See attached screenshot, when processing the messages in a user's folder, when the message is an "eventMessageRequest" then the json is invalid: it contains the "type" key twice and the Json parsers can't deal with it.
screenshot of original json received from MS Graph
abbreviated json: note the "type" key used twice:
[
{
"type": "#microsoft.graph.eventMessageRequest",
"createdDateTime": "2020-07-01T18:21:32Z",
"lastModifiedDateTime": "2020-07-01T18:21:33Z",
"categories": [
],
"receivedDateTime": "2020-07-01T18:21:33Z",
"sentDateTime": "2020-07-01T18:21:29Z",
"hasAttachments": true,
"isDeliveryReceiptRequested": null,
"isReadReceiptRequested": false,
"isRead": true,
"isDraft": false,
"inferenceClassification": "focused",
"meetingMessageType": "meetingRequest",
"type": "singleInstance",
"isOutOfDate": false,
"isAllDay": false,
"isDelegated": false,
"responseRequested": true,
"allowNewTimeProposals": null,
"meetingRequestType": "informationalUpdate"
}
]
Related
Im trying to input a JSON file to GCP Bigquery, but bumping into multiple errors...
I think it is because it is in NOSQL format.
As below the JSON file is in a hashmap the takes multiple hashmaps as the value.
example:
[
{
"value1": {"able" : false, "have" : true, "date": "2022-01-01"},
"value2": {"done" : true, "time" : "12:31"}
}]
If it is hard to input data like this to Bigquery would it be possible to insert table if I change the schema as below?
[
{
"value1": {"able" : false, "have" : true, "done" : Null, "date": "2022-01-01"},
"value2": {"able" : Null, "have" : Null,"done" : true, "date" : Null}
}]
Tried GCP transform with and without a schema.
I'm getting a JSON output from the VirusTotal API, as seen below:
{
"scan_id": "id_goes_here",
"resource": "domain_goes_here",
"url": "http://example.com/",
"response_code": 1,
"scan_date": "2021-01-19 21:28:32",
"permalink": "https://www.virustotal.com/gui/url/id_goes_here/detection/u-id_goes_here",
"verbose_msg": "Scan finished, scan information embedded in this object",
"filescan_id": null,
"positives": 0,
"total": 83,
"scans": {
"CMC Threat Intelligence": {
"detected": false,
"result": "clean site"
},
"CLEAN MX": {
"detected": false,
"result": "clean site"
},
"DNS8": {
"detected": false,
"result": "clean site"
},
"MalwareDomainList": {
"detected": false,
"result": "clean site",
"detail": "http://www.malwaredomainlist.com/mdl.php?search=example.com"
...
I want to go through this JSON and find the entries that have detected as true and if so print the name. As well, print the reference (if there is one)
For example, let's say the entry for MalwareDomainList had detected set to true. So as below:
"MalwareDomainList": {
"detected": true,
"result": "malicious site",
"detail": "http://www.malwaredomainlist.com/mdl.php?search=example.com"
I want to print the name of the list, so in this case, MalwareDomainList and also print the value of detail. As you can see in the above output, not all lists have a detail field.
In python3, I am thinking I need to have something like:
scan_content = json.dumps(response.json(), indent =4 )
for scan in scan_content["scans"]:
if scan["detected"]:
print("Name of blacklist:", scan["name"])
But getting an error since name is not an actual key in the output. Not sure how to alter my above code to achieve the following:
If detected is set to true
Print the name of the list
If applicable, print value in the detail key
Trying the below solution as such:
I have the following:
json_data = json.dumps(response.json(), indent =4 )
scan_content = json_data["scans"]
positive = []
for elem in scan_content.keys():
if scan_content[elem]["detected"] == "true":
positive.append({"Name" : elem, "Detail" : scan_content[elem]["detail"]})
print(*positive)
I get the error
scan_content = json_data["scans"]
TypeError: string indices must be integers
I am having a issue with applying a App Service Authentication to my Web App Slots.
The error i am receiving is the following:
"The template resource 'webapptest1a/authconfig' for type 'Microsoft.WindowsAzure.ResourceStack.Frontdoor.Common.Entities.TemplateGenericProperty`1[System.String]' at line '1' and column '8107' has incorrect segment lengths. A nested resource type must have identical number of segments as its resource name. A root resource type must have segment length one greater than its resource name"
Here is my code, which i think is pretty much correct. I am finding it difficult to find references for Web App slots config. I have the Microsoft Documentation and i followed it, but no luck.
Here is my code:
{
"type": "Microsoft.Web/sites/slots/config",
"name": "[concat(parameters('webAppName'),'/authconfig')]",
"apiVersion": "2018-11-01",
"location": "[resourceGroup().location]",
"dependsOn": [
"[parameters('webAppName')]",
"[concat(parameters('sqlDatabase'), 'constr')]"
],
"properties": {
"enabled": true,
"runtimeVersion": "~1",
"unauthenticatedClientAction": "RedirectToLoginPage",
"tokenStoreEnabled": true,
"allowedExternalRedirectUrls": null,
"defaultProvider": "AzureActiveDirectory",
"clientId": null,
"clientSecret": null,
"clientSecretCertificateThumbprint": null,
"issuer": null,
"allowedAudiences": [
"https://webapptest1a-staging.azurewebsites.net"
],
"additionalLoginParams": null,
"isAadAutoProvisioned": false,
"googleClientId": null,
"googleClientSecret": null,
"googleOAuthScopes": null,
"facebookAppId": null,
"facebookAppSecret": null,
"facebookOAuthScopes": [
""
],
"twitterConsumerKey": null,
"twitterConsumerSecret": null,
"microsoftAccountClientId": null,
"microsoftAccountClientSecret": null,
"microsoftAccountOAuthScopes": [
""
]
}
},
I am really concussed, i have tried many variants but i am not getting close.
I changed the name to few different variants then i was given different errors but in regards to the naming convention.
"name": "[concat(parameters('webAppName'), '/appsettings')]",
I also changed the Depends on twice from:
"[parameters('webAppName')]",
"[concat(parameters('sqlDatabase'), 'constr')]"
To:
"[concat('Microsoft.Web/sites/', parameters('webAppName'))]",
"[concat(parameters('sqlDatabase'), 'constr')]"
I am really stuck! Would love some guidance.
Thank you
As the error says that "A root level resource must have one less segment in the name than the resource type". Here you are passing the incorrect name for the resource. As the segment length of Type is 4, segment length of Name must be 3. So in the config name you must pass the slot name also something like below (You can change the slot name and config name according to your template)
[concat(parameters('webAppName'), '/staging/web')]
Please check the below example for reference:
{
"type": "Microsoft.Web/sites/slots/config",
"apiVersion": "2018-11-01",
"name": "[concat(parameters('webAppName'), '/staging/web')]",
"location": "East US",
"dependsOn": [
"[resourceId('Microsoft.Web/sites/slots', parameters('webAppName'), 'staging')]",
"[resourceId('Microsoft.Web/sites', parameters('webAppName'))]"
],
"properties": {
"numberOfWorkers": 1,
"defaultDocuments": [
"Default.htm",
"Default.html",
"Default.asp",
"index.htm",
"index.html",
"iisstart.htm",
"default.aspx",
"index.php",
"hostingstart.html"
],
"netFrameworkVersion": "v4.0",
"requestTracingEnabled": false,
"remoteDebuggingEnabled": false,
"remoteDebuggingVersion": "VS2019",
"httpLoggingEnabled": false,
"logsDirectorySizeLimit": 35,
"detailedErrorLoggingEnabled": false,
"publishingUsername": "$mytestap345__staging",
"scmType": "None",
"use32BitWorkerProcess": true,
"webSocketsEnabled": false,
"alwaysOn": false,
"managedPipelineMode": "Integrated",
"virtualApplications": [
{
"virtualPath": "/",
"physicalPath": "site\\wwwroot",
"preloadEnabled": false
}
],
"loadBalancing": "LeastRequests",
"experiments": {
"rampUpRules": []
},
"autoHealEnabled": false,
"localMySqlEnabled": false,
"ipSecurityRestrictions": [
{
"ipAddress": "Any",
"action": "Allow",
"priority": 1,
"name": "Allow all",
"description": "Allow all access"
}
],
"scmIpSecurityRestrictions": [
{
"ipAddress": "Any",
"action": "Allow",
"priority": 1,
"name": "Allow all",
"description": "Allow all access"
}
],
"scmIpSecurityRestrictionsUseMain": false,
"http20Enabled": false,
"minTlsVersion": "1.2",
"ftpsState": "AllAllowed",
"reservedInstanceCount": 0
}
}
I have build a small microservice using SpringBoot2 and Spring 5 which has a REST service exposed (HTTP GET Method) and which internally consumes another REST GET service (third party API). Using Postman when I call my service (GET), then I get a JSON response but the problem is I get a complete whole object in response like below :-
[
{
"id": "1",
"name": "Open Catalogue",
"subcategories": [
{
"id": "106",
**"name": "Components",**
"subcategories": [
{
"id": "816",
"name": "Power Supplies",
"subcategories": [
{
"id": "814",
"name": "Rechargeable Batteries",
"subcategories": [],
"sample": {
"empty": true,
"lazy": false,
"async": false
}
},
{
"id": "829",
"name": "Battery Chargers",
"subcategories": [],
"samples": {
"empty": true,
"lazy": false,
"async": false
}
},
My service URL used in post man is like this :-
http://localhost:8080/test-search?searchKey=ball
So my requirement is whenever a user consume this service by a sub-category name then only that sub-category details along with its immediate child details should be returned and not the child of child.
Here searchKey in URL is nothing but a free text to search for a sub-category. For instance when I say :-
http://localhost:8080/test-search?searchKey=Components
then only below details should be returned like this :-
"id": "106",
"name": "Components",
"subcategories": [
{
"id": "816",
"name": "Power Supplies",
Response should not have sub-categories of Power Supplies i.e. "subcategories": [ "id": "814",
"name": "Rechargeable Batteries",
Is there any efficient way to do the filtering while preparing the JSON response or first fetch whole object and then start filtering?
Please advise, thank you
EDIT 2014-05-01: I tried fromJSON first (as suggested below), but that only parsed the first line. I found out that there were commas missing between the brackets of each JSON line so I changed that in TextEdit and saved the file. I also added [ at the beginning of the file and ] at the end and then it worked with JSON. Now the next step: from a list (with embedded lists) to a dataframe (or csv).
I get a data package from edX every now and then on the courses we are evaluating. Some of these are just plain .csv files which are quite easy to handle, others are more difficult for me (not having a CS or programming background).
I have 2 files I want to open and parse into csv files for analysis in R. I have tried many many json2csv tools out there, but to no avail. I also tried the simple methods described here to turn json into csv.
The data is confidential, so I cannot share the entire data set, but will share the first two lines of the file, maybe that helps. The problem is that nowhere I find anything about .mongo files, which to me seems quite strange, do they even exist? Or is this just a JSON file that may be corrupted (which could explain the errors)?
Any suggestions are welcome.
The first 2 lines in one of the .mongo files:
{
"_id": {
"$oid": "52d1e62c350e7a3156000009"
},
"votes": {
"up": [
],
"down": [
],
"up_count": 0,
"down_count": 0,
"count": 0,
"point": 0
},
"visible": true,
"abuse_flaggers": [
],
"historical_abuse_flaggers": [
],
"parent_ids": [
],
"at_position_list": [
],
"body": "the delft university accredited course with the scholarship (fundamentals of water treatment) is supposed to start in about a month's time. But have the scholarship list been published? Any tentative date??",
"course_id": "DelftX/CTB3365x/2013_Fall",
"_type": "Comment",
"endorsed": false,
"anonymous": false,
"anonymous_to_peers": false,
"author_id": "269835",
"comment_thread_id": {
"$oid": "52cd40c5ab40cf347e00008d"
},
"author_username": "tachak59",
"sk": "52d1e62c350e7a3156000009",
"updated_at": {
"$date": 1389487660636
},
"created_at": {
"$date": 1389487660636
}
}{
"_id": {
"$oid": "52d0a66bcb3eee318d000012"
},
"votes": {
"up": [
],
"down": [
],
"up_count": 0,
"down_count": 0,
"count": 0,
"point": 0
},
"visible": true,
"abuse_flaggers": [
],
"historical_abuse_flaggers": [
],
"parent_ids": [
{
"$oid": "52c63278100c07c0d1000028"
}
],
"at_position_list": [
],
"body": "I got it. Thank you!",
"course_id": "DelftX/CTB3365x/2013_Fall",
"_type": "Comment",
"endorsed": false,
"anonymous": false,
"anonymous_to_peers": false,
"parent_id": {
"$oid": "52c63278100c07c0d1000028"
},
"author_id": "2655027",
"comment_thread_id": {
"$oid": "52c4f303b03c4aba51000013"
},
"author_username": "dmoronta",
"sk": "52c63278100c07c0d1000028-52d0a66bcb3eee318d000012",
"updated_at": {
"$date": 1389405803386
},
"created_at": {
"$date": 1389405803386
}
}{
"_id": {
"$oid": "52ceea0cada002b72c000059"
},
"votes": {
"up": [
],
"down": [
],
"up_count": 0,
"down_count": 0,
"count": 0,
"point": 0
},
"visible": true,
"abuse_flaggers": [
],
"historical_abuse_flaggers": [
],
"parent_ids": [
{
"$oid": "5287e8d5906c42f5aa000013"
}
],
"at_position_list": [
],
"body": "if u please send by mail \n",
"course_id": "DelftX/CTB3365x/2013_Fall",
"_type": "Comment",
"endorsed": false,
"anonymous": false,
"anonymous_to_peers": false,
"parent_id": {
"$oid": "5287e8d5906c42f5aa000013"
},
"author_id": "2276302",
"comment_thread_id": {
"$oid": "528674d784179607d0000011"
},
"author_username": "totah1993",
"sk": "5287e8d5906c42f5aa000013-52ceea0cada002b72c000059",
"updated_at": {
"$date": 1389292044203
},
"created_at": {
"$date": 1389292044203
}
}
R doesn't have "native" support for these files but there is a JSON parser with the rjson package. So I might load my .mongo file with:
myfile <- "path/to/myfile.mongo"
myJSON <- readLines(myfile)
myNiceData <- fromJSON(myJSON)
Since RJson converts into a data structure that fits the object being read, you'll have to do some additional snooping but once you have an R data type you shouldn't have any trouble working with it from there.
Another package to consider when parsing JSON data is jsonlite. It will make data frames for you so you can write them to a csv format with write.table or some other applicable method for writing objects.
NOTE: if it is easier to connect to the MongoDB and get the data from a request, then RMongo may be a good bet. The R-Bloggers also made a post about using RMongo that has a nice little walkthrough.
I used RJSON as suggested by #theWanderer and with the help of a colleague wrote the following code to parse the data into columns, choosing the specific columns that are needed, and checking each of the instances if they return the right variables.
Entire workflow:
Checked some of the data in jsonlint - corrected the errors → },{ instead of }{ between each line and [ and ] at the beginning and end of the file
Made a smaller file to play with, containing about 11 JSON lines
Used the code below to parse the datafile - however, checking the different listItems first if they are not lists themselves (that gives problems) // as you will see, I also removed things like \n because that gave errors and added an empty value for parent_id if there is none in the data (otherwise it would mix up the data)
The code to import the .mongo file into R and then parse it into CSV:
library(rjson)
###### set working directory to write out the data file
setwd("/your/favourite/dir/json to csv/")
#never ever convert strings to factors
options(stringsAsFactors = FALSE)
#import the .mongo file to R
temp.data = fromJSON(file="temp.mongo", method="C", unexpected.escape="error")
file.remove("temp.csv") ## removes the old datafile if there is one
## (so the data is not appended to the file,
## but a new file is created)
listItem = temp.data[[1]] ## prepare the listItem the first time
for (listItem in temp.data){
parent_id = ""
if (length(listItem$parent_id)>0){
parent_id = listItem$parent_id
}
write.table(t(c(
listItem$votes$up_count, listItem$visible, parent_id,
gsub("\n", "", listItem$body), listItem$course_id, unlist(listItem["_type"]),
listItem$endorsed, listItem$anonymous, listItem$author_id,
unlist(listItem$comment_thread_id), listItem$author_username,
as.POSIXct(unlist(listItem$created_at)/1000, origin="1970-01-01"))), # end t(), c()
file="temp.csv", sep="\t", append=TRUE, row.names=FALSE, col.names=FALSE)
}