How to Parse this Json String using JObject - json

[
["Sender", "service#mydomain.com"],
["Date", "Sat, 19 Dec 201520:41:31 +0000"],
["X-Mailgun-Sid", "WyI0ZjRjNyIsICJyYWplZXZrbXh4eddHh4eDMzMzMzQHlhaG9vLmNvbSIsICJjNGExZiJd"],
["Received", "by luna.mailgun.net with HTTP; Sat, 19 Dec 2015 20:41:31+0000"],
["Message-Id", "<201512192024131.73374.12565#mydomain.com>"],
["Reply-To", "junky01#hotmail.com"],
["X-Mailgun-Skip-Verification", "false"],
["To", "John Myers <testxxxxxx33333#yahoo.com>"], ["From", "\"Inc.\" <service#mydomain.com>"],
["Subject", "Test subject"],
["Mime-Version", "1.0"],
["Content-Type",
["multipart/mixed", { "boundary": "e43d638b70f04a40889d14f4c8422953" } ]
]
]
When using JObject (VB.net), JObject.parse throws this error:
Error reading JObject from JsonReader. Current JsonReader item is not an object: StartArray. Path '', line 2, position 2.---- at Newtonsoft.Json.Linq.JObject.Load(JsonReader reader) at Newtonsoft.Json.Linq.JObject.Parse(String json)
But when I copy the above string into online Json viewers, they all seem to be parsing it fine.

Your JSON does not represent an object; it is an array (of arrays). Therefore, you cannot use JObject.Parse(). Instead, use JArray.Parse(), which can handle arrays, or use JToken.Parse(), which can handle either arrays or objects.
string json = #"
[
[""Sender"", ""service#mydomain.com""],
[""Date"", ""Sat, 19 Dec 201520:41:31 +0000""],
[""X-Mailgun-Sid"", ""WyI0ZjRjNyIsICJyYWplZXZrbXh4eddHh4eDMzMzMzQHlhaG9vLmNvbSIsICJjNGExZiJd""],
[""Received"", ""by luna.mailgun.net with HTTP; Sat, 19 Dec 2015 20:41:31+0000""],
[""Message-Id"", ""<201512192024131.73374.12565#mydomain.com>""],
[""Reply-To"", ""junky01#hotmail.com""],
[""X-Mailgun-Skip-Verification"", ""false""],
[""To"", ""John Myers <testxxxxxx33333#yahoo.com>""],
[""From"", ""\""Inc.\"" <service#mydomain.com>""],
[""Subject"", ""Test subject""],
[""Mime-Version"", ""1.0""],
[""Content-Type"",
[""multipart/mixed"", { ""boundary"": ""e43d638b70f04a40889d14f4c8422953"" } ]
]
]";
JArray rows = JArray.Parse(json);
foreach (JArray row in rows)
{
Console.WriteLine(string.Join(": ", row.Select(r => r.ToString())));
}
Fiddle: https://dotnetfiddle.net/VRs47S

Related

Table from nested list, struct

I have this json data:
consumption_json = """
{
"count": 48,
"next": null,
"previous": null,
"results": [
{
"consumption": 0.063,
"interval_start": "2018-05-19T00:30:00+0100",
"interval_end": "2018-05-19T01:00:00+0100"
},
{
"consumption": 0.071,
"interval_start": "2018-05-19T00:00:00+0100",
"interval_end": "2018-05-19T00:30:00+0100"
},
{
"consumption": 0.073,
"interval_start": "2018-05-18T23:30:00+0100",
"interval_end": "2018-05-18T00:00:00+0100"
}
]
}
"""
and I would like to covert the results list to an Arrow table.
I have managed this by first converting it to python data structure, using python's json library, and then converting that to an Arrow table.
import json
consumption_python = json.loads(consumption_json)
results = consumption_python['results']
table = pa.Table.from_pylist(results)
print(table)
pyarrow.Table
consumption: double
interval_start: string
interval_end: string
----
consumption: [[0.063,0.071,0.073]]
interval_start: [["2018-05-19T00:30:00+0100","2018-05-19T00:00:00+0100","2018-05-18T23:30:00+0100"]]
interval_end: [["2018-05-19T01:00:00+0100","2018-05-19T00:30:00+0100","2018-05-18T00:00:00+0100"]]
But, for reasons of performance, I'd rather just use pyarrow exclusively for this.
I can use pyarrow's json reader to make a table.
reader = pa.BufferReader(bytes(consumption_json, encoding='ascii'))
table_from_reader = pa.json.read_json(reader)
And 'results' is a struct nested inside a list. (Actually, everything seems to be nested).
print(table_from_reader['results'].type)
list<item: struct<consumption: double, interval_start: timestamp[s], interval_end: timestamp[s]>>
How do I turn this into a table directly?
following this https://stackoverflow.com/a/72880717/3617057
I can get closer...
import pyarrow.compute as pc
flat = pc.list_flatten(table_from_reader["results"])
print(flat)
[
-- is_valid: all not null
-- child 0 type: double
[
0.063,
0.071,
0.073
]
-- child 1 type: timestamp[s]
[
2018-05-18 23:30:00,
2018-05-18 23:00:00,
2018-05-18 22:30:00
]
-- child 2 type: timestamp[s]
[
2018-05-19 00:00:00,
2018-05-18 23:30:00,
2018-05-17 23:00:00
]
]
flat is a ChunkedArray whose underlying arrays are StructArray. To convert it to a table, you need to convert each chunks to a RecordBatch and concatenate them in a table:
pa.Table.from_batches(
[
pa.RecordBatch.from_struct_array(s)
for s in flat.iterchunks()
]
)
If flat is just a StructArray (not a ChunkedArray), you can call:
pa.Table.from_batches(
[
pa.RecordBatch.from_struct_array(flat)
]
)

Django - Access Json

I have json like this below
{
"id": 1,
"interviewer": "hengtw1",
"incidenttwg1": {
"id": 5,
"child_occupation": [
6
],
},
}
How can i access child_occupation array. All i tried is incidenttwg1['child_occupation'] or ['incidenttwg1']['child_occupation']. Anyway its still doesn't work.
Any Help?? Thanks....
check this if your string is valid json in python
and
refer this to know more about json encoder and decoder
import json
# Decoding json
data = json.loads({"id": 1,"interviewer": "hengtw1","incidenttwg1": {"id": 5,"child_occupation": [6]}})
print(data["incidenttwg1"]["child_occupation"])
# this will print [6] (list)
print(data["incidenttwg1"]["child_occupation"][0])
# this will print 6 (list item)

How to fetch an attribute value from a variable, having the content of a JSON response

I'm using the Robot Framework API automation. Here, storing the JSON response in a variable [POSTResp.content]. I.e., "POSTResp.content" has the whole response, as given below. Please help me to get an attribute's value (for ex, value of referenceId) from the stored content.
Example of JSON response:
{
"serviceResponseHeader": {
"responseContext": {
"responseCode": "MS19",
"responseDescription": "Success",
"serviceResponseTimeInGMT": "18 Sep 2018 16:12:43 GMT"
},
"requesterContext": {
"applicationCode": null,
"applicationSubCode": null,
"countryCode": null,
"requesterReferenceNumber": null,
"requestTimeInGMT": "30 Jun 2015 11:54:49 GMT",
"requesterUserIdentity": "23483",
"requesterGroupIdentity": "1620",
"requesterIpAddress": "",
"sessionIdentity": "2536kjhfdashfkhfsab",
"ssoSessionIdentity": "2536kjhfdashfkhfsab",
"requesterAbbreviatedGroupName": "NEWCOMP"
},
"serviceContext": {
"serviceVersionNumber": "1.0",
"serviceCode": "30"
}
},
"getProxyDetailResponseBody": {
"proxyDetails": {
"proxyType": "",
"proxyValue": "20140005K",
"referenceId": "PR18090000847597",
"transactionId": "18091801657466"
}
}
}
I've tried the below ways,
1) ${json} To JSON ${POSTResp.content} true
log to console \n the Proxy ID is ${json["proxyValue"]}
Result: Resolving variable '${json["proxyValue"]}' failed: TypeError: string indices must be integers, not str
2) ${json} Evaluate json.loads(${POSTResp.content}} json
log to console \n the Proxy ID is ${json["proxyValue"]}
Result: failed: SyntaxError: unexpected EOF while parsing (, line 1)
Issues with your two approaches:
1) the library keyword call passes a true argument (well, truth-like) to the pretty_print parameter:
${json} To JSON ${POSTResp.content} true
Looking at the library's source, in that case the keyword does not return a dict object - but a string, a beatified version of the source json. That coincides with the error your received.
Remove the "true" argument and it must return a dict.
2) In the Evaluate surround the variable with triple quotes (python's literal string):
${json} Evaluate json.loads('''${POSTResp.content}'''}
json
Without it, the framework just dumped the variable's value, which raised a python syntax error.
By the way, try not to make your variables with language keywords/library names - like ${json} up there.

F# JSON Type Provider, do not serialize null values

Background
I am using the FSharp.Data JSON Type Provider with a sample that has an array of objects that may have different properties. Here is an illustrative example:
[<Literal>]
let sample = """
{ "input": [
{ "name": "Mickey" },
{ "year": 1928 }
]
}
"""
type InputTypes = JsonProvider< sample >
The JSON Type Provider creates an Input type which has both an Optional Name and an Optional Year property. That works well.
Problem
When I try to pass an instance of this to the web service, I do something like this:
InputTypes.Root(
[|
InputTypes.Input(Some("Mouse"), None)
InputTypes.Input(None, Some(2028))
|]
)
The web service is receiving the following and choking on the nulls.
{
"input": [
{
"name": "Mouse",
"year": null
},
{
"name": null,
"year": 2028
}
]
}
What I Tried
I find that this works:
InputTypes.Root(
[|
InputTypes.Input(JsonValue.Parse("""{ "name": "Mouse" }"""))
InputTypes.Input(JsonValue.Parse("""{ "year": 2028 }"""))
|]
)
It sends this:
{
"input": [
{
"name": "Mouse"
},
{
"year": 2028
}
]
}
However, on my real project, the structures are larger and would require a lot more conditional JSON string building. It kind of defeats the purpose.
Questions
Is there a way to cause the JSON Type Provider to not serialize null properties?
Is there a way to cause the JSON Type Provider to not serialize empty arrays?
As a point of comparison, the Newtonsoft.JSON library has a NullValueHandling attribute.
I don't think there is an easy way to get the JSON formatting in F# Data to drop the null fields - I think the type does not clearly distinguish between what is null and what is missing.
You can fix that by writing a helper function to drop all null fields:
let rec dropNullFields = function
| JsonValue.Record flds ->
flds
|> Array.choose (fun (k, v) ->
if v = JsonValue.Null then None else
Some(k, dropNullFields v) )
|> JsonValue.Record
| JsonValue.Array arr ->
arr |> Array.map dropNullFields |> JsonValue.Array
| json -> json
Now you can do the following and get the desired result:
let json =
InputTypes.Root(
[|
InputTypes.Input(Some("Mouse"), None)
InputTypes.Input(None, Some(2028))
|]
)
json.JsonValue |> dropNullFields |> sprintf "%O"

How to use MapReduce to parse a Json file?

Im new to json format. And Im trying to learn how to parse a Json file and pick up the data in it using a MapReduce programming model. Is there any Json Parser that can read multiple lines in records.
Here is my possible & maximum number of elements that be present in my Json file:
{
"type": "",
"format": "",
"version": "",
"id": "",
"start": "",
"cp": "",
message:{ "proto": "","protoVer": "","cliIP": "","reqPort": "","reqHost": "","reqMethod": "","reqPath": "","reqQuery": "","reqCT": "","reqLen": "","sslVer": "","status": "","redirURL": "","respCT": "","respLen": "","bytes": "","UA": "","fwdHost":},
reqHdr:{"accEnc": "","accLang": "","auth": "","cacheCtl": "","conn": "","contMD5": "","cookie": "","DNT": "","expect": "","ifMatch": "","ifMod": "","ifNone": "","ifRange": "","ifUnmod": "","range": "","referer": "","te": "","upgrade": "","via": "","xFrwdFor": "","xReqWith": ""},
"respHdr": {"accRange": "","allowOrigin": "","age": "","allow": "","cacheCtl": "","conn": "","contEnc": "","contLang": "","contMD5": "","contDisp": "","contRange": "","date": "","eTag": "","expires": "","lastMod": "","link": "","p3p": "","retry": "","server": "","trailer": "","transEnc": "","vary": "","via": "","warning": "","wwwAuth": "","xPwrdBy": "","setCookie": ""},
"netPerf": {"downloadTime": "","originName": "","originIP": "","originInitIP": "","originRetry": "","lastMileRTT": "","midMileLatency": "","netOriginLatency": "","lastMileBW": "","cacheStatus": "","firstByte": "","lastByte": "","asnum": "","network": "","netType": "","edgeIP": ""},
"geo": {"country": "","region": "","city": ""},
"waf" : {"logVer" : "1.0","ipRules" : "","appRules" : "","warn" : "","deny" : ""},
"content": {"custom_name": "custom_value"},
}
These are my sample values in the json file.
`{"type":"cloud_monitor","format":"default","version":"1.0","id":"71101cb85441995d11a43bb","start":"1413585245.921","cp":"254623","message":{"proto":"http","protoVer":"1.1","status":"403","cliIP":"23.79.231.14","reqPort":"80","reqHost":"ksd.metareactor.com","reqMethod":"GET","reqPath":"%2findex.php","reqQuery":"path%3d57%26product_id%3d49%26route%3d%255Cwinnt%255Cwin.ini%2500.","respCT":"text/html","respLen":"286","bytes":"286","UA":"mozilla-saturn","fwdHost":"origin-demo2-akamaized.scoe-sil.net"}`,
"reqHdr":{"accEnc":"gzip,%20deflate","cookie":"PHPSESSID%3dkkqoodvfe0rt9l7lbvqghk6e15%3bcurrency%3dUSD%3blanguage%3den"},"netPerf":{"downloadTime":"8","lastMileRTT":"20","cacheStatus":"0","firstByte":"1","lastByte":"1","asnum":"12222","edgeIP":"184.28.16.109"},"geo":{"country":"US","region":"CA","city":"SANFRANCISCO","lat":"37.7795","long":"-122.4195"},"network":{"edgeIP":"184.28.16.109","asnum":"12222","network":"","networkType":""},"waf":{"ver":"2.0","policy":"qik1_12418","ruleSet":"KRS%201.0","mode":"scr","rsr":"1","dor":"0","oft":"0","riskGroups":":INBOUND-ANOMALY","riskTuples":":-3000002","riskScores":":-1000","pAction":"","pRate":"","warnRules":"3000002","warnSlrs":"ARGS%3aroute","warnData":"d2lubnQvd2luLmluaQ%3d%3d","warnTags":"AKAMAI%2fWEB_ATTACK%2fFILE_INJECTION","denyRules":"INBOUND-ANOMALY","denyData":"U2NvcmU6IDEwMDAsIERFTlkgdGhyZXNob2xkOiAyNSwgQWxlcnQgUnVsZXM6IDMwMDAwMDIsIERlbnkgUnVsZTogLCBMYXN0IE1hdGNoZWQgTWVzc2FnZTogTG9jYWwgU3lzdGVtIEZpbGUgQWNjZXNzIEF0dGVtcHQ%3d"}}
I have a Java Json Parser but I can use it to read a single line. How can I identify a multiple line record in a Json file and use it in the MapReduce code to extract the data.
My Json parser class:
String[] tuple = value.toString().split("\n");
try {
for(int i=0; i<tuple.length; i++) {
JSONObject jsonobj = new JSONObject(tuple[i]);
type = (String) jsonobj.get("type");
format = (String) jsonobj.get("format");
version = (String) jsonobj.get("version");
id = (String) jsonobj.get("id");
start = (String) jsonobj.get("start");
cp = (String) jsonobj.get("cp");
message = (String) jsonobj.get("message");
}
} catch (JSONException e) {
e.printStackTrace();
}
Can anyone help me on writing code to read Json records which are in multiple lines in MapReduce ?