SoapUI JSON Parsing - Get Parent Node using Child value - json

I have a JSON response like below from SoapUI.
{
"-1": {
"startDate": "",
"modifiedBy": "",
"endDate": "",
"projectId": 1,
"build": "",
"projectKey": "TEST",
"started": ""
},
"0": {
"startDate": "",
"modifiedBy": "",
"endDate": "",
"projectId": 2,
"build": "",
"projectKey": "BEST",
"started": ""
},
"2": {
"startDate": "",
"modifiedBy": "",
"endDate": "",
"projectId": 3,
"build": "",
"projectKey": "WORST",
"started": ""
}
}
My requirement is, I have to get the value/node "0" which has the projectkey="BEST" using JsonSurpler or Groovy Script Test Step. The projectkey is now under the node "0". Maybe it will be under "10" or "1000" or "-500".
How to get the parent node using a child node value?

There are different ways to achieve that. These are the simplest I can recollect now:
import groovy.json.JsonSlurper
String string = '''{
"-1": {
"startDate": "",
"modifiedBy": "",
"endDate": "",
"projectId": 1,
"build": "",
"projectKey": "TEST",
"started": ""
},
"0":{
"startDate": "",
"modifiedBy": "",
"endDate": "",
"projectId": 2,
"build": "",
"projectKey": "BEST",
"started": ""
},
"2": {
"startDate": "",
"modifiedBy": "",
"endDate": "",
"projectId": 3,
"build": "",
"projectKey": "WORST",
"started": ""
}
}'''
def json = new JsonSlurper().parseText(string)
assert json.find { it.value?.projectKey == 'BEST' }?.key == '0'
assert json.findResult { k, v -> v?.projectKey == 'BEST' ? k : null } == '0'
This will find the first item which satisfies the condition. If you need all the items which satisfy the condition then use findAll or findResults accordingly.

Related

Accessing specific JSON values in a deluge script

I have a JSON API response that contains multiple entries, with different types of subscriptions and multiple users.
I need to search the list for a "user_name" AND a "subscription", then return any matching "duration". In some cases, there will be more than one "duration" for a user and subscription. I would need the total (sum) of the duration when there is more than one.
For example, here is a part of an example Json I am working with:
[
{
"id": 139387026,
"user_name": "John Smith",
"note": "",
"last_modify": "2022-03-28 14:16:35",
"date": "2022-03-28",
"locked": "0",
"addons_external_id": "",
"description": "",
"info": [
{
"subscription": "basic",
"duration": "22016",
}
]
},
{
"id": 139387027,
"user_name": "John Smith",
"note": "",
"last_modify": "2022-03-28 14:16:35",
"date": "2022-03-28",
"locked": "0",
"addons_external_id": "",
"description": "",
"info": [
{
"subscription": "advanced",
"duration": "10537",
}
]
},
{
"id": 139387028,
"user_name": "Martin Lock",
"note": "",
"last_modify": "2022-03-28 14:16:35",
"date": "2022-03-28",
"locked": "0",
"addons_external_id": "",
"description": "",
"info": [
{
"subscription": "basic",
"duration": "908",
}
]
},
]
So for example, for user_name: "John Smith" and subscription: "advanced", I need to return duration: "10537".
I've used toJsonlist(); to convert it, then used the code below, but it returns all values in the list. I can't figure out how to search for the specific values or add matching entries together.
rows = subscriptions.toJsonlist();
for each row in rows
{
info row;
user_name = row.getJson("user_name");
info "username: " + user_name;
subscription = row.getJson("subscription");
info "subscription: " + subscription;
subscriptionId = row.getJson("subscriptionId");
info "subscription Id: " + subscriptionId;
}
I'm fairly new to programming. Any help is appreciated!
According to your needs , you want to filter your JSON data and get the corresponding value from your filter in user_name and subcription.
Here is the Deluge Script for that. I use clear variable name so that it will not confused you.
//Your Entry Change this based on your filter
input_user_name = "John Smith";
input_subscription = "advanced";
//Your JSON data
json_string_data = '[ { "id": 139387026, "user_name": "John Smith", "note": "", "last_modify": "2022-03-28 14:16:35", "date": "2022-03-28", "locked": "0", "addons_external_id": "", "description": "", "info": [ { "subscription": "basic", "duration": "22016", } ] }, { "id": 139387027, "user_name": "John Smith", "note": "", "last_modify": "2022-03-28 14:16:35", "date": "2022-03-28", "locked": "0", "addons_external_id": "", "description": "", "info": [ { "subscription": "advanced", "duration": "10537", } ] }, { "id": 139387028, "user_name": "Martin Lock", "note": "", "last_modify": "2022-03-28 14:16:35", "date": "2022-03-28", "locked": "0", "addons_external_id": "", "description": "", "info": [ { "subscription": "basic", "duration": "908", } ] } ]';
//Declare the data as JSON
processed_json_data = json_string_data.toJsonlist();
initial_total_duration = 0;//Donot change this
list_of_duration = List();
total_duration_per_username_per_subscription = Map();
for each row in processed_json_data
{
if (row.get("user_name") == input_user_name )
{
info_list = row.get("info").toJSONList();
for each info_row in info_list
{
if (info_row.get("subscription") == input_subscription)
{
info_row_duration = info_row.get("duration").toLong(); // make it integer
list_of_duration.add(info_row_duration);
}
}
}
}
result_map = Map();
//Sum of list_of_duration
for each duration in list_of_duration
{
initial_total_duration = initial_total_duration + duration;
}
result_map.put("user_name",input_user_name);
result_map.put("subscription",input_subscription);
result_map.put("no_of_subscription",list_of_duration.size());
result_map.put("total_duration",initial_total_duration);
info result_map;
And the result should be
{"user_name":"John Smith","subscription":"advanced","no_of_subscription":1,"total_duration":10537}
You can test these script in https://deluge.zoho.com/tryout.
Thanks,
Von

Create Table in Athena From Nested JSON

I have nested JSON of type
[{
"emails": [{
"label": "",
"primary": "",
"relationdef_id": "",
"type": "",
"value": ""
}],
"licenses": [{
"allocated": "",
"parent_type": "",
"parentid": "",
"product_type": "",
"purchased_license_id": "",
"service_type": ""
}, {
"allocated": "",
"parent_type": "",
"parentid": "",
"product_type": "",
"purchased_license_id": "",
"service_type": ""
}]
}, {
"emails": [{
"label": "",
"primary": "",
"relationdef_id": "",
"type": "",
"value": ""
}],
"licenses": [{
"allocated": "2016-04-26 01:46:26",
"parent_type": "",
"parentid": "",
"product_type": "",
"purchased_license_id": "",
"service_type": ""
}]
}]
which is not able to be converted to athena table.
I have tried to update it to list of objects also
{
"emails": [{
"label": "",
"primary": "",
"relationdef_id": "",
"type": "",
"value": ""
}
],
"licenses": [{
"allocated": "",
"parent_type": "",
"parentid": "",
"product_type": "",
"purchased_license_id": "",
"service_type": ""
},{
"allocated": "",
"parent_type": "",
"parentid": "",
"product_type": "",
"purchased_license_id": "",
"service_type": ""
}
]
}
{
"emails": [{
"label": "",
"primary": "",
"relationdef_id": "",
"type": "",
"value": ""
}
],
"licenses": [{
"allocated": "",
"parent_type": "",
"parentid": "",
"product_type": "",
"purchased_license_id": "",
"service_type": ""
}
]
}
{
"emails": [{
"label": "",
"primary": "",
"relationdef_id": "",
"type": "",
"value": ""
}
],
"licenses": [{
"allocated": "",
"parent_type": "",
"parentid": "",
"product_type": "",
"purchased_license_id": "",
"service_type": ""
}
]
}
with Query:
CREATE EXTERNAL TABLE `test_orders1`(
`emails` array<struct<`label`: string, `primary`: string,`relationdef_id`: string,`type`: string, `value`: string>>,
`licenses` array<struct<`allocated`: string, `parent_type`: string, `parentid`: string, `product_type`: string,`purchased_license_id`: string, `service_type`: string>>)
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
WITH SERDEPROPERTIES ( 'ignore.malformed.json' = 'true')
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
but only 1 row is formed.
Is there a way where i can use Nested json of type JSONArray into Athena table?
Or how can I change Nested Json that will work for me?
When querying JSON data Athena requires the files to be formatted with one JSON document per line. It's unclear from your question if this is the case or not, the examples you give are multiline, but perhaps that's only to make the question more clear.
The table DDL you include looks like it should work on the second example data, provided that it is formatted as one document per line, e.g.
{"emails": [{"label": "", "primary": "", "relationdef_id": "", "type": "", "value": ""}], "licenses": [{"allocated": "", "parent_type": "", "parentid": "", "product_type": "", "purchased_license_id": "", "service_type": ""}, { "allocated": "", "parent_type": "", "parentid": "", "product_type": "", "purchased_license_id": "", "service_type": ""}]}
{"emails": [{"label": "", "primary": "", "relationdef_id": "", "type": "", "value": ""}], "licenses": [{"allocated": "", "parent_type": "", "parentid": "", "product_type": "", "purchased_license_id": "", "service_type": ""}]}
{"emails": [{"label": "", "primary": "", "relationdef_id": "", "type": "", "value": ""}], "licenses": [{"allocated": "", "parent_type": "", "parentid": "", "product_type": "", "purchased_license_id": "", "service_type": ""}]}

Develop bargraphs from json object list directly

I have multiple json scripts with a similar format as:
{
"name": "xyz",
"slug": "xyz",
"supplier": "xyz.limited",
"attributes": [
{
"name": "mass",
"productConfiguration": "base",
"description": "",
"value": "0.24",
"minimumValue": "",
"maximumValue": "",
"measurementUnit": "kg",
"uuid": "",
"attributeClassUuid": "",
"productUuid": ""
},
{
"name": "length",
"productConfiguration": "base",
"description": "assumed to be length",
"value": "58.0",
"minimumValue": "",
"maximumValue": "",
"measurementUnit": "mm",
"uuid": "",
"attributeClassUuid": "",
"productUuid": ""
},
{
"name": "height",
"productConfiguration": "base",
"description": "assumed to be height",
"value": "25.0",
"minimumValue": "",
"maximumValue": "",
"measurementUnit": "mm",
"uuid": "",
"attributeClassUuid": "",
"productUuid": ""
},
{
"name": "width",
"productConfiguration": "base",
"description": "assumed to be width",
"value": "58.0",
"minimumValue": "",
"maximumValue": "",
"measurementUnit": "mm",
"uuid": "",
"attributeClassUuid": "",
"productUuid": ""
},
{
"name": "momentum-storage",
"productConfiguration": "base",
"description": "",
"value": "0.050",
"minimumValue": "",
"maximumValue": "",
"measurementUnit": "N m s",
"uuid": "",
"attributeClassUuid": "",
"productUuid": ""
},
{
"name": "maximum-torque",
"productConfiguration": "base",
"description": "",
"value": "0.007",
"minimumValue": "",
"maximumValue": "",
"measurementUnit": "N m",
"uuid": "",
"attributeClassUuid": "",
"productUuid": ""
},
{
"name": "voltage",
"productConfiguration": "base",
"description": "voltage is given in DC current",
"value": "12.0",
"minimumValue": "",
"maximumValue": "",
"measurementUnit": "V",
"uuid": "",
"attributeClassUuid": "",
"productUuid": ""
},
{
"name": "maximum-power",
"productConfiguration": "base",
"description": "",
"value": "9.0",
"minimumValue": "",
"maximumValue": "",
"measurementUnit": "W",
"uuid": "",
"attributeClassUuid": "",
"productUuid": ""
},
{
"name": "lifetime",
"productConfiguration": "base",
"description": "",
"value": "",
"minimumValue": "5.0",
"maximumValue": "",
"measurementUnit": "yr",
"uuid": "",
"attributeClassUuid": "",
"productUuid": ""
}
]
}
Each of these files have different set of attributes defined. I would like to plot the number of times each attribute is repeated across all files. I try to do this using python3.5 functionalities 'os' and 'panda.DataFrame'. However, got lost somewhere! Can use some help with this. Thanks in advance!

Issues parsing a 1GB json file using JSON.NET

I have gotten an application where the input has been scaled up from 50K location records to 1.1 Million location records.
This has caused serious issues as the entire file was previously de-serialized into a single object.
The size of the object is ~1GB for a production like file with 1.1 Million records.
Due to large object GC issues I want to keep the de-serialized object below the 85K mark.
I'm trying to parse out a single location object at a time and de-serialize it so I can control the number of objects
that get de-serialized and in turn control the size of the object. I'm using the Json.Net libraries to do this.
Below is a sample of the JSON file that I'm receiving as a stream into my application.
{
"Locations": [{
"LocationId": "",
"ParentLocationId": "",
"DisplayFlag": "Y",
"DisplayOptions": "",
"DisplayName": "",
"Address": "",
"SecondaryAddress": "",
"City": "",
"State": "",
"PostalCode": "",
"Country": "",
"Latitude": 40.59485,
"Longitude": -73.96174,
"LatLonQuality": 99,
"BusinessLogoUrl": "",
"BusinessUrl": "",
"DisplayText": "",
"PhoneNumber": "",
"VenueGroup": 7,
"VenueType": 0,
"SubVenue": 0,
"IndoorFlag": "",
"OperatorDefined": "",
"AccessPoints": [{
"AccessPointId": "",
"MACAddress": "",
"DisplayFlag": "",
"DisplayOptions": "",
"Latitude": 40.59485,
"Longitude": -73.96174,
"Status": "Up",
"OperatorDefined": "",
"RoamingGroups": [{
"GroupName": ""
},
{
"GroupName": ""
}],
"Radios": [{
"RadioId": "",
"RadioFrequency": "",
"RadioProtocols": [{
"Protocol": ""
}],
"WifiConnections": [{
"BSSID": "",
"ServiceSets": [{
"SSID": "",
"SSID_Broadcasted": ""
}]
}]
}]
}]
},
{
"LocationId": "",
"ParentLocationId": "",
"DisplayFlag": "Y",
"DisplayOptions": "",
"DisplayName": "",
"Address": "",
"SecondaryAddress": "",
"City": "",
"State": "",
"PostalCode": "",
"Country": "",
"Latitude": 40.59485,
"Longitude": -73.96174,
"LatLonQuality": 99,
"BusinessLogoUrl": "",
"BusinessUrl": "",
"DisplayText": "",
"PhoneNumber": "",
"VenueGroup": 7,
"VenueType": 0,
"SubVenue": 0,
"IndoorFlag": "",
"OperatorDefined": "",
"AccessPoints": [{
"AccessPointId": "",
"MACAddress": "",
"DisplayFlag": "",
"DisplayOptions": "",
"Latitude": 40.59485,
"Longitude": -73.96174,
"Status": "Up",
"OperatorDefined": "",
"RoamingGroups": [{
"GroupName": ""
},
{
"GroupName": ""
}],
"Radios": [{
"RadioId": "",
"RadioFrequency": "",
"RadioProtocols": [{
"Protocol": ""
}],
"WifiConnections": [{
"BSSID": "",
"ServiceSets": [{
"SSID": "",
"SSID_Broadcasted": ""
}]
}]
}]
}]
}]
}
I need to be able to pull out the individual Location objects, so that I would be looking at the following
{
"LocationId": "",
"ParentLocationId": "",
"DisplayFlag": "Y",
"DisplayOptions": "",
"DisplayName": "",
"Address": "",
"SecondaryAddress": "",
"City": "",
"State": "",
"PostalCode": "",
"Country": "",
"Latitude": 40.59485,
"Longitude": -73.96174,
"LatLonQuality": 99,
"BusinessLogoUrl": "",
"BusinessUrl": "",
"DisplayText": "",
"PhoneNumber": "",
"VenueGroup": 7,
"VenueType": 0,
"SubVenue": 0,
"IndoorFlag": "",
"OperatorDefined": "",
"AccessPoints": [{
"AccessPointId": "",
"MACAddress": "",
"DisplayFlag": "",
"DisplayOptions": "",
"Latitude": 40.59485,
"Longitude": -73.96174,
"Status": "Up",
"OperatorDefined": "",
"RoamingGroups": [{
"GroupName": ""
},
{
"GroupName": ""
}],
"Radios": [{
"RadioId": "",
"RadioFrequency": "",
"RadioProtocols": [{
"Protocol": ""
}],
"WifiConnections": [{
"BSSID": "",
"ServiceSets": [{
"SSID": "",
"SSID_Broadcasted": ""
}]
}]
}]
}]
}
I'm trying to use the Json.NET JsonTextReader to accomplish this, however I cannot get the reader to contain an entire location in its buffer, due to the size of the records in the stream the reader initially will have down as far as "RadioProtocols", which is mid way through the object, by the time the stream reaches the end of the object, the reader has discarded the start of the object.
The code I'm using to try to get this functionality to work is
var ser = new JsonSerializer();
using (var reader = new JsonTextReader(new StreamReader(stream)))
{
reader.SupportMultipleContent = true;
while (reader.Read())
{
if (reader.TokenType == JsonToken.StartObject && reader.Depth == 2)
{
do
{
reader.Read();
} while (reader.TokenType != JsonToken.EndObject && reader.Depth == 2);
var singleLocation = ser.Deserialize<Locations>(reader);
}
}
}
Any information on this or an alternative to doing it would be greatly appreciated. As a side note, the way our customers send the information cannot change at this time.
When the reader is positioned at the beginning of the object you want to deserialize (an entry in the Locations array in your case), you can just call ser.Deserialize<T>(reader) and it will work, advancing to the end of the object at that level, and no further. Thus the following should iterate through the Location objects in your file, loading each one separately:
public static IEnumerable<T> DeserializeNestedItems<T>(TextReader textReader)
{
var ser = new JsonSerializer();
using (var reader = new JsonTextReader(textReader))
{
reader.SupportMultipleContent = true;
while (reader.Read())
{
if (reader.TokenType == JsonToken.StartObject && reader.Depth == 2)
{
var item = ser.Deserialize<T>(reader);
yield return item;
}
}
}
}
And an example of use using your test string:
Debug.Assert(DeserializeNestedItems<Location>(new StringReader(json)).Count() == 2); // No assert.
var list = DeserializeNestedItems<Location>(new StringReader(json)).SelectMany(l => l.AccessPoints).Select(a => new { a.Latitude, a.Longitude }).ToList();
Debug.WriteLine(JsonConvert.SerializeObject(list, Formatting.Indented));
Which outputs:
[
{
"Latitude": 40.59485,
"Longitude": -73.96174
},
{
"Latitude": 40.59485,
"Longitude": -73.96174
}
]
Note - the Location class comes from posting your JSON to http://json2csharp.com/.
Thanks for all the help, I've managed to get it doing what I want which is de-serializing individual location objects.
If the item is converted to a JObject it will read in the full object and de-serialize it, this can be looped to get the solution.
This is the code that was settled on
while (reader.Read())
{
if (reader.TokenType == JsonToken.StartObject && reader.Depth == 2)
{
location = JObject.Load(reader).ToObject<Location>();
var lv = new LocationValidator(location, FootprintInfo.OperatorId, FootprintInfo.RoamingGroups, true);
var vr = lv.IsValid();
if (vr.Successful)
{
yield return location;
}
else
{
errors.Add(new Error(elNumber, location.LocationId, vr.Error.Field, vr.Error.Detail));
if (errors.Count >= maxErrors)
{
yield break;
}
}
++elNumber;
}
}

How to Parse this Json using Gson and get the field I want?

{
"ws_result":
[
{
"token": "",
"norm_token": "",
"len": "",
"type": "",
"pos": "",
"prop": "",
"stag": "",
"child":
[
{
"token": "",
"norm_token":"",
"len": "",
"type": "",
"pos": "",
"prop": "",
"stag": "",
"child":
[
{
"token": "",
"norm_token":"",
"len": "",
"type": "",
"pos": "",
"prop": "",
"stag": "",
"child": [ ]
},
{
"token": "",
"norm_token":"",
"len": "",
"type": "",
"pos": "",
"prop": "",
"stag": "",
"child": [ ]
}
]
},
{
"token": "",
"norm_token":"",
"len": "",
"type": "",
"pos": "",
"prop": "",
"stag": "",
"child":
[
{
"token": "",
"norm_token":"",
"len": "",
"type": "",
"pos": "",
"prop": "",
"stag": "",
"child": [ ]
}
]
}
]
},
{
"token": "",
"norm_token":"",
"len": "",
"type": "",
"pos": "",
"prop": "",
"stag": "2",
"child": [ ]
},
{
"token": "",
"norm_token": "",
"len": "",
"type": "",
"pos": "",
"prop": "",
"stag": "",
"child": [ ]
}
]
}
Such that some children are empty some is not, and some children contain more children. How do I actually parse this thing and get what I want. I am totally new with Json, and I am trying to use Gson. What I want is to get a value of a token with specific type in the nested Json. Thanks a lot for any help and directions.
I tried use com.google.gson.stream.JsonReader, but ist not working
JsonReader jsonReader = new JsonReader(new StringReader(result));
jsonReader.beginObject();
while(jsonReader.hasNext()){
String field = jsonReader.nextName();
if (field.equals("type")){
System.out.println(jsonReader.nextString());
} else if (field.equals("token")){
System.out.println(jsonReader.nextString());
} else {
jsonReader.skipValue();
}
}
jsonReader.endObject();
Parse your json recursively like this:
http://snipplr.com/view/71742/java-reflection-and-recursive-json-deserializer-using-gson/
private void parse(JsonObject o, PackagingResponse r){
Iterator<Entry<String, JsonElement>> i = o.entrySet().iterator();
while(i.hasNext()){
Entry<String, JsonElement> e = i.next();
JsonElement el = e.getValue();
if(el.isJsonObject())
parse(el.getAsJsonObject(), r);
//......
}
}