How to remove empty json object from nested json structure - json

In my current JSON, I am getting an empty JSON object {} inside dummy_var5. The empty object is inside an array which is inside an object itself.
{
"dummy_var1": "abc",
"dummy_var2": [
{
"item": {
"action": "test",
"po": {
"id": "abc"
},
"ot": "test1",
"id": "1"
}
}
],
"dummy_var3": {
"dummy_var4": [
{
"name": "test",
"value": "test1"
},
{
"name": "test",
"value": "test1"
}
],
"name": "test2"
},
"dummy_var5": [
{
"ref": "test",
"name": "test1",
"type": null
},
{}
],
"dummy_var6": [
{
"role": "test",
"ref": "test1",
"partyDescription": "test2"
}
]
}
Considering this structure does not change, Is there any way to remove this via jolt. We tried using third party tool 'atlasmap' but couldn't achieve the desired result.
Expected output :-
{
"dummy_var1": "abc",
"dummy_var2": [
{
"item": {
"action": "test",
"po": {
"id": "abc"
},
"ot": "test1",
"id": "1"
}
}
],
"dummy_var3": {
"dummy_var4": [
{
"name": "test",
"value": "test1"
},
{
"name": "test",
"value": "test1"
}
],
"name": "test2"
},
"dummy_var5": [
{
"ref": "test",
"name": "test1",
"type": null
}
],
"dummy_var6": [
{
"role": "test",
"ref": "test1",
"partyDescription": "test2"
}
]
}

You can use this single shift transformation spec
[
{
"operation": "shift",
"spec": {
"*": "&", // the attributes other than "dummy_var5"
"dummy_var5": {
"*": {
"*": "&2.[&1].&"
}
}
}
}
]
the match "*":"&" of the line "*": "&2.[&1].&" returns the null value from the leaf node for this level, and so removes the innermost null object {}

Related

A JOLT transformation question about mapping reference

This is an input that needs to be transformed using Jolt Transformation to obtain the expected output.
I am attempting to create a jolt transformation for the below input:
{
"flights": [
{
"id": "123",
"route": "BJS-SIN"
},
{
"id": "456",
"route": "SIN-PEK"
},
{
"id": "789",
"route": "SIN-BJS"
}
],
"prices": [
{
"id": "abc",
"amount": 560
},
{
"id": "def",
"amount": 780
}
],
"solutions": [
{
"price-ref": "abc",
"flights-ref": [
"123",
"456"
]
},
{
"price-ref": "def",
"flights-ref": [
"123",
"789"
]
}
]
}
Desired output would like :
{
"solutions": [
{
"flights": [
{
"id": "123",
"route": "BJS-SIN"
},
{
"id": "456",
"route": "SIN-PEK"
}
],
"price": {
"id": "abc",
"amount": 560
}
},
{
"flights": [
{
"id": "123",
"route": "BJS-SIN"
},
{
"id": "789",
"route": "SIN-BJS"
}
],
"price": {
"id": "def",
"amount": 780
}
}
]
}
As the output, all the data should be constructed refer to the solutions node. How about the mapping expression in JOLT?
all the values from the source json are generated randomly and the keys names are fixed.
I had tried many times but could not find the right spec, so please help.
You can use this spec:
[
{
"operation": "shift",
"spec": {
"*": "&",
"flights|prices": {
"*": {
"*": "&2.#(1,id).&"
}
}
}
},
{
"operation": "shift",
"spec": {
"solutions": {
"*": {
"flights*": {
"*": {
"*": {
"#(6,flights.&)": "&5[&4].flights[&2]"
}
}
},
"price*": {
"*": {
"#(5,prices.&)": "&4[&3].price"
}
}
}
}
}
}
]

Conditional Jolt spec for nested array

I am trying to write Jolt spec for the following input. I need to populate the primaryEmail field based on the condition if primary field is true in the emails array
[
{
"uid": "1234mark",
"name": "mark",
"userName": "markw",
"displayName": "Mark W",
"emails": [
{
"primary": false,
"value": "mark#gmail.com"
},
{
"primary": true,
"value": "mark#hotmail.com"
}
]
},
{
"uid": "9876steve",
"name": "steve",
"userName": "stevew",
"displayName": "Steve W",
"emails": [
{
"primary": false,
"value": "steve#gmail.com"
},
{
"primary": true,
"value": "steve#hotmail.com"
}
]
}
]
The desired output is
[
{
"user": {
"externalId": "1234mark",
"name": "mark",
"userName": "markw",
"displayName": "Mark W",
"primaryEmail": "mark#hotmail.com"
}
},
{
"user": {
"externalId": "9876steve",
"name": "steve",
"userName": "stevew",
"displayName": "Steve W",
"primaryEmail": "steve#hotmail.com"
}
}
]
But I get the following incorrect output since I am not able to populate the primaryEmail field conditionally properly.
[
{
"user": {
"externalId": "1234mark",
"name": "mark",
"userName": "markw",
"displayName": "Mark W"
}
},
{
"user": {
"externalId": "9876steve",
"name": "steve",
"userName": "stevew",
"displayName": "Steve W"
}
}
]
The spec I have created is the following
[
{
"operation": "shift",
"spec": {
"*": {
"uid": "[&1].user.externalId",
"name": "[&1].user.name",
"userName": "[&1].user.userName",
"displayName": "[&1].user.displayName",
"title": "[&1].user.title",
"emails": {
"*": {
"primary": {
"true": {
"#(2,value)": "primaryEmail"
}
}
}
}
}
}
}
]
Could someone please help with this query. Thanks.
What you need is to go 5 levels the three up from the innermost object while adding an extra node called user such as
[
{
"operation": "shift",
"spec": {
"*": {
"uid": "[&1].user.externalId",
"*": "[&1].user.&", // the attributes except for "uid" and "emails" array
"emails": {
"*": {
"primary": {
"true": {
"#(2,value)": "[&5].user.&2Email" // replicate literal "primary" by using &2
}
}
}
}
}
}
}
]
the demo on the site http://jolt-demo.appspot.com/ is

JOLT - Transform Array of JSON objects containing nested Arrays into their respective Array group

I want to transform a JSON input file into expected JSON output file:
My input file:
[
{
"category": [
{
"id": "id1",
"name": "Connected Home & Housewares1"
},
{
"id": "id2",
"name": "Housewares1"
},
{
"id": "id3",
"name": "Household Batteries1"
},
{
"id": "id4",
"name": "Alkaline Batteries1"
}
]
},
{
"category": [
{
"id": "id1",
"name": "Connected Home & Housewares2"
},
{
"id": "id2",
"name": "Housewares2"
},
{
"id": "id3",
"name": "Household Batteries2"
},
{
"id": "id4",
"name": "Alkaline Batteries2"
}
]
}
]
(Above is just an example of 2 records, but it contains ~50k records)
Expected Output:
[
{
"childSKUs": [
{
"divisionName": "Connected Home & Housewares1",
"deptName": "Housewares1",
"className": "Household Batteries1"
}
]
},
{
"childSKUs": [
{
"divisionName": "Connected Home & Housewares2",
"deptName": "Housewares2",
"className": "Household Batteries2"
}
]
}
]
My current JOLT Spec:
[
{
"operation": "shift",
"spec": {
"*": {
"category": {
"0": {
"name": "[0].childSKUs[&1].divisionName"
},
"1": {
"name": "[0].childSKUs[&1].deptName"
},
"2": {
"name": "[0].childSKUs[&1].className"
}
}
}
}
}
]
But it is only giving following output:
[
{
"childSKUs": [
{
"divisionName": [
"Connected Home & Housewares1",
"Connected Home & Housewares2"
]
},
{
"deptName": [
"Housewares1",
"Housewares2"
]
},
{
"className": [
"Household Batteries1",
"Household Batteries2"
]
}
]
}
]
I tried various options by playing in https://jolt-demo.appspot.com but no luck so far. Any help is appreciated. Would also be great if you explain your SPEC how was it done.
Ok, I figured it out myself.
Thanks to #Pokuri's answer here which gave me the idea to arrive at the solution: https://stackoverflow.com/a/55861132/248847
Correct JOLT SPEC for above question:
[
{
"operation": "shift",
"spec": {
"*": {
"category": {
"0": {
"name": "[&3].childSKUs[&1].divisionName"
},
"1": {
"name": "[&3].childSKUs[0].deptName"
},
"2": {
"name": "[&3].childSKUs[0].className"
}
}
}
}
}
]

Getting unique values from nested Array using jq

Trying to get unique values stored in items array for each group. somehow it's always mixed...
My JSON looks like this:
{
"start": 1534425916,
"stop": 1535030716,
"groups": [
{
"group": "transmission",
"data": {
"events": 665762,
},
"items": [
{
"item": "manualni",
"data": {
"events": 389158,
}
},
{
"item": "automaticka",
"data": {
"events": 276604,
}
}
]
},
{
"group": "vat",
"data": {
"events": 671924,
},
"items": [
{
"item": "ne",
"data": {
"events": 346221,
}
},
{
"item": "ano",
"data": {
"events": 325703,
}
}
]
}
]
}
Desired result is the following:
{
"id": "transmission",
"value": [
"manualni",
"automaticka",
]
}
{
"id": "vat",
"value": [
"ne",
"ano"
]
}
Tried with this filter on command line:
| jq '{id: .groups[].group, value: [.groups[].items[].item]}'
Which results in the above mentioned mixed up result:
{
"id": "transmission",
"value": [
"manualni",
"automaticka",
"ne",
"ano"
]
}
{
"id": "vat",
"value": [
"manualni",
"automaticka",
"ne",
"ano"
]
}
Any idea how to receive the uniquified values here? Thanks in advance!
This gets the desired result. I think the manual entry under .[] explains why it works.
jq '.groups[] | {"id": .group, "value": [.items[].item]}'

Jolt: Merge arrays from properties

I'm trying to extract and merge objects from an array contained in some (but not all) of my input elements. Using the JOLT JSON transformation library.
Also, the arrays I'm trying to merge contain objects that don't always have the same properties. One key might be present in some, but not others.
Example is contrived/nonsensical simplification, but has the general shape of our data.
Input:
{
"Widgets": [
{
"Id": "1",
"PetFriendly": "True",
"Features": [
{
"Name": "Easy Button",
"Type": "Button"
},
{
"Name": "Lunch Lever",
"Type": "Food Service",
"MenuItems": [
"Pizza",
"Cheezburger"
]
}
]
},
{
"Id": "2",
"PetFriendly": "True"
},
{
"Id": "3",
"PetFriendly": "False",
"Features": [
{
"Name": "Missles",
"Type": "Attack"
}
]
},
{
"Id": "4",
"PetFriendly": "False",
"Features": [
{
"Name": "Bombs",
"Type": "Attack",
"MenuItems": [
"Rat Poison"
]
}
]
}
]
}
Desired output:
{
"Widgets": [
{
"Id": "1"
"PetFriendly": "True"
},
{
"Id": "2"
"PetFriendly": "True"
},
{
"Id": "3",
"PetFriendly": "False"
},
{
"Id": "4",
"PetFriendly": "False"
}
],
"Features": [
{
"WidgetId": "1",
"Name": "Easy Button",
"Type": "Button"
},
{
"WidgetId": "1",
"Name": "Lunch Lever",
"Type": "Food Service",
"MenuItems": [
"Pizza",
"Cheezburger"
]
},
{
"WidgetId": "3",
"Name": "Missles",
"Type": "Attack"
},
{
"WidgetId": "4",
"Name": "Bombs",
"Type": "Attack",
"MenuItems": [
"Rat Poison"
]
}
]
}
I have tried many transforms with no success, and read all the ShiftR documentation and its unit tests. A little help?
Spec
[
{
"operation": "shift",
"spec": {
"Widgets": {
"*": {
// build the finished "Widgets" output
"Id": "Widgets[&1].Id",
"PetFriendly": "Widgets[&1].PetFriendly",
//
// Process the Features, by pushing the Id
// down into them, but maintain the same doubly
// nested structure.
// Shift works property by property, so first
// fix the properties in side each Features element,
// (pulling ID down).
// Then in a 2nd Shift can accumulate things into array.
"Features": {
"*": {
"#(2,Id)": "temp[&3].Features[&1].WidgetId",
"*": "temp[&3].Features[&1].&"
}
}
}
}
}
},
{
"operation": "shift",
"spec": {
// passthru
"Widgets": "Widgets",
"temp": {
"*": {
"Features": {
// walk thru the doubly nested structure an
// now accumulate all non-null itens into
// the the final Features array.
"*": "Features[]"
}
}
}
}
}
]
Finally got it working with the below spec, BUT it has an undesirable side effect: It leaves empty default arrays. Is there a way to remove empty arrays, or otherwise mark them during the default step so they can be deleted? I checked this GitHub issue but not sure how to translate it to arrays of string. Anyone have a better solution?
[
// First fill in default value for "MenuItems" since not all Features have it.
{
"operation": "default",
"spec": {
"Widgets[]": {
"*": {
"Features[]": {
"*": {
"MenuItems": []
}
}
}
}
}
},
{
// Extract the Features' properties into arrays. The defaults added above ensure that we can merge the arrays into Feature objects as in this example:
// https://github.com/bazaarvoice/jolt/blob/master/jolt-core/src/test/resources/json/shiftr/mergeParallelArrays2_and-do-not-transpose.json.
"operation": "shift",
"spec": {
"Widgets": {
"*": {
"Id": "Widgets[&1].Id",
"PetFriendly": "Widgets[&1].PetFriendly",
"Features": {
"*": {
"#(2,Id)": "temp.WidgetId",
"Name": "temp.Name",
"Type": "temp.Type",
"MenuItems": "temp.MenuItems[]"
}
}
}
}
}
},
// Finally merge the arrays into Feature objects.
{
"operation": "shift",
"spec": {
"Widgets": "Widgets",
"temp": {
"WidgetId": {
"*": "Features[&0].WidgetId"
},
"Name": {
"*": "Features[&0].Name"
},
"Type": {
"*": "Features[&0].Type"
},
"MenuItems": {
"*": "Features[&0].MenuItems"
}
}
}
}
]
Result:
{
"Widgets": [
{
"Id": "1",
"PetFriendly": "True"
},
{
"Id": "2",
"PetFriendly": "True"
},
{
"Id": "3",
"PetFriendly": "False"
},
{
"Id": "4",
"PetFriendly": "False"
}
],
"Features": [
{
"WidgetId": "1",
"Name": "Easy Button",
"Type": "Button",
"MenuItems": []
},
{
"WidgetId": "1",
"Name": "Lunch Lever",
"Type": "Food Service",
"MenuItems": [ "Pizza", "Cheezburger" ]
},
{
"WidgetId": "3",
"Name": "Missles",
"Type": "Attack",
"MenuItems": []
},
{
"WidgetId": "4",
"Name": "Bombs",
"Type": "Attack",
"MenuItems": [ "Rat Poison" ]
}
]
}