I am trying to transform json to complex array. The inner array of INVLOC is not generating the way I want.
My input json is :
{
"valid": "true",
"message": "",
"data": {
"ARINVT01": [
{
"firstName": "andrew",
"lastname": "Gilly",
"INVLOC": {
"data": [
{
"mmm": "MAIN",
"nnn": "0.000"
},
{
"mmm": "A1",
"nnn": "0.000"
}
]
}
},
{
"firstName": "Jack",
"lastname": "watson",
"INVLOC": {
"data": [
{
"mmm": "MAIN",
"nnn": "0.000"
}
]
}
}
]
}
}
Expected output :
[
{
"FNAME": "andrew",
"LNAME": "Gilly",
"INVBALANCES": [
{
"user": "MAIN",
"CURBAL": "0.000"
},
{
"user": "A1",
"CURBAL": "0.000"
}
]
},
{
"FNAME": "Jack",
"LNAME": "watson",
"INVBALANCES": [
{
"user": "A1",
"CURBAL": "0.000"
}
]
}
]
My spec :
[
{
"operation": "shift",
"spec": {
"data": {
"ARINVT01": {
"*": {
"firstName": "[&1].FNAME",
"lastname": "[&1].LNAME",
"INVLOC": {
"data": {
"*": {
"mmm": "[&1].INVBALANCES.[#1].user",
"nnn": "[&1].INVBALANCES.[#1].CURBAL"
}
}
}
}
}
}
}
}
]
Getting output :
[ {
"FNAME" : "andrew",
"LNAME" : "Gilly",
"INVBALANCES" : [ {
"user" : "MAIN"
}, {
"CURBAL" : "0.000"
} ]
}, {
"INVBALANCES" : [ {
"user" : "A1"
}, {
"CURBAL" : "0.000"
} ],
"FNAME" : "Jack",
"LNAME" : "watson"
} ]
Can someone Help me with my spec?
And also if I can get some sort of tutorial in understanding the array transformation of JOLT.
Spec
[
{
"operation": "shift",
"spec": {
"data": {
"ARINVT01": {
"*": { // array index of ARINVT01
//
// &0 is "firstName"
// &1 is the current array index of ARINVT01
"firstName": "[&1].FNAME",
"lastname": "[&1].LNAME",
"INVLOC": {
"data": {
"*": { // array index of data array
//
// &0 is "mmm"
// &1 is the current array index of the data array
// &2 is "data"
// &3 is "INVLOC"
// &4 is the current array index of ARINVT01
// basically the "&" "number" logic is
// how many levels to walk back up the document
// to lookup the currently matched value
"mmm": "[&4].INVBALANCES.[&1].user",
"nnn": "[&4].INVBALANCES.[&1].CURBAL"
}
}
}
}
}
}
}
}
]
Related
In my current JSON, I am getting an empty JSON object {} inside dummy_var5. The empty object is inside an array which is inside an object itself.
{
"dummy_var1": "abc",
"dummy_var2": [
{
"item": {
"action": "test",
"po": {
"id": "abc"
},
"ot": "test1",
"id": "1"
}
}
],
"dummy_var3": {
"dummy_var4": [
{
"name": "test",
"value": "test1"
},
{
"name": "test",
"value": "test1"
}
],
"name": "test2"
},
"dummy_var5": [
{
"ref": "test",
"name": "test1",
"type": null
},
{}
],
"dummy_var6": [
{
"role": "test",
"ref": "test1",
"partyDescription": "test2"
}
]
}
Considering this structure does not change, Is there any way to remove this via jolt. We tried using third party tool 'atlasmap' but couldn't achieve the desired result.
Expected output :-
{
"dummy_var1": "abc",
"dummy_var2": [
{
"item": {
"action": "test",
"po": {
"id": "abc"
},
"ot": "test1",
"id": "1"
}
}
],
"dummy_var3": {
"dummy_var4": [
{
"name": "test",
"value": "test1"
},
{
"name": "test",
"value": "test1"
}
],
"name": "test2"
},
"dummy_var5": [
{
"ref": "test",
"name": "test1",
"type": null
}
],
"dummy_var6": [
{
"role": "test",
"ref": "test1",
"partyDescription": "test2"
}
]
}
You can use this single shift transformation spec
[
{
"operation": "shift",
"spec": {
"*": "&", // the attributes other than "dummy_var5"
"dummy_var5": {
"*": {
"*": "&2.[&1].&"
}
}
}
}
]
the match "*":"&" of the line "*": "&2.[&1].&" returns the null value from the leaf node for this level, and so removes the innermost null object {}
I'm new to Jolt Transform and have some EDI data that I want to transform. I want to keep the original data and append a few extracted elements in order to make it easier for a downstream process to get to. Here's a sample:
Input
{
"id": "2000",
"segments": [
{
"REF02": "xxxxxxxxxxxx",
"REF01": "0F",
"id": "REF"
},
{
"REF02": "yyyyyyyyyyyy",
"REF01": "1L",
"id": "REF"
}
],
"loops": [
{
"id": "2100",
"segments": []
},
{
"id": "2100",
"segments": [],
"loops": [
{
"id": "2300",
"segments": [
{
"DTP01": "348",
"DTP03": "20220101",
"DTP02": "D8",
"id": "DTP"
},
{
"DTP01": "349",
"DTP03": "20221231",
"DTP02": "D8",
"id": "DTP"
}
]
}
]
}
]
}
Desired Output
{
"ext": {
"subscriber": "xxxxxxxxxxxx",
"start": "20220101"
},
"id": "2000",
"segments": [
{
"REF02": "xxxxxxxxxxxx",
"REF01": "0F",
"id": "REF"
},
{
"REF02": "yyyyyyyyyyyy",
"REF01": "1L",
"id": "REF"
}
],
"loops": [
{
"id": "2100",
"segments": []
},
{
"id": "2100",
"segments": [],
"loops": [
{
"id": "2300",
"segments": [
{
"DTP01": "348",
"DTP03": "20220101",
"DTP02": "D8",
"id": "DTP"
},
{
"DTP01": "349",
"DTP03": "20221231",
"DTP02": "D8",
"id": "DTP"
}
]
}
]
}
]
}
That is, I'd like to append the ext element - copying values from the original.
In xpath terms, the extracted values' paths look something like this:
subscriber :
/segments[../id eq "2000"]/..[REF01 eq "0F"]/REF02/text()
start :
/loops[../id eq "2000"]/../segments[../id eq "2100"])[1]/..
[id eq "NM1" and NM101 eq "IL"][DTP01 eq "348"]/DTP03/text()
Thanks for the help.
I ended up with this solution. If there's any feedback on streamlining, making more elegant, let me know.
[
{
"operation": "default",
"spec": {
"dummy": {}
}
},
{
"operation": "shift",
"spec": {
"dummy": {
"#2,segments": {
"*": {
"REF01": {
"0F": {
"#2,REF02": "ext.subscriber"
}
}
}
},
"#2,loops": {
"*": {
"id": {
"2100": {
"#2,loops": {
"*": {
"id": {
"2300": {
"#2,segments": {
"*": {
"DTP01": {
"348": {
"#2,DTP03": "ext.start"
}
}
}
}
}
}
}
}
}
}
}
}
},
"id": "id",
"segments": "segments",
"loops": "loops"
}
}
]
You can replicate the whole content by use of "#": "" within a shift transformation while adding less curly braces to nest the inner stuff, at the same time determining the paths to search to be used in the next spec such as
[
{
"operation": "shift",
"spec": {
"#": "",
"segments": {
"*": {//indexes of the array
"#REF02": "ext.su.#REF01"
}
},
"loops": {
"*": {//indexes of the array
"#loops": {
"*": {//indexes of the array
"#segments": {
"*": {//indexes of the array
"#DTP03": "ext.st.#5,id.#3,id.#DTP01"
}
}
}
}
}
}
}
},
{
"operation": "shift",
"spec": {
"ext": {
"#su.0F": "&1.subscriber",
"#st.2100.2300.348": "&1.start"
},
"*": "&" //replicate the other attributes/arrays/objects
}
}
]
This is an input that needs to be transformed using Jolt Transformation to obtain the expected output.
I am attempting to create a jolt transformation for the below input:
{
"flights": [
{
"id": "123",
"route": "BJS-SIN"
},
{
"id": "456",
"route": "SIN-PEK"
},
{
"id": "789",
"route": "SIN-BJS"
}
],
"prices": [
{
"id": "abc",
"amount": 560
},
{
"id": "def",
"amount": 780
}
],
"solutions": [
{
"price-ref": "abc",
"flights-ref": [
"123",
"456"
]
},
{
"price-ref": "def",
"flights-ref": [
"123",
"789"
]
}
]
}
Desired output would like :
{
"solutions": [
{
"flights": [
{
"id": "123",
"route": "BJS-SIN"
},
{
"id": "456",
"route": "SIN-PEK"
}
],
"price": {
"id": "abc",
"amount": 560
}
},
{
"flights": [
{
"id": "123",
"route": "BJS-SIN"
},
{
"id": "789",
"route": "SIN-BJS"
}
],
"price": {
"id": "def",
"amount": 780
}
}
]
}
As the output, all the data should be constructed refer to the solutions node. How about the mapping expression in JOLT?
all the values from the source json are generated randomly and the keys names are fixed.
I had tried many times but could not find the right spec, so please help.
You can use this spec:
[
{
"operation": "shift",
"spec": {
"*": "&",
"flights|prices": {
"*": {
"*": "&2.#(1,id).&"
}
}
}
},
{
"operation": "shift",
"spec": {
"solutions": {
"*": {
"flights*": {
"*": {
"*": {
"#(6,flights.&)": "&5[&4].flights[&2]"
}
}
},
"price*": {
"*": {
"#(5,prices.&)": "&4[&3].price"
}
}
}
}
}
}
]
I have a JSON input :
{
"Name": "travel.0.0",
"Documents": [
{
"Id": 6732210,
"Name": "Passport",
"Fields": [
{
"Name": "First Name",
"Type": "text"
}
],
"Variants": [
{
"Name": "diplomatic_passport",
"TrainingSamples": [
{
"Name": "First Name",
"Value": "VGhlIFBPU1QsIFBVVCBhbmQgUEFUQ0ggcmVxdWVzdHMgY2FuIGhhdmUgdGhlIHJlcXVlc3QgYm9keSAocGF5bG9hZCksIHN1Y2ggYXMgSlNPTiBvciBYTUwgZGF0YS4gSW4gU3dhZ2dlciB0ZXJtcywgdGhlIHJlcXVlc3QgYm9keSBpcyBjYWxsZWQgYSBib2R5IHBhcmFtZXRlci4gVGhlcmUgY2FuIGJlIG9ubHkgb25lIGJvZHkgcGFyYW1ldGVyLCBhbHRob3VnaCB0aGUgb3BlcmF0aW9uIG1heSBoYXZlIG90aGVyIHBhcmFtZXRlcnMgKHBhdGgsIHF1ZXJ5LCBoZWFkZXIpLg==",
"BoundingBoxes": [
{
"Name": "string",
"Dimensions": [
{
"PageIndex": 0,
"Left": 0,
"Top": 0,
"Right": 0,
"Bottom": 0
}
]
}
]
}
]
}
]
}
]
}
My JOLT Spec is :
[
{
"operation": "shift",
"spec": {
"Name": "ProjectName",
"Documents": {
"*": {
"Name": {
"#": ["DocumentDefinitions[#].Name", "DocumentDefinitions[#].SectionDefinitions[#].Name"]
},
"Variants": {
"*": {
"Name": "DocumentDefinitions[#].SectionDefinitions[#].VarientNames[&1]",
"TrainingSamples": {
"0": {
"Name": "DocumentDefinitions[#].SectionDefinitions[#].ImageSources[#].ImageName.",
"Value": "DocumentDefinitions[#].SectionDefinitions[#].ImageSources[#].ImageValue."
}
}
}
},
"Fields": {
"*": {
"Name": "DocumentDefinitions[#].SectionDefinitions[#].Fields[&1].Name.",
"Type": "DocumentDefinitions[#].SectionDefinitions[#].Fields[&1].Type."
}
}
}
}
}
}
]
My Output is :
{
"ProjectName" : "travel.0.0",
"DocumentDefinitions" : [ {
"Name" : "Passport",
"SectionDefinitions" : [ {
"Name" : "Passport",
"VarientNames" : [ "diplomatic_passport" ],
"ImageSources" : [ {
"ImageName" : "First Name",
"ImageValue" : "VGhlIFBPU1QsIFBVVCBhbmQgUEFUQ0ggcmVxdWVzdHMgY2FuIGhhdmUgdGhlIHJlcXVlc3QgYm9keSAocGF5bG9hZCksIHN1Y2ggYXMgSlNPTiBvciBYTUwgZGF0YS4gSW4gU3dhZ2dlciB0ZXJtcywgdGhlIHJlcXVlc3QgYm9keSBpcyBjYWxsZWQgYSBib2R5IHBhcmFtZXRlci4gVGhlcmUgY2FuIGJlIG9ubHkgb25lIGJvZHkgcGFyYW1ldGVyLCBhbHRob3VnaCB0aGUgb3BlcmF0aW9uIG1heSBoYXZlIG90aGVyIHBhcmFtZXRlcnMgKHBhdGgsIHF1ZXJ5LCBoZWFkZXIpLg=="
} ],
"Fields" : [ {
"Name" : "First Name",
"Type" : "text"
} ]
} ]
} ]
}
This output looks fine , until i have only one JSON object inside my input array of documents . However , when i put another object , it looks somewhat like this which is not desired . It is clubbing all values of both objects together as below :
{
"ProjectName" : "travel.0.0",
"DocumentDefinitions" : [ {
"Name" : [ "Passport", "Passport" ],
"SectionDefinitions" : [ {
"Name" : [ "Passport", "Passport" ],
"VarientNames" : [ [ "diplomatic_passport", "diplomatic_passport" ] ],
"ImageSources" : [ {
"ImageName" : [ "First Name", "First Name" ],
"ImageValue" : [ "VGhlIFBPU1QsIFBVVCBhbmQgUEFUQ0ggcmVxdWVzdHMgY2FuIGhhdmUgdGhlIHJlcXVlc3QgYm9keSAocGF5bG9hZCksIHN1Y2ggYXMgSlNPTiBvciBYTUwgZGF0YS4gSW4gU3dhZ2dlciB0ZXJtcywgdGhlIHJlcXVlc3QgYm9keSBpcyBjYWxsZWQgYSBib2R5IHBhcmFtZXRlci4gVGhlcmUgY2FuIGJlIG9ubHkgb25lIGJvZHkgcGFyYW1ldGVyLCBhbHRob3VnaCB0aGUgb3BlcmF0aW9uIG1heSBoYXZlIG90aGVyIHBhcmFtZXRlcnMgKHBhdGgsIHF1ZXJ5LCBoZWFkZXIpLg==", "VGhlIFBPU1QsIFBVVCBhbmQgUEFUQ0ggcmVxdWVzdHMgY2FuIGhhdmUgdGhlIHJlcXVlc3QgYm9keSAocGF5bG9hZCksIHN1Y2ggYXMgSlNPTiBvciBYTUwgZGF0YS4gSW4gU3dhZ2dlciB0ZXJtcywgdGhlIHJlcXVlc3QgYm9keSBpcyBjYWxsZWQgYSBib2R5IHBhcmFtZXRlci4gVGhlcmUgY2FuIGJlIG9ubHkgb25lIGJvZHkgcGFyYW1ldGVyLCBhbHRob3VnaCB0aGUgb3BlcmF0aW9uIG1heSBoYXZlIG90aGVyIHBhcmFtZXRlcnMgKHBhdGgsIHF1ZXJ5LCBoZWFkZXIpLg==" ]
} ],
"Fields" : [ {
"Name" : [ "First Name", "First Name" ],
"Type" : [ "text", "text" ]
} ]
} ]
} ]
}
Please suggest what changes should i do in my JOLT spec , such that each transformed object in the array , get listed as a seprate object , and not get clubbed .
Check this spec,
[
{
"operation": "shift",
"spec": {
"Name": "ProjectName",
"Documents": {
"*": {
"Name": "DocumentDefinitions[&1].Name",
"Variants": {
"*": {
"Name": "DocumentDefinitions[&3].SectionDefinitions[&1].VarientNames[]",
"TrainingSamples": {
"*": {
"Name": "DocumentDefinitions[&5].SectionDefinitions[&1].ImageSources[&1].ImageName",
"Value": "DocumentDefinitions[&5].SectionDefinitions[&1].ImageSources[&1].ImageValue"
}
}
}
},
"Fields": {
"*": {
"Name": "DocumentDefinitions[&3].SectionDefinitions[&3].Fields[&1].Name",
"Type": "DocumentDefinitions[&3].SectionDefinitions[&3].Fields[&1].Type"
}
}
}
}
}
}
]
Trying to make a Jolt script that will put in a single number line, then a array of strings in one single array and a tag on the end of that array. These are the values that I have been working with.
JSON INPUT
[
{
"foo": "111",
"bar": "222",
"sun": "333",
"ListofStrings": [
"Dog",
"Train"
],
"ID": "BLAH"
},
{
"foo": "999",
"bar": "222",
"sun": "777",
"ListofStrings": [
"CAT",
"PLANE"
],
"ID": "HAHA"
}
]
JOLT SPEC This is what I have been working with that prints out the ListofStrings but this is the one that works stably.
[
{
"operation": "shift",
"spec": {
"*": {
"foo": "input[].number",
"bar": "input[].number",
"sun": "input[].number",
"ListofStrings": "input[].List"
}
}
},
{
"operation": "default",
"spec": {
"app_id": "test",
"input": {
"*": {
"app_id": "test"
}
}
}
}
]
CURRENT OUTPUT
{
"input" : [ {
"number" : "111"
}, {
"number" : "222"
}, {
"number" : "333"
}, {
"List" : [ "Dog", "Train" ]
}, {
"number" : "999"
}, {
"number" : "222"
}, {
"number" : "777"
}, {
"List" : [ "Cat", "Car" ]
} ],
"app_id" : "test"
}
DESIRED OUTPUT
{
"input" : [ {
"number" : "111"
"List" : [ "Dog", "Train" ]
"ID": "BLAH_foo"
}, {
"number" : "222"
"List" : [ "Dog", "Train" ]
"ID": "BLAH_bar"
}, {
"number" : "333"
"List" : [ "Dog", "Train" ]
"ID": "BLAH_sun"
}, {
"number" : "999"
"List" : [ "Cat", "Car" ]
"ID": "HAHA_foo"
}, {
"number" : "222"
"List" : [ "Cat", "Car" ]
"ID": "HAHA_bar"
}, {
"number" : "777"
"List" : [ "Cat", "Car" ]
"ID": "HAHA_sun"
} ],
"app_id" : "test"
}
Check this spec
[
//Converting list to Map
{
"operation": "shift",
"spec": {
"*": {
"ListofStrings": null,
"*": {
"#": "#1.number",
"#(1,ListofStrings)": "#1.list"
}
}
}
},
//Shift the number and list to the input array
{
"operation": "shift",
"spec": {
"*": {
"$": "input[#2].number",
"#(0,list)": "input[#2].List"
}
}
}, {
"operation": "default",
"spec": {
"app_id": "test",
"input": {
"*": {
"app_id": "test"
}
}
}
}
]
Edit 1
Add the ID node to the map using first shift operation "ID": null, and "#(1,ID)": "#1.ID". Then shift the ID node to the input array in the second shift operation "#(0,ID)": "input[#2].ID".
[
//Converting list to Map
{
"operation": "shift",
"spec": {
"*": {
"ListofStrings": null,
"ID": null,
"*": {
"#": "#1.number",
"#(1,ListofStrings)": "#1.list",
"#(1,ID)": "#1.ID"
}
}
}
},
//Shift the number and list to the input array
{
"operation": "shift",
"spec": {
"*": {
"$": "input[#2].number",
"#(0,list)": "input[#2].List",
"#(0,ID)": "input[#2].ID"
}
}
}, {
"operation": "default",
"spec": {
"app_id": "test",
"input": {
"*": {
"app_id": "test"
}
}
}
}
]