JOLT concatenate two strings - json

I have this collection of driver details and I want to use jolt to copy the properties to concatenate first name and last name within a single property called name
{
"drivers_details": [
{
"first_name": "Doru",
"last_name": "Petre",
"cnp": "1641201390687",
"id_series": "RK",
"id_number": "123456"
},
{
"first_name": "GIGI",
"last_name": "FANE",
"cnp": "16412013906871",
"id_series": "RK",
"id_number": "1234567"
}
]
}
I am using this spec for the shift transformation spec
[
{
"operation": "shift",
"spec": {
"drivers_details": {
"*": {
"cnp": "body.drivers[&1].tin",
"id_series": "body.drivers[&1].id_series",
"id_number": "body.drivers[&1].id_number"
}
}
}
}
]
and this one for modify-overwrite, but it doesn't work how I was expecting
{
"operation": "modify-overwrite-beta",
"spec": {
"drivers_details": {
"*": {
"name": "=concat(#(2,first_name),' ',#(2,last_name))"
}
}
}
}
What is the correct way of doing this?

You can use modify transformation as in the following case, and get rid of names other than newly formed attribute name such as
[
{
"operation": "modify-overwrite-beta",
"spec": {
"drivers_details": {
"*": {
"name": "=concat(#(1,first_name),' ',#(1,last_name))"
}
}
}
},
{
"operation": "remove",
"spec": {
"drivers_details": {
"*": {
"*name": ""
}
}
}
},
{ // this last spec stands only for ordering of the attributes if the ordering and putting the "name" at the top matters.
// So, you might get rid of the "remove" transformation above if this is used. Otherwise keep it.
"operation": "shift",
"spec": {
"drivers_details": {
"*": {
"name": "&2[&1].&",
"cnp": "&2[&1].&",
"id*": "&2[&1].&"
}
}
}
}
]
where those attributes(first/last_name) stays right-hand-side at the current level, so just only need to traverse 1 colon(:) which appears as #(1,..)

You can use this spec without the remove operation:
[
{
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"*": {
"name": "=concat(#(1,first_&),' ',#(1,last_&))"
}
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"*": {
"name": "&2[&1].&",
"cnp": "&2[&1].&",
"id_*": "&2[&1].&"
}
}
}
}
]

Related

How to split the nested JSON in Jolt specification

pls help me out wit this jolt specification. pls help me
Notes :
Resourcename is the last element of ResourceId which will be a new attribute that we need to add to the expected output
Tags field needs to be copied and splited as mentioned in the expected output.
Input :
[
{
"ResourceId": "/subscriptions/bb842437aa4/resourceGroups/ECHLABHENKEL/providers/Microsoft.Compute/virtualMachines/pmoapps",
"Tags": "Name\": \"PMOapplication\",\"Owner\": \"Breil sathish"
},
{
"ResourceId": "/subscriptions/bb842437aa4/resourceGroups/HCLTECHLABHENKEL/providers/Microsoft.Compute/virtualMachines/pmoapps",
"Tags": "Name\": \"PMOapplication\",\"Owner\": \"Breil sathish1"
}
]
Expected Output :
[
{
"ResourceId": "/subscriptions/bb842437aa4/resourceGroups/ECHLABHENKEL/providers/Microsoft.Compute/virtualMachines/pmoapps",
"Tags": "Name\": \"PMOapplication\",\"Owner\": \"Breil sathish",
"Resourcename": "pmoapps",
"Name": "PMOapplication",
"Owner": "Breil sathish"
},
{
"ResourceId": "/subscriptions/bb842437aa4/resourceGroups/HCLTECHLABHENKEL/providers/Microsoft.Compute/virtualMachines/pmoapps",
"Tags": "Name\": \"PMOapplication\",\"Owner\": \"Breil sathish1",
"Resourcename": "pmoapps",
"Name": "PMOapplication",
"Owner": "Breil sathish1"
}
]
Thanks,
N Sathish
You can use the following transformation spec as reading the explanations stated at the beginning of each of them such as
[
{ // Convert the value of the attributes to individual arrays with the identical names
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"ResourceName": "=split('/', #(1,ResourceId))",
"tag": "=split(',', #(1,Tags))"
}
}
},
{ // Generate tag0 and tag1 attributes by splitting members of the "tag" array
"operation": "shift",
"spec": {
"*": {
"*": "&1.&",
"tag": {
"*": {
"#": "&3.&2&1"
}
}
}
}
},
{ // Split related strings by colon characters for "tag" array while deriving the last element of "ResourceName" array
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"R*": "=lastElement(#(1,&))",
"tag*": "=split(': ', #(1,&))"
}
}
},
{ // Match components of "tag" array component 1 against component 2
"operation": "shift",
"spec": {
"*": {
"*": "&1.&",
"tag*": {
"#1,&[1]": "&2.&1.#(2,&[0])"
}
}
}
},
{ // Split the values by \" character combination
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"tag*": {
"*": "=split('\"', #(1,&))"
}
}
}
},
{ // Prune undesired values for right-hand-side
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"tag*": {
"*": "=join('', #(1,&))"
}
}
}
},
{ // Prune undesired values for left-hand-side(keys)
"operation": "shift",
"spec": {
"*": {
"*": "[&1].&",
"tag*": {
"\"*\"": "[&2].&(0,1)",
"*\"": "[&2].&(0,1)"
}
}
}
}
]

JOLT JSON transform values from one-to-many to one-to-one

I'm trying to map one key to each value in array to a new array by using JOLT.
Could someone please help give me a solution for this:
My JSON:
[
{
"person_id": "1",
"resources": ["asd", "zxc"]
},
{
"person_id": "2",
"resources": ["ghj", "asd"]
}
]
And my expected JSON:
[
{
"person_id": "1",
"resource": "asd"
},
{
"person_id": "1",
"resource": "zxc"
},
{
"person_id": "2",
"resource": "ghj"
},
{
"person_id": "2",
"resource": "asd"
}
]
I had tried this Jolt Specification
[
{
"operation": "shift",
"spec": {
"*": {
"resources": {
"*": {
"#(2,person_id)": "[&].person_id",
"#": "[&].resource"
}
}
}
}
}
]
But no luck it always maps all values at the same index to 1 array.
You can use two consecutive shift transformation specs by walking through the resources array within the first one such that
[
{
"operation": "shift",
"spec": {
"*": {
"*s": { // this tecnique stands for extracting by the replacement of the asterisk through use of &(2,1) below
"*": {
"#(2,person_id)": "[&3].&1.person_id", // to separate by indexes of the array to generate three level for three independent objects
"#": "[&3].&1.&(2,1)"
}
}
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"*": ""
}
}
}
]
the demo on the site http://jolt-demo.appspot.com/ is
You can use this jolt spec:
[
{
"operation": "shift",
"spec": {
"*": {
"*s": {
"*": {
"#(2,person_id)": "#(1).person_id",
"#": "#(1).&(2,1)"
}
}
}
}
},
{
"operation": "shift",
"spec": {
"*": ""
}
}
]

JoltTransformJson - Json Transformation

i have JSON value as below :
{
"table": "table_name",
"op_type": "U",
"before": {
"AAAA": "1-1111",
"BBBB": "2022-08-31 03:57:01",
"CCCC": "2023-08-31 23:59:59"
},
"after": {
"AAAA": "1-1112",
"BBBB": "2022-08-31 10:10:34"
}
}
i want to do this how can i do?
{
"AAAA": "1-1112",
"BBBB": "2022-08-31 10:10:34",
"CCCC": "2023-08-31 23:59:59"
"changed_columns": "AAAA, BBBB"
}
AAAA: "If you have after.AAAA, take AAAA else before.AAAA", BBBB: "If you have after.BBBB, take BBBB else before.BBBB.
AND I want to add changed_columns field like this :
,"changed_columns": "AAAA, BBBB"
is there a way to do this?
You can use shift operation and getting after values for the first. and then you can using before values. So if keys match with together you have an array with two element.
Now You can get first element to getting after values with modify-overwrite-beta operations and =firstElement function.
[
{
"operation": "shift",
"spec": {
"after": {
"*": {
"$": "changed_columns[]",
"#(1,&)": "&1"
}
},
"before": {
"*": "&"
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"*": "=firstElement(#(1,&))",
"changed_columns": "=join(', ',#(1,changed_columns))"
}
}
]
You can use
cardinality spec lately after using "after|before" as the key in this order to determine the precedence
exchange key-value pairs consecutively twice to determine whether
really changed the components in order to form "changed_columns"
such as
[
{
// multiplex the attributes in order to generate three independent groups
"operation": "shift",
"spec": {
"after|before": { // this order is important to determine the precedence in the upcoming cardinality spec
"*": {
"#": "&",
"#(0)": "l.&",
"*": {
"#1": "f.&2"
}
}
}
}
},
{
// determine whether before vs. after values equal through this and next two specs
"operation": "modify-overwrite-beta",
"spec": {
"l": {
"*": "=lastElement(#(1,&))"
},
"f": {
"*": "=firstElement(#(1,&))"
}
}
},
{
"operation": "shift",
"spec": {
"*": "&",
"l|f": {
"*": {
"$": "lf.#(0)"
}
}
}
},
{
"operation": "shift",
"spec": {
"*": "&",
"lf": {
"*": {
"$": "&2.#(0)"
}
}
}
},
{
// construct an array from those newly formed keys
"operation": "shift",
"spec": {
"*": "&",
"lf": {
"*": {
"$": "changed_columns"
}
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"changed_columns": "=join(', ',#(1,&))"
}
},
{
"operation": "cardinality",
"spec": {
"*": "ONE"
}
},
{
"operation": "sort"
}
]

Unexpected null elements in output array

EDIT: Now after 16 hours break I realized I have been misreading the results for hours - I didn't notice that there is null in all cases, so the behavior is consistent unlike I claim in this question (facepalm). However I decided not to delete this question but only close it as there is already useful answers.
Consider the following input JSON:
{
"widgets": [
{
"type": "FOO",
"id": "F1"
},
{
"type": "ZAP",
"id": "Z1"
},
{
"type": "BAR",
"id": "B1"
}
]
}
The following transformation:
[
{
"operation": "shift",
"spec": {
"widgets": {
"*": {
"type": {
"FOO": {
"#(2,id)": "widgets[&3].fooId"
},
"BAR": {
"#(2,id)": "widgets[&3].barId"
}
}
}
}
}
}
]
creates the expected (correct) output. EDIT I misread this output - I didn't realize there is 3 elements where one element is null but I though there is only 2 non-null elements:
{
"widgets": [
{
"fooId": "F1"
},
null,
{
"barId": "B1"
}
]
}
However the following transformation creates the unexpected output:
[
{
"operation": "shift",
"spec": {
"widgets": {
"*": {
"type": {
"BAR": {
"#(2,id)": "widgets[&3].barId"
}
}
}
}
}
}
]
This is the actual (wrong) output:
{
"widgets": [
null,
null,
{
"barId": "B1"
}
]
}
This is the expected (correct) output:
{
"widgets": [
{
"barId": "B1"
}
]
}
Why sometimes there is null elements in the widgets array and sometimes there is not? How I can avoid them?
Based on the many other SO questions the following operation:
{
"operation": "modify-overwrite-beta",
"spec": {
"*": "=recursivelySquashNulls"
}
}
should remove the null values, but why those are not removed in the following transformation?
[
{
"operation": "shift",
"spec": {
"widgets": {
"*": {
"type": {
"BAR": {
"#(2,id)": "widgets[&3].barId"
}
}
}
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"*": "=recursivelySquashNulls"
}
}
]
I'm observing this behavior in http://jolt-demo.appspot.com/
The first output keeps null as well(I don't know if it should appear in the desired output). Since the index(which is 1) of the object having "type": "ZAP" stays in the middle of the indexes(0,1,2), but not at the end, then doesn't vanish at that level.
Btw, squashNulls especially no impact for the objects nested within array. Indeed, it has some bugs and documented as fixed for version 0.1.6.
You can rather use the following spec which doesn't have square-bracketed indexes such as
[
{
"operation": "shift",
"spec": {
"*": {
"*": {
"type": {
"FOO|BAR": {
"#(2,id)": "&4.&1.&1Id"
}
}
}
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"*": {
"#": "&2[]"
}
}
}
}
]
This spec will help you resolve this query :
1)
Actually recursivelySquashNulls will work if you keep the array in a object for eg: I kept it under test object and made it to previous one.
This spec will resolve your issue :
[
{
"operation": "shift",
"spec": {
"widgets": {
"*": {
"type": {
"BAR": {
"#(2,id)": "widgets[&3].barId"
}
}
}
}
}
}, {
"operation": "shift",
"spec": {
"*": "test.&"
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"*": "=recursivelySquashNulls"
}
},
{
"operation": "shift",
"spec": {
"test": {
"*": "&"
}
}
}
]
It looks like the web application in http://jolt-demo.appspot.com/ is running old version 0.1.1 that in this case behaves a bit differently than the newer versions.
At the moment the latest version available in Maven Central seems to be 0.1.7:
https://central.sonatype.dev/artifact/com.bazaarvoice.jolt/jolt-core/0.1.7/versions
thought the Github release page only claims latest to be 0.1.6:
https://github.com/bazaarvoice/jolt/releases/tag/jolt-0.1.6
I have the impression that the very powerful and useful Jolt library is not getting all the care it deserves :(
squashNull seems to work ok in Java code when using version 0.1.7.
Here is my final transformation that works as expected with http://jolt-demo.appspot.com/ based on answer by Barbaros Özhan:
[
{
"operation": "modify-overwrite-beta",
"spec": {
"widgets": {
"*": {
"type": "=toLower"
}
}
}
},
{
"operation": "shift",
"spec": {
"widgets": {
"*": {
"type": {
"foo|bar": {
"#(2,id)": "&4.&1.&1Id"
}
}
}
}
}
},
{
"operation": "shift",
"spec": {
"widgets": {
"*": {
"#": "&2[]"
}
}
}
}
]

JOLT transformation filter a list about the first string element

I would like to search a list for the first real string element (which could be not a number) and output this.
Input :
{
"firstString": [
"0.20",
"test",
"0.30"
]
}
or
{
"firstString": [
"0.20",
"0,30",
"test"
]
}
Expected Output :
{
"readingS": "test"
}
The order of the element can change, either it comes to 2nd or 3rd placeholder. The list is maximum 3 elements long.
My thought was to go over either the last or middle element, however this does not work. The list is generated just before with the modify-overwrite-beta.
You can still use modify-overwrite-beta transformation along with toInteger conversion such as
[
//Determine the elements whether toInteger conversion is applicable for them
{
"operation": "modify-overwrite-beta",
"spec": {
"arr": "=(#(1,firstString))",
"0": "=toInteger(#(1,arr[0]))",
"1": "=toInteger(#(1,arr[1]))",
"2": "=toInteger(#(1,arr[2]))"
}
},
//The arrays are generated from the elements which are eligible to
//conversion, while not for the others
{
"operation": "shift",
"spec": {
"*": "&",
"arr": { "*": "&" }
}
},
//The arrays are removed, so all numeric ones by reversing
//key-value pairs
{
"operation": "shift",
"spec": {
"*": {
"$": "#(0)"
}
}
},
//Reverse back the pairs
{
"operation": "shift",
"spec": {
"*": {
"$": "readingS[#2]"
}
}
},
//and pick the leftmost element
{
"operation": "cardinality",
"spec": {
"*": "ONE"
}
}
]
You have some values that ended with numbers. So we can match all values that ended with the number, and we have a real string in the last.
[
{
"operation": "shift",
"spec": {
"firstString": {
"*": {
"*0": "",
"*1": "",
"*2": "",
"*3": "",
"*4": "",
"*5": "",
"*6": "",
"*7": "",
"*8": "",
"*9": "",
"*": {
"$": "readingS"
}
}
}
}
}, {
"operation": "shift",
"spec": {
"*": {
"*": "&"
}
}
}
]