Multiplication of values in Jolt Transformation Spec - json

I have input as shown below, If timeMeasurementUnitCode="DAY" then value I need to calculate as value*24*60, if timeMeasurementUnitCode="HOUR" then value value*60
I see there is one similar use case people are looking for, the ticket is opened on Jul 17, 2019: https://github.com/bazaarvoice/jolt/issues/832
Issue 832 seems still open now
Input:
{
"freightCharacteristics": {
"transitDuration": {
"timeMeasurementUnitCode": "DAY",
"value": 6
},
"loadingDuration": {
"timeMeasurementUnitCode": "HOUR",
"value": 6
}
}
}
Expected output: Based on the constant provided in the input as DAY or HOUR, I need to multiple no.of days*hours*minutes to arrive at my requirement as shown below.
{
"transitDurationInMinutes" : 8640,
"loadingDurationInMinutes" : 360
}
]
As per the suggestion from #mattyb, I used divide function to achieve what want to achieve by multiplication, logic is updated as below.
[
{
"operation": "shift",
"spec": {
"freightCharacteristics": {
"transitDuration": {
"timeMeasurementUnitCode": {
"DAY": {
"#(2,value)": "transitDurationInMinutes"
},
"*": {
"#(2,value)": "transitDurationInMinutes"
}
}
},
"loadingDuration": {
"timeMeasurementUnitCode": {
"HOUR": {
"#(2,value)": "loadingDurationInMinutes"
},
"*": {
"#(2,value)": "loadingDurationInMinutes"
}
}
}
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"transitDurationInMinutes": "=divide(#(1,transitDurationInMinutes),0.00069444444)",
"loadingDurationInMinutes": "=divide(#(1,loadingDurationInMinutes),0.01666666666)"
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"transitDurationInMinutes": "=toInteger",
"loadingDurationInMinutes": "=toInteger"
}
}
]

Another option would be making DAY and HOUR literals object keys respectively in order to pick them conditionally within the next step such as
[
{
"operation": "shift",
"spec": {
"*": {
"*": {
"value": "#(1,timeMeasurementUnitCode).&"
}
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"DAY": {
"val": "=divide(1,#(1,value))",// 1/6 will be used to simulate a multiplication
"div": "=divide(1440,#(1,val))",
"transitDurationInMinutes": "=toInteger(#(1,div))"// to eliminate decimal representation which ends with ".0"
},
"HOUR": {
"val": "=divide(1,#(1,value))",
"div": "=divide(60,#(1,val))",
"loadingDurationInMinutes": "=toInteger(#(1,div))"
}
}
},
{// get rid of object keys
"operation": "shift",
"spec": {
"*": {
"*Duration*": "&"
}
}
}
]
the demo on the site http://jolt-demo.appspot.com/ is

Related

How to split the nested JSON in Jolt specification

pls help me out wit this jolt specification. pls help me
Notes :
Resourcename is the last element of ResourceId which will be a new attribute that we need to add to the expected output
Tags field needs to be copied and splited as mentioned in the expected output.
Input :
[
{
"ResourceId": "/subscriptions/bb842437aa4/resourceGroups/ECHLABHENKEL/providers/Microsoft.Compute/virtualMachines/pmoapps",
"Tags": "Name\": \"PMOapplication\",\"Owner\": \"Breil sathish"
},
{
"ResourceId": "/subscriptions/bb842437aa4/resourceGroups/HCLTECHLABHENKEL/providers/Microsoft.Compute/virtualMachines/pmoapps",
"Tags": "Name\": \"PMOapplication\",\"Owner\": \"Breil sathish1"
}
]
Expected Output :
[
{
"ResourceId": "/subscriptions/bb842437aa4/resourceGroups/ECHLABHENKEL/providers/Microsoft.Compute/virtualMachines/pmoapps",
"Tags": "Name\": \"PMOapplication\",\"Owner\": \"Breil sathish",
"Resourcename": "pmoapps",
"Name": "PMOapplication",
"Owner": "Breil sathish"
},
{
"ResourceId": "/subscriptions/bb842437aa4/resourceGroups/HCLTECHLABHENKEL/providers/Microsoft.Compute/virtualMachines/pmoapps",
"Tags": "Name\": \"PMOapplication\",\"Owner\": \"Breil sathish1",
"Resourcename": "pmoapps",
"Name": "PMOapplication",
"Owner": "Breil sathish1"
}
]
Thanks,
N Sathish
You can use the following transformation spec as reading the explanations stated at the beginning of each of them such as
[
{ // Convert the value of the attributes to individual arrays with the identical names
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"ResourceName": "=split('/', #(1,ResourceId))",
"tag": "=split(',', #(1,Tags))"
}
}
},
{ // Generate tag0 and tag1 attributes by splitting members of the "tag" array
"operation": "shift",
"spec": {
"*": {
"*": "&1.&",
"tag": {
"*": {
"#": "&3.&2&1"
}
}
}
}
},
{ // Split related strings by colon characters for "tag" array while deriving the last element of "ResourceName" array
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"R*": "=lastElement(#(1,&))",
"tag*": "=split(': ', #(1,&))"
}
}
},
{ // Match components of "tag" array component 1 against component 2
"operation": "shift",
"spec": {
"*": {
"*": "&1.&",
"tag*": {
"#1,&[1]": "&2.&1.#(2,&[0])"
}
}
}
},
{ // Split the values by \" character combination
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"tag*": {
"*": "=split('\"', #(1,&))"
}
}
}
},
{ // Prune undesired values for right-hand-side
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"tag*": {
"*": "=join('', #(1,&))"
}
}
}
},
{ // Prune undesired values for left-hand-side(keys)
"operation": "shift",
"spec": {
"*": {
"*": "[&1].&",
"tag*": {
"\"*\"": "[&2].&(0,1)",
"*\"": "[&2].&(0,1)"
}
}
}
}
]

JOLT concatenate two strings

I have this collection of driver details and I want to use jolt to copy the properties to concatenate first name and last name within a single property called name
{
"drivers_details": [
{
"first_name": "Doru",
"last_name": "Petre",
"cnp": "1641201390687",
"id_series": "RK",
"id_number": "123456"
},
{
"first_name": "GIGI",
"last_name": "FANE",
"cnp": "16412013906871",
"id_series": "RK",
"id_number": "1234567"
}
]
}
I am using this spec for the shift transformation spec
[
{
"operation": "shift",
"spec": {
"drivers_details": {
"*": {
"cnp": "body.drivers[&1].tin",
"id_series": "body.drivers[&1].id_series",
"id_number": "body.drivers[&1].id_number"
}
}
}
}
]
and this one for modify-overwrite, but it doesn't work how I was expecting
{
"operation": "modify-overwrite-beta",
"spec": {
"drivers_details": {
"*": {
"name": "=concat(#(2,first_name),' ',#(2,last_name))"
}
}
}
}
What is the correct way of doing this?
You can use modify transformation as in the following case, and get rid of names other than newly formed attribute name such as
[
{
"operation": "modify-overwrite-beta",
"spec": {
"drivers_details": {
"*": {
"name": "=concat(#(1,first_name),' ',#(1,last_name))"
}
}
}
},
{
"operation": "remove",
"spec": {
"drivers_details": {
"*": {
"*name": ""
}
}
}
},
{ // this last spec stands only for ordering of the attributes if the ordering and putting the "name" at the top matters.
// So, you might get rid of the "remove" transformation above if this is used. Otherwise keep it.
"operation": "shift",
"spec": {
"drivers_details": {
"*": {
"name": "&2[&1].&",
"cnp": "&2[&1].&",
"id*": "&2[&1].&"
}
}
}
}
]
where those attributes(first/last_name) stays right-hand-side at the current level, so just only need to traverse 1 colon(:) which appears as #(1,..)
You can use this spec without the remove operation:
[
{
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"*": {
"name": "=concat(#(1,first_&),' ',#(1,last_&))"
}
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"*": {
"name": "&2[&1].&",
"cnp": "&2[&1].&",
"id_*": "&2[&1].&"
}
}
}
}
]

Unexpected null elements in output array

EDIT: Now after 16 hours break I realized I have been misreading the results for hours - I didn't notice that there is null in all cases, so the behavior is consistent unlike I claim in this question (facepalm). However I decided not to delete this question but only close it as there is already useful answers.
Consider the following input JSON:
{
"widgets": [
{
"type": "FOO",
"id": "F1"
},
{
"type": "ZAP",
"id": "Z1"
},
{
"type": "BAR",
"id": "B1"
}
]
}
The following transformation:
[
{
"operation": "shift",
"spec": {
"widgets": {
"*": {
"type": {
"FOO": {
"#(2,id)": "widgets[&3].fooId"
},
"BAR": {
"#(2,id)": "widgets[&3].barId"
}
}
}
}
}
}
]
creates the expected (correct) output. EDIT I misread this output - I didn't realize there is 3 elements where one element is null but I though there is only 2 non-null elements:
{
"widgets": [
{
"fooId": "F1"
},
null,
{
"barId": "B1"
}
]
}
However the following transformation creates the unexpected output:
[
{
"operation": "shift",
"spec": {
"widgets": {
"*": {
"type": {
"BAR": {
"#(2,id)": "widgets[&3].barId"
}
}
}
}
}
}
]
This is the actual (wrong) output:
{
"widgets": [
null,
null,
{
"barId": "B1"
}
]
}
This is the expected (correct) output:
{
"widgets": [
{
"barId": "B1"
}
]
}
Why sometimes there is null elements in the widgets array and sometimes there is not? How I can avoid them?
Based on the many other SO questions the following operation:
{
"operation": "modify-overwrite-beta",
"spec": {
"*": "=recursivelySquashNulls"
}
}
should remove the null values, but why those are not removed in the following transformation?
[
{
"operation": "shift",
"spec": {
"widgets": {
"*": {
"type": {
"BAR": {
"#(2,id)": "widgets[&3].barId"
}
}
}
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"*": "=recursivelySquashNulls"
}
}
]
I'm observing this behavior in http://jolt-demo.appspot.com/
The first output keeps null as well(I don't know if it should appear in the desired output). Since the index(which is 1) of the object having "type": "ZAP" stays in the middle of the indexes(0,1,2), but not at the end, then doesn't vanish at that level.
Btw, squashNulls especially no impact for the objects nested within array. Indeed, it has some bugs and documented as fixed for version 0.1.6.
You can rather use the following spec which doesn't have square-bracketed indexes such as
[
{
"operation": "shift",
"spec": {
"*": {
"*": {
"type": {
"FOO|BAR": {
"#(2,id)": "&4.&1.&1Id"
}
}
}
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"*": {
"#": "&2[]"
}
}
}
}
]
This spec will help you resolve this query :
1)
Actually recursivelySquashNulls will work if you keep the array in a object for eg: I kept it under test object and made it to previous one.
This spec will resolve your issue :
[
{
"operation": "shift",
"spec": {
"widgets": {
"*": {
"type": {
"BAR": {
"#(2,id)": "widgets[&3].barId"
}
}
}
}
}
}, {
"operation": "shift",
"spec": {
"*": "test.&"
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"*": "=recursivelySquashNulls"
}
},
{
"operation": "shift",
"spec": {
"test": {
"*": "&"
}
}
}
]
It looks like the web application in http://jolt-demo.appspot.com/ is running old version 0.1.1 that in this case behaves a bit differently than the newer versions.
At the moment the latest version available in Maven Central seems to be 0.1.7:
https://central.sonatype.dev/artifact/com.bazaarvoice.jolt/jolt-core/0.1.7/versions
thought the Github release page only claims latest to be 0.1.6:
https://github.com/bazaarvoice/jolt/releases/tag/jolt-0.1.6
I have the impression that the very powerful and useful Jolt library is not getting all the care it deserves :(
squashNull seems to work ok in Java code when using version 0.1.7.
Here is my final transformation that works as expected with http://jolt-demo.appspot.com/ based on answer by Barbaros Özhan:
[
{
"operation": "modify-overwrite-beta",
"spec": {
"widgets": {
"*": {
"type": "=toLower"
}
}
}
},
{
"operation": "shift",
"spec": {
"widgets": {
"*": {
"type": {
"foo|bar": {
"#(2,id)": "&4.&1.&1Id"
}
}
}
}
}
},
{
"operation": "shift",
"spec": {
"widgets": {
"*": {
"#": "&2[]"
}
}
}
}
]

Jolt to check non-null value and then add a new field

Below is the json input file.
{
"data": {
"no1": 12345,
"cp1": {
"cp1_sub1": "sub1",
"cp2_sub2": "sub2"
},
"cp2": null
}
}
the expectation is like adding a "no1" field inside the "cp1" and "cp2" if it is non-null object.
I have tried with following 2 ways but non of it is working. Is there any way to check null or add a null value with a new key in jolt?
Try 1:
[
{
"operation": "shift",
"spec": {
"data": {
"*": "data.&",
"cp1": {
"#(1,no1)": "data.cp1.no1",
"*": "data.cp1.&"
},
"cp2": {
"#(1,no1)": "data.cp2.no1",
"*": "data.cp2.&"
}
}
}
}
]
Here the problem is it adds "no1" in "cp2".
Try 2:
[
{
"operation": "shift",
"spec": {
"data": {
"*": "data.&",
"cp1": {
"#(1,no1)": "data.cp1.no1",
"*": "data.cp1.&"
},
"cp2": {
"*": {
"#(1,no1)": "data.cp2.no1",
"*": "data.cp2.&"
}
}
}
}
}
]
Here the problem is "cp2" it self removed. If this is correct then how we can add back "cp2": null
If the input is an array like this.
{
"abc": "def",
"data": [
{
"no1": 12345,
"cp1": {
"cp1_sub1": "sub1",
"cp2_sub2": "sub2"
},
"cp2": null
},
{
"no1": 56789,
"cp1": null,
"cp2": {
"cp3_sub1": "sub1",
"cp3_sub2": "sub2"
}
}
]
}
I tried with the same way with #Barbaros answer. but the last step is not working for the array part. Considering cp1 and cp2 as the fixed key and not a dynamic.
[
{
// multiplex the current JSON value in order to use one for logical comparisons later
"operation": "shift",
"spec": {
"data": {
"*": {
"*": "data.[&1].&",
"#(0,cp1)": "data.[&1].temp-cp1",
"#(0,cp2)": "data.[&1].temp-cp2"
}
}
}
},
{
// determine whether the value of the attribute/object is "null"
"operation": "modify-overwrite-beta",
"spec": {
"data": {
"*": {
"temp-cp1": ["=toString", "NuLllLL"],
"temp-cp2": ["=toString", "NuLllLL"]
}
}
}
}
]
We should generate a comparison identifier in order to be prepared for upcoming conditional which will be held within a consequent transformation
[
{
// multiplex the current JSON value in order to use one for logical comparisons later
"operation": "shift",
"spec": {
"data": {
"*": "&1.&",
"#": "New_&1.&"
}
}
},
{
// determine whether the value of the attribute/object is "null"
"operation": "modify-overwrite-beta",
"spec": {
"New_data": {
"data": {
"cp*": ["=toString", "NuLllLL"]
}
}
}
},
{
// use conditional logic based on the results determined within the previous transformation
"operation": "shift",
"spec": {
"New_data": {
"data": {
"cp*": {
"NuLllLL": {
"#(4,data.&1)": "&3.&2"
},
"*": {
"#1": "&3.&2.new_&2",
"#(4,data.no1)": "&3.&2.no1"
},
"*}": { // check out if it's an object
"#(4,data.&1)": "&3.&2",
"#(4,data.no1)": "&3.&2.no1"
}
},
"*": "&1.&"
}
}
}
}
]
the demo on the site http://jolt-demo.appspot.com/ is

conditional JSON transformation in JOLT

I need to process response from SOLR service that looks like following:
{
"data": {
"distanceUnits": "mi", //it can be "mi", "MI", "miles", "km", "KM", etc.
"solrResponse": {
"header": {
"found": 32,
"retrieved": 10,
... //there can be other things like timestamp
},
"results": [
{
"matchScore": "08768",
"indicators" [{...}],
"location": {
... //there are about hundred attributes
"distance": "2.7649" //distance always in km
"similarity": "0.342"
},
...
]
}
The transformation need to return everything from solrResponse practically intact except two things:
similarity need to be renamed to similarityScore and moved up next to matchScore.
If distanceUnit is specified as miles, distance need to be converted to miles (i.e. divided by 1.609)
distance value is to berounded to 2 digits after decimal point and concatenated with distanceUnit.
I have created the following spec:
[
{
"operation": "modify-overwrite-beta",
"spec": {
"data": {
"unit1": "=toLower(#(1,distanceUnit))",
"unit2": "=substring(#(1,unit1),0,1)",
"solrResponse": {
"results": {
"": {
"dist1": "=divideAndRound(2,#(1,distance),0.6215)",
"distanceMiles": "=concat(#(1,dist1), ' ', #(2,distanceUnit))",
"dist2": "=divideAndRound(2,#(1,distance),1.0)",
"distanceKm": "=concat(#(1,dist2), ' ', #(2,distanceUnit))"
}
}
}
}
}
},
{
"operation": "shift",
"spec": {
"data": {
"solrResponse": {
"header": "&1",
"response": {
"": {
"&1.similarity": "similarityScore",
"unit2": {
"m": {
"distanceMiles": "locations.distance"
},
"": {
"distanceKm": "locations.distance"
}
},
"": "&1"
}
}
}
}
}
}
]
Unfortunately it does not work. Please help me.
The Jolt doesn't have a function such as multiply or product, but divide and divideAndRound to be used in a modify transformation spec. Then, we'll have a conditional to filer out whether the provided value for distanceUnit attribute exists within the list(mi,m,Mi,MI) or not such as
[
{
"operation": "modify-overwrite-beta",
"spec": {
"*": { // this outer one represents "locations" list(if there's no other outer level list or object, otherwise replace "*" with the key name "locations")
"*": { // this one stands for "locations" list's indexes
"distanceinMiles_": "=divideAndRound(2,#(1,distance),0.6215040397762585)",
"distanceinMiles": "=concat(#(1,distanceinMiles_),' ',#(1,distanceUnit))",
"distance_": "=divideAndRound(2,#(1,distance),1)",
"distance": "=concat(#(1,distance_),' ',#(1,distanceUnit))"
}
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"*": {
"address": "&2[&1].&",
"city": "&2[&1].&",
"state": "&2[&1].&",
"postalCode": "&2[&1].&",
"distanceUnit": {
"mi|m|Mi|MI": { "#(2,distanceinMiles)": "&4[&3].distance" },
"*": { "#(2,distance)": "&4[&3].distance" }
}
}
}
}
}
]
where
#(1,distanceXx) in the first spec represents traversing one colon(:) (as a Right-Hand-Side element) in order to reach the
original level of the elements #(1,distanceXx), while
#(2,distanceXx) stands for traversing { (object opening "curly" brace) twice to reach the same.
the attributes at the indexes level of "locations" are combined by using [&1] and [&3] respectively
&2 and &4 are substituted respectively to denote the key name
"locations"
the demo on the site http://jolt-demo.appspot.com/ :
Edit (response for your last edit):
Considering the input
{
"data": {
"distanceUnit": "Mi",
"solrResponse": {
"header": {
"found": 32,
"retrieved": 10
},
"results": [
{
"matchScore": "08768",
"location": {
"distance": "2.7649",
"similarity": "0.342"
}
}
]
}
}
}
you can use the following specs
[
{
"operation": "modify-overwrite-beta",
"spec": {
"data": {
"distanceUnit": "=toLower(#(1,&))",
"solrResponse": {
"results": {
"*": {
"location": {
"dist1": "=divideAndRound(2,#(1,distance),0.6215)",
"distanceKm": "=concat(#(1,dist1), ' km')",
"dist2": "=divideAndRound(2,#(1,distance),1.0)",
"distanceMiles": "=concat(#(1,dist2), ' km')"
}
}
}
}
}
}
},
{
"operation": "shift",
"spec": {
"data": {
"*": "&1.&",
"solrResponse": {
"*": "&2.&1.&",
"results": {
"*": {
"*": "&4.&3.&2[&1].&",
"location": {
"#(4,distanceUnit)": {
"mi": { "#(2,distanceKm)": "&7.&6.&5[&4].&3.distance" },
"km": { "#(2,distanceMiles)": "&7.&6.&5[&4].distance" }
},
"similarity": "&5.&4.&3[&2].&Score"
}
}
}
}
}
}
},
{
"operation": "sort"
}
]
the demo is :