JOLT moving from LHS to RHS and flattening - json

I am trying to convert the following JSON with JOLT but struggling, any help would be appreciated.
{
"2021-04-14T00:00:00+02:00": {
"249184": {
"SRAD": null,
"T": -50.00000000000001
},
"249185": {
"SRAD": 0.46133333444595337,
"T": null
}
},
"2021-04-14T00:15:00+02:00": {
"249184": {
"SRAD": null,
"T": -50.00000000000001
},
"249185": {
"SRAD": 0.4593333303928375,
"T": null
}
}
}
Desired output: Note here that the timestamps are repeated for each deviceID along with the SRAD and T values.
{
"timestamp": "2021-04-14T00:00:00+02:00",
"deviceID": 249184,
"SRAD":null,
"T":-50.00000000000001
},
{
"timestamp": "2021-04-14T00:00:00+02:00",
"deviceID": 249185,
"SRAD":0.46133333444595337,
"T":null
},
{
"timestamp": "2021-04-14T00:15:00+02:00",
"deviceID": 249184,
"SRAD":null,
"T":-50.00000000000001
},
{
"timestamp": "2021-04-14T00:15:00+02:00",
"deviceID": 249185,
"SRAD": 0.4593333303928375,
"T":null
}
I have tried a bunch of things but keep going in circles.

Here's an alternate spec with 2 shifts that supports any number of fields inside the deviceId object:
[
{
"operation": "shift",
"spec": {
"*": {
"*": {
"$(1)": "a[].timestamp",
"$": "b[].deviceId",
"#": "c[]"
}
}
}
},
{
"operation": "shift",
"spec": {
"a": {
"*": {
"timestamp": "[&1].timestamp",
"#(2,b[&])": "[&1]"
}
},
"b": null,
"c": {
"*": {
"*": "[&1].&"
}
}
}
}
]

[
// Change null as String, as jolt will not
// process null values
{
"operation": "modify-default-beta",
"spec": {
"*": {
"*": {
"SRAD": "null",
"T": "null"
}
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"*": {
"SRAD": "&4.SRAD",
"T": "&4.T",
"$0": "&4.deviceID",
"$1": "&4.timestamp"
}
}
}
}, {
"operation": "shift",
"spec": {
"*": {
"*": {
"*": {
"#": "&"
}
}
}
}
}, {
"operation": "shift",
"spec": {
"*": {
"0": "[&1].deviceID",
"1": "[&1].timestamp",
"2": "[&1].SRAD",
"3": "[&1].T"
}
}
}
]
You cannot process the JSON having null as value, so null is replaced with "null" using modify-default-beta operation.
Input JSON is grouped by timestamp and deviceid, so deconstruct the json by assigning names to the children nodes and then construct the output JSON.

Related

How to convert string "{}" to empty dictionary {} JSON in Jolt? I am trying to replace "{}" by {} in Jolt

Input JSON value :
{
"age": [
{
"r": "r1",
"d": "{}"
}
]
}
Desired Output :
{
"age": [
{
"r": "r1",
"d": {}
}
]
}
I tried using modify-overwrite-beta but unable to replace string by empty dictionary.
You can use the following shift transformation spec in order to convert {} to null, and then modify transformation spec along with notNull function to return the desired result such as
[
{
"operation": "shift",
"spec": {
"age": {
"*": {
"*": {
"*": { "#1": "&4[&3].&2" },
"{}": "&3[&2].&1"
}
}
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"age": {
"*": {
"*": ["=notNull", {}]
}
}
}
}
]
the demo on the site http://jolt-demo.appspot.com/ is
You can use this spec:
[
{
"operation": "shift",
"spec": {
"*": {
"*": {
"*": "&2.[&1].&",
"d": {
"{}": "&3.[&2].&1.temp"
}
}
}
}
},
{
"operation": "remove",
"spec": {
"age": {
"*": {
"d": {
"*": ""
}
}
}
}
}
]

How can we extract key value from JSON array literal in Jolt Spec/ Transformation

I have a use case, where we're getting JSON data in complicated fashion, i have translate this object as JSON array and now am unable to extract the key/value from the array, and also there is no guarantee that every time we'll receive the Tags.
Please suggest how we can extract this from array.
Input JSON :
[
{
"SourceId": "/Apple/bb842437dd4/sourceGroups/ALPHABAT/providers/Mobile.com/phone/isp",
"Tags": "Name\": \"OMapplication\",\"Owner\": \"Breily",
"Tagscopy": [
"Name\": \"OMapplication\"",
"\"Owner\": \"Breily"
],
"ResourceName": "omapps"
},
{
"SourceId": "/Apple/bb842437dd4/sourceGroups/ALPHABAT/providers/Mobile.com/phone/isp",
"Tags": "mobile-source-usage\": \"apple-cloud",
"Tagscopy": [
"mobile-source-usage\": \"apple-cloud"
],
"ResourceName": "omapps"
}
]
Need to do operation on "Tagscopy", we're getting random data in this.
Desired JSON :
[
{
"SourceId": "/Apple/bb842437dd4/sourceGroups/ALPHABAT/providers/Mobile.com/phone/isp",
"Tags": "Name\": \"OMapplication\",\"Owner\": \"Breily",
"Name": "OMapplication",
"Owner": "Breily",
"ResourceName": "omapps"
},
{
"SourceId": "/Apple/bb842437dd4/sourceGroups/ALPHABAT/providers/Mobile.com/phone/isp",
"Tags": "mobile-source-usage\": \"apple-cloud",
"mobile-source-usage": "apple-cloud",
"ResourceName": "omapps"
}
]
Jolt Spec Used :
[
{
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"SourceIdcopy": "=split('/', #(1,SourceId))",
"Tagscopy": "=split(',', #(1,Tags))",
"SourceName": "=lastElement(#(1,SourceIdcopy))"
}
}
},
{
"operation": "remove",
"spec": {
"*": {
"SourceIdcopy": ""
}
}
},
{
"operation": "modify-default-beta",
"spec": {
"*": {
"*": "&",
"Tagscopy": {
"*": "&"
}
}
}
}
]
You can use the following transformation spec
[
{// Split members of the "Tagscopy" array with integer(0,1) suffixed keys
"operation": "shift",
"spec": {
"*": {
"*": "&1.&",
"Tagscopy": {
"*": {
"#": "&3.&2&1"
}
}
}
}
},
{// Split related strings by colon characters
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"Tagscopy*": "=split(': ', #(1,&))"
}
}
},
{// Match components of those array component1 against component2
"operation": "shift",
"spec": {
"*": {
"*": "&1.&",
"Tagscopy*": {
"#1,&[1]": "&2.&1.#(2,&[0])"
}
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"Tagscopy*": {
"*": "=split('\"', #(1,&))"
}
}
}
},
{// Prune undesired values for right-hand-side
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"Tagscopy*": {
"*": "=join('', #(1,&))"
}
}
}
},
{// Prune undesired values for left-hand-side(keys)
"operation": "shift",
"spec": {
"*": {
"*": "[&1].&",
"Tagscopy*": {
"\"*\"": "[&2].&(0,1)",
"*\"": "[&2].&(0,1)"
}
}
}
}
]

How to split the nested JSON in Jolt specification

pls help me out wit this jolt specification. pls help me
Notes :
Resourcename is the last element of ResourceId which will be a new attribute that we need to add to the expected output
Tags field needs to be copied and splited as mentioned in the expected output.
Input :
[
{
"ResourceId": "/subscriptions/bb842437aa4/resourceGroups/ECHLABHENKEL/providers/Microsoft.Compute/virtualMachines/pmoapps",
"Tags": "Name\": \"PMOapplication\",\"Owner\": \"Breil sathish"
},
{
"ResourceId": "/subscriptions/bb842437aa4/resourceGroups/HCLTECHLABHENKEL/providers/Microsoft.Compute/virtualMachines/pmoapps",
"Tags": "Name\": \"PMOapplication\",\"Owner\": \"Breil sathish1"
}
]
Expected Output :
[
{
"ResourceId": "/subscriptions/bb842437aa4/resourceGroups/ECHLABHENKEL/providers/Microsoft.Compute/virtualMachines/pmoapps",
"Tags": "Name\": \"PMOapplication\",\"Owner\": \"Breil sathish",
"Resourcename": "pmoapps",
"Name": "PMOapplication",
"Owner": "Breil sathish"
},
{
"ResourceId": "/subscriptions/bb842437aa4/resourceGroups/HCLTECHLABHENKEL/providers/Microsoft.Compute/virtualMachines/pmoapps",
"Tags": "Name\": \"PMOapplication\",\"Owner\": \"Breil sathish1",
"Resourcename": "pmoapps",
"Name": "PMOapplication",
"Owner": "Breil sathish1"
}
]
Thanks,
N Sathish
You can use the following transformation spec as reading the explanations stated at the beginning of each of them such as
[
{ // Convert the value of the attributes to individual arrays with the identical names
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"ResourceName": "=split('/', #(1,ResourceId))",
"tag": "=split(',', #(1,Tags))"
}
}
},
{ // Generate tag0 and tag1 attributes by splitting members of the "tag" array
"operation": "shift",
"spec": {
"*": {
"*": "&1.&",
"tag": {
"*": {
"#": "&3.&2&1"
}
}
}
}
},
{ // Split related strings by colon characters for "tag" array while deriving the last element of "ResourceName" array
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"R*": "=lastElement(#(1,&))",
"tag*": "=split(': ', #(1,&))"
}
}
},
{ // Match components of "tag" array component 1 against component 2
"operation": "shift",
"spec": {
"*": {
"*": "&1.&",
"tag*": {
"#1,&[1]": "&2.&1.#(2,&[0])"
}
}
}
},
{ // Split the values by \" character combination
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"tag*": {
"*": "=split('\"', #(1,&))"
}
}
}
},
{ // Prune undesired values for right-hand-side
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"tag*": {
"*": "=join('', #(1,&))"
}
}
}
},
{ // Prune undesired values for left-hand-side(keys)
"operation": "shift",
"spec": {
"*": {
"*": "[&1].&",
"tag*": {
"\"*\"": "[&2].&(0,1)",
"*\"": "[&2].&(0,1)"
}
}
}
}
]

Jolt to check non-null value and then add a new field

Below is the json input file.
{
"data": {
"no1": 12345,
"cp1": {
"cp1_sub1": "sub1",
"cp2_sub2": "sub2"
},
"cp2": null
}
}
the expectation is like adding a "no1" field inside the "cp1" and "cp2" if it is non-null object.
I have tried with following 2 ways but non of it is working. Is there any way to check null or add a null value with a new key in jolt?
Try 1:
[
{
"operation": "shift",
"spec": {
"data": {
"*": "data.&",
"cp1": {
"#(1,no1)": "data.cp1.no1",
"*": "data.cp1.&"
},
"cp2": {
"#(1,no1)": "data.cp2.no1",
"*": "data.cp2.&"
}
}
}
}
]
Here the problem is it adds "no1" in "cp2".
Try 2:
[
{
"operation": "shift",
"spec": {
"data": {
"*": "data.&",
"cp1": {
"#(1,no1)": "data.cp1.no1",
"*": "data.cp1.&"
},
"cp2": {
"*": {
"#(1,no1)": "data.cp2.no1",
"*": "data.cp2.&"
}
}
}
}
}
]
Here the problem is "cp2" it self removed. If this is correct then how we can add back "cp2": null
If the input is an array like this.
{
"abc": "def",
"data": [
{
"no1": 12345,
"cp1": {
"cp1_sub1": "sub1",
"cp2_sub2": "sub2"
},
"cp2": null
},
{
"no1": 56789,
"cp1": null,
"cp2": {
"cp3_sub1": "sub1",
"cp3_sub2": "sub2"
}
}
]
}
I tried with the same way with #Barbaros answer. but the last step is not working for the array part. Considering cp1 and cp2 as the fixed key and not a dynamic.
[
{
// multiplex the current JSON value in order to use one for logical comparisons later
"operation": "shift",
"spec": {
"data": {
"*": {
"*": "data.[&1].&",
"#(0,cp1)": "data.[&1].temp-cp1",
"#(0,cp2)": "data.[&1].temp-cp2"
}
}
}
},
{
// determine whether the value of the attribute/object is "null"
"operation": "modify-overwrite-beta",
"spec": {
"data": {
"*": {
"temp-cp1": ["=toString", "NuLllLL"],
"temp-cp2": ["=toString", "NuLllLL"]
}
}
}
}
]
We should generate a comparison identifier in order to be prepared for upcoming conditional which will be held within a consequent transformation
[
{
// multiplex the current JSON value in order to use one for logical comparisons later
"operation": "shift",
"spec": {
"data": {
"*": "&1.&",
"#": "New_&1.&"
}
}
},
{
// determine whether the value of the attribute/object is "null"
"operation": "modify-overwrite-beta",
"spec": {
"New_data": {
"data": {
"cp*": ["=toString", "NuLllLL"]
}
}
}
},
{
// use conditional logic based on the results determined within the previous transformation
"operation": "shift",
"spec": {
"New_data": {
"data": {
"cp*": {
"NuLllLL": {
"#(4,data.&1)": "&3.&2"
},
"*": {
"#1": "&3.&2.new_&2",
"#(4,data.no1)": "&3.&2.no1"
},
"*}": { // check out if it's an object
"#(4,data.&1)": "&3.&2",
"#(4,data.no1)": "&3.&2.no1"
}
},
"*": "&1.&"
}
}
}
}
]
the demo on the site http://jolt-demo.appspot.com/ is

Json array elements duplication using jolt

I want to know if jolt can handle the following transformation:
{
"interface": [
{
"field": "A",
"ip": [
"1.1.1.1",
"1.1.1.2"
]
},
{
"field": "B",
"ip": [
"1.1.1.3"
]
}
]
}
to
{
"interface": [
{
"field": "A",
"ip": "1.1.1.1"
},
{
"field": "A",
"ip": "1.1.1.2"
},
{
"field": "B",
"ip": "1.1.1.3"
}
]
}
i.e for a JSON array containing a child array, create one version of parent item for each of its child array item.
Can jolt do that?
This can be achieved by doing a two stage shift:
Create an array of arrays
Then flatten the array of arrays
[
{
"operation": "shift",
"spec": {
"interface": {
"*": {
"ip": {
"*": {
"#": "[&3].[&1].ip",
"#(2,field)": "[&3].[&1].field"
}
}
}
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"*": "interface.[]"
}
}
}
]
It is not complete answer, but maybe that helps you somehow or someone else could help?
[
{
"operation": "shift",
"spec": {
"interface": {
"*": {
"ip": {
"*": {
"#": "ip",
"#(2,field)": "field"
}
}
}
}
}
},
{
"operation": "shift",
"spec": {
"ip": {
"*": {
"#": "interface[&1].ip",
"#(2,field)": {
"???": "interface[&2].field" // How to take proper index of an array?
}
}
}
}
}
]