Truncate every string in an array with JOLT Transformation - json

I try to truncate strings in an array with JOLT. The character "^" should be removed from any string that contains it in the first position.
Sample Input:
{
"scores": [
"^aaaa",
"^bbbb",
"cccc",
"^dddd",
"eeee"
]
}
Expected Output:
{
"scores" : [ "aaaa", "bbbb", "cccc", "dddd", "eeee" ]
}
My Spec:
[
{
"operation": "modify-overwrite-beta",
"spec": {
"splittedScores": "=split('\\^',#(1,scores))",
"scores": "=join('',#(1,splittedScores))" // <--- Not working
}
}
]

Seems you'd need one extra shift transformation sandwiched between modify transformations to tame the subarrays such as
[
{
"operation": "modify-overwrite-beta",
"spec": {
"*": "=split('\\^',#(1,&))"
}
},
{
"operation": "shift",
"spec": {
"*": {
"*": "&"
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"*": "=join('',#(1,&))"
}
},
{
"operation": "shift",
"spec": {
"*": {
"#": "scores"
}
}
}
]
the demo on the site http://jolt-demo.appspot.com/ is
or only one shift transformation will be sufficient to use(a straightforward option) such as
[
{
"operation": "shift",
"spec": {
"*": {
"*": {
"^*": {
"$(0,1)": "&3"
},
"*": {
"$": "&3"
}
}
}
}
}
]
splitting by "$(0,1)" wildcard for the object under "^*" node. &3 represents going three levels up to grab the name of the key scores .
the demo on the site http://jolt-demo.appspot.com/ is
If ^ is being removed in the first position really matters, then the second method; if all ^ characters from any strings should be removed, then the first method should be used.

Related

How to split the nested JSON in Jolt specification

pls help me out wit this jolt specification. pls help me
Notes :
Resourcename is the last element of ResourceId which will be a new attribute that we need to add to the expected output
Tags field needs to be copied and splited as mentioned in the expected output.
Input :
[
{
"ResourceId": "/subscriptions/bb842437aa4/resourceGroups/ECHLABHENKEL/providers/Microsoft.Compute/virtualMachines/pmoapps",
"Tags": "Name\": \"PMOapplication\",\"Owner\": \"Breil sathish"
},
{
"ResourceId": "/subscriptions/bb842437aa4/resourceGroups/HCLTECHLABHENKEL/providers/Microsoft.Compute/virtualMachines/pmoapps",
"Tags": "Name\": \"PMOapplication\",\"Owner\": \"Breil sathish1"
}
]
Expected Output :
[
{
"ResourceId": "/subscriptions/bb842437aa4/resourceGroups/ECHLABHENKEL/providers/Microsoft.Compute/virtualMachines/pmoapps",
"Tags": "Name\": \"PMOapplication\",\"Owner\": \"Breil sathish",
"Resourcename": "pmoapps",
"Name": "PMOapplication",
"Owner": "Breil sathish"
},
{
"ResourceId": "/subscriptions/bb842437aa4/resourceGroups/HCLTECHLABHENKEL/providers/Microsoft.Compute/virtualMachines/pmoapps",
"Tags": "Name\": \"PMOapplication\",\"Owner\": \"Breil sathish1",
"Resourcename": "pmoapps",
"Name": "PMOapplication",
"Owner": "Breil sathish1"
}
]
Thanks,
N Sathish
You can use the following transformation spec as reading the explanations stated at the beginning of each of them such as
[
{ // Convert the value of the attributes to individual arrays with the identical names
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"ResourceName": "=split('/', #(1,ResourceId))",
"tag": "=split(',', #(1,Tags))"
}
}
},
{ // Generate tag0 and tag1 attributes by splitting members of the "tag" array
"operation": "shift",
"spec": {
"*": {
"*": "&1.&",
"tag": {
"*": {
"#": "&3.&2&1"
}
}
}
}
},
{ // Split related strings by colon characters for "tag" array while deriving the last element of "ResourceName" array
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"R*": "=lastElement(#(1,&))",
"tag*": "=split(': ', #(1,&))"
}
}
},
{ // Match components of "tag" array component 1 against component 2
"operation": "shift",
"spec": {
"*": {
"*": "&1.&",
"tag*": {
"#1,&[1]": "&2.&1.#(2,&[0])"
}
}
}
},
{ // Split the values by \" character combination
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"tag*": {
"*": "=split('\"', #(1,&))"
}
}
}
},
{ // Prune undesired values for right-hand-side
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"tag*": {
"*": "=join('', #(1,&))"
}
}
}
},
{ // Prune undesired values for left-hand-side(keys)
"operation": "shift",
"spec": {
"*": {
"*": "[&1].&",
"tag*": {
"\"*\"": "[&2].&(0,1)",
"*\"": "[&2].&(0,1)"
}
}
}
}
]

Broke nested dynamic JSON array with JOLT

I'm looking for flattening nested JSON file into SQL ready format.
JSON file's content:
{
"ProductLine": [
"Product 1",
"Product 2"
],
"Purchase": 364,
"Cancel": [
140,
2
]
}
My current transformation:
[
{
"operation": "shift",
"spec": {
"*": {
"*": {
"#": "[#2].&2"
}
}
}
}
]
Desired output:
[
{
"ProductLine": "Product 1",
"Purchase": 364,
"Cancel": 140
},
{
"ProductLine": "Product 2",
"Cancel": 2
}
]
The difficulty is that arrays can change, sometimes "Cancel" can be an array or sometimes "Purchase" block can be nested.
You can use this spec:
If Purchase or cancel be an array or not, this works
[
{
"operation": "cardinality",
"spec": {
"*": "MANY"
}
},
{
"operation": "shift",
"spec": {
"ProductLine": {
"*": {
"*": {
"#1": "[&2].&3",
"#(3,Purchase[&1])": "[&2].Purchase",
"#(3,Cancel[&1])": "[&2].Cancel"
}
}
}
}
}
]
First, change all values to the array. Now you can loop on the ProductLine and get other fields from Purchase and Cancel.
Update: The following answer has been obtained in collaboration with Barbaros Özhan. Special thanks.
[
{
"operation": "cardinality",
"spec": {
"*": "MANY"
}
},
{
"operation": "shift",
"spec": {
"*": {
"*": {
"#": "[#2].&2"
}
}
}
}
]
We can pick Purchase at a different(outer) level such as
[
{
"operation": "shift",
"spec": {
"*": {
"*": {
"#": "[#2].&2"
}
},
"Purchase": "[#].&"// at two level less than the inner object
}
}
]
the demo one the site http://jolt-demo.appspot.com/ is
Edit : Considering array indeterminance for the attributes, you can use the following spec alternatively
[
{ //reform two separate objects
"operation": "shift",
"spec": {
"#": "orj",
"*": "non_array.&.#0[]"
}
},
{ // in order to keep the non-array values as the first component of the newly formed array(s)
"operation": "sort"
},
{
"operation": "shift",
"spec": {
"*": { //the topmost level
"*": { //level for the keys
"*": "&1[]" //match keys and values to convert non-arrays to arrays
}
}
}
},
{// pick the first component for the non-array(s)
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"*": "=firstElement"
}
}
},
{ // apply the original spec after having got individual array values
"operation": "shift",
"spec": {
"*": {
"*": {
"#": "[#2].&2"
}
}
}
},
{ //get rid of the attributes with null values
"operation": "modify-overwrite-beta",
"spec": {
"*": "=recursivelySquashNulls"
}
}
]
or another straightforward alternative would be using your original spec after applying cardinality spec such as
[
{
"operation": "cardinality",
"spec": {
"*": "MANY"
}
},
{
"operation": "shift",
"spec": {
"*": {
"*": {
"#": "[#2].&2"
}
}
}
}
]

JoltTransformJson - How to get column names as values

i have JSON value as below :
{
"table": "table_name",
"op_type": "U",
"before": {
"AAAA": "1-1111",
"BBBB": "2022-08-31 03:57:01"
},
"after": {
"AAAA": "1-1111",
"BBBB": "2022-08-31 10:10:34",
"DDDD": "2023-08-31 23:59:59"
}
}
I want to add column_names field like this :
,"changed_columns": "AAAA,BBBB,DDDD"
is there a way to do this?
You can use the following specs in which the main idea is to arrange the attributes so as to generate an array with unique elements within the an array by using successive shift transformation, then combine them within a modify transformation such as
[
{
// combine common key names for each respective values for the attributes
"operation": "shift",
"spec": {
"before|after": {
"*": {
"$": "&"
}
}
}
},
{
// construct an array from those newly formed keys
"operation": "shift",
"spec": {
"*": {
"$": "changed_columns"
}
}
},
{
// make them comma-separated
"operation": "modify-overwrite-beta",
"spec": {
"*": "=join(',',#(1,&))"
}
}
]
the demo on the site http://jolt-demo.appspot.com/ is
Edit : If your aim is to keep newly generated attribute along with the existing ones, then you can prefer using the following spec
[
{
"operation": "shift",
"spec": {
"*": "&", //else case
"before|after": {
"*": {
"$": "cc.&",
"#": "&2.&"
}
}
}
},
{
"operation": "shift",
"spec": {
"cc": {
"*": {
"$": "changed_columns"
}
},
"*": "&" //else case
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"changed_columns": "=join(',',#(1,&))"
}
}
]

Jolt Transform: Single Value can't be added to an Array

I have the following Jolt Spec
[
{
"operation": "shift",
"spec": {
"*": {
"value": "[]"
}
}
},
{
"operation": "shift",
"spec": {
"*": "values"
}
},
{
"operation": "default",
"spec": {
"values": []
}
}
]
For the following Input:
[
{
"value": 175
},
{
"value": 160
}
]
I get the expected result as following:
{
"values" : [ 175, 160 ]
}
And for the following Input:
[
{
"valueNum": 175
}
]
I again get an expected result as follows:
{
"values" : [ ]
}
But for the following input :
[
{
"value": 175
}
]
I get the following output
{
"values" : 175
}
I want to have the values in an array even if there is just one element in it like below:
{
"values" : [175]
}
Could you please help me fixing my Jolt Spec to get the desired result? Thanks!
Just using a single shift transformation spec as the following one would suffice
[
{
"operation": "shift",
"spec": {
"*": {
"*": "&s[]"
}
}
}
]
where & is a substitution for the key name(value), and s is a suffix to make the word plural
the demoes on the site http://jolt-demo.appspot.com/ are
Edit : If you need an output based on seperating the keys of the attributes to be value or the others(such as valueNum), then still a single spec like the one below would be sufficient :
[
{
"operation": "shift",
"spec": {
"*": {
"value": "&s[]",
"*": "&[]"
}
}
}
]
With the following spec, I was able to achieve all my use cases
[
{
"operation": "shift",
"spec": {
"*": {
"value": "[]"
}
}
},
{
"operation": "shift",
"spec": {
"*": "values[]"
}
},
{
"operation": "default",
"spec": {
"values": []
}
}
]

JOLT transformation filter a list about the first string element

I would like to search a list for the first real string element (which could be not a number) and output this.
Input :
{
"firstString": [
"0.20",
"test",
"0.30"
]
}
or
{
"firstString": [
"0.20",
"0,30",
"test"
]
}
Expected Output :
{
"readingS": "test"
}
The order of the element can change, either it comes to 2nd or 3rd placeholder. The list is maximum 3 elements long.
My thought was to go over either the last or middle element, however this does not work. The list is generated just before with the modify-overwrite-beta.
You can still use modify-overwrite-beta transformation along with toInteger conversion such as
[
//Determine the elements whether toInteger conversion is applicable for them
{
"operation": "modify-overwrite-beta",
"spec": {
"arr": "=(#(1,firstString))",
"0": "=toInteger(#(1,arr[0]))",
"1": "=toInteger(#(1,arr[1]))",
"2": "=toInteger(#(1,arr[2]))"
}
},
//The arrays are generated from the elements which are eligible to
//conversion, while not for the others
{
"operation": "shift",
"spec": {
"*": "&",
"arr": { "*": "&" }
}
},
//The arrays are removed, so all numeric ones by reversing
//key-value pairs
{
"operation": "shift",
"spec": {
"*": {
"$": "#(0)"
}
}
},
//Reverse back the pairs
{
"operation": "shift",
"spec": {
"*": {
"$": "readingS[#2]"
}
}
},
//and pick the leftmost element
{
"operation": "cardinality",
"spec": {
"*": "ONE"
}
}
]
You have some values that ended with numbers. So we can match all values that ended with the number, and we have a real string in the last.
[
{
"operation": "shift",
"spec": {
"firstString": {
"*": {
"*0": "",
"*1": "",
"*2": "",
"*3": "",
"*4": "",
"*5": "",
"*6": "",
"*7": "",
"*8": "",
"*9": "",
"*": {
"$": "readingS"
}
}
}
}
}, {
"operation": "shift",
"spec": {
"*": {
"*": "&"
}
}
}
]