JoltTransformJson - Json Transformation - json

i have JSON value as below :
{
"table": "table_name",
"op_type": "U",
"before": {
"AAAA": "1-1111",
"BBBB": "2022-08-31 03:57:01",
"CCCC": "2023-08-31 23:59:59"
},
"after": {
"AAAA": "1-1112",
"BBBB": "2022-08-31 10:10:34"
}
}
i want to do this how can i do?
{
"AAAA": "1-1112",
"BBBB": "2022-08-31 10:10:34",
"CCCC": "2023-08-31 23:59:59"
"changed_columns": "AAAA, BBBB"
}
AAAA: "If you have after.AAAA, take AAAA else before.AAAA", BBBB: "If you have after.BBBB, take BBBB else before.BBBB.
AND I want to add changed_columns field like this :
,"changed_columns": "AAAA, BBBB"
is there a way to do this?

You can use shift operation and getting after values for the first. and then you can using before values. So if keys match with together you have an array with two element.
Now You can get first element to getting after values with modify-overwrite-beta operations and =firstElement function.
[
{
"operation": "shift",
"spec": {
"after": {
"*": {
"$": "changed_columns[]",
"#(1,&)": "&1"
}
},
"before": {
"*": "&"
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"*": "=firstElement(#(1,&))",
"changed_columns": "=join(', ',#(1,changed_columns))"
}
}
]

You can use
cardinality spec lately after using "after|before" as the key in this order to determine the precedence
exchange key-value pairs consecutively twice to determine whether
really changed the components in order to form "changed_columns"
such as
[
{
// multiplex the attributes in order to generate three independent groups
"operation": "shift",
"spec": {
"after|before": { // this order is important to determine the precedence in the upcoming cardinality spec
"*": {
"#": "&",
"#(0)": "l.&",
"*": {
"#1": "f.&2"
}
}
}
}
},
{
// determine whether before vs. after values equal through this and next two specs
"operation": "modify-overwrite-beta",
"spec": {
"l": {
"*": "=lastElement(#(1,&))"
},
"f": {
"*": "=firstElement(#(1,&))"
}
}
},
{
"operation": "shift",
"spec": {
"*": "&",
"l|f": {
"*": {
"$": "lf.#(0)"
}
}
}
},
{
"operation": "shift",
"spec": {
"*": "&",
"lf": {
"*": {
"$": "&2.#(0)"
}
}
}
},
{
// construct an array from those newly formed keys
"operation": "shift",
"spec": {
"*": "&",
"lf": {
"*": {
"$": "changed_columns"
}
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"changed_columns": "=join(', ',#(1,&))"
}
},
{
"operation": "cardinality",
"spec": {
"*": "ONE"
}
},
{
"operation": "sort"
}
]

Related

How can we extract key value from JSON array literal in Jolt Spec/ Transformation

I have a use case, where we're getting JSON data in complicated fashion, i have translate this object as JSON array and now am unable to extract the key/value from the array, and also there is no guarantee that every time we'll receive the Tags.
Please suggest how we can extract this from array.
Input JSON :
[
{
"SourceId": "/Apple/bb842437dd4/sourceGroups/ALPHABAT/providers/Mobile.com/phone/isp",
"Tags": "Name\": \"OMapplication\",\"Owner\": \"Breily",
"Tagscopy": [
"Name\": \"OMapplication\"",
"\"Owner\": \"Breily"
],
"ResourceName": "omapps"
},
{
"SourceId": "/Apple/bb842437dd4/sourceGroups/ALPHABAT/providers/Mobile.com/phone/isp",
"Tags": "mobile-source-usage\": \"apple-cloud",
"Tagscopy": [
"mobile-source-usage\": \"apple-cloud"
],
"ResourceName": "omapps"
}
]
Need to do operation on "Tagscopy", we're getting random data in this.
Desired JSON :
[
{
"SourceId": "/Apple/bb842437dd4/sourceGroups/ALPHABAT/providers/Mobile.com/phone/isp",
"Tags": "Name\": \"OMapplication\",\"Owner\": \"Breily",
"Name": "OMapplication",
"Owner": "Breily",
"ResourceName": "omapps"
},
{
"SourceId": "/Apple/bb842437dd4/sourceGroups/ALPHABAT/providers/Mobile.com/phone/isp",
"Tags": "mobile-source-usage\": \"apple-cloud",
"mobile-source-usage": "apple-cloud",
"ResourceName": "omapps"
}
]
Jolt Spec Used :
[
{
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"SourceIdcopy": "=split('/', #(1,SourceId))",
"Tagscopy": "=split(',', #(1,Tags))",
"SourceName": "=lastElement(#(1,SourceIdcopy))"
}
}
},
{
"operation": "remove",
"spec": {
"*": {
"SourceIdcopy": ""
}
}
},
{
"operation": "modify-default-beta",
"spec": {
"*": {
"*": "&",
"Tagscopy": {
"*": "&"
}
}
}
}
]
You can use the following transformation spec
[
{// Split members of the "Tagscopy" array with integer(0,1) suffixed keys
"operation": "shift",
"spec": {
"*": {
"*": "&1.&",
"Tagscopy": {
"*": {
"#": "&3.&2&1"
}
}
}
}
},
{// Split related strings by colon characters
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"Tagscopy*": "=split(': ', #(1,&))"
}
}
},
{// Match components of those array component1 against component2
"operation": "shift",
"spec": {
"*": {
"*": "&1.&",
"Tagscopy*": {
"#1,&[1]": "&2.&1.#(2,&[0])"
}
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"Tagscopy*": {
"*": "=split('\"', #(1,&))"
}
}
}
},
{// Prune undesired values for right-hand-side
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"Tagscopy*": {
"*": "=join('', #(1,&))"
}
}
}
},
{// Prune undesired values for left-hand-side(keys)
"operation": "shift",
"spec": {
"*": {
"*": "[&1].&",
"Tagscopy*": {
"\"*\"": "[&2].&(0,1)",
"*\"": "[&2].&(0,1)"
}
}
}
}
]

How to split the nested JSON in Jolt specification

pls help me out wit this jolt specification. pls help me
Notes :
Resourcename is the last element of ResourceId which will be a new attribute that we need to add to the expected output
Tags field needs to be copied and splited as mentioned in the expected output.
Input :
[
{
"ResourceId": "/subscriptions/bb842437aa4/resourceGroups/ECHLABHENKEL/providers/Microsoft.Compute/virtualMachines/pmoapps",
"Tags": "Name\": \"PMOapplication\",\"Owner\": \"Breil sathish"
},
{
"ResourceId": "/subscriptions/bb842437aa4/resourceGroups/HCLTECHLABHENKEL/providers/Microsoft.Compute/virtualMachines/pmoapps",
"Tags": "Name\": \"PMOapplication\",\"Owner\": \"Breil sathish1"
}
]
Expected Output :
[
{
"ResourceId": "/subscriptions/bb842437aa4/resourceGroups/ECHLABHENKEL/providers/Microsoft.Compute/virtualMachines/pmoapps",
"Tags": "Name\": \"PMOapplication\",\"Owner\": \"Breil sathish",
"Resourcename": "pmoapps",
"Name": "PMOapplication",
"Owner": "Breil sathish"
},
{
"ResourceId": "/subscriptions/bb842437aa4/resourceGroups/HCLTECHLABHENKEL/providers/Microsoft.Compute/virtualMachines/pmoapps",
"Tags": "Name\": \"PMOapplication\",\"Owner\": \"Breil sathish1",
"Resourcename": "pmoapps",
"Name": "PMOapplication",
"Owner": "Breil sathish1"
}
]
Thanks,
N Sathish
You can use the following transformation spec as reading the explanations stated at the beginning of each of them such as
[
{ // Convert the value of the attributes to individual arrays with the identical names
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"ResourceName": "=split('/', #(1,ResourceId))",
"tag": "=split(',', #(1,Tags))"
}
}
},
{ // Generate tag0 and tag1 attributes by splitting members of the "tag" array
"operation": "shift",
"spec": {
"*": {
"*": "&1.&",
"tag": {
"*": {
"#": "&3.&2&1"
}
}
}
}
},
{ // Split related strings by colon characters for "tag" array while deriving the last element of "ResourceName" array
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"R*": "=lastElement(#(1,&))",
"tag*": "=split(': ', #(1,&))"
}
}
},
{ // Match components of "tag" array component 1 against component 2
"operation": "shift",
"spec": {
"*": {
"*": "&1.&",
"tag*": {
"#1,&[1]": "&2.&1.#(2,&[0])"
}
}
}
},
{ // Split the values by \" character combination
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"tag*": {
"*": "=split('\"', #(1,&))"
}
}
}
},
{ // Prune undesired values for right-hand-side
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"tag*": {
"*": "=join('', #(1,&))"
}
}
}
},
{ // Prune undesired values for left-hand-side(keys)
"operation": "shift",
"spec": {
"*": {
"*": "[&1].&",
"tag*": {
"\"*\"": "[&2].&(0,1)",
"*\"": "[&2].&(0,1)"
}
}
}
}
]

Broke nested dynamic JSON array with JOLT

I'm looking for flattening nested JSON file into SQL ready format.
JSON file's content:
{
"ProductLine": [
"Product 1",
"Product 2"
],
"Purchase": 364,
"Cancel": [
140,
2
]
}
My current transformation:
[
{
"operation": "shift",
"spec": {
"*": {
"*": {
"#": "[#2].&2"
}
}
}
}
]
Desired output:
[
{
"ProductLine": "Product 1",
"Purchase": 364,
"Cancel": 140
},
{
"ProductLine": "Product 2",
"Cancel": 2
}
]
The difficulty is that arrays can change, sometimes "Cancel" can be an array or sometimes "Purchase" block can be nested.
You can use this spec:
If Purchase or cancel be an array or not, this works
[
{
"operation": "cardinality",
"spec": {
"*": "MANY"
}
},
{
"operation": "shift",
"spec": {
"ProductLine": {
"*": {
"*": {
"#1": "[&2].&3",
"#(3,Purchase[&1])": "[&2].Purchase",
"#(3,Cancel[&1])": "[&2].Cancel"
}
}
}
}
}
]
First, change all values to the array. Now you can loop on the ProductLine and get other fields from Purchase and Cancel.
Update: The following answer has been obtained in collaboration with Barbaros Özhan. Special thanks.
[
{
"operation": "cardinality",
"spec": {
"*": "MANY"
}
},
{
"operation": "shift",
"spec": {
"*": {
"*": {
"#": "[#2].&2"
}
}
}
}
]
We can pick Purchase at a different(outer) level such as
[
{
"operation": "shift",
"spec": {
"*": {
"*": {
"#": "[#2].&2"
}
},
"Purchase": "[#].&"// at two level less than the inner object
}
}
]
the demo one the site http://jolt-demo.appspot.com/ is
Edit : Considering array indeterminance for the attributes, you can use the following spec alternatively
[
{ //reform two separate objects
"operation": "shift",
"spec": {
"#": "orj",
"*": "non_array.&.#0[]"
}
},
{ // in order to keep the non-array values as the first component of the newly formed array(s)
"operation": "sort"
},
{
"operation": "shift",
"spec": {
"*": { //the topmost level
"*": { //level for the keys
"*": "&1[]" //match keys and values to convert non-arrays to arrays
}
}
}
},
{// pick the first component for the non-array(s)
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"*": "=firstElement"
}
}
},
{ // apply the original spec after having got individual array values
"operation": "shift",
"spec": {
"*": {
"*": {
"#": "[#2].&2"
}
}
}
},
{ //get rid of the attributes with null values
"operation": "modify-overwrite-beta",
"spec": {
"*": "=recursivelySquashNulls"
}
}
]
or another straightforward alternative would be using your original spec after applying cardinality spec such as
[
{
"operation": "cardinality",
"spec": {
"*": "MANY"
}
},
{
"operation": "shift",
"spec": {
"*": {
"*": {
"#": "[#2].&2"
}
}
}
}
]

Converting the key variable Case to Camel Case using Jolt

I am getting data from the database sources in the following format with capital keys.
I want to convert keys in camel case using the JOLT transformation.
json which I may be getting does not contains fixed key value pairs. Key can vary case-to-case basis
input json
[
{
"LAST_UPDT_TS": "2018-05-21 07:52:06.0",
"RTRV_TS": "2023-02-08 06:03:03.932108",
"DOC_ID": "1-102GJ8CY",
"PARENT_ASSET_DOC_ID": null,
"ASSET_STATUS": "Inactive",
"CAGE_NUM": "SUBZ6G"
},
{
"LAST_UPDT_TS": "2020-09-09 22:28:25.0",
"RTRV_TS": "2023-02-08 06:03:03.932108",
"DOC_ID": "1-102MDPE7",
"PARENT_ASSET_DOC_ID": null,
"ASSET_STATUS": "Active",
"CAGE_NUM": "012210"
}
]
Expected output
[
{
"lastUpdtTs": "2018-05-21 07:52:06.0",
"rtrvTs": "2023-02-08 06:03:03.932108",
"docId": "1-102GJ8CY",
"ParentAssetDocId": null,
"AssetStatus": "Inactive",
"CageNum": "SUBZ6G"
},
{
"lastUpdtTs": "2020-09-09 22:28:25.0",
"retrieveTimestamp": "2023-02-08 06:03:03.932108",
"docId": "1-102MDPE7",
"ParentAssetDocId": null,
"AssetStatus": "Active",
"CageNum": "012210"
}
]
You can use the following spec
[// Preparation for the next spec for assigning default "null" for null values of which the attributes won't vanish
{
"operation": "default",
"spec": {
"*": {
"*": "null"
}
}
},
{// Exchange key-value pairs
"operation": "shift",
"spec": {
"*": {
"*": {
"$": "&2.&1.#(0)"
}
}
}
},
{// Split values by underscores in order to convert them to independent arrays for each attribute
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"*": {
"*": "=split('_',#(1,&))"
}
}
}
},
{// Convert all values to lowercase letters
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"*": {
"*": {
"*": "=toLower"
}
}
}
}
},
{// Enumerate each components of the arrays
"operation": "shift",
"spec": {
"*": {
"*": {
"*": {
"*": "&3.&2.&1.&"
}
}
}
}
},
{// Split those components letterwise
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"*": {
"*": {
"*": "=split('',#(1,&))"
}
}
}
}
},
{// Enumerate each components of the arrays
"operation": "shift",
"spec": {
"*": {
"*": {
"*": {
"*": {
"*": "&4.&3.&2.&1.&"
}
}
}
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"*": {// Convert only the first letters of the derived pieces to uppercase letters by use of 0th(zeroth) index
"*": {
"*": {
"*": {
"0": "=toUpper"
}
}
}
}
}
},
{// Rearrange the elements to prepare to combine the pieces
"operation": "shift",
"spec": {
"*": {
"*": {
"*": {
"*": {
"*": "&4.&3.&2"
}
}
}
}
}
},
{// Combine the pieces in order to get new values
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"*": {
"*": "=join('',#(1,&))"
}
}
}
},
{
"operation": "shift",
"spec": {// Make values keys, while keys to values
"*": {
"*": {
"*": {
"$": "[&3].#(0)"
}
}
}
}
},
{
"operation": "shift",
"spec": {
"*": {// get back again the real null values
"*": {
"null": "[&2].&1",
"*": {
"#1": "[&3].&2"
}
}
}
}
}
]
There are two immediate handicaps :
Disappearing null values during key-value exchange. Used the first and the last specs to handle this issue.
Identical values which would prevent separation of keys after applied key-value exchange. An extra level of deepness added to specs in order to handle this.

JoltTransformJson - How to get column names as values

i have JSON value as below :
{
"table": "table_name",
"op_type": "U",
"before": {
"AAAA": "1-1111",
"BBBB": "2022-08-31 03:57:01"
},
"after": {
"AAAA": "1-1111",
"BBBB": "2022-08-31 10:10:34",
"DDDD": "2023-08-31 23:59:59"
}
}
I want to add column_names field like this :
,"changed_columns": "AAAA,BBBB,DDDD"
is there a way to do this?
You can use the following specs in which the main idea is to arrange the attributes so as to generate an array with unique elements within the an array by using successive shift transformation, then combine them within a modify transformation such as
[
{
// combine common key names for each respective values for the attributes
"operation": "shift",
"spec": {
"before|after": {
"*": {
"$": "&"
}
}
}
},
{
// construct an array from those newly formed keys
"operation": "shift",
"spec": {
"*": {
"$": "changed_columns"
}
}
},
{
// make them comma-separated
"operation": "modify-overwrite-beta",
"spec": {
"*": "=join(',',#(1,&))"
}
}
]
the demo on the site http://jolt-demo.appspot.com/ is
Edit : If your aim is to keep newly generated attribute along with the existing ones, then you can prefer using the following spec
[
{
"operation": "shift",
"spec": {
"*": "&", //else case
"before|after": {
"*": {
"$": "cc.&",
"#": "&2.&"
}
}
}
},
{
"operation": "shift",
"spec": {
"cc": {
"*": {
"$": "changed_columns"
}
},
"*": "&" //else case
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"changed_columns": "=join(',',#(1,&))"
}
}
]