How to use shift and modify-overwrite-beta in jolt - json

I want to know how to use shift and modify-overwrite-beta for the below input. Please explain how it works with your answer.
Input:
[
{
"data": {
"firstName": "Leanne",
"lastName": "Graham",
"id": 111
}
},
{
"data": {
"firstName": "Ervin",
"lastName": "Howell",
"id": 222
}
}
]
Desired output:
{
"111": {
"fullName": "Leanne Graham"
},
"222": {
"fullName": "Ervin Howell"
}
}

You can solve that in different ways.
1. This spec is more understandable:
Order your list with the id value as a key in the shift operation: #(1,id)
Concat firstName and lastName in the modify-overwrite-beta operation: =concat(#(1,firstName),' ',#(1,lastName))
Remove unused keys in the remove operation: id, firstName, lastName
[
{
"operation": "shift",
"spec": {
"*": { // index of the array: 1, 2
"*": { // data object
"*": "#(1,id).&" // value of id in the current object: 111, 222
}
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"*": { // ids: 111, 222
"fullName": "=concat(#(1,firstName),' ',#(1,lastName))"
}
}
},
{
"operation": "remove",
"spec": {
"*": { // ids: 111, 222
"id": "",
"firstName": "",
"lastName": ""
}
}
}
]
2. This spec is shorter:
Concat firstName and lastName in the modify-overwrite-beta operation: =concat(#(1,firstName),' ',#(1,lastName))
Get the value of id as a key and put fullname in it: key #(1,id).fullName, value #(0,fullName)
[
{
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"*": {
"fullName": "=concat(#(1,firstName),' ',#(1,lastName))"
}
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"*": {
"#(0,fullName)": "#(1,id).fullName"
}
}
}
}
]
The following image can help more:

You can use those consecutive transformation specs
[
{
"operation": "shift",
"spec": {
"*": {// represents the level of the outermost object
"*": {// represents the level of the "data" object
"*": "#1,id.&"//make "id" values key of each object, "&" replicates each attribute's values
}
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"FullName": "=concat(#(1,firstName),' ',#(1,lastName))"// concatenate attributes through adding 1 level deep shift of the tree as staying on the right hand side, and name by the literal "FullName" on the left hand side
}
}
},
{
"operation": "remove",
"spec": {
"*": {
"fi*Name|la*Name|id": "" //delete attributes "id" and the ones ending with "Names" except for the "fullName"
}
}
}
]
as the names suggest shift is used to move the attributes/arrays/objects to their new places, and modify updates on the existing structure

Related

How to split the nested JSON in Jolt specification

pls help me out wit this jolt specification. pls help me
Notes :
Resourcename is the last element of ResourceId which will be a new attribute that we need to add to the expected output
Tags field needs to be copied and splited as mentioned in the expected output.
Input :
[
{
"ResourceId": "/subscriptions/bb842437aa4/resourceGroups/ECHLABHENKEL/providers/Microsoft.Compute/virtualMachines/pmoapps",
"Tags": "Name\": \"PMOapplication\",\"Owner\": \"Breil sathish"
},
{
"ResourceId": "/subscriptions/bb842437aa4/resourceGroups/HCLTECHLABHENKEL/providers/Microsoft.Compute/virtualMachines/pmoapps",
"Tags": "Name\": \"PMOapplication\",\"Owner\": \"Breil sathish1"
}
]
Expected Output :
[
{
"ResourceId": "/subscriptions/bb842437aa4/resourceGroups/ECHLABHENKEL/providers/Microsoft.Compute/virtualMachines/pmoapps",
"Tags": "Name\": \"PMOapplication\",\"Owner\": \"Breil sathish",
"Resourcename": "pmoapps",
"Name": "PMOapplication",
"Owner": "Breil sathish"
},
{
"ResourceId": "/subscriptions/bb842437aa4/resourceGroups/HCLTECHLABHENKEL/providers/Microsoft.Compute/virtualMachines/pmoapps",
"Tags": "Name\": \"PMOapplication\",\"Owner\": \"Breil sathish1",
"Resourcename": "pmoapps",
"Name": "PMOapplication",
"Owner": "Breil sathish1"
}
]
Thanks,
N Sathish
You can use the following transformation spec as reading the explanations stated at the beginning of each of them such as
[
{ // Convert the value of the attributes to individual arrays with the identical names
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"ResourceName": "=split('/', #(1,ResourceId))",
"tag": "=split(',', #(1,Tags))"
}
}
},
{ // Generate tag0 and tag1 attributes by splitting members of the "tag" array
"operation": "shift",
"spec": {
"*": {
"*": "&1.&",
"tag": {
"*": {
"#": "&3.&2&1"
}
}
}
}
},
{ // Split related strings by colon characters for "tag" array while deriving the last element of "ResourceName" array
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"R*": "=lastElement(#(1,&))",
"tag*": "=split(': ', #(1,&))"
}
}
},
{ // Match components of "tag" array component 1 against component 2
"operation": "shift",
"spec": {
"*": {
"*": "&1.&",
"tag*": {
"#1,&[1]": "&2.&1.#(2,&[0])"
}
}
}
},
{ // Split the values by \" character combination
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"tag*": {
"*": "=split('\"', #(1,&))"
}
}
}
},
{ // Prune undesired values for right-hand-side
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"tag*": {
"*": "=join('', #(1,&))"
}
}
}
},
{ // Prune undesired values for left-hand-side(keys)
"operation": "shift",
"spec": {
"*": {
"*": "[&1].&",
"tag*": {
"\"*\"": "[&2].&(0,1)",
"*\"": "[&2].&(0,1)"
}
}
}
}
]

JOLT concatenate two strings

I have this collection of driver details and I want to use jolt to copy the properties to concatenate first name and last name within a single property called name
{
"drivers_details": [
{
"first_name": "Doru",
"last_name": "Petre",
"cnp": "1641201390687",
"id_series": "RK",
"id_number": "123456"
},
{
"first_name": "GIGI",
"last_name": "FANE",
"cnp": "16412013906871",
"id_series": "RK",
"id_number": "1234567"
}
]
}
I am using this spec for the shift transformation spec
[
{
"operation": "shift",
"spec": {
"drivers_details": {
"*": {
"cnp": "body.drivers[&1].tin",
"id_series": "body.drivers[&1].id_series",
"id_number": "body.drivers[&1].id_number"
}
}
}
}
]
and this one for modify-overwrite, but it doesn't work how I was expecting
{
"operation": "modify-overwrite-beta",
"spec": {
"drivers_details": {
"*": {
"name": "=concat(#(2,first_name),' ',#(2,last_name))"
}
}
}
}
What is the correct way of doing this?
You can use modify transformation as in the following case, and get rid of names other than newly formed attribute name such as
[
{
"operation": "modify-overwrite-beta",
"spec": {
"drivers_details": {
"*": {
"name": "=concat(#(1,first_name),' ',#(1,last_name))"
}
}
}
},
{
"operation": "remove",
"spec": {
"drivers_details": {
"*": {
"*name": ""
}
}
}
},
{ // this last spec stands only for ordering of the attributes if the ordering and putting the "name" at the top matters.
// So, you might get rid of the "remove" transformation above if this is used. Otherwise keep it.
"operation": "shift",
"spec": {
"drivers_details": {
"*": {
"name": "&2[&1].&",
"cnp": "&2[&1].&",
"id*": "&2[&1].&"
}
}
}
}
]
where those attributes(first/last_name) stays right-hand-side at the current level, so just only need to traverse 1 colon(:) which appears as #(1,..)
You can use this spec without the remove operation:
[
{
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"*": {
"name": "=concat(#(1,first_&),' ',#(1,last_&))"
}
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"*": {
"name": "&2[&1].&",
"cnp": "&2[&1].&",
"id_*": "&2[&1].&"
}
}
}
}
]

JOLT Specification for converting filed value to field name

I am trying to convert the below JSON payload into a JSON that has the field name as the value of the field:
The jolt file that I have is working if field values are different. But if field values are the same then it is giving an array in the response.
Can you please help to provide jolt specifications for this?
Input JSON Payload:
{
"action": {
"Success": true,
"records": [
{
"Id": "Test_abc",
"SubscriptionID": "ID_1"
},
{
"Id": "Test_abc",
"SubscriptionID": "ID_2"
},
{
"Id": "Test_xyz",
"SubscriptionID": "ID_3"
}
],
"type": "update"
}
}
Expected output:
{
"action": {
"Success": true,
"records": {
"Test_abc": {
"SubscriptionID": "ID_1"
},
"Test_abc": {
"SubscriptionID": "ID_2"
},
"Test_xyz": {
"SubscriptionID": "ID_3"
}
},
"type": "update"
}
}
Solution not found yet.
You can use the following shift transformation spec
[
{
"operation": "shift",
"spec": {
"*": {
"*": "&1.&", // &1 replicates the literal "action"
"records": {
"*": {
"S*": "&3.&2.#(1,Id).&" // &3 replicates the literal "action" (by going three level up the tree), &2 for "records", & replicates the current level attribute
}
}
}
}
}
]
presumingly converting one of the Id value from Test_abc to another one such as Test_def, since the desired output is wrong as a JSON value(There cannot be more than one object with the same tag at the same level)
Your output is wrong. You can't have multiple objects with the same keys.
If You want to support all objects of records array, You should bring them into an array like this spec:
[
{
"operation": "shift",
"spec": {
"*": {
"*": "&1.&",
"records": {
"*": {
"S*ID": "&3.&2[].#(1,Id).&"
}
}
}
}
}
]
But, if you want to have an object as your records in the output, You should remove the duplicate ID like this spec:
[
{
"operation": "shift",
"spec": {
"*": {
"*": "&1.&",
"records": {
"*": {
"S*ID": "&3.&2.#(1,Id).&"
}
}
}
}
},
{
"operation": "cardinality",
"spec": {
"*": {
"records": {
"*": {
"*": "ONE"
}
}
}
}
}
]

JOLT transformation filter a list about the first string element

I would like to search a list for the first real string element (which could be not a number) and output this.
Input :
{
"firstString": [
"0.20",
"test",
"0.30"
]
}
or
{
"firstString": [
"0.20",
"0,30",
"test"
]
}
Expected Output :
{
"readingS": "test"
}
The order of the element can change, either it comes to 2nd or 3rd placeholder. The list is maximum 3 elements long.
My thought was to go over either the last or middle element, however this does not work. The list is generated just before with the modify-overwrite-beta.
You can still use modify-overwrite-beta transformation along with toInteger conversion such as
[
//Determine the elements whether toInteger conversion is applicable for them
{
"operation": "modify-overwrite-beta",
"spec": {
"arr": "=(#(1,firstString))",
"0": "=toInteger(#(1,arr[0]))",
"1": "=toInteger(#(1,arr[1]))",
"2": "=toInteger(#(1,arr[2]))"
}
},
//The arrays are generated from the elements which are eligible to
//conversion, while not for the others
{
"operation": "shift",
"spec": {
"*": "&",
"arr": { "*": "&" }
}
},
//The arrays are removed, so all numeric ones by reversing
//key-value pairs
{
"operation": "shift",
"spec": {
"*": {
"$": "#(0)"
}
}
},
//Reverse back the pairs
{
"operation": "shift",
"spec": {
"*": {
"$": "readingS[#2]"
}
}
},
//and pick the leftmost element
{
"operation": "cardinality",
"spec": {
"*": "ONE"
}
}
]
You have some values that ended with numbers. So we can match all values that ended with the number, and we have a real string in the last.
[
{
"operation": "shift",
"spec": {
"firstString": {
"*": {
"*0": "",
"*1": "",
"*2": "",
"*3": "",
"*4": "",
"*5": "",
"*6": "",
"*7": "",
"*8": "",
"*9": "",
"*": {
"$": "readingS"
}
}
}
}
}, {
"operation": "shift",
"spec": {
"*": {
"*": "&"
}
}
}
]

Jolt Transform MapRecord to JSON

I need to flatten a db using nifi. I read in a table based in a PK ID. For each row the nifi content is shown as a MapRecord. I need to pull each field value out of the MapRecord and make it a json property.
Input:
{
"maxResults": 150,
"total": 89,
"issues": "MapRecord[{issueId=1, firstName=Jack, lastName=Smith}]",
"address": "MapRecord[{addressId=1, street=Mockingbird Lane, town=Timbuktoo}]"
}
Notice in the MapRecord nothing is in quotes. I don't know why this is like this. It is obviously not JSON.
I want the result to look like:
Expected output:
{
"maxResults": 150,
"total": 89,
"firstName": "Jack",
"lastName": "Smith",
"street": "Mockingbird Lane",
"town": "Timbuktoo"
}
Does anyone know how to do this using a JOLT transform?
Thanks.
You can manage it to solve by applying too many transformation specs. In the first step need to extract the key-value pairs. In order to perform this split function within modify-overwrite-beta might be used. At the later steps, need to keep the existing pairs(maxResults&total) and properly combine with the extracted ones using some trick methods such as
[
{
"operation": "modify-overwrite-beta",
"spec": {
"i": "=split('}',#(1,issues))",
"a": "=split('}',#(1,address))"
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"i": "=split(', ',#(1,i))",
"a": "=split(', ',#(1,a))"
}
},
{
"operation": "shift",
"spec": {
"maxResults": "&",
"total": "&",
"i|a": {
"0": { "1|2": { "#": "theRest" } }
}
}
},
{
"operation": "shift",
"spec": {
"*": "arr0.&",
"theRest": { "*": "&" }
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"*": "=split('=',#(1,&))"
}
},
{
"operation": "shift",
"spec": {
"arr0": {
"*": {
"#": "arr[1]",
"$": "arr[0]"
}
},
"*": {
"*": {
"#": "arr.[&1]"
}
}
}
},
{
"operation": "shift",
"spec": {
"*": "&",
"arr": {
"0": null,
"*": {
"*": "[&1].#(2,[0].[&])"
}
}
}
},
{
"operation": "shift",
"spec": {
"0": null,
"*": ""
}
}
]