json - jolt : How to extract common value to key - json

I am trying to convert a JSON to different format using JOLT (using NiFi JoltTransformJson processor).
Input Json
[
{
"date": "202001010000",
"name": "test1",
"val": "1",
"status": "0"
},
{
"date": "202001010000",
"name": "test2",
"val": "2",
"status": "0"
},
{
"date": "202001010001",
"name": "test1",
"val": "3",
"status": "0"
},
{
"date": "202001010001",
"name": "test2",
"val": "4",
"status": "0"
}
]
and I want to Output like
{
"202001010000" : [ {
"name" : "test1",
"val" : "1",
"status" : "0"
}, {
"name" : "test2",
"val" : "2",
"status" : "0"
}
],
"202001010001" : [ {
"name" : "test1",
"val" : "3",
"status" : "0"
}, {
"name" : "test2",
"val" : "4",
"status" : "0"
}
]
}
I'm trying to convert JSON format using Jolt Transform but it can't.


Make the date node value as key for each object in the array.
Remove the date node from the object.
Spec :
[
{
"operation": "shift",
"spec": {
"*": {
"date": {
"#1": "#(2,date)"
}
}
}
},
{
"operation": "remove",
"spec": {
"*": {
"*": {
"date": ""
}
}
}
}
]

Related

How to remove empty json object from nested json structure

In my current JSON, I am getting an empty JSON object {} inside dummy_var5. The empty object is inside an array which is inside an object itself.
{
"dummy_var1": "abc",
"dummy_var2": [
{
"item": {
"action": "test",
"po": {
"id": "abc"
},
"ot": "test1",
"id": "1"
}
}
],
"dummy_var3": {
"dummy_var4": [
{
"name": "test",
"value": "test1"
},
{
"name": "test",
"value": "test1"
}
],
"name": "test2"
},
"dummy_var5": [
{
"ref": "test",
"name": "test1",
"type": null
},
{}
],
"dummy_var6": [
{
"role": "test",
"ref": "test1",
"partyDescription": "test2"
}
]
}
Considering this structure does not change, Is there any way to remove this via jolt. We tried using third party tool 'atlasmap' but couldn't achieve the desired result.
Expected output :-
{
"dummy_var1": "abc",
"dummy_var2": [
{
"item": {
"action": "test",
"po": {
"id": "abc"
},
"ot": "test1",
"id": "1"
}
}
],
"dummy_var3": {
"dummy_var4": [
{
"name": "test",
"value": "test1"
},
{
"name": "test",
"value": "test1"
}
],
"name": "test2"
},
"dummy_var5": [
{
"ref": "test",
"name": "test1",
"type": null
}
],
"dummy_var6": [
{
"role": "test",
"ref": "test1",
"partyDescription": "test2"
}
]
}

You can use this single shift transformation spec
[
{
"operation": "shift",
"spec": {
"*": "&", // the attributes other than "dummy_var5"
"dummy_var5": {
"*": {
"*": "&2.[&1].&"
}
}
}
}
]
the match "*":"&" of the line "*": "&2.[&1].&" returns the null value from the leaf node for this level, and so removes the innermost null object {}

How to change a name of more fields in Json array using Jolt

I am trying to convert the input JSON:
for example , incoming json data structure is
{
"12346565": [
{
"type": "13",
"value": "3",
"score": "5"
},
{
"type": "45",
"value": "1",
"score": "5"
}
],
"12346777": [
{
"type": "41",
"value": "10",
"score": "3"
}
]
}
The expected output is:
{
"12346565": [
{
"model": "13",
"v": "3"
},
{
"model": "45",
"v": "1"
}
],
"12346777": [
{
"model": "41",
"v": "10"
}
]
}
change:
type -> model;
value -> v
remove:
score
Any ideas?

You can use modify and remove transformation specs consecutively such as
[
{
// generate new pairs with values copied from the former attributes
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"*": {
"model": "#(1,type)",
"v": "#(1,value)"
}
}
}
},
{
// get rid of undesired attributes
"operation": "remove",
"spec": {
"*": {
"*": {
"type": "",
"value": "",
"score": ""
}
}
}
}
]
the demo on the site http://jolt-demo.appspot.com/ is

Selectively get certain arrays from inside an array in JSON using JOLT

I've been trying to get some fields out of a very long and complicated json format but not getting the output I want.
MY current spec obtains all events from the event array and lists them in the output. I'm unsure how to select specific events and only output those. Am not quite sure of the syntax
My JSON:
{
"rootid": "19718",
"clloadm": "2021-06-01T22:40:02",
"clload": "2021-06-01T21:21:39",
"date": "2021-05-25T21:52:30",
"events": [
{
"done": {
"id": "e0",
"value": "2021-05-29T08:08:19"
},
"id": "e0_event",
"started": {
"id": "e0",
"value": "2021-05-29T08:08:19"
},
"status": "complete"
},
{
"done": {
"id": "e1",
"value": "2021-05-27T02:20:25"
},
"id": "e1_event",
"started": {
"id": "e1",
"value": "2021-05-27T02:20:25"
},
"status": "complete"
},
{
"done": {
"id": "e2",
"value": "2021-05-29T08:08:19"
},
"id": "e2_event",
"started": {
"id": "e2",
"value": "2021-05-29T08:08:19"
},
"status": "complete"
},
{
"done": {
"id": "e3",
"value": "2021-05-29T08:08:19"
},
"id": "e3_event",
"started": {
"id": "e3",
"value": "2021-05-29T08:08:19"
},
"status": "complete"
},
{
"done": {
"id": "e4",
"value": "2021-05-29T08:08:19"
},
"id": "e4_event",
"started": {
"id": "e4",
"value": "2021-05-29T08:08:19"
},
"status": "complete"
}
],
"ids": [
{
"id": "id",
"source": "source",
"value": "value"
},
{
"id": "new_id",
"source": "new_source",
"value": "value"
}
]
}
My Jolt Spec that gets all events for now:
[
{
"operation": "shift",
"spec": {
"rootid": "rootid",
"clloadm": "clloadm",
"clload": "clload",
"date": "date",
"events": {
"*": {
"*": {
"#value": "#id"
}
}
},
"ids": {
"*": {
"#value": "#id"
}
}
}
}
]
The output I get:
{
"rootid" : "19718",
"clloadm" : "2021-06-01T22:40:02",
"clload" : "2021-06-01T21:21:39",
"date" : "2021-05-25T21:52:30",
"e0" : [ "2021-05-29T08:08:19", "2021-05-29T08:08:19" ],
"e1" : [ "2021-05-27T02:20:25", "2021-05-27T02:20:25" ],
"e2" : [ "2021-05-29T08:08:19", "2021-05-29T08:08:19" ],
"e3" : [ "2021-05-29T08:08:19", "2021-05-29T08:08:19" ],
"e4" : [ "2021-05-29T08:08:19", "2021-05-29T08:08:19" ],
"id" : "value",
"new_id" : "value"
}
The output I would like
{
"rootid" : "19718",
"clloadm" : "2021-06-01T22:40:02",
"clload" : "2021-06-01T21:21:39",
"date" : "2021-05-25T21:52:30",
"e0" : [ "2021-05-29T08:08:19", "2021-05-29T08:08:19" ],
"e4" : [ "2021-05-29T08:08:19", "2021-05-29T08:08:19" ],
"id" : "value",
"new_id" : "value"
}

You can write the individual keys e0 and e4 as conditional cases for #id key while rewriting the rest of the key-value pairs through "*":"&" representation such as
[
{
"operation": "shift",
"spec": {
"*": "&",
"events": {
"*": {
"*": {
"#id": {
"e0": { "#(2,value)": "&" },
"e4": { "#(2,value)": "&" }
}
}
}
},
"ids": {
"*": {
"#value": "#id"
}
}
}
}
]

Jolt Transform - Unable to transform json array into elements are objects with the specified key/value pairs

I'm trying to transform JSON array into elements are objects with the specified key/value pairs
{
"Resource": "sorPersonRole",
"Roleid": "1",
"Timestamp": "2010-06-30 00:00:00.0",
"Release": "Public",
"DOB": "2064-09-05",
"Active": "Y",
"enterprise_id": "a12s33",
"Inactive_enterprise_id": "",
"emp_ID": "123456",
"Inactive_emp_id": "000821972",
"Username": "",
"A_ID": "fsgf1234jhgfs3",
"P_ID": "w123456",
"Is Email Valid": "Y",
"Flag": "N",
"Registered": "spring",
"Description": "mainland corp",
"End Date": null
}
Expected output:
{
"meta" : {
"Resource" : "sorPersonRole",
"Roleid" : "1",
"Timestamp" : "2010-06-30 00:00:00.0",
"Release" : "Public",
"Sorid" : "w123456"
},
"sorAttributes" : {
"DOB" : "2064-09-05",
"Active" : "Y",
"End Date" : null,
"identifiers":
[
{
"type" : "enterprise"
"enterprise_id" : "a12s33",
"Username" : ""
},
{
"type" : "former-enterprise"
"Inactive_enterprise_id" : ""
},
{
"type" : "UID"
"emp_ID" : "123456",
"Inactive_emp_id" : "000821972"
},
{
"type" : "National"
"A_ID" : "fsgf1234jhgfs3"
}
],
"mainLand:com:adhoc" : {
"Is Email Valid" : "Y",
"Flag" : "N",
"Registered" : "spring",
"Description" : "mainland corp"
}
}
}
current Jolt spec: which I am not getting desired output
[
{
"operation": "shift",
"spec": {
"Resource": "meta.&",
"P_ID": "meta.Sorid",
"Roleid": "meta.&",
"Timestamp": "meta.&",
"Release": "meta.&",
"enterprise_id": "sorAttributes.Identifiers.type.enterprise.&",
"Inactive_enterprise_id": "sorAttributes.Identifiers.type.former-enterprise.&",
"emp_ID": "sorAttributes.Identifiers.type.UID.&",
"Inactive_emp_id": "sorAttributes.Identifiers.type.UID.&",
"Username": "sorAttributes.Identifiers.type.enterprise.&",
"A_ID": "sorAttributes.Identifiers.type.National.&",
"Is Email Valid": "sorAttributes.mainLand:com:adhoc.&",
"Flag": "sorAttributes.mainLand:com:adhoc.&",
"Registered": "sorAttributes.mainLand:com:adhoc.&",
"Description": "sorAttributes.mainLand:com:adhoc.&",
"*": "sorAttributes.&"
}
}
]
I have tried the different JsonSpecs provided on different websites, could able to match expected output. Also tried using two-shift operations but no luck, Any help or suggestion will be appreciated.
Thanks.

This can help,
For the nodes to be shifted into the identifier array, shift one level more.
[
{
"operation": "shift",
"spec": {
"Resource": "meta.&",
"Roleid": "meta.&",
"Timestamp": "meta.&",
"Release": "meta.&",
"P_ID": "meta.Sorid",
"DOB": "sorAttributes.&",
"Active": "sorAttributes.&",
"End Date": "sorAttributes.&",
"Is Email Valid": "sorAttributes.mainLand:com:adhoc.&",
"Flag": "sorAttributes.mainLand:com:adhoc.&",
"Registered": "sorAttributes.mainLand:com:adhoc.&",
"Description": "sorAttributes.mainLand:com:adhoc.&",
"enterprise_id": {
"#enterprise": "sorAttributes.identifiers[#2].type",
"#": "sorAttributes.identifiers[#2].&",
"#(1,Username)": "sorAttributes.identifiers[#2].Username"
},
"Inactive_enterprise_id": {
"#former-enterprise": "sorAttributes.identifiers[#2].type",
"#": "sorAttributes.identifiers[#2].&"
},
"Inactive_emp_id": {
"#UID": "sorAttributes.identifiers[#2].type",
"#": "sorAttributes.identifiers[#2].&",
"#(1,emp_ID)": "sorAttributes.identifiers[#2].emp_ID"
},
"A_ID": {
"#National": "sorAttributes.identifiers[#2].type",
"#": "sorAttributes.identifiers[#2].&"
}
}
}, {
"operation": "modify-overwrite-beta",
"spec": {
"*": "=recursivelySquashNulls"
}
}
]

Get outer json key value into very json in the nested json array using JOLT

I have a JSON like this
{
"id": "1234",
"name": "something",
"list": [
{
"A": "Something"
},
{
"B": "Something1"
}
]
}
What I would like is to do is add id and name to both the JSON's present inside list
I've gone through a couple of questions and I couldn't find anywhere where somebody had done this.

I believe the following Shift spec will work:
{
"id|name": "&",
"list": {
"*": {
"#(2,id)": "&2.[&1].id",
"#(2,name)": "&2.[&1].name",
"*": "&2.[&1].&"
}
}
}
With your sample data, the output produced was:
{
"id": "1234",
"name": "something",
"list": [{
"id": "1234",
"name": "something",
"A": "Something"
}, {
"id": "1234",
"name": "something",
"B": "Something1"
}]
}

This spec should give you what you want:
[
{
"operation": "shift",
"spec": {
"list": {
"*": {
"#(2,id)": "&2.[&1].id",
"#(2,name)": "&2.[&1].name",
"*": "&2.[&1].&"
}
}
}
}
]
With your input it gives the following as output:
{
"list" : [ {
"A" : "Something",
"id" : "1234",
"name" : "something"
}, {
"B" : "Something1",
"id" : "1234",
"name" : "something"
} ]
}