How to transform a flat JSON to nested JSON using NiFi - json

Input JSON :
{
"type": "mbrInfo",
"csId": 123456789,
"insTS": "14-07-201911:55",
"seqId": 1234565,
"title": "Mr",
"fName": "Amit",
"mName": "",
"lName": "V",
"suffix": "Engg",
"lvlId": "P",
"lvlType": "LAC",
"acctStatus": "20",
"enrlDT": "2016-08-29",
"vrsnId": 1
}
Expected Output JSON:
{
"type": "mbrInfo",
"csId": 123456789,
"insTS": "14-07-201911:55",
"seqId": 1234565,
"name" : [{
"title": "Mr",
"fName": "Amit",
"mName": "",
"lName": "V",
"suffix": "Engg"}],
"lvlId": "P",
"lvlType": "LAC",
"acctStatus": "20",
"enrlDT": "2016-08-29",
"vrsnId": 1
}.
Currently I am using JOLTtransformJSON processor with JOLT Spec as :
[
{
"operation": "shift",
"spec": {
"name": {
"$": "[#1]",
"#.title": "[#1].title",
"#.fName": "[#1].fName",
"#.mName": "[#1].mName",
"#.lName": "[#1].lName",
"#.suffix": "[#1].suffix"
}
}
}
]
But all I am getting is either NULL or the original JSON (with diff spec) as output.
Thanks in advance.

Is the intent to put all the name fields into a 1-element array containing an object. This JOLT spec puts them into an object at the name field:
[
{
"operation": "shift",
"spec": {
"title": "name.title",
"fName": "name.fName",
"mName": "name.mName",
"lName": "name.lName",
"suffix": "name.suffix",
"*": "&"
}
}
]
...and this spec puts them into a 1-element array at the name field:
[
{
"operation": "shift",
"spec": {
"title": "name[0].title",
"fName": "name[0].fName",
"mName": "name[0].mName",
"lName": "name[0].lName",
"suffix": "name[0].suffix",
"*": "&"
}
}
]
I don't see any other place in the input to get an index into the array, so I just used 0.

Related

JOLT Transformation split objects from JSON with common values

I have an array of objects and I want to split each key from the object into a separate array in JSON.
Input JSON:
[
{
"Company": "Tesla",
"Age": 30.2
},
{
"Company": "Facebook",
"Age": 40.5
},
{
"Company": "Amazon",
"Age": 60
}
]
Expected Output
[
{
"value": "Tesla",
"variable": "Company",
"model": "device"
},
{
"value": "Facebook",
"variable": "Company",
"model": "device"
},
{
"value": "Amazon",
"variable": "Company",
"model": "device"
},
{
"value": 30.2,
"variable": "Age",
"model": "device"
},
{
"value": 40.5,
"variable": "Age",
"model": "device"
},
{
"value": 60,
"variable": "Age",
"model": "device"
}
]
Here the value of the Model is fixed. I have tried with some jolt spec. It's not working.
Any help would be much appreciated!
You can use shift transformation spec such as
[
{
// distinguish the attributes by indexes
"operation": "shift",
"spec": {
"*": {
"*": {
"#": "&1[&2].value", // # represents values of the keys inherited from one upper level, &1 represents keys "Company" or "Age", [&2] represents going tree two levels up to grab the outermost indexes in an array fashion
"$": "&1[&2].variable", // $ represents values of the keys inherited from one upper level
"#device": "&1[&2].model"
}
}
}
},
{
// get rid of object/array labels
"operation": "shift",
"spec": {
"*": {
"*": ""
}
}
}
]
the demo on the site http://jolt-demo.appspot.com/ is
Edit : If the input was
[
{
"Company": "Tesla",
"Age": 30.2,
"Time": "27/10/1992"
},
{
"Company": "Facebook",
"Age": 40.5,
"Time": "27/10/1982"
}
]
as mentioned in the comment, then you could convert the spec to
[
{
"operation": "shift",
"spec": {
"*": {
"Company|Age": {
"#": "&1[&2].value",
"$": "&1[&2].variable",
"#(1,Time)": "&1[&2].DateofBirth",
"#device": "&1[&2].model"
}
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"*": ""
}
}
}
]
in order to get the following desired output :
[
{
"value": "Tesla",
"variable": "Company",
"DateofBirth": "27/10/1992",
"model": "device"
},
{
"value": "Facebook",
"variable": "Company",
"DateofBirth": "27/10/1982",
"model": "device"
},
{
"value": 30.2,
"variable": "Age",
"DateofBirth": "27/10/1992",
"model": "device"
},
{
"value": 40.5,
"variable": "Age",
"DateofBirth": "27/10/1982",
"model": "device"
}
]

JOLT add a field in JSON in the middle of the structure

I try append a new field in a JSON structure,i need it in a determined position, before the array but the SPEC locate the field in the end.
This is my JSON original and the code of transformation written in JOLT TRANSFORMATION web:
INPUT:
{
"Id": ">COS",
"equipment": "ALA",
"elementId": "M15463",
"zone": "AMBA",
"hub": "AVA",
"terminalServer": "XS0156",
"Area": "null",
"timestamp": "1576155950000",
"collectedData": [
{
"name": "llljljiouohh",
"instance": "X1.M1.YE9.ON18",
"value": "5",
"unit": "db"
}
]
}
JSON spec:
[
{
"operation": "default",
"spec": {
"timestamp_dt": "2022-10-14 15:00"
}
}
]
**
result:**
{
"Id": ">COS",
"equipment": "ALA",
"elementId": "M15463",
"zone": "AMBA",
"hub": "AVA",
"terminalServer": "XS0156",
"Area": "null",
"timestamp": "1576155950000",
"collectedData": [
{
"name": "llljljiouohh",
"instance": "X1.M1.YE9.ON18",
"value": "5",
"unit": "db"
}
],
"timestamp_dt": "2022-10-14 15:00"
}
Expected:
{
"Id": ">COS",
"equipment": "ALA",
"elementId": "M15463",
"zone": "AMBA",
"hub": "AVA",
"terminalServer": "XS0156",
"Area": "null",
"timestamp": "1576155950000",
"timestamp_dt": "2022-10-14 15:00",
"collectedData": [
{
"name": "llljljiouohh",
"instance": "X1.M1.YE9.ON18",
"value": "5",
"unit": "db"
}
]
}
Any suggestion please? Thanks!
You can individually write each key-value pairs in the desired order within a shift transformation such as
[
{
"operation": "default",
"spec": {
"timestamp_dt": "2022-10-14 15:00"
}
},
{
"operation": "shift",
"spec": {
"Id": "&",
"equipment": "&",
"elementId": "&",
"zone": "&",
"hub": "&",
"terminalServer": "&",
"Area": "&",
"timestamp": "&",
"timestamp_dt": "&",
"collectedData": "&"
}
}
]

Rearraning of fields in nested JSON using Jolt

I have a requirement where re-arrangement of fields need to be done in a nested JSON which has array objects.
Below is the sample input:
{
"_id": "1234",
"Name": "John",
"City": "New Jersey",
"Order": [
{
"item": "pizza",
"qty": 2
},
{
"item": "coke",
"qty": 2
}
],
"State": "NJ",
"Phone": "67755321",
"Paymenttype": "Card"
}
My Jolt is able to populate all fields, but unable to figure out how to get the 'order data' in same manner in output
My code:
[
{
"operation": "shift",
"spec": {
"_id": "_id",
"Name": "Name",
"State": "State",
"Phone": "Phone"
}
}
]
Expected output:
{
"_id" : "1234",
"Name" : "John",
"State" : "NJ",
"Phone" : "67755321",
"Order": [
{
"item": "pizza",
"qty": 2
},
{
"item": "coke",
"qty": 2
}
],
"Paymenttype": "Card"
}
You can just use ampersand substitutions for the values of the attributes such as
[
{
"operation": "shift",
"spec": {
"_id": "&",
"Name": "&",
"State": "&",
"Phone": "&",
"Order": "&",
"Paymenttype": "&"
}
}
]
the demo on the site https://jolt-demo.appspot.com/ is :
even using "*" wildcards within the keys might be practical(depending on the existing key names) such as
[
{
"operation": "shift",
"spec": {
"_id": "&",
"N*": "&",
"St*": "&",
"Ph*": "&",
"Or*": "&",
"Pa*": "&"
}
}
]

jolt - copy or move a key from nested object to the top level

I'm looking for a way to copy or move a key from nested object to the top level
Input:
{
"id": "123",
"name": "foo",
"details": {
"orderNumber": "456789",
"addr": "N st 124",
"date": "2021-01-01"
}
}
desired output:
{
"id": "123",
"name": "foo",
"orderNumber": "456789",
"details": {
"orderNumber": "456789",
"addr": "N st 124",
"date": "2021-01-01"
}
}
or ideally
{
"id": "123",
"name": "foo",
"orderNumber": "456789",
"details": {
"addr": "N st 124",
"date": "2021-01-01"
}
}
the closest I could get is below transformation, but it converts object to value array
[
{
"operation": "shift",
"spec": {
"id": "id",
"name": "name",
"details": {
"orderNumber": "orderNumber",
"*": "details"
}
}
}
]
You're so close to the result, just a slight change(adding an ampersand) is needed such as
[
{
"operation": "shift",
"spec": {
"id": "id",
"name": "name",
"details": {
"orderNumber": "orderNumber",
"*": "&1.&"
}
}
}
]
in this case the keys keeps on appearing.

how to add whole json as object with jolt

this is my input json
{
"Name": "Back-test job",
"market": "2",
"TimeZone": "257",
"company": "1"
}
and I want to add the whole json as an object like this:
{
"Name": "Back-test job",
"market": "2",
"TimeZone": "257",
"company": "1",
"wholeJson":{
"Name": "Back-test job",
"market": "2",
"TimeZone": "257",
"company": "1"}
}
do you have any idea for spec?
Just call the base json value twice through use of &1, which refers one-level up as having a nested structure "*": {...} within the specification of shift operation, in one of them prepend with the key wholeJson and also apply a sort specification against getting unordered result such as
[
{
"operation": "shift",
"spec": {
"*": {
"#": "&1",
"#(1,&)": "wholeJson.&1"
}
}
},
{
"operation": "sort",
"spec": {
"*": ""
}
}
]