I am using Apache NiFi and receive from an embedded micro over a TCP/IP socket a JSON file of the form:
{
"id": 123456,
"ip": "192.168.1.1",
"t": -12.9,
"T": -23.8,
"variables": [
"user1",
0,
-12.97,
23.87
]
}
and would like to transform it such that the keys of the variables are added as they are known to me as follows:
{
"id": 123456,
"ip": "192.168.1.1",
"t": -12.9,
"T": -23.8,
"variables": [
"username" : "user1",
"valid" : 0,
"temperature 1" : -12.97,
"temperature 2" : 23.87
]
}
and then to be able to access a key value pair such as variables.username.
I have tried using JoltTransformJSON but don't know how to write the spec correctly if it can do it!! I have written the following jolt spec:
[
{
"operation": "shift",
"spec": {
"id": "id",
"ip": "ip",
"t": "t",
"T": "T",
"variables": {
"username": "",
"valid": "",
"temperature 1": "",
"temperature 2": ""
}
}
}
]
I also have tried using UpdateRecord with Record Reader/Writer but passing the correct schema causes an error on the first variable as it can't match "user1" with a key from the initial JSON.
I have managed to work out the Jolt Spec as follows:
[
{
"operation": "shift",
"spec": {
"id": "id",
"ip": "ip",
"t": "t",
"T": "T",
"variables": {
"0": "username",
"1": "valid",
"2": "temperature 1",
"3": "temperature 2"
}
}
}
]
Thank you daggett for your contribution.
Related
I try append a new field in a JSON structure,i need it in a determined position, before the array but the SPEC locate the field in the end.
This is my JSON original and the code of transformation written in JOLT TRANSFORMATION web:
INPUT:
{
"Id": ">COS",
"equipment": "ALA",
"elementId": "M15463",
"zone": "AMBA",
"hub": "AVA",
"terminalServer": "XS0156",
"Area": "null",
"timestamp": "1576155950000",
"collectedData": [
{
"name": "llljljiouohh",
"instance": "X1.M1.YE9.ON18",
"value": "5",
"unit": "db"
}
]
}
JSON spec:
[
{
"operation": "default",
"spec": {
"timestamp_dt": "2022-10-14 15:00"
}
}
]
**
result:**
{
"Id": ">COS",
"equipment": "ALA",
"elementId": "M15463",
"zone": "AMBA",
"hub": "AVA",
"terminalServer": "XS0156",
"Area": "null",
"timestamp": "1576155950000",
"collectedData": [
{
"name": "llljljiouohh",
"instance": "X1.M1.YE9.ON18",
"value": "5",
"unit": "db"
}
],
"timestamp_dt": "2022-10-14 15:00"
}
Expected:
{
"Id": ">COS",
"equipment": "ALA",
"elementId": "M15463",
"zone": "AMBA",
"hub": "AVA",
"terminalServer": "XS0156",
"Area": "null",
"timestamp": "1576155950000",
"timestamp_dt": "2022-10-14 15:00",
"collectedData": [
{
"name": "llljljiouohh",
"instance": "X1.M1.YE9.ON18",
"value": "5",
"unit": "db"
}
]
}
Any suggestion please? Thanks!
You can individually write each key-value pairs in the desired order within a shift transformation such as
[
{
"operation": "default",
"spec": {
"timestamp_dt": "2022-10-14 15:00"
}
},
{
"operation": "shift",
"spec": {
"Id": "&",
"equipment": "&",
"elementId": "&",
"zone": "&",
"hub": "&",
"terminalServer": "&",
"Area": "&",
"timestamp": "&",
"timestamp_dt": "&",
"collectedData": "&"
}
}
]
I need to convert the nested json to flat json by given type of input, if its flat json or nested json gives output as flat json respectively. I haven't seen any correct resource related to the this topic. I given the inputs and outputs below.
I am having some trouble with converting the nested JSON to flat JSON, and didn't get any closer as to what is mentioned problem. I need to transform a JSON structure by using a JOLT spec. I use https://jolt-demo.appspot.com to test the following below. Help me, how can i get the expected out, with this code
Input-1:
{
"MessageType": "CREATION",
"Number": "123",
"Status": "created sucessfully",
"StopSequence": "1",
"Code": [
{
"CodeName": "ABC",
"ShortDescription": "short description about ABC",
"TimeImpact": 234,
"Rank": 1
},
{
"ReasonCodeName": "XYZ",
"ShortDescription": "short description about ABC",
"TimeImpact": 123,
"Rank": 2
}
]
}
Input-2:
{
"MessageType": "UPDATE",
"Number": "345",
"PNumber": "P123",
"Status": "updated sucessfully",
"StopSequence": "2",
"Id": 1234,
"LNumber": "34565",
"DeviceID": "7645235",
"Timestamp": "2015-10-01T16:00:00-05:00",
"Timezone": "US/New_York",
"TimezoneShortName": "EST",
"Unlocode": "XXXX",
}
Expected Output-1 if input 1 passes:
[
{
"MessageType": "CREATION",
"Number": "123",
"Status": "created sucessfully",
"StopSequence": "1",
"Code": "1",
"CodeName": "ABC",
"ShortDescription": "short description about ABC",
"TimeImpact": 234,
"CodeRank": 1
},
{
"MessageType": "CREATION",
"Number": "123",
"Status": "created sucessfully",
"StopSequence": "1",
"Code": "2",
"ReasonCodeName": "XYZ",
"ShortDescription": "short description about ABC",
"TimeImpact": 123,
"Rank": 2
}
]
Expected Output-2 if input 2 passes:
{
"MessageType": "UPDATE",
"PNumber": "P123",
"Status": "updated sucessfully",
"StopSequence": "2",
"Id": 1234,
"Timestamp": "2015-10-01T16:00:00-05:00",
"Timezone": "US/New_York",
}
What you need is to repeat as much as the number of objects of the Code array. So, walk through by them as picking the related values from the out of the array such as
[
{
"operation": "shift",
"spec": {
"Code": {
"*": {
"#(2,MessageType)": "[&1].MessageType", // "#(2,MessageType)" means going up the tree 2 levels to reach the position of "MessageType" , [&1] means combining the values as array of objects depending on the indexes of the array
"#(2,Number)": "[&1].Number",
"#(2,Status)": "[&1].Status",
"#(2,StopSequence)": "[&1].StopSequence",
"#(0,Rank)": "[&1].Code", // to copy the values of the "Rank" to "Code"
"*": "[&1].&"
}
},
"MessageT*|PNu*|St*|Id|Times*|Timezone": "&"
}
}
]
I want to convert JSON format to another format so I use jolt for that. please if there is best from jolt recommend me to it.
when I map this object
{
"id": 1,
"username": "sd4s5d4",
"phone": "111",
"groups": [
{
"id": 1
},
{
"id": 2
}
]
}
the expected output is ok the value of groups is returned as a list
{
"id": 1,
"username": "sd4s5d4",
"phone": "111",
"groups": [ 1, 2 ]
}
but when I map this object
{
"id": 5,
"username": "sd4s5d4",
"phone": "111",
"groups": [
{
"id": 1
}
]
}
it returns
{
"id": 5,
"username": "sd4s5d4",
"phone": "111",
"groups": 1
}
how to make groups in last output to be list even if it one item.
wanted format
{
"id": 5,
"username": "sd4s5d4",
"phone": "111",
"groups": [1]
}
Spec
[
{
"operation": "shift",
"spec": {
"id": "id",
"username": "username",
"phone": "phone",
"groups": {
"*": {
"id": "groups"
}
}
}
}
]
Just replacing "id": "groups" with "id": "groups.[&1]" is enough, btw no need to repeat every element individually, just replace them with proper substitution by using ampersand operator prepended to the integer values which represent the level that provide to the target key (e.g.number of } operators while arriving the related key). So, convert "id": "groups.[&1]" to "id": "&2.[&1]", and use "*": "&", for the other elements such as
[
{
"operation": "shift",
"spec": {
"*": "&",
"groups": {
"*": {
"id": "&2.[&1]"
}
}
}
}
]
I am trying to write a spec to shift a few key in a json object to the very end of the object.
{
"id": "12345",
"timestamp": "2019-10-28 13:24:44.547",
"action": "notify",
"name": "test"
}
to:
{
"id": "12345",
"action": "notify",
"name": "test",
"timestamp": "2019-10-28 13:24:44.547"
}
Appreciate any leads on how to go about doing this using JOLT.
I believe that is an answer, but I'm not so sure that Jolt keeps an order.
[
{
"operation": "shift",
"spec": {
"id": "id",
"action": "action",
"name": "name",
"timestamp": "timestamp"
}
}
]
Input JSON :
{
"type": "mbrInfo",
"csId": 123456789,
"insTS": "14-07-201911:55",
"seqId": 1234565,
"title": "Mr",
"fName": "Amit",
"mName": "",
"lName": "V",
"suffix": "Engg",
"lvlId": "P",
"lvlType": "LAC",
"acctStatus": "20",
"enrlDT": "2016-08-29",
"vrsnId": 1
}
Expected Output JSON:
{
"type": "mbrInfo",
"csId": 123456789,
"insTS": "14-07-201911:55",
"seqId": 1234565,
"name" : [{
"title": "Mr",
"fName": "Amit",
"mName": "",
"lName": "V",
"suffix": "Engg"}],
"lvlId": "P",
"lvlType": "LAC",
"acctStatus": "20",
"enrlDT": "2016-08-29",
"vrsnId": 1
}.
Currently I am using JOLTtransformJSON processor with JOLT Spec as :
[
{
"operation": "shift",
"spec": {
"name": {
"$": "[#1]",
"#.title": "[#1].title",
"#.fName": "[#1].fName",
"#.mName": "[#1].mName",
"#.lName": "[#1].lName",
"#.suffix": "[#1].suffix"
}
}
}
]
But all I am getting is either NULL or the original JSON (with diff spec) as output.
Thanks in advance.
Is the intent to put all the name fields into a 1-element array containing an object. This JOLT spec puts them into an object at the name field:
[
{
"operation": "shift",
"spec": {
"title": "name.title",
"fName": "name.fName",
"mName": "name.mName",
"lName": "name.lName",
"suffix": "name.suffix",
"*": "&"
}
}
]
...and this spec puts them into a 1-element array at the name field:
[
{
"operation": "shift",
"spec": {
"title": "name[0].title",
"fName": "name[0].fName",
"mName": "name[0].mName",
"lName": "name[0].lName",
"suffix": "name[0].suffix",
"*": "&"
}
}
]
I don't see any other place in the input to get an index into the array, so I just used 0.