I'm so close with this but afraid I do need a hand with a jolt transformation. I've done most of the work but can't get the last minute transformation working.
Here is my data:
{
"totalElements": 168,
"columns": {
"dimension": {
"id": "variables/daterangehour",
"type": "time"
},
"columnIds": [
"1"
]
},
"rows": [
{
"itemId": "119050300",
"value": "00:00 2019-06-03",
"data": [
120
]
},
{
"itemId": "119050805",
"value": "05:00 2019-06-08",
"data": [
98
]
},
{
"itemId": "119050923",
"value": "23:00 2019-06-09",
"data": [
172
]
}
]
}
}
This is my Jolt:
[{
"operation": "shift",
"spec": {
"rows": {
"*": {
"value": "[&1].date",
"data": "[&1].data"
}
}
}
}
]
It gives me this result:
[ {
"date" : "00:00 2019-06-03",
"data" : [ 120 ]
}, {
"date" : "22:00 2019-06-09",
"data" : [ 307 ]
}, {
"date" : "23:00 2019-06-09",
"data" : [ 172 ]
} ]
This causes my system issues, I actual need the data field like this:
[ {
"date" : "00:00 2019-06-03",
"data" : "120"
}, {
"date" : "05:00 2019-06-08",
"data" : "98"
} ]
How do I pluck the item out of the array / square brackets? It will only ever be one item in there.
You should go deeper and take value. Is it clear for you?
[
{
"operation": "shift",
"spec": {
"rows": {
"*": {
"value": "[&1].date",
"data": {
"*": "[&2].data"
}
}
}
}
}
]
And if you need Strings add this:
{
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"data": "=toString"
}
}
}
Related
I am using JOLTTransformJson Processor in Nifi.
My input is:
[
{
"col_name": "time",
"data_type": "timestamp",
"is_nullable": true
},
{
"col_name": "otherData",
"data_type": "string",
"is_nullable": false
}
]
I am using the below spec:
[
{
"operation": "shift",
"spec": {
"*": {
"col_name": "name",
"data_type": "type[0]",
"is_nullable": {
"true": "type[1]",
"false": "type[1]"
}
}
}
},
{
"operation": "default",
"spec": {
"*": {
"type[1]": "notnull"
}
}
}
]
Expected Output is :
{
"type": "record",
"name": "table_name",
"fields": [
{
"name": "time",
"type": [
"timestamp",
"null"
]
},
{
"name": "otherData",
"type": [
"string",
"notnull"
]
}
]
}
But getting the below one as the current result by combining all values in array like:
{
"name": [
"time",
"otherData"
],
"type": [
[
"timestamp",
"int"
],
null
]
}
Can someone please help what am I missing.
You can use the following first shift transformation spec through walking by the objects of the array in order to be able to repeatedly apply the techniques :
[
{
"operation": "shift",
"spec": {
"#record": "type", // fixed values formed by using # wildcards left-hand-side
"#table_name": "name",
"*": {
"col_*": "fields[#2].&(0,1)", // replicate the 1st replacement of the asterisk from the current(0th) level
"is_nullable": { // conditional logic starts here
"#(1,data_type)": "fields[#3].type",
"true": {
"#null": "fields[#4].type"
},
"false": {
"#notnull": "fields[#4].type"
}
}
}
}
},
{ //to sort the attributes within the fields array
"operation": "sort",
"spec": {
"fields": ""
}
},
{ //to sort whole result
"operation": "shift",
"spec": {
"type": "&",
"name": "&",
"fields": "&"
}
}
]
the second and third transformation are just added to sort the attributes/arrays as desired
This is an input that needs to be transformed using Jolt Transformation to obtain the expected output.
I am attempting to create a jolt transformation for the below input:
{
"flights": [
{
"id": "123",
"route": "BJS-SIN"
},
{
"id": "456",
"route": "SIN-PEK"
},
{
"id": "789",
"route": "SIN-BJS"
}
],
"prices": [
{
"id": "abc",
"amount": 560
},
{
"id": "def",
"amount": 780
}
],
"solutions": [
{
"price-ref": "abc",
"flights-ref": [
"123",
"456"
]
},
{
"price-ref": "def",
"flights-ref": [
"123",
"789"
]
}
]
}
Desired output would like :
{
"solutions": [
{
"flights": [
{
"id": "123",
"route": "BJS-SIN"
},
{
"id": "456",
"route": "SIN-PEK"
}
],
"price": {
"id": "abc",
"amount": 560
}
},
{
"flights": [
{
"id": "123",
"route": "BJS-SIN"
},
{
"id": "789",
"route": "SIN-BJS"
}
],
"price": {
"id": "def",
"amount": 780
}
}
]
}
As the output, all the data should be constructed refer to the solutions node. How about the mapping expression in JOLT?
all the values from the source json are generated randomly and the keys names are fixed.
I had tried many times but could not find the right spec, so please help.
You can use this spec:
[
{
"operation": "shift",
"spec": {
"*": "&",
"flights|prices": {
"*": {
"*": "&2.#(1,id).&"
}
}
}
},
{
"operation": "shift",
"spec": {
"solutions": {
"*": {
"flights*": {
"*": {
"*": {
"#(6,flights.&)": "&5[&4].flights[&2]"
}
}
},
"price*": {
"*": {
"#(5,prices.&)": "&4[&3].price"
}
}
}
}
}
}
]
I have a JSON:
[
{
"id": 1015380,
"type": "campaign",
"stats": [
{
"message_sends_by_any_user": 13,
"ctr": "0.094",
}
]
},
{
"id": 1015695,
"type": "campaign",
"stats": [
{
"message_sends_by_any_user": 7,
"ctr": "0.091",
}
]
}
]
I want to "unnested" stats list to top level. JOLT config:
[
{
"operation": "shift",
"spec": {
"*": {
"id": "[&1].id",
"type": "[&1].type",
"stats": {
"*": "[&2].&"
}
}
}
}
]
Expected:
[
{
"id": 1015380,
"type": "campaign",
"message_sends_by_any_user": 13,
"ctr": "0.094",
},
{
"id": 1015695,
"type": "campaign",
"message_sends_by_any_user": 7,
"ctr": "0.091"
}
]
But actual output contains array indexes like this:
[
{
"id" : 1015380,
"type" : "campaign",
"0" : {
"message_sends_by_any_user" : 13,
"ctr" : "0.094"
}, ...
How to avoid these indexes?
You can prefer walking through the indexes of the stats list by taking id and type attributes inside such as
[
{
"operation": "shift",
"spec": {
"*": {
"stats": {
"*": {
"#(2,id)": "[&3].id",
"#(2,type)": "[&3].type",
"*": "[&3].&"
}
}
}
}
}
]
the demo on the site http://jolt-demo.appspot.com is
Trying to make a Jolt script that will put in a single number line, then a array of strings in one single array and a tag on the end of that array. These are the values that I have been working with.
JSON INPUT
[
{
"foo": "111",
"bar": "222",
"sun": "333",
"ListofStrings": [
"Dog",
"Train"
],
"ID": "BLAH"
},
{
"foo": "999",
"bar": "222",
"sun": "777",
"ListofStrings": [
"CAT",
"PLANE"
],
"ID": "HAHA"
}
]
JOLT SPEC This is what I have been working with that prints out the ListofStrings but this is the one that works stably.
[
{
"operation": "shift",
"spec": {
"*": {
"foo": "input[].number",
"bar": "input[].number",
"sun": "input[].number",
"ListofStrings": "input[].List"
}
}
},
{
"operation": "default",
"spec": {
"app_id": "test",
"input": {
"*": {
"app_id": "test"
}
}
}
}
]
CURRENT OUTPUT
{
"input" : [ {
"number" : "111"
}, {
"number" : "222"
}, {
"number" : "333"
}, {
"List" : [ "Dog", "Train" ]
}, {
"number" : "999"
}, {
"number" : "222"
}, {
"number" : "777"
}, {
"List" : [ "Cat", "Car" ]
} ],
"app_id" : "test"
}
DESIRED OUTPUT
{
"input" : [ {
"number" : "111"
"List" : [ "Dog", "Train" ]
"ID": "BLAH_foo"
}, {
"number" : "222"
"List" : [ "Dog", "Train" ]
"ID": "BLAH_bar"
}, {
"number" : "333"
"List" : [ "Dog", "Train" ]
"ID": "BLAH_sun"
}, {
"number" : "999"
"List" : [ "Cat", "Car" ]
"ID": "HAHA_foo"
}, {
"number" : "222"
"List" : [ "Cat", "Car" ]
"ID": "HAHA_bar"
}, {
"number" : "777"
"List" : [ "Cat", "Car" ]
"ID": "HAHA_sun"
} ],
"app_id" : "test"
}
Check this spec
[
//Converting list to Map
{
"operation": "shift",
"spec": {
"*": {
"ListofStrings": null,
"*": {
"#": "#1.number",
"#(1,ListofStrings)": "#1.list"
}
}
}
},
//Shift the number and list to the input array
{
"operation": "shift",
"spec": {
"*": {
"$": "input[#2].number",
"#(0,list)": "input[#2].List"
}
}
}, {
"operation": "default",
"spec": {
"app_id": "test",
"input": {
"*": {
"app_id": "test"
}
}
}
}
]
Edit 1
Add the ID node to the map using first shift operation "ID": null, and "#(1,ID)": "#1.ID". Then shift the ID node to the input array in the second shift operation "#(0,ID)": "input[#2].ID".
[
//Converting list to Map
{
"operation": "shift",
"spec": {
"*": {
"ListofStrings": null,
"ID": null,
"*": {
"#": "#1.number",
"#(1,ListofStrings)": "#1.list",
"#(1,ID)": "#1.ID"
}
}
}
},
//Shift the number and list to the input array
{
"operation": "shift",
"spec": {
"*": {
"$": "input[#2].number",
"#(0,list)": "input[#2].List",
"#(0,ID)": "input[#2].ID"
}
}
}, {
"operation": "default",
"spec": {
"app_id": "test",
"input": {
"*": {
"app_id": "test"
}
}
}
}
]
I have json converted from protobuf protocol with following format:
{
"id": "6aa0734f-6d6a-4b95-8a2b-2dde346f9df7",
"measurements": [
{
"ts": "1590331111510000000",
"values": [
{},
{
"name": 1,
"value": -1.8093990087509155
},
{
"name": 2,
"value": 0.35456427931785583
}
],
"parameters": [
"Stat",
"VoltageAnglePhaseB"
]
}
]
}
expected output is:
{
"id" : "6aa0734f-6d6a-4b95-8a2b-2dde346f9df7",
"ts" : "1590331111510000000",
"Stat":-1.8093990087509155,
"VoltageAnglePhaseB":0.35456427931785583
}
I've started my Jolt spec like this:
[
{
"operation": "shift",
"spec": {
"id": "id",
"measurements": {
"*": {
"ts": "ts",
"parameters": {
"*": ""
},
"*": ""
}
}
}
}
]
But facing the problem to extract name and value from the JSON.
Does anyone have an idea?
It can be done with 2 shift operations. First, shift the values to the valuesArr, same way shift the parameters to the parametersArr. Then match the values with the index.
[
{
"operation": "shift",
"spec": {
"id": "id",
"measurements": {
"*": {
"ts": "ts",
"values": {
"*": {
"value": "valuesArr"
}
},
"parameters": "parametersArr"
}
}
}
},
{
"operation": "shift",
"spec": {
"id": "id",
"ts": "ts",
"valuesArr": {
"*": "#(2,parametersArr[&])"
}
}
}
]