I am using JOLTTransformJson Processor in Nifi.
My input is:
[
{
"col_name": "time",
"data_type": "timestamp",
"is_nullable": true
},
{
"col_name": "otherData",
"data_type": "string",
"is_nullable": false
}
]
I am using the below spec:
[
{
"operation": "shift",
"spec": {
"*": {
"col_name": "name",
"data_type": "type[0]",
"is_nullable": {
"true": "type[1]",
"false": "type[1]"
}
}
}
},
{
"operation": "default",
"spec": {
"*": {
"type[1]": "notnull"
}
}
}
]
Expected Output is :
{
"type": "record",
"name": "table_name",
"fields": [
{
"name": "time",
"type": [
"timestamp",
"null"
]
},
{
"name": "otherData",
"type": [
"string",
"notnull"
]
}
]
}
But getting the below one as the current result by combining all values in array like:
{
"name": [
"time",
"otherData"
],
"type": [
[
"timestamp",
"int"
],
null
]
}
Can someone please help what am I missing.
You can use the following first shift transformation spec through walking by the objects of the array in order to be able to repeatedly apply the techniques :
[
{
"operation": "shift",
"spec": {
"#record": "type", // fixed values formed by using # wildcards left-hand-side
"#table_name": "name",
"*": {
"col_*": "fields[#2].&(0,1)", // replicate the 1st replacement of the asterisk from the current(0th) level
"is_nullable": { // conditional logic starts here
"#(1,data_type)": "fields[#3].type",
"true": {
"#null": "fields[#4].type"
},
"false": {
"#notnull": "fields[#4].type"
}
}
}
}
},
{ //to sort the attributes within the fields array
"operation": "sort",
"spec": {
"fields": ""
}
},
{ //to sort whole result
"operation": "shift",
"spec": {
"type": "&",
"name": "&",
"fields": "&"
}
}
]
the second and third transformation are just added to sort the attributes/arrays as desired
Related
I need to merge array data from two different arrays from the Input JSON and make a flat list of items in the Output JSON. The first array contains the key required for the output and the second array contains the value.
So far, all my attempts at a spec haven't even come close, which is why I haven't listed one. Please see the Input and Desired Output below. Thanks for any help!!
Input JSON :
{
"data": [
{
"2": {
"value": "DC1"
},
"3": {
"value": 5
}
},
{
"2": {
"value": "DC2"
},
"3": {
"value": 10
}
}
],
"fields": [
{
"id": 2,
"label": "DataCenter",
"type": "text"
},
{
"id": 3,
"label": "CCount",
"type": "numeric"
}
],
"metadata": {
"numFields": 2,
"numRecords": 2,
"skip": 0,
"totalRecords": 2
}
}
Desired Output JSON:
[
{
"DataCenter": "DC1",
"CCount": "5"
},
{
"DataCenter": "DC2",
"CCount": "10"
}
]
You can use this spec:
[
{
"operation": "shift",
"spec": {
"data": "&",
"fields": {
"*": {
"label": "fields.#(1,id)"
}
}
}
},
{
"operation": "shift",
"spec": {
"data": {
"*": {
"*": {
"*": "[&2].#(4,fields.&1)"
}
}
}
}
}
]
In your desired output CCount value is a string, You can add the below spec to change an integer to a string.
,
{
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"CCount": "=toString"
}
}
}
You might reciprocally match id value of fields array vs. the object keys of the data array within a common object within the first shift transformation spec in order to prepare for the second one in which we tile the attributes into their individual objects such as
[
{
"operation": "shift",
"spec": {
"*": { // represents the node for both "data" and "fields" arrays
"*": {
"*": {
"value": "&1.&",
"#1,label": "#2,id.&"
}
}
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"value": {
"*": "[&].#2,label"
}
}
}
}
]
the demo on the site http://jolt-demo.appspot.com/ is
The Jolt transform should modify the input as shown, I need to push the data aggregated like that in to the DB based on the Key value as identifier.
Input Json :
[
{
"Key": "991641500~2167767723",
"itemNm": "2067875000"
},
{
"Key": "991641500~2167767724",
"itemNm": "2067875085"
},
{
"Key": "991641500~2167767723",
"itemNm": "2067875063"
},
{
"Key": "991641500~2167767724",
"itemNm": "2067875004"
}
]
Output JSON:
The output JSON should be merged as follows, I need to have the key and the contents of those as elements of a JSon Array.
[
{
"Key": "991641500~2167767723",
"Items": [
{
"itemNm": "2067875004"
},
{
"itemNm": "2067875085"
}
]
},
{
"Key": "991641500~2167767724",
"Items": [
{
"itemNm": "2067875000"
},
{
"itemNm": "2067875063"
}
]
}
]
You can use the following transformation spec :
[
{
// group attributes under common objects by their Key(#(1,Key))
"operation": "shift",
"spec": {
"*": {
"Key": "#(1,Key).&",
"itemNm": "#(1,Key).items[&1].&"
}
}
},
{
// get rid of object labels
"operation": "shift",
"spec": {
"*": ""
}
},
{
// get rid of redundant null components of the arrays
"operation": "modify-overwrite-beta",
"spec": {
"*": "=recursivelySquashNulls"
}
},
{
// pick only single one from identical components of each "Key" array
"operation": "cardinality",
"spec": {
"*": {
"Key": "ONE"
}
}
}
]
the demo on the site http://jolt-demo.appspot.com/ is :
I need to transform multiple keys and their values to new JSON spec using jolt.
Input:
{
"product": "monitor",
"ID": "222",
"price": "300"
}
Expected output:
{
"record": [
{
"key": "product",
"value": "monitor"
},
{
"key": "ID",
"value": "222"
},
{
"key": "price",
"value": "300"
}
]
}
Thanks in advance
Convert each node in the input json into key/value and shift to the named object. And then named object is moved to the array.
[
{
"operation": "shift",
"spec": {
"*": {
"#": "&1.value",
"$": "&1.key"
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"#": "records"
}
}
}
]
I have json converted from protobuf protocol with following format:
{
"id": "6aa0734f-6d6a-4b95-8a2b-2dde346f9df7",
"measurements": [
{
"ts": "1590331111510000000",
"values": [
{},
{
"name": 1,
"value": -1.8093990087509155
},
{
"name": 2,
"value": 0.35456427931785583
}
],
"parameters": [
"Stat",
"VoltageAnglePhaseB"
]
}
]
}
expected output is:
{
"id" : "6aa0734f-6d6a-4b95-8a2b-2dde346f9df7",
"ts" : "1590331111510000000",
"Stat":-1.8093990087509155,
"VoltageAnglePhaseB":0.35456427931785583
}
I've started my Jolt spec like this:
[
{
"operation": "shift",
"spec": {
"id": "id",
"measurements": {
"*": {
"ts": "ts",
"parameters": {
"*": ""
},
"*": ""
}
}
}
}
]
But facing the problem to extract name and value from the JSON.
Does anyone have an idea?
It can be done with 2 shift operations. First, shift the values to the valuesArr, same way shift the parameters to the parametersArr. Then match the values with the index.
[
{
"operation": "shift",
"spec": {
"id": "id",
"measurements": {
"*": {
"ts": "ts",
"values": {
"*": {
"value": "valuesArr"
}
},
"parameters": "parametersArr"
}
}
}
},
{
"operation": "shift",
"spec": {
"id": "id",
"ts": "ts",
"valuesArr": {
"*": "#(2,parametersArr[&])"
}
}
}
]
The requirement is to flatten the hierarchical list into single list as shown in output with key's value as KEY and value's value as VALUE in final list. (sample output added).
INPUT:
{
"optionList": [{
"key": "General",
"values": [{
"key": "A",
"value": ["a"]
},
{
"key": "B",
"value": ["b"]
},
{
"key": "C",
"value": ["c"]
}]
}]
}
Required Output :
{
"A":"a",
"B":"b",
"C":"c"
}
Below transformation should work for you:
[
{
"operation": "shift",
"spec": {
"optionList": {
"*": {
"values": {
"*": {
"value": "#(1,key)"
}
}
}
}
}
}, {
"operation": "cardinality",
"spec": {
"*": {
"#": "ONE"
}
}
}
]
See also:
Transpose data in an Array.
CardinalityTransform