I need to transform multiple keys and their values to new JSON spec using jolt.
Input:
{
"product": "monitor",
"ID": "222",
"price": "300"
}
Expected output:
{
"record": [
{
"key": "product",
"value": "monitor"
},
{
"key": "ID",
"value": "222"
},
{
"key": "price",
"value": "300"
}
]
}
Thanks in advance
Convert each node in the input json into key/value and shift to the named object. And then named object is moved to the array.
[
{
"operation": "shift",
"spec": {
"*": {
"#": "&1.value",
"$": "&1.key"
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"#": "records"
}
}
}
]
Related
I am using JOLTTransformJson Processor in Nifi.
My input is:
[
{
"col_name": "time",
"data_type": "timestamp",
"is_nullable": true
},
{
"col_name": "otherData",
"data_type": "string",
"is_nullable": false
}
]
I am using the below spec:
[
{
"operation": "shift",
"spec": {
"*": {
"col_name": "name",
"data_type": "type[0]",
"is_nullable": {
"true": "type[1]",
"false": "type[1]"
}
}
}
},
{
"operation": "default",
"spec": {
"*": {
"type[1]": "notnull"
}
}
}
]
Expected Output is :
{
"type": "record",
"name": "table_name",
"fields": [
{
"name": "time",
"type": [
"timestamp",
"null"
]
},
{
"name": "otherData",
"type": [
"string",
"notnull"
]
}
]
}
But getting the below one as the current result by combining all values in array like:
{
"name": [
"time",
"otherData"
],
"type": [
[
"timestamp",
"int"
],
null
]
}
Can someone please help what am I missing.
You can use the following first shift transformation spec through walking by the objects of the array in order to be able to repeatedly apply the techniques :
[
{
"operation": "shift",
"spec": {
"#record": "type", // fixed values formed by using # wildcards left-hand-side
"#table_name": "name",
"*": {
"col_*": "fields[#2].&(0,1)", // replicate the 1st replacement of the asterisk from the current(0th) level
"is_nullable": { // conditional logic starts here
"#(1,data_type)": "fields[#3].type",
"true": {
"#null": "fields[#4].type"
},
"false": {
"#notnull": "fields[#4].type"
}
}
}
}
},
{ //to sort the attributes within the fields array
"operation": "sort",
"spec": {
"fields": ""
}
},
{ //to sort whole result
"operation": "shift",
"spec": {
"type": "&",
"name": "&",
"fields": "&"
}
}
]
the second and third transformation are just added to sort the attributes/arrays as desired
I need to merge array data from two different arrays from the Input JSON and make a flat list of items in the Output JSON. The first array contains the key required for the output and the second array contains the value.
So far, all my attempts at a spec haven't even come close, which is why I haven't listed one. Please see the Input and Desired Output below. Thanks for any help!!
Input JSON :
{
"data": [
{
"2": {
"value": "DC1"
},
"3": {
"value": 5
}
},
{
"2": {
"value": "DC2"
},
"3": {
"value": 10
}
}
],
"fields": [
{
"id": 2,
"label": "DataCenter",
"type": "text"
},
{
"id": 3,
"label": "CCount",
"type": "numeric"
}
],
"metadata": {
"numFields": 2,
"numRecords": 2,
"skip": 0,
"totalRecords": 2
}
}
Desired Output JSON:
[
{
"DataCenter": "DC1",
"CCount": "5"
},
{
"DataCenter": "DC2",
"CCount": "10"
}
]
You can use this spec:
[
{
"operation": "shift",
"spec": {
"data": "&",
"fields": {
"*": {
"label": "fields.#(1,id)"
}
}
}
},
{
"operation": "shift",
"spec": {
"data": {
"*": {
"*": {
"*": "[&2].#(4,fields.&1)"
}
}
}
}
}
]
In your desired output CCount value is a string, You can add the below spec to change an integer to a string.
,
{
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"CCount": "=toString"
}
}
}
You might reciprocally match id value of fields array vs. the object keys of the data array within a common object within the first shift transformation spec in order to prepare for the second one in which we tile the attributes into their individual objects such as
[
{
"operation": "shift",
"spec": {
"*": { // represents the node for both "data" and "fields" arrays
"*": {
"*": {
"value": "&1.&",
"#1,label": "#2,id.&"
}
}
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"value": {
"*": "[&].#2,label"
}
}
}
}
]
the demo on the site http://jolt-demo.appspot.com/ is
Jolt is new to me and I have been struggling with this issue till the point where I created this post.
I want to turn this:
{
"Properties": [{
"Id": "property1",
"Values": ["randomValue1", "randomValue2"]
}, {
"Id": "property2",
"Values": "randomValue3"
}, {
"Id": "property3",
"Values": "randomValue4"
}]
}
into this
{
"Properties": [{
"Id": "property1",
"Values": "randomValue1"
},{
"Id": "property1",
"Values": "randomValue2"
}, {
"Id": "property2",
"Values": "randomValue3"
}, {
"Id": "property3",
"Values": "randomValue4"
}]
}
The values for each property can be 1 value or an array of an unknown amount of values.
I changed the following json into what is seen in the first json already:
{
"Properties": {
"property1": ["randomValue1", "randomValue1"],
"property2": ["randomValue3"],
"property3": ["randomValue4"]
}
}
Spec:
[{
"operation": "shift",
"spec": {
"Properties": {
"*": {
"*": "Properties[#2].Values",
"$": "Properties[#2].Id"
}
}
}
}]
Property names on RHS are generic and the number of values for a property can differ as well.
Thank you in advanced for taking the time to assist me.
check if this helps:
[
{
"operation": "cardinality",
"spec": {
"Properties": {
"*": {
// normalize values to always be a list
"Values": "MANY"
}
}
}
},
{
"operation": "shift",
"spec": {
"Properties": {
"*": {
"Values": {
"*": {
// create arrays with values and ids
"#": "Values",
"#(2,Id)": "Id"
}
}
}
}
}
},
{
"operation": "shift",
"spec": {
"Values": {
"*": {
// create the final list joining ids and values at the indexes you want
"#": "Properties[&1].Values",
"#(2,Id[#1])": "Properties[&1].Id"
}
}
}
}
]
I have json converted from protobuf protocol with following format:
{
"id": "6aa0734f-6d6a-4b95-8a2b-2dde346f9df7",
"measurements": [
{
"ts": "1590331111510000000",
"values": [
{},
{
"name": 1,
"value": -1.8093990087509155
},
{
"name": 2,
"value": 0.35456427931785583
}
],
"parameters": [
"Stat",
"VoltageAnglePhaseB"
]
}
]
}
expected output is:
{
"id" : "6aa0734f-6d6a-4b95-8a2b-2dde346f9df7",
"ts" : "1590331111510000000",
"Stat":-1.8093990087509155,
"VoltageAnglePhaseB":0.35456427931785583
}
I've started my Jolt spec like this:
[
{
"operation": "shift",
"spec": {
"id": "id",
"measurements": {
"*": {
"ts": "ts",
"parameters": {
"*": ""
},
"*": ""
}
}
}
}
]
But facing the problem to extract name and value from the JSON.
Does anyone have an idea?
It can be done with 2 shift operations. First, shift the values to the valuesArr, same way shift the parameters to the parametersArr. Then match the values with the index.
[
{
"operation": "shift",
"spec": {
"id": "id",
"measurements": {
"*": {
"ts": "ts",
"values": {
"*": {
"value": "valuesArr"
}
},
"parameters": "parametersArr"
}
}
}
},
{
"operation": "shift",
"spec": {
"id": "id",
"ts": "ts",
"valuesArr": {
"*": "#(2,parametersArr[&])"
}
}
}
]
The requirement is to flatten the hierarchical list into single list as shown in output with key's value as KEY and value's value as VALUE in final list. (sample output added).
INPUT:
{
"optionList": [{
"key": "General",
"values": [{
"key": "A",
"value": ["a"]
},
{
"key": "B",
"value": ["b"]
},
{
"key": "C",
"value": ["c"]
}]
}]
}
Required Output :
{
"A":"a",
"B":"b",
"C":"c"
}
Below transformation should work for you:
[
{
"operation": "shift",
"spec": {
"optionList": {
"*": {
"values": {
"*": {
"value": "#(1,key)"
}
}
}
}
}
}, {
"operation": "cardinality",
"spec": {
"*": {
"#": "ONE"
}
}
}
]
See also:
Transpose data in an Array.
CardinalityTransform