I want to transform a JSON into key-value pairs and simultaneously copy a specific single value, i.e. the "timestamp", to all these pairs.
Input JSON:
{
"Timestamp": "2018-05-13T14:57:09",
"first_key": "1023",
"another_key": "1987",
"yet_another_key": "677"
}
Expected output:
[ {
"Timestamp": "2018-05-13T14:57:09.087",
"key": "first_key",
"value": "1023"
},
{
"Timestamp": "2018-05-13T14:57:09.087",
"key": "another_key",
"value": "1987"
},
{
"Timestamp": "2018-05-13T14:57:09.087",
"key": "yet_another_key",
"value": "677"
}]
What I came up with so far is the following JOLT specification. It already generates the key-value pairs for all entries that are not "Timestamp", but how can I copy the value of "Timestamp" into each of these records?
[{
"operation": "shift",
"spec": {
"Timestamp": "[].Timestamp",
"*": {
"$": "[#2].key",
"#": "[#2].value"
}
}
}]
Output of the above JOLT specification:
[ {
"Timestamp" : "2018-05-13T14:57:09"
}, {
"key" : "first_key",
"value" : "1023"
}, {
"key" : "another_key",
"value" : "1987"
}, {
"key" : "yet_another_key",
"value" : "677"
} ]
Spec
[
{
// first separate the Timestamp from the fields that
// are going to be pivoted.
// This is needed because the "[#2]" logic
// isn't doing what you think it is /
// you would end up with a null in your output array.
// Basically the "[#2]" logic as written only works
// when you are cleanly pivoting data. Which you are
// not because "Timestamp" and "first_key" are siblings.
"operation": "shift",
"spec": {
"Timestamp": "Timestamp",
"*": "keysToPivot.&"
}
},
{
"operation": "shift",
"spec": {
"keysToPivot": {
"*": {
// Now that we can cleanly loop thru _only_ the
// keysToPivot, the [#2] logic will work "right".
// Additionally, lookup the Timestamp value
// and add it to each "pivoted" output.
"$": "[#2].key",
"#": "[#2].value",
"#(2,Timestamp)": "[#2].Timestamp"
}
}
}
}
]
Related
I am using JOLTTransformJson Processor in Nifi.
My input is:
[
{
"col_name": "time",
"data_type": "timestamp",
"is_nullable": true
},
{
"col_name": "otherData",
"data_type": "string",
"is_nullable": false
}
]
I am using the below spec:
[
{
"operation": "shift",
"spec": {
"*": {
"col_name": "name",
"data_type": "type[0]",
"is_nullable": {
"true": "type[1]",
"false": "type[1]"
}
}
}
},
{
"operation": "default",
"spec": {
"*": {
"type[1]": "notnull"
}
}
}
]
Expected Output is :
{
"type": "record",
"name": "table_name",
"fields": [
{
"name": "time",
"type": [
"timestamp",
"null"
]
},
{
"name": "otherData",
"type": [
"string",
"notnull"
]
}
]
}
But getting the below one as the current result by combining all values in array like:
{
"name": [
"time",
"otherData"
],
"type": [
[
"timestamp",
"int"
],
null
]
}
Can someone please help what am I missing.
You can use the following first shift transformation spec through walking by the objects of the array in order to be able to repeatedly apply the techniques :
[
{
"operation": "shift",
"spec": {
"#record": "type", // fixed values formed by using # wildcards left-hand-side
"#table_name": "name",
"*": {
"col_*": "fields[#2].&(0,1)", // replicate the 1st replacement of the asterisk from the current(0th) level
"is_nullable": { // conditional logic starts here
"#(1,data_type)": "fields[#3].type",
"true": {
"#null": "fields[#4].type"
},
"false": {
"#notnull": "fields[#4].type"
}
}
}
}
},
{ //to sort the attributes within the fields array
"operation": "sort",
"spec": {
"fields": ""
}
},
{ //to sort whole result
"operation": "shift",
"spec": {
"type": "&",
"name": "&",
"fields": "&"
}
}
]
the second and third transformation are just added to sort the attributes/arrays as desired
I am trying remove a field from array of JSON using JOLT Transformation. below is the input JSON and expected output JSON.
can you suggest the JOLT transformation for the mentioned scenario
Input:
{
"code": 200,
"data": {
"totalCount": 1,
"SCount": 1,
"FCount": 0,
"FSequences": [],
"token": [
328358
]
},
"id": "ce27g26hvamob9lbd0eg"
}
Jolt I'm using:
[
{
"operation": "shift",
"spec": {
"data": {
"*": "&"
},
"status|id": "&"
}
}
]
Expected output:
{
"totalCount" : 1,
"SCount" : 1,
"FCount" : 0,
"FSequences" : "",
"token" : "328358",
"status" : 200,
"id" : "ce27g26hvamob9lbd0eg"
}
You can consecutively apply shift and modify specs such that
[
{
"operation": "shift",
"spec": {
"data": {
"*": "&",
"token": {
"*": "&1" // replicates the tag "token" by grabbing after going up tree one level
}
},
"code": "status", // renaming the attribute
"id": "&"
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"*u*n*": ["=toInteger", ""], // conversion only doesn't apply for "FSequences" among *u*n* tagged attributes
"*o*n": "=toString" // only applied for "token" attribute
}
}
]
I have this JSON:
{
"mapTrue": "true",
"code": [
"X",
"Y",
"Z"
],
"expired": [
"true",
"false",
"true"
]
}
I want to use the "mapTrue" value in order to filter the "code" array.
If "mapTrue": "true", I'll take the 0 and 2 values in "expired" array, therefore I need to output
"code": ["X", "Z"].
Using the same logic, for "mapTrue: "false" I'll return "code": ["Y"].
This specification returns the right "code" array but it doesn't use the "mapTrue" value.
[
{
"operation": "shift",
"spec": {
"expired": {
"*": {
"true": {
"#(3,code[&1])": "code"
}
}
}
}
}
]
That's where my problem is.
This works:
[
{
"operation": "shift",
"spec": {
"expired": "#(1,mapTrue)",
"code": "code"
}
},
{
"operation": "shift",
"spec": {
"*": {
"*": {
"&1": {
"#(3,code[&1])": "code[]"
}
}
}
}
}
]
An alternative option would be using the following specs
[
{
// separate the values for "true", "false" while match "code" with the value of "mapTrue"
"operation": "shift",
"spec": {
"c*": {
"*": "#(2,expired[&])", // walk through the indexes(0,1,2) of "expired" array while matching with the components of "code"
"#(1,mapTrue)": "&1"
}
}
},
{
"operation": "shift",
"spec": {
"c*": {
"*": {
"#(2,&)": "&2" // grab the value going tree up two levels by using "#(2,&)", and copy the main object's label "code" by grabbing it after going tree two levels again
}
}
}
}
]
the demo on the site http://jolt-demo.appspot.com/ is
The Jolt transform should modify the input as shown, I need to push the data aggregated like that in to the DB based on the Key value as identifier.
Input Json :
[
{
"Key": "991641500~2167767723",
"itemNm": "2067875000"
},
{
"Key": "991641500~2167767724",
"itemNm": "2067875085"
},
{
"Key": "991641500~2167767723",
"itemNm": "2067875063"
},
{
"Key": "991641500~2167767724",
"itemNm": "2067875004"
}
]
Output JSON:
The output JSON should be merged as follows, I need to have the key and the contents of those as elements of a JSon Array.
[
{
"Key": "991641500~2167767723",
"Items": [
{
"itemNm": "2067875004"
},
{
"itemNm": "2067875085"
}
]
},
{
"Key": "991641500~2167767724",
"Items": [
{
"itemNm": "2067875000"
},
{
"itemNm": "2067875063"
}
]
}
]
You can use the following transformation spec :
[
{
// group attributes under common objects by their Key(#(1,Key))
"operation": "shift",
"spec": {
"*": {
"Key": "#(1,Key).&",
"itemNm": "#(1,Key).items[&1].&"
}
}
},
{
// get rid of object labels
"operation": "shift",
"spec": {
"*": ""
}
},
{
// get rid of redundant null components of the arrays
"operation": "modify-overwrite-beta",
"spec": {
"*": "=recursivelySquashNulls"
}
},
{
// pick only single one from identical components of each "Key" array
"operation": "cardinality",
"spec": {
"*": {
"Key": "ONE"
}
}
}
]
the demo on the site http://jolt-demo.appspot.com/ is :
Say I have the following Json:
{
"collection": [
{
"sequence": 1,
"value": "event1"
},
{
"sequence": 2,
"value": "event2"
},
{
"sequence": 3,
"value": "event3"
}
]
}
Given that the array is not guaranteed to be sorted by "sequence", how do I extract the element that has the highest value of "sequence"?
the sub-objects of the "collection" array might be sorted by the sequence values by putting them as indexes for reformed array (call "objectWithHighestSequence") such as
[
{
"operation": "shift",
"spec": {
"collection": {
"*": "objectWithHighestSequence[#(0,sequence)]"
}
}
},
{
// only pick the last object after sorting within the shift transformation
"operation": "modify-overwrite-beta",
"spec": {
"*": "=lastElement(#(1,objectWithHighestSequence))"
}
}
]
the demo on the site http://jolt-demo.appspot.com/ is