Jolt transform expand dot as a JSON Object - json

Is there a way in Jolt to convert a dot operator in JSON value to a nested JSON object while transforming the input ?
For example in the below json I would like target_property of id=2 to be converted to a json object as shown in the expected O/P. Any help is appreciated
Input:
{
"name": "Test",
"email": "Test",
"form_id": "123",
"field_list": [
{
"id": 1,
"value": "Test Subject",
"is_custom_field": false,
"target_property": "subject"
},
{
"id": 2,
"value": "Test Description",
"is_custom_field": false,
"target_property": "comment.body"
}
]
}
Jolt Transform tried
[
{
"operation": "shift",
"spec": {
"form_id": "ticket.ticket_form_id",
"name": "ticket.requester.name",
"email": "ticket.requester.email",
"field_list": {
"*": {
"is_custom_field": {
"true": {
"#(2,target_property)": "ticket.custom_fields[&3].id",
"#(2,value)": "ticket.custom_fields[&3].value"
},
"*": {
"#(2,value)": "ticket.#(3,target_property)"
}
}
}
}
}
}
]
Current Output
{
"ticket" : {
"ticket_form_id" : "123",
"requester" : {
"name" : "Test",
"email" : "Test"
},
"subject" : "Test Subject",
"comment.body" : "Test Description"
}
}
Expected Output
{
"ticket": {
"ticket_form_id": "123",
"requester": {
"name": "Test",
"email": "Test"
},
"subject": "Test Subject",
"comment": {
"body": "Test Description"
}
}
}

You can add another shift transformation spec such that
{
"operation": "shift",
"spec": {
"*": {
"*": "&1.&",
"*.*": "&1.&(0,1).&(0,2)"
}
}
}
where
"*.*" represents the attributes having a dot within them. the zeroes
within &(0,..) expressions stand for the current level
1,2 as second arguments of them represent the first and the second pieces of the splitted keys (in this case comment.body)
respectively.

Related

conversion array into array by jolt

I have following json data . I want to transform following data by jolt nifi processor into result data
{
"data": [
{
"source": "Environment Sensors",
"alert_count": "2",
"category": "envs",
"alert_array": {
"alerts": [
{
"name": "neeraj",
"id": "123"
},
{
"name": "arun",
"id": "897"
}
]
}
}
]
}
result data
{
"alert": "2",
"subcategory": "envs",
"alert_array": [
{
"category": "Environment Sensors",
"newName": "neeraj",
"newID": "123"
},
{
"category": "Environment Sensors",
"newName": "arun",
"newID": "897"
}
]
}
Here value of category is the value of source
You can use the following shift transformation spec
[
{
"operation": "shift",
"spec": {
"data": {
"*": {
"alert_count": "alert",
"category": "sub&",
"alert_array": {
"*": {
"*": {
"#(3,category)": "&3[&1].category",
"name": "&3[&1].newName",
"id": "&3[&1].newID"
}
}
}
}
}
}
}
]
first reach the elements of the innermost object, and determine the value of the category element by going 3 levels up along with the others which are alreay located there.

JOLT spec to populate data based on tag presence

I have requirement where input will have one or two arrays based on scenarios:
Input 1:
{
"name": "xyz",
"age": "30",
"addressA": [
{
"id": "111111",
"details": ""
},
{
"id": "111112",
"details": ""
}
],
"addressB": [
{
"id": "222222",
"details": ""
},
{
"id": "222223",
"details": ""
}
]
}
Input 2:
{
"name": "xyz",
"age": "30",
"addressB": [
{
"id": "222222",
"details": ""
},
{
"id": "222223",
"details": ""
}
]
}
So when 'addressA' is present in input json it should use that to give output if its no present then it should use 'addressB' to populate the same
Output expected:
For input 1:
[
{
"id": "111111",
"details": ""
},
{
"id": "111112",
"details": ""
}
]
For Input 2:
[
{
"id": "222222",
"details": ""
},
{
"id": "222223",
"details": ""
}
]
Any help would be appreciated.
I see option in JOLT to have if-else based on string value, but could not find anything to check just if exists
Thanks
Mahendra
You can determine the arrays A and B within the first shift transformation spec, and then use conditional within the second one such as
[
{
"operation": "shift",
"spec": {
"address*": {
"*": "&(1,1)" // in order to determine the labels of the arrays as "A" and "B"
}
}
},
{
"operation": "shift",
"spec": {
"A": {
"#": ""
},
"*": "&"
}
},
{
"operation": "shift",
"spec": {
"*": ""
}
}
]

JOLT transformation remove a specific field

Edit1: My goal is to be able to remove a specific attribute that has a specific value. In my example, the rating attribute should be removed as soon as it has a "?" as content.
"rating": "?" => should be deleted
"rating": "1" => should NOT be deleted
Hi all, is there any way that I don't remove all attributes of a certain type, but only those that have a certain content? Example:
"test1": "123",
"test1":"?"
Here I would like to remove only test1:"?".
Input:
{
"TEST": {
"revision": "2021",
"results": [
{
"parameter": "Abweichung X L/R",
"example": "123",
"rating": "1"
},
{
"parameter": "10001",
"example": "BIN7934659050003696",
"rating": "?"
},
{
"parameter": "1015 Bauteil niO",
"rating": "0"
}
]
}
}
Desired Output:
{
"TEST": {
"revision": "2021",
"results": [
{
"parameter": "Test",
"example": "123456",
"rating": "1"
},
{
"parameter": "10001",
"example": "123"
},
{
"parameter": "Test123",
"rating": "0"
}
]
}
}
All "ratings" except the one with the content "?" remain.
Thanks in advance and many greetings
You can use below shift transformation spec in order to remove the attribute rating with value ?
[
{
"operation": "shift",
"spec": {
"TEST": {
"*": "&1.&",
"results": {
"*": {
"rating": {
"?": {
"#(2,parameter)": "&5.&4.[&3].parameter",
"#(2,example)": "&5.&4.[&3].example"
},
"*": {
"#2": "&5.&4.[&3]"
}
}
}
}
}
}
}
]
where
"*" represents the rest of the keys(revision in this case) other than specified(results)
the &1 at the top and &5s represent the outermost key(TEST)
the &4s represent the array's(results) name
the [&3] s are needed to combine the independent sub-elements due to the indexes of the array(results) as a common center

Unable to form the JOLT schema to transform JSON in NiFi

I am trying to use the jolt JSON to JSON transformation in Apache NiFi. I want to transform one JSON into another format.
Here is my original JSON:
{
"total_rows": 5884,
"offset": 0,
"rows": [
{
"id": "03888c0ab40c32451a018be6b409eba3",
"key": "03888c0ab40c32451a018be6b409eba3",
"value": {
"rev": "1-d5cc089dd8682422962ccab4f24bd21b"
},
"doc": {
"_id": "03888c0ab40c32451a018be6b409eba3",
"_rev": "1-d5cc089dd8682422962ccab4f24bd21b",
"topic": "iot-2/type/home-iot/id/1234/evt/temp/fmt/json",
"payload": {
"temperature": 36
},
"deviceId": "1234",
"deviceType": "home-iot",
"eventType": "temp",
"format": "json"
}
},
{
"id": "03888c0ab40c32451a018be6b409f163",
"key": "03888c0ab40c32451a018be6b409f163",
"value": {
"rev": "1-dee82cbb1b5ffa8a5e974135eb6340c5"
},
"doc": {
"_id": "03888c0ab40c32451a018be6b409f163",
"_rev": "1-dee82cbb1b5ffa8a5e974135eb6340c5",
"topic": "iot-2/type/home-iot/id/1234/evt/temp/fmt/json",
"payload": {
"temperature": 22
},
"deviceId": "1234",
"deviceType": "home-iot",
"eventType": "temp",
"format": "json"
}
}
]
}
I want this to be transformed in the following JSON:
[
{
"temperature":36,
"deviceId":"1234",
"deviceType":"home-iot",
"eventType":"temp"
},
{
"temperature":22,
"deviceId":"1234",
"deviceType":"home-iot",
"eventType":"temp"
}
]
This is what my spec looks like:
[
{
"operation": "shift",
"spec": {
"rows": {
"*": {
"doc": {
"deviceId": "[&1].deviceId",
"deviceType": "[&1].deviceType",
"eventType": "[&1].eventType",
"payload": {
"*": "[&1]"
}
}
}
}
}
}
]
I keep getting a null response. I am new to this and the documentation is not very easy to comprehend. Can somebody please help?
Because you are "down" one more level after the array index, by the time you get to deviceId you are 2 levels away from the index. Replace all the &1s with &2 except for payload. In that case you are another level "down" so you'll want to use &3 for the index. You also need to take whatever is matched by the * (temperature, e.g.) and set the outgoing field name to the same thing, by using & after the array index. Here's the resulting spec:
[
{
"operation": "shift",
"spec": {
"rows": {
"*": {
"doc": {
"deviceId": "[&2].deviceId",
"deviceType": "[&2].deviceType",
"eventType": "[&2].eventType",
"payload": {
"*": "[&3].&"
}
}
}
}
}
}
]

Get outer json key value into very json in the nested json array using JOLT

I have a JSON like this
{
"id": "1234",
"name": "something",
"list": [
{
"A": "Something"
},
{
"B": "Something1"
}
]
}
What I would like is to do is add id and name to both the JSON's present inside list
I've gone through a couple of questions and I couldn't find anywhere where somebody had done this.
I believe the following Shift spec will work:
{
"id|name": "&",
"list": {
"*": {
"#(2,id)": "&2.[&1].id",
"#(2,name)": "&2.[&1].name",
"*": "&2.[&1].&"
}
}
}
With your sample data, the output produced was:
{
"id": "1234",
"name": "something",
"list": [{
"id": "1234",
"name": "something",
"A": "Something"
}, {
"id": "1234",
"name": "something",
"B": "Something1"
}]
}
This spec should give you what you want:
[
{
"operation": "shift",
"spec": {
"list": {
"*": {
"#(2,id)": "&2.[&1].id",
"#(2,name)": "&2.[&1].name",
"*": "&2.[&1].&"
}
}
}
}
]
With your input it gives the following as output:
{
"list" : [ {
"A" : "Something",
"id" : "1234",
"name" : "something"
}, {
"B" : "Something1",
"id" : "1234",
"name" : "something"
} ]
}