i want to use JOLT Transform to convert an dictionary to an JSON Object. Here below i demonstrate it with an example below.
There is in the root an Array which contains the results from different computers. "Computer1", "Computer2", and more.
This structure should be stay. I removed the 2nd and more array elemt which will reoccur in the same way with {...}.
Given Object:
[
{
"name": "Computer1",
"events": {
"counts": [
{
"countType": "CRITICAL",
"count": 5
},
{
"countType": "HIGH",
"count": 12
},
{
"countType": "LOW",
"count": 40
}
]
},
"processes": {
"counts": [
{
"countType": "CRITICAL",
"count": 0
},
{
"countType": "HIGH",
"count": 2
},
{
"countType": "LOW",
"count": 80
}
]
}
},
{
"name": "Computer2",
"events": {...},...
}
]
Desired Output:
[
{
"name": "Computer1",
"events": {
"CRITICAL": 5,
"HIGH": 12,
"LOW": 40
},
"processes": {
"CRITICAL": 0,
"HIGH": 2,
"LOW": 80
}
}
, {
"name": "Computer2",
"events": {...},
...
}
]
Please help to identify the right JOLT spec.
Thanks in advance
Marcus
You can use such a shift transformation specs
[
{
"operation": "shift",
"spec": {
"*": {
"name": "&1.&",
"events|processes": { // pipe represents "OR" logic
"counts": {
"*": {
"#count": "&4.&3.#countType" // "&4" and "&3" represent going four and three level up the tree to grab the indices of the outermost level list and the key name of the objects("events" and "processes") respectively
}
}
}
}
}
},
{
"operation": "shift",
"spec": {
"*": ""
}
}
]
Related
This is an input that needs to be transformed using Jolt Transformation to obtain the expected output.
I am attempting to create a jolt transformation for the below input:
{
"flights": [
{
"id": "123",
"route": "BJS-SIN"
},
{
"id": "456",
"route": "SIN-PEK"
},
{
"id": "789",
"route": "SIN-BJS"
}
],
"prices": [
{
"id": "abc",
"amount": 560
},
{
"id": "def",
"amount": 780
}
],
"solutions": [
{
"price-ref": "abc",
"flights-ref": [
"123",
"456"
]
},
{
"price-ref": "def",
"flights-ref": [
"123",
"789"
]
}
]
}
Desired output would like :
{
"solutions": [
{
"flights": [
{
"id": "123",
"route": "BJS-SIN"
},
{
"id": "456",
"route": "SIN-PEK"
}
],
"price": {
"id": "abc",
"amount": 560
}
},
{
"flights": [
{
"id": "123",
"route": "BJS-SIN"
},
{
"id": "789",
"route": "SIN-BJS"
}
],
"price": {
"id": "def",
"amount": 780
}
}
]
}
As the output, all the data should be constructed refer to the solutions node. How about the mapping expression in JOLT?
all the values from the source json are generated randomly and the keys names are fixed.
I had tried many times but could not find the right spec, so please help.
You can use this spec:
[
{
"operation": "shift",
"spec": {
"*": "&",
"flights|prices": {
"*": {
"*": "&2.#(1,id).&"
}
}
}
},
{
"operation": "shift",
"spec": {
"solutions": {
"*": {
"flights*": {
"*": {
"*": {
"#(6,flights.&)": "&5[&4].flights[&2]"
}
}
},
"price*": {
"*": {
"#(5,prices.&)": "&4[&3].price"
}
}
}
}
}
}
]
I have following json data . I want to transform following data by jolt nifi processor into result data
{
"data": [
{
"source": "Environment Sensors",
"alert_count": "2",
"category": "envs",
"alert_array": {
"alerts": [
{
"name": "neeraj",
"id": "123"
},
{
"name": "arun",
"id": "897"
}
]
}
}
]
}
result data
{
"alert": "2",
"subcategory": "envs",
"alert_array": [
{
"category": "Environment Sensors",
"newName": "neeraj",
"newID": "123"
},
{
"category": "Environment Sensors",
"newName": "arun",
"newID": "897"
}
]
}
Here value of category is the value of source
You can use the following shift transformation spec
[
{
"operation": "shift",
"spec": {
"data": {
"*": {
"alert_count": "alert",
"category": "sub&",
"alert_array": {
"*": {
"*": {
"#(3,category)": "&3[&1].category",
"name": "&3[&1].newName",
"id": "&3[&1].newID"
}
}
}
}
}
}
}
]
first reach the elements of the innermost object, and determine the value of the category element by going 3 levels up along with the others which are alreay located there.
My input JSON is like
{
"common": {
"name": "abc"
},
"details": {
"id": 4,
"node": [
{
"name": "node1",
"array2": []
},
{
"name": "node2",
"array2": [
{
"name": "node2_a2_1"
}
]
},
{
"name": "node3",
"array2": [
{
"name": "node3_a2_1"
},
{
"name": "node3_a2_2"
},
{
"name": "node3_a2_3"
}
]
}
]
}
}
What I want is for each leaf node in array2 (e.g. {"name": "node3_a2_1"}) I will traverse towards the root and add all common items and create an array of JSON objects without any nested object. So, the output I want is like
[
{
"common_name": "abc",
"id": 4,
"node_name": "node2",
"name": "node2_a2_1"
},
{
"common_name": "abc",
"id": 4,
"node_name": "node3",
"name": "node3_a2_1"
},
{
"common_name": "abc",
"id": 4,
"node_name": "node3",
"name": "node3_a2_2"
},
{
"common_name": "abc",
"id": 4,
"node_name": "node3",
"name": "node3_a2_3"
}
]
Could you please suggest how can I do that?
You can walk through the array2 while picking values of each element from their original location in the JSON value.
For example; traverse } four times to grab the value of id by using #(4,id), and use #(3,name)[&1] as the common distinguishing identifier for each attribute such as
[
{
"operation": "shift",
"spec": {
"details": {
"node": {
"*": {
"array2": {
"*": {
"#(5,common.name)": "#(3,name)[&1].common_name",
"#(4,id)": "#(3,name)[&1].id",
"#(2,name)": "#(3,name)[&1].node_name",
"name": "#(3,name)[&1].&"
}
}
}
}
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"*": ""
}
}
}
]
I would like to know in Jolt, how do I count the number of levels I need to moved up to get the required data.
I played around the Jolt spec to convert value in "quantity" from 0 to 1
Input
{
"items": [
{
"product": {
"name": "product1",
"id": "001"
},
"quantity": 1
},
{
"product": {
"name": "product2",
"id": "002"
},
"quantity": 0
},
{
"product": {
"name": "product3",
"id": "003"
},
"quantity": 0
}
]
}
The expected output
{
"items": [
{
"product": {
"name": "product1",
"id": "001"
},
"quantity": 1
},
{
"product": {
"name": "product2",
"id": "002"
},
"quantity": 1
},
{
"product": {
"name": "product3",
"id": "003"
},
"quantity": 1
}
]
}
Jolt spec with some notes for what I understand.
[
{
"operation": "shift",
"spec": {
"items": {
"*": {
"quantity": {
"0": {
"#1": "items.[&3].quantity" //[&3] => I move 3 levels up to items.* to get index?
},
"*": {
"#(2,quantity)": "items.[&3].quantity" //#(2,quantity) => I move 2 levels up to get items.*.quantity value?
}
},
"*": "items.[&1].&" //[&1] => I move 1 level up to items.* to get index?
}
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"items": {
"*": {
"quantity": "=toInteger(#(1,quantity))"
}
}
}
}
]
Do I understand it correctly? Please advice.
Thanks you
As much as I understand, you want to reflect the first quantity value within the items array to all other quantity values of the other objects. Then you can use these shift transformations :
[
{
"operation": "shift",
"spec": {
"items": {
"0": {
"#": "&"
},
"*": {
"#(0,product)": "&.product",
"#(2,&1[0].quantity)": "&.quantity"
}
}
}
},
{
"operation": "shift",
"spec": {
"*": "items"
}
}
]
I am trying to use the jolt JSON to JSON transformation in Apache NiFi. I want to transform one JSON into another format.
Here is my original JSON:
{
"total_rows": 5884,
"offset": 0,
"rows": [
{
"id": "03888c0ab40c32451a018be6b409eba3",
"key": "03888c0ab40c32451a018be6b409eba3",
"value": {
"rev": "1-d5cc089dd8682422962ccab4f24bd21b"
},
"doc": {
"_id": "03888c0ab40c32451a018be6b409eba3",
"_rev": "1-d5cc089dd8682422962ccab4f24bd21b",
"topic": "iot-2/type/home-iot/id/1234/evt/temp/fmt/json",
"payload": {
"temperature": 36
},
"deviceId": "1234",
"deviceType": "home-iot",
"eventType": "temp",
"format": "json"
}
},
{
"id": "03888c0ab40c32451a018be6b409f163",
"key": "03888c0ab40c32451a018be6b409f163",
"value": {
"rev": "1-dee82cbb1b5ffa8a5e974135eb6340c5"
},
"doc": {
"_id": "03888c0ab40c32451a018be6b409f163",
"_rev": "1-dee82cbb1b5ffa8a5e974135eb6340c5",
"topic": "iot-2/type/home-iot/id/1234/evt/temp/fmt/json",
"payload": {
"temperature": 22
},
"deviceId": "1234",
"deviceType": "home-iot",
"eventType": "temp",
"format": "json"
}
}
]
}
I want this to be transformed in the following JSON:
[
{
"temperature":36,
"deviceId":"1234",
"deviceType":"home-iot",
"eventType":"temp"
},
{
"temperature":22,
"deviceId":"1234",
"deviceType":"home-iot",
"eventType":"temp"
}
]
This is what my spec looks like:
[
{
"operation": "shift",
"spec": {
"rows": {
"*": {
"doc": {
"deviceId": "[&1].deviceId",
"deviceType": "[&1].deviceType",
"eventType": "[&1].eventType",
"payload": {
"*": "[&1]"
}
}
}
}
}
}
]
I keep getting a null response. I am new to this and the documentation is not very easy to comprehend. Can somebody please help?
Because you are "down" one more level after the array index, by the time you get to deviceId you are 2 levels away from the index. Replace all the &1s with &2 except for payload. In that case you are another level "down" so you'll want to use &3 for the index. You also need to take whatever is matched by the * (temperature, e.g.) and set the outgoing field name to the same thing, by using & after the array index. Here's the resulting spec:
[
{
"operation": "shift",
"spec": {
"rows": {
"*": {
"doc": {
"deviceId": "[&2].deviceId",
"deviceType": "[&2].deviceType",
"eventType": "[&2].eventType",
"payload": {
"*": "[&3].&"
}
}
}
}
}
}
]