I have a json like this coming from an input port:
{
"url": "blablabla",
"keyword": "foo"
}
then I have to generate a new json to pass to a post call. The new json is something like this:
{
"requests": [
{
"source": "blablabla",
"params": {
"keywords": [
"something"
],
"sub-field1": true
}
}
],
"field1": "1",
"field2": "2",
"field3": false
}
where the array keywords should be replaced with a new array with the value of the previous Json ("foo"). The resulting is:
{
"requests": [
{
"source": "blablabla",
"params": {
"keywords": [
"foo"
],
"sub-field1": true
}
}
],
"field1": "1",
"field2": "2",
"field3": false
}
then i have to call a function via REST API Post.
The problem is that I don't know how to perform the replace of the field value
Try JoltTransformJSON with the following spec:
[
{
"operation": "shift",
"spec": {
"url": "requests[0].source",
"keyword": "requests[0].params.keywords[]",
"#1": "requests[0].params.sub-field1"
}
},
{
"operation": "default",
"spec": {
"field1": "1",
"field2": "2",
"field3": false
}
}
]
Related
In my current JSON, I am getting an empty JSON object {} inside dummy_var5. The empty object is inside an array which is inside an object itself.
{
"dummy_var1": "abc",
"dummy_var2": [
{
"item": {
"action": "test",
"po": {
"id": "abc"
},
"ot": "test1",
"id": "1"
}
}
],
"dummy_var3": {
"dummy_var4": [
{
"name": "test",
"value": "test1"
},
{
"name": "test",
"value": "test1"
}
],
"name": "test2"
},
"dummy_var5": [
{
"ref": "test",
"name": "test1",
"type": null
},
{}
],
"dummy_var6": [
{
"role": "test",
"ref": "test1",
"partyDescription": "test2"
}
]
}
Considering this structure does not change, Is there any way to remove this via jolt. We tried using third party tool 'atlasmap' but couldn't achieve the desired result.
Expected output :-
{
"dummy_var1": "abc",
"dummy_var2": [
{
"item": {
"action": "test",
"po": {
"id": "abc"
},
"ot": "test1",
"id": "1"
}
}
],
"dummy_var3": {
"dummy_var4": [
{
"name": "test",
"value": "test1"
},
{
"name": "test",
"value": "test1"
}
],
"name": "test2"
},
"dummy_var5": [
{
"ref": "test",
"name": "test1",
"type": null
}
],
"dummy_var6": [
{
"role": "test",
"ref": "test1",
"partyDescription": "test2"
}
]
}
You can use this single shift transformation spec
[
{
"operation": "shift",
"spec": {
"*": "&", // the attributes other than "dummy_var5"
"dummy_var5": {
"*": {
"*": "&2.[&1].&"
}
}
}
}
]
the match "*":"&" of the line "*": "&2.[&1].&" returns the null value from the leaf node for this level, and so removes the innermost null object {}
I want to query for the complete Json.
I have Json objects saved in in different documents.
_id: 1,
my_json: {
"field1": "1",
"field2": "2",
"field3": "3",
"field4": "4"
}
_id: 2,
my_json: {
"field1": "11",
"field2": "22",
"field3": "33",
"field4": "44"
}
In my query I want to match for the complete combination not the single field. So I tried the query below but it is returning empty response.
{
"query": {
"bool": {
"must": [
{
"terms": {
"my_json": [
{
"field1": "11",
"field2": "22",
"field3": "33",
"field4": "44"
},
{
"field1": "1",
"field2": "2",
"field3": "3",
"field4": "4"
}
]
}
}
]
}
}
}
I need to work with more and more Json objects like above. So I cannot use the regular query.Hence I tried to match the complte the JSON object.
If you want to match all properties of a JSON object, you can do so but using one term query per field in your JSON, i.e. each field of your JSON must match
{
"query": {
"bool": {
"must": [
{
"term": {
"my_json.field1": "11"
}
},
{
"term": {
"my_json.field2": "22"
}
},
{
"term": {
"my_json.field3": "33"
}
},
{
"term": {
"my_json.field4": "44"
}
}
]
}
}
}
I am trying to use the jolt JSON to JSON transformation in Apache NiFi. I want to transform one JSON into another format.
Here is my original JSON:
{
"total_rows": 5884,
"offset": 0,
"rows": [
{
"id": "03888c0ab40c32451a018be6b409eba3",
"key": "03888c0ab40c32451a018be6b409eba3",
"value": {
"rev": "1-d5cc089dd8682422962ccab4f24bd21b"
},
"doc": {
"_id": "03888c0ab40c32451a018be6b409eba3",
"_rev": "1-d5cc089dd8682422962ccab4f24bd21b",
"topic": "iot-2/type/home-iot/id/1234/evt/temp/fmt/json",
"payload": {
"temperature": 36
},
"deviceId": "1234",
"deviceType": "home-iot",
"eventType": "temp",
"format": "json"
}
},
{
"id": "03888c0ab40c32451a018be6b409f163",
"key": "03888c0ab40c32451a018be6b409f163",
"value": {
"rev": "1-dee82cbb1b5ffa8a5e974135eb6340c5"
},
"doc": {
"_id": "03888c0ab40c32451a018be6b409f163",
"_rev": "1-dee82cbb1b5ffa8a5e974135eb6340c5",
"topic": "iot-2/type/home-iot/id/1234/evt/temp/fmt/json",
"payload": {
"temperature": 22
},
"deviceId": "1234",
"deviceType": "home-iot",
"eventType": "temp",
"format": "json"
}
}
]
}
I want this to be transformed in the following JSON:
[
{
"temperature":36,
"deviceId":"1234",
"deviceType":"home-iot",
"eventType":"temp"
},
{
"temperature":22,
"deviceId":"1234",
"deviceType":"home-iot",
"eventType":"temp"
}
]
This is what my spec looks like:
[
{
"operation": "shift",
"spec": {
"rows": {
"*": {
"doc": {
"deviceId": "[&1].deviceId",
"deviceType": "[&1].deviceType",
"eventType": "[&1].eventType",
"payload": {
"*": "[&1]"
}
}
}
}
}
}
]
I keep getting a null response. I am new to this and the documentation is not very easy to comprehend. Can somebody please help?
Because you are "down" one more level after the array index, by the time you get to deviceId you are 2 levels away from the index. Replace all the &1s with &2 except for payload. In that case you are another level "down" so you'll want to use &3 for the index. You also need to take whatever is matched by the * (temperature, e.g.) and set the outgoing field name to the same thing, by using & after the array index. Here's the resulting spec:
[
{
"operation": "shift",
"spec": {
"rows": {
"*": {
"doc": {
"deviceId": "[&2].deviceId",
"deviceType": "[&2].deviceType",
"eventType": "[&2].eventType",
"payload": {
"*": "[&3].&"
}
}
}
}
}
}
]
Trying to get unique values stored in items array for each group. somehow it's always mixed...
My JSON looks like this:
{
"start": 1534425916,
"stop": 1535030716,
"groups": [
{
"group": "transmission",
"data": {
"events": 665762,
},
"items": [
{
"item": "manualni",
"data": {
"events": 389158,
}
},
{
"item": "automaticka",
"data": {
"events": 276604,
}
}
]
},
{
"group": "vat",
"data": {
"events": 671924,
},
"items": [
{
"item": "ne",
"data": {
"events": 346221,
}
},
{
"item": "ano",
"data": {
"events": 325703,
}
}
]
}
]
}
Desired result is the following:
{
"id": "transmission",
"value": [
"manualni",
"automaticka",
]
}
{
"id": "vat",
"value": [
"ne",
"ano"
]
}
Tried with this filter on command line:
| jq '{id: .groups[].group, value: [.groups[].items[].item]}'
Which results in the above mentioned mixed up result:
{
"id": "transmission",
"value": [
"manualni",
"automaticka",
"ne",
"ano"
]
}
{
"id": "vat",
"value": [
"manualni",
"automaticka",
"ne",
"ano"
]
}
Any idea how to receive the uniquified values here? Thanks in advance!
This gets the desired result. I think the manual entry under .[] explains why it works.
jq '.groups[] | {"id": .group, "value": [.items[].item]}'
I'm trying to extract and merge objects from an array contained in some (but not all) of my input elements. Using the JOLT JSON transformation library.
Also, the arrays I'm trying to merge contain objects that don't always have the same properties. One key might be present in some, but not others.
Example is contrived/nonsensical simplification, but has the general shape of our data.
Input:
{
"Widgets": [
{
"Id": "1",
"PetFriendly": "True",
"Features": [
{
"Name": "Easy Button",
"Type": "Button"
},
{
"Name": "Lunch Lever",
"Type": "Food Service",
"MenuItems": [
"Pizza",
"Cheezburger"
]
}
]
},
{
"Id": "2",
"PetFriendly": "True"
},
{
"Id": "3",
"PetFriendly": "False",
"Features": [
{
"Name": "Missles",
"Type": "Attack"
}
]
},
{
"Id": "4",
"PetFriendly": "False",
"Features": [
{
"Name": "Bombs",
"Type": "Attack",
"MenuItems": [
"Rat Poison"
]
}
]
}
]
}
Desired output:
{
"Widgets": [
{
"Id": "1"
"PetFriendly": "True"
},
{
"Id": "2"
"PetFriendly": "True"
},
{
"Id": "3",
"PetFriendly": "False"
},
{
"Id": "4",
"PetFriendly": "False"
}
],
"Features": [
{
"WidgetId": "1",
"Name": "Easy Button",
"Type": "Button"
},
{
"WidgetId": "1",
"Name": "Lunch Lever",
"Type": "Food Service",
"MenuItems": [
"Pizza",
"Cheezburger"
]
},
{
"WidgetId": "3",
"Name": "Missles",
"Type": "Attack"
},
{
"WidgetId": "4",
"Name": "Bombs",
"Type": "Attack",
"MenuItems": [
"Rat Poison"
]
}
]
}
I have tried many transforms with no success, and read all the ShiftR documentation and its unit tests. A little help?
Spec
[
{
"operation": "shift",
"spec": {
"Widgets": {
"*": {
// build the finished "Widgets" output
"Id": "Widgets[&1].Id",
"PetFriendly": "Widgets[&1].PetFriendly",
//
// Process the Features, by pushing the Id
// down into them, but maintain the same doubly
// nested structure.
// Shift works property by property, so first
// fix the properties in side each Features element,
// (pulling ID down).
// Then in a 2nd Shift can accumulate things into array.
"Features": {
"*": {
"#(2,Id)": "temp[&3].Features[&1].WidgetId",
"*": "temp[&3].Features[&1].&"
}
}
}
}
}
},
{
"operation": "shift",
"spec": {
// passthru
"Widgets": "Widgets",
"temp": {
"*": {
"Features": {
// walk thru the doubly nested structure an
// now accumulate all non-null itens into
// the the final Features array.
"*": "Features[]"
}
}
}
}
}
]
Finally got it working with the below spec, BUT it has an undesirable side effect: It leaves empty default arrays. Is there a way to remove empty arrays, or otherwise mark them during the default step so they can be deleted? I checked this GitHub issue but not sure how to translate it to arrays of string. Anyone have a better solution?
[
// First fill in default value for "MenuItems" since not all Features have it.
{
"operation": "default",
"spec": {
"Widgets[]": {
"*": {
"Features[]": {
"*": {
"MenuItems": []
}
}
}
}
}
},
{
// Extract the Features' properties into arrays. The defaults added above ensure that we can merge the arrays into Feature objects as in this example:
// https://github.com/bazaarvoice/jolt/blob/master/jolt-core/src/test/resources/json/shiftr/mergeParallelArrays2_and-do-not-transpose.json.
"operation": "shift",
"spec": {
"Widgets": {
"*": {
"Id": "Widgets[&1].Id",
"PetFriendly": "Widgets[&1].PetFriendly",
"Features": {
"*": {
"#(2,Id)": "temp.WidgetId",
"Name": "temp.Name",
"Type": "temp.Type",
"MenuItems": "temp.MenuItems[]"
}
}
}
}
}
},
// Finally merge the arrays into Feature objects.
{
"operation": "shift",
"spec": {
"Widgets": "Widgets",
"temp": {
"WidgetId": {
"*": "Features[&0].WidgetId"
},
"Name": {
"*": "Features[&0].Name"
},
"Type": {
"*": "Features[&0].Type"
},
"MenuItems": {
"*": "Features[&0].MenuItems"
}
}
}
}
]
Result:
{
"Widgets": [
{
"Id": "1",
"PetFriendly": "True"
},
{
"Id": "2",
"PetFriendly": "True"
},
{
"Id": "3",
"PetFriendly": "False"
},
{
"Id": "4",
"PetFriendly": "False"
}
],
"Features": [
{
"WidgetId": "1",
"Name": "Easy Button",
"Type": "Button",
"MenuItems": []
},
{
"WidgetId": "1",
"Name": "Lunch Lever",
"Type": "Food Service",
"MenuItems": [ "Pizza", "Cheezburger" ]
},
{
"WidgetId": "3",
"Name": "Missles",
"Type": "Attack",
"MenuItems": []
},
{
"WidgetId": "4",
"Name": "Bombs",
"Type": "Attack",
"MenuItems": [ "Rat Poison" ]
}
]
}