how to use $ and * at the same level in a spec - json

I am new to jolt and whilst i like lots of it one thing thats really hurting me right now is how to use * and $ at the same level in a spec. I have the following desired input and output. But try as i might i cannot seem to transform both the list of action ids (there are the "1" and "2" into attribute values and move the list of action data associated with the id into a sub attribute.
Input
{
"Attr1": "Attr1_data",
"Actions": {
"1": [
"Action data 1 line 1",
"Action data 1 line 2",
"Action data 1 line 3"
],
"2": [
"Action data 2 line 1",
"Action data 2 line 2",
"Action data 2 line 3"
]
},
"Attr2": "Attr2_data"
}
Desired
{
"Attr1": "Attr1_data",
"Action": [
{
"id" : "1",
"data" : [
"Action data 1 line 1",
"Action data 1 line 2",
"Action data 1 line 3"
]
},
{
"id" : "2",
"data" : [
"Action data 2 line 1",
"Action data 2 line 2",
"Action data 2 line 3"
]
}
],
"Attr2": "Attr2_data"
}
using the following spec
[
{
"operation" : "shift",
"spec": {
"Actions": {
"*" : {
"$": "Action[].id"
}
},
"*": "&"
}
}
]
I can generate
{
"Attr1": "Attr1_data",
"Action": [
{
"id": "1"
},
{
"id": "2"
}
],
"Attr2": "Attr2_data"
}
But try as i might i cannot seem to copy the data lines in to a new data attribute.
Can anyone pls point me in the right direction ?

You can convert yours to the following transformation spec
[
{
"operation": "shift",
"spec": {
"*s": { // represents a tag(of an object/array/attribute) with a trailing letter "s". The reason of this reform is to be able use "Action" as the key of the inner array without repeatedly writing it.
"*": {
"$": "&(2,1)[#2].id", // "$" looks one level up and copies the tag name, &(2,1) goes two levels up the tree and pick the first piece separated by asterisk, [#2] goes two level up in order to reach the level of "Actions" array to combine the subelements distributed from that level in arrayic manner with "Action"(&(2,1)) label, and the leaf node "id" stands for tag of the current attribute
"*": "&(2,1)[#2].data"
}
},
"*": "&" // the rest of the attributes(/objects/arrays) other than "Actions"
}
}
]
the demo on the site http://jolt-demo.appspot.com/ is

Related

Combine two values to an object with a JOLT transformation

I am pretty new to Jolt. The source format was generated from some software and does not have a good structure, so I would like to transform this. Each text ("First text" and "Second text") has an associated link ("First text" -> "Link of first text"; "Second text" -> "Link of second text"). How would you do such a transformation? Any help please?
Sample Input:
{
"flow": [
"^First text",
{
"*": ".^.c-0"
},
"^Second text",
{
"*": ".^.c-1"
},
{
"c-0": [
{
"->": "Link of first text"
}
],
"c-1": [
{
"->": "Link of second text"
}
]
}
]
}
Expected Output:
{
"flow": {
"links": [
{
"message": "^First text",
"link": "Link of first text"
},
{
"message": "^Second text",
"link": "Link of second text"
}
]
}
}
What you need is to use shift transformation.
One option is to create individual arrays(lists) within the first,
and then dissipate them to separate objects within the second transformation such as
[
{
"operation": "shift",
"spec": {
"flow": {
"*": {
"*text": { // filter out by the key names ending with "text"
"$": "&3.message" // go 3 levels up to grab the key name "flow"
},
"c*": { // filter out by the key names starting with "c"
"*": {
"*": "&4.link" // go 4 levels up to grab the key name "flow"
}
}
}
}
}
},
{
"operation": "shift",
"spec": {
"*": { // stands for "flow"
"*": {
"*": "&2.links[&].&1" // &2 -> go 2 levels up to grab the key name "flow"
// &1 -> go 1 level up to grab the key names of the arrays
}
}
}
}
]
the demo on the site http://jolt-demo.appspot.com/ is

Split a string and trim a known prefix from each part in a complex JSON structure

I'm dealing with a fairly complex JSON-structure in which a single entry needs to be edited in several places. For example:
[
{
"name": "test 1",
"stuff": {
"properties": {
"id": 0,
"stuff_list": [
{
"entryId": 1,
"description": "- item 1\n- item 2\n- item 3"
},
{
"entryId": 2,
"description": "- item 1\n- item 2\n- item 3"
}
]
}
}
},
{
"name": "test 2",
"stuff": {
"properties": {
"id": 1,
"stuff_list": [
{
"entryId": 1,
"description": null
},
{
"entryId": 2,
"description": "- item 1\n- item 2\n- item 3"
}
]
}
}
}
]
Here I would like to edit each "description"-element: The string needs to be split at each \n and the substrings "^\n?-\s" of each resulting array element need to be removed. So it should result in:
{
"entryId": 1,
"description": ["item 1", "item 2", "item 3"]
}
My first approach is:
jq '.[].stuff.properties.stuff_list[].description | split("\n")' the_file.json
but that's not working in the first place becaue of the null values that can occur at some places. So now I wonder: how can I achieve what I want?
An alternate version using split() on the \n and trimming string - on the left, would be to do
.[].stuff.properties.stuff_list[].description |=
if . != null then
split("\n") | map(ltrimstr("- "))
else
.
end
jqplay - Demo

JOLT Spec to build arrays from group of similarly named keys

Trying to build a JOLT spec to put similarly named keys into an array, and strip off leading 6 characters. In below example, all keys starting with "fd1lk1" would go into array "Link1", keys starting with "fd1lk2" would go into array "Link2", etc.
Thanks for any help!
Source JSON:
{
"EventName": "WidgetFeedImpression",
"WidgetName": "_blah_2019.08.17",
"WidgetID": "5d56ef313db7c300018d9c66",
"WidgetVariationName": "_blah_2019.08.17",
"WidgetVariationID": "5b5f524eb1932300014d0928",
"WidgetTemplate": "blah-six-grid-wth-image",
"fd1lk1Title": "link 1 title",
"fd1lk1Image": "link 1 image",
"fd1lk1TargetURL": "link 1 url",
"fd1lk1Position": "1",
"fd1lk1Id": "fd5da8ce0f8701a3000190efbdlk1",
"fd1lk2Title": "link 2 title",
"fd1lk2Image": "link 2 image",
"fd1lk2TargetURL": "link 2 url",
"fd1lk2Position": "2",
"fd1lk2Id": "fd5da8ce0f8701a3000190efbdlk2",
"gtmcb": "1878625665",
"CreatedAtUtc": "2019-10-24T16:57:01.5274702Z"
}
Desired Output:
{
"EventName": "WidgetFeedImpression",
"WidgetName": "_blah_2019.08.17",
"WidgetID": "5d56ef313db7c300018d9c66",
"WidgetVariationName": "_blah_2019.08.17",
"WidgetVariationID": "5b5f524eb1932300014d0928",
"WidgetTemplate": "blah-six-grid-wth-image",
"Link1" : [ {
"Title" : "link 1 title",
"Image" : "link 1 image",
"TargetURL" : "link 1 url",
"Position" : "1",
"Id" : "fd5da8ce0f8701a3000190efbdlk1"
} ],
"Link2" : [ {
"Title" : "link 2 title",
"Image" : "link 2 image",
"TargetURL" : "link 2 url",
"Position" : "2",
"Id" : "fd5da8ce0f8701a3000190efbdlk2"
} ],
"gtmcb": "1878625665",
"CreatedAtUtc": "2019-10-24T16:57:01.5274702Z"
}
The * in fd1lk*Title is extracted and used with &(0,1):
[
{
"operation": "shift",
"spec": {
"fd1lk*Title": "Link&(0,1).[0].Title",
"fd1lk*Image": "Link&(0,1).[0].Image",
"fd1lk*TargetURL": "Link&(0,1).[0].TargetURL",
"fd1lk*Position": "Link&(0,1).[0].Position",
"fd1lk*Id": "Link&(0,1).[0].Id",
"*": {
"#": "&"
}
}
}
]
See here for the basic concept.

How can I use "not equal" condition while filtering array using JOLT specification

I want to filter JSON array using JOLT transformation, where condition is negative. In the below example I want only records where URL value is not equal to Not Available.
{
"Photos": [
{
"Id": "327703",
"Caption": "TEST>> photo 1",
"Url": "Not Available."
},
{
"Id": "327704",
"Caption": "TEST>> photo 2",
"Url": "http://bob.com/0001/327704/photo.jpg"
},
{
"Id": "327705",
"Caption": "TEST>> photo 3",
"Url": "http://bob.com/0001/327705/photo.jpg"
}
]
}
Take a look on very similar question Removing Elements from array based on a condition. Based on it you can solve it as below:
[
{
"operation": "shift",
"spec": {
"Photos": {
// loop thru all the photos
"*": {
// for each URL
"Url": {
// For "Not Available." do nothing.
"Not Available.": null,
// In other case pass thru
"*": {
"#2": "Photos[]"
}
}
}
}
}
}
]
Generally when you want to negate filter you do a filter and as transformation pass null which skips item.

Jolt reference first element in array as target name

I have been looking at this for a few weeks (in the background) and am stumped on how to convert JSON data approximating a CSV into a tagged set using the NiFi JoltTransformJson processor. What I mean by this is to use the data from the first row of an array in the input as the JSON object name in the output.
As an example I have this input data:
[
[
"Company",
"Retail Cost",
"Percentage"
],
[
"ABC",
"5,368.11",
"17.09%"
],
[
"DEF",
"101.47",
"0.32%"
],
[
"GHI",
"83.79",
"0.27%"
]
]
and what I am trying to get as output is:
[
{
"Company": "ABC",
"Retail Cost": "5,368.11",
"Percentage": "17.09%"
},
{
"Company": "DEF",
"Retail Cost": "101.47",
"Percentage": "0.32%"
},
{
"Company": "GHI",
"Retail Cost": "83.79",
"Percentage": "0.27%"
}
]
I see this as primarily 2 problems: getting access to the content of the first array and then making sure that the output data does not include that first array.
I would love to post a Jolt Specification showing myself getting somewhat close, but the closest gives me the correct shape of output without the correct content. It looks like this:
[
{
"operation": "shift",
"spec": {
"*": {
"*": "[&1].&0"
}
}
}
]
But it results in an output like this:
[ {
"0" : "Company",
"1" : "Retail Cost",
"2" : "Percentage"
}, {
"0" : "ABC",
"1" : "5,368.11",
"2" : "17.09%"
}, {
"0" : "DEF",
"1" : "101.47",
"2" : "0.32%"
}, {
"0" : "GHI",
"1" : "83.79",
"2" : "0.27%"
} ]
Which clearly has the wrong object name and it has 1 too many elements in the output.
Can do it, but wow it is hard to read / looks like terrible regex
Spec
[
{
// this does most of the work, but producs an output
// array with a null in the Zeroth space.
"operation": "shift",
"spec": {
// match the first item in the outer array and do
// nothing with it, because it is just "header" data
// e.g. "Company", "Retail Cost", "Percentage".
// we need to reference it, but not pass it thru
"0": null,
//
// loop over all the rest of the items in the outer array
"*": {
// this is rather confusing
// "*" means match the array indices of the innner array
// and we will write the value at that index "ABC" etc
// to "[&1].#(2,[0].[&])"
// "[&1]" means make the ouput be an array, and at index
// &1, which is the index of the outer array we are
// currently in.
// Then "lookup the key" (Company, Retail Cost) using
// #(2,[0].[&])
// Which is go back up the tree to the root, then
// come back down into the first item of the outer array
// and Index it by the by the array index of the current
// inner array that we are at.
"*": "[&1].#(2,[0].[&])"
}
}
},
{
// We know the first item in the array will be null / junk,
// because the first item in the input array was "header" info.
// So we match the first item, and then accumulate everything
// into a new array
"operation": "shift",
"spec": {
"0": null,
"*": "[]"
}
}
]