Jolt - Merge two arrays and rename fields - json

I have a problem writing objects to an array. Basically I want to merge the arrays and rename the fields, while keeping the objects seperately.
My input json looks like this:
{
"board":[
{
"role":"Head of board",
"id":"111",
"name":"John Snow"
}
],
"leaders":[
{
"role":"Accounting leader",
"id":"222",
"name":"Amanda Johns"
},
{
"role":"HR leader",
"id":"333",
"name":"Frank Smith"
}
]
}
This is my spec: (I am aware that the values in brackets are probably not right)
[
{
"operation":"shift",
"spec":{
"board":{
"*":{
"id":"employees.bosses[#2].emp_num",
"role":"employees.bosses[#2].position",
"name":"employees.bosses[#2].name"
}
},
"leaders":{
"*":{
"id":"employees.bosses[#2].emp_num",
"role":"employees.bosses[#2].position",
"name":"employees.bosses[#2].name"
}
}
}
}
]
and this is my output:
{
"employees": {
"bosses": [ {
"emp_num": ["111", "222"],
"position": ["Head of board", "Accounting leader"],
"name": ["John Snow", "Amanda Johns"]
}, {
"emp_num": "333",
"position": "HR leader",
"name": "Frank Smith"
} ]
}
}
while I expect output that looks like this:
{
"employees": {
"bosses": [ {
"emp_num": "111",
"position": "Head of board",
"name": "John Snow"
}, {
"emp_num": "222",
"position": "Accounting leader",
"name": "Amanda Johns"
}, {
"emp_num": "333",
"position": "HR leader",
"name": "Frank Smith"
} ]
}
}
I have major troubles understanding what to do and how the [#n] work, I would really appreciate any help with fixing my spec and explaination why this does/does not work!

Need to distinguish the indexes of the arrays while combining them. To accomplish this, add a suffix letter or word(here I chose a for a.[&1]) for the first array while renaming all keys as desired within the first shift operation, then apply new extensions employees.bosses in the consecutive shift operation such as
[
{
"operation": "shift",
"spec": {
"board": {
"*": {
"id": "a.[&1].emp_num",
"role": "a.[&1].position",
"name": "a.[&1].name"
}
},
"leaders": {
"*": {
"id": "&1.emp_num",
"role": "&1.position",
"name": "&1.name"
}
}
}
},
{
"operation": "shift",
"spec": {
"*": "employees.bosses"
}
}
]

Related

conversion array into array by jolt

I have following json data . I want to transform following data by jolt nifi processor into result data
{
"data": [
{
"source": "Environment Sensors",
"alert_count": "2",
"category": "envs",
"alert_array": {
"alerts": [
{
"name": "neeraj",
"id": "123"
},
{
"name": "arun",
"id": "897"
}
]
}
}
]
}
result data
{
"alert": "2",
"subcategory": "envs",
"alert_array": [
{
"category": "Environment Sensors",
"newName": "neeraj",
"newID": "123"
},
{
"category": "Environment Sensors",
"newName": "arun",
"newID": "897"
}
]
}
Here value of category is the value of source
You can use the following shift transformation spec
[
{
"operation": "shift",
"spec": {
"data": {
"*": {
"alert_count": "alert",
"category": "sub&",
"alert_array": {
"*": {
"*": {
"#(3,category)": "&3[&1].category",
"name": "&3[&1].newName",
"id": "&3[&1].newID"
}
}
}
}
}
}
}
]
first reach the elements of the innermost object, and determine the value of the category element by going 3 levels up along with the others which are alreay located there.

Create array of simpler JSON objects from a nested JSON using jolt

My input JSON is like
{
"common": {
"name": "abc"
},
"details": {
"id": 4,
"node": [
{
"name": "node1",
"array2": []
},
{
"name": "node2",
"array2": [
{
"name": "node2_a2_1"
}
]
},
{
"name": "node3",
"array2": [
{
"name": "node3_a2_1"
},
{
"name": "node3_a2_2"
},
{
"name": "node3_a2_3"
}
]
}
]
}
}
What I want is for each leaf node in array2 (e.g. {"name": "node3_a2_1"}) I will traverse towards the root and add all common items and create an array of JSON objects without any nested object. So, the output I want is like
[
{
"common_name": "abc",
"id": 4,
"node_name": "node2",
"name": "node2_a2_1"
},
{
"common_name": "abc",
"id": 4,
"node_name": "node3",
"name": "node3_a2_1"
},
{
"common_name": "abc",
"id": 4,
"node_name": "node3",
"name": "node3_a2_2"
},
{
"common_name": "abc",
"id": 4,
"node_name": "node3",
"name": "node3_a2_3"
}
]
Could you please suggest how can I do that?
You can walk through the array2 while picking values of each element from their original location in the JSON value.
For example; traverse } four times to grab the value of id by using #(4,id), and use #(3,name)[&1] as the common distinguishing identifier for each attribute such as
[
{
"operation": "shift",
"spec": {
"details": {
"node": {
"*": {
"array2": {
"*": {
"#(5,common.name)": "#(3,name)[&1].common_name",
"#(4,id)": "#(3,name)[&1].id",
"#(2,name)": "#(3,name)[&1].node_name",
"name": "#(3,name)[&1].&"
}
}
}
}
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"*": ""
}
}
}
]

Jolt transformation array data

I want to transform a JSON using JOLT like this:
Input: {
"array": [
"1","2","3","4"
],
"array2": [
{
"something": "123",
"something1": "Plane"
},
{
"something3": "567",
"something4": "Car"
}
]
}
Into the following format, as you can see from output I need data from both arrays to fit exact param names, not empty one like in the first or existed param names like in the second one.
Output: {
"one_array": [
{
"code": "1",
"description": "",
},
{
"code": "2",
"description": "",
},
{
"code": "3",
"description": "",
},
{
"code": "4",
"description": "",
}
], "other_array": [
{
"id": "123",
"type": "Plane"
},
{
"id": "567",
"type": "Car"
}
]
}
Some clarifications are really appreciated
You can achieve this using 2 shift operations and the default operation as below.
[
{
"operation": "shift",
"spec": {
"array": {
"*": {
"#": "one_array[&].id"
}
},
"array2": {
"*": {
"*": {
"#": "tmp_array[&2]"
}
}
}
}
},
{
"operation": "shift",
"spec": {
"one_array": "one_array",
"tmp_array": {
"*": {
"0": "other_array[&1].id",
"1": "other_array[&1].type"
}
}
}
},
{
"operation": "default",
"spec": {
"one_array[]": {
"*": {
"description": ""
}
}
}
}
]

Jolt: Merge arrays from properties

I'm trying to extract and merge objects from an array contained in some (but not all) of my input elements. Using the JOLT JSON transformation library.
Also, the arrays I'm trying to merge contain objects that don't always have the same properties. One key might be present in some, but not others.
Example is contrived/nonsensical simplification, but has the general shape of our data.
Input:
{
"Widgets": [
{
"Id": "1",
"PetFriendly": "True",
"Features": [
{
"Name": "Easy Button",
"Type": "Button"
},
{
"Name": "Lunch Lever",
"Type": "Food Service",
"MenuItems": [
"Pizza",
"Cheezburger"
]
}
]
},
{
"Id": "2",
"PetFriendly": "True"
},
{
"Id": "3",
"PetFriendly": "False",
"Features": [
{
"Name": "Missles",
"Type": "Attack"
}
]
},
{
"Id": "4",
"PetFriendly": "False",
"Features": [
{
"Name": "Bombs",
"Type": "Attack",
"MenuItems": [
"Rat Poison"
]
}
]
}
]
}
Desired output:
{
"Widgets": [
{
"Id": "1"
"PetFriendly": "True"
},
{
"Id": "2"
"PetFriendly": "True"
},
{
"Id": "3",
"PetFriendly": "False"
},
{
"Id": "4",
"PetFriendly": "False"
}
],
"Features": [
{
"WidgetId": "1",
"Name": "Easy Button",
"Type": "Button"
},
{
"WidgetId": "1",
"Name": "Lunch Lever",
"Type": "Food Service",
"MenuItems": [
"Pizza",
"Cheezburger"
]
},
{
"WidgetId": "3",
"Name": "Missles",
"Type": "Attack"
},
{
"WidgetId": "4",
"Name": "Bombs",
"Type": "Attack",
"MenuItems": [
"Rat Poison"
]
}
]
}
I have tried many transforms with no success, and read all the ShiftR documentation and its unit tests. A little help?
Spec
[
{
"operation": "shift",
"spec": {
"Widgets": {
"*": {
// build the finished "Widgets" output
"Id": "Widgets[&1].Id",
"PetFriendly": "Widgets[&1].PetFriendly",
//
// Process the Features, by pushing the Id
// down into them, but maintain the same doubly
// nested structure.
// Shift works property by property, so first
// fix the properties in side each Features element,
// (pulling ID down).
// Then in a 2nd Shift can accumulate things into array.
"Features": {
"*": {
"#(2,Id)": "temp[&3].Features[&1].WidgetId",
"*": "temp[&3].Features[&1].&"
}
}
}
}
}
},
{
"operation": "shift",
"spec": {
// passthru
"Widgets": "Widgets",
"temp": {
"*": {
"Features": {
// walk thru the doubly nested structure an
// now accumulate all non-null itens into
// the the final Features array.
"*": "Features[]"
}
}
}
}
}
]
Finally got it working with the below spec, BUT it has an undesirable side effect: It leaves empty default arrays. Is there a way to remove empty arrays, or otherwise mark them during the default step so they can be deleted? I checked this GitHub issue but not sure how to translate it to arrays of string. Anyone have a better solution?
[
// First fill in default value for "MenuItems" since not all Features have it.
{
"operation": "default",
"spec": {
"Widgets[]": {
"*": {
"Features[]": {
"*": {
"MenuItems": []
}
}
}
}
}
},
{
// Extract the Features' properties into arrays. The defaults added above ensure that we can merge the arrays into Feature objects as in this example:
// https://github.com/bazaarvoice/jolt/blob/master/jolt-core/src/test/resources/json/shiftr/mergeParallelArrays2_and-do-not-transpose.json.
"operation": "shift",
"spec": {
"Widgets": {
"*": {
"Id": "Widgets[&1].Id",
"PetFriendly": "Widgets[&1].PetFriendly",
"Features": {
"*": {
"#(2,Id)": "temp.WidgetId",
"Name": "temp.Name",
"Type": "temp.Type",
"MenuItems": "temp.MenuItems[]"
}
}
}
}
}
},
// Finally merge the arrays into Feature objects.
{
"operation": "shift",
"spec": {
"Widgets": "Widgets",
"temp": {
"WidgetId": {
"*": "Features[&0].WidgetId"
},
"Name": {
"*": "Features[&0].Name"
},
"Type": {
"*": "Features[&0].Type"
},
"MenuItems": {
"*": "Features[&0].MenuItems"
}
}
}
}
]
Result:
{
"Widgets": [
{
"Id": "1",
"PetFriendly": "True"
},
{
"Id": "2",
"PetFriendly": "True"
},
{
"Id": "3",
"PetFriendly": "False"
},
{
"Id": "4",
"PetFriendly": "False"
}
],
"Features": [
{
"WidgetId": "1",
"Name": "Easy Button",
"Type": "Button",
"MenuItems": []
},
{
"WidgetId": "1",
"Name": "Lunch Lever",
"Type": "Food Service",
"MenuItems": [ "Pizza", "Cheezburger" ]
},
{
"WidgetId": "3",
"Name": "Missles",
"Type": "Attack",
"MenuItems": []
},
{
"WidgetId": "4",
"Name": "Bombs",
"Type": "Attack",
"MenuItems": [ "Rat Poison" ]
}
]
}

Transform JSON-JSON JOLT

I am quite new to JOLT and I need to transform my JSON files to the desired schema. This is my input
[
{
"PK": 12345,
"FULL_NAME":"Amit Prakash",
"BIRTHDATE":"1987-05-25",
"SEX":"M",
"EMAIL": "amprak#mail.com",
"PHONE": "809386731",
"TS":"2015-11-19 14:36:34.0"
},
{
"PK": 12654,
"FULL_NAME": "Rohit Dhand",
"BIRTHDATE":"1979-02-01",
"SEX":"M",
"EMAIL": "rodha#mail.com",
"PHONE": "937013861",
"TS":"2015-11-20 11:03:02.6"
},
...
]
and this is my desired output:
{
"records": [
{
"attribs": [{
"type": "customer",
"reference": "CUST"
}],
"name": "Amit Prakash",
"personal_email": "amprak#mail.com",
"mobile": "809386731",
"id": 12345
},
{
"attribs": [{
"type": "customer",
"reference": "CUST"
}],
"name": "Rohit Dhand",
"personal_email": "rodha#mail.com",
"mobile": "937013861",
"id": 12654
},
...
]
}
So far, I have only managed up to this point:
[
{
"operation": "remove",
"spec": {
"*": {
"BIRTHDATE": "",
"SEX": "",
"TS": ""
}
}
},
{
"operation": "shift",
"spec": {
"*": "records"
}
}
]
But I can't go on from here. I don't know how to rename keys in the output.
Also, what's the alternative to remove operation? remove operation is good if you have fewer keys to exclude than to include, but how about the reverse (few keys to include, more than to exclude within a JSON object)?
Spec
[
{
"operation": "shift",
"spec": {
"*": {
"PK": "records[&1].id",
"PHONE": "records[&1].mobile",
"EMAIL": "records[&1].personal_email",
"FULL_NAME": "records[&1].name"
}
}
},
{
"operation": "default",
"spec": {
"records[]": {
"*": {
"attribs[]": {
"0": {
"type": "customer",
"reference": "CUST"
}
}
}
}
}
}
]
Shift makes copy of your data, whereas the other operations do not. Thus, one way to remove stuff is to just not have it copied across in your shift spec.
Generally the remove is used to get rid of things that would "mess up" the shift.