JSON Transformation using JOLT: Nested Array and Concatenation - json

I have the following input JSON
[
{
"name": "Project1",
"Addresses": [
[
"B24",
"Niyam Street",
"67897",
"New York"
],
[
"A14",
"Prinston Str",
"London"
]
]
},
{
"name": "Project2",
"Addresses": [
[
"123",
"Portland Street",
"234"
],
[
"Lalbag",
"Kolaba"
],
[
"8th Avenue",
"3rd Signal"
]
]
}
]
I would like to transform it into like the below:
[
{
"name": "Project1",
"Addresses": [
"B24 Niyam Street 67897 New York",
"A14 Prinston Str London"
]
},
{
"name": "Project2",
"Addresses": [
"123 Portland Street 234",
"Lalbag Kolaba",
"8th Avenue 3rd Signal"
]
}
]
The Addresses attribute value is a two-dimensional array of dynamic size.
Could you please help me with a valid jolt spec or some hints to achieve it? I am lost. Thank you so much.

You can use combination of shift and modify transformations such as
[
{
"operation": "shift",
"spec": {
"*": {
"*": "&", // the attributes other than "Addresses"
"Addresses": {
"*": "&1.&2.&" // determine the objects with indexed key names(0,1,2...) where &2 represents going two level up the tree to get the outermost indices of the objects of the main array, & for indices of the "Addresses" array, and &1 to keep the key name "Addresses" to transfer to the upcoming specs
}
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"*": {
"*": "=join(' ',#(1,&))" // concatenate all components of each array respectively at once
}
}
}
},
{
// combine the attributes back again
"operation": "shift",
"spec": {
"*": {
"*": "[&].&1"
},
"A*": {
"*": {
"*": "[&1].&2"
}
}
}
}
]
the demo on the site http://jolt-demo.appspot.com/ is

Related

JSON to JSON transform using JOLTTransformJSON NiFi

I am using JOLTTransformJson Processor in Nifi.
My input is:
[
{
"col_name": "time",
"data_type": "timestamp",
"is_nullable": true
},
{
"col_name": "otherData",
"data_type": "string",
"is_nullable": false
}
]
I am using the below spec:
[
{
"operation": "shift",
"spec": {
"*": {
"col_name": "name",
"data_type": "type[0]",
"is_nullable": {
"true": "type[1]",
"false": "type[1]"
}
}
}
},
{
"operation": "default",
"spec": {
"*": {
"type[1]": "notnull"
}
}
}
]
Expected Output is :
{
"type": "record",
"name": "table_name",
"fields": [
{
"name": "time",
"type": [
"timestamp",
"null"
]
},
{
"name": "otherData",
"type": [
"string",
"notnull"
]
}
]
}
But getting the below one as the current result by combining all values in array like:
{
"name": [
"time",
"otherData"
],
"type": [
[
"timestamp",
"int"
],
null
]
}
Can someone please help what am I missing.
You can use the following first shift transformation spec through walking by the objects of the array in order to be able to repeatedly apply the techniques :
[
{
"operation": "shift",
"spec": {
"#record": "type", // fixed values formed by using # wildcards left-hand-side
"#table_name": "name",
"*": {
"col_*": "fields[#2].&(0,1)", // replicate the 1st replacement of the asterisk from the current(0th) level
"is_nullable": { // conditional logic starts here
"#(1,data_type)": "fields[#3].type",
"true": {
"#null": "fields[#4].type"
},
"false": {
"#notnull": "fields[#4].type"
}
}
}
}
},
{ //to sort the attributes within the fields array
"operation": "sort",
"spec": {
"fields": ""
}
},
{ //to sort whole result
"operation": "shift",
"spec": {
"type": "&",
"name": "&",
"fields": "&"
}
}
]
the second and third transformation are just added to sort the attributes/arrays as desired

Jolt Spec for Json that may or may not have Array

Any help is greatly Appreciated.
I Have input JSON that can have Phone in array or it can be blank or it can be missing.
[
{
"Name": "abc",
"Phone": [
{
"office-1": "123",
"home-1": "989"
},
{
"office-1": "456",
"home-1": "999"
}
],
"Email": "abc#123.com"
},
{
"Name": "efg",
"Phone": [],
"Email": "efg#123.com"
},
{
"Name": "xyz",
"Email": "xyz#123.com"
}
]
My Jolt is already able to convert the Phone number array, but it is not working if the label Phone is missing in JSON input.
Expected output:
[
{
"Name": "abc",
"office-1": "123",
"home-1": "989",
"Email": "abc#123.com"
},
{
"Name": "abc",
"office-1": "456",
"home-1": "999",
"Email": "abc#123.com"
},
{
"Name": "efg",
"Email": "efg#123.com"
},
{
"Name": "xyz",
"Email": "xyz#123.com"
}
]
Please help
You can walk through the objects after separating the stuff under Phone node and the others such as
[
{
"operation": "shift",
"spec": {
"*": {
"Phone": {
"*": {
"*": "&3.&1.&",
"#(2,Name)": "&3.&1.Name",
"#(2,Email)": "&3.&1.Email"
}
},
"*": "&1.&1.&" // "else" case
}
}
},
{
// get rid of object labels
"operation": "shift",
"spec": {
"*": {
"*": ""
}
}
},
{
// get rid of duplicated values of some attributes
"operation": "cardinality",
"spec": {
"*": {
"*": "ONE"
}
}
}
]
the demo on the site http://jolt-demo.appspot.com/ is
new sample of data where I have kept non array data in between array data

Jolt Transformation - getting last element by field Sequence

Say I have the following Json:
{
"collection": [
{
"sequence": 1,
"value": "event1"
},
{
"sequence": 2,
"value": "event2"
},
{
"sequence": 3,
"value": "event3"
}
]
}
Given that the array is not guaranteed to be sorted by "sequence", how do I extract the element that has the highest value of "sequence"?
the sub-objects of the "collection" array might be sorted by the sequence values by putting them as indexes for reformed array (call "objectWithHighestSequence") such as
[
{
"operation": "shift",
"spec": {
"collection": {
"*": "objectWithHighestSequence[#(0,sequence)]"
}
}
},
{
// only pick the last object after sorting within the shift transformation
"operation": "modify-overwrite-beta",
"spec": {
"*": "=lastElement(#(1,objectWithHighestSequence))"
}
}
]
the demo on the site http://jolt-demo.appspot.com/ is

Key to value using jolt

I need to transform multiple keys and their values to new JSON spec using jolt.
Input:
{
"product": "monitor",
"ID": "222",
"price": "300"
}
Expected output:
{
"record": [
{
"key": "product",
"value": "monitor"
},
{
"key": "ID",
"value": "222"
},
{
"key": "price",
"value": "300"
}
]
}
Thanks in advance
Convert each node in the input json into key/value and shift to the named object. And then named object is moved to the array.
[
{
"operation": "shift",
"spec": {
"*": {
"#": "&1.value",
"$": "&1.key"
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"#": "records"
}
}
}
]

Transform JSON-JSON JOLT

I am quite new to JOLT and I need to transform my JSON files to the desired schema. This is my input
[
{
"PK": 12345,
"FULL_NAME":"Amit Prakash",
"BIRTHDATE":"1987-05-25",
"SEX":"M",
"EMAIL": "amprak#mail.com",
"PHONE": "809386731",
"TS":"2015-11-19 14:36:34.0"
},
{
"PK": 12654,
"FULL_NAME": "Rohit Dhand",
"BIRTHDATE":"1979-02-01",
"SEX":"M",
"EMAIL": "rodha#mail.com",
"PHONE": "937013861",
"TS":"2015-11-20 11:03:02.6"
},
...
]
and this is my desired output:
{
"records": [
{
"attribs": [{
"type": "customer",
"reference": "CUST"
}],
"name": "Amit Prakash",
"personal_email": "amprak#mail.com",
"mobile": "809386731",
"id": 12345
},
{
"attribs": [{
"type": "customer",
"reference": "CUST"
}],
"name": "Rohit Dhand",
"personal_email": "rodha#mail.com",
"mobile": "937013861",
"id": 12654
},
...
]
}
So far, I have only managed up to this point:
[
{
"operation": "remove",
"spec": {
"*": {
"BIRTHDATE": "",
"SEX": "",
"TS": ""
}
}
},
{
"operation": "shift",
"spec": {
"*": "records"
}
}
]
But I can't go on from here. I don't know how to rename keys in the output.
Also, what's the alternative to remove operation? remove operation is good if you have fewer keys to exclude than to include, but how about the reverse (few keys to include, more than to exclude within a JSON object)?
Spec
[
{
"operation": "shift",
"spec": {
"*": {
"PK": "records[&1].id",
"PHONE": "records[&1].mobile",
"EMAIL": "records[&1].personal_email",
"FULL_NAME": "records[&1].name"
}
}
},
{
"operation": "default",
"spec": {
"records[]": {
"*": {
"attribs[]": {
"0": {
"type": "customer",
"reference": "CUST"
}
}
}
}
}
}
]
Shift makes copy of your data, whereas the other operations do not. Thus, one way to remove stuff is to just not have it copied across in your shift spec.
Generally the remove is used to get rid of things that would "mess up" the shift.