Jolt Transformation to "unpivot" repeating columns into nested JSON array - json

My source data is a huge flatten CSV which contains data that I'd like to normalise and store in a number of separate database tables. I'm wondering if I can use a JOLT transformation to alter the structure of my CSV data which I've already a flat JSON object.
For example, the object below has two 'parent' rows and three repeating pairs of columns which are the 'child' rows:
{
"id": 1,
"name": "zzz",
"code_1": "abc",
"date_1": "2021-01-01",
"code_2": "def",
"date_2": "2021-01-02",
"code_3": "ghi",
"date_3": "2021-01-03"
}
Can I use JOLT to generate the output:
{
"id": 1,
"name": "zzz",
"codes": [
{
"code": "abc",
"date": "2021-01-01"
},
{
"code": "def",
"date": "2021-01-02"
},
{
"code": "ghi",
"date": "2021-01-03"
}
]
}
I've had a play around with the Jolt Playground, but as yet I've not found something which could achieve this goal.
Any hints, tips, or pointers welcome.
Thanks in advance.
UPDATE:
I'm getting closer, with the following spec and output. My codes and dates are in different objects however:
spec:
[
{
"operation": "shift",
"spec": {
"identifier": "id",
"name": "name",
"code*": "codes[].code",
"date*": "codes[].date"
}
}
]
output:
{
"id" : 1,
"name" : "zzz",
"codes" : [ {
"code" : "abc"
}, {
"date" : "2021-01-01"
}, {
"code" : "def"
}, {
"date" : "2021-01-02"
}, {
"code" : "ghi"
}, {
"date" : "2021-01-03"
} ]
}
(Edit: corrected the required output.)

You can use shift transformations twice as the elements are grouped under common numbered keys by _1,_2,_3 in the first step, and then remove keys of those objects while nesting them within codes list such as
[
{
"operation": "shift",
"spec": {
"*": "&",
"code*": { "#": "&(0,1).code" },
"date*": { "#": "&(0,1).date" }
}
},
{
"operation": "shift",
"spec": {
"_*": "codes[]",
"*": "&"
}
}
]

Related

How to dynamically add key and values from one object into another object in an array via jolt

I'm using jolt and I have an input object where I would like to take the keys out of one property and insert them into each object of an array in another property dynamically:
My input:
{
"data": {
"NAN_KEY": 1,
"TEMP": 3
},
"attributes": [
{
"name": "attribute1",
"value": 3
},
{
"name": "attribute2",
"value": 2
}
]
}
The result I'm aiming for:
"attributes": [
{
"name": "attribute1",
"value": 3,
"NAN_KEY": 1,
"TEMP": 3
},
{
"name": "attribute2",
"value": 2,
"NAN_KEY": 1,
"TEMP": 3
}
]
This question was posted previously in this thread
But after using the solution, I realized I needed it to dynamically add the entire object instead of hardcoding the fields
Any help is appreciated!
Yes, it's possible to make it more dynamic such as
[
{
"operation": "shift",
"spec": {
"attributes": {
"*": {
"#2,data": { "*": "[&1].&" }, // go two levels up the tree to grab the values of the "data" array
"*": "[&1].&"
}
}
}
}
]
the demo on the site http://jolt-demo.appspot.com/ is

Using JOLT Transformation Need to remove the sqaure brakets of an object

I am trying remove a field from array of JSON using JOLT Transformation. below is the input JSON and expected output JSON.
can you suggest the JOLT transformation for the mentioned scenario
Input:
{
"code": 200,
"data": {
"totalCount": 1,
"SCount": 1,
"FCount": 0,
"FSequences": [],
"token": [
328358
]
},
"id": "ce27g26hvamob9lbd0eg"
}
Jolt I'm using:
[
{
"operation": "shift",
"spec": {
"data": {
"*": "&"
},
"status|id": "&"
}
}
]
Expected output:
{
"totalCount" : 1,
"SCount" : 1,
"FCount" : 0,
"FSequences" : "",
"token" : "328358",
"status" : 200,
"id" : "ce27g26hvamob9lbd0eg"
}
You can consecutively apply shift and modify specs such that
[
{
"operation": "shift",
"spec": {
"data": {
"*": "&",
"token": {
"*": "&1" // replicates the tag "token" by grabbing after going up tree one level
}
},
"code": "status", // renaming the attribute
"id": "&"
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"*u*n*": ["=toInteger", ""], // conversion only doesn't apply for "FSequences" among *u*n* tagged attributes
"*o*n": "=toString" // only applied for "token" attribute
}
}
]

Copy a value from field to another field using Jolt while preserving the rest of the document

I'm trying to use a JOLT spec to copy the value from a JSON object to another field in the same object and keep the original document (with the new field) .
I'm trying using the shift operator, but I just can't figure out how to preserve the document.
Here's an example of the Object I'm trying to transform:
{
"contact": {
"name": "Foo",
"id": "123456",
"phone": "231-123"
},
"event": {
"name": "create",
"type": "test"
}
}
And the output should be something like this:
{
"contact": {
"name": "Foo",
"id": "123456",
"userId": "123456",
"phone": "231-123"
},
"event": {
"name": "create",
"type": "test"
}
}
In this case I want to copy the value from id to a new field named "userId"
The specs I've tried are:
[
{
"operation": "shift",
"spec": {
"*": "&",
"contact": {
"userId": "contact.id"
}
}
}
]
But that just deletes everything but the event node.
Using a modify transform would suit best for your case such as
[
{
"operation": "modify-overwrite-beta",
"spec": {
"contact": {
"userId": "#(1,id)" // goes 1 level up the tree and grabs the value of the "id" attribute
}
}
}
]
the demo on the site http://jolt-demo.appspot.com/ is :
Btw, using a shift transformation is also possible such as
[
{
"operation": "shift",
"spec": {
"contact": {
"*": "&1.&",
"#(0,id)": "&1.userId" // in this case, we have "0" instead of "1", since no need to go up the tree as identifier of value #(0,id) is on the left hand side, exactly stays at the same level with the tag "id"
},
"*": "&"
}
}
]
but obviously, the first one is simpler.

Jolt Transform - Filter Array Where Fields equal

In a Jolt transform Im trying to filter an array of objects by an id, if equal to an Id on a field outside of the array.
Here is my data
{
"position": {
"positionManagerId": "123"
},
"managers": [
{
"id": "123",
"name": "John"
},
{
"id": "456",
"name": "Jack"
}
]
}
Id like to grab the manager where
managers.id == position.positonManagerId
My output would look like
{
"managerId": "123",
"managerName": "John"
}
You can apply shift transformation twice.
Combine key-value pairs under the common id values as key names within the first one so as to get a list for the searched id value( 123 in this case ) while the other pair(s) won't be lists.
Then use index values(0and1) within the second spec in order to eliminate the pair(s) without list value such as
[
{
"operation": "shift",
"spec": {
"position": {
"*": {
"$": "#(0)"
}
},
"managers": {
"*": {
"#name": "#id"
}
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"0": {
"$1": "managerId"
},
"1": "managerName"
}
}
}
]

JOLT transformation to copy single value along an array

I want to transform a JSON into key-value pairs and simultaneously copy a specific single value, i.e. the "timestamp", to all these pairs.
Input JSON:
{
"Timestamp": "2018-05-13T14:57:09",
"first_key": "1023",
"another_key": "1987",
"yet_another_key": "677"
}
Expected output:
[ {
"Timestamp": "2018-05-13T14:57:09.087",
"key": "first_key",
"value": "1023"
},
{
"Timestamp": "2018-05-13T14:57:09.087",
"key": "another_key",
"value": "1987"
},
{
"Timestamp": "2018-05-13T14:57:09.087",
"key": "yet_another_key",
"value": "677"
}]
What I came up with so far is the following JOLT specification. It already generates the key-value pairs for all entries that are not "Timestamp", but how can I copy the value of "Timestamp" into each of these records?
[{
"operation": "shift",
"spec": {
"Timestamp": "[].Timestamp",
"*": {
"$": "[#2].key",
"#": "[#2].value"
}
}
}]
Output of the above JOLT specification:
[ {
"Timestamp" : "2018-05-13T14:57:09"
}, {
"key" : "first_key",
"value" : "1023"
}, {
"key" : "another_key",
"value" : "1987"
}, {
"key" : "yet_another_key",
"value" : "677"
} ]
Spec
[
{
// first separate the Timestamp from the fields that
// are going to be pivoted.
// This is needed because the "[#2]" logic
// isn't doing what you think it is /
// you would end up with a null in your output array.
// Basically the "[#2]" logic as written only works
// when you are cleanly pivoting data. Which you are
// not because "Timestamp" and "first_key" are siblings.
"operation": "shift",
"spec": {
"Timestamp": "Timestamp",
"*": "keysToPivot.&"
}
},
{
"operation": "shift",
"spec": {
"keysToPivot": {
"*": {
// Now that we can cleanly loop thru _only_ the
// keysToPivot, the [#2] logic will work "right".
// Additionally, lookup the Timestamp value
// and add it to each "pivoted" output.
"$": "[#2].key",
"#": "[#2].value",
"#(2,Timestamp)": "[#2].Timestamp"
}
}
}
}
]