Jolt transform Json with list within list - json

I have a output that I want to jolt transform but I'm having a difficult time doing so
Json Example:
{
"message": [
[
"2019",
"DATE"
],
[
"second",
"ORDINAL"
],
[
"Local",
"PERSON"
],
[
"2019",
"DATE"
],
[
"ISO",
"ORG"
],
[
"Ubuntu",
"PERSON"
]
]
}
I want to make the output look like this
{
"DATE": "2019",
"ORDINAL": "second",
"PERSON": "Local",
"DATE": "2019",
"ORG":"ISO",
"PERSON": "ubuntu"
}
The transformation should start with what displayed below but I get confused because of the list and the changing key and values. The original shows the keys as the capitalized values while the values are the lower cased values, they are backwards in the original.
{
"operation": "shift",
"spec": {
....

The current desired output is not a valid JSON as having duplicate DATE and PERSON keys. What you want might be
{
"DATE" : [ "2019", "2019" ],
"ORDINAL" : "second",
"PERSON" : [ "Local", "Ubuntu" ],
"ORG" : "ISO"
}
Then, you can use the following spec
[
{
"operation": "shift",
"spec": {
"*": {
"*": {
"0": "#(2,[&1][1])"
}
}
}
}
]
where we walk the indexes of the messages array through use of consecutive "*" wildcards, and that forms such a JSON
{
"0" : [ "2019", "DATE" ],
"1" : [ "second", "ORDINAL" ],
"2" : [ "Local", "PERSON" ],
"3" : [ "2019", "DATE" ],
"4" : [ "ISO", "ORG" ],
"5" : [ "Ubuntu", "PERSON" ]
}
and then "0" is used to get the second components, and "#(2,[&1][1])" is used to get the first components of each arrays from that JSON value.

Related

How to append constant array to another array using Jolt spec transformation

I have the following input JSON:
[
{
"id": "id1",
"projectId": "ALL",
"composable": "true",
"schema": {
"properties": {},
"required": [ "name", "db_access" ]
}
},
{
"id": "id2",
"projectId": "ALL",
"composable": "true",
"schema": {
"properties": {},
"required": [ "test", "remote" ]
}
}
]
Desired JSON:
[
{
"id": "id1",
"projectId": "ALL",
"composable": "true",
"schema": {
"properties": {},
"required": [ "name", "db_access", "account", "region" ]
}
},
{
"id": "id2",
"projectId": "ALL",
"composable": "true",
"schema": {
"properties": {},
"required": [ "test", "remote", "account", "region" ]
}
}
]
I wish to append the "account" and "region" Strings to the beginning or end (it doesn't matter) of the "required" array.
How do I achieve that using Jolt spec transformation?
You can use such a shift transformation spec
[
{
"operation": "shift",
"spec": {
"*": {
"*": "[&1].&", // attributes (or objects/arrays other than "schema"
"schema": {
"*": "[&2].&1.&", // attributes (or objects/arrays other than "required"
"required": {
"#region|#account": "[&3].&2.&1", // [&3] : represents going three levels up the tree to get values of the indexes in array-wise manner, &2 : replicates "schema", &1 : one level up to grab "required"
"*": "[&3].&2.&1"
}
}
}
}
}
]
the demo on the site http://jolt-demo.appspot.com/ is

Unable to extract values from a dynamic list of lists in a JSON response in Power Automate

In Power Automate, from the following json response which is obtained from azure-application-insights, I wish to extract each of the sub-lists in the “rows” list. The number of sub-lists would be variable depending upon the responses from the insights.
{
"tables": [
{
"name": "PrimaryResult",
"columns": [
{
"name": "Name",
"type": "dynamic"
},
{
"name": "Question",
"type": "dynamic"
},
{
"name": "Answer",
"type": "dynamic"
],
"rows": [
[
"John",
"when are you going",
"January’ 2022?"
],
[
"Albert",
"who will add deal co-owners for a deal?",
"The authorizers"
],
[
"Mohan",
"co-deal owner name can be added by whom?",
"The approvers”
]
]
}
]
}
I wish to dynamically extract the contents of all the sub-lists, and use each of the extracted outputs separately.

Jolt Transform on Nested JSON Object Array

Would like to flatten JUST the properties nested JSON, but still work for all objects in the input array
Having trouble putting all three together in one spec (type field, geo field, properties field). I wrote specs to do each one individually, but when I combine the specs to be of use together in one object, it produces the wrong output - the array of objects really messes it up.
Thanks!
Input:
[
{
"type": "Feature",
"geometry": {
"type": "MultiLineString",
"coordinates": [
[
[
-11.11,
11.11
]
]
]
},
"properties": {
"tester_email": "tester123#123.com",
"phase_test": "Test 1"
}
},
{
"type": "Feature22222",
"geometry": {
"type": "MultiLineString",
"coordinates": [
[
[
-11.11,
11.11
]
]
]
},
"properties": {
"tester_email": "tester123#123.com",
"phase_test": "Test 1"
}
}
]
JOLT Spec:
[{
"operation": "shift",
"spec": {
"*": {
"type": "[&1].type",
"geometry": "[&1].geometry",
"properties": "[&1]"
}
}
}
]
Current Output:
[ [ {
"type" : "Feature",
"geometry" : {
"type" : "MultiLineString",
"coordinates" : [ [ [ -11.11, 11.11 ] ] ]
}
}, {
"tester_email" : "tester123#123.com",
"phase_test" : "Test 1"
} ], [ {
"type" : "Feature22222",
"geometry" : {
"type" : "MultiLineString",
"coordinates" : [ [ [ -11.11, 11.11 ] ] ]
}
}, {
"tester_email" : "tester123#123.com",
"phase_test" : "Test 1"
} ] ]
Desired Output:
[ {
"type" : "Feature",
"geometry" : {
"type" : "MultiLineString",
"coordinates" : [ [ [ -11.11, 11.11 ] ] ]
},
"tester_email" : "tester123#123.com",
"phase_test" : "Test 1"
},{
"type" : "Feature22222",
"geometry" : {
"type" : "MultiLineString",
"coordinates" : [ [ [ -11.11, 11.11 ] ] ]
},
"tester_email" : "tester123#123.com",
"phase_test" : "Test 1"
} ]
This worked:
[
{
"operation": "remove",
"spec": {
"*": {
"properties": {
"type": ""
}
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"*": "[&1].&",
"properties": {
"*": "[&2].&"
}
}
}
}
]

JQ query output in csv format

I have been trying to extract a csv from the below json file using jq but not able to get so far. Does any experts out here can help?
{
"values": [
{
"resourceId": "xxxx-xxxx-xxx-8b16-xxxxxx",
"property-contents": {
"property-content": [
{
"statKey": "config|name",
"timestamps": [
1517591034069
],
"values": [
"somebname.UNIVERSE.test.com"
]
},
{
"statKey": "summary|guest|ipAddress",
"timestamps": [
1517591034069
],
"values": [
"100.xx.5.xx"
]
},
{
"statKey": "summary|parentCluster",
"timestamps": [
1551120506024
],
"values": [
"UFO-UFO"
]
},
{
"statKey": "summary|parentDatacenter",
"timestamps": [
1551120806021
],
"values": [
"GALAXY-D123"
]
},
{
"statKey": "summary|parentVcenter",
"timestamps": [
1517591334271
],
"values": [
"X-RAY123"
]
},
{
"statKey": "summary|runtime|powerState",
"timestamps": [
1517591034069
],
"values": [
"Powered On"
]
}
]
}
},
..
...
xxx-xxxx-xxx-8b16-xxxxxx,somebname.UNIVERSE.test.com,100.xx.5.xx,UFO-UFO,GALAXY-D123,X-RAY123,Powered On
Expected o/p is:
xxx-xxxx-xxx-8b16-xxxxxx,somebname.UNIVERSE.test.com,100.xx.5.xx,UFO-UFO,GALAXY-D123,X-RAY123,Powered On
Your expected output leaves some things unclear:
The second CSV column contains somebname.UNIVERSE.test.com, which was presumably derived from the section "property-content": [ { ..., "values": [ "somebname.UNIVERSE.test.com" ], ... }. How do you determine which element in the "property-content" list to pick for the second column? Is it because it's the first element? Is it because of its "statKey": "config|name"?
What if the "property-content" list is empty? What if it doesn't have the "statKey" entry you're looking for? What if the "values" list has zero or more than one element? The CSV row can only contain one scalar value. The same question applies for subsequent columns.
Making a wild guess here,
$ jq -r '.values[] | [ .resourceId, (."property-contents"."property-content"[] | .values[]) ] | join(",")' your.json
xxxx-xxxx-xxx-8b16-xxxxxx,somebname.UNIVERSE.test.com,100.xx.5.xx,UFO-UFO,GALAXY-D123,X-RAY123,Powered On
I cannot guarantee (and somewhat doubt) that this works in the general case, but I've been unable to extract a general case from your one example.

Jolt grouping together

I have this input in JSON, I am having difficulty grouping things together:
[
{
"PK": "123",
"SURNAME": "CHEN",
"SEX": "F",
"DATE_OF_BIRTH": "1962-08-29 00:00:00.0",
"PHONE_TYPE": "05",
"PHONE_NO": "12312312",
"OPERATION": "INSERT",
}, {
"PK": "123",
"SURNAME": "CHEN",
"SEX": "F",
"DATE_OF_BIRTH": "1962-08-29 00:00:00.0",
"PHONE_TYPE": "04",
"PHONE_NO": "78787878",
"OPERATION": "UPDATE"
},{
"PK": "456",
"SURNAME": "DEV",
"SEX": "M",
"DATE_OF_BIRTH": "1953-06-06 00:00:00.0",
"PHONE_TYPE": "05",
"PHONE_NO": "34343434",
"OPERATION": "INSERT"
}, {
"CLIENT_ID": "456",
"SURNAME": "DEV",
"SEX": "M",
"DATE_OF_BIRTH": "1953-06-06 00:00:00.0",
"PHONE_TYPE": "02",
"PHONE_NO": "56565656",
"OPERATION": "DELETE",
}
]
And this is the expected output:
{
"Customers": [{
"MatchingProfile": {
"CustomerNumber": "", // leave blank
"DBType": "Oracle",
"DBKey": "123",
"LastName": "CHEN",
"Gender": "Female",
"Birthdate": "1962-08-29",
},
"Contacts": [{
"ContactType": "Fax",
"CountryCode": "", // leave blank
"Phone_Number": "12312312",
"Status": "Active"
}, {
"ContactType": "Mobile",
"CountryCode": "", // leave blank
"PhoneNumber": "78787878",
"Status": "Active"
}
]
},{
"MatchingProfile": {
"CustomerNumber": "", // leave blank
"DBType": "Oracle",
"DBKey": "456",
"LastName": "DEV",
"Gender": "Male",
"Birthdate": "1953-06-06",
},
"Contacts": [{
"ContactType": "Fax",
"CountryCode": "", // leave blank
"PhoneNumber": "34343434",
"Status": "Active"
}, {
"ContactType": "Office",
"CountryCode": "", // leave blank
"PhoneNumber": "56565656",
"Status": "Inactive"
}
]
}
]
}
The SEX from input is "M", "F", plus some other coded values. Corresponding values for the output Gender is "Male", "Female" and left "" (blank) otherwise. (Don't accuse me of being gender-biased, I know, this is a project requirement, okay? Not my call)
The OPERATION from input that is "INSERT" and "UPDATE" will be a corresponding Status: "Active" ; for "DELETE" it will be Status : "Inactive".
Plus the Birthdate output is truncated equivalent of DATE_OF_BIRTH, minus the time.
The PHONE_TYPE are the following: 02 - "Office", 04 - "Mobile", 05 - "Fax" (I purposedly left out the others).
Is it possible to have a mapping for this in Jolt? Can you show a spec? I'm new with Jolt and I am bit confused. This is 10x harder than Excel Pivot.
This is pretty much as close as OOTB Jolt can get. Note Jolt is for changing the structure of your data, not doing custom data mappings of things like "PHONE_TYPE": "04" means "Fax".
Transformed Output
{
"Customers" : [ {
"MatchingProfile" : {
"DBKey" : "123",
"Gender" : "F",
"LastName" : "CHEN",
"Birthdate" : "1962-08-29 00:00:00.0",
"Contacts" : [ {
"ContactType" : "05",
"Phone_Number" : "12312312",
"Status" : "INSERT"
}, {
"ContactType" : "04",
"Phone_Number" : "78787878",
"Status" : "UPDATE"
} ]
}
}, {
"MatchingProfile" : {
"DBKey" : "456",
"Gender" : "M",
"LastName" : "DEV",
"Birthdate" : "1953-06-06 00:00:00.0",
"Contacts" : [ {
"ContactType" : "05",
"Phone_Number" : "34343434",
"Status" : "INSERT"
}, {
"ContactType" : "02",
"Phone_Number" : "56565656",
"Status" : "DELETE"
} ]
}
} ]
}
Jolt Spec
[
// first pivot by the value of SURNAME
{
"operation": "shift",
"spec": {
"*": { // for each item in the array
"SURNAME": { // match SURNAME
"*": { // match any value of SURNAME
"#2": "&[]" // copy the whole record from 2 levels up to the SURNAME as an array, so we know that in the next step it is always an array
}
}
}
}
},
{
"operation": "shift",
"spec": {
"*": { // match CHEN or DEV
"0": {
// only pull pk, sex, dob from the first entry of the SURNAME array so as to not duplicate output
"PK": "Customers[#3].MatchingProfile.DBKey",
"SEX": "Customers[#3].MatchingProfile.Gender",
"SURNAME": "Customers[#3].MatchingProfile.LastName",
"DATE_OF_BIRTH": "Customers[#3].MatchingProfile.Birthdate",
// this does mean that the PHONE_TYPE has to be dealt with twice
// once for the zeroth item, and then once again for the rest
"PHONE_TYPE": "Customers[#3].MatchingProfile.Contacts[0].ContactType",
"PHONE_NO": "Customers[#3].MatchingProfile.Contacts[0].Phone_Number",
"OPERATION": "Customers[#3].MatchingProfile.Contacts[0].Status"
},
"*": {
// handle PHONE_TYPE and friends for the other records
"PHONE_TYPE": "Customers[#3].MatchingProfile.Contacts[&1].ContactType",
"PHONE_NO": "Customers[#3].MatchingProfile.Contacts[&1].Phone_Number",
"OPERATION": "Customers[#3].MatchingProfile.Contacts[&1].Status"
}
}
}
}
]
If if you find Jolt valuable for the pivot and the structure change, then your best bet is to "fixup" your input data array, aka map "PHONE_TYPE": "04" to "Fax", trim the 00:00:00 from the birthday, and then use Jolt to make the nested "Customers[].MatchingProfile.Contacts[]" structure.