Pls help with this jolt transformation.
Note:
If there is field "ServiceFamily" then change the field name to
"tag1"
If there is field "PublisherName" then change the field name to
"tag2"
Input:
[
{
"ServiceFamily": "Compute",
"CostAllocationRuleName": null,
"benefitId": null,
"benefitName": null
},
{
"PublisherName": "Microsoft",
"ChargeType": "Usage",
"Frequency": "UsageBased",
"PricingModel": "OnDemand",
"benefitName": null
}
]
Expected output:
[
{
"Tag1": "Compute",
"CostAllocationRuleName": null,
"benefitId": null,
"benefitName": null
},
{
"Tag2": "Microsoft",
"ChargeType": "Usage",
"Frequency": "UsageBased",
"PricingModel": "OnDemand",
"benefitName": null
}
]
You can use such a shift transformation spec
[
{
"operation": "shift",
"spec": {
"*": {
"ServiceFamily": "[#2].Tag1",
"PublisherName": "[#2].Tag2",
"*": "[#2].&"
}
}
}
]
the demo on the site http://jolt-demo.appspot.com/ is
Alternatively you can use the following one which consecutively applies modify and remove transformation specs
[
{
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"Tag1": "=(#(1,ServiceFamily))",
"Tag2": "=(#(1,PublisherName))"
}
}
},
{
"operation": "remove",
"spec": {
"*": {
"ServiceFamily": "",
"PublisherName": ""
}
}
}
]
the demo on the site http://jolt-demo.appspot.com/ is
Related
I'm coming to the conclusion that Jolt is beyond me.
With this input data:-
{
"cluster_id": "1",
"data": {
"id": 1,
"types": [
{
"incident_id": 10,
"incident_ref": "AAA",
"incident_code": "123",
"incident_date": "2010-11-15T00:01:00Z"
},
{
"incident_id": 20,
"incident_ref": "BBB",
"incident_code": "456",
"incident_date": "2020-11-15T00:01:00Z"
}
]
}
}
Spec:-
[
{
"operation": "shift",
"spec": {
"cluster_id": "id",
"data": {
"types": {
"*": {
"incident_id": "incidents",
"incident_ref": "incidents"
}
}
}
}
}
]
Gives:-
{
"id" : "1",
"incidents" : [ 10, "AAA", 20, "BBB" ]
}
How would I get the result of:-
{
"id" : "1",
"incidents" : [
{"id": 10, "ref": "AAA", "code": "123", date: "2010-11-15T00:01:00Z"},
{"id": 20, "ref": "BBB", "code": "456", date: "2020-11-15T00:01:00Z"},
]
}
Tried a bunch of permutations but getting nowhere!
You can use this spec:
[
{
"operation": "shift",
"spec": {
"*": "id",
"data": {
"types": {
"*": {
"incident_*": "incidents[&1].&(0,1)"
}
}
}
}
}
]
To prevent using incident text in the spec, you can use the below spec:
[
{
"operation": "shift",
"spec": {
"*": "id",
"data": {
"types": {
"*": {
"*_*": "&(0,1)[&1].&(0,2)"
}
}
}
}
}
]
You can use two level of shift transformations by extracting substring from tag names such as
[
{
"operation": "shift",
"spec": {
"*": "id",
"data": {
"types": {
"*": "incidents"
}
}
}
},
{
"operation": "shift",
"spec": {
"id": "&",
"*": {
"*": {
"*_*": "&2[&1].&(0,2)"
}
}
}
}
]
I'm trying to write a spec to do the below transformation using jolt transformation. I need to convert the flat json to nested Json
I am having some trouble with converting the flat JSON to Nested JSON. I have looked at examples and didn't get any closer as to what is mentioned above. I need to transform a JSON structure by using a JOLT spec. I use https://jolt-demo.appspot.com to test the following below.
Input :
[
{
"container_id": "a",
"carrier_scac": "b",
"location": "banglore",
"state": "karnataka",
"country": "India"
},
{
"container_id": "a",
"carrier_scac": "b",
"location": "pune",
"state": "maharashtra",
"country": "India"
},
{
"container_id": "c",
"carrier_scac": "d",
"location": "dharwad",
"state": "kan",
"country": "India"
},
{
"container_id": "c",
"carrier_scac": "d",
"location": "hubli",
"state": "kant",
"country": "India"
}
]
Desired Output:
[
{
"load": {
"container_id": "a",
"carrier_scac": "b",
"stops": [
{
"location": "banglore",
"state": "karnataka"
},
{
"location": "pune",
"state": "maharashtra"
}
]
},
"containerInfo": {
"country": "India"
}
},
{
"load": {
"container_id": "c",
"carrier_scac": "d",
"stops": [
{
"location": "dharwad",
"state": "kan"
},
{
"location": "hubli",
"state": "kant"
}
]
},
"containerInfo": {
"country": "India"
}
}
]
Jolt Spec that I'm using :
[
{
"operation": "shift",
"spec": {
"*": {
"container_id": "#(1,container_id).&",
"carrier_scac": "#(1,container_id).&",
"*": "#(1,container_id).stops[&1].&"
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"*": "=recursivelySquashNulls"
}
},
{
"operation": "cardinality",
"spec": {
"*": {
"container_id": "ONE",
"carrier_scac": "ONE"
}
}
},
{
"operation": "shift",
"spec": {
"*": ""
}
}
]
Current spec is pretty good, just needs some little modifications such as adding load and containerInfo nodes, and shortening a bit as below
[
{
"operation": "shift",
"spec": {
"*": {
"*": "#(1,container_id).load.stops[&1].&", // "else" case
"country": "#(1,container_id).load.containerInfo.&",
"c*": "#(1,container_id).load.&" // the attributes starting with "c" but other than "country"
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"*": "=recursivelySquashNulls"
}
},
{
"operation": "cardinality",
"spec": {
"*": {
"*": {
"c*": "ONE",
"containerInfo": {
"*": "ONE"
}
}
}
}
},
{
"operation": "shift",
"spec": {
"*": ""
}
}
]
I want to transform this JSON:
[
{
"payload": {
"before": {
"order_id": 3110445250,
"data": "2022-06-10",
"ora": "08:08:00",
"test1": "2022-06-10T00:00:00.000Z",
"test2": "2022-06-10T03:00:00.000Z"
},
"after": null,
"source": {
"file": "delta.000002",
"pos": 8362,
"row": 0,
"thread": null,
"query": null
},
"op": "c",
"ts_ms": 1654851760103,
"transaction": null
}
}
]
Desired output:
if "after" is null then remove "after" and rename "before" to "final"
else if "before" is null then remove "before" and rename after to "final"
else if "after" is not null and "before" is not null then remove "before" and rename after to "final"
[
{
"payload": {
"before": {
"order_id": 3110445250,
"data": "2022-06-10",
"ora": "08:08:00",
"test1": "2022-06-10T00:00:00.000Z",
"test2": "2022-06-10T03:00:00.000Z"
},
"source": {
"file": "delta.000002",
"pos": 8362,
"row": 0,
"thread": null,
"query": null
},
"op": "c",
"ts_ms": 1654851760103,
"transaction": null
}
}
]
Thanks a lot!
You can use a conditional logic within a shift transformation after determining the sizes of before/after objects/attributes through use of a modify transformation in which we compute their respective sizes whether they return a value. In cases the value doesn't return, then -1 is assigned to determine the element(object or attribute) to be null such as
[
{
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"payload": {
"before_sz": ["=size(#(1,before))", -1],
"after_sz": ["=size(#(1,after))", -1]
}
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"payload": {
"before_sz": {
"-1": {
"#(2,after_sz)": {
"-1": { "": "" },
"*": { "#(4,after)": "[&6].&5.final" }
}
},
"*": {
"#(2,after_sz)": {
"-1": { "#(4,before)": "[&6].&5.final" },
"*": { "#(4,after)": "[&6].&5.final" }
}
}
},
"*": "[&2].&1.&"
}
}
}
},
{
"operation": "remove",
"spec": {
"*": {
"payload": {
"after_sz": "",
"after": "",
"before_sz": "",
"before": ""
}
}
}
},
{
"operation": "sort"
}
]
that will return
[
{
"payload": {
"final": {
"data": "2022-06-10",
"ora": "08:08:00",
"order_id": 3110445250,
"test1": "2022-06-10T00:00:00.000Z",
"test2": "2022-06-10T03:00:00.000Z"
},
"op": "c",
"source": {
"file": "delta.000002",
"pos": 8362,
"query": null,
"row": 0,
"thread": null
},
"transaction": null,
"ts_ms": 1654851760103
}
}
]
for the current input as a sample case.
PS.: as a missing case for the question : if both after/before are null, then both are removed within this solution.
Edit : If the input is not nested within square brackets(as you asked as an extra case), then convert the spec to this one :
[
{
"operation": "modify-overwrite-beta",
"spec": {
"payload": {
"before_sz": ["=size(#(1,before))", -1],
"after_sz": ["=size(#(1,after))", -1]
}
}
},
{
"operation": "shift",
"spec": {
"payload": {
"before_sz": {
"-1": {
"#(2,after_sz)": {
"-1": { "": "" },
"*": { "#(4,after)": "&5.final" }
}
},
"*": {
"#(2,after_sz)": {
"-1": { "#(4,before)": "&5.final" },
"*": { "#(4,after)": "&5.final" }
}
}
},
"*": "&1.&"
}
}
},
{
"operation": "remove",
"spec": {
"payload": {
"after_sz": "",
"after": "",
"before_sz": "",
"before": ""
}
}
},
{
"operation": "sort"
}
]
I have json converted from protobuf protocol with following format:
{
"id": "6aa0734f-6d6a-4b95-8a2b-2dde346f9df7",
"measurements": [
{
"ts": "1590331111510000000",
"values": [
{},
{
"name": 1,
"value": -1.8093990087509155
},
{
"name": 2,
"value": 0.35456427931785583
}
],
"parameters": [
"Stat",
"VoltageAnglePhaseB"
]
}
]
}
expected output is:
{
"id" : "6aa0734f-6d6a-4b95-8a2b-2dde346f9df7",
"ts" : "1590331111510000000",
"Stat":-1.8093990087509155,
"VoltageAnglePhaseB":0.35456427931785583
}
I've started my Jolt spec like this:
[
{
"operation": "shift",
"spec": {
"id": "id",
"measurements": {
"*": {
"ts": "ts",
"parameters": {
"*": ""
},
"*": ""
}
}
}
}
]
But facing the problem to extract name and value from the JSON.
Does anyone have an idea?
It can be done with 2 shift operations. First, shift the values to the valuesArr, same way shift the parameters to the parametersArr. Then match the values with the index.
[
{
"operation": "shift",
"spec": {
"id": "id",
"measurements": {
"*": {
"ts": "ts",
"values": {
"*": {
"value": "valuesArr"
}
},
"parameters": "parametersArr"
}
}
}
},
{
"operation": "shift",
"spec": {
"id": "id",
"ts": "ts",
"valuesArr": {
"*": "#(2,parametersArr[&])"
}
}
}
]
I have this JSON for input:
{
"id": 1031435,
"event_id": "Formula_257",
"formula_id": 257,
"ts_start": 1583164200084000,
"ts_end": 1583164484960000,
"type": "formula",
"details": {
"6aa0734f-6d6a-4b95-8a2b-2dde346f9df7": {
"PowerActiveTriPhase": 183836912
}
},
"ack_ts": null,
"ack_user": null
}
and I need to get this kind of output:
{
"id": 1031435,
"event_id": "Formula_257",
"formula_id": 257,
"ts_start": 1583164200084000,
"ts_end": 1583164484960000,
"type": "formula",
"equipment_id":"6aa0734f-6d6a-4b95-8a2b-2dde346f9df7",
"parameter":"PowerActiveTriPhase",
"value":183836912,
"ack_ts": null,
"ack_user": null
}
What kind of spec do I need to use?
Thanks a lot!
This should work
[
{
"operation": "shift",
"spec": {
"id": "id",
"event_id": "event_id",
"formula_id": "formula_id",
"ts_start": "ts_start",
"ts_end": "ts_end",
"type": "type",
"details": {
"*": {
"$": "equipment_id",
"*": {
"$": "parameter",
"#": "value"
}
}
},
"ack_ts": "ack_ts",
"ack_user": "ack_user"
}
}
]