I am trying to write a spec to shift a few key in a json object to the very end of the object.
{
"id": "12345",
"timestamp": "2019-10-28 13:24:44.547",
"action": "notify",
"name": "test"
}
to:
{
"id": "12345",
"action": "notify",
"name": "test",
"timestamp": "2019-10-28 13:24:44.547"
}
Appreciate any leads on how to go about doing this using JOLT.
I believe that is an answer, but I'm not so sure that Jolt keeps an order.
[
{
"operation": "shift",
"spec": {
"id": "id",
"action": "action",
"name": "name",
"timestamp": "timestamp"
}
}
]
Related
I want to convert JSON format to another format so I use jolt for that. please if there is best from jolt recommend me to it.
when I map this object
{
"id": 1,
"username": "sd4s5d4",
"phone": "111",
"groups": [
{
"id": 1
},
{
"id": 2
}
]
}
the expected output is ok the value of groups is returned as a list
{
"id": 1,
"username": "sd4s5d4",
"phone": "111",
"groups": [ 1, 2 ]
}
but when I map this object
{
"id": 5,
"username": "sd4s5d4",
"phone": "111",
"groups": [
{
"id": 1
}
]
}
it returns
{
"id": 5,
"username": "sd4s5d4",
"phone": "111",
"groups": 1
}
how to make groups in last output to be list even if it one item.
wanted format
{
"id": 5,
"username": "sd4s5d4",
"phone": "111",
"groups": [1]
}
Spec
[
{
"operation": "shift",
"spec": {
"id": "id",
"username": "username",
"phone": "phone",
"groups": {
"*": {
"id": "groups"
}
}
}
}
]
Just replacing "id": "groups" with "id": "groups.[&1]" is enough, btw no need to repeat every element individually, just replace them with proper substitution by using ampersand operator prepended to the integer values which represent the level that provide to the target key (e.g.number of } operators while arriving the related key). So, convert "id": "groups.[&1]" to "id": "&2.[&1]", and use "*": "&", for the other elements such as
[
{
"operation": "shift",
"spec": {
"*": "&",
"groups": {
"*": {
"id": "&2.[&1]"
}
}
}
}
]
I'm looking for a way to copy or move a key from nested object to the top level
Input:
{
"id": "123",
"name": "foo",
"details": {
"orderNumber": "456789",
"addr": "N st 124",
"date": "2021-01-01"
}
}
desired output:
{
"id": "123",
"name": "foo",
"orderNumber": "456789",
"details": {
"orderNumber": "456789",
"addr": "N st 124",
"date": "2021-01-01"
}
}
or ideally
{
"id": "123",
"name": "foo",
"orderNumber": "456789",
"details": {
"addr": "N st 124",
"date": "2021-01-01"
}
}
the closest I could get is below transformation, but it converts object to value array
[
{
"operation": "shift",
"spec": {
"id": "id",
"name": "name",
"details": {
"orderNumber": "orderNumber",
"*": "details"
}
}
}
]
You're so close to the result, just a slight change(adding an ampersand) is needed such as
[
{
"operation": "shift",
"spec": {
"id": "id",
"name": "name",
"details": {
"orderNumber": "orderNumber",
"*": "&1.&"
}
}
}
]
in this case the keys keeps on appearing.
I'm trying to convert below input json to flatten necessary column names and its values while retaining all metadata.
Below is the input json that I've for my CDC use-case.
{
"type": "update",
"timestamp": 1558346256000,
"binlog_filename": "mysql-bin-changelog.000889",
"binlog_position": 635,
"database": "books",
"table_name": "publishers",
"table_id": 111,
"columns": [
{
"id": 1,
"name": "id",
"column_type": 4,
"last_value": 2,
"value": 2
},
{
"id": 2,
"name": "name",
"column_type": 12,
"last_value": "Suresh",
"value": "Suresh123"
},
{
"id": 3,
"name": "email",
"column_type": 12,
"last_value": "Suresh#yahoo.com",
"value": "Suresh#yahoo.com"
}
]
}
Below is the expected output json
[
{
"type": "update",
"timestamp": 1558346256000,
"binlog_filename": "mysql-bin-changelog.000889",
"binlog_position": 635,
"database": "books",
"table_name": "publishers",
"table_id": 111,
"columns": {
"id": "2",
"name": "Suresh123",
"email": "Suresh#yahoo.com"
}
}
]
I tried the below spec from which I'm able to retrieve columns object but not the rest of the metadata.
[
{
"operation": "shift",
"spec": {
"columns": {
"*": {
"#(value)": "[#1].#(1,name)"
}
}
}
}
]
Any leads would be very much appreciated.
I got the JOLT spec for above transformation. I'm posting it here incase if anyone stumbles upon the something like this.
[
{
"operation": "shift",
"spec": {
"columns": {
"*": {
"#(value)": "columns.#(1,name)"
}
},
"*": "&"
}
}
]
I am trying to use the jolt JSON to JSON transformation in Apache NiFi. I want to transform one JSON into another format.
Here is my original JSON:
{
"total_rows": 5884,
"offset": 0,
"rows": [
{
"id": "03888c0ab40c32451a018be6b409eba3",
"key": "03888c0ab40c32451a018be6b409eba3",
"value": {
"rev": "1-d5cc089dd8682422962ccab4f24bd21b"
},
"doc": {
"_id": "03888c0ab40c32451a018be6b409eba3",
"_rev": "1-d5cc089dd8682422962ccab4f24bd21b",
"topic": "iot-2/type/home-iot/id/1234/evt/temp/fmt/json",
"payload": {
"temperature": 36
},
"deviceId": "1234",
"deviceType": "home-iot",
"eventType": "temp",
"format": "json"
}
},
{
"id": "03888c0ab40c32451a018be6b409f163",
"key": "03888c0ab40c32451a018be6b409f163",
"value": {
"rev": "1-dee82cbb1b5ffa8a5e974135eb6340c5"
},
"doc": {
"_id": "03888c0ab40c32451a018be6b409f163",
"_rev": "1-dee82cbb1b5ffa8a5e974135eb6340c5",
"topic": "iot-2/type/home-iot/id/1234/evt/temp/fmt/json",
"payload": {
"temperature": 22
},
"deviceId": "1234",
"deviceType": "home-iot",
"eventType": "temp",
"format": "json"
}
}
]
}
I want this to be transformed in the following JSON:
[
{
"temperature":36,
"deviceId":"1234",
"deviceType":"home-iot",
"eventType":"temp"
},
{
"temperature":22,
"deviceId":"1234",
"deviceType":"home-iot",
"eventType":"temp"
}
]
This is what my spec looks like:
[
{
"operation": "shift",
"spec": {
"rows": {
"*": {
"doc": {
"deviceId": "[&1].deviceId",
"deviceType": "[&1].deviceType",
"eventType": "[&1].eventType",
"payload": {
"*": "[&1]"
}
}
}
}
}
}
]
I keep getting a null response. I am new to this and the documentation is not very easy to comprehend. Can somebody please help?
Because you are "down" one more level after the array index, by the time you get to deviceId you are 2 levels away from the index. Replace all the &1s with &2 except for payload. In that case you are another level "down" so you'll want to use &3 for the index. You also need to take whatever is matched by the * (temperature, e.g.) and set the outgoing field name to the same thing, by using & after the array index. Here's the resulting spec:
[
{
"operation": "shift",
"spec": {
"rows": {
"*": {
"doc": {
"deviceId": "[&2].deviceId",
"deviceType": "[&2].deviceType",
"eventType": "[&2].eventType",
"payload": {
"*": "[&3].&"
}
}
}
}
}
}
]
I am quite new to JOLT and I need to transform my JSON files to the desired schema. This is my input
[
{
"PK": 12345,
"FULL_NAME":"Amit Prakash",
"BIRTHDATE":"1987-05-25",
"SEX":"M",
"EMAIL": "amprak#mail.com",
"PHONE": "809386731",
"TS":"2015-11-19 14:36:34.0"
},
{
"PK": 12654,
"FULL_NAME": "Rohit Dhand",
"BIRTHDATE":"1979-02-01",
"SEX":"M",
"EMAIL": "rodha#mail.com",
"PHONE": "937013861",
"TS":"2015-11-20 11:03:02.6"
},
...
]
and this is my desired output:
{
"records": [
{
"attribs": [{
"type": "customer",
"reference": "CUST"
}],
"name": "Amit Prakash",
"personal_email": "amprak#mail.com",
"mobile": "809386731",
"id": 12345
},
{
"attribs": [{
"type": "customer",
"reference": "CUST"
}],
"name": "Rohit Dhand",
"personal_email": "rodha#mail.com",
"mobile": "937013861",
"id": 12654
},
...
]
}
So far, I have only managed up to this point:
[
{
"operation": "remove",
"spec": {
"*": {
"BIRTHDATE": "",
"SEX": "",
"TS": ""
}
}
},
{
"operation": "shift",
"spec": {
"*": "records"
}
}
]
But I can't go on from here. I don't know how to rename keys in the output.
Also, what's the alternative to remove operation? remove operation is good if you have fewer keys to exclude than to include, but how about the reverse (few keys to include, more than to exclude within a JSON object)?
Spec
[
{
"operation": "shift",
"spec": {
"*": {
"PK": "records[&1].id",
"PHONE": "records[&1].mobile",
"EMAIL": "records[&1].personal_email",
"FULL_NAME": "records[&1].name"
}
}
},
{
"operation": "default",
"spec": {
"records[]": {
"*": {
"attribs[]": {
"0": {
"type": "customer",
"reference": "CUST"
}
}
}
}
}
}
]
Shift makes copy of your data, whereas the other operations do not. Thus, one way to remove stuff is to just not have it copied across in your shift spec.
Generally the remove is used to get rid of things that would "mess up" the shift.