Please help me write the jolt spec to rename the key with a special character "#". When I place "#" in jolt spec, it is not working. I am getting the below error
Error running the Transform.
JOLT Chainr encountered an exception constructing Transform className:com.bazaarvoice.jolt.Shiftr at index:0.
Invalid key:Company#1 can not have an # other than at the front.
Please help in resolving this.
Input JSON
[
{
"C": "p",
"ID": 1,
"Company#1": "Tesla",
"Age": 30.2,
"Year": 1996,
"Time": "22/10/1996"
},
{
"C": "p",
"ID": 2,
"Company#1": "Facebook",
"Age": 40.5,
"Year": 2001,
"Time": "22/10/2001"
}
]
JOLT Spec
[
{
"operation": "shift",
"spec": {
"*": {
"Company#1": "[#2].CompanyTest",
"Age": "[#2].AgeTest",
"*": "[#2].&"
}
}
}
]
Expected Output
[ {
"C" : "p",
"ID" : 1,
"CompanyTest" : "Tesla",
"AgeTest" : 30.2,
"Year" : 1996,
"Time" : "22/10/1996"
}, {
"C" : "p",
"ID" : 2,
"CompanyTest" : "Facebook",
"AgeTest" : 40.5,
"Year" : 2001,
"Time" : "22/10/2001"
} ]
Need to escape by prepending double backslash such as Company\\#1 in order to prevent getting an error
Related
I have a output that I want to jolt transform but I'm having a difficult time doing so
Json Example:
{
"message": [
[
"2019",
"DATE"
],
[
"second",
"ORDINAL"
],
[
"Local",
"PERSON"
],
[
"2019",
"DATE"
],
[
"ISO",
"ORG"
],
[
"Ubuntu",
"PERSON"
]
]
}
I want to make the output look like this
{
"DATE": "2019",
"ORDINAL": "second",
"PERSON": "Local",
"DATE": "2019",
"ORG":"ISO",
"PERSON": "ubuntu"
}
The transformation should start with what displayed below but I get confused because of the list and the changing key and values. The original shows the keys as the capitalized values while the values are the lower cased values, they are backwards in the original.
{
"operation": "shift",
"spec": {
....
The current desired output is not a valid JSON as having duplicate DATE and PERSON keys. What you want might be
{
"DATE" : [ "2019", "2019" ],
"ORDINAL" : "second",
"PERSON" : [ "Local", "Ubuntu" ],
"ORG" : "ISO"
}
Then, you can use the following spec
[
{
"operation": "shift",
"spec": {
"*": {
"*": {
"0": "#(2,[&1][1])"
}
}
}
}
]
where we walk the indexes of the messages array through use of consecutive "*" wildcards, and that forms such a JSON
{
"0" : [ "2019", "DATE" ],
"1" : [ "second", "ORDINAL" ],
"2" : [ "Local", "PERSON" ],
"3" : [ "2019", "DATE" ],
"4" : [ "ISO", "ORG" ],
"5" : [ "Ubuntu", "PERSON" ]
}
and then "0" is used to get the second components, and "#(2,[&1][1])" is used to get the first components of each arrays from that JSON value.
My source data is a huge flatten CSV which contains data that I'd like to normalise and store in a number of separate database tables. I'm wondering if I can use a JOLT transformation to alter the structure of my CSV data which I've already a flat JSON object.
For example, the object below has two 'parent' rows and three repeating pairs of columns which are the 'child' rows:
{
"id": 1,
"name": "zzz",
"code_1": "abc",
"date_1": "2021-01-01",
"code_2": "def",
"date_2": "2021-01-02",
"code_3": "ghi",
"date_3": "2021-01-03"
}
Can I use JOLT to generate the output:
{
"id": 1,
"name": "zzz",
"codes": [
{
"code": "abc",
"date": "2021-01-01"
},
{
"code": "def",
"date": "2021-01-02"
},
{
"code": "ghi",
"date": "2021-01-03"
}
]
}
I've had a play around with the Jolt Playground, but as yet I've not found something which could achieve this goal.
Any hints, tips, or pointers welcome.
Thanks in advance.
UPDATE:
I'm getting closer, with the following spec and output. My codes and dates are in different objects however:
spec:
[
{
"operation": "shift",
"spec": {
"identifier": "id",
"name": "name",
"code*": "codes[].code",
"date*": "codes[].date"
}
}
]
output:
{
"id" : 1,
"name" : "zzz",
"codes" : [ {
"code" : "abc"
}, {
"date" : "2021-01-01"
}, {
"code" : "def"
}, {
"date" : "2021-01-02"
}, {
"code" : "ghi"
}, {
"date" : "2021-01-03"
} ]
}
(Edit: corrected the required output.)
You can use shift transformations twice as the elements are grouped under common numbered keys by _1,_2,_3 in the first step, and then remove keys of those objects while nesting them within codes list such as
[
{
"operation": "shift",
"spec": {
"*": "&",
"code*": { "#": "&(0,1).code" },
"date*": { "#": "&(0,1).date" }
}
},
{
"operation": "shift",
"spec": {
"_*": "codes[]",
"*": "&"
}
}
]
I have been spending two days trying to do the following, without success.
I have this example JSON-LD document:
{
"#context": [
"https://www.w3.org/2018/credentials/v1",
"https://w3c-ccg.github.io/ldp-bbs2020/context/v1",
"https://mycontext.com/credentials/v1"
],
"type": ["VerifiableCredential"],
"issuer": "did:example:issuer",
"issuanceDate": "2020-01-01T00:00:00Z",
"expirationDate": "2029-12-31T23:59:59Z",
"credentialSubject": {
"id": "did:example:subject",
"type": ["ExampleCredential"],
"values": [
{
"a": 1,
"b": 2,
"c": 3
},
{
"a": 4,
"b": 5,
"c": 6
}
]
}
}
and the following JSON-LD context (hypothetically retrievable at https://mycontext.com/credentials/v1):
{
"#context": {
"#version": 1.1,
"#protected": true,
"schema": "https://schema.org/",
"ExampleCredential": {
"#id": "https://mycontext.com/credentials#ExampleCredential",
"#context": {
"#version": 1.1,
"#protected": true,
"type": "#type",
"values": "#nest",
"a": {
"#id": "https://mycontext.com/credentials#a",
"#type": "schema:example_a",
"#nest": "values"
},
"b": {
"#id": "https://mycontext.com/credentials#b",
"#type": "schema:example_b",
"#nest": "values"
},
"c": {
"#id": "https://mycontext.com/credentials#c",
"#type": "schema:example_c",
"#nest": "values"
}
}
}
}
}
Furthermore, I am trying to use the following JSON-LD frame to filter only the element in the credential that has the field a equal to 1:
{
"#context": [
"https://www.w3.org/2018/credentials/v1",
"https://w3c-ccg.github.io/ldp-bbs2020/context/v1",
"https://mycontext.com/credentials/v1"
],
"#explicit": true,
"type": ["VerifiableCredential"],
"issuer": "did:example:issuer",
"issuanceDate": {},
"expirationDate": {},
"credentialSubject": {
"id": "did:example:subject",
"#explicit": true,
"type": ["ExampleCredential"],
"values": [
{
"a": 1,
"b": {},
"c": {}
}
]
}
}
I am using the jsonld package to perform these operations, and the final result that I would like to get corresponds to the initial document, but with the values field filtered to only contain the first object in the list. This is just the last of the attempts, and I have tried everything from "#container": "#set" to #nest, but clearly I am missing something. What is the right way to format the context and/or the frame so that I can achieve the expected result? Aka, what is the right way to express a relationship in a context in which the child is a list of objects of a specific type?
Thanks!
Input JSON :
{
"type": "mbrInfo",
"csId": 123456789,
"insTS": "14-07-201911:55",
"seqId": 1234565,
"title": "Mr",
"fName": "Amit",
"mName": "",
"lName": "V",
"suffix": "Engg",
"lvlId": "P",
"lvlType": "LAC",
"acctStatus": "20",
"enrlDT": "2016-08-29",
"vrsnId": 1
}
Expected Output JSON:
{
"type": "mbrInfo",
"csId": 123456789,
"insTS": "14-07-201911:55",
"seqId": 1234565,
"name" : [{
"title": "Mr",
"fName": "Amit",
"mName": "",
"lName": "V",
"suffix": "Engg"}],
"lvlId": "P",
"lvlType": "LAC",
"acctStatus": "20",
"enrlDT": "2016-08-29",
"vrsnId": 1
}.
Currently I am using JOLTtransformJSON processor with JOLT Spec as :
[
{
"operation": "shift",
"spec": {
"name": {
"$": "[#1]",
"#.title": "[#1].title",
"#.fName": "[#1].fName",
"#.mName": "[#1].mName",
"#.lName": "[#1].lName",
"#.suffix": "[#1].suffix"
}
}
}
]
But all I am getting is either NULL or the original JSON (with diff spec) as output.
Thanks in advance.
Is the intent to put all the name fields into a 1-element array containing an object. This JOLT spec puts them into an object at the name field:
[
{
"operation": "shift",
"spec": {
"title": "name.title",
"fName": "name.fName",
"mName": "name.mName",
"lName": "name.lName",
"suffix": "name.suffix",
"*": "&"
}
}
]
...and this spec puts them into a 1-element array at the name field:
[
{
"operation": "shift",
"spec": {
"title": "name[0].title",
"fName": "name[0].fName",
"mName": "name[0].mName",
"lName": "name[0].lName",
"suffix": "name[0].suffix",
"*": "&"
}
}
]
I don't see any other place in the input to get an index into the array, so I just used 0.
I am using Apache NiFi and receive from an embedded micro over a TCP/IP socket a JSON file of the form:
{
"id": 123456,
"ip": "192.168.1.1",
"t": -12.9,
"T": -23.8,
"variables": [
"user1",
0,
-12.97,
23.87
]
}
and would like to transform it such that the keys of the variables are added as they are known to me as follows:
{
"id": 123456,
"ip": "192.168.1.1",
"t": -12.9,
"T": -23.8,
"variables": [
"username" : "user1",
"valid" : 0,
"temperature 1" : -12.97,
"temperature 2" : 23.87
]
}
and then to be able to access a key value pair such as variables.username.
I have tried using JoltTransformJSON but don't know how to write the spec correctly if it can do it!! I have written the following jolt spec:
[
{
"operation": "shift",
"spec": {
"id": "id",
"ip": "ip",
"t": "t",
"T": "T",
"variables": {
"username": "",
"valid": "",
"temperature 1": "",
"temperature 2": ""
}
}
}
]
I also have tried using UpdateRecord with Record Reader/Writer but passing the correct schema causes an error on the first variable as it can't match "user1" with a key from the initial JSON.
I have managed to work out the Jolt Spec as follows:
[
{
"operation": "shift",
"spec": {
"id": "id",
"ip": "ip",
"t": "t",
"T": "T",
"variables": {
"0": "username",
"1": "valid",
"2": "temperature 1",
"3": "temperature 2"
}
}
}
]
Thank you daggett for your contribution.