Update Json-Attributes in Apache-Nifi: Jolt - json

I'm a newbie on Apache Nifi and have the following Problem: I would like to transform a json file as follows:
From:
{
"Property1": "x1",
"Property2": "Tag_**2ABC**",
"Property3": "x3",
"Property4": "x4"
}
to:
{
"**2ABC**_Property1": "x1",
"**2ABC**_Property3": "x3",
"**2ABC**_Property4": "x4"
},
it means: taking the value from a certain Attribute to update all other attributes.
I could find examples using JoltTransformer-Processor that works well when the update is only adding a string. But not for my case
What I've done so far: I have set each Attribute using evaluateJSONPath processor. But I just tried quite a lot of possibilities to use the update Attribute processor to do it without success. All my possible tests looked like (within UpdateAttribute):
Property1 --> ${'Property2':substring(4,6)}"_"${'Property1'}
Using Jolt:
[
{"operation": "modify-overwrite-beta",
"spec": {
"Property1": "${'Property2':substring(4,6)}_${'Property1'}"
}
}
]
Which point am I missing here? Thanks in advance!

I don't know about Nifi, but here is how you can do it in Jolt.
Spec
[
{
"operation": "shift",
"spec": {
// match Property2
"Property2": {
"Tag_*": { // capture the nasty "**2ABC**" part to reference later
// go back up the tree to the root
"#2": {
// match and ignore Property2
"Property2": null,
//
// match Property* and use it and the captured
// "prefix" to create the output key
// &(2,1) references the Tag_*, and pull off the "**2ABC**" part
"Property*": "&(2,1)_&"
}
}
}
}
}
]

Related

Rename fields in JOLT transformation

I want to change the name of fields in an array by Jolt. Please explain how it works with your answer.
Input:
{
"referenceName": "***",
"name": "***"
}
Desired output:
{
"ReferenceName": "***",
"ColumnName": "***"
}
You can use this simple spec:
[
{
"operation": "shift",
"spec": {
"referenceName": "ReferenceName",
"name": "ColumnName"
}
}
]
Or you can have simpler:
[
{
"operation": "shift",
"spec": {
"r*": "R&(0,1)",
"n*": "ColumnN&(0,1)"
}
}
]
When you want to change some names in your JSON, You can use the shift operation like the above spec:
r*: match all keys that started with the r like referenceName and * match everything after r that equals eferenceName. You can write R&(0,1) that &(0,1) value is equal to first *.
n*: match all keys that started with the n like name and * match everything after n that is equal to ame. You can write ColumnN&(0,1) that &(0,1) value is equal to first *.
Note: & means the key's value, But when you use the &(0,1), you are saying that in the 0 level or current level please get the first * value. Suppose you had a second * in this level, You should get it like this: &(0,2)

How do we traverse a jolt spec?

this is a simple question,I am new to nifi and jolt. I just wanted to know,how to travserse the jolt spec while using wild card characters.For example,this is an example in jolt demo site,
input is
{
"data": {
"1234": {
"clientId": "12",
"hidden": true
},
"1235": {
"clientId": "35",
"hidden": false
}
}
}
Spec is
[
{
"operation": "shift",
"spec": {
"data": {
"*": {
"hidden": {
"true": {
// if hidden is true, then write the value disabled to the RHS output path
// Also #(3,clientId) means lookup the tree 3 levels, to the "1234" or "1235" level,
// and then come back down down the tree and grabe the value of "clientId"
"#disabled": "clients.#(3,clientId)"
},
"false": {
"#enabled": "clients.#(3,clientId)"
}
}
}
}
}
}
]
and output is
{
"clients" : {
"12" : "disabled",
"35" : "enabled"
}
}
How did we get the above output? like what #(3,clientsid).As far as I understand, it goes 3 levels up.But 3 levels with respect to what..the spec or the input? Either way,how to move 3 levels up,can you please define which are the levels here?
Thanks in advance
Just count the number of each opening curly-braces({) or colons(:) in the backward direction. Whenever they're not independent such as :{, then count this as only one in order to reach to the target key which is "*" wildcard just under "data" key in this case, and for #(3,clientId); first level is crossing the colon(:) next to "#disabled" or "#enabled", second level is crossing opening curly-braces next to those boolean keys for each, and then the third level is crossing opening curly-brace just after "hidden" key to reach the indexes the object with the "data" key.

Support for concat in JSON Jolt in Apache NiFi

Using Apache NiFi I want to add a new field to all the elements within a JSON flow file based on the concatenation of two other fields. I am trying to use the JoltTransformJSON processor for this, however, the Jolt transform I want to use works OK using online tools but does not work within NiFi. I suspect a version issue but there may be something stupid with my Jolt specification.
The input JSON looks like this...
[
{
"id": 485842,
"cc": 1,
"x": 0,
"y": null
},
{
"id": 281733,
"cc": 1,
"x": 0,
"y": 10
},
{
"id": 721412,
"cc": 12,
"x": 0,
"y": null
}
]
The desired output I want is this...
[ {
"id" : 485842,
"cc" : 1,
"x" : 0,
"y" : null,
"id_cc" : "485842_1"
}, {
"id" : 281733,
"cc" : 1,
"x" : 0,
"y" : 10,
"id_cc" : "281733_1"
}, {
"id" : 721412,
"cc" : 12,
"x" : 0,
"y" : null,
"id_cc" : "721412_12"
} ]
And the Jolt transform I use on the online site is...
[{
"operation": "modify-default-beta",
"spec": {
"*": {
"id_cc": "=concat(#(1,id),'_',#(1,cc))"
}
}
}]
In NiFi, I configure the JoltTransformJSON processor to have Modify-Default and I use this slightly modified Jolt Specification...
{
"operation": "modify-default",
"spec": {
"*": {
"id_cc": "=concat(#(1,id),'_',#(1,cc))"
}
}
}
NiFi validates this OK and the process runs. The output JSON consists of a single record only and a new field is added like this
"operation": "modify-default"
Is there a quick fix to the modify-default operation that will resolve this or is there an even easier way using a shift operation?
Thanks in advance for any pointers.
After some help from colleagues, I have found a way to make this work.
Firstly, set the Jolt Transformation DSL property of the JoltTransformJSON processor to be Chain.
Secondly, set the Jolt specification to the following...
[{
"operation": "modify-default-beta",
"spec": {
"*": {
"id_cc": "=concat(#(1,id),'_',#(1,cc))"
}
}
}]
The [] are vital as is the beta in the operation.
Thirdly, ensure the JSON fed into the Jolt processor is an array.
Get all this correct and the expected output will be produced.

Json to Json transforms using json surfer or Jolt with varying attribute name

I had a problem where I had json that looked like this--where the "key" in the json is variable:
{
"rpt":{
"tests": [
{"kerryA":[1,2,3,4]},
{"wallB":[1,2]},
{"MaryA":[1,2,3]}
]
}
}
I wanted it to look like this:
{
"rpt": {
"tests": [
{
"id":"kerryA",
"testTaken":[1,2,3,4]
},
{
"id":"wallB",
"testTaken":[1,2]
},
{
"name":"MaryA",
"testTaken":[1,2,3]
}
]
}
}
I looked at JsonSurfer and Jolt but they didn't seem to support the transform I was looking for. I solved it by rewriting the ObjectNode by recursing through the json with Jackson. While this wasn't bad, I feel like the json to json tranasformation that I looked at did not support the variable "key" well... Other than rewriting and recursing like I did, is there a more elegant transformation solution?
You can achieve it using JOLT with shift operation:
The spec would be:
[{
"operation": "shift",
"spec": {
"rpt": {
"tests" : {
"*": {
"*": {
"$": "rpt.tests[&2].id",
"#": "rpt.tests[&2].testTaken"
}
}
}
}
}
}
]
Take into account that Shiftr treats JSON arrays in the input data as Maps with numeric keys. That is why the double "*" and why rpt.tests[&2].id refers to the element of the array.
You can get more information about the shift operation at: https://github.com/bazaarvoice/jolt/blob/master/jolt-core/src/main/java/com/bazaarvoice/jolt/Shiftr.java
An example to process it in Java would be:
Object inputJSON = JsonUtils.classpathToObject("/input.json");
List<Object> chainrSpecJSON = JsonUtils.classpathToList("/spec.json");
Chainr chainr = Chainr.fromSpec(chainrSpecJSON);
Object transformedOutput = chainr.transform(inputJSON);
System.out.println(JsonUtils.toPrettyJsonString(transformedOutput));
Where input.json is a file in the classpath with your entry data and spec.json is a file with the before code.

Setting value as object in JSON-LD expansion

I try to convert JSON to JSON-LD and was wondering if I could use JSON-LD expansion algorithm for creating my converter. Then I could just specify my schema as a context and run the expansion algorithm for doing the conversion. Problem is that I cannot figure out how to define new value objects in context so that the expansion algorithm would work.
Let's say I have this:
{
"timestamp": "2016-01-08T11:01:38Z"
}
and I want to get this:
{
"prefix:time": {"prefix:start": "2016-01-08T11:01:38Z"}
}
I have tried it using the JSON-LD playground with something like this:
{
"#context": {
"timestamp": {
"#id": "prefix:time",
"#value": {"prefix:start": "#value"}
}
},
"timestamp": "2016-01-08T11:01:38Z"
}
But the expanded result looks like this:
[
{
"prefix:time": [
{
"#value": "2016-01-08T11:01:38Z"
}
]
}
]
Is there any way to use the JSON-LD expansion (or other) algorithm to replace the value with a new JSON object?
This Jolt transform does the transform you described.
[
{
"operation": "shift",
"spec": {
"timestamp": "prefix:time.prefix:start"
}
}
]
You can try it out at http://jolt-demo.appspot.com/