I'm using Apache NiFi and I get some JSON as input. I want to create a new JSON, as follows:
{
"data": ORIGINAL_JSON_HERE,
"new_field_1": "field_1_value",
"new_field_2": "field_2_value"
}
Where field1,field2 does not depend on the JSON, thus the values are not relevant.
In the following demo-site (made by the original creator of the Java Jolt library):
http://jolt-demo.appspot.com/#inception
If I do the following spec:
[
{
"operation": "shift",
"spec": {
"*": "&"
}
}
]
However, when I use the same spec in NiFi JOLT processors, I get null as output...
How can I find the correct spec for NiFi?
The problem was that I wanted the output JSON to have a data field, but I didn't specifiy an operation to create such field. The following works:
[
{
"operation": "shift",
"spec": {
"*": "data.&"
}
},
{
"operation": "default",
"spec": {
"data": {},
}
}
]
Related
I need convert the date in NiFi:
2022-11-22 00:00:00
To:
2022-11-22T00:00:00.000Z (ISO 8601)
can someone help me convert this date?
You can use the following spec within a JoltTransformJSON processor presuming that you need a conversion for the current value of an attribute (namely dt nested within a simple JSON object),
2022-11-22 00:00:00 extracted from the variable $now
[
{
// reformat by adding the milliseconds option
"operation": "default",
"spec": {
"dt": "${now():format('yyyy-MM-ddHH:mm:ss.SSS')}"
}
},
{
// split and recombine the pieces of the attribute's value
"operation": "modify-overwrite-beta",
"spec": {
"date": "=substring(#(1,dt),0,10)",
"time": "=substring(#(1,dt),10,22)",
"dt":"=concat(#(1,date),'T',#(1,time),'Z')"
}
},
{
// pick only theoriginal tag name
"operation": "shift",
"spec": {
"dt": "&"
}
}
]
If you (or can) have this info as an attribute, you can use NiFi's expression language to transform it.
For instance, with something like this: ${my_attribute:toDate("yyyy-MM-dd HH:mm:ss", "Europe/Paris"):format("yyyy-MM-dd'T'HH:mm:ss'Z'")}
Input:
{
"Remarks":"COMMENTS:(87) Test Comments"
}
Expecting Output is below:
{
"Remarks" : "C(87): TestComments"
"Id" : 87
}
I want to replace COMMENTS:(87) string with C(87): and need to get 87 in brackets and print the same in separate attribute "Id".
Can anyone help on this ?
You can use split function along with modify transformation in order to split the string by opening and closing parentheses, and then combine the desired substrings by using concat function. At the last step get rid of the auxiliary elements through use of remove transformation spec such as
[
{
"operation": "modify-overwrite-beta",
"spec": {
"str0": "=split('\\(', #(1,Remarks))",
"str1": "=split('\\)', #(1,str0))",
"str2": "#(1,str1[1])",
"Remarks": "=concat('C(',#(1,str2[0]),'):',#(1,str2[1]))",
"Id": "#(1,str2[0])"
}
},
{
"operation": "remove",
"spec": {
"str*": ""
}
}
]
I had a problem where I had json that looked like this--where the "key" in the json is variable:
{
"rpt":{
"tests": [
{"kerryA":[1,2,3,4]},
{"wallB":[1,2]},
{"MaryA":[1,2,3]}
]
}
}
I wanted it to look like this:
{
"rpt": {
"tests": [
{
"id":"kerryA",
"testTaken":[1,2,3,4]
},
{
"id":"wallB",
"testTaken":[1,2]
},
{
"name":"MaryA",
"testTaken":[1,2,3]
}
]
}
}
I looked at JsonSurfer and Jolt but they didn't seem to support the transform I was looking for. I solved it by rewriting the ObjectNode by recursing through the json with Jackson. While this wasn't bad, I feel like the json to json tranasformation that I looked at did not support the variable "key" well... Other than rewriting and recursing like I did, is there a more elegant transformation solution?
You can achieve it using JOLT with shift operation:
The spec would be:
[{
"operation": "shift",
"spec": {
"rpt": {
"tests" : {
"*": {
"*": {
"$": "rpt.tests[&2].id",
"#": "rpt.tests[&2].testTaken"
}
}
}
}
}
}
]
Take into account that Shiftr treats JSON arrays in the input data as Maps with numeric keys. That is why the double "*" and why rpt.tests[&2].id refers to the element of the array.
You can get more information about the shift operation at: https://github.com/bazaarvoice/jolt/blob/master/jolt-core/src/main/java/com/bazaarvoice/jolt/Shiftr.java
An example to process it in Java would be:
Object inputJSON = JsonUtils.classpathToObject("/input.json");
List<Object> chainrSpecJSON = JsonUtils.classpathToList("/spec.json");
Chainr chainr = Chainr.fromSpec(chainrSpecJSON);
Object transformedOutput = chainr.transform(inputJSON);
System.out.println(JsonUtils.toPrettyJsonString(transformedOutput));
Where input.json is a file in the classpath with your entry data and spec.json is a file with the before code.
I try to convert JSON to JSON-LD and was wondering if I could use JSON-LD expansion algorithm for creating my converter. Then I could just specify my schema as a context and run the expansion algorithm for doing the conversion. Problem is that I cannot figure out how to define new value objects in context so that the expansion algorithm would work.
Let's say I have this:
{
"timestamp": "2016-01-08T11:01:38Z"
}
and I want to get this:
{
"prefix:time": {"prefix:start": "2016-01-08T11:01:38Z"}
}
I have tried it using the JSON-LD playground with something like this:
{
"#context": {
"timestamp": {
"#id": "prefix:time",
"#value": {"prefix:start": "#value"}
}
},
"timestamp": "2016-01-08T11:01:38Z"
}
But the expanded result looks like this:
[
{
"prefix:time": [
{
"#value": "2016-01-08T11:01:38Z"
}
]
}
]
Is there any way to use the JSON-LD expansion (or other) algorithm to replace the value with a new JSON object?
This Jolt transform does the transform you described.
[
{
"operation": "shift",
"spec": {
"timestamp": "prefix:time.prefix:start"
}
}
]
You can try it out at http://jolt-demo.appspot.com/
Please help me in writing the jolt spec to get the expected output with special character "#" as shown below. When I place "#FirstName":"Names.FirstName" in the spec it is not working. Please help in resolving this.
JSON Input:
{
"FirstName": "First"
}
Jolt Spec:
[
{
"operation": "shift",
"spec": {
"FirstName": "Names.FirstName"
}
}
]
Expected Output:
{
"Names" : {
"#FirstName" : "First"
}
}
Updated : As of Jolt version 0.0.21, the "shift" transform now allows special characters to escaped.
This Spec will allow "#FirstName" to be a key in the output.
[
{
"operation": "shift",
"spec": {
"FirstName": "Names.\\#FirstName"
}
}
]