I need convert the date in NiFi:
2022-11-22 00:00:00
To:
2022-11-22T00:00:00.000Z (ISO 8601)
can someone help me convert this date?
You can use the following spec within a JoltTransformJSON processor presuming that you need a conversion for the current value of an attribute (namely dt nested within a simple JSON object),
2022-11-22 00:00:00 extracted from the variable $now
[
{
// reformat by adding the milliseconds option
"operation": "default",
"spec": {
"dt": "${now():format('yyyy-MM-ddHH:mm:ss.SSS')}"
}
},
{
// split and recombine the pieces of the attribute's value
"operation": "modify-overwrite-beta",
"spec": {
"date": "=substring(#(1,dt),0,10)",
"time": "=substring(#(1,dt),10,22)",
"dt":"=concat(#(1,date),'T',#(1,time),'Z')"
}
},
{
// pick only theoriginal tag name
"operation": "shift",
"spec": {
"dt": "&"
}
}
]
If you (or can) have this info as an attribute, you can use NiFi's expression language to transform it.
For instance, with something like this: ${my_attribute:toDate("yyyy-MM-dd HH:mm:ss", "Europe/Paris"):format("yyyy-MM-dd'T'HH:mm:ss'Z'")}
Related
Been at this for 5 hours trying to convert a JSON array of string values to a plain string comma separated value and trim the length via substring.
Can NiFi do this?
e.g.
Starting with
[
"Charlie was here",
"Linus was here",
"Snoopy was here",
"Sally was here"
]
I am trying to convert it to
Charlie was here,Linus was here,Snoopy was here,Sally was here
So if the above value gets stored into an attribute called 'myData'
then I can substring it to shorten the overall length and it does not matter what gets chopped off at the end.
e.g.
myData:substring(0,1024)
I have been trying to use the following processors, various combinations but have not been able to find the correct one to use.
UpdateAttribute
EvaluateJSONPath
SplitJSON
MergeContent
The closest I got is with the splitjson and mergecontent but then the content contains no comma separating the values and I end up with
Charlie was hereLinus was hereSnoopy was hereSally was here
Just about everything I have found posted in here deals with text convert to json but not json convert to text.
What processor am I missing here?
An option would be using JoltTransformJSON processor with the following specification :
[
{
"operation": "shift",
"spec": {
"*": "&1[]"
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"*": "=join(',',#(1,&))"// concatenate all string components separated by comma
}
},
{// derive the unnested string only
"operation": "shift",
"spec": {
"*": ""
}
}
]
the demo on the site http://jolt-demo.appspot.com/ is :
Input json :
{
"userid" : "31b25f023c58c969388991a6b9b4030000000000",
"username_id": "54fca0dd914ae593ef65988b4a3e93cccc590000000000"
}
There is 10 0's ahead of each value. The overall length of string can vary, but it will be appended by 10 0's always. I would like to remove it.
Thus expected output:
{
"userid" : "31b25f023c58c969388991a6b9b403",
"username_id": "54fca0dd914ae593ef65988b4a3e93cccc59"
}
Hence i am looking for the right way using jolt transform to perform this truncation.
You can start with a modify transformation in which the substring function is applied for each value of the attributes after determining their ten reduced length in order to extract the value except for the last ten characters, as it's mentioned ten zeroes are always right-padded, such as
[
{
"operation": "modify-overwrite-beta",
"spec": {
"userid_len": "=size(#(1,userid))",
"userid_len_dif": "=intSum(-10,#(1,userid_len))",
"userid": "=substring(#(1,&),0,#(1,userid_len_dif))",
"username_id_len": "=size(#(1,username_id))",
"username_id_len_dif": "=intSum(-10,#(1,username_id_len))",
"username_id": "=substring(#(1,&),0,#(1,username_id_len_dif))"
}
},
{
"operation": "shift",
"spec": {
"use*id": "&"
}
}
]
the demo on the site http://jolt-demo.appspot.com/ is
Input:
{
"Remarks":"COMMENTS:(87) Test Comments"
}
Expecting Output is below:
{
"Remarks" : "C(87): TestComments"
"Id" : 87
}
I want to replace COMMENTS:(87) string with C(87): and need to get 87 in brackets and print the same in separate attribute "Id".
Can anyone help on this ?
You can use split function along with modify transformation in order to split the string by opening and closing parentheses, and then combine the desired substrings by using concat function. At the last step get rid of the auxiliary elements through use of remove transformation spec such as
[
{
"operation": "modify-overwrite-beta",
"spec": {
"str0": "=split('\\(', #(1,Remarks))",
"str1": "=split('\\)', #(1,str0))",
"str2": "#(1,str1[1])",
"Remarks": "=concat('C(',#(1,str2[0]),'):',#(1,str2[1]))",
"Id": "#(1,str2[0])"
}
},
{
"operation": "remove",
"spec": {
"str*": ""
}
}
]
I'm using Apache NiFi and I get some JSON as input. I want to create a new JSON, as follows:
{
"data": ORIGINAL_JSON_HERE,
"new_field_1": "field_1_value",
"new_field_2": "field_2_value"
}
Where field1,field2 does not depend on the JSON, thus the values are not relevant.
In the following demo-site (made by the original creator of the Java Jolt library):
http://jolt-demo.appspot.com/#inception
If I do the following spec:
[
{
"operation": "shift",
"spec": {
"*": "&"
}
}
]
However, when I use the same spec in NiFi JOLT processors, I get null as output...
How can I find the correct spec for NiFi?
The problem was that I wanted the output JSON to have a data field, but I didn't specifiy an operation to create such field. The following works:
[
{
"operation": "shift",
"spec": {
"*": "data.&"
}
},
{
"operation": "default",
"spec": {
"data": {},
}
}
]
I'm a newbie on Apache Nifi and have the following Problem: I would like to transform a json file as follows:
From:
{
"Property1": "x1",
"Property2": "Tag_**2ABC**",
"Property3": "x3",
"Property4": "x4"
}
to:
{
"**2ABC**_Property1": "x1",
"**2ABC**_Property3": "x3",
"**2ABC**_Property4": "x4"
},
it means: taking the value from a certain Attribute to update all other attributes.
I could find examples using JoltTransformer-Processor that works well when the update is only adding a string. But not for my case
What I've done so far: I have set each Attribute using evaluateJSONPath processor. But I just tried quite a lot of possibilities to use the update Attribute processor to do it without success. All my possible tests looked like (within UpdateAttribute):
Property1 --> ${'Property2':substring(4,6)}"_"${'Property1'}
Using Jolt:
[
{"operation": "modify-overwrite-beta",
"spec": {
"Property1": "${'Property2':substring(4,6)}_${'Property1'}"
}
}
]
Which point am I missing here? Thanks in advance!
I don't know about Nifi, but here is how you can do it in Jolt.
Spec
[
{
"operation": "shift",
"spec": {
// match Property2
"Property2": {
"Tag_*": { // capture the nasty "**2ABC**" part to reference later
// go back up the tree to the root
"#2": {
// match and ignore Property2
"Property2": null,
//
// match Property* and use it and the captured
// "prefix" to create the output key
// &(2,1) references the Tag_*, and pull off the "**2ABC**" part
"Property*": "&(2,1)_&"
}
}
}
}
}
]