Input json :
{
"userid" : "31b25f023c58c969388991a6b9b4030000000000",
"username_id": "54fca0dd914ae593ef65988b4a3e93cccc590000000000"
}
There is 10 0's ahead of each value. The overall length of string can vary, but it will be appended by 10 0's always. I would like to remove it.
Thus expected output:
{
"userid" : "31b25f023c58c969388991a6b9b403",
"username_id": "54fca0dd914ae593ef65988b4a3e93cccc59"
}
Hence i am looking for the right way using jolt transform to perform this truncation.
You can start with a modify transformation in which the substring function is applied for each value of the attributes after determining their ten reduced length in order to extract the value except for the last ten characters, as it's mentioned ten zeroes are always right-padded, such as
[
{
"operation": "modify-overwrite-beta",
"spec": {
"userid_len": "=size(#(1,userid))",
"userid_len_dif": "=intSum(-10,#(1,userid_len))",
"userid": "=substring(#(1,&),0,#(1,userid_len_dif))",
"username_id_len": "=size(#(1,username_id))",
"username_id_len_dif": "=intSum(-10,#(1,username_id_len))",
"username_id": "=substring(#(1,&),0,#(1,username_id_len_dif))"
}
},
{
"operation": "shift",
"spec": {
"use*id": "&"
}
}
]
the demo on the site http://jolt-demo.appspot.com/ is
Related
I want to change the name of fields in an array by Jolt. Please explain how it works with your answer.
Input:
{
"referenceName": "***",
"name": "***"
}
Desired output:
{
"ReferenceName": "***",
"ColumnName": "***"
}
You can use this simple spec:
[
{
"operation": "shift",
"spec": {
"referenceName": "ReferenceName",
"name": "ColumnName"
}
}
]
Or you can have simpler:
[
{
"operation": "shift",
"spec": {
"r*": "R&(0,1)",
"n*": "ColumnN&(0,1)"
}
}
]
When you want to change some names in your JSON, You can use the shift operation like the above spec:
r*: match all keys that started with the r like referenceName and * match everything after r that equals eferenceName. You can write R&(0,1) that &(0,1) value is equal to first *.
n*: match all keys that started with the n like name and * match everything after n that is equal to ame. You can write ColumnN&(0,1) that &(0,1) value is equal to first *.
Note: & means the key's value, But when you use the &(0,1), you are saying that in the 0 level or current level please get the first * value. Suppose you had a second * in this level, You should get it like this: &(0,2)
Been at this for 5 hours trying to convert a JSON array of string values to a plain string comma separated value and trim the length via substring.
Can NiFi do this?
e.g.
Starting with
[
"Charlie was here",
"Linus was here",
"Snoopy was here",
"Sally was here"
]
I am trying to convert it to
Charlie was here,Linus was here,Snoopy was here,Sally was here
So if the above value gets stored into an attribute called 'myData'
then I can substring it to shorten the overall length and it does not matter what gets chopped off at the end.
e.g.
myData:substring(0,1024)
I have been trying to use the following processors, various combinations but have not been able to find the correct one to use.
UpdateAttribute
EvaluateJSONPath
SplitJSON
MergeContent
The closest I got is with the splitjson and mergecontent but then the content contains no comma separating the values and I end up with
Charlie was hereLinus was hereSnoopy was hereSally was here
Just about everything I have found posted in here deals with text convert to json but not json convert to text.
What processor am I missing here?
An option would be using JoltTransformJSON processor with the following specification :
[
{
"operation": "shift",
"spec": {
"*": "&1[]"
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"*": "=join(',',#(1,&))"// concatenate all string components separated by comma
}
},
{// derive the unnested string only
"operation": "shift",
"spec": {
"*": ""
}
}
]
the demo on the site http://jolt-demo.appspot.com/ is :
I need convert the date in NiFi:
2022-11-22 00:00:00
To:
2022-11-22T00:00:00.000Z (ISO 8601)
can someone help me convert this date?
You can use the following spec within a JoltTransformJSON processor presuming that you need a conversion for the current value of an attribute (namely dt nested within a simple JSON object),
2022-11-22 00:00:00 extracted from the variable $now
[
{
// reformat by adding the milliseconds option
"operation": "default",
"spec": {
"dt": "${now():format('yyyy-MM-ddHH:mm:ss.SSS')}"
}
},
{
// split and recombine the pieces of the attribute's value
"operation": "modify-overwrite-beta",
"spec": {
"date": "=substring(#(1,dt),0,10)",
"time": "=substring(#(1,dt),10,22)",
"dt":"=concat(#(1,date),'T',#(1,time),'Z')"
}
},
{
// pick only theoriginal tag name
"operation": "shift",
"spec": {
"dt": "&"
}
}
]
If you (or can) have this info as an attribute, you can use NiFi's expression language to transform it.
For instance, with something like this: ${my_attribute:toDate("yyyy-MM-dd HH:mm:ss", "Europe/Paris"):format("yyyy-MM-dd'T'HH:mm:ss'Z'")}
Input:
{
"Remarks":"COMMENTS:(87) Test Comments"
}
Expecting Output is below:
{
"Remarks" : "C(87): TestComments"
"Id" : 87
}
I want to replace COMMENTS:(87) string with C(87): and need to get 87 in brackets and print the same in separate attribute "Id".
Can anyone help on this ?
You can use split function along with modify transformation in order to split the string by opening and closing parentheses, and then combine the desired substrings by using concat function. At the last step get rid of the auxiliary elements through use of remove transformation spec such as
[
{
"operation": "modify-overwrite-beta",
"spec": {
"str0": "=split('\\(', #(1,Remarks))",
"str1": "=split('\\)', #(1,str0))",
"str2": "#(1,str1[1])",
"Remarks": "=concat('C(',#(1,str2[0]),'):',#(1,str2[1]))",
"Id": "#(1,str2[0])"
}
},
{
"operation": "remove",
"spec": {
"str*": ""
}
}
]
I'm a newbie on Apache Nifi and have the following Problem: I would like to transform a json file as follows:
From:
{
"Property1": "x1",
"Property2": "Tag_**2ABC**",
"Property3": "x3",
"Property4": "x4"
}
to:
{
"**2ABC**_Property1": "x1",
"**2ABC**_Property3": "x3",
"**2ABC**_Property4": "x4"
},
it means: taking the value from a certain Attribute to update all other attributes.
I could find examples using JoltTransformer-Processor that works well when the update is only adding a string. But not for my case
What I've done so far: I have set each Attribute using evaluateJSONPath processor. But I just tried quite a lot of possibilities to use the update Attribute processor to do it without success. All my possible tests looked like (within UpdateAttribute):
Property1 --> ${'Property2':substring(4,6)}"_"${'Property1'}
Using Jolt:
[
{"operation": "modify-overwrite-beta",
"spec": {
"Property1": "${'Property2':substring(4,6)}_${'Property1'}"
}
}
]
Which point am I missing here? Thanks in advance!
I don't know about Nifi, but here is how you can do it in Jolt.
Spec
[
{
"operation": "shift",
"spec": {
// match Property2
"Property2": {
"Tag_*": { // capture the nasty "**2ABC**" part to reference later
// go back up the tree to the root
"#2": {
// match and ignore Property2
"Property2": null,
//
// match Property* and use it and the captured
// "prefix" to create the output key
// &(2,1) references the Tag_*, and pull off the "**2ABC**" part
"Property*": "&(2,1)_&"
}
}
}
}
}
]