In Apache Nifi, i want to split a line of a json file based on the content of a field delemited by comma.
This is an example of my input flowfile :
{
"name":"app",
"os":"linux",
"instance":"instance1,instance2,instance3,instance4"
}
And this is want as output :
{
"name":"app",
"os":"linux",
"instance":"instance1"
},
{
"name":"app",
"os":"linux",
"instance":"instance2"
},
{
"name":"app",
"os":"linux",
"instance":"instance3"
},
{
"name":"app",
"os":"linux",
"instance":"instance4"
}
I need to know if it's possible to realise this task with the joltTransformJson processor or if i must do that with a script, in this case can you please show some similar examples scripts.
Thanks
don't know about jolt.
with ExecuteGroovyProcessor you can do this transformation in a following way:
import groovy.json.*
def ff=session.get()
if(!ff)return
//read stream, convert to reader, parse to objects
def json=ff.read().withReader("UTF-8"){r-> new JsonSlurper().parse(r) }
//transform json
json = json.instance.split(',').collect{e-> json+[instance:e] }
//write
ff.write("UTF-8"){w-> new JsonBuilder(json).writeTo(w)}
//transfer to success
REL_SUCCESS<<ff
The same but for ExecuteScript processor:
import groovy.json.*
def ff=session.get()
if(!ff)return
ff = session.write(ff, {inputStream, outputStream ->
def json=inputStream.withReader("UTF-8"){r-> new JsonSlurper().parse(r) }
json = json.instance.split(',').collect{e-> json+[instance:e] }
outputStream.withWriter("UTF-8"){w-> new JsonBuilder(json).writeTo(w)}
} as StreamCallback)
session.transfer(ff, REL_SUCCESS)
Related
I have the following JSON from a response from a random REST request:
{ pages= { 56206384={ title = Siberia } }
how can I extract the element "title", given that the number 56206384 will change with every new request? is there a way to have a regex expression for any number?
already tried: def title = ParsedResponse.query.pages.(*).title
any ideas? would appreciate the help
{ pages= { 56206384={ title = Siberia } }
how can I extract the element "title", given that the number 56206384
will change with every new request?
If you know the structure will be just like what you have shown, you could do something like this.
import groovy.json.JsonSlurper
def slurper = new JsonSlurper()
def json = slurper.parseText '{ "pages": { "56206384" : { "title": "Siberia" } }'
def title = json.pages.values().title[0]
I'm looking for a way to convert a comma delimited string in a json to an array in NiFi.
Then length of the array is variable. I've tried a few things now, including using the jolt transformer, but the version of jolt with nifi doesn't support split.
My flow file looks like:
{
"arrStr": "abc,def,hij",
"something": "else"
}
And I'm trying to convert it to:
{
"arrStr": ["abc", "def", "hij"],
"something": "else"
}
Update 7/10:
Sorry, should have included that I am on nifi version 1.7.1 which I think has jolt version 0.1.0.
This works
"split" unit test case
https://github.com/bazaarvoice/jolt/blob/7812399d1c955742d81eae363244a2d0ef86cf3b/jolt-core/src/test/resources/json/modifier/functions/stringsSplitTest.json
[
{
"operation": "modify-overwrite-beta",
"spec": {
"arrStr": "=split(',',#(1,arrStr))"
}
}
]
Shout out to #daggett for commenting about groovy. I went with that route and it's real easy to do. I had to use ExecuteScript processor, not the ExecuteGroovy processor, as the ExecuteGroovy processor didn't load up the IOUtils I wanted to use.
import groovy.json.JsonBuilder
import groovy.json.JsonSlurper
import org.apache.commons.io.IOUtils
import java.nio.charset.StandardCharsets
def flowFile = session.get();
if (flowFile == null) {
return;
}
flowFile = session.write(flowFile,
{ inputStream, outputStream ->
def content = IOUtils.toString(inputStream, java.nio.charset.StandardCharsets.UTF_8)
def slurped = new JsonSlurper().parseText(content)
def builder = new JsonBuilder(slurped)
builder.content.arrStr = builder.content.arrStr.split(',')
outputStream.write(builder.toPrettyString().getBytes(StandardCharsets.UTF_8))
} as StreamCallback)
session.transfer(flowFile, ExecuteScript.REL_SUCCESS)
i searched and tried out a lot of things to solve my problem. But it seems that nothing will work. Maybe I just understand it wrong. I have a test.json File looking like this.
{
"TEST-A": [{ "app_id":"aaa" }],
"TEST-B": [{ "app_id":"bbb" }],
"TEST-C": [{ "app_id":"ccc" }]
}
and now i want to edit TEST-B with "xxx". What am i trying for example ist:
import groovy.json.JsonSlurper
def slurper = new groovy.json.JsonSlurper()
def inputFile = new File("../config/test.json")
def inputJSON = new JsonSlurper().parseText(inputFile.text)
def builder = new JsonBuilder(inputJSON)
builder.content.TEST-B[0] = 'xxx'
I thought that i already have a map to edit or do i need to use "assert"?
Greetings Frost.
The following code:
import groovy.json.*
def data = '''\
{
"TEST-A": [{ "app_id":"aaa" }],
"TEST-B": [{ "app_id":"bbb" }],
"TEST-C": [{ "app_id":"ccc" }]
}'''
def parsed = new JsonSlurper().parseText(data)
println "parsed is a java.util.Map: ${parsed instanceof Map}"
parsed.'TEST-B'[0].app_id = 'xxx'
println "map after change: $parsed"
def result = JsonOutput.toJson(parsed)
println "result is a String: ${result instanceof String}"
println "result: $result"
println "pretty result:\n${JsonOutput.prettyPrint(result)}"
when run, prints out:
~> groovy test.groovy
parsed is a java.util.Map: true
map after change: [TEST-A:[[app_id:aaa]], TEST-B:[[app_id:xxx]], TEST-C:[[app_id:ccc]]]
result is a String: true
result: {"TEST-A":[{"app_id":"aaa"}],"TEST-B":[{"app_id":"xxx"}],"TEST-C":[{"app_id":"ccc"}]}
pretty result:
{
"TEST-A": [
{
"app_id": "aaa"
}
],
"TEST-B": [
{
"app_id": "xxx"
}
],
"TEST-C": [
{
"app_id": "ccc"
}
]
}
and I believe accomplishes the essence of what you are trying to do. The thing to understand about JsonSlurper is after parsing the in-data, what you get back is a normal java.util.Map (or possibly a java.util.List depending on the in-data json).
In other words, the parsed variable above is just a Map where the keys are strings and the values are Lists of Maps.
The second thing to keep in mind is that keys like TEST-B are not valid indentifiers in groovy so you can not just write parsed.TEST-B because that would be interpreted as parsed.TEST - B, so you have to quote keys with special characters like this.
After you change the map and assuming you want to get back to a json representation, you have to use something like JsonOutput as in the code above.
I need help with converting an xml to json in Groovy. The xml I have is something like this
def xmlrecords = '''<root>
<node1>abc</node1>
<node2>def</node2>
<node3>hij</node3>
</root>'''
I know this is fairly simple, but I'm struggling to get to grips with groovy. So any help would be very much appreciated.
Thanks.
UPDATE
I know that I can do something like
def xml= new XmlParser().parseText( xmlrecords )
def json = new JsonBuilder()
json.records
{
node1 "abc"
node2 "def"
node3 "hij"
}
println json.toPrettyString()
but what I want to do is access the data of the nodes like this
json.records
{
node1 xml.node1 //xml.node1=abc
node2 xml.node2
node3 xml.node3
}
since the data that each node stores keeps changing for me. Whatever code that I have written above doesn't work and I have been breaking my head over this. So Could you please help me out?
You're pretty much on the right track. You just needed to apply the .text() function on your xml object.
See below
static main(args) {
def xmlrecords = '''<root>
<node1>abc</node1>
<node2>def</node2>
<node3>hij</node3>
</root>'''
def xml= new XmlParser().parseText(xmlrecords)
def json = new JsonBuilder()
json.records
{
node1 xml.node1.text() //xml.node1=abc
node2 xml.node2.text()
node3 xml.node3.text()
}
println json.toPrettyString()
}
Output
{
"records": {
"node1": "abc",
"node2": "def",
"node3": "hij"
}
}
Hope this helps!
Direct XML to JSON conversion can create ugly deeply-nested JSON. You need to transform the XML into the simplified structure you want in the JSON.
def xmlrecords = '''<root>
<node1>abc</node1>
<node2>def</node2>
<node3>hij</node3>
</root>'''
def xml= new XmlParser().parseText( xmlrecords )
// programmatically transform XML into Map with node name and value
def map = new TreeMap()
xml.each {
map.put(it.name(), it.text())
}
def json = JsonOutput.toJson(map)
println JsonOutput.prettyPrint(json)
The JSON output:
{
"node1": "abc",
"node2": "def",
"node3": "hij"
}
I am using groovy to automate some tests on SoapUI, and I wanted to also automate assertions in a way I would get a field's name and value from a *.txt file and check if the wanted field does exist with the wanted value in the SOapUI response.
Suppose I have the following json response:
{
"path" : {
"field" : "My Wanted Value"
}
}
And from my text file I would have the following two strings :
path="path.field"
value="My Wanted Value"
I tried the following :
import groovy.json.JsonSlurper
def response = messageExchange.response.responseContent
def slurper = new JsonSlurper()
def json = slurper.parseText response
assert json.path==value;
But of course it doesn't work.
Any idea how can I get it done please?
Thank you
I think your problem is to access a json value from a path based with . notation, in your case path.field to solve this you can use the follow approach:
import groovy.json.JsonSlurper
def path='path.field'
def value='My Wanted Value'
def response = '''{
"path" : {
"field" : "My Wanted Value"
}
}'''
def json = new JsonSlurper().parseText response
// split the path an iterate over it step by step to
// find your value
path.split("\\.").each {
json = json[it]
}
assert json == value
println json // My Wanted Value
println value // My Wanted Value
Additionally I'm not sure if you're also asking how to read the values from a file, if it's also a requirement you can use ConfigSlurper to do so supposing you've a file called myProps.txt with your content:
path="path.field"
value="My Wanted Value"
You can access it using the follow approach:
import groovy.util.ConfigSlurper
def urlFile = new File('C:/temp/myProps.txt').toURI().toURL()
def config = new ConfigSlurper().parse(urlFile);
println config.path // path.field
println config.value // My Wanted Value
All together (json path + read config from file):
import groovy.json.JsonSlurper
import groovy.util.ConfigSlurper
def response = '''{
"path" : {
"field" : "My Wanted Value"
}
}'''
// get the properties from the config file
def urlFile = new File('C:/temp/myProps.txt').toURI().toURL()
def config = new ConfigSlurper().parse(urlFile);
def path=config.path
def value=config.value
def json = new JsonSlurper().parseText response
// split the path an iterate over it step by step
// to find your value
path.split("\\.").each {
json = json[it]
}
assert json == value
println json // My Wanted Value
println value // My Wanted Value
Hope this helps,