I'm looking for a way to convert a comma delimited string in a json to an array in NiFi.
Then length of the array is variable. I've tried a few things now, including using the jolt transformer, but the version of jolt with nifi doesn't support split.
My flow file looks like:
{
"arrStr": "abc,def,hij",
"something": "else"
}
And I'm trying to convert it to:
{
"arrStr": ["abc", "def", "hij"],
"something": "else"
}
Update 7/10:
Sorry, should have included that I am on nifi version 1.7.1 which I think has jolt version 0.1.0.
This works
"split" unit test case
https://github.com/bazaarvoice/jolt/blob/7812399d1c955742d81eae363244a2d0ef86cf3b/jolt-core/src/test/resources/json/modifier/functions/stringsSplitTest.json
[
{
"operation": "modify-overwrite-beta",
"spec": {
"arrStr": "=split(',',#(1,arrStr))"
}
}
]
Shout out to #daggett for commenting about groovy. I went with that route and it's real easy to do. I had to use ExecuteScript processor, not the ExecuteGroovy processor, as the ExecuteGroovy processor didn't load up the IOUtils I wanted to use.
import groovy.json.JsonBuilder
import groovy.json.JsonSlurper
import org.apache.commons.io.IOUtils
import java.nio.charset.StandardCharsets
def flowFile = session.get();
if (flowFile == null) {
return;
}
flowFile = session.write(flowFile,
{ inputStream, outputStream ->
def content = IOUtils.toString(inputStream, java.nio.charset.StandardCharsets.UTF_8)
def slurped = new JsonSlurper().parseText(content)
def builder = new JsonBuilder(slurped)
builder.content.arrStr = builder.content.arrStr.split(',')
outputStream.write(builder.toPrettyString().getBytes(StandardCharsets.UTF_8))
} as StreamCallback)
session.transfer(flowFile, ExecuteScript.REL_SUCCESS)
Related
Imagine I have a json (in string format) that looked like:
val jsonString: String = "
{
"toys": {
"orange toys": {
"brand":"Toys-R-Us",
"price":"123.45"
}
},
"date":"05-27-1996"
}
"
My question is, how can I get the value for "brand" (Toys-R-Us), in Scala, using the scala.util.parsing.json library? I am assuming that this might require the traversing of the json or maybe even easier, a way to look up the key "brand" and from their obtaining the value.
The scala.util.parsing.json library is outdated, unsafe for open systems, and not included in the latest version of the Scala library.
Use dijon FTW!
import dijon._
import scala.language.dynamics._
val jsonString: String = """
{
"toys": {
"orange toys": {
"brand":"Toys-R-Us",
"price":"123.45"
}
},
"date":"05-27-1996"
}
"""
println(parse(jsonString).toys.`orange toys`.brand)
Here is a runnable code on Scastie that works fine with Scala 2 and Scala 3.
I have this json:
{
"-1": {
"description": "test1"
},
"222": {
"description": "test2"
},
"223": {
"description": "test3"
},
"224": {
"description": "test4"
},
"recordsCount": 4
}
Using this Json path expression: $.. it returns all the Json.
I would like to find the Json path expression that returns only the ID values:
"-1"
"222"
"223"
"224"
Can you help me?
Accessing the property (key) instead of the value is a feature not present in the original JSONPath specifications, and only a few implementation support this feature. So, no luck using the JMeter JsonPath Plugin/Extractor extracting only property names.
For instance, JsonPath-Plus for JavaScript offers the operator ~ for grabbing property names of matching items.
As an alternative, you could use JSR223 Groovy to get the results from JSON:
import groovy.json.JsonSlurper;
import groovy.json.JsonOutput; //if output as JSON is needed
JsonSlurper JSON = new JsonSlurper ();
jsonResponse = JSON.parseText(prev.getResponseDataAsString());
//println json.keySet() //[-1, 222, 223, 224, recordsCount]
vars.put("firstName", response.members[2].firstName.toString());
Set<String> keySet=obj.keySet();
List<String> keys = [];
// iterate over the key set, adding only numbers to result list
for(String key:keySet){
if(key.isNumber())
keys.add(key);
}
//println(keys); //[-1, 222, 223, 224]
def output = JsonOutput.toJson(keys); //convert to JSON string if neede
//println(output); //["-1","222","223","224"]
vars.put("jsonKeys", output.toString());
Demo: Online Groovy sample.
More information:
How to extract data from JSON response using JMETER (See 8.3 JSR223 with Groovy)
Scripting JMeter Assertions in Groovy - A Tutorial
Jayway JsonPath (alternative Java JSONPath library)
i searched and tried out a lot of things to solve my problem. But it seems that nothing will work. Maybe I just understand it wrong. I have a test.json File looking like this.
{
"TEST-A": [{ "app_id":"aaa" }],
"TEST-B": [{ "app_id":"bbb" }],
"TEST-C": [{ "app_id":"ccc" }]
}
and now i want to edit TEST-B with "xxx". What am i trying for example ist:
import groovy.json.JsonSlurper
def slurper = new groovy.json.JsonSlurper()
def inputFile = new File("../config/test.json")
def inputJSON = new JsonSlurper().parseText(inputFile.text)
def builder = new JsonBuilder(inputJSON)
builder.content.TEST-B[0] = 'xxx'
I thought that i already have a map to edit or do i need to use "assert"?
Greetings Frost.
The following code:
import groovy.json.*
def data = '''\
{
"TEST-A": [{ "app_id":"aaa" }],
"TEST-B": [{ "app_id":"bbb" }],
"TEST-C": [{ "app_id":"ccc" }]
}'''
def parsed = new JsonSlurper().parseText(data)
println "parsed is a java.util.Map: ${parsed instanceof Map}"
parsed.'TEST-B'[0].app_id = 'xxx'
println "map after change: $parsed"
def result = JsonOutput.toJson(parsed)
println "result is a String: ${result instanceof String}"
println "result: $result"
println "pretty result:\n${JsonOutput.prettyPrint(result)}"
when run, prints out:
~> groovy test.groovy
parsed is a java.util.Map: true
map after change: [TEST-A:[[app_id:aaa]], TEST-B:[[app_id:xxx]], TEST-C:[[app_id:ccc]]]
result is a String: true
result: {"TEST-A":[{"app_id":"aaa"}],"TEST-B":[{"app_id":"xxx"}],"TEST-C":[{"app_id":"ccc"}]}
pretty result:
{
"TEST-A": [
{
"app_id": "aaa"
}
],
"TEST-B": [
{
"app_id": "xxx"
}
],
"TEST-C": [
{
"app_id": "ccc"
}
]
}
and I believe accomplishes the essence of what you are trying to do. The thing to understand about JsonSlurper is after parsing the in-data, what you get back is a normal java.util.Map (or possibly a java.util.List depending on the in-data json).
In other words, the parsed variable above is just a Map where the keys are strings and the values are Lists of Maps.
The second thing to keep in mind is that keys like TEST-B are not valid indentifiers in groovy so you can not just write parsed.TEST-B because that would be interpreted as parsed.TEST - B, so you have to quote keys with special characters like this.
After you change the map and assuming you want to get back to a json representation, you have to use something like JsonOutput as in the code above.
In Apache Nifi, i want to split a line of a json file based on the content of a field delemited by comma.
This is an example of my input flowfile :
{
"name":"app",
"os":"linux",
"instance":"instance1,instance2,instance3,instance4"
}
And this is want as output :
{
"name":"app",
"os":"linux",
"instance":"instance1"
},
{
"name":"app",
"os":"linux",
"instance":"instance2"
},
{
"name":"app",
"os":"linux",
"instance":"instance3"
},
{
"name":"app",
"os":"linux",
"instance":"instance4"
}
I need to know if it's possible to realise this task with the joltTransformJson processor or if i must do that with a script, in this case can you please show some similar examples scripts.
Thanks
don't know about jolt.
with ExecuteGroovyProcessor you can do this transformation in a following way:
import groovy.json.*
def ff=session.get()
if(!ff)return
//read stream, convert to reader, parse to objects
def json=ff.read().withReader("UTF-8"){r-> new JsonSlurper().parse(r) }
//transform json
json = json.instance.split(',').collect{e-> json+[instance:e] }
//write
ff.write("UTF-8"){w-> new JsonBuilder(json).writeTo(w)}
//transfer to success
REL_SUCCESS<<ff
The same but for ExecuteScript processor:
import groovy.json.*
def ff=session.get()
if(!ff)return
ff = session.write(ff, {inputStream, outputStream ->
def json=inputStream.withReader("UTF-8"){r-> new JsonSlurper().parse(r) }
json = json.instance.split(',').collect{e-> json+[instance:e] }
outputStream.withWriter("UTF-8"){w-> new JsonBuilder(json).writeTo(w)}
} as StreamCallback)
session.transfer(ff, REL_SUCCESS)
I need help with converting an xml to json in Groovy. The xml I have is something like this
def xmlrecords = '''<root>
<node1>abc</node1>
<node2>def</node2>
<node3>hij</node3>
</root>'''
I know this is fairly simple, but I'm struggling to get to grips with groovy. So any help would be very much appreciated.
Thanks.
UPDATE
I know that I can do something like
def xml= new XmlParser().parseText( xmlrecords )
def json = new JsonBuilder()
json.records
{
node1 "abc"
node2 "def"
node3 "hij"
}
println json.toPrettyString()
but what I want to do is access the data of the nodes like this
json.records
{
node1 xml.node1 //xml.node1=abc
node2 xml.node2
node3 xml.node3
}
since the data that each node stores keeps changing for me. Whatever code that I have written above doesn't work and I have been breaking my head over this. So Could you please help me out?
You're pretty much on the right track. You just needed to apply the .text() function on your xml object.
See below
static main(args) {
def xmlrecords = '''<root>
<node1>abc</node1>
<node2>def</node2>
<node3>hij</node3>
</root>'''
def xml= new XmlParser().parseText(xmlrecords)
def json = new JsonBuilder()
json.records
{
node1 xml.node1.text() //xml.node1=abc
node2 xml.node2.text()
node3 xml.node3.text()
}
println json.toPrettyString()
}
Output
{
"records": {
"node1": "abc",
"node2": "def",
"node3": "hij"
}
}
Hope this helps!
Direct XML to JSON conversion can create ugly deeply-nested JSON. You need to transform the XML into the simplified structure you want in the JSON.
def xmlrecords = '''<root>
<node1>abc</node1>
<node2>def</node2>
<node3>hij</node3>
</root>'''
def xml= new XmlParser().parseText( xmlrecords )
// programmatically transform XML into Map with node name and value
def map = new TreeMap()
xml.each {
map.put(it.name(), it.text())
}
def json = JsonOutput.toJson(map)
println JsonOutput.prettyPrint(json)
The JSON output:
{
"node1": "abc",
"node2": "def",
"node3": "hij"
}