Apache Nifi manipulate nested json with Execute Script - json

I am looking for a way to update the given Json's values and keys in a dynamic way. The way the Json is delivered is Always the same(in Terms of structure). The only Thing that differs is the amount of Data that is provided. So for example there could sometimes be 30, sometimes only 10 nestings etc.
…
"ampdata": [
{
"nr": "303",
"code": "JGJGh4958GH",
"status": "AVAILABLE",
"ability": [ "" ],
"type": "wheeled",
"conns": [
{
"nr": "447",
"status": "",
"version": "3",
"format": "sckt",
"amp": "32",
"vol": "400",
"vpower": 22
}
]
}
As Json uses other keys/values than I in my DB, I Need to convert them. Additionally I Need to Change some values if they match explicit strings.
So for example: "Code" has to be renamed to"adrID" and "sckt" should map to the values "bike".
I tried a simple Groovy-Script to remove the key and or Change the value. There is no Problem in changing values, but in changing the key itself. So I tried removing the key and adding a new key. Unfortunately I could not figure out how to add a new key:value to the given json. So how can I add a new pair of key:value or rename the key, if that´s possible. Have a look at my code-example
def flowFile = session.get()
if (!flowFile) return
try {
flowFile = session.write(flowFile,
{ inputStream, outputStream ->
def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
def obj = new JsonSlurper().parseText(text)
def objBuilder = new JsonBuilder(obj)
// Update ingestionDate field with today's date
for(i in 0..obj.data.size()-1){
obj.data[0].remove("postal_code")
objBuilder.data[0].postal_code=5
}
// Output updated JSON
def json = JsonOutput.toJson(obj)
outputStream.write(JsonOutput.prettyPrint(json).getBytes(StandardCharsets.UTF_8))
} as StreamCallback)
flowFile = session.putAttribute(flowFile, "filename", flowFile.getAttribute('filename').tokenize('.')[0]+'_translated.json')
session.transfer(flowFile, REL_SUCCESS)
} catch(Exception e) {
log.error('Error during JSON operations', e)
session.transfer(flowFile, REL_FAILURE)
}

...
def obj = new JsonSlurper().parse(inputStream, "UTF-8")
obj.data.each{e->
def value = e.remove("postal_code")
//set old value with a new key into object
e["postalCode"] = value
}
//write to output
def builder = new JsonBuilder(obj)
outputStream.withWriter("UTF-8"){ it << builder.toPrettyString() }

Related

Groovy Jsonslurper edit value in array

i searched and tried out a lot of things to solve my problem. But it seems that nothing will work. Maybe I just understand it wrong. I have a test.json File looking like this.
{
"TEST-A": [{ "app_id":"aaa" }],
"TEST-B": [{ "app_id":"bbb" }],
"TEST-C": [{ "app_id":"ccc" }]
}
and now i want to edit TEST-B with "xxx". What am i trying for example ist:
import groovy.json.JsonSlurper
def slurper = new groovy.json.JsonSlurper()
def inputFile = new File("../config/test.json")
def inputJSON = new JsonSlurper().parseText(inputFile.text)
def builder = new JsonBuilder(inputJSON)
builder.content.TEST-B[0] = 'xxx'
I thought that i already have a map to edit or do i need to use "assert"?
Greetings Frost.
The following code:
import groovy.json.*
def data = '''\
{
"TEST-A": [{ "app_id":"aaa" }],
"TEST-B": [{ "app_id":"bbb" }],
"TEST-C": [{ "app_id":"ccc" }]
}'''
def parsed = new JsonSlurper().parseText(data)
println "parsed is a java.util.Map: ${parsed instanceof Map}"
parsed.'TEST-B'[0].app_id = 'xxx'
println "map after change: $parsed"
def result = JsonOutput.toJson(parsed)
println "result is a String: ${result instanceof String}"
println "result: $result"
println "pretty result:\n${JsonOutput.prettyPrint(result)}"
when run, prints out:
~> groovy test.groovy
parsed is a java.util.Map: true
map after change: [TEST-A:[[app_id:aaa]], TEST-B:[[app_id:xxx]], TEST-C:[[app_id:ccc]]]
result is a String: true
result: {"TEST-A":[{"app_id":"aaa"}],"TEST-B":[{"app_id":"xxx"}],"TEST-C":[{"app_id":"ccc"}]}
pretty result:
{
"TEST-A": [
{
"app_id": "aaa"
}
],
"TEST-B": [
{
"app_id": "xxx"
}
],
"TEST-C": [
{
"app_id": "ccc"
}
]
}
and I believe accomplishes the essence of what you are trying to do. The thing to understand about JsonSlurper is after parsing the in-data, what you get back is a normal java.util.Map (or possibly a java.util.List depending on the in-data json).
In other words, the parsed variable above is just a Map where the keys are strings and the values are Lists of Maps.
The second thing to keep in mind is that keys like TEST-B are not valid indentifiers in groovy so you can not just write parsed.TEST-B because that would be interpreted as parsed.TEST - B, so you have to quote keys with special characters like this.
After you change the map and assuming you want to get back to a json representation, you have to use something like JsonOutput as in the code above.

when pass string variable to jasonparser jsonobject returns null

To read the json file I have created below method in groovy. I am calling it from another class.
class WUPage{
#Shared
String fileContents ;
def jsonSlurper ;
def jsonObject ;
public String getValueFromJsonFile(String fileName, String jsonKey){
fileContents = new File(fileName).getText('UTF-8')
jsonSlurper = new JsonSlurper()
jsonObject = jsonSlurper.parseText(fileContents)
println " Json Key from method is : " + jsonObject.jsonKey
println " Json Key hardecoded is : " + jsonObject.pipe.id
return jsonObject.jsonKey
}
}
When I am calling this method from another class I am passing file name and the key to it like below
getValueFromJsonFile(jsonFIleName, "pipe.id")
I am getting below output
Json Key from method is : null
Json Key hardecoded is : [India]
From above output second line is correct when key is hardcoded. It seems it's not recognizing key coming from method parameter.
Can you please help me on this.
Json File is :
{
"source": [
{
"id": "manish",
"type": "csv",
"path": "/home/surya/f1.txt",
"delimiter": ",",
"tableName": "table1",
"schema": "f1,f2,f3,f4,f5"
}
],
"pipe": [
{
"id": "India",
"sql": "select f1,f2,f5 from table1"
}
],
"sinks": [
{
"id": "output1",
"type": "aaa",
"path": "/home/surya/out",
"format": "json"
}
]
}
It is because the dot in the key name is not doing what you expected. You didn't show your JSON data in the original post, but I can deduce that there is a top-level object "pipe" that contains an id key. Groovy can't magically know this, so it applies the entire parameter as a top-level key, not as compound deep keys.
You are trying to always get the key jsonKey from that map.
def json = [a: [b: 42]] // just some map, but that's what your JsonSlurper result is
def key = 'a' // start with a simple key
println json.key // wont work, because groovy will access the key `key`
// => null
println json."$key" // works, now
// => [b:42]
key = 'a.b'
println json."$key" // wont work, because you can not access recursive keys like that
// => null
So at that point you have to decide, what to do and how comples your path there can become and what's the source of your path (e.g. is is always a string from somewhere else or is this for developer convenience).
You can Eval a string
You can "split" the path at . and then reduce this list over the map
Instead of a string pass in a Closure, that accesses the data

Extract field id in Vaadin json response by JMeter

Currently I'm trying to extract field id from Vaadin json response to detect all comboboxes (RTFComboBox) and process them further. Example response (cut down):
for(;;);[{
"syncId": 113,
"clientId": 113,
"changes": [],
"state": {
"1273": {
"caption": "",
"styles": ["tokenfield", "tokentextfield"]
},
"1274": {
"styles": ["RTFTokenField"]
},
"1275": {
"width": "185.0px",
"immediate": true,
"styles": ["RTFTokenField", "RTFComboBox", "tiny"],
"registeredEventListeners": ["focus"]
}
}
}]
What I need is to get id number (here 1275), I can deal with "for(;;)" junk but I can't extract proper id values. I was trying to use Regex Extractor but this solution isn't very flexible (or I just can't write proper expression).
Any ideas how to get parent id when styles child array contains 'RTFComboBox'? Maybe here is required some more complex solution, like some script in groovy, than JMeter's JSON Extractor?
I would advice, to write a groovy script in the beanshell processor, parse the JSON and then get all the keys for the state object. Here's the code snippet, please make sure you add the java-json.jar in the JMETER_HOME/lib folder
try {
String jsonString = prev.getResponseDataAsString();
JSONArray jsonArray = new JSONArray(jsonString);
JSONObject object = jsonArray.getJSONObject(0);
JSONObject states = object.getJSONObject("state");
String keys[] = states.getNames(states);
for(int i=0; i< keys.length; i++) {
log.info(keys[i]);
}
} catch (JSONException e) {
e.printStackTrace();
}
I've finally made it using JsonSlurper, this is my JSR223 PostProcessor script:
import groovy.json.JsonSlurper
def jsonSlurper = new JsonSlurper()
def response = jsonSlurper.parseText(prev.getResponseDataAsString().drop(8))
//drop to remove for(;;); junk preventing proper parse
def map = [:]
response.state[0].each {k, v -> if(v.styles != null && v.styles.contains("RTFComboBox")) {map.put(k, v)}}
//map contains key (field id-most important to me) and key (rest of values in node)
If you need some specific value, just in map.put line set v.field, i.e.:
map.put(k, v.immediate)

Getting the name of each field in JSON

I'm trying to parse a random JSON file in Grails.
First I need to get the name of each field
For example, given below JSON file,
{
"abbreviation": "EX",
"guid": "1209812-1l2kj1j-fwefoj9283jf-ae",
"metadata": {
"dataOrigin": "Example"
},
"rooms":
[
],
"site": {
"guid": "1209812-1l2kj1j-fwefoj9283jf-ae"
},
"title": "Example!!"
}
I want to find out the structure of the JSON file(lists of keys maybe), for example I want to save the list of keys such as 'abbreviation', 'guid', 'metadata', 'rooms', 'site', 'title' from this JSON file.
How would I do this?
(We need the name of the keys in order to get the value of that key, so with a arbitrarily structured JSON file I need to find out the keys first)
You can try below code
def filePath = "JSONFILE.json"
def text = new File(filePath).getText()
def json = JSON.parse(text)
def jsonKeys = json.collect{it.key}
println(jsonKeys)
This will print all json keys
From what dmahaptro commented, I figured out how to get all the keys within a JSON object.
Here is a simple sample code I wrote to test it
String jsonFile = new JsonSlurper().parseText(new URL(path to the json file).text)
JSONArray jsonParse = new JSONArray(jsonFile)
int len = jsonParse.length()
def names = []
def keys = []
(0..len-1).each {
JSONObject val = jsonParse.getJSONObject(it)
int numKeys = val.length()
names = val.names()
keys = val.keySet()
(0..numKeys-1).each {
def field = names[it]
println field +" : " + val."${field}"
}
}
This will print the key:value pair given a JSON file.

How to add "data" and "paging" section on JSON marshalling

I know i can customize the JSON response registering JSON marshallers to Domain entities, even i can create profiles with names for different responses.
This is done filling an array that later will be marshalled like:
JSON.registerObjectMarshaller(myDomain) {
def returnArray = [:]
returnArray['id'] = it.id
returnArray['name'] = it.name
returnArray['price'] = it.price
return returnArray
}
What i want is to alter the way it gets marshalled to have two sections like
{
"paging": {
"total": 100
},
"data": [
{
"id": 1,
"description": "description 1",
}
},
...
]
}
I assume i have to implemetn a custom JSON Marshaller but i don't know how to use it for a specific response instead of wide application.
EDIT: I assume i'll need a custom RENDERER apart from the marshaller. Is this one that i don't know how to use for specific response.
What about a simple:
def json = new JSON([ paging: [ total: myArray.totalCount ], data: myArray ])
Your domain objects will be converted with the marshaller you have set up while your paging data will simply be transformed into JSON.