Evaluate value of JSON Key using NIFI - json

I have a scenario where I can have multiple different types of json objects coming into my system. I do not know the object type ahead of time and based upon object type, will route to a different processor in my flow
{
"book": {
"id": "1234",
"name": "book1"
}
}
or
{
"video": {
"id": "3214",
"name": "video1"
}
}
or
{
"magazine": {
"id": "3233",
"name": "magazine1"
}
}
how can I evaluate if the object is a book, or a video, or a magazine to route to the correct processor
I've tried using evaluatejsonpath using the ~ but it just outputs the entire json object
Current flow :

One way is to extract all top level fields using EvaluateJsonPath processor, set the extracted field values to dynamic properties, and use the properties in RouteOnAttribute processor to route the flow to correct downstream processor.
EvaluateJsonPath:
Please don't forget to set
'Destination' to 'flowfile-attribute' and
'Return Type' to 'json'
If EvaluateJsonPath processor could not find the field or element, then the value of dynamic property will be set to empty string.
All we need to do is to use the dynamic properties in RouteOnAttribute processor.
RouteOnAttribute:
Using equals() and not()
or using isEmpty() and not()
Please don't forget to set
'Routing Strategy' to 'Route to Property Name'.
Apache NiFi expression language guide
Example Flow:
I am using PutFile processor as a downstream processor, for an example. It could be any processor.

Related

Validating against dynamic data - JSON Schema - Ajv

I'm trying to create a JSON Schema for something very dynamic. Say I have two pieces of data, and I want one (the source) to determine the validity of the other (the target). Both can change over time, but both will always be an array of objects with known properties. For example:
source.json
[
{ "id": 23, "active": true },
{ "id": 9, "active": false },
{ "id": 6, "active": true }
]
target.json
[
{ "identifier": 6 }
]
The schema I'm trying to create is this: For each active object in the source array, there should be an equivalent object in the target array. A little more formally, given an object in the source array where "active" equals true and "id" equals x, there should be an object in the target array where "identifier" equals x.
In the example above, the target would be invalid because it's missing an object like { "identifier": 23 }.
However, I want to statically define this schema (or something capable of generating it) in a JSON file ahead of time, and this feels pretty tough since the source array can change. I'm using Ajv, and I'm aware that it supports the $data reference, but I'm not sure that's enough to help me here. The other option I could see is creating some kind of schema-generator definition? In concept, it too would be a JSON object I define ahead of time, but at runtime it would be used to safely generate arbitrary schemas based on runtime data such as the source array. However, if a mechanism like this doesn't already exist, trying to implement it myself sounds like a great way to give myself a code-injection vulnerability.
Thanks for your time!

Unable to get a key value called properties "properties": "Value" Groovy

I'm making API requests to a service which returns a JSON object within the body.
I can't seem to get the value of a key called "properties" within groovy.
Everytime I call obj.properties i get the following back
{
"class": "org.json.JSONObject"
}
but if I call just the obj I get the expected JSON object
{
"dummy1": ,
"dummy2": false,
"dummy3": etsad,
"dummy4": asdfw,
"dummy5": qweqwe,
"dummy6": 123123,
"properties": {
"country": UK,
}
}
Likewise if I obj.dummy2 i get false it's only when I obj.properties do I get the above mentioned response
Notice groovy have a special handling for Object's properties, for example for number:
def y = 25
print y.properties
It will print [class:class java.lang.Integer]
So it's part of basic groovy object
See also an answer about getting non-synthetic properties from groovy object
As #daggett comment, you can use
obj.get('properties')
Check out this answer here on how to access the properties of objects.
The reason obj.properties isn't working is most likely due to the fact that every object will have properties, and in your case obj.properties is getting the properties of the JSON object and not the value associated with the key.
Instead of obj.properties, consider obj['properties']

Length is not worked in JSON Extractor in Jmeter

I need to get the count of card from json file. For this I've used $.storedCards.cards.lenght
in JSON Extractor but it doesn't work. There is an error message:
Options AS_PATH_LIST and ALWAYS_RETURN_LIST are not allowed when using path functions!
After that I've tried JSR223 PostProcessor with next script on goovy
def jsonText = '''${AllCards}''' //${AllCards} has json value
def json = new JsonSlurper().parseText(jsonText)
log.info( "Json length---------->"+json.resource.size())
${CardsCount} = props.get("4") //vars.put(json.resource.size.toString())
but there is problem with set value to my variable. Or when i've created variable in Groovy it was impossible to use outside from script.
My json file
"storedCards":
{
"cards":
[
{
"CardId":"123",
"cardBrand":"Visa",
"lastFourDigits":"2968",
},
{
"CardId":"321",
"cardBrand":"Visa",
"lastFourDigits":"2968",
},
..........
],
How can i get the count of card and set to my Variables? what should i use for this?
Your JSON data seems to be invalid. Assuming you have the valid JSON like below, I'm answering your question.
{
"storedCards": {
"cards": [
{
"CardId": "123",
"cardBrand": "Visa",
"lastFourDigits": "2968"
},
{
"CardId": "321",
"cardBrand": "Visa",
"lastFourDigits": "2968"
}
]
}
}
You dont need to write Groovy code, you can resolve this using JSON Extractor. Instead of using length function, use JSON path predicate like this-
$.storedCards.cards[*]
Though Variable you used in JSON Extractor won't give the solution right away, another JMeter function helps - __RandomFromMultipleVars
Excerpt from documentation -
The RandomFromMultipleVars function returns a random value based on the variable values provided by Source Variables.
The variables can be simple or multi-valued as they can be generated by the following extractors:
Boundary Extractor
Regular Expression Extractor
CSS Selector Extractor
JSON Extractor
XPath Extractor
XPath2 Extractor
Multi-value vars are the ones that are extracted when you set -1 for
Match Numbers. This leads to creation of match number variable called
varName_matchNr and for each value to the creation of variable
varName_n where n = 1, 2, 3 etc.
So once you use the predicate, you will get the count in the yourVariableName_matchNr. Example:-
Hope this help.

How do I programmatically create JSON in XQuery in MarkLogic?

I need to build up a JSON node in XQuery in MarkLogic. I know that I can use xdmp:unquote() to parse from a string into a node(). However, I’d like to build the JSON programmatically, without ugly string concatenation. I can use computed element constructors to build XML nodes in XQuery. Is there something similar for JSON nodes?
JSON is implemented in MarkLogic as an extension to the XML data model. MarkLogic 8 introduces object-node, array-node, number-node, boolean-node, and null-node tests and constructors. Thus, in XQuery you can build JSON with computed constructors, just like you would with XML. For example,
object-node {
"key" || fn:string(xdmp:random(100)): array-node { 1, 2, 3 },
"another": object-node { "child": text {'asdf'} },
"lastButNotLeast": boolean-node { fn:true() }
}
will create the JSON,
{
"key47": [1, 2, 3],
"another": {
"child": "asdf"
},
"lastButNotLeast": true
}
Aside: In JavaScript you can build JSON-like structures as JavaScript objects using JavaScript syntax. You can convert a JavaScript object into a JSON node using xdmp.toJSON(). Most builtin functions that require a JSON node, however, will do this conversion automatically, such as xdmp.documentInsert().

TJSONUnMarshal: how to track what is actually unmarshalled

Is there another way to track what is unmarshalled than write own reverter for each field?
I'm updating my local data based on json message and my problem is (simplified):
I'm expecting json like
{ "items": [ { "id":1, "name":"foobar", "price":"12.34" } ] }
which is then unmarshaled to class TItems by
UnMarshaller.TryCreateObject( TItems, TJsonObject( OneJsonElement ), TargetItem )
My problem is that I can't make difference between
{ "items": [ { "id":1, "name":"", "price":"12.34" } ] }
and
{ "items": [ { "id":1, "price":"12.34" } ] }
In both cases name is blank and i'd like to update only those fields that are passed on json message. Of course I could create a reverted for each field, but there are plenty of fields and messages so it's quite huge.
I tried to look REST.Jsonreflect.pas source, but couldn't make sense.
I'm using delphi 10.
In Rest.Json unit there is a TJson class defined that offers several convenience methods like converting objects to JSON and vice versa. Specifically, it has a class function JsonToObject where you can specify options like for example ignore empty strings or ignore empty arrays. I think the TJson class can serve you. For unmarshalling complex business objects you have to write custom converters though.
Actually, my problem was finally simple to solve.
Instead of using TJSONUnMarshal.tryCreateObject I use now TJSONUnMarshal.CreateObject. First one has object parameters declared with out modifier, but CreateObject has Object parameter var modifier, so I was able to
create object, initalize it from database and pass it to CreateObject which only modifies fields in json message.