I'm working on a sample NiFi flow where I get a Json file as input. I use EvaluateJsonPath processor to get the value of the desired path. I've set the destination of EvaluateJsonPath as "flow-file attribute" and I added new properties with the required JsonPath. For ex: Property name: username, value: $.input.username. Now, I will be needing this value in the next processor. So I want to know which processor should I use to read the attributes of the flow-file.
You don't need a special processor to read the attributes of a FlowFile.
If this is your attribute key/value pair.
username : $.input.username
You can read that value like below in any processor property that supports Expression Language.
${username}
Related
It appears that the standard Apache NiFi readers/writers can only parse JSON input based on Avro schema.
Avro schema is limiting for JSON, e.g. it does not allow valid JSON properties starting with digits.
JoltTransformJSON processor can help here (it doesn't impose Avro limitations to how the input JSON may look like), but it seems that this processor does not support batch FlowFiles. It is also not based on the readers and writers (maybe because of that).
Is there a way to read arbitrary valid batch JSON input, e.g. in multi-line form
{"myprop":"myval","12345":"12345",...}
{"myprop":"myval2","12345":"67890",...}
and transform it to other JSON structure, e.g. defined by JSON schema, and e.g. using JSON Patch transformation, without writing my own processor?
Update
I am using Apache NiFi 1.7.1
Update 2
Unfortunately, #Shu's suggestion did work. I am getting same error.
Reduced the case to a single UpdateRecord processor that reads JSON with numeric properties and writes to a JSON without such properties using
myprop : /data/5836c846e4b0f28d05b40202
mapping. Still same error :(
it does not allow valid JSON properties starting with digits?
This bug NiFi-4612 fixed in NiFi-1.5 version, We can use AvroSchemaRegistry to defined your schema and change the
Validate Field Names
false
Then we can have avro schema field names starting with digits.
For more details refer to this link.
Is there a way to read arbitrary valid batch JSON input, e.g. in multi-line form?
This bug NiFi-4456 fixed in NiFi-1.7, if you are not using this version of NiFi then we can do workaround to create an array of json messages with ,(comma delimiter) by using.
Flow:
1.SplitText //split the flowfile with 1 line count
2.MergeRecord //merge the flowfiles into one
3.ConvertRecord
For more details regards to this particular issues refer to this link(i have explained with the flow).
I have simple pipeline using apache nifi and i want to publish some messages in kafka topic using existing kafka puplisher processor.
The problem is how to specify kafka key using apache nifi expression language?
I tired something like ${message:jsonPath('$.key')} but, of course, i got an error because object message does not exist.
I also tried to use filename object which is something like a default object name for input messages, but it didn't help
Using another kafka publisher processor it is possible by setting message key field property, but what about PublishKafka processor?
NiFi expression language can only reference flow file attributes, and cannot directly reference the content (this is done on purpose).
So if you want to use the value of a field from your json document as the key, then you need to first use another processor like EvaluateJsonPath to extract the value of that field into a flow file attribute.
Lets say you have a field "foo" in your json document, you might use EvaluateJsonPath with destination to set to "flow file attributes" and then add a dynamic property like:
foo = $.foo
Then in PublishKafka set the key property to ${foo}.
Keep in mind this only makes sense if you have a single json document per flow file, otherwise if you have multiple then it is unclear what the key is since you can only have one "foo" attribute for the flow file, but many "foo" fields in the content of the flow file.
i have a stream of json in apache nifi that contain dynamic fields (maximum 11 fields) and i want to convert it to csv file.
sample json:
{
"field1":"some text",
"field2":"some text",
"field3":"some text",
"field4":"some text",
"field5":"some text",
"field6":"some text",
"field7":"some text"
}
i don't wanna using replace or json evaluate; how i do it with ConvertRecord?
using this processor is so odd and hard to work...
Clear expression about dynamic fields:
i have 11 fields at total. one record may have contain 7 fields, and next record may contain 11 fields and next 9 fields...
The steps provided below will help you in getting this done.:
Connect your source processor which generates/outputs the JSON files to ConvertRecord.
Configure ConvertRecord and set 'Record Reader' to use JsonTreeReader controller service and 'Record Writer' to use CSVRecordSetWriter controller service
Configure both the controller services and set Schema Registry property to use AvroSchemaRegistry
Configure AvroSchemaRegistry. Go to 'Properties' tab and click the + button which lets you add a dynamic property.
Give some property name (ex: mySchema) and for the value, give the Avro schema expected for your input JSON. (You can use InferAvroSchema processor to generate Avro schema for your JSON)
Configure both JsonTreeReader and CsvRecordSetWriter and set the 'Schema Name' property to the name provided above, in this case, mySchema.
Connect the relationships of ConvertRecord to downstream processors according to your need.
If i have
<xml><name>himasnhu</name><age>24</age></xml>
How can i covert it to
{"name":"himanshu","age":24} .
Thanks.
In Oracle 12.2 you should be able to use:
SELECT JSON_OBJECTAGG( id VALUE text )
FROM XMLTABLE(
'/xml/*'
PASSING XMLTYPE( '<xml><name>himanshu</name></xml>')
COLUMNS id VARCHAR2(200) PATH './name()',
text VARCHAR2(200) PATH './text()'
);
I'm not on a 12c system so this is untested.
In earlier versions you can write a Java function [1] [2] using one of the many Java JSON packages to perform the conversion and then load it into the database using the loadjava utility (or a CREATE JAVA statement) and then use that.
You can use the XML to JSON filter to convert an XML document to a JavaScript Object Notation (JSON) document. For details on the mapping conventions used, see:Github- Mapping convention
Configuration
To configure the XML to JSON filter, specify the following fields:
Name:
Enter a suitable name to reflect the role of this filter.
Automatically insert JSON array boundaries:
Select this option to attempt to automatically reconstruct JSON arrays from the incoming XML document. This option is selected by default.
[Note]
If the incoming XML document includes the processing instruction, the JSON array is reconstructed regardless of this option setting. If the XML document does not contain , and this option is selected, the filter makes an attempt at guessing what should be part of the array by examining the element names.
Is this possible to have a node property as json raw string and to filter on it with cypher ?
I have a node with some defined properties and metadata (json raw string).
I would like to select or filter on those metadata property.
This is something like this :
START movie=node:TYPE_INDEX(Type = 'MOVIE') // Start with the reference
MATCH movie-[t:TAG]->tag
WHERE collect(movie.Metadata).RatingPress > 3
RETURN distinct movie.Label
And metadata are something like this :
{"RatingPress" : "0","RatingSpectator" : 3"}
I have expected to use collect function in order to call the property like this :
collect(movie.Metadata).RatingPress
But, of course it fails...
Is this a way to bind some json string from a node property with cypher ?
Thanks for your help
That's going against the principles of properties. Why not set the properties in the JSON metadata directly on the node?
But to answer your question:
No, cypher has no knowledge about JSON.
We treat the entire Node as a JSON blob. Since Neo4j doesn't support hierarchical properties, we flatten out the JSON into delimited property names on save and unflatten them on read. You can then form Cypher queries on (for example) property name "foo.bar.baz". The queries tend to look a bit funky because you'll need to quote them using single back quotes, but it works.