I need to modify output to from this
"payload": {
"a": "value",
"b": "value",
.....
}
to
"a": "value",
"b": "value",
.....
I was looking at record_transformer but it looks like this is only to for example add new field or modify value of existing json property. Is it possible to extract data like i want to?
You can try something like this [If key names are fixed]
#type record_transformer
<record>
a $${record["payload"]["a"]
remove_keys payload
</record>
Or this plugin for dynamic keys https://github.com/kazegusuri/fluent-plugin-flatten-hash
Related
I want to split and transfer the json data in NiFi, Here is my json structure look like this;
I want to split json by id1,id2 array of json transfer to respective processor group say example processor_group a,b. I tried with evaluate json path $.id1,$.id2 but i didn't get exact solution. Can you please help me out from this issue;
{
"id1": [{
"u_name": "aa"
}, {
"addr": "bb"
}],
"id2": [{
"u_name": "aa"
}, {
"addr": "bb"
}]
}
The processor you're looking for is SplitJSON.
Configure it as follows:
Then, you'll receive two FlowFiles:
First one will contain the id1:
[{
"u_name": "aa"
}, {
"addr": "bb"
}]
second one will contain id2:
[{
"u_name": "aa"
}, {
"addr": "bb"
}]
Here is how to get to the values you want with EvaluateJsonPath:
#varun_rathinam Accessing json in an array object via EvaluateJsonPath can be quite confusing. I also notice the structure of your json is kind of confusing with same values in both. I have adjusted id2 for cc and dd for testing so that I can tell id1 and id2 values apart.
The solution you want is (see template for exact string values):
Notice we use the normal tree for each json object ( $.object ) then access the array ( 0, 1 ) then access the array's objects. Also notice it is possible to access the json object array with or without a . before the [.
Reference:
https://community.cloudera.com/t5/Support-Questions/how-to-extract-fields-in-flow-file-which-are-surrounded-by/m-p/208635
You can also find my template during testing of your issue on my GitHub:
https://github.com/steven-dfheinz/NiFi-Templates/blob/master/NiFI_EvaluateJsonPath_Demo.xml
I'm starting to use JSONata for data transformation and I was wondering if there exists a way to have a file which contains values transformation for some fields in json file.
I will have to do multiple transformation types, but most cases will be to translate a field value from "A" to "B" for example and I will be easier to do that in a file in order to not create new versions of data transformation and just will be necessary to create a new entry in this file.
Regards
You can use $lookup to perform simple mappings. For example, with the following JSON:
{
"mapping": [
{ "a": "a1" },
{ "b": "b1" }
],
"values": [
"a", "b"
]
}
You can map the values using:
values.$lookup($$.mapping, $)
In which case the result will be:
[
"a1",
"b1"
]
Alternatively you can look at $sift which will allow you to write a function to sift through the mappings.
I am currently using the Cassandra-Ruby driver to insert data from a JSON file into an existing table in my database.
the JSON file looks like this:
[
{
"id": "123",
"destination": "234",
"type": "equipment",
"support": "type 1",
"test": "test1"
},
{
"id": "234",
"destination": "123",
"type": "equipment",
"support": "type 1",
"test": "test1"
}
]
I am reading in the file like this:
file = File.read('itemType.json')
data_hash = JSON.parse(file) #return an array of hashes
Iterate through the array and get each hash
and insert each hash onto the table
data_hash.each do |has|
#check the type of each object
#puts has.class #return hash
insert_statement = session.prepare('INSERT INTO keyspace.table JSON ?')
session.execute(insert_statement, [has]) #error occurs here
end
After running this code, I get this error message
in `assert_instance_of': options must be a Hash
I checked that each object being inserted in the table is a hash, so I'm not sure why I'm getting this issue.
You are saying that you are inserting a JSON but you are not, you are trying to insert an object. See this example from the documentation:
INSERT INTO cycling.cyclist_category JSON '{
"category" : "Sprint",
"points" : 700,
"id" : "829aa84a-4bba-411f-a4fb-38167a987cda"
}';
You have to give it a json format if you do it like that.
using .to_json add \ escape character. This gave me error
INSERT INTO organization_metadata JSON '{\"id\":9150,\"destroyed\":false,\"name\":\"ABC\",\"timestamp\":1510541801000000000}';
and the following worked.
INSERT INTO organization_metadata JSON '{"id":9150,"destroyed":false,"name":"ABC","timestamp":1510541801000000000}';
Table of SQLite3 contains 2 column and select command will give below json output :
SELECT COLUMN2 FROM TABLE WHERE COLUMN1='1';
OUTPUT:
{ "categories": null, "settings": { "A": "10", "B": { "C":
["Test1","Test2","Test3"]}, "D": "1.0" } }
Now i wanted to set Value of Key "C" to ["Apple","orange"] .
SQLite3 supports JSON where we can set the value of Key using json_set. But i am not able to do that and need some help on it
Thanks
I would like to know whether, if a JSON Lines file is structured like this:
{"structure1":"some kind of file header"}
{"structure2": [ {"key1":"aaa", "key2":"bbb"}, {"key1":"one", "key2":"two"}]
{"structure2": [ {"key1":"xxx", "key2":"yyy"}, {"key1":"first", "key2":"second"}]
{"structure3":"maybe some kind of file footer"}
Is it considered a non-valid JSONLines format? I looked at the standard at http://jsonlines.org/ and I couldn't see anything one way or the other.
Thank you.
Your lines are each one valid ( but missing a '}' and the end of the second and third).
but if you want to put them all in one file it will be not valid. They have to be in array and separated with , or an object with key/value ( where the value can be array or object )
example of array :
[
{"structure1":"some kind of file header"},
{"structure2": [ {"key1":"aaa", "key2":"bbb"}, {"key1":"one", "key2":"two"}]},
{"structure2": [ {"key1":"xxx", "key2":"yyy"}, {"key1":"first", "key2":"second"}]},
{"structure3":"maybe some kind of file footer"}
]
or an object :
{
"structure1": "some kind of file header",
"structure2": [{
"key1": "aaa",
"key2": "bbb"
}, {
"key1": "one",
"key2": "two"
}],
"structure2": [{
"key1": "xxx",
"key2": "yyy"
}, {
"key1": "first",
"key2": "second"
}],
"structure3": "maybe some kind of file footer"
}
You can test your json on this site to see if valid or not :
http://www.jslint.com/ or http://jsonlint.com/