NiFi JsonRecordSetWriter skipping a nested field - json

Question

I am pulling messages out of a Kafka topic into NiFi, and am seeing problems with the JsonRecordSetWriter not outputting a nested field. If I switch from that writer to a CSV writer I don't have the same problem, but both the Avro and XML writers have the same problem, so it's a problem with the inferred schema, I believe.
Here is my simplified input:
{
"data": [
{
"object": {
"extensions": {
"field1": "TS",
"field2": "howdy"
}
}
},
{
"object": {
"extensions": {
"field1": "TT",
"field3": "something"
}
}
}
]
}
And the output:
[ {
"data" : [ {
"object" : {
"extensions" : {
"field1" : "TS",
"field2" : "howdy"
}
}
}, {
"object" : {
"extensions" : {
"field1" : "TT",
"field2" : null
}
}
} ]
} ]
If I use the CSV writer I get field1 and 2 for the first record and field 1 and 3 for the second record, so the JsonTreeReader is getting the data read from Kafka correctly, it's the JsonRecordSetWriter that isn't writing it out right. Looks like the schema inference engine is reading the first record in the array for it's schema, and then outputting based on that. Field2 is output regardless that it doesn't exist in record2, and field3 is ignored since it didn't exist in record1.
Any suggestions from folks who know more than I?
Thanks in advance for any assistance!

Answer 1

I solved my problem, but didn't figure out why it was happening, unfortunately. The way I solved it was to switch from the ConsumeKafkaRecord processor to the ConsumeKafka processor, which doesn't use the JsonTreeReader and JsonRecordSetWriter controllers to process Json. Since the values I was pulling out of Kafka were already in Json, getting them as a string and going from there (adding an application/json mime.type to ensure they were correctly treated) worked just fine for me.
The problem could also have been solved by creating a schema which contained every field possible, but that would have resulted in a lot of null fields, as the records are sparsely populated - and it was further complicated by the fact that one of my fields started with a symbol, and NiFi uses Avro-formatted schemas everywhere, so I would have had to work around that (a bug fix in NiFi 1.7 allows it, but it has limitations also).
So I'm on my way - not sure if this experience will help others, but if it does, great!

Answer 2

I had almost the same problem with a transformRecord with common jsons. In short, the flow listed an entry in an application event s3 and transformed the jonsons of those events into 2 types:
1 - For an athena table.
2 - For an HTTP call.
Each event file contained more than 200 events, and in the middle of these events it contained one that had a nesting called "traits" and the information from that traits never came (it was an event that rarely came in this "set" of events, about 1 to 2% of the events in the archive, but they are extremely important).
I had to modify the beginning of my flow to make it work.
The flow was:
ListS3 -> FetchS3Object -> JoltTransformRecord -> SplitJson -> EvaluateJsonPath
And stayed:
ListS3 -> FetchS3Object -> ConvertRecord -> SplitJson -> JoltTransformJson -> EvaluateJsonPath
And now it works perfectly.
The #Chrick solution helped me a lot to identify what the error was and, as my problem is a little different, I hope this can help someone.