Is it possible to apply filter on the Nested JSON fields with help of Kafka Streams? If Yes, how those fields can be addressed?
For example,
{
"before":{
"id":1,
"name":"abc"
},
"after":{
"id":1,
"name":"xyz"
}
and now if name is modified in after field I do not want to filter it but fields other than name are getting modified I want to filter that record.
Thank you.
The deserializer of the configured Stream Serde should return your object type. You can then filter just like a regular Java Stream
stream.filter(yourMesssage -> compareCDCRecords(yourMessage.getBefore(), yourMessage.getAfter()))
Related
For context I have Azure Dataflow reading from CosmosDB where the table has two "schemas". By this I mean one format has a field called "data" that is a JSON representation of data I need. The other format also has a field called "data" but the field is just a compressed string and I believe this is what causes the issue I'm having.
When dataflow reads the field from the source, the JSON gets turned into a non-json format through some form of serialization because when the projection gets imported, the data field is being read as a string and not a complex object. (Example below) I suspect this is because the two "data" fields have the same name.
I unfortunately cannot change how the data is being stored in CosmosDB so I cannot change one of the field names to another value.
Is there any way to prevent this, I would like to keep it in JSON format with quotes and colons etc, instead of what I have below.
Example of data in CosmosDB:
{
"Name": "sample",
"ValueInfo": [
{
"Field1": "foo",
"Field2": "bar"
}
]
}
How it looks in ADF
{
Name=sample,
ValueInfo=[
{
Field1=foo,
Field2=bar
}
]
}
I have a JSON like
{
"campaign_key": 316,
"client_key": 127,
"cpn_mid_counter": "24",
"cpn_name": "Bopal",
"cpn_status": "Active",
"clt_name": "Bopal Ventures",
"clt_status": "Active"
}
Expected output
1st JSON :
{
"campaign_key": 316,
"client_key": 127,
"cpn_mid_counter": "24",
"cpn_name": "Bopal",
"cpn_status": "Active"
}
2nd JSON:
{
"clt_name": "Bopal Ventures",
"clt_status": "Active"
}
How do I acheive this by using NIFI? Thanks.
You can do what 'user' had said. The not-so-good thing about that approach is, if you number of fields are increasing, then you are required to add that many JSON Path expression attributes to EvaluateJsonPath and subsequently add that many attributes in ReplaceText.
Instead what I'm proposing is, use QueryRecord with Record Reader set to JsonTreeReader and Record Writer set to JsonRecordSetWriter. And add two dynamic relationship properties as follows:
json1 : SELECT campaign_key, client_key, cpn_mid_counter, cpn_name, cpn_status FROM FLOWFILE
json2 : SELECT clt_name, clt_status FROM FLOWFILE
This approach takes care of reading and writing the output in JSON format. Plus, if you want to add more fields, you just have add the field name in the SQL SELECT statement.
QueryRecord processor lets you execute SQL query against the FlowFile content. More details on this processor can be found here
Attaching screenshots
Karthik,
Use EvaluateJsonPath processor to get those all json Values by using its keys.
Example: $.campaign_key for gets compaign key value and $.clt_name for get clt name.
Like above one you can get all jsons.
Then use ReplaceText Processor for convert single json into two jsons.
{"Compaign_Key":${CompaignKey},...etc}
{"Clt_name":${clt_name}}
It will convert single json into two jsons.
Hope this helpful and let me know if you have issues.
Is there another way to track what is unmarshalled than write own reverter for each field?
I'm updating my local data based on json message and my problem is (simplified):
I'm expecting json like
{ "items": [ { "id":1, "name":"foobar", "price":"12.34" } ] }
which is then unmarshaled to class TItems by
UnMarshaller.TryCreateObject( TItems, TJsonObject( OneJsonElement ), TargetItem )
My problem is that I can't make difference between
{ "items": [ { "id":1, "name":"", "price":"12.34" } ] }
and
{ "items": [ { "id":1, "price":"12.34" } ] }
In both cases name is blank and i'd like to update only those fields that are passed on json message. Of course I could create a reverted for each field, but there are plenty of fields and messages so it's quite huge.
I tried to look REST.Jsonreflect.pas source, but couldn't make sense.
I'm using delphi 10.
In Rest.Json unit there is a TJson class defined that offers several convenience methods like converting objects to JSON and vice versa. Specifically, it has a class function JsonToObject where you can specify options like for example ignore empty strings or ignore empty arrays. I think the TJson class can serve you. For unmarshalling complex business objects you have to write custom converters though.
Actually, my problem was finally simple to solve.
Instead of using TJSONUnMarshal.tryCreateObject I use now TJSONUnMarshal.CreateObject. First one has object parameters declared with out modifier, but CreateObject has Object parameter var modifier, so I was able to
create object, initalize it from database and pass it to CreateObject which only modifies fields in json message.
I am building an iOS app in which one of my API calls returns a large JSON blob that I load into a JSON object using SwiftyJSON. For example, it looks something like this
{
"data": {
"name":"object name",
"id":1,
"description":"short description of object",
"type":"type",
"runs":[
],
}
As part of the app the user can modify things like the name, but the API endpoint for the PATCH call needs to have the runs key removed. Does anyone know how to take a SwiftyJSON JSON object and create a new one that has a subset of keys. For example, I want the JSON blob to look like
{
"data": {
"name":"object name",
"id":1,
"description":"short description of object",
"type":"type"
}
I have spent many hours trying various things with no luck. Any help would be greatly appreciated.
You can grab the associated dictionaryValue from your SwiftyJSON object and use the removeValueForKey method.
Of course it means you have to assign the dictionaryValue to a variable, you can't remove a key in the SwiftyJSON object itself.
Example:
var dataDict = json["data"].dictionaryValue
dataDict.removeValueForKey("runs")
Note: the removeValueForKey method name is a bit misleading; it will remove the key, not just the value.
I've iOS project which is using RestKit 0.21.0 component responsible to get, parse and store in Core Data responses from remote server. In one of the backend JSON response I have something like that:
"response": [
{
"id": 1,
"start_time": "10:00:00",
"end_time": "14:00:00",
"name": "Object name",
"occurrences": [
"2013-09-13T14:00:00",
"2013-09-20T14:00:00",
"2013-09-27T14:00:00"
]
},
.
.
.
]
Generally I'm able to parse and store in Core Data received objects. I've only problem with nested array occurrences.
Do you have any advices how should I properly parse and store this collection?
I guess you want to map it to dates. To do that you generally need a container. You can also simply map to an array of strings and post-process.
1) Array of strings:
Just add an NSArray property to your destination object and map occurrences to it. This would be a transformable attribute in Core Data (could be transient). Now you can iterate the array and create the dates (could be done in willSave).
2) Relationship to dates:
Create a new entity, call it Occurrence. It has a single date property. Use a 'nil' keypath mapping to create instances of this Occurrence entity and map each of the dates to a new instance (the conversion to NSDate will be done for you). You have no identity so your only option would be to use the date as the unique identifier.