I am using MongoDB Realm Webhooks to allow for my react app to fetch data from my MongoDB Database. Everything works fine, however the data that I receive is not raw JSON, in the sense that every integer field in the object has an additional property which shows its data type. To better clarify this, I will show an example of the data that is returned:
stats: {
draws: {$numberInt: "1"}
games: {$numberInt: "271"}
goals: {$numberInt: "417"}
losses: {$numberInt: "23"}
}
Is there a way where I am able to parse the data so it can be formatted without the datatype? For example:
stats: {
draws: 1,
games: 271,
goals: 417,
losses: 23,
}
This would improve code readability for my frontend. I've tried to flatten the object manually using an object flatten function, but what I am dealing with is a large nested object, so it is difficult.
Try BSON.Binary.fromText or similar functions. There's not a lot of documentation of the BSON utilities
Related
I'm consuming a Kafka topic published by another team (so I have very limited influence over the message format). The message has a field that holds an ARRAY of STRUCTS (an array of objects), but if the array has only one value then it just holds that STRUCT (no array, just an object). I'm trying to transform the message using Confluent KSQL. Unfortunately, I cannot figure out how to do this.
For example:
{ "field": {...} } <-- STRUCT (single element)
{ "field": [ {...}, {...} ] } <-- ARRAY (multiple elements)
{ "field": [ {...}, {...}, {...} ] <-- ARRAY (multiple elements)
If I configure the field in my message schema as a STRUCT then all messages with multiple values error. If I configure the field in my message schema as an ARRAY then all messages with a single value error. I could create two streams and merge them, but then my error log will be polluted with irrelevant errors.
I've tried capturing this field as a STRING/VARCHAR which is fine and I can split the messages into two streams. If I do this, then I can parse the single value messages and extract the data I need, but I cannot figure out how to parse the multivalue messages. None of the KSQL JSON functions seem to allow parsing of JSON Arrays out of JSON Strings. I can use EXTRACTJSONFIELD() to extract a particular element of the array, but not all of the elements.
Am I missing something? Is there any way to handle this reasonably?
In my experience, this is one use-case where KSQL just doesn't work. You would need to use Kafka Streams or a plain consumer to deserialize the event as a generic JSON type, then check object.get("field").isArray() or isObject(), and handle accordingly.
Even if you used a UDF in KSQL, the STREAM definition would be required to know ahead of time if you have field ARRAY<?> or field STRUCT<...>
I finally solved this in a roundabout way...
First, I created an initial stream reading the transaction as a stream of bytes using KAFKA format instead of JSON format. This allows me to put a filter conditional filter on the data so I can fork the stream into a version for the single (STRUCT) variation and a version for the multiple (ARRAY) variation.
The initial stream looks like:
CREATE OR REPLACE STREAM `my-topic-stream` (
id STRING KEY,
data BYTES
)
WITH (
KAFKA_TOPIC='my-topic',
VALUE_FORMAT='KAFKA'
);
Forking that stream looks like this with a second for a multiple version filtering for IS NOT NULL:
CREATE OR REPLACE STREAM `my-single-stream`
WITH (
kafka_topic='my-single-topic'
) AS
SELECT *
FROM `my-topic-stream`
WHERE JSON_ARRAY_LENGTH(EXTRACTJSONFIELD(FROM_BYTES(data, 'utf8'), '$.field')) IS NULL;
At this point I can create a schema for both variations, explode field, and merge the two streams back together. I don't know if this can be refined to be more efficient, but this successfully processes the transactions as I wanted.
For example,
if the data in kafka toipc looks like this
{
"header": {
"name": "jake"
},
"body": {
"Data":"!#$%&&"
}
}
So how do I read the value "!#$%&&" from my consumer application? I need to process the data once I get that data
You'll need to consume the data using String Serde, JSON Serde, or define your own.
If you define your own, then you'd call value.getBody().getData(), like any other Java Object, where value is the argument from mapValues, peek, filter, etc. Kafka Streams DSL
For the others, the answer will depend on what JSON library you're using, but the answer isn't unique to Kafka, so read that library's documentation on parsing strings.
Here's one example of consuming using String Serde - https://github.com/confluentinc/kafka-streams-examples/blob/7.1.1-post/src/main/java/io/confluent/examples/streams/JsonToAvroExample.java#L118
I am writing a Swift program to interact with our backend Database. These database tables can change format, and each row is from DynamoDB and is represented by a JSON object. Within the program, it will grab the cache from the server which contains the Database structures and then the current contents of the database. I don't want to hardcode the database structures into the app as that would obviously make this too very hard to maintain. What is the best way to read in these rows into a generic dictionary object with dynamic names that can hold any of the JSON types?
One way you can handle parsing dynamic JSON is to have a key or a set of keys that gives information about which type of model to decode. For example, you can have a key type in your JSON objects:
{
...
"type": "some type"
...
}
and an intermediate model:
struct IntermediateModel: Codable {
let type: String // or enum
}
you first decode all your JSON objects to the intermediate model and then decode them to the actual model type based on the value of type.
I have written some helper protocols that make this easier here.
I have this kind of data in a Kafka Topic:
{..., fields: { "a": "aval", "b": "bval" } }
If I create a Kafka Engine table, I get an error when using a field definition like this:
fields String
because it (correctly) doesn't recognize it as a String:
2018.07.09 17:09:54.362061 [ 27 ] <Error> void DB::StorageKafka::streamThread(): Code: 26, e.displayText() = DB::Exception: Cannot parse JSON string: expected opening quote: (while read the value of key fields): (at row 1)
As ClickHouse does not currently have a Map or JSONObject type, what would be the best way to work over it, provided I don't know in advance the name of the inner fields ("a" or "b" in the example - so I cannot see Nested structures helping)?
Apparently, at the moment ClickHouse does not support complex JSON parsing.
From this answer in ClickHouse Github:
Clickhouse uses quick and dirty JSON parser, which does not how to read complex deep structures. So it can't skip that field as it does not know where that nested structure ends.
Sorry. :/
So you should preprocess your json with some external tools, of you can contribute to Clickhouse and improve JSON parser.
I'm trying to parse a JSON from a rest service. This service does not put the data into the format that I think ItemFileReadStore wants, but I cannot change it. Everything I have found in the dojo library for reading JSON data requires an identifier, which my data does not have. This is the JSON data:
{"ChannelReadResponse":[
{"Event": {"#entityOrigin":"System","#entityId":"0x080e00000136ad8986520af104608052","Name":"Untitled","SymbolCode":"OHVPEV---------","TimeObserved":"2012-04-13T21:09:49.207Z","CreatedUser":"Helpdesk","ModifiedUser":"Helpdesk","CreatedTime":"2012-04-13T21:09:49.207Z","ModifiedTime":"2012-04-17T15:51:12.496Z"},
{"#entityOrigin":"System","#entityId":"0x080e00000136bb54ec770af104608028","Name":"My Event","SymbolCode":"OHVPE----------","Severity":"SIGACT","Outcome":"Effective","TimeObserved":"2012-04-16T14:34:29.796Z","CreatedUser":"Helpdesk","ModifiedUser":"Helpdesk","CreatedTime":"2012-04-16T14:34:29.796Z","ModifiedTime":"2012-04-17T15:50:52.499Z"}
]
,"Channel":{"#writable":"false","#connected":"true","#entityId":"0x080e00000136ad8500760af104608064","Name":"Ozone",
"Members":{"Member":[{"#entityOrigin":"System","#entityRef":"0x080e00000136ad8986520af104608052"},{"#entityOrigin":"System","#entityRef":"0x080e00000136bb54ec770af104608028"}]
}}},
{"Event": {"#entityOrigin":"System","#entityId":"0x080e00000136bc3c92d80af104608042","Name":"From2","SymbolCode":"OHVPE----------","TimeObserved":"2012-04-16T19:43:03.150Z","CreatedUser":"Helpdesk","ModifiedUser":"Helpdesk","CreatedTime":"2012-04-16T19:43:03.150Z","ModifiedTime":"2012-04-16T19:43:03.150Z"},
"Channel": {"#writable":"false","#connected":"true","#entityId":"0x080e00000136bc3c92d80af104608034","Name":"Ozone2",
"Members":{"Member":{"#entityOrigin":"System","#entityRef":"0x080e00000136bc3c92d80af104608042"}}}
]}
]}
Is there any way to work with this data? I specifically want all the Events out of it.
Just massage it into the form that the store wants. For example, if you get the data back in a variable called 'data', you could easily just do:
var json = {
identifier: "#entityId",
items: data
};
Then just use the json object in the store.
I can only think of converting your JSON data to JavaScript Object Literal and then add the ID and Name to the JavaScript Object Literal....then convert it into JSON before passing it to your Dojo Store.
I have faced similar issue but i had the luxury to change my service to return JSON with Identifier and Name. I haven't tried what I wrote above.