I am working on a (.NET) REST API which is returning some JSON data. The consumer of the API is an embedded client. We have been trying to establish the structure of the JSON we will be working with. The format the embedded client wants to use is something I have not seen before in working with JSON. I suggested that it is no "typical" JSON. I was met with the question "Where is 'typical' JSON format documented"?
As an example of JSON I "typically" see:
{
"item" : {
"users": [ ... list of user objects ... ],
"times": [ ... list of time objects ...],
}
}
An example of the non-typical JSON:
{
"item" : [
{
"users": [ ... list of user objects ... ]
},
{
"times": [ ... list of time objects ...]
},
]
}
In the second example, item contains an array of objects, which each contain a property whose value is an array of entities. This is valid JSON. However, I have not encountered another instance of JSON that is structured this way when it is not an arbitrary array of objects but is in fact a set list of properties on the "item" object.
In searching json.org, stackoverflow.com and other places on the interwebs I have not found any guidelines on why the structure of JSON follows the "typical" example above rather than the second example.
Can you provide links to documentation that would provide recommendations for one format or the other above?
Not a link, but just straightforward answer: Items are either indexed (0, 1, 2, ...) or keyed (users, times). No matter what software you use, you can get at indexed or keyed data equally easily and quickly. But not with what you call "non-typical" JSON: To get at the users, I have to iterate through the array and find one dictionary that has a key "users". But there might be two or more dictionaries with that key. So what am I supposed to do then? If you use JSON schema, the "non-typical" JSON is impossible to check. In iOS, in the typical case I write
NSArray* users = itemDict [#"users"];
For the non-typical JSON I have to write
NSArray* users = nil;
for (NSDictionary* dict in itemArray)
if (dict [#"users"] != nil)
users = dict [#"users"];
but that still has no error checking for multiple dicts with the key "users". Which is an error that in the first case isn't even possible. So just tell them what the are asking for is rubbish and creates nothing but unnecessary work. For other software, you probably have the same problems.
Related
I'm consuming a Kafka topic published by another team (so I have very limited influence over the message format). The message has a field that holds an ARRAY of STRUCTS (an array of objects), but if the array has only one value then it just holds that STRUCT (no array, just an object). I'm trying to transform the message using Confluent KSQL. Unfortunately, I cannot figure out how to do this.
For example:
{ "field": {...} } <-- STRUCT (single element)
{ "field": [ {...}, {...} ] } <-- ARRAY (multiple elements)
{ "field": [ {...}, {...}, {...} ] <-- ARRAY (multiple elements)
If I configure the field in my message schema as a STRUCT then all messages with multiple values error. If I configure the field in my message schema as an ARRAY then all messages with a single value error. I could create two streams and merge them, but then my error log will be polluted with irrelevant errors.
I've tried capturing this field as a STRING/VARCHAR which is fine and I can split the messages into two streams. If I do this, then I can parse the single value messages and extract the data I need, but I cannot figure out how to parse the multivalue messages. None of the KSQL JSON functions seem to allow parsing of JSON Arrays out of JSON Strings. I can use EXTRACTJSONFIELD() to extract a particular element of the array, but not all of the elements.
Am I missing something? Is there any way to handle this reasonably?
In my experience, this is one use-case where KSQL just doesn't work. You would need to use Kafka Streams or a plain consumer to deserialize the event as a generic JSON type, then check object.get("field").isArray() or isObject(), and handle accordingly.
Even if you used a UDF in KSQL, the STREAM definition would be required to know ahead of time if you have field ARRAY<?> or field STRUCT<...>
I finally solved this in a roundabout way...
First, I created an initial stream reading the transaction as a stream of bytes using KAFKA format instead of JSON format. This allows me to put a filter conditional filter on the data so I can fork the stream into a version for the single (STRUCT) variation and a version for the multiple (ARRAY) variation.
The initial stream looks like:
CREATE OR REPLACE STREAM `my-topic-stream` (
id STRING KEY,
data BYTES
)
WITH (
KAFKA_TOPIC='my-topic',
VALUE_FORMAT='KAFKA'
);
Forking that stream looks like this with a second for a multiple version filtering for IS NOT NULL:
CREATE OR REPLACE STREAM `my-single-stream`
WITH (
kafka_topic='my-single-topic'
) AS
SELECT *
FROM `my-topic-stream`
WHERE JSON_ARRAY_LENGTH(EXTRACTJSONFIELD(FROM_BYTES(data, 'utf8'), '$.field')) IS NULL;
At this point I can create a schema for both variations, explode field, and merge the two streams back together. I don't know if this can be refined to be more efficient, but this successfully processes the transactions as I wanted.
There are many articles about BizTalk JSON encoder...
I am trying to produce JSON for 3rd party software wanting root level array, like so:
[
{
"property" : "value"
},
{
"property" : "value"
}
]
I am trying to control the output using schema, but I am not able to specify minOccurs and maxOccurs on the root node. I have also tried "Group Max/Min Occurs", with no difference.
Is it not possible to do this?
What about the old newtonsoft hack for adding Array attribute to the output XML? (I have tried this as well, but failed...)
This is similar to BizTalk 2013 - decode JSON array
As per the answers on the other question, you can't have an array at the root node of an XML schema. That one is for receiving a JSON payload with an array at the root, rather then sending one. But your options are similar, you would need to either have a custom pipeline component after the JSON encoder that removes the root or a custom pipeline component that produces the JSON the way you want.
The json data structure for jstree is define in https://github.com/vakata/jstree, here is an example
[ { "text" : "Root node", "children" : [ "Child node 1", "Child node 2" ] } ]
Notably it says
The children key can be used to add children to the branch, it should
be an array
However later on in section Populating the tree using AJAX and lazy loading nodes it shows to use set children to false to indicate when a child has not be processed
[{
"id":1,"text":"Root node","children":[
{"id":2,"text":"Child node 1","children":true},
{"id":3,"text":"Child node 2"}
]
}]
So here we see children used as both as an array and as a boolean
I am using jstree as an example because this is where I encountered the issue, but my question is really a general json question. My question is this, is it valid JSON for the same element in json to be two different types (an array and a boolean)
Structure wise, both are valid JSON packets. This is okay, as JSON is somewhat less stricter than XML(with a XSD or a DTD). As per: https://www.w3schools.com/js/js_json_objects.asp,
JSON objects are surrounded by curly braces {}.
JSON objects are written in key/value pairs.
Keys must be strings, and values must be a valid JSON data type (string, number, object, array, boolean or null).
Keys and values are separated by a colon.
Each key/value pair is separated by a comma.
Having said that, if the sender is allowed to send such JSONs, only caveat is that server side will have to handle this discrepancy upon receiving such different packets. This is a bad-looking-contract, and hence server might need to do extra work to manage it. Server side handling of such incoming JSON packets can become tricky.
See: How do I create JSON data structure when element can be different types in for use by
You could validate whether a JSON is okay or not at https://jsonlint.com/
See more about JSON in this answer: https://stackoverflow.com/a/4862511/945214
It is valid Json. JSON RFC 8259 defines a general syntax but it contains nothing that would allow a tool to identify that two equally named entries are meant to describe the same conceptual thing.
The need to have a criteria to check two JSON structures for instance equality has been one motivation to create something like Json Schema.
I also think it is not too unusual for javascript to provide this kind of mixed data. Sometimes it might help to explicitly convert the javascript object to JSON. Like in JSON.stringify(testObject)
A thing for json validation
https://www.npmjs.com/package/json-validation
https://davidwalsh.name/json-validation.
I have been teaching myself how to parse JSON using SwiftyJSON and the fab PokeApi.co I thought I was getting the hang of it, until I tried to get an array back of a Pokemon's different types. The JSON returned looks like this:
...
],
"base_experience": 155,
"types": [
{
"slot": 2,
"type": {
"url": "https://pokeapi.co/api/v2/type/3/",
"name": "flying"
}
},
{
"slot": 1,
"type": {
"url": "https://pokeapi.co/api/v2/type/1/",
"name": "normal"
}
}
]
}
I need to grab an array of the value for "name" within "types" and then "type". I tried lots of different suggestions from here, the SwiftyJSON guide on their Github and various other sources, and the only thing I could get to work is this:
let typesArray = json["types"].arrayValue.map({$0["type"]["name"].stringValue})
for item in typesArray {
self.currentPokemon.pokemonTypes.append(item.firstUppercased)
print("Added \(item)")
}
I am happy that I have got it to work, but I am desperately trying to get my head around parsing JSON and this just doesn't make sense to me! Could anybody please explain to me what is going on? Is there a clearer/ more readable way of doing this?
Thank you for your time!
JSON
First things first... I think you need to make sure you really understand JSON before you can wrap your head around what you are doing here. JSON objects are essentially a Key / Value object. The Key is a string and the value can be many things: array of more JSON objects, or even as simple as a string.
Best Practices
So one thing I like to do when I have to parse through JSON for any response is create an enum of strings that will have all of the "Keys" in the json. Note that if you do not assign a calue to any case and the type is String when you do pokemonApiParams.name.rawValue it will automatically return the stringified version of your case!
enum pokemonApiParams: String {
case baseExperience
case types
case slot
case type
case url
case name
}
Now when you are working with your JSON you can use this enum to help make things clearer. This is especially nice if your API response ever changed a key. You could just change the enum value and everything would work! Another reason why this is a "best practice" is it takes out these "magic strings" from your code and keeps everything in one logistical place that you can refer to later!
// This would give you 155 in your example
json[pokemonApiParams.baseExperience.rawValue]
Working through your example
Now to the part you were having trouble understanding.
First let's reconstruct your "types array" piece by piece...
let typesArray = json[pokemonApiParams.types.rawValue]
This guy creates a json object that contains the array of "types" in your response. Basic SwiftyJSON objects are great but you can be a little more specific here to get some more usage, hence why your example uses .arrayValue.
let typesArray = json[pokemonApiParams.types.rawValue].arrayValue
Now we have a JSON array object and we could go through it using various different looping functions. However you use the .map function next. This guy is super useful once you get used to it and use it some more!
Understanding .map
Basically we can map each element in our array to be something else, in this case we are using a short hand to go through the array and get only the "name" attribute of each type. So we are mapping from a [JSON] -> String.
The $0 is the shorthand I was talking about. Imagine this as your index in a for loop. The map function will walk through the entirety of the array and $0 is the object in the array that is being looked at.
Check out the map documentation too!
Putting it all together
Looking at your response each types object has a slot and a type.
So back to your example (but using our enum!):
let typesArray = json[pokemonApiParams.types.rawValue].arrayValue
.map {
$0[pokemonApiParams.type][pokemonApiParams.name.rawValue].stringValue
}
Now your typesArray is an array of strings. Another way to write this (which would be a little clearer and maybe easier to understand) is:
let typesArray = json[pokemonApiParams.types.rawValue].arrayValue
.map { (types: JSON) -> String in
types[pokemonApiParams.type.rawValue][pokemonApiParams.name.rawValue].stringValue
}
This is clearer because we have to specify our object (types) instead of using the $0 shorthand and we specify what we are returning (String).
Hopefully this is all correct and clear!
Happy coding and go catch 'em all!
I am tinkering with firebase and curious about the data structure. Browsing to my database, firebase allows me to modify the structure and data in my database. But it seems that firebase only supports objects (and dictionaries for lists).
I want to know if arrays are supported. I would also like to know if dictionary items can be named with integers - the firebase interface only inserts strings as names which makes me concerned about ordering records.
Here is a sample of json created through firebase interface:
{
"dg":{
"users":{
"rein":{
"searches":{
"0":{
"urls":"http://reinpetersen.com,http://www.reinpetersen.com",
"keyphrases":"rein petersen,reinsbrain,programmer turned kitesurfer"
}
}
},
"jacqui":{
"searches":{
"0":{
"urls":"http://www.diving-fiji.com,http://diving-fiji.com",
"keyphrases":"diving marine conservation, diving fiji"
}
}
}
},
"crawl_list":{
"1":{
"urls":"http://www.diving-fiji.com,http://diving-fiji.com",
"keyphrases":"diving marine conservation, diving fiji"
},
"0":{
"urls":"http://reinpetersen.com,http://www.reinpetersen.com",
"keyphrases":"rein petersen,reinsbrain,programmer turned kitesurfer"
}
}
}
}
Obviously, for my lists, I want the dictionary item names to be integers so i can ensure sorting is correct.
You can save arrays into Firebase. For example:
var data = new Firebase(...);
data.set(['x', 'y', 'z']);
Javascript Arrays are essentially just objects with numeric keys. When retrieving data, we automatically detect when a Firebase object has only numeric keys, and we return an array if that is the case.
Note that for storing a list of data to which many people can append, an array is not a good choice, as multiple people writing to the same index in the array can cause conflicts. Instead, we have a "push" function which creates a chronologically-ordered unique ID for your data.
Also, if you're intending to use the array as a way of ordering data, there's a better way to do that using our priorities. See the docs.
The Firebase docs have a pretty good section on how to order your data: Ordered Data.
Just like JSON fields, Firebase fields can only be named with strings. It sounds like what you're looking for is setWithPriority(), which attaches sortable priority data to your fields, or push(), which is guaranteed to give your fields unique names, ordered chronologically. (More on lists and push() here.)
You can also push() or set() arrays. For example,
new Firebase("http://gamma.firebase.com/MyUser").push(["cakes","bulldozers"]);
results in a tree like you'd expect, with MyUser receiving a uniquely named child who has children "0":"cakes" and "1":"bulldozers".