I am trying with no success to insert a byte array in a postgresql json type as one of the json columns. is it possible? does anyone have an example?
You can't, at least natively. JSON only allows 3 basic primative data types: number, string, and boolean. Everything else must be serialized to a string or number.
This means that you have three basic options:
Serialize to hexadecimal. The advantage is that it becomes easy to turn into a bytea in PostgreSQL if you need it.
Serialize to base64. The advantage here is that it uses up less space.
Serialize to a number array. This is not preferred in my view since it is hard to constrain each number to between 0 and 255.
Related
I'm writing a library to deserialize a subset of JSON into predefined Python types.
I want to deserialize arbitrary JSON into an object that quacks like serde-json's Value. However, I don't want it to deserialize into String's, Number's and Bool's - instead when the deserializer hits one of these I would prefer it simply keeps a reference to the respective byte string so I can efficiently (i.e. without the additional type conversion) parse the byte strings into the correct arbitrary Python types. Something like this:
use serde::Deserialize;
use serde_json::value::RawValue;
use serde_json::Map;
#[derive(Deserialize)]
pub enum MyValue<'a> {
Null,
Bytes(&'a RawValue),
Array(Vec<MyValue<'a>>),
Object(Map<String, MyValue<'a>>),
}
This will require writing a lot of traits so that it behaves like Value, and I'm not even sure if it won't just ignore deserializing the structural parts and put everything into a RawValue.
What is the cleanest way to do this?
suppose we are developing an application that pulls Avro records from a source
stream (e.g. Kafka/Kinesis/etc), parses them into JSON, then further processes that
JSON with additional transformations. Further assume these records can have a
varying schema (which we can look up and fetch from a registry).
We would like to use Spark's built in from_avro function, But it is pretty clear that
Spark from_avro wants you to hard code a >Fixed< schema into your code. It doesn't seem
to allow the schema to vary row by incoming row.
That sort of makes sense if you are parsing the Avro to Internal row format.. One would need
a consistent structure for the dataframe. But what if we wanted something like
from_avro which grabbed the bytes from some column in the row and also grabbed the string
representation of the Avro schema from some other column in the row, and then parsed that Avro
into a JSON string.
Does such built-in method exist? Or is such functionality available in a 3rd party library ?
Thanks !
I have a little dilema. I have a backend/Frontend Application that comunicates with a JSON based REST Api.
The backend is written in PHP(Symfony/jmsserializer) and the Frontend in Dart
The communication between these two has a little Problem.
For most List Data the backend responds with a JSON like
[{"id":1,...},{"id:2,....}]
But for some it responds with
{"0":{"id":1,...}, "1":{"id:2,....}}
Now my Question is should the backend respond with the later at all or only with the first?
Problem
You usually have a list of objects. You sometimes get an object with sub-objects as properties.
Underlying issue
JS/JSON-Lists are ordered from 0 upwards which means that if you have PHP-Array which does not respect this rule json_encode will output a JS/JSON-Object instead using the numeric indices as keys.
PHP-Arrays are ordered maps which have more features that the JSON-Lists. Whenever you're using those extra features you won't be able to translate directly into JSON-Lists without loosing some information (ordering, keys, skipped indices, etc.).
PHP-Arrays and JSON-Objects on the other hand are more ore less equivalent in terms of features and can be correctly translated between each other without any loss of information.
Occurence
This happens if you have an initial PHP-Array of values which respects the JS/JSON-List rules but the keys in the list of objects are modified somehow. For example if you have a custom indexing order {"3":{}, "0":{}, "1":{}, "2":{}} or if you have (any) keys that are strings (ie. not numeric).
This always happens if you want to use the numeric id of the object as the numeric index of the list {"123":{"id": 123, "name": "obj"}} even if the numeric ids are in ascending order... so long as they are not starting from 0 upwards it's not a json-list it's a json-object.
Specific case
So my guess is that the PHP code in the backend is doing something like fetching a list of objects but its modifying something about it like inserting them by (string) keys into the array, inserting them in a specific order, removing some of them.
Resolution
The backend can easily fix this by using array_values($listOfObjects) before using json_encode which will reindex the entire list by numeric indices of ascending value.
Arrays and dictionaries are two separate types in JSON ("array" and "object" respectively), but PHP combines this functionality in a single array type.
PHP's json_encode deals with this as follows: an array that only contains numeric keys ($array = ['cat', 'dog']) is serialized as JSON array, an associative array that contains non-numeric keys ($array = ['cat' => 'meow', 'dog' => 'woof']) is serialized as JSON object, including the array's keys in the output.
If you end up with an associative array in PHP, but want to serialize it as a plain array in JSON, just use this to convert it to a numerical array before JSON encoding it: $array = array_values($array);
I am not asking for any libraries to do so and I am just writing code for bson_to_json and json_to_bson.
so here is the BSON specification.
For regular double, doc, array, string, it is fine and it is easy to convert between BSON and JSON.
However, for those particular objects, such as
Timestamp and UTC:
If convert from JSON to BSON, how can I know they are timestamp and utc?
Regex (string, string), JavaScript code with scope (string, doc)
their structures have multiple parts, how can I present the structures in JSON?
Binary data (generic, function, etc)`
How can I present the type of binary data in JSON?
int32 and int64
How can I present them in JSON, so BSON can know which is 32 bit or 64 bit?
Thanks
As we know JSON cannot express objects so you will need to decide how you want the stringified version of the BSON objects (field types) to be represented within the output of your ocaml driver.
Some of the data types are easy, Timestamp is not needed since it is internal to sharding only and Javascript blocks are best left out due to the fact that they are best used only within system.js as saved functions for use in MRs.
You also gotta consider that some of these fields are actually both in and out. What I mean by in and out is that some are used to specify input documents to be serialised to BSON and some are part of output document that need deserialising from BSON into JSON.
Regex is one which will most likely be a field type you send down. As such you will need to serialise your ocaml object to the BSON equivilant of {$regex: 'd', '$options': 'ig'} from /d/ig PCRE representation.
Dates can be represented in JSON by either choosing to use the ISODate string or a timestamp for the representation. The output will be something like {$sec:556675,$usec:6787} and you can convert $sec to the display you need.
Binary data in JSON can be represented by taking the data (if I remember right) property from the output document and then encoding that to base 64 and storing it as a stirng in the field.
int32 and int64 has no real definition between the two in JSON except that 64bit ints will be bigger than 2147483647 so I am unsure if you can keep the data types unique there.
That should help get you started.
I am just starting out with MongoDB and one of the things that I have noticed is that it uses BSON to store data internally. However the documentation is not exactly clear on what BSON is and how it is used in MongoDB. Can someone explain it to me, please?
BSON is the binary encoding of JSON-like documents that MongoDB uses when storing documents in collections. It adds support for data types like Date and binary that aren't supported in JSON.
In practice, you don't have to know much about BSON when working with MongoDB, you just need to use the native types of your language and the supplied types (e.g. ObjectId) of its driver when constructing documents and they will be mapped into the appropriate BSON type by the driver.
What's BSON?
BSON [bee · sahn], short for Binary JSON, is a binary-encoded
serialization of JSON-like documents.
How is it different from JSON?
BSON is designed to be efficient in space, but in some cases is not much more efficient than JSON. In some cases BSON uses even more space than JSON. The reason for this is another of the BSON design goals: traversability. BSON adds some "extra" information to documents, like length of strings and subobjects. This makes traversal faster.
BSON is also designed to be fast to encode and decode. For example, integers are stored as 32 (or 64) bit integers, so they don't need to be parsed to and from text. This uses more space than JSON for small integers, but is much faster to parse.
In addition to compactness, BSON adds additional data types unavailable in JSON, notably the BinData and Date data types.
Source: http://bsonspec.org/
MongoDB represents JSON documents in binary-encoded format so we call it BSON behind the scenes.
BSON extends the JSON model to provide additional data types such as Date and binary which are not supported in JSON also provide ordered fields in order for it to be efficient for encoding and decoding within different languages.
In other word we can say BSON is just binary JSON ( a superset of JSON with some more data types, most importantly binary byte array ).
Mongodb using as a serialization format of JSON include with encoding format for storing and accessing documents. simply we can say BSON is a binary encoded format for JSON data.
for more mongoDB Article : https://om9x.com/blog/bson-vs-json/
MongoDB represents JSON documents in binary-encoded format called BSON behind the scenes. BSON extends the JSON model to provide additional data types and to be efficient for encoding and decoding within different languages.
By using BSON encoding on top of JSON, MongoDB gets the capability of creating indexes on top of values that resides inside the JSON document in raw format. This helps in running efficient analytical queries as NoSQL system were known for having no support for Indexes.
This relatively short article gives a pretty good explanation of BSON and JSON: It talks about some of the problems with JSON, why BSON was invented, what problems it solves compared to JSON and how it could benefit you.
https://www.compose.com/articles/from-json-to-bson-and-back/
In my use case that article told me that serializing to JSON would work for me and I didn't need to serialize to BSON
To stay strictly within the boundaries of the OP question:
What is BSON?
BSON is a specification for a rich set of scalar types (int32, int64, decimal, date, etc.) plus containers (object a.k.a. a map, and array) as they might appear in a byte stream. There is no "native" string form of BSON; it is a byte[] spec. To work with this byte stream, there are many native language implementations available that can turn the byte stream into actual types appropriate for the language. These are called codecs. For example, the Java implementation of a BSON codec into the Document class from MongoDB turns objects into something that implements java.util.Map. Dates are decoded into java.util.Date. Transmitting BSON looks like this in, for example, Java and python:
Java:
import org.bson.*;
MyObject --> get() from MyObject, set() into org.bson.Document --> org.bson.standardCodec.encode(Document) to byte[]
XMIT byte[]
python:
import bson
byte[] --> bson.decode(byte[]) to dict --> get from dict --> do something
There are no to- and from- string calls involved. There is no parser. There is nothing about whitespace and double quotes and escaped characters. Dates, BigDecimal, and arrays of Long captured on the Java side reappear in python as datetime.datetime, Decimal, and array of int.
In comparison, JSON is a string. There is no codec for JSON. Transmitting JSON looks like this:
MyObject --> convert to JSON (now you have a big string with quotes and braces and commas)
XMIT string
parse string to dict (or possibly a class via a framework)
Superficially this looks the same but the JSON specification for scalars has only strings and "number" (leaving out bools and nulls, etc.). There is no direct way to send a long or a BigDecimal from sender to receiver in JSON; they are both just "number". Furthermore, JSON has no type for plain byte array. All non-ASCII data must be base64 or otherwise encoded in a way to protect it and sent as a string. BSON has a byte array type. The producer sets it, the consumer gets it. There is no secondary processing of strings to turn it back into the desired type.
How does MongoDB use BSON?
To start, it is the wire protocol for content. It also is the on-disk format of data. Because varying length types (most notably string) carry length information in the BSON spec, this permits MongoDB to performantly traverse an object (hopping field to field). Finding the object in a collection is more than just BSON including use of indexes.