Are JSON responses ever incomplete because of server errors, or are they designed to fail loudly? Are there any special concerns for transferring very large sets of data over JSON, and can they be mitigated? I'm open to any suggestions.
Transferring JSON over HTTP is no different than transferring any bytes over HTTP.
Yes, server errors can result in incomplete transfers. Imagine turning your server off half way through a transfer. This is true of any network transfer. Your client will fail loudly if there is such an error. You might get a connection time out or an error status code. Either way you will know about it.
There is no practical limit to the amount of data you can transfer as JSON over HTTP. I have transferred 1GB+ of JSON data in a single HTTP request. When making a large transfer you want to be sure to use a streaming API on the server side. Which is to say write to the output stream of the HTTP response while reading the data from your db, rather than reading your data from the DB into RAM entirely and then encoding it to JSON and writing it to the output. This way your client can start processing the response immediately, plus your server wont run out of memory.
Related
I have a web service which handles a significant volume of traffic. This traffic can be in range of millions per minute. The service is hosted on AWS EC2 behind an ELB and uses HTTP APIs. This leads to a good chunk of AWS bill comprising of Data Transfer fees. The Data Transfer Out component is mostly higher since the 50% responses from web service are somewhat large and encoded as JSON in addition to SSL negotiations.
Now gRPC payloads are smaller in size compared to similar data represented as JSON due to binary serialization. So it is possible to save upon the data transfer costs by switching from HTTP APIs to gRPC?
I couldn't find any benchmark/article anywhere correlating AWS Data Transfer costs with HTTP APIs/gRPC services. Even 5-10% savings would be beneficial.
PS: Here the clients accessing the web service are also mine. So it is possible for to make changes on both server side and client side.
Maybe, but probably not. It depends on your actual data.
If you're using HTTP for communications, then there are two components of overall message size: HTTP headers and response body. If the headers represent a significant portion of your overall message size, then it makes more sense to get rid of them by using an alternative layer-7 protocol, such as WebSockets.
If the headers aren't significant, then it depends on what your actual message content is. That's because Protocol Buffers, which is used by gRPC, performs essentially two optimizations:
Replacement of field names with a one- or two-byte value. This can be a big savings, as long as your JSON response doesn't frequently use the same field names (ie, repeated objects). If it does, then using GZip encoding will reduce the average cost of a field name down to somewhere around 5 bytes (my observation with large files, YMMV).
Storage of numeric values in fewer than their normal number of bits. If your message content consists of arrays of numbers, this will be a huge win. If it's mostly text, you won't see much benefit, because the same byte sequence will have to be sent in either case.
Personally, I think switching to WebSockets would be the best first step. That assumes, of course, that these messages are coming from a relatively small number of clients. If every message is from a different client, you won't save anything.
We have a backend application which has to send telemetry data to an event hub. All data have to be serialized into JSON and compressed.
Should we collect all serialized objects into a single newline-delimited JSON or it is better to use one EventData wrapper per object and send it as a batch? Probably compression will work better with newline-delimited json. But will ASA be able to process it?
Asa supports gzip and deflate compression. Every eventhub message can be up to 256 kb including metadata. On processing side, every message has some overhead. So, for same number of records, smaller number of eventhub messages is better. However, this usually means some buffering on the send side. Based on your overall latency requirements and memory footprint requirements on sender, you should batch multiple records into every eventhub message, with compression.
We are up to build API which deals with HBase table. Let's say the API has to methods: api.com/get/to get something out of HBase and api.com/put/ to put a matrix into HBase. We want to put and get matrices of size 200mb.
We can't come to conclusion of how to send data to this API. Do you think it sounds OK to send HTTPS request and represent the 200mb input matrix as JSON and put it to POST parameter?
Can't find any best practices for this case. Thank you.
The payload limits depends of the client and server RAM size and processor.
Teorically there is no limit in the standard (RFC 2616). However is not a good idea to construct a big payload because it probably fails because of one of this reasons:
lost packets on data transmission
limits on server side
limits on client side
The best is try to split your 200mb input matrix in smaller inputs and make multiple requests.
I need to separate Server/Client(now it is one, but have to divided) Program.
So Here is my plan
server : queries to database. send data as type of JSON.
client : receives json data from server.
The thing I worry about is data size. I expect server will send data which almost sizes 200MB. Is it proper size to transfer with http protocol? or should I make this as a file and send via FTP?(but I expect that client will not open extra port for this :<)
P.S
Is there any reference what is the proper size of json data?
Thanks ahead.
There is no limit for the amount of data that can be transmitted over HTTP. HTTP also doesn't care what kind of data you are sending/receiving. It can be audio, video or JSON, so you should be safe.
Moreover, HTTP servers and clients can easily use gzip to make the requests/response more compact, and since JSON is text based, the content can be compressed quite a lot.
In short, there is no problem with your approach.
I dont know whether i am asking valid question
My problem is i want to send large data from one application to another for which i am using JSONP to pass data to handler file which stores it in database.As the data is large i am dividing it in chunks and passing the packets in loop, the more the number of packets the more time it takes to pass complete data which ultimately results in performance issue.(FYI my web server is bit slow)
Is there any way by which i can compress my data and send it at a time rather than sending it in packets.
OR
Any other way by which i can pass my data(large data) from application to another.
Need this ASAP
Thanks in advance.