I have a server and I need to upload files along with some fields from the client to the server. I have currently been using standard multipart/form-data.
I have found however that using multipart/form-data is not ideal. Objects on my server may have other objects nested within them, and thus are represented as a JSON object with other JSON objects embedded within.
I would like for the client to start making POST/PUT requests using a JSON representation exactly like it would expect in a GET request to the server, in a REST-ful manner. This way I don't have to flatten the fields which might be nested a couple layers within the JSON object in order to use multipart/form-data.
Problem is, JSON doesn't represent binary data. Multipart/form-data doesn't seem to have a way to represent fields nested within the values of other fields. But it does have much better handling of file-uploads.
I am at a loss for how to design this. Should I just have the client upload JSON with the fields encoded in base64, and take the 25% hit? Or should I have the JSON object being represented as some sort of "json" variable in a Multipart/form-data request, and have the binary files to be uploaded as another variable?
Should I just have the client upload JSON with the fields encoded in
base64, and take the 25% hit?
The hit will be 33% since 4/3=1.33.
Or should I have the JSON object being represented as some sort of
"json" variable in a Multipart/form-data request, and have the binary
files to be uploaded as another variable?
This should work.
You might also consider this approach: send all files using multipart, then get some identificators of files as a response. Put this identificators in your json and send it anyway you like. This approach might be beneficial if you have many scenarios in which you send files: you might always send them to the server with the same request, then get their identificators; after that do with them what you like.
Related
If you send plain Json formatted un-serialized text from a web api and have it be de-serialized in a C# function using the jsonConvert.DeserializeObject functional? I have a co-worker who created a web api in the cloud and he sends plain text formatted to look like un-serialized Json which I try to use C# functionality to Deserialize but when I try to convert what is sent into Model classes it fails. I am telling him that the C# JsonConvert.SerializeObject must be used or it won't work. Can someone help clarify this with me?
Your colleague is right. JSON strings need to be de-serialized to turn them into objects.
Moreover, the things you call "plain text formatted to look like JSON" are JSON strings. JSON is plain text (in UTF8, with syntax rules).
Let's say you have some data structure in your program that you want to send over the network. The network can only send series of bytes, so whatever your structure was, you need to turn it into that -- you need to serialize it. JSON is one way of doing that, e.g.
'{"example": "some data"}'
is a string containing JSON. It is serialized, it's just a string of bytes to send over the network.
On the receiving end, you need to deserialize it back into some data structure, some type of hash map or dictionary or whatever it's called in C# probably.
If what you try "fails", you could ask a much more specific question showing what you tried with what data and how exactly it failed.
I have a situation where I have to write a api to create a resource and amongst datafields that I need to accept is a string that is basically contents of a html file. As I see it I have a choice between structuring the entire thing as a json object where this field is a string field with urlencoded html string , and having the Content Type as multipart/form-data where each of the fields and the html string (UTF-8 encoded) is a part in the message.
Not using json is something I am not comfortable with as I feel violating the REST standards in not structuring the content of the entity I am about to create thus there is a loss of information for the consumers as they can't tell immediately looking at my api definition about what data to feed to it. But practically multipart/form-data handles stuff like html file content better and more efficient as I will not have to urlencode it and can control the char-encoding also.
What will be a better approach in current context and upholding RESTful principles ? Also are there other trade-offs i should be aware of ? what about parsing a json with a huge string field (~ 200 Kb)embedded?
EDIT :- I was reading some similar questions on SO and one approach that stood out was the 2-step approach of making a first call with metadata to create the entity and then upload the file as an UPDATE process to the created entity wherein we use multipart/form-data. In that context, I guess , what I am asking is how sound is an approach where I send both metadata and the file in a single api call as multipart data , where each metadata field is actually a part in the multipart message as is the file.
The canonical way to upload files to REST API is using the multipart/form-data. As W3 recommendation guide says:
The content type "multipart/form-data" should be used for submitting
forms that contain files, non-ASCII data, and binary data.
Multipart/form-data has advantages over base64 to represent binary data. Is sticked to REST/Http philosophy, and simplify the develop of API clients.
Returning values from Forms: multipart/form-data
W3 Recommendation guide
The good practice is to use multipart/form-data whenever files are uploaded to the server along with database fields. Do not send a base64 JSON string as the request to your Rest API as it might corrupt the file or degrade the performance of your application.
As far as documenting multipart/form-data Rest API for your consumers is concerned you have to force your API consumers to use the same form fields which you have predefined in your web service.
Returning Values from Forms: multipart/form-data
I started using FormData objects everywhere on the client-side, in lieu of regular form input fields, for dynamic REST posts. FormData is presented in a positive light in various tutorials, so I went with it.
However, down the line, this caused me problems when decoding the form data into my Go structs. FormData objects are sent as "multipart/form-data" (regardless of files being sent) and I believe my decoder in Go didn't convert the raw data back to string form. Eventually my SQL queries were throwing panics, as hex data was being sent in where strings should have been.
So with some adjustment, I could use FormData however I've decided to revert to the simple universal recommendation: Use "multipart/form-data" only for special cases like when sending files. Otherwise, just use regular "application/x-www-form-urlencoded".
I have a solution where I need to be able to log multiple JSON Objects to a file. Essentially doing one log file per day. What is the easiest way to write (and later read) these from a single file?
How does MongoDB handle this with BSON? What does it use as a separator between "records"?
Does Protocol Buffers, BSON, MessagePack, etc... offer compression and the record concept? Compression would be a nice benefit.
With protocol buffers you could define the message as follows:
Message JSONObject {
required string JSON = 1;
}
Message DailyJSONLog {
repeated JSONObject JSON = 1;
}
This way you would just read the file from memory and deserialize it. Its essentially the same way for serializing them as well. Once you have the file (serialized DailyJSONLog) on disk, you can easily just append serialized JSONObjects to the end of that file (since the DailyJSONLog message is very simply a repeated field).
The only issue with this is if you have a LOT of messages each day or if you want to start at a certain location during the day (you're not able to easily get to the middle (or arbitrary) of the repeated list).
I've gotten around this by taking a JSONObject, serializing it and then base64 encoding it. I'd store these to a file separating by a new line. This allows you to very easily see how many records are in each file, gain access to any arbitrary JSON object within the file and to trivially keep expanding the file (you can expand the above 'repeated' message as well pretty trivially but it is a one way easy operation...)
Compression is a different topic. Protocol Buffers will not compress strings. If you were to define a pb message to match your JSON message, then you will get the benefit of having pb possibly 'compress' any integers into their [varint][1] encoded format. You will get 'less' compression if you try above base64 encoding route as well.
I have a web application that allows a client to request several image thumbnails at once. The response is currently sent as a JSON-encoded list of objects where each object includes the (encoded) image data. (I'm using JSON.NET; it looks like it's encoding the image data as base64 strings).
If I change my application such that the image data is sent separately from the JSON object, as "binary" (application/octet-stream) content, will this be more efficient?
Or put more simply, is application/octet stream a more compact encoding than base64?
Also, how does this differ from image/jpeg?
OK, so application/octet-stream is essentially binary, which is - of course - more efficient than base64. Something like 30% more efficient in terms of space, and removes the need to encode/decode.
I am trying to send a generated PDF file (Apache FOP) to the client. I know this can be done by writing the array to the response stream and by setting the correct content type, length and so on in the servlet. My problem is that the whole app was built based on the idea that it will only receive/send JSON. In the servlet's service() method, I have this:
response.setContentType("application/json");
reqBroker.process(request, response);
RequestBroker is the class who processes the JSON (jackson processor), everything is generic and I cannot change it. On top of this, I have to receive the JSON from the request correctly, to access the data and generate my pdf. So those two lines are necessary. But when I send the response, I need to have another content type so that the pdf is displayed correctly in the browser.
So far, I am able to send the byte array as part of the JSON, but then I don't know how to display the array as PDF on the client (if smth like this is even possible).
I would like some suggestions on how can I send my pdf and set the right header, without messing with the JSON. Thanks.
JSON and byte arrays don't mix.
Instead, you should create an <iframe> and point it to a URL that returns a raw PDF.
Take a look here:How to send pdf in json, it lists couple of approaches that you can consider. The easiest way is to convert the binary data into string by using Base64 compression. In C#, this would mean a call to Convert.FromBase64String. However this has space overhead as Base64 compression means around +33% more memory. If you can get away with it, this is the least complicated solution. in case additional size is an issue you can think about zipping it up.