I'm looking into an HTTP interface that returns (essentially) a JSON object.
When I access the URL by chrome or firefox, the JSON data is shown with appropriate indents. However, when I download it with curl etc, the data is binary.
I think the browsers know this binary encoding method and show it in a pretty format. (If I save it as a file from the browsers, it is a text file with the indents.)
What do you think this binary encoding is?
(Unfortunately, I can not upload the binary data here...)
[SOLVED]
Browsers send requests with headers but curl doesn't send header by default. That is the reason why I get the different response by these methods. My API returns binarized (compressed) json when called without a header.
You should have a look in the header of the HTTP response message which contains the binary data. There should be values about encoding, content-type and compression.
With this values you can decode the binary data.
Related
I'm developing an API which, given a request-body with certain contents, generates a document that can be in different formats (e.g. pdf, png, etc). Given that the document can be a binary file, the file is base64-encoded and placed within a json response that looks like this:
{
"document": "document base64-encoded"
}
The response might contain other fields but they are not relevant to the question.
My question is:
What is the best way to specify the content-type (e.g. pdf) and the content-encoding (e.g base64) of the field document?
I thought of some options but I would like to hear your recommendation.
My options
Use header parameters Content-Type and Content-Transfer-Encoding: I assume this option would be wrong as these parameters apply to the message itself which in this case is application/json without any encoding.
Specify custom-parameters (e.g. x-document-content-type and x-document-content-tranfer-encoding.
Include such metadata within the response-body like this:
{
"document": {
"content": "document base64-econded",
"type": "pdf",
"econding": "base64"
}
}
Thanks for your input!
There are 2 layers of communication here, each with their own contract/protocol. Your own JSON API protocol, being sent over the HTTP protocol. You should stick to HTTP standards to comply with the contract of that layer, and it is up to you to define the contract of the JSON data.
The mentioned HTTP headers are used to tell the user-agent/client what the MIME-type of the accompanying HTTP body is. This must be application/JSON to comply with the HTTP standard. If you diver from this it will not be valid HTTP traffic anymore and a regular, well configured HTTP client wouldn't be able to consume your API.
The JSON body inside the HTTP message can be filled with whatever you find is necessary for a consumer of your API to use the encoded contents. You are defining your own contract here, so it is really up to your needs and that of the consumers of your api.
Do you use other content encodings as base64? If so, then you can specify the encoding of the data.
Do you provide multiple different file types? Then it's useful to add the content-type of the encoded data. Be aware that the header of the encoded content might already contain information about the type of the contents of the file.
P.s. it would probably be better to provide the "type" as you name it, as a mime-type. This is more common as a known standard for defining content types.
I am writing rest api with Flask. One of my endpoints handles a post request. One field of the request JSON 'audio' is supposed to contain the BASE64 encoded PCM file (audio format). If save this field to a file, it is about 200KB.
It might be too big to copy and paste in Swagger or Postman for testing. Even worse with curl command. Is there any good way to test with really big JSON in the request.
I'd recommend a great tool called Insomnia. You can send point it to any file to send it as a HTTP request. You can change the Content-Type header to be application/json and select your JSON data file. Here's a screenshot of what the program looks like.
So I am reading a JSON file on a remote server via standard node http.get. This JSON contains strings like this: NÖM Portionsbutter PG=20g 100er, but on my server this same string looks like this: N�M Portionsbutter PG=20g 100er.
The problem, I think, lies in the discrepancy between the header of the http request (content-type: text/plain;charset=UTF-8) and the actual encoding of the JSON file (ISO-8859-1).
I tried several versions of fixing this, like using iconv-lite's iconv.decode(data, 'latin1'), but again, these special chars ("Umlaute" in German) show up wrong.
Fun fact: Downloading the file via the browser, inspecting it via file -I file.json and getting text/plain; charset=iso-8859-1 and then using iconv.decode(data, 'latin1') works perfectly fine and the Umlaute are correct.
I am out of ideas right here ... what is the perfect way to properly parse a JSON file like this?
If the server uses an incorrect encoding, it's broken and should be fixed.
(It should use application/json, in which case the charset parameter is undefined/unused).
#Julian Reschke is obviously right but I found a way around this and am now able to read the json with all its Umlaute in it properly.
What I did is use request's encoding option and set it to null, so request is not trying to "parse" the response in any way. I then use iconv.decode(data, 'latin1') to set the proper encoding of the day and then JSON.parse it. works beautifully!
When I'm checking web requests in Chrome's DevTools on Mac (Network tab), I've got the Payload in the following format:
7|0|6|https://www.example.com/app/Basic/|00D1D071AC218DFE91521C012683E911|com.optionfair.client.common.services.nongenerated.RefreshService|getCometUpdates|I|J|1|2|3|4|3|5|6|6|173|VvAwAqy|o$UN|
which is basically separated by vertical bar character (|).
How I can copy or convert above payload from Chrome into some meaningful format such as JSON? Any ideas?
Btw. In this question it looks fine on the screenshot, but in my case, I don't have view parsed and it doesn't look like JSON format at all.
Using Google Chrome on Mac (Version 57.0.2987.133, 64-bit).
Reproducible steps:
Go to this page.
Open DevTools on Network/XHR tab and look for refresh requests.
My goal is to reuse/replicate the POST data in Request Payload in the command-line tool such as curl so it can be recognized (not necessary on the page mentioned above, but I'd like to know the general approach to deal with this blob format). I would expect JSON format, but it's not.
Here you need to look at the request header content-type to determine how this request was encoded before knowing what might parse it:
This is GWT RPC, so it can include serializations of built-in and custom Java Objects, where knowledge of the class is in both server-side Java and transpiled Java running on the client via Google Web Toolkit.
There is no reason for Chrome to understand this format directly, and it need not have a JSON or XML cannonicalization. Fully interpreting these calls to the extent it is possible on the client may require disassembly or introspection tricks against the transpiled client code, assuming the program wasn't transpiled with source maps.
Without digging into the client code, one can interpret the literal rpc while guessing at or probing the definition of classes and their methods by modifying the call.
I have a server and I need to upload files along with some fields from the client to the server. I have currently been using standard multipart/form-data.
I have found however that using multipart/form-data is not ideal. Objects on my server may have other objects nested within them, and thus are represented as a JSON object with other JSON objects embedded within.
I would like for the client to start making POST/PUT requests using a JSON representation exactly like it would expect in a GET request to the server, in a REST-ful manner. This way I don't have to flatten the fields which might be nested a couple layers within the JSON object in order to use multipart/form-data.
Problem is, JSON doesn't represent binary data. Multipart/form-data doesn't seem to have a way to represent fields nested within the values of other fields. But it does have much better handling of file-uploads.
I am at a loss for how to design this. Should I just have the client upload JSON with the fields encoded in base64, and take the 25% hit? Or should I have the JSON object being represented as some sort of "json" variable in a Multipart/form-data request, and have the binary files to be uploaded as another variable?
Should I just have the client upload JSON with the fields encoded in
base64, and take the 25% hit?
The hit will be 33% since 4/3=1.33.
Or should I have the JSON object being represented as some sort of
"json" variable in a Multipart/form-data request, and have the binary
files to be uploaded as another variable?
This should work.
You might also consider this approach: send all files using multipart, then get some identificators of files as a response. Put this identificators in your json and send it anyway you like. This approach might be beneficial if you have many scenarios in which you send files: you might always send them to the server with the same request, then get their identificators; after that do with them what you like.