New Relic currently URL encodes the JSON payload of our webhooks. We correctly include the necessary header, however, I'm wondering if this is best practice - or if the JSON payload should be sent un-encoded (or even with some different encoding)?
It's more common to send JSON directly in the request body with a Content-Type header value of application/json. It's not entirely uncommon to do it the way New Relic does however. GitHub post-receive hook payloads are also encoded this way.
In some languages/frameworks it's easier to extract the JSON data from a parameter than from reading the entire request body since form parameters are automatically parsed out and easily referenced and in some cases reading a request body may involve some more elaborate stream parsing.
An example of that is PHP's syntax for extracting the entire raw request body:
$data = file_get_contents('php://input');
var_dump(json_decode($data));
Versus extracting from a form parameter:
$data = $_POST["payload"];
var_dump(json_decode($data));
For frameworks that easily expose the request body, this isn't as much an issue. Here's the example given from the GitHub docs for handling the hook with Sinatra:
post '/' do
push = JSON.parse(params[:payload])
"I got some JSON: #{push.inspect}"
end
If the JSON body were passed directly, this is what it would look like:
post '/' do
push = JSON.parse(request.body.read)
"I got some JSON: #{push.inspect}"
end
It's mostly just an aesthetic preference, there's no technical advantage to either. With the parameter, you do get the flexibility of sending multiple JSON payloads in a single request or the option to send additional meta data outside the JSON. I would suggest sending a list of JSON objects and including all the meta data in a top level wrapper if that's important to you. Something like this:
{
"notification_id": "akjfd8",
"events": [ ... ]
}
I personally prefer raw JSON bodies, but that's mostly because it makes debugging far easier on sites like RequestBin. Your screenshot does a great job of illustrating how unreadable JSON encoded into a form parameter is.
<rant>
Payloads should be the content type you say they are. If they are sending as application/x-www-form-urlencoded then that is what you should expect to find. The culture of using a Russian doll model of nested encodings is beyond stupid in my opinion.
If you want to send JSON send it as application/json. Is that really so hard?
</rant>
Related
I have a web-hook that calls an API that returns a single plain text response. The response isn't formatted in JSON or xml because the response is a single piece of data, not a map or array. I see plenty of examples of how to extract a field from a JSON response and store it in a marketo token but no example of how to store the whole response payload. Here is a sample response from the API:
alsdfjasdhfalksdhfalksdjalksdk
Note that the response is not anything like this:
{
"field":"alsdfjasdhfalksdhfalksdjalksdk"
}
Maybe some Marketo expert has done this and could share. I'd appreciate it, thanks.
Marketo's parsing of XML and JSON is limited, to say the least--so your webhook does need to declare some sort of key-value pair so the response can be mapped. Given what you're describing, you could likely just hardcode in the surrounding JSON with the right MIME type and be fine.
Basically, Marketo can only handle the following for JSON webhook responses:
Key-value pair
Key can only be defined by placement (no XPath or similar--you'd be stuck with field[1] or similar)
Ideally no nesting, as Marketo doesn't do a great job of reading nested data--largely for the reason mentioned above
I have a situation where I have to write a api to create a resource and amongst datafields that I need to accept is a string that is basically contents of a html file. As I see it I have a choice between structuring the entire thing as a json object where this field is a string field with urlencoded html string , and having the Content Type as multipart/form-data where each of the fields and the html string (UTF-8 encoded) is a part in the message.
Not using json is something I am not comfortable with as I feel violating the REST standards in not structuring the content of the entity I am about to create thus there is a loss of information for the consumers as they can't tell immediately looking at my api definition about what data to feed to it. But practically multipart/form-data handles stuff like html file content better and more efficient as I will not have to urlencode it and can control the char-encoding also.
What will be a better approach in current context and upholding RESTful principles ? Also are there other trade-offs i should be aware of ? what about parsing a json with a huge string field (~ 200 Kb)embedded?
EDIT :- I was reading some similar questions on SO and one approach that stood out was the 2-step approach of making a first call with metadata to create the entity and then upload the file as an UPDATE process to the created entity wherein we use multipart/form-data. In that context, I guess , what I am asking is how sound is an approach where I send both metadata and the file in a single api call as multipart data , where each metadata field is actually a part in the multipart message as is the file.
The canonical way to upload files to REST API is using the multipart/form-data. As W3 recommendation guide says:
The content type "multipart/form-data" should be used for submitting
forms that contain files, non-ASCII data, and binary data.
Multipart/form-data has advantages over base64 to represent binary data. Is sticked to REST/Http philosophy, and simplify the develop of API clients.
Returning values from Forms: multipart/form-data
W3 Recommendation guide
The good practice is to use multipart/form-data whenever files are uploaded to the server along with database fields. Do not send a base64 JSON string as the request to your Rest API as it might corrupt the file or degrade the performance of your application.
As far as documenting multipart/form-data Rest API for your consumers is concerned you have to force your API consumers to use the same form fields which you have predefined in your web service.
Returning Values from Forms: multipart/form-data
I started using FormData objects everywhere on the client-side, in lieu of regular form input fields, for dynamic REST posts. FormData is presented in a positive light in various tutorials, so I went with it.
However, down the line, this caused me problems when decoding the form data into my Go structs. FormData objects are sent as "multipart/form-data" (regardless of files being sent) and I believe my decoder in Go didn't convert the raw data back to string form. Eventually my SQL queries were throwing panics, as hex data was being sent in where strings should have been.
So with some adjustment, I could use FormData however I've decided to revert to the simple universal recommendation: Use "multipart/form-data" only for special cases like when sending files. Otherwise, just use regular "application/x-www-form-urlencoded".
Is it a common practice to create an HTTP POST end point (with the header "application/x-www-form-urlencoded") that takes in a JSON string as the value of the key/value POST data instead of having a list of fields as the POST data? Regardless of being common or not, is it considered bad since it requires more work to stringify the fields into a JSON string?
Example:
user={"username":"bob", "age":1} vs username=bob&age=1 in the POST form data.
This is actually very common - the SOAP messaging format does just this; replacing the standard POST body with a custom messaging format (in the case of SOAP it is XML).
The only thing to consider is that HTTP is pretty ubiquitous these days and although it is trivial to find a JSON library for any language, creating a HTTP body is utterly simple.
So it isn't bad by any means, but if the client is serializing to JSON and then you are deserializing from JSON, would it just be easier to use standard HTTP message bodies?
Recently I run into some weird issue with http header usage ( Adding multiple custom http request headers mystery) To avoid the problem at that time, I have put the fields into json string and add that json string into header instead of adding those fields into separate http headers.
For example, instead of
request.addHeader("UserName", mUserName);
request.addHeader("AuthToken", mAuthorizationToken);
request.addHeader("clientId","android_client");
I have created a json string and add it to the single header
String jsonStr="{\"UserName\":\"myname\",\"AuthToken\":\"123456\",\"clientId\":\"android_client\"}";
request.addHeader("JSonStr",jsonStr);
Since I am new to writing Rest and dealing with the Http stuff, I don't know if my usage is proper or not. I would appreciate some insight into this.
Some links
http://lists.w3.org/Archives/Public/ietf-http-wg/2011OctDec/0133.html
Yes, you may use JSON in HTTP headers, given some limitations.
According to the HTTP spec, your header field-body may only contain visible ASCII characters, tab, and space.
Since many JSON encoders (e.g. json_encode in PHP) will encode invisible or non-ASCII characters (e.g. "é" becomes "\u00e9"), you often don't need to worry about this.
Check the docs for your particular encoder or test it, though, because JSON strings technically allow most any Unicode character. For example, in JavaScript JSON.stringify() does not escape multibyte Unicode, by default. However, you can easily modify it to do so, e.g.
var charsToEncode = /[\u007f-\uffff]/g;
function http_header_safe_json(v) {
return JSON.stringify(v).replace(charsToEncode,
function(c) {
return '\\u'+('000'+c.charCodeAt(0).toString(16)).slice(-4);
}
);
}
Source
Alternatively, you can do as #rocketspacer suggested and base64-encode the JSON before inserting it into the header field (e.g. how JWT does it). This makes the JSON unreadable (by humans) in the header, but ensures that it will conform to the spec.
Worth noting, the original ARPA spec (RFC 822) has a special description of this exact use case, and the spirit of this echoes in later specs such as RFC 7230:
Certain field-bodies of headers may be interpreted according to an
internal syntax that some systems may wish to parse.
Also, RFC 822 and RFC 7230 explicitly give no length constraints:
HTTP does not place a predefined limit on the length of each header field or on the length of the header section as a whole, as described in Section 2.5.
Base64encode it before sending. Just like how JSON Web Token do it.
Here's a NodeJs Example:
const myJsonStr = JSON.stringify(myData);
const headerFriendlyStr = Buffer.from(myJsonStr, 'utf8').toString('base64');
res.addHeader('foo', headerFriendlyStr);
Decode it when you need reading:
const myBase64Str = req.headers['foo'];
const myJsonStr = Buffer.from(myBase64Str, 'base64').toString('utf8');
const myData = JSON.parse(myJsonStr);
Generally speaking you do not send data in the header for a REST API. If you need to send a lot of data it best to use an HTTP POST and send the data in the body of the request. But it looks like you are trying to pass credentials in the header, which some REST API's do use. Here is an example for passing the credentials in a REST API for a service called SMSIfied, which allows you to send SMS text message via the Internet. This example is using basic authentication, which is a a common technique for REST API's. But you will need to use SSL with this technique to make it secure. Here is an example on how to implement basic authentication with WCF and REST.
From what I understand using a json string in the header option is not as much of an abuse of usage as using http DELETE for http GET, thus there has even been proposal to use json in http header. Of course more thorough insights are still welcome and the accepted answer is still to be given.
In the design-first age the OpenAPI specification gives another perspective:
an HTTP header parameter may be an object,
e.g. object X-MyHeader is
{"role": "admin", "firstName": "Alex"}
however, within the HTTP request/response the value is serialized, e.g.
X-MyHeader: role=admin,firstName=Alex
I am just confused with the these terms. Can anybody please provide/explain me brief with an example?
Ajax - "Asynchronous Javascript and XML". Ajax loosely defines a set of technologies to help make web applications present a richer user experience. Data updating and refreshing of the screen is done asynchronously using javascript and xml (or json or just a normal http post).
JSON - "Javascript Object Notation". JSON is like xml in that it can be used to describe objects, but it's more compact and has the advantage of being actual javascript. An object expressed in JSON can be converted into an actual object to be manipulated in javascript code.
By default, Ajax requests have to occur in the same domain of the page where the request originates. JSONP - "JSON with padding" - was created to allow you to request JSON resources from a different domain. (CORS is a newer and better alternative to JSONP.)
REST - "Representational State Transfer". Applications using REST principles have a Url structure and a request/response pattern that revolve around the use of resources. In a pure model, the HTTP Verbs Get, Post, Put and Delete are used to retrieve, create, update and delete resources respectively. Put and Delete are often not used, leaving Get and Post to map to select (GET) and create, update and delete (POST)
Ajax, or more properly, AJAX, stands for Asynchronous Javascript And Xml. Technically it refers to any asynchronous request made by the browser (anything that uses an XmlHttpRequest) on behalf of some script running on the current page, regardless of what content-type is returned. It can also be used to describe a certain pattern of constructing a page/site where most/all of the content is fetched/updated dynamically on the page. When used to describe a data format, "ajax" typically means "xml".
JSON is a data encoding format. The name itself is an acronym for "JavaScript Object Notation". JSON-formatted data looks like:
{"key": "value1", "key2": {"number": 1, "array": [0, 1, 2]}}
JSON data may be fetched by an AJAX request, though it is quite commonly used in other contexts as a lightweight, extensible, and easy-to-parse data exchange format.
JSONP is simply JSON-formatted data wrapped in a callback function. The "P" stands for "with Padding", which is kind of stupid unless you like to think of function calls as "padding". In any case, JSONP data will look like:
someFunction({"key": "value1", "key2": {"number": 1, "array": [0, 1, 2]}});
As such, JSONP is really just a JavaScript snippet, and unlike JSON is not used outside of the context of JavaScript, browsers (or other JavaScript-capable clients), and AJAX requests. The reason for using JSONP is that it allows the same-origin policy to be subverted. A script that was sourced in from site X cannot make a direct request to site Y if site Y is on a different domain from site X. But if site Y's server can send JSONP-formatted responses, then the script from site X can add a new <script> tag to the document that references a URL on site Y, and when the response from site Y is loaded it will invoke some callback function that script X has defined in the document, thus giving script X access to data that was loaded dynamically from site Y.
Note that JSONP data is not (typically) requested using an XmlHttpRequest. It can be done this way, subject to the standard caveats of the same-origin policy, but then you lose the cross-domain magic that makes JSONP useful in the first place.
REST is simply the formal spec/description of how HTTP actually works/is intended to be used. If you understand the concept of a URL being used to request a corresponding resource from a server and the difference between Get and Post then you really know all you need to about REST.
Ajax stand for Asynchronous JavaScript and Xml/XhttpRequet (the X depends and changed since mostly json is used today.
It's a way to execute a request from the page using javascript to the server and receive some response. This response can be anything, json, xml, text, html, etc...
This makes pages look very responsive without having to reload the complete page to execute actions on it. For example posting this answer to your questions. :-)
Json is a data format stands for JavaScrip Object Notation. It's a lighter serialization format than xml and has the advantage to be JavaScript.
JsonP is the next and logical step to using Ajax with Json.
A server will response with JSONP wrapping the Json object in a callback function. The name of the function is passed by the client to the server, usually as a parameter in the querystring.
The P stands for padding, since the server surrounds the json object with the name of the function and the object as an argument.
callback({"name":"my name"});
See: http://en.wikipedia.org/wiki/JSONP for a more detailed explanation.