Recently I run into some weird issue with http header usage ( Adding multiple custom http request headers mystery) To avoid the problem at that time, I have put the fields into json string and add that json string into header instead of adding those fields into separate http headers.
For example, instead of
request.addHeader("UserName", mUserName);
request.addHeader("AuthToken", mAuthorizationToken);
request.addHeader("clientId","android_client");
I have created a json string and add it to the single header
String jsonStr="{\"UserName\":\"myname\",\"AuthToken\":\"123456\",\"clientId\":\"android_client\"}";
request.addHeader("JSonStr",jsonStr);
Since I am new to writing Rest and dealing with the Http stuff, I don't know if my usage is proper or not. I would appreciate some insight into this.
Some links
http://lists.w3.org/Archives/Public/ietf-http-wg/2011OctDec/0133.html
Yes, you may use JSON in HTTP headers, given some limitations.
According to the HTTP spec, your header field-body may only contain visible ASCII characters, tab, and space.
Since many JSON encoders (e.g. json_encode in PHP) will encode invisible or non-ASCII characters (e.g. "é" becomes "\u00e9"), you often don't need to worry about this.
Check the docs for your particular encoder or test it, though, because JSON strings technically allow most any Unicode character. For example, in JavaScript JSON.stringify() does not escape multibyte Unicode, by default. However, you can easily modify it to do so, e.g.
var charsToEncode = /[\u007f-\uffff]/g;
function http_header_safe_json(v) {
return JSON.stringify(v).replace(charsToEncode,
function(c) {
return '\\u'+('000'+c.charCodeAt(0).toString(16)).slice(-4);
}
);
}
Source
Alternatively, you can do as #rocketspacer suggested and base64-encode the JSON before inserting it into the header field (e.g. how JWT does it). This makes the JSON unreadable (by humans) in the header, but ensures that it will conform to the spec.
Worth noting, the original ARPA spec (RFC 822) has a special description of this exact use case, and the spirit of this echoes in later specs such as RFC 7230:
Certain field-bodies of headers may be interpreted according to an
internal syntax that some systems may wish to parse.
Also, RFC 822 and RFC 7230 explicitly give no length constraints:
HTTP does not place a predefined limit on the length of each header field or on the length of the header section as a whole, as described in Section 2.5.
Base64encode it before sending. Just like how JSON Web Token do it.
Here's a NodeJs Example:
const myJsonStr = JSON.stringify(myData);
const headerFriendlyStr = Buffer.from(myJsonStr, 'utf8').toString('base64');
res.addHeader('foo', headerFriendlyStr);
Decode it when you need reading:
const myBase64Str = req.headers['foo'];
const myJsonStr = Buffer.from(myBase64Str, 'base64').toString('utf8');
const myData = JSON.parse(myJsonStr);
Generally speaking you do not send data in the header for a REST API. If you need to send a lot of data it best to use an HTTP POST and send the data in the body of the request. But it looks like you are trying to pass credentials in the header, which some REST API's do use. Here is an example for passing the credentials in a REST API for a service called SMSIfied, which allows you to send SMS text message via the Internet. This example is using basic authentication, which is a a common technique for REST API's. But you will need to use SSL with this technique to make it secure. Here is an example on how to implement basic authentication with WCF and REST.
From what I understand using a json string in the header option is not as much of an abuse of usage as using http DELETE for http GET, thus there has even been proposal to use json in http header. Of course more thorough insights are still welcome and the accepted answer is still to be given.
In the design-first age the OpenAPI specification gives another perspective:
an HTTP header parameter may be an object,
e.g. object X-MyHeader is
{"role": "admin", "firstName": "Alex"}
however, within the HTTP request/response the value is serialized, e.g.
X-MyHeader: role=admin,firstName=Alex
Related
The documentation did not seem to help, and I could not find a lot of post that described the differences in go lang with render.JSON vs a W.Write([]byte). We have a simple chi router that makes a post request and returns the data. I thought the only difference was auto setting content type, but when I use render.JSON it removes the keys.
Can anyone please tell me in a few human words if there is a difference and a good use case to use render vs write? Does anyone have any personal exp, or advice regarding this? I have read the documents, and seen response sent many different ways.
//Current Pref as I can define the headers, etc.
w.WriteHeader(resp.StatusCode)
w.Header().Set("Content-Type", "application/json")
json := []byte(body)
w.Write(json)
//vs (sets content type, but removes the keys* and I am not sure why)
render.JSON(w,r,interface)
See the code for render.JSON. The function encodes the argument to JSON, sets a JSON content type header and writes the JSON to the response. The application can set response headers before calling render.JSON. The application can set the response status by calling render.Status before calling render.JSON.
The ResponseWriter.Write method writes bytes to the response as is.
Note a key difference between render.JSON and ResponseWriter.Write: The former function encodes the argument as JSON and the latter writes the bytes as is.
The render.JSON function encapsulates functionality commonly used by applications. Use the function if you find it convenient.
I have a REST API that accepts and returns JSON data.
A sample request response is a follows
Request
{
"repos": [
"some-repo",
"test-repo<script>alert(1)</script>"
]
}
Response
{
"error": "Error Message",
"repos": [
"test-repo<script>alert(1)</script>"
]
}
Is my API vulnerable for XSS?. From what I understand, since the Content-Type is set to application/json, the API as such is safe from XSS. The client needs to ensure that the output is encoded to prevent any XSS attacks.
To add an additional layer of security, I can add some input encoding/validation in the API layer.
Please let me know if my assessment is right and any other gotchas that I need to be aware of
I think it's right that any XSS issue here is a vulnerability of the client. If the client inserts HTML into a document, then it is its responsibility to apply any neccessary encoding.
The client knows what encoding is required not the server. Different encoding, or no encoding may be needed in different places for the same data. For example:
If a client did something like:
$(div).html("<b>" + repos + "</b>");
then it would be vulnerable to XSS, so repos would need to be HTML encoded here.
But if it did something like:
$(div).append($("<b>").text(repos));
then HTML encoding would have resulted in HTML entity codes being wrongly displayed to the user.
Or if the client wanted to do some processing of the data, it may want the plaintext data first to do the processing, and then encode it later to output it.
Input validation can help too, but the rules for what is valid input may not align with what is safe to use without encoding. Things like ampersands, quotes and brackets can appear in valid text data too. But if your data can't contain these characters, you can reject the input as invalid.
Your API will not be vulnerable to XSS unless it also provides a UI that consumes it. It will be the clients of your API which could be vulnerable - they will need to make sure they correctly encode any data from your API before they use it in any UI.
I think your api is vulnerable, though the script may not be execute. Talking to XSS prevention, we always suggest decode/validation dangerous characters when the api deals the input. There is a common requirement "clean the data before it store in the DB".
As for your situation, the api just response a json, but we don't know where the data in json will be used to. usually the frontend accept the data without any decode/validation, if that, there will be a xss.
you talk about that the client use decode the data they get from your api. Yes I agree, but frontend always "trust" the backend so that they won't decode the data. but the api server should not trust any input from frontend due to this can be controlled/changed by (malicious)users.
I've set up an API in Spring where the client can specify the ids of the objects they want to receive. It returns a JSON list (using the #ResponseBody annotation) of the objects.
But, since the request can be a long list, I've set it up as a POST, where it's received as an object named ProductRequest (using the #RequestBody annotation). This seemingly isn't conformant to the official REST API standard, since posts are official to create new objects, but it seems better to implement it this way since you're not cluttering up the URL with a bunch of ids. Also, I can specify additional parameters customizing the output.
So my question is, can this be considered a valid RESTful design? Post isn't being used to create an object, so it's not strictly conformant to restful. Thoughts?
It is a simple GET request.
#GetMapping(path = "/your-url")
public List<Object> method(#RequestParam(name="ids") List<Long> objectIds) {
}
You can receive your list just as a simple java list.
/api?ids=1,2,3
You can very well do an http POST to "get" the resource(s). There are some very valid examples of doing so,
GraphQL -- All we are doing is sending (POSTing) some Queries in requestbody to "GET" some data in the format that we prefer
Assume that you have to authenticate with a username and password to get a specific data, you will either send it in Header in GET request or you will do a POST to 'GET' a data.
Couldn't that overflow the max allowable for the URL string?
Actually, there is no limitation on the length of a URI. However, practically, you can encounter some.
According to RFC7230 (emphasis mine),
HTTP does not place a predefined limit on the length of a request-line, as described in Section 2.5. A server that receives a method longer than any that it implements SHOULD respond with a 501 (Not Implemented) status code. A server that receives a request-target longer than any URI it wishes to parse MUST respond with a 414 (URI Too Long) status code.
Various ad-hoc limitations on request-line length are found in practice. It is RECOMMENDED that all HTTP senders and recipients support, at a minimum, request-line lengths of 8000 octets.
It clearly suggests that we can't rely on a request-line while we are dealing with your issue.
Can this be considered a valid RESTful design?
Due to the limitation mentioned above, using POST is a valid workaround.
As stated by this Wikipedia article,
The common workaround for these problems is to use POST instead of GET and store the parameters in the request body. The length limits on request bodies are typically much higher than those on URL length.
To completely free you from hesitation, there is a common non-standard header X-Http-Method-Override. It requests a web application to override the method specified in the request with the method given in the header:
X-Http-Method-Override: GET
I am not sure whether Spring handles the header. Though, it has the HiddenHttpMethodFilter filter which seems to do similar work.
I have a situation where I have to write a api to create a resource and amongst datafields that I need to accept is a string that is basically contents of a html file. As I see it I have a choice between structuring the entire thing as a json object where this field is a string field with urlencoded html string , and having the Content Type as multipart/form-data where each of the fields and the html string (UTF-8 encoded) is a part in the message.
Not using json is something I am not comfortable with as I feel violating the REST standards in not structuring the content of the entity I am about to create thus there is a loss of information for the consumers as they can't tell immediately looking at my api definition about what data to feed to it. But practically multipart/form-data handles stuff like html file content better and more efficient as I will not have to urlencode it and can control the char-encoding also.
What will be a better approach in current context and upholding RESTful principles ? Also are there other trade-offs i should be aware of ? what about parsing a json with a huge string field (~ 200 Kb)embedded?
EDIT :- I was reading some similar questions on SO and one approach that stood out was the 2-step approach of making a first call with metadata to create the entity and then upload the file as an UPDATE process to the created entity wherein we use multipart/form-data. In that context, I guess , what I am asking is how sound is an approach where I send both metadata and the file in a single api call as multipart data , where each metadata field is actually a part in the multipart message as is the file.
The canonical way to upload files to REST API is using the multipart/form-data. As W3 recommendation guide says:
The content type "multipart/form-data" should be used for submitting
forms that contain files, non-ASCII data, and binary data.
Multipart/form-data has advantages over base64 to represent binary data. Is sticked to REST/Http philosophy, and simplify the develop of API clients.
Returning values from Forms: multipart/form-data
W3 Recommendation guide
The good practice is to use multipart/form-data whenever files are uploaded to the server along with database fields. Do not send a base64 JSON string as the request to your Rest API as it might corrupt the file or degrade the performance of your application.
As far as documenting multipart/form-data Rest API for your consumers is concerned you have to force your API consumers to use the same form fields which you have predefined in your web service.
Returning Values from Forms: multipart/form-data
I started using FormData objects everywhere on the client-side, in lieu of regular form input fields, for dynamic REST posts. FormData is presented in a positive light in various tutorials, so I went with it.
However, down the line, this caused me problems when decoding the form data into my Go structs. FormData objects are sent as "multipart/form-data" (regardless of files being sent) and I believe my decoder in Go didn't convert the raw data back to string form. Eventually my SQL queries were throwing panics, as hex data was being sent in where strings should have been.
So with some adjustment, I could use FormData however I've decided to revert to the simple universal recommendation: Use "multipart/form-data" only for special cases like when sending files. Otherwise, just use regular "application/x-www-form-urlencoded".
New Relic currently URL encodes the JSON payload of our webhooks. We correctly include the necessary header, however, I'm wondering if this is best practice - or if the JSON payload should be sent un-encoded (or even with some different encoding)?
It's more common to send JSON directly in the request body with a Content-Type header value of application/json. It's not entirely uncommon to do it the way New Relic does however. GitHub post-receive hook payloads are also encoded this way.
In some languages/frameworks it's easier to extract the JSON data from a parameter than from reading the entire request body since form parameters are automatically parsed out and easily referenced and in some cases reading a request body may involve some more elaborate stream parsing.
An example of that is PHP's syntax for extracting the entire raw request body:
$data = file_get_contents('php://input');
var_dump(json_decode($data));
Versus extracting from a form parameter:
$data = $_POST["payload"];
var_dump(json_decode($data));
For frameworks that easily expose the request body, this isn't as much an issue. Here's the example given from the GitHub docs for handling the hook with Sinatra:
post '/' do
push = JSON.parse(params[:payload])
"I got some JSON: #{push.inspect}"
end
If the JSON body were passed directly, this is what it would look like:
post '/' do
push = JSON.parse(request.body.read)
"I got some JSON: #{push.inspect}"
end
It's mostly just an aesthetic preference, there's no technical advantage to either. With the parameter, you do get the flexibility of sending multiple JSON payloads in a single request or the option to send additional meta data outside the JSON. I would suggest sending a list of JSON objects and including all the meta data in a top level wrapper if that's important to you. Something like this:
{
"notification_id": "akjfd8",
"events": [ ... ]
}
I personally prefer raw JSON bodies, but that's mostly because it makes debugging far easier on sites like RequestBin. Your screenshot does a great job of illustrating how unreadable JSON encoded into a form parameter is.
<rant>
Payloads should be the content type you say they are. If they are sending as application/x-www-form-urlencoded then that is what you should expect to find. The culture of using a Russian doll model of nested encodings is beyond stupid in my opinion.
If you want to send JSON send it as application/json. Is that really so hard?
</rant>