I'm using Django REST Framework and all the API calls come from Android and iOS apps. The system works perfectly most of the time, however, when an internal server error happens and I get an email from Django, the POST of the WSGIRequest contains <could not parse> instead of the actual posted JSON data (even though 'CONTENT_TYPE': 'application/json' is also in the header, and the data is sent as JSON).
This is really annoying, as it would be great to see the request body that actually causes the error, not just the stacktrace.
The <could not parse> part is very similar to this question (in the ModPythonRequest part): django request.POST contains <could not parse>, except the actual problem is slightly different. Also the reference link in that question (https://stackoverflow.com/questions/12471661/mod-python-could-not-parse-the-django-post-request-for-blackberry-and-some-andro) seems to have gone down, even though the name looked very promising.
I'm on Django 1.6.2 and DRF 2.3.13.
The POST dictionary of the WSGIRequest is always going to be invalid, because it is intended to hold the parsed form data when the Content-Type is application/x-www-form-urlencoded or multipart/form-data.
The data you want is in the body attribute of the WSGIRequest object, which isn't output when that object is converted to a string to be written to the log.
When using Django REST framework, you will typically want to access request.DATA (which will handle whatever formats you have parsers configured for - defaulting to form content and JSON) instead of Django's standard request.POST, which will only handle form encoded data.
Related
This is a style question about REST API design
I have an API which returns a particular resource, and can return either the fields of the resource as a JSON object, or a PDF representation of the resource. The normal REST way of doing that is to use the same URL, but return either the JSON object or the PDF data depending on the "Accept" header of the request.
This has been fine when calling the API from a client application. But now I am writing a web application, and I want to display the PDF. I can fetch the PDF data with XMLHttpRequest, but there isn't an easy way to display it. (I recall some hack involving passing the whole base64-encoded content in the URL, but that is both flaky and disgusting).
The easy way to display a PDF in a web application is window.open(), but I can't pass an Accept header to that (there are a few questions here for people asking how to do that).
This seems like a potentially common situation. What's the best workaround? Stick ?pdf or /pdf or ?accept=pdf onto the url? Is there a de facto standard? Or is there a solution I haven't thought of (maybe treating "application/pdf" as the default request mime type, and only returning the JSON object if the Accept header is "application/json")?
Actually the more I think of it, the more that last idea (return PDF unless explicitly asked for application/json in the Accept header) seems like the right answer. A web browser can display PDFs (at least usually, these days) and not JSON objects, so a request made natively by a browser should always get the PDF, and REST clients that wants JSON data would normally explicitly ask for application/json, so the API server should not do that by default if it has a better browser-compatible form.
Since it took actually explaining this whole problem into the question box to make me think of the answer I posted the question anyway.
I am trying to stick to the Restful design pattern for both JSON and HTML. My issue is the design for creating a new resource (amongst others, but this is the gist of the issue). IE:
JSON – POST to /resource creates a new resource.
JSON – GET to /resource returns a list of resources.
JSON – GET to /resource/{id} returns a resource.
HTML – POST to /resource creates a new resource.
HTML – GET to /resource returns a list of resources.
HTML – GET to /resource/{id} returns a resource.
All good so far – but I need a HTML form to actually create the data to send to the HTML POST. Obviously POST and GET already do things. I could use one of the below to return the HTML form:
HTML – GET to /resource?CREATE
HTML - GET to /resource?action=CREATE
HTML – GET to /resources/CREATE
But they seem like a kludge and not that intuitive.
Any thoughts or ideas?
EDIT - See my answer to my question below. At present this is (I consider) the best option.
I would indeed use something like /resources/create. If you want to allow for non-numeric identifiers, then this will not work. In that case you can identify a resource with a prefix, such as /resources/resource-{id} and then you can still use /resources/create.
I found this blog post really helpful to make URI scheme decisions: http://blog.2partsmagic.com/restful-uri-design/
In fact, you should leverage content negotiation (CONNEG) when you want to handle several formats within RESTful services.
I mean:
Set the Content-Type header to specify the type of sent data
Set the Accept header to specify the type of data you want to receive
The server resources should leverage these hints to make the appropriate data conversion.
In the case of JSON, the content type would be obviously application/json. For HTML form, you should leverage the content type application/x-www-form-urlencoded (or multipart/form-data if you want to upload files as well). See the specification for more details.
Otherwise, you shouldn't use action in URL since it's not really RESTful. The HTTP verb should determine the action to do on the resource. I mean, to create a resource, the POST method should be used. The GET method aims to retrieve the state of a resource.
For more details, you could have a look at this blog post:
Designing a Web API (i.e. RESTful service).
I have an answer. I'll use standard RESTful POST from a HTML page, but when I have no form parameters sent and my accept header is text/html, I'll send a HTML form to the requestor. Keeps RESTful URI design and allows a clean HTML form + process (2 step).
HTML - POST - /resources (with no form attributes) generates a HTML form
HTML - POST - /resources (with form attributes) adds a resource
JSON - POST - /resources (with form attributes) adds a resource
OK, it's not "strictly" RESTful as I'm POSTing but not creating a new resource so in theory I should use a GET for that, but it's the best of a mismatched design.
If anyone can provide a better solution, I'm still all ears :-)
I'd rather add and endpoint called /templates/ that returns a template/form/whatever you need for given action. It also seems that the server should be unaware of such form existence. It can accept or reject a request and it's client job to submit it in an appropriate format.
I guess that you mix processing the view with preparing RESTful endpoints. The backend site should be completely unaware of the fact that some sort of view/form is required. It's client job to prepare such form.
I have a situation where I have to write a api to create a resource and amongst datafields that I need to accept is a string that is basically contents of a html file. As I see it I have a choice between structuring the entire thing as a json object where this field is a string field with urlencoded html string , and having the Content Type as multipart/form-data where each of the fields and the html string (UTF-8 encoded) is a part in the message.
Not using json is something I am not comfortable with as I feel violating the REST standards in not structuring the content of the entity I am about to create thus there is a loss of information for the consumers as they can't tell immediately looking at my api definition about what data to feed to it. But practically multipart/form-data handles stuff like html file content better and more efficient as I will not have to urlencode it and can control the char-encoding also.
What will be a better approach in current context and upholding RESTful principles ? Also are there other trade-offs i should be aware of ? what about parsing a json with a huge string field (~ 200 Kb)embedded?
EDIT :- I was reading some similar questions on SO and one approach that stood out was the 2-step approach of making a first call with metadata to create the entity and then upload the file as an UPDATE process to the created entity wherein we use multipart/form-data. In that context, I guess , what I am asking is how sound is an approach where I send both metadata and the file in a single api call as multipart data , where each metadata field is actually a part in the multipart message as is the file.
The canonical way to upload files to REST API is using the multipart/form-data. As W3 recommendation guide says:
The content type "multipart/form-data" should be used for submitting
forms that contain files, non-ASCII data, and binary data.
Multipart/form-data has advantages over base64 to represent binary data. Is sticked to REST/Http philosophy, and simplify the develop of API clients.
Returning values from Forms: multipart/form-data
W3 Recommendation guide
The good practice is to use multipart/form-data whenever files are uploaded to the server along with database fields. Do not send a base64 JSON string as the request to your Rest API as it might corrupt the file or degrade the performance of your application.
As far as documenting multipart/form-data Rest API for your consumers is concerned you have to force your API consumers to use the same form fields which you have predefined in your web service.
Returning Values from Forms: multipart/form-data
I started using FormData objects everywhere on the client-side, in lieu of regular form input fields, for dynamic REST posts. FormData is presented in a positive light in various tutorials, so I went with it.
However, down the line, this caused me problems when decoding the form data into my Go structs. FormData objects are sent as "multipart/form-data" (regardless of files being sent) and I believe my decoder in Go didn't convert the raw data back to string form. Eventually my SQL queries were throwing panics, as hex data was being sent in where strings should have been.
So with some adjustment, I could use FormData however I've decided to revert to the simple universal recommendation: Use "multipart/form-data" only for special cases like when sending files. Otherwise, just use regular "application/x-www-form-urlencoded".
Here's my JSON response for http://localhost:8000/characters/api/users/1?format=json
)]}',
{"id":1,"username":"admin","mage_by_user":[3],"mage_last_updated":"2015-02-11T16:13:16.229Z"}
Notice the )]}', on the first line.
Here is my code that gets called to create the JSON:
class UserSerializer(serializers.ModelSerializer):
mage_by_user = serializers.PrimaryKeyRelatedField(
many=True, queryset=Mage.objects.all())
mage_last_updated = serializers.ReadOnlyField(
source='mage_by_user.updated_date')
class Meta:
model = User
fields = ('id', 'username', 'mage_by_user', 'mage_last_updated',)
Further testing:
I've noticed the title of the page is TypeError at <insert url here>.
This happens with all of my endpoints
If I try to access a non-existent object (userId=2 for instance), then renders 'normally' for DRF, e.g:
{
detail: "Not found"
}
Any idea why this would happen?
Those characters are inserted by the Djangular middleware AngularJsonVulnerabilityMiddleware, to inject Json Vulnerability Protection
A JSON vulnerability allows third party website to turn your JSON resource URL into JSONP request under some conditions. To counter this your server can prefix all JSON requests with following string ")]}',\n". Angular will automatically strip the prefix before processing it as JSON.
Unfortunately, it means it breaks various JSON viewers.
Sorry to not be more help, but this looks like something entirely unrelated to REST framework. There's absolutely no way a JSON response there would ever be rendered in that way.
Perhaps you have a custom renderer configured, that's outputting a malformed response, perhaps you have some broken middleware inserting those characters, perhaps its an issue in the client or whatever environment you're making the requests, or perhaps it's something else entirely unrelated to any of those.
I'd start by trying to narrow down the issue as much as possible - remove all the complexity from the view and serializer and attempt to replicate the behavior in a test case.
Most likely there's some sort of unexpected integration issue you're missing or some otherwise obvious code typo that's being overlooked.
We're having a bit of a discussion on the subject of posting data to a REST endpoint. Since the objects are quite complex, the easiest solution is to simply serialize them as JSON and send this in the request body.
Now the question is this: Is this kosher? Or should the JSON be set as a form parameter like data=[JSON]? Or is sending of JSON in the request body just frowned upon for forcing the clients using the application, to send their data via JavaScript instead of letting the browser package it up as application/x-www-form-urlencoded?
I know all three options work. But which are OK? Or at least recommended?
I'd say that both methods will work well
it's important that you stay consistent across your APIs. The option I would personally choose is simply sending the content as application/json.
POST doesn't force you to use application/x-www-form-urlencoded - it's simply something that's used a lot because it's what webbrowsers use.
There is nothing wrong about sending it directly as serialized JSON, for example google does this by default in it's volley library (which obviously is their recommended REST library for android).
If fact, there are plenty of questions on SO about how not to use JSON, but rather perform "normal" POST requests with volley. Which is a bit counter intuitive for beginners, having to overwrite it's base class' getParams() method.
But google having it's own REST library doing this by default, would be my indicator that it is OK.
You can use JSON as part of the request data as the OP had stated all three options work.
The OP needs to support JSON input as it had to support contain complex structural content. However, think of it this way... are you making a request to do something or are you just sending what is basically document data and you just happen to use the POST operation as the equivalent of create new entry.
That being the case, what you have is basically a resource endpoint with CRUDL semantics. Following up on that you're actually not limited to application/json but any type that the resource endpoint is supposed to handle.
For non-resource endpoints
I find that (specifically for JAX-RS) the application/x-www-urlencoded one is better.
Consistency with OAuth 2.0 and OpenID Connect, they use application/x-www-urlencoded.
Easier to annotate the individual fields using Swagger Annotations
Swagger provides more defaults.
Postman generates a nice form for you to fill out and makes things easier to test.
Examples of non-resource endpoints:
Authentication
Authorization
Simple Search (though I would use GET on this one)
Non-simple search where there are many criteria
Sending a message/document (though I would also consider multipart/form-data so I can pass meta data along with the content, but JAX-RS does not have a standard for this one Jersey and RestEasy have their own implementations)