Maybe I'm confusing things or over-complicating them, but I'm struggling to develop a RESTful API that supports both HTML and JSON content types. Take for example a simple user management feature. I would expect to have an API that looks like this:
GET /users: lists all users
GET /users/{id}: views a single user
POST /users: creates a new user
A programmatic client posting to /users with a JSON payload would expect a 201 Created response with a Location header specifying the URL to the newly created user, e.g. /users/1. However, a person creating a user through his web browser would post to the same URL with a form-encoded payload and would expect to be redirected to the user list page, requiring the API to return a 302/303 redirect with a Location header of /users.
From a purely conceptual point of view, I find it surprising that an API would react differently based on the submitted content type, and wonder if this is bad design. Then again, maybe it's a mistake to consider the programmatic API and the web-centric API to be the same API and one shouldn't worry about such concerns and worry more about providing a good experience to the different clients.
What do you think?
You've stumbled upon two separate issues.
One, the typical web browser is a pretty lousy REST client.
Two, web application APIs are not necessarily REST APIs (see #1).
And thus, your conundrum of trying to serve two masters.
Arguably representation has little to do with application semantics when it comes to details such as workflow, particularly if you have equally rich media types (vs a constrained media type such as an image, or something else).
So, in those terms, it's really not appropriate to have the application behave differently given similar media types.
On the other hand, media type IS Yet Another aspect of the request which can influence operation on the back end. You could, for example be requesting an elided "lite" data type that may well not offer links to other parts of the api that a richer media type would, or your authorization level is a factor on what data you can view, as well as what other relations are available, or even what media types are supported at all.
So it's fair that every aspect of the request payload can have impact on the particular semantics and effect of any particular request to the server. In that case, you're scenario is not really off the mark.
In the end, it's down to documentation to clarify your intent as an API designer.
Related
Our business wants to email our customers a survey after they work with support. For internal reasons, we want to ask them the first question in the body of the email. We'd like to have a link for each answer. The link will go to a web service, which will store the answer, then present the rest of the survey.
So far so good.
The challenge I'm running into: making a server-side changed based on an HTTP GET is bad practice, but you can't do a POST from a link. Options seem to be:
Use an HTTP GET instead, even though that's not correct and could cause problems (https://twitter.com/rombulow/status/990684453734203392)
Embed an HTML form in the email and style some buttons to look like links (likely not compatible with a number of email platforms)
Don't include the first question in the email (not possible for business reasons)
Use HTTP GET, but have some sort of mechanism which prevents a link from altering the server state more than once
Does anybody have any better recommendations? Googling hasn't turned up much about this specific situation.
One thing to keep in mind is that HTTP is specifying semantics, not implementation. If you want to change the state of your server on receipt of a GET request, you can. See RFC 7231
This definition of safe methods does not prevent an implementation from including behavior that is potentially harmful, that is not entirely read-only, or that causes side effects while invoking a safe method. What is important, however, is that the client did not request that additional behavior and cannot be held accountable for it. For example, most servers append request information to access log files at the completion of every response, regardless of the method, and that is considered safe even though the log storage might become full and crash the server. Likewise, a safe request initiated by selecting an advertisement on the Web will often have the side effect of charging an advertising account.
Domain agnostic clients are going to assume that GET is safe, which means your survey results could get distorted by web spiders crawling the links, browsers pre-loading resource to reduce the perceived latency, and so on.
Another possibility that works in some cases is to treat the path through the graph as the resource. Each answer link acts like a breadcrumb trail, encoding into itself the history of the clients answers. So a client that answered A and B to the first two questions is looking at /survey/questions/questionThree?AB where the user that answered C to both is looking at /survey/questions/questionThree?CC. In other words, you aren't changing the state of the server, you are just guiding the client through a pre-generated survey graph.
We are developing server with REST API, which accepts and responses with JSON. The problem is, if you need to upload images from client to server.
Note: and also I am talking about a use-case where the entity (user) can have multiple files (carPhoto, licensePhoto) and also have other properties (name, email...), but when you create new user, you don't send these images, they are added after the registration process.
The solutions I am aware of, but each of them have some flaws
1. Use multipart/form-data instead of JSON
good : POST and PUT requests are as RESTful as possible, they can contain text inputs together with file.
cons : It is not JSON anymore, which is much easier to test, debug etc. compare to multipart/form-data
2. Allow to update separate files
POST request for creating new user does not allow to add images (which is ok in our use-case how I said at beginning), uploading pictures is done by PUT request as multipart/form-data to for example /users/4/carPhoto
good : Everything (except the file uploading itself) remains in JSON, it is easy to test and debug (you can log complete JSON requests without being afraid of their length)
cons : It is not intuitive, you cant POST or PUT all variables of entity at once and also this address /users/4/carPhoto can be considered more as a collection (standard use-case for REST API looks like this /users/4/shipments). Usually you cant (and dont want to) GET/PUT each variable of entity, for example users/4/name . You can get name with GET and change it with PUT at users/4. If there is something after the id, it is usually another collection, like users/4/reviews
3. Use Base64
Send it as JSON but encode files with Base64.
good : Same as first solution, it is as RESTful service as possible.
cons : Once again, testing and debugging is a lot worse (the body can have megabytes of data), there is increase in size and also in processing time in both - client and server
I would really like to use solution no. 2, but it has its cons... Anyone can give me a better insight of "what is best" solution?
My goal is to have RESTful services with as much standards included as possible, while I want to keep it as simple as possible.
OP here (I am answering this question after two years, the post made by Daniel Cerecedo was not bad at a time, but the web services are developing very fast)
After three years of full-time software development (with focus also on software architecture, project management and microservice architecture) I definitely choose the second way (but with one general endpoint) as the best one.
If you have a special endpoint for images, it gives you much more power over handling those images.
We have the same REST API (Node.js) for both - mobile apps (iOS/android) and frontend (using React). This is 2017, therefore you don't want to store images locally, you want to upload them to some cloud storage (Google cloud, s3, cloudinary, ...), therefore you want some general handling over them.
Our typical flow is, that as soon as you select an image, it starts uploading on background (usually POST on /images endpoint), returning you the ID after uploading. This is really user-friendly, because user choose an image and then typically proceed with some other fields (i.e. address, name, ...), therefore when he hits "send" button, the image is usually already uploaded. He does not wait and watching the screen saying "uploading...".
The same goes for getting images. Especially thanks to mobile phones and limited mobile data, you don't want to send original images, you want to send resized images, so they do not take that much bandwidth (and to make your mobile apps faster, you often don't want to resize it at all, you want the image that fits perfectly into your view). For this reason, good apps are using something like cloudinary (or we do have our own image server for resizing).
Also, if the data are not private, then you send back to app/frontend just URL and it downloads it from cloud storage directly, which is huge saving of bandwidth and processing time for your server. In our bigger apps there are a lot of terabytes downloaded every month, you don't want to handle that directly on each of your REST API server, which is focused on CRUD operation. You want to handle that at one place (our Imageserver, which have caching etc.) or let cloud services handle all of it.
small 2023 update: If possible, but CDN in front of the pictures, it usually will save you a lot of money and make the pictures even more available (i.e. no issues when peaks happen).
Cons : The only "cons" which you should think of is "not assigned images". User select images and continue with filling other fields, but then he says "nah" and turn off the app or tab, but meanwhile you successfully uploaded the image. This means you have uploaded an image which is not assigned anywhere.
There are several ways of handling this. The most easiest one is "I don't care", which is a relevant one, if this is not happening very often or you even have desire to store every image user send you (for any reason) and you don't want any deletion.
Another one is easy too - you have CRON and i.e. every week and you delete all unassigned images older than one week.
There are several decisions to make:
The first about resource path:
Model the image as a resource on its own:
Nested in user (/user/:id/image): the relationship between the user and the image is made implicitly
In the root path (/image):
The client is held responsible for establishing the relationship between the image and the user, or;
If a security context is being provided with the POST request used to create an image, the server can implicitly establish a relationship between the authenticated user and the image.
Embed the image as part of the user
The second decision is about how to represent the image resource:
As Base 64 encoded JSON payload
As a multipart payload
This would be my decision track:
I usually favor design over performance unless there is a strong case for it. It makes the system more maintainable and can be more easily understood by integrators.
So my first thought is to go for a Base64 representation of the image resource because it lets you keep everything JSON. If you chose this option you can model the resource path as you like.
If the relationship between user and image is 1 to 1 I'd favor to model the image as an attribute specially if both data sets are updated at the same time. In any other case you can freely choose to model the image either as an attribute, updating the it via PUT or PATCH, or as a separate resource.
If you choose multipart payload I'd feel compelled to model the image as a resource on is own, so that other resources, in our case, the user resource, is not impacted by the decision of using a binary representation for the image.
Then comes the question: Is there any performance impact about choosing base64 vs multipart?. We could think that exchanging data in multipart format should be more efficient. But this article shows how little do both representations differ in terms of size.
My choice Base64:
Consistent design decision
Negligible performance impact
As browsers understand data URIs (base64 encoded images), there is no need to transform these if the client is a browser
I won't cast a vote on whether to have it as an attribute or standalone resource, it depends on your problem domain (which I don't know) and your personal preference.
Your second solution is probably the most correct. You should use the HTTP spec and mimetypes the way they were intended and upload the file via multipart/form-data. As far as handling the relationships, I'd use this process (keeping in mind I know zero about your assumptions or system design):
POST to /users to create the user entity.
POST the image to /images, making sure to return a Location header to where the image can be retrieved per the HTTP spec.
PATCH to /users/carPhoto and assign it the ID of the photo given in the Location header of step 2.
There's no easy solution. Each way has their pros and cons . But the canonical way is using the first option: multipart/form-data. As W3 recommendation guide says
The content type "multipart/form-data" should be used for submitting forms that contain files, non-ASCII data, and binary data.
We aren't sending forms,really, but the implicit principle still applies. Using base64 as a binary representation, is incorrect because you're using the incorrect tool for accomplish your goal, in other hand, the second option forces your API clients to do more job in order to consume your API service. You should do the hard work in the server side in order to supply an easy-to-consume API. The first option is not easy to debug, but when you do it, it probably never changes.
Using multipart/form-data you're sticked with the REST/http philosophy. You can view an answer to similar question here.
Another option if mixing the alternatives, you can use multipart/form-data but instead of send every value separate, you can send a value named payload with the json payload inside it. (I tried this approach using ASP.NET WebAPI 2 and works fine).
I am not able to understand if HTML5s Server-sent-events really fit in a ReST architecture. I understand that NOT all aspects of HTML5/HTTP need to fit in a ReST architecture. But I would like to know from experts, which half of HTTP is SSE in (the ReSTful half or the other half !).
One view could be that it is ReSTful, because there is an 'initial' HTTP GET request from the client to the server and the remaining can just be seen as partial-content responses of just a different Content-type ("text/event-stream")
A request sent without any idea of how many responses are going to come as response(events) ? Is that ReSTful ?
Motivation for the question: We are developing the server-side of an app, and we want to support both ReST clients (in general) and Browsers (in particular). While SSEs will work for most of the HTML5 browser clients, we are not sure if SSEs are suitable for support by a pure ReST client. Hence the question.
Edit1:
Was reading Roy Fielding's old article, where he says :
"In other words, a single user request results in a potentially large number of server obligations. As such, a benevolent user can produce a disproportionate load on the publisher or broker that is distributing notifications. On the Internet, we don’t have the luxury of designing just for benevolent users, and thus in HTTP systems we call such requests a denial-of-service exploit.... That is exactly why there is no standard mechanism for notifications in HTTP"
Does that imply SSE is not ReSTful ?
Edit2:
Was going through Twitter's REST API.
While REST puritans might debate if their REST API is really/fully REST, just the title of the section Differences between Streaming and REST seems to suggest that Streaming (and even SSE) cannot be considered ReSTful !? Anyone contends that ?
I think it depends:
Do your server-side events use hypermedia and hyperlinks to describe possible state changes?
The answer to that question is the answer to whether or not they satisfy REST within your application architecture.
Now, the manner in which those events are sent/received may or may not adhere to REST - everything I have read about SSE suggests that they do not. I suspect it will impact several principles, especially layering - though if intermediaries were aware of the semantics of SSE you could probably negate this.
I think this is orthogonal as it's just part of the processing directive for HTML and JavaScript that the browser (via the JavaScript it is running) understands. You should still be able to have client-side application state decoupled from server-side resource state.
Some of the advice I've seen on how to deal with scaling using SSE don't fit REST - i.e. introducing custom headers (modifying the protocol).
How do you respect REST while using SSE?
I'd like to see some kind of
<link rel="event" href="http://example.com/user/1" />
Then the processing directives (including code-on-demand such as JavaScript) of whatever content-type/resource you are working with tell the client how to subscribe and utilize the events made available from such a hyperlink. Obviously, the data of those events should itself be hypermedia containing more hyperlinks that control program flow. (This is where I believe you make the distinction between REST and not-REST).
At some point the browser could become aware of that link relationship - just like a stylesheet and do some of that fancy wire-up for you, so all you do is just listen for events in JavaScript.
While I do think that your application can still fit a REST style around SSE, they are not REST themselves (Since your question was specifically about their use, not their implementation I am trying to be clear about what I am speaking to).
I dislike that the specification uses HTTP because it does away with a lot of the semantics and effectively tunnels an anemic protocol through an otherwise relatively rich one. This is supposedly a benefit but strikes me as selling dinner to pay for lunch.
ReST clients (in general) and Browsers (in particular).
How is your browser not a REST client? Browser are arguably the most REST client of all. It's all the crap we stick in to them via JavaScript that makes then stop adhering to REST. I suspect/fear that as long as we continue to think about our REST-API 'clients' and our browser clients as fundamentally different we will still be stuck in this current state - presumably because all the REST people are looking for a hyperlink that the RPC people have no idea needs to exist ;)
I think SSE can be used by a REST API. According to the Fielding dissertation, we have some architectural constraints the application MUST meet, if we want to call it REST.
client-server architecture: ok - the client triggers while the server does the processing
stateless: ok - we still store client state on the client and HTTP is still a stateless protocol
cache: ok - we have to use no cache header
uniform interface
identification of resources: ok - we use URIs
manipulation of resources through representations: ok - we can use HTTP methods with the same URI
self-descriptive messages: ok, partially - we use content-type header we can add RDF to the data if we want, but there is no standard which describes that the data is RDF coded. we should define a text/event-stream+rdf MIME type or something like that if that is supported.)
hypermedia as the engine of application state: ok - we can send links in the data
layered system: ok - we can add other layers, which can transform the data stream aka. pipes and filters where the pump is the server, the filters are these layers and the sink is the client
code on demand: ok - optional, does not matter
Btw. there is no such rule, that you cannot use different technologies together. So you can use for example a REST API and websockets together if you want, but if the websockets part does not meet at least with the self-descriptive message and the HATEOAS constraints, then the client will be hard to maintain. Scalability can be another problem, since the other constraints are about that.
I'm working on a web application (Java EE backend) which contains a fairly complex input modal. This input modal allows the user to capture data, but it has a bunch of (JavaScript) restrictions, such as mandatory fields, fields only being available if a specific value is entered, etc.
I have to expose this functionality to external systems and allow them to submit this data to my server. These external systems can be both web or client based (but I can assume that the clients will have internet access). My first thought is to provide some kind of definition of the fields and stuff like mandatory to these systems through services, and have them render the input modal however they want. This has been met with resistance though, because the types of fields and restrictions will likely change quite a bit during the next few months of development. These external systems have different deployment timelines, and for this to work we'll have to firstly duplicate all the logic handling these restrictions across all systems, and secondly synchronize our deployments.
An alternative which has been proposed is to have the external systems call my modal through standard HTTP and render it either in an iframe or in an embedded rendered. This solves all of the previous complaints, but it leaves me feeling a little uneasy.
Are there any alternatives we are not thinking of? Maybe some kind of UI schema with existing render libraries for the different platforms? What are your thoughts on the second proposal, any major concerns or is this the "best" solution?
Edit: To clarify, I'll of course still perform backend validation regardless of the frontend decision, as I can't just trust the incoming data.
The constraints that you mention (mandatory fields etc.) really have nothing to do with the user interface. You are also right that it is not a good idea to have your backend render web content.
Your first proposal sounds like a good idea, here's how I would solve the issues you mentioned:
Do all the validation on the backend and send a model object to the client, representing the current state of the UI (field name, type, enabled/disabled, error message etc.).
Keep the client as dumb as possible. It should only be responsible for rendering the model on a window / webpage. Whenever a field is changed and it requires validation, submit the model to the backend for validation and get back a new model to be displayed. (You could optimize this by only returning the fields that changed.)
Doing it this way will keep your validation logic in one place (the backend) and the clients rarely need to be modified.
I have been faced with same issues in several previous projects. Based on this experience I can honestly say that server-side validation is the thing you will likely have to implement to avoid rubbish being committed from client side regardless if it comes from GUI or other third party system via API. You can choose one of available validation frameworks, I used Apache Commons Validator and think it is well, or you can implement your own one. On the other hand client side pre-validation, auto-completion and data look up are the solutions you should have to make human users happy. Do not consider about code duplication, just make your system right way from the business point of view.
The oEmbed specification mentions 2 different ways of finding the oEmbed content of an URL:
Knowing the API endpoint of the website and passing it, through a GET parameter, the URL you want info about, if it matches the URL pattern it declared.
Discovering the URL of the oEmbed version thanks to a <link rel="alternate" type="application/json+oembed" ... /> (or text/xml+oembed) HTML header.
The 2nd ways seems more generic, as you don't have to store and maintain a whole list of providers. Moreover, lists of providers are the sign of a centralized internet, where only a few actors exist. This approach is hardly scalable.
I can see a use for the 1st approach, though, for websites that can parse resources made available by someone else. For example, I can provide an oEmbed version of video pages from website Foo. However, for several reasons, mainly security-related, I wouldn't trust someone who says "I can parse resource X for you" unless X's author is OK with that, which brings us back to approach 2.
So my question is: what did I miss here? What's the use of the 1st method of dealing with oEmbed? For instance, why store (and maintain up-to-date) a whole list of endpoints and patterns like oohEmbed does if you have a generic way of discovering it on-the-fly and for virtually any resource on the internet?
As a very closely related question, which I think may be asked at the same time (please correct me if I'm wrong): what happens if one doesn't provide a central endpoint for oEmbed contents, but rather, say, expect a '?version=oembed' parameter on each URL, that returns the oEmbed version instead of the standard one?
If I recall correctly, supporting both mechanisms was a compromise that we figured would help drive adoption. It's much easier to persuade large web properties to add a single endpoint vs. adding markup (that's irrelevant to most clients) to every response body. It was a pragmatic choice.
Longer term we planned to leverage some of the work Eran Hammer-Lahav was doing around discovery rather than re-inventing it (poorly, again). Unfortunately, his ideas still haven't gotten much traction and the web still lacks a good, standardized way to do this sort of thing.
I was hoping to find an answer here but it looks like everyone else is as confused as we are. The advantage of using option 1 in my opinion is that it only uses 1 json request instead of a potentially expensive html request followed by the json request. You can always use option 2 as a fallback in case you can't match a pattern in your pre-baked list of oEmbed providers.
OEmbed discovery is a major security concern. WordPress for example has a whitelist of supported OEmbed providers.
Suppose that every random URL at the internet can trigger an OEmbed code. That means everyone can hack your site.
Steps:
Create a new site, add an OEmbed discovery.
Post the URL to a form at your site. Now your site perform the OEmbed on my behalf.
Exploit:
by denial-of-service (DOS): e.g. redirect the URL to a tarpit or feed it a 1GB json response.
by cross site scripting (XSS): inject random HTML to pages that other people can see.
by stealing the admin's session-cookie via XSS: now the attacker can login to your CMS to upload files, and exploit even more.
It's XSS to the max, with little to stop it. The only sane thing to do, is whitelisting proper endpoints. That's the oEmbed endpoints are explicitly listed.
If you want something scalable, you might like www.noembed.com and www.embedly.com They provide OEmbed support for various sites which don't do OEmbed themselves.