I am confused on how a REST api can both be hypertext driven, but also machine readable. Say I design an API and some endpoint lists contents of a collection.
GET /api/companies/
The server should return a list with company resources e.g:
/api/companies/adobe
/api/companies/microsoft
/api/companies/apple
One way would be to generate a hypertext (html) page with <a> links to the respective resources. However I would also like to make it easy for a non-browser client to read this list. For example, some client might want to populates a dropdown gui with companies. In this case returning a html document is inappropriate, and it might be better to return a list in JSON or XML format.
It is not clear to me how REST style can satisfy both. Is there a practical solution or examples of a REST api that is both nice to browsers and non-browser clients?
What you're looking for is nowadays referred to as HATEOAS API's. See this question for examples: Actual examples for HATEOAS (REST-architecture)
The ReST architectural style, as originally defined by Roy Fielding, prescribes "hypertext as the engine for application state" as one of the architectural contraints. However, this concept got more or less "lost in translation" when people started equaling "RESTful API's" with "using the HTTP verbs right" (plus a little more, if you're lucky). (Edit: Providing credence for my assertion are the first and second highest-ratest answers in What exactly is RESTful programming? . The first talks only about HTTP verbs).
Some thoughts on your question: (mainly because the subject keeps fascinating me)
In HATEOAS, standardized media types with precise meaning are very important. It's generally thought to be best to reuse an existing media type when possible, to benefit from general understanding and tooling around this. One popular method is using XML, because it offers both general structure for data and a way to define semantics, i.e. through an XML schema or with namespaces. XML in and by itself is more or less meaningless when considering HATEOAS. The same applies for JSON.
For supporting links, you want to choose a media type that either supports links "natively" (i.e. text/html, application/xhtml+xml) or a media type that allows defining what pieces in the document must be interpreted as links through some embedded metadata, such as XML can with for example XLINK. I don't think you could use application/json because JSON by itself has no pre-defined place to define metadata. I do think that it would be possible to design a media type based on json - call it application/x-descriptive-json - that defines up-front that the JSON document returned must consist of a "header" and "body" property where header may contain further specified metadata. You could also design a media type for JSON just to support embedded links. Simpler media type, less extenisble. I wouldn't be surprised if both things I describe already exist in some form.
To be both nice to browsers and non-browser clients, all it takes is respecting the Accept header. You must assume that a client who asks for text/html is truly happy with text/html. This could be an argument for not using text/html as the media type for your non-browser API entry point. In principle, I think it could work though if the only thing you want is links. Good HTML markup can be very well consumed by non-browser clients. HTML also defines way to do paging, through rel="next", rel="previous".
The three biggest problems of a singular media type for both browsers and non-browsers I see are:
you must ensure all your site html is outputted with non-browser consumption in mind, i.e. embed sufficient metadata. Perhaps add hidden links in some places. It's a bit comparable from thinking about accessibility for visually impaired people: Though now, you're designing for a consumer who cannot read English, or any natural language for that matter. :)
there may be lots of markup and content that may essentially irrelevant to a non-browser client. Think of repeating header and footer text, navigation area's that kind of things.
HTML may simply lack the expressiveness you need. In principle, as soon as you go "think up" some conventions specific to your site (i.e. say rel="original-image means the link to the full-size, original image), then you're not doing strictly HATEOS anymore (at least, that's my understanding). HTML leaves no room for defining new meaning to elements. XML does.
A work-around to problem three might be using XHTML, since XHTML, by the virtue of being XML, does allow specifying new kinds of elements through namespaces.
I see #robert_b_clarke mentioning Microformats, which is relevant in this discussion. It's indeed one way of trying to improve accessibility for non-human agents. The main issue with this from a technical point of view is that it essentially relies on "out-of-band" information. Microformats are not part of the text/html spec. In a way, it's comparable to saying: "Hey, if I say that there's a resource with type A and id X, you can access it at mysite.com/A/X." The example I gave with rel=original-image could be called a micro-format as well. But it is a way to go. "State in your API docs: We serve nicely formatted text/html. Our text/html also embeds the following microformats: ..." You can even define your own ones.
I think the following presentation as a nice down-to-earth explanation of HATEOAS:
http://www.slideshare.net/apigee/hateoas-101-opinionated-introduction-to-a-rest-api-style
Edit:
I only now read about HTML5 microdata (because of #robert_b_clarke). It seems like HTML5 does provide a way for supplying additional information beyond what's possible with standard HTML tags. Consider what I wrote dated. :) Edit edit: It's only a draft, phew. ;)
Edit 2
Re a "descriptive JSON" format: This has just been announced http://jsonapi.org/ . They have applied for their own mime type. It's by Yehuda Katz (Ember.js) and Steve Klabnib, who's writing Designing Hypermedia API's.
The HTTP Accept header can be used by clients to request a response in a specific content type. For example, your REST API clients might request JSON data using the following header:
GET http://yourdomain.com/api/companies
Accept: application/json
So your server app can then serve JSON or HTML for the same URL depending on the value of the Accept header. Of course all your REST client apps will have to include that header, which may or may not be practical.
There are numerous alternative approaches, one of which is to serve the same XHTML content to both browsers and client apps. You can use HTML5 microdata or Microformats to embed structured data within HTML. That approach has a number of limitations. API client requests will result in larger, more complicated responses than necessary as they will include a load of stuff that's only usable by a web browser. There are also other differences in behaviour you might like to enforce. For instance you would probably want an unauthorized GET request for a protected resource to result in an HTTP 401 response for a machine client, and a redirect to login page for a web browser.
You may find that the easiest way is to be less principled and serve the human friendly and machine friendly versions of your resources through separate URLs
http://yourdomain.com/companies
http://yourdomain.com/api/companies
I've seen this question answered several ways. Some developers add a request parameter to indicate the format of the response, as in /api/companies/?rtnType=json. This method may be acceptable in a small application. It is a departure from true RESTful theology though.
The better way (in Java at least) is to use something like the Spring Framework. Spring can provide dynamic response formatting based on the media type in the HTTP request. The book "Spring in Action" (Walls, 2011) has an excellent explanation of this in chapter 11. And there are similar ways to accomplish dynamic response formatting in other languages without breaking REST.
Related
The content tends to be mixed with it's form when expressed as a HTML+CSS+JS document. Almost every modern website requires CSS and/or JavaScript to be readable. Most of them are not easy to parse automatically because they relay on web browser to render it. Sections of the document are defined using visual clues, colors and formatting. One can use HTML5 tags like <article> but those are not a part of any bigger structure as far as I know, and still can contain non-content elements.
Websites are basically apps or clients.
Is there any standard that can be used to serve content of a website that has a well defined schema? An API for websites that could be used to express content in the form that is easy to server, parse, store, cryptographically sign...
I'm aware of formats like XML and JSON but I have not managed to find any standardized way to express a blog post as a JSON document.
An example of what I have in mind:
This question can be fetched as an JSON document using Stackexchange API. The result is machine readable and easy to parse but is in not standardized. It reflects details of Stackexchange specific data structures. Other QA website will have different API, with different structure and formats even though both have questions and answers.
There are two important standards out there dealing with the semantic aspect of a web page, like the one you are looking for. Microdata and RDFa. With their aid, you can pick a certain open vocabulary to describe your data or create your own based on them.
With JSON-LD also, you can create a schema for JSON documents like the XML schema is for the XML documents.
This is probably a stupid idea, but I would like to know why.
I'm reading about REST API's, and principles such as HATEOAS. All the time, I'm wondering why people don't just use HTML for the representation of their resources.
Sure, I can think of disadvantages such as parsing difficulties and increased data, but on the other hand, it's a semantical hypermedia language with which you can separate data from presentation. Also, it's human readable and people can interact with it in the browser, follow links, submit forms, etc. It could be used as both an API and UI.
Can anyone explain why it is a terrible idea to use HTML for REST API representations?
The www uses html for REST!
There's nothing wrong with the idea at all. Personally I would congratulate you on your questioning this in the first place, many don't.
Rest does not mandate an application protocol, it's just that JSON/XML has become the standard choice (as HTML is usually hard to parse). If you use a simplified version of HTML you might actually find it more useful than JSON.
I've written several rest applications that accept both application/json and text/html for content negotiation. It allows for easy testing on a browser.
As you mention, it certainly makes HATEOAS easier!
JSON does not (currently) have a standard mechanism for dealing either with HATEOAS or with strong typing (most people use the #class way of specifying what object the json represents). JSON is in my opinion, not finished yet.
XML on the other hand is.. but what is HTML if it isn't a kind of XML?
With html :
<div name="Elvis Presley" id="1" class="com.graceland.elvis.Person">
wife
<span name="country" class="java.lang.String">USA</span>
</div>
Good luck trying to replicate that with Json. Json doesn't effectively handle 'attributes' for starters!
Can anyone explain why it is a terrible idea to use HTML for REST API
representations?
Yes
It is not well formed
How would clients parse the result consistently ?
The markup is verbose
It is not a format meant for consumption by machines. It is a view for humans. REST APIs are meant for machine consumption.
Large responses are bloated and would lead to more network latency.
As for presentation, you cannot assume that the API would be consumed by a browser. What about native Android / iOS apps ?
REST supports all kinds of content included HTML. It's clear that most of RESTful applications and Web APIs are focused on data. So such formats like JSON, XML and YAML are more convenient to build and parse.
But if you want to leverage the Conneg feature (content negociation - based on the header Accept) of REST, you can handle several kinds of content according to the caller:
a browser. Perhaps we would prefer to display an HTML content to display UI for the request. You would have: Accept: text/html.
an application. In this case, you rather expect some structured data. You would have something like that: Accept: application/json, Accept: application/xml, and so on.
In fact, it's up to the RESTful applications. I built RESTful applications that implement conneg and send back several kinds of content according the specified header Accept.
Hope it helps,
Thierry
REST is about communication between machines. HTML contains a lot of GUI elements, it contains CSS, JS, etc... as well. All of these are for humans to display the view. The machines are interested only in the data and its annotation.
Btw. it is possible to use HTML as a data transfer format by REST. For example HAL has (or just had?) a HTML serialization format and Hydra can use HTML as well e.g. with microdata.
If you are talking about HTML which can be used both by the browsers and the REST clients (which extract only the data), then I think it is usually hard to write such a HTML document.
tl;dr: If we assume that XML isn't a terrible idea for a REST API, I think it would be reasonable to use a strict subset of XHTML (JSON is a strict subset of JavaScript), especially if HATEOAS is important to your API.
The fundamental benefit of HTML for a REST API is the <a href=""> and the <form action=""> tags (you can possibly even simplify it down to just the form tag). It's defined to handle Hypermedia and it's the only well understood way of linking documents. You don't have to read a JSON-LD / HAL / Siren spec to understand the structure of the HTML.
Other's here argue against it because HTML contains <h1> tags. But you can use a strict subset of HTML rather than trying to create a superset of JSON. JSON is effectively a strict subset of JavaScript objects. Personally I think this would make an excellent REST API - easy to understand by both humans and machines.
I initially thought that microdata is close to what you want but that only handles the GET for HTTP, you need methods for handling all the other HTTP methods (hence the need for the <form> tag). If you do only care about GET requests I think that might work for you. You asked about JSON-LD in one of your comments and in the Schema.org wikipedia page you can see the similarity between micro data and JSON-LD.
microdata
<div itemscope itemtype="http://schema.org/Movie">
<h1 itemprop="name">Avatar</h1>
<div itemprop="director" itemscope itemtype="http://schema.org/Person">
Director: <span itemprop="name">James Cameron</span>
(born <time itemprop="birthDate" datetime="1954-08-16">August 16, 1954</time>)
</div>
<span itemprop="genre">Science fiction</span>
Trailer
</div>
JSON-LD
<script type="application/ld+json">+schema app
{
"#context": "http://schema.org/",
"#type": "web master",
"name": "schema.org/person",
"Struturedata":
{
"#type": "Person",
"name": "chema mpnrroy josepinedamonroy",
"birthDate": "10/19/1982"
},
"geng": "male",
"Mecanismo":microdata. ".estructuredate./" validador
}
</script>
I think the major issue is that HATEOAS doesn't provide enough tangible benefit to developers, they just want to transfer data not have a self-discoverable API. The self-discovery just isn't that important because someone interfacing with your API only needs to discover the relevant URL once and then as long as your API doesn't change they don't have to care any more. Further even if you did write a fully HATEOAS supported REST API, the main benefit is supposed to be that clients don't need to hard code URLs and so it doesn't matter if you change the URLs. However you've no way of preventing API clients from not hard-coding the URLs and so if you ever do change the structure then you're going to have unhappy clients. Take the web for example, it's a (mostly) properly implemented REST API but link rot is still a major issue because everyone still relies on fixed URLs.
Then if links aren't that important, the simplicity of JSON wins out. Being able to represent both arrays and objects so naturally is hard to argue against. The entire spectrum of programming languages care fundamentally about arrays (lists) and objects (dictionaries/maps). The fact that you can't simply represent an array in XML or HTML is a major drawback.
Another problem against it, is that a large proportion of web developers are programming in JavaScript and then it's a no brainer to interop with JSON and you have to have major benefits to persuade your boss to use something else.
I've seen two primary ways to add JSON REST Hateoas and I'm not sure which is more standard or the pros and cons of each approach.
The typical approach I see (Atom Links) is that the returned entity is appended to with a field named either links or _links. This field is an array of rel=<rel> and href=<href> pairs.
But I've also seen (Link Headers) links put into the header value named "Link". The Link is a collection with the format <hef>; rel=<rel>.
Also, I noticed that in JAX-RS there doesn't seem to add Atom Links with fully qualified hrefs, only paths. By fully qualified I mean scheme and authority included. Is it looked on as bad practice to have a complete URI for the href when using Atom Links for HATEOAS?
All the HATEOAS formats i know use the link relationship RFC https://www.rfc-editor.org/rfc/rfc5988 to abstractly define a link relationship. This rfc describes the Link header which is a fine way for conveying link relationships. Other formats serialize/present links in different ways. _links is probably most associated with the HAL+JSON format, while links is used by Siren and COLLECTION+JSON (which also allows for link headers).
It's all a matter of preference. Many times it comes down to asking if you think of link relationships as metadata of the resource or actually part of the resource. Sometimes it's both. HTML primarily treats them as part of the relationship and has been wildly successful with that. Having them in the response body of the resource makes it very easy to see them in a browser, headers are a little trickier to see.
Regarding URLs being absolute, scheme relative, root relative, path relative. That again is all preference. Something i like to keep in mind is that a resource is not always retrieved from a request, thus relative paths often can be useless. For example, storing a resource in a cache or on disk. Absolute or scheme relative URLs are much more portable across systems and I personally prefer them over root or path relative URLs in most cases. There are interesting scenarios where you may actually want URLs to be relative so they target different destinations depending on the executing environment. There was an interesting discussion on this recently in the HAL forum: https://groups.google.com/forum/#!topic/hal-discuss/_rwYvjLOT7Q
Adding another answer because HATEOAS links in headers have bitten us. If we had the option to do it again we would surely use links in the payload instead of the headers
The main reasons are:
When querying a collection, getting links for each of the records requires some special way to allow you to know which of the records a link is for.
<cry>We misused the title attribute to be a record ID </cry>
You can easily get a 431 Request Header Fields Too Large error if you have too many links.
<scream>Our collections don't send the all the links because of that limitation, which means we have to fetch every instance separately to have access to all the actions.<\scream>
I have a situation where the underlying application provides a UI layer and this in turn has to be rendered as a portlet. However, I do not want all parts of the UI originally presented to be rendered in Portlet.
Proposed solution: Using Datapower for parsing an XML being a norm, I am wondering if it is possible to parse a HTML. I understand HTML may not be always well formed. But if there are very few HTML pages in underlying application, then a contract can be enforced..
Also, if one manages to parse and extract data out of HTML using DP, then the resultant (perhaps and XML) can be used to produce HTML5 with all its goodies.
So question: Is it advisable to use Datapower to parse an HTML page to extract an XML out of it? Prerequisite: number of HTML pages per application could vary in data but not with many pages.
I suspect you will be unable to parse HTML using DataPower. DataPower can parse well-formed XML, but HTML - unless it is explicitly designed as xHTML - is likely to be full of tags that break well-formedness.
Many web pages are full of tags like <br> or <ul><li>Item1<li>Item2<li>Item3</ul>, all of which will cause the parsing to fail.
If you really want to follow your suggested approach, you'll probably need to do something on a more flexible platform such as WAS where you can build (or reuse) a parser that takes care of all of that for you.
If you think about it, this is what your web browser does - it has all the complex rules that turn badly-formed XML tags (i.e. HTML) into a valid DOM structure. It sounds like you may be better off doing manipulation at the level of the DOM rather than the HTML, as that way you can leverage existing, well-tested parsing solutions and focus on the structure of the data. You could do this client-side using JavaScript or you could look at a server-side JavaScript option such as Rhino or PhantomJS.
All of this might be doing things the hard way, though. Have you confirmed whether or not the underlying application has any APIs or web services that IT uses to render the pages, allowing you to get to the data without the existing presentation layer getting in the way?
Cheers,
Chris
Question of parsing and HTML page originates when you want to do some processing over it. If this is the case you can face problems because datapower by default will not allow hyperlinks inside the well formed XML or HTML document [It is considered to be a security risk], however this can be overcome with appropriate settings in XML manager present.
As far as question of HTML page parsing is concerned, Datapower being and ESB layer is expected to provide message format translation and that it indeed does. So design wise it is a good place to do message format translation. Practically however you will face above mentioned problem when you try to parse HTML as XML document.
The parsing can produce any message format model you wish [theoretically] hence you can use the XSLT to achieve what you wish.
Ajitabh
If there is no DTD or schema to validate the H5 document against, how are we supposed to do document validation? And by document validation, I mean "how are we supposed to ensure our html5 documents are both syntactically accurate and structurally sound?" Please help! This is going to become a huge problem for our industry if we have no way to accurately validate HTML5 documents!
Sure, the W3C has an online tool that validates individual pages. But, if I'm creating A LOT of pages (hundreds, say) and I want to validate them in a sort of batch mode, what is the accepted method of ensuring valid structure and syntax? I mean, it seems rather rudimentary to just look at the document and say "yep. that's a valid xml document." What about custom tags? What about tag attributes? It seems like the W3C is leaving us out in the cold a little bit here.
Maybe the best answer will be found in the HTML editor. But then you get DTD/schema fragmentation. Each editor vendor coming up with their own rendition of what a valid structure is.
Maybe the answer is "wait for HTML5 to become official". But I really can't wait for that. I need to start creating and validating content now. I have applications I want to publish that can only be accomplished with html5.
So, any thoughts?
If there is no DTD or schema to validate the H5 document against, how are we supposed to do document validation?
With a specialized HTML5 validator rather then a generic SGML or XML validator.
Obviously, as the specification is still in draft form, the tools that do exist are immature and likely to be out of date or become out of date.
Sure, the W3C has an online tool that validates individual pages. But, if I'm creating A LOT of pages (hundreds, say) and I want to validate them in a sort of batch mode, what is the accepted method of ensuring valid structure and syntax?
Either use a different tool or download the W3C validator and run a local copy. It has a SOAP API so writing a batch validation tool isn't difficult.
What about custom tags?
HTML5 doesn't allow custom elements.
What about tag attributes?
The only custom attributes in HTML5 are data-* attributes, so an HTML 5 validator can recognize them.
It seems like the W3C is leaving us out in the cold a little bit here.
It seems like you expect the state of QA tools for HTML 5 (unfinished) to be up to the same standard as those for HTML 4 (over a decade old). This isn't a realistic expectation.
Maybe the best answer will be found in the HTML editor. But then you get DTD/schema fragmentation. Each editor vendor coming up with their own rendition of what a valid structure is.
The specification is clear (although in flux) even if it isn't expressed in the form of a DTD or schema. If each editor has a different idea of what is valid, then most or all of them are going to be either out of date or just buggy.
Maybe the answer is "wait for HTML5 to become official". But I really can't wait for that. I need to start creating and validating content now. I have applications I want to publish that can only be accomplished with html5.
If you need to live in the bleeding edge, then you have to accept the limitations and risks of doing so.
You might find this question/answer interesting: Will HTML 5 validation be worth the candle? . The answer is written by the developer of http://about.validator.nu/ .
You should start by taking a look at http://about.validator.nu/ .
Some, though not all, of your concerns are addressed there. You can host your own validator, there's a python based submission script, you can use a RESTFUL web service API and there are ways to get validation output in a variety of different forms.
I can't however see a simple way to integrate XHTML5 with other applications of XML such that one can easily create a validator of such compound documents. Not that there's really been a way to do that with earlier versions of XHTML either though.
This is working well for me: https://github.com/hober/html5-el
To get this to work, I renamed the default '/etc/schema/schemas.xml' file in order to move it out of the way and let the 'html5-el' one be used by nxml-mode.
If there is no DTD or schema to validate the H5 document against, how are we supposed to do document validation? And by document validation, I mean "how are we supposed to ensure our html5 documents are both syntactically accurate and structurally sound?" Please help! This is going to become a huge problem for our industry if we have no way to accurately validate HTML5 documents!
If testing pages with either Firefox or Opera, both of those will report errors such as code that is not "well-formed" and mismatched tags. Beyond that, one of the validators such as validator.w3.org or validator.nu will definitely help.
Sure, the W3C has an online tool that validates individual pages. But, if I'm creating A LOT of pages (hundreds, say) and I want to validate them in a sort of batch mode, what is the accepted method of ensuring valid structure and syntax? I mean, it seems rather rudimentary to just look at the document and say "yep. that's a valid xml document."
There are ways to run the W3C validator in batch mode.
What about custom tags? What about tag attributes? It seems like the W3C is leaving us out in the cold a little bit here.
The easy answer to that one is that "custom tags" are simply not considered valid. The Working Group has thoroughly addressed the issue of "distributed extensibility", particularly with respect to allowing "decentralized
parties to create their own languages" and "extension attributes" (http:// lists.w3.org/Archives/Public/public-html/2011Feb/0085.html). There are numerous ways to extend HTML (http://www.whatwg.org/specs/web-apps/current-work/multipage/infrastructure.html#extensibility) but adding custom tags is not one of them. Custom data and microdata attributes should validate fine.
Maybe the answer is "wait for HTML5 to become official". But I really can't wait for that. I need to start creating and validating content now. I have applications I want to publish that can only be accomplished with html5.
Since HTML 5 was stabilized at the end of last year (Dec. 2010), IMO we don't need to wait for it to become an official "recommendation" by the W3C. The stabilized spec provides a solid base that all browser vendors can implement consistently and for the ongoing evolution beyond HTML 5 of the spec, which is now being called the "HTML Living Standard" (Jan. 2011 and later). There is a good diagram of this at http://www.HTML-5.com/html-versions-and-history.html#html-versions (scroll down to see the diagram).