Is there a tool like Google's Protobuf for JSON? I know you can convert from a Protobuf format to JSON but that requires a whole lot of extra serialization/deserialization, and I was wondering if there is some kind of tool that lets you specify the structure of a JSON message and then automatically generates libraries for use in a specified language (direct serialization/deserialization not just a wrapper around Protobuf's JSON formatter class)
I know nearly all languages provide their own in house way of handling JSON, and many higher level ones even allow you to avoid the boiler plate parsing code, but I was looking for a universal tool where you would only need to specify the format once, and then just get the generated libraries for use in multiple languages.
The Protobuf equivalent would be JSON-Schema, but still is language dependent on having a serializer or code generator available, just as Protobuf is.
If you're looking at making a REST-API, then OpenAPI Spec + swagger-codegen could be an option.
Related
Morning!
I've got an app with a config file that's become unwieldy - many switches with no intuition as to which combinations are valid. Right now, all the switches are stored in an XML file. The config file specifies inputs for a large HPC job.
I'm thinking of writing some a formal grammar for a run - that is, the sort of combinations that are acceptable, and from the parsing of it, the switches needed will automatically be inferred. The values would still be read from the XML file, but only when needed.
Is this sort of approach reasonable? How would I go about implementing a grammar without a parser?
If I understand you correctly, you want to implement a Domain Specific Language (DSL), the purpose of which is to specify validation rules for the contents of an XML-based configuration file.
Some people implement a DSL by defining a parser specific to the needs of the DSL. However, some other people shoehorn the semantics of their DSL into the syntax of an existing file format, such as XML or JSON. So if you want to avoid having to write a parser, you could express your DSL in XML syntax.
I'm aware that there are python and powershell methods to convert plain text files, csv's etc.... into json format for upload into NoSQL DBs such as CouchDB.
But according to the CouchDB definitive guide, it makes it seems like there is a native built in way of doing this kind of conversion, without the need for a 3rd party tool.
This older thread appears to hint at this:
Filter and update functions in CouchDB?
This part in particular:
There are other design document functions that are being introduced at the >time of this writing, including _update and _filter that we aren’t covering in >depth here. Filter functions are covered in Chapter 20, Change Notifications. >Imagine a web service that POSTs an XML blob at a URL of your choosing when >particular events occur. PayPal’s instant payment notification is one of >these. With an _update handler, you can POST these directly in CouchDB and it >can parse the XML into a JSON document and save it. The same goes for CSV, >multi-part form, or any other format.
But when I dig deeper I don't find anything concrete.
The supporting wiki link is not clear to me (a beginner with json/NoSQL/curl stuff: http://wiki.apache.org/couchdb/Document_Update_Handlers
Hopefully this is a simple yes/no. And any links to help on this that is better than the above link also appreciated, thank you!
CouchDB supports transforming the internal documents/views into many other formats through the use of show and list functions. It's not a "native" transformation, as you define the transformation yourself, it's not magical.
That being said, there is not a similar mechanism for the reverse (ie: converting some arbitrary format into JSON documents) but you're much better off scripting that with a full-featured language/script and using the bulk docs API to do your imports in batch.
Following is my requirement:
Application A is creating a JSON based on its Java Beans and sending to my Application.
I have to take this JSON and convert it into XML (XSD for this is completely different than my JSON structure) and send to Application B.
Solution 1) I am currently converting this json to xml using json.org library.Then using Apache-xalan and XSL stylesheet, I am converting this to xml format as required by App B.
Solution 2) Converting this json to Java Bean (JB1).Then converting this JB1 to another Java Bean (JB2) as per the xml structure required by Application B.Then convert JB2 to XML for app B.
Solution 3) Using Apache Xalan and Xerces to parse through the input json and make the XML in Java itself without using XSL.
Which is better approach (in simplicity of code, throughput )? As JSON becomes more complex, is it easy to use solution 1 ? Please suggest if there is better approach other than these 3 ?
XSLT 3.0 offers a built-in json-to-xml() function. Once you have the XML, you can easily transform it to your required format. It is implemented in Saxon 9.7 (PE or higher) and I believe in Exselt.
Solution 1: Yes. This is the conventional and best path for both simple and complex JSON and simple or complex targeted XML.
Solution 2: No. There's no reason to introduce Java Beans as an intermediate form, especially if you have no other need for Java Beans. This option unnecessarily introduces transformational and marshalling complexity.
Solution 3: No. Neither Xalan nor Xerces are designed to parse JSON; they are designed to parse XML.
There are sample programs that will map a JSON document into an equivalent XML document and back; I wrote one as a demo for Liberty's support of json-p (javax.json), using an XML vocabulary I called JinX (JSON in XML). That could be used as a pre/post processor wrapped around XSLT, if desired.
Better solutions are possible -- redefine XSLT to operate on JSON trees, for example -- but would take a bit more work.
JSON is, pure-and-simple, "a communications protocol." In other words, "it specifically exists(!) to allow 'arbitrary (JavaScript) data structures' to be conveyed between some-client and some-host," over "the HTTP(S) protocol."
Therefore: "it is not(!) XML," and therefore must never be considered to be "appropriate input to XSLT!"
"Thou shalt not mix Apples and Oranges!"
If you wish to apply "XSLT" technologies to a "JSON-derived" input (which is, by definition, "a data structure ...") then you must first, and "by whatever suitable means," convert that data structure into XML.
According to a post on Stackflow.com called “what’s is JSOn and why would I use it? “web services used XML as their primary data format for transmitting back data, but since JSON appeared, it is preferred method.” Why do must web services use JSON over XML, is because it’s a better method for interchanging?
XML was designed primarily for document formats, e.g. papers in scientific journals. It contains many features that aren't needed for simple data interchange, and these features can get in the way when you are processing XML because they can't be easily represented in Javascript. So the code for processing the XML ends up a lot more complicated than it could be. By contrasts, JSON has an exact match to the data structures Javascript can handle natively. Of course, that problem could in principle be solved by using a language with better XML support than JavaScript - XSLT, for example - but unfortunately XSLT in the browser has never had the same level of investment put into it.
Additionally, for reasons I have never understood, the browser security folks decided that reading JSON from alien web sites (i.e. from a different domain from your HTML page) is safe, but reading XML from alien sites isn't. So if you switch from XML to JSON, you get rid of a lot of cross-site-scripting hassle.
JSON is less verbose and it is sufficient for simple data transmission, i.e. if you do not need any transformations (XSLT).
We are designing a fairly complex REST API, in which most of the I/O are JSON encoded objects with a specific structure. One challenge we have found is to document the API in such a way that makes it easier for clients to post correct input and process output. Because the data of both the input and output requires fairly complex JSON objects, client developers often introduce bugs related to the structure of the I/O objects.
With all of the JSON web API's these days, I would have hoped for a general solution, but I am having a hard time finding one. I looked into json-schema which is a json-validation schema but both the IETF draft and implementations seem to be fairly immature (even though they have been around for a while, which is not a good sign).
A slightly different approach is offered by Protocol Buffers and Apache Avro, where the schema is not used for validation, but actually required for the encoding/decoding of the message. Of these 2, Avro seems to have rather limited documentation and implementations. ProtoBuf seems better, but I am not sure if this is really suitable to use in the browser to call a JSON api?
Now I am starting to doubt if I am looking at this from the right angle. Are there other methods available to make my API a bit more strong-typed'ish? Or is a formal description of a JSON REST/RPC API something that defeats the purpose of using JSON?
Edit: 6 months after this topic we found mongoose, which is very close to what we were lookin for.
Below a reply I received by email from Douglas Crockford.
I am not a believer in schemas as an alternative to input validation.
There are properties that cannot be verified from the syntax. I think
that was one of the ways that XML went wrong.
If your formats are too complex, then I would look at simplifying
them.
Such systems exist and I'm the author of one of them. It is called Piqi-RPC and it does IDL-based validation of the input and output parameters for RPC-style APIs over HTTP.
It supports JSON, XML and Google Protocol Buffers as data representation formats for input and output of HTTP POST requests. Clients can choose to use any of the three formats and specify their choice using the standard Accept and Content-Type HTTP headers.
So, yes, in theory, you are looking in the right direction. However, at the moment, Piqi-RPC supports writing servers only in Erlang and it wouldn't be very useful for you if you use a different stack. I heard that Apache Thrift also supports JSON over HTTP transport, but I haven't checked. Another kind of similar system I know of (also for Erlang) is called UBF. I have heard of libraries for Java that can parse and validate JSON based on Protocol Buffers specification (e.g. http://code.google.com/p/protostuff/).
The idea itself is far from being new, but there aren't many systems that approach it in practice. It is a challenging problem.
Historically, IDLs were used for interface definition and binary data serialization and not so much for validating dynamic data interchange formats (e.g. XML and JSON) which emerged later. Sun-RPC IDL and CORBA IDL fall in the first category. WSDL would be one of few examples covering both areas, but it is a terrible piece of technology and it would be a bad choice for most modern systems. In addition, there are many schema languages (also known as DDLs -- data definition languages), most of which are highly specialized and work with only one representation format, e.g. XML or JSON schemas. Few of those have stable implementations.
The Piqi project and Piqi-RPC, which is based on it, are build around several fairly simple realizations:
DLL doesn't have to be explicitly tied to any particular data representation format or built around it. Instead, such language can be fairly universal and cover wide range of practical use-cases (e.g. cross-language data serialization and data validation) and data formats (e.g. JSON, XML, Protocol Buffers).
IDL for RPC-style communication can be implemented as a thin, mostly syntactic layer on top of the universal DDL.
Such IDL and interface specifications can be transport agnostic.
Speaking of REST-style APIs over HTTP compared to RPC-style APIs over HTTP.
With RPC-style APIs, service developer or an automated system have to validate three things: function name (according to some service naming scheme), input and, if you choose so, output.
In case of REST-style APIs, people get themselves in trouble for no good reason. Now, they have a lot more stuff to validate: arbitrarily complex URL syntax, including dynamic parameters encoded in URL segments (for all HTTP methods) and URL query string (only for HTTP GET method), HTTP method correspondence (whether it should be GET, POST, PUT, DELETE, etc.), HTTP body when some parameters go there (sometimes they do it manually twice for parameters represented in JSON and XML), custom HTTP headers, and separately -- service documentation. Imagine an IDL supporting all that!
XML is better for RESTful services in many ways. It has native linking (<link href=, for all those HATEOAS fans), native language support (lang="en") and a great ecosystem.
It is also better for future proofing and future API refactorings. Converting this:
<profile>
<username>alganet</username>
</profile>
To support more usernames:
<profile>
<username>alganet</username>
<username>alexandre</username>
</profile>
Is much more simpler to do without breaking existing clients using XML. JSON is hard on that.
If you really need JSON, JSON-Schema is the way to go. It's immature, but I don't know anything better on that case. Maybe your consumers could choose between XML and JSON, so they can choose between a small payload (JSON) or RESTful candies (XML) using Content Negotiation.
I'd say the answer to your last question is yes. If you need a way to constrain and document the JSON "schema", why didn't you go with XML in the first place? It is not that much harder to parse, and being able to enforce a schema for it is a great advantage.