I have a following use case :
parse the json from stream (kafka topic)
extract some fields (likely 35 out of 100 fields)
Build a json out of those fields
Publish it to pub/sub for further processing
My implementation is very much java language bound. Can anyone suggest optimal solution for this ? and why is it optimal ?
For json parsing, I am thinking of https://bolerio.github.io/mjson/
Kafka includes Jackson JSON library and includes its own JSON Deserializer that returns a JsonNode class
Alternatively,as listed in the comments, you can use higher level frameworks such as Spring, Vertx, Quarkus, etc to build Kafka consumers
For the listed use case, I would opt for Spark, Flink, or NiFi for integration with PubSub. Each also offering JSON proessing, with NiFi being more advanced in that it can do JSONPath
Related
I want to parse JSON data from a RESTful service.
Unlike a SOAP-based service, where a service consumer can create stubs and skeleton from WSDL, in the case of the RESTful service, the service consumer gets a raw JSON string.
Since the service consumer does not have a Java object matching the JSON structure, we are not able to use the JSON to Java Mappers like GSON, Jackson etc.
One another way is to use parsers like JsonPath, minimal-json, etc which help traversing the JSON structure and read the data.
Is there any better way of reading JSON data?
The official docs for Jackson mention 3 different ways to parse a JSON doc from Java. The first 2 do not require "Java object matching the JSON structure". In Summary :
Streaming API (aka "Incremental parsing/generation") reads and writes JSON content as discrete events.
Tree Model provides a mutable in-memory tree representation of a JSON document. ObjectMapper can build trees that consist of JsonNode nodes.
Data Binding converts JSON to and from POJOs based either on property accessor conventions or annotations.
With simple data binding you convert to and from Java Maps, Lists, Strings, Numbers, Booleans and nulls
With full data binding you convert to and from any Java bean type (as well as "simple" types mentioned above)
Another option is to generate Java Beans from JSON documents. You mileage may vary and you may/probably will have to modify the generated files. There are at least 5 online tools for that purpose that you can try:
http://www.jsonschema2pojo.org/
http://pojo.sodhanalibrary.com/
https://timboudreau.com/blog/json/read
http://jsongen.byingtondesign.com/
http://json2java.azurewebsites.net/
There are also IDE plugins that you can use. For instance this one for Intellij https://plugins.jetbrains.com/idea/plugin/7678-jackson-generator-plugin
The GSON supports work without objects, too. Something as this:
JsonObject propertiesWrapper = new JsonParser().parse(responseContent).getAsJsonObject();
assertNotNull(propertiesWrapper);
propertiesWrapper = propertiesWrapper.getAsJsonObject("properties");
assertNotNull(propertiesWrapper);
JsonArray propertiesArray = propertiesWrapper.getAsJsonArray("property");
assertNotNull(propertiesArray);
assertTrue(propertiesArray.size()>0, "The list of properties should not be empty. ");
The problem is that the work this way is so inconvenient that it is really better to create objects instead.
Jackson has absolutely the same problems, and to greater extent - extremal inconvenient for direct json reading/creation. All its tutorials advice to use POJOs instead, too.
The only really convenient way is use Groovy. Groovy works as an envelope on Java, you can simply write Java code and use Groovy operators at need. And in JSON or XML reading and creation Groovy is incomparably more powerful that Java with all its libraries multiplied on each other! It is even much more convenient than already prepared by somebody else tree structure of ready POJOs.
Does smooks support json output or thirdparty plugin for json?
For example, to do XML-to-JSON or EDI-to-JSON
I see it has json reader/parser, but can't seem to find an output/writer.
TIA!
Not directly, but it would support populating of a Java Object model from e.g. XML or EDI and then you could use something like Jackson to serialize the Java to JSON. So should be easy enough to do once you get the data into a Java Object model.
Also note that you do not need to create an actual Java Object model. You can create what Smooks calls a "virtual object model", which is basically collection types (Maps, Lists etc).
I need to deserialize some JSON objects. I tried to use Tiny-json library, but it's too slow. I tried to use Newtonsoft.Json, but it fails in webplayer with this error:
MissingMethodException: Method not found: 'System.Collections.ObjectModel.KeyedCollection.
What JSON parser do you recommend?
You can try one of these open source solutions:
https://github.com/jacobdufault/fullserializer
https://github.com/mtschoen/JSONObject (https://www.assetstore.unity3d.com/en/#!/content/710) I am using this one most of the times, it's versbose but does its job well, not sure about performance however
Or go with paid ones:
https://www.assetstore.unity3d.com/en/#!/content/11347
Unity 5.3 added Native support of Json Serializer. It is faster than others.
JsonUtility.ToJson to convert a class to Json.
JsonUtility.FromJson to convert Json back to class.
For complete example and information regarding json arrays, see
Serialize and Deserialize Json and Json Array in Unity
I'm want to try changing a SOAP WCF to accept requests and return results in JSON format to make the data traffic less bulky.
I see that JSON requests functions looks like this:
wcfClient.OpenReadAsync(http://yourUrl.com/wcf/service1.svc/GetEmployees)
and do the regular SOAP requests functions instead that looks like :
wcfClient.GetEmployeesAsync();
1) For JSON results, do you need to parse them into an object or is it automatically parsed like SOAP?
2) Is there a way to do this without doing too much work like changing every single WCF calls in the project to looks "JSON-ish"?
To complement Davut's answer - WCF does support building RESTful services, although I agree that the ASP.NET Web API framework in general easier to use than WCF. JSON.NET is a great library, and it has nice deserialization capabilities (e.g., it can easily take the JSON which represent the list of Employee objects and convert them into the actual List<Employee> instance)
But for completeness sake, if you want to use a "normal" WCF client to access WCF-based services which return JSON, you can do it. It's not too straightforward, but you can do that by using a new encoder and behavior which does the conversion. The post at http://blogs.msdn.com/b/carlosfigueira/archive/2010/04/29/consuming-rest-json-services-in-silverlight-4.aspx talks more about it, and has a pointer to a code sample.
In short, it's possible to consume JSON using a WCF client in Silverlight, but due to its complexity it's usually not done, and Davut's option (use a HTTP client such as WebClient to download JSON, then a library such as JSON.NET to parse it into objects) is preferred.
Firstly the idea "make the data traffic less bulky." is good.
Especially for Mobile devices. Beside this don't think that WCF xml causes network issues for PC. XM is the one of most compressible format. By WCF binary it goes as compressed.
For "Is there a way to do this without doing too much work?"
Yes there is a way name on it RESTFul Services(Restless Services). Now Microsoft directly support it by WEBApi.
Also you may use ODATA for filtering,ordering operations
Here are some links,
http://msdn.microsoft.com/en-us/library/system.servicemodel.web.webgetattribute.aspx
http://blogs.msdn.com/b/rjacobs/archive/2010/06/14/how-to-do-api-key-verification-for-rest-services-in-net-4.aspx
ODATA
http://www.odata.org/documentation/uri-conventions#FilterSystemQueryOption
A few practice notes,Some restrictions:
EntityFrameWork entities derived from EntityObject which has IsReferenceType attribute doesn't allow you to JSON serialize. ( I produced POCO objects using an automapper mapped them and serialized json)
WEBAPI support you much think such as WebGet,WebInvoke GetXML Give JSON ,ODATA features(just select and format not allowed.)
Note:In your web request's header you should accept text/json to get really json.
"For JSON results, do you need to parse them into an object or..."
I can say you should try JSON.NET it's portable library works everywhere. When you deserialize with a generic function it returns you the collection you expect.
Hope it helps someone. While discovering these stackoverflow helped me like an assistant.
In my GWT project I need to process json data retrieved from a database via PHP. I have seen the Google examples using JavaScriptObject overlay classes. What I don't understand is why this seems to be the prefered method of processing the json data. Why shouldn't I use all native Java code to pull in the data?
Think about it the other way around: what does it mean to use POJOs? (or native Java classes as you name them)
You have to:
parse the JSON into some Java-accessible structure (e.g. com.google.gwt.json.client.JSONObject, or elemental.json.JsonObject)
create POJOs
fill the POJOs with the data from the parsed JSON structure
now you can forget the parsed JSON structure from step 1
On the other hand, with JavaScriptObject, you use JsonUtil.safeEval and TA-DA! you get your JSON parsed right into a typed Java object!
Now, to deal with JSON, there's also AutoBeans.
Choose your poison.