I'm building a Java application in which there are some JSON objects (in particular AnyJson objects from com.satori.rtm.model.AnyJson) and I want to send these objects in a Kafka Topic. Should I send them in AnyJson o String type? I'm asking this because the constructor KafkaProducer<K, V> seems to have some problem to handle JSON values when (de)serialization.
This in my producer configuration
Properties props= new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaBrokerEndpoint);
props.put(ProducerConfig.CLIENT_ID_CONFIG, "KafkaProducer");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, IntegerSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class);
kafkaProducer= new KafkaProducer<Integer, AnyJson>(props);
where JsonSerializer.class is from org.springframework.kafka.support.serializer.JsonSerializer;
I have seen there are multiple packages that handle the JSON objects (as kafka.utils.json, com.google.gson.JsonObject;, etc).
Then when running
ProducerRecord<Integer, AnyJson> record= new ProducerRecord<Integer, AnyJson>(topic, json);
kafkaProducer.send(record);
I have the following exception
No serializer found for class com.satori.rtm.connection.GsonSerializer$JsonElementWrapper
Any help?
You may want to use org.apache.kafka.common.serialization.ByteArraySerializer for key and value serializations.
Now you should configure your producer record with byte[].
Then use ObjectMapper from Jackson (http://www.baeldung.com/jackson-object-mapper-tutorial ) to convert any json object to byte array.
Related
I'm using Flink to process some JSON-format data coming from some Data Source.
For now, my process is quite simple: extract each element from the JSON-format data and print them into log file.
Here is my piece of code:
// create proper deserializer to deserializer the JSON-format data into ObjectNode
PravegaDeserializationSchema<ObjectNode> adapter = new PravegaDeserializationSchema<>(ObjectNode.class, new JavaSerializer<>());
// create connector to receive data from Pravega
FlinkPravegaReader<ObjectNode> source = FlinkPravegaReader.<ObjectNode>builder()
.withPravegaConfig(pravegaConfig)
.forStream(stream)
.withDeserializationSchema(adapter)
.build();
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
DataStream<ObjectNode> dataStream = env.addSource(source).name("Pravega Stream");
dataStream.???.print();
Saying that the data coming from Pravega is like this: {"name":"titi", "age":18}
As I said, for now I simply need to extract name and age and print them.
So how could I do this?
As my understanding, I need to make some customized codes at ???. I might need to create a custom POJO class which contains ObjectNode. But I don't know how. I've read the official doc of Flink and also tried to google about how to create a custom POJO for Flink but I can't still figure out clearly.
Could you please show me an example?
Why don't You simply use something more meaningful instead of JavaSerializer? Perhaps something from here.
You could then create a POJO with the fields you want to use and simply deserialize JSON data to Your POJO instead of ObjectNode
Also, if there is some specific reason that You need to have ObjectNode on deserialization then You can simply do something like :
//I assume You have created the class named MyPojo
dataStream.map(new MapFunction<ObjectNode, MyPojo>() {
ObjectMapper mapper = new ObjectMapper();
#Override
public MyPojo map(final ObjectNode value) throws Exception {
mapper.readValue(value.asText(), MyPojo.class)
}
})
I have a project where I need to support serialising model classes to JSON using both Jackson and GSON (switchable).
Besides simply converting these model classes to JSON I also need to manipulate the by adding/removed certain keys (as per user input).
I was thinking of converting my model classes (and the logic to remove/add data) to simple java.util.Maps and convert these to JSON. Which works.
To convert these model classes to the java.util.Maps for Jackson I'm doing:
Map<String, Object> jsonMap =
mapper.convertValue(object, new TypeReference<Map<String, Object>() {})
and for GSON:
JsonElement jsonElement = gson.toJsonTree(object);
Map<String, Object> jsonMap =
gson.fromJson(jsonElement, new TypeToken<Map<String, Object>>(){}.getType());
After manipulating the data in the map, I do the following to actually serialize the map to JSON string:
// Jackson
mapper.writeValueAsString(jsonMap);
// GSON
gson.toJson(jsonMap);
Is there a performance hit to convert a model Object to a Map instead of a Jackson JsonNode (mapper.convertValue(object, ..) vs. mapper.valueToTree(object))?
Would it be possible to directly convert a model Object to a Map using GSON? Now I'm first creating a JsonElement and then converting it to a Map.
Jackson's ObjectNode uses LinkedHashMap internally (and ArrayNode uses ArrayList), so there should not be much difference between these approaches. JsonNode model can optimize some things slightly better than "raw" Lists and Maps; but on the other hand there is thin wrapper that adds some memory usage... so in the end things probably even out.
What are the significant diffs between:
org.json.JSONObject and javax.json.JsonObject?
Most importantly are they interchangeable from client to Webservice?
ie. can I send JSONObject to Webservice and have the Webservice believe the type is JsonObject (and vice versa)?
(JSONObject found in the json-20080701.jar of ACRA)
(JsonObject found in C:\glassfish4\glassfish\modules\javax.json.jar)
What are the significant diffs between org.json.JSONObject and
javax.json.JsonObject?
javax.json.JsonObject is included in Java EE 7
javax.json.JsonObject is immutable
org.json.JSONObject has siginificantly more convenience methods
Most importantly are they interchangeable from client to Webservice?
ie. can I send JSONObject to Webservice and have the Webservice
believe the type is JsonObject (and vice versa)?
Of course this should work. It is not the class instance which gets transferred to the webservice, but the JSON data, which is generated from the class instance. On the other side, the JSON data can be parsed back into any kind of object.
Example:
If you have a simple class named Person:
public class Person {
private String name = "Hans";
private int age = 26;
}
This could be transformed into JSON similar to: {"name":"Hans", "age":25}
The generated JSON string is sent to the webservice.
Now, on the other end of your application, or in any other application, this JSON string can be parsed into any class, if you have an appropriate parser. You don't even need Java to parse it.
Is it possible to configure jackson to always add the type of the serialized object to the generated json output.
For example:
package org.acme;
class ClassA
{
String a;
String b;
}
and I want the generated json to be:
["org.acme.ClassA",{"a":"str1","b":"str2"}]
You can do that with enableDefaultTyping() of the ObjectMapper
e.g.
mapper.enableDefaultTyping(DefaultTyping.OBJECT_AND_NON_CONCRETE);
See ObjectMapper API
If your are free to change from Jackson and do not especially need the format to match the one your are showing you can try Genson http://code.google.com/p/genson.
For example if your requirement is to be able to deserialize interfaces or abstract classes based on the original type of the object you serialized you can do:
interface Entity {}
static class Person implements Entity {}
Genson genson = new Genson.Builder().setWithClassMetadata(true).create();
// json will be equal to {"#class":"my.package.Person"}
String json = genson.serialize(new Person());
// and now Genson is able to deserialize it back to Person using the information
// in the Json Object
Person person = (Person) genson.deserialize(json, Entity.class);
Another nice feature is the ability to define aliases for your classes, so you show less information in the json stream but also this allows you to do refactoring without worring of existing json streams (for example if you store it in a database).
Genson genson = new Genson.Builder().addAlias("person", Person.class).create();
// json value is {"#class": "person"}
String json = genson.serialize(new Person());
Have a look at the wiki.
I am trying to host a WCF service that responds to incoming requests by providing a json output stream. I have the following type
[DataContract]
[KnownType(typeof(List<HubCommon>))]
[KnownType(typeof(Music))]
[KnownType(typeof(AppsAndPlugins))]
[KnownType(typeof(Notifications))]
[KnownType(typeof(Scenes))]
[KnownType(typeof(Skins))]
[KnownType(typeof(Ringtones))]
[KnownType(typeof(Alarms))]
[KnownType(typeof(Widgets))]
[KnownType(typeof(Wallpapers))]
[KnownType(typeof(Soundsets))]
public class HubCommon{}
In my *.svc.cs file I do the following
List<HubCommon> hubContent = _ldapFacade.GetResults(query);
MemoryStream stream = new MemoryStream();
DataContractJsonSerializer serializer = new DataContractJsonSerializer(typeof(HubCommon));
serializer.WriteObject(stream,hubContent);
So essentially I am trying to serialize a List to Json but I get the following error on the "WriteObject" execution:-
The server encountered an error processing the request. The exception message is 'Type 'System.Collections.Generic.List`1[[HubContentCore.Domain.HubCommon, HubContentCore, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null]]' with data contract name 'ArrayOfHubCommon:http://schemas.datacontract.org/2004/07/HubContentCore.Domain' is not expected. Add any types not known statically to the list of known types - for example, by using the KnownTypeAttribute attribute or by adding them to the list of known types passed to DataContractSerializer.'
What am I missing here ?
Thanks in advance.
The type of your DataContractJsonSerializer is HubCommon but you are writing an object of type List<HubCommon> and HubCommon is not added to the KnownTypAttribute