I try to use Kafka Stream to convert a topic with String/JSON messages to another topic as Avro messages.
Stream main method:
streamsConfiguration.put(StreamsConfig.KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());
streamsConfiguration.put(StreamsConfig.VALUE_SERDE_CLASS_CONFIG, GenericAvroSerde.class);
final KStreamBuilder builder = new KStreamBuilder();
final Serde<String> stringSerde = Serdes.String();
builder.stream(stringSerde, stringSerde, "testin")
.mapValues(value -> AvroUtil.transform(value))
.to("testout");
final KafkaStreams streams = new KafkaStreams(builder, streamsConfiguration);
streams.start();
Transformation:
public static GenericRecord transform(Object value) {
// ... parse string/json and generate Avro object
String userSchema = "{\"type\":\"record\"," +
"\"name\":\"myrecord\"," +
"\"fields\":[{\"name\":\"f1\",\"type\":\"string\"}]}";
Schema.Parser parser = new Schema.Parser();
Schema schema = parser.parse(userSchema);
GenericRecord avroRecord = new GenericData.Record(schema);
avroRecord.put("f1", "value1");
return avroRecord;
}
And get an exception like this:
Exception in thread "StreamThread-1" java.lang.NoSuchMethodError: com.fasterxml.jackson.annotation.JsonProperty.access()Lcom/fasterxml/jackson/annotation/JsonProperty$Access;
at com.fasterxml.jackson.databind.introspect.JacksonAnnotationIntrospector.findPropertyAccess(JacksonAnnotationIntrospector.java:229)
at com.fasterxml.jackson.databind.introspect.POJOPropertyBuilder$9.withMember(POJOPropertyBuilder.java:545)
at com.fasterxml.jackson.databind.introspect.POJOPropertyBuilder$9.withMember(POJOPropertyBuilder.java:542)
at com.fasterxml.jackson.databind.introspect.POJOPropertyBuilder.fromMemberAnnotationsExcept(POJOPropertyBuilder.java:996)
at com.fasterxml.jackson.databind.introspect.POJOPropertyBuilder.findAccess(POJOPropertyBuilder.java:542)
at com.fasterxml.jackson.databind.introspect.POJOPropertyBuilder.removeNonVisible(POJOPropertyBuilder.java:623)
at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector._removeUnwantedAccessor(POJOPropertiesCollector.java:697)
at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.collectAll(POJOPropertiesCollector.java:298)
at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.getJsonValueMethod(POJOPropertiesCollector.java:169)
at com.fasterxml.jackson.databind.introspect.BasicBeanDescription.findJsonValueMethod(BasicBeanDescription.java:222)
at com.fasterxml.jackson.databind.ser.BasicSerializerFactory.findSerializerByAnnotations(BasicSerializerFactory.java:355)
at com.fasterxml.jackson.databind.ser.BeanSerializerFactory._createSerializer2(BeanSerializerFactory.java:210)
at com.fasterxml.jackson.databind.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:153)
at com.fasterxml.jackson.databind.SerializerProvider._createUntypedSerializer(SerializerProvider.java:1203)
at com.fasterxml.jackson.databind.SerializerProvider._createAndCacheUntypedSerializer(SerializerProvider.java:1157)
at com.fasterxml.jackson.databind.SerializerProvider.findValueSerializer(SerializerProvider.java:481)
at com.fasterxml.jackson.databind.SerializerProvider.findTypedValueSerializer(SerializerProvider.java:679)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:107)
at com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3559)
at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:2927)
at io.confluent.kafka.schemaregistry.client.rest.entities.requests.RegisterSchemaRequest.toJson(RegisterSchemaRequest.java:76)
at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:232)
at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:224)
at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:219)
at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.registerAndGetId(CachedSchemaRegistryClient.java:58)
at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.register(CachedSchemaRegistryClient.java:90)
at io.confluent.kafka.serializers.AbstractKafkaAvroSerializer.serializeImpl(AbstractKafkaAvroSerializer.java:72)
at io.confluent.kafka.serializers.KafkaAvroSerializer.serialize(KafkaAvroSerializer.java:54)
Is this the right approach? I'm new to Kafka Streams and Avro
Just the jackson dependencies were missing:
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.7.8</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>2.7.8</version>
</dependency>
Now it works
Related
I have created my patch endpoint (Json path specified in RFC 6902).
At UI generated by springfox my endpoint is shown, but the model example (only patch) did not show.
To use Json patch in my Spring-boot 2 project I have used that dependency on pom.xml.
<dependencies>
<dependency>
<groupId>com.github.java-json-tools</groupId>
<artifactId>json-patch</artifactId>
<version>1.12</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.11.1</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>io.springfox</groupId>
<artifactId>springfox-swagger2</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>io.springfox</groupId>
<artifactId>springfox-swagger-ui</artifactId>
<scope>compile</scope>
</dependency>
</dependencies>
At my endpoint, my code is:
#RestController
#RequestMapping(value = "/operation", produces = "application/json")
public class IntentController {
#RequestMapping(value = "/{id}",
method = RequestMethod.PATCH,
consumes = "application/json-patch+json")
public void updateValue(#PathVariable Long id, #RequestBody JsonPatch patch){ {
// ... do magic
}
#RequestMapping(value = "/{id}",
method = RequestMethod.GET)
public MyDto getValue(#PathVariable Long id){ {
MyDto dto = service.findById(id);
return dto;
}
#RequestMapping(method = RequestMethod.POST)
public void updateValue(#RequestBody MyDto dto){ {
service.insert(dto);
}
}
My GET and POST endpoints are generated fine with their example models in UI.
Only PATCH doesn't work fine... their example model didn't generate.
The problem lies with JsonPatch object, this object does not have any getter method, so Springfox library could not generate the model for request.
One possible solution may be like , you create a custom MyJsonPatch POJO with getter and setter and create a JsonPatch with the data of MyJsonPatch.
I can't found a solution to my problem, so I decided to use #ApiParam from Swagger to describe that this field is an RFC 6902 implementation.
I'm getting txt file from resource folder(Spring).
and created the File
File file = new File(classLoader.getResource("files/example.txt").getFile());
And I want to convert this file to JsonObject file.
read the content of file using inputstream, then convert the stream to string.. use google gson json library to convert string to json:
http://www.java67.com/2016/10/3-ways-to-convert-string-to-json-object-in-java.html
You could also use Jackson for this. Jackson is one of the most complete JSON libraries around.
If you are using Maven, just include these dependencies:
<!-- https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-core -->
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>2.8.8</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-databind -->
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.8.8</version>
</dependency>
Then you can create an ObjectMapper instance with which you can create a JsonNode (similar to JsonObject) in this way:
ObjectMapper mapper = new ObjectMapper();
JsonNode jsonNode = mapper.readTree(in); // create a tree structure from the JSON
You can do whatever you want with this JsonNode:
jsonNode.fields().forEachRemaining(entry -> {
if(entry.getKey().endsWith(".ID")) {
entry.setValue(new TextNode(UUID.randomUUID().toString()));
}
});
I am learning spring 4.2.4 by writing some webapp code.The idea is to return json file by controller.
I have already posted my questions before couple of days ago and still no i could get the right suggestions for my case. I am trying all the suggestions given by stackoverfolow none of the suggestions could work for me. Here is my controller:
.....
#RequestMapping(value="/getmessages",method=RequestMethod.GET, produces="application/json")
#ResponseBody
public Map<String,Object> getMessage(Principal prinicipal){
List<Message>message=null;
if(prinicipal==null){
message=new ArrayList<Message>();
}
else{
String username=prinicipal.getName();
message=usersService.getMessage(username);
}
Map<String,Object> data= new HashMap<String,Object>();
data.put("message", message);
data.put("number", message.size());
System.out.println("message has to be her\n"+message);
System.out.println("Number message has to be her is..."+message.size());
return data;
}
the message content which is to be retrieved from mysql is propely displayed in console.
The problem is conversion to JSON and return the result. I have been trying by change the jackson 1.9.x jar to jackson-fasterxml-2.x and it does not work. All other possible configuration of servlete also does not work for me.
When I add jackson-fasterxml-databind ....it displays file download dialogue box for filename"getmessages". to download and save...
I am very grateful for your help.
Finally I have solved this problem as follow:
I changed my return type from MAP to String and I converted my MAP to String in side my CONTROLLER as:
Converting my Map to String:
ObjectMapper mapper = new ObjectMapper();
String jsonFromMap = mapper.writeValueAsString(data);
I have also changed my dependencies as:
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>2.2.3</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.2.3</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>2.2.3</version>
Inside #RequestMapping change
produces=application/json
to
produces=text/html
I think the problem is with browser unable to understand application/json and returning Map............
Any how I did what I want to do even-though I do not know how I did it!
I'm a newbie in the theme. How can I return JSON data from my controller, using something like that (using ResponseBody)
#RequestMapping(value = "/ajaxtest", method = RequestMethod.GET)
#ResponseBody
public Set<String> ajaxTest() {
Set<String> records = new HashSet<String>();
records.add("Record #1");
records.add("Record #2");
return records;
}
I tried ro use Jackson, but have http 406 error.
What correct version of Jackson should I use with Spring version 4.0.3 and what is the algoritm of using?
UPD
Ajax call
<button id="btn">Click!</button>
<script>
$("#btn").click(
function sendAjax() {
$.ajax({
url: "/ajaxtest",
dataType: "json",
success: function(data) {
alert(data);
},
error:function() {
alert("error");
}
});
})
</script>
For Converting to json request you must include following 3 jar to your project build path.
Jackson jar is use for convrting HTTP request to jason format.
Also mention headear=content-type=application/json
These are the jar files
jackson-mapper-asl.jar
jackson-core-asl.jar
jackson-jaxrs.jar
If you are using maven, you can include the following dependency in your pom.xml
Jackson Mapper Version 1 which is good enough to convert your object to JSON object:
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
<version>1.9.13</version>
</dependency>
Or you can also go for latest Jackson version,
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.6.0</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>2.6.0</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>2.6.0</version>
</dependency>
Thanks in advance. I am using version 1.9.2
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-core-asl</artifactId>
<version>1.9.2</version>
</dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
<version>1.9.2</version>
</dependency>
org.codehaus.jackson.map.ObjectMapper jsonMapper = new org.codehaus.jackson.map.
jsonMapper.configure(DeserializationConfig.Feature.FAIL_ON_UNKNOWN_PROPERTIES , false);
JsonNode listNode = jsonMapper.readTree(json);
Iterator<JsonNode> it = listNode.getElements();
while (it.hasNext()) {
JsonNode moNode = it.next();
String className = moNode.get("objectClass").asText();
String moNodeText = moNode.toString();
Class<?> moClass = Class.forName(className);
Object ob = jsonMapper.readValue(moNodeText, moClass);
MyObject mo = (MyObject) ob;
moSet.add(mo);
}
But it keep failing pointing to additional fields in json.
(was java.lang.UnsupportedOperationException) (through reference chain: com.xxx.MyObject["additional_field_Json"])
I tried parser but got same error. any help?
thanks
Gopi
2 >>>> nodeText :{"objectclass":"com.xxx.xxx.MetaInfo$Flow",............,"fieldonlyexistinjson":["VALUE1","VALUE2","VALUE3"],"subFlows":[]}
2014-04-23 17:39:48,549 - ERROR main com.xxx.xxx.TestMetaDataCache.testImportMetadataFromJson (TestMetaDataCache.java:86) exception
org.codehaus.jackson.map.JsonMappingException: (was java.lang.UnsupportedOperationException) (through reference chain: com.xxx.xxx.Flow["objectTypes"])
at org.codehaus.jackson.map.JsonMappingException.wrapWithPath(JsonMappingException.java:218)
at org.codehaus.jackson.map.JsonMappingException.wrapWithPath(JsonMappingException.java:183)
at org.codehaus.jackson.map.deser.BeanDeserializer.wrapAndThrow(BeanDeserializer.java:1462)
at org.codehaus.jackson.map.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:699)
at org.codehaus.jackson.map.deser.BeanDeserializer.deserialize(BeanDeserializer.java:580)
at org.codehaus.jackson.map.ObjectMapper._readValue(ObjectMapper.java:2695)
at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1279)
at org.codehaus.jackson.JsonParser.readValueAs(JsonParser.java:1336)
at com.xxx.xxx.TestMetaDataCache.testImportMetadataFromJson(TestMetaDataCache.java:80)
No, that is not the same exception: "UnsupportedOperationException" is not what you get for unrecognized property. Can you include full stack trace for exception?
My guess is that you have an immutable List or Map somewhere, and deserializer is trying to add entries to it.