I am consuming data from a Kafka topic using the following consumer setup:
val consumer = {
val properties = new Properties()
properties.put(ConsumerConfig.GROUP_ID_CONFIG, "some_consumer_group")
properties.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, streamingConfig.getString("kafka.brokers"))
properties.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, classOf[StringDeserializer])
properties.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, classOf[KafkaJsonDeserializer[Sample]])
properties.put("json.value.type",classOf[Sample])
properties.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, streamingConfig.getInt("kafka.maxPollRecords"): java.lang.Integer)
properties.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, streamingConfig.getInt("kafka.sessionTimeoutMs"): java.lang.Integer)
properties.put(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, streamingConfig.getInt("kafka.heartbeatIntervalMs"): java.lang.Integer)
val consumer = new KafkaConsumer[String, Sample](properties)
consumer.subscribe(util.Arrays.asList(topic))
consumer
}
while the payload is represented by the following case class
case class Sample(properties: Map[String, String],eventType: String)
The problem is that during the deserialization the following errors are produced
Caused by: org.apache.kafka.common.errors.SerializationException: com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `org.service.Sample` (no Creators, like default construct, exist): cannot deserialize from Object value (no delegate- or property-based Creator)
at [Source: (byte[])"{}"; line: 1, column: 2]
Caused by: com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `org.service.Sample` (no Creators, like default construct, exist): cannot deserialize from Object value (no delegate- or property-based Creator)
at [Source: (byte[])"{}"; line: 1, column: 2]
If I was using Java the solution for this is straighforward: Just add a default constructor. What happens in the case of Scala case classes though? I know that the Jackson object mappers can be configured with a jackson-module-scala which takes care of case classes but due to the way the KafkaJsonSerializer is configured I cannot mess with its object mappers - this is its configure method:
protected void configure(KafkaJsonSerializerConfig config) {
boolean prettyPrint = config.getBoolean("json.indent.output");
this.objectMapper = new ObjectMapper();
this.objectMapper.configure(SerializationFeature.INDENT_OUTPUT, prettyPrint);
}
So how can I go about this error here?
Related
Intro
I'm sending JSON messages between two backend servers that use different languages. The producing
server creates a variety of JSON messages, wrapped inside a message with metadata.
The wrapping class is Message, The consuming server has to determine which type of message its
receiving based solely on the message contents.
When I try to use a star-projection to
deserialize the message, I get an error.
Example
import kotlinx.serialization.json.Json
#Language("JSON")
val carJson = """
{
"message_type": "some message",
"data": {
"info_type": "Car",
"name": "Toyota"
}
}
""".trimIndent()
// normally I wouldn't know what the Json message would be - so the type is Message<*>
val actualCarMessage = Json.decodeFromString<Message<*>>(carJson)
Error message
Exception in thread "main" java.lang.IllegalArgumentException: Star projections in type arguments are not allowed, but Message<*>
at kotlinx.serialization.SerializersKt__SerializersKt.serializerByKTypeImpl$SerializersKt__SerializersKt(Serializers.kt:81)
at kotlinx.serialization.SerializersKt__SerializersKt.serializer(Serializers.kt:59)
at kotlinx.serialization.SerializersKt.serializer(Unknown Source)
at ExampleKt.main(example.kt:96)
at ExampleKt.main(example.kt)
Class structure
I want to deserialize JSON into a data class, Message, that has a field with a generic type.
import kotlinx.serialization.SerialName
import kotlinx.serialization.Serializable
#Serializable
data class Message<out DataType : SpecificInformation>(
#SerialName("message_type")
val type: String,
#SerialName("data")
val data: DataType,
)
The field is constrained by a sealed interface, SpecificInformation, with some implementations.
import kotlinx.serialization.SerialName
import kotlinx.serialization.Serializable
import kotlinx.serialization.json.JsonClassDiscriminator
#JsonClassDiscriminator("info_type")
sealed interface SpecificInformation {
#SerialName("info_type")
val infoType: String
}
#Serializable
#SerialName("User")
data class UserInformation(
#SerialName("info_type")
override val infoType: String,
val name: String,
) : SpecificInformation
// there are more implementations...
Workaround?
This is a known
issue (kotlinx.serialization/issues/944)
,
so I'm looking for workarounds.
I have control over the JSON structure and libraries - though I have a preference for
kotlinx.serialization.
I can't change that there are two JSON objects, one is inside the other, and the discriminator is
inside the inner-class.
A custom serializer would be great. But I'd prefer to have this configured on the class or file
(with #Serializable(with = ...) or #file:UseSerializers(...)) as using a
custom SerializersModule is not as seamless.
Attempt: JsonContentPolymorphicSerializer
I've written a custom serializer, which only if it's used specifically (which is something I'd like
to avoid). It's also quite clunky, breaks if the data classes change or a new one is added, and
doesn't benefit from the sealed interface.
Can this be improved so that
It can be used generically? Json.decodeFromString<Message<*>>(carJson)
It doesn't have any hard-coded strings?
class MessageCustomSerializer : JsonContentPolymorphicSerializer<Message<*>>(Message::class) {
override fun selectDeserializer(element: JsonElement): DeserializationStrategy<out Message<*>> {
val discriminator = element
.jsonObject["data"]
?.jsonObject?.get("info_type")
?.jsonPrimitive?.contentOrNull
println("found discriminator $discriminator")
val subclassSerializer = when (discriminator?.lowercase()) {
"user" -> UserInformation.serializer()
"car" -> CarInformation.serializer()
else -> throw IllegalStateException("could not find serializer for $discriminator")
}
println("found subclassSerializer $subclassSerializer")
return Message.serializer(subclassSerializer)
}
}
fun main() {
#Language("JSON")
val carJson = """
{
"message_type": "another message",
"data": {
"info_type": "Car",
"brand": "Toyota"
}
}
""".trimIndent()
val actualCarMessage =
Json.decodeFromString(MessageCustomSerializer(), carJson)
val expectedCarMessage = Message("another message", CarInformation("Car", "Toyota"))
require(actualCarMessage == expectedCarMessage) {
println("car json parsing ❌")
}
println("car json parsing ✅")
}
#Serializable(with = ... - infinite loop
I tried applying MessageCustomSerializer directly to Message...
#Serializable(with = MessageCustomSerializer::class)
data class Message<out T : SpecificInformation>(
//...
But then I couldn't access the plugin-generated serializer, and this causes an infinite loop.
return Message.serializer(subclassSerializer) // calls 'MessageCustomSerializer', causes infinite loop
#Serializer(forClass = ...) - not generic
In addition to annotating Message with #Serializable(with = MessageCustomSerializer::class), I
tried
deriving a plugin-generated serializer:
#Serializer(forClass = Message::class)
object MessagePluginGeneratedSerializer : KSerializer<Message<*>>
But this serializer is not generic, and causes an error
java.lang.AssertionError: No such value argument slot in IrConstructorCallImpl: 0 (total=0).
Symbol: MessageCustomSerializer.<init>|-5645683436151566731[0]
at org.jetbrains.kotlin.ir.expressions.IrMemberAccessExpressionKt.throwNoSuchArgumentSlotException(IrMemberAccessExpression.kt:66)
at org.jetbrains.kotlin.ir.expressions.IrFunctionAccessExpression.putValueArgument(IrFunctionAccessExpression.kt:31)
at org.jetbrains.kotlinx.serialization.compiler.backend.ir.IrBuilderExtension$DefaultImpls.irInvoke(GeneratorHelpers.kt:210)
at org.jetbrains.kotlinx.serialization.compiler.backend.ir.SerializableCompanionIrGenerator.irInvoke(SerializableCompanionIrGenerator.kt:35)
You are asking many things here, so I will simply try to give some pointers in regards to the errors you are making which you seem to be stuck on. With those in mind, and reading the documentation I link to, I believe you should be able to resolve the rest yourself.
Polymorphic serialization
Acquaint yourself with kotlinx.serialization polymorphic serialization. When you are trying to serialize Message<*> and DataType you are trying to use polymorphic serialization.
In case you are serializing Message<*> as the root object, specifying PolymorphicSerializer explicitly (as I also posted in the bug report you link to) should work. E.g., Json.decodeFromString( PolymorphicSerializer( Message::class ), carJson ).
P.s. I'm not 100% certain what you are trying to do here is the same as in the bug report. Either way, specifying the serializer explicitely should work, whether or not it is a bug that you shouldn't be required to do so.
The message_type and info_type fields you have in Message and DataType respectively are class discriminators. You need to configure this in your Json settings, and set the correct SerialName on your concrete classes for them to work. Using a different class discriminator per hierarchy is only possible starting from kotlinx.serialization 1.3.0 using #JsonClassDiscriminator.
Overriding plugin-generated serializer
But then I couldn't access the plugin-generated serializer, and this causes an infinite loop.
#Serializable(with = ...) overrides the plugin-generated serializer. If you want to retain the plugin-generated serializer, do not apply with.
When you are serializing the object directly (as the root object), you can still pass a different serializer to use as the first parameter to encode/decode. When you want to override the serializer to use for a specific property nested somewhere in the root object, use #Serializable on the property.
Polymorphism and generic classes
The "No such value argument slot in IrConstructorCallImpl: 0" error is to be expected.
You need to do more work in case you want to specify a serializer for polymorphic generic classes.
With default data types like String, Long, BigDecimal, and etc. Works fine, But with custom data types errors occures.
fun main() {
class Quantity(doubleValue: Double) : BigDecimal(doubleValue) {
override fun toByte(): Byte = super.toInt().toByte()
override fun toChar(): Char = super.toInt().toChar()
override fun toShort(): Short = super.toInt().toShort()
}
data class CustomDataType(
val customDataType: Quantity
)
val testJson = jacksonObjectMapper()
.valueToTree<JsonNode>(
CustomDataType(
Quantity(200.toDouble())
)
).toString()
jacksonObjectMapper().readValue(testJson, CustomDataType::class.java)
}
Output ->
Exception in thread "main" com.fasterxml.jackson.databind.JsonMappingException: Cannot deserialize Class com.test.project.server.utils.ApiJsonTypesKt$main$CustomDataType (of type local/anonymous) as a Bean
at [Source: (String)"{"customDataType":0.00200000000000000004163336342344337026588618755340576171875}"; line: 1, column: 1]
at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:309)
at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:268)
at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
at com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:491)
at com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:4669)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4478)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3434)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3402)
at com.test.project.server.utils.ApiJsonTypesKt.main(ApiJsonTypes.kt:37)
at com.test.project.server.utils.ApiJsonTypesKt.main(ApiJsonTypes.kt)
Caused by: java.lang.IllegalArgumentException: Cannot deserialize Class com.test.project.server.utils.ApiJsonTypesKt$main$CustomDataType (of type local/anonymous) as a Bean
at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.isPotentialBeanType(BeanDeserializerFactory.java:883)
at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:137)
at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:414)
at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:349)
at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:264)
... 9 more
I'm developing cordapp using the example-cordapp project as a reference. I've been able to commit a transaction to the ledger and even run querias on the node to see if it's really there. However, when I try to run query from my Spring Boot application, I get this error.
Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request
processing failed; nested exception is
org.springframework.http.converter.HttpMessageConversionException: JSON mapping problem:
java.util.Collections$UnmodifiableRandomAccessList[0]->net.corda.core.contracts.StateAndRef["state"]-
>net.corda.core.contracts.TransactionState["data"]-
>com.mypackage.states.MyState["party"]; nested exception is
com.fasterxml.jackson.databind.JsonMappingException: object is not an instance of declaring class
(through reference chain: java.util.Collections$UnmodifiableRandomAccessList[0]-
>net.corda.core.contracts.StateAndRef["state"]->net.corda.core.contracts.TransactionState["data"]-
>com.mypackage.states.MyState["party"])] with root cause
java.lang.IllegalArgumentException: object is not an instance of declaring class
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_251]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_251]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~
[na:1.8.0_251]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_251]
Here's the request code
#GetMapping(value = [ "/api/v1/states" ], produces = [MediaType.APPLICATION_JSON_VALUE])
fun getMyIOUs(): ResponseEntity<List<StateAndRef<MyState>>> {
val myStates = proxy.vaultQueryBy<MyState>().states
return ResponseEntity.ok(myStates)
}
And here's the state code
#BelongsToContract(com.sentinel.contract.SharingInformationContract::class)
class SharingInformationState(
val party: Party,
val dataOwnerId: Long,
val dataBuyerId: Long,
override val linearId: UniqueIdentifier = UniqueIdentifier()) : LinearState, QueryableState {
override val participants: List<AbstractParty> = listOf(party)
override fun generateMappedObject(schema: MappedSchema): PersistentState {
return when (schema) {
SharingInformationSchemaV1 -> SharingInformationSchemaV1.PersistentSharingInformation(
party,
dataOwnerId,
dataBuyerId,
linearId.id
)
else -> throw IllegalArgumentException("Unrecognised schema $schema")
}
}
override fun supportedSchemas(): Iterable<MappedSchema> = listOf(SharingInformationSchemaV1)
}
There's little information about this issue on the internet. Some suggest it is connected to the classpath, that something is duplicated there, but I don't know how to check. Also, this error isn't connected to the Party type. I've tried to add #JsonIgnore on a party, but then it throws on the other field. Persistence of this field in mapping schema also doesn't matter. I've tried persisting and not persisting, it changes nothing. Thanks in advance!
I believe this is because you are missing Corda Jackson support library which is required to convert Corda objects to json.
Add this to your dependencies in the build.gradle
compile "net.corda:corda-jackson:$corda_release_version"
https://github.com/corda/samples-java/blob/master/Advanced/auction-cordapp/client/build.gradle#L19
Also, make sure you have a MappingJackson2HttpMessageConverter bean configured.
#Bean
public MappingJackson2HttpMessageConverter mappingJackson2HttpMessageConverter(){
ObjectMapper mapper = JacksonSupport.createDefaultMapper(partyAProxy());
MappingJackson2HttpMessageConverter converter = new MappingJackson2HttpMessageConverter();
converter.setObjectMapper(mapper);
return converter;
}
https://github.com/corda/samples-java/blob/master/Advanced/auction-cordapp/client/src/main/java/net/corda/samples/client/AppConfig.java#L48
The Exception java.lang.IllegalArgumentException: object is not an instance of declaring class is something that happens if a method is called by reflection on an object which is of the wrong type.
In conjunction with jackson that may happen because a generic is lying to you. Here is an example:
class A (val x: String)
class B (val y: String)
class C (val z: List<A>)
ObjectMapper().writeValueAsString(C(listOf(B("x")) as List<A>))
This causes a compile warning, but it compiles and initially runs because of type erasure. However we forcefully injected a List<B> in a place where actually a List<A> is expected. While type erasure does remove quite a bit of information, it does not do so completely. Reflection can still be used to determine that C.z is actually of type List<A>. Jackson uses this information and tries to serialize an object of type A but instead finds an object of type B in the list and fails with the given message.
Check that your data structure actually contains the types that you expect!
I am attempting to parse a json object that contains a list. I am able to parse the list if the field is backed by a Java List, but it fails if the field is backed by a Scala list. What is the difference between parsing into a Scala List vs a Java List, and what do I have to change to be able to parse this into a Scala List?
object JsonParsingExample extends App {
val objectMapper = new ObjectMapper()
// This line succeeds.
objectMapper.readValue("""{"list": ["a","b"]}""", classOf[JavaList])
// This line fails.
objectMapper.readValue("""{"list": ["a","b"]}""", classOf[ScalaList])
}
case class JavaList() {
#JsonProperty(value = "list")
var myList: java.util.ArrayList[String] = null
}
case class ScalaList() {
#JsonProperty(value = "list")
var myList: List[String] = null
}
The error message I receive is:
com.fasterxml.jackson.databind.JsonMappingException: Can not construct instance of scala.collection.immutable.List, problem: abstract types either need to be mapped to concrete types, have custom deserializer, or be instantiated with additional type information
Jackson doesn't know anything about Scala types by default (otherwise it would have to depend on scala-library). To teach it, use jackson-module-scala.
Because the scala.collection.immutable.List is actually an abstract class. Generally when you use List("a", "b", "c") is the object List.apply() which is coming from this line: https://github.com/scala/scala/blob/2.12.x/src/library/scala/collection/immutable/List.scala#L452 and that's actually an inner class (something called scala.collection.immutable.$colon$colon).
Basically I want null arrays to be deserialized as empty collection by Jackson with scala module. As I understand this is not implemented right now so I'm looking for a workaround. I think it could work if I have auxiliary constructor with Option[Seq[Any]] parameter. Like this:
case class Test (id: Int, seq: Seq[Any]){
#JsonCreator
def this(id: Int, seq: Option[Seq[Any]]) = this(id, seq.getOrElse(Seq()))
}
But it doesn't work since Jackson gives me this exception:
Caused by: java.lang.IllegalArgumentException: Conflicting
property-based creators: already had explicitly marked [constructor
for com.test.Main$Test, annotations: [null]], encountered [constructor
for com.test.Main$Test, annotations: {interface
com.fasterxml.jackson.annotation.JsonCreator=#com.fasterxml.jackson.annotation.JsonCreator(mode=DEFAULT)}]
It's interesting that if I remove id field it starts working:
case class Test (seq: Seq[Any]){
#JsonCreator
def this(seq: Option[Seq[Any]]) = this(seq.getOrElse(Seq()))
}
Could somebody explain me what is going on here?
Can I have overloaded constructor as json creator with case classes?
Maybe you see another workaround to my initial problem(deserializing null as empty collection)?
p.s. about my initial problem. I don't want to have Option[Seq[Any]] and I don't want to have getter because my constructor param will have ugly name like _seq and in scala parameter names is also a public API.