I had the bright idea to implement a generic object and attempt to serialize / deserialize it, and received this error:
Serializer for class 'DetailsRequest' is not found.
Mark the class as #Serializable or provide the serializer explicitly.
I thought the #Serializer annotation would have achieved this... ???
The implementation of the data class, and its custom serializer (which is probably overkill), are as follows:
#Serializable(with=DetailsRequestSerializer::class)
data class DetailsRequest<out T>(val details: T)
object DetailsRequestSerializer : KSerializer<DetailsRequest<*>> {
override val descriptor: SerialDescriptor = buildClassSerialDescriptor(
"DetailsRequest") {
when (this::class) {
String::class -> element<String>("details")
Long::class -> element<Long>("details")
Int::class -> element<Int>("details")
Double::class -> element<Double>("details")
Float::class -> element<Float>("details")
else -> element<String>("details")
}
}
override fun serialize(
encoder: Encoder,
value: DetailsRequest<*>
) {
value.details.let {
encoder.encodeStructure(descriptor) {
when (value::class) {
String::class -> encodeStringElement(descriptor, 0, value.details as String)
Long::class -> encodeLongElement(descriptor, 0, value.details as Long)
Int::class -> encodeIntElement(descriptor, 0, value.details as Int)
Double::class -> encodeDoubleElement(descriptor, 0, value.details as Double)
Float::class -> encodeFloatElement(descriptor, 0, value.details as Float)
else -> encodeStringElement(descriptor, 0, value.details as String)
}
}
}
}
override fun deserialize(decoder: Decoder): DetailsRequest<*> {
return when (this::class) {
String::class -> DetailsRequest(decoder.decodeString())
Long::class -> DetailsRequest(decoder.decodeLong())
Int::class -> DetailsRequest(decoder.decodeInt())
Double::class -> DetailsRequest(decoder.decodeDouble())
Float::class -> DetailsRequest(decoder.decodeFloat())
else -> DetailsRequest(decoder.decodeString())
}
}
}
I have written this unit test:
class PlaygroundTest {
private val json = Json {
encodeDefaults = true
isLenient = true
allowSpecialFloatingPointValues = true
allowStructuredMapKeys = true
prettyPrint = true
useArrayPolymorphism = false
ignoreUnknownKeys = true
}
#Test
fun `Details Integration Tests`() {
val input = DetailsRequest(details = "New Record")
val value = json.encodeToString(input)
println("value=[$value]")
val output = value.convertToDataClass<DetailsRequest<String>>()
println("output=[$output]")
}
#OptIn(InternalSerializationApi::class)
internal inline fun <reified R : Any> String.convertToDataClass() =
json.decodeFromString(R::class.serializer(), this)
}
Bonus points: This is a Kotlin Multiplatform project, which I do not believe will impact this situation.
Is this type of situation even possible?
The problem you asked about
Good news: this situation is totally possible, and your code is very nearly correct. Kotlin's builtin serialization is a bit wonky about getting serializers, though, so it is a bit surprising at first that your code snippet doesn't work.
The answer lies in a bit of almost fine print in the documentation[1]:
Constraints
This paragraph explains known (but not all!) constraints of the serializer() implementation. Please note that they are not bugs, but implementation restrictions that we cannot workaround.
[...]
Serializers for classes with generic parameters are ignored by this method
Since DetailsRequest<out T> has that generic out T, this means that accessing its serializer through DetailsRequest::class.serializer() won't work.
Fortunately, kotlinx.serialization provides us a different method[2] to get a serializer for an arbitrary type:
inline fun <T> serializer(): KSerializer<T>
In your case, you could use it like this:
// OptIn no longer needed
internal inline fun <reified R : Any> String.convertToDataClass() =
json.decodeFromString(serializer<R>(), this)
Note that there's an overload of json.decodeFromString that uses serializer() automatically: all you really need is
internal inline fun <reified R : Any> String.convertToDataClass() =
json.decodeFromString<R>(this)
(In fact, at this point, you may even consider removing the convertToDataClass method entirely.)
The problem you didn't ask about
If you run the code with that change, you'll get a different error:
Exception in thread "main" kotlinx.serialization.json.internal.JsonDecodingException: Unexpected JSON token at offset 0: Expected beginning of the string, but got {
JSON input: {
"details": "New Record"
}
This is simply because your implementation of the deserializer is a bit off—it's trying to deserialize DetailsRequest as a single string (such as "New Record") rather than a full JSON object the way you serialized it (such as { "details": "New Record" }).
A full explanation of the issue here would take a bit more space than is reasonable for this answer, but I refer you to the very complete—if a bit long—serialization guide here. You may need to scroll up and read the rest of the Custom Serializers section to ensure you understand it all as well.
A quicker solution
If you mark DetailsRequest with simply #Serializable (without the with argument) and remove the entire DetailsRequestSerializer object, the code will run fine and serialize and deserialize as one would expect. The #Serializable annotation (without a with argument) tells the compiler to autogenerate a serializer, which it does successfully and reasonably in this case.
Unless you have a specific serialization need that this default serializer does not fulfill, I would strongly recommend simply using the default one. It's much cleaner and a lot less work. :-)
[1] You'll have to scroll down a bit; Dokka doesn't allow me to link to a specific declaration of a function. The relevant bit is all the way at the bottom.
[2] Yes, it's on the same page in the documentation. Just don't scroll down this time. :-)
Related
I'm accessing a JSON API which has 2 kinds of endpoints:
the first kind returns a list of objects of the same type (Symptom, ChronicDisease...)
the second kind (a search function) returns a mixed list of objects of different types (those types are the same than can be returned by the first kind of API)
In the second case, each item of the list has a type field telling which is the type of the object. This field doesn't exist in the first case.
I would like to use the default deserializer for the first kind of API and a custom deserializer for the second kind of API. Is it possible?
If I only use the default deserializer, API calls of the first kind will work but I'm unable to perform a search.
If I enable the following deserializer, the search will work but the deserializer is also used when using the first kind of API and it fails because the type field is missing.
Custom deserializer I'd like to use:
class SearchableItemDeserializer : JsonDeserializer<SearchableItem>() {
override fun deserialize(p: JsonParser, ctxt: DeserializationContext): SearchableItem {
val root : JsonNode = p.readValueAsTree()
val type : String = root.get("type").asText()
when(type){
"symptom" -> {
return ObjectMapper().readValue(root.asText(), Symptom::class.java)
}
"symptom_group" -> {
return ObjectMapper().readValue(root.asText(), SymptomGroup::class.java)
}
"diagnosis" -> {
return ObjectMapper().readValue(root.asText(), Diagnose::class.java)
}
"chronic_disease" -> {
return ObjectMapper().readValue(root.asText(), ChronicDisease::class.java)
}
}
throw Exception("Unable to deserialize type $type")
}
}
Interface common to Symptom, SymptomGroup, Diagnose and ChronicDisease:
#JsonDeserialize(using = SearchableItemDeserializer::class)
interface SearchableItem
It's possible. You can extent Converter.Factory to create you custom converter. Probably most dumb and direct way would be to add check for specific retrofit annotation inside "requestBodyConverter" or "responseBodyConverter" methods.
Something like:
class CustomConverter : Converter.Factory() {
override fun responseBodyConverter(type: Type,
annotations: Array<Annotation>,
retrofit: Retrofit): Converter<ResponseBody, *>? {
return responseConverter(*annotations)
.responseBodyConverter(type, annotations, retrofit)
}
private fun responseConverter(vararg methodAnnotations: Annotation): Converter.Factory {
return when {
endpoint1(*methodAnnotations) -> converter1
endpoint2(*methodAnnotations) -> converter2
else -> defaultConverter
}
}
override fun requestBodyConverter(type: Type,
parameterAnnotations: Array<Annotation>,
methodAnnotations: Array<Annotation>,
retrofit: Retrofit): Converter<*, RequestBody>? {
//same approach here
}
fun endpoint1(vararg annotations: Annotation): Boolean {
//condition check here
}
fun endpoint2(vararg annotations: Annotation): Boolean {
//and here (if needed)
}
Just add your endpoints 1/2 implementation (probably just compare #Get() contents with certain pattern or something like that) and repeat same instruction for requestConverter.
When ready, just add it to retrofit:
return Retrofit.Builder()
.baseUrl(url)
.client(client)
.addConverterFactory(CustomConverter())
.build()
In Kotlin, is it possible to declare a generic function type as the return type of a function?
What I want to achieve would look like this in Java:
interface Factory {
static Factory INSTANCE = new FactoryImpl();
<T> T create(String name, Class<T> type);
}
class PrefixedFactory implements Factory {
private final String prefix;
PrefixedFactory(String prefix) {
this.prefix = prefix;
}
#Override
public <T> T create(String name, Class<T> type) {
return Factory.INSTANCE.create(prefix + name, type);
}
}
(Note that in the example I access the Factory instance using the static field to avoid passing a generic function as a parameter, which would present its own problems in Kotlin).
I would like convert the prefixer to a kotlin function, but it seems to be impossible to declare a generic function as the return type:
fun prefixer(prefix: String): <T> (String, KClass<T>) -> T { TODO() }
This of course does not compile. It seems to me that this is a limitation compared to Java's functional interfaces. Is there a way to accomplish this, or a workaround?
(Edit) Clarification
I want the actual result function to be generic. If I do
fun <T: Any> prefixer(prefix: String): (String, KClass<T>) -> T { TODO() }
as the current answers suggest; I don't get a generic function, instead I get (String, KClass<Foo>) -> Foo if I call prefixer<Foo>(""). So that function can only be called with Foo, while the factory function prefixer in that case is generic, the result is not. I hope that clears up the misunderstandings.
My use case is in a Gradle plugin, where I wrote a helper method similar to this one that applies some defaults to each task created:
val myPrefix = "..."
val project: Project = <from context>
fun <T: Task> String.task(type: KClass<T>, doConfig: T.() -> Unit) {
project.tasks.create("$prefix$this", type.java, { it.doConfig() })
}
Note that the project comes in as closure. Now I want to reuse that helper in a different plugin, so I would like to create this function using a factory for different project instances.
You're doing it almost correctly. You only need to define the generic part at the prefixer function directly.
fun <T: Any> prefixer(prefix: String): (String, KClass<T>) -> T { TODO() }
Depending on you actual implementation, you could have a look at the reified keyword.
No, it isn't possible (as far as I know). The technical term for such a type is "higher-kinded type" and very few languages support them, on JVM I only know of Scala.
If someone asked me the same question without having an interface like Factory, I'd suggest creating exactly this interface as a workaround.
The following line does compile:
fun <T : Any> prefixer(prefix: String): (String, KClass<T>) -> T = TODO()
First, the generic deceleration should be right after the fun keyword.
Then it has has to be declared as type Any. The default is Any? but KClass only takes Any.
Although I was disappointed to read #Alexey's answer, I found a more streamlined workaround taking advantage of Kotlin's operators. The following makes it look more like a lambda when used:
private class Prefixer(private val: String) {
operator fun <T> invoke(name: String, type: Class<T>): T {
TODO()
}
}
To use it:
val createMy = Prefixer("MyPrefix")
val result = createMy("Configuration", Configuration::class.java)
Feel free to replace with KClass where necessary. I was actually using this for a slightly different purpose.
I have a case class that's a "Bag of Fields" - lots of fields that represent a business entity. For my use case, the distinction between "", null, and the k/v being totally absent from the JSON object is irrelevant to me. In this case class these fields are already Option[String]s, so I'd like all 3 of those cases to collapse into None. I believe the current behavior of the auto-generated Reads is what I want except for the empty string case.
This BOF is likely to change in the future, so "just implement your own Reads" (as suggested here: make play-json read the empty string as None for a type of Option[T]) or something else where I have to re-enumerate all the fields is a non-starter.
I think what I may need is Play's 'Json Transformers'. It appears trivial to write a transformer that removes entries if they have empty values, but what I cant figure out is how to compose it with the auto-generated Reads implementation.
I imagine what I need is some combinator over Reads and transformers, some signature like: (JSON -> JSON, JSON -> T) -> (JSON -> T). Obviously I've found this page: https://www.playframework.com/documentation/2.5.x/ScalaJsonCombinators, but none of the listed combinators does what I want I believe. Could this combinator be easily implemented? I'd be a little out of my type-fu depth but that would be a great solution if I could get some pointers.
Here's what worked for me:
class RemoveEmpty[T] (reader: Reads[T]) extends Reads[T] {
override def reads(json: JsValue): JsResult[T] = json match {
case JsObject(underlying) => {
reader.reads(JsObject(underlying.filterNot{ case (k, v) => jsonValueEmpty(v) } ))
}
case _ => {
JsError("Non-JsObj passed to RemoveEmpty")
}
}
def jsonValueEmpty(v: JsValue) = v match {
case JsNull | JsString("") => true
case _ => false
}
}
Then you can just use it like this:
implicit val myTypeReads = new RemoveEmpty(Json.reads[MyType])
Consider this example using Play's JSON API (play.api.libs.json):
case class FooJson(
// lots of other fields omitted
location: Option[LocationJson]
)
object FooJson {
implicit val writes = Json.writes[FooJson]
}
and
case class LocationJson(latitude: Double, longitude: Double)
object LocationJson {
implicit val writes = Json.writes[LocationJson]
}
If location is None, the resulting JSON won't have location field at all. This is fine and understadable. But if I wanted for some reason (say, to make my API more self-documenting), how can I explicitly output null in JSON?
{
"location": null
}
I also tried defining the field as location: LocationJson and passing option.orNull to it, but it does not work (scala.MatchError: null at play.api.libs.json.OWrites$$anon$2.writes). For non-custom types such as String or Double, this approach would produce null in JSON output.
So, while using Json.writes[FooJson] as shown above (or something equally simple, i.e. not having to write a custom Writes implementation), is there a clean way to include nulls in JSON?
What I'm asking is analogous to JsonInclude.Include.ALWAYS in the Jackson library (also Jackson's default behaviour). Similarly in Gson this would be trivial
(new GsonBuilder().serializeNulls().create()).
Play 2.4.4
Greg Methvin, a Play committer, wrote this answer to me in a related GitHub issue:
The JSON macros only support one way of encoding optional values,
which is to omit None values from the JSON. This is not a bug but
rather a limitation of the implementation. If you want to include
nulls you're unfortunately going to have to implement your own Writes.
I do think we should try to provide more configurability for the
macros though.
In this case, I'll let Play exclude this field when the value is null, even if it slightly sacrifices API consistency and self-documentability. It's still such a minor thing (in this particular API) that it doesn't warrant uglifying the code as much as a custom Writes would take for a case class with a dozen values.
I'm hoping they do make this more configurable in future Play versions.
Hello from the future.
As of Play 2.7, a fairly simple solution was introduced for automated JSON codecs. We can introduce the appropriate implicit value for JsonConfiguration in the scope for the Format/Reads/Writes. The following configuration will write nulls for empty Options instead of omitting the fields entirely.
import play.api.libs.json._
implicit val config = JsonConfiguration(optionHandlers = OptionHandlers.WritesNull)
implicit val residentWrites = Json.writes[Resident]
Reference
Here's a way to do it:
object MyWrites extends DefaultWrites{
override def OptionWrites[T](implicit fmt: Writes[T]): Writes[Option[T]] = new Writes[Option[T]] {
override def writes(o: Option[T]): JsValue = {
o match {
case Some(a) => Json.toJson(a)(fmt)
case None => JsNull
}
}
}
}
This will overwrite the default implementation which will not create an element. I used this in your sample code:
case class FooJson(
// ...
location: Option[LocationJson]
)
case class LocationJson(latitude: Double, longitude: Double)
object LocationJson {
implicit val writes = Json.writes[LocationJson]
}
implicit val fooJsonWriter: Writes[FooJson] = new Writes[FooJson] {
override def writes(o: FooJson): JsValue = {
JsObject(Seq(
"location" -> Json.toJson(o.location)
// Additional fields go here.
))
}
}
Json.toJson(FooJson(None))
And got this result res0: play.api.libs.json.JsValue = {"location":null}.
if we have null values then we have to add the option with members in case class which will resolve the issue
case class response(
name:String,
age: option[int]
)
object response {
implicit val format = Json.format[response]
}
Here the option is the answer for us. and if we are the JSON response for age is coming as null and this will handle the solution for us.
I need to serialize/deserialize a Scala class with structure something like the following:
#JsonIgnoreProperties(ignoreUnknown = true, value = Array("body"))
case class Example(body: Array[Byte]) {
lazy val isNativeText = bodyIsNativeText
lazy val textEncodedBody = (if (isNativeText) new String(body, "UTF-8") else Base64.encode(body))
def this(isNativeText: Boolean, textEncodedBody: String) = this((if(isNativeText) str.getBytes("UTF-8") else Base64.decode(textEncodedBody)))
def bodyIsNativeText: Boolean = // determine if the body was natively a string or not
}
It's main member is an array of bytes, which MIGHT represent a UTF-8 encoded textual string, but might not. The primary constructor accepts an array of bytes, but there is an alternate constructor which accepts a string with a flag indicating whether this string is base64 encoded binary data, or the actual native text we want to store.
For serializing to a JSON object, I want to store the body as a native string rather than a base64-encoded string if it is native text. That's why I use #JsonIgnoreProperties to not include the body property, and instead have a textEncodedBody that gets echoed out in the JSON.
The problem comes when I try to deserialize it like so:
val e = Json.parse[Example]("""{'isNativeText': true, 'textEncodedBody': 'hello'}""")
I receive the following error:
com.codahale.jerkson.ParsingException: Invalid JSON. Needed [body],
but found [isNativeText, textEncodedBody].
Clearly, I have a constructor that will work...it just is not the default one. How can I force Jerkson to use this non-default constructor?
EDIT: I've attempted to use both the #JsonProperty and #JsonCreator annotation, but jerkson appears to disregard both of those.
EDIT2: Looking over the jerkson case class serialization source code, it looks like a case class method with the same name as its field will be used in the way that a #JsonProperty would function - that is, as a JSON getter. If I could do that, it would solve my problem. Not being super familiar with Scala, I have no idea how to do that; is it possible for a case class to have a user-defined method with the same name as one of its fields?
For reference, here is the code below that leads me to this conclusion...
private val methods = klass.getDeclaredMethods
.filter { _.getParameterTypes.isEmpty }
.map { m => m.getName -> m }.toMap
def serialize(value: A, json: JsonGenerator, provider: SerializerProvider) {
json.writeStartObject()
for (field <- nonIgnoredFields) {
val methodOpt = methods.get(field.getName)
val fieldValue: Object = methodOpt.map { _.invoke(value) }.getOrElse(field.get(value))
if (fieldValue != None) {
val fieldName = methodOpt.map { _.getName }.getOrElse(field.getName)
provider.defaultSerializeField(if (isSnakeCase) snakeCase(fieldName) else fieldName, fieldValue, json)
}
}
json.writeEndObject()
}
Correct me if I'm wrong, but it looks like Jackson/Jerkson will not support arbitrarily nested JSON. There's an example on the wiki that uses nesting, but it looks like the target class must have nested classes corresponding to the nested JSON.
Anyway, if you're not using nesting with your case classes then simply declaring a second case class and a couple implicit conversions should work just fine:
case class Example(body: Array[Byte]) {
// Note that you can just inline the body of bodyIsNativeText here
lazy val isNativeText: Boolean = // determine if the body was natively a string or not
}
case class ExampleRaw(isNativeText: Boolean, textEncodedBody: String)
implicit def exampleToExampleRaw(ex: Example) = ExampleRaw(
ex.isNativeText,
if (ex.isNativeText) new String(ex.body, "UTF-8")
else Base64.encode(ex.body)
)
implicit def exampleRawToExample(raw: ExampleRaw) = Example(
if (raw.isNativeText) raw.textEncodedBody.getBytes("UTF-8")
else Base64.decode(textEncodedBody)
)
Now you should be able to do this:
val e: Example = Json.parse[ExampleRaw](
"""{'isNativeText': true, 'textEncodedBody': 'hello'}"""
)
You could leave the original methods and annotations you added to make the JSON generation continue to work with the Example type, or you could just convert it with a cast:
generate(Example(data): ExampleRaw)
Update:
To help catch errors you might want to do something like this too:
case class Example(body: Array[Byte]) {
// Note that you can just inline the body of bodyIsNativeText here
lazy val isNativeText: Boolean = // determine if the body was natively a string or not
lazy val doNotSerialize: String = throw new Exception("Need to convert Example to ExampleRaw before serializing!")
}
That should cause an exception to be thrown if you accidentally pass an instance of Example instead of ExampleRaw to a generate call.