Trying to deserialize cached json string to data object and getting exception: kotlinx.serialization.json.internal.JsonDecodingException: Expected class kotlinx.serialization.json.JsonObject (Kotlin reflection is not available) as the serialized body of kotlinx.serialization.Polymorphic<List>, but had class kotlinx.serialization.json.JsonArray (Kotlin reflection is not available)
Code used to deserialize
internal inline fun <reified R : Any> String.convertToDataClass() =
Json {
ignoreUnknownKeys = true
}.decodeFromString(R::class.serializer(), this)
Code example:
val jsonString ="""
[{"name1":"value1"}, {"name2":"value2"}]
"""
val dataObject = jsonString.convertToDataClass<List<SomeObject>>()
When going through Ktor pipeline everything works fine but it is breaking on attempt to deserialize the same response body cached as string.
I am aware of that R::class.serializer() is marked as for internal usage but this is the only way known to me how to deserialize generics from string content.
There is a fitting extension function available at kotlinx.serialization.decodeFromString that takes one generic parameter, so you could pass R as generic to that extension.
Check https://github.com/Kotlin/kotlinx.serialization#introduction-and-references. The sample is val obj = Json.decodeFromString<Project>(string), which will fit your needs doing something like this
import kotlinx.serialization.decodeFromString
import kotlinx.serialization.json.Json
internal inline fun <reified R : Any> String.convertToDataClass() =
Json {
ignoreUnknownKeys = true
}.decodeFromString<R>(this)
Related
Problem
I have JSON content in the form of a string, which I first want to traverse programmatically with Jackson. Then, when I have the node of interest, I want to deserialize it.
What I have tried
I have successfully deserialized strings using mapper.readValue, but now I want to perform such an operation on a jsonNode instead of a string.
Libraries
jackson-core:2.9.9
jackson-module-kotlin:2.9.9
Kotlin 1.3.41
kotlin-stdlib-jdk8:1.3.41
Code
package somepackage
import com.fasterxml.jackson.databind.JsonNode
import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.module.kotlin.KotlinModule
import com.fasterxml.jackson.module.kotlin.readValue
import com.fasterxml.jackson.module.kotlin.treeToValue
fun main() {
val mapper = ObjectMapper().registerModule(KotlinModule())
readValueWorksFine(mapper)
treeToValueFails(mapper)
}
fun treeToValueFails(mapper: ObjectMapper) {
val fullJsonContent = """
[{
"product_id":123,
"Comments":
[{
"comment_id": 23,
"message": "Hello World!"
}]
}]
""".trimIndent()
// Traverse to get the node of interest
val commentsNode: JsonNode = mapper.readTree(fullJsonContent).get(0).get("Comments")
// Deserialize
val comments: List<Comment> = mapper.treeToValue<List<Comment>>(commentsNode)
// The line below fails. (I would have expected the exception to be thrown in the line above instead.
// Exception:
// Exception in thread "main" java.lang.ClassCastException: class
// java.util.LinkedHashMap cannot be cast to class somepackage.Comment (java.util.LinkedHashMap is in module
// java.base of loader 'bootstrap'; somepackage.Comment is in unnamed module of loader 'app')
for (comment: Comment in comments) { // This line fails
println(comment.comment_id)
println(comment.message)
}
}
fun readValueWorksFine(mapper: ObjectMapper) {
val commentsJsonContent = """
[{
"comment_id": 23,
"message": "Hello World!"
}]
""".trimIndent()
val comments1: List<Comment> = mapper.readValue<List<Comment>>(commentsJsonContent)
for (comment in comments1) {
println(comment)
}
}
data class Comment(val comment_id: Long, val message: String)
Exception/Output
The code above results in the following exception/output:
Comment(comment_id=23, message=Hello World!)
Exception in thread "main" java.lang.ClassCastException: class java.util.LinkedHashMap cannot be cast to class somepackage.Comment (java.util.LinkedHashMap is in module java.base of loader 'bootstrap'; somepackage.Comment is in unnamed module of loader 'app')
at somepackage.TKt.treeToValueFails(T.kt:39)
at somepackage.TKt.main(T.kt:13)
at somepackage.TKt.main(T.kt)
The problem cause
Even though ObjectMapper.treeToValue is a Kotlin inline extension function with a reified generic parameter (which means that generics are preserved at runtime), it calls the Java ObjectMapper.treeToValue(TreeNode, Class<T>) method. The value passed as Class<T> will loose generic type information for generic types such as List<Comment>, because of type erasure.
So treeToValue can be used for:
mapper.treeToValue<Comment>(commentNode)
but not for:
mapper.treeToValue<List<Comment>>(commentsNode)
Also note that ObjectMapper contains multiple methods that have #SuppressWarnings annotations, which causes some problems not to appear at compile-time, but at run-time.
Solution 1 - use convertValue()
This is the best solution. It uses the Kotlin extension function ObjectMapper.convertValue.
val commentsNode = mapper.readTree(fullJsonContent).get(0).get("Comments")
val comments = mapper.convertValue<List<Comment>>(commentsNode)
Solution 2 - use an ObjectReader
This solution doesn't use jackson-module-kotlin extension functions.
val reader = mapper.readerFor(object : TypeReference<List<Comment>>() {})
val comments: List<Comment> = reader.readValue(commentsNode)
Solution 3 - deserialize in map
Because treeToValue (Kotlin extension function) does work for non-generic types, you can first get the nodes as as list of JsonNodes, and then map each JsonNode to a Comment.
But it's cumbersome that you cannot simply return mapper.treeToValue(it), because that causes type inference compile errors.
val commentsNode = mapper.readTree(fullJsonContent).get(0).get("Comments")
val comments = commentsNode.elements().asSequence().toList().map {
val comment: Comment = mapper.treeToValue(it)
comment
}
I am using spray-json library to deserialize json objects. I want to be able to parse a DateTime object, specified in the nscala-time library, from a json string.
I want to deserialize a json string of this form:
{"updatedAt": "2015-05-18T23:55:49.033Z"}
For this I am defining my own protocol using spray-json's approach. My spray-json protocol goes as follows:
import spray.json._
// I want to de-serialize an object from this library
import com.github.nscala_time.time.Imports._
object CustomProtocol extends DefaultJsonProtocol {
// There are other implicit objects that describe how to format other
// classes before the one below. All of them are independent from each other
implicit object DateTimeFormat extends RootJsonFormat[DateTime] {
def write(dateTime: DateTime) = JsString(dateTime.toString)
def read(json: JsValue) = {
json.asJsObject.getFields("updatedAt") match {
case Seq(JsString(timeStamp)) => DateTime.parse(timeStamp)
case other => deserializationError("DateTime expected, found \n" + other)
}
}
}
My test (using scala-test) goes as follows:
//Previous imports
import CustomProtocol._
test("Can parse DateTime from Json string") {
val DateTimeJson = """ {"updatedAt": "2015-05-18T23:55:49.033Z"} """
val DateTimefromJson = DateTimeJson.parseJson.convertTo[DateTime]
val realDateTime = DateTime.parse("2015-05-18T23:55:49.033Z")
assert(DateTimefromJson.equals(realDateTime))
I get the following compilation error:
{Directory}/src/test/scala/messageJsonParserTest.scala:29: Cannot find
JsonReader or JsonFormat type class for com.github.nscala_time.time.Imports.DateTime
[error] val DateTimefromJson = DateTimeJson.parseJson.convertTo[DateTime]
Can anybody spot the error in my code?
I'm sorry if this question has been asked before. After browsing my mind and the internet for a whole day I have not been able to come up with a solution.
I have an object of akka.http.scaladsl.model.HttpEntity looking like HttpEntity("application/json", {Myjson here})
Is there a way i can fetch my json from the entity without any string manipulations(other than converting to a string and doing a split)
You will need a JSON parser and a glue code between the JSON parser and Akka Http called Unmarshaller.
Akka Http includes unmarshallers for spray-json library. You can find more unmarshallers in hseeberger/akka-http-json library.
If you choose to use spray-json a pseudo code for that would be
case class MyJson(v1: String, v2: Int)
object JsonProtocol extends DefaultJsonProtocol {
implicit val myFormat = jsonFormat2(MyJson)
}
val resp: Future[MyJson] = Unmarshal(response).to[MyJson]
I'm trying to create a resource for 2 services, 1 in application/x-www-form-urlencoded and string payload and the other application/json format with json body.
I have this code:
#POST #Path("/test")
fun test(#Context request: ContainerRequest): Response {
val baos = ByteArrayOutputStream()
request.entityStream.use { it.copyTo(baos) }
val ipnRawData = baos.toString()
var map : Map<String,Any>
map = when (request.headers.getFirst("Content-Type")) {
"application/json" -> objectMapper.convertValue(ipnRawData,Map::class.java) as Map<String,Any>
"application/x-www-form-urlencoded" -> LinkedHashMap()
else -> throw UnsupportedOperationException()
}
//....handle the map
return Response.status(200).build()
}
But when I try to run it with the json option, and body: {"name" :"test"}), I get an error:
"java.lang.IllegalArgumentException: Can not construct instance of java.util.LinkedHashMap: no String-argument constructor/factory method to deserialize from String value ('{
"name" :"test"}')"
Thanks for any help, Yoel
You should use mapper.readValue to deserialize JSON into an object.
Using raw Jackson without the Jackson-Kotlin module:
val map: Map<String, String> = JSON.readValue("""{"name" :"test"}""",
object : TypeReference<Map<String, String>>() {})
This passes in an object expression with superclass TypeReference specifying the the type you are wanting to create with full generics still intact (you method suffers from type erasure).
Instead, if you are using the Jackson-Kotlin module you only need:
val map: Map<String, String> = JSON.readValue("""{"name" :"test"}""")
Since it has helper/extension functions to hide some of the uglier things like TypeReference creation.
You should always use the Jackson-Kotlin module with Kotlin code so that you can instantiate any type of Kotlin object including data classes that have all val parameters and no default constructors, have it understand nullability, and also deal with default values for constructor parameters. A simple stand-alone example:
import com.fasterxml.jackson.module.kotlin.*
val JSON = jacksonObjectMapper() // creates ObjectMapper() and adds Kotlin module in one step
val map: Map<String, String> = JSON.readValue("""{"name" :"test"}""")
Notice the import .* so that it picks up all the extension functions otherwise you need to explicitly import: com.fasterxml.jackson.module.kotlin.readValue
Or in your case the modified code would be:
import com.fasterxml.jackson.module.kotlin.readValue
val objectMapper = jacksonObjectMappe() // instead of ObjectMapper()
...
#POST #Path("/test")
fun test(#Context request: ContainerRequest): Response {
val bodyAsString = request.entityStream.bufferedReader().readText()
val map: Map<String, Any> = when (request.headers.getFirst("Content-Type")) {
"application/json" -> objectMapper.readValue(bodyAsString)
"application/x-www-form-urlencoded" -> LinkedHashMap()
else -> throw UnsupportedOperationException()
}
//....handle the map
return Response.status(200).build()
}
The code has also been cleaned up a little to remove the use of a var and to read the entity stream in a more Kotlin friendly way.
Also note that the Content-Type header may be more complicated, it could contain encoding as well such as:
Content-type: application/json; charset=utf-8
So you may want a utility function that checks if the header is "equal to application/json or starts with application/json;" instead of only an equality check.
Lastly you could pass the request.entityStream directly to objectMapper.readValue and never copy it into a string at all. There are various overloads for readValue that are helpful for these types of inputs.
I had a function in AWS Lambda:
def test(pj: Pojo, context: Context): java.util.List[Document]
that was not initializing the pj with the input JSON values at all.
I found another way of doing AWS Lambda in Scala like this:
import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.module.scala.DefaultScalaModule
val scalaMapper = new ObjectMapper().registerModule(new DefaultScalaModule)
def test(input: InputStream, output: OutputStream): Unit = {
val inputPojo = scalaMapper.readValue(input, classOf[Pojo])
val answer: Seq[Document] = getTheRealAnswer(inputPojo)
val jsonStr = "{ frustration: \"I wish my answer was JSON.\" }"
output.write(jsonStr.getBytes("UTF-8"))
}
and that works, except what I really want to return as an answer is a JSON array of Documents. How should I go about that?
Edit: In my original posting, I wrote: "[the first example] was returning the answer as an error 22. Basically AWS (I think) treated the JSON conversion of the List[Document] as a filename, JSON has plenty of colons, and the error 22 came from colons in filenames not being allowed. Weird." That turned out to be an error in my invocation of the AWS Lambda Function from AWS CLI. I omitted the output filename in the command invocation, and returned JSON was interpreted by AWS CLI as a filename.
Since I wrote this message, I got things to work like this:
def jsonizeDocs(cDocument: Seq[Document]): String = {
val sb=new StringBuilder
for (doc <- cDocument) {
if (sb.nonEmpty) {
sb.append(",")
}
sb.append(doc.toJson)
}
sb.toString
}
Note! This answer is based on a light wrapper I wrote around json4s which I call JSON Extensions
Assuming you are using Scala Objects, import the io.onema.json.Extensions._
import io.onema.json.Extensions._
case class Doc(title: String, content: String)
val listOfDocs = Seq(Doc("Foo", "bar"), Doc("Bar", "Baz"), Doc("Blah", "Bax"))
val json: String = listOfDocs.asJson
println(json)
// [{"title":"Foo","content":"bar"},{"title":"Bar","content":"Baz"},{"title":"Blah","content":"Bax"}]
See the running example here
Now, since you are using a Pojo, you need to import io.onema.json.JavaExtensions._. Assuming you have the following POJO:
public class Document {
private String title;
private String content;
public String getTitle() {return title;}
public String getContent() {return content;}
public void setTitle(String title) { this.title = title;}
public void setContent(String content) {this.content = content;}
}
Use this method in your Scala code like such:
import io.onema.json.JavaExtensions._
import com.example.Document
// ...
def jsonizeDocs(cDocument: Seq[Document]): String = {
val json: String = cDocument.asJson
println(json)
json
}
In AWS Lambda (and to go the other way around) use jsonDecode and a custom object mapper to deserialize to the expected type:
import io.onema.json.JavaExtensions._
import io.onema.json.Mapper
import com.example.Document
val jsonString = """[{"title":"Foo","content":"bar"},{"title":"Bar","content":"Baz"},{"title":"Blah","content":"Bax"}]"""
val mapper: ObjectMapper = Mapper.allowUnknownPropertiesMapper
val doc: Document = jsonString.jsonDecode[Document](mapper)
I have used the method described here quite successfully in a lambda framework that is able to deserialize to AWS lambda events as well as custom types, see a simple example here.
That's it! you can use this library or one of the many JSON serializers in Java or Scala. If you know the type of your objects most libraries will enable you to serialize to JSON and back very easily.