Parse a nested JSON with Kotlinx.Serialization - json

I've been playing with Kotlinx.serialization, and I have been trying to parse a substring:
Given a JSON like:
{
"Parent" : {
"SpaceShip":"Tardis",
"Mark":40
}
}
And my code is something like:
data class SomeClass(
#SerialName("SpaceShip") ship:String,
#SerialName("Mark") mark:Int)
Obviously, Json.nonstrict.parse(SomeClass.serializer(), rawString) will fail because the pair "SpaceShip" and "Mark" are not in the root of the JSON.
How do I make the serializer refer to a subtree of the JSON?
P.S: Would you recommend retrofit instead (because it's older, and maybe more mature)?

#Serializable
data class Parent(
#SerialName("Parent")
val someClass: SomeClass
)
#Serializable
data class SomeClass(
#SerialName("SpaceShip")
val ship: String,
#SerialName("Mark")
val mark: Int
)
fun getSomeClass(inputStream: InputStream): SomeClass {
val json = Json(JsonConfiguration.Stable)
val jsonString = Scanner(inputStream).useDelimiter("\\A").next()
val parent = json.parse(Parent.serializer(), jsonString)
return parent.someClass
}

import kotlinx.serialization.*
import kotlinx.serialization.json.Json
#Serializable
data class Parent(
#SerialName("Parent")
val parent: SomeClass
)
#Serializable
data class SomeClass(
#SerialName("SpaceShip")
val ship:String,
#SerialName("Mark")
val mark:Int
)
fun main() {
val parent = Json.parse(Parent.serializer(), "{\"Parent\":{\"SpaceShip\":\"Tardis\",\"Mark\":40}}")
println(parent)
}

Related

Polymorphic deserialization with kotlinx.serialization in Kotlin/Native

I want to decode a json string containing a list of objects in a polymorphic class structure using kotlinx.serialization in a Kotlin Multiplatform project, but it works only on JVM, not on Native. Here is a minimum reproducible example:
#Serializable
abstract class Project {
abstract val name: String
}
#Serializable
#SerialName("BasicProject")
data class BasicProject(override val name: String): Project()
#Serializable
#SerialName("OwnedProject")
data class OwnedProject(override val name: String, val owner: String) : Project()
fun main() {
val data = Json.decodeFromString<List<Project>>("""
[
{"type":"BasicProject","name":"example"},
{"type":"OwnedProject","name":"kotlinx.serialization","owner":"kotlin"}
]
"""))
}
This works on JVM but throws the following exception on Native:
kotlinx.serialization.SerializationException: Serializer for class ‘Project’ is not found.
Mark the class as #Serializable or provide the serializer explicitly.
On Kotlin/Native explicitly declared serializer should be used for interfaces and enums without #Serializable annotation.message
This problem has been discussed before in the context of encoding and some workarounds have been suggested, e.g. here, but my problem is decoding. Is there a workaround, or do I simply have to implement my own json parser?
You need to explicitly pass respectful serializer and serializersModule:
object ListOfProjectSerializer : KSerializer<List<Project>> by ListSerializer(Project.serializer())
val module = SerializersModule {
polymorphic(Project::class) {
subclass(BasicProject::class)
subclass(OwnedProject::class)
}
}
fun main() {
val data = Json { serializersModule = module }.decodeFromString(
ListOfProjectSerializer,
"""
[
{"type":"BasicProject","name":"example"},
{"type":"OwnedProject","name":"kotlinx.serialization","owner":"kotlin"}
]
"""
)
}

Deserialize JSON array with different values type with kotlinx.serialization library

I'm trying to deserialize following String:
val stringJson = "{\"decomposed\":[\", \",{\"id\":4944372,\"name\":\"Johny\",\"various\":false,\"composer\":false,\"genres\":[]}]}"
Deserialization works fine with following code
#Serializable
data class Artist(
val decomposed: JsonArray
)
fun main() {
val jsonString = "{\"decomposed\":[\", \",{\"id\":4944372,\"name\":\"Johny\",\"various\":false,\"composer\":false,\"genres\":[]}]}"
println(Json.decodeFromString<Artist>(jsonString))
}
But I want to do something like
#Serializable
class Decomposed {
#Serializable
class DecomposedClassValue(val value: DecomposedClass)
#Serializable
class StringValue(val value: String)
}
#Serializable
data class DecomposedClass(
val id: Long? = null,
val name: String? = null,
val various: Boolean? = null,
val composer: Boolean? = null,
val genres: JsonArray? = null
)
#Serializable
data class Artist(
val decomposed: List<Decomposed>
)
fun main() {
val jsonString = "{\"decomposed\":[\", \",{\"id\":4944372,\"name\":\"Johny\",\"various\":false,\"composer\":false,\"genres\":[]}]}"
println(Json.decodeFromString<Artist>(jsonString))
}
But kotlinx.serialization expectedly fails with JsonDecodingException: Unexpected JSON token at offset 15: Expected '{, kind: CLASS'
And I can't figure out how can I rewrite my Decomposed so deserialization work. Can you please help me out?
What you are trying to do is called polymorphic deserialization.
It requires target classes of deserialization to have a common superclass (preferrably sealed):
#Serializable
data class Artist(
val decomposed: List<Decomposed>
)
#Serializable
sealed class Decomposed
#Serializable
class StringValue(val value: String) : Decomposed() //Can't add superclass to String, so we have to create a wrapper class which we could make extend Decomposed
#Serializable
data class DecomposedClass(
val id: Long? = null,
val name: String? = null,
val various: Boolean? = null,
val composer: Boolean? = null,
val genres: JsonArray? = null
) : Decomposed() //DecomposedClassValue is redundant, we may extend DecomposedClass from Decomposed directly
This will allow you to deserialize JSON of the following format:
val jsonString = "{\"decomposed\":[{\"type\":\"StringValue\", \"value\":\",\"}, {\"type\":\"DecomposedClass\", \"id\":4944372,\"name\":\"Johny\",\"various\":false,\"composer\":false,\"genres\":[]}]}"
Since there is no class descriminator in original JSON, serialization library can't determine the actual serializer which should be used to deserialize Kotlin class. You will have to write custom JsonContentPolymorphicSerializer and wire it to Decomposed class; also you have to write custom serializer for StringValue class, as it is represented in JSON as a String, not a JSONObject with a value field of String type:
object DecomposedSerializer : JsonContentPolymorphicSerializer<Decomposed>(Decomposed::class) {
override fun selectDeserializer(element: JsonElement) = when {
element is JsonPrimitive -> StringValue.serializer()
else -> DecomposedClass.serializer()
}
}
object StringValueSerializer : KSerializer<StringValue> {
override val descriptor: SerialDescriptor = buildClassSerialDescriptor("StringValue")
override fun deserialize(decoder: Decoder): StringValue {
require(decoder is JsonDecoder)
val element = decoder.decodeJsonElement()
return StringValue(element.jsonPrimitive.content)
}
override fun serialize(encoder: Encoder, value: StringValue) {
encoder.encodeString(value.value)
}
}
#Serializable(with = DecomposedSerializer::class)
sealed class Decomposed
#Serializable(with = StringValueSerializer::class)
class StringValue(val value: String) : Decomposed()
This will allow you to deserialize JSON of original format.

convert json string to case class object from given json string and type of case class

Requirement is to convert json string to case class object in scala given jsonString and the type of the case class.
I have tried Gson and jackson libraries, but not able to solve the given requirment.
package eg.json
import com.fasterxml.jackson.databind.ObjectMapper
import com.google.gson.Gson
import com.typesafe.scalalogging.LazyLogging
case class Person(name : String, age : Int)
case class Address(street : String, buildingNumber : Int, zipCode : Int)
case class Rent(amount : Double, month : String)
//there are many other case classes
object JsonToObject extends LazyLogging{
import logger._
def toJsonString(ref : Any) : String = {
val gson = new Gson()
val jsonString = gson.toJson(ref)
jsonString
}
def main(args: Array[String]): Unit = {
val person = Person("John", 35)
val jsonString = toJsonString(person)
//here requirement is to convert json string to case class instance, provided the type of case class instance
val gsonObj = toInstanceUsingGson( jsonString, Person.getClass )
debug(s"main : object deserialized using gson : $gsonObj")
val jacksonObj = toInstanceUsingJackson( jsonString, Person.getClass )
debug(s"main : object deserialized using gson : $jacksonObj")
}
def toInstanceUsingGson[T](jsonString : String, caseClassType : Class[T]) : T = {
val gson = new Gson()
val ref = gson.fromJson(jsonString, caseClassType)
ref
}
def toInstanceUsingJackson[T](jsonString : String, caseClassType : Class[T]) : T = {
val mapper = new ObjectMapper()
val ref = mapper.readValue(jsonString, caseClassType)
ref
}
}
Output of execution of above code is :-
01:32:52.369 [main] DEBUG eg.json.JsonToObject$ - main : object deserialized using gson : Person
Exception in thread "main" com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "name" (class eg.json.Person$), not marked as ignorable (0 known properties: ])
at [Source: (String)"{"name":"John","age":35}"; line: 1, column: 10] (through reference chain: eg.json.Person$["name"])
at com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:60)
at com.fasterxml.jackson.databind.DeserializationContext.handleUnknownProperty(DeserializationContext.java:822)
at com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:1152)
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1589)
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownVanilla(BeanDeserializerBase.java:1567)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:294)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4013)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3004)
at eg.json.JsonToObject$.toInstanceUsingJackson(JsonToObject.scala:49)
at eg.json.JsonToObject$.main(JsonToObject.scala:34)
at eg.json.JsonToObject.main(JsonToObject.scala)
Kindly suggest, how to achieve this using gson or jackson, or suggest some other library with sample example.
Above simplified problem is on github :-
https://github.com/moglideveloper/JsonToScalaObject
With Jackson you can do it like this:
import com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper
val mapper = new ObjectMapper() with ScalaObjectMapper
//this line my be needed depending on your case classes
mapper.registerModule(DefaultScalaModule)
def fromJson[T](json: String)(implicit m: Manifest[T]): T = {
mapper.readValue[T](json)
}
I think it is really clean with Jackson lib.
The usage is like this:
val json: String = ???
val personObject: Person = fromJson[Person](json)
Try using circe by Cats.
add circe to your project (https://circe.github.io/circe/ - Quick Start).
create a case class that represent what you want to build from your json.
declare a decoder
https://circe.github.io/circe/codecs/semiauto-derivation.html
https://github.com/circe/circe
import io.circe.parser.decode
import io.circe.syntax._
case class DataToDecode(name : String,
age : Int,
street : String,
buildingNumber : Int,
zipCode : Int,
amount : Double,
month : String)
object DataToDecode{
implicit val dataToDecode: Decoder[DataToDecode] = deriveDecoder
def decodeData(data: Json) : DataToDecode {
data.as[DataToDecode].right.get
}
}
nice example here

How to load csv with nested columns using Apache Spark

I have a csv file:
name,age,phonenumbers
Tom,20,"[{number:100200, area_code:555},{number:100300, area_code:444}]"
Harry,20,"[{number:100400, area_code:555},{number:100500, area_code:666}]"
How can I load this file in Spark to a RDD/Dataset of Person where Person object looks like:
class Person {
String name;
Integer age;
List<Phone> phonenumbers;
class Phone {
int number;
int area_code;
}
}
Unfortunately, the column names for the nested object don't have quotes in your example. Is that truly the case? Because if they DO have quotes (e.g. well-formed JSON) then you could very easily use the from_json function as below:
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
val schema = new ArrayType(new StructType()
.add("number", IntegerType)
.add("area_code", IntegerType), false)
val converted = input.withColumn("phones", from_json('phonenumbers, schema))
If that's not the case, then you'll need to use your own logic to convert the string into the actual nested object, such as:
import org.apache.spark.sql.functions._
case class Phone(number: Int, area_code:Int)
case class Person(name: String, age: Int, phonenumbers: Array[Phone])
val converted = input.map {
case Row(name: String, age: Int, phonenumbers: String) => {
import scala.util.matching.Regex
val phoneFormat = raw"\{number:(\d{6}), area_code:(\d{3})\}".r
val phones = for (m <- phoneFormat.findAllMatchIn(phonenumbers)) yield Phone(m.group(1).toInt, m.group(2).toInt)
Person(name, age, phones.toArray)
}
}

Vector deserialization by using lift-json

How can i deserialize json array using lift-json to scala vector?
For example:
case class Foo(bar: Vector[Bar])
trait Bar {
def value: Int
}
case class Bar1(value: Int) extends Bar
case class Bar2(value: Int) extends Bar
import net.liftweb.json.{ShortTypeHints, Serialization, DefaultFormats}
implicit val formats = new DefaultFormats {
override val typeHintFieldName = "type"
override val typeHints = ShortTypeHints(List(classOf[Foo],classOf[Bar1],classOf[Bar2]))
}
println(Serialization.writePretty(Foo(Vector(Bar1(1), Bar2(5), Bar1(1)))))
The result is:
{
"type":"Foo",
"bar":[{
"type":"Bar1",
"value":1
},{
"type":"Bar2",
"value":5
},{
"type":"Bar1",
"value":1
}]
}
Good. But when i try to deserialize this string
println(Serialization.read[Foo](Serialization.writePretty(Foo(Vector(Bar1(1), Bar2(5), Bar1(1))))))
i get an exception:
net.liftweb.json.MappingException: Parsed JSON values do not match
with class constructor args=List(Bar1(1), Bar2(5), Bar1(1)) arg
types=scala.collection.immutable.$colon$colon constructor=public
test.Foo(scala.collection.immutable.Vector)
It's means that json array associated with scala list, not vector type that defined in class Foo. I know that there is way to create custom serializer by extending net.liftweb.json.Serializer and include it to formats value. But how can i restore type of objects that stores in Vector. I wanna get result of deserializing like this:
Foo(Vector(Bar1(1), Bar2(5), Bar1(1)))
I've often been annoyed by the List-centricness of Lift, and have found myself needing to do similar things in the past. The following is the approach I've used, adapted a bit for your example:
trait Bar { def value: Int }
case class Bar1(value: Int) extends Bar
case class Bar2(value: Int) extends Bar
case class Foo(bar: Vector[Bar])
import net.liftweb.json._
implicit val formats = new DefaultFormats { outer =>
override val typeHintFieldName = "type"
override val typeHints =
ShortTypeHints(classOf[Bar1] :: classOf[Bar2] :: Nil) +
new ShortTypeHints(classOf[Foo] :: Nil) {
val FooName = this.hintFor(classOf[Foo])
override def deserialize = {
case (FooName, foo) => foo \ "bar" match {
case JArray(bars) => Foo(
bars.map(_.extract[Bar](outer, manifest[Bar]))(collection.breakOut)
)
case _ => throw new RuntimeException("Not really a Foo.")
}
}
}
}
Kind of ugly, and could probably be cleaned up a bit, but it works.
You could add an implicit conversion:
implicit def listToVect(list:List[Bar]):Vector[Bar] = list.map(identity)(breakOut)
after that, Serialization.read[Foo] works as expected.