how to parse a complicated json in Scala - json

I am a real newbie in Scala (Using Play framework) and trying to figure out what would be the best way to parse a complicated JSON.
This is an example I have:
{
"id":"test1",
"tmax":270,
"at":2,
"bcat":[
"IAB26",
"IAB25",
"IAB24"
],
"imp":[
{
"id":"1",
"banner":{
"w":320,
"h":480,
"battr":[
10
],
"api":[
]
},
"bidfloor":0.69
}
],
"app":{
"id":"1234",
"name":"Video Games",
"bundle":"www.testapp.com",
"cat":[
"IAB1"
],
"publisher":{
"id":"1111"
}
},
"device":{
"dnt":0,
"connectiontype":2,
"carrier":"Ellijay Telephone Company",
"dpidsha1":"e5f61ae0597d8abee94860d66f7d512aa68d0985",
"dpidmd5":"c1827fe90bae819017dfdb30db7e84fa",
"didmd5":"1f6cb9dc519db8e48cf9592b31cce04e",
"didsha1":"2e6a5d7f5fd1b2b5dea56a80f2b9dc24902a0ca7",
"ifa":"422aeb3a-6507-40b5-9e6f-42a0e14b51be",
"osv":"4.4",
"os":"Android",
"ua":"Mozilla/5.0 (Linux; Android 4.4.2; DL1010Q Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/30.0.0.0 Safari/537.36",
"ip":"24.75.160.74",
"devicetype":1,
"geo":{
"type":1,
"country":"USA",
"city":"Jasper"
}
},
"user":{
"id":"9c2be7c2a3dfe19070f193910e92b2e0"
},
"cur":[
"USD"
]
}
Thank you very much!

I would go with case classes as described in http://json4s.org/#extracting-values e.g.
scala> import org.json4s._
scala> import org.json4s.jackson.JsonMethods._
scala> implicit val formats = DefaultFormats // Brings in default date formats etc.
scala> case class Child(name: String, age: Int, birthdate: Option[java.util.Date])
scala> case class Address(street: String, city: String)
scala> case class Person(name: String, address: Address, children: List[Child])
scala> val json = parse("""
{ "name": "joe",
"address": {
"street": "Bulevard",
"city": "Helsinki"
},
"children": [
{
"name": "Mary",
"age": 5,
"birthdate": "2004-09-04T18:06:22Z"
},
{
"name": "Mazy",
"age": 3
}
]
}
""")
scala> json.extract[Person]
res0: Person = Person(joe,Address(Bulevard,Helsinki),List(Child(Mary,5,Some(Sat Sep 04 18:06:22 EEST 2004)), Child(Mazy,3,None)))

Use implicit to automatically fetch json value with the same name.
if you have a string like
val jsonString =
{ "company": {
"name": "google",
"department": [
{
"name": "IT",
"members": [
{
"firstName": "Max",
"lastName": "Joe",
"age" : 25
},
{
"firstName": "Jean",
"lastName": "Nick",
"age" : 55,
"salary": 100,000
}
]
},
{
"name": "Marketing"
"members": [
{
"firstName": "Mike",
"lastName": "Lucas",
"age" : 43
}
]
},
]
}
}
Then we should do as following to parse it into scala object
val json: JsValue = Json.parse(jsonString) // string to JsValue
case class Person(firstName: String, lastName: String, age: Int, salary: Option[Int]) // keep the lower level object above since you will use them below
implicit personR = Json.reads[Person]
// implicit will read the json attribute if you use the same name as it is in json String. For example "name" : "google" will be read if you have Company(name: String). See the "name" and name:String .
case class Department(name: String, members: Seq[Person])
implicit departmentR = Json.reads[Department]
case class Company(name: String, department: Seq[Department])
implicit val companyR = Json.reads[Company]
val result = json.validate[Company] match {
case s: JsSuccess[Company] => s.get // s.get will return a Company object
case e: JsError => println("error")
}

Don't know if it can be used standalone, but Play Framework has great tools to work with JSON. Take a look here

Related

Scala Json parsing

This is the input json which I am getting which is nested json structure and I don't want to map directly to class, need custom parsing of some the objects as I have made the case classes
{
"uuid": "b547e13e-b32d-11ec-b909-0242ac120002",
"log": {
"Response": {
"info": {
"receivedTime": "2022-02-09T00:30:00Z",
"isSecure": "Yes",
"Data": [{
"id": "75641",
"type": "vendor",
"sourceId": "3",
"size": 53
}],
"Group": [{
"isActive": "yes",
"metadata": {
"owner": "owner1",
"compressionType": "gz",
"comments": "someComment",
"createdDate": "2022-01-11T11:00:00Z",
"updatedDate": "2022-01-12T14:17:55Z"
},
"setId": "1"
},
{
"isActive": "yes",
"metadata": {
"owner": "owner12",
"compressionType": "snappy",
"comments": "someComment",
"createdDate": "2022-01-11T11:00:00Z",
"updatedDate": "2022-01-12T14:17:55Z"
},
"setId": "2"
},
{
"isActive": "yes",
"metadata": {
"owner": "owner123",
"compressionType": "snappy",
"comments": "someComment",
"createdDate": "2022-01-11T11:00:00Z",
"updatedDate": "2022-01-12T14:17:55Z"
},
"setId": "4"
},
{
"isActive": "yes",
"metadata": {
"owner": "owner124",
"compressionType": "snappy",
"comments": "someComments",
"createdDate": "2022-01-11T11:00:00Z",
"updatedDate": "2022-01-12T14:17:55Z"
},
"setId": "4"
}
]
}
}
}
}
Code that I am trying play json also tried circe . Please help ..New to scala world
below is object and case class
case class DataCatalog(uuid: String, data: Seq[Data], metadata: Seq[Metadata])
object DataCatalog {
case class Data(id: String,
type: String,
sourceId: Option[Int],
size: Int)
case class Metadata(isActive: String,
owner: String,
compressionType: String,
comments: String,
createdDate: String,
updatedDate: String
)
def convertJson(inputjsonLine: String): Option[DataCatalog] = {
val result = Try {
//val doc: Json = parse(line).getOrElse(Json.Null)
//val cursor: HCursor = doc.hcursor
//val uuid: Decoder.Result[String] = cursor.downField("uuid").as[String]
val lat = (inputjsonLine \ "uuid").get
DataCatalog(uuid, data, group)
}
//do pattern matching
result match {
case Success(dataCatalog) => Some(dataCatalog)
case Failure(exception) =>
}
}
}
Any parsing api is fine.
If you use Scala Play, for each case class you should have an companion object which will help you a lot with read/write object in/from json:
object Data {
import play.api.libs.json._
implicit val read = Json.reads[Data ]
implicit val write = Json.writes[Data ]
def tupled = (Data.apply _).tupled
}
object Metadata {
import play.api.libs.json._
implicit val read = Json.reads[Metadata ]
implicit val write = Json.writes[Metadata ]
def tupled = (Metadata.apply _).tupled
}
Is required as each companion object to be in same file as the case class. For your json example, you need more case classes because you have a lot of nested objects there (log, Response, info, each of it)
or, you can read the field which you're interested in as:
(jsonData \ "fieldName").as[CaseClassName]
You can try to access the Data value as:
(jsonData \ "log" \ "Response" \ "info" \ "Data").as[Data]
same for Metadata

How to deserialize dynamic Json with pure Kotlin Serialization

I am investigating the use of Kotlin Serialization in my latest project
The Json I have to contend with is as follows:-
{
"countryCodeMapping": {
"ssd": [
{
"name": "South Sudan",
"alpha2": "ss",
"alpha3": "ssd"
}
],
"nic": [
{
"name": "Nicaragua",
"alpha2": "ni",
"alpha3": "nic"
}
],
"fin": [
{
"name": "Finland",
"alpha2": "fi",
"alpha3": "fin"
}
],
"cck": [
{
"name": "Cocos (Keeling) Islands",
"alpha2": "cc",
"alpha3": "cck"
}
],
"tun": [
{
"name": "Tunisia",
"alpha2": "tn",
"alpha3": "tun"
}
],
"uga": [
{
"name": "Uganda",
"alpha2": "ug",
"alpha3": "uga"
}
],
....
"myDescriptions": {
"1": {
"groupId": 1,
"description": "cvbxcvbxcvbxbxb",
"myTypes": [
"cxcvbxb",
"xcvxcbxxx",
"cvbxcvbxcxvcb"
]
},
"2": {
"groupId": 2,
"description": "vxvbxbxbxbx",
"myTypes": [
"xcvb",
"xcvbxcvb"
]
},
"3": {
"groupId": 3,
"description": "xcvbxcvbxcvbxcvb",
"myTypes": [
"xcvbxcvbxcvbx"
]
},
"4": {
"groupId": 4,
"description": "xcvbxcbxcvbxcb",
"myTypes": [
"xcvbxcxcvb"
]
},
"5": {
"groupId": 5,
"description": "xcvbxcvbxcvbxcvbx",
"myTypes": [
"xcvbxcvbxc",
"xcvbxcvbxcvbxb"
]
},
"6": {
"groupId": 6,
"description": "dfgsdfgsdgsdfgsdgsg",
"myTypes": [
I must decode the entire Json string, however I started with the countryCodeMapping
section as follows:-
#Serializable
data class Meta(
#SerialName("countryCodeMapping")
val mapping: CountryCodeMapping
)
#Serializable
data class CountryCodeMapping(val ssd: List<Country>)
#Serializable
data class Country(
#SerialName("name")
val name: String,
#SerialName("alpha2")
val alpha2: String,
#SerialName("alpha3")
val alpha3: String
)
val countryMap: Meta = Json {
isLenient = true
ignoreUnknownKeys = true
allowStructuredMapKeys = true
}.decodeFromString<Meta>(metaJson)
This works, however I obviously only decode Sudan
countryMap = Meta(mapping=CountryCodeMapping(ssd=[Country(name=South Sudan, alpha2=ss, alpha3=ssd)]))
What data class models do I require to enable me to achieve my goal?
UPDATE
The changes I had to make (Based on answer comment) were
#Serializable
data class Meta(
#SerialName("countryCodeMapping")
val mapping: Map<String, List<Country>>
)
#Serializable
data class Country(
#SerialName("name")
val name: String? = null,
#SerialName("alpha2")
val alpha2: String? = null,
#SerialName("alpha3")
val alpha3: String? = null
)
You can just use a regular map here:
val mapping: Map<String, List<Country>>
When you have JSON objects like { "someKey": ..., "otherKey": ... }, you have mainly 2 options to represent them:
use a regular class with properties
use a generic Map<String, ...>
You current approach uses option 1 (with CountryCodeMapping class, and hardcoded ssd property). This approach is best when the keys are fixed and well known.
Approach 2 is better when the keys are dynamic and all the values have the same type, which is your case here.
To implement approach 2, use what #broot suggested: a Map<String, List<Country>> instead of the CountryCodeMapping class:
#Serializable
data class Meta(
#SerialName("countryCodeMapping")
val mapping: Map<String, List<Country>>
)
#Serializable
data class Country(
#SerialName("name")
val name: String,
#SerialName("alpha2")
val alpha2: String,
#SerialName("alpha3")
val alpha3: String
)

Can't handle json tree list using TypeToken + generics with Gson in Kotlin

I have got the json structure as below:
{
"object": "list",
"total": 3,
"data": [
{
"object": "brand",
"id": "15243937043340",
"company": {
"object": "company",
"id": "956936000",
"name": "ABC"
},
"name": "Kindle",
"images": [
"http://www.spacecentrestorage.com/assets/uploads/General/SCS-Slide02-Commercial.jpg"
]
},
{
"object": "brand",
"id": "15243937043340",
"company": {
"object": "company",
"id": "956936000",
"name": "ABC"
},
"name": "Kindle",
"images": [
"http://www.spacecentrestorage.com/assets/uploads/General/SCS-Slide02-Commercial.jpg"
]
},
{
"object": "brand",
"id": "15243937043340",
"company": {
"object": "company",
"id": "956936000",
"name": "ABC"
},
"name": "Kindle",
"images": [
"http://www.spacecentrestorage.com/assets/uploads/General/SCS-Slide02-Commercial.jpg"
]
}
],
"associated": {}
}
And this is my Gson data class mapping :
data class Response (
#SerializedName("object")
val obj: String,
val total: Int,
val data: List<*>,
val associated: Response
)
data class Brand (
#SerializedName("object")
val obj: String,
val id: String,
val name: String,
val images: List<String>,
val company: Company
)
data class Company (
#SerializedName("object")
val obj: String,
val id: String,
val name: String
)
When it comes to extracting the tree as above, I find returned data string becomes Malformed Json and gives MalformedJsonException on $[0].companies.null
I have read about the recursive deserialisation function but it is not working in my case. I resort to deserialise as below , using original method, it causes errors
val response = gson.fromJson(queryResult , Response::class.java)
println("result 2 : $response" )
val dataString = response.data.toString()
println("result 3 : $dataString" )
val brands = Gson().fromJson(dataString, Array<Brand>::class.java).toMutableList()
println("result 4 : $brands" )
I would like to ask :
If returning json component to string, shall all the indents and symbols " be erased ?
To extract all associated object of the elements of the list of objects, what precautions do I have to take for deserialising list of objects using Gson ?
If you set the type parameter of the data list in Response to Brand GSON knows how to deserialise the items of the list.
data class Response (
#SerializedName("object")
val obj: String,
val total: Int,
val data: List<Brand>,
val associated: Response
)
Using this there is no need to parse the items of the list again and you can get all brands like this:
val response = Gson().fromJson(queryResult , Response::class.java)
val dataList = response.data
print("brands: " )
dataList.forEach { println(it) }

How to parse just part of JSON with Klaxon?

I'm trying to parse some JSON to kotlin objects. The JSON looks like:
{
data: [
{ "name": "aaa", "age": 11 },
{ "name": "bbb", "age": 22 },
],
otherdata : "don't need"
}
I just need to data part of the entire JSON, and parse each item to a User object:
data class User(name:String, age:Int)
But I can't find an easy way to do it.
Here's one way you can achieve this
import com.beust.klaxon.Klaxon
import java.io.StringReader
val json = """
{
"data": [
{ "name": "aaa", "age": 11 },
{ "name": "bbb", "age": 22 },
],
"otherdata" : "not needed"
}
""".trimIndent()
data class User(val name: String, val age: Int)
fun main(args: Array<String>) {
val klaxon = Klaxon()
val parsed = klaxon.parseJsonObject(StringReader(json))
val dataArray = parsed.array<Any>("data")
val users = dataArray?.let { klaxon.parseFromJsonArray<User>(it) }
println(users)
}
This will work as long as you can fit the whole json string in memory. Otherwise you may want to look into the streaming API: https://github.com/cbeust/klaxon#streaming-api

Argonaut: decoding a polymorphic array

The JSON object for which I'm trying to write a DecodeJson[T] contains an array of different "types" (meaning the JSON structure of its elements is varying). The only common feature is the type field which can be used to distinguish between the types. All other fields are different. Example:
{
...,
array: [
{ type: "a", a1: ..., a2: ...},
{ type: "b", b1: ...},
{ type: "c", c1: ..., c2: ..., c3: ...},
{ type: "a", a1: ..., a2: ...},
...
],
...
}
Using argonaut, is it possible to map the JSON array to a Scala Seq[Element] where Element is a supertype of suitable case classes of type ElementA, ElementB and so on?
I did the same thing with play-json and it was quite easy (basically a Reads[Element] that evaluates the type field and accordingly forwards to more specific Reads). However, I couldn't find a way to do this with argonaut.
edit: example
Scala types (I wish to use):
case class Container(id: Int, events: List[Event])
sealed trait Event
case class Birthday(name: String, age: Int) extends Event
case class Appointment(start: Long, participants: List[String]) extends Event
case class ... extends Event
JSON instance (not under my control):
{
"id":1674,
"events": {
"data": [
{
"type": "birthday",
"name": "Jones Clinton",
"age": 34
},
{
"type": "appointment",
"start": 1675156665555,
"participants": [
"John Doe",
"Jane Doe",
"Foo Bar"
]
}
]
}
}
You can create a small function to help you build a decoder that handles this format.
See below for an example.
import argonaut._, Argonaut._
def decodeByType[T](encoders: (String, DecodeJson[_ <: T])*) = {
val encMap = encoders.toMap
def decoder(h: CursorHistory, s: String) =
encMap.get(s).fold(DecodeResult.fail[DecodeJson[_ <: T]](s"Unknown type: $s", h))(d => DecodeResult.ok(d))
DecodeJson[T] { c: HCursor =>
val tf = c.downField("type")
for {
tv <- tf.as[String]
dec <- decoder(tf.history, tv)
data <- dec(c).map[T](identity)
} yield data
}
}
case class Container(id: Int, events: ContainerData)
case class ContainerData(data: List[Event])
sealed trait Event
case class Birthday(name: String, age: Int) extends Event
case class Appointment(start: Long, participants: List[String]) extends Event
implicit val eventDecoder: DecodeJson[Event] = decodeByType[Event](
"birthday" -> DecodeJson.derive[Birthday],
"appointment" -> DecodeJson.derive[Appointment]
)
implicit val containerDataDecoder: DecodeJson[ContainerData] = DecodeJson.derive[ContainerData]
implicit val containerDecoder: DecodeJson[Container] = DecodeJson.derive[Container]
val goodJsonStr =
"""
{
"id":1674,
"events": {
"data": [
{
"type": "birthday",
"name": "Jones Clinton",
"age": 34
},
{
"type": "appointment",
"start": 1675156665555,
"participants": [
"John Doe",
"Jane Doe",
"Foo Bar"
]
}
]
}
}
"""
def main(args: Array[String]) = {
println(goodJsonStr.decode[Container])
// \/-(Container(1674,ContainerData(List(Birthday(Jones Clinton,34), Appointment(1675156665555,List(John Doe, Jane Doe, Foo Bar))))))
}