Scala JSON Parsing Nested Array into Case Class - json

I'm using the standard Scala json parsing and I am running into issues with the nested arrays.
Here is sample JSON:
{
"_id": "id_here",
"_rev": "rev_here",
"RandomHosts": [
"randomhosts_1",
"randomhosts_2",
"randomhosts_3",
"randomhosts_4"
],
"FirstObject": {
"Host": "ActualHost",
"Port": 8888,
"DB": 0,
"WFMDB": 1,
"ETLDB": 2,
"HostListPrefix": "Dev1",
"ExtraHostsBands": [
{
"Host": "dev2",
"Port": 2222,
"DB": 0,
"WFMDB": 1,
"ETLDB": 2,
"HostListPrefix": "Dev2"
},
{
"Host": "dev3",
"Port": 3333,
"DB": 0,
"WFMDB": 1,
"ETLDB": 3,
"HostListPrefix": "Dev3"
}
],
"RandomObject":{}
}
}
// I HAVE OTHER IMPORTS AS WELL
import scala.util.parsing.json._;
case class BandClass (
Host: String,
Port:Int,
DB:Int,
WFMDB:Int,
ETLDB:Int,
HostListPrefix:String
);
var jsonString = "<myjson>";
val bandconfig = JSON.parseFull(jsonString);
// THIS WORKS PERFECT AND GIVES ME JOINED STRING
val random_hosts = bandconfig.get.asInstanceOf[Map[String, Any]]("RandomHosts").asInstanceOf[List[String]].mkString(",");
//THIS ALSO WORKS PERFECT AND GIVES ME HOST AND PORT
val firstObject = bandconfig.get.asInstanceOf[Map[String, Any]]("FirstObject").asInstanceOf[Map[String, Any]];
val firstObjectHost = firstObject("Host").asInstanceOf[String]
val firstObjectPort = firstObject("Port").asInstanceOf[Double].toInt
//THIS IS WHERE EVERYTHING FALLS APART
val extraBands = firstObject("ExtraHostBands").asInstanceOf[List[BandClass]]
//EVEN THIS DOESNT WORK
val extraBands2 = firstObject("ExtraHostBands").asInstanceOf[Map[String, Any]]
Caused by: java.lang.ClassCastException: scala.collection.immutable.HashMap$HashTrieMap cannot be cast to $BandClass
I'm not sure how to force that nested json array into my case class. Id even settle for a map or seq or anything I could iterate over to get the host/ports out of the ExtraHostBand json objects.
Can anyone point me in the correct direction to get that json array into case class? I also have access to play-json but cant seem to figure that out either.

Ended up going to play-json and it worked really well. Here is a solution in case someone needs this in the future:
import play.api.libs.json.Json;
import play.api.libs.json;
import play.api.libs.json.Writes;
import play.api.libs.json._;
import play.api.libs._;
import play.api.libs.functional.syntax._;
import play.api.libs.json.Reads._;
case class BandClass (
Host: String,
Port:Int,
DB:Int,
WFMDB:Int,
ETLDB:Int,
HostListPrefix:String
)
case class FirstObject (
Host: String,
Port:Int,
DB:Int,
WFMDB:Int,
ETLDB:Int,
HostListPrefix:String,
ExtraHostsBands: List[BandClass]
)
case class RawConfig (
_id: String,
_rev: String,
RandomHosts: List[String],
FirstObject: FirstObject
)
implicit val bandClassFormat = Json.format[BandClass];
implicit val firstObjectFormat = Json.format[FirstObject];
implicit val rawConfigFormat = Json.format[RawConfig];
val playJsonParse = Json.parse(<myjson>).as[RawConfig];
println("playJSON ID " + playJsonParse._id)
println("playJSON REV " + playJsonParse._rev)
playJsonParse.FirstObject.ExtraHostBands.foreach
{
case(r) => {
println("Host " + r.Host);
println("Host Prefix " + r.HostListPrefix);
println("ETLDB " + r.ETLDB);
}
}

Related

How to add another object in JSON with Ktor

I'm creating API server with Ktor and using exposed SQL.
I have two tables like below:
/// Performance Table
#kotlinx.serialization.Serializable
data class PerformanceDAO(val performanceId: String, val title: String, val actor: String)
object Performances : Table() {
val performanceId: Column<String> = varchar("performanceId", 20)
val title: Column<String> = varchar("title", 50)
override val primaryKey = PrimaryKey(performanceId)
}
//Actor Table
object Actors: Table() {
val performanceId: Column<String> = reference("performanceId", Performances.performanceId)
val name: Column<String> = varchar("name", 10)
}
and I'm trying to get data this way:
class PerformanceDAOImpl : PerformanceDAOFacade {
private fun resultRowToPerformance(row: ResultRow) = PerformanceDAO(
performanceId = row[Performances.performanceId],
title = row[Performances.title],
actor = row[Actors.name],
)
override suspend fun allPerformances(): List<PerformanceDAO> = dbQuery {
Actors.leftJoin(Performances)
.slice(Actors.name, Performances.title, Performances.performanceId)
.selectAll()
.groupBy(Performances.title).map(::resultRowToPerformance)
}
}
and then I got only one actors without a key.
{
"performanceId": "PF13234",
"title": "Harry footer",
"actor": [
"John"
]
},
but I want get data like this
{
"performanceId": "PF13234",
"title": "Harry footer",
"actor": [
{
"name: "John",
"image": "/upload/images/image.jpg"
},
{
"name: "Harry",
"image": "/upload/images/image.jpg"
},
]
},
I want to know how to make sub object in JSON with Exposed SQL!
You have to create a separate data class ActorDTO and map it in your code by making subquery. Also, I would advice to make Exposed Entities and then map them to your DAO, it can be much simpler but will add some boilerplate code.
Please check documentation

Using Json format in Confluent Platform Schema Registry?

I'm trying to put several event types in the same Kafka topic using the JSON format, but in the Producer implementation I'm always getting org.apache.kafka.common.errors.SerializationException: Error serializing JSON message. Seems that the annotation #Schema isn't working as expected is like the schema defined by the annotation isn't enriched properly and in the method that validates the backward compatibility the schema defined by my event has the schemaObj empty and the result is not compatible and fails.
My event:
#Schema(
value = "1",
refs = Array(new SchemaReference(name = "event", subject = "event"))
)
case class Event(#BeanProperty id: String,
#BeanProperty name: String)
Producer:
def send(): Unit = {
val props = new Properties() {
put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092")
put(
ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringSerializer"
)
put(
ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
"io.confluent.kafka.serializers.json.KafkaJsonSchemaSerializer"
)
put("auto.register.schemas", "false")
put("use.latest.version", "true")
put("schema.registry.url", "http://127.0.0.1:8081")
put("json.fail.invalid.schema", "true")
}
val producer = new KafkaProducer[String, Event](props)
val topic = "all-json"
val key = "key1"
val event = Event("id", "name")
val record = new ProducerRecord[String, Event](topic, key, event)
producer.send(record).get
}
By the command line, I can perfectly produce the events. The JSON Schema is modeled by
{
"oneOf": [
{ "$ref": "Event.schema.json" },
{ "$ref": "EventB.schema.json" }
]
}
...
the dependencies of confluent used are the version 6.0.1.
Do you know what is the issue?

Pass DataType of the Class

I have the following data class, which stores two values, JSON and dataType:
data class DataTypeItem(
var json : String = "",
var dataType : Class<*> ?= null
)
I have the list defined in the following way:
val dataTypeList = mutableMapOf<String, DataTypeItem>()
dataTypeList.put( "item_1", DataTypeItem( json1, MyDataType::class.java ) )
dataTypeList.put( "item_2", DataTypeItem( json1, List<MyDataType>::class.java ) )
Please note that in one case I'm using the MyDataType as the DataType and in the other List < MyDataType >.
Now I would like to loop through each of the dataTypeList items and parse JSON for the given data type into it's model:
fun init()
{
dataTypeList.forEach {
dataTypeItem ->
val model = Gson().fromJson( dataTypeItem.value.json, dataTypeItem.value.dataType::class.java )
}
}
I'm using the following model:
data class dataTypeItem(
#SerializedName("sqlId")
val sqlId: String,
#SerializedName("name")
val name: String
)
But I keep getting an Runtime exception:
Attempted to deserialize a java.lang.Class. Forgot to register a type adapter?
In addition, in case it's a list, I need to call toList() on Gson().fromJSON(..):
fun init()
{
dataTypeList.forEach {
dataTypeItem ->
val model;
if( dataTypeItem.value.dataType::class.java is Array )
model = Gson().fromJson( dataTypeItem.value.json, dataTypeItem.value.dataType::class.java ).toList()
else
model = Gson().fromJson( dataTypeItem.value.json, dataTypeItem.value.dataType::class.java )
}
}
How can I pass the dataType dynamically and distinguish if it's a List/Array or straight up class? In addition, whenever I try to call toList(), I get an error that it's undefined.
If I specify the class directly, then it's working fine
var model = Gson().fromJson( json, DataTypeItem::class.java )
or
var model = Gson().fromJson( json, Array<DataTypeItem>::class.java )
but I need to be able to specify it dynamically as an argument
This code works fine:
val dataTypeMap = mapOf(
"item_1" to MyDataTypeItem("""{"sqlId" : "1", "name" : "a"}""", MyDataType::class.java),
"item_2" to MyDataTypeItem("""[{"sqlId" : "1", "name" : "a"}, {"sqlId" : "2", "name" : "b"}]""", Array<MyDataType>::class.java)
)
val result = dataTypeMap.map{ Gson().fromJson(it.value.json, it.value.dataType) }
I renamed DataTypeItem to MyDataTypeItem and dataTypeItem to MyDataType.
Why you need to call toList()? If it is really necessary you can do the following instead:
val result = dataTypeMap.map {
if (it.value.dataType?.isArray == true) Gson().fromJson<Array<*>>(it.value.json, it.value.dataType).toList()
else Gson().fromJson(it.value.json, it.value.dataType)
}

Scala JSON Rapture API throws exception

I am trying to use rapture.io Scala JSON parser to parse a JSON value (rows) that looks like this:
{
rows:
[
[
null,
"2016-11-16T15:43:18.000Z",
{
"p": 1,
"q": 2
},
null,
"Game highlights"
],
[
null,
"2007-10-09T01:52:29.000Z",
{
"p": 21,
"q": 99
},
"blaah",
"Game reviews"
]
}
My code looks like this:
import rapture.io._
import rapture.codec._
import rapture.json._
import rapture.data._
import rapture.uri._
import rapture.net._
import encodings.system
import jsonBackends.jawn._
class NotesDownloader () {
def download(): Unit = {
val src = uri"https://some_url".slurp[Char]
val jsonResponse = Json.parse(src)
val rows = jsonResponse.data.rows
val rowsBean = rows(0).as[Array[Member]]
println(jsonResponse)
}
case class Member(array: Array[Some[String]])
}
When I try to extract the complete data into Member, I get this exception:
Error:(40, 30) not enough arguments for method as: (implicit ext: rapture.data.Extractor[Array[NotesDownloader.this.Member],rapture.json.Json], implicit mode:
rapture.core.Mode[rapture.data.ExtractionMethods])mode.Wrap[Array[NotesDownloader.this.Member],rapture.data.DataGetException].
Unspecified value parameters ext, mode. val rowsBean = value.as[Array[Member]]
what am I missing?
As long as I know, you could use something like this.
Json.parse(str).as[List[Member]]) since it's a list not an a simple array.
The error message tells you that you need values for the implicit parameters ext and mode. It would be something like this:
implicit val ext = ...
implicit val mode = ...
val rowsBean = rows(0).as[Array[Member]] // this uses the above implicits

Cryptic Spray Json error message

I'm receiving the following error message when trying to parse some json:
[info] The future returned an exception of type: spray.httpx.PipelineException, with message:
Vector("eba760a81b177051b0520418b4e10596955adb98196c15367a2467ab66a19b5c", 600, "AN51SPP6iZBHFJ3aux1jtn6MMMD13Gh3t7", 500,
["1BXVXP82f7x9YWdWuCaCYwad8ZoYayyRYt"], "76a91473758c13a91699376abb8fe76931bdd9bdc04ee388ac", false)
(of class scala.collection.immutable.Vector). (AddressUnspentTXORequestTest.scala:14)
and I'm not really sure what that means. Here is the piece of json that I am trying to parse:
[
{
"transaction_hash": "eba760a81b177051b0520418b4e10596955adb98196c15367a2467ab66a19b5c",
"output_index": 1,
"value": 600,
"asset_id": "AN51SPP6iZBHFJ3aux1jtn6MMMD13Gh3t7",
"asset_quantity": 500,
"addresses": [
"1BXVXP82f7x9YWdWuCaCYwad8ZoYayyRYt"
],
"script_hex": "76a91473758c13a91699376abb8fe76931bdd9bdc04ee388ac",
"spent": false,
"confirmations": 31674
},
{
"transaction_hash": "1f9f6224bee8813135aba622693c78a33b3460e4efdb340174f87fdd8c9d4148",
"output_index": 1,
"value": 600,
"asset_id": "AS6tDJJ3oWrcE1Kk3T14mD8q6ycHYVzyYQ",
"asset_quantity": 200000,
"addresses": [
"1BXVXP82f7x9YWdWuCaCYwad8ZoYayyRYt"
],
"script_hex": "76a91473758c13a91699376abb8fe76931bdd9bdc04ee388ac",
"spent": false,
"confirmations": 35895
}
]
and here is the case class that I am trying to parse it into:
case class UnspentTXO(transaction_hash: String, output_index: Int, value: Long,
asset_id: Option[String], asset_quantity: Option[Long], addresses: List[BitcoinAddress],
script_hex: String, spent: Boolean)
The method initiating the request is here :
def getUnspentTXOs(address: Address): Future[List[UnspentTXO]] = {
val pipeline: HttpRequest => Future[List[UnspentTXO]] =
sendReceive ~> unmarshal[List[UnspentTXO]]
pipeline(Get(host + path + address.value + "/unspents"))
}
and finally this is how I am parsing that Json Request:
override def read(value: JsValue): UnspentTXO = {
val Seq(transaction_hash, output_index, locked_satoshies, asset_id, asset_quantity, addresses, script_hex, spent) =
value.asJsObject.getFields("transaction_hash", "value", "asset_id", "asset_quantity", "addresses", "script_hex", "spent")
val assetId = asset_id match {
case JsString(s) => Some(s)
case JsNull => None
case _ => throw new RuntimeException("Asset id should be of type JsString or JsNull, got something else")
}
val assetQuantity = asset_quantity match {
case JsNumber(n) => Some(n.toLong)
case JsNull => None
case _ => throw new RuntimeException("Asset quantity should be JsNull or a JsNumber")
}
// convert JsArray to List[ BitcoinAdress ]
val addressList = addresses match {
case ja: JsArray => {
ja.elements.toList.map( e => BitcoinAddress(e.convertTo[String]))
}
case _ => throw new RuntimeException("address list should be of type JsArray, got something else")
}
UnspentTXO(transaction_hash.convertTo[String], output_index.convertTo[Int], locked_satoshies.convertTo[Long],
assetId, assetQuantity, addressList,
script_hex.convertTo[String], spent.convertTo[Boolean])
}
I think the problem might be that the request is returning a json array instead of just a JSON object, so I am not sure if I am handling that correctly inside of my getUnspentTXOs. The error message seems to be very vague. It seems that spray is trying to wrap the json fields inside of a Vector instead of inside of a UnspentTXO case class. I'm not sure of why this is happening though.
You can not just call convertTo[ Option[ Long ] ] and convertTo[ List [ BitcointAddress ] ].
This is how convertTo is defined,
def convertTo[T :JsonReader]: T = jsonReader[T].read(this)
Which means... only types T for which an implicit evidence of typeclass JsonReader[ T ] is available can be used with convertTo.
Unless you provide appropriate implicit typeclass evidence, you will have to specially handle few cases.
Other than this, spray-json is just too minimalistic... so that so JsObject is just a wrapper on top of Map[ String, JsValue] and getFields is defined as,
def getFields(fieldNames: String*): immutable.Seq[JsValue] =
fieldNames.flatMap(fields.get)(collection.breakOut)
Which means... it will just ignore asked fields which are not present in them map, and only return a sequence of JsValue's corresponding to the asked fields which are present.
Hence optional values have to be checked to be present in the map. Or we will have to pattern-match for each possible case ( which can result in a lot of cases).
override def read(value: JsValue): UnspentTXO = {
val jsObject = value.asJsObject
// get only non-optional values here
val Seq( transaction_hash, output_index, locked_satoshies, addresses,
script_hex, spent ) =
jsObject.getFields( "transaction_hash", "output_index", "value", "addresses", "script_hex", "spent" )
// Assuming you have imported spray.json._, simple types will work.
// have to handle options differently
// or 2 optional values would mean 4 patterns-matchings of sequences like above.
// jsObject.fields is just a Map[ String, JsValue ]
val assetId = jsObject.fields.get( "asset_id" ) match {
case Some( JsString( s ) ) => Some( s )
case None => None
}
val assetQuantity = jsObject.fields.get( "asset_quantity" ) match {
case Some( JsNumber( n ) ) => Some( n.toLong )
case None => None
}
// convert JsArray to List[ BitcoinAdress ]
val addressList = addresses match {
case ja : JsArray => {
ja.elements.toList.map( BitcoinAddress( _.convertTo[ String ] ) )
}
}
UnspentTXO( transaction_hash.convertTo[ String ], output_index.convertTo[ Int ],
locked_satoshies.convertTo[ Long ], assetId, assetQuantity,
addressList, script_hex.convertTo[ String ], spent.convertTo[ Boolean ] )
}
Another way to get convertTo[ Option[ T ] ] and convertTo[ List[ T ] ] working is by importing spray.json.DefaultJsonProtocol._ which provides json-formats for most generally used types. But even then, you must have an implicit evidence of typeclass JsonReader[ T ] in your scope.