I was looking towards PolymorphicAdapter but all the polymorphic example I could find had a key called "type" or something similar that could be use to differentiate the class to use. However in my case I don't have such key. I'm a bit lost on how to parse such a peculiar json.
{
"infos": {
"1588318": {
"id": "1588318",
"id_user": "9701",
"profile_name": "Profile1",
"views": 100
},
"1588319": {
"id": "1588319",
"id_user": "7391",
"profile_name": "Profile2",
"views": 10
},
"1588320": false,
"1588321": {
"id": "1588321",
"deleted": true
}
}
}
data class UserInfo(val infos: Map<String, UserResult>)
sealed class UserResult {
data class UserDeleted(val id: String, val deleted: Boolean): UserResult()
data class UserInfoCard(
val id: String,
val title: String,
#Json(name = "profile_name") val profileName: String,
val views: Int
): UserResult()
}
In the end I didn't find any solution and after discussing with the API manager he said he would update with a key to determine if it's either a profile or a deleted_profile
Related
The meta is simple, but how do I model analysis for Kotlinx-Serialization?
{
"meta": {
"subject": "33306",
"interval": "weekly"
},
"analysis": {
"2021-07-20": {
"dose": "0.6410"
},
"2021-07-16": {
"dose": "0.9570"
},
"2021-07-09": {
"dose": "0.6880"
}
}
}
I have this at the moment.
#Serializable
class Observation(
#SerialName("meta")
val meta: Meta,
#SerialName("analysis")
val analysis: Map<String, Map<String, String>>
)
But I get the error.
kotlinx.serialization.json.internal.JsonDecodingException: Polymorphic serializer was not found for missing class discriminator ('null')
The solution was to model it as a suspend method in Retrofit and wrap its return type in Response.
#GET("/observation")
suspend fun getObservation(
#Query("subject")
subject: String
): Response<Observation>
I have been given a json string that looks like the following one:
{
"dataflows": [
{
"name": "test",
"sources": [
{
"name": "person_inputs",
"path": "/data/input/events/person/*",
"format": "JSON"
}
],
"transformations": [
{
"name": "validation",
"type": "validate_fields",
"params": {
"input": "person_inputs",
"validations": [
{
"field": "office",
"validations": [
"notEmpty"
]
},
{
"field": "age",
"validations": [
"notNull"
]
}
]
}
},
{
"name": "ok_with_date",
"type": "add_fields",
"params": {
"input": "validation_ok",
"addFields": [
{
"name": "dt",
"function": "current_timestamp"
}
]
}
}
],
"sinks": [
{
"input": "ok_with_date",
"name": "raw-ok",
"paths": [
"/data/output/events/person"
],
"format": "JSON",
"saveMode": "OVERWRITE"
},
{
"input": "validation_ko",
"name": "raw-ko",
"paths": [
"/data/output/discards/person"
],
"format": "JSON",
"saveMode": "OVERWRITE"
}
And I have been asked to use it as some kind of recipe for an ETL pipeline, i.e., the data must be extracted from the "path" specifid in the "sources" key, the transformations to be carried out are specified within the "transformations" key and, finally, the transformed data must saved to one of the two specified "sink" keys.
I have decided to convert the json string into a scala map, as follows:
val json = Source.fromFile("path/to/json")
//parse
val parsedJson = jsonStrToMap(json.mkString)
implicit val formats = org.json4s.DefaultFormats
val parsedJson = parse(jsonStr).extract[Map[String, Any]]
so, with that, I get a structure like this one:
which is a map whose first value is a list of maps. I can evaluate parsedJson("dataflows") to get:
which is a list, as expected, but, then I cannot traverse such list, even though I need to in order to get to the sources, transformations and sinks. I have tried using the index of the listto, for example, get its first element, like this: parsedJson("dataflows")(0), but to no avail.
Can anyone please help me traverse this structure? Any help would be much appreciated.
Cheers,
When you evaluate parsedJson("dataflows") a Tuple2 is returned aka a Tuple which has two elements that are accessed with ._1 and ._2
So for dataflows(1)._1 the value returned is "sources" and dataflows(1)._2 is list of maps (List[Map[K,V]) which can be traversed like you would normally traverse elements of a List where each element is Map
Let's deconstruct this for example:
val dataFlowsZero = ("sources", List(Map(42 -> "foo"), Map(42 -> "bar")))
The first element in the Tuple
scala> dataFlowsZero._1
String = sources
The second element in the Tuple
scala> dataFlowsZero._2
List[Map[Int, String]] = List(Map(42 -> foo), Map(42 -> bar))`
Map the keys in each Map in List to a new List
scala> dataFlowsZero._2.map(m => m.keys)
List[Iterable[Int]] = List(Set(42), Set(42))
Map the values in each Map in the List to a new List
scala> dataFlowsZero._2.map(m => m.values)
List[Iterable[String]] = List(Iterable(foo), Iterable(bar))
The best solution is to convert the JSON to the full data structure that you have been provided rather than just Map[String, Any]. This makes it trivial to pick out the data that you want. For example,
val dataFlows = parse(jsonStr).extract[DataFlows]
case class DataFlows(dataflows: List[DataFlow])
case class DataFlow(name: String, sources: List[Source], transformations: List[Transformation], sinks: List[Sink])
case class Source(name: String, path: String, format: String)
case class Transformation(name: String, `type`: String, params: List[Param])
case class Param(input: String, validations: List[Validation])
case class Validation(field: String, validations: List[String])
case class Sink(input: String, name: String, paths: List[String], format: String, saveMode: String)
The idea is to make the JSON handler do most of the work to create a type-safe version of the original data.
I'm using thingspeak and I have successfully got thingspeak to fetch the json data using okhttp but I don't know how to parse it correctly using klaxon.
Here is the code
private fun funButton1() {
println("Attempting to get JSON data!")
val url = "https://api.thingspeak.com/channels/1029606/feeds.json?results=1"
val request = Request.Builder().url(url).build()
val client = OkHttpClient()
client.newCall(request).enqueue(object: Callback {
override fun onResponse(call: Call, response: Response) {
val body = response.body?.string()
println(body)
class feeds (val field1: String)
val result = Klaxon()
.parse<feeds>(body.toString())
textView.text = result
}
override fun onFailure(call: Call, e: IOException) {
println("Failed to execute request!")
}
})
This is the json data from the thingspeak
{
"channel": {
"id": 1029606,
"name": "LED ",
"description": "Acts as a medium for the phone and arduino \r\nRules : 1 = LED ON 0 = LED OFF ",
"latitude": "0.0",
"longitude": "0.0",
"field1": "LED STATUS",
"created_at": "2020-04-01T17:19:03Z",
"updated_at": "2020-04-01T17:20:39Z",
"last_entry_id": 25
},
"feeds": [
{
"created_at": "2020-05-11T02:58:07Z",
"entry_id": 25,
"field1": "1"
}
]
}
Im trying to get the value of field1 which the value is one but I don't know how I'm supposed to do that because im stupid. But I'm hoping that someone could show me how to use klaxon properly to get the json data.
For Klaxon, you'll need to create a class which represent the structure of your JSON.
f.e. if you get a JSON with:
{
"username": "admin",
"password": "admin"
}
you wanna make a class which looks like that:
class myClass(val username:String, val password:String)
Then, you can parse it like you are doing.
For your JSON, you'll need a bigger Class.
For the Sake of simplicity, I'll just make a class for feeds and a class for the channel:
class Feed(val created_at:String, val entry_id: Int, val field1:String)
class Channel(val id: Int, val name: String, val description: String, val latitude: String, val longitude:String, val field1: String, val created_at: String, val updated_at: String, val last_entry_id: Int)
Then you can use this class to parse your JSON:
class Thingspeak(val channel: Channel, val feeds: ArrayList<Feed>)
Please let me know if it worked for you!
I have a similar question with Deserialize json based on fields in .Net (C#), but do it in Scala.
I have an app which streams in 2 types of json objects (Account and User).
Account:
{
"data_type": "account",
"id": 1,
"type": "Trial",
"created_at": 1523982003,
}
User:
{
"data_type": "user",
"id": 1,
"account_id": 1,
"department": "Finance"
"created_at": 1523982122
}
I need to deserialize the two above json objects based on the field data_type in Scala with help of Circe library.
How can I do this?
This snippet works for me in Ammonite:
import $ivy.`io.circe:circe-core_2.12:0.9.3`, io.circe._
import $ivy.`io.circe:circe-generic_2.12:0.9.3`, io.circe.generic._
import $ivy.`io.circe:circe-generic-extras_2.12:0.9.3`, io.circe.generic.extras._
interp.load.plugin.ivy("org.scalamacros" % "paradise_2.12.4" % "2.1.1")
implicit val config: Configuration = Configuration.
default.
withSnakeCaseMemberNames.
withDiscriminator("data_type").
copy(transformConstructorNames = _.toLowerCase)
{
#ConfiguredJsonCodec
sealed trait InputEntity
object InputEntity {
#ConfiguredJsonCodec case class Account(id: Long, `type`: String, createdAt: Long) extends InputEntity
#ConfiguredJsonCodec case class User(id: Long, accountId: Long, department: String, createdAt: Long) extends InputEntity
}
}
import $ivy.`io.circe:circe-parser_2.12:0.9.3`, io.circe.parser._
val accountJson = """
{
"data_type": "account",
"id": 1,
"type": "Trial",
"created_at": 1523982003
}
"""
val account = decode[InputEntity](accountJson)
// account: Either[Error, InputEntity] = Right(Account(1L, "Trial", 1523982003L)
val userJson = """
{
"data_type": "user",
"id": 1,
"account_id": 1,
"department": "Finance",
"created_at": 1523982122
}
"""
val user = decode[InputEntity](userJson)
// user: Either[Error, InputEntity] = Right(User(1L, 1L, "Finance", 1523982122L))
(BTW: you had syntax errors in your JSON examples that would make the parser fail, so I fixed them in code above).
The most important here is
Configuration from io.circe.generic.extras._ that defines discriminator field,
keeping classes as sum type,
if you use annotations for generating codecs replace #JsonCodec with #ConfiguredJsonCodec.
Actually, you could also replace these Strings with enums, and read createdAt as LocalDateTime or similar, but that would be out of scope of this question.
The JSON object for which I'm trying to write a DecodeJson[T] contains an array of different "types" (meaning the JSON structure of its elements is varying). The only common feature is the type field which can be used to distinguish between the types. All other fields are different. Example:
{
...,
array: [
{ type: "a", a1: ..., a2: ...},
{ type: "b", b1: ...},
{ type: "c", c1: ..., c2: ..., c3: ...},
{ type: "a", a1: ..., a2: ...},
...
],
...
}
Using argonaut, is it possible to map the JSON array to a Scala Seq[Element] where Element is a supertype of suitable case classes of type ElementA, ElementB and so on?
I did the same thing with play-json and it was quite easy (basically a Reads[Element] that evaluates the type field and accordingly forwards to more specific Reads). However, I couldn't find a way to do this with argonaut.
edit: example
Scala types (I wish to use):
case class Container(id: Int, events: List[Event])
sealed trait Event
case class Birthday(name: String, age: Int) extends Event
case class Appointment(start: Long, participants: List[String]) extends Event
case class ... extends Event
JSON instance (not under my control):
{
"id":1674,
"events": {
"data": [
{
"type": "birthday",
"name": "Jones Clinton",
"age": 34
},
{
"type": "appointment",
"start": 1675156665555,
"participants": [
"John Doe",
"Jane Doe",
"Foo Bar"
]
}
]
}
}
You can create a small function to help you build a decoder that handles this format.
See below for an example.
import argonaut._, Argonaut._
def decodeByType[T](encoders: (String, DecodeJson[_ <: T])*) = {
val encMap = encoders.toMap
def decoder(h: CursorHistory, s: String) =
encMap.get(s).fold(DecodeResult.fail[DecodeJson[_ <: T]](s"Unknown type: $s", h))(d => DecodeResult.ok(d))
DecodeJson[T] { c: HCursor =>
val tf = c.downField("type")
for {
tv <- tf.as[String]
dec <- decoder(tf.history, tv)
data <- dec(c).map[T](identity)
} yield data
}
}
case class Container(id: Int, events: ContainerData)
case class ContainerData(data: List[Event])
sealed trait Event
case class Birthday(name: String, age: Int) extends Event
case class Appointment(start: Long, participants: List[String]) extends Event
implicit val eventDecoder: DecodeJson[Event] = decodeByType[Event](
"birthday" -> DecodeJson.derive[Birthday],
"appointment" -> DecodeJson.derive[Appointment]
)
implicit val containerDataDecoder: DecodeJson[ContainerData] = DecodeJson.derive[ContainerData]
implicit val containerDecoder: DecodeJson[Container] = DecodeJson.derive[Container]
val goodJsonStr =
"""
{
"id":1674,
"events": {
"data": [
{
"type": "birthday",
"name": "Jones Clinton",
"age": 34
},
{
"type": "appointment",
"start": 1675156665555,
"participants": [
"John Doe",
"Jane Doe",
"Foo Bar"
]
}
]
}
}
"""
def main(args: Array[String]) = {
println(goodJsonStr.decode[Container])
// \/-(Container(1674,ContainerData(List(Birthday(Jones Clinton,34), Appointment(1675156665555,List(John Doe, Jane Doe, Foo Bar))))))
}