JSON decode nested field as Map[String, String] in Scala using circe - json

A circe noob here. I am trying to decode a JSON string to case class in Scala using circe. I want one of the nested fields in the input JSON to be decoded as a Map[String, String] instead of creating a separate case class for it.
Sample code:
import io.circe.parser
import io.circe.generic.semiauto.deriveDecoder
case class Event(
action: String,
key: String,
attributes: Map[String, String],
session: String,
ts: Long
)
case class Parsed(
events: Seq[Event]
)
Decoder[Map[String, String]]
val jsonStr = """{
"events": [{
"ts": 1593474773,
"key": "abc",
"action": "hello",
"session": "def",
"attributes": {
"north_lat": -32.34375,
"south_lat": -33.75,
"west_long": -73.125,
"east_long": -70.3125
}
}]
}""".stripMargin
implicit val eventDecoder = deriveDecoder[Event]
implicit val payloadDecoder = deriveDecoder[Parsed]
val decodeResult = parser.decode[Parsed](jsonStr)
val res = decodeResult match {
case Right(staff) => staff
case Left(error) => error
}
I am ending up with a decoding error on attributes field as follows:
DecodingFailure(String, List(DownField(north_lat), DownField(attributes), DownArray, DownField(events)))
I found an interesting link here on how to decode JSON string to a map here: Convert Json to a Map[String, String]
But I'm having little luck as to how to go about it.
If someone can point me in the right direction or help me out on this that will be awesome.

Let's parse the error :
DecodingFailure(String, List(DownField(geotile_north_lat), DownField(attributes), DownArray, DownField(events)))
It means we should look in "events" for an array named "attributes", and in this a field named "geotile_north_lat". This final error is that this field couldn't be read as a String. And indeed, in the payload you provide, this field is not a String, it's a Double.
So your problem has nothing to do with Map decoding. Just use a Map[String, Double] and it should work.

So you can do something like this:
final case class Attribute(
key: String,
value: String
)
object Attribute {
implicit val attributesDecoder: Decoder[List[Attribute]] =
Decoder.instance { cursor =>
cursor
.value
.asObject
.toRight(
left = DecodingFailure(
message = "The attributes field was not an object",
ops = cursor.history
)
).map { obj =>
obj.toList.map {
case (key, value) =>
Attribute(key, value.toString)
}
}
}
}
final case class Event(
action: String,
key: String,
attributes: List[Attribute],
session: String,
ts: Long
)
object Event {
implicit val eventDecoder: Decoder[Event] = deriveDecoder
}
Which you can use like this:
val result = for {
json <- parser.parse(jsonStr).left.map(_.toString)
obj <- json.asObject.toRight(left = "The input json was not an object")
eventsRaw <- obj("events").toRight(left = "The input json did not have the events field")
events <- eventsRaw.as[List[Event]].left.map(_.toString)
} yield events
// result: Either[String, List[Event]] = Right(
// List(Event("hello", "abc", List(Attribute("north_lat", "-32.34375"), Attribute("south_lat", "-33.75"), Attribute("west_long", "-73.125"), Attribute("east_long", "-70.3125")), "def", 1593474773L))
// )
You can customize the Attribute class and its Decoder, so their values are Doubles or Jsons.

Related

How to create a List from a HTTP response string based on certain conditions?

I am trying to parse the result of a HTTPS request in Scala.
The HTTPS response is a String as follows:
{
"rows":
[
{
"log_forwarding_ip":"",
"device_name":"AD1",
"id":"51",
"mgmt_ip_addr":"192.168.25.150",
"log_forwarding":"1",
"isActive":"0"
},
{
"log_forwarding_ip":"192.168.1.1",
"device_name":"WIN-SRV2019",
"id":"50",
"mgmt_ip_addr":"192.168.25.151",
"log_forwarding":"1",
"isActive":"1"
},
{
"log_forwarding_ip":"129.168.1.2",
"device_name":"PA",
"id":"3",
"mgmt_ip_addr":"192.168.1.161",
"log_forwarding":"1",
"isActive":"1"
}
],
"status":1
}
I have to create a List of all id's where isActive and log_forwarding are both equal to 1.
So far what I have done is:
object syncTables {
def main(args: Array[String]): Unit = {
case class deviceInfo(log_forwarding_ip: String, device_name: String, id: String,
mgmt_ip_addr: String, log_forwarding: String, isActive: String)
try {
val r = requests.get("https://192.168.1.253/api/device/deviceinfo.php", verifySslCerts = false)
if (r.statusCode == 200) {
val x = r.text
println(x)
} else {
println("Error in API call: "+r.statusCode)
}
}
}
}
Now I'm really confused what to do next to achieve my result. I'm totally new to JSON, that's why I don't know which JSON library I should use.
I tried using Play Framework but it seems all pretty complicated to me.
Does Scala offer something like Python's json module where this task can be easily done by using dictionaries and lists.
I'm using Scala 2.11.12 and com.lihaoyi.requests.Any kind of help will be highly appreciated.Thanks in advance.
Use json4s to parse the JSON string. Let's call your JSON input string json, then you can do something like this
import org.json4s._
import org.json4s.jackson.JsonMethods._
case class DeviceInfo(log_forwarding_ip: String, device_name: String, id: String,
mgmt_ip_addr: String, log_forwarding: String, isActive: String)
implicit val formats: Formats = DefaultFormats // Brings in default date formats etc.
val parsedJson = parse(json)
val deviceInfos = parsedJson match {
case JObject(head :: _) =>
head._2
.extract[List[DeviceInfo]]
.filter(_.isActive == 1 && _.log_forwarding == 1)
}
This will output
val res0: List[DeviceInfo] = List(DeviceInfo("192.168.1.1","WIN-SRV2019","50","192.168.25.151","1","1"),DeviceInfo("129.168.1.2","PA","3","192.168.1.161","1","1"))

spray json in scala: deserializing json with unknown fields without losing them

I'm looking for a solution to this but for json spray and my searches and attempts to get this working with json spray have failed thus far.
If I have the following json:
{
"personalData": {
"person": {
"first": "first_name",
"last": "last_name"
},
"contact": {
"phone": "1111111",
"address": {
"line": "123 Main St",
"city": "New York"
}
},
"xx": "yy", // unknown in advanced
"zz": { // unknown in advanced
"aa": "aa",
"bb": "bb",
"cc": {
"dd": "dd",
"ee": "ee"
}
}
}
}
We know for sure that the json will contain person and contact but we don't know what other fields may be created upstream from us that we don't care about/use.
I want to serialize this JSON into a case class containing person and contact but on the other hand, I don't want to lose the other fields (save them in a map so the class will be deserialized to the same json as received).
This is how far I've made it:
case class Address(
line: String,
city: String,
postalCode: Option[String]
)
case class Contact(
phone: String,
address: Address
)
case class Person(
first: String,
last: String
)
case class PersonalData(
person: Person,
contact: Contact,
extra: Map[String, JsValue]
)
implicit val personFormat = jsonFormat2(Person)
implicit val addressFormat = jsonFormat3(Address)
implicit val contactFormat = jsonFormat2(Contact)
implicit val personalDataFormat = new RootJsonFormat[PersonalData] {
def write(personalData: PersonalData): JsValue = {
JsObject(
"person" -> personalData.person.toJson,
"contact" -> personalData.contact.toJson,
// NOT SURE HOW TO REPRESENT extra input
)
}
def read(value: JsValue): CAERequestBEP = ???
}
Can someone help me do this with spray.json instead of play? I've spent such a long time trying to do this and can't seem to make it work.
In order to do that, you need to write your own formatter for PersonalDataFormat:
case class Person(first: String, last: String)
case class Address(line: String, city: String)
case class Contact(phone: String, address: Address)
case class PersonalData(person: Person, contact: Contact, extra: Map[String, JsValue])
case class Entity(personalData: PersonalData)
implicit val personFormat = jsonFormat2(Person)
implicit val addressFormat = jsonFormat2(Address)
implicit val contactFormat = jsonFormat2(Contact)
implicit object PersonalDataFormat extends RootJsonFormat[PersonalData] {
override def read(json: JsValue): PersonalData = {
val fields = json.asJsObject.fields
val person = fields.get("person").map(_.convertTo[Person]).getOrElse(???) // Do error handling instead of ???
val contact = fields.get("contact").map(_.convertTo[Contact]).getOrElse(???) // Do error handling instead of ???
PersonalData(person, contact, fields - "person" - "contact")
}
override def write(personalData: PersonalData): JsValue = {
JsObject(personalData.extra ++ ("person" -> personalData.person.toJson, "contact" -> personalData.contact.toJson))
}
}
implicit val entityFormat = jsonFormat1(Entity)
val jsonResult = jsonString.parseJson.convertTo[Entity]
The result is:
Entity(PersonalData(Person(first_name,last_name),Contact(1111111,Address(123 Main St,New York)),Map(xx -> "yy", zz -> {"aa":"aa","bb":"bb","cc":{}})))
(Assuming the json is not exactly the json above, but a valid similar one)
Code run in Scastie

Deserializing a json object property to a String using kotlinx.serialization

Given json as follows where the structure of the payload object will vary:
{
"id": 1,
"displayName": "Success",
"payload": {
"someProperty": "example",
"someOtherProperty": {
"someNestedProperty": "example"
}
}
}
...using kotlinx.serialization how can I deserialize this into the following data class, where the value of payload should be the raw json string of the payload object.
#Serializable
data class Stub(
val id: Int,
val displayName: String,
val payload: String
)
Struggled to find a way of doing this with Serializers, but it was simple enough to implement manually using JsonElement.
val jsonObject = Json.parseToJsonElement(jsonString).jsonObject
val stub = Stub(
jsonObject["id"]!!.jsonPrimitive.int,
jsonObject["displayName"]!!.jsonPrimitive.content,
jsonObject["payload"]!!.toString()
)
There is a way to handle using JSONTransformingSerializer. It allows you to transform the json prior to deserialization. In this case from a jsonElement into a jsonPrimitive (of type String).
First create a transformer as follows:
object JsonAsStringSerializer: JsonTransformingSerializer<String>(tSerializer = String.serializer()) {
override fun transformDeserialize(element: JsonElement): JsonElement {
return JsonPrimitive(value = element.toString())
}
}
Now apply this transfer to the specific element in your data class by adding...
#Serializable(with = JsonAsStringSerializer::class)
just above the property you want to transform. Like this...
#Serializable
data class Stub(
val id: Int,
val displayName: String,
#Serializable(with = JsonAsStringSerializer::class)
val payload: String
)
The value of payload will be a string:
"{'someProperty': 'example','someOtherProperty': {'someNestedProperty':'example'}"
If you are later trying to deserialize this into different models depending on the structure, check out the JsonContentPolymorphicSerializer feature.

Scala Circe. Encoder type Any

I am trying encode class ResponseResult to json
case class ResponseResult (var Code : Int,
var Message: String,
var Data: Any )
var json = ResponseResult(1, "2", List(3,4,5)).asJson
I get an error could not find implicit value for parameter encoder: io.circe.Encoder[ResponseResult]
Then I create encoder
object ResponseResult {
implicit val encodeResult: Encoder[ResponseResult] =
Encoder.forProduct3[ResponseResult, Int, String, Any]("Code", "Message", "Data") {
case ResponseResult(c, m, d) => (c, m, d.toString)
}
}
I get an error could not find implicit value for parameter encoder: Encoder[Any]
This is my full code
import io.circe.Encoder
import io.circe.generic.auto._
import io.circe.syntax._
object a {
case class ResponseResult (var Code : Int,
var Message: String,
var Data: Any )
object ResponseResult {
implicit val encodeResult: Encoder[ResponseResult] =
Encoder .forProduct3[ResponseResult, Int, String, Any]("Code", "Message", "Data") {
case ResponseResult(c, m, d) => (c, m, d.toString)
}
}
def main(args: Array[String]): Unit = {
var json = ResponseResult(1, "2", List(3,4,5)).asJson
}
}
Anyone can show me How to encode class ResponseResult.
Thank you
Well, you cannot have Codec for Any type, simple as that.
If you want to pass some value there, serialize it and be able to deserialize it later, you have to assume some type. You can defer this assumption by using type parameter:
case class ResponseResult[Data](Code : Int,
Message: String,
Data: Data)
and requiring existence of encoder for this type parameter:
object ResponseResult {
implicit def encodeResult[Data: Encoder]: Encoder[ResponseResult[Data]] =
Encoder.forProduct3[ResponseResult[Data], Int, String, Data]("Code", "Message", "Data") {
case ResponseResult(c, m, d) => (c, m, d)
}
}
then you will be able to serialize data like you want
// inferred to ResponseResult[List[Int]]
val response = ResponseResult(1, "2", List(3,4,5))
// and Circe can generate codec for List[Int] automatically
// so it can now generate codec for ResponseResult[List[Int]]
val json = response.asJson
Actually, you can use Json type for Data.
case class ResponseResult (var Code : Int,
var Message: String,
var Data: io.circe.Json )
so you can pass anything to Data, null too.

In Json4s why does an integer field in a JSON object get automatically converted to a String?

If I have a JSON object like:
{
"test": 3
}
Then I would expect that extracting the "test" field as a String would fail because the types don't line up:
import org.json4s._
import org.json4s.jackson.JsonMethods
import org.json4s.JsonAST.JValue
def getVal[T: Manifest](json: JValue, fieldName: String): Option[T] = {
val field = json findField {
case JField(name, _) if name == fieldName => true
case _ => false
}
field.map {
case (_, value) => value.extract[T]
}
}
val json = JsonMethods.parse("""{"test":3}""")
val value: Option[String] = getVal[String](json, "test") // Was Some(3) but expected None
Is this automatic conversion from a JSON numeric to a String expected in Json4s? If so, are there any workarounds for this where the extracted field has to be of the same type that is specified in the type parameter to the extract method?
This is the default nature of most if not all of the parsers. If you request a value of type T and if the value can be safely cast to that specific type then the library would cast it for you. for instance take a look at the typesafe config with the similar nature of casting Numeric field to String.
import com.typesafe.config._
val config = ConfigFactory parseString """{ test = 3 }"""
val res1 = config.getString("test")
res1: String = 3
if you wanted not to automatically cast Integer/Boolean to String you could do something like this manually checking for Int/Boolean types as shown below.
if(Try(value.extract[Int]).isFailure || Try(value.extract[Boolean]).isFailure) {
throw RuntimeException(s"not a String field. try Int or Boolean")
} else {
value.extract[T]
}
One simple workaround is to create a custom serializer for cases where you want "strict" behavior. For example:
import org.json4s._
val stringSerializer = new CustomSerializer[String](_ => (
{
case JString(s) => s
case JNull => null
case x => throw new MappingException("Can't convert %s to String." format x)
},
{
case s: String => JString(s)
}
))
Adding this serializer to your implicit formats ensures the strict behavior:
implicit val formats = DefaultFormats + stringSerializer
val js = JInt(123)
val str = js.extract[String] // throws MappingException