I am trying to deserialise a json string to an object using jsons but having problems with nested objects, but can't work out the syntax.
As an example the following code attempts to define the data structure as a series of dataclasses but fails to deserialise the nested objects C and D ? The syntax is clearly wrong, but its not clear to me how it should structured
import jsons
from dataclasses import dataclass
#dataclass
class D:
E: str
class C:
id: int
name:str
#dataclass
class test:
A: str
B: int
C: C()
D: D()
jsonString = {"A":"a","B":1,"C":[{"id":1,"name":"one"},{"id":2,"name":"two"}],"D":[{"E":"e"}]}
instance = jsons.load(jsonString, test)
Can anyone indicate the correct way to deserialise the objects from json ?
There are two relatively simple problems with your attempt:
You forgot to decorate C with #dataclass.
Test.C and Test.D aren't defined with types, but with instances of the types. (Further, you want both fields to be lists of the given type, not single instances of each.)
Given the code
import jsons
from dataclasses import dataclass
from typing import List
#dataclass
class D:
E: str
#dataclass # Problem 1 fixed
class C:
id: int
name: str
#dataclass
class Test:
A: str
B: int
C: List[C] # Problem 2 fixed; List[C] not C() or even C
D: List[D] # Problem 2 fixed; List[D], not D() or even D
Then
>>> obj = {"A":"a", "B":1, "C": [{"id": 1,"name": "one"}, {"id": 2, "name": "two"}], "D":[{"E": "e"}]}
>>> jsons.load(obj, Test)
test(A='a', B=1, C=[C(id=1, name='one'), C(id=2, name='two')], D=[D(E='e')])
from dataclasses import dataclass
from typing import List
from validated_dc import ValidatedDC
#dataclass
class D(ValidatedDC):
E: str
#dataclass
class C(ValidatedDC):
id: int
name: str
#dataclass
class Test(ValidatedDC):
A: str
B: int
C: List[C]
D: List[D]
jsonString = {
"A": "a",
"B": 1,
"C": [{"id": 1, "name": "one"}, {"id": 2, "name": "two"}],
"D": [{"E": "e"}]
}
instance = Test(**jsonString)
assert instance.C == [C(id=1, name='one'), C(id=2, name='two')]
assert instance.C[0].id == 1
assert instance.C[1].name == 'two'
assert instance.D == [D(E='e')]
assert instance.D[0].E == 'e'
ValidatedDC: https://github.com/EvgeniyBurdin/validated_dc
You can do something like this:
from collections import namedtuple
# First parameter is the class/tuple name, second parameter
# is a space delimited string of varaibles.
# Note that the variable names should match the keys from
# your dictionary of arguments unless only one argument is given.
A = namedtuple("A", "a_val") # Here the argument `a_val` can be called something else
B = namedtuple("B", "num")
C = namedtuple("C", "id name")
D = namedtuple("D", "E") # This must be `E` since E is the key in the dictionary.
# If you dont want immutable objects to can use full classes
# instead of namedtuples
# A dictionary which matches the name of an object seen in a payload
# to the object we want to create for that name.
object_options = {
"A": A,
"B": B,
"C": C,
"D": D
}
my_objects = [] # This is the list of object we get from the payload
jsonString = {"A":"a","B":1,"C":[{"id":1,"name":"one"},{"id":2,"name":"two"}],"D":[{"E":"e"}]}
for key, val in jsonString.items():
if key in object_options: # If this is a valid object
if isinstance(val, list): # If it is a list of this object
for v in val: # Then we need to add each object in the list
my_objects.append(object_options[key](**v))
elif isinstance(val, dict): # If the object requires a dict then pass the whole dict as arugments
my_objects.append(object_options[key](**val))
else: # Else just add this object with a singular argument.
my_objects.append(object_options[key](val))
print(my_objects)
Output:
[A(a_val='a'), B(num=1), C(id=1, name='one'), C(id=2, name='two'), D(E='e')]
I've finally managed to get this to work by removing the dataClass definition and expanding the class definitions old school.... code as follows...
import jsons
class D:
def __init__(self, E = ""):
self.E = E
class C:
def __init__(self, id = 0, name=""):
self.id = id
self.name = name
class test:
def __init__(self, A = "", B = 0, C = C(), D = D()):
self.A = A
self.B = B
self.C = C
self.D = D
jsonString = {"A":"a","B":1,"C":[{"id":1,"name":"one"},{"id":2,"name":"two"}],"D":[{"E":"e"}]}
instance = jsons.load(jsonString, test)
It now works but is not as clean as with a dataClass. Grateful if anyone can indicate how the original post can be constructed with the dataClass definition.
Related
Is there a way to decode arbitrary json (e.g: We don't know the keys at compile time)?
For example, I need to parse the following json:
{
"Foo": [
"Value 1",
"Value 2"
],
"Bar": [
"Bar Value 1"
],
"Baz": []
}
where the names and number of keys are not known at compile time and may change per GET request. The goal is basically to decode this into a Map String (Array String) type
Is there a way to do this using purescript-argonaut?
You can totally write your own by first parsing the string into Json via jsonParser, and then examining the resulting data structure with the various combinators provided by Argonaut.
But the quickest and simplest way, I think, is to parse it into Foreign.Object (Array String) first, and then convert to whatever your need, like Map String (Array String):
import Data.Argonaut (decodeJson, jsonParser)
import Data.Either (Either)
import Data.Map as Map
import Foreign.Object as F
decodeAsMap :: String -> Either _ (Map.Map String (Array String))
decodeAsMap str = do
json <- jsonParser str
obj <- decodeJson json
pure $ Map.fromFoldable $ (F.toUnfoldable obj :: Array _)
The Map instance of EncodeJSON will generate an array of tuple, you can manually construct a Map and see the encoded json.
let v = Map.fromFoldable [ Tuple "Foo" ["Value1", "Value2"] ]
traceM $ encodeJson v
Output should be [ [ 'Foo', [ 'Value1', 'Value2' ] ] ].
To do the reverse, you need to transform you object to an array of tuple, Object.entries can help you.
An example
// Main.js
var obj = {
foo: ["a", "b"],
bar: ["c", "d"]
};
exports.tuples = Object.entries(obj);
exports.jsonString = JSON.stringify(exports.tuples);
-- Main.purs
module Main where
import Prelude
import Data.Argonaut.Core (Json)
import Data.Argonaut.Decode (decodeJson)
import Data.Argonaut.Parser (jsonParser)
import Data.Either (Either)
import Data.Map (Map)
import Debug.Trace (traceM)
import Effect (Effect)
import Effect.Console (log)
foreign import tuples :: Json
foreign import jsonString :: String
main :: Effect Unit
main = do
let
a = (decodeJson tuples) :: Either String (Map String (Array String))
b = (decodeJson =<< jsonParser jsonString) :: Either String (Map String (Array String))
traceM a
traceM b
traceM $ a == b
I'm new in Scala and do not know how to deal with this Json with json4s:
After parsing the json and extracting the following json through:
val data = json \\ "someKey"
I have a Json like this:
[{"Id":14706061,
"Rcvr":1,
"HasSig":true,
"Sig":80},
{"Id":3425490,
"Rcvr":1,
"HasSig":false,
"Sig": 80}]
Printing it to the console, it returns:
JArray(List(JObject(List((Id,JInt(14706061)), (Rcvr,JInt(1)), (HasSig,JBool(true)), (Sig,JInt(80), Id,JInt(3425490)), (Rcvr,JInt(1)), (HasSig,JBool(false)), (Sig,JInt(80) ))
So, after that I used:
println("show values: " + data.values)
And had:
List(Map(Id -> 14706061, Rcvr -> 1, HasSig -> true, Sig -> 80), Map(Id -> 3425490, Rcvr -> 1, HasSig -> false, Sig -> 80))
But I don't know how to extract each Map from each position of the List.
I also tried to extract to a case class but I had 0 entries:
case class Example (Id: BigInt, Rcvr: Int, HasSig: Boolean, Sig: Int)
case class ExampleList (examples: List[Example])
implicit val formats = DefaultFormats.strict
val dataList = data.extract[ExampleList]
Thanks in advance for your help
PD. If I assign:
val dataList = data.values
The type of dataList (with getClass) is: class scala.collection.immutable.$colon$colon
PD2. SOLUTION
After the:
val data = json \\ "someKey"
I put:
val dataList = data.extract[JArray]
val examples = dataList.values
It returns an iterable Array with its Maps iterable, so fixed.
Checked with:
println("number of elements: " + examples.length)
and
println("show each item: " + examples.foreach(println))
Thanks for taking your time in reading.
If you want to extract into a Case Class instead of a Map, the correct type for extraction is List[Example], instead of ExampleList.
ExampleList has an attribute examples, your json doesn't. That is why you got an empty list.
import org.json4s.native.JsonMethods._
import org.json4s._
implicit val formats = DefaultFormats
val str = """[{"Id":14706061,
"Rcvr":1,
"HasSig":true,
"Sig":80},
{"Id":3425490,
"Rcvr":1,
"HasSig":false,
"Sig": 80}]"""
case class Example (Id: BigInt, Rcvr: Int, HasSig: Boolean, Sig: Int)
val json = parse(str)
val examples = json.extract[List[Example]]
Hope the below code helps.
enter code here
val iList = List(Map("Id" -> 14706061, "Rcvr" -> 1, "HasSig" -> "true", "Sig" -> 80),
Map("Id" -> 3425490, "Rcvr" -> 1, "HasSig" -> false, "Sig" -> 80))
for(i <-0 until iList.size){val lMap = iList(i)println("Id: " + lMap("Id"))}
This question has come up a few times recently, so I'm FAQ-ing it here. Suppose I've got some case classes like this:
import io.circe._, io.circe.generic.semiauto._
object model {
case class A(a: String)
case class B(a: String, i: Int)
case class C(i: Int, b: Boolean)
implicit val encodeA: Encoder[A] = deriveEncoder
implicit val encodeB: Encoder[B] = deriveEncoder
implicit val encodeC: Encoder[C] = deriveEncoder
implicit val decodeA: Decoder[A] = deriveDecoder
implicit val decodeB: Decoder[B] = deriveDecoder
implicit val decodeC: Decoder[C] = deriveDecoder
}
And I want to encode a value that could be any one of these as JSON using circe and Shapeless coproducts.
import io.circe.shapes._, io.circe.syntax._
import shapeless._
import model._
type ABC = A :+: B :+: C :+: CNil
val c: ABC = Coproduct[ABC](C(123, false))
This looks fine at first:
scala> c.asJson
res0: io.circe.Json =
{
"i" : 123,
"b" : false
}
But the problem is that I can never decode a coproduct containing a B element, since any valid JSON document that could be decoded as B can also be decoded as A, and the coproduct decoders provided by circe-shapes try the elements in the order they appear in the coproduct.
scala> val b: ABC = Coproduct[ABC](B("xyz", 123))
b: ABC = Inr(Inl(B(xyz,123)))
scala> val json = b.asJson
json: io.circe.Json =
{
"a" : "xyz",
"i" : 123
}
scala> io.circe.jawn.decode[ABC](json.noSpaces)
res1: Either[io.circe.Error,ABC] = Right(Inl(A(xyz)))
How can I disambiguate the elements of my coproduct in my encoding?
The circe-shapes module encodes ordinary hlists and coproducts without labels, as seen above: a coproduct as the bare JSON representation of the element, and an hlist ends up as just a JSON array (the same as the default tuple encoding):
scala> ("xyz" :: List(1, 2, 3) :: false :: HNil).asJson.noSpaces
res2: String = ["xyz",[1,2,3],false]
In the case of hlists there's no danger of ambiguity because of overlapping names, but for coproducts there is. In both cases, though, you can add labels to the JSON representation using Shapeless's labeling mechanism, which tags values with a type-level symbol.
An hlist with labels is called a "record", and a coproduct with labels is a "union". Shapeless provides special syntax and operations for both of these, and circe-shapes treats both differently from unlabeled hlists or coproducts. For example (assuming the definitions and imports above):
scala> import shapeless.union._, shapeless.syntax.singleton._
import shapeless.union._
import shapeless.syntax.singleton._
scala> type ABCL = Union.`'A -> A, 'B -> B, 'C -> C`.T
defined type alias ABCL
scala> val bL: ABCL = Coproduct[ABCL]('B ->> B("xyz", 123))
bL: ABCL = Inr(Inl(B(xyz,123)))
scala> val jsonL = bL.asJson
jsonL: io.circe.Json =
{
"B" : {
"a" : "xyz",
"i" : 123
}
}
scala> io.circe.jawn.decode[ABCL](jsonL.noSpaces)
res3: Either[io.circe.Error,ABCL] = Right(Inr(Inl(B(xyz,123))))
The encoding for records similarly includes the member names:
scala> ('a ->> "xyz" :: 'b ->> List(1) :: 'c ->> false :: HNil).asJson.noSpaces
res4: String = {"c":false,"b":[1],"a":"xyz"}
In general if you want your hlist and coproduct encodings to include labels (and look like the encodings you'd get from circe-generic for case classes and sealed trait hierarchies), you can use records or coproducts to tell circe-shapes to do this with a minimal amount of extra syntactic and runtime overhead.
I am trying to follow the tutorial https://www.jamesward.com/2012/02/21/play-framework-2-with-scala-anorm-json-coffeescript-jquery-heroku but of course play-scala has changed since the tutorial (as seems to be the case with every tutorial I find). I am using 2.4.3 This requires I actually learn how things work, not necessarily a bad thing.
One thing that is giving me trouble is the getOrElse method.
Here is my Bar.scala model
package models
import play.api.db._
import play.api.Play.current
import anorm._
import anorm.SqlParser._
case class Bar(id: Option[Long], name: String)
object Bar {
val simple = {
get[Option[Long]]("id") ~
get[String]("name") map {
case id~name => Bar(id, name)
}
}
def findAll(): Seq[Bar] = {
DB.withConnection { implicit connection =>
SQL("select * from bar").as(Bar.simple *)
}
}
def create(bar: Bar): Unit = {
DB.withConnection { implicit connection =>
SQL("insert into bar(name) values ({name})").on(
'name -> bar.name
).executeUpdate()
}
}
}
and my BarFormat.scala Json formatter
package models
import play.api.libs.json._
import anorm._
package object Implicits {
implicit object BarFormat extends Format[Bar] {
def reads(json: JsValue):JsResult[Bar] = JsSuccess(Bar(
Option((json \ "id").as[Long]),
(json \ "name").as[String]
))
def writes(bar: Bar) = JsObject(Seq(
"id" -> JsNumber(bar.id.getOrElse(0L)),
"name" -> JsString(bar.name)
))
}
}
and for completeness my Application.scala controller:
package controllers
import play.api.mvc._
import play.api.data._
import play.api.data.Forms._
import javax.inject.Inject
import javax.inject._
import play.api.i18n.{ I18nSupport, MessagesApi, Messages, Lang }
import play.api.libs.json._
import views._
import models.Bar
import models.Implicits._
class Application #Inject()(val messagesApi: MessagesApi) extends Controller with I18nSupport {
val barForm = Form(
single("name" -> nonEmptyText)
)
def index = Action {
Ok(views.html.index(barForm))
}
def addBar() = Action { implicit request =>
barForm.bindFromRequest.fold(
errors => BadRequest,
{
case (name) =>
Bar.create(Bar(None, name))
Redirect(routes.Application.index())
}
)
}
def listBars() = Action { implicit request =>
val bars = Bar.findAll()
val json = Json.toJson(bars)
Ok(json).as("application/json")
}
and routes
# Routes
# This file defines all application routes (Higher priority routes first)
# ~~~~
# Home page
POST /addBar controllers.Application.addBar
GET / controllers.Application.index
GET /listBars controllers.Application.listBars
# Map static resources from the /public folder to the /assets URL path
GET /assets/*file controllers.Assets.versioned(path="/public", file: Asset)
When I try to run my project I get the following error:
now bar.id is defined as an Option[Long] so bar.id.getOrElse(0L) should return a Long as far as I can tell, but it is clearly returning an Any. Can anyone help me understand why?
Thank You!
That's the way type inference works in Scala...
First of all there is an implicit conversion from Int to BigDecimal:
scala> (1 : Int) : BigDecimal
res0: BigDecimal = 1
That conversion allows for Int to be converted before the option is constructed:
scala> Some(1) : Option[BigDecimal]
res1: Option[BigDecimal] = Some(1)
If we try getOrElse on its own where the type can get fixed we get expected type Int:
scala> Some(1).getOrElse(2)
res2: Int = 1
However, this does not work (the problem you have):
scala> Some(1).getOrElse(2) : BigDecimal
<console>:11: error: type mismatch;
found : Any
required: BigDecimal
Some(1).getOrElse(2) : BigDecimal
^
Scala's implicit conversions kick in last, after type inference is performed. That makes sense, because if you don't know the type how would you know what conversions need to be applied. Scala can see that BigDecimal is expected, but it has an Int result based on the type of the Option it has. So it tries to widen the type, can't find anything that matches BigDecimal in Int's type hierarchy and fails with the error.
This works, however because the type is fixed in the variable declaration:
scala> val v = Some(1).getOrElse(2)
v: Int = 1
scala> v: BigDecimal
res4: BigDecimal = 1
So we need to help the compiler somehow - any type annotation or explicit conversion would work. Pick any one you like:
scala> (Some(1).getOrElse(2) : Int) : BigDecimal
res5: BigDecimal = 1
scala> Some(1).getOrElse[Int](2) : BigDecimal
res6: BigDecimal = 1
scala> BigDecimal(Some(1).getOrElse(2))
res7: scala.math.BigDecimal = 1
Here is the signature for Option.getOrElse method:
getOrElse[B >: A](default: ⇒ B): B
The term B >: A expresses that the type parameter B or the abstract type B refer to a supertype of type A, in this case, Any being the supertype of Long:
val l: Long = 10
val a: Any = l
So, we can do something very similar here with getOrElse:
val some: Option[Long] = Some(1)
val value: Any = option.getOrElse("potatos")
val none: Option[Long] = None
val elseValue: Any = none.getOrElse("potatos")
Which brings us to your scenario: the returned type from getOrElse will be a Any and not a BigDecimal, so you will need another way to handle this situation, like using fold, per instance:
def writes(bar: Bar) = {
val defaultValue = BigDecimal(0)
JsObject(Seq(
"id" -> JsNumber(bar.id.fold(defaultValue)(BigDecimal(_))),
"name" -> JsString(bar.name)
))
}
Some other discussions that can help you:
Why is Some(1).getOrElse(Some(1)) not of type Option[Int]?
Option getOrElse type mismatch error
I need for a frontend jquery-component a specific json object like this (ajax response):
[
{"division":"IT", "contacts":[
{“firstname”:”Carl”, “surname”:”Smith”, “empID”:1},
{“firstname”:”Henry”, “surname”:”Miller”, “empID”:2}
]},
{"division":"Sales", "contacts":[
{“firstname”:”Nancy”, “surname”:”McDonald”, “empID”:3},
{“firstname”:”Susan”, “surname”:”McBright”, “empID”:4}
]}
]
In Backend the data is read via anorm (MySQL) and transformed in following object:
List(Map("division" -> "IT", "contacts" -> List(c3,c4)), Map("division" -> "Sales", "contacts" -> List(c3,c4)))
Then I try to serialize the Object to JSon but without success (including implicit Writes converters). Below I made a simplified test case Idea-Worksheet in the same manner:
import play.api.libs.json.{JsValue, Json, Writes}
case class Contact(firstname: String, surname: String, empID: Option[Int])
case class ContactDivisionList(division: String, contacts: Seq[Contact])
implicit val ctWrites = new Writes[Contact] {
def writes(ct: Contact) = Json.obj(
"firstname" -> ct.firstname,
"surname" -> ct.surname,
"empid" -> ct.empID
)
}
implicit val sdlWrites = new Writes[ContactDivisionList] {
def writes(dlist: ContactDivisionList) = Json.obj(
"division" -> dlist.division,
"contacts" -> Json.toJson(dlist.contacts)
)
}
/*
Example
*/
val c1 = Contact("Carl","Smith",Option(1))
val c2 = Contact("Henry","Miller",Option(2))
val c3 = Contact("Nancy","McDonald",Option(3))
val c4 = Contact("Susan","McBright",Option(4))
//Test case 1 ->OK
Json.toJson(List(c1,c2,c3,c4))
//Test case 2 ->OK
val c_comp1=List(Map("contacts" -> List(c1,c2)),Map("contacts" -> List(c3,c4)))
//RESULT --> c_comp1: List[scala.collection.immutable.Map[String,List[Contact]]] = List(Map(contacts -> List(Contact(Carl,Smith,Some(1)), Contact(Henry,Miller,Some(2)))), Map(contacts -> List(Contact(Nancy,McDonald,Some(3)), Contact(Susan,McBright,Some(4)))))
Json.toJson(c_comp1)
//res1: play.api.libs.json.JsValue = [{"contacts": [{"firstname":"Carl","surname":"Smith","empid":1},{"firstname":"Henry","surname":"Miller","empid":2}]},{"contacts":[{"firstname":"Nancy","surname":"McDonald","empid":3},{"firstname":"Susan","surname":"McBright","empid":4}]}]
//Test case 3 ->Fail!!!
val c_comp2 = List(Map("division" -> "IT", "contacts" -> List(c1,c2)),Map("division" -> "Sales", "contacts" -> List(c3,c4)))
//sdlWrites: play.api.libs.json.Writes[ContactDivisionList]{def writes(dlist: ContactDivisionList): play.api.libs.json.JsObject} = $anon$2#3738baec
Json.toJson(c_comp2)
//!!!!!Error messages
/*Error:(39, 13) No Json serializer found for type List[scala.collection.immutable.Map[String,java.io.Serializable]]. Try to implement an implicit Writes or Format for this type.
Json.toJson(c_comp2)
^
Error:(39, 13) not enough arguments for method toJson: (implicit tjs: play.api.libs.json.Writes[List[scala.collection.immutable.Map[String,java.io.Serializable]] ])play.api.libs.json.JsValue.
Unspecified value parameter tjs.
Json.toJson(c_comp2)
^
*/
At the end of the script you can see "Test case 3" that got an error when i execute Json.toJson(c_comp2) -->"No Json serializer found for type..". I try a lot of things but i don't get it right. The only difference to successful "Test case 2" is that i extend the Map with a String-Tuppel.
I hope sombody can help me with that issue, Thx
Best regards
Karsten
Your problem is that the Map you have there is mapping String to the least upper bound of String (your division name) and List[Contact], which happens to be java.io.Serializable.
scala> case class Contact(firstname: String, surname: String, empID: Option[Int])
defined class Contact
scala> case class ContactDivisionList(division: String, contacts: Seq[Contact])
defined class ContactDivisionList
scala> val c1 = Contact("Carl","Smith",Option(1))
c1: Contact = Contact(Carl,Smith,Some(1))
scala> val c2 = Contact("Henry","Miller",Option(2))
c2: Contact = Contact(Henry,Miller,Some(2))
scala> val c3 = Contact("Nancy","McDonald",Option(3))
c3: Contact = Contact(Nancy,McDonald,Some(3))
scala> val c4 = Contact("Susan","McBright",Option(4))
c4: Contact = Contact(Susan,McBright,Some(4))
scala> Map("division" -> "IT", "contacts" -> List(c1,c2))
res10: scala.collection.immutable.Map[String,java.io.Serializable] = Map(division -> IT, contacts -> List(Contact(Carl,Smith,Some(1)), Contact(Henry,Miller,Some(2))))
I'm not entirely sure the nature of your problem, but if you already have List[ContactDivisionList], it's pretty straightforward to serialize that to JSON:
scala> implicit val contactWrites = Json.writes[Contact]
contactWrites: play.api.libs.json.OWrites[Contact] = play.api.libs.json.OWrites$$anon$2#3676af92
scala> implicit val contactDivisionListWrites = Json.writes[ContactDivisionList]
contactDivisionListWrites: play.api.libs.json.OWrites[ContactDivisionList] = play.api.libs.json.OWrites$$anon$2#2999d17d
scala> Json.toJson(List(ContactDivisionList("IT", List(c1,c2)), ContactDivisionList("Sales", List(c3, c4))))
res2: play.api.libs.json.JsValue = [{"division":"IT","contacts":[{"firstname":"Carl","surname":"Smith","empID":1},{"firstname":"Henry","surname":"Miller","empID":2}]},{"division":"Sales","contacts":[{"firstname":"Nancy","surname":"McDonald","empID":3},{"firstname":"Susan","surname":"McBright","empID":4}]}]
It seems to me that you should avoid having that Map in the first place. I've never worked with anorm before, but I think where you need to be looking at is the code that data structure, because at that point, you've lost typesafety. You should ideally work with ContactDivisionList or construct your object using the JsValue cases directly.