Is there a way to decode arbitrary json (e.g: We don't know the keys at compile time)?
For example, I need to parse the following json:
{
"Foo": [
"Value 1",
"Value 2"
],
"Bar": [
"Bar Value 1"
],
"Baz": []
}
where the names and number of keys are not known at compile time and may change per GET request. The goal is basically to decode this into a Map String (Array String) type
Is there a way to do this using purescript-argonaut?
You can totally write your own by first parsing the string into Json via jsonParser, and then examining the resulting data structure with the various combinators provided by Argonaut.
But the quickest and simplest way, I think, is to parse it into Foreign.Object (Array String) first, and then convert to whatever your need, like Map String (Array String):
import Data.Argonaut (decodeJson, jsonParser)
import Data.Either (Either)
import Data.Map as Map
import Foreign.Object as F
decodeAsMap :: String -> Either _ (Map.Map String (Array String))
decodeAsMap str = do
json <- jsonParser str
obj <- decodeJson json
pure $ Map.fromFoldable $ (F.toUnfoldable obj :: Array _)
The Map instance of EncodeJSON will generate an array of tuple, you can manually construct a Map and see the encoded json.
let v = Map.fromFoldable [ Tuple "Foo" ["Value1", "Value2"] ]
traceM $ encodeJson v
Output should be [ [ 'Foo', [ 'Value1', 'Value2' ] ] ].
To do the reverse, you need to transform you object to an array of tuple, Object.entries can help you.
An example
// Main.js
var obj = {
foo: ["a", "b"],
bar: ["c", "d"]
};
exports.tuples = Object.entries(obj);
exports.jsonString = JSON.stringify(exports.tuples);
-- Main.purs
module Main where
import Prelude
import Data.Argonaut.Core (Json)
import Data.Argonaut.Decode (decodeJson)
import Data.Argonaut.Parser (jsonParser)
import Data.Either (Either)
import Data.Map (Map)
import Debug.Trace (traceM)
import Effect (Effect)
import Effect.Console (log)
foreign import tuples :: Json
foreign import jsonString :: String
main :: Effect Unit
main = do
let
a = (decodeJson tuples) :: Either String (Map String (Array String))
b = (decodeJson =<< jsonParser jsonString) :: Either String (Map String (Array String))
traceM a
traceM b
traceM $ a == b
Related
I am trying to deserialise a json string to an object using jsons but having problems with nested objects, but can't work out the syntax.
As an example the following code attempts to define the data structure as a series of dataclasses but fails to deserialise the nested objects C and D ? The syntax is clearly wrong, but its not clear to me how it should structured
import jsons
from dataclasses import dataclass
#dataclass
class D:
E: str
class C:
id: int
name:str
#dataclass
class test:
A: str
B: int
C: C()
D: D()
jsonString = {"A":"a","B":1,"C":[{"id":1,"name":"one"},{"id":2,"name":"two"}],"D":[{"E":"e"}]}
instance = jsons.load(jsonString, test)
Can anyone indicate the correct way to deserialise the objects from json ?
There are two relatively simple problems with your attempt:
You forgot to decorate C with #dataclass.
Test.C and Test.D aren't defined with types, but with instances of the types. (Further, you want both fields to be lists of the given type, not single instances of each.)
Given the code
import jsons
from dataclasses import dataclass
from typing import List
#dataclass
class D:
E: str
#dataclass # Problem 1 fixed
class C:
id: int
name: str
#dataclass
class Test:
A: str
B: int
C: List[C] # Problem 2 fixed; List[C] not C() or even C
D: List[D] # Problem 2 fixed; List[D], not D() or even D
Then
>>> obj = {"A":"a", "B":1, "C": [{"id": 1,"name": "one"}, {"id": 2, "name": "two"}], "D":[{"E": "e"}]}
>>> jsons.load(obj, Test)
test(A='a', B=1, C=[C(id=1, name='one'), C(id=2, name='two')], D=[D(E='e')])
from dataclasses import dataclass
from typing import List
from validated_dc import ValidatedDC
#dataclass
class D(ValidatedDC):
E: str
#dataclass
class C(ValidatedDC):
id: int
name: str
#dataclass
class Test(ValidatedDC):
A: str
B: int
C: List[C]
D: List[D]
jsonString = {
"A": "a",
"B": 1,
"C": [{"id": 1, "name": "one"}, {"id": 2, "name": "two"}],
"D": [{"E": "e"}]
}
instance = Test(**jsonString)
assert instance.C == [C(id=1, name='one'), C(id=2, name='two')]
assert instance.C[0].id == 1
assert instance.C[1].name == 'two'
assert instance.D == [D(E='e')]
assert instance.D[0].E == 'e'
ValidatedDC: https://github.com/EvgeniyBurdin/validated_dc
You can do something like this:
from collections import namedtuple
# First parameter is the class/tuple name, second parameter
# is a space delimited string of varaibles.
# Note that the variable names should match the keys from
# your dictionary of arguments unless only one argument is given.
A = namedtuple("A", "a_val") # Here the argument `a_val` can be called something else
B = namedtuple("B", "num")
C = namedtuple("C", "id name")
D = namedtuple("D", "E") # This must be `E` since E is the key in the dictionary.
# If you dont want immutable objects to can use full classes
# instead of namedtuples
# A dictionary which matches the name of an object seen in a payload
# to the object we want to create for that name.
object_options = {
"A": A,
"B": B,
"C": C,
"D": D
}
my_objects = [] # This is the list of object we get from the payload
jsonString = {"A":"a","B":1,"C":[{"id":1,"name":"one"},{"id":2,"name":"two"}],"D":[{"E":"e"}]}
for key, val in jsonString.items():
if key in object_options: # If this is a valid object
if isinstance(val, list): # If it is a list of this object
for v in val: # Then we need to add each object in the list
my_objects.append(object_options[key](**v))
elif isinstance(val, dict): # If the object requires a dict then pass the whole dict as arugments
my_objects.append(object_options[key](**val))
else: # Else just add this object with a singular argument.
my_objects.append(object_options[key](val))
print(my_objects)
Output:
[A(a_val='a'), B(num=1), C(id=1, name='one'), C(id=2, name='two'), D(E='e')]
I've finally managed to get this to work by removing the dataClass definition and expanding the class definitions old school.... code as follows...
import jsons
class D:
def __init__(self, E = ""):
self.E = E
class C:
def __init__(self, id = 0, name=""):
self.id = id
self.name = name
class test:
def __init__(self, A = "", B = 0, C = C(), D = D()):
self.A = A
self.B = B
self.C = C
self.D = D
jsonString = {"A":"a","B":1,"C":[{"id":1,"name":"one"},{"id":2,"name":"two"}],"D":[{"E":"e"}]}
instance = jsons.load(jsonString, test)
It now works but is not as clean as with a dataClass. Grateful if anyone can indicate how the original post can be constructed with the dataClass definition.
I have input json like
{"a": "x", "b": "y", "c": "z", .... }
I want to convert this json to a Map like Map[String, String]
so basically a map of key value pairs.
How can I do this using circe?
Note that I don't know what keys "a", "b", "c" will be present in Json. All I know is that they will always be strings and never any other data type.
I looked at Custom Decoders here https://circe.github.io/circe/codecs/custom-codecs.html but they work only when you know the tag names.
I found an example to do this in Jackson. but not in circe
import com.fasterxml.jackson.module.scala.DefaultScalaModule
import com.fasterxml.jackson.databind.ObjectMapper
val data = """
{"a": "x", "b", "y", "c": "z"}
"""
val mapper = new ObjectMapper
mapper.registerModule(DefaultScalaModule)
mapper.readValue(data, classOf[Map[String, String]])
While the solutions in the other answer work, they're much more verbose than necessary. Off-the-shelf circe provides an implicit Decoder[Map[String, String]] instance, so you can just write the following:
scala> val doc = """{"a": "x", "b": "y", "c": "z"}"""
doc: String = {"a": "x", "b": "y", "c": "z"}
scala> io.circe.parser.decode[Map[String, String]](doc)
res0: Either[io.circe.Error,Map[String,String]] = Right(Map(a -> x, b -> y, c -> z))
The Decoder[Map[String, String]] instance is defined in the Decoder companion object, so it's always available—you don't need any imports, other modules, etc. Circe provides instances like this for most standard library types with reasonable instances. If you want to decode a JSON array into a List[String], for example, you don't need to build your own Decoder[List[String]]—you can just use the one in implicit scope that comes from the Decoder companion object.
This isn't just a less verbose way to solve this problem, it's the recommended way to solve it. Manually constructing an explicit decoder instance and converting from Either to Try to compose parsing and decoding operations is both unnecessary and error-prone (if you do need to end up with Try or Option or whatever, it's almost certainly best to do that at the end).
Assuming:
val rawJson: String = """{"a": "x", "b": "y", "c": "z"}"""
This works:
import io.circe.parser._
val result: Try[Map[String, String]] = parse(rawJson).toTry
.flatMap(json => Try(json.asObject.getOrElse(sys.error("Not a JSON Object"))))
.flatMap(jsonObject => Try(jsonObject.toMap.map{case (name, value) => name -> value.asString.getOrElse(sys.error(s"Field '$name' is not a JSON string"))}))
val map: Map[String, String] = result.get
println(map)
Or with using Decoder:
import io.circe.Decoder
val decoder = Decoder.decodeMap(KeyDecoder.decodeKeyString, Decoder.decodeString)
val result = for {
json <- parse(rawJson).toTry
map <- decoder.decodeJson(json).toTry
} yield map
val map = result.get
println(map)
You can test following invalid inputs and see, what exception will be thrown:
val rawJson: String = """xxx{"a": "x", "b": "y", "c": "z"}""" // invalid JSON
val rawJson: String = """[1,2,3]""" // not a JSON object
val rawJson: String = """{"a": 1, "b": "y", "c": "z"}""" // not all values are string
I have a string in scala which in terms of formatting, it is a json, for example
{"name":"John", "surname":"Doe"}
But when I generate this value it is initally a string. I need to convert this string into a json but I cannot change the output of the source. So how can I do this conversion in Scala? (I cannot use the Play Json library.)
If you have strings as
{"name":"John", "surname":"Doe"}
and if you want to save to elastic as mentioned here then you should use parseRaw instead of parseFull.
parseRaw will return you JSONType and parseFull will return you map
You can do as following
import scala.util.parsing.json._
val jsonString = "{\"name\":\"John\", \"surname\":\"Doe\"}"
val parsed = JSON.parseRaw(jsonString).get.toString()
And then use the jsonToEs api as
sc.makeRDD(Seq(parsed)).saveJsonToEs("spark/json-trips")
Edited
As #Aivean pointed out, when you already have json string from source, you won't be needing to convert to json, you can just do
if jsonString is {"name":"John", "surname":"Doe"}
sc.makeRDD(Seq(jsonString)).saveJsonToEs("spark/json-trips")
You can use scala.util.parsing.json to convert JSON in string format to JSON (which is basically HashMap datastructure),
eg.
scala> import scala.util.parsing.json._
import scala.util.parsing.json._
scala> val json = JSON.parseFull("""{"name":"John", "surname":"Doe"}""")
json: Option[Any] = Some(Map(name -> John, surname -> Doe))
To navigate the json format,
scala> json match { case Some(jsonMap : Map[String, Any]) => println(jsonMap("name")) case _ => println("json is empty") }
John
nested json example,
scala> val userJsonString = """{"name":"John", "address": { "perm" : "abc", "temp" : "zyx" }}"""
userJsonString: String = {"name":"John", "address": { "perm" : "abc", "temp" : "zyx" }}
scala> val json = JSON.parseFull(userJsonString)
json: Option[Any] = Some(Map(name -> John, address -> Map(perm -> abc, temp -> zyx)))
scala> json.map(_.asInstanceOf[Map[String, Any]]("address")).map(_.asInstanceOf[Map[String, String]]("perm"))
res7: Option[String] = Some(abc)
I have a String with some arbitrary JSON in it. I want to construct a JsObject with my JSON string as a JSON object value, not a string value. For example, assuming my arbitrary string is a boring {} I want {"key": {}} and not {"key": "{}"}.
Here's how I'm trying to do it.
val myString = "{}"
Json.obj(
"key" -> Json.parse(myString)
)
The error I get is
type mismatch; found :
scala.collection.mutable.Buffer[scala.collection.immutable.Map[String,java.io.Serializable]]
required: play.api.libs.json.Json.JsValueWrapper
I'm not sure what to do about that.
"{}" is an empty object.
So, to get {"key": {}} :
Json.obj("key" -> Json.obj())
Update:
Perhaps you have an old version of Play. This works under Play 2.3.x:
scala> import play.api.libs.json._
scala> Json.obj("foo" -> Json.parse("{}"))
res2: play.api.libs.json.JsObject = {"foo":{}}
I was wondering if there is a parser or an easy way to iterate through a json object without knowing the keys/schema of the json ahead of time in scala. I took a look at a few libraries like json4s, but it seems to still require knowing the schema ahead of time before extracting the fields. I just want to iterate over each field, extract the fields and print out their values something like:
json.foreachkey(key -> println(key +":" + json.get(key))
In Play Json you'll initially parse your json into a JsValue; you can then pattern-match this to determine if it is a JsObject (note that you can find the fields of this using fields or value), a JsArray (again, note the value), or a primitive such as JsString or JsNull
def parse(jsVal: JsValue) {
jsVal match {
case json: JsObject =>
case json: JsArray =>
case json: JsString =>
...
}
}
If by json you mean any JValue, then json4s seems to have this functionality out of the box:
scala> import org.json4s.JsonDSL._
import org.json4s.JsonDSL._
scala> import org.json4s.native.JsonMethods._
import org.json4s.native.JsonMethods._
scala> val json = parse(""" { "numbers" : [1, 2, 3, 4] } """)
json: org.json4s.JValue = JObject(List((numbers,JArray(List(JInt(1), JInt(2), JInt(3), JInt(4))))))
scala> compact(render(json))
res1: String = {"numbers":[1,2,3,4]}
Use liftweb, it allows you to parse the json first into JValue - then extract native scala objects from it, no matter the schema:
val jsonString = """{"menu": {
| "id": "file",
| "value": "File",
| "popup": {
| "menuitem": [
| {"value": "New", "onclick": "CreateNewDoc()"},
| {"value": "Open", "onclick": "OpenDoc()"},
| {"value": "Close", "onclick": "CloseDoc()"}
| ]
| }
|}}""".stripMargin
val jVal: JValue = parse(jsonString)
jVal.values
>>> Map(menu -> Map(id -> file, value -> File, popup -> Map(menuitem -> List(Map(value -> New, onclick -> CreateNewDoc()), Map(value -> Open, onclick -> OpenDoc()), Map(value -> Close, onclick -> CloseDoc())))))