Using Scala and json4s (Maybe Im missing a golden fish library or something)
I am trying to add some list (or array) of strings to a JSON so in the end looks like:
{"already":"here",..."listToAdd":["a","b",c"]}
The fact is that I already have the String in a JObject and the list of strings in an Array[String] (but it could be changed to List if needed). So I followed the docat json4s.org which states that:
Any seq produces JSON array.
scala> val json = List(1, 2, 3)
scala> compact(render(json))
res0: String = [1,2,3]
Tuple2[String, A] produces field.
scala> val json = ("name" -> "joe")
scala> compact(render(json))
res1: String = {"name":"joe"}
And when trying it, it gives:
Error:(15, 28) type mismatch;
found : (String, String)
required: org.json4s.JValue
which expands to) org.json4s.JsonAST.JValue
println(compact(render(idJSON)))
Using Scala 2.11.4
Json4s 3.2.11 (Jackson)
You have to additionally import some implicit conversion methods:
import org.json4s.JsonDSL._
These will convert the Scala objects into the library's Json AST.
Related
I have a string in scala which in terms of formatting, it is a json, for example
{"name":"John", "surname":"Doe"}
But when I generate this value it is initally a string. I need to convert this string into a json but I cannot change the output of the source. So how can I do this conversion in Scala? (I cannot use the Play Json library.)
If you have strings as
{"name":"John", "surname":"Doe"}
and if you want to save to elastic as mentioned here then you should use parseRaw instead of parseFull.
parseRaw will return you JSONType and parseFull will return you map
You can do as following
import scala.util.parsing.json._
val jsonString = "{\"name\":\"John\", \"surname\":\"Doe\"}"
val parsed = JSON.parseRaw(jsonString).get.toString()
And then use the jsonToEs api as
sc.makeRDD(Seq(parsed)).saveJsonToEs("spark/json-trips")
Edited
As #Aivean pointed out, when you already have json string from source, you won't be needing to convert to json, you can just do
if jsonString is {"name":"John", "surname":"Doe"}
sc.makeRDD(Seq(jsonString)).saveJsonToEs("spark/json-trips")
You can use scala.util.parsing.json to convert JSON in string format to JSON (which is basically HashMap datastructure),
eg.
scala> import scala.util.parsing.json._
import scala.util.parsing.json._
scala> val json = JSON.parseFull("""{"name":"John", "surname":"Doe"}""")
json: Option[Any] = Some(Map(name -> John, surname -> Doe))
To navigate the json format,
scala> json match { case Some(jsonMap : Map[String, Any]) => println(jsonMap("name")) case _ => println("json is empty") }
John
nested json example,
scala> val userJsonString = """{"name":"John", "address": { "perm" : "abc", "temp" : "zyx" }}"""
userJsonString: String = {"name":"John", "address": { "perm" : "abc", "temp" : "zyx" }}
scala> val json = JSON.parseFull(userJsonString)
json: Option[Any] = Some(Map(name -> John, address -> Map(perm -> abc, temp -> zyx)))
scala> json.map(_.asInstanceOf[Map[String, Any]]("address")).map(_.asInstanceOf[Map[String, String]]("perm"))
res7: Option[String] = Some(abc)
This question has come up a few times recently, so I'm FAQ-ing it here. Suppose I've got some case classes like this:
import io.circe._, io.circe.generic.semiauto._
object model {
case class A(a: String)
case class B(a: String, i: Int)
case class C(i: Int, b: Boolean)
implicit val encodeA: Encoder[A] = deriveEncoder
implicit val encodeB: Encoder[B] = deriveEncoder
implicit val encodeC: Encoder[C] = deriveEncoder
implicit val decodeA: Decoder[A] = deriveDecoder
implicit val decodeB: Decoder[B] = deriveDecoder
implicit val decodeC: Decoder[C] = deriveDecoder
}
And I want to encode a value that could be any one of these as JSON using circe and Shapeless coproducts.
import io.circe.shapes._, io.circe.syntax._
import shapeless._
import model._
type ABC = A :+: B :+: C :+: CNil
val c: ABC = Coproduct[ABC](C(123, false))
This looks fine at first:
scala> c.asJson
res0: io.circe.Json =
{
"i" : 123,
"b" : false
}
But the problem is that I can never decode a coproduct containing a B element, since any valid JSON document that could be decoded as B can also be decoded as A, and the coproduct decoders provided by circe-shapes try the elements in the order they appear in the coproduct.
scala> val b: ABC = Coproduct[ABC](B("xyz", 123))
b: ABC = Inr(Inl(B(xyz,123)))
scala> val json = b.asJson
json: io.circe.Json =
{
"a" : "xyz",
"i" : 123
}
scala> io.circe.jawn.decode[ABC](json.noSpaces)
res1: Either[io.circe.Error,ABC] = Right(Inl(A(xyz)))
How can I disambiguate the elements of my coproduct in my encoding?
The circe-shapes module encodes ordinary hlists and coproducts without labels, as seen above: a coproduct as the bare JSON representation of the element, and an hlist ends up as just a JSON array (the same as the default tuple encoding):
scala> ("xyz" :: List(1, 2, 3) :: false :: HNil).asJson.noSpaces
res2: String = ["xyz",[1,2,3],false]
In the case of hlists there's no danger of ambiguity because of overlapping names, but for coproducts there is. In both cases, though, you can add labels to the JSON representation using Shapeless's labeling mechanism, which tags values with a type-level symbol.
An hlist with labels is called a "record", and a coproduct with labels is a "union". Shapeless provides special syntax and operations for both of these, and circe-shapes treats both differently from unlabeled hlists or coproducts. For example (assuming the definitions and imports above):
scala> import shapeless.union._, shapeless.syntax.singleton._
import shapeless.union._
import shapeless.syntax.singleton._
scala> type ABCL = Union.`'A -> A, 'B -> B, 'C -> C`.T
defined type alias ABCL
scala> val bL: ABCL = Coproduct[ABCL]('B ->> B("xyz", 123))
bL: ABCL = Inr(Inl(B(xyz,123)))
scala> val jsonL = bL.asJson
jsonL: io.circe.Json =
{
"B" : {
"a" : "xyz",
"i" : 123
}
}
scala> io.circe.jawn.decode[ABCL](jsonL.noSpaces)
res3: Either[io.circe.Error,ABCL] = Right(Inr(Inl(B(xyz,123))))
The encoding for records similarly includes the member names:
scala> ('a ->> "xyz" :: 'b ->> List(1) :: 'c ->> false :: HNil).asJson.noSpaces
res4: String = {"c":false,"b":[1],"a":"xyz"}
In general if you want your hlist and coproduct encodings to include labels (and look like the encodings you'd get from circe-generic for case classes and sealed trait hierarchies), you can use records or coproducts to tell circe-shapes to do this with a minimal amount of extra syntactic and runtime overhead.
Looking at this helpful answer on encoding a List[A] in the Play JSON library,
I tried to use Json#arr:
import play.api.libs.json._
scala> Json.arr( 1, 2, 3 )
res21: play.api.libs.json.JsArray = [1,2,3]
All works well so far, but what if I have a List[Int]:
scala> Json.arr( List(1,2,3) )
res22: play.api.libs.json.JsArray = [[1,2,3]] // not what I wanted
I attempted to pass the List[Int] as var-args (if that's the right term):
scala> Json.arr( List(1,2,3): _* )
<console>:18: error: type mismatch;
found : List[Int]
required: Seq[play.api.libs.json.Json.JsValueWrapper]
Json.arr( List(1,2,3): _* )
^
But that did not work.
Then, I tried to create a List[JsValueWrapper], and then pass that, via var-args, to Json#arr:
scala> List(1,2,3).map(Json.toJsFieldJsValueWrapper)
<console>:18: error: No Json serializer found for type T. Try to implement an implicit Writes or Format for this type.
List(1,2,3).map(Json.toJsFieldJsValueWrapper)
^
Although that failed, the following works when applied to a single Int:
scala> Json.toJsFieldJsValueWrapper(1)
res27: play.api.libs.json.Json.JsValueWrapper = JsValueWrapperImpl(1)
scala> Json.arr(res27)
res28: play.api.libs.json.JsArray = [1]
How can I pass a List[Int] into Json.arr to get an output of [1,2,3], i.e. a JsArray consisting of three JsNumber's?
You could use Json.toJson, validate as a JsArray, or return an empty one if invalid:
scala> val list = List(1, 2, 3)
list: List[Int] = List(1, 2, 3)
scala> Json.toJson(list).validate[JsArray].getOrElse(JsArray())
res6: play.api.libs.json.JsArray = [1,2,3]
You could also modify your lambda slightly with Json.arr:
scala> Json.arr(list.map(Json.toJsFieldJsValueWrapper(_)): _*)
res10: play.api.libs.json.JsArray = [1,2,3]
It seems as though without the parentheses, the compiler is having trouble inferring what T is. This is because the compiler tries to resolve the implicit before eta-expanding the anonymous function, so it doesn't know what argument will be passed to Json.toJsFieldJsValueWrapper yet. Json.toJsFieldJsValueWrapper(_) is fundamentally different because it expands to a slightly different Function1:
x => Json.toJsFieldJsValueWrapper(x)
Here the compiler knows that x will be an Int, so the implicit Writes[Int] is resolved.
How about just using Json.toJson and casting to a JsArray (if you need to)?
scala> Json.toJson(List(1,2,3)).as[JsArray]
res0: play.api.libs.json.JsArray = [1,2,3]
I have a String with some arbitrary JSON in it. I want to construct a JsObject with my JSON string as a JSON object value, not a string value. For example, assuming my arbitrary string is a boring {} I want {"key": {}} and not {"key": "{}"}.
Here's how I'm trying to do it.
val myString = "{}"
Json.obj(
"key" -> Json.parse(myString)
)
The error I get is
type mismatch; found :
scala.collection.mutable.Buffer[scala.collection.immutable.Map[String,java.io.Serializable]]
required: play.api.libs.json.Json.JsValueWrapper
I'm not sure what to do about that.
"{}" is an empty object.
So, to get {"key": {}} :
Json.obj("key" -> Json.obj())
Update:
Perhaps you have an old version of Play. This works under Play 2.3.x:
scala> import play.api.libs.json._
scala> Json.obj("foo" -> Json.parse("{}"))
res2: play.api.libs.json.JsObject = {"foo":{}}
Using an existing JsValue, I'm parsing its values to fill out a new JSON object.
val json: JsValue = getJson(...)
val newJson: JsObject = Json.obj(
"Name" -> (json \ "personName").asOpt[String].getOrElse("")
)
However, I get the following compile-time error on the third line:
No Json deserializer found for type Object.
Looking at getOrElse's docs, why does "Object," rather than "String," get returned in my above code?
final def getOrElse[B >: A](default: ⇒ B): B
I admit that I don't understand B >: A.