I'm having trouble parsing JSON with json4s.jackson in spark-shell. The same thing works fine in sbt repl.
I wonder if there's a workaround for the version of Spark I'm using.
spark-shell v1.6, scala v2.10.5
sbt repl scala v 2.11.8
.
The following example demonstrates the problem.
sbt repl works as expected for all examples.
spark-shell chokes and gives an error for val c. What's weird is that it seems to choke on Option[Int] or Option[Double] but it works fine for Option[A] where A is a class.
.
import org.json4s.JsonDSL._
import org.json4s.jackson.JsonMethods.{render,compact,pretty}
import org.json4s.DefaultFormats
import org.json4s.jackson.JsonMethods._
import org.json4s.{JValue, JObject}
implicit val formats = DefaultFormats
class A(val a: Int, val aa: Int)
class B(val b: Int, val optA: Option[A]=None)
class C(val b: Int, val bb: Option[Int]=None)
val jb_optA_nested: JObject = ("b" -> 5) ~ ("optA" -> ("a" -> 999) ~ ("aa" -> 1000))
val jb_optA_not_nested: JObject = ("b" -> 5) ~ ("a" -> 999) ~ ("aa" -> 1000)
val jb_optA_none: JObject = ("b" -> 5)
val jc: JObject = ("b" -> 5) ~ ("bb" -> 100)
val b_nested = jb_optA_nested.extract[B] // works as expected in both (optA=Some(A(999,1000)))
val b_not_nested = jb_optA_not_nested.extract[B] // works as expected in both (optA=None)
val b_none = jb_optA_none.extract[B] // works as expected in both (optA=None)
val c = jc.extract[C] // error in spark-shell; works fine in sbt repl
The error generated is: org.json4s.package$MappingException: Can't find constructor for C
The only real difference I can find (other than scala versions) is that in spark-shell... it chokes on Option[native types] and seems to work on Option[user-defined classes]. But maybe that's coincidence.
In posts like this... JSON4s can't find constructor w/spark I see comments where people suggest the class structure doesn't match the JSON... but to me, class C and val jc look identical.
Also of note... this error persists when I harden class defs and functions in a .JAR and import defs into spark-shell from the jar instead of defining in the repl. sometimes that's relevant for spark 1.6, but doesn't seem to be here.
Have you tried:
class C(val b: Int, val bb: Option[java.lang.Integer]=None)
I've had issues with the Scala Int before with Json4s - although I can't recall exactly what it was.
Also doing a case class is worthwhile with the Int - any reason you prefer a regular class? I don't see any vars.
Related
Using Scala and json4s (Maybe Im missing a golden fish library or something)
I am trying to add some list (or array) of strings to a JSON so in the end looks like:
{"already":"here",..."listToAdd":["a","b",c"]}
The fact is that I already have the String in a JObject and the list of strings in an Array[String] (but it could be changed to List if needed). So I followed the docat json4s.org which states that:
Any seq produces JSON array.
scala> val json = List(1, 2, 3)
scala> compact(render(json))
res0: String = [1,2,3]
Tuple2[String, A] produces field.
scala> val json = ("name" -> "joe")
scala> compact(render(json))
res1: String = {"name":"joe"}
And when trying it, it gives:
Error:(15, 28) type mismatch;
found : (String, String)
required: org.json4s.JValue
which expands to) org.json4s.JsonAST.JValue
println(compact(render(idJSON)))
Using Scala 2.11.4
Json4s 3.2.11 (Jackson)
You have to additionally import some implicit conversion methods:
import org.json4s.JsonDSL._
These will convert the Scala objects into the library's Json AST.
Looking at this helpful answer on encoding a List[A] in the Play JSON library,
I tried to use Json#arr:
import play.api.libs.json._
scala> Json.arr( 1, 2, 3 )
res21: play.api.libs.json.JsArray = [1,2,3]
All works well so far, but what if I have a List[Int]:
scala> Json.arr( List(1,2,3) )
res22: play.api.libs.json.JsArray = [[1,2,3]] // not what I wanted
I attempted to pass the List[Int] as var-args (if that's the right term):
scala> Json.arr( List(1,2,3): _* )
<console>:18: error: type mismatch;
found : List[Int]
required: Seq[play.api.libs.json.Json.JsValueWrapper]
Json.arr( List(1,2,3): _* )
^
But that did not work.
Then, I tried to create a List[JsValueWrapper], and then pass that, via var-args, to Json#arr:
scala> List(1,2,3).map(Json.toJsFieldJsValueWrapper)
<console>:18: error: No Json serializer found for type T. Try to implement an implicit Writes or Format for this type.
List(1,2,3).map(Json.toJsFieldJsValueWrapper)
^
Although that failed, the following works when applied to a single Int:
scala> Json.toJsFieldJsValueWrapper(1)
res27: play.api.libs.json.Json.JsValueWrapper = JsValueWrapperImpl(1)
scala> Json.arr(res27)
res28: play.api.libs.json.JsArray = [1]
How can I pass a List[Int] into Json.arr to get an output of [1,2,3], i.e. a JsArray consisting of three JsNumber's?
You could use Json.toJson, validate as a JsArray, or return an empty one if invalid:
scala> val list = List(1, 2, 3)
list: List[Int] = List(1, 2, 3)
scala> Json.toJson(list).validate[JsArray].getOrElse(JsArray())
res6: play.api.libs.json.JsArray = [1,2,3]
You could also modify your lambda slightly with Json.arr:
scala> Json.arr(list.map(Json.toJsFieldJsValueWrapper(_)): _*)
res10: play.api.libs.json.JsArray = [1,2,3]
It seems as though without the parentheses, the compiler is having trouble inferring what T is. This is because the compiler tries to resolve the implicit before eta-expanding the anonymous function, so it doesn't know what argument will be passed to Json.toJsFieldJsValueWrapper yet. Json.toJsFieldJsValueWrapper(_) is fundamentally different because it expands to a slightly different Function1:
x => Json.toJsFieldJsValueWrapper(x)
Here the compiler knows that x will be an Int, so the implicit Writes[Int] is resolved.
How about just using Json.toJson and casting to a JsArray (if you need to)?
scala> Json.toJson(List(1,2,3)).as[JsArray]
res0: play.api.libs.json.JsArray = [1,2,3]
I am trying to follow the tutorial https://www.jamesward.com/2012/02/21/play-framework-2-with-scala-anorm-json-coffeescript-jquery-heroku but of course play-scala has changed since the tutorial (as seems to be the case with every tutorial I find). I am using 2.4.3 This requires I actually learn how things work, not necessarily a bad thing.
One thing that is giving me trouble is the getOrElse method.
Here is my Bar.scala model
package models
import play.api.db._
import play.api.Play.current
import anorm._
import anorm.SqlParser._
case class Bar(id: Option[Long], name: String)
object Bar {
val simple = {
get[Option[Long]]("id") ~
get[String]("name") map {
case id~name => Bar(id, name)
}
}
def findAll(): Seq[Bar] = {
DB.withConnection { implicit connection =>
SQL("select * from bar").as(Bar.simple *)
}
}
def create(bar: Bar): Unit = {
DB.withConnection { implicit connection =>
SQL("insert into bar(name) values ({name})").on(
'name -> bar.name
).executeUpdate()
}
}
}
and my BarFormat.scala Json formatter
package models
import play.api.libs.json._
import anorm._
package object Implicits {
implicit object BarFormat extends Format[Bar] {
def reads(json: JsValue):JsResult[Bar] = JsSuccess(Bar(
Option((json \ "id").as[Long]),
(json \ "name").as[String]
))
def writes(bar: Bar) = JsObject(Seq(
"id" -> JsNumber(bar.id.getOrElse(0L)),
"name" -> JsString(bar.name)
))
}
}
and for completeness my Application.scala controller:
package controllers
import play.api.mvc._
import play.api.data._
import play.api.data.Forms._
import javax.inject.Inject
import javax.inject._
import play.api.i18n.{ I18nSupport, MessagesApi, Messages, Lang }
import play.api.libs.json._
import views._
import models.Bar
import models.Implicits._
class Application #Inject()(val messagesApi: MessagesApi) extends Controller with I18nSupport {
val barForm = Form(
single("name" -> nonEmptyText)
)
def index = Action {
Ok(views.html.index(barForm))
}
def addBar() = Action { implicit request =>
barForm.bindFromRequest.fold(
errors => BadRequest,
{
case (name) =>
Bar.create(Bar(None, name))
Redirect(routes.Application.index())
}
)
}
def listBars() = Action { implicit request =>
val bars = Bar.findAll()
val json = Json.toJson(bars)
Ok(json).as("application/json")
}
and routes
# Routes
# This file defines all application routes (Higher priority routes first)
# ~~~~
# Home page
POST /addBar controllers.Application.addBar
GET / controllers.Application.index
GET /listBars controllers.Application.listBars
# Map static resources from the /public folder to the /assets URL path
GET /assets/*file controllers.Assets.versioned(path="/public", file: Asset)
When I try to run my project I get the following error:
now bar.id is defined as an Option[Long] so bar.id.getOrElse(0L) should return a Long as far as I can tell, but it is clearly returning an Any. Can anyone help me understand why?
Thank You!
That's the way type inference works in Scala...
First of all there is an implicit conversion from Int to BigDecimal:
scala> (1 : Int) : BigDecimal
res0: BigDecimal = 1
That conversion allows for Int to be converted before the option is constructed:
scala> Some(1) : Option[BigDecimal]
res1: Option[BigDecimal] = Some(1)
If we try getOrElse on its own where the type can get fixed we get expected type Int:
scala> Some(1).getOrElse(2)
res2: Int = 1
However, this does not work (the problem you have):
scala> Some(1).getOrElse(2) : BigDecimal
<console>:11: error: type mismatch;
found : Any
required: BigDecimal
Some(1).getOrElse(2) : BigDecimal
^
Scala's implicit conversions kick in last, after type inference is performed. That makes sense, because if you don't know the type how would you know what conversions need to be applied. Scala can see that BigDecimal is expected, but it has an Int result based on the type of the Option it has. So it tries to widen the type, can't find anything that matches BigDecimal in Int's type hierarchy and fails with the error.
This works, however because the type is fixed in the variable declaration:
scala> val v = Some(1).getOrElse(2)
v: Int = 1
scala> v: BigDecimal
res4: BigDecimal = 1
So we need to help the compiler somehow - any type annotation or explicit conversion would work. Pick any one you like:
scala> (Some(1).getOrElse(2) : Int) : BigDecimal
res5: BigDecimal = 1
scala> Some(1).getOrElse[Int](2) : BigDecimal
res6: BigDecimal = 1
scala> BigDecimal(Some(1).getOrElse(2))
res7: scala.math.BigDecimal = 1
Here is the signature for Option.getOrElse method:
getOrElse[B >: A](default: ⇒ B): B
The term B >: A expresses that the type parameter B or the abstract type B refer to a supertype of type A, in this case, Any being the supertype of Long:
val l: Long = 10
val a: Any = l
So, we can do something very similar here with getOrElse:
val some: Option[Long] = Some(1)
val value: Any = option.getOrElse("potatos")
val none: Option[Long] = None
val elseValue: Any = none.getOrElse("potatos")
Which brings us to your scenario: the returned type from getOrElse will be a Any and not a BigDecimal, so you will need another way to handle this situation, like using fold, per instance:
def writes(bar: Bar) = {
val defaultValue = BigDecimal(0)
JsObject(Seq(
"id" -> JsNumber(bar.id.fold(defaultValue)(BigDecimal(_))),
"name" -> JsString(bar.name)
))
}
Some other discussions that can help you:
Why is Some(1).getOrElse(Some(1)) not of type Option[Int]?
Option getOrElse type mismatch error
I have the following code that works in a console app when referencing "org.reactivemongo" %% "play2-reactivemongo" % "0.10.5.0.akka23"
when I update the reference to "org.reactivemongo" % "play2-reactivemongo_2.11" % "0.11.0.play23-M3" I get:
No Json serializer as JsObject found for type play.api.libs.json.JsObject. Try to implement an implicit OWrites or OFormat for this type.
import org.joda.time.DateTime
import reactivemongo.bson.BSONObjectID
import play.modules.reactivemongo.json.BSONFormats._
case class GoogleToken
(
id: Option[BSONObjectID],
name: String,
emailAddress: String,
refreshToken: String,
expires: DateTime
)
object GoogleToken {
import play.api.libs.json.Json
// Generates Writes and Reads
implicit val googleTokenFormat = Json.format[GoogleToken]
}
and then
val collection = db.collectionJSONCollection
val query = Json.obj()
val cursor = collection.find(query).
cursor[GoogleToken](ReadPreference.nearest).
collect[List]()
What am I doing wrong?
The final release of ReactiveMongo 0.11 has been published ("org.reactivemongo" %% "play2-reactivemongo" % "0.11.0.play23").
As indicated on the updated documentation, for the default BSON/JSON conversions, it's recommended to have: import play.modules.reactivemongo.json._, ImplicitBSONHandlers._.
In my case, I was feeding ReactiveMongo (insert) with a JsValue instead than a JsObject. In order to fix it, behind adding import play.modules.reactivemongo.json._, I also had to change my implicit Writes in OWrites:
from
implicit val myWrites: Writes[A] = new Writes[A] {
def writes(a: A) = Json.obj(...)
to
implicit val myWrites: OWrites[A] = new OWrites[A] { <-- NOTE THE 'O' before 'Writes'
def writes(a: A) = Json.obj(...)
Mine worked out after adding:
import play.modules.reactivemongo.json._
import play.modules.reactivemongo.json.collection._
try to add
import reactivemongo.play.json._
For me adding this import worked.
import play.modules.reactivemongo.json._
I am trying to write a Map in key as int to json string but I am not able to do so:
import org.json4s._
import org.json4s.jackson.JsonMethods._
import org.json4s.JsonDSL._
object MyObject {
def main(args: Array[String]) {
// Works fine
//val myMap = Map("a" -> List(3,4), "b" -> List(7,8))
// Does not work
val myMap = Map(4 -> Map("a" -> 5))
val jsonString = pretty(render(myMap))
println(jsonString)
}
I am receiving the following error:
[error] /my_stuff/my_file.scala:14: overloaded method value render with alternatives:
[error] (value: org.json4s.JValue)org.json4s.JValue <and>
[error] (value: org.json4s.JValue)(implicit formats: org.json4s.Formats)org.json4s.JValue
[error] cannot be applied to (scala.collection.immutable.Map[Int,scala.collection.immutable.Map[String,Int]])
[error] val jsonString = pretty(render(myMap))
[error] ^
[error] one error found
[error] (compile:compileIncremental) Compilation failed
I vaguely understand the error message, it looks like render expects JValue as an input, and I am not providing it, but I don't the first case either, and the code works as I expect.
How do I write such map to json string?
Edit: My source of confusion
I am mostly a python programmer, and in python
In [1]: import json
In [2]: wrong = {2: 5}
In [3]: with open("wrong.json") as f:
...: json.dump(wrong, f)
works perfectly fine, of course python stringifies the 2.
I think it is an expected result. If you check the json specification you will see that you need to use strings for the names of the elements.
So I am afraid you will need something like:
val myMap = Map("4" -> Map("a" -> 5))