When trying to insert a MongoDBObject that contains a JsNumber
val obj: DBObject = getDbObj // contains a "JsNumber()"
collection.insert(obj)
the following error occurs:
[error] play - Cannot invoke the action, eventually got an error: java.lang.IllegalArgumentException: can't serialize class scala.math.BigDecimal
I tried to replace the JsNumber with an Int, but I got the same error.
EDIT
Error can be reproduced via this test code. Full code in scalatest (https://gist.github.com/kman007us/6617735)
val collection = MongoConnection()("test")("test")
val obj: JsValue = Json.obj("age" -> JsNumber(100))
val q = MongoDBObject("name" -> obj)
collection.insert(q)
There are no registered handlers for Plays JSON implementation - you could add handlers to automatically translate plays Js Types to BSON types. However, that wont handle mongodb extended json which has a special structure dealing with non native json types eg: date and objectid translations.
An example of using this is:
import com.mongodb.util.JSON
val obj: JsValue = Json.obj("age" -> JsNumber(100))
val doc: DBObject = JSON.parse(obj.toString).asInstanceOf[DBObject]
For an example of a bson transformer see the joda time transformer.
It seems that casbah driver isn't compatible with Plays's JSON implementation. If I look through the cashbah code than it seems that you must use a set of MongoDBObject objects to build your query. The following snippet should work.
val collection = MongoConnection()("test")("test")
val obj = MongoDBObject("age" -> 100)
val q = MongoDBObject("name" -> obj)
collection.insert(q)
If you need the compatibility with Play's JSON implementation then use ReactiveMongo and Play-ReactiveMongo.
Edit
Maybe this Gist can help to convert JsValue objects into MongoDBObject objects.
Related
I have a text file with json value. and this gets read into a DF
{"name":"Michael"}
{"name":"Andy", "age":30}
I want to infer the schema dynamically for each line while Streaming and store it in separate locations(tables) depending on its schema.
unfortunately while I try to read the value.schema it still shows as String. Please help on how to do it on Streaming as RDD is not allowed in streaming.
I wanted to use the following code which doesnt work as the value is still read as String format.
val jsonSchema = newdf1.select("value").as[String].schema
val df1 = newdf1.select(from_json($"value", jsonSchema).alias("value_new"))
val df2 = df1.select("value_new.*")
I even tried to use,
schema_of_json("json_schema"))
val jsonSchema: String = newdf.select(schema_of_json(col("value".toString))).as[String].first()
still no hope.. Please help..
You can load the data as textFile, create case class for person and parse every json string to Person instance using json4s or gson, then creating the Dataframe as follows:
case class Person(name: String, age: Int)
val jsons = spark.read.textFile("/my/input")
val persons = jsons.map{json => toPerson(json) //instead of 'toPerson' actually parse with json4s or gson to return Person instance}
val df = sqlContext.createDataFrame(persons)
Deserialize json to case class using json4s:
https://commitlogs.com/2017/01/14/serialize-deserialize-json-with-json4s-in-scala/
Deserialize json to case class using gson:
https://alvinalexander.com/source-code/scala/scala-case-class-gson-json-object-deserialization-and-scalatra
Using jackson library I read json data from a file (each row of file is a JSON object) an parse it to a map object of String and Any. My goal is to save specified keys (id and text) to a collection.
val input = scala.io.Source.fromFile("data.json").getLines()
val mapper = new ObjectMapper() with DefaultScalaModule
val data_collection = mutable.HashMap.empty[Int, String]
for (i <- input){
val parsedJson = mapper.readValue[Map[String, Any]](i)
data_collection.put(
parsedJson.get("id"),
parsedJson.get("text")
)
But as the values in the parsedJson map have the Any type, getting some keys like id and text, it returns Some(value) not just the value with the appropriate type. I expect the values for the id key to be Integer and values for the text to be String.
Running the code I got the error:
Error:(31, 23) type mismatch;
found : Option[Any]
required: Int
parsedJson.get("id"),
Here is a sample of JSON data in the file:
{"text": "Hello How are you", "id": 1}
Is it possible in Scala to parse id values to Int and text values to String, or at least convert Some(value) to value with type Int or String?
If you want to get a plain value from a Map instead of a Option you can use the () (apply) method - However it will throw an exception if the key is not found.
Second, Scala type system is static not dynamic, if you have an Any that's it, it won't change to Int or String at runtime, and the compiler will fail - Nevertheless, you can cast them using the asInstanceOf[T] method, but again if type can't be casted to the target type it will throw an exception.
Please note that even if you can make your code work with the above tricks, that code wouldn't be what you would expect in Scala. There are ways to make the code more typesafe (like pattern matching), but parsing a Json to a typesafe object is an old problem, I'm sure jackson provides a way to parse a json into case class that represent your data. If not take a look to circe it does.
Try the below code :
import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.module.scala.DefaultScalaModule
import
com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper
val input = scala.io.Source.fromFile("data.json").getLines()
val mapper = new ObjectMapper() with ScalaObjectMapper
mapper.registerModule(DefaultScalaModule)
val obj = mapper.readValue[Map[String, Any]](input)
val data_collection = mutable.HashMap.empty[Int, String]
for (i <- c) {
data_collection.put(
obj.get("id").fold(0)(_.toString.toInt),
obj.get("text").fold("")(_.toString)
)
}
println(data_collection) // Map(1 -> Hello How are you)
I am using Play Framework and I am trying to convert a Scala object to a JSON string.
Here is my code where I get my object:
val profile: Future[List[Profile]] = profiledao.getprofile(profileId);
The object is now in the profile value.
Now I want to convert that profile object which is a Future[List[Profile]] to JSON data and then convert that data into a JSON string then write into a file.
Here is the code that I wrote so far:
val jsondata = Json.toJson(profile)
Jackson.toJsonString(jsondata)
This is how I am trying to convert into JSON data but it is giving me the following output:
{"empty":false,"traversableAgain":true}
I am using the Jackson library to do the conversion.
Can someone help me with this ?
Why bother with Jackson? If you're using Play, you have play-json available to you, which uses Jackson under the hood FWIW:
First, you need an implicit Reads to let play-json know how to serialize Profile. If Profile is a case class, you can do this:
import play.api.libs.json._
implicit val profileFormat = Json.format[Profile]
If not, define your own Reads like this.
Then since getprofile (which should follow convention and be getProfile) returns Future[List[Profile]], you can do this to get a JsValue:
val profilesJson = profiledao.getprofile(profileId).map(toJson)
(profiledao should also be profileDao.)
In the end, you can wrap this in a Result like Ok and return that from your controller.
I am new to Scala and Play, and I ask for help with this simple example. I tried to search for solution by myself, but I did not succeed.
I am trying to do the example from from Mastering Play Framework for Scala book, the one about extending Json parser (Pages 29-30).
The environment I use is:
Scala: 2.11.7
Play: 2.5.8
Activator: 1.3.10
The code is:
case class Subscription(emailId: String, interval: Long)
In controller:
import play.api.libs.json.Json
import play.api.libs.json.JsValue
import play.api.libs.json.Writes
.....
val parseAsSubscription = parse.using {
request =>
parse.json.map {
body =>
val emailId:String = (body \ "emailId").as[String]
val fromDate:Long = (body \ "fromDate").as[Long]
Subscription(emailId, fromDate)
}
}
implicit val subWrites:Writes[Subscription] = Json.writes[Subscription]
def getSub = Action(parseAsSubscription) {
request =>
val subscription: Subscription = request.body
Ok(Json.toJson(Subscription))
}
The line: Ok(Json.toJson(Subscription)) gives an error
No Json serializer found for type models.Subscription.type. Try to
implement an implicit Writes or Format for this type.
This is odd, because Writes object is defined one row above. Thus, I tried to pass it to toJson method explicitly:
Ok(Json.toJson(Subscription)(subWrites))
It gave me a different error, which partially explained why existing Writes object did not suit:
type mismatch;
found:
play.api.libs.json.Writes[models.Subscription]
required:
play.api.libs.json.Writes[models.Subscription.type]
However, I don't understand the nature of this error and what models.Subscription.type is .
I used to do a similar thing in a different example, and it worked just fine.
Any help will be appreciated.
You're trying to serialize the type Subscription, rather than the request body, which you stored as the value subscription. Try replacing the last line with Ok(Json.toJson(subscription)).
I'm using playframework 2.1.0 with Anorm to query a db.
I want to serialize the result to json without going through any interim objects/case classes.
this is what the flow looks like:
Using anorm:
DB.withConnection { implicit c =>
val q = SQL(""" long query goes here """)
q().toList
}
then I take this result and transform it from a List[SqlRow] to List[Map[String,Any]].
String,Any is the column name, value (Object/Any)
val asMap = info.toList.map(row => scala.collection.immutable.Map(row.asMap.toSeq:_*))
The i'd like to jsonize this.
I tried some json libs : GSON, spray-json, playframework json lib.
But none of them seem to work with Any out of the box.
I tried writing implicit writer for the Any type with some pattern matching, but the problem is that this writer always overtakes all the other writes so the json is not produced correctly.
Advise?
How would you suggest transforming a result from Anorm to Json? without any interim domain models.
found a solution, not the best, using FlexJson.
The annoying thing is that FlexJson is not very scala oriented so scala collections and some scala types need to be converted to the equivalent Java type.
val info:List[SqlRow] = loadInfoFromDB using Anorm
//transform scala maps to java maps
val asMap: List[util.Map[String, Any]] = info.toList.map(row => JavaConversions.mapAsJavaMap(row.asMap))
//create the basic FlexJson serializer
val flexJson: JSONSerializer = new flexjson.JSONSerializer()
//register a Option transformer so it can serialize Options correctly
flexJson.transform(new flexjson.transformer.AbstractTransformer {
def transform(`object`: Any) {
`object`.asInstanceOf[Option[_]] match {
case None => getContext.write("null")
case Some(b:Any) => getContext.transform(b)
}
}
},classOf[Option[_]])
//finally convert the scala List to java List and use this serializer on it.
val infoJsn: String = flexJson.deepSerialize(JavaConversions.seqAsJavaList(asMap))