I am new to scala and to the playframework, but so far it is great. I am having trouble figuring out how to turn a list of data into json (or really any complex structure). This isn't a real-world example, but here is what I am trying to do. Get some data from a database.
scala> val result:List[(Long,String)] = DB.withConnection { implicit c =>
SQL("select * from users").as(
long("id")~str("uid") map(flatten)*)
}
result: List[(Long, String)] = List((3,397a73ee5150429786863db144341bb3), (4,2850760dc9024c16bea6c8c65f409821), (5,636ee2bf758e4f699f27890ac55d7db2))
I would like to be able to then turn that into json and return it. based on this doc, it looks like i need to iterate through and call toJson on the result
http://www.playframework.org/documentation/2.0/ScalaJson
But, In practice i am having trouble doing it. Is this even the correct approach? Is there some scala concept that would make this simple? I see some examples using case classes, but I haven't quite wrapped my head around that concept yet.
I don't really expect this to work but, i guess I am conceptually trying to do something like this
scala> toJson(Map("response" -> result))
<console>:27: error: No Json deserializer found for type scala.collection.immutable.Map[java.lang.String,List[(Long, String)]]. Try to implement an implicit Writes or Format for this type.
toJson(Map("response" -> result))
Thanks
As said, you can write you own implicit Writes to do that but you can also rely on the existing Writes and just retrieve your data as a List[Map[String, Any]] and apply toJson on it:
val simple = {
get[Pk[Long]]("user.id") ~
get[Long]("user.uid") map {
case id~uid => Map("id" -> id.get.toString, "uid" -> uid.toString)
}
}
val result:List[Map(String,String)] = DB.withConnection { implicit c =>
SQL("select * from users").as(User.simple *)
}
Related
I have some JSON coming from an external API which I have no control over. Part of the JSON is formatted like this:
{
"room_0": {
"area_sq_ft": 151.2
},
"room_1": {
"area_sq_ft": 200.0
}
}
Instead of using an array like they should have, they've used room_n for a key to n number of elements. Instead of creating a case class with room_0, room_1, room_2, etc., I want to convert this to a Seq[Room] where this is my Room case class:
case class Room(area: Double)
I am using Reads from play.api.libs.json for converting other parts of the JSON to case classes and would prefer to use Reads for this conversion. How could I accomplish that?
Here's what I've tried.
val sqFtReads = (__ \ "size_sq_ft").read[Double]
val roomReads = (__ \ "size_sq_ft").read[Seq[Room]](sqFtReads).map(Room)
cmd19.sc:1: overloaded method value read with alternatives:
(t: Seq[$sess.cmd17.Room])play.api.libs.json.Reads[Seq[$sess.cmd17.Room]] <and>
(implicit r: play.api.libs.json.Reads[Seq[$sess.cmd17.Room]])play.api.libs.json.Reads[Seq[$sess.cmd17.Room]]
cannot be applied to (play.api.libs.json.Reads[Double])
val roomReads = (__ \ "size_sq_ft").read[Seq[Room]](sqFtReads).map(Room)
A tricky little challenge but completely achievable with Reads.
First, Reads[Room] - i.e. the converter for a single Room instance:
val roomReads = new Reads[Room] {
override def reads(json: JsValue): JsResult[Room] = {
(json \ "area_sq_ft").validate[Double].map(Room(_))
}
}
Pretty straightforward; we peek into the JSON and try to find a top-level field called area_sq_ft which validates as a Double. If it's all good, we return the populated Room instance as needed.
Next up, the converter for your upstream object that in good Postel's Law fashion, you are cleaning up for your own consumers.
val strangeObjectReads = new Reads[Seq[Room]] {
override def reads(json: JsValue): JsResult[Seq[Room]] = {
json.validate[JsObject].map { jso =>
val roomsSortedNumerically = jso.fields.sortBy { case (name, contents) =>
val numericPartOfRoomName = name.dropWhile(!_.isDigit)
numericPartOfRoomName.toInt
}
roomsSortedNumerically.map { case (name, contents) =>
contents.as[Room](roomReads)
}
}
}
}
The key thing here is the json.validate[JsObject] around the whole lot. By mapping over this we get the JsResult that we need to wrap the whole thing, plus, we can get access to the fields inside the JSON object, which is defined as a Seq[(String, JsValue)].
To ensure we put the fields in the correct order in the output sequence, we do a little bit of string manipulation, getting the numeric part of the room_1 string, and using that as the sortBy criteria. I'm being a bit naive here and assuming your upstream server won't do anything nasty like skip room numbers!
Once you've got the rooms sorted numerically, we can just map over them, converting each one with our roomReads converter.
You've probably noticed that my custom Reads implementations are most definitely not one-liners. This comes from bitter experience dealing with oddball upstream JSON formats. Being a bit verbose, using a few more variables and breaking things up a bit pays off big time when that upstream server changes its JSON format suddenly!
I am trying to query an API which returns a JSON array (e.g. [{"name":"obj1", "value":5}, {"name":"obj2", "value":2}]) and process the result, which gets parsed as an Option[List[Map[String,Any]]]. However, I am not sure how to properly extract each Map, since the types are erased at runtime.
import scala.util.parsing.json._
import scalaj.http._
val url = "https://api.github.com/users/torvalds/repos"
val req = Http(url).asString
val parsed = JSON.parseFull(req.body) match {
case Some(data) => data match {
case list: List[_] => list
case _ => sys.error("Result is not a list.")
}
case None => sys.error("Invalid JSON received.")
}
parsed.foreach{
case x: Map[_,_] => x.get("full_name") // error here
}
The error occurs because I cannot apply the function with a String key type. However, because of type erasure, the key and value type are unknown, and specifying that it's a String map throws compiler warnings.
Am I going about things the wrong way? Or maybe I'd have better luck with a different HTTP/JSON library?
You can replace your last line with:
parsed.collect{ case x: Map[_,_] => x.asInstanceOf[Map[String,Any]].get("full_name") }
We sort of "cheat" here since we know the keys in a JSON are always Strings.
As for your last question, if you need something lightweight, I think what you have here is as simple as it gets.
Take a look at this SO post if you want to do something more powerful with your pattern matching.
I have made a small app using json and reactiveMongo which inserts
Students Information.
object Applications extends Controller{
val studentDao = StudentDaoAndEntity
val studentqueryReader: Reads[JsObject] = implicitly[Reads[JsObject]]
def saveStudent = Action.async(parse.json) { request =>
request.body.validate[StudentInfo].map {
k => studentDao.insertStudent(k).map {
l => Ok("Successfully inserted")
}
}.getOrElse(Future.successful(BadRequest("Invalid Json"))
In databse
object StudentDaoAndEntity {
val sreader: Reads[StudentInfo] = Json.reads[StudentInfo]
val swriter: Writes[StudentInfo] = Json.writes[StudentInfo]
val studentqueryReader: Reads[JsObject] = implicitly[Reads[JsObject]]
def db = ReactiveMongoPlugin.db
def collection: JSONCollection = db[JSONCollection]("student")
def insertStudent(student: StudentInfo): Future[JsObject]= {
val modelToJsObj = swriter.writes(student).as[JsObject]
collection.insert(modelToJsObj) map (_ => modelToJsObj)
}
This works fine. Now I need to get all data I have inserted. How can I
do that? I am not asking for code but for Idea.
First of all: it seems that you are using Play-ReactiveMongo (as far as i know, JSONCollection is not part of ReactiveMongo itself). If this is the case, then your code is unnecessarily complex. Instead of doing the JSON conversions manually, you just can pass your StudentInfo objects directly to insert. Minimal example:
val studentInfo: StudentInfo = ...
def collection: JSONCollection = db[JSONCollection]("student")
collection.insert(studentInfo)
That's the elegant part of the Play plugin. Yes, MongoDB persists data as JSON (or BSON, to be more precise), but you don't have do deal with it. Just make sure the implicit Writes (or Reads, in case of querying) is in scope, as well as other necessary imports (e.g. play.modules.reactivemongo.json._).
Now I need to get all data I have inserted. How can I do that? I am
not asking for code but for Idea.
Well, you want to have a look at the documentation (scroll down for examples), it's quite simple and there is not more to it. In your case, it could look like this:
// perform query via cursor
val cursor: Cursor[StudentInfo] =
collection.find(Json.obj("lastName" -> "Regmi")).cursor[StudentInfo]
// gather results as list
val futureStudents: Future[List[StudentInfo]] = cursor.collect[List]()
In this case, you get all students with the last name Regmi. If you really want to retrieve all students, then you probably need to pass an empty JsObject as your query. Again, it's not necessary to deal with JSON conversions, as long as the implicit Reads is in scope.
Here is the Complete Answer of my own Question
package controllers
def findAll=Action.async {
val cursor = Json.obj()
StudentDaoAndEntity.findAllStudent(cursor) map {
case Nil => Ok("Student Not Found")
case l:Seq[JsObject] => Ok(Json.toJson(l))
}
}
def findAllStudent(allStd: JsObject): Future[Seq[JsObject]] = {
// gather all the JsObjects in a list
collection.find(allStd).cursor[JsObject].collect[List]()
}
I have a simple single key-valued Map(K,V) myDictionary that is populated by my program and at the end I want to write it as JSON format string in a text file - as I would need parse them later.
I was using this code earlier,
Some(new PrintWriter(outputDir+"/myDictionary.json")).foreach{p => p.write(compact(render(decompose(myDictionary)))); p.close}
I found it to be slower as the input size increased. Later, I used this var out = new
var out = new PrintWriter(outputDir+"/myDictionary.json");
out.println(scala.util.parsing.json.JSONObject(myDictionary.toMap).toString())
This is proving to be bit faster.
I have run this for sample input and found that this is faster than my earlier approach. I assuming my input map size would reach at least a million values( >1GB text file) (K,V) hence I want to make sure that I follow the faster and memory efficient approach for Map serialization process.What are other approaches that you would recommend,that I can look into to optimize this.
The JSON support in the standard Scala library is probably not the best choice. Unfortunately the situation with JSON libraries for Scala is a bit confusing, there are many alternatives (Lift JSON, Play JSON, Spray JSON, Twitter JSON, Argonaut, ...), basically one library for each day of the week... I suggest you have a look at these at least to see if any of them is easier to use and more performative.
Here is an example using Play JSON which I have chosen for particular reasons (being able to generate formats with macros):
object JsonTest extends App {
import play.api.libs.json._
type MyDict = Map[String, Int]
implicit object MyDictFormat extends Format[MyDict] {
def reads(json: JsValue): JsResult[MyDict] = json match {
case JsObject(fields) =>
val b = Map.newBuilder[String, Int]
fields.foreach {
case (k, JsNumber(v)) => b += k -> v.toInt
case other => return JsError(s"Not a (string, number) pair: $other")
}
JsSuccess(b.result())
case _ => JsError(s"Not an object: $json")
}
def writes(m: MyDict): JsValue = {
val fields: Seq[(String, JsValue)] = m.map {
case (k, v) => k -> JsNumber(v)
} (collection.breakOut)
JsObject(fields)
}
}
val m = Map("hallo" -> 12, "gallo" -> 34)
val serial = Json.toJson(m)
val text = Json.stringify(serial)
println(text)
val back = Json.fromJson[MyDict](serial)
assert(back == JsSuccess(m), s"Failed: $back")
}
While you can construct and deconstruct JsValues directly, the main idea is to use a Format[A] where A is the type of your data structure. This puts more emphasis on type safety than the standard Scala-Library JSON. It looks more verbose, but in end I think it's the better approach.
There are utility methods Json.toJson and Json.fromJson which look for an implicit format of the type you want.
On the other hand, it does construct everything in-memory and it does duplicate your data structure (because for each entry in your map you will have another tuple (String, JsValue)), so this isn't necessarily the most memory efficient solution, given that you are operating in the GB magnitude...
Jerkson is a Scala wrapper for the Java JSON library Jackson. The latter apparently has the feature to stream data. I found this project which says it adds streaming support. Play JSON in turn is based on Jerkson, so perhaps you can even figure out how to stream your object with that. See also this question.
I'm using playframework 2.1.0 with Anorm to query a db.
I want to serialize the result to json without going through any interim objects/case classes.
this is what the flow looks like:
Using anorm:
DB.withConnection { implicit c =>
val q = SQL(""" long query goes here """)
q().toList
}
then I take this result and transform it from a List[SqlRow] to List[Map[String,Any]].
String,Any is the column name, value (Object/Any)
val asMap = info.toList.map(row => scala.collection.immutable.Map(row.asMap.toSeq:_*))
The i'd like to jsonize this.
I tried some json libs : GSON, spray-json, playframework json lib.
But none of them seem to work with Any out of the box.
I tried writing implicit writer for the Any type with some pattern matching, but the problem is that this writer always overtakes all the other writes so the json is not produced correctly.
Advise?
How would you suggest transforming a result from Anorm to Json? without any interim domain models.
found a solution, not the best, using FlexJson.
The annoying thing is that FlexJson is not very scala oriented so scala collections and some scala types need to be converted to the equivalent Java type.
val info:List[SqlRow] = loadInfoFromDB using Anorm
//transform scala maps to java maps
val asMap: List[util.Map[String, Any]] = info.toList.map(row => JavaConversions.mapAsJavaMap(row.asMap))
//create the basic FlexJson serializer
val flexJson: JSONSerializer = new flexjson.JSONSerializer()
//register a Option transformer so it can serialize Options correctly
flexJson.transform(new flexjson.transformer.AbstractTransformer {
def transform(`object`: Any) {
`object`.asInstanceOf[Option[_]] match {
case None => getContext.write("null")
case Some(b:Any) => getContext.transform(b)
}
}
},classOf[Option[_]])
//finally convert the scala List to java List and use this serializer on it.
val infoJsn: String = flexJson.deepSerialize(JavaConversions.seqAsJavaList(asMap))