I'm using playframework 2.1.0 with Anorm to query a db.
I want to serialize the result to json without going through any interim objects/case classes.
this is what the flow looks like:
Using anorm:
DB.withConnection { implicit c =>
val q = SQL(""" long query goes here """)
q().toList
}
then I take this result and transform it from a List[SqlRow] to List[Map[String,Any]].
String,Any is the column name, value (Object/Any)
val asMap = info.toList.map(row => scala.collection.immutable.Map(row.asMap.toSeq:_*))
The i'd like to jsonize this.
I tried some json libs : GSON, spray-json, playframework json lib.
But none of them seem to work with Any out of the box.
I tried writing implicit writer for the Any type with some pattern matching, but the problem is that this writer always overtakes all the other writes so the json is not produced correctly.
Advise?
How would you suggest transforming a result from Anorm to Json? without any interim domain models.
found a solution, not the best, using FlexJson.
The annoying thing is that FlexJson is not very scala oriented so scala collections and some scala types need to be converted to the equivalent Java type.
val info:List[SqlRow] = loadInfoFromDB using Anorm
//transform scala maps to java maps
val asMap: List[util.Map[String, Any]] = info.toList.map(row => JavaConversions.mapAsJavaMap(row.asMap))
//create the basic FlexJson serializer
val flexJson: JSONSerializer = new flexjson.JSONSerializer()
//register a Option transformer so it can serialize Options correctly
flexJson.transform(new flexjson.transformer.AbstractTransformer {
def transform(`object`: Any) {
`object`.asInstanceOf[Option[_]] match {
case None => getContext.write("null")
case Some(b:Any) => getContext.transform(b)
}
}
},classOf[Option[_]])
//finally convert the scala List to java List and use this serializer on it.
val infoJsn: String = flexJson.deepSerialize(JavaConversions.seqAsJavaList(asMap))
Related
I have a text file with json value. and this gets read into a DF
{"name":"Michael"}
{"name":"Andy", "age":30}
I want to infer the schema dynamically for each line while Streaming and store it in separate locations(tables) depending on its schema.
unfortunately while I try to read the value.schema it still shows as String. Please help on how to do it on Streaming as RDD is not allowed in streaming.
I wanted to use the following code which doesnt work as the value is still read as String format.
val jsonSchema = newdf1.select("value").as[String].schema
val df1 = newdf1.select(from_json($"value", jsonSchema).alias("value_new"))
val df2 = df1.select("value_new.*")
I even tried to use,
schema_of_json("json_schema"))
val jsonSchema: String = newdf.select(schema_of_json(col("value".toString))).as[String].first()
still no hope.. Please help..
You can load the data as textFile, create case class for person and parse every json string to Person instance using json4s or gson, then creating the Dataframe as follows:
case class Person(name: String, age: Int)
val jsons = spark.read.textFile("/my/input")
val persons = jsons.map{json => toPerson(json) //instead of 'toPerson' actually parse with json4s or gson to return Person instance}
val df = sqlContext.createDataFrame(persons)
Deserialize json to case class using json4s:
https://commitlogs.com/2017/01/14/serialize-deserialize-json-with-json4s-in-scala/
Deserialize json to case class using gson:
https://alvinalexander.com/source-code/scala/scala-case-class-gson-json-object-deserialization-and-scalatra
I am using Play Framework and I am trying to convert a Scala object to a JSON string.
Here is my code where I get my object:
val profile: Future[List[Profile]] = profiledao.getprofile(profileId);
The object is now in the profile value.
Now I want to convert that profile object which is a Future[List[Profile]] to JSON data and then convert that data into a JSON string then write into a file.
Here is the code that I wrote so far:
val jsondata = Json.toJson(profile)
Jackson.toJsonString(jsondata)
This is how I am trying to convert into JSON data but it is giving me the following output:
{"empty":false,"traversableAgain":true}
I am using the Jackson library to do the conversion.
Can someone help me with this ?
Why bother with Jackson? If you're using Play, you have play-json available to you, which uses Jackson under the hood FWIW:
First, you need an implicit Reads to let play-json know how to serialize Profile. If Profile is a case class, you can do this:
import play.api.libs.json._
implicit val profileFormat = Json.format[Profile]
If not, define your own Reads like this.
Then since getprofile (which should follow convention and be getProfile) returns Future[List[Profile]], you can do this to get a JsValue:
val profilesJson = profiledao.getprofile(profileId).map(toJson)
(profiledao should also be profileDao.)
In the end, you can wrap this in a Result like Ok and return that from your controller.
How can one send, using the Play! framework, a JSON response that is formatted to be human-readable?
For example, I'm looking for something like:
def handleGET(path:String) = Action{ implicit request =>
val json = doSomethingThatReturnsAJson(path,request)
request.getQueryString("pretty") match {
case Some(_) => //some magic that will beautify the response
case None => Ok(json)
}
}
My search led me to JSON pretty-print, which was not very helpful on it's own, but it did say the ability should be integrated in future versions. That was play 2.1.X, so, I guess it already exists somewhere in the 2.2X version of play?
Play framework has pretty printing support built-in:
import play.api.libs.json.Json
Json.prettyPrint(aJsValue)
So in your case, it would be sufficient to do the following:
def handleGET(path:String) = Action { implicit request =>
val json = doSomethingThatReturnsAJson(path, request)
request.getQueryString("pretty") match {
case Some(_) => Ok(Json.prettyPrint(json)).as(ContentTypes.JSON)
case None => Ok(json)
}
}
You can use Gson to pretty print Json string, don't know about scala; but here is a Java example which you can convert to scala and use it:
Gson gson = new GsonBuilder().setPrettyPrinting().create();
String jsonStr = gson.toJson(obj);
System.out.println(jsonStr);
I have a simple single key-valued Map(K,V) myDictionary that is populated by my program and at the end I want to write it as JSON format string in a text file - as I would need parse them later.
I was using this code earlier,
Some(new PrintWriter(outputDir+"/myDictionary.json")).foreach{p => p.write(compact(render(decompose(myDictionary)))); p.close}
I found it to be slower as the input size increased. Later, I used this var out = new
var out = new PrintWriter(outputDir+"/myDictionary.json");
out.println(scala.util.parsing.json.JSONObject(myDictionary.toMap).toString())
This is proving to be bit faster.
I have run this for sample input and found that this is faster than my earlier approach. I assuming my input map size would reach at least a million values( >1GB text file) (K,V) hence I want to make sure that I follow the faster and memory efficient approach for Map serialization process.What are other approaches that you would recommend,that I can look into to optimize this.
The JSON support in the standard Scala library is probably not the best choice. Unfortunately the situation with JSON libraries for Scala is a bit confusing, there are many alternatives (Lift JSON, Play JSON, Spray JSON, Twitter JSON, Argonaut, ...), basically one library for each day of the week... I suggest you have a look at these at least to see if any of them is easier to use and more performative.
Here is an example using Play JSON which I have chosen for particular reasons (being able to generate formats with macros):
object JsonTest extends App {
import play.api.libs.json._
type MyDict = Map[String, Int]
implicit object MyDictFormat extends Format[MyDict] {
def reads(json: JsValue): JsResult[MyDict] = json match {
case JsObject(fields) =>
val b = Map.newBuilder[String, Int]
fields.foreach {
case (k, JsNumber(v)) => b += k -> v.toInt
case other => return JsError(s"Not a (string, number) pair: $other")
}
JsSuccess(b.result())
case _ => JsError(s"Not an object: $json")
}
def writes(m: MyDict): JsValue = {
val fields: Seq[(String, JsValue)] = m.map {
case (k, v) => k -> JsNumber(v)
} (collection.breakOut)
JsObject(fields)
}
}
val m = Map("hallo" -> 12, "gallo" -> 34)
val serial = Json.toJson(m)
val text = Json.stringify(serial)
println(text)
val back = Json.fromJson[MyDict](serial)
assert(back == JsSuccess(m), s"Failed: $back")
}
While you can construct and deconstruct JsValues directly, the main idea is to use a Format[A] where A is the type of your data structure. This puts more emphasis on type safety than the standard Scala-Library JSON. It looks more verbose, but in end I think it's the better approach.
There are utility methods Json.toJson and Json.fromJson which look for an implicit format of the type you want.
On the other hand, it does construct everything in-memory and it does duplicate your data structure (because for each entry in your map you will have another tuple (String, JsValue)), so this isn't necessarily the most memory efficient solution, given that you are operating in the GB magnitude...
Jerkson is a Scala wrapper for the Java JSON library Jackson. The latter apparently has the feature to stream data. I found this project which says it adds streaming support. Play JSON in turn is based on Jerkson, so perhaps you can even figure out how to stream your object with that. See also this question.
I have a string that is a Json array of two objects.
> val ss = """[ {"key1" :"value1"}, {"key2":"value2"}]"""
I want to use the Play Json libraries to deserialize it and create a map from the key values to the objects.
def deserializeJsonArray(ss:String):Map[String, JsValue] = ???
// Returns Map("value1" -> {"key1" :"value1"}, "value2" -> {"key2" :"value2"})
How do I write the deserializeJsonArray function? This seems like it should be easy, but I can't figure it out from either the Play documentation or the REPL.
I'm a bit rusty, so please forgive the mess. Perhaps another overflower can come in here and clean it up for me.
This solution assumes that the JSON is an array of objects, and each of the objects contains exactly one key-value pair. I would highly recommend spicing it up with some error handling and/or pattern matching to validate the parsed JSON string.
def deserializeJsonArray(ss: String): Map[String, JsValue] = {
val jsObjectSeq: Seq[JsObject] = Json.parse(ss).as[Seq[JsObject]]
val jsValueSeq: Seq[JsValue] = Json.parse(ss).as[Seq[JsValue]]
val keys: Seq[String] = jsObjectSeq.map(json => json.keys.head)
(keys zip jsValueSeq).toMap
}