I could not find what is mistake in following code? - json

Scenario:
I have following case class:
case class Student(firstName:String, lastName: String)
I need to write reads and writes for Student. The json I provide is Sequence of Student.
For Example:
{
"College": "Abc",
"student" : [{"firstName" : "Jack", "lastName":"Starc"},
{"firstName" : "Nicolas", "lastName":"Pooran"}
]
}
I have written my reads and writes as:
implicit val studentFormat = Json.format[Student]
implicit val studentRead = Json.reads[Student]
implicit val studentWrite = Json.writes[Student]
implicit val studentReadSeq = Reads.seq(studentRead)
implicit val studentWriteSeq = Writes.seq(studentWrite)
Now I have to make a type parser and check whether the student is array or simple object. Here key i.e. Student can be Students or StudentInfo. So I have to make a parser on the basis of value provided in json.
For this I have done as following:
def studentCheck(jsonValue: JsObject) = {
var modifiedJson = Json.obj()
for ((key, value) <- jsonValue.value) {
if(value.validate[Student].isSuccess ) {
val json =
studentFormat.writes(value.validate[Student].get).as[JsObject]
modifiedJson.+(key, json)
}
else if(studentReadSeq.reads(value).isSuccess) {
//My Code will be here
modifiedJson
}
else {
println("Error")
modifiedJson.+(key,value)
}
}
}
val studentJson = Json.obj(
"college" -> "ABC",
"student" -> Json.arr(
Json.obj("firstName" -> "Jack", "lastName" -> "Starc"),
Json.obj("firstName" -> "Nicolas", "entity" -> "Pooran")
)
)
studentCheck(studentJson)
The problem I get here is, even provided List of Students the first case i.e. if statement is executed instead of elseif. How can I validate so it satisfy all the condition, i.e if Student object is provided if statement is executed and if List of Student is provided elseif statement is executed.

You have a much better, safer, and functional way to validate a json.
Let's assume you have a College case class:
case class College(college: String, students: List[Student])
You can have a reader such as this:
object College{
implicit val collegeReads = Reads[College] = (
(JsPath \ "college").read[String] and
(JsPath \ "students").read[List[Student]
) (College.apply _)
}
Then in order to validate it, you can do:
def foo(jsonValue: JsObject)={
jsonValue.validate[College].fold(
errors => ,//handle parsing errors
collage => //your code when the parsing is successfull.
)
}

Related

Creating JSON file with for loop in scala

My requirement is to convert two string and create a JSON file(using spray JSON) and save in a resource directory.
one input string contains the ID and other input strings contain the score and topic
id = "alpha1"
inputstring = "science 30 math 24"
Expected output JSON is
{“ContentID”: “alpha1”,
“Topics”: [
{"Score" : 30, "TopicID" : "Science" },
{ "Score" : 24, "TopicID" : "math”}
]
}
below is the approach I have taken and am stuck in the last place
Define the case class
case class Topic(Score: String, TopicID: String)
case class Model(contentID: String, topic: Array[Topic])
implicit val topicJsonFormat: RootJsonFormat[Topic] = jsonFormat2(Topic)
implicit val modelJsonFormat: RootJsonFormat[Model] = jsonFormat2(Model)
Parsing the input string
val a = input.split(" ").zipWithIndex.collect{case(v,i) if (i % 2 == 0) =>
(v,i)}.map(_._1)
val b = input.split(" ").zipWithIndex.collect{case(v,i) if (i % 2 != 0) =>
(v,i)}.map(_._1)
val result = a.zip(b)
And finally transversing through result
paired foreach {case (x,y) =>
val tClass = Topic(x, y)
val mClassJsonString = Topic(x, y).toJson.prettyPrint
out1.write(mClassJsonString.toString)
}
And the file is generated as
{"Score" : 30, "TopicID" : "Science" }
{ "Score" : 24, "TopicID" : "math”}
The problem is I am not able to add the contentID as needed above.
Adding ContentId inside foreach is making contentID added multiple time.
You're calling toJson inside foreach creating strings and then you're appending it to buffer.
What you probably wanted to do is to create a class (ADT) hierarchy first and then serialize it:
val topics = paired.map(Topic)
//toArray might be not necessary if topics variable is already an array
val model = Model("alpha1", topics.toArray)
val json = model.toJson.prettyPrint
out1.write(json.toString)

Scala Json - List of Json

I have the below code and the output from my program. However, I could not create a List of Json (desired output) given below. What kind changes do I need to do in the existing code?
case class Uiresult(AccountNo: String, Name: String)
val json = parse(jsonString)
val elements = (json \\ "_source").children
for (acct <- elements) {
val m = acct.extract[Source]
val res = write(Uiresult(m.accountNo, (m.firstName + m.lastName))
println(res)
}
Output from current program:
{"AccountNo":"1234","Name":"Augustin John"}
{"AccountNo":"1235","Name":"Juliet Paul"}
{"AccountNo":"1236","Name":"Sebastin Arul"}
Desired output:
[
{"AccountNo":"1234","Name":"Augustin John"},
{"AccountNo":"1235","Name":"Juliet Paul"},
{"AccountNo":"1236","Name":"Sebastin Arul"}
]
To create a list for the for comprehension, use the yield keyword. This will return the values from the iterations and create a list for you, which you then can assign to a val.
val list = for (acct <- elements) yield {
val m = acct.extract[Source]
val res = write(Uiresult(m.accountNo, (m.firstName + m.lastName))
res
}
This can be written even shorter,
val list = for (acct <- elements) yield {
val m = acct.extract[Source]
write(Uiresult(m.accountNo, (m.firstName + m.lastName))
}
The type of list (Array, List, Seq, etc.) will be determined by the type of elements. Other data structures such as set dictionaries are also possible to use in this way.
To print the output into the exact format as in the "desired output" above, use mkString.
println(list.mkString("[\n", ",\n", "\n]"))

Accessing a Single Value from Parsed JObject in Scala (Jackson, json4s)

I have an object like this:
val aa = parse(""" { "vals" : [[1,2,3,4], [4,5,6,7], [8,9,6,3]] } """)
I want to access the value '1' in the first JArray.
println(aa.values ???)
How is this done?
Thanks
One way would be :
val n = (aa \ "vals")(0)(0).extract[Int]
println(n)
Another way is to parse the whole json using a case class :
implicit val formats = DefaultFormats
case class Numbers(vals: List[List[Int]])
val numbers = aa.extract[Numbers]
This way you can access the first value of the first list however you like :
for { list <- numbers.vals.headOption; hd <- list.headOption } println(hd)
// or
println(numbers.vals.head.head)
// or ...

explode json array in schema rdd

I have a json like :
{"name":"Yin", "address":[{"city":"Columbus","state":"Ohio"},{"city":"Columbus","state":"Ohio"}]}
{"name":"Michael", "address":[{"city":null, "state":"California"},{"city":null, "state":"California"}]}
here address is an array and if i use sqlContext.jsonfile i get the data in schema rdd as follows :
[Yin , [(Columbus , Ohio) , (Columbus , Ohio)]
[Micheal , [(null, California) , (null, California)]
I want to explode the array present and want the data in the following format in schema rdd :
[Yin, Columbus, Ohio]
[Yin, Columbus, Ohio]
[Micheal, null, California]
[Micheal, null, California]
I am using spark SQL
The typical suggestion is drop out of sql for this, but if you want to stay in SQL, here is an answer I got from asking this on the mailing list (nabble isn't showing the response for some reason):
From Michael Armbrust
You can do want with lateral view explode (using HiveContext), but what seems to be missing is that jsonRDD converts json objects into structs (fixed keys with a fixed order) and fields in a struct are accessed using a .
val myJson = sqlContext.jsonRDD(sc.parallelize("""{"foo":[{"bar":1},{"baz":2}]}""" :: Nil))
myJson.registerTempTable("JsonTest")​
val result = sql("SELECT f.bar FROM JsonTest LATERAL VIEW explode(foo) a AS f").collect()
myJson: org.apache.spark.sql.DataFrame = [foo: array<struct<bar:bigint,baz:bigint>>]
result: Array[org.apache.spark.sql.Row] = Array([1], [null])
In Spark 1.3 you can also hint to jsonRDD that you'd like the json objects converted into Maps (non-uniform keys) instead of structs, by manually specifying the schema of your JSON.
import org.apache.spark.sql.types._
val schema =
StructType(
StructField("foo", ArrayType(MapType(StringType, IntegerType))) :: Nil)
​
sqlContext.jsonRDD(sc.parallelize("""{"foo":[{"bar":1},{"baz":2}]}""" :: Nil), schema).registerTempTable("jsonTest")
​
val withSql = sql("SELECT a FROM jsonTest LATERAL VIEW explode(foo) a AS a WHERE a['bar'] IS NOT NULL").collect()
​
val withSpark = sql("SELECT a FROM jsonTest LATERAL VIEW explode(foo) a AS a").rdd.filter {
case Row(a: Map[String, Int]) if a.contains("bar") => true
case _: Row => false
}.collect()
schema: org.apache.spark.sql.types.StructType = StructType(StructField(foo,ArrayType(MapType(StringType,IntegerType,true),true),true))
withSql: Array[org.apache.spark.sql.Row] = Array([Map(bar -> 1)])
withSpark: Array[org.apache.spark.sql.Row] = Array([Map(bar -> 1)])

Play framework - save data to database by parameters in URL

I'm new to the Play Framework, and Scala language. I want to save some data to database only by running URL with specified parameters.
For example I want to run url like:
/DeviceData?device_ID=1&insertDate=2013-01-01&windDirection=50&device_ID=1&insertDate=2013-01-02&windDirection=5
and after that in the database two new records would be inserted (with Device_ID, insertDate and windDirection).
Right now I'm trying to save only one record at once (I don't know how to read list of elements and save them) but event that it's not working. There is no error, it's just not inserted.
DeviceData model
case class DeviceData(data_ID: Long, device_ID: Long, insertDate: String, windDirection: Double)
object DeviceData{
var deviceDataList = new HashMap[Long, DeviceData]
var data_ID = 0L
def nextId(): Long = { data_ID += 1; data_ID}
def createDeviceData(device_ID: Long, insertDate: String, windDirection: Double) :Unit = {
DB.withConnection { implicit connection =>
SQL(
"""
INSERT INTO devicedata(device_ID, insertDate, windDirection)
VALUES ({device_ID}, {insertDate}, {windDirection})
"""
).
on("device_ID" -> device_ID, "insertDate" -> insertDate, "windDirection" -> windDirection).
executeInsert()
}
}
def list(): List[DeviceData] = { deviceDataList.values.toList }
}
DeviceDatas controller
object DeviceDatas extends Controller {
val deviceDataForm = Form(
tuple(
"device_ID" -> of[Long],
"insertDate" -> nonEmptyText,
"windDirection" -> of[Double]
)
)
def listDeviceData() = Action {
Ok(views.html.deviceData(DeviceData.list(), deviceDataForm))
}
def createDeviceData(device_ID: Long, insertDate: String, windDirection: Double) = Action { implicit request =>
deviceDataForm.bindFromRequest.fold(
errors => BadRequest(views.html.deviceData(DeviceData.list(), errors)),
{ case (device_ID, insertDate, windDirection) => {
DeviceData.createDeviceData(device_ID, insertDate, windDirection)
Redirect(routes.DeviceDatas.listDeviceData)
}
}
)
}
}
deviceData.scala.html - it's simple one, just to check if there is any new inserted record.
#(deviceDatas: List[DeviceData], deviceDataForm: Form[(Long, String, Double)])
#import helper._
#main("DeviceDatas"){
<h3>#deviceDatas.size DeviceData(s)</h3>
}
routes file for /deviceDatas
GET /deviceDatas controllers.DeviceDatas.listDeviceData
POST /deviceDatas controllers.DeviceDatas.createDeviceData(device_ID: Long, insertDate: String, windDirection: Double)
Could You help me with that how to insert the data into database, and if there is any possibility to put list of elements with few records to insert. Also what's the best way to insert DateTime (yyyy-MM-dd hh:mm:ss) into URL parameters in Play Framework? I'm stuck and I don't know how to do it.
UPDATED
Thanks Zim-Zam O'Pootertoot for the answer. Unfortunately I need to use parameters, because I'm sending the data through the router. But anyway one more thanks to You because I'll use json in the future.
I decided to not use List of parameter as I said before, but for one new record I'm sending one request (for example: to add 6 new records to the database I need to run 6 times URL on the router:
/DeviceData?device_ID=1&insertDate=2013-01-01&windDirection=50
And my problem was solved by changing the route file to:
GET /deviceDatas controllers.DeviceDatas.listDeviceData
GET /deviceDatas controllers.DeviceDatas.createDeviceData(device_ID: Long, insertDate: String, windDirection: Double)
To pass in data for multiple records, and also to pass in DateTime data, send the data in the request's json body instead of as url params
http://www.playframework.com/documentation/2.2.x/ScalaBodyParsers
http://www.playframework.com/documentation/2.2.x/ScalaJson
Action(parse.json) { implicit request =>
(request.body \ "records") match {
case arr: JsArray => arr.value.foreach(json => {
val deviceId = (json \ "device_ID").as[Long]
val date = (json \ "insertDate").as[String]
val windDirection = (json \ "windDirection").as[Double]
// insert data in database
})
case _ => throw new IllegalArgumentException("Invalid Json: records must be a JsArray")
}}
The json for your records might look something like
{"records" : [
{"device_ID" : 123, "insertDate" : "2014-03-01 12:00:00", "windDirection" : 123.45},
{"device_ID" : 456, "insertDate" : "2014-03-02 12:00:00", "windDirection" : 54.321}]}