I have an object like this:
val aa = parse(""" { "vals" : [[1,2,3,4], [4,5,6,7], [8,9,6,3]] } """)
I want to access the value '1' in the first JArray.
println(aa.values ???)
How is this done?
Thanks
One way would be :
val n = (aa \ "vals")(0)(0).extract[Int]
println(n)
Another way is to parse the whole json using a case class :
implicit val formats = DefaultFormats
case class Numbers(vals: List[List[Int]])
val numbers = aa.extract[Numbers]
This way you can access the first value of the first list however you like :
for { list <- numbers.vals.headOption; hd <- list.headOption } println(hd)
// or
println(numbers.vals.head.head)
// or ...
Related
My requirement is to convert two string and create a JSON file(using spray JSON) and save in a resource directory.
one input string contains the ID and other input strings contain the score and topic
id = "alpha1"
inputstring = "science 30 math 24"
Expected output JSON is
{“ContentID”: “alpha1”,
“Topics”: [
{"Score" : 30, "TopicID" : "Science" },
{ "Score" : 24, "TopicID" : "math”}
]
}
below is the approach I have taken and am stuck in the last place
Define the case class
case class Topic(Score: String, TopicID: String)
case class Model(contentID: String, topic: Array[Topic])
implicit val topicJsonFormat: RootJsonFormat[Topic] = jsonFormat2(Topic)
implicit val modelJsonFormat: RootJsonFormat[Model] = jsonFormat2(Model)
Parsing the input string
val a = input.split(" ").zipWithIndex.collect{case(v,i) if (i % 2 == 0) =>
(v,i)}.map(_._1)
val b = input.split(" ").zipWithIndex.collect{case(v,i) if (i % 2 != 0) =>
(v,i)}.map(_._1)
val result = a.zip(b)
And finally transversing through result
paired foreach {case (x,y) =>
val tClass = Topic(x, y)
val mClassJsonString = Topic(x, y).toJson.prettyPrint
out1.write(mClassJsonString.toString)
}
And the file is generated as
{"Score" : 30, "TopicID" : "Science" }
{ "Score" : 24, "TopicID" : "math”}
The problem is I am not able to add the contentID as needed above.
Adding ContentId inside foreach is making contentID added multiple time.
You're calling toJson inside foreach creating strings and then you're appending it to buffer.
What you probably wanted to do is to create a class (ADT) hierarchy first and then serialize it:
val topics = paired.map(Topic)
//toArray might be not necessary if topics variable is already an array
val model = Model("alpha1", topics.toArray)
val json = model.toJson.prettyPrint
out1.write(json.toString)
I have the below code and the output from my program. However, I could not create a List of Json (desired output) given below. What kind changes do I need to do in the existing code?
case class Uiresult(AccountNo: String, Name: String)
val json = parse(jsonString)
val elements = (json \\ "_source").children
for (acct <- elements) {
val m = acct.extract[Source]
val res = write(Uiresult(m.accountNo, (m.firstName + m.lastName))
println(res)
}
Output from current program:
{"AccountNo":"1234","Name":"Augustin John"}
{"AccountNo":"1235","Name":"Juliet Paul"}
{"AccountNo":"1236","Name":"Sebastin Arul"}
Desired output:
[
{"AccountNo":"1234","Name":"Augustin John"},
{"AccountNo":"1235","Name":"Juliet Paul"},
{"AccountNo":"1236","Name":"Sebastin Arul"}
]
To create a list for the for comprehension, use the yield keyword. This will return the values from the iterations and create a list for you, which you then can assign to a val.
val list = for (acct <- elements) yield {
val m = acct.extract[Source]
val res = write(Uiresult(m.accountNo, (m.firstName + m.lastName))
res
}
This can be written even shorter,
val list = for (acct <- elements) yield {
val m = acct.extract[Source]
write(Uiresult(m.accountNo, (m.firstName + m.lastName))
}
The type of list (Array, List, Seq, etc.) will be determined by the type of elements. Other data structures such as set dictionaries are also possible to use in this way.
To print the output into the exact format as in the "desired output" above, use mkString.
println(list.mkString("[\n", ",\n", "\n]"))
Scenario:
I have following case class:
case class Student(firstName:String, lastName: String)
I need to write reads and writes for Student. The json I provide is Sequence of Student.
For Example:
{
"College": "Abc",
"student" : [{"firstName" : "Jack", "lastName":"Starc"},
{"firstName" : "Nicolas", "lastName":"Pooran"}
]
}
I have written my reads and writes as:
implicit val studentFormat = Json.format[Student]
implicit val studentRead = Json.reads[Student]
implicit val studentWrite = Json.writes[Student]
implicit val studentReadSeq = Reads.seq(studentRead)
implicit val studentWriteSeq = Writes.seq(studentWrite)
Now I have to make a type parser and check whether the student is array or simple object. Here key i.e. Student can be Students or StudentInfo. So I have to make a parser on the basis of value provided in json.
For this I have done as following:
def studentCheck(jsonValue: JsObject) = {
var modifiedJson = Json.obj()
for ((key, value) <- jsonValue.value) {
if(value.validate[Student].isSuccess ) {
val json =
studentFormat.writes(value.validate[Student].get).as[JsObject]
modifiedJson.+(key, json)
}
else if(studentReadSeq.reads(value).isSuccess) {
//My Code will be here
modifiedJson
}
else {
println("Error")
modifiedJson.+(key,value)
}
}
}
val studentJson = Json.obj(
"college" -> "ABC",
"student" -> Json.arr(
Json.obj("firstName" -> "Jack", "lastName" -> "Starc"),
Json.obj("firstName" -> "Nicolas", "entity" -> "Pooran")
)
)
studentCheck(studentJson)
The problem I get here is, even provided List of Students the first case i.e. if statement is executed instead of elseif. How can I validate so it satisfy all the condition, i.e if Student object is provided if statement is executed and if List of Student is provided elseif statement is executed.
You have a much better, safer, and functional way to validate a json.
Let's assume you have a College case class:
case class College(college: String, students: List[Student])
You can have a reader such as this:
object College{
implicit val collegeReads = Reads[College] = (
(JsPath \ "college").read[String] and
(JsPath \ "students").read[List[Student]
) (College.apply _)
}
Then in order to validate it, you can do:
def foo(jsonValue: JsObject)={
jsonValue.validate[College].fold(
errors => ,//handle parsing errors
collage => //your code when the parsing is successfull.
)
}
I have some response data in JSON format:
{ "items" :
[
{"commentId" : "28444760","userId" : "142607","userIP" : "","userName" : "Rock'n'Roll ","userAvatar" : "/i/default_userpic.png ","userIsBlock" : false,"userBanDate" : "",
"userCountry" : "http://ikor.ill.in.ua/f/UA.gif","date" : "16.02.2017, 17:07","text" : "txt","rate" : 2,"isLikeExist" : false,"childComments" : []}
]
}
and I want to parse it to lists.
For example, to extract commentId I use:
val js = parse(json)\\"items"
val commentId:List[String] = js\\"commentId"\ classOf[JString]
and I get list with id
when I tried parsing date I got:
List(16.02.2017, 17:24, 16.02.2017, 17:23)
How I can return the date list in the format List("date time")?
This is how you can parse date/times with a format specification:
def parseDT(s: String) = {
val fmt = "dd.MM.yyyy, HH:mm"
val df = java.time.format.DateTimeFormatter.ofPattern(fmt)
java.time.LocalDateTime.parse(s, df)
}
So after you get the dates from JSON (as strings) parse them:
val dates = datesFromJSON.map(parse(_))
If all you want is to remove the comma from the date strings, you can do
val dates = datesFromJSON.map(s => s.replaceAll(",",""))
The solution in my case is:
val dateFormat = new SimpleDateFormat("dd.MM.yyyy, HH:MM")
val date:List[java.util.Date] = (js\\"date").children.map(x=>dateFormat.parse(x.values.toString))
I have a json like :
{"name":"Yin", "address":[{"city":"Columbus","state":"Ohio"},{"city":"Columbus","state":"Ohio"}]}
{"name":"Michael", "address":[{"city":null, "state":"California"},{"city":null, "state":"California"}]}
here address is an array and if i use sqlContext.jsonfile i get the data in schema rdd as follows :
[Yin , [(Columbus , Ohio) , (Columbus , Ohio)]
[Micheal , [(null, California) , (null, California)]
I want to explode the array present and want the data in the following format in schema rdd :
[Yin, Columbus, Ohio]
[Yin, Columbus, Ohio]
[Micheal, null, California]
[Micheal, null, California]
I am using spark SQL
The typical suggestion is drop out of sql for this, but if you want to stay in SQL, here is an answer I got from asking this on the mailing list (nabble isn't showing the response for some reason):
From Michael Armbrust
You can do want with lateral view explode (using HiveContext), but what seems to be missing is that jsonRDD converts json objects into structs (fixed keys with a fixed order) and fields in a struct are accessed using a .
val myJson = sqlContext.jsonRDD(sc.parallelize("""{"foo":[{"bar":1},{"baz":2}]}""" :: Nil))
myJson.registerTempTable("JsonTest")
val result = sql("SELECT f.bar FROM JsonTest LATERAL VIEW explode(foo) a AS f").collect()
myJson: org.apache.spark.sql.DataFrame = [foo: array<struct<bar:bigint,baz:bigint>>]
result: Array[org.apache.spark.sql.Row] = Array([1], [null])
In Spark 1.3 you can also hint to jsonRDD that you'd like the json objects converted into Maps (non-uniform keys) instead of structs, by manually specifying the schema of your JSON.
import org.apache.spark.sql.types._
val schema =
StructType(
StructField("foo", ArrayType(MapType(StringType, IntegerType))) :: Nil)
sqlContext.jsonRDD(sc.parallelize("""{"foo":[{"bar":1},{"baz":2}]}""" :: Nil), schema).registerTempTable("jsonTest")
val withSql = sql("SELECT a FROM jsonTest LATERAL VIEW explode(foo) a AS a WHERE a['bar'] IS NOT NULL").collect()
val withSpark = sql("SELECT a FROM jsonTest LATERAL VIEW explode(foo) a AS a").rdd.filter {
case Row(a: Map[String, Int]) if a.contains("bar") => true
case _: Row => false
}.collect()
schema: org.apache.spark.sql.types.StructType = StructType(StructField(foo,ArrayType(MapType(StringType,IntegerType,true),true),true))
withSql: Array[org.apache.spark.sql.Row] = Array([Map(bar -> 1)])
withSpark: Array[org.apache.spark.sql.Row] = Array([Map(bar -> 1)])