I have some response data in JSON format:
{ "items" :
[
{"commentId" : "28444760","userId" : "142607","userIP" : "","userName" : "Rock'n'Roll ","userAvatar" : "/i/default_userpic.png ","userIsBlock" : false,"userBanDate" : "",
"userCountry" : "http://ikor.ill.in.ua/f/UA.gif","date" : "16.02.2017, 17:07","text" : "txt","rate" : 2,"isLikeExist" : false,"childComments" : []}
]
}
and I want to parse it to lists.
For example, to extract commentId I use:
val js = parse(json)\\"items"
val commentId:List[String] = js\\"commentId"\ classOf[JString]
and I get list with id
when I tried parsing date I got:
List(16.02.2017, 17:24, 16.02.2017, 17:23)
How I can return the date list in the format List("date time")?
This is how you can parse date/times with a format specification:
def parseDT(s: String) = {
val fmt = "dd.MM.yyyy, HH:mm"
val df = java.time.format.DateTimeFormatter.ofPattern(fmt)
java.time.LocalDateTime.parse(s, df)
}
So after you get the dates from JSON (as strings) parse them:
val dates = datesFromJSON.map(parse(_))
If all you want is to remove the comma from the date strings, you can do
val dates = datesFromJSON.map(s => s.replaceAll(",",""))
The solution in my case is:
val dateFormat = new SimpleDateFormat("dd.MM.yyyy, HH:MM")
val date:List[java.util.Date] = (js\\"date").children.map(x=>dateFormat.parse(x.values.toString))
Related
My requirement is to convert two string and create a JSON file(using spray JSON) and save in a resource directory.
one input string contains the ID and other input strings contain the score and topic
id = "alpha1"
inputstring = "science 30 math 24"
Expected output JSON is
{“ContentID”: “alpha1”,
“Topics”: [
{"Score" : 30, "TopicID" : "Science" },
{ "Score" : 24, "TopicID" : "math”}
]
}
below is the approach I have taken and am stuck in the last place
Define the case class
case class Topic(Score: String, TopicID: String)
case class Model(contentID: String, topic: Array[Topic])
implicit val topicJsonFormat: RootJsonFormat[Topic] = jsonFormat2(Topic)
implicit val modelJsonFormat: RootJsonFormat[Model] = jsonFormat2(Model)
Parsing the input string
val a = input.split(" ").zipWithIndex.collect{case(v,i) if (i % 2 == 0) =>
(v,i)}.map(_._1)
val b = input.split(" ").zipWithIndex.collect{case(v,i) if (i % 2 != 0) =>
(v,i)}.map(_._1)
val result = a.zip(b)
And finally transversing through result
paired foreach {case (x,y) =>
val tClass = Topic(x, y)
val mClassJsonString = Topic(x, y).toJson.prettyPrint
out1.write(mClassJsonString.toString)
}
And the file is generated as
{"Score" : 30, "TopicID" : "Science" }
{ "Score" : 24, "TopicID" : "math”}
The problem is I am not able to add the contentID as needed above.
Adding ContentId inside foreach is making contentID added multiple time.
You're calling toJson inside foreach creating strings and then you're appending it to buffer.
What you probably wanted to do is to create a class (ADT) hierarchy first and then serialize it:
val topics = paired.map(Topic)
//toArray might be not necessary if topics variable is already an array
val model = Model("alpha1", topics.toArray)
val json = model.toJson.prettyPrint
out1.write(json.toString)
I have a json string and I wast to be able to parse it to get the 'key' values.
jsonString = {"id":2279,
"name":"Test",
"description":null,
"tags":[],
"keys":[{
"key":"WI1MX6XAWSY03X8Y",
"flag":true},
{"key":"BK2Q18T8RSN6VODR",
"flag":false}]}
I want to be able to parse this string and get values for both the keys.
Currently I'm doing:
val details = JSON.parseFull(jsonString)
val keys = details.get.asInstanceOf[Map[String, Any]]("keys")
println(keys)
keys here is:
List(Map(key -> 3JP11GJ5OOGOVV5N, flag -> true), Map(key -> F49M347FOHYKBT9, flag -> false))
Please let me know how i can get both the 'key' values.
There is nothing related to JSON actually, you just have to do:
val keysValues = key.map(k => k("key"))
Scenario:
I have following case class:
case class Student(firstName:String, lastName: String)
I need to write reads and writes for Student. The json I provide is Sequence of Student.
For Example:
{
"College": "Abc",
"student" : [{"firstName" : "Jack", "lastName":"Starc"},
{"firstName" : "Nicolas", "lastName":"Pooran"}
]
}
I have written my reads and writes as:
implicit val studentFormat = Json.format[Student]
implicit val studentRead = Json.reads[Student]
implicit val studentWrite = Json.writes[Student]
implicit val studentReadSeq = Reads.seq(studentRead)
implicit val studentWriteSeq = Writes.seq(studentWrite)
Now I have to make a type parser and check whether the student is array or simple object. Here key i.e. Student can be Students or StudentInfo. So I have to make a parser on the basis of value provided in json.
For this I have done as following:
def studentCheck(jsonValue: JsObject) = {
var modifiedJson = Json.obj()
for ((key, value) <- jsonValue.value) {
if(value.validate[Student].isSuccess ) {
val json =
studentFormat.writes(value.validate[Student].get).as[JsObject]
modifiedJson.+(key, json)
}
else if(studentReadSeq.reads(value).isSuccess) {
//My Code will be here
modifiedJson
}
else {
println("Error")
modifiedJson.+(key,value)
}
}
}
val studentJson = Json.obj(
"college" -> "ABC",
"student" -> Json.arr(
Json.obj("firstName" -> "Jack", "lastName" -> "Starc"),
Json.obj("firstName" -> "Nicolas", "entity" -> "Pooran")
)
)
studentCheck(studentJson)
The problem I get here is, even provided List of Students the first case i.e. if statement is executed instead of elseif. How can I validate so it satisfy all the condition, i.e if Student object is provided if statement is executed and if List of Student is provided elseif statement is executed.
You have a much better, safer, and functional way to validate a json.
Let's assume you have a College case class:
case class College(college: String, students: List[Student])
You can have a reader such as this:
object College{
implicit val collegeReads = Reads[College] = (
(JsPath \ "college").read[String] and
(JsPath \ "students").read[List[Student]
) (College.apply _)
}
Then in order to validate it, you can do:
def foo(jsonValue: JsObject)={
jsonValue.validate[College].fold(
errors => ,//handle parsing errors
collage => //your code when the parsing is successfull.
)
}
I have an object like this:
val aa = parse(""" { "vals" : [[1,2,3,4], [4,5,6,7], [8,9,6,3]] } """)
I want to access the value '1' in the first JArray.
println(aa.values ???)
How is this done?
Thanks
One way would be :
val n = (aa \ "vals")(0)(0).extract[Int]
println(n)
Another way is to parse the whole json using a case class :
implicit val formats = DefaultFormats
case class Numbers(vals: List[List[Int]])
val numbers = aa.extract[Numbers]
This way you can access the first value of the first list however you like :
for { list <- numbers.vals.headOption; hd <- list.headOption } println(hd)
// or
println(numbers.vals.head.head)
// or ...
I have a json like :
{"name":"Yin", "address":[{"city":"Columbus","state":"Ohio"},{"city":"Columbus","state":"Ohio"}]}
{"name":"Michael", "address":[{"city":null, "state":"California"},{"city":null, "state":"California"}]}
here address is an array and if i use sqlContext.jsonfile i get the data in schema rdd as follows :
[Yin , [(Columbus , Ohio) , (Columbus , Ohio)]
[Micheal , [(null, California) , (null, California)]
I want to explode the array present and want the data in the following format in schema rdd :
[Yin, Columbus, Ohio]
[Yin, Columbus, Ohio]
[Micheal, null, California]
[Micheal, null, California]
I am using spark SQL
The typical suggestion is drop out of sql for this, but if you want to stay in SQL, here is an answer I got from asking this on the mailing list (nabble isn't showing the response for some reason):
From Michael Armbrust
You can do want with lateral view explode (using HiveContext), but what seems to be missing is that jsonRDD converts json objects into structs (fixed keys with a fixed order) and fields in a struct are accessed using a .
val myJson = sqlContext.jsonRDD(sc.parallelize("""{"foo":[{"bar":1},{"baz":2}]}""" :: Nil))
myJson.registerTempTable("JsonTest")
val result = sql("SELECT f.bar FROM JsonTest LATERAL VIEW explode(foo) a AS f").collect()
myJson: org.apache.spark.sql.DataFrame = [foo: array<struct<bar:bigint,baz:bigint>>]
result: Array[org.apache.spark.sql.Row] = Array([1], [null])
In Spark 1.3 you can also hint to jsonRDD that you'd like the json objects converted into Maps (non-uniform keys) instead of structs, by manually specifying the schema of your JSON.
import org.apache.spark.sql.types._
val schema =
StructType(
StructField("foo", ArrayType(MapType(StringType, IntegerType))) :: Nil)
sqlContext.jsonRDD(sc.parallelize("""{"foo":[{"bar":1},{"baz":2}]}""" :: Nil), schema).registerTempTable("jsonTest")
val withSql = sql("SELECT a FROM jsonTest LATERAL VIEW explode(foo) a AS a WHERE a['bar'] IS NOT NULL").collect()
val withSpark = sql("SELECT a FROM jsonTest LATERAL VIEW explode(foo) a AS a").rdd.filter {
case Row(a: Map[String, Int]) if a.contains("bar") => true
case _: Row => false
}.collect()
schema: org.apache.spark.sql.types.StructType = StructType(StructField(foo,ArrayType(MapType(StringType,IntegerType,true),true),true))
withSql: Array[org.apache.spark.sql.Row] = Array([Map(bar -> 1)])
withSpark: Array[org.apache.spark.sql.Row] = Array([Map(bar -> 1)])