I need to convert a string json in an object using scala.
I tried this, but I cannot access the prop Name, for example
import scala.util.parsing.json._
val parsed = JSON.parseFull("""{"Name":"abc", "age":10}""")
How can I do to get an string json and convert to an object? Thanks
val parsed = JSON.parseFull("""{"Name":"abc", "age":10}""")
val parsed = JSON.parseFull(parsed)
try it
Related
I have a Binary json array string which I need to parse in Spark scala.
This is what I am doing:
import com.github.wnameless.json.flattener.JsonFlattener
def extractBlob(jsonString: String,
defaultValue: String = ""): String = {
JsonFlattener
.flatten(jsonString)
.toString
}
val extractBlobAsText = udf((jS: String) => extractBlob(jS))
val df1 =
df.withColumn("extracted",extractBlobAsText(unhex(col("hex").cast(StringType))))
df1.show(false)
But I am getting this error
com.eclipsesource.json.ParseException: Unexpected end of input at 1:1 at
com.eclipsesource.json.JsonParser.error(JsonParser.java:490) at
com.eclipsesource.json.JsonParser.expected(JsonParser.java:484) at
com.eclipsesource.json.JsonParser.readValue(JsonParser.java:193) at
com.eclipsesource.json.JsonParser.parse(JsonParser.java:152) at
com.eclipsesource.json.JsonParser.parse(JsonParser.java:91) at
com.eclipsesource.json.Json.parse(Json.java:295) at
com.github.wnameless.json.flattener.JsonFlattener.<init>(JsonFlattener.java:144) at
com.github.wnameless.json.flattener.JsonFlattener.flatten(JsonFlattener.java:100) at
notebook0.Cell5$285.extractBlob(Cell5:12) at
Also, I do not know the JSON schema. All I know is that it is a Json array. So I also need to do explode later.
I have a text file with json value. and this gets read into a DF
{"name":"Michael"}
{"name":"Andy", "age":30}
I want to infer the schema dynamically for each line while Streaming and store it in separate locations(tables) depending on its schema.
unfortunately while I try to read the value.schema it still shows as String. Please help on how to do it on Streaming as RDD is not allowed in streaming.
I wanted to use the following code which doesnt work as the value is still read as String format.
val jsonSchema = newdf1.select("value").as[String].schema
val df1 = newdf1.select(from_json($"value", jsonSchema).alias("value_new"))
val df2 = df1.select("value_new.*")
I even tried to use,
schema_of_json("json_schema"))
val jsonSchema: String = newdf.select(schema_of_json(col("value".toString))).as[String].first()
still no hope.. Please help..
You can load the data as textFile, create case class for person and parse every json string to Person instance using json4s or gson, then creating the Dataframe as follows:
case class Person(name: String, age: Int)
val jsons = spark.read.textFile("/my/input")
val persons = jsons.map{json => toPerson(json) //instead of 'toPerson' actually parse with json4s or gson to return Person instance}
val df = sqlContext.createDataFrame(persons)
Deserialize json to case class using json4s:
https://commitlogs.com/2017/01/14/serialize-deserialize-json-with-json4s-in-scala/
Deserialize json to case class using gson:
https://alvinalexander.com/source-code/scala/scala-case-class-gson-json-object-deserialization-and-scalatra
I am trying to get a map of columnName->values from a dataframe.I tried
val g=dataFrame.limit(limit)
val p=g.columns.map(i=>(i,g.select(i).map(_.get(0)).collect()))
and
val g=dataFrame.limit(limit)
val p=g.columns.map(i=>(i->g.select(i).map(_.get(0)).collect()))
But bot gives me an Array[String,Array[Any]]
I want to get a map[String,Array[Any]]
I also tried .toMap at the end to convert array to map,
val g=dataFrame.limit(limit)
val p=g.columns.map(i=>(i,g.select(i).map(_.get(0)).collect())).toMap
val gson=new Gson
gson.toJson(p)
but this gives me json string of the form
{"key1":"eq_site_deductible","value1":[0.0,0.0,0.0,],"key2":"county","value2":["CLAY COUNTY","CLAY COUNTY","Mary county"]}
I want to get a json string of the form {"eq_site_deductible":[value array],"county":[value array]}
If you just need the json, you don't need to convert it into map.
Below snippet can be used to write the dataframe content into a json file
dataFrame.write.format("json").save("result.json")
Or if you need the json string to be processed further in your code, you can use dataframe.toJSON to get the RDD[String], in which String will be the json
I have json string. I converted it to JValue using net.liftweb.JsonParser
val x : JValue = parse(json)
Then i modified the value of a field called "name" using replace()
x.replace("name" :: Nil, JString("Tim"))
Question is how do i convert this JValue back to a json String
You can simply use this
import net.liftweb.json._
compact(render(x))
Which will give you a json string version of the JValue object in this form
String = {"name":"Tim"}
In latest version 3.3.0 as of now in 2018 use below to convert from JsonAST.JValue to json string:
import net.liftweb.json._
compactRender(jValue)
I have a JsValue(spray.json.JsValue) which is in JSON format. I need to convert this JSON value to a MongodbObject using bson.util. How can I do this ?
You've probably got an answer for this already, but since i was just looking for a way to do the same conversion I thought I'd leave an answer. This worked for me.
You can convert it to a String using spray.json and use the JSON.parse provided by com.mongodb.util.JSON.
The trick is to remove the additional double quotes at the start and end of the string so that JSON.parse recognises it as a json object instead of a JSON String.
import spray.json._
import DefaultJsonProtocol._
val json = "{'foo':'baa'}"
val jsValue = json.toJson
val slicedJson = jsValue.toString().slice(1, jsValue.toString().length - 1)
val dbObject = JSON.parse(slicedJson).asInstanceOf[DBObject]