I have a JSON string like this:
{"id":"111","name":"abc","ids":["740"],"data":"abc"}
I want to cut off the field "ids", however I don't know apriori the values like ["740"]. So, it might be e.g. ["888,222"] or whatever. The goal is to get the json string without the field "ids".
How to do it? Should I use JackMapper?
EDIT:
I tried to use JackMapper as JacksMapper.readValue[Map[String, String]](jsonString)to get only fields that I need. But the problem is that"ids":["740"]` throws the parsing error because it's an array. So, I decided to cut off this field before parsing, though it's an ugly solution and ideally I just want to parse the json string into Map.
Not sure what JackMapper is, but if other libraries are allowed, my personal favourites would be:
Play-JSON:
val jsonString = """{"id":"111","name":"abc","ids":["740"],"data":"abc"}"""
val json = Json.parse(jsonString).as[JsObject]
val newJson = json - "ids"
Circe:
import io.circe.parser._
val jsonString = """{"id":"111","name":"abc","ids":["740"],"data":"abc"}"""
val json = parse(jsonString).right.get.asObject.get // not handling errors
val newJson = json.remove("ids")
Note that this is the minimal example to get you going which doesn't handle bad input etc.
Related
override def accessToken(): ServiceCall[RequestTokenLogIn, Done] = {
request=>
val a=request.oauth_token.get
val b=request.oauth_verifier.get
val url=s"https://api.twitter.com/oauth/access_token?oauth_token=$a&oauth_verifier=$b"
ws.url(url).withMethod("POST").get().map{
res=>
println(res.body)
}
The output which I am getting on terminal is
oauth_token=xxxxxxxxx&oauth_token_secret=xxxxxxx&user_id=xxxxxxxxx&screen_name=xxxxx
I want to convert this response in json format.like
{
oauth_token:"",
token_secret:"",
}
When Calling res.json.toString its not converting into jsValue.
Is there any other way or am I missing something?
According to the documentation twitter publishes, it seems that the response is not a valid json. Therefore you cannot convert it automagically.
As I see it you have 2 options, which you are not going to like. In both options you have to do string manipulations.
The first option, which I like less, is actually building the json:
print(s"""{ \n\t"${res.body.replace("=", "\": \"").replace("&", "\"\n\t\"")}" \n}""")
The second option, is to extract the variables into a case class, and let play-json build the json string for you:
case class TwitterAuthToken(oauth_token: String, oauth_token_secret: String, user_id: Long, screen_name: String)
object TwitterAuthToken {
implicit val format: OFormat[TwitterAuthToken] = Json.format[TwitterAuthToken]
}
val splitResponse = res.body.split('&').map(_.split('=')).map(pair => (pair(0), pair(1))).toMap
val twitterAuthToken = TwitterAuthToken(
oauth_token = splitResponse("oauth_token"),
oauth_token_secret = splitResponse("oauth_token_secret"),
user_id = splitResponse("user_id").toLong,
screen_name = splitResponse("screen_name")
)
print(Json.toJsObject(twitterAuthToken))
I'll note that Json.toJsObject(twitterAuthToken) returns JsObject, which you can serialize, and deserialize.
I am not familiar with any option to modify the delimiters of the json being parsed by play-json. Given an existing json you can manipulate the paths from the json into the case class. But that is not what you are seeking for.
I am not sure if it is requires, but in the second option you can define user_id as long, which is harder in the first option.
I have a text file with json value. and this gets read into a DF
{"name":"Michael"}
{"name":"Andy", "age":30}
I want to infer the schema dynamically for each line while Streaming and store it in separate locations(tables) depending on its schema.
unfortunately while I try to read the value.schema it still shows as String. Please help on how to do it on Streaming as RDD is not allowed in streaming.
I wanted to use the following code which doesnt work as the value is still read as String format.
val jsonSchema = newdf1.select("value").as[String].schema
val df1 = newdf1.select(from_json($"value", jsonSchema).alias("value_new"))
val df2 = df1.select("value_new.*")
I even tried to use,
schema_of_json("json_schema"))
val jsonSchema: String = newdf.select(schema_of_json(col("value".toString))).as[String].first()
still no hope.. Please help..
You can load the data as textFile, create case class for person and parse every json string to Person instance using json4s or gson, then creating the Dataframe as follows:
case class Person(name: String, age: Int)
val jsons = spark.read.textFile("/my/input")
val persons = jsons.map{json => toPerson(json) //instead of 'toPerson' actually parse with json4s or gson to return Person instance}
val df = sqlContext.createDataFrame(persons)
Deserialize json to case class using json4s:
https://commitlogs.com/2017/01/14/serialize-deserialize-json-with-json4s-in-scala/
Deserialize json to case class using gson:
https://alvinalexander.com/source-code/scala/scala-case-class-gson-json-object-deserialization-and-scalatra
From a separate system I get a String parameter "messageJson" whose content is in the form:
{"agent1":"smith","agent2":"brown","agent3":{"agent3_1":"jones","agent3_2":"johnson"}}
To use it in my program I parse it with JsonSlurper.
def myJson = new JsonSlurper().parseText(messageJson)
But the resulting Json has the form:
[agent1:smith, agent2:brown, agent3:[agent3_1:jones, agent3_2:johnson]]
Note the square brackets and the lack of double quotes. How can I parse messageJson so that the original structure is kept?
Ok, thanks to the hint by cfrick, I was able to find a solution. In case anyone else has a similar problem, all I needed to do was using JsonOutput in the end to convert the map back to a Json
I.E. :
def myJson = new JsonSlurper().parseText(messageJson)
myJson << [agent4:"jane"]
def backToJson = JsonOutput.toJson(myJson)
I am using Play Framework and I am trying to convert a Scala object to a JSON string.
Here is my code where I get my object:
val profile: Future[List[Profile]] = profiledao.getprofile(profileId);
The object is now in the profile value.
Now I want to convert that profile object which is a Future[List[Profile]] to JSON data and then convert that data into a JSON string then write into a file.
Here is the code that I wrote so far:
val jsondata = Json.toJson(profile)
Jackson.toJsonString(jsondata)
This is how I am trying to convert into JSON data but it is giving me the following output:
{"empty":false,"traversableAgain":true}
I am using the Jackson library to do the conversion.
Can someone help me with this ?
Why bother with Jackson? If you're using Play, you have play-json available to you, which uses Jackson under the hood FWIW:
First, you need an implicit Reads to let play-json know how to serialize Profile. If Profile is a case class, you can do this:
import play.api.libs.json._
implicit val profileFormat = Json.format[Profile]
If not, define your own Reads like this.
Then since getprofile (which should follow convention and be getProfile) returns Future[List[Profile]], you can do this to get a JsValue:
val profilesJson = profiledao.getprofile(profileId).map(toJson)
(profiledao should also be profileDao.)
In the end, you can wrap this in a Result like Ok and return that from your controller.
I have a JsValue(spray.json.JsValue) which is in JSON format. I need to convert this JSON value to a MongodbObject using bson.util. How can I do this ?
You've probably got an answer for this already, but since i was just looking for a way to do the same conversion I thought I'd leave an answer. This worked for me.
You can convert it to a String using spray.json and use the JSON.parse provided by com.mongodb.util.JSON.
The trick is to remove the additional double quotes at the start and end of the string so that JSON.parse recognises it as a json object instead of a JSON String.
import spray.json._
import DefaultJsonProtocol._
val json = "{'foo':'baa'}"
val jsValue = json.toJson
val slicedJson = jsValue.toString().slice(1, jsValue.toString().length - 1)
val dbObject = JSON.parse(slicedJson).asInstanceOf[DBObject]