override def accessToken(): ServiceCall[RequestTokenLogIn, Done] = {
request=>
val a=request.oauth_token.get
val b=request.oauth_verifier.get
val url=s"https://api.twitter.com/oauth/access_token?oauth_token=$a&oauth_verifier=$b"
ws.url(url).withMethod("POST").get().map{
res=>
println(res.body)
}
The output which I am getting on terminal is
oauth_token=xxxxxxxxx&oauth_token_secret=xxxxxxx&user_id=xxxxxxxxx&screen_name=xxxxx
I want to convert this response in json format.like
{
oauth_token:"",
token_secret:"",
}
When Calling res.json.toString its not converting into jsValue.
Is there any other way or am I missing something?
According to the documentation twitter publishes, it seems that the response is not a valid json. Therefore you cannot convert it automagically.
As I see it you have 2 options, which you are not going to like. In both options you have to do string manipulations.
The first option, which I like less, is actually building the json:
print(s"""{ \n\t"${res.body.replace("=", "\": \"").replace("&", "\"\n\t\"")}" \n}""")
The second option, is to extract the variables into a case class, and let play-json build the json string for you:
case class TwitterAuthToken(oauth_token: String, oauth_token_secret: String, user_id: Long, screen_name: String)
object TwitterAuthToken {
implicit val format: OFormat[TwitterAuthToken] = Json.format[TwitterAuthToken]
}
val splitResponse = res.body.split('&').map(_.split('=')).map(pair => (pair(0), pair(1))).toMap
val twitterAuthToken = TwitterAuthToken(
oauth_token = splitResponse("oauth_token"),
oauth_token_secret = splitResponse("oauth_token_secret"),
user_id = splitResponse("user_id").toLong,
screen_name = splitResponse("screen_name")
)
print(Json.toJsObject(twitterAuthToken))
I'll note that Json.toJsObject(twitterAuthToken) returns JsObject, which you can serialize, and deserialize.
I am not familiar with any option to modify the delimiters of the json being parsed by play-json. Given an existing json you can manipulate the paths from the json into the case class. But that is not what you are seeking for.
I am not sure if it is requires, but in the second option you can define user_id as long, which is harder in the first option.
In my Play+Scala (2.5.x, 2.11.11) application I am using Case classes to convert JSON to and from (client<->application and application<->DB). Using inbuilt Reads/Writes the conversion becomes completely automatic.
The issue comes when DB returns a null value for a field and the corresponding JSON output does not include that key:value pair. For example:
def getUserDetails (id: Option[String]) = Action.async {
dbch.dbProd.withConnection { implicit connection =>
val result: Option[SignupUserDetails] =
SQL"""select * from #$USR_LISTUSERS where (User_Pk = ${id})""".as(SignupUserDetailsParser.singleOpt)
......
}
}
Case class:
case class SignupUserDetails(User_Pk: Option[String], Emailid: Option[String], First_Name: Option[String], Last_Name: Option[String], Country: Option[String])
Parser:
implicit val SignupUserDetailsParser = Macro.namedParser[SignupUserDetails]
I found a similar question where it is mentioned that this behaviour should occur only for custom types but not for String type:
Explicitly output JSON null in case of missing optional value
But it does not work for String type as well.
I am trying to programmatically enforce schema(json) on textFile which looks like json. I tried with jsonFile but the issue is for creating a dataframe from a list of json files, spark has to do a 1 pass through the data to create a schema for the dataframe. So it needs to parse all the data which is taking longer time (4 hours since my data is zipped and of size TBs). So I want to try reading it as textFile and enforce schema to get interested fields alone to later query on the resulting data frame. But I am not sure how do I map it to the input. Can some give me some reference on how do I map schema to json like input.
input :
This is the full schema :
records: org.apache.spark.sql.DataFrame = [country: string, countryFeatures: string, customerId: string, homeCountry: string, homeCountryFeatures: string, places: array<struct<freeTrial:boolean,placeId:string,placeRating:bigint>>, siteName: string, siteId: string, siteTypeId: string, Timestamp: bigint, Timezone: string, countryId: string, pageId: string, homeId: string, pageType: string, model: string, requestId: string, sessionId: string, inputs: array<struct<inputName:string,inputType:string,inputId:string,offerType:string,originalRating:bigint,processed:boolean,rating:bigint,score:double,methodId:string>>]
But I am only interested in few fields like :
res45: Array[String] = Array({"requestId":"bnjinmm","siteName":"bueller","pageType":"ad","model":"prepare","inputs":[{"methodId":"436136582","inputType":"US","processed":true,"rating":0,"originalRating":1},{"methodId":"23232322","inputType":"UK","processed":falase,"rating":0,"originalRating":1}]
val records = sc.textFile("s3://testData/sample.json.gz")
val schema = StructType(Array(StructField("requestId",StringType,true),
StructField("siteName",StringType,true),
StructField("model",StringType,true),
StructField("pageType",StringType,true),
StructField("inputs", ArrayType(
StructType(
StructField("inputType",StringType,true),
StructField("originalRating",LongType,true),
StructField("processed",BooleanType,true),
StructField("rating",LongType,true),
StructField("methodId",StringType,true)
),true),true)))
val rowRDD = ??
val inputRDD = sqlContext.applySchema(rowRDD, schema)
inputRDD.registerTempTable("input")
sql("select * from input").foreach(println)
Is there any way to map this ? Or do I need to use son parser or something. I want to use textFile only because of the constraints.
Tried with :
val records =sqlContext.read.schema(schema).json("s3://testData/test2.gz")
But keeping getting the error :
<console>:37: error: overloaded method value apply with alternatives:
(fields: Array[org.apache.spark.sql.types.StructField])org.apache.spark.sql.types.StructType <and>
(fields: java.util.List[org.apache.spark.sql.types.StructField])org.apache.spark.sql.types.StructType <and>
(fields: Seq[org.apache.spark.sql.types.StructField])org.apache.spark.sql.types.StructType
cannot be applied to (org.apache.spark.sql.types.StructField, org.apache.spark.sql.types.StructField, org.apache.spark.sql.types.StructField, org.apache.spark.sql.types.StructField, org.apache.spark.sql.types.StructField, org.apache.spark.sql.types.StructField)
StructField("inputs",ArrayType(StructType(StructField("inputType",StringType,true), StructField("originalRating",LongType,true), StructField("processed",BooleanType,true), StructField("rating",LongType,true), StructField("score",DoubleType,true), StructField("methodId",StringType,true)),true),true)))
^
It can load with following code with predefined schema, spark don't need to go through the file in ZIP file. The code in the question has ambiguity.
import org.apache.spark.sql.types._
val input = StructType(
Array(
StructField("inputType",StringType,true),
StructField("originalRating",LongType,true),
StructField("processed",BooleanType,true),
StructField("rating",LongType,true),
StructField("score",DoubleType,true),
StructField("methodId",StringType,true)
)
)
val schema = StructType(Array(
StructField("requestId",StringType,true),
StructField("siteName",StringType,true),
StructField("model",StringType,true),
StructField("inputs",
ArrayType(input,true),
true)
)
)
val records =sqlContext.read.schema(schema).json("s3://testData/test2.gz")
Not all the fields need to be provided. While it's good to provide all if possible.
Spark try best to parse all, if some row is not valid. It will add _corrupt_record as a column which contains the whole row.
While if it's plained json file file.
There is a http response which is a JSON string
{"id":"12345","dob":"01\/01\/1991","first_name":"Joe","gender":"male"}
Which needs to be instantiated into this class
case class UserRow(id: Long, firstName: String, lastName: String, dob: Long, gender: String)
I tried parsing the JSON into a map
val result = parseFull(response)
println(result)
Output
Some(Map(dob -> 01/01/1991, id -> 12345, first_name -> Joe, gender -> male))
Trying to get
map.get("id").toString().toLong //Throws a NumberFormatException
Dob should be converted to millis (EPOC) of type Long. Help is appreciated
Basic Answer
You can use https://github.com/json4s/json4s or another JSON-lib. And use a serializer.
I needed to adapt some parts of the JSON and the case class.
The id now really is a number
Fields that are optional (lastName) is not provided can be made optional by changing the type to Option in the case class
the names need to match exactly : last_name -> lastName
For handling the time in the dob field you could try this extension:
// Joda Time
implicit val formats = org.json4s.DefaultFormats ++ org.json4s.ext.JodaTimeSerializers.all
Some code examples
import org.json4s._
import org.json4s.native.Serialization
implicit val formats = Serialization.formats(NoTypeHints)
val jsonExample = """{"id":12345,"firstName":"Joe","gender":"male"}"""
case class UserRow(id: Long, firstName: String, lastName: Option[String], dob: Option[Long], gender: String)
Usage Example
scala> Serialization.read[UserRow](jsonExample)
res5: UserRow = UserRow(12345,Joe,None,None,male)
Extended Answer
The JSON provided leads to several problems, that could only be solved using a handcrafted deserializer https://github.com/json4s/json4s#serialization
How does I deserialise data like this into a case class like this:
case class SoundCloudUser (
id: Int,
permalink: String,
username: String,
country: String,
full_name: String,
city: String,
description: String)
(that is, where the case class has less constructor arguments than the JSON has values)
I tried creating a FieldSerializer to do this, but I could only work out how to ignore fields when serialising, not deserialising.
As long as the fields in the JSON data are a superset of the fields in your case class, you don't need to do anything special to ignore the fields in the JSON data that aren't in your case class. It should "just work". Are you getting any errors?