First of all - sorry for stupid question.
I got some Json string from DB and want to parse all of them with json4s:
val df = sqlContext.sql("SELECT * FROM analytic.test").repartition(22)
val df_base = df.map(f => {
implicit val formats = DefaultFormats
val jsonString = f(5).toString
val tempJSON = parse(jsonString)
val mainJsonArray = tempJSON \ "events"
(
f(2).toString,
makeEventArray(mainJsonArray)
)
}).cache()
All good, i got Json's, but sometimes in DB occurs some failed Json, that take me to error :
com.fasterxml.jackson.core.JsonParseException: Unexpected end-of-input: was expecting closing '"' for name
First question - How can I evade this row with corrupt Json and continue my program?
I trying surround parse with try\catch, but in this case:
var tempJSON = json4s.JsonAST.JValue
try {
tempJSON = parse(f(5).toString)
} catch {
case e: Exception => println("Error on JSON parser. " + e )
}
But taking error:
Error:(51, 25) type mismatch;
found: org.json4s.JValue (which expands to) org.json4s.JsonAST.JValue
required: org.json4s.JsonAST.JValue.type tempJSON = parse(f(5).toString)
^
Second question - How to declare tempJson right?
Or I must validate Json before parse? How?
You can use Try for it:
val tempJSON = Try(parse(f(5).toString))
So now, you can match it:
tempJSON match {
case Success(yourValue) => ???
case Failure(exception) => println(exception.getMessage)
}
Or, if you don't need the exception, you would convert it to Option:
tempJSON.toOption
You'll get None or Some(value).
I don't know json4s but it probably exists, like in Play Json, a validate function returning something like a JsError or a JsSuccess.
Otherwise, an other way to go is to return an Option[JValue] (if you don't want to deal with exceptions), i.e. :
def function: Option[JValue] = {
try {
Option(parse(f(5).toString))
} catch {
case e: Exception => None
}
}
If you want to catch all parsing errors and silently skip them, which is risky and ill advised in most cases.
You can simply do:
df.flatMap(f => Try(do something).toOption)
You may however prefer, earlier validation preventing this to begin with, catching a more specific error only, logging errors etc.
Related
I have a server set up to send messages over a local host port. I am trying to decode the serialized json messages sent by the server and get this error.
Error decoding message: kotlinx.serialization.json.internal.JsonDecodingException: Unexpected JSON token at offset 55: Expected EOF after parsing, but had instead at path: $
JSON input: .....mber":13,"Timestamp":5769784} .....
The Racer State messages are formatted in JSON as follows: { “SensorId”: “value”, “RacerBibNumber” : “value”, “Timestamp” : “value” }, where the value’s are character string representations of the field values. I have also tried changing my RacerStatus Class to take String instead of Int but to a similar error. Am I missing something here? The symbol that is missing in the error was not able to be copied over so I know it's not UTF-8.
I have also added
val inputString = bytes.toString(Charsets.UTF_8)
println("Received input: $inputString")
This gets
Received input: {"SensorId":0,"RacerBibNumber":5254,"Timestamp":3000203}
with a bunch of extraneous symbols at the end.
data class RacerStatus(
var SensorId: Int,
var RacerBibNumber: Int,
var Timestamp: Int
) {
fun encode(): ByteArray {
return Json.encodeToString(serializer(), this).toByteArray()
}
companion object {
fun decode(bytes: ByteArray): RacerStatus {
print(bytes[0])
try {
val mstream = ByteArrayInputStream(bytes)
return Json.decodeFromStream<RacerStatus>(mstream)
} catch (e: SerializationException) {
println("Error decoding message: $e")
return RacerStatus(0, 0, 0)
}
// return Json.decodeFromString(serializer(), mstream.readBytes().toString())
}
}
}
So I found an answer to my question. I added a regex to include just the json components I know my json contains.
val str = bytes.toString(Charsets.UTF_8)
val re = Regex("[^A-Za-z0-9{}:,\"\"]")
return Json.decodeFromString<RacerStatus>(re.replace(str,""))
I thought that Charsets.UTF_8 would remove the misc characters but it did not. Is there a more intiuative solution? Also is there a regex that would cover all possible values in json?
The below code compiles, but throws an error: Exception in thread "main" scala.MatchError:[{"id":6430758,"name":...] (of class play.api.libs.json.JsArray). How can I read JSON for the given link by taking the items list in it and only 5 elements?
import play.api.libs.json._
def getProjects: List[Map[String, Any]] = {
val iter = getJSON("https://api.github.com/search/repositories?q=scala")
val json: JsValue = Json.parse(iter.get mkString "\n")
val projects = (json \ "items") match {
case l: List[Map[String, Any]] => l take 5
}
projects
}
def getJSON(url: String): Try[Iterator[String]] =
Try(Source.fromURL(url).getLines) recover {
case e: FileNotFoundException =>
throw new AppException(s"Requested page does not exist: ${e.getMessage}.")
case e: MalformedURLException =>
throw new AppException(s"Please make sure to enter a valid URL: ${e.getMessage}.")
case _ => throw new AppException("An unexpected error has occurred.")
}
Since you're using Play, you should work within its JsValue abstraction rather than jumping out to a Map[String, Any].
The reason your match is failing is because json \ "items" isn't a Map[String, Any], it's a JsValue. Ideally, you know the structure of your JSON (what your schema for project is) and you can deserialize to that:
case class Project(id: Long, name: String, ...)
object Project {
implicit val fmt = Json.format[Project]
}
val projects = WS.get("https://api.github.com/search/repositories?q=scala").map { response =>
response.json.validate[Map[String, Project]].map(_ take 5)
}
That leaves you with a Future[JsResult[Map[String, Project]]]. The outer type is Future because the operation is inherently asynchronous, JsResult will be either a JsSuccess with your Map[String, Project] or a JsError containing the reason(s) your JSON couldn't be validated.
It feels quick and dirty, but if that's really what you're wanting to do then you can try:
val listOfMaps: Seq[Map[String, String]] =
(res1 \ "items").as[JsArray].value.map { jsobj =>
jsobj.as[JsObject].value.map { case (key, value) =>
key -> value.toString
}
}.take(5)
A better option would be to create a case class with they keys and types that you are expecting and write a Reads to parse the Json to that case class. See https://www.playframework.com/documentation/2.3.x/ScalaJsonCombinators. Then you would have a list of your case class and you can easily take 5 from there.
Im trying to implement validation by using this article ScalaJsonCombinators
Basically i want to get the value if exist and if not return null
val nextPage: JsResult[JsValue] = (ch.\("paging").\("next")).validate[JsValue]
val nextUrl: String = nextPage match {
case s: JsSuccess[String] => s.get
case e: JsError => null
}
I have tow issue`s
the first is a warning
Warning:(99, 19) non-variable type argument String in type pattern play.api.libs.json.JsSuccess[String] is unchecked since it is eliminated by erasure
case s: JsSuccess[String] => s.get
^
the second is an error because the string is a URI with special
characters im getting a scheme error
val nextPage: JsResult[JsValue] = (ch.("paging").("next")).validate[JsValue]
val nextUrl: String = nextPage match {
case s: JsSuccess[String] => s.toString
case e: JsError => null
}
java.lang.IllegalArgumentException: Illegal character in scheme name at index 9: JsSuccess(https://graph.facebook.com/v2.2/396697410351933/feed?limit=200&access_token=623501864400497%7CGumAg3k6Eu
What will be to correct way to validate this element without using serialization.??
thanks,
miki
You're validating incorrectly. calling validate[JsValue] on a JsValue is meaningless, because it will always result in JsSuccess[JsValue]. The other problem is you're then trying to pattern match a JsSuccess[String] from a JsSuccess[JsValue], which you cannot do for many reasons. One, the type argument is erased at runtime as noted by the compiler warning. And two, a JsSuccess[String] can't be a JsSuccess[JsValue], the types aren't related.
What you really need is validate[String].
Probably impossible to debug unless you provide the relevant JSON.
There's a more elegant way of doing this, though. Since you don't seem to care about the failures (you're discarding them for null), you can just use asOpt. Here is where I say that if there's a chance there is no next, then nextUrl should be Option[String] instead of String, and you should never ever ever use null in Scala.
val nextUrl: Option[String] = (ch \ "paging" \ "next").asOpt[String]
If you for some reason must use null:
val nextUrl: String = (ch \ "paging" \ "next").asOpt[String].getOrElse(null)
I'm trying to make server which can handle HTTP requests with Json. Sorry if I haver. I'm just still new to all this.
So, I made function which takes JsValue and working with it.
def find(info: JsValue) = {
val req = Search.makeRequest("person",info)
val result = DB.withConnection { implicit c =>
SQL(req).as(person *)
}
Json.toJson(result)
}
Then I do something like this:
val test = Json.parse("""{"name":"John"}""")
Person.find(test)
It works fine. But then I try to call this function with HTTP request:
routes file:
GET /findperson controllers.PersonController.findPerson(info: String)
controller:
def findPerson(info: String) = Action {
Ok(Person.find(Json.parse(info)))
}
actual request:
http://localhost:9000/findperson?info="""{"name":"John"}"""
I get:
Exception: Malformed JSON: Got a sequence of JsValue outside an array or an object.
Can someone tell me how do it right? Please.
Although I agree with #Ryan that it is an unusual thing to want to do, I think the problem you are having is actually as the message says, "Malformed JSON". Remember that your url parameter is not source code, it's a simple string. There's no need to escape the quotation marks. So try using the url:
http://localhost:9000/findperson?info={"name":"John"}
JSON to Parse: http://www.dota2.com/jsfeed/heropickerdata?v=18874723138974056&l=english
Hero Class and JSON Serialization
case class Hero(
var id:Option[Int],
name: String,
bio: String,
var trueName:Option[String]
){}
implicit val modelReader: Reads[Hero] = Json.reads[Hero]
Reading Data
val future: Future[play.api.libs.ws.Response] = WS.url("http://www.dota2.com/jsfeed/heropickerdata?v=18874723138974056&l=english").get()
val json = Json.parse(Await.result(future,5 seconds).body).as[Map[String, Hero]]
var i = 1
json.foreach(p => {
p._2.trueName = Some(p._1)
p._2.id = Some(i)
p._2.commitToDatabase
i += 1
})
I need to get the id of each hero. The order of heros in the json matches their id. Obviously a map is unordered and wont work. Does anyone have any other ideas?
I have tried to use a LinkedHashMap. I even tried to make an implicit Reads for LinkedHashMap but I've failed. If anyone thinks that this is the answer then would you please give me some guidance?
It keeps just saying "No Json deserializer found for type scala.collection.mutable.LinkedHashMap[String,models.Hero]. Try to implement an implicit Reads or Format for this type.". I have the trait imported into the file i'm trying to read from. I have a funny feeling that the last line in my Reads is the problem. i think I can't just do the asInstanceOf, however I have no other ideas of how to do this reads.
LinkedHashMap Implicit Reads Code: http://pastebin.com/cf5NpSCX
You can try extracting data in order from the JsObject returned by Json.parse directly, possibly like this:
val json = Json.parse(Await.result(future,5 seconds).body)
val heroes: Map[String, Hero] = json match {
case obj: JsObject =>
obj.fields.zipWithIndex.map{ case ((name: String, heroJson: JsValue), id) =>
heroJson.asOpt[Hero].map{ _.copy(id = Some(id)) }
}.flatten.toMap
case _ = > Seq.empty
}
I don't believe you'll need an order-preserving map anymore since the ids are generated and fixed.