If I try Http Get Response {"ReturnValue":""},
this Code make error.
Caused by: spray.json.DeserializationException: Expected List as
JsArray, but got {"ReturnValue":""}
import spray.httpx.SprayJsonSupport._
import spray.json.DefaultJsonProtocol
import spray.http._
import spray.client.pipelining._
import scala.concurrent.duration._
import scala.concurrent.{ Await, Future }
import akka.actor.ActorSystem
import scala.concurrent.ExecutionContext.Implicits.global
class ApiHelper extends DefaultJsonProtocol {
case class Robot(name: String, color: Option[String], amountOfArms: Int)
implicit val RobotFormat = jsonFormat3(Robot)
def CallAPI(httpMethod: String, subURL: String): String = {
val apiLocation = "~~~"
val timeout = 5.seconds
implicit val system = ActorSystem("robotClient")
return httpMethod match {
case "GET" =>
val pipeline: HttpRequest => Future[List[Robot]] = sendReceive ~> unmarshal[List[Robot]]
val f: Future[List[Robot]] = pipeline(Get(s"$apiLocation"+subURL))
val robots = Await.result(f, timeout)
println(s"Got the list of robots: $robots")
return "hello"
}
}
}
Caused by: spray.json.DeserializationException: Expected List as JsArray, but got {"ReturnValue":""} at
spray.json.package$.deserializationError(package.scala:23) at
spray.json.CollectionFormats$$anon$1.read(CollectionFormats.scala:29)
at
spray.json.CollectionFormats$$anon$1.read(CollectionFormats.scala:25)
at
spray.httpx.SprayJsonSupport$$anonfun$sprayJsonUnmarshaller$1.applyOrElse(SprayJsonSupport.scala:37)
at
spray.httpx.SprayJsonSupport$$anonfun$sprayJsonUnmarshaller$1.applyOrElse(SprayJsonSupport.scala:34)
at
scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at
spray.httpx.unmarshalling.Unmarshaller$$anon$1$$anonfun$unmarshal$1.apply(Unmarshaller.scala:29)
at
spray.httpx.unmarshalling.SimpleUnmarshaller.protect(SimpleUnmarshaller.scala:40)
at
spray.httpx.unmarshalling.Unmarshaller$$anon$1.unmarshal(Unmarshaller.scala:29)
at
spray.httpx.unmarshalling.SimpleUnmarshaller.apply(SimpleUnmarshaller.scala:29)
at
spray.httpx.unmarshalling.SimpleUnmarshaller.apply(SimpleUnmarshaller.scala:23)
at
spray.httpx.unmarshalling.UnmarshallerLifting$$anon$3.apply(UnmarshallerLifting.scala:35)
at
spray.httpx.unmarshalling.UnmarshallerLifting$$anon$3.apply(UnmarshallerLifting.scala:34)
at
spray.httpx.unmarshalling.UnmarshallerLifting$$anon$2.apply(UnmarshallerLifting.scala:30)
at
spray.httpx.unmarshalling.UnmarshallerLifting$$anon$2.apply(UnmarshallerLifting.scala:29)
at
spray.httpx.unmarshalling.package$PimpedHttpResponse.as(package.scala:51)
at
spray.httpx.ResponseTransformation$$anonfun$unmarshal$1.apply(ResponseTransformation.scala:33)
... 13 more
Is there any way to get Json Object?
You can provide and use your own implementation of unmarshal which would construct JsValue instead of List[Robot]. JsValue would represent either valid response (list of robots) or arbitrary json response (or probably more custom object types).
def unmarshal: HttpResponse ⇒ JsValue =
response ⇒
if (response.status.isSuccess)
response.as[List[Robot]] match {
case Right(value) ⇒ value.toJson
case Left(error: MalformedContent) ⇒
response.as[JsObject] match {
case Right(value) ⇒ value.toJson
case Left(error) => throw new PipelineException(error.toString)
}
case Left(error) ⇒ throw new PipelineException(error.toString)
}
else throw new UnsuccessfulResponseException(response.status)
After the future (call to pipeline) returns JsValue you can try to convert it back again to List[Robot] in a controlled way (e.g. within a Try block) and in case of failure handle it as a custom json response.
Related
I'm new to Akka HTTP and I want to get rid of unnecessary fields from a JSON response and take only the necessary fields. For example, I use this endpoint to get the response and it contains a bunch of fields. For the moment I only need 'name' and 'versions'. I would like to know how to deserialize this into a case class containing only 'name' and 'versions'. I coded the following lines to get the response as a string.
import akka.http.scaladsl.Http
import akka.http.scaladsl.model.{HttpRequest, HttpResponse}
import akka.stream.scaladsl.{Flow, Sink, Source}
import akka.stream.{ActorMaterializer, OverflowStrategy}
import scala.concurrent.Future
import scala.concurrent.duration.DurationInt
import scala.language.postfixOps
import scala.util.{Failure, Success}
object SoftwareRegistry extends App {
implicit val system = ActorSystem("NPMRegistry")
implicit val materializer = ActorMaterializer()
import system.dispatcher
case class NPMPackage(name: String)
// reading the packages
val filename = "B:\\Scala\\NPMRegistry\\src\\main\\resources\\packages.txt"
val bufferedSource = scala.io.Source.fromFile(filename)
val listOfPackages: List[NPMPackage] = (for (line <- bufferedSource.getLines) yield {
NPMPackage(line.trim)
}).toList
bufferedSource.close()
// source
val sourceList = Source(listOfPackages)
// sink
val sink = Sink.foreach[NPMPackage] { p =>
// https request
val responseFuture: Future[HttpResponse] =
Http().singleRequest(HttpRequest(uri = s"https://registry.npmjs.org/${p.name}"))
val x = responseFuture
.flatMap(_.entity.toStrict(2 seconds))
.map(_.data.utf8String)
x.onComplete {
case Success(res) => println(res)
case Failure(_) => sys.error("Something went wrong")
}
}
// flow to slow things down and streaming sink to time-delayed operations
val bufferedFlow = Flow[NPMPackage]
.buffer(10, overflowStrategy = OverflowStrategy.backpressure)
.throttle(1, 3 seconds)
sourceList.async
.via(bufferedFlow).async
.to(sink)
.run()
}
And it prints the following output
For parsing json you need to use some library. In akka-http docs they use spray-json. Add the following dependency to your build.sbt with appropriate akkaHttpVersion.
"com.typesafe.akka" %% "akka-http-spray-json" % akkaHttpVersion
Now you need serializers and deserializers for your data. I am using a simple model, change it as needed.
trait Formatter extends DefaultJsonProtocol {
implicit object jsonFormat extends JsonFormat[Versions] {
override def read(json: JsValue): Versions = json match {
case JsObject(fields) =>
Versions(fields.keys.toList)
}
override def write(obj: Versions): JsValue = JsonParser(obj.toString)
}
implicit val formatterPackage: RootJsonFormat[Package] = jsonFormat2(Package)
case class Package(name: String, versions: Versions)
case class Versions(versions: List[String])
}
Finally sink:
//needed import with others
import spray.json._
object SoftwareRegistry extends App with Formatter {
//existing code
//---------
val sink = Sink.foreach[NPMPackage] { p =>
// https request
val responseFuture: Future[HttpResponse] =
Http().singleRequest(HttpRequest(uri = s"https://registry.npmjs.org/${p.name}"))
val packages = responseFuture
.flatMap(
_.entity
.dataBytes
.via(JsonFraming.objectScanner(Int.MaxValue))
.map(_.utf8String)
.map(_.parseJson.convertTo[Package])
.toMat(Sink.seq)(Keep.right)
.run()
)
packages.onComplete {
case Success(res) => println(res)
case Failure(_) => sys.error("Something went wrong")
}
}
//existing code
//---------
}
I have encountered a weird situation.
I m trying to build a method that takes a type and a JSON
and build it into a case class instance and if needed auto-complete missing key values.
So far I managed to do everything separately but not altogether.
The case class with its defaults:
case class Foo(a: String = "empty String", b: Option[Int] = Some(1))
and when I do the conversion:
import io.circe.generic.extras.auto._
import io.circe.generic.extras.Configuration
import io.circe.parser.decode
implicit val customConfig: Configuration = Configuration.default.withDefaults
println(decode[Foo]("{}"))
this is the output I get:
Right(Foo(empty String,Some(1)))
and this is working as I expected
but when I put it into a generic method it required a to be an option due to the error:
Exception in thread "main" java.lang.ExceptionInInitializerError
Caused by: DecodingFailure(Attempt to decode value on failed cursor, List(DownField(a)))
so I`m changing the case class to be
case class Foo(a: Option[String] = Some("empty String"), b: Option[Int] = Some(1))
and add the decoder:
object Foo{
implicit val decoder:Decoder[Foo] = deriveDecoder[Foo]
}
to the method:
import io.circe.Decoder
import io.circe.parser.decode
def convertToObj[T](jsonStr: String)(implicit decoder: Decoder[T]): T = {
decode[T](jsonStr)
match {
case Right(value) => value
case Left(error) => throw error
}
}
println(convertToObj[Foo]("{}"))
and the output is:
Foo(None,None)
so now I have lost my default values that I put and not able to use the automatic decoder as well.
How can I combine my two wishes into one approach?
You would need to do something like:
package foo.bar
import io.circe.Decoder
import io.circe.generic.extras.semiauto
import io.circe.generic.extras.Configuration
import io.circe.parser.decode
case class Foo(a: String = "empty String", b: Option[Int] = Some(1))
object Foo {
implicit val customConfig: Configuration = Configuration.default.withDefaults
implicit val decoder: Decoder[Foo] = semiauto.deriveConfiguredDecoder[Foo]
}
object TestApp extends App {
def convertToObj[T](jsonStr: String)(implicit decoder: Decoder[T]): T =
decode[T](jsonStr) match {
case Right(value) => value
case Left(error) => throw error
}
println(convertToObj[Foo]("{}"))
}
However, you can have circe automatically derive your decoder for you, so you can get away with less boilerplate:
package foo.bar
import io.circe.Decoder
import io.circe.generic.extras.auto._
import io.circe.generic.extras.Configuration
import io.circe.parser.decode
case class Foo(a: String = "empty String", b: Option[Int] = Some(1))
object TestApp extends App {
implicit val customConfig: Configuration = Configuration.default.withDefaults
def convertToObj[T](jsonStr: String)(implicit decoder: Decoder[T]): T =
decode[T](jsonStr) match {
case Right(value) => value
case Left(error) => throw error
}
println(convertToObj[Foo]("{}"))
}
Both of these examples give me output: Foo(empty String,Some(1))
NOTE:
method deriveDecoder in object semiauto is deprecated (since 0.12.0): Use deriveConfiguredDecoder
I am not much familier with spray json, but I have to convert the below json into Array[myTest]
Below is the code, but it doesnt work. It throws the following errors: How do I fix them?
Error:(19, 54) Cannot find JsonReader or JsonFormat type class for Array[A$A61.this.myTest]
lazy val converted= trainingDataRef.toJson.convertTo[Array[myTest]]
^
Error:(19, 54) not enough arguments for method convertTo: (implicit evidence$1: spray.json.JsonReader[Array[A$A61.this.myTest]])Array[A$A61.this.myTest].
Unspecified value parameter evidence$1.
lazy val converted= trainingDataRef.toJson.convertTo[Array[myTest]]
^
Error:(10, 61) could not find implicit value for evidence parameter of type spray.json.DefaultJsonProtocol.JF[Map[String,Any]]
implicit val format: RootJsonFormat[myTest] = jsonFormat3(myTest.apply)
^
Code: ^
import spray.json.DefaultJsonProtocol._
import spray.json._
case class myTest (
id: String,
classDetails: Map[String, Any],
school: Map[String, Any])
object myTest {
implicit val format: RootJsonFormat[myTest] = jsonFormat3(myTest.apply)
}
val trainingDataRef = """[{"id":"my-id","classDetails":{"sec":"2","teacher":"John"},"school":{"name":"newschool"}}]"""
println(trainingDataRef.getClass)
val converted= trainingDataRef.toJson.convertTo[Array[myTest]]
println(converted)
spray-json has good documentation, try take a look there. Basically, you have to define your case classes and implement JsonFormat for them:
import spray.json.DefaultJsonProtocol._
import spray.json._
case class ClassDetails(sec: String, teacher: String)
object ClassDetails {
implicit val format: RootJsonFormat[ClassDetails] = jsonFormat2(ClassDetails.apply)
}
case class School(name: String)
object School {
implicit val format: RootJsonFormat[School] = jsonFormat1(School.apply)
}
case class ClassInfo
(
id: String,
classDetails: ClassDetails,
school: School
)
object ClassInfo {
implicit object ClassInfoFormat extends RootJsonFormat[ClassInfo] {
def write(c: ClassInfo): JsValue = JsObject(
"id" -> JsString(c.id),
"classDetails" -> c.classDetails.toJson,
"school" -> c.school.toJson
)
def read(value: JsValue): ClassInfo = {
value.asJsObject.getFields("id", "classDetails", "school") match {
case Seq(JsString(name), details, school) =>
new ClassInfo(name, details.convertTo[ClassDetails], school.convertTo[School])
case _ => throw new DeserializationException("ClassInfo expected")
}
}
}
}
val json = """[{"id":"my-id","classDetails":{"sec":"2","teacher":"John"},"school":{"name":"newschool"}}]"""
// JSON string to case classes
val classInfos = json.parseJson.convertTo[Seq[ClassInfo]]
classInfos.zipWithIndex.foreach { case (c, idx) =>
println(s"$idx => $c")
}
println
// Seq[ClassInfo] to JSON
println(s"$classInfos: ")
println(classInfos.toJson.prettyPrint)
I am trying to invoke the google geocoding api and retrieve the response.
lazy val geoCodingConnectionFlow: Flow[HttpRequest, HttpResponse, Any] =
Http().outgoingConnectionHttps(config.getString("services.geoCodingApiHost"), config.getInt("services.geoCodingApiPort"))
def geoCodingRequest(request: HttpRequest): Future[HttpResponse] = Source.single(request).via(geoCodingConnectionFlow).runWith(Sink.head)
/**
* This call to google service is limited
* #see https://developers.google.com/maps/documentation/geocoding/#Limits
*/
def ?(l: GeoLocation)(implicit ec: ExecutionContext): Future[Either[String, List[Result]]] = {
val latlang = s"17.3644264,78.3896741"
import org.json4s.native.Serialization
import org.json4s.NoTypeHints
import akka.http.scaladsl.model._
import akka.http.scaladsl.unmarshalling._
implicit val materializer = ActorMaterializer()
implicit val executor = system.dispatcher
implicit val formats = Serialization.formats(NoTypeHints)
geoCodingRequest(RequestBuilding.Get(s"${config.getString("services.geoCodingApiUrlPart")}?latlng=$latlang&key=${config.getString("services.geoCodingApiKey")}")).flatMap { response =>
val nonBinaryType = ContentTypes.`application/json`
def responseEntity: HttpEntity = response.entity
response.status match {
case OK if (response.entity.contentType == ContentTypes.`application/json`) => Unmarshal(response.entity).to[List[Result]].map(Right(_))
case BadRequest => Future.successful(Left(s"$latlang: incorrect Latitude and Longitude format"))
case _ => Unmarshal(response.entity).to[String].flatMap { entity =>
val error = s"Google GeoCoding request failed with status code ${response.status} and entity $entity"
Future.failed(new IOException(error))
}
}
}
}
}
I am getting the following compilation error when trying to execute this!
Service.scala:78: could not find implicit value for parameter um: akka.http.scaladsl.unmarshalling.Unmarshaller[akka.http.scaladsl.model.ResponseEntity,List[com.thenewmotion.geocode.Result]]
case OK if(response.entity.contentType == ContentTypes.`application/json`)=> Unmarshal(response.entity).to[List[Result]].map(Right(_))
Please help me to get the result parsed into the following Result case classes:
package com.thenewmotion.geocode
case class Address(
long_name: String,
short_name: String,
types: List[String]
)
case class Result(
address_components: List[Address],
formatted_address: String,
types: List[String]
)
case class Response(
results: List[Result],
status: Status
) {
def allResults = status match {
case Ok => Right(results)
case e: Error => Left(e)
}
}
/** #see https://developers.google.com/maps/documentation/geocoding/#StatusCodes */
sealed trait Status
case object Ok extends Status
sealed trait Error extends Status
case object ZeroResults extends Error
case object OverQuotaLimit extends Error
case object Denied extends Error
case object InvalidRequest extends Error
case class OtherError(description: String) extends Error
^
As said in your error message, you need to provide an implicit Unmarshaller[akka.http.scaladsl.model.ResponseEntity,List[com.thenewmotion.geocode.Result]] otherwise the framework won't know how to convert the response entity into your model List[com.thenewmotion.geocode.Result].
Alternatively, you can use the built in unmarshaller to convert the entity to String first, then use spray-json to parse the json string into the target model:
import akka.http.scaladsl.unmarshalling.Unmarshal
import spray.json._
implicit val modelJsonReader = new JsonReader[List[com.thenewmotion.geocode.Result]] {
// See https://github.com/spray/spray-json on how to implement JsonReader
}
def parseJson(str: String): List[com.thenewmotion.geocode.Result] = {
// parse using spray-json
str.parseJson.convertTo[List[com.thenewmotion.geocode.Result]]
}
response.status match {
case OK if (response.entity.contentType == ContentTypes.`application/json`) =>
Unmarshal(response.entity).to[String].map { jsonString =>
Right(parseJson(jsonString))
}
case BadRequest => Future.successful(Left(s"$latlang: incorrect Latitude and Longitude format"))
case _ => Unmarshal(response.entity).to[String].flatMap { entity =>
val error = s"Google GeoCoding request failed with status code ${response.status} and entity $entity"
Future.failed(new IOException(error))
}
}
I'm using my own implicit implementation of JSON serializer and deserializer for my case object
My case class looks like (it's just a code snippet)
sealed trait MyTrait
case object MyCaseClass extends MyTrait
I want to write my own ser. and deser. of JSON for MyTrait
implicit val myTraitFormat = new JsonFormat[MyTrait] {
override def read(json: JsValue): MyTrait = json.asJsObject.getFields("value") match {
case Seq(JsString("TEST")) ⇒ MyCaseClass
case _ ⇒ throw new DeserializationException(s"$json is not a valid extension of my trait")
}
override def write(myTrait: MyTrait): JsValue = {
myTrait match {
case MyCaseClass => JsObject("value" -> JsString("TEST"))
}
}
}
Now my test is failing by throwing DeserializationException:
"The my JSON format" when {
"deserializing a JSON" must {
"return correct object" in {
val json = """{"value": "TEST"}""".asJson
json.convertTo[MyTrait] must equal (MyCaseClass)
}
}
}
Obviously json.asJsObject.getFields("value")can not be matched to Seq(JsString("TEST")). Maybe this is related to using traits?
But I have found example on official spray-json site https://github.com/spray/spray-json#providing-jsonformats-for-other-types
Any ideas how to properly match on field in JsObject?
Thanks!
Best
Write variable name instead of string:
override def read(json: JsValue): MyCaseClass = json.asJsObject.getFields("value") match {
case Seq(JsString(value)) ⇒ MyCaseClass
case _ ⇒ throw new DeserializationException(s"$json is not a valid case class")
}
Does that work for you?
Update:
Yes, your test is wrong. I tried it with the following (note parseJson instead of asJson) and it worked:
scala> val json = """{"value": "TEST"}""".parseJson
json: spray.json.JsValue = {"value":"TEST"}
scala> json.convertTo[MyCaseClass]
res2: MyCaseClass = MyCaseClass()
Update 2: Tried it with trait:
scala> import spray.json._
import spray.json._
scala> import spray.json.DefaultJsonProtocol._
import spray.json.DefaultJsonProtocol._
[Your code]
scala> val json = """{"value": "TEST"}""".parseJson
json: spray.json.JsValue = {"value":"TEST"}
scala> json.convertTo[MyTrait]
res1: MyTrait = MyCaseClass