I have an endpoint we can call /test that internally fetches data from a 3rd party API and then wants to do some transformation before returning a response. Where I am hung up is this 3rd party API is returning gzipped JSON and I can't decode it (yet). I found the decodeRequest directive but it seems I have to use this in my routing and I am a level deeper here. I have an internal method I call once I receive a GET to my endpoint /test which is named do3rdPartyAPIRequest where I build up an HttpRequest and pass to Http().singleRequest() so then in return I have a Future[HttpResponse] which is where I think I want to be but I am stuck here.
With some local APIs I built and consumed in a similar fashion I didn't encode my responses so typically with a Future[HttpResponse] I check the response status and go into conversion to JSON via Unmarshal but this one needs an extra step as far as I know before transforming to JSON. I realize this question is very similar to this one however that is spray specific and I haven't been able to translate this answer into current akka http
Finally figured this out - this might not be the absolute best to get a bytestring from response but it works .. Turns out you can use Gzip class
and you have two options
Gzip.decode
Gzip.decoderFlow
Here are my examples in case this helps you:
def getMyDomainObject(resp: HttpResponse):Future[MyDomain] = {
for {
byteString <- resp.entity.dataBytes.runFold(ByteString(""))(_ ++ _)
decompressedBytes <- Gzip.decode(byteString)
result <- Unmarshal(decompressedBytes).to[MyDomain]
} yield result
}
def getMyDomainObjectVersion2(resp:HttpResponse):Future[MyDomain] = {
resp.entity.dataBytes
.via(Gzip.decoderFlow)
.runWith(Sink.head)
.flatMap(Unmarshal(_).to[MyDomain])
}
You would expect that compressed content would be managed by akka-http by default, but it only provides PredefinedFromEntityUnmarshallers and inside entity there is not information about Content-encoding header.
To solve this you have to implement your own Unmarshaller and to have it in scope
Example:
implicit val gzipMessageUnmarshaller = Unmarshaller(ec => {
(msg: HttpMessage) => {
val `content-encoding` = msg.getHeader("Content-Encoding")
if (`content-encoding`.isPresent && `content-encoding`.get().value() == "gzip") {
val decompressedResponse = msg.entity.transformDataBytes(Gzip.decoderFlow)
Unmarshal(decompressedResponse).to[String]
} else {
Unmarshal(msg).to[String]
}
}
})
Related
My professor wants me to write a little tutorial on how to deploy Ballerina services. So I'm trying to learn it. I'm using Version 1.2 and I'm a bit overwhelmed by the concept of taint checking and the variable types...
I'm trying to write a minimal REST-Service with an endpoint that requests json data from another api and then uses that JSON to do stuff.
What's working so far is the following:
service tutorial on new http:Listener(9090) {
// Resource functions are invoked with the HTTP caller and the incoming request as arguments.
resource function getName(http:Caller caller, http:Request req) {
http:Client clientEP = new("https://api.scryfall.com/");
var resp = clientEP->get("/cards/random");
if (resp is http:Response) {
var payload = resp.getJsonPayload();
if (payload is json) {
var result = caller->respond(<#untainted>(payload));
} else {
log:printError("");
}
} else {
log:printError("");
}
}
That responds with the JSON that is returned from https://api.scryfall.com/cards/random
But lets now say, that I want to access a single value from that JSON. e.G. "name".
If I try to access it like this: payload["name"]
I get: invalid operation: type 'json' does not support indexing
I just figured out that it works if I create a map first like that:
map mp = <map>payload;
If I then access mp["name"] it works. BUT WHY? What is the json type good for if you still have to create a map and then cast the payload? And how would I access json inside the json? For example mp["data"][0]... invalid operation: type 'json' does not support indexing again...
And I'm still trying to understand the concept of taint checking....
do I just cast everything that is tainted to <#untainted> after checking the content?
Sometimes I really do not get what the documentation is trying to tell me....
I would recommend you to use Ballerina Swan Lake versions. Swan Lake versions contain enhancements to various language features. Here is a sample code that covers your use case. You can download Swan Lake Alpha2 at https://ballerina.io/
import ballerina/io;
import ballerina/http;
service tutorial on new http:Listener(9090) {
resource function get payload() returns json|error {
http:Client clientEP = check new ("https://api.scryfall.com/");
json payload = <json> check clientEP -> get("/cards/random", targetType = json);
// Processing the json payload
// Here the type of the `payload.name` expression is json | error
// You can eliminate with check: it returns from this resource with this error
json nameField = check payload.name;
io:println(nameField);
// You can traverse the json tree as follows
json standardLegality = check payload.legalities.standard;
io:println(standardLegality);
// colors is an array
// The cast is necessary because `check payload.colors` gives you a json
json colors = <json[]> check payload.colors;
io:println(colors);
// Responding with the complete payload recived from api.scryfall.com
return payload;
}
}
Taint analysis helps you to write code with no security vulnerabilities. However, we've disabled taint analysis in Swan Lake versions. If you want to enable it, you can use the option --taint-check with bal build
I am using play-ws standalone to consume REST service in scala.
val data = Json.obj("message" -> "How are you?")
wsClient.url("http://localhost:5000/token").post(data).map { response =>
val statusText: String = response.statusText
println(response.body)
}
When i run this, i get the following error,
Cannot find an instance of play.api.libs.json.JsObject to WSBody. Define a BodyWritable[play.api.libs.json.JsObject] or extend play.api.libs.ws.ahc.DefaultBodyWritables
wsClient.url("http://localhost:5000/token").post(data).map { response =>
It tells to define a bodywritable. I have read the documentation but cud't get the "BodyWritable". I am new to scala. Anybody help me please. Thanks in advance.
You need to import BodyWritables for json objects, Add following import statements to your source file
import play.api.libs.ws.JsonBodyReadables._
import play.api.libs.ws.JsonBodyWritables._
For more information have a look at official documentation
The current accepted answer does not work in Scala Play 2.7.x (possibly some earlier versions as well).
I couldn't find it in the docs, but you need to explicitly call asScala on the ws object. For example:
val data = Json.obj("message" -> "How are you?")
ws
.asScala()
.url("http://someurl.com")
.post(data)
.map(response => {
//do something with response
})
Note: this also returns a scala future instead of a java completion stage.
My app supports protobuf and JSON serialzation. For JSON serialization I use com.trueaccord.scalapb.json.JsonFormat, my dtos are generated from proto definitions.
The com.trueaccord serializer wraps option types to JSON objects which is causing issues for some clients so I want to be able to support org.json4s without braking the existing clients.
I would like to be able to pick a serializer based on a custom http header called JFORMAT. The idea is that if this header is sent I will use json4s otherwise I will use the trueaccord serializer.
I managed to create a Unmarshaller which can pick a request serializer based on a header value:
Unmarshaller.withMaterializer[HttpRequest, T](_ => implicit mat => {
case request: HttpRequest =>
val entity = request.entity
entity.dataBytes.runFold(ByteString.empty)(_ ++ _).map(data => {
entity.contentType match {
case `applicationJsonContentType` =>
val jsFormat = {
val header = request.headers.find(h => h.name() == jsonFormatHeaderName)
if (header.isEmpty) "1.0" else header.get.value()
}
val charBuffer = Unmarshaller.bestUnmarshallingCharsetFor(entity)
val jsonText = data.decodeString(charBuffer.nioCharset().name())
val dto = if(jsFormat == "2.0") {
write[T](value)(formats) // New Formatter
} else {
JsonFormat.fromJsonString[T](jsonText) // Old Formatter
}
dto
case `protobufContentType` =>
companion.parseFrom(CodedInputStream.newInstance(data.asByteBuffer)) // Proto Formatter
case _ =>
throw UnsupportedContentTypeException(applicationJsonContentType, protobufContentType)
}
})
I want to do the same with my Marshaller which I use with Marshaller.oneOf and the JSON handling one looks like:
Marshaller.withFixedContentType(contentType) { value =>
val jsonText = JsonSerializer.toJsonString[T](value)
HttpEntity(contentType, jsonText)
}
Is there a way to construct a Mashaller which is aware of the request http headers? The Akka HTTP docs don't have any examples and I cannot make sense of the PredefinedToRequestMarshallers.
Do I need to combine multiple marshallers somehow or can I append some metadata to a context during the request serialization I can use later in the Marshaller? I want to avoid appending meta to my dto if possible or using a custom content type like application/vnd.api+json
There are lots of other useful info I could use from the request when I format the response like Accept-Encoding, custom headers like unique request id to create a correlation id, I could add JSONP support by reading the callback query parmeter, etc.
To clarify: I need a solution to use the Mashaller, subclass of it or a custom version created by a factory method or maybe multiple Marshallers chained together. Marshaller.withFixedContentType already using the Accept header so there must be a way. I added added bounty to reward a solution to a specific challenge. I am ware of hacks and workarounds and I asked the question because I need a clean solution solving a specific scenario.
Custom Marshallers section mentions Marshaller.oneOf overloaded methods, that seems to be what you want:
Helper for creating a "super-marshaller" from a number of
"sub-marshallers". Content-negotiation determines, which
"sub-marshaller" eventually gets to do the job.
The Marshaller companion object has many methods that receive a Seq[HttpHeader]. You can look into their implementations as well.
I don't have the time to look into the source code myself, but if this is not enough to put you on the right path, let me know.
Edit:
How about?
get {
optionalHeaderValueByName("JFORMAT") { format =>
complete {
format match {
case Some(f) => "Complete with json4s"
case _ => "Complete with trueaccord"
}
}
}
}
I have implemented Play framework's WebSocket so as to perform server communication using a WebSocket instead of Http. I have created a function as WebSocket.using[JsValue]. My json response is stored inside a Future[JsValue] variable and I am trying to fetch and return the json value from within Future[JsValue] variable. However I have been unable to return the json data from the Future[JsValue] variable. When I tried creating the WebSocket function as WebSocket.using[Future[JsValue]], in this case I was unable to create a json FrameFormatter for it.
def socketTest = WebSocket.using[JsValue] { request =>
val in = Iteratee.ignore[JsValue]
val out = Enumerator[JsValue](
Json.toJson(futureJsonVariable)
).andThen(Enumerator.eof)
(in, out)
}
futureJsonVariable is a variable of type Future[JsValue] In the above code the error at runtime is No Json serializer found for type scala.concurrent.Future[play.api.libs.json.JsValue]. Try to implement an implicit Writes or Format for this type. How can I return a json using a WebSocket method in Scala ? How can it be achieved using an Actor class instance ? If anyone knows best available online tutorials for WebSocket in Play framework. Any help is appreciated.
Use tryAccept to return either the result of the future when it is redeemed, or an error:
def socketTest = WebSocket.tryAccept[JsValue] { request =>
futureJsonVariable.map { json =>
val in = Iteratee.ignore[JsValue]
val out = Enumerator(json).andThen(Enumerator.eof)
Right((in, out))
} recover {
case err => Left(InternalServerError(err.getMessage))
}
}
This is similar to using but returns a Future[Either[Result, (Iteratee[A, _], Enumerator[A])]]. The Either[Result, ...] allows you to handle the case where something unexpected occurs calculating the future value A by providing a play.api.mvc.Result in the Left branch. The corollary is that you need to also wrap the "happy path" (where nothing goes wrong) in Right, in this case the iteratee/enumerator tuple you'd ordinarily return from using.
You can do something similar with the tryAcceptWithActor function.
I have spent too much time trying to debug the following issue but I am not sure where the problem is occurring.
Issue: Getting 400 Bad Request, Invalid Json as response with following exception:
com.fasterxml.jackson.databind.JsonMappingException: No content to map due to end-of-input
at [Source: [B#6ee503c9; line: 1, column: 1]
at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:164) ~[jackson-databind.jar:2.2.2]
at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:2931) ~[jackson-databind.jar:2.2.2]
at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:2846) ~[jackson-databind.jar:2.2.2]
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:1569) ~[jackson-databind.jar:2.2.2]
at play.api.libs.json.JacksonJson$.parseJsValue(JsValue.scala:480) ~[play-json_2.10.jar:2.2.0]
at play.api.libs.json.Json$.parse(Json.scala:27) ~[play-json_2.10.jar:2.2.0]
Method to be tested in my controller:
def createArticle(id: String) =
Action.async(parse.json) { implicit request =>
(request.body \ "content").asOpt[String].map {
............
............
}.getOrElse(BadRequest("Invalid request body"))
}
Corresponding Unit Test:
"create article" in {
running(FakeApplication()) {
val postJson = Json.obj("content" -> "article content")
val result = resource.createArticle(ARTICE_ID)(FakeRequest(POST, controllers.routes.ArticleResource.create(ARTICLE_ID).url).withJsonBody(postJson).withHeaders(CONTENT_TYPE -> "application/json").run
status(result) must equalTo OK
}
}
I read the discussion here but none of the suggestions there helped.
I had similar problems, but didn't solve them so far (elegantly, anyway..). As my environment was java instead of scala, I can just give a hunch. I think that it's possible that when you send post it's done asynchronously (Action.async at your create article method), so you'd possibly need to wait at test code result before trying to see if it's OK.
I had a similar problem and the solution is pointed here. In my case it was because I using response.asJson() twice, and as described by #jroper
Considering that the body of an HTTP response is a stream, and not
something that you want to necessarily buffer in memory, then it makes
sense that accessing the body twice (regardless of what format you are
accessing the body in) would not be supported.
Try instantiating your request inside the method definition.
This is how it's working for me (example using POST with JSON body and result as JSON):
"process" should {
"should be valid" in {
val request = FakeRequest(POST, "/").withJsonBody(Json.parse(""" {
"id":1,
"name":"prod 1",
"price":55.55
}"""))
val result: Future[Result] = controller.process.apply(request)
val json = contentAsJson(result)
status(result) must be(CREATED)
(json \ "id").as[Int] mustBe 1
// .. more assertions and rest of code ...
Here the code deals with a JsValue and query it's nodes to check if the returned values are matching the desired mock data output