I am trying to serialize a kotlin.collections.Map from Json using Ktor and I continue to get an error, basically telling me it doesn't know how. I assumed this case was basic.
val beans = beans {
bean("oauthClient") {
HttpClient(CIO) {
expectSuccess = true
install(DefaultRequest){
url("${env["oauth.url"]!!}/ms_oauth/oauth2/endpoints/oauthservice/tokens")
}
install(Auth) {
basic {
credentials {
BasicAuthCredentials(username = env["client.id"]!!, password = env["client.secret"]!!)
}
}
}
install(ContentNegotiation) {
json()
}
}
}
bean {
val oauthResults: Map<String,Any> = runBlocking { // blows up here
ref<HttpClient>("oauthClient").submitForm("${env["oauth.url"]!!}/ms_oauth/oauth2/endpoints/oauthservice/tokens",
Parameters.build {
append("grant_type", "password")
append("scope", "bug.rest.public")
append("username", env["ssoUsername"]!!)
append("password", env["ssoPassword"]!!)
}
).body()
}
}
}
error:
Caused by: io.ktor.client.call.NoTransformationFoundException:
No transformation found: class io.ktor.utils.io.ByteBufferChannel -> class kotlin.collections.Map
Do I have to explicitly enable something? Or am I making a different mistake?
Update
I've since tried to marshal it into a data object and I get the same error, but referencing that object:
#Serializable
data class Oauth constructor(
#SerialName("expires_in")
val expiration: Int,
#SerialName("token_type")
val tokenType:String,
#SerialName("access_token")
val accessToken: String
)
error:
io.ktor.client.call.NoTransformationFoundException:
No transformation found: class io.ktor.utils.io.ByteBufferChannel -> class com.company.Oauth
I'm trying to fetch and deserialize some data that is being hosted on github.
{
"Meals": [
{
"id": "1598044e-5259-11e9-8647-d663bd870b02",
"name": "Tomato pasta",
"quantity": [{
"quantity": 1 },
{
"quantity": 2
},
{
"quantity": 3
}],
"availableFromDate": "1605802429",
"expiryDate": "1905802429",
"info": "Vegetarian",
"hot": false,
"locationLat": 57.508865,
"locationLong": -6.292,
"distance": null
},
{
"id": "2be2d854-a067-43ec-a488-2e69f0f2a624",
"name": "Pizza",
"quantity": [{
"quantity": 1 },
{
"quantity": 2
},
{
"quantity": 3
}
],
"availableFromDate": "1605802429",
"expiryDate": "1905902429",
"info": "Meat",
"hot": false,
"locationLat": 51.509465,
"locationLong": -0.135392,
"distance": null
}
]
}
If I spin up a json-server locally then it works perfectly, so I know that my data class is not the problem. However when I try to do the same thing from that github link I get this error:
Error Domain=KotlinException Code=0 "No transformation found: class io.ktor.utils.io.ByteChannelNative -> class kotlin.collections.List
I have a feeling it might be to do with setting a ContentType or something along those lines but I haven't had any success specifying that so far.
Here is my code to make the request:
class MealApi {
private val httpClient = HttpClient {
install(JsonFeature) {
val json = Json { ignoreUnknownKeys = true }
serializer = KotlinxSerializer(json)
}
}
suspend fun getAllMeals(): List<Meal> {
return httpClient.get(endpoint)
}
}
and here is my data class just for completeness:
#Serializable
data class Meal(
#SerialName("id")
val id: String,
#SerialName("name")
val name: String,
#SerialName("quantity")
val quantity: List<Quantity>,
#SerialName("availableFromDate")
var availableFromDate: String,
#SerialName("expiryDate")
var expiryDate: String,
#SerialName("info")
val info: String,
#SerialName("hot")
val hot: Boolean,
#SerialName("locationLat")
val locationLat: Float,
#SerialName("locationLong")
val locationLong: Float,
#SerialName("distance")
var distance: Double? = null
)
#Serializable
data class Quantity(
#SerialName("quantity")
val quantity: Int
)
UPDATE
I've found that this server https://gitcdn.link/ allows you to serve your raw github files with the right Content-Type.
I've searched a lot how to change the server response headers (to change the plain/text one to application/json) but it seems that ktor actually doesn't allow to do that:
https://youtrack.jetbrains.com/issue/KTOR-617
https://youtrack.jetbrains.com/issue/KTOR-580
A nice way should be to allow the ResponseObserver to change the server response headers and pass through the modify response. But you can't actually.
Your problem depends, as you pointed out, from the fact that the raw github page provides an header Content-Type=plain/text instead of ContentType=application/json.
So IRL when you are running your API in a real server this won't occur as you'll take care to put the right content type at server level.
But if you want a workaround to this you could rewrite your api call in this way:
suspend fun getAllMealsWithFallback() {
var meals: Meals? = null
try {
meals = httpClient.get(endpoint)
} catch (e: NoTransformationFoundException) {
val mealsString: String = httpClient.get(endpoint)
val json = kotlinx.serialization.json.Json {
ignoreUnknownKeys = true
}
meals = json.decodeFromString(mealsString)
} finally {
println("Meals: ${meals?.meals}")
}
}
I had to add this class to conform to the json text you have provided in the github link.
#Serializable
data class Meals(
#SerialName("Meals")
val meals: List<Meal>,
)
Try this:
install(JsonFeature) {
serializer = KotlinxSerializer(KotlinJson { ignoreUnknownKeys = true })
acceptContentTypes = acceptContentTypes + ContentType.Any
}
If you'd like to accept all content types. Or Use ContentType.Text.Any, ContentType.Text.Html if you preferred.
In case the issue is the Content-Type:
You can alter the list of response content types, for which the KotlinxSerializer gets active, by extending the JsonFeature block to:
install(JsonFeature) {
val json = Json { ignoreUnknownKeys = true }
serializer = KotlinxSerializer(json)
receiveContentTypeMatchers += object : ContentTypeMatcher {
override fun contains(contentType: ContentType): Boolean =
contentType == ContentType("text", "plain")
}
}
If you're using Ktor 2.0, you would need ContentNegotiation plugin instead of JsonFeature.
For example if you use Gson:
install(io.ktor.client.plugins.contentnegotiation.ContentNegotiation) {
gson()
}
As a workaround for ktor version 2.0.3 you can create you own AppContentNegotiation class and in the scope.responsePipeline.intercept provide needed contentType
/**
* A plugin that serves two primary purposes:
* - Negotiating media types between the client and server. For this, it uses the `Accept` and `Content-Type` headers.
* - Serializing/deserializing the content in a specific format when sending requests and receiving responses.
* Ktor supports the following formats out-of-the-box: `JSON`, `XML`, and `CBOR`.
*
* You can learn more from [Content negotiation and serialization](https://ktor.io/docs/serialization-client.html).
*/
public class AppContentNegotiation internal constructor(
internal val registrations: List<Config.ConverterRegistration>
) {
/**
* A [ContentNegotiation] configuration that is used during installation.
*/
public class Config : Configuration {
internal class ConverterRegistration(
val converter: ContentConverter,
val contentTypeToSend: ContentType,
val contentTypeMatcher: ContentTypeMatcher,
)
internal val registrations = mutableListOf<ConverterRegistration>()
/**
* Registers a [contentType] to a specified [converter] with an optional [configuration] script for a converter.
*/
public override fun <T : ContentConverter> register(
contentType: ContentType,
converter: T,
configuration: T.() -> Unit
) {
val matcher = when (contentType) {
ContentType.Application.Json -> JsonContentTypeMatcher
else -> defaultMatcher(contentType)
}
register(contentType, converter, matcher, configuration)
}
/**
* Registers a [contentTypeToSend] and [contentTypeMatcher] to a specified [converter] with
* an optional [configuration] script for a converter.
*/
public fun <T : ContentConverter> register(
contentTypeToSend: ContentType,
converter: T,
contentTypeMatcher: ContentTypeMatcher,
configuration: T.() -> Unit
) {
val registration = ConverterRegistration(
converter.apply(configuration),
contentTypeToSend,
contentTypeMatcher
)
registrations.add(registration)
}
private fun defaultMatcher(pattern: ContentType): ContentTypeMatcher =
object : ContentTypeMatcher {
override fun contains(contentType: ContentType): Boolean =
contentType.match(pattern)
}
}
/**
* A companion object used to install a plugin.
*/
#KtorDsl
public companion object Plugin : HttpClientPlugin<Config, AppContentNegotiation > {
public override val key: AttributeKey<AppContentNegotiation> =
AttributeKey("ContentNegotiation")
override fun prepare(block: Config.() -> Unit): AppContentNegotiation {
val config = Config().apply(block)
return AppContentNegotiation(config.registrations)
}
override fun install(plugin: AppContentNegotiation, scope: HttpClient) {
scope.requestPipeline.intercept(HttpRequestPipeline.Transform) { payload ->
val registrations = plugin.registrations
registrations.forEach { context.accept(it.contentTypeToSend) }
if (subject is OutgoingContent || DefaultIgnoredTypes.any { it.isInstance(payload) }) {
return#intercept
}
val contentType = context.contentType() ?: return#intercept
if (payload is Unit) {
context.headers.remove(HttpHeaders.ContentType)
proceedWith(EmptyContent)
return#intercept
}
val matchingRegistrations =
registrations.filter { it.contentTypeMatcher.contains(contentType) }
.takeIf { it.isNotEmpty() } ?: return#intercept
if (context.bodyType == null) return#intercept
context.headers.remove(HttpHeaders.ContentType)
// Pick the first one that can convert the subject successfully
val serializedContent = matchingRegistrations.firstNotNullOfOrNull { registration ->
registration.converter.serialize(
contentType,
contentType.charset() ?: Charsets.UTF_8,
context.bodyType!!,
payload
)
} ?: throw ContentConverterException(
"Can't convert $payload with contentType $contentType using converters " +
matchingRegistrations.joinToString { it.converter.toString() }
)
proceedWith(serializedContent)
}
scope.responsePipeline.intercept(HttpResponsePipeline.Transform) { (info, body) ->
if (body !is ByteReadChannel) return#intercept
if (info.type == ByteReadChannel::class) return#intercept
// !!!!!!! Provide desired content type here
val contentType = <HERE> // default implementation is - context.response.contentType() ?: return#intercept
val registrations = plugin.registrations
val matchingRegistrations = registrations
.filter { it.contentTypeMatcher.contains(contentType) }
.takeIf { it.isNotEmpty() } ?: return#intercept
// Pick the first one that can convert the subject successfully
val parsedBody = matchingRegistrations.firstNotNullOfOrNull { registration ->
registration.converter
.deserialize(context.request.headers.suitableCharset(), info, body)
} ?: return#intercept
val response = HttpResponseContainer(info, parsedBody)
proceedWith(response)
}
}
}
}
And then install it for HttpClient
HttpClient {
install(AppContentNegotiation) {
json(json)
addDefaultResponseValidation()
}
}
In my case, this exception was thrown when I was attempting to access the body() potion of the HTTP response as such:
val httpResponse = httpClient.post(urlString) { ... }
val body = httpResponse.body<YourExpectedSerializableResponseType>
In the happy path scenario, the server would return a body that matched YourExpectedSerializableResponseType and everything would work as expected.
However, for a particular edge case that returned a different (still considered successful) status code, the server returned an empty response body. Since the client was expecting a response body and in this case, there wasn't any, this exception was thrown because it could not serialize an empty response body to the expected type YourExpectedSerializableResponseType.
My recommendation: In addition to ensuring your server is returning the type you expect to serialize/consume on your client, confirm your server is actually returning an object.
Internal Dialogue: I wonder if this exception could be more clear in this case as the issue is more so that the client expected a response body to exist and less so that an empty response body ("") couldn't be serialized into an expected type - especially given that an empty response body isn't even valid JSON. Hm. 🤔
I'm running spark job with kafka-clients:2.2.1 and
spark-streaming-kafka-0-10_2.11:2.4.3
The job is running in the following mode:
val scc: StreamingContext = new StreamingContext(spark, "180 seconds")
val kafkaSink: Broadcast[KafkaSink] = ...
val stream = KafkaUtils.createDirectStream(...)
stream //. do some logic
.foreachRDD { (rdd, time) =>
p.foreach { row =>
val rowJson = convertRowToJSON(row)
kafkaSink.value.send(kafkaOptions.topic, rowJson) }
}
The job is running but for some mini-batches, there are failed tasks with following exceptions:
java.lang.NullPointerException
at org.apache.kafka.clients.producer.KafkaProducer.waitOnMetadata(KafkaProducer.java:1010)
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:916)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:897)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:774)
at com.outbrain.recs.rtap.kafka.KafkaSink.send(KafkaSink.scala:42)
at
What could be the reason for those exceptions?
I'm new to the play framework generally and how to use it with Scala. I want to build a proxy for big Json objects. I achieved so far that the json is stored in a cache and if it is not there, requested from a webservice.
However when two requests are coming in, targeting the same end point (webservice and path are identicall) only one call should be performed and the other request should wait for the result of the first call. At the moment it is performing a call to the service with every request.
This is my controller:
#Singleton
class CmsProxyController #Inject()(val cmsService: CmsProxyService) extends Controller {
implicit def ec : ExecutionContext = play.api.libs.concurrent.Execution.defaultContext
def header(path: String) = Action.async { context =>
cmsService.head(path) map { title =>
Ok(Json.obj("title" -> title))
}
}
def teaser(path: String) = Action.async { context =>
cmsService.teaser(path) map { res =>
Ok(res).as(ContentTypes.JSON)
}
}
}
This is the service:
trait CmsProxyService {
def head(path: String): Future[String]
def teaser(path: String): Future[String]
}
#Singleton
class DefaultCmsProxyService #Inject()(cache: CacheApi, cmsCaller: CmsCaller) extends CmsProxyService {
private val BASE = "http://foo.com"
private val CMS = "bar/rest/"
private val log = Logger("application")
override def head(path: String) = {
query(url(path), "$.payload[0].title")
}
override def teaser(path: String) = {
query(url(path), "$.payload[0].content.teaserText")
}
private def url(path: String) = s"${BASE}/${CMS}/${path}"
private def query(url: String, jsonPath: String): Future[String] = {
val key = s"${url}?${jsonPath}"
val payload = findInCache(key)
if (payload.isDefined) {
log.debug("found payload in cache")
Future.successful(payload.get)
} else {
val queried = parse(fetch(url)) map { json =>
JSONPath.query(jsonPath, json).as[String]
}
queried.onComplete(value => saveInCache(key, value.get))
queried
}
}
private def parse(fetched: Future[String]): Future[JsValue] = {
fetched map { jsonString =>
Json.parse(jsonString)
}
}
//retrieve the requested value from the cache or from ws
private def fetch(url: String): Future[String] = {
val body = findInCache(url)
if (body.isDefined) {
log.debug("found body in cache")
Future.successful(body.get)
} else {
cmsCaller.call(url)
}
}
private def findInCache(key: String): Option[String] = cache.get(key)
private def saveInCache(key: String, value: String, duration: FiniteDuration = 5.minutes) = cache.set(key, value, 5.minutes)
}
And finally the call to the webservice:
trait CmsCaller {
def call(url: String): Future[String]
}
#Singleton
class DefaultCmsCaller #Inject()(wsClient: WSClient) extends CmsCaller {
import scala.concurrent.ExecutionContext.Implicits.global
//keep those futures which are currently requested
private val calls: Map[String, Future[String]] = TrieMap()
private val log = Logger("application")
override def call(url: String): Future[String] = {
if(calls.contains(url)) {
Future.successful("ok")
}else {
val f = doCall(url)
calls put(url, f)
f
}
}
//do the final call
private def doCall(url: String): Future[String] = {
val request = ws(url)
val response = request.get()
val mapped = mapResponse(response)
mapped.onComplete(_ => cmsCalls.remove(url))
mapped
}
private def ws(url: String): WSRequest = wsClient.url(url)
//currently executed with every request
private def mapResponse(f: Future[WSResponse]): Future[String] = {
f.onComplete(_ => log.debug("call completed"))
f map {res =>
val status = res.status
log.debug(s"ws called, response status: ${status}")
if (status == 200) {
res.body
} else {
""
}
}
}
}
My question is: How can only one call to the webservice beeing executed? Even if there are several requests to the same target. I don't want to block it, the other request (not sure if I use the right word here) shall just be informed that there is already a webservice call on the way.
The request to head and teaser, see controller, shall perform only one call to the webservice.
Simple answer using Scala lazy keyword
def requestPayload(): String = ??? //do something
#Singleton
class SimpleCache #Inject() () {
lazy val result: Future[String] = requestPayload()
}
//Usage
#Singleton
class SomeController #Inject() (simpleCache: SimpleCache) {
def action = Action { req =>
simpleCache.result.map { result =>
Ok("success")
}
}
}
First request will trigger the rest call and all the other requests will use the cached result. Use map and flatMap to chain the requests.
Complicated answer using Actors
Use Actor to queue requests and Cache the result of the first successful request json result. All the other requests will read the result of the first request.
case class Request(value: String)
class RequestManager extends Actor {
var mayBeResult: Option[String] = None
var reqs = List.empty[(ActorRef, Request)]
def receive = {
case req: Request =>
context become firstReq
self ! req
}
def firstReq = {
case req: Request =>
process(req).onSuccess { value =>
mayBeResult = Some(value)
context become done
self ! "clear_pending_reqs"
}
context become processing
}
def processing = {
case req: Request =>
//queue requests
reqs = reqs ++ List(sender -> req)
}
def done = {
case "clear_pending_reqs" =>
reqs.foreach { case (sender, _) =>
//send value to the sender
sender ! value.
}
}
}
handle the case where the first request fails. In the above code block if the first request fails then actor will never go to the done state.
I solved my problem with a synchronization of the cache in the service. I'm not sure if this an elegant solution, but it works for me.
trait SyncCmsProxyService {
def head(path: String): String
def teaser(path: String): String
}
#Singleton
class DefaultSyncCmsProxyService #Inject()(implicit cache: CacheApi, wsClient: WSClient) extends SyncCmsProxyService with UrlBuilder with CacheAccessor{
private val log = Logger("application")
override def head(path: String) = {
log.debug("looking for head ...")
query(url(path), "$.payload[0].title")
}
override def teaser(path: String) = {
log.debug("looking for teaser ...")
query(url(path), "$.payload[0].content.teaserText")
}
private def query(url: String, jsonPath: String) = {
val key = s"${url}?${jsonPath}"
val payload = findInCache(key)
if (payload.isDefined) {
payload.get
}else{
val json = Json.parse(body(url))
val queried = JSONPath.query(jsonPath, json).as[String]
saveInCache(key, queried)
}
}
private def body(url: String) = {
cache.synchronized {
val body = findInCache(url)
if (body.isDefined) {
log.debug("found body in cache")
body.get
} else {
saveInCache(url, doCall(url))
}
}
}
private def doCall(url : String): String = {
import scala.concurrent.ExecutionContext.Implicits.global
log.debug("calling...")
val req = wsClient.url(url).get()
val f = req map { res =>
val status = res.status
log.debug(s"endpoint called! response status: ${status}")
if (status == 200) {
res.body
} else {
""
}
}
Await.result(f, 15.seconds)
}
}
Note that I omitted the traits UrlBuilder and CacheAccessor here because they are trivial.
In the following code i get the nullpointer exception only in some cases and the JSON is the same every time. How to resolve this
Error
errors.GrailsExceptionResolver - NullPointerException occurred when processing request: [POST] /school/sd/
Cannot set property 'school' on null object. Stacktrace follows:
Message: Cannot set property 'school' on null object
controller
def save() {
if (!requestIsJson()) {
respondNotAcceptable()
return
}
println request.GSON
def sInstance = new School(request.GSON)
println "got here"
if (sInstance.save(flush: true)) {
respondCreated sInstance
} else {
respondUnprocessableEntity sInstance
}
def resp = RestClientHelper.createExpGroup(sInstance)
}
In order to create Grails object from JSON it's easy to use special converter.
import grails.converters.JSON
.......
class Controller {
def doSomthing = {
def myDomain = new MyDomain(JSON.parse(params.myDomain))
//Save domain object
myDomain.save(flush:true)
}
}
Grails converter reference.