I'm trying to put several event types in the same Kafka topic using the JSON format, but in the Producer implementation I'm always getting org.apache.kafka.common.errors.SerializationException: Error serializing JSON message. Seems that the annotation #Schema isn't working as expected is like the schema defined by the annotation isn't enriched properly and in the method that validates the backward compatibility the schema defined by my event has the schemaObj empty and the result is not compatible and fails.
My event:
#Schema(
value = "1",
refs = Array(new SchemaReference(name = "event", subject = "event"))
)
case class Event(#BeanProperty id: String,
#BeanProperty name: String)
Producer:
def send(): Unit = {
val props = new Properties() {
put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092")
put(
ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringSerializer"
)
put(
ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
"io.confluent.kafka.serializers.json.KafkaJsonSchemaSerializer"
)
put("auto.register.schemas", "false")
put("use.latest.version", "true")
put("schema.registry.url", "http://127.0.0.1:8081")
put("json.fail.invalid.schema", "true")
}
val producer = new KafkaProducer[String, Event](props)
val topic = "all-json"
val key = "key1"
val event = Event("id", "name")
val record = new ProducerRecord[String, Event](topic, key, event)
producer.send(record).get
}
By the command line, I can perfectly produce the events. The JSON Schema is modeled by
{
"oneOf": [
{ "$ref": "Event.schema.json" },
{ "$ref": "EventB.schema.json" }
]
}
...
the dependencies of confluent used are the version 6.0.1.
Do you know what is the issue?
Related
I'm creating API server with Ktor and using exposed SQL.
I have two tables like below:
/// Performance Table
#kotlinx.serialization.Serializable
data class PerformanceDAO(val performanceId: String, val title: String, val actor: String)
object Performances : Table() {
val performanceId: Column<String> = varchar("performanceId", 20)
val title: Column<String> = varchar("title", 50)
override val primaryKey = PrimaryKey(performanceId)
}
//Actor Table
object Actors: Table() {
val performanceId: Column<String> = reference("performanceId", Performances.performanceId)
val name: Column<String> = varchar("name", 10)
}
and I'm trying to get data this way:
class PerformanceDAOImpl : PerformanceDAOFacade {
private fun resultRowToPerformance(row: ResultRow) = PerformanceDAO(
performanceId = row[Performances.performanceId],
title = row[Performances.title],
actor = row[Actors.name],
)
override suspend fun allPerformances(): List<PerformanceDAO> = dbQuery {
Actors.leftJoin(Performances)
.slice(Actors.name, Performances.title, Performances.performanceId)
.selectAll()
.groupBy(Performances.title).map(::resultRowToPerformance)
}
}
and then I got only one actors without a key.
{
"performanceId": "PF13234",
"title": "Harry footer",
"actor": [
"John"
]
},
but I want get data like this
{
"performanceId": "PF13234",
"title": "Harry footer",
"actor": [
{
"name: "John",
"image": "/upload/images/image.jpg"
},
{
"name: "Harry",
"image": "/upload/images/image.jpg"
},
]
},
I want to know how to make sub object in JSON with Exposed SQL!
You have to create a separate data class ActorDTO and map it in your code by making subquery. Also, I would advice to make Exposed Entities and then map them to your DAO, it can be much simpler but will add some boilerplate code.
Please check documentation
I'm trying to fetch and deserialize some data that is being hosted on github.
{
"Meals": [
{
"id": "1598044e-5259-11e9-8647-d663bd870b02",
"name": "Tomato pasta",
"quantity": [{
"quantity": 1 },
{
"quantity": 2
},
{
"quantity": 3
}],
"availableFromDate": "1605802429",
"expiryDate": "1905802429",
"info": "Vegetarian",
"hot": false,
"locationLat": 57.508865,
"locationLong": -6.292,
"distance": null
},
{
"id": "2be2d854-a067-43ec-a488-2e69f0f2a624",
"name": "Pizza",
"quantity": [{
"quantity": 1 },
{
"quantity": 2
},
{
"quantity": 3
}
],
"availableFromDate": "1605802429",
"expiryDate": "1905902429",
"info": "Meat",
"hot": false,
"locationLat": 51.509465,
"locationLong": -0.135392,
"distance": null
}
]
}
If I spin up a json-server locally then it works perfectly, so I know that my data class is not the problem. However when I try to do the same thing from that github link I get this error:
Error Domain=KotlinException Code=0 "No transformation found: class io.ktor.utils.io.ByteChannelNative -> class kotlin.collections.List
I have a feeling it might be to do with setting a ContentType or something along those lines but I haven't had any success specifying that so far.
Here is my code to make the request:
class MealApi {
private val httpClient = HttpClient {
install(JsonFeature) {
val json = Json { ignoreUnknownKeys = true }
serializer = KotlinxSerializer(json)
}
}
suspend fun getAllMeals(): List<Meal> {
return httpClient.get(endpoint)
}
}
and here is my data class just for completeness:
#Serializable
data class Meal(
#SerialName("id")
val id: String,
#SerialName("name")
val name: String,
#SerialName("quantity")
val quantity: List<Quantity>,
#SerialName("availableFromDate")
var availableFromDate: String,
#SerialName("expiryDate")
var expiryDate: String,
#SerialName("info")
val info: String,
#SerialName("hot")
val hot: Boolean,
#SerialName("locationLat")
val locationLat: Float,
#SerialName("locationLong")
val locationLong: Float,
#SerialName("distance")
var distance: Double? = null
)
#Serializable
data class Quantity(
#SerialName("quantity")
val quantity: Int
)
UPDATE
I've found that this server https://gitcdn.link/ allows you to serve your raw github files with the right Content-Type.
I've searched a lot how to change the server response headers (to change the plain/text one to application/json) but it seems that ktor actually doesn't allow to do that:
https://youtrack.jetbrains.com/issue/KTOR-617
https://youtrack.jetbrains.com/issue/KTOR-580
A nice way should be to allow the ResponseObserver to change the server response headers and pass through the modify response. But you can't actually.
Your problem depends, as you pointed out, from the fact that the raw github page provides an header Content-Type=plain/text instead of ContentType=application/json.
So IRL when you are running your API in a real server this won't occur as you'll take care to put the right content type at server level.
But if you want a workaround to this you could rewrite your api call in this way:
suspend fun getAllMealsWithFallback() {
var meals: Meals? = null
try {
meals = httpClient.get(endpoint)
} catch (e: NoTransformationFoundException) {
val mealsString: String = httpClient.get(endpoint)
val json = kotlinx.serialization.json.Json {
ignoreUnknownKeys = true
}
meals = json.decodeFromString(mealsString)
} finally {
println("Meals: ${meals?.meals}")
}
}
I had to add this class to conform to the json text you have provided in the github link.
#Serializable
data class Meals(
#SerialName("Meals")
val meals: List<Meal>,
)
Try this:
install(JsonFeature) {
serializer = KotlinxSerializer(KotlinJson { ignoreUnknownKeys = true })
acceptContentTypes = acceptContentTypes + ContentType.Any
}
If you'd like to accept all content types. Or Use ContentType.Text.Any, ContentType.Text.Html if you preferred.
In case the issue is the Content-Type:
You can alter the list of response content types, for which the KotlinxSerializer gets active, by extending the JsonFeature block to:
install(JsonFeature) {
val json = Json { ignoreUnknownKeys = true }
serializer = KotlinxSerializer(json)
receiveContentTypeMatchers += object : ContentTypeMatcher {
override fun contains(contentType: ContentType): Boolean =
contentType == ContentType("text", "plain")
}
}
If you're using Ktor 2.0, you would need ContentNegotiation plugin instead of JsonFeature.
For example if you use Gson:
install(io.ktor.client.plugins.contentnegotiation.ContentNegotiation) {
gson()
}
As a workaround for ktor version 2.0.3 you can create you own AppContentNegotiation class and in the scope.responsePipeline.intercept provide needed contentType
/**
* A plugin that serves two primary purposes:
* - Negotiating media types between the client and server. For this, it uses the `Accept` and `Content-Type` headers.
* - Serializing/deserializing the content in a specific format when sending requests and receiving responses.
* Ktor supports the following formats out-of-the-box: `JSON`, `XML`, and `CBOR`.
*
* You can learn more from [Content negotiation and serialization](https://ktor.io/docs/serialization-client.html).
*/
public class AppContentNegotiation internal constructor(
internal val registrations: List<Config.ConverterRegistration>
) {
/**
* A [ContentNegotiation] configuration that is used during installation.
*/
public class Config : Configuration {
internal class ConverterRegistration(
val converter: ContentConverter,
val contentTypeToSend: ContentType,
val contentTypeMatcher: ContentTypeMatcher,
)
internal val registrations = mutableListOf<ConverterRegistration>()
/**
* Registers a [contentType] to a specified [converter] with an optional [configuration] script for a converter.
*/
public override fun <T : ContentConverter> register(
contentType: ContentType,
converter: T,
configuration: T.() -> Unit
) {
val matcher = when (contentType) {
ContentType.Application.Json -> JsonContentTypeMatcher
else -> defaultMatcher(contentType)
}
register(contentType, converter, matcher, configuration)
}
/**
* Registers a [contentTypeToSend] and [contentTypeMatcher] to a specified [converter] with
* an optional [configuration] script for a converter.
*/
public fun <T : ContentConverter> register(
contentTypeToSend: ContentType,
converter: T,
contentTypeMatcher: ContentTypeMatcher,
configuration: T.() -> Unit
) {
val registration = ConverterRegistration(
converter.apply(configuration),
contentTypeToSend,
contentTypeMatcher
)
registrations.add(registration)
}
private fun defaultMatcher(pattern: ContentType): ContentTypeMatcher =
object : ContentTypeMatcher {
override fun contains(contentType: ContentType): Boolean =
contentType.match(pattern)
}
}
/**
* A companion object used to install a plugin.
*/
#KtorDsl
public companion object Plugin : HttpClientPlugin<Config, AppContentNegotiation > {
public override val key: AttributeKey<AppContentNegotiation> =
AttributeKey("ContentNegotiation")
override fun prepare(block: Config.() -> Unit): AppContentNegotiation {
val config = Config().apply(block)
return AppContentNegotiation(config.registrations)
}
override fun install(plugin: AppContentNegotiation, scope: HttpClient) {
scope.requestPipeline.intercept(HttpRequestPipeline.Transform) { payload ->
val registrations = plugin.registrations
registrations.forEach { context.accept(it.contentTypeToSend) }
if (subject is OutgoingContent || DefaultIgnoredTypes.any { it.isInstance(payload) }) {
return#intercept
}
val contentType = context.contentType() ?: return#intercept
if (payload is Unit) {
context.headers.remove(HttpHeaders.ContentType)
proceedWith(EmptyContent)
return#intercept
}
val matchingRegistrations =
registrations.filter { it.contentTypeMatcher.contains(contentType) }
.takeIf { it.isNotEmpty() } ?: return#intercept
if (context.bodyType == null) return#intercept
context.headers.remove(HttpHeaders.ContentType)
// Pick the first one that can convert the subject successfully
val serializedContent = matchingRegistrations.firstNotNullOfOrNull { registration ->
registration.converter.serialize(
contentType,
contentType.charset() ?: Charsets.UTF_8,
context.bodyType!!,
payload
)
} ?: throw ContentConverterException(
"Can't convert $payload with contentType $contentType using converters " +
matchingRegistrations.joinToString { it.converter.toString() }
)
proceedWith(serializedContent)
}
scope.responsePipeline.intercept(HttpResponsePipeline.Transform) { (info, body) ->
if (body !is ByteReadChannel) return#intercept
if (info.type == ByteReadChannel::class) return#intercept
// !!!!!!! Provide desired content type here
val contentType = <HERE> // default implementation is - context.response.contentType() ?: return#intercept
val registrations = plugin.registrations
val matchingRegistrations = registrations
.filter { it.contentTypeMatcher.contains(contentType) }
.takeIf { it.isNotEmpty() } ?: return#intercept
// Pick the first one that can convert the subject successfully
val parsedBody = matchingRegistrations.firstNotNullOfOrNull { registration ->
registration.converter
.deserialize(context.request.headers.suitableCharset(), info, body)
} ?: return#intercept
val response = HttpResponseContainer(info, parsedBody)
proceedWith(response)
}
}
}
}
And then install it for HttpClient
HttpClient {
install(AppContentNegotiation) {
json(json)
addDefaultResponseValidation()
}
}
In my case, this exception was thrown when I was attempting to access the body() potion of the HTTP response as such:
val httpResponse = httpClient.post(urlString) { ... }
val body = httpResponse.body<YourExpectedSerializableResponseType>
In the happy path scenario, the server would return a body that matched YourExpectedSerializableResponseType and everything would work as expected.
However, for a particular edge case that returned a different (still considered successful) status code, the server returned an empty response body. Since the client was expecting a response body and in this case, there wasn't any, this exception was thrown because it could not serialize an empty response body to the expected type YourExpectedSerializableResponseType.
My recommendation: In addition to ensuring your server is returning the type you expect to serialize/consume on your client, confirm your server is actually returning an object.
Internal Dialogue: I wonder if this exception could be more clear in this case as the issue is more so that the client expected a response body to exist and less so that an empty response body ("") couldn't be serialized into an expected type - especially given that an empty response body isn't even valid JSON. Hm. 🤔
I have the following data class, which stores two values, JSON and dataType:
data class DataTypeItem(
var json : String = "",
var dataType : Class<*> ?= null
)
I have the list defined in the following way:
val dataTypeList = mutableMapOf<String, DataTypeItem>()
dataTypeList.put( "item_1", DataTypeItem( json1, MyDataType::class.java ) )
dataTypeList.put( "item_2", DataTypeItem( json1, List<MyDataType>::class.java ) )
Please note that in one case I'm using the MyDataType as the DataType and in the other List < MyDataType >.
Now I would like to loop through each of the dataTypeList items and parse JSON for the given data type into it's model:
fun init()
{
dataTypeList.forEach {
dataTypeItem ->
val model = Gson().fromJson( dataTypeItem.value.json, dataTypeItem.value.dataType::class.java )
}
}
I'm using the following model:
data class dataTypeItem(
#SerializedName("sqlId")
val sqlId: String,
#SerializedName("name")
val name: String
)
But I keep getting an Runtime exception:
Attempted to deserialize a java.lang.Class. Forgot to register a type adapter?
In addition, in case it's a list, I need to call toList() on Gson().fromJSON(..):
fun init()
{
dataTypeList.forEach {
dataTypeItem ->
val model;
if( dataTypeItem.value.dataType::class.java is Array )
model = Gson().fromJson( dataTypeItem.value.json, dataTypeItem.value.dataType::class.java ).toList()
else
model = Gson().fromJson( dataTypeItem.value.json, dataTypeItem.value.dataType::class.java )
}
}
How can I pass the dataType dynamically and distinguish if it's a List/Array or straight up class? In addition, whenever I try to call toList(), I get an error that it's undefined.
If I specify the class directly, then it's working fine
var model = Gson().fromJson( json, DataTypeItem::class.java )
or
var model = Gson().fromJson( json, Array<DataTypeItem>::class.java )
but I need to be able to specify it dynamically as an argument
This code works fine:
val dataTypeMap = mapOf(
"item_1" to MyDataTypeItem("""{"sqlId" : "1", "name" : "a"}""", MyDataType::class.java),
"item_2" to MyDataTypeItem("""[{"sqlId" : "1", "name" : "a"}, {"sqlId" : "2", "name" : "b"}]""", Array<MyDataType>::class.java)
)
val result = dataTypeMap.map{ Gson().fromJson(it.value.json, it.value.dataType) }
I renamed DataTypeItem to MyDataTypeItem and dataTypeItem to MyDataType.
Why you need to call toList()? If it is really necessary you can do the following instead:
val result = dataTypeMap.map {
if (it.value.dataType?.isArray == true) Gson().fromJson<Array<*>>(it.value.json, it.value.dataType).toList()
else Gson().fromJson(it.value.json, it.value.dataType)
}
I am trying to use rapture.io Scala JSON parser to parse a JSON value (rows) that looks like this:
{
rows:
[
[
null,
"2016-11-16T15:43:18.000Z",
{
"p": 1,
"q": 2
},
null,
"Game highlights"
],
[
null,
"2007-10-09T01:52:29.000Z",
{
"p": 21,
"q": 99
},
"blaah",
"Game reviews"
]
}
My code looks like this:
import rapture.io._
import rapture.codec._
import rapture.json._
import rapture.data._
import rapture.uri._
import rapture.net._
import encodings.system
import jsonBackends.jawn._
class NotesDownloader () {
def download(): Unit = {
val src = uri"https://some_url".slurp[Char]
val jsonResponse = Json.parse(src)
val rows = jsonResponse.data.rows
val rowsBean = rows(0).as[Array[Member]]
println(jsonResponse)
}
case class Member(array: Array[Some[String]])
}
When I try to extract the complete data into Member, I get this exception:
Error:(40, 30) not enough arguments for method as: (implicit ext: rapture.data.Extractor[Array[NotesDownloader.this.Member],rapture.json.Json], implicit mode:
rapture.core.Mode[rapture.data.ExtractionMethods])mode.Wrap[Array[NotesDownloader.this.Member],rapture.data.DataGetException].
Unspecified value parameters ext, mode. val rowsBean = value.as[Array[Member]]
what am I missing?
As long as I know, you could use something like this.
Json.parse(str).as[List[Member]]) since it's a list not an a simple array.
The error message tells you that you need values for the implicit parameters ext and mode. It would be something like this:
implicit val ext = ...
implicit val mode = ...
val rowsBean = rows(0).as[Array[Member]] // this uses the above implicits
To begin with I would like to say sorry for long post, and I really appreciate those who still look into my problem.
I have a controller that should return a json-response with a structure like:
{
result: [
{
key: value,
key: value,
key: value,
key: [
{
key: value,
key: value,
key: value
},...
]
},....
]
}
However I have problems getting the Writes to work as I want.
Note. I will add comments under the line where I have problem.
object APIController extends Controller {
def feed() = Action {
val objects = repo.getObjects().toList
Ok(Json.toJson(Json.obj("result" -> Class_1.apply(objects).result)))
}
first off, if I don't make a Json.obj("result" -> List[objects]) the result key isn't shown in the JSON-result. If I add a Writer for that I get errors saying that the List[objects] must have a Writer. But if I write it like above it doesn't need a Writer for the List[objects]
case class Class_1 (result: Seq[Class_2])
object Class_1 {
def apply(objs: List[Object]) = {
var result:ListBuffer[Class_2] = ListBuffer[Class_2]()
for(obj <- objs) feedResult += Class_2.apply(code)
new Class_1(result.toList)
}
}
*this is where I would put the Writer for Class_1. But if I do this like
implicit val class1Writer = new Writes[Class_1] {
override def writes(o: Class_1): JsValue = Json.obj("result" -> o.result)
} I get the problems I mentioned earlier, that I suddenly need a Writes for a List[objects] of that type*
case class Class_2 (id: Long, id2: Long, listOfStuff: Seq[Class_3])
object Class_2 {
def apply(obj: Object) = {
var items: ListBuffer[Class_3] = ListBuffer[Class_3]()
for(obj1 <- obj.getListOfStuff()) items += Class_3.apply(obj1)
new Class_2(obj.firstID, obj.secID, items.toList)
}
}
implicit val class2Writes = new Writes[Class_2] {
override def writes(o: Class_2): JsValue = {
Json.obj(
"id" -> o.id,
"id2" -> o.id2,
"items" -> o.listOfStuff
)
}
}
*the "items" -> o.listOfStuff says it needs a Writes for a List[types in the list] but I have a Writes for the objects in the list (Class_3) and I don't need a Writes for when serializing a list of objects from Class_2, why is it behaving like this?*
case class Class_3 (id: Long, text: String)
object Class_3 {
def apply(obj: Object) = {
new Class_3(obj.id, obj.funnyText)
}
}
implicit val class3Writer = new Writes[Class_3] {
override def writes(o: Class_3): JsValue = {
Json.obj(
"id" -> o.id,
"text" -> o.text
)
}
}
}
The error I get from this code is:
No Json deserializer found for type Seq[Class_3]. Try to implement an implicit Writes or Format for this type.
[error] "items" -> o.listOfStuff
[error] ^
If I remove this line in the Writes it compiles and works.
And I think that's weird since the first list I serialize doesn't have a Writer, only for the objects in the list.
Does anyone know why it behaves like this?
What should I do to accomplish what I'm after? (I hope you see what I'm trying to do)
Thanks in advance.
Just put the implicit val class3Writer ahead of class2Writes