I'm trying to put several event types in the same Kafka topic using the JSON format, but in the Producer implementation I'm always getting org.apache.kafka.common.errors.SerializationException: Error serializing JSON message. Seems that the annotation #Schema isn't working as expected is like the schema defined by the annotation isn't enriched properly and in the method that validates the backward compatibility the schema defined by my event has the schemaObj empty and the result is not compatible and fails.
My event:
#Schema(
value = "1",
refs = Array(new SchemaReference(name = "event", subject = "event"))
)
case class Event(#BeanProperty id: String,
#BeanProperty name: String)
Producer:
def send(): Unit = {
val props = new Properties() {
put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092")
put(
ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringSerializer"
)
put(
ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
"io.confluent.kafka.serializers.json.KafkaJsonSchemaSerializer"
)
put("auto.register.schemas", "false")
put("use.latest.version", "true")
put("schema.registry.url", "http://127.0.0.1:8081")
put("json.fail.invalid.schema", "true")
}
val producer = new KafkaProducer[String, Event](props)
val topic = "all-json"
val key = "key1"
val event = Event("id", "name")
val record = new ProducerRecord[String, Event](topic, key, event)
producer.send(record).get
}
By the command line, I can perfectly produce the events. The JSON Schema is modeled by
{
"oneOf": [
{ "$ref": "Event.schema.json" },
{ "$ref": "EventB.schema.json" }
]
}
...
the dependencies of confluent used are the version 6.0.1.
Do you know what is the issue?
I tried convert to data class to map. (Under the code is just example code)
data class User (
val name : String = "",
val age : Int = 0,
val deviceGroup: MutableSet<DeviceGroup> = mutableSetOf()
)
data class DeviceGroup (
val name : String = "",
val deviceLink : MutableSet<DeviceLink> = mutableSetOf()
)
data class DeviceLink (
val id : Int = 0,
val device : Device
)
data class Device (
val devId : Int = 0,
val name : String = ""
)
fun main (request : HttpServletRequest) {
val currentUser = request.session.getAttribute("user") as User
val data = userRepository.findByName(currentUser.name)
// return currentUser
// result is {name="test", age=17, deviceGroup = [{name="group1"}, {name="group2"}]}
// I want deserialization data class to Map
val response = data.deviceGroup.toMap()
response.deivceGroup.forEach {
// And add new key, pair
it.add(Map<String, MutableSet<Device>>("devices", mutableSetOf()))
// Lastly, I want put in the value
deviceGroupRepository.findByName(it.name).deviceLink.forEach {
it.devices.add(this)
}
}
return response
}
if just return the data value, that's result is "{name="test", age=17, deviceGroup = [{name="group1"}, {name="group2"}]}"
How to convert to data class to Map object and add new key pair?
Use associate to turn a collection into a Map
The Kotlin standard library provides a function called associate which will take a collection of objects and transform them into a map. It takes one argument, which is a function specifying what the keys and values of the map should be.
For example, in your case, you would call it like this:
val response = data.deviceGroup.associate { it.name to it.deviceLink }
It will return a Map<String, MutableSet<DeviceLink>> where the key is the name of the device group and the value is the deviceLink set.
The easiest way to add new values is simply to append them with the + operator.
val response = data.deviceGroup.associate {
it.name to it.deviceLink
} + mapOf("device" to emptySet())
If you need more control than that, you could use .toMutableMap() so new entries can be added using put.
val response = data.deviceGroup.associate {
it.name to it.deviceLink
}.toMutableMap()
response.put("device", emptySet())
JSON Data Parsing Using Retofit2 and Rxjava2. This Data get In ArrayList successfully. its ArrayList Size is Nine but its display only two Record in Table. After Two Record its Kotlin.NullPointerException.
JSON Data:
{"success":1,"salesGst":[{"Cmp_Name":"ABC","GSTIN":"AAAA","FirmName":"SALES GJ","ChallanNo":"1","ChallanDate":"2019-03-15 00:00:00","ChallanAmount":"2778.75","TaxTotal":"2778.75","InvoiceType":"Retail Invoice","CGSTTotal":"0.0","PartyGST":"CDE","SGSTTotal":"0.0","IGSTTotal":"0.0"},{"Cmp_Name":"ABC","GSTIN":"AAAA","FirmName":"SALES GJ","ChallanNo":"1","ChallanDate":"2019-03-13 00:00:00","ChallanAmount":"2203.0","TaxTotal":"2118.5","InvoiceType":"Tax Invoice","CGSTTotal":"52.96","PartyGST":"CDE","SGSTTotal":"52.96","IGSTTotal":"0.0"},{"Cmp_Name":"ABC","GSTIN":"AAAA","FirmName":"VIKAS","ChallanNo":"2","ChallanDate":"2019-03-16 00:00:00","ChallanAmount":"6975.0","TaxTotal":"6975.0","InvoiceType":"Retail Invoice","CGSTTotal":"0.0","PartyGST":null,"SGSTTotal":"0.0","IGSTTotal":"0.0"},{"Cmp_Name":"ABC","GSTIN":"AAAA","FirmName":"SALES MH","ChallanNo":"2","ChallanDate":"2019-03-13 00:00:00","ChallanAmount":"420.0","TaxTotal":"403.75","InvoiceType":"Tax Invoice","CGSTTotal":"0.0","PartyGST":"ABC","SGSTTotal":"0.0","IGSTTotal":"20.19"},{"Cmp_Name":"ABC","GSTIN":"AAAA","FirmName":"SALES GJ","ChallanNo":"3","ChallanDate":"2019-03-14 00:00:00","ChallanAmount":"4788.0","TaxTotal":"4560.0","InvoiceType":"Tax Invoice","CGSTTotal":"114.0","PartyGST":"CDE","SGSTTotal":"114.0","IGSTTotal":"0.0"},{"Cmp_Name":"ABC","GSTIN":"AAAA","FirmName":"SALES GJ","ChallanNo":"4","ChallanDate":"2019-03-15 00:00:00","ChallanAmount":"241.9","TaxTotal":"230.38","InvoiceType":"Tax Invoice","CGSTTotal":"5.76","PartyGST":"CDE","SGSTTotal":"5.76","IGSTTotal":"0.0"},{"Cmp_Name":"ABC","GSTIN":"AAAA","FirmName":"SALES GJ","ChallanNo":"5","ChallanDate":"2019-03-15 00:00:00","ChallanAmount":"5563.68","TaxTotal":"5101.5","InvoiceType":"Tax Invoice","CGSTTotal":"231.28","PartyGST":"CDE","SGSTTotal":"231.28","IGSTTotal":"0.0"},{"Cmp_Name":"ABC","GSTIN":"AAAA","FirmName":"SALES GJ","ChallanNo":"6","ChallanDate":"2019-03-16 00:00:00","ChallanAmount":"13238.0","TaxTotal":"12459.25","InvoiceType":"Tax Invoice","CGSTTotal":"389.29","PartyGST":"CDE","SGSTTotal":"389.29","IGSTTotal":"0.0"},{"Cmp_Name":"ABC","GSTIN":"AAAA","FirmName":"SALES MH","ChallanNo":"7","ChallanDate":"2019-03-16 00:00:00","ChallanAmount":"2074.0","TaxTotal":"1975.0","InvoiceType":"Tax Invoice","CGSTTotal":"0.0","PartyGST":"ABC","SGSTTotal":"0.0","IGSTTotal":"98.75"}]}
Please Guide Me,After Getting How to Show in TableLayout.
In ArrayList Nine Record but in Table show only Two Record another seven record is not display. in third record taxtotal give kotlin.nullpointerException. what missing?
private fun displaySalesGSTData(salesGSt : List<SalesGST>) {
salesGST = SalesGST()
tvSalesCompanyName.setText(salesGSt.get(1).Cmp_Name)
tvGSTIN.setText(salesGSt.get(1).GSTIN)
val rowHeader = TableRow(this#Sales)
rowHeader.setBackgroundColor(Color.parseColor("#c0c0c0"))
rowHeader.setLayoutParams(TableLayout.LayoutParams(TableLayout.LayoutParams.MATCH_PARENT,
TableLayout.LayoutParams.WRAP_CONTENT))
val headerText = arrayOf<String>("Sr.No.", "Invoice Type", "Bill No.", "Bill Date", "Firm Name", "GST NO","TAX Total","CGST","SGST","IGST","Net Amount")
for (c in headerText)
{
val tv = TextView(this#Sales)
tv.setLayoutParams(TableRow.LayoutParams(TableRow.LayoutParams.WRAP_CONTENT,
TableRow.LayoutParams.WRAP_CONTENT))
tv.setGravity(Gravity.CENTER)
// tv.setBackgroundResource(R.drawable.table_header)
tv.setTextColor(Color.parseColor("#3F51B5"))
tv.setTextSize(18F)
tv.setPadding(5, 5, 5, 5)
tv.setText(c)
rowHeader.addView(tv)
}
tableMarks.addView(rowHeader)
for (j in 0 until salesGSt.size)
{
/*val jsonObject1 = jsonArray.getJSONObject(j)
val date = jsonObject1.getString("ExamDate")
val inputFormatter1 = SimpleDateFormat("yyyy-MM-dd HH:mm:ss")
val date1 = inputFormatter1.parse(date)
val outputFormatter1 = SimpleDateFormat("dd-MMM-yyyy")
ExamDate = outputFormatter1.format(date1)*/
/* String replaceDate = date.replace("/Date(", "").replace(")/", "");
Long getDate = Long.valueOf(replaceDate);
ExamDate = dateFormat.format(getDate);*/
/*Subject = jsonObject1.getString("subject")
ExamName = jsonObject1.getString("ExamName")
TotalMark = jsonObject1.getLong("TotalMarks")
PassingMark = jsonObject1.getLong("PassingMarks")
Mark = jsonObject1.getLong("Marks")*/
var fName : String = salesGSt.get(j).FirmName!!
var invoice : String = salesGSt.get(j).InvoiceType!!
var bill_no : String = salesGSt.get(j).ChallanNo!!
var bill_date : String = salesGSt.get(j).ChallanDate!!
var gst_no : String = salesGSt.get(j).PartyGST!!
var tax_total : Double = salesGSt.get(j).TaxTotal!!.toDouble()
var cgst : String = salesGSt.get(j).CGSTTotal!!
var igst : String = salesGSt.get(j).IGSTTotal!!
var sgst : String = salesGSt.get(j).SGSTTotal!!
var net_amount : String = salesGSt.get(j).ChallanAmount!!
var sr : Int = j + 1
// dara rows
val row = TableRow(this#Sales)
row.setLayoutParams(TableLayout.LayoutParams(TableLayout.LayoutParams.MATCH_PARENT,
TableLayout.LayoutParams.WRAP_CONTENT))
val colText = arrayOf<String>(sr.toString(),(invoice), bill_no, bill_date, fName, gst_no, tax_total.toString(),cgst,sgst,igst,net_amount)
for (text in colText)
{
val tv = TextView(this#Sales)
tv.setLayoutParams(TableRow.LayoutParams(TableRow.LayoutParams.WRAP_CONTENT,
TableRow.LayoutParams.WRAP_CONTENT))
tv.setGravity(Gravity.CENTER)
// tv.setBackgroundResource(R.drawable.table_shape)
tv.setTextSize(18F)
tv.setTextColor(Color.parseColor("#3F51B5"))
tv.setPadding(5, 5, 5, 5)
tv.setText(text)
row.addView(tv)
}
tableMarks.addView(row)
}
}
The 3rd item, salesGst[2], is "PartyGST": null. Your json deserializer library won't handle non-null fields as it's written in Java. I assume you have a data class where PartGST is defined as non-null, yet the deserializer will still parse it as null. Therefore, when you access PartyGST then you will receive a NullPointerException because Kotlin is expecting it to be non-null. This is a good article, which explains in more detail:
I've trusted you! You promised no null pointer exceptions!
A solution to this would be to have two models. The DTO (for JSON response) where all fields are optional and your internal model (used by your app), where you define which fields you want to be optional. Then you can have a mapper to handle the case where a field is null in the DTO, but non-null in your internal model:
// Models for your API response
data class SalesGstsDTO(val gsts: List<GstDTO>)
data class GstDTO(val name: String?, val surname: String?)
// Internal models used by your app
data class SalesGsts(val gsts: List<Gst>)
data class Gst(val name: String, val surname: String?)
class SalesGstDTOMapper {
fun mapToSalesGsts(salesGstsDTO: SalesGstsDTO): SalesGsts {
val gsts = mutableListOf<Gst>()
salesGstsDTO.gsts.map {
val name = it.name ?: return#map // Skips this item. You could handle this how you wish
val surname = it.surname
val gst = Gst(name, surname)
gsts.add(gst)
}
return SalesGsts(gsts)
}
}
This also allows you to decouple your app from the JSON response.
To begin with I would like to say sorry for long post, and I really appreciate those who still look into my problem.
I have a controller that should return a json-response with a structure like:
{
result: [
{
key: value,
key: value,
key: value,
key: [
{
key: value,
key: value,
key: value
},...
]
},....
]
}
However I have problems getting the Writes to work as I want.
Note. I will add comments under the line where I have problem.
object APIController extends Controller {
def feed() = Action {
val objects = repo.getObjects().toList
Ok(Json.toJson(Json.obj("result" -> Class_1.apply(objects).result)))
}
first off, if I don't make a Json.obj("result" -> List[objects]) the result key isn't shown in the JSON-result. If I add a Writer for that I get errors saying that the List[objects] must have a Writer. But if I write it like above it doesn't need a Writer for the List[objects]
case class Class_1 (result: Seq[Class_2])
object Class_1 {
def apply(objs: List[Object]) = {
var result:ListBuffer[Class_2] = ListBuffer[Class_2]()
for(obj <- objs) feedResult += Class_2.apply(code)
new Class_1(result.toList)
}
}
*this is where I would put the Writer for Class_1. But if I do this like
implicit val class1Writer = new Writes[Class_1] {
override def writes(o: Class_1): JsValue = Json.obj("result" -> o.result)
} I get the problems I mentioned earlier, that I suddenly need a Writes for a List[objects] of that type*
case class Class_2 (id: Long, id2: Long, listOfStuff: Seq[Class_3])
object Class_2 {
def apply(obj: Object) = {
var items: ListBuffer[Class_3] = ListBuffer[Class_3]()
for(obj1 <- obj.getListOfStuff()) items += Class_3.apply(obj1)
new Class_2(obj.firstID, obj.secID, items.toList)
}
}
implicit val class2Writes = new Writes[Class_2] {
override def writes(o: Class_2): JsValue = {
Json.obj(
"id" -> o.id,
"id2" -> o.id2,
"items" -> o.listOfStuff
)
}
}
*the "items" -> o.listOfStuff says it needs a Writes for a List[types in the list] but I have a Writes for the objects in the list (Class_3) and I don't need a Writes for when serializing a list of objects from Class_2, why is it behaving like this?*
case class Class_3 (id: Long, text: String)
object Class_3 {
def apply(obj: Object) = {
new Class_3(obj.id, obj.funnyText)
}
}
implicit val class3Writer = new Writes[Class_3] {
override def writes(o: Class_3): JsValue = {
Json.obj(
"id" -> o.id,
"text" -> o.text
)
}
}
}
The error I get from this code is:
No Json deserializer found for type Seq[Class_3]. Try to implement an implicit Writes or Format for this type.
[error] "items" -> o.listOfStuff
[error] ^
If I remove this line in the Writes it compiles and works.
And I think that's weird since the first list I serialize doesn't have a Writer, only for the objects in the list.
Does anyone know why it behaves like this?
What should I do to accomplish what I'm after? (I hope you see what I'm trying to do)
Thanks in advance.
Just put the implicit val class3Writer ahead of class2Writes
I want to be able to parse a string to an object that I can access using the dot notation e.g. myobject.property, instead of the array notation e.g. myobject['property']. The array notation works fine. Here's what I have so far.
I have some XML:
<level1 name="level1name">
<level2 type="level2Type">
<entry>level2entry</entry>
<entry>level2entry</entry>
</level2>
</level1>
Which converts to the JSON:
{
"level1": {
"name": "level1name",
"level2": {
"type": "level2Type",
"entry": [
"level2entry",
"level2entry"
]
}
}
}
I have the following Dart code:
Object jsonObject = JSON.parse("""{
"level1": {
"name": "level1name",
"level2": {
"type": "level2Type",
"entry": [
"level2entry",
"level2entry"
]
}
}
}
""");
print("my test 1 == ${jsonObject}");
print("my test 2 == ${jsonObject['level1']}");
print("my test 3 == ${jsonObject['level1']['name']}");
which produce the (desired) output:
my test 1 == {level1: {name: level1name, level2: {type: level2Type, entry: [level2entry, level2entry]}}}
my test 2 == {name: level1name, level2: {type: level2Type, entry: [level2entry, level2entry]}}
my test 3 == level1name
But when I try:
print("my test 1 == ${jsonObject.level1}");
I get the following:
Exception: NoSuchMethodException : method not found: 'get:level1'
Receiver: {level1: {name: level1name, level2: {type: level2Type, entry: [level2entry, level2entry]}}}
Arguments: []
Stack Trace: 0. Function: 'Object.noSuchMethod' url: 'bootstrap' line:717 col:3
Ideally, I want an object that I can access using the dot notation and without the compiler giving warning about Object not having property. I tried the following:
class MyJSONObject extends Object{
Level1 _level1;
Level1 get level1() => _level1;
set level1(Level1 s) => _level1 = s;
}
class Level1 {
String _name;
String get name() => _name;
set name(String s) => _name = s;
}
...
MyJSONObject jsonObject = JSON.parse("""{
"level1": {
"name": "level1name",
"level2": {
"type": "level2Type",
"entry": [
"level2entry",
"level2entry"
]
}
}
}
""");
...
print("my test 1 == ${jsonObject.level1.name}");
but instead of giving me 'level1name' as hoped, I get:
Exception: type 'LinkedHashMapImplementation<String, Dynamic>' is not a subtype of type 'MyJSONObject' of 'jsonObject'.
What am I doing wrong here? Is there any way to do what I'm trying? Thanks.
At the moment, JSON.parse only returns Lists (array), Maps, String, num, bool, and null
(api ref).
I suspect that until reflection makes it way into the language, it won't be able to re-construct objects based upon the keys found in json.
You could, however, create a constructor in your MyJsonObject which took a string, called JSON.parse internally, and assigned the various values.
Something like this works in the dart editor:
#import("dart:json");
class Level2 {
var type;
var entry;
}
class Level1 {
var name;
var level2;
}
class MyJSONObject {
Level1 level1;
MyJSONObject(jsonString) {
Map map = JSON.parse(jsonString);
this.level1 = new Level1();
level1.name = map['level1']['name'];
level1.level2 = new Level2();
level1.level2.type = map['level1']['level2']['type'];
//etc...
}
}
main() {
var obj = new MyJSONObject(json);
print(obj.level1.level2.type);
}
A non trivial version would needs some loops and possible recursion if you had deeper nested levels.
Update: I've hacked together a non-trivial version (inspired by the post below), it's up on github (also taking Seth's comments re the constructor):
Chris is completely right. I will only add that the JSON parser could be modified to return a little richer object (something like JsonMap instead of pure Map) that could allow jsonObj.property by implementing noSuchMethod. That would obviously perform worse than jsonObj['property'].