I am trying to produce JSON records from my Scala Producer code to Kafka topic. It is successfully generated, however I am not able to register the schema and do schema evolution compatibility checks.
I am not able to find any proper code/doc references. How do I register my JSON schema and consume by connecting to schema registry client and check for the compatibilities.
Any suggestions please? (more about what am trying Class io.confluent.kafka.serializers.json.KafkaJsonSchemaSerializer could not be found)
Compatibilities are checked server-side automatically upon producing, which in-turn registers the schemas, by default.
You provide schema.registry.url to the Producer and Consumer properties when using the clients and the JSONSchema(De)Serializer classes
Related
I want to use apache avro schema's for data serialization and deserialization.
I want to use it with json encoding.
I want to put several of this serialized objects using different schemas to the same "source" (it's a kafka topic).
When I read it back I have the need to be able to resolve the right schema for the current data entry.
But the serialized data don't have any schema information in it. And to test all possible schema's for compatibility (kind of a duck typing approach) would be pretty unclean and error prone (for data which fits to multiple schemas it would be unclear which one to take)
I'm currently thought about putting the namespace and object name inside the json data programatically. But such a solution would not belong to the avro standard and it would open a new error scenario where it's possible to put the wrong schema namespace and/or object name inside the data.
I'm wondering if there would be a better way. Or if I have a general flaw in my solution.
Background: I want to use it for kafka message but don't want to use the schema registry (don't wan't to have a new single point of failure). I also still want to have KSQL support which is only available for json format or for using avro with the schema registry)
I publish messages to a kafka topic (outputTopic) in a format which my subscribers can understand. I now wish to modify the format of these messages in a way which will break the existing topic consumers.
For example, I post objects, serialised in json format, but need to change the objects, and therefore the schema.
What is the best way to manage this type of change? Should I alter the producer so that it publishes to a new topic (outputTopic2)? Are there any better ways to manage this?
Avro schemas will not solve the problem. You could update the consumers to handle both old and new versions whether or not there are schemas.
Instead, keep your producer as it is. Deploy an updated version that reads from the same data source and publishes data to a new topic with the new updated format.
Allow consumers to migrate from the old version to the new version before finally killing the old one.
One clean way to do according to me, is to use Schema registry / Apache Avro. Depending on how you use it, it will help you to guarantee backward / forward compatibility.
I'm preparing laboratory script for presenting features of of OCB, however
there is a misconception on how OCB references to Data Models defined by
FIWARE. I'm aware of parameter 'type', which is supplied along with new
entity. The problem is connected to OCB. How does it difference between data models?
I can add variables from different data model which don't match. I
can't find the controller for it and if there is no any that's really shame. The problem I got is connected with situation:
Creating entity with declared type for instance: Device
Adding some parameter, which is not specified in Device schema. It could be dateLastWatering from Garden data model.
It doesn't make any sense, how the OCB reacts to that?
From my experience it just "swallows" the updated content as long as semantic of command is correct.
I received information that a restriction can be implemented in application layer, however in my opinion it should be supplied along with OCB already.
That way model can be mistaken and the example is not really educational. It
could be useful if after defining a model for entity, OCB could interact
somehow with user and check, or debug what it receives. That way it would be
more professional I guess and fore sure more safe to use. If OCB doesn't have
that feature is there a possibility for further development?
Orion Broker is totally agnostic with regards to Data Models. Orion is not intended to check schemas or perform data validation. In fact, it is a nice feature that the schema of the Data is free and up to applications.
However, FIWARE defines Data Models to promote harmonization in certain domains, for instance smart cities. So it is nice to reuse or extend existing data models for the sake of data portability.
Currently developing an app for observing changes in a JCR Repository and replicate this changes to another repository. I can't rely on a cluster JCR cause the two repositories won't be on the same network, and the connection is not reliable, so my implementation takes care of the communication problem using a REST API between the two servers, and give some fault tolerance guarantees. The problem is I need to serialize nodes, preferably in a JSON format, and parse that JSON into a Node on the other side.
I've tried Apache Sling, using some internal classes it can serialize perfectly the node into a JSON format, but I can't seem to find a way to deserialize into a Node Object on the other side. Any ideas?
The Apache Sling POST Servlet can import JSON content, see "importing content structures" at http://sling.apache.org/documentation/bundles/manipulating-content-the-slingpostservlet-servlets-post.html
Another option is the new replication module that has recently been contributed, that might be just what you need. It's at http://svn.apache.org/repos/asf/sling/trunk/contrib/extensions/replication/ , there might not be any docs yet but if you ask on the Sling users mailing list I'm sure you'll get help on how to use it.
I have a use case where I need to validate JSON objects against a schema that can change real time..
Let me explain my requirements..
I persist JSON objects (MongoDB).
Before persisting I MUST validate the data type of some of the
fields of JSON objects (mentioned in #1) against a schema.
I persist the schema in mongodb.
I always validate the JSON objects against the latest schema available in db. (so I dont think it matters much even if the schema can change in real time for me it is kinda static).
I am using a J2EE stack (Spring Framework).
Can anyone guide me here..?
Another way of doing it is to use an external library https://github.com/fge/json-schema-validator to do the work for you. The one I proposed supports draft 4 of JSON Schema.
The IBM DataPower appliance has JSON Schema validation support. This will allow you to offload validation to an appliance that is designed for it along with routing of data within te enterprise.