Integration of schema registry - json

Question

I am developing an application(B) which uses Schema Registry and allows clients(A) to send data to any other platform(C). General design:
Client A -> Platform B -> Destination C
So I am not necessarily publishing messages to kafka.
Is there any documentation on how to integrate and use Schema registry with a custom application which is not publishing to kafka? All the documentation I could find is around using registry with kafka.
For this setup, I am finding it difficult to finalize the serialization format to use- if I use avro/protobuf, it requires a corresponding class/message in application B. This means clients will need to update B every time a new schema is added or an old one updated. For JSON, such a class should not be needed (I guess?) since it is schemaless.

Answer 1

What is the intended schema format for your solution? If a binary structured format, then serializing key/value (Java Objects - > byte Array), you definitely need a Schema Registry integrated in your application.
Refer : https://docs.confluent.io/platform/current/schema-registry/index.html
Again for #2, Schema Registry provides you a better Schema Management capability in terms of Schema Evolution and compatibility checks, so you need not to worry much about updating your KAFKA Clients whenever a schema evolves.

Answer 2

Answer to your questions:
Documentation in Schema Registry API Reference. Here is an example how to add key/value schemas for Platform B.
Once the key/value schemas are setup in the Schema Registry, then you have to configure the connector properties to serialize/deserialize the message.
Connector properties to serialize/deserialize avro messages.
(Connector sink example)
{
"name": "eraseme_connector_schema_test_v01",
"config": {
"connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
"tasks.max": "1",
"key.converter": "io.confluent.connect.avro.AvroConverter",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"topics": "eraseme_topic_schema_test_v05",
"connection.url": "jdbc:mysql://xxx.xxxx.xxxx",
"connection.user": "USER",
"connection.password": "PASSOWRD",
"insert.mode": "upsert",
"delete.enabled": "true",
"pk.mode": "record_key",
"pk.fields": "YOUR_KEY_1, YOUR_KEY_1,...,YOUR_KEY_N",
"auto.create": "true",
"auto.evolve": "true",
"value.converter.schema.registry.url": "http://schema-registry:8081",
"key.converter.schema.registry.url": "http://schema-registry:8081"
}
}

Answer 3

All the documentation I could find is around using registry with kafka
Because the Confluent Registry stores schemas in a Kafka topic. So, you'll therefore need a Kafka installation, even if it is not used by your services. Note: There are other Schema Registry implementations.
finding it difficult to finalize the serialization format to use
Doesn't really matter; the Confluent Registry also supports defining your own. What actually matters is how supported that format is within your clients. E.g. Avro support in Golang or .NET isn't the greatest.
requires a corresponding class/message in application B. This means clients will need to update B every time a new schema is added or an old one updated
So? Do you never upgrade other 3rd party libraries for new methods/fixes? Why would you treat your schemas any different?
The Registry enforces backwards compatible changes, by default. Schema upgrades aren't required unless you actually need the new data being sent.
For JSON, such a class should not be needed (I guess?) since it is schemaless
The Confluent Schema Registry uses JSONSchema, so it is not schemaless.
Kafka itself supports a form of JSON in the Connect API that has a schema, but it is not integrated with any Registry, and as such cannot guarantee any schema evolution rules.
any documentation on how to integrate and use Schema registry with a custom application which is not publishing to kafka
You'd use the REST API directly.