I need to validate messages on a brokers side.
I run cp-server (merely ran cp-all-in-one compose file).
created a topic
set confluent.value.schema.validation to true
registered a schema (JSON)
produced a message
It always fails. Why validation fails? Should I change configuration?
Schema:
{
"$id": "http://example.com/models/data-item-definition.json",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"description": "test data item 1",
"properties": {
"array_val": {
"items": {
"type": "string"
},
"type": [
"array",
"null"
]
},
"int_val": {
"type": "integer"
},
"string_val": {
"type": "string"
}
},
"required": [
"string_val",
"int_val"
],
"title": "data item",
"type": "object"
}
Message:
{
"string_val": "text",
"int_val": 10,
"array_val": ["one", "two", "three"]
}
An issue is that if confluent.value.schema.validation is true then to produce a message we need to send a magic byte and a schema ID at the beginning.
See Wire Format.
Related
I have a JSON object like:
{
"result": [
{
"name" : "abc",
"email": "abc.test#mail.com"
},
{
"name": "def",
"email": "def.test#mail.com"
},
{
"name": "xyz",
"email": "abc.test#mail.com"
}
]
}
and schema for this:
{
"definitions": {},
"$schema": "http://json-schema.org/draft-07/schema#",
"$id": "https://example.com/object1607582431.json",
"title": "Root",
"type": "object",
"required": [
"result"
],
"properties": {
"result": {
"$id": "#root/result",
"title": "Result",
"type": "array",
"default": [],
"uniqueItems": true,
"items": {
"$id": "#root/result/items",
"title": "Items",
"type": "object",
"required": [
"name",
"email"
],
"properties": {
"name": {
"$id": "#root/result/items/name",
"title": "Name",
"type": "string"
},
"email": {
"$id": "#root/result/items/email",
"title": "Email",
"type": "string"
}
}
}
}
}
}
I am looking for an option to check uniqueness for email irrespective of name. How I can validate that every email should be unique?
You can't. There are no keywords that let you compare one particular data value against another, other than uniqueItems, which compares an array element in toto against another.
The JsonSchema specification does not currently support this.
You can see the active GitHub issue here: https://github.com/json-schema-org/json-schema-vocabularies/issues/22
However, there are various extensions of JsonSchema that do validate unique fields within lists of objects.
If you happen to be using Python you can use the package (I created) JsonVL. It can be installed with pip install jsonvl and then run with jsonvl data.json schema.json.
Code examples in the GitHub repo: https://github.com/gregorybchris/jsonvl
I have following schema:
{
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Bulk Create Entity",
"title": "Bulk Create Entity",
"type": "object",
"properties": {
"idempotence_key": {
"description": "Idempotence Key",
"type": "string"
},
"requested_by": {
"description": "Requested By",
"type": "string"
},
"updated_by_process_id": {
"description": "Process id which is creating this entity.",
"type": "string"
},
"entity_creation_requests": {
"description": "Entity creation requests.",
"type": "array",
"minItems": 1,
"items": {
"title": "Entity Creation Request",
"type": "object",
"properties": {
"type": {
"description": "Entity Type",
"type": "string"
},
"taxonomies": {
"description": "Entity Taxonomies",
"type": "array",
"items": {
"title": "Taxonomy",
"type": "object",
"properties": {
"name": {
"description": "Taxonomy Name",
"type": "string"
},
"value": {
"description": "Taxonomy Value",
"type": "string"
}
}
}
}
}
}
}
},
"required": [
"idempotence_key",
"requested_by",
"updated_by_process_id",
"entity_creation_requests"
]
}
Here, the root level payload is an object which has a key "entity_creation_requests" which is an array of objects which in turn have an array property "taxonomies" which contains a list of key-value pairs.
Now depending on the "type" of the request under "entity_creation_requests", I want to validate the presence of certain keys in the taxonomies list.
For example, for the creation request of type "product", I want keys "MRP", "seller" etc. to be present in the taxonomies list.
Can we achieve this using JSON schema validator?
{Here is something I have created: https://codebeautify.org/jsonviewer/cb80c728}
Are there any other alternatives?
I am using this in an spring boot application (Java).
I have the following schema:
{
"$schema": "http://json-schema.org/schema#",
"$id": "http://api.hobnob.social/schemas/users/create.json",
"title": "Create User Schema",
"description": "For validating client-provided create user object",
"type": "object",
"properties": {
"email": {
"type": "string",
"format": "email"
},
"password": { "type": "string" },
"profile": { "$ref": "profile.json#" }
},
"required": ["email", "password"],
"additionalProperties": false
}
{
"$schema": "http://json-schema.org/schema#",
"$id": "http://api.hobnob.social/schemas/users/profile.json",
"title": "User Profile Schema",
"description": "For validating client-provided user profile object when creating and/or updating an user",
"type": "object",
"properties": {
"bio": { "type": "string" },
"summary": { "type": "string" },
"name": {
"type": "object",
"properties": {
"first": { "type": "string" },
"last": { "type": "string" },
"middle": { "type": "string" }
},
"additionalProperties": false
}
},
"additionalProperties": false
}
I am using ajv to validate against it. I am getting the expected results in almost all cases. But when validating a json with either the bio or summary fields included (with type of string), no response comes from ajv at all.
E.g. I attempt to validate
{
"email": "e#ma.il",
"password": "password",
"profile": {
"name": {
"first": "firstname"
},
"bio":"this is a bio"
}
}
and no response at all comes back.
I tried consolidating the schema but that made no difference. I'm hoping I have made some simple beginner mistake that someone may spot! I have spent many hours trying to work out what is going wrong, but after all my debugging I am no further forward.
I got this working somehow, but not sure why it started working.
In my test script I added a line to delete the test index from elasticsearch. After that, all tests passed. I then removed the new line from my test script to see if it would stop working again, but it didn't.
I'm guessing the problem was somehow related to elasticsearch...
I'm trying to use this avro schema
{
"type": "record",
"name": "ComplianceEntity",
"namespace": "com.linkedin.events.metadata",
"fields": [
{
"name": "fieldPath",
"type": "string"
},
{
"name": "complianceDataType",
"type": {
"type": "enum",
"name": "ComplianceDataType",
"symbols": [
"NONE",
"MEMBER_ID"
],
"symbolDocs": {
"NONE": "None of the following types apply",
"MEMBER_ID": "ID for LinkedIn members"
}
}
},
{
"name": "complianceDataTypeUrn",
"type": [
"null",
"string"
],
"default": null
},
{
"name": "fieldFormat",
"type": [
"null",
{
"type": "enum",
"name": "FieldFormat",
"symbols": [
"NUMERIC"
],
"symbolDocs": {
"NUMERIC": "Numerical format, 12345"
},
"doc": "The field format"
}
]
},
{
"name": "securityClassification",
"type": "SecurityClassification"
},
{
"name": "valuePattern",
"default": null,
"type": [
"null",
"string"
]
}
]
}
To generate and avro file using the avro-tools:
java -jar ./avro-tools-1.8.2.jar compile schema ComplianceEntity.avsc .
But I am getting the following error message:
Exception in thread "main" org.apache.avro.SchemaParseException: "SecurityClassification" is not a defined name. The type of the "securityClassification" field must be a defined name or a {"type": ...} expression.
Could anyone tell, why SecurityClassification is not identified as a defined name?
You are using it as type of your field, however you are not defining it properly like for complianceDataType, that's the reason why you are getting the avro exception
{
"name": "securityClassification",
"type": "SecurityClassification"
}
Make sure that if you have more than 1 Schema, you pass all of them, especially dependency schemas. It is supported from AVRO 1.5.3 https://issues.apache.org/jira/browse/AVRO-877.
java -jar ./avro-tools-1.8.2.jar compile schema SecurityClassification.avsc ComplianceEntity.avsc .
I have the JSON schemas below:
Schema A:
{
"$schema": "http://json-schema.org/draft-04/hyper-schema#",
"title": "A",
"type": "object",
"properties": {
"id": {
"type": "string"
},
"name": {
"type": "string"
},
"entityBId": {
"type": "string"
}
},
"required": [
"name",
"entityBId"
],
"links": [
{
"rel": "B",
"href": "myapi/EntityB?id={entityBId}"
}
]
}
Schema B :
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "B",
"type": "object",
"properties": {
"id": {
"type": "string"
},
"name": {
"type": "string"
}
},
"required": [
"name"
]
}
I'm trying to figure out if there is a way to run something like a integrity check based in a JSON Schema with external links/references.
For instance: When I receive an Object A with a entityBId = 1, I'd like to fetch this entity B running a GET in the endpoint declared in the links href and check if there is a valid object from the received id.
It will run like a deep validation and can be useful in a scenario without a defined DB schema.
As suggested by Raj Kamal, I made my own "link validation".
There is a function that looks for link attribute on schemas and validate the externals references directly on database and throw an exception if a valid reference is not found.