Use JSON schema in Pubsub topic subscription - json

Can we create Pubsub topic subscription with a json schema and not with a avro schema file?
does the file need to be of extension ".avsc"? can we use ".json" instead?
Data is avro.
AVSC files contents looks like:
{
"type" : "record",
"name" : "twitter_schema",
"namespace" : "com.miguno.avro",
"fields" : [ {
"name" : "username",
"type" : "string",
"doc" : "Name of the user account on Twitter.com"
}, {
"name" : "tweet",
"type" : "string",
"doc" : "The content of the user's Twitter message"
}, {
"name" : "timestamp",
"type" : "long",
"doc" : "Unix epoch time in seconds"
} ],
"doc:" : "A basic schema for storing Twitter messages"
}
Json files contents looks like:
{
"schema": {
"fields": [{
"name": "username",
"type": "string",
"doc": "Name of the user account on Twitter.com"
}, {
"name": "tweet",
"type": "string",
"doc": "The content of the user's Twitter message"
}, {
"name": "timestamp",
"type": "long",
"doc": "Unix epoch time in seconds"
}]
}
}
Getting the below error when i tried to create subscription with json schema.

Only Avro and Protocol Buffer schemas are supported with Cloud Pub/Sub schemas. The extension of the file does not matter. Note that the Avro schema is valid JSON. The second schema is not a valid AVRO schema, which is why an error is being returned.

Related

How to declare an optional field in Google Pub/Sub Topic Schema?

I'm trying to create a Pub/Sub Schema Topic on AVRO respecting the indications on the documentation with "default" : null indication.
I declared my optional field this way :
{
"name": "myField",
"type": ["null","string"],
"default": null
}
The error I get :
Incorrect token in the stream. Expected: Object start, found String
Do you have any idea on how to solve this ?
Not sure if this is exactly what you would like, but I would probably try something like this:
{
"type": "record",
"name": "SomeName",
"fields": [
{
"name": "myField",
"type": ["string", "null"]
}
]
}
as defined in Apache Avro specification and described in PubSub Schema creation
#Snowfire, you have mentioned in the comment that you choose to continue with schema-less topic.
You can follow this document to create Pub/Sub Schema.
To create a schema of Avro type you can try this:
{
"type": "record",
"name": "Avro",
"fields": [
{
"name": "testname",
"type": ["null", "string"],
"default": null
},
{
"name": "testId",
"type": ["null", "int"],
"default": null
}
]
}
I tested below message against the schema and it works.
{
"testname": {
"string": "Jack"
},
"testId": {
"int": 101
}
}

PROPERTY_REMOVED_FROM_CLOSED_CONTENT_MODEL for optional field

Use draft-7 json schema CLOSED_CONTENT_MODEL and BACKWARD compatibility and confluent schema registry 7.2.1-post.
deps:
implementation 'com.github.victools:jsonschema-generator:4.26.0'
implementation 'io.confluent:kafka-schema-registry:7.2.1'
schema example
{
"type" : "object",
"properties" : {
"name" : {
"type" : "string",
"description" : "String"
},
"timeDescription" : {
"type" : "string",
"description" : "String"
},
},
"required" : [ "name"],
"additionalProperties" : false
}
I am trying to find incompatibilities between schema via
{{base_url}}/compatibility/subjects/subject/versions/latest?verbose=true
(timeDescription is absent but not required)
{
"type" : "object",
"properties" : {
"name" : {
"type" : "string",
"description" : "String"
},
},
"required" : [ "name"],
"additionalProperties" : false
}
and see
```json
{
"is_compatible": false,
"messages": [
"Found incompatible change: Difference{jsonPath='#/properties/timeDescription', type=PROPERTY_REMOVED_FROM_CLOSED_CONTENT_MODEL}"
]
}
How to disable incompatibilities for case when optional field exists in one schema and absent in another?
Thanks for your answers.
It is not possible to enable or disable compatibilities. You might need select different Schema type.

cygnus mongo sink and meta data storing

I tried to stored this meta data entity but seems like cygnus is only storing entity data no meta data was stored in the data base.
Here is how I update my entity using NGSI v1 updateContext
{
"contextElements": [
{
"type": "dummyMeta",
"isPattern": "false",
"id": "dummyMeta",
"attributes": [
{
"name": "dummy",
"type": "float",
"value": "26.5",
"metadatas": [
{
"name": "accuracy",
"type": "float",
"value": "1"
}
]
}
]
}
],
"updateAction": "APPEND"
}
Here is the payload subscription:
{
"entities": [
{
"id": "dummyMeta",
"type": "dummyMeta",
"isPattern": "false"
}
],
"attributes": [
"dummy"
]
,
"reference": "http://cygnusserver.ddns.net:5050/notify",
"duration":"P1M",
"notifyConditions": [
{
"type": "ONCHANGE",
"condValues": [
"dummy"
]
}
],
"throttling": "PT5S"
}
here is how it is stored in the data base
> db['kura_/egmmqtt_dummyMeta_dummyMeta'].find().sort({$natural:-1})
{ "_id" : ObjectId("57c929d8902531258a3c6ed0"), "recvTime" : ISODate("2016-09-02T07:27:18.331Z"), "attrName" : "dummy", "attrType" : "float", "attrValue" : "26.5" }
{ "_id" : ObjectId("57c92990902531258a3c6ecc"), "recvTime" : ISODate("2016-09-02T07:26:04.148Z"), "attrName" : "dummy", "attrType" : "float", "attrValue" : "26.5" }
What am I missing to be able to store the whole information (data and metadata) about the attribute?
Thanks in advance for your help!
MongoDB sink does not save the metadata by design. It is a requirement by our internal products currently using Cygnus.
Being said that, I think it should not be very difficult to modify the code by your side in order to save the metadata.
Alternatively, I can create an issue about optionally save metadata when configured through a configuration parameter. Nevertheless, I cannot commit with an implementation date.

Validate Json Schema Draft V4

I have a Json schema as below , when i validate it on a online validatior like http://jsonschemalint.com/draft4/# it says its valid.I changed "type" to "object" , "array" , "string" and kept the rest of the structure same .still it says valid schema. My understanding was when
type=object there should be "properties"
when type=array there should be"items"
{
"title": "Example Schema",
"type": "string",
"properties": {
"firstName": {
"type": "string"
},
"lastName": {
"type": "string"
}
},
"required": [
"firstName",
"lastName"
]
}
According to the docs:
Some validation keywords only apply to one or more primitive types.
When the primitive type of the instance cannot be validated by a given
keyword, validation for this keyword and instance SHOULD succeed.
So the behavior you mention is correct.

Json Schema example for oneOf objects

I am trying to figure out how oneOf works by building a schema which validates two different object types. For example a person (firstname, lastname, sport) and vehicles (type, cost).
Here are some sample objects:
{"firstName":"John", "lastName":"Doe", "sport": "football"}
{"vehicle":"car", "price":20000}
The question is what have I done wrongly and how can I fix it. Here is the schema:
{
"description": "schema validating people and vehicles",
"$schema": "http://json-schema.org/draft-04/schema#",
"type": "object",
"required": [ "oneOf" ],
"properties": { "oneOf": [
{
"firstName": {"type": "string"},
"lastName": {"type": "string"},
"sport": {"type": "string"}
},
{
"vehicle": {"type": "string"},
"price":{"type": "integer"}
}
]
}
}
When I try to validate it in this parser:
https://json-schema-validator.herokuapp.com/
I get the following error:
[ {
"level" : "fatal",
"message" : "invalid JSON Schema, cannot continue\nSyntax errors:\n[ {\n \"level\" : \"error\",\n \"schema\" : {\n \"loadingURI\" : \"#\",\n \"pointer\" : \"/properties/oneOf\"\n },\n \"domain\" : \"syntax\",\n \"message\" : \"JSON value is of type array, not a JSON Schema (expected an object)\",\n \"found\" : \"array\"\n} ]",
"info" : "other messages follow (if any)"
}, {
"level" : "error",
"schema" : {
"loadingURI" : "#",
"pointer" : "/properties/oneOf"
},
"domain" : "syntax",
"message" : "JSON value is of type array, not a JSON Schema (expected an object)",
"found" : "array"
} ]
Try this:
{
"description" : "schema validating people and vehicles",
"type" : "object",
"oneOf" : [
{
"type" : "object",
"properties" : {
"firstName" : {
"type" : "string"
},
"lastName" : {
"type" : "string"
},
"sport" : {
"type" : "string"
}
}
},
{
"type" : "object",
"properties" : {
"vehicle" : {
"type" : "string"
},
"price" : {
"type" : "integer"
}
},
"additionalProperties":false
}
]
}
oneOf need to be used inside a schema to work.
Inside properties, it's like another property called "oneOf" without the effect you want.