Creating an Avro schema for a simple json - json

I'm trying to build an avro schema for the following json:
{
"id":1234,
"my_name_field": "my_name",
"extra_data": {
"my_long_value": 1234567890,
"my_message_string": "Hello World!",
"my_int_value": 777,
"some_new_field": 1
}
}
The value for 'id' and 'my_name_field' are known but the fields in 'extra_data' dynamically change and are unknown.
The avro schema I had in mind is:
{
"name":"my_record",
"type":"record",
"fields":[
{"name":"id", "type":"int", "default":0},
{"name":"my_name_field", "type":"string", "default":"NoName"},
{ "name":"extra_data", "type":{"type":"map", "values":["null","int","long","string"]} }
]
}
My first idea was to make 'extra_data' a record with a map, but this does not work:
{ "name":"extra_data", "type":{"type":"map", "values":["null","int","long","string"]} }
I get:
AvroTypeException: Expected start-union. Got VALUE_NUMBER_INT
apache gives some nice examples in https://cwiki.apache.org/confluence/display/Hive/AvroSerDe but none seem to do the job.
This is the unit test I run to check:
public class AvroTest {
#Test
public void readRecord() throws IOException {
String event="{\"id\":1234,\"my_name_field\":\"my_name\",\"extra_data\":{\"my_long_value\":1234567890,\"my_message_string\":\"Hello World!\",\"my_int_value\":777,\"some_new_field\":1}}";
SchemaRegistry<Schema> registry = new com.linkedin.camus.schema.MySchemaRegistry();
DecoderFactory decoderFactory = DecoderFactory.get();
ObjectMapper mapper = new ObjectMapper();
GenericDatumReader<GenericData.Record> reader = new GenericDatumReader<GenericData.Record>();
Schema schema = registry.getLatestSchemaByTopic("record_topic").getSchema();
reader.setSchema(schema);
HashMap hashMap = mapper.readValue(event, HashMap.class);
long now = Long.valueOf(hashMap.get("now").toString())*1000;
GenericData.Record read = reader.read(null, decoderFactory.jsonDecoder(schema, event));
}
Would appreciate help with this,
Thanks.

If the list of extra data fields is really unknown using multiple optional value fields may help, like this:
{
"name":"my_record",
"type":"record",
"fields":[
{"name":"id", "type":"int", "default":0},
{"name":"my_name_field", "type":"string", "default":"NoName"},
{"name":"extra_data", "type": "array", "items": {
{"name": "extra_data_entry", "type":"record", "fields": [
{"name":"extra_data_field_name", "type": "string"},
{"name":"extra_data_field_type", "type": "string"},
{"name":"extra_data_field_value_string", "type": ["null", "string"]},
{"name":"extra_data_field_value_int", "type": ["null", "int"]},
{"name":"extra_data_field_value_long", "type": ["null", "long"]}
]}
}}
]
}
Then you can select the extra_data_field_value_* value based on the extra_data_field_type for that field.

Related

RealmList<String> as a JSON Schema - Mongo DB Realm

I have a simple model class from which I need to generate the schema on Mongo DB Atlas. But I'm having troubles when it comes to defining RealmList<String> inside a JSON schema. If I insert "array" as a bsonType, I get an error. What should I write instead?
Model class:
class Note : RealmObject {
#PrimaryKey
var _id: ObjectId = ObjectId.create()
var title: String = ""
var description: String = ""
var images: RealmList<String> = realmListOf()
var date: RealmInstant = RealmInstant.from(System.currentTimeMillis(),0)
}
Current Schema:
{
"bsonType": "object",
"properties": {
"_id": {
"bsonType": "objectId"
},
"title": {
"bsonType": "string"
},
"description": {
"bsonType": "string"
},
"images": {
"bsonType": "array"
},
"date": {
"bsonType": "date"
}
},
"required": [
"_id",
"title",
"description",
"images",
"date"
],
"title": "Note"
}
I am not sure which mode you're using but if you're in development mode, when you add an object in the SDK, the server will automatically generate a matching object, as long as the changes are additive, like adding a new object or property
In the queston, the 'images' bson definition looks incomplete
"images": {
"bsonType": "array"
},
While it is an array, it's an array of strings so I believe it should look more like this
"images": {
"bsonType": "array",
"items": {
"bsonType": "string"
}
}
Where the type of items is defined as a string

parsing json object with number as its key fields?

I'm trying to parse json into kotlin objects but the problem is that its key fields are numbers any idea how can parse them , I've tried serialized name but still facing problem.
The json response looks like this :
{
"Id": [{
"1": {
"name": "name1",
"class": "11a"
}
},
{
"2": {
"name": "name2",
"class": "11b"
}
}
]
}
I'm using gson and the main thing i'm trying to do is to store this number fields as some other string objects.
You can parse them into a list of maps, then "map" those to your data classes instead:
val input = """{
"Id": [{
"1": {
"name": "name1",
"class": "11a"
}
},
{
"2": {
"name": "name2",
"class": "11b"
}
}
]
}"""
val gson = Gson()
val parsed: Map<String, List<Map<String, Any>>> =
gson.fromJson(input, (object : TypeToken<Map<String, List<Map<String, Any>>>>(){}).type)
println(parsed["Id"]?.get(0)?.get("1")) // {name=name1, class=11a}
It will have some nasty generic signature, though.
If you're working with Kotlin, take a look at Klaxon, it will improve your experience.

How do I get a configurationSection as string?

I am working with .NETCore 2.0 I want to load a complete configuration-section as string.
To be specific, I want to do Json-Schema validation and my schema is stored in appsettings.json:
{
...
"schemas": {
"project": {
"title": "Project",
"type": "object",
"required": [ "param1" ],
"additionalProperties": false,
"properties": {
"param1": {
"type": "string"
},
"param2": {
...
}
}
}
},
...
}
Now I want to load configuration-section "schemas.project" as string and let Json.NET Schema do the schema parsing.
Something like this:
var schemaString = this.configuration.GetSection("schemas.project").Get<string>();
var schema = JSchema.Parse(schemaString);
...
Is there a way to load a complete configuration-section as string? Otherwise I'll read in the schema-file as string..

Full Json match with RestAssured

I'm using REST-Assured to test some API. My API clearly respond with a JSON and according to the doc if this is the response:
{
"id": "390",
"data": {
"leagueId": 35,
"homeTeam": "Norway",
"visitingTeam": "England",
},
"odds": [{
"price": "1.30",
"name": "1"
},
{
"price": "5.25",
"name": "X"
}]
}
I could test like this:
#Test
public void givenUrl_whenSuccessOnGetsResponseAndJsonHasRequiredKV_thenCorrect() {
get("/events?id=390")
.then()
.statusCode(200)
.assertThat()
.body("data.leagueId", equalTo(35));
}
Surely this is readable but I would a full comparison of the JSON (i.e.: this is the JSON response; this is a canned JSON - a resource file would be perfect - are those JSON equals?). Does REST-Assured offer something like that or I need to make it manually.
Use RestAssured's JsonPath to parse the json file into a Map and then compare it with Hamcrest Matchers. This way the order etc didn't matter.
import static org.hamcrest.Matchers.equalTo;
import io.restassured.path.json.JsonPath;
...
JsonPath expectedJson = new JsonPath(new File("/path/to/expected.json"));
given()
...
.then()
.body("", equalTo(expectedJson.getMap("")));
Karate is exactly what you are looking for - you can perform a full equality match of a JSON payload in one step.
And for the cases where you have dynamic values (generated keys, timestamps) Karate provides a very elegant way for you to ignore (or just validate the format of) these keys.
One the primary motivations for creating Karate was to come up with a better alternative to REST-assured. You can refer to this document which can help you evaluate Karate and make a case for it in your organization: Karate vs REST-assured.
REST Assured does not support JSON comparison, only schema and parts of the body as you have in your question. What you can do is using Hamcrest's JSON comparitorSameJSONAs in Hamcrest JSON SameJSONAs
If somebody is looking for method without parsing json-file.
You can check the body size at the beginning using Matchers.aMapWithSize(size), and then check the contents as usual.
Example:
#Test
public void getAccount_forbidden_whenUserIsAnonymous() {
RestAssured
.get("/account")
.then()
.statusCode(HttpServletResponse.SC_FORBIDDEN)
.body("", Matchers.aMapWithSize(5),
"timestamp", Matchers.notNullValue(),
"status", Matchers.equalTo(HttpServletResponse.SC_FORBIDDEN),
"error", Matchers.equalTo("Forbidden"),
"message", Matchers.equalTo("Access Denied"),
"path", Matchers.equalTo("/account"));
}
You can use Validate with JSON SCHEMA in RestAssured.
Try this code:
// Base Test [BaseTest.java]
public class BaseTest {
protected RequestSpecification requestSpecificationToMerge = new RequestSpecBuilder()
.setBaseUri("BASE URL")
.setContentType(ContentType.JSON)
.build();
#BeforeMethod
public void setFilter() {
RestAssured.filters(new AllureRestAssured());
}
}
// Test [Home.java]
public class Home extends BaseTest {
#Test(priority = 0)
public void getHome() {
given()
.spec(requestSpecificationToMerge)
.basePath("/your endpoint")
.when()
.get()
.then()
.log().body()
.body(matchesJsonSchemaInClasspath("home.json"));
}
// JSON SCHEMA [home.json]
{
"type": "object",
"required": [
"data",
"meta",
"status"
],
"properties": {
"data": {
"type": ["array", "null"],
"items": {
"type": "object",
"required": [
"id",
"title",
"sorting"
],
"properties": {
"id": {
"type": "integer"
},
"title": {
"type": "string"
},
"sorting": {
"type": "integer"
}
}
}
},
"meta": {
"type": ["object", "null"],
"required": [
"pagination"
],
"items": {
"type": "object",
"required": [
"current_page",
"per_page",
"total",
"total_page"
],
"properties": {
"current_page": {
"type": "integer"
},
"per_page": {
"type": "integer"
},
"total": {
"type": "integer"
},
"total_page": {
"type": "integer"
}
}
}
},
"status": {
"type": "object",
"required": [
"code",
"message_client",
"message_server"
],
"properties": {
"code": {
"type": "integer",
"enum": [
200,
404
]
},
"message_client": {
"type": "string"
},
"message_server": {
"type": "string"
}
}
}
}
}
Easy way:
String s = "{\"ip\": \"your_ip\"}";
given().log().all().contentType(ContentType.JSON).get("http://ip.jsontest.com/").then().log().all().assertThat().body(containsString(s))
Not easy way: you can create custom matcher or use jsonpath, it has options to comapre jsons.
Apparently, rest-assured only provides capabilities to validate the schema as described here.
However, it's quite simple to make an exact comparison using jackson-databind and junit.
We should write a function that compares a body returned by rest-assured with a file in the resources directory
import org.junit.Assert;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.json.JsonMapper;
import com.fasterxml.jackson.databind.node.ObjectNode;
void assertJsonEquals(String expectedJson, ResponseBodyExtractionOptions actualBody) throws IOException {
Assert.assertNotNull("The request returned no body", expectedJson);
final ObjectMapper mapper = new ObjectMapper();
Assert.assertEquals(
mapper.readTree(Objects.requireNonNull(getClass().getClassLoader().getResource(expectedJsonPath)).openStream().readAllBytes()),
mapper.readTree(body.asByteArray())
);
}
Then, use it as shown below
final ResponseBodyExtractionOptions actualBody = given()
.accept("application/json")
.contentType(MediaType.APPLICATION_JSON)
.when()
.get("...")
.then()
.extract().body();
assertJsonEquals("expected-payload.json", actualBody);
You can make use of JSONAssert Library to match the entire JSON Response. I recently wrote a blog on how to achieve it.
Below is the basic usage on how to use the library:
JSONAssert.assertEquals(expectedResponse, actualResponse, JSONCompareMode.LENIENT);

How to get a list of readOnly fields from a JsonSchema in Java

I am trying to validate a JSON input against a schema using the json-schema-validator library: https://github.com/fge/json-schema-validator. The validation works fine. However, one of my requirement is to make sure that the input doesn't contain any readonly fields. I have marked the fields as readonly in the json schema but there's no good way to know which fields are readonly in the java code.
Here's an example schema:
{
"type": "object",
"$schema": "http://json-schema.org/draft-03/schema",
"id": "http://jsonschema.net",
"required":false,
"properties": {
"partId": {
"type": "string",
"final":true,
"minLength": 1,
"maxLength": 36
},
"name": {
"type": "string",
"required": true
},
"forSaleDate": {
"type":"string",
"format":"date",
"final":true,
"readOnly":true
}
}
}
Here's the code I used for validation:
JsonNode jsonSchemaNode = null;
try {
jsonSchemaNode = JsonLoader.fromString(schemaString);
} catch (JsonProcessingException e) {
log.error("Invalid JSON schema");
throw e;
}
JsonSchema jsonSchema = factory.fromSchema(jsonSchemaNode);
JsonNode json = null;
try {
json = mapper.readTree(example);
} catch (JsonProcessingException e) {
log.error("Invalid JSON string");
}
if (json != null) {
ValidationReport validationReport = jsonSchema.validate(json);
}
The last field in my schema "forSaleDate" is a readonly field. In my validation code, the JSON input will get parsed into a JsonNode and I can call the "has(fieldName)" method on that JsonNode to see if it contains a particular field. The idea is to do this for all the readonly fields. However, the tricky part is to figure out which fields are readonly. The schema in the above code is represented as JsonSchema and it does not have a method that would return all the readonly fields. Did anyone come across this sort of validation? If yes, what was your approach?