How to generate all test cases from a JSON schema - json

I am developing a system where we create documents that are created from json-files. The json files are described by a json schema of the following kind, where a key either could have a static default value or have one of multiple enumerated values:
{
"name" : { "default" : "John Doe" },
"bilingual" : { "type" : "boolean" },
"kind_of_document" : {
"type" : "integer",
"enum" : [0, 1, 2]
}
}
What I want to do is to create test files for all possible combinations of values from the schema. In the above case there would be six different json files:
{ "name" : "John Doe", "bilingual" : true, "kind_of_document" : 0 }
{ "name" : "John Doe", "bilingual" : true, "kind_of_document" : 1 }
{ "name" : "John Doe", "bilingual" : true, "kind_of_document" : 2 }
{ "name" : "John Doe", "bilingual" : false, "kind_of_document" : 0 }
{ "name" : "John Doe", "bilingual" : false, "kind_of_document" : 1 }
{ "name" : "John Doe", "bilingual" : false, "kind_of_document" : 2 }
This is a simplified version; in a normal case there may be 20-30 keys, of which most have default values, but there may still 5-10 keys that can have multiple values. They number of keys may vary from one document to another.
The problem:
How do I iterate over the list of keys to generate all possible combinations of values? I suppose that some form of recursion is the way to go, but I can't figure out what the algorithm should be. It seems to me that this should be a fairly common problem, so somebody has most likely solved it before.
I have tried googling for it, but I don't know how to formulate the query to avoid just general questions about generating combinations of list elements, such as
How to get all possible combinations of a list’s elements?
I have also looked at itertools, but I am not sure how it could me solve the problem:
https://docs.python.org/3/library/itertools.html#itertools.permutations
I am writing this in python, but of course the general algorithm for solving this would be language independent.

Related

How can i match fields with wildcards using jq?

I have a JSON object of the following form:
{
"Task11c-0-20181209-12:59:30-65611" : {
"attributes" : {
"configname" : "Task11c",
"datetime" : "20181209-12:59:30",
"experiment" : "Task11c",
"inifile" : "lab1.ini",
"iterationvars" : "",
"iterationvarsf" : "",
"measurement" : "",
"network" : "Manhattan1_1C",
"processid" : "65611",
"repetition" : "0",
"replication" : "#0",
"resultdir" : "results",
"runnumber" : "0",
"seedset" : "0"
},
......
},
......
"Task11b-12-20181209-13:03:17-65612" : {
....
....
},
.......
}
I reported only the first part, but in general I have many other sub-objects which match a string like Task11c-0-20181209-12:59:30-65611. They all have in common the initial word Task. I want to extract the processid from each sub-object. I'm trying to use a wildcard like in bash, but it seems not to be possible.
I also read about the match() function, but it works with strings and not json objects.
Thanks for the support.
Filter keys that start with Test and get only the attribute of your choice using the select() expression
jq 'to_entries[] | select(.key|startswith("Task")).value.attributes.processid' json

Back-converting from JSON draft 4 to JSON draft 3

I have a bit of a weird situation. I am running a JSON schema validator that doesn't support draft 4, and due to corporate wonkiness I'm stuck with it instead of replacing it. Our developers are giving me schemas in draft 4 format, so I have to go through by hand and back-convert it, particularly the required fields.
This has all worked fine up until I hit something like this (consider this pseudo-code; I'm still getting the hang of JSON):
"items": {
"type": "array",
"required" : true,
"items": [
{...},
"required": ["0", "1"] // This bit right here
],
}
I'm told it basically says, "The first two items in the array are required." But I can't find a way to express that in JSON draft 3. Is that even supported, and if so, how would you express it?
required is a keyword that only has meaning for object instances, not for arrays.
The way to indicate that an array must have at least 2 items is through minItems keyword in both Draft3 and Draft4.
If you need to express any other schema just for the first and second item in an array, you do it by having two schemas in the items array. For instance, the following schema requires that properties "0" and "1" are included in the first and second items in the array.
For Draft 3:
"items" : [{
"properties" : {
"0" : {"required" : true},
"1" : {"required" : true}
}
}, {
"properties" : {
"0" : {"required" : true},
"1" : {"required" : true}
}
}
]
And Draft 4:
"items" : [{
"required" : ["0", "1"]
}, {
"required" : ["0", "1"]
}
]

Migrating from MySQL to Couchbase Server (NoSQL)

I am planning to move some of my RDBMS tables to a NoSQL environment. I have different modules and associated tables as shown below.
1. General
News table
Trainings table
Knowledge table
2. Application_x
Clicks table
Crashes table
Note : This is a sample scenario
In this case, this is how i defined Nosql (Couchbase server) structure. Please correct me if this is wrong.
The purpose of migration is mainly intended for searching.
{
"General":[
"news" : [{
"id" : "123",
"title" : "test title"
},
{
"id" : "345",
"title" : "test title 2"
}],
"trainings" :[{
"id" : "1",
"name" : "training 1"
},
{
"id" : "2",
"name" : "training 2"
}],
"knowledge" :[{
"id" : "1",
"categ" : "programming"
},
{
"id" : "2",
"categ" : "management"
}]
],
"Application_x": [
"clicks" : [{
"userid" : "1",
"area" : "1850",
},
{
"userid" : "2",
"area" : "258",
}],
"crashes" :[{
"userid" : "1",
"severity" : "1",
},
{
"userid" : "2",
"severity" : "8",
}]
]
}
Can someone correct me if my approach is not correct ?
Thanks in advance,
Tismon Varghese.
By reading your question, I am left scratching my head as to what your approach is. In a NoSQL database (such as couchbase) - you don't have the idea of tables and columns. Each object is serialized to JSON and stored in plain text. Yes, this creates duplication, but the drawbacks of duplication are greatly outweighed by the benefits of scalability.
In this example, using Couchbase, you would probably want to create one Couchbase bucket per application. That way, should you need to migrate to a different Couchbase cluster at a later date, a minimal amount of configuration is required. Each row in your tables gets created as a separate object in Couchbase. There is no need to separate the object types within the bucket.
This blog entry contains some detailed instructions on how to migrate from mySQL to Couchbase.
On a side note, I might recommend using Couchbase combined with Elasticsearch.

Is there any way to preserve the order while generating patch for json files ?

I am new to Json stuff i.e. JSON PATCH.
I have scenario where I need to figure out between two version of Json files of same object, for that I am using json-patch-master.
But unfortunately the patch generated interpreting it differently i.e. the order differently hence getting unexpected/invalid results.
Could anyone help me how to preserve the order while generating Json Patch ?
**Here is the actual example.
Original Json file :**
[ {
"name" : "name1",
"roolNo" : "1"
}, {
"name" : "name2",
"roolNo" : "2"
}, {
"name" : "name3",
"roolNo" : "3"
}, {
"name" : "name4",
"roolNo" : "4"
} ]
**Modified/New Json file: i.e. removed 2nd node of original file.**
[ {
"name" : "name1",
"roolNo" : "1"
}, {
"name" : "name3",
"roolNo" : "3"
}, {
"name" : "name4",
"roolNo" : "4"
} ]
**Patch/Diff Generated :**
[ {"op":"remove","path":"/3"},
{"op":"replace","path":"/1/name","value":"name3"},
{"op":"replace","path":"/1/roolNo","value":"3"},
{"op":"replace","path":"/2/name","value":"name4"},
{"op":"replace","path":"/2/roolNo","value":"4"}]
Very time I generate Diff/Patch it is giving different path/diff results.
And moreover the interpretation is different i.e. order is not preserving.
**Is there any way to get expected results i.e. [ {"op":"remove","path":"/1"} ] , in other words generated a patch/diff based some order so will get what is expected. ?
How to handle this kind of scenario ?**
Please help me.
Thank you so much.
~Shyam
We are currently working on this issue in Starcounter-Jack/JSON-Patch.
It seems to work nice with native Array.Observe- http://jsfiddle.net/tomalec/p4s7aw96/.
Try Starcounter-Jack/JSON-Patch issues/65_ArrayObserve branch
we will release it as new version once shim and performance will be checked.
Feel free to add you comments at JSON-Patch issue board

MongoDB AND Comparison Fails

I have a Collection named StudentCollection with two documents given below,
> db.studentCollection.find().pretty()
{
"_id" : ObjectId("52d7c0c744b4dd77efe93df7"),
"regno" : 101,
"name" : "Ajeesh",
"gender" : "Male",
"docs" : [
"voterid",
"passport",
"drivinglic"
]
}
{
"_id" : ObjectId("52d7c6a144b4dd77efe93df8"),
"regno" : 102,
"name" : "Sathish",
"gender" : "Male",
"dob" : ISODate("2013-12-09T21:05:00Z")
}
Why does the below query returns a document when it doesn't fulfil the criteria which I gave in find command. I know it's a bad & stupid query for AND comparison. I tried this with MySQL and it doesn't return anything as expected but why does NOSQL makes problem. I hope it's considering the last field for comparison.
> db.studentCollection.find({regno:101,regno:102}).pretty()
{
"_id" : ObjectId("52d7c6a144b4dd77efe93df8"),
"regno" : 102,
"name" : "Sathish",
"gender" : "Male",
"dob" : ISODate("2013-12-09T21:05:00Z")
}
Can anyone brief why does Mongodb works this way?
MongoDB leverages JSON/BSON and names should be unique (http://www.ietf.org/rfc/rfc4627.txt # 2.2.) Found this in another post How to generate a JSON object dynamically with duplicate keys? . I am guessing the value for 'regno' gets overridden to '102' in your case.
If what you want is an OR query, try the following:
db.studentCollection.find ( { $or : [ { "regno" : "101" }, {"regno":"102"} ] } );
Or even better, use $in:
db.studentCollection.find ( { "regno" : { $in: ["101", "102"] } } );
Hope this helps!
Edit : Typo!
MongoDB converts your query to a Javascript document. Since you have not mentioned anything for $and condition in your document, your query clause is getting overwritten by the last value which is "regno":"102". Hence you get last document as result.
If you want to use an $and, you may use any of the following:
db.studentCollection.find({$and:[{regno:"102"}, {regno:"101"}]});
db.studentCollection.find({regno:{$gte:"101", $lte:"102"}});