json schema validation does not work for dependencies? - json

As far as I understand, using:
dependencies:
cake:
- eggs
- flour
in a JSON schema validation (I actually parse the schema in YAML, but it should not really matter) should forbid the presence of a cake without an entry for eggs and flour. So this should be rejected:
receip:
cake: crepes
while this should be accepted:
receip:
eggs: 3 eggs, given by farmer
flour: 500g
cake: crepes
Unfortunately, both cases are accepted in my case. Any idea what I made wrong? I also tried to add another level required as proposed by jsonSchema attribute conditionally required but the problem is the same.
MWE
In case it helps, here are all the files that I'm using.
debug_schema.yml:
$schema: https://json-schema.org/draft/2020-12/schema
type: object
properties:
receip:
type: object
properties:
eggs:
type: string
flour:
type: string
cake:
type: string
dependencies:
cake:
- eggs
- flour
b.yml:
receip:
## I'd like to forbid a cake without eggs and flour
## Should not validate until I uncomment:
# eggs: 3 eggs, given by farmer
# flour: 500g
cake: crepes
validate_yaml.py
#!/usr/bin/env python3
import yaml
from jsonschema import validate, ValidationError
import argparse
# Instantiate the parser
parser = argparse.ArgumentParser(description='This tool not only checks if the YAML file is correct in term'
+ ' of syntex, but it also checks if the structure of the YAML file follows'
+ ' the expectations of this project (e.g. that it contains a list of articles,'
+ ' with each articles having a title etc…).')
# Optional argument
parser.add_argument('--schema', type=str, default="schema.yml",
help='Path to the schema')
parser.add_argument('--yaml', type=str, default="science_zoo.yml",
help='Path to the YAML file to verify')
args = parser.parse_args()
class YAMLNotUniqueKey(Exception):
pass
# By default, PyYAML does not raise an error when two keys are identical, which is an issue here
# see https://github.com/yaml/pyyaml/issues/165#issuecomment-430074049
class UniqueKeyLoader(yaml.SafeLoader):
def construct_mapping(self, node, deep=False):
mapping = set()
for key_node, value_node in node.value:
key = self.construct_object(key_node, deep=deep)
if key in mapping:
raise YAMLNotUniqueKey(f"Duplicate key {key!r} found.")
mapping.add(key)
return super().construct_mapping(node, deep)
with open(args.schema, "r") as schema_stream:
try:
schema = yaml.load(schema_stream, UniqueKeyLoader)
with open(args.yaml, "r") as yaml_stream:
try:
validate(yaml.load(yaml_stream, UniqueKeyLoader), schema)
print("The YAML file is correct and follows the schema, congrats! :D")
except ValidationError as exc:
print(exc.message)
print("The problem is located in the JSON at the path {}.".format(exc.json_path))
except YAMLNotUniqueKey as exc:
print("ERROR: {}".format(exc))
print("Rename in the YAML one the two instances of this key to ensure all keys are unique.")
except yaml.YAMLError as exc:
print("Errors when trying to load the YAML file:")
print(exc)
except YAMLNotUniqueKey as exc:
print("ERROR: {}".format(exc))
print("Rename in the schema one the two instances of this key to ensure all keys are unique.")
except yaml.YAMLError as exc:
print("Errors when trying to load the schema file:")
print(exc)
Command:
./validate_yaml.py --yaml b.yml --schema debug_schema.yml

Oh, so it seems that this keyword has been deprecated, and split into dependentRequired (see e.g. this example) and dependentSchemas (see e.g. this example). Just using dependentRequired solves the issue:
$schema: https://json-schema.org/draft/2020-12/schema
type: object
properties:
receip:
type: object
properties:
eggs:
type: string
flour:
type: string
cake:
type: string
dependentRequired:
cake:
- eggs
- flour

Related

How to use a link rule reference in an ordered choice expression in a TextX grammar?

I am new to TextX. I am trying to create a grammar for defining data types that have fields that could be of a simple type or of the type of another data type. The grammar description is:
Library:
data_types *= DataType
;
DataType: name=ID "{"
fields*=Field
"}" ;
Field: type=([DataType] | ID) name=ID;
//Type: [DataType] | ID;
An example of a model following this grammar would be
vec {
int64 a
int64 b
int64 c
}
matrix {
vec a
vec b
}
I want to link the type of the field to either a data type that is already declared, or to some simple string. However, when compiling the above grammar with textx generate dummy.tx --target dot, I get the error Error: None:9:13: error: Expected ''((\\')|[^'])*'' or '"((\\")|[^"])*"' or re_match or rule_ref or '[' at position dummy.tx:(9, 13) => 'eld: type=*([DataType'..
Is there any way to accomplish what I want? I have tried putting the type declaration in a separate block, as seen in the comment, but that did not help. Any suggestion or hint would be highly appreciated.
A standard approach is to use custom classes and create builtins for all types that are not created by users themselves. It is best to show how it is done in code using your example. Note the use of registration so that the language with registered builtins can be available to textx CLI command. Also, see the entity example as the same techniques is used there.
from textx import metamodel_from_str
from textx.registration import (language, register_language,
metamodel_for_language)
# We use registration support to register language
# This way it will be available to textx CLI command
#language('library', '.lib')
def library_lang():
"Library language."
grammar = r'''
Library:
data_types *= DataType
;
DataType: name=ID "{"
fields*=Field
"}" ;
Field: type=[Type] name=ID;
Type: DataType | BuiltInType;
BuiltInType: name=ID;
'''
# Here we use our class for builtin types so we can
# make our own instances for builtin types.
# See textX Entity example for more.
class BuiltInType:
def __init__(self, parent, name):
self.parent = parent
self.name = name
# Create all builtin instances.
builtins = {
'int64': BuiltInType(None, 'int64'),
}
return metamodel_from_str(grammar,
# Register custom classes and builtins.
classes=[BuiltInType],
builtins=builtins)
# This should really be done through setup.{cfg,py}
# Here it is done through registration API for an example to
# be self-contained.
register_language(library_lang)
model_str = r'''
vec {
int64 a
int64 b
int64 c
}
matrix {
vec a
vec b
}
'''
# Now we can get registered language metamodel by name and
# parse our model.
model = metamodel_for_language('library').model_from_str(model_str)
# ... do something with the model
assert len(model.data_types) == 2
assert model.data_types[0].name == 'vec'
assert model.data_types[0].fields[0].name == 'a'
assert model.data_types[0].fields[0].type.name == 'int64'

How to efficiently parse JSON data with multiple keys in Python 2.7?

I'm writing a script that will check the CVS COVID vaccine availability for cities in my state of VA. I have been successful getting the data I'm looking for, but my code is hard coded in some areas. I'm specifically asking for help improving my code in the areas number 1 & 2 below:
The JSON file can be found here:
https://www.cvs.com//immunizations/covid-19-vaccine.vaccine-status.VA.json?vaccineinfo
I'm trying to access the data in the responsePayloadData key. The only way I could figure out how to do this is to make it the only key. For that reason, I deleted the other key responseMetaData:
#remove the key that we don't need
del obj['responseMetaData']
I'm also not sure how to dynamically loop through the VA items without hard coding the number of cities I know are there in the data:
for x, y in obj.items():
for a in range(34):
Here's the full code:
import requests
import json
import time
from datetime import datetime
import urllib2
try:
import indigo
except:
pass
strAvail = "False"
strAvailCity = "None"
try:
# download raw json object from CVS Virginia Website
url = "https://www.cvs.com//immunizations/covid-19-vaccine.vaccine-status.VA.json?vaccineinfo"
data = urllib2.urlopen(url).read().decode()
except urllib2.HTTPError, err:
return {"error": err.reason, "error_code": err.code}
# parse json object
obj = json.loads(data)
# remove the key that we don't need
del obj['responseMetaData']
# loop through the JSON dictionary and check availability
# status options: {"Fully Booked", "Available"}
for x, y in obj.items():
for a in range(34):
# print('City: ' + y['data']['VA'][a]['city'])
# print('Total Available: ' + y['data']['VA'][a]['totalAvailable'])
# print('Percent Available: ' + y['data']['VA'][a]['pctAvailable'])
# print('Status: ' + y['data']['VA'][a]['status'])
# print("------------------------------")
# If there is availability anywhere in the state, take some action.
if y['data']['VA'][a]['status'] == "Available":
strAvail = True
strAvailCity = y['data']['VA'][a]['city']
# Log timestamp for this check to the JSON
now = datetime.now()
strDateTime = now.strftime("%m/%d/%Y %I:%M %p")
EDIT: Since the JSON is not available outside the US. I've pasted it below:
{"responsePayloadData":{"currentTime":"2021-02-11T14:55:00.470","data":{"VA":[{"totalAvailable":"1","city":"ABINGDON","state":"VA","pctAvailable":"0.19%","status":"Fully Booked"},{"totalAvailable":"0","city":"ALEXANDRIA","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"ARLINGTON","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"BEDFORD","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"BLACKSBURG","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"CHARLOTTESVILLE","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"CHATHAM","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"CHESAPEAKE","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"1","city":"DANVILLE","state":"VA","pctAvailable":"0.19%","status":"Fully Booked"},{"totalAvailable":"2","city":"DUBLIN","state":"VA","pctAvailable":"0.39%","status":"Fully Booked"},{"totalAvailable":"0","city":"FAIRFAX","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"FREDERICKSBURG","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"GAINESVILLE","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"HAMPTON","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"HARRISONBURG","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"LEESBURG","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"LYNCHBURG","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"MARTINSVILLE","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"MECHANICSVILLE","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"MIDLOTHIAN","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},
{"totalAvailable":"0","city":"NEWPORT NEWS","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"NORFOLK","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"PETERSBURG","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"PORTSMOUTH","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"RICHMOND","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"ROANOKE","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},
{"totalAvailable":"0","city":"ROCKY MOUNT","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"STAFFORD","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"SUFFOLK","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},
{"totalAvailable":"0","city":"VIRGINIA BEACH","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"WARRENTON","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"WILLIAMSBURG","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"WINCHESTER","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"WOODSTOCK","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"}]}},"responseMetaData":{"statusDesc":"Success","conversationId":"Id-beb5f68730b34e6aa3bbc1fd927ea12b","refId":"Id-b4a7256078789eb59b8912b4","operation":"getInventorybyCity","statusCode":"0000"}}
Regarding problem 1, you can just access the data by key. You don't need to delete the other key:
payload = obj['responsePayloadData']
For the second problem, you can just iterate over the items in the list associated with obj['data']['VA']:
for city in payload['data']['VA']:
print(city)
{'city': 'ABINGDON',
'pctAvailable': '0.19%',
'state': 'VA',
'status': 'Fully Booked',
'totalAvailable': '1'}
{'city': 'ALEXANDRIA',
'pctAvailable': '0.00%',
'state': 'VA',
'status': 'Fully Booked',
'totalAvailable': '0'}
...

Elixir - JasonHelpers - How can I send a keyword list to json_map?

I have a data structure that I want to convert to json and preserve the key order.
For example:
%{ x: 1, a: 5} should be converted to "{\"x\": 1, \"a\": 5}"
Poison does it without any problem. But when I upgrade to Jason, it changes to "{\"a\": 5, \"x\": 1}".
So I use JasonHelpers json_map to preserve the order like this:
Jason.Helpers.json_map([x: 1, a: 5])
It creates a fragment with correct order.
However, when I use a variable to do this:
list = [x: 1, a: 5]
Jason.Helpers.json_map(list)
I have an error:
** (Protocol.UndefinedError) protocol Enumerable not implemented for {:list, [line: 15], nil} of type Tuple.
....
QUESTION: How can I pass a pre-calculated list into Jason.Helpers.json_map ?
The calculation is complicated, so I don't want to repeat the code just to use json_map, but use the function that returns a list.
json_map/1 is a macro, from its docs:
Encodes a JSON map from a compile-time keyword.
It is designed for compiling JSON at compile-time, which is why it doesn't work with your runtime variable.
Support for encoding keyword lists was added to the Jason library a year ago, but it looks like it hasn't been pushed to hex yet. I managed to get it work by pulling the latest code from github:
defp deps do
[{:jason, git: "https://github.com/michalmuskala/jason.git"}]
end
Then by creating a struct that implements Jason.Encoder (adapted from this solution by the Jason author):
defmodule OrderedObject do
defstruct [:value]
def new(value), do: %__MODULE__{value: value}
defimpl Jason.Encoder do
def encode(%{value: value}, opts) do
Jason.Encode.keyword(value, opts)
end
end
end
Now we can encode objects with ordered keys:
iex(1)> Jason.encode!(OrderedObject.new([x: 1, a: 5]))
"{\"x\":1,\"a\":5}"
I don't know if this is part of the public API or just an implementation detail, but it appears you have some control of the order when implementing the Jason.Encoder protocol for a struct.
Let's say you've defined an Ordered struct:
defmodule Ordered do
#derive {Jason.Encoder, only: [:a, :x]}
defstruct [:a, :x]
end
If you encode the struct, the "a" key will be before the "x" key:
iex> Jason.encode!(%Ordered{a: 5, x: 1})
"{\"a\":5,\"x\":1}"
Let's reorder the keys we pass in to the :only option:
defmodule Ordered do
#derive {Jason.Encoder, only: [:x, :a]}
defstruct [:a, :x]
end
If we now encode the struct, the "x" key will be before the "a" key:
iex> Jason.encode!(%Ordered{a: 5, x: 1})
"{\"x\":1,\"a\":5}"

Emit Python embedded object as native JSON in YAML document

I'm importing webservice tests from Excel and serialising them as YAML.
But taking advantage of YAML being a superset of JSON I'd like the request part of the test to be valid JSON, i.e. to have delimeters, quotes and commas.
This will allow us to cut and paste requests between the automated test suite and manual test tools (e.g. Postman.)
So here's how I'd like a test to look (simplified):
- properties:
METHOD: GET
TYPE: ADDRESS
Request URL: /addresses
testCaseId: TC2
request:
{
"unitTypeCode": "",
"unitNumber": "15",
"levelTypeCode": "L",
"roadNumber1": "810",
"roadName": "HAY",
"roadTypeCode": "ST",
"localityName": "PERTH",
"postcode": "6000",
"stateTerritoryCode": "WA"
}
In Python, my request object has a dict attribute called fields which is the part of the object to be serialised as JSON. This is what I tried:
import yaml
def request_presenter(dumper, request):
json_string = json.dumps(request.fields, indent=8)
return dumper.represent_str(json_string)
yaml.add_representer(Request, request_presenter)
test = Test(...including embedded request object)
serialised_test = yaml.dump(test)
I'm getting:
- properties:
METHOD: GET
TYPE: ADDRESS
Request URL: /addresses
testCaseId: TC2
request: "{
\"unitTypeCode\": \"\",\n
\"unitNumber\": \"15\",\n
\"levelTypeCode": \"L\",\n
\"roadNumber1\": \"810\",\n
\"roadName\": \"HAY\",\n
\"roadTypeCode\": \"ST\",\n
\"localityName\": \"PERTH\",\n
\"postcode\": \"6000\",\n
\"stateTerritoryCode\": \"WA\"\n
}"
...only worse because it's all on one line and has white space all over the place.
I tried using the | style for literal multi-line strings which helps with the line breaks and escaped quotes (it's more involved but this answer was helpful.) However, escaped or multiline, the result is still a string that will need to be parsed separately.
How can I stop PyYaml analysing the JSON block as a string and make it just accept a block of text as part of the emitted YAML? I'm guessing it's something to do with overriding the emitter but I could use some help. If possible I'd like to avoid post-processing the serialised test to achieve this.
Ok, so this was the solution I came up with. Generate the YAML with a placemarker ahead of time. The placemarker marks the place where the JSON should be inserted, and also defines the root-level indentation of the JSON block.
import os
import itertools
import json
def insert_json_in_yaml(pre_insert_yaml, key, obj_to_serialise):
marker = '%s: null' % key
marker_line = line_of_first_occurrence(pre_insert_yaml, marker)
marker_indent = string_indent(marker_line)
serialised = json.dumps(obj_to_serialise, indent=marker_indent + 4)
key_with_json = '%s: %s' % (key, serialised)
serialised_with_json = pre_insert_yaml.replace(marker, key_with_json)
return serialised_with_json
def line_of_first_occurrence(basestring, substring):
"""
return line number of first occurrence of substring
"""
lineno = lineno_of_first_occurrence(basestring, substring)
return basestring.split(os.linesep)[lineno]
def string_indent(s):
"""
return indentation of a string (no of spaces before a nonspace)
"""
spaces = ''.join(itertools.takewhile(lambda c: c == ' ', s))
return len(spaces)
def lineno_of_first_occurrence(basestring, substring):
"""
return line number of first occurrence of substring
"""
return basestring[:basestring.index(substring)].count(os.linesep)
embedded_object = {
"unitTypeCode": "",
"unitNumber": "15",
"levelTypeCode": "L",
"roadNumber1": "810",
"roadName": "HAY",
"roadTypeCode": "ST",
"localityName": "PERTH",
"postcode": "6000",
"stateTerritoryCode": "WA"
}
yaml_string = """
---
- properties:
METHOD: GET
TYPE: ADDRESS
Request URL: /addresses
testCaseId: TC2
request: null
after_request: another value
"""
>>> print(insert_json_in_yaml(yaml_string, 'request', embedded_object))
- properties:
METHOD: GET
TYPE: ADDRESS
Request URL: /addresses
testCaseId: TC2
request: {
"unitTypeCode": "",
"unitNumber": "15",
"levelTypeCode": "L",
"roadNumber1": "810",
"roadName": "HAY",
"roadTypeCode": "ST",
"localityName": "PERTH",
"postcode": "6000",
"stateTerritoryCode": "WA"
}
after_request: another value

Does the value "NO" gets parsed as "False" in MySQL when entered through Django's fixtures?

I'm currently using Django 1.2.4 and MySQL 5.1 on Ubuntu 9.10. The model is:
# project/cream/models.py
class IceCream(models.Model):
name = models.CharField(max_length=100)
code = models.CharField(max_length=5)
def __unicode__(self):
return u'%s - %s' % (self.code, self.name)
The fixtures data in a project/cream/fixtures/data.yaml file is:
- model: cream.icecream
pk: 1
fields:
name: Strawberry
code: ST
- model: cream.icecream
pk: 2
fields:
name: Noir Chocolat
code: NO
From the project folder, I invoke the command:
python manage.py loaddata cream/fixtures/data.yaml
The data is successfully loaded in the database but they look like the following:
False - Noir Chocolat
ST - Strawberry
Notice how the first entry is False instead of NO. Does anyone know how to fix this issue in my fixtures?
NO is treated as False because PyYaml implicitly detects that as a boolean value, as seen in resolver.py. If you want it to be the actual string "NO", try putting it in quotes ("").