Does the value "NO" gets parsed as "False" in MySQL when entered through Django's fixtures? - mysql

I'm currently using Django 1.2.4 and MySQL 5.1 on Ubuntu 9.10. The model is:
# project/cream/models.py
class IceCream(models.Model):
name = models.CharField(max_length=100)
code = models.CharField(max_length=5)
def __unicode__(self):
return u'%s - %s' % (self.code, self.name)
The fixtures data in a project/cream/fixtures/data.yaml file is:
- model: cream.icecream
pk: 1
fields:
name: Strawberry
code: ST
- model: cream.icecream
pk: 2
fields:
name: Noir Chocolat
code: NO
From the project folder, I invoke the command:
python manage.py loaddata cream/fixtures/data.yaml
The data is successfully loaded in the database but they look like the following:
False - Noir Chocolat
ST - Strawberry
Notice how the first entry is False instead of NO. Does anyone know how to fix this issue in my fixtures?


NO is treated as False because PyYaml implicitly detects that as a boolean value, as seen in resolver.py. If you want it to be the actual string "NO", try putting it in quotes ("").

Related

json schema validation does not work for dependencies?

As far as I understand, using:
dependencies:
cake:
- eggs
- flour
in a JSON schema validation (I actually parse the schema in YAML, but it should not really matter) should forbid the presence of a cake without an entry for eggs and flour. So this should be rejected:
receip:
cake: crepes
while this should be accepted:
receip:
eggs: 3 eggs, given by farmer
flour: 500g
cake: crepes
Unfortunately, both cases are accepted in my case. Any idea what I made wrong? I also tried to add another level required as proposed by jsonSchema attribute conditionally required but the problem is the same.
MWE
In case it helps, here are all the files that I'm using.
debug_schema.yml:
$schema: https://json-schema.org/draft/2020-12/schema
type: object
properties:
receip:
type: object
properties:
eggs:
type: string
flour:
type: string
cake:
type: string
dependencies:
cake:
- eggs
- flour
b.yml:
receip:
## I'd like to forbid a cake without eggs and flour
## Should not validate until I uncomment:
# eggs: 3 eggs, given by farmer
# flour: 500g
cake: crepes
validate_yaml.py
#!/usr/bin/env python3
import yaml
from jsonschema import validate, ValidationError
import argparse
# Instantiate the parser
parser = argparse.ArgumentParser(description='This tool not only checks if the YAML file is correct in term'
+ ' of syntex, but it also checks if the structure of the YAML file follows'
+ ' the expectations of this project (e.g. that it contains a list of articles,'
+ ' with each articles having a title etc…).')
# Optional argument
parser.add_argument('--schema', type=str, default="schema.yml",
help='Path to the schema')
parser.add_argument('--yaml', type=str, default="science_zoo.yml",
help='Path to the YAML file to verify')
args = parser.parse_args()
class YAMLNotUniqueKey(Exception):
pass
# By default, PyYAML does not raise an error when two keys are identical, which is an issue here
# see https://github.com/yaml/pyyaml/issues/165#issuecomment-430074049
class UniqueKeyLoader(yaml.SafeLoader):
def construct_mapping(self, node, deep=False):
mapping = set()
for key_node, value_node in node.value:
key = self.construct_object(key_node, deep=deep)
if key in mapping:
raise YAMLNotUniqueKey(f"Duplicate key {key!r} found.")
mapping.add(key)
return super().construct_mapping(node, deep)
with open(args.schema, "r") as schema_stream:
try:
schema = yaml.load(schema_stream, UniqueKeyLoader)
with open(args.yaml, "r") as yaml_stream:
try:
validate(yaml.load(yaml_stream, UniqueKeyLoader), schema)
print("The YAML file is correct and follows the schema, congrats! :D")
except ValidationError as exc:
print(exc.message)
print("The problem is located in the JSON at the path {}.".format(exc.json_path))
except YAMLNotUniqueKey as exc:
print("ERROR: {}".format(exc))
print("Rename in the YAML one the two instances of this key to ensure all keys are unique.")
except yaml.YAMLError as exc:
print("Errors when trying to load the YAML file:")
print(exc)
except YAMLNotUniqueKey as exc:
print("ERROR: {}".format(exc))
print("Rename in the schema one the two instances of this key to ensure all keys are unique.")
except yaml.YAMLError as exc:
print("Errors when trying to load the schema file:")
print(exc)
Command:
./validate_yaml.py --yaml b.yml --schema debug_schema.yml

Oh, so it seems that this keyword has been deprecated, and split into dependentRequired (see e.g. this example) and dependentSchemas (see e.g. this example). Just using dependentRequired solves the issue:
$schema: https://json-schema.org/draft/2020-12/schema
type: object
properties:
receip:
type: object
properties:
eggs:
type: string
flour:
type: string
cake:
type: string
dependentRequired:
cake:
- eggs
- flour

Airflow 1.x TriggerDagRunOperator pass {{ds}} as conf

In Airflow 1.x (not 2.x), I want DAG1 to trigger DAG2.
and want to pass DAG1's templated date {{ds}} as a 'conf' dict parameter like {"day":"{{ds}}"} to DAG2, so that DAG2 can access it via {{dag_run.conf['day']}}
But DAG1 just ends up passing the literal string '{{ds}}' instead of '2021-12-03'
DAG2 uses an SSHOperator, not PythonOperator (for which a solution seems to exist)
DAG 1:
from airflow.operators import TriggerDagRunOperator
def fn_day(context, dagrun_order):
dagrun_order.payload = {"day": "{{ds}}"}
return dagrun_order
trig = TriggerDagRunOperator(
trigger_dag_id="ssh",
task_id='trig',
python_callable=fn_day,
dag=dag)
DAG 2 :
from airflow.contrib.operators.ssh_operator import SSHOperator
ssh = SSHOperator(
ssh_conn_id='ssh_vm',
task_id='echo',
command="echo {{dag_run.conf['day']}}"
)

How to efficiently parse JSON data with multiple keys in Python 2.7?

I'm writing a script that will check the CVS COVID vaccine availability for cities in my state of VA. I have been successful getting the data I'm looking for, but my code is hard coded in some areas. I'm specifically asking for help improving my code in the areas number 1 & 2 below:
The JSON file can be found here:
https://www.cvs.com//immunizations/covid-19-vaccine.vaccine-status.VA.json?vaccineinfo
I'm trying to access the data in the responsePayloadData key. The only way I could figure out how to do this is to make it the only key. For that reason, I deleted the other key responseMetaData:
#remove the key that we don't need
del obj['responseMetaData']
I'm also not sure how to dynamically loop through the VA items without hard coding the number of cities I know are there in the data:
for x, y in obj.items():
for a in range(34):
Here's the full code:
import requests
import json
import time
from datetime import datetime
import urllib2
try:
import indigo
except:
pass
strAvail = "False"
strAvailCity = "None"
try:
# download raw json object from CVS Virginia Website
url = "https://www.cvs.com//immunizations/covid-19-vaccine.vaccine-status.VA.json?vaccineinfo"
data = urllib2.urlopen(url).read().decode()
except urllib2.HTTPError, err:
return {"error": err.reason, "error_code": err.code}
# parse json object
obj = json.loads(data)
# remove the key that we don't need
del obj['responseMetaData']
# loop through the JSON dictionary and check availability
# status options: {"Fully Booked", "Available"}
for x, y in obj.items():
for a in range(34):
# print('City: ' + y['data']['VA'][a]['city'])
# print('Total Available: ' + y['data']['VA'][a]['totalAvailable'])
# print('Percent Available: ' + y['data']['VA'][a]['pctAvailable'])
# print('Status: ' + y['data']['VA'][a]['status'])
# print("------------------------------")
# If there is availability anywhere in the state, take some action.
if y['data']['VA'][a]['status'] == "Available":
strAvail = True
strAvailCity = y['data']['VA'][a]['city']
# Log timestamp for this check to the JSON
now = datetime.now()
strDateTime = now.strftime("%m/%d/%Y %I:%M %p")
EDIT: Since the JSON is not available outside the US. I've pasted it below:
{"responsePayloadData":{"currentTime":"2021-02-11T14:55:00.470","data":{"VA":[{"totalAvailable":"1","city":"ABINGDON","state":"VA","pctAvailable":"0.19%","status":"Fully Booked"},{"totalAvailable":"0","city":"ALEXANDRIA","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"ARLINGTON","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"BEDFORD","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"BLACKSBURG","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"CHARLOTTESVILLE","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"CHATHAM","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"CHESAPEAKE","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"1","city":"DANVILLE","state":"VA","pctAvailable":"0.19%","status":"Fully Booked"},{"totalAvailable":"2","city":"DUBLIN","state":"VA","pctAvailable":"0.39%","status":"Fully Booked"},{"totalAvailable":"0","city":"FAIRFAX","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"FREDERICKSBURG","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"GAINESVILLE","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"HAMPTON","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"HARRISONBURG","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"LEESBURG","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"LYNCHBURG","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"MARTINSVILLE","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"MECHANICSVILLE","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"MIDLOTHIAN","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},
{"totalAvailable":"0","city":"NEWPORT NEWS","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"NORFOLK","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"PETERSBURG","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"PORTSMOUTH","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"RICHMOND","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"ROANOKE","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},
{"totalAvailable":"0","city":"ROCKY MOUNT","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"STAFFORD","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"SUFFOLK","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},
{"totalAvailable":"0","city":"VIRGINIA BEACH","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"WARRENTON","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"WILLIAMSBURG","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"WINCHESTER","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"WOODSTOCK","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"}]}},"responseMetaData":{"statusDesc":"Success","conversationId":"Id-beb5f68730b34e6aa3bbc1fd927ea12b","refId":"Id-b4a7256078789eb59b8912b4","operation":"getInventorybyCity","statusCode":"0000"}}

Regarding problem 1, you can just access the data by key. You don't need to delete the other key:
payload = obj['responsePayloadData']
For the second problem, you can just iterate over the items in the list associated with obj['data']['VA']:
for city in payload['data']['VA']:
print(city)
{'city': 'ABINGDON',
'pctAvailable': '0.19%',
'state': 'VA',
'status': 'Fully Booked',
'totalAvailable': '1'}
{'city': 'ALEXANDRIA',
'pctAvailable': '0.00%',
'state': 'VA',
'status': 'Fully Booked',
'totalAvailable': '0'}
...

Grails - Bootstrapping test data fails

I'm encounter a strange error and I don't know how to fix it. I'm creating test entities in Bootstrap.groovy:
def init = { servletContext ->
if (Environment.current != Environment.TEST) {
servletContext.reloadId = new Date().getTime() // for reloading css
// create some insurance companies
InsuranceCompany ozp = InsuranceCompany.findOrSaveWhere(code: "207", acronym: "OZP", name: "Oborová zdravotní pojišťovna zaměstnanců bank, pojišťoven a stavebnictví")
InsuranceCompany vzp = InsuranceCompany.findOrSaveWhere(code: "111", acronym: "VZP", name: "Všeobecná zdravotní pojišťovna České republiky")
InsuranceCompany vozp = InsuranceCompany.findOrSaveWhere(code: "201", acronym: "VOZP", name: "Vojenská zdravotní pojišťovna České republiky")
Patient miladavrana = Patient.findOrSaveWhere(title: "JUDr", firstName: "Milada", lastName: "Vraná", RC: "0307080846", insuranceCompany: vozp,address: new Address(street: "5.Května", number: 568, city: "Korouhev", PSC: 16422))
Patient jiristarecek = Patient.findOrSaveWhere(title: "", firstName: "Jiří", lastName: "Stareček", RC: "9457174705", insuranceCompany: cpzp,address: new Address(street: "Nad Schody", number: 180, city: "Velký Valtinov", PSC: 51521))
}
Actually the bootstrap file is quite longer, it has almost 1000 lines of test data. Everything works ok if I comment 2/3 of those data. When I have all of these data uncomment and grails is trying to create it, it fails:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x000000000262b036, pid=4152, tid=8016
#
# JRE version: 6.0_29-b11
# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.4-b02 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# j BootStrap$_closure1.doCall(Ljava/lang/Object;)Ljava/lang/Object;+87
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#
--------------- T H R E A D ---------------
Current thread (0x000000000d985800): JavaThread "pool-7-thread-1" [_thread_in_Java, id=8016, stack(0x000000000bcf0000,0x000000000bdf0000)]
siginfo: ExceptionCode=0xc0000005, reading address 0x0000000138c797cf
Registers:
RAX=0x000000000bdedd78, RBX=0x000000000bdedd78, RCX=0x00000000d9d1e488, RDX=0x00000000d9d0abff
RSP=0x000000000bdedd88, RBP=0x000000000bdeddb8, RSI=0x000000000d985800, RDI=0x000000000ee83ed0
R8 =0x0000000000000004, R9 =0x00000000d4b4e890, R10=0x000000006dfc4f80, R11=0x000000000ee83f38
R12=0x0000000000000000, R13=0x00000000d9d1d8ce, R14=0x000000000bdee5b0, R15=0x000000000d985800
RIP=0x000000000262b036, EFLAGS=0x0000000000010287
this is part of the log the error generates. I'm using Grails 2.0.1 & MySQL 5.5.21 as a database. Anyone could hep or has this error before? Thank you very much.
Cheers,
Lojza

Try increasing the heap allocation to your JVM through JAVAOPTS.

Is there a tool to check database integrity in Django?

The MySQL database powering our Django site has developed some integrity problems; e.g. foreign keys that refer to nonexistent rows. I won't go into how we got into this mess, but I'm now looking at how to fix it.
Basically, I'm looking for a script that scans all models in the Django site, and checks whether all foreign keys and other constraints are correct. Hopefully, the number of problems will be small enough so they can be fixed by hand.
I could code this up myself but I'm hoping that somebody here has a better idea.
I found django-check-constraints but it doesn't quite fit the bill: right now, I don't need something to prevent these problems, but to find them so they can be fixed manually before taking other steps.
Other constraints:
Django 1.1.1 and upgrading has been determined to break things
MySQL 5.0.51 (Debian Lenny), currently with MyISAM tables
Python 2.5, might be upgradable but I'd rather not right now
(Later, we will convert to InnoDB for proper transaction support, and maybe foreign key constraints on the database level, to prevent similar problems in the future. But that's not the topic of this question.)

I whipped up something myself. The management script below should be saved in myapp/management/commands/checkdb.py. Make sure that intermediate directories have an __init__.py file.
Usage: ./manage.py checkdb for a full check; use --exclude app.Model or -e app.Model to exclude the model Model in the app app.
from django.core.management.base import BaseCommand, CommandError
from django.core.management.base import NoArgsCommand
from django.core.exceptions import ObjectDoesNotExist
from django.db import models
from optparse import make_option
from lib.progress import with_progress_meter
def model_name(model):
return '%s.%s' % (model._meta.app_label, model._meta.object_name)
class Command(BaseCommand):
args = '[-e|--exclude app_name.ModelName]'
help = 'Checks constraints in the database and reports violations on stdout'
option_list = NoArgsCommand.option_list + (
make_option('-e', '--exclude', action='append', type='string', dest='exclude'),
)
def handle(self, *args, **options):
# TODO once we're on Django 1.2, write to self.stdout and self.stderr instead of plain print
exclude = options.get('exclude', None) or []
failed_instance_count = 0
failed_model_count = 0
for app in models.get_apps():
for model in models.get_models(app):
if model_name(model) in exclude:
print 'Skipping model %s' % model_name(model)
continue
fail_count = self.check_model(app, model)
if fail_count > 0:
failed_model_count += 1
failed_instance_count += fail_count
print 'Detected %d errors in %d models' % (failed_instance_count, failed_model_count)
def check_model(self, app, model):
meta = model._meta
if meta.proxy:
print 'WARNING: proxy models not currently supported; ignored'
return
# Define all the checks we can do; they return True if they are ok,
# False if not (and print a message to stdout)
def check_foreign_key(model, field):
foreign_model = field.related.parent_model
def check_instance(instance):
try:
# name: name of the attribute containing the model instance (e.g. 'user')
# attname: name of the attribute containing the id (e.g. 'user_id')
getattr(instance, field.name)
return True
except ObjectDoesNotExist:
print '%s with pk %s refers via field %s to nonexistent %s with pk %s' % \
(model_name(model), str(instance.pk), field.name, model_name(foreign_model), getattr(instance, field.attname))
return check_instance
# Make a list of checks to run on each model instance
checks = []
for field in meta.local_fields + meta.local_many_to_many + meta.virtual_fields:
if isinstance(field, models.ForeignKey):
checks.append(check_foreign_key(model, field))
# Run all checks
fail_count = 0
if checks:
for instance in with_progress_meter(model.objects.all(), model.objects.count(), 'Checking model %s ...' % model_name(model)):
for check in checks:
if not check(instance):
fail_count += 1
return fail_count
I'm making this a community wiki because I welcome any and all improvements to my code!

Thomas' answer is great but is now a bit out of date.
I have updated it as a gist to support Django 1.8+.