proper Django ORM syntax to make this code work in MySQL - mysql

I have the following django code working on an sqlite database but for some unknown reason I get a syntax error if I change the backend to MySQL...does django's ORM treat filtering differently in MySQL?
def wsjson(request,imei):
wstations = WS.objects.annotate(latest_wslog_date=Max('wslog__date'),latest_wslog_time=Max('wslog__time'))
logs = WSLog.objects.filter(date__in=[b.latest_wslog_date for b in wstations],time__in=[b.latest_wslog_time for b in wstations],imei__exact=imei)
data = serializers.serialize('json',logs)
return HttpResponse(data,'application/javascript')
The code basically gets the latest logs from WSlog corresponding to each record in WS and serializes it to json.
Models are defined as:
class WS(models.Model):
name = models.CharField(max_length=20)
imei = models.CharField(max_length=15)
description = models.TextField()
def __unicode__(self):
return self.name
class WSLog(models.Model):
imei = models.CharField(max_length=15)
date = models.DateField()
time = models.TimeField()
data1 = models.DecimalField(max_digits=8,decimal_places=3)
data2 = models.DecimalField(max_digits=8,decimal_places=3)
WS = models.ForeignKey(WS)
def __unicode__(self):
return self.imei

Ok finally managed to fix it with code below
def wsjson(request,imei):
wstations = WS.objects.annotate(latest_wslog_date=Max('wslog__date'),latest_wslog_time=Max('wslog__time'))logs = WSLog.objects.filter(date__in=[a.latest_wslog_date for a in wstations],time__in=[b.latest_wslog_time for b in wstations],imei__exact=imei)
data = serializers.serialize('json',logs)
return HttpResponse(data,'application/javascript')
the weird part was the original code worked as expected in an ubuntu machine running MySql but failed in a windows machine (also running MySql)

Related

Getting error when trying to insert JSON field in Postgres

I have the following schema in django.
class function(models.Model):
func_id = models.IntegerField(primary_key=True)
func_name = models.CharField(max_length=30)
func_args = JSONField()
func_version = models.CharField(max_length=20)
func_desc = models.CharField(max_length=500)
user = models.ForeignKey(user_data,on_delete=models.CASCADE)
container_path = models.CharField(max_length=200)
class Meta:
db_table = 'function'
When I am trying to run the following query in postgres shell
INSERT INTO
function(func_id,func_name,func_args,func_version,func_desc,user_id,container_path)
VALUES (101,'Sum',{"input1":"a","input2":"b"},'1.7','blahblah',105,'/path');
I am getting below error:
ERROR: syntax error at or near "{"
LINE 1: ...nc_desc,user_id,container_path) VALUES (101,'Sum',{"input1":...
Any clue where I am going wrong?
Try enclosing your JSON in single quotes '{...}'. Even though JSONField supposedly works with python dictionaries, when doing raw SQL, postgresql won't understand that syntax.

Iterating through multiline input, and match to database items

I need help iterating through input to a webapp I'm writing, which looks like:
The users will be inputting several hundred (or thousands) of urls pasted from excel documents, each on a new line like this. Thus far, as you can see, I've created the input page, an output page, and written the code to query the database.
from flask import Flask,render_template, request
from flask_sqlalchemy import SQLAlchemy
from urllib.parse import urlparse
from sqlalchemy.ext.declarative import declarative_base
app = Flask(__name__)
app.config["DEBUG"] = True
app.config["SECRET_KEY"] = "secret_key_here"
db = SQLAlchemy(app)
SQLALCHEMY_DATABASE_URI = db.create_engine(connector_string_here))
app.config[SQLALCHEMY_DATABASE_URI] = SQLALCHEMY_DATABASE_URI
app.config["SQLALCHEMY_POOL_RECYCLE"] = 299
app.config["SQLALCHEMY_TRACK_MODIFICATIONS"] = False
db.Model = declarative_base()
class Scrapers(db.Model):
__tablename__ = "Scrapers"
id = db.Column(db.Integer, primary_key = True)
scraper_dom = db.Column(db.String(255))
scraper_id = db.Column(db.String(128))
db.Model.metadata.create_all(SQLALCHEMY_DATABASE_URI)
Session = db.sessionmaker()
Session.configure(bind=SQLALCHEMY_DATABASE_URI)
session = Session()
scrapers = session.query(Scrapers.scraper_dom, Scrapers.scraper_id).all()
#app.route("/", methods=["GET","POST"])
def index():
if request.method == "Get":
return render_template("url_page.html")
else:
return render_template("url_page.html")
#app.route("/submit", methods=["GET","POST"])
def submit():
sites = [request.form["urls"]]
for site in sites:
que = urlparse(site).netloc
return render_template("submit.html", que=que)
#scrapers.filter(Scrapers.scraper_dom.in_(
#next(x.scraper_id for x in scrapers if x.matches(self.fnetloc))
As is apparent, this is incomplete. I've omitted previous attempts at matching the input, as I realized I had issues iterating through the input. At first, I could only get it to print all of the input instead of iterating over it. And now, it prints like this:
Which is just repeating the urlparse(site).netloc for the first line of input, some random number of times. It is parsing correctly and returning the actual value I will need to use later (for each urlparse(site).netloc match scraper_dom and return associated scraper_id). Now, though, I've tried using input() but kept getting errors with [request.form["urls"]] not being an iterable.
Please help, it'd be much appreciated.
Output of sites:
New output with:
que = [urlparse(site).netloc for site in request.form["urls"].split('\n')]

Django equivalent for MySQL bit datatype

I am facing issues with "makeflag" field which is bit(1) type in my database(MySQL). I have tried using booleanField and bit1booleanfield with below syntax. But i am getting error with both. when i try POST request with json data on this model,
I get error as
"Data too long for column" on passing 1 or 0 as value.
And when i give true or false as value, then i get 400 Bad Request.
Can someone please help me understand how can i post data using django and json for bit field (of mysql).
makeflag=models.BooleanField(db_column='MakeFlag', default=1)
makeflag=Bit1BooleanField()
My model is the next:
class Product(models.Model):
productid = models.AutoField(db_column='ProductID', primary_key=True)
name = models.CharField(db_column='Name', max_length=50)
productnumber = models.CharField(db_column='ProductNumber', max_length=25)
makeflag = models.TextField(db_column='MakeFlag', max_length=1)
color = models.CharField(db_column='Color', max_length=15, blank=True)
safetystocklevel = models.SmallIntegerField(db_column='SafetyStockLevel')
reorderpoint = models.SmallIntegerField(db_column='ReorderPoint')
standardcost = models.FloatField(db_column='StandardCost')
You probably need to use django-mysql for mysql specific functionality. Have a look docs for bit here

Implementing MySQL "generated columns" on Django 1.8/1.9

I discovered the new generated columns functionality of MySQL 5.7, and wanted to replace some properties of my models by those kind of columns. Here is a sample of a model:
class Ligne_commande(models.Model):
Quantite = models.IntegerField()
Prix = models.DecimalField(max_digits=8, decimal_places=3)
Discount = models.DecimalField(max_digits=5, decimal_places=3, blank=True, null=True)
#property
def Prix_net(self):
if self.Discount:
return (1 - self.Discount) * self.Prix
return self.Prix
#property
def Prix_total(self):
return self.Quantite * self.Prix_net
I defined generated field classes as subclasses of Django fields (e.g. GeneratedDecimalField as a subclass of DecimalField). This worked in a read-only context and Django migrations handles it correctly, except a detail : generated columns of MySQL does not support forward references and django migrations does not respect the order the fields are defined in a model, so the migration file must be edited to reorder operations.
After that, trying to create or modify an instance returned the mysql error : 'error totally whack'. I suppose Django tries to write generated field and MySQL doesn't like that. After taking a look to django code I realized that, at the lowest level, django uses the _meta.local_concrete_fields list and send it to MySQL. Removing the generated fields from this list fixed the problem.
I encountered another problem: during the modification of an instance, generated fields don't reflect the change that have been made to the fields from which they are computed. If generated fields are used during instance modification, as in my case, this is problematic. To fix that point, I created a "generated field descriptor".
Here is the final code of all of this.
Creation of generated fields in the model, replacing the properties defined above:
Prix_net = mk_generated_field(models.DecimalField, max_digits=8, decimal_places=3,
sql_expr='if(Discount is null, Prix, (1.0 - Discount) * Prix)',
pyfunc=lambda x: x.Prix if not x.Discount else (1 - x.Discount) * x.Prix)
Prix_total = mk_generated_field(models.DecimalField, max_digits=10, decimal_places=2,
sql_expr='Prix_net * Quantite',
pyfunc=lambda x: x.Prix_net * x.Quantite)
Function that creates generated fields. Classes are dynamically created for simplicity:
from django.db.models import fields
def mk_generated_field(field_klass, *args, sql_expr=None, pyfunc=None, **kwargs):
assert issubclass(field_klass, fields.Field)
assert sql_expr
generated_name = 'Generated' + field_klass.__name__
try:
generated_klass = globals()[generated_name]
except KeyError:
globals()[generated_name] = generated_klass = type(generated_name, (field_klass,), {})
def __init__(self, sql_expr, pyfunc=None, *args, **kwargs):
self.sql_expr = sql_expr
self.pyfunc = pyfunc
self.is_generated = True # mark the field
# null must be True otherwise migration will ask for a default value
kwargs.update(null=True, editable=False)
super(generated_klass, self).__init__(*args, **kwargs)
def db_type(self, connection):
assert connection.settings_dict['ENGINE'] == 'django.db.backends.mysql'
result = super(generated_klass, self).db_type(connection)
# double single '%' if any because it will clash with later Django format
sql_expr = re.sub('(?<!%)%(?!%)', '%%', self.sql_expr)
result += ' GENERATED ALWAYS AS (%s)' % sql_expr
return result
def deconstruct(self):
name, path, args, kwargs = super(generated_klass, self).deconstruct()
kwargs.update(sql_expr=self.sql_expr)
return name, path, args, kwargs
generated_klass.__init__ = __init__
generated_klass.db_type = db_type
generated_klass.deconstruct = deconstruct
return generated_klass(sql_expr, pyfunc, *args, **kwargs)
The function to register generated fields in a model. It must be called at django start-up, for example in the ready method of the AppConfig of the application.
from django.utils.datastructures import ImmutableList
def register_generated_fields(model):
local_concrete_fields = list(model._meta.local_concrete_fields[:])
generated_fields = []
for field in model._meta.fields:
if hasattr(field, 'is_generated'):
local_concrete_fields.remove(field)
generated_fields.append(field)
if field.pyfunc:
setattr(model, field.name, GeneratedFieldDescriptor(field.pyfunc))
if generated_fields:
model._meta.local_concrete_fields = ImmutableList(local_concrete_fields)
And the descriptor. Note that it is used only if a pyfunc is defined for the field.
class GeneratedFieldDescriptor(object):
attr_prefix = '_GFD_'
def __init__(self, pyfunc, name=None):
self.pyfunc = pyfunc
self.nickname = self.attr_prefix + (name or str(id(self)))
def __get__(self, instance, owner):
if instance is None:
return self
if hasattr(instance, self.nickname) and not instance.has_changed:
return getattr(instance, self.nickname)
return self.pyfunc(instance)
def __set__(self, instance, value):
setattr(instance, self.nickname, value)
def __delete__(self, instance):
delattr(instance, self.nickname)
Note the instance.has_changed that must tell if the instance is being modified. If found a solution for this here.
I have done extensive tests of my application and it works fine, but I am far from using all django functionalities. My question is: could this settings clash with some use cases of django ?

How to get field value after distincting with foreinkey

I'm beginner in Djanog and trying to display values that distinct result with foreinkey.
Here are my env and example model.
Django 1.8
Mysql5
Python2.7
class a_group(models.Model):
num = models.AutoField(primary_key=True)
title = models.CharField(max_length=50)
def __unicode__(self):
return self.title
class b_group(models.Model):
no = models.AutoField(primary_key=True)
group = models.ForeignKey(a_group)
And then I tried to distinct with group field like this.
g = b_group.objects.values('group').distinct()
But, as mentioned at here https://docs.djangoproject.com/en/dev/ref/models/querysets/#values , it only return pk, not title.
Is there anyway to get title field value also?
You can also refer to fields on related models with reverse relations through OneToOneField,ForeignKey and ManyToManyField attributes, you can do as follow:
g = b_group.objects.values('group__title').distinct()
to access a field of related model Django use by convention double underscore.