This question is a follow-up to this one.
I'm running a Django application on top of a MySQL (actually MariaDB) database.
My Django Model looks like this:
from django.db import models
from django.db.models import Count, Sum
class myModel(models.Model):
my_string = models.CharField(max_length=32,)
my_date = models.DateTimeField()
#staticmethod
def get_stats():
logger.info(myModel.objects.values('my_string').annotate(
count=Count("my_string"),
sum1=Sum('my_date'),
sum2=Sum(# WHAT GOES HERE?? #),
)
)
When I call get_stats, it gives me the count and the sum1.
However, for sum2, I want the sum of the following Database expression for each matching row: my_date + 0 (this converts it to a true integer value).
What should I put in the expression above to get that sum returned in sum2?
When I change it to sum2=Sum(F('my_date')), I get the following exception: http://gist.github.com/saqib-zmi/50bf572a972bae5d2871
Not sure, but try F() expression
from datetime import timedelta
myModel.objects.annotate(
count=Count("my_string"),
sum1=Sum('my_date'),
sum2=Sum(F('my_date') + timedelta(days=1))
).values('my_string', 'count', 'sum1', 'sum2')
https://docs.djangoproject.com/en/dev/ref/models/queries/#f-expressions
Related
flask-sqlalchemy model:
from sqlalchemy import extract
from sqlalchemy.sql import func
from app import db
class Entry(db.Model):
date_order = db.Column(db.Date, nullable=False)
version_number = db.Column(db.Integer, nullable=False, default=1)
#hybrid_property
def display_name(self):
return f"{self.date_order.year} ({self.version_number})"
#display_name.expression
def display_name(cls):
return func.concat(extract("year", cls.date_order), " (", cls.version_number, ")")
This works with MySQL. I had to use func.concat because with the simple addition it would cast the date year to an integer and just add them together instead of concatenation. I tested with my custom API and Flask shell:
In [1]: dp = Entry.query.first().display_name
In [2]: Entry.query.filter_by(display_name=dp).all()
Out[2]: [...returns a bunch of entries with that display name...]
But my testing environment runs an SQLite instance. I have this unit test:
# `create_entry` fixture commits the instance to db
def test_user_display_name_expression(create_entry):
entry = create_entry(date_order=date(2022, 11, 11), version_number=3)
filtered = Entry.query.filter_by(display_name="2022 (3)").one()
assert filtered == entry
This returns an error:
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such function: concat
Is there any way to create this concatenation expression so both SQL implementation would query on it?
After many inputs from
PChemGuy I managed to find the issue:
Neither func.concat or adding non-casted properties work as concatenation.
Instead we need sqlalchemy.sql.expression.cast to cast every property to sqlalchemy.String:
from sqlalchemy import extract, String
from sqlalchemy.sql.expression import cast
#display_name.expression
def display_name(cls):
return (
cast(extract("year", cls.date_order), String)
+ " ("
+ cast(cls.version_number, String)
+ ")"
)
Both MysQL and SQLite can understand this.
I am having some problems with my UPDATE statement with MySQL.Connector.
My code apparently works sometimes and others not, and I don't understand why.
I made a test function, for one row, which is very similar to my other main function.
import mysql.connector
from datetime import datetime
def connect():
return mysql.connector.connect(host="xxxxx.xxx", user="xxx", passwd="xxxxxx", db="xxx")
def test():
mydb = connect()
mycursor = mydb.cursor()
sql = "SELECT MAX(value) FROM test"
mycursor.execute(sql)
date = datetime.now().strftime("%d-%m-%Y %H:%M:%S")
for x in mycursor.fetchall():
updateSql = "UPDATE test SET date=%s WHERE value=%s"
vals = (date, x[0])
mycursor.execute(updateSql, vals)
mydb.commit()
print(vals)
print(mycursor.rowcount)
test()
This code doesn't seem to be working, as the printed rowCount value is 0.
My vals are correctly displayed : ('12-01-2020 16:47:15', 'testValue')
However, what is shown on the database is : '00-00-0000 00:00:00' .
Any ideas on how to solve this?
Edit: just found the answer.
I was using the wrong formatting.
date = datetime.now().strftime("%d-%m-%Y %H:%M:%S") wasn't properly recognized.
The DATETIME type in mySql databases requires the format to be the following:
date = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
with the Year in the first place, Month in the second one, and Day in the third one.
If this standard isn't followed then the date will be converted into 0000:00:00 00:00:00 .
I need create sequence but in generic case not using Sequence class.
USN = Column(Integer, nullable = False, default=nextusn, server_onupdate=nextusn)
, this funcion nextusn is need generate func.max(table.USN) value of rows in model.
I try using this
class nextusn(expression.FunctionElement):
type = Numeric()
name = 'nextusn'
#compiles(nextusn)
def default_nextusn(element, compiler, **kw):
return select(func.max(element.table.c.USN)).first()[0] + 1
but the in this context element not know element.table. Exist way to resolve this?
this is a little tricky, for these reasons:
your SELECT MAX() will return NULL if the table is empty; you should use COALESCE to produce a default "seed" value. See below.
the whole approach of inserting the rows with SELECT MAX is entirely not safe for concurrent use - so you need to make sure only one INSERT statement at a time invokes on the table or you may get constraint violations (you should definitely have a constraint of some kind on this column).
from the SQLAlchemy perspective, you need your custom element to be aware of the actual Column element. We can achieve this either by assigning the "nextusn()" function to the Column after the fact, or below I'll show a more sophisticated approach using events.
I don't understand what you're going for with "server_onupdate=nextusn". "server_onupdate" in SQLAlchemy doesn't actually run any SQL for you, this is a placeholder if for example you created a trigger; but also the "SELECT MAX(id) FROM table" thing is an INSERT pattern, I'm not sure that you mean for anything to be happening here on an UPDATE.
The #compiles extension needs to return a string, running the select() there through compiler.process(). See below.
example:
from sqlalchemy import Column, Integer, create_engine, select, func, String
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.sql.expression import ColumnElement
from sqlalchemy.schema import ColumnDefault
from sqlalchemy.ext.compiler import compiles
from sqlalchemy import event
class nextusn_default(ColumnDefault):
"Container for a nextusn() element."
def __init__(self):
super(nextusn_default, self).__init__(None)
#event.listens_for(nextusn_default, "after_parent_attach")
def set_nextusn_parent(default_element, parent_column):
"""Listen for when nextusn_default() is associated with a Column,
assign a nextusn().
"""
assert isinstance(parent_column, Column)
default_element.arg = nextusn(parent_column)
class nextusn(ColumnElement):
"""Represent "SELECT MAX(col) + 1 FROM TABLE".
"""
def __init__(self, column):
self.column = column
#compiles(nextusn)
def compile_nextusn(element, compiler, **kw):
return compiler.process(
select([
func.coalesce(func.max(element.column), 0) + 1
]).as_scalar()
)
Base = declarative_base()
class A(Base):
__tablename__ = 'a'
id = Column(Integer, default=nextusn_default(), primary_key=True)
data = Column(String)
e = create_engine("sqlite://", echo=True)
Base.metadata.create_all(e)
# will normally pre-execute the default so that we know the PK value
# result.inserted_primary_key will be available
e.execute(A.__table__.insert(), data='single row')
# will run the default expression inline within the INSERT
e.execute(A.__table__.insert(), [{"data": "multirow1"}, {"data": "multirow2"}])
# will also run the default expression inline within the INSERT,
# result.inserted_primary_key will not be available
e.execute(A.__table__.insert(inline=True), data='single inline row')
Trying to port some SqlAlchemy to Django and I've got this tricky little bit:
version = Column(
BIGINT,
default=literal_column(
'UNIX_TIMESTAMP() * 1000000 + MICROSECOND(CURRENT_TIMESTAMP)'
),
nullable=False)
What's the best option for porting the literal_column bit to Django? Best idea I've got so far is a function to set as the default that executes the same raw sql, but I'm not sure if there's an easier way? My google-foo is failing me there.
Edit: the reason we need to use a timestamp created by mysql is that we are measuring how out of date something is (so we need to actually know time) and we want, for correctness, to have only one time-stamping authority (so that we don't introduce error using python functions that look at system times, which could be different across servers).
At present I've got:
def get_current_timestamp(self):
cursor = connection.cursor()
cursor.execute("SELECT UNIX_TIMESTAMP() * 1000000 + MICROSECOND(CURRENT_TIMESTAMP)")
row = cursor.fetchone()
return row
version = models.BigIntegerField(default=get_current_timestamp)
which, at this point, sounds like my best/only option.
If you don't care about having a central time authority:
import time
version = models.BigIntegerField(
default = lambda: int(time.time()*1000000) )
To bend the database to your will:
from django.db.models.expressions import ExpressionNode
class NowInt(ExpressionNode):
""" Pass this in the same manner you would pass Count or F objects """
def __init__(self):
super(Now, self).__init__(None, None, False)
def evaluate(self, evaluator, qn, connection):
return '(UNIX_TIMESTAMP() * 1000000 + MICROSECOND(CURRENT_TIMESTAMP))', []
### Model
version = models.BigIntegerField(default=NowInt())
because expression nodes are not callables, the expression will be evaluated database side.
I have recently started using SQLALCHEMY to query a my-sql database. I want to generate a select statement that uses the "INTO OUTFILE <file>" syntax to export query results to a test file. For example:
SELECT *
FROM table
INTO OUTFILE '/tmp/export.txt';
Is there a way to generate the "INTO OUTFILE..." clause using SQLALCHEMY?
If not, can I subclass one of the SQLALCHEMY classes so I can build that clause myself?
Thanks.
I did some thinking and poking around the examples on the SQLAlchemy site and figured it out. (Also posted to sql-alchemy user reciptes)
from sqlalchemy import *
from sqlalchemy.sql.expression import Executable, ClauseElement
from sqlalchemy.ext import compiler
class SelectIntoOutfile(Executable, ClauseElement):
def __init__(self, select, file):
self.select = select
self.file = file
#compiler.compiles(SelectIntoOutfile)
def compile(element, compiler, **kw):
return "%s INTO OUTFILE '%s'" % (
compiler.process(element.select), element.file
)
e = SelectIntoOutfile(select([s.dim_date_table]).where(s.dim_date_table.c.Year==2009), '/tmp/test.txt')
print e
eng.execute(e)