Django annotate with frequency - mysql

(django 3.2.12, python 3.9.3, MySQL 8.0.28)
Imagine models like the following:
class User(models.Model):
email = models.EmailField(...)
created_datetime = models.DatetimeField(...)
class UserLog(models.Model):
created_datetime = models.DatetimeField(...)
user = models.ForeignKey('user.User' ...)
login = models.BooleanField('Log in' ..)
And the following query, destined to annotate each user in the queryset with the frequency of their logs(when log.login=True):
users = User.objects.filter(
Q(...)
).annotate(
login_count=Count('userlog', filter=Q(userlog__login=True)),
login_duration_over=Now() - F('created_datetime'),
login_frequency=ExpressionWrapper(
F('login_duration_over') / F('login_count'),
output_field=models.DurationField()
),
)
This results in a SQL error:
(1064, "You have an error in your SQL syntax;)
The generated SQL (fragment for login_frequency) looks like this:
(
INTERVAL TIMESTAMPDIFF(
MICROSECOND,
`user_user`.`created_datetime`,
CURRENT_TIMESTAMP
) MICROSECOND / (
COUNT(
CASE WHEN `user_userlog`.`login` THEN `user_userlog`.`id` ELSE NULL END
)
)
) AS `login_frequency`,
and MySQL does not seem to like it. A similar code works on SQLlite and, I am told on PG.
What is wrong with the ExpressionWrapper on MySQL, any idea?

Found a workaround:
users = User.objects.filter(
Q(...)
).annotate(
login_count=Count('userlog', filter=Q(userlog__login=True)),
login_duration_over=Now() - F('created_datetime'),
login_frequency=Cast(
ExpressionWrapper(
Cast(F('login_duration_over'), output_field=models.BigIntegerField()) / F('login_count'),
output_field=models.BigIntegerField()
),
output_field=models.DurationField()
)
)
this forces the DIVIDE operation to be performed db-side on bigints and once that is done, cast it back to a timedelta.
MySQL stopped screaming and the results are correct.
Even though that work, this feels ugly. Could there not be a better way?

Related

Select SQL statement from multiple database

My question has 2 parts, and i am unsure if it is my query which is causing the error or data adapter.
Part 1. I have a mySQL query which works fine on SQLyog, the query involves selection from tables existing in different databases. And both databases has been set to have the same accecss rights for the user account used at the connection string.
Below will be my query.
SELECT `portaldb`.`users`.`full_name` AS 'Name of User'
,`Systemrevamp`.`System_countries`.`CountryName` AS 'Quoted For'
,`Systemrevamp`.`uniquequote`.`UniqueQuote` AS 'System Quote ID'
, IF (
LEFT(`Systemrevamp`.`uniquequote`.`username`, 1) = ' '
,'Web Access'
,'Bulk Upload'
) AS 'Type'
,DATE_FORMAT(`Systemrevamp`.`uniquequote`.`insertedon`, '%d-%b-%Y') AS 'Quoted On'
FROM `portaldb`.`users`
INNER JOIN `Systemrevamp`.`uniquequote` ON TRIM(`Systemrevamp`.`uniquequote`.`UserName`) = `portaldb`.`users`.`usrname`
INNER JOIN `Systemrevamp`.`System_countries` ON `Systemrevamp`.`System_countries`.`Code` = `Systemrevamp`.`uniquequote`.`CountryCode`
INNER JOIN `portaldb`.`permission_details` ON `portaldb`.`permission_details`.`user_ID` = `portaldb`.`users`.`user_ID`
WHERE `portaldb`.`permission_details`.`group_ID` = '5'
AND `Systemrevamp`.`uniquequote`.`insertedon` >= (NOW() - INTERVAL 3 MONTH)
ORDER BY `portaldb`.`users`.`full_name` ASC,`Systemrevamp`.`uniquequote`.`insertedon` ASC
Visual studio keeps telling me there is syntax error and to check the version of SQL, but I am able to retrieve at the database.
Part 2. Below is a snippet of my code. My question for this part is, if the above query is used, what source table should I put for the data adapter to fill at 'XXX'?
myDataAdapter = New MySqlDataAdapter(strSQL, myConnection)
allUserDataset = New DataSet()
myDataAdapter.Fill(allUserDataset, "XXX")
gvAllQuotes.DataSource = allUserDataset
gvAllQuotes.DataBind()
Please let me know if more information is needed. Thank you.

SQL Field Update Syntax Issues with complex selection logic

The query below works fine and is what I need.
I want to add a new keyword onto "Keywords" e.g.
UPDATE bugs SET bug.keywords = CONCAT(bug.keywords, ', Report:DevProcess')
However no matter where I place this in the logic below, I get a syntax error.
The web examples I have seen and in Stackoverflow are for simple
Update ... WHERE .... examples.
SET #StartDate = '2016-03-01';
SET #EndDate = '2016-03-31';
SELECT
bugs_activity.bug_id,
bug.status_whiteboard AS Whiteboard,
bug.keywords AS Keywords,
bug.bug_status,
bug.resolution,
SUM(CASE WHEN fd.name = 'bug_status' AND (bugs_activity.added = 'VERIFIED' OR bugs_activity.added = 'CLOSED') THEN 1 ELSE 0 END) AS ClosedCount,
MIN(CASE WHEN fd.name = 'bug_status' AND bugs_activity.added = 'VERIFIED' THEN bug_when ELSE NULL END) AS verifiedDate,
MIN(CASE WHEN fd.name = 'bug_status' AND bugs_activity.added = 'CLOSED' THEN bug_when ELSE NULL END) AS closedDate
FROM bugs_activity
INNER JOIN bugs bug
ON bugs_activity.bug_id = bug.bug_id
INNER JOIN fielddefs fd
ON bugs_activity.fieldid = fd.id
WHERE
(bugs_activity.bug_when BETWEEN '2015-09-01' AND #EndDate)
AND (Keywords LIKE '%Region:Europe%')
AND NOT (Keywords LIKE '%Report:DevProcess%')
GROUP BY bug_id
HAVING
ClosedCount > 0
AND (
(verifiedDate IS NOT NULL AND verifiedDate >= #StartDate)
OR (verifiedDate IS NULL AND (closedDate IS NOT NULL AND closedDate >= #StartDate))
)
Additional info from questions:
Linqpad SQL talking to MySQL DB
From linqpad - You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'UPDATE bugs SET bug.keywords = CONCAT(bug.keywords, ', Report:DevProcess')' at line 20
I removed GroupBy line, same error.
CONCAT is what a web search tells me How to prepend a string to a column value in MySQL?
In the selected "bugs" I want to "UPDATE bugs SET bug.keywords = CONCAT(bug.keywords, ', Report:DevProcess') "
Bugzilla has historically for a long time used multiple keywords. It is what it is whether good or bad.
== 31/05/2016 update ==
I simplified the query and got past the syntax error, however no update. I confirmed account had DB write access by using the read account which produced an access denied error.
-- this shows the one record
SELECT bug_id
FROM bugs
WHERE (bugs.bug_status = 'VERIFIED') AND (bugs.status_whiteboard LIKE '%Leiden%') and (bugs.keywords LIKE '%Region:Europe%') AND NOT (bugs.keywords LIKE '%Report:DevProcess%')
-- this runs without an error but shows no records updated
UPDATE bugs SET bugs.keywords = CONCAT(bugs.keywords, ', Report:DevProcess')
WHERE (bugs.bug_status = 'VERIFIED') AND (bugs.status_whiteboard LIKE '%Leiden%') and (bugs.keywords LIKE '%Region:Europe%') AND NOT (bugs.keywords LIKE '%Report:DevProcess%')
The simpler syntax does "work", i.e. no syntax error however the write did not work initially. This turned out to be because the keyword schema required the keyword to be pre-defined. Adding the keyword in resulting in the record being updated.
-- this runs without an error but shows no records updated
UPDATE bugs SET bugs.keywords = CONCAT(bugs.keywords, ', Report:DevProcess')
WHERE (bugs.bug_status = 'VERIFIED') AND (bugs.status_whiteboard LIKE '%Leiden%') and (bugs.keywords LIKE '%Region:Europe%') AND NOT (bugs.keywords LIKE '%Report:DevProcess%')

SQLAlchemy MySQL IF Statement

I'm in the middle of converting an old legacy PHP system to Flask + SQLAlchemy and was wondering how I would construct the following:
I have a model:
class Invoice(db.Model):
paidtodate = db.Column(DECIMAL(10,2))
fullinvoiceamount = db.Column(DECIMAL(10,2))
invoiceamount = db.Column(DECIMAL(10,2))
invoicetype = db.Column(db.String(10))
acis_cost = db.Column(DECIMAL(10,2))
The query I need to run is:
SELECT COUNT(*) AS the_count, sum(if(paidtodate>0,paidtodate,if(invoicetype='CPCN' or invoicetype='CPON' or invoicetype='CBCN' or invoicetype='CBON' or invoicetype='CPUB' or invoicetype='CPGU' or invoicetype='CPSO',invoiceamount,
fullinvoiceamount))) AS amount,
SUM(acis_cost) AS cost, (SUM(if(paidtodate>0,paidtodate,invoiceamount))-SUM(acis_cost)) AS profit FROM tblclientinvoices
Is there an SQLAlchemyish way to construct this query? - I've tried googling for Mysql IF statments with SQlAlchemy but drew blanks.
Many thanks!
Use func(documentation) to generate SQL function expression:
qry = select([
func.count().label("the_count"),
func.sum(func.IF(
Invoice.paidtodate>0,
Invoice.paidtodate,
# #note: I prefer using IN instead of multiple OR statements
func.IF(Invoice.invoicetype.in_(
("CPCN", "CPON", "CBCN", "CBON", "CPUB", "CPGU", "CPSO",)
),
Invoice.invoiceamount,
Invoice.fullinvoiceamount)
)
).label("amount"),
func.sum(Invoice.acis_cost).label("Cost"),
(func.sum(func.IF(
Invoice.paidtodate>0,
Invoice.paidtodate,
Invoice.invoiceamount
))
- func.sum(Invoice.acis_cost)
).label("Profit"),
],
)
rows = session.query(qry).all()
for row in rows:
print row

SQL fails in PHP but works in command line mysql

SQL Code:
SELECT id, album_date AS timestamp, CONCAT((SELECT detail_value
FROM people_db.user_details_tbl WHERE detail_field = 'first_name' AND user_id = pictures_db.albums.owner), ' uploaded pictures!') AS title_html
FROM pictures_db.albums
WHERE id IN
(SELECT DISTINCT(album_id)
FROM pictures_db.album_pics
WHERE pic_id IN
(SELECT DISTINCT(picture_id)
FROM pictures_db.picture_access_tbl
WHERE grantee_group_id IN
(SELECT group_id
FROM people_db.group_membership_tbl
WHERE member_id = '2'
)
)
);
PHP Code:
$albums_sql = mysql_query("SELECT id, album_date AS timestamp, CONCAT((SELECT detail_value
FROM people_db.user_details_tbl
WHERE detail_field = 'first_name' AND user_id = pictures_db.albums.owner), ' uploaded pictures!') AS title_html
FROM pictures_db.albums
WHERE id IN (
SELECT DISTINCT(album_id)
FROM pictures_db.album_pics
WHERE pic_id IN (
SELECT DISTINCT(picture_id)
FROM pictures_db.picture_access_tbl
WHERE grantee_group_id IN (
SELECT group_id
FROM people_db.group_membership_tbl
WHERE member_id = '2'
)
)
)") or die(mysql_error());
When the PHP is run, the error is: Table 'pictures_db.albums' doesn't exist
I tried executing as the same user, regranted all permissions, and even flushed privileges. Works in shell, not in PHP.
Any ideas?
The error message is quite clear: MySQL sees the database pictures_db but not the table albums.
That could be due to permissions, but you seem to have checked that thoroughly.
Another possible reason is that the connection string you're using in PHP points to a different database instance than the one you're using at the command line. Perhaps the connection string still points to a different environment, such as DEV but you're in QA or points to an old test version of the database?
Do you call mysql_select_db() before running the query?
mysql_select_db('pictures_db');

Django ORM Creates Phantom Alias in SQL Join

I am executing the following code (names changed to protect the innocent, so the model structure might seem weird):
memberships =
models.Membership.objects.filter(
degree__gt=0.0,
group=request.user.get_profile().group
)
exclude_count =
memberships.filter(
member__officerships__profile=request.user.get_profile()
).count()
if exclude_officers_with_profile:
memberships = memberships.exclude(
member__officerships__profile=request.user.get_profile()
)
total_count = memberships.count()
which at this point results in:
OperationalError at /
(1054, "Unknown column 'U1.id' in 'on clause'")
The SQL generated is:
SELECT
COUNT(*)
FROM
`membership`
WHERE (
`membership`.`group_id` = 2 AND
`membership`.`level` > 0.0 AND
NOT (
`membership`.`member_id`
IN (
SELECT
U2.`member_id`
FROM
`membership` U0 INNER JOIN `officers` U2
ON (U1.`id` = U2.`member_id`)
WHERE U2.`profile_id` = 2
)
)
)
It appears that the SQL Join's ON statement is referencing an alias that hasn't been defined. Any ideas why!? I dropped my MySQL database and re-synced the tables from my models to try and ensure that there weren't any inconsistencies there.
The structure of the models I'm using are:
class Membership(models.Model):
member = models.ForeignKey(Member, related_name='memberships')
group = models.ForeignKey(Group, related_name='memberships')
level = models.FloatField(default=0.0)
class Member(models.Model):
name = models.CharField(max_length=32)
class Officer(models.Model):
member = models.ForeignKey(Member, related_name='officerships')
profile = models.ForeignKey(UserProfile)
class UserProfile(models.Model)
group = models.ForeignKey(Group)
class Group(models.Model)
pass
I think this may be a symptom of:
http://code.djangoproject.com/ticket/9188
which was fixed as of django revision 9589, I think. Now how to figure out which revision I'm working from...
Confirmed. When I implemented the change referenced in the ticket above:
http://code.djangoproject.com/changeset/9589
my error went away.