Django querysets: Excluding NULL values across multiple joins - mysql

I'm trying to avoid using extra() here, but haven't found a way to get the results I want using Django's other queryset methods.
My models relationships are as follows:
Model: Enrollment
FK to Course
FK to User
FK to Mentor (can be NULL)
Model: Course
FK to CourseType
In a single query: given a User, I'm trying to get all of the CourseTypes they have access to. A User has access to a CourseType if they have an Enrollment with both a Course of that CourseType AND an existing Mentor.
This user has 2 Enrollments: one in a Course for CourseType ID 6, and the other for a Course for CourseType ID 7, but her enrollment for CourseType ID 7 does not have a mentor, so she does not have access to CourseType ID 7.
user = User.objects.get(pk=123)
This works fine: Get all of the CourseTypes that the user has enrollments for, but don't (yet) query for the mentor requirement:
In [28]: CourseType.objects.filter(course__enrollment__user=user).values('pk')
Out[28]: [{'pk': 6L}, {'pk': 7L}]
This does not give me the result I want: Excluding enrollments with NULL mentor values. I want it to return only ID 6 since that is the only enrollment with a mentor, but it returns an empty queryset:
In [29]: CourseType.objects.filter(course__enrollment__user=user).exclude(course__enrollment__mentor=None).values('pk')
Out[29]: []
Here's the generated SQL for the last queryset that isn't returning what I want it to:
SELECT `courses_coursetype`.`id` FROM `courses_coursetype` INNER JOIN `courses_course` ON ( `courses_coursetype`.`id` = `courses_course`.`course_type_id` ) INNER JOIN `store_enrollment` ON ( `courses_course`.`id` = `store_enrollment`.`course_id` ) WHERE (`store_enrollment`.`user_id` = 3877 AND NOT (`courses_coursetype`.`id` IN (SELECT U0.`id` AS `id` FROM `courses_coursetype` U0 LEFT OUTER JOIN `courses_course` U1 ON ( U0.`id` = U1.`course_type_id` ) LEFT OUTER JOIN `store_enrollment` U2 ON ( U1.`id` = U2.`course_id` ) WHERE U2.`mentor_id` IS NULL)))
The problem, it seems, is that in implementing the exclude(), Django is creating a subquery which is excluding more rows than I want excluded.
To get the desired results, I had to use extra() to explicitly exclude NULL Mentor values in the WHERE clause:
In [36]: CourseType.objects.filter(course__enrollment__user=user).extra(where=['store_enrollment.mentor_id IS NOT NULL']).values('pk')
Out[36]: [{'pk': 6L}]
Is there a way to get this result without using extra()? If not, should I file a ticket with Django per the docs? I looked at the existing tickets and searched for this issue but unfortunately came up short.
I'm using Django 1.7.10 with MySQL.
Thanks!

Try using isnull.
CourseType.objects.filter(
course__enrollment__user=user,
course__enrollment__mentor__isnull=False,
).values('pk')

Instead of exclude() you can create complex queries using Q(), or in your case ~Q():
filter_q = Q(course__enrollment__user=user) | ~Q(course__enrollment__mentor=None)
CourseType.objects.filter(filter_q).values('pk')
This might lead to a different SQL statement.
See docs:
https://docs.djangoproject.com/en/3.2/topics/db/queries/#complex-lookups-with-q-objects

Related

Mysql Query for fetching records using single Query from three tables

We have three table
table 1- app ( id , name )
table 2- appPlayer ( id , name )
table 3- appPlayerSession ( id , appId , appPlayerId , version)
my Current query is:
SELECT (select name from app k where k.id= aps.appId) AS appName,version,appId,count(version) FROM appPlayerSession aps GROUP BY appId,version,appName
we need to count the session users for each game with same version, and also woth the object of all users data using single mysql query.
Current Result using my query, but we also need players for each app..
As you havent given your expected result and on basis of your requirement you can do something this.it may be enhanced as per your requirement.
SELECT (select name from app k where k.id= aps.appId) AS appName,version,appId,(select P.name from appPlayer P where P.id=aps.appPlayerid) as appPlayerName, count(version) FROM appPlayerSession aps GROUP BY appId,version,appName,appPlayerName
Also check fiddle as per your requirement created as you havent given any data set and its on my assumption.
http://sqlfiddle.com/#!9/30fe4f/1
New Sql as per your new added requirement-
select X.appname,X.version,X.appid,GROUP_CONCAT(distinct X.appPlayerName order by X.appPlayerName) as Users ,
sum(X.vercount)
from (SELECT (select name from app k where k.id= aps.appId)
AS appName,version,appId,
(select P.name from appPlayer P where P.id=aps.appPlayerid)
as appPlayerName, count(version)as vercount
FROM appPlayerSession aps
GROUP BY appId,version,appName,appPlayerName) X
group by X.appname,X.version,X.appid
New fiddle -http://sqlfiddle.com/#!9/13646c/5
You can use JOIN in sql to connect with multiple tables and fetch result
Below is the format :
SELECT t1.col, 
       t3.col 
FROM   table1 
       JOIN table2 
         ON table1.primarykey = table2.foreignkey 
       JOIN table3 
         ON table2.primarykey = table3.foreignkey 
In your case :
SELECT app.col, 
       appPlayer.col,
appPlayerSession.col 
FROM   app 
       JOIN appPlayer 
         ON app.id = appPlayer.appId
       JOIN appPlayerSession 
         ON appPlayer.id = appPlayerSession.appPlayerId
Hope this is helpful.
One suggestion . It is not a standard to use camelCase for table and column names. snake_case is preferred widely.

How to create update statement using results of sub-query where tables have same column name?

working on resolving issue where we need to update several thousand rows where there is an admin entity that isnt verified yet but there is a matching admin entity name in the verified list.
i have script written to pull the rows from that joins together the 2 tables. this gives me the list of rows that need update.
SELECT
pi.id AS `Primary Key for PI`,
pi.admin_entity_id,
pi.admin_entity_unverified,
asa.id AS `Admin Setup Accounts ID`,
asa.name AS `Admin Entity-Verified`
FROM
p_information pi
LEFT JOIN admin_setup_accounts asa ON asa.name =
pi.admin_entity_unverified
WHERE
pi.admin_entity_unverified IS NOT NULL
AND pi.admin_entity_unverified <> ''
AND pi.admin_entity_id IS NULL
AND pi.admin_entity_unverified IN (select name FROM
admin_setup_accounts)
http://sqlfiddle.com/#!9/a8c11a/3/0
This query returns list of unverified entries (pi.admin_entity_unverified) that match a verified list (asa.name) and the ID (asa.id) for each of those verified names.
i now need to do an update of all these matching rows and update the pi tables admin_entity_id with the corresponding admin setup accounts id (asa.id), so im thinking of doing like
update p_information pi
inner join (
SELECT
pi.id AS `Primary Key for PI`,
pi.admin_entity_id,
pi.admin_entity_unverified,
asa.id AS `Admin_Setup_ Accounts_ID`,
asa.name AS `Admin Auditing Entity-Verified`
FROM
p_information pi
LEFT JOIN admin_setup_accounts asa ON asa.name =
pi.admin_auditing_entity_unverified
WHERE
pi.admin_auditing_entity_unverified IS NOT NULL
AND pi.admin_auditing_entity_unverified <> ''
AND pi.admin_auditing_entity_id IS NULL
AND pi.admin_auditing_entity_unverified IN (select name FROM
admin_setup_accounts)) upi
ON upi.id = pi.id
SET pi.admin_entity_id = (this needs to be the asa.id AS `Admin Setup Accounts ID` returned from the subquery (upi)
but since pi and asa both use the id column inside the subquery how can i list those outside the query ? or is there another way to get this update done using the query? let me know
thanks
well i think i figured it out, i was thinking i couldnt use the column aliases outside the sub query, but it appears to work just fine. i do the set command as set pi.admin_entity_id = upi.Admin_Setup_ Accounts_ID and it updates correctly . :) oh well guess shouldnt overthink it

Graded Assignment [Zaption] Database

Most specifically, I'm having trouble returning from LibreOffice Base [HSQLdb] a list of grades organized by (1) class, (2) assignment, (3) student's last name.
I want this output so I can run a script to copy the grades from the database to an online gradebook (which doesn't have an API [sadface])
I suspect several possible causes for this problem:
My relational structure may need tweaking.
I somehow need to implement a "student ID." On Zaption, Students make their submissions under whatever "ZaptionName" they choose to use. I then manually match ZaptionName to RosterFullName in the second table shown.
Zaption allows multiple submissions by the same "student" for the same assignment. Because multiple submissions are allowed, I run a FilterLowMultiples query to select the highest grade for that assignment for that student.
FilterLowMultiples:
SELECT MAX( "Grade" ) "Grade", "RosterFullName",
"Assignment", MAX( "ZaptionName" ) "ZapName"
FROM "SelectAssignment"
GROUP BY "RosterFullName", "Assignment"
SelectAssignment is below for reference:
SELECT "GradedAssignments"."Assignment", "Roster"."RosterFullName",
"GradedAssignments"."Grade", "ZaptionNames"."ZaptionName"
FROM "Roster", "ClassIndex", "GradedAssignments", "ZaptionNames"
WHERE "Roster"."Class" = "ClassIndex"."Class"
AND "GradedAssignments"."ZaptionName" = "ZaptionNames"."ZaptionName"
AND "ZaptionNames"."RosterFullName" = "Roster"."RosterFullName"
AND ( "GradedAssignments"."Assignment" = 'YouKnowWhatever')
My query to PullAssignmentGrades is as follows, but sorting by assignment fails, as there is no assignment by default unless that student submitted one, so the row is blank and that student falls to the bottom of the sort, which is bad for the transfer-to-online script I run.
SELECT "Roster"."RosterFirstName", "ClassIndex"."Class",
"Roster"."RosterFullName", "ClassIndex"."ClassLevel",
"FilterLowMultiples"."Grade", "FilterLowMultiples"."ZapName",
"FilterLowMultiples"."Assignment", "FilterLowMultiples"."Grade",
"FilterLowMultiples"."Assignment", "ClassIndex"."ClassDisplayOrder",
"Roster"."RosterLastName"
FROM "ClassIndex", "FilterLowMultiples", "Roster"
ORDER BY "Roster"."RosterFirstName" ASC,
"FilterLowMultiples"."Grade" DESC,
"FilterLowMultiples"."Assignment" ASC,
"ClassIndex"."ClassDisplayOrder" ASC,
"Roster"."RosterLastName" ASC
Use a LEFT JOIN in your query for SelectAssignment so you don't drop students who didn't do a particular assignment. Optionally you can use COALESCE on the potentially NULL values from the "GradedAssignments" table to assign a grade of 0 or I. Like so:
SELECT 'YouKnowWhatever' AS "Assignment", "Roster"."RosterFullName",
COALESCE("GradedAssignments"."Grade",0), "ZaptionNames"."ZaptionName"
FROM "Roster"
INNER JOIN "ClassIndex" ON "Roster"."Class" = "ClassIndex"."Class"
INNER JOIN "ZaptionNames" ON "ZaptionNames"."RosterFullName" = "Roster"."RosterFullName"
LEFT JOIN "GradedAssignments" ON ("GradedAssignments"."ZaptionName" = "ZaptionNames"."ZaptionName"
AND "GradedAssignments"."Assignment" = 'YouKnowWhatever')

Sql Result IN a Query

dont blame for the database design.I am not its database architect. I am the one who has to use it in current situation
I hope this will be understandable.
I have 3 tables containing following data with no foreign key relationship b/w them:
groups
groupId groupName
1 Admin
2 Editor
3 Subscriber
preveleges
groupId roles
1 1,2
2 2,3
3 1
roles
roleId roleTitle
1 add
2 edit
Query:
SELECT roles
from groups
LEFT JOIN preveleges ON (groups.groupId=preveleges.groupId)
returns specific result i.e roles.
Problem: I wanted to show roleTitle instead of roles in the above query.
I am confused how to relate table roles with this query and returns required result
I know it is feasible with coding but i want in SQL.Any suggestion will be appreciated.
SELECT g.groupName,
GROUP_CONCAT(r.roleTitle
ORDER BY FIND_IN_SET(r.roleId, p.roles))
AS RoleTitles
FROM groups AS g
LEFT JOIN preveleges AS p
ON g.groupId = p.groupId
LEFT JOIN roles AS r
ON FIND_IN_SET(r.roleId, p.roles)
GROUP BY g.groupName ;
Tested at: SQL-FIDDLE
I would change the data structure it self. Since It's not normalised, there are multiple elements in a single column.
But it is possible with SQL, if for some (valid) reason you can't change the DB.
A simple "static" solution:
SELECT REPLACE(REPLACE(roles, '1', 'add'), '2', 'edit') from groups
LEFT JOIN preveleges ON(groups.groupId=preveleges.groupId)
A more complex but still ugly solution:
CREATE FUNCTION ReplaceRoleIDWithName (#StringIds VARCHAR(50))
RETURNS VARCHAR(50)
AS
BEGIN
DECLARE #RoleNames VARCHAR(50)
SET #RoleNames = #StringIds
SELECT #RoleNames = REPLACE(#RoleNames, CAST(RoleId AS VARCHAR(50)), roleTitle)
FROM roles
RETURN #RoleNames
END
And then use the function in the query
SELECT ReplaceRoleIDWithName(roles) from groups
LEFT JOIN preveleges ON(groups.groupId=preveleges.groupId)
It is possible without function, but this is more readable. Made without editor so it's not tested in anyway.
You also tagged the question with PostgreSQL and it's actually quite easy with Postgres to work around this broken design:
SELECT grp.groupname, r.roletitle
FROM groups grp
join (
select groupid, cast(regexp_split_to_table(roles, ',') as integer) as role_id
from privileges
) as privs on privs.groupid = grp.groupid
join roles r on r.roleid = privs.role_id;
SQLFiddle: http://sqlfiddle.com/#!12/5e87b/1
(Note that I changed the incorrectly spelled name preveleges to the correct spelling privileges)
But you should really, really re-design your data model!
Fixing your design also enables you to define foreign key constraints and validate the input. In your current model, the application would probably break (just as my query would), if someone inserted the value 'one,two,three' into the roles table.
Edit
To complete the picture, using Postgres's array handling the above could be slightly simplified using a similar approach as MySQL's find_in_set()
select grp.groupname, r.roletitle
from groups grp
join privileges privs on grp.groupid = privs.groupid
join roles r on r.roleid::text = any (string_to_array(privs.roles, ','))
In both cases if all role titles should be shown as a comma separated list, the string_agg() function could be used (which is equivalent to MySQL's group_concat()
select grp.groupname, string_agg(r.roletitle, ',')
from groups grp
join privileges privs on grp.groupid = privs.groupid
join roles r on r.roleid::text = any (string_to_array(privs.roles, ','))
group by grp.groupname

Two columns within an ISNULL function in MySql

I have two tables, one is for individuals and the other is for their company.
Each table has a column for locale, this is mandatory for the company, but not for the individual. The idea seems to be that if the individual doesn't set a preference, they are assumed to be in the locale of their company.
I would like to select the locale for the individual, using the company default if the individual locale is null and I thought of doing the following (which I don't think is possible in MySql)...
SELECT
ISNULL(individual.Locale, company.Locale) `Locale`
FROM
individual
INNER JOIN
company ON company.CompanyId = individual.CompanyId
WHERE
individual.IndividualId = 1
Is there a nice way to do this - or am I just going to end up sending both Locale's back and making the decision in the code?
You can use the COALESCE() function which returns the first non-NULL value among its arguments. This function can also be used in most other RDBMS like SQL-Server, Oracle 9, Postgres 8:
SELECT
COALESCE(individual.Locale, company.Locale) AS Locale
FROM
individual
INNER JOIN
company ON company.CompanyId = individual.CompanyId
WHERE
individual.IndividualId = 1
You pretty much had it, what you want is IFNULL()
SELECT
IFNULL(individual.Locale, company.Locale) `Locale`
FROM
individual
INNER JOIN
company ON company.CompanyId = individual.CompanyId
WHERE
individual.IndividualId = 1
Try the CASE...WHEN...THEN statement
SELECT
(CASE
WHEN individual.Locale IS NULL THEN company.Locale
ELSE individual.Locale
END) 'Locale'
FROM
individual
INNER JOIN
company ON company.CompanyId = individual.CompanyId
WHERE
individual.IndividualId = 1