Here is the code from django docs that explains the use of managers.
class PollManager(models.Manager):
def with_counts(self):
from django.db import connection
cursor = connection.cursor()
cursor.execute("""
SELECT p.id, p.question, p.poll_date, COUNT(*)
FROM polls_opinionpoll p, polls_response r
WHERE p.id = r.poll_id
GROUP BY p.id, p.question, p.poll_date
ORDER BY p.poll_date DESC""")
result_list = []
for row in cursor.fetchall():
p = self.model(id=row[0], question=row[1], poll_date=row[2])
p.num_responses = row[3]
result_list.append(p)
return result_list
class OpinionPoll(models.Model):
question = models.CharField(max_length=200)
poll_date = models.DateField()
objects = PollManager()
class Response(models.Model):
poll = models.ForeignKey(OpinionPoll)
person_name = models.CharField(max_length=50)
response = models.TextField()
I have two questions based on this code:
1) where is r.poll_id coming from? I understand Response has foreignKey relationship to OpinionPoll. In order to JOIN OpinionPoll table with Response table, I need to join on their id.
HOwever to access the poll id in Response, I would do r.poll.id.
Is the syntax, r.poll_id, a MySQL syntax.
why GROUP BY p.id, p.question, p.poll_date? why GROUP BY p.id alone is not sufficient?
2) Is it possible to turn the above raw SQL query into a django ORM query?If so how would that look like?
I am not a SQL guy. so bear with me, if this sounds stupid
EDIT:
If I want to create OpinionPoll and Response tables outside of Django, how will SQL statment for create look like?
In the Django shell, when I run
python manage.py sqlall appname
I get the following:
BEGIN;
CREATE TABLE "myapp_opinionpoll" (
"id" integer NOT NULL PRIMARY KEY,
"question" varchar(200) NOT NULL,
"poll_date" date NOT NULL
)
;
CREATE TABLE "myapp_response" (
"id" integer NOT NULL PRIMARY KEY,
"poll_id" integer NOT NULL REFERENCES "myapp_opinionpoll" ("id"),
"person_name" varchar(50) NOT NULL,
"response" text NOT NULL
)
;
CREATE INDEX "larb_response_70f78e6b" ON "myapp_response" ("poll_id");
COMMIT;
I see something like REFERENCES "myapp_opinionpoll" and CREATE INDEXabove. I am not sure
if this is how in SQL it is done?
[1] Django model will create foreign keys like fieldname_id as the field in mysql. So you see the field poll = models.ForeignKey(OpinionPoll) creates this field.
About GROUP BY, because these fields are exactly what selected, except for the aggregate function, grouping them exactly can make them distinct.
[2] Try this, I didn't debug, but may helps:
from django.db.models import Count
OpinionPoll.objects.annotate(num_responses=Count('response'))
For more about aggregation, see the docs: https://docs.djangoproject.com/en/1.6/topics/db/aggregation/
Related
I am working with a legacy MySQL database in Kotlin using the Exposed library. I've got it working fine with MySQL (simple queries work as expected). I'm running MySQL version 5.7.26, but I don't think the problem is related to MySQL itself.
I have two tables, events and event_years. The relevant columns in events are the id (int, primary key), and name (varchar 255). Event_years contains an id (int, primary key), year (datetime), and event_id (int, foreign key), among other things that are not relevant to this question.
An event may have zero or more rows in event_years referring to it. I would like to select the event name, and year, and order the results by year.
I was able to achieve this using the mysql CLI like this:
SELECT e.id, e.name, y.id, y.date
FROM events e, event_years y
WHERE e.id = y.event_id
ORDER BY y.date;
In Kotlin, I have created objects for Events and EventYears:
object Events : Table("events") {
val id = integer("id").autoIncrement().primaryKey()
val name = varchar("name", length = 255)
}
object EventYears : Table("event_years") {
val id = integer("id").autoIncrement().primaryKey()
val eventId = integer("event_id").uniqueIndex()
val date = date("date")
}
I have then tried the following query:
val res = EventYears.innerJoin(Events)
.slice(Events.name, Events.id, EventYears.date, EventYears.id)
.select { EventYears.eventId eq Events.id }
.groupBy(EventYears.date)
I expected the result to be an Iterable object containing these values (like the ones I've received when doing queries without joining), but an exception was raised:
java.lang.IllegalStateException: Cannot join with foo.bar.Events#b307e119 as there is no matching primary key/ foreign key pair and constraint missing
Thank you in advance for any help. I've read the Exposed documentation twice now and still can not understand why this is not working.
You should define your eventId as val eventId = reference("film", Events).uniqueIndex() then you'll be possible to use your query without an exception.
The other way is to provide both columns explicitly in innerJoin
val res = EventYears.innerJoin(Events, {EventYears.eventId}, {Events.id})
.slice(Events.name, Events.id, EventYears.date, EventYears.id)
.selectAll()
.groupBy(EventYears.date)
Try specifying the constraint :
val joinResult = Events.join(EventYears, JoinType.INNER, additionalConstraint = {
EventYears.eventId eq Events.id
Then select as follows
joinResult.select({ EventYears.eventId eq Events.id }) {
val eventId = it[Events.id]
val eventYearsId = it[EventYears.eventId]
}
I'm trying to avoid using extra() here, but haven't found a way to get the results I want using Django's other queryset methods.
My models relationships are as follows:
Model: Enrollment
FK to Course
FK to User
FK to Mentor (can be NULL)
Model: Course
FK to CourseType
In a single query: given a User, I'm trying to get all of the CourseTypes they have access to. A User has access to a CourseType if they have an Enrollment with both a Course of that CourseType AND an existing Mentor.
This user has 2 Enrollments: one in a Course for CourseType ID 6, and the other for a Course for CourseType ID 7, but her enrollment for CourseType ID 7 does not have a mentor, so she does not have access to CourseType ID 7.
user = User.objects.get(pk=123)
This works fine: Get all of the CourseTypes that the user has enrollments for, but don't (yet) query for the mentor requirement:
In [28]: CourseType.objects.filter(course__enrollment__user=user).values('pk')
Out[28]: [{'pk': 6L}, {'pk': 7L}]
This does not give me the result I want: Excluding enrollments with NULL mentor values. I want it to return only ID 6 since that is the only enrollment with a mentor, but it returns an empty queryset:
In [29]: CourseType.objects.filter(course__enrollment__user=user).exclude(course__enrollment__mentor=None).values('pk')
Out[29]: []
Here's the generated SQL for the last queryset that isn't returning what I want it to:
SELECT `courses_coursetype`.`id` FROM `courses_coursetype` INNER JOIN `courses_course` ON ( `courses_coursetype`.`id` = `courses_course`.`course_type_id` ) INNER JOIN `store_enrollment` ON ( `courses_course`.`id` = `store_enrollment`.`course_id` ) WHERE (`store_enrollment`.`user_id` = 3877 AND NOT (`courses_coursetype`.`id` IN (SELECT U0.`id` AS `id` FROM `courses_coursetype` U0 LEFT OUTER JOIN `courses_course` U1 ON ( U0.`id` = U1.`course_type_id` ) LEFT OUTER JOIN `store_enrollment` U2 ON ( U1.`id` = U2.`course_id` ) WHERE U2.`mentor_id` IS NULL)))
The problem, it seems, is that in implementing the exclude(), Django is creating a subquery which is excluding more rows than I want excluded.
To get the desired results, I had to use extra() to explicitly exclude NULL Mentor values in the WHERE clause:
In [36]: CourseType.objects.filter(course__enrollment__user=user).extra(where=['store_enrollment.mentor_id IS NOT NULL']).values('pk')
Out[36]: [{'pk': 6L}]
Is there a way to get this result without using extra()? If not, should I file a ticket with Django per the docs? I looked at the existing tickets and searched for this issue but unfortunately came up short.
I'm using Django 1.7.10 with MySQL.
Thanks!
Try using isnull.
CourseType.objects.filter(
course__enrollment__user=user,
course__enrollment__mentor__isnull=False,
).values('pk')
Instead of exclude() you can create complex queries using Q(), or in your case ~Q():
filter_q = Q(course__enrollment__user=user) | ~Q(course__enrollment__mentor=None)
CourseType.objects.filter(filter_q).values('pk')
This might lead to a different SQL statement.
See docs:
https://docs.djangoproject.com/en/3.2/topics/db/queries/#complex-lookups-with-q-objects
We have a table of data that looks like:
CREATE TABLE objects (
id BIGSERIAL NOT NULL,
typeid BIGINT NOT NULL,
fullobject JSON,
PRIMARY KEY (id),
...
);
and
CREATE TABLE types (
id BIGSERIAL NOT NULL,
type VARCHAR(255),
PRIMARY KEY (id),
...
);
in the objects.fullobject column is JSON data for users and orgs like:
// id: 1
{
...
"type":"1", // user
"orgs":[
{"id":"2", "position":"foo"},
{"id":"2", "position":"bar"},
{"id":"3", "position":"foo"},
]
}
// id: 2
{
...
"type":"2", // org
"name":"Org 1"
}
// id: 3
{
...
"type":"2", // org
"name":"Org 2"
}
The question is, if I want to find a user based on the name of the organisation, how do I do the join?
I'm not sure if I can create the right index on the org data inside a user.
The two solutions I can think of are:
Create a single text attribute that contains the id's of the orgs for a user, then do a join based of that (e.g. "searchableOrg": "|org1|org2|").
Then the query looks like:
SELECT * from objects user
INNER JOIN types usertype ON usertype.id = user.typeid
INNER JOIN objects org ON json_extract_path(user.fullobject, 'searchableOrg') LIKE '%|' || org.id || '|%'
INNER JOIN types usertype ON usertype.id = user.typeid
WHERE json_extract_path(org.fullobject, 'name') LIKE '%whatever%'
AND usertype.type = 'user'
AND orgtype.type = 'org'
Here I can have indexes on json_extract_path(user.fullobject, 'searchableOrg') and json_extract_path(org.fullobject, 'name').
However this doesn't work if we add things like a start or end date to the user's organisation membership and need to further filter the join on that.
Create a trigger that when a user is created/update/deleted modifies a table (userorgtable) that contains the user org membership and do the join off of that.
I haven't looked at triggers yet, but hopefully the query would be something like:
SELECT * from objects user
INNER JOIN userorgtable userorg ON user.id = userorg.userid
INNER JOIN types usertype ON usertype.id = user.typeid
INNER JOIN objects org ON userorg.orgid = org.id
INNER JOIN types orgtype ON orgtype.id = org.typeid
WHERE json_extract_path(org.fullobject, 'name') LIKE '%whatever%'
AND usertype.type = 'user'
AND orgtype.type = 'org'
In this case, if we needed to add further attributes to the join (such as start and end dates), we can just put them in the userorgtable and have add conditions like:
AND userorg.startdate < CURRENT_DATE()
Is there another option? Perhaps using more of the Postgres JSON functions?
Thanks in advance
Say I have a simple blog entry model in Django:
class Entry(models.Model):
author = models.ForeignKey(Author)
topic = models.ForeignKey(Topic)
entry = models.CharField(max_length=50, default='')
Now say I want to query for a author or topic, but exclude a particular topic altogether.
entry_list = Entry.objects.filter(Q(author=12)|Q(topic=123)).exclude(topic=666)
Sinmple enough, but I've found that this raw SQL contains a join on the topic table, even though it doesn't have to be used:
SELECT `blog_entry`.`id`
FROM `blog_entry`
LEFT OUTER JOIN `blog_topic`
ON (`blog_entry`.`topic_id` = `blog_topic`.`id`)
WHERE ((`blog_entry`.`author_id` = 12
OR `blog_entry`.`topic_id` = 123
)
AND NOT ((`blog_topic`.`id` = 666
AND NOT (`blog_topic`.`id` IS NULL)
AND `blog_topic`.`id` IS NOT NULL
))
)
Why is that? How can I get Django to query only the column ids and not join tables? I've tried the following but it give a FieldError:
entry_list = Entry.objects.filter(Q(author_id=12)|Q(topic_id=123)).exclude(topic_id=666)
i wonder whether this is a bug.
trying a similar example, i get no join when putting the exclude before the filter (but i do get it using your order)
I am very frustrated from linq to sql when dealing with many to many relationship with the skip extension. It doesn't allow me to use joinned queries. Not sure it is the case for SQL server 2005 but I am currently using SQL Server 2000.
Now I consider to write a store procedure to fetch a table that is matched by two tables e.g. Album_Photo (Album->Album_Photo<-Photo) and Photo table and only want the Photos data so I match the Album's ID with Album_Photo and use that ID to match the photo. In the store procedure I am just fetch all the joinned data. After that in the linq to sql, I create a new Album object.
e.g.
var albums = (from r in result
where (modifier_id == r.ModifierID || user_id == r.UserID)
select new Album() {
Name = r.Name,
UserID = r.UserID,
ModifierID = r.ModifierID,
ID = r.ID,
DateCreated = r.DateCreated,
Description = r.Description,
Filename = r.Filename
}).AsQueryable();
I used the AsQueryable to get the result as a IQueryable rather than IEnumerable. Later I want to do something with the collection, it gives me this error:
System.InvalidOperationException: The query results cannot be enumerated more than once.
It sounds like you have a situation where the query has already executed by the time you are want to filter it later in your code.
Can you do something like...
var albums = (blah blah blah).AsQueryable().Where(filterClause) when you have enough info to process
what happens if you try albums.where(filter) later on in the code? Is this what you are trying?