Sum of averages raw query - mysql

Question

I have the following code that I have to optimize:
These are the models:
class Question(models.Model):
question_id = models.CharField(max_length=20)
label = models.CharField(max_length=255, verbose_name='Question')
class Property(models.Model):
name = models.CharField(max_length=200)
class Response(models.Model):
question = models.ForeignKey(Question, on_delete=models.CASCADE)
submit_date = models.DateTimeField()
score = models.IntegerField(null=True, blank=True)
is_null = models.BooleanField(default=False)
ignore = models.BooleanField(default=False)
property = models.ForeignKey(Property, on_delete=models.CASCADE)
class Plan(models.Model):
name = models.CharField(max_length=100)
questions = models.ManyToManyField(Question, through='PlanQuestion')
start_date = models.DateField(null=True)
completion_date = models.DateField(null=True)
class PlanQuestion(models.Model):
question = models.ForeignKey(Question, on_delete=models.CASCADE)
plan = models.ForeignKey(Plan, on_delete=models.CASCADE)
I first iterate over the plans then plan questions like this:
plans = Plan.objects.filter(
start_date__isnull=False, completion_date__isnull=False
)
for plan in plans:
plan_questions = plan.questions.through.objects.filter(plan=plan)
for plan_question in plan_questions:
# run the below query for each plan_question here
In the above code for each plan question this query is run to calculate the average of score:
SELECT AVG(score) AS average_score
FROM Response WHERE question_id=%(question_id)s
AND DATE(submit_date) >= %(stard_date)s AND DATE(submit_date) <= %(end_date)s
The problem is that:
If let us say Plan1 has 5 questions:
P1 => Avg(Q1) + Avg(Q2) + Avg(Q3) + Avg(Q4) + Avg(Q5)
The query is run for each question which calculates the average score for each response (one question can have many responses) so for P1, 5 queries are run, and let us say it takes 0.5 seconds to execute one query then it would take 2.5 seconds (5 * 0.5) to run 5 queries for one plan. Now If we increase the number of Plans each having 5 questions then it would increase the time exponentially.
I want a way to reduce the number of these queries so that for each question I don't have to run queries separately. How to combine all the queries of question in one query ?. Maybe I can use union but I don't get how would I write a single query using that or maybe there might be a better solution than a union.
I also tried to add prefech_related but that did no improvement.
Edit:
Create Tables:
CREATE TABLE `Response` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`question_id` int(11) NOT NULL,
`score` int(11) DEFAULT NULL
PRIMARY KEY (`id`),
KEY `Response_25110688` (`question_id`),
CONSTRAINT `question_id_refs_id_2dd82bdb` FOREIGN KEY (`question_id`) REFERENCES `Question` (`id`),
) ENGINE=InnoDB AUTO_INCREMENT=157533450 DEFAULT CHARSET=latin1
CREATE TABLE `Question` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`question_id` varchar(20) NOT NULL,
`label` varchar(255) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=353 DEFAULT CHARSET=latin1
CREATE TABLE `Plan` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`start_date` date DEFAULT NULL,
`completion_date` date DEFAULT NULL
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=687 DEFAULT CHARSET=latin1
CREATE TABLE `PlanQuestion` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`plan_id` int(11) NOT NULL,
`question_id` int(11) NOT NULL,
PRIMARY KEY (`id`),
KEY `PlanQuestion_plan_id_de8df699_fk_Plan_id` (`plan_id`),
KEY `PlanQuestion_question_id_49c10d5b_fk_Question_id` (`question_id`),
CONSTRAINT `PlanQuestion_plan_id_de8df699_fk_Plan_id` FOREIGN KEY (`plan_id`) REFERENCES `Plan` (`id`),
CONSTRAINT `PlanQuestion_question_id_49c10d5b_fk_Question_id` FOREIGN KEY (`question_id`) REFERENCES `Question` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=2130 DEFAULT CHARSET=latin1
CREATE TABLE `Property` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(200) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=188651 DEFAULT CHARSET=latin1
Here is the full query:
SELECT id, COUNT(*) AS count, AVG(int_val) AS int_average
FROM Response WHERE question_id=%(question_id)s
AND property_id=%(property_id)s and is_null=0
AND Response.ignore=0 AND DATE(submit_date) >= %(stard_date)s
AND DATE(submit_date) <= %(end_date)s

Answer 1

This does not make a lot of sense:
SELECT id, COUNT(*) AS count, AVG(int_val) AS int_average
FROM Response
WHERE question_id=%(question_id)s
AND DATE(submit_date) >= %(stard_date)s
AND DATE(submit_date) <= %(end_date)s
Without a GROUP BY, the COUNT and AVG will be totals for the one "question_id". But then if there is a different id for each row, which id are you hoping for?
OK, assuming id is removed, it needs this composite index with the columns in this order:
INDEX(question_id, submit_date)
Meanwhile, remove INDEX(question_id) because it will be in the way.
Sorry, but sometimes performance requires changes.
Secondly... "for plan_question in plan_questions" implies that you want that to be run for every "question"?
Then get rid of the loop and do all the work at the same time:
SELECT question_id, COUNT(*) AS count, AVG(int_val) AS int_average
FROM Response
WHERE DATE(submit_date) >= %(start_date)s
AND DATE(submit_date) <= %(end_date)s
GROUP BY question_id
This will return one row per question; then you can loop through the resultset to deliver the output.
Good news: Even if you don't add the above index, this will work better than what you have now.
Also... cur_date = datetime.now().date() could be removed from the app code; instead, use simply CURDATE() in SQL to get just the date or NOW() to get the date+time.
Indexing Getting rid of "for plan_question in plan_questions" will be the biggest benefit. The query (as I wrote it) already benefits from the index on question_id. However, adding INDEX(submit_date) might run faster if the date range is narrow.
If there are other clauses in the WHERE, we need to see them. There may be other indexes to suggest.
More
SELECT id, COUNT(*) AS count
FROM response
-- (and not JOINing to any other tables)
GROUP BY id;
This query always has a count of 1 because each id occurs in response exactly once.
SELECT
-- (without id)
COUNT(*) AS count
FROM response
-- (and not JOINing to any other tables)
-- (without GROUP BY)
;
This query always returns exactly 1 row.
Still More
Based on
WHERE question_id=%(question_id)s
AND property_id=%(property_id)s and is_null=0
AND Response.ignore=0 AND DATE(submit_date)...
you need
INDEX(question_id, property_id, is_null, ignore)
and drop INDEX(question_id).
But... My statement about doing a single query instead of an app loop still stands.
JOINing to Plan
SELECT r.question_id,
COUNT(*) AS count,
AVG(r.int_val) AS int_average,
p.plan -- perhaps you want to say which "plan" is involved?
FROM Plans AS p
JOIN PLanQuestions AS pq ON pq.plan_id = p.plan_id
JOIN Responses AS r ON r.question_id = pq.question_id
WHERE p.... -- optionally filter on which plans to include
AND pq.... -- optionally filter on the other columns in pq
AND r.... -- optionally filter on which responses to include
ORDER BY ... -- optionally sort the results by any column(s) in any table(s)
And remove the two single-column indexes in PlanQuestions, replacing them by two 2-column indexes:
INDEX(plan_id, question_id),
INDEX(question_id, plan_id)
AND DATE(submit_date) <= %(end_date)s
GROUP BY question_id
Sargable
DATE(submit_date) >= "..." is "not sargable" This means that an index involving col cannot help with the test. Since submit_date is of datatype DATE, this is semantically identical and faster:
submit_date >= "..."