I am using test DB from https://github.com/datacharmer/test_db. It has a moderate size of 160 Mb. To run queries I use MySQL Workbench.
Following code runs in 0.015s
SELECT *
FROM employees INNER JOIN salaries ON employees.emp_no = salaries.emp_no
The similar code with GROUP BY added runs for 15.0s
SELECT AVG(salary), gender
FROM employees INNER JOIN salaries ON employees.emp_no = salaries.emp_no
GROUP BY gender
I checked the execution plan for both queries and found that in both cases query cost is similar and is about 600k. I should add that the employee table has 300K rows and the salary table is about 3M rows.
Can anyone suggest a reason why the difference in the execution time is soo huge? I need this explanation to understand the way SQL works better.
Problem solution: As I found due to comments and answers the problem was related to me not noticing that in the case of the first query my IDE was limiting result to 1000 rows. That's how I got 0.015s. In reality, it takes 10.0s to make a join in my case. If the index for gender is created (indices for employees.emp_no and salaries.emp_no already exist in this DB) it takes 10.0s to make join and group by. Without index for gender second query takes 18.0s.
The EXPLAIN for the first query shows that it does a table-scan (type=ALL) of 300K rows from employees, and for each one, does a partial primary key (type=ref) lookup to 1 row (estimated) in salaries.
mysql> explain SELECT * FROM employees
INNER JOIN salaries ON employees.emp_no = salaries.emp_no;
+----+-------------+-----------+------+---------------+---------+---------+----------------------------+--------+-------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-----------+------+---------------+---------+---------+----------------------------+--------+-------+
| 1 | SIMPLE | employees | ALL | PRIMARY | NULL | NULL | NULL | 299113 | NULL |
| 1 | SIMPLE | salaries | ref | PRIMARY | PRIMARY | 4 | employees.employees.emp_no | 1 | NULL |
+----+-------------+-----------+------+---------------+---------+---------+----------------------------+--------+-------+
The EXPLAIN for the second query (actually a sensible query to compute AVG() as you mentioned in your comment) shows something additional:
mysql> EXPLAIN SELECT employees.gender, AVG(salary) FROM employees
INNER JOIN salaries ON employees.emp_no = salaries.emp_no
GROUP BY employees.gender;
+----+-------------+-----------+------+---------------+---------+---------+----------------------------+--------+---------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-----------+------+---------------+---------+---------+----------------------------+--------+---------------------------------+
| 1 | SIMPLE | employees | ALL | PRIMARY | NULL | NULL | NULL | 299113 | Using temporary; Using filesort |
| 1 | SIMPLE | salaries | ref | PRIMARY | PRIMARY | 4 | employees.employees.emp_no | 1 | NULL |
+----+-------------+-----------+------+---------------+---------+---------+----------------------------+--------+---------------------------------+
See the Using temporary; Using filesort in the Extra field? That means that the query has to build a temp table to accumulate the AVG() results per group. It has to use a temp table because MySQL can't know that it will scan all the rows for each gender together, so it must assume it will need to maintain running totals independently as it scans rows. It doesn't seem like that would be a big problem to track two (in this case) gender totals, but suppose it were postal code or something like that?
Creating a temp table is a pretty expensive operation. It means writing data, not only reading it as the first query does.
If we could make an index that orders by gender, then MySQL's optimizer would know it can scan all those rows with the same gender together. So it can calculate the running total of one gender at a time, then once it's done scanning one gender, calculate the AVG(salary) and then be guaranteed no further rows for that gender will be scanned. Therefore it can skip building a temp table.
This index helps:
mysql> alter table employees add index (gender, emp_no);
Now the EXPLAIN of the same query shows that it will do an index scan (type=index) which visits the same number of entries, but it'll scan in a more helpful order for the calculation of the aggregate AVG().
Same query, but no Using temporary note:
mysql> EXPLAIN SELECT employees.gender, AVG(salary) FROM employees
INNER JOIN salaries ON employees.emp_no = salaries.emp_no
GROUP BY employees.gender;
+----+-------------+-----------+-------+----------------+---------+---------+----------------------------+--------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-----------+-------+----------------+---------+---------+----------------------------+--------+-------------+
| 1 | SIMPLE | employees | index | PRIMARY,gender | gender | 5 | NULL | 299113 | Using index |
| 1 | SIMPLE | salaries | ref | PRIMARY | PRIMARY | 4 | employees.employees.emp_no | 1 | NULL |
+----+-------------+-----------+-------+----------------+---------+---------+----------------------------+--------+-------------+
And executing this query is a lot faster:
+--------+-------------+
| gender | AVG(salary) |
+--------+-------------+
| M | 63838.1769 |
| F | 63769.6032 |
+--------+-------------+
2 rows in set (1.06 sec)
The addition of the GROUP BY clause could easily explain the big performance drop that you are seeing.
From the documentation :
The most general way to satisfy a GROUP BY clause is to scan the whole table and create a new temporary table where all rows from each group are consecutive, and then use this temporary table to discover groups and apply aggregate functions (if any).
The additional cost incurred by the grouping process can be very expensive. Also, grouping happens even if no aggregate function is used.
If you don’t need an aggregate function, don’t group. If you do, ensure that you have a single index that references all grouped columns, as suggested by the documentation :
In some cases, MySQL is able to do much better than that and avoid creation of temporary tables by using index access.
PS : please note that « SELECT * ... GROUP BY »-like statements are not supported since MySQL 5.7.5 (unless you turn off option ONLY_FULL_GROUP_BY)
There is another reason as well as what GMB points out. Basically, you are probably looking at the timing of the first query until it returns the first row. I doubt it is returning all the rows in 0.015 seconds.
The second query with the GROUP BY needs to process all the data to derive the results.
If you added an ORDER BY (which requires processing all the data) to the first query, then you would see a similar performance drop.
I am using MySQL 5.6 on FreeBSD and have just recently switched from using MyISAM tables to InnoDB to gain advances of foreign key constraints and transactions.
After the switch, I discovered that a query on a table with 100,000 rows that was previously taking .003 seconds, was now taking 3.6 seconds. The query looked like this:
SELECT *
-> FROM USERS u
-> JOIN MIGHT_FLOCK mf ON (u.USER_ID = mf.USER_ID)
-> WHERE u.STATUS = 'ACTIVE' AND u.ACCESS_ID >= 8 ORDER BY mf.STREAK DESC LIMIT 0,100
I noticed that if I removed the ORDER BY clause, the execution time dropped back down to .003 seconds, so the problem is obviously in the sorting.
I then discovered that if I added back the ORDER BY but removed indexes on the columns referred to in the query (STATUS and ACCESS_ID), the query execution time would take the normal .003 seconds.
Then I discovered that if I added back the indexes on the STATUS and ACCESS_ID columns, but used IGNORE INDEX (STATUS,ACCESS_ID), the query would still execute in the normal .003 seconds.
Is there something about InnoDB and sorting results when referencing an indexed column in a WHERE clause that I don't understand?
Or am I doing something wrong?
EXPLAIN for the slow query returns the following results:
+----+-------------+-------+--------+--------------------------+---------+---------+---------------------+-------+---------------------------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+--------------------------+---------+---------+---------------------+-------+---------------------------------------------------------------------+
| 1 | SIMPLE | u | ref | PRIMARY,STATUS,ACCESS_ID | STATUS | 2 | const | 53902 | Using index condition; Using where; Using temporary; Using filesort |
| 1 | SIMPLE | mf | eq_ref | PRIMARY | PRIMARY | 4 | PRO_MIGHT.u.USER_ID | 1 | NULL |
+----+-------------+-------+--------+--------------------------+---------+---------+---------------------+-------+---------------------------------------------------------------------+
EXPLAIN for the fast query returns the following results:
+----+-------------+-------+--------+---------------+---------+---------+----------------------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+---------------+---------+---------+----------------------+------+-------------+
| 1 | SIMPLE | mf | index | PRIMARY | STREAK | 2 | NULL | 100 | NULL |
| 1 | SIMPLE | u | eq_ref | PRIMARY | PRIMARY | 4 | PRO_MIGHT.mf.USER_ID | 1 | Using where |
+----+-------------+-------+--------+---------------+---------+---------+----------------------+------+-------------+
Any help would be greatly appreciated.
In the slow case, MySQL is making an assumption that the index on STATUS will greatly limit the number of users it has to sort through. MySQL is wrong. Presumably most of your users are ACTIVE. MySQL is picking up 50k user rows, checking their ACCESS_ID, joining to MIGHT_FLOCK, sorting the results and taking the first 100 (out of 50k).
In the fast case, you have told MySQL it can't use either index on USERS. MySQL is using its next-best index, it is taking the first 100 rows from MIGHT_FLOCK using the STREAK index (which is already sorted), then joining to USERS and picking up the user rows, then checking that your users are ACTIVE and have an ACCESS_ID at or above 8. This is much faster because only 100 rows are read from disk (x2 for the two tables).
I would recommend:
drop the index on STATUS unless you frequently need to retrieve INACTIVE users (not ACTIVE users). This index is not helping you.
Read this question to understand why your sorts are so slow. You can probably tune InnoDB for better sort performance to prevent these kind of problems.
If you have very few users with ACCESS_ID at or above 8 you should see a dramatic improvement already. If not you might have to use STRAIGHT_JOIN in your select clause.
Example below:
SELECT *
FROM MIGHT_FLOCK mf
STRAIGHT_JOIN USERS u ON (u.USER_ID = mf.USER_ID)
WHERE u.STATUS = 'ACTIVE' AND u.ACCESS_ID >= 8 ORDER BY mf.STREAK DESC LIMIT 0,100
STRAIGHT_JOIN forces MySQL to access the MIGHT_FLOCK table before the USERS table based on the order in which you specify those two tables in the query.
To answer the question "Why did the behaviour change" you should start by understanding the statistics that MySQL keeps on each index: http://dev.mysql.com/doc/refman/5.6/en/myisam-index-statistics.html. If statistics are not up to date or if InnoDB is not providing sufficient information to MySQL, the query optimiser can (and does) make stupid decisions about how to join tables.
I'm using Drupal 6 with MySQL version 5.0.95 and at an impasse where one of my queries which displays content based on most recent article date slows down and because of the frequency of being used kills the site performance altogether. The query in question is as below:
SELECT n.nid,
n.title,
ma.field_article_date_format_value,
ma.field_article_summary_value
FROM node n
INNER JOIN content_type_article ma ON n.nid=ma.nid
INNER JOIN term_node tn ON n.nid=tn.nid
WHERE tn.tid= 153
AND n.status=1
ORDER BY ma.field_article_date_format_value DESC
LIMIT 0, 11;
The EXPLAIN of the query shows the below result:
+----+-------------+-------+--------+--------------------------+---------+---------+----------------------+-------+---------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+--------------------------+---------+---------+----------------------+-------+---------------------------------+
| 1 | SIMPLE | tn | ref | PRIMARY,nid | PRIMARY | 4 | const | 19006 | Using temporary; Using filesort |
| 1 | SIMPLE | ma | ref | nid,ix_article_date | nid | 4 | drupal_mm_stg.tn.nid | 1 | |
| 1 | SIMPLE | n | eq_ref | PRIMARY,node_status_type | PRIMARY | 4 | drupal_mm_stg.ma.nid | 1 | Using where |
+----+-------------+-------+--------+--------------------------+---------+---------+----------------------+-------+---------------------------------+
This query seemed relatively simple and straight forward and retrieves articles which belong to a category (term) 153 and are of status 1 (published). But apparently Using temporary table and Using filesort means the query is bound to fail from what I've learnt browsing about it.
Removing field_article_date_format_value from the ORDER BY clause solves the Using temporary; Using filesort reduces the query execution time but is required and cannot be traded off, unfortunately same holds equally true for the site performance.
My hunch is that most of the trouble comes from the term_node table which maps articles to categories and is a many-many relationship table meaning if article X is associated to 5 categories C1....C5 it will have 5 entries in that table, this table is from out-of-the-box drupal.
Dealing with heavy DB content is something new to me and going through some of the similar queries (
When ordering by date desc, "Using temporary" slows down query,
MySQL performance optimization: order by datetime field) I tried to create a composite index for the content_type_article whose datetime field is used in the ORDER BY clause along with another key (nid) in it and tried to FORCE INDEX.
SELECT n.nid, n.title,
ma.field_article_date_format_value,
ma.field_article_summary_value
FROM node n
INNER JOIN content_type_article ma FORCE INDEX (ix_article_date) ON n.nid=ma.nid
INNER JOIN term_node tn ON n.nid=tn.nid
WHERE tn.tid= 153
AND n.status=1
ORDER BY ma.field_article_date_format_value DESC
LIMIT 0, 11;
The result and the following EXPLAIN query did not seem to help much
+----+-------------+-------+--------+--------------------------+-----------------+---------+----------------------+-------+---------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+--------------------------+-----------------+---------+----------------------+-------+---------------------------------+
| 1 | SIMPLE | tn | ref | PRIMARY,nid | PRIMARY | 4 | const | 18748 | Using temporary; Using filesort |
| 1 | SIMPLE | ma | ref | ix_article_date | ix_article_date | 4 | drupal_mm_stg.tn.nid | 1 | |
| 1 | SIMPLE | n | eq_ref | PRIMARY,node_status_type | PRIMARY | 4 | drupal_mm_stg.ma.nid | 1 | Using where |
+----+-------------+-------+--------+--------------------------+-----------------+---------+----------------------+-------+---------------------------------+
The fields n.nid, ca.nid, ma.field_article_date_format_value are all indexed. Querying the DB with Limit 0,11 takes approximately 7-10 seconds with the ORDER BY clause but without it the query barely takes a second. The database engine is MyISAM. Any help on this would be greatly appreciated.
Any answer that could help me in getting this query like a normal one (at the same speed as a query without sort by date) would be great. My attempts with creating a composite query as a combination of nid and field_article_date_format_value and use in the query did not help the cause. I'm open to providing additional info on the problem and any new suggestions.
Taking a look at your query and the explain, it seems like having the n.status=1 in the where clause is making the search very inefficient because you need to return the whole set defined by the joins and then apply the status = 1. Try starting the join from the term_node table that is inmediately filtered by the WHERE and then make the joins adding the status condition immediately. Give it a try and please tell me how it goes.
SELECT n.nid, n.title,
ma.field_article_date_format_value,
ma.field_article_summary_value
FROM term_node tn
INNER JOIN node n ON n.nid=tn.nid AND n.status=1
INNER JOIN content_type_article ma FORCE INDEX (ix_article_date) ON n.nid=ma.nid
WHERE tn.tid= 153
ORDER BY ma.field_article_date_format_value DESC
LIMIT 0, 11;
Using temporary; Using filesort means only that MySQL needs to construct a temporary result table and sort it to get the result you need. This is often a consequence of the ORDER BY ... DESC LIMIT 0,n construct you're using to get the latest postings. In itself it's not a sign of failure. See this: http://www.mysqlperformanceblog.com/2009/03/05/what-does-using-filesort-mean-in-mysql/
Here are some things to try. I am not totally sure they'll work; it's hard to know without having your data to experiment with.
Is there a BTREE index on content_type_article.field_article_date_format_value ? If so, that may help.
Do you HAVE to display the 11 most recent articles? Or can you display the 11 most recent articles that have appeared in the last week or month? If so you could add this line to your WHERE clause. It would filter your stuff by date rather than having to look all the way back to the beginning of time for matching articles. This will be especially helpful if you have a long-established Drupal site.
AND ma.field_article_date_format_value >= (CURRENT_TIME() - INTERVAL 1 MONTH)
First, try to flip the order of the INNER JOIN operations. Second, incorporate the tid=153 into the join criterion. This MAY reduce the size of the temp table you need to sort. All together my suggestions are as follows:
SELECT n.nid,
n.title,
ma.field_article_date_format_value,
ma.field_article_summary_value
FROM node n
INNER JOIN term_node tn ON (n.nid=tn.nid AND tn.tid = 153)
INNER JOIN content_type_article ma ON n.nid=ma.nid
WHERE n.status=1
AND ma.field_article_date_format_value >= (CURRENT_TIME() - INTERVAL 1 MONTH)
ORDER BY ma.field_article_date_format_value DESC
LIMIT 0, 11;
Those are
1) Covering indexes
I think the simple answer may be "covering indexes".
Especially on the content_type_article table. The "covering index" has the expression in the ORDER BY as the leading column, and includes all of the columns that are being referenced by the query. Here's the index I created (on my test table):
CREATE INDEX ct_article_ix9
ON content_type_article
(field_article_date_format_value, nid, field_article_summary_value);
And here's an excerpt of the EXPLAIN I get from the query (after I build example tables, using the InnoDB engine, including a covering index on each table):
_type table type key ref Extra
------ ----- ----- -------------- ----------- ------------------------
SIMPLE ma index ct_article_ix9 NULL Using index
SIMPLE n ref node_ix9 ma.nid Using where; Using index
SIMPLE tn ref term_node_ix9 n.nid,const Using where; Using index
Note that there's no 'Using filesort' shown in the plan, and the plan shows 'Using index' for each table referenced in the query, which basically means that all of the data needed by the query is retrieved from the index pages, with no need to reference any pages from the underlying table. (Your tables have a lot more rows than my test tables, but if you can get an explain plan that looks like this, you may get better performance.)
For completeness, here's the entire EXPLAIN output:
+----+-------------+-------+-------+---------------+----------------+---------+---------------------+------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+---------------+----------------+---------+-------- ------------+------+--------------------------+
| 1 | SIMPLE | ma | index | NULL | ct_article_ix9 | 27 | NULL | 1 | Using index |
| 1 | SIMPLE | n | ref | node_ix9 | node_ix9 | 10 | testps.ma.nid,const | 11 | Using where; Using index |
| 1 | SIMPLE | tn | ref | term_node_ix9 | term_node_ix9 | 10 | testps.n.nid,const | 11 | Using where; Using index |
+----+-------------+-------+-------+---------------+----------------+---------+---------------------+------+--------------------------+
3 rows in set (0.00 sec)
I made no changes to your query, except to omit the FORCE INDEX hint. Here are the other two "covering indexes" that I created on the other two tables referenced in the query:
CREATE INDEX node_ix9
ON node (`nid`,`status`,`title`);
CREATE INDEX term_node_ix9
ON term_node (nid,tid);
(Note that if nid is the clustering key on the node table, you may not need the covering index on the node table.)
2) Use correlated subqueries in place of joins?
If the previous idea doesn't improve anything, then, as another alternative, since the original query is returning a maximum of 11 rows, you might try rewriting the query to avoid the join operations, and instead make use of correlated subqueries. Something like the query below.
Note that this query differs significantly from the original query. The difference is that with this query, a row from the context_type_article table will be returned only one time. With the query using the joins, a row from that table could be matched to multiple rows from node and term_node tables, which would return that same row more than once. This may be viewed as either desirable or undesirable, it really depends on the cardinality, and whether the resultset meets the specification.
SELECT ( SELECT n2.nid
FROM node n2
WHERE n2.nid = ma.nid
AND n2.status = 1
LIMIT 1
) AS `nid`
, ( SELECT n3.title
FROM node n3
WHERE n3.nid = ma.nid
AND n3.status = 1
LIMIT 1
) AS `title`
, ma.field_article_date_format_value
, ma.field_article_summary_value
FROM content_type_article ma
WHERE EXISTS
( SELECT 1
FROM node n1
WHERE n1.nid = ma.nid
AND n1.status = 1
)
AND EXISTS
( SELECT 1
FROM term_node tn
WHERE tn.nid = ma.nid
AND tn.tid = 153
)
ORDER BY ma.field_article_date_format_value DESC
LIMIT 0,11
(Sometimes, a query using this type of "orrelated subquery" can have considerably WORSE performance than an equivalent query that does join operations. But in some cases, a query like this can actually perform better, especially given a very limited number of rows being returned.)
Here's the explain output for that query:
+----+--------------------+-------+-------+---------------+----------------+---------+---------------------+------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+--------------------+-------+-------+---------------+----------------+---------+---------------------+------+--------------------------+
| 1 | PRIMARY | ma | index | NULL | ct_article_ix9 | 27 | NULL | 11 | Using where; Using index |
| 5 | DEPENDENT SUBQUERY | tn | ref | term_node_ix9 | term_node_ix9 | 10 | testps.ma.nid,const | 13 | Using where; Using index |
| 4 | DEPENDENT SUBQUERY | n1 | ref | node_ix9 | node_ix9 | 10 | testps.ma.nid,const | 12 | Using where; Using index |
| 3 | DEPENDENT SUBQUERY | n3 | ref | node_ix9 | node_ix9 | 10 | testps.ma.nid,const | 12 | Using where; Using index |
| 2 | DEPENDENT SUBQUERY | n2 | ref | node_ix9 | node_ix9 | 10 | testps.ma.nid,const | 12 | Using where; Using index |
+----+--------------------+-------+-------+---------------+----------------+---------+---------------------+------+--------------------------+
5 rows in set (0.00 sec)
Note that again, each access is 'Using index', which means the query is satisfied directly from index pages, rather than having to visit any data pages in the underlying table.
Example tables
Here are the example tables (along with the indexes) that I built and populated, based on the information from your question:
CREATE TABLE `node` (`id` INT PRIMARY KEY, `nid` INT, `title` VARCHAR(10),`status` INT);
CREATE INDEX node_ix9 ON node (`nid`,`status`,`title`);
INSERT INTO `node` VALUES (1,1,'foo',1),(2,2,'bar',0),(3,3,'fee',1),(4,4,'fi',0),(5,5,'fo',1),(6,6,'fum',0),(7,7,'derp',1);
INSERT INTO `node` SELECT id+7,nid+7,title,`status` FROM node;
INSERT INTO `node` SELECT id+14,nid+14,title,`status` FROM node;
INSERT INTO `node` SELECT id+28,nid+28,title,`status` FROM node;
INSERT INTO `node` SELECT id+56,nid+56,title,`status` FROM node;
CREATE TABLE content_type_article (id INT PRIMARY KEY, nid INT, field_article_date_format_value DATETIME, field_article_summary_value VARCHAR(10));
CREATE INDEX ct_article_ix9 ON content_type_article (field_article_date_format_value, nid, field_article_summary_value);
INSERT INTO content_type_article VALUES (1001,1,'2012-01-01','foo'),(1002,2,'2012-01-02','bar'),(1003,3,'2012-01-03','fee'),(1004,4,'2012-01-04','fi'),(1005,5,'2012-01-05','fo'),(1006,6,'2012-01-06','fum'),(1007,7,'2012-01-07','derp');
INSERT INTO content_type_article SELECT id+7,nid+7, DATE_ADD(field_article_date_format_value,INTERVAL 7 DAY),field_article_summary_value FROM content_type_article;
INSERT INTO content_type_article SELECT id+14,nid+14, DATE_ADD(field_article_date_format_value,INTERVAL 14 DAY),field_article_summary_value FROM content_type_article;
INSERT INTO content_type_article SELECT id+28,nid+28, DATE_ADD(field_article_date_format_value,INTERVAL 28 DAY),field_article_summary_value FROM content_type_article;
INSERT INTO content_type_article SELECT id+56,nid+56, DATE_ADD(field_article_date_format_value,INTERVAL 56 DAY),field_article_summary_value FROM content_type_article;
CREATE TABLE term_node (id INT, tid INT, nid INT);
CREATE INDEX term_node_ix9 ON term_node (nid,tid);
INSERT INTO term_node VALUES (2001,153,1),(2002,153,2),(2003,153,3),(2004,153,4),(2005,153,5),(2006,153,6),(2007,153,7);
INSERT INTO term_node SELECT id+7, tid, nid+7 FROM term_node;
INSERT INTO term_node SELECT id+14, tid, nid+14 FROM term_node;
INSERT INTO term_node SELECT id+28, tid, nid+28 FROM term_node;
INSERT INTO term_node SELECT id+56, tid, nid+56 FROM term_node;
MySQL is "optimizing" your query so that it selects from the term_node table first, even though you are specifying to select from node first. Not knowing the data, I'm not sure which is the optimal way. The term_node table is certainly where your performance issues are since ~19,000 records is being selected from there.
Limits without ORDER BY are almost always faster because MySQL stops as soon as it finds the specified limit. With an ORDER BY, it first has to find all the records and sort them, then get the specified limit.
The simple thing to try is moving your WHERE condition into the JOIN clause, which is where it should be. That filter is specific to the table being joined. This will make sure MySQL doesn't optimize it incorrectly.
INNER JOIN term_node tn ON n.nid=tn.nid AND tn.tid=153
A more complicated thing is to do a SELECT on the term_node table and JOIN on that. That's called a DERIVED TABLE and you will see it defined as such in the EXPLAIN. Since you said it was a many-to-many, I added a DISTINCT parameter to reduce the numbers of records to join on.
SELECT ...
FROM node n
INNER JOIN content_type_article ma FORCE INDEX (ix_article_date) ON n.nid=ma.nid
INNER JOIN (SELECT DISTINCT nid FROM term_node WHERE tid=153) tn ON n.nid=tn.nid
WHERE n.status=1
ORDER BY ma.field_article_date_format_value DESC
LIMIT 0,11
MySQL 5.0 has some limitations with derived tables, so this may not work. Although there are work arounds.
You really want to avoid the sort operation happening at all if you can by taking advantage of a pre-sorted index.
To find out if this is possible, imagine your data denormalised into a single table, and ensure that everything that must be included in your WHERE clause is specifiable with a SINGLE VALUE. e.g. if you must use an IN clause on one of the columns, then sorting is inevitable.
Here's a screenshot of some sample data:
So, if you DID have your data denormalised, you could query on tid and status using single values and then sort by date descending. That would mean the following index in that case would work perfectly:
create index ix1 on denormalisedtable(tid, status, date desc);
If you had this, your query would only hit the top 10 rows and would never need to sort.
So - how do you get the same performance WITHOUT denormalising...
I think you should be able to use the STRAIGHT_JOIN clause to force the order that MySQL selects from the tables - you want to get it to select from the table you are SORTING last.
Try this:
SELECT n.nid,
n.title,
ma.field_article_date_format_value,
ma.field_article_summary_value
FROM node n
STRAIGHT_JOIN term_node tn ON n.nid=tn.nid
STRAIGHT_JOIN content_type_article ma ON n.nid=ma.nid
WHERE tn.tid= 153
AND n.status=1
ORDER BY ma.field_article_date_format_value DESC
LIMIT 0, 11;
The idea is to get MySQL to select from the node table and then from the term_node table and THEN FINALLY from the content_type_article table (the table containing the column you are sorting on).
This last join is your most important one and you want it to happen using an index so that the LIMIT clause can work without needing to sort the data.
This single index MIGHT do the trick:
create index ix1 on content_type_article(nid, field_article_date_format_value desc);
or
create index ix1 on content_type_article(nid, field_article_date_format_value desc, field_article_summary_value);
(for a covering index)
I say MIGHT, because I don't know enough about the MySQL optimiser to know if it's clever enough to handle the multiple 'nid' column values that will be getting fed into the content_type_article without having to resort the data.
Logically, it should be able to work quickly - e.g. if 5 nid values are getting fed into the final content_type_article table, then it should be able to get the top 10 of each directly from the index and merge the results together then pick the final top 10, meaning a total of 50 rows read from this table insted of the full 19006 that you're seeing currently.
Let me know how it goes.
If it works for you, further optimisation will be possible using covering indexes on the other tables to speed up the first two joins.
I am trying to speed up this django app (note: I didn't design this... just stuck maintaining it) and the biggest bottle neck seems to be these queries that are being generated by the admin. We have a content class that 4-5 other sub-classes inherit from and anytime the master list is pulled up in the admin a query like this is generated:
SELECT `content_content`.`id`,
`content_content`.`issue_id`,
`content_content`.`slug`,
`content_content`.`section_id`,
`content_content`.`priority`,
`content_content`.`group_id`,
`content_content`.`rotatable`,
`content_content`.`pub_status`,
`content_content`.`created_on`,
`content_content`.`modified_on`,
`content_content`.`old_pk`,
`content_content`.`content_type_id`,
`content_image`.`content_ptr_id`,
`content_image`.`caption`,
`content_image`.`kicker`,
`content_image`.`pic`,
`content_image`.`crop_x`,
`content_image`.`crop_y`,
`content_image`.`crop_side`,
`content_issue`.`id`,
`content_issue`.`special_issue_name`,
`content_issue`.`web_publish_date`,
`content_issue`.`issue_date`,
`content_issue`.`fm_name`,
`content_issue`.`arts_name`,
`content_issue`.`comments`,
`content_section`.`id`,
`content_section`.`name`,
`content_section`.`audiodizer_id`
FROM `content_image`
INNER
JOIN `content_content`
ON `content_image`.`content_ptr_id` = `content_content`.`id`
INNER
JOIN `content_issue`
ON `content_content`.`issue_id` = `content_issue`.`id`
INNER
JOIN `content_section`
ON `content_content`.`section_id` = `content_section`.`id`
WHERE NOT ( `content_content`.`pub_status` = -1 )
ORDER BY `content_issue`.`issue_date` DESC LIMIT 30
I ran an EXPLAIN on this and got the following:
+----+-------------+-----------------+--------+-------------------------------------------------------------------------------------------------+---------+---------+--------------------------------------+-------+---------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-----------------+--------+-------------------------------------------------------------------------------------------------+---------+---------+--------------------------------------+-------+---------------------------------+
| 1 | SIMPLE | content_image | ALL | PRIMARY | NULL | NULL | NULL | 40499 | Using temporary; Using filesort |
| 1 | SIMPLE | content_content | eq_ref | PRIMARY,issue_id,content_content_issue_id,content_content_section_id,content_content_pub_status | PRIMARY | 4 | content_image.content_ptr_id | 1 | Using where |
| 1 | SIMPLE | content_section | eq_ref | PRIMARY | PRIMARY | 4 | content_content.section_id | 1 | |
| 1 | SIMPLE | content_issue | eq_ref | PRIMARY | PRIMARY | 4 | content_content.issue_id | 1 | |
+----+-------------+-----------------+--------+-------------------------------------------------------------------------------------------------+---------+---------+--------------------------------------+-------+---------------------------------+
Now, from what I've read, I need to somehow figure out how to make the query to content_image not be terrible; however, I'm drawing a blank on where to start.
Currently, judging by the execution plan, MySQL is starting with content_image, retrieving all rows, and only thereafter using primary keys on the other tables: content_image has a foreign key to content_content, and content_content has foreign keys to content_issue and content_section. Also, only after all the joins are complete can it make much use of the ORDER BY content_issue.issue_date DESC LIMIT 30, since it can't tell which of these joins might fail, and therefore, how many records from content_issue will really be needed before it can get the first thirty rows of output.
So, I would try the following:
Change JOIN content_issue to JOIN (SELECT * FROM content_issue ORDER BY issue_date DESC LIMIT 30) content_issue. This will allow MySQL, if it starts with content_issue and works its way to the other tables, to grab a very small subset of content_issue.
Note: properly speaking, this changes the semantics of the query: it means that only records from at most the last 30 content_issues will be retrieved, and therefore that if some of those issues don't have published contents with images, then fewer than 30 records will be retrieved. I don't have enough information about your data to gauge whether this change of semantics would actually change the results you get.
Also note: I'm not suggesting to remove the ORDER BY content_issue.issue_date DESC LIMIT 30 from the end of the query. I think you want it in both places.
Add an index on content_issue.issue_date, to optimize the above subquery.
Add an index on content_image.content_ptr_id, so MySQL can work its way from content_content to content_image without doing a full table scan.