I have a table with some two or three million rows…
mysql> select count(*) from tbl;
+----------+
| count(*) |
+----------+
| 2615889 |
+----------+
1 row in set (1.23 sec)
mysql> show indexes from tbl;
+-------+------------+-------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-------+------------+-------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| tbl | 0 | PRIMARY | 1 | tbl_id | A | 2284627 | NULL | NULL | | BTREE | | |
| ...
| tbl | 1 | tbl_fld | 1 | fld | A | 2284627 | NULL | NULL | YES | BTREE | | |
+-------+------------+-------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
6 rows in set (0.30 sec)
…and for the following query I seem to do drastically better (namely, I wind up using an index) if I add an order by clause…
mysql> explain select * from tbl
-> where fld in (select fld from tbl group by fld having count(*)>1)
-> limit 1000;
+----+--------------------+-------+-------+---------------+---------+---------+------+---------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+--------------------+-------+-------+---------------+---------+---------+------+---------+-------------+
| 1 | PRIMARY | tbl | ALL | NULL | NULL | NULL | NULL | 2328333 | Using where |
| 2 | DEPENDENT SUBQUERY | tbl | index | NULL | tbl_fld | 15 | NULL | 1 | Using index |
+----+--------------------+-------+-------+---------------+---------+---------+------+---------+-------------+
2 rows in set (0.00 sec)
mysql> explain select * from tbl
-> where fld in (select fld from tbl group by fld having count(*)>1)
-> order by fld limit 1000;
+----+--------------------+-------+-------+---------------+---------+---------+------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+--------------------+-------+-------+---------------+---------+---------+------+------+-------------+
| 1 | PRIMARY | tbl | index | NULL | tbl_fld | 15 | NULL | 1000 | Using where |
| 2 | DEPENDENT SUBQUERY | tbl | index | NULL | tbl_fld | 15 | NULL | 1 | Using index |
+----+--------------------+-------+-------+---------------+---------+---------+------+------+-------------+
2 rows in set (0.00 sec)
… why is that?
From a quick glance, it looks like it's using a key in the second query which is a good thing so that it will prevent a full scan of the table.
LIMIT should not be used without ORDER BY as then it is not well-defined which rows will be returned. And as you found out ORDER BY .. LIMIT optimization does not work (is not implemented at all AFAIK), whereas when you add ORDER BY, the optimization kicks in and MySQL stops executing after enough rows are found.
Related
Last week I installed some additional database monitoring and have since come to discover that a full 30% of our database load is spent on a single query on a single table (which currently has some 6 million rows):
delete FROM mdl_grade_items_history WHERE timemodified < ?
In a testing environment, I tried to make some schema changes:
Running EXPLAIN on this query indicates that every time this query is run, a full table scan is done.
EXPLAIN DELETE FROM mdl_grade_items_history WHERE timemodified < '1490528405';
+----+-------------+-------------------------+------------+------+---------------+------+---------+------+--------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------------------------+------------+------+---------------+------+---------+------+--------+----------+-------------+
| 1 | DELETE | mdl_grade_items_history | NULL | ALL | NULL | NULL | NULL | NULL | 140784 | 100.00 | Using where |
+----+-------------+-------------------------+------------+------+---------------+------+---------+------+--------+----------+-------------+
1 row in set (0.00 sec)
Checking EXPLAIN for a (very similar) SELECT query shows a similar situation.
EXPLAIN SELECT id FROM mdl_grade_items_history WHERE timemodified < '1490528405';
+----+-------------+-------------------------+------------+------+---------------+------+---------+------+--------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------------------------+------------+------+---------------+------+---------+------+--------+----------+-------------+
| 1 | SIMPLE | mdl_grade_items_history | NULL | ALL | NULL | NULL | NULL | NULL | 140784 | 33.33 | Using where |
+----+-------------+-------------------------+------------+------+---------------+------+---------+------+--------+----------+-------------+
1 row in set, 1 warning (0.01 sec)
Checking the table definition, there does not seem to be an index on timemodified
SHOW INDEX FROM mdl_grade_items_history;
+-------------------------+------------+-------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-------------------------+------------+-------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| mdl_grade_items_history | 0 | PRIMARY | 1 | id | A | 140784 | NULL | NULL | | BTREE | | |
| mdl_grade_items_history | 1 | mdl_graditemhist_act_ix | 1 | action | A | 2 | NULL | NULL | | BTREE | | |
| mdl_grade_items_history | 1 | mdl_graditemhist_old_ix | 1 | oldid | A | 17170 | NULL | NULL | | BTREE | | |
| mdl_grade_items_history | 1 | mdl_graditemhist_cou_ix | 1 | courseid | A | 1065 | NULL | NULL | YES | BTREE | | |
| mdl_grade_items_history | 1 | mdl_graditemhist_cat_ix | 1 | categoryid | A | 2300 | NULL | NULL | YES | BTREE | | |
| mdl_grade_items_history | 1 | mdl_graditemhist_sca_ix | 1 | scaleid | A | 6 | NULL | NULL | YES | BTREE | | |
| mdl_grade_items_history | 1 | mdl_graditemhist_out_ix | 1 | outcomeid | A | 1 | NULL | NULL | YES | BTREE | | |
| mdl_grade_items_history | 1 | mdl_graditemhist_log_ix | 1 | loggeduser | A | 30 | NULL | NULL | YES | BTREE | | |
+-------------------------+------------+-------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
8 rows in set (0.00 sec)
So I tried to add one (both via CREATE INDEX and ALTER TABLE .. ADD INDEX)
CREATE INDEX `mdl_gradeitemhist_tim_ix` ON `mdl_grade_items_history` (`timemodified`);
ALTER TABLE `mdl_grade_items_history` ADD INDEX `mdl_gradeitemhist_tim_ix` (`timemodified`);
In both instances, the SELECT query was affected (note the change in type)
EXPLAIN `SELECT` id FROM mdl_grade_items_history WHERE timemodified < '1490528405';
+----+-------------+-------------------------+------------+-------+--------------------------+--------------------------+---------+------+-------+----------+--------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------------------------+------------+-------+--------------------------+--------------------------+---------+------+-------+----------+--------------------------+
| 1 | SIMPLE | mdl_grade_items_history | NULL | range | mdl_gradeitemhist_tim_ix | mdl_gradeitemhist_tim_ix | 9 | NULL | 70206 | 100.00 | Using where; Using index |
+----+-------------+-------------------------+------------+-------+--------------------------+--------------------------+---------+------+-------+----------+--------------------------+
1 row in set, 1 warning (0.00 sec)
But not the DELETE query.
EXPLAIN DELETE FROM mdl_grade_items_history WHERE timemodified < '1490528405';
+----+-------------+-------------------------+------------+------+--------------------------+------+---------+------+--------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------------------------+------------+------+--------------------------+------+---------+------+--------+----------+-------------+
| 1 | DELETE | mdl_grade_items_history | NULL | ALL | mdl_gradeitemhist_tim_ix | NULL | NULL | NULL | 140412 | 100.00 | Using where |
+----+-------------+-------------------------+------------+------+--------------------------+------+---------+------+--------+----------+-------------+
1 row in set (0.00 sec)
What have I done wrong? What else could I try?
Low cardinality indexes (action, scaleid, outcomeid) are almost never used. Get rid of them.
Having a large number of single-column indexes is a red flag. Please learn about the power and benefit of "composite" indexes. (Not relevent for the select/delete mentioned here, but probably relevant for other queries.)
Extra indexes on a table slightly slow down INSERTs and DELETEs since the indexes need to (eventually) be updated.
Extra indexes slow down UPDATEs if an indexed column is modified.
CREATE INDEX and ALTER TABLE ADD INDEX do the same thing; you probably have a redundant index now.
The EXPLAINs are different because (1) SELECT and DELETE do different things and (2) EXPLAIN is not very sophisticated.
Deleting a large number of rows takes a lot of effort -- Keep in mind that the deleted rows are hung onto in case of a ROLLBACK. Only after COMMIT can the rows really be removed. (With autocommit=ON, there is an implicit COMMIT.)
Tips on large deletes:
Deleting in chunks
Using PARTITIONs for very efficient deletion of time series
I have a table that stores tags in the JSON format for each page. Current table has about 1,000,000 records.
When I run
SELECT COUNT(*) FROM alltags WHERE tagtype = 'pages' AND JSON_SEARCH(tags, 'one', 'Hampden') IS NOT NULL;
the query takes about 20 seconds
+----------+
| COUNT(*) |
+----------+
| 23500 |
+----------+
1 row in set (19.20 sec)
The query is slow regardless if I use InnoDB or MyISAM engine.
I got indexes for tagtype, tags and tagtype with tags columns. I need the exact number of results for pagination. Is there a way I can speed up the SELECT COUNT query? Selecting actual data is pretty instant.
SELECT id FROM alltags WHERE tagtype = 'pages' AND JSON_SEARCH(tags, 'one', 'Hampden') IS NOT NULL LIMIT 0,10;
+------+
| id |
+------+
| 5072 |
| 5075 |
| 5078 |
| 5081 |
| 5084 |
| 5087 |
| 5090 |
| 5093 |
| 5096 |
| 5099 |
+------+
10 rows in set (0.01 sec)
These are the indexes
SHOW INDEX FROM alltags;
+---------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+---------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| alltags | 0 | PRIMARY | 1 | id | A | 896649 | NULL | NULL | | BTREE | | |
| alltags | 1 | tagtype | 1 | tagtype | A | 2 | NULL | NULL | | BTREE | | |
| alltags | 1 | keyword | 1 | keyword | A | 33 | NULL | NULL | | BTREE | | |
| alltags | 1 | tags | 1 | tags | A | 89665 | NULL | NULL | YES | BTREE | | |
+---------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
EXPLAIN results:
Select data
EXPLAIN SELECT id FROM alltags WHERE tagtype = 'pages' AND JSON_SEARCH(tags, 'one', 'Hampden') IS NOT NULL;
+----+-------------+---------+------------+------+---------------+---------+---------+-------+--------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+---------+------------+------+---------------+---------+---------+-------+--------+----------+-------------+
| 1 | SIMPLE | alltags | NULL | ref | tagtype | tagtype | 7 | const | 892372 | 100.00 | Using where |
+----+-------------+---------+------------+------+---------------+---------+---------+-------+--------+----------+-------------+
1 row in set, 1 warning (0.00 sec)
Select count
EXPLAIN SELECT COUNT(*) FROM alltags WHERE tagtype = 'pages' AND JSON_SEARCH(tags, 'one', 'Hampden') IS NOT NULL;
+----+-------------+---------+------------+------+---------------+---------+---------+-------+--------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+---------+------------+------+---------------+---------+---------+-------+--------+----------+-------------+
| 1 | SIMPLE | alltags | NULL | ref | tagtype | tagtype | 7 | const | 892372 | 100.00 | Using where |
+----+-------------+---------+------------+------+---------------+---------+---------+-------+--------+----------+-------------+
1 row in set, 1 warning (0.00 sec)
This question already has answers here:
SELECT vs UPDATE performance with index
(7 answers)
Closed 8 years ago.
I'm trying to improve my query so that it doesn't take so long. Is there anything I can try?
I'm using InnoDB.
My table:
mysql> describe hunted_place_review_external_urls;
+--------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+--------------+------+-----+---------+----------------+
| id | bigint(20) | NO | PRI | NULL | auto_increment |
| worker_id | varchar(255) | YES | MUL | NULL | |
| queued_at | bigint(20) | YES | MUL | NULL | |
| external_url | varchar(255) | NO | | NULL | |
| place_id | varchar(63) | NO | MUL | NULL | |
| source_id | varchar(63) | NO | | NULL | |
| successful | tinyint(1) | NO | | 0 | |
+--------------+--------------+------+-----+---------+----------------+
mysql> show index from hunted_place_review_external_urls;
+-----------------------------------+------------+--------------------------------------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-----------------------------------+------------+--------------------------------------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| hunted_place_review_external_urls | 0 | PRIMARY | 1 | id | A | 5118685 | NULL | NULL | | BTREE | | |
| hunted_place_review_external_urls | 1 | worker_id | 1 | worker_id | A | 5118685 | NULL | NULL | YES | BTREE | | |
| hunted_place_review_external_urls | 1 | queued_at | 1 | queued_at | A | 5118685 | NULL | NULL | YES | BTREE | | |
| hunted_place_review_external_urls | 1 | worker_id_and_queued_at | 1 | worker_id | A | 5118685 | NULL | NULL | YES | BTREE | | |
| hunted_place_review_external_urls | 1 | worker_id_and_queued_at | 2 | queued_at | A | 5118685 | NULL | NULL | YES | BTREE | | |
| hunted_place_review_external_urls | 1 | place_id_source_id_external_url_successful | 1 | place_id | A | 5118685 | NULL | NULL | | BTREE | | |
| hunted_place_review_external_urls | 1 | place_id_source_id_external_url_successful | 2 | source_id | A | 5118685 | NULL | NULL | | BTREE | | |
| hunted_place_review_external_urls | 1 | place_id_source_id_external_url_successful | 3 | external_url | A | 5118685 | NULL | NULL | | BTREE | | |
| hunted_place_review_external_urls | 1 | place_id_source_id_external_url_successful | 4 | successful | A | 5118685 | NULL | NULL | | BTREE | | |
+-----------------------------------+------------+--------------------------------------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
My query:
mysql> select count(*) from hunted_place_review_external_urls;
+----------+
| count(*) |
+----------+
| 4217356 |
+----------+
1 row in set (0.96 sec)
mysql> select count(*) from hunted_place_review_external_urls where worker_id is null;
+----------+
| count(*) |
+----------+
| 772626 |
+----------+
1 row in set (0.27 sec)
mysql> update hunted_place_review_external_urls set worker_id = "123" where worker_id is null order by queued_at asc limit 1;
Query OK, 1 row affected (4.80 sec)
Rows matched: 1 Changed: 1 Warnings: 0
Why is the update query taking 4s even though I have both single and composite index on queued_at and worker_id? This never happened before when the number of rows with worker_id = null was much lower. With ~200k rows instead of 780k rows, it would only take a few milliseconds.
Note the equivalent query with SELECT instead of UPDATE is extremely fast:
mysql> select * from hunted_place_review_external_urls where worker_id is null order by queued_at asc limit 1;
1 row in set (0.00 sec)
My queued_at values are timestamps expressed in number of milliseconds, such as 1398210069531
I've tried dropping my single indices on worker_id and queued_at but the problem remains:
mysql> drop index queued_at on hunted_place_review_external_urls;
Query OK, 0 rows affected (3.75 sec)
mysql> drop index worker_id on hunted_place_review_external_urls;
Query OK, 0 rows affected (3.75 sec)
mysql> show index from hunted_place_review_external_urls;
+-----------------------------------+------------+--------------------------------------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-----------------------------------+------------+--------------------------------------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| hunted_place_review_external_urls | 0 | PRIMARY | 1 | id | A | 5118685 | NULL | NULL | | BTREE | | |
| hunted_place_review_external_urls | 1 | worker_id_and_queued_at | 1 | worker_id | A | 5118685 | NULL | NULL | YES | BTREE | | |
| hunted_place_review_external_urls | 1 | worker_id_and_queued_at | 2 | queued_at | A | 5118685 | NULL | NULL | YES | BTREE | | |
| hunted_place_review_external_urls | 1 | place_id_source_id_external_url_successful | 1 | place_id | A | 5118685 | NULL | NULL | | BTREE | | |
| hunted_place_review_external_urls | 1 | place_id_source_id_external_url_successful | 2 | source_id | A | 5118685 | NULL | NULL | | BTREE | | |
| hunted_place_review_external_urls | 1 | place_id_source_id_external_url_successful | 3 | external_url | A | 5118685 | NULL | NULL | | BTREE | | |
| hunted_place_review_external_urls | 1 | place_id_source_id_external_url_successful | 4 | successful | A | 5118685 | NULL | NULL | | BTREE | | |
+-----------------------------------+------------+--------------------------------------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
Here's my EXPLAIN SELECT statement. I'm using an old version of MYSQL that doesn't support EXPLAIN UPDATE:
mysql> explain select * from hunted_place_review_external_urls where worker_id is null order by queued_at asc limit 1;
+----+-------------+-----------------------------------+------+-------------------------+-------------------------+---------+-------+---------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-----------------------------------+------+-------------------------+-------------------------+---------+-------+---------+-------------+
| 1 | SIMPLE | hunted_place_review_external_urls | ref | worker_id_and_queued_at | worker_id_and_queued_at | 768 | const | 1587282 | Using where |
+----+-------------+-----------------------------------+------+-------------------------+-------------------------+---------+-------+---------+-------------+
1 row in set (0.00 sec)
This is your query:
update hunted_place_review_external_urls
set worker_id = "123"
where worker_id is null
order by queued_at asc
limit 1;
It first has to find the row use for updating, and that requires applying the where clause and the order by clause. Either it does all the work (scanning the table and then sorting), or it uses an index. The proper index would be hunted_place_review_external_urls(worker_id, queued_at). You can add more columns at the end, but these must be the first two columns, and in that order.
EDIT:
Given that the select is fast, try this version:
update hunted_place_review_external_urls toupdate join
(select
from hunted_place_review_external_urls
where worker_id is null
order by queued_at asc
limit 1
) l
on toupdate.id = l.id
set toupdate.worker_id = '123';
I'm not sure why the indexes would be used properly here but not in the update, but hopefully this will work.
Compare the Time of Drop Index, Create Index and update.
You may notice a correlation.
When you are simple performing SELECT Queries, indexes are USEFUL, cause they speed things up.
When you are performing UPDATE or DELETE Statements - Indexes are bad, cause they slow things down! Whenever you change a value of an indexed column, MySQL needs to rebuild the index for any subsequent row. (assuming you are ALWAYS fetching the oldest entry - that means: Reindex ALL reamaining 772625 Rows.)
Try to drop the index on worker_id and see the Update Performance. If worker_id is not indexed, the update will be way faster. (Finding the entry to update is still as fast as before, because it mainly depends on the sorting performed on the indexed column queued_at and a very small subset of unindexed null values on the worker_id, matching the desired queued_at value)
I just created a dummy db and tested your setup:
With 1.000.000 Rows and both - the single index on worker_id and the composite index on worker_id|queued at, a select looks like:
SELECT * FROM `tbl` WHERE ISNULL( worker_ID ) ORDER BY queued_at ASC LIMIT 1
and the performance:
Query took 0.3360 sec
Trying to modify the worker_id in the way you do, results in:
UPDATE `tbl` SET worker_id=1 WHERE ISNULL(worker_ID) ORDER BY queued_at ASC LIMIT 1
performance:
1 row affected. (Query took 7.9592 sec)
Droping both the indexes (single and composite) on worker_id, then the same query results in:
1 row affected. (Query took 1.4364 sec)
(generated 50.000 rows per insert, so they have the same date, so the index is not "perfect" for searching, so "real" data may perform way better.)
I need your help optimizing a query. One table is a log table which has millions of entries and I try to break my query to < 1s. My query should give an overall overview and should be quick therefore. I'm sure I could make multiple simple queries over the list with help of a script. But what a script can do, can mysql I think - I hope at least. And maybe not all parts are best used, but I'm stuck in a query which makes a temporary table and filesort (which I found out is really bad). As reading around I found out to use some neat and well placed indexes but now I'm stuck in a specific point.
Let me show you my final query with its results:
SELECT
ps.SERVER_ID,
ps.FULLNAME,
SUM(CASE WHEN pml.ID_TYPE = 3 THEN 1 ELSE 0 END) 'amount_warning',
SUM(CASE WHEN pml.ID_TYPE = 4 THEN 1 ELSE 0 END) 'amount_error',
SUM(CASE WHEN pml.ID_TYPE = 5 THEN 1 ELSE 0 END) 'amount_alert',
SUM(CASE WHEN pml.ID_TYPE = 7 THEN 1 ELSE 0 END) 'amount_critical'
FROM
PAR_SERVER ps
INNER JOIN
PAR_MONITORINGv2_LOG pml ON ps.SERVER_ID = pml.SERVER_ID
WHERE
pml.CREATED_DATE > date_sub( NOW( ) , INTERVAL 7 DAY )
GROUP BY
ps.SERVER_ID;
Here is what I get:
mysql> [thequeryabove]
[...]
59 rows in set (11.69 sec)
mysql> explain [thequeryabove]
+----+-------------+-------+--------+-----------------------------+---------+---------+---------------------------+---------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+-----------------------------+---------+---------+---------------------------+---------+----------------------------------------------+
| 1 | SIMPLE | pml | ALL | SERVER_ID,SERVER_ID-ID_TYPE | NULL | NULL | NULL | 4014447 | Using where; Using temporary; Using filesort |
| 1 | SIMPLE | ps | eq_ref | PRIMARY | PRIMARY | 4 | database.pml.SERVER_ID | 1 | |
+----+-------------+-------+--------+-----------------------------+---------+---------+---------------------------+---------+----------------------------------------------+
2 rows in set (0.00 sec)
Here's my current table setup:
mysql> describe PAR_SERVER;
+----------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+----------------+--------------+------+-----+---------+----------------+
| SERVER_ID | int(255) | NO | PRI | NULL | auto_increment |
| FULLNAME | varchar(255) | YES | | NULL | |
| SHORTNAME | varchar(255) | YES | MUL | NULL | |
+----------------+--------------+------+-----+---------+----------------+
3 rows in set (0.00 sec)
mysql> show indexes from PAR_SERVER;
+------------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+------------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| PAR_SERVER | 0 | PRIMARY | 1 | SERVER_ID | A | 142 | NULL | NULL | | BTREE | |
| PAR_SERVER | 1 | shortname | 1 | SHORTNAME | A | 142 | NULL | NULL | YES | BTREE | |
+------------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
2 rows in set (0.00 sec)
mysql> select count(*) from PAR_SERVER;
+----------+
| count(*) |
+----------+
| 142 |
+----------+
1 row in set (0.00 sec)
mysql> describe PAR_MONITORINGv2_LOG;
+--------------+----------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+----------+------+-----+---------+----------------+
| ID | int(11) | NO | PRI | NULL | auto_increment |
| ID_TYPE | int(11) | NO | MUL | NULL | |
| ID_SERVICE | int(11) | NO | MUL | NULL | |
| SERVER_ID | int(11) | NO | MUL | NULL | |
| MESSAGE | tinytext | NO | | NULL | |
| CREATED_DATE | datetime | NO | | NULL | |
+--------------+----------+------+-----+---------+----------------+
6 rows in set (0.00 sec)
mysql> show indexes from PAR_MONITORINGv2_LOG;
+----------------------+------------+-------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+----------------------+------------+-------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| PAR_MONITORINGv2_LOG | 0 | PRIMARY | 1 | ID | A | 3998188 | NULL | NULL | | BTREE | |
| PAR_MONITORINGv2_LOG | 1 | ID_TYPE | 1 | ID_TYPE | A | 7 | NULL | NULL | | BTREE | |
| PAR_MONITORINGv2_LOG | 1 | ID_SERVICE | 1 | ID_SERVICE | A | 5 | NULL | NULL | | BTREE | |
| PAR_MONITORINGv2_LOG | 1 | SERVER_ID | 1 | SERVER_ID | A | 66 | NULL | NULL | | BTREE | |
| PAR_MONITORINGv2_LOG | 1 | SERVER_ID-ID_TYPE | 1 | SERVER_ID | A | 66 | NULL | NULL | | BTREE | |
| PAR_MONITORINGv2_LOG | 1 | SERVER_ID-ID_TYPE | 2 | ID_TYPE | A | 258 | NULL | NULL | | BTREE | |
+----------------------+------------+-------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
6 rows in set (0.00 sec)
mysql> select count(*) from PAR_MONITORINGv2_LOG;
+----------+
| count(*) |
+----------+
| 3998386 |
+----------+
1 row in set (0.00 sec)
Here are time results breaking my query step by step down. I may going step by step up after fixing each part taking so long. But for now only the query with runtime of 2.30 sec is currently interesting for this question.
mysql> SELECT ps.SERVER_ID, ps.FULLNAME FROM PAR_SERVER ps INNER JOIN PAR_MONITORINGv2_LOG pml ON ps.SERVER_ID = pml.SERVER_ID WHERE pml.CREATED_DATE > date_sub( NOW( ) , INTERVAL 7 DAY ) GROUP BY ps.SERVER_ID;
[...]
59 rows in set (6.41 sec)
mysql> explain [thequeryabove]
+----+-------------+-------+--------+-----------------------------+---------+---------+---------------------------+---------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+-----------------------------+---------+---------+---------------------------+---------+----------------------------------------------+
| 1 | SIMPLE | pml | ALL | SERVER_ID,SERVER_ID-ID_TYPE | NULL | NULL | NULL | 4014788 | Using where; Using temporary; Using filesort |
| 1 | SIMPLE | ps | eq_ref | PRIMARY | PRIMARY | 4 | database.pml.SERVER_ID | 1 | |
+----+-------------+-------+--------+-----------------------------+---------+---------+---------------------------+---------+----------------------------------------------+
2 rows in set (0.00 sec)
mysql> SELECT ps.SERVER_ID, ps.FULLNAME FROM PAR_SERVER ps INNER JOIN PAR_MONITORINGv2_LOG pml ON ps.SERVER_ID = pml.SERVER_ID GROUP BY ps.SERVER_ID;
[...]
59 rows in set (2.30 sec)
mysql> explain [thequeryabove]
+----+-------------+-------+--------+-----------------------------+-----------+---------+---------------------------+---------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+-----------------------------+-----------+---------+---------------------------+---------+----------------------------------------------+
| 1 | SIMPLE | pml | index | SERVER_ID,SERVER_ID-ID_TYPE | SERVER_ID | 4 | NULL | 4015694 | Using index; Using temporary; Using filesort |
| 1 | SIMPLE | ps | eq_ref | PRIMARY | PRIMARY | 4 | database.pml.SERVER_ID | 1 | |
+----+-------------+-------+--------+-----------------------------+-----------+---------+---------------------------+---------+----------------------------------------------+
2 rows in set (0.00 sec)
mysql> SELECT pml.SERVER_ID FROM PAR_MONITORINGv2_LOG pml GROUP BY pml.SERVER_ID;
[...]
65 rows in set (0.00 sec)
mysql> explain [thequeryabove]
+----+-------------+-------+-------+---------------+-----------+---------+------+------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+---------------+-----------+---------+------+------+--------------------------+
| 1 | SIMPLE | pml | range | NULL | SERVER_ID | 4 | NULL | 67 | Using index for group-by |
+----+-------------+-------+-------+---------------+-----------+---------+------+------+--------------------------+
1 row in set (0.00 sec)
I was able to improve the query a lot by defining an index for (SERVER_ID, ID_TYPE) as my following example query confirms:
mysql> SELECT count(*) 'count_warnings' FROM PAR_MONITORINGv2_LOG pml WHERE pml.SERVER_ID = 191 AND pml.ID_TYPE = 3 GROUP BY pml.SERVER_ID;
[...]
1 row in set (0.01 sec)
mysql> explain [thequeryabove]
+----+-------------+-------+------+-------------------------------------+-------------------+---------+-------------+-------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+-------------------------------------+-------------------+---------+-------------+-------+-------------+
| 1 | SIMPLE | pml | ref | ID_TYPE,SERVER_ID,SERVER_ID-ID_TYPE | SERVER_ID-ID_TYPE | 8 | const,const | 10254 | Using index |
+----+-------------+-------+------+-------------------------------------+-------------------+---------+-------------+-------+-------------+
1 row in set (0.00 sec)
I'm stuck now in the most broked down query with a long execution time of 2.30 sec. I don't know how to use indexes for such a query not having any where clause.
Your query will definitely benefit the most from adding composite index on PAR_MONITORINGv2_LOG(CREATED_DATE, SERVER_ID,ID_TYPE). However, I suggest even simple index on CREATED_DATE will improve performance a lot.
I have the following InnoDB table, which acts kinda weird under MySQL 5.1.49
(mysql Ver 14.14 Distrib 5.1.49, for debian-linux-gnu (x86_64) using readline 6.1)
mysql> desc forum_favorite;
+-----------+----------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+----------------------+------+-----+---------+-------+
| id_member | smallint(5) unsigned | YES | MUL | NULL | |
| id_topic | int(10) unsigned | YES | MUL | NULL | |
+-----------+----------------------+------+-----+---------+-------+
2 rows in set (0.00 sec)
mysql> show index from forum_favorite;
+----------------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+----------------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| forum_favorite | 1 | id_member | 1 | id_member | A | 2134 | NULL | NULL | YES | BTREE | |
| forum_favorite | 1 | id_topic | 1 | id_topic | A | 3201 | NULL | NULL | YES | BTREE | |
+----------------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
2 rows in set (0.00 sec)
Now, check the query:
mysql> SELECT id_topic FROM forum_favorite WHERE (id_member = 2);
+----------+
| id_topic |
+----------+
| 1249 |
| 20209 |
| 91878 |
| 99026 |
| 90257 |
| 1179 |
| 1179 |
+----------+
7 rows in set (0.00 sec)
When I search for a specific topic with a given member, it gives an empty result set. WHY?
mysql> select * from forum_favorite where id_member = 2 and id_topic = 1249;
Empty set (0.00 sec)
But when I search for another topic, it comes back ok...
It can be found without id_member in the where clause:
mysql> select * from forum_favorite where id_topic = 1249;
+-----------+----------+
| id_member | id_topic |
+-----------+----------+
| 2 | 1249 |
+-----------+----------+
1 rows in set (0.00 sec)
Indexes:
mysql> show index from forum_favorite;
+----------------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+----------------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| forum_favorite | 1 | id_member | 1 | id_member | A | 2134 | NULL | NULL | YES | BTREE | |
| forum_favorite | 1 | id_topic | 1 | id_topic | A | 3201 | NULL | NULL | YES | BTREE | |
+----------------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
2 rows in set (0.00 sec)
Explain on the incriminating query:
mysql> explain select * from forum_favorite where id_member = 2 and id_topic = 1249;
+----+-------------+----------------+-------------+--------------------+--------------------+---------+------+------+---------------------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+----------------+-------------+--------------------+--------------------+---------+------+------+---------------------------------------------------------------+
| 1 | SIMPLE | forum_favorite | index_merge | id_member,id_topic | id_member,id_topic | 3,5 | NULL | 1 | Using intersect(id_member,id_topic); Using where; Using index |
+----+-------------+----------------+-------------+--------------------+--------------------+---------+------+------+---------------------------------------------------------------+
1 row in set (0.00 sec)
Explain on the query, with another topicid. WHAT THE HELL???
mysql> explain select * from forum_favorite where (id_member = 2) and id_topic = 20209;
+----+-------------+----------------+------+--------------------+----------+---------+-------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+----------------+------+--------------------+----------+---------+-------+------+-------------+
| 1 | SIMPLE | forum_favorite | ref | id_member,id_topic | id_topic | 5 | const | 1 | Using where |
+----+-------------+----------------+------+--------------------+----------+---------+-------+------+-------------+
1 row in set (0.00 sec
I have migrated from an old mysql 4.x database to mysql 5.1 recently, where the above queries gave consistent results.
What could be the problem?!?! What makes the optimizer go bonkers?
Have you tried to run OPTIMIZE on the table?
UPD:
For InnoDB tables, OPTIMIZE TABLE is mapped to ALTER TABLE, which rebuilds the table to update index statistics and free unused space in the clustered index.
I think the problem is actually in Using intersect(id_member,id_topic);. I would extend the id_topic index to (id_topic, id_member) so that it used a single key for the lookup instead of index merge (which is actually rather rare in MySQL).