Looking to find out some ideas on how to optimise this query - its pretty awkward with several range searches
SELECT t1.id, t1.custom_track_id, t1.deleted, t1.last_modified, t1.client_id
FROM tracks t1
INNER JOIN tracks t2 ON t1.custom_track_id = t2.custom_track_id
AND t1.last_modified > t2.last_modified
AND t1.deleted !=0
AND t2.deleted =0
AND t2.client_id
IN ( 103, 115, 116 )
WHERE t1.client_id
IN ( 103, 115, 116 )
All the fields its looking to find and joining on are INT fields.
Indexes (yes theres a few dodgy ones in there) :
+--------+------------+------------------------------------------------+--------------+----------------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+--------+------------+------------------------------------------------+--------------+----------------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| tracks | 0 | PRIMARY | 1 | id | A | 566045 | NULL | NULL | | BTREE | | |
| tracks | 1 | client_id | 1 | client_id | A | 589 | NULL | NULL | | BTREE | | |
| tracks | 1 | featured | 1 | featured | A | 16 | NULL | NULL | | BTREE | | |
| tracks | 1 | system_status | 1 | system_status | A | 20 | NULL | NULL | | BTREE | | |
| tracks | 1 | created_by | 1 | created_by | A | 225 | NULL | NULL | | BTREE | | |
| tracks | 1 | custom_track_id | 1 | custom_track_id | A | 283022 | NULL | NULL | | BTREE | | |
| tracks | 1 | custom_track_id | 2 | custom_artist_id | A | 283022 | NULL | NULL | | BTREE | | |
| tracks | 1 | counterpoint_id | 1 | counterpoint_id | A | 113209 | NULL | NULL | YES | BTREE | | |
| tracks | 1 | system_status_2 | 1 | system_status | A | 20 | NULL | NULL | | BTREE | | |
| tracks | 1 | composition | 1 | composition | A | 13 | NULL | NULL | YES | BTREE | | |
| tracks | 1 | published_start_date | 1 | published_start_date | A | 13 | NULL | NULL | YES | BTREE | | |
| tracks | 1 | published_end_date | 1 | published_end_date | A | 13 | NULL | NULL | YES | BTREE | | |
| tracks | 1 | deleted | 1 | deleted | A | 20 | NULL | NULL | | BTREE | | |
| tracks | 1 | restricted_access_required | 1 | restricted_access_required | A | 13 | NULL | NULL | | BTREE | | |
| tracks | 1 | mv_clientid_deleted_featured_restrictedaccess | 1 | client_id | A | 336 | NULL | NULL | | BTREE | | |
| tracks | 1 | mv_clientid_deleted_featured_restrictedaccess | 2 | custom_track_id | A | 336 | NULL | NULL | | BTREE | | |
| tracks | 1 | mv_clientid_deleted_featured_restrictedaccess | 3 | deleted | A | 336 | NULL | NULL | | BTREE | | |
| tracks | 1 | mv_clientid_deleted_featured_restrictedaccess | 4 | last_modified | A | 336 | NULL | NULL | | BTREE | | |
| tracks | 1 | mv_clientid_customtrackid_deleted_lastmodified | 1 | client_id | A | 336 | NULL | NULL | | BTREE | | |
| tracks | 1 | mv_clientid_customtrackid_deleted_lastmodified | 2 | custom_track_id | A | 336 | NULL | NULL | | BTREE | | |
| tracks | 1 | mv_clientid_customtrackid_deleted_lastmodified | 3 | deleted | A | 336 | NULL | NULL | | BTREE | | |
| tracks | 1 | mv_clientid_customtrackid_deleted_lastmodified | 4 | last_modified | A | 336 | NULL | NULL | | BTREE | | |
+--------+------------+------------------------------------------------+--------------+----------------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
And the EXPLAIN :
---+---------+-------------------------------------------------+-------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+--------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------+---------+-------------------------------------------------+-------+--------------------------+
| 1 | SIMPLE | t1 | range | client_id,custom_track_id,deleted,mv_clientid_deleted_featured_restrictedaccess,mv_clientid_customtrackid_deleted_lastmodified | mv_clientid_deleted_featured_restrictedaccess | 4 | NULL | 16018 | Using where; Using index |
| 1 | SIMPLE | t2 | ref | client_id,custom_track_id,deleted,mv_clientid_deleted_featured_restrictedaccess,mv_clientid_customtrackid_deleted_lastmodified | custom_track_id | 302 | synchtank_shared_application.t1.custom_track_id | 2 | Using where |
+----+-------------+-------+-------+--------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------+---------+-------------------------------------------------+-------+--------------------------+
So looking for ways to either optimise the query or the indexes. Im also curious why of the 2 composite indexes it could match on, it chooses the one without last_modified in it.
EXPLAIN for Stephan :
+----+-------------+-------+-------+-----------------------------------------------+-----------------------------------------------+---------+------+-------+---------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+-----------------------------------------------+-----------------------------------------------+---------+------+-------+---------------------------------------------+
| 1 | SIMPLE | t1 | range | client_id | client_id | 4 | NULL | 8111 | Using where |
| 1 | SIMPLE | t2 | range | mv_clientid_deleted_featured_restrictedaccess | mv_clientid_deleted_featured_restrictedaccess | 4 | NULL | 16018 | Using where; Using index; Using join buffer |
+----+-------------+-------+-------+-----------------------------------------------+-----------------------------------------------+---------+------+-------+---------------------------------------------+
You can try to force mysql to use the indexes:
SELECT
t1.id,
t1.custom_track_id,
t1.deleted,
t1.last_modified,
t1.client_id
FROM
tracks t1 FORCE INDEX(`client_id`)
INNER JOIN tracks t2 FORCE INDEX(`mv_clientid_deleted_featured_restrictedaccess`)
ON t1.custom_track_id = t2.custom_track_id
AND t1.last_modified > t2.last_modified
AND t2.deleted = 0
AND t2.client_id IN ( 103, 115, 116 )
WHERE
t1.client_id IN ( 103, 115, 116 )
AND t1.deleted !=0
Usually the reason why mysql don't uses the most efficient index is due to the false stats in cardinality that is why its a good practice to run OPTIMIZE TABLE on big tables with lots of columns and indexes .
Related
For some reason, MySQL chooses completely wrong indexes. It feels like it doesn't check which index is the best for the query.
Some indexes on the contacts table:
+----------+------------+---------------------------------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+----------+------------+---------------------------------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| contacts | 0 | PRIMARY | 1 | id | A | 2227424 | NULL | NULL | | BTREE | | |
| contacts | 1 | idx_contacts_date_modfied | 1 | date_modified | A | 261152 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_contacts_id_del | 1 | id | A | 2228229 | NULL | NULL | | BTREE | | |
| contacts | 1 | idx_contacts_id_del | 2 | deleted | A | 2228229 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_contacts_date_entered | 1 | date_entered | A | 286622 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_cont_last_first | 1 | last_name | A | 783981 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_cont_last_first | 2 | first_name | A | 1434526 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_cont_last_first | 3 | deleted | A | 1434526 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_contacts_del_last | 1 | deleted | A | 1 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_contacts_del_last | 2 | last_name | A | 830164 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_cont_del_reports | 1 | deleted | A | 1 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_cont_del_reports | 2 | reports_to_id | A | 1 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_cont_del_reports | 3 | last_name | A | 830164 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_reports_to_id | 1 | reports_to_id | A | 1 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_del_id_user | 1 | deleted | A | 1 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_del_id_user | 2 | id | A | 2228229 | NULL | NULL | | BTREE | | |
| contacts | 1 | idx_del_id_user | 3 | assigned_user_id | A | 2228229 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_cont_assigned | 1 | assigned_user_id | A | 2 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_contact_title | 1 | title | A | 1 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_contact_mkto_id | 1 | mkto_id | A | 1 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_contacts_first_last | 1 | first_name | A | 265736 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_contacts_first_last | 2 | last_name | A | 1453136 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_contacts_first_last | 3 | deleted | A | 1453136 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_contacts_assigned_del | 1 | assigned_user_id | A | 2 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_contacts_assigned_del | 2 | deleted | A | 2 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_contacts_tmst_id | 1 | team_set_id | A | 1 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_contacts_tmst_id | 2 | deleted | A | 1 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_del_date_modified_id | 1 | deleted | A | 1 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_del_date_modified_id | 2 | date_modified | A | 265687 | NULL | NULL | YES | BTREE | | |
+----------+------------+---------------------------------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
The query:
SELECT SQL_NO_CACHE contacts.id,
contacts.date_modified contacts__date_modified
FROM contacts
INNER JOIN
(SELECT tst.team_set_id
FROM team_sets_teams tst
INNER JOIN team_memberships team_membershipscontacts ON (team_membershipscontacts.team_id = tst.team_id)
AND (team_membershipscontacts.user_id = '5daa2e92-c347-11e9-afc5-525400a80916')
AND (team_membershipscontacts.deleted = 0)
GROUP BY tst.team_set_id) contacts_tf ON contacts_tf.team_set_id = contacts.team_set_id
LEFT JOIN contacts_cstm contacts_cstm ON contacts_cstm.id_c = contacts.id
WHERE contacts.deleted = 0
ORDER BY contacts.date_modified DESC,
contacts.id DESC
LIMIT 21;
For some reason, the compiler chooses index idx_contacts_del_last which contains field which is not even in the query! and the query takes around 2 minutes (2M rows).
When I force idx_contacts_date_modfied or idx_del_date_modified_id index, the query takes 0.5s.
For fun, I tried to delete the index idx_contacts_del_last and add it again. After that, mysql chose a DIFFERENT index - idx_reports_to_id implying that MySQL does not even try to choose optimal index and probably chooses the first index which it sees... From my observation the first index, which have field deleted as the first field and was added first is chosen.
So I dropped and recreated all indexes except the one I want the query to use and it finally chooses the correct one. But now I need a different query to use a different index and it still keeps using the one I didn't recreate.
Is there some setting which can make mysql look more thoroughly on index optimization? I Use mysql 5.7.6
EDIT:
The query is system generated and I can't alter it
Explain:
+----+-------------+--------------------------+------------+--------+--------------------------------------------------------------------------------------------------------------------------------+----------------------------+---------+-------------------------------------------+---------+----------+---------------------------------------------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+--------------------------+------------+--------+--------------------------------------------------------------------------------------------------------------------------------+----------------------------+---------+-------------------------------------------+---------+----------+---------------------------------------------------------------------+
| 1 | PRIMARY | contacts | NULL | ref | idx_contacts_del_last,idx_cont_del_reports,idx_del_id_user,idx_contacts_tmst_id,idx_del_date_modified,idx_del_date_modified_id | idx_contacts_del_last | 2 | const | 1114111 | 100.00 | Using temporary; Using filesort |
| 1 | PRIMARY | <derived2> | NULL | ALL | NULL | NULL | NULL | NULL | 2 | 50.00 | Using where; Using join buffer (Block Nested Loop) |
| 1 | PRIMARY | contacts_cstm | NULL | eq_ref | PRIMARY | PRIMARY | 144 | sugarcrm.contacts.id | 1 | 100.00 | Using index |
| 2 | DERIVED | team_membershipscontacts | NULL | ref | idx_team_membership,idx_teammemb_team_user,idx_del_team_user | idx_team_membership | 145 | const | 2 | 99.36 | Using index condition; Using where; Using temporary; Using filesort |
| 2 | DERIVED | tst | NULL | ref | idx_ud_set_id,idx_ud_team_id,idx_ud_team_set_id,idx_ud_team_id_team_set_id | idx_ud_team_id_team_set_id | 144 | sugarcrm.team_membershipscontacts.team_id | 1 | 100.00 | Using index |
+----+-------------+--------------------------+------------+--------+--------------------------------------------------------------------------------------------------------------------------------+----------------------------+---------+-------------------------------------------+---------+----------+---------------------------------------------------------------------+
Turns out there is a bug in mysql https://bugs.mysql.com/bug.php?id=69721
After setting
SET SESSION optimizer_switch='block_nested_loop=off';
The queries fly as a charm.
Have these indexes (in the column-order given):
team_membershipscontacts: (user_id, deleted, team_id)
contacts: (team_set_id, deleted)
team_sets_teams: (team_id, team_set_id)
Remove this; it seems to be a waste of effort:
LEFT JOIN contacts_cstm contacts_cstm ON contacts_cstm.id_c = contacts.id
This issue is fixed in MySQL 8.0.20 (and possibly MySQL 8.0.18 as well however that had a slightly different issue).
I have this query and it takes ages (about 10 min) to complete.
SELECT COUNT(DISTINCT(column)) var,
column2 var2
FROM table
WHERE column<>'X' and
column2 between 'Y' and 'Z'
GROUP BY var2
ORDER BY var DESC
Any ideas how to optimize for speed? I tried with indexes but still slow. Maybe they are not set properly. Y and Z are timestamps, if it matters, and X is something that is not needed at all for this query but is in the table for it is needed for other queries from the same app. The table is very large - millions of rows, and it is yet to grow.
Edit: Here is the EXPLAIN result from an example:
mysql> EXPLAIN SELECT COUNT(DISTINCT(ip)) v, geo n from idevaff_iptracking where geo<>'XX' and stamp between '1525122000' and '1543615199' group by n order by v desc;
+------+-------------+--------------------+-------+------------------------+--------------+---------+------+---------+-----------------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+--------------------+-------+------------------------+--------------+---------+------+---------+-----------------------------------------------------------+
| 1 | SIMPLE | idevaff_iptracking | range | stamp,geo,geo_stamp_ip | geo_stamp_ip | 9 | NULL | 3469323 | Using where; Using index; Using temporary; Using filesort |
+------+-------------+--------------------+-------+------------------------+--------------+---------+------+---------+-----------------------------------------------------------+
1 row in set (0.00 sec)
Table locums are as follows:
id,acct_id,ip,refer,stamp,hit_time,hit_date,src1,src2,split,sub_id,tid1,tid2,tid3,tid4,target_url,geo.
Indexes are as follows:
mysql> SHOW INDEX FROM idevaff_iptracking
-> ;
+--------------------+------------+--------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+--------------------+------------+--------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| idevaff_iptracking | 0 | PRIMARY | 1 | id | A | 6775984 | NULL | NULL | | BTREE | | |
| idevaff_iptracking | 1 | acct_id_ip | 1 | acct_id | A | 2 | NULL | NULL | | BTREE | | |
| idevaff_iptracking | 1 | acct_id_ip | 2 | ip | A | 6775984 | NULL | NULL | YES | BTREE | | |
| idevaff_iptracking | 1 | ip | 1 | ip | A | 6775984 | NULL | NULL | YES | BTREE | | |
| idevaff_iptracking | 1 | stamp | 1 | stamp | A | 6775984 | NULL | NULL | | BTREE | | |
| idevaff_iptracking | 1 | acct_id | 1 | acct_id | A | 4 | NULL | NULL | | BTREE | | |
| idevaff_iptracking | 1 | geo | 1 | geo | A | 440 | NULL | NULL | YES | BTREE | | |
| idevaff_iptracking | 1 | tid1 | 1 | tid1 | A | 276 | NULL | NULL | YES | BTREE | | |
| idevaff_iptracking | 1 | tid2 | 1 | tid2 | A | 514 | NULL | NULL | YES | BTREE | | |
| idevaff_iptracking | 1 | tid3 | 1 | tid3 | A | 34 | NULL | NULL | YES | BTREE | | |
| idevaff_iptracking | 1 | tid4 | 1 | tid4 | A | 5623 | NULL | NULL | YES | BTREE | | |
| idevaff_iptracking | 1 | acct_id_stamp_ip | 1 | acct_id | A | 744 | NULL | NULL | | BTREE | | |
| idevaff_iptracking | 1 | acct_id_stamp_ip | 2 | stamp | A | 6775984 | NULL | NULL | | BTREE | | |
| idevaff_iptracking | 1 | acct_id_stamp_ip | 3 | ip | A | 6775984 | NULL | NULL | YES | BTREE | | |
| idevaff_iptracking | 1 | geo_stamp_ip | 1 | geo | A | 22362 | NULL | NULL | YES | BTREE | | |
| idevaff_iptracking | 1 | geo_stamp_ip | 2 | stamp | A | 6775984 | NULL | NULL | | BTREE | | |
| idevaff_iptracking | 1 | geo_stamp_ip | 3 | ip | A | 6775984 | NULL | NULL | YES | BTREE | | |
| idevaff_iptracking | 1 | acct_id_tid1_stamp | 1 | acct_id | A | 658 | NULL | NULL | | BTREE | | |
| idevaff_iptracking | 1 | acct_id_tid1_stamp | 2 | tid1 | A | 11866 | NULL | NULL | YES | BTREE | | |
| idevaff_iptracking | 1 | acct_id_tid1_stamp | 3 | stamp | A | 6775984 | NULL | NULL | | BTREE | | |
| idevaff_iptracking | 1 | acct_id_tid2_stamp | 1 | acct_id | A | 2 | NULL | NULL | | BTREE | | |
| idevaff_iptracking | 1 | acct_id_tid2_stamp | 2 | tid2 | A | 18666 | NULL | NULL | YES | BTREE | | |
| idevaff_iptracking | 1 | acct_id_tid2_stamp | 3 | stamp | A | 6775984 | NULL | NULL | | BTREE | | |
| idevaff_iptracking | 1 | acct_id_tid3_stamp | 1 | acct_id | A | 2 | NULL | NULL | | BTREE | | |
| idevaff_iptracking | 1 | acct_id_tid3_stamp | 2 | tid3 | A | 1832 | NULL | NULL | YES | BTREE | | |
| idevaff_iptracking | 1 | acct_id_tid3_stamp | 3 | stamp | A | 6775984 | NULL | NULL | | BTREE | | |
| idevaff_iptracking | 1 | acct_id_tid4_stamp | 1 | acct_id | A | 2 | NULL | NULL | | BTREE | | |
| idevaff_iptracking | 1 | acct_id_tid4_stamp | 2 | tid4 | A | 5060 | NULL | NULL | YES | BTREE | | |
| idevaff_iptracking | 1 | acct_id_tid4_stamp | 3 | stamp | A | 6775984 | NULL | NULL | | BTREE | | |
+--------------------+------------+--------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
29 rows in set (0.00 sec)
Add this composite index:
INDEX(column2, column)
If that does not suffice, we need to see SHOW CREATE TABLE in order to discuss further. (geo_stamp_ip is not as good.)
It is usually a mistake to splay an array (the tid's) across columns.
EXPLAIN FORMAT=JSON
SELECT COUNT(DISTINCT ip) v, geo n
from idevaff_iptracking
where geo<>'XX'
and stamp between '1525122000' AND '1543615199'
group by n
order by v desc;
Some of the indexes are redundant. In general, INDEX(a) can be removed if you have INDEX(a,b). (For example: acct_id_ip)
I'm having an issue with the speed of a query I wan't to do.
The rooms table has 2685588 rows, and the room_active has less than 100, but it isn't slow with normal queries.
The query is:
SELECT rooms.*, room_active.active_users
FROM rooms LEFT JOIN room_active ON (room_active.roomid = rooms.id)
WHERE caption LIKE '%room%' ORDER BY room_active.active_users LIMIT 50
Runtime: 4.3s
It takes more than 5 seconds to execute when it should take less than one like does this one:
SELECT rooms.*, room_active.active_users
FROM rooms LEFT JOIN room_active ON (room_active.roomid = rooms.id)
WHERE caption LIKE '%room%' LIMIT 50
Runtime: 0.3s
The describe for the first query is this:
+----+-------------+-------------+--------+----------------+---------+---------+----------------+---------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------------+--------+----------------+---------+---------+----------------+---------+----------------------------------------------+
| 1 | SIMPLE | rooms | ALL | NULL | NULL | NULL | NULL | 2210576 | Using where; Using temporary; Using filesort |
| 1 | SIMPLE | room_active | eq_ref | PRIMARY,roomid | PRIMARY | 4 | xukys.rooms.id | 1 | NULL |
+----+-------------+-------------+--------+----------------+---------+---------+----------------+---------+----------------------------------------------+
Indexes:
+-------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| rooms | 0 | PRIMARY | 1 | id | A | 2210613 | NULL | NULL | | BTREE | | |
| rooms | 1 | owner | 1 | owner | A | 2210613 | NULL | NULL | | BTREE | | |
| rooms | 1 | roomtype | 1 | roomtype | A | 18 | NULL | NULL | | BTREE | | |
| rooms | 1 | caption | 1 | caption | A | 2210613 | NULL | NULL | | BTREE | | |
| rooms | 1 | category | 1 | category | A | 18 | NULL | NULL | | BTREE | | |
| rooms | 1 | users_now | 1 | users_now | A | 18 | NULL | NULL | | BTREE | | |
| rooms | 1 | tags | 1 | tags | A | 18 | NULL | NULL | | BTREE | | |
| rooms | 1 | users_max | 1 | users_max | A | 18 | NULL | NULL | | BTREE | | |
| rooms | 1 | description | 1 | description | A | 368435 | NULL | NULL | | BTREE | | |
| rooms | 1 | state | 1 | state | A | 18 | NULL | NULL | | BTREE | | |
| rooms | 1 | model_name | 1 | model_name | A | 18 | NULL | NULL | | BTREE | | |
| rooms | 1 | public_ccts | 1 | public_ccts | A | 18 | NULL | NULL | | BTREE | | |
| rooms | 1 | score | 1 | score | A | 18 | NULL | NULL | | BTREE | | |
| rooms | 1 | password | 1 | password | A | 18 | NULL | NULL | | BTREE | | |
| rooms | 1 | icon_bg | 1 | icon_bg | A | 954 | NULL | NULL | | BTREE | | |
| rooms | 1 | icon_fg | 1 | icon_fg | A | 18 | NULL | NULL | | BTREE | | |
| rooms | 1 | badge_id | 1 | badge_id | A | 804 | NULL | NULL | | BTREE | | |
+-------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
+-------------+------------+--------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-------------+------------+--------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| room_active | 0 | PRIMARY | 1 | roomid | A | 116 | NULL | NULL | | BTREE | | |
| room_active | 0 | roomid | 1 | roomid | A | 116 | NULL | NULL | | BTREE | | |
| room_active | 1 | active_users | 1 | active_users | A | 29 | NULL | NULL | | BTREE | | |
+-------------+------------+--------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
The ORDER BY is removing the advantage of LIMIT 50 because to be able to order by the 50 first elements he has to fetch all the records.
maybe if you tell us what exactly you want, we can help with the query. For example do you really need to do a LEFT JOIN and get all the rooms who don't have a room_active ?
not sure if it helps, but try this one
SELECT rooms.*, room_active.active_users
FROM room_active ra JOIN rooms r ON (room_active.roomid = rooms.id)
WHERE r.caption LIKE '%room%' ORDER BY ra.active_users LIMIT 50
It will ignore the rooms that don't have a room_active, reducing the fetched records
Follow following steps and check performance.
Create index on field 'caption' for its first 4 characters and on 'active_users'.
Instead of "rooms.*", select fields explicitly.
Use "caption LIKE 'room%'" instead of "caption LIKE '%room%'".
If this still slow, you should create a view for your joined query and then perform simple select statement on your view.
1 you might want a index perhaps on the order?
2 You don't need %room% do you? room% would be faster, your effectivly searching all files if you don't
3 don't left join if you can and only want active? instead inner join because it will require less joins (it doesnt join the ones that are null)
4 dont select .* unless needed?
5. Maybe you can use a differnt function such as Contains if you are using oracle
6. Put a index on caption its being mass searched thats probably the problem
I have a problem where the correct index is no being used.
I have the following indexes on an innodb table (about 500,000 rows) :
+-------------+------------+------------------------+--------------+------------------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+-------------+------------+------------------------+--------------+------------------------+-----------+-------------+----------+--------+------+------------+---------+
| osdate_user | 0 | PRIMARY | 1 | id | A | 419700 | NULL | NULL | | BTREE | |
| osdate_user | 0 | email | 1 | email | A | 419700 | NULL | NULL | | BTREE | |
| osdate_user | 0 | username | 1 | username | A | 419700 | NULL | NULL | YES | BTREE | |
| osdate_user | 1 | lastvisit | 1 | lastvisit | A | 419700 | NULL | NULL | | BTREE | |
| osdate_user | 1 | active | 1 | active | A | 8 | NULL | NULL | YES | BTREE | |
| osdate_user | 1 | gender | 1 | gender | A | 88 | NULL | NULL | YES | BTREE | |
| osdate_user | 1 | regdate | 1 | regdate | A | 419700 | NULL | NULL | | BTREE | |
| osdate_user | 1 | lastupdate | 1 | lastupdate | A | 419700 | NULL | NULL | YES | BTREE | |
| osdate_user | 1 | password | 1 | password | A | 419700 | NULL | NULL | | BTREE | |
| osdate_user | 1 | age | 1 | age | A | 190 | NULL | NULL | YES | BTREE | |
| osdate_user | 1 | is_new | 1 | is_new | A | 8 | NULL | NULL | YES | BTREE | |
| osdate_user | 1 | private_photos | 1 | private_photos | A | 8 | NULL | NULL | | BTREE | |
| osdate_user | 1 | pictures_cnt | 1 | pictures_cnt | A | 10 | NULL | NULL | YES | BTREE | |
| osdate_user | 1 | pictures_cnt | 2 | private_photos | A | 10 | NULL | NULL | | BTREE | |
| osdate_user | 1 | status | 1 | status | A | 19 | NULL | NULL | | BTREE | |
| osdate_user | 1 | status | 2 | active | A | 19 | NULL | NULL | YES | BTREE | |
| osdate_user | 1 | status | 3 | gender | A | 19 | NULL | NULL | YES | BTREE | |
| osdate_user | 1 | status | 4 | age | A | 19 | NULL | NULL | YES | BTREE | |
| osdate_user | 1 | status | 5 | country | A | 7630 | NULL | NULL | YES | BTREE | |
| osdate_user | 1 | status | 6 | city | A | 46633 | NULL | NULL | YES | BTREE | |
| osdate_user | 1 | status | 7 | pictures_cnt | A | 83940 | NULL | NULL | YES | BTREE | |
| osdate_user | 1 | status | 8 | private_photos | A | 139900 | NULL | NULL | | BTREE | |
| osdate_user | 1 | status | 9 | lang | A | 209850 | NULL | NULL | | BTREE | |
| osdate_user | 1 | status | 10 | is_new | A | 209850 | NULL | NULL | YES | BTREE | |
+-------------+------------+------------------------+--------------+------------------------+-----------+-------------+----------+--------+------+------------+---------+
This query :
EXPLAIN EXTENDED SELECT user.id, user.active
FROM osdate_user user
WHERE user.active =1
AND user.status = 'active'
AND user.gender = 'M'
AND user.age
BETWEEN 19
AND 35
AND user.pictures_cnt >0
AND user.private_photos = '0'
ORDER BY user.lastvisit DESC
LIMIT 0 , 24
Displays the following plan :
+----+-------------+---------+-------------+--------------------------------------------------------------+------------------------------+---------+---------------------------+-------+----------+----------------------------------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+---------+-------------+--------------------------------------------------------------+------------------------------+---------+---------------------------+-------+----------+----------------------------------------------------------------------------+
| 1 | SIMPLE | user | index_merge | PRIMARY,active,gender,age,private_photos,pictures_cnt,status | gender,private_photos,active | 4,1,2 | NULL | 28204 | 100.00 | Using intersect(gender,private_photos,active); Using where; Using filesort |
| 1 | SIMPLE | userext | eq_ref | userid | userid | 4 | db_name.user.id | 1 | 100.00 | Using index |
+----+-------------+---------+-------------+--------------------------------------------------------------+------------------------------+---------+---------------------------+-------+----------+----------------------------------------------------------------------------+
My question is why isn't the query using the index "status"
When I force it to use the index "status" - it takes 0.2s, if I don't it takes 2.5s
Any help much appreciated. regards.
u have several indexes that overlap the fields they index.
For example
pictures_cnt and status
The optimizer "sees" your query as closer to use the fields of the pictures_cnt rather than the status index. If u will have more fields in the where that belongs to the status index, it might decide to use that index instead of picture_cnt
Do notice, it can't use both.
If u think the optimizer is wrong (as is the case on uncommon occasions) u can force it to use the index u want it to use. do benchmark it though.
After searching a bit, I found some recommendations regarding not using columns that allow null in indexes. I changed the columns in the status index to all be non-null, and now that index is being used.
As is usually the case, the simplest and cleanest solutions are the best.
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
I have a database containing around 5million rows and am having issues with a query taking a long time (over a minute). I was hoping this information would be enough for someone to give me some suggestions, if I need to post more information just let me know.
Thank you in advance for any advice
EXPLAIN SELECT count( * ) AS count
FROM vtiger_time
INNER JOIN vtiger_crmentity ON vtiger_crmentity.crmid = vtiger_time.timeid
INNER JOIN vtiger_crmentityrel ON ( vtiger_crmentityrel.relcrmid = vtiger_crmentity.crmid
OR vtiger_crmentityrel.crmid = vtiger_crmentity.crmid )
LEFT JOIN vtiger_users ON vtiger_users.id = vtiger_crmentity.smownerid
LEFT JOIN vtiger_groups ON vtiger_groups.groupid = vtiger_crmentity.smownerid
WHERE vtiger_crmentity.deleted =0
AND (
vtiger_crmentityrel.crmid =211294
OR vtiger_crmentityrel.relcrmid =211294
)
+----+-------------+---------------------+-------------+-------------------------------------------------+----------------+---------+--------------------------------+-------+------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+---------------------+-------------+-------------------------------------------------+----------------+---------+--------------------------------+-------+------------------------------------------+
| 1 | SIMPLE | vtiger_crmentityrel | index_merge | crmid,relcrmid | crmid,relcrmid | 4,4 | NULL | 5881 | Using union(crmid,relcrmid); Using where |
| 1 | SIMPLE | vtiger_crmentity | ref | PRIMARY,deleted,deleted_2,crmentity_multi_index | deleted_2 | 4 | const | 84424 | Using where; Using index |
| 1 | SIMPLE | vtiger_users | eq_ref | PRIMARY | PRIMARY | 4 | crm.vtiger_crmentity.smownerid | 1 | Using index |
| 1 | SIMPLE | vtiger_groups | eq_ref | PRIMARY | PRIMARY | 4 | crm.vtiger_crmentity.smownerid | 1 | Using index |
| 1 | SIMPLE | vtiger_time | eq_ref | PRIMARY,timeid | PRIMARY | 4 | crm.vtiger_crmentity.crmid | 1 | Using index |
+----+-------------+---------------------+-------------+-------------------------------------------------+----------------+---------+--------------------------------+-------+------------------------------------------+
Additionally these are my currently set indexes
+------------------+------------+----------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+------------------+------------+----------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| vtiger_crmentity | 0 | PRIMARY | 1 | crmid | A | 755968 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 0 | crmid | 1 | crmid | A | 755968 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | crmentity_IDX0 | 1 | smcreatorid | A | 15 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | crmentity_IDX1 | 1 | smownerid | A | 15 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | crmentity_IDX2 | 1 | modifiedby | A | 15 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | deleted | 1 | deleted | A | 15 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | deleted | 2 | smownerid | A | 15 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | smownerid | 1 | smownerid | A | 199 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | smownerid | 2 | deleted | A | 199 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | deleted_2 | 1 | deleted | A | 15 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | deleted_2 | 2 | smownerid | A | 15 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | smownerid_2 | 1 | smownerid | A | 385 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | smownerid_2 | 2 | deleted | A | 758 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | crm_ownerid_del_setype_idx | 1 | smownerid | A | 15 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | crm_ownerid_del_setype_idx | 2 | deleted | A | 15 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | crm_ownerid_del_setype_idx | 3 | setype | A | 613 | NULL | NULL | YES | BTREE | |
| vtiger_crmentity | 1 | crmentity_multi_index | 1 | crmid | A | 755968 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | crmentity_multi_index | 2 | smownerid | A | 755968 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | crmentity_multi_index | 3 | deleted | A | 755968 | NULL | NULL | | BTREE | |
+------------------+------------+----------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
+---------------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+---------------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| vtiger_crmentityrel | 1 | crmid | 1 | crmid | A | 223960 | NULL | NULL | | BTREE | |
| vtiger_crmentityrel | 1 | relcrmid | 1 | relcrmid | A | 12442 | NULL | NULL | | BTREE | |
+---------------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
+--------------+------------+---------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+--------------+------------+---------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+
| vtiger_users | 0 | PRIMARY | 1 | id | A | 39 | NULL | NULL | | BTREE | |
| vtiger_users | 1 | idx_user_name | 1 | user_name | A | 39 | NULL | NULL | YES | BTREE | |
| vtiger_users | 1 | user_password | 1 | user_password | A | 39 | NULL | NULL | YES | BTREE | |
+--------------+------------+---------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+
+---------------+------------+---------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+---------------+------------+---------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| vtiger_groups | 0 | PRIMARY | 1 | groupid | A | 5 | NULL | NULL | | BTREE | |
| vtiger_groups | 1 | groupname | 1 | groupname | A | 5 | NULL | NULL | YES | BTREE | |
| vtiger_groups | 1 | idx_groups_123group | 1 | groupname | A | 5 | NULL | NULL | YES | BTREE | |
+---------------+------------+---------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
+----------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+----------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| vtiger_time | 0 | PRIMARY | 1 | timeid | A | 591772 | NULL | NULL | | BTREE | |
| vtiger_time | 0 | timeid | 1 | timeid | A | 591772 | NULL | NULL | | BTREE | |
| vtiger_time | 1 | relatedto | 1 | relatedto | A | 1405 | NULL | NULL | YES | BTREE | |
| vtiger_time | 1 | date_start | 1 | date_start | A | 7129 | NULL | NULL | YES | BTREE | |
| vtiger_time | 1 | relatedto_2 | 1 | relatedto | A | 3269 | NULL | NULL | YES | BTREE | |
+----------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
+------------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+------------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| vtiger_timecf | 0 | PRIMARY | 1 | timeid | A | 591324 | NULL | NULL | | BTREE | |
| vtiger_timecf | 0 | timeid | 1 | timeid | A | 591324 | NULL | NULL | | BTREE | |
| vtiger_timecf | 1 | timeid_2 | 1 | timeid | A | 591324 | NULL | NULL | | BTREE | |
+------------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
After adding the following index
ALTER TABLE vtiger_crmentity ADD INDEX TMP_deletion_smownerid_crmid (smownerid, deleted,crmid);
The new explain:
+----+-------------+---------------------+-------------+--------------------------------------------------------------------------+----------------+---------+--------------------------------+-------+------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+---------------------+-------------+--------------------------------------------------------------------------+----------------+---------+--------------------------------+-------+------------------------------------------+
| 1 | SIMPLE | vtiger_crmentityrel | index_merge | crmid,relcrmid | crmid,relcrmid | 4,4 | NULL | 5891 | Using union(crmid,relcrmid); Using where |
| 1 | SIMPLE | vtiger_crmentity | ref | PRIMARY,crmid,deleted,deleted_2,crmentity_multi_index,_deletion_crmid | deleted | 4 | const | 84424 | Using where; Using index |
| 1 | SIMPLE | vtiger_users | eq_ref | PRIMARY | PRIMARY | 4 | crm.vtiger_crmentity.smownerid | 1 | Using index |
| 1 | SIMPLE | vtiger_groups | eq_ref | PRIMARY | PRIMARY | 4 | crm.vtiger_crmentity.smownerid | 1 | Using index |
| 1 | SIMPLE | vtiger_time | eq_ref | PRIMARY,timeid | PRIMARY | 4 | crm.vtiger_crmentity.crmid | 1 | Using index |
+----+-------------+---------------------+-------------+--------------------------------------------------------------------------+----------------+---------+--------------------------------+-------+------------------------------------------+
Rewriting the query would help, but since you cannot do that, you may benefit from a composite index on vtiger_crmentity.crmid and vtiger_crmentity.deleted. But since it isn't using one of the the index you currently have on vtiger_crmentity.crmid, it might not use that new one. If it doesn't, try adding vtiger_crmentity.crmid to the vtiger_crmentity.deleted index. Since it is already using the index on vtiger_crmentity.deleted, this will make the index a covering index so the query will not need to read from the table.
As I mentioned in my comment, there are a lot of duplicate indexes. The won't affect the query for this question, but they will slow down inserts and increase table size. The following indexes are redundant:
vtiger_crmentity.crmid
vtiger_crmentity.smownerid
vtiger_crmentity.smownerid_2
vtiger_crmentity.deleted_2
vtiger_groups.idx_groups_123group
vtiger_time.timeid
vtiger_time.relatedto_2
vtiger_timecf.timeid
vtiger_timcfe.timeid_2
You also shouldn't need the index on the user password column since you shouldn't be looking for users by password.
Try adding this index:
ALTER TABLE vtiger_crmentityrel ADD INDEX ix_crmentityrel_crmid_relcrmid (crmid,relcrmid);
This should allow the first row in the explain to be satisfied completely by the two indexes on the crmentityrel table and should save you a few thousand lookups.
All other parts of the execution plan are using covering indexes so if this doesn't help, I'm not sure there is much more you can do without restructuring the data or the query.