Why is MySQL not using my index ? Optimizing mysql select query - mysql

I am trying to optimize a MySQL select request:
SELECT * FROM `sales`
WHERE ((sales.private = false AND (sales.buyer_id IS NULL OR NOT sales.buyer_id=142)
AND (sales.merchand_id IS NULL OR NOT sales.merchand_id=142)
AND (sales.private_item = false) )
AND ((sales.buyer_id=32 OR sales.merchand_id=32)
AND (sales.admin=0 AND NOT sales.type IN ('book'))))
ORDER BY sales.created_at DESC, sales.id DESC LIMIT 0, 10;
The schema of the table is
mysql> SHOW columns from sales;
+------------------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------------------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| type | varchar(255) | YES | MUL | NULL | |
| buyer_id | int(11) | YES | MUL | NULL | |
| merchand_id | int(11) | YES | MUL | NULL | |
| private | tinyint(1) | YES | | 0 | |
| admin | tinyint(1) | YES | | 0 | |
| created_at | datetime | YES | | NULL | |
| updated_at | datetime | YES | | NULL | |
| country_id | int(11) | YES | MUL | 0 | |
| private_item | tinyint(1) | YES | | 0 | |
+------------------------+--------------+------+-----+---------+----------------+
The indexes are:
mysql> show indexes from sales;
+-----------------+------------+--------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+-----------------+------------+--------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| sales | 0 | PRIMARY | 1 | id | A | 286509 | NULL | NULL | | BTREE | |
| sales | 1 | index_sales_on_type | 1 | type | A | 123 | NULL | NULL | YES | BTREE | |
| sales | 1 | index_sales_on_buyer_id | 1 | buyer_id | A | 40929 | NULL | NULL | YES | BTREE | |
| sales | 1 | index_sales_on_merchand_id | 1 | merchand_id | A | 40929 | NULL | NULL | YES | BTREE | |
| sales | 1 | index_sales_on_country_id | 1 | country_id | A | 6 | NULL | NULL | YES | BTREE | |
| sales | 1 | index_sales_on_type_and_country_id | 1 | type | A | 151 | NULL | NULL | YES | BTREE | |
| sales | 1 | index_sales_on_type_and_country_id | 2 | country_id | A | 428 | NULL | NULL | YES | BTREE | |
| sales | 1 | index_sales_viewed | 1 | buyer_id | A | 35813 | NULL | NULL | YES | BTREE | |
| sales | 1 | index_sales_viewed | 2 | merchand_id | A | 286509 | NULL | NULL | YES | BTREE | |
| sales | 1 | index_sales_viewed | 3 | private_item| A | 285009 | NULL | NULL | YES | BTREE | |
| sales | 1 | index_sales_viewed | 4 | admin | A | 285009 | NULL | NULL | YES | BTREE | |
| sales | 1 | index_sales_viewed | 5 | type | A | 285009 | NULL | NULL | YES | BTREE | |
| sales | 1 | index_sales_viewed | 6 | private | A | 285009 | NULL | NULL | YES | BTREE | |
| sales | 1 | index_sales_viewed | 7 | created_at | A | 285009 | NULL | NULL | YES | BTREE | |
+-------+------------+------------------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
When doing the query it uses the index_sales_on_type_and_country_id even if there is no country_id in the query...
The query takes 2.5 seconds with this index.
But when I use USE INDEX(index_sales_viewed) it goes down to 0.2 seconds.
Here is the EXPLAIN of the query:
+----+-------------+-----------------+------+----------------------------------------------+------+---------+------+--------+---------------------
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-----------------+------+----------------------------------------------+------+---------+------+--------+---------------------
| 1 | SIMPLE | sales | range | see bellow | index_sales_on_type_and_country_id | 258 | NULL | 208725 | Using where; Using filesort |
+----+-------------+-----------------+------+----------------------------------------------+------+---------+------+--------+---------------------
the possible keys are :
index_sales_on_type,
index_sales_on_buyer_id,
index_sales_on_merchand_id,
index_sales_on_type_and_country_id,
index_sales_public_recent_activity
Why doesn't MySQL use index_sales_viewed by default? Could there be a better index?
Thank you!

This is wrong use on NULL, please change all the column used in the index to be NOT NULL
refer this When to use NULL in MySQL tables
official documentation
If this column is NULL, there are no relevant indexes. In this case, you may be able to improve the performance of your query by examining the WHERE clause to check whether it refers to some column or columns that would be suitable for indexing. If so, create an appropriate index and check the query with EXPLAIN again
Mysql chosen index index_sales_on_type_and_country_id because you are not compare with NULL value

Related

MySQL performance issue comparing 2 DateTime fields on the same table

My feed_listingjob has 2 datetime fields:
+------------+-------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------+-------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| data | longtext | NO | | NULL | |
| meta_data | longtext | NO | | NULL | |
| state | varchar(25) | NO | | NULL | |
| error | longtext | NO | | NULL | |
| job_id | int(11) | NO | MUL | NULL | |
| created_at | datetime(6) | NO | MUL | NULL | |
| updated_at | datetime(6) | NO | MUL | NULL | |
| es_sync_at | datetime(6) | YES | MUL | NULL | |
+------------+-------------+------+-----+---------+----------------+
updated_at and es_sync_at both are indexed individually as below:
mysql> show indexes from feed_listingjob;
+-----------------+------------+--------------------------------------------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-----------------+------------+--------------------------------------------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| feed_listingjob | 0 | PRIMARY | 1 | id | A | 64534 | NULL | NULL | | BTREE | | |
| feed_listingjob | 1 | feed_listingjob_job_id_4c3b1b514481f269_fk_feed_importjob_id | 1 | job_id | A | 2081 | NULL | NULL | | BTREE | | |
| feed_listingjob | 1 | feed_listingjob_fde81f11 | 1 | created_at | A | 64534 | NULL | NULL | | BTREE | | |
| feed_listingjob | 1 | feed_listingjob_afd1a1a8 | 1 | updated_at | A | 64534 | NULL | NULL | | BTREE | | |
| feed_listingjob | 1 | feed_listingjob_381895a2 | 1 | es_sync_at | A | 2 | NULL | NULL | YES | BTREE | | |
+-----------------+------------+--------------------------------------------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
5 rows in set (0.00 sec)
And my query is not using the indexes:
mysql> explain SELECT `feed_listingjob`.`id` FROM `feed_listingjob` WHERE `feed_listingjob`.`es_sync_at` < `feed_listingjob`.`updated_at` LIMIT 10;
+----+-------------+-----------------+------+---------------+------+---------+------+-------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-----------------+------+---------------+------+---------+------+-------+-------------+
| 1 | SIMPLE | feed_listingjob | ALL | NULL | NULL | NULL | NULL | 53534 | Using where |
+----+-------------+-----------------+------+---------------+------+---------+------+-------+-------------+
1 row in set (0.01 sec)
Can someone please tell me why and how to optimise this query?
The index can't be used here because feed_listingjob.updated_at is not a constant. Which means every row would have to be examined to check the condition. One way to use the index would be to have another column which stores the difference between es_sync_at and updated_at timestamps. If you store it as difference = (es_sync_at - updated_at timestamps) then the query becomes (difference < 0).

why is mysql not using the right index?

I have a problem where the correct index is no being used.
I have the following indexes on an innodb table (about 500,000 rows) :
+-------------+------------+------------------------+--------------+------------------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+-------------+------------+------------------------+--------------+------------------------+-----------+-------------+----------+--------+------+------------+---------+
| osdate_user | 0 | PRIMARY | 1 | id | A | 419700 | NULL | NULL | | BTREE | |
| osdate_user | 0 | email | 1 | email | A | 419700 | NULL | NULL | | BTREE | |
| osdate_user | 0 | username | 1 | username | A | 419700 | NULL | NULL | YES | BTREE | |
| osdate_user | 1 | lastvisit | 1 | lastvisit | A | 419700 | NULL | NULL | | BTREE | |
| osdate_user | 1 | active | 1 | active | A | 8 | NULL | NULL | YES | BTREE | |
| osdate_user | 1 | gender | 1 | gender | A | 88 | NULL | NULL | YES | BTREE | |
| osdate_user | 1 | regdate | 1 | regdate | A | 419700 | NULL | NULL | | BTREE | |
| osdate_user | 1 | lastupdate | 1 | lastupdate | A | 419700 | NULL | NULL | YES | BTREE | |
| osdate_user | 1 | password | 1 | password | A | 419700 | NULL | NULL | | BTREE | |
| osdate_user | 1 | age | 1 | age | A | 190 | NULL | NULL | YES | BTREE | |
| osdate_user | 1 | is_new | 1 | is_new | A | 8 | NULL | NULL | YES | BTREE | |
| osdate_user | 1 | private_photos | 1 | private_photos | A | 8 | NULL | NULL | | BTREE | |
| osdate_user | 1 | pictures_cnt | 1 | pictures_cnt | A | 10 | NULL | NULL | YES | BTREE | |
| osdate_user | 1 | pictures_cnt | 2 | private_photos | A | 10 | NULL | NULL | | BTREE | |
| osdate_user | 1 | status | 1 | status | A | 19 | NULL | NULL | | BTREE | |
| osdate_user | 1 | status | 2 | active | A | 19 | NULL | NULL | YES | BTREE | |
| osdate_user | 1 | status | 3 | gender | A | 19 | NULL | NULL | YES | BTREE | |
| osdate_user | 1 | status | 4 | age | A | 19 | NULL | NULL | YES | BTREE | |
| osdate_user | 1 | status | 5 | country | A | 7630 | NULL | NULL | YES | BTREE | |
| osdate_user | 1 | status | 6 | city | A | 46633 | NULL | NULL | YES | BTREE | |
| osdate_user | 1 | status | 7 | pictures_cnt | A | 83940 | NULL | NULL | YES | BTREE | |
| osdate_user | 1 | status | 8 | private_photos | A | 139900 | NULL | NULL | | BTREE | |
| osdate_user | 1 | status | 9 | lang | A | 209850 | NULL | NULL | | BTREE | |
| osdate_user | 1 | status | 10 | is_new | A | 209850 | NULL | NULL | YES | BTREE | |
+-------------+------------+------------------------+--------------+------------------------+-----------+-------------+----------+--------+------+------------+---------+
This query :
EXPLAIN EXTENDED SELECT user.id, user.active
FROM osdate_user user
WHERE user.active =1
AND user.status = 'active'
AND user.gender = 'M'
AND user.age
BETWEEN 19
AND 35
AND user.pictures_cnt >0
AND user.private_photos = '0'
ORDER BY user.lastvisit DESC
LIMIT 0 , 24
Displays the following plan :
+----+-------------+---------+-------------+--------------------------------------------------------------+------------------------------+---------+---------------------------+-------+----------+----------------------------------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+---------+-------------+--------------------------------------------------------------+------------------------------+---------+---------------------------+-------+----------+----------------------------------------------------------------------------+
| 1 | SIMPLE | user | index_merge | PRIMARY,active,gender,age,private_photos,pictures_cnt,status | gender,private_photos,active | 4,1,2 | NULL | 28204 | 100.00 | Using intersect(gender,private_photos,active); Using where; Using filesort |
| 1 | SIMPLE | userext | eq_ref | userid | userid | 4 | db_name.user.id | 1 | 100.00 | Using index |
+----+-------------+---------+-------------+--------------------------------------------------------------+------------------------------+---------+---------------------------+-------+----------+----------------------------------------------------------------------------+
My question is why isn't the query using the index "status"
When I force it to use the index "status" - it takes 0.2s, if I don't it takes 2.5s
Any help much appreciated. regards.
u have several indexes that overlap the fields they index.
For example
pictures_cnt and status
The optimizer "sees" your query as closer to use the fields of the pictures_cnt rather than the status index. If u will have more fields in the where that belongs to the status index, it might decide to use that index instead of picture_cnt
Do notice, it can't use both.
If u think the optimizer is wrong (as is the case on uncommon occasions) u can force it to use the index u want it to use. do benchmark it though.
After searching a bit, I found some recommendations regarding not using columns that allow null in indexes. I changed the columns in the status index to all be non-null, and now that index is being used.
As is usually the case, the simplest and cleanest solutions are the best.

MYSQL query is slow using group by with order by

I have a query that is using a group by and an order by, but it's very slow! I need some help to get the index correct. This is the query I'm running:
select *
from puresen_mv_shop.cache_deals
where feature_ids REGEXP 'i,t,d'
and phone_cost > 100.00
group by handset_numeric_id
order by popularity
LIMIT 0,10;
explain:
1 SIMPLE cache_deals index popularity 5 10 6635320.00 Using where; Using temporary
This runs in about 3 seconds, but I need it under a second. There are over 600k rows in the table. Here is the table structure:
mysql> describe cache_deals
-> ;
+-------------------------+---------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------------------+---------------+------+-----+---------+-------+
| deal_id | varchar(100) | NO | | NULL | |
| deal_id_replication | varchar(150) | NO | PRI | | |
| handset_numeric_id | int(10) | YES | MUL | NULL | |
| handset_url | varchar(100) | YES | | NULL | |
| image_url | varchar(50) | YES | MUL | NULL | |
| handset_id | varchar(50) | YES | | NULL | |
| phone_cost | decimal(10,2) | YES | | NULL | |
| tariff_numeric_id | int(10) | YES | MUL | NULL | |
| tariff_id | varchar(100) | YES | MUL | NULL | |
| tariff_name | varchar(100) | YES | | NULL | |
| network_name | varchar(20) | YES | | NULL | |
| network_numeric_id | int(5) | YES | MUL | NULL | |
| term | int(5) | YES | MUL | NULL | |
| minutes | int(5) | YES | MUL | NULL | |
| texts | int(5) | YES | MUL | NULL | |
| data | int(5) | YES | MUL | NULL | |
| org_line_rental | decimal(10,2) | YES | MUL | NULL | |
| effective_monthly_cost | decimal(10,2) | YES | MUL | NULL | |
| free_gift_id | int(10) | YES | MUL | NULL | |
| free_gift_name | varchar(50) | YES | | NULL | |
| cashback | int(5) | YES | MUL | NULL | |
| free_lr | int(5) | YES | MUL | NULL | |
| half_lr | int(5) | YES | MUL | NULL | |
| clearance_flag | int(5) | YES | MUL | NULL | |
| manufacturer_numeric_id | int(5) | YES | MUL | NULL | |
| manufacturer_name | varchar(50) | YES | | NULL | |
| full_handset_name | varchar(100) | YES | | NULL | |
| popularity | int(20) | YES | MUL | NULL | |
| handset_colour | varchar(50) | YES | MUL | NULL | |
| feature_ids | varchar(100) | YES | | NULL | |
| operating_system | varchar(30) | YES | MUL | NULL | |
+-------------------------+---------------+------+-----+---------+-------+
Here are the indexes I've currently got on this table:
mysql> show index from cache_deals;
+-------------+------------+-------------------------+--------------+-------------------------+-----------+-------------+----------+--------+------+------------
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type
+-------------+------------+-------------------------+--------------+-------------------------+-----------+-------------+----------+--------+------+------------
| cache_deals | 0 | PRIMARY | 1 | deal_id_replication | A | 663532 | NULL | NULL | | BTREE
| | |
| cache_deals | 1 | handset_numeric_id | 1 | handset_numeric_id | A | 759 | NULL | NULL | YES | BTREE
| | |
| cache_deals | 1 | network_numeric_id | 1 | network_numeric_id | A | 8 | NULL | NULL | YES | BTREE
| | |
| cache_deals | 1 | tariff_numeric_id | 1 | tariff_numeric_id | A | 1091 | NULL | NULL | YES | BTREE
| | |
| cache_deals | 1 | manufacturer_numeric_id | 1 | manufacturer_numeric_id | A | 23 | NULL | NULL | YES | BTREE
| | |
| cache_deals | 1 | operating_system | 1 | operating_system | A | 42 | NULL | NULL | YES | BTREE
| | |
| cache_deals | 1 | image_url | 1 | image_url | A | 755 | NULL | NULL | YES | BTREE
| | |
| cache_deals | 1 | tariff_id | 1 | tariff_id | A | 1091 | NULL | NULL | YES | BTREE
| | |
| cache_deals | 1 | term | 1 | term | A | 7 | NULL | NULL | YES | BTREE
| | |
| cache_deals | 1 | minutes | 1 | minutes | A | 26 | NULL | NULL | YES | BTREE
| | |
| cache_deals | 1 | texts | 1 | texts | A | 14 | NULL | NULL | YES | BTREE
| | |
| cache_deals | 1 | data | 1 | data | A | 14 | NULL | NULL | YES | BTREE
| | |
| cache_deals | 1 | org_line_rental | 1 | org_line_rental | A | 128 | NULL | NULL | YES | BTREE
| | |
| cache_deals | 1 | effective_monthly_cost | 1 | effective_monthly_cost | A | 2147 | NULL | NULL | YES | BTREE
| | |
| cache_deals | 1 | free_gift_id | 1 | free_gift_id | A | 105 | NULL | NULL | YES | BTREE
| | |
| cache_deals | 1 | cashback | 1 | cashback | A | 2 | NULL | NULL | YES | BTREE
| | |
| cache_deals | 1 | free_lr | 1 | free_lr | A | 2 | NULL | NULL | YES | BTREE
| | |
| cache_deals | 1 | half_lr | 1 | half_lr | A | 2 | NULL | NULL | YES | BTREE
| | |
| cache_deals | 1 | clearance_flag | 1 | clearance_flag | A | 2 | NULL | NULL | YES | BTREE
| | |
| cache_deals | 1 | handset_colour | 1 | handset_colour | A | 17 | NULL | NULL | YES | BTREE
| | |
| cache_deals | 1 | popularity | 1 | popularity | A | 718 | NULL | NULL | YES | BTREE
| | |
+-------------+------------+-------------------------+--------------+-------------------------+-----------+-------------+----------+--------+------+------------
Am I indexing this table correctly?
Thanks for you help.
That REGEXP is killing the potential performance since it can't use an index. Can that part of the query be written some other way or possibly normalized to another table? If you can, that will probably help. Other than that, a composite index on phone_cost and handset_numeric_id should help. Also adding popularity to that index might help, but MySQL is already unhappy with the query since you're not really supposed to be able to work with columns that aren't in the GROUP BY clause.
As a side note, the data types of the columns could be more efficient. int(5) works the same as MEDIUMINT due to the limit on the INT except that MEDIUMINT is one byte smaller. Also, int(20) doesn't make sense since the INT type can't be that large.
select [name the columns you actually want returned]
from puresen_mv_shop.cache_deals
where feature_ids = 'i,t,d'
and phone_cost > 100.00
order
by popularity
LIMIT 0,10;
Index features_id or (features_id,phone_cost)

adding an additional item in SQL statement causes slow performance

I have a sql statement that looks like this:
SELECT colID
FROM tableName
WHERE ColDateStart <='$lowerDate'
AND ColDateStart>='$upperDate'
AND ColVcamID='$id1'
AND ColVlviID='$id2'
AND ColSomeID='$id3';
All the columns in the WHERE statment are indexed columns.
When I run this it takes over a second. However when I run this without other Id3, the performance is considerably improved (0.03 seconds).
When I run explain, with otherId3, it uses an index merge using otherId1 and otherId3. However when I remove the otherId3, it uses the single index of otherId2.
Why does adding otherId3 make an impact on the performance?
Table Structure:
+----------------------+-------------+------+-----+---------------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+----------------------+-------------+------+-----+---------------------+----------------+
| ColID | int(11) | NO | PRI | NULL | auto_increment |
| ColCustID | int(11) | NO | MUL | 0 | |
| ColCarrID | int(11) | NO | MUL | NULL | |
| ColTariID | int(11) | NO | MUL | 0 | |
| ColCarrierRef | varchar(30) | NO | MUL | | |
| ColNumbID | int(11) | NO | MUL | 0 | |
| ColVlviID | int(11) | NO | MUL | NULL | |
| ColVcamID | int(11) | NO | MUL | NULL | |
| ColSomeID | int(11) | NO | MUL | NULL | |
| ColVlnsID | int(11) | NO | MUL | NULL | |
| ColNGNumber | varchar(12) | NO | | | |
| ColOrigNumber | varchar(16) | NO | MUL | NULL | |
| ColCLIRestrictedFlag | int(2) | NO | | NULL | |
| ColOrigLocality | varchar(11) | NO | MUL | | |
| ColOrigAreaCode | varchar(11) | NO | MUL | | |
| ColTermNumber | varchar(16) | NO | MUL | NULL | |
| ColBatchNumber | varchar(10) | NO | | | |
| ColDateStart | date | NO | MUL | 0000-00-00 | |
| ColDateClear | date | NO | | 0000-00-00 | |
| ColTimeStart | time | NO | | 00:00:00 | |
| ColTimeClear | time | NO | | 00:00:00 | |
| ColCallLength | time | NO | | 00:00:00 | |
| ColRingLength | time | NO | | 00:00:00 | |
| ColEffectiveFlag | smallint(1) | NO | MUL | NULL | |
| ColUnansweredFlag | smallint(1) | NO | MUL | NULL | |
| ColEngagedFlag | smallint(1) | NO | | NULL | |
| ColRecID | int(11) | NO | MUL | NULL | |
| ColCreatedUserID | int(11) | NO | | 0 | |
| ColCreatedDatetime | datetime | NO | MUL | 0000-00-00 00:00:00 | |
| ColDirection | int(1) | NO | MUL | NULL | |
+----------------------+-------------+------+-----+---------------------+----------------+
Indexes
+-------+------------+-------------------------------+--------------+---------------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+-------+------------+-------------------------------+--------------+---------------------+-----------+-------------+----------+--------+------+------------+---------+
| tableName | 0 | PRIMARY | 1 | ColID | A | 18031283 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_ColCustID | 1 | ColCustID | A | 1339 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_ColNumbID | 1 | ColNumbID | A | 24366 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colOrigNumber | 1 | colOrigNumber | A | 4507820 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colOrigLocality | 1 | colOrigLocality | A | 36873 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colOrigAreaCode | 1 | colOrigAreaCode | A | 696 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colTermNumber | 1 | colTermNumber | A | 137643 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colDateStart | 1 | colDateStart | A | 3639 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colEffectiveFlag | 1 | colEffectiveFlag | A | 2 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colUnansweredFlag | 1 | colUnansweredFlag | A | 2 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colEngagedFlag | 1 | colUnansweredFlag | A | 2 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colTariID | 1 | colTariID | A | 91 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_CustID_DateStart | 1 | colCustID | A | 1339 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_CustID_DateStart | 2 | colDateStart | A | 693510 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_NumbID_DateStart | 1 | colNumbID | A | 24366 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_NumbID_DateStart | 2 | colDateStart | A | 4507820 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colRecID | 1 | colRecID | A | 214658 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colCarrierRef | 1 | colCarrierRef | A | 6010427 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colCustID_colTermNumber | 1 | colCustID | A | 1339 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colCustID_colTermNumber | 2 | colTermNumber | A | 143105 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colCreatedDatetime | 1 | colCreatedDatetime | A | 474507 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colDirection | 1 | colDirection | A | 2 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colVlviID | 1 | colVlviID | A | 4133 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colSomeID | 1 | colSomeID | A | 10 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colVcamID | 1 | colVcamID | A | 7 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colVlnsID | 1 | colVlnsID | A | 18 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colCarrID | 1 | colCarrID | A | 4 | NULL | NULL | | BTREE | |
+-------+------------+-------------------------------+--------------+---------------------+-----------+-------------+----------+--------+------+------------+---------+
First up, you have far too may indexes. Most of those indexes are likely pointless. If you aren't using a specific index for WHERE criteria, JOINs, or ORDERing, then remove it as it just slows things down.
Next up, for your query you specify 3 columns that are always in the query comprising 1x DATE and 2x INT columns. The DATE column should be first as a date range is pretty fast on an index, and then the two INTs. This gives a starting point of the following 3-column index
CREATE INDEX searchIndex
ON tableName (ColDateStart,ColVlviID,ColVcamID)
USING BTREE;
More information on that available here : http://dev.mysql.com/doc/refman/5.0/en/multiple-column-indexes.html
I ordered the columns that way intentionally. ColDateStart for the date range filtering, followed by the other columns in decreasing order of cardinality. I selected ColVlviID as a second column as it has a cardinality of 4133 compared to ColVcamID having a cardinality of 7. This will allow MySQL to more efficiently reduce the matching rows.
Now, assuming colSomeID is the last column, I might instead do the following
CREATE INDEX searchIndex_someID
ON tableName (ColDateStart,ColVlviID,ColVcamID,ColSomeID)
USING BTREE;
This 3-column index will help MySQL get down to the applicable dataset before checking for that last ID. Optioanlly you could add a 4th column to that index at the end, if you are commonly filtering in another particular INT column.
As an aside, you may want to consider the following instead of the dateCol criteria
SELECT colID
FROM tableName
WHERE ColDateStart BETWEEN DATE('$lowerDate') AND DATE('$upperDate')
AND ColVcamID=$id1
AND ColVlviID=$id2
AND ColSomeID=$id3
The above all of course assumes that you are sanitising the variables before executing the query. I have removed the quotes from the $idx variables, as they should be numeric and therefore do not need to be entered as strings.
SELECT id
FROM tableName FORCE INDEX(`idx_otherId2`)
WHERE dateCol <='$lowerDate'
AND dateCol>='$upperDate'
AND otherId1='$id1'
AND otherId2='$id2'
AND otherId3='$id3';

MySQL Queries with a range on the Join (speed issue)

I'm trying to run the following query:
SELECT formatted_journeys.*, MAX(speed)
FROM formatted_journeys
JOIN tracker.g_log
ON imeiid = vehicle
AND g_logid BETWEEN start_g_log AND end_g_log
GROUP BY id
however, it seems to be extremely slow. Here's the explain for that query.
+----+-------------+--------------------+------+-------------------------------+-------+---------+-----------------------------------+------+----------+---------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+--------------------+------+-------------------------------+-------+---------+-----------------------------------+------+----------+---------------------------------+
| 1 | SIMPLE | formatted_journeys | ALL | vehicle,start_g_log,end_g_log | NULL | NULL | NULL | 824 | 100.00 | Using temporary; Using filesort |
| 1 | SIMPLE | g_log | ref | PRIMARY,Dupes | Dupes | 4 | motrak.formatted_journeys.vehicle | 1985 | 100.00 | Using where |
+----+-------------+--------------------+------+-------------------------------+-------+---------+-----------------------------------+------+----------+---------------------------------+
2 rows in set, 1 warning (0.02 sec)
The tables are as follows:
formatted_journeys:
+-----------------+------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------------+------------------+------+-----+---------+----------------+
| id | int(11) unsigned | NO | PRI | NULL | auto_increment |
| start_g_log | int(11) | YES | MUL | NULL | |
| end_g_log | int(11) | YES | MUL | NULL | |
| start_latitude | decimal(18,12) | YES | | NULL | |
| start_longitude | decimal(18,12) | YES | | NULL | |
| end_latitude | decimal(18,12) | YES | | NULL | |
| end_longitude | decimal(18,12) | YES | | NULL | |
| start_location | text | YES | | NULL | |
| end_location | text | YES | | NULL | |
| distance | decimal(10,5) | YES | | NULL | |
| start_date | datetime | YES | | NULL | |
| end_date | datetime | YES | | NULL | |
| vehicle | int(11) | YES | MUL | NULL | |
| private | bit(1) | NO | | b'0' | |
+-----------------+------------------+------+-----+---------+----------------+
Indices:
+--------------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+--------------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| formatted_journeys | 0 | PRIMARY | 1 | id | A | 830 | NULL | NULL | | BTREE | |
| formatted_journeys | 1 | vehicle | 1 | vehicle | A | 4 | NULL | NULL | YES | BTREE | |
| formatted_journeys | 1 | start_g_log | 1 | start_g_log | A | 830 | NULL | NULL | YES | BTREE | |
| formatted_journeys | 1 | end_g_log | 1 | end_g_log | A | 830 | NULL | NULL | YES | BTREE | |
+--------------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
tracker.g_log:
+-----------+------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------+------------------+------+-----+---------+----------------+
| g_logid | int(10) unsigned | NO | PRI | NULL | auto_increment |
| imeiid | int(10) unsigned | NO | MUL | NULL | |
| latitude | decimal(18,12) | YES | | NULL | |
| longitude | decimal(18,12) | YES | | NULL | |
| speed | int(4) | YES | MUL | NULL | |
| bearing | int(4) | YES | | NULL | |
| distance | decimal(10,5) | YES | | NULL | |
| eventcode | int(10) | YES | | NULL | |
| status | int(10) | YES | | NULL | |
| date | datetime | YES | | NULL | |
+-----------+------------------+------+-----+---------+----------------+
Indices:
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| g_log | 0 | PRIMARY | 1 | g_logid | A | 31760 | NULL | NULL | | BTREE | |
| g_log | 0 | Dupes | 1 | imeiid | A | 16 | NULL | NULL | | BTREE | |
| g_log | 0 | Dupes | 2 | date | A | 31760 | NULL | NULL | YES | BTREE | |
| g_log | 0 | Dupes | 3 | eventcode | A | 31760 | NULL | NULL | YES | BTREE | |
| g_log | 1 | speed | 1 | speed | A | 423 | NULL | NULL | YES | BTREE | |
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
Now I know that the filesort isn't a good thing but how would I get rid of it?
You will hardly get rid of Using temporary; Using filesort, but the query might be faster.
First, try to rewrite the query as:
SELECT *
FROM (
SELECT id, MAX(speed) as max_speed
FROM formatted_journeys
JOIN tracker.g_log
ON imeiid = vehicle
AND g_logid BETWEEN start_g_log AND end_g_log
GROUP BY id) as maxspeeds
JOIN formatted_journeys USING (id);
Then, you can try to force the query to use a coverage index, though that is not easy for the query.
First try: add a composite index (vehicle, start_g_log, end_g_log) and look in explain whether it is used (you should see "Using index" there)