I am not sure why this query isn't using an index on the table world_cities. The indexes are the same and in the same order. The data types are of the same type and same length. The only difference is the key_name.
So why does it not use the key?
if I run the query I get: 318824 rows in set (2 min 51.30 sec)
Here is what I have done:
explain select * from zip_codes zc
inner join world_cities wc on
zc.city = wc.city
and zc.country = wc.country
and zc.region = wc.region;
+----+-------------+-------+------+---------------------+---------------------+---------+---------------------------------------------------------------+---------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+---------------------+---------------------+---------+---------------------------------------------------------------+---------+-------------+
| 1 | SIMPLE | wc | ALL | city_country_region | NULL | NULL | NULL | 3173958 | |
| 1 | SIMPLE | zc | ref | country_region_city | country_region_city | 165 | largedbapi.wc.city,largedbapi.wc.country,largedbapi.wc.region | 1 | Using where |
+----+-------------+-------+------+---------------------+---------------------+---------+---------------------------------------------------------------+---------+-------------+
mysql> show indexes from world_cities where Key_name like '%city%';
+--------------+------------+---------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+--------------+------------+---------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| world_cities | 1 | city_country_region | 1 | city | A | NULL | NULL | NULL | YES | BTREE | |
| world_cities | 1 | city_country_region | 2 | country | A | NULL | NULL | NULL | YES | BTREE | |
| world_cities | 1 | city_country_region | 3 | region | A | NULL | NULL | NULL | YES | BTREE | |
+--------------+------------+---------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
mysql> show indexes from zip_codes where Key_name like '%city%';
+-----------+------------+---------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+-----------+------------+---------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| zip_codes | 1 | country_region_city | 1 | city | A | 218424 | NULL | NULL | YES | BTREE | |
| zip_codes | 1 | country_region_city | 2 | country | A | 218424 | NULL | NULL | YES | BTREE | |
| zip_codes | 1 | country_region_city | 3 | region | A | 436849 | NULL | NULL | YES | BTREE | |
+-----------+------------+---------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
Your city_country_region key on world_cities has a NULL cardinality, while a non-NULL one is required for it to be use. Running analyze on it should fix it.
ANALYZE TABLE world_cities
Also have a look at this post for more info on NULL cardinality.
Related
Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 3 years ago.
Improve this question
I am using the following query which is very slow despite having indices, can someone help?
table a has close to 50k entries and table b about 2 million
i tried creating index on batchid in table 2 but it didnt help
select distinct x, y
from t1 a,t2 b
where b.batch_id = a.batch_id and b.sm_id = a.smid and b.status = 'new' and target_userid = *some value*
limit 10
here is the explain output
+----+-------------+-------+------+-----------------------------------+---------------+---------+------------------------------+---------+------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+-----------------------------------+---------------+---------+------------------------------+---------+------------------------------+
| 1 | SIMPLE | b | ALL | smreconhistory_sm_id_status_index | NULL | NULL | NULL | 2033644 | Using where; Using temporary |
| 1 | SIMPLE | a | ref | smid_batch_id,idx_batch_id | smid_batch_id | 16 | sync.b.sm_id,sync.b.batch_id | 1 | Using where; Distinct |
+----+-------------+-------+------+-----------------------------------+---------------+---------+------------------------------+---------+--------------------
index for table 1
+-----------------+------------+----------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-----------------+------------+----------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| SMReconJobEntry | 0 | PRIMARY | 1 | id | A | 49323 | NULL | NULL | | BTREE | | |
| SMReconJobEntry | 1 | smreconjobentry_ugid_index | 1 | ugid | A | 49323 | NULL | NULL | | BTREE | | |
| SMReconJobEntry | 1 | smid_batch_id | 1 | smid | A | 49323 | NULL | NULL | | BTREE | | |
| SMReconJobEntry | 1 | smid_batch_id | 2 | batch_id | A | 49323 | NULL | NULL | | BTREE | | |
| SMReconJobEntry | 1 | idx_batch_id | 1 | batch_id | A | 49323 | NULL | NULL | | BTREE | | |
+-----------------+------------+----------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+--
for table 2
+----------------+------------+-----------------------------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+----------------+------------+-----------------------------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| SMReconHistory | 0 | PRIMARY | 1 | id | A | 2033644 | NULL | NULL | | BTREE | | |
| SMReconHistory | 1 | smreconhistory_sm_id_status_index | 1 | sm_id | A | 18 | NULL | NULL | YES | BTREE | | |
| SMReconHistory | 1 | smreconhistory_sm_id_status_index | 2 | status | A | 18 | NULL | NULL | | BTREE | | |
| SMReconHistory | 1 | gmailunit_id_foreign | 1 | gmailunit_id | A | 2033644 | NULL | NULL | YES | BTREE | | |
+----------------+------------+-----------------------------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
Do not use find_in_set, replace function, it will calculate every time, I would suggest to use IN with static values.
x IN ('a', 'b', 'c')
I am working on large tables of my database (mysql). Some of my queries are running more than 5 minutes. Here is an example of my query that is running slow:
select b.DESCRIPTION collateral_type, a.description brand, a.year,
a.model, a.plate_number, d.description fuel, a.chassis_number,
a.engine_number, c.description as color, insurance_name
from lms_loan_application_collateral a inner join
lms_collateral_type b
on a.COLLATERAL_TYPE_ID = b.id left join
lms_color c
on a.color_id = c.id left join
lms_fuel_type d
on a.fuel_type_id = d.id inner join
lms_loan_application e
on e.id = a.loan_application_id inner join
lms_dlr_dtl f
on e.code = f.lano inner join
lms_loanapp_dtl g
on f.lano = g.lano
where b.description in ('Motorcycle', 'Automotive', 'Heavy Equipment');
here is a mysql explain of the query
+----+-------------+-------+-------+--------------------------------------------------------+-------------------------+---------+------------+-------+----------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+--------------------------------------------------------+-------------------------+---------+------------+-------+----------------------------------------------------+
| 1 | SIMPLE | g | index | lms_loanapp_dtl_lano | lms_loanapp_dtl_lano | 23 | NULL | 23432 | Using where; Using index |
| 1 | SIMPLE | f | ref | lano | lano | 23 | new.g.LANO | 1 | Using index |
| 1 | SIMPLE | b | ALL | lms_collateral_type_description,lms_collateral_type_id | NULL | NULL | NULL | 11 | Using where; Using join buffer (Block Nested Loop) |
| 1 | SIMPLE | a | ref | collateral_type_id | collateral_type_id | 5 | new.b.ID | 4067 | Using index condition |
| 1 | SIMPLE | e | ref | lms_loan_application_id | lms_loan_application_id | 153 | func | 1 | Using index condition; Using where |
| 1 | SIMPLE | c | ALL | LMS_color_id | NULL | NULL | NULL | 20 | Range checked for each record (index map: 0x1) |
| 1 | SIMPLE | d | ALL | LMS_fuel_type_id | NULL | NULL | NULL | 4 | Using where; Using join buffer (Block Nested Loop) |
+----+-------------+-------+-------+--------------------------------------------------------+-------------------------+---------+------------+-------+----------------------------------------------------+
under the key column, there are NULL values. I am not sure if mysql uses the index keys I have created.
Below is the list of indexes I created for the involved tables:
SHOW INDEX FROM lms_loan_application_collateral FROM new;
+---------------------------------+------------+---------------------+--------------+---------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+---------------------------------+------------+---------------------+--------------+---------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| lms_loan_application_collateral | 1 | color_id | 1 | color_id | A | 36 | NULL | NULL | YES | BTREE | | |
| lms_loan_application_collateral | 1 | fuel_type_id | 1 | fuel_type_id | A | 6 | NULL | NULL | YES | BTREE | | |
| lms_loan_application_collateral | 1 | loan_application_id | 1 | loan_application_id | A | 89493 | NULL | NULL | YES | BTREE | | |
| lms_loan_application_collateral | 1 | collateral_type_id | 1 | collateral_type_id | A | 22 | NULL | NULL | YES | BTREE | | |
+---------------------------------+------------+---------------------+--------------+---------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
SHOW INDEX FROM lms_collateral_type FROM new;
+---------------------+------------+---------------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+---------------------+------------+---------------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| lms_collateral_type | 1 | lms_collateral_type_description | 1 | DESCRIPTION | A | 11 | NULL | NULL | YES | BTREE | | |
| lms_collateral_type | 1 | lms_collateral_type_id | 1 | ID | A | 11 | NULL | NULL | YES | BTREE | | |
+---------------------+------------+---------------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
SHOW INDEX FROM lms_color FROM new;
+-----------+------------+--------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-----------+------------+--------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| lms_color | 1 | LMS_color_id | 1 | id | A | 20 | NULL | NULL | | BTREE | | |
+-----------+------------+--------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
SHOW INDEX FROM lms_fuel_type FROM new;
+---------------+------------+------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+---------------+------------+------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| lms_fuel_type | 1 | LMS_fuel_type_id | 1 | id | A | 4 | NULL | NULL | YES | BTREE | | |
+---------------+------------+------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
SHOW INDEX FROM lms_loan_application FROM new;
+----------------------+------------+-------------------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+----------------------+------------+-------------------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| lms_loan_application | 1 | lmspis_id | 1 | PIS_ID | A | 1878 | NULL | NULL | YES | BTREE | | |
| lms_loan_application | 1 | loan_type_id | 1 | LOAN_TYPE_ID | A | 6 | NULL | NULL | YES | BTREE | | |
| lms_loan_application | 1 | lms_loan_application_id | 1 | ID | A | 1878 | NULL | NULL | YES | BTREE | | |
+----------------------+------------+-------------------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
SHOW INDEX FROM lms_dlr_dtl FROM new;
+-------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| lms_dlr_dtl | 0 | PRIMARY | 1 | LDDID | A | 90066 | NULL | NULL | | BTREE | | |
| lms_dlr_dtl | 1 | lano | 1 | LANO | A | 90066 | NULL | NULL | YES | BTREE | | |
+-------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
SHOW INDEX FROM lms_loanapp_dtl FROM new;
+-----------------+------------+----------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-----------------+------------+----------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| lms_loanapp_dtl | 0 | PRIMARY | 1 | LLADID | A | 23432 | NULL | NULL | | BTREE | | |
| lms_loanapp_dtl | 1 | lms_loanapp_dtl_lano | 1 | LANO | A | 23432 | NULL | NULL | YES | BTREE | | |
+-----------------+------------+----------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
The tables which indicates the type as ALL does not utilise the indexes. Create indexes on id column on tables represents in c and d and one index on both description and id columns on b.
create index ix_id on lms_color(id);
create index ix_id on lms_fuel_type(id);
create index ix_description_id on lms_collateral_type(description,id);
MySQL is using ALL for your WHERE because of low cardinality. So MySQL thinks it is cheaper to just doing table scan over 11 entries than using binary search with index.
The real problem is that you do not filter using column whose index has higher cardinality (lms_loanapp_dtl_lano and collateral_type_id).
I've tried to create a two-column index on my table this way:
CREATE INDEX prod_hash_index ON components(producer_normalized, hash);
Which results in not one, but two indexes (when I do SHOW INDEXES FROM components;):
+------------+------------+-----------------+--------------+---------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+------------+------------+-----------------+--------------+---------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| components | 0 | PRIMARY | 1 | id | A | 1570231 | NULL | NULL | | BTREE | | |
| components | 1 | item_num | 1 | item_num | A | 1570231 | NULL | NULL | | BTREE | | |
| components | 1 | fs_index | 1 | first_symbol | A | 303 | NULL | NULL | | BTREE | | |
| components | 1 | prod_hash_index | 1 | producer_normalized | A | 18 | NULL | NULL | YES | BTREE | | |
| components | 1 | prod_hash_index | 2 | hash | A | 1570231 | NULL | NULL | | BTREE | | |
+------------+------------+-----------------+--------------+---------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
What does it mean? Did MySQL create an additional one-column index?
you read in result panel every row, with index name and column in the index.
About prod_has_index you have, first row about producer_normalized column in position one
and another row for has column in position (seq_in_index) 2.
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
I have a database containing around 5million rows and am having issues with a query taking a long time (over a minute). I was hoping this information would be enough for someone to give me some suggestions, if I need to post more information just let me know.
Thank you in advance for any advice
EXPLAIN SELECT count( * ) AS count
FROM vtiger_time
INNER JOIN vtiger_crmentity ON vtiger_crmentity.crmid = vtiger_time.timeid
INNER JOIN vtiger_crmentityrel ON ( vtiger_crmentityrel.relcrmid = vtiger_crmentity.crmid
OR vtiger_crmentityrel.crmid = vtiger_crmentity.crmid )
LEFT JOIN vtiger_users ON vtiger_users.id = vtiger_crmentity.smownerid
LEFT JOIN vtiger_groups ON vtiger_groups.groupid = vtiger_crmentity.smownerid
WHERE vtiger_crmentity.deleted =0
AND (
vtiger_crmentityrel.crmid =211294
OR vtiger_crmentityrel.relcrmid =211294
)
+----+-------------+---------------------+-------------+-------------------------------------------------+----------------+---------+--------------------------------+-------+------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+---------------------+-------------+-------------------------------------------------+----------------+---------+--------------------------------+-------+------------------------------------------+
| 1 | SIMPLE | vtiger_crmentityrel | index_merge | crmid,relcrmid | crmid,relcrmid | 4,4 | NULL | 5881 | Using union(crmid,relcrmid); Using where |
| 1 | SIMPLE | vtiger_crmentity | ref | PRIMARY,deleted,deleted_2,crmentity_multi_index | deleted_2 | 4 | const | 84424 | Using where; Using index |
| 1 | SIMPLE | vtiger_users | eq_ref | PRIMARY | PRIMARY | 4 | crm.vtiger_crmentity.smownerid | 1 | Using index |
| 1 | SIMPLE | vtiger_groups | eq_ref | PRIMARY | PRIMARY | 4 | crm.vtiger_crmentity.smownerid | 1 | Using index |
| 1 | SIMPLE | vtiger_time | eq_ref | PRIMARY,timeid | PRIMARY | 4 | crm.vtiger_crmentity.crmid | 1 | Using index |
+----+-------------+---------------------+-------------+-------------------------------------------------+----------------+---------+--------------------------------+-------+------------------------------------------+
Additionally these are my currently set indexes
+------------------+------------+----------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+------------------+------------+----------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| vtiger_crmentity | 0 | PRIMARY | 1 | crmid | A | 755968 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 0 | crmid | 1 | crmid | A | 755968 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | crmentity_IDX0 | 1 | smcreatorid | A | 15 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | crmentity_IDX1 | 1 | smownerid | A | 15 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | crmentity_IDX2 | 1 | modifiedby | A | 15 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | deleted | 1 | deleted | A | 15 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | deleted | 2 | smownerid | A | 15 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | smownerid | 1 | smownerid | A | 199 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | smownerid | 2 | deleted | A | 199 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | deleted_2 | 1 | deleted | A | 15 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | deleted_2 | 2 | smownerid | A | 15 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | smownerid_2 | 1 | smownerid | A | 385 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | smownerid_2 | 2 | deleted | A | 758 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | crm_ownerid_del_setype_idx | 1 | smownerid | A | 15 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | crm_ownerid_del_setype_idx | 2 | deleted | A | 15 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | crm_ownerid_del_setype_idx | 3 | setype | A | 613 | NULL | NULL | YES | BTREE | |
| vtiger_crmentity | 1 | crmentity_multi_index | 1 | crmid | A | 755968 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | crmentity_multi_index | 2 | smownerid | A | 755968 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | crmentity_multi_index | 3 | deleted | A | 755968 | NULL | NULL | | BTREE | |
+------------------+------------+----------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
+---------------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+---------------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| vtiger_crmentityrel | 1 | crmid | 1 | crmid | A | 223960 | NULL | NULL | | BTREE | |
| vtiger_crmentityrel | 1 | relcrmid | 1 | relcrmid | A | 12442 | NULL | NULL | | BTREE | |
+---------------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
+--------------+------------+---------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+--------------+------------+---------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+
| vtiger_users | 0 | PRIMARY | 1 | id | A | 39 | NULL | NULL | | BTREE | |
| vtiger_users | 1 | idx_user_name | 1 | user_name | A | 39 | NULL | NULL | YES | BTREE | |
| vtiger_users | 1 | user_password | 1 | user_password | A | 39 | NULL | NULL | YES | BTREE | |
+--------------+------------+---------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+
+---------------+------------+---------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+---------------+------------+---------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| vtiger_groups | 0 | PRIMARY | 1 | groupid | A | 5 | NULL | NULL | | BTREE | |
| vtiger_groups | 1 | groupname | 1 | groupname | A | 5 | NULL | NULL | YES | BTREE | |
| vtiger_groups | 1 | idx_groups_123group | 1 | groupname | A | 5 | NULL | NULL | YES | BTREE | |
+---------------+------------+---------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
+----------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+----------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| vtiger_time | 0 | PRIMARY | 1 | timeid | A | 591772 | NULL | NULL | | BTREE | |
| vtiger_time | 0 | timeid | 1 | timeid | A | 591772 | NULL | NULL | | BTREE | |
| vtiger_time | 1 | relatedto | 1 | relatedto | A | 1405 | NULL | NULL | YES | BTREE | |
| vtiger_time | 1 | date_start | 1 | date_start | A | 7129 | NULL | NULL | YES | BTREE | |
| vtiger_time | 1 | relatedto_2 | 1 | relatedto | A | 3269 | NULL | NULL | YES | BTREE | |
+----------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
+------------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+------------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| vtiger_timecf | 0 | PRIMARY | 1 | timeid | A | 591324 | NULL | NULL | | BTREE | |
| vtiger_timecf | 0 | timeid | 1 | timeid | A | 591324 | NULL | NULL | | BTREE | |
| vtiger_timecf | 1 | timeid_2 | 1 | timeid | A | 591324 | NULL | NULL | | BTREE | |
+------------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
After adding the following index
ALTER TABLE vtiger_crmentity ADD INDEX TMP_deletion_smownerid_crmid (smownerid, deleted,crmid);
The new explain:
+----+-------------+---------------------+-------------+--------------------------------------------------------------------------+----------------+---------+--------------------------------+-------+------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+---------------------+-------------+--------------------------------------------------------------------------+----------------+---------+--------------------------------+-------+------------------------------------------+
| 1 | SIMPLE | vtiger_crmentityrel | index_merge | crmid,relcrmid | crmid,relcrmid | 4,4 | NULL | 5891 | Using union(crmid,relcrmid); Using where |
| 1 | SIMPLE | vtiger_crmentity | ref | PRIMARY,crmid,deleted,deleted_2,crmentity_multi_index,_deletion_crmid | deleted | 4 | const | 84424 | Using where; Using index |
| 1 | SIMPLE | vtiger_users | eq_ref | PRIMARY | PRIMARY | 4 | crm.vtiger_crmentity.smownerid | 1 | Using index |
| 1 | SIMPLE | vtiger_groups | eq_ref | PRIMARY | PRIMARY | 4 | crm.vtiger_crmentity.smownerid | 1 | Using index |
| 1 | SIMPLE | vtiger_time | eq_ref | PRIMARY,timeid | PRIMARY | 4 | crm.vtiger_crmentity.crmid | 1 | Using index |
+----+-------------+---------------------+-------------+--------------------------------------------------------------------------+----------------+---------+--------------------------------+-------+------------------------------------------+
Rewriting the query would help, but since you cannot do that, you may benefit from a composite index on vtiger_crmentity.crmid and vtiger_crmentity.deleted. But since it isn't using one of the the index you currently have on vtiger_crmentity.crmid, it might not use that new one. If it doesn't, try adding vtiger_crmentity.crmid to the vtiger_crmentity.deleted index. Since it is already using the index on vtiger_crmentity.deleted, this will make the index a covering index so the query will not need to read from the table.
As I mentioned in my comment, there are a lot of duplicate indexes. The won't affect the query for this question, but they will slow down inserts and increase table size. The following indexes are redundant:
vtiger_crmentity.crmid
vtiger_crmentity.smownerid
vtiger_crmentity.smownerid_2
vtiger_crmentity.deleted_2
vtiger_groups.idx_groups_123group
vtiger_time.timeid
vtiger_time.relatedto_2
vtiger_timecf.timeid
vtiger_timcfe.timeid_2
You also shouldn't need the index on the user password column since you shouldn't be looking for users by password.
Try adding this index:
ALTER TABLE vtiger_crmentityrel ADD INDEX ix_crmentityrel_crmid_relcrmid (crmid,relcrmid);
This should allow the first row in the explain to be satisfied completely by the two indexes on the crmentityrel table and should save you a few thousand lookups.
All other parts of the execution plan are using covering indexes so if this doesn't help, I'm not sure there is much more you can do without restructuring the data or the query.
I am going to join two tables by using a single position in one table to the range (represented by two columns) in another table.
However, the performance is too slow, which is about 20 mins.
I have tried adding the index on the table or changing the query.
But the performance is still poor.
So, I am asking for optimization of the joining speed.
The following is the query to MySQL.
mysql> SELECT `inVar`.chrom, `inVar`.pos, `openChrom_K562`.score
-> FROM `inVar`
-> LEFT JOIN `openChrom_K562`
-> ON (
-> `inVar`.chrom=`openChrom_K562`.chrom AND
-> `inVar`.pos BETWEEN `openChrom_K562`.chromStart AND `openChrom_K562`.chromEnd
-> );
inVar and openChrom_K562 are the tables I used.
inVar stores the single position in each row.
openChrom_K562 stores the range information indicated by chromStart and chromEnd.
inVar contains 57902 rows and openChrom_K562 has 137373 rows respectively.
Fields on the tables.
mysql> DESCRIBE inVar;
+-------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------+-------------+------+-----+---------+-------+
| chrom | varchar(31) | NO | PRI | NULL | |
| pos | int(10) | NO | PRI | NULL | |
+-------+-------------+------+-----+---------+-------+
mysql> DESCRIBE openChrom_K562;
+------------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+------------+-------------+------+-----+---------+-------+
| chrom | varchar(31) | NO | MUL | NULL | |
| chromStart | int(10) | NO | MUL | NULL | |
| chromEnd | int(10) | NO | | NULL | |
| score | int(10) | NO | | NULL | |
+------------+-------------+------+-----+---------+-------+
Index built in the tables
mysql> SHOW INDEX FROM inVar;
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| inVar | 0 | PRIMARY | 1 | chrom | A | NULL | NULL | NULL | | BTREE | |
| inVar | 0 | PRIMARY | 2 | pos | A | 57902 | NULL | NULL | | BTREE | |
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
mysql> SHOW INDEX FROM openChrom_K562;
+----------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+----------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| openChrom_K562 | 1 | start_end | 1 | chromStart | A | 137373 | NULL | NULL | | BTREE | |
| openChrom_K562 | 1 | start_end | 2 | chromEnd | A | 137373 | NULL | NULL | | BTREE | |
| openChrom_K562 | 1 | chrom_only | 1 | chrom | A | 22 | NULL | NULL | | BTREE | |
| openChrom_K562 | 1 | chrom_start | 1 | chrom | A | 22 | NULL | NULL | | BTREE | |
| openChrom_K562 | 1 | chrom_start | 2 | chromStart | A | 137373 | NULL | NULL | | BTREE | |
| openChrom_K562 | 1 | chrom_end | 1 | chrom | A | 22 | NULL | NULL | | BTREE | |
| openChrom_K562 | 1 | chrom_end | 2 | chromEnd | A | 137373 | NULL | NULL | | BTREE | |
+----------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
Execution plan on MySQL
mysql> EXPLAIN SELECT `inVar`.chrom, `inVar`.pos, score FROM `inVar` LEFT JOIN `openChrom_K562` ON ( inVar.chrom=openChrom_K562.chrom AND `inVar`.pos BETWEEN chromStart AND chromEnd );
+----+-------------+----------------+-------+--------------------------------------------+------------+---------+-----------------+-------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+----------------+-------+--------------------------------------------+------------+---------+-----------------+-------+-------------+
| 1 | SIMPLE | inVar | index | NULL | PRIMARY | 37 | NULL | 57902 | Using index |
| 1 | SIMPLE | openChrom_K562 | ref | start_end,chrom_only,chrom_start,chrom_end | chrom_only | 33 | tmp.inVar.chrom | 5973 | |
+----+-------------+----------------+-------+--------------------------------------------+------------+---------+-----------------+-------+-------------+
It seems it only optimizes by looking chrom in two tables. Then do the brute-force comparing in the tables.
Is there any ways to do the further optimization like indexing on the position?
(It is my first time posting the question, sorry for the poor posting quality.)
chrom_only is likely to be a bad index selection for your join as you only have chrom 22 values.
If I have interpreted this right the query should be faster if using start_end
SELECT `inVar`.chrom, `inVar`.pos, `openChrom_K562`.score
FROM `inVar`
LEFT JOIN `openChrom_K562`
USE INDEX (`start_end`)
ON (
`inVar`.chrom=`openChrom_K562`.chrom AND
`inVar`.pos BETWEEN `openChrom_K562`.chromStart AND `openChrom_K562`.chromEnd
)