here is my explain result.
+----+-------------+----------------+--------+-------------------------------------------------------+-------------+---------+-------------------------------------+------+----------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+----------------+--------+-------------------------------------------------------+-------------+---------+-------------------------------------+------+----------------------------------------------------+
| 1 | SIMPLE | client_address | ALL | client_id | NULL | NULL | NULL | 1619 | Using where; Using temporary; Using filesort |
| 1 | SIMPLE | clients | eq_ref | PRIMARY,contract_id,contract_id_2 | PRIMARY | 3 | defenselaw.client_address.client_id | 1 | Using where |
| 1 | SIMPLE | user_infos | eq_ref | user_id | user_id | 3 | defenselaw.clients.client_rep_id | 1 | Using where |
| 1 | SIMPLE | users | eq_ref | PRIMARY | PRIMARY | 2 | defenselaw.user_infos.user_id | 1 | Using where; Using index |
| 1 | SIMPLE | contracts | eq_ref | PRIMARY | PRIMARY | 3 | defenselaw.clients.contract_id | 1 | NULL |
| 1 | SIMPLE | documents | eq_ref | contract_id,contract_id_2,contract_id_3,contract_id_4 | contract_id | 3 | defenselaw.clients.contract_id | 1 | Using where |
| 1 | SIMPLE | client_info | ref | client_id,client_id_2,client_id_3,index5 | client_id | 3 | defenselaw.client_address.client_id | 1 | Using where |
| 1 | SIMPLE | location | ALL | PRIMARY | NULL | NULL | NULL | 1 | Using where; Using join buffer (Block Nested Loop) |
+----+-------------+----------------+--------+-------------------------------------------------------+-------------+---------+-------------------------------------+------+----------------------------------------------------+
here is the query.
SELECT
contracts.contract_id,
documents.documentKey,
documents.document_name,
documents.notify_accountant,
clients.client_first_name,
clients.client_middle_name,
clients.client_last_name,
clients.client_rep_id,
client_address.client_street,
client_address.client_apartment,
client_address.client_city,
client_address.client_state,
client_address.client_zip,
client_info.client_email,
client_info.client_home_phone,
client_info.client_work_phone,
client_info.client_cell_phone,
client_info.client_fax_number,
contracts.contract_agreement_date,
user_infos.user_first_name,
user_infos.user_last_name,
location.location
FROM
`contracts`
LEFT JOIN
`clients`
ON clients.contract_id = contracts.contract_id
INNER JOIN
`client_address`
ON client_address.client_id = clients.client_id
INNER JOIN
`client_info`
ON client_info.client_id = client_address.client_id
LEFT JOIN
`documents`
ON contracts.contract_id = documents.contract_id
LEFT JOIN
`user_infos`
ON user_infos.user_id = clients.client_rep_id
LEFT JOIN
`users` ON
users.user_id = user_infos.user_id
LEFT JOIN
`location`
ON location.location_id = user_infos.location_id
WHERE client_info.client_role = 'primary'
AND documents.archive = 0
ORDER BY
documents.created_on DESC
Could someone explain to me why my index on client_address isn't being used? I am lost here as I just don't know what to do. It's not that complicated of a query its just rather large. Yes, stackoverflow, I understand the post is mostly code. What more do you want from me? How much more can I say about this other than my query isn't using an index and its causing a really slow query here.
+----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| client_address | CREATE TABLE `client_address` (
`client_address_id` mediumint(12) NOT NULL AUTO_INCREMENT,
`client_id` mediumint(12) NOT NULL,
`client_street` varchar(40) COLLATE utf8_bin NOT NULL,
`client_apartment` varchar(20) COLLATE utf8_bin NOT NULL,
`client_city` varchar(80) COLLATE utf8_bin NOT NULL,
`client_state` varchar(80) COLLATE utf8_bin NOT NULL,
`client_zip` varchar(80) COLLATE utf8_bin NOT NULL,
PRIMARY KEY (`client_address_id`),
KEY `client_id` (`client_id`)
) ENGINE=InnoDB AUTO_INCREMENT=2167 DEFAULT CHARSET=utf8 COLLATE=utf8_bin |
+----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Related
I have a particular MariaDB Query that utilizes some joins and some IN ('...') conditions.
Generally it returns results in < 2sec on large data sets (~50M records), however when a very large number of options are presented in the IN condition, (Eg. 1000+ IN options) the query takes 5+ hours and the logic completely changes when running an ANALYZE on the queries.
Looking to understand why this is the case and suggestions on how I might be able to resolve the bottleneck. Presently thinking the simplest option may be to drop the IN condition completely and filter the results in PHP instead of SQL, as if the IN condition is dropped results are returned in <1s on same tables.
ANALYZE results from query where small IN set used.
+------+-------------+-------+-------+------------------------+----------+---------+--------------------+-------+---------+----------+------------+---------------------------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | r_rows | filtered | r_filtered | Extra |
+------+-------------+-------+-------+------------------------+----------+---------+--------------------+-------+---------+----------+------------+---------------------------------------------------------------------+
| 1 | SIMPLE | t1 | range | calldate,call_id | calldate | 7 | NULL | 13400 | 7162.00 | 100.00 | 33.01 | Using index condition; Using where; Using temporary; Using filesort |
| 1 | SIMPLE | t2 | ref | PRIMARY,called,call_id | call_id | 4 | crimson.t1.call_id | 1 | 3.21 | 100.00 | 35.24 | Using where |
| 1 | SIMPLE | d1 | ref | digits,leg_id | leg_id | 4 | crimson.t2.xid | 1 | 1.94 | 100.00 | 0.58 | Using index condition; Using where |
| 1 | SIMPLE | g1 | ref | call_id | call_id | 4 | crimson.t1.call_id | 1 | 3.00 | 100.00 | 0.00 | Using where |
+------+-------------+-------+-------+------------------------+----------+---------+--------------------+-------+---------+----------+------------+---------------------------------------------------------------------+
4 rows in set (1.154 sec)
ANALYZE results from the same query and conditions where 1200 IN set has been used.
+------+--------------+-------------+--------+------------------------+---------+---------+--------------------+------+---------+----------+------------+---------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | r_rows | filtered | r_filtered | Extra |
+------+--------------+-------------+--------+------------------------+---------+---------+--------------------+------+---------+----------+------------+---------------------------------+
| 1 | PRIMARY | <subquery2> | ALL | distinct_key | NULL | NULL | NULL | 1222 | 1222.00 | 100.00 | 100.00 | Using temporary; Using filesort |
| 1 | PRIMARY | d1 | ref | digits,leg_id | digits | 29 | tvc_0._col_1 | 5 | 6192.72 | 100.00 | 100.00 | Using index condition |
| 1 | PRIMARY | t2 | eq_ref | PRIMARY,called,call_id | PRIMARY | 8 | crimson.d1.leg_id | 1 | 1.00 | 100.00 | 36.73 | Using where |
| 1 | PRIMARY | g1 | ref | call_id | call_id | 4 | crimson.t2.call_id | 1 | 3.32 | 100.00 | 0.05 | Using where |
| 1 | PRIMARY | t1 | ref | calldate,call_id | call_id | 4 | crimson.t2.call_id | 1 | 5.19 | 100.00 | 0.03 | Using where |
| 2 | MATERIALIZED | <derived3> | ALL | NULL | NULL | NULL | NULL | 1222 | 1222.00 | 100.00 | 100.00 | |
| 3 | DERIVED | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | No tables used |
+------+--------------+-------------+--------+------------------------+---------+---------+--------------------+------+---------+----------+------------+---------------------------------+
7 rows in set (5 hours 16 min 16.738 sec)
ANALYZE without any IN condition at all.
+------+-------------+-------+-------+------------------------+----------+---------+--------------------+-------+---------+----------+------------+---------------------------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | r_rows | filtered | r_filtered | Extra |
+------+-------------+-------+-------+------------------------+----------+---------+--------------------+-------+---------+----------+------------+---------------------------------------------------------------------+
| 1 | SIMPLE | t1 | range | calldate,call_id | calldate | 7 | NULL | 13400 | 7162.00 | 100.00 | 33.01 | Using index condition; Using where; Using temporary; Using filesort |
| 1 | SIMPLE | t2 | ref | PRIMARY,called,call_id | call_id | 4 | crimson.t1.call_id | 1 | 3.21 | 100.00 | 35.24 | Using where |
| 1 | SIMPLE | g1 | ref | call_id | call_id | 4 | crimson.t1.call_id | 1 | 3.57 | 100.00 | 0.06 | Using where |
| 1 | SIMPLE | d1 | ref | leg_id | leg_id | 4 | crimson.t2.xid | 1 | 1.33 | 100.00 | 100.00 | Using index condition |
+------+-------------+-------+-------+------------------------+----------+---------+--------------------+-------+---------+----------+------------+---------------------------------------------------------------------+
4 rows in set (0.093 sec)
Example Tables:
CREATE TABLE `digit_dial_map_x` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`leg_id` int(11) NOT NULL,
`sequence` int(2) NOT NULL,
`digits` varchar(26) DEFAULT NULL,
`category` varchar(2) NOT NULL DEFAULT 'M',
INDEX `leg_id` (`leg_id`),
INDEX `digits` (`digits`),
INDEX `category` (`category`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
CREATE TABLE `call_legs_x` (
`xid` bigint(20) NOT NULL AUTO_INCREMENT,
`call_id` int(11) NOT NULL,
`calldate` date NOT NULL,
`start_time` time DEFAULT NULL,
`duration_hr` varchar(6) DEFAULT NULL,
`duration_min` varchar(2) DEFAULT NULL,
`duration_sec` varchar(2) DEFAULT NULL,
`calling` varchar(7) DEFAULT NULL,
`called` varchar(7) DEFAULT NULL,
`ans` varchar(2) DEFAULT NULL,
`ans_time` varchar(4) DEFAULT NULL,
`digits_dialed` varchar(26) DEFAULT NULL,
`digits_actual` varchar(26) DEFAULT NULL,
`ani` varchar(20) DEFAULT NULL,
`dnis` varchar(10) DEFAULT NULL,
`extn` varchar(10) DEFAULT NULL,
`trans_conf` varchar(2) DEFAULT NULL,
`third_party` varchar(7) DEFAULT NULL,
`sysid` varchar(3) DEFAULT NULL,
`call_log_id` varchar(12) DEFAULT NULL,
`assoc_log_id` varchar(12) DEFAULT NULL,
`raw_id` int(11) NOT NULL,
`leg` varchar(2) DEFAULT NULL,
`call_start` datetime DEFAULT NULL,
`call_end` datetime DEFAULT NULL,
`call_start_utc` datetime DEFAULT NULL,
`call_end_utc` datetime DEFAULT NULL,
INDEX `calldate` (`calldate`, `start_time`),
INDEX `called` (`called`),
INDEX `call_id` (`call_id`),
INDEX `digits_dialed` (`digits_dialed`),
INDEX `raw_id` (`raw_id`),
INDEX `call_start` (`call_start`),
INDEX `call_end` (`call_end`),
INDEX `call_start_utc` (`call_start`),
INDEX `call_end_utc` (`call_end`),
INDEX `calling` (`calling`),
INDEX `ans_time` (`ans_time`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
Example Query (omitted the 1200 IN options):
SELECT t1.call_id, t2.ans_time, t2.ans, ((t2.duration_hr * 3600) + (t2.duration_min *60) + t2.duration_sec) as duration, t2.digits_dialed, t2.digits_actual, t2.dnis, t2.trans_conf, t1.ani, t1.calling, t2.called, d1.digits, g1.extn
FROM call_legs_55 as t1
JOIN call_legs_55 as t2 ON t1.call_id=t2.call_id
JOIN digit_dial_map_55 as d1 ON t2.xid=d1.leg_id
JOIN call_legs_55 as g1 ON t1.call_id=g1.call_id
WHERE (t1.calldate BETWEEN '2019-11-25' AND '2019-11-25') AND NOT ((t1.calldate = '2019-11-25') AND (t1.start_time < '00:00:00')) AND NOT((t1.calldate = '2019-11-25') AND (t1.start_time > '24:00:00'))
AND (t1.calling IN ('T6001','T6002') )
AND d1.digits IN ('...')
AND t2.called !='X9999'
AND t1.calling != 'X9999'
AND t1.calling != ''
AND t2.ans_time != ''
AND (g1.extn IN ('52043','52042','52132','52116') AND g1.extn != t1.calling)
GROUP BY CONCAT(t1.call_id, g1.extn);
digit_dial_map_x is a many:many mapping table, correct?? It's indexes are very inefficient.
Recommend:
PRIMARY KEY(leg_id, digits),
INDEX(digits, leg_id)
More discussion: http://mysql.rjweb.org/doc.php/index_cookbook_mysql#many_to_many_mapping_table
(The table did not happen to be such.)
I have the following query. the main tables are company_reports and project_con_messages with 770,000 and 1,040,000 records respectively.
When I run this query it takes about 20 seconds and when I remove the last table company_par_user_settings it takes less than 0.5 seconds.
company_par_user_settings (short name: pus) has about 200,000 records and is meant to show user settings for each company_partner. on company_par_user_settings table we have composite unique index key on partner_id and user_id fields. I also removed the index and replaced with simple indexes on partner_id and user_id but at the end, it didn't make any big difference in running time.
SELECT *
FROM company_reports rep
LEFT JOIN system_users usr ON rep.user_id=usr.id
LEFT JOIN company_rep_subjects sbj ON rep.subject_id=sbj.id
INNER JOIN company_partners par ON rep.partner_id=par.id
LEFT JOIN project_con_messages mes ON rep.message_id=mes.id
LEFT JOIN company_par_user_settings pus ON par.id=pus.partner_id AND 1=pus.user_id
WHERE 1=1
ORDER BY rep.id DESC
LIMIT 0,50
Here below I added the explain on the above query:
|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
| 1 | SIMPLE | rep | NULL | ALL | partner_id | NULL | NULL | NULL | 772236 | 100.00 | Using temporary; Using filesort |
| 1 | SIMPLE | par | NULL | eq_ref | PRIMARY | PRIMARY | 3 | portal_ebrahim.rep.partner_id | 1 | 100.00 | NULL |
| 1 | SIMPLE | usr | NULL | eq_ref | PRIMARY | PRIMARY | 2 | portal_ebrahim.rep.user_id | 1 | 100.00 | Using where |
| 1 | SIMPLE | sbj | NULL | eq_ref | PRIMARY | PRIMARY | 2 | portal_ebrahim.rep.subject_id | 1 | 100.00 | NULL |
| 1 | SIMPLE | mes | NULL | eq_ref | PRIMARY | PRIMARY | 4 | portal_ebrahim.rep.message_id | 1 | 100.00 | NULL |
| 1 | SIMPLE | pus | NULL | ALL | NULL | NULL | NULL | NULL | 191643 | 100.00 | Using where; Using join buffer (Block Nested Loop) |
|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
I appreciate if someone help me on the right indexes or any other solutions that make the query run faster.
Edit:
Here is the show table for company_par_user_settings
CREATE TABLE `company_par_user_settings` (
`id` mediumint(9) NOT NULL AUTO_INCREMENT,
`partner_id` mediumint(8) unsigned NOT NULL,
`user_id` smallint(5) unsigned NOT NULL,
`access` tinyint(1) unsigned NOT NULL DEFAULT '0' COMMENT '0-Not specified',
`access_category` tinyint(1) unsigned NOT NULL DEFAULT '0',
`notify` tinyint(1) unsigned NOT NULL DEFAULT '0',
`stars` tinyint(1) unsigned NOT NULL DEFAULT '0',
PRIMARY KEY (`id`),
UNIQUE KEY `partner_id` (`partner_id`,`user_id`),
KEY `stars` (`stars`)
) ENGINE=MyISAM AUTO_INCREMENT=198729 DEFAULT CHARSET=utf8 COLLATE=utf8_persian_ci
Our database is set up so that we have a credentials table that hold multiple different types of credentials (logins and the like). There's also a credential_pairs table that associates some of these types together (for instance, a user may have a password and security token).
In an attempt to see if a pair match, there is the following query:
SELECT DISTINCT cp.credential_id FROM credential_pairs AS cp
INNER JOIN credentials AS c1 ON (cp.primary_credential_id = c1.credential_id)
INNER JOIN credentials AS c2 ON (cp.secondary_credential_id = c2.credential_id)
WHERE c1.data = AES_ENCRYPT('Some Value 1', 'encryption key')
AND c2.data = AES_ENCRYPT('Some Value 2', 'encryption key');
This query works fine and gives us exactly what we need. HOWEVER, it is constantly showing in the slow query log (possibly due to lack of indexes?). When I ask MySQL to "explain" the query it gives me:
+----+-------------+-------+------+--------------------------------------------------------+---------------------+---------+-------+-------+--------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+--------------------------------------------------------+---------------------+---------+-------+-------+--------------------------------+
| 1 | SIMPLE | c1 | ref | credential_id_UNIQUE,credential_id,ix_credentials_data | ix_credentials_data | 22 | const | 1 | Using where; Using temporary |
| 1 | SIMPLE | c2 | ref | credential_id_UNIQUE,credential_id,ix_credentials_data | ix_credentials_data | 22 | const | 1 | Using where |
| 1 | SIMPLE | cp | ALL | NULL | NULL | NULL | NULL | 69197 | Using where; Using join buffer |
+----+-------------+-------+------+--------------------------------------------------------+---------------------+---------+-------+-------+--------------------------------+
I have a feeling that last entry (where it shows 69197 rows) is probably the problem, but I am FAR from a DBA... help?
credentials table:
CREATE TABLE `credentials` (
`hidden_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`credential_id` varchar(255) NOT NULL,
`data` blob NOT NULL,
`credential_status` varchar(100) NOT NULL,
`insert_date` datetime NOT NULL,
`insert_user` int(10) unsigned NOT NULL,
`update_date` datetime DEFAULT NULL,
`update_user` int(10) unsigned DEFAULT NULL,
`delete_date` datetime DEFAULT NULL,
`delete_user` int(10) unsigned DEFAULT NULL,
`is_deleted` tinyint(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`hidden_id`,`credential_id`),
UNIQUE KEY `credential_id_UNIQUE` (`credential_id`),
KEY `credential_id` (`credential_id`),
KEY `data` (`data`(10)),
KEY `credential_status` (`credential_status`(10))
) ENGINE=InnoDB AUTO_INCREMENT=1572 DEFAULT CHARSET=utf8;
credential_pairs Table:
CREATE TABLE `credential_pairs` (
`hidden_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`credential_id` varchar(255) NOT NULL,
`primary_credential_id` varchar(255) NOT NULL,
`secondary_credential_id` varchar(255) NOT NULL,
`is_deleted` tinyint(1) DEFAULT NULL,
PRIMARY KEY (`hidden_id`,`credential_id`),
KEY `primary_credential_id` (`primary_credential_id`(10)),
KEY `secondary_credential_id` (`secondary_credential_id`(10))
) ENGINE=InnoDB AUTO_INCREMENT=500 DEFAULT CHARSET=latin1;
credentials Indexes:
+-------------+------------+----------------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+-------------+------------+----------------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+
| credentials | 0 | PRIMARY | 1 | hidden_id | A | 186235 | NULL | NULL | | BTREE | |
| credentials | 0 | PRIMARY | 2 | credential_id | A | 186235 | NULL | NULL | | BTREE | |
| credentials | 0 | credential_id_UNIQUE | 1 | credential_id | A | 186235 | NULL | NULL | | BTREE | |
| credentials | 1 | credential_id | 1 | credential_id | A | 186235 | NULL | NULL | | BTREE | |
| credentials | 1 | ix_credentials_data | 1 | data | A | 186235 | 20 | NULL | | BTREE | |
+-------------+------------+----------------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+
credential_pair Indexes:
+------------------+------------+---------------------------------------------+--------------+-------------------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+------------------+------------+---------------------------------------------+--------------+-------------------------+-----------+-------------+----------+--------+------+------------+---------+
| credential_pairs | 0 | PRIMARY | 1 | hidden_id | A | 69224 | NULL | NULL | | BTREE | |
| credential_pairs | 0 | PRIMARY | 2 | credential_id | A | 69224 | NULL | NULL | | BTREE | |
| credential_pairs | 1 | ix_credential_pairs_credential_id | 1 | credential_id | A | 69224 | 36 | NULL | | BTREE | |
| credential_pairs | 1 | ix_credential_pairs_primary_credential_id | 1 | primary_credential_id | A | 69224 | 36 | NULL | | BTREE | |
| credential_pairs | 1 | ix_credential_pairs_secondary_credential_id | 1 | secondary_credential_id | A | 69224 | 36 | NULL | | BTREE | |
+------------------+------------+---------------------------------------------+--------------+-------------------------+-----------+-------------+----------+--------+------+------------+---------+
UPDATE NOTES:
AFAICT: The DISTINCT was superfluous... nothing really needed it, so I dropped it. In an attempt to follow Fabrizio's advice to get a where on the credential_pairs lookup I then altered the statement to read as:
SELECT credential_id
FROM credential_pairs cp
WHERE cp.primary_credential_id = (SELECT credential_id FROM credentials WHERE data = AES_ENCRYPT('value 1','enc_key')) AND
cp.secondary_credential_id = (SELECT credential_id FROM credentials WHERE data = AES_ENCRYPT('value 2','enc_key'))
And.... nothing. The statement takes just as long and the explain looks pretty much the same. So, I added an index to the primary and secondary columns with:
ALTER TABLE credential_pairs ADD INDEX `idx_credential_pairs__primary_and_secondary`(`primary_credential_id`, `secondary_credential_id`);
And... nothing.
+----+-------------+-------------+-------+---------------------+---------------------------------------------+---------+------+-------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------------+-------+---------------------+---------------------------------------------+---------+------+-------+--------------------------+
| 1 | PRIMARY | cp | index | NULL | idx_credential_pairs__primary_and_secondary | 514 | NULL | 69217 | Using where; Using index |
| 3 | SUBQUERY | credentials | ref | ix_credentials_data | ix_credentials_data | 22 | | 1 | Using where |
| 2 | SUBQUERY | credentials | ref | ix_credentials_data | ix_credentials_data | 22 | | 1 | Using where |
+----+-------------+-------------+-------+---------------------+---------------------------------------------+---------+------+-------+--------------------------+
It says it's using the index, but it still looks like it's table scanning. So, I added a joint key (as per a'r's comment below) with:
ALTER TABLE credential_pairs ADD KEY (primary_credential_id, secondary_credential_id);
And... same result as with the index (are these functionally the same?).
The DISTINCT is what is generating the "Use temporary", you usually want to avoid those when possible
Plus you are scanning the whole credential_pair table as you do not have any conditions against it so no indexes are used and the whole table is returned before applying the WHERE
hope it makes sense
EDIT/ADD
Try by starting from a different table, if I understand correctly, you have Table A, a Table B and a Table AB and you are starting the select from AB, try to start it from A
I haven't tested this, but you could try:
SELECT cp.credential_id
FROM credentials AS c1
LEFT JOIN credential_pairs AS cp ON (c1.credential_id = cp.primary_credential_id)
LEFT JOIN credentials AS c2 ON (cp.secondary_credential_id = c2.credential_id)
WHERE
c1.data = AES_ENCRYPT('Some Value 1', 'encryption key')
AND c2.data = AES_ENCRYPT('Some Value 2', 'encryption key');
I have had luck in the past by moving select tables around
(edited) For more details about the app it self, please, also see:
Simple but heavy application consuming a lot of resources. How to Optimize?
(The adopted solution was use both joins and fulltext search)
I have the following query running up to roughly 500.000 rows in 25 seconds. If I remove the ORDER, it takes 0.5 seconds.
Fisrt test
Keeping the ORDER and removing all t. and tu. columns, the query takes 7 seconds.
Second test
If I add or remove an INDEX to the i.created_at field the response time remain the same.
QUERY:
**EDITED: I'VE NOTICED THAT BOTH GROUP BY AND ORDER BY SLOW DOWN THE QUERY (I've also achieve a little gain in the query changing the joins. The gain was to 10secs, but at all, the problem remains). With the modification, the EXPLAIN have stopped to return filesort, but stills returning "using temporary" **
SELECT SQL_NO_CACHE
DISTINCT `i`.`id`,
`i`.`entity`,
`i`.`created_at`,
`i`.`collected_at`,
`t`.`status_id` AS `twt_status_id`,
`t`.`user_id` AS `twt_user_id`,
`t`.`content` AS `twt_content`,
`tu`.`id` AS `twtu_id`,
`tu`.`screen_name` AS `twtu_screen_name`,
`tu`.`profile_image` AS `twtu_profile_image`
FROM `mtrt_items` AS `i`
LEFT JOIN `mtrt_users` AS `u` ON i.user_id =u.id
LEFT JOIN `twt_tweets_content` AS `t` ON t.id =i.id
LEFT JOIN `twt_users` AS `tu` ON u.id = tu.id
INNER JOIN `mtrt_items_searches` AS `r` ON i.id =r.item_id
INNER JOIN `mtrt_searches` AS `s` ON s.id =r.search_id
INNER JOIN `mtrt_searches_groups` AS `sg` ON sg.search_id =s.id
INNER JOIN `mtrt_search_groups` AS `g` ON sg.group_id =g.id
INNER JOIN `account_clients` AS `c` ON g.client_id =c.id
ORDER BY `i`.`created_at` DESC
LIMIT 100 OFFSET 0
Here is the EXPLAIN (EDITED):
+----+-------------+-------+--------+--------------------+-----------+---------+------------------------+------+------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+--------------------+-----------+---------+------------------------+------+------------------------------+
| 1 | SIMPLE | c | index | PRIMARY | PRIMARY | 4 | NULL | 1 | Using index; Using temporary |
| 1 | SIMPLE | g | ref | PRIMARY,client_id | client_id | 4 | clubr_new.c.id | 3 | Using index |
| 1 | SIMPLE | sg | ref | group_id,search_id | group_id | 4 | clubr_new.g.id | 1 | Using index |
| 1 | SIMPLE | s | eq_ref | PRIMARY | PRIMARY | 4 | clubr_new.sg.search_id | 1 | Using index |
| 1 | SIMPLE | r | ref | search_id,item_id | search_id | 4 | clubr_new.s.id | 4359 | Using where |
| 1 | SIMPLE | i | eq_ref | PRIMARY | PRIMARY | 8 | clubr_new.r.item_id | 1 | |
| 1 | SIMPLE | u | eq_ref | PRIMARY | PRIMARY | 8 | clubr_new.i.user_id | 1 | Using index |
| 1 | SIMPLE | t | eq_ref | PRIMARY | PRIMARY | 4 | clubr_new.i.id | 1 | |
| 1 | SIMPLE | tu | eq_ref | PRIMARY | PRIMARY | 8 | clubr_new.u.id | 1 | |
+----+-------------+-------+--------+--------------------+-----------+---------+------------------------+------+------------------------------+
Here is the mtrt_items table:
+--------------+-------------------------------------------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+-------------------------------------------------------+------+-----+---------+----------------+
| id | bigint(20) | NO | PRI | NULL | auto_increment |
| entity | enum('twitter','facebook','youtube','flickr','orkut') | NO | MUL | NULL | |
| user_id | bigint(20) | NO | MUL | NULL | |
| created_at | datetime | NO | MUL | NULL | |
| collected_at | datetime | NO | | NULL | |
+--------------+-------------------------------------------------------+------+-----+---------+----------------+
CREATE TABLE `mtrt_items` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`entity` enum('twitter','facebook','youtube','flickr','orkut') COLLATE utf8_unicode_ci NOT NULL,
`user_id` bigint(20) NOT NULL,
`created_at` datetime NOT NULL,
`collected_at` datetime NOT NULL,
PRIMARY KEY (`id`),
KEY `mtrt_user_id` (`user_id`),
KEY `entity` (`entity`),
KEY `created_at` (`created_at`),
CONSTRAINT `mtrt_items_ibfk_1` FOREIGN KEY (`user_id`) REFERENCES `mtrt_users` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=309650 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
The twt_tweets_content is MyISAM and is also used for fulltext searches:
+-----------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+--------------+------+-----+---------+-------+
| id | int(11) | NO | PRI | NULL | |
| user_id | int(11) | NO | MUL | NULL | |
| status_id | varchar(100) | NO | MUL | NULL | |
| content | varchar(200) | NO | MUL | NULL | |
+-----------+--------------+------+-----+---------+-------+
Instead of placing the Order By into the main query, wrap it, like so:
SELECT * FROM (
... your query
) ORDER BY `created at`
Take a look at the query plan. You will find that in your case, the sort is performed on your table mtrt_items before the outer join is performed. In the rewrite I've partially provided, the sort is applied after the outer joins, and is applied on a much smaller set.
UPDATE
Assuming that the LIMIT is being applied to a large set (500,000?), it looks like you can perform the top before doing any of the joins.
SELECT * from (
SELECT
`id`, ... `created_at`, ...
ORDER BY `i`.`created_at` DESC
LIMIT 100 OFFSET 0) as i
LEFT JOIN `mtrt_users` AS `u` ON i.user_id =u.id
LEFT JOIN `twt_tweets_content` AS `t` ON t.id =i.id
LEFT JOIN `twt_users` AS `tu` ON t.user_id = tu.id
INNER JOIN `mtrt_items_searches` AS `r` ON i.id =r.item_id
INNER JOIN `mtrt_searches` AS `s` ON s.id =r.search_id
INNER JOIN `mtrt_searches_groups` AS `sg` ON sg.search_id =s.id
INNER JOIN `mtrt_search_groups` AS `g` ON sg.group_id =g.id
INNER JOIN `account_clients` AS `c` ON g.client_id =c.id
GROUP BY i.id
Don't include the VARCHAR/TEXT fields in your initial query. This will create the TEMPORARY table required for the sorting, using the MEMORY engine and this will increase the efficiency dramatically. You can collect the text fields later using another query, without any sorting, simply with a condition on the PRIMARY KEY field and merge the data in your script (assuming that you are using one).
Also get rid of any JOINs (INNER or OUTER) that you don't actually take any data from.
I'm not really sure where to start with cutting down this query in the search section of my site so that it doesn't take so long. You run a search on a variety of tables, and get filtered results back mainly from the "Items" table, and currently it (and searched like it) take sometimes over 10 seconds. Part of the problem is that the php code that composes the sql is dirty as hell, of course, and I'm not used to having huge queries that I need to optimize in general, but what techniques can I use to determine where to add indexes in a query like this?
The query
SELECT
items.ItemId,
items.Name,
items.BrandCode,
items.BrandCategoryId,
items.CatalogPage,
items.PriceRetail,
items.PriceSell,
items.PriceHold,
items.Descr,
items.GenderId,
products.ImagetnURL,
products.FlagDefault,
products.ProductId,
products.Code AS ProductCode,
products.Name AS ProductName,
brands.Name AS BrandName,
items.FlagStatus AS ItemFlagStatus
FROM
items,
products,
brands,
productsizes,
searchsizechartsizes,
sizechartsizes
WHERE
items.ItemId = products.ItemId AND
items.BrandCode = brands.Code AND
items.FlagStatus != 'U' AND
products.FlagStatus != 'U' AND
items.TypeId = '10' AND
searchsizechartsizes.SearchSizeChartId = '11' AND
searchsizechartsizes.Size = sizechartsizes.Size AND
sizechartsizes.SizeChartId = productsizes.SizeChartId AND
productsizes.ProductId = products.ProductId
GROUP BY
items.ItemId
ORDER BY
items.Name
LIMIT
0, 15;
Logs show 14 second execution time, and 3 million rows examined
# Query_time: 14 Lock_time: 0 Rows_sent: 15 Rows_examined: 2901565
Summary of query results
+--------+----------------------+-----------+-----------------+-------------+-------------+-----------+-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+----------------------------+-------------+-----------+-------------+------------------------------------------+-------------------+----------------+
| ItemId | Name | BrandCode | BrandCategoryId | CatalogPage | PriceRetail | PriceSell | PriceHold | Descr | GenderId | ImagetnURL | FlagDefault | ProductId | ProductCode | ProductName | BrandName | ItemFlagStatus |
+--------+----------------------+-----------+-----------------+-------------+-------------+-
| 3376 | 10-inch Pull On Boot | RW | 2801 | 24 | 189.99 | 189.99 | 189.99 | Full grain brown leather - blah blah blah | 1 | images/rw/rw-2249tn.jpg | Y | 4345 | 2249 | Brown Full Grain Turbo Vegas Leather | Red Wing Work | A |
| 9340 | 11 | RR | NULL | 1 | 300.00 | 300.00 | 300.00 | The Engineer 11" boot from Red Wing Shoes� blah blah blah... | 1 | images/rr/rr-2990tn.jpg | Y | 16749 | 2990 | Black Harness Calfskin | Red Wing Heritage | A
~~~~~~ other results here redacted for space reasons ~~~~~~
15 rows in set (13.33 sec)
Explain of the query
mysql> explain SELECT items.ItemId, items.Name, items.BrandCode, items.BrandCategoryId, items.CatalogPage, items.PriceRetail, items.PriceSell, items.PriceHold, items.Descr, items.GenderId, products.ImagetnURL, products.FlagDefault, products.ProductId, products.Code as ProductCode, products.Name as ProductName, brands.Name as BrandName, items.FlagStatus as ItemFlagStatus FROM items, products, brands, productsizes, searchsizechartsizes, sizechartsizes WHERE items.ItemId = products.ItemId AND items.BrandCode = brands.Code AND items.FlagStatus != 'U' AND products.FlagStatus != 'U' AND items.TypeId = '10' AND (searchsizechartsizes.SearchSizeChartId = '11' AND searchsizechartsizes.Size = sizechartsizes.Size AND sizechartsizes.SizeChartId = productsizes.SizeChartId AND productsizes.ProductId = products.ProductId) group by items.ItemId ORDER BY items.Name LIMIT 0, 15;
+----+-------------+----------------------+--------+------------------------------------------------+-------------+---------+----------------------------------------+------+---------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+----------------------+--------+------------------------------------------------+-------------+---------+----------------------------------------+------+---------------------------------+
| 1 | SIMPLE | searchsizechartsizes | ref | PRIMARY,Size | PRIMARY | 4 | const | 2 | Using temporary; Using filesort |
| 1 | SIMPLE | brands | ALL | NULL | NULL | NULL | NULL | 40 | |
| 1 | SIMPLE | sizechartsizes | ref | Size,SizeChartId | Size | 33 | shermanbros.searchsizechartsizes.Size | 217 | Using where |
| 1 | SIMPLE | productsizes | ref | ProductId,SizeChartId | SizeChartId | 5 | shermanbros.sizechartsizes.SizeChartId | 17 | Using where |
| 1 | SIMPLE | products | eq_ref | PRIMARY,FlagStatus,flagstatusanddefault,ItemId | PRIMARY | 4 | shermanbros.productsizes.ProductId | 1 | Using where |
| 1 | SIMPLE | items | eq_ref | PRIMARY,BrandCode,TypeId,FlagStatus,ItemsIndex | PRIMARY | 4 | shermanbros.products.ItemId | 1 | Using where |
+----+-------------+----------------------+--------+------------------------------------------------+-------------+---------+----------------------------------------+------+---------------------------------+
6 rows in set (0.02 sec)
Structure of the items table
mysql> show create table items;
+-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| items | CREATE TABLE `items` (
`ItemId` int(11) NOT NULL auto_increment,
`Code` varchar(25) default NULL,
`Name` varchar(100) default NULL,
`BrandCode` char(2) default NULL,
`CatalogPage` int(3) default NULL,
`BrandCategoryId` int(11) default NULL,
`TypeId` int(11) default NULL,
`StyleId` int(11) default NULL,
`GenderId` int(11) default NULL,
`PriceRetail` decimal(6,2) default NULL,
`PriceSell` decimal(6,2) default NULL,
`PriceHold` decimal(6,2) default NULL,
`Cost` decimal(6,2) default NULL,
`PriceNote` longtext,
`FlagTaxable` char(1) default NULL,
`FlagStatus` char(1) default NULL,
`FlagFeatured` char(1) default NULL,
`MaintFlagStatus` char(1) default NULL,
`Descr` longtext,
`DescrNote` longtext,
`ImagetnURL` varchar(50) default NULL,
`ImagefsURL` varchar(50) default NULL,
`ImagelsURL` varchar(50) default NULL,
`DateCreated` date NOT NULL default '0000-00-00',
`DateStatus` timestamp NOT NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP,
`AdminNote` text,
PRIMARY KEY (`ItemId`),
KEY `BrandCode` (`BrandCode`),
KEY `Name` (`Name`),
KEY `TypeId` (`TypeId`),
KEY `StyleId` (`StyleId`),
KEY `GenderId` (`GenderId`),
KEY `FlagStatus` (`FlagStatus`),
KEY `ItemsIndex` (`TypeId`,`FlagStatus`,`ItemId`)
) ENGINE=MyISAM AUTO_INCREMENT=10216 DEFAULT CHARSET=latin1 |
+-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
Existing indexes on the items table
mysql> show indexes from items;
+-------+------------+------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+-------+------------+------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| items | 0 | PRIMARY | 1 | ItemId | A | 8678 | NULL | NULL | | BTREE | |
| items | 1 | BrandCode | 1 | BrandCode | A | 36 | NULL | NULL | YES | BTREE | |
| items | 1 | Name | 1 | Name | A | 8678 | NULL | NULL | YES | BTREE | |
| items | 1 | TypeId | 1 | TypeId | A | 17 | NULL | NULL | YES | BTREE | |
| items | 1 | StyleId | 1 | StyleId | A | 41 | NULL | NULL | YES | BTREE | |
| items | 1 | GenderId | 1 | GenderId | A | 3 | NULL | NULL | YES | BTREE | |
| items | 1 | FlagStatus | 1 | FlagStatus | A | 6 | NULL | NULL | YES | BTREE | |
| items | 1 | ItemsIndex | 1 | TypeId | A | 17 | NULL | NULL | YES | BTREE | |
| items | 1 | ItemsIndex | 2 | FlagStatus | A | 52 | NULL | NULL | YES | BTREE | |
| items | 1 | ItemsIndex | 3 | ItemId | A | 8678 | NULL | NULL | | BTREE | |
+-------+------------+------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
10 rows in set (0.00 sec)
It seems to have an index on various columns already, but those indexes still leave it slow with regards to this search query. And one multi-index on typeid, flagstats, itemid that's there to optimize a seperate query.
And yeah, as you can see, the items table itself is a beast, which probably doesn't help.
Looking at your FROM and WHERE clauses:
FROM
items,
products,
brands,
productsizes,
searchsizechartsizes,
sizechartsizes
WHERE
items.ItemId = products.ItemId AND
items.BrandCode = brands.Code AND
items.FlagStatus != 'U' AND
products.FlagStatus != 'U' AND
items.TypeId = '10' AND
searchsizechartsizes.SearchSizeChartId = '11' AND
searchsizechartsizes.Size = sizechartsizes.Size AND
sizechartsizes.SizeChartId = productsizes.SizeChartId AND
productsizes.ProductId = products.ProductId
you have put all of the join logic into the WHERE clause, which is unhelpful when reading the query. Using explicit join syntax instead, we can rewrite as the following:
FROM
items
JOIN products ON items.ItemId = products.ItemId
JOIN brands ON items.BrandCode = brands.Code
JOIN productsizes ON products.ProductId = productsizes.ProductId
JOIN sizechartsizes ON sizechartsizes.SizeChartId = productsizes.SizeChartId
JOIN searchsizechartsizes ON sizechartsizes.Size = searchsizechartsizes.Size
WHERE
items.FlagStatus != 'U' AND
items.TypeId = '10' AND
products.FlagStatus != 'U' AND
searchsizechartsizes.SearchSizeChartId = '11'
This makes it more clear what is going on.
It seems like you have a good candidate index for this query in ItemsIndex. (By the way - the index on TypeID alone is redundant and should be dropped, because TypeID is a left prefix of ItemsIndex.) Unfortunately MySQL's query optimiser seems to think it makes more sense to do the join back-to-front, starting with table searchsizechartsizes. I'm not sure why it would do this, but I notice that table brands apparently has no usable keys defined on it. It might be a good idea to look into this.