I'm not really sure where to start with cutting down this query in the search section of my site so that it doesn't take so long. You run a search on a variety of tables, and get filtered results back mainly from the "Items" table, and currently it (and searched like it) take sometimes over 10 seconds. Part of the problem is that the php code that composes the sql is dirty as hell, of course, and I'm not used to having huge queries that I need to optimize in general, but what techniques can I use to determine where to add indexes in a query like this?
The query
SELECT
items.ItemId,
items.Name,
items.BrandCode,
items.BrandCategoryId,
items.CatalogPage,
items.PriceRetail,
items.PriceSell,
items.PriceHold,
items.Descr,
items.GenderId,
products.ImagetnURL,
products.FlagDefault,
products.ProductId,
products.Code AS ProductCode,
products.Name AS ProductName,
brands.Name AS BrandName,
items.FlagStatus AS ItemFlagStatus
FROM
items,
products,
brands,
productsizes,
searchsizechartsizes,
sizechartsizes
WHERE
items.ItemId = products.ItemId AND
items.BrandCode = brands.Code AND
items.FlagStatus != 'U' AND
products.FlagStatus != 'U' AND
items.TypeId = '10' AND
searchsizechartsizes.SearchSizeChartId = '11' AND
searchsizechartsizes.Size = sizechartsizes.Size AND
sizechartsizes.SizeChartId = productsizes.SizeChartId AND
productsizes.ProductId = products.ProductId
GROUP BY
items.ItemId
ORDER BY
items.Name
LIMIT
0, 15;
Logs show 14 second execution time, and 3 million rows examined
# Query_time: 14 Lock_time: 0 Rows_sent: 15 Rows_examined: 2901565
Summary of query results
+--------+----------------------+-----------+-----------------+-------------+-------------+-----------+-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+----------------------------+-------------+-----------+-------------+------------------------------------------+-------------------+----------------+
| ItemId | Name | BrandCode | BrandCategoryId | CatalogPage | PriceRetail | PriceSell | PriceHold | Descr | GenderId | ImagetnURL | FlagDefault | ProductId | ProductCode | ProductName | BrandName | ItemFlagStatus |
+--------+----------------------+-----------+-----------------+-------------+-------------+-
| 3376 | 10-inch Pull On Boot | RW | 2801 | 24 | 189.99 | 189.99 | 189.99 | Full grain brown leather - blah blah blah | 1 | images/rw/rw-2249tn.jpg | Y | 4345 | 2249 | Brown Full Grain Turbo Vegas Leather | Red Wing Work | A |
| 9340 | 11 | RR | NULL | 1 | 300.00 | 300.00 | 300.00 | The Engineer 11" boot from Red Wing Shoes� blah blah blah... | 1 | images/rr/rr-2990tn.jpg | Y | 16749 | 2990 | Black Harness Calfskin | Red Wing Heritage | A
~~~~~~ other results here redacted for space reasons ~~~~~~
15 rows in set (13.33 sec)
Explain of the query
mysql> explain SELECT items.ItemId, items.Name, items.BrandCode, items.BrandCategoryId, items.CatalogPage, items.PriceRetail, items.PriceSell, items.PriceHold, items.Descr, items.GenderId, products.ImagetnURL, products.FlagDefault, products.ProductId, products.Code as ProductCode, products.Name as ProductName, brands.Name as BrandName, items.FlagStatus as ItemFlagStatus FROM items, products, brands, productsizes, searchsizechartsizes, sizechartsizes WHERE items.ItemId = products.ItemId AND items.BrandCode = brands.Code AND items.FlagStatus != 'U' AND products.FlagStatus != 'U' AND items.TypeId = '10' AND (searchsizechartsizes.SearchSizeChartId = '11' AND searchsizechartsizes.Size = sizechartsizes.Size AND sizechartsizes.SizeChartId = productsizes.SizeChartId AND productsizes.ProductId = products.ProductId) group by items.ItemId ORDER BY items.Name LIMIT 0, 15;
+----+-------------+----------------------+--------+------------------------------------------------+-------------+---------+----------------------------------------+------+---------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+----------------------+--------+------------------------------------------------+-------------+---------+----------------------------------------+------+---------------------------------+
| 1 | SIMPLE | searchsizechartsizes | ref | PRIMARY,Size | PRIMARY | 4 | const | 2 | Using temporary; Using filesort |
| 1 | SIMPLE | brands | ALL | NULL | NULL | NULL | NULL | 40 | |
| 1 | SIMPLE | sizechartsizes | ref | Size,SizeChartId | Size | 33 | shermanbros.searchsizechartsizes.Size | 217 | Using where |
| 1 | SIMPLE | productsizes | ref | ProductId,SizeChartId | SizeChartId | 5 | shermanbros.sizechartsizes.SizeChartId | 17 | Using where |
| 1 | SIMPLE | products | eq_ref | PRIMARY,FlagStatus,flagstatusanddefault,ItemId | PRIMARY | 4 | shermanbros.productsizes.ProductId | 1 | Using where |
| 1 | SIMPLE | items | eq_ref | PRIMARY,BrandCode,TypeId,FlagStatus,ItemsIndex | PRIMARY | 4 | shermanbros.products.ItemId | 1 | Using where |
+----+-------------+----------------------+--------+------------------------------------------------+-------------+---------+----------------------------------------+------+---------------------------------+
6 rows in set (0.02 sec)
Structure of the items table
mysql> show create table items;
+-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| items | CREATE TABLE `items` (
`ItemId` int(11) NOT NULL auto_increment,
`Code` varchar(25) default NULL,
`Name` varchar(100) default NULL,
`BrandCode` char(2) default NULL,
`CatalogPage` int(3) default NULL,
`BrandCategoryId` int(11) default NULL,
`TypeId` int(11) default NULL,
`StyleId` int(11) default NULL,
`GenderId` int(11) default NULL,
`PriceRetail` decimal(6,2) default NULL,
`PriceSell` decimal(6,2) default NULL,
`PriceHold` decimal(6,2) default NULL,
`Cost` decimal(6,2) default NULL,
`PriceNote` longtext,
`FlagTaxable` char(1) default NULL,
`FlagStatus` char(1) default NULL,
`FlagFeatured` char(1) default NULL,
`MaintFlagStatus` char(1) default NULL,
`Descr` longtext,
`DescrNote` longtext,
`ImagetnURL` varchar(50) default NULL,
`ImagefsURL` varchar(50) default NULL,
`ImagelsURL` varchar(50) default NULL,
`DateCreated` date NOT NULL default '0000-00-00',
`DateStatus` timestamp NOT NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP,
`AdminNote` text,
PRIMARY KEY (`ItemId`),
KEY `BrandCode` (`BrandCode`),
KEY `Name` (`Name`),
KEY `TypeId` (`TypeId`),
KEY `StyleId` (`StyleId`),
KEY `GenderId` (`GenderId`),
KEY `FlagStatus` (`FlagStatus`),
KEY `ItemsIndex` (`TypeId`,`FlagStatus`,`ItemId`)
) ENGINE=MyISAM AUTO_INCREMENT=10216 DEFAULT CHARSET=latin1 |
+-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
Existing indexes on the items table
mysql> show indexes from items;
+-------+------------+------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+-------+------------+------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| items | 0 | PRIMARY | 1 | ItemId | A | 8678 | NULL | NULL | | BTREE | |
| items | 1 | BrandCode | 1 | BrandCode | A | 36 | NULL | NULL | YES | BTREE | |
| items | 1 | Name | 1 | Name | A | 8678 | NULL | NULL | YES | BTREE | |
| items | 1 | TypeId | 1 | TypeId | A | 17 | NULL | NULL | YES | BTREE | |
| items | 1 | StyleId | 1 | StyleId | A | 41 | NULL | NULL | YES | BTREE | |
| items | 1 | GenderId | 1 | GenderId | A | 3 | NULL | NULL | YES | BTREE | |
| items | 1 | FlagStatus | 1 | FlagStatus | A | 6 | NULL | NULL | YES | BTREE | |
| items | 1 | ItemsIndex | 1 | TypeId | A | 17 | NULL | NULL | YES | BTREE | |
| items | 1 | ItemsIndex | 2 | FlagStatus | A | 52 | NULL | NULL | YES | BTREE | |
| items | 1 | ItemsIndex | 3 | ItemId | A | 8678 | NULL | NULL | | BTREE | |
+-------+------------+------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
10 rows in set (0.00 sec)
It seems to have an index on various columns already, but those indexes still leave it slow with regards to this search query. And one multi-index on typeid, flagstats, itemid that's there to optimize a seperate query.
And yeah, as you can see, the items table itself is a beast, which probably doesn't help.
Looking at your FROM and WHERE clauses:
FROM
items,
products,
brands,
productsizes,
searchsizechartsizes,
sizechartsizes
WHERE
items.ItemId = products.ItemId AND
items.BrandCode = brands.Code AND
items.FlagStatus != 'U' AND
products.FlagStatus != 'U' AND
items.TypeId = '10' AND
searchsizechartsizes.SearchSizeChartId = '11' AND
searchsizechartsizes.Size = sizechartsizes.Size AND
sizechartsizes.SizeChartId = productsizes.SizeChartId AND
productsizes.ProductId = products.ProductId
you have put all of the join logic into the WHERE clause, which is unhelpful when reading the query. Using explicit join syntax instead, we can rewrite as the following:
FROM
items
JOIN products ON items.ItemId = products.ItemId
JOIN brands ON items.BrandCode = brands.Code
JOIN productsizes ON products.ProductId = productsizes.ProductId
JOIN sizechartsizes ON sizechartsizes.SizeChartId = productsizes.SizeChartId
JOIN searchsizechartsizes ON sizechartsizes.Size = searchsizechartsizes.Size
WHERE
items.FlagStatus != 'U' AND
items.TypeId = '10' AND
products.FlagStatus != 'U' AND
searchsizechartsizes.SearchSizeChartId = '11'
This makes it more clear what is going on.
It seems like you have a good candidate index for this query in ItemsIndex. (By the way - the index on TypeID alone is redundant and should be dropped, because TypeID is a left prefix of ItemsIndex.) Unfortunately MySQL's query optimiser seems to think it makes more sense to do the join back-to-front, starting with table searchsizechartsizes. I'm not sure why it would do this, but I notice that table brands apparently has no usable keys defined on it. It might be a good idea to look into this.
Related
I have a nested query that deletes a row in table terms only if exactly one row in definitions.term_id is found. It works but it takes like 9 seconds on my system. Im looking to optimize the query.
DELETE FROM terms
WHERE id
IN(
SELECT term_id
FROM definitions
WHERE term_id = 1234
GROUP BY term_id
HAVING COUNT(term_id) = 1
)
The database is only about 4000 rows. If I separate the query into 2 independent queries, it takes about 0.1 each
terms
+-------+------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------+------------------+------+-----+---------+----------------+
| id | int(11) unsigned | NO | PRI | NULL | auto_increment |
| term | varchar(50) | YES | | NULL | |
+-------+------------------+------+-----+---------+----------------+
definitions
+----------------+------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+----------------+------------------+------+-----+---------+----------------+
| id | int(11) unsigned | NO | PRI | NULL | auto_increment |
| term_id | int(11) | YES | | NULL | |
| definition | varchar(500) | YES | | NULL | |
| example | varchar(500) | YES | | NULL | |
| submitter_name | varchar(50) | YES | | NULL | |
| approved | int(1) | YES | MUL | 0 | |
| created_at | timestamp | YES | | NULL | |
| updated_at | timestamp | YES | | NULL | |
| votos | int(3) | NO | | NULL | |
+----------------+------------------+------+-----+---------+----------------+
To speed up the process, please consider creating an index on the relevant field:
CREATE INDEX term_id ON terms (term_id)
How about using correlated sub query using exists and try,
DELETE FROM terms t
WHERE id = 1234
AND EXISTS (SELECT 1
FROM definitions d
WHERE d.term_id = t.term_id
GROUP BY term_id
HAVING COUNT(term_id) = 1)
It's often quicker to create a new table retaining only the rows you wish to keep. That said, I'd probably write this as follows, and provide indexes as appropriate.
DELETE
FROM terms t
JOIN
( SELECT term_id
FROM definitions
WHERE term_id = 1234
GROUP
BY term_id
HAVING COUNT(*) = 1
) x
ON x.term_id = t.id
Hehe; this may be a kludgy way to do it:
DELETE ... WHERE id = ( SELECT ... )
but without any LIMIT or other constraints.
I'm depending on getting an error something like "subquery returned more than one row" in order to prevent the DELETE being performed if multiple rows match.
I have the following ugly query, which runs okay but not great, on my local machine (1.4 secs, running v5.7). On the server I'm using, which is running an older version of MySQL (v5.5), the query just hangs. It seems to get caught on "Copying to tmp table":
SELECT
SQL_CALC_FOUND_ROWS
DISTINCT p.parcel_number,
p.street_number,
p.street_name,
p.site_address_city_state,
p.number_of_units,
p.number_of_stories,
p.bedrooms,
p.bathrooms,
p.lot_area_sqft,
p.cost_per_sq_ft,
p.year_built,
p.sales_date,
p.sales_price,
p.id
FROM (
SELECT APN, property_case_detail_id FROM property_inspection AS pi
GROUP BY APN, property_case_detail_id
HAVING
COUNT(IF(status='Resolved Date', 1, NULL)) = 0
) as open_cases
JOIN property AS p
ON p.parcel_number = open_cases.APN
LIMIT 0, 1000;
mysql> show processlist;
+-------+-------------+-----------+--------------+---------+------+----------------------+------------------------------------------------------------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-------+-------------+-----------+--------------+---------+------+----------------------+------------------------------------------------------------------------------------------------------+
| 21120 | headsupcity | localhost | lead_housing | Query | 21 | Copying to tmp table | SELECT
SQL_CALC_FOUND_ROWS
DISTINCT p.parcel_number,
p.street_numbe |
| 21121 | headsupcity | localhost | lead_housing | Query | 0 | NULL | show processlist |
+-------+-------------+-----------+--------------+---------+------+----------------------+------------------------------------------------------------------------------------------------------+
2 rows in set (0.00 sec)
Explains are different on my local machine and on the server, and I'm assuming the only reason my query runs at all on my local machine, is because of the key that is automatically created on the derived table:
Explain (local):
+----+-------------+------------+------------+------+---------------+-------------+---------+------------------------------+---------+----------+---------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+------------+------------+------+---------------+-------------+---------+------------------------------+---------+----------+---------------------------------+
| 1 | PRIMARY | p | NULL | ALL | NULL | NULL | NULL | NULL | 40319 | 100.00 | Using temporary |
| 1 | PRIMARY | <derived2> | NULL | ref | <auto_key0> | <auto_key0> | 8 | lead_housing.p.parcel_number | 40 | 100.00 | NULL |
| 2 | DERIVED | pi | NULL | ALL | NULL | NULL | NULL | NULL | 1623978 | 100.00 | Using temporary; Using filesort |
+----+-------------+------------+------------+------+---------------+-------------+---------+------------------------------+---------+----------+---------------------------------+
Explain (server):
+----+-------------+------------+------+---------------+------+---------+------+---------+------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------+------+---------------+------+---------+------+---------+------------------------------------------+
| 1 | PRIMARY | p | ALL | NULL | NULL | NULL | NULL | 41369 | Using temporary |
| 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 122948 | Using where; Distinct; Using join buffer |
| 2 | DERIVED | pi | ALL | NULL | NULL | NULL | NULL | 1718586 | Using temporary; Using filesort |
+----+-------------+------------+------+---------------+------+---------+------+---------+------------------------------------------+
Schemas:
mysql> explain property_inspection;
+-------------------------+--------------+------+-----+-------------------+-----------------------------+
| Field | Type | Null | Key | Default | Extra |
+-------------------------+--------------+------+-----+-------------------+-----------------------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| lblCaseNo | int(11) | NO | MUL | NULL | |
| APN | bigint(10) | NO | MUL | NULL | |
| date | varchar(50) | NO | | NULL | |
| status | varchar(500) | NO | | NULL | |
| property_case_detail_id | int(11) | YES | MUL | NULL | |
| case_type_id | int(11) | YES | MUL | NULL | |
| date_modified | timestamp | NO | | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
| update_status | tinyint(1) | YES | | 1 | |
| created_date | datetime | NO | | NULL | |
+-------------------------+--------------+------+-----+-------------------+-----------------------------+
10 rows in set (0.02 sec)
mysql> explain property; (not all columns, but you get the gist)
+----------------------------+--------------+------+-----+-------------------+-----------------------------+
| Field | Type | Null | Key | Default | Extra |
+----------------------------+--------------+------+-----+-------------------+-----------------------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| parcel_number | bigint(10) | NO | | 0 | |
| date_modified | timestamp | NO | | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
| created_date | datetime | NO | | NULL | |
+----------------------------+--------------+------+-----+-------------------+-----------------------------+
Variables that might be relevant:
tmp_table_size: 16777216
innodb_buffer_pool_size: 8589934592
Any ideas on how to optimize this, and any idea why the explains are so different?
Since this is where the Optimizers are quite different, let's try to optimize
SELECT APN, property_case_detail_id FROM property_inspection AS pi
GROUP BY APN, property_case_detail_id
HAVING
COUNT(IF(status='Resolved Date', 1, NULL)) = 0
) as open_cases
Give this a try:
SELECT ...
FROM property AS p
WHERE NOT EXISTS ( SELECT 1 FROM property_inspection
WHERE status = 'Resolved Date'
AND p.parcel_number = APN )
ORDER BY ??? -- without this, the `LIMIT` is unpredictable
LIMIT 0, 1000;
or...
SELECT ...
FROM property AS p
LEFT JOIN property_inspection AS pi ON p.parcel_number = pi.APN
WHERE pi.status = 'Resolved Date'
AND pi.APN IS NULL
ORDER BY ??? -- without this, the `LIMIT` is unpredictable
LIMIT 0, 1000;
Index:
property_inspection: INDEX(status, parcel_number) -- in either order
MySQL 5.5 and 5.7 are quite different and the later has better optimizer so there is no surprise that explain plans are different.
You'd better provide SHOW CREATE TABLE property; and SHOW CREATE TABLE property_inspection; outputs as it will show indexes that are on your tables.
Your sub-query is the issue.
- Server tries to process 1.6M rows with no index and grouping everything.
- Having is quite expensive operation so you'd better avoid it, expecially in sub-queries.
- Grouping in this case is bad idea. You do not need the aggregation/counting. You need to check if the 'Resolved Date' status is just exists
Based on the information provided I'd recommend:
- Alter table property_inspection to reduce length of status column.
- Add index on the column. Use covering index (APN, property_case_detail_id, status) if possible (in this columns order).
- Change query to something like this:
SELECT
SQL_CALC_FOUND_ROWS
DISTINCT p.parcel_number,
...
p.id
FROM
property_inspection AS `pi1`
INNER JOIN property AS p ON (
p.parcel_number = `pi1`.APN
)
LEFT JOIN (
SELECT
`pi2`.property_case_detail_id
, `pi2`. APN
FROM
property_inspection AS `pi2`
WHERE
`status` = 'Resolved Date'
) AS exclude ON (
exclude.APN = `pi1`.APN
AND exclude.property_case_detail_id = `pi1`.property_case_detail_id
)
WHERE
exclude.APN IS NULL
LIMIT
0, 1000;
here is my explain result.
+----+-------------+----------------+--------+-------------------------------------------------------+-------------+---------+-------------------------------------+------+----------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+----------------+--------+-------------------------------------------------------+-------------+---------+-------------------------------------+------+----------------------------------------------------+
| 1 | SIMPLE | client_address | ALL | client_id | NULL | NULL | NULL | 1619 | Using where; Using temporary; Using filesort |
| 1 | SIMPLE | clients | eq_ref | PRIMARY,contract_id,contract_id_2 | PRIMARY | 3 | defenselaw.client_address.client_id | 1 | Using where |
| 1 | SIMPLE | user_infos | eq_ref | user_id | user_id | 3 | defenselaw.clients.client_rep_id | 1 | Using where |
| 1 | SIMPLE | users | eq_ref | PRIMARY | PRIMARY | 2 | defenselaw.user_infos.user_id | 1 | Using where; Using index |
| 1 | SIMPLE | contracts | eq_ref | PRIMARY | PRIMARY | 3 | defenselaw.clients.contract_id | 1 | NULL |
| 1 | SIMPLE | documents | eq_ref | contract_id,contract_id_2,contract_id_3,contract_id_4 | contract_id | 3 | defenselaw.clients.contract_id | 1 | Using where |
| 1 | SIMPLE | client_info | ref | client_id,client_id_2,client_id_3,index5 | client_id | 3 | defenselaw.client_address.client_id | 1 | Using where |
| 1 | SIMPLE | location | ALL | PRIMARY | NULL | NULL | NULL | 1 | Using where; Using join buffer (Block Nested Loop) |
+----+-------------+----------------+--------+-------------------------------------------------------+-------------+---------+-------------------------------------+------+----------------------------------------------------+
here is the query.
SELECT
contracts.contract_id,
documents.documentKey,
documents.document_name,
documents.notify_accountant,
clients.client_first_name,
clients.client_middle_name,
clients.client_last_name,
clients.client_rep_id,
client_address.client_street,
client_address.client_apartment,
client_address.client_city,
client_address.client_state,
client_address.client_zip,
client_info.client_email,
client_info.client_home_phone,
client_info.client_work_phone,
client_info.client_cell_phone,
client_info.client_fax_number,
contracts.contract_agreement_date,
user_infos.user_first_name,
user_infos.user_last_name,
location.location
FROM
`contracts`
LEFT JOIN
`clients`
ON clients.contract_id = contracts.contract_id
INNER JOIN
`client_address`
ON client_address.client_id = clients.client_id
INNER JOIN
`client_info`
ON client_info.client_id = client_address.client_id
LEFT JOIN
`documents`
ON contracts.contract_id = documents.contract_id
LEFT JOIN
`user_infos`
ON user_infos.user_id = clients.client_rep_id
LEFT JOIN
`users` ON
users.user_id = user_infos.user_id
LEFT JOIN
`location`
ON location.location_id = user_infos.location_id
WHERE client_info.client_role = 'primary'
AND documents.archive = 0
ORDER BY
documents.created_on DESC
Could someone explain to me why my index on client_address isn't being used? I am lost here as I just don't know what to do. It's not that complicated of a query its just rather large. Yes, stackoverflow, I understand the post is mostly code. What more do you want from me? How much more can I say about this other than my query isn't using an index and its causing a really slow query here.
+----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| client_address | CREATE TABLE `client_address` (
`client_address_id` mediumint(12) NOT NULL AUTO_INCREMENT,
`client_id` mediumint(12) NOT NULL,
`client_street` varchar(40) COLLATE utf8_bin NOT NULL,
`client_apartment` varchar(20) COLLATE utf8_bin NOT NULL,
`client_city` varchar(80) COLLATE utf8_bin NOT NULL,
`client_state` varchar(80) COLLATE utf8_bin NOT NULL,
`client_zip` varchar(80) COLLATE utf8_bin NOT NULL,
PRIMARY KEY (`client_address_id`),
KEY `client_id` (`client_id`)
) ENGINE=InnoDB AUTO_INCREMENT=2167 DEFAULT CHARSET=utf8 COLLATE=utf8_bin |
+----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I have the following MySQL query:
SELECT pool.username
FROM pool
LEFT JOIN sent ON pool.username = sent.username
AND sent.campid = 'YA1LGfh9'
WHERE sent.username IS NULL
AND pool.gender = 'f'
AND (`location` = 'united states' OR `location` = 'us' OR `location` = 'usa');
The problem is that the pool table contains millions of rows and this query takes over 12 minutes to complete. I realize that in this query, the entire left table (pool) is being scanned. The pool table has an auto incremented id row.
I would like to split this query into multiple queries so that rather than scanning the entire pool table I scan 1000 rows at a time and in the next query I would pick up where I left off (1000-2000,2000-3000) and so on using the id column to keep track.
How can I specify this in my query? Please show examples if you know the answer. Thank you.
here are my indexes if it helps:
mysql> show index from main.pool;
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| pool | 0 | PRIMARY | 1 | id | A | 9275039 | NULL | NULL | | BTREE | |
| pool | 1 | username | 1 | username | A | 9275039 | NULL | NULL | | BTREE | |
| pool | 1 | source | 1 | source | A | 1 | NULL | NULL | | BTREE | |
| pool | 1 | location | 1 | location | A | 38168 | NULL | NULL | | BTREE | |
| pool | 1 | pdex | 1 | gender | A | 2 | NULL | NULL | | BTREE | |
| pool | 1 | pdex | 2 | username | A | 9275039 | NULL | NULL | | BTREE | |
| pool | 1 | pdex | 3 | id | A | 9275039 | NULL | NULL | | BTREE | |
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
8 rows in set (0.00 sec)
mysql> show index from main.sent;
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| sent | 0 | PRIMARY | 1 | primary_key | A | 351 | NULL | NULL | | BTREE | |
| sent | 1 | username | 1 | username | A | 175 | NULL | NULL | | BTREE | |
| sent | 1 | sdex | 1 | campid | A | 7 | NULL | NULL | | BTREE | |
| sent | 1 | sdex | 2 | username | A | 351 | NULL | NULL | | BTREE | |
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
and here is the explain for my query:
----------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+---------------+------+---------+-------+---------+--------------------------------------+
| 1 | SIMPLE | pool | ref | location,pdex | pdex | 5 | const | 6084332 | Using where |
| 1 | SIMPLE | sent | index | sdex | sdex | 309 | NULL | 351 | Using where; Using index; Not exists |
+----+-------------+-------+-------+---------------+------+---------+-------+---------+--------------------------------------+
here is the structure of the pool table:
| pool | CREATE TABLE `pool` (
`id` int(20) NOT NULL AUTO_INCREMENT,
`username` varchar(50) CHARACTER SET utf8 NOT NULL,
`source` varchar(10) CHARACTER SET utf8 NOT NULL,
`gender` varchar(1) CHARACTER SET utf8 NOT NULL,
`location` varchar(50) CHARACTER SET utf8 NOT NULL,
PRIMARY KEY (`id`),
KEY `username` (`username`),
KEY `source` (`source`),
KEY `location` (`location`),
KEY `pdex` (`gender`,`username`,`id`)
) ENGINE=MyISAM AUTO_INCREMENT=9327026 DEFAULT CHARSET=latin1 |
here is the structure of the sent table:
| sent | CREATE TABLE `sent` (
`primary_key` int(50) NOT NULL AUTO_INCREMENT,
`username` varchar(50) NOT NULL,
`from` varchar(50) NOT NULL,
`campid` varchar(255) NOT NULL,
`timestamp` int(20) NOT NULL,
PRIMARY KEY (`primary_key`),
KEY `username` (`username`),
KEY `sdex` (`campid`,`username`)
) ENGINE=MyISAM AUTO_INCREMENT=352 DEFAULT CHARSET=latin1 |
This produces a syntax error but this WHERE clause in the beginning is what im after:
SELECT pool.username
FROM pool
WHERE id < 1000
LEFT JOIN sent ON pool.username = sent.username
AND sent.campid = 'YA1LGfh9'
WHERE sent.username IS NULL
AND pool.gender = 'f'
AND (location = 'united states' OR location = 'us' OR location = 'usa');
Splitting your query does not sound like the right approach.
The better way would be to fetch some records from your existing query, send your messages and then continue fetching.
Your query could benefit from another compound index on
pool( location, gender, username )
This should allow to run your complete query from sdex and your new index.
If you really want to split the query, an easy approach could be to
SELECT MIN(id), MAX(id) FROM pool
and then loop from min to max in steps of 1000, and add id >= r AND id < r+1000 to your query.
This could return 0 rows if you have gaps, but it would never return more than 1000 rows at once. A different compound index on pool including (id, location, gender and maybe username) could help for this query.
Looks like its using pool.location
Could try adding an index on gender, might no be a lot of help though.
Rationalising location to a country code in your data, and indexing that would probably be useful.
But the first index to add looks to be campid to me, that could make a serious dent in the number of records it has to test.
Our database is set up so that we have a credentials table that hold multiple different types of credentials (logins and the like). There's also a credential_pairs table that associates some of these types together (for instance, a user may have a password and security token).
In an attempt to see if a pair match, there is the following query:
SELECT DISTINCT cp.credential_id FROM credential_pairs AS cp
INNER JOIN credentials AS c1 ON (cp.primary_credential_id = c1.credential_id)
INNER JOIN credentials AS c2 ON (cp.secondary_credential_id = c2.credential_id)
WHERE c1.data = AES_ENCRYPT('Some Value 1', 'encryption key')
AND c2.data = AES_ENCRYPT('Some Value 2', 'encryption key');
This query works fine and gives us exactly what we need. HOWEVER, it is constantly showing in the slow query log (possibly due to lack of indexes?). When I ask MySQL to "explain" the query it gives me:
+----+-------------+-------+------+--------------------------------------------------------+---------------------+---------+-------+-------+--------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+--------------------------------------------------------+---------------------+---------+-------+-------+--------------------------------+
| 1 | SIMPLE | c1 | ref | credential_id_UNIQUE,credential_id,ix_credentials_data | ix_credentials_data | 22 | const | 1 | Using where; Using temporary |
| 1 | SIMPLE | c2 | ref | credential_id_UNIQUE,credential_id,ix_credentials_data | ix_credentials_data | 22 | const | 1 | Using where |
| 1 | SIMPLE | cp | ALL | NULL | NULL | NULL | NULL | 69197 | Using where; Using join buffer |
+----+-------------+-------+------+--------------------------------------------------------+---------------------+---------+-------+-------+--------------------------------+
I have a feeling that last entry (where it shows 69197 rows) is probably the problem, but I am FAR from a DBA... help?
credentials table:
CREATE TABLE `credentials` (
`hidden_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`credential_id` varchar(255) NOT NULL,
`data` blob NOT NULL,
`credential_status` varchar(100) NOT NULL,
`insert_date` datetime NOT NULL,
`insert_user` int(10) unsigned NOT NULL,
`update_date` datetime DEFAULT NULL,
`update_user` int(10) unsigned DEFAULT NULL,
`delete_date` datetime DEFAULT NULL,
`delete_user` int(10) unsigned DEFAULT NULL,
`is_deleted` tinyint(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`hidden_id`,`credential_id`),
UNIQUE KEY `credential_id_UNIQUE` (`credential_id`),
KEY `credential_id` (`credential_id`),
KEY `data` (`data`(10)),
KEY `credential_status` (`credential_status`(10))
) ENGINE=InnoDB AUTO_INCREMENT=1572 DEFAULT CHARSET=utf8;
credential_pairs Table:
CREATE TABLE `credential_pairs` (
`hidden_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`credential_id` varchar(255) NOT NULL,
`primary_credential_id` varchar(255) NOT NULL,
`secondary_credential_id` varchar(255) NOT NULL,
`is_deleted` tinyint(1) DEFAULT NULL,
PRIMARY KEY (`hidden_id`,`credential_id`),
KEY `primary_credential_id` (`primary_credential_id`(10)),
KEY `secondary_credential_id` (`secondary_credential_id`(10))
) ENGINE=InnoDB AUTO_INCREMENT=500 DEFAULT CHARSET=latin1;
credentials Indexes:
+-------------+------------+----------------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+-------------+------------+----------------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+
| credentials | 0 | PRIMARY | 1 | hidden_id | A | 186235 | NULL | NULL | | BTREE | |
| credentials | 0 | PRIMARY | 2 | credential_id | A | 186235 | NULL | NULL | | BTREE | |
| credentials | 0 | credential_id_UNIQUE | 1 | credential_id | A | 186235 | NULL | NULL | | BTREE | |
| credentials | 1 | credential_id | 1 | credential_id | A | 186235 | NULL | NULL | | BTREE | |
| credentials | 1 | ix_credentials_data | 1 | data | A | 186235 | 20 | NULL | | BTREE | |
+-------------+------------+----------------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+
credential_pair Indexes:
+------------------+------------+---------------------------------------------+--------------+-------------------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+------------------+------------+---------------------------------------------+--------------+-------------------------+-----------+-------------+----------+--------+------+------------+---------+
| credential_pairs | 0 | PRIMARY | 1 | hidden_id | A | 69224 | NULL | NULL | | BTREE | |
| credential_pairs | 0 | PRIMARY | 2 | credential_id | A | 69224 | NULL | NULL | | BTREE | |
| credential_pairs | 1 | ix_credential_pairs_credential_id | 1 | credential_id | A | 69224 | 36 | NULL | | BTREE | |
| credential_pairs | 1 | ix_credential_pairs_primary_credential_id | 1 | primary_credential_id | A | 69224 | 36 | NULL | | BTREE | |
| credential_pairs | 1 | ix_credential_pairs_secondary_credential_id | 1 | secondary_credential_id | A | 69224 | 36 | NULL | | BTREE | |
+------------------+------------+---------------------------------------------+--------------+-------------------------+-----------+-------------+----------+--------+------+------------+---------+
UPDATE NOTES:
AFAICT: The DISTINCT was superfluous... nothing really needed it, so I dropped it. In an attempt to follow Fabrizio's advice to get a where on the credential_pairs lookup I then altered the statement to read as:
SELECT credential_id
FROM credential_pairs cp
WHERE cp.primary_credential_id = (SELECT credential_id FROM credentials WHERE data = AES_ENCRYPT('value 1','enc_key')) AND
cp.secondary_credential_id = (SELECT credential_id FROM credentials WHERE data = AES_ENCRYPT('value 2','enc_key'))
And.... nothing. The statement takes just as long and the explain looks pretty much the same. So, I added an index to the primary and secondary columns with:
ALTER TABLE credential_pairs ADD INDEX `idx_credential_pairs__primary_and_secondary`(`primary_credential_id`, `secondary_credential_id`);
And... nothing.
+----+-------------+-------------+-------+---------------------+---------------------------------------------+---------+------+-------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------------+-------+---------------------+---------------------------------------------+---------+------+-------+--------------------------+
| 1 | PRIMARY | cp | index | NULL | idx_credential_pairs__primary_and_secondary | 514 | NULL | 69217 | Using where; Using index |
| 3 | SUBQUERY | credentials | ref | ix_credentials_data | ix_credentials_data | 22 | | 1 | Using where |
| 2 | SUBQUERY | credentials | ref | ix_credentials_data | ix_credentials_data | 22 | | 1 | Using where |
+----+-------------+-------------+-------+---------------------+---------------------------------------------+---------+------+-------+--------------------------+
It says it's using the index, but it still looks like it's table scanning. So, I added a joint key (as per a'r's comment below) with:
ALTER TABLE credential_pairs ADD KEY (primary_credential_id, secondary_credential_id);
And... same result as with the index (are these functionally the same?).
The DISTINCT is what is generating the "Use temporary", you usually want to avoid those when possible
Plus you are scanning the whole credential_pair table as you do not have any conditions against it so no indexes are used and the whole table is returned before applying the WHERE
hope it makes sense
EDIT/ADD
Try by starting from a different table, if I understand correctly, you have Table A, a Table B and a Table AB and you are starting the select from AB, try to start it from A
I haven't tested this, but you could try:
SELECT cp.credential_id
FROM credentials AS c1
LEFT JOIN credential_pairs AS cp ON (c1.credential_id = cp.primary_credential_id)
LEFT JOIN credentials AS c2 ON (cp.secondary_credential_id = c2.credential_id)
WHERE
c1.data = AES_ENCRYPT('Some Value 1', 'encryption key')
AND c2.data = AES_ENCRYPT('Some Value 2', 'encryption key');
I have had luck in the past by moving select tables around