I have two SQL queries that I need for my application:
SELECT *
FROM a
LEFT JOIN b AS b1
ON a.source_id = b1.id AND a.source_type = 'xxxxx'
LEFT JOIN b AS b2
ON a.source_id = b2.id AND a.source_type = 'xxxxx'
WHERE (b1.user_id = 1 OR b2.user_id = 1);
EXPLAIN gives the following:
|----|-------------|-------|------------|--------|----------------------------|---------|---------|----------------------|--------|----------|--------------------------------------------|
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
|----|-------------|-------|------------|--------|----------------------------|---------|---------|----------------------|--------|----------|--------------------------------------------|
| 1 | SIMPLE | a | NULL | ALL | NULL | NULL | NULL | NULL | 437816 | 100.00 | NULL |
|----|-------------|-------|------------|--------|----------------------------|---------|---------|----------------------|--------|----------|--------------------------------------------|
| 1 | SIMPLE | b1 | NULL | ALL | PRIMARY | NULL | NULL | NULL | 6 | 100.00 | Using where; Using join buffer (hash join) |
|----|-------------|-------|------------|--------|----------------------------|---------|---------|----------------------|--------|----------|--------------------------------------------|
| 1 | SIMPLE | b2 | NULL | eq_ref | PRIMARY,index_b_on_user_id | PRIMARY | 8 | database.a.source_id | 1 | 100.00 | Using where |
|----|-------------|-------|------------|--------|----------------------------|---------|---------|----------------------|--------|----------|--------------------------------------------|
and
SELECT *
FROM a
LEFT JOIN b AS b1
ON a.source_id = b1.id AND a.source_type = 'xxxxx'
LEFT JOIN b AS b2
ON a.source_id = b2.id AND a.source_type = 'xxxxx'
WHERE (b1.date = '2021-03-09' OR b2.date = '2021-03-09');
EXPLAIN gives the following
|----|-------------|-------|------------|--------|---------------|---------|---------|----------------------|--------|----------|--------------------------------------------|
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
|----|-------------|-------|------------|--------|---------------|---------|---------|----------------------|--------|----------|--------------------------------------------|
| 1 | SIMPLE | a | NULL | ALL | NULL | NULL | NULL | NULL | 437816 | 100.00 | NULL |
|----|-------------|-------|------------|--------|---------------|---------|---------|----------------------|--------|----------|--------------------------------------------|
| 1 | SIMPLE | b1 | NULL | ALL | PRIMARY | NULL | NULL | NULL | 6 | 100.00 | Using where; Using join buffer (hash join) |
|----|-------------|-------|------------|--------|---------------|---------|---------|----------------------|--------|----------|--------------------------------------------|
| 1 | SIMPLE | b2 | NULL | eq_ref | PRIMARY | PRIMARY | 8 | database.a.source_id | 1 | 100.00 | Using where |
|----|-------------|-------|------------|--------|---------------|---------|---------|----------------------|--------|----------|--------------------------------------------|
The indexes are as follows:
|-------|------------|-----------------------------|--------------|-------------|-----------|-------------|----------|--------|------|------------|---------|---------------|---------|------------|
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment | Visible | Expression |
|-------|------------|-----------------------------|--------------|-------------|-----------|-------------|----------|--------|------|------------|---------|---------------|---------|------------|
| b | 0 | PRIMARY | 1 | id | A | 6 | NULL | NULL | | BTREE | | | YES | NULL |
|-------|------------|-----------------------------|--------------|-------------|-----------|-------------|----------|--------|------|------------|---------|---------------|---------|------------|
| b | 1 | index_b_on_date_and_user_id | 1 | date | A | 5 | NULL | NULL | YES | BTREE | | | YES | NULL |
|-------|------------|-----------------------------|--------------|-------------|-----------|-------------|----------|--------|------|------------|---------|---------------|---------|------------|
| b | 1 | index_b_on_date_and_user_id | 2 | user_id | A | 5 | NULL | NULL | YES | BTREE | | | YES | NULL |
|-------|------------|-----------------------------|--------------|-------------|-----------|-------------|----------|--------|------|------------|---------|---------------|---------|------------|
| b | 1 | index_b_on_user_id | 1 | user_id | A | 1 | NULL | NULL | YES | BTREE | | | YES | NULL |
|-------|------------|-----------------------------|--------------|-------------|-----------|-------------|----------|--------|------|------------|---------|---------------|---------|------------|
| b | 1 | index_b_on_date | 1 | date | A | 5 | NULL | NULL | YES | BTREE | | | YES | NULL |
|-------|------------|-----------------------------|--------------|-------------|-----------|-------------|----------|--------|------|------------|---------|---------------|---------|------------|
| a | 0 | PRIMARY | 1 | id | A | 657 | NULL | NULL | | BTREE | | | YES | NULL |
|-------|------------|-----------------------------|--------------|-------------|-----------|-------------|----------|--------|------|------------|---------|---------------|---------|------------|
| a | 1 | index_a_on_source_id | 1 | source_id | A | 554 | NULL | NULL | YES | BTREE | | | YES | NULL |
|-------|------------|-----------------------------|--------------|-------------|-----------|-------------|----------|--------|------|------------|---------|---------------|---------|------------|
I understand that the queries don't really make sense since they're joining the same table on the same data, but this is a simplified version of a larger query that still exhibits the same behaviour.
The first query with the date runs in 0.0021 seconds, however the second runs in about 2 seconds. I'd like to know why there is such a large discrepancy between the two similar queries and if possible I'd like to improve the bad query. b is indexed on both date and company_id so I'm struggling to see where the difference comes from apart from the data type.
An interesting thing I noticed while playing with the query was that if I removed the OR from the second query so that it became:
SELECT *
FROM a
LEFT JOIN b AS b1
ON a.source_id = b1.id AND a.source_type = 'xxxxx'
LEFT JOIN b AS b2
ON a.source_id = b2.id AND a.source_type = 'xxxxx'
WHERE b1.date = '2021-03-09';
then it only takes 0.00055 seconds to execute.
Related
For some reason, MySQL chooses completely wrong indexes. It feels like it doesn't check which index is the best for the query.
Some indexes on the contacts table:
+----------+------------+---------------------------------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+----------+------------+---------------------------------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| contacts | 0 | PRIMARY | 1 | id | A | 2227424 | NULL | NULL | | BTREE | | |
| contacts | 1 | idx_contacts_date_modfied | 1 | date_modified | A | 261152 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_contacts_id_del | 1 | id | A | 2228229 | NULL | NULL | | BTREE | | |
| contacts | 1 | idx_contacts_id_del | 2 | deleted | A | 2228229 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_contacts_date_entered | 1 | date_entered | A | 286622 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_cont_last_first | 1 | last_name | A | 783981 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_cont_last_first | 2 | first_name | A | 1434526 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_cont_last_first | 3 | deleted | A | 1434526 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_contacts_del_last | 1 | deleted | A | 1 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_contacts_del_last | 2 | last_name | A | 830164 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_cont_del_reports | 1 | deleted | A | 1 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_cont_del_reports | 2 | reports_to_id | A | 1 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_cont_del_reports | 3 | last_name | A | 830164 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_reports_to_id | 1 | reports_to_id | A | 1 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_del_id_user | 1 | deleted | A | 1 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_del_id_user | 2 | id | A | 2228229 | NULL | NULL | | BTREE | | |
| contacts | 1 | idx_del_id_user | 3 | assigned_user_id | A | 2228229 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_cont_assigned | 1 | assigned_user_id | A | 2 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_contact_title | 1 | title | A | 1 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_contact_mkto_id | 1 | mkto_id | A | 1 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_contacts_first_last | 1 | first_name | A | 265736 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_contacts_first_last | 2 | last_name | A | 1453136 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_contacts_first_last | 3 | deleted | A | 1453136 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_contacts_assigned_del | 1 | assigned_user_id | A | 2 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_contacts_assigned_del | 2 | deleted | A | 2 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_contacts_tmst_id | 1 | team_set_id | A | 1 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_contacts_tmst_id | 2 | deleted | A | 1 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_del_date_modified_id | 1 | deleted | A | 1 | NULL | NULL | YES | BTREE | | |
| contacts | 1 | idx_del_date_modified_id | 2 | date_modified | A | 265687 | NULL | NULL | YES | BTREE | | |
+----------+------------+---------------------------------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
The query:
SELECT SQL_NO_CACHE contacts.id,
contacts.date_modified contacts__date_modified
FROM contacts
INNER JOIN
(SELECT tst.team_set_id
FROM team_sets_teams tst
INNER JOIN team_memberships team_membershipscontacts ON (team_membershipscontacts.team_id = tst.team_id)
AND (team_membershipscontacts.user_id = '5daa2e92-c347-11e9-afc5-525400a80916')
AND (team_membershipscontacts.deleted = 0)
GROUP BY tst.team_set_id) contacts_tf ON contacts_tf.team_set_id = contacts.team_set_id
LEFT JOIN contacts_cstm contacts_cstm ON contacts_cstm.id_c = contacts.id
WHERE contacts.deleted = 0
ORDER BY contacts.date_modified DESC,
contacts.id DESC
LIMIT 21;
For some reason, the compiler chooses index idx_contacts_del_last which contains field which is not even in the query! and the query takes around 2 minutes (2M rows).
When I force idx_contacts_date_modfied or idx_del_date_modified_id index, the query takes 0.5s.
For fun, I tried to delete the index idx_contacts_del_last and add it again. After that, mysql chose a DIFFERENT index - idx_reports_to_id implying that MySQL does not even try to choose optimal index and probably chooses the first index which it sees... From my observation the first index, which have field deleted as the first field and was added first is chosen.
So I dropped and recreated all indexes except the one I want the query to use and it finally chooses the correct one. But now I need a different query to use a different index and it still keeps using the one I didn't recreate.
Is there some setting which can make mysql look more thoroughly on index optimization? I Use mysql 5.7.6
EDIT:
The query is system generated and I can't alter it
Explain:
+----+-------------+--------------------------+------------+--------+--------------------------------------------------------------------------------------------------------------------------------+----------------------------+---------+-------------------------------------------+---------+----------+---------------------------------------------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+--------------------------+------------+--------+--------------------------------------------------------------------------------------------------------------------------------+----------------------------+---------+-------------------------------------------+---------+----------+---------------------------------------------------------------------+
| 1 | PRIMARY | contacts | NULL | ref | idx_contacts_del_last,idx_cont_del_reports,idx_del_id_user,idx_contacts_tmst_id,idx_del_date_modified,idx_del_date_modified_id | idx_contacts_del_last | 2 | const | 1114111 | 100.00 | Using temporary; Using filesort |
| 1 | PRIMARY | <derived2> | NULL | ALL | NULL | NULL | NULL | NULL | 2 | 50.00 | Using where; Using join buffer (Block Nested Loop) |
| 1 | PRIMARY | contacts_cstm | NULL | eq_ref | PRIMARY | PRIMARY | 144 | sugarcrm.contacts.id | 1 | 100.00 | Using index |
| 2 | DERIVED | team_membershipscontacts | NULL | ref | idx_team_membership,idx_teammemb_team_user,idx_del_team_user | idx_team_membership | 145 | const | 2 | 99.36 | Using index condition; Using where; Using temporary; Using filesort |
| 2 | DERIVED | tst | NULL | ref | idx_ud_set_id,idx_ud_team_id,idx_ud_team_set_id,idx_ud_team_id_team_set_id | idx_ud_team_id_team_set_id | 144 | sugarcrm.team_membershipscontacts.team_id | 1 | 100.00 | Using index |
+----+-------------+--------------------------+------------+--------+--------------------------------------------------------------------------------------------------------------------------------+----------------------------+---------+-------------------------------------------+---------+----------+---------------------------------------------------------------------+
Turns out there is a bug in mysql https://bugs.mysql.com/bug.php?id=69721
After setting
SET SESSION optimizer_switch='block_nested_loop=off';
The queries fly as a charm.
Have these indexes (in the column-order given):
team_membershipscontacts: (user_id, deleted, team_id)
contacts: (team_set_id, deleted)
team_sets_teams: (team_id, team_set_id)
Remove this; it seems to be a waste of effort:
LEFT JOIN contacts_cstm contacts_cstm ON contacts_cstm.id_c = contacts.id
This issue is fixed in MySQL 8.0.20 (and possibly MySQL 8.0.18 as well however that had a slightly different issue).
controller
#from = params[:from] ? 4.hours.since(Time.zone.parse(params[:from])) : today_start_time
#to = params[:to] ? 4.hours.since(Time.zone.parse(params[:to])) : #from
#franchise = current_user_accessible_franchises_filtered
#orders_type = Restaurant::DELIVERY_TYPES.detect { |t| t == params[:orders_type]} || Restaurant::DELIVERY_TYPES.first
#limit = (params[:limit] || 250).to_i
#to = 48.hours.since(#to) if #to == #from && #from == today_start_time
start_time = #from.utc.strftime('%Y-%m-%d %H:%M')
end_time = 24.hours.since(#to).utc.strftime('%Y-%m-%d %H:%M')
#orders = Order
.includes(:address, :franchise, :customer, :ordered_items, :rests, :driver)
.where(
"((orders.created_at >= ? AND orders.created_at <= ?) OR (orders.delivery_target >= ? AND orders.delivery_target <= ?)) AND orders.franchise_id in (?) AND orders.delivery_type = ?",
start_time, end_time, start_time, end_time, #franchise, #orders_type
)
.order("orders.status desc, orders.id desc")
log
(23732.8ms) SELECT COUNT(DISTINCT orders.id) FROM orders LEFT
OUTER JOIN addresses ON addresses.id = orders.address_id
LEFT OUTER JOIN franchises ON franchises.id =
orders.franchise_id LEFT OUTER JOIN customers ON
customers.id = orders.customer_id LEFT OUTER JOIN
ordered_items ON ordered_items.order_id = orders.id LEFT
OUTER JOIN ordered_items ordered_items_orders_join ON
ordered_items_orders_join.order_id = orders.id LEFT OUTER JOIN
restaurants ON restaurants.id =
ordered_items_orders_join.restaurant_id LEFT OUTER JOIN drivers
ON drivers.id = orders.driver_id WHERE (((orders.created_at >=
'2015-03-01 10:00' AND orders.created_at <= '2015-03-04 10:00') OR
(orders.delivery_target >= '2015-03-01 10:00' AND
orders.delivery_target <= '2015-03-04 10:00')) AND orders.franchise_id
in
(3,31,4,22,37,2,36,17,34,30,19,20,21,18,27,13,25,16,35,24,32,33,1,12,28,23,14,26,29,8,11)
AND orders.delivery_type = 'Mr. Delivery Restaurant')
EXPLAIN EXTENDED
mysql> EXPLAIN EXTENDED
SELECT DISTINCT `orders`.id
FROM `orders`
LEFT OUTER JOIN `addresses` ON `addresses`.`id` = `orders`.`address_id`
LEFT OUTER JOIN `franchises` ON `franchises`.`id` = `orders`.`franchise_id`
LEFT OUTER JOIN `customers` ON `customers`.`id` = `orders`.`customer_id`
LEFT OUTER JOIN `ordered_items` ON `ordered_items`.`order_id` = `orders`.`id`
LEFT OUTER JOIN `ordered_items` `ordered_items_orders_join`
ON `ordered_items_orders_join`.`order_id` = `orders`.`id`
LEFT OUTER JOIN `restaurants` ON `restaurants`.`id` = `ordered_items_orders_join`.`restaurant_id`
LEFT OUTER JOIN `drivers` ON `drivers`.`id` = `orders`.`driver_id`
WHERE ( ( (orders.created_at >= '2015-02-01 10:00'
AND orders.created_at <= '2015-02-04 10:00')
OR (orders.delivery_target >= '2015-02-01 10:00'
AND orders.delivery_target <= '2015-02-04 10:00') )
AND orders.franchise_id in (3,31,4,22,37,2,36,17,34,30,
19,20,21,18,27,13,25,16,35,24,32,33,1,12,28,23,14,26,29, 8,11 )
AND orders.delivery_type = 'Mr. Delivery Restaurant'
)
ORDER BY orders.status desc,
orders.id desc
LIMIT 250;
+----+-------------+---------------------------+--------+------------------------------------------------------------------------------------------------------------------------------+---------------------------------+---------+---------------------------------------------------------+--------+----------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+---------------------------+--------+------------------------------------------------------------------------------------------------------------------------------+---------------------------------+---------+---------------------------------------------------------+--------+----------+----------------------------------------------+
| 1 | SIMPLE | orders | ref | index_orders_on_created_at,index_orders_on_delivery_target,index_orders_on_franchise_id,index_orders_on_delivery_type,my_idx | index_orders_on_delivery_type | 77 | const | 549769 | 100.00 | Using where; Using temporary; Using filesort |
| 1 | SIMPLE | addresses | eq_ref | PRIMARY | PRIMARY | 4 | mrd_staging_new.orders.address_id | 1 | 100.00 | Using index; Distinct |
| 1 | SIMPLE | franchises | eq_ref | PRIMARY | PRIMARY | 4 | mrd_staging_new.orders.franchise_id | 1 | 100.00 | Using index; Distinct |
| 1 | SIMPLE | customers | eq_ref | PRIMARY | PRIMARY | 4 | mrd_staging_new.orders.customer_id | 1 | 100.00 | Using index; Distinct |
| 1 | SIMPLE | ordered_items | ref | index_ordered_items_on_order_id | index_ordered_items_on_order_id | 5 | mrd_staging_new.orders.id | 2 | 100.00 | Using index; Distinct |
| 1 | SIMPLE | ordered_items_orders_join | ref | index_ordered_items_on_order_id | index_ordered_items_on_order_id | 5 | mrd_staging_new.orders.id | 2 | 100.00 | Distinct |
| 1 | SIMPLE | restaurants | eq_ref | PRIMARY | PRIMARY | 4 | mrd_staging_new.ordered_items_orders_join.restaurant_id | 1 | 100.00 | Using index; Distinct |
| 1 | SIMPLE | drivers | eq_ref | PRIMARY | PRIMARY | 4 | mrd_staging_new.orders.driver_id | 1 | 100.00 | Using index; Distinct |
+----+-------------+---------------------------+--------+------------------------------------------------------------------------------------------------------------------------------+---------------------------------+---------+---------------------------------------------------------+--------+----------+----------------------------------------------+
8 rows in set, 1 warning (0.00 sec)
indices on Orders table
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+--------+------------+---------------------------------------+--------------+-----------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| orders | 0 | PRIMARY | 1 | id | A | 935943 | NULL | NULL | | BTREE | | |
| orders | 1 | index_orders_on_delivered_at | 1 | delivered_at | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | index_orders_on_updated_at | 1 | updated_at | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | index_orders_on_created_at | 1 | created_at | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | index_orders_on_pickup_at | 1 | pickup_at | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | index_orders_on_coupon_id | 1 | coupon_id | A | 14856 | NULL | NULL | YES | BTREE | | |
| orders | 1 | index_orders_on_driver_id | 1 | driver_id | A | 4159 | NULL | NULL | YES | BTREE | | |
| orders | 1 | index_orders_on_delivery_target | 1 | delivery_target | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | index_orders_on_franchise_id | 1 | franchise_id | A | 48 | NULL | NULL | YES | BTREE | | |
| orders | 1 | idx_for_customers_report | 1 | customer_id | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | idx_for_customers_report | 2 | delivered_at | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | idx_for_customers_report | 3 | created_at | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | idx_for_customers_report | 4 | status | A | 935943 | NULL | NULL | | BTREE | | |
| orders | 1 | idx_for_customers_report | 5 | franchise_id | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | orders_fetch_index_for_revenue_report | 1 | id | A | 935943 | NULL | NULL | | BTREE | | |
| orders | 1 | orders_fetch_index_for_revenue_report | 2 | created_at | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | orders_fetch_index_for_revenue_report | 3 | delivered_at | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | orders_fetch_index_for_revenue_report | 4 | status | A | 935943 | NULL | NULL | | BTREE | | |
| orders | 1 | orders_fetch_index_for_revenue_report | 5 | franchise_id | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | orders_fetch_index_for_revenue_report | 6 | customer_id | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | orders_fetch_index_for_revenue_report | 7 | address_id | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | index_orders_on_address_id | 1 | address_id | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | index_orders_on_customer_id | 1 | customer_id | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | index_orders_on_delivery_type | 1 | delivery_type | A | 4 | NULL | NULL | | BTREE | | |
| orders | 1 | index_orders_on_status | 1 | status | A | 8 | NULL | NULL | | BTREE | | |
| orders | 1 | index_orders_on_should_pay_restaurant | 1 | should_pay_restaurant | A | 4 | NULL | NULL | YES | BTREE | | |
| orders | 1 | my_idx | 1 | created_at | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | my_idx | 2 | delivery_target | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | my_idx | 3 | franchise_id | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | my_idx | 4 | delivery_type | A | 935943 | NULL | NULL | | BTREE | | |
| orders | 1 | my_idx | 5 | status | A | 935943 | NULL | NULL | | BTREE | | |
| orders | 1 | my_idx | 6 | id | A | 935943 | NULL | NULL | | BTREE | | |
total quantity
mysql> select count(*) from orders;
+----------+
| count(*) |
+----------+
| 965520 |
+----------+
1 row in set (0.32 sec)
I have tried to add a composite index
create index my_idx on orders(created_at, delivery_target, franchise_id, delivery_type);
even
create index my_idx on orders(created_at, delivery_target, franchise_id, delivery_type, status, id);
but without success.
Any ideas?
INDEX(delivery_type, -- because it is "= constant"
franchise_id) -- the only other viable column
Indexing any column hiding in an OR is virtually futile.
I do question your ORDER BY status while doing DISTINCT id and no mention of status.
You may be able to speed it up by turning the OR into a UNION:
( SELECT orders.status, orders.id
FROM ...
JOIN ...
WHERE orders.created_at ...
AND ...
ORDER BY ...
LIMIT 250
)
UNION DISTINCT
( SELECT orders.status, orders.id
FROM ...
JOIN ...
WHERE orders.delivery_target ... -- note difference
AND ...
ORDER BY ...
LIMIT 250
)
ORDER BY status DESC, id DESC -- yes, again
LIMIT 250; -- yes, again
But for this reformulation to work well, there needs to be new indexes. I am not sure whether these
INDEX(delivery_type, franchise_id, created_at),
INDEX(delivery_type, franchise_id, delivery_target),
or these
INDEX(delivery_type, created_at),
INDEX(delivery_type, delivery_target),
would work better. Add all 4; do EXPLAIN to see which ones it uses.
When GROUP BY uid and GROUP BY anonym_id are placed into queries the execution time problem appears > 2sec.
The question is - how to optimize/override this query with effect like GROUP BY statement?
( SELECT u.id as uid, u.name, u.avatar, u.avatar_date, u.driver,
u.vehicle, m.msg, m.removed, (m.date DIV 1000) AS date, m.from_id = 1 AS outbox,
CASE WHEN m.read_state IS NULL THEN 1 ELSE 0 END AS read_state, d.anonym_id
FROM dialog as d CROSS JOIN messages AS m CROSS JOIN users AS u
WHERE (m.id=d.mid AND ((d.uid1 = 1 and u.id = d.uid2) or (d.uid2 = 1 and u.id = d.uid1)))
GROUP BY uid
ORDER BY d.id DESC )
UNION
( SELECT u.id as uid, u.name, u.avatar, u.avatar_date, u.driver, u.vehicle,
m.msg, m.removed, (m.date DIV 1000) AS date, m.from_id = 1 AS outbox,
CASE WHEN m.read_state IS NULL THEN 1 ELSE 0 END AS read_state,
d.anonym_id
FROM dialog as d FORCE KEY(anonym_id) CROSS JOIN messages AS m ON d.mid=m.id
CROSS JOIN users AS u ON u.id=d.uid2
WHERE d.anonym_id=100001
GROUP BY d.uid2
ORDER BY d.id DESC )
LIMIT 0, 20
mysql> show index from users;
+-------+------------+-------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-------+------------+-------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| users | 0 | PRIMARY | 1 | id | A | 606396 | NULL | NULL | | BTREE | | |
| users | 1 | number | 1 | number | A | 606396 | NULL | NULL | | BTREE | | |
| users | 1 | name | 1 | name | A | 121279 | NULL | NULL | | BTREE | | |
| users | 1 | show_phone | 1 | show_phone | A | 10 | NULL | NULL | | BTREE | | |
| users | 1 | show_mail | 1 | show_mail | A | 14 | NULL | NULL | | BTREE | | |
| users | 1 | show_on_map | 1 | show_on_map | A | 18 | NULL | NULL | | BTREE | | |
| users | 1 | show_on_map | 2 | map_activity | A | 606396 | NULL | NULL | | BTREE | | |
| users | 1 | udid | 1 | udid | A | 606396 | NULL | NULL | YES | BTREE | | |
| users | 1 | blocked | 1 | blocked | A | 10 | NULL | NULL | | BTREE | | |
+-------+------------+-------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
mysql> show index from dialog;
+--------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+--------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| dialog | 0 | PRIMARY | 1 | id | A | 2964484 | NULL | NULL | | BTREE | | |
| dialog | 1 | uid1 | 1 | uid1 | A | 211748 | NULL | NULL | | BTREE | | |
| dialog | 1 | uid1 | 2 | uid2 | A | 2964484 | NULL | NULL | | BTREE | | |
| dialog | 1 | uid2 | 1 | uid2 | A | 1482242 | NULL | NULL | | BTREE | | |
| dialog | 1 | anonym_id | 1 | anonym_id | A | 18 | NULL | NULL | | BTREE | | |
| dialog | 1 | idx_mid | 1 | mid | A | 200 | NULL | NULL | | BTREE | | |
+--------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
mysql> show index from messages;
+----------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+----------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| messages | 0 | PRIMARY | 1 | id | A | 23227116 | NULL | NULL | | BTREE | | |
| messages | 1 | user_id_2 | 1 | user_id | A | 188 | NULL | NULL | | BTREE | | |
| messages | 1 | user_id_2 | 2 | read_state | A | 188 | NULL | NULL | | BTREE | | |
| messages | 1 | user_id_2 | 3 | removed | A | 188 | NULL | NULL | | BTREE | | |
| messages | 1 | from_id | 1 | from_id | A | 188 | NULL | NULL | | BTREE | | |
| messages | 1 | from_id | 2 | to_number | A | 188 | NULL | NULL | | BTREE | | |
+----------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
EXPLAIN EXTENDED
1 PRIMARY d index_merge uid1,uid2,idx_mid uid1,uid2 4,4 26 100.00 Using sort_union(uid1,uid2); Using where; Using temporary; Using filesort
1 PRIMARY u ALL PRIMARY 625866 100.00 Using where; Using join buffer
1 PRIMARY m eq_ref PRIMARY PRIMARY 4 numbers.d.mid 1 100.00
2 UNION d ref anonym_id anonym_id 4 const 1 100.00
2 UNION u eq_ref PRIMARY PRIMARY 4 numbers.d.uid2 1 100.00
2 UNION m eq_ref PRIMARY PRIMARY 4 numbers.d.mid 1 100.00
UNION RESULT <union1,2> ALL
Here is the first table 'tbl1':
+---------+---------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+---------+---------------------+------+-----+---------+----------------+
| val | varchar(45) | YES | MUL | NULL | |
| id | bigint(20) unsigned | NO | PRI | NULL | auto_increment |
+---------+---------------------+------+-----+---------+----------------+
With its indexes:
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| tbl1 | 0 | PRIMARY | 1 | id | A | 201826018 | NULL | NULL | | BTREE | |
| tbl1 | 1 | val | 1 | val | A | 2147085 | NULL | NULL | YES | BTREE | |
| tbl1 | 1 | id_val | 1 | id | A | 201826018 | NULL | NULL | | BTREE | |
| tbl1 | 1 | id_val | 2 | val | A | 201826018 | NULL | NULL | YES | BTREE | |
| tbl1 | 1 | val_id | 1 | val | A | 2147085 | NULL | NULL | YES | BTREE | |
| tbl1 | 1 | val_id | 2 | id | A | 201826018 | NULL | NULL | | BTREE | |
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
(The reason for some extra indexing is this: http://bit.ly/KWx1Xz.)
The second table is just about the same. Here are its index cardinalities though:
+--------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+--------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| tbl2 | 0 | PRIMARY | 1 | id | A | 201826018 | NULL | NULL | | BTREE | |
| tbl2 | 1 | val | 1 | val | A | 881336 | NULL | NULL | YES | BTREE | |
| tbl2 | 1 | id_val | 1 | id | A | 201826018 | NULL | NULL | | BTREE | |
| tbl2 | 1 | id_val | 2 | val | A | 201826018 | NULL | NULL | YES | BTREE | |
| tbl2 | 1 | val_id | 1 | val | A | 881336 | NULL | NULL | YES | BTREE | |
| tbl2 | 1 | val_id | 2 | id | A | 201826018 | NULL | NULL | | BTREE | |
+--------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
The task is to inner-join them on the val column and get the id's list (and to do it in 1 second).
Here is the 'join' approach:
SELECT tbl1.id FROM tbl1 JOIN tbl2 ON tbl1.val = 'iii' AND tbl2.val = 'iii' AND tbl1.id = tbl2.id;
Result: 10831 rows in set (55.15 sec)
Query explain:
+----+-------------+--------+--------+----------------------------------+---------+---------+---------------------------+------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+--------+--------+----------------------------------+---------+---------+---------------------------+------+--------------------------+
| 1 | SIMPLE | tbl1 | ref | PRIMARY,val,id_val,val_id | val_id | 138 | const | 5160 | Using where; Using index |
| 1 | SIMPLE | tbl2 | eq_ref | PRIMARY,val,id_val,val_id | PRIMARY | 8 | search_test.tbl1.id | 1 | Using where |
+----+-------------+--------+--------+----------------------------------+---------+---------+---------------------------+------+--------------------------+
And here is the 'in' approach:
SELECT id FROM tbl1 WHERE val = 'iii' and id IN (SELECT id FROM tbl2 WHERE val = 'iii');
Result: 10831 rows in set (1 min 10.15 sec)
Explain:
+----+--------------------+--------+-----------------+---------------------------------+---------+---------+-------+------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+--------------------+--------+-----------------+---------------------------------+---------+---------+-------+------+--------------------------+
| 1 | PRIMARY | tbl1 | ref | val,val_id | val_id | 138 | const | 8553 | Using where; Using index |
| 2 | DEPENDENT SUBQUERY | tbl2 | unique_subquery | PRIMARY,val,id_val,val_id | PRIMARY | 8 | func | 1 | Using where |
+----+--------------------+--------+-----------------+---------------------------------+---------+---------+-------+------+--------------------------+
So, here is the question: how to tweak this query to let MySQL accomplish it in a second?
OK I have tested this on 30,000+ records per table and it runs pretty darn quick.
As it currently stands you're performing a join on two massive tables presently but if you scan for matches on 'val' on each table first that will reduce the size of your join sets substantially.
I originally posted this answer as a set of subqueries but I didn't realize that MySQL is painfully slow at nested subqueries since it executes from the outside in. However if you define the subqueries as views it runs them from the inside out.
So, first create the views.
CREATE VIEW tbl1_iii AS (
SELECT * FROM tbl1 WHERE val='iii'
);
CREATE VIEW tbl2_iii AS (
SELECT * FROM tbl2 WHERE val='iii'
);
Then run the query.
SELECT tbl1_iii.id from tbl1_iii,tbl2_iii
WHERE tbl1_iii.id = tbl2_iii.id;
Lightning.
SELECT tbl1.id FROM tbl1 JOIN tbl2 ON tbl1.id = tbl2.id and tbl1.val = tbl2.val
where tbl1.val = 'iii';
I need to find the best index for this query:
SELECT c.id id, type
FROM Content c USE INDEX (type_proc_last_cat)
LEFT JOIN Battles b ON c.id = b.id
WHERE type = 1
AND processing_status = 1
AND category IN (13, 19)
AND status = 4
ORDER BY last_change DESC
LIMIT 100";
The tables look like this:
mysql> describe Content;
+-------------------+---------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------------+---------------------+------+-----+---------+-------+
| id | bigint(20) unsigned | NO | PRI | NULL | |
| type | tinyint(3) unsigned | NO | MUL | NULL | |
| category | bigint(20) unsigned | NO | | NULL | |
| processing_status | tinyint(3) unsigned | NO | | NULL | |
| last_change | int(10) unsigned | NO | | NULL | |
+-------------------+---------------------+------+-----+---------+-------+
mysql> show indexes from Content;
+---------+------------+---------------------+--------------+-------------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+---------+------------+---------------------+--------------+-------------------+-----------+-------------+----------+--------+------+------------+---------+
| Content | 0 | PRIMARY | 1 | id | A | 4115 | NULL | NULL | | BTREE | |
| Content | 1 | type_proc_last_cat | 1 | type | A | 4 | NULL | NULL | | BTREE | |
| Content | 1 | type_proc_last_cat | 2 | processing_status | A | 20 | NULL | NULL | | BTREE | |
| Content | 1 | type_proc_last_cat | 3 | last_change | A | 4115 | NULL | NULL | | BTREE | |
| Content | 1 | type_proc_last_cat | 4 | category | A | 4115 | NULL | NULL | | BTREE | |
+---------+------------+---------------------+--------------+-------------------+-----------+-------------+----------+--------+------+------------+---------+
mysql> describe Battles;
+---------------------+---------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+---------------------+---------------------+------+-----+---------+-------+
| id | bigint(20) unsigned | NO | PRI | NULL | |
| status | tinyint(4) unsigned | NO | | NULL | |
| status_last_changed | int(11) unsigned | NO | | NULL | |
+---------------------+---------------------+------+-----+---------+-------+
mysql> show indexes from Battles;
+---------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+---------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Battles | 0 | PRIMARY | 1 | id | A | 1215 | NULL | NULL | | BTREE | |
| Battles | 0 | id_status | 1 | id | A | 1215 | NULL | NULL | | BTREE | |
| Battles | 0 | id_status | 2 | status | A | 1215 | NULL | NULL | | BTREE | |
+---------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
And I get output like this:
mysql> explain
-> SELECT c.id id, type
-> FROM Content c USE INDEX (type_proc_last_cat)
-> LEFT JOIN Battles b USE INDEX (id_status) ON c.id = b.id
-> WHERE type = 1
-> AND processing_status = 1
-> AND category IN (13, 19)
-> AND status = 4
-> ORDER BY last_change DESC
-> LIMIT 100;
+----+-------------+-------+--------+--------------------+--------------------+---------+-----------------------+------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+--------------------+--------------------+---------+-----------------------+------+--------------------------+
| 1 | SIMPLE | c | ref | type_proc_last_cat | type_proc_last_cat | 2 | const,const | 1352 | Using where; Using index |
| 1 | SIMPLE | b | eq_ref | id_status | id_status | 9 | wtm_master.c.id,const | 1 | Using where; Using index |
+----+-------------+-------+--------+--------------------+--------------------+---------+-----------------------+------+--------------------------+
The trouble is the rows count for the Content table. It appears MySQL is unable to effectively use both last_change and category in the type_proc_last_cat index. If I switch the order of last_change and category, fewer rows are selected but it results in a filesort for the ORDER BY, meaning it pulls all the matching rows from the database. That is worse, since there are 100,000+ rows in both tables.
Tables are both InnoDB, so keep in mind that the PRIMARY key is appended to every other index. So the index the index type_proc_last_cat above behaves likes it's on (type, processing_status, last_change, category, id). I am aware I could change the PRIMARY key for Battles to (id, status) and drop the id_status index (and I may just do that).
Edit: Any value for type, category, processing_status, and status is less than 20% of the total values. last_change and status_last_change are unix timestamps.
Edit: If I use a different index with category and last_change in reverse order, I get this:
mysql> show indexes from Content;
+---------+------------+---------------------+--------------+-------------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+---------+------------+---------------------+--------------+-------------------+-----------+-------------+----------+--------+------+------------+---------+
| Content | 0 | PRIMARY | 1 | id | A | 4115 | NULL | NULL | | BTREE | |
| Content | 1 | type_proc_cat_last | 1 | type | A | 6 | NULL | NULL | | BTREE | |
| Content | 1 | type_proc_cat_last | 2 | processing_status | A | 26 | NULL | NULL | | BTREE | |
| Content | 1 | type_proc_cat_last | 3 | category | A | 228 | NULL | NULL | | BTREE | |
| Content | 1 | type_proc_cat_last | 4 | last_change | A | 4115 | NULL | NULL | | BTREE | |
+---------+------------+---------------------+--------------+-------------------+-----------+-------------+----------+--------+------+------------+---------+
mysql> explain SELECT c.id id, type FROM Content c USE INDEX (type_proc_cat_last) LEFT JOIN Battles b
USE INDEX (id_status) ON c.id = b.id WHERE type = 1 AND processing_status = 1 AND category IN (13, 19) AND status = 4 ORDER BY last_change DESC LIMIT 100;
+----+-------------+-------+-------+--------------------+--------------------+---------+-----------------------+------+------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+--------------------+--------------------+---------+-----------------------+------+------------------------------------------+
| 1 | SIMPLE | c | range | type_proc_cat_last | type_proc_cat_last | 10 | NULL | 165 | Using where; Using index; Using filesort |
| 1 | SIMPLE | b | ref | id_status | id_status | 9 | wtm_master.c.id,const | 1 | Using where; Using index |
+----+-------------+-------+-------+--------------------+--------------------+---------+-----------------------+------+------------------------------------------+
The filesort worries me as it tells me MySQL pulls all the matching rows first, before sorting. This will be a big problem when there are 100,000+.
rows field in EXPLAIN doesn't reflect the number of rows that actually were read. It reflects the number of rows that possible will be affected. Also it doesn't rely on LIMIT, because LIMIT is applied after the plan has been calculated.
So you don't need to worry about it.
Also I would suggest you to swap last_change and category in type_proc_last_cat so mysql can try to use the last index part (last_change) for sorting purposes.