Optimize query to GROUP BY uid with UNION - mysql

When GROUP BY uid and GROUP BY anonym_id are placed into queries the execution time problem appears > 2sec.
The question is - how to optimize/override this query with effect like GROUP BY statement?
( SELECT u.id as uid, u.name, u.avatar, u.avatar_date, u.driver,
u.vehicle, m.msg, m.removed, (m.date DIV 1000) AS date, m.from_id = 1 AS outbox,
CASE WHEN m.read_state IS NULL THEN 1 ELSE 0 END AS read_state, d.anonym_id
FROM dialog as d CROSS JOIN messages AS m CROSS JOIN users AS u
WHERE (m.id=d.mid AND ((d.uid1 = 1 and u.id = d.uid2) or (d.uid2 = 1 and u.id = d.uid1)))
GROUP BY uid
ORDER BY d.id DESC )
UNION
( SELECT u.id as uid, u.name, u.avatar, u.avatar_date, u.driver, u.vehicle,
m.msg, m.removed, (m.date DIV 1000) AS date, m.from_id = 1 AS outbox,
CASE WHEN m.read_state IS NULL THEN 1 ELSE 0 END AS read_state,
d.anonym_id
FROM dialog as d FORCE KEY(anonym_id) CROSS JOIN messages AS m ON d.mid=m.id
CROSS JOIN users AS u ON u.id=d.uid2
WHERE d.anonym_id=100001
GROUP BY d.uid2
ORDER BY d.id DESC )
LIMIT 0, 20
mysql> show index from users;
+-------+------------+-------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-------+------------+-------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| users | 0 | PRIMARY | 1 | id | A | 606396 | NULL | NULL | | BTREE | | |
| users | 1 | number | 1 | number | A | 606396 | NULL | NULL | | BTREE | | |
| users | 1 | name | 1 | name | A | 121279 | NULL | NULL | | BTREE | | |
| users | 1 | show_phone | 1 | show_phone | A | 10 | NULL | NULL | | BTREE | | |
| users | 1 | show_mail | 1 | show_mail | A | 14 | NULL | NULL | | BTREE | | |
| users | 1 | show_on_map | 1 | show_on_map | A | 18 | NULL | NULL | | BTREE | | |
| users | 1 | show_on_map | 2 | map_activity | A | 606396 | NULL | NULL | | BTREE | | |
| users | 1 | udid | 1 | udid | A | 606396 | NULL | NULL | YES | BTREE | | |
| users | 1 | blocked | 1 | blocked | A | 10 | NULL | NULL | | BTREE | | |
+-------+------------+-------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
mysql> show index from dialog;
+--------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+--------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| dialog | 0 | PRIMARY | 1 | id | A | 2964484 | NULL | NULL | | BTREE | | |
| dialog | 1 | uid1 | 1 | uid1 | A | 211748 | NULL | NULL | | BTREE | | |
| dialog | 1 | uid1 | 2 | uid2 | A | 2964484 | NULL | NULL | | BTREE | | |
| dialog | 1 | uid2 | 1 | uid2 | A | 1482242 | NULL | NULL | | BTREE | | |
| dialog | 1 | anonym_id | 1 | anonym_id | A | 18 | NULL | NULL | | BTREE | | |
| dialog | 1 | idx_mid | 1 | mid | A | 200 | NULL | NULL | | BTREE | | |
+--------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
mysql> show index from messages;
+----------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+----------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| messages | 0 | PRIMARY | 1 | id | A | 23227116 | NULL | NULL | | BTREE | | |
| messages | 1 | user_id_2 | 1 | user_id | A | 188 | NULL | NULL | | BTREE | | |
| messages | 1 | user_id_2 | 2 | read_state | A | 188 | NULL | NULL | | BTREE | | |
| messages | 1 | user_id_2 | 3 | removed | A | 188 | NULL | NULL | | BTREE | | |
| messages | 1 | from_id | 1 | from_id | A | 188 | NULL | NULL | | BTREE | | |
| messages | 1 | from_id | 2 | to_number | A | 188 | NULL | NULL | | BTREE | | |
+----------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
EXPLAIN EXTENDED
1 PRIMARY d index_merge uid1,uid2,idx_mid uid1,uid2 4,4 26 100.00 Using sort_union(uid1,uid2); Using where; Using temporary; Using filesort
1 PRIMARY u ALL PRIMARY 625866 100.00 Using where; Using join buffer
1 PRIMARY m eq_ref PRIMARY PRIMARY 4 numbers.d.mid 1 100.00
2 UNION d ref anonym_id anonym_id 4 const 1 100.00
2 UNION u eq_ref PRIMARY PRIMARY 4 numbers.d.uid2 1 100.00
2 UNION m eq_ref PRIMARY PRIMARY 4 numbers.d.mid 1 100.00
UNION RESULT <union1,2> ALL

Related

what index to use for query

controller
#from = params[:from] ? 4.hours.since(Time.zone.parse(params[:from])) : today_start_time
#to = params[:to] ? 4.hours.since(Time.zone.parse(params[:to])) : #from
#franchise = current_user_accessible_franchises_filtered
#orders_type = Restaurant::DELIVERY_TYPES.detect { |t| t == params[:orders_type]} || Restaurant::DELIVERY_TYPES.first
#limit = (params[:limit] || 250).to_i
#to = 48.hours.since(#to) if #to == #from && #from == today_start_time
start_time = #from.utc.strftime('%Y-%m-%d %H:%M')
end_time = 24.hours.since(#to).utc.strftime('%Y-%m-%d %H:%M')
#orders = Order
.includes(:address, :franchise, :customer, :ordered_items, :rests, :driver)
.where(
"((orders.created_at >= ? AND orders.created_at <= ?) OR (orders.delivery_target >= ? AND orders.delivery_target <= ?)) AND orders.franchise_id in (?) AND orders.delivery_type = ?",
start_time, end_time, start_time, end_time, #franchise, #orders_type
)
.order("orders.status desc, orders.id desc")
log
(23732.8ms) SELECT COUNT(DISTINCT orders.id) FROM orders LEFT
OUTER JOIN addresses ON addresses.id = orders.address_id
LEFT OUTER JOIN franchises ON franchises.id =
orders.franchise_id LEFT OUTER JOIN customers ON
customers.id = orders.customer_id LEFT OUTER JOIN
ordered_items ON ordered_items.order_id = orders.id LEFT
OUTER JOIN ordered_items ordered_items_orders_join ON
ordered_items_orders_join.order_id = orders.id LEFT OUTER JOIN
restaurants ON restaurants.id =
ordered_items_orders_join.restaurant_id LEFT OUTER JOIN drivers
ON drivers.id = orders.driver_id WHERE (((orders.created_at >=
'2015-03-01 10:00' AND orders.created_at <= '2015-03-04 10:00') OR
(orders.delivery_target >= '2015-03-01 10:00' AND
orders.delivery_target <= '2015-03-04 10:00')) AND orders.franchise_id
in
(3,31,4,22,37,2,36,17,34,30,19,20,21,18,27,13,25,16,35,24,32,33,1,12,28,23,14,26,29,8,11)
AND orders.delivery_type = 'Mr. Delivery Restaurant')
EXPLAIN EXTENDED
mysql> EXPLAIN EXTENDED
SELECT DISTINCT `orders`.id
FROM `orders`
LEFT OUTER JOIN `addresses` ON `addresses`.`id` = `orders`.`address_id`
LEFT OUTER JOIN `franchises` ON `franchises`.`id` = `orders`.`franchise_id`
LEFT OUTER JOIN `customers` ON `customers`.`id` = `orders`.`customer_id`
LEFT OUTER JOIN `ordered_items` ON `ordered_items`.`order_id` = `orders`.`id`
LEFT OUTER JOIN `ordered_items` `ordered_items_orders_join`
ON `ordered_items_orders_join`.`order_id` = `orders`.`id`
LEFT OUTER JOIN `restaurants` ON `restaurants`.`id` = `ordered_items_orders_join`.`restaurant_id`
LEFT OUTER JOIN `drivers` ON `drivers`.`id` = `orders`.`driver_id`
WHERE ( ( (orders.created_at >= '2015-02-01 10:00'
AND orders.created_at <= '2015-02-04 10:00')
OR (orders.delivery_target >= '2015-02-01 10:00'
AND orders.delivery_target <= '2015-02-04 10:00') )
AND orders.franchise_id in (3,31,4,22,37,2,36,17,34,30,
19,20,21,18,27,13,25,16,35,24,32,33,1,12,28,23,14,26,29, 8,11 )
AND orders.delivery_type = 'Mr. Delivery Restaurant'
)
ORDER BY orders.status desc,
orders.id desc
LIMIT 250;
+----+-------------+---------------------------+--------+------------------------------------------------------------------------------------------------------------------------------+---------------------------------+---------+---------------------------------------------------------+--------+----------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+---------------------------+--------+------------------------------------------------------------------------------------------------------------------------------+---------------------------------+---------+---------------------------------------------------------+--------+----------+----------------------------------------------+
| 1 | SIMPLE | orders | ref | index_orders_on_created_at,index_orders_on_delivery_target,index_orders_on_franchise_id,index_orders_on_delivery_type,my_idx | index_orders_on_delivery_type | 77 | const | 549769 | 100.00 | Using where; Using temporary; Using filesort |
| 1 | SIMPLE | addresses | eq_ref | PRIMARY | PRIMARY | 4 | mrd_staging_new.orders.address_id | 1 | 100.00 | Using index; Distinct |
| 1 | SIMPLE | franchises | eq_ref | PRIMARY | PRIMARY | 4 | mrd_staging_new.orders.franchise_id | 1 | 100.00 | Using index; Distinct |
| 1 | SIMPLE | customers | eq_ref | PRIMARY | PRIMARY | 4 | mrd_staging_new.orders.customer_id | 1 | 100.00 | Using index; Distinct |
| 1 | SIMPLE | ordered_items | ref | index_ordered_items_on_order_id | index_ordered_items_on_order_id | 5 | mrd_staging_new.orders.id | 2 | 100.00 | Using index; Distinct |
| 1 | SIMPLE | ordered_items_orders_join | ref | index_ordered_items_on_order_id | index_ordered_items_on_order_id | 5 | mrd_staging_new.orders.id | 2 | 100.00 | Distinct |
| 1 | SIMPLE | restaurants | eq_ref | PRIMARY | PRIMARY | 4 | mrd_staging_new.ordered_items_orders_join.restaurant_id | 1 | 100.00 | Using index; Distinct |
| 1 | SIMPLE | drivers | eq_ref | PRIMARY | PRIMARY | 4 | mrd_staging_new.orders.driver_id | 1 | 100.00 | Using index; Distinct |
+----+-------------+---------------------------+--------+------------------------------------------------------------------------------------------------------------------------------+---------------------------------+---------+---------------------------------------------------------+--------+----------+----------------------------------------------+
8 rows in set, 1 warning (0.00 sec)
indices on Orders table
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+--------+------------+---------------------------------------+--------------+-----------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| orders | 0 | PRIMARY | 1 | id | A | 935943 | NULL | NULL | | BTREE | | |
| orders | 1 | index_orders_on_delivered_at | 1 | delivered_at | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | index_orders_on_updated_at | 1 | updated_at | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | index_orders_on_created_at | 1 | created_at | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | index_orders_on_pickup_at | 1 | pickup_at | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | index_orders_on_coupon_id | 1 | coupon_id | A | 14856 | NULL | NULL | YES | BTREE | | |
| orders | 1 | index_orders_on_driver_id | 1 | driver_id | A | 4159 | NULL | NULL | YES | BTREE | | |
| orders | 1 | index_orders_on_delivery_target | 1 | delivery_target | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | index_orders_on_franchise_id | 1 | franchise_id | A | 48 | NULL | NULL | YES | BTREE | | |
| orders | 1 | idx_for_customers_report | 1 | customer_id | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | idx_for_customers_report | 2 | delivered_at | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | idx_for_customers_report | 3 | created_at | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | idx_for_customers_report | 4 | status | A | 935943 | NULL | NULL | | BTREE | | |
| orders | 1 | idx_for_customers_report | 5 | franchise_id | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | orders_fetch_index_for_revenue_report | 1 | id | A | 935943 | NULL | NULL | | BTREE | | |
| orders | 1 | orders_fetch_index_for_revenue_report | 2 | created_at | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | orders_fetch_index_for_revenue_report | 3 | delivered_at | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | orders_fetch_index_for_revenue_report | 4 | status | A | 935943 | NULL | NULL | | BTREE | | |
| orders | 1 | orders_fetch_index_for_revenue_report | 5 | franchise_id | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | orders_fetch_index_for_revenue_report | 6 | customer_id | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | orders_fetch_index_for_revenue_report | 7 | address_id | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | index_orders_on_address_id | 1 | address_id | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | index_orders_on_customer_id | 1 | customer_id | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | index_orders_on_delivery_type | 1 | delivery_type | A | 4 | NULL | NULL | | BTREE | | |
| orders | 1 | index_orders_on_status | 1 | status | A | 8 | NULL | NULL | | BTREE | | |
| orders | 1 | index_orders_on_should_pay_restaurant | 1 | should_pay_restaurant | A | 4 | NULL | NULL | YES | BTREE | | |
| orders | 1 | my_idx | 1 | created_at | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | my_idx | 2 | delivery_target | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | my_idx | 3 | franchise_id | A | 935943 | NULL | NULL | YES | BTREE | | |
| orders | 1 | my_idx | 4 | delivery_type | A | 935943 | NULL | NULL | | BTREE | | |
| orders | 1 | my_idx | 5 | status | A | 935943 | NULL | NULL | | BTREE | | |
| orders | 1 | my_idx | 6 | id | A | 935943 | NULL | NULL | | BTREE | | |
total quantity
mysql> select count(*) from orders;
+----------+
| count(*) |
+----------+
| 965520 |
+----------+
1 row in set (0.32 sec)
I have tried to add a composite index
create index my_idx on orders(created_at, delivery_target, franchise_id, delivery_type);
even
create index my_idx on orders(created_at, delivery_target, franchise_id, delivery_type, status, id);
but without success.
Any ideas?
INDEX(delivery_type, -- because it is "= constant"
franchise_id) -- the only other viable column
Indexing any column hiding in an OR is virtually futile.
I do question your ORDER BY status while doing DISTINCT id and no mention of status.
You may be able to speed it up by turning the OR into a UNION:
( SELECT orders.status, orders.id
FROM ...
JOIN ...
WHERE orders.created_at ...
AND ...
ORDER BY ...
LIMIT 250
)
UNION DISTINCT
( SELECT orders.status, orders.id
FROM ...
JOIN ...
WHERE orders.delivery_target ... -- note difference
AND ...
ORDER BY ...
LIMIT 250
)
ORDER BY status DESC, id DESC -- yes, again
LIMIT 250; -- yes, again
But for this reformulation to work well, there needs to be new indexes. I am not sure whether these
INDEX(delivery_type, franchise_id, created_at),
INDEX(delivery_type, franchise_id, delivery_target),
or these
INDEX(delivery_type, created_at),
INDEX(delivery_type, delivery_target),
would work better. Add all 4; do EXPLAIN to see which ones it uses.

Optimize sql statement with OR in where clause

running this query on 5Gb sized messages table.
The problem is that the execution takes > 3 minutes.
SELECT m.id FROM messages m
LEFT JOIN dialog d on m.id=d.mid
WHERE (SELECT count(*)
FROM dialog
WHERE (m.from_id=uid1 and m.user_id=uid2)
OR (m.from_id=uid2 and m.user_id=uid1))=0 && read_state=0
LIMIT 100
I understand, that this is a bad practice to search by NESTED SELECT IN WHERE CLAUSE, but yet didn`t find another way to select such rows. Tried to split OR to 2 UNION statements, but it was long either.
messages table structure:
+------------+-------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------+-------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| from_id | int(11) | NO | MUL | NULL | |
| user_id | int(11) | NO | MUL | NULL | |
| group_id | int(11) | NO | | NULL | |
| to_number | varchar(30) | NO | MUL | NULL | |
| msg | text | NO | | NULL | |
| image | varchar(20) | NO | | NULL | |
| date | bigint(20) | NO | | NULL | |
| read_state | tinyint(1) | NO | | 0 | |
| removed | tinyint(1) | NO | | NULL | |
+------------+-------------+------+-----+---------+----------------+
dialog table structure
+-----------+------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------+------------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| uid1 | int(11) | NO | MUL | NULL | |
| uid2 | int(11) | NO | MUL | NULL | |
| mid | int(11) | NO | | NULL | |
| anonym_id | int(10) unsigned | NO | | NULL | |
+-----------+------------------+------+-----+---------+----------------+
mysql> show index from messages;
+----------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+----------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| messages | 0 | PRIMARY | 1 | id | A | 12560908 | NULL | NULL | | BTREE | | |
| messages | 1 | to_number | 1 | to_number | A | 161037 | NULL | NULL | | BTREE | | |
| messages | 1 | from_id | 1 | from_id | A | 157011 | NULL | NULL | | BTREE | | |
| messages | 1 | from_id | 2 | to_number | A | 169742 | NULL | NULL | | BTREE | | |
| messages | 1 | user_id_2 | 1 | user_id | A | 314022 | NULL | NULL | | BTREE | | |
| messages | 1 | user_id_2 | 2 | read_state | A | 380633 | NULL | NULL | | BTREE | | |
| messages | 1 | user_id_2 | 3 | removed | A | 392528 | NULL | NULL | | BTREE | | |
+----------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
mysql> show index from dialog;
+--------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+--------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| dialog | 0 | PRIMARY | 1 | id | A | 3125615 | NULL | NULL | | BTREE | | |
| dialog | 1 | uid1 | 1 | uid1 | A | 520935 | NULL | NULL | | BTREE | | |
| dialog | 1 | uid1 | 2 | uid2 | A | 3125615 | NULL | NULL | | BTREE | | |
| dialog | 1 | uid2 | 1 | uid2 | A | 1562807 | NULL | NULL | | BTREE | | |
+--------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
EXPLAIN EXTENDED
+----+--------------------+--------+-------+---------------+------+---------+------+----------+----------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+--------------------+--------+-------+---------------+------+---------+------+----------+----------+--------------------------+
| 1 | PRIMARY | m | ALL | NULL | NULL | NULL | NULL | 22190398 | 100.00 | Using where |
| 1 | PRIMARY | d | ALL | NULL | NULL | NULL | NULL | 3125621 | 100.00 | |
| 2 | DEPENDENT SUBQUERY | dialog | index | uid1,uid2 | uid1 | 8 | NULL | 3125621 | 100.00 | Using where; Using index |
+----+--------------------+--------+-------+---------------+------+---------+------+----------+----------+--------------------------+
The first thought is to change the subquery query to use not exists instead of count(*). The second is to split this into two separate subqueries. The third is to add indexes:
create index idx_messages_read_state_4 on messages_read_state(user_id, from_id, user_id, id);
create index idx_dialog_2 on dialog(uid1, uid2)
And the fourth is to remove the left join to dialog in the outer query. No fields from dialog are being used and the left join means it is not being used for filtering.
The query is then:
select m.id
from messages m
where m.read_state = 0 and
not exists (select 1
from dialog d
where m.from_id = d.uid1 and m.user_id = d.uid2
) and
not exists (select 1
from dialog d
where m.from_id = d.uid2 and m.user_id = d.uid1
)
limit 100;

Mysql query optimisation - range searches

Looking to find out some ideas on how to optimise this query - its pretty awkward with several range searches
SELECT t1.id, t1.custom_track_id, t1.deleted, t1.last_modified, t1.client_id
FROM tracks t1
INNER JOIN tracks t2 ON t1.custom_track_id = t2.custom_track_id
AND t1.last_modified > t2.last_modified
AND t1.deleted !=0
AND t2.deleted =0
AND t2.client_id
IN ( 103, 115, 116 )
WHERE t1.client_id
IN ( 103, 115, 116 )
All the fields its looking to find and joining on are INT fields.
Indexes (yes theres a few dodgy ones in there) :
+--------+------------+------------------------------------------------+--------------+----------------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+--------+------------+------------------------------------------------+--------------+----------------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| tracks | 0 | PRIMARY | 1 | id | A | 566045 | NULL | NULL | | BTREE | | |
| tracks | 1 | client_id | 1 | client_id | A | 589 | NULL | NULL | | BTREE | | |
| tracks | 1 | featured | 1 | featured | A | 16 | NULL | NULL | | BTREE | | |
| tracks | 1 | system_status | 1 | system_status | A | 20 | NULL | NULL | | BTREE | | |
| tracks | 1 | created_by | 1 | created_by | A | 225 | NULL | NULL | | BTREE | | |
| tracks | 1 | custom_track_id | 1 | custom_track_id | A | 283022 | NULL | NULL | | BTREE | | |
| tracks | 1 | custom_track_id | 2 | custom_artist_id | A | 283022 | NULL | NULL | | BTREE | | |
| tracks | 1 | counterpoint_id | 1 | counterpoint_id | A | 113209 | NULL | NULL | YES | BTREE | | |
| tracks | 1 | system_status_2 | 1 | system_status | A | 20 | NULL | NULL | | BTREE | | |
| tracks | 1 | composition | 1 | composition | A | 13 | NULL | NULL | YES | BTREE | | |
| tracks | 1 | published_start_date | 1 | published_start_date | A | 13 | NULL | NULL | YES | BTREE | | |
| tracks | 1 | published_end_date | 1 | published_end_date | A | 13 | NULL | NULL | YES | BTREE | | |
| tracks | 1 | deleted | 1 | deleted | A | 20 | NULL | NULL | | BTREE | | |
| tracks | 1 | restricted_access_required | 1 | restricted_access_required | A | 13 | NULL | NULL | | BTREE | | |
| tracks | 1 | mv_clientid_deleted_featured_restrictedaccess | 1 | client_id | A | 336 | NULL | NULL | | BTREE | | |
| tracks | 1 | mv_clientid_deleted_featured_restrictedaccess | 2 | custom_track_id | A | 336 | NULL | NULL | | BTREE | | |
| tracks | 1 | mv_clientid_deleted_featured_restrictedaccess | 3 | deleted | A | 336 | NULL | NULL | | BTREE | | |
| tracks | 1 | mv_clientid_deleted_featured_restrictedaccess | 4 | last_modified | A | 336 | NULL | NULL | | BTREE | | |
| tracks | 1 | mv_clientid_customtrackid_deleted_lastmodified | 1 | client_id | A | 336 | NULL | NULL | | BTREE | | |
| tracks | 1 | mv_clientid_customtrackid_deleted_lastmodified | 2 | custom_track_id | A | 336 | NULL | NULL | | BTREE | | |
| tracks | 1 | mv_clientid_customtrackid_deleted_lastmodified | 3 | deleted | A | 336 | NULL | NULL | | BTREE | | |
| tracks | 1 | mv_clientid_customtrackid_deleted_lastmodified | 4 | last_modified | A | 336 | NULL | NULL | | BTREE | | |
+--------+------------+------------------------------------------------+--------------+----------------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
And the EXPLAIN :
---+---------+-------------------------------------------------+-------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+--------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------+---------+-------------------------------------------------+-------+--------------------------+
| 1 | SIMPLE | t1 | range | client_id,custom_track_id,deleted,mv_clientid_deleted_featured_restrictedaccess,mv_clientid_customtrackid_deleted_lastmodified | mv_clientid_deleted_featured_restrictedaccess | 4 | NULL | 16018 | Using where; Using index |
| 1 | SIMPLE | t2 | ref | client_id,custom_track_id,deleted,mv_clientid_deleted_featured_restrictedaccess,mv_clientid_customtrackid_deleted_lastmodified | custom_track_id | 302 | synchtank_shared_application.t1.custom_track_id | 2 | Using where |
+----+-------------+-------+-------+--------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------+---------+-------------------------------------------------+-------+--------------------------+
So looking for ways to either optimise the query or the indexes. Im also curious why of the 2 composite indexes it could match on, it chooses the one without last_modified in it.
EXPLAIN for Stephan :
+----+-------------+-------+-------+-----------------------------------------------+-----------------------------------------------+---------+------+-------+---------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+-----------------------------------------------+-----------------------------------------------+---------+------+-------+---------------------------------------------+
| 1 | SIMPLE | t1 | range | client_id | client_id | 4 | NULL | 8111 | Using where |
| 1 | SIMPLE | t2 | range | mv_clientid_deleted_featured_restrictedaccess | mv_clientid_deleted_featured_restrictedaccess | 4 | NULL | 16018 | Using where; Using index; Using join buffer |
+----+-------------+-------+-------+-----------------------------------------------+-----------------------------------------------+---------+------+-------+---------------------------------------------+
You can try to force mysql to use the indexes:
SELECT
t1.id,
t1.custom_track_id,
t1.deleted,
t1.last_modified,
t1.client_id
FROM
tracks t1 FORCE INDEX(`client_id`)
INNER JOIN tracks t2 FORCE INDEX(`mv_clientid_deleted_featured_restrictedaccess`)
ON t1.custom_track_id = t2.custom_track_id
AND t1.last_modified > t2.last_modified
AND t2.deleted = 0
AND t2.client_id IN ( 103, 115, 116 )
WHERE
t1.client_id IN ( 103, 115, 116 )
AND t1.deleted !=0
Usually the reason why mysql don't uses the most efficient index is due to the false stats in cardinality that is why its a good practice to run OPTIMIZE TABLE on big tables with lots of columns and indexes .

Slow query with where like, order by and limit

I'm having an issue with the speed of a query I wan't to do.
The rooms table has 2685588 rows, and the room_active has less than 100, but it isn't slow with normal queries.
The query is:
SELECT rooms.*, room_active.active_users
FROM rooms LEFT JOIN room_active ON (room_active.roomid = rooms.id)
WHERE caption LIKE '%room%' ORDER BY room_active.active_users LIMIT 50
Runtime: 4.3s
It takes more than 5 seconds to execute when it should take less than one like does this one:
SELECT rooms.*, room_active.active_users
FROM rooms LEFT JOIN room_active ON (room_active.roomid = rooms.id)
WHERE caption LIKE '%room%' LIMIT 50
Runtime: 0.3s
The describe for the first query is this:
+----+-------------+-------------+--------+----------------+---------+---------+----------------+---------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------------+--------+----------------+---------+---------+----------------+---------+----------------------------------------------+
| 1 | SIMPLE | rooms | ALL | NULL | NULL | NULL | NULL | 2210576 | Using where; Using temporary; Using filesort |
| 1 | SIMPLE | room_active | eq_ref | PRIMARY,roomid | PRIMARY | 4 | xukys.rooms.id | 1 | NULL |
+----+-------------+-------------+--------+----------------+---------+---------+----------------+---------+----------------------------------------------+
Indexes:
+-------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| rooms | 0 | PRIMARY | 1 | id | A | 2210613 | NULL | NULL | | BTREE | | |
| rooms | 1 | owner | 1 | owner | A | 2210613 | NULL | NULL | | BTREE | | |
| rooms | 1 | roomtype | 1 | roomtype | A | 18 | NULL | NULL | | BTREE | | |
| rooms | 1 | caption | 1 | caption | A | 2210613 | NULL | NULL | | BTREE | | |
| rooms | 1 | category | 1 | category | A | 18 | NULL | NULL | | BTREE | | |
| rooms | 1 | users_now | 1 | users_now | A | 18 | NULL | NULL | | BTREE | | |
| rooms | 1 | tags | 1 | tags | A | 18 | NULL | NULL | | BTREE | | |
| rooms | 1 | users_max | 1 | users_max | A | 18 | NULL | NULL | | BTREE | | |
| rooms | 1 | description | 1 | description | A | 368435 | NULL | NULL | | BTREE | | |
| rooms | 1 | state | 1 | state | A | 18 | NULL | NULL | | BTREE | | |
| rooms | 1 | model_name | 1 | model_name | A | 18 | NULL | NULL | | BTREE | | |
| rooms | 1 | public_ccts | 1 | public_ccts | A | 18 | NULL | NULL | | BTREE | | |
| rooms | 1 | score | 1 | score | A | 18 | NULL | NULL | | BTREE | | |
| rooms | 1 | password | 1 | password | A | 18 | NULL | NULL | | BTREE | | |
| rooms | 1 | icon_bg | 1 | icon_bg | A | 954 | NULL | NULL | | BTREE | | |
| rooms | 1 | icon_fg | 1 | icon_fg | A | 18 | NULL | NULL | | BTREE | | |
| rooms | 1 | badge_id | 1 | badge_id | A | 804 | NULL | NULL | | BTREE | | |
+-------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
+-------------+------------+--------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-------------+------------+--------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| room_active | 0 | PRIMARY | 1 | roomid | A | 116 | NULL | NULL | | BTREE | | |
| room_active | 0 | roomid | 1 | roomid | A | 116 | NULL | NULL | | BTREE | | |
| room_active | 1 | active_users | 1 | active_users | A | 29 | NULL | NULL | | BTREE | | |
+-------------+------------+--------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
The ORDER BY is removing the advantage of LIMIT 50 because to be able to order by the 50 first elements he has to fetch all the records.
maybe if you tell us what exactly you want, we can help with the query. For example do you really need to do a LEFT JOIN and get all the rooms who don't have a room_active ?
not sure if it helps, but try this one
SELECT rooms.*, room_active.active_users
FROM room_active ra JOIN rooms r ON (room_active.roomid = rooms.id)
WHERE r.caption LIKE '%room%' ORDER BY ra.active_users LIMIT 50
It will ignore the rooms that don't have a room_active, reducing the fetched records
Follow following steps and check performance.
Create index on field 'caption' for its first 4 characters and on 'active_users'.
Instead of "rooms.*", select fields explicitly.
Use "caption LIKE 'room%'" instead of "caption LIKE '%room%'".
If this still slow, you should create a view for your joined query and then perform simple select statement on your view.
1 you might want a index perhaps on the order?
2 You don't need %room% do you? room% would be faster, your effectivly searching all files if you don't
3 don't left join if you can and only want active? instead inner join because it will require less joins (it doesnt join the ones that are null)
4 dont select .* unless needed?
5. Maybe you can use a differnt function such as Contains if you are using oracle
6. Put a index on caption its being mass searched thats probably the problem

MySQL explain - require index suggestions / feedback [closed]

Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
I have a database containing around 5million rows and am having issues with a query taking a long time (over a minute). I was hoping this information would be enough for someone to give me some suggestions, if I need to post more information just let me know.
Thank you in advance for any advice
EXPLAIN SELECT count( * ) AS count
FROM vtiger_time
INNER JOIN vtiger_crmentity ON vtiger_crmentity.crmid = vtiger_time.timeid
INNER JOIN vtiger_crmentityrel ON ( vtiger_crmentityrel.relcrmid = vtiger_crmentity.crmid
OR vtiger_crmentityrel.crmid = vtiger_crmentity.crmid )
LEFT JOIN vtiger_users ON vtiger_users.id = vtiger_crmentity.smownerid
LEFT JOIN vtiger_groups ON vtiger_groups.groupid = vtiger_crmentity.smownerid
WHERE vtiger_crmentity.deleted =0
AND (
vtiger_crmentityrel.crmid =211294
OR vtiger_crmentityrel.relcrmid =211294
)
+----+-------------+---------------------+-------------+-------------------------------------------------+----------------+---------+--------------------------------+-------+------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+---------------------+-------------+-------------------------------------------------+----------------+---------+--------------------------------+-------+------------------------------------------+
| 1 | SIMPLE | vtiger_crmentityrel | index_merge | crmid,relcrmid | crmid,relcrmid | 4,4 | NULL | 5881 | Using union(crmid,relcrmid); Using where |
| 1 | SIMPLE | vtiger_crmentity | ref | PRIMARY,deleted,deleted_2,crmentity_multi_index | deleted_2 | 4 | const | 84424 | Using where; Using index |
| 1 | SIMPLE | vtiger_users | eq_ref | PRIMARY | PRIMARY | 4 | crm.vtiger_crmentity.smownerid | 1 | Using index |
| 1 | SIMPLE | vtiger_groups | eq_ref | PRIMARY | PRIMARY | 4 | crm.vtiger_crmentity.smownerid | 1 | Using index |
| 1 | SIMPLE | vtiger_time | eq_ref | PRIMARY,timeid | PRIMARY | 4 | crm.vtiger_crmentity.crmid | 1 | Using index |
+----+-------------+---------------------+-------------+-------------------------------------------------+----------------+---------+--------------------------------+-------+------------------------------------------+
Additionally these are my currently set indexes
+------------------+------------+----------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+------------------+------------+----------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| vtiger_crmentity | 0 | PRIMARY | 1 | crmid | A | 755968 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 0 | crmid | 1 | crmid | A | 755968 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | crmentity_IDX0 | 1 | smcreatorid | A | 15 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | crmentity_IDX1 | 1 | smownerid | A | 15 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | crmentity_IDX2 | 1 | modifiedby | A | 15 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | deleted | 1 | deleted | A | 15 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | deleted | 2 | smownerid | A | 15 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | smownerid | 1 | smownerid | A | 199 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | smownerid | 2 | deleted | A | 199 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | deleted_2 | 1 | deleted | A | 15 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | deleted_2 | 2 | smownerid | A | 15 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | smownerid_2 | 1 | smownerid | A | 385 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | smownerid_2 | 2 | deleted | A | 758 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | crm_ownerid_del_setype_idx | 1 | smownerid | A | 15 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | crm_ownerid_del_setype_idx | 2 | deleted | A | 15 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | crm_ownerid_del_setype_idx | 3 | setype | A | 613 | NULL | NULL | YES | BTREE | |
| vtiger_crmentity | 1 | crmentity_multi_index | 1 | crmid | A | 755968 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | crmentity_multi_index | 2 | smownerid | A | 755968 | NULL | NULL | | BTREE | |
| vtiger_crmentity | 1 | crmentity_multi_index | 3 | deleted | A | 755968 | NULL | NULL | | BTREE | |
+------------------+------------+----------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
+---------------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+---------------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| vtiger_crmentityrel | 1 | crmid | 1 | crmid | A | 223960 | NULL | NULL | | BTREE | |
| vtiger_crmentityrel | 1 | relcrmid | 1 | relcrmid | A | 12442 | NULL | NULL | | BTREE | |
+---------------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
+--------------+------------+---------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+--------------+------------+---------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+
| vtiger_users | 0 | PRIMARY | 1 | id | A | 39 | NULL | NULL | | BTREE | |
| vtiger_users | 1 | idx_user_name | 1 | user_name | A | 39 | NULL | NULL | YES | BTREE | |
| vtiger_users | 1 | user_password | 1 | user_password | A | 39 | NULL | NULL | YES | BTREE | |
+--------------+------------+---------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+
+---------------+------------+---------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+---------------+------------+---------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| vtiger_groups | 0 | PRIMARY | 1 | groupid | A | 5 | NULL | NULL | | BTREE | |
| vtiger_groups | 1 | groupname | 1 | groupname | A | 5 | NULL | NULL | YES | BTREE | |
| vtiger_groups | 1 | idx_groups_123group | 1 | groupname | A | 5 | NULL | NULL | YES | BTREE | |
+---------------+------------+---------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
+----------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+----------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| vtiger_time | 0 | PRIMARY | 1 | timeid | A | 591772 | NULL | NULL | | BTREE | |
| vtiger_time | 0 | timeid | 1 | timeid | A | 591772 | NULL | NULL | | BTREE | |
| vtiger_time | 1 | relatedto | 1 | relatedto | A | 1405 | NULL | NULL | YES | BTREE | |
| vtiger_time | 1 | date_start | 1 | date_start | A | 7129 | NULL | NULL | YES | BTREE | |
| vtiger_time | 1 | relatedto_2 | 1 | relatedto | A | 3269 | NULL | NULL | YES | BTREE | |
+----------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
+------------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+------------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| vtiger_timecf | 0 | PRIMARY | 1 | timeid | A | 591324 | NULL | NULL | | BTREE | |
| vtiger_timecf | 0 | timeid | 1 | timeid | A | 591324 | NULL | NULL | | BTREE | |
| vtiger_timecf | 1 | timeid_2 | 1 | timeid | A | 591324 | NULL | NULL | | BTREE | |
+------------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
After adding the following index
ALTER TABLE vtiger_crmentity ADD INDEX TMP_deletion_smownerid_crmid (smownerid, deleted,crmid);
The new explain:
+----+-------------+---------------------+-------------+--------------------------------------------------------------------------+----------------+---------+--------------------------------+-------+------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+---------------------+-------------+--------------------------------------------------------------------------+----------------+---------+--------------------------------+-------+------------------------------------------+
| 1 | SIMPLE | vtiger_crmentityrel | index_merge | crmid,relcrmid | crmid,relcrmid | 4,4 | NULL | 5891 | Using union(crmid,relcrmid); Using where |
| 1 | SIMPLE | vtiger_crmentity | ref | PRIMARY,crmid,deleted,deleted_2,crmentity_multi_index,_deletion_crmid | deleted | 4 | const | 84424 | Using where; Using index |
| 1 | SIMPLE | vtiger_users | eq_ref | PRIMARY | PRIMARY | 4 | crm.vtiger_crmentity.smownerid | 1 | Using index |
| 1 | SIMPLE | vtiger_groups | eq_ref | PRIMARY | PRIMARY | 4 | crm.vtiger_crmentity.smownerid | 1 | Using index |
| 1 | SIMPLE | vtiger_time | eq_ref | PRIMARY,timeid | PRIMARY | 4 | crm.vtiger_crmentity.crmid | 1 | Using index |
+----+-------------+---------------------+-------------+--------------------------------------------------------------------------+----------------+---------+--------------------------------+-------+------------------------------------------+
Rewriting the query would help, but since you cannot do that, you may benefit from a composite index on vtiger_crmentity.crmid and vtiger_crmentity.deleted. But since it isn't using one of the the index you currently have on vtiger_crmentity.crmid, it might not use that new one. If it doesn't, try adding vtiger_crmentity.crmid to the vtiger_crmentity.deleted index. Since it is already using the index on vtiger_crmentity.deleted, this will make the index a covering index so the query will not need to read from the table.
As I mentioned in my comment, there are a lot of duplicate indexes. The won't affect the query for this question, but they will slow down inserts and increase table size. The following indexes are redundant:
vtiger_crmentity.crmid
vtiger_crmentity.smownerid
vtiger_crmentity.smownerid_2
vtiger_crmentity.deleted_2
vtiger_groups.idx_groups_123group
vtiger_time.timeid
vtiger_time.relatedto_2
vtiger_timecf.timeid
vtiger_timcfe.timeid_2
You also shouldn't need the index on the user password column since you shouldn't be looking for users by password.
Try adding this index:
ALTER TABLE vtiger_crmentityrel ADD INDEX ix_crmentityrel_crmid_relcrmid (crmid,relcrmid);
This should allow the first row in the explain to be satisfied completely by the two indexes on the crmentityrel table and should save you a few thousand lookups.
All other parts of the execution plan are using covering indexes so if this doesn't help, I'm not sure there is much more you can do without restructuring the data or the query.