Mysql. JOIN to small table blocks using indexes on the main one - mysql

MySQL version 5.7.17
When I use JOIN for small tables, MySQL don't allow me to use index on the main table.
Compare:
EXPLAIN SELECT `ko_base_client_orders`.*
FROM `ko_base_client_orders`
LEFT JOIN `ko_base_new_perspe` AS `ko_of`
ON (`ko_of`.`id` = `ko_base_client_orders`.`office`)
ORDER BY `order_time` DESC
LIMIT 10 OFFSET 0;
+----+-------------+-----------------------+------------+-------+---------------+---------+---------+------+-------+----------+-----------------------------------------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-----------------------+------------+-------+---------------+---------+---------+------+-------+----------+-----------------------------------------------------------------+
| 1 | SIMPLE | ko_base_client_orders | NULL | ALL | NULL | NULL | NULL | NULL | 60737 | 100.00 | Using temporary; Using filesort |
| 1 | SIMPLE | ko_of | NULL | index | PRIMARY | PRIMARY | 4 | NULL | 2 | 100.00 | Using where; Using index; Using join buffer (Block Nested Loop) |
+----+-------------+-----------------------+------------+-------+---------------+---------+---------+------+-------+----------+-----------------------------------------------------------------+
2 rows in set, 1 warning (0.00 sec)
and the same query with index hint:
EXPLAIN SELECT `ko_base_client_orders`.*
FROM `ko_base_client_orders`
LEFT JOIN `ko_base_new_perspe` AS `ko_of`
FORCE INDEX FOR JOIN (`PRIMARY`)
ON (`ko_of`.`id` = `ko_base_client_orders`.`office`)
ORDER BY `order_time` DESC
LIMIT 10 OFFSET 0;
+----+-------------+-----------------------+------------+--------+---------------+------------+---------+---------------------------------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-----------------------+------------+--------+---------------+------------+---------+---------------------------------+------+----------+-------------+
| 1 | SIMPLE | ko_base_client_orders | NULL | index | NULL | order_time | 6 | NULL | 10 | 100.00 | NULL |
| 1 | SIMPLE | ko_of | NULL | eq_ref | PRIMARY | PRIMARY | 4 | e3.ko_base_client_orders.office | 1 | 100.00 | Using index |
+----+-------------+-----------------------+------------+--------+---------------+------------+---------+---------------------------------+------+----------+-------------+
In second case MySQL uses key for order_time, but in first example it ignores index and scans the entire table.
I can not predict which table would be small, and which would not (it differs from project to project), so I don't want to use index hints all the time. Is there another solution for this problem?

Related

MySQL: Out of sort memory, consider increasing server sort buffer size

I cant find explanation of MySQL behavior. Examples are below.
Fields name and name_full are text type and field price_steps is json type. None of them has index.
SELECT
name,
name_full,
price_steps
FROM
`lots`
WHERE
EXISTS (
SELECT
*
FROM
`categories`
INNER JOIN `category_lot` ON `categories`.`id` = `category_lot`.`category_id`
WHERE
`lots`.`id` = `category_lot`.`lot_id`
AND `category_id` IN (25)
)
ORDER BY
`created_at` DESC
LIMIT 31 OFFSET 0
MySQL throws error: [Err] 1038 - Out of sort memory, consider increasing server sort buffer size.
Ok, let it be.
Then I added extra field into select part.
SUBSTRING(`name_full`, 1, 200000000000) as name_full2
and query runs successfully (Why? Extra field should lead to extra memory allocation, isnt it?).
Then I decided to make query heavier and replace string
AND `category_id` IN (25)
with
AND `category_id` IN (1,2,3,4,5,6,7,8,9,10, 25)
and the query also finishes successfully.
Count of rows with category = 25 is only about 250, but with categories in (1,2,3,4,5,6,7,8,9,10, 25) is about 40000 rows. This must lead to extra memory demands, but mysql doesnt throw error. Why?
Any explanation to this paradox? Thanks in advance!
UPDATE1
EXPLAIN with failing query
mysql> EXPLAIN SELECT name, name_full, price_steps FROM `lots` WHERE EXISTS ( SELECT * FROM `categories` INNER JOIN `category_lot` ON `categories`.`id` = `category_lot`.`category_id` WHERE `lots`.`id` = `category_lot`.`lot_id` AND `category_id` IN(25) ) ORDER BY `created_at` DESC;
+----+-------------+--------------+------------+--------+--------------------------------------------------------------+----------------------------------+---------+---------------------------+------+----------+----------------------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+--------------+------------+--------+--------------------------------------------------------------+----------------------------------+---------+---------------------------+------+----------+----------------------------------------------+
| 1 | SIMPLE | categories | NULL | const | PRIMARY | PRIMARY | 8 | const | 1 | 100.00 | Using index; Using temporary; Using filesort |
| 1 | SIMPLE | category_lot | NULL | ref | category_lot_lot_id_foreign,category_lot_category_id_foreign | category_lot_category_id_foreign | 8 | const | 1099 | 100.00 | Start temporary |
| 1 | SIMPLE | lots | NULL | eq_ref | PRIMARY | PRIMARY | 8 | torgs.category_lot.lot_id | 1 | 100.00 | End temporary |
+----+-------------+--------------+------------+--------+--------------------------------------------------------------+----------------------------------+---------+---------------------------+------+----------+----------------------------------------------+
3 rows in set, 2 warnings (0.00 sec)
EXPLAIN with success query (added 4th field)
mysql> EXPLAIN SELECT name, name_full, price_steps,SUBSTRING(`name_full`, 1, 200000000000) as name_full2 FROM `lots` WHERE EXISTS ( SELECT * FROM `categories` INNER JOIN `category_lot` ON `categories`.`id` = `category_lot`.`category_id` WHERE `lots`.`id` = `category_lot`.`lot_id` AND `category_id` IN(25) ) ORDER BY `created_at` DESC;
+----+-------------+--------------+------------+--------+--------------------------------------------------------------+----------------------------------+---------+---------------------------+------+----------+----------------------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+--------------+------------+--------+--------------------------------------------------------------+----------------------------------+---------+---------------------------+------+----------+----------------------------------------------+
| 1 | SIMPLE | categories | NULL | const | PRIMARY | PRIMARY | 8 | const | 1 | 100.00 | Using index; Using temporary; Using filesort |
| 1 | SIMPLE | category_lot | NULL | ref | category_lot_lot_id_foreign,category_lot_category_id_foreign | category_lot_category_id_foreign | 8 | const | 1099 | 100.00 | Start temporary |
| 1 | SIMPLE | lots | NULL | eq_ref | PRIMARY | PRIMARY | 8 | torgs.category_lot.lot_id | 1 | 100.00 | End temporary |
+----+-------------+--------------+------------+--------+--------------------------------------------------------------+----------------------------------+---------+---------------------------+------+----------+----------------------------------------------+
3 rows in set, 2 warnings (0.00 sec)
EXPLAIN with success query (added 4th field and category_id in (1,2,3,4,5,6,7,8,9,10,25))
mysql> EXPLAIN SELECT name, name_full, price_steps, SUBSTRING(`name_full`, 1, 200000000000) as name_full2 FROM `lots` WHERE EXISTS ( SELECT * FROM `categories` INNER JOIN `category_lot` ON `categories`.`id` = `category_lot`.`category_id` WHERE `lots`.`id` = `category_lot`.`lot_id` AND `category_id` IN(1,2,3,4,5,6,7,8,9,10,25) ) ORDER BY `created_at` DESC;
+----+--------------+--------------+------------+--------+--------------------------------------------------------------+----------------------------------+---------+---------------------+------+----------+---------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+--------------+--------------+------------+--------+--------------------------------------------------------------+----------------------------------+---------+---------------------+------+----------+---------------------------------+
| 1 | SIMPLE | <subquery2> | NULL | ALL | NULL | NULL | NULL | NULL | NULL | 100.00 | Using temporary; Using filesort |
| 1 | SIMPLE | lots | NULL | eq_ref | PRIMARY | PRIMARY | 8 | <subquery2>.lot_id | 1 | 100.00 | NULL |
| 2 | MATERIALIZED | categories | NULL | range | PRIMARY | PRIMARY | 8 | NULL | 11 | 100.00 | Using where; Using index |
| 2 | MATERIALIZED | category_lot | NULL | ref | category_lot_lot_id_foreign,category_lot_category_id_foreign | category_lot_category_id_foreign | 8 | torgs.categories.id | 1883 | 100.00 | NULL |
+----+--------------+--------------+------------+--------+--------------------------------------------------------------+----------------------------------+---------+---------------------+------+----------+---------------------------------+
4 rows in set, 2 warnings (0.00 sec)
UPDATE2
+-------+------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Level | Code | Message |
+-------+------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Note | 1276 | Field or reference 'torgs.lots.id' of SELECT #2 was resolved in SELECT #1 |
| Note | 1003 | /* select#1 */ select `torgs`.`lots`.`name` AS `name`,`torgs`.`lots`.`name_full` AS `name_full`,`torgs`.`lots`.`price_steps` AS `price_steps`,substr(`torgs`.`lots`.`name_full`,1,200000000000) AS `name_full2` from `torgs`.`lots` semi join (`torgs`.`categories` join `torgs`.`category_lot`) where ((`torgs`.`category_lot`.`category_id` = `torgs`.`categories`.`id`) and (`torgs`.`lots`.`id` = `<subquery2>`.`lot_id`) and (`torgs`.`categories`.`id` in (1,2,3,4,5,6,7,8,9,10,25))) order by `torgs`.`lots`.`created_at` desc |
+-------+------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
2 rows in set (0.00 sec)

Using ORDER BY ... LIMIT 20; makes query extremely slow

I'm looking into an issue with slow queries, and I can't figure out why using LIMIT 20 on this query makes it a factor 100 slower.
The query is as follows:
SELECT DISTINCT e.eventid,e.clock,e.ns,e.objectid,e.acknowledged,er1.r_eventid
FROM events e JOIN event_recovery er1 ON er1.eventid=e.eventid
WHERE e.source='0' AND e.object='0' AND e.objectid='115779'
AND e.eventid<='65535859' AND e.value='1'
ORDER BY e.eventid
LIMIT 20;
EXPLAIN:
Without LIMIT 20:
EXPLAIN SELECT DISTINCT e.eventid... ORDER BY e.eventid;
+------+-------------+-------+--------+---------------------------+----------+---------+-------------------------+-------+---------------------------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+-------+--------+---------------------------+----------+---------+-------------------------+-------+---------------------------------------------------------------------+
| 1 | SIMPLE | e | ref | PRIMARY,events_1,events_2 | events_1 | 16 | const,const,const | 15792 | Using index condition; Using where; Using temporary; Using filesort |
| 1 | SIMPLE | er1 | eq_ref | PRIMARY | PRIMARY | 8 | zabbix_server.e.eventid | 1 | |
+------+-------------+-------+--------+---------------------------+----------+---------+-------------------------+-------+---------------------------------------------------------------------+
(4256 rows in set (0.050 sec) when executed without EXPLAIN)
With LIMIT 20:
EXPLAIN SELECT DISTINCT e.eventid... ORDER BY e.eventid LIMIT 20;
+------+-------------+-------+--------+---------------------------+---------+---------+-------------------------+---------+------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+-------+--------+---------------------------+---------+---------+-------------------------+---------+------------------------------+
| 1 | SIMPLE | e | range | PRIMARY,events_1,events_2 | PRIMARY | 8 | NULL | 2969589 | Using where; Using temporary |
| 1 | SIMPLE | er1 | eq_ref | PRIMARY | PRIMARY | 8 | zabbix_server.e.eventid | 1 | |
+------+-------------+-------+--------+---------------------------+---------+---------+-------------------------+---------+------------------------------+
(20 rows in set (9.269 sec) when executed without EXPLAIN)
Why is it not using the keys for the LIMIT 20?
I'm using MariaDB 10.3.11 on Debian 9
EDIT:
Forcing it to use the INDEX makes it fast, but why doesn't the MySQL optimizer realise this?
EXPLAIN SELECT DISTINCT e.eventid,e.clock,e.ns,e.objectid,e.acknowledged,er1.r_eventid FROM events e USE INDEX (events_1) JOIN event_recovery er1 ON er1.eventid=e.eventid WHERE e.source='0' AND e.object='0' AND e.objectid='115779' AND e.eventid<='65535859' AND e.value='1' ORDER BY e.eventid LIMIT 20;
+------+-------------+-------+--------+---------------+----------+---------+-------------------------+-------+---------------------------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+-------+--------+---------------+----------+---------+-------------------------+-------+---------------------------------------------------------------------+
| 1 | SIMPLE | e | ref | events_1 | events_1 | 16 | const,const,const | 15820 | Using index condition; Using where; Using temporary; Using filesort |
| 1 | SIMPLE | er1 | eq_ref | PRIMARY | PRIMARY | 8 | zabbix_server.e.eventid | 1 | |
+------+-------------+-------+--------+---------------+----------+---------+-------------------------+-------+---------------------------------------------------------------------+

Query uses temporary table without force index

Query
SELECT SQL_NO_CACHE contacts.id,
contacts.date_modified contacts__date_modified
FROM contacts
INNER JOIN
(SELECT tst.team_set_id
FROM team_sets_teams tst
INNER JOIN team_memberships team_membershipscontacts ON (team_membershipscontacts.team_id = tst.team_id)
AND (team_membershipscontacts.user_id = '5daa2e92-c347-11e9-afc5-525400a80916')
AND (team_membershipscontacts.deleted = 0)
GROUP BY tst.team_set_id) contacts_tf ON contacts_tf.team_set_id = contacts.team_set_id
LEFT JOIN contacts_cstm contacts_cstm ON contacts_cstm.id_c = contacts.id
WHERE contacts.deleted = 0
ORDER BY contacts.date_modified DESC,
contacts.id DESC
LIMIT 21;
Takes extremely long (2 minutes on 2M records). I cant change this query, since it is system generated
This is it's explain:
+----+-------------+--------------------------+------------+--------+-------------------------------------------------------------------------------------------------------+----------------------------+---------+-------------------------------------------+---------+----------+---------------------------------------------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+--------------------------+------------+--------+-------------------------------------------------------------------------------------------------------+----------------------------+---------+-------------------------------------------+---------+----------+---------------------------------------------------------------------+
| 1 | PRIMARY | contacts | NULL | ref | idx_contacts_tmst_id,idx_del_date_modified,idx_contacts_del_last,idx_cont_del_reports,idx_del_id_user | idx_del_date_modified | 2 | const | 1113718 | 100.00 | Using temporary; Using filesort |
| 1 | PRIMARY | <derived3> | NULL | ALL | NULL | NULL | NULL | NULL | 2 | 50.00 | Using where; Using join buffer (Block Nested Loop) |
| 1 | PRIMARY | contacts_cstm | NULL | eq_ref | PRIMARY | PRIMARY | 144 | sugarcrm.contacts.id | 1 | 100.00 | Using index |
| 3 | DERIVED | team_membershipscontacts | NULL | ref | idx_team_membership,idx_teammemb_team_user,idx_del_team_user | idx_team_membership | 145 | const | 2 | 99.36 | Using index condition; Using where; Using temporary; Using filesort |
| 3 | DERIVED | tst | NULL | ref | idx_ud_set_id,idx_ud_team_id,idx_ud_team_set_id,idx_ud_team_id_team_set_id | idx_ud_team_id_team_set_id | 144 | sugarcrm.team_membershipscontacts.team_id | 1 | 100.00 | Using index |
+----+-------------+--------------------------+------------+--------+-------------------------------------------------------------------------------------------------------+----------------------------+---------+-------------------------------------------+---------+----------+---------------------------------------------------------------------+
But when I use force index(idx_del_date_modified) (which is the same index used in explain), the query takes just 0.01s and I get slightly different explain.
+----+-------------+--------------------------+------------+--------+----------------------------------------------------------------------------+----------------------------+---------+-------------------------------------------+---------+----------+---------------------------------------------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+--------------------------+------------+--------+----------------------------------------------------------------------------+----------------------------+---------+-------------------------------------------+---------+----------+---------------------------------------------------------------------+
| 1 | PRIMARY | contacts | NULL | ref | idx_del_date_modified | idx_del_date_modified | 2 | const | 1113718 | 100.00 | Using where |
| 1 | PRIMARY | <derived2> | NULL | ALL | NULL | NULL | NULL | NULL | 2 | 50.00 | Using where |
| 1 | PRIMARY | contacts_cstm | NULL | eq_ref | PRIMARY | PRIMARY | 144 | sugarcrm.contacts.id | 1 | 100.00 | Using index |
| 2 | DERIVED | team_membershipscontacts | NULL | ref | idx_team_membership,idx_teammemb_team_user,idx_del_team_user | idx_team_membership | 145 | const | 2 | 99.36 | Using index condition; Using where; Using temporary; Using filesort |
| 2 | DERIVED | tst | NULL | ref | idx_ud_set_id,idx_ud_team_id,idx_ud_team_set_id,idx_ud_team_id_team_set_id | idx_ud_team_id_team_set_id | 144 | sugarcrm.team_membershipscontacts.team_id | 1 | 100.00 | Using index |
+----+-------------+--------------------------+------------+--------+----------------------------------------------------------------------------+----------------------------+---------+-------------------------------------------+---------+----------+---------------------------------------------------------------------+
The first query uses temporary table and file sort, but the query with force index uses just where. Shouldn't the queries be the same? Why is the query with force index so much faster - used index is still the same.
According to MySQL manual:
Temporary tables can be created under conditions such as these:
If there is an ORDER BY clause and a different GROUP BY clause, or if
the ORDER BY or GROUP BY contains columns from tables other than the
first table in the join queue, a temporary table is created.
DISTINCT combined with ORDER BY may require a temporary table.
If you use the SQL_SMALL_RESULT option, MySQL uses an in-memory
temporary table, unless the query also contains elements (described
later) that require on-disk storage.
Likely, you have better performance because in MySQL there is the query optimizer component.
If you create index the query optimizer could not use the index column even though the index exists.
Using force index(..) you are forcing MySql to use index, instead.
Please consider a detailed example here.

How to optimise a complicated mysql query?

When I EXPLIAN this:
EXPLIAN SELECT m.*,m.id AS mid FROM movie_category mc
LEFT JOIN movie m ON m.id=mc.movie_id
RIGHT JOIN movie_area ma ON ma.movie_id=mc.movie_id
LEFT JOIN area a ON a.id=ma.area_id
LEFT JOIN category c ON c.id=mc.category_id
WHERE 1 and ma.area_id>0
GROUP BY mid
ORDER BY m.read_count desc LIMIT 0,36;
I got this result:
+----+-------------+-------+------------+--------+-----------------+----------+---------+----------------------+-------+----------+---------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+--------+-----------------+----------+---------+----------------------+-------+----------+---------------------------------+
| 1 | SIMPLE | ma | NULL | ALL | NULL | NULL | NULL | NULL | 15545 | 100.00 | Using temporary; Using filesort |
| 1 | SIMPLE | mc | NULL | ref | movie_id | movie_id | 5 | flask.ma.movie_id | 2 | 100.00 | NULL |
| 1 | SIMPLE | m | NULL | eq_ref | PRIMARY,year_id | PRIMARY | 4 | flask.ma.movie_id | 1 | 100.00 | NULL |
| 1 | SIMPLE | a | NULL | eq_ref | PRIMARY | PRIMARY | 4 | flask.ma.area_id | 1 | 100.00 | Using index |
| 1 | SIMPLE | c | NULL | eq_ref | PRIMARY | PRIMARY | 4 | flask.mc.category_id | 1 | 100.00 | Using index |
+----+-------------+-------+------------+--------+-----------------+----------+---------+----------------------+-------+----------+---------------------------------+
5 rows in set, 1 warning (0.00 sec)
How to optimise this query? I really down't know what to do, help me out.
edit:
From the explain result,first line's Extra is "Using temporary; Using filesort",is not good.And second line's Extra and third line's Extra are all NULL, are also not good.
ps:
The query span 0.91 seconds,is very bad.How to add index to prove the query speed.
Improve schema
Without seeing the schemas involved, I will assume you have two many-to-many mapping table (mc, ma) that are inefficiently defined.
Follow the rules here ; you will gain some efficiency.
Get rid of unnecessary tables
category and movie_category are not really used in the query, so remove them from the query.
I assume this is a "generated" query? Then make the generation a little more sophisticated!

Adding LIMIT to query increases query time by over 1000%

I am running the following query:
SELECT
MyField,
COUNT(*) AS MyCount
FROM
MyTable
NATURAL JOIN
AnotherTable
WHERE
Timestamp >= 1000 AND Timestamp <= 10000
GROUP BY
MyField
ORDER BY
MyCount DESC;
This runs fine and takes about 6 seconds to complete. If I want to limit the result to show only the 20 highest MyCounts, I add LIMIT 20 on to the end of the query. Suddenly it takes 6 minutes to complete!
The EXPLAIN output for the original query:
+----+-------------+-------------+--------+---------------------------+---------+---------+---------------------------+---------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------------+--------+---------------------------+---------+---------+---------------------------+---------+----------------------------------------------+
| 1 | SIMPLE | MyTable | ALL | mytable_fkey | NULL | NULL | NULL | 6858209 | Using where; Using temporary; Using filesort |
| 1 | SIMPLE | AnotherTable| eq_ref | PRIMARY | PRIMARY | 4 | test.MyTable.FKeyID | 1 | Using index |
+----+-------------+-------------+--------+---------------------------+---------+---------+---------------------------+---------+----------------------------------------------+
The EXPLAIN output for the query with LIMIT 20:
+----+-------------+-------------+--------+---------------------------+-------------------------+---------+---------------------------+---------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------------+--------+---------------------------+-------------------------+---------+---------------------------+---------+----------------------------------------------+
| 1 | SIMPLE | MyTable | index | mytable_fkey | myfield_timestamp_index | 771 | NULL | 6858209 | Using where; Using temporary; Using filesort |
| 1 | SIMPLE | AnotherTable| eq_ref | PRIMARY | PRIMARY | 4 | test.MyTable.FKeyID | 1 | Using index |
+----+-------------+-------------+--------+---------------------------+-------------------------+---------+---------------------------+---------+----------------------------------------------+
What is the explanation for this? Is there a better way I can limit the number of rows?
If you see Using temporary; Using filesort in your EXPLAIN output, you are probably missing a suitable index and you're getting killed because of it.
Make sure your JOIN and GROUP BY fields are both available in the same index.