As established in some other questions here, using "subquery" in MySQL delete causes it to be slower, while identical "select" query performs fast:
MariaDB [as_01_import]> explain select * from invoice_payment where invoice_id in (select id from dochead where system_id = 5786);
+------+-------------+-----------------+------+---------------------------------------+----------------------------+---------+-------------------------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+-----------------+------+---------------------------------------+----------------------------+---------+-------------------------+------+-------------+
| 1 | PRIMARY | dochead | ref | PRIMARY,dochead_system_id | dochead_system_id | 4 | const | 891 | Using index |
| 1 | PRIMARY | invoice_payment | ref | invoice_payment_invoice_fk,invoice_id | invoice_payment_invoice_fk | 4 | as_01_import.dochead.id | 1 | |
+------+-------------+-----------------+------+---------------------------------------+----------------------------+---------+-------------------------+------+-------------+
MariaDB [as_01_import]> explain delete from invoice_payment where invoice_id in (select id from dochead where system_id = 5786);
+------+--------------------+-----------------+-----------------+---------------------------+---------+---------+------+---------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+--------------------+-----------------+-----------------+---------------------------+---------+---------+------+---------+-------------+
| 1 | PRIMARY | invoice_payment | ALL | NULL | NULL | NULL | NULL | 1235451 | Using where |
| 2 | DEPENDENT SUBQUERY | dochead | unique_subquery | PRIMARY,dochead_system_id | PRIMARY | 4 | func | 1 | Using where |
+------+--------------------+-----------------+-----------------+---------------------------+---------+---------+------+---------+-------------+
2 rows in set (0.44 sec)
Knowing that JOIN can use indexes, I would like to ask experts:
What prevents MySQL / MariaDB from using indexes in DELETE with SUBQUERY? Is that an implementation problem or is there a conceptual problem? Are there any plans to solve this? Is the same problem affecting other SQL vendors?
Subqueries are derived tables and are not materialized. They are manifested in temporary tables.
As I wrote in this answer:
The document Derived Tables in MySQL 5.7 describes it well for
versions 5.6 and 5.7, where the latter will provide no penalty due to
the change in materialized derived table output being incorporated
into the outer query. In prior versions, substantial overhead was
endured with temporary tables with the derived.
Do not use IN ( SELECT ... ). Instead, use a multi-table DELETE as described here: http://dev.mysql.com/doc/refman/5.5/en/delete.html
Doing so will make use of indexes where applicable.
Related
I have a working, nice, indexed SQL query aggregating notes (sum of ints) for all my users and others stuffs. This is "query A".
I want to use this aggregated notes in others queries, say "query B".
If I create a View based on "query A", will the indexes of the original query will be used when needed if I join it in "query B" ?
Is that true for MySQL ? For others flavors of SQL ?
Thanks.
In MySQL, you cannot create an index on a view. MySQL uses indexes of the underlying tables when you query data against the views that use the merge algorithm. For the views that use the temptable algorithm, indexes are not utilized when you query data against the views.
https://www.percona.com/blog/2007/08/12/mysql-view-as-performance-troublemaker/
Here's a demo table. It has a userid attribute column and a note column.
mysql> create table t (id serial primary key, userid int not null, note int, key(userid,note));
If you do an aggregation to get the sum of note per userid, it does an index-scan on (userid, note).
mysql> explain select userid, sum(note) from t group by userid;
+----+-------------+-------+-------+---------------+--------+---------+------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+---------------+--------+---------+------+------+-------------+
| 1 | SIMPLE | t | index | userid | userid | 9 | NULL | 1 | Using index |
+----+-------------+-------+-------+---------------+--------+---------+------+------+-------------+
1 row in set (0.00 sec)
If we create a view for the same query, then we can see that querying the view uses the same index on the underlying table. Views in MySQL are pretty much like macros — they just query the underlying table.
mysql> create view v as select userid, sum(note) from t group by userid;
Query OK, 0 rows affected (0.03 sec)
mysql> explain select * from v;
+----+-------------+------------+-------+---------------+--------+---------+------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------+-------+---------------+--------+---------+------+------+-------------+
| 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 2 | NULL |
| 2 | DERIVED | t | index | userid | userid | 9 | NULL | 1 | Using index |
+----+-------------+------------+-------+---------------+--------+---------+------+------+-------------+
2 rows in set (0.00 sec)
So far so good.
Now let's create a table to join with the view, and join to it.
mysql> create table u (userid int primary key, name text);
Query OK, 0 rows affected (0.09 sec)
mysql> explain select * from v join u using (userid);
+----+-------------+------------+-------+---------------+-------------+---------+---------------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------+-------+---------------+-------------+---------+---------------+------+-------------+
| 1 | PRIMARY | u | ALL | PRIMARY | NULL | NULL | NULL | 1 | NULL |
| 1 | PRIMARY | <derived2> | ref | <auto_key0> | <auto_key0> | 4 | test.u.userid | 2 | NULL |
| 2 | DERIVED | t | index | userid | userid | 9 | NULL | 1 | Using index |
+----+-------------+------------+-------+---------------+-------------+---------+---------------+------+-------------+
3 rows in set (0.01 sec)
I tried to use hints like straight_join to force it to read v then join to u.
mysql> explain select * from v straight_join u on (v.userid=u.userid);
+----+-------------+------------+-------+---------------+--------+---------+------+------+----------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------+-------+---------------+--------+---------+------+------+----------------------------------------------------+
| 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 7 | NULL |
| 1 | PRIMARY | u | ALL | PRIMARY | NULL | NULL | NULL | 1 | Using where; Using join buffer (Block Nested Loop) |
| 2 | DERIVED | t | index | userid | userid | 9 | NULL | 7 | Using index |
+----+-------------+------------+-------+---------------+--------+---------+------+------+----------------------------------------------------+
"Using join buffer (Block Nested Loop)" is MySQL's terminology for "no index used for the join." It's just looping over the table the hard way -- by reading batches of rows from start to finish of the table.
I tried to use force index to tell MySQL that type=ALL is to be avoided.
mysql> explain select * from v straight_join u force index(PRIMARY) on (v.userid=u.userid);
+----+-------------+------------+--------+---------------+---------+---------+----------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------+--------+---------------+---------+---------+----------+------+-------------+
| 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 7 | NULL |
| 1 | PRIMARY | u | eq_ref | PRIMARY | PRIMARY | 4 | v.userid | 1 | NULL |
| 2 | DERIVED | t | index | userid | userid | 9 | NULL | 7 | Using index |
+----+-------------+------------+--------+---------------+---------+---------+----------+------+-------------+
Maybe this is using an index for the join? But it's weird that table u is before table t in the EXPLAIN. I'm frankly not sure how to understand what it's doing, given the order of rows in this EXPLAIN report. I would expect the joined table should come after the primary table of the query.
I only put a few rows of data into each table. One might get some different EXPLAIN results with a larger representative sample of test data. I'll leave that to you to try.
We have two tables - the first is relatively big (contact table) 250k rows and the second is small(user table, < 10 rows). On mysql 5.6 version I have next explain result:
EXPLAIN SELECT
o0_.id AS id_0,
o8_.first_name,
o8_.last_name
FROM
contact o0_
LEFT JOIN user o8_ ON o0_.user_owner_id = o8_.id
LIMIT
25 OFFSET 100
+----+-------------+-------+-------+---------------+----------------------+---------+------+--------+----------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+---------------+----------------------+---------+------+--------+----------------------------------------------------+
| 1 | SIMPLE | o0_ | index | NULL | IDX_403263ED9EB185F9 | 5 | NULL | 253030 | Using index |
| 1 | SIMPLE | o8_ | ALL | PRIMARY | NULL | NULL | NULL | 5 | Using where; Using join buffer (Block Nested Loop) |
+----+-------------+-------+-------+---------------+----------------------+---------+------+--------+----------------------------------------------------+
2 rows in set (0,00 sec)
When i use force index for join:
EXPLAIN SELECT
o0_.id AS id_0,
o8_.first_name,
o8_.last_name
FROM
contact o0_
LEFT JOIN user o8_ force index for join(`PRIMARY`) ON o0_.user_owner_id = o8_.id
LIMIT
25 OFFSET 100
or adding indexes on fields which appears in select clause (first_name, last_name) on user table:
alter table user add index(first_name, last_name);
Explain result changes to this:
+----+-------------+-------+--------+---------------+----------------------+---------+-------------------------+--------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+---------------+----------------------+---------+-------------------------+--------+-------------+
| 1 | SIMPLE | o0_ | index | NULL | IDX_403263ED9EB185F9 | 5 | NULL | 253030 | Using index |
| 1 | SIMPLE | o8_ | eq_ref | PRIMARY | PRIMARY | 4 | o0_.user_owner_id | 1 | NULL |
+----+-------------+-------+--------+---------------+----------------------+---------+-------------------------+--------+-------------+
2 rows in set (0,00 sec)
On mysql 5.5 version I have same explain result without additional indexes:
+----+-------------+-------+--------+---------------+----------------------+---------+-------------------------+--------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+---------------+----------------------+---------+-------------------------+--------+-------------+
| 1 | SIMPLE | o0_ | index | NULL | IDX_403263ED9EB185F9 | 5 | NULL | 255706 | Using index |
| 1 | SIMPLE | o8_ | eq_ref | PRIMARY | PRIMARY | 4 | o0_.user_owner_id | 1 | |
+----+-------------+-------+--------+---------------+----------------------+---------+-------------------------+--------+-------------+
2 rows in set (0.00 sec)
Why i need force use PRIMARY index or add extra indexes on mysql 5.6 version?
Same behavior occurs with other selects, when join small tables.
If you have a table with so few rows, it may actually be faster to do a full table scan, than going to an index, locate the records and then go back to the table. If you have other fields in the user table apart from the 3 in the query, then you may consider adding a covering index, but franly, I do not think that any of this would have significant affect on the speed of the query.
I have two tables, both with around 200,000 records. I want to join them on KEY which is a string.
Both tables have an index KEY, VALUE. When I run:
SELECT vpn, t1_sku, t2_sku
FROM first_inventory
LEFT JOIN second_inventory USING (vpn)
It is really fast. Here is the explain output:
----------------------------------------------------------------------------------------------------------------------------
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
|----|-------------|------------------|----------|-----------------|---------|-----------|---------|--------|--------------|
| 1 | SIMPLE | first_inventory | index | | t1_vpn | 158 | | 204437 | Using index |
| 1 | SIMPLE | second_inventory | ref | t2_vpn | t2_vpn | 387 | func | 1 | Using index |
----------------------------------------------------------------------------------------------------------------------------
However, when I do a RIGHT JOIN it takes forever and never finishes: "SELECT vpn, t1_sku, t2_sku FROM first_inventory RIGHT JOIN second_inventory USING (vpn)". Here is the explain output:
----------------------------------------------------------------------------------------------------------------------------
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
|----|-------------|------------------|----------|-----------------|---------|-----------|---------|--------|--------------|
| 1 | SIMPLE | second_inventory | index | | t1_vpn | 773 | | 220996 | Using index |
| 1 | SIMPLE | first_inventory | index | | t2_vpn | 158 | | 204437 | Using index |
----------------------------------------------------------------------------------------------------------------------------
I can see that the RIGHT JOIN query appears to be looking at 220996 x 204437 rows, while the LEFT JOIN only seems to be looking at 204437 rows, but I can't figure out why since I've made the same indexes on both tables.
At the end of both tables it says:
PRIMARY KEY ('id'),
KEY 't1_vpn' ('vpn','t1_sku') USING BTREE
and
PRIMARY KEY ('id'),
KEY 't2_vpn' ('vpn','t2_sku') USING BTREE
Any idea why the RIGHT JOIN query is taking so long?
The problem was that there was UTF-8 content in one of the tables. I changed everything to Latin1 and it runs fast both ways now.
See: this discussion
In the web page that I'm working on I need to show some statistics based on a different user details which are in three tables. So I have the following query that I join to more different tables:
SELECT *
FROM `user` `u`
LEFT JOIN `subscriptions` `s` ON `u`.`user_id` = `s`.`user_id`
LEFT JOIN `devices` `ud` ON `u`.`user_id` = `ud`.`user_id`
GROUP BY `u`.`user_id`
When I execute the query with LIMIT 1000 it takes about 0.05 seconds and since I'm using the data from all the three tables in a lot of queries I've decided to put it inside a VIEW:
CREATE VIEW `user_details` AS ( the same query from above )
And now when I run:
SELECT * FROM user_details LIMIT 1000
it takes about 7-10 seconds.
So my question is can I do something to optimize the view because the query seems to be pretty quick or I should the whole query instead of the view ?
Edit: this is what EXPLAIN SELECT * FROM user_details returns
+----+-------------+------------+--------+----------------+----------------+---------+------------------------+--------+-------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------+--------+----------------+----------------+---------+------------------------+--------+-------+
| 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 322666 | |
| 2 | DERIVED | u | index | NULL | PRIMARY | 4 | NULL | 372587 | |
| 2 | DERIVED | s | eq_ref | PRIMARY | PRIMARY | 4 | db_users.u.user_id | 1 | |
| 2 | DERIVED | ud | ref | device_id_name | device_id_name | 4 | db_users.u.user_id | 1 | |
+----+-------------+------------+--------+----------------+----------------+---------+------------------------+--------+-------+
4 rows in set (8.67 sec)
this is what explain retuns for the query:
+----+-------------+-------+--------+----------------+----------------+---------+------------------------+--------+-------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+----------------+----------------+---------+------------------------+--------+-------+
| 1 | SIMPLE | u | index | NULL | PRIMARY | 4 | NULL | 372587 | |
| 1 | SIMPLE | s | eq_ref | PRIMARY | PRIMARY | 4 | db_users.u.user_id | 1 | |
| 1 | SIMPLE | ud | ref | device_id_name | device_id_name | 4 | db_users.u.user_id | 1 | |
+----+-------------+-------+--------+----------------+----------------+---------+------------------------+--------+-------+
3 rows in set (0.00 sec)
Views and joins are extremely bad if it comes to performance. This is more or less true for all relational database management systems. Sounds strange, since that is what those systems are designed for, but it is true nevertheless.
Try to avoid the joins if this is a query in heavy usage on your page: instead create a real table (not a view) that is filled from the three tables. you can automate that process using triggers. So each time an entry is inserted into one of the original tables the triggers takes care that the data is propagated to the physical user_details table.
This strategy certainly means a one time investment for the setup, but you definitely will get a much better performance.
Below is the EXPLAIN for the query in question..
mysql> EXPLAIN SELECT orders_0.id, orders_0.created_at, orders_0.total_amount, orders_0.delivery_date, orders_0.customer_id, orders_0.items_summary, orders_0.site_id, orders_0.depot_id, orders_0.region_id, addresses_0.country, orders_0.payment_status, orders_0.created_by, orders_0.status, orders_0.delivered_by
-> FROM production.addresses addresses_0, production.orders orders_0
-> WHERE orders_0.delivery_address_id = addresses_0.id AND ((orders_0.status Not In ('cancelled','checkout','expired')) AND (orders_0.delivery_date>{d '2011-10-31'}));
+----+-------------+-------------+--------+------------------------------------------+---------+---------+-----------------------------------------------+--------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------------+--------+------------------------------------------+---------+---------+-----------------------------------------------+--------+-------------+
| 1 | SIMPLE | orders_0 | ALL | status,delivery_date,delivery_address_id | NULL | NULL | NULL | 929330 | Using where |
| 1 | SIMPLE | addresses_0 | eq_ref | PRIMARY | PRIMARY | 4 | production.orders_0.delivery_address_id | 1 | |
+----+-------------+-------------+--------+------------------------------------------+---------+---------+-----------------------------------------------+--------+-------------+
2 rows in set (0.00 sec)
mysql>
I have indexes on the three columns it's trying to do where conditions on but sadly it's not trying to use an index merge. Is the only option to create a multi-column index or is there something that's putting MYSQL off using those indexes?
You have to create a multi-columns index.
For example, to do that in Phpmyadmin, you have to check this different fields of your table and click the 'index' button below the table structure.