MySQL optimization. Why option2 so faster than option1 - mysql

Question

I wrote a query thas was taking way too much time (32 minutes) so I tried other methods to find one faster.
I finally wrote another one taking under 5 seconds
The problem is that I don't understand my optimization.
Can someone explain how it happen to be that much faster.
hugeTable has 494 500 rows
smallTable1 has 983 rows
smallTable2 has 983 rows
cursor.execute('UPDATE hugeTable dst,
(
SELECT smallTable1.hugeTableId, smallTable2.valueForHugeTable
FROM smallTable2
INNER JOIN smallTable1 ON smallTable1.id = smallTable2.id
-- This select represent 983 rows
)src
SET dst.columnToUpdate = src.valueForHugeTable
WHERE dst.id2 = %s AND dst.id = src.hugeTableId;', inputId2)
-- Condition : dst.id2 = %s alone target 983 rows.
-- Combinasion of : dst.id2 = %s AND dst.id = src.hugeTableId target a single unique row.
-- This query takes 32 minutes
And here is a way to do the exact same request with more steps, but way faster:
-- First create a temporary table to hold (983) rows from hugeTable that has to be updated
cursor.execute('CREATE TEMPORARY TABLE tmpTable AS
SELECT * from hugeTable
WHERE id2 = %s;', inputid)
-- Update the rows into tmpTable instead of into hugeTable
cursor.execute('UPDATE tmpTable dst,
(
SELECT smallTable1.hugeTableId, smallTable2.valueForHugeTable
FROM smallTable2
INNER JOIN smallTable1 ON smallTable1.id = smallTable2.id
-- This select represent 983 rows
)src
SET dst.columnToUpdate = src.valueForHugeTable
WHERE dst.id = src.hugeTableId;')
-- Then delete the (983) rows we want to update
cursor.execute('DELETE FROM hugeTable WHERE id2 = %s;', inputId2)
-- And create new rows replacing the above deleled ones with rows from tmpTable
cursor.execute('INSERT INTO hugeTable SELECT * FROM tmpTable;')
-- This takes litle under 5 seconds.
I would like to know why the first method takes so much time.
Understanding this will help me getting a new MySql level up.

Answer 1

Add a composite index to dst: INDEX(id2, id) (in either order).
More
Case 1:
UPDATE hugeTable dst,
( SELECT smallTable1.hugeTableId, smallTable2.valueForHugeTable
FROM smallTable2
INNER JOIN smallTable1 ON smallTable1.id = smallTable2.id
)src SET dst.columnToUpdate = src.valueForHugeTable
WHERE dst.id2 = 1234
AND dst.id = src.hugeTableId;
Case 2:
CREATE TEMPORARY TABLE tmpTable AS
SELECT *
from hugeTable
WHERE id2 = 1234;
UPDATE tmpTable dst,
( SELECT smallTable1.hugeTableId, smallTable2.valueForHugeTable
FROM smallTable2
INNER JOIN smallTable1 ON smallTable1.id = smallTable2.id
)src SET dst.columnToUpdate = src.valueForHugeTable
WHERE dst.id = src.hugeTableId;
Without knowing the MySQL version and seeing the EXPLAINs, I can only guess at why they are so different...
The subquery ( SELECT ... JOIN ... ) may or may not be 'materialized' into an implicit temp table. (Newer versions are better at doing such.)
Such a materialized subquery may or may not have an index created for it. (Again, new versions are better.)
If there are no adequate indexes on either dst or src, then the amount of 'effort' is the product of the sizes of the two tables. Note that in Case 2, dst is much smaller. (This may be the answer you are looking for.)
If the tables are not fully cached in RAM, one could artificially involve more I/O than the other. An I/O-bound query is often 10 times as slow as a the same query when it is fully cached in RAM. (This is less likely to be the answer, but may be part of the answer.)
Having a 3-table UPDATE would probably eliminate some of the issues above. And it may (or may not) eliminate the timing difference.
For further discussion, please provide
MySQL version
SHOW CREATE TABLE -- for each table
How big is innodb_buffer_pool_size
SHOW TABLE STATUS -- for each table
EXPLAIN UPDATE ... -- for each UPDATE -- requires at least 5.6
What percentage of the table has ( id2 = inputId2 )?