Outside query of subqueries is extremely slow (Mysql) - mysql

Question

I have a aggregate query with two levels deep subqueries. What is strange is that the two subqueries run acceptably fast but the outside query unacceptably slow.
The basic idea behind the query is to use a table to find all elements linked to a key, selected by one of the elements queries. This resultant set should then be provided to the outside query that will match it according to its own keys/indexes.
Here with all outputs and statements:
We start with the two table definitions
CREATE TABLE `table1` (
`id1` int(11) NOT NULL DEFAULT '0',
`id2` int(11) NOT NULL,
`value` int(11) DEFAULT '0',
PRIMARY KEY (`id1`,`id2`),
KEY `k_id1` (`id1`),
KEY `k_id2` (`id2`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
CREATE TABLE `lookuptable1` (
`id3` int(11) NOT NULL,
`id4` int(11) NOT NULL,
PRIMARY KEY (`id3`,`id4`),
UNIQUE KEY `id4_idx` (`id4`),
KEY `id3_idx` (`id3`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
The inside subquery with it's own subquery
SELECT lt1.id4
FROM lookuptable1 lt1
WHERE lt1.id3 = (SELECT pt1.id3
FROM lookuptable1 pt1
WHERE pt1.id4 = 5960)
+-----------+
| id4 |
+-----------+
| 5960 |
| 17215 |
| 3625734 |
| 9312798 |
+-----------+
4 rows in set (0.00 sec)
As you can see: Fast enough.
But the outside query is where the bad bottleneck lies.
Complete query
SELECT
t1.id1,
sum(t1.value)
FROM table1 t1
WHERE t1.id2 = 3 AND t1.id1 IN
(
SELECT lt1.id4
FROM lookuptable1 lt1
WHERE lt1.id3 = (SELECT pt1.id3
FROM lookuptable1 pt1
WHERE pt1.id4 = 5960)
);
+-----------+-----------------------+
| id 1. | sum(t1.value) |
+-----------+-----------------------+
| 9312798 | 0 |
+-----------+-----------------------+
1 row in set (8.01 sec)
That is 8 seconds too slow
herewith the Explain extended for this query:
+----+--------------------+-------+--------+-------------------+-------------+---------+------------+---------+----------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+--------------------+-------+--------+-------------------+-------------+---------+------------+---------+----------+--------------------------+
| 1 | PRIMARY | t1 | index | NULL. | PRIMARY | 8 | NULL. | 1454343 | 100.00 | Using where |
| 2 | DEPENDENT SUBQUERY | lt1 | eq_ref | PRIMARY,id3,id4 | PRIMARY | 8 | const,func | 1 | 100.00 | Using where; Using index |
| 3 | SUBQUERY | pt1 | const | id4 | id4_idx | 4 | | 1 | 100.00 | Using index |
+----+--------------------+-------+--------+-------------------+-------------+---------+------------+---------+----------+--------------------------+
As I understand from this, the outside query doesn't actually use the index that it could.
What could we possibly be doing wrong in this query. Surely it should be running much much faster.
I tried running the outside query with the subqueries' result copy-pasted inside the IN clause (in other words the subqueries aren't run. It runs normally fast. Here's the explain extended then:
+----+-------------+-------+-------+----------------+---------+---------+------+------+----------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+-------+----------------+---------+---------+------+------+----------+-------------+
| 1 | SIMPLE | t1 | range | PRIMARY,k_id1 | PRIMARY | 4 | NULL | 5 | 100.00 | Using where |
+----+-------------+-------+-------+----------------+---------+---------+------+------+----------+-------------+
Oh yeah. This is running on MySQL 5.5

Answer 1

you could avoid the IN clause using an inner join
SELECT
t1.id1,
sum(t1.value)
FROM table1 t1
INNER JOIN (
SELECT lt1.id4
FROM lookuptable1 lt1
WHERE lt1.id3 = (SELECT pt1.id3
FROM lookuptable1 pt1
WHERE pt1.id4 = 5960)
) t on t.id4 = t1.id1 and t1.id2 = 3
and this could improve your query ..
be sure you have a proper index on table1 (id1, id2)