Mysql: Optimize query against huge tables - mysql

Question

My db tables are growing very fast (and will continue), at this time I have a problem with this query (well, others too):
select user_id from post where user_id not in (select id from user)
What I need is the new ids that are in post table and there are not in user table.
Here is the explain:
> mysql> explain select user_id from post where user_id not in (select
> id from user);
>
+----+--------------------+-------+-----------------+---------------+---------+---------+------+----------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref |rows | Extra |
+----+--------------------+-------+-----------------+---------------+---------+---------+------+----------+-------------+
| 1 | PRIMARY | post | ALL | NULL |NULL | NULL | NULL | 16076920 | Using where |
| 2 | DEPENDENT SUBQUERY | user | unique_subquery | PRIMARY | PRIMARY | 8 | func | 1 | Using index |
+----+--------------------+-------+-----------------+---------------+---------+---------+------+----------+-------------+
I have tried this other too:
SELECT p.user_id FROM post p LEFT JOIN user u ON p.user_id=u.id WHERE u.id IS NULL;
The explain:
mysql> EXPLAIN SELECT p.user_id FROM post p LEFT JOIN user u ON p.user_id=u.id WHERE u.id IS NULL;
+----+-------------+-------+--------+---------------+---------+---------+-----------------+----------+--------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+---------------+---------+---------+-----------------+----------+--------------------------------------+
| 1 | SIMPLE | p | ALL | NULL | NULL | NULL | NULL | 14323335 | |
| 1 | SIMPLE | u | eq_ref | PRIMARY | PRIMARY | 8 | ghost.p.user_id | 1 | Using where; Using index; Not exists |
+----+-------------+-------+--------+---------------+---------+---------+-----------------+----------+--------------------------------------+
Both queries have to see the entire table post, and it is very huge:
post table: 16077899 entries
user table: 9657158 entries
The query would take several minutes (more than 30min) to perform, any tip ?
Thanks!

Answer 1

I think you should do two things...
Make sure you have and index on post (user_id)
Add the DISTINCT keyword to your query, like this:
SELECT DISTINCT user_id
FROM post
WHERE user_id NOT IN (SELECT id FROM user)
Check the new EXPLAIN PLAN.

Answer 2

I am no expert in database, but your first query explain didn't use a Index on userid, do you have any index built on user_id field in post table? If not just create that. Also, you can try group by/distinct to filter your users, as first query will otherwise return multiple userid from post. This increase speed in all.