Optimise comparing data in two big MySQL tables - mysql

Question

How could I optimise query, which will find all records, which:
have activation_request.date_confirmed not null
and
do not have related string value in another table: activation_request.email =
user.username shouldn't return any record
I tried:
SELECT email
FROM activation_request l
LEFT JOIN user r ON r.username = l.email
WHERE l.date_confirmed is not null
AND r.username IS NULL
and
SELECT email
FROM activation_request
WHERE date_confirmed is not null
AND NOT EXISTS (SELECT 1
FROM user
WHERE user.username = activation_request.email
)
but both tables have xxx.xxx.xxx records hence after all night running those queries unfortunatelly I haven't got any results.
Create statements:
CREATE TABLE `activation_request` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`version` bigint(20) NOT NULL,
`date_confirmed` datetime DEFAULT NULL,
`email` varchar(255) NOT NULL,
(...)
PRIMARY KEY (`id`),
KEY `emailIdx` (`email`),
KEY `reminderSentIdx` (`date_reminder_sent`),
KEY `idx_resent_needed` (`date_reminder_sent`,`date_confirmed`),
) ENGINE=InnoDB AUTO_INCREMENT=103011867 DEFAULT CHARSET=utf8;
CREATE TABLE `user` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`version` bigint(20) NOT NULL,
`username` varchar(255) NOT NULL,
(...)
PRIMARY KEY (`id`),
UNIQUE KEY `Q52plW9W7TJWZcLj00K3FmuhwMSw4F7vmxJGyjxz5iiINVR9fXyacEoq4rHppb` (`username`),
) ENGINE=InnoDB AUTO_INCREMENT=431400048 DEFAULT CHARSET=latin1;
Explain for LEFT JOIN:
[[id:1, select_type:SIMPLE, table:l, type:ALL, possible_keys:null,
key:null, key_len:null, ref:null, rows:49148965, Extra:Using where],
[id:1, select_type:SIMPLE, table:r, type:index, possible_keys:null,
key:Q52plW9W7TJWZcLj00K3FmuhwMSw4F7vmxJGyjxz5iiINVR9fXyacEoq4rHppb,
key_len:257, ref:null, rows:266045508, Extra:Using where; Not exists;
Using index; Using join buffer (Block Nested Loop)]] [[id:1,
select_type:SIMPLE, table:l, type:ALL, possible_keys:null, key:null,
key_len:null, ref:null, rows:49148965, Extra:Using where], [id:1,
select_type:SIMPLE, table:r, type:index, possible_keys:null,
key:Q52plW9W7TJWZcLj00K3FmuhwMSw4F7vmxJGyjxz5iiINVR9fXyacEoq4rHppb,
key_len:257, ref:null, rows:266045508, Extra:Using where; Not exists;
Using index; Using join buffer (Block Nested Loop)]]
After adding indexes on staging db (with slightly less data, but the same structure) query is now running ~24h and still no results):
$ show processlist;
| Id | User | Host | db | Command | Time | State | Info
| 64 | root | localhost | staging_db | Query | 110072 | Sending data | SELECT ar.email FROM activation_request ar WHERE ar.date_confirmed is not null AND NOT EXISTS (SELE |
Mysql version:
$ select version();
5.6.16-1~exp1
All other commands on the list are Sleep so there is no other query running and possibly disturbing/locking rows.

Answer 1

For this query:
SELECT ar.email
FROM activation_request ar
WHERE ar.date_confirmed is not null AND
NOT EXISTS (SELECT 1
FROM user u
WHERE u.username = ar.email
)
I would recommend indexes on activation_request(date_confirmed, email) and user(username).
Unless you have a really humongous amount of data, though, your problem may be that tables are locked.