SQL optimization: Finding first unwatched video using no subselect - mysql

Question

My question. I think I have a performance killing subquery but can't prove it. My first tries using JOIN failed. Can somebody provide a more high performing solution or confirm that this is indeed acceptable as it is?
I have two tables, one that contains a todo-list (joblist) and one that tracks every user's progress (userprogress). A job can but must not be to watch a video. (It's a site for e-learning.)
When videos have been watched they are automatically set to 'finished' on an enum field. Users may also skip videos manually (status = 'skipped').
Table structures are provided below.
To get the first video that a user have not watched at all (no record in userprogress) or has begun to watch (status = 'begun') I am using this query.
I have set indexes on ever field that is being used for selection or ordering. However I am unsure if they are all needed.
The SELECT statement has two parts
An inner subselect, where I fetch all seen or skipped videos
The main statement, where I fetch the first video not among the ones found by (1)
There is a named parameter for PHP (:email), to avoid SQL-injection.
SELECT jl.where_to_do_it FROM joblist AS jl
INNER JOIN userprogress AS up
ON (jl.joblistID = up.joblistID)
WHERE jl.what_to_do = 'video'
AND jl.joblistID NOT IN
(
SELECT injl.joblistID
FROM joblist AS injl
INNER JOIN userprogress AS inup
ON (injl.joblistID = inup.joblistID)
WHERE
(inup.status = 'finished' OR inup.status = 'skipped')
AND
inup.email = :email
AND
injl.what_to_do = 'video'
)
ORDER BY jl.joborder ASC
LIMIT 0,1
This is the output from EXPLAIN, which I need some help understanding.
id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY jl ref PRIMARY,what_to_do what_to_do 602 const 9 Using where; Using filesort
1 PRIMARY up ref joblistID joblistID 3 jl.joblistID 1 Using index
2 DEP-SUB injl eq_ref PRIMARY,what_to_do PRIMARY 3 func 1 Using where
2 DEP-SUB inup eq_ref nodup,email,joblistID,status nodup 455 const,func 1 Using where
The create table commands:
CREATE TABLE IF NOT EXISTS `joblist` (
`joblistID` mediumint(10) unsigned NOT NULL AUTO_INCREMENT,
`what_to_do` varchar(200) COLLATE utf8_swedish_ci NOT NULL,
`where_to_do_it` varchar(100) COLLATE utf8_swedish_ci NOT NULL,
`joborder` mediumint(6) NOT NULL,
`track` enum('fast','slow','bonus') COLLATE utf8_swedish_ci NOT NULL DEFAULT 'slow',
`chapter` tinyint(11) unsigned NOT NULL COMMENT 'What book chapter it relates to',
PRIMARY KEY (`joblistID`),
KEY `nodupjobs` (`joborder`,`chapter`),
KEY `what_to_do` (`what_to_do`),
KEY `where_to_do_it` (`where_to_do_it`),
KEY `joborder` (`joborder`),
KEY `track` (`track`),
KEY `chapter` (`chapter`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_swedish_ci COMMENT='Suggested working order';
CREATE TABLE IF NOT EXISTS `userprogress` (
`upID` int(10) unsigned NOT NULL AUTO_INCREMENT,
`email` varchar(150) COLLATE utf8_swedish_ci NOT NULL COMMENT 'user id',
`joblistID` mediumint(9) unsigned NOT NULL COMMENT 'foreign key',
`progressdata` varchar(300) COLLATE utf8_swedish_ci DEFAULT NULL COMMENT 'JSON object describing progress',
`percentage_complete` tinyint(3) unsigned DEFAULT NULL,
`status` enum('begun','skipped','finished') COLLATE utf8_swedish_ci DEFAULT 'begun',
`lastupdate` datetime NOT NULL,
`approved` datetime DEFAULT NULL,
PRIMARY KEY (`upID`),
UNIQUE KEY `nodup` (`email`,`joblistID`),
KEY `email` (`email`),
KEY `joblistID` (`joblistID`),
KEY `status` (`status`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_swedish_ci COMMENT='Keep track of what the user has done';

Answer 1

Yes, you are correct. IN and NOT IN are particularly bad performers in mysql. Here is a revised version:
SELECT jl.where_to_do_it
FROM joblist jl INNER JOIN
userprogress up
ON (jl.joblistID = up.joblistID)
WHERE jl.what_to_do = 'video' and
not exists (
(SELECT 1
FROM joblist injl INNER JOIN
userprogress inup
ON (injl.joblistID = inup.joblistID)
WHERE (inup.status = 'finished' OR inup.status = 'skipped') and
inup.email = :email and
injl.what_to_do = 'video' and
ini1.joblistid = j1.joblistid
)
ORDER BY jl.joborder ASC
LIMIT 0,1

Answer 2

Looks like you're running in circles... Your subquery is looking for Videos with status finished or skipped and then in the outter query lookin for the ones that don't have that status, I would change that for a condition like this
SELECT jl.where_to_do_it FROM joblist AS jl
INNER JOIN userprogress AS up
ON (jl.joblistID = up.joblistID)
WHERE jl.what_to_do = 'video'
AND up.status <> 'finished' AND inup.status <> 'skipped'
AND up.email = :email
AND jl.what_to_do = 'video'
Or maybe I understand wrong, anyway the problem seems to be the NOT IN (i will not suggest to use this ever) instead try changing the subquery in the condition and do a Left join with it and add a condition And SQ.joblistID IS NULL, something like this
SELECT jl.where_to_do_it FROM joblist AS jl
INNER JOIN userprogress AS up
ON (jl.joblistID = up.joblistID)
LEFT JOIN (
SELECT injl.joblistID
FROM joblist AS injl
INNER JOIN userprogress AS inup
ON (injl.joblistID = inup.joblistID)
WHERE
(inup.status = 'finished' OR inup.status = 'skipped')
AND
inup.email = :email
AND
injl.what_to_do = 'video'
) SQ ON jl.joblistID = SQ.joblistID
WHERE jl.what_to_do = 'video'
AND SQ.joblistID IS NULL
ORDER BY jl.joborder ASC
But i think the first option will work...
Hope it helps