Optimize DELETE FROM mdl_grade_items_history - mysql

Question

Last week I installed some additional database monitoring and have since come to discover that a full 30% of our database load is spent on a single query on a single table (which currently has some 6 million rows):
delete FROM mdl_grade_items_history WHERE timemodified < ?
In a testing environment, I tried to make some schema changes:
Running EXPLAIN on this query indicates that every time this query is run, a full table scan is done.
EXPLAIN DELETE FROM mdl_grade_items_history WHERE timemodified < '1490528405';
+----+-------------+-------------------------+------------+------+---------------+------+---------+------+--------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------------------------+------------+------+---------------+------+---------+------+--------+----------+-------------+
| 1 | DELETE | mdl_grade_items_history | NULL | ALL | NULL | NULL | NULL | NULL | 140784 | 100.00 | Using where |
+----+-------------+-------------------------+------------+------+---------------+------+---------+------+--------+----------+-------------+
1 row in set (0.00 sec)
Checking EXPLAIN for a (very similar) SELECT query shows a similar situation.
EXPLAIN SELECT id FROM mdl_grade_items_history WHERE timemodified < '1490528405';
+----+-------------+-------------------------+------------+------+---------------+------+---------+------+--------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------------------------+------------+------+---------------+------+---------+------+--------+----------+-------------+
| 1 | SIMPLE | mdl_grade_items_history | NULL | ALL | NULL | NULL | NULL | NULL | 140784 | 33.33 | Using where |
+----+-------------+-------------------------+------------+------+---------------+------+---------+------+--------+----------+-------------+
1 row in set, 1 warning (0.01 sec)
Checking the table definition, there does not seem to be an index on timemodified
SHOW INDEX FROM mdl_grade_items_history;
+-------------------------+------------+-------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-------------------------+------------+-------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| mdl_grade_items_history | 0 | PRIMARY | 1 | id | A | 140784 | NULL | NULL | | BTREE | | |
| mdl_grade_items_history | 1 | mdl_graditemhist_act_ix | 1 | action | A | 2 | NULL | NULL | | BTREE | | |
| mdl_grade_items_history | 1 | mdl_graditemhist_old_ix | 1 | oldid | A | 17170 | NULL | NULL | | BTREE | | |
| mdl_grade_items_history | 1 | mdl_graditemhist_cou_ix | 1 | courseid | A | 1065 | NULL | NULL | YES | BTREE | | |
| mdl_grade_items_history | 1 | mdl_graditemhist_cat_ix | 1 | categoryid | A | 2300 | NULL | NULL | YES | BTREE | | |
| mdl_grade_items_history | 1 | mdl_graditemhist_sca_ix | 1 | scaleid | A | 6 | NULL | NULL | YES | BTREE | | |
| mdl_grade_items_history | 1 | mdl_graditemhist_out_ix | 1 | outcomeid | A | 1 | NULL | NULL | YES | BTREE | | |
| mdl_grade_items_history | 1 | mdl_graditemhist_log_ix | 1 | loggeduser | A | 30 | NULL | NULL | YES | BTREE | | |
+-------------------------+------------+-------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
8 rows in set (0.00 sec)
So I tried to add one (both via CREATE INDEX and ALTER TABLE .. ADD INDEX)
CREATE INDEX `mdl_gradeitemhist_tim_ix` ON `mdl_grade_items_history` (`timemodified`);
ALTER TABLE `mdl_grade_items_history` ADD INDEX `mdl_gradeitemhist_tim_ix` (`timemodified`);
In both instances, the SELECT query was affected (note the change in type)
EXPLAIN `SELECT` id FROM mdl_grade_items_history WHERE timemodified < '1490528405';
+----+-------------+-------------------------+------------+-------+--------------------------+--------------------------+---------+------+-------+----------+--------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------------------------+------------+-------+--------------------------+--------------------------+---------+------+-------+----------+--------------------------+
| 1 | SIMPLE | mdl_grade_items_history | NULL | range | mdl_gradeitemhist_tim_ix | mdl_gradeitemhist_tim_ix | 9 | NULL | 70206 | 100.00 | Using where; Using index |
+----+-------------+-------------------------+------------+-------+--------------------------+--------------------------+---------+------+-------+----------+--------------------------+
1 row in set, 1 warning (0.00 sec)
But not the DELETE query.
EXPLAIN DELETE FROM mdl_grade_items_history WHERE timemodified < '1490528405';
+----+-------------+-------------------------+------------+------+--------------------------+------+---------+------+--------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------------------------+------------+------+--------------------------+------+---------+------+--------+----------+-------------+
| 1 | DELETE | mdl_grade_items_history | NULL | ALL | mdl_gradeitemhist_tim_ix | NULL | NULL | NULL | 140412 | 100.00 | Using where |
+----+-------------+-------------------------+------------+------+--------------------------+------+---------+------+--------+----------+-------------+
1 row in set (0.00 sec)
What have I done wrong? What else could I try?

Answer 1

Low cardinality indexes (action, scaleid, outcomeid) are almost never used. Get rid of them.
Having a large number of single-column indexes is a red flag. Please learn about the power and benefit of "composite" indexes. (Not relevent for the select/delete mentioned here, but probably relevant for other queries.)
Extra indexes on a table slightly slow down INSERTs and DELETEs since the indexes need to (eventually) be updated.
Extra indexes slow down UPDATEs if an indexed column is modified.
CREATE INDEX and ALTER TABLE ADD INDEX do the same thing; you probably have a redundant index now.
The EXPLAINs are different because (1) SELECT and DELETE do different things and (2) EXPLAIN is not very sophisticated.
Deleting a large number of rows takes a lot of effort -- Keep in mind that the deleted rows are hung onto in case of a ROLLBACK. Only after COMMIT can the rows really be removed. (With autocommit=ON, there is an implicit COMMIT.)
Tips on large deletes:
Deleting in chunks
Using PARTITIONs for very efficient deletion of time series