How to purge big MySQL database old entries from single column? - mysql

Question

I need to remove old database entries payload, while keeping other data (id and other properties) of same entries.
Table in question has message_id column (which consists of a datestamp concatenated with other info), content column (which is BLOB, and it makes over 90% of database total size) and some other columns that we have no use for in this case.
I've first tried running simple update with condition:
UPDATE LOW_PRIORITY repository SET content="" WHERE SUBSTR( message_id, 6, 6 )<201601 AND message_box = "IN";
I extract a YYYYMM from every entry message_id, and if it's older than a chosen cutoff month - I replace content with an empty string.
Database is over 25GB in size, and holds almost 2KK entries in my table, and is running on a very modest hardware, and my query failed with error after running for some time:
ERROR 2013 (HY000): Lost connection to MySQL server during query
Usually I try to avoid changing database variables, but i knew this error also pops up when you try restoring database from a large dumpfile, therefore I went and updated setting to handle 100MB packet size:
set global max_allowed_packet=104857600;
Re-running my UPDATE query resulted in a new error:
ERROR 2013 (HY000): Lost connection to MySQL server during query
As I have mentioned before - my MySQL server runs on a very modest hardware, and I'd prefer not to modify settings that could make server exceed available resources, therefore instead of increasing all available timeout database variables, I've decided to run my query in smaller chunks with a query like this:
UPDATE LOW_PRIORITY repository SET content="" WHERE message_id in (select message_id from(select message_id from repository where SUBSTR( message_id, 6, 6 )<201603 AND message_box = "IN" limit 0, 1000)as temp);
This query fails with an error:
ERROR 1206 (HY000): The total number of locks exceeds the lock table size
It also fails with a same query when limited even to single line with "limit 1"!
Do I use pagination incorrectly, or is there another better way of doing this?
*DB is running an a virtual Ubuntu server with dual core Intel CPU with 1GB of RAM and 100GB HDD. I't completely adequate for it's daily tasks, and I'd really like not to increase specs for just this one query.

Answer 1

You are trying to trick mysql into doing something it doesn't want (using limit in an in-statement) in a complicated way (complicated = more resources). That is not wrong, but you can just write
UPDATE LOW_PRIORITY repository SET content=""
WHERE content <> ""
and SUBSTR( message_id, 6, 6 ) < 201603 AND message_box = "IN"
limit 1000;
This will update the first 1000 old rows that still have content in it.

Answer 2

I would imagine your #1 problem here is that your WHERE condition will not be able to use an index on message_id field.
Why not simply do:
WHERE message_id < 20160100* ...
Assuming this is integer field, 201512** would be less the 201601** anyway so there would be no change in your outcome. But removing the substring function would allow you to use index on that field.