mysql stored procedure to pop from a database based queue - mysql

Question

We have a system that has a database based queue for processing items in threads instead of real time. It's currently implemented in Mybatis calling a this stored procedure in mysql:
DROP PROCEDURE IF EXISTS pop_invoice_queue;
DELIMITER ;;
CREATE PROCEDURE pop_invoice_queue(IN compId int(11), IN limitRet int(11)) BEGIN
SELECT LAST_INSERT_ID(id) as value, InvoiceQueue.* FROM InvoiceQueue
WHERE companyid = compId
AND (lastPopDate is null OR lastPopDate < DATE_SUB(NOW(), INTERVAL 3 MINUTE)) LIMIT limitRet FOR UPDATE;
UPDATE InvoiceQueue SET lastPopDate=NOW() WHERE id=LAST_INSERT_ID();
END;;
DELIMITER ;
The problem is that this pops N items from the queue but only updates the lastPopDate value for the last item popped off the queue. So if we call this stored procedure with limitRet = 5, it will pop five items off the queue and start working on them but only the fifth item will have a lastPopDate set so when the next thread comes and pops off the queue it will get items 1-4 and item 6.
How can we get this to update all N records 'popped' off the database?

Answer 1

If you are willing to add a BIGINT field to the table via:
ALTER TABLE InvoiceQueue
ADD uuid BIGINT NULL DEFAULT NULL,
INDEX ix_uuid (uuid);
then you can do the update first, and select the records updated, via:
CREATE PROCEDURE pop_invoice_queue(IN compId int(11), IN limitRet int(11))
BEGIN
SET #uuid = UUID_SHORT();
UPDATE InvoiceQueue
SET uuid = #uuid,
lastPopDate = NOW()
WHERE companyid = compId
AND uuid IS NULL
AND (lastPopDate IS NULL OR lastPopDate < NOW() - INTERVAL 3 MINUTE)
ORDER BY
id
LIMIT limitRet;
SELECT *
FROM InvoiceQueue
WHERE uuid = #uuid
FOR UPDATE;
END;;
For the UUID_SHORT() function to return unique values, it should be called no more than 16 million times a second per machine. Visit here for more details.
For performance, you may want to alter the lastPopDate field to be NOT NULL as the OR clause will cause your query to not use an index, even if one is available:
ALTER TABLE InvoiceQueue
MODIFY lastPopDate DATETIME NOT NULL DEFAULT '0000-00-00';
Then, if you do not already have one, you could add an index on the companyid/lastPopDate/uuid fields, as follows:
ALTER TABLE InvoiceQueue
ADD INDEX ix_company_lastpop (companyid, lastPopDate, uuid);
Then you can remove the OR clause from your UPDATE query:
UPDATE InvoiceQueue
SET uuid = #uuid,
lastPopDate = NOW()
WHERE companyid = compId
AND lastPopDate < NOW() - INTERVAL 3 MINUTE
ORDER BY
id
LIMIT limitRet;
which will use the index you just created.

Answer 2

Since mysql has neither collection nor output/returning clause, my suggestion is to use temporary tables. Something like :
CREATE TEMPORARY TABLE temp_data
SELECT LAST_INSERT_ID(id) as value, InvoiceQueue.* FROM InvoiceQueue
WHERE companyid = compId
AND (lastPopDate is null OR lastPopDate < DATE_SUB(NOW(), INTERVAL 3 MINUTE)) LIMIT limitRet FOR UPDATE;
UPDATE InvoiceQueue
INNER JOIN temp_data ON (InvoiceQueue.PKColumn = temp_data.PKColumn)
SET lastPopDate=NOW();
SELECT * FROM temp_data ;
DROP TEMPORARY TABLE temp_data;
Also, I surmise such select ... for update can cause deadlocks (surely, if the procedure is called from different sessions) - as far as I know order in which rows get locked is not guaranteed (even if you had order by, rows might be locked in different order). I'd recommend to double check documentation.