I have a stored procedure in mysql thats to perform a task that needs to be synchronized such that if two application calls the stored procedure, only one can access a section of code to perform the task, keeping the other one to get blocked until the first one finishes the task.
DELIMITER $$
CREATE PROCEDURE SP_GEN_ID(IN NAME VARCHAR(20))
BEGIN
DECLARE maxLen int default 0;
START TRANSACTION;
#the section of code that needs to be synchronized
COMMIT
END;
$$
DELIMITER ;
So, if two applications call the stored procedure simultaneously, the task has to be synchronized.
a. But Start TRANSACTION and COMMIT did NOT synchronize the execution.
b. And LOCK TABLES tableA can not be used in stored procedure to ensure the synchronization too.
c. I tried to synchronize the stored procedure call in application level. I used
boost_interprocess scoped_lock lock();
It worked perfectly fine in boost 1.41
But interprocess locking mutex is not supported in the boost 1.34 library, which is what is available in my case.
Is there a way to synchronize the stored procedure section of code such that when two calls are made simultaneously, one gets blocked before the other gets executed?
(added the following)
edited code: to give an idea what I am trying to perform in the synchronized block of the stored procedure.
It gets the last assigned id, and increment it by one and check whether it is not used for someother 'name' record.
When a valid id is found, update the last assigned id record table and then associate that with the 'name' given.
DELIMITER $$
CREATE PROCEDURE SP_GEN_ID(IN NAME VARCHAR(20))
BEGIN
DECLARE maxLen int default 0;
START TRANSACTION;
#the section of code that needs to be synchronized
SELECT lastid into lastgenerated FROM DB_last_id WHERE key = 'NAME_ID';
findid_loop:
LOOP
set lastid = lastid + 1;
#this is to check whether it was assigned with someother name before.
IF not EXISTS (SELECT 1 FROM user_name_id WHERE name_id = lastgenerated) then
set nameid = lastgenerated;
set found = true;
LEAVE findid_loop;
END IF;
#for loop limit check
IF (counter < loopLimit) then
set counter = counter + 1;
ITERATE findid_loop;
ELSE
#reached the limit and not found.
LEAVE findid_loop;
END IF;
END LOOP findid_loop;
#if a valid id, update last id and assign to name.
IF (found) THEN
#update the id.
update DB_last_id set lastid = nameid where key = 'NAME_ID';
insert into user_name_id values (nameid ,name);
ELSE
#return an empty string for the application to take action on the failure.
set nameid = '';
END IF;
#end transaction here.
COMMIT
END;
$$
DELIMITER ;
As mentioned in my comments above, you should find that a transaction is sufficient for most needs; however, if you need to explicitly wait until the other call has completed, use GET_LOCK(str,timeout):
Tries to obtain a lock with a name given by the string str, using a timeout of timeout seconds. Returns 1 if the lock was obtained successfully, 0 if the attempt timed out (for example, because another client has previously locked the name), or NULL if an error occurred (such as running out of memory or the thread was killed with mysqladmin kill). If you have a lock obtained with GET_LOCK(), it is released when you execute RELEASE_LOCK(), execute a new GET_LOCK(), or your connection terminates (either normally or abnormally). Locks obtained with GET_LOCK() do not interact with transactions. That is, committing a transaction does not release any such locks obtained during the transaction.
This function can be used to implement application locks or to simulate record locks. Names are locked on a server-wide basis. If a name has been locked by one client, GET_LOCK() blocks any request by another client for a lock with the same name. This enables clients that agree on a given lock name to use the name to perform cooperative advisory locking. But be aware that it also enables a client that is not among the set of cooperating clients to lock a name, either inadvertently or deliberately, and thus prevent any of the cooperating clients from locking that name. One way to reduce the likelihood of this is to use lock names that are database-specific or application-specific. For example, use lock names of the form db_name.str or app_name.str.
mysql> SELECT GET_LOCK('lock1',10);
-> 1
mysql> SELECT IS_FREE_LOCK('lock2');
-> 1
mysql> SELECT GET_LOCK('lock2',10);
-> 1
mysql> SELECT RELEASE_LOCK('lock2');
-> 1
mysql> SELECT RELEASE_LOCK('lock1');
-> NULL
The second RELEASE_LOCK() call returns NULL because the lock 'lock1' was automatically released by the second GET_LOCK() call.
If multiple clients are waiting for a lock, the order in which they will acquire it is undefined and depends on factors such as the thread library in use. In particular, applications should not assume that clients will acquire the lock in the same order that they issued the lock requests.
Note
Before MySQL 5.5.3, if a client attempts to acquire a lock that is already held by another client, it blocks according to the timeout argument. If the blocked client terminates, its thread does not die until the lock request times out.
This function is unsafe for statement-based replication. Beginning with MySQL 5.5.1, a warning is logged if you use this function when binlog_format is set to STATEMENT. (Bug #47995)
Starting a transaction with START TRANSACTION does not actually start it. The first table access following START TRANSACTION does. Opening a transaction isn't also a mean for concurrency control. If you need just that, you can rely on the advisory locks system MySQL provides through GET_LOCK(), RELEASE_LOCK(), and a few other related functions.
An alternative way to implement concurrency control, through transactions this time, would be by relying on exclusive row locks. Since SELECT statements are non-locking in InnoDB, issuing such query starts a transaction, however it neither sets any locks nor respects any pre-existing ones. To have a SELECT statement actually block if there is an earlier transaction operating on the same information (row), you have to use FOR UPDATE clause. For example:
START TRANSACTION;
SELECT lastid into lastgenerated FROM DB_last_id WHERE key = 'NAME_ID' FOR UPDATE;
...
With this construction there will never be two concurrent transactions operating on the same 'NAME_ID' past the SELECT statement that was explicitly told to perform a locking read.
Related
I am trying to write a mySQL stored procedure that does the following:
1. insert a new row into table A;
2. read the new row from table A;
3. using information gleaned from step 2, insert a new row into table B
I need step 2 to wait until step 1 is finished, as table B uses one of the columns - to be more specific the autoincrement index - of table A as a foreign key. I cannot know what that value is until the new row has been inserted.
Usually, step 2 does take place after step 1 is finished, but I don't want any hiccups caused by a slow insertion or any other unforeseen factors.
On second thought, I probably also need mySQL to block any new insertions into table A until I have finished reading from it.
Any ideas? Thank you very much!
You can use the GET_LOCK() and RELEASE_LOCK() functions in MySQL to block or delay the execution of a query until a previous one has been completed within a stored procedure.
The GET_LOCK() function can be used to acquire a named lock, which prevents other sessions from acquiring the same lock until the lock is released. If the lock is already held by another session, the GET_LOCK() function will block until the lock is released or a timeout is reached.
The RELEASE_LOCK() function can be used to release a previously acquired named lock.
Here is an example of how to use GET_LOCK() and RELEASE_LOCK() in a stored procedure:
DELIMITER $$
CREATE PROCEDURE my_procedure()
BEGIN
DECLARE lock_name VARCHAR(255) DEFAULT 'my_lock';
-- Acquire the lock
SELECT GET_LOCK(lock_name, 60); -- Wait up to 60 seconds for the lock to be released
-- Execute your query here
-- Release the lock
SELECT RELEASE_LOCK(lock_name);
END$$
DELIMITER ;
In this example, the GET_LOCK() function is called with a lock name and a timeout value of 60 seconds. If the lock is already held by another session, the query will block until the lock is released or the timeout is reached.
After the lock is acquired, you can execute your query. Once the query is complete, you can release the lock by calling the RELEASE_LOCK() function with the same lock name.
It's important to note that you should use unique lock names to avoid conflicts with other sessions. Also, be careful not to hold the lock for too long, as this can lead to blocking and contention issues in your database.
You can use a transaction. This is exactly what they are for: avoiding inconsistent database states.
Also, for your step 2, you can use the LAST_INSERT_ID function. Even if you do not use a transaction, this function will return the last ID inserted by the current connection; if other connections insert a row, it won't affect the result of LAST_INSERT_ID() in your connection.
So:
INSERT INTO people (name) VALUES ('John');
INSERT INTO addresses (person_id, address) VALUES (LAST_INSERT_ID(), 'There');
is likely sufficient for what you need; if you have more complex logic, transactions will help:
BEGIN TRANSACTION;
-- do anything
COMMIT;
A transaction is treated as a logical unit; other modifications to the database will either happen before or after the transaction, not during one.
I want to have a stored procedure that does the following:
Locks a table
Checks for a value in it
Updates same table based on that value
Unlocks the table
If an error occurs between 1 and 4, will the table be unlocked? Or do I need to capture the error somehow? (how?)
Is there a better way to do this?
You can't lock a table within a stored procedure in MySQL.
SQL Statements Not Permitted in Stored Routines
Stored routines cannot contain arbitrary SQL statements. The following statements are not permitted:
The locking statements LOCK TABLES and UNLOCK TABLES.
— http://dev.mysql.com/doc/refman/5.6/en/stored-program-restrictions.html
If you are using InnoDB, then you can accomplish your purpose by locking the rows of interest using locking reads with SELECT ... FOR UPDATE. When you hit an error and roll back the transaction, the rows are unlocked automatically.
I wrote about this in detail in this recent answer where the question involved avoiding conflicting inserts but the underlying concept is the same whether you know the row you want already exists, or whether it might or might not exist.
Have you considered using transactions with a try-catch block? See this:
BEGIN TRAN
SAVE TRAN S1 -- Savepoint so any rollbacks will only affect this transaction
BEGIN TRY
/* Do your work in here */
END TRY
BEGIN CATCH
ROLLBACK TRAN S1 -- rollback just this transaction
SET #ErrorMessage = ERROR_MESSAGE()
SET #Severity = ERROR_SEVERITY()
SET #State = ERROR_STATE()
RAISERROR(#ErrorMessage, #Severity, #State) -- re-throw error if needed
END CATCH
Let's say we execute...
SELECT * FROM MY_TABLE FOR UPDATE
...and there is more than one row in MY_TABLE.
Theoretically, if two concurrent transactions execute this statement, but it happens to traverse (and therefore lock) the rows in different order, a deadlock may occur. For example:
Transaction 1: Locks row A.
Transaction 2: Locks row B.
Transaction 1: Attempts to lock row B and blocks.
Transaction 2: Attempts to lock row A and deadlocks.
The way to resolve this is to use ORDER BY to ensure rows are always locked in the same order.
So, my question is: will this theoretical deadlock ever occur in practice? I know there are ways to artificially induce it, but could it ever occur in the normal operation? Should we just always use ORDER BY, or it's actually safe to omit it?
I'm primarily interested in behavior of Oracle and MySQL/InnoDB, but comments on other DBMSes would be helpful as well.
--- EDIT ---
Here is how to reproduce a deadlock under Oracle when locking order is not the same:
Create the test table and fill it with some test data...
CREATE TABLE DEADLOCK_TEST (
ID INT PRIMARY KEY,
A INT
);
INSERT INTO DEADLOCK_TEST SELECT LEVEL, 1 FROM DUAL CONNECT BY LEVEL <= 10000;
COMMIT;
...from one client session (I used SQL Developer), run the following block:
DECLARE
CURSOR CUR IS
SELECT * FROM DEADLOCK_TEST
WHERE ID BETWEEN 1000 AND 2000
ORDER BY ID
FOR UPDATE;
BEGIN
WHILE TRUE LOOP
FOR LOCKED_ROW IN CUR LOOP
UPDATE DEADLOCK_TEST
SET A = -99999999999999999999
WHERE CURRENT OF CUR;
END LOOP;
ROLLBACK;
END LOOP;
END;
/
From a different client session (I simply started one more instance of SQL Developer), run that same block, but with DESC in the ORDER BY. After few seconds, you'll get the:
ORA-00060: deadlock detected while waiting for resource
BTW, you'll likely achieve the same result by completely removing the ORDER BY (so both blocks are identical), and adding the...
ALTER SESSION SET OPTIMIZER_INDEX_COST_ADJ = 1;
...in front of one block but...
ALTER SESSION SET OPTIMIZER_INDEX_COST_ADJ = 10000;
...in front of the other (so Oracle chooses different execution plans and likely fetches the rows in different order).
This illustrates that locking is indeed done as rows are fetched from the cursor (and not for the whole result-set at once when the cursor is opened).
Your example in your question shows that the order of locking depends upon the access method. This access path is not directly decided by the ORDER BY clause of the query, there are many factors that can influence this access path. Therefore, you can't prevent a deadlock just by adding an ORDER BY because you could still have two distinct access paths. In fact by running your test case with the order by and changing the session parameters I was able to cause two session to run into an ORA-60 with the same query.
If the sessions involved have no other lock pending, locking the rows in the same order in all sessions will prevent deadlocks but how can you reliably force this order? Note that this would only prevent this very special case of deadlock anyway. You could still get deadlocks with multiple queries in each session or different plans.
In practice this case is really special and shouldn't happen often anyway: if you're worried about deadlocks, I still think there are easier methods to prevent them.
The easiest way to prevent a deadlock is to use either FOR UPDATE NOWAIT or FOR UPDATE WAIT X (although WAIT X can still trigger a deadlock with values of X superior to the deadlock detection mechanism, currently 3 seconds as of 11g I believe -- thanks #APC for the correction).
In other words, both transactions should ask: give me those rows and lock them but if another user already has a lock return an error instead of waiting indefinitely. It is the indefinite waiting that causes deadlocks.
In practice I would say that most applications with real person users would rather receive an error immediately than have a transaction wait indefinitely for another transaction to finish. I would consider FOR UPDATE without NOWAIT only for non-critical batch jobs.
I think you have misunderstood how FOR UPDATE works. It acquires the locks when the cursor is activated ;that is, when the SELECT is issued.
So, running your query, Transaction 1 will lock the entire table (because you haven't specified a WHERE clause). Transaction 2 will either hang or fail (depending on what you've specified in the WAIT clause) regardless of whether Transaction 1 has issued any DML against the selected set of records. If fact, Transaction 1 doesn't even have to fetch any records; Transaction 2 will hurl ORA-00054 once Transaction 1 has opened the FOR UPDATE cursor.
The deadlock scenario you describe is the classic outcome of an application which uses optimistic locking (i.e. assumes it will be able to acquire a lock when it needs to). The whole point of FOR UPDATE is that it is a pessimistic locking strategy: grab all the locks potentially required now in order to guarantee successful processing in the future.
The inestimable Mr Kyte provides the crucial insight in his blog:
"deadlock detection trumps a waiting period"
In my code I was using NOWAIT in the FOR UPDATE clause of the cursor used in the second session:
cursor c10000 is
select * from order_lines
where header_id = 1234
for update;
cursor c1 is
select * from order_lines
where header_id = 1234
and line_id = 9999
for update nowait;
Consequently Session 2 fails immediately and hurls ORA-00054.
However the OP doesn't specify anything, in which case the second session will wait indefinitely for the row to be released. Except that it doesn't, because after a while deadlock detection kicks in and terminates the command with extreme prejudice i.e. ORA-00060. If they had specified a short wait period - say WAIT 1 - they would have seen ORA-30006: resource busy.
Note that this happens regardless of whether we use the verbose syntax...
open c10000;
loop
fetch c10000 into r;
or the snazzier....
for r in c10000 loop
And it really doesn't matter whether Session 1 has fetched the row of interest when Session 2 starts.
tl;dr
So the key thing is, ORDER BY doesn't solve anything. The first session to issue FOR UPDATE grabs all the records in the result set. Any subsequent session attempting to update any of those records will fail with either ORA-00054, ORA-30006 or ORA-00060, depending on whether they specified NOWAIT, WAIT n or nothing.... unless the first session releases the locks before the WAIT period times out or deadlock detection kicks in.
Here is a worked example. I am using an autonmous transaction to simulate a second session. The effect is the same but the output is easier to read.
declare
cursor c1 is
select * from emp
where deptno = 10
for update;
procedure s2
is
cursor c2 is
select * from emp
where empno = 7934 -- one of the employees in dept 10
for update
-- for update nowait
-- for update wait 1
;
x_deadlock exception;
pragma exception_init( x_deadlock, -60);
x_row_is_locked exception;
pragma exception_init( x_row_is_locked, -54);
x_wait_timeout exception;
pragma exception_init( x_wait_timeout, -30006);
pragma autonomous_transaction;
begin
dbms_output.put_line('session 2 start');
for r2 in c2 loop
dbms_output.put_line('session 2 got '||r2.empno);
update emp
set sal = sal * 1.1
where current of c2;
dbms_output.put_line('session 2 update='||sql%rowcount);
end loop;
rollback;
exception
when x_deadlock then
dbms_output.put_line('session 2: deadlock exception');
when x_row_is_locked then
dbms_output.put_line('session 2: nowait exception');
when x_wait_timeout then
dbms_output.put_line('session 2: wait timeout exception');
end s2;
begin
for r1 in c1 loop
dbms_output.put_line('session 1 got '||r1.empno);
s2;
end loop;
end;
/
In this version I have specified a straightfor update in the second session. This is the configuration the OP uses and as can be seen from the output hurls because a deadlock has been detected:
session 1 got 7782
session 2 start
session 2: deadlock exception
session 1 got 7839
session 2 start
session 2: deadlock exception
session 1 got 7934
session 2 start
session 2: deadlock exception
PL/SQL procedure successfully completed.
What this clearly demonstrates is
The first session has locked the whole result set from the go-get, because the second session never gets a lock on that one row, even when the first session has not yet retrieved it.
The Deadlock detected exception is hurled even though the second session has not been able to update anything.
1. The Deadlock detected exception is hurled even though the first session does not update any of the fetched wows.
The code is easily modifiable to demonstrate the different behaviours of the FOR UPDATE variants.
I have a stored procedure in mysql thats to perform a task that needs to be synchronized such that if two application calls the stored procedure, only one can access a section of code to perform the task, keeping the other one to get blocked until the first one finishes the task.
DELIMITER $$
CREATE PROCEDURE SP_GEN_ID(IN NAME VARCHAR(20))
BEGIN
DECLARE maxLen int default 0;
START TRANSACTION;
#the section of code that needs to be synchronized
COMMIT
END;
$$
DELIMITER ;
So, if two applications call the stored procedure simultaneously, the task has to be synchronized.
a. But Start TRANSACTION and COMMIT did NOT synchronize the execution.
b. And LOCK TABLES tableA can not be used in stored procedure to ensure the synchronization too.
c. I tried to synchronize the stored procedure call in application level. I used
boost_interprocess scoped_lock lock();
It worked perfectly fine in boost 1.41
But interprocess locking mutex is not supported in the boost 1.34 library, which is what is available in my case.
Is there a way to synchronize the stored procedure section of code such that when two calls are made simultaneously, one gets blocked before the other gets executed?
(added the following)
edited code: to give an idea what I am trying to perform in the synchronized block of the stored procedure.
It gets the last assigned id, and increment it by one and check whether it is not used for someother 'name' record.
When a valid id is found, update the last assigned id record table and then associate that with the 'name' given.
DELIMITER $$
CREATE PROCEDURE SP_GEN_ID(IN NAME VARCHAR(20))
BEGIN
DECLARE maxLen int default 0;
START TRANSACTION;
#the section of code that needs to be synchronized
SELECT lastid into lastgenerated FROM DB_last_id WHERE key = 'NAME_ID';
findid_loop:
LOOP
set lastid = lastid + 1;
#this is to check whether it was assigned with someother name before.
IF not EXISTS (SELECT 1 FROM user_name_id WHERE name_id = lastgenerated) then
set nameid = lastgenerated;
set found = true;
LEAVE findid_loop;
END IF;
#for loop limit check
IF (counter < loopLimit) then
set counter = counter + 1;
ITERATE findid_loop;
ELSE
#reached the limit and not found.
LEAVE findid_loop;
END IF;
END LOOP findid_loop;
#if a valid id, update last id and assign to name.
IF (found) THEN
#update the id.
update DB_last_id set lastid = nameid where key = 'NAME_ID';
insert into user_name_id values (nameid ,name);
ELSE
#return an empty string for the application to take action on the failure.
set nameid = '';
END IF;
#end transaction here.
COMMIT
END;
$$
DELIMITER ;
As mentioned in my comments above, you should find that a transaction is sufficient for most needs; however, if you need to explicitly wait until the other call has completed, use GET_LOCK(str,timeout):
Tries to obtain a lock with a name given by the string str, using a timeout of timeout seconds. Returns 1 if the lock was obtained successfully, 0 if the attempt timed out (for example, because another client has previously locked the name), or NULL if an error occurred (such as running out of memory or the thread was killed with mysqladmin kill). If you have a lock obtained with GET_LOCK(), it is released when you execute RELEASE_LOCK(), execute a new GET_LOCK(), or your connection terminates (either normally or abnormally). Locks obtained with GET_LOCK() do not interact with transactions. That is, committing a transaction does not release any such locks obtained during the transaction.
This function can be used to implement application locks or to simulate record locks. Names are locked on a server-wide basis. If a name has been locked by one client, GET_LOCK() blocks any request by another client for a lock with the same name. This enables clients that agree on a given lock name to use the name to perform cooperative advisory locking. But be aware that it also enables a client that is not among the set of cooperating clients to lock a name, either inadvertently or deliberately, and thus prevent any of the cooperating clients from locking that name. One way to reduce the likelihood of this is to use lock names that are database-specific or application-specific. For example, use lock names of the form db_name.str or app_name.str.
mysql> SELECT GET_LOCK('lock1',10);
-> 1
mysql> SELECT IS_FREE_LOCK('lock2');
-> 1
mysql> SELECT GET_LOCK('lock2',10);
-> 1
mysql> SELECT RELEASE_LOCK('lock2');
-> 1
mysql> SELECT RELEASE_LOCK('lock1');
-> NULL
The second RELEASE_LOCK() call returns NULL because the lock 'lock1' was automatically released by the second GET_LOCK() call.
If multiple clients are waiting for a lock, the order in which they will acquire it is undefined and depends on factors such as the thread library in use. In particular, applications should not assume that clients will acquire the lock in the same order that they issued the lock requests.
Note
Before MySQL 5.5.3, if a client attempts to acquire a lock that is already held by another client, it blocks according to the timeout argument. If the blocked client terminates, its thread does not die until the lock request times out.
This function is unsafe for statement-based replication. Beginning with MySQL 5.5.1, a warning is logged if you use this function when binlog_format is set to STATEMENT. (Bug #47995)
Starting a transaction with START TRANSACTION does not actually start it. The first table access following START TRANSACTION does. Opening a transaction isn't also a mean for concurrency control. If you need just that, you can rely on the advisory locks system MySQL provides through GET_LOCK(), RELEASE_LOCK(), and a few other related functions.
An alternative way to implement concurrency control, through transactions this time, would be by relying on exclusive row locks. Since SELECT statements are non-locking in InnoDB, issuing such query starts a transaction, however it neither sets any locks nor respects any pre-existing ones. To have a SELECT statement actually block if there is an earlier transaction operating on the same information (row), you have to use FOR UPDATE clause. For example:
START TRANSACTION;
SELECT lastid into lastgenerated FROM DB_last_id WHERE key = 'NAME_ID' FOR UPDATE;
...
With this construction there will never be two concurrent transactions operating on the same 'NAME_ID' past the SELECT statement that was explicitly told to perform a locking read.
Is there a way to execute low priority updates in MySQL, when using InnoDB?
I am running a very high load application where there may easily be literally thousands of users trying to concurrently update the same data records. This is mostly session-based statistical information, much of which could be ignored in case there is a wait time associated with the request. I'd like to be able to check whether some table/row is locked, and if so just not pass an update query to the server. Is that possible?
did you try setting low_priority_updates=1 in your my.cnf file? This should give select queries priority when an update or insert would otherwise lock the table.
If you say so that there is a time limit with the request you could use a Stored Procedure to skip certain updates
Something like this:
DELIMITER $$
DROP PROCEDURE IF EXISTS `updateStats` $$
CREATE PROCEDURE updateStats ()
BEGIN
DECLARE _B smallint(1) DEFAULT 0;
DECLARE _SECONDS INT DEFAULT 1;
-- http://dev.mysql.com/doc/refman/5.0/en/lock-tables-and-transactions.html
SELECT GET_LOCK('myLabel1',_SECONDS) INTO _B;
IF _B = 1 THEN
UPDATE table SET ........;
SLEEP(_SECONDS);
SELECT RELEASE_LOCK('myLabel1') INTO _B;
END IF;
END
This will make sure that if you got the Lock, that lasts for _SECONDS you make sure no other procedure runs the same code in that time frame.
The sleep is needed to keep the lock for 1 second (as if the SP terminates sooner, the lock is released)
You can also add an else node to the if, so it the stored procedure cannot update, to do custom code, like add to queue.
Suppose you want to write into the live table only in interval of 1 second to not load it too much, probably you are having a lot of indexes on it. On the else node you could update a second table that acts as a queue, and the queue is emptied in the IF true node, when you also make the update.
So your user application doesn't wait for the update to complete, and doesn't care if it doesn't complete, might this be a suitable context for using a background processing manager such as Gearman?