Is there a way to execute low priority updates in MySQL, when using InnoDB?
I am running a very high load application where there may easily be literally thousands of users trying to concurrently update the same data records. This is mostly session-based statistical information, much of which could be ignored in case there is a wait time associated with the request. I'd like to be able to check whether some table/row is locked, and if so just not pass an update query to the server. Is that possible?
did you try setting low_priority_updates=1 in your my.cnf file? This should give select queries priority when an update or insert would otherwise lock the table.
If you say so that there is a time limit with the request you could use a Stored Procedure to skip certain updates
Something like this:
DELIMITER $$
DROP PROCEDURE IF EXISTS `updateStats` $$
CREATE PROCEDURE updateStats ()
BEGIN
DECLARE _B smallint(1) DEFAULT 0;
DECLARE _SECONDS INT DEFAULT 1;
-- http://dev.mysql.com/doc/refman/5.0/en/lock-tables-and-transactions.html
SELECT GET_LOCK('myLabel1',_SECONDS) INTO _B;
IF _B = 1 THEN
UPDATE table SET ........;
SLEEP(_SECONDS);
SELECT RELEASE_LOCK('myLabel1') INTO _B;
END IF;
END
This will make sure that if you got the Lock, that lasts for _SECONDS you make sure no other procedure runs the same code in that time frame.
The sleep is needed to keep the lock for 1 second (as if the SP terminates sooner, the lock is released)
You can also add an else node to the if, so it the stored procedure cannot update, to do custom code, like add to queue.
Suppose you want to write into the live table only in interval of 1 second to not load it too much, probably you are having a lot of indexes on it. On the else node you could update a second table that acts as a queue, and the queue is emptied in the IF true node, when you also make the update.
So your user application doesn't wait for the update to complete, and doesn't care if it doesn't complete, might this be a suitable context for using a background processing manager such as Gearman?
Related
I'm using an Aurora DB (ie MySQL version 5.6.10) as a queue, and I'm using a stored procedure to pull records out of a table in batches. The sproc works with the following steps...
Select the next batch of data into a temptable
Write the IDs from the records from the temp table into to a log table
Output the records
Once a record has been added to the log, the sproc won't select it again next time it's called, so multiple servers can call this sproc, and both deal with batches of data from the queue without stepping on each others toes.
The sproc runs in a fraction of a second, but my company is now spinning up servers automatically, and these cloned servers are calling the sproc at exactly the same time, and the result is the same records are being selected twice
Is there a way I can make this sproc be limited to one call at a time? Ideally, any additional calls should wait until the first call is finished, and then they can run
Unfortunately, I have very little experience working with MySQL, so I'm not really sure where to start. I'd much appreciate it if anyone could point me in the right direction
This is a job for MySQL table locking. Try something like this. (You didn't show us your queries so there's a lot of guesswork here.)
SET autocommit = 0;
LOCK TABLES logtable WRITE;
CREATE TEMPORARY TABLE temptable AS
SELECT whatever FROM whatevertable FOR UPDATE;
INSERT INTO logtable (id)
SELECT id FROM temptable;
COMMIT;
UNLOCK TABLES;
If more than one connection tries to run this sequence concurrently, one will wait for the other's UNLOCK TABLES; to proceed. You say your SP is quick, so probably nobody will notice the short wait.
Pro tip: When you have the same timed code running on lots of servers, it's best to put in a short random delay before running the job. That way the shared resources (like your MySQL database) won't get hammered by a whole lot of requests precisely timed to be simultaneous.
I am using a stored procedure to update a table with doing some calculations. What I am worrying about is if multiple users access this stored procedure concurrently, my calculations will be wrong. I don't know this is possible as I don't have much knowledge about MySQL. What I want to achieve is allow access to stored procedure only one user at a time. Is this situation handled by default or do I have to do some extra works.
There's no way to control concurrent access to a stored procedure in MySQL. You'll need to control concurrent access to the appropriate row of the table instead. This isn't hard.
Here's what you do:
First, use InnoDB or some other transaction-capable access method. MyISAM won't work.
Second, identify some particular row in some table around which the operation revolves. For example, you might be updating a row with id number 17 in a table called 'calc' as part of your computation. You'll SELECT this row FOR UPDATE, and that will prevent other
Third, write your stored procedure code like this:
/* make sure you'll clean up correctly if something goes wrong */
DECLARE EXIT HANDLER FOR SQLEXCEPTION ROLLBACK;
/* begin a transaction */
START TRANSACTION;
...
/* lock the row you want to update, or to use to control concurrency */
SELECT id FROM calc WHERE id=17 FOR UPDATE;
...
/* do your calculations and updates */
...
/* when you're done, commit the transaction */
COMMIT;
The cool thing about this is that each concurrent access to the particular row will delay the completion of the SELECT ... FOR UPDATE until the preceding one finishes its COMMIT. If you have concurrent clients updating different rows they will run concurrently.
I have a stored procedure in mysql thats to perform a task that needs to be synchronized such that if two application calls the stored procedure, only one can access a section of code to perform the task, keeping the other one to get blocked until the first one finishes the task.
DELIMITER $$
CREATE PROCEDURE SP_GEN_ID(IN NAME VARCHAR(20))
BEGIN
DECLARE maxLen int default 0;
START TRANSACTION;
#the section of code that needs to be synchronized
COMMIT
END;
$$
DELIMITER ;
So, if two applications call the stored procedure simultaneously, the task has to be synchronized.
a. But Start TRANSACTION and COMMIT did NOT synchronize the execution.
b. And LOCK TABLES tableA can not be used in stored procedure to ensure the synchronization too.
c. I tried to synchronize the stored procedure call in application level. I used
boost_interprocess scoped_lock lock();
It worked perfectly fine in boost 1.41
But interprocess locking mutex is not supported in the boost 1.34 library, which is what is available in my case.
Is there a way to synchronize the stored procedure section of code such that when two calls are made simultaneously, one gets blocked before the other gets executed?
(added the following)
edited code: to give an idea what I am trying to perform in the synchronized block of the stored procedure.
It gets the last assigned id, and increment it by one and check whether it is not used for someother 'name' record.
When a valid id is found, update the last assigned id record table and then associate that with the 'name' given.
DELIMITER $$
CREATE PROCEDURE SP_GEN_ID(IN NAME VARCHAR(20))
BEGIN
DECLARE maxLen int default 0;
START TRANSACTION;
#the section of code that needs to be synchronized
SELECT lastid into lastgenerated FROM DB_last_id WHERE key = 'NAME_ID';
findid_loop:
LOOP
set lastid = lastid + 1;
#this is to check whether it was assigned with someother name before.
IF not EXISTS (SELECT 1 FROM user_name_id WHERE name_id = lastgenerated) then
set nameid = lastgenerated;
set found = true;
LEAVE findid_loop;
END IF;
#for loop limit check
IF (counter < loopLimit) then
set counter = counter + 1;
ITERATE findid_loop;
ELSE
#reached the limit and not found.
LEAVE findid_loop;
END IF;
END LOOP findid_loop;
#if a valid id, update last id and assign to name.
IF (found) THEN
#update the id.
update DB_last_id set lastid = nameid where key = 'NAME_ID';
insert into user_name_id values (nameid ,name);
ELSE
#return an empty string for the application to take action on the failure.
set nameid = '';
END IF;
#end transaction here.
COMMIT
END;
$$
DELIMITER ;
As mentioned in my comments above, you should find that a transaction is sufficient for most needs; however, if you need to explicitly wait until the other call has completed, use GET_LOCK(str,timeout):
Tries to obtain a lock with a name given by the string str, using a timeout of timeout seconds. Returns 1 if the lock was obtained successfully, 0 if the attempt timed out (for example, because another client has previously locked the name), or NULL if an error occurred (such as running out of memory or the thread was killed with mysqladmin kill). If you have a lock obtained with GET_LOCK(), it is released when you execute RELEASE_LOCK(), execute a new GET_LOCK(), or your connection terminates (either normally or abnormally). Locks obtained with GET_LOCK() do not interact with transactions. That is, committing a transaction does not release any such locks obtained during the transaction.
This function can be used to implement application locks or to simulate record locks. Names are locked on a server-wide basis. If a name has been locked by one client, GET_LOCK() blocks any request by another client for a lock with the same name. This enables clients that agree on a given lock name to use the name to perform cooperative advisory locking. But be aware that it also enables a client that is not among the set of cooperating clients to lock a name, either inadvertently or deliberately, and thus prevent any of the cooperating clients from locking that name. One way to reduce the likelihood of this is to use lock names that are database-specific or application-specific. For example, use lock names of the form db_name.str or app_name.str.
mysql> SELECT GET_LOCK('lock1',10);
-> 1
mysql> SELECT IS_FREE_LOCK('lock2');
-> 1
mysql> SELECT GET_LOCK('lock2',10);
-> 1
mysql> SELECT RELEASE_LOCK('lock2');
-> 1
mysql> SELECT RELEASE_LOCK('lock1');
-> NULL
The second RELEASE_LOCK() call returns NULL because the lock 'lock1' was automatically released by the second GET_LOCK() call.
If multiple clients are waiting for a lock, the order in which they will acquire it is undefined and depends on factors such as the thread library in use. In particular, applications should not assume that clients will acquire the lock in the same order that they issued the lock requests.
Note
Before MySQL 5.5.3, if a client attempts to acquire a lock that is already held by another client, it blocks according to the timeout argument. If the blocked client terminates, its thread does not die until the lock request times out.
This function is unsafe for statement-based replication. Beginning with MySQL 5.5.1, a warning is logged if you use this function when binlog_format is set to STATEMENT. (Bug #47995)
Starting a transaction with START TRANSACTION does not actually start it. The first table access following START TRANSACTION does. Opening a transaction isn't also a mean for concurrency control. If you need just that, you can rely on the advisory locks system MySQL provides through GET_LOCK(), RELEASE_LOCK(), and a few other related functions.
An alternative way to implement concurrency control, through transactions this time, would be by relying on exclusive row locks. Since SELECT statements are non-locking in InnoDB, issuing such query starts a transaction, however it neither sets any locks nor respects any pre-existing ones. To have a SELECT statement actually block if there is an earlier transaction operating on the same information (row), you have to use FOR UPDATE clause. For example:
START TRANSACTION;
SELECT lastid into lastgenerated FROM DB_last_id WHERE key = 'NAME_ID' FOR UPDATE;
...
With this construction there will never be two concurrent transactions operating on the same 'NAME_ID' past the SELECT statement that was explicitly told to perform a locking read.
I have a stored procedure in mysql thats to perform a task that needs to be synchronized such that if two application calls the stored procedure, only one can access a section of code to perform the task, keeping the other one to get blocked until the first one finishes the task.
DELIMITER $$
CREATE PROCEDURE SP_GEN_ID(IN NAME VARCHAR(20))
BEGIN
DECLARE maxLen int default 0;
START TRANSACTION;
#the section of code that needs to be synchronized
COMMIT
END;
$$
DELIMITER ;
So, if two applications call the stored procedure simultaneously, the task has to be synchronized.
a. But Start TRANSACTION and COMMIT did NOT synchronize the execution.
b. And LOCK TABLES tableA can not be used in stored procedure to ensure the synchronization too.
c. I tried to synchronize the stored procedure call in application level. I used
boost_interprocess scoped_lock lock();
It worked perfectly fine in boost 1.41
But interprocess locking mutex is not supported in the boost 1.34 library, which is what is available in my case.
Is there a way to synchronize the stored procedure section of code such that when two calls are made simultaneously, one gets blocked before the other gets executed?
(added the following)
edited code: to give an idea what I am trying to perform in the synchronized block of the stored procedure.
It gets the last assigned id, and increment it by one and check whether it is not used for someother 'name' record.
When a valid id is found, update the last assigned id record table and then associate that with the 'name' given.
DELIMITER $$
CREATE PROCEDURE SP_GEN_ID(IN NAME VARCHAR(20))
BEGIN
DECLARE maxLen int default 0;
START TRANSACTION;
#the section of code that needs to be synchronized
SELECT lastid into lastgenerated FROM DB_last_id WHERE key = 'NAME_ID';
findid_loop:
LOOP
set lastid = lastid + 1;
#this is to check whether it was assigned with someother name before.
IF not EXISTS (SELECT 1 FROM user_name_id WHERE name_id = lastgenerated) then
set nameid = lastgenerated;
set found = true;
LEAVE findid_loop;
END IF;
#for loop limit check
IF (counter < loopLimit) then
set counter = counter + 1;
ITERATE findid_loop;
ELSE
#reached the limit and not found.
LEAVE findid_loop;
END IF;
END LOOP findid_loop;
#if a valid id, update last id and assign to name.
IF (found) THEN
#update the id.
update DB_last_id set lastid = nameid where key = 'NAME_ID';
insert into user_name_id values (nameid ,name);
ELSE
#return an empty string for the application to take action on the failure.
set nameid = '';
END IF;
#end transaction here.
COMMIT
END;
$$
DELIMITER ;
As mentioned in my comments above, you should find that a transaction is sufficient for most needs; however, if you need to explicitly wait until the other call has completed, use GET_LOCK(str,timeout):
Tries to obtain a lock with a name given by the string str, using a timeout of timeout seconds. Returns 1 if the lock was obtained successfully, 0 if the attempt timed out (for example, because another client has previously locked the name), or NULL if an error occurred (such as running out of memory or the thread was killed with mysqladmin kill). If you have a lock obtained with GET_LOCK(), it is released when you execute RELEASE_LOCK(), execute a new GET_LOCK(), or your connection terminates (either normally or abnormally). Locks obtained with GET_LOCK() do not interact with transactions. That is, committing a transaction does not release any such locks obtained during the transaction.
This function can be used to implement application locks or to simulate record locks. Names are locked on a server-wide basis. If a name has been locked by one client, GET_LOCK() blocks any request by another client for a lock with the same name. This enables clients that agree on a given lock name to use the name to perform cooperative advisory locking. But be aware that it also enables a client that is not among the set of cooperating clients to lock a name, either inadvertently or deliberately, and thus prevent any of the cooperating clients from locking that name. One way to reduce the likelihood of this is to use lock names that are database-specific or application-specific. For example, use lock names of the form db_name.str or app_name.str.
mysql> SELECT GET_LOCK('lock1',10);
-> 1
mysql> SELECT IS_FREE_LOCK('lock2');
-> 1
mysql> SELECT GET_LOCK('lock2',10);
-> 1
mysql> SELECT RELEASE_LOCK('lock2');
-> 1
mysql> SELECT RELEASE_LOCK('lock1');
-> NULL
The second RELEASE_LOCK() call returns NULL because the lock 'lock1' was automatically released by the second GET_LOCK() call.
If multiple clients are waiting for a lock, the order in which they will acquire it is undefined and depends on factors such as the thread library in use. In particular, applications should not assume that clients will acquire the lock in the same order that they issued the lock requests.
Note
Before MySQL 5.5.3, if a client attempts to acquire a lock that is already held by another client, it blocks according to the timeout argument. If the blocked client terminates, its thread does not die until the lock request times out.
This function is unsafe for statement-based replication. Beginning with MySQL 5.5.1, a warning is logged if you use this function when binlog_format is set to STATEMENT. (Bug #47995)
Starting a transaction with START TRANSACTION does not actually start it. The first table access following START TRANSACTION does. Opening a transaction isn't also a mean for concurrency control. If you need just that, you can rely on the advisory locks system MySQL provides through GET_LOCK(), RELEASE_LOCK(), and a few other related functions.
An alternative way to implement concurrency control, through transactions this time, would be by relying on exclusive row locks. Since SELECT statements are non-locking in InnoDB, issuing such query starts a transaction, however it neither sets any locks nor respects any pre-existing ones. To have a SELECT statement actually block if there is an earlier transaction operating on the same information (row), you have to use FOR UPDATE clause. For example:
START TRANSACTION;
SELECT lastid into lastgenerated FROM DB_last_id WHERE key = 'NAME_ID' FOR UPDATE;
...
With this construction there will never be two concurrent transactions operating on the same 'NAME_ID' past the SELECT statement that was explicitly told to perform a locking read.
I want to write a procedure that will handle the insert of data into 2 tables. If the insert should fail in either one then the whole procedure should fail. I've tried this many different ways and cannot get it to work. I've purposefully made my second insert fail but the data is inserted into the first table anyway.
I've tried to nest IF statements based on the rowcount but even though the data fails on the second insert, the data is still being inserted into the first table. I'm looking for a total number of 2 affected rows.
Can someone please show me how to handle multiple inserts and rollback if one of them fails? A short example would be nice.
If you are using InnoDB tables (or other compatible engine) you can use the Transaction feature of MySQL that allows you to do exactly what you want.
Basically you start the transaction
do the queries checking for the result
If every result is OK you call the CONMIT
else you call the ROLLBACK to void all the queries within the transaction.
You can read and article about with examples here.
HTH!
You could try turning autocommit off. It might be automatically committing your first insert even though you haven't explicitly committed the transaction that's been started:
SET autocommit = 0;
START TRANSACTION
......