We have a large MyISAM table to which rows get inserted to the bottom of the table only.
While doing some benchmarks, i realized that selects do not (always) lock other inserts to that same table. However, when the inserts are coming from a stored procedure/function they will by locked by the select.
Why is that?
To demonstrate this behavior:
CREATE TABLE Foo (
ID INT NOT NULL AUTO_INCREMENT,
Bar VARCHAR(200),
PRIMARY KEY(ID)) ENGINE=MyISAM;
--INSERT into Foo 10M rows
DELIMITER $$
DROP PROCEDURE IF EXISTS InsertProc$$
CREATE PROCEDURE InsertProc(IN vBar VARCHAR(255))
BEGIN
INSERT Foo(Bar) VALUES (vBar);
END$$
DELIMITER ;
Run the following query:
SELECT Count(*) FROM Foo WHERE INSTR(Bar, 'abcdefg') > 0;
While that Select is running, open a new connection and run the following insert query:
INSERT Foo(Bar) VALUES ('xyz1234');
That Insert will run and return right away, However if i run the following query:
CALL InsertProc('xyz1234');
Now the query locks and waits for the select to complete.
MySql Version: 5.0.51 running on Window Server 2K3
Thank you.
-- UPDATE
Here is the profile output:
Insert Direct:
(initialization) 0.0000432
checking permissions 0.0000074
Opening tables 0.0000077
System lock 0.0000032
Table lock 0.0000025
init 0.000021
update 0.0002365
end 0.0000382
query end 0.000002
freeing items 0.0000057
closing tables 0.0000022
logging slow query 0.0000005
Insert via Procedure:
(initialization) 0.0000285
Opening tables 0.0004325
System lock 0.0000022
Table lock 0.0002957
checking permissions 0.0000047
Opening tables 0.000004
System lock 0.0000017
Table lock 3.2365122
init 0.0000422
update 0.000251
end 0.0000025
query end 0.000003
closing tables 0.00004
query end 0.0000074
freeing items 0.0000074
logging slow query 0.000001
cleaning up 0.5790915
Why does the procedure open and "Table lock" twice?
This issue was submitted as a bug:
http://bugs.mysql.com/bug.php?id=58689
MyIASM for any particular reason? InnoDB tables usually have much better locking characteristics.
Speculation: perhaps the locking/mutex handling on the AUTO_INCREMENT field on MyISAM tables is stricter when stored procedures are used.
To rule it out, could you set up a test where ID wasn't an AUTO_INCREMENT field?
Have you given INSERT DELAYED a try if you application might allow for it?
Related
This is my procedure
use my_database;
drop procedure if exists sel_upd_ports;
delimiter $$
create procedure sel_upd_ports()
begin
drop temporary table if exists oneone;
start transaction;
create temporary table oneone as (select id,tt from table where status=0 limit 10 for update);
update table a join oneone b on a.id=b.id set status=1;
commit;
select * from oneone;
drop temporary table oneone;
end$$
delimiter ;
this is what I need:
it has to be a procedure
update+select queries should be working in transaction, so my application can be concurrent
procedure should return a result set
this procedure works, but I can't get rid of an error
ERROR 1746 (HY000): Can't update table 'table' while 'oneone' is being created.
and it goes away as soon as i delete a lock "for update".
See documentation:
13.2.9 SELECT Syntax
...
In addition, you cannot use FOR UPDATE as part of the SELECT in a
statement such as CREATE TABLE new_table SELECT ... FROM old_table .... (If you attempt to do so, the statement is rejected with the
error Can't update table 'old_table' while 'new_table' is being
created.) This is a change in behavior from MySQL 5.5 and earlier,
which permitted CREATE TABLE ... SELECT statements to make changes in
tables other than the table being created.
...
See MySQL 5.6 Release Notes:
Changes in MySQL 5.6.2 (2011-04-11, Developer Milestone) :: Bugs
Fixed
Incompatible Change; Replication: It is no longer possible to issue a CREATE TABLE ... SELECT statement which changes any tables other than the table being created. Any such statement is not executed and
instead fails with an error.
One consequence of this change is that FOR UPDATE may no longer be
used at all with the SELECT portion of a CREATE TABLE ... SELECT.
This means that, prior to upgrading from a previous release, you
should rewrite any CREATE TABLE ... SELECT statements that cause
changes in other tables so that the statements no longer do so.
This change also has implications for statement-based replication
between a MySQL 5.6 (or later slave) and a master running a previous
version of MySQL. In such a case, if a CREATE TABLE ... SELECT
statement on the master that causes changes in other tables succeeds
on the master, the statement nonetheless fails on the slave, causing
replication to stop. To keep this from happening, you should either
use row-based replication, or rewrite the offending statement before
running it on the master. (Bug #11749792, Bug #11745361, Bug #39804,
Bug #55876)
References: See also: Bug #47899.
I'm experiencing some very strange transactional behaviour in my MYSQL application.
I've managed to reduce the problem down to a small isolated test case, the code for which I’ve included below:
-- Setup a new environment
SET GLOBAL TRANSACTION ISOLATION LEVEL READ COMMITTED;
DROP DATABASE IF EXISTS `testDB`;
CREATE DATABASE `testDB`;
USE `testDB`;
-- Create a table I want two procedure calls to interact with
CREATE TABLE `tbl_test` (
`id` INT(10) UNSIGNED NOT NULL
, PRIMARY KEY (`id`)
);
-- A second table purely to demonstrate the issue
CREATE TABLE `tbl_test2` (
`id` INT(10) UNSIGNED NOT NULL
);
DELIMITER $$
DROP PROCEDURE IF EXISTS `sp_test` $$
CREATE PROCEDURE `sp_test` ()
BEGIN
START TRANSACTION;
-- CRAZY LINE
SELECT * FROM `tbl_test2`;
-- Insert ignore so both calls don’t try to insert the same row
INSERT IGNORE INTO `tbl_test` (`id`) VALUES (1);
-- Sleep added to make it possible to run concurrently manually
SELECT SLEEP(1) INTO #rubbish;
-- The result I am interested in
SELECT COUNT(*) FROM `tbl_test`;
COMMIT;
END $$
DELIMITER ;
Steps to Reproduce:
Run in the above script to create a test database, two tables and a stored procedure.
In two separate connections, as near to simultaneously as possible, run the stored procedures (you can increase the SLEEP time if you need longer):
USE `testDB`;
CALL sp_test ();
The Problem
When executed concurrently over two separate connections the SELECT COUNT(*) FROM `tbl_test`; statement returns different values for the two calls.
When I follow the steps above, I get back 1 from the first of the two procedure calls and 0 from the second.
My understanding of transactional behaviour and table locking is that when the first call reaches the INSERT statement it will create a lock. The second procedure call will reach the same line but must then wait until the transaction from the first call has been committed. Increasing the sleep time reinforces this idea as the second call will take twice as long to complete. If this is the case however, then the second procedure call should pick up the insert from the first call and both results should be equal to 1.
TL;DR
I'm expecting both to equal 1
Note that I am using READ_COMMITTED as my transaction isolation level.
I've tested this using MYSQL server and MariaDB
Further Weirdness
So at this point I assumed my understanding was incorrect. However, I then noticed that by removing the line SELECT * FROM `tbl_test2`; the results suddenly produced the expected values!
I've been experimenting with the script but essentially, including a SELECT statement to any table within the database before the INSERT line causes unanticipated results. I have absolutely no idea why this is the case.
Questions
Is my understanding of the expected transactional behaviour correct?
Why on earth does the SELECT statement to an unrelated table cause the transactional locking to fail?
If anyone can shed some light on this I would be very grateful!
MySQL provides an automatic mechanism to increment record IDs. This is OK for many purposes, but I need to be able to use sequences as offered by ORACLE. Obviously, there is no point in creating a table for that purpose.
The solution SHOULD be simple:
1) Create a table to hosts all the needed sequences,
2) Create a function that increases the value of a specific sequence and returns the new value,
3) Create a function that returns the current value of a sequence.
In theory, it looks simple... BUT...
When increasing the value of a sequence (much the same as nextval in Oracle), you need to prevent other sessions to perform this operation (or even fetch the current value) till the updated is completed.
Two theoretical options:
a - Use an UPDATE statement that would return the new value in a single shot, or
b - Lock the table between the UPDATE and SELECT.
Unfortunately, it would appear that MySQL does not allow to lock tables within functions / procedures, and while trying to make the whole thing in a single statement (like UPDATE... RETURNING...) you must use #-type variables which survive the completion of the function/procedure.
Does anyone have an idea/working solution for this?
Thanks.
The following is a simple example with a FOR UPDATE intention lock. A row-level lock with the INNODB engine. The sample shows four rows for next available sequences that will not suffer from the well-known INNODB Gap Anomaly (the case where gaps occur after failed usage of an AUTO_INCREMENT).
Schema:
-- drop table if exists sequences;
create table sequences
( id int auto_increment primary key,
sectionType varchar(200) not null,
nextSequence int not null,
unique key(sectionType)
) ENGINE=InnoDB;
-- truncate table sequences;
insert sequences (sectionType,nextSequence) values
('Chassis',1),('Engine Block',1),('Brakes',1),('Carburetor',1);
Sample code:
START TRANSACTION; -- Line1
SELECT nextSequence into #mine_to_use from sequences where sectionType='Carburetor' FOR UPDATE; -- Line2
select #mine_to_use; -- Line3
UPDATE sequences set nextSequence=nextSequence+1 where sectionType='Carburetor'; -- Line4
COMMIT; -- Line5
Ideally you do not have a Line3 or bloaty code at all which would delay other clients on a Lock Wait. Meaning, get your next sequence to use, perform the update (the incrementing part), and COMMIT, ASAP.
The above in a stored procedure:
DROP PROCEDURE if exists getNextSequence;
DELIMITER $$
CREATE PROCEDURE getNextSequence(p_sectionType varchar(200),OUT p_YoursToUse int)
BEGIN
-- for flexibility, return the sequence number as both an OUT parameter and a single row resultset
START TRANSACTION;
SELECT nextSequence into #mine_to_use from sequences where sectionType=p_sectionType FOR UPDATE;
UPDATE sequences set nextSequence=nextSequence+1 where sectionType=p_sectionType;
COMMIT; -- get it and release INTENTION LOCK ASAP
set p_YoursToUse=#mine_to_use; -- set the OUT parameter
select #mine_to_use as yourSeqNum; -- also return as a 1 column, 1 row resultset
END$$
DELIMITER ;
Test:
set #myNum:= -1;
call getNextSequence('Carburetor',#myNum);
+------------+
| yourSeqNum |
+------------+
| 4 |
+------------+
select #myNum; -- 4
Modify the stored procedure accordingly for you needs, such as having only 1 of the 2 mechanisms for retrieving the sequence number (either the OUT parameter or the result set). In other words, it is easy to ditch the OUT parameter concept.
If you do not adhere to ASAP release of the LOCK (which obviously is not needed after the update), and proceed to perform time consuming code, prior to the release, then the following can occur after a timeout period for other clients awaiting a sequence number:
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting
transaction
Hopefully this is never an issue.
show variables where variable_name='innodb_lock_wait_timeout';
MySQL Manual Page for innodb_lock_wait_timeout.
On my system at the moment it has a value of 50 (seconds). A wait of more than a second or two is probably unbearable in most situations.
Also of interest during TRANSACTIONS is that section of the output from the following command:
SHOW ENGINE INNODB STATUS;
I need to write a MySQL stored procedure that will execute multiple queries/logic. However, there is a table that is used in this procedure which is also used by other stored procedure. I need to prevent the two procedures from reading/writing to the same table while one is in the middle of an execution.
I need somehow to lock the shared table so no other procedure can use it until locking procedure unlocks it.
Additionally, there are other tables involved in the procedure that I do not want to lock at all. The lock should be placed on some tables and not all of them. All of my tables are using InnoDB storage engine.
Here is what I tried
DELIMITER $$
CREATE PROCEDURE `sp_test` ()
BEGIN
SET autocommit = 0;
LOCK TABLES db.table WRITE, db.table READ;
START TRANSACTION;
END
... Query 1 SELECT * FROM table1...
... Query 2 UPDATE table1...
... Query 3 INSERT INTO table1...
... Query 4 INSERT INTO table1...
... Query 5 UPDATE table1...
... CALL db.pro1; -- this second procedure makes updated to table1
... Query 6 SELECT * FROM table1...
COMMIT;
UNLOCK TABLES;
But this is giving me the following error
ERROR 1314: LOCK is not allowed in stored procedures
How can I lock the shared table in the stored procedure to prevent 2 processes from using the it until the first running procedure finish executing?
I am wondering if it is necessary to use locking in most likely concurrent environment and how in following case. Using MySQL database server with InnoDB engine
Let's say I have a table
CREATE TABLE `A` (
`id` INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
`m_id` INT NOT NULL, -- manual id
`name` VARCHAR(10)
) ENGINE=INNODB;
And the procedure
CREATE PROCEDURE `add_record`(IN _NAME VARCHAR(10))
BEGIN
DECLARE _m_id INT;
DECLARE EXIT HANDLER FOR SQLEXCEPTION ROLLBACK;
START TRANSACTION;
SELECT (`m_id` + 1) INTO _m_id FROM `A` WHERE `id` = (SELECT MAX(`id`) FROM `A`);
INSERT INTO `A`(`m_id`, `name`) VALUES(_m_id, _NAME);
COMMIT;
END$$
Like you see the fact is that I am increasing m_id manually and concurrent transactions are most likely happening. I can't make my mind if database might become in inconsistent state. Also using FOR UPDATE and LOCK IN SHARE MODE has no point in this situation as transaction deals with new records and has nothing to do with updates on a specific row. Further LOCK TABLES are not allowed in stored procedures and is quite insufficient.
So, my question is how to avoid inconsistent state in marked scenario if it is possible to happen actually. Any advice will be grateful
transaction deals with new records and has nothing to do with updates on a specific row
Such a new record is known as a phantom:
phantom
A row that appears in the result set of a query, but not in the result set of an earlier query. For example, if a query is run twice within a transaction, and in the meantime, another transaction commits after inserting a new row or updating a row so that it matches the WHERE clause of the query.
This occurrence is known as a phantom read. It is harder to guard against than a non-repeatable read, because locking all the rows from the first query result set does not prevent the changes that cause the phantom to appear.
Among different isolation levels, phantom reads are prevented by the serializable read level, and allowed by the repeatable read, consistent read, and read uncommitted levels.
So to prevent phantoms from occurring on any statement, one can simply set the transaction isolation level to be SERIALIZABLE. InnoDB implements this using next-key locks, which not only locks the records that your queries match but also locks the gaps between those records.
The same can be accomplished on a per-statement basis by using locking reads, such as you describe in your question: LOCK IN SHARE MODE or FOR UPDATE (the former allows concurrent sessions to read the matching records while the lock is in place, whilst the latter does not).
First, a sequence table
CREATE TABLE m_id_sequence (
id integer primary key auto_increment
);
and then alter the procedure to get the next m_id from the sequence table
DELIMITER $$
CREATE PROCEDURE `add_record`(IN _NAME VARCHAR(10))
BEGIN
DECLARE _m_id INT;
DECLARE EXIT HANDLER FOR SQLEXCEPTION ROLLBACK;
START TRANSACTION;
INSERT INTO m_id_sequence VALUES ();
SET _m_id = LAST_INSERT_ID();
INSERT INTO `A`(`m_id`, `name`) VALUES(_m_id, _NAME);
COMMIT;
END$$
DELIMITER ;