I'm seeing duplicate rows in my history table - however the transaction that updates it first does delete from ; and then re-inserts the data for each one. The other queries are correct (and work most of the time just fine).
How is it that the select statement can see an inconsistent view of the data?
create my_history_table (
`a` int(11),
`b` int(11),
`timestamp` datetime
) ENGINE=InnoDB;
We update this table as needed with this:
set autocommit=0;
set transaction isolation level read committed;
start transaction;
delete from my_history_table;
<loop on cursor inserting results for each>
commit;
In another stored procedure we read these results back to the client:
select * from my_history_table join anotherTable using (a);
So the answer was that delete from ; was timing out on getting a lock but as of MySQL 5.0.31, that does not automatically roll back the transaction. So it was doing the inserts without doing the delete and commiting that mess.
Related
I have the following table:
CREATE TABLE `accounts` (
`name` varchar(50) NOT NULL,
`balance` int NOT NULL,
PRIMARY KEY (`name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
And it has two accounts in it. "Bob" has a balance of 100. "Jim" has a balance of 200.
I run this query to transfer 50 from Jim to Bob:
SET TRANSACTION ISOLATION LEVEL REPEATABLE READ;
BEGIN;
SELECT * FROM accounts;
SELECT SLEEP(10);
SET #bobBalance = (SELECT balance FROM accounts WHERE name = 'bob' FOR UPDATE);
SET #jimBalance = (SELECT balance FROM accounts WHERE name = 'jim' FOR UPDATE);
UPDATE accounts SET balance = #bobBalance + 50 WHERE name = 'bob';
UPDATE accounts SET balance = #jimBalance - 50 WHERE name = 'jim';
COMMIT;
While that query is sleeping, I run the following query in a different session to set Jim's balance to 500:
UPDATE accounts SET balance = 500 WHERE name = 'jim';
What I thought would happen is that this would cause a bug. The transaction would set Jim's balance to 150, because the first read in the transaction (before the SLEEP) would establish a snapshot in which Jim's balance is 200, and that snapshot would be used in the later query to get Jim's balance. So we would subtract 50 from 200 even though Jim's balance has actually been changed to 500 by the other query.
But that's not what happens. Actually, the end result is correct. Bob has 150 and Jim has 450. But I don't understand why this is.
The MySQL documentation says about Repeatable Read:
This is the default isolation level for InnoDB. Consistent reads within the same transaction read the snapshot established by the first read. This means that if you issue several plain (nonlocking) SELECT statements within the same transaction, these SELECT statements are consistent also with respect to each other. See Section 15.7.2.3, “Consistent Nonlocking Reads”.
So what am I missing here? Why does it seem like the SELECT statements in the transaction are not all using a snapshot established by the first SELECT statement?
The repeatable-read behavior only works for non-locking SELECT queries. It reads from the snapshot established by the first query in the transaction.
But any locking SELECT query reads the latest committed version of the row, as if you had started your transaction in READ-COMMITTED isolation level.
A SELECT is implicitly a locking read if it's involved in any kind of SQL statement that modifies data.
For example:
INSERT INTO table2 SELECT * FROM table1 WHERE ...;
The above locks examined rows in table1, even though the statement is just copying them to table2.
SET #myvar = (SELECT ... FROM table1 WHERE ...);
This is also copying a value from table1, into a variable. It locks the examined row in table1.
Likewise SELECT statements that are invoked in a trigger, or as part of a multi-table UPDATE or DELETE, and so on. Anytime the SELECT is part of a larger statement that modifies any data (in a table or in a variable), it locks the rows examined by the SELECT.
And therefore it's a locking read, and behaves like an UPDATE with respect to which row version it reads.
I have an complex database. I can simplify it like that:
The table a:
CREATE TABLE a
(
Id int(10) unsigned NOT NULL AUTO_INCREMENT,
A int(11) DEFAULT NULL,
CalcUniqId int(11) DEFAULT NULL,
PRIMARY KEY (Id)
) ENGINE=InnoDB;
CREATE TRIGGER a_before_ins_tr before INSERT on a
FOR EACH ROW
BEGIN
select Max(CalcUniqId) from A into #MaxCalcUniqId;
set new.CalcUniqId=IfNull(#MaxCalcUniqId,1)+1;
END $
It works like that:
start transaction
insert into A(A)
... insert in other tables. It take between 30 and 60 seconds
commit;
The problem is, the trigger returns the same CalcUniqId for all transaction that run at the same time.
Is there any solution or work arround.
Is this a solution:
start transaction;
SET SESSION TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;
insert into A(A) values(10);
SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ;
....
commit;
Man can run this test:
Session 1:
Step1: start transaction;
Step2: insert into A(A) values(1);
Step3: commit;
Session 2:
Step1: start transaction;
Step2: insert into A(A) values(2);
Step3: commit;
Run in session 1 the steps 1,2 and in session 2 the steps 1,2. than step 3 in both. After that do
select Id, A, CalcUniqId from a;
both have the same CalcUniqId=2.
Change the SELECT in the Trigger to this:
select Max(CalcUniqId) from A into #MaxCalcUniqId
FOR UPDATE; -- add this
That tells the transaction that you intend to change the value; that blocks the other transactions from changing it.
This will probably lead to your 30-60 sec transactions being run one after another. And probably dying due to exceeding lock_wait_timeout. Rather than increasing that setting (which is already "too high"), please explain the bigger picture. Perhaps we can concoct a workaround that gets the 'correct' value and runs in parallel.
I am wondering if it is necessary to use locking in most likely concurrent environment and how in following case. Using MySQL database server with InnoDB engine
Let's say I have a table
CREATE TABLE `A` (
`id` INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
`m_id` INT NOT NULL, -- manual id
`name` VARCHAR(10)
) ENGINE=INNODB;
And the procedure
CREATE PROCEDURE `add_record`(IN _NAME VARCHAR(10))
BEGIN
DECLARE _m_id INT;
DECLARE EXIT HANDLER FOR SQLEXCEPTION ROLLBACK;
START TRANSACTION;
SELECT (`m_id` + 1) INTO _m_id FROM `A` WHERE `id` = (SELECT MAX(`id`) FROM `A`);
INSERT INTO `A`(`m_id`, `name`) VALUES(_m_id, _NAME);
COMMIT;
END$$
Like you see the fact is that I am increasing m_id manually and concurrent transactions are most likely happening. I can't make my mind if database might become in inconsistent state. Also using FOR UPDATE and LOCK IN SHARE MODE has no point in this situation as transaction deals with new records and has nothing to do with updates on a specific row. Further LOCK TABLES are not allowed in stored procedures and is quite insufficient.
So, my question is how to avoid inconsistent state in marked scenario if it is possible to happen actually. Any advice will be grateful
transaction deals with new records and has nothing to do with updates on a specific row
Such a new record is known as a phantom:
phantom
A row that appears in the result set of a query, but not in the result set of an earlier query. For example, if a query is run twice within a transaction, and in the meantime, another transaction commits after inserting a new row or updating a row so that it matches the WHERE clause of the query.
This occurrence is known as a phantom read. It is harder to guard against than a non-repeatable read, because locking all the rows from the first query result set does not prevent the changes that cause the phantom to appear.
Among different isolation levels, phantom reads are prevented by the serializable read level, and allowed by the repeatable read, consistent read, and read uncommitted levels.
So to prevent phantoms from occurring on any statement, one can simply set the transaction isolation level to be SERIALIZABLE. InnoDB implements this using next-key locks, which not only locks the records that your queries match but also locks the gaps between those records.
The same can be accomplished on a per-statement basis by using locking reads, such as you describe in your question: LOCK IN SHARE MODE or FOR UPDATE (the former allows concurrent sessions to read the matching records while the lock is in place, whilst the latter does not).
First, a sequence table
CREATE TABLE m_id_sequence (
id integer primary key auto_increment
);
and then alter the procedure to get the next m_id from the sequence table
DELIMITER $$
CREATE PROCEDURE `add_record`(IN _NAME VARCHAR(10))
BEGIN
DECLARE _m_id INT;
DECLARE EXIT HANDLER FOR SQLEXCEPTION ROLLBACK;
START TRANSACTION;
INSERT INTO m_id_sequence VALUES ();
SET _m_id = LAST_INSERT_ID();
INSERT INTO `A`(`m_id`, `name`) VALUES(_m_id, _NAME);
COMMIT;
END$$
DELIMITER ;
I have a stored procedure , its contents are as follows:
-- --------------------------------------------------------------------------------
-- Routine DDL
-- Note: comments before and after the routine body will not be stored by the server
-- --------------------------------------------------------------------------------
DELIMITER $$
CREATE DEFINER=`MailMe`#`%` PROCEDURE `sp_archivev3`()
BEGIN
INSERT INTO
send.sgev3_archive(a_bi,
b_vc,
c_int,
d_int,
e_vc,
f_vc,
g_vc,
h_vc,
i_dt,
j_vc,
k_vc,
l_vc,
m_dt,
n_vch,
o_bit)
SELECT a_bi,
b_vc,
c_int,
d_int,
e_vc,
f_vc,
g_vc,
h_vc,
i_dt,
j_vc,
k_vc,
l_vc,
m_dt,
n_vch,
o_bit
FROM send.sgev3
WHERE m_dt BETWEEN '2014-06-09' AND CURDATE();
END
Since, my query is inserting the records into send.sgev3_archive from send.sgev3. I want to do one more thing. I want to delete the records present in the send.sgev3 table after selecting and inserting the same into send.sgev3_archive. Should I write the DELETE query right below the SELECT query in my code above? Just wanted to confirm as I don't want to mess up my real data and accidently delete any records without getting it copied. Please advise.
Yes exactly. Include a DELETE statement saying
DELETE FROM send.sgev3
WHERE m_dt BETWEEN '2014-06-09' AND CURDATE();
To be more sure that INSERT does completes before DELETE invokes; wrap the INSERT and DELETE in a Transaction Block saying
START TRANSACTION;
INSERT INTO send.sgev3_archive ...
SELECT ... FROM send.sgev3
COMMIT;
You can as well handle error condition in your procedure and ROLLBACK the entire transaction by using exit handler in stored procedure. Below post already shows an way to do the same. Take a look.
How can I use transactions in my MySQL stored procedure?
MySQL Rollback in transaction
EDIT:
why transaction is necessary? Can't I just proceed like the way I have
mentioned in my question?
Instead of explaining you why; let's show you an example (Quite resemble your scenario)
Let's say you have a table named parent declared as
create table parent(id int not null auto_increment primary key,
`name` varchar(10),city varchar(10));
Insert some records to it
insert into parent(`name`,city) values('sfsdfd','sdfsdfdf'),('sfsdfd','sdfsdfdf'),('sfsdfd',null)
Now, you have another table named child defined as below (Notice the last column has not null constraint)
create table child(id int not null auto_increment primary key,
`name` varchar(10),city varchar(10) not null)
Now execute both the below statement (what you are currently doing)
insert into child(`name`,city) select * from parent;
delete from parent;
Result: INSERT will fail due to the not null constraint in child table but delete will succeed.
To avoid this exact scenario you need Transaction in place. so that, if INSERT fails you don't go for delete at all.
A pseudo code on how you handle this in transaction
start transaction
insert into child(`name`,city) select * from parent;
if(ERROR)
rollback
exit from stored proc
else
commit
delete from parent;
SideNote: exit from stored proc can be implemented using LEAVE
I have a PHP/5.2 driven application that uses transactions under MySQL/5.1 so it can rollback multiple inserts if an error condition is met. I have different reusable functions to insert different type of items. So far so good.
Now I need to use table locking for some of the inserts. As the official manual suggests, I'm using SET autocommit=0 instead of START TRANSACTION so LOCK TABLES does not issue an implicit commit. And, as documented, unlocking tables implicitly commits any active transaction:
http://dev.mysql.com/doc/refman/5.1/en/lock-tables-and-transactions.html
And here lies the problem: if I simply avoid UNLOCK TABLES, it happens that the second call to LOCK TABLES commits pending changes!
It appears that the only way is to perform all necessary LOCK TABLES in a single statement. That's a mainteinance nightmare.
Does this issue have a sensible workaround?
Here's a little test script:
DROP TABLE IF EXISTS test;
CREATE TABLE test (
test_id INT(10) UNSIGNED NOT NULL AUTO_INCREMENT,
random_number INT(10) UNSIGNED NOT NULL,
PRIMARY KEY (test_id)
)
COLLATE='utf8_spanish_ci'
ENGINE=InnoDB;
-- No table locking: everything's fine
START TRANSACTION;
INSERT INTO test (random_number) VALUES (ROUND(10000* RAND()));
SELECT * FROM TEST ORDER BY test_id;
ROLLBACK;
SELECT * FROM TEST ORDER BY test_id;
-- Table locking: everything's fine if I avoid START TRANSACTION
SET autocommit=0;
INSERT INTO test (random_number) VALUES (ROUND(10000* RAND()));
SELECT * FROM TEST ORDER BY test_id;
ROLLBACK;
SELECT * FROM TEST ORDER BY test_id;
SET autocommit=1;
-- Table locking: I cannot nest LOCK/UNLOCK blocks
SET autocommit=0;
LOCK TABLES test WRITE;
INSERT INTO test (random_number) VALUES (ROUND(10000* RAND()));
SELECT * FROM TEST ORDER BY test_id;
ROLLBACK;
UNLOCK TABLES; -- Implicit commit
SELECT * FROM TEST ORDER BY test_id;
SET autocommit=1;
-- Table locking: I cannot chain LOCK calls ether
SET autocommit=0;
LOCK TABLES test WRITE;
INSERT INTO test (random_number) VALUES (ROUND(10000* RAND()));
SELECT * FROM TEST ORDER BY test_id;
-- UNLOCK TABLES;
LOCK TABLES test WRITE; -- Implicit commit
INSERT INTO test (random_number) VALUES (ROUND(10000* RAND()));
SELECT * FROM TEST ORDER BY test_id;
-- UNLOCK TABLES;
ROLLBACK;
SELECT * FROM TEST ORDER BY test_id;
SET autocommit=1;
Apparently, LOCK TABLES cannot be fixed to play well with transactions. A workaround is to replace it with SELECT .... FOR UPDATE. You don't need any special syntax (you can use regular START TRANSACTION) and it works as expected:
START TRANSACTION;
SELECT COUNT(*) FROM foo FOR UPDATE; -- Lock issued
INSERT INTO foo (foo_name) VALUES ('John');
SELECT COUNT(*) FROM bar FOR UPDATE; -- Lock issued, no side effects
ROLLBACK; -- Rollback works as expected
Please note that COUNT(*) is just an example, you can normally use the SELECT statement to fetch data you actually need ;-)
(This information was provided by Frank Heikens.)