In an attempt to overcome deadlocks on the combat server, I reached a dead end. There are 2 tables:
First:
create table table_1
(
id int auto_increment
primary key,
data1 text null,
data2 text null
);
Second:
create table table_2
(
id int auto_increment
primary key,
t1_id int null,
data1 text null,
data2 text null,
constraint table_2_table_1_id_fk
foreign key (t1_id) references table_1 (id)
on update cascade on delete cascade
);
For tests, there are 30 records in table_1, 60 in table_2 (every 2 records from table_2 refer by key to 1 record in table_1).
Next, a simple php script that updates some records from table_1, by condition from table_2 in an infinite loop:
<?php
$db = new PDO("mysql:host=127.0.0.1;dbname=test_db;", 'debian-sys-maint', 'pass', [
PDO::MYSQL_ATTR_INIT_COMMAND => "SET NAMES 'utf8'",
PDO::ATTR_DEFAULT_FETCH_MODE => PDO::FETCH_ASSOC,
PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION
]);
$random = rand(0, 9);
while (true) {
$db->exec("
UPDATE table_1
INNER JOIN table_2 on table_1.id = table_2.t1_id and table_2.data2 like '%$random%'
SET table_1.data1 = table_1.data1 + 1,
table_1.data2 = table_1.data2 + 1;");
}
So, I run this PHP script in 150 instances, and a Deadlock error is generated. I tried to fix it by modifying the query as follows:
UPDATE table_1
INNER JOIN (SELECT t1_id FROM table_2 WHERE table_2.data2 like '%$random%' ORDER BY t1_id) table_2 on table_1.id = table_2.t1_id
SET table_1.data1 = table_1.data1 + 1,
table_1.data2 = table_1.data2 + 1;
WITH tmp (id) AS (SELECT t1_id FROM table_2 WHERE table_2.data2 like '%$random%' ORDER BY t1_id)
UPDATE table_1
INNER JOIN tmp on table_1.id = tmp.id
SET table_1.data1 = table_1.data1 + 1,
table_1.data2 = table_1.data2 + 1;
UPDATE table_1
SET table_1.data1 = table_1.data1 + 1,
table_1.data2 = table_1.data2 + 1
WHERE table_1.id in (SELECT t1_id FROM table_2 WHERE table_2.data2 like '%$random%' ORDER BY t1_id);
UPDATE table_1, (SELECT t1_id FROM table_2 WHERE table_2.data2 like '%$random%' ORDER BY t1_id) tmp1
SET table_1.data1 = table_1.data1 + 1,
table_1.data2 = table_1.data2 + 1
WHERE table_1.id = tmp1.t1_id;
Actually the first question: I don't understand where the deadlock comes from, if the records are always sorted in the same order, the race condition is excluded. This is what the SHOW ENGINE INNODB STATUS; returns when catching a deadlock:
------------------------
LATEST DETECTED DEADLOCK
------------------------
2022-05-10 01:56:01 140233769219840
*** (1) TRANSACTION:
TRANSACTION 3529589, ACTIVE 1 sec starting index read
mysql tables in use 2, locked 2
LOCK WAIT 7 lock struct(s), heap size 1128, 90 row lock(s)
MySQL thread id 20, OS thread handle 140233708357376, query id 173 localhost 127.0.0.1 debian-sys-maint executing
UPDATE table_1
INNER JOIN table_2 on table_1.id = table_2.t1_id and table_2.data2 like '%9%'
SET table_1.data1 = table_1.data1 + 1,
table_1.data2 = table_1.data2 + 1
*** (1) HOLDS THE LOCK(S):
RECORD LOCKS space id 2 page no 4 n bits 224 index PRIMARY of table `test_db`.`table_1` trx id 3529589 lock_mode X locks rec but not gap
.....................
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 2 page no 4 n bits 224 index PRIMARY of table `test_db`.`table_1` trx id 3529589 lock_mode X locks rec but not gap waiting
.....................
*** (2) TRANSACTION:
TRANSACTION 3529887, ACTIVE 0 sec starting index read
mysql tables in use 2, locked 2
LOCK WAIT 5 lock struct(s), heap size 1128, 52 row lock(s)
MySQL thread id 118, OS thread handle 140229428811520, query id 444 localhost 127.0.0.1 debian-sys-maint executing
UPDATE table_1
INNER JOIN table_2 on table_1.id = table_2.t1_id and table_2.data2 like '%7%'
SET table_1.data1 = table_1.data1 + 1,
table_1.data2 = table_1.data2 + 1
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 2 page no 4 n bits 224 index PRIMARY of table `test_db`.`table_1` trx id 3529887 lock_mode X locks rec but not gap
.....................
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 2 page no 4 n bits 224 index PRIMARY of table `test_db`.`table_1` trx id 3529887 lock_mode X locks rec but not gap waiting
.....................
*** WE ROLL BACK TRANSACTION (2)
The only way I managed to solve the deadlock was to create a temporary table that would contain a ready-made set of keys that would be affected by UPDATE immediately from table_1:
DROP TABLE IF EXISTS tmp1;
CREATE TEMPORARY TABLE tmp1
SELECT table_2.t1_id FROM table_2 WHERE table_2.data2 like '%$random%'
UPDATE table_1
INNER JOIN tmp1 on table_1.id = tmp1.t1_id
SET table_1.data1 = table_1.data1 + 1,
table_1.data2 = table_1.data2 + 1;
Or with the same success I can create a copy of table_2 in a temporary table, it also does not cause deadlocks:
DROP TABLE IF EXISTS tmp1;
CREATE TEMPORARY TABLE tmp1
SELECT * FROM table_2 WHERE 1;
UPDATE table_1
INNER JOIN tmp1 on table_1.id = tmp1.t1_id and tmp1.data2 like '%$random%'
SET table_1.data1 = table_1.data1 + 1,
table_1.data2 = table_1.data2 + 1;
The second question is: Why does this option not cause deadlocks? As I understand it, it is the records from table_1 that are blocked, in this case they are blocked in the same way, only the search does not take place according to table_2, but according to the temporary table tmp1.
It would seem that my problem is solved, deadlocks are not called, but I did not like the solution with a temporary table and I continued my tests during which I came across a very strange thing that finally drove me to a dead end. If you start and complete the transaction yourself, then deadlocks do not appear:
BEGIN;
UPDATE table_1
INNER JOIN table_2 on table_1.id = table_2.t1_id and table_2.data2 like '%$random%'
SET table_1.data1 = table_1.data1 + 1,
table_1.data2 = table_1.data2 + 1;
COMMIT;
This is probably the last and most exciting question for me... I have an autocommit enabled, why if I don't explicitly start and complete the transaction, then deadlocks come out? PHP is known to work in one thread, for 1 script there is exactly 1 connection to the database, all 150 scripts work in parallel
I think this is because of the isolation level. The default is REPEATABLE READ and the second clause maybe the problem.
For a unique index with a unique search condition, InnoDB locks only the index record found, not the gap before it.
For other search conditions, InnoDB locks the index range scanned, using gap locks or next-key locks to block insertions by other sessions into the gaps covered by the range. For information about gap locks and next-key locks, see Section 15.7.1, “InnoDB Locking”.
From dev.mysql
So, try SERIALIZABLE
This level is like REPEATABLE READ, but InnoDB implicitly converts all plain SELECT statements to SELECT ... FOR SHARE if autocommit is disabled. If autocommit is enabled, the SELECT is its own transaction. It therefore is known to be read only and can be serialized if performed as a consistent (nonlocking) read and need not block for other transactions. (To force a plain SELECT to block if other transactions have modified the selected rows, disable autocommit.)
I think it is simply that
table_2.data2 like '%$random%'
has a leading wildcard, thereby necessitating a full table scan. In doing so, it either locks too many rows, or bumps into the other thread.
Note: "90 row lock(s)"
Can the LIKE be improved and provide a suitable INDEX? Perhaps FULLTEXT would be useful here.
How can I select Ids of updated records in mysql;
Is there any way to do something like this
select Id from (update tblA set col1=false where col2 > 5 )
You can't, you have to select them before updating.
If your table uses InnoDB storage engine, use SELECT FOR UPDATE to lock selected rows and prevent other clients from inserting, updating or deleting selected rows.
START TRANSACTION;
SELECT id FROM tblA WHERE col2 > 5 FOR UPDATE;
UPDATE tblA SET col1 = false WHERE col2 > 5;
COMMIT;
Another alternative is to set some column to a value which is unique to your client (e.g. process id) and when select using that value.
But it requires an additional column.
UPDATE tblA SET col1 = false, process_id = 12345 WHERE col2 > 5 AND process_id IS NULL;
SELECT id FROM tblA WHERE process_id = 12345
Is there a way in mysql that I can find the number of rows that get locked when a certain query runs? Eg. for a query, what is the number of rows locked:-
UPDATE xyz SET ARCHIVE = 1 , LAST_MODIFIED = CURRENT_TIMESTAMP WHERE ID = '123' AND ARCHIVE = 0;
Assume in this case, there is a index on ID and Archive is part of primary key.
BEGIN;
# lock
UPDATE xyz SET ARCHIVE = 1 , LAST_MODIFIED = CURRENT_TIMESTAMP WHERE ID = '123' AND ARCHIVE = 0;
# returns locked rows (X)
SELECT trx_rows_locked FROM information_schema.innodb_trx;
# release
COMMIT;
I have a table where there is a column that contains values like this - ["12"]. The table has 28 million records in it. When I perform the below, it takes forever and eventually just errors out and I receive this message - "ERROR 1206 (HY000) at line 1: The total number of locks exceeds the lock table size"
This is the update statement:
update table set category_ids = REPLACE(REPLACE(category_ids,'"]',''),'["','');
Basically, just trying to remove the [" and "] from the values in that column.
Is there a more efficient/better way to do this?
I have not tested this but just guessing.
select limited rows for update, and then update.
select * from my_table where col like '[%]' limit 0, 100000 for update;
update my_table set col=replace(.... where ... limit 100000;
on the next iteration you can try like
select * from my_table where col like '[%]' limit 100001, 100000 for update;
update my_table set col=replace(.... where ... limit 100000;
This practice can control the number of locks permitted.
I need an MySQL Skript which does the following:
delete chunks of the database until it has deleted all link_id's greater then 10000
exmaple:
x = 10000
DELETE FROM pligg_links WHERE link_id > x and link_id < x+10000
x = x + 10000
...
So it would delete
DELETE FROM pligg_links WHERE link_id > 10000 and link_id < 20000
then
DELETE FROM pligg_links WHERE link_id > 20000 and link_id < 30000
until all id's less then 10000 have been removed
I need this because the database is very very big (more then a gig)
thank in advance
You could use the LIMIT statement to regulate how many items to delete in one step:
DELETE FROM pligg_links
WHERE link_id > 10000
LIMIT 1000;
http://dev.mysql.com/doc/refman/5.1/en/delete.html says:
The MySQL-specific LIMIT row_count option to DELETE tells the server
the maximum number of rows to be deleted before control is returned to
the client. This can be used to ensure that a given DELETE statement
does not take too much time. You can simply repeat the DELETE
statement until the number of affected rows is less than the LIMIT
value.
You could determine the number of deleted rows by using SELECT ROW_COUNT(); if you would like automatize deletion:
http://dev.mysql.com/doc/refman/5.0/en/information-functions.html#function_row-count
what is the problem in ::
DELETE FROM pligg_links WHERE link_id > 10000
There is another way of doing it:(Just execute these 3 queries in order)
Step1 : Insert into pligg_links_temp select * from pligg_links where link_id < 10000;
Step2 : Drop pligg_links;
Step3 : Insert into pligg_links select * from pligg_links_temp