In mysql document https://dev.mysql.com/doc/refman/8.0/en/innodb-locks-set.html. It says
SELECT ... FOR UPDATE and SELECT ... FOR SHARE statements that use a
unique index acquire locks for scanned rows, and release the locks for
rows that do not qualify for inclusion in the result set (for example,
if they do not meet the criteria given in the WHERE clause).
But in practice, it does not work as I expected. Here is my test.
Create a new Table Person.
CREATE TABLE `Persion` (
`id` bigint NOT NULL,
`name` varchar(100) NOT NULL,
`age` bigint NOT NULL,
PRIMARY KEY (`id`),
KEY `Persion_name_IDX` (`name`) USING BTREE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
Insert following data.
id
name
age
10
bob
19
11
cat
20
This is my sql statement.
//transaction 1
start transaction;
SELECT * from test.Persion p where id IN (10,11) and age=19 for UPDATE;
//transaction 2
UPDATE test.Persion set name='tx6' where id=11;
In transaction 1, when I execute the select...for update sql, it should not lock the id=11 record, because it does't match the age=19 condition. So my second transaction should not be blocked. But in fact, it is!
So I want to know what happend. Did I understand the document correct?
I read the documentation the same way you do, and I think it's unclear or incorrect. I repeated your test and I got the same result you did — the row with id 11 remained locked by the first session.
Then I tested your case after setting the following in the first session:
SET transaction_isolation='READ-COMMITTED';
This allows the second session to update the row.
This makes me think locking follows the same rule for updates in READ-COMMITTED isolation level, described on this page: https://dev.mysql.com/doc/refman/8.0/en/innodb-transaction-isolation-levels.html
READ COMMITTED
Using READ COMMITTED has additional effects:
For UPDATE or DELETE statements, InnoDB holds locks only for rows that it updates or deletes. Record locks for nonmatching rows are released after MySQL has evaluated the WHERE condition.
A large table (~10.5M rows) has been causing issues lately. I previously modified my application to use temporary tables for faster selects, but was still having issues due to UPDATE statements. Today I implemented partitions so that the writes happen more quickly, but now my temporary tables error. Its purpose is to group events, placing the first event ID of a set in the EVENT_ID column. Example: writing 4 events beginning at 1000 would result in events 1000, 1001, 1002, 1003, all with an EVENT_ID of 1000. I have tried to do away with the UPDATE statements, but that would require too much refactoring, so it is not an option. Here is the table definition:
CREATE TABLE `all_events` (
`ID` bigint NOT NULL AUTO_INCREMENT,
`EVENT_ID` bigint unsigned DEFAULT NULL,
`LAST_UPDATE` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`EMPLOYEE_ID` int unsigned NOT NULL,
`QUANTITY` float unsigned NOT NULL,
`OPERATORS` float unsigned NOT NULL DEFAULT '0',
`SECSEARNED` decimal(10,2) unsigned NOT NULL DEFAULT '0.00' COMMENT 'for all parts in QUANTITY',
`SECSBURNED` decimal(10,2) unsigned NOT NULL DEFAULT '0.00',
`YR` smallint unsigned NOT NULL DEFAULT (year(curdate())),
PRIMARY KEY (`ID`,`YR`),
KEY `LAST_UPDATE` (`LAST_UPDATE`),
KEY `EMPLOYEE_ID` (`EMPLOYEE_ID`),
KEY `EVENT_ID` (`EVENT_ID`)
) ENGINE=InnoDB AUTO_INCREMENT=17464583 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
/*!50100 PARTITION BY RANGE (`YR`)
(PARTITION p2015 VALUES LESS THAN (2016) ENGINE = InnoDB,
PARTITION p2016 VALUES LESS THAN (2017) ENGINE = InnoDB,
PARTITION p2017 VALUES LESS THAN (2018) ENGINE = InnoDB,
PARTITION p2018 VALUES LESS THAN (2019) ENGINE = InnoDB,
PARTITION p2019 VALUES LESS THAN (2020) ENGINE = InnoDB,
PARTITION p2020 VALUES LESS THAN (2021) ENGINE = InnoDB,
PARTITION p2021 VALUES LESS THAN (2022) ENGINE = InnoDB,
PARTITION p2022 VALUES LESS THAN (2023) ENGINE = InnoDB,
PARTITION p2023 VALUES LESS THAN MAXVALUE ENGINE = InnoDB) */
Now in my application when running a report the statement:
CREATE TEMPORARY TABLE IF NOT EXISTS ape ENGINE=MEMORY AS
SELECT * FROM all_events
WHERE LAST_UPDATE BETWEEN '2022-05-01 00:00:00' AND CURRENT_TIMESTAMP()
Produces the error: 'Specified storage engine' is not supported for default value expressions.
Is there a way to still use temporary tables with ENGINE=MEMORY, or is there another high performance engine I can use? The statement worked until the partitioning was implemented. InnoDB is the only engine my tables can be in due to the MySQL implementation, and it has been InnoDB since before partitioning.
Edit: When removing ENGINE=MEMORY it does work, but running SHOW CREATE TABLE tells me that it's using InnoDB. I would prefer the performance increase of MEMORY vs InnoDB.
Second Edit:
The MySQL server has been crashing 2 to 3 times daily, and every time I catch it I find this error:
TRANSACTION 795211228, ACTIVE 0 sec fetching rows
mysql tables in use 13, locked 13
LOCK WAIT 866 lock struct(s), heap size 106704, 4800 row lock(s), undo log entries 1
MySQL thread id 5032986, OS thread handle 140442167994112, query id 141216988 myserver 192.168.1.100 my-user Searching rows for update
UPDATE `all_events` SET `EVENT_ID`=LAST_INSERT_ID() WHERE `EVENT_ID` IS NULL
RECORD LOCKS space id 30558 page no 16 n bits 792 index EVENT_ID of table `mydb`.`all_events` trx id 795211228 lock_mode X
It's running Galera Cluster with 3 nodes. Node 3 is the main, becomes unavailable, and 1 comes offline to resync 3. I fail over to 2 and we're usually good until it catches up, but it's causing downtime. The temp tables I'm using are for faster reads, the partitioning is my attempt at improving write performance.
Third edit:
Added example SELECT - note there are fields not in the table definition, I reduced what was displayed for simplicity of the post, but all fields in the SELECT do in fact exist.
CREATE TEMPORARY TABLE IF NOT EXISTS allpe AS
SELECT * FROM all_events
WHERE LAST_UPDATE BETWEEN ? AND ?;
CREATE TEMPORARY TABLE IF NOT EXISTS ap1 AS SELECT * FROM allpe;
CREATE TEMPORARY TABLE IF NOT EXISTS ap2 AS SELECT * FROM allpe;
SELECT PART_NUMBER, WORKCENTER_NAME, SUM(SECSEARNED) AS EARNED, SUM(SECSBURNED) AS BURNED, SUM(QUANTITY) AS QUANTITY, (
SELECT SUM(ap1.SECSEARNED)
FROM ap1
WHERE ap1.PART_NUMBER = ape.PART_NUMBER AND ap1.WORKCENTER_ID = ape.WORKCENTER_ID
) AS EARNEDALL, (
SELECT SUM(ap2.SECSBURNED)
FROM ap2
WHERE ap2.PART_NUMBER = ape.PART_NUMBER AND ap2.WORKCENTER_ID = ape.WORKCENTER_ID
) AS BURNEDALL
FROM allpe ape
WHERE EMPLOYEE_ID = ?
GROUP BY PART_NUMBER, WORKCENTER_ID, WORKCENTER_NAME, EMPLOYEE_ID
ORDER BY EARNED;
DROP TEMPORARY TABLE allpe;
DROP TEMPORARY TABLE ap1;
DROP TEMPORARY TABLE ap2;
Fourth edit:
Writing inside of stored procedure - this is not in a loop, but multiple rows can come from multiple joins to employee_presence, so I cannot get the ID and store it for writing subsequent rows.
INSERT INTO `all_events`(`EVENT_ID`,`LAST_UPDATE`,`PART_NUMBER`, `WORKCENTER_ID`,`XPPS_WC`, `EMPLOYEE_ID`,`WORKCENTER_NAME`, `QUANTITY`, `LEVEL_PART_NUMBER`,`OPERATORS`,`SECSEARNED`,`SECSBURNED`)
SELECT NULL,NOW(),NEW.PART_NUMBER,NEW.ID,OLD.XPPS_WC,ep.EMPLOYEE_ID,NEW.NAME,(NEW.PARTS_MADE-OLD.PARTS_MADE)*WorkerContrib(ep.EMPLOYEE_ID,OLD.ID),IFNULL(NEW.LEVEL_PART_NUMBER,NEW.PART_NUMBER),WorkerCount(NEW.ID)*WorkerContrib(ep.EMPLOYEE_ID,OLD.ID),WorkerContrib(ep.EMPLOYEE_ID,OLD.ID)*CreditSeconds,WorkerCount(NEW.ID)*WorkerContrib(ep.EMPLOYEE_ID,OLD.ID)*IFNULL(TIMESTAMPDIFF(SECOND, GREATEST(NEW.LAST_PART_TIME,NEW.JOB_START_TIME), now()),0)
FROM employee_presence ep WHERE ep.WORKCENTER_ID=OLD.ID;
UPDATE `all_events` SET `EVENT_ID`=LAST_INSERT_ID() WHERE `WORKCENTER_ID`=NEW.ID AND `EVENT_ID` IS NULL;
I would suppose to read the following link from dev.MySQL.com
You cannot use CREATE TEMPORARY TABLE ... LIKE to create an empty
table based on the definition of a table that resides in the mysql
tablespace, InnoDB system tablespace (innodb_system), or a general
tablespace. The tablespace definition for such a table includes a
TABLESPACE attribute that defines the tablespace where the table
resides, and the aforementioned tablespaces do not support temporary
tables. To create a temporary table based on the definition of such a
table, use this syntax instead:
CREATE TEMPORARY TABLE new_tbl SELECT * FROM orig_tbl LIMIT 0;
So it seems the correct syntax for your case will be:
CREATE TEMPORARY TABLE ape
SELECT * FROM all_events
WHERE...
In the current issue the problematic column is YR smallint unsigned NOT NULL DEFAULT (year(curdate())). This DEFAULT value is not legal for a column which is used in partitioning expression. The error will be "Constant, random or timezone-dependent expressions in (sub)partitioning function are not allowed ...".
And only when you fix this by removing the partitioning then you'll receive an error "'Specified storage engine' is not supported for default value expressions".
CREATE TABLE .. SELECT inherits main columns properties from source tables.
In the current issue the problematic column is YR smallint unsigned NOT NULL DEFAULT (year(curdate())) again. The column in temptable must inherit main properties, including DEFAULT expression - but this expression is not allowed for MEMORY engine.
As the error suggests, the expression default does not work with the MEMORY storage engine.
One solution would be to remove that default from your all_events.yr column.
The other solution is to create an empty temporary table initially as an InnoDB table, then use ALTER TABLE to remove the expression default and convert to MEMORY engine before filling it with data.
Example:
mysql> create temporary table t as select * from all_events where false;
mysql> alter table t alter column yr drop default, engine=memory;
mysql> insert into t select * from all_events;
Sufficient? If I am not mistaken, this is equivalent to what your SELECT finds (no temp tables needed):
SELECT PART_NUMBER, WORKCENTER_ID, WORKCENTER_NAME, EMPLOYEE_ID,
SUM(SECSEARNED) AS TOT_EARNED,
SUM(SECSBURNED) AS TOT_BURNED,
SUM(QUANTITY) AS TOT_QUANTITY
FROM all_events
WHERE EMPLOYEE_ID = ?
AND LAST_UPDATE >= '2022-05-01'
GROUP BY PART_NUMBER, WORKCENTER_ID, WORKCENTER_NAME;
For performance, it would need this.
INDEX(EMPLOYEE_ID, LAST_UPDATE)
Also, removing the partitioning might speed it up a little more.
else (Notes on other fixes to the path you have taken)
Since yr is not needed, avoid it by changing '*' to a list of needed columns in
CREATE TEMPORARY TABLE IF NOT EXISTS ape ENGINE=MEMORY AS
SELECT * FROM all_events
WHERE LAST_UPDATE BETWEEN '2022-05-01 00:00:00' AND CURRENT_TIMESTAMP()
WHERE ap2.PART_NUMBER = ape.PART_NUMBER AND ap2.WORKCENTER_ID = ape.WORKCENTER_ID
Add this composite index to all_events:
INDEX(PART_NUMBER, WORKCENTER_ID)
That will probably suffice to make the query fast enough without the temp tables.
Also add thatallpe` after building it.
If you are running MySQL 8.0, you can use WITH instead of needing the two extra temp tables.
I have MyIsam table with few records (about 20):
CREATE TABLE `_cm_dtstd_37` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`NUMBER` int(10) NOT NULL COMMENT 'str',
`DESCRIPTION` char(32) NOT NULL COMMENT 'str',
PRIMARY KEY (`id`),
UNIQUE KEY `PHONE` (`NUMBER`)
) ENGINE=MyISAM AUTO_INCREMENT=15 DEFAULT CHARSET=utf8 COMMENT='==CORless Numbers=='
Single insert:
INSERT IGNORE INTO _cm_dtstd_37 VALUES(NULL, 55555, '55555')
takes very long time to execute (about 5 to 7 minutes) and makes MySql server put every next query on 'wait' state. No other query (even those that read/write other tables) is executed until first INSERT is done.
I have no idea how to debug this and where to search for any clue.
All inserts to another tables work well, whole database works great when not inserting to feral table.
That is one big reason for moving from MyISAM to InnoDB.
MyISAM allows multiple simultaneous reads (SELECT), but any type of write locks the entire table, even against writes.
InnoDB uses "row locking", so most simultaneous accesses to a table have no noticeable impact on each other.
Say I have this table:
CREATE TABLE `test` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`number` int(10) UNSIGNED NOT NULL,
PRIMARY KEY (id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Do I need some kind of lock if I insert data like this:
INSERT INTO test(number)
SELECT COALESCE(MAX(number), 0) + 1 FROM test;
In other words, if I have this statement executed in parallel multiple times, should I be worried that the same number could be inserted twice? I obviously will create a UNIQUE key (which will in fact be a composite key, that's why the classic AUTO INCREMENT feature does not fit my needs), but in that case should I be worry that a UNIQUE CONSTRAINT error might be thrown?
By default InnoDB uses auto-commit mode, so each query is a single transaction. So it will automatically perform the necessary locking to prevent duplication.
I have recently switched my project tables to InnoDB (thinking the relations would be a nice thing to have). I'm using a PHP script to index about 500 products at a time.
A table storing word/ids association:
CREATE TABLE `windex` (
`word` varchar(64) NOT NULL,
`wid` int(10) unsigned NOT NULL AUTO_INCREMENT,
`count` int(11) unsigned NOT NULL DEFAULT '1',
PRIMARY KEY (`wid`),
UNIQUE KEY `word` (`word`)
) ENGINE=InnoDB AUTO_INCREMENT=324551 DEFAULT CHARSET=latin1
Another table stores product id/word id associations:
CREATE TABLE `indx_0` (
`wid` int(7) unsigned NOT NULL,
`pid` int(7) unsigned NOT NULL,
UNIQUE KEY `wid` (`wid`,`pid`),
KEY `pid` (`pid`),
CONSTRAINT `indx_0_ibfk_1` FOREIGN KEY (`wid`) REFERENCES `windex` (`wid`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `indx_0_ibfk_2` FOREIGN KEY (`pid`) REFERENCES `product` (`ID`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=latin1
The script was tested using MyISAM and it indexes products relatively fast (much, much faster than InnoDB). First time running in InnoDB it was ridiculously slow but after nesting more values together I ended up speeding it up by a lot (but not enough).
I would assume innodb would be much faster for this type of thing because of rowlevel locks but that's not the case.
I construct a query that looks something like:
SELECT
title,keywords,upc,...
FROM product
WHERE indexed = 0
LIMIT 500
I create a loop and fill an array with all the words that need to be added to windex and all the word id/product id pairs that need to be added to indx_0.
Because innodb keeps increasing my auto-increment values whenever i do a "REPLACE INTO" or "INSERT IGNORE INTO" that fails because of duplicate values, I need to make sure the values I add don't already exist. To do that I first select all values that exist using a query like such:
SELECT wid,word
FROM windex
WHERE
word = "someword1" or word = "someword2" or word = "someword3" ... ...
Then I filter out my array against the results which exist so all the new words I add are 100% new.
This takes about 20% of overall execution time. The other 80% goes into adding the pair values into indx_0, for which there are many more values.
Here's an example of what I get.
0.4806 seconds to select products. (0.4807 sec total).
0.0319 seconds to gather 500 items. (0.5126 sec total).
5.2396 seconds to select windex values for comparison. (5.7836 sec total).
1.8986 seconds to update count. (7.6822 sec total).
0.0641 seconds to add 832 windex records. (7.7464 sec total).
17.2725 seconds to add index of 3435 pid/wid pairs. (25.7752 sec total).
Operation took 26.07 seconds to index 500 products.
The 3435 pairs are being all executed in a single query such as:
INSERT INTO indx_0(pid,wid)
VALUES (1,4),(3,9),(9,2)... ... ...
Why is InnoDB so much slower than MyISAM in my case?
InnoDB provides more complex keys structure than MyIsam (FOREIGN KEYS) and regenerating keys is really slow in InnoDB. You should enclose all update/insert statements into one transactions (those are actually quite fast in InnoDB, once I had about 300 000 insert queries on InnoDb table with 2 indexes and it took around 30 minutes, once I enclosed every 10 000 inserts into BEGIN TRANSACTION and COMMIT it took less than 2 minutes).
I recommend to use:
BEGIN TRANSACTION;
SELECT ... FROM products;
UPDATE ...;
INSERT INTO ...;
INSERT INTO ...;
INSERT INTO ...;
COMMIT;
This will cause InnoDB to refresh indexes just once not few hundred times.
Let me know if it worked
I had a similar problem and it seems InnoDB has by default innodb_flush_log_at_trx_commit enabled which flushes every insert/update query on your hdd log file. The writing speed of your hard disk is a bottleneck for this process.
So try to modify your mysql config file
`innodb_flush_log_at_trx_commit = 0`
Restart mysql service.
I experienced about x100 speedup on inserts.