I need to create an index on a large InnoDB production table and want to do this without locking the table in any way. I am using MySQL 5.6 (.38-83.90).
I tried
create index my_index on my_table(col1, col2);
Neither columns are primary keys. col1 is a foreign key.
Well, this totally locked the table. Other queries were stalled with "Waiting for table metadata lock" bringing my website to its knees. I had to kill the create index query.
From this https://dev.mysql.com/doc/refman/5.6/en/innodb-create-index-overview.html I thought that it would not lock the table: "... no syntax changes are required... The table remains available for read and write operations while the index is being created or dropped."
I see that I can set LOCK=NONE or LOCK=SHARED, but I don't see that it should be necessary or, if it is, which one I need to use.
"You can specify LOCK=NONE to assert that concurrent DML is permitted during the DDL operation. MySQL automatically permits concurrent DML when possible."
"You can specify LOCK=SHARED to assert that concurrent queries are permitted during a DDL operation. MySQL automatically permits concurrent queries when possible."
None of the limitations https://dev.mysql.com/doc/refman/5.6/en/innodb-create-index-limitations.html seem to apply to my case.
What am I missing?
My guess (just a guess) is that you are missing the ALGORITHM=INPLACE clause on the CREATE INDEX statement.
CREATE INDEX my_index ON my_table(col1, col2) ALGORITHM=INPLACE ;
^^^^^^^^^^^^^^^^^
Also be aware of transactions acquiring and holding metadata locks.
https://dev.mysql.com/doc/refman/5.6/en/metadata-locking.html
Any transaction that has referenced my_table will continue to hold a metadata lock on that table until the transaction is committed or rolled back. I suggest checking the TRANSACTIONS section of SHOW ENGINE INNODB STATUS output.
Related
I have a MySQL table (call it 'my_table') with a composed primary key with 4 columns (call them 'a', 'b', 'c' and 'd').
At least one time I encountered a deadlock on parallel asynchronous EJB calls calling 'DELETE FROM my_table where a=? and b=?' with different values, so I started to look into how InnoDB table locking works.
I've found no clear documentation on how table locking works with composed keys. Is the whole table locked by the delete, despite the fact that there's no overlap among the actual rows being deleted?
Do I need to do a select to recover the values for c and d and delete batches using the whole primary key?
This is in the context of a complex application which works with 4 different databases. Only MySQL seems to have this issue.
InnoDB never locks the entire table for DML statements. (Unless the DML is hitting all rows.)
There are other locks for DDL statements, such as when ALTER TABLE is modifying/adding columns/indexes/etc. (Some of these have been greatly sped up in MySQL 8.0.)
There is nothing special about a composite key wrt locking.
There is a thing called a "gap lock". For various reasons, the "gap" between two values in the index will be locked. This prevents potential conflicts such as inserting the same new value that does not yet exist, and there is a uniqueness constraint.
Since the PRIMARY KEY is a unique key, you may have hit something like that.
If practical, do SHOW ENGINE INNODB STATUS; to see whether the lock is "gap" or not.
Another thing that can happen is that a lock can start out being weak, then escalate to "eXclusive". This can lead to a deadlock.
Do I need to do a select to recover the values for c and d and delete batches using the whole primary key?
I think you need to explain more precisely what you are doing. Provide the query. Provide SHOW CREATE TABLE.
InnoDB's lock handling is possibly unique to MySQL. It has some quirks. Sometimes it is a bit greedy about what it locks; to compensate, it is possibly faster than the competition.
In any case, check for deadlocks (and timeouts) and deal with them. The hope that these problems are rare enough that having to deal with them is not too much a performance burden.
DELETE FROM my_table where a=? and b=? means that potentially a large number of rows are being deleted. That means that the undo log and MVCC need to do a lot of work. Hence, I recommend trying not to delete (or update) more than 1K rows at a time.
Does
ALTER TABLE sample ADD COLUMN `hasItem` tinyint(1) DEFAULT NULL
lock the entire table?
Short answer: For MySQL < 5.6 locks are required. From 5.6 on, and using InnoDB, locks are not required for many ALTER TABLE operations including adding a column.
If you're using MySQL 5.5 or older, it will get a read lock for the whole operation and then a brief write lock at the end.
From the MySQL documentation for ALTER TABLE...
In most cases, ALTER TABLE makes a temporary copy of the original table... While ALTER TABLE is executing, the original table is readable by other sessions (with the exception noted shortly). Updates and writes to the table that begin after the ALTER TABLE operation begins are stalled until the new table is ready...
The exception referred to earlier is that ALTER TABLE blocks reads (not just writes) at the point where it is ready to install a new version of the table .frm file, discard the old file, and clear outdated table structures from the table and table definition caches. At this point, it must acquire an exclusive lock.
Which is to say, when adding a column it read locks the table for most of the operation, then gets a write lock at the end.
MySQL 5.6 added the Online DDL to InnoDB which speeds up and improves many things such as altering tables and indexes. Adding a column to a table will no longer require table locks except possibly brief exclusive locks at the start and end of the operation.
It should happen automatically, but to be sure set ALGORITHM=inplace and LOCK=none to your ALTER TABLE statement.
There is one exception...
InnoDB tables created before MySQL 5.6 do not support ALTER TABLE ... ALGORITHM=INPLACE for tables that include temporal columns (DATE, DATETIME or TIMESTAMP) and have not been rebuilt using ALTER TABLE ... ALGORITHM=COPY.
I have a script that runs 24-7 on a table to perform necessary functions on it. However, when it is running, it is almost impossible to do an ALTER TABLE ADD INDEX statement, as it seems like it just hangs indefinitely. Is there any way around this? How should I go about adding this index?
The Alter table statement is getting a metadata lockout. You cannot perform your alter statement while another transaction is in process on the same table. Since your script runs 24-7, it is not possible to do what you are asking.
To ensure transaction serializability, the server must not permit one session to perform a data definition language (DDL) statement on a table that is used in an uncompleted explicitly or implicitly started transaction in another session. The server achieves this by acquiring metadata locks on tables used within a transaction and deferring release of those locks until the transaction ends. A metadata lock on a table prevents changes to the table's structure. This locking approach has the implication that a table that is being used by a transaction within one session cannot be used in DDL statements by other sessions until the transaction ends.
You can read more about this Here at dev.mysql.
Version 5.6 has ALTER TABLE ... ALGORITHM=INLACE ... to do ADD INDEX and several other ALTERs without blocking everything.
pt-online-table-alter (from Percona.com) can do it in older versions of MySQL. It uses a TRIGGER.
I am running MySQL 5.6.13 and I would like to run a CREATE INDEX ... BTREE statement on my production database.
The table is InnoDB and has ~ 4 million rows, and I would like very much not to lock it.
According to the docs, it appears as if this statement will not completely lock my table and return quickly. But, I wanted a second opinion before I made this change.
Would it be safe to create this index?
By default, InnoDB in MySQL 5.6 will perform a read lock while creating the index, so you can still have other concurrent clients SELECT from the table, but not do insert/update/delete on that table while the index is being created.
You can optionally allow the index creation to be completely online and not even do the read lock:
ALTER TABLE my_table ADD INDEX a (a), LOCK=NONE;
See http://dev.mysql.com/doc/refman/5.6/en/innodb-create-index-overview.html for more details about online DDL statements in MySQL.
Also see this blog posted today from a MySQL Community Manager: Top 10 advances to availability since MySQL 5.5
PS: It's not necessary to specify BTREE for the index type. InnoDB supports only BTREE indexes, so it ignores that option.
When I update a particularly large table, the update times out because the table is locked while the indexes rebuild. Is there any way to rebuild the index ONLINE (i.e. Oracle) so the update does not timeout?
Simple answer: No, there is no way.
More complex answer: You can emulate online index addition by using statement-based replication and adding the index to the slave first, then making it the master. This is why people use packages like http://mysql-mmm.org/.
optimize table rebuilds indexes online
OPTIMIZE TABLE uses online DDL for regular and partitioned InnoDB tables, which reduces downtime for concurrent DML operations. The table rebuild triggered by OPTIMIZE TABLE and performed under the cover by ALTER TABLE ... FORCE is completed in place. An exclusive table lock is only taken briefly during the prepare phase and the commit phase of the operation. During the prepare phase, metadata is updated and an intermediate table is created. During the commit phase, table metadata changes are committed.
Source: https://dev.mysql.com/doc/refman/8.0/en/optimize-table.html
But! I've heard much about that you should not rebuild indexes in InnoDB because they are always up to date. Google a bit about that.
pt-online-schema-change can be used to optimize a table. OPTIMIZE TABLE is effectively a noop ALTER TABLE.
pt-online-schema-change --alter "ENGINE=InnoDB" D=sakila,t=actor