I have a Spring / JDBC application that relies heavily on MySQL rollback for unit tests. I'm finding that if I perform certain DDL operations - even on temporary tables - in those transactions, the rollback fails even on normal DML statements. For example:
#Test
#Rollback(true)
public void testRollbackProblem() {
template.update("create temporary table foo (id INTEGER )");
template.update("update forms set form_name = 'blah' where form_id = 1412");
template.update("alter table foo add (name text)");
}
After this test completes, that middle statement will be persisted and not rolled back. Is there a way to prevent that? Perhaps certain parameters passed to the alter statement?
This is MySQL limitations as far I know. CREATE TABLE and ALTER TABLE statements cause an implicit commit and can not be rolled back:
The CREATE TABLE statement in InnoDB is processed as a single transaction. This means that a ROLLBACK from the user does not undo CREATE TABLE statements the user made during that transaction.
See also MySQL documentation: https://dev.mysql.com/doc/refman/5.7/en/implicit-commit.html
Updated:
Because you're using CREATE TEMPORARY TABLE it actually shouldn't do implicit commit but it happens because when you're executing ALTER TABLE:
CREATE TABLE and DROP TABLE statements do not commit a transaction if the TEMPORARY keyword is used. (This does not apply to other operations on temporary tables such as ALTER TABLE and CREATE INDEX, which do cause a commit.) However, although no implicit commit occurs, neither can the statement be rolled back, which means that the use of such statements causes transactional atomicity to be violated. For example, if you use CREATE TEMPORARY TABLE and then roll back the transaction, the table remains in existence.
Related
I have a table where it is constantly being sent UPDATES/INSERTS from other processes, and I am trying to perform a transaction to my SQL table where I rotate the table and move some of the last values into the new table that I just created:
BEGIN TRANSACTION;
CREATE TABLE temp LIKE sales;
RENAME TABLE sales TO sales_05_04_19, temp TO sales
INSERT INTO sales SELECT * FROM sales_05_04_19 WHERE time > 1556953200000;
COMMIT;
But it does not appear to be blocking these UPDATES/INSERTS and some seem to actually make it through to the newly created sales table before the transaction's INSERT occurs. This causes me to get the error on the transaction insert:
(1062, "Duplicate entry '1' for key 'PRIMARY'")
I thought that this transaction would block the UPDATES/INSERTS until it commits, but that doesn't seem to be the case here.
So I feel that I would need to acquire a lock. How would I go about doing this (if that is the right approach to fixing this)?
If you want to prevent updates on a table while you do this kind of thing, you'll need to LOCK TABLES rather than use a transaction. I suggest creating your new table, locking both it and the old one, doing the rename switcheroo, doing your insert, then releasing the locks. There is no need for the transaction. Transactions avoid inconsistency, but they do not guarantee order, and DDL statements like create and rename table are not transaction-safe in any case.
We've got an InnoDB table with 70 million rows, and we have been trying to run an alter table statement to modify and add a couple of columns. The query seems to have altered the table, and is now in the state of 'committing alter table to storage engine'.
START TRANSACTION;
ALTER TABLE table
MODIFY COLUMN column1 int(11) NOT NULL DEFAULT 0,
MODIFY COLUMN column2 tinyint(1) NOT NULL DEFAULT 1,
ADD COLUMN column3 int(11),
ADD COLUMN column4 int(11) NOT NULL DEFAULT 1,
ADD COLUMN column5 varchar(255);
COMMIT;
This has been running overnight, and is at 19 hours at the current time. We do not have the performance schema enabled so cannot look at an estimated time of completion. My concern lies as to what the query is doing and whether the query will rollback if killed. I have seen other questions relate to queries that are stuck in copying to tmp tables, or awaiting a table lock. However I cannot find anything about being stuck while the alter table is committing.
Is it safe to kill a query in this state, and if the query is killed, will it rollback successfully?
The server is running MariaDB 10.2
From the documentation:
Some statements cannot be rolled back. In general, these include data definition language (DDL) statements, such as those that create or drop databases, those that create, drop, or alter tables or stored routines.
You should design your transactions not to include such statements. If you issue a statement early in a transaction that cannot be rolled back, and then another statement later fails, the full effect of the transaction cannot be rolled back in such cases by issuing a ROLLBACK statement.
I implemented the ALGORITHM=INPLACE and LOCK=NONE for InnoDB in MySQL 5.6.
Depending on the previous table definition, this operation could imply ALGORITHM=INPLACE, or it could fall back to ALGORITHM=COPY.
Starting with MariaDB 10.3 (MDEV-11369), ADD COLUMN would be instantaneous; before that, the table would have to be rebuilt. (The syntax ALGORITHM=INPLACE is very misleading.)
Starting with MariaDB 10.2.13 and 10.3.5 (MDEV-11415), ALGORITHM=COPY will no longer write undo log records for the individual rows, and the rollback (in case of client disconnect or killed server) should be much faster.
As the ALTER TABLE is a DDL statement, it causes an implicit commit when it is executed. This means that it cannot be rolled back but interrupting the DDL (by killing the connection) will cause the already applied changes to be rolled back in a controlled manner.
Given that you are using the default ALTER TABLE operation (no ALGORITHM defined), cancelling it should be relatively fast as all it has to do, at least according to my knowledge, is to discard the new copy of the table.
I need to create an index on a large InnoDB production table and want to do this without locking the table in any way. I am using MySQL 5.6 (.38-83.90).
I tried
create index my_index on my_table(col1, col2);
Neither columns are primary keys. col1 is a foreign key.
Well, this totally locked the table. Other queries were stalled with "Waiting for table metadata lock" bringing my website to its knees. I had to kill the create index query.
From this https://dev.mysql.com/doc/refman/5.6/en/innodb-create-index-overview.html I thought that it would not lock the table: "... no syntax changes are required... The table remains available for read and write operations while the index is being created or dropped."
I see that I can set LOCK=NONE or LOCK=SHARED, but I don't see that it should be necessary or, if it is, which one I need to use.
"You can specify LOCK=NONE to assert that concurrent DML is permitted during the DDL operation. MySQL automatically permits concurrent DML when possible."
"You can specify LOCK=SHARED to assert that concurrent queries are permitted during a DDL operation. MySQL automatically permits concurrent queries when possible."
None of the limitations https://dev.mysql.com/doc/refman/5.6/en/innodb-create-index-limitations.html seem to apply to my case.
What am I missing?
My guess (just a guess) is that you are missing the ALGORITHM=INPLACE clause on the CREATE INDEX statement.
CREATE INDEX my_index ON my_table(col1, col2) ALGORITHM=INPLACE ;
^^^^^^^^^^^^^^^^^
Also be aware of transactions acquiring and holding metadata locks.
https://dev.mysql.com/doc/refman/5.6/en/metadata-locking.html
Any transaction that has referenced my_table will continue to hold a metadata lock on that table until the transaction is committed or rolled back. I suggest checking the TRANSACTIONS section of SHOW ENGINE INNODB STATUS output.
I have a script that runs 24-7 on a table to perform necessary functions on it. However, when it is running, it is almost impossible to do an ALTER TABLE ADD INDEX statement, as it seems like it just hangs indefinitely. Is there any way around this? How should I go about adding this index?
The Alter table statement is getting a metadata lockout. You cannot perform your alter statement while another transaction is in process on the same table. Since your script runs 24-7, it is not possible to do what you are asking.
To ensure transaction serializability, the server must not permit one session to perform a data definition language (DDL) statement on a table that is used in an uncompleted explicitly or implicitly started transaction in another session. The server achieves this by acquiring metadata locks on tables used within a transaction and deferring release of those locks until the transaction ends. A metadata lock on a table prevents changes to the table's structure. This locking approach has the implication that a table that is being used by a transaction within one session cannot be used in DDL statements by other sessions until the transaction ends.
You can read more about this Here at dev.mysql.
Version 5.6 has ALTER TABLE ... ALGORITHM=INLACE ... to do ADD INDEX and several other ALTERs without blocking everything.
pt-online-table-alter (from Percona.com) can do it in older versions of MySQL. It uses a TRIGGER.
Are sql statements such as CREATE TABLE tbl_name ..... allowed in transactions.
For example:
begin;
CREATE TABLE .......;
sdfghjk;
rollback;
The table is still created despite a statement in the transaction failing, and a rollback at the end. Is there a way to prevent the table from being created if a statement in the transaction fails?
DDL statements are allowed within transactions, but are not generally impacted by the transactions. From the MySQL Documentation on what can and cannot be rolled back:
Some statements cannot be rolled back. In general, these include data
definition language (DDL) statements, such as those that create or
drop databases, those that create, drop, or alter tables or stored
routines.
You should design your transactions not to include such statements. If
you issue a statement early in a transaction that cannot be rolled
back, and then another statement later fails, the full effect of the
transaction cannot be rolled back in such cases by issuing a ROLLBACK
statement.
Source
If you still need to use table you can do create temporary table..... It doesn't commit transaction but will be deleted when connection will be closed.