Using liquibase v3.6.3 on MySQL. If I understood correctly, every CHANGESET is run into a SQL transaction by default. However, seems to me that transactions are being commites in a CHANGE basis. When running this script
databaseChangeLog:
- changeSet:
id: changeset-
changes:
- renameTable:
oldTableName: old_table
newTableName: new_table
- addColumn:
columns:
- column:
name: test_column_name
type: varchar(255)
tableName: other_table
If the addColumn tag fails because some SQL exception (i.e constraint check or other), then the databasechangelog table won't be updated, which I don't expect to, as the changeset failed. However, the firs statement DID pass and my table is now called new_table.
Of course, if I correct the problem causing the second one to fail and retry the update, it will fail because old_table doesn't exist anymore.
I'm aware of this paragraph in the liquibase documentation
Liquibase attempts to execute each changeSet in a transaction that is
committed at the end, or rolled back if there is an error. Some
databases will auto-commit statements which interferes with this
transaction setup and could lead to an unexpected database state.
Therefore, it is usually best to have just one change per changeSet
unless there is a group of non-auto-committing changes that you want
applied as a transaction such as inserting data.
https://www.liquibase.org/documentation/changeset.html
but I don't really understand it. Auto-commit means auto commiting A TRANSACTION. If all the changeset is wrapped in a transaction, why are there only some changes passing? Should liquibase rollback the whole transaction?
Any best practices for this? Can't we manually set transactions in liquibase?
It is not Liquibase that is committing a changeset partially.
I have worked with many databases and a basic concept for all the databases I used, is that a transaction combines data modifications (DML) only.
DDL is never part of a transaction. It is always executed immediately and an open transaction is automatically committed before it is executed.
The reason for that is that the rollback command of a database can handle data modifications only. It can't handle DDL. And if a rollback is not possible anymore then keeping the transaction open becomes useless.
So, Liquibase does create a transaction and commits all changes at the end as the doucmentation states. But that works only if the changeset contains DML only, no DDL.
And because of that DDL and DML should never be mixed in one changeset and every DDL statement should be in a separate changeset. Otherwise there is no way that Liquibase can prevent a changeset from partially succeeding and causing trouble when trying to rollback.
Mysql (and many other relational databases) has implicit commit concept. Most of database trigger the commit implicitly (just like you call COMMIT yourself) to end the current active transaction before(or after) executing the DDL statements.
Liquibase tries to apply specified changes of one changeset under single transaction. In your case, there are two changes and both are DDL statements (RENAME TABLE and ALTER TABLE), under one change set. Both statements will trigger the implicit commit which would leave the database inconsistent state if later statement fails.
More information on mysql implicit commit on their website including the comprehensive list of SQL statements which trigger the implicit commits.
Hope it helps.
Related
I want to test a script what it does when MySQL transaction fails. So I want to simulate transaction failure on COMMIT.
Image the script opens a transaction, makes some queries, and then commits the transaction. I want to implant some extra query to the transaction, to ensure the transaction will fail on COMMIT. So I can test how the script recovers from the transation failure.
I don't want to use explicit ROLLBACK. I want to simulate some real-life case when a commit fails. The exact DB structure is unimportant. It may adapt to the solution.
Edit: I need the transaction to fail on COMMIT, not on some prior query. Therefore answers that rollback prior to COMMIT are not what I want. Such as this one: How to simulate a transaction failure?
My unsuccessful attempts:
Inserting a row with invalid PK or FK fails immediately with insert. Temporarily disabling FK checks with FOREIGN_KEY_CHECKS=0 won't help as they won't be rechecked on COMMIT. If it was psql, defferable constraints would help. But not in mysql.
Opening two parallel transactions and inserting a row with the same PK (or any column with unique constaint) in both transactions locks the later transaction on insert and waits for the former transaction. So the transaction rolls back on insert not on commit.
So I believe you can try the following
Two phase commit : Use two database ( First and second) and make
use of two phase commit. When the commit statement is supposed to be
executed you can shutdown the second db. This way your commit
operation will fail and transaction will rollback.
You executed several inserts and before your commit your database server dies. A transaction may fail if it doesn't receive commit .
Hopefully it helps!
I made a wrong update query in my table.
I forgot to make an id field in the WHERE clause.
So that updated all my rows.
How to recover that?
I didn't have a backup....
There are two lessons to be learned here:
Backup data
Perform UPDATE/DELETE statements within a transaction, so you can use ROLLBACK if things don't go as planned
Being aware of the transaction (autocommit, explicit and implicit) handling for your database can save you from having to restore data from a backup.
Transactions control data manipulation statement(s) to ensure they are atomic. Being "atomic" means the transaction either occurs, or it does not. The only way to signal the completion of the transaction to database is by using either a COMMIT or ROLLBACK statement (per ANSI-92, which sadly did not include syntax for creating/beginning a transaction so it is vendor specific). COMMIT applies the changes (if any) made within the transaction. ROLLBACK disregards whatever actions took place within the transaction - highly desirable when an UPDATE/DELETE statement does something unintended.
Typically individual DML (Insert, Update, Delete) statements are performed in an autocommit transaction - they are committed as soon as the statement successfully completes. Which means there's no opportunity to roll back the database to the state prior to the statement having been run in cases like yours. When something goes wrong, the only restoration option available is to reconstruct the data from a backup (providing one exists). In MySQL, autocommit is on by default for InnoDB - MyISAM doesn't support transactions. It can be disabled by using:
SET autocommit = 0
An explicit transaction is when statement(s) are wrapped within an explicitly defined transaction code block - for MySQL, that's START TRANSACTION. It also requires an explicitly made COMMIT or ROLLBACK statement at the end of the transaction. Nested transactions is beyond the scope of this topic.
Implicit transactions are slightly different from explicit ones. Implicit transactions do not require explicity defining a transaction. However, like explicit transactions they require a COMMIT or ROLLBACK statement to be supplied.
Conclusion
Explicit transactions are the most ideal solution - they require a statement, COMMIT or ROLLBACK, to finalize the transaction, and what is happening is clearly stated for others to read should there be a need. Implicit transactions are OK if working with the database interactively, but COMMIT statements should only be specified once results have been tested & thoroughly determined to be valid.
That means you should use:
SET autocommit = 0;
START TRANSACTION;
UPDATE ...;
...and only use COMMIT; when the results are correct.
That said, UPDATE and DELETE statements typically only return the number of rows affected, not specific details. Convert such statements into SELECT statements & review the results to ensure correctness prior to attempting the UPDATE/DELETE statement.
Addendum
DDL (Data Definition Language) statements are automatically committed - they do not require a COMMIT statement. IE: Table, index, stored procedure, database, and view creation or alteration statements.
Sorry man, but the chances of restoring an overwritten MySQL database are usually close to zero. Different from deleting a file, overwriting a record actually and physically overwrites the existing data in most cases.
To be prepared if anything comes up here, you should stop your MySQL server, and make a copy of the physical directory containing the database so nothing can get overwritten further: A simple copy+paste of the data folder to a different location should do.
But don't get your hopes up - I think there's nothing that can be done really.
You may want to set up a frequent database backup for the future. There are many solutions around; one of the simplest, most reliable and easiest to automate (using at or cron in Linux, or the task scheduler in Windows) is MySQL's own mysqldump.
Sorry to say that, but there is no way to restore the old field values without a backup.
Don't shoot the messenger...
Do you have binlogs enabled? You can recover by accessing the binlogs.
I saw in multiple answers here and on google that rollback a transaction implies only the rollback of the last command, and i read also that implies ALL commands. (neither both documented or referenced by)
That I need to do is create a store procedure that insert/update on table A, get the last ID of A, insert that ID into B, get the last id of B, insert it into C, etc, etc, etc.
I want to know which is the method to commit or rollback all commands in the transaction, in order to start the transaction and if something fails, get back everything as the original.
SQL code with IF error and last_id will be preciated, because also I saw a LOT of differents ways to get the last id and I don't know which is better.
By the way, all tables are InnoDB
Kind regards,
If you BEGIN a transaction then nothing will get applied until you COMMIT it. Dropping your connection or issuing a ROLLBACK is the same as never committing it.
This is, of course, presuming you have autocommit set on, which is usually the case.
You can roll-back individual commands if you wrap them as transactions as well.
More information is available in the documentation.
Keep in mind that MyISAM and other engines do not support transactions, where InnoDB does. Further, only INSERT, UPDATE, DELETE and REPLACE statements are able to be rolled back. Other things, like alterations to the schema, are not.
As documented under START TRANSACTION, COMMIT, and ROLLBACK Syntax:
These statements provide control over use of transactions:
[ deletia ]
ROLLBACK rolls back the current transaction, canceling its changes.
I wrote a tool for our project, for applying sql update files that were committed, to the DB. Whenever run (on deployment), it calculates the list of update files which need to be applied, and applies them iniside a transaction.
Recently I became aware of an issue: mysql would implicitly commit a transaction, whenever DDL statements (like create) are executed. http://dev.mysql.com/doc/refman/5.0/en/implicit-commit.html
This is an issue for me, as sometimes an sql update file contains several statements, which as I understand will result in committing the transaction in the middle of executing the update file. This is a problem, because whenever a subsequent update will fail (which happens from time to time) I want to be able to rollback the transaction, or at least track which update files where applied (completely) and which were not.
Is there a way around the implicit transactions issue? I.e. is there a way to rollback a sequence of DDL statements whenever one of them fail?
Any other suggestions how I can handle the issue?
Thanks
Gidi
No. MySQL does not support transactional DDL. You either need to separate your DDL statements from DML statements, or perhaps try to use migration tool like RuckUsing
I made a wrong update query in my table.
I forgot to make an id field in the WHERE clause.
So that updated all my rows.
How to recover that?
I didn't have a backup....
There are two lessons to be learned here:
Backup data
Perform UPDATE/DELETE statements within a transaction, so you can use ROLLBACK if things don't go as planned
Being aware of the transaction (autocommit, explicit and implicit) handling for your database can save you from having to restore data from a backup.
Transactions control data manipulation statement(s) to ensure they are atomic. Being "atomic" means the transaction either occurs, or it does not. The only way to signal the completion of the transaction to database is by using either a COMMIT or ROLLBACK statement (per ANSI-92, which sadly did not include syntax for creating/beginning a transaction so it is vendor specific). COMMIT applies the changes (if any) made within the transaction. ROLLBACK disregards whatever actions took place within the transaction - highly desirable when an UPDATE/DELETE statement does something unintended.
Typically individual DML (Insert, Update, Delete) statements are performed in an autocommit transaction - they are committed as soon as the statement successfully completes. Which means there's no opportunity to roll back the database to the state prior to the statement having been run in cases like yours. When something goes wrong, the only restoration option available is to reconstruct the data from a backup (providing one exists). In MySQL, autocommit is on by default for InnoDB - MyISAM doesn't support transactions. It can be disabled by using:
SET autocommit = 0
An explicit transaction is when statement(s) are wrapped within an explicitly defined transaction code block - for MySQL, that's START TRANSACTION. It also requires an explicitly made COMMIT or ROLLBACK statement at the end of the transaction. Nested transactions is beyond the scope of this topic.
Implicit transactions are slightly different from explicit ones. Implicit transactions do not require explicity defining a transaction. However, like explicit transactions they require a COMMIT or ROLLBACK statement to be supplied.
Conclusion
Explicit transactions are the most ideal solution - they require a statement, COMMIT or ROLLBACK, to finalize the transaction, and what is happening is clearly stated for others to read should there be a need. Implicit transactions are OK if working with the database interactively, but COMMIT statements should only be specified once results have been tested & thoroughly determined to be valid.
That means you should use:
SET autocommit = 0;
START TRANSACTION;
UPDATE ...;
...and only use COMMIT; when the results are correct.
That said, UPDATE and DELETE statements typically only return the number of rows affected, not specific details. Convert such statements into SELECT statements & review the results to ensure correctness prior to attempting the UPDATE/DELETE statement.
Addendum
DDL (Data Definition Language) statements are automatically committed - they do not require a COMMIT statement. IE: Table, index, stored procedure, database, and view creation or alteration statements.
Sorry man, but the chances of restoring an overwritten MySQL database are usually close to zero. Different from deleting a file, overwriting a record actually and physically overwrites the existing data in most cases.
To be prepared if anything comes up here, you should stop your MySQL server, and make a copy of the physical directory containing the database so nothing can get overwritten further: A simple copy+paste of the data folder to a different location should do.
But don't get your hopes up - I think there's nothing that can be done really.
You may want to set up a frequent database backup for the future. There are many solutions around; one of the simplest, most reliable and easiest to automate (using at or cron in Linux, or the task scheduler in Windows) is MySQL's own mysqldump.
Sorry to say that, but there is no way to restore the old field values without a backup.
Don't shoot the messenger...
Do you have binlogs enabled? You can recover by accessing the binlogs.