I want to copy a live production database into my local development database. Is there a way to do this without locking the production database?
I'm currently using:
mysqldump -u root --password=xxx -h xxx my_db1 | mysql -u root --password=xxx -h localhost my_db1
But it's locking each table as it runs.
Does the --lock-tables=false option work?
According to the man page, if you are dumping InnoDB tables you can use the --single-transaction option:
--lock-tables, -l
Lock all tables before dumping them. The tables are locked with READ
LOCAL to allow concurrent inserts in the case of MyISAM tables. For
transactional tables such as InnoDB and BDB, --single-transaction is
a much better option, because it does not need to lock the tables at
all.
For innodb DB:
mysqldump --single-transaction=TRUE -u username -p DB
This is ages too late, but good for anyone that is searching the topic. If you're not innoDB, and you're not worried about locking while you dump simply use the option:
--lock-tables=false
The answer varies depending on what storage engine you're using. The ideal scenario is if you're using InnoDB. In that case you can use the --single-transaction flag, which will give you a coherent snapshot of the database at the time that the dump begins.
--skip-add-locks helped for me
To dump large tables, you should combine the --single-transaction option with --quick.
http://dev.mysql.com/doc/refman/5.1/en/mysqldump.html#option_mysqldump_single-transaction
This is about as late compared to the guy who said he was late as he was to the original answer, but in my case (MySQL via WAMP on Windows 7), I had to use:
--skip-lock-tables
For InnoDB tables use flag --single-transaction
it dumps the consistent state of the database at the time when BEGIN
was issued without blocking any applications
MySQL DOCS
http://dev.mysql.com/doc/refman/5.1/en/mysqldump.html#option_mysqldump_single-transaction
Honestly, I would setup replication for this, as if you don't lock tables you will get inconsistent data out of the dump.
If the dump takes longer time, tables which were already dumped might have changed along with some table which is only about to be dumped.
So either lock the tables or use replication.
mysqldump -uuid -ppwd --skip-opt --single-transaction --max_allowed_packet=1G -q db | mysql -u root --password=xxx -h localhost db
When using MySQL Workbench, at Data Export, click in Advanced Options and uncheck the "lock-tables" options.
Due to https://dev.mysql.com/doc/refman/5.7/en/mysqldump.html#option_mysqldump_lock-tables :
Some options, such as --opt (which is enabled by default), automatically enable --lock-tables. If you want to override this, use --skip-lock-tables at the end of the option list.
If you use the Percona XtraDB Cluster -
I found that adding
--skip-add-locks
to the mysqldump command
Allows the Percona XtraDB Cluster to run the dump file
without an issue about LOCK TABLES commands in the dump file.
Another late answer:
If you are trying to make a hot copy of server database (in a linux environment) and the database engine of all tables is MyISAM you should use mysqlhotcopy.
Acordingly to documentation:
It uses FLUSH TABLES, LOCK TABLES, and cp or scp to make a database
backup. It is a fast way to make a backup of the database or single
tables, but it can be run only on the same machine where the database
directories are located. mysqlhotcopy works only for backing up
MyISAM and ARCHIVE tables.
The LOCK TABLES time depends of the time the server can copy MySQL files (it doesn't make a dump).
As none of these approaches worked for me, I simply did a:
mysqldump [...] | grep -v "LOCK TABLE" | mysql [...]
It will exclude both LOCK TABLE <x> and UNLOCK TABLES commands.
Note: Hopefully your data doesn't contain that string in it!
Related
This may seem like a very dumb question but I didn't learn it in any other way and I just want to have some clarification.
I started to use MySQL a while ago and in order to test various scenarios, I back up my databases. I used MySQL dump for that:
Export:
mysqldump -hSERVER -uUSER -pPASSWORD --all-databases > filename.sql
Import:
mysql -hSERVER -uUSER -pPASSWORD < filename.sql
Easy enough and it worked quite well up until now, when I noticed a little problem with this "setup": It does not fully "reset" the databases and tables. If, for example, there is an additional table added AFTER a dump file has been created, that additional table will not disappear if you import the same dump file. It essentially only "corrects" tables already there and recreates any databaes or tables missing, but does not remove any additional tables, which happen to have names that are not in the dump file.
What I want to do is to completely reset all the databases on a server when I import such a dump file. What would be the best solution? Is there a special import function reserved for that purpose or do I have to delete the databases myself first? Or is that a bad idea?
You can use the parameter --add-drop-database to add a "drop database" statement to the dump before each "create database" statement.
e.g.
mysqldump -hSERVER -uUSER -pPASSWORD --all-databases --add-drop-database >filename.sql
see here for details.
There's nothing magic about the dump and restore processes you describe. mysqldump writes out SQL statements that describe the current state of the database or databases you are dumping. It has to fetch a list of tables in each database you're dumping, then it has to read the tables one by one and write them out as SQL. On databases of any size, this takes time.
So, if you create a new table while mysqldump is running, it may not pick up that new table. Similarly, if your application software changes contents of tables while mysqldump is running, those changes may or may not show up in the backup.
You can look at the .sql files mysqldump writes out to see what they have picked up. If you want to be sure that your dumped .sql files are perfect, you need to run mysqldump on a quiet server -- one where nobody is running data definition language.
MySQL hot backup solutions are available. You may need to look into that.
The OP may want look into
mysql_install_db
if they want a fresh start with the post-install default
settings before restoring one or more dumped DBs. For
production servers, another useful script is:
mysql_secure_installation
Also, they may prefer to dump the DB(s) they created separately:
mysqldump -hSERVER -uUSER -pPASSWORD --database foo > foo.sql
to avoid inadvertently changing the internal DBs:
mysql, information_schema, performance_schema.
I have a root access to a mysql server, I need to dump ALL the database inside the server.
I tried with a simple mysqldump, but the server and pc seems blocked due to the large size of the databases and tables. Can I "optimize" this DUMP avoiding locking the server (and PC) ?
Thank you so much!
EDIT:
I want to EXPORT all the databases from a Mysql Server.
I need to understand what options to pass to mysqldump to avoid blocking:
The Mysql Server <---- it CAN'T go down
The PC that is goind to do this DUMP
You can disable locking:
mysqldump --skip-lock-tables
Of course you will not be able to create a consistent dump that way, so I would not recommend to use that option.
When only using MyISAM and ARCHIVE tables you might want to consider using mysqlhotcopy (included with a regular mysql package). There is similar software for other table engines like InnoDB available.
Another option is using a replication slave for backup.
Fire the dump command from commandline. :
mysqldump <other mysqldump options> --routines > outputfile.sql
If we want to backup ONLY the stored procedures and triggers and not the mysql tables and data then we should run something like:
mysqldump --routines --no-create-info --no-data --no-create-db --skip-opt <database> > outputfile.sql
If you need to import them to another db/server you will have to run something like:
mysql <database> < outputfile.sql
I need to backup the whole of a MySQL database with the information about all users and their permissions and passwords.
I see the options on http://www.igvita.com/2007/10/10/hands-on-mysql-backup-migration/,
but what should be the options to backup all of the MySQL database with all users and passwords and permissions and all database data?
Just a full backup of MySQL so I can import later on another machine.
At it's most basic, the mysqldump command you can use is:
mysqldump -u$user -p$pass -S $socket --all-databases > db_backup.sql
That will include the mysql database, which will have all the users/privs tables.
There are drawbacks to running this on a production system as it can cause locking. If your tables are small enough, it may not have a significant impact. You will want to test it first.
However, if you are running a pure InnoDB environment, you can use the --single-transaction flag which will create the dump in a single transaction (get it) thus preventing locking on the database. Note, there are corner cases where the initial FLUSH TABLES command run by the dump can lock the tables. If that is the case, kill the dump and restart it. I would also recommend that if you are using this for backup purposes, use the --master-data flag as well to get the binary log coordinates from where the dump was taken. That way, if you need to restore, you can import the dump file and then use the mysqlbinlog command to replay the binary log files from the position where this dump was taken.
If you'd like to transfer also stored procedures and triggers it's may be worth to use
mysqldump --all-databases --routines --triggers
if you have master/slave replication you may dump their settings
--dump-slave and/or --master-data
Oneliner suitable for daily backups of all your databases:
mysqldump -u root -pVeryStrongPassword --all-databases | gzip -9 > ./DBBackup.$(date +"%d.%m.%Y").sql.gz
If put in cron it will create files in format DBBackup.09.07.2022.sql.gz on a daily basis.
I've exported a mysqldump of a database with InnoDB tables and foreign key relationships in them, using the --single-transaction flag (that I read somewhere I should use for InnoDB). No problems.
But when trying to import that dump into another existing database (same database, different server) I get all sorts of errors when trying to drop the tables because it would break the InnoDB relationships.
I also read that I should use foreign_key_checks=0 to avoid this, but this is a server variable, not part of the dump process. So I'm trying to figure out how to automate all this since I have a script that backs up the DB, it was working when all we had were MyISAM tables:
mysqldump -u user -p'password' --single-transaction -q database | ssh user#backup.com mysql -u user -p'password' database
Thanks.
You can dump into a file, add the required SET FOREIGN_KEY_CHECKS=0; in that file, and then feed the file to mysql.
It turns out that the mysqldump file is smart enough to detect that they are InnoDB tables and puts the appropriate comments at the top of the file. My problem was that when I exported through PHPMyAdmin it didn't put the correct comments on the file, hence causing all this trouble.
Thanks for your response.
You can also add to the mysql command line when restoring without editing the original file. This is very useful as mysql backups can become huge, and editing a GB+ file takes lots of CPU time versus adding this to the commandline,
mysql -D YourDatabaseName -u YourUserName -p --init-command="set ##foreign_key_checks=0"<YourBackupDumpFile.sql
I'm new to MySQL and I'm figuring out the best way to perform an on-line hot logical backup using mysqldump. This page suggests this command line:
mysqldump --single-transaction --flush-logs --master-data=2
--all-databases > backup_sunday_1_PM.sql
but... if you read the documentation carefully you find that:
While a --single-transaction dump is in process, to ensure a valid dump file
(correct table contents and binary log position), no other connection should use
the following statements: ALTER TABLE, DROP TABLE, RENAME TABLE, TRUNCATE TABLE. A
consistent read is not isolated from those statements, so use of them on a table to
be dumped can cause the SELECT performed by mysqldump to retrieve the table contents
to obtain incorrect contents or fail.
So, is there any way to prevent this possible dump corruption scenario?
I.e. a commands that could block those statements temporarily.
PS: MySQL bug entry on this subject http://bugs.mysql.com/bug.php?id=27850
Open a mysql command window and issue this command:
mysql> FLUSH TABLES WITH READ LOCK;
This will lock all tables in all databases on this MySQL instance until you issue UNLOCK TABLES (or terminate the client connection that holds these read locks).
To confirm this, you can open another command window and try to do an ALTER, DROP, RENAME or TRUNCATE. These commands hang, waiting for the read lock to be released. Hit Ctrl-C to terminate the waiting.
But while the tables have a read lock, you can still perform a mysqldump backup.
The FLUSH TABLES WITH READ LOCK command may be the same as using the --lock-all-tables option of mysqldump. It's not totally clear, but this doc seems to support it:
Another use for UNLOCK TABLES is to
release the global read lock acquired
with FLUSH TABLES WITH READ LOCK.
Both FLUSH TABLES WITH READ LOCK and --lock-all-tables use the phrase "global read lock," so I think it's likely that these do the same thing. Therefore, you should be able to use that option to mysqldump and protect against concurrent ALTER, DROP, RENAME, and TRUNCATE.
Re. your comment: The following is from Guilhem Bichot in the MySQL bug log that you linked to:
Hi. --lock-all-tables calls FLUSH
TABLES WITH READ LOCK. Thus it is
expected to block ALTER, DROP, RENAME,
or TRUNCATE (unless there is a bug or
I'm wrong). However, --lock-all-tables
--single-transaction cannot work (mysqldump throws an error message):
because lock-all-tables locks all
tables of the server against writes
for the duration of the backup,
whereas single-transaction is intended
to let writes happen during the backup
(by using a consistent-read SELECT in
a transaction), they are incompatible
in nature.
From this, it sounds like you cannot get concurrent access during a backup, and simultaneously block ALTER, DROP, RENAME and TRUNCATE.
I thought the same thing reading that part of the documentation though, I found more information:
4.5.4. mysqldump — A Database Backup Program
http://dev.mysql.com/doc/en/mysqldump.html
For InnoDB tables, mysqldump provides a way of making an online
backup:
shell> mysqldump --all-databases --single-transaction > all_databases.sql
This backup acquires a global read lock on all tables (using FLUSH
TABLES WITH READ LOCK) at the beginning of the dump. As soon as this
lock has been acquired, the binary log coordinates are read and the
lock is released. If long updating statements are running when the
FLUSH statement is issued, the MySQL server may get stalled until
those statements finish. After that, the dump becomes lock free and
does not disturb reads and writes on the tables. If the update
statements that the MySQL server receives are short (in terms of
execution time), the initial lock period should not be noticeable,
even with many updates.
There is a conflict with the --opt and --single-transaction options:
--opt
This option is shorthand. It is the same as specifying
--add-drop-table --add-locks --create-options --disable-keys --extended-insert --lock-tables --quick --set-charset. It should give you a fast dump operation and produce a dump file that can be reloaded
into a MySQL server quickly.
The --opt option is enabled by default. Use --skip-opt to disable it.
If I understand your question correctly you want the actual data and the DDL (Data Definition Language) together, because if you only want the DDL you would use --no-data. More information about this can be found at:
http://dev.mysql.com/doc/workbench/en/wb-reverse-engineer-create-script.html
Use the --databases option with mysqldump if you wish to create the
database as well as all its objects. If there is no CREATE DATABASE
db_name statement in your script file, you must import the database
objects into an existing schema or, if there is no schema, a new
unnamed schema is created.
As suggested by The Definitive Guide to MySQL 5 By Michael Kofler I would suggest the follow options:
--skip-opt
--single-transaction
--add-drop-table
--create-options
--quick
--extended-insert
--set-charset
--disable-keys
Additionally, not mentioned is --order-by-primary
Also if you are using the --databases option, you should also use --add-drop-database especially if combined with this answer If you are backing up databases that are connect on different networks you may need to use the --compress option.
So a mysqldump command (without using the --compress, --databases, or --add-drop-database options) would be :
mysqldump --skip-opt --order-by-primary --single-transaction --add-drop-table --create-options --quick --extended-insert --set-charset -h db_host -u username --password="myPassword" db_name | mysql --host=other_host db_name
I removed the reference to --disable-keys that was given in the book as it is not effective with InnoDB as i understand it. The MySql manual states:
For each table, surround the INSERT statements with /*!40000 ALTER
TABLE tbl_name DISABLE KEYS /; and /!40000 ALTER TABLE tbl_name
ENABLE KEYS */; statements. This makes loading the dump file faster
because the indexes are created after all rows are inserted. This
option is effective only for nonunique indexes of MyISAM tables.
I also found this bug report http://bugs.mysql.com/bug.php?id=64309 which has comments on the bottom from Paul DuBois who also wrote a few books to which I have no reference on this specific issue other than those comments found within that bug report.
Now to create the "Ultimate Backup" I would suggest to consider something along the lines of this shell script
https://github.com/red-ant/mysql-svn-backup/blob/master/mysql-svn.sh
You can't get a consistent dump without locking tables. I just do mine during a time of day that the 2 minutes it takes to do the dump isn't noticed.
One solution is to do replication, then back up the slave instead of the master. If the slave misses writes during the backup, it will just catch up later. This will also leave you with a live backup server in case the master fails. Which is nice.
Hi that's late for any answer but the solution arrived in MariaDB with BACKUP STAGE, open source time is relativist :)