This may seem like a very dumb question but I didn't learn it in any other way and I just want to have some clarification.
I started to use MySQL a while ago and in order to test various scenarios, I back up my databases. I used MySQL dump for that:
Export:
mysqldump -hSERVER -uUSER -pPASSWORD --all-databases > filename.sql
Import:
mysql -hSERVER -uUSER -pPASSWORD < filename.sql
Easy enough and it worked quite well up until now, when I noticed a little problem with this "setup": It does not fully "reset" the databases and tables. If, for example, there is an additional table added AFTER a dump file has been created, that additional table will not disappear if you import the same dump file. It essentially only "corrects" tables already there and recreates any databaes or tables missing, but does not remove any additional tables, which happen to have names that are not in the dump file.
What I want to do is to completely reset all the databases on a server when I import such a dump file. What would be the best solution? Is there a special import function reserved for that purpose or do I have to delete the databases myself first? Or is that a bad idea?
You can use the parameter --add-drop-database to add a "drop database" statement to the dump before each "create database" statement.
e.g.
mysqldump -hSERVER -uUSER -pPASSWORD --all-databases --add-drop-database >filename.sql
see here for details.
There's nothing magic about the dump and restore processes you describe. mysqldump writes out SQL statements that describe the current state of the database or databases you are dumping. It has to fetch a list of tables in each database you're dumping, then it has to read the tables one by one and write them out as SQL. On databases of any size, this takes time.
So, if you create a new table while mysqldump is running, it may not pick up that new table. Similarly, if your application software changes contents of tables while mysqldump is running, those changes may or may not show up in the backup.
You can look at the .sql files mysqldump writes out to see what they have picked up. If you want to be sure that your dumped .sql files are perfect, you need to run mysqldump on a quiet server -- one where nobody is running data definition language.
MySQL hot backup solutions are available. You may need to look into that.
The OP may want look into
mysql_install_db
if they want a fresh start with the post-install default
settings before restoring one or more dumped DBs. For
production servers, another useful script is:
mysql_secure_installation
Also, they may prefer to dump the DB(s) they created separately:
mysqldump -hSERVER -uUSER -pPASSWORD --database foo > foo.sql
to avoid inadvertently changing the internal DBs:
mysql, information_schema, performance_schema.
Related
I'm attempting to dump all the databases from a 500Gb RDS instance into a smaller instance (100Gb). I have a lot of user permissions saved so I need to dump the mysql table.
mysqldump -h hostname -u username -ppassword --all-databases > dump.sql
Now when I try to upload the data to my new instance I get the following error:
mysql -h hostname -u username -ppassword < dump.sql`
ERROR 1044 (42000) at line 2245: Access denied for user 'staging'#'%' to database 'mysql'
I would just use a database snapshot to accomplish this, but my instance is smaller in size.
As a sanity check, I tried dumping the data into the original instance but got the same error. Can someone please advise on what I should do here? Thanks!
You may need to do the databases individually, or at least remove the mysql schema from the existing file (perhaps using grep to find the line counts for the USE database; statements and then sed to trim out the troublesome section, or see below), and then generate a dump file that doesn't monkey with the table structures or the proprietary RDS triggers in the MySQL schema.
I have not tried to restore the full mysql schema onto an RDS instance, but I can certainly see where it would go awry with the customizations in RDS and the lack of SUPER privilege... but it seems like these options on mysqldump should get you close, at least.
mysqldump --no-create-info # don't try to drop and recreate the mysql schema tables
--skip-triggers # RDS has proprietary triggers in the mysql schema
--insert-ignore # write INSERT IGNORE statements to ignore duplicates
--databases mysql # only one database, "mysql"
--skip-lock-tables # don't generate statements to LOCK TABLES/UNLOCK TABLES during restore
--single-transaction # to avoid locking up the source instance during the dump
If this is still too aggressive, then you will need to resort to dumping only the rows from the specific tables whose content you need to preserve ("user" and the other grant tables).
THERE IS NO WARRANTY on the following, but it's one from my collection. It's a one-liner that reads "old_dumpfile.sql" and writes "new_dumpfile.sql"... but switching the output off when it sees the USE or CREATE DATABASE statements with `mysql` on the same line, and switching it back on again the next time such a statement occurs without `mysql` in it. This will need to be modified if your dump file also has the DROP DATABASE statements in it, or you could generate a new dumpfile with --skip-add-drop-database.
Running your existing dump file through this should essentially remove only the mysql schema from that file, allowing you to easily restore it manually, first, and then let the rest of the database data flow in more smoothly.
perl -pe 'if (/(^USE\s|^CREATE\sDATABASE.*\s)`mysql`/) { $x = 1; } elsif (/^USE\s`/ || /^CREATE\sDATABASE/) { $x = 0; }; $_ = "" if $x;' old_dumpfile.sql > new_dumpfile.sql
I guess you can try to use workbench. There is a migration function there, create the smaller instance (100GB) first, then use that migration feature to migrate from 500GB to the 100GB one see if it works.
I have had too many access denied issues with the RDS MySQL. So running below command on RDS is my way out:
GRANT ALL ON `%`.* to '<type_the_usernamne_here>'#'%';
I am not sure whether this will be helpful in your case. But it has always been a life saviour for me.
I need to restore a dumped database, but without discarding existing rows in tables.
To dump I use:
mysqldump -u root --password --databases mydatabase > C:\mydatabase.sql
To restore I do not use the mysql command, since it will discard all existing rows, but instead mysqlimport should do the trick, obviously. But how? Running:
mysqlimport -u root -p mydatabase c:\mydatabase.sql
says "table mydatabase.mydatabase does not exist". Why does it look for tables? How to restore dump with entire database without discarding existing rows in existing tables? I could dump single tables if mysqlimport wants it.
What to do?
If you are concerned with stomping over existing rows, you need to mysqldump it as follows:
MYSQLDUMP_OPTIONS="--no-create-info --skip-extended-insert"
mysqldump -uroot --ppassword ${MYSQLDUMP_OPTIONS} --databases mydatabase > C:\mydatabase.sql
This will do the following:
remove CREATE TABLE statements and use only INSERTs.
It will INSERT exactly one row at a time. This helps mitigate rows with duplicate keys
With the mysqldump performed in this manner, now you can import like this
mysql -uroot -p --force -Dtargetdb < c:\mydatabase.sql
Give it a Try !!!
WARNING : Dumping with --skip-extended-insert will make the mysqldump really big, but at least you can control each duplicate done one by one. This will also increase the length of time the reload of the mysqldump is done.
I would edit the mydatabase.sql file in a text editor, dropping the lines that reference dropping tables or deleting rows, then manually import the file normally using the mysql command as normal.
mysql -u username -p databasename < mydatabase.sql
The mysqlimport command is designed for dumps created with the mysql command SELECT INTO OUTFILE rather than direct database dumps.
This sounds like it is much more complicated than you are describing.
If you do a backup the way you describe, it has all the records in your database. Then you say that you do not want to delete existing rows from your database and load from the backup? Why? The reason why the backup file (the output from mysqldump) has the drop and create table commands is to ensure that you don't wind up with two copies of your data.
The right answer is to load the mysqldump output file using the mysql client. If you don't want to do that, you'll have to explain why to get a better answer.
I'm using the following command to create an incremental backup in MySQL
mysqldump -uusername -ppassword db_name --flush-logs > D:\dbname_incremental_backup.sql
However the sql file is as big as a complete backup, and obviously importing it takes a long time as well. Could anybody tell me how to create incremental backups and import just the new data from each incremental backup rather than the whole database again?
I have read all the related articles in dev.mysql.com but still can not understand how to do it.
mysqldump only creates full backups. There's no built-in functionality for incremental backups.
For that sort of thing you probably want Percona xtrabackup but that will only work with InnoDB tables. This is usually not an issue since using MyISAM tables is considered extremely harmful.
By default a mysql dump will drop tables making an incremental update impossible. If you open up the resulting file, you will see something like:
DROP TABLE IF EXISTS `some_table_name`;
You can create a dump without dumping and creating new tables using the --no-create-info option. To make your dump friendly to incremental imports, you should also use --skip-extended-import which will break inserts out into one insert statement per row. Combined with using --force on the import will mean that inserts for rows that exist will fail but the import will continue. You will end up seeing errors in the logs for rows that already exist, but new rows will be inserted as desired.
You should be able to export with the following command (I also recommend not typing the password in the command so that it won't appear in your history)
mysqldump -u username -p --no-create-info --skip-extended-insert db_name --flush-logs > D:\dbname_incremental_backup.sql
You can then import with the following command:
mysql -u username -p --force db_name < D:\dbname_incremental_backup.sql
I need to backup the whole of a MySQL database with the information about all users and their permissions and passwords.
I see the options on http://www.igvita.com/2007/10/10/hands-on-mysql-backup-migration/,
but what should be the options to backup all of the MySQL database with all users and passwords and permissions and all database data?
Just a full backup of MySQL so I can import later on another machine.
At it's most basic, the mysqldump command you can use is:
mysqldump -u$user -p$pass -S $socket --all-databases > db_backup.sql
That will include the mysql database, which will have all the users/privs tables.
There are drawbacks to running this on a production system as it can cause locking. If your tables are small enough, it may not have a significant impact. You will want to test it first.
However, if you are running a pure InnoDB environment, you can use the --single-transaction flag which will create the dump in a single transaction (get it) thus preventing locking on the database. Note, there are corner cases where the initial FLUSH TABLES command run by the dump can lock the tables. If that is the case, kill the dump and restart it. I would also recommend that if you are using this for backup purposes, use the --master-data flag as well to get the binary log coordinates from where the dump was taken. That way, if you need to restore, you can import the dump file and then use the mysqlbinlog command to replay the binary log files from the position where this dump was taken.
If you'd like to transfer also stored procedures and triggers it's may be worth to use
mysqldump --all-databases --routines --triggers
if you have master/slave replication you may dump their settings
--dump-slave and/or --master-data
Oneliner suitable for daily backups of all your databases:
mysqldump -u root -pVeryStrongPassword --all-databases | gzip -9 > ./DBBackup.$(date +"%d.%m.%Y").sql.gz
If put in cron it will create files in format DBBackup.09.07.2022.sql.gz on a daily basis.
I've exported a mysqldump of a database with InnoDB tables and foreign key relationships in them, using the --single-transaction flag (that I read somewhere I should use for InnoDB). No problems.
But when trying to import that dump into another existing database (same database, different server) I get all sorts of errors when trying to drop the tables because it would break the InnoDB relationships.
I also read that I should use foreign_key_checks=0 to avoid this, but this is a server variable, not part of the dump process. So I'm trying to figure out how to automate all this since I have a script that backs up the DB, it was working when all we had were MyISAM tables:
mysqldump -u user -p'password' --single-transaction -q database | ssh user#backup.com mysql -u user -p'password' database
Thanks.
You can dump into a file, add the required SET FOREIGN_KEY_CHECKS=0; in that file, and then feed the file to mysql.
It turns out that the mysqldump file is smart enough to detect that they are InnoDB tables and puts the appropriate comments at the top of the file. My problem was that when I exported through PHPMyAdmin it didn't put the correct comments on the file, hence causing all this trouble.
Thanks for your response.
You can also add to the mysql command line when restoring without editing the original file. This is very useful as mysql backups can become huge, and editing a GB+ file takes lots of CPU time versus adding this to the commandline,
mysql -D YourDatabaseName -u YourUserName -p --init-command="set ##foreign_key_checks=0"<YourBackupDumpFile.sql