I am trying to export my database using mysqldump and the sql file should satisfy the following conditions
The file should not contain data from table_x ( Keep the structure )
Delete/skip data that
is older than 10 days from table_y ( Keep the structure )
The conditions may increase in future for different tables.
This dump file will be used on local environment and will work as replacement for production database.
And Is there a way to write all these conditions inside a file?
mysqldump is what you want:
http://dev.mysql.com/doc/refman/5.7/en/mysqldump.html
mysqldump --single-transaction --host=localhost --user=MyUser --password=MyPassword MyDatabase MyTable --where="mydatefield > 'insert-10 day old date here'"
mysqldump --single-transaction --host=localhost --user=MyUser --password=MyPassword --ignore-table=MyDatabase.MyTable --ignore-tableMyDatabase.MySecondIgnoredTable
This answer could be improved by changing the 'insert-10 day old date here' placeholder with a backtick bash command that is curdate - 10 days.
Specifying the password on the command line can be a security risk and is discouraged, especially in scripts or shared hosting environment. Check the help file above to see the more complex process of creating a credentials file to use in the command.
If you have MyISAM tables instead of InnoDB you'll want to switch --single-transaction with --skip-add-locks
I realize this question is from a year ago, but someone might still find useful.
Related
This may seem like a very dumb question but I didn't learn it in any other way and I just want to have some clarification.
I started to use MySQL a while ago and in order to test various scenarios, I back up my databases. I used MySQL dump for that:
Export:
mysqldump -hSERVER -uUSER -pPASSWORD --all-databases > filename.sql
Import:
mysql -hSERVER -uUSER -pPASSWORD < filename.sql
Easy enough and it worked quite well up until now, when I noticed a little problem with this "setup": It does not fully "reset" the databases and tables. If, for example, there is an additional table added AFTER a dump file has been created, that additional table will not disappear if you import the same dump file. It essentially only "corrects" tables already there and recreates any databaes or tables missing, but does not remove any additional tables, which happen to have names that are not in the dump file.
What I want to do is to completely reset all the databases on a server when I import such a dump file. What would be the best solution? Is there a special import function reserved for that purpose or do I have to delete the databases myself first? Or is that a bad idea?
You can use the parameter --add-drop-database to add a "drop database" statement to the dump before each "create database" statement.
e.g.
mysqldump -hSERVER -uUSER -pPASSWORD --all-databases --add-drop-database >filename.sql
see here for details.
There's nothing magic about the dump and restore processes you describe. mysqldump writes out SQL statements that describe the current state of the database or databases you are dumping. It has to fetch a list of tables in each database you're dumping, then it has to read the tables one by one and write them out as SQL. On databases of any size, this takes time.
So, if you create a new table while mysqldump is running, it may not pick up that new table. Similarly, if your application software changes contents of tables while mysqldump is running, those changes may or may not show up in the backup.
You can look at the .sql files mysqldump writes out to see what they have picked up. If you want to be sure that your dumped .sql files are perfect, you need to run mysqldump on a quiet server -- one where nobody is running data definition language.
MySQL hot backup solutions are available. You may need to look into that.
The OP may want look into
mysql_install_db
if they want a fresh start with the post-install default
settings before restoring one or more dumped DBs. For
production servers, another useful script is:
mysql_secure_installation
Also, they may prefer to dump the DB(s) they created separately:
mysqldump -hSERVER -uUSER -pPASSWORD --database foo > foo.sql
to avoid inadvertently changing the internal DBs:
mysql, information_schema, performance_schema.
I need to restore a dumped database, but without discarding existing rows in tables.
To dump I use:
mysqldump -u root --password --databases mydatabase > C:\mydatabase.sql
To restore I do not use the mysql command, since it will discard all existing rows, but instead mysqlimport should do the trick, obviously. But how? Running:
mysqlimport -u root -p mydatabase c:\mydatabase.sql
says "table mydatabase.mydatabase does not exist". Why does it look for tables? How to restore dump with entire database without discarding existing rows in existing tables? I could dump single tables if mysqlimport wants it.
What to do?
If you are concerned with stomping over existing rows, you need to mysqldump it as follows:
MYSQLDUMP_OPTIONS="--no-create-info --skip-extended-insert"
mysqldump -uroot --ppassword ${MYSQLDUMP_OPTIONS} --databases mydatabase > C:\mydatabase.sql
This will do the following:
remove CREATE TABLE statements and use only INSERTs.
It will INSERT exactly one row at a time. This helps mitigate rows with duplicate keys
With the mysqldump performed in this manner, now you can import like this
mysql -uroot -p --force -Dtargetdb < c:\mydatabase.sql
Give it a Try !!!
WARNING : Dumping with --skip-extended-insert will make the mysqldump really big, but at least you can control each duplicate done one by one. This will also increase the length of time the reload of the mysqldump is done.
I would edit the mydatabase.sql file in a text editor, dropping the lines that reference dropping tables or deleting rows, then manually import the file normally using the mysql command as normal.
mysql -u username -p databasename < mydatabase.sql
The mysqlimport command is designed for dumps created with the mysql command SELECT INTO OUTFILE rather than direct database dumps.
This sounds like it is much more complicated than you are describing.
If you do a backup the way you describe, it has all the records in your database. Then you say that you do not want to delete existing rows from your database and load from the backup? Why? The reason why the backup file (the output from mysqldump) has the drop and create table commands is to ensure that you don't wind up with two copies of your data.
The right answer is to load the mysqldump output file using the mysql client. If you don't want to do that, you'll have to explain why to get a better answer.
I'm using the following command to create an incremental backup in MySQL
mysqldump -uusername -ppassword db_name --flush-logs > D:\dbname_incremental_backup.sql
However the sql file is as big as a complete backup, and obviously importing it takes a long time as well. Could anybody tell me how to create incremental backups and import just the new data from each incremental backup rather than the whole database again?
I have read all the related articles in dev.mysql.com but still can not understand how to do it.
mysqldump only creates full backups. There's no built-in functionality for incremental backups.
For that sort of thing you probably want Percona xtrabackup but that will only work with InnoDB tables. This is usually not an issue since using MyISAM tables is considered extremely harmful.
By default a mysql dump will drop tables making an incremental update impossible. If you open up the resulting file, you will see something like:
DROP TABLE IF EXISTS `some_table_name`;
You can create a dump without dumping and creating new tables using the --no-create-info option. To make your dump friendly to incremental imports, you should also use --skip-extended-import which will break inserts out into one insert statement per row. Combined with using --force on the import will mean that inserts for rows that exist will fail but the import will continue. You will end up seeing errors in the logs for rows that already exist, but new rows will be inserted as desired.
You should be able to export with the following command (I also recommend not typing the password in the command so that it won't appear in your history)
mysqldump -u username -p --no-create-info --skip-extended-insert db_name --flush-logs > D:\dbname_incremental_backup.sql
You can then import with the following command:
mysql -u username -p --force db_name < D:\dbname_incremental_backup.sql
Situation: our production mysql database makes a daily dump into a .sql file. I'd like to keep a shadow database that is relatively up to date.
I know that to create a mysql database from a .sql file, one uses:
mysql -u USERNAME -p DATABASENAME < FILE.SQL
For our db, this took 4-5 hours. Needless to say, I'd like to cut that down, and I'm wondering if there's a way to just update the db with what's new/changed. On Day 2, is there a way to just update my shadow database with the new .sql file dumped from the production db?
MySQL Replication is the way to go.
But, in cases, where that is not possible, use the following procedure:
Have a modifed timestamp column in all your tables and update this value whenever a row is inserted/changed.
Use the following mysqldump options to take the incremental SQL file (this uses REPLACE commands instead of insertcommands, since the existing record will be updated in the backup database).
Keep a timestamp value somewhere placed in the file system. and use it in the where condition. MDFD_DATE is the column name on which you need to filter. On successful backup, update the value stored in the file.
skip-tz-utc prevents MSQL from automatically adjusting the timestamp values, based on your timezone.
mysqldump --databases db1,db2 --user=user --password=password --no-create-info --no-tablespaces --replace --skip-tz-utc --lock-tables --add-locks --compact --where=MDFD_DATE>='2012-06-13 23:09:42' --log-error=dump_error.txt --result-file=result.sql
Use the new sql file and run it in your server.
Limitations:
This method will not work if some records are deleted in your database. You need to manually delete them from the backup databases. Otherwise, keep a DEL_FLAG column and update it to 'Y' in production for deleted records and use this condition to delete records in the backup databases.
This problem can be solved using mysql synchronization.
Some links to guide you:
http://www.howtoforge.com/mysql_database_replication
Free MySQL synchronization tool
https://launchpad.net/mysql-proxy
https://www.google.com.br/search?q=mysql+synchronization
I need to backup the whole of a MySQL database with the information about all users and their permissions and passwords.
I see the options on http://www.igvita.com/2007/10/10/hands-on-mysql-backup-migration/,
but what should be the options to backup all of the MySQL database with all users and passwords and permissions and all database data?
Just a full backup of MySQL so I can import later on another machine.
At it's most basic, the mysqldump command you can use is:
mysqldump -u$user -p$pass -S $socket --all-databases > db_backup.sql
That will include the mysql database, which will have all the users/privs tables.
There are drawbacks to running this on a production system as it can cause locking. If your tables are small enough, it may not have a significant impact. You will want to test it first.
However, if you are running a pure InnoDB environment, you can use the --single-transaction flag which will create the dump in a single transaction (get it) thus preventing locking on the database. Note, there are corner cases where the initial FLUSH TABLES command run by the dump can lock the tables. If that is the case, kill the dump and restart it. I would also recommend that if you are using this for backup purposes, use the --master-data flag as well to get the binary log coordinates from where the dump was taken. That way, if you need to restore, you can import the dump file and then use the mysqlbinlog command to replay the binary log files from the position where this dump was taken.
If you'd like to transfer also stored procedures and triggers it's may be worth to use
mysqldump --all-databases --routines --triggers
if you have master/slave replication you may dump their settings
--dump-slave and/or --master-data
Oneliner suitable for daily backups of all your databases:
mysqldump -u root -pVeryStrongPassword --all-databases | gzip -9 > ./DBBackup.$(date +"%d.%m.%Y").sql.gz
If put in cron it will create files in format DBBackup.09.07.2022.sql.gz on a daily basis.