I need mysqldump including the data, but in database there are also couple of tables with FEDERATED engine. The mysqldump includes INSERTs for FEDERATED tables too, what makes for exemaple "Duplicate entry" errors (because the table is already filled with data).
I can dump structure and data separately and then remove INSERTs for FEDERATED tables,
but this is not exactly the same as "make automatically reliable backups"
How to backup and then restore Mysql databases with FEDERATED tables?
After a quick look at the diff, the solution to this is to compile your own patched version of mysqldump that by default skips INSERTs for FEDERATED tables. I have not tried to apply it myself so YMMV.
http://bugs.mysql.com/bug.php?id=32038
Related
this isn't my exact situation but close enough for description.
On a MySQL 8.0.28 instance on an ubuntu 22.08 server, I have a database 'important' that contains only MyISAM tables. Within MySQL I:
DROP DATABASE important;
then continue to 'do stuff' with other databases etc. Some time later, I decide I want that 'important' database back (without restoring other databases). Luckily I have a physical backup of the MySQL data folder, specifically including the 'important' directory.
If I stop the server, copy the 'important' directory back in place, then restart the server I hoped (and was the case under MySQL 5) that the database would re-appear, but it doesn't.
As far as I can sell, MySQL 5 used to physically scan the directory to look for databases whereas MySQL 8 seems to have a separate idea of what databases exist. To clarify, I am only using MySQL 8.0.28 in this scenario - the backup was made under the same version as I'm restoring to.
My question is, how can I make this database reappear? Perhaps that's "How can I make MySQL 8 rescan its database of databases"?
I think you're in uncharted territory.
It's true that in MySQL 5.x, you could create a schema simply with mkdir, creating a subdirectory under the data directory. Within that directory, each MyISAM table has three files: tablename.MYD, tablename.MYI, and tablename.frm. The frm file has metadata about the table and its columns, so the MyISAM storage engine could discover new tables simply by reading their respective frm file.
This was not possible for InnoDB in 5.x, because InnoDB had an internal data dictionary stored in the ibdata1 tablespace in the data directory. The data dictionary had to maintain duplicate metadata about InnoDB tables.
In MySQL 8.0, they eliminated frm files completely. Tables of all storage engines are managed by the InnoDB data dictionary. There is no interface to the data dictionary besides DDL statements like CREATE/ALTER/DROP. There are several benefits to this change.
This means you can't use external shell commands like mkdir and cp to create schemas and tables. You have to use DDL to communicate to the data dictionary. It's not only stored on disk in the InnoDB tablespace, but it's also cached as objects in the MySQL Server process. Using DDL updates both.
I have not tried this with MyISAM tables (I haven't used MyISAM tables if I can avoid it for over ten years), but in theory you could use CREATE SCHEMA and CREATE TABLE to recreate empty tables that match the definition of the tables you're restoring. Then outside of MySQL, use cp to copy the tables you want to restore over the empty data and index files. I would shut down MySQL Server before doing this, to hopefully avoid any race conditions or conflicting with file caches.
I have no idea if that will work. I would test these steps first on a test instance (not your production instance).
We have a script that "rotates"/archive the Syslog tables in MySQL. This script:
at Linux level, renames the "MyISAM" tables files then compress them
then
inside MySQL, rename these tables
The 2 steps are "online". No MySQLd restart is required.
Now I built a new Syslog database in MariaDB (Debian Stretch). The tables are using InnoDB and not MyISAM. This script fails at the 2nd execution to rename the table inside MySQL after moving the file:
ERROR 1050 (42S01): Table 'SystemEvents_1' already exists
A reference of the table is kept somewhere (tablespace internal system table?) which prevents from doing that.
My question:
would it work if I migrate my tables to the ARIA engine with transactional=0?
Thanks, Vince
I think it is no longer possible.
I converted my tables to MyISAM (and even Aria with transactional=0) and had the same error message. I think the best is to use mysqldump to export the tables instead of directly renaming the filesystem files. Less convenient but mysqldump will always work regardless the choosen engine.
I have DB InnoDb innodb_db_1. I have turned on innodb_file_per_table.
If I go to var/lib/mysql/innodb_db_1/ I will find files table_name.ibd, table_name.frm, db.opt.
Now, I'm trying to copy these files to another DB for example to innodb_db_2(var/lib/mysql/innodb_db_2/) but nothing happened.
But if my DB will be MyIsam, I can copy in such way and everything be ok.
What suggestions to move the DB by copying the file of InnoDb DB?
Even when you use file-per-table, the tables keep some of their data and metadata in /var/lib/mysql/ibdata1. So you can't just move .ibd files to a new MySQL instance.
You'll have to backup and restore your database. You can use:
mysqldump, included with MySQL, reliable but slow.
mydumper, a community contributed substitute for mysqldump, this supports compression and parallel execution and other neat features.
Percona XtraBackup, which is free and performs high-speed physical backups of InnoDB (and also supports other storage engines). This is recommended to minimize interruption to your live operations, and also if your database is large.
Re your comment:
No, you cannot just copy .ibd files. You cannot turn off the requirement for ibdata1. This file includes, among other things, a data dictionary which you can think of like a table of contents for a book. It tells InnoDB what tables you have, and which physical file they reside in.
If you just move a .ibd file into another MySQL instance, this does not add it to that instance's data dictionary. So InnoDB has no idea to look in the new file, or which logical table it goes with.
If you want a workaround, you could ALTER TABLE mytable ENGINE=MyISAM, move that file and its .frm to another instance, and then ALTER TABLE mytable ENGINE=InnoDB to change it back. Remember to FLUSH TABLES WITH READ LOCK before you move MyISAM files.
But these steps are not for beginners. It would be a lot safer for you to use the backup & restore method unless you know what you're doing. I'm trying to save you some grief.
There is an easy procedure to move the whole Mysql InnoDB from pc A to pc B.
The conditions to perform the procedure are:
You need to have innodb_file_per_table option set
You need to be able to make a shutdown of the database
In my case i had to move whole 150Gb MySql database (the biggest table had aprox. 60Gb). Making sqldumps and loading them back was not an option (too slow).
So what I did was I made a "cold backup" of the mysql database (mysql doc) and then simply move files to another computer.
The steps to make after moving the databse are described here dba stackexchange.
I am writing this, because (assuming you are able to follow mentioned conditions) this is by far the fastest (and probalby the easiest) method to move a (large) MySql InnoDb and nobody mentioned it yet.
You can copy MyISAM tables all day long (safely, as long as they are flushed and locked or the server is stopped) but you can't do this with InnoDB, because the two storage engines handle tables and tablespaces very differently.
MyISAM automatically discovers tables by iterating the files in the directory named for the database.
InnoDB has an internal data dictionary stored in the system tablespace (ibdata1). Not only do the tables have to be consistent, there are identifiers in the .ibd files that must match what the data dictionary has stored internally.
Prior to MySQL 5.6, with the introduction of transportable tablespaces, this wasn't a supported operation. If you are using MySQL 5.6, the link provides you with information on how this works.
The alternatives:
use mysqldump [options] database_name > dumpfile.sql without the --databases option, which will dump the tables in the specified database but will omit any DATABASE commands (DROP DATABASE, CREATE DATABASE and USE), some or all of which, based on the combination of options specified, are normally added to the dump file. You can then import this with mysql [options] < dumpfile.sql.
CREATE TABLE db2.t1 LIKE db1.t1; INSERT INTO db2.t1 SELECT * FROM db1.t1; (for each table; you'll have to add any foreign key constraints back in)
ALTER TABLE on each table, changing it to MyISAM, then flushing and locking the tables with FLUSH TABLES WITH READ LOCK;, copying them over, then altering everything back to InnoDB. Not the greatest idea, since you'll lose all foreign key declarations and have to add them back on the original tables, but it is an alternative.
As far as I know, "hot copying" table files is a very bad idea (I've done it two times, and only made it work with MyISAM tables, and i did it only because I had no other choice).
My personal recomendation is: Use mysqldump. On your shell:
mysqldump -h yourHost -u yourUser -pYourPassword yourDatabase yourTable > dumpFile.sql
To copy the data from a dump file to another database, on your shell:
mysql -h yourHost -u yourUser -pYourPassword yourNewDatabase < dumpFile.sql
Check: mysqldump — A Database Backup Program.
If you insist on copying InnoDB files by hand, please read this: Backing Up and Recovering an InnoDB Database
I have a question relating to the backing up of multiple MySQL innodb databases which have cross database relations with one another. A concern would be the relational integrity of the data during backup and restore of the databases.
The use case for this would be a situation where one "master" database is been used to hold top level data such as User information for multiple separate domains which cannot be consolidated into one database. Setting this up is not an issue as MySQL allows cross database relations with innodb foreign keys.
However, how would one properly back up the master and secondary databases (assuming they are on the same server) without breaking the foreign keys during back up.
I am looking for a way to some how capture a "snap shot" in time of those databases in that instant or if that is not possible to preserve the relations during backup time...
I hope I understand the question correctly.
Suppose you have two databases (db1 and db2) where a foreign key constraint from one or more tables in db2 references something in db1.
If you want to make sure everything is point-in-time consistent when making a backup, you can let mysqldump do it for you. Whenever you run mysqldump, you issue the --single-transaction option like this:
mysqldump -u... -p... --single-transaction --databases db1 db2 > db1_db2.sql
An internal checkpoint is created so that db1 and db2 are consistent in terms of referential integrity and in point-in-time EVEN IF DATA IS BEING ENTERED INTO db1 AND db2 DURING THE BACKUP.
During the restore of a mysqldump, foreign keys are disabled to allow a clean, fast reload reload. You can trust the foreign keys being disabled during restore IF AND ONLY IF the mysqldump was made using the --single-transaction option.
I need to dump a database from a shared hosting that somehow doesn't have mysqldump installed. In fact, I only have mysql and mysqladmin available from the whole set of MySQL utilities.
Is it doable or I'll need to resort to installing something like phpMyAdmin?
You could use the following methods (from Database Backups in the documentation)
Making Backups by Copying Files
MyISAM tables are stored as files, so it is easy to do a backup by copying files. To get a consistent backup, do a LOCK TABLES on the relevant tables, followed by FLUSH TABLES for the tables. You need only a read lock; this allows other clients to continue to query the tables while you are making a copy of the files in the database directory. The FLUSH TABLES statement is needed to ensure that the all active index pages are written to disk before you start the backup.
FLUSH TABLES WITH READ LOCK;
Closes all open tables and locks all tables for all databases with a read lock until you explicitly release the lock by executing UNLOCK TABLES. This is very convenient way to get backups if you have a file system such as Veritas that can take snapshots in time.
UNLOCK TABLES;
Making Delimited-Text File Backups
To create a text file containing a table's data, you can use:
SELECT * INTO OUTFILE 'file_name' FROM tbl_name
This method works for any kind of data file, but saves only table data, not the table structure.
To reload the output file, use"
LOAD DATA INFILE
how about shutting down the server and copying the datadir itself?
You can get SQLYog. It has a back up database as SQL Dump option for each database.
Anyways,
I had to resort to using the Sypex dumper, a web-based tool for fast (really fast, much faster than phpMyAdmin for instance) MySQL database dumping. It's in Russian, but the interface is fairly obvious.
You can connect to a server remotely with mysqldump. For example:
mysqldump -u poweruser -h remote.mysql.host database
Maatkit seems quite appropriate for this with mk-parallel-dump and mk-parallel-restore.