We have a script that "rotates"/archive the Syslog tables in MySQL. This script:
at Linux level, renames the "MyISAM" tables files then compress them
then
inside MySQL, rename these tables
The 2 steps are "online". No MySQLd restart is required.
Now I built a new Syslog database in MariaDB (Debian Stretch). The tables are using InnoDB and not MyISAM. This script fails at the 2nd execution to rename the table inside MySQL after moving the file:
ERROR 1050 (42S01): Table 'SystemEvents_1' already exists
A reference of the table is kept somewhere (tablespace internal system table?) which prevents from doing that.
My question:
would it work if I migrate my tables to the ARIA engine with transactional=0?
Thanks, Vince
I think it is no longer possible.
I converted my tables to MyISAM (and even Aria with transactional=0) and had the same error message. I think the best is to use mysqldump to export the tables instead of directly renaming the filesystem files. Less convenient but mysqldump will always work regardless the choosen engine.
Related
this isn't my exact situation but close enough for description.
On a MySQL 8.0.28 instance on an ubuntu 22.08 server, I have a database 'important' that contains only MyISAM tables. Within MySQL I:
DROP DATABASE important;
then continue to 'do stuff' with other databases etc. Some time later, I decide I want that 'important' database back (without restoring other databases). Luckily I have a physical backup of the MySQL data folder, specifically including the 'important' directory.
If I stop the server, copy the 'important' directory back in place, then restart the server I hoped (and was the case under MySQL 5) that the database would re-appear, but it doesn't.
As far as I can sell, MySQL 5 used to physically scan the directory to look for databases whereas MySQL 8 seems to have a separate idea of what databases exist. To clarify, I am only using MySQL 8.0.28 in this scenario - the backup was made under the same version as I'm restoring to.
My question is, how can I make this database reappear? Perhaps that's "How can I make MySQL 8 rescan its database of databases"?
I think you're in uncharted territory.
It's true that in MySQL 5.x, you could create a schema simply with mkdir, creating a subdirectory under the data directory. Within that directory, each MyISAM table has three files: tablename.MYD, tablename.MYI, and tablename.frm. The frm file has metadata about the table and its columns, so the MyISAM storage engine could discover new tables simply by reading their respective frm file.
This was not possible for InnoDB in 5.x, because InnoDB had an internal data dictionary stored in the ibdata1 tablespace in the data directory. The data dictionary had to maintain duplicate metadata about InnoDB tables.
In MySQL 8.0, they eliminated frm files completely. Tables of all storage engines are managed by the InnoDB data dictionary. There is no interface to the data dictionary besides DDL statements like CREATE/ALTER/DROP. There are several benefits to this change.
This means you can't use external shell commands like mkdir and cp to create schemas and tables. You have to use DDL to communicate to the data dictionary. It's not only stored on disk in the InnoDB tablespace, but it's also cached as objects in the MySQL Server process. Using DDL updates both.
I have not tried this with MyISAM tables (I haven't used MyISAM tables if I can avoid it for over ten years), but in theory you could use CREATE SCHEMA and CREATE TABLE to recreate empty tables that match the definition of the tables you're restoring. Then outside of MySQL, use cp to copy the tables you want to restore over the empty data and index files. I would shut down MySQL Server before doing this, to hopefully avoid any race conditions or conflicting with file caches.
I have no idea if that will work. I would test these steps first on a test instance (not your production instance).
$ user=jocular; cat ~/list|while read db; do echo rm -vi /var/lib/mysql/$user_$db; done
That is what I came up with but my instructor gave me this feedback :
Removing MySQL databases in this manner can cause catastrophic problems for the MySQL server that can lead to loss of the MySQL server. Remove a database using InnoDB tables in this fashion and attempt to restore it from a backup to learn more.
What would be the safest command to remove the unmapped databases ?
Your instructor is right, InnoDB makes it harder to manipulate tables and databases using shell tools. The reason is that InnoDB manages a "data dictionary" inside the ibdata1 file, which catalogs the databases and tables and which tablespace files they belong in. If you move or rename or delete files in the shell, InnoDB's data dictionary now references out-of-date information, and subsequently trying to use those table names or database names runs into conflicts.
Sort of like when you get a new phone, and you keep getting calls from friends of the former owner of that phone number.
If you use SQL or other MySQL commands to drop the database, InnoDB makes sure to update its data dictionary and keep it in sync with reality.
user=jocular; cat ~/list | while read db; do mysqladmin drop "${user}_${db}" ; done
Mysqladmin prompts you before dropping a database, since there's no undoing that change. But you can also use the -f option to force dropping without prompting. Just be careful you don't drop the wrong database!
I have DB InnoDb innodb_db_1. I have turned on innodb_file_per_table.
If I go to var/lib/mysql/innodb_db_1/ I will find files table_name.ibd, table_name.frm, db.opt.
Now, I'm trying to copy these files to another DB for example to innodb_db_2(var/lib/mysql/innodb_db_2/) but nothing happened.
But if my DB will be MyIsam, I can copy in such way and everything be ok.
What suggestions to move the DB by copying the file of InnoDb DB?
Even when you use file-per-table, the tables keep some of their data and metadata in /var/lib/mysql/ibdata1. So you can't just move .ibd files to a new MySQL instance.
You'll have to backup and restore your database. You can use:
mysqldump, included with MySQL, reliable but slow.
mydumper, a community contributed substitute for mysqldump, this supports compression and parallel execution and other neat features.
Percona XtraBackup, which is free and performs high-speed physical backups of InnoDB (and also supports other storage engines). This is recommended to minimize interruption to your live operations, and also if your database is large.
Re your comment:
No, you cannot just copy .ibd files. You cannot turn off the requirement for ibdata1. This file includes, among other things, a data dictionary which you can think of like a table of contents for a book. It tells InnoDB what tables you have, and which physical file they reside in.
If you just move a .ibd file into another MySQL instance, this does not add it to that instance's data dictionary. So InnoDB has no idea to look in the new file, or which logical table it goes with.
If you want a workaround, you could ALTER TABLE mytable ENGINE=MyISAM, move that file and its .frm to another instance, and then ALTER TABLE mytable ENGINE=InnoDB to change it back. Remember to FLUSH TABLES WITH READ LOCK before you move MyISAM files.
But these steps are not for beginners. It would be a lot safer for you to use the backup & restore method unless you know what you're doing. I'm trying to save you some grief.
There is an easy procedure to move the whole Mysql InnoDB from pc A to pc B.
The conditions to perform the procedure are:
You need to have innodb_file_per_table option set
You need to be able to make a shutdown of the database
In my case i had to move whole 150Gb MySql database (the biggest table had aprox. 60Gb). Making sqldumps and loading them back was not an option (too slow).
So what I did was I made a "cold backup" of the mysql database (mysql doc) and then simply move files to another computer.
The steps to make after moving the databse are described here dba stackexchange.
I am writing this, because (assuming you are able to follow mentioned conditions) this is by far the fastest (and probalby the easiest) method to move a (large) MySql InnoDb and nobody mentioned it yet.
You can copy MyISAM tables all day long (safely, as long as they are flushed and locked or the server is stopped) but you can't do this with InnoDB, because the two storage engines handle tables and tablespaces very differently.
MyISAM automatically discovers tables by iterating the files in the directory named for the database.
InnoDB has an internal data dictionary stored in the system tablespace (ibdata1). Not only do the tables have to be consistent, there are identifiers in the .ibd files that must match what the data dictionary has stored internally.
Prior to MySQL 5.6, with the introduction of transportable tablespaces, this wasn't a supported operation. If you are using MySQL 5.6, the link provides you with information on how this works.
The alternatives:
use mysqldump [options] database_name > dumpfile.sql without the --databases option, which will dump the tables in the specified database but will omit any DATABASE commands (DROP DATABASE, CREATE DATABASE and USE), some or all of which, based on the combination of options specified, are normally added to the dump file. You can then import this with mysql [options] < dumpfile.sql.
CREATE TABLE db2.t1 LIKE db1.t1; INSERT INTO db2.t1 SELECT * FROM db1.t1; (for each table; you'll have to add any foreign key constraints back in)
ALTER TABLE on each table, changing it to MyISAM, then flushing and locking the tables with FLUSH TABLES WITH READ LOCK;, copying them over, then altering everything back to InnoDB. Not the greatest idea, since you'll lose all foreign key declarations and have to add them back on the original tables, but it is an alternative.
As far as I know, "hot copying" table files is a very bad idea (I've done it two times, and only made it work with MyISAM tables, and i did it only because I had no other choice).
My personal recomendation is: Use mysqldump. On your shell:
mysqldump -h yourHost -u yourUser -pYourPassword yourDatabase yourTable > dumpFile.sql
To copy the data from a dump file to another database, on your shell:
mysql -h yourHost -u yourUser -pYourPassword yourNewDatabase < dumpFile.sql
Check: mysqldump — A Database Backup Program.
If you insist on copying InnoDB files by hand, please read this: Backing Up and Recovering an InnoDB Database
Good day to you.
I used my MySQL server with "innodb_file_per_table" option, and now server is crashed. I want to recover this server using this way:
Uninstall old MySQL
Install new MySQL
Add "innodb_file_per_table" in MySQL configuration
Copy databases folders (only my, not mysql) from old MySQL/data to new MySQL/data
In every folder I have two files, .frm and .ibd, and looks like this files have all data from my databases tables.
But, after copying, tables in this databases didn't work: when I try to open some table, I get error: Table xxx doesn't exist in engine.
I tried REPAIR command, but it isn't helpful.
If you know how to finish my way of repair — please help.
...I know that I need to copy ibdata1 also, but it looks so dead for recovery, so I try way that I try.
REPAIR command won't help with InnoDB.
If you're lucky enough the best you can do is:
1. Start MySQL with innodb_force_recovery=4 (try 5, 6 values if InnoDB fails to start). Make sure innodb_purge_threads=0.
2. Dump the database with mysqldump tool. Yes, it may be slow, but there is no other choice.
3. Create new empty InnoDB table space and reload the dump.
If MySQL fails to start with innodb_force_recovery=6, then recovery from backups is the only option. Well, you can fetch records from *.ibd files, but this is tedious job - google data recovery tool from percona
UPDATE: Data recovery toolkit moved to GitHub
You need to copy everything, not only the data folder
For example, without the ibdata file, mysql don't know where the tables are stored.
https://serverfault.com/questions/487159/what-is-the-ibdata1-file-in-my-var-lib-mysql-directory
I'm trying to run a rather large query that is supposed to run nightly to populate a table. I'm getting an error saying Incorrect key file for table '/var/tmp/#sql_201e_0.MYI'; try to repair it but the storage engine I'm using (whatever the default is, I guess?) doesn't support repairing tables.
how do I fix this so I can run the query?
You must change the location of MySQL's temporary folder which is '/tmp' in most cases to a location with a bigger disk space. Change it in MySQL's config file.
Basically your server is running out of disk space where /tmp is located.
You'll need to run this command from the MySQL prompt:
REPAIR TABLE tbl_name USE_FRM;
From MySQL's documentation on the Repair command:
The USE_FRM option is available for use if the .MYI index file is missing or if its header is corrupted. This option tells MySQL not to trust the information in the .MYI file header and to re-create it using information from the .frm file. This kind of repair cannot be done with myisamchk.
Your query is generating a result set so large that it needs to build a temporary table either to hold some of the results or some intermediate product used in generating the result.
The temporary table is being generated in /var/tmp. This temporary table would appear to have been corrupted. Perhaps the device the temporary table was being built on ran out of space. However, usually this would normally result in an "out of space" error. Perhaps something else running on your machine has clobbered the temporary table.
Try reworking your query to use less space, or try reconfiguring your database so that a larger or safer partition is used for temporary tables.
MySQL Manual - B.5.4.4. Where MySQL Stores Temporary Files
The storage engine (MyISAM) DOES support repair table. You should be able to repair it.
If the repair fails then it's a sign that the table is very corrupted, you have no choice but to restore it from backups.
If you have other systems (e.g. non-production with same software versions and schema) with an identical table then you might be able to fix it with some hackery (copying the frm an MYI files, followed by a repair).
In essence, the trick is to avoid getting broken tables in the first place. This means always shutting your db down cleanly, never having it crash and never having hardware or power problems. In practice this isn't very likely, so if durability matters you may want to consider a more crash-safe storage engine.
Simple "REPAIR the table" from PHPMYADMIN solved this problem for me.
go to phpmyadmin
open problematic table
go to Operations tab (in my version of PMA)
at the bottom you will find "Repair table" link
In my case, there was a disc space issue. I deleted some unwanted war files from my server and it worked after that.
REPAIR TABLE tbl_name USE_FRM;
Command only run when MySQL 'Storage Engine' type should be 'MyISAM'
Hope this helps
this issue is because of low storage space availability of a particular drive(c:\ or d:\ etc.,), release some memory then it will work.
Thanks
Saikumar.P
This happenes might be because you ran out of disk storage and the mysql files and starting files got corrupted
The solution to be tried as below
First we will move the tmp file to somewhere with larger space
Step 1: Copy your existing /etc/my.cnf file to make a backup
cp /etc/my.cnf{,.back-`date +%Y%m%d`}
Step 2: Create your new directory, and set the correct permissions
mkdir /home/mysqltmpdir
chmod 1777 /home/mysqltmpdir
Step 3: Open your /etc/my.cnf file
nano /etc/my.cnf
Step 4: Add below line under the [mysqld] section and save the file
tmpdir=/home/mysqltmpdir
Secondly you need to remove or error files and logs from the /var/lib/mysql/ib_* that means to remove anything that starts by "ib"
rm /var/lib/mysql/ibdata1 and rm /var/lib/mysql/ibda.... and so on
Thirdly you will need to make sure that there is a pid file available to have the database to write in
Step 1 you need to edit /etc/my.cnf
pid-file= /var/run/mysqld/mysqld.pid
Step 2 create the directory with the file to point to
mkdir /var/run/mysqld
touch /var/run/mysqld/mysqld.pid
chown -R mysql:mysql /var/run/mysqld
Last step restart mysql server
/etc/init.d/mysql restart
I just resolved a similar issue “Incorrect key file:\bonga_process\alarms.MYI’:try to repair it”
How I resolved it:
I started the server configuration application, ServerConfiguration.exe
Maitenance tab
Click on Repair tables command button
Prompts to delete files to free disk space appear with Manage Storage command
button
Click on Manage Storage command, this will run a take you to some files you need to delete to free more disk space. Delete any useless files.
Restart the system auditor application
Apply proper charset and collation to database, table and columns/fields.
I creates database and table structure using sql queries from one server to another.
it creates database structure as follows:
database with charset of "utf8", collation of "utf8_general_ci"
tables with charset of "utf8" and collation of "utf8_bin".
table columns / fields have charset "utf8" and collation of "utf8_bin".
I change collation of table and column to utf8_general_ci, and it resolves the error.
Change to MyISAM engine and run this command
REPAIR TABLE tbl_name USE_FRM;