looking to optimize mysql5.5 databases for INNODB. I have tried the link
Note: The databases also contains MYISAM tables.
Howto: Clean a mysql InnoDB storage engine?
As per the instructions I have dropped all databases as well as ibdata* and ib_logfile* and made changes in my.cnf like
[mysqld]
innodb_file_per_table
innodb_flush_method=O_DIRECT
innodb_log_file_size=1G
innodb_buffer_pool_size=4G
Now on restoring the backup, not only mysql storing INNODB tables in separate files, but it is doing it for MYISAM tables as well.
This is resulting in MYISAM database unusable.
when I am trying to access the website having MYISAM tables I am getting the error - Error displaying the error page: Application Instantiation Error
Can't makeout why mysql is storing MYISAM tables in separate files.
Thanks.
AFAIK MyISAM always stores tables in their own files (3 files per table: .MYI, .MYD and .frm).
It is not allowed to remove the ibdata file. Do you have a backup of that file? To move from single file to file per table you need to do as stated here: http://dev.mysql.com/doc/refman/5.0/en/innodb-multiple-tablespaces.html (see under "Enabling and Disabling Multiple Tablespaces")
Related
this isn't my exact situation but close enough for description.
On a MySQL 8.0.28 instance on an ubuntu 22.08 server, I have a database 'important' that contains only MyISAM tables. Within MySQL I:
DROP DATABASE important;
then continue to 'do stuff' with other databases etc. Some time later, I decide I want that 'important' database back (without restoring other databases). Luckily I have a physical backup of the MySQL data folder, specifically including the 'important' directory.
If I stop the server, copy the 'important' directory back in place, then restart the server I hoped (and was the case under MySQL 5) that the database would re-appear, but it doesn't.
As far as I can sell, MySQL 5 used to physically scan the directory to look for databases whereas MySQL 8 seems to have a separate idea of what databases exist. To clarify, I am only using MySQL 8.0.28 in this scenario - the backup was made under the same version as I'm restoring to.
My question is, how can I make this database reappear? Perhaps that's "How can I make MySQL 8 rescan its database of databases"?
I think you're in uncharted territory.
It's true that in MySQL 5.x, you could create a schema simply with mkdir, creating a subdirectory under the data directory. Within that directory, each MyISAM table has three files: tablename.MYD, tablename.MYI, and tablename.frm. The frm file has metadata about the table and its columns, so the MyISAM storage engine could discover new tables simply by reading their respective frm file.
This was not possible for InnoDB in 5.x, because InnoDB had an internal data dictionary stored in the ibdata1 tablespace in the data directory. The data dictionary had to maintain duplicate metadata about InnoDB tables.
In MySQL 8.0, they eliminated frm files completely. Tables of all storage engines are managed by the InnoDB data dictionary. There is no interface to the data dictionary besides DDL statements like CREATE/ALTER/DROP. There are several benefits to this change.
This means you can't use external shell commands like mkdir and cp to create schemas and tables. You have to use DDL to communicate to the data dictionary. It's not only stored on disk in the InnoDB tablespace, but it's also cached as objects in the MySQL Server process. Using DDL updates both.
I have not tried this with MyISAM tables (I haven't used MyISAM tables if I can avoid it for over ten years), but in theory you could use CREATE SCHEMA and CREATE TABLE to recreate empty tables that match the definition of the tables you're restoring. Then outside of MySQL, use cp to copy the tables you want to restore over the empty data and index files. I would shut down MySQL Server before doing this, to hopefully avoid any race conditions or conflicting with file caches.
I have no idea if that will work. I would test these steps first on a test instance (not your production instance).
I have a MyISAM database with MyISAM and InnoDB tables, I want to migrate this database to another server, For MyISAM simply copying the .frm .MYI and .MYD works but for the InnoDB tables how they can be moved to new server?
Assuming that the configuration and versions are identical, and that the source server is completely shut down, you can move the ibdata file and all of the *.frm and *.ibd files. But it isn't advised.
The portable way to move all (or some) of the databases is to dump them using mysqldump and restore them on the target using mysql.
I physically moved the database and now I am not able to restore Innodb database.
I am getting this error-
140131 13:03:41 [ERROR] Cannot find or open table table_t1/t1#P#p0 from
the internal data dictionary of InnoDB though the .frm file for the
table exists. Maybe you have deleted and recreated InnoDB data
files but have forgotten to delete the corresponding .frm files
of InnoDB tables, or you have moved .frm files to another database?
or, the table contains indexes that this version of the engine
doesn't support.
See http://dev.mysql.com/doc/refman/5.5/en/innodb-troubleshooting.html
how you can resolve the problem.
Help me to restore this db
Put it back and move it through the SQL interface (e.g. RENAME TABLE).
I'm the lucker owner of a webhotel where the host changes settings without telling.
When thats said,
I have some tables in my database that are running with InnoDB engine.
But over the night the host have disabled InnoDB, so I cant convert it to MyISAM with ALTER command.
Anyway I can get the data out of the database, or convert it to MyISAM when InnoDB is disabled?
Only thing I see all the time is,
#1033 - Incorrect information in file: 'file.frm'
Thanks.
Unfortunately, you need to have InnoDB enabled so that MySQL could read the data for conversion.
To recover the data on another instance, you would need ibdata* files from MySQL root data directory as well as all *.ibd files from your database directory (if your MySQL setup had innodb_file_per_table enabled).
Is it possible to clean a mysql innodb storage engine so it is not storing data from deleted tables?
Or do I have to rebuild a fresh database every time?
Here is a more complete answer with regard to InnoDB. It is a bit of a lengthy process, but can be worth the effort.
Keep in mind that /var/lib/mysql/ibdata1 is the busiest file in the InnoDB infrastructure. It normally houses six types of information:
Table Data
Table Indexes
MVCC (Multiversioning Concurrency Control) Data
Rollback Segments
Undo Space
Table Metadata (Data Dictionary)
Double Write Buffer (background writing to prevent reliance on OS caching)
Insert Buffer (managing changes to non-unique secondary indexes)
See the Pictorial Representation of ibdata1
InnoDB Architecture
Many people create multiple ibdata files hoping for better disk-space management and performance, however that belief is mistaken.
Can I run OPTIMIZE TABLE ?
Unfortunately, running OPTIMIZE TABLE against an InnoDB table stored in the shared table-space file ibdata1 does two things:
Makes the table’s data and indexes contiguous inside ibdata1
Makes ibdata1 grow because the contiguous data and index pages are appended to ibdata1
You can however, segregate Table Data and Table Indexes from ibdata1 and manage them independently.
Can I run OPTIMIZE TABLE with innodb_file_per_table ?
Suppose you were to add innodb_file_per_table to /etc/my.cnf (my.ini). Can you then just run OPTIMIZE TABLE on all the InnoDB Tables?
Good News : When you run OPTIMIZE TABLE with innodb_file_per_table enabled, this will produce a .ibd file for that table. For example, if you have table mydb.mytable witha datadir of /var/lib/mysql, it will produce the following:
/var/lib/mysql/mydb/mytable.frm
/var/lib/mysql/mydb/mytable.ibd
The .ibd will contain the Data Pages and Index Pages for that table. Great.
Bad News : All you have done is extract the Data Pages and Index Pages of mydb.mytable from living in ibdata. The data dictionary entry for every table, including mydb.mytable, still remains in the data dictionary (See the Pictorial Representation of ibdata1). YOU CANNOT JUST SIMPLY DELETE ibdata1 AT THIS POINT !!! Please note that ibdata1 has not shrunk at all.
InnoDB Infrastructure Cleanup
To shrink ibdata1 once and for all you must do the following:
Dump (e.g., with mysqldump) all databases into a .sql text file (SQLData.sql is used below)
Drop all databases (except for mysql and information_schema) CAVEAT : As a precaution, please run this script to make absolutely sure you have all user grants in place:
mkdir /var/lib/mysql_grants
cp /var/lib/mysql/mysql/* /var/lib/mysql_grants/.
chown -R mysql:mysql /var/lib/mysql_grants
Login to mysql and run SET GLOBAL innodb_fast_shutdown = 0; (This will completely flush all remaining transactional changes from ib_logfile0 and ib_logfile1)
Shutdown MySQL
Add the following lines to /etc/my.cnf (or my.ini on Windows)
[mysqld]
innodb_file_per_table
innodb_flush_method=O_DIRECT
innodb_log_file_size=1G
innodb_buffer_pool_size=4G
(Sidenote: Whatever your set for innodb_buffer_pool_size, make sure innodb_log_file_size is 25% of innodb_buffer_pool_size.
Also: innodb_flush_method=O_DIRECT is not available on Windows)
Delete ibdata* and ib_logfile*, Optionally, you can remove all folders in /var/lib/mysql, except /var/lib/mysql/mysql.
Start MySQL (This will recreate ibdata1 [10MB by default] and ib_logfile0 and ib_logfile1 at 1G each).
Import SQLData.sql
Now, ibdata1 will still grow but only contain table metadata because each InnoDB table will exist outside of ibdata1. ibdata1 will no longer contain InnoDB data and indexes for other tables.
For example, suppose you have an InnoDB table named mydb.mytable. If you look in /var/lib/mysql/mydb, you will see two files representing the table:
mytable.frm (Storage Engine Header)
mytable.ibd (Table Data and Indexes)
With the innodb_file_per_table option in /etc/my.cnf, you can run OPTIMIZE TABLE mydb.mytable and the file /var/lib/mysql/mydb/mytable.ibd will actually shrink.
I have done this many times in my career as a MySQL DBA. In fact, the first time I did this, I shrank a 50GB ibdata1 file down to only 500MB!
Give it a try. If you have further questions on this, just ask. Trust me; this will work in the short term as well as over the long haul.
CAVEAT
At Step 6, if mysql cannot restart because of the mysql schema begin dropped, look back at Step 2. You made the physical copy of the mysql schema. You can restore it as follows:
mkdir /var/lib/mysql/mysql
cp /var/lib/mysql_grants/* /var/lib/mysql/mysql
chown -R mysql:mysql /var/lib/mysql/mysql
Go back to Step 6 and continue
UPDATE 2013-06-04 11:13 EDT
With regard to setting innodb_log_file_size to 25% of innodb_buffer_pool_size in Step 5, that's blanket rule is rather old school.
Back on July 03, 2006, Percona had a nice article why to choose a proper innodb_log_file_size. Later, on Nov 21, 2008, Percona followed up with another article on how to calculate the proper size based on peak workload keeping one hour's worth of changes.
I have since written posts in the DBA StackExchange about calculating the log size and where I referenced those two Percona articles.
Aug 27, 2012 : Proper tuning for 30GB InnoDB table on server with 48GB RAM
Jan 17, 2013 : MySQL 5.5 - Innodb - innodb_log_file_size higher than 4GB combined?
Personally, I would still go with the 25% rule for an initial setup. Then, as the workload can more accurate be determined over time in production, you could resize the logs during a maintenance cycle in just minutes.
The InnoDB engine does not store deleted data. As you insert and delete rows, unused space is left allocated within the InnoDB storage files. Over time, the overall space will not decrease, but over time the 'deleted and freed' space will be automatically reused by the DB server.
You can further tune and manage the space used by the engine through an manual re-org of the tables. To do this, dump the data in the affected tables using mysqldump, drop the tables, restart the mysql service, and then recreate the tables from the dump files.
I follow this guide for a complete reset (as root):
mysqldump --all-databases --single-transaction | gzip -c > /tmp/mysql.all.sql.gz
service mysql stop
mv /var/lib/mysql /var/lib/mysql.old; mkdir -m700 /var/lib/mysql; chown mysql:mysql /var/lib/mysql
mysql_install_db # mysql 5.5
mysqld --initialize-insecure # mysql 5.7
service mysql start
zcat /tmp/mysql.all.sql.gz | mysql
service mysql restart
What nobody seems to mention is the impact innodb_undo_log_truncate setting can have.
Take a look at my answer at How to shrink/purge ibdata1 file in MySQL.