I am using innodb. I have a problem with app performance. When I run mysqlturner.pl I get:
[--] Data in InnoDB tables: 8G (Tables: 1890)
[!!] Total fragmented tables: 1890
Ok, i have run mysqlcheck -p --optimize db.
I have diceded that innodb_file_per_table is disabled. The last 2 years a database wasn't reindex. How I can do it?
Make mysqldump?
Stop mysql serice?
insert enable innodb_file_per_table into my.cnf?
Start mysql serice?
Import from mysqldump?
run mysqlcheck -p --optimize db?
Will everything be ok?
Tables fragmented
Bogus.
All InnoDB tables are always (according to that tool) fragmented.
In reality, only 1 table in 1000 needs to be defragmented.
OPTIMIZE TABLE foo; will defrag a single table. But, again, I say "don't bother".
ibdata1 bloated
On the other hand... If you are concerned about the size of ibdata1 and Data_free (SHOW TABLE STATUS) shows that a large chunk of ibdata1 is "free", then the only cure is painful: As already mentioned: dump, stop, remove ibdata1, restart, reload.
But... If you don't have enough disk space to dump everything, you are in deep weeds.
You have 8GB of tables? (Sum Data_length and Index_length across all InnoDB tables.) If ibdata1 is, say 10GB, don't bother. If it is 100GB, then you have a lot of wasted space. But if you are not running out of disk space, again I say "don't bother".
If you are running out of disk space and ibdata1 seems to have a lot "free", then do the dump etc. But additionally: If you have 1890 tables, probably most are "tiny". Maybe a few are somewhat big? Maybe some table fluctuates dramatically in size (eg, add a million rows, then delete most of them)? I suggest having innodb_file_per_table ON when creating big or fluctuating tables; OFF for tiny tables.
Tiny tables take less space when living in ibdata1; big/fluctuating tables can be cleaned up if needed via OPTIMIZE if ever needed.
Yes that would do it. Take a backup before you start and test the entire process out first on an copy of the host, esp. the exact mysqldump command, you don't want databases like 'information_schema' in the dump.
Also ensure that other services cannot connect to your db once you start the export - any data they change would be lost during the import.
There are detailed instructions in this previous question:
Howto: Clean a mysql InnoDB storage engine?
Related
I have a wamp local server with only MyISAM tables, Total size is 350 MB and increases by about 10 MB per day. Would like to have a hourly backup solution. As recently WAMP server crashed and lost data.Should I create a trigger or schedule a task or use some utility it would be great if someone can guide. So that database backup is takeon either on local network or on different partition of server HDD. Have mirror RAID enabled on the server.
Since InnoDB does not lose data in a crash, and since you are talking about backing up a lot of data, and you are in the 'early stages', the only real answer to your question is to switch to InnoDB.
When the server crashes, MyISAM almost always leaves some indexes in need of REPAIR TABLE. This problem vanishes with InnoDB.
If the server crashes in the middle of a multi-row INSERT, UPDATE, or DELETE on a MyISAM table, some rows will be done and other rows won't be. And you have no way to know how much was done. With InnoDB, the entire query is "rolled back", making it as if it had not occurred. Very predictable.
See my tips
on converting.
For better recovery from a crash where the server cannot be restarted at all, consider Replication.
looking to optimize mysql5.5 databases for INNODB. I have tried the link
Note: The databases also contains MYISAM tables.
Howto: Clean a mysql InnoDB storage engine?
As per the instructions I have dropped all databases as well as ibdata* and ib_logfile* and made changes in my.cnf like
[mysqld]
innodb_file_per_table
innodb_flush_method=O_DIRECT
innodb_log_file_size=1G
innodb_buffer_pool_size=4G
Now on restoring the backup, not only mysql storing INNODB tables in separate files, but it is doing it for MYISAM tables as well.
This is resulting in MYISAM database unusable.
when I am trying to access the website having MYISAM tables I am getting the error - Error displaying the error page: Application Instantiation Error
Can't makeout why mysql is storing MYISAM tables in separate files.
Thanks.
AFAIK MyISAM always stores tables in their own files (3 files per table: .MYI, .MYD and .frm).
It is not allowed to remove the ibdata file. Do you have a backup of that file? To move from single file to file per table you need to do as stated here: http://dev.mysql.com/doc/refman/5.0/en/innodb-multiple-tablespaces.html (see under "Enabling and Disabling Multiple Tablespaces")
I have a huge MySQL InnoDB database (about 15 Go, and 100M rows), on a Debian server.
I have to save my database every two hours in another server, but without affect performances.
I looked at the MySQL replication, but it does not correspond to the fact that I look for, because I also want to protect of problems which the application could possibly cause.
What would be the best way of dealing with it?
Thank you very much!
I think you need incremental backups.
You can use Percona XtraBackup to make fast incremental backups. This works only if your database uses only InnoDB tables.
Refer to the documentation about how to create incremental backups:
http://www.percona.com/doc/percona-xtrabackup/howtos/recipes_ibkx_inc.html
Have you looked at writing a script that uses mysqldump to dump the contents of the DB, transfers it over to the backup DB (piping it to SSH would work), and inserts it via the command line?
There are options so that mysqldump won't lock the tables and so won't degrade performance too much.
Want to copy some big tables which are used very often from main server to local server in production mode ofc.
Is it safe?
Some suggestions, tools are welcome. :)
I'm guessing from asking about MYI tables that all of your tables are MyISAM tables and not InnoDb ones. Each MyISAM table is made up of three files: .frm, .MYD and .MYI they contain the structure, data and index respectively.
People advise against copying these raw files from running systems but I've found that as long as you're sure nothing's writing to the tables then copying them works fine (I've done this more times than I care to remember).
If you're doing this on a replica then just stop replication on the slave before copying. If it's just a single server or the master I'd recommend running FLUSH TABLES WITH READ LOCK before you start copying files, this will prevent any process from writing to the tables. When you're done release the lock with UNLOCK TABLES.
I would always recommend doing a CHECK TABLE on the tables you've copied in this manner. I think this is what I've used in the past mysqlcheck --all-tables --fast --auto-repair
If you're using LVM on the server then taking an LVM snapshot could be another way of getting hold of a clean snapshot.
If you're going to be doing this regularly I'd recommend either using replication to keep the local server up to date or to setup a slave which you can use for taking backups (as it's not the main database there's no problem with you stopping it, dumping tables, etc)
Yes it's alright if you
Shut the mysql server down cleanly before you copy the files, or at least take a global read lock
Take the .frm .MYI and .MYD files at the same time.
Have identical my.cnf and run the same MySQL version on each system
So it's basically ok, but not necessarily a good idea.
If you have different versions of mysql (You can only generally move to a more recent version, not an older one), or are running with a significantly different my.cnf (i.e. any of the fulltext indexing parameters differ, and you have fulltext indexes), you may need to rebuild the table. Rebuilding the table can be done on the destination server with ALTER TABLE blah ENGINE=MyISAM; this might still be a bit faster than a mysqldump / restore.
Is it possible to clean a mysql innodb storage engine so it is not storing data from deleted tables?
Or do I have to rebuild a fresh database every time?
Here is a more complete answer with regard to InnoDB. It is a bit of a lengthy process, but can be worth the effort.
Keep in mind that /var/lib/mysql/ibdata1 is the busiest file in the InnoDB infrastructure. It normally houses six types of information:
Table Data
Table Indexes
MVCC (Multiversioning Concurrency Control) Data
Rollback Segments
Undo Space
Table Metadata (Data Dictionary)
Double Write Buffer (background writing to prevent reliance on OS caching)
Insert Buffer (managing changes to non-unique secondary indexes)
See the Pictorial Representation of ibdata1
InnoDB Architecture
Many people create multiple ibdata files hoping for better disk-space management and performance, however that belief is mistaken.
Can I run OPTIMIZE TABLE ?
Unfortunately, running OPTIMIZE TABLE against an InnoDB table stored in the shared table-space file ibdata1 does two things:
Makes the table’s data and indexes contiguous inside ibdata1
Makes ibdata1 grow because the contiguous data and index pages are appended to ibdata1
You can however, segregate Table Data and Table Indexes from ibdata1 and manage them independently.
Can I run OPTIMIZE TABLE with innodb_file_per_table ?
Suppose you were to add innodb_file_per_table to /etc/my.cnf (my.ini). Can you then just run OPTIMIZE TABLE on all the InnoDB Tables?
Good News : When you run OPTIMIZE TABLE with innodb_file_per_table enabled, this will produce a .ibd file for that table. For example, if you have table mydb.mytable witha datadir of /var/lib/mysql, it will produce the following:
/var/lib/mysql/mydb/mytable.frm
/var/lib/mysql/mydb/mytable.ibd
The .ibd will contain the Data Pages and Index Pages for that table. Great.
Bad News : All you have done is extract the Data Pages and Index Pages of mydb.mytable from living in ibdata. The data dictionary entry for every table, including mydb.mytable, still remains in the data dictionary (See the Pictorial Representation of ibdata1). YOU CANNOT JUST SIMPLY DELETE ibdata1 AT THIS POINT !!! Please note that ibdata1 has not shrunk at all.
InnoDB Infrastructure Cleanup
To shrink ibdata1 once and for all you must do the following:
Dump (e.g., with mysqldump) all databases into a .sql text file (SQLData.sql is used below)
Drop all databases (except for mysql and information_schema) CAVEAT : As a precaution, please run this script to make absolutely sure you have all user grants in place:
mkdir /var/lib/mysql_grants
cp /var/lib/mysql/mysql/* /var/lib/mysql_grants/.
chown -R mysql:mysql /var/lib/mysql_grants
Login to mysql and run SET GLOBAL innodb_fast_shutdown = 0; (This will completely flush all remaining transactional changes from ib_logfile0 and ib_logfile1)
Shutdown MySQL
Add the following lines to /etc/my.cnf (or my.ini on Windows)
[mysqld]
innodb_file_per_table
innodb_flush_method=O_DIRECT
innodb_log_file_size=1G
innodb_buffer_pool_size=4G
(Sidenote: Whatever your set for innodb_buffer_pool_size, make sure innodb_log_file_size is 25% of innodb_buffer_pool_size.
Also: innodb_flush_method=O_DIRECT is not available on Windows)
Delete ibdata* and ib_logfile*, Optionally, you can remove all folders in /var/lib/mysql, except /var/lib/mysql/mysql.
Start MySQL (This will recreate ibdata1 [10MB by default] and ib_logfile0 and ib_logfile1 at 1G each).
Import SQLData.sql
Now, ibdata1 will still grow but only contain table metadata because each InnoDB table will exist outside of ibdata1. ibdata1 will no longer contain InnoDB data and indexes for other tables.
For example, suppose you have an InnoDB table named mydb.mytable. If you look in /var/lib/mysql/mydb, you will see two files representing the table:
mytable.frm (Storage Engine Header)
mytable.ibd (Table Data and Indexes)
With the innodb_file_per_table option in /etc/my.cnf, you can run OPTIMIZE TABLE mydb.mytable and the file /var/lib/mysql/mydb/mytable.ibd will actually shrink.
I have done this many times in my career as a MySQL DBA. In fact, the first time I did this, I shrank a 50GB ibdata1 file down to only 500MB!
Give it a try. If you have further questions on this, just ask. Trust me; this will work in the short term as well as over the long haul.
CAVEAT
At Step 6, if mysql cannot restart because of the mysql schema begin dropped, look back at Step 2. You made the physical copy of the mysql schema. You can restore it as follows:
mkdir /var/lib/mysql/mysql
cp /var/lib/mysql_grants/* /var/lib/mysql/mysql
chown -R mysql:mysql /var/lib/mysql/mysql
Go back to Step 6 and continue
UPDATE 2013-06-04 11:13 EDT
With regard to setting innodb_log_file_size to 25% of innodb_buffer_pool_size in Step 5, that's blanket rule is rather old school.
Back on July 03, 2006, Percona had a nice article why to choose a proper innodb_log_file_size. Later, on Nov 21, 2008, Percona followed up with another article on how to calculate the proper size based on peak workload keeping one hour's worth of changes.
I have since written posts in the DBA StackExchange about calculating the log size and where I referenced those two Percona articles.
Aug 27, 2012 : Proper tuning for 30GB InnoDB table on server with 48GB RAM
Jan 17, 2013 : MySQL 5.5 - Innodb - innodb_log_file_size higher than 4GB combined?
Personally, I would still go with the 25% rule for an initial setup. Then, as the workload can more accurate be determined over time in production, you could resize the logs during a maintenance cycle in just minutes.
The InnoDB engine does not store deleted data. As you insert and delete rows, unused space is left allocated within the InnoDB storage files. Over time, the overall space will not decrease, but over time the 'deleted and freed' space will be automatically reused by the DB server.
You can further tune and manage the space used by the engine through an manual re-org of the tables. To do this, dump the data in the affected tables using mysqldump, drop the tables, restart the mysql service, and then recreate the tables from the dump files.
I follow this guide for a complete reset (as root):
mysqldump --all-databases --single-transaction | gzip -c > /tmp/mysql.all.sql.gz
service mysql stop
mv /var/lib/mysql /var/lib/mysql.old; mkdir -m700 /var/lib/mysql; chown mysql:mysql /var/lib/mysql
mysql_install_db # mysql 5.5
mysqld --initialize-insecure # mysql 5.7
service mysql start
zcat /tmp/mysql.all.sql.gz | mysql
service mysql restart
What nobody seems to mention is the impact innodb_undo_log_truncate setting can have.
Take a look at my answer at How to shrink/purge ibdata1 file in MySQL.