MySQL: reducing ibdata file size for MyISAM tables - mysql

My Question is actually very similar to this one and also includes a good answer for a case with InnoDB Engine tables:
https://dba.stackexchange.com/questions/8982/is-there-any-best-way-to-reduce-the-size-of-ibdata-in-mysql/8983#8983,
I have noticed that drop schema do not shrink ibdata files , so i have looked for a methods to configure the DB so that the size will be reduced after deleting a schema.
i have found many links talking about InnoDB and the way to save table per file so that the .frm file it self will contain the table data and it will be reduced.
But what happens with MyISAM tables (with more than 5G table size).

ibdata1 and MyISAM are mutually exclusive.
First thing you should do is count how many tables use both storage engines:
SELECT COUNT(1) EngineCount,engine
FROM information_schema.tables
WHERE table_schema NOT IN ('information_schema','performance_schema','mysql')
GROUP BY engine;
If SOME tables are InnoDB:
Perform my CleanUp of InnoDB
Howto: Clean a mysql InnoDB storage engine?
https://dba.stackexchange.com/questions/8982/is-there-any-best-way-to-reduce-the-size-of-ibdata-in-mysql/8983#8983
If you have only MyISAM tables and no InnoDB tables:
First, eliminate any traces of InnoDB
Do the following:
STEP01) Add this to my.cnf
[mysqld]
skip-innodb
STEP02) service mysql restart
STEP03) rm -f /var/lib/mysql/ibdata1 /var/lib/mysql/ib_logfile*
After these steps, you can perform a Compression of Each MyISAM tables like this:
For the table mydb.mytable that is MyISAM, just run one of the following:
OPTIMIZE TABLE mydb.mytable;
ALTER TABLE mydb.mytable ENGINE=MyISAM; ANALYZE TABLE mydb.mytable;
If you want to defrag all your MyISAM tables, here is a shell script to do so...
MYSQL_USER=root
MYSQL_PASS=rootpassword
MYSQL_CONN="-u${MYSQL_USER} -p${MYSQL_PASS}"
SQL="SELECT CONCAT('OPTIMIZE TABLE ',table_schema,'.',table_name,';') "
SQL="${SQL} FROM information_schema.tables "
SQL="${SQL} WHERE engine='MyISAM' AND table_schema NOT IN "
SQL="${SQL} ('information_schema','performance_schema','mysql')"
mysql ${MYSQL_CONN} -ANe"${SQL}" > GlobalMyISAMOptmizeTable.sql
less GlobalMyISAMOptmizeTable.sql
Once you trust the script visually, just run it
mysql ${MYSQL_CONN} < GlobalMyISAMOptmizeTable.sql
Give it a Try !!!
UPDATE 2012-07-25 09:52 EDT
I would like to clarify one of my suggestions for compression of MyISAM
I said earlier
OPTIMIZE TABLE mydb.mytable;
ALTER TABLE mydb.mytable ENGINE=MyISAM; ANALYZE TABLE mydb.mytable;
These commands are mechanically identical. OPTIMIZE TABLE performs a defrag of the MyISAM table and then runs ANALYZE TABLE to compute fresh index statistics.
Mechanically speaking, this is what ALTER TABLE mydb.mytable ENGINE=MyISAM; does:
CREATE TABLE mydb.mytabletmp LIKE mydb.mytable;
INSERT INTO mydb.mytabletmp SELECT * FROM mydb.mytable;
ALTER TABLE mydb.mytable RENAME mydb.mytablezap;
ALTER TABLE mydb.mytabletmp RENAME mydb.mytable;
DROP TABLE mydb.mytablezap;

As mentioned MyISAM shouldn't be using ibdata.
What are your innodb settings in your my.cnf, or in the MySQL shell type:
SHOW VARIABLES LIKE "%innodb%";
Are these variables set?
innodb_data_home_dir
innodb_data_file_path
If you are not using INNODB at all you should be able to safely remove the ibdata and ib_logfile(s) and restart MySQL. Normally though removing these without dropping the tables first will cause issues.
See How to shrink/purge ibdata1 file in MySQL
If you do have a MyISAM table over 5Gigs, it's recommended that you use INNODB anyway. (Anything over 4 Gigs).
To troubleshoot you could try and add
skip-innodb
in the my.cnf if you are not using INNODB at all.

Related

MySQL defragment InnoDB tables

After running mysqltuner, I received this output:
Total fragmented tables: 284
To defragment I tried these:
1. ALTER TABLE tbl_name ENGINE=INNODB
2. Another way to perform a defragmentation operation is to use mysqldump to dump the table to a text file, drop the table, and reload it from the dump file.
Both of the methods are described in MySQL documentation.
But none of the methods helped. Mysqltuner still says that tables are fragmented.
Note: innodb_file_per_table is OFF in my configurations.
How can I defragment my InnoDB tables?
ALTER TABLE tbl_name ENGINE=INNODB
Is the only way to defragment a innodb table.
ALGORITHM=INPLACE can help us to do it online.
As of MySQL 5.5.11, you can also use ALTER TABLE tbl_name FORCE
to perform a “null” alter operation that rebuilds the table.
Previously the FORCE option was recognized but ignored.
https://dev.mysql.com/doc/refman/5.5/en/alter-table.html
I had the same issue and the solution that worked for me was doing analyze after alter:
alter table xyz engine=InnoDB;
analyze table xyz;

Innodb table with many deletes and inserts - is there any disk space wasted?

I have a big (500GB) innodb table that I'm performing many inserts as well as deletes on.
innodb_file_per_table is set in my var configuration. I know it might contain a lot of 'wasted' disk space, due to the nature of innodb, question is how much is expected and if there is a way to find out that number in my specific table. From what I understand it re-uses the not-used space that is created during delete so if I keep doing inserts I shouldn't experience any major diff when performing a re-claim procedure? As a thumb rule, what are the high and low limits for the disk space waste and what usage patterns define it?
If you have innodb_file_per_table enabled, I have some good news. You can do a lot more than defrag the table. You can shrink the table. In order to find out how much space you will reclaim follow this procedure:
For these steps, let's use the InnoDB table mydb.mytable
GIVEN_DB=mydb
GIVEN_TB=mytable
MYSQL_CONN="-root -p...."
SQLSTMT="SELECT data_length+index_length FROM information_schema.tables WHERE"
SQLSTMT="${SQLSTMT} table_schema='${GIVEN_DB}' AND table_name='${GIVEN_TB}'"
IBDSIZE=`ls -l /var/lib/mysql/${GIVEN_DB}/${GIVEN_TB}.ibd | awk '{print $5}'`
TBLSIZE=`mysql ${MYSQL_CONN} --skip-column-names -Ae"${SQLSTMT}"`
(( FRGSIZE = IBDSIZE - TBLSIZE ))
FRAGPCT=`echo ${FRGSIZE}00/${IBDSIZE}|bc`
echo ${IBDSIZE} ${TBLSIZE} ${FRGSIZE} ${FRAGPCT}
This will print 4 numbers
IBDSIZE : Size of InnoDB Individual Tablespace from the OS Viewpoint
TBLSIZE : Size of InnoDB Table from the INFORMATION_SCHEMA Database
FRGSIZE : Difference of IBDSIZE and TBLSIZE (in Bytes)
FRAGPCT : Percentage Difference of IBDSIZE and TBLSIZE
If you do not like how high FRAGPCT is, there are three ways to shrink mydb.mytable:
OPTION 1 : OPTIMIZE TABLE mydb.mytable;
OPTION 2 : ALTER TABLE mydb.mytable ENGINE=InnoDB;
OPTION 3 : Run the Compression in Stages
CREATE TABLE mydb.mytabletmp LIKE mydb.mytable;
INSERT INTO mydb.mytabletmp SELECT * FROM mydb.mytable;
DROP TABLE mydb.mytable;
ALTER TABLE mydb.mytabletmp RENAME mydb.mytable;
Give it a Try !!!
MySQL / InnoDB provides no function to 'free up' space, but it will re-use the space from the deleted rows for future insert statements.
Running OPTIMIZE TABLE (more information) periodically will help to maximize the amount you can re-use as well as increase I/O performance (it's basically like defrag for your database).

If i change mysql engine from Myisam to innodb, will it affect on my data

I am new to Mysql .
will it affect my data on server if i change mysql engine from Myisam to innodb.
Thanks
Changing engine from MyISAM to INNODB should not affect your data, but safe side you can easily take backup of your table before changine engine.
Taking backup:
CREATE TABLE backup_table LIKE your_table;
INSERT INTO backup_table SELECT * FROM your_table;
It may affect the performance of your queries. You need to configure Innodb specific System variables. e.g.
innodb_buffer_pool_size = 16G
innodb_additional_mem_pool_size = 2G
innodb_log_file_size = 1G
innodb_log_buffer_size = 8M
Changing engine to INNODB:
ALTER TABLE table_name ENGINE=INNODB;
I found two caveats when converting MyISAM tables to InnoDB: row size and support for full-text indexes
I ran into an issue converting some tables from an off-the-shelf application from MyISAM to InnoDB because of the maximum record size. MyISAM supports longer rows than InnoDB does. The maximum in a default InnoDB installation is about 8000 bytes. You can work around this by TRUNCATE'ing a table that fails, but this will bite you later on the INSERT. You might have to break your data up into multiple tables or restructure it with variable length column types such as TEXT (which can be slower).
A stock Innodb installation doesn't support FULLTEXT indexes. This may or may not impact your application. For one application's table I was converting, we decided to look in other fields for the data we needed rather than doing full text scans. (I did an "ALTER TABLE DROP INDEX..." on it to remove the FULLTEXT index before converting to InnoDB.) I wouldn't recommend full-text indexes for a write-heavy table anyway.
If converting a big table full of data with "ALTER TABLE..." works on the first try, you're probably okay.
http://dev.mysql.com/doc/refman/5.5/en/innodb-restrictions.html
Lastly, if you want to convert from MyISAM to InnoDB on a running system that is read heavy (where UPDATEs and INSERTs are rare), you can run this without interrupting your users. (The RENAME runs atomically. It will back out all of the changes if any of them don't work.)
CREATE TABLE mytable_ind AS SELECT * FROM mytable;
ALTER TABLE mytable_ind ENGINE=InnoDB;
RENAME TABLE mytable TO mytable_myi, mytable_ind TO mytable;
DROP TABLE mytable_myi;

MySQL default database and how to make existing non-InnoDB become InnoDB

I am in the beginning stages of a project and I have so far been using the default MySQL database.
By the way, does the default database have name?
My question is how I can change the existing tables to be utf-8 and InnoDB without deleting the current ones and making new tables. Is there an alter table to make the table utf-8 and InnoDB?
Thanks,
Alex
MyISAM is the default Storage Engine for MySQL (until 5.5.5, at which point InnoDB became the default). There is no concept of a default database.
To make an existing table use InnoDB, use the following:
ALTER TABLE tbl_name ENGINE = InnoDB;
To change the character set of an existing table to utf8, use the following:
ALTER TABLE tbl_name CONVERT TO CHARACTER SET utf8;
If you want to play it safe with changing a table to InnoDB, I have a nice suggestion:
Create a new table as InnoDB and load it
For this example you have a MyISAM table called mytable in the mydb database. You can perform the following:
use mydb
CREATE TABLE mytable_innodb LIKE mytable;
ALTER TABLE mytable_innodb ENGINE=InnoDB;
INSERT INTO mytable_innodb SELECT * FROM mytable;
ALTER TABLE mytable RENAME mytable_myisam;
ALTER TABLE mytable_innodb RENAME mytable;
That's it. The new table is InnoDB plus you have a backup of the original table in MyISAM with its original contents and layout. You are free to perform whatever other conversions you need on the new InnoDB table.
CAVEAT
Make sure you optimize InnoDB
InnoDB performance tweaks
Howto: Clean a mysql InnoDB storage engine?
How to Safely Change InnoDB Log File Size
Major Differences Between InnoDB and MyISAM

Sql : Side effects of creating/deleting databases and tables on the fly

I need to simulate sql by creating a wrapper over mysql(customer requirement :P), and hence my application requires to create/drop tables(and possibly databases) during runtime.
The frequency of such create/drop operations will not be very high. I'm not a database expert, but I believe that such operations could lead to some side-effects over long term.
Is it advisable to do go ahead with these creation/deletion of databases and what are the possible complications I can run into?
This is only a problem under two scenarios
SCENARIO #1
For InnoDB tables, the innodb buffer pool should be optimally set to the sum of all data pags and index pages that make up InnoDB tables.
Even worse can be that innodb_file_per_table is disabled (default)
This will produce a file called /var/lib/mysql/ibdata1 which can grow and never shrink. This is true no matter how many times you drop and create databases.
If one forgets to make the necessary changes in /etc/my.cnf, this could also expose innodb buffer pool to under-utilization until the data fills back up.
Changes to make for InnoDB are straightforward.
Run this query
SELECT CONCAT(KeyBuf,'M') BufferPoolSetting FROM (SELECT CEILING(SumInnoDB/POWER(1024,2)) KeyBuf FROM (SELECT SUM(data_length+index_length) SumInnoDB FROM information_schema.tables WHERE engine='InnoDB' and table_schema NOT IN ('information_schema','mysql')) A) AA;
The output of this query should be used as the innodb_buffer_pool_size in /etc/my.cnf just before you drop all databases and create new ones.
SCENARIO #2
For MyISAM tables, the key buffer should be optimally set to the sum of all .MYI files.
If one forgets to make the necessary changes in /etc/my.cnf, this could also expose MyISAM key cache (key buffer) to under-utilization until the data fills back up.
Changes to make for MyISAM are straightforward.
Run this query
SELECT CONCAT(KeyBuf,'M') KeyBufferSetting FROM (SELECT CEILING(SumIndexes/POWER(1024,2)) KeyBuf FROM (SELECT SUM(index_length) SumIndexes FROM information_schema.tables WHERE engine='MyISAM' and table_schema NOT IN ('information_schema','mysql')) A) AA;
The output of this query should be used as the key_buffer_size in /etc/my.cnf just before you drop all databases and create new ones.