How can I avoid "repair with keycache" in MySQL? - mysql

Question

This is really maddening. I have followed every instruction for settings that I have found on the interwebs, and I can't get past this.
Basically, I have a table with about 8 million rows. I need to create a backup of this table like so:
create table mytable_backup like mytable
And that takes several hours on my production server, which is an Amazon EC2 instance running through EngineYard. It takes only minutes on my MacBook Pro. This is another one of those annoying things that MySQL does in the background, and you can't guess how it is making the decision to do something so stupidly slow.
BTW, there is over 330G available in the tmp directory, so that is not the issue.
But here is what "free -m" yields:
deploy#domU-12-31-39-02-35-31 ~ $ free -m
total used free shared buffers cached
Mem: 1740 1728 11 0 14 1354
-/+ buffers/cache: 359 1380
Swap: 895 2 893
I don't know how to read that, but the "11" under the free column doesn't look very good.
I am running:
Server version: 5.0.51-log Gentoo Linux mysql-community-5.0.51
Here is my configuration file:
# /etc/mysql/my.cnf: The global mysql configuration file.
# $Header: /var/cvsroot/gentoo-x86/dev-db/mysql/files/my.cnf-4.1,v 1.3 2006/05/05 19:51:40 chtekk Exp $
# The following options will be passed to all MySQL clients
[client]
port = 3306
[mysql]
character-sets-dir=/usr/share/mysql/charsets
default-character-set=utf8
[mysqladmin]
character-sets-dir=/usr/share/mysql/charsets
default-character-set=utf8
[mysqlcheck]
character-sets-dir=/usr/share/mysql/charsets
default-character-set=utf8
[mysqldump]
character-sets-dir=/usr/share/mysql/charsets
default-character-set=utf8
[mysqlimport]
character-sets-dir=/usr/share/mysql/charsets
default-character-set=utf8
[mysqlshow]
character-sets-dir=/usr/share/mysql/charsets
default-character-set=utf8
[myisamchk]
character-sets-dir=/usr/share/mysql/charsets
[myisampack]
character-sets-dir=/usr/share/mysql/charsets
[mysqld_safe]
err-log = /db/mysql/log/mysql.err
# To allow table cache to be raised
open-file-limit = 4096
[mysqld]
max_connections = 300
innodb_file_per_table = 1
log-slow-queries = /db/mysql/log/slow_query.log
long_query_time = 2000000
ft_min_word_len = 3
max_heap_table_size = 64M
tmp_table_size = 64M
server-id = 1
log-bin = /db/mysql/master-bin
log-bin-index = /db/mysql/master-bin.index
# END master/slave configuration
character-set-server = utf8
default-character-set = utf8
user = mysql
port = 3306
socket = /var/run/mysqld/mysqld.sock
pid-file = /var/run/mysqld/mysqld.pid
log-error = /db/mysql/log/mysqld.err
basedir = /usr
datadir = /db/mysql
key_buffer = 32M
max_allowed_packet = 32M
table_cache = 1024
thread_cache = 512
sort_buffer_size = 100M
net_buffer_length = 64K
read_buffer_size = 1M
read_rnd_buffer_size = 1M
myisam_sort_buffer_size = 100M
myisam_max_sort_file_size = 2G
myisam_repair_threads = 1
language = /usr/share/mysql/english
# security:
# using "localhost" in connects uses sockets by default
# skip-networking
# bind-address = 127.0.0.1
# point the following paths to different dedicated disks
tmpdir = /mnt/mysql/tmp
# log-update = /path-to-dedicated-directory/hostname
# you need the debug USE flag enabled to use the following directives,
# if needed, uncomment them, start the server and issue
# #tail -f /tmp/mysqld.sql /tmp/mysqld.trace
# this will show you *exactly* what's happening in your server ;)
#log = /tmp/mysqld.sql
#gdb
#debug = d:t:i:o,/tmp/mysqld.trace
#one-thread
# the rest of the innodb config follows:
# don't eat too much memory, we're trying to be safe on 64Mb boxes
# you might want to bump this up a bit on boxes with more RAM
innodb_buffer_pool_size = 1275M
# this is the default, increase it if you have lots of tables
innodb_additional_mem_pool_size = 16M
#
# i'd like to use /var/lib/mysql/innodb, but that is seen as a database :-(
# and upstream wants things to be under /var/lib/mysql/, so that's the route
# we have to take for the moment
#innodb_data_home_dir = /var/lib/mysql/
#innodb_log_arch_dir = /var/lib/mysql/
#innodb_log_group_home_dir = /var/lib/mysql/
# you may wish to change this size to be more suitable for your system
# the max is there to avoid run-away growth on your machine
innodb_data_file_path = ibdata1:20M:autoextend
# we keep this at around 25% of of innodb_buffer_pool_size
# sensible values range from 1MB to (1/innodb_log_files_in_group*innodb_buffer_pool_size)
innodb_log_file_size = 96M
# this is the default, increase it if you have very large transactions going on
innodb_log_buffer_size = 8M
# this is the default and won't hurt you
# you shouldn't need to tweak it
innodb_log_files_in_group = 2
# see the innodb config docs, the other options are not always safe
# This is not good for performance when used with bin_sync. Disabling.
innodb_flush_log_at_trx_commit = 2
innodb_flush_method = O_DIRECT
innodb_lock_wait_timeout = 50
query_cache_size = 16M
query_cache_type = 1
[mysqldump]
quick
max_allowed_packet = 16M
[mysql]
# uncomment the next directive if you are not familiar with SQL
#safe-updates
[isamchk]
key_buffer = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M
[myisamchk]
key_buffer = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M
ft_min_word_len = 3
[mysqlhotcopy]
interactive-timeout

Answer 1

For what it's worth, 11 megs free is perfectly fine. That's 11 megs of memory not being used for anything, and "wasted" as far as the hardware is concerned. The real number is the "1380" used in caches, PLUS the 11 megs unused. Caches can be blown away as necessary.
Your system has nearly 1400 MB of RAM available.

Answer 2

You could try
create table backup_table as (select * from production table) engine=myisam
This should create the table with only the data and none of the keys. You can then add the keys on by doing
alter table backup_table add index(column_name)
I've done this successfully several times, and it is usually a factor of 2 times faster than inserting with the keys in place.

Answer 3

You have to look your setting for myisam_max_sort_file_size and myisam_sort_buffer_size
If the sum of all the keys is less that myisam_max_sort_file_size, a sort will, in the worst case, land in a MyISAM table, which is a good thing.
Otherwise, it will revert to the keycache. That means loading the necessary .MYI index pages into the keycache and traversing those index pages in memory. Nobody wants that !!!!
Your current setting for this variable says 2G.
Look at the keys being built. Add them up. If the sum of all the key sizes exceed 2G, keycache all the way !!! You will have to up this value. You could up this value for the session to 4G with
SET myisam_max_sort_file_size = 1024 * 1024 * 1024 * 4;
SET myisam_sort_buffer_size = 1024 * 1024 * 1024 * 4;
or you could plant the number directly like this:
SET myisam_max_sort_file_size = 4294967296;
SET myisam_sort_buffer_size = 4294967296;
before doing the ENABLE KEYS;
If you are just interested in backing up the data, why index it to begin with ??? Try using the ARCHIVE storage engine. It has no indexing whatsoever. Do the following:
CREATE TABLE mytable_backup LIKE mytable;
ALTER TABLE mytable_backup ENGINE=ARCHIVE;
INSERT INTO mytable_backup SELECT * FROM mytable;
I also noticed you are using Amazon EC2. I have never been in EC2 before. Run this command:
SHOW ENGINES;
+------------+---------+----------------------------------------------------------------+--------------+------+------------+
| Engine | Support | Comment | Transactions | XA | Savepoints |
+------------+---------+----------------------------------------------------------------+--------------+------+------------+
| InnoDB | DEFAULT | Supports transactions, row-level locking, and foreign keys | YES | YES | YES |
| MRG_MYISAM | YES | Collection of identical MyISAM tables | NO | NO | NO |
| BLACKHOLE | YES | /dev/null storage engine (anything you write to it disappears) | NO | NO | NO |
| CSV | YES | CSV storage engine | NO | NO | NO |
| MEMORY | YES | Hash based, stored in memory, useful for temporary tables | NO | NO | NO |
| FEDERATED | YES | Federated MySQL storage engine | NO | NO | NO |
| ARCHIVE | YES | Archive storage engine | NO | NO | NO |
| MyISAM | YES | Default engine as of MySQL 3.23 with great performance | NO | NO | NO |
+------------+---------+----------------------------------------------------------------+--------------+------+------------+
If the ARCHIVE storage engine appears in the list and Support is Yes, you have the option to backup to an ARCHIVE table. If not, you must get the myisam_max_sort_file_size and myisam_sort_buffer_size adjusted.