I've searched a lot, it's said that I have to edit my.cnf file to change the stopword list.
I renamed my-medium.cnf to my.cnf and added the ft_query_expansion_limit and ft_stopword_file conditions. I have restarted mySQL. But it is not taking effect. I dont have admin privileges.
# The MySQL server
[mysqld]
port = 3306
socket = /tmp/mysql.sock
skip-external-locking
key_buffer_size = 16M
max_allowed_packet = 1M
table_open_cache = 64
sort_buffer_size = 512K
net_buffer_length = 8K
read_buffer_size = 256K
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M
ft_query_expansion_limit = 10
ft_stopword_file = 'D:/mysql/stopword.txt'
mysql> show variables like '%ft%';
+--------------------------+----------------+
| Variable_name | Value |
+--------------------------+----------------+
| ft_boolean_syntax | + -><()~*:""&| |
| ft_max_word_len | 84 |
| ft_min_word_len | 4 |
| ft_query_expansion_limit | 20 |
| ft_stopword_file | (built-in) |
+--------------------------+----------------+
5 rows in set (0.00 sec)
What can I do to modify the stopword list?
In my.ini text file (MySQL) :
ft_stopword_file = "" or link an empty file "empty_stopwords.txt"
ft_min_word_len = 2
// set your minimum length, but be aware that shorter words (3,2) will increase the query time dramatically, especially if the fulltext indexed column fields are large.
Save the file, restart the server.
The next step should be to repair the indexes with this query:
REPAIR TABLE tbl_name QUICK.
However, this will not work if you table is using InnoDB storage engine. You will have to change it to MyISAM :
ALTER TABLE t1 ENGINE = MyISAM;
So, once again:
1. Edit my.ini file and save
2. Restart your server (this cannot be done dynamically)
3. Change the table engine (if needed) ALTER TABLE tbl_name ENGINE = MyISAM;
4. Perform repair REPAIR TABLE tbl_name QUICK.
Be aware that InnoDB and MyISAM have their speed differences. One read faster, other writes faster ( read more about that on the internet )
Related
I developed an application connected to mariaDB (Ver 15.1 Distrib 10.1.31-MariaDB, for Win32) in DelphiXE8.
I want to improve query performance.
Describe the simplified scenario:
de_User Table (innoDB) (rows 81762)
ID_U INT PRIMARY KEY
Name VARCHAR(30)
INDEX ID_U, Name
de_doc Table (innoDB) (rows 260452)
IDD INT PRIMARY KEY
DataFi Date
UserID INT
...
INDEX IDD, UserID, DataFi
----
CONSTRAINT UserID_LK
FOREIGN KEY de_Doc (UserID)
REFERENCES de_User (ID_U)
ON DELETE CASCADE
ON UPDATE CASCADE
my query
select User.*, Doc.LastDoc
FROM de_Users AS Us
LEFT JOIN (
SELECT UserID,MAX(DataFi) AS LastDoc
FROM de_doc
GROUP BY UserID
) as Doc on Doc.UserID = Us.ID_U
ORDER BY Us.Name ASC, Doc.LastDoc DESC;
--
EXPLAIN select ...
+------+-------------+----------------+-------+---------------+---------------+---------+----------------+--------+---------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+----------------+-------+---------------+---------------+---------+----------------+--------+---------------------------------+
| 1 | PRIMARY | de_User | ALL | NULL | NULL | NULL | NULL | 81762 | Using temporary; Using filesort |
| 1 | PRIMARY | <derived2> | ref | key0 | key0 | 5 | Base.Us.ID_U | 10 | |
| 2 | DERIVED | de_Doc | index | NULL | UserID_LK| 4 | NULL | 260452 | |
+------+-------------+----------------+-------+---------------+---------------+---------+----------------+--------+---------------------------------+
my.ini
...
# The MySQL server
[mysqld]
...
key_buffer = 4096M
key_buffer_size=1024M
table_open_cache = 2048
query_cache_size = 128M
max_connections = 100
...
max_allowed_packet = 256M
sort_buffer_size = 4096M
net_buffer_length = 16M
read_buffer_size = 256M
myisam_sort_buffer_size = 256M
log_error = "mysql_error.log"
...
# Comment the following if you are using InnoDB tables
innodb_data_home_dir = "C:/xampp/mysql/data"
innodb_data_file_path = ibdata1:10M:autoextend
innodb_log_group_home_dir = "C:/xampp/mysql/data"
innodb_log_arch_dir = "C:/xampp/mysql/data"
## You can set .._buffer_pool_size up to 50 - 80 %
## of RAM but beware of setting memory usage too high
innodb_buffer_pool_size = 2048M
# DEPRECATED innodb_additional_mem_pool_size = 1024M
## Set .._log_file_size to 25 % of buffer pool size
innodb_log_file_size = 512M
innodb_log_buffer_size = 128M
innodb_flush_log_at_trx_commit = 1
innodb_lock_wait_timeout = 50
...
thread_concurrency = 4
...
[isamchk]
key_buffer = 1024M
sort_buffer_size = 256M
read_buffer = 8M
write_buffer = 16M
[myisamchk]
key_buffer = 1024M
sort_buffer_size = 256M
read_buffer = 8M
write_buffer = 8M
TEST phpmyadmin:
83705 total, the query employed 1,0000 sec.
if I remove "order by Doc.LastDoc DESC" it is very fast
83705 total, the query employed 0,0000 sec.
TEST in my application developed with delphiEX8
view table all rows 2,8 sec.
if I remove "order by Doc.LastDoc DESC" it is very fast
view table all rows 1,8 sec.
How can I improve performance?
Suggestions for your my.ini [mysqld] SECTION
sort_buffer_size=2M # from 4096M (4G) of RAM per connection, next 2 are per connect also
read_buffer_size=256K # from 256M to reduce volume of data retrieved by 99%
read_rnd_buffer_size=256K # from ? to a reasonable size
These three could be dynamically set (as root) with SET GLOBAL variable_name=value replace K with *1024 and M with *1024*1024 for Kbytes and Megabytes, please. Please post positive/negative results after a full BUSINESS DAY of uptime.
This is ambiguous: INDEX IDD, UserID, DataFi
Probably User.* was supposed to be Us.*? Be aware that "simplifying" a query may turn it into a different problem.
Probably LEFT JOIN is unnecessary; use JOIN.
You need this composite INDEX(UserID, LastDoc)
Do you really want 82K rows in the output? What will the client do with that much data? I ask because if the client will further digest the results, maybe such would be better done in SQL.
When timing, be sure to avoid the Query cache by using SELECT SQL_NO_CACHE.
phpmyadmin probably tacks on a LIMIT, thereby changing what the Optimizer will do!
ORDER BY t1.a, t2.b (different tables) makes it impossible to use an index for ordering. This will prevent any sort of short-circuiting of the query.
Changing these values in my.ini, in phpmyadmin here is the improved result.
The time it takes to populate the grid in my Delphi application, now 1.9 sec compared to before 2.8 sec.
my pc has 8Gb RAM;
Can I reduce the time to populate the grid in Delphi? Maybe I have to make a new request for this.
innodb_buffer_pool_size = 2048M
# Set .._log_file_size to 25 % of buffer pool size
BEFORE
innodb_log_file_size = 64M
(83705 del total, The query employed 1,0000 sec.)
AFTER
innodb_log_file_size = 512M
(83705 del total, The query employed 0,0000 sec.)
If your goal is "grouwise-max", then you left out a clause:
select User.*, Doc.LastDoc
FROM de_Users AS Us
LEFT JOIN
(
SELECT UserID,MAX(DataFi) AS LastDoc
FROM de_doc
GROUP BY UserID
) as Doc ON Doc.UserID = Us.ID_U
AND Doc.LastDoc = Us.DataFi -- this was missing
ORDER BY Us.Name ASC, Doc.LastDoc DESC;
That will also lead to many fewer rows being delivered, hence addressing the performance question.
Try this query and check whether the output is same as your query
select Us.*, max(Doc.DataFi) as LastDoc
FROM de_Users AS Us
LEFT JOIN de_doc as Doc on Doc.UserID = Us.ID_U
group by Us.ID_U
ORDER BY Us.Name ASC, LastDoc DESC;
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 3 years ago.
Improve this question
I have a MySQL with a few 100,000 tables. This is currently the best design for my system, since these tables are not related to each other and select queries will only be down on a single table. In addition, users will most likely not access the same tables very often.
I have 16GB RAM, but after about a day, MySQL consumes about 90% of that, with my total memory use of the system being at 99-100%. I tried numerous things, but simply can't get the memory usage down.
My innodb_buffer_pool_size is currently at 8GB, but I had it at 1G with the same issue. I also tried reducing open_files_limit but that didn't help either.
Here is my output for
SHOW GLOBAL STATUS LIKE '%Open_%';
+----------------------------+----------+
| Variable_name | Value |
+----------------------------+----------+
| Com_show_open_tables | 0 |
| Innodb_num_open_files | 431 |
| Open_files | 0 |
| Open_streams | 0 |
| Open_table_definitions | 615 |
| Open_tables | 416 |
| Opened_files | 4606655 |
| Opened_table_definitions | 4598528 |
| Opened_tables | 4661002 |
| Slave_open_temp_tables | 0 |
| Table_open_cache_hits | 30024782 |
| Table_open_cache_misses | 4661002 |
| Table_open_cache_overflows | 4660579 |
+----------------------------+----------+
And here is my mysqld config:
sql-mode=''
innodb_buffer_pool_size = 8G
open_files_limit=100000
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
bind-address = 127.0.0.1
key_buffer_size = 16M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 8
myisam-recover-options = BACKUP
query_cache_limit = 1M
query_cache_size = 16M
log_error = /var/log/mysql/error.log
expire_logs_days = 10
max_binlog_size = 100M
Anyone know how to efficiently handle these thousands of tables?
ADDITIONAL INFO
A) Mysqld: https://pastebin.com/PTiz6uRD
B) SHOW GLOBAL STATUS: https://pastebin.com/K4sCmvFz
C) SHOW GLOBAL VARIABLES: https://pastebin.com/Cc64BAUw
D) MySQLTuner: https://pastebin.com/zLzayi56
E) SHOW ENGINE INNODB STATUS: https://pastebin.com/hHDuw6gY
F) top: https://pastebin.com/6WYnSnPm
TEST AFTER SERVER REBOOT (LITTLE MEMORY CONSUMED):
A) ulimit -a: https://pastebin.com/FmPrAKHU
B) iostat -x: https://pastebin.com/L0G7H8s4
C) df -h: https://pastebin.com/d3EttR19
D) MySQLTuner: https://pastebin.com/T3DYDLg8
First block to remove will be in Linux command line,
ulimit -n 124000 to get past the current limit of 1024 Open Files
this can be done NOW, no shutdown/restart of Linux is required for this to be active.
Suggestions to consider for your my.cnf [mysqld] section
table_open_cache=10000 # from 431 available today to a practical upper limit
table_definition_cache=10000 # from 615 available today to a practical upper limit
thread_cache_size=100 # from 8 for V8 refman CAP suggested to avoid OOM
max_heap_table_size=64M # from 16M to reduce created_tmp_disk_tables
tmp_table_size=64M # from 16M should always be equal to max_heap_table_size
innodb_lru_scan_depth=100 # from 1024 to reduce CPU workload every SECOND
innodb_log_buffer_size=512M # from 50M to avoid log rotation every 7 minutes
Considering your situation, I would SKIP the one a day rule and monitor before
next Global Variable change. Make all cnf changes. Stop/start services required because you are limited to open_files_limit in MySQL even though you requested 100,000 ulimit caused runtime to limit you to 1024.
If you copy paste the block to the END of the MySQLD section,
remove same named variables above the block in MYSQLD section only,
you will get rid of the 'multiple variable confusion' for the next analyst.
Please view profile for contact information and get in touch.
I know there are plenty of information on this topic, but I think I've tried everything I can, and don't know any new ideas (if any?) about increasing my MySQL database performance.
Situation: I use eTesting platform (taotesting if anybody knows it). It uses MySQL database, with 8 tables. At this moment one of those tables has ~500k rows. Others are either empty or has ~10-15 rows. At first mysql performance was terrible using this platform, then I decided to convert MyISAM tables to InnoDB and make some my.cnf changes. This seemed to have improved performance, but not as much as I wanted.
Server has 1 CPU / 4 cores. 6 GB RAM. It's not dedicated to MySQL, it also hosts PHP/apache/nginx.
What's more, there are about 80% more selects from database then inserts/updates/deletes.
Any ideas how to further (if possible) improve mysql configuration are welcome .
Here's my.cnf:
#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html
# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
# Here is entries for some specific programs
# The following values assume you have at least 32M ram
# This was formally known as [safe_mysqld]. Both versions are currently parsed.
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
#
# * Basic Settings
#
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address = 127.0.0.1
#
# * Fine Tuning
#
key_buffer_size = 32M
max_allowed_packet = 16M
thread_stack = 256K
thread_cache_size = 50
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover = BACKUP
max_connections = 200
#table_cache = 1M
sort_buffer_size = 1M
read_buffer_size = 1M
join_buffer_size = 1M
#thread_concurrency = 8
max_heap_table_size = 64M
tmp_table_size = 64M
#
# * Query Cache Configuration
#
query_cache_limit = 2M
query_cache_size = 2M
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
#general_log_file = /var/log/mysql/mysql.log
#general_log = 1
#
# Error log - should be very few entries.
#
log_error = /var/log/mysql/error.log
#
# Here you can see queries with especially long duration
log_slow_queries = /var/log/mysql/mysql-slow.log
long_query_time = 1
#log-queries-not-using-indexes
#log = /var/log/mysql/testing_req_nec.log
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
# other settings you may need to change.
#server-id = 1
#log_bin = /var/log/mysql/mysql-bin.log
expire_logs_days = 10
max_binlog_size = 100M
#binlog_do_db = include_database_name
#binlog_ignore_db = include_database_name
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_log_file_size = 1G
innodb_log_buffer_size = 16M
innodb_buffer_pool_instances = 8
innodb_flush_log_at_trx_commit = 0
innodb_file_per_table = 1
innodb_read_io_threads = 64
innodb_write_io_threads = 64
innodb_additional_mem_pool_size = 16M
innodb_max_dirty_pages_pct = 90
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
[mysql]
#no-auto-rehash # faster start of mysql but no tab completition
[isamchk]
key_buffer_size = 16M
#
# * IMPORTANT: Additional settings that can override those from this file!
# The files must end with '.cnf', otherwise they'll be ignored.
#
#open_files_limit = 8192
!includedir /etc/mysql/conf.d/
EDIT:
Ok, I thought actual numbers were not needed since I said that performance I got wasn't enough for me. I'm using Jmeter to do performance testing. One example would be: Student can login, select test, submit his answers, and end test. I've tried doing this with ~30 students, and here's what I got:
Login: ~2 seconds (best case scenario), 8s (worst case)
Select test: ~5s - 7s (best case), ~20s - 25s worst case.
Submit answers to question: ~5 - 7s (best case), ~30s worst case.
Test ending, same as other submits.
See now I would like to have best case scenario for more students (more threads). Problem is that this eTesting platform doesn't use tradidional relational DB model (it uses RDF triples if you've heard) and stores them into MySQL tables. There are a lot of queries, one submit ~80 queries. Sof if test has ~15 items, 1 student sends ~2k queries.
And I've tried EXPLAIN. Can't really help since I can't change eTesting platform source code without breaking everything else, nor can I change table structure (besides changing it's engine, maybe some indexes?)
EDIT:
submit queries example:
http://codeviewer.org/view/code:3d10
tables structure:
http://codeviewer.org/view/code:3d11
mysql> show variables LIKE '%buffer_pool%';
+------------------------------+------------+
| Variable_name | Value |
+------------------------------+------------+
| innodb_buffer_pool_instances | 8 |
| innodb_buffer_pool_size | 4294967296 |
+------------------------------+------------+
2 rows in set (0.00 sec)
Explain on one of more complex queries:
EXPLAIN SELECT count(*) as count FROM statements WHERE (predicate = 'http://www.tao.lu/Ontologies/TAODelivery.rdf#DeliveryExecutionDelivery' AND (object = 'https://etestas.nec.lt/tao_ssl_dev.rdf#i139266227459751316')) AND subject IN (SELECT subject FROM statements WHERE (predicate = 'http://www.tao.lu/Ontologies/TAODelivery.rdf#DeliveryExecutionSubject' AND (object = 'https://etestas.nec.lt/tao_ssl_dev.rdf#i1392637693114892'))) AND subject IN (SELECT subject FROM statements WHERE predicate = 'http://www.w3.org/1999/02/22-rdf-syntax-ns#type' AND object in ('http://www.tao.lu/Ontologies/TAODelivery.rdf#DeliveryExecution'));
+----+--------------------+------------+----------------+---------------+------+---------+-------------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+--------------------+------------+----------------+---------------+------+---------+-------------+------+-------------+
| 1 | PRIMARY | statements | ref | k_po | k_po | 990 | const,const | 1 | Using where |
| 3 | DEPENDENT SUBQUERY | statements | index_subquery | k_sp,k_po | k_sp | 990 | func,const | 1 | Using where |
| 2 | DEPENDENT SUBQUERY | statements | index_subquery | k_sp,k_po | k_sp | 990 | func,const | 1 | Using where |
+----+--------------------+------------+----------------+---------------+------+---------+-------------+------+-------------+
You can check MySQL Tuner and Tuning Primer, both scripts can help you optimize your server configuration
The InnoDB performance measures varies according to Hardware,OS and also various factors.You can check out this for the detailed InnoDB Performance
For the past 4 days MySQL keeps crashing on running scripts, like once / day
this is the error log
key_buffer_size=134217728
read_buffer_size=1048576
max_used_connections=39
max_threads=100
threads_connected=34
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 336508 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
thd: 0x92025f38
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x95dce36c thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x2d) [0x6b65ad]
/usr/sbin/mysqld(handle_segfault+0x494) [0x3823d4]
[0x110400]
/usr/sbin/mysqld(MYSQLparse(void*)+0x6aa) [0x3b42da]
/usr/sbin/mysqld(mysql_parse(THD*, char const*, unsigned int, char const**)+0x23e) [0x39ce6e]
/usr/sbin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0xf35) [0x39df25]
/usr/sbin/mysqld(do_command(THD*)+0xf3) [0x39f0e3]
/usr/sbin/mysqld(handle_one_connection+0x2a0) [0x38dbd0]
/lib/tls/i686/cmov/libpthread.so.0(+0x596e) [0x93d96e]
/lib/tls/i686/cmov/libc.so.6(clone+0x5e) [0xd78a4e]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x86982ef4 is an invalid pointer
thd->thread_id=2906
thd->killed=NOT_KILLED
The box runs on 2GB RAM, by my calculations it shouldn't have the problem with max memory. I've specifically lowered the memory requirements to a minimum but still getting the errors.
mysql> show variables like "sort_buffer%";
+------------------+---------+
| Variable_name | Value |
+------------------+---------+
| sort_buffer_size | 1048576 |
+------------------+---------+
It crashed today on this SQL query
ALTER TABLE FieldDefaultValue MODIFY value_field varchar(2000) CHARACTER SET utf8 collate utf8_bin;
Anyone got any similar experience ?
EDIT:
The table in question actually doesn't contain much data, the database has much larger tables:
mysql> desc fielddefaultvalue;
+----------------------+---------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+----------------------+---------------+------+-----+---------+----------------+
| fielddefaultvalue_Id | bigint(20) | NO | PRI | NULL | auto_increment |
| version | bigint(20) | NO | | NULL | |
| value_field | varchar(2000) | YES | MUL | NULL | |
| optimistic_version | bigint(20) | NO | | NULL | |
| property_fk | bigint(20) | YES | MUL | NULL | |
| esg_fk | bigint(20) | YES | MUL | NULL | |
+----------------------+---------------+------+-----+---------+----------------+
6 rows in set (0.02 sec)
mysql> select count(*) from fielddefaultvalue;
+----------+
| count(*) |
+----------+
| 690 |
+----------+
1 row in set (0.00 sec)
It also fails on multiple inserts (400-500) of little data, but not all the time, the same script can run properly once or crash it
EDIT 2: After crash recovery the error log also reports:
InnoDB: which exceeds the log group capacity 9433498.
InnoDB: If you are using big BLOB or TEXT rows, you must set the
InnoDB: combined size of log files at least 10 times bigger than the
InnoDB: largest such row.
my.cnf
lower_case_table_names = 1
key_buffer = 16M
key_buffer_size = 128M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 8
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover = BACKUP
max_connections = 100
table_cache = 512
thread_concurrency = 4
sort_buffer_size = 1M
read_buffer_size = 1M
table_open_cache = 512
read_rnd_buffer_size = 8M
innodb_file_per_table = 1
open_files_limit = 65536
default_character_set=utf8
query_cache_limit = 1M
query_cache_size = 64M
expire_logs_days = 10
max_binlog_size = 250M
innodb_buffer_pool_size = 256M
innodb_additional_mem_pool_size = 20M
EDIT: 5 hours later
It just crashed again on the same "regular" script, it's a 25.000 line update script on a date column.
Same error message:
InnoDB: Log scan progressed past the checkpoint lsn 186 4056481576
110620 17:30:52 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Read
Funny thing is, I've ran this script today and didn't fail, but it did now.
The most likely explanation is running out of address space; please post your entire my.cnf.
Running 32-bit OS in production is not a good idea.
However, what you should do is:
Reproduce the fault on the same MySQL version on a non-production machine
Check that you are using a properly supported, current build from Oracle. If you are not, then install one of those and reproduce the problem. If you are running Redhat (or similar), then you can use Oracle's RPMs. They also provide some other distributions' packages and binaries in a tar.gz file. Your package vendor may patch MySQL with some dodgy patches. I never run OEM MySQL builds in production.
You seem to be running 32-bit. Ensure that you're not running out of address-space.
If you can reproduce the bug using a standard Oracle build on a supported operating system, you are not running out of memory / adress space and there is no hardware fault, then you can submit the bug to Oracle.
The best idea is to reproduce the test-case with the minimum amount of data / table size.
Sounds like your innodb_log_file_size is not big enough - try with 256 MB in my.cnf:
innodb_log_file_size=256M
You need to shut it down cleanly, remove the old logfiles, then restart - mysql will recreate new log files.
Strange... I don't know how optimized the ALTER TABLE actually is on MySQL. Perhaps it consumes a lot of power. If the table contains a lot of data, try to move all your data into a temporary table and empty the main one. Then do your alter table and push the data back. If it has to do work on each row then you can split the work up like this and do a few records at a time.
This is really maddening. I have followed every instruction for settings that I have found on the interwebs, and I can't get past this.
Basically, I have a table with about 8 million rows. I need to create a backup of this table like so:
create table mytable_backup like mytable
And that takes several hours on my production server, which is an Amazon EC2 instance running through EngineYard. It takes only minutes on my MacBook Pro. This is another one of those annoying things that MySQL does in the background, and you can't guess how it is making the decision to do something so stupidly slow.
BTW, there is over 330G available in the tmp directory, so that is not the issue.
But here is what "free -m" yields:
deploy#domU-12-31-39-02-35-31 ~ $ free -m
total used free shared buffers cached
Mem: 1740 1728 11 0 14 1354
-/+ buffers/cache: 359 1380
Swap: 895 2 893
I don't know how to read that, but the "11" under the free column doesn't look very good.
I am running:
Server version: 5.0.51-log Gentoo Linux mysql-community-5.0.51
Here is my configuration file:
# /etc/mysql/my.cnf: The global mysql configuration file.
# $Header: /var/cvsroot/gentoo-x86/dev-db/mysql/files/my.cnf-4.1,v 1.3 2006/05/05 19:51:40 chtekk Exp $
# The following options will be passed to all MySQL clients
[client]
port = 3306
[mysql]
character-sets-dir=/usr/share/mysql/charsets
default-character-set=utf8
[mysqladmin]
character-sets-dir=/usr/share/mysql/charsets
default-character-set=utf8
[mysqlcheck]
character-sets-dir=/usr/share/mysql/charsets
default-character-set=utf8
[mysqldump]
character-sets-dir=/usr/share/mysql/charsets
default-character-set=utf8
[mysqlimport]
character-sets-dir=/usr/share/mysql/charsets
default-character-set=utf8
[mysqlshow]
character-sets-dir=/usr/share/mysql/charsets
default-character-set=utf8
[myisamchk]
character-sets-dir=/usr/share/mysql/charsets
[myisampack]
character-sets-dir=/usr/share/mysql/charsets
[mysqld_safe]
err-log = /db/mysql/log/mysql.err
# To allow table cache to be raised
open-file-limit = 4096
[mysqld]
max_connections = 300
innodb_file_per_table = 1
log-slow-queries = /db/mysql/log/slow_query.log
long_query_time = 2000000
ft_min_word_len = 3
max_heap_table_size = 64M
tmp_table_size = 64M
server-id = 1
log-bin = /db/mysql/master-bin
log-bin-index = /db/mysql/master-bin.index
# END master/slave configuration
character-set-server = utf8
default-character-set = utf8
user = mysql
port = 3306
socket = /var/run/mysqld/mysqld.sock
pid-file = /var/run/mysqld/mysqld.pid
log-error = /db/mysql/log/mysqld.err
basedir = /usr
datadir = /db/mysql
key_buffer = 32M
max_allowed_packet = 32M
table_cache = 1024
thread_cache = 512
sort_buffer_size = 100M
net_buffer_length = 64K
read_buffer_size = 1M
read_rnd_buffer_size = 1M
myisam_sort_buffer_size = 100M
myisam_max_sort_file_size = 2G
myisam_repair_threads = 1
language = /usr/share/mysql/english
# security:
# using "localhost" in connects uses sockets by default
# skip-networking
# bind-address = 127.0.0.1
# point the following paths to different dedicated disks
tmpdir = /mnt/mysql/tmp
# log-update = /path-to-dedicated-directory/hostname
# you need the debug USE flag enabled to use the following directives,
# if needed, uncomment them, start the server and issue
# #tail -f /tmp/mysqld.sql /tmp/mysqld.trace
# this will show you *exactly* what's happening in your server ;)
#log = /tmp/mysqld.sql
#gdb
#debug = d:t:i:o,/tmp/mysqld.trace
#one-thread
# the rest of the innodb config follows:
# don't eat too much memory, we're trying to be safe on 64Mb boxes
# you might want to bump this up a bit on boxes with more RAM
innodb_buffer_pool_size = 1275M
# this is the default, increase it if you have lots of tables
innodb_additional_mem_pool_size = 16M
#
# i'd like to use /var/lib/mysql/innodb, but that is seen as a database :-(
# and upstream wants things to be under /var/lib/mysql/, so that's the route
# we have to take for the moment
#innodb_data_home_dir = /var/lib/mysql/
#innodb_log_arch_dir = /var/lib/mysql/
#innodb_log_group_home_dir = /var/lib/mysql/
# you may wish to change this size to be more suitable for your system
# the max is there to avoid run-away growth on your machine
innodb_data_file_path = ibdata1:20M:autoextend
# we keep this at around 25% of of innodb_buffer_pool_size
# sensible values range from 1MB to (1/innodb_log_files_in_group*innodb_buffer_pool_size)
innodb_log_file_size = 96M
# this is the default, increase it if you have very large transactions going on
innodb_log_buffer_size = 8M
# this is the default and won't hurt you
# you shouldn't need to tweak it
innodb_log_files_in_group = 2
# see the innodb config docs, the other options are not always safe
# This is not good for performance when used with bin_sync. Disabling.
innodb_flush_log_at_trx_commit = 2
innodb_flush_method = O_DIRECT
innodb_lock_wait_timeout = 50
query_cache_size = 16M
query_cache_type = 1
[mysqldump]
quick
max_allowed_packet = 16M
[mysql]
# uncomment the next directive if you are not familiar with SQL
#safe-updates
[isamchk]
key_buffer = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M
[myisamchk]
key_buffer = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M
ft_min_word_len = 3
[mysqlhotcopy]
interactive-timeout
For what it's worth, 11 megs free is perfectly fine. That's 11 megs of memory not being used for anything, and "wasted" as far as the hardware is concerned. The real number is the "1380" used in caches, PLUS the 11 megs unused. Caches can be blown away as necessary.
Your system has nearly 1400 MB of RAM available.
You could try
create table backup_table as (select * from production table) engine=myisam
This should create the table with only the data and none of the keys. You can then add the keys on by doing
alter table backup_table add index(column_name)
I've done this successfully several times, and it is usually a factor of 2 times faster than inserting with the keys in place.
You have to look your setting for myisam_max_sort_file_size and myisam_sort_buffer_size
If the sum of all the keys is less that myisam_max_sort_file_size, a sort will, in the worst case, land in a MyISAM table, which is a good thing.
Otherwise, it will revert to the keycache. That means loading the necessary .MYI index pages into the keycache and traversing those index pages in memory. Nobody wants that !!!!
Your current setting for this variable says 2G.
Look at the keys being built. Add them up. If the sum of all the key sizes exceed 2G, keycache all the way !!! You will have to up this value. You could up this value for the session to 4G with
SET myisam_max_sort_file_size = 1024 * 1024 * 1024 * 4;
SET myisam_sort_buffer_size = 1024 * 1024 * 1024 * 4;
or you could plant the number directly like this:
SET myisam_max_sort_file_size = 4294967296;
SET myisam_sort_buffer_size = 4294967296;
before doing the ENABLE KEYS;
If you are just interested in backing up the data, why index it to begin with ??? Try using the ARCHIVE storage engine. It has no indexing whatsoever. Do the following:
CREATE TABLE mytable_backup LIKE mytable;
ALTER TABLE mytable_backup ENGINE=ARCHIVE;
INSERT INTO mytable_backup SELECT * FROM mytable;
I also noticed you are using Amazon EC2. I have never been in EC2 before. Run this command:
SHOW ENGINES;
+------------+---------+----------------------------------------------------------------+--------------+------+------------+
| Engine | Support | Comment | Transactions | XA | Savepoints |
+------------+---------+----------------------------------------------------------------+--------------+------+------------+
| InnoDB | DEFAULT | Supports transactions, row-level locking, and foreign keys | YES | YES | YES |
| MRG_MYISAM | YES | Collection of identical MyISAM tables | NO | NO | NO |
| BLACKHOLE | YES | /dev/null storage engine (anything you write to it disappears) | NO | NO | NO |
| CSV | YES | CSV storage engine | NO | NO | NO |
| MEMORY | YES | Hash based, stored in memory, useful for temporary tables | NO | NO | NO |
| FEDERATED | YES | Federated MySQL storage engine | NO | NO | NO |
| ARCHIVE | YES | Archive storage engine | NO | NO | NO |
| MyISAM | YES | Default engine as of MySQL 3.23 with great performance | NO | NO | NO |
+------------+---------+----------------------------------------------------------------+--------------+------+------------+
If the ARCHIVE storage engine appears in the list and Support is Yes, you have the option to backup to an ARCHIVE table. If not, you must get the myisam_max_sort_file_size and myisam_sort_buffer_size adjusted.