Currently I have 3 servers running, 2 remote, Main server running MySQL 5.5.24 and backup running MySQL 5.1.63 and 1 local running MySQL 5.5.20 for development. The main server is used for reporting purposes; currently have 144 tables (MyISAM and InnoDB), and about 80gb of data, the larger tables have about 13 million rows each one.
On the remote servers I have very limited access through webmin, because of this I can’t make the changes directly, I need to ask to IT department for them.
I’m having some serious performance issues on the Main server since a MySQL crash we had last month. The server was running a development release which had some disk writing bugs, after the recovery we installed the MySQL 5.5.23 version and that’s when the performance issues began. We recently installed MySQL 5.5.24 (lastest stable version) but we still have the performance issues
Here are the servers configurations, all are dedicated servers:
Main:
Operating system: Gentoo Linux
Kernel and CPU: Linux 2.6.31-gentoo-r6 on x86_64
Processor information: Intel(R) Xeon(TM) CPU 2.66GHz, 8 cores
Real memory (ram): 23.55 GB total
Virtual memory: 3.74 GB total
Local disk space: 463.38 GB total (about 30% free)
Backup:
Operating system: Gentoo Linux
Kernel and CPU: Linux 2.6.31-gentoo-r6 on x86_64
Processor information: Intel(R) Core(TM)2 Duo CPU E6405 # 2.13GHz, 2 cores
Real memory (ram): 15.68 GB total
Virtual memory: 16.01 GB total
Local disk space: 4.73 TB total (about 80% free)
Development:
Operating system: Ubuntu Linux 9.04
Kernel and CPU: Linux 2.6.32-33-server on x86_64
Processor information: Intel(R) Core(TM)2 Duo CPU E6550 # 2.33GHz, 2 cores
Real memory (ram): 1.95 GB total
Virtual memory: 5.65 GB total
Local disk space: 141.15 GB total (about 5% free)
All the servers have the same my.cnf configuration, except for the innodb_buffer_pool_size
my.cnf
[mysqld]
back_log = 50
max_connections = 100
max_connect_errors = 10
table_open_cache = 2048
max_allowed_packet = 16M
binlog_cache_size = 1M
max_heap_table_size = 64M
read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M
thread_cache_size = 8
thread_concurrency = 8
query_cache_size = 64M
query_cache_limit = 2M
ft_min_word_len = 4
default-storage-engine = MYISAM
thread_stack = 262K
stored_program_cache= 1024
transaction_isolation = REPEATABLE-READ
tmp_table_size = 64M
binlog_format=mixed
slow_query_log
long_query_time = 2
key_buffer_size = 512M
bulk_insert_buffer_size = 64M
myisam_sort_buffer_size = 128M
myisam_max_sort_file_size = 10G
myisam_repair_threads = 1
myisam_recover
innodb_additional_mem_pool_size = 16M
innodb_buffer_pool_size = 8G
innodb_data_file_path = ibdata1:10M:autoextend
innodb_write_io_threads = 8
innodb_read_io_threads = 8
innodb_thread_concurrency = 16
innodb_flush_log_at_trx_commit = 1
innodb_log_buffer_size = 8M
innodb_log_file_size = 256M
innodb_log_files_in_group = 3
innodb_max_dirty_pages_pct = 90
innodb_lock_wait_timeout = 240
server-id = 1
event_scheduler = 1
[mysqldump]
quick
max_allowed_packet = 16M
[mysql]
no-auto-rehash
[myisamchk]
key_buffer_size = 512M
sort_buffer_size = 512M
read_buffer = 8M
write_buffer = 8M
[mysqlhotcopy]
interactive-timeout
[mysqld_safe]
open-files-limit = 8192
I’ve tried several large Stored Procedures with lots of columns and data and the backup server is about 10x faster than the main server, and running them on development server its about 5x faster than main.
Since the main server and the backup server are on the same remote location and through a VPN I know there are no connection speed issues.
I can’t post the Stored Procedures code, but I know the issue is on the final SELECT since the creation of all temporary tables are pretty fast, and I can see in the MySQL monitor (using SHOW FULL PROCESSLIST) that the SELECT is taking 95% of the time. Most of the Stored Procedures works with tons of data, with multiples joins.
Any Ideas why the main server could be having so bad performance even it’s the most powerful?
We've been looking for reported performance bugs of the latest versions, but what we found didn't help at all http://bugs.mysql.com/bug.php?id=44585
I'll really appreciate any help
Would this question better be asked on server fault? It's my first question
Sorry about my english.
Are the same tables involved in all of the problematic SELECT statements in the Stored Procedures you tried? Perhaps the index statistics need to be rebuilt on some of them - check out the OPTIMIZE TABLE command.
Finally we moved the DB to another server, new version of MySQL
Related
I'm running a distributed app in the cloud where clients keep open multiple long-running connections to a MySQL (Percona) server.
After updating to the latest Percona release, the maximum number of simultaneously connected clients has dropped significantly. In the past, with the previous Percona version, it was used successfully with up to 8K connections, now it struggles to get above 3K.
By "struggles" I mean that when the connection bottleneck has been reached, connecting to MySQL times out even in the command line. When I do manage to connect, show processlist doesn't show any hanging queries or locks. So the connections that are established work just fine. The Apache webserver serving the PHP webapp used to monitor the project also hangs indefinitely.
Here's the my.cnf I use, patched together from various guides.
[mysqld]
open_files_limit = 16384
table_open_cache = 16384
character_set_server = utf8mb4
max_connections = 16384
expire_logs_days = 10
max_binlog_size = 100M
innodb_open_files = 16384
innodb_file_per_table = 1
innodb_read_io_threads = 4
innodb_write_io_threads = 4
innodb_thread_concurrency = 0
innodb_log_file_size = 128M
innodb_open_files = 4000
innodb_flush_method = O_DIRECT
innodb_buffer_pool_instances = 1
thread_pool_size = 16
local_infile = 1
skip-name-resolve
thread_cache_size = 16384
thread_handling = pool-of-threads
innodb_buffer_pool_size = 512M
innodb_buffer_pool_instances = 1
innodb_log_buffer_size = 64M
And the Percona version I'm running:
mysqld --version
Ver 8.0.15-6 for debian-linux-gnu on x86_64 (Percona Server (GPL), Release '6', Revision '63abd08')
OS: Ubuntu 18.04.2 LTS
Hardware:
CPU: Intel(R) Xeon(R) Platinum 8175M CPU # 2.50GHz
RAM: 32G
The queries I'm running are all based on unique indexes (no joins), tables are all InnoDB. The same configuration worked fine until the update to Percona 8.
The my.cnf settings I had to remove after the upgrade as they were no longer supported are:
innodb_locks_unsafe_for_binlog = 0
query_cache_size = 0
query_cache_type = 0
Things I tried:
I ran mysqltuner.pl but didn't get any relevant advice from it.
ulimit -n and ulimit -s show values significantly higher than what I'm using (1048576 and 16384 specifically).
Any other suggestions?
I am on linux box running only mysql with 8 cores and 16GB ram. All connections come from web server on another machine in the same network running php with codeigniter.
I cannot get more than 150 connections on mysql.
My my.cnf is:
[mysqld]
user=mysql
port = 3306
socket = /tmp/mysql.sock
datadir = /usr/local/mysql/var/
skip-external-locking
max_connections=500
max_user_connections=500
open-files-limit = 500
key_buffer_size = 2048M
max_allowed_packet = 32M
table_open_cache = 512
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 16M
myisam_sort_buffer_size = 128M
thread_cache_size = 250
table_definition_cache = 1024
query_cache_size = 32M
query_cache_limit = 32M
table_cache=1024
max_heap_table_size=1024M
key_buffer=2048M
wait_timeout=60
thread_concurrency = 16
long_query_time = 1
tmp_table_size=256M
show status returns that max_connections and max_user_connections to be 500.
Since mysql is saying that connection limit is 500, I thought there are other setting in PHP, apache or codeigniter that is limiting the requests to mysql, but I cannot find any. I've searched google for few days trying to find answers without any luck.
Are there limits set on any of the above mentioned software? I will post configs if necessary.
Thank you.
Check the Max number of Apache worker processes (ServerLimit and MaxClients) in httpd.conf. Assuming a fixed number of connections per worker, you might be maxing out your number of workers, so nothing is requesting new MySQL connections.
Sorry, just found this on Google, but your config is foobar
max_connections=500
open-files-limit = 500
table_open_cache = 512
table_definition_cache = 1024
open-files-limit: 500 #; not good!
table_open_cache: 512... * 2+ open-files per table = 1024 open-files used minimum
max_connections: 500 each open connection starts a new file handle to buffer.
Your mysql will open some 130 tables just to start (core tables) which leaves you with a mere 240 file handles to share between data queries and connections. For each connection with a table query, 3+ file handles are consumed (connection, data table, index file(s)). That maxes your open-files long before you get 150 connections. open-files-limit needs to start >2048 for that kind of DB usage.
more help:
show global status like '%open%';
show global status like '%onnect%';
see for yourself how many files/connections are actually in use. some operating systems (Windows XP) hard limit files/network connections. try googling "mysql 'YOUR-OS' 150 connections" and see if that is the limiting factor.
I have a Xeon 2.0Ghz server (12 cores) with 16GB memory, running Apache and mySQL for a website with around 50,000 records in InnoDB (Percona). My queries used to return in about 0.17 to 0.25 seconds, then I ran the Percona tools mySQL optimizer, uploaded the new my.cnf file and suddenly the same queries are taking 1.20 to 1.30 seconds, so about 5x longer.
What did I do wrong? Here are my old and new my.cnf files"
NEW:
[mysqld]
default_storage_engine = InnoDB
key_buffer_size = 32M
myisam_recover = FORCE,BACKUP
max_allowed_packet = 16M
max_connect_errors = 1000000
log_bin = /var/lib/mysql/mysql-bin
expire_logs_days = 14
sync_binlog = 1
tmp_table_size = 32M
max_heap_table_size = 32M
query_cache_type = 0
query_cache_size = 0
max_connections = 200
thread_cache_size = 50
open_files_limit = 65535
table_definition_cache = 1024
table_open_cache = 2048
innodb_flush_method = O_DIRECT
innodb_log_files_in_group = 2
innodb_log_file_size = 256M
innodb_flush_log_at_trx_commit = 1
innodb_file_per_table = 1
innodb_buffer_pool_size = 12G
log_error = /var/lib/mysql/mysql-error.log
log_queries_not_using_indexes = 1
slow_query_log = 1
slow_query_log_file = /var/lib/mysql/mysql-slow.log
OLD:
[mysqld]
innodb_buffer_pool_size = 12000M
innodb_log_file_size = 256M
innodb_flush_method = O_DIRECT
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 16M
innodb_additional_mem_pool_size = 20M
innodb_thread_concurrency = 20
read_rnd_buffer_size=50M
query_cache_size=128M
query_cache_type=1
tmp_table_size=512M
wait_timeout=90
query_cache_limit=64M
key_buffer_size=128M
max_heap_table_size=512M
max_allowed_packet=32M
log_slow_queries
log-queries-not-using-indexes
long_query_time = 1
Are you swapping at all after running for a while?
You might try turning down your innodb_buffer_pool_size since you say the server is also running Apache. At the moment it looks like MySQL has the potential to use up all the server's memory for itself and leave nothing for the OS and Apache.
Try setting innodb_buffer_pool_size to 8G and then set innodb_log_file_size to 2G.
You can probably up your innodb_thread_concurrency as well, but since it isn't a dedicated MySQL server it may be fine at the default of 8. It depends on what CPU you have but the docs say:
The correct value for this variable is dependent on environment and
workload. You will need to try a range of different values to
determine what value works for your applications. A recommended value
is 2 times the number of CPUs plus the number of disks.
So play around with that and see what works best.
Also, is your database larger than the amount of RAM you have or could your entire DB fit in memory?
Just keep in mind that since you are running Apache on the same server, Apache is going to want to create a bunch of its own threads and consume as much memory as required for all the server processes and if you're running something like PHP that's going to take up memory as well.
You're going to have to find a good balance where both Apache and MySQL can both perform at maximum capacity on the same system but where neither one uses so much memory that the other has to swap.
Additional ways you can troubleshoot or profile performance would be to check your slow query log and run explains on the slow queries. In addition, you can install the Percona toolkit and run pt-query-digest to analyze your performance. Read the docs here.
Days of searching but still no solution.
My situation
I am using this command to dump a 16+Gb (growing) database (all but 2 tables are InnoDB)
The dump is run via cron once a day and sits in a small perl script
mysqldump -uuser -pxxx --single-transaction --quick DBNAME > DBBACKUP.sql
Now the website has 2 main tables that are very heavy read/write (low load when the dump is actually happening in the morning however)
The DB has 54 tables (most under 10-20 MB)
Table 1 is 71 Million rows (9GB) InnoDB table
Table 2 is 25 million rows (3GB) InnoDB table
Now when the dump runs, the application (whist accessing Table 1 or 2) is fine, until around 3 minutes into the dump when I expect Table 1 starts to backup, the website when submitting data to be inserted into Table 1 hangs for around 9 minutes, then works again, and then Table 2 starts to be dumped and the application that uses Table 2 starts to hang.
--single-transaction should not lock tables
--quick should not buffer in memory
Lastly, the CPU and MEM %'s are very low while the dump is happening.
So has anyone seen this and know of a solution or have any suggestions at all?
SYSTEM SPECS
Processor: Dual Intel Xeon E5506 Quad Core
RAM: 12 Gb DDR3
HD1: 4 147 Gb SAS drives in a RAID 10 array
HD2: 500 Gb SATA Backup Drive
OS: Linux OS - CentOS 5 64-bit
MY.CNF
max_connections = 500
safe-show-database
skip-locking
key_buffer = 128M
max_allowed_packet = 16M
table_cache = 256
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 2M
myisam_sort_buffer_size = 64M
thread_cache_size = 8
query_cache_size= 64M
thread_concurrency = 8
wait_timeout = 30
innodb_file_per_table
innodb_log_file_size = 10485760
open_files_limit = 8192
ft_min_word_len = 3
log-error=/var/log/mysql/mysql-error.log
log-slow-queries = /var/log/mysql/mysql-slow.log
long_query_time = 7
[mysqldump]
quick
max_allowed_packet = 16M
[mysql]
no-auto-rehash
[isamchk]
key_buffer = 128M
sort_buffer_size = 128M
read_buffer = 2M
write_buffer = 2M
[myisamchk]
key_buffer = 128M
sort_buffer_size = 128M
read_buffer = 2M
write_buffer = 2M
Thank You.
OK Looks like I found the issue.
Basically, my DB has 4 myiisam table and around 20 innodb tables, the myissam table are under 2 Mb in size (total) but were causing the locks :(
So I am lucky enough to be able to dump the innodb tables (as above excluding the myissam tables) separately (and keep consistency) from the myissam tables, and no more locking :)
If anybody has other solutions feel free to post.
As mentioned here you could setup replication (master/slave) and backup the slave (with or without locking) and have no fear of changes, thus getting around the locking issue.
You can find a guide to setup replication here:
http://dev.mysql.com/doc/refman/5.0/en/replication-howto.html
I recently got a new dedicated MySQL machine. Now it's running fine, but sometimes it gets slowed down a lot by queries that state: Copying to tmp table. It seems to happen randomly.
The machine has 12GB of DDR3 ram, and runs in a RAID10 setup with (4x 15k RPM SAS drives).
This machine hosts 5 databases, all between 1 and 8gb in size each. Reads / Writes: 66% / 34%
Below is my my.cnf file. If anyone has performance optimization tips, I would love to hear them.
[mysqld]
skip-name-resolve
datadir=/var/lib/mysql
#socket=/tmp/mysql.sock
log-error=/var/log/mysqld.log
old_passwords=0
max_connections = 1500
table_cache = 1024
max_allowed_packet = 16M
sort_buffer_size = 2M
thread_cache = 8
thread_concurrency = 32
query_cache_size = 0M
query_cache_type = 0
default-storage-engine = innodb
transaction_isolation = REPEATABLE-READ
tmp_table_size = 256M
long_query_time = 3
log_slow_queries = 1
innodb_additional_mem_pool_size=48M
innodb_flush_log_at_trx_commit=2
innodb_log_buffer_size=32M
innodb_buffer_pool_size=6G
innodb_autoinc_lock_mode=2
innodb_io_capacity=500
innodb_read_io_threads=16
innodb_write_io_threads=8
innodb_buffer_pool_size = 5000M
innodb_lock_wait_timeout = 300
innodb_max_dirty_pages_pct = 90
innodb_thread_concurrency =32
It seems I have found the solution myself. max_heap_table_size wasnt set. this limited tmp_table_size. i have now set both values on 512m.