We are running multiple scripts that calculates data. The script was written in PHP(Laravel) and they were triggered by a cron(every minute). What I have noticed is when the cron triggers more than 300 of those process, mysql crashes "Connection Refused". This issue doesn't happen when there are less than 300 processes. I already have increased max_connections to 1000 and back_log t0 500 in my.cnf but the issue still persists(of course MySQL service was restarted). I have seen someone say that back_log is also at OS level but I can't find any article on how to adjust it. Any thoughts?
Here's some config values:
max_connections = 1000
back_log = 500
connect_timeout = 5
wait_timeout = 600
max_allowed_packet = 64M
thread_cache_size = 256
sort_buffer_size = 128M
bulk_insert_buffer_size = 128M
tmp_table_size = 128M
max_heap_table_size = 1G
myisam_recover_options = BACKUP
key_buffer_size = 128M
#open-files-limit = 2000
table_open_cache = 400
myisam_sort_buffer_size = 512M
concurrent_insert = 2
read_buffer_size = 8M
read_rnd_buffer_size = 4M
innodb_buffer_pool_size = 80G
innodb_log_buffer_size = 256M
innodb_file_per_table = 1
innodb_open_files = 400
innodb_io_capacity = 400
innodb_flush_method = O_DIRECT
I think it could be a performance issue, Did you monitor the mysql server cpu usage?
Related
I have a Drupal 7 application running a large Mysql database. It's built on AWS instance (30GB RAM, 8 CPU). Below is the my.cnf copy.
Issue is that my application gets noticeably fast when I restart the mysql which makes me think that current configuration isn't optimal and once the caches get filled, they slow down the server speed.
Any experts who could point out the issue here?
[mysqld]
# moving datadir to /media/db ebs storage
datadir=/media/db/mysql
socket=/var/lib/mysql/mysql.sock
tmpdir=/dev/shm/
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
local-infile=0
log-warnings=2
skip-external-locking
bind-address = 127.0.0.1
expire_logs_days = 1
max_binlog_size = 100M
log-bin=bin.log
log-bin-index=bin-log.index
binlog_format=row
key_buffer_size = 512M
max_allowed_packet = 32M
thread_stack = 292K
myisam_sort_buffer_size = 64M
thread_cache_size = 50
myisam-recover = BACKUP
max_connections = 100
max_user_connections = 150
table_cache = 16384
table_open_cache = 10240
table_definition_cache = 4096
thread_concurrency = 8
join_buffer_size = 4M
query_cache_type = 1
query_cache_limit = 256K
query_cache_min_res_unit = 2k
query_cache_size = 200M
innodb_buffer_pool_size= 8G
tmp_table_size=64M
max_heap_table_size=64M
wait_timeout=50
interactive_timeout=50
connect_timeout=10
open-files-limit = 65535
innodb_flush_log_at_trx_commit = 0
innodb_large_prefix=true
innodb_file_format=barracuda
innodb_file_per_table=true
table_open_cache = 5000
innodb_log_buffer_size= 32M
innodb_log_file_size = 512m
read_rnd_buffer_size = 8M
transaction-isolation = READ-COMMITTED
innodb_lock_wait_timeout = 25
innodb_write_io_threads = 16
innodb_sync_spin_loops = 200
innodb_spin_wait_delay = 24
innodb_read_io_threads = 16
innodb_flush_method = O_DIRECT
innodb_buffer_pool_instances = 8
innodb_autoinc_lock_mode = 2
[mysqldump]
max_allowed_packet = 32M
[isamchk]
key_buffer = 32M
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
set-variable=long_query_time=1
long_query_time=10
slow_query_log_file =/var/log/mysql/mysql-slow.log
Rate Per Second=RPS - Suggestions for your my.cnf [mysqld] section
read_rnd_buffer_size=256K # from 8M to reduce handler_read_rnd_next RPS
innodb_lru_scan_depth=100 # from 1024 to reduce CPU cycles used every SECOND
query_cache_min_res_unit=512 # from 2K to store more RESULTS in same query_cache_size
The BEST to you.
I have a web server with Apache and MySQL running on AWS EC2 t2.small with Windows 2012 Server. AWS EC2 t2.small characteristics:
RAM 2 GB (used 65%)
1 CPU 2.50 GHz (used 1%)
Now MySQL process (mysqld.exe) uses 400 MB of RAM (too much for me).
MySQL current settings are (my.ini):
key_buffer = 16M
max_allowed_packet = 16M
sort_buffer_size = 512K
net_buffer_length = 8K
read_buffer_size = 256K
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M
tmp-table-size = 32M
max-heap-table-size = 32M
max-connections = 500
thread-cache-size = 50
open-files-limit = 65535
table-definition-cache = 1024
table-open-cache = 2048
query_cache_type = 1
query_cache_limit = 256K
query_cache_min_res_unit = 2k
query_cache_size = 80M
innodb-log-files-in-group = 2
innodb-log-file-size = 64M
innodb-flush-log-at-trx-commit = 1
innodb-file-per-table = 1
innodb_buffer_pool_size = 128M
innodb_additional_mem_pool_size = 2M
innodb_log_file_size = 5M
innodb_log_buffer_size = 8M
innodb_flush_log_at_trx_commit = 1
innodb_lock_wait_timeout = 50
Database is formed by 20 InnoDB tables and they are composed with 5/10 columns. The server has a low traffic.
How can I optimize my settings to be suitable with EC2 t2.small (2GB RAM)?
You have innodb_buffer_pool_size twice in your config. It should be with underscores, but check which one gets used with:
show variables like 'innodb_buffer_pool_size';
You could try halving innodb_buffer_pool_size and query_cache_size. Try if performance is ok with query_cache_size=0 too.
I just moved server and am working on the my.cnf to optimize it for my pc, if someone can give a review on it, as the site is loading very slow on initial connect. In Pingdom, it shows it as "WAIT", and it starts loading site after about 8 seconds.
Here's my My.cnf
[mysqld]
datadir="/var/lib/mysql"
socket="/var/lib/mysql/mysql.sock"
user=mysql
open_files_limit=26672
innodb_file_per_table=1
symbolic-links=0
default-storage-engine=MyISAM
tmp_table_size = 1G
key_buffer_size = 2G
max_heap_table_size = 2G
query_cache_size = 2G
max_connections = 500
thread_cache_size = 5G
table_definition_cache = 2G
max_allowed_packet=64M
table_cache=2G
max_connect_errors = 10
binlog_cache_size = 2G
read_buffer_size = 2G
read_rnd_buffer_size = 5G
sort_buffer_size = 2G
bulk_insert_buffer_size = 3G
join_buffer_size = 3G
myisam_sort_buffer_size = 4G
thread_cache_size = 2G
myisam_max_sort_file_size = 4G
thread_concurrency = 8
query_cache_size = 1G
query_cache_limit = 10G
innodb_additional_mem_pool_size = 1G
innodb_buffer_pool_size = 2G
innodb_data_file_path = ibdata1:10M:autoextend
innodb_file_io_threads = 4
innodb_thread_concurrency = 16
innodb_flush_log_at_trx_commit = 1
innodb_log_buffer_size = 8M
innodb_log_file_size = 1G
innodb_log_files_in_group = 3
innodb_max_dirty_pages_pct = 90
innodb_lock_wait_timeout = 120
myisam_sort_buffer_size=2G
[mysqld_safe]
log="/var/log/mysqld.log"
log-error="/var/log/mysqld.error.log"
[myisamchk]
sort_buffer_size = 5M
read_buffer = 50M
write_buffer = 50M
All advice/ criticism are welcomed, i am an amatuer in this and have made this by reading up blogs and other resources.
I recommend you to use Percona's Configuration Wizard for MySQL:
https://tools.percona.com/wizard
And to check if you really need:
1) MyISAM engine;
2) query cache;
[mysqld]
open_files_limit=10000
skip-name-resolve
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
max_heap_table_size = 2GB
query_cache_limit = 256M
key_buffer_size = 6GB
innodb_table_locks = 0
innodb_buffer_pool_size = 100M
join_buffer_size = 32M
max_connections = 400
max_user_connections = 1400
#skip-bdb
key_buffer = 48M
# max_allowed_packet = 68M
max_allowed_packet = 256M
tmp_table_size = 3GB
table_cache = 800
sort_buffer_size = 1M
net_buffer_length = 8K
read_buffer_size = 1M
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M
server-id = 1
wait_timeout = 28800
query_cache_size = 50M
thread_cache_size = 1024
long_query_time = 4
We have this config on 15GB RAM VPS server, which is used for MySQL only. The problem is that the server is using only 3GB of RAM during the heaviest traffic.
Can anyone suggest what should be adjusted in config to use all available RAM?
mysqltuner.pl says this:
[!!] Maximum possible memory usage: 15.0G (103% of installed RAM)
I am currently running on a cloud server, WITH ukfast
CPU GenuineIntel, Intel(R) Xeon(R) CPU E5-2620 0 # 2.00GHz
Version Parallels Plesk Panel v11.0.9_build110120608.16 os_CentOS 6
OS CentOS 6.3 (Final) 64-bit
RAM: 6GB
I have 12,000 SKU's but never get over 30 vistors on the site at one time and I am trying to optimise my.cnf file:
This is what I currently have:
log_slow_queries = /var/log/mysql-slow.log
long_query_time = 5
key_buffer = 32M
max_allowed_packet = 16M
table_cache = 1024
sort_buffer_size = 48M
read_buffer_size = 92M
read_rnd_buffer_size = 100M
myisam_sort_buffer_size = 100M
myisam_max_sort_file_size = 1G
myisam_max_extra_sort_file_size = 1G
myisam_repair_threads = 1
thread_cache_size = 32
query_cache_type = 1
query_cache_size = 200M
query_cache_limit = 2M
max_connections=50
wait_timeout=120
tmp_table_size = 256M
max_heap_table_size = 64M
innodb_buffer_pool_size = 650M
innodb_additional_mem_pool_size = 24M
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid}
is there anything I am missing or anything I should increase or decrease?
any thoughts / advice would be great.
I have face this problem before.
Please try this
key_buffer = 32M
max_allowed_packet = 16M
table_cache = 1024
sort_buffer_size = 48M
read_buffer_size = 92M
read_rnd_buffer_size = 104M
myisam_sort_buffer_size = 100M
myisam_max_sort_file_size = 1024m
myisam_max_extra_sort_file_size = 1024m
myisam_repair_threads = 1
thread_cache_size = 32
query_cache_type = 1
query_cache_size = 256M
query_cache_limit = 2M
max_connections=50
wait_timeout=60
tmp_table_size = 256M
max_heap_table_size = 64M
innodb_buffer_pool_size = 650M
innodb_additional_mem_pool_size = 24M
or you have to change or upgrade your hosting to VPS server.
Houp this help.
^ ^