I am having a 4 GB vps box. At any time there are not more than 40 visitors browsing 10-15 page simultaneously. Suddenly 2 days back server cpu usage touching almost 2000%. Strange thing is when i stop the apache, it still does not go down. below is the top command result.
PID USER PR NI VIRT RES SHR S CPU MEM TIME+ COMMAND
23644 mysql 15 0 12.7g 742m 6840 S 1812.0 18.1 1107:15 mysqld
14076 xyzaz 19 0 194m 16m 7348 S 7.1 0.4 0:00.22 php
14080 xyzaz 18 0 195m 17m 7276 S 6.5 0.4 0:00.20 php
i have troubleshooted every thing but no luck. I dont know it was working fine but suddenly how it changed to such a high value.
below is the output of my.cnf
[mysqld]
local-infile=0
max_connections=500
key_buffer_size=128M
myisam_sort_buffer_size=64M
join_buffer_size=2M
read_buffer_size=1M
sort_buffer_size=2M
max_heap_table_size=16M
table_cache=1800
thread_cache_size=384
interactive_timeout=25
wait_timeout=7200
connect_timeout=10
max_allowed_packet=268435456
max_connect_errors=100
query_cache_limit=4M
query_cache_size=128M
query_cache_type=1
tmp_table_size=64M
skip-name-resolve
innodb_buffer_pool_size=229M
default-storage-engine=MyISAM
innodb_file_per_table=1
[mysqld_safe]
open_files_limit = 6553
[mysqldump]
quick
max_allowed_packet=16M
[myisamchk]
key_buffer_size=64M
sort_buffer=64M
read_buffer=16M
write_buffer=16M
Any help is really appriciated
Related
I have a problem importing a 13Gb database on my server with the following configuration:
CPU: Intel Atom C2350 - 1.7 GHz - 2 core(s)
RAM: 4GB - DDR3
Hard Drive(s): 1x 128GB (SSD SATA)
After following many mysql configurations on stackoverflow, I have reduced all the values in the my.cnf configuration file:
[mysqld]
sql_mode=""
key_buffer_size = 50M
innodb_buffer_pool_size = 100M
innodb_buffer_pool_instances = 2
innodb_log_file_size = 50M
innodb_write_io_threads = 2
innodb_flush_log_at_trx_commit = 0
sort_buffer_size = 50M
read_buffer_size = 50M
read_rnd_buffer_size = 50M
join_buffer_size = 50M
max_connections = 50M
wait_timeout = 600
max_allowed_packet = 64M
However the server still runs out of memory. Current swap is 1Gb I have tried increasing it to 7Gb, but that didn't do much.
So to summarize when I run the import of the dump file it works for some time and then the process gets killed because it runs out of memory. What elements do I need to change/add in my.cnf file for it to continue the process ? I don't mind if it takes 3, 4 days to complete.
Here is the error message:
Out of memory (Needed 2098545784 bytes)
mysql: Out of memory (Needed 2601694880 bytes)
Here are the logs of memory usage just before the process got killed:
total used free shared buff/cache available
Mem: 3942 3763 108 0 70 22
Swap: 1048 1037 11
PID %CPU %MEM VSZ RSS TTY STAT START TIME
16944 91.6 90.1 4206248 3638948 pts/0 R 14:00 6:01
Thanks in advance.
Host with 32GB RAM. Mysql 5.7.21 works in Docker container, runs with the following command:
docker run --restart=always --name mydb5721 -v ... -e MYSQL_ROOT_PASSWORD=... -e MYSQL_USER=... -e MYSQL_PASSWORD=... -p ...:3306 --memory 14G --memory-swap 14G --health-interval=10s --health-timeout=10s --health-retries=3 --health-cmd='/bin/bash /var/lib/mysql/healthcheck.sh' mysql:5.7.21 --verbose &
so without swap and with 14GB RAM.
Memory usage almost always grows and is never cleared, even after my large requests are completed. And it crashes when exceeds limit 14GB. "Flush query cache" never changes this graph, "flush tables" usually not greatly affected and then grows fast again. When the db-requests are large and heavy, the limit is reached quickly, otherwise - slowly.
/etc/mysql/mysql.conf.d:
[mysqld]
innodb_force_recovery = 6
user=root
max_allowed_packet=1500M
bind-address=0.0.0.0
key_buffer_size=1024M
max_connections=200
table_open_cache=64
query_cache_limit=4M
query_cache_size=512M
innodb_buffer_pool_size=7G
server-id=2
log-slave-updates=1
innodb_log_buffer_size = 16M
innodb_log_file_size = 250M
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
datadir = /var/lib/mysql
log-error = /var/log/mysql/error.log
symbolic-links=0
skip-host-cache
skip-name-resolve
sql-mode="NO_AUTO_VALUE_ON_ZERO"
innodb_file_per_table = 1
table_open_cache = 2048
innodb_open_files = 2048
sort_buffer_size = 128M
read_buffer_size = 128M
read_rnd_buffer_size = 1M
thread_stack = 128M
query_cache_type = 0
thread_cache_size = 32
max_heap_table_size = 256M
tmp_table_size = 1G
innodb_buffer_pool_instances = 4
innodb_read_io_threads = 8
innodb_write_io_threads = 8
performance_schema = 0
innodb_flush_log_at_trx_commit = 2
slow_query_log=1
long_query_time=40
slow-query-log-file=/var/log/mysql/slow_queries.log
general_log=0
innodb_flush_method=O_DIRECT
innodb_tmpdir=/tmp
secure-file-priv = ""
My main questions:
- is the Docker guilty or Mysql (cause I never have this problem with mysql without docker)?
- why memory never releases even after my queries are completed?
- will the solution migration to MariaDB?
Your ulimit -a report indicates Open files limit of 1024. In LX, ulimit -n 24000 will enable more file handles for MySQL to use.
For the new limit to be persistent over LX shutdown, restart review this URL
https://glassonionblog.wordpress.com/2013/01/27/increase-ulimit-and-file-descriptors-limit/
Your details may be slightly different.
Review my recent COMMENT about DOCKER missing CLOSE() to release resources.
Rate Per Second=RPS Suggestions to consider for your my.cnf [mysqld] section
# 20180924 1442 mysqlservertuning.com
read_rnd_buffer_size=256K # from 1M to reduce handler_read_rnd_next RPS
read_buffer_size=256K # from 128M will increase handler_read_next RPS
innodb_io_capacity_max=20000 # from 2000 to take advantage of SSD IOPS capacity
innodb_io_capacity=10000 # from 200 to take advantage of SSD IOPS capacity
tmp_table_size-256M # from 1G to be matched to max_heap_table_size
key_buffer_size=64M # from 1G to conserve RAM, key_blocks_used is less than 1%
innodb_lru_scan_depth=100 # from 1024 to conserve CPU cycles every SECOND
innodb_thread_concurrency=24 # from 0 (unlimited) for your 16 cpu's reported by iostat
table_open_cache=4096 # from 2048 to reduce opened_tables count
table_definition_cache=2048 # from 1424 to reduce opened_table_definitions count
query_cache_size=0 # from ~512M because query_cache_type=0 (off) to conserve RAM
query_cache_limit=0 # from ~4M because query_cache_type=0 (off) to conserve RAM
Considering your situation, backup your current my.cnf, copy the whole block, starting
with # date time, my web address to END of [mysqld] section, lead SAME NAMED variable with # AND spacebar to disable the line prior to new BLOCK of variables, stop service / start service.
Until the docker world gets the CLOSE() in place, you will still have crashes, but a little bit later in the day. Many of your PER CONNECTION variables were OVERPROVISIONED, some may still be, but likely survivable.
For additional suggestions please view my profile, Network profile for contact information including my Skype ID. Look forward to hearing from you.
I tend to think that it was mysql v5 bug cause when I upgrade mysql to v8.0.12 the situation became good - memory regularly releases as expected even during the heavy request.
Hi are trying to optimize MySQL for a huge traffic web server, we are using Apache 2.4 , and MySQL 5.6 version.
We have put these parameters in MySQL configuration file
cat /etc/my.cnf
[mysqld]
skip-external-locking
key_buffer = 500M
max_allowed_packet=64M
max_connections = 2000
max_user_connections = 50000
wait_timeout=180
connect_timeout=120
table_open_cache=4096
table_definition_cache=4096
myisam_sort_buffer_size = 64M
thread_cache_size = 128
query_cache_size = 190M
query_cache_limit = 3M
open_files_limit=10000
log_error = /var/log/mysql/error.log
default-storage-engine=MyISAM
[mysqld]
log-bin=mysql-bin
binlog-do-db=******
server-id=1
[mysql]
no-auto-rehash
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
replicate-do-db=*****
auto_increment_increment = 2
auto_increment_offset = 1
replicate-same-server-id = 0
slave-skip-errors=all
[mysqld]
slave-skip-errors=1062
skip-slave-start
query_cache_type=1
[mysqld]
innodb_lock_wait_timeout=120
slave_compressed_protocol=1
query_cache_size = 190M
The QC is inefficient when it is big; limit to about 50M.
slave-skip-errors=1062
This is a flavor of "sweeping bugs under the rug".
default-storage-engine=MyISAM
passé.
key_buffer = 500M
If you do stick with MyISAM, and this server is running just MySQL, then use 1000M.
server-id=1
Make sure each server has a different value.
But that is only configuration. If you have slow queries, let's see them and try to improve them.
I have an Ubuntu server and 10 sites with different framework / CMS.
My problem is that when I open a site with a big CMS integrated with MySQL DB, Apache2 using 100% CPU, in these sites I have a page speed between 10 - 20 seconds (to render HTML page) and I really don't know why. (but all PHP frameworks without a MySQL connection work fine).
With my server support manager we see all works well in my server (no I/O problem or others), and we think there is an issue in Apache / MySQL config.
I haven't crashed DB tables and optimized all innodb tables.
This is a top snapshot when I load this slow CMS from my browser:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13307 www-data 20 0 90116 42m 22m R 100 4.2 0:15.23 apache2
2224 root 20 0 37932 8148 1724 S 8 0.8 212:21.99 newrelic-daemon
13422 mysql 20 0 186m 47m 6280 S 0 4.7 0:11.20 mysqld
13889 root 20 0 2640 1132 860 R 0 0.1 0:00.03 top
1 root 20 0 2796 1124 792 S 0 0.1 0:06.28 init
Newrelic screenshot problem (not only with wordpress but all CMS): Newrelic screenshot
My CPU info:
$ head /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 45
model name : Intel(R) Xeon(R) CPU E5-2670 0 # 2.60GHz
stepping : 7
cpu MHz : 2600.086
cache size : 20480 KB
fdiv_bug : no
hlt_bug : no
My Memory info:
$ head /proc/meminfo
MemTotal: 1028476 kB
MemFree: 571424 kB
Buffers: 17920 kB
Cached: 273312 kB
SwapCached: 824 kB
Active: 232768 kB
Inactive: 180264 kB
Active(anon): 147336 kB
Inactive(anon): 5292 kB
Active(file): 85432 kB
My Apache Config:
LockFile ${APACHE_LOCK_DIR}/accept.lock
PidFile ${APACHE_PID_FILE}
Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15
<IfModule mpm_prefork_module>
StartServers 2
MinSpareServers 5
MaxSpareServers 10
MaxClients 50
MaxRequestsPerChild 0
</IfModule>
<IfModule mpm_worker_module>
StartServers 2
MinSpareThreads 25
MaxSpareThreads 75
ThreadLimit 64
ThreadsPerChild 25
MaxClients 150
MaxRequestsPerChild 0
</IfModule>
<IfModule mpm_event_module>
StartServers 2
MinSpareThreads 5
MaxSpareThreads 10
ThreadLimit 64
ThreadsPerChild 25
MaxClients 50
MaxRequestsPerChild 0
</IfModule>
User ${APACHE_RUN_USER}
Group ${APACHE_RUN_GROUP}
AccessFileName .htaccess
<Files ~ "^\.ht">
Order allow,deny
Deny from all
Satisfy all
</Files>
DefaultType text/plain
HostnameLookups Off
ErrorLog ${APACHE_LOG_DIR}/error.log
LogLevel warn
Include mods-enabled/*.load
Include mods-enabled/*.conf
Include httpd.conf
Include ports.conf
LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %O" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
Include conf.d/
Include sites-enabled/
MySQL Config:
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
user = mysql
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
skip-external-locking
key_buffer = 32M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 286
interactive_timeout = 25
wait_timeout = 1000
myisam-recover = BACKUP
myisam_sort_buffer_size = 32MB
max_connections = 400
read_buffer_size = 1M
sort_buffer_size = 2M
table_cache = 400
thread_concurrency = 10
query_cache_limit = 64M
query_cache_size = 32M
general_log_file = /var/log/mysql/mysql.log
log_error = /var/log/mysql/error.log
log_slow_queries = /var/log/mysql/mysql-slow.log
long_query_time = 4
log-queries-not-using-indexes
log_bin = /var/log/mysql/mysql-bin.log
expire_logs_days = 10
max_binlog_size = 100M
bind-address = 127.0.0.1
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
[mysql]
[isamchk]
key_buffer = 32M
!includedir /etc/mysql/conf.d/
How to solve the issue?
There's a whole lot wrong with your Apache config - but that's probably irrelevant to the problem you are currently having, as is the state of your DBMS - based on the information you provided it's apache where things are going wrong.
You've no told us which MPM Apache is using (which is critical to understanding the problem) nor which CMS.
The information I've said you've omitted is just the tip of te iceberg - investigating and resolving performance issues is really, really complex - it can take months (much of which is spent gathering information) - you're very unlikely to get a definitive answer here.
You need to place more info. For example, if you try to display in wordpress 1000000 of posts and especially if you get properties of this posts in separate querys, you can get 100% CPU Usage for a big time.
Try to find a script, that cause this problem, montor your mysql (show processlist) in high usage moment and others.
I'd like to ask anyone with a bit of experience on an opinion of my current my.cnf settings and what optimizations should be done.
Server specs are:
CPU: AMD Opteron™ 3280, 8 Cores, 8x 2.4 GHz
RAM: 16 GB DDR3
2x 2,000 GB SATA II-HDD, 7.2k - RAID 1 Software
Cloudlinux + Cpanel installed
Apache + Mysql 5.5 + NginxCP + eAccelerator
The current my.cnf file is:
[client]
port=3306
socket="/var/lib/mysql/mysql.sock"
[mysqld]
innodb_additional_mem_pool_size=16M
innodb_buffer_pool_size=10G
innodb_file_per_table=1
innodb_log_buffer_size=4M
innodb_flush_log_at_trx_commit=2
log-bin=mysql-bin
myisam_sort_buffer_size=64M
expire_logs_days=7
query_cache_size=128M
thread_cache_size=12
max_allowed_packet=15M
skip-federated
table_definition_cache=2048
local-infile=0
table_cache=8192
max_connections=60
read_buffer_size=2M
slow_query_log=1
slow_query_log_file="/var/log/slow_queries.log"
thread_concurrency=16
sort_buffer_size=2M
port=3306
join_buffer_size=16M
key_buffer_size=600M
query_cache_limit=10M
socket="/var/lib/mysql/mysql.sock"
skip-external-locking
query-cache-type=1
long_query_time=5
default-storage-engine=InnoDB
tmp_table_size=384M
max_heap_table_size=384M
[myisamchk]
read_buffer=2M
key_buffer=256M
sort_buffer_size=256M
write_buffer=2M
[mysqldump]
max_allowed_packet=16M
DBAs recommended to use the Percona Wizard first o all https://tools.percona.com/wizard
Secondly follow this guide: http://www.percona.com/blog/2014/01/28/10-mysql-settings-to-tune-after-installation/