I have a problem importing a 13Gb database on my server with the following configuration:
CPU: Intel Atom C2350 - 1.7 GHz - 2 core(s)
RAM: 4GB - DDR3
Hard Drive(s): 1x 128GB (SSD SATA)
After following many mysql configurations on stackoverflow, I have reduced all the values in the my.cnf configuration file:
[mysqld]
sql_mode=""
key_buffer_size = 50M
innodb_buffer_pool_size = 100M
innodb_buffer_pool_instances = 2
innodb_log_file_size = 50M
innodb_write_io_threads = 2
innodb_flush_log_at_trx_commit = 0
sort_buffer_size = 50M
read_buffer_size = 50M
read_rnd_buffer_size = 50M
join_buffer_size = 50M
max_connections = 50M
wait_timeout = 600
max_allowed_packet = 64M
However the server still runs out of memory. Current swap is 1Gb I have tried increasing it to 7Gb, but that didn't do much.
So to summarize when I run the import of the dump file it works for some time and then the process gets killed because it runs out of memory. What elements do I need to change/add in my.cnf file for it to continue the process ? I don't mind if it takes 3, 4 days to complete.
Here is the error message:
Out of memory (Needed 2098545784 bytes)
mysql: Out of memory (Needed 2601694880 bytes)
Here are the logs of memory usage just before the process got killed:
total used free shared buff/cache available
Mem: 3942 3763 108 0 70 22
Swap: 1048 1037 11
PID %CPU %MEM VSZ RSS TTY STAT START TIME
16944 91.6 90.1 4206248 3638948 pts/0 R 14:00 6:01
Thanks in advance.
Related
Host with 32GB RAM. Mysql 5.7.21 works in Docker container, runs with the following command:
docker run --restart=always --name mydb5721 -v ... -e MYSQL_ROOT_PASSWORD=... -e MYSQL_USER=... -e MYSQL_PASSWORD=... -p ...:3306 --memory 14G --memory-swap 14G --health-interval=10s --health-timeout=10s --health-retries=3 --health-cmd='/bin/bash /var/lib/mysql/healthcheck.sh' mysql:5.7.21 --verbose &
so without swap and with 14GB RAM.
Memory usage almost always grows and is never cleared, even after my large requests are completed. And it crashes when exceeds limit 14GB. "Flush query cache" never changes this graph, "flush tables" usually not greatly affected and then grows fast again. When the db-requests are large and heavy, the limit is reached quickly, otherwise - slowly.
/etc/mysql/mysql.conf.d:
[mysqld]
innodb_force_recovery = 6
user=root
max_allowed_packet=1500M
bind-address=0.0.0.0
key_buffer_size=1024M
max_connections=200
table_open_cache=64
query_cache_limit=4M
query_cache_size=512M
innodb_buffer_pool_size=7G
server-id=2
log-slave-updates=1
innodb_log_buffer_size = 16M
innodb_log_file_size = 250M
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
datadir = /var/lib/mysql
log-error = /var/log/mysql/error.log
symbolic-links=0
skip-host-cache
skip-name-resolve
sql-mode="NO_AUTO_VALUE_ON_ZERO"
innodb_file_per_table = 1
table_open_cache = 2048
innodb_open_files = 2048
sort_buffer_size = 128M
read_buffer_size = 128M
read_rnd_buffer_size = 1M
thread_stack = 128M
query_cache_type = 0
thread_cache_size = 32
max_heap_table_size = 256M
tmp_table_size = 1G
innodb_buffer_pool_instances = 4
innodb_read_io_threads = 8
innodb_write_io_threads = 8
performance_schema = 0
innodb_flush_log_at_trx_commit = 2
slow_query_log=1
long_query_time=40
slow-query-log-file=/var/log/mysql/slow_queries.log
general_log=0
innodb_flush_method=O_DIRECT
innodb_tmpdir=/tmp
secure-file-priv = ""
My main questions:
- is the Docker guilty or Mysql (cause I never have this problem with mysql without docker)?
- why memory never releases even after my queries are completed?
- will the solution migration to MariaDB?
Your ulimit -a report indicates Open files limit of 1024. In LX, ulimit -n 24000 will enable more file handles for MySQL to use.
For the new limit to be persistent over LX shutdown, restart review this URL
https://glassonionblog.wordpress.com/2013/01/27/increase-ulimit-and-file-descriptors-limit/
Your details may be slightly different.
Review my recent COMMENT about DOCKER missing CLOSE() to release resources.
Rate Per Second=RPS Suggestions to consider for your my.cnf [mysqld] section
# 20180924 1442 mysqlservertuning.com
read_rnd_buffer_size=256K # from 1M to reduce handler_read_rnd_next RPS
read_buffer_size=256K # from 128M will increase handler_read_next RPS
innodb_io_capacity_max=20000 # from 2000 to take advantage of SSD IOPS capacity
innodb_io_capacity=10000 # from 200 to take advantage of SSD IOPS capacity
tmp_table_size-256M # from 1G to be matched to max_heap_table_size
key_buffer_size=64M # from 1G to conserve RAM, key_blocks_used is less than 1%
innodb_lru_scan_depth=100 # from 1024 to conserve CPU cycles every SECOND
innodb_thread_concurrency=24 # from 0 (unlimited) for your 16 cpu's reported by iostat
table_open_cache=4096 # from 2048 to reduce opened_tables count
table_definition_cache=2048 # from 1424 to reduce opened_table_definitions count
query_cache_size=0 # from ~512M because query_cache_type=0 (off) to conserve RAM
query_cache_limit=0 # from ~4M because query_cache_type=0 (off) to conserve RAM
Considering your situation, backup your current my.cnf, copy the whole block, starting
with # date time, my web address to END of [mysqld] section, lead SAME NAMED variable with # AND spacebar to disable the line prior to new BLOCK of variables, stop service / start service.
Until the docker world gets the CLOSE() in place, you will still have crashes, but a little bit later in the day. Many of your PER CONNECTION variables were OVERPROVISIONED, some may still be, but likely survivable.
For additional suggestions please view my profile, Network profile for contact information including my Skype ID. Look forward to hearing from you.
I tend to think that it was mysql v5 bug cause when I upgrade mysql to v8.0.12 the situation became good - memory regularly releases as expected even during the heavy request.
I have been working on a separate server to configure the mariadb but still queries are taking sooo long time to return and mostly not returning.
Mariadb version: 10.1.31-1
Server has 50GB Ram and 2200 Ghz * 10 core
Centos7 as OP
The target database currently has very huge innodb tables and not indexed well. there is no any other table type just innodb.
Currently i am making some nested select queries and saving results to another empty clone database with insert into or replace into.
as an example:
REPLACE INTO db1.table1
SELECT * FROM db0.table1 WHERE ART_ID IN (
SELECT ART_ID FROM db0.table2 WHERE BRA_ID IN (
SELECT BRA_ID FROM db0.table3 WHERE BRAND IN (
SELECT BRAND FROM db0.table4 WHERE ID IN (... bunch of comma seperated ids)
)
)
);
The only connection is from local and/or just one remote connection.
The only task of the server is to filter the huge data to another db, no web server etc.
I put the /etc/my.cnf file down below. I made lots of tests on this file so you may see some useless additions.
The question is: Is there any miss configurations in the cnf file; and is there any other configuration tips etc to run those bunch of queries in reasonable time.
It was working better but after i change the configurations or / and update the server everything got much slower. (yes i update the tables as well after updating db version)
# The MySQL server
[mysqld]
performance_schema=0
bind-address = 0.0.0.0
max_connections = 32
socket = /var/lib/mysql/mysql.sock
skip-external-locking
max_allowed_packet = 1024M
connect_timeout=300
wait_timeout=1200
interactive_timeout=300
key_buffer_size = 2G
bulk_insert_buffer_size=256M
sort_buffer_size = 32M
#read_buffer_size = 8M
#read_rnd_buffer_size = 32M
myisam_sort_buffer_size = 32M
join_buffer_size=32M
#max_heap_table_size= 8G
tmp_table_size=8G
concurrent_insert=2
#external-locking=FALSE
#open_files_limit=50000
default-storage-engine=innodb
innodb_file_per_table=1
#lower_case_table_names=1
# Try number of CPU's*2 for thread_concurrency
innodb_write_io_threads=10
innodb_read_io_threads=20
#innodb_buffer_pool_instances=8
#thread_stack = 1M
#thread_cache_size = -1
innodb_buffer_pool_size = 16G
#query_cache_limit = 1073741824
#query_cache_size = 1073741824
#query_cache_type = 1
query_cache_size=0
query_cache_type=0
#innodb_additional_mem_pool_size = 20M
# Set .._log_file_size to 25 % of buffer pool size
innodb_fast_shutdown=0
#innodb_log_file_size = 1G
#innodb_log_buffer_size = 8M
#innodb_flush_log_at_trx_commit = 1
#innodb_lock_wait_timeout = 0
innodb_lock_wait_timeout = 1073741823
innodb_thread_concurrency=0
innodb_commit_concurrency=0
innodb_flush_method=O_DSYNC
innodb_log_compressed_pages = 0
#innodb_flush_neighbors = 0
#innodb_adaptive_hash_index_partitions = 8
log-bin=mysql-bin
server-id = 1
[mysqldump]
quick
max_allowed_packet = 16M
[mysql]
no-auto-rehash
[isamchk]
key_buffer = 384M
sort_buffer = 384M
read_buffer = 256M
write_buffer = 256M
[myisamchk]
key_buffer = 384M
sort_buffer = 384M
read_buffer = 256M
write_buffer = 256M
[mysqlhotcopy]
interactive-timeout
I used percona wizard and merge the advices.
the current my.cnf is below.
I run the php application yesterday night (2018-03-05 22:41:30 UTC), but still mysql could not return any result.
the application is only adding limit 10000 and offset (i did not want to start from beginning, i am currently tracking the last offset and starting from that point.)
currently i am planning to delete and reinstall mariadb :(
(it will be emotional reaction)
# Generated by Percona Configuration Wizard (http://tools.percona.com/) version REL5-20120208
[mysql]
# CLIENT #
port = 9999
socket = /var/lib/mysql/mysql.sock
[mysqld]
# GENERAL #
port = 9999
user = mysql
default-storage-engine = InnoDB
socket = /var/lib/mysql/mysql.sock
pid-file = /var/lib/mysql/mysql.pid
# MyISAM # - there is no myisam table
key-buffer-size = 32M
myisam-recover-options = FORCE,BACKUP
# SAFETY #
max-allowed-packet = 16M
max-connect-errors = 1000000
skip-name-resolve
# DATA STORAGE #
datadir = /var/lib/mysql/
# BINARY LOGGING #
log-bin = /var/lib/mysql/mysql-bin
expire-logs-days = 14
sync-binlog = 1
# CACHES AND LIMITS #
tmp-table-size = 32M
max-heap-table-size = 32M
query-cache-type = 0
query-cache-size = 0
max-connections = 30
#thread-cache-size = 16
open-files-limit = 65535
table-definition-cache = 1024
table-open-cache = 2048
# INNODB #
innodb-flush-method = O_DIRECT
innodb-log-files-in-group = 2
innodb-log-file-size = 5G
innodb-flush-log-at-trx-commit = 1
innodb-file-per-table = 1
innodb-buffer-pool-size = 42G
innodb-buffer-pool-instances = 42
thread_cache_size=100
innodb_lru_scan_depth=100
innodb_purge_threads=4
innodb_read_io_threads=64
innodb_write_io_threads=64
innodb_thread_concurrency=0
max_seeks_for_key=32
max_write_lock_count=16
thread_concurrency=35
innodb_fast_shutdown=0
innodb_file_per_table=1
default-storage-engine=innodb
concurrent_insert=2
join_buffer_size=32M
tmp_table_size=8G
read_buffer_size = 8M
# LOGGING #
log-error = /var/lib/mysql/mysql-error.log
log-queries-not-using-indexes = 0
slow-query-log = 1
slow-query-log-file = /var/lib/mysql/mysql-slow.log
The my.cnf data in the Question does not match SHOW GLOBAL VARIABLES; values.
Please use this link
https://mariadb.com/kb/en/library/configuring-mariadb-with-mycnf/
to help get accurate matched values for assistance.
Then, Suggestions to consider for your my.cnf [mysqld] section include:
innodb_buffer_pool_size=8G # from > 40G (95% of RAM will not work well)
innodb_lru_scan_depth=100 # from 1024 see refman every second digging too deep
log_warnings=2 # from 1 to include connection errors more detail
max_connect_errors=10 # from 1,000,000 no need to waste a million cycles
thread_cache_size=100 # from 16 CAP per V8 MySQ to be prepared for volume
have_symlink=NO # to protect you server, unless you NEED it
innodb_purge_threads=4 # from 1 to accommodate when needed
innodb_read_io_threads=64 # from 4 see dba.stackexhange.com Q 5666 9/12/11 Rolando
innodb_write_io_threads=64 # from 4 per Rolando to use multi-core
innodb_thrad_concurrency=0 # another part of multi-core enabling
max_seeks_for_key=32 # from huge number, not found in 32, will not be found
max_write_lock_count=16 # give RD opportunity after nn locks
thread_concurrency=35 # to limit concurrent processing load
#Erce, To be applied in the correct active my.cnf, please. Withdrawn my offer to redo your mysqld section.
I used percona wizard and merge the advices. the current my.cnf is below.
I run the php application yesterday night (2018-03-05 22:41:30 UTC), but still mysql could not return any result.
the application is only adding limit 10000 and offset (i did not want to start from beginning, i am currently tracking the last offset and starting from that point.)
[mysql]
# CLIENT #
port = 9999
socket = /var/lib/mysql/mysql.sock
[mysqld]
# GENERAL #
port = 9999
user = mysql
default-storage-engine = InnoDB
socket = /var/lib/mysql/mysql.sock
pid-file = /var/lib/mysql/mysql.pid
# MyISAM # - there is no myisam table
key-buffer-size = 32M
myisam-recover-options = FORCE,BACKUP
# SAFETY #
max-allowed-packet = 16M
max-connect-errors = 1000000
skip-name-resolve
# DATA STORAGE #
datadir = /var/lib/mysql/
# BINARY LOGGING #
log-bin = /var/lib/mysql/mysql-bin
expire-logs-days = 14
sync-binlog = 1
# CACHES AND LIMITS #
tmp-table-size = 32M
max-heap-table-size = 32M
query-cache-type = 0
query-cache-size = 0
max-connections = 30
#thread-cache-size = 16
open-files-limit = 65535
table-definition-cache = 1024
table-open-cache = 2048
# INNODB #
innodb-flush-method = O_DIRECT
innodb-log-files-in-group = 2
innodb-log-file-size = 5G
innodb-flush-log-at-trx-commit = 1
innodb-file-per-table = 1
innodb-buffer-pool-size = 42G
innodb-buffer-pool-instances = 42
thread_cache_size=100
innodb_lru_scan_depth=100
innodb_purge_threads=4
innodb_read_io_threads=64
innodb_write_io_threads=64
innodb_thread_concurrency=0
max_seeks_for_key=32
max_write_lock_count=16
thread_concurrency=35
innodb_fast_shutdown=0
innodb_file_per_table=1
default-storage-engine=innodb
concurrent_insert=2
join_buffer_size=32M
tmp_table_size=8G
read_buffer_size = 8M
# LOGGING #
log-error = /var/lib/mysql/mysql-error.log
log-queries-not-using-indexes = 0
slow-query-log = 1
slow-query-log-file = /var/lib/mysql/mysql-slow.log
#erce
These THREE lines NEED TO BE REMOVED (just before LOGGING area)
join_buffer_size=32M
tmp_table_size=8G
read_buffer_size = 8M
LOGGING
tmp_table_size was accurately matched earlier with max_heap_table_size.
This line later takes charge and get you unbalanced which is BAD.
The join_buffer_size and read_buffer_size are killing your RAM footprint.
Use mysqlcalculator.com to understand why, please.
The performance problem is primarily in the formulation of the query, not necessarily in the tuning or indexes.
Query Formulation
In general, turn IN ( SELECT ... ) into JOIN.
Tackling the last step:
SELECT bra_id
FROM t3
JOIN t4 ON t3.brand = t4.brand
WHERE t4.id IN (...commalist...)
with indexes:
t4: INDEX(id, brand)
t3: INDEX(brand, bra_id)
This starts with t4, looking for the various ids, then mapping them to brands. Then it reaches into t3 by brand to get bra_id.
I'll let you practice by doing the next two steps to get through t2 to t1.
Tuning advice
Don't blindly increase values in my.cnf, it can get you into trouble.
innodb_buffer_pool_size = 35G -- the most important
42G is probably OK -- if you don't have too many other apps running on the same server. 42 instances may work. Some say 16 instances should be the max.
The rest of the tunables can probably be left alone.
max_heap_table_size= 8G
tmp_table_size=8G
Those are dangerously high. With them, 8GB could be allocated once (or more than once) per connection. But you have only 50GB of RAM. It would be better to set them to no more than 1% of RAM (500M or less; 32M should be just fine).
#lower_case_table_names=1
Leave commented out. Any attempt to override what is needed for the OS is likely to get you into deep weeds.
port = 9999
What was wrong with the default 3306?
Other
"application grabbed 10000 items in 6 hours. for the next block of items 8 hour passed" -- Please provide SHOW CREATE TABLE for the 4 tables being read, plus the table being written to. And tell us how big the tables are. I suspect you are thrashing over index updates in the target table. There is a simple solution.
Check to see if mysqld is swapping.
I guess i need to change select query style, i will try the offered query alternatives because it did not work, queries are still to slow (i assume)
But these are current server stats (49GB ram, 10 core) and mysqls virtual memory is about 47~GB which is fine, no any other app is running. I think its alright ?
tuning-primer.sh
https://pastebin.com/bcjS1PWf
mysqltuner.pl
https://pastebin.com/mXUTLPQ6
my.cnf
https://pastebin.com/9jMLzhZS
Suggestions for your my.cnf-ini [mysqld] section using data posted 3/15/18
thread_cahce_size=40 # from 20, 8 required for overhead, room to wiggle
innodb_io_capacity=400 # from 200 new iostat -x will confirm use
# read_buffer_size=8388608 # lead with # for default of 128K - extreme at 8M
read_rnd_buffer_size=128K # from 256K will affect handler_read_rnd_next
sync_binlog=32 # from 1 sec frequency, unless you fear recovery required
# join_buffer_size=4M # lead with # for default of 128K
key_buffer_size=1M # from 32M less than 1M used - primarily Innodb tables
innodb_adapative_max_sleep_delay=15 # from 150000 (15 secs - n0 other users)
innodb_buffer_pool_instances=8 # from 42 to minimize mgmt. overhead
innodb_buffer_pool_size=40G # from 44G allow OS a little breathing room
innodb_change_buffer_max_size=3 # from 25% - no need to set aside 10G when not changing anything significant
innodb_doublewrite=OFF # from ON you are not a PROD effort - crash recovery NA for run of this instance
innodb_print_all_deadlocks=ON # from OFF you always need in error log
innodb_read_ahead_threshold=8 # from 56, why wait to read NXT extent?
innodb_stats_sample_pages=32 # from 8 for more accurate index cardinality
log_warnings=2 # from 1 to record addl connection error details
max_join_size=1000000000 # from more huge number to 1B rows max for join
sql_select_limit=1000000000 # from more huge number to 1B rows selected
query_alloc_block_size=32K # from 16K - minimize RAM alloc AvgQrySz 20K
query_cache_type=0 # for OFF, not used
query_cache_size=0 # from 8M not needed for one time selections
query_cache_limit=1K # from 256K, just to keep the framework, in case QC needed
query_cache_min_res_unit=512 # from 2K for optimal QC storage when QC used
query_prealloc_size=32K # from 24K to minimize RAM alloc for parsing
transaction_alloc_block_size=32K # from 8K to minimize RAM alloc
transaction_prealloc_size=32K # from 4K to minimize RAM alloc
updatable_views_with_limit=NO # from YES to reduce handler_external_lock count
this one could have to go back to YES - it is DYNAMIC variable
Please find your ONE com_stmt_prepare CODE and ensure it has CLOSE before leaving the session to release resources per refman PREPARE or mysql_stmt_prepare for API.
Once you are running, before running your troubled query, consider using
ALTER TABLE tbl_name ENGINE=INNODB
for each table to take advantage of additional pages for better cardinality,
then try your long running query.
I am have vps setup with ubuntu 14 server. I have lamp stack installed. Everything works perfect but mysql server stops frequently and i have to restart it manually. Here is my my.cnf mysql config file.
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html
# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
# Here is entries for some specific programs
# The following values assume you have at least 32M ram
# This was formally known as [safe_mysqld]. Both versions are currently parsed.
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
#
# * Basic Settings
#
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address = 127.0.0.1
#
# * Fine Tuning
#
key_buffer = 16M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 8
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover = BACKUP
#max_connections = 100
#table_cache = 64
#thread_concurrency = 10
#
# * Query Cache Configuration
#
query_cache_limit = 1M
query_cache_size = 16M
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
#general_log_file = /var/log/mysql/mysql.log
#general_log = 1
#
# Error log - should be very few entries.
#
log_error = /var/log/mysql/error.log
#
# Here you can see queries with especially long duration
#log_slow_queries = /var/log/mysql/mysql-slow.log
#long_query_time = 2
#log-queries-not-using-indexes
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
# other settings you may need to change.
#server-id = 1
#log_bin = /var/log/mysql/mysql-bin.log
expire_logs_days = 10
max_binlog_size = 100M
#binlog_do_db = include_database_name
#binlog_ignore_db = include_database_name
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
[mysql]
#no-auto-rehash # faster start of mysql but no tab completition
[isamchk]
key_buffer = 16M
#
# * IMPORTANT: Additional settings that can override those from this file!
# The files must end with '.cnf', otherwise they'll be ignored.
#
!includedir /etc/mysql/conf.d/
And following is my mysqltuner output.
-------- General Statistics ------------------------------------------------
[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.5.49-0ubuntu0.14.04.1
[OK] Operating on 64-bit architecture
-------- Storage Engine Statistics -------------------------------------------
[--] Status: +Archive -BDB -Federated +InnoDB -ISAM -NDBCluster
[--] Data in MyISAM tables: 38K (Tables: 34)
[--] Data in PERFORMANCE_SCHEMA tables: 0B (Tables: 17)
[--] Data in InnoDB tables: 11M (Tables: 101)
[!!] Total fragmented tables: 101
-------- Security Recommendations -------------------------------------------
[OK] All database users have passwords assigned
-------- Performance Metrics -------------------------------------------------
[--] Up for: 1h 13m 27s (2K q [0.464 qps], 130 conn, TX: 5M, RX: 332K)
[--] Reads / Writes: 81% / 19%
[--] Total buffers: 192.0M global + 2.7M per thread (151 max threads)
[!!] Maximum possible memory usage: 597.8M (121% of installed RAM)
[OK] Slow queries: 0% (0/2K)
[OK] Highest usage of available connections: 7% (12/151)
[OK] Key buffer size / total MyISAM indexes: 16.0M/196.0K
[OK] Query cache efficiency: 33.9% (493 cached / 1K selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 77 sorts)
[!!] Temporary tables created on disk: 33% (166 on disk / 498 total)
[OK] Thread cache hit rate: 90% (12 created / 130 connections)
[OK] Table cache hit rate: 94% (178 open / 189 opened)
[OK] Open file limit used: 11% (116/1K)
[OK] Table locks acquired immediately: 100% (1K immediate / 1K locks)
[OK] InnoDB data size / buffer pool: 11.9M/128.0M
-------- Recommendations -----------------------------------------------------
General recommendations:
Run OPTIMIZE TABLE to defragment tables for better performance
MySQL started within last 24 hours - recommendations may be inaccurate
Reduce your overall MySQL memory footprint for system stability
Enable the slow query log to troubleshoot bad queries
When making adjustments, make tmp_table_size/max_heap_table_size equal
Reduce your SELECT DISTINCT queries without LIMIT clauses
Variables to adjust:
*** MySQL's maximum memory usage is dangerously high ***
*** Add RAM before increasing MySQL buffer variables ***
tmp_table_size (> 16M)
max_heap_table_size (> 16M)
And here is my memory uses
Every 5.0s: free -m Sun Jul 3 03:12:12 2016
total used free shared buffers cached
Mem: 490 476 13 64 10 183
-/+ buffers/cache: 281 208
Swap: 0 0 0
What can be wrong here. Is there some misconfig on my my.cnf or something else is killing mysql. Please help me to find the issue.
Its possible that mysql runs out of memory. It happens when your system lacks swapping ( i.e. When memory gets exhausted, programs can't be transferred from main memory to a "swap" file in order to prevent system from memory failures ).
Swap explanation from Linux.com:
Linux divides its physical RAM (random access memory) into chucks of
memory called pages. Swapping is the process whereby a page of memory
is copied to the preconfigured space on the hard disk, called swap
space, to free up that page of memory. The combined sizes of the
physical memory and the swap space is the amount of virtual memory
available.
You can tweak "innodb_buffer_pool_size" to reduce indexing and caching footprints in the memory.
Ideally, you set the size of the buffer pool to as large a value as
practical, leaving enough memory for other processes on the server to
run without excessive paging. The larger the buffer pool, the more
InnoDB acts like an in-memory database, reading data from disk once
and then accessing the data from memory during subsequent reads.
Buffer pool size is configured using the innodb_buffer_pool_size
configuration option.
According to MySql documentation:
Solution:
Add this under [mysqld] : innodb_buffer_pool_size=64M
then make a swap file:
dd if=/dev/zero of=/swapfile bs=100M count=40
mkswap /swapfile
swapon /swapfile
and add this to /etc/fstab:
/swapfile none swap sw 0 0
I am having a 4 GB vps box. At any time there are not more than 40 visitors browsing 10-15 page simultaneously. Suddenly 2 days back server cpu usage touching almost 2000%. Strange thing is when i stop the apache, it still does not go down. below is the top command result.
PID USER PR NI VIRT RES SHR S CPU MEM TIME+ COMMAND
23644 mysql 15 0 12.7g 742m 6840 S 1812.0 18.1 1107:15 mysqld
14076 xyzaz 19 0 194m 16m 7348 S 7.1 0.4 0:00.22 php
14080 xyzaz 18 0 195m 17m 7276 S 6.5 0.4 0:00.20 php
i have troubleshooted every thing but no luck. I dont know it was working fine but suddenly how it changed to such a high value.
below is the output of my.cnf
[mysqld]
local-infile=0
max_connections=500
key_buffer_size=128M
myisam_sort_buffer_size=64M
join_buffer_size=2M
read_buffer_size=1M
sort_buffer_size=2M
max_heap_table_size=16M
table_cache=1800
thread_cache_size=384
interactive_timeout=25
wait_timeout=7200
connect_timeout=10
max_allowed_packet=268435456
max_connect_errors=100
query_cache_limit=4M
query_cache_size=128M
query_cache_type=1
tmp_table_size=64M
skip-name-resolve
innodb_buffer_pool_size=229M
default-storage-engine=MyISAM
innodb_file_per_table=1
[mysqld_safe]
open_files_limit = 6553
[mysqldump]
quick
max_allowed_packet=16M
[myisamchk]
key_buffer_size=64M
sort_buffer=64M
read_buffer=16M
write_buffer=16M
Any help is really appriciated
what best config for my.cnf file i have big size sql
Total processors: 4
Processor #1
Vendor
GenuineIntel
Name
Intel(R) Core(TM) i7-3930K CPU # 3.20GHz
Speed
3200.058 MHz
Cache
12288 KB
Memory Information
Memory: 14243308k/15728640k available
There is no straight forward answer of this question as one configuration can be suitable for one but not for other..
If you are using innodb engine then You can start with below configuration initially and can optimize as per your requirement/problems-
default_storage_engine = InnoDB
innodb_file_per_table
lower_case_table_names = 1
wait_timeout = 120
innodb_buffer_pool_size = 10G
key_buffer_size = 20M
max_allowed_packet = 64M
max_connections = 100
query_cache_type = 1
query_cache_size = 20M
query_cache_limit = 2M
read_buffer_size = 2M
sort_buffer_size = 2M
join_buffer_size = 2M
tmp_table_size = 512M
max_heap_table_size = 512M
for the mysql optimization, I will suggest you please try with MySQL tuner script. You can try it with the following command
wget https://launchpad.net/mysql-tuning-primer/trunk/1.6-r1/+download/tuning-primer.sh
chmod 755 tuning-primer.sh
./tuning-primer.sh