My mysqld process consumes 232% CPU and and there 14000+ connections
(I'm a little new to this thing but following Stack Overflow for assistance).
top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3112 mysql 20 0 7061444 1.397g 15848 S 232.6 8.9 1138:06 mysqld
System:
Ubuntu 18.04,
16GB RAM,
8 Core CPU,
120GB Disk
and MySQL version 5.7.25
mysql> show status like 'Conn%';
+-----------------------------------+-------+
| Variable_name | Value |
+-----------------------------------+-------+
| Connection_errors_accept | 0 |
| Connection_errors_internal | 0 |
| Connection_errors_max_connections | 0 |
| Connection_errors_peer_address | 0 |
| Connection_errors_select | 0 |
| Connection_errors_tcpwrap | 0 |
| Connections | 14007 |
+-----------------------------------+-------+
7 rows in set (0.01 sec)
And show variables like "%timeout%"
mysql> show variables like "%timeout%";
+-----------------------------+----------+
| Variable_name | Value |
+-----------------------------+----------+
| connect_timeout | 10 |
| delayed_insert_timeout | 300 |
| have_statement_timeout | YES |
| innodb_flush_log_at_timeout | 1 |
| innodb_lock_wait_timeout | 50 |
| innodb_rollback_on_timeout | OFF |
| interactive_timeout | 28800 |
| lock_wait_timeout | 31536000 |
| net_read_timeout | 30 |
| net_write_timeout | 60 |
| rpl_stop_slave_timeout | 31536000 |
| slave_net_timeout | 60 |
| wait_timeout | 28800 |
+-----------------------------+----------+
13 rows in set (0.01 sec)
And mysqld.cnf settings
[mysqld]
# Skip reverse DNS lookup of clients
skip-name-resolve
default-storage-engine=InnoDB
max_allowed_packet=500M
max_connections = 256
interactive_timeout=7200
wait_timeout=7200
innodb_file_per_table=1
innodb_buffer_pool_size = 8G
innodb_buffer_pool_instances = 4
innodb_log_file_size = 1G
innodb_flush_log_at_trx_commit = 1
innodb_flush_method = O_DIRECT
innodb_open_files=5000
innodb_io_capacity=2000
innodb_io_capacity_max=4000
innodb_old_blocks_time=2000
open_files_limit=50000
query_cache_type = 1
query_cache_min_res_unit = 1M
query_cache_limit = 1M
query_cache_size = 50M
tmp_table_size= 256M
max_heap_table_size= 256M
#key_buffer_size = 128M
thread_stack = 128K
thread_cache_size = 32
slow-query-log = 1
slow-query-log-file = /var/lib/mysql/mysql-slow.log
long_query_time = 1
Note: Corrected above mysqld.cnf values to match with below reports attached
Additional Info:
htop:- https://pastebin.com/43f4b3fK
top:- https://pastebin.com/rTh1XvUt
GLOBAL VARIABLES: https://pastebin.com/K2fgKwEv (Complete)
INNODB STATUS:- https://pastebin.com/nGrZjHAg
Mysqltuner:- https://pastebin.com/ZNYieJj8
[SHOW FULL PROCESSLIST], [ulimit -a], [iostat -xm], [lscpu] :- https://pastebin.com/mrnyQrXf
Server freezes when multiple db transaction is being carried out. Is there a lock like thing or any configuration flaws?
(Background: This is a WordPress blog and nobody else is accessing it right now. I somehow imported a 115K posts from an old blog but struck here with this CPU ghost)
Rate Per Second = RPS - Suggestions to consider for your my.cnf [mysqld] section,
innodb_lru_scan_depth=100 # from 1024 to reduce 90% of cpu cycles used for function every SECOND
innodb_io_capacity=3500 # from 2000 to enable higher IOPS on your SSD devices
innodb_flushing_avg_loops=5 # from 30 to reduce innodb_buffer_pool_pages_dirty overhead - count was 3183 in SGStatus
read_buffer_size=256K # from 128K to reduce handler_read_next RPS of 277,134
read_rnd_buffer_size=192K # from 256K to reduce handler_read_rnd_next RPS of 778
There are many more opportunities to improve performance through Global Variables.
Disclaimer: I am the author of web site mentioned in my profile, Network profile that includes contact information.
A likely cause of high CPU and poor performance is the poor schema for wp_postmeta. I discuss the remedy here.
Meanwhile, "you can't tune your way out of a performance problem". I did glance at the settings -- all are "reasonable".
Related
We run a CentOS DirectAdmin install with MariaDB 10.2.14 where on Magento is installed.
Currently our DB locks very often when a process runs, so all other processes are waiting until the current process finishes. This is quite a problem, because for example, also the adding to cart process is waiting in that case and people can not order.
How can we prevent the DB from being locked so long and solve this issue?
Server:
6x Intel Xeon
32GB RAM
500GB SSD
My.cnf:
[mysqld]
bind-address = 127.0.0.1
local-infile=0
innodb_file_per_table=1
innodb_file_format=barracuda
slow_query_log = 1
slow_query_log_file=/var/log/mysql-log-slow-queries.log
key_buffer = 250M
key_buffer_size = 250M
max_allowed_packet = 128M
table_cache = 512
sort_buffer_size = 7M
read_buffer_size = 7M
read_rnd_buffer_size = 7M
myisam_sort_buffer_size = 64M
tmp_table_size = 190M
query_cache_type = 1
query_cache_size = 220M
query_cache_limit = 512M
thread_cache_size = 150
max_connections = 225
wait_timeout = 300
innodb_buffer_pool_size = 7G
max_heap_table_size =180M
innodb_log_buffer_size = 36M
join_buffer_size = 32M
innodb_buffer_pool_instances = 7
long_query_time = 15
table_definition_cache = 4K
open_files_limit = 60K
table_open_cache = 50767
innodb_log_file_size= 128M
innodb_lock_wait_timeout = 700
Suggestions to consider for your my.cnf [mysqld] section
The following lead with # to disable or REMOVE to allow defaults to serve your requirements
Some of these are already mentioned by Rick James in earlier comment.
. key_buffer
. key_buffer_size
. table_cache
. sort_buffer_size
. read_buffer_size
. read_rnd_buffer_size
. MyISAM_sort_buffer_size
. join_buffer_size
. long_query_time
. innodb_lock_wait_timeout
make these changes or add lines to your my.cnf for
query_cache_type=0 # from 1 to turn OFF QC and conserve CPU cycles
query_cache_size=0 # from 220M to conserve RAM for more useful work
query_cache_limit=0 # from 512M to conserve RAM for more useful work
thread_cache_size=100 # from 150 V8 refman suggested CAP to avoid OOM
innodb_lru_scan_depth=100 # from 1024 to minimum to conserve CPU every SECOND
innodb_flush_neighbors=0 # from 1 no need to waste CPU cycles when using SSD
innodb_io_capacity_max=10000 # from 2000 since you have SSD
innodb_io_capacity=5000 # from 200 to use more of your SSD capability
for additional assistance, please check my profile, clk Network profile for contact info.
MySQL will wait a certain amount of time for the lock to be removed before it gives up and throws that error. If you are able to track when you are seeing these error messages down to any consistent time of the day, you should look at what else the server is doing at that time - for instance is a database backup running. By doing this you should be able to narrow down possibilities for what processes could be creating the lock although it's not always that straight forward to do - likely to be a bit of trial and error.
Sometimes deadlock issues can be caused on the database.The reason behind this issue is if you are running a lot of custom scripts and killing the scripts before the database connection gets chance to close.
If you can login to MySQL from CLI and run the following command
SHOW PROCESSLIST;
you will get the following output
+———+—————–+——————-+—————–+———+——+——-+——————+———–+—————+———–+
| Id | User | Host | db | Command | Time | State | Info | Rows_sent | Rows_examined | Rows_read |
+———+—————–+——————-+—————–+———+——+——-+——————+———–+—————+———–+
| 6794372 | db_user| 111.11.0.65:21532 | db_name| Sleep | 3800 | | NULL | 0 | 0 | 0 |
| 6794475 | db_user| 111.11.0.65:27488 | db_name| Sleep | 3757 | | NULL | 0 | 0 | 0 |
| 6794550 | db_user| 111.11.0.65:32670 | db_name| Sleep | 3731 | | NULL | 0 | 0 | 0 |
| 6794797 | db_user| 111.11.0.65:47424 | db_name | Sleep | 3639 | | NULL | 0 | 0 | 0 |
| 6794909 | db_user| 111.11.0.65:56029 | db_name| Sleep | 3591 | | NULL | 0 | 0 | 0 |
| 6794981 | db_user| 111.11.0.65:59201 | db_name| Sleep | 3567 | | NULL | 0 | 0 | 0 |
| 6795096 | db_user| 111.11.0.65:2390 | db_name| Sleep | 3529 | | NULL | 0 | 0 | 0 |
| 6795270 | db_user| 111.11.0.65:10125 | db_name | Sleep | 3473 | | NULL | 0 | 0 | 0 |
| 6795402 | db_user| 111.11.0.65:18407 | db_name| Sleep | 3424 | | NULL | 0 | 0 | 0 |
| 6795701 | db_user| 111.11.0.65:35679 | db_name| Sleep | 3330 | | NULL | 0 | 0 | 0 |
| 6800436 | db_user| 111.11.0.65:57815 | db_name| Sleep | 1860 | | NULL | 0 | 0 | 0 |
| 6806227 | db_user| 111.11.0.67:20650 | db_name| Sleep | 188 | | NULL | 1 | 0 | 0 |
| 6806589 | db_user| 111.11.0.65:36618 | db_name| Query | 0 | NULL | SHOW PROCESSLIST | 0 | 0 | 0 |
| 6806742 | db_user| 111.11.0.75:38717 | db_name| Sleep | 0 | | NULL | 0 | 0 | 0 |
| 6806744 | db_user| 111.11.0.75:38819 | db_name| Sleep | 0 | | NULL | 61 | 61 | 61 |
+———+—————–+——————-+—————–+———+——+——-+——————+———–+—————+———–+
15 rows in set (0.00 sec)
You can see as an example
6794372 the command is sleep and time is 3800. This is preventing other operations.
These processes should be killed 1 by 1 using the command.
KILL 6794372;
Once you have killed all the sleeping connections, things should start working as normal again
These are deprecated; their names have changed. Remove them:
key_buffer = 250M
table_cache = 512
These are higher than they should be:
key_buffer_size = 250M
query_cache_size = 220M
thread_cache_size = 150
long_query_time = 15
table_definition_cache = 4K
table_open_cache = 50767
innodb_lock_wait_timeout = 700
The last one may be the villain. It implies that you have some loooong transactions. This is a design flaw in your code. Find a way to make the transactions shorter. If you need help, describe what you are doing to us.
I feel that 5 is plenty long for a transaction.
Do you sometimes get this?
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
I recently upgraded MySQL 5.1 to 5.7.8rc.
I have a unique issue with "sending data" status during profiling. It takes more time than expected for every complex or union queries (some time for simple queries also). I tried with best optimized as well as original query. Googled and tried all the possible configurations but no luck.
Query performs supper fast in MySQL 5.1, but not in 5.7.
Also tried with table optimization, analyze, repair etc..
some details for reference:
OS: Centos 6.9 64 bit
MySQL: 5.7.8 rc
CPU: 4
RAM: 64 GB
Data volume: 450 GB
Type: Dedicated VM
Query Profiling:
Status Duration
starting 0.000515
checking permissions 0.000023
checking permissions 0.000016
checking permissions 0.000014
checking permissions 0.000014
checking permissions 0.000014
checking permissions 0.000014
checking permissions 0.000014
checking permissions 0.000014
checking permissions 0.000016
checking permissions 0.000014
checking permissions 0.000019
Opening tables 0.000079
init 0.000325
System lock 0.000068
optimizing 0.000079
statistics 0.001888
preparing 0.000151
Creating tmp table 0.000128
Sorting result 0.000027
optimizing 0.000034
statistics 0.000064
preparing 0.000047
Creating tmp table 0.000047
Sorting for group 0.000030
optimizing 0.000018
statistics 0.000022
preparing 0.000022
Creating tmp table 0.000043
Sorting for order 0.000023
executing 0.000016
Sending data 4.015596
Creating sort index 0.004766
executing 0.000010
Sending data 0.000159
executing 0.000008
Sending data 0.000025
Creating sort index 0.000349
end 0.000010
query end 0.000024
removing tmp table 0.000013
query end 0.000011
removing tmp table 0.000008
query end 0.000009
removing tmp table 0.000011
query end 0.000009
removing tmp table 0.000007
query end 0.000010
removing tmp table 0.000008
query end 0.000006
closing tables 0.000026
freeing items 0.000039
removing tmp table 0.000009
freeing items 0.000067
logging slow query 0.000047
cleaning up 0.000042
Execution Plan:
+----+--------------+-------------+------------+--------+-----------------------------------------------------+--------------------+---------+--------------------------------------+------+----------+------------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+--------------+-------------+------------+--------+-----------------------------------------------------+--------------------+---------+--------------------------------------+------+----------+------------------------------------+
| 1 | PRIMARY | TIGM_GRP | NULL | index | PRIMARY,FLD_PARENT_GROUP_ID | PRIMARY | 2 | NULL | 87 | 0.33 | Using where |
| 1 | PRIMARY | TIPLD | NULL | range | FLD_PRICE_LEVEL_ID | FLD_PRICE_LEVEL_ID | 3 | NULL | 26 | 10.00 | Using index condition; Using where |
| 1 | PRIMARY | TIPLM | NULL | eq_ref | PRIMARY | PRIMARY | 2 | TIPLD.FLD_PRICE_LEVEL_ID | 1 | 10.00 | Using where |
| 1 | PRIMARY | TIGL | NULL | ref | FLD_GROUP_ID,fld_item_id | FLD_GROUP_ID | 2 | TIGM_GRP.FLD_GROUP_ID | 404 | 100.00 | NULL |
| 1 | PRIMARY | TIM | NULL | eq_ref | PRIMARY,FLD_ITEM_TYPE,FLD_ADDON_ID,FLD_PRODUCT_TYPE | PRIMARY | 4 | TIGL.FLD_ITEM_ID | 1 | 50.00 | Using where |
| 1 | PRIMARY | TSIAM | NULL | eq_ref | PRIMARY | PRIMARY | 4 | TIM.FLD_ADDON_ID | 1 | 10.00 | Using where |
| 1 | PRIMARY | TSIARM | NULL | ref | FLD_ADDON_ID | FLD_ADDON_ID | 5 | TIM.FLD_ADDON_ID | 1 | 10.00 | Using index condition; Using where |
| 1 | PRIMARY | TIPD | NULL | ref | FLD_ITEM_ID,FLD_PRICE_LEVEL_ID | FLD_ITEM_ID | 2 | TIM.FLD_ITEM_ID | 3 | 1.35 | Using index condition; Using where |
| 1 | PRIMARY | TIGM_PARENT | NULL | eq_ref | PRIMARY | PRIMARY | 2 | TIGM_GRP.FLD_PARENT_GROUP_ID | 1 | 100.00 | Using index |
| 2 | UNION | TIGM_GRP | NULL | index | PRIMARY,FLD_PARENT_GROUP_ID | PRIMARY | 2 | NULL | 87 | 0.06 | Using where |
| 2 | UNION | TIGM_PARENT | NULL | eq_ref | PRIMARY | PRIMARY | 2 | TIGM_GRP.FLD_PARENT_GROUP_ID | 1 | 100.00 | Using index |
|NULL| UNION RESULT | <union1,2> | NULL | ALL | NULL | NULL | NULL | NULL | NULL | NULL | Using temporary; Using filesort |
+----+--------------+-------------+------------+--------+-----------------------------------------------------+--------------------+---------+--------------------------------------+------+----------+------------------------------------+
my.cnf:
[mysql]
# CLIENT #######################################################################
port = 3306
socket = /var/lib/mysql/mysql.sock
[mysqld]
# GENERAL ######################################################################
user = mysql
port = 3306
socket = /var/lib/mysql/mysql.sock
server_id = 32108
default_storage_engine = InnoDB
pid_file = /var/run/mysqld/mysqld.pid
optimizer_prune_level = 0
optimizer_search_depth = 0
max_length_for_sort_data = 8388608 #New
net_buffer_length = 1048576 #New
back_log = 80
symbolic-links = 0
log_bin_trust_function_creators = 1
net_read_timeout = 10 #90
net_write_timeout = 10 #120
net_retry_count = 30
thread_stack = 512K #192K
long_query_time = 10
tmpdir = /tmp
# MyISAM #######################################################################
key_buffer_size = 64M
read_buffer_size = 32M
read_rnd_buffer_size = 32M
bulk_insert_buffer_size = 16M
myisam_sort_buffer_size = 128M
myisam_max_sort_file_size = 1G
myisam_repair_threads = 1
memlock
max_allowed_packet = 32M
max_connect_errors = 100
sql_mode = STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
sysdate-is-now = 1
explicit_defaults_for_timestamp = 1
innodb = FORCE
# Password policy disabled as per communication 05-Sep2017
validate_password = OFF
# DATA STORAGE ##################################################################
datadir = /var/lib/mysql/
# BINARY LOGGING ################################################################
log-bin = /var/lib/mysql/mysql-bin
expire-logs-days = 14
sync-binlog = 1
# REPLICATION ###################################################################
skip-slave-start = 1
relay-log = /var/log/mysql-realy-logs/relay-bin
slave-skip-errors = 1062 #,1053,1032,1237,1146
slave-net-timeout = 60
relay_log_purge = 1
# CACHES AND LIMITS #############################################################
tmp-table-size = 256M
max-heap-table-size = 256M
query_cache_min_res_unit = 12288 #8192 #New
query-cache-type = 1
query-cache-size = 32M #102400 #0 #256M
max-connections = 150
thread-cache-size = 10 #-1 #Auto resized
open-files-limit = 65535
table_definition_cache = 2000
table_open_cache = 4096 #3092
table_open_cache_instances = 4
sort_buffer_size = 128M
join_buffer_size = 512M
binlog_cache_size = 16M
query_cache_limit = 4M
# INNODB ########################################################################
innodb_fast_shutdown = 1
innodb_flush_method = O_DIRECT
innodb_log_group_home_dir = /mysql_redo_logs/mysql_redo_logs
innodb_log_files_in_group = 2
innodb_log_file_size = 1G
innodb_flush_log_at_trx_commit = 2
innodb_file_per_table = 1 #ON
innodb_buffer_pool_size = 32G #20G
innodb_buffer_pool_instances = 32
innodb_log_buffer_size = 64M
innodb_adaptive_hash_index = 1 #ON
innodb_thread_concurrency = 300 # "0" is default and means infinite (as and when needed). #250 #32
innodb_thread_sleep_delay = 1
innodb_flush_neighbors = 1
innodb_sync_array_size = 832 # Default is 768
skip-innodb_doublewrite #New
innodb_page_cleaners = 32 # Must be =innodb_buffer_pool_instances
innodb_sort_buffer_size = 512M
innodb_read_io_threads = 64
innodb_write_io_threads = 16
#innodb_concurrency_tickets = 429496729
innodb_max_dirty_pages_pct = 90
innodb_lock_wait_timeout = 10 #80
innodb_compression_level = 0 #New
innodb_lru_scan_depth = 512 #1024 #Default
# LOGGING #######################################################################
log_error = /var/lib/mysql/mysql-error.log
log_queries_not_using_indexes = 1
slow_query_log = 1
log_error_verbosity = 3
[mysqld_safe]
open-files-limit = 4096
malloc-lib = /usr/lib64/libtcmalloc_minimal.so.4
[mysqldump]
quick
max_allowed_packet = 64M
So far I have tried most of the options, currently I started the service in caching mode for quick response.
Can you please help me fix this "sending data" delay issue
Everyone
I think I figured out the issue.
1) 5.6 the optimization engine and plan generated is a result mixture of meta-data and user query (non relational joins). engine trusts user and get best plan.
But 5.7 does not trust soft relations between the tables though they are indexed.
2) 5.7 expects explicit data integrity and do not believe on user.
POC:
I created the same tables involved with hard FK references, wala ! found a miraculous response. the same query which executed without FK took 6.085 Sec (sending data),
and the query with data integrity enforced took 0.05 Sec (with no change in indexing).
So I think 5.7 expects strong data integrity constraints explicitly.
A lot of Optimizer changes happened between 5.1 and 5.7. For one thing, ICP is new; I see it several times in the EXPLAIN. We need to see the query to discuss further. If possible, get the EXPLAIN from 5.1.
Meanwhile, here are some minor comments on my.cnf:
thread_stack = 512K #192K
Generally, it is not useful to increase this setting.
optimizer_prune_level = 0
optimizer_search_depth = 0
What prompted you to set to 0?
innodb = FORCE
deprecated in 5.7.5; suggest removing before it becomes an error.
slave-skip-errors = 1062 #,1053,1032,1237,1146
Sweeping gremlins under the rug -- you will stub your toes on they lump.
innodb_buffer_pool_size = 32G #20G
Unless you have some large apps on the same machine, 44G might be a little better.
innodb_page_cleaners = 32 # Must be =innodb_buffer_pool_instances
re the comment -- Technically, it is auto-limited to pool_instances; see https://dev.mysql.com/doc/refman/8.0/en/innodb-parameters.html#sysvar_innodb_page_cleaners (Thanks, WilsonHauck)
log_queries_not_using_indexes = 1
In my opinion, this clutters the slowlog. The interesting entries are those that exceed long_query_time. Do you have any interesting queries in the slowlog?
I am getting Too many connections error from Magento.
I have increased the max_connection to 1000 but I am still getting the error.
I contacted hosting provider and they asked me to use command show processlist; and review my coding.
When I ran the command, I only saw few active connections (about 4 to 5 connection). Therefore, I have no clue how to fix the problem.
I have increased the max_connection to 1500 and I am getting create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug error now.
Could anyone can help me with this situation please?
I am grateful for your help and time.
This is my my.cnf
key_buffer = 384M
max_allowed_packet = 1M
table_cache = 1024
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 8M
myisam_sort_buffer_size = 64M
thread_cache_size = 16
query_cache_type = 1
query_cache_size = 48M
log_slow_queries=/var/log/mysqld.slowquery.log
max_connections=1000
wait_timeout=120
tmp_table_size = 64M
max_heap_table_size = 64M
innodb_buffer_pool_size = 2048M
innodb_additional_mem_pool_size = 20M
open_files_limit=34484
#
And this is show proccesslist
+-------+-----------+-----------+--------------------+----------------+-------+--------------------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-------+-----------+-----------+--------------------+----------------+-------+--------------------+------------------+
| 4729 | root | localhost | abc_def| Sleep | 13093 | | NULL |
| 16282 | eximstats | localhost | eximstats | Sleep | 84 | | NULL |
| 16283 | DELAYED | localhost | eximstats | Delayed insert | 84 | Waiting for INSERT | |
| 16343 | root | localhost | NULL | Query | 0 | NULL | show processlist |
+-------+-----------+-----------+--------------------+----------------+-------+--------------------+------------------+
4 rows in set (0.00 sec)
You can increase max connections, but just so much
In essence there are too many connections being started, or not closed
So you can
increase max connections to allow more connections being started
Reduce wait_timeout for the connections that have gone away to free up again
Investigate where all these connections request are coming from? (can often be bots, or may bots at once, some index update or other cronjob etc)
thanks, hope it helps
In my configuration innodb_flush_method=O_DSYNC from O-DIRECT reduces about 75% the iowait, and accordingly this the load. Should I set another variables besides innodb_flush_method to reduce more the iowait?
My configuration file is:
[mysqld]
innodb_file_per_table=1
query_cache_size=128M
thread_cache_size=64
key_buffer_size=32M
max_allowed_packet=16M
table_cache=1024
table_definition_cache=8192
wait_timeout=20
max_user_connections=25
innodb_flush_method=O_DSYNC
open_files_limit=16384
myisam_sort_buffer_size=2M
collation_server=utf8_unicode_ci
character_set_server=utf8
tmp_table_size = 384M
max_heap_table_size = 384M
innodb_buffer_pool_size=64M
innodb_thread_concurrency=8
max_connections=125
I have a database with 100 Innodb tables, 3 of them has about 25000 records, the others has no significant records. The average queries in peak time is about 160, the majority is SELECT
innodb_buffer_pool_size
Major problem is innodb_buffer_pool_size is too small. Recommandation is set to 50~75% of main memory.
innodb_buffer_pool_size=64M
I strongly recommand that you should increase it's value.
Generally speaking, O_DIRECT is little bit fast because InnoDB Buffer Pool caches Data+Index, So with O_DIRECT disabled File System Page Cache is faster. MySQL Manaual says
( http://dev.mysql.com/doc/refman/5.1/en/innodb-parameters.html#sysvar_innodb_flush_method)
Depending on hardware configuration, setting innodb_flush_method to O_DIRECT can either have either a positive or negative effect on performance. Benchmark your particular configuration to decide which setting to use.
But in my experience, there was no significant difference between O_DIRECT and O_DSYNC. Both SSD and HDD are test.
Anyway you should increase innodb_buffer_pool_size.
Calculating innodb buffer pool hit ratio
mysql> SHOW GLOBAL STATUS LIKE '%innodb%';
+---------------------------------------+-------------+
| Variable_name | Value |
+---------------------------------------+-------------+
.....
.....
| Innodb_buffer_pool_read_requests | 11054273949 |
| Innodb_buffer_pool_reads | 135237 |
| Innodb_buffer_pool_wait_free | 0 |
....
innodb buffer pool hit ratio = ((Innodb_buffer_pool_read_requests) / (Innodb_buffer_pool_read_requests + Innodb_buffer_pool_reads)) * 100
E.g. above examples,
hit ratio = (11054273949 / (11054273949 + 135237)) * 100 = 99.99%
I think this value is too small in your case.
query_cache_size
"the majority is SELECT"
If most queries are SELECT and update query is rare, I think increasing query_cache_size is very helpful for you.
Could you post your query cache status as follows?
mysql> show global status like 'Qc%';
+-------------------------+------------+
| Variable_name | Value |
+-------------------------+------------+
| Qcache_free_blocks | 13 |
| Qcache_free_memory | 1073403104 |
| Qcache_hits | 217949 |
| Qcache_inserts | 337009 |
| Qcache_lowmem_prunes | 0 |
| Qcache_not_cached | 2122598 |
| Qcache_queries_in_cache | 68 |
| Qcache_total_blocks | 167 |
+-------------------------+------------+
mysql> show global status like 'com_select%';
+---------------+---------+
| Variable_name | Value |
+---------------+---------+
| Com_select | 3292531 |
+---------------+---------+
1 row in set (0.00 sec)
Calculating innodb buffer pool hit ratio
query cache hit ratio = ((Qcache_hits) / (Qcache_hits + Com_select)) * 100
first, figure out your query cache hit ratio.
I was running benchmark test in mysql server using mysqlslap. But sometimes the last thread in the pool is not executing and closing the connection. It keeps in sleep state.
mysql> show full processlist;
+-----+------+-----------------------+-----------+---------+------+-------+-----------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-----+------+-----------------------+-----------+---------+------+-------+-----------------------+
| 2 | root | 122.178.231.xxx:15832 | mysqlslap | Sleep | 740 | | NULL |
| 106 | root | localhost | NULL | Query | 0 | NULL | show full processlist |
+-----+------+-----------------------+-----------+---------+------+-------+-----------------------+
2 rows in set (0.00 sec)
Any idea? It happens often :(
This was the command I executed.
mysqlslap -h IP_address -u root --auto-generate-sql --concurrency=100
-pPASSWORD -vvv --number-of-queries=2500
my.cnf file is as follows,
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
skip-bdb
max_connections = 500
query_cache_type = 1
query_cache_size = 128M
sort_buffer_size = 64M
key_buffer_size = 425M
query_cache_limit = 1024M
thread_cache_size = 4M
innodb_buffer_pool_size = 1360M
innodb_log_file_size = 512M
long_query_time = 5
innodb_thread_concurrency = 8
innodb_lock_wait_timeout = 300
max_allowed_packet = 128M
net_read_timeout = 240
net_write_timeout = 240
table_lock_wait_timeout =240
connect_timeout = 30
log-warnings = 2
log = /var/log/mysqld.log
log-error = /var/log/mysqld.error.log
log-slow-queries = /var/log/mysql_slow_queries.log