I am getting Too many connections error from Magento.
I have increased the max_connection to 1000 but I am still getting the error.
I contacted hosting provider and they asked me to use command show processlist; and review my coding.
When I ran the command, I only saw few active connections (about 4 to 5 connection). Therefore, I have no clue how to fix the problem.
I have increased the max_connection to 1500 and I am getting create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug error now.
Could anyone can help me with this situation please?
I am grateful for your help and time.
This is my my.cnf
key_buffer = 384M
max_allowed_packet = 1M
table_cache = 1024
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 8M
myisam_sort_buffer_size = 64M
thread_cache_size = 16
query_cache_type = 1
query_cache_size = 48M
log_slow_queries=/var/log/mysqld.slowquery.log
max_connections=1000
wait_timeout=120
tmp_table_size = 64M
max_heap_table_size = 64M
innodb_buffer_pool_size = 2048M
innodb_additional_mem_pool_size = 20M
open_files_limit=34484
#
And this is show proccesslist
+-------+-----------+-----------+--------------------+----------------+-------+--------------------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-------+-----------+-----------+--------------------+----------------+-------+--------------------+------------------+
| 4729 | root | localhost | abc_def| Sleep | 13093 | | NULL |
| 16282 | eximstats | localhost | eximstats | Sleep | 84 | | NULL |
| 16283 | DELAYED | localhost | eximstats | Delayed insert | 84 | Waiting for INSERT | |
| 16343 | root | localhost | NULL | Query | 0 | NULL | show processlist |
+-------+-----------+-----------+--------------------+----------------+-------+--------------------+------------------+
4 rows in set (0.00 sec)
You can increase max connections, but just so much
In essence there are too many connections being started, or not closed
So you can
increase max connections to allow more connections being started
Reduce wait_timeout for the connections that have gone away to free up again
Investigate where all these connections request are coming from? (can often be bots, or may bots at once, some index update or other cronjob etc)
thanks, hope it helps
Related
My mysqld process consumes 232% CPU and and there 14000+ connections
(I'm a little new to this thing but following Stack Overflow for assistance).
top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3112 mysql 20 0 7061444 1.397g 15848 S 232.6 8.9 1138:06 mysqld
System:
Ubuntu 18.04,
16GB RAM,
8 Core CPU,
120GB Disk
and MySQL version 5.7.25
mysql> show status like 'Conn%';
+-----------------------------------+-------+
| Variable_name | Value |
+-----------------------------------+-------+
| Connection_errors_accept | 0 |
| Connection_errors_internal | 0 |
| Connection_errors_max_connections | 0 |
| Connection_errors_peer_address | 0 |
| Connection_errors_select | 0 |
| Connection_errors_tcpwrap | 0 |
| Connections | 14007 |
+-----------------------------------+-------+
7 rows in set (0.01 sec)
And show variables like "%timeout%"
mysql> show variables like "%timeout%";
+-----------------------------+----------+
| Variable_name | Value |
+-----------------------------+----------+
| connect_timeout | 10 |
| delayed_insert_timeout | 300 |
| have_statement_timeout | YES |
| innodb_flush_log_at_timeout | 1 |
| innodb_lock_wait_timeout | 50 |
| innodb_rollback_on_timeout | OFF |
| interactive_timeout | 28800 |
| lock_wait_timeout | 31536000 |
| net_read_timeout | 30 |
| net_write_timeout | 60 |
| rpl_stop_slave_timeout | 31536000 |
| slave_net_timeout | 60 |
| wait_timeout | 28800 |
+-----------------------------+----------+
13 rows in set (0.01 sec)
And mysqld.cnf settings
[mysqld]
# Skip reverse DNS lookup of clients
skip-name-resolve
default-storage-engine=InnoDB
max_allowed_packet=500M
max_connections = 256
interactive_timeout=7200
wait_timeout=7200
innodb_file_per_table=1
innodb_buffer_pool_size = 8G
innodb_buffer_pool_instances = 4
innodb_log_file_size = 1G
innodb_flush_log_at_trx_commit = 1
innodb_flush_method = O_DIRECT
innodb_open_files=5000
innodb_io_capacity=2000
innodb_io_capacity_max=4000
innodb_old_blocks_time=2000
open_files_limit=50000
query_cache_type = 1
query_cache_min_res_unit = 1M
query_cache_limit = 1M
query_cache_size = 50M
tmp_table_size= 256M
max_heap_table_size= 256M
#key_buffer_size = 128M
thread_stack = 128K
thread_cache_size = 32
slow-query-log = 1
slow-query-log-file = /var/lib/mysql/mysql-slow.log
long_query_time = 1
Note: Corrected above mysqld.cnf values to match with below reports attached
Additional Info:
htop:- https://pastebin.com/43f4b3fK
top:- https://pastebin.com/rTh1XvUt
GLOBAL VARIABLES: https://pastebin.com/K2fgKwEv (Complete)
INNODB STATUS:- https://pastebin.com/nGrZjHAg
Mysqltuner:- https://pastebin.com/ZNYieJj8
[SHOW FULL PROCESSLIST], [ulimit -a], [iostat -xm], [lscpu] :- https://pastebin.com/mrnyQrXf
Server freezes when multiple db transaction is being carried out. Is there a lock like thing or any configuration flaws?
(Background: This is a WordPress blog and nobody else is accessing it right now. I somehow imported a 115K posts from an old blog but struck here with this CPU ghost)
Rate Per Second = RPS - Suggestions to consider for your my.cnf [mysqld] section,
innodb_lru_scan_depth=100 # from 1024 to reduce 90% of cpu cycles used for function every SECOND
innodb_io_capacity=3500 # from 2000 to enable higher IOPS on your SSD devices
innodb_flushing_avg_loops=5 # from 30 to reduce innodb_buffer_pool_pages_dirty overhead - count was 3183 in SGStatus
read_buffer_size=256K # from 128K to reduce handler_read_next RPS of 277,134
read_rnd_buffer_size=192K # from 256K to reduce handler_read_rnd_next RPS of 778
There are many more opportunities to improve performance through Global Variables.
Disclaimer: I am the author of web site mentioned in my profile, Network profile that includes contact information.
A likely cause of high CPU and poor performance is the poor schema for wp_postmeta. I discuss the remedy here.
Meanwhile, "you can't tune your way out of a performance problem". I did glance at the settings -- all are "reasonable".
I recently upgraded MySQL 5.1 to 5.7.8rc.
I have a unique issue with "sending data" status during profiling. It takes more time than expected for every complex or union queries (some time for simple queries also). I tried with best optimized as well as original query. Googled and tried all the possible configurations but no luck.
Query performs supper fast in MySQL 5.1, but not in 5.7.
Also tried with table optimization, analyze, repair etc..
some details for reference:
OS: Centos 6.9 64 bit
MySQL: 5.7.8 rc
CPU: 4
RAM: 64 GB
Data volume: 450 GB
Type: Dedicated VM
Query Profiling:
Status Duration
starting 0.000515
checking permissions 0.000023
checking permissions 0.000016
checking permissions 0.000014
checking permissions 0.000014
checking permissions 0.000014
checking permissions 0.000014
checking permissions 0.000014
checking permissions 0.000014
checking permissions 0.000016
checking permissions 0.000014
checking permissions 0.000019
Opening tables 0.000079
init 0.000325
System lock 0.000068
optimizing 0.000079
statistics 0.001888
preparing 0.000151
Creating tmp table 0.000128
Sorting result 0.000027
optimizing 0.000034
statistics 0.000064
preparing 0.000047
Creating tmp table 0.000047
Sorting for group 0.000030
optimizing 0.000018
statistics 0.000022
preparing 0.000022
Creating tmp table 0.000043
Sorting for order 0.000023
executing 0.000016
Sending data 4.015596
Creating sort index 0.004766
executing 0.000010
Sending data 0.000159
executing 0.000008
Sending data 0.000025
Creating sort index 0.000349
end 0.000010
query end 0.000024
removing tmp table 0.000013
query end 0.000011
removing tmp table 0.000008
query end 0.000009
removing tmp table 0.000011
query end 0.000009
removing tmp table 0.000007
query end 0.000010
removing tmp table 0.000008
query end 0.000006
closing tables 0.000026
freeing items 0.000039
removing tmp table 0.000009
freeing items 0.000067
logging slow query 0.000047
cleaning up 0.000042
Execution Plan:
+----+--------------+-------------+------------+--------+-----------------------------------------------------+--------------------+---------+--------------------------------------+------+----------+------------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+--------------+-------------+------------+--------+-----------------------------------------------------+--------------------+---------+--------------------------------------+------+----------+------------------------------------+
| 1 | PRIMARY | TIGM_GRP | NULL | index | PRIMARY,FLD_PARENT_GROUP_ID | PRIMARY | 2 | NULL | 87 | 0.33 | Using where |
| 1 | PRIMARY | TIPLD | NULL | range | FLD_PRICE_LEVEL_ID | FLD_PRICE_LEVEL_ID | 3 | NULL | 26 | 10.00 | Using index condition; Using where |
| 1 | PRIMARY | TIPLM | NULL | eq_ref | PRIMARY | PRIMARY | 2 | TIPLD.FLD_PRICE_LEVEL_ID | 1 | 10.00 | Using where |
| 1 | PRIMARY | TIGL | NULL | ref | FLD_GROUP_ID,fld_item_id | FLD_GROUP_ID | 2 | TIGM_GRP.FLD_GROUP_ID | 404 | 100.00 | NULL |
| 1 | PRIMARY | TIM | NULL | eq_ref | PRIMARY,FLD_ITEM_TYPE,FLD_ADDON_ID,FLD_PRODUCT_TYPE | PRIMARY | 4 | TIGL.FLD_ITEM_ID | 1 | 50.00 | Using where |
| 1 | PRIMARY | TSIAM | NULL | eq_ref | PRIMARY | PRIMARY | 4 | TIM.FLD_ADDON_ID | 1 | 10.00 | Using where |
| 1 | PRIMARY | TSIARM | NULL | ref | FLD_ADDON_ID | FLD_ADDON_ID | 5 | TIM.FLD_ADDON_ID | 1 | 10.00 | Using index condition; Using where |
| 1 | PRIMARY | TIPD | NULL | ref | FLD_ITEM_ID,FLD_PRICE_LEVEL_ID | FLD_ITEM_ID | 2 | TIM.FLD_ITEM_ID | 3 | 1.35 | Using index condition; Using where |
| 1 | PRIMARY | TIGM_PARENT | NULL | eq_ref | PRIMARY | PRIMARY | 2 | TIGM_GRP.FLD_PARENT_GROUP_ID | 1 | 100.00 | Using index |
| 2 | UNION | TIGM_GRP | NULL | index | PRIMARY,FLD_PARENT_GROUP_ID | PRIMARY | 2 | NULL | 87 | 0.06 | Using where |
| 2 | UNION | TIGM_PARENT | NULL | eq_ref | PRIMARY | PRIMARY | 2 | TIGM_GRP.FLD_PARENT_GROUP_ID | 1 | 100.00 | Using index |
|NULL| UNION RESULT | <union1,2> | NULL | ALL | NULL | NULL | NULL | NULL | NULL | NULL | Using temporary; Using filesort |
+----+--------------+-------------+------------+--------+-----------------------------------------------------+--------------------+---------+--------------------------------------+------+----------+------------------------------------+
my.cnf:
[mysql]
# CLIENT #######################################################################
port = 3306
socket = /var/lib/mysql/mysql.sock
[mysqld]
# GENERAL ######################################################################
user = mysql
port = 3306
socket = /var/lib/mysql/mysql.sock
server_id = 32108
default_storage_engine = InnoDB
pid_file = /var/run/mysqld/mysqld.pid
optimizer_prune_level = 0
optimizer_search_depth = 0
max_length_for_sort_data = 8388608 #New
net_buffer_length = 1048576 #New
back_log = 80
symbolic-links = 0
log_bin_trust_function_creators = 1
net_read_timeout = 10 #90
net_write_timeout = 10 #120
net_retry_count = 30
thread_stack = 512K #192K
long_query_time = 10
tmpdir = /tmp
# MyISAM #######################################################################
key_buffer_size = 64M
read_buffer_size = 32M
read_rnd_buffer_size = 32M
bulk_insert_buffer_size = 16M
myisam_sort_buffer_size = 128M
myisam_max_sort_file_size = 1G
myisam_repair_threads = 1
memlock
max_allowed_packet = 32M
max_connect_errors = 100
sql_mode = STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
sysdate-is-now = 1
explicit_defaults_for_timestamp = 1
innodb = FORCE
# Password policy disabled as per communication 05-Sep2017
validate_password = OFF
# DATA STORAGE ##################################################################
datadir = /var/lib/mysql/
# BINARY LOGGING ################################################################
log-bin = /var/lib/mysql/mysql-bin
expire-logs-days = 14
sync-binlog = 1
# REPLICATION ###################################################################
skip-slave-start = 1
relay-log = /var/log/mysql-realy-logs/relay-bin
slave-skip-errors = 1062 #,1053,1032,1237,1146
slave-net-timeout = 60
relay_log_purge = 1
# CACHES AND LIMITS #############################################################
tmp-table-size = 256M
max-heap-table-size = 256M
query_cache_min_res_unit = 12288 #8192 #New
query-cache-type = 1
query-cache-size = 32M #102400 #0 #256M
max-connections = 150
thread-cache-size = 10 #-1 #Auto resized
open-files-limit = 65535
table_definition_cache = 2000
table_open_cache = 4096 #3092
table_open_cache_instances = 4
sort_buffer_size = 128M
join_buffer_size = 512M
binlog_cache_size = 16M
query_cache_limit = 4M
# INNODB ########################################################################
innodb_fast_shutdown = 1
innodb_flush_method = O_DIRECT
innodb_log_group_home_dir = /mysql_redo_logs/mysql_redo_logs
innodb_log_files_in_group = 2
innodb_log_file_size = 1G
innodb_flush_log_at_trx_commit = 2
innodb_file_per_table = 1 #ON
innodb_buffer_pool_size = 32G #20G
innodb_buffer_pool_instances = 32
innodb_log_buffer_size = 64M
innodb_adaptive_hash_index = 1 #ON
innodb_thread_concurrency = 300 # "0" is default and means infinite (as and when needed). #250 #32
innodb_thread_sleep_delay = 1
innodb_flush_neighbors = 1
innodb_sync_array_size = 832 # Default is 768
skip-innodb_doublewrite #New
innodb_page_cleaners = 32 # Must be =innodb_buffer_pool_instances
innodb_sort_buffer_size = 512M
innodb_read_io_threads = 64
innodb_write_io_threads = 16
#innodb_concurrency_tickets = 429496729
innodb_max_dirty_pages_pct = 90
innodb_lock_wait_timeout = 10 #80
innodb_compression_level = 0 #New
innodb_lru_scan_depth = 512 #1024 #Default
# LOGGING #######################################################################
log_error = /var/lib/mysql/mysql-error.log
log_queries_not_using_indexes = 1
slow_query_log = 1
log_error_verbosity = 3
[mysqld_safe]
open-files-limit = 4096
malloc-lib = /usr/lib64/libtcmalloc_minimal.so.4
[mysqldump]
quick
max_allowed_packet = 64M
So far I have tried most of the options, currently I started the service in caching mode for quick response.
Can you please help me fix this "sending data" delay issue
Everyone
I think I figured out the issue.
1) 5.6 the optimization engine and plan generated is a result mixture of meta-data and user query (non relational joins). engine trusts user and get best plan.
But 5.7 does not trust soft relations between the tables though they are indexed.
2) 5.7 expects explicit data integrity and do not believe on user.
POC:
I created the same tables involved with hard FK references, wala ! found a miraculous response. the same query which executed without FK took 6.085 Sec (sending data),
and the query with data integrity enforced took 0.05 Sec (with no change in indexing).
So I think 5.7 expects strong data integrity constraints explicitly.
A lot of Optimizer changes happened between 5.1 and 5.7. For one thing, ICP is new; I see it several times in the EXPLAIN. We need to see the query to discuss further. If possible, get the EXPLAIN from 5.1.
Meanwhile, here are some minor comments on my.cnf:
thread_stack = 512K #192K
Generally, it is not useful to increase this setting.
optimizer_prune_level = 0
optimizer_search_depth = 0
What prompted you to set to 0?
innodb = FORCE
deprecated in 5.7.5; suggest removing before it becomes an error.
slave-skip-errors = 1062 #,1053,1032,1237,1146
Sweeping gremlins under the rug -- you will stub your toes on they lump.
innodb_buffer_pool_size = 32G #20G
Unless you have some large apps on the same machine, 44G might be a little better.
innodb_page_cleaners = 32 # Must be =innodb_buffer_pool_instances
re the comment -- Technically, it is auto-limited to pool_instances; see https://dev.mysql.com/doc/refman/8.0/en/innodb-parameters.html#sysvar_innodb_page_cleaners (Thanks, WilsonHauck)
log_queries_not_using_indexes = 1
In my opinion, this clutters the slowlog. The interesting entries are those that exceed long_query_time. Do you have any interesting queries in the slowlog?
I managed (with some help from here) to setup a replication from a MASTER server running mysql 5.6 (centos 6) to a slave running Mariadb 10.1.22 (Centos 7).
My issue now is this, i have another server with the exact mariadb version and specs but its replication is not catching up, instead it is increasing.
When started it was 48000 seconds behind and quickly dropped to 46000 after a few minutes. After that it is steadily increasing. ATM of writing almost back to 48K seconds
Show full processlist; shows the sql thread is spending up to 8 seconds running Update_rows_log_event::ha_update_row(-1) back to back which from all the google search i cannot find what it means.
MariaDB [(none)]> show full processlist;
+-----+------------------+---------------------------------------+--------------+---------+------+------------------------------------------+-----------------------+----------+
| Id | User | Host | db | Command | Time | State | Info | Progress |
+-----+------------------+---------------------------------------+--------------+---------+------+------------------------------------------+-----------------------+----------+
| 3 | system user | | NULL | Connect | 3640 | Queueing master event to the relay log | NULL | 0.000 |
| 2 | system user | | NULL | Connect | 5 | Update_rows_log_event::ha_update_row(-1) | NULL | 0.000 |
Also i caught a simple UPDATE table SET timestamp = NOW() WHERE static_ip = 'a-valid-ip' AND process_id = '13217' taking up to 6 seconds while the table has the static_ip and process_id columns as PK and the command takes 0.078 seconds when executed directly.
Contents of /etc/my.cnf
[mysqld]
max_allowed_packet = 1G
max_connections = 600
thread_cache_size = 16
query_cache_size = 64M
tmp_table_size= 512M
max_heap_table_size= 512M
wait_timeout=60
#Innodb Settings
innodb_file_per_table=1
innodb_buffer_pool_size = 25G
innodb_log_file_size = 2048M
innodb_flush_log_at_trx_commit = 0
innodb_file_format = Barracuda
innodb_flush_neighbors = 0
#Log
log-error =/var/log/error.log
tmpdir = /dev/shm
#Replication SLAVE
server-id=6
slave-skip-errors=1062
my.cnf is same as the server that is running OK except for the slave-id.
Any suggestions/help on what is happening?
Thank you.
From help from the guys at mariadb the ha_update_rows was not relevant and the reason for the slowness was dual disk failure on the machine.
[root#ser3 ~]# dd if=/dev/zero of=/tmp/output conv=fdatasync bs=384k count=1k;
1024+0 records in
1024+0 records out
402653184 bytes (403 MB) copied, 43.1096 s, 9.3 MB/s
This is an SSD.
I have a long running query that I'd like to speed up. The result of the query is a new table.
The tables are all MYISAM and it is running on a large EC2 instance (m2.4xlarge, 64GB RAM).
System usage looks like this:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17438 mysql 20 0 32.7g 7.1g 7420 S 2 10.6 1:35.82 mysqld
Relevant portion of my cnf:
key_buffer = 32768M
max_allowed_packet = 96M
thread_stack = 192K
thread_cache_size = 8
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 8M
table_cache = 512
thread_concurrency = 8
bulk_insert_buffer_size = 2048M
max_write_lock_count = 1
# ~1/4 of memory of machine
max_heap_table_size = 16384M
tmp_table_size = 16384M
# ~1/4 of memory
myisam_sort_buffer_size = 17179869184
When I run this simple query, it takes much longer than I think it should and memory and CPU usage on the machine is low.
The query explain plan looks like this:
mysql> explain SELECT encounter_id
-> FROM encounters e, sampled_patients sp
-> WHERE e.patient_id = sp.patient_id;
+----+-------------+-------+-------+---------------+------------+---------+--------------------+---------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+---------------+------------+---------+--------------------+---------+-------------+
| 1 | SIMPLE | sp | index | patient_id | patient_id | 4 | NULL | 1537954 | Using index |
| 1 | SIMPLE | e | ref | patient_id | patient_id | 5 | noah.sp.patient_id | 1 | Using where |
+----+-------------+-------+-------+---------------+------------+---------+--------------------+---------+-------------+
2 rows in set (0.00 sec)
It has to look through ~1.5M rows, but it's indexed. How can I speed this up?
I was running benchmark test in mysql server using mysqlslap. But sometimes the last thread in the pool is not executing and closing the connection. It keeps in sleep state.
mysql> show full processlist;
+-----+------+-----------------------+-----------+---------+------+-------+-----------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-----+------+-----------------------+-----------+---------+------+-------+-----------------------+
| 2 | root | 122.178.231.xxx:15832 | mysqlslap | Sleep | 740 | | NULL |
| 106 | root | localhost | NULL | Query | 0 | NULL | show full processlist |
+-----+------+-----------------------+-----------+---------+------+-------+-----------------------+
2 rows in set (0.00 sec)
Any idea? It happens often :(
This was the command I executed.
mysqlslap -h IP_address -u root --auto-generate-sql --concurrency=100
-pPASSWORD -vvv --number-of-queries=2500
my.cnf file is as follows,
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
skip-bdb
max_connections = 500
query_cache_type = 1
query_cache_size = 128M
sort_buffer_size = 64M
key_buffer_size = 425M
query_cache_limit = 1024M
thread_cache_size = 4M
innodb_buffer_pool_size = 1360M
innodb_log_file_size = 512M
long_query_time = 5
innodb_thread_concurrency = 8
innodb_lock_wait_timeout = 300
max_allowed_packet = 128M
net_read_timeout = 240
net_write_timeout = 240
table_lock_wait_timeout =240
connect_timeout = 30
log-warnings = 2
log = /var/log/mysqld.log
log-error = /var/log/mysqld.error.log
log-slow-queries = /var/log/mysql_slow_queries.log