mysql thread not executing. Still in sleep state - mysql

I was running benchmark test in mysql server using mysqlslap. But sometimes the last thread in the pool is not executing and closing the connection. It keeps in sleep state.
mysql> show full processlist;
+-----+------+-----------------------+-----------+---------+------+-------+-----------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-----+------+-----------------------+-----------+---------+------+-------+-----------------------+
| 2 | root | 122.178.231.xxx:15832 | mysqlslap | Sleep | 740 | | NULL |
| 106 | root | localhost | NULL | Query | 0 | NULL | show full processlist |
+-----+------+-----------------------+-----------+---------+------+-------+-----------------------+
2 rows in set (0.00 sec)
Any idea? It happens often :(
This was the command I executed.
mysqlslap -h IP_address -u root --auto-generate-sql --concurrency=100
-pPASSWORD -vvv --number-of-queries=2500
my.cnf file is as follows,
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
skip-bdb
max_connections = 500
query_cache_type = 1
query_cache_size = 128M
sort_buffer_size = 64M
key_buffer_size = 425M
query_cache_limit = 1024M
thread_cache_size = 4M
innodb_buffer_pool_size = 1360M
innodb_log_file_size = 512M
long_query_time = 5
innodb_thread_concurrency = 8
innodb_lock_wait_timeout = 300
max_allowed_packet = 128M
net_read_timeout = 240
net_write_timeout = 240
table_lock_wait_timeout =240
connect_timeout = 30
log-warnings = 2
log = /var/log/mysqld.log
log-error = /var/log/mysqld.error.log
log-slow-queries = /var/log/mysql_slow_queries.log

Related

MySQL Master-Master replication issue

I have MySQL 5.7 installed on two servers and I configured both servers' mysqld.cnf file as shown below
Server1 configuration.
[mysqld]
bind-address = 192.168.0.1
server_id = 1
log_bin = /var/log/mysql/mysql-bin.log
log_bin_index = /var/log/mysql/mysql-bin.log.index
relay_log = /var/log/mysql/mysql-relay-bin
relay_log_index = /var/log/mysql/mysql-relay-bin.index
expire_logs_days = 10
max_binlog_size = 100M
log_slave_updates = 1
auto-increment-increment = 2
auto-increment-offset = 1
Server2 configuration.
[mysqld]
bind-address = 192.168.0.2
server_id = 2
log_bin = /var/log/mysql/mysql-bin.log
log_bin_index = /var/log/mysql/mysql-bin.log.index
relay_log = /var/log/mysql/mysql-relay-bin
relay_log_index = /var/log/mysql/mysql-relay-bin.index
expire_logs_days = 10
max_binlog_size = 100M
log_slave_updates = 1
auto-increment-increment = 2
auto-increment-offset = 2
Then when I get the out put from Server1 with the below:
mysql> SHOW MASTER STATUS;
+------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql-bin.000006 | 459 | | | |
+------------------+----------+--------------+------------------+-------------------+
Then I go to run and get the output as per below at Server2:
mysql> STOP SLAVE;
mysql> CHANGE MASTER TO master_host='192.168.0.1', master_port=3306,
master_user='replicauser',
master_password='somestrongpassword', master_log_file='mysql-bin.000006',
master_log_pos=459;
mysql> START SLAVE;
mysql> SHOW MASTER STATUS;
+------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql-bin.000007 | 154 | | | |
+------------------+----------+--------------+------------------+-------------------+
So what is the missing to complete my solution ?
Edit:
The missing part was just restart for both servers

Mysqld consumes 232% CPU

My mysqld process consumes 232% CPU and and there 14000+ connections
(I'm a little new to this thing but following Stack Overflow for assistance).
top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3112 mysql 20 0 7061444 1.397g 15848 S 232.6 8.9 1138:06 mysqld
System:
Ubuntu 18.04,
16GB RAM,
8 Core CPU,
120GB Disk
and MySQL version 5.7.25
mysql> show status like 'Conn%';
+-----------------------------------+-------+
| Variable_name | Value |
+-----------------------------------+-------+
| Connection_errors_accept | 0 |
| Connection_errors_internal | 0 |
| Connection_errors_max_connections | 0 |
| Connection_errors_peer_address | 0 |
| Connection_errors_select | 0 |
| Connection_errors_tcpwrap | 0 |
| Connections | 14007 |
+-----------------------------------+-------+
7 rows in set (0.01 sec)
And show variables like "%timeout%"
mysql> show variables like "%timeout%";
+-----------------------------+----------+
| Variable_name | Value |
+-----------------------------+----------+
| connect_timeout | 10 |
| delayed_insert_timeout | 300 |
| have_statement_timeout | YES |
| innodb_flush_log_at_timeout | 1 |
| innodb_lock_wait_timeout | 50 |
| innodb_rollback_on_timeout | OFF |
| interactive_timeout | 28800 |
| lock_wait_timeout | 31536000 |
| net_read_timeout | 30 |
| net_write_timeout | 60 |
| rpl_stop_slave_timeout | 31536000 |
| slave_net_timeout | 60 |
| wait_timeout | 28800 |
+-----------------------------+----------+
13 rows in set (0.01 sec)
And mysqld.cnf settings
[mysqld]
# Skip reverse DNS lookup of clients
skip-name-resolve
default-storage-engine=InnoDB
max_allowed_packet=500M
max_connections = 256
interactive_timeout=7200
wait_timeout=7200
innodb_file_per_table=1
innodb_buffer_pool_size = 8G
innodb_buffer_pool_instances = 4
innodb_log_file_size = 1G
innodb_flush_log_at_trx_commit = 1
innodb_flush_method = O_DIRECT
innodb_open_files=5000
innodb_io_capacity=2000
innodb_io_capacity_max=4000
innodb_old_blocks_time=2000
open_files_limit=50000
query_cache_type = 1
query_cache_min_res_unit = 1M
query_cache_limit = 1M
query_cache_size = 50M
tmp_table_size= 256M
max_heap_table_size= 256M
#key_buffer_size = 128M
thread_stack = 128K
thread_cache_size = 32
slow-query-log = 1
slow-query-log-file = /var/lib/mysql/mysql-slow.log
long_query_time = 1
Note: Corrected above mysqld.cnf values to match with below reports attached
Additional Info:
htop:- https://pastebin.com/43f4b3fK
top:- https://pastebin.com/rTh1XvUt
GLOBAL VARIABLES: https://pastebin.com/K2fgKwEv (Complete)
INNODB STATUS:- https://pastebin.com/nGrZjHAg
Mysqltuner:- https://pastebin.com/ZNYieJj8
[SHOW FULL PROCESSLIST], [ulimit -a], [iostat -xm], [lscpu] :- https://pastebin.com/mrnyQrXf
Server freezes when multiple db transaction is being carried out. Is there a lock like thing or any configuration flaws?
(Background: This is a WordPress blog and nobody else is accessing it right now. I somehow imported a 115K posts from an old blog but struck here with this CPU ghost)
Rate Per Second = RPS - Suggestions to consider for your my.cnf [mysqld] section,
innodb_lru_scan_depth=100 # from 1024 to reduce 90% of cpu cycles used for function every SECOND
innodb_io_capacity=3500 # from 2000 to enable higher IOPS on your SSD devices
innodb_flushing_avg_loops=5 # from 30 to reduce innodb_buffer_pool_pages_dirty overhead - count was 3183 in SGStatus
read_buffer_size=256K # from 128K to reduce handler_read_next RPS of 277,134
read_rnd_buffer_size=192K # from 256K to reduce handler_read_rnd_next RPS of 778
There are many more opportunities to improve performance through Global Variables.
Disclaimer: I am the author of web site mentioned in my profile, Network profile that includes contact information.
A likely cause of high CPU and poor performance is the poor schema for wp_postmeta. I discuss the remedy here.
Meanwhile, "you can't tune your way out of a performance problem". I did glance at the settings -- all are "reasonable".

About the user filtering of mysql percona audit plugin

I have a mysql server. I use Percona and I use the audit log plugin. I have no problem with this.
But there are too many active users in the sub-structure using mysql. I just want to keep the audit logs of real users. I want to use the include and exclude properties for this.
When I apply the following percona article, it fails because it cannot find the mysql variable. Does not add new variable.
Any advice on this?
https://www.percona.com/doc/percona-server/LATEST/management/audit_log_plugin.html#filtering-by-user
I have added the following line to my.cnf file, but again nothing has changed.
audit_log_include_accounts = 'test # localhost'
[root#mysql-server ~]# cat /etc/redhat-release
CentOS Linux release 7.5.1804 (Core)
[root#mysql-server ~]# yum list installed |grep "percona"
Percona-Server-client-57.x86_64 5.7.22-22.1.el7 #percona
Percona-Server-server-57.x86_64 5.7.11-4.1.el7 #percona
Percona-Server-shared-57.x86_64 5.7.22-22.1.el7 #percona
Percona-Server-shared-compat-57.x86_64 5.7.22-22.1.el7 #percona
percona-toolkit.x86_64 3.0.10-1.el7 #percona
[root#mysql-server ~]# mysql -V
mysql Ver 14.14 Distrib 5.7.22-22, for Linux (x86_64) using 6.2
mysql> SHOW VARIABLES LIKE 'audit%';
+---------------------------+--------------------------+
| Variable_name | Value |
+---------------------------+--------------------------+
| audit_log_buffer_size | 1048576 |
| audit_log_file | /var/log/mysql/audit.log |
| audit_log_flush | OFF |
| audit_log_format | JSON |
| audit_log_handler | FILE |
| audit_log_policy | ALL |
| audit_log_rotate_on_size | 10485760 |
| audit_log_rotations | 10 |
| audit_log_strategy | ASYNCHRONOUS |
| audit_log_syslog_facility | LOG_USER |
| audit_log_syslog_ident | percona-audit |
| audit_log_syslog_priority | LOG_INFO |
+---------------------------+--------------------------+
12 rows in set (0.01 sec)
mysql> SET GLOBAL audit_log_include_accounts = 'test#localhost';
ERROR 1193 (HY000): Unknown system variable 'audit_log_include_accounts'
mysql>
mysql> SET GLOBAL audit_log_include_accounts = NULL;
ERROR 1193 (HY000): Unknown system variable 'audit_log_include_accounts'
[root#mysql-server ~]# cat /etc/my.cnf
[client]
port = 3306
socket = /var/lib/mysql/mysql.sock
[isamchk]
key_buffer_size = 16M
[mysqld]
audit_log_file = /var/log/mysql/audit.log
audit_log_format = JSON
audit_log_policy = ALL
audit_log_rotate_on_size = 10M
audit_log_rotations = 10
basedir = /usr
bind-address = 0.0.0.0
binlog-format = ROW
datadir = /var/lib/mysql
default-storage-engine = innodb
expire_logs_days = 7
explicit_defaults_for_timestamp = 1
innodb-buffer-pool-size = 1G
innodb-file-per-table = 1
innodb-flush-log-at-trx-commit = 2
innodb-flush-method = O_DIRECT
innodb-log-file-size = 512M
innodb-log-files-in-group = 2
key_buffer_size = 16M
log-error = /var/log/mysql/mysqld.log
log-queries-not-using-indexes = 0
max-connect-errors = 1000000
max-connections = 1000
max-heap-table-size = 512M
max_allowed_packet = 16M
max_binlog_size = 100M
open-files-limit = 65535
pid-file = /var/run/mysqld/mysqld.pid
port = 3306
query-cache-type = 1
query_cache_limit = 12M
query_cache_size = 16M
server_id = 1
skip-external-locking
skip-name-resolve
slave-skip-errors = 1062,1053
slow-query-log = 1
slow-query-log-file = /var/log/mysql/mysql-slow.log
socket = /var/lib/mysql/mysql.sock
ssl = false
ssl-ca = /etc/mysql/cacert.pem
ssl-cert = /etc/mysql/server-cert.pem
ssl-key = /etc/mysql/server-key.pem
sync-binlog = 0
sysdate-is-now = 1
table-definition-cache = 4096
table-open-cache = 4096
thread_cache_size = 8
thread_stack = 256K
tmp_table_size = 16M
tmpdir = /tmp
user = mysql
[mysqld-5.0]
myisam-recover = BACKUP
[mysqld-5.1]
myisam-recover = BACKUP
[mysqld-5.5]
myisam-recover = BACKUP
[mysqld-5.6]
myisam-recover-options = BACKUP
[mysqld-5.7]
myisam-recover-options = BACKUP
[mysqld_safe]
log-error = /var/log/mysql/mysqld_safe.log
nice = 0
socket = /var/lib/mysql/mysql.sock
[mysqldump]
max_allowed_packet = 16M
quick
quote-names
!includedir /etc/my.cnf.d
I found the cause of the problem. The version needs to be upgraded. When I upgraded the version, the variables came automatically and I can make changes.
However, the supported version is 5.7 in the documentation. But not exactly.
my current version: '5.7.11-4.1.el7'
The version I upgraded to: '5.7.24-27.1.el7'

Will tarantool helps in OpenVZ?

I have a small OpenVZ container.
2 cores and 4096MB RAM.
I have mysql database (overal size is 80MB InnoDb)
When i do 100 queries like INSERT ON DUPLICATE KEY UPDATE, few of them executes longer than 1 seconds, always
mysql> SHOW PROFILE;
+----------------------+----------+
| Status | Duration |
+----------------------+----------+
| checking permissions | 0.000040 |
| creating table | 0.000056 |
| After create | 0.011363 |
| query end | 1.231525 |
| freeing items | 0.000089 |
| logging slow query | 0.000019 |
| cleaning up | 0.000005 |
+----------------------+----------+
7 rows in set (0.00 sec)
When i turned off binary logging, it helped. So the problem maybe in hard drive. And occurs when binary log is writing during the query executing.
Will it help, if i replace mysql with the tarantool? As i know tarantool also writes binlogs.
I have percona mysql
mysql> select version();
+-----------------+
| version() |
+-----------------+
| 5.6.25-73.1-log |
+-----------------+
1 row in set (0.01 sec)
Here is the my.cnf
[client]
default-character-set = utf8mb4
[mysql]
# CLIENT #
port = 3306
socket = /var/lib/mysql/mysql.sock
default-character-set = utf8mb4
[mysqld]
character-set-client-handshake = FALSE
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci
# GENERAL #
user = mysql
default-storage-engine = InnoDB
socket = /var/lib/mysql/mysql.sock
pid-file = /var/lib/mysql/mysql.pid
# MyISAM #
key-buffer-size = 32M
myisam-recover = FORCE,BACKUP
# SAFETY #
max-allowed-packet = 16M
max-connect-errors = 1000000
# DATA STORAGE #
datadir = /var/lib/mysql/
# BINARY LOGGING #
#log-bin = /var/lib/mysql/mysql-bin
#expire-logs-days = 14
#sync-binlog = 1
# CACHES AND LIMITS #
tmp-table-size = 128M
max-heap-table-size = 128M
query-cache-type = 1
query-cache-size = 32M
max-connections = 1000
thread-cache-size = 128
open-files-limit = 65535
table-definition-cache = 1024
table-open-cache = 2048
# INNODB #
innodb-flush-method = O_DIRECT
innodb-log-files-in-group = 2
innodb-log-file-size = 256M
innodb_log_buffer_size = 32M
innodb-flush-log-at-trx-commit = 0
innodb-file-per-table = 1
innodb-buffer-pool-size = 1G
innodb_buffer_pool_instances = 1
# LOGGING #
log-error = /var/lib/mysql/mysql-error1.log
log-queries-not-using-indexes = 1
slow-query-log = 1
slow-query-log-file = /var/lib/mysql/mysql-slow.log
long_query_time = 1
log-queries-not-using-indexes = 1
#PERCONA
log_slow_verbosity = microtime,query_plan,innodb
Yes Tarantool write 'binlogs' (WAL - write-ahead logging).
Definitely tarantool are more faster than mysql.
But probably mysql need more tuning. Sorry I'm not know much about mysql tuning.
Tarantool should help you out here.
It delivers sub 0.001s latencies even with transaction log turned on. Here is the piece of code that helps you testing Tarantool under heavy read/write workloads: https://gist.github.com/danikin/a5ddc6fe0cedc6257853.

Magento - Too many connections

I am getting Too many connections error from Magento.
I have increased the max_connection to 1000 but I am still getting the error.
I contacted hosting provider and they asked me to use command show processlist; and review my coding.
When I ran the command, I only saw few active connections (about 4 to 5 connection). Therefore, I have no clue how to fix the problem.
I have increased the max_connection to 1500 and I am getting create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug error now.
Could anyone can help me with this situation please?
I am grateful for your help and time.
This is my my.cnf
key_buffer = 384M
max_allowed_packet = 1M
table_cache = 1024
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 8M
myisam_sort_buffer_size = 64M
thread_cache_size = 16
query_cache_type = 1
query_cache_size = 48M
log_slow_queries=/var/log/mysqld.slowquery.log
max_connections=1000
wait_timeout=120
tmp_table_size = 64M
max_heap_table_size = 64M
innodb_buffer_pool_size = 2048M
innodb_additional_mem_pool_size = 20M
open_files_limit=34484
#
And this is show proccesslist
+-------+-----------+-----------+--------------------+----------------+-------+--------------------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-------+-----------+-----------+--------------------+----------------+-------+--------------------+------------------+
| 4729 | root | localhost | abc_def| Sleep | 13093 | | NULL |
| 16282 | eximstats | localhost | eximstats | Sleep | 84 | | NULL |
| 16283 | DELAYED | localhost | eximstats | Delayed insert | 84 | Waiting for INSERT | |
| 16343 | root | localhost | NULL | Query | 0 | NULL | show processlist |
+-------+-----------+-----------+--------------------+----------------+-------+--------------------+------------------+
4 rows in set (0.00 sec)
You can increase max connections, but just so much
In essence there are too many connections being started, or not closed
So you can
increase max connections to allow more connections being started
Reduce wait_timeout for the connections that have gone away to free up again
Investigate where all these connections request are coming from? (can often be bots, or may bots at once, some index update or other cronjob etc)
thanks, hope it helps