I'm trying to setup mysql mts replication on 5.7.
I'm replicating one database using LOGICAL_CLOCK:
mysql> show variables like 'slave_parallel%';
+------------------------+---------------+
| Variable_name | Value |
+------------------------+---------------+
| slave_parallel_type | LOGICAL_CLOCK |
| slave_parallel_workers | 4 |
+------------------------+---------------+
2 rows in set (0.00 sec)
But only one thread is active, others just wait:
mysql> show processlist;
+--------+-------------+-----------------------+--------------------+---------+--------+---------------------------------------------+----------------------------------------------------------------------------+-----------+---------------+
| Id | User | Host | db | Command | Time | State | Info | Rows_sent | Rows_examined |
+--------+-------------+-----------------------+--------------------+---------+--------+---------------------------------------------+----------------------------------------------------------------------------+-----------+---------------+
| 8103 | system user | | NULL | Connect | 72189 | Waiting for master to send event | NULL | 0 | 0 |
| 8104 | system user | | NULL | Connect | 0 | Waiting for dependent transaction to commit | NULL | 0 | 0 |
| 8105 | system user | | db_repl | Connect | 344764 | System lock | DELETE FROM docs WHERE id = 21099810 | 0 | 0 |
| 8106 | system user | | NULL | Connect | 344765 | Waiting for an event from Coordinator | NULL | 0 | 0 |
| 8107 | system user | | NULL | Connect | 344771 | Waiting for an event from Coordinator | NULL | 0 | 0 |
| 8108 | system user | | NULL | Connect | 344813 | Waiting for an event from Coordinator | NULL | 0 | 0 |
| 600637 | sphinx | localhost | mysql | Query | 0 | starting | show processlist | 0 | 0 |
+--------+-------------+-----------------------+--------------------+---------+--------+---------------------------------------------+----------------------------------------------------------------------------+-----------+---------------+
7 rows in set (0.00 sec)
How I can debug my configuration to see why threads are not active?
P.S. I have high loaded master and have big slave delay because slave has only one thread for replication, so I need to increase slave threads to decrease "Seconds_behind_master".
Related
MySQL 5.7.17 MGR deploy in single-primary mode, 3 node all one one machine, the same configuration.
And then we test insert on the primary node and observe that the slave nodes have high delay with the write node, even primary node finish the insert test, slave node data is incresing!
Why MySQL Group Replication slave nodes have high delay with write node?
here is the my.cnf:
[mysqld]
datadir=/dba/mysql/data/s1
basedir=/dba/mysql/mysql-5.7/
port=24801
socket=/dba/mysql/data/s1/s1.sock
server_id=1
gtid_mode=ON
enforce_gtid_consistency=ON
master_info_repository=TABLE
relay_log_info_repository=TABLE
binlog_checksum=NONE
log_slave_updates=ON
log_bin=binlog
binlog_format=ROW
transaction_write_set_extraction=XXHASH64
loose-group_replication_group_name="aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
loose-group_replication_start_on_boot=off
loose-group_replication_local_address= "127.0.0.1:24901"
loose-group_replication_group_seeds= "127.0.0.1:24901,127.0.0.1:24902,127.0.0.1:24903"
loose-group_replication_bootstrap_group= off
loose-group_replication_single_primary_mode=true
loose-group_replication_enforce_update_everywhere_checks=false
slave_parallel_type=LOGICAL_CLOCK
slave_preserve_commit_order=1
slave_parallel_workers=4
and the MGR config :
mysql> show variables like '%group_replication%';
+----------------------------------------------------+-------------------------------------------------+
| Variable_name | Value |
+----------------------------------------------------+-------------------------------------------------+
| group_replication_allow_local_disjoint_gtids_join | OFF |
| group_replication_allow_local_lower_version_join | OFF |
| group_replication_auto_increment_increment | 7 |
| group_replication_bootstrap_group | OFF |
| group_replication_components_stop_timeout | 31536000 |
| group_replication_compression_threshold | 1000000 |
| group_replication_enforce_update_everywhere_checks | OFF |
| group_replication_flow_control_applier_threshold | 25000 |
| group_replication_flow_control_certifier_threshold | 25000 |
| group_replication_flow_control_mode | QUOTA |
| group_replication_force_members | |
| group_replication_group_name | aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa |
| group_replication_group_seeds | 127.0.0.1:24901,127.0.0.1:24902,127.0.0.1:24903 |
| group_replication_gtid_assignment_block_size | 1000000 |
| group_replication_ip_whitelist | AUTOMATIC |
| group_replication_local_address | 127.0.0.1:24901 |
| group_replication_poll_spin_loops | 0 |
| group_replication_recovery_complete_at | TRANSACTIONS_APPLIED |
| group_replication_recovery_reconnect_interval | 60 |
| group_replication_recovery_retry_count | 10 |
| group_replication_recovery_ssl_ca | |
| group_replication_recovery_ssl_capath | |
| group_replication_recovery_ssl_cert | |
| group_replication_recovery_ssl_cipher | |
| group_replication_recovery_ssl_crl | |
| group_replication_recovery_ssl_crlpath | |
| group_replication_recovery_ssl_key | |
| group_replication_recovery_ssl_verify_server_cert | OFF |
| group_replication_recovery_use_ssl | OFF |
| group_replication_single_primary_mode | ON |
| group_replication_ssl_mode | DISABLED |
| group_replication_start_on_boot | OFF |
+----------------------------------------------------+-------------------------------------------------+
32 rows in set (0.01 sec)
I have just closed a MySQL shell by clicking x on my command prompt and I am now curious what happens to database connections when you close the window.
I know that the MySQL server runs on port 3306. If I understand correctly, the server waits for connection requests and passes them to a new thread/available thread for managing the connection.
How does the server know when the connection has been terminated, so that the thread can be released or put back into a pool of available threads?
Scenarios such as:
Blue Screen during a connection with a MySQL server on a remote connection.
Program terminated before it could call a disconnect request.
From the mysql client:
show processlist;
+-----+------+-----------------+---------------+---------+-------+-------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-----+------+-----------------+---------------+---------+-------+-------+------------------+
| 2 | root | localhost:6509 | so_gibberish2 | Sleep | 105 | | NULL |
| 3 | root | localhost:6510 | so_gibberish2 | Sleep | 105 | | NULL |
| 820 | root | localhost:40286 | stackoverflow | Sleep | 13457 | | NULL |
| 821 | root | localhost:40287 | stackoverflow | Sleep | 966 | | NULL |
| 827 | root | localhost:42254 | so_gibberish2 | Query | 0 | init | show processlist |
| 831 | root | localhost:44036 | stackoverflow | Sleep | 230 | | NULL |
+-----+------+-----------------+---------------+---------+-------+-------+------------------+
select connection_id();
+-----------------+
| connection_id() |
+-----------------+
| 827 |
+-----------------+
Now hit [x] as you say to client the mysql client that ran the above.
Below is from Mysql Workbench:
show processlist;
+-----+------+-----------------+---------------+---------+-------+-------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-----+------+-----------------+---------------+---------+-------+-------+------------------+
| 2 | root | localhost:6509 | so_gibberish2 | Sleep | 105 | | NULL |
| 3 | root | localhost:6510 | so_gibberish2 | Sleep | 105 | | NULL |
| 820 | root | localhost:40286 | stackoverflow | Sleep | 13457 | | NULL |
| 821 | root | localhost:40287 | stackoverflow | Sleep | 966 | | NULL |
+-----+------+-----------------+---------------+---------+-------+-------+------------------+
select connection_id();
+-----------------+
| connection_id() |
+-----------------+
| 3 |
+-----------------+
So the closing of a program naturally closes any open sockets it has.
Honestly, the latter output looks more like:
But I always use the mysql client for output formatting for tables which would throw off the results. So there you have it as an image.
I have an issue with a rails 4 application that leaves idle mysql connections. While we look at the code to determine the cause, I at least want to be able to automatically eliminate the offending processes. The mysql documentation says setting wait_timeout and interactive_timeout will close inactive connections so I've set them both to 300. However, as can be seen below, there are various processes that exceed the 300 limit. I've confirmed that the settings are indeed 300. Why isn't mysql eliminating these processes?
mysql> show processlist;
+-----+--------------+-----------+--------------+---------+------+-------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-----+--------------+-----------+--------------+---------+------+-------+------------------+
| 143 | application | localhost | application | Sleep | 1500 | | NULL |
| 144 | application | localhost | application | Sleep | 1499 | | NULL |
| 145 | application | localhost | application | Sleep | 0 | | NULL |
| 146 | application | localhost | application | Sleep | 780 | | NULL |
| 147 | application | localhost | application | Sleep | 783 | | NULL |
| 148 | application | localhost | application | Sleep | 0 | | NULL |
| 149 | application | localhost | application | Sleep | 7 | | NULL |
| 150 | application | localhost | application | Sleep | 3 | | NULL |
| 151 | application | localhost | application | Sleep | 8 | | NULL |
| 152 | application | localhost | application | Sleep | 3 | | NULL |
| 155 | root | localhost | NULL | Query | 0 | NULL | show processlist |
+-----+--------------+-----------+--------------+---------+------+-------+------------------+
11 rows in set (0.00 sec)
try run mysql query kill 143;
this works for me
I've started to see very high cpu usage on mysql on my CentOS server.
It used to run at between 20%-30% but now will spike into the 400% at times.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9169 mysql 20 0 2232m 66m 6816 S 140.6 0.9 17:37.03 mysqld
when i do a show processlist i dont see much running
mysql> show processlist;
+-------+------+-----------+----------+---------+------+-------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-------+------+-----------+----------+---------+------+-------+------------------+
| 53 | sqluser | localhost | mydns | Sleep | 0 | | NULL |
| 449 | sqluser | localhost | DB1 | Sleep | 0 | | NULL |
| 16151 | sqluser | localhost | DB2 | Sleep | 105 | | NULL |
| 16473 | sqluser | localhost | DB2 | Sleep | 61 | | NULL |
| 16641 | sqluser | localhost | DB2 | Sleep | 39 | | NULL |
| 16729 | sqluser | localhost | DB2 | Sleep | 29 | | NULL |
| 16733 | sqluser | localhost | DB2 | Sleep | 29 | | NULL |
| 16815 | sqluser | localhost | DB2 | Sleep | 15 | | NULL |
| 16867 | sqluser | localhost | DB2 | Sleep | 9 | | NULL |
| 16886 | sqluser | localhost | NULL | Query | 0 | NULL | show processlist |
| 16910 | sqluser | localhost | DB2 | Sleep | 5 | | NULL |
| 16919 | sqluser | localhost | DB1 | Sleep | 4 | | NULL |
| 16935 | sqluser | localhost | DB1 | Sleep | 2 | | NULL |
| 16937 | sqluser | localhost | DB1 | Sleep | 2 | | NULL |
| 16939 | sqluser | localhost | DB1 | Sleep | 1 | | NULL |
| 16941 | sqluser | localhost | DB1 | Sleep | 1 | | NULL |
| 16951 | sqluser | localhost | DB1 | Sleep | 1 | | NULL |
| 16952 | sqluser | localhost | DB2 | Sleep | 1 | | NULL |
| 16955 | sqluser | localhost | DB1 | Sleep | 1 | | NULL |
| 16960 | sqluser | localhost | DB1 | Sleep | 0 | | NULL |
| 16965 | sqluser | localhost | DB1 | Sleep | 0 | | NULL |
+-------+------+-----------+----------+---------+------+-------+------------------+
21 rows in set (0.00 sec)
Is there anything else i can do to see what is causing such high usage?
First of all, you need to get some more data, in order to do that I usually use some tools like:
Mytop
Innotop
Mytop has some great ways of viewing active connections and queries.
Innotop has great ways of seeing what is using your table spaces, buffer pools, etc.
You don't have to use these tools to gather the data. You could use the Percona Toolkit tools, they have some great ones for really drilling down into the knitty gritty of MySQL usage.
Don't forget that there are logs to check. Specifically you should check on the slow log. To find if and where that is being stored, you can run following:
First make sure it's turned on:
MySQL> SHOW VARIABLES LIKE 'slow_query_log';
Then check where it's located on your file system:
MySQL> SHOW VARIABLES LIKE 'slow_query_log_file';
Using the Percona tool pt-query-digest on the slow log has proved to be helpful for me.
It gives a cohesive list of what queries are using the most time (which usually means resource intensive queries). You can view the usage of the tool here: PT-Query-Digest
You may find some really slow queries, if so I would evaluate how they are being used and see if adding an index, or optimizing them is needed.
Once you gather these data pieces you should have a good picture of what's happening to your MySQL instance. Of course there's many more tools and techniques, but I find these to be sufficient to form a good baseline of performance.
My database seems to have many open connections despite having no users on httpd.
mysql> SHOW PROCESSLIST;
+-------+----------+--------------------+---------------------+---------+------+-------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-------+----------+--------------------+---------------------+---------+------+-------+------------------+
| 22865 | rdsadmin | localhost:51066 | mysql | Sleep | 8 | | NULL |
| 25098 | ubuntu | 123.45.67.89:48060 | database_name_here! | Sleep | 350 | | NULL |
| 25099 | ubuntu | 123.45.67.89:48069 | database_name_here! | Sleep | 3731 | | NULL |
| 25126 | ubuntu | 123.45.67.89:49558 | database_name_here! | Sleep | 349 | | NULL |
| 25161 | ubuntu | 123.45.67.89:50770 | database_name_here! | Sleep | 289 | | NULL |
| 26583 | ubuntu | 123.45.67.89:37593 | database_name_here! | Sleep | 4550 | | NULL |
| 29729 | ubuntu | 123.45.67.89:49280 | database_name_here! | Sleep | 953 | | NULL |
| 29737 | ubuntu | 123.45.67.89:49489 | database_name_here! | Sleep | 953 | | NULL |
| 29738 | ubuntu | 123.45.67.89:49490 | database_name_here! | Sleep | 349 | | NULL |
| 29755 | ubuntu | 123.45.67.89:50674 | database_name_here! | Sleep | 2751 | | NULL |
| 29759 | ubuntu | 123.45.67.89:50892 | database_name_here! | Sleep | 600 | | NULL |
| 35600 | ubuntu | 123.45.67.89:49677 | NULL | Query | 0 | NULL | SHOW PROCESSLIST |
+-------+----------+--------------------+---------------------+---------+------+-------+------------------+
I don't understand how there are so many connections. I'm using amazon EC2 with RDS, apache/mysql/PHP with PDO. I have persistent connections "on", but I thought this simply cached the connection. It seems somehow 10 connections remain open constantly even though there are 1-2 users on my site at any given time.
What is the cause of this, and how can I fix it? Is this normal behavior, are there a minimum amount of connections that MySQL keeps open to handle different users? I cannot seem to find any information on how to analyze what is going on.
Usually you'll have a connection pool from your application server (Apache httpd, nginx, etc.) to your database. These stay open all the time so there's no reconnection required when a request comes in. Remember, connecting to the database can take several seconds, but using an active connection takes zero time.
Why would you want to "fix" this? It's a feature of a modern application stack. If they're not running queries, there's no issue.