I have an issue with a rails 4 application that leaves idle mysql connections. While we look at the code to determine the cause, I at least want to be able to automatically eliminate the offending processes. The mysql documentation says setting wait_timeout and interactive_timeout will close inactive connections so I've set them both to 300. However, as can be seen below, there are various processes that exceed the 300 limit. I've confirmed that the settings are indeed 300. Why isn't mysql eliminating these processes?
mysql> show processlist;
+-----+--------------+-----------+--------------+---------+------+-------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-----+--------------+-----------+--------------+---------+------+-------+------------------+
| 143 | application | localhost | application | Sleep | 1500 | | NULL |
| 144 | application | localhost | application | Sleep | 1499 | | NULL |
| 145 | application | localhost | application | Sleep | 0 | | NULL |
| 146 | application | localhost | application | Sleep | 780 | | NULL |
| 147 | application | localhost | application | Sleep | 783 | | NULL |
| 148 | application | localhost | application | Sleep | 0 | | NULL |
| 149 | application | localhost | application | Sleep | 7 | | NULL |
| 150 | application | localhost | application | Sleep | 3 | | NULL |
| 151 | application | localhost | application | Sleep | 8 | | NULL |
| 152 | application | localhost | application | Sleep | 3 | | NULL |
| 155 | root | localhost | NULL | Query | 0 | NULL | show processlist |
+-----+--------------+-----------+--------------+---------+------+-------+------------------+
11 rows in set (0.00 sec)
try run mysql query kill 143;
this works for me
Related
I have just closed a MySQL shell by clicking x on my command prompt and I am now curious what happens to database connections when you close the window.
I know that the MySQL server runs on port 3306. If I understand correctly, the server waits for connection requests and passes them to a new thread/available thread for managing the connection.
How does the server know when the connection has been terminated, so that the thread can be released or put back into a pool of available threads?
Scenarios such as:
Blue Screen during a connection with a MySQL server on a remote connection.
Program terminated before it could call a disconnect request.
From the mysql client:
show processlist;
+-----+------+-----------------+---------------+---------+-------+-------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-----+------+-----------------+---------------+---------+-------+-------+------------------+
| 2 | root | localhost:6509 | so_gibberish2 | Sleep | 105 | | NULL |
| 3 | root | localhost:6510 | so_gibberish2 | Sleep | 105 | | NULL |
| 820 | root | localhost:40286 | stackoverflow | Sleep | 13457 | | NULL |
| 821 | root | localhost:40287 | stackoverflow | Sleep | 966 | | NULL |
| 827 | root | localhost:42254 | so_gibberish2 | Query | 0 | init | show processlist |
| 831 | root | localhost:44036 | stackoverflow | Sleep | 230 | | NULL |
+-----+------+-----------------+---------------+---------+-------+-------+------------------+
select connection_id();
+-----------------+
| connection_id() |
+-----------------+
| 827 |
+-----------------+
Now hit [x] as you say to client the mysql client that ran the above.
Below is from Mysql Workbench:
show processlist;
+-----+------+-----------------+---------------+---------+-------+-------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-----+------+-----------------+---------------+---------+-------+-------+------------------+
| 2 | root | localhost:6509 | so_gibberish2 | Sleep | 105 | | NULL |
| 3 | root | localhost:6510 | so_gibberish2 | Sleep | 105 | | NULL |
| 820 | root | localhost:40286 | stackoverflow | Sleep | 13457 | | NULL |
| 821 | root | localhost:40287 | stackoverflow | Sleep | 966 | | NULL |
+-----+------+-----------------+---------------+---------+-------+-------+------------------+
select connection_id();
+-----------------+
| connection_id() |
+-----------------+
| 3 |
+-----------------+
So the closing of a program naturally closes any open sockets it has.
Honestly, the latter output looks more like:
But I always use the mysql client for output formatting for tables which would throw off the results. So there you have it as an image.
I'm trying to setup mysql mts replication on 5.7.
I'm replicating one database using LOGICAL_CLOCK:
mysql> show variables like 'slave_parallel%';
+------------------------+---------------+
| Variable_name | Value |
+------------------------+---------------+
| slave_parallel_type | LOGICAL_CLOCK |
| slave_parallel_workers | 4 |
+------------------------+---------------+
2 rows in set (0.00 sec)
But only one thread is active, others just wait:
mysql> show processlist;
+--------+-------------+-----------------------+--------------------+---------+--------+---------------------------------------------+----------------------------------------------------------------------------+-----------+---------------+
| Id | User | Host | db | Command | Time | State | Info | Rows_sent | Rows_examined |
+--------+-------------+-----------------------+--------------------+---------+--------+---------------------------------------------+----------------------------------------------------------------------------+-----------+---------------+
| 8103 | system user | | NULL | Connect | 72189 | Waiting for master to send event | NULL | 0 | 0 |
| 8104 | system user | | NULL | Connect | 0 | Waiting for dependent transaction to commit | NULL | 0 | 0 |
| 8105 | system user | | db_repl | Connect | 344764 | System lock | DELETE FROM docs WHERE id = 21099810 | 0 | 0 |
| 8106 | system user | | NULL | Connect | 344765 | Waiting for an event from Coordinator | NULL | 0 | 0 |
| 8107 | system user | | NULL | Connect | 344771 | Waiting for an event from Coordinator | NULL | 0 | 0 |
| 8108 | system user | | NULL | Connect | 344813 | Waiting for an event from Coordinator | NULL | 0 | 0 |
| 600637 | sphinx | localhost | mysql | Query | 0 | starting | show processlist | 0 | 0 |
+--------+-------------+-----------------------+--------------------+---------+--------+---------------------------------------------+----------------------------------------------------------------------------+-----------+---------------+
7 rows in set (0.00 sec)
How I can debug my configuration to see why threads are not active?
P.S. I have high loaded master and have big slave delay because slave has only one thread for replication, so I need to increase slave threads to decrease "Seconds_behind_master".
I've started to see very high cpu usage on mysql on my CentOS server.
It used to run at between 20%-30% but now will spike into the 400% at times.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9169 mysql 20 0 2232m 66m 6816 S 140.6 0.9 17:37.03 mysqld
when i do a show processlist i dont see much running
mysql> show processlist;
+-------+------+-----------+----------+---------+------+-------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-------+------+-----------+----------+---------+------+-------+------------------+
| 53 | sqluser | localhost | mydns | Sleep | 0 | | NULL |
| 449 | sqluser | localhost | DB1 | Sleep | 0 | | NULL |
| 16151 | sqluser | localhost | DB2 | Sleep | 105 | | NULL |
| 16473 | sqluser | localhost | DB2 | Sleep | 61 | | NULL |
| 16641 | sqluser | localhost | DB2 | Sleep | 39 | | NULL |
| 16729 | sqluser | localhost | DB2 | Sleep | 29 | | NULL |
| 16733 | sqluser | localhost | DB2 | Sleep | 29 | | NULL |
| 16815 | sqluser | localhost | DB2 | Sleep | 15 | | NULL |
| 16867 | sqluser | localhost | DB2 | Sleep | 9 | | NULL |
| 16886 | sqluser | localhost | NULL | Query | 0 | NULL | show processlist |
| 16910 | sqluser | localhost | DB2 | Sleep | 5 | | NULL |
| 16919 | sqluser | localhost | DB1 | Sleep | 4 | | NULL |
| 16935 | sqluser | localhost | DB1 | Sleep | 2 | | NULL |
| 16937 | sqluser | localhost | DB1 | Sleep | 2 | | NULL |
| 16939 | sqluser | localhost | DB1 | Sleep | 1 | | NULL |
| 16941 | sqluser | localhost | DB1 | Sleep | 1 | | NULL |
| 16951 | sqluser | localhost | DB1 | Sleep | 1 | | NULL |
| 16952 | sqluser | localhost | DB2 | Sleep | 1 | | NULL |
| 16955 | sqluser | localhost | DB1 | Sleep | 1 | | NULL |
| 16960 | sqluser | localhost | DB1 | Sleep | 0 | | NULL |
| 16965 | sqluser | localhost | DB1 | Sleep | 0 | | NULL |
+-------+------+-----------+----------+---------+------+-------+------------------+
21 rows in set (0.00 sec)
Is there anything else i can do to see what is causing such high usage?
First of all, you need to get some more data, in order to do that I usually use some tools like:
Mytop
Innotop
Mytop has some great ways of viewing active connections and queries.
Innotop has great ways of seeing what is using your table spaces, buffer pools, etc.
You don't have to use these tools to gather the data. You could use the Percona Toolkit tools, they have some great ones for really drilling down into the knitty gritty of MySQL usage.
Don't forget that there are logs to check. Specifically you should check on the slow log. To find if and where that is being stored, you can run following:
First make sure it's turned on:
MySQL> SHOW VARIABLES LIKE 'slow_query_log';
Then check where it's located on your file system:
MySQL> SHOW VARIABLES LIKE 'slow_query_log_file';
Using the Percona tool pt-query-digest on the slow log has proved to be helpful for me.
It gives a cohesive list of what queries are using the most time (which usually means resource intensive queries). You can view the usage of the tool here: PT-Query-Digest
You may find some really slow queries, if so I would evaluate how they are being used and see if adding an index, or optimizing them is needed.
Once you gather these data pieces you should have a good picture of what's happening to your MySQL instance. Of course there's many more tools and techniques, but I find these to be sufficient to form a good baseline of performance.
For some reason,the connections to my local database are put to sleep, even though the database is not actually doing anything:
mysql> SHOW FULL PROCESSLIST;
+-----+------+-----------+-----+---------+------+-------+-----------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-----+------+-----------+-----+---------+------+-------+-----------------------+
| 85 | root | localhost | *** | Sleep | 810 | | NULL |
| 88 | root | localhost | *** | Sleep | 662 | | NULL |
| 89 | root | localhost | *** | Sleep | 586 | | NULL |
| 93 | root | localhost | *** | Sleep | 692 | | NULL |
| 98 | root | localhost | *** | Sleep | 719 | | NULL |
| 102 | root | localhost | *** | Sleep | 545 | | NULL |
| 182 | root | localhost | *** | Query | 0 | init | SHOW FULL PROCESSLIST |
| 263 | root | localhost | *** | Sleep | 809 | | NULL |
| 275 | root | localhost | *** | Sleep | 660 | | NULL |
| 279 | root | localhost | *** | Sleep | 584 | | NULL |
| 282 | root | localhost | *** | Sleep | 1 | | NULL |
| 283 | root | localhost | *** | Sleep | 544 | | NULL |
+-----+------+-----------+-----+---------+------+-------+-----------------------+
12 rows in set (0.00 sec)
No new connections are being made and there's no lock, but somehow the requests are not being processed. Any idea what could be causing this?
In this context Sleep simply means that the thread is idle. Here's the definition from the manual:
Sleep
The thread is waiting for the client to send a new statement to it.
By default PHP is configured to allow persistent MySQL connections. That's why you're seeing all these sleeping connections, just waiting to be re-used.
Whether you want to use persistent links or not is a different question, but if you want to disable it then you need to make a change to your PHP.ini
mysql.allow_persistent = Off
My database seems to have many open connections despite having no users on httpd.
mysql> SHOW PROCESSLIST;
+-------+----------+--------------------+---------------------+---------+------+-------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-------+----------+--------------------+---------------------+---------+------+-------+------------------+
| 22865 | rdsadmin | localhost:51066 | mysql | Sleep | 8 | | NULL |
| 25098 | ubuntu | 123.45.67.89:48060 | database_name_here! | Sleep | 350 | | NULL |
| 25099 | ubuntu | 123.45.67.89:48069 | database_name_here! | Sleep | 3731 | | NULL |
| 25126 | ubuntu | 123.45.67.89:49558 | database_name_here! | Sleep | 349 | | NULL |
| 25161 | ubuntu | 123.45.67.89:50770 | database_name_here! | Sleep | 289 | | NULL |
| 26583 | ubuntu | 123.45.67.89:37593 | database_name_here! | Sleep | 4550 | | NULL |
| 29729 | ubuntu | 123.45.67.89:49280 | database_name_here! | Sleep | 953 | | NULL |
| 29737 | ubuntu | 123.45.67.89:49489 | database_name_here! | Sleep | 953 | | NULL |
| 29738 | ubuntu | 123.45.67.89:49490 | database_name_here! | Sleep | 349 | | NULL |
| 29755 | ubuntu | 123.45.67.89:50674 | database_name_here! | Sleep | 2751 | | NULL |
| 29759 | ubuntu | 123.45.67.89:50892 | database_name_here! | Sleep | 600 | | NULL |
| 35600 | ubuntu | 123.45.67.89:49677 | NULL | Query | 0 | NULL | SHOW PROCESSLIST |
+-------+----------+--------------------+---------------------+---------+------+-------+------------------+
I don't understand how there are so many connections. I'm using amazon EC2 with RDS, apache/mysql/PHP with PDO. I have persistent connections "on", but I thought this simply cached the connection. It seems somehow 10 connections remain open constantly even though there are 1-2 users on my site at any given time.
What is the cause of this, and how can I fix it? Is this normal behavior, are there a minimum amount of connections that MySQL keeps open to handle different users? I cannot seem to find any information on how to analyze what is going on.
Usually you'll have a connection pool from your application server (Apache httpd, nginx, etc.) to your database. These stay open all the time so there's no reconnection required when a request comes in. Remember, connecting to the database can take several seconds, but using an active connection takes zero time.
Why would you want to "fix" this? It's a feature of a modern application stack. If they're not running queries, there's no issue.