I have been running a dashboard for weeks without issue. Suddenly tonight the site keeps crashing due to MYSQL processes overtaking the CPU and RAM. Can someone help me understand why this may have been ok for weeks and now is crashing the server?
Server is pretty basic - 4gb RAM, 2 cores, SSD HDD> I think the key for me is that this has not been an issue for months and now is without having made any new changes.
[root#host ~]# mysqladmin --defaults-extra-file=/usr/local/directadmin/conf/my.cnf proc stat
+------+-------------------+-----------+-------------------+---------+------+-----------------------------+------------------------------------------------------------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+------+-------------------+-----------+-------------------+---------+------+-----------------------------+------------------------------------------------------------------------------------------------------+
| 5 | event_scheduler | localhost | | Daemon | 22 | Waiting for next activation | |
| 184 | food_example-site | localhost | food_example-site | Query | 143 | executing | SELECT * FROM orders_new WHERE DeliveryDate >= (DATE_SUB(CURDATE(),INTERVAL 1 DAY)) |
| 195 | food_example-site | localhost | food_example-site | Sleep | 4044 | | |
| 200 | food_example-site | localhost | food_example-site | Sleep | 4036 | | |
| 575 | food_example-site | localhost | food_example-site | Sleep | 2559 | | |
| 579 | food_example-site | localhost | food_example-site | Sleep | 2548 | | |
| 607 | food_example-site | localhost | food_example-site | Sleep | 2435 | | |
| 622 | food_example-site | localhost | food_example-site | Sleep | 2375 | | |
| 638 | food_example-site | localhost | food_example-site | Sleep | 2318 | | |
| 680 | food_example-site | localhost | food_example-site | Sleep | 2165 | | |
| 686 | food_example-site | localhost | food_example-site | Sleep | 2140 | | |
| 688 | food_example-site | localhost | food_example-site | Sleep | 2134 | | |
| 702 | food_example-site | localhost | food_example-site | Query | 154 | executing | SELECT * FROM orders_new WHERE DeliveryDate >= (DATE_SUB(CURDATE(),INTERVAL 1 DAY)) |
| 800 | food_example-site | localhost | food_example-site | Sleep | 1813 | | |
| 837 | food_example-site | localhost | food_example-site | Sleep | 1654 | | |
| 865 | food_example-site | localhost | food_example-site | Sleep | 1533 | | |
| 867 | food_example-site | localhost | food_example-site | Sleep | 1528 | | |
| 922 | food_example-site | localhost | food_example-site | Sleep | 1288 | | |
| 1189 | food_example-site | localhost | food_example-site | Sleep | 160 | | |
| 1192 | food_example-site | localhost | food_example-site | Sleep | 149 | | |
| 1194 | food_example-site | localhost | food_example-site | Sleep | 147 | | |
| 1196 | da_sso_gZ45Hr1bu | localhost | food_example-site | Connect | 141 | executing | REPLACE INTO orders_new
SELECT
pm.post_id,
pm.post_id AS 'Order_#',
M |
| 1204 | food_example-site | localhost | food_example-site | Sleep | 105 | | |
| 1212 | da_sso_gZ45Hr1bu | localhost | food_example-site | Connect | 82 | executing | REPLACE INTO orders_new
SELECT
pm.post_id,
pm.post_id AS 'Order_#',
M |
| 1216 | food_example-site | localhost | food_example-site | Sleep | 75 | | |
| 1231 | food_example-site | localhost | food_example-site | Sleep | 45 | | |
| 1236 | food_example-site | localhost | food_example-site | Sleep | 27 | | |
| 1238 | da_sso_gZ45Hr1bu | localhost | food_example-site | Connect | 22 | executing | REPLACE INTO orders_new
SELECT
pm.post_id,
pm.post_id AS 'Order_#',
M |
| 1240 | food_example-site | localhost | food_example-site | Sleep | 15 | | |
| 1244 | da_admin | localhost | | Query | 0 | starting | show processlist |
+------+-------------------+-----------+-------------------+---------+------+-----------------------------+------------------------------------------------------------------------------------------------------+
Uptime: 5887 Threads: 30 Questions: 22748 Slow queries: 0 Opens: 859 Flush tables: 3 Open tables: 780 Queries per second avg: 3.864
[root#host ~]#
Have you been INSERTing records into your tables during those weeks where everything worked well? If so, your tables may have become large enough that you need to add indexes for your queries to keep working efficiently. This is a very common consequence of success in new database applications. Look up the " slow query log. " Figure out which queries are hammering your system. Then ask other questions if you need help optimizing them.
Your orders_new table may need an index on its DeliveryDate column.
Try this and see if it helps.
CREATE INDEX orderdate ON orders_new (DeliveryDate);
And, it looks like your REPLACE INTO query DELETEs and then INSERTs every row in your table whenever you run it. That is bound to slow things down. Find another way to do what you need to do there, or at least put a WHERE clause in that query. If you can use UPDATE ... ON DUPLICATE KEY INSERT instead, do that.
Related
'top' gives:
top - 13:07:35 up 827 days, 8:18, 2 users, load average: 5,26, 5,38, 5,45
KiB Mem: 32848512 total, 32265240 used, 583272 free, 261908 buffers
KiB Swap: 1046520 total, 313784 used, 732736 free, 28190744 cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23927 mysql 20 0 917m 243m 8152 S 553,7 0,8 65:09.27 mysqld
mysql> SHOW FULL PROCESSLIST;
+------+--------+-----------------+-------------+---------+------+--------------+--------------------------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+------+--------+-----------------+-------------+---------+------+--------------+--------------------------------------------------------------------+
| 600 | oneill | localhost | NULL | Query | 0 | NULL | SHOW FULL PROCESSLIST |
| 956 | oneill | 127.0.0.1:42219 | FoxPoker | Sleep | 0 | | NULL |
| 1318 | oneill | 127.0.0.1:44207 | FoxPoker | Sleep | 0 | | NULL |
| 1319 | oneill | 127.0.0.1:44210 | FoxPoker | Sleep | 0 | | NULL |
| 1321 | oneill | 127.0.0.1:44215 | FoxPoker | Sleep | 0 | | NULL |
| 1489 | oneill | 127.0.0.1:45218 | FoxPoker | Sleep | 0 | | NULL |
| 1490 | oneill | 127.0.0.1:45217 | FoxPoker | Sleep | 0 | | NULL |
| 1700 | oneill | 127.0.0.1:46246 | FoxPoker | Sleep | 0 | | NULL |
| 1701 | oneill | 127.0.0.1:46248 | FoxPoker | Sleep | 0 | | NULL |
| 2203 | oneill | 127.0.0.1:49218 | FoxPoker | Sleep | 0 | | NULL |
| 3262 | oneill | 127.0.0.1:54855 | FoxPoker | Sleep | 0 | | NULL |
| 3263 | oneill | 127.0.0.1:54858 | FoxPokerDev | Sleep | 0 | | NULL |
| 3462 | oneill | 127.0.0.1:56029 | FoxPoker | Sleep | 0 | | NULL |
| 4269 | oneill | 127.0.0.1:60551 | FoxPoker | Sleep | 0 | | NULL |
| 4945 | oneill | 127.0.0.1:36032 | FoxPoker | Query | 3 | Sending data | SELECT COUNT(deviceId) FROM devices WHERE BINARY deviceId="pMGmpY" |
| 4950 | oneill | 127.0.0.1:36073 | FoxPoker | Query | 1 | Sending data | SELECT COUNT(deviceId) FROM devices WHERE BINARY deviceId="tAf994" |
| 4952 | oneill | 127.0.0.1:36083 | FoxPoker | Query | 1 | Sending data | SELECT COUNT(deviceId) FROM devices WHERE BINARY deviceId="ea6Kv2" |
| 4954 | oneill | 127.0.0.1:36097 | FoxPoker | Query | 1 | Sending data | SELECT COUNT(deviceId) FROM devices WHERE BINARY deviceId="yFYvur" |
| 4955 | oneill | 127.0.0.1:36098 | FoxPoker | Query | 1 | Sending data | SELECT COUNT(deviceId) FROM devices WHERE BINARY deviceId="73qTcv" |
| 4956 | oneill | 127.0.0.1:36099 | FoxPoker | Query | 1 | Sending data | SELECT COUNT(deviceId) FROM devices WHERE BINARY deviceId="q64MfG" |
+------+--------+-----------------+-------------+---------+------+--------------+--------------------------------------------------------------------+
mysql> show status like '%onn%';
+--------------------------+-------+
| Variable_name | Value |
+--------------------------+-------+
| Aborted_connects | 1 |
| Connections | 5528 |
| Max_used_connections | 25 |
| Ssl_client_connects | 0 |
| Ssl_connect_renegotiates | 0 |
| Ssl_finished_connects | 0 |
| Threads_connected | 19 |
+--------------------------+-------+
7 rows in set (0.00 sec)
(Connections keep on growing.)
mysqladmin status -h localhost -u -p
Uptime: 924 Threads: 21 Questions: 92803 Slow queries: 1677 Opens: 1025 Flush tables: 1 Open tables: 265 Queries per second avg: 100.436
I restared mysql, optimized all tables and nothing changed.
Any idea ?
Login to MySQL and check for expensive queries using:
SHOW FULL PROCESSLIST;
Find the offending sql that's causing the 1600 slow queries and performance tune it.
You can also look to enable slow query logging via the below documentation.
https://dev.mysql.com/doc/refman/5.7/en/slow-query-log.html
I have just closed a MySQL shell by clicking x on my command prompt and I am now curious what happens to database connections when you close the window.
I know that the MySQL server runs on port 3306. If I understand correctly, the server waits for connection requests and passes them to a new thread/available thread for managing the connection.
How does the server know when the connection has been terminated, so that the thread can be released or put back into a pool of available threads?
Scenarios such as:
Blue Screen during a connection with a MySQL server on a remote connection.
Program terminated before it could call a disconnect request.
From the mysql client:
show processlist;
+-----+------+-----------------+---------------+---------+-------+-------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-----+------+-----------------+---------------+---------+-------+-------+------------------+
| 2 | root | localhost:6509 | so_gibberish2 | Sleep | 105 | | NULL |
| 3 | root | localhost:6510 | so_gibberish2 | Sleep | 105 | | NULL |
| 820 | root | localhost:40286 | stackoverflow | Sleep | 13457 | | NULL |
| 821 | root | localhost:40287 | stackoverflow | Sleep | 966 | | NULL |
| 827 | root | localhost:42254 | so_gibberish2 | Query | 0 | init | show processlist |
| 831 | root | localhost:44036 | stackoverflow | Sleep | 230 | | NULL |
+-----+------+-----------------+---------------+---------+-------+-------+------------------+
select connection_id();
+-----------------+
| connection_id() |
+-----------------+
| 827 |
+-----------------+
Now hit [x] as you say to client the mysql client that ran the above.
Below is from Mysql Workbench:
show processlist;
+-----+------+-----------------+---------------+---------+-------+-------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-----+------+-----------------+---------------+---------+-------+-------+------------------+
| 2 | root | localhost:6509 | so_gibberish2 | Sleep | 105 | | NULL |
| 3 | root | localhost:6510 | so_gibberish2 | Sleep | 105 | | NULL |
| 820 | root | localhost:40286 | stackoverflow | Sleep | 13457 | | NULL |
| 821 | root | localhost:40287 | stackoverflow | Sleep | 966 | | NULL |
+-----+------+-----------------+---------------+---------+-------+-------+------------------+
select connection_id();
+-----------------+
| connection_id() |
+-----------------+
| 3 |
+-----------------+
So the closing of a program naturally closes any open sockets it has.
Honestly, the latter output looks more like:
But I always use the mysql client for output formatting for tables which would throw off the results. So there you have it as an image.
I have an issue with a rails 4 application that leaves idle mysql connections. While we look at the code to determine the cause, I at least want to be able to automatically eliminate the offending processes. The mysql documentation says setting wait_timeout and interactive_timeout will close inactive connections so I've set them both to 300. However, as can be seen below, there are various processes that exceed the 300 limit. I've confirmed that the settings are indeed 300. Why isn't mysql eliminating these processes?
mysql> show processlist;
+-----+--------------+-----------+--------------+---------+------+-------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-----+--------------+-----------+--------------+---------+------+-------+------------------+
| 143 | application | localhost | application | Sleep | 1500 | | NULL |
| 144 | application | localhost | application | Sleep | 1499 | | NULL |
| 145 | application | localhost | application | Sleep | 0 | | NULL |
| 146 | application | localhost | application | Sleep | 780 | | NULL |
| 147 | application | localhost | application | Sleep | 783 | | NULL |
| 148 | application | localhost | application | Sleep | 0 | | NULL |
| 149 | application | localhost | application | Sleep | 7 | | NULL |
| 150 | application | localhost | application | Sleep | 3 | | NULL |
| 151 | application | localhost | application | Sleep | 8 | | NULL |
| 152 | application | localhost | application | Sleep | 3 | | NULL |
| 155 | root | localhost | NULL | Query | 0 | NULL | show processlist |
+-----+--------------+-----------+--------------+---------+------+-------+------------------+
11 rows in set (0.00 sec)
try run mysql query kill 143;
this works for me
I've started to see very high cpu usage on mysql on my CentOS server.
It used to run at between 20%-30% but now will spike into the 400% at times.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9169 mysql 20 0 2232m 66m 6816 S 140.6 0.9 17:37.03 mysqld
when i do a show processlist i dont see much running
mysql> show processlist;
+-------+------+-----------+----------+---------+------+-------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-------+------+-----------+----------+---------+------+-------+------------------+
| 53 | sqluser | localhost | mydns | Sleep | 0 | | NULL |
| 449 | sqluser | localhost | DB1 | Sleep | 0 | | NULL |
| 16151 | sqluser | localhost | DB2 | Sleep | 105 | | NULL |
| 16473 | sqluser | localhost | DB2 | Sleep | 61 | | NULL |
| 16641 | sqluser | localhost | DB2 | Sleep | 39 | | NULL |
| 16729 | sqluser | localhost | DB2 | Sleep | 29 | | NULL |
| 16733 | sqluser | localhost | DB2 | Sleep | 29 | | NULL |
| 16815 | sqluser | localhost | DB2 | Sleep | 15 | | NULL |
| 16867 | sqluser | localhost | DB2 | Sleep | 9 | | NULL |
| 16886 | sqluser | localhost | NULL | Query | 0 | NULL | show processlist |
| 16910 | sqluser | localhost | DB2 | Sleep | 5 | | NULL |
| 16919 | sqluser | localhost | DB1 | Sleep | 4 | | NULL |
| 16935 | sqluser | localhost | DB1 | Sleep | 2 | | NULL |
| 16937 | sqluser | localhost | DB1 | Sleep | 2 | | NULL |
| 16939 | sqluser | localhost | DB1 | Sleep | 1 | | NULL |
| 16941 | sqluser | localhost | DB1 | Sleep | 1 | | NULL |
| 16951 | sqluser | localhost | DB1 | Sleep | 1 | | NULL |
| 16952 | sqluser | localhost | DB2 | Sleep | 1 | | NULL |
| 16955 | sqluser | localhost | DB1 | Sleep | 1 | | NULL |
| 16960 | sqluser | localhost | DB1 | Sleep | 0 | | NULL |
| 16965 | sqluser | localhost | DB1 | Sleep | 0 | | NULL |
+-------+------+-----------+----------+---------+------+-------+------------------+
21 rows in set (0.00 sec)
Is there anything else i can do to see what is causing such high usage?
First of all, you need to get some more data, in order to do that I usually use some tools like:
Mytop
Innotop
Mytop has some great ways of viewing active connections and queries.
Innotop has great ways of seeing what is using your table spaces, buffer pools, etc.
You don't have to use these tools to gather the data. You could use the Percona Toolkit tools, they have some great ones for really drilling down into the knitty gritty of MySQL usage.
Don't forget that there are logs to check. Specifically you should check on the slow log. To find if and where that is being stored, you can run following:
First make sure it's turned on:
MySQL> SHOW VARIABLES LIKE 'slow_query_log';
Then check where it's located on your file system:
MySQL> SHOW VARIABLES LIKE 'slow_query_log_file';
Using the Percona tool pt-query-digest on the slow log has proved to be helpful for me.
It gives a cohesive list of what queries are using the most time (which usually means resource intensive queries). You can view the usage of the tool here: PT-Query-Digest
You may find some really slow queries, if so I would evaluate how they are being used and see if adding an index, or optimizing them is needed.
Once you gather these data pieces you should have a good picture of what's happening to your MySQL instance. Of course there's many more tools and techniques, but I find these to be sufficient to form a good baseline of performance.
For some reason,the connections to my local database are put to sleep, even though the database is not actually doing anything:
mysql> SHOW FULL PROCESSLIST;
+-----+------+-----------+-----+---------+------+-------+-----------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-----+------+-----------+-----+---------+------+-------+-----------------------+
| 85 | root | localhost | *** | Sleep | 810 | | NULL |
| 88 | root | localhost | *** | Sleep | 662 | | NULL |
| 89 | root | localhost | *** | Sleep | 586 | | NULL |
| 93 | root | localhost | *** | Sleep | 692 | | NULL |
| 98 | root | localhost | *** | Sleep | 719 | | NULL |
| 102 | root | localhost | *** | Sleep | 545 | | NULL |
| 182 | root | localhost | *** | Query | 0 | init | SHOW FULL PROCESSLIST |
| 263 | root | localhost | *** | Sleep | 809 | | NULL |
| 275 | root | localhost | *** | Sleep | 660 | | NULL |
| 279 | root | localhost | *** | Sleep | 584 | | NULL |
| 282 | root | localhost | *** | Sleep | 1 | | NULL |
| 283 | root | localhost | *** | Sleep | 544 | | NULL |
+-----+------+-----------+-----+---------+------+-------+-----------------------+
12 rows in set (0.00 sec)
No new connections are being made and there's no lock, but somehow the requests are not being processed. Any idea what could be causing this?
In this context Sleep simply means that the thread is idle. Here's the definition from the manual:
Sleep
The thread is waiting for the client to send a new statement to it.
By default PHP is configured to allow persistent MySQL connections. That's why you're seeing all these sleeping connections, just waiting to be re-used.
Whether you want to use persistent links or not is a different question, but if you want to disable it then you need to make a change to your PHP.ini
mysql.allow_persistent = Off