I have a very basic question regarding MySQL Workbench Client Connections window.In that window a Command column and a Time column is shown.If the Command column value is Sleep and the Time column value is very large(say 1500), does that mean that the client connection object has not been used for quite sometime? Also what are the meanings of "Threads Connected", "Threads Running","Total Connections" etc?
A sample MySQL Workbench screenshot of Client Connections real time is shown below:
It basically utilizes output of SHOW PROCESSLIST command.
Command Column: It basically implies the type of action happening in a particular connected thread. In the example screenshot: Sleep means that a thread is connected, but not firing any query as of now. Query means that a query is being executed. That is why we have more Threads Connected, but less number of Threads Running (Query command being run). Some threads are in the process of Connecting. Check more details here.
Time Column: The time in seconds that the thread has been in its current state.
Threads Connected: Number of MySQL client connections open to the server at the moment. So, for example, in our application code, when we do a mysqli_connect, it opens a connection to the Server. In this particular case, it also basically implies that 15 client sessions (most of them originating from application code) are executing simultaneously right now.
Threads Running: Out of these 15 connections, 4 are actually in the process of executing a query.
Total Connection: Total connections made to the server till date (since last server restart I believe).
Connection Limit: Maximum number of connections that can be made simultaneously. Default value of this is 151. In our case, we have increased it to 512, due to server capacity available.
Related
I have a Django project hosted on AWS Lambda function.
This microservice uses pymysql to connect to AWS Aurora RDS database.
This service executes one and only one query over and over again.
1 in 200 times the query fails with EOF packet error.
In order to investigate this issue i have implemented a "repeater" which would repeat the same query if one fails (maximum 2 repeats with 0.25 seconds delay).
Once again, in a rare ocasion, a query has failed and I expected to see a successful query after the first reattempt. However it failed in all 3 consecutive calls with all DIFFERENT error messages.
Error messages (in order):
AssertionError: Protocol error, expecting EOF
django.db.utils.InternalError: Packet sequence number wrong - got 24 expected 1
django.db.utils.InterfaceError: (0, '')
These are errors from 3 separate queries executed against MySql Aurora RDS database. (I just wanted to ephesize that indeed it is not a stack trace but rather different query errors).
More useful info:
The microservice uses Django ORM to create queries.
The database is in Master-Slave configuration and those queries go against a Slave database.
The parameters observed in Master and Slave databases (such as CPU usage, free RAM space, various latencies, various throughputs, etc.) are completely normal and do not indicate any potential errors.
It is not a multithreaded environment.
Error stack traces:
Complete stack for *EOF error*:
https://pastebin.com/BracLTZX
Complete stack for *Packet sequence error*:
https://pastebin.com/fYmRGh69
Complete stack for *Interface error*:
https://pastebin.com/bstG1r2q
I am running two EC2 instances on AWS to serve my application one for each. Each application can open up to 100 connections to MySql for default. For database I use RDS and t2.medium instance which can handle 312 connections at a time.
In general, my connection size does not get larger than 20. When I start sending notifications to the users to come to the application, the connection size increases a lot (which is expected.). In some cases, MySql connections increases unexpectedly and my application starts to throw PoolExhaustedException:
PoolExhaustedException: [pool-7-thread-92] Timeout: Pool empty. Unable to fetch a connection in 30 seconds, none available[size:100; busy:100; idle:0; lastwait:30000].
When I check the database connections from Navicat, I see that there are about 200 connections and all of them are sleeping. I do not understand why the open connections are not used. I use standard Spring Data Jpa to save and read my entities which means I do not open or close the connections manually.
Unless I shut down one of the instance and the mysql connections are released, both instances do not response at all.
You can see the mysql connection size change graphic here, and a piece of log here.
I've a MySQL 5.1.41 Server installed on a Ubuntu machine. I get connected to it through Workbench from my Windows machine over TCP/IP. I run a bigger query, after 900 seconds I got the below message, (there is no wait_timeout defined in the server's configuration file my.cnf)
Error Code: 2013. Lost connection to MySQL server during query
But when I look into the process list by using show processlist; command, I can still see my query running.
I got this link http://dev.mysql.com/doc/refman/5.0/en/gone-away.html where I found the below lines,
The problem on Windows is that in some cases MySQL does not get an
error from the OS when writing to the TCP/IP connection to the server,
but instead gets the error when trying to read the answer from the
connection.
I'm not sure whether this is the reason for my observation.
Please clarify me on this.
Thanks in advance!!
Closing connection is not a reason to stop a query. A query might be update, or kind of transaction, or select with output to remote (server) file.
Closed connection is just is just means, that you will not receive any data from DBMS after executing query (data, timings - nothing).
The reason of closing connection could be different, as SO-User posted. Try increasing
on server side:
wait_timeout
max_allowed_packet
on client side:
any kinds of timeout you find in your client (i.e. that SO-User suggests)
Do not forget to reload DBMS config and restart client (for sure)
In MySQL WorkBench we have an option to change timeout.
Find it under
Edit → Preferences → SQL Editor → DBMS connection read time out (in seconds): 600
Changed the value to 6000 or something higher.
Update
Lost connection to MySQL server
There are three likely causes for this error message.
Usually it indicates network connectivity trouble and you should check
the condition of your network if this error occurs frequently. If the
error message includes “during query,” this is probably the case you
are experiencing.
Sometimes the “during query” form happens when millions of rows are
being sent as part of one or more queries. If you know that this is
happening, you should try increasing net_read_timeout from its default
of 30 seconds to 60 seconds or longer, sufficient for the data
transfer to complete.
More rarely, it can happen when the client is attempting the initial
connection to the server. In this case, if your connect_timeout value
is set to only a few seconds, you may be able to resolve the problem
by increasing it to ten seconds, perhaps more if you have a very long
distance or slow connection. You can determine whether you are
experiencing this more uncommon cause by using SHOW GLOBAL STATUS LIKE
'Aborted_connects'. It will increase by one for each initial
connection attempt that the server aborts. You may see “reading
authorization packet” as part of the error message; if so, that also
suggests that this is the solution that you need.
If the cause is none of those just described, you may be experiencing
a problem with BLOB values that are larger than max_allowed_packet,
which can cause this error with some clients. Sometime you may see an
ER_NET_PACKET_TOO_LARGE error, and that confirms that you need to
increase max_allowed_packet.
Doc link: Error lost connection
and also check here
for some reason when I open a connection the the Percona MySQL database on my HostGator website, after fetching the query, it will disconnect/ close the connection about 10 seconds later.
I typically wouldn't care, but HeidiSQL freezes up, preventing exporting or sorting the returned rows with it's UI unless I connect again.
Any thoughts on making the connection last longer? is it something I can do myself, or will it require a dedicated server or some upgrade? (I'm currently on a shared one). Thanks!
Sounds like it may be the "wait" timeout on the MySQL connection.
SHOW VARIABLES LIKE 'wait_timeout'
That's the amount of time (in seconds) that MySQL will leave the session (the database connection) open while it's idle, waiting for another statement to be issued. After this amount of time expires, MySQL can close the connection.
You should be able to change this for a session, to change the timeout to 5 minutes
SET wait_timeout = 300
Verify the setting with the SHOW VARIABLES statement again.
NOTE: This is per connection. It only affects the current session. Every new connection will inherit their own wait_timeout value from the global setting.
(This is only a guess. There's insufficient information in the question to make a precise diagnosis. It could be something other than MySQL server that's closing the database connection, e.g. it could be your connection pool settings (if you are using a connection pool).
We have 8 phusion passengers with 20 connections each. should be 160 max connection. Today our Mysql connection crossed 300 and our server stopped responding.
What would happen if the thread dies unnaturally ? how do db connections associated with it get cleaned-up?
How to debug this type of scenario ?
What would happen if the thread dies unnaturally ?
If you are using your statements under transaction then all statements under un-successful transactions will be rolled back but if you are executing your statements individually then how many have been executed will be saved in db but other un-successful will not and data can be in-consistant.
how do db connections associated with it get cleaned-up?
As per my assumption your application is opening connections more than required and also not closing properly. In this case if you check connections in mysql admininstrator then you will get so many connections in sleep mode. So you can kill all sleep connections to clean them.
How to debug this type of scenario ?
Step1: Enable general logs on your server:
Step2: execute below command in any gui tool like sqlyog etc.
show full processlist;
Step3: Take first sleep process from above command:
Step3: find/grep above process id (suppose it is 235798) in your general log. You can use below command.
$ cat /var/log/mysqldquery.log | grep 235798
Note: there can be different file name and path for your general log file.
Above command will show you few lines, check if you are closing connection at the end, should show "quite" statement at the end of line. In this way you need to check few processes those are in sleep mode and you can jugde which type statements/from which module opening extra connectiions and not closing and accordingly you can take action.