We have 8 phusion passengers with 20 connections each. should be 160 max connection. Today our Mysql connection crossed 300 and our server stopped responding.
What would happen if the thread dies unnaturally ? how do db connections associated with it get cleaned-up?
How to debug this type of scenario ?
What would happen if the thread dies unnaturally ?
If you are using your statements under transaction then all statements under un-successful transactions will be rolled back but if you are executing your statements individually then how many have been executed will be saved in db but other un-successful will not and data can be in-consistant.
how do db connections associated with it get cleaned-up?
As per my assumption your application is opening connections more than required and also not closing properly. In this case if you check connections in mysql admininstrator then you will get so many connections in sleep mode. So you can kill all sleep connections to clean them.
How to debug this type of scenario ?
Step1: Enable general logs on your server:
Step2: execute below command in any gui tool like sqlyog etc.
show full processlist;
Step3: Take first sleep process from above command:
Step3: find/grep above process id (suppose it is 235798) in your general log. You can use below command.
$ cat /var/log/mysqldquery.log | grep 235798
Note: there can be different file name and path for your general log file.
Above command will show you few lines, check if you are closing connection at the end, should show "quite" statement at the end of line. In this way you need to check few processes those are in sleep mode and you can jugde which type statements/from which module opening extra connectiions and not closing and accordingly you can take action.
Related
I have small web/mail server with apache/mariadb. Last week we changed some of the WWW code and to make it work I changed in php.ini line :
max_input_vars to 5000 (now 4000, it was 1000 at the start)
And it seems changed something because our mariadb 10.3.28 starts making problems.
It just stops reciving any information.
Restart of mysql (and httpd) helps for 24h now ...
Log:
2022-10-05 14:28:58 2796199 [Warning] Aborted connection 2796199 to db: 'ACTIVEDB' user: 'USER' host: 'localhost' (Got an error reading communication packets)
This kind of warnings shows up sometimes but now we got dozens every hour.
In PHP i decrased max_input_vars, in my.cnf I addedd
max_allowed_packet = 124M
max_connections = 400
log_warnings = 3
Everything was at default values before.
Log level was for some time at level 4 but it stared to get too big without any time give to "crush".
Disk is nvme 500GB, Intel and shows no problems.
I would like to hear :
how to check/connect mariadb when it looks inactive
what and how (step by step) to check
Thanks all
This is not an answer, but too long for a comment.
The error "Aborted connection ... (Got an error reading communication packets)" occurs if a client disconnected without sending a COM_CLOSE notification to the server before.
This behavior is easily reproducible, e.g. by starting the command line client and killing the command line client from another session. Depending on the log_warning level, the server will write a log entry and increase the server status variable aborted_clients (or aborted_connects if this happens during connection handshake).
Here are only a few possible reasons:
Before 10.2.3 the default log_warning level was 1 (no logging of aborted_connections), since 10.2.4 default value is 2 (log aborted connections) - if the server was recently upgraded from 10.2.3 or lower the problem may have already existed in previous installation, but was not written into log file.
The PHP script(s) doesn't close the connection: As soon the script is ready with its work, make sure that all transactions were committed, memory (result sets) were freed, and the connection was closed properly.
A timeout occurred, e.g. wait_timeout was set too low and exceeded or PHP's max_execution_time exceeded (and script was killed) or net_read/write_timeouts occurred.
DNS problems: In this case enable skip-name-resolve, use IP's and verify against IP's.
Network or firewall problems
I would like to thank for sugestions and your time.
Looks like there was a problem with one table, I don't know how/why but after some time it was set up as READ-ONLY. There was no information in log about it up to level 3 (level 4 generated too much information for me :( )
The DB works some time fine (expect for one table) and in time it looks like it just hangs whole DB.
The case is still "under investigation".
About "damaged" table :
listed and read-only related thinks works fine
insert/change hangs the whole DB for 2-3 minutes and then gets back to work
after few "hangs" the DB just freezes
there was nothing strange in logs level 3
copy table to new and then change names to switch tables works (I hope so)
If anyone have any idea how to check table would be great (standard operatinos like check, analyse did nothing).
Thanks again.
I have a very basic question regarding MySQL Workbench Client Connections window.In that window a Command column and a Time column is shown.If the Command column value is Sleep and the Time column value is very large(say 1500), does that mean that the client connection object has not been used for quite sometime? Also what are the meanings of "Threads Connected", "Threads Running","Total Connections" etc?
A sample MySQL Workbench screenshot of Client Connections real time is shown below:
It basically utilizes output of SHOW PROCESSLIST command.
Command Column: It basically implies the type of action happening in a particular connected thread. In the example screenshot: Sleep means that a thread is connected, but not firing any query as of now. Query means that a query is being executed. That is why we have more Threads Connected, but less number of Threads Running (Query command being run). Some threads are in the process of Connecting. Check more details here.
Time Column: The time in seconds that the thread has been in its current state.
Threads Connected: Number of MySQL client connections open to the server at the moment. So, for example, in our application code, when we do a mysqli_connect, it opens a connection to the Server. In this particular case, it also basically implies that 15 client sessions (most of them originating from application code) are executing simultaneously right now.
Threads Running: Out of these 15 connections, 4 are actually in the process of executing a query.
Total Connection: Total connections made to the server till date (since last server restart I believe).
Connection Limit: Maximum number of connections that can be made simultaneously. Default value of this is 151. In our case, we have increased it to 512, due to server capacity available.
I've a MySQL 5.1.41 Server installed on a Ubuntu machine. I get connected to it through Workbench from my Windows machine over TCP/IP. I run a bigger query, after 900 seconds I got the below message, (there is no wait_timeout defined in the server's configuration file my.cnf)
Error Code: 2013. Lost connection to MySQL server during query
But when I look into the process list by using show processlist; command, I can still see my query running.
I got this link http://dev.mysql.com/doc/refman/5.0/en/gone-away.html where I found the below lines,
The problem on Windows is that in some cases MySQL does not get an
error from the OS when writing to the TCP/IP connection to the server,
but instead gets the error when trying to read the answer from the
connection.
I'm not sure whether this is the reason for my observation.
Please clarify me on this.
Thanks in advance!!
Closing connection is not a reason to stop a query. A query might be update, or kind of transaction, or select with output to remote (server) file.
Closed connection is just is just means, that you will not receive any data from DBMS after executing query (data, timings - nothing).
The reason of closing connection could be different, as SO-User posted. Try increasing
on server side:
wait_timeout
max_allowed_packet
on client side:
any kinds of timeout you find in your client (i.e. that SO-User suggests)
Do not forget to reload DBMS config and restart client (for sure)
In MySQL WorkBench we have an option to change timeout.
Find it under
Edit → Preferences → SQL Editor → DBMS connection read time out (in seconds): 600
Changed the value to 6000 or something higher.
Update
Lost connection to MySQL server
There are three likely causes for this error message.
Usually it indicates network connectivity trouble and you should check
the condition of your network if this error occurs frequently. If the
error message includes “during query,” this is probably the case you
are experiencing.
Sometimes the “during query” form happens when millions of rows are
being sent as part of one or more queries. If you know that this is
happening, you should try increasing net_read_timeout from its default
of 30 seconds to 60 seconds or longer, sufficient for the data
transfer to complete.
More rarely, it can happen when the client is attempting the initial
connection to the server. In this case, if your connect_timeout value
is set to only a few seconds, you may be able to resolve the problem
by increasing it to ten seconds, perhaps more if you have a very long
distance or slow connection. You can determine whether you are
experiencing this more uncommon cause by using SHOW GLOBAL STATUS LIKE
'Aborted_connects'. It will increase by one for each initial
connection attempt that the server aborts. You may see “reading
authorization packet” as part of the error message; if so, that also
suggests that this is the solution that you need.
If the cause is none of those just described, you may be experiencing
a problem with BLOB values that are larger than max_allowed_packet,
which can cause this error with some clients. Sometime you may see an
ER_NET_PACKET_TOO_LARGE error, and that confirms that you need to
increase max_allowed_packet.
Doc link: Error lost connection
and also check here
for some reason when I open a connection the the Percona MySQL database on my HostGator website, after fetching the query, it will disconnect/ close the connection about 10 seconds later.
I typically wouldn't care, but HeidiSQL freezes up, preventing exporting or sorting the returned rows with it's UI unless I connect again.
Any thoughts on making the connection last longer? is it something I can do myself, or will it require a dedicated server or some upgrade? (I'm currently on a shared one). Thanks!
Sounds like it may be the "wait" timeout on the MySQL connection.
SHOW VARIABLES LIKE 'wait_timeout'
That's the amount of time (in seconds) that MySQL will leave the session (the database connection) open while it's idle, waiting for another statement to be issued. After this amount of time expires, MySQL can close the connection.
You should be able to change this for a session, to change the timeout to 5 minutes
SET wait_timeout = 300
Verify the setting with the SHOW VARIABLES statement again.
NOTE: This is per connection. It only affects the current session. Every new connection will inherit their own wait_timeout value from the global setting.
(This is only a guess. There's insufficient information in the question to make a precise diagnosis. It could be something other than MySQL server that's closing the database connection, e.g. it could be your connection pool settings (if you are using a connection pool).
I'm setting up a PrestaShop installation on a development server which is a GCE instance and using Cloud SQL as a database server. Everything works just fine except one thing: whenever there is a long period of inactivity on the site, the first page load after that always gives me this error:
Link to database cannot be established: SQLSTATE[HY000] [2003]
If I refresh the page the error is gone and never appears again until I stop using the site for an hour or so. It almost looks like database instance is going into sleep mode or something like that.
The reason I mentioned Prestashop is the fact that I never get this error when using Adminer or connecting to the database from mysql console client.
With the per use billing model, instances are spun down after a 15 minute timeout to save you money. They then take a few seconds to be spun up when next accessed. It may be the Prestashop is timing out on these first requests (though I have no experience with that application).
Try changing your instance to a package billing, which has a 12 hour timeout, to see if this helps
https://developers.google.com/cloud-sql/faq#how_usage_calculated
According to GCE documentation,
Once a connection has been established with an instance, traffic is permitted in both directions over that connection, until the connection times out after 10 minutes of inactivity
I suspect that might be the cause. To get around it, you can try to lower the tcp keepalive time.
Refer here: https://cloud.google.com/sql/docs/compute-engine-access
To keep long-lived unused connections alive, you can set the TCP keepalive. The following commands set the TCP keepalive value to one minute and make the configuration permanent across instance reboots.
# Display the current tcp_keepalive_time value.
$ cat /proc/sys/net/ipv4/tcp_keepalive_time
# Set tcp_keepalive_time to 60 seconds and make it permanent across reboots.
$ echo 'net.ipv4.tcp_keepalive_time = 60' | sudo tee -a /etc/sysctl.conf
# Apply the change.
$ sudo /sbin/sysctl --load=/etc/sysctl.conf
# Display the tcp_keepalive_time value to verify the change was applied.
$ cat /proc/sys/net/ipv4/tcp_keepalive_time