I have Django RestFrameWork application in which we are using sqlalchemy library for MySql connection.
engine = create_engine('mysql+mysqldb://username:password#hostaddress/'
'DBname', pool_recycle=1800,
connect_args={'connect_timeout': 1800}, pool_size=10, max_overflow=10, pool_pre_ping=True)
connection = engine.connect()
As the API usage increases the Mysql is creating new connections and count of threads_connected keeps growing. After reaching max value it is throwing Too many connections error. In show processList many process will be in sleep mode. If we restart the app all the connections will be reset. The following chart indicates no.of connections v/s time. How to fix this issue.
You must close connections after you've finished using them because if you don't, the connection stays open until the webserver closes it which might take a lot of time.
The best practice would be using a connection pool. Because opening and closing connections are too heavy and decreases performance. Even if you're using a connection pool you must let the connection go after you've used it.
Related
I'm looking into aborted connection -
2022-11-21T20:10:43.215738Z 640870 [Note] Aborted connection 640870 to db: '' user: '' host: '10.0.0.**' (Got timeout reading communication packets)
My understanding is that I need to figure out whether it is an interactive or not connection, and increase wait_timeout (or interactive_timeout) accordingly. If it has no effect, then I'll need to adjust net_read_timeout or net_write_timeout and see.
I'd like to ask:
Is there a meta table that I can query for the connection type
(interactive or not)?
There are how-to's on the internet on adjusting wait_timeout (or
interactive_timeout) and all of them have rebooting the database as
the last step. Is that really required? Given that immediate effect
is not required, the sessions are supposed to come and go, and new
sessions will pick up the new value (after the system value is set),
I suppose if there is a way to track how many connections are left
with the old values, then it will be ok?
Finally, can someone suggest any blog (strategy) on handling aborted
connection or adjusting the timeout values?
Thank you!
RDS MySQL version 5.7
There is only one client that sets the interactive flag by default: the mysql command-line client. All other client tools and connectors do not set this flag by default. You can choose to set the interactive flag, because it's a flag in the MySQL client API mysql_real_connect(). So you would know if you did it. In some connectors, you aren't calling the MySQL client API directly, and it isn't even an option to set this flag.
So for practical purposes, you can ignore the difference between wait_timeout and interactive_timeout, unless you're trying to tune the timeout of the mysql client in a shell window.
You should never need to restart the MySQL Server. The timeout means the client closed the session after there has been no activity for wait_timeout seconds. The default value is 28800, which is 8 hours.
The proper way of handling this in application code is to catch exceptions, reconnect if necessary, and then retry whatever query was interrupted.
Some connectors have an auto-reconnect option. Auto-reconnect does not automatically retry the query.
In many applications, you are borrowing a connection from a connection pool, and the connection pool manager is supposed to test the connection before returning it to the caller. For example running SELECT 1; is a common test. The action of testing the connection causes a reconnect if the connection was not used for 8 hours.
If you don't use a connection pool (for example if your client program is PHP, which doesn't support connection pools as far as I know), then your client opens a new connection on request, so naturally it can't be idle for 8 hours if it's a new connection. Then the connection is closed as the request finishes, and presumably this request lasts less than 8 hours.
So this comes up only if your client opens a long-lived MySQL connection that is inactive for periods of 8 hours or more. In such cases, it's your responsibility to test the connection and reopen it if necessary before running a query.
I am running two EC2 instances on AWS to serve my application one for each. Each application can open up to 100 connections to MySql for default. For database I use RDS and t2.medium instance which can handle 312 connections at a time.
In general, my connection size does not get larger than 20. When I start sending notifications to the users to come to the application, the connection size increases a lot (which is expected.). In some cases, MySql connections increases unexpectedly and my application starts to throw PoolExhaustedException:
PoolExhaustedException: [pool-7-thread-92] Timeout: Pool empty. Unable to fetch a connection in 30 seconds, none available[size:100; busy:100; idle:0; lastwait:30000].
When I check the database connections from Navicat, I see that there are about 200 connections and all of them are sleeping. I do not understand why the open connections are not used. I use standard Spring Data Jpa to save and read my entities which means I do not open or close the connections manually.
Unless I shut down one of the instance and the mysql connections are released, both instances do not response at all.
You can see the mysql connection size change graphic here, and a piece of log here.
for some reason when I open a connection the the Percona MySQL database on my HostGator website, after fetching the query, it will disconnect/ close the connection about 10 seconds later.
I typically wouldn't care, but HeidiSQL freezes up, preventing exporting or sorting the returned rows with it's UI unless I connect again.
Any thoughts on making the connection last longer? is it something I can do myself, or will it require a dedicated server or some upgrade? (I'm currently on a shared one). Thanks!
Sounds like it may be the "wait" timeout on the MySQL connection.
SHOW VARIABLES LIKE 'wait_timeout'
That's the amount of time (in seconds) that MySQL will leave the session (the database connection) open while it's idle, waiting for another statement to be issued. After this amount of time expires, MySQL can close the connection.
You should be able to change this for a session, to change the timeout to 5 minutes
SET wait_timeout = 300
Verify the setting with the SHOW VARIABLES statement again.
NOTE: This is per connection. It only affects the current session. Every new connection will inherit their own wait_timeout value from the global setting.
(This is only a guess. There's insufficient information in the question to make a precise diagnosis. It could be something other than MySQL server that's closing the database connection, e.g. it could be your connection pool settings (if you are using a connection pool).
With the node-mysql module, there are two connection options - a single connection and a connection pool. What is the best way to set up a connection to a MySQL database, using a single global connection for all requests, or creating a pool of connections and taking one from the pool for each request? Or is there a better way to do this? Will I run in to problems using just a single shared connection for all requests?
Maintaining a single connection for the whole app might be a little bit tricky.
Normally, You want to open a connection to your mysql instance, and wait for it to be established.
From this point you can start using the database (maybe start a HTTP(S) server, process the requests and query the database as needed.)
The problem is when the connection gets destroyed (ex. due to a network error).
Since you're using one connection for the whole application, you must reconnect to MySQL and somehow queue all queries while the connection is being established. It's relatively hard to implement such functionality properly.
node-mysql has a built-in pooler. A pooler, creates a few connections and keeps them in a pool. Whenever you want to close a connection obtained from the pool, the pooler returns it to the pool instead of actually closing it. Connections on the pool can be reused on next open calls.
IMO using a connection pool, is obviously simpler and shouldn't affect the performance much.
i have web app running on GF 3.1.1 using MYSQL connection pool. When i check connections to DB using : "show processlist;" i can see everytime that only one connection is used. Why is this happening? There are a lot of threads that do something with DB. Please can someone answer my question?
Thank you very much.
You're connection pool is a pool of available connections. You might have 32 connections available but that doesn't mean it will connect them all. It will manage the connections for you. As demand increases the number of open connections will increase till your max is reached. Likewise as demand decreases your open connections will decrease down to your minimum.
Assuming your queries are fast GF is probably deciding that 1 connection is all that's needed.
Having said all that, the default min connection is 8 so log into your Admin console and look at your connection pool settings. The number of open connections should be at least what the min is set too.