I am running two EC2 instances on AWS to serve my application one for each. Each application can open up to 100 connections to MySql for default. For database I use RDS and t2.medium instance which can handle 312 connections at a time.
In general, my connection size does not get larger than 20. When I start sending notifications to the users to come to the application, the connection size increases a lot (which is expected.). In some cases, MySql connections increases unexpectedly and my application starts to throw PoolExhaustedException:
PoolExhaustedException: [pool-7-thread-92] Timeout: Pool empty. Unable to fetch a connection in 30 seconds, none available[size:100; busy:100; idle:0; lastwait:30000].
When I check the database connections from Navicat, I see that there are about 200 connections and all of them are sleeping. I do not understand why the open connections are not used. I use standard Spring Data Jpa to save and read my entities which means I do not open or close the connections manually.
Unless I shut down one of the instance and the mysql connections are released, both instances do not response at all.
You can see the mysql connection size change graphic here, and a piece of log here.
Related
I'm looking into aborted connection -
2022-11-21T20:10:43.215738Z 640870 [Note] Aborted connection 640870 to db: '' user: '' host: '10.0.0.**' (Got timeout reading communication packets)
My understanding is that I need to figure out whether it is an interactive or not connection, and increase wait_timeout (or interactive_timeout) accordingly. If it has no effect, then I'll need to adjust net_read_timeout or net_write_timeout and see.
I'd like to ask:
Is there a meta table that I can query for the connection type
(interactive or not)?
There are how-to's on the internet on adjusting wait_timeout (or
interactive_timeout) and all of them have rebooting the database as
the last step. Is that really required? Given that immediate effect
is not required, the sessions are supposed to come and go, and new
sessions will pick up the new value (after the system value is set),
I suppose if there is a way to track how many connections are left
with the old values, then it will be ok?
Finally, can someone suggest any blog (strategy) on handling aborted
connection or adjusting the timeout values?
Thank you!
RDS MySQL version 5.7
There is only one client that sets the interactive flag by default: the mysql command-line client. All other client tools and connectors do not set this flag by default. You can choose to set the interactive flag, because it's a flag in the MySQL client API mysql_real_connect(). So you would know if you did it. In some connectors, you aren't calling the MySQL client API directly, and it isn't even an option to set this flag.
So for practical purposes, you can ignore the difference between wait_timeout and interactive_timeout, unless you're trying to tune the timeout of the mysql client in a shell window.
You should never need to restart the MySQL Server. The timeout means the client closed the session after there has been no activity for wait_timeout seconds. The default value is 28800, which is 8 hours.
The proper way of handling this in application code is to catch exceptions, reconnect if necessary, and then retry whatever query was interrupted.
Some connectors have an auto-reconnect option. Auto-reconnect does not automatically retry the query.
In many applications, you are borrowing a connection from a connection pool, and the connection pool manager is supposed to test the connection before returning it to the caller. For example running SELECT 1; is a common test. The action of testing the connection causes a reconnect if the connection was not used for 8 hours.
If you don't use a connection pool (for example if your client program is PHP, which doesn't support connection pools as far as I know), then your client opens a new connection on request, so naturally it can't be idle for 8 hours if it's a new connection. Then the connection is closed as the request finishes, and presumably this request lasts less than 8 hours.
So this comes up only if your client opens a long-lived MySQL connection that is inactive for periods of 8 hours or more. In such cases, it's your responsibility to test the connection and reopen it if necessary before running a query.
I have Django RestFrameWork application in which we are using sqlalchemy library for MySql connection.
engine = create_engine('mysql+mysqldb://username:password#hostaddress/'
'DBname', pool_recycle=1800,
connect_args={'connect_timeout': 1800}, pool_size=10, max_overflow=10, pool_pre_ping=True)
connection = engine.connect()
As the API usage increases the Mysql is creating new connections and count of threads_connected keeps growing. After reaching max value it is throwing Too many connections error. In show processList many process will be in sleep mode. If we restart the app all the connections will be reset. The following chart indicates no.of connections v/s time. How to fix this issue.
You must close connections after you've finished using them because if you don't, the connection stays open until the webserver closes it which might take a lot of time.
The best practice would be using a connection pool. Because opening and closing connections are too heavy and decreases performance. Even if you're using a connection pool you must let the connection go after you've used it.
I have aplications that connect to a remote server (MySQL 5.5 on Windows Server 2012), at first I started receiving "too many connections" message which I solved by increasing MAX_CONNECTION value in my.inf to 500, then I start getting "can't create new thread" message so I decrease decrease timeouts to avoid idle connections using a socket, which didn't completely work. Now I get odd messages like 'file not found', as soon as I restart the service I stop getting the messages and everything works correctly.
The problem occurs when the server reaches around 170 connections at the same time.
Is there some configuration I'm missing?, I really don't know what info you need to give me a hint to fix this. I mean, there are servers that accept a lot morw of connections at the same time, right? waht I'm missing.
RAM and CPU of the system dosen't reach 35-40% at max connections (170).
Edit: Error occur at 2 'places', when running a query or at the attempt of conennection, it's like the MySQL service rejects the attempt. VB6 is the language used in the client app (ODBC connector). The app opens, executes and closes the connection.
Note: I have full control over client app and server config.
I am creating PDF documents on AWS server with Sidekiq for processing this job on background.
While the process of creating the PDF file, the [Rails] application is pooling database to check out whether the PDF file was created or not (the interval: 2 seconds).
This morning I got this error message on the Sidekiq side:
ActiveRecord::ConnectionTimeoutError: could not obtain a database connection within 5.000 seconds (waited 5.000 seconds)
I am using Amazon RDS with MySQL on it.
As a temporary solution, I increased the pool parameter from 10 to 30 in database.yml, however I realize this is just a temporary patch.
How to fix it properly?
Thank you
I think that your solution is actually the correct one.
ActiveRecord::ConnectionPool is thread based, i.e. it tries to obtain a separate connection for each thread that wants to work with the database. If there are more threads wanting to access the database then the total size of the connection pool (configured with the pool option in database.yml), ConnectionPool tries to wait up to 5 seconds by default if a connection from some other thread is freed. After these 5 seconds time out, the ActiveRecord::ConnectionTimeoutError exception is raised.
Now, Sidekiq uses 25 worker threads by default. So, under higher load, it is perfectly possible that there will be up to 25 jobs (threads) trying to access the db at the same time. If your pool was set to 10, the excess workers had to wait for the other ones to complete and probably some thread had to wait too long.
So, either enlarge the size of the connection pool to at least a little higher value then 25 (the number of sidekiq workers), just as you did, or run your sidekiq with less workers by running it like sidekiq -c 5. Finally, always ensure that you allow enough incoming connections on the MySQL side (by default it's over 100).
Polling generally doesn't scale as a solution.
Without knowing more about the structure of your application I would be tempted to leverage some concurrency constructs from a gem like concurrent-ruby.
Specifically, the creation of a PDF file maps quite closely to the concept of a Future or a Promise
I would look at rearchitecting your worker:
Your PDF generation code should be a Promise. It should open a db connection only long enough to write the resulting PDF to the database; not while it is doing pdf generation as well.
Your main application code should spin up a PDF generation promise on Sidekiq as usual. Instead of polling the database; this code simply waits for the Promise to complete or fail; if the promise completes succesfully the PDF is in the database, if it fails you have an exception trace etc.
As always, ymmv
I am using Hibernate with c3p0 as pooling provider. I have set its max size as 50 only. Now, I performed load testing of my application with 1000 concurrent threads accessing database continuously and with mysql max_connections as 2000. I am getting proper responses from the application but sometimes I face socket exception error.
So, first thing is if my pooling size is 50 only, how 1000 connections are managed by hibernate ? Does it mean that 50 connections are being taken from the pool and rest of the connections are created? Also, why I must be getting socket exception like connection reset exception?
if you've set things up properly and c3p0's maxPoolSize is 50, then if 1000 clients hit the pool, 50 will get Connections initially and the rest will wait() briefly until Connections are returned by the first cohort. the pool's job, in collaboration with your application which should hold Connections as briefly as possible, is to ensure that a limited number of Connections are efficiently shared.
if you are seeing occasional connection reset / socket exception, you probably ought to configure some Connection testing:
http://www.mchange.com/projects/c3p0/index.html#configuring_connection_testing
the latest prerelease version has some more direct advice about connection testing; you can download that or read the html source starting here:
https://github.com/swaldman/c3p0/blob/master/src/doc/index.html#L1071