We just migrated from dbcp to tomcat jdbc connection pooling.
We tried the system in load and received the following exception:
java.sql.SQLException: [IA1856] Timeout: Pool empty. Unable to fetch a connection in 1 seconds, none available[size:125; busy:90; idle:0; lastwait:1000].
at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:632)
at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:174)
at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:124)
at com.inneractive.model.mappings.BasicPersistenceEntityMapping.getConnection(BasicPersistenceEntityMapping.java:233)
at com.inneractive.model.mappings.BasicPersistenceEntityMapping.callWithConnection(BasicPersistenceEntityMapping.java:243)
at com.inneractive.model.mappings.PersistenceEntityMapping.get(PersistenceEntityMapping.java:194)
at com.inneractive.model.data.client.ClientUtils.GetClientByExamples(ClientUtils.java:353)
at com.inneractive.client.ExternalAdRingsClientStart.getClientInfoByRequestParametersOrInsert(ExternalAdRingsClientStart.java:1329)
at com.inneractive.client.ExternalAdRingsClientStart.newClientSession(ExternalAdRingsClientStart.java:245)
at com.inneractive.simpleM2M.web.SimpleM2MProtocolBean.generateCampaign(SimpleM2MProtocolBean.java:235)
at com.inneractive.simpleM2M.web.SimpleM2MProtocolBean.generateCampaign(SimpleM2MProtocolBean.java:219)
at com.inneractive.simpleM2M.web.AdsServlet.doGet(AdsServlet.java:175)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:396)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Notice this:
[size:125; busy:90; idle:0; lastwait:1000]
Where are the connections that are not busy?
The busy number kept going down after this,
but we still didnt manage to get any connections.
Any ideas?
Configuration:
<Resource auth="Container" driverClassName="com.mysql.jdbc.Driver"
factory="org.apache.tomcat.jdbc.pool.DataSourceFactory" loginTimeout="10000"
maxActive="35" maxIdle="35" maxWait="1000" name="jdbc/mysql"
password="-----" testOnBorrow="true" testOnReturn="false" type="javax.sql.DataSource"
url="jdbc:mysql://localhost:3306/my_db?elideSetAutoCommits=true&useDynamicCharsetInfo=false&rewriteBatchedStatements=true&useLocalSessionState=true&useLocalTransactionState=true&alwaysSendSetIsolation=false&cacheServerConfiguration=true&noAccessToProcedureBodies=true&useUnicode=true&characterEncoding=UTF-8"
username="root" validationQuery="SELECT 1"/>
env: ubuntu and tomcat 6. db - mysql
Taking a look at the source of ConnectionPool.java you seem to hit this code snippet in the borrowConnection() method:
//we didn't get a connection, lets see if we timed out
if (con == null) {
if ((System.currentTimeMillis() - now) >= maxWait) {
throw new SQLException("[" + Thread.currentThread().getName()+"] " +
"Timeout: Pool empty. Unable to fetch a connection in " + (maxWait / 1000) +
" seconds, none available["+busy.size()+" in use].");
} else {
//no timeout, lets try again
continue;
}
}
So according to this, your connection is Null.
The value of con is retrieved on the line:
PooledConnection con = idle.poll();
if you track the code, you will see idle is (depending on your configuration, but by default) FairBlockingQueue. You may checkout the implementation for hints.
In general you always have to close ResultSets, Statements, and Connections and used connections should be correctly released back to the pool.
Not doing so correctly may result in connections never been closed => never being available again for reuse (connection pool "leaks").
I suggest you construct some detailed logging over the state of the pool and monitor it to isolate the problem.
Some guidelines from Apache for preventing database connection pool leaks:
removeAbandoned="true"
abandoned database connections are removed and recycled
removeAbandonedTimeout="60"
set the number of seconds a database connection has been idle before it is considered abandoned
logAbandoned="true"
log a stack trace of the code which abandoned the database connection resources. Keep in mind that "logging of abandoned Connections adds overhead for every Connection borrow because a stack trace has to be generated."
I still think slightly increasing the maxWait value (1200, 1500, 1700 - just experiment, there will be no difference in the response times from user perspective) will clear those rare cases, in which you still have problems.
"where are the connections that are not busy ?"
It sounds like they've been dropped and for some reason your connection pool isn't trying to reconnect them.
Add this to the URL that you're connecting to:
autoReconnect=true
And add this as a property to the resource should cause dead connections to be reconnected automatically.
validationQuery="SELECT 1"
Also this should allow you to see connections being dropped:
logAbandoned="true"
There's multiple similar questions on stack overflow.
Tomcat connection pooling,idle connections,and connection creation
JDBC Connection pool not reopening Connections in tomcat
However it may also be that you're not releasing the connections completely which is the cause of them dying.
JDBC MySql connection pooling practices to avoid exhausted connection pool
seems to be a bug in the pool, size variable is incremented, then trying to create connection,
but if creation fails... we have size value large and no actual connections in pool - terrible :
//if we get here, see if we need to create one
//this is not 100% accurate since it doesn't use a shared
//atomic variable - a connection can become idle while we are creating
//a new connection
if (size.get() < getPoolProperties().getMaxActive()) {
//atomic duplicate check
if (size.addAndGet(1) > getPoolProperties().getMaxActive()) {
//if we got here, two threads passed through the first if
size.decrementAndGet();
} else {
//create a connection, we're below the limit
return createConnection(now, con, username, password);
}
} //end if
Related
I'm using Slick3.1.1 + HikariCP2.5.1. My config is:
rdsConfig = {
url = "jdbc:mysql://mydb.........us-west-2.rds.amazonaws.com:3306/owlschema"
driver = "com.mysql.jdbc.Driver"
connectionPool = HikariCP
maxConnections = 222
minConnections = 30
keepAliveConnection = true
properties = {
user = "me"
password = "mydarksecret"
}
numThreads = 40
}
I'm running around 1 query per 3 seconds, with each query taking < 0.4s. At first everything runs fine, but after about 2 hours, HikariCP starts closing connections, resulting in errors about 'no operations allowed after connection closed':
15:20:38.288 DEBUG [] [rdsConfig-8] com.zaxxer.hikari.pool.HikariPool - rdsConfig - Timeout failure stats (total=30, active=0, idle=30, waiting=0)
15:20:38.290 DEBUG [] [rdsConfig connection closer] com.zaxxer.hikari.pool.PoolBase - rdsConfig - Closing connection com.mysql.jdbc.JDBC4Connection#229960c: (connection is evicted or dead)
15:20:38.333 DEBUG [] [rdsConfig connection closer] com.zaxxer.hikari.pool.PoolBase - rdsConfig - Closing connection com.mysql.jdbc.JDBC4Connection#229960c failed
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_77]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) ~[na:1.8.0_77]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) ~[na:1.8.0_77]
at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[na:1.8.0_77]
Is there some other config setting I need to avoid this? I don't understand why HikariCP is closing connections at all, and if it wants to, shouldn't it be only supplying my code with non-closed connections? It has a reputation for running well with its default settings so I'm confused as to why it's having problems. Thanks.
HikariCP determined that the connection was dead, i.e. (connection is evicted or dead), and therefore attempted to close it. The driver then said, "Sorry the connection is already closed", which is not unexpected.
You might think, "Why do you need to close a dead connection?" Well, maybe it was only temporarily unavailable (or slow) so the validation test failed, but the connection is still "alive" from the driver's perspective. Closing, or at least attempting to, is essential to allow the driver opportunity to cleanup resources.
HikariCP closes connections in five cases:
The connection failed validation. This is invisible to your application. The connection is retired and replaced. You would see a log message to the effect of Failed to validate connection....
A connection was idle for longer than idleTimeout. This is invisible to your application. The connection is retired and replaced. You would see a closure reason of (connection has passed idleTimeout).
A connection reached its maxLifetime. This is invisible to your application. The connection is retired and replaced. You would see a closure reason of (connection has passed maxLifetime), or if the connection is in use at the time of reaching maxLifetime you would see (connection is evicted or dead) at a later time.
The user manually evicted a connection. This is invisible to your application. The connection is retired and replaced. You would see a closure reason of (connection evicted by user).
A JDBC call threw an unrecoverable SQLException. This should be visible to your application. You would see a closure reason of (connection is broken).
There are quite a few variables here. I do not know what HikariCP default settings might be altered by Slick, in addition to user specified settings. You do not show surrounding logs, so I cannot tell if any other related issues. There is strangeness in the fact that your configuration shows 222 connections, but the pool stats logged at the Timeout Failure are (total=30, active=0, idle=30, waiting=0), so it appears that RDS may be capping you (?).
I suggest opening an issue on Github, attaching the log messages containing the pool settings at startup, attaching the section of log for the one minute preceding the exception, and grepping the log file for any other relevant warnings.
we constantly face issues with the connections to MySQL hosted by ClearDB. We have a dedicated plan which offers more then 300+ connections for our application.
I know the CBR on ClearDB site automatically closes an inactive connection after 60s.
The (Spring) application runs in Tomcat and uses a ConnectionPool with the following settings:
org.apache.tomcat.jdbc.pool.DataSource dataSource = new org.apache.tomcat.jdbc.pool.DataSource();
dataSource.setDriverClassName("com.mysql.jdbc.Driver");
dataSource.setUrl(serviceInfo.getJdbcUrl());
dataSource.setUsername(serviceInfo.getUserName());
dataSource.setPassword(serviceInfo.getPassword());
dataSource.setInitialSize(10);
dataSource.setMaxActive(30);
dataSource.setMaxIdle(30);
dataSource.setTimeBetweenEvictionRunsMillis(34000);
dataSource.setMinEvictableIdleTimeMillis(55000);
dataSource.setTestOnBorrow(true);
dataSource.setTestWhileIdle(true);
dataSource.setValidationInterval(34000);
dataSource.setValidationQuery("SELECT 1");
The error we see in our stack is:
2015-01-13T13:36:22.75+0100 [App/0] OUT The last packet successfully received from the server was 90,052 milliseconds ago. The last packet sent successfully to the server was 90,051 milliseconds ago.; nested exception is com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
2015-01-13T13:36:22.75+0100 [App/0] OUT The last packet successfully received from the server was 90,052 milliseconds ago. The last packet sent successfully to the server was 90,051 milliseconds ago.
2015-01-13T13:36:22.75+0100 [App/0] OUT ... 52 common frames omitted
2015-01-13T13:36:22.75+0100 [App/0] OUT Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
2015-01-13T13:36:22.75+0100 [App/0] OUT at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2914) ~[mysql-connector-java-5.1.33.jar:5.1.33]
2015-01-13T13:36:22.75+0100 [App/0] OUT at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3337) ~[mysql-connector-java-5.1.33.jar:5.1.33]
2015-01-13T13:36:22.75+0100 [App/0] OUT ... 64 common frames omitted
Do you have any ideas what could be causing this or did you have similar experiences with ClearDB and maybe moved somewhere else?
unfortunate I'm out of any ideas, any help is really appreciated.
The error you listed looks a lot like your connection has been disconnected on the remote end (i.e. by ClearDb). 60s is a pretty short window for idle connections, so I'd suggest a few changes to your pool config.
1.) Set initialSize and minIdle (defaults to initialSize) intentionally low. This will keep the number of idle connections low. Less idle connections means there's more of a chance the connection will be reused before the 60s window expires.
2.) You don't need maxIdle here. It defaults to maxActive.
3.) Set timeBetweenEvictionRunsMillis lower. This sets how often the pool will check for idle connections. The default of 5s is probably fine.
4.) Lower minEvictableIdleTimeMillis. This is the minimum amount of time the connection will be in the pool before it can be evicted. It doesn't mean it will be evicted exactly when it's this old though. If the idle check just ran and your connection is minEvictableIdleTimeMillis - 1s old, it will have to wait for the next check to evict the connection (i.e timeBetweenEvictionRunsMillis). If you're using the default timeBetweenEvictionRunsMillis of 5s, setting this to 50s should give it plenty of time.
5.) Set the validationInterval lower. This determines how long the pool will wait since the last successful validation before it validates the connection again. I'd go with something between 2 and 5s. It's high enough that you'll get some benefit when you're busy, and low enough that it won't cause you to miss validation on bad connections.
6.) I'd also suggest that you enable removeAbandoned and logAbandoned, with removeAbandonedTimeout set to something like 5 or 10s (most web apps shouldn't hold the db connection for that long). This will eliminate the possibility that your web app is holding the connection in an idle state for more than 60s, then trying to use it again.
I have a Java EE application running in GlassFish on EC2, with a MySQL database on Amazon RDS.
I am trying to configure the JDBC connection pool to in order to minimize downtime in case of database failover.
My current configuration isn't working correctly during a Multi-AZ failover, as the standby database instance appears to be available in a couple of minutes (according to the AWS console) while my GlassFish instance remains stuck for a long time (about 15 minutes) before resuming work.
The connection pool is configured like this:
asadmin create-jdbc-connection-pool --restype javax.sql.ConnectionPoolDataSource \
--datasourceclassname com.mysql.jdbc.jdbc2.optional.MysqlConnectionPoolDataSource \
--isconnectvalidatereq=true --validateatmostonceperiod=60 --validationmethod=auto-commit \
--property user=$DBUSER:password=$DBPASS:databaseName=$DBNAME:serverName=$DBHOST:port=$DBPORT \
MyPool
If I use a Single-AZ db.m1.small instance and reboot the database from the console, GlassFish will invalidate the broken connections, throw some exceptions and then reconnect as soon the database is available. In this setup I get less than 1 minute of downtime.
If I use a Multi-AZ db.m1.small instance and reboot with failover from the AWS console, I see no exception at all. The server halts completely, with all incoming requests timing out. After 15 minutes I finally get this:
Communication failure detected when attempting to perform read query outside of a transaction. Attempting to retry query. Error was: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.3.2.v20111125-r10461): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 940,715 milliseconds ago. The last packet sent successfully to the server was 935,598 milliseconds ago.
It appears as if each HTTP thread gets blocked on an invalid connection without getting an exception and so there's no chance to perform connection validation.
Downtime in the Multi-AZ case is always between 15-16 minutes, so it looks like a timeout of some sort but I was unable to change it.
Things I have tried without success:
connection leak timeout/reclaim
statement leak timeout/reclaim
statement timeout
using a different validation method
using MysqlDataSource instead of MysqlConnectionPoolDataSource
How can I set a timeout on stuck queries so that connections in the pool are reused, validated and replaced?
Or how can I let GlassFish detect a database failover?
As I commented before, it is because the sockets that are open and connected to the database don't realize the connection has been lost, so they stayed connected until the OS socket timeout is triggered, which I read might be usually in about 30 minutes.
To solve the issue you need to override the socket Timeout in your JDBC Connection String or in the JDNI COnnection Configuration/Properties to define the socketTimeout param to a smaller time.
Keep in mind that any connection longer than the value defined will be killed, even if it is being used (I haven't been able to confirm this, is what I read).
The other two parameters I mention in my comment are connectTimeout and autoReconnect.
Here's my JDBC Connection String:
jdbc:(...)&connectTimeout=15000&socketTimeout=60000&autoReconnect=true
I also disabled Java's DNS cache by doing
java.security.Security.setProperty("networkaddress.cache.ttl" , "0");
java.security.Security.setProperty("networkaddress.cache.negative.ttl" , "0");
I do this because Java doesn't honor the TTL's, and when the failover takes place, the DNS is the same but the IP changes.
Since you are using an Application Server, the parameters to disable DNS cache must be passed to the JVM when starting the glassfish with -Dnet and not the application itself.
I have a grails app that has flurries of high activity, but then often periods of inactivity that can last several hours to over night. I notice that the first users in the morning get the following type of exception, and I believe this is due to the connections in the pool going stale and MYSql database closing them.
I've found conflicting information in Googling about whether using Connector/J connection property 'autoReconnect=true' is a good idea (and whether or not the client will still get an exception even if the connection is then restored), or whether to set other properties that will periodically evict or refresh idle connections, test on borrow, etc. Grails uses DBCP underneath. I currently have a simple config as below, and am looking for an answer on how to best ensure that any connection grabbed out of the pool after a long inactive period is valid and not closed.
dataSource {
pooled = true
dbCreate = "update"
url = "jdbc:mysql://my.ip.address:3306/databasename"
driverClassName = "com.mysql.jdbc.Driver"
dialect = org.hibernate.dialect.MySQL5InnoDBDialect
username = "****"
password = "****"
properties {
//what should I add here?
}
}
Exception
2012-06-20 08:40:55,150 [http-bio-8443-exec-1] ERROR transaction.JDBCTransaction - JDBC begin failed
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 64,129,968 milliseconds ago. The last packet sent successfully to the server was 64,129,968 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3851)
...... Lots more .......
Caused by: java.sql.SQLException: Already closed.
at org.apache.commons.dbcp.PoolableConnection.close(PoolableConnection.java:114)
The easiest is to configure the connection pool to specify the query to be run to test the connection before it is passed to the application:
validationQuery="select 1 as dbcp_connection_test"
testOnBorrow=true
This same "connection validation" query can be run on other events. I'm not sure of the defaults for these:
testOnReturn=true
testWhileIdle=true
There are also configuration settings that limit the "age" of idle connections in the pool, which can be useful if idle connections are being closed at the server end.
minEvictableIdleTimeMillis
timeBetweenEvictionRunsMillis
http://commons.apache.org/dbcp/configuration.html
I don't know if it is the best way to handle database connection, but I had the same problems as you described. I tried a lot and ended up with the c3p0 connection pool.
Using c3p0 you could force your app to refresh the database connection after a certain time.
Place the c3p0.jar into your lib folder and add your configuration to conf/spring/resources.groovy.
My resources.groovy looks like this:
import com.mchange.v2.c3p0.ComboPooledDataSource
import org.codehaus.groovy.grails.commons.ConfigurationHolder as CH
beans = {
/**
* c3P0 pooled data source that forces renewal of DB connections of certain age
* to prevent stale/closed DB connections and evicts excess idle connections
* Still using the JDBC configuration settings from DataSource.groovy
* to have easy environment specific setup available
*/
dataSource(ComboPooledDataSource) { bean ->
bean.destroyMethod = 'close'
//use grails' datasource configuration for connection user, password, driver and JDBC url
user = CH.config.dataSource.username
password = CH.config.dataSource.password
driverClass = CH.config.dataSource.driverClassName
jdbcUrl = CH.config.dataSource.url
//force connections to renew after 4 hours
maxConnectionAge = 4 * 60 * 60
//get rid too many of idle connections after 30 minutes
maxIdleTimeExcessConnections = 30 * 60
}
}
I'm using tomcat 7 and the tomcat jdbc connection pool to dish out mysql connections.
During night times we don't have any activity so all connections become idle for longer than 8 hours and are dropped by mysql. (mysql's wait_timeout default).
We use the following pool configuration:
<Resource name="jdbc/dbName"
auth="Container"
factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
type="javax.sql.DataSource"
maxActive="50"
maxIdle="30"
maxWait="5000"
driverClassName="com.mysql.jdbc.Driver"
validationQuery="SELECT 1"
testOnBorrow="true"
testWhileIdle="true"
timeBetweenEvictionRunsMillis="10000"
removeAbandoned="true"
removeAbandonedTimeout="60"
logAbandoned="true"
username="xxx"
password="xxx"
url="jdbc:mysql://host:3306/xxx"/>
I was expecting the EvictionPolicy to remove idle connections way before they ever get closed by MySql. Somehow after one day we get the following exception:
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed.Connection was implicitly closed by the driver.
I guess this problem should be something the jdbc connection pool can fix, but there are many configuration properties and I haven't used this pool before. Anybody got a good set of properties to configure the pool to not dish out closed connections?
Kind regards,
Albert
Solved it. Turned out it wasn't a pooling problem after all. We were using squeryl and lift together which isn't a happy combi (just yet). Connections got closed before returned to the pool.
Ditching lift's DB connection management in favor of squeryl's solved it.