Play! 2.0 - BoneCP Returning Closed Connections - mysql

I have an interesting issue which I have not been able to resolve. I am using Play! 2.0.4 and using the integrated BoneCP connection pool to get the DB connections. However, for some reason, BoneCP keeps returning closed connections.
Database Server: Amazon RDS MySQL 5, default timeout settings (which should be 8 hours...)
My Play Datasource configuration looks as follows:
db.default.driver=com.mysql.jdbc.Driver
db.default.url="jdbc:mysql://{server}/{schema}?autoReconnect=true&useUnicode=yes&characterEncoding=UTF-8"
db.default.partitionCount=4
db.default.idleConnectionTestPeriod=2 minutes
I had assumed setting the idleConnectionTestPeriod to 2 minutes surely would have prevented BoneCP from returning closed connections, but it hasn't.
Every so often, I get the following stack trace in my logs:
Exception in thread "pool-6-thread-25" java.sql.SQLException: Connection is closed!
at com.jolbox.bonecp.ConnectionHandle.checkClosed(ConnectionHandle.java:350)
at com.jolbox.bonecp.ConnectionHandle.setReadOnly(ConnectionHandle.java:1089)
at play.api.db.BoneCPApi$$anon$1.onCheckOut(DB.scala:328)
at com.jolbox.bonecp.BoneCP.getConnection(BoneCP.java:514)
at com.jolbox.bonecp.BoneCPDataSource.getConnection(BoneCPDataSource.java:114)
at play.api.db.DBApi$class.getConnection(DB.scala:64)
at play.api.db.BoneCPApi.getConnection(DB.scala:273)
at play.api.db.DB$$anonfun$getConnection$1.apply(DB.scala:129)
at play.api.db.DB$$anonfun$getConnection$1.apply(DB.scala:129)
at scala.Option.map(Option.scala:133)
at play.api.db.DB$.getConnection(DB.scala:129)
at play.api.db.DB.getConnection(DB.scala)
at play.db.DB.getConnection(DB.java:50)
at play.db.DB.getConnection(DB.java:43)
at play.db.DB.getConnection(DB.java:29)
at com.edatasource.inboxtracker.tasks.TrackSiteEventActionTask.run(TrackSiteEventActionTask.java:23)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Does anybody know how I can fix this issue? Currently, I've had to wrap the DB.getConnection() in a try/catch and just catch the exception thrown by BoneCP and retry until I retrieve a valid connection. Seems like that should be unnecessary.
Thanks for any help.

Please try with 0.8.0-beta1. There was a bug related to this.

Related

mysql connection Can not read response from server

MySQL 5.7, a transaction is running but thread is sleeping, client request(tomcat) is blocking, it will last for many many seconds, after killing connection in MySQL, tomcat receives below exception:
org.springframework.dao.RecoverableDataAccessException:
### Error querying database. Cause: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 852,932 milliseconds ago. The last packet sent successfully to the server was 857,937 milliseconds ago.
at org.springframework.jdbc.support.SQLExceptionSubclassTranslator.doTranslate(SQLExceptionSubclassTranslator.java:98)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:82)
at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:73)
at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:446)
at com.sun.proxy.$Proxy59.selectList(Unknown Source)
at org.mybatis.spring.SqlSessionTemplate.selectList(SqlSessionTemplate.java:230)
......
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:989)
at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:2222)
at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1982)
at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:3407)
at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:470)
at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:3109)
at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2334)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2733)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2549)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1861)
at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1192)
......
Caused by: java.io.EOFException: Can not read response from server. Expected to read 20,481 bytes, read 19,682 bytes before connection was unexpectedly lost.
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3008)
at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:2205)
I use alibaba druid Connection Pool, testOnBorrow=true, mysql java driver version is 5.1.40
The above exception is thrown after connection is killed, tomcat is blocking until the connection is killed.
This case occurs several times in production evn, it is hard to repeat in develop env.
As the caused exception is Can not read response from server. Expected to read 20,481 bytes, read 19,682 bytes before connection was unexpectedly lost., client is waiting for more data from server, so I guess the connection borrowed from pool is valid at first, but why can't reading more data from server?
BTW: we recently use MySQL /*+ MAX_EXECUTION_TIME(xxx) */ optimizer hint. MySQL will throw Exception if query is timeout, I don't know whether it is related with my problem, but I guess it should not.
Presumably your query from your servlet ran for longer than your MAX_EXECUTION_TIME. At any rate, it looks like your query timed out when it had been running for 853 seconds.
And, your servlet crashed. The error trace you provided shows what happened.
If you want your servlet to recover from having its JDBC connections killed, you must program it to catch an exception when you KILL its database process, and then repeat the query.
You'll have to do some experimenting to get this right.
You can force the failure in development by using a much shorter MAX_EXECUTION_TIME. Or you can access your development MySQL server with a command client, look up the processid of the process serving your spring servlet, and KILL it manually.
The root cause: I believe your query takes far too long. You must optimize it somehow. Look at query-performance for ideas about that.

ATG Connection Threads

I am facing an issue where I am getting the following error:
CONTAINER:atg.repositoryException; SOURCE:java.sql.SQLException: Unexpected exception while enlisting XAConnection java.sql.SQLException: XA error: XAResource.XAER_RMFAIL start() failed on resource'ATGProductionDS_atg11':XAResource.XAER_RMFAIL: Resource manager is unavailable.
How do I solve this?
Do ATG connection threads get closed implicitly or do we have to close it explicitly?
Do ATG connection threads get closed after updateItem() and addItem() methods implicitly?
How can we close an ATG session thread explicitly?
There are potentially a number of causes for this issue. There is a support document in Oracle support portal which helps understand where the issue is (You will have to register for Oracle Support access)
Slow running query which is never timing out due to a misconfiguration
The timeouts for your datasource(s) are not setup correctly between WebLogic and the ATG application

JDBC4 Communications Exception

I seem to have a problem, where I get a communications exception when I try to write to the database. It seems to happen after a perioid of inactivity, but I'm not sure.
Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.1.v20150605-31e8258): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 40,404,396 milliseconds ago. The last packet sent successfully to the server was 40,404,396 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
I tried to set the timeout higher, and added autoReconnect=true to the connection string. When the exception is thrown, it does retry 4 times, and then stops.
The fun thing is, that the database is located within the same server as the application-server. How come this happens, and how do I fix this?
I really hope you guys can help me!
Best regards,
Ben
EDIT
As pointed out, this questions is seen multiple places. Unfortunately the proposed solutions didn't help me. Either they had an actual communications failure (Not running mysql local on the server) or they fixed it with another connection string. I've tried all that was said, and still getting the error.
Latest error from the weekend is shown here: http://pastebin.com/wMb7Ygwd

BoneCP connection closed unexpectedly

I'm integrating DataNucleus with BoneCP-0.8.0-rc2 and I'm getting this exception, randomly:
javax.jdo.JDODataStoreException: No operations allowed after connection closed.
at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:421)
at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:230)
After reading this post, I have set
datanucleus.connectionPool.maxConnectionAgeInSeconds = 170 (seconds)
Other properties that I use:
datanucleus.connectionPool.minPoolSize=0
datanucleus.connectionPool.maxPoolSize=8
The local MySQL server where I tested this property has wait_timeout=28800 (seconds).
Since I added this new property, I'm getting the above exception more often than before.
Since the exception doesn't explicitly specify that the connection is closed by the driver, I assume it was closed by the connection manager.
Do you have any other clue what might cause this exception?

Grails and MySQL Connection Exception

I have a grails application that is in production now. This morning I was alerted to the fact that the server was not resolving. Tomcat was spinning and spinning. I researched and it looks like it has to do with MySQL causing the connection to timeout after 8 hours of inactivity. I have found examples on stackoverflow of people having similar problems. However, all of these people mention that if they hit the server again and the connection is refreshed. For me, the site was down entirely and Tomcat wouldn't respond. Does it sound like something else could be at play here?
Last Exception in Tomcat log
2011-Aug-30 23:58:43,283 [TP-Processor19] org.hibernate.util.JDBCExceptionReporter
ERROR The last packet successfully received from the server was 37,118,147 milliseconds ago. The last packet sent successfully to the server was 37,122,138 milliseconds ago. \
is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing \
the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
2011-Aug-30 23:58:43,290 [TP-Processor19] org.codehaus.groovy.grails.web.errors.GrailsExceptionResolver
ERROR Exception occurred when processing request: [GET] /picks/ncaafb
Stacktrace follows:
java.net.SocketException: Connection timed out
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3302)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1940)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2113)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2568)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2113)
at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2275)
at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2275)
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
at sportsdb.Season.getCurrentNCAAFootballSeason(Season.groovy:93)
at PicksController$_closure2.doCall(PicksController.groovy:60)
at PicksController$_closure2.doCall(PicksController.groovy)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:774)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:896)
at java.lang.Thread.run(Thread.java:662)
2011-Aug-30 23:58:43,315 [TP-Processor19] org.hibernate.util.JDBCExceptionReporter
ERROR Already closed.
2011-Aug-30 23:58:43,315 [TP-Processor19] org.hibernate.util.JDBCExceptionReporter
ERROR Already closed.
2011-Aug-30 23:58:43,316 [TP-Processor19] org.codehaus.groovy.grails.web.servlet.GrailsDispatcherServlet
ERROR HandlerInterceptor.afterCompletion threw exception
org.hibernate.exception.GenericJDBCException: Cannot release connection
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:774)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:896)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.sql.SQLException: Already closed.
at org.apache.commons.dbcp.PoolableConnection.close(PoolableConnection.java:114)
at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.close(PoolingDataSource.java:191)
at $Proxy7.close(Unknown Source)
... 6 more
My plan is to implement the solution mentioned in the link above, but I wanted to make sure nothing else visibly fishy was going on since we have a somewhat different result (their connections are refreshing and mine are not).
If you're using a Tomcat JNDI DataSource look at some of the parameters you can set on the datasource such as testOnBorrow. If validation fails the connection will be dropped from the pool. There is some performance overhead that you will incur by testing connections, but it should fix problems like this. If you have minIdle/maxIdle set high that would explain why you continue to experience the problem while reconnecting fixes it for other people.