Percona connection fails after timeout - mysql

We have multimodule project with database, and sometimes we recieve this exception:
org.springframework.dao.DataAccessResourceFailureException: PreparedStatementCallback; SQL [SELECT data FROM table]; No operations allowed after connection closed.; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed.
at org.springframework.jdbc.support.SQLExceptionSubclassTranslator.doTranslate(SQLExceptionSubclassTranslator.java:79)
...
at java.lang.Thread.run(Thread.java:745)
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed.
at sun.reflect.GeneratedConstructorAccessor108.newInstance(Unknown Source)
... 87 common frames omitted
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 51,072,384 milliseconds ago. The last packet sent successfully to the server was 51,072,384 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
at sun.reflect.GeneratedConstructorAccessor89.newInstance(Unknown Source)
... 78 common frames omitted
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
... 83 common frames omitted
Google suggested that it could be because of database closes connections when no queries in time specified in wait_timeout variable.
So I reprodused this situation in that way:
Use Percona server 5.7.16, SpringBoot 1.3.3
Set timeouts in mysql to demonstrate problem (in production sets default value 8 hours):
SET GLOBAL interactive_timeout = 15;
SET GLOBAL wait_timeout = 15;
Set heartbeat for mysql in springboot properties:
spring.datasource.validationQuery=SELECT 1
spring.datasource.testWhileIdle = true
Run test:
#Test
#DatabaseSetup("/....xml")
public void test() throws Exception {
mockMvc
.perform(get("/request_that_fetches_data_from_db"))
.andExpect(status().isOk());
sleep(1000 * 60);
mockMvc
.perform(get("/request_that_fetches_data_from_db"))
.andExpect(status().isOk());
}
Result is:
java.sql.SQLException: Could not retrieve transation read-only status server
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:957)
...
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 40 414 milliseconds ago. The last packet sent successfully to the server was 1 milliseconds ago.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
...
Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2957)
... 46 more
I looked at mysql general log. There are query SELECT 1 passes every 10 seconds, but when after that recieve business query, then straight recieve command quit. Error log was empty for testing period.
What is wrong and how to fix this problem without increasing wait_timeout?
P.S.jdbc:mysql://url?autoReconnect=true don't works too.

Related

mysql connection Can not read response from server

MySQL 5.7, a transaction is running but thread is sleeping, client request(tomcat) is blocking, it will last for many many seconds, after killing connection in MySQL, tomcat receives below exception:
org.springframework.dao.RecoverableDataAccessException:
### Error querying database. Cause: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 852,932 milliseconds ago. The last packet sent successfully to the server was 857,937 milliseconds ago.
at org.springframework.jdbc.support.SQLExceptionSubclassTranslator.doTranslate(SQLExceptionSubclassTranslator.java:98)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:82)
at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:73)
at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:446)
at com.sun.proxy.$Proxy59.selectList(Unknown Source)
at org.mybatis.spring.SqlSessionTemplate.selectList(SqlSessionTemplate.java:230)
......
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:989)
at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:2222)
at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1982)
at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:3407)
at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:470)
at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:3109)
at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2334)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2733)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2549)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1861)
at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1192)
......
Caused by: java.io.EOFException: Can not read response from server. Expected to read 20,481 bytes, read 19,682 bytes before connection was unexpectedly lost.
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3008)
at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:2205)
I use alibaba druid Connection Pool, testOnBorrow=true, mysql java driver version is 5.1.40
The above exception is thrown after connection is killed, tomcat is blocking until the connection is killed.
This case occurs several times in production evn, it is hard to repeat in develop env.
As the caused exception is Can not read response from server. Expected to read 20,481 bytes, read 19,682 bytes before connection was unexpectedly lost., client is waiting for more data from server, so I guess the connection borrowed from pool is valid at first, but why can't reading more data from server?
BTW: we recently use MySQL /*+ MAX_EXECUTION_TIME(xxx) */ optimizer hint. MySQL will throw Exception if query is timeout, I don't know whether it is related with my problem, but I guess it should not.
Presumably your query from your servlet ran for longer than your MAX_EXECUTION_TIME. At any rate, it looks like your query timed out when it had been running for 853 seconds.
And, your servlet crashed. The error trace you provided shows what happened.
If you want your servlet to recover from having its JDBC connections killed, you must program it to catch an exception when you KILL its database process, and then repeat the query.
You'll have to do some experimenting to get this right.
You can force the failure in development by using a much shorter MAX_EXECUTION_TIME. Or you can access your development MySQL server with a command client, look up the processid of the process serving your spring servlet, and KILL it manually.
The root cause: I believe your query takes far too long. You must optimize it somehow. Look at query-performance for ideas about that.

After a day grails program hangs. Need solution

Here is the log. Problem occurse after a day with no interaction whatsoever. Is it the asynchronous mail plugin? is it a setting? Who can help me with this, any help is appreciated.
Grails 2.4.0, ubuntu 16.04
The last packet successfully received from the server was 278.019 milliseconds ago. The last packet sent successfully to the server was 218.021 milliseconds ago.
ERROR 2017-01-19 10:37:47,642 interceptor.TransactionInterceptor Application exception overridden by rollback exception
org.springframework.dao.RecoverableDataAccessException: Hibernate operation: could not prepare statement; SQL [update async_mail_mess set status=? where end_date<? and (status=? or status=?)]; Communications link failure
The last packet successfully received from the server was 278.019 milliseconds ago. The last packet sent successfully to the server was 218.021 milliseconds ago.; nested exception is com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 278.019 milliseconds ago. The last packet sent successfully to the server was 218.021 milliseconds ago.
at grails.plugin.asyncmail.AsynchronousMailPersistenceService$_updateExpiredMessages_closure4.doCall(AsynchronousMailPersistenceService.groovy:66)
at org.grails.datastore.gorm.GormStaticApi.withTransaction(GormStaticApi.groovy:814)
at org.grails.datastore.gorm.GormStaticApi.withTransaction(GormStaticApi.groovy:714)
at grails.plugin.asyncmail.AsynchronousMailPersistenceService.updateExpiredMessages(AsynchronousMailPersistenceService.groovy:63)
at grails.plugin.asyncmail.ExpiredMessagesCollectorJob.execute(ExpiredMessagesCollectorJob.groovy:13)
at grails.plugins.quartz.GrailsJobFactory$GrailsJob.execute(GrailsJobFactory.java:104)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 278.019 milliseconds ago. The last packet sent successfully to the server was 218.021 milliseconds ago.
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3853)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2471)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2651)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2671)
at com.mysql.jdbc.ConnectionImpl.setAutoCommit(ConnectionImpl.java:5024)
... 8 more
Caused by: java.net.SocketException: Broken pipe (Write failed)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3834)
... 12 more
ERROR 2017-01-19 10:37:47,642 listeners.ExceptionPrinterJobListener Exception occurred in job: Grails Job
org.quartz.JobExecutionException: org.springframework.transaction.TransactionSystemException: Could not roll back Hibernate transaction; nested exception is org.hibernate.TransactionException: rollback failed [See nested exception: org.springframework.transaction.TransactionSystemException: Could not roll back Hibernate transaction; nested exception is org.hibernate.TransactionException: rollback failed]
at grails.plugins.quartz.GrailsJobFactory$GrailsJob.execute(GrailsJobFactory.java:111)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: org.springframework.transaction.TransactionSystemException: Could not roll back Hibernate transaction; nested exception is org.hibernate.TransactionException: rollback failed
at grails.plugin.asyncmail.ExpiredMessagesCollectorJob.execute(ExpiredMessagesCollectorJob.groovy:13)
at grails.plugins.quartz.GrailsJobFactory$GrailsJob.execute(GrailsJobFactory.java:104)
... 2 more
Caused by: org.hibernate.TransactionException: rollback failed
... 4 more
Caused by: org.hibernate.TransactionException: unable to rollback against JDBC connection
... 4 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Communications link failure during rollback(). Transaction resolution unknown.
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.Util.getInstance(Util.java:386)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1013)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:982)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927)
at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4757)
... 4 more
ERROR 2017-01-19 10:38:09,620 spi.SqlExceptionHelper Communications link failure
It could be because of two reasons
1.) your grails box got disconnected from the network.
2.) Your database server stops.
You can add configuration parameters to your dataSource config to ensure the DB connection is kept alive
dataSource {
properties {
validationQuery=SELECT 1 FROM DUAL
validationQueryTimeout=3
validationInterval=15000
}
}
The validationQuery will depend on your database vendor e.g. the above works with Oracle

Communications link failure to MySQL during long LOCK TABLES

I am encountering some strange behaviour. I have a Java programm that permanently writes data to a MySQL table using prepared statements and batch inserts. If some other process issues a LOCK TABLES t WRITE on the same table, and releases it shortly thereafter (after a few minutes), the Java program continues as expected. However, when the locking sustains for a longer period (more than 30 minutes), the Java program looses connection and does not continue. After 2 hours, it fails with a Communications link failure. The insert statement that is pending during table lock is executed after the lock is released, but the connection is gone afterwards.
Here are the details:
it happens on both MySQL 5.0.51a and 5.1.66
I'm using the most recent JDBC driver mysql-connector-5.1.25-bin.jar
the wait_timeout is set to 28800 (8 hours)
the stack trace of the Communications link failure and the Java program are given below
Does anyone know what's going on here? Are there any timeouts I need to set/increase?
The exception thrown after two hours:
Exception in thread "main" java.sql.BatchUpdateException: Communications link failure
The last packet successfully received from the server was 7.260.436 milliseconds ago. The last packet sent successfully to the server was 7.210.431 milliseconds ago.
at com.mysql.jdbc.PreparedStatement.executeBatchedInserts(PreparedStatement.java:1836)
at com.mysql.jdbc.PreparedStatement.executeBatch(PreparedStatement.java:1456)
at com.mysql.jdbc.CallableStatement.executeBatch(CallableStatement.java:2499)
at main.LockTest.main(LockTest.java:26)
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 7.260.436 milliseconds ago. The last packet sent successfully to the server was 7.210.431 milliseconds ago.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1121)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3670)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3559)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4110)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2570)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2731)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2815)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2155)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2458)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2375)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2359)
at com.mysql.jdbc.PreparedStatement.executeBatchedInserts(PreparedStatement.java:1792)
... 3 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:114)
at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:161)
at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:189)
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3116)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3570)
... 13 more
The full Java program:
package main;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.SQLException;
import java.util.Date;
public class LockTest {
private static final String CONNECTION_PATTERN = "jdbc:mysql://%s/?user=%s&password=%s"
+ "&autoReconnect=true&rewriteBatchedStatements=true&characterEncoding=utf8";
private static final String QUERY = "INSERT INTO test.lock_test (random_text) VALUES (?)";
public static void main(String[] args) throws SQLException, InterruptedException {
Connection con = DriverManager.getConnection(String.format(CONNECTION_PATTERN, "host", "user", "pw"));
PreparedStatement ps = con.prepareCall(QUERY);
int i = 0;
while (true) {
i++;
ps.setString(1, Long.toString(System.currentTimeMillis()));
ps.addBatch();
if (i % 10 == 0) {
ps.executeBatch();
System.out.println(new Date() + ": inserting 10 rows");
}
Thread.sleep(5000);
}
}
}
Finally, I found out about the real reason. MySQL's wait_timeout has nothing to do with problem--as long as a statement is running (no matter for how long), the database session is not in the state SLEEP and hence the session will never be closed by MySQL.
The reason seems to be the Linux operating system (Debian 6 or 7), which closes the tcp connection after being idle for 2 hours (the concrete mechanism is unknown to me, maybe someone can elaborate on this?).
To avoid the described timeouts, one needs to send more frequent tcp keep alive packets. To do so, one has to decrease /proc/sys/net/ipv4/tcp_keepalive_time (I decreased it from 7,200 to 60). cat 60 > /proc/sys/net/ipv4/tcp_keepalive_time does this temporarily; to retain it after a system reboot, one has to set net.ipv4.tcp_keepalive_time=60 in /etc/sysctl.conf.
I just search the same question. I think my answer can help you .
view your exception
The last packet successfully received from the server was 7.260.436 milliseconds ago.
That's apparently means that 7.260.436 milliseconds < wait_timeout is set to 28800
2.If your application throws exception because the wait_timeout is too small. The exception show like this:
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 46,208,149 milliseconds ago. The last packet sent successfully to the server was 46,208,150 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
; SQL []; The last packet successfully received from the server was 46,208,149 milliseconds ago.
The last packet sent successfully to the server was 46,208,150 milliseconds ago.
is longer than the server configured value of 'wait_timeout'.
You should consider either expiring and/or testing connection validity before use in your application,
increasing the server configured values for client timeouts,
or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.;
that means 46,208,150 milliseconds ago > wait_timeout
Answer !!!
Be attention to the following key word "Communications link failure" ,this is mostly about MySQl variables: net_write_timeout or net_read_timeout。
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
the default value of net_write_timeout and net_read_timeout is small ,you can execute
mysql> show variables like '%timeout%'; to find the result.

Grails and MySQL Connection Exception

I have a grails application that is in production now. This morning I was alerted to the fact that the server was not resolving. Tomcat was spinning and spinning. I researched and it looks like it has to do with MySQL causing the connection to timeout after 8 hours of inactivity. I have found examples on stackoverflow of people having similar problems. However, all of these people mention that if they hit the server again and the connection is refreshed. For me, the site was down entirely and Tomcat wouldn't respond. Does it sound like something else could be at play here?
Last Exception in Tomcat log
2011-Aug-30 23:58:43,283 [TP-Processor19] org.hibernate.util.JDBCExceptionReporter
ERROR The last packet successfully received from the server was 37,118,147 milliseconds ago. The last packet sent successfully to the server was 37,122,138 milliseconds ago. \
is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing \
the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
2011-Aug-30 23:58:43,290 [TP-Processor19] org.codehaus.groovy.grails.web.errors.GrailsExceptionResolver
ERROR Exception occurred when processing request: [GET] /picks/ncaafb
Stacktrace follows:
java.net.SocketException: Connection timed out
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3302)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1940)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2113)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2568)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2113)
at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2275)
at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2275)
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
at sportsdb.Season.getCurrentNCAAFootballSeason(Season.groovy:93)
at PicksController$_closure2.doCall(PicksController.groovy:60)
at PicksController$_closure2.doCall(PicksController.groovy)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:774)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:896)
at java.lang.Thread.run(Thread.java:662)
2011-Aug-30 23:58:43,315 [TP-Processor19] org.hibernate.util.JDBCExceptionReporter
ERROR Already closed.
2011-Aug-30 23:58:43,315 [TP-Processor19] org.hibernate.util.JDBCExceptionReporter
ERROR Already closed.
2011-Aug-30 23:58:43,316 [TP-Processor19] org.codehaus.groovy.grails.web.servlet.GrailsDispatcherServlet
ERROR HandlerInterceptor.afterCompletion threw exception
org.hibernate.exception.GenericJDBCException: Cannot release connection
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:774)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:896)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.sql.SQLException: Already closed.
at org.apache.commons.dbcp.PoolableConnection.close(PoolableConnection.java:114)
at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.close(PoolingDataSource.java:191)
at $Proxy7.close(Unknown Source)
... 6 more
My plan is to implement the solution mentioned in the link above, but I wanted to make sure nothing else visibly fishy was going on since we have a somewhat different result (their connections are refreshing and mine are not).
If you're using a Tomcat JNDI DataSource look at some of the parameters you can set on the datasource such as testOnBorrow. If validation fails the connection will be dropped from the pool. There is some performance overhead that you will incur by testing connections, but it should fix problems like this. If you have minIdle/maxIdle set high that would explain why you continue to experience the problem while reconnecting fixes it for other people.

MySQL 5.1.32-community and JDK 1.6.0_16

I have this real weird exception when issuing a basic SELECT, using MySQL 5.1.32-community from JDK 1.6.0_16 code. The conditions to repeat : at any place in my app, if i change the date of my OS (WindowsXP SP3) and after that hit some refresh button, that is reloading info from database, while doing this multiple times (usually 2-3 times are enough) i get this error:
Last packet sent to the server was 0 ms ago.
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
Last packet sent to the server was 0 ms ago.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1074)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3134)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1818)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1961)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2537)
at com.mysql.jdbc.ConnectionImpl.setTransactionIsolation(ConnectionImpl.java:5148)
followed by some of my own code, and continued by this :
Caused by: java.net.SocketException: Software caused connection abort: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3119)
... 128 more
Now, i have already google it and i found zero solutions. I have already tried using "127.0.0.1" instead of "localhost" with no results. Also, i was able to test this on MySQL 4.12 and JRE 1.4 and the problem is inexistent there. I am using plain JDBC, single connection to DB (not pooled connections) and my SQL tables are using InnoDB type.
Any help will be highly appreciated as i am pretty stuck there by this phenomenon.
Based on what you described, the driver uses the OS time to time out a connection or network operation.
The workaround is likely just to catch the exception and retry setting up the connection.