Website is running at the client's server and we got the following exception...
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 45915 seconds ago.The last packet sent successfully to the server was 45915 seconds ago, which is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
at sun.reflect.GeneratedConstructorAccessor268.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1074)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3246)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1917)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2060)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2542)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1734)
at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1885)
at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:186)
at org.hibernate.loader.Loader.getResultSet(Loader.java:1787)
at org.hibernate.loader.Loader.doQuery(Loader.java:674)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:236)
at org.hibernate.loader.Loader.doList(Loader.java:2220)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2104)
at org.hibernate.loader.Loader.list(Loader.java:2099)
at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:94)
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1569)
at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:283)
at org.hibernate.impl.CriteriaImpl.uniqueResult(CriteriaImpl.java:305)
.........
My question is how JDBC driver decide the seconds 45915? Is there any variable stored in MySQL server for connection of particular database? How to get details of last established connection to MySQL and other information related to connection.
Related
MySQL 5.7, a transaction is running but thread is sleeping, client request(tomcat) is blocking, it will last for many many seconds, after killing connection in MySQL, tomcat receives below exception:
org.springframework.dao.RecoverableDataAccessException:
### Error querying database. Cause: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 852,932 milliseconds ago. The last packet sent successfully to the server was 857,937 milliseconds ago.
at org.springframework.jdbc.support.SQLExceptionSubclassTranslator.doTranslate(SQLExceptionSubclassTranslator.java:98)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:82)
at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:73)
at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:446)
at com.sun.proxy.$Proxy59.selectList(Unknown Source)
at org.mybatis.spring.SqlSessionTemplate.selectList(SqlSessionTemplate.java:230)
......
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:989)
at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:2222)
at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1982)
at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:3407)
at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:470)
at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:3109)
at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2334)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2733)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2549)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1861)
at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1192)
......
Caused by: java.io.EOFException: Can not read response from server. Expected to read 20,481 bytes, read 19,682 bytes before connection was unexpectedly lost.
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3008)
at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:2205)
I use alibaba druid Connection Pool, testOnBorrow=true, mysql java driver version is 5.1.40
The above exception is thrown after connection is killed, tomcat is blocking until the connection is killed.
This case occurs several times in production evn, it is hard to repeat in develop env.
As the caused exception is Can not read response from server. Expected to read 20,481 bytes, read 19,682 bytes before connection was unexpectedly lost., client is waiting for more data from server, so I guess the connection borrowed from pool is valid at first, but why can't reading more data from server?
BTW: we recently use MySQL /*+ MAX_EXECUTION_TIME(xxx) */ optimizer hint. MySQL will throw Exception if query is timeout, I don't know whether it is related with my problem, but I guess it should not.
Presumably your query from your servlet ran for longer than your MAX_EXECUTION_TIME. At any rate, it looks like your query timed out when it had been running for 853 seconds.
And, your servlet crashed. The error trace you provided shows what happened.
If you want your servlet to recover from having its JDBC connections killed, you must program it to catch an exception when you KILL its database process, and then repeat the query.
You'll have to do some experimenting to get this right.
You can force the failure in development by using a much shorter MAX_EXECUTION_TIME. Or you can access your development MySQL server with a command client, look up the processid of the process serving your spring servlet, and KILL it manually.
The root cause: I believe your query takes far too long. You must optimize it somehow. Look at query-performance for ideas about that.
We're having users created under Secondary Userstore(JDBC Userstore). Similarly, we have an application called MyApplication created in API Store. When users are trying to login to that MyApplication by invoking /token API which was provided by WSO2 even with correct username (in the format of TESTDOMAIN/testuser) and password also. Sometimes login is getting failed by returning a response with 400 Bad Request:
{
"error_description": "Error when handling event : PRE_AUTHENTICATION",
"error": "invalid_grant"
}
And, in the IDM Audit.log, the error was like shown below:
WARN {AUDIT_LOG}- Initiator=wso2.system.user Action=Authentication Target=TESTDOMAIN/testuser Data=null Outcome=Failure Error={"Error Message":"Un-expected error while pre-authenticating, Error when handling event : PRE_AUTHENTICATION","Error Code":"31002"}
After 5 attempts of user login, the user is getting logged in successfully without any problem.
I'm not getting any clue and not understanding why this login failure happens randomly.
Please provide your solutions/ideas regarding this issue.
UPDATED:
After enabling user core debug logs and some other logs which seems to be relevant to this issue. During authentication failure, I could see following wso2carbon.log:
DEBUG {org.wso2.carbon.user.core.jdbc.JDBCUserStoreManager} - Error occurred while checking existence of values.
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 733,140 milliseconds ago. The last packet sent successfully to the server was 733,140 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:115)
... 113 more
DEBUG {org.wso2.carbon.identity.oauth2.token.AccessTokenIssuer} - Error occurred while validating grant
org.wso2.carbon.identity.oauth2.IdentityOAuth2Exception: Error when handling event : PRE_AUTHENTICATION
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 733,140 milliseconds ago. The last packet sent successfully to the server was 733,140 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
As stated by #senthalan in the comments, let's try adding "autoReconnect=true" to the end of the connection URL.
Additionally, please verify you are having the following recommended values under connection configurations for your MySQL datasources in the master-datasources.xml. (As described in [1])
<definition type="RDBMS">
<configuration>
<url>jdbc:mysql://localhost:3306/umdb?autoReconnect=true</url>
<username>regadmin</username>
<password>regadmin</password>
<driverClassName>com.mysql.jdbc.Driver</driverClassName>
<maxActive>80</maxActive>
<maxWait>60000</maxWait>
<minIdle>5</minIdle>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1</validationQuery>
<validationInterval>30000</validationInterval>
<defaultAutoCommit>false</defaultAutoCommit>
Also, we can increase the number of max_connections from the DB side as described in [2].
mysql> SET GLOBAL max_connections = 500;
Query OK, 0 rows affected (0.00 sec)
[1] https://docs.wso2.com/display/ADMIN44x/Changing+to+MySQL
[2] https://stackoverflow.com/a/19991390/2910841
I published a backend project in google App Engine since 1 year, and all seems good, 1 week ago this Exception started to throw in logs sometimes (not always) during executing a JPA select query in Cloud SQL and sometimes the query is just returning the result successfully:
Internal Exception: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 309,979 milliseconds ago. The last packet sent successfully to the server was 309,981 milliseconds ago.
Error Code: 0
Call: SELECT ID, created_time, CURRENCY, IP, NAME FROM country_table
Query: ReadAllQuery(name="Country.findAll" referenceClass=Country sql="SELECT ID, created_time, CURRENCY, IP, NAME FROM country_table")
and sometimes this exception is stopping some queries to be executed:
The last packet successfully received from the server was 219,443 milliseconds ago. The last packet sent successfully to the server was 219,446 milliseconds ago.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:44)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1117)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3851)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2471)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2651)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2739)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2149)
at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2313)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeSelect(DatabaseAccessor.java:1007)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:642)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:558)
at org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:2002)
at org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:570)
at org.eclipse.persistence.sessions.server.ClientSession.executeCall(ClientSession.java:250)
This is because MySQL closed the connection.
Probably JPA will automatically reconnect to the database. If the transaction result is correct, you can ignore this message.
Or you can extend the timer until connection close with interactive_timeout and wait_timeout on developer console. This can reduce the message.
my application (runs on a Tomcat server) uses the atomikos connection pool to connect with a mysql database. everything works fine except that the connection will
be shutdown if leave the application server not used for some hours. below is the error message I got when operate the application server again after this happens:
:58:28 AM RusticiSoftware.ScormContentPlayer.Util.Logger LogInfo
INFO: Parsing metadata
Aug 15, 2013 9:58:28 AM RusticiSoftware.ScormContentPlayer.DataHelp.JdbcDataHelper ExecuteReturnDbRows
INFO: ExecuteReturnDbRows: failed - The last packet successfully received from the server was 59,735,409 milliseconds ago. The last packet sent successfully to the server was 59,735,409 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 59,735,409 milliseconds ago. The last packet sent successfully to the server was 59,735,409 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1121)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3871)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2484)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2664)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2815)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2155)
at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1379)
at RusticiSoftware.ScormContentPlayer.DataHelp.JdbcDataHelper.ExecuteReturnDbRows(JdbcDataHelper.java:453)
..................................
.................................
.................................
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3852)
... 57 more
I do set the autoReconnect to true in my jndi parameters but looks it doesn't work.
<Resource name="jdbc/ScormEngineDB" auth="Container"
type="javax.sql.DataSource" driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3306/wgea_scorm?charset=utf8&useUnicode=true&characterEncoding=utf-8&autoReconnect=true"
factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
username="username" password="password" maxActive="20" maxIdle="10" pinGlobalTxToPhysicalConnection="true" testQuery="select 1"
maxWait="-1" />
I also set the log in the mysql side and find out that the test query (select 1) actually was not sent to mysql because the connection is closed. now I have to restart the application server every morning when the problem happens.
any ideas about this?
thanks
Finally I find out that Tomcat connection pool is used rather than Atomikos connection pool. So Tomcat connection pool parameters should be used in the JNDI configuration. It should be like:
<Resource name="jdbc/ScormEngineDB" auth="Container"
type="javax.sql.DataSource" driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3306/wgea_scorm?charset=utf8&useUnicode=true&characterEncoding=utf-8&autoReconnect=true"
factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
username="username" password="password" maxActive="20" maxIdle="10" autoReconnectForConnectionPools="true"
autoReconnectForPools="true" pinGlobalTxToPhysicalConnection="true"
<!-- below are Tomcat connection pool parameters-->
testOnBorrow="true" logValidationErrors="true" validationQuery="select 1" testWhileIdle="true"
testOnConnect="true" validationInterval="3000000" maxWait="-1" />
the validationInterval parameter can be set to a value that is shorter than the database connection timeout so that the connection can be kept alive.
Regards the autoConnection parameter, many say that it is not recommended so it can be removed from the above JNDI configuration. Refer http://tomcat.10.x6.nabble.com/connection-autoReconnect-td4340944.html for more information
I have this real weird exception when issuing a basic SELECT, using MySQL 5.1.32-community from JDK 1.6.0_16 code. The conditions to repeat : at any place in my app, if i change the date of my OS (WindowsXP SP3) and after that hit some refresh button, that is reloading info from database, while doing this multiple times (usually 2-3 times are enough) i get this error:
Last packet sent to the server was 0 ms ago.
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
Last packet sent to the server was 0 ms ago.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1074)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3134)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1818)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1961)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2537)
at com.mysql.jdbc.ConnectionImpl.setTransactionIsolation(ConnectionImpl.java:5148)
followed by some of my own code, and continued by this :
Caused by: java.net.SocketException: Software caused connection abort: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3119)
... 128 more
Now, i have already google it and i found zero solutions. I have already tried using "127.0.0.1" instead of "localhost" with no results. Also, i was able to test this on MySQL 4.12 and JRE 1.4 and the problem is inexistent there. I am using plain JDBC, single connection to DB (not pooled connections) and my SQL tables are using InnoDB type.
Any help will be highly appreciated as i am pretty stuck there by this phenomenon.
Based on what you described, the driver uses the OS time to time out a connection or network operation.
The workaround is likely just to catch the exception and retry setting up the connection.