Getting communication link failure error in mysql on Heroku project - mysql

I am working on a Spring hibernate project which is on Heroku and uses ClearDB. When my page is idle for some time (hardly a minute) and then I send some request to the server I get the error as follows:
HTTP ERROR 500
Problem accessing /authenticate.action. Reason:
Communications link failure
Last packet sent to the server was 16 ms ago.; nested exception is org.hibernate.exception.JDBCConnectionException: Communications link failure
Last packet sent to the server was 16 ms ago.
Caused by:
org.springframework.dao.DataAccessResourceFailureException: Communications link failure
Last packet sent to the server was 16 ms ago.; nested exception is org.hibernate.exception.JDBCConnectionException: Communications link failure
Last packet sent to the server was 16 ms ago.
at org.springframework.orm.hibernate3.SessionFactoryUtils.convertHibernateAccessException(SessionFactoryUtils.java:631)
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.translateExceptionIfPossible(HibernateJpaDialect.java:104)
at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.translateExceptionIfPossible(AbstractEntityManagerFactoryBean.java:403)
at org.springframework.dao.support.ChainedPersistenceExceptionTranslator.translateExceptionIfPossible(ChainedPersistenceExceptionTranslator.java:58)
at org.springframework.dao.support.DataAccessUtils.translateIfNecessary(DataAccessUtils.java:213)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:163)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at sun.proxy.$Proxy33.authenticate(Unknown Source)
Caused by: org.hibernate.exception.JDBCConnectionException: Communications link failure
Last packet sent to the server was 16 ms ago.
at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:131)
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:49)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:125)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:110)
at org.hibernate.engine.jdbc.internal.proxy.AbstractStatementProxyHandler.continueInvocation(AbstractStatementProxyHandler.java:129)
at org.hibernate.engine.jdbc.internal.proxy.AbstractProxyHandler.invoke(AbstractProxyHandler.java:81)
at sun.proxy.$Proxy75.executeQuery(Unknown Source)
I tried looking up on the net for answers and I made some configuration settings as follows:
Heres my datasource configuration:
<beans:bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource">
<beans:property name="url" value="#{ 'jdbc:mysql://' + #dbUrl.getHost() + #dbUrl.getPath() + '?reconnect=true'}"/>
<beans:property name="username" value="#{ #dbUrl.getUserInfo().split(':')[0] }"/>
<beans:property name="password" value="#{ #dbUrl.getUserInfo().split(':')[1] }"/>
<beans:property name="minEvictableIdleTimeMillis" value="1800000"/>
I still get the error every time I leave the page idle for a minute or so. I dont get this error if my requests to the server are non stop.
My understanding is that, this is something related to idle connections in connection pool. I never got this error when I had the database running on my local machine. Its only on Heroku that I get this error. Has anybody else got a similar problem?
Thanks,
Kiran

Related

org.javalite.activejdbc.DBException: com.mysql.jdbc.CommunicationsException: Communications link failure due to underlying exception:

Getting org.javalite.activejdbc.DBException frequently while connecting to DB using activejdbc**
Please help to know why this is happening frequently??
exception occurred while getting user :
org.javalite.activejdbc.DBException:
com.mysql.jdbc.CommunicationsException: Communications link failure
due to underlying exception:
** BEGIN NESTED EXCEPTION ** >
java.net.SocketException
MESSAGE: Connection reset by peer: socket write error>
STACKTRACE:>
java.net.SocketException: Connection reset by peer: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(Unknown Source)
at java.net.SocketOutputStream.write(Unknown Source)
at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
at java.io.BufferedOutputStream.flush(Unknown Source)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:2637)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1554)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1665)
at com.mysql.jdbc.Connection.execSQL(Connection.java:3176)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1153)
at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1266)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:353)
at org.javalite.activejdbc.DB.find(DB.java:505)
at org.javalite.activejdbc.LazyList.hydrate(LazyList.java:329)
at org.javalite.activejdbc.AbstractLazyList.isEmpty(AbstractLazyList.java:42)
at org.javalite.activejdbc.ModelDelegate.findFirst(ModelDelegate.java:223)
at com.nb.ats.db.model.UserModel.findFirst(UserModel.java:2358)
** END NESTED EXCEPTION **>
Last packet sent to the server was 0 ms ago.,*
I am using the activejdbc-1.4.14.j7-20180227.053828-4.jar with MYSQLDB
Please help to know why this is happening frequently??
And how to fix this error?
Thanks in advance
"Connection reset by peer.." means your application loses a connection to the database. You may have a network or some other connectivity issue. ActiveJDBC has nothing to so with it,

hibernate connection pool with mysql and tomcat 7

I am stuck in this problem since last 5 days but couldn't find any proper solution.
I have added zeroDateTimeBehavior=convertToNull&autoReconnect=true also with mysql connection url.
org.hibernate.exception.JDBCConnectionException: Communications link failure due to underlying exception:
** BEGIN NESTED EXCEPTION **
java.io.EOFException
STACKTRACE:
java.io.EOFException
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1865)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2256)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2756)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1536)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1626)
at com.mysql.jdbc.Connection.execSQL(Connection.java:3031)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:943)
at

Communications link failure

I got the following Error when keycloak is start and trying to login from admin console..
It takes to much time it trying to hit server directly and gives error when trying to hit from load balancer..
Can any one tell me how to remove this error.
12:46:31,369 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (Timer-2) Communications link failure
The last packet successfully received from the server was 1 milliseconds ago. The last packet sent successfully to the server was 0 milliseconds ago.
12:46:31,370 ERROR [org.keycloak.services.scheduled.ScheduledTaskRunner] (Timer-2) Failed to run scheduled task ClearExpiredEvents: org.hibernate.exception.JDBCConnectionException: could not extract ResultSet
at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:132)
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:49)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:126)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:112)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:91)
at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.getResultSet(AbstractLoadPlanBasedLoader.java:449)
at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeQueryStatement(AbstractLoadPlanBasedLoader.java:202)
at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeLoad(AbstractLoadPlanBasedLoader.java:137)
at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeLoad(AbstractLoadPlanBasedLoader.java:102)
at org.hibernate.loader.collection.plan.AbstractLoadPlanBasedCollectionInitializer.initialize(AbstractLoadPlanBasedCollectionInitializer.java:100)
at org.hibernate.persister.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:693)
at org.hibernate.event.internal.DefaultInitializeCollectionEventListener.onInitializeCollection(DefaultInitializeCollectionEventListener.java:92)
at org.hibernate.internal.SessionImpl.initializeCollection(SessionImpl.java:1933)
at org.hibernate.collection.internal.AbstractPersistentCollection$4.doWork(AbstractPersistentCollection.java:558)
at org.hibernate.collection.internal.AbstractPersistentCollection.withTemporarySessionIfNeeded(AbstractPersistentCollection.java:260)
at org.hibernate.collection.internal.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:554)
at org.hibernate.collection.internal.AbstractPersistentCollection.read(AbstractPersistentCollection.java:142)
at org.hibernate.collection.internal.AbstractPersistentCollection$1.doWork(AbstractPersistentCollection.java:171)
at org.hibernate.collection.internal.AbstractPersistentCollection$1.doWork(AbstractPersistentCollection.java:156)
at org.hibernate.collection.internal.AbstractPersistentCollection.withTemporarySessionIfNeeded(AbstractPersistentCollection.java:260)
at org.hibernate.collection.internal.AbstractPersistentCollection.readSize(AbstractPersistentCollection.java:155)
at org.hibernate.collection.internal.PersistentMap.size(PersistentMap.java:140)
at java.util.HashMap.putMapEntries(HashMap.java:500)
at java.util.HashMap.putAll(HashMap.java:784)
at org.keycloak.models.jpa.ClientAdapter.getProtocolMappers(ClientAdapter.java:304)
at org.keycloak.models.cache.entities.CachedClient.<init>(CachedClient.java:75)
at org.keycloak.models.cache.entities.CachedRealm.<init>(CachedRealm.java:201)
at org.keycloak.models.cache.infinispan.DefaultCacheRealmProvider.getRealm(DefaultCacheRealmProvider.java:153)
at org.keycloak.models.cache.infinispan.DefaultCacheRealmProvider.getRealms(DefaultCacheRealmProvider.java:193)
at org.keycloak.services.scheduled.ClearExpiredEvents.run(ClearExpiredEvents.java:16)
at org.keycloak.services.scheduled.ScheduledTaskRunner.run(ScheduledTaskRunner.java:27)
at org.keycloak.timer.basic.BasicTimerProvider$1.run(BasicTimerProvider.java:29)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 1 milliseconds ago. The last packet sent successfully to the server was 0 milliseconds ago.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1117)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3589)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3478)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4019)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2490)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2651)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2734)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2155)
at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2322)
at org.jboss.jca.adapters.jdbc.CachedPreparedStatement.executeQuery(CachedPreparedStatement.java:111)
at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:462)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:82)
... 29 more
Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3039)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3489)
... 39 more
So after spending quite a good amount of time I found that the connection url for Data base is wrong.After putting right Connection url it working fine.

spring-boot web app loses ability to connect to MySQL / RDS after a while

I have a normal spring boot 1.2.x web app with an embedded Tomcat 7.x container and connected to an RDS instance (running MySQL 5.6). If the application is idle for a period of time (8 hours?) and then it receives a request it throws the following exception
** BEGIN NESTED EXCEPTION **
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException
MESSAGE: The last packet successfully received from the server was39320 seconds ago.The last packet sent successfully to the server was 39320 seconds ago, whi
ch is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your ap
plication, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this proble
m.
STACKTRACE:
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was39320 seconds ago.The last packet sent succe
ssfully to the server was 39320 seconds ago, which is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or t
esting connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection pr
operty 'autoReconnect=true' to avoid this problem.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
... trimmed more of the stacktrace ...
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3227)
... 119 more
** END NESTED EXCEPTION **
In my application.yml (where configurations for datasource, hibernate, etc. can be set) I have the following (this is part of what I get when I call the management API /env
"applicationConfig: [classpath:/application.yml]#rds-profile":{
"spring.profiles":"rds-profile",
"spring.datasource.driverClassName":"com.mysql.jdbc.Driver",
"spring.datasource.url":"jdbc:mysql://rds-host:3306/mydb?user=mysqlusername&password=****",
"spring.datasource.schema":"classpath:/schema.sql",
"spring.datasource.username":"mysqlusername",
"spring.datasource.password":"******",
"spring.datasource.testOnBorrow":true,
"spring.datasource.validationQuery":"SELECT 1",
"spring.datasource.continueOnError":true,
"spring.datasource.timeBetweenEvictionRunsMillis":5000,
"spring.datasource.minEvictableIdleTimeMillis":5000,
"spring.datasource.max-active":500,
"spring.jpa.database-platform":"org.hibernate.dialect.MySQL5InnoDBDialect",
"spring.jpa.database":"MYSQL",
"spring.jpa.show-sql":false,
"spring.jpa.generate-ddl":false,
"spring.jpa.hibernate.ddl-auto":"none",
"spring.jpa.hibernate.dialect":"org.hibernate.dialect.MySQL5InnoDBDialect"
},
Curiously, when I call the management API "/configprops" I get this (I don't know if this is the root of the problem?
"spring.datasource.CONFIGURATION_PROPERTIES":{
"prefix":"spring.datasource",
"properties":{
"platform":"all",
"data":null,
"driverClassName":"com.mysql.jdbc.Driver",
"password":"******",
"url":"jdbc:mysql://rds-host:3306/mydb?user=mysqlusername&password=****",
"schema":"classpath:/schema.sql",
"username":"mysqlusername",
"jndiName":null,
"xa":{
"dataSourceClassName":null,
"properties":{
}
},
"continueOnError":true,
"sqlScriptEncoding":null,
"separator":";",
"initialize":true
}
},
The question is: given the above configurations and details, why is it that I am still getting the "wait_timeout" exception? I would expect the connections to be tested when borrowed and I would expect the JDBC connection pool to create valid connections if none are available... so why is my application running out of valid connections after (8 hours?) or inactivity?
Thank you.
If you are using auto-configuration to define RDS connection from the property file like this:
cloud.aws.rds.testdb.password=testdbpwd
cloud.aws.rds.testdb.username=testdbuser
cloud.aws.rds.testdb.databaseName=testdb
spring boot datasource auto-configuration will not work even you put these(or tomcat datasource conf) to your configuration file:
spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver
spring.datasource.test-on-borrow: true
spring.datasource.validation-query: SELECT 1 FROM DUAL
spring.datasource.log-validation-errors: true
I think this is the reason why you cannot validate your connections in the pool, before using them.
You need to override postProcessAfterInitialization method to set pool properties of the TomcatJdbcDataSourceFactory bean like this:
#Component
public class PoolConfiguration implements BeanPostProcessor {
#Override
public Object postProcessAfterInitialization(Object bean, String beanName) throws BeansException {
if (bean instanceof TomcatJdbcDataSourceFactory) {
TomcatJdbcDataSourceFactory tomcatJdbcDataSourceFactory = (TomcatJdbcDataSourceFactory) bean;
tomcatJdbcDataSourceFactory.setTestOnBorrow(true);
tomcatJdbcDataSourceFactory.setTestWhileIdle(true);
tomcatJdbcDataSourceFactory.setValidationQuery("SELECT 1");
}
return bean;
}
}
I could not find any other solution for this.By the way this might be a bug of spring-cloud-aws-autoconfigure packet.
Good Luck!
Try using this as well
spring.datasource.test-while-idle=true
spring.datasource.validation-interval=5000

how to use c3p0 with node failover test

We have setup MySQL NDB Cluster with three nodes and here is the details of Setup.
Node1 - MultithreadedDataNode1, SQLNode1
Node2 - MultithreadedDataNode2, SQLNode2
Node3 - Management Node
we are testing a few failover scenarios to ensure that the setup works. On the client side we are using MySQLConnector v5.1.30 along with c3p0 connection pooling. Client is configured with Multimaster parameters. So if one of the node or application goes down, JDBC queries should be executed on another node and continue to respond.
But we are observing abnormal behavior when SQLNode(Port: 3306) goes down. The connections are not failed over to another node.
The configurations are as given below:
<bean id="myDataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource"
destroy-method="close">
<property name="driverClass" value="com.mysql.jdbc.ReplicationDriver" /> <!-- useConfigs=clusterBase -->
<property name="jdbcUrl" value="jdbc:mysql:replication://192.168.102.22,192.168.102.23/hhmefep?autoReconnect=true&failOverReadOnly=false&roundRobinLoadBalance=true" />
<property name="user" value="root"/>
<property name="password" value="changeit"/>
<property name="acquireIncrement" value="2" />
<property name="minPoolSize" value="50" />
<property name="maxPoolSize" value="100" />
<property name="maxStatements" value="2000" />
<property name="maxIdleTime" value="3600" />
<property name="connectionTesterClassName" value="com.mysql.jdbc.integration.c3p0.MysqlConnectionTester" />
<property name="testConnectionOnCheckin" value="true" />
<property name="testConnectionOnCheckout" value="false" />
<property name="idleConnectionTestPeriod" value="30" />
</bean>
When I start the client, these two nodes are up and running.
After that I kill the mysqld process from first node (192.168.102.22), I get following stackTrace,
org.hibernate.exception.JDBCConnectionException: could not execute query
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:99)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
at org.hibernate.loader.Loader.doList(Loader.java:2536)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2276)
at org.hibernate.loader.Loader.list(Loader.java:2271)
at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:119)
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1716)
at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:347)
at com.mi.persistence.dao.cdr.BganActiveCallsDaoImpl.findBganActiveCallsBy_Imsi_Nsapi_Apn(BganActiveCallsDaoImpl.java:53)
at sun.reflect.GeneratedMethodAccessor283.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy23.findBganActiveCallsBy_Imsi_Nsapi_Apn(Unknown Source)
at com.mi.persistence.service.cdr.BganActiveCallsServiceImpl.findBganActiveCallsBy_Imsi_Nsapi_Apn(BganActiveCallsServiceImpl.java:22)
at sun.reflect.GeneratedMethodAccessor282.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy24.findBganActiveCallsBy_Imsi_Nsapi_Apn(Unknown Source)
at com.mi.persistence.service.callhandler.BganAccessRequestServiceImpl.getPreeptedCalls(BganAccessRequestServiceImpl.java:384)
at com.mi.taskobjects.externalsigobj.radiusobj.BGANAccessRequest.process(BGANAccessRequest.java:175)
at com.mi.entities.scheduler.SchWorkerThread.run(SchWorkerThread.java:165)
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 19,578 milliseconds ago. The last packet sent successfully to the server was 0 milliseconds ago.
at sun.reflect.GeneratedConstructorAccessor66.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:409)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1127)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3715)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3604)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4155)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2615)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2776)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2838)
at com.mysql.jdbc.LoadBalancedMySQLConnection.execSQL(LoadBalancedMySQLConnection.java:159)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2082)
at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2212)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:76)
at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:208)
at org.hibernate.loader.Loader.getResultSet(Loader.java:1953)
at org.hibernate.loader.Loader.doQuery(Loader.java:802)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:274)
at org.hibernate.loader.Loader.doList(Loader.java:2533)
... 42 more
Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3161)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3615)
... 56 more
Apr 24, 2014 2:13:13 PM com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector run
WARNING: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector#18138d9 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
Apr 24, 2014 2:13:13 PM com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector run
WARNING: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector#18138d9 -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 3
Active Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask#2dbdcf (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask#1088a28 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask#1a67efd (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
Pending Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask#1db547d
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask#87fa45
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask#13fcd69
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#1c6dc6d
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#b198d1
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#1a7ce01
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#1c5be25
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#1469706
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#1654503
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#195f1b4
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#12f893d
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#1db4400
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#4d7ab3
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#b29aa5
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#1c6fb4f
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#9d2c8e
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#d843e0
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StmtAcquireTask#504b76
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#18a408c
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#193e4a4
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#87d60d
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#1f6002a
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#184e113
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#d4bbb0
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#6e4a4c
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#cfc9f4
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#1d3c0f3
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#120989c
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#1034f45
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#497a1b
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#1ef8e0b
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#568dc9
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#76bb9c
Pool thread stack traces:
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
java.lang.Thread.sleep(Native Method)
com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2396)
com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2326)
com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:832)
com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
java.lang.reflect.Constructor.newInstance(Constructor.java:513)
com.mysql.jdbc.Util.handleNewInstance(Util.java:409)
com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:417)
com.mysql.jdbc.LoadBalancingConnectionProxy.createConnectionForHost(LoadBalancingConnectionProxy.java:383)
com.mysql.jdbc.RandomBalanceStrategy.pickConnection(RandomBalanceStrategy.java:75)
com.mysql.jdbc.LoadBalancingConnectionProxy.pickNewConnection(LoadBalancingConnectionProxy.java:711)
com.mysql.jdbc.LoadBalancingConnectionProxy.dealWithInvocationException(LoadBalancingConnectionProxy.java:415)
com.mysql.jdbc.LoadBalancingConnectionProxy.invoke(LoadBalancingConnectionProxy.java:654)
com.mysql.jdbc.LoadBalancingConnectionProxy.invoke(LoadBalancingConnectionProxy.java:549)
$Proxy22.ping(Unknown Source)
com.mysql.jdbc.ReplicationConnection.ping(ReplicationConnection.java:986)
sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
com.mchange.v2.c3p0.impl.NewProxyConnection.rawConnectionOperation(NewProxyConnection.java:1315)
com.mysql.jdbc.integration.c3p0.MysqlConnectionTester.activeCheckConnection(MysqlConnectionTester.java:78)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:368)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishResourceOnCheckin(C3P0PooledConnectionPool.java:301)
com.mchange.v2.resourcepool.BasicResourcePool.attemptRefurbishResourceOnCheckin(BasicResourcePool.java:1606)
com.mchange.v2.resourcepool.BasicResourcePool.access$200(BasicResourcePool.java:32)
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask.run(BasicResourcePool.java:1228)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
java.lang.Thread.sleep(Native Method)
com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2396)
com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2326)
com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:832)
com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
java.lang.reflect.Constructor.newInstance(Constructor.java:513)
com.mysql.jdbc.Util.handleNewInstance(Util.java:409)
com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:417)
com.mysql.jdbc.LoadBalancingConnectionProxy.createConnectionForHost(LoadBalancingConnectionProxy.java:383)
com.mysql.jdbc.RandomBalanceStrategy.pickConnection(RandomBalanceStrategy.java:75)
com.mysql.jdbc.LoadBalancingConnectionProxy.pickNewConnection(LoadBalancingConnectionProxy.java:711)
com.mysql.jdbc.LoadBalancingConnectionProxy.dealWithInvocationException(LoadBalancingConnectionProxy.java:415)
com.mysql.jdbc.LoadBalancingConnectionProxy.invoke(LoadBalancingConnectionProxy.java:654)
com.mysql.jdbc.LoadBalancingConnectionProxy.invoke(LoadBalancingConnectionProxy.java:549)
$Proxy22.ping(Unknown Source)
com.mysql.jdbc.ReplicationConnection.ping(ReplicationConnection.java:986)
sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
com.mchange.v2.c3p0.impl.NewProxyConnection.rawConnectionOperation(NewProxyConnection.java:1315)
com.mysql.jdbc.integration.c3p0.MysqlConnectionTester.activeCheckConnection(MysqlConnectionTester.java:78)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:368)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishResourceOnCheckin(C3P0PooledConnectionPool.java:301)
com.mchange.v2.resourcepool.BasicResourcePool.attemptRefurbishResourceOnCheckin(BasicResourcePool.java:1606)
com.mchange.v2.resourcepool.BasicResourcePool.access$200(BasicResourcePool.java:32)
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask.run(BasicResourcePool.java:1228)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
java.lang.Thread.sleep(Native Method)
com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2396)
com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2326)
com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:832)
com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
java.lang.reflect.Constructor.newInstance(Constructor.java:513)
com.mysql.jdbc.Util.handleNewInstance(Util.java:409)
com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:417)
com.mysql.jdbc.LoadBalancingConnectionProxy.createConnectionForHost(LoadBalancingConnectionProxy.java:383)
com.mysql.jdbc.RandomBalanceStrategy.pickConnection(RandomBalanceStrategy.java:75)
com.mysql.jdbc.LoadBalancingConnectionProxy.pickNewConnection(LoadBalancingConnectionProxy.java:711)
com.mysql.jdbc.LoadBalancingConnectionProxy.dealWithInvocationException(LoadBalancingConnectionProxy.java:415)
com.mysql.jdbc.LoadBalancingConnectionProxy.invoke(LoadBalancingConnectionProxy.java:654)
com.mysql.jdbc.LoadBalancingConnectionProxy.invoke(LoadBalancingConnectionProxy.java:549)
$Proxy22.ping(Unknown Source)
com.mysql.jdbc.ReplicationConnection.ping(ReplicationConnection.java:986)
sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
com.mchange.v2.c3p0.impl.NewProxyConnection.rawConnectionOperation(NewProxyConnection.java:1315)
com.mysql.jdbc.integration.c3p0.MysqlConnectionTester.activeCheckConnection(MysqlConnectionTester.java:78)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:368)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishResourceOnCheckin(C3P0PooledConnectionPool.java:301)
com.mchange.v2.resourcepool.BasicResourcePool.attemptRefurbishResourceOnCheckin(BasicResourcePool.java:1606)
com.mchange.v2.resourcepool.BasicResourcePool.access$200(BasicResourcePool.java:32)
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask.run(BasicResourcePool.java:1228)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Please correct me if I am missing anything ..
Thanks,
Parimal
So, what is happening is clear: c3p0 is testing Connections on check-in. those tests are hanging, neither succeeding nor failing with an Exception, eventually sataurating and deadlocking the Thread pool.
The question I can't answer for sure is why the Connection tests are hanging. It's clear from the stack traces that what they are trying to do is reconnect, which is what you want to happen, but to your replicated (slave) server. Instead, reconnections are failing, and the Threads are falling asleep. Looking at the source to mysql's ConnectionImpl class, connectWithRetry sleeps initialTimout seconds (default 2) and retries up to maxReconnects time (default 3). by default, this means that things should hang for only up to 6 seconds. It looks like you may be experiencing longer hangs than this, though.
One workaround would be to just set autoReconnect=false. Then Connection tests will probably just fail, c3p0 will try to acquire a new Connection, which will hopefully succeed from the failover server.
Other things to try would be to reduce initialTimeout and maxConnects to make failures faster. Again, if the internal reconnect fails, c3p0 will reacquire dead Connections, which should now point to the failover server.
Ultimately, there are a couple of unknowns here: (1) Is your replication server, the one that your Connections should reconnect with, live and up and running? Because the clear issue is that the mysql driver is failing to establish a new Connection upon failure, as real question is why com.mysql.jdbc.ReplicationDriver is not behaving as advertised, and connecting to the alternative server. One possibility is that there's just a problem with the alternative server. Please verify that you can connect directly to that [ jdbc:mysql://192.168.102.23/hhmefep ]. If not, that's your problem! (2) If you can connect to the alt server, we still don't know whether the failover logic prior to connectWithRetries(...) is actually working, that is, is it trying to connect to the replicated server? (One thing that might be worth doing is to intentionally shut down the replicated server, so that there is nothing to failover too. Does the application fail more cleanly?)
I guess, as a practical matter, I'd verify that the replication server is functioning, reduce initialTimout and maxReconnects (maybe to 1 and 1), and try again. If that doesn't work, I'd set autoReconnect to false. If you do set autoReconnect to false, you can use the c3p0 parameter maxConnectionAge to ensure that, when the main server comes back, Connections are (fairly) promptly replaced by Connections back to the master. Set maxCoonnectionAge to something like a few minutes (not a few seconds, or you'll churn through Connections and reduce performance).
good luck!