resume for XID raised 0: unknown - mysql

I am using Spring 3.0.5, Hibernate 3.6.7, Atomikos TransactionEssentials 3.7.0 and MySQL 5.5
I recently faced the problem, that connections in my connectionpool timedout after 8 hours and were reset by the server causing a message "resume for XID '???' raised 0: unknown
Here is my datasource configuration:
<bean id="myDataSource"
class="com.atomikos.jdbc.AtomikosDataSourceBean"
init-method="init"
destroy-method="close"
depends-on="myConfigurer">
<property name="uniqueResourceName" value="myPUDataSource" />
<property name="xaDataSourceClassName" value="$CONF{database.XAdriver}" />
<property name="poolSize" value="10" />
<property name="xaProperties">
<props>
<prop key="user">$CONF{database.user}</prop>
<prop key="password">$CONF{database.password}</prop>
<prop key="URL">$CONF{database.url}</prop>
<prop key="pinGlobalTxToPhysicalConnection">true</prop>
<prop key="autoReconnect">true</prop>
</props>
</property>
</bean>
As I was investigating I found out that the option autoReconnect=true only reconnects my faulted connection after a delay of 2 seconds (which is configurabe I think).
Browsing the web I found solutions suggesting to increase wait_timeout in the MySQL-Server, which I think is not a real solution.
The application should be able to handle dead connections and reconnect automatically, because there may be other issues causing a connection loss, too.
(And I don't want to make any directives to Servers whatever kind the application uses).
At last I found a good solution for this, which I will post as answer to help people facing the same or similar problems.

By inserting the line:
<property name="testQuery" value="SELECT 1"/>
on the data source's configuration, it seems the connection is tested before use and reconnected if it's dead.
I tried it with wait_timeout=60 and even with a restart of the MySQL-Server while my application was running...
Result: no more Exceptions and Errors!

Related

Spring MVC & Hibernate stops working after 8 hours

The following xml config file stops working after 8 hours of inactivity with this Error:
ERROR: Already closed.
25-may-2017 8:45:23 org.apache.catalina.core.StandardWrapperValve invoke
Servlet.service() for servlet [dispatcher] in the context with path [/system] threw exception [Request processing failed; nested exception is org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is org.hibernate.TransactionException: JDBC begin transaction failed: ] with root cause:
java.net.SocketException: **Software caused connection abort: recv failed**
Below is the Configuration file:
<context:component-scan base-package="es.company.system.persistence" />
<bean id="sessionFactory" class="org.springframework.orm.hibernate4.LocalSessionFactoryBean">
<property name="dataSource" ref="dataSource" />
<property name="packagesToScan" value="es.mypackage.model"></property>
<property name="hibernateProperties">
<props>
<prop key="hibernate.dialect">org.hibernate.dialect.MySQL5Dialect</prop>
<prop key="hibernate.show_sql">false</prop>
</props>
</property>
</bean>
<bean id="dataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource" destroy-method="close">
<property name="connection.driver_class" value="com.mysql.jdbc.Driver" />
<property name="connection.url" value="jdbc:mysql://localhost:3306/system" />
<property name="connection.username" value="???" />
<property name="connection.password" value="???" />
<property name="hibernate.c3p0.min_size" value="5" />
<property name="hibernate.c3p0.max_size" value="20" />
<property name="hibernate.c3p0.timeout" value="1800"/>
<property name="hibernate.c3p0.max_statements" value="50" />
<property name="hibernate.c3p0.testConnectionOnCheckout" value="true" />
<property name="hibernate.c3p0.privilegeSpawnedThreads" value="true" />
<property name="hibernate.c3p0.contextClassLoaderSource" value="library" />
</bean>
<bean id="transactionManager"
class="org.springframework.orm.hibernate4.HibernateTransactionManager">
<property name="sessionFactory" ref="sessionFactory" />
</bean>
<bean id="persistenceExceptionTranslationPostProcessor"
class="org.springframework.dao.annotation.PersistenceExceptionTranslationPostProcessor" />
<tx:annotation-driven transaction-manager="transactionManager" />
I'm currently using Spring MVC 4.3.8 and Hibernate 4.1.9. with C3P0 conections pool for MySQL. I thought with C3P0 this would be solved, but it's not.
Anyone knows what's wrong?
Thanks!!!
Software caused connection abort: recv failed --
This usually means that there was a network error, such as a TCP timeout.
also set timeout param with default = 0 for hibernate.c3p0.timeout
hibernate.c3p0.timeout – When an idle connection is removed from the
pool (in second). Hibernate default: 0, never expire.
set autoReconnect , hibernate.c3p0.autoReconnect = true or as url connection param : jdbc:mysql://localhost:3306/test?autoReconnect=true
Add < property name="hibernate.c3p0.preferredTestQuery">SELECT 1</property>
and check it log that c3p0 execute it(if not add real select count(1) from real table with limit 1). Also check your network timeout
I have experienced this issues some time google cloud sql where the SQL server breaks all the connection after idle time out of 10 hrs. But in the application we still have the sessionFactory instance and can create sessions but when you actually fire a db query it fails with broken connection exception.
None of the below attempts worked.
1. auto reconnect set in connection url
2. auto reconnect, timeout, validation properties in xml file.
3. Wrote a polling service to make a jdbc call to keep the connection active for every 8 hrs (<10 hrs idle time out period)
4. Closed and opened new sessionFactory for every db operation.
Finally i have created a polling service which will shutdown the sessiongFactory and create a new one.
NOTE: Calling close method on sessionFactory object is not sufficient. You need to explicit set it to null;
This may be a workaround but it saved me from day restart of the application.
I was not using spring just rest service with hibernate.

java.sql.SQLException: Already closed

We have a webapp running in production on tomcat with a MySQL back-end. All was fine for sometime, then suddenly we started getting this exception java.sql.SQLException: Already closed.
The entire stack trace is:
DEBUG [org.springframework.jdbc.datasource.DataSourceUtils] Fetching JDBC Connection from DataSource
DEBUG [org.springframework.jdbc.datasource.DataSourceUtils] Returning JDBC Connection to DataSource
DEBUG [org.springframework.jdbc.datasource.DataSourceUtils] Could not close JDBC Connection
java.sql.SQLException: Already closed.
at org.apache.commons.dbcp.PoolableConnection.close(PoolableConnection.java:114)
at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.close(PoolingDataSource.java:191)
at org.springframework.jdbc.datasource.DataSourceUtils.doReleaseConnection(DataSourceUtils.java:333)
at org.springframework.jdbc.datasource.DataSourceUtils.releaseConnection(DataSourceUtils.java:294)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:405)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:428)
at com.nokia.analytics.aws.aggregate.service.importer.DBInsert.truncateTable(DBInsert.java:135)
at com.blah.analytics.aggregate.service.importer.AggregateCollector.pullAndInsert(AggregateCollector.java:85)
at com.blah.analytics.aggregate.service.importer.AggregateCollector.call(AggregateCollector.java:96)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
We are using org.apache.commons.dbcp.BasicDataSource as our datasource. I searched quite a bit but to no avail.
It doesn't occur always and hence is very hard to reproduce. It seems a problem with db connection pooling. Somewhere it was suggested to set this param as negative. Currently we are not changing of those parameters (all have default vals).
What approach should we follow to avoid it?
EDIT:
The relevant code is in (DBInsert.java)
133: String sql = "DELETE FROM "+tableName;
134: logger.debug(sql);
135: this.jdbcTemplate.execute(sql);
(133-135 are line nos. which are specified in the exception)
My datasource config:
<bean id="bisToolDataSource" class="org.apache.commons.dbcp.BasicDataSource"
destroy-method="close">
<property name="driverClassName" value="com.mysql.jdbc.Driver" />
<property name="url"
value="${url}/blah_db?verifyServerCertificate=false&useSSL=true&requireSSL=true" />
<property name="username" value="${uname}" />
<property name="password" value="${passwd}" />
</bean>
The cause of this problem is connection isn't used in a long time, add testOnBorrow and validationQuery property to your datasource configuration then your application will work fine.
Good luck:)
As user NobodyElse pointed out, the problem was related to connection pooling. I was using org.apache.commons.dbcp.BasicDataSource as datasource. The nature of the application is such that there is spurt of connections at some fixed time in the day and no connections at all for the entire day. So due to this connections in the pool were getting stale and when next day application tried to connect to DB, we were getting this exception.
There are basically two solutions to this:
The one pointed out by NobodyElse, that is to use testOnBorrow; details can be found here
The other solution (which I employed for our app) is to turn off pooling completely. Note do this only when the application is not DB intensive (which was true in our case). So I switched to org.springframework.jdbc.datasource.DriverManagerDataSource. The config for which it seems to be working fine is:
<bean id="dataSource" class="org.springframework.jdbc.datasource.DriverManagerDataSource">
<property name="driverClassName" value="com.mysql.jdbc.Driver" />
<property name="url"
value="${url}/blah_db?verifyServerCertificate=false&useSSL=true&requireSSL=true" />
<property name="username" value="${uname}" />
<property name="password" value="${passwd}" />
</bean>

Hibernate, c3p0, and Mysql, spring

Many related issues are there at stack overflow but still i am putting this because still i haven't clear solution and exact reason for my problem.
I hope here i get complete solution of my problem and help full for All
I am using c3p0, MySQL, hibernate(J PA) and spring in my application.
Listing of main jar files i am using in my application is:
c3p0-0.9.2.jar
mchange-commons-java-0.2.3.3.jar
hibernate-commons-annotations-3.0.0.ga.jar
hibernate-entity manager.jar
hibernate-tools.jar
hibernate-validator-4.0.2.GA.jar
hibernate3.jar
mysql-connector-java-5.1.13-bin.jar
c3p0 settings in my persistent.XML is like this:
<property name="hibernate.c3p0.timeout" value="300"/>
<property name="hibernate.c3p0.max_statements" value="50"/>
<property name="hibernate.c3p0.idle_test_period" value="300"/>
Here that somewhere i read that hibernate.c3p0.idle_test_period must
not be higher of the hibernate.c3p0.timeout. In my case both are equal.
Is i need to change anything ?
how can i solve this problem ?
Server works fine for some hours(may be till 8 hours not exactly) and gives following error. error log is:
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure
The last packet successfully received from the server was 35,019,246 milliseconds
ago. The last packet sent successfully to the server was 0 milliseconds ago.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:409)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1118)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3055)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2941)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3489)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1959)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2113)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2568)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2113)
at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2275)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery
(NewProxyPreparedStatement.java:116)
at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:186)
at org.hibernate.loader.Loader.getResultSet(Loader.java:1787)
at org.hibernate.loader.Loader.doQuery(Loader.java:674)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:236)
at org.hibernate.loader.Loader.doList(Loader.java:2220)
... 39 more
Caused by: java.io.EOFException: Can not read response from server.
Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2502)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2952)
... 52 more
... 39 more
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3302)
... 50 more
Here I am showing the snapshot of my persistent.xml:-
<properties>
<property name="hibernate.connection.username" value="****"/>
<property name="hibernate.connection.driver_class" value="com.mysql.jdbc.Driver"/>
<property name="hibernate.connection.password" value="***************"/>
<property name="hibernate.connection.url" value="jdbc:mysql://xxx.xxx.x.xx:3306/projectname?autoReconnect=true"/>
<!--Connection Pooling c3p0 configuration-->
<!--Minimum number of JDBC connections in the pool. Hibernate default: 1-->
<property name="hibernate.c3p0.min_size" value="5"/>
<!--Maximum number of JDBC connections in the pool. Hibernate default: 100-->
<property name="hibernate.c3p0.max_size" value="20"/>
<!--When an idle connection is removed from the pool (in second). Hibernate default: 0, never expire.-->
<property name="hibernate.c3p0.timeout" value="300"/>
<!--Number of prepared statements will be cached. Increase performance. Hibernate default: 0 , caching is disable.-->
<property name="hibernate.c3p0.max_statements" value="50"/>
<!--idle time in seconds before a connection is automatically validated. Hibernate default: 0-->
<property name="hibernate.c3p0.idle_test_period" value="300"/>
</properties>
I use singleton object of connection class in my application.
public static void createConnection()
{
if (em != null)
{
Connection.em.clear();
}
if (em == null) {
new Connection();
}
Thanks in advance. Hope i will get my solution.
--Om--
A singleton Connection object is a bad idea (at least it is in any concurrent, multi-client application). It is certainly inconsistent with a Connection pool, the philosophy of which is that Connections should be acquired from the pool when they are needed and returned to the pool [by calling close()] as soon as a unit of work is done.
Right now, your singleton Connection will timeout, and there is nothing c3p0 can do about it. Connection pools manage and test Connections when they are checked into the pool. When they are checked out, the pool tries to stay out of the way and not interfere with the client.
==> All of that said, it's unclear what your "Connection" class is here. It looks like it is not a java.sql.Connection after all, because you have a static field called "em" in Connection. So it is hard to interpret what's going on.
Broadly, you want a single c3p0 DataSource, and to acquire Connections as-needed and to close() them as promptly as possible.
To prevent Connection timeouts like you are seeing, the best approach a Connection testing regime. The easiest approach would be to set...
hibernate.c3p0.validate=true
hibernate.c3p0.preferredTestQuery=SELECT 1
Once you have that working, you can enhance the performance a bit by using testConnectionOnCheckin and idleConnectionTestPeriod (or hibernate.c3p0.idle_test_period). See http://www.mchange.com/projects/c3p0/#configuring_connection_testing
It makes no sense to have your idleConnectionTestPeriod (hibernate.c3p0.idle_test_period) and maxIdleTime (hibernate.c3p0.timeout) be the same. Idle connections will never be tested; they'll expire before they get the chance. idleConnectionTestPeriod (hibernate.c3p0.idle_test_period), if you set it at all, should be much shorter than maxIdleTime (hibernate.c3p0.timeout).
Good luck!
As this is the first hit in Google I'll share what brought me the solution to this problem showing up on our production servers: http://drglennn.blogspot.nl/2009/05/javasqlsqlexception-communication-link.html
What made this a nasty bug is that the application will work by ommitting:
<property name="connection.provider_class">org.hibernate.connection.C3P0ConnectionProvider</property>
in your persistence-unit config but it will not use the C3P0 Connection Pool and will therefore generate connection failures.
Caused by: CommunicationsException: Communications link failure
That exception means Database isn't reachable.
Since you are able to connect to Database I guess issue might be
1) DB server is down at that time
2) DB server has run out of connections.
Try connecting to Database using some other tool, MySQL Workbench or from console, at the time you get this exception
You should rater configure you a datasource. Otherwise there is much trouble with the interaction of spring an c3o0.
<bean id="dataSource_core" class="com.mchange.v2.c3p0.ComboPooledDataSource" destroy-method="close">
<property name="driverClass" value="${hibernate.connection.driver_class}" />
<property name="jdbcUrl" value="${hibernate.connection.url}" />
<property name="user" value="${hibernate.connection.username}" />
<property name="password" value="${hibernate.connection.password}" />
<property name="minPoolSize" value="${hibernate.connection.minPoolSize}" />
<property name="maxPoolSize" value="${hibernate.connection.maxPoolSize}" />
<property name="maxIdleTime" value="${hibernate.connection.maxIdleTime}" />
<property name="maxStatements" value="${hibernate.connection.maxStmts}" />
<property name="testConnectionOnCheckout" value="${hibernate.connection.testCo}" />
<property name="testConnectionOnCheckin" value="${hibernate.connection.testCo}" />
<property name="preferredTestQuery" value="${hibernate.connection.testSQL}" />
<property name="idleConnectionTestPeriod" value="${hibernate.connection.testPeriod}" />
</bean>

Getting error "unable to find valid certification path to requested target" when using java Mailing Service with Velocity Engine

I am using Velocity enguine and java mailing service to send mails. i am using SMTP protocal.. here are my configurations for that
bean id="mailSender" class="org.springframework.mail.javamail.JavaMailSenderImpl"/
property name="host" value="lotus.acme.com"/
property name="port" value="25"/
property name="username" value="edosrvc"/
property name="password" value="wipro#123"/
property name="javaMailProperties"
props
<!-- Use SMTP transport protocol -->
prop key="mail.transport.protocol">smtp</prop>
<!-- Use SMTP-AUTH to authenticate to SMTP server -->
<prop key="mail.smtp.auth">true</prop>
<prop key="mail.smtp.ssl.trust">*</prop>
<!-- Use TLS to encrypt communication with SMTP server -->
<prop key="mail.smtp.starttls.enable">true</prop>
<prop key="mail.debug">true</prop>
</props>
</property>
</bean>
<bean id="esi" class="com.example.NotificationServiceImpl">
<property name="mailSender" ref="mailSender" />
<property name="velocityEngine" ref="velocityEngine" />
</bean>
<bean id="velocityEngine"class="org.springframework.ui.velocity.VelocityEngineFactoryBean">
<property name="velocityProperties">
<value>
resource.loader=class
class.resource.loader.class=org.apache.velocity.runtime.resource.loader.ClasspathResourceLoader
</value>
</property>
</bean>
<bean id="templateMessage" class="org.springframework.mail.SimpleMailMessage">
<property name="from" value="samplesrvc#acme.com"></property>
</bean>
i have tried including this property also
*
even then i am getting the belo error
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:174)
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:238)
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:280)
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:191)
at sun.security.validator.Validator.validate(Validator.java:218)
can anyone please help me on this issue.
I am very new to these protocals and server certificates.
so any one can please tell me how to get my work done soon.
Thanks in advance.
What version of JavaMail are you using?
If you turn on Session debugging, what does the debugging output show?
Also, try setting the System property "mail.socket.debug" to "true" for additional debugging output related to certificates.
I don't know anything about Velocity so I have no idea how those property settings in your code are getting to JavaMail, but it seems quite possible that they aren't, which would explain the problem. The debugging output will help verify what's really going on.

c3p0 connection management problems

I'm using Spring 3.0.2, Hibernate 3.5.0 and c3p0 0.9.1.2 and I'm having a ton of errors when it comes down to retrieving connections and commiting transactions. Here's my configuration of c3p0:
<bean id="dataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource" destroy-method="close">
<property name="driverClass" value="${jdbc.driverClassName}"/>
<property name="jdbcUrl" value="${jdbc.url}"/>
<property name="properties">
<props>
<prop key="c3p0.acquireIncrement">5</prop>
<prop key="c3p0.maxIdleTime">30</prop>
<prop key="c3p0.idleConnectionTestPeriod">20</prop>
<prop key="c3p0.maxPoolSize">100</prop>
<prop key="c3p0.maxStatements">0</prop>
<prop key="c3p0.minPoolSize">1</prop>
<prop key="user">${jdbc.username}</prop>
<prop key="password">${jdbc.password}</prop>
</props>
</property>
</bean>
Up until recently I got this error pretty often:
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
Last packet sent to the server was 0 ms ago.
Recently I also get these:
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Connection.close() has already been called. Invalid operation in this state.
java.sql.SQLException: Connections could not be acquired from the underlying database!
Any suggestions?
That's probably because connections are being timed out. Checkout idle_test_period setting or use bonecp http://jolbox.com which has sensible defaults instead.