I am using hibernate to put an object into DB. The DB is MYSQL community edition.
The whole update runs into transaction.
The code that does it is this:
getHibernateTemplate().saveOrUpdate(order);
There is no problem when the order is small, but when I tested the scenario with close to 1 000 000 items I ran into this issue:
11-22#12:56:48 DEBUG PersistOrderServiceImpl [flow.ottoImportOrderPlacementInboundFlow.1] - saving order instance::
[Order [orderId=080661, vatOrderNumber=SODE000001, orderDate=Tue Nov 08 10:12:37 CET 2011, shippingMethod=STANDARD, ....
11-22#13:02:12 ERROR JDBCTransaction [flow.ottoImportOrderPlacementInboundFlow.1] - JDBC rollback failed
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
.....
....
11-22#13:02:12 ERROR TransactionInterceptor [flow.ottoImportOrderPlacementInboundFlow.1] - Application exception overridden by rollback exception
java.lang.NullPointerException
at com.mysql.jdbc.Field.setConnection(Field.java:972)
at com.mysql.jdbc.StatementImpl.getGeneratedKeysInternal(StatementImpl.java:1912)
at com.mysql.jdbc.StatementImpl.getGeneratedKeysInternal(StatementImpl.java:1905)
at com.mysql.jdbc.StatementImpl.getGeneratedKeys(StatementImpl.java:1885)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.getGeneratedKeys(NewProxyPreparedStatement.java:1749)
at org.hibernate.id.IdentityGenerator$GetGeneratedKeysDelegate.executeAndExtract(IdentityGenerator.java:97)
at org.hibernate.id.insert.AbstractReturningDelegate.performInsert(AbstractReturningDelegate.java:57)
......
......
It seems that the problem is manifesting after around 5 minutes and 24 seconds or 324 seconds total.
I cannot find anything in MySQL log about it.
It seems it is not memory related, because when I lower the memory the exception is different:
java.lang.OutOfMemoryError: GC overhead limit exceeded
I don't know if that is some kind of bug in Hibernate of MySQL or it is some setting that limits the time that a an update/connection/transaction can run.
I could not find anything related in MySQL documentation or in Hibernate settings.
I would appreciate any help with debugging or resolving this issue.
RESOLVED: Problem was in the file c3p0.properties that was in my tomcat lib.
It contains: c3p0.unreturnedConnectionTimeout=300 which is more or less the time after which I get the error. After I removed it the update passed, no problem.
The database connection pool is configured with a timeout value. Once a transaction has started (borrowed a connection from the pool) and does not finish (does not release the connection to the pool) before this timeout value, the connection is closed by the database connection pool (marked as abandoned). When hibernate tries to commit the transaction (note that hibernate has no way of knowing if the connection has been closed here), the driver thoughs an exception. Try to break down your transaction into further smaller transactions (Commit the items in batches). Once all the orders have been inserted into the DB, insert your Order object. However, your application code should take care of a batch commit failing and the roll-back logic thereafter.
Related
MySQL 5.7, a transaction is running but thread is sleeping, client request(tomcat) is blocking, it will last for many many seconds, after killing connection in MySQL, tomcat receives below exception:
org.springframework.dao.RecoverableDataAccessException:
### Error querying database. Cause: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 852,932 milliseconds ago. The last packet sent successfully to the server was 857,937 milliseconds ago.
at org.springframework.jdbc.support.SQLExceptionSubclassTranslator.doTranslate(SQLExceptionSubclassTranslator.java:98)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:82)
at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:73)
at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:446)
at com.sun.proxy.$Proxy59.selectList(Unknown Source)
at org.mybatis.spring.SqlSessionTemplate.selectList(SqlSessionTemplate.java:230)
......
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:989)
at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:2222)
at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1982)
at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:3407)
at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:470)
at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:3109)
at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2334)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2733)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2549)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1861)
at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1192)
......
Caused by: java.io.EOFException: Can not read response from server. Expected to read 20,481 bytes, read 19,682 bytes before connection was unexpectedly lost.
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3008)
at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:2205)
I use alibaba druid Connection Pool, testOnBorrow=true, mysql java driver version is 5.1.40
The above exception is thrown after connection is killed, tomcat is blocking until the connection is killed.
This case occurs several times in production evn, it is hard to repeat in develop env.
As the caused exception is Can not read response from server. Expected to read 20,481 bytes, read 19,682 bytes before connection was unexpectedly lost., client is waiting for more data from server, so I guess the connection borrowed from pool is valid at first, but why can't reading more data from server?
BTW: we recently use MySQL /*+ MAX_EXECUTION_TIME(xxx) */ optimizer hint. MySQL will throw Exception if query is timeout, I don't know whether it is related with my problem, but I guess it should not.
Presumably your query from your servlet ran for longer than your MAX_EXECUTION_TIME. At any rate, it looks like your query timed out when it had been running for 853 seconds.
And, your servlet crashed. The error trace you provided shows what happened.
If you want your servlet to recover from having its JDBC connections killed, you must program it to catch an exception when you KILL its database process, and then repeat the query.
You'll have to do some experimenting to get this right.
You can force the failure in development by using a much shorter MAX_EXECUTION_TIME. Or you can access your development MySQL server with a command client, look up the processid of the process serving your spring servlet, and KILL it manually.
The root cause: I believe your query takes far too long. You must optimize it somehow. Look at query-performance for ideas about that.
I'm using Liquibase in all my projects, I really love the way it handles db updates, but recently I'm having this issue:
liquibase : Successfully acquired change log lock
liquibase : Successfully released change log lock
liquibase : Could not release lock
liquibase.exception.LockException: liquibase.exception.DatabaseException: liquibase.exception.DatabaseException: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Communications link failure during rollback(). Transaction resolution unknown.
at liquibase.lockservice.StandardLockService.releaseLock(StandardLockService.java:283) ~[liquibase-core-3.5.5.jar!/:na]
at liquibase.Liquibase.update(Liquibase.java:218) [liquibase-core-3.5.5.jar!/:na]
at liquibase.Liquibase.update(Liquibase.java:192) [liquibase-core-3.5.5.jar!/:na]
.
.
.
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Communications link failure during rollback(). Transaction resolution unknown.
.
.
liquibase : Failed to restore the auto commit to true
.
.
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 16,913 milliseconds ago. The last packet sent successfully to the server was 89 milliseconds ago.
.
.
Caused by: java.lang.NullPointerException: null
at com.mysql.jdbc.MysqlIO.clearInputStream(MysqlIO.java:899) ~[mysql-connector-java-5.1.46.jar!/:5.1.46]
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2477) ~[mysql-connector-java-5.1.46.jar!/:5.1.46]
I have 3 apps running on my server. One of them running for about 2years. The second running for 2 months and the third one is new.
They are all Spring Boot applications.
This started happening when I want to update the jar of the second app, I stopped the old jar, run the new one and I can not get it to start with the error above.
Then I run the old jar of the same app and it started just fine. I try stopping and restarting this old jar and a few times I get the error above but most of the times it started just fine.
I try restarting the server (all three apps start as services with system boot) but none of them successfully start because of the same error. Not even the one running for 2 years. I stop all other apps and try again with this long-lasting app, it kept failing until it started.
Could it be a memory issue? I'm using a DigitalOcean droplet with 1 Core and 2gb RAM.
Also, what I've notice is that when it successfully start the logs looks like:
liquibase : Successfully acquired change log lock
liquibase : Reading from myDataBase.DATABASECHANGELOG
liquibase : Successfully released change log lock
Note the statement between the acquisition and release of the lock. I also suspect about a timing issue, between acquiring the lock and reading the changelog. But don't know if I can increase that time or how can I do it.
UPDATE
I don't known the source of the issue, but after updating the liquibase-core to v3.7.0 all the apps started correctly
UPDATE 2
About the previous update, I was testing as is not that the problem is solved, is just less frequent. But it has happen
IMPORTANT UPDATE 3
After some digging I realize that by the same time this happen, the team was playing around with DB Connection Pool. The application.properties files contain this:
spring.datasource.tomcat.initial-size=5
spring.datasource.tomcat.max-wait=20000
spring.datasource.tomcat.max-active=5
spring.datasource.tomcat.max-idle=5
spring.datasource.tomcat.min-idle=5
spring.datasource.tomcat.default-auto-commit=true
Sadly I don't understand much about DB Pool. But what raise my attention was the last line about the auto-commit. The last time this error pops up. I disable all those lines and the app started fine. I don't know if it was just coincidence or not.
Thank you very much!
UPDATE 4
I'm back to square one... I don't know how to solve this any clue would be appreciatted
UPDATE 5
I updated mysql-connector-java to v8.0.15. My server has MySQL v5.6.33. I'm still having this issue and now I can not restart my app.
After the above change, now the exception ends with:
The last packet successfully received from the server was 10,036 milliseconds ago. The last packet sent successfully to the server was 10,036 milliseconds ago.
.
.
.
Caused by: java.io.IOException: Socket is closed
at com.mysql.cj.protocol.AbstractSocketConnection.getMysqlInput(AbstractSocketConnection.java:72) ~[mysql-connector-java-8.0.15.jar!/:8.0.15]
at com.mysql.cj.protocol.a.NativeProtocol.clearInputStream(NativeProtocol.java:833) ~[mysql-connector-java-8.0.15.jar!/:8.0.15]
... 92 common frames omitted
UPDATE 6 - WORKAROUND SOLUTION
In a moment of clarity I decided to disable liquibase from my project, by setting liquibase.enabled=false in the application.properties. Now I can start/stop/restart the app anytime with no problem. As I only use liquibase to mantain the DataBase, and the changes I made are executed (the problem is when releasing the lock), I will enable Liquibase just to update the schema and then disable it to run the app
UPDATE 7 - I think I've solve it
I kept stoping and starting the apps over and over and every time I did it I get the Socket is closed so I began to make my head around a timing issue... From the exceptions thrown I notice that every time it says that:
The last packet successfully received from the server was xxxx milliseconds ago.
The smallest value I get of those milliseconds ago was 10.085 and by chance I have some logs of the same app from time ago when this issue never pop-up. From those logs statements I meassure the time between acquiring the change-log-lock and releasing it. To my surprise it was around 9 seconds and never above that.
So I login to mysql console and issue:
show variables LIKE '%timeout%';
Which gave me a list of the defined timeouts. And there it was connect_timeout was the smallest and set to 10 seconds. It has to be it.
I googled how to set that value. I've try creating a file /etc/my.cnf and adding to it:
[mysqld]
connect_timeout=20
Then restart mysql with:
sudo /etc/init.d/mysql restart
But when I log back to the mysql console the value was still 10. So in the mysql console I issued this other command:
SET GLOBAL connect_timeout=20;
And the value was updated (I don't know if this will persist after a restart)
I restart mysql again and... voilà! Now the apps start normally with liquibase enabled. I'm happy! :-)
I kept stoping and starting the apps over and over and every time I did it I get the Socket is closed so I began to make my head around a timing issue... From the exceptions thrown I notice that every time it says that:
The last packet successfully received from the server was xxxx milliseconds ago.
The smallest value I get of those milliseconds ago was 10.085 and by chance I have some logs of the same app from time ago when this issue never pop-up. From those logs statements I meassure the time between acquiring the change-log-lock and releasing it. To my surprise it was around 9 seconds and never above that.
So I login to mysql console and issue:
show variables LIKE '%timeout%';
Which gave me a list of the defined timeouts. And there it was connect_timeout was the smallest and set to 10 seconds. It has to be it.
I googled how to set that value. I've try creating a file /etc/my.cnf and adding to it:
[mysqld]
connect_timeout=20
Then restart mysql with:
sudo /etc/init.d/mysql restart
But when I log back to the mysql console the value was still 10. So in the mysql console I issued this other command:
SET GLOBAL connect_timeout=20;
And the value was updated (I don't know if this will persist after a full server restart)
I restart mysql again and... voilà! Now the apps start normally with liquibase enabled. I'm happy! :-)
We were working with MySQL 8.0.19. For me, it were only a query which used to fail with the exception Caused by: java.io.IOException: Socket is closed.
Found that for MySQL 8.0.19, there was a bug reported with a similar stack trace of mine - https://bugs.mysql.com/bug.php?id=99234
After upgrading MySQL server to 8.0.21, it started working ! No other changes.
Hope it helps for folks who have spent quite some time with selected query(ies) not working and having the same error, and tried out various other options.
have you solved it yet?
I had a similar problem and solved it by forcing Liquibase to execute in synchronous mode.
Was your spring boot app generated by jhipster by any chance? If so, take a look at DatabaseConfiguration and change from AsyncSpringLiquibase to SpringLiquibase
Here is the log of this exception:
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 1,409,240 milliseconds ago. The last packet sent successfully to the server was 1,409,267 milliseconds ago.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:989)
at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:2229)
at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1989)
at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:3410)
at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:470)
at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:3112)
at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2341)
...
Caused by: java.io.EOFException: Can not read response from server. Expected to read 7 bytes, read 5 bytes before connection was unexpectedly lost.
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3011)
at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:2212)
I know this exception is quite normal, and I've googled it and got a lot of solutions. However, none of those solutions fix my problem. Well, it's just a simple java application, not a java web application, and I didn't use any connecting pool but simply used JDBC. My mysql version is 5.7.12. And the mysql is running on a windows server, while the java application is running on linux. I have checked the 'wait_timeout' for mysql and it's 28800, which is much larger than 1409240 ms. So the problem should probably not be caused by this issue. I've also checked the tcp connection wait time on my linux, it's 7200s, still much bigger than 1409240 ms. I also tried add '?autoReconnect=true' to the JDBC url, but it still made no difference. And I'm sure that there is nothing wrong with the accessibility of my server, cause the connection does work for several minutes before the problem occurs.I've almost tried any thing I can do. However, the problem still persists. What should I do? Is there any posibility that the problem is caused by the windows firewall?
edit:
This problem occurs when executing a SELECT query tries to select all data items from a table whose size is 10.9 G. Maybe this table is just too big so that the sentence ResultSet rs = stmt.executeQuery(sql); cannot be done. But I've checked the 'max_execution_time' variable of mysql, it's set to 0, indicating there is no restriction of execution time.
Welcome to TCP/IP. Lots of things can cause loss of a TCP connection, especially one that has been idle for a while. One such thing, as you mention, is a firewall. The connection can be knocked down by some network entity even if both the client and the server agree it should be kept alive. Connection loss likelihood goes up when client and server are not on the same local network.
Figuring out why means doing lots of packet monitoring at various places in the network. That can use up a lot of time and effort and not teach you much. Plus, learning to read wireshark output is a real task.
Most client-server programmers who need long-lasting connections use some kind of keepalive operation to avoid having the connection sit idle for too long. Keepalive operations send something and get something back. In your case you could do this the easy way by issuing a SELECT NOW() query (or some other round-trip no-op) once every minute or two while your client sits otherwise idle.
The best way to handle this kind of thing is to open the database connection when you need it, then close it when you're done. If you use the connection pooling feature you can open and close your pooled connection upon every query, and still avoid churning the physical connections. The JDBC connector and MySQL server code are optimized for this approach; it is probably the best way to go.
I had com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure reading more on the error message I realized it had to do with my SSL setting in the connection string.
When you have SSL enabled in your connection string for MySQL.Please make sure your date and time are correct or in sync with the current date. If not the connection will be hit with javax.net.ssl.SSLHandshakeException resulting in The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
. prompting you to do something about your date. This is how I solved mine. I had to set my date and time correctly. Sometimes the error messages say a lot about the problem.
Scenario:
We have a scheduler which is using JDBC Job Store. Quartz version is 2.1.2.
The job which is being scheduling is also updating a database.
The database is same for both quartz and the job itself and is hosted in MySQL Server. Both application tables and quartz tables are stored in the same database.
Connection pool is different for both application and quartz. In the application we are using spring for connection pooling and quartz is forced to use connection pooling via quartz.properties.
Here is the snippet of quartz.properties
org.quartz.dataSource.qzDS.driver = com.mysql.jdbc.Driver
org.quartz.dataSource.qzDS.URL = jdbc:mysql://localhost:3306/dbname?autoReconnect=true
org.quartz.dataSource.qzDS.user = dbuser
org.quartz.dataSource.qzDS.password =dbpassword
org.quartz.dataSource.qzDS.maxConnections = 30
org.quartz.datasource.qzDS.validationQuery = select 1
#org.quartz.datasource.qzDS.minEvictableIdleTimeMillis=21600000
#org.quartz.datasource.qzDS.timeBetweenEvictionRunsMillis=1800000
#org.quartz.datasource.qzDS.numTestsPerEviction=-1
#org.quartz.datasource.qzDS.testWhileIdle=true
org.quartz.datasource.qzDS.debugUnreturnedConnectionStackTraces=true
org.quartz.datasource.qzDS.unreturnedConnectionTimeout=120
org.quartz.datasource.qzDS.initialPoolSize=5
org.quartz.datasource.qzDS.minPoolSize=5
org.quartz.datasource.qzDS.maxPoolSize=30
org.quartz.datasource.qzDS.acquireIncrement=5
org.quartz.datasource.qzDS.maxIdleTime=120
org.quartz.datasource.qzDS.validateOnCheckout=true
Database is clustered with MASTER-MASTER replication on two servers and they are being used via virtual IP everywhere in the application and quartz.
Scheduler i.e. quartz is also clustered on the same two machines where MySQL is clustered.
The problem:
One of the servers (till now we have got the problem with backup server machine) is occasionally throwing database connection error while calling notifyJobStoreJobComplete method. This is causing the job to stay in BLOCKED state even if the job itself has successfully completed but quartz was unable to update its status.
Questions:
What can be the cause of the problem?
How to move the BLOCKED jobs into WAITING state so that the jobs can be run on their next scheduled time at least. Direct editing the QRTZ_SIMPLE_TRIGGERS tables would not be a good solution, even if it works.
EDIT: To bump up the question.
the error during notifyJobStoreJobComplete is: org.quartz.impl.jdbcjobstore.JobStoreTX - Failed to override connection auto commit/transaction isolation.
[java] com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 619,082,686 milliseconds ago. The last packet sent successfully to the server was 619,082,686 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
I think main problem was communication link failure by MySQL which we solved it by increasing 'wait_timeout' to 14 days and as our maintenance is scheduled in every 15 days, we restart the each of MySQL server is our DB cluster (We have Master-Master replication in place). With approach we haven't get any communication link failure after that. In fact some time we don't restart the server in every 15 days but still no error(touch wood). :)
And as far as Quartz triggers being locked in BLOCKED state, we updated the quartz to 2.1.4 which possibly has the fix for the almost same problem. After the quartz update, we have faced the triggers being in BLOCKED state very very less frequent.
We are still unable to find out how to get the trigger out of BLOCKED state without directly modifying the quartz tables. Whenever we face this problem, we manually remove the entry for BLOCKED trigger from the qrtz_fired_triggers table and it solves the problem. I think enterprise version of quartz may have this feature from some web UI.
I have hibernate 3.3, c3p0, MySql 5.1, and Spring.
The MySQL connections in my service calls are consistently being closed after ~39 minutes. The natural running time of my service call is on the order of ~5 hours.
I've tried changing various c3p0 config, etc, to avoid the 39 minute cap. No luck.
Is there a more direct, systematic way to log or troubleshoot this? i.e. can I find out why the connection is being closed, and by whom, at which layer?
Update: stack trace
24 Oct 2010 02:22:12,262 [WARN] 012e323c-df4b-11df-89ed-97e9a9c1ac19 (Foobar Endpoint : 3) org.hibernate.util.JDBCExceptionReporter: SQL Error: 0, SQLState: 08003
24 Oct 2010 02:22:12,264 [ERROR] 012e323c-df4b-11df-89ed-97e9a9c1ac19 (Foobar Endpoint : 3) org.hibernate.util.JDBCExceptionReporter: No operations allowed after connection closed.
24 Oct 2010 02:22:12,266 [ERROR] 012e323c-df4b-11df-89ed-97e9a9c1ac19 (Foobar Endpoint : 3) org.hibernate.event.def.AbstractFlushingEventListener: Could not synchronize database state with session
I have hibernate 3.3, c3p0, MySql 5.1, and Spring. The MySQL connections in my service calls are consistently being closed after ~39 minutes. The natural running time of my service call is on the order of ~5 hours.
I'm not sure I understood. Do you have processes that are supposed to run for 5 hours but currently get aborted after ~39mn (or probably 2400 seconds). Can you confirm? What is previously working? Did you change anything?
Meanwhile, here are some ideas:
start with the database (see B.5.2.11. Communication Errors and Aborted Connections)
start the server with the --log-warnings option and check the logs for suspicious messages
see if you can reproduce the problem using a MySQL client from the db host
if it works, do the same thing from the app server machine
it if works, you'll know MySQL is ok
move at the app server level
activate logging (of Hibernate and C3P0) to get a full stack trace and/or more hints about the culprit
also please show your C3P0 configuration settings
And don't forget that C3P0's configuration when using Hibernate is very specific and some settings must go in a c3p0.properties file.
Auto reconnect configuration
http://hibernatedb.blogspot.com/2009/05/automatic-reconnect-from-hibernate-to.html