Hi I am getting the following exception frequently in glassfish can any one help me on this.
The log message is null.
The log message is null.
java.sql.SQLException: Error in allocating a connection. Cause: java.lang.IllegalStateException: Wrong XAState: 0
at com.sun.gjc.spi.base.DataSource.getConnection(DataSource.java:115)
at com.sify.beaserv.bean.PollBoxBean.getNode(PollBoxBean.java:754)
at com.sify.beaserv.bean.PollBoxBean.buildAlarm(PollBoxBean.java:317)
at com.sify.beaserv.bean.PollBoxBean.processNode(PollBoxBean.java:652)
at com.sify.beaserv.bean.PollBoxBean.onMessage(PollBoxBean.java:60)
at sun.reflect.GeneratedMethodAccessor185.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sun.enterprise.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1011)
at com.sun.enterprise.security.SecurityUtil.invoke(SecurityUtil.java:175)
at com.sun.ejb.containers.BaseContainer.invokeTargetBeanMethod(BaseContainer.java:2929)
at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4020)
at com.sun.ejb.containers.MessageBeanContainer.deliverMessage(MessageBeanContainer.java:1111)
at com.sun.ejb.containers.MessageBeanListenerImpl.deliverMessage(MessageBeanListenerImpl.java:74)
at com.sun.enterprise.connectors.inflow.MessageEndpointInvocationHandler.invoke(MessageEndpointInvocationHandler.java:179)
at $Proxy76.onMessage(Unknown Source)
at com.sun.messaging.jms.ra.OnMessageRunner.run(OnMessageRunner.java:258)
at com.sun.enterprise.connectors.work.OneWork.doWork(OneWork.java:76)
at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:555)
The exception message Wrong XAState: 0 suggests that you're trying to use an XA driver, which is correct for two-phase commit. Is that true? Do you know which JDBC driver and which database vendor you're using? Do you get this message right at start up, or is it after you've been running successfully for a while?
Related
Getting org.javalite.activejdbc.DBException frequently while connecting to DB using activejdbc**
Please help to know why this is happening frequently??
exception occurred while getting user :
org.javalite.activejdbc.DBException:
com.mysql.jdbc.CommunicationsException: Communications link failure
due to underlying exception:
** BEGIN NESTED EXCEPTION ** >
java.net.SocketException
MESSAGE: Connection reset by peer: socket write error>
STACKTRACE:>
java.net.SocketException: Connection reset by peer: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(Unknown Source)
at java.net.SocketOutputStream.write(Unknown Source)
at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
at java.io.BufferedOutputStream.flush(Unknown Source)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:2637)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1554)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1665)
at com.mysql.jdbc.Connection.execSQL(Connection.java:3176)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1153)
at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1266)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:353)
at org.javalite.activejdbc.DB.find(DB.java:505)
at org.javalite.activejdbc.LazyList.hydrate(LazyList.java:329)
at org.javalite.activejdbc.AbstractLazyList.isEmpty(AbstractLazyList.java:42)
at org.javalite.activejdbc.ModelDelegate.findFirst(ModelDelegate.java:223)
at com.nb.ats.db.model.UserModel.findFirst(UserModel.java:2358)
** END NESTED EXCEPTION **>
Last packet sent to the server was 0 ms ago.,*
I am using the activejdbc-1.4.14.j7-20180227.053828-4.jar with MYSQLDB
Please help to know why this is happening frequently??
And how to fix this error?
Thanks in advance
"Connection reset by peer.." means your application loses a connection to the database. You may have a network or some other connectivity issue. ActiveJDBC has nothing to so with it,
When I tried to log in AIX server, through a java code/application, I am getting the below exception,
INFO: J2SSH:KEY_EXCHANGE_FAILURE
com.maverick.ssh.SshException: EOF received from remote side [Unknown cause]
at com.maverick.ssh2.TransportProtocol.b(Unknown Source)
at com.maverick.ssh2.TransportProtocol.j(Unknown Source)
at com.maverick.ssh2.TransportProtocol.nextMessage(Unknown Source)
at com.maverick.ssh.components.jce.DiffieHellmanGroup1Sha1.performClientExchange(Unknown Source)
at com.maverick.ssh2.TransportProtocol.e(Unknown Source)
at com.maverick.ssh2.TransportProtocol.processMessage(Unknown Source)
at com.maverick.ssh2.TransportProtocol.startTransportProtocol(Unknown Source)
at com.maverick.ssh2.Ssh2Client.connect(Unknown Source)
at com.maverick.ssh.SshConnector.connect(Unknown Source)
at com.maverick.ssh.SshConnector.connect(Unknown Source)
Server details :
Server type : AIX
Mode of connection : SSH
Authentication : password based authentication. Not key based.
I surfed in internet, but not able get clear answer for this issue. Please some one help me. Thanks in advance.
The java code in question is using the J2SSH Maverick API from SSHTOOLS Limited and is throwing the exception during key exchange. This specific error means the remote server disconnected whilst key exchange was being performed.
There are many potential reasons for this. It's likely that the server did not like something in the key exchange messages and disconnected or even seg-faulted. The resolution would be to ensure your using the latest version of the API above and if it continues to be a problem contact the company for support.
I am getting the following error when running sonar-runner-2.3 (with sonar-3.7). Any help is greatly appreciated. I have double checked the database configuration in both the sonar server and sonar runner and it looks good. Also, searching for similar errors on stackoverflow hasn't helped.
ERROR: Error during Sonar runner execution
org.sonar.runner.impl.RunnerException: Unable to execute Sonar
at org.sonar.runner.impl.BatchLauncher$1.delegateExecution(BatchLauncher.java:91)
at org.sonar.runner.impl.BatchLauncher$1.run(BatchLauncher.java:75)
at java.security.AccessController.doPrivileged(Native Method)
at org.sonar.runner.impl.BatchLauncher.doExecute(BatchLauncher.java:69)
at org.sonar.runner.impl.BatchLauncher.execute(BatchLauncher.java:50)
at org.sonar.runner.api.EmbeddedRunner.doExecute(EmbeddedRunner.java:102)
at org.sonar.runner.api.Runner.execute(Runner.java:90)
at org.sonar.runner.Main.executeTask(Main.java:70)
at org.sonar.runner.Main.execute(Main.java:59)
at org.sonar.runner.Main.main(Main.java:41)
Caused by: org.sonar.core.persistence.BadDatabaseVersion: Unknown database status: FRESH_INSTALL
I have an INNODB engine type table that intermittently throws the exception below.
From what I have read so far, the reason for this exception is one of the following
A full disk
Setting my.cnf innodb_force_recovery property to = 4
Corrupted table
But none of these reasons apply in my case.
15:04:04,391 WARN [org.hibernate.engine.jdbc.spi.SqlExceptionHelper]
(http--10.10.70.31-8080-2) SQL Error: 1030, SQLState: HY000
15:04:04,391 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper]
(http--10.10.70.31-8080-2) Got error -1 from storage engine
15:04:04,410 ERROR
[com.ds.document.service.DocumentGenerationServiceImpl]
(http--10.10.70.31-8080-2) Error when persisting document
[AF-Worksheet] to case_document:
javax.persistence.PersistenceException: o
rg.hibernate.exception.GenericJDBCException: Got error -1 from storage
engine
at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1377)
[hibernate-entitymanager-4.1.7.Final.jar:]
at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1300)
[hibernate-entitymanager-4.1.7.Final.jar:]
at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1306)
[hibernate-entitymanager-4.1.7.Final.jar:]
at org.hibernate.ejb.AbstractEntityManagerImpl.merge(AbstractEntityManagerImpl.java:888)
[hibernate-entitymanager-4.1.7.Final.jar:]
at sun.reflect.GeneratedMethodAccessor595.invoke(Unknown Source) [:1.6.0_14]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[:1.6.0_14]
at java.lang.reflect.Method.invoke(Method.java:597) [:1.6.0_14]
at org.springframework.orm.jpa.SharedEntityManagerCreator$SharedEntityManagerInvocationHandler.invoke(SharedEntityManagerCreator.java:240)
[spring-orm-3.1.1.RELEASE.jar:]
at $Proxy265.merge(Unknown Source) at com.ds.uadj.model.dao.CaseDocumentDAO.addOrUpdateCaseDocument(CaseDocumentDAO.java:266)
[uadj-model-0.0.1-SNAPSHOT.jar:]
at com.ds.uadj.model.service.CaseDocumentationServiceImpl.uploadDocument(CaseDocumentationServiceImpl.java:56)
[uadj-model-0.0.1-SNAPSHOT.jar:]
The intermittent/sporadic behavior baffled me.
Any ideas why this is happening?
Apparently there is a MySQL bug on 5.6.10 version, which I use.
http://bugs.mysql.com/bug.php?id=68722
It seems stable now after the upgrade.
We are using JBoss 5.1 w/ MDB backed by ActiveMQ RAR.
When a message on a queue is consumed and performs some database operations which then result in a deadlock, the deadlock is essentially hosing the entire instance of JBoss until it is restarted. By hosed, any subsequent messages consumed on that queue all fail with the follow exception:
Caused by: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Cannot open connection
The deadlock exception never references my code, which in turn make it very difficult for me to catch and handle.
For example, here is an exception of a deadlock exception:
2012-06-18 18:52:19,848 WARN [JDBCExceptionReporter] : SQL Error: 1213, SQLState: 40001
2012-06-18 18:52:19,848 ERROR [JDBCExceptionReporter] : Deadlock found when trying to get lock; try restarting transaction
2012-06-18 18:52:19,850 ERROR [AbstractFlushingEventListener] : Could not synchronize database state with session
org.hibernate.exception.LockAcquisitionException: Could not execute JDBC batch update
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:105)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:275)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:266)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:168)
at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321)
at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:50)
at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1027)
at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:365)
at org.hibernate.ejb.AbstractEntityManagerImpl$1.beforeCompletion(AbstractEntityManagerImpl.java:504)
at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:101)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:269)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:89)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:177)
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1423)
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:137)
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.endTransaction(MessageInflowLocalProxy.java:435)
at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.finish(MessageInflowLocalProxy.java:314)
at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.after(MessageInflowLocalProxy.java:230)
at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:136)
at $Proxy677.afterDelivery(Unknown Source)
at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:128)
at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:69)
at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:224)
at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:897)
at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:169)
at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:205)
at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:260)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.sql.BatchUpdateException: Deadlock found when trying to get lock; try restarting transaction
at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:2013)
at com.mysql.jdbc.PreparedStatement.executeBatch(PreparedStatement.java:1449)
at com.mysql.jdbc.jdbc2.optional.StatementWrapper.executeBatch(StatementWrapper.java:721)
at org.jboss.resource.adapter.jdbc.WrappedStatement.executeBatch(WrappedStatement.java:774)
at org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:70)
at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:268)
... 29 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:407)
at com.mysql.jdbc.Util.getInstance(Util.java:382)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1064)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3603)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3535)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1989)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2150)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2626)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2119)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2415)
at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1976)
... 34 more
2012-06-18 18:52:19,851 WARN [arjLoggerI18N] : [com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator_2] TwoPhaseCoordinator.beforeCompletion - failed for com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple#480671ab
javax.persistence.PersistenceException: org.hibernate.exception.LockAcquisitionException: Could not execute JDBC batch update
at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:614)
at org.hibernate.ejb.AbstractEntityManagerImpl$1.beforeCompletion(AbstractEntityManagerImpl.java:513)
at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:101)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:269)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:89)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:177)
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1423)
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:137)
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.endTransaction(MessageInflowLocalProxy.java:435)
at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.finish(MessageInflowLocalProxy.java:314)
at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.after(MessageInflowLocalProxy.java:230)
at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:136)
at $Proxy677.afterDelivery(Unknown Source)
at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:128)
at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:69)
at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:224)
at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:897)
at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:169)
at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:205)
at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:260)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: org.hibernate.exception.LockAcquisitionException: Could not execute JDBC batch update
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:105)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:275)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:266)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:168)
at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321)
at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:50)
at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1027)
at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:365)
at org.hibernate.ejb.AbstractEntityManagerImpl$1.beforeCompletion(AbstractEntityManagerImpl.java:504)
... 22 more
Caused by: java.sql.BatchUpdateException: Deadlock found when trying to get lock; try restarting transaction
at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:2013)
at com.mysql.jdbc.PreparedStatement.executeBatch(PreparedStatement.java:1449)
at com.mysql.jdbc.jdbc2.optional.StatementWrapper.executeBatch(StatementWrapper.java:721)
at org.jboss.resource.adapter.jdbc.WrappedStatement.executeBatch(WrappedStatement.java:774)
at org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:70)
at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:268)
... 29 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:407)
at com.mysql.jdbc.Util.getInstance(Util.java:382)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1064)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3603)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3535)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1989)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2150)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2626)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2119)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2415)
at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1976)
... 34 more
2012-06-18 18:52:19,912 WARN [TxConnectionManager] : Connection error occured: org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener#6acc2da9[state=NORMAL mc=org.jboss.resource.adapter.jdbc.xa.XAManagedConnection#2c9e906 handles=0 lastUse=1340059939649 permit=true trackByTx=true mcp=org.jboss.resource.connectionmanager.JBossManagedConnectionPool$OnePool#10015060 context=org.jboss.resource.connectionmanager.InternalManagedConnectionPool#4643d6d5 xaResource=org.jboss.resource.adapter.jdbc.xa.XAManagedConnection#2c9e906 txSync=null]
com.mysql.jdbc.jdbc2.optional.MysqlXAException: XA_RBDEADLOCK: Transaction branch was rolled back: deadlock was detected
at com.mysql.jdbc.jdbc2.optional.MysqlXAConnection.mapXAExceptionFromSQLException(MysqlXAConnection.java:605)
at com.mysql.jdbc.jdbc2.optional.MysqlXAConnection.dispatchCommand(MysqlXAConnection.java:584)
at com.mysql.jdbc.jdbc2.optional.MysqlXAConnection.end(MysqlXAConnection.java:479)
at org.jboss.resource.adapter.jdbc.xa.XAManagedConnection.end(XAManagedConnection.java:246)
at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelAbort(XAResourceRecord.java:396)
at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:3270)
at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:3248)
at com.arjuna.ats.arjuna.coordinator.BasicAction.Abort(BasicAction.java:1933)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:97)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:177)
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1423)
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:137)
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.endTransaction(MessageInflowLocalProxy.java:435)
at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.finish(MessageInflowLocalProxy.java:314)
at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.after(MessageInflowLocalProxy.java:230)
at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:136)
at $Proxy677.afterDelivery(Unknown Source)
at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:128)
at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:69)
at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:224)
at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:897)
at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:169)
at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:205)
at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:260)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
2012-06-18 18:52:19,914 INFO [ServerSessionImpl:153] : Endpoint failed to process message. Reason: Endpoint after delivery notification failure
I can catch the subsequent errors (the errors on subsequent messages to the queue):
Caused by: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Cannot open connection
But I'm not even sure what to do with it, maybe I can get a new EntityManager that isn't hosed, but I'm getting via Injection to begin with... The only way I know to fix this error is to restart.
I assume that the initial Deadlock is happening as part of ending the transaction in the Queue which is why it's not happening in my code, but does any idea of a way I can handle this gracefully?
Update:
All DataSources are MySQL XA
In transaction-jboss-beans.xml we have transactionTimeout set to 300
There is no way to handle container transaction exception gracefully.
It decides to rollback whatever the reason is and you have no control against it. A complex option to get notified about such exceptions and rollback events may be to write a JCA connector which enrolls fake resources into the distributed transaction manager for each MDB transaction.
I would recommend you to investigate what happens and fix the trouble from its source.
Here is a dead-lock scenario, it always involve multiple threads: if two pieces of code use the same entities but in different order, a first thread will lock resource A (a row, a table or a Java monitor) and wait for B when a second thread have already locked resource B and now waits for A.
So a distributed dead-lock is possible between Java monitors and Database resources and that case is complex to diagnose. The full system is stuck until the transaction timeouts at both ends, Java thread dumps and database sessions and locks must be scanned.
In your case, as MySQL detects the dead-lock itself, it means only database resources are involved.
To help, I guess you should reduce the number of SQL queries run in your batch update for one JMS message, one transaction. With concurrent messages consumed at the same time, working on transaction applying large number of rows, a deadlock situation is more likely to occur.
But it is strange that your JBoss server is stuck after rollback because of a DB-only dead-lock:
either you have exhausted the number of JDBC connections from the DataSource pool
or there is no more MySQL server-side connection available
or all message beans are in dead-lock situation too
the transaction timeout may be too high to get rollbacks in a reasonable time frame
So either you reduce MDB concurrency by down sizing the corresponding pool, either you reduce the number of updates per transaction - maybe even as small as one update per JMS message...
I was able to resolve this by isolating each transaction using the following method:
Annotate the MDB with:
#TransactionManagement(value = TransactionManagementType.BEAN)
Thus telling the MDB that the Bean itself will manage the transaction and not the container (outside of my code), next, inject the necessary resources:
#Resource
MessageDrivenContext mc;
UserTransaction tx;
To create and manage my own transactions
tx = mc.getUserTransaction();
tx.begin();
//Do work
tx.commit();
The above step can be repeated for each block of code.
Using very granular transactions I was able to trace where in my code the lock / race condition was happening. Once I solved that I was able to back out some of the more granular transaction management.