RedisConnectionFailureException cleanupExpiredSessions
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out
Caused by: java.net.SocketTimeoutException: Read timed out
Related
It happened occasionly and there was no something that mad sense.
This was happend during the process of Generation to Accesstoken which fetch user info from Mysql
After this [Communications link failure] the Accesstoken was generated but lack of the user infos which lead to continous problems.
2023-01-30 23:28:41,460 WARN [org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory] (ForkJoinPool.commonPool-worker-15) IJ030027: Destroying connection that is not valid, due to the following exception: com.mysql.cj.jdbc.ConnectionImpl#4d85783d: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
The last packet successfully received from the server was 1,653,313 milliseconds ago. The last packet sent successfully to the server was 1,653,315 milliseconds ago.
Caused by: com.mysql.cj.exceptions.CJCommunicationsException:
2023-01-30 23:28:41,461 WARN [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (ForkJoinPool.commonPool-worker-15) IJ000621: Destroying connection that could not be validated: org.jboss.jca.core.connectionmanager.listener.TxConnectionListener#7277f22f[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection#27f95c6 connection handles=0 lastReturned=1675090868148 lastValidated=1675090868145 lastCheckedOut=1675090868145 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool#203b3fab mcp=SemaphoreConcurrentLinkedQueueManagedConnectionPool#56d2adbb[pool=KeycloakDS] xaResource=LocalXAResourceImpl#64dd4124[connectionListener=7277f22f connectionManager=6418d3f warned=false currentXid=null productName=MySQL productVersion=5.7.35-log jndiName=java:jboss/datasources/KeycloakDS] txSync=null]
2023-01-30 23:28:41,467 WARN [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (default task-175) SQL Error: 0, SQLState: null
2023-01-30 23:28:41,467 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (default task-175) IJ031050: The result set is closed
2023-01-30 23:28:41,467 ERROR [org.keycloak.services.error.KeycloakErrorHandler] (default task-175) Uncaught server error: com.fasterxml.jackson.databind.JsonMappingException: could not initialize a collection: [org.keycloak.models.jpa.entities.UserEntity.attributes#39235fc0-85dc-4270-bd76-cad600d2e3e2]
Caused by: org.hibernate.exception.GenericJDBCException: could not initialize a collection: [org.keycloak.models.jpa.entities.UserEntity.attributes#39235fc0-85dc-4270-bd76-cad600d2e3e2]
Caused by: java.sql.SQLException: IJ031050: The result set is closed
Caused by: org.keycloak.models.ModelException: org.keycloak.models.ModelException: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not read entity state from ResultSet : EntityKey[org.keycloak.models.jpa.entities.UserEntity#39235fc0-85dc-4270-bd76-cad600d2e3e2]
Caused by: org.keycloak.models.ModelException: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not read entity state from ResultSet : EntityKey[org.keycloak.models.jpa.entities.UserEntity#39235fc0-85dc-4270-bd76-cad600d2e3e2]
Caused by: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not read entity state from ResultSet : EntityKey[org.keycloak.models.jpa.entities.UserEntity#39235fc0-85dc-4270-bd76-cad600d2e3e2]
Caused by: org.hibernate.exception.GenericJDBCException: Could not read entity state from ResultSet : EntityKey[org.keycloak.models.jpa.entities.UserEntity#39235fc0-85dc-4270-bd76-cad600d2e3e2]
Caused by: java.sql.SQLException: IJ031050: The result set is closed
Versions Info:
Java: openjdk version "11.0.12"
keycloak: 14.0.0
MySql: 5.7
Recently we upgraded our spring boot application from 2.3.4 to 2.6.7
Our spring boot application uses Aurora DB instance with Maria DB driver.
According to latest AWS documentation,https://aws.amazon.com/blogs/database/using-the-mariadb-jdbc-driver-with-amazon-aurora-with-mysql-compatibility/, Maria DB driver is no longer compatible with Aurora DB.
Hence we decided to switch it to AWS Mysql driver with Hikari connection pool.
After the change, long running queries are timing out resulting in socket timeout exception.
We tried with the jdbc url parameter connectionTimeout=0 and socketTimeout=0 but still we are getting socket timeout exception.
Any suggestions here?
Exception stack trace:
org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.hibernate.exception.JDBCConnectionException: could not extract ResultSet
at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:112)
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:37)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:67)
at org.hibernate.loader.Loader.getResultSet(Loader.java:2322)
at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2075)
at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2037)
at org.hibernate.loader.Loader.doQuery(Loader.java:956)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:357)
at org.hibernate.loader.Loader.doList(Loader.java:2868)
at org.hibernate.loader.Loader.doList(Loader.java:2850)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2682)
at org.hibernate.loader.Loader.list(Loader.java:2677)
at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:540)
at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:400)
at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:219)
at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1459)
at org.hibernate.query.internal.AbstractProducedQuery.doList(AbstractProducedQuery.java:1649)
at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1617)
at org.hibernate.query.Query.getResultList(Query.java:165)
at sun.reflect.GeneratedMethodAccessor346.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.orm.jpa.SharedEntityManagerCreator$DeferredQueryInvocationHandler.invoke(SharedEntityManagerCreator.java:406)
at com.sun.proxy.$Proxy439.getResultList(Unknown Source)
at com.querydsl.jpa.impl.AbstractJPAQuery.getResultList(AbstractJPAQuery.java:191)
at com.querydsl.jpa.impl.AbstractJPAQuery.fetch(AbstractJPAQuery.java:243)
at com.scacap.baader.recon.api.recon.baader.RkkReconNoFilterRepositoryImpl.findAllUnknownKeywords(RkkReconNoFilterRepositoryImpl.java:43)
at com.scacap.baader.recon.api.recon.baader.RkkReconNoFilterRepositoryImpl$$FastClassBySpringCGLIB$$eafd8e8e.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137)
... 152 common frames omitted
Caused by: software.aws.rds.jdbc.mysql.shading.com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
The last packet successfully received from the server was 48,065 milliseconds ago. The last packet sent successfully to the server was 48,066 milliseconds ago.
at software.aws.rds.jdbc.mysql.shading.com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(SQLError.java:174)
at software.aws.rds.jdbc.mysql.shading.com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:64)
at software.aws.rds.jdbc.mysql.shading.com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:953)
at software.aws.rds.jdbc.mysql.shading.com.mysql.cj.jdbc.ClientPreparedStatement.executeQuery(ClientPreparedStatement.java:1009)
at sun.reflect.GeneratedMethodAccessor337.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at software.aws.rds.jdbc.mysql.shading.com.mysql.cj.jdbc.ha.ConnectionProxy$JdbcInterfaceProxy.lambda$invoke$0(ConnectionProxy.java:322)
at software.aws.rds.jdbc.mysql.shading.com.mysql.cj.jdbc.ha.plugins.DefaultConnectionPlugin.execute(DefaultConnectionPlugin.java:75)
at software.aws.rds.jdbc.mysql.shading.com.mysql.cj.jdbc.ha.plugins.NodeMonitoringConnectionPlugin.execute(NodeMonitoringConnectionPlugin.java:173)
at software.aws.rds.jdbc.mysql.shading.com.mysql.cj.jdbc.ha.plugins.failover.FailoverConnectionPlugin.execute(FailoverConnectionPlugin.java:230)
at software.aws.rds.jdbc.mysql.shading.com.mysql.cj.jdbc.ha.plugins.ConnectionPluginManager.execute(ConnectionPluginManager.java:133)
at software.aws.rds.jdbc.mysql.shading.com.mysql.cj.jdbc.ha.ConnectionProxy$JdbcInterfaceProxy.invoke(ConnectionProxy.java:319)
at com.sun.proxy.$Proxy437.executeQuery(Unknown Source)
at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52)
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:57)
... 182 common frames omitted
Caused by: software.aws.rds.jdbc.mysql.shading.com.mysql.cj.exceptions.CJCommunicationsException: Communications link failure
The last packet successfully received from the server was 48,065 milliseconds ago. The last packet sent successfully to the server was 48,066 milliseconds ago.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at software.aws.rds.jdbc.mysql.shading.com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61)
at software.aws.rds.jdbc.mysql.shading.com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:105)
at software.aws.rds.jdbc.mysql.shading.com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:151)
at software.aws.rds.jdbc.mysql.shading.com.mysql.cj.exceptions.ExceptionFactory.createCommunicationsException(ExceptionFactory.java:167)
at software.aws.rds.jdbc.mysql.shading.com.mysql.cj.protocol.a.NativeProtocol.readMessage(NativeProtocol.java:520)
at software.aws.rds.jdbc.mysql.shading.com.mysql.cj.protocol.a.NativeProtocol.checkErrorMessage(NativeProtocol.java:700)
at software.aws.rds.jdbc.mysql.shading.com.mysql.cj.protocol.a.NativeProtocol.sendCommand(NativeProtocol.java:639)
at software.aws.rds.jdbc.mysql.shading.com.mysql.cj.protocol.a.NativeProtocol.sendQueryPacket(NativeProtocol.java:987)
at software.aws.rds.jdbc.mysql.shading.com.mysql.cj.NativeSession.execSQL(NativeSession.java:666)
at software.aws.rds.jdbc.mysql.shading.com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:930)
... 196 common frames omitted
Caused by: javax.net.ssl.SSLException: Socket closed
at sun.security.ssl.Alert.createSSLException(Alert.java:127)
at sun.security.ssl.TransportContext.fatal(TransportContext.java:348)
at sun.security.ssl.TransportContext.fatal(TransportContext.java:291)
at sun.security.ssl.TransportContext.fatal(TransportContext.java:286)
at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1594)
at sun.security.ssl.SSLSocketImpl.access$400(SSLSocketImpl.java:73)
at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:994)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at software.aws.rds.jdbc.mysql.shading.com.mysql.cj.protocol.FullReadInputStream.readFully(FullReadInputStream.java:64)
at software.aws.rds.jdbc.mysql.shading.com.mysql.cj.protocol.a.SimplePacketReader.readHeaderLocal(SimplePacketReader.java:81)
at software.aws.rds.jdbc.mysql.shading.com.mysql.cj.protocol.a.SimplePacketReader.readHeader(SimplePacketReader.java:63)
at software.aws.rds.jdbc.mysql.shading.com.mysql.cj.protocol.a.SimplePacketReader.readHeader(SimplePacketReader.java:45)
at software.aws.rds.jdbc.mysql.shading.com.mysql.cj.protocol.a.TimeTrackingPacketReader.readHeader(TimeTrackingPacketReader.java:52)
at software.aws.rds.jdbc.mysql.shading.com.mysql.cj.protocol.a.TimeTrackingPacketReader.readHeader(TimeTrackingPacketReader.java:41)
at software.aws.rds.jdbc.mysql.shading.com.mysql.cj.protocol.a.MultiPacketReader.readHeader(MultiPacketReader.java:54)
at software.aws.rds.jdbc.mysql.shading.com.mysql.cj.protocol.a.MultiPacketReader.readHeader(MultiPacketReader.java:44)
at software.aws.rds.jdbc.mysql.shading.com.mysql.cj.protocol.a.NativeProtocol.readMessage(NativeProtocol.java:514)
... 201 common frames omitted
Caused by: java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.jav
I tried to deploy a Java and Maven function app to Microsoft Azure. The deployment goes well but when I try to run the program it gets all these errors. I start getting all these communication errors. Everything works locally but when published it doesn't. I don't know what info I should give here. But if someone has got the same problem it would be appreciated if someone could help. I can provide more info if asked but don't know what I should put here.
> Result: Failure
Exception: KeyException: Key does not exist.
Stack: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.microsoft.azure.functions.worker.broker.JavaMethodInvokeInfo.invoke(JavaMethodInvokeInfo.java:22)
at com.microsoft.azure.functions.worker.broker.JavaMethodExecutor.execute(JavaMethodExecutor.java:54)
at com.microsoft.azure.functions.worker.broker.JavaFunctionBroker.invokeMethod(JavaFunctionBroker.java:51)
at com.microsoft.azure.functions.worker.handler.InvocationRequestHandler.execute(InvocationRequestHandler.java:33)
at com.microsoft.azure.functions.worker.handler.InvocationRequestHandler.execute(InvocationRequestHandler.java:10)
at com.microsoft.azure.functions.worker.handler.MessageHandler.handle(MessageHandler.java:45)
at com.microsoft.azure.functions.worker.JavaWorkerClient$StreamingMessagePeer.lambda$onNext$0(JavaWorkerClient.java:92)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.hibernate.exception.GenericJDBCException: Unable to acquire JDBC Connection
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:47)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:97)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:109)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:136)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getConnectionForTransactionManagement(LogicalConnectionManagedImpl.java:254)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:262)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:214)
at org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:56)
at org.hibernate.internal.AbstractSharedSessionContract.beginTransaction(AbstractSharedSessionContract.java:409)
at io.madtrix.producerutil.dao.TokenRepository.getMadtrixTokenForManagerAccountAuthenticatedSourceWithClientId(TokenRepository.java:139)
at io.madtrix.MessageHandler.<init>(MessageHandler.java:38)
at io.madtrix.TwitterProducer.run(TwitterProducer.java:31)
... 16 more
Caused by: java.sql.SQLException: Connections could not be acquired from the underlying database!
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:118)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:692)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:140)
at org.hibernate.c3p0.internal.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:72)
at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:35)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:106)
... 25 more
Caused by: com.mchange.v2.resourcepool.CannotAcquireResourceException: A ResourcePool could not acquire a resource from its primary factory or source.
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1469)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:644)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:554)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutAndMarkConnectionInUse(C3P0PooledConnectionPool.java:758)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:685)
... 29 more
Caused by: java.sql.SQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:108)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:95)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:87)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:61)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:71)
at com.mysql.cj.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:932)
at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:857)
at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:444)
at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:230)
at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:226)
at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138)
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125)
at com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44)
at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870)
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)
Caused by: com.mysql.cj.exceptions.CJCommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.GeneratedConstructorAccessor36.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:59)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:103)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:149)
at com.mysql.cj.exceptions.ExceptionFactory.createCommunicationsException(ExceptionFactory.java:165)
at com.mysql.cj.protocol.a.NativeProtocol.negotiateSSLConnection(NativeProtocol.java:355)
at com.mysql.cj.protocol.a.NativeAuthenticationProvider.negotiateSSLConnection(NativeAuthenticationProvider.java:789)
at com.mysql.cj.protocol.a.NativeAuthenticationProvider.proceedHandshakeWithPluggableAuthentication(NativeAuthenticationProvider.java:499)
at com.mysql.cj.protocol.a.NativeAuthenticationProvider.connect(NativeAuthenticationProvider.java:217)
at com.mysql.cj.protocol.a.NativeProtocol.connect(NativeProtocol.java:1411)
at com.mysql.cj.NativeSession.connect(NativeSession.java:165)
at com.mysql.cj.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:876)
... 13 more
Caused by: javax.net.ssl.SSLException: java.security.ProviderException: java.security.KeyException: Key does not exist.
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1964)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1921)
at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1904)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1420)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
at com.mysql.cj.protocol.ExportControlled.performTlsHandshake(ExportControlled.java:213)
at com.mysql.cj.protocol.StandardSocketFactory.performTlsHandshake(StandardSocketFactory.java:206)
at com.mysql.cj.protocol.a.NativeSocketConnection.performTlsHandshake(NativeSocketConnection.java:99)
at com.mysql.cj.protocol.a.NativeProtocol.negotiateSSLConnection(NativeProtocol.java:350)
... 19 more
Caused by: java.security.ProviderException: java.security.KeyException: Key does not exist.
at sun.security.mscapi.RSACipher.doFinal(RSACipher.java:310)
at sun.security.mscapi.RSACipher.engineDoFinal(RSACipher.java:334)
at javax.crypto.Cipher.doFinal(Cipher.java:2164)
at java.security.Signature$CipherAdapter.engineVerify(Signature.java:1319)
at java.security.SignatureSpi.engineVerify(SignatureSpi.java:278)
at java.security.Signature$Delegate.engineVerify(Signature.java:1228)
at java.security.Signature.verify(Signature.java:704)
at sun.security.ssl.RSASignature.engineVerify(RSASignature.java:182)
at sun.security.ssl.RSASignature.engineVerify(RSASignature.java:175)
at java.security.Signature$Delegate.engineVerify(Signature.java:1222)
at java.security.Signature.verify(Signature.java:655)
at sun.security.ssl.HandshakeMessage$ECDH_ServerKeyExchange.<init>(HandshakeMessage.java:1118)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:284)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
at sun.security.ssl.Handshaker.process_record(Handshaker.java:987)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
... 24 more
Caused by: java.security.KeyException: Key does not exist.
at sun.security.mscapi.RSACipher.encryptDecrypt(Native Method)
at sun.security.mscapi.RSACipher.doFinal(RSACipher.java:294)
... 41 more
The answer was in the connection to MySQL. It used SSL=true but when changed to SSL=false it started working normally.
Here is the hive metastore error log, I have tried autoReconnect=true in JDBC url, it takes no effect. Do you have any idea about it?
17/08/20 00:24:06 [pool-3-thread-9487]: ERROR metastore.RetryingHMSHandler: Retrying HMSHandler after 5000 ms (attempt 2 of 10) with error: javax.jdo.JDODataStoreException: Could not retrieve transation read-only status server
at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
at org.datanucleus.api.jdo.JDOPersistenceManager.getDataStoreConnection(JDOPersistenceManager.java:2259)
at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.trySetAnsiQuotesForMysql(MetaStoreDirectSql.java:166)
at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.<init>(MetaStoreDirectSql.java:107)
at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:252)
at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:223)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:58)
at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:497)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:475)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_tables(HiveMetaStore.java:2835)
at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
at com.sun.proxy.$Proxy3.get_tables(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_tables.getResult(ThriftHiveMetastore.java:8102)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_tables.getResult(ThriftHiveMetastore.java:8086)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.hadoop.hive.metastore.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:48)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
NestedThrowablesStackTrace:
java.sql.SQLException: Could not retrieve transation read-only status server
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:957)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:896)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:885)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:860)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:877)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:873)
at com.mysql.jdbc.ConnectionImpl.isReadOnly(ConnectionImpl.java:3629)
at com.mysql.jdbc.ConnectionImpl.isReadOnly(ConnectionImpl.java:3598)
at com.jolbox.bonecp.ConnectionHandle.isReadOnly(ConnectionHandle.java:867)
at org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getConnection(ConnectionFactoryImpl.java:422)
at org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getXAResource(ConnectionFactoryImpl.java:378)
at org.datanucleus.store.connection.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:328)
at org.datanucleus.store.connection.AbstractConnectionFactory.getConnection(AbstractConnectionFactory.java:94)
at org.datanucleus.store.rdbms.RDBMSStoreManager.getNucleusConnection(RDBMSStoreManager.java:1357)
at org.datanucleus.api.jdo.JDOPersistenceManager.getDataStoreConnection(JDOPersistenceManager.java:2245)
at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.trySetAnsiQuotesForMysql(MetaStoreDirectSql.java:166)
at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.<init>(MetaStoreDirectSql.java:107)
at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:252)
at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:223)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:58)
at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:497)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:475)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_tables(HiveMetaStore.java:2835)
at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
at com.sun.proxy.$Proxy3.get_tables(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_tables.getResult(ThriftHiveMetastore.java:8102)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_tables.getResult(ThriftHiveMetastore.java:8086)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.hadoop.hive.metastore.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:48)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 1,412,672 milliseconds ago. The last packet sent successfully to the server was 1,412,672 milliseconds ago. is longer than the server configured value of 'interactive_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
at sun.reflect.GeneratedConstructorAccessor61.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:981)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3652)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2460)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2625)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2547)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2505)
at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1370)
at com.mysql.jdbc.ConnectionImpl.isReadOnly(ConnectionImpl.java:3623)
... 33 more
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3634)
... 39 more
I have installed Hadoop 2.7.1 and Hive 1.2.1 using instructions from tutorialspoint.
http://www.tutorialspoint.com/hive/hive_installation.htm
While starting hive, I'm getting error:
Logging initialized using configuration in jar:file:/usr/local/hive/lib/hive-common-1.2.1.jar!/hive-log4j.properties
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503)
... 8 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)
... 14 more
Caused by: javax.jdo.JDOFatalDataStoreException: Unable to open a test connection to the given database. JDBC url = jdbc:derby://localhost:1527/metastore_db;create=true, username = APP. Terminating connection pool (set lazyInit to true if you expect to start your database after your app). Original Exception: ------
java.sql.SQLException: No suitable driver found for jdbc:derby://localhost:1527/metastore_db;create=true
.....
------
NestedThrowables:
java.sql.SQLException: Unable to open a test connection to the given database. JDBC url = jdbc:derby://localhost:1527/metastore_db;create=true, username = APP. Terminating connection pool (set lazyInit to true if you expect to start your database after your app). Original Exception: ------
java.sql.SQLException: No suitable driver found for jdbc:derby://localhost:1527/metastore_db;create=true
.....
------
Caused by: java.sql.SQLException: Unable to open a test connection to the given database. JDBC url = jdbc:derby://localhost:1527/metastore_db;create=true, username = APP. Terminating connection pool (set lazyInit to true if you expect to start your database after your app). Original Exception: ------
java.sql.SQLException: No suitable driver found for jdbc:derby://localhost:1527/metastore_db;create=true
------
Caused by: java.sql.SQLException: No suitable driver found for jdbc:derby://localhost:1527/metastore_db;create=true
.....