When running Apache drill in kubernetes cluster, this exception is throwed a moment after startup:
Starting drillbit, logging to /opt/drill/log/drillbit.out
Exception in thread "main" org.apache.drill.exec.exception.DrillbitStartupException: Could not get canonical hostname.
at org.apache.drill.exec.server.BootStrapContext.getCanonicalHostName(BootStrapContext.java:169)
at org.apache.drill.exec.server.BootStrapContext.<init>(BootStrapContext.java:81)
at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:161)
at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:518)
at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:498)
at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:494)
Caused by: java.net.UnknownHostException: aks-nodepool1-20640609-1: aks-nodepool1-20640609-1: Try again
at java.net.InetAddress.getLocalHost(InetAddress.java:1505)
at org.apache.drill.exec.server.BootStrapContext.getCanonicalHostName(BootStrapContext.java:167)
... 5 more
Caused by: java.net.UnknownHostException: aks-nodepool1-20640609-1: Try again
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getLocalHost(InetAddress.java:1500)
aks-nodepool1-20640609-1 is name of the node, where the container is running.
Every other application than Drill is running ok on this stack.
How can this be fixed ...?
Looks like InetAddress.getLocalHost().getCanonicalHostName() could not get canonical host name, you can investigate why this is happening, maybe you need to configure something in your env. Another option is to use custom host name, it can be set using env variable: DRILL_HOST_NAME -> System.getenv("DRILL_HOST_NAME").
Related
I am following the developer guide to deploy SCDF to a minikube cluster on my local machine. Used the helm chart approach. Was able to get it working with the defaults. The default deploys a mariadb in the cluster. I wanted to change it to use an external mysql db that is running in a docker container in my machine (outside the cluster). Followed the recommendations to change the values.yaml to enable the external DB and attributes for external DB connection (URI, dbname, user/pwd etc.).
Then deployed using "helm install my-release -f values.yaml bitnami/spring-cloud-dataflow"
The SCDF pod (& skipper pod) errors out because it can't find the mysql jdbc driver. kubectl logs on the pods show the following error: "java.lang.IllegalStateException: Cannot load driver class: com.mysql.cj.jdbc.Driver"
How do I include the mysql jdbc driver in the image for SCDF that gets deployed (or resolve this problem). I read SCDF already includes the drivers for std databases (true ?). New to helm/k8s so apologies if solution is obvious... Other posts on similar error all talk about including this in the pom.xml . But this is not a dependency issue with my (task) app but SCDF itself.
thanks
-------------------- more detail on the exception stack ---------------
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSource' defined in class path resource [org/springframework/boot/autoconfigure/jdbc/DataSourceConfiguration$Hikari.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.zaxxer.hikari.HikariDataSource]: Factory method 'dataSource' threw exception; nested exception is java.lang.IllegalStateException: Cannot load driver class: com.mysql.cj.jdbc.Driver
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:657)
...
Looks like I just need to specify the mariadb jdbc driver in the values.yaml file for mysql as well. That got past the load class error.
But still would like to know how to prevent class load error for a different driver specified when SCDF is deployed to k8s via the helm chart.
I'm trying to deploy SCDF 2.6.0 to Openshift.
I can verify DB schema is updated successfully, but seems like the Tomcat failed to start with below error and I have no idea what is going on.
Caused by: java.lang.IllegalArgumentException: standardService.connector.startFailed
Caused by: org.apache.catalina.LifecycleException: Protocol handler start failed
Caused by: java.net.SocketException: Permission denied
Steps reproduce
Use MariaDB, then import the *.yaml in below sequence
server-roles.yaml
server-rolebinding.yaml
service-account.yaml
server-config.yaml (make sure to change the DB connection here )
server-svc.yaml
server-deployment.yaml
I uploaded all the yml and full log file in my repo :
https://github.com/gry77/app-issue-repo/tree/master/Openshift-SCDF-issue/k8s-config
Apparently this error gone after I changed the server port from 80 to something else.
so just change the server.port in the environment to other then 80
OpenShift will not allow you to run containers as privileged by default, so you'll need to specifically allow that using a SecurityContextConstraint. There is a good documentation on how to get SCDF to run on OpenShift here: https://donovanmuller.blog/spring-cloud-dataflow-server-openshift/docs/1.1.0.RELEASE/reference/htmlsingle/#_creating_and_configuring_service_accounts
Basically, you'll need to add the anyuid SCC to the ServiceAccount running the Pods:
oc adm policy add-scc-to-user anyuid system:serviceaccount:scdf:scdf
SnappyData v.0-5
I am logged into Ubuntu as a non-root user, 'foo'.
SnappyData directory/install is owned by 'foo' user and 'foo' group.
I am starting ALL nodes (locator,lead,server) with a script here:
SNAPPY_HOME/sbin/snappy-start-all.sh
Locator starts.
Server starts.
Lead dies with this error.
16/07/21 23:12:26.883 UTC serverConnector INFO JobFileDAO:
rootDir is /tmp/spark-jobserver/filedao/data 16/07/21 23:12:26.888 UTC
serverConnector ERROR JobServer$: Unable to start Spark
JobServer: java.lang.reflect.InvocationTargetException at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at spark.jobserver.JobServer$.start(JobServer.scala:69) at
io.snappydata.impl.LeadImpl.startAddOnServices(LeadImpl.scala:283) at
io.snappydata.impl.LeadImpl$.invokeLeadStartAddonService(LeadImpl.scala:360)
at
io.snappydata.ToolsCallbackImpl$.invokeLeadStartAddonService(ToolsCallbackImpl.scala:28)
at
org.apache.spark.sql.SnappyContext$.invokeServices(SnappyContext.scala:1362)
at
org.apache.spark.sql.SnappyContext$.initGlobalSnappyContext(SnappyContext.scala:1340)
at org.apache.spark.sql.SnappyContext.(SnappyContext.scala:104)
at org.apache.spark.sql.SnappyContext.(SnappyContext.scala:95)
at
org.apache.spark.sql.SnappyContext$.newSnappyContext(SnappyContext.scala:1221)
at
org.apache.spark.sql.SnappyContext$.apply(SnappyContext.scala:1249)
at
org.apache.spark.scheduler.SnappyTaskSchedulerImpl.postStartHook(SnappyTaskSchedulerImpl.scala:25)
at org.apache.spark.SparkContext.(SparkContext.scala:601) at
io.snappydata.impl.LeadImpl.start(LeadImpl.scala:129) at
io.snappydata.impl.ServerImpl.start(ServerImpl.scala:32) at
io.snappydata.tools.LeaderLauncher.startServerVM(LeaderLauncher.scala:91)
at
com.pivotal.gemfirexd.tools.internal.GfxdServerLauncher.connect(GfxdServerLauncher.java:174)
at
com.gemstone.gemfire.internal.cache.CacheServerLauncher$AsyncServerLauncher.run(CacheServerLauncher.java:1003)
at java.lang.Thread.run(Thread.java:745) Caused by:
java.io.FileNotFoundException:
/tmp/spark-jobserver/filedao/data/jars.data (Permission denied) at
java.io.FileOutputStream.open0(Native Method) at
java.io.FileOutputStream.open(FileOutputStream.java:270) at
java.io.FileOutputStream.(FileOutputStream.java:213) at
spark.jobserver.io.JobFileDAO.init(JobFileDAO.scala:90) at
spark.jobserver.io.JobFileDAO.(JobFileDAO.scala:30) ... 22 more
16/07/21 23:12:26.891 UTC Distributed system shutdown hook
INFO snappystore: VM is exiting - shutting down distributed system
Do I need to be a different user to start the Lead node? Use 'sudo'? Configure a property to tell Spark to use a directory 'foo' has permission to? Create this directory myself ahead of time?
It seems that the current owner of /tmp/spark-jobserver is some other user. Check the permissions on that directory and delete it.
If multiple users will be running leads on the same machine, you can configure the job-server directories to be elsewhere like mentioned here. The relevant properties can be found in application.conf source. This is probably more trouble than worth, so for now it will be easier to just ensure a single user starts the lead nodes on a machine.
We shall be fixing the default to be inside work/ directory in next release (SNAP-69).
hibernate 3.6.8 final
c3p0 jar that came with hibernate 3.6.8 package -> c3p0-0.9.1.jar
1
15
40
0
5
2
The app seems to be working fine, however I get massive log calls with the following stacktrace:
org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already. Could not load com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1566)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at com.mysql.jdbc.Util.getInstance(Util.java:386)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1013)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:982)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2411)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2153)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:792)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
at sun.reflect.GeneratedConstructorAccessor38.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:381)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305)
at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:134)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:152)
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1074)
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1061)
at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1796)
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:620)
Any information on how to remove that INFO log would be very much helpful thanks!
UPDATE: Is this a critical error? Or can should I just ignore it?
After a search on the web regarding this issue, I found some similar issues reported by several people. All of them point to a common problem: Threads. Basically, if you start new threads in your application (either in your code or by using a third party tool like Quartz, you have to make sure that all of the threads are stopped appropriately when the application is undeployed from the server. Here are some quotes from the searches:
Mikolaj Rydzewski wrote:
It looks like after webapp's instance has been undeployed, background quartz thread wants to do something and then exception occurs.
Another (and better explanation) on jspwiki.org:
It is possible that this is caused by Tomcat unsuccessfully reloading the web application. The app is unloaded, but all threads don't get shut down properly. As a result, when the threads try to run, they get clobbered by the fact that Tomcat has shut down its classloader, and an error is logged.
So, in order to solve this issue you have to make sure all threads started by your application will be stopped at application undeployment (or redeployment, it's the same). You can do this by registering a ServletContextListener to your application server and stopping your threads inside contextDestroyed(ServletContextEvent) method.
If you are using log4j, change your logging settings to something like this (the word ERROR replace INFO) :
log4j.rootLogger=ERROR, file, stdout
log4j.logger.org.hibernate=ERROR
They are located in the the log4j.properties file in yr project.
OK I switched to boneCP, c3p0 does not really seem to work for java6!!!
I have created a play application. When i am running the application i'm getting this error both in dev and prod
07:59:58,445 INFO ~ Precompiling ...
08:00:08,884 ERROR ~ Unable to set localhost. This prevents creation of a GUID. Cause was: sys3: sys3: Name or service not known
java.net.UnknownHostException: sys3: sys3: Name or service not known
at java.net.InetAddress.getLocalHost(InetAddress.java:1438)
at net.sf.ehcache.Cache.<clinit>(Cache.java:155)
at net.sf.ehcache.config.ConfigurationHelper.createCache(ConfigurationHelper.java:298)
at net.sf.ehcache.config.ConfigurationHelper.createDefaultCache(ConfigurationHelper.java:214)
at net.sf.ehcache.CacheManager.configure(CacheManager.java:552)
at net.sf.ehcache.CacheManager.init(CacheManager.java:323)
at net.sf.ehcache.CacheManager.<init>(CacheManager.java:279)
at net.sf.ehcache.CacheManager.create(CacheManager.java:641)
at play.cache.EhCacheImpl.<init>(EhCacheImpl.java:31)
at play.cache.EhCacheImpl.newInstance(EhCacheImpl.java:41)
at play.cache.Cache.init(Cache.java:241)
at play.Play.start(Play.java:511)
at play.Play.init(Play.java:300)
at play.server.Server.main(Server.java:158)
Caused by: java.net.UnknownHostException: sys3: Name or service not known
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:866)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1258)
at java.net.InetAddress.getLocalHost(InetAddress.java:1434)
... 13 more
I don't know what is this, what this will do. How to solve this issue. Pl help me.
java is unable to resolve mynap3 as localhost, so you'll have to add the line
127.0.0.1 localhost mynap3
to your /etc/hosts file
you can do this by executing
sudo bash
echo "127.0.0.1 localhost mynap3" >> /etc/hosts