Running Apache Drill on OpenShift fails at startup - openshift

Question

does anyway have experience on running apache drill on openshift?
I'm having a problem where I'm trying to run the standard docker-image in openshift
https://hub.docker.com/r/apache/drill
really just the embedded mode without trying to configure anything.
it works fine if I run it in my local docker desktop daemon and it's totally fine.
but when I deploy it to openshift it becomes totally weird by not even starting up.
my deployment yaml is very straight forward:
apiVersion: apps/v1
kind: Deployment
metadata:
name: apache-drill
labels:
app.kubernetes.io/component: apache-drill-svc
spec:
replicas: 1
selector:
matchLabels:
template:
metadata:
labels:
app.kubernetes.io/component: apache-drill-svc
spec:
containers:
- name: apache-drill
image: apache/drill
imagePullPolicy: IfNotPresent
resources:
requests:
cpu: 1
memory: 1Gi
limits:
cpu: 4
memory: 4Gi
But once it launches this is the feedback that it fails to start because of the log directories:
I believe it might be because of this guidelines regarding the security.
https://docs.openshift.com/container-platform/3.11/creating_images/guidelines.html#openshift-specific-guidelines
so because of that I've tried to create my own image wrapping the official one:
https://hub.docker.com/r/cstmgl/adrill/tags (probably there are better ways to do this)
anyway I still get an error even thought the image also works final in my local docker daemon but if I try it on OpenShift I get this error:
apiVersion: apps/v1
kind: Deployment
metadata:
name: apache-drill
labels:
app.kubernetes.io/component: apache-drill-svc
spec:
replicas: 1
template:
metadata:
labels:
app.kubernetes.io/component: apache-drill-svc
spec:
containers:
- name: apache-drill
image: cstmgl/adrill:1.18.0
imagePullPolicy: IfNotPresent
resources:
requests:
cpu: 1
memory: 1Gi
limits:
cpu: 4
memory: 4Gi
Any idea what I'm doing wrong? I'll update this soon with the actual startup error it's rather basic but I believe it's a problem with the user running the application just not sure how to get around it.
ERROR: Drill config file not readable: /opt/drill/conf/drill-override.conf - Wrong user?
Anyway I changed that image also to something like this in my dockerfile
FROM cstmgl/adrill:1.18.0
ARG APP_ROOT=/opt/drill
RUN chmod -R ugo+rw ${APP_ROOT}
RUN echo ${UID}
USER ${UID}
WORKDIR ${DRILL_HOME}
ENTRYPOINT [ "drill-embedded" ]
but still not working, this is the log:
Error: Failure in starting embedded Drillbit: org.apache.drill.exec.exception.DrillbitStartupException: Failed to login. (state=,code=0)
java.sql.SQLException: Failure in starting embedded Drillbit: org.apache.drill.exec.exception.DrillbitStartupException: Failed to login.
at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:143)
at org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:67)
at org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:67)
at org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
at org.apache.drill.jdbc.Driver.connect(Driver.java:75)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:135)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:192)
at sqlline.Commands.connect(Commands.java:1364)
at sqlline.Commands.connect(Commands.java:1244)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
at sqlline.SqlLine.dispatch(SqlLine.java:730)
at sqlline.SqlLine.initArgs(SqlLine.java:410)
at sqlline.SqlLine.begin(SqlLine.java:515)
at sqlline.SqlLine.start(SqlLine.java:267)
at sqlline.SqlLine.main(SqlLine.java:206)
Caused by: org.apache.drill.exec.exception.DrillbitStartupException: Failed to login.
at org.apache.drill.exec.server.BootStrapContext.login(BootStrapContext.java:161)
at org.apache.drill.exec.server.BootStrapContext.<init>(BootStrapContext.java:82)
at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:171)
at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:135)
at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:133)
... 18 more
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: javax.security.auth.login.LoginException: java.lang.NullPointerException: invalid null input: name
at com.sun.security.auth.UnixPrincipal.<init>(UnixPrincipal.java:71)
at com.sun.security.auth.module.UnixLoginModule.login(UnixLoginModule.java:133)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:1926)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1837)
at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:710)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:660)
at org.apache.drill.exec.server.BootStrapContext.login(BootStrapContext.java:156)
at org.apache.drill.exec.server.BootStrapContext.<init>(BootStrapContext.java:82)
at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:171)
at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:135)
at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:133)
at org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:67)
at org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:67)
at org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
at org.apache.drill.jdbc.Driver.connect(Driver.java:75)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:135)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:192)
at sqlline.Commands.connect(Commands.java:1364)
at sqlline.Commands.connect(Commands.java:1244)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
at sqlline.SqlLine.dispatch(SqlLine.java:730)
at sqlline.SqlLine.initArgs(SqlLine.java:410)
at sqlline.SqlLine.begin(SqlLine.java:515)
at sqlline.SqlLine.start(SqlLine.java:267)
at sqlline.SqlLine.main(SqlLine.java:206)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1847)
at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:710)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:660)
at org.apache.drill.exec.server.BootStrapContext.login(BootStrapContext.java:156)
... 22 more
Caused by: javax.security.auth.login.LoginException: java.lang.NullPointerException: invalid null input: name
at com.sun.security.auth.UnixPrincipal.<init>(UnixPrincipal.java:71)
at com.sun.security.auth.module.UnixLoginModule.login(UnixLoginModule.java:133)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:1926)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1837)
at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:710)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:660)
at org.apache.drill.exec.server.BootStrapContext.login(BootStrapContext.java:156)
at org.apache.drill.exec.server.BootStrapContext.<init>(BootStrapContext.java:82)
at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:171)
at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:135)
at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:133)
at org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:67)
at org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:67)
at org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
at org.apache.drill.jdbc.Driver.connect(Driver.java:75)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:135)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:192)
at sqlline.Commands.connect(Commands.java:1364)
at sqlline.Commands.connect(Commands.java:1244)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
at sqlline.SqlLine.dispatch(SqlLine.java:730)
at sqlline.SqlLine.initArgs(SqlLine.java:410)
at sqlline.SqlLine.begin(SqlLine.java:515)
at sqlline.SqlLine.start(SqlLine.java:267)
at sqlline.SqlLine.main(SqlLine.java:206)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:856)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:1926)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1837)
... 25 more
Apache Drill 1.18.0
"Two things are infinite: the universe and Drill; and I'm not sure about the universe."

Answer 1

I have come across this problem myself and after some hard googling, reading this post a few times, and some OpenShift documentation I stumbled upon the way to resolve this.
The problem you are seeing in the logs indicates there is a problem with the name, if you look into sqlline.log you will see that user.name and user.home are empty - it looks something like this:
INFO o.apache.drill.exec.server.Drillbit - Drillbit environment: user.name=?
INFO o.apache.drill.exec.server.Drillbit - Drillbit environment: user.home=?
It seems not everyone knows this but basically OpenShift generates user IDs upon pod creation and the user does not have a home or a name, but in an older version of documentation from OpenShift they provide a way to generate 'default' for the username:
https://docs.openshift.com/container-platform/3.10/creating_images/guidelines.html#openshift-specific-guidelines
Specifically you want this uid_entrypoint script in your image that you can run to generate default for your 100xxxxx user:
https://github.com/RHsyseng/container-rhel-examples/blob/master/starter-arbitrary-uid/bin/uid_entrypoint
You will need to make sure that when building your image that you change the permission of /etc/passswd in order to be able to modify it when you start your image
After I did this I was able to start Drill and you should see the user.name and user.home updated in the log file sqlline.log:
INFO o.apache.drill.exec.server.Drillbit - Drillbit environment: user.name=default
INFO o.apache.drill.exec.server.Drillbit - Drillbit environment: user.home=/

Answer 2

Your application needs root access try this
Either you can add User root in dockerfile just after FROM statement.
or
oc adm policy add-scc-to-user anyuid -z default -n <yournamespace>