Apache Drill: Failure setting up ZK for client - apache-drill

Question

I am testing Apache Drill with a two server cluster.
Let's say their external IPs are:
1.1.1.1
2.2.2.2
I first setup Zookeeper to run on both, and when I do the status command I get positive response:
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper-3.4.8/bin/../conf/zoo.cfg
Mode: leader
The way I have my zoo.cfg to get it working was like this:
Server 1:
// other default values omitted
clientPort=2181
server.1=0.0.0.0:2888:3888
server.2=2.2.2.2:2888:3888
Server 2:
// other default values omitted
clientPort=2181
server.1=1.1.1.1:2888:3888
server.2=0.0.0.0:2888:3888
Next I wanted to get Drill running with this cluster, so I modify the drill-override.conf file for the 2 servers as follows:
Server 1:
drill.exec: {
cluster-id: "test",
zk.connect: "1.1.1.1:2181,2.2.2.2:2181"
}
Server 2:
drill.exec: {
cluster-id: "test",
zk.connect: "2.2.2.2:2181,1.1.1.1:2181"
}
I can start a drillbit on both servers, and when I do status I get this response on both servers:
drillbit is running.
But when I then try to open the console via bin/drill-conf I get this stack trace:
Error: Failure in connecting to Drill: org.apache.drill.exec.rpc.RpcException: Failure setting up ZK for client. (state=,code=0)
java.sql.SQLException: Failure in connecting to Drill: org.apache.drill.exec.rpc.RpcException: Failure setting up ZK for client.
at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:159)
at org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:64)
at org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
at net.hydromatic.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:126)
at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213)
at sqlline.Commands.connect(Commands.java:1083)
at sqlline.Commands.connect(Commands.java:1015)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
at sqlline.SqlLine.dispatch(SqlLine.java:742)
at sqlline.SqlLine.initArgs(SqlLine.java:528)
at sqlline.SqlLine.begin(SqlLine.java:596)
at sqlline.SqlLine.start(SqlLine.java:375)
at sqlline.SqlLine.main(SqlLine.java:268)
Caused by: org.apache.drill.exec.rpc.RpcException: Failure setting up ZK for client.
at org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:208)
at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:151)
... 18 more
Caused by: java.io.IOException: Failure to connect to the zookeeper cluster service within the allotted time of 10000 milliseconds.
at org.apache.drill.exec.coord.zk.ZKClusterCoordinator.start(ZKClusterCoordinator.java:123)
at org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:206)
... 19 more
apache drill 1.7.0
"start your sql engine"
Why would drill fail to connect to the ZK cluster, which is running just fine?
All ports are open between these two boxes.

Answer 1

Pre-Requisites
Prerequisites for starting drill in distributed mode:
(Required) Running Oracle JDK version 7
(Required) Running a ZooKeeper quorum
(Recommended) Running a Hadoop cluster
(Recommended) Using DNS
Configuration
As your server IP address:
Server 1 - 1.1.1.1
Server 2 - 2.2.2.2
Put same configuration in zoo.cfg in both Server 1 and Server 2
clientPort=2181
server.1=1.1.1.1:2888:3888
server.2=2.2.2.2:2888:3888
Similarly same configuration in drill-override.conf for both the servers
drill.exec: {
cluster-id: "test",
zk.connect: "1.1.1.1:2181,2.2.2.2:2181"
}
Starting Drill
Start drillbit on all the cluster nodes using
bin/drillbit.sh start
Using Drill
Web UI:
Open web UI using any node address. For example:
1.1.1.1:8047
Via Shell:
Fire bin/drill-localhost command and drill shell will appear.
Verify Installation
From drill shell or UI fire
SELECT * FROM sys.drillbits;
Drill lists information about the Drillbits that are running
Stopping Drill
Fire command
bin/drillbit.sh stop