Troubleshoot DCHQ Host - containers

Question

I have been running DCHQ.io (On-Prem) for a few months now with no major issue.
My Container Hosts environment looks like this:
DCHQ-VM-Host
RAM: 14gb / CPU 4 / 100gb Storage
10.21.38.165
host0
This is where DCHQ resides
Docker-VM-1
RAM: 8gb / CPU 2
10.21.36.201
host1
Where LB containers are hosted
Docker-Metal-2
RAM 96gb / CPU 12
10.21.39.71
host2
Where APP containers are hosted
Docker-Metal-4
RAM 96gb / CPU 12
10.21.38.170
host4
Where DB containers are hosted
Today, while attempting to deploy the 3-Tier Java (ApacheHTTP – Tomcat – MySQL) application template for a POC at work, host-4 went offline.
Couple days ago I converted host-1 from a BM machine to a VM. Therefore, I removed that host from DCHQ and added it back using the same name (host-1) but this time as a VM ona different ESX server. Not sure if this has something to do with host-4 throwing the error below. As a result, no template involving host-4 can be deployed so as a workaround I'm using host-1 and host-2 to deploy.
I have tried restarting the host-4, deactivating/activating host-4 from within DCHQ UI and restarting the agent on host-4 but to no avail.
My last resort is to remove and reinstall the client on host-4 but I wanted to post it here first. I have also email DCHQ support with this issue.
The DCHQ log shows the following error :
2016-01-29 15:56:31.026 INFO 1217 --- [pool-3-thread-2] c.d.a.o.impl.ImagePullQueueProcessor : Processing pull req
2016-01-29 15:56:31.028 INFO 1217 --- [pool-3-thread-2] c.d.a.o.impl.TemplateOperationsImpl : Received pull request for image [mysql:latest] registry [null]
2016-01-29 15:56:31.028 INFO 1217 --- [pool-3-thread-2] c.d.a.o.impl.DockerClientBuilderUtil : Using public repo since username [null] or password is empty
2016-01-29 15:56:31.043 ERROR 1217 --- [pool-15-thread-1] c.g.d.core.async.ResultCallbackTemplate : Error during callback
org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:10624 [/127.0.0.1] failed: Connection refused
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:71)
at com.github.dockerjava.jaxrs.connector.ApacheConnector.apply(ApacheConnector.java:443)
at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:246)
at org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:683)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:424)
at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:679)
at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:435)
at org.glassfish.jersey.client.JerseyInvocation$Builder.post(JerseyInvocation.java:338)
at com.github.dockerjava.jaxrs.async.POSTCallbackNotifier.response(POSTCallbackNotifier.java:29)
at com.github.dockerjava.jaxrs.async.AbstractCallbackNotifier.call(AbstractCallbackNotifier.java:45)
at com.github.dockerjava.jaxrs.async.AbstractCallbackNotifier.call(AbstractCallbackNotifier.java:22)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
... 25 common frames omitted
2016-01-29 15:56:31.044 ERROR 1217 --- [pool-3-thread-2] c.d.a.o.impl.TemplateOperationsImpl : Error pulling image [mysql] response logs []
2016-01-29 15:56:31.046 INFO 1217 --- [pool-3-thread-2] c.d.a.o.impl.ImagePullQueueProcessor : Finished processing pull req
2016-01-29 15:58:39.687 WARN 1217 --- [pool-3-thread-1] c.d.a.o.impl.SysInfoMonitorService : org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:10624 [/127.0.0.1] failed: Connection refused
2016-01-29 15:58:39.688 ERROR 1217 --- [pool-3-thread-2] c.d.a.o.impl.MachineOperationsImpl : org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:10624 [/127.0.0.1] failed: Connection refused
Thank you in advance,
Rod

Answer 1

Thanks for reporting this issue. Feel free to report this on our issue tracker.
https://github.com/dchqinc/dchq-on-premise-issue-tracker/issues
Please make sure that the information in the application.properties file has not changed. Make sure that the server key matches whatever the DCHQ UI shows for host-4.
vi /opt/dchq/config/application.properties
You can then restart the agent:
service dchq stop
ps -ef | grep dchq
## forcefully kill any DCHQ process that may not have stopped otherwise using kill -9
service dchq start
Lastly -- can you please test the connection from the UI? This will help us narrow down the issue.
For any help deploying Docker Compose applications, please refer to our documentation: http://dchq.co/docker-compose.html