Resource monitor using hyperledger fabric caliper - Error: socket hang up - containers

Question

Following Configuration is used to monitor the resource utilization of the docker container which is in a remote machine using a hyperledger caliper (0.4.2).
test:
name: sample fabric network
description: sample Network Benchmark
workers:
type: local
number: 5
rounds:
- label: readAsset
description: Using Chaincode Asset Management contract
txNumber: 10000
rateControl:
type: fixed-load
opts:
tps: 200
workload:
module: workload/readAsset.js
arguments:
contractId: asset-management
monitors:
resource:
- module: docker
options:
interval: 5
containers:
- http://34.233.177.187:7051/peer0.org1.example.com
It gives an error of Error retrieving remote containers: Error: socket hang up as below and does not generate any resource monitoring-related report. Although, the transaction is submitted to a remote network using caliper from where I am trying to monitor the resource utilization of container peer0.org1.example.com. The above-defined Machine IP and the name of the container are defined in the Connection profile.
Log from caliper console
2021.02.17-03:46:22.319 info [caliper] [worker-orchestrator] 5 workers prepared, progressing to test phase.
2021.02.17-03:46:22.945 error [caliper] [monitor-docker] Error retrieving remote containers: Error: socket hang up
2021.02.17-03:46:22.947 info [caliper] [round-orchestrator] Monitors successfully started
2021.02.17-03:46:22.948 info [caliper] [worker-message-handler] Worker#0 is starting Round#0
2021.02.17-03:46:22.952 info [caliper] [worker-message-handler] Worker#1 is starting Round#0
2021.02.17-03:46:22.956 info [caliper] [caliper-worker] Worker #1 starting workload loop
2021.02.17-03:46:22.958 info [caliper] [caliper-worker] Worker #0 starting workload loop
2021.02.17-03:46:22.961 info [caliper] [worker-message-handler] Worker#2 is starting Round#0
2021.02.17-03:46:22.967 info [caliper] [worker-message-handler] Worker#3 is starting Round#0
2021.02.17-03:46:22.970 info [caliper] [caliper-worker] Worker #2 starting workload loop
2021.02.17-03:46:22.973 info [caliper] [worker-message-handler] Worker#4 is starting Round#0
2021.02.17-03:46:22.975 info [caliper] [caliper-worker] Worker #3 starting workload loop
2021.02.17-03:46:22.977 info [caliper] [caliper-worker] Worker #4 starting workload loop
2021.02.17-03:46:27.352 info [caliper] [default-observer] [readAsset Round 0 Transaction Info] - Submitted: 15 Succ: 10 Fail:0 Unfinished:5
2021.02.17-03:46:32.322 info [caliper] [default-observer] [readAsset Round 0 Transaction Info] - Submitted: 45 Succ: 30 Fail:0 Unfinished:15
2021.02.17-03:46:37.322 info [caliper] [default-observer] [readAsset Round 0 Transaction Info] - Submitted: 70 Succ: 60 Fail:0 Unfinished:10
2021.02.17-03:46:42.323 info [caliper] [default-observer] [readAsset Round 0 Transaction Info] - Submitted: 89 Succ: 79 Fail:0 Unfinished:10
2021.02.17-03:46:47.323 info [caliper] [default-observer] [readAsset Round 0 Transaction Info] - Submitted: 111 Succ: 103 Fail:0 Unfinished:8
2021.02.17-03:46:52.323 info [caliper] [default-observer] [readAsset Round 0 Transaction Info] - Submitted: 128 Succ: 120 Fail:0 Unfinished:8
2021.02.17-03:46:57.323 info [caliper] [default-observer] [readAsset Round 0 Transaction Info] - Submitted: 153 Succ: 148 Fail:0 Unfinished:5
2021.02.17-03:47:02.323 info [caliper] [default-observer] [readAsset Round 0 Transaction Info] - Submitted: 173 Succ: 165 Fail:0 Unfinished:8
2021.02.17-03:47:07.324 info [caliper] [default-observer] [readAsset Round 0 Transaction Info] - Submitted: 194 Succ: 189 Fail:0 Unfinished:5
2021.02.17-03:47:12.324 info [caliper] [default-observer] [readAsset Round 0 Transaction Info] - Submitted: 214 Succ: 205 Fail:0 Unfinished:9
2021.02.17-03:47:17.325 info [caliper] [default-observer] [readAsset Round 0 Transaction Info] - Submitted: 233 Succ: 223 Fail:0 Unfinished:10
2021.02.17-03:47:22.325 info [caliper] [default-observer] [readAsset Round 0 Transaction Info] - Submitted: 252 Succ: 239 Fail:0 Unfinished:13
2021.02.17-03:47:27.325 info [caliper] [default-observer] [readAsset Round 0 Transaction Info] - Submitted: 269 Succ: 267 Fail:0 Unfinished:2
20
When monitoring is done using the local machine then it works. for that, I used the below configuration, and its works successfully for local docker container. The configuration is shown below:
monitors:
resource:
- module: docker
options:
interval: 5
containers:
- all
Is there any more configuration to monitor the remote container resource
utilization in caliper?
Thank you!

Answer 1

Caliper uses the dockerode#3.1.0 module to connect to containers(https://www.npmjs.com/package/dockerode) the socket hang up would indicate that a connection has failed to be created due to a timeout or similar.
There are two options available:
Debug the dockerode based connection - perhaps increasing the log level in caliper will help here? It would be relatively easy to craft up a small app using dockerode#3.1.0 to check the ability of it to connect to the passed container URL - if you are able to connect, and the same URL does not work in caliper then there is a bug that needs to be fixed
Swap out to a Prometheus based monitor that uses CAdvisor to monitor container statistics (you would get Grafana for free here too)