Couchbase created huge GSI index files - couchbase

Question

We faced an issue with Couchbase, where it generated 150Gb secondary index file for a bucket which has only 26.4GB data.
We got an error in the night:
2018-09-13T22:47:18.552Z+05:30 [Error] IndexerSettingsManager: metakv notifier failed (unexpected EOF)..Restarting
2018/09/13 22:47:18 revrpc: Got error (EOF) and will retry in 1s
panic: 2018-09-13T22:47:24.896Z+05:30 [Error] IndexerSettingsManager: metakv notifier failed (Get http://127.0.0.1:8091/_metakv/?feed=continuous: CBAuth database is stale: last reason: EOF)..Restarting
CBAuth database is stale: last reason: EOF
After that indexer was only publishing these logs:
2018-09-14T03:53:45.284Z+05:30 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8340 Milliseconds Vbucket 984
2018-09-14T03:53:45.314Z+05:30 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8370 Milliseconds Vbucket 984
2018-09-14T03:53:45.344Z+05:30 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8400 Milliseconds Vbucket 984
2018-09-14T03:53:45.374Z+05:30 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8430 Milliseconds Vbucket 984
2018-09-14T03:53:45.404Z+05:30 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8460 Milliseconds Vbucket 984
2018-09-14T03:53:45.434Z+05:30 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8490 Milliseconds Vbucket 984
2018-09-14T03:53:45.464Z+05:30 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8520 Milliseconds Vbucket 984
2018-09-14T03:53:45.494Z+05:30 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8550 Milliseconds Vbucket 984
2018-09-14T03:53:45.524Z+05:30 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8580 Milliseconds Vbucket 984
2018-09-14T03:53:45.554Z+05:30 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8610 Milliseconds Vbucket 984
2018-09-14T03:53:45.584Z+05:30 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8640 Milliseconds Vbucket 984
2018-09-14T03:53:45.614Z+05:30 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8670 Milliseconds Vbucket 984
2018-09-14T03:53:45.644Z+05:30 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8700 Milliseconds Vbucket 984
2018-09-14T03:53:45.674Z+05:30 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8730 Milliseconds Vbucket 984
2018-09-14T03:53:45.7Z+05:30 [Info] logWriterStat:: 5308082947943612037 FlushedCount 33730000 QueuedCount 7840
However it was I guess writing index file also in background and it created following files, which cause 100% disk full.
sudo ls -lrth /storage/1/couchbase/index/#2i/myBucket_idx_5308082947943612037_0.index
total 181G
-rw-rw---- 1 couchbase couchbase 31G Sep 14 03:53 data.fdb.418
-rw-rw---- 1 couchbase couchbase 150G Sep 14 03:59 data.fdb.417
Couchbase version Community: 4.0.0-4051-1
Please let us know, how to debug it further, what can be cause of this?
I see few issues related to this, but not much help from them: MB-19145, MB-20464, MB-18912, MB-14962, MB-25086
Edit:
index on this table:
"indexes": {
"datastore_id": "http://127.0.0.1:8091",
"id": "40d3a82d781dad05",
"index_key": [],
"is_primary": true,
"keyspace_id": "myTable",
"name": "#primary",
"namespace_id": "default",
"state": "online",
"using": "gsi"
},
{
"indexes": {
"datastore_id": "http://127.0.0.1:8091",
"id": "b870f85dc1cadd5",
"index_key": [
"(meta().`id`)"
],
"keyspace_id": "myTable",
"name": "idx",
"namespace_id": "default",
"state": "online",
"using": "gsi"
}