Custom Cluster Tests Exit Abnormally and Leave Dangling Containers

Moved from GitHub dgraph/5897

Posted by darkn3rd:

When running custom cluster tests, occasionally the script exits abnormally and doesn’t clean up.

What version of Dgraph are you using?


Have you tried reproducing the issue with the latest release?


What is the hardware spec (RAM, OS)?

Ubuntu Bionic Beaver

Steps to reproduce the issue (command/config used to run Dgraph).

cd $GOPATH/src/
git clone
cd dgraph
git checkout release/v20.03
export GO111MODULE=on
./ -C 2>&1 | tee /tmp/v20.03.custom_cluster.$(date +%Y%m%d_%H%M).txt

Expected behaviour and actual result.


The expected behaviour would be a message about test passing or failing and no left over containers running.

INFO: Running tests in directory graphql/e2e/schema_subscribe
ok	0.327s
INFO: Stopping cluster
Stopped alpha3
Stopped alpha2
Stopped zero1
Stopped alpha1
Removed alpha3
Removed alpha2
Removed zero1
Removed alpha1
INFO: Tests completed in 10m 28s
INFO: All tests passed!


It looks like the script or subshell is exited abnormally and left containers behind.

INFO: Running tests in directory graphql/e2e/schema_subscribe
Rebuilding dgraph ...
Commit SHA256: 54940e2aea1acf788d4179495bfd995e3eef1706
Old SHA256: 1b618157b1495e48a175073b8afd53cab0954ebac72a4378b9e4880a09b585a9
New SHA256: 1b618157b1495e48a175073b8afd53cab0954ebac72a4378b9e4880a09b585a9
Removing orphan container "zeroAdmin"
Removing orphan container "alphaAdmin"
Creating alpha1 ... done 
Creating zero1  ... done 
Creating alpha2 ... done 
Creating alpha3 ... done waiting 60 seconds for localhost:6180 localhost:6180 is available after 0 seconds waiting 60 seconds for localhost:9180 localhost:9180 is available after 0 seconds
...[Decoder]: Using assembly version of decoder

INFO: Running tests in directory graphql/e2e/schema_subscribe
ok	0.549s

You can see 3 alphas + 1 zero container left behind with docker ps | awk '{ print $NF }'.

Related PR: