Dgraph does not properly stop during clean shutdown

Issues Dgraph
, , , ,
1

Report a Dgraph Bug

What version of Dgraph are you using?

v20.03.4

Have you tried reproducing the issue with the latest release?

Yes

What is the hardware spec (RAM, OS)?

Linux (Kubernetes)

Steps to reproduce the issue (command/config used to run Dgraph).

  1. Run a 6-node cluster.
  2. Stop the cluster (send a SIGTERM to all processes)
  3. See that the Alphas are stuck in the “Terminating” state for 10 minutes.

Commands to reproduce on Kubernetes locally:

  1. Run a local cluster via kind
  2. Install Dgraph via Helm:
helm repo add dgraph https://charts.dgraph.io
helm install my-dgraph-release dgraph/dgraph

Then, after the cluster has started properly, delete the release to stop the cluster:

helm delete my-dgraph-release

Expected behaviour and actual result.

Dgraph should terminate after it has shut down cleanly. The logs show the following:

Stopping raftwal store...
Stopping worker server...

And then it is stuck running until Kubernetes forcibly kills the instance.

2 Likes
2

Similar to Dgraph doesn't shutdown on ctrl+c and Zero restarts on SIGINT

1 Like
3

This is very serious in general operations since bygone Unix days to current cloud native infrastructure; Responding to signals essential for container management (k8s, docker, nomad), service supervision (systemd, runit, supervisor, etc), change configuration, monitoring, etc.

4

Fixed by: fix(shutdown): Force exit if CTRL-C is caught before initialization by jarifibrahim · Pull Request #6359 · dgraph-io/dgraph · GitHub

2 Likes