Alpha stuck at Raft.Ready took too long to process

Moved from GitHub dgraph/5946

Posted by jarifibrahim:

What version of Dgraph are you using?

Have you tried reproducing the issue with the latest release?

Yes, master is on https://github.com/dgraph-io/dgraph/commit/156bc23bca4ef941ed5b1e84638961764bd59f27

Steps to reproduce the issue (command/config used to run Dgraph).

Run 6 node dgraph cluster and run flock against it.
Flock was running for a week before it ran in to this issue.

Expected behaviour and actual result.

I expected dgraph to work normally.

The actual result was that dgraph got stuck in the Raft.Ready took too long to process forever. For the first few days active mutations were failing. Later I stopped the mutations hoping the cluster will recover by it never recovered.

alpha3    | W0712 13:39:31.410790      19 draft.go:1222] Raft.Ready took too long to process: Timer Total: 984ms. Breakdown: [{advance 983ms} {disk 1ms} {proposals 0s}] Num entries: 0. MustSync: false
alpha3    | W0712 13:39:31.882951      19 draft.go:1222] Raft.Ready took too long to process: Timer Total: 472ms. Breakdown: [{advance 470ms} {disk 1ms} {proposals 0s}] Num entries: 1. MustSync: true
alpha1    | W0712 13:39:33.340034      19 draft.go:1222] Raft.Ready took too long to process: Timer Total: 563ms. Breakdown: [{advance 562ms} {disk 1ms} {proposals 0s}] Num entries: 0. MustSync: false
alpha3    | W0712 13:39:33.522478      19 draft.go:1222] Raft.Ready took too long to process: Timer Total: 1.035s. Breakdown: [{advance 1.034s} {disk 1ms} {proposals 0s}] Num entries: 0. MustSync: false
alpha3    | W0712 13:39:34.005682      19 draft.go:1222] Raft.Ready took too long to process: Timer Total: 483ms. Breakdown: [{advance 483ms} {disk 0s} {proposals 0s}] Num entries: 0. MustSync: false
alpha1    | W0712 13:39:34.553391      19 draft.go:1222] Raft.Ready took too long to process: Timer Total: 563ms. Breakdown: [{advance 562ms} {disk 0s} {proposals 0s}] Num entries: 0. MustSync: false
alpha1    | W0712 13:39:35.965456      19 draft.go:1222] Raft.Ready took too long to process: Timer Total: 856ms. Breakdown: [{advance 855ms} {disk 0s} {proposals 0s}] Num entries: 0. MustSync: false
alpha3    | W0712 13:39:36.064111      19 draft.go:1222] Raft.Ready took too long to process: Timer Total: 1.509s. Breakdown: [{advance 1.509s} {disk 0s} {proposals 0s}] Num entries: 0. MustSync: false
alpha1    | W0712 13:39:36.561155      19 draft.go:1222] Raft.Ready took too long to process: Timer Total: 596ms. Breakdown: [{advance 595ms} {disk 0s} {proposals 0s}] Num entries: 0. MustSync: false

See logs here (truncated logs) - https://gist.github.com/jarifibrahim/67b2e854349945adc396f4b9c49e051d