Report a Dgraph Bug
What version of Dgraph are you using?
$ dgraph version Dgraph version : v21.12.0 Dgraph codename : zion Dgraph SHA-256 : 078c75df9fa1057447c8c8afc10ea57cb0a29dfb22f9e61d8c334882b4b4eb37 Commit SHA-1 : d62ed5f15 Commit timestamp : 2021-12-02 21:20:09 +0530 Branch : HEAD Go version : go1.17.3 jemalloc enabled : true
Have you tried reproducing the issue with the latest release?
What is the hardware spec (RAM, OS)?
k8s 15x 16c 64GiB ram on GKE
Steps to reproduce the issue (command/config used to run Dgraph).
filed Panic on tablet move v21.12.0 but seems like it is more broad than just during a tablet move. Seems like any transaction may be removed from a internal map when it should not be and will cause a crashloop that will bring down entire groups since they share transaction atomicity via raft.
2021/12/22 06:56:16 Unable to find txn with start ts: 2178082 github.com/dgraph-io/dgraph/x.AssertTruef /ext-go/1/src/github.com/dgraph-io/dgraph/x/error.go:107 github.com/dgraph-io/dgraph/worker.(*node).applyMutations /ext-go/1/src/github.com/dgraph-io/dgraph/worker/draft.go:707 github.com/dgraph-io/dgraph/worker.(*node).applyCommitted /ext-go/1/src/github.com/dgraph-io/dgraph/worker/draft.go:744 github.com/dgraph-io/dgraph/worker.(*node).processApplyCh.func1 /ext-go/1/src/github.com/dgraph-io/dgraph/worker/draft.go:931 github.com/dgraph-io/dgraph/worker.(*node).processApplyCh.func2 /ext-go/1/src/github.com/dgraph-io/dgraph/worker/draft.go:970 github.com/dgraph-io/dgraph/worker.(*node).processApplyCh /ext-go/1/src/github.com/dgraph-io/dgraph/worker/draft.go:1025 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1581
This is just after a node restart in that group - I think a snapshot is being sent from one node to the other and it just explodes. When this happens to the whole group at once thats it - game over, have to restore from backup.