Predicate move keeps repeating, but not completing, every 5 minutes forcing ingest to be stopped

Users Dgraph
1

An Alpha is logging that it is moving a predicate, however it attempts to move the same predicate every 5 minutes. While this is going on, ingest has crawled to nearly a complete stop, except for a few records that can slip in.

Any idea what might be going on or where to lock for more information?
Alpha Logs

dgraph_alpha4.1.mntsdoqergd0@dgraph-4    | I0117  20:53:10.172402    1 predicate_move.go:220] Was instructed to delete tablet: 0-foo
dgraph_alpha4.1.mntsdoqergd0@dgraph-4    | I0117  20:53:10.174534    1 index.go:1245] Dropping predicate: [0-foo]
dgraph_alpha4.1.mntsdoqergd0@dgraph-4    | I0117  20:53:10.172565    1 log.go:33] DropPrefix called for [
dgraph_alpha4.1.mntsdoqergd0@dgraph-4    | foo]
dgraph_alpha4.1.mntsdoqergd0@dgraph-4    | I0117  20:53:10.174586    1 log.go:33] Writes flushed. Stopping compactions now... 
dgraph_alpha4.1.mntsdoqergd0@dgraph-4    | I0117  20:53:10.174725    1 log.go:33] No prefixes to drop 
dgraph_alpha4.1.mntsdoqergd0@dgraph-4    | I0117  20:53:10.174736    1 log.go:33] Resuming writes 
dgraph_alpha4.1.mntsdoqergd0@dgraph-4    | I0117  20:53:10.174749    1 log.go:33] Deleting schema for predicate: [0-foo]
... Logs about Block cache metrics, Raft progress, Calculates tablet sizes..
Almost exactly five minutes after the Deleting schema for predicate, it starts over again.

Zero Logs

dgraph_zero2.1.q02t93f5k72f@dgraph-3    | I0117  20:48:10.159738    1 zero.go:823] Tablet: 0-foo does not belong to group: 1. Sending delete instruction.
1 Like
2

We resolved this by calling the /moveTablet?tablet=name&group=2 API on the predicated that was failing to finish the move. We had to move it 2 times. Once back to the original Group, but the error simply moved, then back to the group Dgraph initially wanted it to move to. After moving it twice, the error no longer presented itself and ingestion worked again.

Doc reference:
https://dgraph.io/docs/deploy/dgraph-zero/

2 Likes