Dgraph alpha crash "Was the raft log corrupted, truncated, or lost?"

Users Dgraph
,
1

My Docker swarm based dgraph cluster fell over this morning while running a large mutation. I am unable to get the logs from the failed container as the swarm removed the container, but the next 4 attempts to start the container failed with the error below.

panic: tocommit(2309607) is out of range [lastIndex(2309601)]. Was the raft log corrupted, truncated, or lost?

goroutine 424 [running]:
log.Panicf(0x1de0710, 0x5d, 0xc00b7749a0, 0x2, 0x2)
	/usr/local/go/src/log/log.go:358 +0xc5
github.com/dgraph-io/dgraph/x.(*ToGlog).Panicf(0x2e0fef8, 0x1de0710, 0x5d, 0xc00b7749a0, 0x2, 0x2)
	/ext-go/1/src/github.com/dgraph-io/dgraph/x/log.go:40 +0x53
go.etcd.io/etcd/raft.(*raftLog).commitTo(0xc0003ae150, 0x233de7)
	/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190228193606-a943ad0ee4c9/raft/log.go:203 +0x135
go.etcd.io/etcd/raft.(*raft).handleHeartbeat(0xc000324a00, 0x8, 0x17, 0x16, 0xc, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190228193606-a943ad0ee4c9/raft/raft.go:1324 +0x54
go.etcd.io/etcd/raft.stepFollower(0xc000324a00, 0x8, 0x17, 0x16, 0xc, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190228193606-a943ad0ee4c9/raft/raft.go:1269 +0x439
go.etcd.io/etcd/raft.(*raft).Step(0xc000324a00, 0x8, 0x17, 0x16, 0xc, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190228193606-a943ad0ee4c9/raft/raft.go:971 +0x1218
go.etcd.io/etcd/raft.(*node).run(0xc00b676000, 0xc000324a00)
	/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190228193606-a943ad0ee4c9/raft/node.go:357 +0x1178
created by go.etcd.io/etcd/raft.RestartNode
	/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190228193606-a943ad0ee4c9/raft/node.go:246 +0x346
[Sentry] 2021/03/31 07:19:04 Sending fatal event [e160d81942b746a98f618e1c6785481e] to o318308.ingest.sentry.io project: 1805390

Now in the logs I get lots of this in the logs

2021-03-31 09:41:58	
E0331 08:41:58.475084      15 run.go:772] Error while retrieving cors origins: GetCorsOrigins returned 2 results
2021-03-31 09:41:57	
I0331 08:41:57.642175      15 admin.go:686] Error reading GraphQL schema: found multiple nodes for GraphQL schema.

Any ideas how I fix it?

Thanks

Dean

Dgraph metadata

dgraph version

Dgraph version : v20.11.2
Dgraph codename : tchalla-2
Dgraph SHA-256 : 0153cb8d3941ad5ad107e395b347e8d930a0b4ead6f4524521f7a525a9699167
Commit SHA-1 : 94f3a0430
Commit timestamp : 2021-02-23 13:07:17 +0530
Branch : HEAD
Go version : go1.15.5
jemalloc enabled : true

2

You should start from scratch. This looks like you are mixing files, doing tests, and not cleaning the paths. You should make sure that you always start “clean” in a new env.