@iluminae any progress with your setup?
I have the same problem: all 3 alphas in group crashes with the same error:
I0128 08:56:19.765476 18 groups.go:867] Got address of a Zero leader: intouch-dgraph-zero-1.intouch-dgraph-zero-headless.staging.svc.cluster.local:5080
I0128 08:56:19.766039 18 groups.go:881] Starting a new membership stream receive from intouch-dgraph-zero-1.intouch-dgraph-zero-headless.staging.svc.cluster.local:5080.
I0128 08:56:19.768307 18 groups.go:898] Received first state update from Zero: counter:73698 groups:<key:1 value:<members:<key:1 value:<id:1 group_id:1 addr:"intouch-dgraph-alpha-0.intouch-dgraph-alpha-headless.staging.svc.cluster.local:7080" last_update:1643105936 > > members:<key:2 value:<id:2 group_id:1 addr:"intouch-dgraph-alpha-1.intouch-dgraph-alpha-headless.staging.svc.cluster.local:7080" last_update:1643337197 > > members:<key:3 value:<id:3 group_id:1 addr:"intouch-dgraph-alpha-2.intouch-dgraph-alpha-headless.staging.svc.cluster.local:7080" last_update:1643337206 > > tablets:<key:"0-assetState" value:<group_id:1 predicate:"0-assetState" on_disk_bytes:6201115 uncompressed_bytes:12128311 > > tablets:<key:"0-base64EncodedDefinition" value:<group_id:1 predicate:"0-base64EncodedDefinition" on_disk_bytes:138314480 uncompressed_bytes:582461588 > > tablets:<key:"0-base64EncodedWorkflowConfiguration" value:<group_id:1 predicate:"0-base64EncodedWorkflowConfiguration" on_disk_bytes:9819 uncompressed_bytes:32627 > > tablets:<key:"0-bitmap$init$0" value:<group_id:1 predicate:"0-bitmap$init$0" > > tablets:<key:"0-checkedOutBy" value:<group_id:1 predicate:"0-checkedOutBy" on_disk_bytes:207 uncompressed_bytes:65 > > tablets:<key:"0-childOf" value:<group_id:1 predicate:"0-childOf" on_disk_bytes:2774 uncompressed_bytes:24312 > > tablets:<key:"0-color" value:<group_id:1 predicate:"0-color" on_disk_bytes:190 uncompressed_bytes:58 > > tablets:<key:"0-contentType" value:<group_id:1 predicate:"0-contentType" on_disk_bytes:2059 uncompressed_bytes:5021 > > tablets:<key:"0-created" value:<group_id:1 predicate:"0-created" on_disk_bytes:288 uncompressed_bytes:457 > > tablets:<key:"0-createdBy" value:<group_id:1 predicate:"0-createdBy" on_disk_bytes:4924838 uncompressed_bytes:11617351 > > tablets:<key:"0-deliveryMethods" value:<group_id:1 predicate:"0-deliveryMethods" on_disk_bytes:471 uncompressed_bytes:1204 > > tablets:<key:"0-description" value:<group_id:1 predicate:"0-description" on_disk_bytes:10845 uncompressed_bytes:61598 > > tablets:<key:"0-dgraph.acl.rule" value:<group_id:1 predicate:"0-dgraph.acl.rule" > > tablets:<key:"0-dgraph.cors" value:<group_id:1 predicate:"0-dgraph.cors" > > tablets:<key:"0-dgraph.drop.op" value:<group_id:1 predicate:"0-dgraph.drop.op" > > tablets:<key:"0-dgraph.graphql.p_query" value:<group_id:1 predicate:"0-dgraph.graphql.p_query" > > tablets:<key:"0-dgraph.graphql.p_sha256hash" value:<group_id:1 predicate:"0-dgraph.graphql.p_sha256hash" > > tablets:<key:"0-dgraph.graphql.schema" value:<group_id:1 predicate:"0-dgraph.graphql.schema" > > tablets:<key:"0-dgraph.graphql.schema_created_at" value:<group_id:1 predicate:"0-dgraph.graphql.schema_created_at" > > tablets:<key:"0-dgraph.graphql.schema_history" value:<group_id:1 predicate:"0-dgraph.graphql.schema_history" > > tablets:<key:"0-dgraph.graphql.xid" value:<group_id:1 predicate:"0-dgraph.graphql.xid" > > tablets:<key:"0-dgraph.group.acl" value:<group_id:1 predicate:"0-dgraph.group.acl" > > tablets:<key:"0-dgraph.password" value:<group_id:1 predicate:"0-dgraph.password" > > tablets:<key:"0-dgraph.rule.permission" value:<group_id:1 predicate:"0-dgraph.rule.permission" > > tablets:<key:"0-dgraph.rule.predicate" value:<group_id:1 predicate:"0-dgraph.rule.predicate" > > tablets:<key:"0-dgraph.type" value:<group_id:1 predicate:"0-dgraph.type" on_disk_bytes:5681552 uncompressed_bytes:13055820 > > tablets:<key:"0-dgraph.user.group" value:<group_id:1 predicate:"0-dgraph.user.group" > > tablets:<key:"0-dgraph.xid" value:<group_id:1 predicate:"0-dgraph.xid" > > tablets:<key:"0-inWorkflow" value:<group_id:1 predicate:"0-inWorkflow" on_disk_bytes:283 uncompressed_bytes:446 > > tablets:<key:"0-locks" value:<group_id:1 predicate:"0-locks" on_disk_bytes:129336 uncompressed_bytes:656025 > > tablets:<key:"0-logger" value:<group_id:1 predicate:"0-logger" > > tablets:<key:"0-name" value:<group_id:1 predicate:"0-name" on_disk_bytes:3984283 uncompressed_bytes:5017783 > > tablets:<key:"0-nextVersion" value:<group_id:1 predicate:"0-nextVersion" on_disk_bytes:351421 uncompressed_bytes:2037143 > > tablets:<key:"0-nodeType" value:<group_id:1 predicate:"0-nodeType" on_disk_bytes:804367 uncompressed_bytes:4415210 > > tablets:<key:"0-ownerId" value:<group_id:1 predicate:"0-ownerId" on_disk_bytes:277 uncompressed_bytes:449 > > tablets:<key:"0-parentOf" value:<group_id:1 predicate:"0-parentOf" on_disk_bytes:296810 uncompressed_bytes:1429209 > > tablets:<key:"0-previousVersion" value:<group_id:1 predicate:"0-previousVersion" on_disk_bytes:264 uncompressed_bytes:423 > > tablets:<key:"0-processId" value:<group_id:1 predicate:"0-processId" > > tablets:<key:"0-publishedTo" value:<group_id:1 predicate:"0-publishedTo" on_disk_bytes:1867862 uncompressed_bytes:14688251 > > tablets:<key:"0-qaReviewer" value:<group_id:1 predicate:"0-qaReviewer" > > tablets:<key:"0-sourceOf" value:<group_id:1 predicate:"0-sourceOf" on_disk_bytes:195 uncompressed_bytes:61 > > tablets:<key:"0-staticId" value:<group_id:1 predicate:"0-staticId" on_disk_bytes:308 uncompressed_bytes:479 > > tablets:<key:"0-subtype" value:<group_id:1 predicate:"0-subtype" on_disk_bytes:196 uncompressed_bytes:60 > > tablets:<key:"0-tenant" value:<group_id:1 predicate:"0-tenant" on_disk_bytes:75020 uncompressed_bytes:638136 > > tablets:<key:"0-textRepresentation" value:<group_id:1 predicate:"0-textRepresentation" on_disk_bytes:5958 uncompressed_bytes:18225 > > tablets:<key:"0-uid" value:<group_id:1 predicate:"0-uid" > > tablets:<key:"0-updated" value:<group_id:1 predicate:"0-updated" on_disk_bytes:878016 uncompressed_bytes:6049651 > > tablets:<key:"0-usedBy" value:<group_id:1 predicate:"0-usedBy" on_disk_bytes:193 uncompressed_bytes:59 > > tablets:<key:"0-uses" value:<group_id:1 predicate:"0-uses" on_disk_bytes:245 uncompressed_bytes:412 > > tablets:<key:"0-validFrom" value:<group_id:1 predicate:"0-validFrom" on_disk_bytes:203 uncompressed_bytes:62 > > tablets:<key:"0-validTo" value:<group_id:1 predicate:"0-validTo" on_disk_bytes:192 uncompressed_bytes:60 > > tablets:<key:"0-versionId" value:<group_id:1 predicate:"0-versionId" on_disk_bytes:123306 uncompressed_bytes:412075 > > tablets:<key:"0-workflowState" value:<group_id:1 predicate:"0-workflowState" > > snapshot_ts:682184 checksum:10996207276505420656 > > zeros:<key:1 value:<id:1 addr:"intouch-dgraph-zero-0.intouch-dgraph-zero-headless.staging.svc.cluster.local:5080" > > zeros:<key:2 value:<id:2 addr:"intouch-dgraph-zero-1.intouch-dgraph-zero-headless.staging.svc.cluster.local:5080" leader:true > > zeros:<key:3 value:<id:3 addr:"intouch-dgraph-zero-2.intouch-dgraph-zero-headless.staging.svc.cluster.local:5080" > > maxUID:27278073 maxTxnTs:690000 maxRaftId:3 cid:"70c16084-935e-4cd6-a783-155a039fff7a" license:<maxNodes:18446744073709551615 expiryTs:1645064788 enabled:true >
2022/01/28 08:56:19 Unable to find txn with start ts: 682333
github.com/dgraph-io/dgraph/x.AssertTruef
/ext-go/1/src/github.com/dgraph-io/dgraph/x/error.go:107
github.com/dgraph-io/dgraph/worker.(*node).applyMutations
/ext-go/1/src/github.com/dgraph-io/dgraph/worker/draft.go:707
github.com/dgraph-io/dgraph/worker.(*node).applyCommitted
/ext-go/1/src/github.com/dgraph-io/dgraph/worker/draft.go:744
github.com/dgraph-io/dgraph/worker.(*node).processApplyCh.func1
/ext-go/1/src/github.com/dgraph-io/dgraph/worker/draft.go:931
github.com/dgraph-io/dgraph/worker.(*node).processApplyCh
/ext-go/1/src/github.com/dgraph-io/dgraph/worker/draft.go:1020
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1581
It’s second time I got this, the first time it was for only 1 node and restoring it with fresh storage helped.