After I tried to remove the failed node, the restarted node prompted repeated raft Id
++ hostname -f
+ dgraph alpha --my=dgraph-alpha-2.dgraph-alpha.crm-test.svc.cluster.local:7080 --zero dgraph-zero-0.dgraph-zero.crm-test.svc.cluster.local:5080,dgraph-zero-1.dgraph-zero.crm-test.svc.cluster.local:5080,dgraph-zero-2.dgraph-zero.crm-test.svc.cluster.local:5080
[Decoder]: Using assembly version of decoder
Page Size: 4096
[Sentry] 2021/03/18 11:38:34 Integration installed: ContextifyFrames
[Sentry] 2021/03/18 11:38:34 Integration installed: Environment
[Sentry] 2021/03/18 11:38:34 Integration installed: Modules
[Sentry] 2021/03/18 11:38:34 Integration installed: IgnoreErrors
[Decoder]: Using assembly version of decoder
Page Size: 4096
[Sentry] 2021/03/18 11:38:34 Integration installed: ContextifyFrames
[Sentry] 2021/03/18 11:38:34 Integration installed: Environment
[Sentry] 2021/03/18 11:38:34 Integration installed: Modules
[Sentry] 2021/03/18 11:38:34 Integration installed: IgnoreErrors
I0318 11:38:35.172320 19 sentry_integration.go:48] This instance of Dgraph will send anonymous reports of panics back to Dgraph Labs via Sentry. No confidential information is sent. These reports help improve Dgraph. To opt-out, restart your instance with the --enable_sentry=false flag. For more info, see https://dgraph.io/docs/howto/#data-handling.
I0318 11:38:35.379017 19 init.go:107]
Dgraph version : v20.11.2
Dgraph codename : tchalla-2
Dgraph SHA-256 : 0153cb8d3941ad5ad107e395b347e8d930a0b4ead6f4524521f7a525a9699167
Commit SHA-1 : 94f3a0430
Commit timestamp : 2021-02-23 13:07:17 +0530
Branch : HEAD
Go version : go1.15.5
jemalloc enabled : true
For Dgraph official documentation, visit https://dgraph.io/docs/.
For discussions about Dgraph , visit http://discuss.dgraph.io.
Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2020 Dgraph Labs, Inc.
I0318 11:38:35.379045 19 run.go:696] x.Config: {PortOffset:0 QueryEdgeLimit:1000000 NormalizeNodeLimit:10000 MutationsNQuadLimit:1000000 PollInterval:1s GraphqlExtension:true GraphqlDebug:false GraphqlLambdaUrl:}
I0318 11:38:35.379086 19 run.go:697] x.WorkerConfig: {TmpDir:t ExportPath:export NumPendingProposals:256 Tracing:0.01 MyAddr:dgraph-alpha-2.dgraph-alpha.crm-test.svc.cluster.local:7080 ZeroAddr:[dgraph-zero-0.dgraph-zero.crm-test.svc.cluster.local:5080 dgraph-zero-1.dgraph-zero.crm-test.svc.cluster.local:5080 dgraph-zero-2.dgraph-zero.crm-test.svc.cluster.local:5080] TLSClientConfig:<nil> TLSServerConfig:<nil> RaftId:0 WhiteListedIPRanges:[] MaxRetries:-1 StrictMutations:false AclEnabled:false AbortOlderThan:5m0s SnapshotAfter:10000 ProposedGroupId:0 StartTime:2021-03-18 11:38:34.735965472 +0000 UTC m=+0.014797633 LudicrousMode:false LudicrousConcurrency:2000 EncryptionKey:**** LogRequest:0 HardSync:false}
I0318 11:38:35.379133 19 run.go:698] worker.Config: {PostingDir:p PostingDirCompression:1 PostingDirCompressionLevel:0 WALDir:w MutationsMode:0 AuthToken: PBlockCacheSize:697932185 PIndexCacheSize:375809638 WalCache:0 HmacSecret:**** AccessJwtTtl:0s RefreshJwtTtl:0s CachePercentage:0,65,35,0 CacheMb:0}
I0318 11:38:35.379282 19 log.go:295] Found file: 37 First Index: 0
I0318 11:38:35.380563 19 storage.go:132] Init Raft Storage with snap: 0, first: 1, last: 0
I0318 11:38:35.380585 19 server_state.go:76] Setting Posting Dir Compression Level: 0
I0318 11:38:35.380591 19 server_state.go:120] Opening postings BadgerDB with options: {Dir:p ValueDir:p SyncWrites:false NumVersionsToKeep:2147483647 ReadOnly:false Logger:0x2e0fef8 Compression:1 InMemory:false MemTableSize:67108864 BaseTableSize:2097152 BaseLevelSize:10485760 LevelSizeMultiplier:10 TableSizeMultiplier:2 MaxLevels:7 ValueThreshold:1024 NumMemtables:5 BlockSize:4096 BloomFalsePositive:0.01 BlockCacheSize:697932185 IndexCacheSize:375809638 NumLevelZeroTables:5 NumLevelZeroTablesStall:15 ValueLogFileSize:1073741823 ValueLogMaxEntries:1000000 NumCompactors:4 CompactL0OnClose:false ZSTDCompressionLevel:0 VerifyValueChecksum:false EncryptionKey:[] EncryptionKeyRotationDuration:240h0m0s BypassLockGuard:false ChecksumVerificationMode:0 DetectConflicts:false managedTxns:false maxBatchCount:0 maxBatchSize:0}
I0318 11:38:35.387582 19 log.go:34] All 0 tables opened in 0s
I0318 11:38:35.387922 19 log.go:34] Discard stats nextEmptySlot: 0
I0318 11:38:35.387944 19 log.go:34] Set nextTxnTs to 0
I0318 11:38:35.388016 19 log.go:34] Deleting empty file: p/000036.vlog
E0318 11:38:35.388564 19 groups.go:1143] Error during SubscribeForUpdates for prefix "\x00\x00\vdgraph.cors\x00": Unable to find any servers for group: 1. closer err: <nil>
I0318 11:38:35.388580 19 groups.go:99] Current Raft Id: 0x3
I0318 11:38:35.388783 19 worker.go:104] Worker listening at address: [::]:7080
E0318 11:38:35.389923 19 groups.go:1143] Error during SubscribeForUpdates for prefix "\x00\x00\x15dgraph.graphql.schema\x00": Unable to find any servers for group: 1. closer err: <nil>
I0318 11:38:35.389938 19 run.go:519] Bringing up GraphQL HTTP API at 0.0.0.0:8080/graphql
I0318 11:38:35.389952 19 run.go:520] Bringing up GraphQL HTTP admin API at 0.0.0.0:8080/admin
I0318 11:38:35.389983 19 run.go:552] gRPC server started. Listening on port 9080
I0318 11:38:35.389996 19 run.go:553] HTTP server started. Listening on port 8080
I0318 11:38:35.489457 19 pool.go:162] CONNECTING to dgraph-zero-0.dgraph-zero.crm-test.svc.cluster.local:5080
I0318 11:38:35.493871 19 pool.go:162] CONNECTING to dgraph-zero-1.dgraph-zero.crm-test.svc.cluster.local:5080
[Sentry] 2021/03/18 11:38:35 Sending fatal event [bf638e8034de402888a1bd942a576349] to o318308.ingest.sentry.io project: 1805390
2021/03/18 11:38:35 rpc error: code = Unknown desc = REUSE_RAFTID: Duplicate Raft ID 3 to removed member: id:3 group_id:1 addr:"dgraph-alpha-2.dgraph-alpha.crm-test.svc.cluster.local:7080" last_update:1615991006