Report a Dgraph Bug
Not sure if this is a bug or user-error, but I accidentally used the latest image tag on my 3-node kubernetes deployment. It upgraded from v20.07.2 to v20.11.0 yesterday and ran into errors with version number 7 not 8. I understand that’s because of storage compatibility, this isn’t the issue I’m reporting.
I’ve set both alpha and zero replicas down to 0, downgraded the StatefulSets back to v20.07.2 and then scaled my zero’s StatefulSet back to 3 replicas. Now I get an error on the first zero, here are the logs:
Dgraph version : v20.07.2
Dgraph codename : shuri-2
Dgraph SHA-256 : a927845127dab735c24727d5a24af411168771b55236aec50f0b987e8c0ac910
Commit SHA-1 : a7bc16d56
Commit timestamp : 2020-10-22 10:17:53 -0700
Branch : HEAD
Go version : go1.14.4
For Dgraph official documentation, visit https://dgraph.io/docs/.
For discussions about Dgraph , visit https://discuss.dgraph.io.
To say hi to the community , visit https://dgraph.slack.com.
Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2020 Dgraph Labs, Inc.
I0123 21:51:45.158537 19 run.go:127] Setting up grpc listener at: 0.0.0.0:5080
I0123 21:51:45.159438 19 run.go:127] Setting up http listener at: 0.0.0.0:6080
I0123 21:51:45.160751 19 run.go:298] Opening zero BadgerDB with options: {Dir:zw ValueDir:zw SyncWrites:false TableLoadingMode:2 ValueLogLoadingMode:2 NumVersionsToKeep:1 ReadOnly:false Truncate:true Logger:0xc000
badger 2021/01/23 21:51:45 INFO: All 1 tables opened in 5ms
badger 2021/01/23 21:51:45 INFO: Replaying file id: 6 at offset: 42785522
badger 2021/01/23 21:51:45 INFO: Replay took: 4.921µs
I0123 21:51:45.208055 19 node.go:149] Setting raft.Config to: &{ID:1 peers:[] learners:[] ElectionTick:20 HeartbeatTick:1 Storage:0xc000112050 Applied:1444061 MaxSizePerMsg:262144 MaxCommittedSizePerReady:67108864
I0123 21:51:45.208368 19 node.go:307] Found Snapshot.Metadata: {ConfState:{Nodes:[1 2 3] Learners:[] XXX_unrecognized:[]} Index:1444061 Term:3 XXX_unrecognized:[]}
I0123 21:51:45.208468 19 node.go:318] Found hardstate: {Term:7 Vote:0 Commit:1445344 XXX_unrecognized:[]}
I0123 21:51:45.209336 19 node.go:327] Group 0 found 1283 entries
I0123 21:51:45.209362 19 raft.go:515] Restarting node for dgraphzero
I0123 21:51:45.209427 19 node.go:186] Setting conf state to nodes:1 nodes:2 nodes:3
I0123 21:51:45.209593 19 pool.go:160] CONNECTING to dgraph-alpha-0.dgraph-alpha.default.svc.cluster.local:7080
I0123 21:51:45.209619 19 pool.go:160] CONNECTING to dgraph-alpha-1.dgraph-alpha.default.svc.cluster.local:7080
I0123 21:51:45.209642 19 pool.go:160] CONNECTING to dgraph-alpha-2.dgraph-alpha.default.svc.cluster.local:7080
I0123 21:51:45.209665 19 pool.go:160] CONNECTING to dgraph-zero-1.dgraph-zero.default.svc.cluster.local:5080
I0123 21:51:45.209724 19 pool.go:160] CONNECTING to dgraph-zero-2.dgraph-zero.default.svc.cluster.local:5080
I0123 21:51:45.210211 19 log.go:34] 1 became follower at term 7
I0123 21:51:45.210517 19 log.go:34] newRaft 1 [peers: [1,2,3], term: 7, commit: 1445344, applied: 1444061, lastindex: 1445344, lastterm: 7]
[Sentry] 2021/01/23 21:51:45 Sending fatal event [1118306852724b18a70f595e9e64d2cc] to o318308.ingest.sentry.io project: 1805390
2021/01/23 21:51:45 proto: ZeroProposal: illegal tag 0 (wire type 0)
github.com/dgraph-io/dgraph/x.Check
/ext-go/1/src/github.com/dgraph-io/dgraph/x/error.go:42
github.com/dgraph-io/dgraph/dgraph/cmd/zero.run
/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/zero/run.go:324
github.com/dgraph-io/dgraph/dgraph/cmd/zero.init.0.func1
/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/zero/run.go:79
github.com/spf13/cobra.(*Command).execute
/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:70
main.main
/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/main.go:78
runtime.main
/usr/local/go/src/runtime/proc.go:203
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1373
What is the hardware spec (RAM, OS)?
Each zero/alpha is on a 4CPU 16GB node
Expected behaviour and actual result.
I can downgrade back to v20.07