Hello, I have no idea how to reproduce this, so I’ll just say what happened:
First I received "ResourceExhausted desc = grpc: received message larger than max (XXXXXXX vs 4194304)"}"
out of nowhere.
My project is loading data from an online API into Dgraph whenever I run it. I saw a GitHub issue suggesting it should be fixed in versions later than Jul 3, 2017. Since I kept getting this error no matter how many times I restarted the program (had previously restarted it many times with no issues), then I decided to stop the docker-compose containers.
After restarting the containers then I first got these errors repeated over and over:
server_1 | 2018/12/13 09:17:45 Error while retrieving timestamps: rpc error: code = Unknown desc = Assigning IDs is only allowed on leader… Will retry…
zero_1 | 2018/12/13 09:17:45 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
zero_1 | 2018/12/13 09:17:48 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
zero_1 | 2018/12/13 09:17:51 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
zero_1 | 2018/12/13 09:17:54 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
And after restarting again (multiple times) then I only get the ‘no leader on term 3’ error:
zero_1 | 2018/12/13 11:43:38 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
zero_1 | 2018/12/13 11:43:41 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
zero_1 | 2018/12/13 11:43:44 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
zero_1 | 2018/12/13 11:43:47 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
zero_1 | 2018/12/13 11:43:50 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
zero_1 | 2018/12/13 11:43:53 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
zero_1 | 2018/12/13 11:43:56 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
zero_1 | 2018/12/13 11:43:59 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
zero_1 | 2018/12/13 11:44:00 oracle.go:92: Purged below ts:43063, len(o.commits):1, len(o.rowCommit):0
zero_1 | 2018/12/13 11:44:02 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
zero_1 | 2018/12/13 11:44:05 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
zero_1 | 2018/12/13 11:44:08 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
zero_1 | 2018/12/13 11:44:11 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
These are my zero logs:
docker logs 57c99f8ed498
Dgraph version : v1.0.7
Commit SHA-1 : f1803442
Commit timestamp : 2018-08-10 13:00:21 -0700
Branch : HEADFor Dgraph official documentation, visit https://docs.dgraph.io.
For discussions about Dgraph , visit http://discuss.dgraph.io.
To say hi to the community , visit https://dgraph.slack.com.Licensed under Apache 2.0 + Commons Clause. Copyright 2015-2018 Dgraph Labs, Inc.
Setting up grpc listener at: 0.0.0.0:5080
Setting up http listener at: 0.0.0.0:6080
2018/12/13 09:24:07 node.go:218: Found Snapshot, Metadata: {ConfState:{Nodes: XXX_unrecognized:} Index:4131 Term:3 XXX_unrecognized:}
2018/12/13 09:24:07 node.go:229: Found hardstate: {Term:3 Vote:1 Commit:4396 XXX_unrecognized:}
2018/12/13 09:24:07 node.go:238: Group 0 found 266 entries
2018/12/13 09:24:07 raft.go:355: Restarting node for dgraphzero
2018/12/13 09:24:08 pool.go:162: Echo error from server:7080. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = “transport: Error while dialing dial tcp 172.20.0.3:7080: connect: connection refused”
2018/12/13 09:24:08 pool.go:108: == CONNECTED ==> Setting server:7080
2018/12/13 09:24:08 raft.go:567: INFO: 1 became follower at term 3
2018/12/13 09:24:08 raft.go:315: INFO: newRaft 1 [peers: , term: 3, commit: 4396, applied: 4131, lastindex: 4396, lastterm: 3]
Running Dgraph zero…
2018/12/13 09:24:08 zero.go:333: Got connection request: id:1 addr:“server:7080”
2018/12/13 09:24:08 zero.go:442: Connected: id:1 addr:“server:7080”
2018/12/13 09:24:08 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
2018/12/13 09:24:11 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
2018/12/13 09:24:12 zero.go:507: Error while applying proposal in update stream While proposing error: context deadline exceeded
2018/12/13 09:24:14 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
2018/12/13 09:24:17 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
2018/12/13 09:24:20 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
2018/12/13 09:24:23 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
… this no leader error repeat for as long as I run it …
These are my server logs:
docker logs 9d61b6d7e93b
Dgraph version : v1.0.7
Commit SHA-1 : f1803442
Commit timestamp : 2018-08-10 13:00:21 -0700
Branch : HEADFor Dgraph official documentation, visit https://docs.dgraph.io.
For discussions about Dgraph , visit http://discuss.dgraph.io.
To say hi to the community , visit https://dgraph.slack.com.Licensed under Apache 2.0 + Commons Clause. Copyright 2015-2018 Dgraph Labs, Inc.
2018/12/13 09:24:08 server.go:117: Setting Badger option: ssd
2018/12/13 09:24:08 server.go:133: Setting Badger table load option: mmap
2018/12/13 09:24:08 server.go:146: Setting Badger value log load option: none
2018/12/13 09:24:08 server.go:157: Opening postings Badger DB with options: {Dir:p ValueDir:p SyncWrites:true TableLoadingMode:2 ValueLogLoadingMode:2 NumVersionsToKeep:2147483647 MaxTableSize:67108864 LevelSizeMultiplier:10 MaxLevels:7 ValueThreshold:32 NumMemtables:5 NumLevelZeroTables:5 NumLevelZeroTablesStall:10 LevelOneSize:268435456 ValueLogFileSize:1073741824 ValueLogMaxEntries:1000000 NumCompactors:3 managedTxns:false DoNotCompact:false maxBatchCount:0 maxBatchSize:0 ReadOnly:false Truncate:true}
2018/12/13 09:24:08 groups.go:78: Current Raft Id: 1
2018/12/13 09:24:08 gRPC server started. Listening on port 9080
2018/12/13 09:24:08 HTTP server started. Listening on port 8080
2018/12/13 09:24:08 worker.go:89: Worker listening at address: [::]:7080
2018/12/13 09:24:08 pool.go:108: == CONNECTED ==> Setting zero:5080
2018/12/13 09:24:08 groups.go:105: Connected to group zero. Assigned group: 0
2018/12/13 09:24:08 draft.go:74: Node ID: 1 with GroupID: 1
2018/12/13 09:24:08 node.go:218: Found Snapshot, Metadata: {ConfState:{Nodes: XXX_unrecognized:} Index:36857 Term:3 XXX_unrecognized:}
2018/12/13 09:24:08 node.go:229: Found hardstate: {Term:3 Vote:1 Commit:37748 XXX_unrecognized:}
2018/12/13 09:24:08 node.go:238: Group 1 found 892 entries
2018/12/13 09:24:08 draft.go:964: Restarting node for group: 1
2018/12/13 09:24:08 raft.go:567: INFO: 1 became follower at term 3
2018/12/13 09:24:08 raft.go:315: INFO: newRaft 1 [peers: , term: 3, commit: 37748, applied: 36857, lastindex: 37748, lastterm: 3]
2018/12/13 09:24:08 draft.go:411: Creating snapshot at index: 36857. ReadTs: 42498.
2018/12/13 09:24:08 groups.go:496: Got address of a Zero server: zero:5080
2018/12/13 09:24:08 mutation.go:174: Done schema update predicate:“nodeType” value_type:STRING directive:INDEX tokenizer:“exact”
2018/12/13 09:24:08 mutation.go:174: Done schema update predicate:“date” value_type:DATETIME directive:INDEX tokenizer:“day”
… more schema updates …
This is my docker-compose file:
version: “3.2”
services:
zero:
image: dgraph/dgraph:latest
volumes:
- ./_dgraphdata:/dgraph
ports:
- 5080:5080
- 6080:6080
restart: on-failure
command: dgraph zero --my=zero:5080
server:
image: dgraph/dgraph:latest
volumes:
- ./_dgraphdata:/dgraph
ports:
- 8080:8080
- 9080:9080
restart: on-failure
command: dgraph server --my=server:7080 --lru_mb=2048 --zero=zero:5080
ratel:
image: dgraph/dgraph:latest
ports:
- 8000:8000
command: dgraph-ratel
Not sure if it’s relevant, but the _dgraphdata
folder (total size 350MB) is owned by root while the project is run by user (but that’s always been the case without issues).