So many rpc error : code = DeadlineExceeded desc = context deadline exceeded

Hi,
When I run the dgraph, there are so many rpc errors, as below:

I0118 15:09:42.488943   44827 pool.go:215] Connection established with 10.237.7.231:5080
I0118 15:09:42.488998   44827 pool.go:215] Connection established with 10.237.7.230:7083
I0118 15:09:42.489107   44827 pool.go:215] Connection established with 10.237.7.232:7083
E0118 15:39:42.485876   44827 pool.go:204] Echo error from 10.237.7.231:5080. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0118 15:39:42.486024   44827 pool.go:204] Echo error from 10.237.7.230:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
I0118 15:39:42.491800   44827 pool.go:215] Connection established with 10.237.7.230:7083
I0118 15:39:42.491987   44827 pool.go:215] Connection established with 10.237.7.231:5080
W0118 16:07:11.615756   44827 node.go:352] No healthy connection to node Id: 3 addr: [10.237.7.232:7083], err: Unhealthy connection
E0118 16:07:11.616109   44827 pool.go:204] Echo error from 10.237.7.232:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
I0118 16:07:11.618927   44827 pool.go:215] Connection established with 10.237.7.232:7083
E0118 16:24:51.542813   44827 pool.go:204] Echo error from 10.237.7.232:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
I0118 16:24:52.004703   44827 pool.go:215] Connection established with 10.237.7.232:7083
E0118 16:24:52.547590   44827 draft.go:443] Lastcommit 39802 > current 28792. This would cause some commits to be lost.
E0118 16:27:12.542670   44827 pool.go:204] Echo error from 10.237.7.232:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0118 16:27:12.542820   44827 pool.go:204] Echo error from 10.237.7.231:5080. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
W0118 16:27:12.542830   44827 node.go:352] No healthy connection to node Id: 3 addr: [10.237.7.232:7083], err: Unhealthy connection
E0118 16:27:12.543024   44827 pool.go:204] Echo error from 10.237.7.230:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
I0118 16:27:12.544722   44827 pool.go:215] Connection established with 10.237.7.232:7083
I0118 16:27:12.544911   44827 pool.go:215] Connection established with 10.237.7.231:5080
I0118 16:27:12.544950   44827 pool.go:215] Connection established with 10.237.7.230:7083

And the error make the RAFT between alpha nodes unstable, I think the timeout is short, the code

1s , is it too short?And if I can config the timeout by myself, I think it is cool.

1s should be sufficient to Echo between nodes. How far are these machines – halfway across the world? Are you sure these network errors are not due to some config issues?

The three alpha nodes are under the same switch, the ping delay between them is less than 0.1ms.

Is there something wrong whit the config of my dgraph cluster?

Zero:

dgraph zero --my 10.237.7.231:5080 --replicas 3 --log_dir /export/data/dgraph

Alpha:

dgraph alpha --debugmode --pending_proposals 20 --max_retries 10 --my 10.237.7.231:7083 --lru_mb 163840 --zero 10.237.7.231:5080 -o 3  --log_dir /export/data/dgraph
dgraph alpha --debugmode --pending_proposals 20 --max_retries 10 --my 10.237.7.230:7083 --lru_mb 163840 --zero 10.237.7.231:5080 -o 3  --log_dir /export/data/dgraph
dgraph alpha --debugmode --pending_proposals 20 --max_retries 10 --my 10.237.7.232:7083 --lru_mb 163840 --zero 10.237.7.231:5080 -o 3  --log_dir /export/data/dgraph

The config looks ok. Can you paste the full logs?

https://github.com/hhtlxhhxy/dgraph-log/blob/master/dgraph.alpha.tar.gz

The alpha node in 10.237.7.232.
Thank you very much!

Hello mrjn,
To test if there is something wrong with the timeout in UpdateHealthyStatus, I clone the dgraph code from github, and checkout the tag v1.0.12-rc5. Then I change the timeout from 1s to 10s , then I build a new dgraph binary , and instead the binary I used before. There seems to be less error.

New binary:

[hehaitao3@A01-R06-I7-232-J33GKR6 dgraph]$ dgraph-new version

Dgraph version   : v1.0.12-rc6
Commit SHA-1     :
Commit timestamp :
Branch           : hehaitao/fix_health_timeout
Go version       : go1.11.1

For Dgraph official documentation, visit https://docs.dgraph.io.
For discussions about Dgraph     , visit https://discuss.dgraph.io.
To say hi to the community       , visit https://dgraph.slack.com.

Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2018 Dgraph Labs, Inc.

New binary log:

[hehaitao3@A01-R06-I7-232-J33GKR6 dgraph]$ cat  dgraph-new.ERROR|grep DeadlineExceeded
E0121 16:17:10.281438   77853 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0121 16:17:40.759539   77853 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0121 16:20:45.759165   77853 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0121 16:22:04.759190   77853 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0121 17:50:00.766776   77853 pool.go:204] Echo error from 10.237.7.230:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0121 19:40:27.759302   77853 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0121 20:13:59.759299   77853 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0121 20:14:00.755302   77853 pool.go:204] Echo error from 10.237.7.231:5080. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0121 20:14:01.484338   77853 groups.go:636] While sending membership update: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0121 20:14:21.759190   77853 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0121 23:05:20.759164   77853 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0122 02:36:48.486784   77853 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0122 02:36:48.486971   77853 pool.go:204] Echo error from 10.237.7.230:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0122 02:37:21.086037   77853 pool.go:204] Echo error from 10.237.7.231:5080. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0122 02:44:51.759674   77853 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0122 04:20:34.310761   77853 pool.go:204] Echo error from 10.237.7.230:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0122 04:21:22.759439   77853 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0122 04:25:04.759165   77853 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0122 05:19:44.759274   77853 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0122 05:49:06.759306   77853 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0122 08:59:11.759255   77853 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0122 10:11:02.047545   77853 pool.go:204] Echo error from 10.237.7.230:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0122 10:24:12.759276   77853 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0122 10:34:52.759089   77853 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0122 10:35:14.759149   77853 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
[hehaitao3@A01-R06-I7-232-J33GKR6 dgraph]$ cat  dgraph-new.ERROR|grep DeadlineExceeded|wc -l
25
[hehaitao3@A01-R06-I7-232-J33GKR6 dgraph]$

Old binary log:

E0120 16:16:08.463222  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 16:17:01.463306  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 16:17:47.463335  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 16:54:13.473504  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 16:54:21.447099  139969 pool.go:204] Echo error from 10.237.7.231:5080. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 16:54:21.463205  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 16:54:31.463275  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 16:54:37.463214  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 16:54:39.463247  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 16:56:49.463231  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 17:20:00.246351  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 17:20:02.447647  139969 pool.go:204] Echo error from 10.237.7.231:5080. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 17:20:02.678359  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 17:20:07.534880  139969 pool.go:204] Echo error from 10.237.7.231:5080. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 17:20:07.534989  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 17:20:07.535004  139969 pool.go:204] Echo error from 10.237.7.230:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 17:20:30.447056  139969 pool.go:204] Echo error from 10.237.7.231:5080. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 17:20:30.463259  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 17:20:45.463190  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 17:20:50.463186  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 17:21:00.463169  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 17:21:03.446990  139969 pool.go:204] Echo error from 10.237.7.231:5080. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 17:21:17.463237  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 17:21:25.463191  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 17:32:47.463244  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 17:35:17.463309  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 17:39:41.465065  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 17:39:41.522029  139969 pool.go:204] Echo error from 10.237.7.231:5080. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 17:39:53.772889  139969 pool.go:204] Echo error from 10.237.7.230:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 17:39:53.773279  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 17:39:57.442494  139969 pool.go:204] Echo error from 10.237.7.231:5080. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 17:40:06.014029  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 17:40:10.127721  139969 pool.go:204] Echo error from 10.237.7.230:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 17:40:15.176120  139969 pool.go:204] Echo error from 10.237.7.230:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 17:40:15.176218  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 17:40:15.176274  139969 pool.go:204] Echo error from 10.237.7.231:5080. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 17:56:37.463259  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 18:03:39.463319  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 18:03:52.463288  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 18:03:59.463277  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 18:08:35.463213  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 18:09:21.463211  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 18:10:17.463289  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 18:34:07.842982  139969 pool.go:204] Echo error from 10.237.7.231:5080. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 18:34:10.276296  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 18:34:10.276368  139969 pool.go:204] Echo error from 10.237.7.230:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 18:39:03.090659  139969 pool.go:204] Echo error from 10.237.7.230:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 18:39:12.639953  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 18:39:12.639959  139969 pool.go:204] Echo error from 10.237.7.231:5080. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 18:39:39.463441  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 18:40:10.464185  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 18:40:26.463371  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 18:41:43.463216  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 18:41:59.463237  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 18:42:03.463336  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 18:43:52.707130  139969 pool.go:204] Echo error from 10.237.7.230:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 18:43:52.707286  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 18:51:45.463244  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 18:51:49.463336  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 18:51:59.463294  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 19:04:09.463361  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 19:04:14.701861  139969 pool.go:204] Echo error from 10.237.7.230:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 19:04:28.146527  139969 pool.go:204] Echo error from 10.237.7.231:5080. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 19:06:41.463294  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 19:07:47.463307  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 19:21:39.463226  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 19:21:45.463310  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 19:22:07.463310  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 19:22:29.463248  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 19:22:33.463252  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 19:29:35.643046  139969 pool.go:204] Echo error from 10.237.7.231:5080. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 19:29:41.358028  139969 pool.go:204] Echo error from 10.237.7.230:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 19:29:41.482968  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 19:44:54.463142  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 19:45:07.463275  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 19:47:04.463499  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 19:47:28.463291  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 19:55:36.804218  139969 pool.go:204] Echo error from 10.237.7.230:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 19:55:36.804288  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 19:55:36.804446  139969 pool.go:204] Echo error from 10.237.7.231:5080. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 20:16:53.463321  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 20:17:01.463232  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 20:19:16.715622  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 20:19:25.151088  139969 pool.go:204] Echo error from 10.237.7.231:5080. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 20:19:25.151105  139969 pool.go:204] Echo error from 10.237.7.230:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 20:19:25.151458  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 20:19:44.281587  139969 pool.go:204] Echo error from 10.237.7.231:5080. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 20:19:44.281668  139969 pool.go:204] Echo error from 10.237.7.230:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 20:19:44.281844  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 20:27:22.463238  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 20:27:31.463223  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0120 20:28:06.463175  139969 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = 

I captured a small portion of the error logs. At the same time range, the number of the error log is 396. And the new binary is 25.
I think to increase the timeout is effective.

The main question is, why are there connection errors anyways? You said, these machines are on the same rack.

Can you run a ping from one of these alphas to other servers, and paste that log as well (ensure that the timestamps are in the same timezone and match)? That way, we can try to determine if this is a connection issue, or is this something specific to Dgraph?

the alpha error log:

E0122 12:14:10.083323   76857 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0122 12:14:18.083198   76857 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0122 12:14:28.083152   76857 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0122 12:14:35.419328   76857 groups.go:869] Error in oracle delta stream. Error: rpc error: code = Canceled desc = context canceled
E0122 12:14:41.083193   76857 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0122 12:14:48.083247   76857 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0122 12:14:58.389756   76857 groups.go:869] Error in oracle delta stream. Error: rpc error: code = Canceled desc = context canceled
E0122 12:21:17.083762   76857 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0122 12:22:28.189989   76857 groups.go:869] Error in oracle delta stream. Error: rpc error: code = Canceled desc = context canceled
E0122 12:22:28.189797   76857 pool.go:204] Echo error from 10.237.7.231:5080. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0122 12:22:29.352068   76857 pool.go:204] Echo error from 10.237.7.232:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0122 12:27:56.728473   76857 groups.go:869] Error in oracle delta stream. Error: rpc error: code = Canceled desc = context canceled
E0122 12:28:25.083196   76857 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0122 12:37:59.083484   76857 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0122 12:45:55.083246   76857 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0122 12:58:25.083267   76857 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0122 13:02:27.083230   76857 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0122 13:02:29.083274   76857 pool.go:204] Echo error from 10.237.7.231:7083. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded

the ping log:

64 bytes from 10.237.7.231: icmp_seq=2976 ttl=64 time=0.088 ms  2019-01-22 12:14:00
64 bytes from 10.237.7.231: icmp_seq=2977 ttl=64 time=0.053 ms  2019-01-22 12:14:01
64 bytes from 10.237.7.231: icmp_seq=2978 ttl=64 time=0.090 ms  2019-01-22 12:14:02
64 bytes from 10.237.7.231: icmp_seq=2979 ttl=64 time=0.107 ms  2019-01-22 12:14:03
64 bytes from 10.237.7.231: icmp_seq=2980 ttl=64 time=0.067 ms  2019-01-22 12:14:04
64 bytes from 10.237.7.231: icmp_seq=2981 ttl=64 time=0.056 ms  2019-01-22 12:14:05
64 bytes from 10.237.7.231: icmp_seq=2982 ttl=64 time=0.055 ms  2019-01-22 12:14:06
64 bytes from 10.237.7.231: icmp_seq=2983 ttl=64 time=0.089 ms  2019-01-22 12:14:07
64 bytes from 10.237.7.231: icmp_seq=2984 ttl=64 time=0.062 ms  2019-01-22 12:14:08
64 bytes from 10.237.7.231: icmp_seq=2985 ttl=64 time=0.056 ms  2019-01-22 12:14:09
64 bytes from 10.237.7.231: icmp_seq=2986 ttl=64 time=0.099 ms  2019-01-22 12:14:10
64 bytes from 10.237.7.231: icmp_seq=2987 ttl=64 time=0.128 ms  2019-01-22 12:14:11
64 bytes from 10.237.7.231: icmp_seq=2988 ttl=64 time=0.100 ms  2019-01-22 12:14:12
64 bytes from 10.237.7.231: icmp_seq=2989 ttl=64 time=0.093 ms  2019-01-22 12:14:13
64 bytes from 10.237.7.231: icmp_seq=2990 ttl=64 time=0.115 ms  2019-01-22 12:14:14
64 bytes from 10.237.7.231: icmp_seq=2991 ttl=64 time=0.057 ms  2019-01-22 12:14:15
64 bytes from 10.237.7.231: icmp_seq=2992 ttl=64 time=0.066 ms  2019-01-22 12:14:16
64 bytes from 10.237.7.231: icmp_seq=2993 ttl=64 time=0.065 ms  2019-01-22 12:14:17
64 bytes from 10.237.7.231: icmp_seq=2994 ttl=64 time=0.072 ms  2019-01-22 12:14:18
64 bytes from 10.237.7.231: icmp_seq=2995 ttl=64 time=0.052 ms  2019-01-22 12:14:19
64 bytes from 10.237.7.231: icmp_seq=2996 ttl=64 time=0.054 ms  2019-01-22 12:14:20
64 bytes from 10.237.7.231: icmp_seq=2997 ttl=64 time=0.068 ms  2019-01-22 12:14:21
64 bytes from 10.237.7.231: icmp_seq=2998 ttl=64 time=0.068 ms  2019-01-22 12:14:22
64 bytes from 10.237.7.231: icmp_seq=2999 ttl=64 time=0.075 ms  2019-01-22 12:14:23
64 bytes from 10.237.7.231: icmp_seq=3000 ttl=64 time=0.063 ms  2019-01-22 12:14:24
64 bytes from 10.237.7.231: icmp_seq=3001 ttl=64 time=0.085 ms  2019-01-22 12:14:25
64 bytes from 10.237.7.231: icmp_seq=3002 ttl=64 time=0.061 ms  2019-01-22 12:14:26
64 bytes from 10.237.7.231: icmp_seq=3003 ttl=64 time=0.061 ms  2019-01-22 12:14:27
64 bytes from 10.237.7.231: icmp_seq=3004 ttl=64 time=0.122 ms  2019-01-22 12:14:28
64 bytes from 10.237.7.231: icmp_seq=3005 ttl=64 time=0.047 ms  2019-01-22 12:14:29
64 bytes from 10.237.7.231: icmp_seq=3006 ttl=64 time=0.047 ms  2019-01-22 12:14:30
64 bytes from 10.237.7.231: icmp_seq=3007 ttl=64 time=0.055 ms  2019-01-22 12:14:31
64 bytes from 10.237.7.231: icmp_seq=3008 ttl=64 time=0.080 ms  2019-01-22 12:14:32
64 bytes from 10.237.7.231: icmp_seq=3009 ttl=64 time=0.087 ms  2019-01-22 12:14:33
64 bytes from 10.237.7.231: icmp_seq=3010 ttl=64 time=0.058 ms  2019-01-22 12:14:34
64 bytes from 10.237.7.231: icmp_seq=3011 ttl=64 time=0.071 ms  2019-01-22 12:14:35
64 bytes from 10.237.7.231: icmp_seq=3012 ttl=64 time=0.049 ms  2019-01-22 12:14:36
64 bytes from 10.237.7.231: icmp_seq=3013 ttl=64 time=0.056 ms  2019-01-22 12:14:37
64 bytes from 10.237.7.231: icmp_seq=3014 ttl=64 time=0.062 ms  2019-01-22 12:14:38
64 bytes from 10.237.7.231: icmp_seq=3015 ttl=64 time=0.069 ms  2019-01-22 12:14:39
64 bytes from 10.237.7.231: icmp_seq=3016 ttl=64 time=0.118 ms  2019-01-22 12:14:40
64 bytes from 10.237.7.231: icmp_seq=3017 ttl=64 time=0.067 ms  2019-01-22 12:14:41
64 bytes from 10.237.7.231: icmp_seq=3018 ttl=64 time=0.070 ms  2019-01-22 12:14:42
64 bytes from 10.237.7.231: icmp_seq=3019 ttl=64 time=0.077 ms  2019-01-22 12:14:43
64 bytes from 10.237.7.231: icmp_seq=3020 ttl=64 time=0.104 ms  2019-01-22 12:14:44
64 bytes from 10.237.7.231: icmp_seq=3021 ttl=64 time=0.069 ms  2019-01-22 12:14:45
64 bytes from 10.237.7.231: icmp_seq=3022 ttl=64 time=0.071 ms  2019-01-22 12:14:46
64 bytes from 10.237.7.231: icmp_seq=3023 ttl=64 time=0.063 ms  2019-01-22 12:14:47
64 bytes from 10.237.7.231: icmp_seq=3024 ttl=64 time=0.050 ms  2019-01-22 12:14:48
64 bytes from 10.237.7.231: icmp_seq=3025 ttl=64 time=0.055 ms  2019-01-22 12:14:49
64 bytes from 10.237.7.231: icmp_seq=3026 ttl=64 time=0.059 ms  2019-01-22 12:14:50
64 bytes from 10.237.7.231: icmp_seq=3027 ttl=64 time=0.063 ms  2019-01-22 12:14:51
64 bytes from 10.237.7.231: icmp_seq=3028 ttl=64 time=0.107 ms  2019-01-22 12:14:52
64 bytes from 10.237.7.231: icmp_seq=3029 ttl=64 time=0.055 ms  2019-01-22 12:14:53
64 bytes from 10.237.7.231: icmp_seq=3030 ttl=64 time=0.049 ms  2019-01-22 12:14:54
64 bytes from 10.237.7.231: icmp_seq=3031 ttl=64 time=0.054 ms  2019-01-22 12:14:55
64 bytes from 10.237.7.231: icmp_seq=3032 ttl=64 time=0.068 ms  2019-01-22 12:14:56
64 bytes from 10.237.7.231: icmp_seq=3033 ttl=64 time=0.074 ms  2019-01-22 12:14:57
64 bytes from 10.237.7.231: icmp_seq=3034 ttl=64 time=0.061 ms  2019-01-22 12:14:58
64 bytes from 10.237.7.231: icmp_seq=3035 ttl=64 time=0.059 ms  2019-01-22 12:14:59
64 bytes from 10.237.7.231: icmp_seq=3036 ttl=64 time=0.071 ms  2019-01-22 12:15:00
64 bytes from 10.237.7.231: icmp_seq=3037 ttl=64 time=0.065 ms  2019-01-22 12:15:01
64 bytes from 10.237.7.231: icmp_seq=3038 ttl=64 time=0.126 ms  2019-01-22 12:15:02



64 bytes from 10.237.7.231: icmp_seq=3476 ttl=64 time=0.098 ms  2019-01-22 12:22:20
64 bytes from 10.237.7.231: icmp_seq=3477 ttl=64 time=0.109 ms  2019-01-22 12:22:21
64 bytes from 10.237.7.231: icmp_seq=3478 ttl=64 time=0.115 ms  2019-01-22 12:22:22
64 bytes from 10.237.7.231: icmp_seq=3479 ttl=64 time=0.098 ms  2019-01-22 12:22:23
64 bytes from 10.237.7.231: icmp_seq=3480 ttl=64 time=0.144 ms  2019-01-22 12:22:24
64 bytes from 10.237.7.231: icmp_seq=3481 ttl=64 time=0.092 ms  2019-01-22 12:22:25
64 bytes from 10.237.7.231: icmp_seq=3482 ttl=64 time=0.105 ms  2019-01-22 12:22:26
64 bytes from 10.237.7.231: icmp_seq=3483 ttl=64 time=0.096 ms  2019-01-22 12:22:27
64 bytes from 10.237.7.231: icmp_seq=3484 ttl=64 time=0.119 ms  2019-01-22 12:22:28
64 bytes from 10.237.7.231: icmp_seq=3485 ttl=64 time=0.100 ms  2019-01-22 12:22:29
64 bytes from 10.237.7.231: icmp_seq=3486 ttl=64 time=0.087 ms  2019-01-22 12:22:30
64 bytes from 10.237.7.231: icmp_seq=3487 ttl=64 time=0.090 ms  2019-01-22 12:22:31
64 bytes from 10.237.7.231: icmp_seq=3488 ttl=64 time=0.115 ms  2019-01-22 12:22:32
64 bytes from 10.237.7.231: icmp_seq=3489 ttl=64 time=0.092 ms  2019-01-22 12:22:33
64 bytes from 10.237.7.231: icmp_seq=3490 ttl=64 time=0.104 ms  2019-01-22 12:22:34
64 bytes from 10.237.7.231: icmp_seq=3491 ttl=64 time=0.095 ms  2019-01-22 12:22:35

there are no packet loss

Wao. That looks like an issue with Grpc. Can you file a bug against Dgraph, and we’ll try and investigate why are Echos timing out?

OK, Thanks very much!

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.