Help with dgraph cluster

Devv · June 3, 2017, 10:27pm

First i ran the first on host A
./dgraph --groups "0,1" --idx 1 --my "172.16.6.15:8080" --bindall

Then attempted to connect to it from a second node, host B

./dgraph --groups "1" --idx 3 --peer "172.16.6.15:8529" --port 8529 -debugmode --bindall

all i get are errors:

Dgraph version   : v0.7.7
Commit SHA-1     : aebe813
Commit timestamp : 2017-06-02 14:33:22 +1000
Branch           : HEAD

Starting commit routine.
2017/06/03 22:21:36 worker.go:90: Worker listening at address: [::]:12345
2017/06/03 22:21:36 main.go:1019: grpc server started.
2017/06/03 22:21:36 main.go:1020: http server started.
2017/06/03 22:21:36 main.go:1021: Server listening on port 8529
2017/06/03 22:21:36 conn.go:99: While trying to connect to "172.16.6.15:8080", got error: rpc error: code = Unimplemented desc = unknown service protos.Worker
Unable to sync memberships. No valid connection
Last update raft index for membership information is zero. Syncing...
Unable to sync memberships. No valid connection
Last update raft index for membership information is zero. Syncing...
Unable to sync memberships. No valid connection

the connection seems fine from host B.

# telnet 172.16.6.15 8080 
Trying 172.16.6.15...
Connected to 172.16.6.15.
Escape character is '^]'.
^]
telnet> quit
Connection closed.

what is going on with : Unimplemented desc = unknown service protos.Worker ? is the cluster functional?
this is on linux (fedora 25), on go1.7.5.

mrjn · June 4, 2017, 10:35am

As the error message says, you don’t have a valid connection to the peer. If you look carefully, you set the peer as --peer "172.16.6.15:8529" for the second instance, while you ran the first instance on port 8080. The peer should be --peer ":8080".

Devv · June 4, 2017, 12:06pm

Seems i may have done a poor job at copy paste: here is the second attempt:

for the first node

./dgraph --groups "0,1" --idx 1 --my "172.16.6.15:8529" --bindall --port 8529 

Dgraph version   : v0.7.7
Commit SHA-1     : aebe813
Commit timestamp : 2017-06-02 14:33:22 +1000
Branch           : HEAD

Starting commit routine.
2017/06/04 12:04:15 worker.go:90: Worker listening at address: [::]:12345
2017/06/04 12:04:15 main.go:1019: grpc server started.
2017/06/04 12:04:15 main.go:1020: http server started.
2017/06/04 12:04:15 main.go:1021: Server listening on port 8529
Node with GroupID: 0, ID: 1
Node with GroupID: 1, ID: 1
Found hardstate: {Data:[] Metadata:{ConfState:{Nodes:[] XXX_unrecognized:[]} Index:0 Term:0 XXX_unrecognized:[]} XXX_unrecognized:[]}
Group 0 found 62 entries
Restarting node for group: 0
raft2017/06/04 12:04:15 INFO: 1 became follower at term 13
raft2017/06/04 12:04:15 INFO: newRaft 1 [peers: [], term: 13, commit: 62, applied: 0, lastindex: 62, lastterm: 13]
Found Snapshot: {Data:[9 1 0 0 0 0 0 0 0 16 1 26 16 49 55 50 46 49 54 46 54 46 49 53 58 56 53 50 57] Metadata:{ConfState:{Nodes:[1] XXX_unrecognized:[]} Index:9 Term:9 XXX_unrecognized:[]} XXX_unrecognized:[]}
Found hardstate: {Data:[9 1 0 0 0 0 0 0 0 16 1 26 16 49 55 50 46 49 54 46 54 46 49 53 58 56 53 50 57] Metadata:{ConfState:{Nodes:[1] XXX_unrecognized:[]} Index:9 Term:9 XXX_unrecognized:[]} XXX_unrecognized:[]}
Group 1 found 0 entries
Restarting node for group: 1
raft2017/06/04 12:04:15 INFO: 1 became follower at term 9
raft2017/06/04 12:04:15 INFO: newRaft 1 [peers: [1], term: 9, commit: 9, applied: 9, lastindex: 9, lastterm: 9]
2017/06/04 12:04:15 draft.go:444: group: 0 Addr: "172.16.6.15:8529" leader: true dead: false
----------------------------
====== APPLYING MEMBERSHIP UPDATE: {NodeId:1 Addr:172.16.6.15:8529 Leader:true RaftIdx:62}
----------------------------
Group: 0. List: [{NodeId:1 Addr:172.16.6.15:8529 Leader:true RaftIdx:62}]
2017/06/04 12:04:15 draft.go:444: group: 1 Addr: "172.16.6.15:8529" leader: false dead: false
----------------------------
====== APPLYING MEMBERSHIP UPDATE: {NodeId:1 Addr:172.16.6.15:8529 Leader:false RaftIdx:3}

For the second node:

./dgraph --groups "1" --idx 3 --peer "172.16.6.15:8529" --port 8529 -debugmode --bindall

Dgraph version   : v0.7.7
Commit SHA-1     : aebe813
Commit timestamp : 2017-06-02 14:33:22 +1000
Branch           : HEAD

Starting commit routine.
2017/06/04 12:05:19 worker.go:90: Worker listening at address: [::]:12345
2017/06/04 12:05:19 main.go:1019: grpc server started.
2017/06/04 12:05:19 main.go:1020: http server started.
2017/06/04 12:05:20 main.go:1021: Server listening on port 8529
2017/06/04 12:05:20 conn.go:99: While trying to connect to "172.16.6.15:8529", got error: rpc error: code = Unimplemented desc = unknown service protos.Worker
Unable to sync memberships. No valid connection
Last update raft index for membership information is zero. Syncing...
Unable to sync memberships. No valid connection
Last update raft index for membership information is zero. Syncing...
Unable to sync memberships. No valid connection
Last update raft index for membership information is zero. Syncing...
Unable to sync memberships. No valid connection
Last update raft index for membership information is zero. Syncing...
Unable to sync memberships. No valid connection
Last update raft index for membership information is zero. Syncing...
Unable to sync memberships. No valid connection
Last update raft index for membership information is zero. Syncing...
Unable to sync memberships. No valid connection
Last update raft index for membership information is zero. Syncing...
Unable to sync memberships. No valid connection
Last update raft index for membership information is zero. Syncing...

whats with the lines:

conn.go:99: While trying to connect to "172.16.6.15:8529", got error: rpc error: code = Unimplemented desc = unknown service protos.Worker and Unable to sync memberships. No valid connection?

mrjn · June 5, 2017, 3:44am

You didn’t set the my flag for the second node. The port is the same as the first one, so, I presume they are on different machines. The my flag is important, so the node can tell others how to connect to itself.

Devv · June 5, 2017, 9:19am

Still no progress with that:

./dgraph --groups "1" --idx 3 --peer "172.16.6.15:8529" --port 8529 -debugmode --bindall --my "172.16.6.23:8529"  

Dgraph version   : v0.7.7
Commit SHA-1     : aebe813
Commit timestamp : 2017-06-02 14:33:22 +1000
Branch           : HEAD

Starting commit routine.
2017/06/05 09:17:21 worker.go:90: Worker listening at address: [::]:12345
2017/06/05 09:17:21 main.go:1019: grpc server started.
2017/06/05 09:17:21 main.go:1020: http server started.
2017/06/05 09:17:21 main.go:1021: Server listening on port 8529
2017/06/05 09:17:21 conn.go:99: While trying to connect to "172.16.6.15:8529", got error: rpc error: code = Unimplemented desc = unknown service protos.Worker
Unable to sync memberships. No valid connection
Last update raft index for membership information is zero. Syncing...
Unable to sync memberships. No valid connection
Last update raft index for membership information is zero. Syncing...
Unable to sync memberships. No valid connection
Last update raft index for membership information is zero. Syncing...

Im not sure whats up, but the ports are open and reachable from the second node:

# telnet 172.16.6.15 8529 
Trying 172.16.6.15...
Connected to 172.16.6.15.
Escape character is '^]'.
^]
telnet> 
[root@d dgraph]# telnet 172.16.6.15  12345 
Trying 172.16.6.15...
Connected to 172.16.6.15.
Escape character is '^]'.
^]
telnet>

mrjn · June 5, 2017, 12:29pm

There’s a different problem now. It’s that there is no protos.Worker. @pawan can you have a look at this?

pawan · June 6, 2017, 5:44am

@Devv The error is a bit misleading. The flags seem to have incorrect value.

First node

./dgraph --groups "0,1" --idx 1 --my "172.16.6.15:12345" --bindall

Second node

./dgraph --groups "1" --idx 3 --peer "172.16.6.15:12345" --port 8529 --bindall

The important thing is that the port in my and peer is the internal workerport (having a default value of 12345) which is different from the port for HTTP communication (passed using --port flag)

I am assuming that you are running the two Dgraph servers on different machines? If not then you would need to provide a different value to the workerport flag for the second node like

Second node

./dgraph --groups "1" --idx 3 --peer "172.16.6.15:12345" --port 8529 --workerport 12346 --bindall

Devv · June 6, 2017, 9:19am

Pawan,
yes im runing them on two separate vms,
thanks, i tried that and the errors are gone,
Is there another way to set the worker ports IP:PORT combination? without the --bindall it binds to localhost, even when --my is given a non localhost IP::PORT combination.

pawan · June 7, 2017, 4:52am

There is no way to pass in the ip in the workerport flag. I think we should bind to the given address if you pass in one through a --my flag and --bindall is not set. I will create an issue.

system · July 7, 2017, 4:52am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Error - Unable to join cluster via dgraphzero Users	3	801	January 17, 2018
Dgraph runs into a error loop and freezes the host Users	20	2096	February 21, 2018
While doing live loader import i'm getting below error and server is not connecting Dgraph dgraph , untagged	2	758	December 28, 2020
Cannot connect to dgraph in VM from Host Users	13	2291	February 21, 2018
Unable to join cluster via dgraphzero after upgrading to 1.0.3 Users	6	735	April 8, 2018

Help with dgraph cluster

Related Topics