Help with dgraph cluster


(Devon) #1

First i ran the first on host A
./dgraph --groups "0,1" --idx 1 --my "172.16.6.15:8080" --bindall

Then attempted to connect to it from a second node, host B

./dgraph --groups "1" --idx 3 --peer "172.16.6.15:8529" --port 8529 -debugmode --bindall

all i get are errors:

Dgraph version   : v0.7.7
Commit SHA-1     : aebe813
Commit timestamp : 2017-06-02 14:33:22 +1000
Branch           : HEAD

Starting commit routine.
2017/06/03 22:21:36 worker.go:90: Worker listening at address: [::]:12345
2017/06/03 22:21:36 main.go:1019: grpc server started.
2017/06/03 22:21:36 main.go:1020: http server started.
2017/06/03 22:21:36 main.go:1021: Server listening on port 8529
2017/06/03 22:21:36 conn.go:99: While trying to connect to "172.16.6.15:8080", got error: rpc error: code = Unimplemented desc = unknown service protos.Worker
Unable to sync memberships. No valid connection
Last update raft index for membership information is zero. Syncing...
Unable to sync memberships. No valid connection
Last update raft index for membership information is zero. Syncing...
Unable to sync memberships. No valid connection

the connection seems fine from host B.

# telnet 172.16.6.15 8080 
Trying 172.16.6.15...
Connected to 172.16.6.15.
Escape character is '^]'.
^]
telnet> quit
Connection closed.

what is going on with : Unimplemented desc = unknown service protos.Worker ? is the cluster functional?
this is on linux (fedora 25), on go1.7.5.


(Manish R Jain) #2

As the error message says, you don’t have a valid connection to the peer. If you look carefully, you set the peer as --peer "172.16.6.15:8529" for the second instance, while you ran the first instance on port 8080. The peer should be --peer ":8080".


(Devon) #3

Seems i may have done a poor job at copy paste: here is the second attempt:

for the first node

./dgraph --groups "0,1" --idx 1 --my "172.16.6.15:8529" --bindall --port 8529 

Dgraph version   : v0.7.7
Commit SHA-1     : aebe813
Commit timestamp : 2017-06-02 14:33:22 +1000
Branch           : HEAD

Starting commit routine.
2017/06/04 12:04:15 worker.go:90: Worker listening at address: [::]:12345
2017/06/04 12:04:15 main.go:1019: grpc server started.
2017/06/04 12:04:15 main.go:1020: http server started.
2017/06/04 12:04:15 main.go:1021: Server listening on port 8529
Node with GroupID: 0, ID: 1
Node with GroupID: 1, ID: 1
Found hardstate: {Data:[] Metadata:{ConfState:{Nodes:[] XXX_unrecognized:[]} Index:0 Term:0 XXX_unrecognized:[]} XXX_unrecognized:[]}
Group 0 found 62 entries
Restarting node for group: 0
raft2017/06/04 12:04:15 INFO: 1 became follower at term 13
raft2017/06/04 12:04:15 INFO: newRaft 1 [peers: [], term: 13, commit: 62, applied: 0, lastindex: 62, lastterm: 13]
Found Snapshot: {Data:[9 1 0 0 0 0 0 0 0 16 1 26 16 49 55 50 46 49 54 46 54 46 49 53 58 56 53 50 57] Metadata:{ConfState:{Nodes:[1] XXX_unrecognized:[]} Index:9 Term:9 XXX_unrecognized:[]} XXX_unrecognized:[]}
Found hardstate: {Data:[9 1 0 0 0 0 0 0 0 16 1 26 16 49 55 50 46 49 54 46 54 46 49 53 58 56 53 50 57] Metadata:{ConfState:{Nodes:[1] XXX_unrecognized:[]} Index:9 Term:9 XXX_unrecognized:[]} XXX_unrecognized:[]}
Group 1 found 0 entries
Restarting node for group: 1
raft2017/06/04 12:04:15 INFO: 1 became follower at term 9
raft2017/06/04 12:04:15 INFO: newRaft 1 [peers: [1], term: 9, commit: 9, applied: 9, lastindex: 9, lastterm: 9]
2017/06/04 12:04:15 draft.go:444: group: 0 Addr: "172.16.6.15:8529" leader: true dead: false
----------------------------
====== APPLYING MEMBERSHIP UPDATE: {NodeId:1 Addr:172.16.6.15:8529 Leader:true RaftIdx:62}
----------------------------
Group: 0. List: [{NodeId:1 Addr:172.16.6.15:8529 Leader:true RaftIdx:62}]
2017/06/04 12:04:15 draft.go:444: group: 1 Addr: "172.16.6.15:8529" leader: false dead: false
----------------------------
====== APPLYING MEMBERSHIP UPDATE: {NodeId:1 Addr:172.16.6.15:8529 Leader:false RaftIdx:3}

For the second node:

./dgraph --groups "1" --idx 3 --peer "172.16.6.15:8529" --port 8529 -debugmode --bindall

Dgraph version   : v0.7.7
Commit SHA-1     : aebe813
Commit timestamp : 2017-06-02 14:33:22 +1000
Branch           : HEAD

Starting commit routine.
2017/06/04 12:05:19 worker.go:90: Worker listening at address: [::]:12345
2017/06/04 12:05:19 main.go:1019: grpc server started.
2017/06/04 12:05:19 main.go:1020: http server started.
2017/06/04 12:05:20 main.go:1021: Server listening on port 8529
2017/06/04 12:05:20 conn.go:99: While trying to connect to "172.16.6.15:8529", got error: rpc error: code = Unimplemented desc = unknown service protos.Worker
Unable to sync memberships. No valid connection
Last update raft index for membership information is zero. Syncing...
Unable to sync memberships. No valid connection
Last update raft index for membership information is zero. Syncing...
Unable to sync memberships. No valid connection
Last update raft index for membership information is zero. Syncing...
Unable to sync memberships. No valid connection
Last update raft index for membership information is zero. Syncing...
Unable to sync memberships. No valid connection
Last update raft index for membership information is zero. Syncing...
Unable to sync memberships. No valid connection
Last update raft index for membership information is zero. Syncing...
Unable to sync memberships. No valid connection
Last update raft index for membership information is zero. Syncing...
Unable to sync memberships. No valid connection
Last update raft index for membership information is zero. Syncing...

whats with the lines:

conn.go:99: While trying to connect to "172.16.6.15:8529", got error: rpc error: code = Unimplemented desc = unknown service protos.Worker and Unable to sync memberships. No valid connection?


(Manish R Jain) #4

You didn’t set the my flag for the second node. The port is the same as the first one, so, I presume they are on different machines. The my flag is important, so the node can tell others how to connect to itself.


(Devon) #5

Still no progress with that:

./dgraph --groups "1" --idx 3 --peer "172.16.6.15:8529" --port 8529 -debugmode --bindall --my "172.16.6.23:8529"  

Dgraph version   : v0.7.7
Commit SHA-1     : aebe813
Commit timestamp : 2017-06-02 14:33:22 +1000
Branch           : HEAD

Starting commit routine.
2017/06/05 09:17:21 worker.go:90: Worker listening at address: [::]:12345
2017/06/05 09:17:21 main.go:1019: grpc server started.
2017/06/05 09:17:21 main.go:1020: http server started.
2017/06/05 09:17:21 main.go:1021: Server listening on port 8529
2017/06/05 09:17:21 conn.go:99: While trying to connect to "172.16.6.15:8529", got error: rpc error: code = Unimplemented desc = unknown service protos.Worker
Unable to sync memberships. No valid connection
Last update raft index for membership information is zero. Syncing...
Unable to sync memberships. No valid connection
Last update raft index for membership information is zero. Syncing...
Unable to sync memberships. No valid connection
Last update raft index for membership information is zero. Syncing...

Im not sure whats up, but the ports are open and reachable from the second node:

# telnet 172.16.6.15 8529 
Trying 172.16.6.15...
Connected to 172.16.6.15.
Escape character is '^]'.
^]
telnet> 
[root@d dgraph]# telnet 172.16.6.15  12345 
Trying 172.16.6.15...
Connected to 172.16.6.15.
Escape character is '^]'.
^]
telnet>

(Manish R Jain) #6

There’s a different problem now. It’s that there is no protos.Worker. @pawan can you have a look at this?


(Pawan Rawal) #7

@Devv The error is a bit misleading. The flags seem to have incorrect value.

First node

./dgraph --groups "0,1" --idx 1 --my "172.16.6.15:12345" --bindall

Second node

./dgraph --groups "1" --idx 3 --peer "172.16.6.15:12345" --port 8529 --bindall

The important thing is that the port in my and peer is the internal workerport (having a default value of 12345) which is different from the port for HTTP communication (passed using --port flag)

I am assuming that you are running the two Dgraph servers on different machines? If not then you would need to provide a different value to the workerport flag for the second node like

Second node

./dgraph --groups "1" --idx 3 --peer "172.16.6.15:12345" --port 8529 --workerport 12346 --bindall

(Devon) #8

Pawan,
yes im runing them on two separate vms,
thanks, i tried that and the errors are gone,
Is there another way to set the worker ports IP:PORT combination? without the --bindall it binds to localhost, even when --my is given a non localhost IP::PORT combination.


(Pawan Rawal) #9

There is no way to pass in the ip in the workerport flag. I think we should bind to the given address if you pass in one through a --my flag and --bindall is not set. I will create an issue.


(system) #10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.