Dgraph runs into a error loop and freezes the host

Every dgraph zero and server errors the same message repetitively, even after restart the cluster with existing database.

Dgraph zero:
raft.go:1070: INFO: 1 no leader at term 7; dropping index reading msg

dgraph server:
Unable to sync memberships. Error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
groups.go:673: Error in oracle delta stream. Error: rpc error: code = Unknown desc = Node is no longer leader.

Can you file a Github issue with a way to replicate this?

Sorry, I couldn’t find a nice way to reproduce this.
Basically, I ran all dgraph instances in a kubernetes pod. There are 5 dgraph zeros and 20 dgraph servers in the pod. Each of them runs in its own container. Each server and zero has an unique group ID assigned via flag idx

After ingesting a large amount of data ~31GB in dgraph database, I tried to make a copy of the database:

  1. stop the running dgraph pod.
  2. copy the database.
  3. restart the pod.

I saw following logs with spiking CPU usage:
similar logs for all dgraph zero:

Setting up grpc listener at: 0.0.0.0:7080
Setting up http listener at: 0.0.0.0:8080
2018/01/15 22:41:37 node.go:233: Found Snapshot, Metadata: {ConfState:{Nodes:[1 2 3 4 5] XXX_unrecognized:[]} Index:27053 Term:3 XXX_unrecognized:[]}
2018/01/15 22:41:37 node.go:248: Found hardstate: {Term:3 Vote:1 Commit:27881 XXX_unrecognized:[]}
2018/01/15 22:41:37 node.go:260: Group 0 found 828 entries
2018/01/15 22:41:37 raft.go:400: Restarting node for dgraphzero
2018/01/15 22:41:37 pool.go:167: Echo error from localhost:7104. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7104
2018/01/15 22:41:37 pool.go:167: Echo error from localhost:7098. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7098
2018/01/15 22:41:37 pool.go:167: Echo error from localhost:7094. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7094
2018/01/15 22:41:37 pool.go:167: Echo error from localhost:7102. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7102
2018/01/15 22:41:37 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7092
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7093
2018/01/15 22:41:37 pool.go:167: Echo error from localhost:7100. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7100
2018/01/15 22:41:37 pool.go:167: Echo error from localhost:7108. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7108
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7107
2018/01/15 22:41:37 pool.go:167: Echo error from localhost:7099. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7099
2018/01/15 22:41:37 pool.go:167: Echo error from localhost:7096. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7096
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7103
2018/01/15 22:41:37 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7090
2018/01/15 22:41:37 pool.go:167: Echo error from localhost:7105. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7105
2018/01/15 22:41:37 pool.go:167: Echo error from localhost:7095. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7095
2018/01/15 22:41:37 pool.go:167: Echo error from localhost:7101. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7101
2018/01/15 22:41:37 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7091
2018/01/15 22:41:37 pool.go:167: Echo error from localhost:7097. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7097
2018/01/15 22:41:37 pool.go:167: Echo error from localhost:7106. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7106
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7109
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7081
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7082
2018/01/15 22:41:37 pool.go:167: Echo error from localhost:7083. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7083
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7084
2018/01/15 22:41:37 raft.go:567: INFO: 1 became follower at term 3
2018/01/15 22:41:37 raft.go:315: INFO: newRaft 1 [peers: [1,2,3,4,5], term: 3, commit: 27881, applied: 27053, lastindex: 27881, lastterm: 3]
Running Dgraph zero...
2018/01/15 22:41:38 zero.go:316: Got connection request: id:18 addr:"localhost:7098" 
2018/01/15 22:41:38 zero.go:412: Connected
2018/01/15 22:41:38 zero.go:316: Got connection request: id:27 addr:"localhost:7107" 
2018/01/15 22:41:38 zero.go:412: Connected
2018/01/15 22:41:38 zero.go:316: Got connection request: id:12 addr:"localhost:7092" 
2018/01/15 22:41:38 zero.go:412: Connected
2018/01/15 22:41:38 zero.go:316: Got connection request: id:29 addr:"localhost:7109" 
2018/01/15 22:41:38 zero.go:412: Connected
2018/01/15 22:41:38 zero.go:316: Got connection request: id:21 addr:"localhost:7101" 
2018/01/15 22:41:38 zero.go:412: Connected
2018/01/15 22:41:38 zero.go:316: Got connection request: id:23 addr:"localhost:7103" 
2018/01/15 22:41:38 zero.go:412: Connected
2018/01/15 22:41:38 zero.go:316: Got connection request: id:22 addr:"localhost:7102" 
2018/01/15 22:41:38 zero.go:412: Connected
2018/01/15 22:41:38 zero.go:316: Got connection request: id:19 addr:"localhost:7099" 
2018/01/15 22:41:38 zero.go:412: Connected
2018/01/15 22:41:38 zero.go:316: Got connection request: id:10 addr:"localhost:7090" 
2018/01/15 22:41:38 zero.go:412: Connected
2018/01/15 22:41:38 zero.go:316: Got connection request: id:24 addr:"localhost:7104" 
2018/01/15 22:41:38 zero.go:412: Connected
2018/01/15 22:41:39 zero.go:316: Got connection request: id:17 addr:"localhost:7097" 
2018/01/15 22:41:39 zero.go:412: Connected
2018/01/15 22:41:39 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
2018/01/15 22:41:39 zero.go:316: Got connection request: id:13 addr:"localhost:7093" 
2018/01/15 22:41:39 zero.go:412: Connected
2018/01/15 22:41:39 zero.go:316: Got connection request: id:14 addr:"localhost:7094" 
2018/01/15 22:41:39 zero.go:412: Connected
2018/01/15 22:41:39 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
2018/01/15 22:41:39 zero.go:316: Got connection request: id:15 addr:"localhost:7095" 
2018/01/15 22:41:39 zero.go:412: Connected
2018/01/15 22:41:39 zero.go:316: Got connection request: id:20 addr:"localhost:7100" 
2018/01/15 22:41:39 zero.go:412: Connected
2018/01/15 22:41:39 zero.go:316: Got connection request: id:16 addr:"localhost:7096" 
2018/01/15 22:41:39 zero.go:412: Connected
2018/01/15 22:41:39 zero.go:316: Got connection request: id:25 addr:"localhost:7105" 
2018/01/15 22:41:39 zero.go:412: Connected
2018/01/15 22:41:40 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
2018/01/15 22:41:40 zero.go:316: Got connection request: id:28 addr:"localhost:7108" 
2018/01/15 22:41:40 zero.go:412: Connected
2018/01/15 22:41:40 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
2018/01/15 22:41:41 zero.go:316: Got connection request: id:11 addr:"localhost:7091" 
2018/01/15 22:41:41 zero.go:412: Connected
2018/01/15 22:41:41 zero.go:316: Got connection request: id:26 addr:"localhost:7106" 
2018/01/15 22:41:41 zero.go:412: Connected
2018/01/15 22:41:42 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
2018/01/15 22:41:43 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
2018/01/15 22:41:44 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
2018/01/15 22:41:45 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
2018/01/15 22:41:46 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
2018/01/15 22:41:46 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
2018/01/15 22:41:46 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
2018/01/15 22:41:46 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
2018/01/15 22:41:46 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
2018/01/15 22:41:47 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
2018/01/15 22:41:47 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg

Similar logs for all dgraph server

2018/01/15 22:41:40 groups.go:457: Unable to sync memberships. Error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/01/15 22:41:40 groups.go:673: Error in oracle delta stream. Error: rpc error: code = Unknown desc = Node is no longer leader.
2018/01/15 22:41:41 groups.go:457: Unable to sync memberships. Error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/01/15 22:41:41 groups.go:673: Error in oracle delta stream. Error: rpc error: code = Unknown desc = Node is no longer leader.
2018/01/15 22:41:42 raft.go:749: INFO: e is starting a new election at term 3
2018/01/15 22:41:42 raft.go:580: INFO: e became candidate at term 4
2018/01/15 22:41:42 raft.go:664: INFO: e received MsgVoteResp from e at term 4
2018/01/15 22:41:42 raft.go:621: INFO: e became leader at term 4
2018/01/15 22:41:42 node.go:301: INFO: raft.node: e elected leader e at term 4
2018/01/15 22:41:42 groups.go:457: Unable to sync memberships. Error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/01/15 22:41:42 groups.go:673: Error in oracle delta stream. Error: rpc error: code = Unknown desc = Node is no longer leader.
2018/01/15 22:41:43 groups.go:457: Unable to sync memberships. Error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/01/15 22:41:43 groups.go:673: Error in oracle delta stream. Error: rpc error: code = Unknown desc = Node is no longer leader.
2018/01/15 22:41:44 groups.go:457: Unable to sync memberships. Error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/01/15 22:41:44 groups.go:673: Error in oracle delta stream. Error: rpc error: code = Unknown desc = Node is no longer leader.
2018/01/15 22:41:45 groups.go:457: Unable to sync memberships. Error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/01/15 22:41:45 groups.go:673: Error in oracle delta stream. Error: rpc error: code = Unknown desc = Node is no longer leader.
2018/01/15 22:41:46 groups.go:457: Unable to sync memberships. Error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/01/15 22:41:46 groups.go:673: Error in oracle delta stream. Error: rpc error: code = Unknown desc = Node is no longer leader.
2018/01/15 22:41:47 groups.go:457: Unable to sync memberships. Error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/01/15 22:41:47 groups.go:673: Error in oracle delta stream. Error: rpc error: code = Unknown desc = Node is no longer leader.

When I test the similar setup locally in terminal with much smaller scale (3 zeros and 3 servers) and data set (~50MB)
I noticed when a dgraph zero starts the term increases everytime and the peers array is empty :
INFO: newRaft 1 [peers: [], term: 45, commit: 171, applied: 0, lastindex: 171, lastterm: 45]
whereas in the kubernetes pod the peers array is [1,2,3,4,5] and the term does not increases :
INFO: newRaft 1 [peers: [1,2,3,4,5], term: 3, commit: 27881, applied: 27053, lastindex: 27881, lastterm: 3]

I managed to partially reproduce this issue:
dgraph version: v1.0.1

  1. Set up a cluster locally with 3 dgraph zeros, 3 dgraph servers, and dgraph ratel:
dgraph zero --my="localhost:7080" --wal=zw --bindall=true
dgraph zero --my="localhost:7081" --peer "localhost:7080" --wal=zw --port_offset=1 --bindall=true --idx 2
dgraph zero --my="localhost:7082" --peer "localhost:7080" --wal=zw --port_offset=2 --bindall=true --idx 3
dgraph server --zero="localhost:7080" --my="localhost:7090" --bindall=true --memory_mb=2048 --port_offset=10 --idx=10
dgraph server --zero="localhost:7080" --my="localhost:7091" --bindall=true --memory_mb=2048 --port_offset=11 --idx=11
dgraph server --zero="localhost:7080" --my="localhost:7092" --bindall=true --memory_mb=2048 --port_offset=12 --idx=12
dgraph-ratel -p 8085
  1. Set up the schema in the web UI:
name: string @index(exact) .
type: string @index(exact) .
kind: string @index(exact) .
  1. Obtain data via a small go program:
package main

import (
	"flag"
	"fmt"
	"math/rand"
	"strconv"
	"time"
)

var horizon int
var density float64

func main() {
	density = *flag.Float64("density", 0.01, "Roughly density of func nodes in the tree.")
	d := flag.Int("depth", 11, "Depth of the tree.")
	o := flag.Int("order", 3, "Order of the tree - ie, 2 gives a binary tree.")
	flag.Parse()
	rand.Seed(time.Now().UnixNano())
	fmt.Println(`
 {
    set {`)
	WriteNaryTree(0, *d-1, *o, "root")
	fmt.Println(`
	}
}`[1:])
}

// WriteNaryTree writes quads forming a tree of depth and order n.
func WriteNaryTree(currentDepth, d, n int, parent string) {
	horizon++
	id := strconv.Itoa(horizon)
	fmt.Printf("_:%s <name> \"%s\" .\n", id, id)
	fmt.Printf("_:%s <type> \"foo\" .\n", id)
	if parent != "" {
		fmt.Printf("_:%s <kid> _:%s .\n", parent, id)
	}

	if rand.Float64() < density {
		fmt.Printf("_:%s <kind> \"%s\" .\n", id, "bar")
	}

	if d > currentDepth {
		for i := 0; i < n; i++ {
			WriteNaryTree(currentDepth+1, d, n, id)
		}
	}
}
  1. Use data output from the go program to run the mutation in the web UI
  2. Wait for the rebalancing to complete.
  3. Terminate all dgraph zeros and dgraph servers
  4. Restart dgraph servers then dgraph zerors
  5. Might need to repeat steps 6-7 for a few times before one or more dgraph termite themselves after following logs:
2018/01/16 12:23:38 gRPC server started.  Listening on port 9090
2018/01/16 12:23:38 HTTP server started.  Listening on port 8090
2018/01/16 12:23:38 worker.go:99: Worker listening at address: [::]:7090
2018/01/16 12:23:38 pool.go:118: == CONNECT ==> Setting localhost:7080
2018/01/16 12:23:38 groups.go:117: Connected to group zero. Connection state: member:<id:10 addr:"localhost:7090" > state:<counter:180 groups:<key:1 value:<members:<key:10 value:<id:10 group_id:1 addr:"localhost:7090" leader:true last_update:1516058396 > > tablets:<key:"_dummy_" value:<group_id:1 predicate:"_dummy_" force:true space:19 > > tablets:<key:"_predicate_" value:<group_id:1 predicate:"_predicate_" > > tablets:<key:"type" value:<group_id:1 predicate:"type" space:6898672 > > > > groups:<key:2 value:<members:<key:11 value:<id:11 group_id:2 addr:"localhost:7091" leader:true last_update:1516058434 > > tablets:<key:"name" value:<group_id:2 predicate:"name" force:true space:8923661 > > > > groups:<key:3 value:<members:<key:12 value:<id:12 group_id:3 addr:"localhost:7092" leader:true last_update:1516057762 > > tablets:<key:"kid" value:<group_id:3 predicate:"kid" space:1154035 > > tablets:<key:"kind" value:<group_id:3 predicate:"kind" space:43273 > > > > zeros:<key:1 value:<id:1 addr:"localhost:7080" > > zeros:<key:2 value:<id:2 addr:"localhost:7081" leader:true > > zeros:<key:3 value:<id:3 addr:"localhost:7082" > > maxLeaseId:98574 maxTxnTs:10000 > 
2018/01/16 12:23:38 pool.go:118: == CONNECT ==> Setting localhost:7081
2018/01/16 12:23:38 pool.go:118: == CONNECT ==> Setting localhost:7082
2018/01/16 12:23:38 pool.go:118: == CONNECT ==> Setting localhost:7091
2018/01/16 12:23:38 pool.go:118: == CONNECT ==> Setting localhost:7092
2018/01/16 12:23:38 draft.go:140: Node ID: 10 with GroupID: 1
2018/01/16 12:23:38 node.go:233: Found Snapshot, Metadata: {ConfState:{Nodes:[10] XXX_unrecognized:[]} Index:14 Term:4 XXX_unrecognized:[]}
2018/01/16 12:23:38 node.go:248: Found hardstate: {Term:7 Vote:10 Commit:19 XXX_unrecognized:[]}
2018/01/16 12:23:38 node.go:260: Group 1 found 5 entries
2018/01/16 12:23:38 draft.go:657: Restarting node for group: 1
2018/01/16 12:23:38 raft.go:567: INFO: a became follower at term 7
2018/01/16 12:23:38 raft.go:315: INFO: newRaft a [peers: [a], term: 7, commit: 19, applied: 14, lastindex: 19, lastterm: 7]
2018/01/16 12:23:40 raft.go:749: INFO: a is starting a new election at term 7
2018/01/16 12:23:40 raft.go:580: INFO: a became candidate at term 8
2018/01/16 12:23:40 raft.go:664: INFO: a received MsgVoteResp from a at term 8
2018/01/16 12:23:40 raft.go:621: INFO: a became leader at term 8
2018/01/16 12:23:40 node.go:301: INFO: raft.node: a elected leader a at term 8
2018/01/16 12:24:23 groups.go:673: Error in oracle delta stream. Error: rpc error: code = Unknown desc = Server is being shut down.
2018/01/16 12:24:24 groups.go:665: Error while calling Oracle rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/16 12:24:25 groups.go:665: Error while calling Oracle rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/16 12:24:26 groups.go:673: Error in oracle delta stream. Error: rpc error: code = Unknown desc = Node is no longer leader.
2018/01/16 12:24:26 groups.go:457: Unable to sync memberships. Error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/01/16 12:24:34 groups.go:673: Error in oracle delta stream. Error: rpc error: code = Unknown desc = Node is no longer leader.
2018/01/16 12:24:35 groups.go:457: Unable to sync memberships. Error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/01/16 12:24:35 groups.go:673: Error in oracle delta stream. Error: rpc error: code = Unknown desc = Node is no longer leader.
2018/01/16 12:24:36 groups.go:457: Unable to sync memberships. Error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/01/16 12:24:36 groups.go:673: Error in oracle delta stream. Error: rpc error: code = Unknown desc = Node is no longer leader.
2018/01/16 12:24:37 groups.go:457: Unable to sync memberships. Error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/01/16 12:24:37 groups.go:673: Error in oracle delta stream. Error: rpc error: code = Unknown desc = Node is no longer leader.
2018/01/16 12:24:38 groups.go:457: Unable to sync memberships. Error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/01/16 12:26:34 unexpected meta: 0

Hey @jzhu077

The log.Fatal caused by unexpected meta: 0 is a known issue and has been fixed in master. We will be doing a release today which will have the fix.

We are looking at the other errors.

@jzhu077 Does leader election finally happen after all the dropping index reading msg?

Also, how do you copy the database? Are you making sure that Zero and Dgraph Servers restart with the same directories (p and w for Server and zw for Zero)?

No. It constantly logs the same error message.
I copied the database(p,w and zw) via cp from the current location to the backup directory, so I assume the database stays as it is during the process.

Also, I was wondering which one is the expected behaviour?
INFO: newRaft 1 [peers: [], term: 45, commit: 171, applied: 0, lastindex: 171, lastterm: 45]
or
INFO: newRaft 1 [peers: [1,2,3,4,5], term: 3, commit: 27881, applied: 27053, lastindex: 27881, lastterm: 3]

Should peers be [1,2,3] if I have 3 dgraph zeros or it should be empty at the time it starts?

Empty peers shouldn’t happen, can you please share the logs of zero when you are running locally.

zero 1 logs

Setting up grpc listener at: 0.0.0.0:7080
Setting up http listener at: 0.0.0.0:8080
2018/01/17 09:05:00 node.go:248: Found hardstate: {Term:448 Vote:1 Commit:221 XXX_unrecognized:[]}
2018/01/17 09:05:00 node.go:260: Group 0 found 221 entries
2018/01/17 09:05:00 raft.go:400: Restarting node for dgraphzero
2018/01/17 09:05:00 raft.go:567: INFO: 1 became follower at term 448
2018/01/17 09:05:00 raft.go:315: INFO: newRaft 1 [peers: [], term: 448, commit: 221, applied: 0, lastindex: 221, lastterm: 54]
Running Dgraph zero...
2018/01/17 09:05:00 node.go:127: Setting conf state to nodes:1 
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7081. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:118: == CONNECT ==> Setting localhost:7081
2018/01/17 09:05:00 node.go:127: Setting conf state to nodes:1 nodes:2 
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7082. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:118: == CONNECT ==> Setting localhost:7082
2018/01/17 09:05:00 node.go:127: Setting conf state to nodes:1 nodes:2 nodes:3 
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:118: == CONNECT ==> Setting localhost:7091
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:118: == CONNECT ==> Setting localhost:7092
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:118: == CONNECT ==> Setting localhost:7090
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:03 raft.go:749: INFO: 1 is starting a new election at term 448
2018/01/17 09:05:03 raft.go:580: INFO: 1 became candidate at term 449
2018/01/17 09:05:03 raft.go:664: INFO: 1 received MsgVoteResp from 1 at term 449
2018/01/17 09:05:03 raft.go:651: INFO: 1 [logterm: 54, index: 221] sent MsgVote request to 2 at term 449
2018/01/17 09:05:03 raft.go:651: INFO: 1 [logterm: 54, index: 221] sent MsgVote request to 3 at term 449
2018/01/17 09:05:03 node.go:191: 		SENDING: MsgVote 1-->2
2018/01/17 09:05:03 node.go:191: 		SENDING: MsgVote 1-->3
2018/01/17 09:05:03 node.go:324: No healthy connection found to node Id: 3, err: Unhealthy connection
2018/01/17 09:05:03 node.go:324: No healthy connection found to node Id: 2, err: Unhealthy connection
2018/01/17 09:05:07 raft.go:749: INFO: 1 is starting a new election at term 449
2018/01/17 09:05:07 raft.go:580: INFO: 1 became candidate at term 450
2018/01/17 09:05:07 raft.go:664: INFO: 1 received MsgVoteResp from 1 at term 450
2018/01/17 09:05:07 raft.go:651: INFO: 1 [logterm: 54, index: 221] sent MsgVote request to 2 at term 450
2018/01/17 09:05:07 raft.go:651: INFO: 1 [logterm: 54, index: 221] sent MsgVote request to 3 at term 450
2018/01/17 09:05:07 node.go:191: 		SENDING: MsgVote 1-->2
2018/01/17 09:05:07 node.go:191: 		SENDING: MsgVote 1-->3
2018/01/17 09:05:07 node.go:324: No healthy connection found to node Id: 3, err: Unhealthy connection
2018/01/17 09:05:07 node.go:324: No healthy connection found to node Id: 2, err: Unhealthy connection
2018/01/17 09:05:10 raft.go:749: INFO: 1 is starting a new election at term 450
2018/01/17 09:05:10 raft.go:580: INFO: 1 became candidate at term 451
2018/01/17 09:05:10 raft.go:664: INFO: 1 received MsgVoteResp from 1 at term 451
2018/01/17 09:05:10 raft.go:651: INFO: 1 [logterm: 54, index: 221] sent MsgVote request to 2 at term 451
2018/01/17 09:05:10 raft.go:651: INFO: 1 [logterm: 54, index: 221] sent MsgVote request to 3 at term 451
2018/01/17 09:05:10 node.go:191: 		SENDING: MsgVote 1-->2
2018/01/17 09:05:10 node.go:191: 		SENDING: MsgVote 1-->3
2018/01/17 09:05:10 node.go:324: No healthy connection found to node Id: 3, err: Unhealthy connection
2018/01/17 09:05:10 node.go:324: No healthy connection found to node Id: 2, err: Unhealthy connection
2018/01/17 09:05:10 oracle.go:375: No healthy connection found to leader of group 3
2018/01/17 09:05:10 pool.go:167: Echo error from localhost:7081. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 raft.go:749: INFO: 1 is starting a new election at term 451
2018/01/17 09:05:12 raft.go:580: INFO: 1 became candidate at term 452
2018/01/17 09:05:12 raft.go:664: INFO: 1 received MsgVoteResp from 1 at term 452
2018/01/17 09:05:12 raft.go:651: INFO: 1 [logterm: 54, index: 221] sent MsgVote request to 2 at term 452
2018/01/17 09:05:12 raft.go:651: INFO: 1 [logterm: 54, index: 221] sent MsgVote request to 3 at term 452
2018/01/17 09:05:12 node.go:191: 		SENDING: MsgVote 1-->2
2018/01/17 09:05:12 node.go:191: 		SENDING: MsgVote 1-->3
2018/01/17 09:05:12 node.go:324: No healthy connection found to node Id: 3, err: Unhealthy connection
2018/01/17 09:05:12 node.go:324: No healthy connection found to node Id: 2, err: Unhealthy connection
2018/01/17 09:05:15 node.go:536: RECEIVED: MsgVote 2-->1
2018/01/17 09:05:15 raft.go:731: INFO: 1 [term: 452] ignored a MsgVote message with lower term from 2 [term: 55]
2018/01/17 09:05:15 raft.go:749: INFO: 1 is starting a new election at term 452
2018/01/17 09:05:15 raft.go:580: INFO: 1 became candidate at term 453
2018/01/17 09:05:15 raft.go:664: INFO: 1 received MsgVoteResp from 1 at term 453
2018/01/17 09:05:15 raft.go:651: INFO: 1 [logterm: 54, index: 221] sent MsgVote request to 2 at term 453
2018/01/17 09:05:15 raft.go:651: INFO: 1 [logterm: 54, index: 221] sent MsgVote request to 3 at term 453
2018/01/17 09:05:15 node.go:191: 		SENDING: MsgVote 1-->2
2018/01/17 09:05:15 node.go:191: 		SENDING: MsgVote 1-->3
2018/01/17 09:05:15 node.go:324: No healthy connection found to node Id: 3, err: Unhealthy connection
2018/01/17 09:05:15 node.go:324: No healthy connection found to node Id: 2, err: Unhealthy connection
2018/01/17 09:05:18 node.go:536: RECEIVED: MsgVote 3-->1
2018/01/17 09:05:18 raft.go:731: INFO: 1 [term: 453] ignored a MsgVote message with lower term from 3 [term: 55]
2018/01/17 09:05:18 raft.go:749: INFO: 1 is starting a new election at term 453
2018/01/17 09:05:18 raft.go:580: INFO: 1 became candidate at term 454
2018/01/17 09:05:18 raft.go:664: INFO: 1 received MsgVoteResp from 1 at term 454
2018/01/17 09:05:18 raft.go:651: INFO: 1 [logterm: 54, index: 221] sent MsgVote request to 2 at term 454
2018/01/17 09:05:18 raft.go:651: INFO: 1 [logterm: 54, index: 221] sent MsgVote request to 3 at term 454
2018/01/17 09:05:18 node.go:191: 		SENDING: MsgVote 1-->2
2018/01/17 09:05:18 node.go:191: 		SENDING: MsgVote 1-->3
2018/01/17 09:05:18 node.go:324: No healthy connection found to node Id: 2, err: Unhealthy connection
2018/01/17 09:05:18 node.go:324: No healthy connection found to node Id: 3, err: Unhealthy connection
2018/01/17 09:05:19 node.go:536: RECEIVED: MsgVote 2-->1
2018/01/17 09:05:19 raft.go:731: INFO: 1 [term: 454] ignored a MsgVote message with lower term from 2 [term: 56]
2018/01/17 09:05:20 oracle.go:375: No healthy connection found to leader of group 2
2018/01/17 09:05:20 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:20 zero.go:316: Got connection request: id:10 addr:"localhost:7090" 
2018/01/17 09:05:20 zero.go:412: Connected
2018/01/17 09:05:22 node.go:536: RECEIVED: MsgVote 2-->1
2018/01/17 09:05:22 raft.go:731: INFO: 1 [term: 454] ignored a MsgVote message with lower term from 2 [term: 57]
2018/01/17 09:05:22 node.go:536: RECEIVED: MsgVote 3-->1
2018/01/17 09:05:22 raft.go:731: INFO: 1 [term: 454] ignored a MsgVote message with lower term from 3 [term: 56]
2018/01/17 09:05:22 raft.go:749: INFO: 1 is starting a new election at term 454
2018/01/17 09:05:22 raft.go:580: INFO: 1 became candidate at term 455
2018/01/17 09:05:22 raft.go:664: INFO: 1 received MsgVoteResp from 1 at term 455
2018/01/17 09:05:22 raft.go:651: INFO: 1 [logterm: 54, index: 221] sent MsgVote request to 2 at term 455
2018/01/17 09:05:22 raft.go:651: INFO: 1 [logterm: 54, index: 221] sent MsgVote request to 3 at term 455
2018/01/17 09:05:22 node.go:191: 		SENDING: MsgVote 1-->2
2018/01/17 09:05:22 node.go:191: 		SENDING: MsgVote 1-->3
2018/01/17 09:05:22 node.go:536: RECEIVED: MsgVoteResp 3-->1
2018/01/17 09:05:22 node.go:536: RECEIVED: MsgVoteResp 2-->1
2018/01/17 09:05:22 raft.go:664: INFO: 1 received MsgVoteResp from 3 at term 455
2018/01/17 09:05:22 raft.go:1013: INFO: 1 [quorum:2] has received 2 MsgVoteResp votes and 0 vote rejections
2018/01/17 09:05:22 raft.go:621: INFO: 1 became leader at term 455
2018/01/17 09:05:22 node.go:301: INFO: raft.node: 1 elected leader 1 at term 455
2018/01/17 09:05:22 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:22 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgProp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgProp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgProp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgProp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgProp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgProp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgProp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgProp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgProp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgProp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgProp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgProp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 zero.go:316: Got connection request: id:11 addr:"localhost:7091" 
2018/01/17 09:05:23 zero.go:412: Connected
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgReadIndex 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgProp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgReadIndexResp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgReadIndex 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgReadIndexResp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgReadIndex 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgReadIndexResp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgReadIndex 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgReadIndexResp 1-->2
2018/01/17 09:05:24 node.go:536: RECEIVED: MsgProp 2-->1
2018/01/17 09:05:24 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:24 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:24 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:24 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:24 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:24 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:24 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:24 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgReadIndex 2-->1
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgReadIndex 2-->1
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgReadIndexResp 1-->2
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgReadIndexResp 1-->2
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgProp 2-->1
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgReadIndex 2-->1
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgReadIndex 2-->1
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgReadIndexResp 1-->2
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgReadIndexResp 1-->2
2018/01/17 09:05:26 zero.go:316: Got connection request: id:12 addr:"localhost:7092" 
2018/01/17 09:05:26 zero.go:412: Connected
2018/01/17 09:05:26 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:26 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:26 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:26 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:26 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:26 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:26 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:26 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:26 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:26 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:26 node.go:536: RECEIVED: MsgReadIndex 2-->1
2018/01/17 09:05:26 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:26 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:26 node.go:191: 		SENDING: MsgReadIndexResp 1-->2
2018/01/17 09:05:26 node.go:536: RECEIVED: MsgReadIndex 2-->1
2018/01/17 09:05:26 node.go:191: 		SENDING: MsgReadIndexResp 1-->2
2018/01/17 09:05:29 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:29 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:29 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:29 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:29 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:29 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:29 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:29 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:29 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:29 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:29 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:29 node.go:536: RECEIVED: MsgReadIndex 2-->1
2018/01/17 09:05:29 node.go:191: 		SENDING: MsgReadIndexResp 1-->2
2018/01/17 09:05:29 node.go:536: RECEIVED: MsgReadIndex 2-->1
2018/01/17 09:05:29 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:29 node.go:191: 		SENDING: MsgReadIndexResp 1-->2
2018/01/17 09:05:30 oracle.go:375: No healthy connection found to leader of group 3
2018/01/17 09:05:30 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure

zero 2 logs:

Setting up grpc listener at: 0.0.0.0:7081
Setting up http listener at: 0.0.0.0:8081
2018/01/17 09:05:12 node.go:248: Found hardstate: {Term:54 Vote:2 Commit:221 XXX_unrecognized:[]}
2018/01/17 09:05:12 node.go:260: Group 0 found 221 entries
2018/01/17 09:05:12 raft.go:400: Restarting node for dgraphzero
2018/01/17 09:05:12 raft.go:567: INFO: 2 became follower at term 54
2018/01/17 09:05:12 raft.go:315: INFO: newRaft 2 [peers: [], term: 54, commit: 221, applied: 0, lastindex: 221, lastterm: 54]
Running Dgraph zero...
2018/01/17 09:05:12 pool.go:118: == CONNECT ==> Setting localhost:7080
2018/01/17 09:05:12 node.go:127: Setting conf state to nodes:1 
2018/01/17 09:05:12 node.go:127: Setting conf state to nodes:1 nodes:2 
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7082. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:118: == CONNECT ==> Setting localhost:7082
2018/01/17 09:05:12 node.go:127: Setting conf state to nodes:1 nodes:2 nodes:3 
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:118: == CONNECT ==> Setting localhost:7090
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:118: == CONNECT ==> Setting localhost:7091
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:118: == CONNECT ==> Setting localhost:7092
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:15 raft.go:749: INFO: 2 is starting a new election at term 54
2018/01/17 09:05:15 raft.go:580: INFO: 2 became candidate at term 55
2018/01/17 09:05:15 raft.go:664: INFO: 2 received MsgVoteResp from 2 at term 55
2018/01/17 09:05:15 raft.go:651: INFO: 2 [logterm: 54, index: 221] sent MsgVote request to 3 at term 55
2018/01/17 09:05:15 raft.go:651: INFO: 2 [logterm: 54, index: 221] sent MsgVote request to 1 at term 55
2018/01/17 09:05:15 node.go:191: 		SENDING: MsgVote 2-->3
2018/01/17 09:05:15 node.go:191: 		SENDING: MsgVote 2-->1
2018/01/17 09:05:15 node.go:324: No healthy connection found to node Id: 3, err: Unhealthy connection
2018/01/17 09:05:18 node.go:536: RECEIVED: MsgVote 3-->2
2018/01/17 09:05:18 raft.go:772: INFO: 2 [logterm: 54, index: 221, vote: 2] rejected MsgVote from 3 [logterm: 54, index: 221] at term 55
2018/01/17 09:05:18 node.go:191: 		SENDING: MsgVoteResp 2-->3
2018/01/17 09:05:18 node.go:324: No healthy connection found to node Id: 3, err: Unhealthy connection
2018/01/17 09:05:19 raft.go:749: INFO: 2 is starting a new election at term 55
2018/01/17 09:05:19 raft.go:580: INFO: 2 became candidate at term 56
2018/01/17 09:05:19 raft.go:664: INFO: 2 received MsgVoteResp from 2 at term 56
2018/01/17 09:05:19 raft.go:651: INFO: 2 [logterm: 54, index: 221] sent MsgVote request to 3 at term 56
2018/01/17 09:05:19 raft.go:651: INFO: 2 [logterm: 54, index: 221] sent MsgVote request to 1 at term 56
2018/01/17 09:05:19 node.go:191: 		SENDING: MsgVote 2-->3
2018/01/17 09:05:19 node.go:191: 		SENDING: MsgVote 2-->1
2018/01/17 09:05:19 node.go:324: No healthy connection found to node Id: 3, err: Unhealthy connection
2018/01/17 09:05:22 raft.go:749: INFO: 2 is starting a new election at term 56
2018/01/17 09:05:22 raft.go:580: INFO: 2 became candidate at term 57
2018/01/17 09:05:22 raft.go:664: INFO: 2 received MsgVoteResp from 2 at term 57
2018/01/17 09:05:22 raft.go:651: INFO: 2 [logterm: 54, index: 221] sent MsgVote request to 1 at term 57
2018/01/17 09:05:22 raft.go:651: INFO: 2 [logterm: 54, index: 221] sent MsgVote request to 3 at term 57
2018/01/17 09:05:22 node.go:191: 		SENDING: MsgVote 2-->1
2018/01/17 09:05:22 node.go:191: 		SENDING: MsgVote 2-->3
2018/01/17 09:05:22 node.go:324: No healthy connection found to node Id: 3, err: Unhealthy connection
2018/01/17 09:05:22 node.go:536: RECEIVED: MsgVote 3-->2
2018/01/17 09:05:22 raft.go:731: INFO: 2 [term: 57] ignored a MsgVote message with lower term from 3 [term: 56]
2018/01/17 09:05:22 oracle.go:375: No healthy connection found to leader of group 1
2018/01/17 09:05:22 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:22 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:22 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:22 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:22 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:22 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:22 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:22 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:22 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:22 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:22 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:22 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:22 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:22 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:22 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:22 node.go:536: RECEIVED: MsgVote 1-->2
2018/01/17 09:05:22 raft.go:708: INFO: 2 [term: 57] received a MsgVote message with higher term from 1 [term: 455]
2018/01/17 09:05:22 raft.go:567: INFO: 2 became follower at term 455
2018/01/17 09:05:22 raft.go:763: INFO: 2 [logterm: 54, index: 221, vote: 0] cast MsgVote for 1 [logterm: 54, index: 221] at term 455
2018/01/17 09:05:22 node.go:191: 		SENDING: MsgVoteResp 2-->1
2018/01/17 09:05:22 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:22 node.go:301: INFO: raft.node: 2 elected leader 1 at term 455
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgProp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgProp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgProp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgProp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgProp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgProp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgReadIndex 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgProp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgReadIndexResp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgReadIndex 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgReadIndexResp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgReadIndex 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgReadIndexResp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgReadIndex 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgReadIndexResp 1-->2
2018/01/17 09:05:24 node.go:191: 		SENDING: MsgProp 2-->1
2018/01/17 09:05:24 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:24 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:24 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgReadIndex 2-->1
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgReadIndex 2-->1
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgReadIndexResp 1-->2
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgReadIndexResp 1-->2
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgProp 2-->1
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgReadIndex 2-->1
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgReadIndex 2-->1
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgReadIndexResp 1-->2
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgReadIndexResp 1-->2
2018/01/17 09:05:26 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:26 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:26 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:26 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:26 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:26 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:26 node.go:191: 		SENDING: MsgReadIndex 2-->1
2018/01/17 09:05:26 node.go:191: 		SENDING: MsgReadIndex 2-->1
2018/01/17 09:05:26 node.go:536: RECEIVED: MsgReadIndexResp 1-->2
2018/01/17 09:05:26 node.go:536: RECEIVED: MsgReadIndexResp 1-->2
2018/01/17 09:05:29 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:29 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:29 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:29 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:29 node.go:191: 		SENDING: MsgReadIndex 2-->1
2018/01/17 09:05:29 node.go:191: 		SENDING: MsgReadIndex 2-->1
2018/01/17 09:05:29 node.go:536: RECEIVED: MsgReadIndexResp 1-->2
2018/01/17 09:05:29 node.go:536: RECEIVED: MsgReadIndexResp 1-->2
2018/01/17 09:05:32 oracle.go:375: No healthy connection found to leader of group 2
2018/01/17 09:06:23 node.go:191: 		SENDING: MsgReadIndex 2-->1
2018/01/17 09:06:23 node.go:536: RECEIVED: MsgReadIndexResp 1-->2
2018/01/17 09:06:23 node.go:191: 		SENDING: MsgReadIndex 2-->1
2018/01/17 09:06:23 node.go:536: RECEIVED: MsgReadIndexResp 1-->2
2018/01/17 09:07:23 node.go:191: 		SENDING: MsgReadIndex 2-->1
2018/01/17 09:07:23 node.go:536: RECEIVED: MsgReadIndexResp 1-->2
2018/01/17 09:07:23 node.go:191: 		SENDING: MsgReadIndex 2-->1
2018/01/17 09:07:23 node.go:536: RECEIVED: MsgReadIndexResp 1-->2

zero 3 logs:

Setting up grpc listener at: 0.0.0.0:7082
Setting up http listener at: 0.0.0.0:8082
2018/01/17 09:05:15 node.go:248: Found hardstate: {Term:54 Vote:1 Commit:221 XXX_unrecognized:[]}
2018/01/17 09:05:15 node.go:260: Group 0 found 221 entries
2018/01/17 09:05:15 raft.go:400: Restarting node for dgraphzero
2018/01/17 09:05:15 raft.go:567: INFO: 3 became follower at term 54
2018/01/17 09:05:15 raft.go:315: INFO: newRaft 3 [peers: [], term: 54, commit: 221, applied: 0, lastindex: 221, lastterm: 54]
Running Dgraph zero...
2018/01/17 09:05:16 pool.go:118: == CONNECT ==> Setting localhost:7080
2018/01/17 09:05:16 node.go:127: Setting conf state to nodes:1 
2018/01/17 09:05:16 pool.go:118: == CONNECT ==> Setting localhost:7081
2018/01/17 09:05:16 node.go:127: Setting conf state to nodes:1 nodes:2 
2018/01/17 09:05:16 node.go:127: Setting conf state to nodes:1 nodes:2 nodes:3 
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:118: == CONNECT ==> Setting localhost:7091
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:118: == CONNECT ==> Setting localhost:7090
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:118: == CONNECT ==> Setting localhost:7092
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:18 raft.go:749: INFO: 3 is starting a new election at term 54
2018/01/17 09:05:18 raft.go:580: INFO: 3 became candidate at term 55
2018/01/17 09:05:18 raft.go:664: INFO: 3 received MsgVoteResp from 3 at term 55
2018/01/17 09:05:18 raft.go:651: INFO: 3 [logterm: 54, index: 221] sent MsgVote request to 1 at term 55
2018/01/17 09:05:18 raft.go:651: INFO: 3 [logterm: 54, index: 221] sent MsgVote request to 2 at term 55
2018/01/17 09:05:18 node.go:191: 		SENDING: MsgVote 3-->1
2018/01/17 09:05:18 node.go:191: 		SENDING: MsgVote 3-->2
2018/01/17 09:05:22 raft.go:749: INFO: 3 is starting a new election at term 55
2018/01/17 09:05:22 raft.go:580: INFO: 3 became candidate at term 56
2018/01/17 09:05:22 raft.go:664: INFO: 3 received MsgVoteResp from 3 at term 56
2018/01/17 09:05:22 raft.go:651: INFO: 3 [logterm: 54, index: 221] sent MsgVote request to 1 at term 56
2018/01/17 09:05:22 raft.go:651: INFO: 3 [logterm: 54, index: 221] sent MsgVote request to 2 at term 56
2018/01/17 09:05:22 node.go:191: 		SENDING: MsgVote 3-->1
2018/01/17 09:05:22 node.go:191: 		SENDING: MsgVote 3-->2
2018/01/17 09:05:22 node.go:536: RECEIVED: MsgVote 1-->3
2018/01/17 09:05:22 raft.go:708: INFO: 3 [term: 56] received a MsgVote message with higher term from 1 [term: 455]
2018/01/17 09:05:22 raft.go:567: INFO: 3 became follower at term 455
2018/01/17 09:05:22 raft.go:763: INFO: 3 [logterm: 54, index: 221, vote: 0] cast MsgVote for 1 [logterm: 54, index: 221] at term 455
2018/01/17 09:05:22 node.go:191: 		SENDING: MsgVoteResp 3-->1
2018/01/17 09:05:22 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:22 node.go:301: INFO: raft.node: 3 elected leader 1 at term 455
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgProp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgProp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgProp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgProp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgProp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgProp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:24 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:24 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:24 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:24 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:24 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:25 oracle.go:375: No healthy connection found to leader of group 1
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:26 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:26 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:26 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:26 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:26 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:26 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:26 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:26 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:26 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:26 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:26 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:26 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:26 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:26 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:26 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:26 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:26 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:29 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:29 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:29 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:29 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:29 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:29 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:29 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:29 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:35 oracle.go:375: No healthy connection found to leader of group 3
2018/01/17 09:08:00 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:08:00 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:08:00 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:08:00 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:08:00 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:08:00 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:08:00 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:08:00 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:08:00 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:08:00 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:08:00 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:08:00 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:08:00 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:08:01 node.go:191: 		SENDING: MsgAppResp 3-->1

Is there a way to recover from the no leader issue?

The cluster should automatically recover ideally. I will go through the logs and try to see what is the issue and get back to you

Any update on the issue? Also, what would be the ETA on the patch release for the issue log.Fatal caused by unexpected meta: 0 ? Just need to plan ahead :slight_smile:

Yeah, we are doing a release today(v1.0.2) which contains the fix. ETA 2-3 hrs.

Is there no nighly today ?

root@ubu-16:/home/dgraph# curl https://nightly.dgraph.io -sSf | bash
curl: (22) The requested URL returned error: 404 Not Found

Solved

Oh sorry - saw there is a new version in main

You can get the nightly build with the following command.
curl https://get.dgraph.io -sSf | bash -s nightly

We have updated the docs to reflect the same

Thank you - first idea was there is none because of the new version in main

Is there any update on the issue?