Dgraph runs into a error loop and freezes the host

jzhu077 · December 23, 2017, 7:55pm

Every dgraph zero and server errors the same message repetitively, even after restart the cluster with existing database.

Dgraph zero:
raft.go:1070: INFO: 1 no leader at term 7; dropping index reading msg

dgraph server:
Unable to sync memberships. Error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
groups.go:673: Error in oracle delta stream. Error: rpc error: code = Unknown desc = Node is no longer leader.

mrjn · December 24, 2017, 1:39am

Can you file a Github issue with a way to replicate this?

jzhu077 · January 15, 2018, 11:02pm

Sorry, I couldn’t find a nice way to reproduce this.
Basically, I ran all dgraph instances in a kubernetes pod. There are 5 dgraph zeros and 20 dgraph servers in the pod. Each of them runs in its own container. Each server and zero has an unique group ID assigned via flag idx

After ingesting a large amount of data ~31GB in dgraph database, I tried to make a copy of the database:

stop the running dgraph pod.
copy the database.
restart the pod.

I saw following logs with spiking CPU usage:
similar logs for all dgraph zero:

Setting up grpc listener at: 0.0.0.0:7080
Setting up http listener at: 0.0.0.0:8080
2018/01/15 22:41:37 node.go:233: Found Snapshot, Metadata: {ConfState:{Nodes:[1 2 3 4 5] XXX_unrecognized:[]} Index:27053 Term:3 XXX_unrecognized:[]}
2018/01/15 22:41:37 node.go:248: Found hardstate: {Term:3 Vote:1 Commit:27881 XXX_unrecognized:[]}
2018/01/15 22:41:37 node.go:260: Group 0 found 828 entries
2018/01/15 22:41:37 raft.go:400: Restarting node for dgraphzero
2018/01/15 22:41:37 pool.go:167: Echo error from localhost:7104. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7104
2018/01/15 22:41:37 pool.go:167: Echo error from localhost:7098. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7098
2018/01/15 22:41:37 pool.go:167: Echo error from localhost:7094. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7094
2018/01/15 22:41:37 pool.go:167: Echo error from localhost:7102. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7102
2018/01/15 22:41:37 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7092
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7093
2018/01/15 22:41:37 pool.go:167: Echo error from localhost:7100. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7100
2018/01/15 22:41:37 pool.go:167: Echo error from localhost:7108. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7108
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7107
2018/01/15 22:41:37 pool.go:167: Echo error from localhost:7099. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7099
2018/01/15 22:41:37 pool.go:167: Echo error from localhost:7096. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7096
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7103
2018/01/15 22:41:37 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7090
2018/01/15 22:41:37 pool.go:167: Echo error from localhost:7105. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7105
2018/01/15 22:41:37 pool.go:167: Echo error from localhost:7095. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7095
2018/01/15 22:41:37 pool.go:167: Echo error from localhost:7101. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7101
2018/01/15 22:41:37 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7091
2018/01/15 22:41:37 pool.go:167: Echo error from localhost:7097. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7097
2018/01/15 22:41:37 pool.go:167: Echo error from localhost:7106. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7106
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7109
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7081
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7082
2018/01/15 22:41:37 pool.go:167: Echo error from localhost:7083. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7083
2018/01/15 22:41:37 pool.go:118: == CONNECT ==> Setting localhost:7084
2018/01/15 22:41:37 raft.go:567: INFO: 1 became follower at term 3
2018/01/15 22:41:37 raft.go:315: INFO: newRaft 1 [peers: [1,2,3,4,5], term: 3, commit: 27881, applied: 27053, lastindex: 27881, lastterm: 3]
Running Dgraph zero...
2018/01/15 22:41:38 zero.go:316: Got connection request: id:18 addr:"localhost:7098" 
2018/01/15 22:41:38 zero.go:412: Connected
2018/01/15 22:41:38 zero.go:316: Got connection request: id:27 addr:"localhost:7107" 
2018/01/15 22:41:38 zero.go:412: Connected
2018/01/15 22:41:38 zero.go:316: Got connection request: id:12 addr:"localhost:7092" 
2018/01/15 22:41:38 zero.go:412: Connected
2018/01/15 22:41:38 zero.go:316: Got connection request: id:29 addr:"localhost:7109" 
2018/01/15 22:41:38 zero.go:412: Connected
2018/01/15 22:41:38 zero.go:316: Got connection request: id:21 addr:"localhost:7101" 
2018/01/15 22:41:38 zero.go:412: Connected
2018/01/15 22:41:38 zero.go:316: Got connection request: id:23 addr:"localhost:7103" 
2018/01/15 22:41:38 zero.go:412: Connected
2018/01/15 22:41:38 zero.go:316: Got connection request: id:22 addr:"localhost:7102" 
2018/01/15 22:41:38 zero.go:412: Connected
2018/01/15 22:41:38 zero.go:316: Got connection request: id:19 addr:"localhost:7099" 
2018/01/15 22:41:38 zero.go:412: Connected
2018/01/15 22:41:38 zero.go:316: Got connection request: id:10 addr:"localhost:7090" 
2018/01/15 22:41:38 zero.go:412: Connected
2018/01/15 22:41:38 zero.go:316: Got connection request: id:24 addr:"localhost:7104" 
2018/01/15 22:41:38 zero.go:412: Connected
2018/01/15 22:41:39 zero.go:316: Got connection request: id:17 addr:"localhost:7097" 
2018/01/15 22:41:39 zero.go:412: Connected
2018/01/15 22:41:39 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
2018/01/15 22:41:39 zero.go:316: Got connection request: id:13 addr:"localhost:7093" 
2018/01/15 22:41:39 zero.go:412: Connected
2018/01/15 22:41:39 zero.go:316: Got connection request: id:14 addr:"localhost:7094" 
2018/01/15 22:41:39 zero.go:412: Connected
2018/01/15 22:41:39 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
2018/01/15 22:41:39 zero.go:316: Got connection request: id:15 addr:"localhost:7095" 
2018/01/15 22:41:39 zero.go:412: Connected
2018/01/15 22:41:39 zero.go:316: Got connection request: id:20 addr:"localhost:7100" 
2018/01/15 22:41:39 zero.go:412: Connected
2018/01/15 22:41:39 zero.go:316: Got connection request: id:16 addr:"localhost:7096" 
2018/01/15 22:41:39 zero.go:412: Connected
2018/01/15 22:41:39 zero.go:316: Got connection request: id:25 addr:"localhost:7105" 
2018/01/15 22:41:39 zero.go:412: Connected
2018/01/15 22:41:40 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
2018/01/15 22:41:40 zero.go:316: Got connection request: id:28 addr:"localhost:7108" 
2018/01/15 22:41:40 zero.go:412: Connected
2018/01/15 22:41:40 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
2018/01/15 22:41:41 zero.go:316: Got connection request: id:11 addr:"localhost:7091" 
2018/01/15 22:41:41 zero.go:412: Connected
2018/01/15 22:41:41 zero.go:316: Got connection request: id:26 addr:"localhost:7106" 
2018/01/15 22:41:41 zero.go:412: Connected
2018/01/15 22:41:42 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
2018/01/15 22:41:43 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
2018/01/15 22:41:44 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
2018/01/15 22:41:45 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
2018/01/15 22:41:46 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
2018/01/15 22:41:46 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
2018/01/15 22:41:46 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
2018/01/15 22:41:46 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
2018/01/15 22:41:46 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
2018/01/15 22:41:47 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg
2018/01/15 22:41:47 raft.go:1070: INFO: 1 no leader at term 3; dropping index reading msg

Similar logs for all dgraph server

2018/01/15 22:41:40 groups.go:457: Unable to sync memberships. Error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/01/15 22:41:40 groups.go:673: Error in oracle delta stream. Error: rpc error: code = Unknown desc = Node is no longer leader.
2018/01/15 22:41:41 groups.go:457: Unable to sync memberships. Error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/01/15 22:41:41 groups.go:673: Error in oracle delta stream. Error: rpc error: code = Unknown desc = Node is no longer leader.
2018/01/15 22:41:42 raft.go:749: INFO: e is starting a new election at term 3
2018/01/15 22:41:42 raft.go:580: INFO: e became candidate at term 4
2018/01/15 22:41:42 raft.go:664: INFO: e received MsgVoteResp from e at term 4
2018/01/15 22:41:42 raft.go:621: INFO: e became leader at term 4
2018/01/15 22:41:42 node.go:301: INFO: raft.node: e elected leader e at term 4
2018/01/15 22:41:42 groups.go:457: Unable to sync memberships. Error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/01/15 22:41:42 groups.go:673: Error in oracle delta stream. Error: rpc error: code = Unknown desc = Node is no longer leader.
2018/01/15 22:41:43 groups.go:457: Unable to sync memberships. Error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/01/15 22:41:43 groups.go:673: Error in oracle delta stream. Error: rpc error: code = Unknown desc = Node is no longer leader.
2018/01/15 22:41:44 groups.go:457: Unable to sync memberships. Error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/01/15 22:41:44 groups.go:673: Error in oracle delta stream. Error: rpc error: code = Unknown desc = Node is no longer leader.
2018/01/15 22:41:45 groups.go:457: Unable to sync memberships. Error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/01/15 22:41:45 groups.go:673: Error in oracle delta stream. Error: rpc error: code = Unknown desc = Node is no longer leader.
2018/01/15 22:41:46 groups.go:457: Unable to sync memberships. Error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/01/15 22:41:46 groups.go:673: Error in oracle delta stream. Error: rpc error: code = Unknown desc = Node is no longer leader.
2018/01/15 22:41:47 groups.go:457: Unable to sync memberships. Error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/01/15 22:41:47 groups.go:673: Error in oracle delta stream. Error: rpc error: code = Unknown desc = Node is no longer leader.

When I test the similar setup locally in terminal with much smaller scale (3 zeros and 3 servers) and data set (~50MB)
I noticed when a dgraph zero starts the term increases everytime and the peers array is empty :
INFO: newRaft 1 [peers: [], term: 45, commit: 171, applied: 0, lastindex: 171, lastterm: 45]
whereas in the kubernetes pod the peers array is [1,2,3,4,5] and the term does not increases :
INFO: newRaft 1 [peers: [1,2,3,4,5], term: 3, commit: 27881, applied: 27053, lastindex: 27881, lastterm: 3]

jzhu077 · January 16, 2018, 12:00am

I managed to partially reproduce this issue:
dgraph version: v1.0.1

Set up a cluster locally with 3 dgraph zeros, 3 dgraph servers, and dgraph ratel:

dgraph zero --my="localhost:7080" --wal=zw --bindall=true
dgraph zero --my="localhost:7081" --peer "localhost:7080" --wal=zw --port_offset=1 --bindall=true --idx 2
dgraph zero --my="localhost:7082" --peer "localhost:7080" --wal=zw --port_offset=2 --bindall=true --idx 3
dgraph server --zero="localhost:7080" --my="localhost:7090" --bindall=true --memory_mb=2048 --port_offset=10 --idx=10
dgraph server --zero="localhost:7080" --my="localhost:7091" --bindall=true --memory_mb=2048 --port_offset=11 --idx=11
dgraph server --zero="localhost:7080" --my="localhost:7092" --bindall=true --memory_mb=2048 --port_offset=12 --idx=12
dgraph-ratel -p 8085

Set up the schema in the web UI:

name: string @index(exact) .
type: string @index(exact) .
kind: string @index(exact) .

Obtain data via a small go program:

package main

import (
	"flag"
	"fmt"
	"math/rand"
	"strconv"
	"time"
)

var horizon int
var density float64

func main() {
	density = *flag.Float64("density", 0.01, "Roughly density of func nodes in the tree.")
	d := flag.Int("depth", 11, "Depth of the tree.")
	o := flag.Int("order", 3, "Order of the tree - ie, 2 gives a binary tree.")
	flag.Parse()
	rand.Seed(time.Now().UnixNano())
	fmt.Println(`
 {
    set {`)
	WriteNaryTree(0, *d-1, *o, "root")
	fmt.Println(`
	}
}`[1:])
}

// WriteNaryTree writes quads forming a tree of depth and order n.
func WriteNaryTree(currentDepth, d, n int, parent string) {
	horizon++
	id := strconv.Itoa(horizon)
	fmt.Printf("_:%s <name> \"%s\" .\n", id, id)
	fmt.Printf("_:%s <type> \"foo\" .\n", id)
	if parent != "" {
		fmt.Printf("_:%s <kid> _:%s .\n", parent, id)
	}

	if rand.Float64() < density {
		fmt.Printf("_:%s <kind> \"%s\" .\n", id, "bar")
	}

	if d > currentDepth {
		for i := 0; i < n; i++ {
			WriteNaryTree(currentDepth+1, d, n, id)
		}
	}
}

Use data output from the go program to run the mutation in the web UI
Wait for the rebalancing to complete.
Terminate all dgraph zeros and dgraph servers
Restart dgraph servers then dgraph zerors
Might need to repeat steps 6-7 for a few times before one or more dgraph termite themselves after following logs:

2018/01/16 12:23:38 gRPC server started.  Listening on port 9090
2018/01/16 12:23:38 HTTP server started.  Listening on port 8090
2018/01/16 12:23:38 worker.go:99: Worker listening at address: [::]:7090
2018/01/16 12:23:38 pool.go:118: == CONNECT ==> Setting localhost:7080
2018/01/16 12:23:38 groups.go:117: Connected to group zero. Connection state: member:<id:10 addr:"localhost:7090" > state:<counter:180 groups:<key:1 value:<members:<key:10 value:<id:10 group_id:1 addr:"localhost:7090" leader:true last_update:1516058396 > > tablets:<key:"_dummy_" value:<group_id:1 predicate:"_dummy_" force:true space:19 > > tablets:<key:"_predicate_" value:<group_id:1 predicate:"_predicate_" > > tablets:<key:"type" value:<group_id:1 predicate:"type" space:6898672 > > > > groups:<key:2 value:<members:<key:11 value:<id:11 group_id:2 addr:"localhost:7091" leader:true last_update:1516058434 > > tablets:<key:"name" value:<group_id:2 predicate:"name" force:true space:8923661 > > > > groups:<key:3 value:<members:<key:12 value:<id:12 group_id:3 addr:"localhost:7092" leader:true last_update:1516057762 > > tablets:<key:"kid" value:<group_id:3 predicate:"kid" space:1154035 > > tablets:<key:"kind" value:<group_id:3 predicate:"kind" space:43273 > > > > zeros:<key:1 value:<id:1 addr:"localhost:7080" > > zeros:<key:2 value:<id:2 addr:"localhost:7081" leader:true > > zeros:<key:3 value:<id:3 addr:"localhost:7082" > > maxLeaseId:98574 maxTxnTs:10000 > 
2018/01/16 12:23:38 pool.go:118: == CONNECT ==> Setting localhost:7081
2018/01/16 12:23:38 pool.go:118: == CONNECT ==> Setting localhost:7082
2018/01/16 12:23:38 pool.go:118: == CONNECT ==> Setting localhost:7091
2018/01/16 12:23:38 pool.go:118: == CONNECT ==> Setting localhost:7092
2018/01/16 12:23:38 draft.go:140: Node ID: 10 with GroupID: 1
2018/01/16 12:23:38 node.go:233: Found Snapshot, Metadata: {ConfState:{Nodes:[10] XXX_unrecognized:[]} Index:14 Term:4 XXX_unrecognized:[]}
2018/01/16 12:23:38 node.go:248: Found hardstate: {Term:7 Vote:10 Commit:19 XXX_unrecognized:[]}
2018/01/16 12:23:38 node.go:260: Group 1 found 5 entries
2018/01/16 12:23:38 draft.go:657: Restarting node for group: 1
2018/01/16 12:23:38 raft.go:567: INFO: a became follower at term 7
2018/01/16 12:23:38 raft.go:315: INFO: newRaft a [peers: [a], term: 7, commit: 19, applied: 14, lastindex: 19, lastterm: 7]
2018/01/16 12:23:40 raft.go:749: INFO: a is starting a new election at term 7
2018/01/16 12:23:40 raft.go:580: INFO: a became candidate at term 8
2018/01/16 12:23:40 raft.go:664: INFO: a received MsgVoteResp from a at term 8
2018/01/16 12:23:40 raft.go:621: INFO: a became leader at term 8
2018/01/16 12:23:40 node.go:301: INFO: raft.node: a elected leader a at term 8
2018/01/16 12:24:23 groups.go:673: Error in oracle delta stream. Error: rpc error: code = Unknown desc = Server is being shut down.
2018/01/16 12:24:24 groups.go:665: Error while calling Oracle rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/16 12:24:25 groups.go:665: Error while calling Oracle rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/16 12:24:26 groups.go:673: Error in oracle delta stream. Error: rpc error: code = Unknown desc = Node is no longer leader.
2018/01/16 12:24:26 groups.go:457: Unable to sync memberships. Error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/01/16 12:24:34 groups.go:673: Error in oracle delta stream. Error: rpc error: code = Unknown desc = Node is no longer leader.
2018/01/16 12:24:35 groups.go:457: Unable to sync memberships. Error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/01/16 12:24:35 groups.go:673: Error in oracle delta stream. Error: rpc error: code = Unknown desc = Node is no longer leader.
2018/01/16 12:24:36 groups.go:457: Unable to sync memberships. Error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/01/16 12:24:36 groups.go:673: Error in oracle delta stream. Error: rpc error: code = Unknown desc = Node is no longer leader.
2018/01/16 12:24:37 groups.go:457: Unable to sync memberships. Error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/01/16 12:24:37 groups.go:673: Error in oracle delta stream. Error: rpc error: code = Unknown desc = Node is no longer leader.
2018/01/16 12:24:38 groups.go:457: Unable to sync memberships. Error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/01/16 12:26:34 unexpected meta: 0

pawan · January 16, 2018, 1:01am

Hey @jzhu077

The log.Fatal caused by unexpected meta: 0 is a known issue and has been fixed in master. We will be doing a release today which will have the fix.

We are looking at the other errors.

pawan · January 16, 2018, 1:07am

@jzhu077 Does leader election finally happen after all the dropping index reading msg?

Also, how do you copy the database? Are you making sure that Zero and Dgraph Servers restart with the same directories (p and w for Server and zw for Zero)?

jzhu077 · January 16, 2018, 1:11am

No. It constantly logs the same error message.
I copied the database(p,w and zw) via cp from the current location to the backup directory, so I assume the database stays as it is during the process.

Also, I was wondering which one is the expected behaviour?
INFO: newRaft 1 [peers: [], term: 45, commit: 171, applied: 0, lastindex: 171, lastterm: 45]
or
INFO: newRaft 1 [peers: [1,2,3,4,5], term: 3, commit: 27881, applied: 27053, lastindex: 27881, lastterm: 3]

Should peers be [1,2,3] if I have 3 dgraph zeros or it should be empty at the time it starts?

janardhan · January 16, 2018, 4:27am

Empty peers shouldn’t happen, can you please share the logs of zero when you are running locally.

jzhu077 · January 16, 2018, 8:07pm

zero 1 logs

Setting up grpc listener at: 0.0.0.0:7080
Setting up http listener at: 0.0.0.0:8080
2018/01/17 09:05:00 node.go:248: Found hardstate: {Term:448 Vote:1 Commit:221 XXX_unrecognized:[]}
2018/01/17 09:05:00 node.go:260: Group 0 found 221 entries
2018/01/17 09:05:00 raft.go:400: Restarting node for dgraphzero
2018/01/17 09:05:00 raft.go:567: INFO: 1 became follower at term 448
2018/01/17 09:05:00 raft.go:315: INFO: newRaft 1 [peers: [], term: 448, commit: 221, applied: 0, lastindex: 221, lastterm: 54]
Running Dgraph zero...
2018/01/17 09:05:00 node.go:127: Setting conf state to nodes:1 
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7081. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:118: == CONNECT ==> Setting localhost:7081
2018/01/17 09:05:00 node.go:127: Setting conf state to nodes:1 nodes:2 
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7082. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:118: == CONNECT ==> Setting localhost:7082
2018/01/17 09:05:00 node.go:127: Setting conf state to nodes:1 nodes:2 nodes:3 
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:118: == CONNECT ==> Setting localhost:7091
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:118: == CONNECT ==> Setting localhost:7092
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:118: == CONNECT ==> Setting localhost:7090
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:00 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:03 raft.go:749: INFO: 1 is starting a new election at term 448
2018/01/17 09:05:03 raft.go:580: INFO: 1 became candidate at term 449
2018/01/17 09:05:03 raft.go:664: INFO: 1 received MsgVoteResp from 1 at term 449
2018/01/17 09:05:03 raft.go:651: INFO: 1 [logterm: 54, index: 221] sent MsgVote request to 2 at term 449
2018/01/17 09:05:03 raft.go:651: INFO: 1 [logterm: 54, index: 221] sent MsgVote request to 3 at term 449
2018/01/17 09:05:03 node.go:191: 		SENDING: MsgVote 1-->2
2018/01/17 09:05:03 node.go:191: 		SENDING: MsgVote 1-->3
2018/01/17 09:05:03 node.go:324: No healthy connection found to node Id: 3, err: Unhealthy connection
2018/01/17 09:05:03 node.go:324: No healthy connection found to node Id: 2, err: Unhealthy connection
2018/01/17 09:05:07 raft.go:749: INFO: 1 is starting a new election at term 449
2018/01/17 09:05:07 raft.go:580: INFO: 1 became candidate at term 450
2018/01/17 09:05:07 raft.go:664: INFO: 1 received MsgVoteResp from 1 at term 450
2018/01/17 09:05:07 raft.go:651: INFO: 1 [logterm: 54, index: 221] sent MsgVote request to 2 at term 450
2018/01/17 09:05:07 raft.go:651: INFO: 1 [logterm: 54, index: 221] sent MsgVote request to 3 at term 450
2018/01/17 09:05:07 node.go:191: 		SENDING: MsgVote 1-->2
2018/01/17 09:05:07 node.go:191: 		SENDING: MsgVote 1-->3
2018/01/17 09:05:07 node.go:324: No healthy connection found to node Id: 3, err: Unhealthy connection
2018/01/17 09:05:07 node.go:324: No healthy connection found to node Id: 2, err: Unhealthy connection
2018/01/17 09:05:10 raft.go:749: INFO: 1 is starting a new election at term 450
2018/01/17 09:05:10 raft.go:580: INFO: 1 became candidate at term 451
2018/01/17 09:05:10 raft.go:664: INFO: 1 received MsgVoteResp from 1 at term 451
2018/01/17 09:05:10 raft.go:651: INFO: 1 [logterm: 54, index: 221] sent MsgVote request to 2 at term 451
2018/01/17 09:05:10 raft.go:651: INFO: 1 [logterm: 54, index: 221] sent MsgVote request to 3 at term 451
2018/01/17 09:05:10 node.go:191: 		SENDING: MsgVote 1-->2
2018/01/17 09:05:10 node.go:191: 		SENDING: MsgVote 1-->3
2018/01/17 09:05:10 node.go:324: No healthy connection found to node Id: 3, err: Unhealthy connection
2018/01/17 09:05:10 node.go:324: No healthy connection found to node Id: 2, err: Unhealthy connection
2018/01/17 09:05:10 oracle.go:375: No healthy connection found to leader of group 3
2018/01/17 09:05:10 pool.go:167: Echo error from localhost:7081. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 raft.go:749: INFO: 1 is starting a new election at term 451
2018/01/17 09:05:12 raft.go:580: INFO: 1 became candidate at term 452
2018/01/17 09:05:12 raft.go:664: INFO: 1 received MsgVoteResp from 1 at term 452
2018/01/17 09:05:12 raft.go:651: INFO: 1 [logterm: 54, index: 221] sent MsgVote request to 2 at term 452
2018/01/17 09:05:12 raft.go:651: INFO: 1 [logterm: 54, index: 221] sent MsgVote request to 3 at term 452
2018/01/17 09:05:12 node.go:191: 		SENDING: MsgVote 1-->2
2018/01/17 09:05:12 node.go:191: 		SENDING: MsgVote 1-->3
2018/01/17 09:05:12 node.go:324: No healthy connection found to node Id: 3, err: Unhealthy connection
2018/01/17 09:05:12 node.go:324: No healthy connection found to node Id: 2, err: Unhealthy connection
2018/01/17 09:05:15 node.go:536: RECEIVED: MsgVote 2-->1
2018/01/17 09:05:15 raft.go:731: INFO: 1 [term: 452] ignored a MsgVote message with lower term from 2 [term: 55]
2018/01/17 09:05:15 raft.go:749: INFO: 1 is starting a new election at term 452
2018/01/17 09:05:15 raft.go:580: INFO: 1 became candidate at term 453
2018/01/17 09:05:15 raft.go:664: INFO: 1 received MsgVoteResp from 1 at term 453
2018/01/17 09:05:15 raft.go:651: INFO: 1 [logterm: 54, index: 221] sent MsgVote request to 2 at term 453
2018/01/17 09:05:15 raft.go:651: INFO: 1 [logterm: 54, index: 221] sent MsgVote request to 3 at term 453
2018/01/17 09:05:15 node.go:191: 		SENDING: MsgVote 1-->2
2018/01/17 09:05:15 node.go:191: 		SENDING: MsgVote 1-->3
2018/01/17 09:05:15 node.go:324: No healthy connection found to node Id: 3, err: Unhealthy connection
2018/01/17 09:05:15 node.go:324: No healthy connection found to node Id: 2, err: Unhealthy connection
2018/01/17 09:05:18 node.go:536: RECEIVED: MsgVote 3-->1
2018/01/17 09:05:18 raft.go:731: INFO: 1 [term: 453] ignored a MsgVote message with lower term from 3 [term: 55]
2018/01/17 09:05:18 raft.go:749: INFO: 1 is starting a new election at term 453
2018/01/17 09:05:18 raft.go:580: INFO: 1 became candidate at term 454
2018/01/17 09:05:18 raft.go:664: INFO: 1 received MsgVoteResp from 1 at term 454
2018/01/17 09:05:18 raft.go:651: INFO: 1 [logterm: 54, index: 221] sent MsgVote request to 2 at term 454
2018/01/17 09:05:18 raft.go:651: INFO: 1 [logterm: 54, index: 221] sent MsgVote request to 3 at term 454
2018/01/17 09:05:18 node.go:191: 		SENDING: MsgVote 1-->2
2018/01/17 09:05:18 node.go:191: 		SENDING: MsgVote 1-->3
2018/01/17 09:05:18 node.go:324: No healthy connection found to node Id: 2, err: Unhealthy connection
2018/01/17 09:05:18 node.go:324: No healthy connection found to node Id: 3, err: Unhealthy connection
2018/01/17 09:05:19 node.go:536: RECEIVED: MsgVote 2-->1
2018/01/17 09:05:19 raft.go:731: INFO: 1 [term: 454] ignored a MsgVote message with lower term from 2 [term: 56]
2018/01/17 09:05:20 oracle.go:375: No healthy connection found to leader of group 2
2018/01/17 09:05:20 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:20 zero.go:316: Got connection request: id:10 addr:"localhost:7090" 
2018/01/17 09:05:20 zero.go:412: Connected
2018/01/17 09:05:22 node.go:536: RECEIVED: MsgVote 2-->1
2018/01/17 09:05:22 raft.go:731: INFO: 1 [term: 454] ignored a MsgVote message with lower term from 2 [term: 57]
2018/01/17 09:05:22 node.go:536: RECEIVED: MsgVote 3-->1
2018/01/17 09:05:22 raft.go:731: INFO: 1 [term: 454] ignored a MsgVote message with lower term from 3 [term: 56]
2018/01/17 09:05:22 raft.go:749: INFO: 1 is starting a new election at term 454
2018/01/17 09:05:22 raft.go:580: INFO: 1 became candidate at term 455
2018/01/17 09:05:22 raft.go:664: INFO: 1 received MsgVoteResp from 1 at term 455
2018/01/17 09:05:22 raft.go:651: INFO: 1 [logterm: 54, index: 221] sent MsgVote request to 2 at term 455
2018/01/17 09:05:22 raft.go:651: INFO: 1 [logterm: 54, index: 221] sent MsgVote request to 3 at term 455
2018/01/17 09:05:22 node.go:191: 		SENDING: MsgVote 1-->2
2018/01/17 09:05:22 node.go:191: 		SENDING: MsgVote 1-->3
2018/01/17 09:05:22 node.go:536: RECEIVED: MsgVoteResp 3-->1
2018/01/17 09:05:22 node.go:536: RECEIVED: MsgVoteResp 2-->1
2018/01/17 09:05:22 raft.go:664: INFO: 1 received MsgVoteResp from 3 at term 455
2018/01/17 09:05:22 raft.go:1013: INFO: 1 [quorum:2] has received 2 MsgVoteResp votes and 0 vote rejections
2018/01/17 09:05:22 raft.go:621: INFO: 1 became leader at term 455
2018/01/17 09:05:22 node.go:301: INFO: raft.node: 1 elected leader 1 at term 455
2018/01/17 09:05:22 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:22 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgProp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgProp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgProp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgProp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgProp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgProp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgProp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgProp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgProp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgProp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgProp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgProp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 zero.go:316: Got connection request: id:11 addr:"localhost:7091" 
2018/01/17 09:05:23 zero.go:412: Connected
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgReadIndex 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgProp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgReadIndexResp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgReadIndex 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgReadIndexResp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgReadIndex 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgReadIndexResp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgReadIndex 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgReadIndexResp 1-->2
2018/01/17 09:05:24 node.go:536: RECEIVED: MsgProp 2-->1
2018/01/17 09:05:24 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:24 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:24 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:24 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:24 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:24 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:24 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:24 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgReadIndex 2-->1
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgReadIndex 2-->1
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgReadIndexResp 1-->2
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgReadIndexResp 1-->2
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgProp 2-->1
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgReadIndex 2-->1
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgReadIndex 2-->1
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgReadIndexResp 1-->2
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgReadIndexResp 1-->2
2018/01/17 09:05:26 zero.go:316: Got connection request: id:12 addr:"localhost:7092" 
2018/01/17 09:05:26 zero.go:412: Connected
2018/01/17 09:05:26 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:26 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:26 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:26 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:26 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:26 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:26 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:26 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:26 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:26 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:26 node.go:536: RECEIVED: MsgReadIndex 2-->1
2018/01/17 09:05:26 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:26 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:26 node.go:191: 		SENDING: MsgReadIndexResp 1-->2
2018/01/17 09:05:26 node.go:536: RECEIVED: MsgReadIndex 2-->1
2018/01/17 09:05:26 node.go:191: 		SENDING: MsgReadIndexResp 1-->2
2018/01/17 09:05:29 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:29 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:29 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:29 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:29 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:29 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:29 node.go:191: 		SENDING: MsgApp 1-->2
2018/01/17 09:05:29 node.go:191: 		SENDING: MsgApp 1-->3
2018/01/17 09:05:29 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:29 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:29 node.go:536: RECEIVED: MsgAppResp 3-->1
2018/01/17 09:05:29 node.go:536: RECEIVED: MsgReadIndex 2-->1
2018/01/17 09:05:29 node.go:191: 		SENDING: MsgReadIndexResp 1-->2
2018/01/17 09:05:29 node.go:536: RECEIVED: MsgReadIndex 2-->1
2018/01/17 09:05:29 node.go:536: RECEIVED: MsgAppResp 2-->1
2018/01/17 09:05:29 node.go:191: 		SENDING: MsgReadIndexResp 1-->2
2018/01/17 09:05:30 oracle.go:375: No healthy connection found to leader of group 3
2018/01/17 09:05:30 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure

jzhu077 · January 16, 2018, 8:08pm

zero 2 logs:

Setting up grpc listener at: 0.0.0.0:7081
Setting up http listener at: 0.0.0.0:8081
2018/01/17 09:05:12 node.go:248: Found hardstate: {Term:54 Vote:2 Commit:221 XXX_unrecognized:[]}
2018/01/17 09:05:12 node.go:260: Group 0 found 221 entries
2018/01/17 09:05:12 raft.go:400: Restarting node for dgraphzero
2018/01/17 09:05:12 raft.go:567: INFO: 2 became follower at term 54
2018/01/17 09:05:12 raft.go:315: INFO: newRaft 2 [peers: [], term: 54, commit: 221, applied: 0, lastindex: 221, lastterm: 54]
Running Dgraph zero...
2018/01/17 09:05:12 pool.go:118: == CONNECT ==> Setting localhost:7080
2018/01/17 09:05:12 node.go:127: Setting conf state to nodes:1 
2018/01/17 09:05:12 node.go:127: Setting conf state to nodes:1 nodes:2 
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7082. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:118: == CONNECT ==> Setting localhost:7082
2018/01/17 09:05:12 node.go:127: Setting conf state to nodes:1 nodes:2 nodes:3 
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:118: == CONNECT ==> Setting localhost:7090
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:118: == CONNECT ==> Setting localhost:7091
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:118: == CONNECT ==> Setting localhost:7092
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:12 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:15 raft.go:749: INFO: 2 is starting a new election at term 54
2018/01/17 09:05:15 raft.go:580: INFO: 2 became candidate at term 55
2018/01/17 09:05:15 raft.go:664: INFO: 2 received MsgVoteResp from 2 at term 55
2018/01/17 09:05:15 raft.go:651: INFO: 2 [logterm: 54, index: 221] sent MsgVote request to 3 at term 55
2018/01/17 09:05:15 raft.go:651: INFO: 2 [logterm: 54, index: 221] sent MsgVote request to 1 at term 55
2018/01/17 09:05:15 node.go:191: 		SENDING: MsgVote 2-->3
2018/01/17 09:05:15 node.go:191: 		SENDING: MsgVote 2-->1
2018/01/17 09:05:15 node.go:324: No healthy connection found to node Id: 3, err: Unhealthy connection
2018/01/17 09:05:18 node.go:536: RECEIVED: MsgVote 3-->2
2018/01/17 09:05:18 raft.go:772: INFO: 2 [logterm: 54, index: 221, vote: 2] rejected MsgVote from 3 [logterm: 54, index: 221] at term 55
2018/01/17 09:05:18 node.go:191: 		SENDING: MsgVoteResp 2-->3
2018/01/17 09:05:18 node.go:324: No healthy connection found to node Id: 3, err: Unhealthy connection
2018/01/17 09:05:19 raft.go:749: INFO: 2 is starting a new election at term 55
2018/01/17 09:05:19 raft.go:580: INFO: 2 became candidate at term 56
2018/01/17 09:05:19 raft.go:664: INFO: 2 received MsgVoteResp from 2 at term 56
2018/01/17 09:05:19 raft.go:651: INFO: 2 [logterm: 54, index: 221] sent MsgVote request to 3 at term 56
2018/01/17 09:05:19 raft.go:651: INFO: 2 [logterm: 54, index: 221] sent MsgVote request to 1 at term 56
2018/01/17 09:05:19 node.go:191: 		SENDING: MsgVote 2-->3
2018/01/17 09:05:19 node.go:191: 		SENDING: MsgVote 2-->1
2018/01/17 09:05:19 node.go:324: No healthy connection found to node Id: 3, err: Unhealthy connection
2018/01/17 09:05:22 raft.go:749: INFO: 2 is starting a new election at term 56
2018/01/17 09:05:22 raft.go:580: INFO: 2 became candidate at term 57
2018/01/17 09:05:22 raft.go:664: INFO: 2 received MsgVoteResp from 2 at term 57
2018/01/17 09:05:22 raft.go:651: INFO: 2 [logterm: 54, index: 221] sent MsgVote request to 1 at term 57
2018/01/17 09:05:22 raft.go:651: INFO: 2 [logterm: 54, index: 221] sent MsgVote request to 3 at term 57
2018/01/17 09:05:22 node.go:191: 		SENDING: MsgVote 2-->1
2018/01/17 09:05:22 node.go:191: 		SENDING: MsgVote 2-->3
2018/01/17 09:05:22 node.go:324: No healthy connection found to node Id: 3, err: Unhealthy connection
2018/01/17 09:05:22 node.go:536: RECEIVED: MsgVote 3-->2
2018/01/17 09:05:22 raft.go:731: INFO: 2 [term: 57] ignored a MsgVote message with lower term from 3 [term: 56]
2018/01/17 09:05:22 oracle.go:375: No healthy connection found to leader of group 1
2018/01/17 09:05:22 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:22 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:22 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:22 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:22 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:22 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:22 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:22 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:22 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:22 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:22 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:22 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:22 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:22 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:22 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:22 node.go:536: RECEIVED: MsgVote 1-->2
2018/01/17 09:05:22 raft.go:708: INFO: 2 [term: 57] received a MsgVote message with higher term from 1 [term: 455]
2018/01/17 09:05:22 raft.go:567: INFO: 2 became follower at term 455
2018/01/17 09:05:22 raft.go:763: INFO: 2 [logterm: 54, index: 221, vote: 0] cast MsgVote for 1 [logterm: 54, index: 221] at term 455
2018/01/17 09:05:22 node.go:191: 		SENDING: MsgVoteResp 2-->1
2018/01/17 09:05:22 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:22 node.go:301: INFO: raft.node: 2 elected leader 1 at term 455
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgProp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgProp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgProp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgProp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgProp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgProp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgReadIndex 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgProp 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgReadIndexResp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgReadIndex 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgReadIndexResp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgReadIndex 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgReadIndexResp 1-->2
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgReadIndex 2-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgReadIndexResp 1-->2
2018/01/17 09:05:24 node.go:191: 		SENDING: MsgProp 2-->1
2018/01/17 09:05:24 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:24 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:24 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgReadIndex 2-->1
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgReadIndex 2-->1
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgReadIndexResp 1-->2
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgReadIndexResp 1-->2
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgProp 2-->1
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgReadIndex 2-->1
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgReadIndex 2-->1
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgReadIndexResp 1-->2
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgReadIndexResp 1-->2
2018/01/17 09:05:26 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:26 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:26 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:26 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:26 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:26 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:26 node.go:191: 		SENDING: MsgReadIndex 2-->1
2018/01/17 09:05:26 node.go:191: 		SENDING: MsgReadIndex 2-->1
2018/01/17 09:05:26 node.go:536: RECEIVED: MsgReadIndexResp 1-->2
2018/01/17 09:05:26 node.go:536: RECEIVED: MsgReadIndexResp 1-->2
2018/01/17 09:05:29 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:29 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:29 node.go:536: RECEIVED: MsgApp 1-->2
2018/01/17 09:05:29 node.go:191: 		SENDING: MsgAppResp 2-->1
2018/01/17 09:05:29 node.go:191: 		SENDING: MsgReadIndex 2-->1
2018/01/17 09:05:29 node.go:191: 		SENDING: MsgReadIndex 2-->1
2018/01/17 09:05:29 node.go:536: RECEIVED: MsgReadIndexResp 1-->2
2018/01/17 09:05:29 node.go:536: RECEIVED: MsgReadIndexResp 1-->2
2018/01/17 09:05:32 oracle.go:375: No healthy connection found to leader of group 2
2018/01/17 09:06:23 node.go:191: 		SENDING: MsgReadIndex 2-->1
2018/01/17 09:06:23 node.go:536: RECEIVED: MsgReadIndexResp 1-->2
2018/01/17 09:06:23 node.go:191: 		SENDING: MsgReadIndex 2-->1
2018/01/17 09:06:23 node.go:536: RECEIVED: MsgReadIndexResp 1-->2
2018/01/17 09:07:23 node.go:191: 		SENDING: MsgReadIndex 2-->1
2018/01/17 09:07:23 node.go:536: RECEIVED: MsgReadIndexResp 1-->2
2018/01/17 09:07:23 node.go:191: 		SENDING: MsgReadIndex 2-->1
2018/01/17 09:07:23 node.go:536: RECEIVED: MsgReadIndexResp 1-->2

jzhu077 · January 16, 2018, 8:09pm

zero 3 logs:

Setting up grpc listener at: 0.0.0.0:7082
Setting up http listener at: 0.0.0.0:8082
2018/01/17 09:05:15 node.go:248: Found hardstate: {Term:54 Vote:1 Commit:221 XXX_unrecognized:[]}
2018/01/17 09:05:15 node.go:260: Group 0 found 221 entries
2018/01/17 09:05:15 raft.go:400: Restarting node for dgraphzero
2018/01/17 09:05:15 raft.go:567: INFO: 3 became follower at term 54
2018/01/17 09:05:15 raft.go:315: INFO: newRaft 3 [peers: [], term: 54, commit: 221, applied: 0, lastindex: 221, lastterm: 54]
Running Dgraph zero...
2018/01/17 09:05:16 pool.go:118: == CONNECT ==> Setting localhost:7080
2018/01/17 09:05:16 node.go:127: Setting conf state to nodes:1 
2018/01/17 09:05:16 pool.go:118: == CONNECT ==> Setting localhost:7081
2018/01/17 09:05:16 node.go:127: Setting conf state to nodes:1 nodes:2 
2018/01/17 09:05:16 node.go:127: Setting conf state to nodes:1 nodes:2 nodes:3 
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:118: == CONNECT ==> Setting localhost:7091
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:118: == CONNECT ==> Setting localhost:7090
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:118: == CONNECT ==> Setting localhost:7092
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7090. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:16 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:18 raft.go:749: INFO: 3 is starting a new election at term 54
2018/01/17 09:05:18 raft.go:580: INFO: 3 became candidate at term 55
2018/01/17 09:05:18 raft.go:664: INFO: 3 received MsgVoteResp from 3 at term 55
2018/01/17 09:05:18 raft.go:651: INFO: 3 [logterm: 54, index: 221] sent MsgVote request to 1 at term 55
2018/01/17 09:05:18 raft.go:651: INFO: 3 [logterm: 54, index: 221] sent MsgVote request to 2 at term 55
2018/01/17 09:05:18 node.go:191: 		SENDING: MsgVote 3-->1
2018/01/17 09:05:18 node.go:191: 		SENDING: MsgVote 3-->2
2018/01/17 09:05:22 raft.go:749: INFO: 3 is starting a new election at term 55
2018/01/17 09:05:22 raft.go:580: INFO: 3 became candidate at term 56
2018/01/17 09:05:22 raft.go:664: INFO: 3 received MsgVoteResp from 3 at term 56
2018/01/17 09:05:22 raft.go:651: INFO: 3 [logterm: 54, index: 221] sent MsgVote request to 1 at term 56
2018/01/17 09:05:22 raft.go:651: INFO: 3 [logterm: 54, index: 221] sent MsgVote request to 2 at term 56
2018/01/17 09:05:22 node.go:191: 		SENDING: MsgVote 3-->1
2018/01/17 09:05:22 node.go:191: 		SENDING: MsgVote 3-->2
2018/01/17 09:05:22 node.go:536: RECEIVED: MsgVote 1-->3
2018/01/17 09:05:22 raft.go:708: INFO: 3 [term: 56] received a MsgVote message with higher term from 1 [term: 455]
2018/01/17 09:05:22 raft.go:567: INFO: 3 became follower at term 455
2018/01/17 09:05:22 raft.go:763: INFO: 3 [logterm: 54, index: 221, vote: 0] cast MsgVote for 1 [logterm: 54, index: 221] at term 455
2018/01/17 09:05:22 node.go:191: 		SENDING: MsgVoteResp 3-->1
2018/01/17 09:05:22 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:22 node.go:301: INFO: raft.node: 3 elected leader 1 at term 455
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgProp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgProp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgProp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgProp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgProp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgProp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:23 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:23 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:24 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:24 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:24 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:24 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:24 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:25 oracle.go:375: No healthy connection found to leader of group 1
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:25 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:25 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:26 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:26 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:26 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:26 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:26 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:26 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:26 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:26 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:26 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:26 pool.go:167: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:26 pool.go:167: Echo error from localhost:7091. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/01/17 09:05:26 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:26 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:26 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:26 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:26 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:26 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:29 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:29 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:29 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:29 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:29 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:29 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:29 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:05:29 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:05:35 oracle.go:375: No healthy connection found to leader of group 3
2018/01/17 09:08:00 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:08:00 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:08:00 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:08:00 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:08:00 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:08:00 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:08:00 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:08:00 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:08:00 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:08:00 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:08:00 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:08:00 node.go:536: RECEIVED: MsgApp 1-->3
2018/01/17 09:08:00 node.go:191: 		SENDING: MsgAppResp 3-->1
2018/01/17 09:08:01 node.go:191: 		SENDING: MsgAppResp 3-->1

jzhu077 · January 16, 2018, 10:24pm

Is there a way to recover from the no leader issue?

janardhan · January 16, 2018, 11:40pm

The cluster should automatically recover ideally. I will go through the logs and try to see what is the issue and get back to you

jzhu077 · January 17, 2018, 3:19am

Any update on the issue? Also, what would be the ETA on the patch release for the issue log.Fatal caused by unexpected meta: 0 ? Just need to plan ahead

pawan · January 17, 2018, 4:08am

Yeah, we are doing a release today(v1.0.2) which contains the fix. ETA 2-3 hrs.

Kbrochier · January 17, 2018, 3:07pm

Is there no nighly today ?

root@ubu-16:/home/dgraph# curl https://nightly.dgraph.io -sSf | bash
curl: (22) The requested URL returned error: 404 Not Found

Kbrochier · January 17, 2018, 3:50pm

Solved

Oh sorry - saw there is a new version in main

janardhan · January 17, 2018, 10:16pm

You can get the nightly build with the following command.
curl https://get.dgraph.io -sSf | bash -s nightly

We have updated the docs to reflect the same

Kbrochier · January 17, 2018, 10:35pm

Thank you - first idea was there is none because of the new version in main

jzhu077 · January 22, 2018, 3:45am

Is there any update on the issue?

Topic		Replies	Views
Frequent Zero leadership change Issues dgraph , kind:bug	2	444	May 21, 2021
Getting issue with Zero cluster Dgraph kind:question , dgraph	2	343	January 27, 2021
Mutation failed because Dgraph execution: Unhealthy connection GraphQL dgraph	3	648	December 28, 2020
Error - Unable to join cluster via dgraphzero Users	3	842	January 17, 2018
After the alpha leader's data is corrupted, the node cannot join the cluster as a new node Dgraph kind:question	2	613	May 10, 2021

Dgraph runs into a error loop and freezes the host

Related topics