Nil Pointer Errors When Running High Volume of Data

Users Dgraph
1

I am running Dgraph 1.0.5 in a cluster of 5, replication set to 3, with one dgraphzero. Twice now, I have seen my responses hang when calling dgraph, and my dgraph02 machine dies (same size as every other machine) The logs before the error were similar both times:

First time:

Jun 05 19:00:21 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]: 2018/06/05 19:00:21 groups.go:356: Serving tablet for: z
Jun 05 19:00:21 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]: 2018/06/05 19:00:21 groups.go:356: Serving tablet for: z
Jun 05 19:00:21 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]: 2018/06/05 19:00:21 groups.go:356: Serving tablet for: EVENT_externalId
Jun 05 19:00:33 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]: 2018/06/05 19:00:33 draft.go:865: Writing snapshot at index: 2208590, applied mark: 2210482
Jun 05 19:01:33 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]: 2018/06/05 19:01:33 draft.go:865: Writing snapshot at index: 2215465, applied mark: 2219335
Jun 05 19:02:03 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]: 2018/06/05 19:02:03 index.go:937: Dropping predicate: [EVENT_id]
Jun 05 19:02:03 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]: 2018/06/05 19:02:03 draft.go:865: Writing snapshot at index: 2217694, applied mark: 2223958
Jun 05 19:02:08 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]: 2018/06/05 19:02:08 schema.go:63: Deleting schema for predicate: [EVENT_id]
Jun 05 19:02:33 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]: 2018/06/05 19:02:33 draft.go:865: Writing snapshot at index: 2225987, applied mark: 2229784
Jun 05 19:03:33 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]: 2018/06/05 19:03:33 draft.go:865: Writing snapshot at index: 2238177, applied mark: 2238563
Jun 05 19:04:33 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]: 2018/06/05 19:04:33 draft.go:865: Writing snapshot at index: 2242721, applied mark: 2247884
Jun 05 19:05:33 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]: 2018/06/05 19:05:33 draft.go:865: Writing snapshot at index: 2254370, applied mark: 2257943
Jun 05 19:06:03 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]: 2018/06/05 19:06:03 draft.go:865: Writing snapshot at index: 2261189, applied mark: 2263826
Jun 05 19:07:03 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]: 2018/06/05 19:07:03 draft.go:865: Writing snapshot at index: 2264801, applied mark: 2274094
Jun 05 19:07:33 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]: 2018/06/05 19:07:33 draft.go:865: Writing snapshot at index: 2275280, applied mark: 2278525
Jun 05 19:08:03 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]: 2018/06/05 19:08:03 draft.go:865: Writing snapshot at index: 2277740, applied mark: 2285825
Jun 05 19:08:26 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]: panic: runtime error: invalid memory address or nil pointer dereference
Jun 05 19:08:26 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x10125b8]
Jun 05 19:08:26 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]: goroutine 23306549 [running]:
Jun 05 19:08:26 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]: github.com/dgraph-io/dgraph/posting.(*List).Value(0x0, 0x4da1b8, 0x0, 0x0, 0x0, 0x0, 0x0)
Jun 05 19:08:26 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]:         /home/travis/gopath/src/github.com/dgraph-io/dgraph/posting/list.go:935 +0x58
Jun 05 19:08:26 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]: github.com/dgraph-io/dgraph/worker.handleCompareFunction.func1(0x4aa3b, 0x0, 0xc4c061a5b0)
Jun 05 19:08:26 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]:         /home/travis/gopath/src/github.com/dgraph-io/dgraph/worker/task.go:911 +0x31f
Jun 05 19:08:26 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]: github.com/dgraph-io/dgraph/algo.ApplyFilter(0xc49fc12b00, 0xc50d58d838)
Jun 05 19:08:26 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]:         /home/travis/gopath/src/github.com/dgraph-io/dgraph/algo/uidlist.go:24 +0xb3
Jun 05 19:08:26 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]: github.com/dgraph-io/dgraph/worker.handleCompareFunction(0x1a821e0, 0xc4a87f42a0, 0xc4777c4600, 0x1, 0xc494fbad10, 0xc4afbba000, 0x4da1b8, 0x0)
Jun 05 19:08:26 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]:         /home/travis/gopath/src/github.com/dgraph-io/dgraph/worker/task.go:888 +0x4d2
Jun 05 19:08:26 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]: github.com/dgraph-io/dgraph/worker.helpProcessTask(0x1a821e0, 0xc4a87f42a0, 0xc4777c4600, 0x1, 0x0, 0x0, 0x139b2d8)
Jun 05 19:08:26 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]:         /home/travis/gopath/src/github.com/dgraph-io/dgraph/worker/task.go:715 +0x429
Jun 05 19:08:26 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]: github.com/dgraph-io/dgraph/worker.processTask(0x1a821e0, 0xc4a87f42a0, 0xc4777c4600, 0x1, 0xc4604be700, 0xc4777c4600, 0xc46886cb48)
Jun 05 19:08:26 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]:         /home/travis/gopath/src/github.com/dgraph-io/dgraph/worker/task.go:626 +0x1be
Jun 05 19:08:26 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]: github.com/dgraph-io/dgraph/worker.ProcessTaskOverNetwork(0x1a821e0, 0xc4a87f42a0, 0xc4777c4600, 0x0, 0xffffffffffffffff, 0xc42002d400)
Jun 05 19:08:26 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]:         /home/travis/gopath/src/github.com/dgraph-io/dgraph/worker/task.go:149 +0x546
Jun 05 19:08:26 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]: github.com/dgraph-io/dgraph/query.ProcessGraph(0x1a821e0, 0xc4a87f42a0, 0xc4bf386000, 0x0, 0xc4b3487bc0)
Jun 05 19:08:26 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]:         /home/travis/gopath/src/github.com/dgraph-io/dgraph/query/query.go:1885 +0x1cbd
Jun 05 19:08:26 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]: created by github.com/dgraph-io/dgraph/query.(*QueryRequest).ProcessQuery
Jun 05 19:08:26 dgraph-sigstr-02.c.charlotte-161616.internal docker[5822]:         /home/travis/gopath/src/github.com/dgraph-io/dgraph/query/query.go:2518 +0x964

Second Time:

2018/06/05 20:10:24 draft.go:845: Couldn't take snapshot, txn watermark: [2462749], applied watermark: [2479024]
Jun 05 20:10:54 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]: 2018/06/05 20:10:54 draft.go:865: Writing snapshot at index: 2474870, applied mark: 2484122
Jun 05 20:11:24 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]: 2018/06/05 20:11:24 draft.go:865: Writing snapshot at index: 2489341, applied mark: 2489673
Jun 05 20:12:24 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]: 2018/06/05 20:12:24 draft.go:865: Writing snapshot at index: 2501377, applied mark: 2501399
Jun 05 20:12:54 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]: 2018/06/05 20:12:54 draft.go:865: Writing snapshot at index: 2502262, applied mark: 2504029
Jun 05 20:13:24 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]: 2018/06/05 20:13:24 draft.go:865: Writing snapshot at index: 2506531, applied mark: 2506543
Jun 05 20:13:54 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]: 2018/06/05 20:13:54 draft.go:865: Writing snapshot at index: 2509138, applied mark: 2509178
Jun 05 20:14:24 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]: 2018/06/05 20:14:24 draft.go:865: Writing snapshot at index: 2510225, applied mark: 2516378
Jun 05 20:15:24 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]: 2018/06/05 20:15:24 draft.go:865: Writing snapshot at index: 2513595, applied mark: 2520166
Jun 05 20:15:54 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]: 2018/06/05 20:15:54 draft.go:865: Writing snapshot at index: 2519810, applied mark: 2522126
Jun 05 20:16:24 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]: 2018/06/05 20:16:24 draft.go:865: Writing snapshot at index: 2525757, applied mark: 2525956
Jun 05 20:18:54 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]: 2018/06/05 20:18:54 index.go:937: Dropping predicate: [PROFILE_id]
Jun 05 20:18:56 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]: 2018/06/05 20:18:56 schema.go:63: Deleting schema for predicate: [PROFILE_id]
Jun 05 20:19:54 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]: 2018/06/05 20:19:54 draft.go:865: Writing snapshot at index: 2535073, applied mark: 2535083
Jun 05 20:20:24 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]: 2018/06/05 20:20:24 draft.go:865: Writing snapshot at index: 2536331, applied mark: 2536342
Jun 05 20:20:54 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]: 2018/06/05 20:20:54 draft.go:865: Writing snapshot at index: 2537539, applied mark: 2537549
Jun 05 20:21:24 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]: 2018/06/05 20:21:24 draft.go:865: Writing snapshot at index: 2537776, applied mark: 2538640
Jun 05 20:22:24 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]: 2018/06/05 20:22:24 draft.go:865: Writing snapshot at index: 2541897, applied mark: 2541914
Jun 05 20:22:54 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]: 2018/06/05 20:22:54 draft.go:865: Writing snapshot at index: 2544234, applied mark: 2544252
Jun 05 20:23:24 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]: 2018/06/05 20:23:24 draft.go:865: Writing snapshot at index: 2545097, applied mark: 2546227
Jun 05 20:24:24 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]: 2018/06/05 20:24:24 draft.go:865: Writing snapshot at index: 2549771, applied mark: 2549781
Jun 05 20:25:24 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]: 2018/06/05 20:25:24 draft.go:865: Writing snapshot at index: 2549773, applied mark: 2549854
Jun 05 20:26:24 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]: 2018/06/05 20:26:24 draft.go:865: Writing snapshot at index: 2549849, applied mark: 2549859
Jun 05 20:26:54 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]: 2018/06/05 20:26:54 draft.go:865: Writing snapshot at index: 2550765, applied mark: 2550811
Jun 05 20:26:54 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]: panic: runtime error: invalid memory address or nil pointer dereference
Jun 05 20:26:54 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x10125b8]
Jun 05 20:26:54 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]: goroutine 37578642 [running]:
Jun 05 20:26:54 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]: github.com/dgraph-io/dgraph/posting.(*List).Value(0x0, 0x568d32, 0x0, 0x0, 0x0, 0x0, 0x0)
Jun 05 20:26:54 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]:         /home/travis/gopath/src/github.com/dgraph-io/dgraph/posting/list.go:935 +0x58
Jun 05 20:26:54 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]: github.com/dgraph-io/dgraph/worker.handleCompareFunction.func1(0x4aa3b, 0x0, 0xc43ac359f0)
Jun 05 20:26:54 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]:         /home/travis/gopath/src/github.com/dgraph-io/dgraph/worker/task.go:911 +0x31f
Jun 05 20:26:54 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]: github.com/dgraph-io/dgraph/algo.ApplyFilter(0xc42c1643c0, 0xc4a73b9838)
Jun 05 20:26:54 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]:         /home/travis/gopath/src/github.com/dgraph-io/dgraph/algo/uidlist.go:24 +0xb3
Jun 05 20:26:54 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]: github.com/dgraph-io/dgraph/worker.handleCompareFunction(0x1a821e0, 0xc48fa2e1b0, 0xc4924b6d00, 0x1, 0xc4683d8210, 0xc485ebcc60, 0x568d32, 0x0)
Jun 05 20:26:54 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]:         /home/travis/gopath/src/github.com/dgraph-io/dgraph/worker/task.go:888 +0x4d2
Jun 05 20:26:54 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]: github.com/dgraph-io/dgraph/worker.helpProcessTask(0x1a821e0, 0xc48fa2e1b0, 0xc4924b6d00, 0x1, 0x0, 0x0, 0x139b2d8)
Jun 05 20:26:54 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]:         /home/travis/gopath/src/github.com/dgraph-io/dgraph/worker/task.go:715 +0x429
Jun 05 20:26:54 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]: github.com/dgraph-io/dgraph/worker.processTask(0x1a821e0, 0xc48fa2e1b0, 0xc4924b6d00, 0x1, 0xc42b12a200, 0xc4924b6d00, 0xc549e64b48)
Jun 05 20:26:54 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]:         /home/travis/gopath/src/github.com/dgraph-io/dgraph/worker/task.go:626 +0x1be
Jun 05 20:26:54 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]: github.com/dgraph-io/dgraph/worker.ProcessTaskOverNetwork(0x1a821e0, 0xc48fa2e1b0, 0xc4924b6d00, 0x0, 0xffffffffffffffff, 0x8ce846)
Jun 05 20:26:54 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]:         /home/travis/gopath/src/github.com/dgraph-io/dgraph/worker/task.go:149 +0x546
Jun 05 20:26:54 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]: github.com/dgraph-io/dgraph/query.ProcessGraph(0x1a821e0, 0xc48fa2e1b0, 0xc4d7c84840, 0x0, 0xc43f4929c0)
Jun 05 20:26:54 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]:         /home/travis/gopath/src/github.com/dgraph-io/dgraph/query/query.go:1885 +0x1cbd
Jun 05 20:26:54 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]: created by github.com/dgraph-io/dgraph/query.(*QueryRequest).ProcessQuery
Jun 05 20:26:54 dgraph-sigstr-02.c.charlotte-161616.internal docker[8052]:         /home/travis/gopath/src/github.com/dgraph-io/dgraph/query/query.go:2518 +0x964
2

After it restarted the first time, it immediately ran out of memory. I have the mb set to 20G with 56G of ram actually available per machine. Should be more than enough?

Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: 2018/06/05 20:08:22 raft.go:567: INFO: 2 became follower at term 17
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: 2018/06/05 20:08:22 log.go:120: INFO: found conflict at index 2461890 [existing term: 15, conflicting term: 17]
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: 2018/06/05 20:08:22 log_unstable.go:131: INFO: replace the unstable entries from index 2461890
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: 2018/06/05 20:08:22 node.go:303: INFO: raft.node: 2 changed leader from 2 to 3 at term 17
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: fatal error: runtime: out of memory
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: runtime stack:
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: runtime.throw(0x136cde9, 0x16)
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         /home/travis/.gimme/versions/go1.9.4.linux.amd64/src/runtime/panic.go:605 +0x95
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: runtime.sysMap(0xd05b040000, 0x2350000, 0x100381600, 0x1b1cc38)
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         /home/travis/.gimme/versions/go1.9.4.linux.amd64/src/runtime/mem_linux.go:216 +0x1d0
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: runtime.(*mheap).sysAlloc(0x1b03680, 0x2350000, 0x1002a43c1)
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         /home/travis/.gimme/versions/go1.9.4.linux.amd64/src/runtime/malloc.go:470 +0xd7
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: runtime.(*mheap).grow(0x1b03680, 0x11a1, 0x0)
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         /home/travis/.gimme/versions/go1.9.4.linux.amd64/src/runtime/mheap.go:887 +0x60
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: runtime.(*mheap).allocSpanLocked(0x1b03680, 0x11a1, 0x1b1cc48, 0xc4794e7c80)
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         /home/travis/.gimme/versions/go1.9.4.linux.amd64/src/runtime/mheap.go:800 +0x334
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: runtime.(*mheap).alloc_m(0x1b03680, 0x11a1, 0xc53fdf0101, 0xc4260c4180)
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         /home/travis/.gimme/versions/go1.9.4.linux.amd64/src/runtime/mheap.go:666 +0x118
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: runtime.(*mheap).alloc.func1()
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         /home/travis/.gimme/versions/go1.9.4.linux.amd64/src/runtime/mheap.go:733 +0x4d
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: runtime.systemstack(0x7f250ce7ee10)
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         /home/travis/.gimme/versions/go1.9.4.linux.amd64/src/runtime/asm_amd64.s:360 +0xab
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: runtime.(*mheap).alloc(0x1b03680, 0x11a1, 0x7f250c010101, 0x8d1674)
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         /home/travis/.gimme/versions/go1.9.4.linux.amd64/src/runtime/mheap.go:732 +0xa1
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: runtime.largeAlloc(0x2340ffc, 0x7f2531820101, 0x100000001099a0e)
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         /home/travis/.gimme/versions/go1.9.4.linux.amd64/src/runtime/malloc.go:827 +0x98
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: runtime.mallocgc.func1()
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         /home/travis/.gimme/versions/go1.9.4.linux.amd64/src/runtime/malloc.go:722 +0x46
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: runtime.systemstack(0xc4203bb300)
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         /home/travis/.gimme/versions/go1.9.4.linux.amd64/src/runtime/asm_amd64.s:344 +0x79
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: runtime.mstart()
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         /home/travis/.gimme/versions/go1.9.4.linux.amd64/src/runtime/proc.go:1135
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: goroutine 41459889 [running]:
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: runtime.systemstack_switch()
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         /home/travis/.gimme/versions/go1.9.4.linux.amd64/src/runtime/asm_amd64.s:298 fp=0xc54985d5b8 sp=0xc54985d5b0 pc=0x917fc0
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: runtime.mallocgc(0x2340ffc, 0x1218360, 0x1, 0x926862)
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         /home/travis/.gimme/versions/go1.9.4.linux.amd64/src/runtime/malloc.go:721 +0x7ae fp=0xc54985d660 sp=0xc54985d5b8 pc=0x8cdcee
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: runtime.makeslice(0x1218360, 0x0, 0x2340ffc, 0x1a7efe0, 0xc4463c6d60, 0x13518a0)
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         /home/travis/.gimme/versions/go1.9.4.linux.amd64/src/runtime/slice.go:54 +0x77 fp=0xc54985d690 sp=0xc54985d660 pc=0x9017a7
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: google.golang.org/grpc.protoCodec.marshal(0x12dc5a0, 0xc4463c6d60, 0xc4d6f22000, 0xc485278e80, 0x8bd8d2, 0xc426626058, 0x2, 0xc426626058)
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         /home/travis/gopath/src/google.golang.org/grpc/codec.go:59 +0x7e fp=0xc54985d6e8 sp=0xc54985d690 pc=0xe5daae
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: google.golang.org/grpc.protoCodec.Marshal(0x12dc5a0, 0xc4463c6d60, 0x8e7aa8, 0x756ea13cf17695, 0x512d0a387d522bdb, 0xc54985d7c0, 0x8bd9ad)
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         /home/travis/gopath/src/google.golang.org/grpc/codec.go:73 +0x6e fp=0xc54985d768 sp=0xc54985d6e8 pc=0xe5dc4e
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: google.golang.org/grpc.(*protoCodec).Marshal(0x1b1b0c0, 0x12dc5a0, 0xc4463c6d60, 0x3, 0xc42001a600, 0xc42001a670, 0xc54985d888, 0x8cd817)
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         <autogenerated>:1 +0x48 fp=0xc54985d7b0 sp=0xc54985d768 pc=0xe76f18
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: google.golang.org/grpc.encode(0x1a819e0, 0x1b1b0c0, 0x12dc5a0, 0xc4463c6d60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         /home/travis/gopath/src/google.golang.org/grpc/rpc_util.go:327 +0x2a5 fp=0xc54985d8a8 sp=0xc54985d7b0 pc=0xe62365
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: google.golang.org/grpc.sendRequest(0x1a821e0, 0xc50cd524b0, 0x0, 0x0, 0x1a819e0, 0x1b1b0c0, 0x0, 0x0, 0x0, 0x0, ...)
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         /home/travis/gopath/src/google.golang.org/grpc/call.go:119 +0x180 fp=0xc54985d9a0 sp=0xc54985d8a8 pc=0xe56bc0
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: google.golang.org/grpc.invoke(0x1a821e0, 0xc50cd524b0, 0x136dfd1, 0x18, 0x12dc5a0, 0xc4463c6d60, 0x12dc5a0, 0xc4463c6d80, 0xc436758280, 0x0, ...)
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         /home/travis/gopath/src/google.golang.org/grpc/call.go:277 +0x803 fp=0xc54985dd70 sp=0xc54985d9a0 pc=0xe57b93
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: google.golang.org/grpc.(*ClientConn).Invoke(0xc436758280, 0x1a821a0, 0xc47ec23e60, 0x136dfd1, 0x18, 0x12dc5a0, 0xc4463c6d60, 0x12dc5a0, 0xc4463c6d80, 0x0, ...)
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         /home/travis/gopath/src/google.golang.org/grpc/call.go:150 +0x19c fp=0xc54985ddf8 sp=0xc54985dd70 pc=0xe5726c
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: google.golang.org/grpc.Invoke(0x1a821a0, 0xc47ec23e60, 0x136dfd1, 0x18, 0x12dc5a0, 0xc4463c6d60, 0x12dc5a0, 0xc4463c6d80, 0xc436758280, 0x0, ...)
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         /home/travis/gopath/src/google.golang.org/grpc/call.go:158 +0xc1 fp=0xc54985de78 sp=0xc54985ddf8 pc=0xe57361
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: github.com/dgraph-io/dgraph/protos/intern.(*raftClient).RaftMessage(0xc4f1126150, 0x1a821a0, 0xc47ec23e60, 0xc4463c6d60, 0x0, 0x0, 0x0, 0xc5114c8228, 0xc50b541f78, 0xc2f9ee)
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         /home/travis/gopath/src/github.com/dgraph-io/dgraph/protos/intern/internal.pb.go:1950 +0xcb fp=0xc54985df00 sp=0xc54985de78 pc=0xe9cfbb
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: github.com/dgraph-io/dgraph/conn.(*Node).doSendMessage.func1(0x1a823e0, 0xc4f1126150, 0x1a821a0, 0xc47ec23e60, 0xc4463c6d60, 0xc431552690, 0xc47ec23f80)
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         /home/travis/gopath/src/github.com/dgraph-io/dgraph/conn/node.go:323 +0x8b fp=0xc54985dfa8 sp=0xc54985df00 pc=0x109c9db
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: runtime.goexit()
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         /home/travis/.gimme/versions/go1.9.4.linux.amd64/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc54985dfb0 sp=0xc54985dfa8 pc=0x91abf1
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: created by github.com/dgraph-io/dgraph/conn.(*Node).doSendMessage
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         /home/travis/gopath/src/github.com/dgraph-io/dgraph/conn/node.go:322 +0x1a6
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: goroutine 1 [semacquire, 59 minutes]:
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: sync.runtime_Semacquire(0xc42048028c)
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         /home/travis/.gimme/versions/go1.9.4.linux.amd64/src/runtime/sema.go:56 +0x39
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: sync.(*WaitGroup).Wait(0xc420480280)
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         /home/travis/.gimme/versions/go1.9.4.linux.amd64/src/sync/waitgroup.go:131 +0x72
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: github.com/dgraph-io/dgraph/dgraph/cmd/server.setupServer()
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         /home/travis/gopath/src/github.com/dgraph-io/dgraph/dgraph/cmd/server/run.go:272 +0x633
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: github.com/dgraph-io/dgraph/dgraph/cmd/server.run()
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         /home/travis/gopath/src/github.com/dgraph-io/dgraph/dgraph/cmd/server/run.go:356 +0x903
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: github.com/dgraph-io/dgraph/dgraph/cmd/server.init.0.func1(0xc420091440, 0xc42039c7d0, 0x0, 0x1)
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         /home/travis/gopath/src/github.com/dgraph-io/dgraph/dgraph/cmd/server/run.go:54 +0x52
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: github.com/dgraph-io/dgraph/vendor/github.com/spf13/cobra.(*Command).execute(0xc420091440, 0xc42039c7b0, 0x1, 0x1, 0xc420091440, 0xc42039c7b0)
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         /home/travis/gopath/src/github.com/dgraph-io/dgraph/vendor/github.com/spf13/cobra/command.go:702 +0x2c6
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: github.com/dgraph-io/dgraph/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x1a6dc20, 0xc42014fee8, 0x8fcb30, 0x1a752f0)
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         /home/travis/gopath/src/github.com/dgraph-io/dgraph/vendor/github.com/spf13/cobra/command.go:783 +0x30e
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: github.com/dgraph-io/dgraph/vendor/github.com/spf13/cobra.(*Command).Execute(0x1a6dc20, 0x1a752f0, 0x15355931c89ab601)
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         /home/travis/gopath/src/github.com/dgraph-io/dgraph/vendor/github.com/spf13/cobra/command.go:736 +0x2b
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: github.com/dgraph-io/dgraph/dgraph/cmd.Execute()
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]:         /home/travis/gopath/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:42 +0x31
Jun 05 20:08:22 dgraph-sigstr-02.c.charlotte-161616.internal docker[6535]: main.main()
3

I am also frequently getting

rpc error: code = Unknown desc = : Unable to read from value log: {Fid:0 Len:84 Offset:782730681}: Unable to find log file. Please retry

when running queries.