Fatal error: runtime: out of memory when live loader

Users Dgraph
1

hi
I’ve deployed DGraph based on the documentation using Docker Swarm
All three machines are 2-core 4G

The following error occurred while using the Live Loader import

Found 1 data file(s) to process
Processing data file "A100_unlisted_1.json.gz"
[10:41:03Z] Elapsed: 05s Txns: 0 N-Quads: 0 N-Quads/s [last 5s]:     0 Aborts: 0
[10:41:08Z] Elapsed: 10s Txns: 0 N-Quads: 0 N-Quads/s [last 5s]:     0 Aborts: 0
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x189bbf8, 0x16)
        /usr/local/go/src/runtime/panic.go:774 +0x72
runtime.sysMap(0xc054000000, 0x44000000, 0x24a4298)
        /usr/local/go/src/runtime/mem_linux.go:169 +0xc5
runtime.(*mheap).sysAlloc(0x248af20, 0x411d6000, 0x7f168dffaca8, 0x97bc27)
        /usr/local/go/src/runtime/malloc.go:701 +0x1cd
runtime.(*mheap).grow(0x248af20, 0x208eb, 0xffffffff)
        /usr/local/go/src/runtime/mheap.go:1255 +0xa3
runtime.(*mheap).allocSpanLocked(0x248af20, 0x208eb, 0x24a42a8, 0x0)
        /usr/local/go/src/runtime/mheap.go:1170 +0x266
runtime.(*mheap).alloc_m(0x248af20, 0x208eb, 0x7f16c0c10101, 0x0)
        /usr/local/go/src/runtime/mheap.go:1022 +0xc2
runtime.(*mheap).alloc.func1()
        /usr/local/go/src/runtime/mheap.go:1093 +0x4c
runtime.(*mheap).alloc(0x248af20, 0x208eb, 0x7f168d010101, 0x7f16c0c19558)
        /usr/local/go/src/runtime/mheap.go:1092 +0x8a
runtime.largeAlloc(0x411d516f, 0x990101, 0x7f16c0c19558)
        /usr/local/go/src/runtime/malloc.go:1138 +0x97
runtime.mallocgc.func1()
        /usr/local/go/src/runtime/malloc.go:1033 +0x46
runtime.systemstack(0x0)
        /usr/local/go/src/runtime/asm_amd64.s:370 +0x66
runtime.mstart()
        /usr/local/go/src/runtime/proc.go:1146

goroutine 141 [running]:
runtime.systemstack_switch()
        /usr/local/go/src/runtime/asm_amd64.s:330 fp=0xc0001b1b28 sp=0xc0001b1b20 pc=0x9a1a10
runtime.mallocgc(0x411d516f, 0x1699b60, 0x1001, 0x1000)
        /usr/local/go/src/runtime/malloc.go:1032 +0x895 fp=0xc0001b1bc8 sp=0xc0001b1b28 pc=0x94ffa5
runtime.makeslice(0x1699b60, 0x411d516f, 0x411d516f, 0x1000)
        /usr/local/go/src/runtime/slice.go:49 +0x6c fp=0xc0001b1bf8 sp=0xc0001b1bc8 pc=0x98a3ec
bytes.makeSlice(0x411d516f, 0x0, 0x0, 0x0)
        /usr/local/go/src/bytes/buffer.go:229 +0x77 fp=0xc0001b1c60 sp=0xc0001b1bf8 pc=0xa42e37
bytes.(*Buffer).grow(0xc0003b9b90, 0x1, 0x0)
        /usr/local/go/src/bytes/buffer.go:142 +0x15b fp=0xc0001b1cb0 sp=0xc0001b1c60 pc=0xa4277b
bytes.(*Buffer).WriteByte(0xc0003b9b90, 0x1461d36, 0x0, 0x0)
        /usr/local/go/src/bytes/buffer.go:267 +0x8c fp=0xc0001b1cd8 sp=0xc0001b1cb0 pc=0xa4306c
bytes.(*Buffer).WriteRune(0xc0003b9b90, 0xc000000036, 0x1, 0x0, 0x0)
        /usr/local/go/src/bytes/buffer.go:279 +0x137 fp=0xc0001b1d18 sp=0xc0001b1cd8 pc=0xa431d7
github.com/dgraph-io/dgraph/chunker.slurpQuoted(0xc00007db60, 0xc0003b9b90, 0x1, 0x0)
        /tmp/go/src/github.com/dgraph-io/dgraph/chunker/chunk.go:328 +0x58 fp=0xc0001b1d58 sp=0xc0001b1d18 pc=0x1461de8
github.com/dgraph-io/dgraph/chunker.(*jsonChunker).consumeMap(0xc0001847a0, 0xc00007db60, 0xc0003b9b90, 0x0, 0x0)
        /tmp/go/src/github.com/dgraph-io/dgraph/chunker/chunk.go:276 +0xec fp=0xc0001b1da8 sp=0xc0001b1d58 pc=0x14619dc
github.com/dgraph-io/dgraph/chunker.(*jsonChunker).Chunk(0xc0001847a0, 0xc00007db60, 0xc000588500, 0xc0001800e0, 0xc0003b9b00)
        /tmp/go/src/github.com/dgraph-io/dgraph/chunker/chunk.go:194 +0xf7 fp=0xc0001b1e28 sp=0xc0001b1da8 pc=0x1461227
github.com/dgraph-io/dgraph/dgraph/cmd/live.(*loader).processLoadFile(0xc0001800e0, 0x1a849e0, 0xc00003a0c8, 0xc00007db60, 0x1a7b9e0, 0xc0001847a0, 0x1, 0x2f)
        /tmp/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/run.go:308 +0x10a fp=0xc0001b1e90 sp=0xc0001b1e28 pc=0x150d40a
github.com/dgraph-io/dgraph/dgraph/cmd/live.(*loader).processFile(0xc0001800e0, 0x1a849e0, 0xc00003a0c8, 0x7ffe8b457910, 0x17, 0x0, 0x0)
        /tmp/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/run.go:235 +0x1c5 fp=0xc0001b1f58 sp=0xc0001b1e90 pc=0x150d1c5
github.com/dgraph-io/dgraph/dgraph/cmd/live.run.func2(0xc00065c0c0, 0xc0001800e0, 0x1a849e0, 0xc00003a0c8, 0x7ffe8b457910, 0x17)
        /tmp/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/run.go:440 +0x56 fp=0xc0001b1fb0 sp=0xc0001b1f58 pc=0x150fe06
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0001b1fb8 sp=0xc0001b1fb0 pc=0x9a3ae1
created by github.com/dgraph-io/dgraph/dgraph/cmd/live.run
        /tmp/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/run.go:439 +0xa43

Thank you for your help

2

Hey @gumupaier, can you please list the steps to reproduce this error? (Dgraph version is 1.2.2 I believe)

3

hi @namanj this error occurred occasionally but not every time. I deployed dgraph in the form of docker Swarm according to the official document, and then directly carried out data import (a large amount, about 10 million), and the error occurred.

4

Thanks @gumupaier, I am working over it.

5

hi @Naman I have found the rule. When the amount of data in the json file is large, the error will occur every time.

1 Like
6

@gumupaier, how big is your data file and have you figured out some rough estimate of threshold at which error occurs?

7

hey @Naman The file size is as follows, if the amount of data is about 2 million per file

image
image1532×374 153 KB