DGraph crashes, SIGILL: illegal instruction when doing mutations


Report a Dgraph Bug

What version of Dgraph are you using?

Dgraph Version
$ dgraph version
 
server_1  | Dgraph version   : v20.03.6
server_1  | Dgraph SHA-256   : 1a94e66946a64c81064535928cab87f80805d43872be71887bf471a31897bc92
server_1  | Commit SHA-1     : b842d2e81
server_1  | Commit timestamp : 2020-10-22 10:17:46 -0700
server_1  | Branch           : HEAD
server_1  | Go version       : go1.14.4

Have you tried reproducing the issue with the latest release?

Yes

What is the hardware spec (RAM, OS)?

Steps to reproduce the issue (command/config used to run Dgraph).

server_1  | I0415 06:38:50.406323      18 groups.go:495] Serving tablet for: inventory.interface.history
server_1  | I0415 06:38:50.408314      18 groups.go:495] Serving tablet for: inventory.driver.history
server_1  | I0415 06:38:50.414439      18 groups.go:495] Serving tablet for: inventory.protocol.history
server_1  | SIGILL: illegal instruction
server_1  | PC=0x1251e05 m=14 sigcode=2
server_1  |
server_1  | goroutine 1516 [running]:
server_1  | github.com/dgryski/go-groupvarint.Decode4(0xc04d419548, 0x4, 0x4, 0xc05120d400, 0x11, 0x20)
server_1  |     /go/pkg/mod/github.com/dgryski/go-groupvarint@v0.0.0-20190318181831-5ce5df8ca4e1/decode_amd64.s:16 +0x25 fp=0xc04d4194c0 sp=0xc04d4194b8 pc=0x1251e05
server_1  | github.com/dgraph-io/dgraph/codec.(*Decoder).UnpackBlock(0xc05131c270, 0xc00014c000, 0x7f4c8111e108, 0x0)
server_1  |     /ext-go/1/src/github.com/dgraph-io/dgraph/codec/codec.go:157 +0x20d fp=0xc04d419590 sp=0xc04d4194c0 pc=0x12525fd
server_1  | github.com/dgraph-io/dgraph/codec.(*Decoder).Seek(0xc05131c270, 0x0, 0x1, 0xc04d419700, 0xc051305d50, 0xc051305d50)
server_1  |     /ext-go/1/src/github.com/dgraph-io/dgraph/codec/codec.go:195 +0x2c2 fp=0xc04d419610 sp=0xc04d419590 pc=0x1252bb2
server_1  | github.com/dgraph-io/dgraph/posting.(*pIterator).init(0xc04d4197d0, 0xc051268ae0, 0x0, 0x0, 0x2, 0x2)
server_1  |     /ext-go/1/src/github.com/dgraph-io/dgraph/posting/list.go:144 +0x1a1 fp=0xc04d4196e0 sp=0xc04d419610 pc=0x1347d01
server_1  | github.com/dgraph-io/dgraph/posting.(*List).iterate(0xc051268ae0, 0xffffffffffffffff, 0x0, 0xc04d419930, 0x203014, 0x203014)
server_1  |     /ext-go/1/src/github.com/dgraph-io/dgraph/posting/list.go:666 +0x111 fp=0xc04d419848 sp=0xc04d4196e0 pc=0x134aba1
server_1  | github.com/dgraph-io/dgraph/posting.(*List).encode(0xc051268ae0, 0xc05123b8a0, 0xffffffffffffffff, 0xc0003d4901, 0x0, 0x0)
server_1  |     /ext-go/1/src/github.com/dgraph-io/dgraph/posting/list.go:945 +0x1e6 fp=0xc04d419988 sp=0xc04d419848 pc=0x134c536
server_1  | github.com/dgraph-io/dgraph/posting.(*List).rollup(0xc051268ae0, 0xffffffffffffffff, 0x1, 0x0, 0x2b2d4e8, 0x0)
server_1  |     /ext-go/1/src/github.com/dgraph-io/dgraph/posting/list.go:997 +0x20a fp=0xc04d4199f8 sp=0xc04d419988 pc=0x134ca2a
server_1  | github.com/dgraph-io/dgraph/posting.(*List).Rollup(0xc051268ae0, 0x0, 0x0, 0x0, 0x0, 0x0)
server_1  |     /ext-go/1/src/github.com/dgraph-io/dgraph/posting/list.go:816 +0xaf fp=0xc04d419b40 sp=0xc04d4199f8 pc=0x134b62f
server_1  | github.com/dgraph-io/dgraph/posting.(*incrRollupi).rollUpKey(0x26d5840, 0xc04da2ee00, 0xc050e1d590, 0x24, 0x24, 0x0, 0x0)
server_1  |     /ext-go/1/src/github.com/dgraph-io/dgraph/posting/mvcc.go:72 +0x90 fp=0xc04d419c00 sp=0xc04d419b40 pc=0x1352750
server_1  | github.com/dgraph-io/dgraph/posting.(*incrRollupi).Process.func1()
server_1  |     /ext-go/1/src/github.com/dgraph-io/dgraph/posting/mvcc.go:127 +0x241 fp=0xc04d419cf0 sp=0xc04d419c00 pc=0x1359ff1
server_1  | github.com/dgraph-io/dgraph/posting.(*incrRollupi).Process(0x26d5840, 0xc04da25230)
server_1  |     /ext-go/1/src/github.com/dgraph-io/dgraph/posting/mvcc.go:152 +0x42e fp=0xc04d419fd0 sp=0xc04d419cf0 pc=0x1352f7e
server_1  | runtime.goexit()
server_1  |     /usr/local/go/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc04d419fd8 sp=0xc04d419fd0 pc=0x9ba811
server_1  | created by github.com/dgraph-io/dgraph/worker.(*node).startTask
server_1  |     /ext-go/1/src/github.com/dgraph-io/dgraph/worker/draft.go:132 +0x60d
server_1  |
server_1  | goroutine 1 [semacquire, 965 minutes]:
server_1  | sync.runtime_Semacquire(0xc000210238)
server_1  |     /usr/local/go/src/runtime/sema.go:56 +0x42
server_1  | sync.(*WaitGroup).Wait(0xc000210230)
server_1  |     /usr/local/go/src/sync/waitgroup.go:130 +0x64
server_1  | github.com/dgraph-io/dgraph/dgraph/cmd/alpha.setupServer(0xc000650060)
server_1  |     /ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:535 +0xc48
server_1  | github.com/dgraph-io/dgraph/dgraph/cmd/alpha.run()
server_1  |     /ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:750 +0x1261
server_1  | github.com/dgraph-io/dgraph/dgraph/cmd/alpha.init.2.func1(0xc0000e5b80, 0xc0003082d0, 0x0, 0x5)
server_1  |     /ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:91 +0x62
server_1  | github.com/spf13/cobra.(*Command).execute(0xc0000e5b80, 0xc000308280, 0x5, 0x5, 0xc0000e5b80, 0xc000308280)
server_1  |     /go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830 +0x29d
server_1  | github.com/spf13/cobra.(*Command).ExecuteC(0x26ec5a0, 0x0, 0x0, 0x0)
server_1  |     /go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914 +0x2fb
server_1  | github.com/spf13/cobra.(*Command).Execute(...)
server_1  |     /go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
server_1  | github.com/dgraph-io/dgraph/dgraph/cmd.Execute()
server_1  |     /ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:70 +0x80
server_1  | main.main()
server_1  |     /ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/main.go:78 +0xb0
server_1  |

..... UNABLE TO PASTE WHOLE STACKTRACE .....

server_1  | goroutine 1513 [select, 965 minutes]:
server_1  | github.com/dgraph-io/badger/v2.(*DB).Subscribe(0xc0003d4800, 0x1c30780, 0xc04d93a7b0, 0xc04d874bc0, 0xc04d6d68c0, 0x1, 0x1, 0x95eec8, 0x10)
server_1  |     /go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.3-0.20200927102853-5e3d4b9926a7/db.go:1734 +0x259
server_1  | github.com/dgraph-io/dgraph/worker.(*grpcWorker).Subscribe(0xc0001a0080, 0xc04d962080, 0x1c39e60, 0xc04d6cad50, 0xc0001a0080, 0xc0000de360)
server_1  |     /ext-go/1/src/github.com/dgraph-io/dgraph/worker/worker.go:80 +0xc8
server_1  | github.com/dgraph-io/dgraph/protos/pb._Worker_Subscribe_Handler(0x198f740, 0xc0001a0080, 0x1c37280, 0xc04d5c89c0, 0xc0000de3f8, 0xc000219100)
server_1  |     /ext-go/1/src/github.com/dgraph-io/dgraph/protos/pb/pb.pb.go:5988 +0x109
server_1  | google.golang.org/grpc.(*Server).processStreamingRPC(0xc0003a66e0, 0x1c3d6a0, 0xc04d960000, 0xc000219100, 0xc0006ffc50, 0x26e85e0, 0x0, 0x0, 0x0)
server_1  |     /go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:1199 +0xd21
server_1  | google.golang.org/grpc.(*Server).handleStream(0xc0003a66e0, 0x1c3d6a0, 0xc04d960000, 0xc000219100, 0x0)
server_1  |     /go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:1279 +0xcd6
server_1  | google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc04d92cf10, 0xc0003a66e0, 0x1c3d6a0, 0xc04d960000, 0xc000219100)
server_1  |     /go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:710 +0xa1
server_1  | created by google.golang.org/grpc.(*Server).serveStreams.func1
server_1  |     /go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:708 +0xa1
server_1  |
server_1  | rax    0xc04d419548
server_1  | rbx    0x2732a00
server_1  | rcx    0x2732a00
server_1  | rdx    0xf
server_1  | rdi    0xc05120d40a
server_1  | rsi    0xc05126a978
server_1  | rbp    0xc04d419580
server_1  | rsp    0xc04d4194b8
server_1  | r8     0xc05120d400
server_1  | r9     0x11
server_1  | r10    0x20
server_1  | r11    0x0
server_1  | r12    0xffffffffffffffff
server_1  | r13    0xa1
server_1  | r14    0xa0
server_1  | r15    0x100
server_1  | rip    0x1251e05
server_1  | rflags 0x10206
server_1  | cs     0x33
server_1  | fs     0x0
server_1  | gs     0x0

Expected behaviour and actual result.


Expected it not to crash…

Experience Report for Feature Request

Note: Feature requests are judged based on user experience and modeled on Go Experience Reports. These reports should focus on the problems: they should not focus on and need not propose solutions.

What you wanted to do

What you actually did

Why that wasn’t great, with examples

Any external references to support your case

Could it be related to this?

This is an old server…

Are you using a x86_64 machine? Illegal instructions are rare nowadays with most processors being x86_64.

How old is old? What is the processor?

Try to recompile from source on the target server. It worked for me, as the go compiler detects the CPU and omits unavailable processor instructions.