Panics when opening a corrupted Badger DB

Issues Badger
1

What version of Go are you using (go version)?

$ go version
go version go1.16.2 windows/amd64

What operating system are you using?

Windows 10 1903 (OS build 18362.1082)

What version of Badger are you using?

github.com/dgraph-io/badger/v3 v3.2011.1

Does this issue reproduce with the latest master?

Yes

Steps to Reproduce the issue

Simply open the DB with

badger.Open(badger.DefaultOptions(dbPath).WithVerifyValueChecksum(true))

The corrupted DB can be found in my gdrive (2.6MB):

What Badger options were set?

badger.DefaultOptions(dbPath)
WithVerifyValueChecksum(true)

What did you do?

Calling badger.Open()

What did you expect to see?

Return an error showing the DB is corrupted

What did you see instead?

Panic in another goroutine, which I cannot recover.

panic: runtime error: index out of range [3] with length 0 [recovered]
	panic: 
== Recovering from initIndex crash ==
File Info: [ID: 38, Size: 408, Zeros: 408]
isEnrypted: false checksumLen: 0 checksum:  indexLen: 0 index: [] 
== Recovered ==


goroutine 80 [running]:
github.com/dgraph-io/badger/v3/table.(*Table).initBiggestAndSmallest.func1.1(0xc0001c1aa0)
	C:/gocode/pkg/mod/github.com/dgraph-io/badger/v3@v3.2011.1/table/table.go:351 +0x165
github.com/dgraph-io/badger/v3/table.(*Table).initBiggestAndSmallest.func1(0xc0001d6180)
	C:/gocode/pkg/mod/github.com/dgraph-io/badger/v3@v3.2011.1/table/table.go:397 +0x6e
panic(0x1d77ca0, 0xc0003bcd68)
	C:/go/src/runtime/panic.go:971 +0x49a
github.com/google/flatbuffers/go.GetInt32(...)
	C:/gocode/pkg/mod/github.com/google/flatbuffers@v1.12.0/go/encode.go:85
github.com/google/flatbuffers/go.GetUOffsetT(...)
	C:/gocode/pkg/mod/github.com/google/flatbuffers@v1.12.0/go/encode.go:121
github.com/dgraph-io/badger/v3/fb.GetRootAsTableIndex(...)
	C:/gocode/pkg/mod/github.com/dgraph-io/badger/v3@v3.2011.1/fb/TableIndex.go:14
github.com/dgraph-io/badger/v3/table.(*Table).readTableIndex(0xc0001d6180, 0x0, 0x8, 0xc0001c1a40)
	C:/gocode/pkg/mod/github.com/dgraph-io/badger/v3@v3.2011.1/table/table.go:704 +0x491
github.com/dgraph-io/badger/v3/table.(*Table).initIndex(0xc0001d6180, 0x0, 0x0, 0xc0002a3260)
	C:/gocode/pkg/mod/github.com/dgraph-io/badger/v3@v3.2011.1/table/table.go:462 +0x545
github.com/dgraph-io/badger/v3/table.(*Table).initBiggestAndSmallest(0xc0001d6180, 0x0, 0x0)
	C:/gocode/pkg/mod/github.com/dgraph-io/badger/v3@v3.2011.1/table/table.go:401 +0xab
github.com/dgraph-io/badger/v3/table.OpenTable(0xc000434280, 0x0, 0x200000, 0x0, 0x0, 0x3f847ae147ae147b, 0x1000, 0x0, 0x1, 0xc00053a080, ...)
	C:/gocode/pkg/mod/github.com/dgraph-io/badger/v3@v3.2011.1/table/table.go:308 +0x36e
github.com/dgraph-io/badger/v3.newLevelsController.func1(0xc00043e180, 0xc0002b57b0, 0xc000589000, 0xc0002b5798, 0xc000178d10, 0x7, 0x7, 0xc00050a780, 0x21, 0x6, ...)
	C:/gocode/pkg/mod/github.com/dgraph-io/badger/v3@v3.2011.1/levels.go:150 +0x2d8
created by github.com/dgraph-io/badger/v3.newLevelsController
	C:/gocode/pkg/mod/github.com/dgraph-io/badger/v3@v3.2011.1/levels.go:129 +0xc2b
2

hey @lukeng , it looks like all your sst files are empty.

du -sc db/*             
4	db/000002.sst
4	db/000004.sst
4	db/000006.sst
4	db/000008.sst
4	db/000010.sst
4	db/000012.sst
4	db/000014.sst

Did you modify these files? Badger shouldn’t be creating empty files.

3

Hi, I did not touch the files in the directory.

May it be created when I add and remove keys? I am using the DB as a cache and will periodically remove keys.

The crash is quite rare and it happens after I kill the app by force, not sure if it is related.

4

@lukeng The SST files are created/deleted by compaction but having 0 size SST is uncommon.
I think somehow your crash caused this.

@lukeng is there any way I can reproduce this problem? If you have a script or a program that I can use to reproduce this it would be very helpful.

5

Hi @ibrahim , I recently encountered a very similar problem, our program killed by power lost, it paniced when it tried to reopen the broken db, here is the stack trace

panic: runtime error: index out of range [3] with length 0 [recovered]
        panic:
== Recovering from initIndex crash ==
File Info: [ID: 28, Size: 2475, Zeros: 2475]
isEnrypted: false checksumLen: 0 checksum:  indexLen: 0 index: []
== Recovered ==


goroutine 469 [running]:
github.com/dgraph-io/badger/v3/table.(*Table).initBiggestAndSmallest.func1.1(0xc000a4e5d0)
        /home/yongsheng/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2011.1/table/table.go:351 +0x125
github.com/dgraph-io/badger/v3/table.(*Table).initBiggestAndSmallest.func1(0xc000a74180)
        /home/yongsheng/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2011.1/table/table.go:397 +0x616
panic(0x1261fc0, 0xc000a660a8)
        /usr/lib/go/src/runtime/panic.go:965 +0x1b9
github.com/google/flatbuffers/go.GetInt32(...)
        /home/yongsheng/go/pkg/mod/github.com/google/flatbuffers@v1.12.0/go/encode.go:85
github.com/google/flatbuffers/go.GetUOffsetT(...)
        /home/yongsheng/go/pkg/mod/github.com/google/flatbuffers@v1.12.0/go/encode.go:121
github.com/dgraph-io/badger/v3/fb.GetRootAsTableIndex(...)
        /home/yongsheng/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2011.1/fb/TableIndex.go:14
github.com/dgraph-io/badger/v3/table.(*Table).readTableIndex(0xc000a74180, 0x0, 0x8, 0xc000a4e5a0)
        /home/yongsheng/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2011.1/table/table.go:704 +0x230
github.com/dgraph-io/badger/v3/table.(*Table).initIndex(0xc000a74180, 0x203000, 0x8, 0x203000)
        /home/yongsheng/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2011.1/table/table.go:462 +0x28c
github.com/dgraph-io/badger/v3/table.(*Table).initBiggestAndSmallest(0xc000a74180, 0x0, 0x0)
        /home/yongsheng/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2011.1/table/table.go:401 +0x85
github.com/dgraph-io/badger/v3/table.OpenTable(0xc000a483e0, 0x0, 0x200000, 0x0, 0x0, 0x3f847ae147ae147b, 0x1000, 0x0, 0x1, 0xc007efac80, ...)
        /home/yongsheng/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2011.1/table/table.go:308 +0x227
github.com/dgraph-io/badger/v3.newLevelsController.func1(0xc00294de00, 0xc003043d10, 0xc002970000, 0xc003043d08, 0xc000465340, 0x7, 0x7, 0xc000a3c0f0, 0x42, 0xc007ebdc00, ...)
        /home/yongsheng/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2011.1/levels.go:150 +0x218
created by github.com/dgraph-io/badger/v3.newLevelsController
        /home/yongsheng/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2011.1/levels.go:129 +0x645

I think write a program that keep writing data and at the same time, we keep sending SIGKILL to the program may reproduce this (haven’t prove it), I will add some code, when I managed to reproduce it by a simple example.

6

@ibrahim I failed to reproduce this problem without force power off, it seems taskkill /F(force kill on windows) can not reproduce this. anyway, I use qemu, create a windows virtual machine, run this code:

then shutdown → force off the virtual machine, and, got a broken db, here is the sample: https://1drv.ms/u/s!AiwOVdacuzTqgokfRvsFNTCmP63Qzg?e=yOVFCY

hope it helps

8

I’m seeing this with version v3.2103.2.
Can’t catch it so the application crashes.

This is a 2K zip of a DB that will cause the issue.
https://drive.google.com/file/d/10vKLwsFypH_AR0g957lj3YwA05V8wxsz/view?usp=sharing

9

I encountered the same problem when opening a corrupted db. Badger DB should not just panic when the db is corrupted. It cannot be captured and breaks the whole application. @ibrahim please fix this.

1 Like
10

The issue of corrupted BadgerDb running on a Windows box is very common.
Have seen it once in Linux as well. On Windows, it’s hard to avoid.
Now I’m exporting a backup every 5min and have an auto recovery.
That only works if the error is detected and panic is avoided.
Sometimes the DB comes up with keys missing or just panics out.

11

Maybe the database open → fetch/update/delete → close in all db operate func works…