Unable to read some data

Issues Badger
1

What version of Go are you using (go version)?

go version go1.14.4 linux/amd64

What operating system are you using?

Ubuntu/Linux

What version of Badger are you using?

v2.0.3 → v2.2007.2

Does this issue reproduce with the latest master?

Not sure how to reproduce (sorry!)

Steps to Reproduce the issue

Not clear how to repo, but I can provide details on the setup and stack trace to the panic. Maybe someone with more context on the code can understand what is causing the issue.

  • Run badgerDB on a Linux host for months
  • Small host CPU/memory, occasionally resource-starved. Small disk size and has run out of disk space a few times.
  • Processing a write-heavy workload with TTL on entries

Started noticing some data is not accessible at specific keys (panic or error when trying to access)

What Badger options were set?

WithCompression(options.None).
WithMaxBfCacheSize(10 * 1024).
WithMaxCacheSize(0).
WithValueLogFileSize(valueLogSize()).
WithMaxTableSize(maxTableSize()).
WithLoadBloomsOnOpen(false).
WithKeepL0InMemory(false).
WithNumLevelZeroTables(1).
WithNumLevelZeroTablesStall(2).
WithTruncate(true).
WithValueLogLoadingMode(options.FileIO).
WithTableLoadingMode(options.FileIO)

What did you do?

Try accessing data with a key

What did you expect to see?

The data

What did you see instead?

Runtime panic from badgerDB source code on v2.03

line":"73","message":"[PANIC RECOVER] runtime error: slice bounds out of range [:768016] with capacity 1761 goroutine 2131853 [running]:\recover.go:71
 +0xee\npanic(0xd76bc0, 0xc009cbf420)\n\t/usr/local/go/src/runtime/panic.go:679
 +0x1b2\ngithub.com/dgraph-io/badger/v2.(*valueLog).Read(0xc0000acf60, 0x6ec00001755, 0xc01275973f, 0xc0200edc60, 0xc, 0x5522b3, 0x410c06, 0x0, 0x0, 0x1)\n\t/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.3/value.go:1523
 +0x60c\ngithub.com/dgraph-io/badger/v2.(*Item).yieldItemValue(0xc01b7bd1e0, 0xc, 0xc00d849300, 0x0, 0x10, 0xc, 0x5522b3)\n\t/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.3/iterator.go:170
 +0x11c\ngithub.com/dgraph-io/badger/v2.(*Item).ValueCopy(0xc01b7bd1e0, 0x16bcab0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\n\t/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.3/iterator.go:127

<application stack trace here>

Upgrading to v2.2007.2 gives another error, still unable to access data:

badger 2020/10/21 01:12:07 ERROR: Invalid read: vp: {Fid:5973 Len:1772 Offset:309696319}
badger 2020/10/21 01:12:07 ERROR: Unable to read: Key: [<key byte values here>], Version : 5579443,
				meta: 66, userMeta: 0
1 Like
4

@Naman can you look into this?

7

Hey @KlotzAndrew, thanks for reporting this.

  • Can you please provide the memory specs of the machine as well?
  • Also, can you check if all these are the options that you set? I see these versions don’t have options like MaxBfCacheSize.
8

Hey! Thanks for taking a look

  • Running on a Digital Ocean 1vCPU 2GB droplet

These settings changed updating versions otherwise, those settings are the full list

2.0.3 used:

WithMaxBfCacheSize(10 * 1024).
WithMaxCacheSize(0).

v2.2007.2 changed those 2 lines to:

WithBlockCacheSize(0).
WithIndexCacheSize(10 * 1024).