What version of Go are you using (go version)?
$ go version go version go1.15.7 linux/amd64
What operating system are you using?
Fedora 34
What version of Badger are you using?
github.com/dgraph-io/badger/v3 v3.2103.2
Does this issue reproduce with the latest master?
N/A
Steps to Reproduce the issue
Reproducing would be hard I use github.com/nathanhack/redisbadger, and it’s after a long time of adding 100M> keys.
What Badger options were set?
badger.DefaultOptions()
and
opts := badger.DefaultIteratorOptions
opts.PrefetchValues = false
What did you do?
Added 100M> keys and during a call to
.Next() on an iterator there was a crash.
What did you expect to see?
The next key
What did you see instead?
goroutine 71060 [running]:
github.com/dgraph-io/badger/v3/table.(*blockIterator).setIdx.func1(0xc34bfd9270, 0x2e6, 0x100058f)
/home/wnhack/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.2/table/iterator.go:97 +0x35f
panic(0xb3af40, 0xc32554a7a0)
/home/wnhack/programs/go/go1.15.7/src/runtime/panic.go:969 +0x1b9
github.com/dgraph-io/badger/v3/table.(*blockIterator).setIdx(0xc34bfd9270, 0x1)
/home/wnhack/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.2/table/iterator.go:101 +0x599
github.com/dgraph-io/badger/v3/table.(*blockIterator).next(…)
/home/wnhack/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.2/table/iterator.go:168
github.com/dgraph-io/badger/v3/table.(*Iterator).next(0xc34bfd9260)
/home/wnhack/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.2/table/iterator.go:344 +0x1b9
github.com/dgraph-io/badger/v3/table.(*Iterator).Next(0xc34bfd9260)
/home/wnhack/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.2/table/iterator.go:406 +0x3e
github.com/dgraph-io/badger/v3/table.(*ConcatIterator).Next(0xc64479a460)
/home/wnhack/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.2/table/iterator.go:529 +0x33
github.com/dgraph-io/badger/v3/table.(*node).next(0xc4c46efc70)
/home/wnhack/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.2/table/merge_iterator.go:81 +0x3d
github.com/dgraph-io/badger/v3/table.(*MergeIterator).Next(0xc4c46efc30)
/home/wnhack/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.2/table/merge_iterator.go:157 +0x38
github.com/dgraph-io/badger/v3/table.(*node).next(0xc4c46efd20)
/home/wnhack/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.2/table/merge_iterator.go:79 +0x72
github.com/dgraph-io/badger/v3/table.(*MergeIterator).Next(0xc4c46efce0)
/home/wnhack/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.2/table/merge_iterator.go:157 +0x38
github.com/dgraph-io/badger/v3/table.(*node).next(0xc4c46efdd0)
/home/wnhack/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.2/table/merge_iterator.go:79 +0x72
github.com/dgraph-io/badger/v3/table.(*MergeIterator).Next(0xc4c46efd90)
/home/wnhack/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.2/table/merge_iterator.go:157 +0x38
github.com/dgraph-io/badger/v3/table.(*node).next(0xc4c46efe80)
/home/wnhack/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.2/table/merge_iterator.go:79 +0x72
github.com/dgraph-io/badger/v3/table.(*MergeIterator).Next(0xc4c46efe40)
/home/wnhack/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.2/table/merge_iterator.go:157 +0x38
github.com/dgraph-io/badger/v3.(*Iterator).parseItem(0xc4c46efef0, 0x201)
/home/wnhack/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.2/iterator.go:690 +0x464
github.com/dgraph-io/badger/v3.(*Iterator).Next(0xc4c46efef0)
/home/wnhack/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.2/iterator.go:594 +0x133
github.com/nathanhack/redisbadger/cmd.glob..func1.2(0xc6f300, 0xc032743570, 0xc439783f40, 0x43, 0x50, 0xc6447a4360, 0x6, 0x6)
/home/wnhack/projects/redisbadger/cmd/root.go:236 +0x18b7
github.com/nathanhack/redcon/v2.handle.func2(0xc032743570, 0xc0004d8080, 0x1, 0xa)
/home/wnhack/go/pkg/mod/github.com/nathanhack/redcon/v2@v2.0.2/redcon.go:451 +0x74
github.com/nathanhack/redcon/v2.handle(0xc0004d8080, 0xc032743570)
/home/wnhack/go/pkg/mod/github.com/nathanhack/redcon/v2@v2.0.2/redcon.go:464 +0x78
created by github.com/nathanhack/redcon/v2.serve
/home/wnhack/go/pkg/mod/github.com/nathanhack/redcon/v2@v2.0.2/redcon.go:400 +0x5e5
Prior to that the following warning and errors occurred:
badger 2021/11/22 22:13:25 WARNING: Block cache might be too small. Metrics: hit: 242565850 miss: 19804696042 keys-added: 4501982243 keys-updated: 477362 keys-
evicted: 4501910174 cost-added: 16518719755552 cost-evicted: 16518451320120 sets-dropped: 0 sets-rejected: 15248187658 gets-dropped: 3903021056 gets-kept: 1613
9483520 gets-total: 20047261892 hit-ratio: 0.01
badger 2021/11/22 22:13:25 WARNING: Cache life expectancy (in seconds):
– Histogram:
Min value: 0
Max value: 13037
Count: 4501909318
50p: 4.00
75p: 8.00
90p: 16.00
[0, 2) 1609768636 35.76% 35.76%
[2, 4) 852565433 18.94% 54.70%
[4, 8) 1317067793 29.26% 83.95%
[8, 16) 681898330 15.15% 99.10%
[16, 32) 37568076 0.83% 99.93%
[32, 64) 2214420 0.05% 99.98%
[64, 128) 616101 0.01% 100.00%
[128, 256) 132841 0.00% 100.00%
[256, 512) 51511 0.00% 100.00%
[512, 1024) 16029 0.00% 100.00%
[1024, 2048) 8685 0.00% 100.00%
[2048, 4096) 929 0.00% 100.00%
[4096, 8192) 415 0.00% 100.00%
[8192, 16384) 119 0.00% 100.00%
panic: runtime error: slice bounds out of range [:16778639] with capacity 3504 [recovered]
panic: ==== Recovered====
Table ID: 302000
Block ID: 511
Entry Idx: 1
Data len: 3470
StartOffset: 742
EndOffset: 16778639
EntryOffsets len: 5
EntryOffsets: [0 742 16778639 2105 2787]
Question
Is there anyway to repair the DB so I can recover the data?
The DB on startup now has the following error:
badger 2021/11/23 10:11:30 ERROR: Received err: Opening table: “badger/318542.sst” error: failed to initialize table error: failed to read index. error: failed
to verify checksum for table: badger/318542.sst error: actual: 2698524383, expected: 3872913041 error: checksum mismatch
github.com/dgraph-io/badger/v3/y.init
/home/wnhack/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.2/y/checksum.go:29
runtime.doInit
/home/wnhack/programs/go/go1.15.7/src/runtime/proc.go:5652
runtime.doInit
/home/wnhack/programs/go/go1.15.7/src/runtime/proc.go:5647
runtime.doInit
/home/wnhack/programs/go/go1.15.7/src/runtime/proc.go:5647
runtime.doInit
/home/wnhack/programs/go/go1.15.7/src/runtime/proc.go:5647
runtime.main
/home/wnhack/programs/go/go1.15.7/src/runtime/proc.go:191
runtime.goexit
/home/wnhack/programs/go/go1.15.7/src/runtime/asm_amd64.s:1374. Cleaning up…
FATA[0037] Opening table: “badger/318542.sst” error: failed to initialize table error: failed to read index. error: failed to verify checksum for table: badger
/318542.sst error: actual: 2698524383, expected: 3872913041 error: checksum mismatch
github.com/dgraph-io/badger/v3/y.init
/home/wnhack/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.2/y/checksum.go:29
runtime.doInit
/home/wnhack/programs/go/go1.15.7/src/runtime/proc.go:5652
runtime.doInit
/home/wnhack/programs/go/go1.15.7/src/runtime/proc.go:5647
runtime.doInit
/home/wnhack/programs/go/go1.15.7/src/runtime/proc.go:5647
runtime.doInit
/home/wnhack/programs/go/go1.15.7/src/runtime/proc.go:5647
runtime.main
/home/wnhack/programs/go/go1.15.7/src/runtime/proc.go:191
runtime.goexit
/home/wnhack/programs/go/go1.15.7/src/runtime/asm_amd64.s:1374
exit status 1
Is there a way to “repair” the DB and/or recover the data?
I just tried looking for help on the https://dgraph.io/docs/badger web site but I’m getting a 502 (I’ll check again later).
Any help recovering the data would be greatly appreciated.