Crash on decoding value using CLI

Issues Badger
,
1

Moved from GitHub badger/1399

Posted by ammmir:

What version of Go are you using (go version)?

$ go version
go version go1.14.3 darwin/amd64

What operating system are you using?

MacOS X

What version of Badger are you using?

2.0.3

Does this issue reproduce with the latest master?

Not tested

What Badger options were set?

default

What did you do?

Database was migrated from manifest v4 to v7 (badger 1.6.0 or thereabouts to 2.0.3) via badger backup/restore.

badger info --dir db --show-keys --with-prefix 019a4730021f457ba087a255a58f49 --read-only=false

What did you see instead?

╰─➤  /tmp/badger2 info --dir db --show-keys --with-prefix 019a4730021f457ba087a255a58f49 --read-only=false
Listening for /debug HTTP requests at port: 8080

[2020-07-08T15:58:40+09:00] MANIFEST       50 B MA
[        6 minutes earlier] 000008.sst    15 MB L1
[       40 minutes earlier] 000000.vlog   96 MB VL
[       40 minutes earlier] 000002.vlog  1.3 GB VL
[                      now] 000003.vlog   32 MB VL

[EXTRA]
[2020-07-08T15:18:08+09:00] KEYREGISTRY    28 B

[Summary]
Level 0 size:          0 B
Level 1 size:        15 MB
Total index size:    15 MB
Value log size:     1.4 GB

Abnormalities:
1 extra file.
0 missing files.
0 empty files.
0 truncated manifests.
badger 2020/07/08 15:58:42 INFO: All 1 tables opened in 10ms
badger 2020/07/08 15:58:42 INFO: Replaying file id: 3 at offset: 32266481
badger 2020/07/08 15:58:42 INFO: Replay took: 5.337µs
badger 2020/07/08 15:58:42 DEBUG: Value log discard stats empty
Only choosing keys with prefix:
00000000  01 9a 47 30 02 1f 45 7b  a0 87 a2 55 a5 8f 49     |..G0..E{...U..I|
badger 2020/07/08 15:58:42 INFO: Got compaction priority: {level:0 score:1.73 dropPrefix:[]}
panic: runtime error: slice bounds out of range [:12] with capacity 0

goroutine 1 [running]:
github.com/dgraph-io/badger/v2.(*valuePointer).Decode(...)
	/Users/amir/src/badger/structs.go:61
github.com/dgraph-io/badger/v2.(*Item).EstimatedSize(...)
	/Users/amir/src/badger/iterator.go:247
github.com/dgraph-io/badger/v2/badger/cmd.printKey(0xc00019c6e0, 0xc0001b0000, 0x10, 0x10)
	/Users/amir/src/badger/badger/cmd/info.go:211 +0x61c
github.com/dgraph-io/badger/v2/badger/cmd.showKeys(0xc000330000, 0xc0000c8100, 0xf, 0x20, 0x0, 0x0)
	/Users/amir/src/badger/badger/cmd/info.go:151 +0x257
github.com/dgraph-io/badger/v2/badger/cmd.handleInfo(0x4aa0840, 0xc0001888a0, 0x0, 0x6, 0x0, 0x0)
	/Users/amir/src/badger/badger/cmd/info.go:120 +0x4ce
github.com/spf13/cobra.(*Command).execute(0x4aa0840, 0xc000188840, 0x6, 0x6, 0x4aa0840, 0xc000188840)
	/Users/amir/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:826 +0x453
github.com/spf13/cobra.(*Command).ExecuteC(0x4aa0fc0, 0x403aee0, 0x4a5c728, 0x0)
	/Users/amir/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
	/Users/amir/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/dgraph-io/badger/v2/badger/cmd.Execute()
	/Users/amir/src/badger/badger/cmd/root.go:40 +0x31
main.main()
	/Users/amir/src/badger/badger/main.go:41 +0x54

If --with-prefix is omitted it doesn’t crash.

2

jarifibrahim commented :

There was a bug in backup restore code which could’ve caused this Ensure `bitValuePointer` flag is cleared for LSM entry values written… · dgraph-io/badger@9459a24 · GitHub .

@ammmir would it be possible for you to do a backup restore from your v1.6x badger to the master version?

3

ammmir commented :

Thanks for the quick reply @jarifibrahim – I’ve pulled master and tried to restore the backup, but with the same result:


Listening for /debug HTTP requests at port: 8080

[2020-07-09T12:23:04+09:00] MANIFEST       50 B MA
[         20 hours earlier] 000008.sst    15 MB L1
[         21 hours earlier] 000000.vlog   96 MB VL
[         21 hours earlier] 000002.vlog  1.3 GB VL
[                      now] 000003.vlog   32 MB VL

[EXTRA]
[2020-07-08T15:18:08+09:00] KEYREGISTRY    28 B

[Summary]
Level 0 size:          0 B
Level 1 size:        15 MB
Total index size:    15 MB
Value log size:     1.4 GB

Abnormalities:
1 extra file.
0 missing files.
0 empty files.
0 truncated manifests.
badger 2020/07/09 12:24:03 INFO: All 1 tables opened in 9ms
badger 2020/07/09 12:24:03 INFO: Replaying file id: 3 at offset: 32266481
badger 2020/07/09 12:24:03 INFO: Replay took: 7.614µs
Only choosing keys with prefix:
00000000  01 9a 47 30 02 1f 45 7b  a0 87 a2 55 a5 8f 49     |..G0..E{...U..I|
badger 2020/07/09 12:24:03 INFO: Got compaction priority: {level:0 score:1.73 dropPrefixes:[]}
panic: runtime error: slice bounds out of range [:12] with capacity 0

goroutine 1 [running]:
github.com/dgraph-io/badger/v2.(*valuePointer).Decode(...)
	/Users/amir/src/badger/structs.go:61
github.com/dgraph-io/badger/v2.(*Item).EstimatedSize(...)
	/Users/amir/src/badger/iterator.go:251
github.com/dgraph-io/badger/v2/badger/cmd.printKey(0xc005720210, 0xc005702200, 0x10, 0x10)
	/Users/amir/src/badger/badger/cmd/info.go:211 +0x61c
github.com/dgraph-io/badger/v2/badger/cmd.showKeys(0xc0001ca400, 0xc000097160, 0xf, 0x20, 0x0, 0x0)
	/Users/amir/src/badger/badger/cmd/info.go:151 +0x257
github.com/dgraph-io/badger/v2/badger/cmd.handleInfo(0x4aa9860, 0xc000180960, 0x0, 0x6, 0x0, 0x0)
	/Users/amir/src/badger/badger/cmd/info.go:120 +0x44d
github.com/spf13/cobra.(*Command).execute(0x4aa9860, 0xc000180900, 0x6, 0x6, 0x4aa9860, 0xc000180900)
	/Users/amir/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:826 +0x453
github.com/spf13/cobra.(*Command).ExecuteC(0x4aa9fe0, 0x403b070, 0x4a65728, 0x0)
	/Users/amir/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
	/Users/amir/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/dgraph-io/badger/v2/badger/cmd.Execute()
	/Users/amir/src/badger/badger/cmd/root.go:40 +0x31
main.main()
	/Users/amir/src/badger/badger/main.go:41 +0x54

Should I regenerate the backup?

4

jarifibrahim commented :

Can you make a copy of your data directory and share it with me? I’d like to reproduce the bug on my end.

@ammmir Yes, try doing a new backup and restore but please keep the old directory.

5

ammmir commented :

@jarifibrahim unfortunately, I can’t send any files since it’s customer data. is there anything else that might help you?

if nothing else, I’ll write my own backup tool to export the data from v1 and reimport to v2, since all I really care about are the keys and values, TTL, and UserMeta.