What version of Go are you using (go version)?
$ go version 1.16
What operating system are you using?
linux/windows
What version of Badger are you using?
v3
Does this issue reproduce with the latest master?
yes
Steps to Reproduce the issue
func TestRawDB(t *testing.T) {
dir, err := ioutil.TempDir("", "badger-test")
if err != nil {
panic("failed to create temp dir")
}
options := badger.DefaultOptions(dir)
options.Dir = dir
options.CompactL0OnClose = true
options.ValueThreshold = 1 << 9
options.NumVersionsToKeep = 1
db, err := badger.Open(options)
if err != nil {
fmt.Println(err)
panic("failed db open")
}
var keys [][]byte
randValue := make([]byte, 1<<12)
for i := 0; i < 1<<17; i++ {
randKey := make([]byte, 40)
_, err := rand.Read(randKey)
if err != nil {
panic("failed to read rand")
}
keys = append(keys, randKey)
err = db.Update(func(txn *badger.Txn) error {
e := badger.NewEntry(randKey, randValue)
err := txn.SetEntry(e)
return err
})
}
for j := 0; j < 4; j++ {
for i := 0; i < len(keys); i++ {
err = db.Update(func(txn *badger.Txn) error {
e := badger.NewEntry(keys[i], randValue)
err := txn.SetEntry(e)
return err
})
}
}
time.Sleep(2 * time.Minute) // https://github.com/sahib/brig/pull/96#discussion_r576542931
err = db.RunValueLogGC(0.5)
if err != nil {
fmt.Println(err)
}
db.Close()
}
What Badger options were set?
| options := badger.DefaultOptions(dir) | |
|---|---|
| options.Dir = dir | |
| options.CompactL0OnClose = true | |
| options.ValueThreshold = 1 << 9 | |
| options.NumVersionsToKeep = 1 |
What did you do?
Wrote a test to repeatedly write the same value to a key to observe GC run and reap the previous values of the key.
What did you expect to see?
After many repeated writes on a fixed set of keys, running GC would purge the old log values.
What did you see instead?
GC does not reap anything and disk usage stays the same. GC fails as there are no discardStats available on the time of running RunValueLogGC (needs call to updateDiscardStats on the vlog). It seems that discardStats are only available after doCompact runs, which only happens if the L0 LSM fills.
Ex: https://github.com/dgraph-io/badger/blob/master/levels.go#L857
https://github.com/dgraph-io/badger/blob/master/levels.go#L553
Repeatedly writing over the same set of keys does not result in L0 growth despite the value log filling with junk. Close() with option CompactL0OnClose skips the L0 full check and calls updateDiscardStats, however, Open()/Close() on a live db seems undesirable. https://github.com/dgraph-io/badger/blob/master/db.go#L604