Large vlog file > 2Gb while setting 2 key value pairs

Issues Badger
1

What version of Go are you using (go version)?

go version go1.16.5 linux/amd64

What operating system are you using?

Fedora 33 (Custom Build)

What version of Badger are you using?

v3.2103.0

Does this issue reproduce with the latest master?

Haven’t tried that.

Steps to Reproduce the issue

var storeDB utils.SDB
type DeploymentConfig struct {
        Name    string
        Version string
}
name := "testDC"
var dc utils.DeploymentConfig
dc.Name = "testMeta"
dc.Version = "00.00.0.init"

err := storeDB.Set(name, dc)
if err != nil {
  Log.Error(err)
  return err
}

type (
  SDB interface {
    Get(key string) (value *DeploymentConfig, err error)
    Set(key string, value *DeploymentConfig) error
    Delete(key string) error
    Close() error
  }
  StoreDB struct {
    db *badger.DB
   }
)

func (sdb *StoreDB) Set(key string, value *DeploymentConfig) error {
  err := sdb.db.Update(func(txn *badger.Txn) error {
    DBS, err := sdb.Get(key)
    if err != nil {
      Log.Debug_f("key [%s] not found: may be initial insert with key", key)
    }
    var updatedValue = *value
    mergo.Merge(&updatedValue, DBS)
    err = txn.Set([]byte(key), updatedValue.encodeDeploymentConfig())
    Log.Info(value.Name)
    if err != nil {
      Log.Error_f("Unable to set key [%s] with value [%s]\n", key, value)
      return err
    }
    return err
  })
  return err
}

func InitDB(dbFile string) (SDB, error) {
  bdb, err := badger.Open(badger.DefaultOptions(dbFile))
  if err != nil {
    Log.Error(err)
  }
  sdb := &StoreDB{
    db: bdb,
  }
  return sdb, err
}

What Badger options were set?

badger.DefaultOptions

What did you do?

Insert two key value pairs. The daemon has been running for approx. 12 hours.

What did you expect to see?

An appropriately sized database, the dataset is < 1k

What did you see instead?

./badger info --dir .
Listening for /debug HTTP requests at port: 8080
jemalloc enabled: false
Using Go memory

[     2021-06-16T06:53:17Z] MANIFEST      128 B MA
[            1 day earlier] 000003.sst    303 B L0
[            1 day earlier] 000004.sst    303 B L0
[            1 day earlier] 000005.sst    289 B L0
[                      now] 000007.sst    299 B L0
[            1 day earlier] 000006.sst    435 B L6
[                      now] 000021.vlog    20 B VL
[                      now] 000022.vlog  2.0 GiB VL

[EXTRA]
[2021-06-14T19:09:08Z] .bash_history 5.4 KiB
[2021-04-21T18:49:52Z] .kube        4.0 KiB
[2021-04-16T23:40:05Z] .rancher     4.0 KiB
[2021-04-29T05:49:40Z] .ssh         4.0 KiB
[2021-06-16T06:53:17Z] 00022.mem    128 MiB
[2021-06-11T09:28:20Z] DISCARD      1.0 MiB
[2021-06-11T09:28:20Z] KEYREGISTRY    28 B
[2021-06-16T06:53:17Z] LOCK            6 B
[2021-06-16T07:00:23Z] badger       16 MiB
[2021-04-16T23:10:44Z] libvirt      4.0 KiB
[2021-06-11T09:18:44Z] nohup.out       0 B
[2021-04-21T03:20:49Z] setup_env.sh   49 B

[Summary]
Level 0 size:      1.2 KiB
Level 1 size:          0 B
Level 2 size:          0 B
Level 3 size:          0 B
Level 4 size:          0 B
Level 5 size:          0 B
Level 6 size:        435 B
Total SST size:    1.6 KiB
Value log size:    2.0 GiB

Abnormalities:
12 extra files.
0 missing files.
0 empty files.
0 truncated manifests.
Error: failed to open database err: while opening memtables error: while opening fid: 22 error: while updating skiplist error: end offset: 20 < size: 134217728 error: Log truncate required to run DB. This might result in data loss
github.com/dgraph-io/badger/v3.init
	/home/hickersonj/Downloads/badger-3.2103.0/errors.go:101
runtime.doInit
	/usr/local/go/src/runtime/proc.go:6309
runtime.doInit
	/usr/local/go/src/runtime/proc.go:6286
runtime.doInit
	/usr/local/go/src/runtime/proc.go:6286
runtime.main
	/usr/local/go/src/runtime/proc.go:208
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1371
2 Likes