Strange behavior on DB.Load

Users Badger
,
1

I’m seeing what appear to be stale reads after ingesting a backup with DB.Load. My use case frequently overwrites the same key. When I iterate through the protobuf encoded backup directly (code borrowed from DB.Load, but just printing entries), the highest version of the key matches what I would expect based on my application behavior. However when I load this backup into a badger instance, the version of the key I get is many versions earlier.

Not sure if I’m missing something or if this might be a bug. If it seems like a bug I’m happy to create github issue with more information

2

It’s looking to me like this is a bug - from what I see, DB.Load returns before the db has completed loading the backup, and if you access it immediately you may see stale results.

Attached is a backup that reproduces this issue along with code snippet below. Replace the path strings with wherever you download the snapshot. (Apparently new users can’t upload attachments)

Badger Version: github.com/dgraph-io/badger/v2 v2.2007.2

Uncomment the time.Sleep and the reader finds the correct highest version

package main

import (
	"bufio"
	"encoding/binary"
	"github.com/dgraph-io/badger/v2"
	"github.com/dgraph-io/badger/v2/pb"
	"github.com/golang/protobuf/proto"
	"io"
	"io/ioutil"
	"log"
	"os"
)

func main() {
	cpk := []byte{0, 18, 99, 104, 101, 99, 107, 112, 111, 105, 110, 116, 0, 1}
	r, _ := os.Open("/Users/michaelschiff/Desktop/example.bkp")

	dir, err := ioutil.TempDir("", "badger")
	if err != nil {
		log.Fatal(err)
	}
	config := badger.DefaultOptions(dir)
	config.MaxTableSize = 8 << 20
	db, err := badger.Open(config)
	if err != nil {
		log.Fatalf("Unable to open badgerDB: %v", err)
	}

	if err := db.Load(r, 1); err != nil {
		log.Fatal(err)
	}

	//time.Sleep(5 * time.Second)

	if err := db.View(func(txn *badger.Txn) error {
		item, err := txn.Get(cpk)
		if err != nil {
			return err
		}
		log.Printf("checkpoint version %d\n", item.Version())
		return nil
	}); err != nil {
		log.Fatal(err)
	}

	rr, _ := os.Open("/Users/michaelschiff/Desktop/example.bkp")
	br := bufio.NewReaderSize(rr, 16<<10)
	unmarshalBuf := make([]byte, 1<<10)

	var v uint64
	for {
		var sz uint64
		err := binary.Read(br, binary.LittleEndian, &sz)
		if err == io.EOF {
			break
		} else if err != nil {
			log.Fatal(err)
		}

		if cap(unmarshalBuf) < int(sz) {
			unmarshalBuf = make([]byte, sz)
		}

		if _, err = io.ReadFull(br, unmarshalBuf[:sz]); err != nil {
			log.Fatal(err)
		}

		list := &pb.KVList{}
		if err := proto.Unmarshal(unmarshalBuf[:sz], list); err != nil {
			log.Fatal(err)
		}

		for _, kv := range list.Kv {
			if v < kv.Version {
				v = kv.Version
			}
		}
	}
	log.Printf("higest version in backup %d\n", v)
}

Example output (without the sleep):

badger 2021/01/30 22:07:41 INFO: All 0 tables opened in 0s
badger 2021/01/30 22:07:41 INFO: Storing value log head: {Fid:0 Len:148 Offset:7898237}
2021/01/30 22:07:41 checkpoint version 339350
badger 2021/01/30 22:07:41 INFO: Storing value log head: {Fid:0 Len:35 Offset:14673892}
2021/01/30 22:07:41 higest version in backup 388192

Example output (with sleep):

badger 2021/01/30 22:08:40 INFO: All 0 tables opened in 0s
badger 2021/01/30 22:08:40 INFO: Storing value log head: {Fid:0 Len:148 Offset:7898237}
badger 2021/01/30 22:08:40 INFO: Storing value log head: {Fid:0 Len:35 Offset:14673892}
2021/01/30 22:08:45 checkpoint version 388192
2021/01/30 22:08:45 higest version in backup 388192
3

@chewxy @ibrahim anyway to update my permissions to allow me to upload the sample backup that reproduces this issue? I have been trying to produce a synthetic data-set that will trigger the issue, but have not yet been able.

4

This backup should reproduce the issue I’ve observed https://we.tl/t-LuHLPT7XnH (the backup file was too large to upload directly to discuss.dgraph.io so have to link a weshare)

5

Hey @michaelschiff, it looks like the issue got resolved by perf(GC): Remove move keys (#1539) · dgraph-io/badger@0a5046f · GitHub . I’ll have to investigate to figure out what actually changed but if you run badger v3.2011.1, you should be fine.

This issue is fixed on the latest release of badger v3.2011.1

6

@ibrahim thanks for the update - Ill upgrade and report back if I see anything else

7

@ibrahim @chewxy I was using a few of the options described here: Get started — that appear to have been removed in v3 - specifically LoadBloomsOnOpen and MaxTableSize. Have these options been replaced by others or is there not a way to configure this anymore?

1 Like
8

@michaelschiff MaxTableSize is splitted into MemTableSize and BaseTableSize. Use the default values if you’re unsure what to set.

As for LoadBloomsOnOpen, that option has been removed because we no longer load bloom filters in memory. They are mmaped. You can remove that option from your code.

1 Like
9

Okay thanks Ill try this

1 Like