RAM usage at high write load.

Users Badger
1

hi,

Thanks for open surcing your great work. I tried to follow the docs and wrote a simple test code to see how DB will behave under high writeload. Code is listed below.

I deployed this code on AWS c6gd.medium machine. It hase 1 CPU and 2GB RAM. I was trying to write 1000000 KV pairs, with 32 byte key and ~30000 byte value.

However, the code crushed. When DB was commiting, it started to use a lot of RAM , and the process is killed. I should note that on my home 8GB RAM machine the operation goes through.

This behaviour is quite surprising to me. I would think that DB can run low in RAM if necessary and do the write operaton even on low RAM machine.

I hope I just made some mistake somewhere, as this is my first use of Badger.

Question: what are the RAM requirements for this particular workload, and if there is some bug or mistake in the code, and if what I observed is the expected behavior?

best,
Stan

package main

import (
“fmt”
“log”
“math/rand”
“time”

badger "github.com/dgraph-io/badger"

"./src/utils"

)

func main() {
// Open the Badger database located in the /tmp/badger directory.
// It will be created if it doesn’t exist.

dir0 := "bad"
fil := dir0 + "/zal.txt"

utils.MakeDir0(dir0)

db, err := badger.Open(badger.DefaultOptions(fil))
if err != nil {
	log.Fatal(err)
}
defer db.Close()
// Your code here…

var rs = rand.NewSource(time.Now().UnixNano())
var ra = rand.New(rs)

N := 100000
big := utils.Generate_id_long(ra, N)

updates := make(map[string]string)

for i := 0; i < 1000000; i++ {
	s := utils.Generate_id(ra)
	j := rand.Intn(2 * N / 3)
	j2 := j + N/3
	if j2 > N {
		j2 = N
	}
	v := big[j:j2]
	//fmt.Println(s)
	updates[s] = v
	if i%1000 == 0 {
		fmt.Println(i)
	}
}
fmt.Println("updating db")

txn := db.NewTransaction(true)

for k := range updates {
	v := updates[k]
	if err := txn.Set([]byte(k), []byte(v)); err == badger.ErrTxnTooBig {
		_ = txn.Commit()
		txn = db.NewTransaction(true)
		_ = txn.Set([]byte(k), []byte(v))
	}
}
_ = txn.Commit()

}