Lower memory usage

Issues Badger
1

What are the recommendation to lower memory usage? I’ve found Clarify "Memory usage" section in readme · Issue #1304 · dgraph-io/badger · GitHub and then Get started — but those looks outdated as Options.ValueLogLoadingMode doesn’t exists anymore

1 Like
2

Same problem here, were used to use “FileIO” mode under low memory environment, but v3 removed those options and value log cannot be changed to smaller than 1GB (1<<20).

We have a device that has only 2GB ram available, and the value log alone mmaps 2GB, which makes badger totally unusable.

@Naman any solution to this?

3

Hey @maddie , the mmap of 2GB vlog files should not make badger unusable on low memory machines. As far as I understand, we can mmap ~2<<64B on 64-bit machine.

4

@Naman Thanks for your response.

I wrote a simple program, as below:

package main

import (
	"log"

	"github.com/dgraph-io/badger/v3"
)

func main() {
	opt := badger.DefaultOptions("/tmp/test")
	db, err := badger.Open(opt)
	if err != nil {
		log.Fatal(err)
	}
	defer db.Close()
}

Then this is what I get when running the program:

$ go build -o badgerv3-test -ldflags "-w -s" -trimpath ./main.go
$ ./badgerv3-test
badger 2021/03/09 11:42:35 INFO: All 0 tables opened in 0s
badger 2021/03/09 11:42:35 INFO: Discard stats nextEmptySlot: 0
badger 2021/03/09 11:42:35 INFO: Set nextTxnTs to 0
badger 2021/03/09 11:42:35 ERROR: Received err: Error while creating log file in valueLog.open error: while opening file: /tmp/test/000001.vlog error: cannot allocate memory
while mmapping /tmp/test/000001.vlog with size: 2147483646
github.com/dgraph-io/ristretto/z.OpenMmapFileUsing
	github.com/dgraph-io/ristretto@v0.0.4-0.20210122082011-bb5d392ed82d/z/file.go:59
github.com/dgraph-io/ristretto/z.OpenMmapFile
	github.com/dgraph-io/ristretto@v0.0.4-0.20210122082011-bb5d392ed82d/z/file.go:86
github.com/dgraph-io/badger/v3.(*logFile).open
	github.com/dgraph-io/badger/v3@v3.2011.1/memtable.go:556
github.com/dgraph-io/badger/v3.(*valueLog).createVlogFile
	github.com/dgraph-io/badger/v3@v3.2011.1/value.go:515
github.com/dgraph-io/badger/v3.(*valueLog).open
	github.com/dgraph-io/badger/v3@v3.2011.1/value.go:571
github.com/dgraph-io/badger/v3.Open
	github.com/dgraph-io/badger/v3@v3.2011.1/db.go:334
main.main
	command-line-arguments/main.go:11
runtime.main
	runtime/proc.go:225
runtime.goexit
	runtime/asm_386.s:1315. Cleaning up...
2021/03/09 11:42:35 During db.vlog.open error: Error while creating log file in valueLog.open error: while opening file: /tmp/test/000001.vlog error: cannot allocate memory
while mmapping /tmp/test/000001.vlog with size: 2147483646
github.com/dgraph-io/ristretto/z.OpenMmapFileUsing
	github.com/dgraph-io/ristretto@v0.0.4-0.20210122082011-bb5d392ed82d/z/file.go:59
github.com/dgraph-io/ristretto/z.OpenMmapFile
	github.com/dgraph-io/ristretto@v0.0.4-0.20210122082011-bb5d392ed82d/z/file.go:86
github.com/dgraph-io/badger/v3.(*logFile).open
	github.com/dgraph-io/badger/v3@v3.2011.1/memtable.go:556
github.com/dgraph-io/badger/v3.(*valueLog).createVlogFile
	github.com/dgraph-io/badger/v3@v3.2011.1/value.go:515
github.com/dgraph-io/badger/v3.(*valueLog).open
	github.com/dgraph-io/badger/v3@v3.2011.1/value.go:571
github.com/dgraph-io/badger/v3.Open
	github.com/dgraph-io/badger/v3@v3.2011.1/db.go:334
main.main
	command-line-arguments/main.go:11
runtime.main
	runtime/proc.go:225
runtime.goexit
	runtime/asm_386.s:1315

This is running under 32-bit Linux

$ uname -a
Linux localhost 3.18.22-rtai-cpac #80 SMP PREEMPT Sat Jun 10 16:07:08 CST 2017 i686 i686 i686 GNU/Linux

$ free
             total       used       free     shared    buffers     cached
Mem:       1962036    1892244      69792      84216     145084     537044
-/+ buffers/cache:    1210116     751920
Swap:      1989628      20028    1969600

FWIW, it was working fine with FileIO option in v2.

The workaround I’m using for now is forking the repo and remove the lower limit check for ValueLogFileSize in db.go#L171:

	// ValueLogFileSize should be stricly LESS than 2<<30 otherwise we will
	// overflow the uint32 when we mmap it in OpenMemtable.
-	if !(opt.ValueLogFileSize < 2<<30 && opt.ValueLogFileSize >= 1<<20) {
+	if !(opt.ValueLogFileSize < 2<<30) {
		return ErrValueLogSize
	}

… and setting ValueLogFileSize to a lower value (in my case 1<<15) and it works without panic, although the actual functional/performance impact is yet to be determined.

5

@maddie , we don’t actively support 32-bit machines.
You can reduce the valuelog threshold using WithValueLogFileSize db option while opening DB. You don’t need to remove the lower limit check I believe.

6

@Naman I tried to use WithValueLogFileSize to set it to 1<<15, but as you can see in db.go the lower limit check is opt.ValueLogFileSize >= 1<<20, which returns error ErrValueLogSize. That’s why I had to fork the repo and removed the check.

Does that mean it can actually be smaller than 1<<20? If so, what should the minimum be? I’m happy to submit a PR for this change.

7

Any suggestions? @Naman

8

Hi @maddie, I don’t think there will be anything wrong in reducing the lower bound check. Just that you will not be able to insert entries which have values bigger than this threshold.

1<<20 (1MB looks like a sane check) and should work on 32-bit machine as well. So, running your program with a value log threshold of 1<<20 should work.

9

Oh yes, indeed. I was confused.

Thanks!