Memory issue during stream operation

Issues Ristretto
,
1

What version of Go are you using (go version)?

$ go version
1.16.0

What operating system are you using?

Linux. It is happening inside docker image: gcr.io \ distroless \ static@sha256:c6d5981545ce1406d33e61434c61e9452dad93ecd8397c41e89036ef977a88f4

What version of ristretto are you using?

v3.2011.1

Does this issue reproduce with the latest master?

v3.2011.1 is the latest stable version.

Steps to Reproduce the issue

Run the application and wait until it is killed as Out of memory.

What ristretto options were set?

Default options. Only path and log level are set.

What did you do?

My application is running a stream operation regularly that scans iterates values and my application is killed due to memory after some period. I checked pprof output:

(pprof) top
Showing nodes accounting for 531.20MB, 99.08% of 536.13MB total
Dropped 19 nodes (cum <= 2.68MB)
Showing top 10 nodes out of 18
      flat  flat%   sum%        cum   cum%
     448MB 83.56% 83.56%      448MB 83.56%  github.com\dgraph-io\ristretto\z.Calloc (inline)
   83.20MB 15.52% 99.08%    83.20MB 15.52%  github.com\dgraph-io\badger\v3\skl.newArena (inline)
         0     0% 99.08%       85MB 15.85%  github.com\bgokden\veri\data.(*Data).InitData
         0     0% 99.08%       85MB 15.85%  github.com\bgokden\veri\data.(*Dataset).CreateIfNotExists
         0     0% 99.08%       85MB 15.85%  github.com\bgokden\veri\data.(*Dataset).GetOrCreateIfNotExists
         0     0% 99.08%       85MB 15.85%  github.com\bgokden\veri\node.(*Node).CreateDataIfNotExists
         0     0% 99.08%       85MB 15.85%  github.com\bgokden\veri\veriservice._VeriService_CreateDataIfNotExists_Handler
         0     0% 99.08%    83.20MB 15.52%  github.com\dgraph-io\badger\v3.(*DB).newMemTable
         0     0% 99.08%    83.20MB 15.52%  github.com\dgraph-io\badger\v3.(*DB).openMemTable
         0     0% 99.08%      448MB 83.56%  github.com\dgraph-io\badger\v3.(*Stream).Orchestrate.func1

Most of the memory is being used by github.com/dgraph-io/ristretto/z.Calloc
I don’t know if there is a way to free it.

What did you expect to see?

This memory is freed after some period.

What did you see instead?

Continuously increasing memory.

Sorry I had to convert all slashes with reverse slash.

2

I’m trying to understand: is this a problem with Badger that you are facing?

Can you tell me what your application is doing ?

3

@bgokden can you try to run badger with jemalloc?
You will have to install jemalloc locally. Please see this

And then, you’ll have to build your go program with jemalloc. To do that, run go build -tags=jemalloc ... .

You should see a significant improvement in badger’s memory utilization with jemalloc

1 Like
4

@ibrahim can we create a makefile for Badger? Or document in Readme about jemalloc?

5

This looks like a solution that I can apply. I will try and reply here.

6

I think this can be called the solution.
I added jemalloc and compiled with jemalloc.
I was using an alpine image but the current alpine doesn’t have support so I moved to debian image.
Also, I was compiling with CGO_ENABLED=0, removed it.
Base memory usage of the application increased but it is stable now.
I was trying to keep my application with no C dependencies and pure go.

Thank you for showing me the right direction.

1 Like
7

@ibrahim can we create a makefile for Badger? Or document in Readme about jemalloc?

Badger is used as a library so the users won’t benefit even if there was a makefile. We can document this in the README file and badger’s documentation.

I’ll get it done.