TTL does not work on keys if I don't call db.Close()

Users Badger
1

Hi,
I am trying to use Badger as an embedded key value db. My requirements are very simple. I have very light write and read requirements. All I need to do is to use a GO based key-value db which badger provides me. Thanks for writing embedded KV db in GO.
I have an application where I open Badger db, and never closes it unless it crashes. So effectively I don’t call db.Close(). I am getting some events constantly which I store in DB. If I get the duplicate event again, I detect it and I don’t call SET on the same key again. So, effectively, I won’t have multiple versions of the key. Also, when I set the Key I set TTL value for the key and I want to make sure that the key expires once TTL value is over irrespective of my application crashes and recover again or runs forever.
Since I don’t close db, I don’t see *.sst file on the disk, hence my TTL doesn’t work after my app crash. I see all of the expired keys after my application recovers. Is there any way to dump sst files irrespective of calling db.Close() ?

2

@vgupta-mickey which version of badger are you using?
We have recently fixed similar issue TTL not restored from vlog · Issue #708 · dgraph-io/badger · GitHub. Please have a look.

3

I am using v1.5.4.

When I run dep ensure, it checkout the vendor directory with v1.5.4 .

4

Can you try master and see if the issue is still there?

5

It works. Thanks.
Another question : I don’t update my keys ever, does it make sense for me to run garbage collector periodically? Basically, I have only one version of the keys.
Also, I will have maximum only 100,000 keys and each key might have 1K value. I am not sure if the default option settings are right for me or not? I don’t see any document for tuning the default options.

6

Since you are never updating any keys, GC wont do much here.
100,000 should be a small number for badger. Hence default settings should work fine. But you can try to tweak these. Here is the file that mentions it badger/options.go at master · dgraph-io/badger · GitHub.
Let me know if you have any doubt with any param in options.

7

Is there anyway I can force to dump .sst file w/o closing the db?

8

You can try to reduce size of Memtable(MaxTableSize in options, currently its 64MB), in that case it will be filled soon and pushed to disk as SST. Hence you will have multiple SSTs. But latest Memtable will still in memory, which can be pushed to disk on close.

9

Great Thanks. I will try.

10

Hi, I have few questions on Badger which is not clear from the docs.

  1. I have one process which opens badger db /tmp/badgerdb and doing reads and writes. I want to run another process to open the same db for read only to analyze the DB? Is it possible?
  2. Do we have a way of dumping all the keys from the database?
  3. How can I connect to database from remote to analyze the database table
  4. Do I need to run ValueLogGC periodically or it is done automatically once in a while?
11

@vgupta-mickey Please find the answers below:

  1. No. Badger can be opened by a single process at a time.
  2. Yes. You can use info tool provided by Badger for this. (You can run badger info --show-keys --dir=badgerdir).
  3. Badger is an embedded key value store. It does not provide any way to connect to it from remote. You need to get Badger directory content to perform any analysis on it.
  4. It is not done automatically. You need to run it periodically. You can see an example on Badger README.
12

Thanks for the response.There is no --show-keys option, instead of --show-tables.

  1. Also, what does it mean by Listening for /debug HTTP request at port 8080? who opens port 8080?

  2. Also, what does it mean by Abnormalities:

Abnormalities:
1 extra file.
0 missing files.
0 empty files.
0 truncated manifests.

  1. my application runs forever, so I don’t close the database ever. The only time I close my application when It crashes. is it implement ok?
13
  1. You can try latest master. We have just included –show-keys options.
  2. Badger exposes /debug endpoint, if you want to collect profiling information(mostly pprof endpoints).
  3. info command analyses Badger directory. It checks for validity of all the files and reports any missing or extra file. It logs all the file related information in manifest.
  4. Yes, this is fine. However if you want to release space occupied by deleted entries in Badger, you can try to run periodic garbage collection.