What triggers compactions of LSM trees in badgerDB?

Users Badger
1

Hi Guys,
We are evaluating using badgerdb v3 here at Mixpanel and trying to figure out how compactions work. What triggers compactions currently ? Does it only runs on db.Close() or it happens periodically based on some trigger? Is there a way to trigger compactions manually?
Thanks,
Harshal

2

Hey @harshalchaudhari, welcome to the community and thanks for reaching out.
In badger, compactions run in the background periodically by a number of goroutines(configurable via NumCompactors db option). Currently, there is no way to trigger compaction manually. We didn’t feel the need to do so. Running compaction on db close is configurable via db option CompactL0OnClose.

How does compaction works?

  1. A bunch of goroutines do compaction in background periodically.
  2. One of those go routine gives high priority to L0 → L1 compaction to avoid stalling of writes.
  3. We calculate the level sizes and the target level sizes which is based on idea Dynamic Level Sizes. Dynamic Level Size for Level-Based Compaction | RocksDB
  4. Then, the compaction priority is decided among the levels. We try to maintain a healthy LSM tree. Based on Leveled Compaction · facebook/rocksdb Wiki · GitHub
  5. The compaction is triggered for those levels. One of the table from L(i-1) is picked and merged with overlapping tables from L(i). This single compaction is optimized by doing subcompactions of non-overlapping ranges in parallel. See Leveled Compaction · facebook/rocksdb Wiki · GitHub

Feel free to ask follow up questions.