Iterator perfomance degradation

Users Badger
1

I’m using BadgerDB as backend for persistent queue (to be open sourced soon). It works the following way - when pushing to queue - entry is inserted to Badger with key equal to some number from sequence. When popping out of queue - I run iterator (when iterator ended - run new one) and opening transaction for deleting the item from DB. Afterwards when some processing on queue item is done and item is Ack-ed - I commit the transaction with deletion.

The problem I faced is that iterator performance is degrading over time(even when queue pop rate is equal to push rate ~1k req/s). And from what I see - it is because LSM tree is growing even taking into account that tombstoned items are removed during compaction - some of them which are not yet removed got moved L0 → L1 and then after their removal they stay there for even longer time for L1->L2 compaction and so on.

The only thing that helps to bring performance back to normal is restarting service and running db.Flatten() on startup that unfortunately can’t be run while db is taking updates.

My questions are:

  1. Could you suggest how can I fix my issue?
    My current options are
			WithNumLevelZeroTables(1).
			WithNumLevelZeroTablesStall(2).
			WithTruncate(true).
			WithKeepL0InMemory(true),
  1. If it’s not doable with current setup - can db.Flatten be modified to have db.prepareToDrop() so it can be run during processing (while temporary locking db for writes)
1 Like
2

ping @ibrahim - I literally just watched a talk you gave on iterators.

3

Hey @sev3ryn, thank you for investigating the issue. The increase in latency is because the deleted items are not being removed. This might be happening because you have older versions of the same item on higher LSM tree levels (0 is the lowest). The deleted entries are removed only when they’re at the same level as their un-deleted versions.

We stop the compactions while running DB.Flatten. This would also block the memtable flush (if L0 doesn’t have space) and so your writes would also get blocked. @sev3ryn Have you tried running DB.Flatten on an online badger instance?

4

Thanks for the answer @ibrahim,

looks like I’ve found a way to pause all the writes in my library while I’m running db.Flatten.

The only info I’m missing

  1. Is transaction Discard is considered a write?
  2. Is any operation in transaction before Commit is considered a write?
5

No. Discard will drop the non-committed writes in the transaction

No. If you don’t commit, badger doesn’t consider it as a write. (The same transaction can read it but other transactions cannot)