I’m using BadgerDB as backend for persistent queue (to be open sourced soon). It works the following way - when pushing to queue - entry is inserted to Badger with key equal to some number from sequence. When popping out of queue - I run iterator (when iterator ended - run new one) and opening transaction for deleting the item from DB. Afterwards when some processing on queue item is done and item is Ack-ed - I commit the transaction with deletion.
The problem I faced is that iterator performance is degrading over time(even when queue pop rate is equal to push rate ~1k req/s). And from what I see - it is because LSM tree is growing even taking into account that tombstoned items are removed during compaction - some of them which are not yet removed got moved L0 -> L1 and then after their removal they stay there for even longer time for L1->L2 compaction and so on.
The only thing that helps to bring performance back to normal is restarting service and running db.Flatten() on startup that unfortunately can’t be run while db is taking updates.
My questions are:
- Could you suggest how can I fix my issue?
My current options are
WithNumLevelZeroTables(1). WithNumLevelZeroTablesStall(2). WithTruncate(true). WithKeepL0InMemory(true),
- If it’s not doable with current setup - can db.Flatten be modified to have db.prepareToDrop() so it can be run during processing (while temporary locking db for writes)