Slow compaction progress and high memory usage

Moved from GitHub badger/1370

Posted by jarifibrahim:

What version of Badger are you using?

https://github.com/dgraph-io/badger/commit/c45d966681d4a058df1190b10f1b5f60c9ab0b6d

Steps to Reproduce the issue

Run badger benchmark write -m 1000 command.

What did you see instead?

The compactions don’t seem to make progress very slow and L0 gets accumulated with data

badger 2020/06/15 16:45:07 INFO: Got compaction priority: {level:0 score:48.4 dropPrefix:[]}
badger 2020/06/15 16:45:07 INFO: Running for level: 0
badger 2020/06/15 16:45:07 INFO: Got compaction priority: {level:1 score:60.551320023834705 dropPrefix:[]}

badger 2020/06/15 16:27:51 INFO: Got compaction priority: {level:1 score:60.551320023834705 dropPrefix:[]}
Time elapsed: 01s, bytes written: 67 MB, speed: 67 MB/sec, entries written: 419420, speed: 419420/sec
badger 2020/06/15 16:27:52 INFO: Got compaction priority: {level:1 score:60.551320023834705 dropPrefix:[]}
Time elapsed: 02s, bytes written: 101 MB, speed: 50 MB/sec, entries written: 629130, speed: 314565/sec
badger 2020/06/15 16:27:53 INFO: Got compaction priority: {level:1 score:60.551320023834705 dropPrefix:[]}
Time elapsed: 03s, bytes written: 155 MB, speed: 52 MB/sec, entries written: 970782, speed: 323594/sec
badger 2020/06/15 16:27:54 INFO: Got compaction priority: {level:1 score:60.551320023834705 dropPrefix:[]}

A score of 1 means we’ve filled the level to 100% capacity. The problem here is that because of commit Improve write stalling on level 0 and 1 · dgraph-io/badger@3747be5 · GitHub the writes do not get stalled and levels get filled up. My badger directory has 200+ level 0 tables and they were being compacted with the entire L1 (which has 300+ tables).