Disable compression still compressing

Issues Badger
1

Seeing a bunch of compression logs even after disabling compression:

badger.DefaultOptions(configs.Settings.BadgerFolder).
    WithCompression(options.None)

badger 2022/01/06 13:15:36 INFO: [6] [E] LOG Compact 6->7 (1, 5 -> 5 tables with 2 splits). [1957277 . 2006545 1684842 1684861 1684863 2006571 .] -> [2007209 2007219 2007220 2007211 2007212 .], took 20.658s

What version of Go are you using (go version)?

❯ go version
go version go1.17.3 darwin/arm64

What operating system are you using?

mac os

What version of Badger are you using?

github.com/dgraph-io/badger/v3 v3.2103.2

Does this issue reproduce with the latest master?

Yes

Steps to Reproduce the issue

Same database from this:

	badgerOptions := badger.DefaultOptions(configs.Settings.BadgerFolder).
		WithLogger(logutil.NewRaftLoggerZap(common_logger.Log)).
		WithCompactL0OnClose(configs.Settings.BadgerCompress).
		WithNumCompactors(configs.Settings.BadgerNumCompactors).
		WithBlockCacheSize(4096 << 20).
		WithMaxLevels(8)

to this:

	badgerOptions := badger.DefaultOptions(configs.Settings.BadgerFolder).
		WithLogger(logutil.NewRaftLoggerZap(common_logger.Log)).
		WithCompactL0OnClose(configs.Settings.BadgerCompress).
		WithNumCompactors(configs.Settings.BadgerNumCompactors).
		WithBlockCacheSize(4096 << 20).
		WithMaxLevels(8)

	badgerOptions = badgerOptions.
		WithBlockCacheSize(0).
		WithCompression(options.None)

What Badger options were set?

What did you do?

What did you expect to see?

No compression

What did you see instead?

bunch of these:

badger 2022/01/06 13:15:29 INFO: [5] [E] LOG Compact 5->6 (1, 0 -> 1 tables with 1 splits). [2000886 . .] -> [2007189 .], took 14.152s
badger 2022/01/06 13:15:30 INFO: [7] [E] LOG Compact 5->6 (1, 0 -> 2 tables with 1 splits). [2000885 . .] -> [2007199 2007200 .], took 14.526s
badger 2022/01/06 13:15:30 INFO: [4] [E] LOG Compact 4->5 (1, 8 -> 8 tables with 3 splits). [2006505 . 2007049 2007050 2006101 2006106 2006108 2005997 2006039 2006053 .] -> [2007188 2007192 2007193 2007191 2007195 2007196 2007190 2007194 .], took 15.412s
badger 2022/01/06 13:15:35 INFO: [2] [E] LOG Compact 5->6 (1, 5 -> 7 tables with 2 splits). [2000908 . 1967910 1967918 1967934 1967913 1967926 .] -> [2007185 2007197 2007202 2007203 2007187 2007198 2007201 .], took 22.7s
badger 2022/01/06 13:15:36 INFO: [8] [E] LOG Compact 5->6 (1, 0 -> 1 tables with 1 splits). [2000878 . .] -> [2007228 .], took 11.425s
badger 2022/01/06 13:15:36 INFO: [6] [E] LOG Compact 6->7 (1, 5 -> 5 tables with 2 splits). [1957277 . 2006545 1684842 1684861 1684863 2006571 .] -> [2007209 2007219 2007220 2007211 2007212 .], took 20.658s
badger 2022/01/06 13:15:37 INFO: [9] [E] LOG Compact 6->7 (1, 4 -> 6 tables with 2 splits). [1957288 . 1647406 1647413 1646879 2006652 .] -> [2007215 2007218 2007221 2007222 2007216 2007217 .], took 19.239s
2

Those messages are compaction not compression.

Here is a diagram from Wikipedia visualizing LSM compaction:

image

4

I see, any way to speed that up??

5

That is usually bottlenecked in disk IO but it is possible it is bottlenecked by CPU - you will want to profile or monitor the system with prometheus to learn more about your system’s performance bottlenecks.