I’m currently developing a project where we need the ability to resend messages to client who have briefly disconnected from our service. For arguments sake, let’s say we need to keep 10 minutes of data cache, arranged by topic id. Data replay requests will come in and ask for items anywhere within that time window.
I have this working, using BadgerDB in managed mode. Rather than using a composite key for each item, I’m setting the max versions of the key to infinity and using the topic id as the key. When someone asks for data on a topic, I create a new key iterator which returns all the versions of the key. It works fantastically! Badger is performing very well, and most replay request are fulfilled in microseconds.
I’m using the managed db features to discard data that is outside the cache time window (also setting expiration on the items themselves). Again this works well
The issue I am facing is with the value log gc. With this approach, it seems that If I do not do frequent value log GCs, application memory seems to get out of whack- continually growing until a a gc is performed.
This is not a huge issue in and of itself, I just run GCs periodically. However, the GC process itself seems very processor intensive. If I’m understanding the value logs correctly, with the pattern I’m using, each individual value log file will become invalid all at once. There would be no need to re-write it as all the values within would be discardable.
Is there a way to optimize the GC for this scenario? I’m using 128MB for the value log file size, hoping that a smaller file would help badger recognize this scenario. Even with that, each successful gc takes around 4-5 seconds. Not a huge deal, except that with the rate data is coming in, I am basically in perpetual value log gcs, tying up and entire CPU
Fo reference, my dbConfig Options look like :
dbOpts := badger.DefaultOptions(dbName). WithNumVersionsToKeep(math.MaxInt32). // using these version to track messages per channel WithSyncWrites(false). WithMaxCacheSize(2 << 30). // 2GB cache to accomodate encryption latency WithValueLogFileSize(128 << 20). WithTruncate(true). WithNumCompactors(int(txnConcurrency)). WithLogger(log.StandardLogger()). WithEncryptionKey(randEncryptionKey()). WithEncryptionKeyRotationDuration(time.Hour)