I’ve implemented an abstraction layer on-top of badger with convenience functions like nodes and lists. I’m trying to get paging correct and am hoping to get some expert opinions if this is the best way to go about this.
The BadgerNode has a prefix on the iterator to make sure that only valid keys are parsed and not more nodes. One optimization would be if I could exclude certain keys, specifically keys that end with a suffix. We have key prefix, why not key suffix?
Thanks!
edit:
Another optimization might be to store the node keys as another entity and retrieve them on node creation. Although, I like the simplicity of the nodes only writing to the database if the nodes themselves have values stored, if something creates a database node but never writes to it, there is no database access.
We can’t iterate over keys with a suffix due to design principle of LSM tree. Badger uses LSM tree as underlying data structure. In each level (except L0), keys are sorted. This is to make read/write efficient. Comparison of 2 byte-slices, be default, happens from left to right hence allowing badger optimising read over keys with a prefix.
My solution is to create an entity prefix so the db and nodes can iterate over each nodes actual data separately of each other. I’ve created backup and restore methods with version control so the db looks for the version setting, if it is lower than current, it backs up to a zip.Writer, closes the database, renames the folder structure, creates a new databases, then imports the backups with the new entity prefix. Seems to work pretty good.