Request option to remove and/or disallow multiple versions of keys when the values are identical

Issues Badger
1

When using a database which allows multiple versions of a key, I would like an option which does not keep multiple copies when both the key and value are identical

db, err = badger.Open(badger.DefaultOptions(path).WithNumVersionsToKeep(math.MaxInt64))
wb := db.NewWriteBatch()
wb.Set([]byte("k1"),[]byte("v1"))
wb.Flush()
wb.Set([]byte("k1"),[]byte("v1"))
wb.Flush()
wb.Set([]byte("k1"),[]byte("v2"))
wb.Flush()
//three entries are made: k1, v1; k1, v1; k1, v2
//would like option for only two to be written: k1, v1; k1 v2

Additionally, there could be a method that removes identical versions of the same key.

2

Would it be a problem to just grab the latest version?

I don’t exactly understand what is your end goal. Perhaps we should start with that and see?

3

@chewxy this is for the same problem described at Request option to write multiple versions in a single batch - #3 by songmelted

to quote:

I have generators which produce numeric values. I have different classes of generators. Each generator has a series of input parameters.

I am producing large volumes of generator+input vs output pairs.

Outputs from a single generator are not guaranteed to be unique.

I am interested in matching outputs from one generator to another. Then analyzing which inputs lead to identical outputs from another generator.

The output values are the keys (unfortunate nomenclature). The input parameters are the values.

I then search for matching outputs (keys) and record the inputs which generated them.

Frequently, a single generator will produce the same output for different inputs within a single write batch. I need to capture each of these as they are unique and meaningful to me.

The issue at hand this time is that sometimes different generator-input combinations produce the same output value not because they are truly unique, but they are actually identical in every way.

As I am producing generator-input combinations, I can reduce them to a “standard” form so that identical combinations look identical. I would like to not write those to the database as they are not meaningful - they are literally identical - no new information. OTOH, I do want to keep any truly unique generator-input combinations even if they have the same output (key).

So, I want to be able to write and keep unique key-value pairs even when the key is the same, but I want to reject any key-value pair where both the key and the value are identical.

I can hack something together, but it won’t be anything efficient. Perhaps there is an efficient way with the existing framework?