# Badger Documentation limited or Feature removed?

**URL:** <https://discuss.dgraph.io/t/badger-documentation-limited-or-feature-removed/11676>\
**Category:** Badger\
**Tags:** kind:question, status:accepted\
**Created:** [November 30, 2020, 8:06am UTC](https://discuss.dgraph.io/t/badger-documentation-limited-or-feature-removed/11676 "2020-11-30T08:06:10Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![rocky](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/rocky/32/1162_2.png) [@rocky](https://discuss.dgraph.io/u/rocky)\
**Post date:** [November 30, 2020, 8:06am UTC](https://discuss.dgraph.io/t/badger-documentation-limited-or-feature-removed/11676/1 "2020-11-30T08:06:10Z")

</div>

Hi,

I was trying to explore a `badger` documentation for the `reverse iterator` option to fetch a value in reverse order so that the last inserted is first fetched, but I could not find it any way:

I tried searching:

Github: [GitHub - dgraph-io/badger: Fast key-value DB in Go.](https://github.com/dgraph-io/badger/)  
Badger Docs: [Get started —](https://dgraph.io/docs/badger/get-started)

Nowhere there it mentions reverse iterator - I am surprised to understand if the feature has been removed from badger?

Thanks

---

<div class="post-metadata">

**Author:** ![Naman](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/naman/32/3498_2.png) [@Naman](https://discuss.dgraph.io/u/Naman)\
**Post date:** [December 1, 2020, 7:02pm UTC](https://discuss.dgraph.io/t/badger-documentation-limited-or-feature-removed/11676/2 "2020-12-01T19:02:33Z")

</div>

You can have a look at examples in `txn_test.go`  
Here is an example code:

```auto
func TestReverse(t *testing.T) {
	dir, err := ioutil.TempDir("", "badger-test")
	require.NoError(t, err)
	defer os.RemoveAll(dir)

	ops := getTestOptions(dir).WithNumVersionsToKeep(math.MaxInt32)
	db, err := Open(ops)
	require.NoError(t, err)
	for i := 0; i < 10; i++ {
		require.NoError(t, db.Update(func(txn *Txn) error {
			require.NoError(t, txn.Set([]byte("key"), []byte(fmt.Sprintf("%05d", i))))
			return nil
		}))
	}
	fmt.Println("Reverse")
	require.NoError(t, db.View(func(txn *Txn) error {
		iopts := DefaultIteratorOptions
		iopts.Reverse = true
		iopts.AllVersions = true
		iopts.Prefix = []byte("key")
		it := txn.NewIterator(iopts)
		defer it.Close()
		for it.Rewind(); it.ValidForPrefix(iopts.Prefix); it.Next() {
			x, _ := it.Item().ValueCopy(nil)
			fmt.Printf("%+v value:%s\n", it.Item(), x)
		}
		return nil
	}))
	fmt.Println("No reverse")
	require.NoError(t, db.View(func(txn *Txn) error {
		iopts := DefaultIteratorOptions
		iopts.Reverse = false
		iopts.AllVersions = true
		iopts.Prefix = []byte("key")
		it := txn.NewIterator(iopts)
		defer it.Close()
		for it.Rewind(); it.ValidForPrefix(iopts.Prefix); it.Next() {
			x, _ := it.Item().ValueCopy(nil)
			fmt.Printf("%+v value:%s\n", it.Item(), x)
		}
		return nil
	}))
	require.NoError(t, db.Close())
}

```

It outputs:

```auto
Reverse
key="key", version=1, meta=40 value:00000
key="key", version=2, meta=40 value:00001
key="key", version=3, meta=40 value:00002
key="key", version=4, meta=40 value:00003
key="key", version=5, meta=40 value:00004
key="key", version=6, meta=40 value:00005
key="key", version=7, meta=40 value:00006
key="key", version=8, meta=40 value:00007
key="key", version=9, meta=40 value:00008
key="key", version=10, meta=40 value:00009
No reverse
key="key", version=10, meta=40 value:00009
key="key", version=9, meta=40 value:00008
key="key", version=8, meta=40 value:00007
key="key", version=7, meta=40 value:00006
key="key", version=6, meta=40 value:00005
key="key", version=5, meta=40 value:00004
key="key", version=4, meta=40 value:00003
key="key", version=3, meta=40 value:00002
key="key", version=2, meta=40 value:00001
key="key", version=1, meta=40 value:00000

```

---

<div class="post-metadata">

**Author:** ![chewxy](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/chewxy/32/4339_2.png) [@chewxy](https://discuss.dgraph.io/u/chewxy)\
**Post date:** [December 1, 2020, 9:59pm UTC](https://discuss.dgraph.io/t/badger-documentation-limited-or-feature-removed/11676/3 "2020-12-01T21:59:46Z")

</div>

ping [@docs](https://discuss.dgraph.io/groups/docs)

---

<div class="post-metadata">

**Author:** ![rocky](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/rocky/32/1162_2.png) [@rocky](https://discuss.dgraph.io/u/rocky)\
**Post date:** [December 4, 2020, 5:42pm UTC](https://discuss.dgraph.io/t/badger-documentation-limited-or-feature-removed/11676/4 "2020-12-04T17:42:37Z")

</div>

@Naman - No, I dont’ need reverse for value, but for KEY.

I want to store some data in key with unique date and time and I want to list all value based on date on reverse order.

Date-Time is mentioned on key. in the format “2006-01-02-12-06-10”

---

<div class="post-metadata">

**Author:** ![Naman](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/naman/32/3498_2.png) [@Naman](https://discuss.dgraph.io/u/Naman)\
**Post date:** [December 8, 2020, 8:52am UTC](https://discuss.dgraph.io/t/badger-documentation-limited-or-feature-removed/11676/5 "2020-12-08T08:52:04Z")

</div>

> [@rocky](#):
>
> No, I dont’ need reverse for value, but for KEY.

To clarify, the example that I posted was to demo iterating over the same key with different versions. Be default the only the latest one is fetched. If you set `iopts.AllVersions = true`, then you get all versions (latest → oldest). If you also set `iopts.Reverse = true`, you get all versions (oldest → latest). This has nothing to do with `Value`. It is all to do with `Version`.

If your whole DB contains Keys in format “YYYY-MM-DD-hh-mm-ss”, then you can simply iterate over the DB by something like this. This would give you all your keys in ascending order of your Keys.

```go
for itr.Rewind(); itr.Valid(); itr.Next() {
  // Do work
}

```

---

<div class="post-metadata">

**Author:** ![rocky](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/rocky/32/1162_2.png) [@rocky](https://discuss.dgraph.io/u/rocky)\
**Post date:** [December 8, 2020, 6:00pm UTC](https://discuss.dgraph.io/t/badger-documentation-limited-or-feature-removed/11676/6 "2020-12-08T18:00:09Z")

</div>

When you mention “version” - Do you mean, Each time the same key is updated, they are saved separately? If so, I don’t want this - I do not want to store multiple value (version based) - It is just going to take unlawful storage, How can I avoid it?

---

<div class="post-metadata">

**Author:** ![Naman](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/naman/32/3498_2.png) [@Naman](https://discuss.dgraph.io/u/Naman)\
**Post date:** [December 8, 2020, 6:48pm UTC](https://discuss.dgraph.io/t/badger-documentation-limited-or-feature-removed/11676/7 "2020-12-08T18:48:11Z")

</div>

> [@rocky](#):
>
> How can I avoid it?

You need not. 🙂  
`NumVersionsToKeep` is 1 by default in badger. I set it to `math.MaxInt32` for the sake of example.

> [@Naman](#):
>
> ```auto
> ops := getTestOptions(dir).WithNumVersionsToKeep(math.MaxInt32)
> 
> ```

---

<div class="post-metadata">

**Author:** ![rocky](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/rocky/32/1162_2.png) [@rocky](https://discuss.dgraph.io/u/rocky)\
**Post date:** [December 16, 2020, 10:35pm UTC](https://discuss.dgraph.io/t/badger-documentation-limited-or-feature-removed/11676/8 "2020-12-16T22:35:23Z")

</div>

Hi, It’s confusing to see the answer from the core team without sharing much detail.

You didn’t answer -

- Each time a key is updated, are they stored separately along with the previous value?

If so, I don’t want to store the previous value. I want to overwrite it, right away. Is there any such option with badger?

- Is there any performance difference in overwriting?
- how long does the old value remains inside my database file, If I don’t overwrite or if badger has no such option overwrite?
- Are they cleaned up using an auto garbage collector? or Do I need to run the garbage collector myself to fresh it?

Thanks

---

<div class="post-metadata">

**Author:** ![Naman](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/naman/32/3498_2.png) [@Naman](https://discuss.dgraph.io/u/Naman)\
**Post date:** [December 18, 2020, 11:52am UTC](https://discuss.dgraph.io/t/badger-documentation-limited-or-feature-removed/11676/9 "2020-12-18T11:52:14Z")

</div>

Sorry for the lack of clarity.

Badger, a key-value database is inspired from [RocksDB](https://github.com/facebook/rocksdb/wiki) and uses [LSM](https://en.wikipedia.org/wiki/Log-structured_merge-tree) tree as the underlying data structure. LSM tree is optimized for heavy-write workloads which is performed by performing sequential writes. This means that a new entry(key-value) would be inserted into the Tree even if the key for it already exists.  
For cleaning up of tree (removing old/stale data), it has to rely upon [Compaction](https://github.com/facebook/rocksdb/wiki/Leveled-Compaction). This compaction process runs periodically, identifies the stale data and cleans up the tree.

Badger also provides a mechanism to keep multiple versions of same key (by setting `NumVersionsToKeep`). This is useful for various purposes. [Dgraph](https://github.com/dgraph-io/dgraph) uses this property of Badger.

For simplicity let’s assume you want to keep a single version of a Key, i.e., if you update a key with a new value, you don’t care about the older value. You want that older value to be cleaned up.

> [@rocky](#):
>
> Each time a key is updated, are they stored separately along with the previous value?

Each time you set a key-value pair (say key1: value2), a new entry(key-value pair) is inserted into the DB. As you have set `NumVersionsToKeep=1` (its the default in badger), the older entry(key1: value1) for the same key is marked as stale. This value is no longer visible to you and would eventually be cleaned up by the compaction process. Now if you do `txn.Get(key1)` you would get `value2`.

> [@rocky](#):
>
> Is there any performance difference in overwriting?

The performance difference is only equivalent to inserting a new Key Value. While the memory would eventually be cleaned. But if your values are very large (`>1KB`) and you are frequently updating that, then garbage collection would take some time.

> [@rocky](#):
>
> how long does the old value remains inside my database file, If I don’t overwrite or if badger has no such option overwrite?

No time constraint as such. But it would eventually be cleaned up maintaining your DB in a healthy state.

> [@rocky](#):
>
> Are they cleaned up using an auto garbage collector? or Do I need to run the garbage collector myself to fresh it?

Automatic garbage collector (compaction + value log GC) will clean that up for you. You don’t need to run it yourself.

Please feel free to ask further clarifications.

---

<div class="post-metadata">

**Author:** ![rocky](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/rocky/32/1162_2.png) [@rocky](https://discuss.dgraph.io/u/rocky)\
**Post date:** [December 18, 2020, 8:34pm UTC](https://discuss.dgraph.io/t/badger-documentation-limited-or-feature-removed/11676/10 "2020-12-18T20:34:16Z")

</div>

Now, this should go to either [@badger](https://discuss.dgraph.io/groups/badger)[@docs](https://discuss.dgraph.io/groups/docs) or FAQ .

That is some information.

I have just one thing that shook me little:

> [@Naman](#):
>
> But if your values are very large (`>1KB`) and you are frequently updating that, then garbage collection would take some time.

`1KB` is considered as very large value. Your post that shares above information is larger than 1 KB, is it very large value?

I have article based community and with badger requirement and my average value size would be 2 KB - 10 KB or more ( in some case ) ~ Are they really considered `very large` values?

**Thank you for clarifying doubts.**

Thanks

---

<div class="post-metadata">

**Author:** ![Naman](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/naman/32/3498_2.png) [@Naman](https://discuss.dgraph.io/u/Naman)\
**Post date:** [December 21, 2020, 5:45pm UTC](https://discuss.dgraph.io/t/badger-documentation-limited-or-feature-removed/11676/11 "2020-12-21T17:45:16Z")

</div>

> [@rocky](#):
>
> `1KB` is considered as very large value. Your post that shares above information is larger than 1 KB, is it very large value?

The default value Is set to `1KB`. You can set the [`ValueThreshold`](https://github.com/dgraph-io/badger/blob/afaf02216411616cde51caaecff85a66c323f00f/options.go#L318-L327) to higher value as per your needs.

There are no benchmarks regarding this Threshold. It depends upon data access etc.

With higher value threshold, LSM tree would be bigger for same amount of data and each table will contain less entries. So, search speed would be slightly slower. I suggest you to not deviate too much from the DefaultValue.