# Feature Request: Allow scanning value log sequentially in file order **URL:** \ **Category:** Badger\ **Tags:** badger, kind:feature, good-first-issue\ **Created:** [June 14, 2020, 6:27pm UTC](https://discuss.dgraph.io/t/feature-request-allow-scanning-value-log-sequentially-in-file-order/8676 "2020-06-14T18:27:52Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![diggy](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/diggy/32/3666_2.png) [@diggy](https://discuss.dgraph.io/u/diggy)\ **Post date:** [June 14, 2020, 6:27pm UTC](https://discuss.dgraph.io/t/feature-request-allow-scanning-value-log-sequentially-in-file-order/8676/1 "2020-06-14T18:27:52Z")
**Moved from GitHub [badger/1368](https://github.com/dgraph-io/badger/issues/1368)** _Posted by_ [eloff](https://github.com/eloff): I’m considering running BadgerDB on HDDs for a use case where I only insert items, and then sometimes scan the entire database. Obviously with limited random access abilities of a HDD, these scans would be many orders of magnitude more efficient if they could walk the value log sequentially in file order. Am I right that there isn’t currently a way to do this or did I miss something? Could such an iterator be implemented easily in BadgerDB? I think so from what I know of the architecture, but I’m not familiar with the internals. Interestingly, if there was a feature like [Feature Request: add a hook to customize SST compaction · Issue #1367 · dgraph-io/badger · GitHub](https://github.com/dgraph-io/badger/issues/1367) one could maybe abuse the callback and the compaction to not only scan the entire value log in order, but to compact it at the same time. ---
**Author:** ![diggy](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/diggy/32/3666_2.png) [@diggy](https://discuss.dgraph.io/u/diggy)\ **Post date:** [June 18, 2020, 11:45am UTC](https://discuss.dgraph.io/t/feature-request-allow-scanning-value-log-sequentially-in-file-order/8676/2 "2020-06-18T11:45:01Z")
[jarifibrahim](https://github.com/jarifibrahim) _commented_ : Hey @eloff, I’d like to understand this better. > these scans would be many orders of magnitude more efficient if they could walk the value log sequentially in file order. Badger stores data in sorted form. How would reading a value log file help you? Also, value log is the write ahead log file and sst is where the index and keys are stored. ---
**Author:** ![diggy](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/diggy/32/3666_2.png) [@diggy](https://discuss.dgraph.io/u/diggy)\ **Post date:** [June 20, 2020, 11:26pm UTC](https://discuss.dgraph.io/t/feature-request-allow-scanning-value-log-sequentially-in-file-order/8676/3 "2020-06-20T23:26:44Z")
[eloff](https://github.com/eloff) _commented_ : Hi @jarifibrahim, I might have an incorrect mental model here, so please bear with me. What I’m thinking is the value log contains all key-value pairs, which may also be duplicated in the SST (or just the key and a pointer into the value log). From what I understand the value log is the write-ahead log, so it must have the keys as well. So if one has a use case where you want to scan all key-value pairs, and the values tend to be stored in the value log and not the SST, then it’s more efficient to walk the value log in file-order then to walk the keys in the SST and look up the value via random access. The limitations are of course the data won’t be ordered by key and you’d need to handle deleted key-values manually., When you find the key-value pair you don’t know at that moment if you’ll find a delete later on in the value log. If your database is append-only then you don’t have to worry about that at all. My use case specifically is conducting a brute force search over all items in an append-only database. Scanning the value log sequentially would be the most efficient way to do that and would allow the OS to perform aggressive read-ahead. On a spinning rust type device, the performance difference would be multiple orders of magnitude. ---
**Author:** ![diggy](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/diggy/32/3666_2.png) [@diggy](https://discuss.dgraph.io/u/diggy)\ **Post date:** [July 2, 2020, 10:04am UTC](https://discuss.dgraph.io/t/feature-request-allow-scanning-value-log-sequentially-in-file-order/8676/4 "2020-07-02T10:04:21Z")
[jarifibrahim](https://github.com/jarifibrahim) _commented_ : @eloff Thanks for the detailed explanation. What you’re looking for is similar to `vlog.Iterate`. [badger/value.go at e013bfd25899ec014d7117fa6e94b17a114d9921 · dgraph-io/badger · GitHub](https://github.com/dgraph-io/badger/blob/e013bfd25899ec014d7117fa6e94b17a114d9921/value.go#L403-L405) We should be able to wrap this function in a public API and expose it so that people can iterate the value log file in a sequential manner. This has to be a read-only operation. ---
**Author:** ![eloff](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/eloff/32/4030_2.png) [@eloff](https://discuss.dgraph.io/u/eloff)\ **Post date:** [July 20, 2020, 1:36pm UTC](https://discuss.dgraph.io/t/feature-request-allow-scanning-value-log-sequentially-in-file-order/8676/5 "2020-07-20T13:36:59Z")
That looks exactly like what I’m talking about, having a public API around that would be great. ---
**Author:** ![Anurag](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/anurag/32/2599_2.png) [@Anurag](https://discuss.dgraph.io/u/Anurag)\ **Post date:** [July 20, 2020, 1:59pm UTC](https://discuss.dgraph.io/t/feature-request-allow-scanning-value-log-sequentially-in-file-order/8676/6 "2020-07-20T13:59:46Z")
@ibrahim this looks like a good first issue for @Naman or anyone else starting to work on Badger. Do you have an ETA in mind for this? ---
**Author:** ![ibrahim](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/ibrahim/32/6387_2.png) [@ibrahim](https://discuss.dgraph.io/u/ibrahim)\ **Post date:** [July 20, 2020, 2:20pm UTC](https://discuss.dgraph.io/t/feature-request-allow-scanning-value-log-sequentially-in-file-order/8676/7 "2020-07-20T14:20:49Z")
I’ve tagged this as a good first issue.