I was reading about ways in which we could transfer posting list data from one instance to another using an RPC. This would be required in RAFT to move data around. RocksDB recommends Checkpoints for efficient Snapshots but gorocksdb doesn’t support them. So we would have to use normal Snapshots fo…

I think we use very few functions of RocksDB and gorocksdb gives us a lot more than we need. I wonder if it makes sense to maintain our own wrapper over RocksDB, just like what cockroachDB does I believe. Writing this wrapper would probably be easier than trying to SWIG RocksDB classes. And we can …

I would be happy moving away from gorocksdb. Also, if we can rely upon the C++ version of RocksDB, instead of C, that’d be pure awesomeness. C++ API is a lot more readable than the C one – and as you figured out @jchiu , also lot more up to date.

[image] pawan: which can do a Write or a BatchWrite then Async BatchWrites are the most performant. We should totally use them. I think we should use checkpoints. While @jchiu figures out a way to provide those, just add a TODO for yourself to switch to it, and do the iteration ITMT. [im…

Just wanted to provide an update on checkpointing. We already have a PR last week to support checkpointing, but there was a question about whether we can still write to the original store when checkpointing. The short answer is: Yes. Details: I tried reading RocksDB code but decide that it is bett…

Any further questions about this? @mrjn @pawan Shall we resolve this PR? Add snapshot and checkpoint dgraph-io:master ← dgraph-io:feature/rdb opened 02:28AM - 15 Sep 16 UTC (UTC) …

[image] jchiu: Note that the final result is “old” not “new” and we start writing “new” only towards the end of checkpointing. So we start writing old towards the end and not new, right? Here it almost seems like Save blocks until the goroutine above has finished writing. Is this behavior pre…

I doubt save checkpoint is blocked by writing. In a previous run (not documented here), I tried to SetOne a lot more and I can see that “Done saving checkpoint” gets printed before a few more SetOnes get printed. What I think is happening is that checkpoint doesn’t block anything. It just saves wha…

[image] jchiu: if i >= 110000 { st.SetOne(byte(“aaaaa”), byte(“old”)) } else { st.SetOne(byte(“aaaaa”), byte(“new”)) } Ok, but to me you seem to be setting old at the end. Hence my doubt that does it block. Though as you said if you do more SetOne, it saves an intermediate state. So that …

Errr yes, I make a mistake… got the signs flipped. Thanks for being alert @pawan :slight_smile: Here is an update. Code: package main import ( "fmt" "math/rand" "strconv" "time" "github.com/dgraph-io/dgraph/store" "github.com/dgraph-io/dgraph/x" ) const ( dbPath1 = "/tmp/testdbs" dbPat…

Summary of this finding and how Checkpoint and snapshot work, @pawan , @jchiu ?

Snapshots/Backup of RocksDB

Users

jchiu September 19, 2016, 2:56am 6

Any further questions about this? @mrjn @pawan

Shall we resolve this PR?

Thanks!

Topic		Replies	Views
Predicate Moves using RAFT Users	12	1954	September 22, 2016
Supporting transactions in Badger Badger	12	1649	September 24, 2017
Why we choose Badger over RocksDB in Dgraph - Dgraph Blog Blog	8	2076	April 14, 2021
QUESTION: Best way to create snapshots from badger Badger kind:question	1	504	December 9, 2020
Introducing Badger: A fast key-value store written natively in Go - Dgraph Blog Blog	32	6518	September 18, 2023

Snapshots/Backup of RocksDB

Related topics