Understanding Raft WAL Logs

[pawan (Pawan Rawal)]

This is regarding the change to commit logs to make them compatible with RAFT. So I have been going through https://github.com/coreos/etcd/tree/master/contrib/raftexample to understand how Raft works in general and what part the WAL play. These guys have built up an example which has a key value store backed by Raft. It took rather long to wrap my head around this but I feel I have a much better understanding of the code there now.

1. From what I understand WAL is used to bring back data from the disk(WAL files) into memory(raftStorage in example) https://github.com/coreos/etcd/blob/master/contrib/raftexample/raft.go#L160 when a node starts, restarts.

2. All requests that come to the master are written to the WAL and the raftStorage. Later when they are committed(Proposition is accepted by a quorum), the are sent over the commit channel which then updates the KV store.

WAL also has the ability to store snapshots where while replaying your can open WAL from a particular snapshot and read entries from that point on.

The WAL files are of 64M each and also store HardState(https://godoc.org/github.com/coreos/etcd/raft/raftpb#HardState) apart from the entries. I have to now figure that how we use and modify our commit logs so that they work with RAFT.

[mrjn (Manish R Jain)]

This is pretty accurate. Our commit logs are also 50MB each or something.

[pawan (Pawan Rawal)]

@mrjn Could you also please summarize the discussion we had yesterday? One of the things we decided was that we would get rid of the cache.

Also for reference

RAFT based Changes to commit logs

For Dgraph, RocksDB is our state machine. When we commit our dirty posting lists to RocksDB, and once we have confirmed sync to disk, we can discard the logs. This mechanism is great because we can avoid storing our commit logs. Each mutation would get written to RAFT logs, from where they’d get applied to RocksDB (state machine). As we merge our dirty posting lists, we can discard the older RAFT logs. In the case of server crash and restart, it would proactively go through all pending logs an…

[mrjn (Manish R Jain)]

Yeah, we don’t need cache because we need to read the logs rarely. I’ll summarize the discussion we had now.

Predicate Moves using RAFT

Predicate moves Every predicate would be part of a 3-node cluster. When mutation arrives, it would be directed to the master of the cluster. The master would append the mutation to it’s write-ahead log (and synced to disk in parallel with the network calls coming next). It would then send this mutation off to it’s followers, and once they confirm, the mutation would be applied to Posting List. Periodically, we need to delete older logs, to keep their size in check. To do this, we’ll run this al…

[pawan (Pawan Rawal)]

From what I understand about Raft.

Everytime, a mutation comes in, it is written to the in-memory storage similar to this https://github.com/coreos/etcd/blob/master/contrib/raftexample/raft.go#L293. It is also written to the WAL. MemoryStorage actually implements the storage interface using an in-memory array.

Whenever a server boots up, logs from the WAL are replayed and data written to the memory storage. I think we need to implement the Storage interface(https://godoc.org/github.com/coreos/etcd/raft#Storage) for our WAL logs.

[pawan (Pawan Rawal)]

My present understanding regarding this is that

1. We have multiple Raft groups(each containing 3 nodes and storing data for a predicate). Note that a node doesn’t mean an instance here. An instance could act and typically would act as nodes for multiple predicates.

2. Now each node has a memory storage and a WAL(this means a log per predicate). If I am thinking correct here then we could just use wal package - github.com/coreos/etcd/wal - Go Packages. Whenever a mutation comes, it is redirected to master of the group of nodes which form the Raft group for the predicate. Master writes it to its logs, sends it to other nodes(they write it to their logs), gets consensus and then the mutation is applied to all nodes.

3. From what I understand RAFT library can be configured to use the memory storage(raft package - github.com/coreos/etcd/raft - Go Packages) for transmitting the entries to other instances. WAL is used to load data into memory in case a node restarts.

4. Whenever a predicate has to be transferred to a new instance, we just make it part of the RAFT group for the predicate and data can then be streamed using Predicate RPC + the storage.

References

RAFT based Changes to commit logs

For Dgraph, RocksDB is our state machine. When we commit our dirty posting lists to RocksDB, and once we have confirmed sync to disk, we can discard the logs. This mechanism is great because we can avoid storing our commit logs. Each mutation would get written to RAFT logs, from where they’d get applied to RocksDB (state machine). As we merge our dirty posting lists, we can discard the older RAFT logs. In the case of server crash and restart, it would proactively go through all pending logs an…