For Dgraph, RocksDB is our state machine. When we commit our dirty posting lists to RocksDB, and once we have confirmed sync to disk, we can discard the logs.
This mechanism is great because we can avoid storing our commit logs. Each mutation would get written to RAFT logs, from where they’d get applied to RocksDB (state machine). As we merge our dirty posting lists, we can discard the older RAFT logs.
In the case of server crash and restart, it would proactively go through all pending logs and apply them to the posting lists (state machine), if they haven’t already been applied. So, when we bring a posting list in memory during query execution, we wouldn’t have to go through our commit logs to ensure it has the latest entries. This would make our query execution faster.
In fact, we can probably just use memory-based RAFT logs. Because, even if a server crashes, the other servers in the cluster can bring it back up to the latest update log. Though, it might be safer to store them on disk in case multiple servers crash at the same time. So, we don’t lose history.
@minions: Read through this and see if you have any suggestions.