Master-slave replication suggestions

Issues Badger
,
1

Moved from GitHub badger/1403

Posted by ammmir:

This is more of an application question than anything else, so please feel free to direct me elsewhere. Many thanks for this wonderful project!

I run a small service, KVdb, that I’d like to improve the read latency from various points around the Internet. Initially, I’m leaning into a single master + multiple slave setup as a stepping stone to a more proper distributed system :slight_smile: The Badger API docs lead me to the Stream framework, which is used by the backup and restore functionality, so perhaps something like this would work?

Assumptions/tradeoffs:

  • Flat topology of 5~10 slaves
  • Master-slave network latencies of 50~200ms
  • Reads should be fast, writes can be an order of magnitude slower
  • All data will be blindly replicated for now

On the master, for each slave that connects:

  1. Create a new Stream
  2. Select keys since the slave’s last version timestamp
  3. Send keys over the network to the slave

On the slave:

  1. Connect to master
  2. Request key stream since last version timestamp
  3. Use KVLoader to ingest keys into our local database
  4. Repeat

On the slave:

  • Redirect write activity to master. (I don’t want to go down the Raft rabbit hole at this point.)
  • Refuse reads/softfail if slave is too far behind master (e.g., time delta) depending on desired consistency

Am I oversimplifying, plain wrong, or is this at least in the right direction?

2

jarifibrahim commented :

Your approach could work but distributed systems are very interesting and in your architecture, a single failure in master will stop the world on your platform.

You should use GitHub - BBVA/raft-badger: Raft backend implementation using BadgerDB and then take periodic snapshots. I think this would be the simplest way of running badger in distributed manner.

If you want to build something very simple, go with the approach you’ve mentioned. If you want something that takes care of failover and faults, use GitHub - BBVA/raft-badger: Raft backend implementation using BadgerDB (I have not looked at the code. I found it on the internet).