Hey @minions, @jchiu,
I’m trying to figure out of our shard move would keep both the shard and the index consistent.
- When we copy over the shard, we copy directly the RocksDB data. This does mean that some PLs might still be in memory, and not flushed out to RocksDB; when we do the copy. However, when the mutations flow through, they should be able to rectify the missing data, and bring it back up to the right state. So, I think shard move works.
There’re three ways to handle index.
- We copy over the indexing data as well. Note that index is generated via goroutines which might be written out later. So, the indexing data wouldn’t be consistent either. Now, when the mutations flow through, they would do mods to the index; but I’m not sure that they’d be idempotent. For e.g.
Data: SET [1, name, Tom hanks]
Now assume that this part of the data was copied over just fine during RocksDB copy, but the index wasn’t. So, the PL exists with “Tom hanks” as value.
Now we run AddMutationWithIndex with this fact, it would see that the value already exists, and not mutate it; which would also not update the index.
This then leads to an inconsistent index, which would never be fixed.
The second way is
- We ignore all the indexing data; and force everything via AddMutationWithIndex. That would then require us to not write directly to RocksDB, but pass everything as a mutation; which is going to be a lot more expensive than what we’re currently doing.
A third way is to have a
syncIndex function which can look over all the data, and sync up the index. We can then run this after the move, and also as a periodic thread to ensure no inconsistencies lie between data and index.