How does zero move the larger shard in the group to other groups?

tss · July 26, 2020, 10:25am

When the amount of data in a group is large, will zero move the larger shard in the group to other groups? How did it move? Does the copy on the copy also move?

Naman · July 26, 2020, 2:02pm

Yes, Zero moves the shards from larger size groups to smaller size groups (see --rebalanceInterval flag). Periodically after that interval, it checks if rebalancing would be useful. The rebalancing decision is based on heuristics (currently it just uses size of groups, maybe query traffic could be another parameter to try on later).

The logic for that decision can be seen here in the code. To summarize:

Sort the groups on the basis of size.
Decide srcGroup: Find the active group (from largest → smallest), and see if the size difference with the smallest group is big enough that it would be advantageous to move.
Decide tablet from srcGroup: Finds a tablet as big a possible such that on moving it dstGroup’s size is less than or equal to srcGroup.

You can see it here. Feel free for follow-ups.

github.com

dgraph-io/dgraph/blob/master/dgraph/cmd/zero/tablet.go#L38-L57


      
          /*
          Steps to move predicate p from g1 to g2.
          Design change:
          • If you’re not the leader, don’t talk to zero.
          • Let the leader send you updates via proposals.
          
          MOVE:
          • Dgraph zero would decide that G1 should not serve P, G2 should serve it.
          
          Phase I:
          • Zero would tell G1 to move P's data before timestamp T1 to G2 (Endpoint: Zero → G1)
          • G1 would do a call to G2, and start streaming.
          • Before G2 starts accepting, it should delete any current keys for P.
          • G1 should tell Zero whether it succeeded or failed. (Endpoint: G1 → Zero)
          • If it succeeds, then based on the time taken for the above run, zero decides to do another run of
            Phase I, or move to Phase II.
          
          Phase II:
          • Zero would propose that G1 is read-only for predicate P. This would propagate to the cluster.
          • Zero would tell G1 to move rest of the P's data (with T1<=ts<=T2) to G2 (Endpoint: Zero → G1)

Can you please elaborate on what you mean by this?

Topic		Replies	Views
More about Dgraph Zero - Deploy Documentation	0	451	August 28, 2020
Can we select the custom group for any predicate to control the sharding? Dgraph	1	426	May 22, 2020
Shard zero and alpha nodes in a federated flavour Dgraph kind:question	2	488	August 29, 2020
Cluster Setup - Deploy Documentation	1	630	June 24, 2023
Dgraph Zero not moving tablets by itself Dgraph	2	636	July 29, 2020

How does zero move the larger shard in the group to other groups?

Related topics