First, it is important to note that Dgraph shards data by predicate, not by group. This means that each predicate is assigned to a specific shard, and the data for that predicate is stored on the alpha nodes that serve that shard. Groups are used to organize the alpha nodes within the Dgraph cluster(following the RAFT logic), in a way that allows each group to serve a specific set of shards.
In Dgraph, a group is a set of alpha nodes that are assigned to serve a specific set of shards. Each shard contains a number of replicas, which is determined by the Replicaset setting in the configuration. For example, if the Replicaset is set to 3, each shard will have three replicas, and the data for that shard will be replicated across three alpha nodes.
The number of groups in a Dgraph cluster is determined by the Replicaset setting in the zero nodes. For example, if the number of alpha nodes is 18 and the Replicaset is set to 3, the cluster will create six groups (18/3), with each group containing three alpha nodes.
One important aspect of Dgraph cluster management is shard rebalancing. This process is used to ensure that each group has a balanced workload and is not overloaded with too many shards. This is important because an overloaded group can lead to performance issues and potential failures within the cluster. Additionally, the process of shard rebalancing in Dgraph is not primarily concerned with ensuring an even distribution of data across the cluster.
Another important consideration when managing Dgraph clusters is the use of the --replicas flag to specify the number of replicas for each group. This flag determines the size(the amount of Alpha instances needed to form a group) of the Raft group (or quorum) that is responsible for ensuring the consistency of data within the group. It is generally recommended to use an odd number of replicas to avoid potential issues with achieving consensus within the group.
To view the number of groups in your cluster, you can check the state endpoint on any alpha node, or you can view the “Cluster” panel in the Ratel dashboard. This will show you the current state of the cluster, including the number of groups and the number of nodes in each group. You can use this information to confirm that your cluster is properly configured and operating as expected.
PS. In Dgraph, data is sharded based on the data itself, rather than on entities as is done in many other distributed databases. This means that each Alpha in a Dgraph cluster contains a subset of the data, and all the relationships and attributes associated with that data. Including the index tables.