Can dgraph do the following use case?

Users
,
1

Hi,

Can dgraph handle the following use case:

  • 1000 clients
  • Each writing 1 mb every 0.25seconds (4mb/s)

Hence 4000 mutations each being 1mb being sent a second. And if so, what cluster spec? Currently I am requiring a 32 core 6-ha setup on gcp just to service 100 clients (10% error rate and two thirds of the compute time is spent retrying on dgraph read timeouts).

I appreciate I am being very light on the details but looking more for a: [your crazy, yes but will require serious optimization work, yes but you need a massive resource setup, yes this is easy and any standard deployment supports this].

2

This is 4 gigabit per second? mean, 500 megabyte/s?

That depends on context a lot. A cluster without chard, replicating by 6. Will choke. You will not be able to grow vertically - Dgraph is made to grow horizontally. You need more shards to handle the load and also do a load balance.

If you have 4 groups each with a good SSD. You would use 125 MB/s per shard. Still, I think a lot. Maybe 8 groups would be better. That would give 62MB/s usage. Completely theoretical and arbitrary numbers. It will depend on factors of the planning schema itself.

The more groups you have, the more separate physical machines in each group. Best. Do not try to grow vertically.

You could even have all shards on the same machine, but you would need to give each instance its own SSD so there is no IO bottleneck.

3

Thank you @MichelDiz, very helpful and makes sense and gives good next steps to investigate.

1 Like
4

@MichelDiz for my interest could you say a few more words on vertical vs horizontal scaling? In general, redundancy aside, I have taken the approach of going for larger alphas and faster disks first, before going for multiple alphas when the machines start to get really large (e.g. after roughly 48 cores). I do this to simplify deployments and rule out things such as network bottlenecks etc. Could you say if/why you think horizontal scaling is generally a better approach?

5

Dgraph was created with the intention of growing horizontally. Hence the name “Distributed Graph”. All logic in the Dgraph code is to work with a distributed system. So the focus is horizontal. And not vertical.

I’ve already noticed that Dgraph gets very idle on super machines. Unless you have many instances on the same machine. It won’t consume all the machine’s resources (CPU, IO). So it’s no use trying for more memory, more processing. It will consume memory and so on, but it won’t do any good. He will be big, fat and slow. But if you focus on distributing it, you will earn much more.

This shouldn’t be happening. The instances must be close to each other. If you have a highly distributed cluster, the network cost is negligible. And when you focus on queries with multiblocks. You will see a gain.

Vertical is just exaggerated. While a vertical cluster is a bodybuilder, a highly distributed cluster is a swarm of bees killing it.

The distribution level of the Dgraph is at the predicate level. Queries are distributed picking predicates here and there. The more groups and more distinct machines, the better.

I find it strange that people read the Dgraph paper and opt for a simple cluster that doesn’t use and abuse the horizontal concept that Dgraph was made for.

I would recommend building a highly distributed cluster and doing periodic backups.

6

Thanks for the input, and totally agree with the sentiment. I think the thing I need to bottom out is what is the best point to shift from the bodybuilder to bees :grinning_face_with_smiling_eyes: . It feels like the documents could suggest/recommend a maximum instance size before you should distribute over vertical scaling. E.g. going from 16G to 32G should be vertical, but going to 64G should be 2x32G alphas etc

1 Like
7

I think the recommendation in the doc is the minimum for a simple cluster with a few million predicates. If you try to make a cluster with less than 32GB in prod, you will suffer a lot. I remember testing with 21 million RDF needed on average 29GB of RAM. But in general other processes were idle with time.

This question changes if you have more Alphas. Because the load pressure will be much lower. So you won’t need for 32GB per machine. I think 16GB is enough - maybe less depending on your day to day context. Obviously if you do a good load balance of mutations you wont have RAM problems.

Imagine the following situation. You are doing an intensive load. And it has 10 indexed predicates. Dgraph is indexing them at the same time as the demand is increasing. And it consumes more RAM and CPU. For a single, even beefy machine this is going to be a lot of work. Now if you divide the ten predicates among 10 machines. You will have a much smoother load.

PS. vertical vs horizontal scaling is like “Centralized vs Decentralized”

1 Like
8

At very high rates, everything needs to be tuned and measured more carefully. Those are very high rates - 0.5GB/sec will add up to about 40TB/day if it is sustained (and perhaps petabytes in the long term depending on update vs insert). This is indeed where horizontal scalability is essential.

Running at 10% of future capacity as you are now is a great place to start because it is already starting to show what the limiting factors will be that determine performance at scale for your particular data model, data size, and queries.

For such an intensive system, I suggest doing quite a lot of perf testing and optimization - including analyzing queries and data model, and certainly understanding the retries and errors you already see. It could be your cluster is under-provisioned, but it could also be that you have some inefficiencies in code or queries.

If you’re a paying customer I suggest a support ticket to investigate the existing errors or retries to start out. If there’s anything obvious we can get insight by checking logs and metrics, and if you feel you need a more substantial perf analysis we can do that via consulting services.

9

Hi, Back to looking at this again. If I want to partition the data into 9 partitions/shards, with each shard replicated three times for safety, would I do something like below (I get the feeling the below will actually create 3 shards/partitions/groups each replicated 3 times)? Also, Can I have something like 3 zeros and 9 alphas (or should you always have number of alphas == number of zeros)?

image:
tag: “v22.0.0”
alpha:
replicaCount: 9
zero:
replicaCount: 3
shardReplicaCount: 3

Second question: If i want to verify the number of groups in the cluster?

10

First, it is important to note that Dgraph shards data by predicate, not by group. This means that each predicate is assigned to a specific shard, and the data for that predicate is stored on the alpha nodes that serve that shard. Groups are used to organize the alpha nodes within the Dgraph cluster(following the RAFT logic), in a way that allows each group to serve a specific set of shards.

In Dgraph, a group is a set of alpha nodes that are assigned to serve a specific set of shards. Each shard contains a number of replicas, which is determined by the Replicaset setting in the configuration. For example, if the Replicaset is set to 3, each shard will have three replicas, and the data for that shard will be replicated across three alpha nodes.

The number of groups in a Dgraph cluster is determined by the Replicaset setting in the zero nodes. For example, if the number of alpha nodes is 18 and the Replicaset is set to 3, the cluster will create six groups (18/3), with each group containing three alpha nodes.

One important aspect of Dgraph cluster management is shard rebalancing. This process is used to ensure that each group has a balanced workload and is not overloaded with too many shards. This is important because an overloaded group can lead to performance issues and potential failures within the cluster. Additionally, the process of shard rebalancing in Dgraph is not primarily concerned with ensuring an even distribution of data across the cluster.

Another important consideration when managing Dgraph clusters is the use of the --replicas flag to specify the number of replicas for each group. This flag determines the size(the amount of Alpha instances needed to form a group) of the Raft group (or quorum) that is responsible for ensuring the consistency of data within the group. It is generally recommended to use an odd number of replicas to avoid potential issues with achieving consensus within the group.

To view the number of groups in your cluster, you can check the state endpoint on any alpha node, or you can view the “Cluster” panel in the Ratel dashboard. This will show you the current state of the cluster, including the number of groups and the number of nodes in each group. You can use this information to confirm that your cluster is properly configured and operating as expected.

PS. In Dgraph, data is sharded based on the data itself, rather than on entities as is done in many other distributed databases. This means that each Alpha in a Dgraph cluster contains a subset of the data, and all the relationships and attributes associated with that data. Including the index tables.