Thanks for the reply!
What really helps is a good storage and RAM. Try NVME tho.
We run the 3 alphas on one machine with 128GB of RAM and an SSD with 2.5TB of space using DigitalOcean. That machine has 24 cores available to it CPU wise. We are using Docker and Docker Compose right now to manage everything. We can’t host the machine ourselves but I’d assume the likes of DigitalOcean are probably using the best hardware for the job.
Now, while I understand we would better off with each alpha on a separate machine, only one alpha had a spike in CPU and the others were hovering around 10% so I don’t believe the machine itself was under any sort of crazy stress.
Which part of dgraph helps each type of situation
What I meant to ask is, are there ways to tune Dgraph to be more suited for writes vs reads. If so, which options (Badger options, no expanding edges, etc) affect which operation (read, write) positively or negatively?
Try 6 Alphas
Are you saying to try 6 alphas, with the replication factor set to 3 or simply to use 6 alphas and have every alpha be a replica? If you mean 6 alphas, 3 replicas, I have a question about setting that up. Currently, we set up our graph using the bulk loader and copy the p directory into each alpha and start the servers up. This works well and is fairly simple.
However, when I try to set the reduce_shards=2, map_shards=4 and copy the 0/p directory from the bulk loader into the first 3 alphas and the 1/p directory into the next three, nothing works. The graph seems to delete predicates and move things around and my queries return no data.
To quote the docs https://docs.dgraph.io/deploy/#bulk-loader
Reduce shards : Before running the bulk load, you need to decide how many Alpha groups will be running when the cluster starts. The number of Alpha groups will be the same number of reduce shards you set with the --reduce_shards flag. For example, if your cluster will run 3 Alpha with 3 replicas per group, then there is 1 group and --reduce_shards should be set to 1. If your cluster will run 6 Alphas with 3 replicas per group, then there are 2 groups and --reduce_shards should be set to 2. Because --reduce_shards was set to 2, there are two sets of p directories: one in ./out/0 directory and another in the ./out/1 directory. Once the output is created, they can be copied to all the servers that will run Dgraph Alphas. Each Dgraph Alpha must have its own copy of the group’s p directory output. Each replica of the first group should have its own copy of ./out/0/p , each replica of the second group should have its own copy of ./out/1/p , and so on.
That paragraph does not really explain much, at least for me. I understand that the groups will eventually be auto-balanced but I’ve had issues with that before since it takes awhile for all the data to copy over from one alpha to another.
don’t get it, are you talking about Bulk Loader or Live Loader? If you are using Live Loader and your server already has the Schema that you want. Just do not pass Schema again on the new load.
We use the bulk loader with a rather larger (1GB+) RDF file for initial setup. When we upgrade to new Dgraph versions that require an export and import (due to compatability changes like 1.0.10), we just use the bulk loader with an RDF we generate via Postgres instead of exporting the data out of Dgraph and back in.
All I meant is that it seems like we have a lot of data (more than the movie data set) and I was not sure if having that much data represented a problem for writes or if write performance is supposed to remain constant no matter the size of the graph.
UPDATE:
I get the following error with 6 alphas, 3 replicas.
Schema not defined for predicate: id.
However the state of my zero clearly shows it there.
{
counter: "1051",
groups: {
1: {
members: {
1: {
id: "1",
groupId: 1,
addr: "cluster0-node2:7082",
leader: true,
lastUpdate: "1541605474"
},
2: {
id: "2",
groupId: 1,
addr: "cluster0-node0:7080"
},
3: {
id: "3",
groupId: 1,
addr: "cluster0-node1:7081"
}
},
tablets: {
_predicate_: {
groupId: 1,
predicate: "_predicate_",
space: "31"
},
id: {
groupId: 1,
predicate: "id"
},
matchOrder: {
groupId: 1,
predicate: "matchOrder",
space: "270219578"
}
}
},
2: {
members: {
4: {
id: "4",
groupId: 2,
addr: "cluster0-node4:7084",
leader: true,
lastUpdate: "1541605478"
},
5: {
id: "5",
groupId: 2,
addr: "cluster0-node3:7083"
},
6: {
id: "6",
groupId: 2,
addr: "cluster0-node4:7085"
}
},
tablets: {
email: {
groupId: 2,
predicate: "email",
space: "249666916"
},
rawLocation: {
groupId: 2,
predicate: "rawLocation",
space: "134326287"
},
title: {
groupId: 2,
predicate: "title",
space: "298198008"
}
}
}
},
zeros: {
1: {
id: "1",
addr: "zero:5080",
leader: true
}
},
maxLeaseId: "10250000",
maxTxnTs: "10000",
maxRaftId: "6",
cid: "d227d288-0d05-4b3d-a773-4aa3d81fee63"
}