Liveloader has the ability to perform a load in a distributed way. This way we avoid having OOM due to high load on a single Alpha Node (Causing congestion). In this text I will demonstrate how you can use this feature via Kubernetes.
This same process works for Docker, Docker Swarm, Bare Metal and so on. Of course, in each one is slightly different.
The reference Yaml is https://github.com/dgraph-io/dgraph/blob/master/contrib/config/kubernetes/dgraph-ha.yaml
In this small tutorial I’m assuming that you already have a k8s context running. Be it GKE, Docker k8s or even Minikube.
First create the setup:
kubectl create -f dgraph-ha.yaml
After that you need to create services exposing each gRPC from Alphas. Because load in a distributed way is not possible through Load Balance (as it is). So you need to expose each Alpha temporarily.
Create a expose_dgraph.yaml
apiVersion: v1
kind: Service
metadata:
name: dgraph-zero-0-grpc-public
labels:
app: dgraph-zero
spec:
type: LoadBalancer
ports:
- port: 5080
targetPort: 5080
name: zero-grpc
selector:
statefulset.kubernetes.io/pod-name: dgraph-zero-0
---
apiVersion: v1
kind: Service
metadata:
name: dgraph-alpha-0-grpc-public
labels:
app: dgraph-alpha
spec:
type: LoadBalancer
ports:
- port: 9080
targetPort: 9080
name: alpha-grpc
selector:
statefulset.kubernetes.io/pod-name: dgraph-alpha-0
---
apiVersion: v1
kind: Service
metadata:
name: dgraph-alpha-1-grpc-public
labels:
app: dgraph-alpha
spec:
type: LoadBalancer
ports:
- port: 9080
targetPort: 9080
name: alpha-grpc
selector:
statefulset.kubernetes.io/pod-name: dgraph-alpha-1
---
apiVersion: v1
kind: Service
metadata:
name: dgraph-alpha-2-grpc-public
labels:
app: dgraph-alpha
spec:
type: LoadBalancer
ports:
- port: 9080
targetPort: 9080
name: alpha-grpc
selector:
statefulset.kubernetes.io/pod-name: dgraph-alpha-2
---
Run:
kubectl create -f expose_dgraph.yaml
This will create a public gRPC endpoint and when your cloud service finishes creating the Endpoints/IPs you can use it to start your load.
./dgraph live -f your.rdf -s your.schema -z 38.239.155.115:5080
-a "33.63.128.148:9080,35.138.12.15:9080,35.225.60.18:9080"
Each URI/IP must be comma separated and can not contain spaces.
After load you need to remove the services that exposes Dgraph:
kubectl delete -f expose_dgraph.yaml
BTW, It is recommended that you modify dgraph-ha.yaml according to your needs and not expose the Dgraph directly to the public. dgraph-ha.yaml exposes by default the Dgraph via load balance. Keep that in mind.
PS. After the publication of this topic. We gonna work in a deep and technical blog post for this.
Cheers.
) blog post sounds exactly like the issue we’re trying to solve.