Create PVC disks for dgraph servers and copy data from bulk process to each one PVC disk for dgraph-servers (about 111 GB)
Start dgraph servers
After that, first server always start the snapshot sending process:
....
Sending SNAPSHOT: Time elapsed: 05m24s
....
After while, the first server is restarted becuase all memory is consumed and cluster is in unbalanced state and dgraph server does not reply to queries (or some times return invalid results).
How I can setup replicated cluster with bulk data?
How much memory do you have available? Technically you need to make sure that all Dgraph Servers are communicating directly without limitations with Dgraph Zero.
I recommend that you use always the same Dgraph Zero from the beginning of the process.
Please share your Zero and Server instances settings/commands. Are they exactly the same as YAML?
You can bulkload for one instance and connect the rest of your servers without bulkload them. Eventually Dgraph will sync Through Dgraph Zero. it Take time but will.
each server has 16 GB of RAM na 250 GB of HDD and 4 CPU cores.
I do not know that I should not start zero servers in HA mode, but only in one instance for initial loading. I will try it.
What is surprised for me is, that server 0 in stateful set was during night restarted two times and servers 1 and 2 one time and now cluster is responding for every query (I hope, that all data is correctly replicated :)).
Here are logs from serevers (I don’t know if it could help, but I try to get as much infromation I have).
My k8s configuration is:
# There are 4 public services exposed, users can use:
# dgraph-zero-public - To load data using Live & Bulk Loaders
# dgraph-server-public - To connect clients and for HTTP APIs
# dgraph-ratel-public - For Dgraph UI
# dgraph-server-x-http-public - Use for debugging & profiling
apiVersion: v1
kind: Service
metadata:
name: dgraph-zero-public
labels:
app: dgraph-zero
spec:
type: LoadBalancer
ports:
- port: 5080
targetPort: 5080
name: zero-grpc
- port: 6080
targetPort: 6080
name: zero-http
selector:
app: dgraph-zero
---
apiVersion: v1
kind: Service
metadata:
name: dgraph-server-public
labels:
app: dgraph-server
spec:
type: LoadBalancer
ports:
- port: 8080
targetPort: 8080
name: server-http
- port: 9080
targetPort: 9080
name: server-grpc
selector:
app: dgraph-server
---
# This service is created in-order to debug & profile a specific server.
# You can create one for each server that you need to profile.
# For a more general HTTP APIs use the above service instead.
apiVersion: v1
kind: Service
metadata:
name: dgraph-ratel-public
labels:
app: dgraph-ratel
spec:
type: NodePort
ports:
- port: 8000
targetPort: 8000
name: ratel-http
selector:
app: dgraph-ratel
---
# This is a headless service which is neccessary for discovery for a dgraph-zero StatefulSet.
# https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#creating-a-statefulset
apiVersion: v1
kind: Service
metadata:
name: dgraph-zero
labels:
app: dgraph-zero
spec:
ports:
- port: 5080
targetPort: 5080
name: zero-grpc
clusterIP: None
selector:
app: dgraph-zero
---
# This is a headless service which is neccessary for discovery for a dgraph-server StatefulSet.
# https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#creating-a-statefulset
apiVersion: v1
kind: Service
metadata:
name: dgraph-server
labels:
app: dgraph-server
spec:
ports:
- port: 7080
targetPort: 7080
name: server-grpc-int
clusterIP: None
selector:
app: dgraph-server
---
# This StatefulSet runs 3 Dgraph Zero.
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: dgraph-zero
spec:
serviceName: "dgraph-zero"
replicas: 3
selector:
matchLabels:
app: dgraph-zero
template:
metadata:
labels:
app: dgraph-zero
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- dgraph-zero
topologyKey: kubernetes.io/hostname
containers:
- name: zero
image: dgraph/dgraph:latest
imagePullPolicy: IfNotPresent
ports:
- containerPort: 5080
name: zero-grpc
- containerPort: 6080
name: zero-http
volumeMounts:
- name: datadir
mountPath: /dgraph
command:
- bash
- "-c"
- |
set -ex
[[ `hostname` =~ -([0-9]+)$ ]] || exit 1
ordinal=${BASH_REMATCH[1]}
idx=$(($ordinal + 1))
if [[ $ordinal -eq 0 ]]; then
dgraph zero --my=$(hostname -f):5080 --idx $idx --replicas 3
else
dgraph zero --my=$(hostname -f):5080 --peer dgraph-zero-0.dgraph-zero.default.svc.cluster.local:5080 --idx $idx --replicas 3
fi
terminationGracePeriodSeconds: 60
volumes:
- name: datadir
persistentVolumeClaim:
claimName: datadir
updateStrategy:
type: RollingUpdate
volumeClaimTemplates:
- metadata:
name: datadir
annotations:
volume.alpha.kubernetes.io/storage-class: anything
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: 5Gi
---
# This StatefulSet runs 6 Dgraph Server forming two server groups, 3 servers in each group.
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: dgraph-server
spec:
serviceName: "dgraph-server"
replicas: 3
selector:
matchLabels:
app: dgraph-server
template:
metadata:
labels:
app: dgraph-server
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- dgraph-server
topologyKey: kubernetes.io/hostname
containers:
- name: server
image: dgraph/dgraph:latest
imagePullPolicy: IfNotPresent
ports:
- containerPort: 7080
name: server-grpc-int
- containerPort: 8080
name: server-http
- containerPort: 9080
name: server-grpc
volumeMounts:
- name: datadir
mountPath: /dgraph
command:
- bash
- "-c"
- |
set -ex
dgraph server --my=$(hostname -f):7080 --lru_mb 4096 --zero dgraph-zero-0.dgraph-zero.default.svc.cluster.local:5080
terminationGracePeriodSeconds: 60
volumes:
- name: datadir
persistentVolumeClaim:
claimName: datadir
updateStrategy:
type: RollingUpdate
volumeClaimTemplates:
- metadata:
name: datadir
annotations:
volume.alpha.kubernetes.io/storage-class: anything
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: 250Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: dgraph-ratel
labels:
app: dgraph-ratel
spec:
selector:
matchLabels:
app: dgraph-ratel
template:
metadata:
labels:
app: dgraph-ratel
spec:
containers:
- name: ratel
image: dgraph/dgraph:latest
ports:
- containerPort: 8000
command:
- dgraph-ratel