Cannot load bulk data into replicated cluster

selmeci · May 30, 2018, 8:19pm

Hi,

I cannot load data into replicated cluster in kubernetes env.

My configuration is:
3 x m5.xlarge on AWS
1 x m5.xlarge for migration and bulk task
Cluster is setup according to dgraph-ha.yaml

Steps:

I start only zero servers.
start bulk process on migration server

docker run -d --name=bulk -v /home/ubuntu/migration/:/dgraph dgraph/dgraph:latest dgraph bulk -r rdfs/ -s schema.dgraph --zero=ZERO-ADDRESS:5080 --reduce_shards=1

Create PVC disks for dgraph servers and copy data from bulk process to each one PVC disk for dgraph-servers (about 111 GB)
Start dgraph servers

After that, first server always start the snapshot sending process:

....
Sending SNAPSHOT: Time elapsed: 05m24s
....

After while, the first server is restarted becuase all memory is consumed and cluster is in unbalanced state and dgraph server does not reply to queries (or some times return invalid results).

How I can setup replicated cluster with bulk data?

selmeci · May 30, 2018, 8:27pm

Logs after restart from server-0

2018/05/30 20:25:04 gRPC server started.  Listening on port 9080
2018/05/30 20:25:04 HTTP server started.  Listening on port 8080
2018/05/30 20:25:04 worker.go:89: Worker listening at address: [::]:7080
2018/05/30 20:25:04 groups.go:78: Current Raft Id: 1
2018/05/30 20:25:04 pool.go:108: == CONNECT ==> Setting dgraph-zero-0.dgraph-zero.default.svc.cluster.local:5080
2018/05/30 20:25:04 groups.go:105: Connected to group zero. Assigned group: 0
2018/05/30 20:25:04 pool.go:108: == CONNECT ==> Setting dgraph-server-2.dgraph-server.default.svc.cluster.local:7080
2018/05/30 20:25:04 pool.go:108: == CONNECT ==> Setting dgraph-server-1.dgraph-server.default.svc.cluster.local:7080
2018/05/30 20:25:04 pool.go:108: == CONNECT ==> Setting dgraph-zero-1.dgraph-zero.default.svc.cluster.local:5080
2018/05/30 20:25:04 pool.go:108: == CONNECT ==> Setting dgraph-zero-2.dgraph-zero.default.svc.cluster.local:5080
2018/05/30 20:25:04 draft.go:170: Node ID: 1 with GroupID: 1
2018/05/30 20:25:04 node.go:228: Found hardstate: {Term:2 Vote:1 Commit:4 XXX_unrecognized:[]}
2018/05/30 20:25:04 node.go:240: Group 1 found 4 entries
2018/05/30 20:25:04 draft.go:926: Restarting node for group: 1
2018/05/30 20:25:04 raft.go:567: INFO: 1 became follower at term 2
2018/05/30 20:25:04 raft.go:315: INFO: newRaft 1 [peers: [], term: 2, commit: 4, applied: 0, lastindex: 4, lastterm: 2]
2018/05/30 20:25:04 node.go:118: Setting conf state to nodes:1 
2018/05/30 20:25:04 mutation.go:181: Done schema update predicate:"_predicate_" value_type:STRING list:true 
2018/05/30 20:25:04 mutation.go:181: Done schema update predicate:"_share_hash_" value_type:STRING directive:INDEX tokenizer:"exact" 
2018/05/30 20:25:07 raft.go:749: INFO: 1 is starting a new election at term 2
2018/05/30 20:25:07 raft.go:580: INFO: 1 became candidate at term 3
2018/05/30 20:25:07 raft.go:664: INFO: 1 received MsgVoteResp from 1 at term 3
2018/05/30 20:25:07 raft.go:621: INFO: 1 became leader at term 3
2018/05/30 20:25:07 node.go:301: INFO: raft.node: 1 elected leader 1 at term 3
2018/05/30 20:25:15 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04s, bytes sent: 0 B, bytes/sec 0
2018/05/30 20:25:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05s, bytes sent: 0 B, bytes/sec 0
2018/05/30 20:25:20 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10s, bytes sent: 0 B, bytes/sec 0
2018/05/30 20:25:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09s, bytes sent: 0 B, bytes/sec 0
2018/05/30 20:25:25 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15s, bytes sent: 0 B, bytes/sec 0
2018/05/30 20:25:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15s, bytes sent: 0 B, bytes/sec 0
2018/05/30 20:25:30 predicate.go:250: Sending SNAPSHOT: Time elapsed: 20s, bytes sent: 0 B, bytes/sec 0
2018/05/30 20:25:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 19s, bytes sent: 0 B, bytes/sec 0

MichelDiz · May 30, 2018, 11:49pm

Hi,

How much memory do you have available? Technically you need to make sure that all Dgraph Servers are communicating directly without limitations with Dgraph Zero.

I recommend that you use always the same Dgraph Zero from the beginning of the process.

Please share your Zero and Server instances settings/commands. Are they exactly the same as YAML?

You can bulkload for one instance and connect the rest of your servers without bulkload them. Eventually Dgraph will sync Through Dgraph Zero. it Take time but will.

selmeci · May 31, 2018, 7:46am

Hi Michel,

each server has 16 GB of RAM na 250 GB of HDD and 4 CPU cores.
I do not know that I should not start zero servers in HA mode, but only in one instance for initial loading. I will try it.

What is surprised for me is, that server 0 in stateful set was during night restarted two times and servers 1 and 2 one time and now cluster is responding for every query (I hope, that all data is correctly replicated :)).

Here are logs from serevers (I don’t know if it could help, but I try to get as much infromation I have).

My k8s configuration is:

# There are 4 public services exposed, users can use:
#       dgraph-zero-public - To load data using Live & Bulk Loaders
#       dgraph-server-public - To connect clients and for HTTP APIs
#       dgraph-ratel-public - For Dgraph UI
#       dgraph-server-x-http-public - Use for debugging & profiling
apiVersion: v1
kind: Service
metadata:
  name: dgraph-zero-public
  labels:
    app: dgraph-zero
spec:
  type: LoadBalancer
  ports:
  - port: 5080
    targetPort: 5080
    name: zero-grpc
  - port: 6080
    targetPort: 6080
    name: zero-http
  selector:
    app: dgraph-zero
---
apiVersion: v1
kind: Service
metadata:
  name: dgraph-server-public
  labels:
    app: dgraph-server
spec:
  type: LoadBalancer
  ports:
  - port: 8080
    targetPort: 8080
    name: server-http
  - port: 9080
    targetPort: 9080
    name: server-grpc
  selector:
    app: dgraph-server
---
# This service is created in-order to debug & profile a specific server.
# You can create one for each server that you need to profile.
# For a more general HTTP APIs use the above service instead.
apiVersion: v1
kind: Service
metadata:
  name: dgraph-ratel-public
  labels:
    app: dgraph-ratel
spec:
  type: NodePort
  ports:
  - port: 8000
    targetPort: 8000
    name: ratel-http
  selector:
    app: dgraph-ratel
---
# This is a headless service which is neccessary for discovery for a dgraph-zero StatefulSet.
# https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#creating-a-statefulset
apiVersion: v1
kind: Service
metadata:
  name: dgraph-zero
  labels:
    app: dgraph-zero
spec:
  ports:
  - port: 5080
    targetPort: 5080
    name: zero-grpc
  clusterIP: None
  selector:
    app: dgraph-zero
---
# This is a headless service which is neccessary for discovery for a dgraph-server StatefulSet.
# https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#creating-a-statefulset
apiVersion: v1
kind: Service
metadata:
  name: dgraph-server
  labels:
    app: dgraph-server
spec:
  ports:
  - port: 7080
    targetPort: 7080
    name: server-grpc-int
  clusterIP: None
  selector:
    app: dgraph-server
---
# This StatefulSet runs 3 Dgraph Zero.
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: dgraph-zero
spec:
  serviceName: "dgraph-zero"
  replicas: 3
  selector:
    matchLabels:
      app: dgraph-zero
  template:
    metadata:
      labels:
        app: dgraph-zero
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - dgraph-zero
              topologyKey: kubernetes.io/hostname
      containers:
      - name: zero
        image: dgraph/dgraph:latest
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 5080
          name: zero-grpc
        - containerPort: 6080
          name: zero-http
        volumeMounts:
        - name: datadir
          mountPath: /dgraph
        command:
          - bash
          - "-c"
          - |
            set -ex
            [[ `hostname` =~ -([0-9]+)$ ]] || exit 1
            ordinal=${BASH_REMATCH[1]}
            idx=$(($ordinal + 1))
            if [[ $ordinal -eq 0 ]]; then
              dgraph zero --my=$(hostname -f):5080 --idx $idx --replicas 3
            else
              dgraph zero --my=$(hostname -f):5080 --peer dgraph-zero-0.dgraph-zero.default.svc.cluster.local:5080 --idx $idx --replicas 3
            fi
      terminationGracePeriodSeconds: 60
      volumes:
      - name: datadir
        persistentVolumeClaim:
          claimName: datadir
  updateStrategy:
    type: RollingUpdate
  volumeClaimTemplates:
  - metadata:
      name: datadir
      annotations:
        volume.alpha.kubernetes.io/storage-class: anything
    spec:
      accessModes:
        - "ReadWriteOnce"
      resources:
        requests:
          storage: 5Gi
---
# This StatefulSet runs 6 Dgraph Server forming two server groups, 3 servers in each group.
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: dgraph-server
spec:
  serviceName: "dgraph-server"
  replicas: 3
  selector:
    matchLabels:
      app: dgraph-server
  template:
    metadata:
      labels:
        app: dgraph-server
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - dgraph-server
              topologyKey: kubernetes.io/hostname
      containers:
      - name: server
        image: dgraph/dgraph:latest
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 7080
          name: server-grpc-int
        - containerPort: 8080
          name: server-http
        - containerPort: 9080
          name: server-grpc
        volumeMounts:
        - name: datadir
          mountPath: /dgraph
        command:
          - bash
          - "-c"
          - |
            set -ex
            dgraph server --my=$(hostname -f):7080 --lru_mb 4096 --zero dgraph-zero-0.dgraph-zero.default.svc.cluster.local:5080
      terminationGracePeriodSeconds: 60
      volumes:
      - name: datadir
        persistentVolumeClaim:
          claimName: datadir
  updateStrategy:
    type: RollingUpdate
  volumeClaimTemplates:
  - metadata:
      name: datadir
      annotations:
        volume.alpha.kubernetes.io/storage-class: anything
    spec:
      accessModes:
        - "ReadWriteOnce"
      resources:
        requests:
          storage: 250Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dgraph-ratel
  labels:
    app: dgraph-ratel
spec:
  selector:
    matchLabels:
      app: dgraph-ratel
  template:
    metadata:
      labels:
        app: dgraph-ratel
    spec:
      containers:
      - name: ratel
        image: dgraph/dgraph:latest
        ports:
        - containerPort: 8000
        command:
          - dgraph-ratel