Load Data using Dgraph w/ Kubernetes

Hi Everyone,
I am new to DGraph and was trying to do the bulk loader in kubernetes. I found the yml file from the github https://raw.githubusercontent.com/dgraph-io/dgraph/master/contrib/config/kubernetes/dgraph-ha/dgraph-ha.yaml. In this yml file in the comments sections it is mentioned to do the bulk load we can create an init container and copy the data from our local repository into pod’s repository dgraph repository. So I have downloaded the 1million.rdf.gz file form the github and tried loading it into the dgraph but I am unable to do so. I had to modify the Statefulset service to deployment. My alpha deployment looks likes this:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    io.kompose.service: alpha
  name: alpha
spec:
  replicas: 1
  selector:
    matchLabels:
      io.kompose.service: alpha
  template:
    metadata:
      labels:
        io.kompose.service: alpha
    spec:
      initContainers:
        - name: init-alpha
          image: dgraph/dgraph:latest
          command:
            - bash
            - "-c"
            - |
              trap "exit" SIGINT SIGTERM
              echo "Write to /dgraph/doneinit when ready."
              until [ -f /dgraph/doneinit ]; do sleep 2; done
          volumeMounts:
            - name: alpha-claim0
              mountPath: /dgraph
      containers:
      - args:
        - dgraph
        - alpha
        - --my=alpha:7080
        - --lru_mb=2048
        - --zero=zero:5080
        image: dgraph/dgraph:latest
        name: alpha
        ports:
        - containerPort: 8080
        - containerPort: 9080
        volumeMounts:
        - mountPath: /dgraph
          name: alpha-claim0
      volumes:
      - name: alpha-claim0
        persistentVolumeClaim:
          claimName: alpha-claim0
status: {}

After I deploy using the above mentioned file I use the following commands to copy my data file. I have made a p directory where i have included my 100million.rdf.gz file.

kubectl cp /root/dgraph/docker-compose/p myalphapodname:/dgraph/ -c init-alpha

kubectl exec myalphapodname -c init-alpha touch /dgraph/doneinit

But I am unable to see my data in the ratel UI. Am I missing something? If this is a silly question please forgive me. I am trying kubernetes and dgraph first time in my life and I need to deliver it asap. I did all the research before posting here. Please if anyone could help me with this that would really be great.

Thank you

Hey @saugat

No question is too silly. We all learn thru asking questions. Welcome to the community by the way.

Quick one: is your Ratel connected to the correct IP/port? Have you done the appropriate whitelisting ?

Also, @joaquin, I had a quick look thru of things, and couldn’t spot any mistakes. Ideas?

@chewxy
Thank you for saying there is no silly questions. So I had literally put down the 100million.rdf.gz file in my p directory on further research I got to know i need to run the some bulkloader commands

dgraph bulk -r goldendata.rdf -s goldendata.schema --http localhost:8090 --zero localhost:5080

and this will create the out/0/p and need to copy this p folder into alpha … If this is right process where do I need to run the above command in my zero ?

Um, I think you need the live loader, not the bulk loader. - Bulk loader is for before you have the cluster up and running. But once it’s up and running you should use the live loader

Yeah, Liveloader should be the chosen one for this task. But you can also use Bulkloader, in that case, you have to start the Alphas after the bulk. You can’t start the bulkload with Alphas running. Just the zero, also, you have to preserve the Zero instance (never delete the Zero volume).

I believe that if you are running inside a pod, you should use zero:5080 or something similar coming from the SVC.

One more thing, if you gonna use the Liveloader, you have to check your provider. If it is AWS, GCP or similar you have to expose the Alpha and the Zero gRPCs ports. Unless you do the liveloader inside a pod.

Hi @saugat

I wouldn’t recommend using a Deployment controller, as the nodes are not sticky and will use a randomly named pod. StatefulSets are useful for stateful apps with zero and alpha.

A quick way to get started w/ K8S would be to use helm chart and then use live loader:

RELEASE="my-release"
helm repo add dgraph https://charts.dgraph.io
helm install $RELEASE dgraph/dgraph

After all the pods are up, you can use port forward to make them available on localhost.

RELEASE="my-release"
export RATEL_POD=$(kubectl get pods \
  --selector "component=ratel,release=$RELEASE" \
  --output jsonpath="{.items[0].metadata.name}"
)
export ALPHA_POD=$(kubectl get pods \
  --selector "statefulset.kubernetes.io/pod-name=alpha-0,release=$RELEASE" \
  --output jsonpath="{.items[0].metadata.name}"
) 
export ZERO_POD=$(kubectl get pods \
  --selector "statefulset.kubernetes.io/pod-name=zero-0,release=$RELEASE" 
  --output jsonpath="{.items[0].metadata.name}"
)

## For Ratel (HTTP)
kubectl port-forward $ALPHA_POD 8080:8080 &
kubectl port-forward $RATEL_POD 8000:8000 &

## For LiveLoader (GRPC)
kubectl port-forward $ALPHA_POD 9080:9080 &
kubectl port-forward $ZERO_POD 5080:5080 &

With these available at localhost, you can use live loader:

dgraph live \
 --files 1million.rdf.gz \
 --schema 1million.schema \
 --alpha localhost:9080 \
 --zero localhost:5080

And also you can visit http://localhost:8000 for Ratel. When configuring which alpha to use, select http://localhost:8080.

Hope that helps.

Joaquin

Hey Joaquin,

I have Dgraph stateful cluster run on AWS EKS, and I’ve successfully exported data and schema to a S3 bucket.

Also, I have a network load balancer listening on 9080 and 5080 and I’m able to use jump server to forward the connection to localhost:9080 and localhost: 5080, the localhost is my laptop.

Question is: Where should I run dgraph live command? Since I don’t have any Dgraph application running on my laptop.

I did the following things:

  1. shell into an Alpha pod
kubectl exec --stdin --tty dgraph-release-dgraph-alpha-0 --namespace myrelease  -- /bin/bash
  1. Ran dgraph version which works.

  2. Ran the import command,

dgraph live --files s3://s3.us-east-1.amazonaws.com/myrelease-dgraph-data-backup/dgraph.r3887409.u0318.2131

error says:

Running transaction with dgraph endpoint: 127.0.0.1:9080
2024/03/22 15:46:06 context deadline exceeded
Unable to connect to zero, Is it running at 127.0.0.1:5080?

I guess it must be that Zero service is not running in the same pod as the current Alpha pod so 127.0.0.1:5080 is not recognizable.

  1. Then I specified the zero service in the command.
dgraph live \
--zero http://dgraph-release-dgraph-zero.myrelease.svc:5080 \
--files s3://s3.us-east-1.amazonaws.com/myrelease-dgraph-data-backup/dgraph.r3887409.u0318.2131

Also

dgraph live \
--zero http://dgraph-release-dgraph-zero-headless.myrelease.svc:5080 \
--files s3://s3.us-east-1.amazonaws.com/myrelease-dgraph-data-backup/dgraph.r3887409.u0318.2131

I got the same error says that unable to connect to zero. I am pretty sure that Zero pod and service is running well.

Running transaction with dgraph endpoint: 127.0.0.1:9080
2024/03/22 15:57:01 context deadline exceeded
Unable to connect to zero, Is it running at http://dgraph-release-dgraph-zero.myrelease.svc:5080?

Running transaction with dgraph endpoint: 127.0.0.1:9080
2024/03/22 15:59:36 context deadline exceeded
Unable to connect to zero, Is it running at http://dgraph-release-dgraph-zero-headless.myrelease.svc:5080?
  1. Replace the http svc address with cluster IP, connecting to zero succeeded and data is imported, but still I don’t like executing Dgraph live import in this way, hopefully you know a better way to do it.

For this, the optimal way would be to run a utility pod using the same dgraph image and make sure that the same service account is used. Then you can test connectivity using the FQDN of an alpha pod and an zero pod†, and if you install AWS cli, you can test S3 bucket connectivity as well††. For the former, I use curl for the state path (zero is 6080, alpha is 8080). I can also use gprcurl for testing grpc.

† To get the FQDN of one of a unique pod, run hostname -f on the desired pod. I noticed you were trying to use the service proxy to randomly select a pod. It may be better to choose a specific pod for liveloader.
†† I use IRSA to setup perms for S3, so having a pod running in the context of the dgraph service account with aws cli is useful for testing permissions, just in case this was ever an issue.

I did these steps, however the zero pod seems to have a infinite tcp error:

  1. List my dgraph cluster pod in eks.
kubectl get pod -n myrelease

NAME                                            READY   STATUS    RESTARTS   AGE
dgraph-release-dgraph-alpha-0                   1/1     Running   0          89m
dgraph-release-dgraph-zero-0                    1/1     Running   0          54m

Cluster is healthy by checking path /health of Alpha pod listening on port 8080.

[{"instance":"alpha","address":"dgraph-release-dgraph-alpha-0.dgraph-release-dgraph-alpha-headless.neosight.svc.cluster.local:7080","status":"healthy","group":"1","version":"v23.1.0","uptime":5948,"lastEcho":1711246631,"ongoing":["opRollup"],"ee_features":["backup_restore","cdc"],"max_assigned":21840}]

Using curl command to apply schema works.

  1. Forward Alph and Zero port
kubectl port-forward dgraph-release-dgraph-alpha-0 9080:9080 -n myrelease
Forwarding from 127.0.0.1:9080 -> 9080
Forwarding from [::1]:9080 -> 9080

kubectl port-forward dgraph-release-dgraph-zero-0 5080:5080 -n myrelease
Forwarding from 127.0.0.1:5080 -> 5080
Forwarding from [::1]:5080 -> 5080
  1. Run dgraph standalone in docker container on my laptop
docker run --name dgraph-utility -d -p "8080:8080" dgraph/standalone:latest
  1. Copy schema and data files into container
docker cp /Users/Ann.Zhang/Downloads/g01.schema dgraph-utility:/g01.schema
docker cp /Users/Ann.Zhang/Downloads/g01.rdf.gz dgraph-utility:/g01.rdf.gz
  1. Shell into the container
docker exec -i -t dgraph-utility /bin/bash 
  1. Start live loading data
dgraph live \
--alpha localhost:9080 \
--zero localhost:5080 \
--schema /g01.schema \
--files /g01.rdf.gz

Message print out in container:

Dgraph version   : v23.1.0
Dgraph codename  : dgraph
Dgraph SHA-256   : 2b0d2fb977807f9d681c3a8e5f67a6fb133c99c772009158107aa6b1ac4cbd10
Commit SHA-1     : 2b18d19
Commit timestamp : 2023-08-17 13:27:10 -0500
Branch           : HEAD
Go version       : go1.19.12
jemalloc enabled : true

For Dgraph official documentation, visit https://dgraph.io/docs.
For discussions about Dgraph     , visit https://discuss.dgraph.io.
For fully-managed Dgraph Cloud   , visit https://dgraph.io/cloud.

Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2023 Dgraph Labs, Inc.



Running transaction with dgraph endpoint: localhost:9080

Processing schema file "/g01.schema"
Processed schema file "/g01.schema"

Found 1 data file(s) to process
Processing data file "/g01.rdf.gz"
[02:21:30Z] Elapsed: 05s Txns: 71 N-Quads: 71000 N-Quads/s [last 5s]: 14200 Aborts: 0
[02:21:35Z] Elapsed: 10s Txns: 140 N-Quads: 140000 N-Quads/s [last 5s]: 13800 Aborts: 0
[02:21:40Z] Elapsed: 15s Txns: 186 N-Quads: 186000 N-Quads/s [last 5s]:  9200 Aborts: 0

Error logs in dgraph-release-dgraph-zero-0. 10.178.11.193 is the IP Address of dgraph-release-dgraph-zero-0.

kubectl logs -f dgraph-release-dgraph-zero-0 -n myrelease

E0324 02:21:37.198103       1 x.go:352] Could not send error msg=write tcp 10.178.11.193:6080->10.178.11.175:58134: write: broken pipe code=ErrorNoData due to http error write tcp 10.178.11.193:6080->10.178.11.175:58134: write: broken pipe
E0324 02:21:47.197544       1 x.go:352] Could not send error msg=write tcp 10.178.11.193:6080->10.178.11.175:45494: write: broken pipe code=ErrorNoData due to http error write tcp 10.178.11.193:6080->10.178.11.175:45494: write: broken pipe
E0324 02:21:57.197362       1 x.go:352] Could not send error msg=write tcp 10.178.11.193:6080->10.178.11.175:53224: write: broken pipe code=ErrorNoData due to http error write tcp 10.178.11.193:6080->10.178.11.175:53224: write: broken pipe
E0324 02:22:07.197324       1 x.go:352] Could not send error msg=write tcp 10.178.11.193:6080->10.178.11.175:52942: write: broken pipe code=ErrorNoData due to http error write tcp 10.178.11.193:6080->10.178.11.175:52942: write: broken pipe
E0324 02:22:17.197429       1 x.go:352] Could not send error msg=write tcp 10.178.11.193:6080->10.178.11.175:56626: write: broken pipe code=ErrorNoData due to http error write tcp 10.178.11.193:6080->10.178.11.175:56626: write: broken pipe
E0324 02:22:27.197550       1 x.go:352] Could not send error msg=write tcp 10.178.11.193:6080->10.178.11.175:41134: write: broken pipe code=ErrorNoData due to http error write tcp 10.178.11.193:6080->10.178.11.175:41134: write: broken pipe
E0324 02:22:37.197504       1 x.go:352] Could not send error msg=write tcp 10.178.11.193:6080->10.178.11.175:37462: write: broken pipe code=ErrorNoData due to http error write tcp 10.178.11.193:6080->10.178.11.175:37462: write: broken pipe