# Load Data using Dgraph w/ Kubernetes **URL:** \ **Category:** Users\ **Tags:** kind:question, area:bulk-loader, area:kubernetes\ **Created:** [October 2, 2020, 1:56am UTC](https://discuss.dgraph.io/t/load-data-using-dgraph-w-kubernetes/10732 "2020-10-02T01:56:51Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![saugat](https://avatars.discourse-cdn.com/v4/letter/s/46a35a/32.png) [@saugat](https://discuss.dgraph.io/u/saugat)\ **Post date:** [October 2, 2020, 1:56am UTC](https://discuss.dgraph.io/t/load-data-using-dgraph-w-kubernetes/10732/1 "2020-10-02T01:56:51Z")
Hi Everyone, I am new to DGraph and was trying to do the bulk loader in kubernetes. I found the yml file from the github [https://raw.githubusercontent.com/dgraph-io/dgraph/master/contrib/config/kubernetes/dgraph-ha/dgraph-ha.yaml](https://raw.githubusercontent.com/dgraph-io/dgraph/master/contrib/config/kubernetes/dgraph-ha/dgraph-ha.yaml). In this yml file in the comments sections it is mentioned to do the bulk load we can create an init container and copy the data from our local repository into pod’s repository dgraph repository. So I have downloaded the 1million.rdf.gz file form the github and tried loading it into the dgraph but I am unable to do so. I had to modify the Statefulset service to deployment. My alpha deployment looks likes this: ```auto apiVersion: apps/v1 kind: Deployment metadata: labels: io.kompose.service: alpha name: alpha spec: replicas: 1 selector: matchLabels: io.kompose.service: alpha template: metadata: labels: io.kompose.service: alpha spec: initContainers: - name: init-alpha image: dgraph/dgraph:latest command: - bash - "-c" - | trap "exit" SIGINT SIGTERM echo "Write to /dgraph/doneinit when ready." until [-f /dgraph/doneinit]; do sleep 2; done volumeMounts: - name: alpha-claim0 mountPath: /dgraph containers: - args: - dgraph - alpha - --my=alpha:7080 - --lru_mb=2048 - --zero=zero:5080 image: dgraph/dgraph:latest name: alpha ports: - containerPort: 8080 - containerPort: 9080 volumeMounts: - mountPath: /dgraph name: alpha-claim0 volumes: - name: alpha-claim0 persistentVolumeClaim: claimName: alpha-claim0 status: {} ``` After I deploy using the above mentioned file I use the following commands to copy my data file. I have made a p directory where i have included my 100million.rdf.gz file. ```auto kubectl cp /root/dgraph/docker-compose/p myalphapodname:/dgraph/ -c init-alpha kubectl exec myalphapodname -c init-alpha touch /dgraph/doneinit ``` But I am unable to see my data in the ratel UI. Am I missing something? If this is a silly question please forgive me. I am trying kubernetes and dgraph first time in my life and I need to deliver it asap. I did all the research before posting here. Please if anyone could help me with this that would really be great. Thank you ---
**Author:** ![chewxy](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/chewxy/32/4339_2.png) [@chewxy](https://discuss.dgraph.io/u/chewxy)\ **Post date:** [October 2, 2020, 2:27am UTC](https://discuss.dgraph.io/t/load-data-using-dgraph-w-kubernetes/10732/2 "2020-10-02T02:27:58Z")
Hey @saugat No question is too silly. We all learn thru asking questions. Welcome to the community by the way. Quick one: is your Ratel connected to the correct IP/port? Have you done the [appropriate whitelisting](https://dgraph.io/docs/deploy/dgraph-administration/#whitelisting-admin-operations) ? Also, @joaquin, I had a quick look thru of things, and couldn’t spot any mistakes. Ideas? ---
**Author:** ![saugat](https://avatars.discourse-cdn.com/v4/letter/s/46a35a/32.png) [@saugat](https://discuss.dgraph.io/u/saugat)\ **Post date:** [October 2, 2020, 2:34am UTC](https://discuss.dgraph.io/t/load-data-using-dgraph-w-kubernetes/10732/3 "2020-10-02T02:34:30Z")
@chewxy Thank you for saying there is no silly questions. So I had literally put down the 100million.rdf.gz file in my p directory on further research I got to know i need to run the some bulkloader commands ```auto dgraph bulk -r goldendata.rdf -s goldendata.schema --http localhost:8090 --zero localhost:5080 ``` and this will create the out/0/p and need to copy this p folder into alpha … If this is right process where do I need to run the above command in my zero ? ---
**Author:** ![chewxy](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/chewxy/32/4339_2.png) [@chewxy](https://discuss.dgraph.io/u/chewxy)\ **Post date:** [October 2, 2020, 2:43am UTC](https://discuss.dgraph.io/t/load-data-using-dgraph-w-kubernetes/10732/4 "2020-10-02T02:43:20Z")
Um, I think you need the [live loader](https://dgraph.io/docs/deploy/fast-data-loading/#live-loader), not the bulk loader. - Bulk loader is for before you have the cluster up and running. But once it’s up and running you should use the live loader ---
**Author:** ![MichelDiz](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/micheldiz/32/11873_2.png) [@MichelDiz](https://discuss.dgraph.io/u/MichelDiz)\ **Post date:** [October 2, 2020, 2:48am UTC](https://discuss.dgraph.io/t/load-data-using-dgraph-w-kubernetes/10732/5 "2020-10-02T02:48:40Z")
Yeah, Liveloader should be the chosen one for this task. But you can also use Bulkloader, in that case, you have to start the Alphas after the bulk. You can’t start the bulkload with Alphas running. Just the zero, also, you have to preserve the Zero instance (never delete the Zero volume). > [@saugat](#): > > `localhost:5080` I believe that if you are running inside a pod, you should use `zero:5080` or something similar coming from the SVC. One more thing, if you gonna use the Liveloader, you have to check your provider. If it is AWS, GCP or similar you have to expose the Alpha and the Zero gRPCs ports. Unless you do the liveloader inside a pod. ---
**Author:** ![joaquin](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/joaquin/32/2619_2.png) [@joaquin](https://discuss.dgraph.io/u/joaquin)\ **Post date:** [October 2, 2020, 7:51am UTC](https://discuss.dgraph.io/t/load-data-using-dgraph-w-kubernetes/10732/6 "2020-10-02T07:51:00Z")
Hi @saugat I wouldn’t recommend using a Deployment controller, as the nodes are not sticky and will use a randomly named pod. StatefulSets are useful for stateful apps with zero and alpha. A quick way to get started w/ K8S would be to use helm chart and then use live loader: ```bash RELEASE="my-release" helm repo add dgraph https://charts.dgraph.io helm install $RELEASE dgraph/dgraph ``` After all the pods are up, you can use port forward to make them available on localhost. ```bash RELEASE="my-release" export RATEL_POD=$(kubectl get pods \ --selector "component=ratel,release=$RELEASE" \ --output jsonpath="{.items[0].metadata.name}" ) export ALPHA_POD=$(kubectl get pods \ --selector "statefulset.kubernetes.io/pod-name=alpha-0,release=$RELEASE" \ --output jsonpath="{.items[0].metadata.name}" ) export ZERO_POD=$(kubectl get pods \ --selector "statefulset.kubernetes.io/pod-name=zero-0,release=$RELEASE" --output jsonpath="{.items[0].metadata.name}" ) ## For Ratel (HTTP) kubectl port-forward $ALPHA_POD 8080:8080 & kubectl port-forward $RATEL_POD 8000:8000 & ## For LiveLoader (GRPC) kubectl port-forward $ALPHA_POD 9080:9080 & kubectl port-forward $ZERO_POD 5080:5080 & ``` With these available at localhost, you can use live loader: ```bash dgraph live \ --files 1million.rdf.gz \ --schema 1million.schema \ --alpha localhost:9080 \ --zero localhost:5080 ``` And also you can visit [http://localhost:8000](http://localhost:8000) for Ratel. When configuring which alpha to use, select [http://localhost:8080](http://localhost:8080). Hope that helps. Joaquin ---
**Author:** ![Ann\_Zhang](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/ann_zhang/32/10634_2.png) [@Ann\_Zhang](https://discuss.dgraph.io/u/Ann_Zhang)\ **Post date:** [March 22, 2024, 3:41pm UTC](https://discuss.dgraph.io/t/load-data-using-dgraph-w-kubernetes/10732/7 "2024-03-22T15:41:17Z")
Hey Joaquin, I have Dgraph stateful cluster run on AWS EKS, and I’ve successfully exported data and schema to a S3 bucket. ![Screenshot 2024-03-22 at 11.34.13 AM](https://canada1.discourse-cdn.com/flex007/uploads/dgraph/original/2X/4/46573c7d5635f17a716379df414986be6df23ecf.png) Also, I have a network load balancer listening on `9080` and `5080` and I’m able to use jump server to forward the connection to `localhost:9080` and `localhost: 5080`, the localhost is my laptop. **Question is: Where should I run `dgraph live` command? Since I don’t have any Dgraph application running on my laptop.** I did the following things: 1. shell into an Alpha pod ```auto kubectl exec --stdin --tty dgraph-release-dgraph-alpha-0 --namespace myrelease -- /bin/bash ``` 1. Ran `dgraph version` which works. 2. Ran the import command, ```auto dgraph live --files s3://s3.us-east-1.amazonaws.com/myrelease-dgraph-data-backup/dgraph.r3887409.u0318.2131 ``` error says: ```auto Running transaction with dgraph endpoint: 127.0.0.1:9080 2024/03/22 15:46:06 context deadline exceeded Unable to connect to zero, Is it running at 127.0.0.1:5080? ``` I guess it must be that Zero service is not running in the same pod as the current Alpha pod so `127.0.0.1:5080` is not recognizable. 1. Then I specified the zero service in the command. ```auto dgraph live \ --zero http://dgraph-release-dgraph-zero.myrelease.svc:5080 \ --files s3://s3.us-east-1.amazonaws.com/myrelease-dgraph-data-backup/dgraph.r3887409.u0318.2131 ``` Also ```auto dgraph live \ --zero http://dgraph-release-dgraph-zero-headless.myrelease.svc:5080 \ --files s3://s3.us-east-1.amazonaws.com/myrelease-dgraph-data-backup/dgraph.r3887409.u0318.2131 ``` I got the same error says that unable to connect to zero. I am pretty sure that Zero pod and service is running well. ```auto Running transaction with dgraph endpoint: 127.0.0.1:9080 2024/03/22 15:57:01 context deadline exceeded Unable to connect to zero, Is it running at http://dgraph-release-dgraph-zero.myrelease.svc:5080? Running transaction with dgraph endpoint: 127.0.0.1:9080 2024/03/22 15:59:36 context deadline exceeded Unable to connect to zero, Is it running at http://dgraph-release-dgraph-zero-headless.myrelease.svc:5080? ``` 1. Replace the http svc address with cluster IP, connecting to `zero` succeeded and data is imported, but still I don’t like executing Dgraph live import in this way, hopefully you know a better way to do it. ---
**Author:** ![joaquin](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/joaquin/32/2619_2.png) [@joaquin](https://discuss.dgraph.io/u/joaquin)\ **Post date:** [March 22, 2024, 7:31pm UTC](https://discuss.dgraph.io/t/load-data-using-dgraph-w-kubernetes/10732/8 "2024-03-22T19:31:32Z")
For this, the optimal way would be to run a utility pod using the same dgraph image and make sure that the same service account is used. Then you can test connectivity using the FQDN of an alpha pod and an zero pod†, and if you install AWS cli, you can test S3 bucket connectivity as well††. For the former, I use curl for the state path (zero is 6080, alpha is 8080). I can also use gprcurl for testing grpc. † To get the FQDN of one of a unique pod, run **`hostname -f`** on the desired pod. I noticed you were trying to use the service proxy to randomly select a pod. It may be better to choose a specific pod for liveloader. †† I use IRSA to setup perms for S3, so having a pod running in the context of the dgraph service account with aws cli is useful for testing permissions, just in case this was ever an issue. ---
**Author:** ![Ann\_Zhang](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/ann_zhang/32/10634_2.png) [@Ann\_Zhang](https://discuss.dgraph.io/u/Ann_Zhang)\ **Post date:** [March 24, 2024, 2:23am UTC](https://discuss.dgraph.io/t/load-data-using-dgraph-w-kubernetes/10732/9 "2024-03-24T02:23:00Z")
I did these steps, however the zero pod seems to have a infinite tcp error: 1. List my dgraph cluster pod in eks. ```auto kubectl get pod -n myrelease NAME READY STATUS RESTARTS AGE dgraph-release-dgraph-alpha-0 1/1 Running 0 89m dgraph-release-dgraph-zero-0 1/1 Running 0 54m ``` Cluster is healthy by checking path `/health` of Alpha pod listening on port `8080`. ```auto [{"instance":"alpha","address":"dgraph-release-dgraph-alpha-0.dgraph-release-dgraph-alpha-headless.neosight.svc.cluster.local:7080","status":"healthy","group":"1","version":"v23.1.0","uptime":5948,"lastEcho":1711246631,"ongoing":["opRollup"],"ee_features":["backup_restore","cdc"],"max_assigned":21840}] ``` Using curl command to apply schema works. 1. Forward Alph and Zero port ```auto kubectl port-forward dgraph-release-dgraph-alpha-0 9080:9080 -n myrelease Forwarding from 127.0.0.1:9080 -> 9080 Forwarding from [::1]:9080 -> 9080 kubectl port-forward dgraph-release-dgraph-zero-0 5080:5080 -n myrelease Forwarding from 127.0.0.1:5080 -> 5080 Forwarding from [::1]:5080 -> 5080 ``` 1. Run dgraph standalone in docker container on my laptop ```auto docker run --name dgraph-utility -d -p "8080:8080" dgraph/standalone:latest ``` 1. Copy schema and data files into container ```auto docker cp /Users/Ann.Zhang/Downloads/g01.schema dgraph-utility:/g01.schema docker cp /Users/Ann.Zhang/Downloads/g01.rdf.gz dgraph-utility:/g01.rdf.gz ``` 1. Shell into the container ```auto docker exec -i -t dgraph-utility /bin/bash ``` 1. Start live loading data ```auto dgraph live \ --alpha localhost:9080 \ --zero localhost:5080 \ --schema /g01.schema \ --files /g01.rdf.gz ``` Message print out in container: ```auto Dgraph version : v23.1.0 Dgraph codename : dgraph Dgraph SHA-256 : 2b0d2fb977807f9d681c3a8e5f67a6fb133c99c772009158107aa6b1ac4cbd10 Commit SHA-1 : 2b18d19 Commit timestamp : 2023-08-17 13:27:10 -0500 Branch : HEAD Go version : go1.19.12 jemalloc enabled : true For Dgraph official documentation, visit https://dgraph.io/docs. For discussions about Dgraph , visit https://discuss.hypermode.com. For fully-managed Dgraph Cloud , visit https://dgraph.io/cloud. Licensed variously under the Apache Public License 2.0 and Dgraph Community License. Copyright 2015-2023 Dgraph Labs, Inc. Running transaction with dgraph endpoint: localhost:9080 Processing schema file "/g01.schema" Processed schema file "/g01.schema" Found 1 data file(s) to process Processing data file "/g01.rdf.gz" [02:21:30Z] Elapsed: 05s Txns: 71 N-Quads: 71000 N-Quads/s [last 5s]: 14200 Aborts: 0 [02:21:35Z] Elapsed: 10s Txns: 140 N-Quads: 140000 N-Quads/s [last 5s]: 13800 Aborts: 0 [02:21:40Z] Elapsed: 15s Txns: 186 N-Quads: 186000 N-Quads/s [last 5s]: 9200 Aborts: 0 ``` Error logs in `dgraph-release-dgraph-zero-0`. `10.178.11.193` is the IP Address of `dgraph-release-dgraph-zero-0`. ```auto kubectl logs -f dgraph-release-dgraph-zero-0 -n myrelease E0324 02:21:37.198103 1 x.go:352] Could not send error msg=write tcp 10.178.11.193:6080->10.178.11.175:58134: write: broken pipe code=ErrorNoData due to http error write tcp 10.178.11.193:6080->10.178.11.175:58134: write: broken pipe E0324 02:21:47.197544 1 x.go:352] Could not send error msg=write tcp 10.178.11.193:6080->10.178.11.175:45494: write: broken pipe code=ErrorNoData due to http error write tcp 10.178.11.193:6080->10.178.11.175:45494: write: broken pipe E0324 02:21:57.197362 1 x.go:352] Could not send error msg=write tcp 10.178.11.193:6080->10.178.11.175:53224: write: broken pipe code=ErrorNoData due to http error write tcp 10.178.11.193:6080->10.178.11.175:53224: write: broken pipe E0324 02:22:07.197324 1 x.go:352] Could not send error msg=write tcp 10.178.11.193:6080->10.178.11.175:52942: write: broken pipe code=ErrorNoData due to http error write tcp 10.178.11.193:6080->10.178.11.175:52942: write: broken pipe E0324 02:22:17.197429 1 x.go:352] Could not send error msg=write tcp 10.178.11.193:6080->10.178.11.175:56626: write: broken pipe code=ErrorNoData due to http error write tcp 10.178.11.193:6080->10.178.11.175:56626: write: broken pipe E0324 02:22:27.197550 1 x.go:352] Could not send error msg=write tcp 10.178.11.193:6080->10.178.11.175:41134: write: broken pipe code=ErrorNoData due to http error write tcp 10.178.11.193:6080->10.178.11.175:41134: write: broken pipe E0324 02:22:37.197504 1 x.go:352] Could not send error msg=write tcp 10.178.11.193:6080->10.178.11.175:37462: write: broken pipe code=ErrorNoData due to http error write tcp 10.178.11.193:6080->10.178.11.175:37462: write: broken pipe ```