Alpha is failing on start

I set up a Dgraph cluster locally using minikube, 3x Alphas, 3x Zeros - everything was fine. Now I have scaled down all Alphas to 0

kubectl scale statefulset proj-graph-engine --replicas=0

then removed all the pvcs and pvs related to those Alphas and now when I’m scaling up the Alphas I get

...
I1119 10:26:06.414946      16 draft.go:1505] Calling IsPeer
E1119 10:26:06.415704      16 draft.go:1538] Error while calling hasPeer: error while joining cluster: rpc error: code = Unknown desc = No node has been set up yet. Retrying...
...

I’m using Dgraph v20.03.1

1 Like

I have started a new cluster using Dgraph v20.03.6 and now after dropping PVCs and scaling Alphas up this shows up in the logs:

[pod/proj-graph-engine-zero-0/proj-graph-engine-zero] I1119 11:44:46.514916      17 zero.go:440] Connected: cluster_info_only:true
[pod/proj-graph-engine-1/proj-graph-engine] I1119 11:44:47.374876      15 draft.go:1543] Calling IsPeer
[pod/proj-graph-engine-1/proj-graph-engine] E1119 11:44:47.380827      15 draft.go:1576] Error while calling hasPeer: error while joining cluster: rpc error: code = Unknown desc = No node has been set up yet. Retrying...
[pod/proj-graph-engine-zero-0/proj-graph-engine-zero] I1119 11:44:47.517125      17 zero.go:422] Got connection request: cluster_info_only:true
[pod/proj-graph-engine-0/proj-graph-engine] E1119 11:44:47.523360      18 draft.go:1576] Error while calling hasPeer: Unable to reach leader in group 1. Retrying...

And again, using Dgraph v20.07.2 and the same operation, scale down, remove PVC, scale up and then

[pod/proj-graph-engine-0/proj-graph-engine] I1119 11:55:31.892497      16 draft.go:1584] Calling IsPeer
[pod/proj-graph-engine-0/proj-graph-engine] E1119 11:55:31.894925      16 draft.go:1617] Error while calling hasPeer: error while joining cluster: rpc error: code = Unknown desc = No node has been set up yet. Retrying...
[pod/proj-graph-engine-1/proj-graph-engine] I1119 11:55:31.990508      17 draft.go:1584] Calling IsPeer
[pod/proj-graph-engine-1/proj-graph-engine] E1119 11:55:31.996177      17 draft.go:1617] Error while calling hasPeer: error while joining cluster: rpc error: code = Unknown desc = No node has been set up yet. Retrying...
[pod/proj-graph-engine-2/proj-graph-engine] I1119 11:55:32.865837      32 draft.go:1584] Calling IsPeer
[pod/proj-graph-engine-2/proj-graph-engine] E1119 11:55:32.879186      32 draft.go:1617] Error while calling hasPeer: error while joining cluster: rpc error: code = Unknown desc = No node has been set up yet. Retrying...
[pod/proj-graph-engine-0/proj-graph-engine] I1119 11:55:32.896428      16 draft.go:1584] Calling IsPeer
[pod/proj-graph-engine-0/proj-graph-engine] E1119 11:55:32.904441      16 draft.go:1617] Error while calling hasPeer: error while joining cluster: rpc error: code = Unknown desc = No node has been set up yet. Retrying...
[pod/proj-graph-engine-1/proj-graph-engine] I1119 11:55:32.996402      17 draft.go:1584] Calling IsPeer
[pod/proj-graph-engine-1/proj-graph-engine] E1119 11:55:33.000037      17 draft.go:1617] Error while calling hasPeer: error while joining cluster: rpc error: code = Unknown desc = No node has been set up yet. Retrying...
[pod/proj-graph-engine-2/proj-graph-engine] I1119 11:55:33.880791      32 draft.go:1584] Calling IsPeer
[pod/proj-graph-engine-2/proj-graph-engine] E1119 11:55:33.886389      32 draft.go:1617] Error while calling hasPeer: error while joining cluster: rpc error: code = Unknown desc = No node has been set up yet. Retrying...
[pod/proj-graph-engine-0/proj-graph-engine] I1119 11:55:33.904971      16 draft.go:1584] Calling IsPeer
[pod/proj-graph-engine-0/proj-graph-engine] E1119 11:55:33.912895      16 draft.go:1617] Error while calling hasPeer: error while joining cluster: rpc error: code = Unknown desc = No node has been set up yet. Retrying...
[pod/proj-graph-engine-1/proj-graph-engine] I1119 11:55:34.004921      17 draft.go:1584] Calling IsPeer
[pod/proj-graph-engine-1/proj-graph-engine] E1119 11:55:34.010401      17 draft.go:1617] Error while calling hasPeer: error while joining cluster: rpc error: code = Unknown desc = No node has been set up yet. Retrying...

This means the official Upgrade Database procedure won’t work anymore

I have accepted this to someone on the team to take a look.

I am looking at this right now, starting with v20.03.6.

@lukaszlenart Did you remove all the Alphas but keep the Zeros? If you’re looking to restart the cluster from scratch you’ll want to start from a clean slate (i.e., new data directories) for all Zeros and Alphas.

@lukaszlenart For this explicit process, using the dgraph helm chart, you could the following:

helm install pge --set image.tag=v20.03.6 dgraph/dgraph

## Scale Down Cluster and Delete Data + State
kubectl scale statefulset pge-dgraph-alpha --replicas=0
kubectl scale statefulset pge-dgraph-zero --replicas=0
kubectl delete pvc --selector release=pge

## Scale Up Cluster Starting with Zeros
kubectl scale statefulset pge-dgraph-zero --replicas=3

## Wait until 3 x healthy zero nodes
kubectl scale statefulset pge-dgraph-alpha --replicas=3

For the bulk loader, on an empty cluster, you would want to use an init container for bulk loader.

Generally, for immutable infrastructure patterns, It may be easier to just delete the statefulsets and recreate them from scratch again. With a helm chart used above, that process would be:

helm delete pge
kubectl delete pvc --selector release=pge

@dmai yes, I removed just Alphas, but then I removed Alphas and Zeros - problem persisted. The main issue is that, when you copied in all files from bulk loader to all the Alphas and then shut them down, they will start complaining in logs about file already exists and the cluster is broken.

@joaquin what do you mean by initContainers? copy data in?

@lukaszlenart Correct. On each of the Alphas, you’d have an initContainer, then do a spin-loop until you finish the bulk load and move directory created to p.

      command:
        - bash
        - "-c"
        - |
          trap "exit" SIGINT SIGTERM
          echo "Write to /dgraph/doneinit when ready."
          until [ -f /dgraph/doneinit ]; do sleep 2; done

Then kubectl cp file(s) into initContainer on alpha-0 pod (or from within the initContainer, curl it down), do the bulk load, touch /dgraph/doneinit. Do this same process for alpha-1, then alpha-2.

@joaquin thanks a lot, that should work!

@lukaszlenart As an example, I added initContainer automation in the current master of dgraph helm chart. If you wanted to use this, you could do the following.

Get the Chart

git clone https://github.com/dgraph-io/charts.git

REL="pge"
helm install "REL" \
 --set image.tag=v20.03.6 \
 --set alpha.initContainers.init.enabled=true \
 ./charts/charts/dgraph/

Copy Data to InitContainer

I used a sample dataset:

mkdir 1million && pushd 1million
PREFIX=https://github.com/dgraph-io/benchmarks/raw/master/data/
FILES=(1million.schema 1million.rdf.gz)

for FILE in ${FILES[*]}; do
  curl --silent --location --remote-name $PREFIX/$FILE
done

popd

Then I ran this process on alpha 0, 1, 2.

NUM=0
REL="pge"

kubectl cp ./1million/ $REL-dgraph-alpha-$NUM:/dgraph -c $REL-dgraph-alpha-init
kubectl exec -ti $REL-dgraph-alpha-$NUM -c $REL-dgraph-alpha-init -- bash

## inside initContainer
REL="pge"
dgraph bulk \
 --files /dgraph/1million/1million.rdf.gz \
 --schema /dgraph/1million/1million.schema \
 --zero $REL-dgraph-zero-0.$REL-dgraph-zero-headless.default.svc.cluster.local:5080

mv /dgraph/out/0/p /dgraph
touch doneinit