# Advice on handling corrupt alpha pod

**URL:** <https://discuss.dgraph.io/t/advice-on-handling-corrupt-alpha-pod/18150>\
**Category:** Dgraph\
**Tags:** kind:question, dgraph\
**Created:** [January 5, 2023, 11:57am UTC](https://discuss.dgraph.io/t/advice-on-handling-corrupt-alpha-pod/18150 "2023-01-05T11:57:50Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![xbreid](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/xbreid/32/10095_2.png) [@xbreid](https://discuss.dgraph.io/u/xbreid)\
**Post date:** [January 5, 2023, 11:57am UTC](https://discuss.dgraph.io/t/advice-on-handling-corrupt-alpha-pod/18150/1 "2023-01-05T11:57:51Z")

</div>

Hello,

We had an incident today where an alpha pod started throwing a similar error to what was found in this topic: [LOG Compact FAILED with error: MANIFEST removes non-existing table 15777621,](http://discuss.hypermode.com/t/log-compact-failed-with-error-manifest-removes-non-existing-table-15777621/14982)

After restarted the cluster, most of our replicas began to stabilize, however the alpha pod originally impacted is restarting constantly throwing the following error:

```auto
12:27:33.499 2023/01/05 11:27:33 file does not exist for table 17230931
12:27:33.499 Error while creating badger KV posting store 

```

My assumption is that we need to remove this pod, and it’s PVC following these steps: [https://dgraph.io/docs/deploy/kubernetes/#removing-a-dgraph-pod](https://dgraph.io/docs/deploy/kubernetes/#removing-a-dgraph-pod)

However, after reviewing the Zero endpoints: [https://dgraph.io/docs/v21.03/deploy/dgraph-zero/#endpoints](https://dgraph.io/docs/v21.03/deploy/dgraph-zero/#endpoints)

It’s mentioned that you cannot use the same `idx` on the restarted alpha pod. Does this mean we simply cannot restart the alpha pod after removing the PVC, in this case `dgraph-alpha-1`?

Unfortunately, it will be rather difficult for us to change the `idx` value of that pod. We are also unable to delete the PVC, without first removing the pod entirely.

We had the idea to scale down our dgraph cluster to 1 alpha and 1 zero, remove the PVC, and remove the alpha from Zero via the endpoint. Then scale it back up to 3 alpha/zero. Is there any issues with doing it this way?

Currently our leaders are on alpha-2 and zero-2, if we scale down to 1 of each, will the leaders be re-elected accordingly?

Please let me know your thoughts.

---

<div class="post-metadata">

**Author:** ![xbreid](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/xbreid/32/10095_2.png) [@xbreid](https://discuss.dgraph.io/u/xbreid)\
**Post date:** [January 5, 2023, 1:20pm UTC](https://discuss.dgraph.io/t/advice-on-handling-corrupt-alpha-pod/18150/2 "2023-01-05T13:20:54Z")

</div>

It looks like we were able to resolve the issue by doing the following:

```
- cordon the node wich runs dgraph-alpha-1 and dgraph-zero-2 kubectl cordon $NODE_NAME

- kubectl delete pod dgraph-alpha-1 (this was the faulty corrupted dgraph pod)

- delete the pvc of dgraph-alpha-1 kubectl delete pvc datadir-dgraph-alpha-1 (this was the faulty corrupted dgraph pvc)

- uncordon the node from step 1

- That would relaunch dgraph-alpha-1 (the missing pod) on the node with a clean PVC created upon launch.

- The join process started, data started to rebuild from a snapshot of other alpha nodes
```