So we have a kubernetes cluster, with 3 alpha replicas and 3 zero replicas.
We have an issue where two alpha pods have gone down, and for some reason the leader alpha has lost it’s leadership. So upon attempting to bring the other pods back up, they no longer can find the leader. Unfortunately the PVCs of the two downed pods have been destroyed. We want to try to rebuild a new cluster from this backup, but the one pod that was still up (lost it’s leadership) is failing to elect a leader because the other two pods are down.
We cannot even trigger a backup on the pod because i guess it’s stuck in this re-election state.
Does anyone have any tips for what we can do here? Will the p directory in the pod that is still up work as a backup that we can copy to the other pods?
Or is there a way to manually elect the leader back to the pod that is up?
Any insights would be appreciated.
Best.