I have just tried five alpha in each group (docker cluster). When I delete the fifth alpha, the cluster becomes unstable.
The conclusion I have now is that the cluster can make stable queries only if more than half of the alpha in each group is healthy.
But in k8s, any alpha failure will bring instability. According to the statefulset details, I guess it may be because k8s always thinks that the alpha pod is running.
First I removed four alphas in ratel, Then I told k8s that I only need 6 alphas now.
kubectl -n graphql scale statefulset.apps/graphql-dgraph-alpha --replicas=6
kubectl -n graphql delete pod graphql-dgraph-alpha-9
kubectl -n graphql delete pod graphql-dgraph-alpha-8
kubectl -n graphql delete pod graphql-dgraph-alpha-4
kubectl -n graphql delete pod graphql-dgraph-alpha-3
Then, 8 and 9 will not restart, but 3 and 4 will restart continuously, but always fail.
The alpha statefulset details:
Replicas: 6 desired | 10 total
Update Strategy: RollingUpdate
Pods Status: 8 Running / 0 Waiting / 0 Succeeded / 0 Failed
This way does not seem to work. k8s always creates pods by sequence number.
If I delete /dgraph/doneinit as I did before, then delete the pod, the pod will not restart continuously, but the cluster is still unstable.