# Changing replication **URL:** \ **Category:** Dgraph\ **Tags:** kind:question\ **Created:** [October 7, 2020, 12:16pm UTC](https://discuss.dgraph.io/t/changing-replication/10826 "2020-10-07T12:16:07Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![paulrostorp](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/paulrostorp/32/1091_2.png) [@paulrostorp](https://discuss.dgraph.io/u/paulrostorp)\ **Post date:** [October 7, 2020, 12:16pm UTC](https://discuss.dgraph.io/t/changing-replication/10826/1 "2020-10-07T12:16:07Z")
What is the official procedure for scaling up/down an existing dgraph cluster ? As an exercise, I’m currently looking at scaling down a dgraph deployment running on kubernetes from 3 zeros and 3 alphas =\> 1 zero & 1 alpha. I’ve done this in the past and found myself in a situation where the alpha keeps crashing because it cannot connect to a leader. I did not resolve the issue at the time, just scrapped the volumes and started fresh. But now, I would like some advice on the best method. Thanks ---
**Author:** ![aman-bansal](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/aman-bansal/32/4475_2.png) [@aman-bansal](https://discuss.dgraph.io/u/aman-bansal)\ **Post date:** [October 7, 2020, 5:50pm UTC](https://discuss.dgraph.io/t/changing-replication/10826/2 "2020-10-07T17:50:05Z")
Hi @paulrostorp, you can use `/removeNode?id={id}&group={gid}` endpoint available at dgraph zero. More can be found [here](https://dgraph.io/docs/deploy/dgraph-zero). ---
**Author:** ![paulrostorp](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/paulrostorp/32/1091_2.png) [@paulrostorp](https://discuss.dgraph.io/u/paulrostorp)\ **Post date:** [October 9, 2020, 10:22am UTC](https://discuss.dgraph.io/t/changing-replication/10826/3 "2020-10-09T10:22:39Z")
Hi @aman-bansal I tried this and now I’m dealing with context deadline exceeded errors on any query ---
**Author:** ![paulrostorp](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/paulrostorp/32/1091_2.png) [@paulrostorp](https://discuss.dgraph.io/u/paulrostorp)\ **Post date:** [October 9, 2020, 12:28pm UTC](https://discuss.dgraph.io/t/changing-replication/10826/4 "2020-10-09T12:28:10Z")
I think it might be due to the fact that my alpha is not a leader: /state ````auto { "counter": "8981", "groups": { "1": { "members": { "1": { "id": "1", "groupId": 1, "addr": "dgraph-alpha-0.dgraph-alpha.starcards.svc.cluster.local:7080", "lastUpdate": "1601898387" } }, "tablets": { [...] } }, "zeros": { "1": { "id": "1", "addr": "dgraph-zero-0.dgraph-zero.starcards.svc.cluster.local:5080", "leader": true } }, "maxLeaseId": "180000", "maxTxnTs": "2680000", "maxRaftId": "3", "removed": [ { "id": "2", "addr": "dgraph-zero-1.dgraph-zero.starcards.svc.cluster.local:5080" }, { "id": "3", "addr": "dgraph-zero-2.dgraph-zero.starcards.svc.cluster.local:5080" }, { "id": "2", "groupId": 1, "addr": "dgraph-alpha-1.dgraph-alpha.starcards.svc.cluster.local:7080", "leader": true, "lastUpdate": "1602161263" }, { "id": "3", "groupId": 1, "addr": "dgraph-alpha-2.dgraph-alpha.starcards.svc.cluster.local:7080", "lastUpdate": "1602160485" } ], "cid": "f8a3101b-3c25-4dc4-b76f-360437ad697f", "license": { "maxNodes": "18446744073709551615", "expiryTs": "1591861766" } }``` ```` ---
**Author:** ![aman-bansal](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/aman-bansal/32/4475_2.png) [@aman-bansal](https://discuss.dgraph.io/u/aman-bansal)\ **Post date:** [October 11, 2020, 11:19am UTC](https://discuss.dgraph.io/t/changing-replication/10826/5 "2020-10-11T11:19:57Z")
Hi @paulrostorp, you are right. The issue is because you have removed the alpha leader. First of all, `removeNode` endpoint is not specifically for the downscaling dgraph cluster. It’s meant to replace the faulty node. But with few tricks, it can be used to downscale the cluster. But we have to ensure that cluster doesn’t go into an unstable state because of loss of consensus. So when you set up 2N+1 number of nodes in a RAFT setup, the cluster can handle N failures. So removing the first node (either Leader or follower) doesn’t have any adverse effect on cluster. But removing the second node which is the leader leads to faulty quorum and thus an unstable cluster. ---
**Author:** ![paulrostorp](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/paulrostorp/32/1091_2.png) [@paulrostorp](https://discuss.dgraph.io/u/paulrostorp)\ **Post date:** [October 11, 2020, 11:36am UTC](https://discuss.dgraph.io/t/changing-replication/10826/6 "2020-10-11T11:36:39Z")
@aman-bansal Is there a way to resolve the issue, or do I have to wipe the volumes and restore ? Also, shouldn’t the removeNode endpoint trigger a re-election or shouldn’t there be a way to manually trigger it ? ---
**Author:** ![dmai](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/dmai/32/1254_2.png) [@dmai](https://discuss.dgraph.io/u/dmai)\ **Post date:** [October 11, 2020, 4:34pm UTC](https://discuss.dgraph.io/t/changing-replication/10826/7 "2020-10-11T16:34:28Z")
As @aman-bansal said, `/removeNode` is meant to replace unhealthy nodes. That means it’s not used to remove the current leader of the group. Doing so can make the existing members stuck trying to connect if the leader was suddenly removed _and_ there’s no longer a majority. The process to call `/removeNode` only to remove followers (leaders are presumably active and healthy) for both Dgraph Zero and Dgraph Alpha groups. #### Manual recovery If your cluster is still stuck, you can wipe the volumes and restore. Otherwise, you can undergo some manual recovery steps by keeping one of the Alpha p directories. 1. Check `/state` for the `maxLeaseId` and `maxTxnTs` information (see [docs](https://dgraph.io/docs/deploy/dgraph-zero/#more-about-state-endpoint) about /state). 2. Keep a p directory around and remove other volumes. 3. Start the Zeros. - Call `/assign?what=uids&num=N` where `num` is set to the value for `maxLeaseId` from step 1. This sets the UID lease for blank UID assignment. - Call `/assign?what=timestamps&num=N` where `num` is set to the value for `maxTxnTs` from step 1. This sets the latest txn timestamp. 4. Copy the p directory to the respective Alpha volumes. 5. Start the Alphas. This is similar to the steps for bulk loading where bulk loader outputs p directory that you can then copy to the Alpha instances (step 4). ---
**Author:** ![paulrostorp](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/paulrostorp/32/1091_2.png) [@paulrostorp](https://discuss.dgraph.io/u/paulrostorp)\ **Post date:** [October 11, 2020, 4:39pm UTC](https://discuss.dgraph.io/t/changing-replication/10826/8 "2020-10-11T16:39:39Z")
Thanks @dmai, this is very useful information.