Add/remove alphas/zero

Users Dgraph
1

What I want to do

Write correct procedure for admins and conduct test replacement for fault nodes in case an incident.

Unfortunately I couldn’t find right procedure for replacement fault node (alpha for example) to new one. I can remove it with removeNode. Bur can’t understand how to add new into old group.

And also how to change count of replics (from 3 to 2) or count of shards. Is there anywhere any kind of paper for that theme?

I had 1 group with 3 zero and 5 alphas in 2 groups per 3 alphas. So I had 2 shards and 3 replicas.

Then I removed 2 alpha nodes from group2 by:
curl “localhost:6080/removeNode?group=2&id=6”
curl “localhost:6080/removeNode?group=2&id=4”
on leader zero and removed alpha3 and alpha5

Than added in config instead of alpha3 and alpha5:

  alpha31:
    ulimits:
      nproc: 1048576
      nofile:
        soft: 1048576
        hard: 1048576
    image: dgraph/dgraph:v22.0.2
    hostname: "alpha31"
    working_dir: /dgraph
    volumes:
       - /media/psf/1/alpha/alpha31:/dgraph
    ports:
      - 8086:8086
      - 9086:9086
    networks:
      - dgraph
    command: dgraph alpha --my=alpha31:7086 --security whitelist=0.0.0.0/0 --zero=zero1:5080,zero2:5081,zero3:5082 -o 6
 alpha51:
    ulimits:
      nproc: 1048576
      nofile:
        soft: 1048576
        hard: 1048576
    image: dgraph/dgraph:v22.0.2
    hostname: "alpha51"
    working_dir: /dgraph
    volumes:
       - /media/psf/1/alpha/alpha51:/dgraph
    ports:
      - 8087:8087
      - 9087:9087
    networks:
      - dgraph
    command: dgraph alpha --my=alpha51:7087 --security whitelist=0.0.0.0/0 --zero=zero1:5080,zero2:5081,zero3:5082 -o 7

and finally have got:

image
image2484×1194 264 KB

but in logs still have records with attempts to connect to “old” already removed alphas:

alpha1_1   | I0221 16:28:27.798546      15 log.go:34] 5 is starting a new election at term 995
alpha1_1   | I0221 16:28:27.798624      15 log.go:34] 5 became pre-candidate at term 995
alpha1_1   | I0221 16:28:27.798636      15 log.go:34] 5 received MsgPreVoteResp from 5 at term 995
alpha1_1   | I0221 16:28:27.798660      15 log.go:34] 5 [logterm: 995, index: 12642025] sent MsgPreVote request to 6 at term 995
alpha1_1   | I0221 16:28:27.798683      15 log.go:34] 5 [logterm: 995, index: 12642025] sent MsgPreVote request to 4 at term 995
zero2_1    | I0221 16:28:27.969901      15 raft.go:807] Skipping creating a snapshot. Num groups: 2, Num checkpoints: 1
zero1_1    | E0221 16:28:28.242016      16 pool.go:311] CONN: Unable to connect with alpha5:7084 : rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup alpha5 on 127.0.0.11:53: no such host"
zero3_1    | E0221 16:28:28.241998      14 pool.go:311] CONN: Unable to connect with alpha3:7082 : rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup alpha3 on 127.0.0.11:53: no such host"
zero2_1    | I0221 16:28:28.743059      15 zero.go:506] Got connection request: cluster_info_only:true 
zero2_1    | I0221 16:28:28.744190      15 zero.go:531] Connected: cluster_info_only:true 
alpha31_1  | E0221 16:28:28.744975      13 draft.go:1818] Error while calling hasPeer: Unable to reach leader in group 2. Retrying...
zero2_1    | I0221 16:28:28.744437      15 zero.go:506] Got connection request: cluster_info_only:true 
zero2_1    | I0221 16:28:28.745338      15 zero.go:531] Connected: cluster_info_only:true 
alpha51_1  | E0221 16:28:28.745855      19 draft.go:1818] Error while calling hasPeer: Unable to reach leader in group 2. Retrying...
alpha1_1   | W0221 16:28:28.799631      15 node.go:424] Unable to send message to peer: 0x4. Error: Do not have address of peer 0x4
alpha1_1   | W0221 16:28:28.799664      15 node.go:424] Unable to send message to peer: 0x6. Error: Do not have address of peer 0x6
alpha31_1  | E0221 16:28:28.877063      13 admin.go:862] namespace: 0. Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
alpha51_1  | E0221 16:28:29.024984      19 admin.go:862] namespace: 0. Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
zero1_1    | E0221 16:28:29.569017      16 pool.go:311] CONN: Unable to connect with alpha5:7084 : rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup alpha5 on 127.0.0.11:53: no such host"
zero2_1    | I0221 16:28:29.746161      15 zero.go:506] Got connection request: cluster_info_only:true 
zero2_1    | I0221 16:28:29.747214      15 zero.go:531] Connected: cluster_info_only:true

And despite it looks like all data on place (according picture the same amount) I couldn’t perform any request to db: it stops by timeout (I tried to increase).

2

I wonder if you have logs of Zero when did the remove of alphas. I have a feeling something went wrong while trying to remove the alphas. Is it possible for you to post the output of the following command (please update the endpoint correctly, it should hit 6080 port on any of the zeros) -

curl -s localhost:6080/state
3

curl -s localhost:6080/state

{
  "counter": "7504436",
  "groups": {
    "1": {
      "members": {
        "1": {
          "id": "1",
          "groupId": 1,
          "addr": "alpha6:7085",
          "leader": false,
          "amDead": false,
          "lastUpdate": "1677048160",
          "learner": false,
          "clusterInfoOnly": false,
          "forceGroupId": false
        },
        "2": {
          "id": "2",
          "groupId": 1,
          "addr": "alpha4:7083",
          "leader": false,
          "amDead": false,
          "lastUpdate": "1676996141",
          "learner": false,
          "clusterInfoOnly": false,
          "forceGroupId": false
        },
        "3": {
          "id": "3",
          "groupId": 1,
          "addr": "alpha2:7081",
          "leader": true,
          "amDead": false,
          "lastUpdate": "1677048164",
          "learner": false,
          "clusterInfoOnly": false,
          "forceGroupId": false
        }
      },
      "tablets": {
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000description": {
          "groupId": 1,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000description",
          "force": false,
          "onDiskBytes": "128724",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "199723"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000dgraph.drop.op": {
          "groupId": 1,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000dgraph.drop.op",
          "force": false,
          "onDiskBytes": "0",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "0"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000dgraph.graphql.p_query": {
          "groupId": 1,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000dgraph.graphql.p_query",
          "force": false,
          "onDiskBytes": "0",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "0"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000dgraph.graphql.schema": {
          "groupId": 1,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000dgraph.graphql.schema",
          "force": false,
          "onDiskBytes": "0",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "0"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000dgraph.graphql.xid": {
          "groupId": 1,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000dgraph.graphql.xid",
          "force": false,
          "onDiskBytes": "0",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "0"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000dgraph.type": {
          "groupId": 1,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000dgraph.type",
          "force": false,
          "onDiskBytes": "99448976",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "548015925"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000fkey": {
          "groupId": 1,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000fkey",
          "force": false,
          "onDiskBytes": "392857764",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "613926336"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000fname": {
          "groupId": 1,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000fname",
          "force": false,
          "onDiskBytes": "132397940",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "301729896"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000layerDigest": {
          "groupId": 1,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000layerDigest",
          "force": false,
          "onDiskBytes": "197556",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "312625"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000layerSize": {
          "groupId": 1,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000layerSize",
          "force": false,
          "onDiskBytes": "17737",
          "remove": false,
          "readOnly": false,
          "moveTs": "1371647",
          "uncompressedBytes": "33955"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000mode": {
          "groupId": 1,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000mode",
          "force": false,
          "onDiskBytes": "77964980",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "254656691"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000pInLayer": {
          "groupId": 1,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000pInLayer",
          "force": false,
          "onDiskBytes": "79910",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "272515"
        }
      },
      "snapshotTs": "15088534",
      "checksum": "5966991399447526092",
      "checkpointTs": "0"
    },
    "2": {
      "members": {
        "5": {
          "id": "5",
          "groupId": 2,
          "addr": "alpha1:7080",
          "leader": false,
          "amDead": false,
          "lastUpdate": "1676989603",
          "learner": false,
          "clusterInfoOnly": false,
          "forceGroupId": false
        },
        "7": {
          "id": "7",
          "groupId": 2,
          "addr": "alpha31:7086",
          "leader": false,
          "amDead": false,
          "lastUpdate": "0",
          "learner": false,
          "clusterInfoOnly": false,
          "forceGroupId": false
        },
        "8": {
          "id": "8",
          "groupId": 2,
          "addr": "alpha51:7087",
          "leader": false,
          "amDead": false,
          "lastUpdate": "0",
          "learner": false,
          "clusterInfoOnly": false,
          "forceGroupId": false
        }
      },
      "tablets": {
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000arch": {
          "groupId": 2,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000arch",
          "force": false,
          "onDiskBytes": "0",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "0"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000buildhost": {
          "groupId": 2,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000buildhost",
          "force": false,
          "onDiskBytes": "0",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "0"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000buildtime": {
          "groupId": 2,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000buildtime",
          "force": false,
          "onDiskBytes": "0",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "0"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000buildtime2": {
          "groupId": 2,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000buildtime2",
          "force": false,
          "onDiskBytes": "0",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "0"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000countLayers": {
          "groupId": 2,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000countLayers",
          "force": false,
          "onDiskBytes": "0",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "0"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000fInLayer": {
          "groupId": 2,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000fInLayer",
          "force": false,
          "onDiskBytes": "82343886",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "167901285"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000group": {
          "groupId": 2,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000group",
          "force": false,
          "onDiskBytes": "0",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "0"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000imageName": {
          "groupId": 2,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000imageName",
          "force": false,
          "onDiskBytes": "0",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "0"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000inImage": {
          "groupId": 2,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000inImage",
          "force": false,
          "onDiskBytes": "0",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "0"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000inTag": {
          "groupId": 2,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000inTag",
          "force": false,
          "onDiskBytes": "0",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "0"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000lImageName": {
          "groupId": 2,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000lImageName",
          "force": false,
          "onDiskBytes": "0",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "0"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000license": {
          "groupId": 2,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000license",
          "force": false,
          "onDiskBytes": "0",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "0"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000linkname": {
          "groupId": 2,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000linkname",
          "force": false,
          "onDiskBytes": "81258132",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "251186991"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000modtime": {
          "groupId": 2,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000modtime",
          "force": false,
          "onDiskBytes": "104723905",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "281812311"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000name": {
          "groupId": 2,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000name",
          "force": false,
          "onDiskBytes": "0",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "0"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000next": {
          "groupId": 2,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000next",
          "force": false,
          "onDiskBytes": "0",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "0"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000path": {
          "groupId": 2,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000path",
          "force": false,
          "onDiskBytes": "258422334",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "516516670"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000pkgid": {
          "groupId": 2,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000pkgid",
          "force": false,
          "onDiskBytes": "0",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "0"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000platform": {
          "groupId": 2,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000platform",
          "force": false,
          "onDiskBytes": "0",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "0"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000release": {
          "groupId": 2,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000release",
          "force": false,
          "onDiskBytes": "0",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "0"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000scanStart": {
          "groupId": 2,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000scanStart",
          "force": false,
          "onDiskBytes": "0",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "0"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000scanStatus": {
          "groupId": 2,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000scanStatus",
          "force": false,
          "onDiskBytes": "16287",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "45462"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000scanStop": {
          "groupId": 2,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000scanStop",
          "force": false,
          "onDiskBytes": "0",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "0"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000size": {
          "groupId": 2,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000size",
          "force": false,
          "onDiskBytes": "172686153",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "361156003"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000summary": {
          "groupId": 2,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000summary",
          "force": false,
          "onDiskBytes": "0",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "0"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000tagDigest": {
          "groupId": 2,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000tagDigest",
          "force": false,
          "onDiskBytes": "0",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "0"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000tagKey": {
          "groupId": 2,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000tagKey",
          "force": false,
          "onDiskBytes": "0",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "0"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000tagName": {
          "groupId": 2,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000tagName",
          "force": false,
          "onDiskBytes": "0",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "0"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000tagSize": {
          "groupId": 2,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000tagSize",
          "force": false,
          "onDiskBytes": "0",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "0"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000url": {
          "groupId": 2,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000url",
          "force": false,
          "onDiskBytes": "0",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "0"
        },
        "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000version": {
          "groupId": 2,
          "predicate": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000version",
          "force": false,
          "onDiskBytes": "0",
          "remove": false,
          "readOnly": false,
          "moveTs": "0",
          "uncompressedBytes": "0"
        }
      },
      "snapshotTs": "15019433",
      "checksum": "6077109718681509729",
      "checkpointTs": "0"
    }
  },
  "zeros": {
    "1": {
      "id": "1",
      "groupId": 0,
      "addr": "zero1:5080",
      "leader": false,
      "amDead": false,
      "lastUpdate": "0",
      "learner": false,
      "clusterInfoOnly": false,
      "forceGroupId": false
    },
    "2": {
      "id": "2",
      "groupId": 0,
      "addr": "zero2:5081",
      "leader": true,
      "amDead": false,
      "lastUpdate": "0",
      "learner": false,
      "clusterInfoOnly": false,
      "forceGroupId": false
    },
    "3": {
      "id": "3",
      "groupId": 0,
      "addr": "zero3:5082",
      "leader": false,
      "amDead": false,
      "lastUpdate": "0",
      "learner": false,
      "clusterInfoOnly": false,
      "forceGroupId": false
    }
  },
  "maxUID": "3580000",
  "maxTxnTs": "15150000",
  "maxNsID": "0",
  "maxRaftId": "8",
  "removed": [
    {
      "id": "6",
      "groupId": 2,
      "addr": "alpha3:7082",
      "leader": false,
      "amDead": false,
      "lastUpdate": "1676634924",
      "learner": false,
      "clusterInfoOnly": false,
      "forceGroupId": false
    },
    {
      "id": "4",
      "groupId": 2,
      "addr": "alpha5:7084",
      "leader": false,
      "amDead": false,
      "lastUpdate": "1676645733",
      "learner": false,
      "clusterInfoOnly": false,
      "forceGroupId": false
    }
  ],
  "cid": "91b87819-76d8-43d5-93da-abbea0f6428e",
  "license": {
    "user": "",
    "maxNodes": "18446744073709551615",
    "expiryTs": "1679062234",
    "enabled": true
  }
}
4

One more question, could you explain the sequence in which you added and removed the nodes? I see you have removed 4 & 6 and added 5,7 & 8.

5

The 5th (alpha1 with id5) - it was the last working one host in Group2. So I didn’t touch it. Just removed 2 dead (alpha3 with id6 and alpha5 with id4) and then add 2 new (alpha31 and alpha51).

I removed host alpha3 with id 6 and host alpha5 with id4:
curl “localhost:6080/removeNode?group=2&id=6”
curl “localhost:6080/removeNode?group=2&id=4”