Group size always reported as 0 in tablet logs

I have dgraph cluster setup using docker in following way :-
Machina A - 1 zero, 3 alphas
Machine B - 3 alphas

Machines are 4 cpu 16gb memory. Here is my alpha configuration (attached for one of the alphas, others have similar config).

lru_mb: 4096
zero: zero:5080
my: alpha1:7080
postings: /home/ubuntu/server1/p
wal: /home/ubuntu/server1/w
expose_trace: true
telemetry: true
max_retries: 3
mutations: allow
normalize_node_limit: 1000
query_edge_limit: 1000
idx: 1
badger.tables: disk
badger.vlog: disk

and zero configuration -

my: zero:5080
replicas: 3
idx: 1
v: 2
expose_trace: true

I bring up the setup and the alphas connect to the zero and the output of /state endpoint is

{
    "counter": "31",
    "groups": {
        "1": {
            "members": {
                "1": {
                    "id": "1",
                    "groupId": 1,
                    "addr": "ip1:7080",
                    "leader": true,
                    "lastUpdate": "1596088454"
                },
                "2": {
                    "id": "2",
                    "groupId": 1,
                    "addr": "ip1:7081"
                },
                "3": {
                    "id": "3",
                    "groupId": 1,
                    "addr": "ip1:7082"
                }
            },
            "tablets": {
                "device": {
                    "groupId": 1,
                    "predicate": "device"
                },
                "dgraph.graphql.schema": {
                    "groupId": 1,
                    "predicate": "dgraph.graphql.schema"
                },
                "dgraph.type": {
                    "groupId": 1,
                    "predicate": "dgraph.type"
                },
                "entity_type": {
                    "groupId": 1,
                    "predicate": "entity_type"
                },
                "entity_value": {
                    "groupId": 1,
                    "predicate": "entity_value"
                },
                "phone": {
                    "groupId": 1,
                    "predicate": "phone"
                }
            },
            "checksum": "10116038444952767405"
        },
        "2": {
            "members": {
                "4": {
                    "id": "4",
                    "groupId": 2,
                    "addr": "ip2:7080"
                },
                "5": {
                    "id": "5",
                    "groupId": 2,
                    "addr": "ip2:7081",
                    "leader": true,
                    "lastUpdate": "1596088475"
                },
                "6": {
                    "id": "6",
                    "groupId": 2,
                    "addr": "ip2:7082"
                }
            },
            "tablets": {
                "dgraph.graphql.xid": {
                    "groupId": 2,
                    "predicate": "dgraph.graphql.xid"
                }
            },
            "checksum": "4375140596800650063"
        }
    },
    "zeros": {
        "1": {
            "id": "1",
            "addr": "ip1:5080",
            "leader": true
        }
    },
    "maxLeaseId": "10000",
    "maxTxnTs": "10000",
    "cid": "d6fec2d9-89a6-409d-a7b1-8d7871c7247d",
    "license": {
        "maxNodes": "18446744073709551615",
        "expiryTs": "1598680456",
        "enabled": true
    }
}

The schema I am using is

<entity_type>: string .
<entity_value>: string @index(exact) @count .
<phone>: [uid] @count @reverse .
<device>: [uid] @count @reverse .

No matter how many nodes I insert, I always get zero logs like this -

Groups sorted by size: [{gid:1 size:0} {gid:2 size:0}]

I0730 06:02:10.600360      12 tablet.go:213] size_diff 0
I0730 06:10:10.600219      12 tablet.go:208]

Groups sorted by size: [{gid:1 size:0} {gid:2 size:0}]

I0730 06:10:10.600324      12 tablet.go:213] size_diff 0
I0730 06:18:10.600344      12 tablet.go:208]

Groups sorted by size: [{gid:1 size:0} {gid:2 size:0}]

I0730 06:18:10.600949      12 tablet.go:213] size_diff 0
I0730 06:26:10.600326      12 tablet.go:208]

Groups sorted by size: [{gid:1 size:0} {gid:2 size:0}]

I0730 06:26:10.600380      12 tablet.go:213] size_diff 0
I0730 06:34:10.600285      12 tablet.go:208]

Groups sorted by size: [{gid:1 size:0} {gid:2 size:0}]

I0730 06:34:10.600316      12 tablet.go:213] size_diff 0

The size diff never comes greater than 0, and that’s why I think the tablet movement is not happening. Filed a similar issue here (Dgraph Zero not moving tablets by itself) . What am I doing wrong in this setup.

Moreover with above configuration, I seem to hit a dead-end after inserting 20-25 mins. The performance has degraded on the machine on which the queries are going (in this case, machine A as it has all the tablets).



As for machine B, it is sitting idly and no load is being transferred to it



CPU profile for machine A

Heap Profile for machine A

Heap profile for machine B

CPU profile for machine B

Attaching zero and alpha logs from both machines. alpha and zero are from machine A, alpha2 is from machine B.
alpha.txt (95.8 KB) alpha2.txt (15.3 KB) zero.txt (32.2 KB)

Hi @praneelrathore

I’m so sorry no one has replied so far. Is this still an issue? The master branch of Dgraph has a bunch of improvements. If this is still an issue, I was wondering if you could test it out?

Hey @chewxy

No this is not a problem anymore. I had some issues which were solved in this thread by Dgraph team. This issue can be closed.