I have a 6 nodes cluster (3 zeros, 3 alphas). I have 1 group.
I run dgraph version 20.11.3 on kubernetes cluster.
I found out some data inconsistency between nodes - the result differs when I run the query: 2 alphas returns the same data, 1 different:
for ip in 10.220.21.176 10.220.10.18 10.220.41.131; do curl -X POST -H 'Content-Type: application/dql' -s -d @dgraph_query.json http://$ip:8080/query | jq . | wc -l; done 3291 1585 3291
/state shows the same for all nodes. No errors in logs.
The only difference about the inconsistent node is
"forceGroupId": true iin the status. What does it mean BTW?
How is it even possible? How to fix such situation? How to diagnose it in the future if I don’t see any errors?
Possible important info: the data was populated on the nodes with
dgraph bulk. Is the output of this command consistent? Because I ran this on each alpha. Maybe I should run it only on 1 node and copy the
p dir over all nodes? Is it possible this caused this situation?
A lot of questions but I’m quite new in dgraph and trying to understand what’s going on there