I would like to see memory profile of the alpha that’s about to OOM (I’m guessing the OOM happened and now the nodes are stable).
I think the problem is not related to oom. Earlier the instance crashed and I thought it was this. Now seeing the logs, I don’t think this is the issue.
Did you restart them?
Yes I did reboot the ec2 instance as it was unresponsive.
Also, there are too many connection errors. Did you change the IP:Port or something after starting the cluster?
No didn’t change the ip:port. Not sure why it is reporting these logs.
If it works, your queries should be working.
Yes my queries started working.
The logs were actually set to a rotation policy, so these might not tell us the clear picture.
So I inserted about half a million users, when this happened. If I start inserting other half now, I am quite sure this will happen again. What should we do if such case happens in production? I still think that dgraph moved a heavy predicate to the other instance, and that caused the unresponsiveness. What is the optimal way in which we can distribute predicates? The data sizes are attached in the image in the question.