Metric dgraph_max_assigned_ts alert

What I want to do

The synchronization timestamp status of the Dgraph instance node is abnormal. The following indicator monitoring often shows warnings with a difference of more than 1000. What is causing this?

max(dgraph_max_assigned_ts )  - min(dgraph_max_assigned_ts) 

What I did

Deployment configuration:
Three zeros and three alphas are deployed in K8s using default startup parameters.
–replicas Set to 3
Alarm configuration comes from

Dgraph metadata

dgraph version
Dgraph version   : v20.11.2
Dgraph codename  : tchalla-2
Commit timestamp : 2021-02-23 13:07:17 +0530
Branch           : HEAD
Go version       : go1.15.5
jemalloc enabled : true

Indicator data at that time:

I would like to ask, if this alarm is not handled, will there be any problems?

Hi, @chewxy
Please help look at the reasons.

It’s a measure of intragroup latency. If you have a group of instances {A, B, C, D, E, F}, the one with the highest maxAssignedTS is usually the leader. It simply tells you how much behind a follower is.

So your alphas are about 2 seconds out of sync with each other in the cluster. Queries could be blocked while waiting for other alphas to catch up.

Well, what causes this out of sync? Because it is normal for me to view the indicator data here, each alpha node will receive the client’s request and get the latest timestamp.