# Extremely slow Oracle().WaitForTs() during mutation **URL:** \ **Category:** Dgraph\ **Created:** [June 10, 2021, 4:17am UTC](https://discuss.dgraph.io/t/extremely-slow-oracle-waitforts-during-mutation/14539 "2021-06-10T04:17:52Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![iluminae](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/iluminae/32/7367_2.png) [@iluminae](https://discuss.dgraph.io/u/iluminae)\ **Post date:** [June 10, 2021, 4:17am UTC](https://discuss.dgraph.io/t/extremely-slow-oracle-waitforts-during-mutation/14539/1 "2021-06-10T04:17:52Z")
I am trying to get to the bottom of severe performance issues on our dgraph system. I have traces indicating that `posting.Oracle().WaitForTs()` ([here](https://github.com/dgraph-io/dgraph/blob/a77bbe8ae0d42697a38069a9749cfe71c2dafbe6/worker/task.go#L886)) is taking several seconds on what seems to be most of my queries: ```auto 166.75ms: funcId=847459680 funcName=processTaskhca.xid message=Start. 166.75ms: message=Waiting for startTs: 16052571 at node: 3, gid: 1 4.85s: message=Done waiting for maxAssigned. Attr: "\x00\x00\x00\x00\x00\x00\x00\x00hca.xid" ReadTs: 16052571 Max: 16052571 4.87s: message=Done waiting for checksum match ``` What does this indicate? dgraph version v21.03, 1group, 3alphas, 20 cores 27GiB ram each, 2TiB disk in gke. ---
**Author:** ![MichelDiz](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/micheldiz/32/11873_2.png) [@MichelDiz](https://discuss.dgraph.io/u/MichelDiz)\ **Post date:** [June 10, 2021, 2:08pm UTC](https://discuss.dgraph.io/t/extremely-slow-oracle-waitforts-during-mutation/14539/2 "2021-06-10T14:08:07Z")
> [@iluminae](#): > > is taking several seconds on what seems to be most of my queries are the other instances (Alphas) healthy? Or having some limitation(like resources)? ---
**Author:** ![iluminae](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/iluminae/32/7367_2.png) [@iluminae](https://discuss.dgraph.io/u/iluminae)\ **Post date:** [June 10, 2021, 3:08pm UTC](https://discuss.dgraph.io/t/extremely-slow-oracle-waitforts-during-mutation/14539/3 "2021-06-10T15:08:28Z")
I do see the alphas using 18 cores of 20 possible, but this is more than the 16 dgraph has available in dgraph cloud, so I feel like thats a good amount. The prometheus stat `dgraph_num_queries_total` shows a total of ~1.7k queries per second being served, which does not sound like too much. Seems like traffic is being spread across the 3 alphas in the group decently. Google shows no disk throttling on the pvcs. We have like 800MB/s throughput quota on the 2TiB disk. ---
**Author:** ![MichelDiz](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/micheldiz/32/11873_2.png) [@MichelDiz](https://discuss.dgraph.io/u/MichelDiz)\ **Post date:** [June 10, 2021, 3:13pm UTC](https://discuss.dgraph.io/t/extremely-slow-oracle-waitforts-during-mutation/14539/4 "2021-06-10T15:13:18Z")
Are you feeling any loss in perf? Some of the logs are just noise for users. They are used for engineers be guided in some issues. But isn’t always a sign of anything too serious. ---
**Author:** ![iluminae](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/iluminae/32/7367_2.png) [@iluminae](https://discuss.dgraph.io/u/iluminae)\ **Post date:** [June 10, 2021, 3:17pm UTC](https://discuss.dgraph.io/t/extremely-slow-oracle-waitforts-during-mutation/14539/5 "2021-06-10T15:17:06Z")
> [@iluminae](#): > > I am trying to get to the bottom of severe performance issues on our dgraph system My production system has 50 million messages in a backlog to insert, and the fastest it can do it is ~3k nquads/s. The backlog is growing by the minute, but the 3k/s is a real problem. When we scale tested dgraph locally we were able to get ~130k nquads/s so I am trying to understand what the extreme slow down is caused by. ---
**Author:** ![MichelDiz](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/micheldiz/32/11873_2.png) [@MichelDiz](https://discuss.dgraph.io/u/MichelDiz)\ **Post date:** [June 10, 2021, 3:32pm UTC](https://discuss.dgraph.io/t/extremely-slow-oracle-waitforts-during-mutation/14539/6 "2021-06-10T15:32:52Z")
What cloud service are you using? my guess is that (In my opinion) containers systems has some drawbacks related to IO. In comparison with a Bare metal machine and using Dgraph’s binary directly on the machine is unfair as Dgraph has total use of the machine, as in a containerized model the container system controls everything. So it is a matter of tunning the Docker/K8s or it is a real limitation that you could avoid by spreading your cluster horizontally more and more. ---
**Author:** ![iluminae](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/iluminae/32/7367_2.png) [@iluminae](https://discuss.dgraph.io/u/iluminae)\ **Post date:** [June 10, 2021, 4:04pm UTC](https://discuss.dgraph.io/t/extremely-slow-oracle-waitforts-during-mutation/14539/7 "2021-06-10T16:04:03Z")
I am using GKE (google). My scale tests were also in k8s. I suppose I can attempt to scale down and horizontally scale the system to be many groups. It is a bit rough to do that on production, but I am rather desperate for performance at this point. ---
**Author:** ![dmai](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/dmai/32/1254_2.png) [@dmai](https://discuss.dgraph.io/u/dmai)\ **Post date:** [June 10, 2021, 4:05pm UTC](https://discuss.dgraph.io/t/extremely-slow-oracle-waitforts-during-mutation/14539/8 "2021-06-10T16:05:48Z")
Hi @iluminae. Are you seeing these numbers in a Dgraph Cloud backend? ---
**Author:** ![iluminae](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/iluminae/32/7367_2.png) [@iluminae](https://discuss.dgraph.io/u/iluminae)\ **Post date:** [June 10, 2021, 4:06pm UTC](https://discuss.dgraph.io/t/extremely-slow-oracle-waitforts-during-mutation/14539/9 "2021-06-10T16:06:44Z")
no, this is running in my GKE. ---
**Author:** ![dmai](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/dmai/32/1254_2.png) [@dmai](https://discuss.dgraph.io/u/dmai)\ **Post date:** [June 10, 2021, 4:25pm UTC](https://discuss.dgraph.io/t/extremely-slow-oracle-waitforts-during-mutation/14539/10 "2021-06-10T16:25:56Z")
And what are the machine specs for Zero? > [@iluminae](#): > > dgraph version v21.03, 1group, 3alphas, 20 cores 27GiB ram each, 2TiB disk in gke. Are these numbers for both Alphas and Zeros? You can check the dgraph\_pending\_proposals\_total metric to see if there’s a bunch of proposals queuing up in the system which would cause more delays. ---
**Author:** ![iluminae](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/iluminae/32/7367_2.png) [@iluminae](https://discuss.dgraph.io/u/iluminae)\ **Post date:** [June 10, 2021, 4:29pm UTC](https://discuss.dgraph.io/t/extremely-slow-oracle-waitforts-during-mutation/14539/11 "2021-06-10T16:29:26Z")
The zeros are 3x(4c 10GiB) and have extremely low utilization. Here is the kubectl top pods output (i have boosted my alphas to 30c 27GiB ram). ```auto NAME CPU(cores) MEMORY(bytes) graphdb-dgraph-alpha-0 27186m 25486Mi graphdb-dgraph-alpha-1 27924m 25054Mi graphdb-dgraph-alpha-2 28195m 25605Mi graphdb-dgraph-zero-0 208m 192Mi graphdb-dgraph-zero-1 204m 190Mi graphdb-dgraph-zero-2 200m 647Mi ``` Here is pending proposals: ![Screen Shot 2021-06-10 at 12.26.28 PM](https://canada1.discourse-cdn.com/flex007/uploads/dgraph/original/2X/8/8386717c2f2b46fd60cf87f1ae9d52ebc4eced64.png)