429 throttle limit - cloud dedicated instance

Users Dgraph Cloud
1

I have a dedicated cloud instance. Occasionally, I see this message from dgraph in my apps’ logs: 429 Too Many Requests. Please throttle your requests. I’d like to know more about it.

  1. what is the limit?
  2. is this a configurable limit in dedicated instances in cloud?
  3. does the limit apply to a single client or all the clients together? (for context, I have multiple server apps using dql to talk to dgraph)

Thanks!

3 Likes
2

We see this too, in a cluster that doesn’t appear to be under stress. I suspect it’s due to the way dgraph handles a mixed read/write scenario and our dev team leaves the cluster over-provisioned in an attempt to compensate for this. I’m not convinced we understand the core issue.

So in addition to the above questions I would add;

  • what is the recommended way to diagnose which queries are triggering these 429s?
  • If dgraph is struggling in a mixed read/write scenario, what are best practices we can put in place to compensate?
  • is there a reference implementation (in any language, but we use typescript), that shows how we should handle these 429s which are not based on any fixed rate limits that I’m aware of?
2 Likes
3

We’re running into this same 429 Too Many Requests. Please throttle your requests. on our project. We are using a self-hosted instance of Dgraph in Kubernetes. This is only an issue on the Dgraph 21.12 release and seems related to this PR. That change seems to be configured to only show the 429 error when exceeding 100k pending requests, which we definitely do not believe we are doing.

For now we are downgrading to 21.03.2 where this is not an issue.

Regardless, would appreciate someone from Dgraph letting us know the answers to the questions posted by @RickSalmon and @azafoo above.

1 Like
4

Dgraph Cloud backends have max pending query limits set to prevent the backend from getting overloaded. That said, new dedicated backends these days are now automatically configured with higher limits. @RickSalmon your dedicated backend has now been updated and you shouldn’t see the too many requests message as often.

The limit applies for all clients. The limit is specifically per Alpha in a backend.

@azafoo @jj0b This 429 Too Many Requests error happens when the current pending queries count reaches the configured limit. The metrics for dgraph_pending_queries_total shows the current pending queries for any particular Dgraph Alpha. Also, you can inspect the traces to see the current list of running or high-latency queries.

1 Like
5

thank you, @dmai. it seems like the best approach from the client-side is to enable a retry mechanism if a 429 is received.

1 Like