Hey there,
I installed dgraph + prometheus + grafana in a docker-compose stack and posted a schema. The “processed queries” metrics is constantly increasing (see picture below) without doing anything (I do no queries and no mutations). Is that a normal behavior? If yes, what is it doing in the background?
Hey @chewxy
I am using 20.11.2. I am doing nothing with the setup above (no queries / mutations initiated by me). There are some custom queries and mutations in the schema connecting to another graphql server.
I will try to setup an example later today
I think the reason is the following: we use a lot of subscriptions
e.g. we use dgraph like this
docker-compose.yml
version: '3.7'
volumes:
dgraph_data:
prometheus_data:
grafana_data:
services:
zero:
image: dgraph/dgraph:v20.11.2
volumes:
- dgraph_data:/dgraph
ports:
- 5080:5080
- 6080:6080
restart: on-failure
command: dgraph zero --my=zero:5080
alpha:
image: dgraph/dgraph:v20.11.2
volumes:
- dgraph_data:/dgraph
ports:
- 8080:8080
- 9080:9080
restart: on-failure
command: dgraph alpha --whitelist 0.0.0.0/0 --my=alpha:7080 --zero=zero:5080
dgraph.graphql
type Product @withSubscription {
productID: ID!
name: String @search(by: [term])
reviews: [Review] @hasInverse(field: about)
}
type Customer @withSubscription {
username: String! @id @search(by: [hash, regexp])
reviews: [Review] @hasInverse(field: by)
}
type Review @withSubscription {
id: ID!
about: Product!
by: Customer!
comment: String @search(by: [fulltext])
rating: Int @search
}
post with
curl -X POST localhost:8080/admin/schema --data-binary '@dgraph.graphql'
subscribe to ws://localhost:8080/graphql (e.g. using altair client):
subscription {
queryProduct {
name
}
}
and check total queries:
watch -n 1 curl 'localhost:8080/debug/prometheus_metrics |grep "dgraph_num_queries_total"'
example grafana graph:
shouldn’t subscriptions do the opposite? less queries and only push changes? For big databases this is really bad
@chewxy any chance that GraphQL subscriptions are translated into DQL polling?
When I think about a chat application with 10k users and everyone is subscribing to a chat, this would mean that the advertised 10k QPS of Dgraph is maxed out easily. Probably with a lot less users, when “standard” queries and mutations are added to that.
If that is indeed true (DQL polling), are event-driven subscriptions on the roadmap yet? This would be very important for medium- to large-scale applications in my opinion.
Yes, that is expected behavior for subscriptions. We also have event-driven lambda webhooks that you can set up. These get triggered anytime there is an add/update or delete mutation that happens. Have you had a look at https://dgraph.io/docs/graphql/lambda/webhook/ yet?
I think @marcown was talking about DQL polling - if that is exposed as an API to the end user. It’d be very useful for “subscriptions” in the DQL world, and I recalled you were working on something like that
Yeah, you can subscribe to custom DQL queries as well. Here is an example on how to do that https://dgraph.io/docs/graphql/subscriptions/#subscriptions-to-custom-dql
@pawan
My problem is the following:
As the number of subscriptions is increasing the number of internal DQL Polling requests per second are increasing. It would be better to have subscriptions only event driven, this would significantly reduce the total number of queries per second → less overhead as @maaft wrote
Is this on the roadmap yet?
I understand, that’s not on the roadmap for now. You could also look into increasing the polling interval if the number of queries per second is a concern or use the lambda webhook which are event driven.
Is there an easy way to reduce the polling interval?
EDIT:
I found --graphql_poll_interval 1 on alpha nodes, is possible to set it per subscription?
@pawan what about my chat example with 10k users?
When I think about a chat application with 10k users and everyone is subscribing to a chat, this would mean that the advertised 10k QPS of Dgraph is maxed out easily. Probably with a lot less users, when “standard” queries and mutations are added to that.
Am I correct with this assumption?
Yeah. We need to build an event driven system here. That was on our todo list.
Great to hear!
In the meantime I guess we could increase throughput by deploying more alphas?
Yup, you could do that. Also, not sure if we’re doing best effort queries or not for the streaming. That’s the other thing we should do.
CC: @hardik
Out of curiosity: What happens when this happens in-between the DQL polling interval? The initial state is: Foo.name = “Bob”
---- polling paused ----
Foo.name = “Alice”Foo.name = “Bob”---- polling continued ----
Are you using a diff to check if you need to trigger a subscription event? If yes, one need to be very careful setting the DQL pollling interval to large values.
The way subscriptions are working right now – they’d run the query, and compare the results with a cached version (or a hash I think) of the previous result. If there’s a difference, that result would be sent out. Otherwise, nothing gets sent.
Okay, thanks for clarification. We definitely need event driven subscriptions then or dgraph users will run into performance issues when they scale their apps.