Count Queries in GraphQL

[amaster507 (Anthony Master)]

I believe the number one use case will be counting for paginate data. What would be nice if possible is a way to use the same filter and results with cascade applied.

Two separate suggestions:

1. Use the same filter:

If I pass in values and build a pretty complex filter and do some pagination I would want to get a total count of the filtered results that way I can quickly do a complete pagination UI display with a first, ...current..., last option. Right now there is no good way to do this without getting all of the ids and counting them in a response which is making the client do extra work. But bottom line, I would want a way to signify to use the same filter|results from a block above. I know graphql really makes this complicated, but I think it would be possible with the graphql parser and DQL generator script that is working behind the scenes. Maybe something like this:

# generated queries
type Query {
  "Placeholder for Counted Nodes, used by @count directive"
  Count: Int
}
# example Query
{
  getData(
    filter: {
      # really complex filter
    }
    first: 10
  ) @count(query: Count) {
    id
  }
  Count # should ignore the first and offset filter properties
  allData:getData @count(query: allDataCount) {
    id
  }
  allDataCount:Count
}

the Count query could then just be a placeholder type that returns 0 unless a count directive utilizes it with or without an alias. And maybe even some smart logic in here that if only a single @count directive is given and only a single Count query is given, then place the Count in that return. I think this is making sense, at least to me it does.

EDIT: I am not sure in my after thought if the aliases are available at the deeper level or not? Does the GraphQL client strip off the aliases and then reattach them with the returned results, or are the aliases available to the resolvers as well? I haven’t wrote a resolver for many months now which makes me quite happy!

2. Work somehow with applying cascade. To explain my point here, examine this query:

{
  queryData(first:5) @cascade {
    id
    metaData(filter: {id:["0x1"]}) {
      id
    }
  }
}

This query gets only the data that are the parents of the metaData with the id <0x1>. I would want to be able to count this as a total result disregarding the first directive while honoring the cascade directive. But keep in mind that @cascade still does not work with pagination, so this may be a limiting factor.

[rajas (Rajas Vanjape)]

Thanks for your comments @amaster507 .

@amaster507

a way to use the same filter and results with cascade applied.

Yes, it will be possible to apply count along with cascade.

@amaster507

I would want to get a total count of the filtered results that way

With the current design of count queries, this could be handled by making two queries, first a count query to get the number of UIDs with given conditions and then a query with pagination to obtain the result. These queries could also be run simultaneously.

@amaster507

disregarding the first directive while honoring the cascade directive

Why would you want to disregard the first directive. With the current design, it will be possible to count with cascade directive. Querying for other fields will have to be done separately.

A more general question is to whether to have separate count queries or to somehow accomodate the count part in the query itself as it is done in DQL.

[rajas (Rajas Vanjape)]

Possible Alternative for countData type queries

The method to have separate queries of the type countData has the following drawbacks:

1. The query has only one field in their return type which is count of UIDs matching filter. It cannot have any other fields of the Data type.
2. For most purposes, a countData query will have to be accompanied by queryData query. As countData query does not return anything more than count of UIDs. It may not prove that useful as it is intended to be.

To solve this problem of countData type query, the alternative is to make all queryData type queries return an extra field count . This field will contain the count of UIDs satisfying the given filter condition.

This approach has the following advantages:

1. Any other fields of Data type could also be queried along with this count variable. Making it easier to get all needed data using a single queryData query rather than using 2 queries, countData and queryData.
2. This way of having a count variable in queryData is more consistent with the way count is handled in DQL. There is no separate countData query in DQL. Rather, a count(UID) variable is present which could be used to get a count of matching UIDs. This will make rewriting to DQL easier.

This approach has the following disadvantage:

1. This will be a breaking change as the return type of queries of the form, queryData will change from [Data] to a structure containing int (to store count) and Data. This will have to be implemented and tested with greater care to avoid any problems with current structure.

Sample Example:

type Data {
	id: ID!
    name: String!
	intList: [Int]
	stringList: [String]
 	metaData: [Data]
}

The following queryData query will be generated in output schema from the above input schema.

query {
    queryData(filter: DataFilter): (Int, [Data])
}

Example GraphQL count query:

query {
    queryData(filter: \*Some Filter*\) {
        count
        name
        stringList
    }
}

The above GraphQL query would be rewritten to the following DQL query.

query {
    queryData(func: type(Data)) @filter(/* rewritten filter condtition */)) {
        count: count(uid)
        name
        stringList
    }
}

@amaster507, I will like to hear your comments about this and whether this solves some of the concerns mentioned by you.

[amaster507 (Anthony Master)]

I think this really simplifies a lot. However in regards to:

@rajas

This will be a breaking change as the return type of queries of the form, queryData will change from [Data] to a structure containing int (to store count) and Data . This will have to be implemented and tested with greater care to avoid any problems with current structure.

I don’t think there has been much decisions made about how breaking changes will be handled and in this early fragile stage, it is probably best to stay away from breaking changes.

Would it be possible to put the count inside of the generated type itself?

Instead of:

type Data {
  id: ID!
  name: String!
  intList: [Int]
  stringList: [String]
   metaData: [Data]
}
query {
  queryData(filter: DataFilter): (Int, [Data]) 
  # I think the correct annotation would be {Int, [Data]}
}

It could easily be:

type Data {
  count: Int # a generated field with a new reserved keyword
  id: ID!
  name: String!
  intList: [Int]
  stringList: [String]
  metaData: [Data]
}
query {
    queryData(filter: DataFilter): [Data]
}

@rajas

Why would you want to disregard the first directive. With the current design, it will be possible to count with cascade directive. Querying for other fields will have to be done separately.

Sorry, bad terminology, I did not mean the 1st directive but rather the first: Int filter. If I limit my results to the first page of 10, I would want the count for all and not just the first page, because I already know that is <=10.

[pawan (Pawan Rawal)]

@amaster507

Would it be possible to put the count inside of the generated type itself?

That would not really work or if it works then it would be similar to a DQL output which I believe is a bit unintuitive where the first object returns the count and next ones the actual data. As an example, the result of the DQL query

{
  directors(func: gt(count(director.film), 5)) {
    totalDirectors : count(uid)
    name@en
  }
}

is

{
  "data": {
    "directors": [
      {
        "totalDirectors": 7712
      },
      {
        "name@en": "Buster Keaton"
      },
      {
        "name@en": "Andrei Tarkovsky"
      }
...

I believe we should not go with this design.

In the design that @rajas is suggesting, the schema would be like below. It is a breaking change yes but has the pros that mentioned.

type QueryDataResponse {
  count: Int64
  data: [Data]
}

query {
  queryData(filter: DataFilter): QueryDataResponse
}

@amaster507

Sorry, bad terminology, I did not mean the 1st directive but rather the first: Int filter. If I limit my results to the first page of 10, I would want the count for all and not just the first page, because I already know that is <=10.

That is right but I don’t think we should disregard the first argument if count is given. That is also incompatible with how DQL works. If the user just wanted the complete count, they can always do a separate query without the first argument.

[rajas (Rajas Vanjape)]

Another Possible Alternative to count queries

The approach of having a count variable inside queryData queries has the following disadvantages:

1. It is a breaking change and will change queryData behaviour drastically.
2. It also brings in other restrictions of not having any fields with the prefix of count , as count<FieldName> will be used to denote the count of predicate edges for <FieldName> field.

To avoid these two issues, as discussed with @pawan yesterday, we propose another approach to count queries.

Count at root

We can have another query of the name aggregate<type name>. This query will include fields for count and any other aggregation fields like sum or avg. For our current use-case, the updated schema would look like

input DataAggregateResult {
  count: Int
  ... 
  ...
  // Other fields for doing things like avg, sum, min, max etc.
}

query {
    aggregateData(filter: DataFilter): DataAggregateResult
}

Then the following GraphQL query

query {
  aggregateData(filter: {...}) {
    count
  }
}

would be rewritten to the following DQL query.

query {
    aggregateData(func: type(Data)) @filter(/* rewritten filter condtition */)) {
        count: count(uid)
    }
}

All query aggregation fields like sum, avg to be added later would be a part of return type of aggregateData query.

Count for a child

Along with the aggregateData query, to handle count of predicate edges and other aggregation fields, we propose having a <predicate_name>_aggregate field inside queryData query. This will solve the problem of having too many count<predicate_name> type fields inside the type.

For the following GraphQL schema

type Data {
  name: String!
  metaData: [Metadata]
}

Example:

query {
    queryData {
        name
        metaData_aggregate {
            count
        }
    }
}

The above query would be rewritten to DQL as follows.

query {
    queryData(func: type(Data))  {
        Data.name
        count(Data.metaData)   /* returns number of items in metaData list*/
    }
}

Note that the count(Data.metaData) field would be made a part of metaData_aggregate field while returning GraphQL response.

Other aggregation fields like min, max, sum could then be added later on to metaData_aggregate.
This approach is similar to how Aggregation queries are handled in other GraphQL providers.

@michaelcompton, I will like to hear your comments about this and any concerns you may have about the proposed solution.

[michaelcompton]

Yeah, I think, in general we probably should have wrapped our queries in a response type (like we ‘wisely’ did for mutations), that would allow not having to have the aggregateData query. Doesn’t really hurt anything, just means a bit of interface bloat.

This looks like a decent approach to me.

input DataAggregateResult {
  count: Int
  ...  // Other fields for doing things like avg, sum, min, max etc.
}

query {
    aggregateData(filter: DataFilter): DataAggregateResult
}

for those “Other fields for doing things like avg, sum, min, max etc.”, we’ll have to do something like either have a nested sum { ... } that contains all the fields you can sum, or do sum_fieldName to get all the alternatives. Same for count for a child. Nested seems more in keeping with what we’ve done for filters, etc.

The other way to think about it might be with some sort of @custom that allowed to add dql snippets that would be compiled into the query. That’s kinda attractive cause it’s completely generic and could support math, but would, I expect, be both harder to implement and harder to use for non dql experts.

I’l also just add my usual refrain that I see GraphQL as about an app’s data interface and not really about a query language, so I’d rather have a way to say ‘hey, add a count for this bit here’, rather than always compiling in all the possibilities everywhere.

[rajas (Rajas Vanjape)]

Extending Count to Aggregate queries

As mentioned in the above posts that the aggregateData query would later on be extended to other aggregate functions like avg, min, max, sum , here are some examples of GraphQL to DQL queries on how that will work.

sum, avg functions would be allowed on field which are of the type Int or Float . sum would have the same return type as the value on which it operates. avg would always be of the type Float to cater to cases in which the sum of values is not divisible by number of predicates.

min , max functions would operate on fields which are of the type Int , Float, String, Datetime and would have the same return type as the field on which it operates.

Aggregate queries at Root level

For the following GraphQL schema

type Data {
  name: String!
  metaData: [Metadata]
}

The following DataAggregateResult input type will be generated

input DataAggregateResult {
  count: Int
  min_name: String
  max_name: String
}

The following GraphQL query

query {
    aggregateData(filter: DataFilter) {
        min_name
    }
}

would translate to the following DQL query

{
  var(func: type(Data)) @filter(/* rewritten filter condition */) {
    a as name
  }

  me() {
    min(val(a))
  }
}

Note that as aggregate functions and count are treated differently in DQL queries. GraphQL queries with both count and aggregate functions would be rewritten to multiple DQL queries and combined together before returning.

Aggregate queries at other levels

query {
    queryData {
        name
        metaData_aggregate {
            min_metadata_field  // metadata_field is some field in type metadata of type string
        }
    }
}

The above GraphQL query would translate to the following DQL query

{
  data as var(func: type(Data))  {
    metaData {
        field as metadata_field
    }
    minField as min(val(field))
  }

  me(func: uid(data)) {
    name
    val(minField)
  }
}

A Note about count queries

It was mentioned in the first post that scalar fields like intList: [Int] would also support count operations. But, with the new model, it won’t be possible to support this as the AggregateData input field would only have a single count variable and all other count functions of fields like metaData will take place at a different level (inside metaData_aggregate field).

As counting of such scalar arrays is supported in DQL, one would still be able to get count of scalar arrays using custom DQL.

[rajas (Rajas Vanjape)]

Count Queries for GraphQL have been pushed into master today. Count Queries will be a part of 20.11 release. Note that work on other aggregate queries like average, maximum, minimum is still under progress.

Related PRs:

1. https://github.com/dgraph-io/dgraph/pull/6834
2. https://github.com/dgraph-io/dgraph/pull/6786

[maaft]

That’s great news!

However I’m missing an __aggregate field like in the following example:

queryPosts {
   __aggregate {
      count
   }
}

With the current implementation I can only do:

getUser(id: $userId) {
   postsAggregate {
      count
   }
}

which is problematic because it makes counting filtered queries hard. E.g. counting only posts with certain properties would include use of @cascade and forces me to return unwanted data (e.g. id of posts).

getUser(id: $userId) @cascade {
   posts(filter: {
      likes: {gt: 10}
   }) {
      id
   }
   postsAggregate {
      count
   }
}

After all, the goal of dedicated count queries is speed. Otherwise I could just query all IDs by myself and count on client-side.

[rajas (Rajas Vanjape)]

Count Queries have two types, counting at root and counting at other levels. If you want to count posts with certain features, you could use count(aggregate) queries at root.
In your case, it will look like this:

aggregatePosts(filter:/*Apply some filter*/){
  count
}

This will report count of all Posts satisfying the given filter.

[maaft]

Ah great, I overlooked this query! Thank you very much!

Will @auth permissions apply to these queries?

[rajas (Rajas Vanjape)]

Yes, they do.

[maaft]

Wonderful!

Now let me filter for child-properties and I’m beyond satisfied! Is there an Issue for child-property filtering? I can’t seem to find one.

[rajas (Rajas Vanjape)]

Filtering for child properties on Count Queries is also possible.

Example:

getUser(id: $userId) {
   postsAggregate(filter: {
      likes: {gt: 10}
   }) {
      count
   }
}

This will give the count of posts which have likes greater than 10 for the User .

[maaft]

That’s awesome.

But I can’t count Posts that have childs that have certain properties, correct?
Like:

getUser(id: $userId) {
   postsAggregate(filter: {
      statistics: {
         likes: {gt: 10}
      }
   }) {
      count
   }
}

By the way, thank you for answering all this stuff while no documentation is available!

[rajas (Rajas Vanjape)]

Yes, That won’t be possible.
Filtering on properties of child is not possible for other queries as well (not just count) .

[maaft]

Yes, I noticed that. Do we have an issue for this already? I couldn’t find one.

[amaster507 (Anthony Master)]

Filter on non-scalar fields or relations (Types)

Consider the schema type User { id: ID! pets: [Badger] } type Badger { id: ID! owner: User! @hasInverse(field: pets) age: Int! @search } Dgraph generates CRUD schema for the types with filter only on scalar types. What if I want to query all users who own boomer badgers (age > 40 years)? It would be great if I could just query queryUser(filter: {pets: {age: {gt: 40}}}) { id pets {id} }

[rajas (Rajas Vanjape)]

Aggregate Queries at root and child levels have been pushed to master. We will soon be updating related documentation.
Related PRs: https://github.com/dgraph-io/dgraph/pull/6985 , https://github.com/dgraph-io/dgraph/pull/7022 .

Note that Aggregate Queries (min, max, sum, avg) are not going to be part of 20.11 .
Count Queries are going to be part of 20.11