Proposal Nested Object Filters for GraphQL rewritten as var blocks in DQL

A proposal of how to implement Nested Object Filters in GraphQL

This has been discussed before multiple times in great detail by a number of developers needing this functionality. Understanding a little of how the GraphQL gets rewritten to DQL, I know that the solution to this is complex and not a quick fix like many other items on the road map. I am working on tackling this again in my app making the filtering process easier for users to find the data they are looking for.

Our app context: We have three primary types of data (Contacts, Tasks, Events) that are interconnected through many different attributes that include but not limited to Addresses, Contact Info, Notes, Messages, Financial Transactions. For the sake of making the app more flexible and future proof we have tried to simplify any data piece to it’s own type. So this makes many of the types with fewer predicates but more edges. In return though, this makes filtering more complex :frowning:

Without posting my entire schema, here is a few parts to understand the idea (access logic removed for simple examples):

type Contact {
  id: ID
  hasAddresses: [HasAddress] @hasInverse(field: for)
  firstName: String @search(...)
  lastName: String @search(...)
}
type HasAddress {
  id: ID
  isPrimary: Boolean @search
  isDNC: Boolean @search
  isMailing: Boolean @search
  isPhysical: Boolean @search
  address: Address
  for: Contact!
}
type Address {
  id: ID
  line1: String @search(...)
  line2: String @search(...)
  city: City
  state: State
  usedBy: [HasAddress] @hasInverse(field: address)
}
type City {
  id: ID
  name: String @search(...)
  usedBy: [Address] @hasInverse(field: city)
  inState: State @hasInverse(field: hasCities)
}
type State {
  name: String @id
  hasCities: [City]
  usedBy: [Address] @hasInverse(field: state)
}

This schema allows multiple Contacts to share a single instance of an address so when it updates it updates for all while at the same time allowing that address to be a primary address for some, a mailing address for others and DNC for others. A complicated example but a real world example from our app and schema.

queryContact accepts the param filter: ContactFilter
ContactFilter accepts primarily field: {operator: value} with a few variations adding in:
id: [ID]
and: [ContactFilter]
or: [ContactFilter]
not: ContactFilter

An example is:

filter: {and: [{firstName: {eq: "Anthony"}},{lastName: {eq: "Master"}}]}

This generates DQL somewhat like:

query {
  queryContact (func: type(Contact)) @filter(eq(Contact.firstName, "Anthony") AND eq(Contact.lastName, "Master")) {
    # ...fields
  }
}

What I would like to see is expand the ContactFilter to also accept edge: EdgeTypeFilter

An example then would be:

filter: {and: [
  {hasAddresses: 
    {address: 
      {state: 
        {name: {eq: "Oklahoma"}}
      }
    }
  },
  {firstName: {eq: "Anthony"}},
  {lastName: {eq: "Master"}}
]}

But then here is the complexity, Rewriting this to DQL.

Ineffectively and very badly using cascade it would look like:

query {
  var filter1 (func: type(Contact)) @cascade {
    HasAddress.address {
      Address.state(eq(State.name, "Oklahoma")) {
        uid
      }
    }
  }
  queryContact (func: type(Contact)) @filter(uid(filter1) AND eq(Contact.firstName, "Anthony") AND eq(Contact.lastName, "Master")) {
    # ...fields
  }
}

This would in essence cause a bunch of extra work by the server which is why I believe it is not implemented yet. GraphQL → DQL rewriting is already capable of writing variable blocks, but eating up performance goes against the core mission of Dgraph. If a feature gets added like this, making it work fast, would be a problem and almost not possible depending on data and schemas.

But what if the feature was added with support only under special conditions. Since GraphQL does not map the reverse directive, it would rely completely on the hasInverse directive being on the edge you wanted to filter.

With inverse edges, the GraphQL could get rewritten to DQL:

query {
  var (func: eq(State.name, "Oklahoma")) @filter(type(State)) {
    State.usedBy {
      Address.usedBy {
        var filter1 as HasAddress.for
      }
    }
  }
  queryContact (func: type(Contact)) @filter(uid(filter1) AND eq(Contact.firstName, "Anthony") AND eq(Contact.lastName, "Master")) {
    # ...fields
  }
}

This follows the best practice to start with the smallest root function possible and traverse larger instead of starting larger and traversing smaller.

This is where I am at so far but I need to move forward and make my application more dynamic. Right now to build these inverse DQL statements, I have to write them by hand and save them in Slash with properties for where they start where they end, what options they have etc. These took a long time to write a little over 50 of these statements. But what I need to do is write a couple hundred of these DQL statements for var blocks somehow dynamically. UNLESS, @core-devs can somehow do this within the GraphQL → DQL rewriting process


Our other idea at hand is to write a bunch of GraphQL statements and then join them dynamically on the client side. Now that we are not getting charged on Slash per query, we can run more queries per user with less worries. If a user then wants half a dozen or more filters each filter would be its own GraphQL statement that returns IDs that get used by the final query. We are doing it with DQL in the middle right now to reduce all of the filter queries to one query with var blocks.

@pbassham anything you want to add from our conversations about this?

9 Likes

Thanks for the detailed post @amaster507. We are looking into how can we support it.

1 Like

Hi @pawan !
How is this going?

This feature would be a game changer…

I know the team has gotten may requests for this but wanted to throw my hat in the ring that this would be a massive improvement for our use case. Thanks for the great work Dgraph team!

2 Likes

This is an absolutely critical feature for our team, and it’s preventing us from using Dgraph to the extent we wish we could. Would love to know when this is slated for dev @pawan!

So the Dgraph team is actively working on this since a few weeks ago. Hopefully it will be done right with speed enhancements and at least two children.

If that is the case, I personally don’t mind waiting.

GraphQL Competition

  • Hasura / Nhost.io - Provides it out of the box for PostgresSQL
  • MongoDb - does not provide it. If you think about it, you can see why not as a NoSQL database is everything but relational
  • Fauna - barely provides basic filtering, so does not provide it. Need to use FQL for relations.

So only one out of three of the most popular GraphQL platforms even support it. There are many other databases that can support it, neo4j i.e., but not out of the box. You have to write your own with something like Prisma.io. As you can see, most databases that use GraphQL are SQL based, which means they will support the nested filters.

Our friends at Dgraph on the other hand, are writing this in Go, not typescript, so we know it will be super fast.

This is one of those features that makes a database a real option for real companies, versus a database to mess around with in your free time. We know under the hood this can be done.

J

2 Likes