How to dynamically build a query

So, I have been experimenting with dgraph and am trying to adapt dgraph (dql) queries to our current business use case.

I am using:

  1. dql
  2. dgraph-js

Example

We want to query users based off a set of criteria. Below is a simple mongo pipeline to illustrate this.

Users
.aggregate([
{ $match: { $and: [{ favoriteDrink: XXXX }, { favoriteFood: XXXX}] } },
])
andQueries.push({ favoriteCountry: XXXX })

As you can see, I am building the query with an $and operator. I have no idea on how many criteria I may want but in mongo I have the flexibility to gradually just add this into the list.

There doesn’t seem to be a way to specify this in a database query in dgraph. Any ideas?

I get the feeling I may need to generate a string with valid query in dgraph-js using simple connecting filter – which is not ideal. Best I find out solution at db-level before digging this rabbit hole.

I saw an interesting approach in the best answer here but its tied to graphql and requires @skip and @include directives to be available (which is not unless I use dgraph Graphql). The solution also doesn’t entirely work the way I want because I’d still need to know the maximum amount of criteria to be filtered by.

Been here and tackled this one…

There are several different ways to do it, and I even use several ways depending on my use case at the time.

Barriers to cross:

  1. Solve the basic syntax
  2. Decide if filters are single level or multi-level and use var blocks when needed
  3. Build it dynamically with your favorite scripting language (ie: JS)

1. Solve the basic syntax

This is actually much easier in DQL right now than it is in GQL. In DQL, all you need to do is have a single root function and then join the other filters with conjunctions in a filter directive:

query {
  Users(func: type(User)) @filter(eq(favoriteDrink,"XXXX") AND eq(favoriteFood,"XXXX") AND eq(favoriteCountry:"XXXX")) {
    uid
  }
}

So here we can just add the type function as the root and use the AND conjunction to join the filter functions together in a filter directive. You can also use OR and NOT. Use parenthesis where needed to make specific enclosures.

Now here everything is on a single level and all with AND, what if we need to filter on multiple levels and use a mix of conjunctions?

2. Decide if filters are single level or multi-level and use var blocks when needed

To filter for data at different levels with a mix of conjunctions, the idea is to create a var block for every filter and then combine them in the end. Note: This may degrade performance based on use cases.

query {
  var1 as var(func: eq(favoriteColor,"X"))
  var(func: eq(interest,"Y")) {
    var2 as forUser
  }
  var(func: eq(city,"Z")) {
    usedInAddress {
      var3 as forUser
    }
  }
  Users(func: type(User)) @filter((uid(var1) OR uid(var2)) AND uid(var3)) {
    uid
  }
}

In this example we are looking for users with either the same favoriteColor (X) or a similar edge to a common interest (Y) and within a city (Z) that is used by addresses that are for the user. The var blocks go a long way to solve these advanced queries. I would test every var block to see if it performs better top to bottom or bottom to top. Some blocks may work better if flipped upside down.

3. Build it dynamically with your favorite scripting language (ie: JS)

Here you can simply use your favorite scripting language to build the query. I will just illustrate the simple format where var blocks and nested queries are not needed. (not a full working example)

const type = "User"
const conj = "AND"
const filters = []

filters.push({pred:"favoriteDrink",op:"eq",val:"XXXX"})
filters.push({pred:"favoriteFood",op:"eq",val:"XXXX"})
filters.push({pred:"favoriteCountry",op:"eq",val:"XXXX"})

let query += `query { ${type}s(func: type(${type})) @filter(`

filters.forEach((f,i) => {
  const c = (i>0) ? ` ${conj} ` : ""
  query += `${c}${f.op}(${f.pred},"${f.val}")`
})

query += ") { uid } }"
1 Like

Thank you @amaster507 for helping me make progress. It works. Particularly, the use of var blocks.

Making query strings are very unreliable. Is there a way of validating before posting to the api (on the dgraph-js side)?

Here’s query builder written that I’ve written in TypeScript.

It’s not complete yet, but see if it meets your requirements.

3 Likes

This is pretty neat. And I think it actually covers my use case with using spread operators and whatever currying is implemented in your OR/AND functions.

1 Like

Pretty neat trick @amaster507 and @zura . I’ve used a query builder (in Go, so I can leverage the dgraph libs) and assemble components together.

@zura maybe you wanna announce it in the Showcase category when the library is finished/released?

Hi @amaster507, @zura, @chewxy just wanted to share my thoughts as well:
What could work well additionally is to extend these framework(s) to use Dgraph itself as a configuration store.
It should be quite easy to store each s-p-o in the example below as a graph.

filters.push({pred:"favoriteDrink",op:"eq",val:"XXXX"})
filters.push({pred:"favoriteFood",op:"eq",val:"XXXX"})
filters.push({pred:"favoriteCountry",op:"eq",val:"XXXX"})

The clients merely need to access their configuration pieces, traverse the configuration and build out a query. This makes the entire process language agnostic, adds reuse and makes it easier to configure (reviews, move between environments etc.).

1 Like

I did a showcase some time back. It wasn’t finished then and it isn’t now, but I think it’s usable even though it lacks some features. I don’t have much time since I’m actively applying for jobs, so current version is as good as it gets for now :slight_smile: If anyone would like to contribute, pull requests are welcome.

@anand

The goal of the library is to make it very easy to have reusable parts and assemble them.

For your example:

filters.push(eq('favoriteDrink', 'XXXX'));
filters.push(eq('favoriteFood', 'XXXX'));
filters.push(eq('favoriteCountry', 'XXXX'));

Then you can use those filters in a query or edge filters:

const q = query()
  .filter(and(...filters));

A :heart: Brought me back to this post… I have since switched to a more complex way of building multiple GraphQL query blocks, normalizing the responses to get IDs, and then logically combining these on the client side to form a final query input for the actual query.

But this is the real reason I am back here. I can’t remember if this inspired me or just confirmed my inspiration, but yes, we store the users saved filters in the database themselves as self referencing filters:

type Filter {
  id : ID
  isMain: Boolean
  name: String
  conjunction: CONJUNCTION
  type: String
  paths: String
  subFilters: [Filter]
}
enum CONJUNCTION {
  AND
  OR
  NOT
}

paths is a stringified array such ["firstName","eq","Anthony"] that is the easiest way I found to store the actual filter params and still able to query/update.
The type and isMain allows me to quickly query all root filters for a specific type.