Webhook (Lambda) on add/update/delete mutations

Motivation


Users of Dgraph’s auto-generated GraphQL CRUD API, use the auto-generated mutations from the client-side to mutate(add/update/delete) data in the DB. Once the data has been mutated in the DB, they often want to listen to the changes that were done by the mutation and do some actions based on those changes. For example:

  • Sending those changes to a 3rd-party tool like DataDog.
  • Maintaining a changelog/audit-log on their own.
  • Auto-filling some data: When a user adds/updates an address, a function is triggered to verify the address against another API and fill out extra data with DQL such as linking Geo data from a 3rd party API.

Recently we have introduced CDC, but that sends the changes at a predicate level. While in GraphQL, the basic entities are types. So, with webhooks on mutations, we want to send all the changes that were done on a type in a single mutation as one event to the webhook. The user will have the ability to write some user-defined code in JavaScript to handle the events.

We aim for this to be part of the 21.03 release.

User Impact


Webhooks will greatly enhance the experience for users of the GraphQL API. Slash GraphQL users will be able to seamlessly enjoy this feature without any extra efforts.

Implementation


We plan to implement it in a similar manner to @lambda. At present, for @lambda following JSON HTTP payload is sent to the lambda server URL configured with Alpha:

{
  "resolver": "<Typename.FieldName>",
  "X-Dgraph-AccessToken": "<ACL-JWT-Token>",
  "authHeader": {
    "key": "<auth-header-name>",
    "value": "<auth-header-value == JWT>"
  },
  "parents": [{...}, {...}, ...],
  "args": {...},
}

where parents are sent for fields and args are sent for queries/mutations.

Similarly, after a mutation has been executed and committed to the DB, we will send the following JSON HTTP payload to the pre-configured lambda server URL:

{
  "resolver": "$webhook",
  "X-Dgraph-AccessToken": "<ACL-JWT-Token>",
  "authHeader": {
    "key": "<auth-header-name>",
    "value": "<auth-header-value == JWT>"
  },
  "event": {
    "__typename": "<Typename>",
    "operation": "<one-of: add/update/delete>",
    "commitTs": <uint64, the commitTs of the mutation>
    "add": {
      "rootUIDs": [<list-of-UIDs-that-were-created-for-root-nodes-in-this-mutation>],
      "input": [<AddTypeInput: i.e. all the data that was received as part of the `input` argument>]
    },
    "update": {
      "rootUIDs": [<list-of-UIDs-of-root-nodes-for-which-somethig-was-set/removed-in-this-mutation>],
      "setPatch": <TypePatch: the object that was received as the patch for set>,
      "removePatch": <TypePatch: the object that was received as the patch for remove>
    },
    "delete": {
      "rootUIDs": [<list-of-UIDs-of-root-nodes-which-were-deleted-in-this-mutation>]
    }
  }
}

Where:

  • only one of add/update/delete would be present inside the event key.
  • $webhook is chosen as the value for resolver as no type in GraphQL can start with a $.
  • Initially, we will start by sending the rootUIDs for any kind of operation, along with sending out input for add mutations and patch for update mutations as shown above. Then later we may extend the capability, wherein the case of add mutations, the input can itself contain the UIDs for the root input nodes mapped correctly to each node.
  • The order of rootUIDs isn’t the same as the order of input for add mutations.
  • The initial guarantee will be that any event would be delivered at most once to the webhook. It might be improved in the future.
  • Also, note that there is no guarantee as to the order in which the events will be delivered to the webhook, i.e., an update mutation that was committed after an add mutation may be delivered before the add mutation. One can use the commitTs in such cases to figure out if they have seen a higher commitTs for the same rootUID.

These events could be exposed to the user by adding lambda function like hooks in their lambda JS code, like this:

// If there is a handler registered for any specific operation on a type,
// the incoming webhook payloads for that type+operation pair will be forwarded to that handler
self.addWebHook({
  "Type1.add": type1AddHandler,
  "Type1.update": type1UpdateHandler,
  "Type2.delete": type2DeleteHandler,
  ...
})

And, the signature of a webhook handler will look like:

function webHookHandler({event, dql, graphql, authHeader}) {
 // user-defined code here
}

The choice of for which operations to send updates to webhook can be made by using the following new directive on a type:

"""
Type User has been configured to receive webhook events for add and delete mutations.
"""
type User @lambdaOnMutate(
  add: true,
  update: false,
  delete: true
) {
  id: ID!
  name: String
  # ... other fields
}

By default, no webhook events will be sent unless this directive overrides that behavior.

GOTCHA

A point to note is that doing DQL mutations won’t trigger these webhooks, these will be triggered only when someone does GraphQL mutations. That means, if the user is making GraphQL calls inside their webhook handlers, that would generate webhook events too, leading to recursion. For handling this scenario, we might consider the following approaches in a future release:

  • Send a particular header in the GraphQL request to disable the webhook trigger. The GraphQL API currently exposed in lambda can do this by exposing a third boolean argument in the graphql(query: String, variables: Object, disableWebhook: Boolean) call. The header may need to be like a JWT so that not anyone can send that header. Or, actually, this could just be part of extensions in the GraphQL request instead of being a header, but still like a JWT so that we are sure it hasn’t been meddled with. We can take the JWT signing information as webhook configuration in the schema.
  • The auto-generated mutations can have a flag to do the same. But, that would enable any client to disable webhook, which may not be intended.

The first approach seems suitable for this, open to discussions.

Note: At present, the user of webhooks is expected to handle this recursion scenario.

References


  • None yet

cc: @pawan @gja @amaster507 @vvbalaji

2 Likes

@pbassham, I added some original use cases on here that go along with what you are doing with our backend and filling out data using Google API. Interested in your feedback here too.

Good that we are explicitly asking users to list the types they are interested in instead of making it a replacement for CDC.

Would it make sense to use ACL to limit who can send this request instead of worrying about from where it came?

@gja, would you need some mechanism to throttle the number of events at the level of alpha or would you have enough lambda resources provisioned for a multi-tenant user.

It depends on WHO is generating the ACL token. For example, If in Slash ACL token is transparent to the Slash User, i.e., it is Slash who is managing this and any Application Client doesn’t have to send the ACL token, then this information can be put in the token.

But, If Slash is not managing this, and ACL token has to come from an Application Client then the issue is that an Application client can’t be trusted. Because, then they can always generate a token using ACL login which disables the webhook, and that won’t be an expected behavior by the Slash User.

I guess, 2nd approach is what is being used.

It is only the Slash User who should be able to disable the webhooks if needed. So, for that all we need is an HMAC which establishes the authenticity of the Slash User, and so a separate JWT with its security config in the GraphQL schema seems the best approach to me for this.

EDIT: ACL may not be enabled in all cases, so this information can’t be merged with the ACL JWT.

Great feature… I’m banking on this one for the upcoming release.

1 Like

This has been merged to master and will be part of v21.03 release.
Related PRs:

  1. https://github.com/dgraph-io/dgraph/pull/7494
  2. feat: Add support for webhook by vardhanapoorv · Pull Request #14 · dgraph-io/dgraph-lambda · GitHub
3 Likes

Nice! Does this include pre-hooks for validation purposes?

1 Like

I just gave it a spin locally and it works great. Thanks.

1 Like

No, it is only post-mutation hooks.

Pre-hooks over lambda will add cost in terms of mutation latency, as they would require Dgraph to suspend the mutation until the network call from lambda doesn’t return.

I think, for validation, we should be supporting that through some directive that would allow listing down the constraints on the input in a manner that they can be executed on Dgraph Alpha itself.

1 Like

@abhimanyusinghgaur - I entirely disagree with you on pre-hooks!

1.) You still have not made @auth directive useable (nor has anyone responded to many of my posts here about this). Are there plans to make it usable in the near future?

2.) It may cost time in mutation latency, but so do lambda mutation resolvers, which does the same thing, just requires MORE work by creating a new mutation. Does that mean you should get rid of that action? Pre-hooks would be optional, as are post-hooks, and are needed.

You guys need to bring this back up with your team. Again, an optional feature that I believe MOST users here with any app of complexity would agree with me on.

Besides the many validation reasons, I don’t want to change something, then change it back (if I even have enough information to change back deleted nodes) just because I don’t have access to a before logic…

Right now, this can’t even be done.

Thanks,

J

4 Likes

The original concept was a hook for both pre and post. I understand that it may incur a cost of performance, but isn’t that for the end user to decide if they want that added cost at the exchange of this feature.

I agree with @jdgamble555 on this one too!

5 Likes