Unify data and compute (micro-service) in GraphQL

Issues Dgraph
,
1

Moved from GitHub dgraph/4442

Posted by marvin-hansen:

Experience Report

Currently, I am building a rapidly growing GraphQL API with neo4j and some related tooling which allows me to just write a schema and the tooling generates the GraphQL API as well as DB CRUD operations. While the tooling is terribly ugly, it gets the job done 90% of the time. For the 10%, I had to resort to custom resolvers. This worked fine until I hit legacy SOAP service integration, which becomes very complex so I ended up querying the DB before using some templating to generate XML queries. Nothing terrible, but nothing fun either.

However, while looking at about half a dozen custom resolvers, I noticed a very distinct pattern:

  1. GraphQL entrance point (Say, AllPartnerProductsSortedByRegionSale)
  2. Querying the internal DB to get some keys required to construct an external query
  3. Build & execute an external query
  4. Apply filtering or sorting
  5. Return a GraphQL response

In many ways, the pattern of getting an internal GraphQL API call that requires information from an external resource is very common in practice and usually handled somewhere between the DB and frontend and, more often than not, handled as microservices.

What you wanted to do

I actually wanted to abstract away the boundaries between data & compute, and I wanted to abstract away the boundaries between internal & external resources.

What you actually did

I actually did most of that by re-formulating business processes as GraphQL endpoints and then implemented the underlying workflow through custom GraphQL resovers that were querying the internal DB as well as external resources & legacy services.

Why that wasn’t great, with examples

There are a couple of practical problems when doing just that with neo4j / GrandStack:

  1. Tooling relies on node.js / JS => Debugging disaster
  2. DB Quering can only be done through the public GrapQL API => Http overhead
  3. Daisy-chaining custom resolvers, well, you don’t do that => Very ugly.

Beyond that, there is a more fundamental issue:

Interweaving micro-services that integrate external resources with an internal data-graph, while technically feasible, isn’t adequately solved.

Neo4j comes with an APOC to call REST services, which only works with a cypher query. Technically, one can combine an APOC Rest call with the GrandStack & custom GraphQL resolver to expose REST as GraphQL while interweaving it with local data, but it would easily win a nomination for the ugliest hack possible let alone the fragility of patch-working together so many unrelated stuff.

The next best is ArangoDB that comes with FOXX, which does exactly this, embedding micro-services within their DB, but again, it uses JS (!) for building those microservices, but more importantly, exposing those mixed functionalities as one unified Graph hasn’t been solved adequately.

Also, I do not believe that embedding microservices inside a DB is a particularly terrific idea in terms of isolation and scalability when all it really takes is to call micro-service from within the DB.

What would be truly great?

A simple addition to DGraph consisting of just three simple bits:

  1. A stored procedure-like DB function that can call a microservices.
    A simple Go plugin, for example, would be sufficient.

  2. A simple @external(“ServiceName”) directive, which can be placed within the DB schema

  3. An update to the query executor that links everything together by dispatching from the directive to the DB function to call the connected micro-service.

Specifically, when a query hits @external(“ServiceName”) @external directive entity, the query executor calls the corresponding DB function “ServiceName”, which then calls the micros-service and returns the value so that the query can be completed with either a result or the external part left empty in case something went wrong.

If you think clearly about it, a lot of microservices really only fetch data, which then gets combined with internal data, so doing exactly this during a GraphQL query already massively simplifies system integration, maintenance, and quite frankly, reduce complexity.

Taking the converged data & compute graph one step further, say a query returns a set of users and their recent purchases, that result can, then again stuffed into a GraphQL endpoint, which yields a machine learning WS endpoint that applies, say a page-rank, and returns the top-3
cat-related-shopping-addicted-users.

From a system engineering perspective, storing micro-service call signatures inside the DB makes a lot of sense, and leveraging this information to bind these calls to an @external directive to access a micro-service during query-time makes even more sense and, practically speaking, allows a very rapid build-up of a converged knowledge & compute graph that almost recursively reads its data, computes it, and stores results back to it.

Even interactions with large, distributed ML systems, say your Apache Spark, becomes a whole lot less of an integration issue once accessible during query time.

To the best of my knowledge, there is no graph database with native, first-class-citizen, support of calling external resources and accessing these external resources during query time.

For people who build systems rapidly and rely on external REST services to do so, this would be a complete game-changer in terms of streamlined development productivity.

Any external references to support your case

A recent discussion of the idea of converging data & compute graph:

Foxx

neo4j / Grandstack
https://grandstack.io/

Neo4J REST/APOC

2

imkleats commented :

I generally like this idea. To add a few use-cases to what Marvin has already listed that a general implementation could support:

  • Event Triggers & Webhooks (see Hasura)
  • An unobtrusive entry-point for open-source or user-defined stored procedure libraries.
  • Foreign Data Wrappers (see Postgres)

The reason I say “generally” above is because I’m still grappling with how Dgraph fits in a Microservices Architecture (MSA) model in that it seems like a distributed, ACID graph database promotes having a shared database pattern that would otherwise be viewed as an MSA anti-pattern for other types of data layers, leaving me to rethink what “new” patterns might emerge.

So, I’m not entirely sure whether it’s more appropriate to primarily implement some of these things (specifically the foreign data integrations) through a service mesh or edge proxy itself (i.e. no need to reinvent the service discovery wheel) and to secondarily have a seamless integration between Dgraph and the mesh/proxy. On the other hand, features like event triggers, webhooks and stored procs would rightly fit within Dgraph’s scope; there could be value in engineering those latter things in a way that facilitates the seamless integration for the former in any case.

Edit: Adding a link to a reference paper - “A Simplified Database Pattern for the Microservice Architecture”.

The distributed architecture of Dgraph might already facilitate the use of the Zero as a control plane for lightweight sidecars that would offer object interface functionality on the external resources. These sidecars could be also then be integrated with the sidecars for any existing service mesh if needed.

3

pawanrawal commented :

@marvin-hansen Thanks for providing a detailed report and what you’d like to see being supported in Dgraph. We do have plans to provide custom GraphQL resolvers to support the use-case you are talking about here, i.e querying and aggregating information from external services. Since multiple users have asked for it, we’ll look into prioritizing it.

4

marvin-hansen commented :

@pawanrawal Thanks a lot for the update and the outlook.

I’m actually migrating a neo4j DB to DGraph right now mainly for concurrency & scalability reason so your assurance helps a lot to move fast forward. The initial results are mind-boggling and I have a very hard time justifying keeping neo after the migration. Sometimes, hard numbers and facts speak for themselves.

Inspired by @imkleats comment, for the time being, I am using Hasura as a layer to seamlessly integrate DGraph with various web and legacy services. I don’t mind unrooting it later once DGraph is ready, but for the time being, I have some hard deadlines to meet for getting things into production.

That said, reliable custom GraphQL resolvers for cross-origin querying&mutating would immediately become the reason we would trash all middleware and use DGraph as single unified data & compute graph. Maintainability and fast evolution remain king when you have to move fast forward.