Building an API around Dgraph application

[forstmeier (John Forstmeier)]

Hi!

I have a question regarding overall API construction. I’m planning on using Dgraph as the core resource behind an application I’m working on but I also plan on “wrapping” the database in some additional functionality. To that end, I’m curious how to best go about the “wrapping.”

In order to provide programmatic access to third parties as well as my own UI, I see two initial approaches.

1. Expose the /graphql endpoint directly
   a. Pro: provides maximum flexibility
   b. Con: reduces my “wrapper” API control and possibly faces limitations of GraphQL vs Dgraph’s GraphQL± which would be available through say a client library implementation (this second point I’m definitely not 100% on)
2. Create a wrapper REST API
   a. Pro: greater “wrapper” API control, utilizes GraphQL± via a client library
   b. Con: increases required changes as database/API change

This documentat points out that when Dgraph is launched, it provides a /graphql endpoint that allows for unconstrained query/mutation operations to be run against the database using GraphQL. My core question is: what should I be considering when wrapping the Dgraph database with my own API?

I’m still in the process of familiarizing myself with the platform so any feedback, recommendations, clarifications, would be greatly appreciated. Thanks!

Best,
John

[MichelDiz]

@forstmeier

Expose the /graphql endpoint directly

If this expose is to “the world”, It is not recommended.

@forstmeier

Create a wrapper REST API

There is a new feature called “custom logic” for GraphQL. With it, you can make calls to lambdas, REST and other activities that can be done via HTTP. Perhaps it will be useful for you.

@forstmeier

what should I be considering when wrapping the Dgraph database with my own API ?

Related to what exactly?

Cheers.

[forstmeier (John Forstmeier)]

There is a new feature called “custom logic” for GraphQL.

Do you have a resource on this? And is it compatible with Dgraph’s GraphQL±?

I was a bit broad with my last question. I’m trying to understand how to best utilize the flexibility of a Dgraph while also maintaining business logic on it - again, very broad. What I’ve sort of settled on after thinking about it is:

• provide “fixed” queries for known objects (like org or user)
• provide “dynamic” queries that read the schema and build “filter options” that, once selected, are used by the business logic to build actual queries and return values (if that makes sense)

Is there an easy way to read the full schema/index as a JSON object using the Go client for Dgraph?

For example, let’s say I had a schema that looked something like this (pseudocode):

[
  Org {
    name string
    users [User
      name string
    ]
    items [Item
      name string
    ]
  }
]

A schema where there could be many Org types which have fields that contain many User/Item types. How could I query for an answer like “Type Org has fields called users that contain type User list and items that contains type Item list”. As far as I can tell the Types defined in the schema just show the clustering of predicates (and define indices) while the schema shows UID values but not the associated Type.

Thanks for the responses!

Best,
John

[MichelDiz]

@forstmeier

Do you have a resource on this? And is it compatible with Dgraph’s GraphQL±?

Check https://github.com/dgraph-io/dgraph/tree/pawanrawal/custom-logic
it’s pretty new and it is not implemented yet.

The branch was deleted. Due merge works. In my last comment on this topic I show a way to “dig” latest work on it.

@forstmeier

And is it compatible with Dgraph’s GraphQL±?

Not sure, but I guess it will be. You could potentially use the HTTP API to do graphql+- queries.

@forstmeier

I’m trying to understand how to best utilize the flexibility of a Dgraph while also maintaining business logic on it

It is not clear if you are asking about Dgraph or Dgraph’s GraphQL. I gonna assume that your questions are related to GraphQL tho.

@forstmeier

maintaining business logic on it

Well, Dgraph’s GraphQL has automatic “CRUD” queries and mutations based on your GraphQL Schema (I mean, the Schema GraphQL with Dgraph params on it).

PS. Dgraph itself doesn’t have “automatic CRUD”. In fact, nothing is automatic in Dgraph (only the background features of course). You have to define everything according to your data model. And learn the query language (of course).

@forstmeier

provide “fixed” queries for known objects (like org or user )

There’s no such feature. Maybe, after April, we gonna have some engineers working on custom graphql +- queries defined in the Schema itself. Which means “Fixed queries”. Let’s put a BIG maybe here.

@forstmeier

provide “dynamic” queries that read the schema and build “filter options” that, once selected, are used by the business logic to build actual queries and return values (if that makes sense)

The GraphQL feature does this. You have to provide a GraphQL schema with Dgraph’s params. But it is not so “wise” it is a “CRUD” pattern. Check these docs https://graphql.dgraph.io/docs/

You will still have to deal with your business logic, as each case is different.

@forstmeier

Is there an easy way to read the full schema/index as a JSON object using the Go client for Dgraph?

I’m not a Go dev, but I guess that yes, there is a way via Query. See Query Language - Query language
But is this a GraphQL question or Dgraph question? Graphql has its own way of doing this, it has built-in documentation out of the box.

@forstmeier

As far as I can tell the Types defined in the schema just show the clustering of predicates (and define indices) while the schema shows UID values but not the associated Type.

This feels like a Dgraph question. Can you confirm this?
Another question, have you used Dgraph or Dgraph’s GraphQL before?

If your question is related to Dgraph the answer is one, if it is to GraphQL it is another similar answer. But technically in both cases the query is similar to the pseudocode that you shared.

Cheers.

[acastle (AC)]

I opted (for my prototype at least) to not use the prebuilt Clients

• I’m running dgraph and only exposing the /graphql± endpoints to localhost.
• I then built a a RESTful API layer of my own creation to handle authentication, business logic, rate limiting, the usual; this then talks to dgraph and is exposed externally.
• I then built a website that consumes the RESTful APIs as “normal”.

The downsides with this approach are exactly as you describe; when I make a change to the schema (which happens often as I’m learning dgraph) then I need to change the API layer and the website as well.

I am finding it to be a great learning exercise though; knowing how to write queries and mutations by hand is helping me understand dgraph better than using an abstraction layer like a client. I am however duplicating what a client likely makes much easier - writing dynamic queries for sorting, filtering, pagination and so on.

I would not take my current implementation to production as-is; again great as a learning exercise though.

[forstmeier (John Forstmeier)]

I think I’m conflating concepts together because this is new to me.

Most of my focus, at the moment, is understanding the flavor of GraphQL (GraphQL±) produced by the company Dgraph. However, given some of the other documentation, I’m also interested in understanding the “pure” GraphQL option (if I understand that correctly).

Basically, here are a few points related to the query languages:

• What is the difference between “regular” GraphQL and GraphQL±?
• If there’s a notable difference, is it only possible to use the supported clients (e.g. Go) to run GraphQL queries/mutations and not “regular” GraphQL?
• The documentation resources you shared in your last response were helpful - can you help me understand the difference between the documents discussing the Dgraph query language?
• I’m definitely interested in the documentation you provided because defining the schema in “pure” GraphQL seemed to be more in-line with what I’m looking for where types are explicitly linked as predicates on other types in the definition.

On the database side:

• The “primary” endpoint on a Dgraph database seems to be geared towards receiving GraphQL± queries while the /graphql endpoint is designed to receive “regular” GraphQL - is that correct?
• Understanding this difference and the trade offs (if there are any) is something I’d definitely like.

To answer your question:

This feels like a Dgraph question. Can you confirm this?
Yes, as far as my only recent understanding of the concepts, it is a Dgraph question.

To answer your other question:

Another question, have you used Dgraph or Dgraph’s GraphQL before?
Yes, I’ve followed the tutorials and spun up local Dgraph databases in Docker and interacted with the database vai the Ratel UI.

Thank you very, very much for your responses - it’s helping me to understand better this too. Once I get better acquainted with the project, I’d love to contribute to the Go client on GitHub!

Best,
John

[forstmeier (John Forstmeier)]

Thanks for the response!

That’s close to what I think I’ll end up doing. I’m trying to work through some fancy dancing around providing some “dynamic” querying (while still wrapping it in some business logic) but we’ll have to see how it goes.

[MichelDiz]

@forstmeier

What is the difference between “regular” GraphQL and GraphQL±?

GraphQL+- is inspired in the “pure” GraphQL. In structure, they are very similar, but Graphql+- has “forked” a lot from Graphql’s specs. You can read more I can say about it here V2 and grphql vs graphql+- - #2 by MichelDiz

@forstmeier

If there’s a notable difference, is it only possible to use the supported clients (e.g. Go) to run GraphQL queries/mutations and not “regular” GraphQL?

All clients created by Dgraph runs only GraphQL+-. For “Pure GraphQL” you have to use any the several GraphQL clients out there. Like the ones from Apollo.

@forstmeier

can you help me understand the difference between the documents discussing the Dgraph query language?

The link from other conversations can explain the difference between the two languages. The docs I’ve shared are about start running the Graphql on Dgraph. I’m not sure what else to say about it.

@forstmeier

I’m definitely interested in the documentation you provided because defining the schema in “pure” GraphQL seemed to be more in-line with what I’m looking for where types are explicitly linked as predicates on other types in the definition.

Cool, We have no specific documentation on Graphql. You need to read the documentation for the Graphql spec. https://spec.graphql.org/

There are hundreds of GraphQL courses and tutorials out there. I would recommend that you study GraphQL before starting with Dgraph. And then go back to studying Dgraph. The concepts are very similar, you will not have difficulty.

@forstmeier

The “primary” endpoint on a Dgraph database seems to be geared towards receiving GraphQL± queries while the /graphql endpoint is designed to receive “regular” GraphQL - is that correct?

Yes, GraphQL+- is the main lang of Dgraph. And /graphql endpoint receives only “pure graphql”.

Cheers.

[forstmeier (John Forstmeier)]

Amazing info, thank you very much! Exactly what I needed. I’ll pivot over to play primarily with the /graphql endpoint/third-party clients going forward (I’m relatively familiar with GraphQL but could stand to take a refresher). Thanks again, this was super helpful!

[denpun (Dennis Mohan Punjabi)]

@MichelDiz

Check https://github.com/dgraph-io/dgraph/tree/pawanrawal/custom-logic

I cannot access the above mentioned branch. Getting a 404.
Is it publicly accessible?

Essentially, what I gather from previous comments is that:

^^ Traditional model is as above.

With Dgraph’s GQL support and Custom Logic Support, we are hoping to see if we can do:

Where data is only returned if Auth + Business logic conditions are met.

Is my understanding correct?

Thanks for all your efforts.

[MichelDiz]

The branch was deleted. But you can find the latest work by searching the branches like this bellow.

@denpun

Is my understanding correct?

I believe so, there will be an exclusive feature for Auth as well. I believe that its implementation includes the basic needs of Auth.

[MichelDiz]

BTW, I think I gave a BIG spoiler… nv, I hope it do not break the “climate” of surprise (not really now hehehe) and excitement about the new features xD

[rlmh (Luis Herrera)]

Are there any plans of exposing the graphql feature as a package (node/go)? The graphql features is fantastic but with business logic one would like to add middleware before returning result to the client who sent the query/mutation over gql. Not to compare but I’m thinking something like what postgraphile offers.

[MichelDiz]

@rlmh

Are there any plans of exposing the graphql feature as a package (node/go)?

I don’t think so. I can’t see how, in Dgraph context.

@rlmh

one would like to add middleware before returning result to the client who sent the query/mutation over gql.

You could use some merge schema thing or federated features. There are a lot of approaches. But that is a question for @michaelcompton .

[denpun (Dennis Mohan Punjabi)]

@MichelDiz

BTW, I think I gave a BIG spoiler… nv, I hope it do not break the “climate” of surprise (not really now hehehe) and excitement about the new features xD

Hah. Nice.
Good to know basic auth will be implemented.

@MichelDiz

The branch was deleted. But you can find the latest work by searching the branches like this bellow.

Great. Will take a look.

@MichelDiz

You could use some merge schema thing or federated features. There are a lot of approaches. But that is a question for

One way is also doing something like this with custom logic
Having custom logic handlers be called possibly twice

1. pre-data-fetch and
2. post-data-fetch

See image below.

Obviously overhead may be high but if it can be simplified some way, as @rlmh suggested, it may make things easy.

My diagram is a simplified just to explain things, actual implementation would need to be different to avoid high overhead. But I am guessing this would meet a lot of needs for data massaging and custom business logic.

Another question: Is this custom logic module an enterprise only feature?