Dgraph, Microservices, Time series, Scalability and CI/CD

[amaster507 (Anthony Master)]

@tvvignesh

• In the microservices world, the recommendation is to have 1 db per service to enable scaling. Should we do that for dgraph as well cause then it will create isolated graphs without any connection in between different instances.

I would not personally recommend it. Dgraph is already built to shard and scale horizontally instead of vertically. I would recommend keeping connected data, well… connected as much as possible.

@tvvignesh

Let’s say we have isolated instances of dgraph, is it possible to refer to the nodes running on another dgraph instance? While we may be violating the microservices principles doing this, I was wondering how to have a unified view of the data graph of all the services together but still allow the flexibility of multiple teams to handle their own database pipelines.

Yes possible with the graphql endpoint and the @custom directive. You could even query data off from other non graph endpoints. Just FYI though, that data would not be expandable beyond the first retrieval so you have to get any of the nested data in a single request.

@tvvignesh

Is something similar possible in dgraph where I have multiple dgraph instances running each with their own graph and I do a query to graph using GraphQL± and you go back and run the individual GraphQL queries in the relevant GraphDB, aggregate and give back the response? This would enable push for graph database with microservices support.

see above.

@tvvignesh

If I have to see what the cons of DGrapgh are, what should I look at before using it in production?

This is really hard because it is based a lot on use case. For me, the biggest challenge is going to be doing client side custom filtering of deeply nested data at multiple levels. As for generally speaking, dgraph has outperformed our previous running MySQL 8.0.17 on RDS for even the simple data lookups on a single table. If you are comparing relational dbs to graph though and even care about the n+1 problem, then graphs will win out every single time!

The query language will probably be most peoples biggest challenge, it involves a different way of thinking than with traditional relational dbs. With SQL like queries you do SELECT [fields] FROM [tables [w/joins]] WHERE [filters] HAVING [special case filters] [pagination] this is totally different with graph databases of any kind. You literally select the fields you want and apply the filters at every level down through the graph. So making the jump for new comers will most likely be easier that don’t have any relational db experience then those making the jump. However if you take it for what it is, you literally ask for the data you want in the shape that you want it. Designing your schema is of the utmost importance to get it right early in the game.

@tvvignesh

Should I go for external time series stores?

Not my expertise here, sorry. I will say it works for basic datetime tracking and plays really well with UI providing the datetime in the correct format without needing to convert it to a javascript time object like I did with MySQL. I use times for events, tasks, and activity tracking, though there is no currently native support for createdAt and updatedAt fields which has to currently be resolved on the UI at this time.

@tvvignesh

So, is there any thoughts you have about this?

Sure, do it on Slash and pay-as-you-go.

@tvvignesh

you can geo-shard for MySQL. How can I do it with DGraph?

umm, idk sorry. All I know is that Dgraph does automatic sharding but not sure about any geo-sharding… guess someone up the chain can give more info here.

@tvvignesh

When you talk about databases, there are things to consider like handling database migrations/schema changes, rollback, roll forward and so on. […] How can we do this in DGraph?

I know this is tagged Dgraph, but I eventually foresee most users being mainly on the GraphQL endpoint so that is where I am answering this from, and I know that dgraph schema is vaguely the same under the hood.

Changing schema does not change data. However it can change access to that data. I am doing schema control in my github repo with my main UI. This will handle any kind of schema rollbacks or change control needed. Any data that needs to change along the way with a schema change will need to be done with custom scripts. Roll back on data changes may not be possible sometimes. If you delete data it is gone and a db rollback is not available unless you restore from a backup as far as I know.

@tvvignesh

So, how would you view this? How to ensure that a person using DGraph is not locked in to the ecosystem but rather embraces it and also is given the choice to move to something else later if needed?

Well, seeing that Dgraph made Graphql± and created Badger from scratch for their own pony show, they probably view it pretty highly. having this said, if any graph database is worth their salt, they would be able to import a well constructed graphql data with full schema. I foresee the future of graphql being the main graph query language. And not just a query language but a full functional db language. Dgraph is leaps and bounds beyond anybody else here and they have a lot of catching up to do. How can Dgraph use graphql as a query language, because they built their whole database structure around it.

@tvvignesh

Does DGraph have something inbuilt to solve this problem if you are doing multiple graph queries or should I still use dataloader to handle this?

Search for liveloader and bulkloader in the Dgraph docs. EDIT: sorry, I misread this as imports instead of queries. You are talking about something different here.