Struggling getting a production quality DGraph instance setup

A recommended, simple, secure, upgradable community based approach to production

TLDR: I am really struggling getting a production quality DGraph instance setup as an App Developer. It would be great if the DGraph team could provided a simple (or as simple as possible) paint by numbers style scripted setup for running DGraph in production. One which includes security considerations, persistent data store setup, backups, logging, and provide upgrade scripts as DGraph is updated. I see this as some type of intermediate step between now and when you have a 3rd party offering DGraph as a service (my preference).

Failing that just some more help on the best direction to go would be great. :slight_smile:

Background

I am working on a startup/sabbatical called Helperific. The /sabbatical part is because I have some learning goals, like building it as a truly scalable system, and exploring unfamiliar new to me tech like Graph Databases, Lambda etc.

DGraph as it ticks all my boxes when researching what DB: Scalable, Distributed, Graph Databases (New to me), Open Source (Apache). All awesome!

Getting going with docker compose, which I was unfamiliar with at the time was easy!

My Ideal Setup

For my app I ideally want a High Availability, Multi Region setup, this feels like something unique to Dgraph that would be hard for most other databases. My app is small so 6 small t2.micro servers in 3 regions, + 3x Zero I guess, different availability zones (2 zones in each region) and 3 different Lambda deployments pointing to them would be cool.

My Struggles

So I have been doing lots of courses to try to get my head around Docker, AWS Networking, Kubernetes and deployment as code solutions such as Terraform.

I think I want to run Dgraph in a private VPC to secure it, as soon as you do this though, you then add complexity as you need a jump box to use Docker Machine for a single instance setup. So then I question if this is be best option.

I was also thinking maybe just setting up docker images preconfigured and using AWS ECS to deploy them to known IP Addresses to get my 6 image cluster. For upgrades I would create the 6 new images deploy those then move the data across, but I am just not sure this feel right.

Then I figured I should look more into Kubernetes and I setup a Kubernetes cluster using KOPs, by default it sets itself up in a public subnet, in just one region and I am not sure if I should be setting it up in a private subnet or the public subnet is fine, as it manages it’s own network security. Also the internet seems to indicate Kubernetes is not a great fit for DB’s but that might not apply to distributed dgraph.

As you can see there are so many deployment options, Single Host, Multi Host, Docker, Docker Swarm, Kubernetes. Within these there is HA and Replicated Cluster. I am struggling to pick one.

Also I have not even started trying to backup and upgrade to a newer version so far I have just been blowing away the data.

My Real Need

For now I just need to be able to get some type of dev, prod box up that I can easily update as new versions of dgraph are released, and eventually transition to a hosted solution one that has some backup script running and some type of monitoring, it probably does not even need to be distributed, even though that would be much cooler!

My suggested solutions

While I am sure all the deployment options are valuable and the best option depends on your setup. I feel that if you could provide a “best for most people” Wirecutter style recommendation, and step through this suggested setup in granular detail that would be very helpful, between now and when you have a hosted version.

Steps like configure you VPC, Create Custom Docker Images, Allocate IP Addresses, Create Bastion Box, Add this recommended monitoring solution. Add this script to backup to S3 periodically, Run this script to upgrade. Paint by numbers.

I am happy to put some time in learning a recommended stack, Docker, Docker Swarm, Kubernetes, etc, I would even consider moving providers from AWS to Google Cloud or whatever service if this helps, as the rest of my app is easy to setup in any provider.

I am thinking maybe you could create a Udemy course based on these steps, if you choose Kubernetes, Docker, Terraform as the preferred platform you could use an existing course as a prerequisite, so you don’t have to cover Docker / Kubernetes / Terraform basics, just the Setup and management of Dgraph. Either you could do the course yourself, or maybe you could find an existing author to work with on creating one, if you don’t have time.

What do you think?

I would be happy to help with this in any way I can!

3 Likes

Hi @vespertilian,

Thank you for your feedback! Would you be willing to help out writing out a few blogs or best practice documents on these topics to benefit the DGraph community?

Perhaps, extending the section we currently have in our documentation on using Kubernetes - https://docs.dgraph.io/deploy#using-kubernetes-v1-8-4

Thank you,
DGraph Product Management

1 Like

@Graphpinto

Sure, I am happy to try to write something up, when I work out what to do, but I am an app developer and not really an expert with infrastructure, so what I type up could just be wrong, and I don’t really know which option to pick, that’s what I was hoping to get help with.

When I google Kubernetes and database, it’s mostly people saying don’t waste your time on containers for database. Maybe it’s different for DGraph because it’s distributed?

http://patrobinson.github.io/2017/12/16/should-i-run-a-database-in-kubernetes/

What does Dgraph recommend as a stack for running the DB in production? Is it Docker Swarm or Kubernetes, Bare metal, or neither? What tech would your company choose if you were building an an app?

What is the recommend way to currently secure it in a cloud environment is the Kubernetes network enough? Or do I need to also run it in a private subnet?

I don’t think I am the best person to make these calls as I am just not as familiar with network infrastructure as I am hoping you guys are.

Hi @vespertilian,

Kubernetes is quite hot, and is used by several large companies to successfully deployed many stateful production apps - Kubernetes : Stateful Apps in Production and Distributed Database Orchestration - YouTube

Here is a link as to how you might use Kubernetes with DGraph - https://docs.dgraph.io/master/deploy/#using-kubernetes-v1-8-4

Thank you,
Pinto

2 Likes

@Graphpinto

Sorry for the late reply but deployment was just sucking up all my time and bumming me out a little. So I switched back to working on the app itself.

Thanks for the link, it’s cool that Kubernetes is hot right now, but is it the best solution for DGraph?

If anything the video just highlights how overwhelming deployment feels, another dev ops task to learn. I guess this is one of the disadvantages of a distributed DB when you are trying to do the ops for it yourself, it’s really 3 servers + zero you have to manage.

From your docs this feels super complex for every point release.

  • Start an export
  • Ensure it’s successful
  • Bring down the cluster
  • Run Dgraph using new data directories.
  • Reload the data via bulk loader.
  • If all looks good, you can delete the old directories (export serves as an insurance)

Especially because: “It is up to the user to retrieve the right export files from the servers in the cluster. Dgraph does not copy files to the server that initiated the export.”

That’s a lot of work for a solo developer. Especially from a list of bullet points to use as the initial guide.

Anyway I digress.

I am still a big fan of Dgraph and I am sure all of this will be solved with time. So I think I will post in another thread asking for a more concrete timeline on some type of paid option, because I think I am spending too much time on db ops and not enough working on my app, and business value. The app is not ready yet, so maybe I just keep developing it and wait for the hosted service, or I switch to something less ideal than a graph database but more widely deployed.

I’d recommend keeping the setup very simple. One Zero, one Alpha. Probably more than sufficient if you’re just building out the app. The more instances are added to the cluster, the more complex ops become. I doubt you need to go there to k8s and other orchestration systems, if you’re just starting out.

2 Likes

Just an update, I have setup DGraph just on a single AWS ECS instance, and tested backing it up to EFS. I am happy enough with this setup for now. I will try to write up the process for other users in the next couple of weeks.

I think your right, a single machine will do for now, and I can ideally be the first person to signup for your paid service.

3 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.