Dgraph-js does not offer graceful way to failover client endpoints/stubs

Issues Dgraph Clients
, ,
1

I’m getting ready to use Dgraph in production, but have run into the following issue:

I currently have a geographically distributed Dgraph cluster over several continents. At any point in time, one or more servers in this cluster may become unavailable (due to upgrade, maintenance, or other considerations). It would be prudent to have a Dgraph client implementation that:

  • Will automatically attempt to send query/mutations to other servers if one server is deemed unavailable, and will only fail if all servers are unreachable.
  • Will take a given domain name (e.g. “consul-dgraph-us-west”) and continuously resolve the name to follow any changes in IP addresses/service availability. This is useful for service discovery and health checking systems such as Consul.
  • Prioritizes certain servers for queries/mutations, and only attempts to contact other servers if the prioritized servers are unavailable. This will reduce round-trip time lag when dealing with various geographically disparate regions.

From what I gather the following code can be used to add multiple endpoints to the Dgraph js client:

const dgraph = require("dgraph-js")
const grpc = require("grpc")
const stub1 = new dgraph.DgraphClientStub(
	"dgraph-alpha1:9080",
	grpc.credentials.createInsecure(),
)
const stub2 = new dgraph.DgraphClientStub(
	"dgraph-alpha2:9080",
	grpc.credentials.createInsecure(),
)
const client = new dgraph.DgraphClient(stub1, stub2)

But from a quick reading of the source code, it seems that the client then uses a round-robin approach to choose which stub to use. If a stub endpoint is unreachable, the given query/mutation fails, and the client does not automatically failover to another stub. Please let me know if this understanding is incorrect.

Thank you!

2

Interesting, are you using a single cluster across continents or they are all individuals clusters per continent? if so, can you share your experience with this? seems complicated to deal. As Dgraph treats each node part of a single structure. And perhaps the idea of upgrading “parts” of the cluster can be dangerous if a single breaking thing is added as a feature.

That’s a kind of experience that others might like to read. For example this experience Transitioning from a Relational DB to a Graph DB

Dgraph does this internally. It talks to all nodes and “keep in touch” to see who he can count on. Also, the Alphas now supports multiple zeros. It means if your main zero is down, it could try to reach others to keep the cluster healthy.

About this, on the client-side. I had commented this internally some time ago. Following my own bias, I think this is an important behave, not just for possible overseas configurations.

My idea was something like

const clientStub = new dgraph.DgraphClientStub(
  ["localhost:9080", 
   "localhost:9081", 
   "localhost:9082", 
   "localhost:9083", 
   "localhost:9084", 
   "localhost:9085"],
  grpc.credentials.createInsecure(),
);

Dgraph does it internally. The first Alpha you hit, will try to reach other Alphas, and the first one who responded will be who serves. If you need to know how it works check this video out https://www.youtube.com/watch?v=Bg4rlmabevM&t=2s

I’m not sure, but he should support it. For example, Liveloader, is able to receive multiple Alphas addresses separated by comma and it will connect to all and send mutations to them. This pattern should be possible in the client javascript.

I mean, you should be able to create a “liveloader” in JS and have the same behavior it has in the golang version. But obviously, it wouldn’t have the same performance.

3

Thank you for your quick reply.

Our project requires high uptime requirements with high levels of data security. We currently have a Scylla cluster in place, but this leaves much to be desired in terms of query complexity. Although, it’s perfect for low-latency queries.

Luckily, our requirements do not place very heavy restrictions on latency for more complex queries. As a result, we decided to place Dgraph nodes over multiple continents, form them into a single cluster, and ignore the latency implications. We use a microservices architecture to collect information from our Pulsar pipeline (as a sink) and then feed the information into Dgraph. This has multiple advantages:

  • We can ensure that once data is written to the Pulsar ensemble, it will eventually make it to Dgraph.
  • We can process any data over many different nodes (deriving relationships, etc. before it’s sent to Dgraph), which we can scale up and down using our microservices architecture.
  • We can continuously retry any task (and if necessary) keep any data stored in Pulsar until a given task can be completed (i.e. circuit breaker capabilities).
  • Tasks can be processed over time and build up in the Pulsar queue/pipeline. This helps with sudden spikes in processing requests.
  • We can then schedule queries to take place on the Dgraph cluster, output them at a later time, and deliver them wherever they need to go.

As a result, we typically use rolling upgrades on our servers. When upgrading, we’d upgrade a single server as a canary, see if it the process works, and rollback if it doesn’t, then continue upgrading other servers. We’d like to do the same with our Dgraph nodes.

From my understanding, Dgraph will serve results from the current leader after connecting to any Alpha. If the current leader is in my geographical region, but I connect to the Alpha in another region, the query would be forwarded back to my current geographic region, resulting in unnecessary latency.

It seems that the individual dgraph.DgraphClientStub object also supports the alter, query, and mutate functions, so it wouldn’t be too much trouble to write our own policy on figuring out which stub to use for queries, along with automatic retries. We also had to write our own policy for the Cassandra js client as well, but it would be nice to see the official client support this to some degree.

Thank you for your input. :slight_smile:

1 Like
4

For anyone else who decides to go down this route, we implemented a quick-and-dirty system to automatically reconnect to other Dgraph nodes to minimize downtime to virtually nothing. We’ll describe the methodology we used.

We resolve a given set of names via the following:

const dns = require('dns')
dns.resolve4(e.endpoint, (err, addrs) => {
    /* implementation here */
})

We then create a client using a stub from a randomly picked server in our region:

const endpoint = _.sample(servers) /* uses lodash */
dgraph_stub_pkg = {
	endpoint,
	stub: new dgraph.DgraphClientStub(
		`${endpoint}:9080`,
		grpc.credentials.createInsecure(),
	),
	in_flight: false, /* used for metrics */
	closing: false /* and clean-up */
}
dgraph_stub_pkg.client =
	new dgraph.DgraphClient(dgraph_stub_pkg.stub)

We then test to see if a given query/mutation fails using the following:

dgraph_op_func(dgraph_stub_pkg.client).then((res) => {
	resolve(res) /*success, resolve promise here*/
}).catch((err) => {
	// Reference: https://github.com/grpc/grpc/blob/master/doc/statuscodes.md
	// 14 => UNAVAILABLE
	if (err && err.code && err.details && err.code === 14) {
		retry(dgraph_op_func) /* retry logic here */
			.then((res) => resolve(res))
			.catch((err) => reject(err))
	} else {
		reject(err) /* some other error */
	}
})

We end up keeping a cache around of temporarily unavailable addresses, and tell the system to retest the connection to those addresses periodically.

We tested the technique by killing random docker instances over and over. Attached is an image of our tests.

image
image2302×694 530 KB

2 Likes