Load Balancing via a headless service

Users Dgraph
#1

I noticed that when I start multiple alphas, my application (conencting via Dgo) only puts load on a single alpha. Having ready the very useful explanation at Load balancing and scaling long-lived connections in Kubernetes I understand this is due to the long-lived connections used by gRPC/HTTP2, i.e. my app connects to a random alpha pod and will continue to use that one pod for all queries.

It seems that a headless service would fix this, as gRPC can natively connect to all pods of a headless service simultaneously, maintain/update the connection pool, and distribute requests across these connections.

There is a headless alpha service by default in your Helm charts, but this is for internal connections only I think. I would therefore like to make the default (9080) service optionally headless also. Alternative is to use a service mesh like LinkerD, but that comes with other considerations and overheads.

Please let me know if this is a sensible choice and I can submit a PR to your Helm repo to make this option templated. I suspect this would also help the liveloader tool as that uses gRPC?

Thanks for any help, Rob

#2

Ping @dmai and @joaquin to take a look.

#3

By default, a Kubernetes LoadBalancer service doesn’t do gRPC load balancing. You’ll need to use a different proxy for this (e.g., Envoy, Nginx, Traefik) that supports gRPC load balancing.

In Helm chart, we have examples of setting up Ingress for Nginx: charts/ingress-nginx.yaml at master · dgraph-io/charts · GitHub

And you can see this how-to on load balancing queries with Nginx directly: https://dgraph.io/docs/howto/load-balancing-nginx

#4

@dmai My app runs inside the k8s cluster also, does it make sense to route this traffic through an ingress proxy to DGraph? Seeing the arguments here client side load balancing is more appropriate to my set up I think, especially because performance is critical.

Via a headless service I beleive the load would also be balanced without ingress, though the destination addresses do not update automatically (e.g. see https://medium.com/google-cloud/loadbalancing-grpc-for-kubernetes-cluster-services-3ba9a8d8fc03). GRPC xDS loadbalancing could be another option for best of both worlds.