Dgraph cluster cannot init

Issues Dgraph
1

Report a Dgraph Bug

What version of Dgraph are you using?

Dgraph Version l

Kubernetes latest

Have you tried reproducing the issue with the latest release?

Yes

What is the hardware spec (RAM, OS)?

CPU: AMD Opteron 2374 HE (8) @ 2.200GHz (*2)
OS: Ubuntu 21.04 x86_64
Memory:16 GB
4 nodes 1 is master, all running same hardware and OS

Steps to reproduce the issue (command/config used to run Dgraph).

dgraph.yaml - Pastebin.com apply to bare metal cluster with Flannel and MetalLB

Expected behaviour and actual result.

expected:
normal logs
what I got:
I0708 20:02:51.407673      20 pool.go:162] CONNECTING to dgraph-zero-0.dgraph-zero.default.svc.cluster.local:5080
E0708 20:02:52.313929      20 groups.go:1181] Error during SubscribeForUpdates for prefix "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x15dgraph.graphql.schema\x00": Unable to find any servers for group: 1. closer err: <nil>
E0708 20:02:53.315047      20 groups.go:1181] Error during SubscribeForUpdates for prefix "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x15dgraph.graphql.schema\x00": Unable to find any servers for group: 1. closer err: <nil>
E0708 20:02:54.316027      20 groups.go:1181] Error during SubscribeForUpdates for prefix "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x15dgraph.graphql.schema\x00": Unable to find any servers for group: 1. closer err: <nil>
E0708 20:02:55.316793      20 groups.go:1181] Error during SubscribeForUpdates for prefix "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x15dgraph.graphql.schema\x00": Unable to find any servers for group: 1. closer err: <nil>
E0708 20:02:56.313487      20 admin.go:857] namespace: 0. Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
E0708 20:02:56.317744      20 groups.go:1181] Error during SubscribeForUpdates for prefix "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x15dgraph.graphql.schema\x00": Unable to find any servers for group: 1. closer err: <nil>
E0708 20:02:57.318996      20 groups.go:1181] Error during SubscribeForUpdates for prefix "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x15dgraph.graphql.schema\x00": Unable to find any servers for group: 1. closer err: <nil>
E0708 20:02:58.319831      20 groups.go:1181] Error during SubscribeForUpdates for prefix "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x15dgraph.graphql.schema\x00": Unable to find any servers for group: 1. closer err: <nil>
E0708 20:02:59.320796      20 groups.go:1181] Error during SubscribeForUpdates for prefix "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x15dgraph.graphql.schema\x00": Unable to find any servers for group: 1. closer err: <nil>
E0708 20:03:00.322090      20 groups.go:1181] Error during SubscribeForUpdates for prefix "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x15dgraph.graphql.schema\x00": Unable to find any servers for group: 1. closer err: <nil>
E0708 20:03:01.314348      20 admin.go:857] namespace: 0. Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
E0708 20:03:01.322541      20 groups.go:1181] Error during SubscribeForUpdates for prefix "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x15dgraph.graphql.schema\x00": Unable to find any servers for group: 1. closer err: <nil>
I0708 20:03:01.609234      20 pool.go:162] CONNECTING to dgraph-zero-1.dgraph-zero.default.svc.cluster.local:5080
E0708 20:03:02.323184      20 groups.go:1181] Error during SubscribeForUpdates for prefix "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x15dgraph.graphql.schema\x00": Unable to find any servers for group: 1. closer err: <nil>
E0708 20:03:03.324424      20 groups.go:1181] Error during SubscribeForUpdates for prefix "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x15dgraph.graphql.schema\x00": Unable to find any servers for group: 1. closer err: <nil>
E0708 20:03:04.324624      20 groups.go:1181] Error during SubscribeForUpdates for prefix "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x15dgraph.graphql.schema\x00": Unable to find any servers for group: 1. closer err: <nil>
E0708 20:03:05.325497      20 groups.go:1181] Error during SubscribeForUpdates for prefix "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x15dgraph.graphql.schema\x00": Unable to find any servers for group: 1. closer err: <nil>
E0708 20:03:06.315747      20 admin.go:857] namespace: 0. Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
E0708 20:03:06.326249      20 groups.go:1181] Error during SubscribeForUpdates for prefix "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x15dgraph.graphql.schema\x00": Unable to find any servers for group: 1. closer err: <nil>
E0708 20:03:07.327379      20 groups.go:1181] Error during SubscribeForUpdates for prefix "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x15dgraph.graphql.schema\x00": Unable to find any servers for group: 1. closer err: <nil>
E0708 20:03:08.327585      20 groups.go:1181] Error during SubscribeForUpdates for prefix "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x15dgraph.graphql.schema\x00": Unable to find any servers for group: 1. closer err: <nil>
E0708 20:03:09.328575      20 groups.go:1181] Error during SubscribeForUpdates for prefix "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x15dgraph.graphql.schema\x00": Unable to find any servers for group: 1. closer err: <nil>
E0708 20:03:10.329089      20 groups.go:1181] Error during SubscribeForUpdates for prefix "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x15dgraph.graphql.schema\x00": Unable to find any servers for group: 1. closer err: <nil>
E0708 20:03:11.316152      20 admin.go:857] namespace: 0. Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
E0708 20:03:11.329784      20 groups.go:1181] Error during SubscribeForUpdates for prefix "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x15dgraph.graphql.schema\x00": Unable to find any servers for group: 1. closer err: <nil>
W0708 20:03:11.408492      20 pool.go:267] Connection lost with dgraph-zero-0.dgraph-zero.default.svc.cluster.local:5080. Error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing
2

Clean up the whole thing in K8s, make sure all bound paths are clean. Start over. Start from scratch is always good, some typo can happen and we could never know that was it or not.

3

Are you sure that changing 2 lines to LoadBalancer in the public services could have caused this? The cluster did the same thing with the helm installation. Ratel couldn’t ever connect to it, even with the helm installation .

4

Not sure what you mean, I didn’t mention LoadBalancer. I said just to start from scratch.

5

Good news is that it’s not crashing, however now it is not responding to any input
and dgraph-alpha is not responding to any wellness checks.
Checked the logs… same errors

6

I’ll switch to Cilium to see if it was flannel

7

It was flannel