Strange behavior of dgraph cluster in kubernetes

Users Dgraph
1

Greetings!

The problem summary%

We run a Dgraph cluster in Kubernetes. 3 alpha, 1 zero, –replicas 3.

Predicate writes appear to succeed (no client‑side write errors), but subsequent reads do not return the predicate data. Only object uids.

The schema already defines the relevant predicates and types, no errors in strict mode.

Persistent volumes are mounted and healthy; network connectivity between pods is OK.

No errors in alpha or zero logs.

The /state of alpha nodes returns one group with 3 members

Same configuration (schema, queries, etc) running locally with docker-compose works perfectly fine.

Any help would be greatly appreciated.

2

My first reaction was that a schema is not applied, as the “uid but no predicates” issue points to that. But as you say, the schema is applied.

First steps I’d take:

  1. Can you query for predicates: has(predicate) {uid predicate}, for instance
  2. I’d turn on verbose logging in one of the alphas (-v=3) to check for anything unusual
3

has(predicate) {uid predicate} returns nothing

it seems to be related to the infrastructure somehow, another thing noticed - “lastUpdate”: “0” in the states of the alphas (all except leader)

is there a way to check network connectivity between alpha nodes except telnet, which seems to work fine?

4

uid()is an identity function. It will always show that any uid exists (if allocated by zero). So it is no indicator for a successful mutation.

5

In this case, you entered an actual predicate name that exists in your schema, right? Not the value “predicate"

In the logs of every Alpha, you should see something like “Connected to zero” and an assigned group/id. If you only see retries to connect to Zero, they’re not joined.

6

@vnium,

Note that I wasn’t using the uid function, rather the uid predicate. The full query would be something like this

{
find_predicate(func: has(name)) { uid name }
}

…if the predicate name exists in the graph, it will be associated to one or more uids.

7

My comment was in regards to the quoted text from @dmitry. You are right that for the predicate query it has no impact.

8

Yes, i have used correct predicates from our schema.

And those queries worked perfectly on our local cluster, so the problem seems to be kubernetes related.

9

Right, if you’re not seeing “Connected to zero” in the alphas, then it’s a connection problem within your k8s setup.

Have a look at the k8s deployment docs if you haven’t. Also this forum has lots of discussions around k8s deployments. Also, GPTv5 is pretty adept at k8s configuration and troubleshooting steps.