Query based on eq() stopped working for one predicate

Users Dgraph
1

We are running Dgraph 1.2 on a k8s cluster and the below query worked few hours ago and not it stopped.

{
  result(func: eq(assetState,"submitted"), first: 10) @filter(eq(nodeType,"rule")) {
    uid 
    staticId
    nodeType
    name
    assetState
  }
}

assetState and nodeType are string predicates with hash index, these are enums tbh.

When I switch filters, it works

{
  result(func: eq(nodeType,"rule"), first: 10) @filter(eq(assetState,"submitted")) {
    uid 
    staticId
    nodeType
    name
    assetState
  }
}

What’s wrong here?

Thanks in advance
Lukasz

2

Restarting (removing) Alpha pods helped with eq(nodeType,"rule") but it still doesn’t work for eq(nodeType,"variable") - this is a show stopper tbh :\

3

And now is back again, eq(nodeType,"rule") doesn’t work again - it started happening after few hours from restart

4

can you share details about specs and configs of your cluster?

5

We use the official Dgraph Chart (an older version which was copied from https://github.com/dgraph-io/dgraph/tree/master/contrib/config/kubernetes/helm) and we have 3 Alphas, 3 Zeros - all the same configuration in three clusters, below is our values.yaml.

Does using an older Chart can be a problem?

dgraph:
  alpha:
    podAnnotations:
      prometheus.io/collectionInterval: 2s
      prometheus.io/path: /debug/prometheus_metrics
      prometheus.io/port: "8080"
      prometheus.io/scrape: "true"
    resources:
      requests:
        cpu: 100m
        memory: 200Mi
  fullnameOverride: in-graph-engine
  zero:
    resources:
      requests:
        cpu: 100m
        memory: 200Mi
6

Feels like it was moved, there’s nothing in that link. Also, the specs are important.

7

I think we are sing this but 5 months older - is this the case?

8

@MichelDiz any hints?

9

I’ll get back to you as soon as I can.

10

I exported and re-imported data but the problem persisted. I dropped an index for that predicate and it started to work. And again I defined the index of type hash and it doesn’t work.

I know that you have added support for @filter on non-indexed predicates in 1.2 but it looks like it’s buggy :frowning:

11

And one more thing, replacing eq with regexp like below solves the problem, but this is rather a hack

{
  result(func: eq(assetState,"submitted"), first: 10) @filter(regexp(nodeType,/^rule$/i)) {
    uid 
    staticId
    nodeType
    name
    assetState
  }
}
12

Hey, I need specs.

Maybe dataset samples or even a mirror of your schema. So I can try to create similar samples and test it on my end.

One thing that is worth mentioning is that all Dgraph features have a battery of tests that they need to do to pass. Overall we caught 70% of the bugs in the tests. Only bugs are almost impossible to detect via tests passes. Or not predicted usage by dev core.

So, I need to have a way to reproduce in different contexts. But first I need to understand your context. So that I can compare with mine.

If I can’t reproduce it. It is hard to tell that this is a Dgraph issue.

13

I did again export/wipe out the cluster/import and now it looks ok, so maybe in my previous attempt to the same I made a mistake and forgot to cleanup alpha’s/zero’s volume or so. We are going to monitor this issue and also I will try re-import those data locally and verify if I can reproduce the issue.

Thanks a lot for your support!

1 Like
14

Hi, I am having a similar issue since last upgrade of the dgraph/standalone docker image. I don’t have a clue what’s happening, but suddenly eq stops working in my simple example:

  • schema has one Type “State” referencing itself:
state: string @index(hash) .
states: [uid] @count .

type State {
  state
  states
}
  • run a loop generating 30k nodes one by one, each referencing between 0 and 250 other State nodes
  • while loop is running executing this query in Ratel finds the start node, and using it’s UID in the test query finds the same node
start(func: type(State)) @filter(eq(state, "start")) {
    uid
    state
    count(states)
  }
    
  test(func: uid(<ABOVE START UID>)) {
    uid
    state
    count(states)
  }

Result:
{
  "data": {
    "start": [
     {
        "uid": "0x1",
        "state": "start",
        "count(states)": 65
      }
    ],
    "test": [
      {
        "uid": "0x1",
        "state": "start",
        "count(states)": 65
      }
    ]

At some point when executing this query again only the test block finds the node while the start block suddenly can’t find any node anymore. Even if the test block clearly shows the searched node exists with the exact state string, the start block stays empty.

I’ve spent way too much time checking my really simple saving loop here where I do not manipulate schema but only save above type with references.

Hope this helps, this drove me crazy for last two days…

15

And back, it started happening again after a few hours.

Not sure what kind of specs do you need but here is a excerpt of our schema:

...
<nodeType>: string @index(exact) .
<assetState>: string @index(exact) .
<staticId>: string @index(exact) .
...

nodeType can be one of the following:

...
layout
content
variable
rule
...

assetState can be one of the following:

archived
submitted
draft

staticId is an UUID, all the predicates form one type - Asset

The following query doesn’t work (an empty result is returned):

{
  result(func: has(staticId)) @filter(eq(nodeType,"variable")) {
    uid
  }
}

where this works:

{
  result(func: has(staticId)) @filter(regexp(nodeType,/^variable$/)) {
    uid
  }
}

One important note: this doesn’t happen on the very beginning when I started a new cluster and imported data. It will start breaking after a few hours (3-5h).

Just checked our second cluster and I discovered the same problem, these two affected clusters are running a significant larger number of data, ~ 500MB, there are 2758 (2558 on the second cluster) of nodeType predicates, compared to 294 in our dev cluster.

Thanks in advance
Lukasz

16

One more important thing, that query based on @filter(eq(nodeType,"..")) doesn’t work for rule but works for variable - just a different value.

17

This seems to be the related issued here:

18

the proposed workaround from above issue isn’t working for my simple setup:

  • start with a clean dgraph/standalone:latest docker image, no schema setup yet
  • using the go client create 30k nodes all with same dgraph.type one by one, each referencing between 0 and 250 of the other nodes
  • run the schema update adding the single type and a hash index to a string field
  • the log shows the successful schema update: Done schema update predicate:“state” value_type:STRING directive:INDEX tokenizer:“hash”

But still, searching for a node using eq(state, “some”) doesn’t yield any results.
I’m not using Datetime tokenizer in my schema and going through the dgraph logs there is no error or warning at all.

19

After adding new node with edge nodeType set to variable the query from above that was not returning anything started to show correct results:

{
  result(func: has(staticId)) @filter(eq(nodeType,"variable")) {
    uid
  }
}
20

just noticed the same, query starts working after saving the same type of node with the predicate in question again…