Divergent data on Alpha nodes when deleting predicates

Issues Dgraph
, ,
1

Moved from GitHub dgraph/5532

Posted by lukaszlenart:

What version of Dgraph are you using?

Started on 1.2.0, then tested with 1.2.3 and now using 20.03.1

Have you tried reproducing the issue with the latest release?

Yes

What is the hardware spec (RAM, OS)?

K8s cluster on AWS, 3xAlpha, 3xZero, 1 group

Steps to reproduce the issue (command/config used to run Dgraph).

I didn’t try to duplicate this behaviour on some other data set, yet I’m planning to do so

Expected behaviour and actual result.

I’m running Dgraph 20.03.1 and struggling with divergent data on Alpha nodes, I was able narrow the problem to deleted predicates, I mean, when we perform a copy operation in our system, we set assetState to locked and at the end we remove the assetState edge. Yet I was able to filter out those affected nodes in Dgraph, using the same query on two different Alphas, here is the query:

{
  q(func: eq(staticId, "99822edc-febd-472a-8882-f78a0fe88092")) {
    uid
    parentOf @filter (not has(assetState)) {
      uid
      assetState
    }
  }
}

result on one Alpha is

{
  "data": {
    "q": [
      {
        "uid": "0x5",
        "parentOf": [
          {
            "uid": "0xc378"
          },
          {
            "uid": "0xc8b0"
          },
          {
            "uid": "0x50af6"
          },
          {
            "uid": "0x50b46"
          }
        ]
      }
    ]
  },
  "extensions": {
...

and on another

{
  "data": {
    "q": [
      {
        "uid": "0x5",
        "parentOf": [
          {
            "uid": "0xc378"
          },
          {
            "uid": "0xc8b0"
          },
          {
            "uid": "0xcbeb"
          },
          {
            "uid": "0xdad0"
          },
          {
            "uid": "0x50967"
          },
          {
            "uid": "0x5096a"
          },
          {
            "uid": "0x50af6"
          },
          {
            "uid": "0x50b46"
          },
          {
            "uid": "0x50b49"
          }
        ]
      }
    ]
  },
  "extensions": {
...

if we do not remove the assetState but just changing it to submitted, everything works fine.

2

This is somehow related to deleting data. Once I changed implementation in our system to update a node instead removing it, the problem is gone.

Yet we still encountering this problem in other areas of our system and just avoiding delete isn’t an option.