Deleting data of type int causes the cluster to crash


Report a Dgraph Bug

What version of Dgraph are you using?

v20.11.0

Have you tried reproducing the issue with the latest release?

yes

What is the hardware spec (RAM, OS)?

ubuntu18.04(256GB RAM) & mac(16GB RAM)

Steps to reproduce the issue (command/config used to run Dgraph).

  1. Start a cluster, it has 1 zero, 6 alpha, 2 groups, each group of 3 alpha, 1 ratel.
  2. Alter schema
id: int @index(int)  .
followers: [uid] @reverse  .

type twitter_user {
    id
    followers
}

  1. Add a node
{
  set {
    _:user <id> "9999" .
    _:user <dgraph.type> "twitter_user" .
  }
}
  1. Query this node
{
  xx(func: eq(id,9999)) {
    uid
    dgraph.type
    expand(_all_)
  }
}

result:

{
    "xx": [
      {
        "uid": "0x2",
        "dgraph.type": [
          "twitter_user"
        ],
        "id": 9999
      }
    ]
  }
  1. Delete this node
{
  delete {
  	<0x2> * * .
  }
}

Expected behaviour and actual result.

Expected behaviour

Query the node again and return an empty JSON.

Actual result

Three alpha in group 0 crash, output the following log:

panic: interface conversion: interface {} is string, not int64

goroutine 205 [running]:
github.com/dgraph-io/dgraph/tok.IntTokenizer.Tokens(...)
	/root/go/src/github.com/dgraph-io/dgraph/tok/tok.go:194
github.com/dgraph-io/dgraph/tok.BuildTokens(0x1cf6d80, 0xc00079c4d0, 0x21a8900, 0x2cb4708, 0x21a8900, 0x2cb4708, 0xc00079c4d0, 0x0, 0x0)
	/root/go/src/github.com/dgraph-io/dgraph/tok/tok.go:108 +0x63
github.com/dgraph-io/dgraph/worker.generateTokenKeys(0xc00baaa000, 0xc00079c4b0, 0x1, 0x1, 0x2, 0xc00079c400, 0x1, 0x1, 0xc0005fc760)
	/root/go/src/github.com/dgraph-io/dgraph/worker/executor.go:83 +0x237
github.com/dgraph-io/dgraph/worker.generateConflictKeys(0x219f6e0, 0xc000048080, 0xc0007fa440, 0xc00b8928a0)
	/root/go/src/github.com/dgraph-io/dgraph/worker/executor.go:120 +0x47d
github.com/dgraph-io/dgraph/worker.(*executor).processMutationCh(0xc0004089b0, 0x219f6e0, 0xc000048080, 0xc000720f60)
	/root/go/src/github.com/dgraph-io/dgraph/worker/executor.go:221 +0x445
created by github.com/dgraph-io/dgraph/worker.(*executor).getChannel
	/root/go/src/github.com/dgraph-io/dgraph/worker/executor.go:276 +0x130
[Sentry] 2021/01/06 06:24:04 Sending fatal event [271b772d2e3e457fbb96f4fec0c29598] to o318308.ingest.sentry.io project: 1805390
[Sentry] 2021/01/06 06:24:06 Buffer flushed successfully.

reproduce this issue

I started to test this issue on k8s, and later I used my computer to deploy the cluster using docker to reproduce the issue.

Here is my docker compose file.

version: "3.2"

services:
  zero1:
    image: dgraph/dgraph:v20.11.0
    volumes:
      - ./dgraph_data/zero1:/dgraph
    ports:
      - 5980:5080
      - 6980:6080
    networks:
      - dgraph
    healthcheck:
      test: ["CMD", "curl", "-IL", "localhost:6080/state"]
      interval: 10s
      start_period: 10s
      timeout: 5s
      retries: 3
    command: dgraph zero --my=zero1:5080 --replicas 3
  alpha1:
    image: dgraph/dgraph:v20.11.0
    depends_on: 
      zero1:
        condition: service_healthy
    volumes:
      - ./dgraph_data/alpha1:/dgraph
    ports:
      - 8981:8080
      - 9981:9080
    networks:
      - dgraph
    healthcheck:
      test: ["CMD", "curl", "-IL", "localhost:8080"]
      interval: 10s
      start_period: 10s
      timeout: 5s
      retries: 3
    command: dgraph alpha --whitelist=0.0.0.0:255.255.255.255 --my=alpha1:7080 --zero=zero1:5080 --lru_mb=1024 --enable_sentry=false --ludicrous_mode --mutations_nquad_limit=20000000 --query_edge_limit=10000000000 --badger.compression=zstd:5
  alpha2:
    image: dgraph/dgraph:v20.11.0
    depends_on: 
      alpha1:
        condition: service_healthy
    volumes:
      - ./dgraph_data/alpha2:/dgraph
    ports:
      - 8982:8080
      - 9982:9080
    networks:
      - dgraph
    healthcheck:
      test: ["CMD", "curl", "-IL", "localhost:8080"]
      interval: 10s
      start_period: 10s
      timeout: 5s
      retries: 3
    command: dgraph alpha --whitelist=0.0.0.0:255.255.255.255 --my=alpha2:7080 --zero=zero1:5080 --enable_sentry=false  --ludicrous_mode --mutations_nquad_limit=20000000 --query_edge_limit=10000000000 --badger.compression=zstd:5
  alpha3:
    image: dgraph/dgraph:v20.11.0
    depends_on: 
      alpha2:
        condition: service_healthy
    volumes:
      - ./dgraph_data/alpha3:/dgraph
    ports:
      - 8983:8080
      - 9983:9080
    networks:
      - dgraph
    healthcheck:
      test: ["CMD", "curl", "-IL", "localhost:8080"]
      interval: 10s
      start_period: 10s
      timeout: 5s
      retries: 3
    command: dgraph alpha --whitelist=0.0.0.0:255.255.255.255 --my=alpha3:7080 --zero=zero1:5080 --enable_sentry=false --ludicrous_mode --mutations_nquad_limit=20000000 --query_edge_limit=10000000000 --badger.compression=zstd:5
  alpha4:
    image: dgraph/dgraph:v20.11.0
    depends_on: 
      alpha3:
        condition: service_healthy
    volumes:
      - ./dgraph_data/alpha4:/dgraph
    ports:
      - 8984:8080
      - 9984:9080
    networks:
      - dgraph
    healthcheck:
      test: ["CMD", "curl", "-IL", "localhost:8080"]
      interval: 10s
      start_period: 10s
      timeout: 5s
      retries: 3
    command: dgraph alpha --whitelist=0.0.0.0:255.255.255.255 --my=alpha4:7080 --zero=zero1:5080 --enable_sentry=false --ludicrous_mode --mutations_nquad_limit=20000000 --query_edge_limit=10000000000 --badger.compression=zstd:5
  alpha5:
    image: dgraph/dgraph:v20.11.0
    depends_on: 
      alpha4:
        condition: service_healthy
    volumes:
      - ./dgraph_data/alpha5:/dgraph
    ports:
      - 8985:8080
      - 9985:9080
    networks:
      - dgraph
    healthcheck:
      test: ["CMD", "curl", "-IL", "localhost:8080"]
      interval: 10s
      start_period: 10s
      timeout: 5s
      retries: 3
    command: dgraph alpha --whitelist=0.0.0.0:255.255.255.255 --my=alpha5:7080 --zero=zero1:5080 --enable_sentry=false --ludicrous_mode --mutations_nquad_limit=20000000 --query_edge_limit=10000000000 --badger.compression=zstd:5
  alpha6:
    image: dgraph/dgraph:v20.11.0
    depends_on: 
      alpha5:
        condition: service_healthy
    volumes:
      - ./dgraph_data/alpha6:/dgraph
    ports:
      - 8986:8080
      - 9986:9080
    networks:
      - dgraph
    healthcheck:
      test: ["CMD", "curl", "-IL", "localhost:8080"]
      interval: 10s
      start_period: 10s
      timeout: 5s
      retries: 3
    command: dgraph alpha --whitelist=0.0.0.0:255.255.255.255 --my=alpha6:7080 --zero=zero1:5080 --enable_sentry=false --ludicrous_mode --mutations_nquad_limit=20000000 --query_edge_limit=10000000000 --badger.compression=zstd:5
  ratel:
    image: dgraph/dgraph:v20.11.0
    ports:
      - 8900:8000
    networks:
      - dgraph
    healthcheck:
      test: ["CMD", "curl", "-IL", "localhost:8000"]
      interval: 10s
      start_period: 10s
      timeout: 5s
      retries: 3
    command: dgraph-ratel
# volumes:
  # data-volume:
networks:
  dgraph:

Why int type?

Because when I change the id in the schema to string type, this issue will not occur.

id: string @index(term)  .
followers: [uid] @reverse  .

type twitter_user {
    id
    followers
}

Why cluster?

The stand-alone dgraph I deployed will not have this issue.

This looks like an easy to fix crash. I’m accepting this as a bug. @Valdanito we’ll try to fix this soon.

1 Like

Thanks :+1: