Regression: v21.03 crashes on query with orderdesc

Report a Dgraph Bug

What version of Dgraph are you using?

All of dgraph/dgraph:master, dgraph/dgraph:v21.03.0 and dgraph/dgraph:v21.03.0-12-g44005bc82 generate error, but dgraph/dgraph:v20.11.2 works OK

Dgraph Version v21.03
$ dgraph version
 
PASTE YOUR RESULTS HERE

Have you tried reproducing the issue with the latest release?

Yes

What is the hardware spec (RAM, OS)?

archlinux 32GB RAM

Steps to reproduce the issue (command/config used to run Dgraph).

I do automatic testing (do new schema, then importing new data - starting from scratch, so no garbage data) which do work for v20.11 but crashed v21.03. Unfortunately I do not have isolated small test case, but hopefully following description is enough.

Following query crashes v21.03:

queryGardenTicket(func: uid(GardenTicketRoot), orderdesc: GardenTicket.created_at) @cascade(GardenTicket.user) {
  uid
}

while the same query without orderdesc works OK:

queryGardenTicket(func: uid(GardenTicketRoot)) @cascade(GardenTicket.user) {
    uid
}

when there is a record as a result.
There is no crash when there is no result, or when using v20.11

Expected behaviour and actual result.

The dgraph alpha crashes with the following stack trace:

dgraph_1         | 2021/05/20 00:52:25 1 0
dgraph_1         | github.com/dgraph-io/dgraph/x.AssertTruef
dgraph_1         | 	/ext-go/1/src/github.com/dgraph-io/dgraph/x/error.go:107
dgraph_1         | github.com/dgraph-io/dgraph/worker.intersectBucket
dgraph_1         | 	/ext-go/1/src/github.com/dgraph-io/dgraph/worker/sort.go:684
dgraph_1         | github.com/dgraph-io/dgraph/worker.sortWithIndex
dgraph_1         | 	/ext-go/1/src/github.com/dgraph-io/dgraph/worker/sort.go:286
dgraph_1         | github.com/dgraph-io/dgraph/worker.processSort.func2
dgraph_1         | 	/ext-go/1/src/github.com/dgraph-io/dgraph/worker/sort.go:515
dgraph_1         | runtime.goexit
dgraph_1         | 	/usr/local/go/src/runtime/asm_amd64.s:1371

Expected result is no crash, as it worked for v20.11

1 Like

Do you have a schema for me to reproduce?

Yes, I managed to extract the case. Graphql Schema:

enum GardenTicketStatus {
  new
  reserved
  active
  cancelled
  past
  unavailable
  error
}

"User object"
type User @secret(field:"password") 
@auth(
  query:{ or:[
   {rule: "{$role_admin:{eq:\"true\"}}" },
   {rule: "{$internal:{eq:\"true\"}}" },
   {rule:
    """
      query($USER: String!) {
        queryUser(filter:{login:{eq:$USER}}) {
          login
        }
      }
    """
   }
  ]},
  update:{ or:[
   {rule: "{$role_admin:{eq:\"true\"}}" },
   {rule: "{$internal:{eq:\"true\"}}" },
   {rule:
    """
      query($USER: String!) {
        queryUser(filter:{login:{eq:$USER}}) {
          login
        }
      }
    """
   }
  ]},
  delete:{ or:[
   {rule: "{$role_admin:{eq:\"true\"}}" },
   {rule: "{$internal:{eq:\"true\"}}" },
  ]}
)
{
  login: String! @id @search(by: [hash])

  first_name: String! @search(by: [hash])
  last_name: String! @search(by: [hash])
  full_name: String @lambda
  mobile_phone: String @search(by: [hash])

  created_at: DateTime! @search
  modified_at: DateTime! @search

  garden_tickets: [GardenTicket]
}
type GardenTicket @withSubscription
 @auth(
  query:{ or:[
   {rule:
    """
      query($USER: String!) {
        queryGardenTicket @cascade(fields:["user"]){
          user (filter:{login:{eq:$USER}}) {login}
        }
      }
    """
   },
   {rule: "{$role_admin:{eq:\"true\"}}" }
  ]}
)

 {
  id: ID!
  date: String!
  start_hour: String! @search(by: [hash])
  end_hour: String! @search(by: [hash])
  valid_from: DateTime! @search
  valid_to: DateTime! @search
  created_at: DateTime! @search
  user: User! @hasInverse(field:garden_tickets)
  status: GardenTicketStatus! @search(by: [hash])
}

# Dgraph.Authorization {"VerificationKey":"secret","Header":"X-Auth-Token","Namespace":"mydomain.com","Algo":"HS256","Audience":["garden"]}

Sample data:

mutation ($now:DateTime!="2021-05-05T10:00:00Z") {

  addUser(input:[
    {login:"admin",password:"admin1",
      created_at:"2021-01-01T00:00:00Z",
      modified_at:"2021-01-01T00:00:00Z",
      first_name:"Admin",
      last_name:"Admin",
    },
    {login:"john",password:"secret",
      created_at:"2021-01-01T00:00:00Z",
      modified_at:"2021-01-01T00:00:00Z",
      first_name:"John",
      last_name:"Doe",
    },
  ]) {numUids}

  addGardenTicket(input:{
    date:"2021-05-05",
    start_hour:"12:00",
    end_hour:"13:00",
    status:reserved,
    created_at:$now,
    valid_to:$now,
    valid_from:$now,
    user:{login:"john"}
  }) {
    numUids
    gardenTicket {id date created_at status start_hour user {login first_name last_name}}
  }

}

Failing DQL query:

 query {
   queryGardenTicket(func: uid(GardenTicketRoot), orderdesc: GardenTicket.created_at) @cascade(GardenTicket.user) {
   uid
    GardenTicket.user  {uid User.login}
   }
   GardenTicketRoot as var(func: uid(GardenTicket_6))
   GardenTicket_6 as var(func: type(GardenTicket)) @filter((eq(GardenTicket.status, "new", "reserved", "active") AND ge(GardenTicket.valid_to, "2020-05-20T02:28:52+02:00")))
   var(func: uid(GardenTicketRoot)) {
     User_2 as GardenTicket.user @filter(eq(User.login, "john"))
   }
   User_1 as var(func: uid(User_2)) @filter(uid(User_Auth5))
   var(func: uid(User_1)) {
    User.email
   }
   User_Auth5 as var(func: uid(User_2)) @filter(eq(User.login, "john")) @cascade {
     User.login : User.login
   }   
 }

Not failing DQL query:

 query {
   queryGardenTicket(func: uid(GardenTicketRoot))  @cascade(GardenTicket.user) {
   uid
    GardenTicket.user  {uid User.login}
   }
   GardenTicketRoot as var(func: uid(GardenTicket_6))
   GardenTicket_6 as var(func: type(GardenTicket)) @filter((eq(GardenTicket.status, "new", "reserved", "active") AND ge(GardenTicket.valid_to, "2020-05-20T02:28:52+02:00")))
   var(func: uid(GardenTicketRoot)) {
     User_2 as GardenTicket.user @filter(eq(User.login, "john"))
   }
   User_1 as var(func: uid(User_2)) @filter(uid(User_Auth5))
   var(func: uid(User_1)) {
    User.email
   }
   User_Auth5 as var(func: uid(User_2)) @filter(eq(User.login, "john")) @cascade {
     User.login : User.login
   }   
 }

Note: I am using JWT with payload "mydomain.com": {"role_admin":"true"} so @auth rule is satisfied. Not sure if it matters.

I have accepted this as a bug. @hardik fyi

1 Like

I ran into this same issue, and came up with a fairly minimal example to reproduce the crash:

schema

<category.name>: string .
<category.item>: [uid] .
<item.name>: string @index(exact) .

mutation

{
  set {
    _:category <category.name> "category" .
    _:item1 <item.name> "item 1" .
    _:item2 <item.name> "item 2" .
    _:category <category.item> _:item1 .
    _:category <category.item> _:item2 .
  }
}

query

query ItemsInCategory(
  $categoryUID: string
) {
  itemsInCategory(func: uid($categoryUID)) @cascade {
    category.item (orderdesc: item.name) {
      uid
    }
  }
}

To prevent the crash, you can

  • remove @cascade from the query
  • remove (orderdesc: item.name) from the query
  • remove @index(exact) from item.name in the schema

I hope this might help anyone looking into solving this issue.

1 Like

I also run into the same issue with that minimal graphql setup:

schema:

type Foo {
  id:   ID!
  date: DateTime! @search(by: [hour])
  bar:  Bar!      @hasInverse(field: foo)
}

type Bar {
  id:  ID!
  foo: Foo! @hasInverse(field: bar)
}

Inject some data:

mutation addFooAndBar {
  addBar(input: {foo: {date: "2021-05-27T17:04:30Z"}}) {
    bar {
      id
      foo {
        id
        date
      }
    }
    numUids
  }
}

query that crash alpha:

query MyQuery {
  queryFoo(order: {asc: date})@cascade(fields:["bar"]) {
    bar {
      id
    }
  }
}

I used the last standalone docker image 21.03:

Dgraph version   : v21.03.0
Dgraph codename  : rocket
Dgraph SHA-256   : b4e4c77011e2938e9da197395dbce91d0c6ebb83d383b190f5b70201836a773f
Commit SHA-1     : a77bbe8ae
Commit timestamp : 2021-04-07 21:36:38 +0530
Branch           : HEAD
Go version       : go1.16.2
jemalloc enabled : true

For Dgraph official documentation, visit https://dgraph.io/docs.
For discussions about Dgraph     , visit https://discuss.dgraph.io.
For fully-managed Dgraph Cloud   , visit https://dgraph.io/cloud.

Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2021 Dgraph Labs, Inc.
1 Like

@chewxy any update on that one?
Can we follow the bug you opened on a bug tracker?

I tried that on https://cloud.dgraph.io/ as it moved on 21.03 and the error seems to be there too.

I got a 502 error with the failing query:

$ curl -v 'https://withered-wave.eu-central-1.aws.cloud.dgraph.io/graphql' \
  -X POST \
  -H 'content-type: application/json' \
  --data '{
    "query": "{ queryFoo(order: {asc: date})@cascade(fields:[\"bar\"]) { bar { id } } }"
  }'
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 3.65.76.212:443...
* Connected to withered-wave.eu-central-1.aws.cloud.dgraph.io (3.65.76.212) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:

*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.eu-central-1.aws.cloud.dgraph.io
*  start date: Apr 14 12:35:08 2021 GMT
*  expire date: Jul 13 12:35:08 2021 GMT
*  subjectAltName: host "withered-wave.eu-central-1.aws.cloud.dgraph.io" matched cert's "*.eu-central-1.aws.cloud.dgraph.io"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55afe4163970)
> POST /graphql HTTP/2
> Host: withered-wave.eu-central-1.aws.cloud.dgraph.io
> user-agent: curl/7.77.0
> accept: */*
> content-type: application/json
> content-length: 94
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* We are completely uploaded and fine
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 502
< server: Caddy
< content-length: 0
< date: Fri, 04 Jun 2021 07:15:46 GMT
<
* Connection #0 to host withered-wave.eu-central-1.aws.cloud.dgraph.io left intact

a working query returns a 200:

$ curl -v 'https://withered-wave.eu-central-1.aws.cloud.dgraph.io/graphql' \
  -X POST \
  -H 'content-type: application/json' \
  --data '{
    "query": "{ queryFoo(order: {asc: date}) { bar { id } } }"
  }'
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 3.64.168.155:443...
* Connected to withered-wave.eu-central-1.aws.cloud.dgraph.io (3.64.168.155) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.eu-central-1.aws.cloud.dgraph.io
*  start date: Apr 14 12:35:08 2021 GMT
*  expire date: Jul 13 12:35:08 2021 GMT
*  subjectAltName: host "withered-wave.eu-central-1.aws.cloud.dgraph.io" matched cert's "*.eu-central-1.aws.cloud.dgraph.io"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55ffa9b68970)
> POST /graphql HTTP/2
> Host: withered-wave.eu-central-1.aws.cloud.dgraph.io
> user-agent: curl/7.77.0
> accept: */*
> content-type: application/json
> content-length: 68
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* We are completely uploaded and fine
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 200
< access-control-allow-credentials: true
< access-control-allow-headers: X-Dgraph-AccessToken, X-Dgraph-AuthToken, Content-Type, Content-Length, Accept-Encoding, Cache-Control, X-CSRF-Token, X-Auth-Token, X-Requested-With, sentry-trace, DG-Auth
< access-control-allow-methods: POST, OPTIONS
< access-control-allow-origin: *
< content-type: application/json
< date: Fri, 04 Jun 2021 07:17:54 GMT
< graphql-toucheduids: 3
< server: Caddy
< content-length: 437
<
* Connection #0 to host withered-wave.eu-central-1.aws.cloud.dgraph.io left intact
{"data":{"queryFoo":[{"bar":{"id":"0x9c4f5c1"}}]},"extensions":{"touched_uids":3,"tracing":{"version":1,"startTime":"2021-06-04T07:17:54.147804484Z","endTime":"2021-06-04T07:17:54.152054709Z","duration":4250235,"execution":{"resolvers":[{"path":["queryFoo"],"parentType":"Query","fieldName":"queryFoo","returnType":"[Foo]","startOffset":105135,"duration":4140283,"dgraph":[{"label":"query","startOffset":143985,"duration":4099697}]}]}}}}%  
1 Like

Hi @doude there’s no progress on this yet. @hardik can give a clearer picture why

Isn’t this kind of blocker for 21.03? You can kill the server with a simple query :frowning: Any suggested workaround?

3 Likes

@hardik Any word on this?

This issue has been fixed on Release/21.03 branch, so can you guys try it out and see if it helps? Also the fix would be available in 21.03.1 and will be patched in cloud soon.
cc: @minhaj

1 Like

Thanks! Is there a docker image released with this patch?
Edit: works with dgraph/dgraph:master

Works for me, thanks!