Cannot retrieve posting for UID from list with key. readTs less than minTs

Issues Dgraph
, , ,
1

Moved from GitHub dgraph/5435

Posted by danielmai:

What version of Dgraph are you using?

v20.03.1

Have you tried reproducing the issue with the latest release?

Yes

What is the hardware spec (RAM, OS)?

Ubuntu Linux

Steps to reproduce the issue (command/config used to run Dgraph).

These steps don’t reproduce the issue all the time, but I do see it sometimes.

  1. Run 6-node Dgraph cluster.
  2. Live load 21-million movie data set.
  3. Run a query, e.g.,
       {
  director(func: allofterms(name@en, "steven spielberg")) {
    name@en
    director.film {
      name@en
      initial_release_date
      country {
        name@en
      }
      starring {
        performance.actor {
          name@en
        }
        performance.character {
          name@en
        }
      }
      genre {
        name@en
      }
    }
  }
}
  4. Restart the same Alpha you ran a query on
  5. Run the same query on the restarted Alpha.

Expected behaviour and actual result.

When running step 4, I sometimes get this error

Error Name: t
Message: : cannot find value without language tag from list with key 0000026964000000000000009c41: cannot retrieve posting for UID 18446744073709551615 from list with key 0000026964000000000000009c41: readTs: 61825 less than minTs: 85561 for key: "\x00\x00\x02id\x00\x00\x00\x00\x00\x00\x00\x9cA"
Raw Error:
{
  "name": "t",
  "url": "http://a3c58ff3185ec11eab20b027ea48a6ae-164055270.us-west-2.elb.amazonaws.com:8080/query?timeout=20s&debug=true&ro=true&be=true",
  "errors": [
    {
      "message": ": cannot find value without language tag from list with key 0000026964000000000000009c41: cannot retrieve posting for UID 18446744073709551615 from list with key 0000026964000000000000009c41: readTs: 61825 less than minTs: 85561 for key: \"\\x00\\x00\\x02id\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x9cA\"",
      "extensions": {
        "code": "ErrorInvalidRequest"
      }
    }
  ]
}

After some amount of time (could be seconds or hours), the error goes away and the query works as expected.

2

danielmai commented :

I saw this in test.sh as well.

INFO: Running test for github.com/dgraph-io/dgraph/dgraph/cmd/counter
[Decoder]: Using assembly version of decoder
--- FAIL: TestBestEffortTs (0.67s)
    increment_test.go:157: Initial value: 1
    increment_test.go:58: Last value written by increment in loop: 2
    increment_test.go:131: Last value read by best effort: 1
    increment_test.go:58: Last value written by increment in loop: 3
    increment_test.go:235: 
        	Error Trace:	increment_test.go:235
        	Error:      	Received unexpected error:
        	            	rpc error: code = Unknown desc = : cannot find value without language tag from list with key 00000b636f756e7465722e76616c0000000000000f44ce: cannot retrieve posting for UID 18446744073709551615 from list with key 00000b636f756e7465722e76616c0000000000000f44ce: readTs: 2482 less than minTs: 2483 for key: "\x00\x00\vcounter.val\x00\x00\x00\x00\x00\x00\x0fD\xce"
        	            	while doing query
        	            	github.com/dgraph-io/dgraph/dgraph/cmd/counter.queryCounter
        	            		/home/dmai/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/counter/increment.go:88
        	            	github.com/dgraph-io/dgraph/dgraph/cmd/counter.TestBestEffortTs
        	            		/home/dmai/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/counter/increment_test.go:234
        	            	testing.tRunner
        	            		/usr/local/go/src/testing/testing.go:992
        	            	runtime.goexit
        	            		/usr/local/go/src/runtime/asm_amd64.s:1373
        	Test:       	TestBestEffortTs
3

martinmr commented :

That seems like it could be a temporary error due to rollup. Notice that the timestamps only differ by one., indicating that the rollup is recent. Retrying in this situation would probably help.

4

Willem520 commented :

when I run query, I got the similar result
{"errors":[{"message":": rpc error: code = Unknown desc = readTs: 6 less than minTs: 11 for key: \"\\x00\\x00\\x03gid\\x02\\x00\\v\\xbf?L\\x93\\x1d|/P.\\xb1n\\x81i\u003cA\\x89$\\x12\\xf5\\r\\x1a\\x1e\\xbe\\xd8@\\xa1=D\\x13\\xac59\"","extensions":{"code":"ErrorInvalidRequest"}}],"data":null}
I only can query schema or uid

5

danielmai commented :

@Willem520 Are you running Dgraph with ludicrous mode enabled? And did you see this error after Dgraph Alpha restarted?

I can easily see this when running ludicrous mode since there’s a known bug where write-ahead log snapshots aren’t created when ludicrous mode is on, so an Alpha restart causes a long list of write-ahead log entries to be replayed.

6

Willem520 commented :

@Willem520 Are you running Dgraph with ludicrous mode enabled? And did you see this error after Dgraph Alpha restarted?

I can easily see this when running ludicrous mode since there’s a known bug where write-ahead log snapshots aren’t created when ludicrous mode is on, so an Alpha restart causes a long list of write-ahead log entries to be replayed.

I run Dgraph without ludicrous mode. I got the above result when I bulk a large dataset which cost about 2 hour.

7

ashish-goswami commented :

Hey @Willem520, in which version of Dgraph did you see above issue?

8

Willem520 commented :

Hey @Willem520, in which version of Dgraph did you see above issue?
Hi @ashish-goswami ,I used v1.2.1

9

Fixed by: Use the readTs for reads via LocalCache by manishrjain · Pull Request #6317 · dgraph-io/dgraph · GitHub

10

I get this error, my Dgraph version info like this

Dgraph version   : v20.03.4
Dgraph SHA-256   : ea782f7687004901e2fb4d0a11904417787217ce156104e4f333460966682a86
Commit SHA-1     : 80e6c199e
Commit timestamp : 2020-07-23 11:17:00 -0700
Branch           : HEAD
Go version       : go1.14.4

Do I need to upgrade to solve this problem?

11

Hey @xxm404, the fix for this issue is yet to be released. We’ll be releasing v20.03.6 this month and that will have the fix.

If you’re keen to try it now, you can use v20.03.6-rc1 tag. Please be aware that this isn’t a released version and it could have some bugs.

12

@ibrahim Thanks for the reply. I will update to fix this problem when v20.03.6 releasing.

1 Like