# Cannot retrieve posting for UID from list with key. readTs less than minTs

**URL:** <https://discuss.dgraph.io/t/cannot-retrieve-posting-for-uid-from-list-with-key-readts-less-than-mints/8533>\
**Category:** Dgraph\
**Tags:** dgraph, status:accepted, kind:bug, area:usability\
**Created:** [May 13, 2020, 3:29pm UTC](https://discuss.dgraph.io/t/cannot-retrieve-posting-for-uid-from-list-with-key-readts-less-than-mints/8533 "2020-05-13T15:29:06Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![diggy](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/diggy/32/3666_2.png) [@diggy](https://discuss.dgraph.io/u/diggy)\
**Post date:** [May 13, 2020, 3:29pm UTC](https://discuss.dgraph.io/t/cannot-retrieve-posting-for-uid-from-list-with-key-readts-less-than-mints/8533/1 "2020-05-13T15:29:06Z")

</div>

**Moved from GitHub [dgraph/5435](https://github.com/dgraph-io/dgraph/issues/5435)**

_Posted by_ [danielmai](https://github.com/danielmai):

### What version of Dgraph are you using?

v20.03.1

### Have you tried reproducing the issue with the latest release?

Yes

### What is the hardware spec (RAM, OS)?

Ubuntu Linux

### Steps to reproduce the issue (command/config used to run Dgraph).

These steps don’t reproduce the issue all the time, but I do see it sometimes.

1. Run 6-node Dgraph cluster.
2. Live load 21-million movie data set.
3. Run a query, e.g.,

```auto
   {
  director(func: allofterms(name@en, "steven spielberg")) {
    name@en
    director.film {
      name@en
      initial_release_date
      country {
        name@en
      }
      starring {
        performance.actor {
          name@en
        }
        performance.character {
          name@en
        }
      }
      genre {
        name@en
      }
    }
  }
}

```

4. Restart the same Alpha you ran a query on
5. Run the same query on the restarted Alpha.

### Expected behaviour and actual result.

When running step 4, I sometimes get this error

```auto
Error Name: t
Message: : cannot find value without language tag from list with key 0000026964000000000000009c41: cannot retrieve posting for UID 18446744073709551615 from list with key 0000026964000000000000009c41: readTs: 61825 less than minTs: 85561 for key: "\x00\x00\x02id\x00\x00\x00\x00\x00\x00\x00\x9cA"
Raw Error:
{
  "name": "t",
  "url": "http://a3c58ff3185ec11eab20b027ea48a6ae-164055270.us-west-2.elb.amazonaws.com:8080/query?timeout=20s&debug=true&ro=true&be=true",
  "errors": [
    {
      "message": ": cannot find value without language tag from list with key 0000026964000000000000009c41: cannot retrieve posting for UID 18446744073709551615 from list with key 0000026964000000000000009c41: readTs: 61825 less than minTs: 85561 for key: \"\\x00\\x00\\x02id\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x9cA\"",
      "extensions": {
        "code": "ErrorInvalidRequest"
      }
    }
  ]
}

```

After some amount of time (could be seconds or hours), the error goes away and the query works as expected.

---

<div class="post-metadata">

**Author:** ![diggy](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/diggy/32/3666_2.png) [@diggy](https://discuss.dgraph.io/u/diggy)\
**Post date:** [May 28, 2020, 10:10pm UTC](https://discuss.dgraph.io/t/cannot-retrieve-posting-for-uid-from-list-with-key-readts-less-than-mints/8533/2 "2020-05-28T22:10:37Z")

</div>

[danielmai](https://github.com/danielmai) _commented_ :

I saw this in test.sh as well.

```auto
INFO: Running test for github.com/dgraph-io/dgraph/dgraph/cmd/counter
[Decoder]: Using assembly version of decoder
--- FAIL: TestBestEffortTs (0.67s)
    increment_test.go:157: Initial value: 1
    increment_test.go:58: Last value written by increment in loop: 2
    increment_test.go:131: Last value read by best effort: 1
    increment_test.go:58: Last value written by increment in loop: 3
    increment_test.go:235: 
        	Error Trace:	increment_test.go:235
        	Error: Received unexpected error:
        	            	rpc error: code = Unknown desc = : cannot find value without language tag from list with key 00000b636f756e7465722e76616c0000000000000f44ce: cannot retrieve posting for UID 18446744073709551615 from list with key 00000b636f756e7465722e76616c0000000000000f44ce: readTs: 2482 less than minTs: 2483 for key: "\x00\x00\vcounter.val\x00\x00\x00\x00\x00\x00\x0fD\xce"
        	            	while doing query
        	            	github.com/dgraph-io/dgraph/dgraph/cmd/counter.queryCounter
        	            		/home/dmai/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/counter/increment.go:88
        	            	github.com/dgraph-io/dgraph/dgraph/cmd/counter.TestBestEffortTs
        	            		/home/dmai/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/counter/increment_test.go:234
        	            	testing.tRunner
        	            		/usr/local/go/src/testing/testing.go:992
        	            	runtime.goexit
        	            		/usr/local/go/src/runtime/asm_amd64.s:1373
        	Test: TestBestEffortTs

```

---

<div class="post-metadata">

**Author:** ![diggy](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/diggy/32/3666_2.png) [@diggy](https://discuss.dgraph.io/u/diggy)\
**Post date:** [May 28, 2020, 11:54pm UTC](https://discuss.dgraph.io/t/cannot-retrieve-posting-for-uid-from-list-with-key-readts-less-than-mints/8533/3 "2020-05-28T23:54:57Z")

</div>

[martinmr](https://github.com/martinmr) _commented_ :

That seems like it could be a temporary error due to rollup. Notice that the timestamps only differ by one., indicating that the rollup is recent. Retrying in this situation would probably help.

---

<div class="post-metadata">

**Author:** ![diggy](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/diggy/32/3666_2.png) [@diggy](https://discuss.dgraph.io/u/diggy)\
**Post date:** [May 29, 2020, 9:56am UTC](https://discuss.dgraph.io/t/cannot-retrieve-posting-for-uid-from-list-with-key-readts-less-than-mints/8533/4 "2020-05-29T09:56:53Z")

</div>

[Willem520](https://github.com/Willem520) _commented_ :

when I run query, I got the similar result  
`{"errors":[{"message":": rpc error: code = Unknown desc = readTs: 6 less than minTs: 11 for key: \"\\x00\\x00\\x03gid\\x02\\x00\\v\\xbf?L\\x93\\x1d|/P.\\xb1n\\x81i\u003cA\\x89$\\x12\\xf5\\r\\x1a\\x1e\\xbe\\xd8@\\xa1=D\\x13\\xac59\"","extensions":{"code":"ErrorInvalidRequest"}}],"data":null}`  
I only can query schema or uid

---

<div class="post-metadata">

**Author:** ![diggy](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/diggy/32/3666_2.png) [@diggy](https://discuss.dgraph.io/u/diggy)\
**Post date:** [June 10, 2020, 6:02am UTC](https://discuss.dgraph.io/t/cannot-retrieve-posting-for-uid-from-list-with-key-readts-less-than-mints/8533/5 "2020-06-10T06:02:17Z")

</div>

[danielmai](https://github.com/danielmai) _commented_ :

@Willem520 Are you running Dgraph with ludicrous mode enabled? And did you see this error _after_ Dgraph Alpha restarted?

I can easily see this when running ludicrous mode since there’s a known bug where write-ahead log snapshots aren’t created when ludicrous mode is on, so an Alpha restart causes a long list of write-ahead log entries to be replayed.

---

<div class="post-metadata">

**Author:** ![diggy](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/diggy/32/3666_2.png) [@diggy](https://discuss.dgraph.io/u/diggy)\
**Post date:** [June 10, 2020, 7:04am UTC](https://discuss.dgraph.io/t/cannot-retrieve-posting-for-uid-from-list-with-key-readts-less-than-mints/8533/6 "2020-06-10T07:04:44Z")

</div>

[Willem520](https://github.com/Willem520) _commented_ :

> @Willem520 Are you running Dgraph with ludicrous mode enabled? And did you see this error _after_ Dgraph Alpha restarted?
> 
> I can easily see this when running ludicrous mode since there’s a known bug where write-ahead log snapshots aren’t created when ludicrous mode is on, so an Alpha restart causes a long list of write-ahead log entries to be replayed.

I run Dgraph without ludicrous mode. I got the above result when I bulk a large dataset which cost about 2 hour.

---

<div class="post-metadata">

**Author:** ![diggy](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/diggy/32/3666_2.png) [@diggy](https://discuss.dgraph.io/u/diggy)\
**Post date:** [June 10, 2020, 8:10am UTC](https://discuss.dgraph.io/t/cannot-retrieve-posting-for-uid-from-list-with-key-readts-less-than-mints/8533/7 "2020-06-10T08:10:41Z")

</div>

[ashish-goswami](https://github.com/ashish-goswami) _commented_ :

Hey @Willem520, in which version of Dgraph did you see above issue?

---

<div class="post-metadata">

**Author:** ![diggy](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/diggy/32/3666_2.png) [@diggy](https://discuss.dgraph.io/u/diggy)\
**Post date:** [June 11, 2020, 8:33am UTC](https://discuss.dgraph.io/t/cannot-retrieve-posting-for-uid-from-list-with-key-readts-less-than-mints/8533/8 "2020-06-11T08:33:34Z")

</div>

[Willem520](https://github.com/Willem520) _commented_ :

> Hey @Willem520, in which version of Dgraph did you see above issue?  
> Hi @ashish-goswami ,I used v1.2.1

---

<div class="post-metadata">

**Author:** ![LGalatin](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/lgalatin/32/2829_2.png) [@LGalatin](https://discuss.dgraph.io/u/LGalatin)\
**Post date:** [September 30, 2020, 8:55pm UTC](https://discuss.dgraph.io/t/cannot-retrieve-posting-for-uid-from-list-with-key-readts-less-than-mints/8533/9 "2020-09-30T20:55:47Z")

</div>

Fixed by: [Use the readTs for reads via LocalCache by manishrjain · Pull Request #6317 · dgraph-io/dgraph · GitHub](https://github.com/dgraph-io/dgraph/pull/6317/files)

---

<div class="post-metadata">

**Author:** ![xxm404](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/xxm404/32/4182_2.png) [@xxm404](https://discuss.dgraph.io/u/xxm404)\
**Post date:** [October 16, 2020, 3:31am UTC](https://discuss.dgraph.io/t/cannot-retrieve-posting-for-uid-from-list-with-key-readts-less-than-mints/8533/10 "2020-10-16T03:31:57Z")

</div>

I get this error, my Dgraph version info like this

```auto
Dgraph version : v20.03.4
Dgraph SHA-256 : ea782f7687004901e2fb4d0a11904417787217ce156104e4f333460966682a86
Commit SHA-1 : 80e6c199e
Commit timestamp : 2020-07-23 11:17:00 -0700
Branch : HEAD
Go version : go1.14.4

```

Do I need to upgrade to solve this problem?

---

<div class="post-metadata">

**Author:** ![ibrahim](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/ibrahim/32/6387_2.png) [@ibrahim](https://discuss.dgraph.io/u/ibrahim)\
**Post date:** [October 16, 2020, 4:32pm UTC](https://discuss.dgraph.io/t/cannot-retrieve-posting-for-uid-from-list-with-key-readts-less-than-mints/8533/11 "2020-10-16T16:32:30Z")

</div>

Hey @xxm404, the fix for this issue is yet to be released. We’ll be releasing v20.03.6 this month and that will have the fix.

If you’re keen to try it now, you can use `v20.03.6-rc1` tag. Please be aware that this isn’t a released version and it could have some bugs.

---

<div class="post-metadata">

**Author:** ![xxm404](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/xxm404/32/4182_2.png) [@xxm404](https://discuss.dgraph.io/u/xxm404)\
**Post date:** [October 17, 2020, 3:06am UTC](https://discuss.dgraph.io/t/cannot-retrieve-posting-for-uid-from-list-with-key-readts-less-than-mints/8533/12 "2020-10-17T03:06:45Z")

</div>

@ibrahim Thanks for the reply. I will update to fix this problem when v20.03.6 releasing.
