After and orderdesc: question on large datasets, ordering, and pagination

Users Dgraph
,
1

To facilitate better answering of questions, if you have a question, please fill in the following info. Otherwise, please delete the template.

What I want to do

I want to be able to page data (preferably using after) after sorting. For example, after I obtain the latest data, sorted via orderdesc: dateCreated, I’ll use after to fetch data older than the last data in the list using its uid.

I understand that after is very efficient compared to offset because the former has O(1) complexity according to the documentation. However, I find it weird (or rather quite annoying) that I cannot page my data using after in Dgraph when I set orderdesc on time-series data.

Is there something similar to after that has a “lookback” ordering strategy (e.g., maybe before), or do I have no choice but to use offset in this case?

Thanks in advance!

Dgraph metadata

dgraph version

Dgraph version : v23.1.0
Dgraph codename : dgraph
Dgraph SHA-256 : 2b0d2fb977807f9d681c3a8e5f67a6fb133c99c772009158107aa6b1ac4cbd10
Commit SHA-1 : 2b18d19
Commit timestamp : 2023-08-17 13:27:10 -0500
Branch : HEAD
Go version : go1.19.12
jemalloc enabled : true

2

Ha!, just realized you can’t even do this with orderasc either. I never really utilized after, but this makes it useless if there is no way that I see to use it in conjunction with ordering at all. I even tried using a block to do the ordering and pagination in separate blocks, and it still did not work as expected.

# DQL
query {
  sortedAndPaginated(
    func: has(name),
    orderasc: name,
    after: 0xfffd8d7297c1984f
  ) {
    uid
    name
  }
  chars as sorted(func: has(name), orderasc: name) {
    uid
    name
  }
  paginated(
    func: uid(chars),
    after: 0xfffd8d7297c1984f # Luke Skywalker
  ) {
    uid
    name
  }
}

// Response
{
  data:{
    sortedAndPaginated: [
      {
        uid:"0xfffd8d7297c19851",
        name:"Han Solo"
      },
      {
        uid:"0xfffd8d7297c19850",
        name:"Princess Leia"
      }
    ]
    sorted:[
      {
        uid:"0xfffd8d7297c19851",
        name:"Han Solo"
      },
      {
        uid:"0xfffd8d7297c1984f",
        name:"Luke Skywalker"
      },
      {
        uid:"0xfffd8d7297c19850",
        name:"Princess Leia"
      }
    ],
    paginated: [
      {
        uid:"0xfffd8d7297c19850",
        name:"Princess Leia"
      },
      {
        uid:"0xfffd8d7297c19851",
        name:"Han Solo"
      }
    ]
  }
}
1 Like
3

I have not tried orderBy with after. I generally use after: with natural ordering to process UIDs for a bulk operation such as a specialized export, schema update etc.

If you just want consistent paging, you might instead use start: and offset. If you are concerned about updates perturbing the results, use a long-running transaction via client.newTransaction() and make all calls within that read transaction. All retrieval will use a single point-in-time snapshot so results don’t change during paging.

See also: Java - DQL or other client libraries for transaction management