Add pagination to schema query

Issues Dgraph
, , , ,
1

Moved from GitHub dgraph/5810

Posted by EnricoMi:

Experience Report

What you wanted to do

I have a huge schema of 235k predicates and my application has to retrieve that schema from dgraph first.I am using

schema { predicate type }

Why that wasn’t great, with examples

The response becomes a 17 MB gRPC response message, so I get the following exception:

Caused by: java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: gRPC message exceeds maximum size 4194304: 17476444
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
	at io.dgraph.DgraphAsyncClient.lambda$runWithRetries$2(DgraphAsyncClient.java:181)
	... 6 more

I can set the max inbound message size via maxInboundMessageSize to a higher value, but no matter what value limit I set, there might be a schema that breaks this setting. I’d rather like a solution that 1) works for any schema (size) and 2) has a small limited memory footprint.

Some pagination of the schema result would be great, so that I can iterate over the schema and retrieve it in batches. Should be combined with retrieving predicates or types only (see issue #5811).

Any external references to support your case

Discussed in the forum.