Experience Report for Feature Request
What you wanted to do
Use grpc stream of graph membership as result of graph query.
What you actually did
Used a unary endpoint with a deep json response.
Why that wasn’t great, with examples
A deep json tree is good to relay hierarchy - This should remain the default API of dgraph, but I suggest a new API is created that streams a result in a “graphstream” format.
GraphStream is a Java library I have come across that is built to stream graph membership. I am not referring to using the library itself or even the protocol, but I do not have a better name for it. Fundamentally, that library normalizes a graph into a stream of several distinct message types:
- node addition,
- node removal,
- edge addition,
- edge removal,
- graph/node/edge attribute addition,
- graph/node/edge attribute change,
- graph/node/edge attribute removal.
- step
My proposal is to implement the above message types on a gRPC stream. Using a gRPC stream would allow the result to leave the dgraph server and return memory sooner for large responses. As soon as you are sure about the membership in your subgraph and connections, you can send the response on this stream in one of the message types above.
Further, a long-lived gRPC connection could be made with the same subsystem as GraphQL subscriptions that could send changes (including removal) of the result graph on that connection.
Any external references to support your case
Again, I am not suggesting using this library at all - just the idea. Still using gRPC and adding some user-facing streaming endpoints.