Go back ui page (port 8000) , make query and not response. The error: context deadline exceeded (see image)
Detail:
I have Dgraph Database with huge data (over 10 million nodes). After i change price predicate from string to float (current with over 5 million nodes has price data) and index it, Dgraph server has been freeze and i can’t query or mutate. This is the error:
It seem to be hang forever and can’t excute any query.
I have restart Dgraph service, but still got the problem.
What happen, is it the bug on Dgraph when run with huge data?
This is not a bug. Unfortunately, in order to keep all our consistency guarantees, Dgraph blocks while rebuilding an index. So adding an index to a predicate with lots of data might result in Dgraph being unavailable for a while. Restarting the server doesn’t help because the index rebuild process starts right after restart. It’s better to just let it complete.
Two strategies to reduce the impact:
Think though your schema and desired queries thoroughly before you start adding data. This minimizes the need to change the indices/schema.
If the need arises, perform the change at some point with low traffic (e.g at night).
Unfortunately this is the current state of affairs. Once version 1.1 is released we can start looking into improving this process.
This is a really unfortunate state to be in. There should be some way to cancel building the index since it’s can be hideously slow. I left it running for 10 hours without it finishing and doesn’t seem to be any way to view the progress.
From a recent blog post “One of the cornerstones of Dgraph is that it allows a flexible schema, which can be modified in a live system, without any downtime. This involves changing data types and adding or removing indices with a single ALTER command to match the needs of an application developer.” https://blog.dgraph.io/post/indexing-in-dgraph/
This post states that there is no downtime which according to this issue is not the case. Ideally this would happen in the background like how CockroachDB does it and only use the index once it’s fully built. CockroachDB also provides info on status of the index build. Online Schema Changes | CockroachDB Docs
Yes. We should be fixing this problem. Allow queries to use the index once it’s built and also allow indices to be built using disk, instead of entirely in memory. Feel free to file an issue to track that. We’ll get to it with priority after v1.1 release. CC: @campoy .
For the time being, if you use best effort queries, you’d still be able to get the queries to work.
