I am currently testing out dgraph for a small project around customer data. Everything is fine loading small amounts of data:
e.g. one table has 19k rows (~38k nodes), loads them fine
another table has 52k rows (~52k nodes), loads them fine
I noticed, however, that when loading slightly larger tables, alpha crashes at around 180k rows and doesnt even recover. I have to start alpha again and restart my data load. I tried chunking it to 10k rows each load but it crashes at around the same amount of data ~180k rows (~180k nodes) (the whole table has about 1.8M rows, so its not even scratching the surface). I have far bigger tables to load (~300M rows in total) and Im not sure whats causing the crash. I scan the stdout and it would have lines that say mem flush around the time it crashed.
Is this due to a small configuration? Im running a single zero/single alpha on windows (yes, i know, why am i on windows, existing servers) (company devops is stingy and wont give me a few linux instances). I forgot to capture the error message. I’ll be sure to capture them next time.
After I restart alpha, i would get random error: “applying proposal. error cannot retrieve posting for uid <> from list with key” scattered through out the stdout. Is this something to be worried about?
I kind of feel you will ask me to load it via the offline method but its regular for us to have 200k new rows of data, at minimum, per day. It would not be feasible for me to stop-start dgraph so i can use the offline loader.
It would be really useful for us if you provide us the logs as and when you get it. I assume that you are using a live loader. You can look at the troubleshooting in the meanwhile. We would be happy if you provide us the exact logs.
As experienced earlier, alpha crashed at more or less the same amount of rows. The following are alpha last few lines. Im trying to get the full stderr/stdout. Any help towards solving this would be appreciated!
We have seen this issue earlier as well. We have a ticket open for this and is being investigated. @ibrahim , fyi.
Would you be willing to share your sample data and the steps that can reliably reproduce this?
Also how are you loading this into dgraph? Curl, Dgo, Ratel or something else?
And I believe you are on v20.03.0 and as you mentioned “running on windows aws ec2 i3.2xlarge with 61gb of ram (im currently sharing it with some other application)”
Unfortunately, its customer email and Im constrained by privacy laws so wont be able to share data. I am loading through dgo.
At a high level, it starts with a query to sql database and goes row by row. It firsts upserts the email address to get the uid, and then updates the node with other details (scalar) or edges to other nodes.
What it was doing here was upserting the email address before it crashed. as it reaches the 10k threshold, it will collate the uids and update the email node with other relevant data and updates the node. Once its finished with the update, it will drop the dgraph and sql connections and create new ones for the next 10k rows. It will repeat until the max keys in sql and in dgraph match.
There are about 1.8 million emails to be stored. It crashes at 180k.