Moved from GitHub dgraph/5760
Posted by amaster507:
Suggested by MichelDiz I have thought about giving a “cooldown” (via flag) to liveloader to reduce the stress on the Alphas and Zeros - giving time to the GC do its work. This would be nice for users who have low resources available. Maybe add it to bulkloader too.
Idea: When using live loader to import mass data offer a flag
--cooldown [seconds Int]
TL;DR; Honor live loader process completion over process speed.
How this helps me: My environment is running on the recommended 8Gb of RAM but when trying to load 6 Million Rows of data using the live loader, it crashes Alpha for OOM around 60% of completion. When alpha crashes it does restart but the live loader stops processing with a “trasnport is closing” message. If there was a way to limit the input rate and give a time to cool down and process what has been submitted every X seconds then I believe it would yield a completed process with expected results. I would rather see completion overall instead of overall throughput without completion.
Alternatively: If a flag
-memory-limit [Megabyte Int] could be implemented to watch memory consumption and throttle the input rate as to what that memory could sustain. This would allow a better user experience as a user could set a memory threshold and the live loader could use up to that amount and then throttle input to keep under that threshold to keep from running into OOM issues.
How to Throttle: I am not the expert here, but I would think that throttling the chunk size would be one attainable solution or adding in a timeout between chunks may be another option.