During reduce process, got this error
How can I add buffers amount?
Hey @jokk33, Can you please let us know the version of dgraph you are using. Also, can you please post the full stack trace, along with the panic.
Any update?
Hey @jokk33, it looks like we’re allocating more than 32 GB of buffers and that’s causing the failure. It seems like Dgraph is using more memory than what we anticipate it to use.
How large is your dataset? This issue shows up only on a specific kind of dataset, I guess.
1 Like
Hi, data contain 2 billion nodes and 4 billion edges. Total rdf.gz is 34G.
Will a large single predicate cause this error? (I have around 2 billion user_id)
With jemalloc optimazation,I used to successfully bulk load 300G data (test data, the amount of each predicate is 1.8 million).
So, it is weird to get this panic, can you help me work out？
Data schema is quite simple, user generate order, order contain product:
<product_id>: string @index(exact) .
<user_id>: string @index(exact) .
<order_id>: string @index(exact) .
<generate>: [uid] @count @reverse .
<contain>: [uid] @count @reverse .
Also get this warning during reduce process.
Bulk command:
nohup ./dgraph bulk --badger.cache_mb=128 -f=/data/dgraph_test/ -s=/data/s.schema --ignore_errors -z=xx.xx.xx.xx:5080 --tmp=/data/tmp1 --out=/data/out1 --map_shards=12 --reduce_shards=3 --replace_out -j=12 --log_dir=/data/log1 &
Hey @jokk33, the warning message is about the cache size and it shouldn’t have an effect on the crash. We’re working on the bug and I’ll post an update here once we have a fix for it.
Thanks a lot, yes, after I add --badger.cache_mb=102400, I can successfully reach 100% but still got this error:
Hey @jokk33, @Naman will share a PR with you soon. You can run that PR locally and it should fix this issue.
1 Like
Appreciate it!
1 Like
Hey @jokk33, fix(allocator): bring in increased buffers for allocator from ristretto by NamanJain8 · Pull Request #7046 · dgraph-io/dgraph · GitHub should fix the issue.
See fix(allocator): increase number of buffers from 32 to 64 in allocator by NamanJain8 · Pull Request #234 · dgraph-io/ristretto · GitHub description as why it might have crashed even at around ~13GB of allocator size.
It would be best if it is possible for you to confirm that the issue is resolved.
2 Likes
