I’ve been testing the Bulk Loader with that
--xidmap option for some time now. It doesn’t work well. The problem is that it consumes way too much memory. I never managed to load more than 4% of the total data. At that point the process has consumed 400G RAM and crashes.
Running the Bulk Loader right now without the
--xidmap option. This seems to progress just fine.
Blank nodes live in the transaction context, right? What does this translate to with the Bulk Loader?
Our main entities have id:s that we use to construct their blank nodes. Then entities have attributes that may reference other entities (using that id / blank node). The entities also contain/reference nested complex types. These need to be modelled as separate nodes, and therefore also need to be assigned blank nodes at insertion (we just generate something random). This is just something temporary necessary to model the structures.
I don’t understand what needs to be done after using the Bulk Loader without the
--xidmap result in such high memory consumption? seems a bit excessive?
Dgraph version : v20.11.0
Dgraph codename : tchalla
Dgraph SHA-256 : 8acb886b24556691d7d74929817a4ac7d9db76bb8b77de00f44650931a16b6ac
Commit SHA-1 : c4245ad55
Commit timestamp : 2020-12-16 15:55:40 +0530
Branch : HEAD
Go version : go1.15.5
jemalloc enabled : true