hi,
I used sample data to test the import on my personal computer.
Respectively used 3 ways :
live loader --new_uids=false,
ludicrous mode live loader --new_uids=false
ludicrous mode live loader --new_uids=true
The remaining parameters are default values.
In each case, I imported the same data three times repeatedly, and found that the import rate dropped severely. And when it was imported for the fourth time, the imported logs began to show a lot of timeouts.
Will the import rate continue to decline? If I want to maintain the dynamic import rate at 50,000 per second, what is the correspondence between the hardware configuration and the amount of data?
After the data is imported, I find that the “id” is gone. Is there a corresponding relationship between the id in the data and the id in the final query? Can I only find it through attributes?
There are some background tasks that Dgraph does to give some guarantees. But there are ways to “hack” it, I have shared some of them here Sharing some numbers from the ludicrous mode. And I think you can get way better results with this machine you have. I got 200k N-Quads/s. Not sure about your numbers. Has your graph 70k N-Quads/s?
It seems that the --conc and --batch parameters play a key role.
When using node normal to test live load, I first performed an incremental test on these two parameters, and the conclusion was that the default values were very good, because adjusting them had little effect or even reduced the load rate.
But in the ludicrous mode, they exerted great power , what caused this?
Yes, in the first minutes of the first half, yes they are. But some Dgraph tasks can still slightly decrease the consistency of the results. Even so, you will still have great results with ludicrous mode.
Basically ludicrous mode unleashes the real power of Dgraph. All ACID guarantees are removed from it. There are no transactions and so on. Thus, there is no consumption of resources to guarantee such standards. That is why we see such exorbitant responses. And the more resources you add, the more you can get out of it.