How to commit transactions as batch?

Issues Dgraph Clients
,
1

Moved from GitHub pydgraph/148

Posted by heshuguo:

As titled.

2

anurags92 commented :

Do you want to load a bunch of data together? You can try Bulk Loader for that. Also checkout http://discuss.dgraph.io, much of our community lives there and you can get faster answers.

3

heshuguo commented :

Okay. Thanks.

4

heshuguo commented :

just checked the doc about bulk loader. it seems not work for my case.
Here is from doc:
Dgraph Bulk Loader serves a similar purpose to the Dgraph Live Loader, but can only be used to load data into a new cluster. It cannot be run on an existing Dgraph cluster. Dgraph Bulk Loader is considerably faster than the Dgraph Live Loader and is the recommended way to perform the initial import of large datasets into Dgraph.

here is my case:
about 50000 transactions per second to insert to dgraph, is there a good way to make it faster?

5

anurags92 commented :

I think it should work but 50K txn per second would be stretching the system. From the benchmarks done in 2017, we achieved 46.7K txn per second. I do not understand why would bulk loader not work for your use case?

6

heshuguo commented :

I think it should work but 50K txn per second would be stretching the system. From the benchmarks done in 2017, we achieved 46.7K txn per second. I do not understand why would bulk loader not work for your use case?
Here is the explanation for bulk loader.
Dgraph Bulk Loader serves a similar purpose to the Dgraph Live Loader, but can only be used to load data into a new cluster. It cannot be run on an existing Dgraph cluster.

7

anurags92 commented :

You mean you have an existing cluster in which you want to bulk commit the transactions?

8

heshuguo commented :

You mean you have an existing cluster in which you want to bulk commit the transactions?

yes, I am trying to use dgraph live to import rdf data

9

anurags92 commented :

I don’t think live loader can handle that kind of load. :frowning:

10

parasssh commented :

Based on recent tests, Live loader can do upto 10K rdfs / sec on average. However, a lot will depend on the data set and the XIDs as well.

11

heshuguo commented :

I don’t think live loader can handle that kind of load. :frowning:

Any suggestion?

12

anurags92 commented :

I think the only way would be to use bulk loader.

13

MichelDiz commented :

If you don’t mind about ACID, you can use Ludicrous mode (With Liveloader) and get 100k/RDFs or more. I got 400k/RDFs on my Ryzen 2700X with NVMe. See Sharing some numbers from the ludicrous mode