Batch import large amount of data

Users Dgraph
1

I Want to Do

I’m using pydgraph to import data into dgraph.What makes me sad is that it takes a long time to import a large amount of data . Every time I import data one by one, I want to know whether dgraph can batch import data?

Import data in this way：
https://github.com/dgraph-io/pydgraph#running-a-mutation

Code :

for file in file_list:
    get_data = ExcelData(data_path, sheetname)
    datas = get_data.readExcel()
    for row in range(int(len(datas))):
        txn = self.client.txn()
        value = list(datas[row].values())
        item_info = self.check_info(value[0])
        try:
            item = {
                'uid': '{}'.format(item_info),
                'dgraph.type': 'obj',
                'id': value[0],
                'name': value[1],
                'grade': value[2],
                'class': value[3],
                'region': value[4],
                'org': value[5],
            }
            print(item)
            response = txn.mutate(set_obj=item)
            txn.commit()
        finally:
            txn.discard()

Dgraph Metadata

dgraph version V20.03
2

Have you tried bulk loader: https://dgraph.io/docs/deploy/fast-data-loading/bulk-loader/ ?

3

I need to import the data into the existing cluster .So you know,this is not in line with my needs :rofl:

4

Try uploading multiple nodes per mutation. From my experience Dgraph seems to be fastest between batch sizes of 100kb-250kb. Alternatively the live loader might work for you as it can upload to an existing cluster.

5

Hi @Soultrans
Live loader is the recommended solution to load into a running cluster. Have you given it a try?

6

I didn’t try to do that.For some reason I had to use pydgrap.Using live loader for data conversion is too cumbersome for me