Batch import large amount of data

I Want to Do

I’m using pydgraph to import data into dgraph.What makes me sad is that it takes a long time to import a large amount of data . Every time I import data one by one, I want to know whether dgraph can batch import data?

Import data in this way:

Code :

for file in file_list:
    get_data = ExcelData(data_path, sheetname)
    datas = get_data.readExcel()
    for row in range(int(len(datas))):
        txn = self.client.txn()
        value = list(datas[row].values())
        item_info = self.check_info(value[0])
            item = {
                'uid': '{}'.format(item_info),
                'dgraph.type': 'obj',
                'id': value[0],
                'name': value[1],
                'grade': value[2],
                'class': value[3],
                'region': value[4],
                'org': value[5],
            response = txn.mutate(set_obj=item)

Dgraph Metadata

dgraph version V20.03

Have you tried bulk loader: ?

I need to import the data into the existing cluster .So you know,this is not in line with my needs :rofl:

Try uploading multiple nodes per mutation. From my experience Dgraph seems to be fastest between batch sizes of 100kb-250kb. Alternatively the live loader might work for you as it can upload to an existing cluster.

Hi @Soultrans
Live loader is the recommended solution to load into a running cluster. Have you given it a try?

I didn’t try to do that.For some reason I had to use pydgrap.Using live loader for data conversion is too cumbersome for me