Bulk data uploader gets killed silently

Users
1

I am trying to ingest data into my dgraph instance on my local machine. The dgraph bulk uploader gets killed silently. Here are the logs of bulk uplaoder -

docker exec -it dgraph dgraph bulk -r mydata/data.rdf -s mydata/dgraph.schema  --http localhost:8000 --zero=localhost:5080
{
	"RDFDir": "mydata/data.rdf",
	"SchemaFile": "mydata/dgraph.schema",
	"DgraphsDir": "out",
	"TmpDir": "tmp",
	"NumGoroutines": 4,
	"MapBufSize": 67108864,
	"ExpandEdges": true,
	"SkipMapPhase": false,
	"CleanupTmp": true,
	"NumShufflers": 1,
	"Version": false,
	"StoreXids": false,
	"ZeroAddr": "localhost:5080",
	"HttpAddr": "localhost:8000",
	"IgnoreErrors": false,
	"MapShards": 1,
	"ReduceShards": 1
}
The bulk loader needs to open many files at once. This number depends on the size of the data set loaded, the map file output size, and the level of indexing. 100,000 is adequate for most data set sizes. See `man ulimit` for details of how to change the limit.
Current max open files limit: 1048576
2018/10/22 05:51:57 loader.go:77: Connecting to zero at localhost:5080
2018/10/22 05:51:57 Replaying from value pointer: {Fid:0 Len:0 Offset:0}
2018/10/22 05:51:57 Iterating file id: 0
2018/10/22 05:51:57 Iteration took: 771.329µs
processing file (1 out of 1): mydata/data.rdf
MAP 01s rdf_count:2.690k err_count:0.000 rdf_speed:2.671k/sec edge_count:7.988k edge_speed:7.931k/sec
MAP 02s rdf_count:251.7k err_count:0.000 rdf_speed:125.4k/sec edge_count:751.1k edge_speed:374.2k/sec
MAP 03s rdf_count:585.7k err_count:0.000 rdf_speed:194.7k/sec edge_count:1.753M edge_speed:582.8k/sec
MAP 04s rdf_count:871.5k err_count:0.000 rdf_speed:217.4k/sec edge_count:2.608M edge_speed:650.7k/sec
MAP 05s rdf_count:1.188M err_count:0.000 rdf_speed:237.1k/sec edge_count:3.474M edge_speed:693.5k/sec
MAP 06s rdf_count:1.440M err_count:0.000 rdf_speed:239.6k/sec edge_count:4.120M edge_speed:685.6k/sec
MAP 07s rdf_count:1.537M err_count:0.000 rdf_speed:219.2k/sec edge_count:4.348M edge_speed:620.3k/sec
MAP 08s rdf_count:1.786M err_count:0.000 rdf_speed:223.0k/sec edge_count:4.858M edge_speed:606.5k/sec
MAP 09s rdf_count:2.136M err_count:0.000 rdf_speed:236.8k/sec edge_count:5.718M edge_speed:633.7k/sec
MAP 10s rdf_count:2.314M err_count:0.000 rdf_speed:230.9k/sec edge_count:6.123M edge_speed:610.9k/sec
MAP 11s rdf_count:2.361M err_count:0.000 rdf_speed:214.1k/sec edge_count:6.256M edge_speed:567.3k/sec
MAP 12s rdf_count:2.416M err_count:0.000 rdf_speed:200.8k/sec edge_count:6.421M edge_speed:533.5k/sec
MAP 13s rdf_count:2.427M err_count:0.000 rdf_speed:186.1k/sec edge_count:6.453M edge_speed:494.7k/sec
MAP 14s rdf_count:2.435M err_count:0.000 rdf_speed:173.3k/sec edge_count:6.476M edge_speed:460.9k/sec
MAP 15s rdf_count:2.440M err_count:0.000 rdf_speed:162.1k/sec edge_count:6.493M edge_speed:431.4k/sec
MAP 16s rdf_count:2.450M err_count:0.000 rdf_speed:152.6k/sec edge_count:6.523M edge_speed:406.3k/sec
MAP 17s rdf_count:2.460M err_count:0.000 rdf_speed:144.3k/sec edge_count:6.553M edge_speed:384.2k/sec
MAP 18s rdf_count:2.472M err_count:0.000 rdf_speed:136.9k/sec edge_count:6.589M edge_speed:364.9k/sec
MAP 19s rdf_count:2.484M err_count:0.000 rdf_speed:130.4k/sec edge_count:6.624M edge_speed:347.6k/sec
MAP 20s rdf_count:2.502M err_count:0.000 rdf_speed:124.8k/sec edge_count:6.679M edge_speed:333.0k/sec
MAP 21s rdf_count:2.516M err_count:0.000 rdf_speed:119.5k/sec edge_count:6.721M edge_speed:319.2k/sec
MAP 22s rdf_count:2.531M err_count:0.000 rdf_speed:114.8k/sec edge_count:6.765M edge_speed:306.7k/sec
MAP 23s rdf_count:2.544M err_count:0.000 rdf_speed:110.3k/sec edge_count:6.803M edge_speed:295.0k/sec
MAP 24s rdf_count:2.557M err_count:0.000 rdf_speed:106.3k/sec edge_count:6.842M edge_speed:284.4k/sec
MAP 25s rdf_count:2.561M err_count:0.000 rdf_speed:102.2k/sec edge_count:6.856M edge_speed:273.6k/sec
MAP 26s rdf_count:2.571M err_count:0.000 rdf_speed:98.63k/sec edge_count:6.884M edge_speed:264.1k/sec
MAP 27s rdf_count:2.586M err_count:0.000 rdf_speed:95.53k/sec edge_count:6.931M edge_speed:256.0k/sec
MAP 28s rdf_count:2.614M err_count:0.000 rdf_speed:93.11k/sec edge_count:7.014M edge_speed:249.8k/sec
MAP 29s rdf_count:2.634M err_count:0.000 rdf_speed:90.37k/sec edge_count:7.073M edge_speed:242.7k/sec
✘-KILL ~/dgraph/mydata
2

I was able to successfully bulk upload after setting -j 1. Thanks to @amanmangal for helping me out.
Some error message would have been helpful.

3

In this case, the process was killed by the OS or the VM. Dgraph doesn’t get any more information about why it was killed than whatever is shown.

4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.