# Bulk load fails

**URL:** <https://discuss.dgraph.io/t/bulk-load-fails/5627>\
**Category:** Dgraph\
**Created:** [December 12, 2019, 6:29pm UTC](https://discuss.dgraph.io/t/bulk-load-fails/5627 "2019-12-12T18:29:10Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![guoweis](https://avatars.discourse-cdn.com/v4/letter/g/3ab097/32.png) [@guoweis](https://discuss.dgraph.io/u/guoweis)\
**Post date:** [December 12, 2019, 6:29pm UTC](https://discuss.dgraph.io/t/bulk-load-fails/5627/1 "2019-12-12T18:29:10Z")

</div>

I’m trying to do bulk load but running into this

**Error on Zero**  
I1212 18:21:11.291695 1 assign.go:41] Updated Lease id: 18446055125946300485. Txn Ts: 110001  
W1212 18:21:12.993017 1 node.go:669] [0x1] Read index context timed out  
W1212 18:22:02.253909 1 raft.go:729] Raft.Ready took too long to process: Timer Total: 280ms. Breakdown: [{disk 277ms} {advance 1ms} {proposals 0s}]. Num entries: 0. MustSync: false  
W1212 18:22:05.934343 1 raft.go:729] Raft.Ready took too long to process: Timer Total: 299ms. Breakdown: [{disk 146ms} {proposals 47ms} {advance 10ms}]. Num entries: 0. MustSync: false  
W1212 18:22:13.235066 1 raft.go:729] Raft.Ready took too long to process: Timer Total: 1.204s. Breakdown: [{disk 965ms} {proposals 212ms} {advance 8ms}]. Num entries: 0. MustSync: false  
W1212 18:22:16.532623 1 node.go:669] [0x1] Read index context timed out  
W1212 18:22:27.271759 1 node.go:669] [0x1] Read index context timed out  
W1212 18:22:21.670279 1 raft.go:729] Raft.Ready took too long to process: Timer Total: 831ms. Breakdown: [{disk 803ms} {proposals 24ms} {advance 0s}]. Num entries: 0. MustSync: false  
W1212 18:22:49.268878 1 node.go:669] [0x1] Read index context timed out  
W1212 18:23:01.828706 1 node.go:669] [0x1] Read index context timed out  
W1212 18:23:04.633381 1 raft.go:729] Raft.Ready took too long to process: Timer Total: 5.149s. Breakdown: [{disk 5.127s} {proposals 19ms} {advance 1ms}]. Num entries: 0. MustSync: false  
W1212 18:23:04.945402 1 raft.go:729] Raft.Ready took too long to process: Timer Total: 224ms. Breakdown: [{disk 96ms} {advance 81ms} {proposals 27ms}]. Num entries: 0. MustSync: false  
W1212 18:23:51.959628 1 node.go:669] [0x1] Read index context timed out  
E1212 18:24:19.559844 1 node.go:654] Error while trying to call ReadIndex: context deadline exceeded  
E1212 18:24:34.035801 1 node.go:707] [0x1] While trying to do lin read index: context deadline exceeded

**This is my dgraph zero**  
Dgraph version : v1.1.0  
Dgraph SHA-256 : 7d4294a80f74692695467e2cf17f74648c18087ed7057d798f40e1d3a31d2095  
Commit SHA-1 : ef7cdb28  
Commit timestamp : 2019-09-04 00:12:51 -0700  
Branch : HEAD  
Go version : go1.12.7

When running this, I have one instance of zero, no alpha running at that time.

**The command line**  
docker exec -it dgraph dgraph bulk -f mail.rdf.gz -s schema.txt --map\_shards=8 --reduce\_shards=4 --http localhost:8000 --zero=localhost:5080

This is running on my mac book pro, with 32G memory.

**This is output from the bulk loader**  
[Decoder]: Using assembly version of decoder  
I1212 18:21:29.948590 24 init.go:98]

Dgraph version : v1.1.0  
Dgraph SHA-256 : 7d4294a80f74692695467e2cf17f74648c18087ed7057d798f40e1d3a31d2095  
Commit SHA-1 : ef7cdb28  
Commit timestamp : 2019-09-04 00:12:51 -0700  
Branch : HEAD  
Go version : go1.12.7

Licensed variously under the Apache Public License 2.0 and Dgraph Community License.  
Copyright 2015-2018 Dgraph Labs, Inc.

{  
“DataFiles”: “mail.rdf.gz”,  
“DataFormat”: “”,  
“SchemaFile”: “schema.txt”,  
“OutDir”: “./out”,  
“ReplaceOutDir”: false,  
“TmpDir”: “tmp”,  
“NumGoroutines”: 2,  
“MapBufSize”: 67108864,  
“SkipMapPhase”: false,  
“CleanupTmp”: true,  
“NumReducers”: 1,  
“Version”: false,  
“StoreXids”: false,  
“ZeroAddr”: “localhost:5080”,  
“HttpAddr”: “localhost:8000”,  
“IgnoreErrors”: false,  
“CustomTokenizers”: “”,  
“NewUids”: false,  
“MapShards”: 8,  
“ReduceShards”: 4  
}  
Connecting to zero at localhost:5080  
Processing file (1 out of 1): mail.rdf.gz  
[18:21:31Z] MAP 01s nquad\_count:110.6k err\_count:0.000 nquad\_speed:109.9k/sec edge\_count:185.7k edge\_speed:184.5k/sec  
[18:21:32Z] MAP 02s nquad\_count:250.2k err\_count:0.000 nquad\_speed:124.7k/sec edge\_count:420.3k edge\_speed:209.4k/sec  
[18:21:33Z] MAP 03s nquad\_count:371.5k err\_count:0.000 nquad\_speed:123.5k/sec edge\_count:623.8k edge\_speed:207.4k/sec  
[18:21:34Z] MAP 04s nquad\_count:514.9k err\_count:0.000 nquad\_speed:128.4k/sec edge\_count:864.6k edge\_speed:215.6k/sec  
[18:21:35Z] MAP 05s nquad\_count:653.5k err\_count:0.000 nquad\_speed:130.4k/sec edge\_count:1.097M edge\_speed:219.1k/sec  
[18:21:36Z] MAP 06s nquad\_count:785.9k err\_count:0.000 nquad\_speed:130.8k/sec edge\_count:1.320M edge\_speed:219.6k/sec  
[18:21:37Z] MAP 07s nquad\_count:918.0k err\_count:0.000 nquad\_speed:130.9k/sec edge\_count:1.542M edge\_speed:219.9k/sec  
[18:21:38Z] MAP 08s nquad\_count:1.034M err\_count:0.000 nquad\_speed:129.1k/sec edge\_count:1.736M edge\_speed:216.7k/sec  
[18:21:39Z] MAP 09s nquad\_count:1.156M err\_count:0.000 nquad\_speed:128.3k/sec edge\_count:1.941M edge\_speed:215.4k/sec  
GC: 19. InUse: 719 MB. Idle: 53 MB  
[18:21:40Z] MAP 10s nquad\_count:1.284M err\_count:0.000 nquad\_speed:128.3k/sec edge\_count:2.156M edge\_speed:215.4k/sec  
[18:21:41Z] MAP 11s nquad\_count:1.398M err\_count:0.000 nquad\_speed:126.9k/sec edge\_count:2.347M edge\_speed:213.1k/sec  
[18:21:42Z] MAP 12s nquad\_count:1.563M err\_count:0.000 nquad\_speed:130.1k/sec edge\_count:2.625M edge\_speed:218.5k/sec  
[18:21:43Z] MAP 13s nquad\_count:1.686M err\_count:0.000 nquad\_speed:129.6k/sec edge\_count:2.832M edge\_speed:217.6k/sec  
[18:21:44Z] MAP 14s nquad\_count:1.790M err\_count:0.000 nquad\_speed:127.7k/sec edge\_count:3.006M edge\_speed:214.5k/sec  
[18:21:45Z] MAP 15s nquad\_count:1.949M err\_count:0.000 nquad\_speed:129.8k/sec edge\_count:3.273M edge\_speed:218.0k/sec  
[18:21:46Z] MAP 16s nquad\_count:2.068M err\_count:0.000 nquad\_speed:129.1k/sec edge\_count:3.472M edge\_speed:216.8k/sec  
[18:21:47Z] MAP 17s nquad\_count:2.190M err\_count:0.000 nquad\_speed:128.7k/sec edge\_count:3.677M edge\_speed:216.1k/sec  
[18:21:48Z] MAP 18s nquad\_count:2.344M err\_count:0.000 nquad\_speed:130.1k/sec edge\_count:3.936M edge\_speed:218.5k/sec  
[18:21:49Z] MAP 19s nquad\_count:2.470M err\_count:0.000 nquad\_speed:129.9k/sec edge\_count:4.148M edge\_speed:218.1k/sec  
GC: 23. InUse: 1.5 GB. Idle: 44 MB  
[18:21:50Z] MAP 20s nquad\_count:2.558M err\_count:0.000 nquad\_speed:127.7k/sec edge\_count:4.295M edge\_speed:214.5k/sec  
[18:21:51Z] MAP 21s nquad\_count:2.710M err\_count:0.000 nquad\_speed:128.9k/sec edge\_count:4.551M edge\_speed:216.5k/sec  
[18:21:52Z] MAP 22s nquad\_count:2.843M err\_count:0.000 nquad\_speed:129.1k/sec edge\_count:4.774M edge\_speed:216.8k/sec  
[18:21:53Z] MAP 23s nquad\_count:2.965M err\_count:0.000 nquad\_speed:128.8k/sec edge\_count:4.980M edge\_speed:216.3k/sec  
[18:21:54Z] MAP 24s nquad\_count:3.050M err\_count:0.000 nquad\_speed:127.0k/sec edge\_count:5.122M edge\_speed:213.2k/sec  
[18:21:55Z] MAP 25s nquad\_count:3.151M err\_count:0.000 nquad\_speed:125.9k/sec edge\_count:5.292M edge\_speed:211.5k/sec  
[18:21:56Z] MAP 26s nquad\_count:3.244M err\_count:0.000 nquad\_speed:124.7k/sec edge\_count:5.448M edge\_speed:209.3k/sec  
[18:21:57Z] MAP 27s nquad\_count:3.348M err\_count:0.000 nquad\_speed:123.9k/sec edge\_count:5.621M edge\_speed:208.0k/sec  
[18:21:58Z] MAP 28s nquad\_count:3.440M err\_count:0.000 nquad\_speed:122.7k/sec edge\_count:5.776M edge\_speed:206.1k/sec  
[18:21:59Z] MAP 29s nquad\_count:3.528M err\_count:0.000 nquad\_speed:121.5k/sec edge\_count:5.923M edge\_speed:204.1k/sec  
GC: 24. InUse: 2.0 GB. Idle: 3.8 MB  
[18:22:00Z] MAP 30s nquad\_count:3.596M err\_count:0.000 nquad\_speed:119.7k/sec edge\_count:6.038M edge\_speed:201.1k/sec  
[18:22:01Z] MAP 31s nquad\_count:3.682M err\_count:0.000 nquad\_speed:118.7k/sec edge\_count:6.183M edge\_speed:199.3k/sec  
[18:22:02Z] MAP 32s nquad\_count:3.725M err\_count:0.000 nquad\_speed:116.2k/sec edge\_count:6.255M edge\_speed:195.1k/sec  
[18:22:03Z] MAP 33s nquad\_count:3.729M err\_count:0.000 nquad\_speed:112.5k/sec edge\_count:6.262M edge\_speed:188.8k/sec  
[18:22:04Z] MAP 34s nquad\_count:3.752M err\_count:0.000 nquad\_speed:109.8k/sec edge\_count:6.301M edge\_speed:184.3k/sec  
[18:22:05Z] MAP 35s nquad\_count:3.760M err\_count:0.000 nquad\_speed:106.8k/sec edge\_count:6.313M edge\_speed:179.3k/sec  
[18:22:06Z] MAP 36s nquad\_count:3.766M err\_count:0.000 nquad\_speed:104.0k/sec edge\_count:6.324M edge\_speed:174.6k/sec  
[18:22:07Z] MAP 37s nquad\_count:3.783M err\_count:0.000 nquad\_speed:101.4k/sec edge\_count:6.352M edge\_speed:170.3k/sec  
[18:22:08Z] MAP 38s nquad\_count:3.788M err\_count:0.000 nquad\_speed:98.86k/sec edge\_count:6.360M edge\_speed:166.0k/sec  
GC: 25. InUse: 1.7 GB. Idle: 545 MB  
[18:22:09Z] MAP 39s nquad\_count:3.788M err\_count:0.000 nquad\_speed:95.96k/sec edge\_count:6.360M edge\_speed:161.1k/sec  
[18:22:12Z] MAP 42s nquad\_count:3.790M err\_count:0.000 nquad\_speed:89.28k/sec edge\_count:6.363M edge\_speed:149.9k/sec  
[18:22:13Z] MAP 43s nquad\_count:3.790M err\_count:0.000 nquad\_speed:86.82k/sec edge\_count:6.364M edge\_speed:145.8k/sec  
[18:22:16Z] MAP 46s nquad\_count:3.790M err\_count:0.000 nquad\_speed:81.07k/sec edge\_count:6.364M edge\_speed:136.1k/sec  
[18:22:18Z] MAP 48s nquad\_count:3.790M err\_count:0.000 nquad\_speed:78.69k/sec edge\_count:6.364M edge\_speed:132.1k/sec  
GC: 25. InUse: 1.7 GB. Idle: 543 MB  
[18:22:19Z] MAP 51s nquad\_count:3.790M err\_count:0.000 nquad\_speed:73.87k/sec edge\_count:6.364M edge\_speed:124.0k/sec  
[18:22:49Z] MAP 01m19s nquad\_count:3.790M err\_count:0.000 nquad\_speed:47.68k/sec edge\_count:6.364M edge\_speed:80.05k/sec  
[18:22:50Z] MAP 01m21s nquad\_count:3.790M err\_count:0.000 nquad\_speed:46.76k/sec edge\_count:6.365M edge\_speed:78.51k/sec  
GC: 26. InUse: 1.7 GB. Idle: 613 MB  
[18:22:58Z] MAP 01m32s nquad\_count:3.793M err\_count:0.000 nquad\_speed:41.05k/sec edge\_count:6.365M edge\_speed:68.87k/sec  
[18:23:19Z] MAP 01m49s nquad\_count:3.793M err\_count:0.000 nquad\_speed:34.73k/sec edge\_count:6.369M edge\_speed:58.32k/sec  
[18:23:31Z] MAP 02m01s nquad\_count:3.793M err\_count:0.000 nquad\_speed:31.26k/sec edge\_count:6.369M edge\_speed:52.50k/sec  
[18:23:55Z] MAP 02m25s nquad\_count:3.793M err\_count:0.000 nquad\_speed:26.13k/sec edge\_count:6.369M edge\_speed:43.87k/sec  
[18:23:56Z] MAP 02m26s nquad\_count:3.793M err\_count:0.000 nquad\_speed:25.94k/sec edge\_count:6.369M edge\_speed:43.55k/sec  
[18:23:57Z] MAP 02m29s nquad\_count:3.794M err\_count:0.000 nquad\_speed:25.34k/sec edge\_count:6.371M edge\_speed:42.54k/sec  
GC: 27. InUse: 1.7 GB. Idle: 613 MB  
[18:24:22Z] MAP 02m52s nquad\_count:3.794M err\_count:0.000 nquad\_speed:21.95k/sec edge\_count:6.371M edge\_speed:36.85k/sec  
[18:24:27Z] MAP 02m58s nquad\_count:3.794M err\_count:0.000 nquad\_speed:21.26k/sec edge\_count:6.371M edge\_speed:35.69k/sec  
[18:24:31Z] MAP 03m01s nquad\_count:3.795M err\_count:0.000 nquad\_speed:20.86k/sec edge\_count:6.372M edge\_speed:35.02k/sec

---

<div class="post-metadata">

**Author:** ![Willem520](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/willem520/32/2503_2.png) [@Willem520](https://discuss.dgraph.io/u/Willem520)\
**Post date:** [December 13, 2019, 8:40am UTC](https://discuss.dgraph.io/t/bulk-load-fails/5627/2 "2019-12-13T08:40:59Z")

</div>

bulk loader would use a large memory，if your dataset is too large,it maybe occur error to fail

---

<div class="post-metadata">

**Author:** ![ashishgoswami](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/ashishgoswami/32/3416_2.png) [@ashishgoswami](https://discuss.dgraph.io/u/ashishgoswami)\
**Post date:** [December 16, 2019, 7:44am UTC](https://discuss.dgraph.io/t/bulk-load-fails/5627/3 "2019-12-16T07:44:40Z")

</div>

Hey @guoweis, I would like to know few things to proceed further:

- What was the error at client side when bulk loader failed? It is not clear from logs.
- How much data are you trying to insert(size, count)?
- If possible please provide memory/cpu usage before failure.

We have done some optimisations in bulk loader recently, please try to run it on master(or v1.1.1-rc2) and let us know if the issue still occurs.