Checking Bulk/Liveloader improvement. (Draft)

Users Dgraph
1

Dataset: 21M

Dgraph test on v20.03.0

Sys TMP files on => WDS500G3XHC-00SJG0
Run on => Kingston SKC1000240G

nquad_speed:212.9k/sec
edge_speed:958.7k/sec
plist_speed:278.8k/sec
edge_speed:2.136M/sec

Total: 02m53s

Dgraph test on v20.07.0

Sys TMP files on => WDS500G3XHC-00SJG0
Run on => Kingston SKC1000240G

nquad_speed:255.2k/sec
edge_speed:1.007M/sec
plist_speed:271.5k/sec
edge_speed:2.168M/sec

Total: 02m49s

Dgraph test on MASTER BRANCH with jemalloc

Sys TMP files on => WDS500G3XHC-00SJG0
Run on => Kingston SKC1000240G

nquad_speed:963.4k/sec
edge_speed:2.556M/sec
plist_speed:856.8k/sec
edge_speed:4.263M/sec

Total: 01m57s

Note: Feels like it uses less RAM and also cleans up faster.

Logs:

[23:27:26-0300] MAP 50s nquad_count:21.24M err_count:0.000 nquad_speed:424.7k/sec edge_count:102.3M edge_speed:2.046M/sec GoMem MB: 2280 CMem MB: 4096
[23:27:27-0300] MAP 51s nquad_count:21.24M err_count:0.000 nquad_speed:416.4k/sec edge_count:102.3M edge_speed:2.006M/sec GoMem MB: 2280 CMem MB: 4096
[23:27:28-0300] MAP 52s nquad_count:21.24M err_count:0.000 nquad_speed:408.3k/sec edge_count:102.3M edge_speed:1.967M/sec GoMem MB: 2280 CMem MB: 4096
[23:27:29-0300] MAP 53s nquad_count:21.24M err_count:0.000 nquad_speed:400.7k/sec edge_count:102.3M edge_speed:1.930M/sec GoMem MB: 2280 CMem MB: 4096
[23:27:30-0300] MAP 54s nquad_count:21.24M err_count:0.000 nquad_speed:393.3k/sec edge_count:102.3M edge_speed:1.894M/sec GoMem MB: 2280 CMem MB: 4096
[23:27:31-0300] MAP 55s nquad_count:21.24M err_count:0.000 nquad_speed:386.0k/sec edge_count:102.3M edge_speed:1.859M/sec GoMem MB: 2280 CMem MB: 4096
[23:27:32-0300] MAP 56s nquad_count:21.24M err_count:0.000 nquad_speed:379.2k/sec edge_count:102.3M edge_speed:1.827M/sec GoMem MB: 2280 CMem MB: 4096
[23:27:33-0300] MAP 57s nquad_count:21.24M err_count:0.000 nquad_speed:372.6k/sec edge_count:102.3M edge_speed:1.795M/sec GoMem MB: 2280 CMem MB: 4096
[23:27:34-0300] MAP 58s nquad_count:21.24M err_count:0.000 nquad_speed:366.1k/sec edge_count:102.3M edge_speed:1.763M/sec GoMem MB: 2280 CMem MB: 4096
[23:27:35-0300] MAP 59s nquad_count:21.24M err_count:0.000 nquad_speed:360.0k/sec edge_count:102.3M edge_speed:1.734M/sec GoMem MB: 2280 CMem MB: 4096
[23:27:36-0300] MAP 01m00s nquad_count:21.24M err_count:0.000 nquad_speed:354.0k/sec edge_count:102.3M edge_speed:1.705M/sec GoMem MB: 2280 CMem MB: 4096
[23:27:37-0300] MAP 01m01s nquad_count:21.24M err_count:0.000 nquad_speed:348.1k/sec edge_count:102.3M edge_speed:1.677M/sec GoMem MB: 2280 CMem MB: 4096

___ Begin jemalloc statistics ___
Version: "5.2.1-531-gd243b4ec487224248172547643630f7a5fb5e84d"
Build-time option settings
  config.cache_oblivious: true
  config.debug: false
  config.fill: true
  config.lazy_lock: false
  config.malloc_conf: ""
  config.opt_safety_checks: false
  config.prof: true
  config.prof_libgcc: false
  config.prof_libunwind: false
  config.stats: true
  config.utrace: false
  config.xmalloc: false
Run-time option settings
  opt.abort: false
  opt.abort_conf: false
  opt.confirm_conf: false
  opt.retain: false
  opt.dss: "secondary"
  opt.narenas: 64
  opt.percpu_arena: "disabled"
  opt.oversize_threshold: 8388608
  opt.metadata_thp: "disabled"
  opt.dirty_decay_ms: 10000 (arenas.dirty_decay_ms: 10000)
  opt.muzzy_decay_ms: 0 (arenas.muzzy_decay_ms: 0)
  opt.lg_extent_max_active_fit: 6
  opt.junk: "false"
  opt.zero: false
  opt.tcache: true
  opt.lg_tcache_max: 15
  opt.tcache_nslots_small_min: 20
  opt.tcache_nslots_small_max: 200
  opt.tcache_nslots_large: 20
  opt.lg_tcache_nslots_mul: 1
  opt.tcache_gc_incr_bytes: 65536
  opt.tcache_gc_delay_bytes: 0
  opt.lg_tcache_flush_small_div: 1
  opt.lg_tcache_flush_large_div: 1
  opt.thp: "not supported"
  opt.prof: false
  opt.prof_prefix: "jeprof"
  opt.prof_active: true (prof.active: false)
  opt.prof_thread_active_init: true (prof.thread_active_init: false)
  opt.lg_prof_sample: 19 (prof.lg_sample: 0)
  opt.prof_accum: false
  opt.lg_prof_interval: -1
  opt.prof_gdump: false
  opt.prof_final: false
  opt.prof_leak: false
  opt.stats_print: false
  opt.stats_print_opts: ""
  opt.stats_print: false
  opt.stats_print_opts: ""
  opt.stats_interval: -1
  opt.stats_interval_opts: ""
  opt.zero_realloc: "strict"
Profiling settings
  prof.thread_active_init: false
  prof.active: false
  prof.gdump: false
  prof.interval: 0
  prof.lg_sample: 0
Arenas: 65
Quantum size: 16
Page size: 4096
Maximum thread-cached size class: 32768
Number of bin size classes: 36
Number of thread-cache bin size classes: 41
Number of large size classes: 196
Allocated: 5371041760, active: 5371461632, metadata: 24531344 (n_thp 0), resident: 5430169600, mapped: 5526704128, retained: 0
Count of realloc(non-null-ptr, 0) calls: 0
Background threads: 0, num_runs: 0, run_interval: 0 ns
--- End jemalloc statistics ---

Draining the last batch
Final Histogram of buffer sizes:
 -- Histogram:
Min value: 0
Max value: 65673429
Mean: 2224296.36
[0 B, 1.0 MiB) 1066 58.60%
[1.0 MiB, 2.0 MiB) 124 6.82%
[2.0 MiB, 4.0 MiB) 232 12.75%
[4.0 MiB, 8.0 MiB) 275 15.12%
[8.0 MiB, 16 MiB) 119 6.54%
[16 MiB, 32 MiB) 1 0.05%
[32 MiB, 64 MiB) 2 0.11%
 --

[23:28:30-0300] REDUCE 01m54s 100.00% edge_count:102.3M edge_speed:4.651M/sec plist_count:18.85M plist_speed:856.8k/sec. Num Encoding MBs: 0. GoMem MB: 561 CMem MB: 8640
[23:28:31-0300] REDUCE 01m55s 100.00% edge_count:102.3M edge_speed:4.449M/sec plist_count:18.85M plist_speed:819.6k/sec. Num Encoding MBs: 0. GoMem MB: 618 CMem MB: 5760
[23:28:32-0300] REDUCE 01m56s 100.00% edge_count:102.3M edge_speed:4.263M/sec plist_count:18.85M plist_speed:785.4k/sec. Num Encoding MBs: 0. GoMem MB: 664 CMem MB: 2880
[23:28:33-0300] REDUCE 01m57s 100.00% edge_count:102.3M edge_speed:4.093M/sec plist_count:18.85M plist_speed:754.0k/sec. Num Encoding MBs: 0. GoMem MB: 702 CMem MB: 240
Writing split lists back to the main DB now
[23:28:33-0300] REDUCE 01m57s 100.00% edge_count:102.3M edge_speed:4.035M/sec plist_count:18.85M plist_speed:743.5k/sec. Num Encoding MBs: 0. GoMem MB: 717 CMem MB: 0
2

Liveloader

Dgraph test on MASTER BRANCH with jemalloc

Zero, Alpha, and Live with default settings.

TMP files on   => WDS500G3XHC-00SJG0
Run on         => Kingston SKC1000240G

[18:25:10-0300] Elapsed: 13m25s Txns: 20954 N-Quads: 20953870 N-Quads/s [last 5s]: 10200 Aborts: 0
[18:25:15-0300] Elapsed: 13m30s Txns: 20995 N-Quads: 20994870 N-Quads/s [last 5s]:  8200 Aborts: 0
[18:25:20-0300] Elapsed: 13m35s Txns: 21048 N-Quads: 21047870 N-Quads/s [last 5s]: 10600 Aborts: 0
[18:25:25-0300] Elapsed: 13m40s Txns: 21105 N-Quads: 21104870 N-Quads/s [last 5s]: 11400 Aborts: 0
[18:25:30-0300] Elapsed: 13m45s Txns: 21157 N-Quads: 21156870 N-Quads/s [last 5s]: 10400 Aborts: 0
[18:25:35-0300] Elapsed: 13m50s Txns: 21204 N-Quads: 21203870 N-Quads/s [last 5s]:  9400 Aborts: 0
Number of TXs run            : 21240
Number of N-Quads processed  : 21239870
Time spent                   : 14m1.081613914s
N-Quads processed per second : 25255

Second test ludicrous_mode on

Dgraph test on MASTER BRANCH with jemalloc

Zero, Alpha, and Live with default settings and ludicrous mode.

Dgraph version   : v20.11.0-gf7c88ee16
Dgraph codename  : unnamed
Dgraph SHA-256   : ac7c59f9424580544a4dd5b1048c2d18b91e90b1481308d08c626c402ce8c15e
Commit SHA-1     : f7c88ee16
Commit timestamp : 2020-09-16 19:20:15 +0530
Branch           : master
Go version       : go1.14.4
jemalloc enabled : true

TMP files on   => WDS500G3XHC-00SJG0
Run on         => Kingston SKC1000240G

The Raft, compaction, and snapshot drag down the whole performance.

[18:28:59-0300] Elapsed: 05s Txns: 1000 N-Quads: 1000000 N-Quads/s [last 5s]: 200000 Aborts: 0
[18:29:04-0300] Elapsed: 10s Txns: 2009 N-Quads: 2009000 N-Quads/s [last 5s]: 201800 Aborts: 0
[18:29:09-0300] Elapsed: 15s Txns: 2978 N-Quads: 2978000 N-Quads/s [last 5s]: 193800 Aborts: 0
[18:29:14-0300] Elapsed: 20s Txns: 3920 N-Quads: 3920000 N-Quads/s [last 5s]: 188400 Aborts: 0
[18:29:19-0300] Elapsed: 25s Txns: 4881 N-Quads: 4881000 N-Quads/s [last 5s]: 192200 Aborts: 0
[18:29:24-0300] Elapsed: 30s Txns: 5700 N-Quads: 5700000 N-Quads/s [last 5s]: 163800 Aborts: 0
[18:29:29-0300] Elapsed: 35s Txns: 6679 N-Quads: 6679000 N-Quads/s [last 5s]: 195800 Aborts: 0
[18:29:34-0300] Elapsed: 40s Txns: 7600 N-Quads: 7600000 N-Quads/s [last 5s]: 184200 Aborts: 0
[18:29:39-0300] Elapsed: 45s Txns: 8374 N-Quads: 8374000 N-Quads/s [last 5s]: 154800 Aborts: 0
[18:29:44-0300] Elapsed: 50s Txns: 9200 N-Quads: 9200000 N-Quads/s [last 5s]: 165200 Aborts: 0
[18:29:49-0300] Elapsed: 55s Txns: 10200 N-Quads: 10200000 N-Quads/s [last 5s]: 200000 Aborts: 0
[18:29:54-0300] Elapsed: 01m00s Txns: 11138 N-Quads: 11138000 N-Quads/s [last 5s]: 187600 Aborts: 0
[18:29:59-0300] Elapsed: 01m05s Txns: 12000 N-Quads: 12000000 N-Quads/s [last 5s]: 172400 Aborts: 0
[18:30:04-0300] Elapsed: 01m10s Txns: 12900 N-Quads: 12900000 N-Quads/s [last 5s]: 180000 Aborts: 0
[18:30:09-0300] Elapsed: 01m15s Txns: 13800 N-Quads: 13800000 N-Quads/s [last 5s]: 180000 Aborts: 0
[18:30:14-0300] Elapsed: 01m20s Txns: 14796 N-Quads: 14796000 N-Quads/s [last 5s]: 199200 Aborts: 0
[18:30:19-0300] Elapsed: 01m25s Txns: 15721 N-Quads: 15721000 N-Quads/s [last 5s]: 185000 Aborts: 0
[18:30:24-0300] Elapsed: 01m30s Txns: 16619 N-Quads: 16619000 N-Quads/s [last 5s]: 179600 Aborts: 0
[18:30:29-0300] Elapsed: 01m35s Txns: 17520 N-Quads: 17520000 N-Quads/s [last 5s]: 180200 Aborts: 0
[18:30:34-0300] Elapsed: 01m40s Txns: 18500 N-Quads: 18500000 N-Quads/s [last 5s]: 196000 Aborts: 0
[18:30:39-0300] Elapsed: 01m45s Txns: 19250 N-Quads: 19250000 N-Quads/s [last 5s]: 150000 Aborts: 0
[18:30:44-0300] Elapsed: 01m50s Txns: 19470 N-Quads: 19470000 N-Quads/s [last 5s]: 44000 Aborts: 0
[18:30:49-0300] Elapsed: 01m55s Txns: 19710 N-Quads: 19710000 N-Quads/s [last 5s]: 48000 Aborts: 0
[18:30:54-0300] Elapsed: 02m00s Txns: 19950 N-Quads: 19950000 N-Quads/s [last 5s]: 48000 Aborts: 0
[18:30:59-0300] Elapsed: 02m05s Txns: 20200 N-Quads: 20200000 N-Quads/s [last 5s]: 50000 Aborts: 0
[18:31:04-0300] Elapsed: 02m10s Txns: 20390 N-Quads: 20390000 N-Quads/s [last 5s]: 38000 Aborts: 0
[18:31:09-0300] Elapsed: 02m15s Txns: 20551 N-Quads: 20551000 N-Quads/s [last 5s]: 32200 Aborts: 0
[18:31:14-0300] Elapsed: 02m20s Txns: 20605 N-Quads: 20605000 N-Quads/s [last 5s]: 10800 Aborts: 0
[18:31:19-0300] Elapsed: 02m25s Txns: 20649 N-Quads: 20649000 N-Quads/s [last 5s]:  8800 Aborts: 0
[18:31:24-0300] Elapsed: 02m30s Txns: 20701 N-Quads: 20701000 N-Quads/s [last 5s]: 10400 Aborts: 0
[18:31:29-0300] Elapsed: 02m35s Txns: 20742 N-Quads: 20742000 N-Quads/s [last 5s]:  8200 Aborts: 0
[18:31:34-0300] Elapsed: 02m40s Txns: 20806 N-Quads: 20806000 N-Quads/s [last 5s]: 12800 Aborts: 0
[18:31:39-0300] Elapsed: 02m45s Txns: 20861 N-Quads: 20861000 N-Quads/s [last 5s]: 11000 Aborts: 0
[18:31:44-0300] Elapsed: 02m50s Txns: 20916 N-Quads: 20916000 N-Quads/s [last 5s]: 11000 Aborts: 0
[18:31:49-0300] Elapsed: 02m55s Txns: 20968 N-Quads: 20968000 N-Quads/s [last 5s]: 10400 Aborts: 0
[18:31:54-0300] Elapsed: 03m00s Txns: 21028 N-Quads: 21028000 N-Quads/s [last 5s]: 12000 Aborts: 0
[18:31:59-0300] Elapsed: 03m05s Txns: 21070 N-Quads: 21070000 N-Quads/s [last 5s]:  8400 Aborts: 0
[18:32:04-0300] Elapsed: 03m10s Txns: 21124 N-Quads: 21124000 N-Quads/s [last 5s]: 10800 Aborts: 0
[18:32:09-0300] Elapsed: 03m15s Txns: 21165 N-Quads: 21165000 N-Quads/s [last 5s]:  8200 Aborts: 0
[18:32:14-0300] Elapsed: 03m20s Txns: 21206 N-Quads: 21206000 N-Quads/s [last 5s]:  8200 Aborts: 0

Number of TXs run            : 21240
Number of N-Quads processed  : 21239870
Time spent                   : 3m27.675780401s
N-Quads processed per second : 102608

Third test ludicrous_mode on

command:
$GOPATH/bin/dgraph alpha --ludicrous_mode --abort_older_than=20m --snapshot_after=20000000

[18:39:00-0300] Elapsed: 05s Txns: 961 N-Quads: 961000 N-Quads/s [last 5s]: 192200 Aborts: 0
[18:39:05-0300] Elapsed: 10s Txns: 1956 N-Quads: 1956000 N-Quads/s [last 5s]: 199000 Aborts: 0
[18:39:10-0300] Elapsed: 15s Txns: 2900 N-Quads: 2900000 N-Quads/s [last 5s]: 188800 Aborts: 0
....
[18:41:35-0300] Elapsed: 02m40s Txns: 20762 N-Quads: 20762000 N-Quads/s [last 5s]:  8400 Aborts: 0
[18:41:40-0300] Elapsed: 02m45s Txns: 20817 N-Quads: 20817000 N-Quads/s [last 5s]: 11000 Aborts: 0
[18:41:45-0300] Elapsed: 02m50s Txns: 20867 N-Quads: 20867000 N-Quads/s [last 5s]: 10000 Aborts: 0
[18:41:50-0300] Elapsed: 02m55s Txns: 20917 N-Quads: 20917000 N-Quads/s [last 5s]: 10000 Aborts: 0
[18:41:55-0300] Elapsed: 03m00s Txns: 20977 N-Quads: 20977000 N-Quads/s [last 5s]: 12000 Aborts: 0
[18:42:00-0300] Elapsed: 03m05s Txns: 21027 N-Quads: 21027000 N-Quads/s [last 5s]: 10000 Aborts: 0
[18:42:05-0300] Elapsed: 03m10s Txns: 21067 N-Quads: 21067000 N-Quads/s [last 5s]:  8000 Aborts: 0
[18:42:10-0300] Elapsed: 03m15s Txns: 21117 N-Quads: 21117000 N-Quads/s [last 5s]: 10000 Aborts: 0
[18:42:15-0300] Elapsed: 03m20s Txns: 21169 N-Quads: 21169000 N-Quads/s [last 5s]: 10400 Aborts: 0
[18:42:20-0300] Elapsed: 03m25s Txns: 21210 N-Quads: 21210000 N-Quads/s [last 5s]:  8200 Aborts: 0
Number of TXs run            : 21240
Number of N-Quads processed  : 21239870
Time spent                   : 3m32.215832788s
N-Quads processed per second : 100188