Performance Testing Stability Issues with v21.12.0

Issues Dgraph
1

I just started testing with version 21.12.0 this week. I’m trying to use Live Loader with the 21million dataset from the benchmarks repository. It seems like gets about halfway through (much faster than 21.0.3) but then one of the Alpha processes dies.

Has anybody else been able to successfully test with this data set?

I also see this error in my alpha logs:

Running node command: node t/index.js
E0218 21:47:25.284928   25414 run.go:535] Failed to start lambda server at port: 20000. Got err: exec: "node": executable file not found in $PATH

I attempted to resolve this by installing nodejs on my alpha servers (which wasn’t required for 21.0.3), but the alpha.log becomes full of node.js code and what looks like Chinese characters, making the log unreadable.

What version of Dgraph are you using?

Dgraph Version

Dgraph version : v21.12.0
Dgraph codename : zion
Dgraph SHA-256 : 078c75df9fa1057447c8c8afc10ea57cb0a29dfb22f9e61d8c334882b4b4eb37
Commit SHA-1 : d62ed5f15
Commit timestamp : 2021-12-02 21:20:09 +0530
Branch : HEAD
Go version : go1.17.3
jemalloc enabled : true

Have you tried reproducing the issue with the latest release?

This occurs with 21.12.0. It works with the previous version 21.0.3

What is the hardware spec (RAM, OS)?

Linux AWS c5.2xlarge - 16GB RAM

Steps to reproduce the issue (command/config used to run Dgraph).

dgraph live
–files $DATADIR/21million.json.gz
–schema $DATADIR/release/release.schema
–alpha $ALPHALEADER
–zero $ZEROLEADER --conc 20

Expected behaviour and actual result.

Expecting to see the data successfully loaded, but the alpha process dies, resulting in:

Connection has been possibly interrupted. Got error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp <alphaleaderip>:9080: connect: connection refused". Will retry after 9s.

1 Like
2

Additional information -

After the Alpha process dies, Live Loader continues to try reconnecting. So I went ahead and restarted the Alpha process on the node, which results in the following from Live:

Error while mutating: 429 Too Many Requests. Please throttle your requests s.Code Unknown

3

Here’s some information from the alpha log when the process dies:


I0218 22:52:20.543980    9941 log.go:34] L0 was stalled for 2.664s
I0218 22:52:20.600014    9941 log.go:34] [0] [E] LOG Compact 0->4 (15, 5 -> 14 tables with 2 splits). [10848 10872 10879 10883 10888 10892 10893 10896 10899 10901 10904 10908 10914 10916 10919 . 10866 10849 10851 10853 10860 .] -> [10924 10926 10929 10931 10933 10935 10940 10941 10942 10946 10947 10923 10928 10930 .], took 2.521s
I0218 22:52:22.054801    9941 run.go:533] Running node command: node t/index.js
E0218 22:52:22.074723    9941 run.go:535] Failed to start lambda server at port: 20000. Got err: exec: "node": executable file not found in $PATH
I0218 22:52:22.399149    9941 log.go:34] [3] [E] LOG Compact 5->6 (1, 8 -> 9 tables with 3 splits). [10812 . 10346 10378 10396 10368 10409 10419 10377 10591 .] -> [10927 10939 10948 10949 10937 10961 10983 10936 10963 .], took 4.446s
I0218 22:52:23.120812    9941 log.go:34] [2] [E] LOG Compact 4->5 (1, 9 -> 9 tables with 3 splits). [10926 . 10903 10909 10910 10902 10915 10921 10922 10885 10890 .] -> [10959 10971 10977 10966 10989 10993 10996 10965 10984 .], took 2.577s
I0218 22:52:24.265717    9941 run.go:533] Running node command: node t/index.js
E0218 22:52:24.266413    9941 run.go:535] Failed to start lambda server at port: 20000. Got err: exec: "node": executable file not found in $PATHI0218 22:52:25.742481    9941 log.go:34] L0 was stalled for 3.402s
I0218 22:52:25.742358    9941 log.go:34] [0] [E] LOG Compact 5->6 (1, 11 -> 11 tables with 4 splits). [10834 . 10747 10760 10761 10736 10750 10759 10741 10754 10654 10674 10994 .] -> [11006 11020 11023 11001 11010 11016 11005 11017 11025 11009 11011 .], took 2.844s
I0218 22:52:26.121342    9941 log.go:34] [2] [E] LOG Compact 5->6 (1, 1 -> 1 tables with 1 splits). [10958 . 10932 .] -> [11028 .], took 3s
I0218 22:52:26.390401    9941 run.go:533] Running node command: node t/index.js
E0218 22:52:26.390440    9941 run.go:535] Failed to start lambda server at port: 20000. Got err: exec: "node": executable file not found in $PATH
I0218 22:52:26.862005    9941 log.go:34] [3] [E] LOG Compact 5->6 (1, 4 -> 5 tables with 2 splits). [10900 . 10340 10344 10343 10365 .] -> [11018 11029 11043 11019 11022 .], took 3.493s
I0218 22:52:28.454762    9941 run.go:533] Running node command: node t/index.js
E0218 22:52:28.468463    9941 run.go:535] Failed to start lambda server at port: 20000. Got err: exec: "node": executable file not found in $PATH
I0218 22:52:29.803966    9941 log.go:34] [2] [E] LOG Compact 5->6 (1, 4 -> 4 tables with 2 splits). [10989 . 10474 10475 09883 09879 .] -> [11070 11088 11076 11078 .], took 2.737s
I0218 22:52:30.098650    9941 groups.go:493] Serving tablet for: 0-writer.film
I0218 22:52:30.105668    9941 groups.go:493] Serving tablet for: 0-writer.film
I0218 22:52:30.106759    9941 groups.go:493] Serving tablet for: 0-writer.film
I0218 22:52:30.107235    9941 groups.go:493] Serving tablet for: 0-writer.film
I0218 22:52:30.108805    9941 groups.go:493] Serving tablet for: 0-writer.film
I0218 22:52:30.294498    9941 log.go:34] Handover found 186 B data in current memtable. Pushing to flushChan.
I0218 22:52:30.494693    9941 run.go:533] Running node command: node t/index.js
E0218 22:52:30.494742    9941 run.go:535] Failed to start lambda server at port: 20000. Got err: exec: "node": executable file not found in $PATH
I0218 22:52:30.589382    9941 log.go:34] [3] [E] LOG Compact 5->6 (1, 7 -> 6 tables with 3 splits). [10907 . 11019 11022 10370 10118 10164 10171 10175 .] -> [11071 11080 11074 11093 11101 11059 .], took 3.727s
I0218 22:52:32.592169    9941 run.go:533] Running node command: node t/index.js
E0218 22:52:32.592975    9941 run.go:535] Failed to start lambda server at port: 20000. Got err: exec: "node": executable file not found in $PATH
I0218 22:52:33.063174    9941 log.go:34] L0 was stalled for 3.252s