xxm404
July 31, 2020, 7:55am
1
I`m doing a benchmark for Dgraph, my cluster consists of 3alpha and 3 zero, alpha deploy in AWS m5.large instance, zero deploy in AWS t3.medium instance, my test request include 1 query and 2 mutate. My cluster just have 40qps throughput, benchmark result like this
1. wrk parameters： wrk2 -t5 -c20 -R40 -d2m
Thread Stats Avg Stdev Max +/- Stdev
Latency 363.30ms 196.14ms 1.43s 73.15%
Req/Sec 19.42 4.29 31.00 77.21%
Latency Distribution (HdrHistogram - Recorded Latency)
50.000% 329.73ms
75.000% 453.38ms
90.000% 606.21ms
99.000% 1.06s
99.900% 1.24s
99.990% 1.44s
99.999% 1.44s
100.000% 1.44s
#[Mean = 363.302, StdDeviation = 196.141]
#[Max = 1434.624, Total count = 4384]
#[Buckets = 27, SubBuckets = 2048]
----------------------------------------------------------
4785 requests in 2.00m, 1.38MB read
Requests/sec: 39.86
Transfer/sec: 11.79KB
alpha cpu %48 iops 366
zero cpu %34 iops 244
2. wrk parameters： wrk2 -t5 -c40 -R80 -d2m
50.000% 973.82ms
75.000% 1.11s
90.000% 1.26s
99.000% 1.54s
99.900% 1.70s
99.990% 1.85s
99.999% 1.85s
100.000% 1.85s
#[Mean = 978.974, StdDeviation = 217.198]
#[Max = 1853.440, Total count = 4508]
#[Buckets = 27, SubBuckets = 2048]
----------------------------------------------------------
4973 requests in 2.00m, 1.44MB read
Requests/sec: 41.42
Transfer/sec: 12.26KB
alpha cpu 46% iops 277
zero cpu 46% iops 156
why are resources not being fully utilized？
@dmitry did a benchmark as well.
Did I hit 1B+ Transactions?
Maybe the link can help!
xxm404
July 31, 2020, 9:07am
3
@abhijit-kar Thanks for the reply, This is my query result
{"data":{"code":"Success","message":"Done","uids":[]},"extensions":{"server_latency":{"parsing_ns":30952,"processing_ns":1583801668},"txn":{"start_ts":415734,"commit_ts":415798,"preds":["1-ww\/attr\/enterFee","1-ww\/attr\/socialLevel","1-ww\/attr\/storeTime","1-_predicate_","1-ww\/attr\/manageLevel","1-ww\/attr\/tryFlag"]}}}
processing_ns:1583801668 looks like too long, but my dgraph instance load is not very high.
seanlaff
July 31, 2020, 1:29pm
5
What do your inserts look like with regards to predicates? Ie do you have lots of predicates with high cardinality of values, or just a couple predicates that you write to heavily?
xxm404
August 4, 2020, 4:20am
6
@seanlaff Thanks for the reply，my scheme like this
<xid>:string @index(hash) @upsert .
<rel1>:uid @reverse .
<rel2>:uid @reverse .
<attr1>:int @index(int) .
<attr2>:int @index(int) .
<attr3>:int @index(int) .
<attr4>:int @index(int) .
<attr5>:int @index(int) .
<attr6>:int @index(int) .
<attr7>:int @index(int) .
My test request process is
1. check xid exists
2. create node
3. set attr
Dgraph looks like just can provide 40qps when I scale dgraph instance（more vpu,more mem）, its still just provide 40qps,im confused.
pawan
August 4, 2020, 6:14am
7
xxm404:
wrk2 -t5 -c20 -R40 -d2m
What do these parameters mean?
Also have you tried increasing the concurrency for your requests? Can you share the exact queries and mutations with as script on how to reproduce these results? The throughput numbers depend on your queries/mutations and the machine that you are using. If you can share exact steps to reproduce then we could help here.
xxm404
August 4, 2020, 10:13am
8
@pawan Thanks for the reply.
wrk2 -t5 -c20 -R40 -d2m
wrk2 is a modern HTTP benchmarking tool. wrk2
Parameters mean :
-t, --threads <N> Number of threads to use
-c, --connections <N> Connections to keep open
-d, --duration <T> Duration of test
-R, --rate <T> work rate (throughput)
in requests/sec (total)
[Required Parameter]
Test model:
wrk2 -> PHP service -> dgraph
Test env:
dgraph v1.0.16 deploy in AWS m5d.xlarge( single host )
m5d.xlarge instance has 130G NVMe local storge
Test scheme:
<xid>:string @index(hash) @upsert .
<rel1>:uid @reverse .
<rel2>:uid @reverse .
<attr1>:int @index(int) .
<attr2>:int @index(int) .
<attr3>:int @index(int) .
<attr4>:int @index(int) .
<attr5>:int @index(int) .
<attr6>:int @index(int) .
<attr7>:int @index(int) .
Test request include query/mutate:
Start a transaction
curl -H "Content-Type: application/graphql+-" -X POST 172.30.29.135:8087/query -d '
{
q(func: eq(xid, ["ww_u_167507121152"])) { uid xid }
}
'
curl -H "X-Dgraph-MutationType: json" -X POST 172.30.29.135:8087/mutate?startTs=0 -d '
{
"set":[
{
"xid":"ww_u_167507121152",
"uid":"_:ww_u_167507121152"
}
]
}
'
Commit transaction
curl -H "Content-Type: application/graphql+-" -X POST 172.30.29.135:8087/query -d '
{
q(func: eq(xid, ["ww_u_167507121152"])) { uid xid }
}
'
curl -H "X-Dgraph-MutationType: json" -X POST 172.30.29.135:8087/mutate?startTs=0 -d '
{
"set":[
{
"attr1":49800,
"attr2":100,
"attr3":100,
"attr4":0,
"attr5":1596526586,
"uid":"0x9d99f4"
}
]
}
Test result:
wrk2 -t2 -c5 -R80 -d2m
Latency Distribution (HdrHistogram - Uncorrected Latency (measured without taking delayed starts into account))
50.000% 66.50ms
75.000% 84.42ms
90.000% 101.57ms
99.000% 139.01ms
99.900% 536.06ms
99.990% 1.23s
99.999% 1.31s
100.000% 1.31s
----------------------------------------------------------
7141 requests in 2.00m, 1.89MB read
Requests/sec: 59.50
Transfer/sec: 16.15KB
cpu 40%
DiskWriteOps 1500
You can try to reproduce. Thank you so much.
pawan
August 4, 2020, 11:18am
9
So I noticed that you specify R (rate) as 40/sec and you are getting the same as throughput which seems good to me. Maybe you should try with larger values of R to test the limits here?
Also you seem to be using a really old version of Dgraph. I’d recommend using the latest version as we would have made many improvements since 1.0.16.
xxm404
August 5, 2020, 12:36pm
10
@pawan When i change R to 80, Test result:
wrk2 -t2 -c5 -R80 -d2m
Latency Distribution (HdrHistogram - Uncorrected Latency (measured without taking delayed starts into account))
50.000% 66.50ms
75.000% 84.42ms
90.000% 101.57ms
99.000% 139.01ms
99.900% 536.06ms
99.990% 1.23s
99.999% 1.31s
100.000% 1.31s
----------------------------------------------------------
7141 requests in 2.00m, 1.89MB read
Requests/sec: 59.50
Transfer/sec: 16.15KB
cpu 40%
DiskWriteOps 1500
cpu and disk does not appear to be a bottleneck,
I will run the same test case at latest version.