Slow data ingestion?

Users Dgraph
#1

Cloud: AKS
K8S version: 1.20.7
Alpha nodes: 64 GB, NVMe SSD 1.9 T (Standard_L8s_v2 SKU)
Cluster setup: one zero, 3 alpha
Alpha replica count: 3

Live Upload command snippet

dgraph live --files /root/dgraph/data/1642449041 --schema /root/dgraph/schema/my_schema.rdf --alpha host.docker.internal:9080 --zero host.docker.internal:5080 --format=rdf --upsertPredicate "xid" -b 1000

Simple (165MB of RDFs) test of Live Uploader stats
Number of TXs run : 55757
Number of N-Quads processed : 55670191
Time spent : 2h9m22.150593615s
N-Quads processed per second : 7172

Should this be considered slow?

#2

Yea seems slow for that hardware. host.docker.internal what is this? Doesn’t seem like how you might do this from kubernetes. Are you running all 4 of these pods all on the same node or something?

#3

host.docker.internal this is required because, I’m using
kubectl port-forward from local machine.

Also, I tried to use Live uploader from zero. It errored out with below message

Error while processing schema file "my_scheme.rdf": rpc error: code = Unknown desc = unauthorized ip address: 10.155.2.4
rpc error: code = Unknown desc = unauthorized ip address: 10.155.2.4#

Per this [post]( How to live load data from a different server on alpha node hosted on EKS - Users / Dgraph - Discuss Dgraph), do whitelist when using live uploader from zero?

#4

Yea I assume your slowness is because of the port-forward and WAN (most likely)

The whitelist is necessary to perform admin tasks on the alphas, so you can either add the whitelist for your pod/service subnet or run this from an alpha.