Liveloader hangs on large dataset

Issues Dgraph
1

What I want to do

I want to use “dgraph live” tool to load my JSON data.

What I did

I ran the v20.11.2 Linux binary “dgraph” tool with 4.8K json file in upsertPredicate mode and it worked just fine:

"dgraph live -a 1.2.3.4:1234 -z 1.2.3.4:1234 -U “xid” -f data.json -s schema_with_xid.txt

I ran this same command but with 5M nodes (and their properties/edges) and it prints a status message similar to the following endlessly:

[19:24:14Z] Elapsed: 02h15m45s Txns: 0 N-Quads: 0 N-Quads/s [last 5s]: 0 Aborts: 0

Dgraph metadata

dgraph version

[Decoder]: Using assembly version of decoder

Page Size: 4096

Dgraph version : v20.11.2

Dgraph codename : tchalla-2

Dgraph SHA-256 : 0153cb8d3941ad5ad107e395b347e8d930a0b4ead6f4524521f7a525a9699167

Commit SHA-1 : 94f3a0430

Commit timestamp : 2021-02-23 13:07:17 +0530

Branch : HEAD

Go version : go1.15.5

jemalloc enabled : true

For Dgraph official documentation, visit https://dgraph.io/docs/.

For discussions about Dgraph , visit http://discuss.dgraph.io.

Licensed variously under the Apache Public License 2.0 and Dgraph Community License.

Copyright 2015-2020 Dgraph Labs, Inc.

2

Can you share the real values? If it is not a public IP it is fine to share. The ports are truly important to see. Also, please share a sample of your data, to understand if it is compatible with Dgraph.

3

I updated this to be an issue and changed the title. Again, I echo @MichelDiz - it would help us reproduce the issue if we could have the data file.

4

Regarding ports, we have a basic GKE deployment with 3 alphas, 1 zero, and internal LBs in front of the pods. The IPs/ports presumably are fine because the data load with small JSON file worked.

I can’t share data. Perhaps I can run the tool locally in debug mode to provide more info?

5

Don’t need to be the real data. Just a similar one.

What tool? I didn’t know that Liveloader has a debug mode.

6

@nitsuj were you able to resolve the issue? I am facing similar issue.

7

@aditya_shelar,

This is a pretty old thread. May be best to create a new topic with your exact command and the output from your attempt. Also consider adding the --verbose flag to the live loader