Context deadline exceeded using LiveLoader (DGraph in AWS server)

Report a Dgraph Bug

What version of Dgraph are you using?

Dgraph Version: v20.11.2
$ dgraph version
 
PASTE YOUR RESULTS HERE

Have you tried reproducing the issue with the latest release?

Yes

What is the hardware spec (RAM, OS)?

Linux/UNIX - AWS Instance
4 vCPUs , Memory = 16Gb

Steps to reproduce the issue (command/config used to run Dgraph).

I created a network in my local machine that worked perfectly with LiveLoader, loading a RDF file of ~10Gb. When reproducing the same steps in DGraph installed in a AWS instance (same DGraph version, same schema, same RDF file) I got the error of “Context deadline exceeded”.

Schema:

<card.creation_time>: string @index(exact) .
<card.fingerprint>: string @index(exact) .
<customer.creation_time>: string @index(exact) .
<customer.id>: int @index(int) .
<device.creation_time>: string @index(exact) .
<device.fingerprint>: string @index(exact) .
<dgraph.drop.op>: string .
<dgraph.graphql.p_query>: string @index(sha256) .
<dgraph.graphql.schema>: string .
<dgraph.graphql.xid>: string @index(exact) @upsert .
<email.address>: string @index(exact) .
<email.creation_time>: string @index(exact) .
<link_customer_card>: [uid] @count @reverse .
<link_customer_device>: [uid] @count @reverse .
<link_customer_email>: [uid] @count @reverse .
<link_customer_phone>: [uid] @count @reverse .
<link_customer_promo>: [uid] @count @reverse .
<link_referee_mgm>: [uid] @count @reverse .
<link_referred_mgm>: [uid] @count @reverse .
<mgm.creation_time>: string @index(exact) .
<mgm.id>: int @index(int) .
<mgm.referee>: int @index(int) .
<mgm.referred>: int @index(int) .
<phone.creation_time>: string @index(exact) .
<phone.number>: string @index(exact) .
<promo.creation_time>: string @index(exact) .
<promo.id>: int @index(int) .
type <card> {
	card.creation_time
	card.fingerprint
	~link_customer_card
}
type <customer> {
	customer.creation_time
	customer.id
	link_customer_phone
	link_customer_email
	link_customer_device
	link_customer_promo
	link_referee_mgm
	link_referred_mgm
	link_customer_card
}
type <device> {
	device.creation_time
	device.fingerprint
	~link_customer_device
}
type <dgraph.graphql> {
	dgraph.graphql.schema
	dgraph.graphql.xid
}
type <dgraph.graphql.persisted_query> {
	dgraph.graphql.p_query
}
type <email> {
	email.address
	email.creation_time
	~link_customer_email
}
type <mgm> {
	mgm.id
	mgm.referee
	mgm.referred
	mgm.creation_time
	~link_referee_mgm
	~link_referred_mgm
}
type <phone> {
	phone.creation_time
	phone.number
	~link_customer_phone
}
type <promo> {
	promo.id
	promo.creation_time
	~link_customer_promo
}

All ports of the AWS instance are open and listening, but for some reason when running LiveLoader with the exact same RDF file, the error triggers.
The command call used is:
dgraph live -f filename.rdf -a http://IP:9080

Just in case, I also tried with all combinations of ports of alpha and zero, despite LiveLoader works with alpha:9080 and zero:5080:
dgraph live -f filename.rdf -a http://IP:9080 -z http://IP:5080

The outcome of the call is always this one:

[Decoder]: Using assembly version of decoder
Page Size: 4096
I0707 10:13:08.578815    1770 init.go:107] 

Dgraph version   : v20.11.2
Dgraph codename  : tchalla-2
Dgraph SHA-256   : ccbae03a5c877ba24f501940e4e08d1897f928d0c6579a534148d36749149f9b
Commit SHA-1     : 94f3a0430
Commit timestamp : 2021-02-23 13:07:17 +0530
Branch           : HEAD
Go version       : go1.15.5
jemalloc enabled : false

For Dgraph official documentation, visit https://dgraph.io/docs/.
For discussions about Dgraph     , visit http://discuss.dgraph.io.

Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2020 Dgraph Labs, Inc.



Running transaction with dgraph endpoint: http://IP:9080
While trying to setup connection: context deadline exceeded. Retrying...
2021/07/07 10:13:19 Could not setup connection after 1 retries
github.com/dgraph-io/dgraph/x.Fatalf
	/ext-go/1/src/github.com/dgraph-io/dgraph/x/error.go:120
github.com/dgraph-io/dgraph/x.GetDgraphClient
	/ext-go/1/src/github.com/dgraph-io/dgraph/x/x.go:926
github.com/dgraph-io/dgraph/dgraph/cmd/live.run
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/run.go:632
github.com/dgraph-io/dgraph/dgraph/cmd/live.init.0.func1
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/run.go:127
github.com/spf13/cobra.(*Command).execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:72
main.main
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/main.go:102
runtime.main
	/usr/local/go/src/runtime/proc.go:204
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1374

Example RDF:

# -- CUSTOMER

	_:customer1 <customer.id> “111111” .

# -- PHONE

	_:phone3 <phone.number> “333333” .
	_:phone3 <dgraph.type> "phone" .
	_:customer1 <link_customer_phone> _:phone3 (since="2021-01-01") .

# -- EMAIL

	_:email5emailcom <email.address> "5@email.com" .
	_:email5emailcom <dgraph.type> "email" .
	_:customer1 <link_customer_email> _:email5emailcom (since="2021-01-13") .

Expected behaviour and actual result.

LiveLoader finished successfully, since both schema and RDF file are exactly the same ones than in local machine, were it worked perfectly.

Let me know if I forgot something (I can provide a piece of the RDF file . I’ve been trying to troubleshoot this for a while without any luck, so any advise or help would be much appreciated.

Thanks in advance and keep up the good work DGraph team!

1 Like

This means that the cluster isn’t reachable. Check the address used. Check if you need to add some firewall rule, of if you need some DNS config, or expose in Docker(?).

Hey @MichelDiz, many thanks for your suggestions and apologies for the (super) late reply, crazy days. There are no firewalls configured and ports were already exposed for the set of IPs we’re using. In fact I can successfully connect to Ratel remotely and change stuff there, the problem for some reason is just in the LiveLoader. Not sure if it could be a shady configuration issue…
I’ll reach back in case I find a solution.