How to import data into new cluster

Soultrans · September 22, 2020, 2:06am

By the way,How can I import data on a new cluster:

docker exec -it dgraph dgraph live -f 1million.rdf.gz --alpha localhost:9081 --zero localhost:5081 -c 1

That doesn’t seem to work

rajas · September 22, 2020, 5:37am

Linking a related issue, Live loader not working in docker .
The problem seems to be that docker runs its separate network in which containers are accessible using their service names and not localhost.
Let us know if you face any other issues while running live loader with docker.

Soultrans · September 22, 2020, 7:28am

OK,many thanks!
By the way, in “volumes: - /tmp/data:/dgraph” what does “/dgraph” mean？Isn’t it a containerFile?
Why can’t I change it to “test”?

# This Docker Compose file can be used to quickly bootup Dgraph Zero
# and Alpha in different Docker containers.

# It mounts /tmp/data on the host machine to /dgraph within the
# container. You can change /tmp/data to a more appropriate location.
# Run `docker-compose up` to start Dgraph.

version: "3.2"
services:
  zero:
    image: dgraph/dgraph:latest
    volumes:
      - /tmp/data:/dgraph
    ports:
      - 5080:5080
      - 6080:6080
    restart: on-failure
    command: dgraph zero --my=zero:5080
  alpha:
    image: dgraph/dgraph:latest
    volumes:
      - /tmp/data:/dgraph
    ports:
      - 8080:8080
      - 9080:9080
    restart: on-failure
    command: dgraph alpha --my=alpha:7080 --lru_mb=2048 --zero=zero:5080
  ratel:
    image: dgraph/dgraph:latest
    ports:
      - 8000:8000
    command: dgraph-ratel

Soultrans · September 22, 2020, 8:11am

It doesn’t seem to work either.

dgraph live -f 1million.rdf.gz -s schema -a alphatest:9082

server@server-dgraph:~/dgraph$ dgraph live -f 1million.rdf.gz -s schema -a alphatest:9082
[Decoder]: Using assembly version of decoder
I0922 16:06:19.387219   24053 init.go:99] 

Dgraph version   : v20.03.3
Dgraph SHA-256   : 08424035910be6b6720570427948bab8352a0b5a6d59a0d20c3ec5ed29533121
Commit SHA-1     : fa3c19120
Commit timestamp : 2020-06-02 16:47:25 -0700
Branch           : HEAD
Go version       : go1.14.1

For Dgraph official documentation, visit https://docs.dgraph.io.
For discussions about Dgraph     , visit http://discuss.dgraph.io.
To say hi to the community       , visit https://dgraph.slack.com.

Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2020 Dgraph Labs, Inc.



Running transaction with dgraph endpoint: alphatest:9082
While trying to setup connection: context deadline exceeded. Retrying...
2020/09/22 16:06:30 Could not setup connection after 1 retries
github.com/dgraph-io/dgraph/x.Fatalf
	/ext-go/1/src/github.com/dgraph-io/dgraph/x/error.go:114
github.com/dgraph-io/dgraph/x.GetDgraphClient
	/ext-go/1/src/github.com/dgraph-io/dgraph/x/x.go:793
github.com/dgraph-io/dgraph/dgraph/cmd/live.run
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/run.go:440
github.com/dgraph-io/dgraph/dgraph/cmd/live.init.0.func1
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/run.go:120
github.com/spf13/cobra.(*Command).execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:70
main.main
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/main.go:78
runtime.main
	/usr/local/go/src/runtime/proc.go:203
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1373

rajas · September 22, 2020, 10:42am

Hi @Soultrans,

The /dgraph is the folder inside docker container where /tmp/data from the host machine gets mounted. You may read more about docker volumes here.

The error which you are facing looks similar to the linked issue. Is the alpha node being run as at alphatest:9082 . As per the docker compose file provide, it looks like this should be alpha:9080 instead.

Soultrans · September 22, 2020, 11:17am

Thanks,@Rajas
I’m running multiple clusters on the same server.Now I would like to use the data provided by the official
And the docker compose file as follows：

version: "3.2"
services:
  zerotest:
    image: dgraph/dgraph:latest
    volumes:
      - /home/server/dgraph_data/test:/dgraph
    ports:
      - 5082:5082
      - 6082:6082
    restart: on-failure
    command: dgraph zero --my=zerotest:5082 -o 2
  alphatest:
    image: dgraph/dgraph:latest
    volumes:
      - /home/server/dgraph_data/test:/dgraph
    ports:
      - 8082:8082
      - 9082:9082
    restart: on-failure
    command: dgraph alpha --my=alphatest:7082 -o 2 --lru_mb=2048 --zero=zerotest:5082
  rateltest:
    image: dgraph/dgraph:latest
    ports:
      - 8002:8002
    command: dgraph-ratel

So,I just want to import this data on port 9082 .

rajas · September 22, 2020, 11:49am

Are you able to access ratel on localhost:8002 . I see that it has not been provided with any additional flags to point it to a running alpha node.

Also, how are you running dgraph live command. From host ? or from the docker container ?

Soultrans · September 22, 2020, 11:53am

I can’t access ratel on localhost:8002. I’m running dgraph live on host.

rajas · September 22, 2020, 11:57am

Can you try running dgraph live in one of the containers ? or by using docker exec from the host machine ?

ratel is not accessible because by default it tries to reach dgraph server on localhost:8080. You will have to provide it the location of alpha server using --addr flag along with the command for dgraph ratel.

Soultrans · September 22, 2020, 12:02pm

I want to achieve this by running in host, as I am not very familiar with Docker.
And how can I access it at 8002?
Just like this？

command: dgraph alpha --my=alphatest:7082 -o 2 --lru_mb=2048 --zero=zerotest:5082

I can access port 8082、9082，port 8002 cannot be accessed.Why is this happening
Running inside the container：
“zero” should be “zerotest” ,it should run at 127.0.0.1:5082

root@93e0aa005bfa:/dgraph# dgraph live -f 1million.rdf.gz -s schema -a alphatest:9082
[Decoder]: Using assembly version of decoder
I0922 12:23:39.404698      53 init.go:99] 

Dgraph version   : v20.03.3
Dgraph SHA-256   : 08424035910be6b6720570427948bab8352a0b5a6d59a0d20c3ec5ed29533121
Commit SHA-1     : fa3c19120
Commit timestamp : 2020-06-02 16:47:25 -0700
Branch           : HEAD
Go version       : go1.14.1

For Dgraph official documentation, visit https://docs.dgraph.io.
For discussions about Dgraph     , visit http://discuss.dgraph.io.
To say hi to the community       , visit https://dgraph.slack.com.

Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2020 Dgraph Labs, Inc.
Running transaction with dgraph endpoint: alphatest:9082
2020/09/22 12:23:49 context deadline exceeded
Unable to connect to zero, Is it running at 127.0.0.1:5080?
github.com/dgraph-io/dgraph/x.Checkf
	/ext-go/1/src/github.com/dgraph-io/dgraph/x/error.go:51
github.com/dgraph-io/dgraph/dgraph/cmd/live.setup
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/run.go:383
github.com/dgraph-io/dgraph/dgraph/cmd/live.run
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/run.go:443
github.com/dgraph-io/dgraph/dgraph/cmd/live.init.0.func1
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/run.go:120
github.com/spf13/cobra.(*Command).execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:70
main.main
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/main.go:78
runtime.main
	/usr/local/go/src/runtime/proc.go:203
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1373

rajas · September 22, 2020, 12:43pm

By default the live loader is trying to connect to zero at 127.0.0.1:5080 . Can you pass the correct address of zero (zerotest:5082) using --zero flag to the live loader.

For more information on dgraph live loader flags, you may run dgraph live --help .

Soultrans · September 22, 2020, 12:54pm

dgraph live -f 1million.rdf.gz -s schema -a alphatest:9082 -z zerotest:5082

root@93e0aa005bfa:/dgraph# dgraph live -f 1million.rdf.gz -s schema -a alphatest:9082 -z zerotest:5082
[Decoder]: Using assembly version of decoder
I0922 12:57:17.709947     142 init.go:99] 

Dgraph version   : v20.03.3
Dgraph SHA-256   : 08424035910be6b6720570427948bab8352a0b5a6d59a0d20c3ec5ed29533121
Commit SHA-1     : fa3c19120
Commit timestamp : 2020-06-02 16:47:25 -0700
Branch           : HEAD
Go version       : go1.14.1

For Dgraph official documentation, visit https://docs.dgraph.io.
For discussions about Dgraph     , visit http://discuss.dgraph.io.
To say hi to the community       , visit https://dgraph.slack.com.

Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2020 Dgraph Labs, Inc.



Running transaction with dgraph endpoint: alphatest:9082

Processing schema file "schema"
2020/09/22 12:57:17 open schema: no such file or directory

That doesn’t seem to work

rajas · September 22, 2020, 1:00pm

@Soultrans, The error is different this time. The error now is that the live loader was not able to find schema file . Can you check if there exists a schema file at /dgraph/schema inside the container from where this command is being run ?
Can you point the right location of schema file to -s flag.

Soultrans · September 22, 2020, 1:03pm

there isn’t exists a schema file at /dgraph/schema inside the container.
I downloaded the data set ”1million.rdf.gz“ from the official website，I just want to import it.
Like this example

rajas · September 22, 2020, 2:25pm

In that case, don’t provide -s flag.

You may read more details about live loader over here .

Soultrans · September 23, 2020, 2:15am

Thanks @rajas
I still want to know how to connect to port 8002.Can you give me some concrete examples?

By the way,is this success?Why can’t I see any data at Ratel.Or is it an enterprise feature?

root@93e0aa005bfa:/dgraph# dgraph live -f 1million.rdf.gz  -a alphatest:9082 -z zerotest:5082
[Decoder]: Using assembly version of decoder
I0923 02:35:09.423704     202 init.go:99] 

Dgraph version   : v20.03.3
Dgraph SHA-256   : 08424035910be6b6720570427948bab8352a0b5a6d59a0d20c3ec5ed29533121
Commit SHA-1     : fa3c19120
Commit timestamp : 2020-06-02 16:47:25 -0700
Branch           : HEAD
Go version       : go1.14.1

For Dgraph official documentation, visit https://docs.dgraph.io.
For discussions about Dgraph     , visit http://discuss.dgraph.io.
To say hi to the community       , visit https://dgraph.slack.com.

Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2020 Dgraph Labs, Inc.



Running transaction with dgraph endpoint: alphatest:9082
Found 1 data file(s) to process
Processing data file "1million.rdf.gz"
2020/09/23 02:35:09 EOF

github.com/dgraph-io/dgraph/x.Check
	/ext-go/1/src/github.com/dgraph-io/dgraph/x/error.go:42
github.com/dgraph-io/dgraph/chunker.FileReader
	/ext-go/1/src/github.com/dgraph-io/dgraph/chunker/chunk.go:372
github.com/dgraph-io/dgraph/dgraph/cmd/live.(*loader).processFile
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/run.go:257
github.com/dgraph-io/dgraph/dgraph/cmd/live.run.func2
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/run.go:482
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1373

MichelDiz · September 23, 2020, 4:01am

Hey, your YAML sets Ratel to - 8002:8002 which

Should not be changed at all, as Ratel have nothing to do with Dgraph’s port offset.
You should use the kind of forwarding in the docker compose. (HOST:CONTAINER) so, the port on your YAML should be - 8002:8000. Therefore, as Ratel is being served in the port 8000, not 8002, you got nothing in the port combination you have used.

Nope.

There is no enterprise feature on your context. We have only

Nothing more. This error you got means something else.

Delete the file and download it again. Feels like it is currupted.

Soultrans · September 23, 2020, 5:39am

I did this operation

wget “https://github.com/dgraph-io/tutorial/blob/master/resources/1million.rdf.gz?raw=true” -O 1million.rdf.gz -q

I still don’t see what you mean @MichelDiz .Isn’t the data from the official source？

MichelDiz · September 23, 2020, 6:54am

which part?

yes, it is. Sometimes we can have bad packages from the internet and create a corrupted file. But that is probably not the case. Just a guess.

If you have a way to reproduce this would be nice. Is quite hard to tell by “EOF”, which in Unix like systems means “Bad End of File”.

Soultrans · September 23, 2020, 7:04am

Well, I’ll keep trying. Thank you @MichelDiz

Topic		Replies	Views
How to load the rdf data? Users	7	1469	January 27, 2018
Live loader not working in docker Users	5	1378	March 5, 2020
Trouble loading 1M dataset Dgraph	2	740	October 21, 2018
Bulk load - New server(alpha) isn't available Dgraph	2	638	November 16, 2018
Live uploader: No data files found in Dgraph dgraph	6	633	January 18, 2022

How to import data into new cluster

Related topics