I installed dgraph using docker toolbox on Windows 7. But when i tried to load a rdf file using the below command, i am getting an error.
PS C:\Users\esgazai\dgraph> docker exec -it dgraph dgraphloader -r 1million.rdf.gz
Dgraph version : v0.8.1
Commit SHA-1 : d86498f7
Commit timestamp : 2017-08-16 12:53:17 +1000
Branch : HEAD
Processing 1million.rdf.gz
2017/08/18 05:18:57 open 1million.rdf.gz: no such file or directory
Hi @Abbas
This is because you didn’t mount the volume from within your machine (the host) to the Docker container. I am aware that you have an open issue about mounting volumes in Windows. In the mean-time, I have another solution which would help you try out queries.
docker exec -it dgraph bash
cd /dgraph
wget "https://github.com/dgraph-io/tutorial/blob/master/resources/1million.rdf.gz?raw=true" -O 1million.rdf.gz -q
dgraphloader -r 1million.rdf.gz
Hope that helps.
Hi Pawan,
Really appreciate your response. As a workaround I had already downloaded the dgraphloader code and built all dependencies within my local golang environment. Got stuck in some gcc dependencies but resolved it.
But now when i downloaded the goldendata.rdf.gz and loaded it using my local dgraphloader version, it hanged and stopped responding. The dgraph container is neither getting stopped or killed…It did process a lot of records:
[Request: 1121] Total RDFs done: 1111879 RDFs per second: 310 Time Elapsed: 59m50s
But could not complete the process…In the docker container console, i am getting below statements:
2017/08/21 08:26:36 node.go:402: WARN: A tick missed to fire. Node blocks too long!
2017/08/21 08:26:36 node.go:402: WARN: A tick missed to fire. Node blocks too long!
2017/08/21 08:26:36 node.go:402: WARN: A tick missed to fire. Node blocks too long!
2017/08/21 08:26:36 node.go:402: WARN: A tick missed to fire. Node blocks too long!
2017/08/21 08:26:36 node.go:402: WARN: A tick missed to fire. Node blocks too long!
2017/08/21 08:26:36 node.go:402: WARN: A tick missed to fire. Node blocks too long!
2017/08/21 08:26:36 node.go:402: WARN: A tick missed to fire. Node blocks too long!
2017/08/21 08:26:36 node.go:402: WARN: A tick missed to fire. Node blocks too long!
2017/08/21 08:26:36 node.go:402: WARN: A tick missed to fire. Node blocks too long!
2017/08/21 08:26:36 node.go:402: WARN: A tick missed to fire. Node blocks too long!
2017/08/21 08:26:36 node.go:402: WARN: A tick missed to fire. Node blocks too long!
@Abbas: Is this issue reproducible ? I am not entirely sure it might be the case that the writes to disk are very very slow or blocked. Can you please check the size of p and w directory.(du -csh *)
12K p
16K w
28K total
@Abbas: Is this issue reproducible ?
@janardhan this time i tried the steps Pawan provided by opening container’s bash and this time the container crashed:
github.com/dgraph-io/dgraph/query.(*QueryRequest).ProcessWithMutation(0xc442a078
c0, 0x13dd4e0, 0xc46529b980, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc45bd6fa70, …)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/query/query.go:2446
+0x5c7
github.com/dgraph-io/dgraph/dgraph.(*Server).Run(0x146a998, 0x7f66dff4c868, 0xc4
6529b980, 0xc45bd6ed50, 0xc4468bbe40, 0x0, 0x0)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/dgraph/server.go:184
+0x8d4
github.com/dgraph-io/dgraph/protos._Dgraph_Run_Handler(0xcfe6a0, 0x146a998, 0x7f
66dff4c800, 0xc44d9078f0, 0xc452accaf0, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/protos/graphresponse
.pb.go:816 +0x28d
google.golang.org/grpc.(*Server).processUnaryRPC(0xc429968000, 0x13e0720, 0xc429
992180, 0xc4360ba500, 0xc420160870, 0x13c7d20, 0x0, 0x0, 0x0)
/home/travis/gopath/src/google.golang.org/grpc/server.go:806 +0xc41
google.golang.org/grpc.(*Server).handleStream(0xc429968000, 0x13e0720, 0xc429992
180, 0xc4360ba500, 0x0)
/home/travis/gopath/src/google.golang.org/grpc/server.go:1006 +0x15a6
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc429f280b0, 0xc429968000
, 0x13e0720, 0xc429992180, 0xc4360ba500)
/home/travis/gopath/src/google.golang.org/grpc/server.go:552 +0xa9
created by google.golang.org/grpc.(*Server).serveStreams.func1
/home/travis/gopath/src/google.golang.org/grpc/server.go:553 +0xa1
goroutine 490846 [semacquire]:
sync.runtime_Semacquire(0xc463d8cadc)
/home/travis/.gimme/versions/go1.8.linux.amd64/src/runtime/sema.go:47 +0
x34
sync.(*WaitGroup).Wait(0xc463d8cad0)
/home/travis/.gimme/versions/go1.8.linux.amd64/src/sync/waitgroup.go:131
+0x7a
github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.(*KV).BatchSetAsy
nc.func1(0xc4780c4180, 0x1, 0x1, 0xc478335440)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/vendor/github.com/dg
raph-io/badger/kv.go:838 +0x66
created by github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.(*KV).
BatchSetAsync
/home/travis/gopath/src/github.com/dgraph-io/dgraph/vendor/github.com/dg
raph-io/badger/kv.go:846 +0x8d
goroutine 475271 [semacquire]:
sync.runtime_SemacquireMutex(0xc420382584)
/home/travis/.gimme/versions/go1.8.linux.amd64/src/runtime/sema.go:62 +0
x34
sync.(*Mutex).Lock(0xc420382580)
/home/travis/.gimme/versions/go1.8.linux.amd64/src/sync/mutex.go:87 +0x9
d
github.com/dgraph-io/dgraph/posting.(*listCache).Get(0xc420382580, 0xc45f941970,
0x10, 0x0)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/posting/lru.go:131 +
0x3a
github.com/dgraph-io/dgraph/posting.GetOrCreate(0xc4780d51a0, 0x10, 0x10, 0xc400
000001, 0x10)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/posting/lists.go:316
+0x8b
github.com/dgraph-io/dgraph/posting.addCountMutation(0x13dd520, 0xc42b67ab70, 0x
c478616680, 0x26, 0xc45f941bb0, 0xc44d478d00)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/posting/index.go:262
+0xca
github.com/dgraph-io/dgraph/posting.updateCount(0x13dd520, 0xc42b67ab70, 0xc42fd
db718, 0x8, 0x26, 0x27, 0x1872ee4f45bf9925, 0x0, 0xb2fcea, 0xc420382580)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/posting/index.go:286
+0xfa
github.com/dgraph-io/dgraph/posting.(*List).AddMutationWithIndex(0xc472b23c70, 0
x13dd520, 0xc42b67ab70, 0xc456828880, 0x0, 0x0)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/posting/index.go:353
+0x7aa
github.com/dgraph-io/dgraph/worker.runMutations(0x7f66dff4c800, 0xc42b67ab70, 0x
c45c3dd000, 0x7d0, 0x900, 0xc42fcf22f0, 0x7f66dff4c800)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/worker/mutation.go:8
2 +0x422
github.com/dgraph-io/dgraph/worker.(*node).processMutation(0xc420076240, 0x7f66d
ff4c800, 0xc42b67ab70, 0x1b1, 0xc42e034740, 0x435b08, 0xdfbfe0)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/worker/draft.go:518
+0x142
github.com/dgraph-io/dgraph/worker.(*node).process(0xc420076240, 0x1b1, 0xc430f3
f400, 0xc4202d2d20)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/worker/draft.go:566
+0x1d2
created by github.com/dgraph-io/dgraph/worker.(*node).processApplyCh
/home/travis/gopath/src/github.com/dgraph-io/dgraph/worker/draft.go:653
+0x7f8
goroutine 480107 [semacquire]:
sync.runtime_SemacquireMutex(0xc420382584)
/home/travis/.gimme/versions/go1.8.linux.amd64/src/runtime/sema.go:62 +0
x34
sync.(*Mutex).Lock(0xc420382580)
/home/travis/.gimme/versions/go1.8.linux.amd64/src/sync/mutex.go:87 +0x9
d
github.com/dgraph-io/dgraph/posting.(*listCache).Get(0xc420382580, 0xc42f0c3cd8,
0x20, 0x0)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/posting/lru.go:131 +
0x3a
github.com/dgraph-io/dgraph/posting.GetOrCreate(0xc478334160, 0x20, 0x20, 0xc400
000001, 0x20)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/posting/lists.go:316
+0x8b
github.com/dgraph-io/dgraph/worker.runMutations(0x7f66dff4c800, 0xc463ad55c0, 0x
c46697f000, 0x7d0, 0x900, 0xc43ea81a20, 0x7f66dff4c800)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/worker/mutation.go:7
5 +0x380
github.com/dgraph-io/dgraph/worker.(*node).processMutation(0xc420076240, 0x7f66d
ff4c800, 0xc463ad55c0, 0x1e4, 0xc438bed440, 0x435b08, 0xdfbfe0)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/worker/draft.go:518
+0x142
github.com/dgraph-io/dgraph/worker.(*node).process(0xc420076240, 0x1e4, 0xc464e0
8900, 0xc4202d2d20)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/worker/draft.go:566
Hey Abbas
Is that the full error trace? How much memory does your machine have?
i have pasted whatever stack trace is visible on the console. let me know where can i get the full stack trace ?My machine has 8GB RAM and I had allocated --memory_mb as 2048 while running the container.
FYI, I have run the dgraphloader again using my local golang setup and not the docker bash. it is running fine albeit a little slowly.
C:\Abbas_New\GoWork\src\main>dgraphloader -r C:\Users\esgazai\dgraph\1million.rdf.gz
Processing C:\Users\esgazai\dgraph\1million.rdf.gz
[Request: 251] Total RDFs done: 151000 RDFs per second: 266 Time Elapsed: 9m27s
Dgraph crashed again…
/home/travis/gopath/src/github.com/dgraph-io/dgraph/posting/lru.go:82 +0x3d
github.com/dgraph-io/dgraph/posting.GetOrCreate(0xc43f2cd500, 0x1d, 0x1d, 0xc400000001, 0x1d)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/posting/lists.go:330 +0x19d
github.com/dgraph-io/dgraph/worker.runMutations(0x7f88846f90b0, 0xc427a26f90, 0xc45f7fe800, 0x7d0, 0x900, 0xc43916ca30, 0x7f88846f90b0)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/worker/mutation.go:75 +0x380
github.com/dgraph-io/dgraph/worker.(*node).processMutation(0xc4201d2360, 0x7f88846f90b0, 0xc427a26f90, 0xdf, 0xc46993d480, 0x435b08, 0xdfbfe0)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/worker/draft.go:518 +0x142
github.com/dgraph-io/dgraph/worker.(*node).process(0xc4201d2360, 0xdf, 0xc439b51e60, 0xc420272cc0)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/worker/draft.go:566 +0x1d2
created by github.com/dgraph-io/dgraph/worker.(*node).processApplyCh
/home/travis/gopath/src/github.com/dgraph-io/dgraph/worker/draft.go:653 +0x7f8
goroutine 77119 [semacquire]:
sync.runtime_SemacquireMutex(0xc44da0e444)
/home/travis/.gimme/versions/go1.8.linux.amd64/src/runtime/sema.go:62 +0x34
sync.(*Mutex).Lock(0xc44da0e440)
/home/travis/.gimme/versions/go1.8.linux.amd64/src/sync/mutex.go:87 +0x9d
github.com/dgraph-io/dgraph/posting.(*listCache).PutIfMissing(0xc44da0e440, 0xc43f4c0300, 0x10, 0xc48e319520, 0x0)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/posting/lru.go:82 +0x3d
github.com/dgraph-io/dgraph/posting.GetOrCreate(0xc43f4ee430, 0x10, 0x10, 0xc400000001, 0x10)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/posting/lists.go:330 +0x19d
github.com/dgraph-io/dgraph/worker.runMutations(0x7f88846f90b0, 0xc486d4aea0, 0xc48aab4000, 0x7d0, 0x900, 0xc43dad3a60, 0x7f88846f90b0)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/worker/mutation.go:75 +0x380
github.com/dgraph-io/dgraph/worker.(*node).processMutation(0xc4201d2360, 0x7f88846f90b0, 0xc486d4aea0, 0x15d, 0xc47362b340, 0x435b08, 0xdfbfe0)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/worker/draft.go:518 +0x142
github.com/dgraph-io/dgraph/worker.(*node).process(0xc4201d2360, 0x15d, 0xc43e10c600, 0xc420272cc0)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/worker/draft.go:566 +0x1d2
created by github.com/dgraph-io/dgraph/worker.(*node).processApplyCh
/home/travis/gopath/src/github.com/dgraph-io/dgraph/worker/draft.go:653 +0x7f8
goroutine 60770 [running]:
goroutine running on other thread; stack unavailable
created by github.com/dgraph-io/dgraph/worker.(*node).processApplyCh
/home/travis/gopath/src/github.com/dgraph-io/dgraph/worker/draft.go:653 +0x7f8
goroutine 60755 [semacquire]:
sync.runtime_SemacquireMutex(0xc44da0e444)
/home/travis/.gimme/versions/go1.8.linux.amd64/src/runtime/sema.go:62 +0x34
sync.(*Mutex).Lock(0xc44da0e440)
/home/travis/.gimme/versions/go1.8.linux.amd64/src/sync/mutex.go:87 +0x9d
github.com/dgraph-io/dgraph/posting.(*listCache).PutIfMissing(0xc44da0e440, 0xc43f2dfb00, 0x1d, 0xc48e01f790, 0x0)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/posting/lru.go:82 +0x3d
github.com/dgraph-io/dgraph/posting.GetOrCreate(0xc43f30ebe0, 0x1d, 0x1d, 0xc400000001, 0x1d)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/posting/lists.go:330 +0x19d
github.com/dgraph-io/dgraph/worker.runMutations(0x7f88846f90b0, 0xc457637440, 0xc45375d800, 0x7d0, 0x900, 0xc4384ccd90, 0x7f88846f90b0)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/worker/mutation.go:75 +0x380
github.com/dgraph-io/dgraph/worker.(*node).processMutation(0xc4201d2360, 0x7f88846f90b0, 0xc457637440, 0x73, 0xc453d1f880, 0x435b08, 0xdfbfe0)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/worker/draft.go:518 +0x142
github.com/dgraph-io/dgraph/worker.(*node).process(0xc4201d2360, 0x73, 0xc4398fe020, 0xc420272cc0)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/worker/draft.go:566 +0x1d2
created by github.com/dgraph-io/dgraph/worker.(*node).processApplyCh
/home/travis/gopath/src/github.com/dgraph-io/dgraph/worker/draft.go:653 +0x7f8
goroutine 60160 [semacquire]:
sync.runtime_SemacquireMutex(0xc44da0e444)
/home/travis/.gimme/versions/go1.8.linux.amd64/src/runtime/sema.go:62 +0x34
sync.(*Mutex).Lock(0xc44da0e440)
/home/travis/.gimme/versions/go1.8.linux.amd64/src/sync/mutex.go:87 +0x9d
github.com/dgraph-io/dgraph/posting.(*listCache).Get(0xc44da0e440, 0xc468727958, 0xe, 0x0)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/posting/lru.go:131 +0x3a
github.com/dgraph-io/dgraph/posting.GetOrCreate(0xc43f4d0060, 0xe, 0xe, 0x1, 0xc43d03eda0)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/posting/lists.go:316 +0x8b
github.com/dgraph-io/dgraph/posting.addIndexMutation(0x13dd520, 0xc46862af60, 0xc48fa0c880, 0xc43f259c68, 0x6, 0x0, 0x0)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/posting/index.go:122 +0x179
github.com/dgraph-io/dgraph/posting.addIndexMutations(0x13dd520, 0xc46862af60, 0xc4688e8100, 0x9, 0xca1280, 0xc43f4319c0, 0x1, 0x0, 0xc43f186ce3)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/posting/index.go:103 +0x1a1
github.com/dgraph-io/dgraph/posting.(*List).AddMutationWithIndex(0xc48dbdb110, 0x13dd520, 0xc46862af60, 0xc4688e8100, 0x0, 0x0)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/posting/index.go:363 +0x6b1
github.com/dgraph-io/dgraph/worker.runMutations(0x7f88846f90b0, 0xc46862af60, 0xc468826800, 0x7d0, 0x900, 0xc43d03eda0, 0x7f88846f90b0)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/worker/mutation.go:82 +0x422
github.com/dgraph-io/dgraph/worker.(*node).processMutation(0xc4201d2360, 0x7f88846f90b0, 0xc46862af60, 0x11, 0xc466c14740, 0xc45f49c500, 0xc45f49c500)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/worker/draft.go:518 +0x142
github.com/dgraph-io/dgraph/worker.(*node).process(0xc4201d2360, 0x11, 0xc43b216fa0, 0xc420272cc0)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/worker/draft.go:566 +0x1d2
created by github.com/dgraph-io/dgraph/worker.(*node).processApplyCh
/home/travis/gopath/src/github.com/dgraph-io/dgraph/worker/draft.go:653 +0x7f8
goroutine 60144 [semacquire]:
sync.runtime_SemacquireMutex(0xc44da0e444)
/home/travis/.gimme/versions/go1.8.linux.amd64/src/runtime/sema.go:62 +0x34
sync.(*Mutex).Lock(0xc44da0e440)
/home/travis/.gimme/versions/go1.8.linux.amd64/src/sync/mutex.go:87 +0x9d
github.com/dgraph-io/dgraph/posting.(*listCache).Get(0xc44da0e440, 0xc46b5e1958, 0xe, 0x0)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/posting/lru.go:131 +0x3a
github.com/dgraph-io/dgraph/posting.GetOrCreate(0xc43f59c080, 0xe, 0xe, 0x1, 0xc43d512130)
/home/travis/gopath/src/github.com/dgraph-io/dgraph/posting/lists.go:316 +0x8b
This is pretty strange. We regularly load 21Million RDF data set and it loads without problems. Are you running a SSD?
No I am not running a SSD. FYI, I also tried to increase --memory_mb to 3048 but still the dgraph is crashing.
Maybe because i am using Docker on Windows.
@pawan is there any way i can increase the space allocated for p and w ? just wanted to try that.
There is no fixed space allocated for p and w. They are only limited by the space of the disk. While we resolve this for you on Windows is it possible for you to try Dgraph on a Ubuntu VM?
yeah i guess thats the only option i have. The 21million RDF you were referring to was on Windows or Ubuntu ?
Ubuntu. All of our development and testing happens on it. We have even loaded > billion RDF’s on Ubuntu with ease.
@Abbas
It’s slow if we use a shared directory between host and container. The recommended way is to use docker data volumes. I have updated the docs for the same.
https://docs.dgraph.io/master/get-started/#on-non-linux-distributions.
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.