Problem after live loader

Issues
, ,
1

Hi. I was loading data using the live loader. The process just stopped suddenly and the output was like this:

Error while mutating: Only leader can decide to commit or abort s.Code Unknown
Error while mutating: Only leader can decide to commit or abort s.Code Unknown
Error while mutating: Only leader can decide to commit or abort s.Code Unknown
Error while mutating: Only leader can decide to commit or abort s.Code Unknown
Error while mutating: Only leader can decide to commit or abort s.Code Unknown
[19:23:52+0200] Elapsed: 08m40s Txns: 6712 N-Quads: 6711678 N-Quads/s [last 5s]:  4936 Aborts: 5
[19:23:57+0200] Elapsed: 08m45s Txns: 6791 N-Quads: 6790678 N-Quads/s [last 5s]: 15800 Aborts: 5
[19:24:02+0200] Elapsed: 08m50s Txns: 6870 N-Quads: 6869678 N-Quads/s [last 5s]: 15800 Aborts: 5
[19:24:07+0200] Elapsed: 08m55s Txns: 6986 N-Quads: 6985678 N-Quads/s [last 5s]: 23200 Aborts: 5
[19:24:12+0200] Elapsed: 09m00s Txns: 7051 N-Quads: 7050327 N-Quads/s [last 5s]: 12930 Aborts: 5
[19:24:17+0200] Elapsed: 09m05s Txns: 7132 N-Quads: 7131327 N-Quads/s [last 5s]: 16200 Aborts: 5
[19:24:22+0200] Elapsed: 09m10s Txns: 7259 N-Quads: 7258327 N-Quads/s [last 5s]: 25400 Aborts: 5
[19:24:27+0200] Elapsed: 09m15s Txns: 7384 N-Quads: 7383327 N-Quads/s [last 5s]: 25000 Aborts: 5
[19:24:32+0200] Elapsed: 09m20s Txns: 7510 N-Quads: 7509327 N-Quads/s [last 5s]: 25200 Aborts: 5
[19:24:37+0200] Elapsed: 09m25s Txns: 7605 N-Quads: 7604327 N-Quads/s [last 5s]: 19000 Aborts: 5
[19:24:42+0200] Elapsed: 09m30s Txns: 7679 N-Quads: 7678327 N-Quads/s [last 5s]: 14800 Aborts: 5
[19:24:47+0200] Elapsed: 09m35s Txns: 7723 N-Quads: 7722327 N-Quads/s [last 5s]:  8800 Aborts: 5
[19:24:52+0200] Elapsed: 09m40s Txns: 7756 N-Quads: 7755327 N-Quads/s [last 5s]:  6600 Aborts: 5
[19:24:57+0200] Elapsed: 09m45s Txns: 7765 N-Quads: 7764327 N-Quads/s [last 5s]:  1800 Aborts: 5
[19:25:02+0200] Elapsed: 09m50s Txns: 7765 N-Quads: 7764327 N-Quads/s [last 5s]:     0 Aborts: 5
[19:25:07+0200] Elapsed: 09m55s Txns: 7765 N-Quads: 7764327 N-Quads/s [last 5s]:     0 Aborts: 5
[19:25:12+0200] Elapsed: 10m00s Txns: 7765 N-Quads: 7764327 N-Quads/s [last 5s]:     0 Aborts: 5
[19:25:17+0200] Elapsed: 10m05s Txns: 7765 N-Quads: 7764327 N-Quads/s [last 5s]:     0 Aborts: 5
[19:25:22+0200] Elapsed: 10m10s Txns: 7798 N-Quads: 7797327 N-Quads/s [last 5s]:  6600 Aborts: 5
[19:25:27+0200] Elapsed: 10m15s Txns: 7857 N-Quads: 7856142 N-Quads/s [last 5s]: 11763 Aborts: 5
[19:25:32+0200] Elapsed: 10m20s Txns: 7956 N-Quads: 7955142 N-Quads/s [last 5s]: 19800 Aborts: 5
[19:25:37+0200] Elapsed: 10m25s Txns: 8109 N-Quads: 8108142 N-Quads/s [last 5s]: 30600 Aborts: 5
[19:25:42+0200] Elapsed: 10m30s Txns: 8236 N-Quads: 8235142 N-Quads/s [last 5s]: 25400 Aborts: 5
[19:25:47+0200] Elapsed: 10m35s Txns: 8330 N-Quads: 8329142 N-Quads/s [last 5s]: 18800 Aborts: 5
[19:25:52+0200] Elapsed: 10m40s Txns: 8423 N-Quads: 8422142 N-Quads/s [last 5s]: 18600 Aborts: 5
[19:25:57+0200] Elapsed: 10m45s Txns: 8527 N-Quads: 8525736 N-Quads/s [last 5s]: 20719 Aborts: 5
[19:26:02+0200] Elapsed: 10m50s Txns: 8609 N-Quads: 8607736 N-Quads/s [last 5s]: 16400 Aborts: 5
[19:26:07+0200] Elapsed: 10m55s Txns: 8711 N-Quads: 8708763 N-Quads/s [last 5s]: 20205 Aborts: 5
[19:26:12+0200] Elapsed: 11m00s Txns: 8808 N-Quads: 8805763 N-Quads/s [last 5s]: 19400 Aborts: 5
Connection has been possibly interrupted. Got error: rpc error: code = Unavailable desc = transport is closing. Will retry after 22s.
panic: rpc error: code = Unavailable desc = transport is closing

goroutine 87 [running]:
github.com/dgraph-io/dgraph/dgraph/cmd/live.(*loader).upsertUids(0xc000516000, 0xc049e90000, 0x3e8, 0x3e8)
        /ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/run.go:394 +0xed3
github.com/dgraph-io/dgraph/dgraph/cmd/live.(*loader).processLoadFile.func1(0xc00010e3c0, 0xc000516000, 0xc000316000)
        /ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/run.go:552 +0x56e
created by github.com/dgraph-io/dgraph/dgraph/cmd/live.(*loader).processLoadFile

I’ve been dealing with this kind of issues and have to split my files in order to load data. I am not sure if the problem are memory leaks.
However, after i encounter this error, the alpha instance just became corrupted. Logs ouput looks like this:

May 10 20:06:29 bitcoin bash[671997]: I0510 20:06:29.327842  672017 run.go:756] x.Config: {PortOffset:0 Limit:query-edge=1000000; mutations-nquad=1000000; max-pending-queries=10000; mutations=allow; normalize-n>
May 10 20:06:29 bitcoin bash[671997]: I0510 20:06:29.327906  672017 run.go:757] x.WorkerConfig: {TmpDir:t ExportPath:export Trace:ratio=0.01; jaeger=; datadog= MyAddr: ZeroAddr:[localhost:5080] TLSClientConfig:>
May 10 20:06:29 bitcoin bash[671997]: I0510 20:06:29.328050  672017 run.go:758] worker.Config: {PostingDir:/var/lib/dgraph/p WALDir:/var/lib/dgraph/w MutationsMode:0 AuthToken: HmacSecret:**** AccessJwtTtl:0s R>
May 10 20:06:29 bitcoin bash[671997]: I0510 20:06:29.328312  672017 log.go:295] Found file: 22 First Index: 630001
May 10 20:06:29 bitcoin bash[671997]: I0510 20:06:29.328385  672017 log.go:295] Found file: 23 First Index: 660001
May 10 20:06:29 bitcoin bash[671997]: I0510 20:06:29.328429  672017 log.go:295] Found file: 24 First Index: 690001
May 10 20:06:29 bitcoin bash[671997]: I0510 20:06:29.328547  672017 storage.go:125] Init Raft Storage with snap: 651118, first: 651119, last: 693490
May 10 20:06:29 bitcoin bash[671997]: I0510 20:06:29.328592  672017 server_state.go:141] Opening postings BadgerDB with options: {Dir:/var/lib/dgraph/p ValueDir:/var/lib/dgraph/p SyncWrites:false NumVersionsToK>
May 10 20:06:32 bitcoin bash[671997]: I0510 20:06:32.058642  672017 log.go:34] All 956 tables opened in 198ms
May 10 20:06:32 bitcoin bash[671997]: I0510 20:06:32.194064  672017 log.go:34] Discard stats nextEmptySlot: 0
May 10 20:06:35 bitcoin bash[671997]: W0510 20:06:35.920485  672017 log.go:36] [Compactor: 2] LOG Compact FAILED with error: while running compactions for: {span:0xc02248a000 compactorId:2 t:{baseLevel:3 target>
May 10 20:06:35 bitcoin bash[671997]: W0510 20:06:35.920674  672017 log.go:36] While running doCompact: while running compactions for: {span:0xc02248a000 compactorId:2 t:{baseLevel:3 targetSz:[0 10485760 104857>
May 10 20:06:39 bitcoin bash[671997]: W0510 20:06:39.247276  672017 log.go:36] [Compactor: 2] LOG Compact FAILED with error: while running compactions for: {span:0xc000585c80 compactorId:2 t:{baseLevel:3 target>

I am running only an instance of alpha and zero, nothing else. Im not using docker, they are installed as services in my ubuntu.
I don’t know what to do about this. Any idea?

2

Can you check if the Zero instance is dead during the load? This looks like the case.

Also, share some stats of you env.

Cheers!

3

Hi!

I cannot perform loads anymore. When I try to execute the live loader again it gives me this error:

Dgraph version   : v21.03.0
Dgraph codename  : rocket
Dgraph SHA-256   : b4e4c77011e2938e9da197395dbce91d0c6ebb83d383b190f5b70201836a773f
Commit SHA-1     : a77bbe8ae
Commit timestamp : 2021-04-07 21:36:38 +0530
Branch           : HEAD
Go version       : go1.16.2
jemalloc enabled : true

For Dgraph official documentation, visit https://dgraph.io/docs.
For discussions about Dgraph     , visit http://discuss.dgraph.io.
For fully-managed Dgraph Cloud   , visit https://dgraph.io/cloud.

Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2021 Dgraph Labs, Inc.



Running transaction with dgraph endpoint: 127.0.0.1:9080
While trying to setup connection: context deadline exceeded. Retrying...
2021/05/12 17:53:25 Could not setup connection after 1 retries
github.com/dgraph-io/dgraph/x.Fatalf
        /ext-go/1/src/github.com/dgraph-io/dgraph/x/error.go:120
github.com/dgraph-io/dgraph/x.GetDgraphClient
        /ext-go/1/src/github.com/dgraph-io/dgraph/x/x.go:1044
github.com/dgraph-io/dgraph/dgraph/cmd/live.run
        /ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/run.go:780
github.com/dgraph-io/dgraph/dgraph/cmd/live.init.0.func1
        /ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/run.go:134
github.com/spf13/cobra.(*Command).execute
        /go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
        /go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
        /go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
        /ext-go/1/src/github
.com/dgraph-io/dgraph/dgraph/cmd/root.go:78
main.main
        /ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/main.go:99
runtime.main
        /usr/local/go/src/runtime/proc.go:225
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1371

Cannot query the database either.

Im using dgraph alpha v21.03 on Ubuntu 20.04 installed with the installation script.
Dgraph service is executed with --cache size-mb 2048.
CPU 8 cores, 32 GB RAM

4

Is your Alpha reachable? is it localhost?

5

Nope, it is localhost

6

Please, share the Alpha logs.