Problem after live loader

Hi. I was loading data using the live loader. The process just stopped suddenly and the output was like this:

Error while mutating: Only leader can decide to commit or abort s.Code Unknown
Error while mutating: Only leader can decide to commit or abort s.Code Unknown
Error while mutating: Only leader can decide to commit or abort s.Code Unknown
Error while mutating: Only leader can decide to commit or abort s.Code Unknown
Error while mutating: Only leader can decide to commit or abort s.Code Unknown
[19:23:52+0200] Elapsed: 08m40s Txns: 6712 N-Quads: 6711678 N-Quads/s [last 5s]:  4936 Aborts: 5
[19:23:57+0200] Elapsed: 08m45s Txns: 6791 N-Quads: 6790678 N-Quads/s [last 5s]: 15800 Aborts: 5
[19:24:02+0200] Elapsed: 08m50s Txns: 6870 N-Quads: 6869678 N-Quads/s [last 5s]: 15800 Aborts: 5
[19:24:07+0200] Elapsed: 08m55s Txns: 6986 N-Quads: 6985678 N-Quads/s [last 5s]: 23200 Aborts: 5
[19:24:12+0200] Elapsed: 09m00s Txns: 7051 N-Quads: 7050327 N-Quads/s [last 5s]: 12930 Aborts: 5
[19:24:17+0200] Elapsed: 09m05s Txns: 7132 N-Quads: 7131327 N-Quads/s [last 5s]: 16200 Aborts: 5
[19:24:22+0200] Elapsed: 09m10s Txns: 7259 N-Quads: 7258327 N-Quads/s [last 5s]: 25400 Aborts: 5
[19:24:27+0200] Elapsed: 09m15s Txns: 7384 N-Quads: 7383327 N-Quads/s [last 5s]: 25000 Aborts: 5
[19:24:32+0200] Elapsed: 09m20s Txns: 7510 N-Quads: 7509327 N-Quads/s [last 5s]: 25200 Aborts: 5
[19:24:37+0200] Elapsed: 09m25s Txns: 7605 N-Quads: 7604327 N-Quads/s [last 5s]: 19000 Aborts: 5
[19:24:42+0200] Elapsed: 09m30s Txns: 7679 N-Quads: 7678327 N-Quads/s [last 5s]: 14800 Aborts: 5
[19:24:47+0200] Elapsed: 09m35s Txns: 7723 N-Quads: 7722327 N-Quads/s [last 5s]:  8800 Aborts: 5
[19:24:52+0200] Elapsed: 09m40s Txns: 7756 N-Quads: 7755327 N-Quads/s [last 5s]:  6600 Aborts: 5
[19:24:57+0200] Elapsed: 09m45s Txns: 7765 N-Quads: 7764327 N-Quads/s [last 5s]:  1800 Aborts: 5
[19:25:02+0200] Elapsed: 09m50s Txns: 7765 N-Quads: 7764327 N-Quads/s [last 5s]:     0 Aborts: 5
[19:25:07+0200] Elapsed: 09m55s Txns: 7765 N-Quads: 7764327 N-Quads/s [last 5s]:     0 Aborts: 5
[19:25:12+0200] Elapsed: 10m00s Txns: 7765 N-Quads: 7764327 N-Quads/s [last 5s]:     0 Aborts: 5
[19:25:17+0200] Elapsed: 10m05s Txns: 7765 N-Quads: 7764327 N-Quads/s [last 5s]:     0 Aborts: 5
[19:25:22+0200] Elapsed: 10m10s Txns: 7798 N-Quads: 7797327 N-Quads/s [last 5s]:  6600 Aborts: 5
[19:25:27+0200] Elapsed: 10m15s Txns: 7857 N-Quads: 7856142 N-Quads/s [last 5s]: 11763 Aborts: 5
[19:25:32+0200] Elapsed: 10m20s Txns: 7956 N-Quads: 7955142 N-Quads/s [last 5s]: 19800 Aborts: 5
[19:25:37+0200] Elapsed: 10m25s Txns: 8109 N-Quads: 8108142 N-Quads/s [last 5s]: 30600 Aborts: 5
[19:25:42+0200] Elapsed: 10m30s Txns: 8236 N-Quads: 8235142 N-Quads/s [last 5s]: 25400 Aborts: 5
[19:25:47+0200] Elapsed: 10m35s Txns: 8330 N-Quads: 8329142 N-Quads/s [last 5s]: 18800 Aborts: 5
[19:25:52+0200] Elapsed: 10m40s Txns: 8423 N-Quads: 8422142 N-Quads/s [last 5s]: 18600 Aborts: 5
[19:25:57+0200] Elapsed: 10m45s Txns: 8527 N-Quads: 8525736 N-Quads/s [last 5s]: 20719 Aborts: 5
[19:26:02+0200] Elapsed: 10m50s Txns: 8609 N-Quads: 8607736 N-Quads/s [last 5s]: 16400 Aborts: 5
[19:26:07+0200] Elapsed: 10m55s Txns: 8711 N-Quads: 8708763 N-Quads/s [last 5s]: 20205 Aborts: 5
[19:26:12+0200] Elapsed: 11m00s Txns: 8808 N-Quads: 8805763 N-Quads/s [last 5s]: 19400 Aborts: 5
Connection has been possibly interrupted. Got error: rpc error: code = Unavailable desc = transport is closing. Will retry after 22s.
panic: rpc error: code = Unavailable desc = transport is closing

goroutine 87 [running]:
github.com/dgraph-io/dgraph/dgraph/cmd/live.(*loader).upsertUids(0xc000516000, 0xc049e90000, 0x3e8, 0x3e8)
        /ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/run.go:394 +0xed3
github.com/dgraph-io/dgraph/dgraph/cmd/live.(*loader).processLoadFile.func1(0xc00010e3c0, 0xc000516000, 0xc000316000)
        /ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/run.go:552 +0x56e
created by github.com/dgraph-io/dgraph/dgraph/cmd/live.(*loader).processLoadFile

I’ve been dealing with this kind of issues and have to split my files in order to load data. I am not sure if the problem are memory leaks.
However, after i encounter this error, the alpha instance just became corrupted. Logs ouput looks like this:

May 10 20:06:29 bitcoin bash[671997]: I0510 20:06:29.327842  672017 run.go:756] x.Config: {PortOffset:0 Limit:query-edge=1000000; mutations-nquad=1000000; max-pending-queries=10000; mutations=allow; normalize-n>
May 10 20:06:29 bitcoin bash[671997]: I0510 20:06:29.327906  672017 run.go:757] x.WorkerConfig: {TmpDir:t ExportPath:export Trace:ratio=0.01; jaeger=; datadog= MyAddr: ZeroAddr:[localhost:5080] TLSClientConfig:>
May 10 20:06:29 bitcoin bash[671997]: I0510 20:06:29.328050  672017 run.go:758] worker.Config: {PostingDir:/var/lib/dgraph/p WALDir:/var/lib/dgraph/w MutationsMode:0 AuthToken: HmacSecret:**** AccessJwtTtl:0s R>
May 10 20:06:29 bitcoin bash[671997]: I0510 20:06:29.328312  672017 log.go:295] Found file: 22 First Index: 630001
May 10 20:06:29 bitcoin bash[671997]: I0510 20:06:29.328385  672017 log.go:295] Found file: 23 First Index: 660001
May 10 20:06:29 bitcoin bash[671997]: I0510 20:06:29.328429  672017 log.go:295] Found file: 24 First Index: 690001
May 10 20:06:29 bitcoin bash[671997]: I0510 20:06:29.328547  672017 storage.go:125] Init Raft Storage with snap: 651118, first: 651119, last: 693490
May 10 20:06:29 bitcoin bash[671997]: I0510 20:06:29.328592  672017 server_state.go:141] Opening postings BadgerDB with options: {Dir:/var/lib/dgraph/p ValueDir:/var/lib/dgraph/p SyncWrites:false NumVersionsToK>
May 10 20:06:32 bitcoin bash[671997]: I0510 20:06:32.058642  672017 log.go:34] All 956 tables opened in 198ms
May 10 20:06:32 bitcoin bash[671997]: I0510 20:06:32.194064  672017 log.go:34] Discard stats nextEmptySlot: 0
May 10 20:06:35 bitcoin bash[671997]: W0510 20:06:35.920485  672017 log.go:36] [Compactor: 2] LOG Compact FAILED with error: while running compactions for: {span:0xc02248a000 compactorId:2 t:{baseLevel:3 target>
May 10 20:06:35 bitcoin bash[671997]: W0510 20:06:35.920674  672017 log.go:36] While running doCompact: while running compactions for: {span:0xc02248a000 compactorId:2 t:{baseLevel:3 targetSz:[0 10485760 104857>
May 10 20:06:39 bitcoin bash[671997]: W0510 20:06:39.247276  672017 log.go:36] [Compactor: 2] LOG Compact FAILED with error: while running compactions for: {span:0xc000585c80 compactorId:2 t:{baseLevel:3 target>

I am running only an instance of alpha and zero, nothing else. Im not using docker, they are installed as services in my ubuntu.
I don’t know what to do about this. Any idea?

Can you check if the Zero instance is dead during the load? This looks like the case.

Also, share some stats of you env.

Cheers!

Hi!

I cannot perform loads anymore. When I try to execute the live loader again it gives me this error:

Dgraph version   : v21.03.0
Dgraph codename  : rocket
Dgraph SHA-256   : b4e4c77011e2938e9da197395dbce91d0c6ebb83d383b190f5b70201836a773f
Commit SHA-1     : a77bbe8ae
Commit timestamp : 2021-04-07 21:36:38 +0530
Branch           : HEAD
Go version       : go1.16.2
jemalloc enabled : true

For Dgraph official documentation, visit https://dgraph.io/docs.
For discussions about Dgraph     , visit https://discuss.dgraph.io.
For fully-managed Dgraph Cloud   , visit https://dgraph.io/cloud.

Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2021 Dgraph Labs, Inc.



Running transaction with dgraph endpoint: 127.0.0.1:9080
While trying to setup connection: context deadline exceeded. Retrying...
2021/05/12 17:53:25 Could not setup connection after 1 retries
github.com/dgraph-io/dgraph/x.Fatalf
        /ext-go/1/src/github.com/dgraph-io/dgraph/x/error.go:120
github.com/dgraph-io/dgraph/x.GetDgraphClient
        /ext-go/1/src/github.com/dgraph-io/dgraph/x/x.go:1044
github.com/dgraph-io/dgraph/dgraph/cmd/live.run
        /ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/run.go:780
github.com/dgraph-io/dgraph/dgraph/cmd/live.init.0.func1
        /ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/run.go:134
github.com/spf13/cobra.(*Command).execute
        /go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
        /go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
        /go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
        /ext-go/1/src/github
.com/dgraph-io/dgraph/dgraph/cmd/root.go:78
main.main
        /ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/main.go:99
runtime.main
        /usr/local/go/src/runtime/proc.go:225
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1371

Cannot query the database either.

Im using dgraph alpha v21.03 on Ubuntu 20.04 installed with the installation script.
Dgraph service is executed with --cache size-mb 2048.
CPU 8 cores, 32 GB RAM

Is your Alpha reachable? is it localhost?

Nope, it is localhost

Please, share the Alpha logs.