Key not Found Error

Dave_Aitel · September 19, 2021, 2:31pm

When doing a rather large import of some data (just using the Python API) I ended up having a lot of weird errors (see below). I’m not sure what caused them, or what the effects of this failure was (did my edges not get created because the indexer fell behind?!?). Do I have TOO MANY edges on a single node?

Version from Docker: dgraph/standalone:v21.03.0

Thanks in advance! I’m going to try 21.03.1 just to see if it by chance fixes this error. I saw there was one other person who reported this in the discuss repo, but no answers for them.

W0911 23:59:49.522646 41 mvcc.go:148] Error failed when calling List.rollup: while encoding: cannot iterate through the list: cannot initialize iterator when calling List.iterate: cannot read initial list part for list with base key 000000000000000000000a656d61696c5f74657874020a6c6167: could not read list part with key 040000000000000000000a656d61696c5f74657874020a6c61670000000000000001: Key not found rolling up key [0 0 0 0 0 0 0 0 0 0 10 101 109 97 105 108 95 116 101 120 116 2 10 108 97 103]
W0911 23:59:49.702710 41 mvcc.go:148] Error failed when calling List.rollup: while encoding: cannot iterate through the list: cannot initialize iterator when calling List.iterate: cannot read initial list part for list with base key 000000000000000000000a656d61696c5f74657874020a736974: could not read list part with key 040000000000000000000a656d61696c5f74657874020a7369740000000000000001: Key not found rolling up key [0 0 0 0 0 0 0 0 0 0 10 101 109 97 105 108 95 116 101 120 116 2 10 115 105 116]
W0911 23:59:50.575229 41 mvcc.go:148] Error failed when calling List.rollup: while encoding: cannot iterate through the list: cannot initialize iterator when calling List.iterate: cannot read initial list part for list with base key 000000000000000000000a656d61696c5f74657874020a73742d: could not read list part with key 040000000000000000000a656d61696c5f74657874020a73742d0000000000000001: Key not found rolling up key [0 0 0 0 0 0 0 0 0 0 10 101 109 97 105 108 95 116 101 120 116 2 10 115 116 45]
W0911 23:59:52.365229 41 mvcc.go:148] Error failed when calling List.rollup: while encoding: cannot iterate through the list: cannot initialize iterator when calling List.iterate: cannot read initial list part for list with base key 000000000000000000000a656d61696c5f74657874020a32323b: could not read list part with key 040000000000000000000a656d61696c5f74657874020a32323b0000000000000001: Key not found rolling up key [0 0 0 0 0 0 0 0 0 0 10 101 109 97 105 108 95 116 101 120 116 2 10 50 50 59]
W0911 23:59:52.500303 41 mvcc.go:148] Error failed when calling List.rollup: while encoding: cannot iterate through the list: cannot initialize iterator when calling List.iterate: cannot read initial list part for list with base key 000000000000000000000a656d61696c5f74657874020a726f62: could not read list part with key 040000000000000000000a656d61696c5f74657874020a726f620000000000000001: Key not found rolling up key [0 0 0 0 0 0 0 0 0 0 10 101 109 97 105 108 95 116 101 120 116 2 10 114 111 98]

ahsan · September 19, 2021, 7:40pm

@Dave_Aitel This issue has been fixed and will be shipped in the next patch release of 21.03. You’d not see this issue in the master build.

Dave_Aitel · September 19, 2021, 7:43pm

That’s awesome news. Is there a way to docker the new version now? (I assume there is…but I just want to make sure I can test with this installed!)

Dave_Aitel · September 19, 2021, 8:43pm

Ah, nevermind, you just switch the version for “master” and it comes right up. Although this isn’t compatible with the old DB directory.

Dave_Aitel · September 20, 2021, 3:09am

@ahsan On Master I get this error instead:

19306 319310 318814 .] -> [319587 319598 319604 319579 319589 319600 319602 319578 319581 .], took 2.206s
I0920 02:49:13.750365      25 log.go:34] [3] [E] LOG Compact 5->6 (1, 4 -> 4 tables with 2 splits). [319378 . 316157 316188 316189 319407 .] -> [319569 319646 319559 319561 .], took 5.932s
I0920 02:49:14.774285      25 log.go:34] [0] [E] LOG Compact 5->6 (1, 5 -> 6 tables with 2 splits). [318835 . 319551 314367 314347 318455 318475 .] -> [319631 319661 319668 319626 319650 319652 .], took 3.413s
I0920 02:49:17.219042      25 log.go:34] [1] [E] LOG Compact 4->5 (1, 8 -> 9 tables with 3 splits). [319584 . 319031 319498 319506 319507 319505 319518 319519 319471 .] -> [319694 319711 319722 319724 319693 319710 319714 319696 319700 .], took 2.279s
panic: runtime error: makeslice: len out of range

goroutine 163 [running]:
github.com/dgraph-io/badger/v3.(*levelsController).fillTables(0xc0001c81c0, 0xc0025df780, 0xc000044000)
	/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.0.0-20210916095630-2a9a524b649b/levels.go:1387 +0x453
github.com/dgraph-io/badger/v3.(*levelsController).doCompact(0xc0001c81c0, 0x1, 0x5, 0x3ff01c1b0dee34e4, 0x3ff01c1b0dee34e4, 0x0, 0x0, 0x0, 0x3, 0xc028316d00, ...)
	/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.0.0-20210916095630-2a9a524b649b/levels.go:1529 +0x91e
github.com/dgraph-io/badger/v3.(*levelsController).runCompactor.func2(0x5, 0x3ff01c1b0dee34e4, 0x3ff01c1b0dee34e4, 0x0, 0x0, 0x0, 0x3, 0xc028316d00, 0x7, 0x7, ...)
	/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.0.0-20210916095630-2a9a524b649b/levels.go:466 +0x7a
github.com/dgraph-io/badger/v3.(*levelsController).runCompactor.func3(0xc0025dff30)
	/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.0.0-20210916095630-2a9a524b649b/levels.go:489 +0x158
github.com/dgraph-io/badger/v3.(*levelsController).runCompactor(0xc0001c81c0, 0x1, 0xc0026571a0)
	/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.0.0-20210916095630-2a9a524b649b/levels.go:518 +0x362
created by github.com/dgraph-io/badger/v3.(*levelsController).startCompact
	/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.0.0-20210916095630-2a9a524b649b/levels.go:355 +0x89
W0920 02:49:17.959385      10 sentry_integration.go:140] unable to read CID from file /tmp/dgraph-alpha-cid-sentry open /tmp/dgraph-alpha-cid-sentry: no such file or directory. Skip
[Sentry] 2021/09/20 02:49:17 Sending fatal event [19d4a7a86abc4e9093693229399e10df] to o318308.ingest.sentry.io project: 5208688
[Sentry] 2021/09/20 02:49:18 Buffer flushed successfully.

ahsan · September 20, 2021, 6:33am

@Dave_Aitel Master is still in the development phase, we didn’t know about this bug. Thanks for letting us know. We are working to fix this. We’ll let you know.

ahsan · September 20, 2021, 7:03am

@Dave_Aitel Can you please share us the output of dgraph version and also the scenario in which you got this error?

ahsan · September 20, 2021, 7:12am

Also, can you please share your machine specs?

Dave_Aitel · September 20, 2021, 2:25pm

Hi! Thanks for your response! Much appreciated!

This is a VM on a high-powered gaming laptop that I’m using for testing. The VM itself has 23G of RAM in it. But I have only allocated 1 CPU to it, so it is not fast.

I can’t say EXACTLY what crashed it, but my script is quite simple: It has a long list of data and it does simple mutations to input them (from the Python API) and then it also queries it so it can connect lists of UIDs as edges. It’s generally not a SMALL amount of data.

[Sentry] 2021/09/20 14:14:20 Integration installed: IgnoreErrors
I0920 14:14:21.285634      23 sentry_integration.go:48] This instance of Dgraph will send anonymous reports of panics back to Dgraph Labs via Sentry. No confidential information is sent. These reports help improve Dgraph. To opt-out, restart your instance with the --telemetry "sentry=false;" flag. For more info, see https://dgraph.io/docs/howto/#data-handling.
I0920 14:14:21.303971      30 sentry_integration.go:48] This instance of Dgraph will send anonymous reports of panics back to Dgraph Labs via Sentry. No confidential information is sent. These reports help improve Dgraph. To opt-out, restart your instance with the --telemetry "sentry=false;" flag. For more info, see https://dgraph.io/docs/howto/#data-handling.
I0920 14:14:22.189659      23 init.go:110] 

Dgraph version   : v21.09.0-rc2-13-gf6e8779a3
Dgraph codename  : unnamed
Dgraph SHA-256   : 42f94bcd9127fe464482d1dcd04d6aea4746340253d41ddf5ac905a43395c66b
Commit SHA-1     : f6e8779a3
Commit timestamp : 2021-09-16 16:15:27 -0700
Branch           : master
Go version       : go1.16.2
jemalloc enabled : true

For Dgraph official documentation, visit https://dgraph.io/docs.
For discussions about Dgraph     , visit http://discuss.dgraph.io.
For fully-managed Dgraph Cloud   , visit https://dgraph.io/cloud.

Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2021 Dgraph Labs, Inc.

I0920 14:14:22.191083      23 run.go:254] Setting Config to: {raft:0xc000012d78 telemetry:0xc000012d48 limit:0xc000012d80 bindall:true portOffset:0 numReplicas:1 peer: w:zw rebalanceInterval:480000000000 tlsClientConfig:<nil> audit:<nil> limiterConfig:0xc000045b50}
I0920 14:14:22.191224      23 run.go:144] Setting up grpc listener at: 0.0.0.0:5080
I0920 14:14:22.191455      23 run.go:144] Setting up http listener at: 0.0.0.0:6080
I0920 14:14:22.191739      23 log.go:295] Found file: 7 First Index: 180001
I0920 14:14:22.192688      23 storage.go:125] Init Raft Storage with snap: 205820, first: 205821, last: 206528
I0920 14:14:22.192881      23 node.go:152] Setting raft.Config to: &{ID:1 peers:[] learners:[] ElectionTick:20 HeartbeatTick:1 Storage:0xc00017ef00 Applied:205820 MaxSizePerMsg:262144 MaxCommittedSizePerReady:67108864 MaxUncommittedEntriesSize:0 MaxInflightMsgs:256 CheckQuorum:false PreVote:true ReadOnlyOption:0 Logger:0x3939e90 DisableProposalForwarding:false}
I0920 14:14:22.193738      23 node.go:310] Found Snapshot.Metadata: {ConfState:{Nodes:[1] Learners:[] XXX_unrecognized:[]} Index:205820 Term:2 XXX_unrecognized:[]}
I0920 14:14:22.193954      23 node.go:321] Found hardstate: {Term:2 Vote:1 Commit:206528 XXX_unrecognized:[]}
I0920 14:14:22.214746      30 init.go:110] 

Dgraph version   : v21.09.0-rc2-13-gf6e8779a3
Dgraph codename  : unnamed
Dgraph SHA-256   : 42f94bcd9127fe464482d1dcd04d6aea4746340253d41ddf5ac905a43395c66b
Commit SHA-1     : f6e8779a3
Commit timestamp : 2021-09-16 16:15:27 -0700
Branch           : master
Go version       : go1.16.2
jemalloc enabled : true

For Dgraph official documentation, visit https://dgraph.io/docs.
For discussions about Dgraph     , visit http://discuss.dgraph.io.
For fully-managed Dgraph Cloud   , visit https://dgraph.io/cloud.

Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2021 Dgraph Labs, Inc.


I0920 14:14:22.216163      30 run.go:901] x.Config: {PortOffset:0 Limit:mutations-nquad=1000000; disallow-drop=false; query-timeout=0ms; txn-abort-after=5m; max-pending-queries=64; max-retries=-1; mutations=allow; normalize-node=10000; shared-instance=false; query-edge=1000000 LimitMutationsNquad:1000000 LimitQueryEdge:1000000 BlockClusterWideDrop:false LimitNormalizeNode:10000 QueryTimeout:0s MaxRetries:-1 SharedInstance:false GraphQL:{Introspection:true Debug:false Extensions:true PollInterval:1s} Lambda:{Url: Num:1 Port:20000 RestartAfter:10s}}
I0920 14:14:22.216319      30 run.go:902] x.WorkerConfig: {TmpDir:t ExportPath:export Trace:ratio=0.01; jaeger=; datadog= MyAddr: ZeroAddr:[localhost:5080] TLSClientConfig:<nil> TLSServerConfig:<nil> Raft:idx=; group=; learner=false; snapshot-after-entries=10000; snapshot-after-duration=30m; pending-proposals=256 Badger:{Dir: ValueDir: SyncWrites:false NumVersionsToKeep:1 ReadOnly:false Logger:0xc0001befd0 Compression:1 InMemory:false MetricsEnabled:true NumGoroutines:8 MemTableSize:67108864 BaseTableSize:2097152 BaseLevelSize:10485760 LevelSizeMultiplier:10 TableSizeMultiplier:2 MaxLevels:7 VLogPercentile:0 ValueThreshold:1048576 NumMemtables:15 BlockSize:4096 BloomFalsePositive:0.01 BlockCacheSize:322122547 IndexCacheSize:214748364 NumLevelZeroTables:5 NumLevelZeroTablesStall:15 ValueLogFileSize:1073741823 ValueLogMaxEntries:1000000 NumCompactors:4 CompactL0OnClose:false LmaxCompaction:false ZSTDCompressionLevel:0 VerifyValueChecksum:false EncryptionKey:[] EncryptionKeyRotationDuration:240h0m0s BypassLockGuard:false ChecksumVerificationMode:0 AllowStopTheWorld:true DetectConflicts:true NamespaceOffset:-1 ExternalMagicVersion:0 managedTxns:false maxBatchCount:0 maxBatchSize:0 maxValueThreshold:0} WhiteListedIPRanges:[{Lower:0.0.0.0 Upper:255.255.255.255}] StrictMutations:false AclEnabled:false HmacSecret:**** AbortOlderThan:5m0s ProposedGroupId:0 StartTime:2021-09-20 14:14:19.284176054 +0000 UTC m=+0.948071941 Security:token=; whitelist=0.0.0.0/0 EncryptionKey:**** LogRequest:0 HardSync:false Audit:false}
I0920 14:14:22.216610      30 run.go:903] worker.Config: {PostingDir:p WALDir:w MutationsMode:1 AuthToken: HmacSecret:**** AccessJwtTtl:0s RefreshJwtTtl:0s CachePercentage:50,30,20 CacheMb:1024 Audit:<nil> ChangeDataConf:file=; kafka=; sasl_user=; sasl_password=; ca_cert=; client_cert=; client_key=; sasl-mechanism=PLAIN;}
I0920 14:14:22.216868      30 log.go:295] Found file: 23 First Index: 660001
I0920 14:14:22.217111      30 log.go:295] Found file: 24 First Index: 690001
I0920 14:14:22.217271      30 log.go:295] Found file: 25 First Index: 720001
I0920 14:14:22.217424      30 log.go:295] Found file: 26 First Index: 750001
I0920 14:14:22.217599      30 storage.go:125] Init Raft Storage with snap: 685591, first: 685592, last: 763557
I0920 14:14:22.217737      30 server_state.go:140] Opening postings BadgerDB with options: {Dir:p ValueDir:p SyncWrites:false NumVersionsToKeep:2147483647 ReadOnly:false Logger:0x3939e90 Compression:1 InMemory:false MetricsEnabled:true NumGoroutines:8 MemTableSize:67108864 BaseTableSize:2097152 BaseLevelSize:10485760 LevelSizeMultiplier:10 TableSizeMultiplier:2 MaxLevels:7 VLogPercentile:0 ValueThreshold:1048576 NumMemtables:15 BlockSize:4096 BloomFalsePositive:0.01 BlockCacheSize:322122547 IndexCacheSize:214748364 NumLevelZeroTables:5 NumLevelZeroTablesStall:15 ValueLogFileSize:1073741823 ValueLogMaxEntries:1000000 NumCompactors:4 CompactL0OnClose:false LmaxCompaction:false ZSTDCompressionLevel:0 VerifyValueChecksum:false EncryptionKey:[] EncryptionKeyRotationDuration:240h0m0s BypassLockGuard:false ChecksumVerificationMode:0 AllowStopTheWorld:true DetectConflicts:false NamespaceOffset:1 ExternalMagicVersion:1 managedTxns:false maxBatchCount:0 maxBatchSize:0 maxValueThreshold:0}
I0920 14:14:22.226278      23 node.go:326] Group 0 found 26528 entries
I0920 14:14:22.228171      23 raft.go:621] Restarting node for dgraphzero
I0920 14:14:22.228368      23 node.go:189] Setting conf state to nodes:1 
I0920 14:14:22.230667      23 pool.go:164] CONN: Connecting to localhost:7080
I0920 14:14:22.230831      23 log.go:34] 1 became follower at term 2
I0920 14:14:22.230963      23 log.go:34] newRaft 1 [peers: [1], term: 2, commit: 206528, applied: 205820, lastindex: 206528, lastterm: 2]
I0920 14:14:22.234404      23 raft.go:476] Could not apply proposal, ignoring: p.MaxUID=0, p.MaxTxnTs=770000p.MaxNsID=0, maxUID=70000 maxTxnTs=770000 maxNsID=0
I0920 14:14:22.236169      23 run.go:388] Running Dgraph Zero...
I0920 14:14:22.314181      30 log.go:34] All 315 tables opened in 43ms
E0920 14:14:22.492768      30 log.go:32] Received err: Level validation err: Levels Controller err: Inter: Biggest(j-1)[315605] 
00000000  00 00 00 00 00 00 00 00  00 00 0a 65 6d 61 69 6c  |...........email|
00000010  5f 74 65 78 74 02 0a 30  0a 4d ff ff ff ff ff f5  |_text..0.M......|
00000020  96 bc                                             |..|

 vs Smallest(j)[315636]: 
00000000  00 00 00 00 00 00 00 00  00 00 0a 65 6d 61 69 6c  |...........email|
00000010  5f 74 65 78 74 02 0a 20  4b 4e ff ff ff ff ff f4  |_text.. KN......|
00000020  da f0                                             |..|

: level=6 j=24 numTables=237
github.com/dgraph-io/badger/v3.(*levelHandler).validate
	/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.0.0-20210916095630-2a9a524b649b/util.go:55
github.com/dgraph-io/badger/v3.(*levelsController).validate
	/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.0.0-20210916095630-2a9a524b649b/util.go:33
github.com/dgraph-io/badger/v3.newLevelsController
	/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.0.0-20210916095630-2a9a524b649b/levels.go:180
github.com/dgraph-io/badger/v3.Open
	/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.0.0-20210916095630-2a9a524b649b/db.go:344
github.com/dgraph-io/badger/v3.OpenManaged
	/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.0.0-20210916095630-2a9a524b649b/managed_db.go:26
github.com/dgraph-io/dgraph/worker.(*ServerState).initStorage
	/ext-go/1/src/github.com/dgraph-io/dgraph/worker/server_state.go:143
github.com/dgraph-io/dgraph/worker.InitServerState
	/ext-go/1/src/github.com/dgraph-io/dgraph/worker/server_state.go:78
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.run
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:905
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.init.1.func1
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:101
github.com/spf13/cobra.(*Command).execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:79
main.main
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/main.go:112
runtime.main
	/usr/local/go/src/runtime/proc.go:225
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1371. Cleaning up...
[Sentry] 2021/09/20 14:14:22 Sending fatal event [bbff13d966eb4b01b03d32dc6734cfb3] to o318308.ingest.sentry.io project: 5208688
2021/09/20 14:14:22 Level validation err: Levels Controller err: Inter: Biggest(j-1)[315605] 
00000000  00 00 00 00 00 00 00 00  00 00 0a 65 6d 61 69 6c  |...........email|
00000010  5f 74 65 78 74 02 0a 30  0a 4d ff ff ff ff ff f5  |_text..0.M......|
00000020  96 bc                                             |..|

 vs Smallest(j)[315636]: 
00000000  00 00 00 00 00 00 00 00  00 00 0a 65 6d 61 69 6c  |...........email|
00000010  5f 74 65 78 74 02 0a 20  4b 4e ff ff ff ff ff f4  |_text.. KN......|
00000020  da f0                                             |..|

: level=6 j=24 numTables=237
github.com/dgraph-io/badger/v3.(*levelHandler).validate
	/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.0.0-20210916095630-2a9a524b649b/util.go:55
github.com/dgraph-io/badger/v3.(*levelsController).validate
	/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.0.0-20210916095630-2a9a524b649b/util.go:33
github.com/dgraph-io/badger/v3.newLevelsController
	/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.0.0-20210916095630-2a9a524b649b/levels.go:180
github.com/dgraph-io/badger/v3.Open
	/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.0.0-20210916095630-2a9a524b649b/db.go:344
github.com/dgraph-io/badger/v3.OpenManaged
	/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.0.0-20210916095630-2a9a524b649b/managed_db.go:26
github.com/dgraph-io/dgraph/worker.(*ServerState).initStorage
	/ext-go/1/src/github.com/dgraph-io/dgraph/worker/server_state.go:143
github.com/dgraph-io/dgraph/worker.InitServerState
	/ext-go/1/src/github.com/dgraph-io/dgraph/worker/server_state.go:78
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.run
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:905
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.init.1.func1
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:101
github.com/spf13/cobra.(*Command).execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:79
main.main
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/main.go:112
runtime.main
	/usr/local/go/src/runtime/proc.go:225
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1371
Error while creating badger KV posting store
github.com/dgraph-io/dgraph/x.Checkf
	/ext-go/1/src/github.com/dgraph-io/dgraph/x/error.go:51
github.com/dgraph-io/dgraph/worker.(*ServerState).initStorage
	/ext-go/1/src/github.com/dgraph-io/dgraph/worker/server_state.go:144
github.com/dgraph-io/dgraph/worker.InitServerState
	/ext-go/1/src/github.com/dgraph-io/dgraph/worker/server_state.go:78
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.run
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:905
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.init.1.func1
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:101
github.com/spf13/cobra.(*Command).execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:79
main.main
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/main.go:112
runtime.main
	/usr/local/go/src/runtime/proc.go:225
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1371

ahsan · September 22, 2021, 11:37am

Is it possible for you to share with us the p directory to help us run further investigations? How big it is?

Dave_Aitel · September 22, 2021, 3:06pm

I ended up deleting it because I couldn’t run DGraph - but I’ll regenerate it with my script and let you know!

Dave_Aitel · September 23, 2021, 5:04am

I regenerated the crash - and my p directory is 2.8G. Unsure how to share that

I0923 04:17:48.080760 22 log.go:34] [1] [E] LOG Compact 5->6 (1, 5 → 6 tables with 2 splits). [254009 . 250297 250075 250103 250114 250053 .] → [254497 254531 254541 254545 254490 254520 .], took 3.079s
panic: runtime error: makeslice: len out of range

goroutine 8128290 [running]:
github.com/dgraph-io/badger/v3.(*levelsController).fillTables(0xc0001c80e0, 0xc0064e9780, 0xc000044000)
/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.0.0-20210916095630-2a9a524b649b/levels.go:1387 +0x453
github.com/dgraph-io/badger/v3.(*levelsController).doCompact(0xc0001c80e0, 0x2, 0x5, 0x3ff01f0fc20025e7, 0x3ff01f0fc20025e7, 0x0, 0x0, 0x0, 0x3, 0xc02a1cb2c0, …)
/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.0.0-20210916095630-2a9a524b649b/levels.go:1529 +0x91e
github.com/dgraph-io/badger/v3.(*levelsController).runCompactor.func2(0x5, 0x3ff01f0fc20025e7, 0x3ff01f0fc20025e7, 0x0, 0x0, 0x0, 0x3, 0xc02a1cb2c0, 0x7, 0x7, …)
/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.0.0-20210916095630-2a9a524b649b/levels.go:466 +0x7a
github.com/dgraph-io/badger/v3.(*levelsController).runCompactor.func3(0xc0064e9f30)
/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.0.0-20210916095630-2a9a524b649b/levels.go:489 +0x158
github.com/dgraph-io/badger/v3.(*levelsController).runCompactor(0xc0001c80e0, 0x2, 0xc03158ea80)
/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.0.0-20210916095630-2a9a524b649b/levels.go:518 +0x362
created by github.com/dgraph-io/badger/v3.(*levelsController).startCompact
/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.0.0-20210916095630-2a9a524b649b/levels.go:355 +0x89
W0923 04:17:49.362548 8 sentry_integration.go:140] unable to read CID from file /tmp/dgraph-alpha-cid-sentry open /tmp/dgraph-alpha-cid-sentry: no such file or directory. Skip
[Sentry] 2021/09/23 04:17:49 Sending fatal event [a888635bf9534ef7bfdfba5df81d4167] to o318308.ingest.sentry.io project: 5208688
[Sentry] 2021/09/23 04:17:49 Too many requests, backing off till: 2021-09-23 04:18:09.718519699 +0000 UTC m=+73091.918170657
[Sentry] 2021/09/23 04:17:49 Buffer flushed successfully.

ahsan · September 23, 2021, 7:43am

Can you please tar it and upload to google drive (or any other cloud storage). Once we download it, you can free up your cloud space.

Or, if your data is synthetic. Is it possible for me to run your script on my end?

Dave_Aitel · September 23, 2021, 3:43pm

I can but probably not till tonight or Friday morning. Do you have an email address so that I can send you the link privately?

ahsan · September 23, 2021, 4:16pm

Yes. Please send it to - ahsan@dgraph.io

Thanks a lot.

Dave_Aitel · September 24, 2021, 3:01pm

Sent!

-dave

ahsan · September 24, 2021, 3:14pm

Ack! I got it. Thanks a lot.

Naman · September 28, 2021, 9:04am

@Dave_Aitel seems like you are running it on HA cluster. Can you please freshly reproduce it and capture the logs as well.

Dave_Aitel · September 29, 2021, 3:21am

I can - but tomorrow. Much too tired tonight.

Dave_Aitel · September 29, 2021, 1:08pm

Ok - this is what I have. Do you want me to just do this and run my import code again?

dave@dave-VirtualBox:~/$ cat rundockers.sh
#!/bin/sh
ulimit -S -n 1048576
docker run --rm -it -p “8080:8080” -p “9080:9080” -v ~/dgraph:/dgraph “dgraph/standalone:master”

Topic		Replies	Views
Rollup - Error in readListPart() Dgraph status:accepted , ticket:created	14	1368	January 14, 2022
New crash bug? Dgraph kind:question , dgraph , kind:bug	4	655	December 13, 2021
Error: file with ID: X not found Dgraph dgraph , kind:bug	6	844	April 20, 2021
How to view logs in dgraph,cannot retrieve UIDs from list with key Dgraph	7	545	January 25, 2021
Dgraph v1.2.2 release Announce	0	582	March 19, 2020

Key not Found Error

Related Topics