raulk
1
Under what circumstances is this known to happen?
2020-11-20T20:23:39.601Z WARN badgerbs v2@v2.2007.2/levels.go:1005 [Compactor: 0] LOG Compact FAILED with error: MANIFEST removes non-existing table 164: {elog:0xc01e3c24e0 thisLevel:0xc00202c900 nextLevel:0xc00202c960 top:[0xc01aa13960 0xc0339f82a0 0xc0229adc00 0xc035db8620 0xc029a7b6c0 0xc01fc7a000 0xc01a6380e0 0xc01c02dea0 0xc015708c40 0xc021cacee0 0xc0200810a0 0xc020c27dc0 0xc01ace7c00 0xc02b1cc620 0xc0385fad20] bot:[0xc038b4ba40 0xc032c05180 0xc01a68ce00 0xc0226202a0 0xc0161a69a0 0xc037f120e0 0xc0320ae540 0xc02284d5e0] thisRange:{left:[] right:[] inf:true} nextRange:{left:[33 98 97 100 103 101 114 33 104 101 97 100 0 0 0 0 0 0 0 0] right:[47 98 108 111 99 107 115 47 85 68 83 65 69 73 72 90 90 90 87 79 79 82 88 69 77 72 80 68 79 67 69 83 67 84 86 72 79 88 90 74 66 86 79 53 84 90 70 66 73 80 52 68 84 52 52 87 81 87 75 79 85 65 90 51 52 85 255 255 255 255 255 255 255 255] inf:false} thisSize:0 dropPrefixes:[]}
2020-11-20T20:23:39.601Z WARN badgerbs v2@v2.2007.2/levels.go:411 While running doCompact: MANIFEST removes non-existing table 164
korjavin2
(Korjavin Ivan)
2
I’ve got the same and my alpha node can’t start now.
W0506 08:51:36.496078 22 log.go:36] While running doCompact: MANIFEST removes non-existing table 4684521
W0506 08:51:39.037852 22 log.go:36] [Compactor: 1] LOG Compact FAILED with error: MANIFEST removes non-existing table 4684521: {span:0xc00c0ca100 compactorId:1 t:{baseLevel:2 targetSz:[0 10485760 10485760 23078611 230786118 2307861185 23078611853] fileSz:[67108864 2097152 2097152 4194304 8388608 16777216 33554432]} p:{level:0 score:3 adjusted:3.2947788531481423 dropPrefixes:[] t:{baseLevel:2 targetSz:[0 10485760 10485760 23078611 230786118 2307861185 23078611853] fileSz:[67108864 2097152 2097152 4194304 8388608 16777216 33554432]}} thisLevel:0xc00060f020 nextLevel:0xc00060f0e0 top:[0xc04b411140 0xc014ef1440 0xc052bb0a80 0xc078c59200 0xc06a1b2540 0xc0314392c0 0xc0061d75c0 0xc020b62000 0xc03b7a2540 0xc064470f00 0xc037887e00 0xc024ca8000 0xc04d9a9680 0xc032b758c0 0xc022d70240] bot:[0xc0333a6b40 0xc053706540 0xc02de96000 0xc019e18240] thisRange:{left:[0 0 0 0 0 0 0 0 0 0 3 120 105 100 2 2 82 49 78 73 68 88 53 52 76 79 65 69 76 85 0 0 0 0 0 0 0 0] right:[4 0 0 0 0 0 0 0 0 0 32 101 98 97 121 46 108 105 115 116 105 110 103 46 98 101 108 111 110 103 115 95 116 111 95 99 97 116 101 103 111 114 121 4 0 0 0 0 11 16 85 75 0 0 0 0 13 57 202 2 255 255 255 255 255 255 255 255] inf:false size:0} nextRange:{left:[0 0 0 0 0 0 0 0 0 0 21 101 98 97 121 46 108 105 115 116 105 110 103 46 108 111 99 97 116 105 111 110 2 10 116 99 104 0 0 0 0 0 0 0 0] right:[4 0 0 0 0 0 0 0 0 0 11 100 103 114 97 112 104 46 116 121 112 101 2 2 101 98 97 121 46 108 105 115 116 105 110 103 0 0 0 0 11 104 132 188 255 255 255 255 255 255 255 255] inf:false size:0} splits:[] thisSize:0 dropPrefixes:[]}
W0506 08:51:39.037996 22 log.go:36] While running doCompact: MANIFEST removes non-existing table 4684521
W0506 08:51:41.607176 22 log.go:36] [Compactor: 1] LOG Compact FAILED with error: MANIFEST removes non-existing table 4684521: {span:0xc007376a00 compactorId:1 t:{baseLevel:2 targetSz:[0 10485760 10485760 23078611 230786118 2307861185 23078611853] fileSz:[67108864 2097152 2097152 4194304 8388608 16777216 33554432]} p:{level:0 score:3 adjusted:3.2947788531481423 dropPrefixes:[] t:{baseLevel:2 targetSz:[0 10485760 10485760 23078611 230786118 2307861185 23078611853] fileSz:[67108864 2097152 2097152 4194304 8388608 16777216 33554432]}} thisLevel:0xc00060f020 nextLevel:0xc00060f0e0 top:[0xc04b411140 0xc014ef1440 0xc052bb0a80 0xc078c59200 0xc06a1b2540 0xc0314392c0 0xc0061d75c0 0xc020b62000 0xc03b7a2540 0xc064470f00 0xc037887e00 0xc024ca8000 0xc04d9a9680 0xc032b758c0 0xc022d70240] bot:[0xc0333a6b40 0xc053706540 0xc02de96000 0xc019e18240] thisRange:{left:[0 0 0 0 0 0 0 0 0 0 3 120 105 100 2 2 82 49 78 73 68 88 53 52 76 79 65 69 76 85 0 0 0 0 0 0 0 0] right:[4 0 0 0 0 0 0 0 0 0 32 101 98 97 121 46 108 105 115 116 105 110 103 46 98 101 108 111 110 103 115 95 116 111 95 99 97 116 101 103 111 114 121 4 0 0 0 0 11 16 85 75 0 0 0 0 13 57 202 2 255 255 255 255 255 255 255 255] inf:false size:0} nextRange:{left:[0 0 0 0 0 0 0 0 0 0 21 101 98 97 121 46 108 105 115 116 105 110 103 46 108 111 99 97 116 105 111 110 2 10 116 99 104 0 0 0 0 0 0 0 0] right:[4 0 0 0 0 0 0 0 0 0 11 100 103 114 97 112 104 46 116 121 112 101 2 2 101 98 97 121 46 108 105 115 116 105 110 103 0 0 0 0 11 104 132 188 255 255 255 255 255 255 255 255] inf:false size:0} splits:[] thisSize:0 dropPrefixes:[]}
W0506 08:51:41.607325 22 log.go:36] While running doCompact: MANIFEST removes non-existing table 4684521
W0506 08:51:44.201397 22 log.go:36] [Compactor: 1] LOG Compact FAILED with error: MANIFEST removes non-existing table 4684521: {span:0xc029dfb700 compactorId:1 t:{baseLevel:2 targetSz:[0 10485760 10485760 23078611 230786118 2307861185 23078611853] fileSz:[67108864 2097152 2097152 4194304 8388608 16777216 33554432]} p:{level:0 score:3 adjusted:3.2947788531481423 dropPrefixes:[] t:{baseLevel:2 targetSz:[0 10485760 10485760 23078611 230786118 2307861185 23078611853] fileSz:[67108864 2097152 2097152 4194304 8388608 16777216 33554432]}} thisLevel:0xc00060f020 nextLevel:0xc00060f0e0 top:[0xc04b411140 0xc014ef1440 0xc052bb0a80 0xc078c59200 0xc06a1b2540 0xc0314392c0 0xc0061d75c0 0xc020b62000 0xc03b7a2540 0xc064470f00 0xc037887e00 0xc024ca8000 0xc04d9a9680 0xc032b758c0 0xc022d70240] bot:[0xc0333a6b40 0xc053706540 0xc02de96000 0xc019e18240] thisRange:{left:[0 0 0 0 0 0 0 0 0 0 3 120 105 100 2 2 82 49 78 73 68 88 53 52 76 79 65 69 76 85 0 0 0 0 0 0 0 0] right:[4 0 0 0 0 0 0 0 0 0 32 101 98 97 121 46 108 105 115 116 105 110 103 46 98 101 108 111 110 103 115 95 116 111 95 99 97 116 101 103 111 114 121 4 0 0 0 0 11 16 85 75 0 0 0 0 13 57 202 2 255 255 255 255 255 255 255 255] inf:false size:0} nextRange:{left:[0 0 0 0 0 0 0 0 0 0 21 101 98 97 121 46 108 105 115 116 105 110 103 46 108 111 99 97 116 105 111 110 2 10 116 99 104 0 0 0 0 0 0 0 0] right:[4 0 0 0 0 0 0 0 0 0 11 100 103 114 97 112 104 46 116 121 112 101 2 2 101 98 97 121 46 108 105 115 116 105 110 103 0 0 0 0 11 104 132 188 255 255 255 255 255 255 255 255] inf:false size:0} splits:[] thisSize:0 dropPrefixes:[]}
W0506 08:51:44.201520 22 log.go:36] While running doCompact: MANIFEST removes non-existing table 4684521
After adding a lot of resources and restarting I’ve got
I0506 09:08:07.849888 21 storage.go:125] Init Raft Storage with snap: 1314625848, first: 1314625849, last: 1314774484
I0506 09:08:07.849905 21 server_state.go:141] Opening postings BadgerDB with options: {Dir:p ValueDir:p SyncWrites:false NumVersionsToKeep:2147483647 ReadOnly:false Logger:0x32c8038 Compression:1 InMemory:false MetricsEnabled:true NumGoroutines:8 MemTableSize:67108864 BaseTableSize:2097152 BaseLevelSize:10485760 LevelSizeMultiplier:10 TableSizeMultiplier:2 MaxLevels:7 VLogPercentile:0 ValueThreshold:1048576 NumMemtables:5 BlockSize:4096 BloomFalsePositive:0.01 BlockCacheSize:697932185 IndexCacheSize:375809638 NumLevelZeroTables:5 NumLevelZeroTablesStall:15 ValueLogFileSize:1073741823 ValueLogMaxEntries:1000000 NumCompactors:4 CompactL0OnClose:false LmaxCompaction:false ZSTDCompressionLevel:0 VerifyValueChecksum:false EncryptionKey:[] EncryptionKeyRotationDuration:240h0m0s BypassLockGuard:false ChecksumVerificationMode:0 DetectConflicts:false NamespaceOffset:1 managedTxns:false maxBatchCount:0 maxBatchSize:0 maxValueThreshold:0}
E0506 09:08:09.847222 21 log.go:32] Received err: file does not exist for table 4679668. Cleaning up...
[Sentry] 2021/05/06 09:08:09 Sending fatal event [2f384924cf264f9fbd53d297c5c9fb10] to o318308.ingest.sentry.io project: 1805390
2021/05/06 09:08:09 file does not exist for table 4679668
Error while creating badger KV posting store
github.com/dgraph-io/dgraph/x.Checkf
/ext-go/1/src/github.com/dgraph-io/dgraph/x/error.go:51
github.com/dgraph-io/dgraph/worker.(*ServerState).initStorage
/ext-go/1/src/github.com/dgraph-io/dgraph/worker/server_state.go:145
github.com/dgraph-io/dgraph/worker.InitServerState
/ext-go/1/src/github.com/dgraph-io/dgraph/worker/server_state.go:80
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.run
/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:760
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.init.2.func1
/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:93
github.com/spf13/cobra.(*Command).execute
/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:78
main.main
/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/main.go:99
runtime.main
/usr/local/go/src/runtime/proc.go:225
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1371