Hi team, Continuously facing these 3 errors while running badger. What will be the solution to this?
- Error 1
badger 2023/09/21 12:38:37 ERROR: Got error: While opening memtable: /data/dedupe/data/table_50/Pentagram/00067.mem error: while opening file: /data/dedupe/data/table_50/Pentagram/00067.mem error: open /data/dedupe/data/table_50/Pentagram/00067.mem: no such file or directory
unable to open: /data/dedupe/data/table_50/Pentagram/00067.mem
github.com/dgraph-io/ristretto/z.OpenMmapFile
/data/badger_broadband/badger-cluster/vendor/github.com/dgraph-io/ristretto/z/file.go:80
github.com/dgraph-io/badger/v4.(*logFile).open
/data/badger_broadband/badger-cluster/vendor/github.com/dgraph-io/badger/v4/memtable.go:554
github.com/dgraph-io/badger/v4.(*DB).openMemTable
/data/badger_broadband/badger-cluster/vendor/github.com/dgraph-io/badger/v4/memtable.go:130
github.com/dgraph-io/badger/v4.(*DB).newMemTable
/data/badger_broadband/badger-cluster/vendor/github.com/dgraph-io/badger/v4/memtable.go:151
github.com/dgraph-io/badger/v4.(*DB).ensureRoomForWrite
/data/badger_broadband/badger-cluster/vendor/github.com/dgraph-io/badger/v4/db.go:995
github.com/dgraph-io/badger/v4.(*DB).writeRequests
/data/badger_broadband/badger-cluster/vendor/github.com/dgraph-io/badger/v4/db.go:829
github.com/dgraph-io/badger/v4.(*DB).doWrites.func1
/data/badger_broadband/badger-cluster/vendor/github.com/dgraph-io/badger/v4/db.go:884
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598 for id: 67
badger 2023/09/21 12:38:37 ERROR: writeRequests: writeRequests err: cannot create new mem table error: newMemTable error: While opening memtable: /data/dedupe/data/table_50/Pentagram/00067.mem error: while opening file: /data/dedupe/data/table_50/Pentagram/00067.mem error: open /data/dedupe/data/table_50/Pentagram/00067.mem: no such file or directory
unable to open: /data/dedupe/data/table_50/Pentagram/00067.mem
github.com/dgraph-io/ristretto/z.OpenMmapFile
/data/badger_broadband/badger-cluster/vendor/github.com/dgraph-io/ristretto/z/file.go:80
github.com/dgraph-io/badger/v4.(*logFile).open
/data/badger_broadband/badger-cluster/vendor/github.com/dgraph-io/badger/v4/memtable.go:554
github.com/dgraph-io/badger/v4.(*DB).openMemTable
/data/badger_broadband/badger-cluster/vendor/github.com/dgraph-io/badger/v4/memtable.go:130
github.com/dgraph-io/badger/v4.(*DB).newMemTable
/data/badger_broadband/badger-cluster/vendor/github.com/dgraph-io/badger/v4/memtable.go:151
github.com/dgraph-io/badger/v4.(*DB).ensureRoomForWrite
/data/badger_broadband/badger-cluster/vendor/github.com/dgraph-io/badger/v4/db.go:995
github.com/dgraph-io/badger/v4.(*DB).writeRequests
/data/badger_broadband/badger-cluster/vendor/github.com/dgraph-io/badger/v4/db.go:829
github.com/dgraph-io/badger/v4.(*DB).doWrites.func1
/data/badger_broadband/badger-cluster/vendor/github.com/dgraph-io/badger/v4/db.go:884
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
badger 2023/09/21 12:38:38 ERROR: Failure while flushing memtable to disk: error while creating table err: while creating table: /data/dedupe/data/table_50/Pentagram/000133.sst error: open /data/dedupe/data/table_50/Pentagram/000133.sst: no such file or directory
unable to open: /data/dedupe/data/table_50/Pentagram/000133.sst
github.com/dgraph-io/ristretto/z.OpenMmapFile
/data/badger_broadband/badger-cluster/vendor/github.com/dgraph-io/ristretto/z/file.go:80
github.com/dgraph-io/badger/v4/table.CreateTable
/data/badger_broadband/badger-cluster/vendor/github.com/dgraph-io/badger/v4/table/table.go:259
github.com/dgraph-io/badger/v4.(*DB).handleFlushTask
/data/badger_broadband/badger-cluster/vendor/github.com/dgraph-io/badger/v4/db.go:1061
github.com/dgraph-io/badger/v4.(*DB).flushMemtable
/data/badger_broadband/badger-cluster/vendor/github.com/dgraph-io/badger/v4/db.go:1083
github.com/dgraph-io/badger/v4.Open.func5
/data/badger_broadband/badger-cluster/vendor/github.com/dgraph-io/badger/v4/db.go:354
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598. Retrying...
total time: cluster time for 50 took 13m33.967768745s
panic: cannot create new mem table error: newMemTable error: While opening memtable: /data/dedupe/data/table_50/Pentagram/00067.mem error: while opening file: /data/dedupe/data/table_50/Pentagram/00067.mem error: open /data/dedupe/data/table_50/Pentagram/00067.mem: no such file or directory
unable to open: /data/dedupe/data/table_50/Pentagram/00067.mem
github.com/dgraph-io/ristretto/z.OpenMmapFile
/data/badger_broadband/badger-cluster/vendor/github.com/dgraph-io/ristretto/z/file.go:80
github.com/dgraph-io/badger/v4.(*logFile).open
/data/badger_broadband/badger-cluster/vendor/github.com/dgraph-io/badger/v4/memtable.go:554
github.com/dgraph-io/badger/v4.(*DB).openMemTable
/data/badger_broadband/badger-cluster/vendor/github.com/dgraph-io/badger/v4/memtable.go:130
github.com/dgraph-io/badger/v4.(*DB).newMemTable
/data/badger_broadband/badger-cluster/vendor/github.com/dgraph-io/badger/v4/memtable.go:151
github.com/dgraph-io/badger/v4.(*DB).ensureRoomForWrite
/data/badger_broadband/badger-cluster/vendor/github.com/dgraph-io/badger/v4/db.go:995
github.com/dgraph-io/badger/v4.(*DB).writeRequests
/data/badger_broadband/badger-cluster/vendor/github.com/dgraph-io/badger/v4/db.go:829
github.com/dgraph-io/badger/v4.(*DB).doWrites.func1
/data/badger_broadband/badger-cluster/vendor/github.com/dgraph-io/badger/v4/db.go:884
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
goroutine 3487 [running]:
cluster/badger_cluster.(*Cluster).AddMemberAddress(0xc490f28df0, {0xbf7d98, 0xc000194000}, 0x32?, 0x32, {0xc48e35f800, 0x2f, 0x7f521b3908f0?})
/data/badger_broadband/badger-cluster/badger_cluster/cluster.go:69 +0x276
main.makeCluster({0xbf7d98, 0xc000194000}, 0x32)
/data/badger_broadband/badger-cluster/app/create_cluster.go:99 +0x3ea
main.worker(...)
/data/badger_broadband/badger-cluster/app/create_cluster.go:59
main.main.func1()
/data/badger_broadband/badger-cluster/app/create_cluster.go:45 +0x8e
created by main.main
/data/badger_broadband/badger-cluster/app/create_cluster.go:42 +0x7e
- Error 2
panic: runtime error: slice bounds out of range [-4169358568:] [recovered]
panic: runtime error: slice bounds out of range [-4169358568:]
panic:
== Recovering from initIndex crash ==
File Info: [ID: 95, Size: 69121215, Zeros: 0]
isEnrypted: false checksumLen: 4238479779
== Recovered ==
goroutine 6932842 [running]:
github.com/dgraph-io/badger/v4/table.(*Table).initBiggestAndSmallest.func1.1()
/data/badger_broadband/badger_grpc/vendor/github.com/dgraph-io/badger/v4/table/table.go:353 +0xa8
panic({0xba9420, 0xc4324db2d8})
/usr/local/go/src/runtime/panic.go:884 +0x212
github.com/dgraph-io/ristretto/z.(*MmapFile).Bytes(...)
/data/badger_broadband/badger_grpc/vendor/github.com/dgraph-io/ristretto/z/file.go:116
github.com/dgraph-io/badger/v4/table.(*Table).read(...)
/data/badger_broadband/badger_grpc/vendor/github.com/dgraph-io/badger/v4/table/table.go:420
github.com/dgraph-io/badger/v4/table.(*Table).readNoFail(0xd0a740?, 0xc4a7b68810?, 0xbfe74d?)
/data/badger_broadband/badger_grpc/vendor/github.com/dgraph-io/badger/v4/table/table.go:424 +0xd0
github.com/dgraph-io/badger/v4/table.(*Table).initBiggestAndSmallest.func1()
/data/badger_broadband/badger_grpc/vendor/github.com/dgraph-io/badger/v4/table/table.go:383 +0x2ff
panic({0xba9420, 0xc4324db2c0})
/usr/local/go/src/runtime/panic.go:884 +0x212
github.com/dgraph-io/ristretto/z.(*MmapFile).Bytes(...)
/data/badger_broadband/badger_grpc/vendor/github.com/dgraph-io/ristretto/z/file.go:116
github.com/dgraph-io/badger/v4/table.(*Table).read(...)
/data/badger_broadband/badger_grpc/vendor/github.com/dgraph-io/badger/v4/table/table.go:420
github.com/dgraph-io/badger/v4/table.(*Table).readNoFail(0x6?, 0x0?, 0x9?)
/data/badger_broadband/badger_grpc/vendor/github.com/dgraph-io/badger/v4/table/table.go:424 +0xd0
github.com/dgraph-io/badger/v4/table.(*Table).initIndex(0xc47e194180)
/data/badger_broadband/badger_grpc/vendor/github.com/dgraph-io/badger/v4/table/table.go:445 +0x109
github.com/dgraph-io/badger/v4/table.(*Table).initBiggestAndSmallest(0xc47e194180)
/data/badger_broadband/badger_grpc/vendor/github.com/dgraph-io/badger/v4/table/table.go:403 +0x7f
github.com/dgraph-io/badger/v4/table.OpenTable(0xc31e8f4160, {0x0, 0x1, 0x1000000, 0xf33333, 0x0, 0x3f847ae147ae147b, 0x1000, 0x0, 0x1, ...})
/data/badger_broadband/badger_grpc/vendor/github.com/dgraph-io/badger/v4/table/table.go:309 +0x278
github.com/dgraph-io/badger/v4/table.CreateTable({0xc167146b80, 0x33}, 0xccef5a4090)
/data/badger_broadband/badger_grpc/vendor/github.com/dgraph-io/badger/v4/table/table.go:273 +0x305
github.com/dgraph-io/badger/v4.(*levelsController).subcompact.func4(0xccef5a4090, 0xc025c8b6e0?)
/data/badger_broadband/badger_grpc/vendor/github.com/dgraph-io/badger/v4/levels.go:846 +0x11b
created by github.com/dgraph-io/badger/v4.(*levelsController).subcompact
/data/badger_broadband/badger_grpc/vendor/github.com/dgraph-io/badger/v4/levels.go:836 +0x707
- Error 3
panic: runtime error: invalid memory address or nil pointer dereference
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x941247]
goroutine 2551 [running]:
github.com/dgraph-io/badger/v4/table.(*Table).offsetsLength(...)
/data/badger_broadband/badger_grpc/vendor/github.com/dgraph-io/badger/v4/table/table.go:133
github.com/dgraph-io/badger/v4/table.(*Table).DecrRef(0xc67b5ca9c0)
/data/badger_broadband/badger_grpc/vendor/github.com/dgraph-io/badger/v4/table/table.go:169 +0x87
github.com/dgraph-io/badger/v4.decrRefs(...)
/data/badger_broadband/badger_grpc/vendor/github.com/dgraph-io/badger/v4/level_handler.go:178
github.com/dgraph-io/badger/v4.(*levelsController).compactBuildTables.func6()
/data/badger_broadband/badger_grpc/vendor/github.com/dgraph-io/badger/v4/levels.go:955 +0x56
github.com/dgraph-io/badger/v4.(*levelsController).runCompactDef.func1()
/data/badger_broadband/badger_grpc/vendor/github.com/dgraph-io/badger/v4/levels.go:1431 +0x25
panic({0xb11c60, 0x11587a0})
/usr/local/go/src/runtime/panic.go:884 +0x213
github.com/dgraph-io/badger/v4/fb.(*TableIndex).StaleDataSize(0x0)
/data/badger_broadband/badger_grpc/vendor/github.com/dgraph-io/badger/v4/fb/TableIndex.go:132 +0x1d
github.com/dgraph-io/badger/v4/table.(*Table).StaleDataSize(0x0?)
/data/badger_broadband/badger_grpc/vendor/github.com/dgraph-io/badger/v4/table/table.go:667 +0x1e
github.com/dgraph-io/badger/v4.(*levelHandler).addSize(...)
/data/badger_broadband/badger_grpc/vendor/github.com/dgraph-io/badger/v4/level_handler.go:214
github.com/dgraph-io/badger/v4.(*levelHandler).replaceTables(0xc242f49140, {0xc48d01a248, 0x1, 0x4208767?}, {0xc6fd7f5a98, 0x1, 0xc48d02b800?})
/data/badger_broadband/badger_grpc/vendor/github.com/dgraph-io/badger/v4/level_handler.go:137 +0x2b3
github.com/dgraph-io/badger/v4.(*levelsController).runCompactDef(_, _, _, {0xc48bea9250, 0x1, {0x4, {0xc48d02b7c0, 0x7, 0x7}, {0xc48d02b800, ...}}, ...})
/data/badger_broadband/badger_grpc/vendor/github.com/dgraph-io/badger/v4/levels.go:1460 +0x43b
github.com/dgraph-io/badger/v4.(*levelsController).doCompact(0xc116de2a80, 0x1, {0x5, 0x3ffd1ba93016f62f, 0x3ffd1ba93016f62f, {0x0, 0x0, 0x0}, {0x4, {0xc48d02b7c0, ...}, ...}})
/data/badger_broadband/badger_grpc/vendor/github.com/dgraph-io/badger/v4/levels.go:1544 +0x538
github.com/dgraph-io/badger/v4.(*levelsController).runCompactor.func2({0x5, 0x3ffd1ba93016f62f, 0x3ffd1ba93016f62f, {0x0, 0x0, 0x0}, {0x4, {0xc48d02b7c0, 0x7, 0x7}, ...}})
/data/badger_broadband/badger_grpc/vendor/github.com/dgraph-io/badger/v4/levels.go:465 +0x78
github.com/dgraph-io/badger/v4.(*levelsController).runCompactor.func3()
/data/badger_broadband/badger_grpc/vendor/github.com/dgraph-io/badger/v4/levels.go:488 +0x158
github.com/dgraph-io/badger/v4.(*levelsController).runCompactor(0xc116de2a80, 0x1, 0xc2026ffef0)
/data/badger_broadband/badger_grpc/vendor/github.com/dgraph-io/badger/v4/levels.go:517 +0x382
created by github.com/dgraph-io/badger/v4.(*levelsController).startCompact
/data/badger_broadband/badger_grpc/vendor/github.com/dgraph-io/badger/v4/levels.go:354 +0x4e