Badgerdb crashing on save

Ravindra_Sharma · June 16, 2022, 11:44am

What version of Go are you using (`go version`)?

$ go version 1.18.3

What operating system are you using?

Ubuntu

What version of Badger are you using?

v3 v3.2103.2

Does this issue reproduce with the latest master?

yes

Steps to Reproduce the issue

Following Code

func Save(db *badger.DB) fiber.Handler {
	return func(ctx *fiber.Ctx) error {
		// response end client receives
		response := client.Response{
			Success:   false,
			Key:       "",
			Value:     "",
			Message:   "",
			CreatedAt: "",
		}
		entry := new(client.Payload)
		if err := ctx.BodyParser(&entry); err != nil {
			response.Message = helpers.ErrorPrint(err.Error())
			return ctx.Status(400).JSON(response)
		}
		if !helpers.ValidateMinLength(entry.Key, 4) {
			response.Message = helpers.ErrorPrint(locale.LEN1001 + " min: 4")
			return ctx.Status(400).JSON(response)
		}
		if !helpers.ValidateMaxLength(entry.Key, 100) {
			response.Message = helpers.ErrorPrint(locale.LEN1001 + " max: 64")
			return ctx.Status(400).JSON(response)
		}
		if !helpers.ValidateMinLength(entry.Value, 2) {
			response.Message = helpers.ErrorPrint(locale.LEN1001 + " min: 2")
			return ctx.Status(400).JSON(response)
		}
		createdAt := time.Now().Format("2006-01-02 15:04:05")
		err := db.Update(func(txn *badger.Txn) error {
			if err := txn.Set([]byte(entry.Key), []byte(entry.Value)); err == badger.ErrTxnTooBig {
				_ = txn.Commit()
				txn = db.NewTransaction(true)
				_ = txn.Set([]byte(entry.Key), []byte(entry.Value))
			}
			_ = txn.Set([]byte(entry.Key+"-date"), []byte(createdAt))
			return nil
		})

		if err != nil {
			response.Message = helpers.ErrorPrint(err.Error())
			return ctx.Status(400).JSON(response)
		}
		response.Success = true
		response.Message = "done"
		response.Key = entry.Key
		response.Value = entry.Value
		response.CreatedAt = createdAt
		return ctx.Status(200).JSON(response)
	}

}

What Badger options were set?

options := badger.DefaultOptions("")
options.NumVersionsToKeep = 1
options.ValueDir = /root/playingdb/db/
options.Dir = /root/playingdb/db/
options.InMemory = false
options.BypassLockGuard = true

What did you do?

Saving the data in badger it is not working

What did you expect to see?

successfully saved

What did you see instead?

Crash

unexpected fault address 0xad2328
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x2 addr=0xad2328 pc=0x411283]

goroutine 12 [running]:
runtime.throw({0xa8d679?, 0x0?})
        /snap/go/9848/src/runtime/panic.go:992 +0x71 fp=0xc000ac6430 sp=0xc000ac6400 pc=0x436431
runtime.sigpanic()
        /snap/go/9848/src/runtime/signal_unix.go:825 +0x305 fp=0xc000ac6480 sp=0xc000ac6430 pc=0x44baa5
runtime.mapiterinit(0x0?, 0x0?, 0x0?)
        /snap/go/9848/src/runtime/map.go:821 +0x23 fp=0xc000ac64a0 sp=0xc000ac6480 pc=0x411283
reflect.mapiterinit(0x0?, 0x0?, 0x0?)
        /snap/go/9848/src/runtime/map.go:1373 +0x19 fp=0xc000ac64c8 sp=0xc000ac64a0 pc=0x460399
github.com/gofiber/fiber/v2/internal/go-json/encoder/vm.Run(0xc0006ce9c0, {0xc0000be800?, 0xc0006e9780?, 0x0?}, 0x0?)
        /root/playingDB/vendor/github.com/gofiber/fiber/v2/internal/go-json/encoder/vm/vm.go:425 +0x1fb7 fp=0xc000ac9938 sp=0xc000ac64c8 pc=0x889d37
github.com/gofiber/fiber/v2/internal/go-json.encodeRunCode(0xa10e80?, {0xc0000be800?, 0xfc19e8?, 0xc001563978?}, 0x1?)
        /root/playingDB/vendor/github.com/gofiber/fiber/v2/internal/go-json/encode.go:307 +0x68 fp=0xc000ac9970 sp=0xc000ac9938 pc=0x8b8dc8
github.com/gofiber/fiber/v2/internal/go-json.encode(0xc0006ce9c0, {0xa10e80, 0xc000590180})
        /root/playingDB/vendor/github.com/gofiber/fiber/v2/internal/go-json/encode.go:232 +0x21c fp=0xc000ac99f0 sp=0xc000ac9970 pc=0x8b8cbc
github.com/gofiber/fiber/v2/internal/go-json.marshal({0xa10e80, 0xc000590180}, {0x0, 0x0, 0x414a79?})
        /root/playingDB/vendor/github.com/gofiber/fiber/v2/internal/go-json/encode.go:147 +0xba fp=0xc000ac9a50 sp=0xc000ac99f0 pc=0x8b897a
github.com/gofiber/fiber/v2/internal/go-json.MarshalWithOption(...)
        /root/playingDB/vendor/github.com/gofiber/fiber/v2/internal/go-json/json.go:186
github.com/gofiber/fiber/v2/internal/go-json.Marshal({0xa10e80?, 0xc000590180?})
        /root/playingDB/vendor/github.com/gofiber/fiber/v2/internal/go-json/json.go:171 +0x2a fp=0xc000ac9a88 sp=0xc000ac9a50 pc=0x8b8e4a
github.com/gofiber/fiber/v2.(*Ctx).JSON(0xc0000e82c0, {0xa10e80?, 0xc000590180?})
        /root/playingDB/vendor/github.com/gofiber/fiber/v2/ctx.go:651 +0x3c fp=0xc000ac9ae0 sp=0xc000ac9a88 pc=0x99489c
main.main.func1(0xc0000e82c0, {0xb8b200?, 0xc0000a0b70?})
        /root/playingDB/main.go:47 +0x10b fp=0xc000ac9b18 sp=0xc000ac9ae0 pc=0x9a726b
github.com/gofiber/fiber/v2.(*App).ErrorHandler(0xc0001221a0, 0xc0000e82c0, {0xb8b200, 0xc0000a0b70})
        /root/playingDB/vendor/github.com/gofiber/fiber/v2/app.go:905 +0x1d3 fp=0xc000ac9be0 sp=0xc000ac9b18 pc=0x98ed93
github.com/gofiber/fiber/v2.(*App).handler(0xc0001221a0, 0x4b9a97?)
        /root/playingDB/vendor/github.com/gofiber/fiber/v2/router.go:157 +0x105 fp=0xc000ac9c20 sp=0xc000ac9be0 pc=0x99ee05
github.com/gofiber/fiber/v2.(*App).handler-fm(0xc0006d6000?)
        <autogenerated>:1 +0x2c fp=0xc000ac9c40 sp=0xc000ac9c20 pc=0x9a318c
github.com/valyala/fasthttp.(*Server).serveConn(0xc000318000, {0xb902e8?, 0xc0000a60d0})
        /root/playingDB/vendor/github.com/valyala/fasthttp/server.go:2278 +0x12e7 fp=0xc000ac9ec8 sp=0xc000ac9c40 pc=0x95be07
github.com/valyala/fasthttp.(*Server).serveConn-fm({0xb902e8?, 0xc0000a60d0?})
        <autogenerated>:1 +0x39 fp=0xc000ac9ef0 sp=0xc000ac9ec8 pc=0x96acd9
github.com/valyala/fasthttp.(*workerPool).workerFunc(0xc0000c6780, 0xc000068300)
        /root/playingDB/vendor/github.com/valyala/fasthttp/workerpool.go:223 +0xa9 fp=0xc000ac9fa0 sp=0xc000ac9ef0 pc=0x9673a9
github.com/valyala/fasthttp.(*workerPool).getCh.func1()
        /root/playingDB/vendor/github.com/valyala/fasthttp/workerpool.go:195 +0x38 fp=0xc000ac9fe0 sp=0xc000ac9fa0 pc=0x967138
runtime.goexit()
        /snap/go/9848/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000ac9fe8 sp=0xc000ac9fe0 pc=0x466021
created by github.com/valyala/fasthttp.(*workerPool).getCh
        /root/playingDB/vendor/github.com/valyala/fasthttp/workerpool.go:194 +0x1b0

goroutine 1 [IO wait, 2 minutes]:
internal/poll.runtime_pollWait(0x7f1a282811d8, 0x72)
        /snap/go/9848/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc000120580?, 0x30?, 0x0)
        /snap/go/9848/src/internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /snap/go/9848/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Accept(0xc000120580)
        /snap/go/9848/src/internal/poll/fd_unix.go:614 +0x22c
net.(*netFD).accept(0xc000120580)
        /snap/go/9848/src/net/fd_unix.go:172 +0x35
net.(*TCPListener).accept(0xc0000a0a50)
        /snap/go/9848/src/net/tcpsock_posix.go:139 +0x28
net.(*TCPListener).Accept(0xc0000a0a50)
        /snap/go/9848/src/net/tcpsock.go:288 +0x3d
github.com/valyala/fasthttp.acceptConn(0xc000318000, {0xb8e228, 0xc0000a0a50}, 0xc00155ba88)
        /root/playingDB/vendor/github.com/valyala/fasthttp/server.go:1849 +0x62
github.com/valyala/fasthttp.(*Server).Serve(0xc000318000, {0xb8e228?, 0xc0000a0a50})
        /root/playingDB/vendor/github.com/valyala/fasthttp/server.go:1764 +0x4ca
github.com/gofiber/fiber/v2.(*App).Listen(0xc0001221a0, {0xc000599d83?, 0xc000032088?})
        /root/playingDB/vendor/github.com/gofiber/fiber/v2/app.go:690 +0xbd
main.main()
        /root/playingDB/main.go:142 +0x3ae

goroutine 19 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x0?)
        /root/playingDB/vendor/github.com/golang/glog/glog.go:882 +0x6a
created by github.com/golang/glog.init.0
        /root/playingDB/vendor/github.com/golang/glog/glog.go:410 +0x1bf

goroutine 20 [select, 2 minutes]:
github.com/dgraph-io/badger/v3/y.(*WaterMark).process(0xc0001d4ab0, 0xc0001d4a80)
        /root/playingDB/vendor/github.com/dgraph-io/badger/v3/y/watermark.go:214 +0x285
created by github.com/dgraph-io/badger/v3/y.(*WaterMark).Init
        /root/playingDB/vendor/github.com/dgraph-io/badger/v3/y/watermark.go:72 +0xaa

goroutine 21 [select, 2 minutes]:
github.com/dgraph-io/badger/v3/y.(*WaterMark).process(0xc0001d4ae0, 0xc0001d4a80)
        /root/playingDB/vendor/github.com/dgraph-io/badger/v3/y/watermark.go:214 +0x285
created by github.com/dgraph-io/badger/v3/y.(*WaterMark).Init
        /root/playingDB/vendor/github.com/dgraph-io/badger/v3/y/watermark.go:72 +0xaa

goroutine 22 [select]:
github.com/dgraph-io/ristretto/z.(*AllocatorPool).freeupAllocators(0xc0000a0f60)
        /root/playingDB/vendor/github.com/dgraph-io/ristretto/z/allocator.go:385 +0x150
created by github.com/dgraph-io/ristretto/z.NewAllocatorPool
        /root/playingDB/vendor/github.com/dgraph-io/ristretto/z/allocator.go:324 +0xc5

goroutine 6 [select, 2 minutes]:
github.com/dgraph-io/ristretto.(*defaultPolicy).processItems(0xc000072040)
        /root/playingDB/vendor/github.com/dgraph-io/ristretto/policy.go:102 +0x91
created by github.com/dgraph-io/ristretto.newDefaultPolicy
        /root/playingDB/vendor/github.com/dgraph-io/ristretto/policy.go:86 +0x156

goroutine 7 [select]:
github.com/dgraph-io/ristretto.(*Cache).processItems(0xc000120000)
        /root/playingDB/vendor/github.com/dgraph-io/ristretto/cache.go:452 +0x15e
created by github.com/dgraph-io/ristretto.NewCache
        /root/playingDB/vendor/github.com/dgraph-io/ristretto/cache.go:207 +0x696

goroutine 8 [select, 2 minutes]:
github.com/dgraph-io/badger/v3.(*DB).monitorCache(0xc0000bb200, 0xc0001d4870)
        /root/playingDB/vendor/github.com/dgraph-io/badger/v3/db.go:469 +0x18a
created by github.com/dgraph-io/badger/v3.Open
        /root/playingDB/vendor/github.com/dgraph-io/badger/v3/db.go:311 +0xc8b

goroutine 9 [select, 2 minutes]:
github.com/dgraph-io/badger/v3.(*DB).updateSize(0xc0000bb200, 0xc0001d4cf0)
        /root/playingDB/vendor/github.com/dgraph-io/badger/v3/db.go:1171 +0x158
created by github.com/dgraph-io/badger/v3.Open
        /root/playingDB/vendor/github.com/dgraph-io/badger/v3/db.go:331 +0xe8c

goroutine 26 [select]:
github.com/dgraph-io/badger/v3.(*levelsController).runCompactor(0xc00010e000, 0x0, 0xc0001d4d50)
        /root/playingDB/vendor/github.com/dgraph-io/badger/v3/levels.go:508 +0x30d
created by github.com/dgraph-io/badger/v3.(*levelsController).startCompact
        /root/playingDB/vendor/github.com/dgraph-io/badger/v3/levels.go:354 +0x4e

goroutine 27 [select]:
github.com/dgraph-io/badger/v3.(*levelsController).runCompactor(0xc00010e000, 0x1, 0xc0001d4d50)
        /root/playingDB/vendor/github.com/dgraph-io/badger/v3/levels.go:508 +0x30d
created by github.com/dgraph-io/badger/v3.(*levelsController).startCompact
        /root/playingDB/vendor/github.com/dgraph-io/badger/v3/levels.go:354 +0x4e

goroutine 28 [select]:
github.com/dgraph-io/badger/v3.(*levelsController).runCompactor(0xc00010e000, 0x2, 0xc0001d4d50)
        /root/playingDB/vendor/github.com/dgraph-io/badger/v3/levels.go:508 +0x30d
created by github.com/dgraph-io/badger/v3.(*levelsController).startCompact
        /root/playingDB/vendor/github.com/dgraph-io/badger/v3/levels.go:354 +0x4e

goroutine 29 [select]:
github.com/dgraph-io/badger/v3.(*levelsController).runCompactor(0xc00010e000, 0x3, 0xc0001d4d50)
        /root/playingDB/vendor/github.com/dgraph-io/badger/v3/levels.go:508 +0x30d
created by github.com/dgraph-io/badger/v3.(*levelsController).startCompact
        /root/playingDB/vendor/github.com/dgraph-io/badger/v3/levels.go:354 +0x4e

goroutine 30 [chan receive, 2 minutes]:
github.com/dgraph-io/badger/v3.(*DB).flushMemtable(0xc0000bb200, 0x0?)
        /root/playingDB/vendor/github.com/dgraph-io/badger/v3/db.go:1078 +0xb2
github.com/dgraph-io/badger/v3.Open.func5()
        /root/playingDB/vendor/github.com/dgraph-io/badger/v3/db.go:357 +0x25
created by github.com/dgraph-io/badger/v3.Open
        /root/playingDB/vendor/github.com/dgraph-io/badger/v3/db.go:356 +0x107c

goroutine 31 [select, 2 minutes]:
github.com/dgraph-io/badger/v3.(*vlogThreshold).listenForValueThresholdUpdate(0xc0000c88c0)
        /root/playingDB/vendor/github.com/dgraph-io/badger/v3/value.go:1172 +0x11a
created by github.com/dgraph-io/badger/v3.Open
        /root/playingDB/vendor/github.com/dgraph-io/badger/v3/db.go:380 +0x16dc

goroutine 32 [select, 2 minutes]:
github.com/dgraph-io/badger/v3.(*DB).doWrites(0xc0000bb200, 0xc0001d4ea0)
        /root/playingDB/vendor/github.com/dgraph-io/badger/v3/db.go:900 +0x236
created by github.com/dgraph-io/badger/v3.Open
        /root/playingDB/vendor/github.com/dgraph-io/badger/v3/db.go:387 +0x179c

goroutine 33 [chan receive, 2 minutes]:
github.com/dgraph-io/badger/v3.(*valueLog).waitOnGC(0xc0000bb3f8, 0x0?)
        /root/playingDB/vendor/github.com/dgraph-io/badger/v3/value.go:1079 +0x7d
created by github.com/dgraph-io/badger/v3.Open
        /root/playingDB/vendor/github.com/dgraph-io/badger/v3/db.go:391 +0x185b

goroutine 34 [select, 2 minutes]:
github.com/dgraph-io/badger/v3.(*publisher).listenForUpdates(0xc0001d4ba0, 0xc0001d4f00)
        /root/playingDB/vendor/github.com/dgraph-io/badger/v3/publisher.go:73 +0x150
created by github.com/dgraph-io/badger/v3.Open
        /root/playingDB/vendor/github.com/dgraph-io/badger/v3/db.go:395 +0x190a

goroutine 35 [chan receive, 2 minutes]:
main.main.func5()
        /root/playingDB/main.go:118 +0x94
created by main.main
        /root/playingDB/main.go:113 +0x365

goroutine 11 [syscall, 2 minutes]:
os/signal.signal_recv()
        /snap/go/9848/src/runtime/sigqueue.go:151 +0x2f
os/signal.loop()
        /snap/go/9848/src/os/signal/signal_unix.go:23 +0x19
created by os/signal.Notify.func1.1
        /snap/go/9848/src/os/signal/signal.go:151 +0x2a

goroutine 36 [sleep]:
time.Sleep(0x2540be400)
        /snap/go/9848/src/runtime/time.go:194 +0x12e
github.com/valyala/fasthttp.(*workerPool).Start.func2()
        /root/playingDB/vendor/github.com/valyala/fasthttp/workerpool.go:66 +0x56
created by github.com/valyala/fasthttp.(*workerPool).Start
        /root/playingDB/vendor/github.com/valyala/fasthttp/workerpool.go:58 +0xd9

goroutine 13 [sleep]:
time.Sleep(0x3b9aca00)
        /snap/go/9848/src/runtime/time.go:194 +0x12e
github.com/valyala/fasthttp.updateServerDate.func1()
        /root/playingDB/vendor/github.com/valyala/fasthttp/header.go:1590 +0x1e
created by github.com/valyala/fasthttp.updateServerDate
        /root/playingDB/vendor/github.com/valyala/fasthttp/header.go:1588 +0x25
exit status 2

Ravindra_Sharma · June 17, 2022, 9:14am

bump ~ This started happening after having 100K get requests in 1 min. All “set” failed on DB

Ravindra_Sharma · June 17, 2022, 3:28pm

@mrjn @pawan - Please share any input on why BadgerDB is not storing any data?

Ravindra_Sharma · June 17, 2022, 3:29pm

@MichelDiz @dmai - Please share any input on why BadgerDB is not storing any data?

vnium · June 17, 2022, 7:18pm

From the logs it looks like the segmentation fault is originating from gojson.

amaster507 · June 18, 2022, 3:51am

None of those people work at Dgraph anymore. Try with @akon but I don’t think anyone is really supporting badger at all at the moment.

Ravindra_Sharma · June 18, 2022, 4:45am

is badgerdb deprecated now?

jarifibrahim · July 7, 2022, 11:18am

@Ravindra_Sharma you might want to try printing the error from set/commit function?
From the stack trace it doesn’t look like badger crashed. The segmentation fault points to some JSON code.

Topic		Replies	Views
[badger v3] updateDiscardStats unexpected fault address Badger kind:bug	3	620	September 28, 2021
Panic when concurrently closing DB while some goroutines doing reads Badger status:accepted , kind:bug , ticket:created	0	627	May 20, 2021
Badger crashes on full disk Badger	4	919	October 6, 2022
Panics when opening a corrupted Badger DB Badger	8	1776	January 18, 2023
Reusing Byte Slice in Transaction causes Bugs with Subscriptions Badger kind:bug	1	504	July 15, 2022