Bad Magic - unable to open database

Users Dgraph
1

Hi

I’m getting the following when I try to open a database.

This is a new database, it’s within Docker on a Mac - it works for a few hours, then stops responding. When I stop and restart, the database won’t open. Is there any way to recover the data?

Dgraph version   : v20.07.0
Dgraph codename  : shuri
Dgraph SHA-256   : 4cd320fc6eab163ab68602a5122a6c82c8467c2ed5ac93478d5f40d44eec71c4
Commit SHA-1     : d65e20530
Commit timestamp : 2020-07-28 15:31:37 -0700
Branch           : HEAD
Go version       : go1.14.4

For Dgraph official documentation, visit https://dgraph.io/docs/.
For discussions about Dgraph     , visit http://discuss.dgraph.io.
To say hi to the community       , visit https://dgraph.slack.com.

Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2020 Dgraph Labs, Inc.


I1210 17:08:26.038488      42 run.go:108] Setting up grpc listener at: 0.0.0.0:5080
I1210 17:08:26.038809      42 run.go:108] Setting up http listener at: 0.0.0.0:6080
[Sentry] 2020/12/10 17:08:26 Sending fatal event [303fdca24312455aac5d77c21fe85101] to o318308.ingest.sentry.io project: 1805390
2020/12/10 17:08:26 manifest has bad magic
github.com/dgraph-io/badger/v2.init
        /go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200718033852-37ee16d8ad1c/manifest.go:339
runtime.doInit
        /usr/local/go/src/runtime/proc.go:5420
runtime.doInit
        /usr/local/go/src/runtime/proc.go:5415
runtime.doInit
        /usr/local/go/src/runtime/proc.go:5415
runtime.doInit
        /usr/local/go/src/runtime/proc.go:5415
runtime.doInit
        /usr/local/go/src/runtime/proc.go:5415
runtime.doInit
        /usr/local/go/src/runtime/proc.go:5415
runtime.main
        /usr/local/go/src/runtime/proc.go:190
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1373
Error while creating badger KV WAL store
github.com/dgraph-io/dgraph/x.Checkf
        /ext-go/1/src/github.com/dgraph-io/dgraph/x/error.go:51
github.com/dgraph-io/dgraph/worker.(*ServerState).initStorage
        /ext-go/1/src/github.com/dgraph-io/dgraph/worker/server_state.go:148
github.com/dgraph-io/dgraph/worker.InitServerState
        /ext-go/1/src/github.com/dgraph-io/dgraph/worker/server_state.go:54
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.run
        /ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:658
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.init.2.func1
        /ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:95
github.com/spf13/cobra.(*Command).execute
        /go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
        /go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
        /go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
        /ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:70
main.main
        /ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/main.go:78
runtime.main
        /usr/local/go/src/runtime/proc.go:203
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1373
2

Can you share the YAML?

3

Simply run from the command-line (it’s a training instance).

docker run --name dgraph --rm -it -p 8080:8080 -p 9080:9080 -p 8000:8000 -v $(pwd)/dgraph:/dgraph dgraph/standalone:latest

Docker version: 19.03.13
MacOS: 10.14.6, MacPro 32GiB RAM

4

This is coming from the OSX filesystem or are you using another?

I also use macOS daily, and I never saw this before. Understand why this happens is crucial. Are you able to reproduce it at any time? Would you mind to share in DM all the steps to reproduce it?

I think so, the debug tool you can retrieve keys and several other things. But I never try recovering anything before. Maybe you can do a copy of the p, zw, w files. And edit or delete the manifest and try again. But not sure if that works for real.

@dmai can you answer that?

5

It’s running on OSX. It’s really weird: I’m writing an in-house training course for Dgraph and used the REACT Todo subscription model as one of the examples.

The database has only the Todo schema and perhaps a couple of records (added via Insomnia mutation post) in it. I went to write up a few steps, came back and the database was unresponsive. CTRL-C and restart and the error appeared as above. I then deleted the directory and started again - it ran; I left it for a couple of hours, and it was dead again - same error. The third time, I posted the above.

The drive has ample space, it’s running standalone, and there’s nothing weird running on the machine as far as I can tell.

I would welcome the opportunity to recover the database rather than kill it and recreate as I can include this in our emergency recovery procedures for the enterprise servers. What’s the debug tool? I tried temporarily renaming the MANIFEST to MANIFEST.old and restarted - it wouldn’t start.

1 Like
6

Thanks for the answer.

Create a new dgraph instance and copy its Manifest to that one you wanna recover.

Here Using the Debug Tool - Howto

7

Thanks - the MANIFEST trick allowed me to get it started again. I’ll leave it running but doing nothing for the next couple of hours and see if it dies again.

Not sure how I missed the debug tool … I’ll have a look at that too,

8

Nice!
After doing the trick, I would export my data and start from scratch.