Badger error after upgrading from 0.8.1 to 0.8.2


(Jeff Hull) #1

UPDATE3: I reverted to 0.8.1 for now by manually extracting the contents of https://github.com/dgraph-io/dgraph/releases/download/v0.8.1/dgraph-linux-amd64-v0.8.1.tar.gz to /usr/local/bin

UPDATE2: is there a way to tell the canned install script to install v0.8.1 instead of latest? I tried the nightly dev build also but no luck there.

UPDATE: I tried wiping out the WSL instance by running lxrun /uninstall /full" and reinstalling, but I still get the same error.

Hi All,

After re-running “curl https://get.dgraph.io -sSf | bash” to upgrade dgraph (with super user permissions), I get the below error, “Error while creating badger KV WAL store: exec format error”… I’m running on WSL bash. Any ideas?

…~$ sudo dgraph --memory_mb 2048

Dgraph version : v0.8.2
Commit SHA-1 : 4269f91f
Commit timestamp : 2017-09-27 14:49:31 +1000
Branch : HEAD

2017/10/03 15:16:42 Error while creating badger KV WAL store error: exec format error
Unable to mmap RDWR log file
github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.(*valueLog).openOrCreateFiles
/home/travis/gopath/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/value.go:576
github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.(*valueLog).Open
/home/travis/gopath/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/value.go:626
github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.NewKV
/home/travis/gopath/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/kv.go:175
github.com/dgraph-io/dgraph/dgraph.(*ServerState).initStorage
/home/travis/gopath/src/github.com/dgraph-io/dgraph/dgraph/server.go:70
github.com/dgraph-io/dgraph/dgraph.NewServerState
/home/travis/gopath/src/github.com/dgraph-io/dgraph/dgraph/server.go:55
main.main
/home/travis/gopath/src/github.com/dgraph-io/dgraph/cmd/dgraph/main.go:758
runtime.main
/home/travis/.gimme/versions/go1.9.linux.amd64/src/runtime/proc.go:185
runtime.goexit
/home/travis/.gimme/versions/go1.9.linux.amd64/src/runtime/asm_amd64.s:2337
Unable to open value log
github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.(*valueLog).Open
/home/travis/gopath/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/value.go:627
github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.NewKV
/home/travis/gopath/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/kv.go:175
github.com/dgraph-io/dgraph/dgraph.(*ServerState).initStorage
/home/travis/gopath/src/github.com/dgraph-io/dgraph/dgraph/server.go:70
github.com/dgraph-io/dgraph/dgraph.NewServerState
/home/travis/gopath/src/github.com/dgraph-io/dgraph/dgraph/server.go:55
main.main
/home/travis/gopath/src/github.com/dgraph-io/dgraph/cmd/dgraph/main.go:758
runtime.main
/home/travis/.gimme/versions/go1.9.linux.amd64/src/runtime/proc.go:185
runtime.goexit
/home/travis/.gimme/versions/go1.9.linux.amd64/src/runtime/asm_amd64.s:2337


(Pawan Rawal) #2

Ah, this is related to the mmaping issue on Windows in Badger. This was fixed and the next release which should happen this week should have this. v0.8.1 still supports variables in mutations. So you could try using that until we do the next release.


(Jeff Hull) #3

Hey @pawan, just installed v0.8.3, but still getting a badger error (below). Is this fix supposed to be in 0.8.3?


(Pawan Rawal) #4

Hey @tamethecomplex

If you are upgrading from v0.8.1 then you would have to export and reimport the data into v0.8.3 because the data format has changed on disk.


(Jeff Hull) #5

Thanks pawan, forgot that I was running in a directory that already had data. Tried creating a new directory but got the original error:


(Pawan Rawal) #6

Oh, that’s strange. @deepak wasn’t this fixed in badger?


(Deepak Jois) #7

@tamethecomplex Can you confirm that you are running Badger inside WSL (Windows Subsystem for Linux) ?

Reading through the comments, it appears the problem is with mmap-ing inside WSL. I can describe what I guess is the problem in more detail, but I am afraid I can’t offer a good solution unless we spend some time investigating how mmap works inside WSL.

We made changes to switch completely to using mmap for value log in Badger. The mmap system call works differently under Windows, from Linux. So in Badger, we do some things slightly differently on Windows when we mmap a file. However, with WSL it gets a bit more complicated. Badger thinks it’s on Linux and tries to mmap accordingly. But the mmap call is probably being emulated in some way on top of Windows (because this is WSL). That seems to be the cause of this error. It is difficult to say more without digging deeper into WSL and running some basic mmap tests.

The best suggestion I can offer at the moment is to either use the Windows binary directly or switch to a Linux VM.


(Pawan Rawal) #8

Please try using this binary and let me know if it works. If it does, then I can update the release with binaries for Windows.


(Jeff Hull) #9

Hey @pawan, do you want me to wget this into WSL and run it there?

@deepak, yes correct, I’m running this in WSL in Windows 10.


(Jeff Hull) #10

Tried running in Windows Powershell (maybe doing this wrong because I have always run in WSL):

Tried downloading into WSL and running it there:


(Pawan Rawal) #11

Yeah, that’s what I meant. Looks like its working for you. Dgraphzero is a new binary which must be run from v0.8.3 before running Dgraph. It handles the dgraph cluster.

Here is the dgraphzero binary for Windows

You can find instructions to run dgraphzero and dgraph at https://docs.dgraph.io/get-started/. Let us know if you are able to get dgraph running successfully.


(Jeff Hull) #12

@pawan, I was able to get it up and running using the executables you provided:


(Jeff Hull) #13

Update on this. I’m able to run dgraph easily with this .bat script:

> start PowerShell.exe -NoExit -Command "cd C:/dgraph/dgraph83/zero;  rm w/LOCK; ./dgraphzero.exe"
> start PowerShell.exe -NoExit -Command "Start-Sleep -m 7000; cd C:/dgraph/dgraph83;  rm w/LOCK; rm p/LOCK; ./dgraph.exe --memory_mb 2048 --peer localhost:8888"

but for some reason, when I terminate dgraph and dgraphzero, the locks for each are not released. So I have to manually delete the locks (presumably dangerous) before starting up again.

I’m not able to use dgraph-live-loader from WSL because of the existing badger problem, but will report back with success of loading exported data into the new v0.8.3 once I have the v0.8.3 dgraph-live-loader Windows binary.


(Pawan Rawal) #14

Hey @tamethecomplex

The LOCK file issue seems to only be present for dgraphzero and not dgraph. I am pushing a fix for that. Also, I am automating uploading of binaries for Windows platform.

In the meantime, I have uploaded the binaries here https://github.com/dgraph-io/dgraph/releases/tag/v0.8.3.


(Jeff Hull) #15

Hey Pawan, I grabbed the windows .exe’s I see on the release page now, and dgraph and dgraphzero are both working as expected, except the lock files (for each) both still do not get deleted when the process is terminated. Not a huge deal for me personally as this is a development only environment (production env’t will be upgraded a little later and is Ubuntu EC2) and I can delete the files before running.

I was also able to get dgraph-live-loader to work with no problems.


(Pawan Rawal) #16

That’s great @tamethecomplex

How do you terminate the processes? The files should be deleted ideally if the process is terminated cleanly.


(Jeff Hull) #17

I was just manually closing the cmd.exe running each process…and until now I didn’t realize that this causes an “unclean” exit rather than terminating the job cleanly first. A “ctrl + c” apparently is a clean exit and resluts in deleted LOCK files. So I guess this can be chalked up to user error :slight_smile:


(Deepak Jois) #18

There is a WSL update in the Fall Creator Update where an mmap related bug has been fixed: https://github.com/Microsoft/BashOnWindows/issues/658

Maybe it would fix this problem with DGraph as well, but someone would need to try it out.


(Jeff Hull) #19

Thanks @deepak, Will grab the update when I can and report back with results.


(Jeff Hull) #20

@deepak, just tried this on a windows store installed version of Ubuntu on WSL / windows fall creators update, but unfortunately still get the same error:

jhull@####:~$ dgraphzero -w zw
Setting up listener at: localhost:8888
Setting up listener at: localhost:8889
2017/10/20 07:33:53 Error while opening WAL store error: exec format error
Unable to mmap RDWR log file
github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.NewKV
        /home/travis/gopath/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/kv.go:272
main.main
        /home/travis/gopath/src/github.com/dgraph-io/dgraph/cmd/dgraphzero/main.go:174
runtime.main
        /home/travis/.gimme/versions/go1.9.linux.amd64/src/runtime/proc.go:185
runtime.goexit
        /home/travis/.gimme/versions/go1.9.linux.amd64/src/runtime/asm_amd64.s:2337

NOTE: I’m in Windows 10 Pro version 1709, Build number 16299.19