Fix badger builds on 32 bit machines

Issues Badger
, , , ,
1

Moved from GitHub badger/1397

Posted by jarifibrahim:

Badger tests fail reliably on 32 bit machines and they need to be fixed.
The builds were enabled in Enable cross-compiled 32bit tests on TravisCI by slyon · Pull Request #1392 · dgraph-io/badger · GitHub but the build is failing currently.

Most of the failures have the following pattern

=== RUN   TestRotate
badger 2020/06/30 16:00:14 INFO: All 0 tables opened in 0s
badger 2020/06/30 16:00:14 INFO: Got compaction priority: {level:0 score:1.73 dropPrefixes:[]}
badger 2020/06/30 16:00:14 INFO: All 0 tables opened in 0s
badger 2020/06/30 16:00:14 INFO: Replaying file id: 0 at offset: 0
badger 2020/06/30 16:00:14 INFO: Replay took: 2.962µs
badger 2020/06/30 16:00:14 INFO: Got compaction priority: {level:0 score:1.73 dropPrefixes:[]}
badger 2020/06/30 16:00:14 INFO: All 0 tables opened in 0s
badger 2020/06/30 16:00:14 INFO: Replaying file id: 0 at offset: 0
badger 2020/06/30 16:00:14 INFO: Replay took: 2.936µs
badger 2020/06/30 16:00:14 INFO: Got compaction priority: {level:0 score:1.73 dropPrefixes:[]}
badger 2020/06/30 16:00:14 INFO: All 0 tables opened in 0s
badger 2020/06/30 16:00:14 INFO: Replaying file id: 0 at offset: 0
badger 2020/06/30 16:00:14 INFO: Replay took: 3.116µs
    TestRotate: rotate_test.go:91: 
        	Error Trace:	rotate_test.go:91
        	Error:      	Received unexpected error:
        	            	Map log file. Path=/tmp/badger-test181776523/000001.vlog. Error=cannot allocate memory
        	            	During db.vlog.open
        	            	github.com/dgraph-io/badger/v2/y.Wrapf
        	            		/home/travis/gopath/src/github.com/slyon/badger/y/error.go:82
        	            	github.com/dgraph-io/badger/v2.Open
        	            		/home/travis/gopath/src/github.com/slyon/badger/db.go:390
        	            	github.com/dgraph-io/badger/v2/badger/cmd.TestRotate
        	            		/home/travis/gopath/src/github.com/slyon/badger/badger/cmd/rotate_test.go:90
        	            	testing.tRunner
        	            		/home/travis/.gimme/versions/go1.14.linux.amd64/src/testing/testing.go:992
        	            	runtime.goexit
        	            		/home/travis/.gimme/versions/go1.14.linux.amd64/src/runtime/asm_386.s:1337
        	Test:       	TestRotate
--- FAIL: TestRotate (0.13s)

See Travis CI - Test and Deploy Your Code with Confidence for failed 32 bit build.

2

jarifibrahim commented :

Related to Test suite failure on 32 bit architectures, like armhf or x86 · Issue #1384 · dgraph-io/badger · GitHub

3

hiepd commented :

Would like to look into this issue if no one is looking at it. Thanks!

4

jarifibrahim commented :

@hiepd feel free to pick it up.

5

Wanted to point out this discussion that may offer some insight into fixing the problem:
Possible 32bit test fix
Reposted here:

I have looked at the issue, and have been able to get tests to pass by requesting smaller amounts of memory from mmap when running 32bit tests. This can be done conditionally compiling a value for the log file size:

// +build 368

package badger

// Log file size when running tests on 32bit system.
const testOptionLogFileSize = 1<<26 - 1

Note: !386 file not shown here, but sets value to 1<<30 - 1

Then, set this as a test option:

func getTestOptions(dir string) Options {
	opt := DefaultOptions(dir).
		WithMaxTableSize(1 << 15). // Force more compaction.
		WithLevelOneSize(4 << 15). // Force more compaction.
		WithSyncWrites(false)
	if !*mmap {
		return opt.WithValueLogLoadingMode(options.FileIO)
	}
	return opt.WithValueLogFileSize(testOptionLogFileSize)
}

There are a number of other places where tests use DefaultOptions instead of test options, and the option will need to be set in those places as well.

This change allows all the tests to pass that were previously failing, due to mmap not being able to allocate the requested memory. However, results varied per test system, and the value may need to be adjusted.

Of course, disabling mmap (use options.FileIO instead) for 32bit tests also avoids this problem, but then then mmap + 32bit cannot really be supported is it is not being tested.