1.6.0 perf regression compared to 1.5.3?

Moved from GitHub badger/939

Posted by burmanm:

What version of Go are you using (go version)?

$ go version
go version go1.12.6 linux/amd64

What version of Badger are you using?

1.5.3 and 1.6.0

Does this issue reproduce with the latest master?

What are the hardware specifications of the machine (RAM, OS, Disk)?

Haswell & Sandy Bridge, both tested. Linux 5.1.11-200.fc29 and 5.1.15-1-MANJARO. 12GB / 16GB machines, SSDs in both.

What did you do?

Port Jaeger from 1.5.3 to 1.6.0 and ran benchmark.

What did you expect to see?

Identical performance.

What did you see instead?

Performance was 50% worse with 1.6.0 compared to 1.5.3.

The code writes multiple spans in a single Update transaction:

entriesToStore := make([]*badger.Entry, 0, len(span.Tags)+4+len(span.Process.Tags)+len(span.Logs)*4)

// Removed the code that makes those entries

	err = w.store.Update(func(txn *badger.Txn) error {
		// Write the entries
		for i := range entriesToStore {
			err = txn.SetEntry(entriesToStore[i])
			if err != nil {
				// Most likely primary key conflict, but let the caller check this
				return err
		return nil

The database is initialized as follows (1.6.0 version):

dir, _ := ioutil.TempDir("", "badger")
opts := badger.DefaultOptions(dir).WithValueDir(dir).WithSyncWrites(false)

What I’m seeing with 1.5.3 is 100k writes (100k transactions, not 100k keys) done in 10 seconds and with 1.6.0 it takes 15 seconds. I’m not entirely sure what’s the cause here, there’s no single reason in the profile outputs (multiple things are slower, such as Txn.Commit). Both versions use the same amount of time in the syscalls, so I can’t see any difference in the disk section either.

I assume the point of those newer faster APIs wasn’t to require using them to get same performance as older versions?

campoy commented :

Thanks for the report, @burmanm!

I know @martinmr ran some benchmarks - I wonder if we can reproduce this regression too.
Can you give it a try, Martin?