Okay, so I did a couple more things. Tried with the GC inside the loop, but it just made all of them worse, and comparatively, they were the same… so ignore that for now.
The second thing I did was to also iterate over the resulting structure (nl in code), and make that part of the timed loop. This was interesting, because this made FB much worse.
$ go test --bench . --benchmem ~/go/src/github.com/dgraph-io/experiments/flats
Flatb k:10 sz:104
Fixed k:10 sz:82
Proto k:10 sz:100
Flatb k:100 sz:824
Fixed k:100 sz:803
Proto k:100 sz:999
Flatb k:1000 sz:8024
Fixed k:1000 sz:8003
Proto k:1000 sz:9987
Flatb k:10000 sz:80024
Fixed k:10000 sz:80004
Proto k:10000 sz:99936
Flatb k:100000 sz:800024
Fixed k:100000 sz:800004
Proto k:100000 sz:999216
Flatb k:1000000 sz:8000024
Fixed k:1000000 sz:8000005
Proto k:1000000 sz:9992219
Flatb k:10000000 sz:80000024
Fixed k:10000000 sz:80000005
Proto k:10000000 sz:99920961
BenchmarkToAndFrom/Flatb-10-4 2000000 850 ns/op 440 B/op 12 allocs/op
BenchmarkToAndFrom/Fixed-10-4 3000000 354 ns/op 424 B/op 7 allocs/op
BenchmarkToAndFrom/Proto-10-4 2000000 607 ns/op 440 B/op 7 allocs/op
BenchmarkToAndFrom/Flatb-100-4 300000 4289 ns/op 3128 B/op 18 allocs/op
BenchmarkToAndFrom/Fixed-100-4 1000000 1671 ns/op 3832 B/op 10 allocs/op
BenchmarkToAndFrom/Proto-100-4 300000 4304 ns/op 3960 B/op 10 allocs/op
BenchmarkToAndFrom/Flatb-1000-4 50000 36327 ns/op 26680 B/op 24 allocs/op
BenchmarkToAndFrom/Fixed-1000-4 100000 12521 ns/op 32760 B/op 13 allocs/op
BenchmarkToAndFrom/Proto-1000-4 50000 38696 ns/op 35064 B/op 13 allocs/op
BenchmarkToAndFrom/Flatb-10000-4 5000 357612 ns/op 452920 B/op 32 allocs/op
BenchmarkToAndFrom/Fixed-10000-4 10000 140472 ns/op 549624 B/op 22 allocs/op
BenchmarkToAndFrom/Proto-10000-4 5000 424528 ns/op 574200 B/op 22 allocs/op
BenchmarkToAndFrom/Flatb-100000-4 300 4071541 ns/op 3615040 B/op 38 allocs/op
BenchmarkToAndFrom/Fixed-100000-4 1000 1851157 ns/op 6259455 B/op 32 allocs/op
BenchmarkToAndFrom/Proto-100000-4 300 4243123 ns/op 6456063 B/op 32 allocs/op
BenchmarkToAndFrom/Flatb-1000000-4 50 31685823 ns/op 27765057 B/op 44 allocs/op
BenchmarkToAndFrom/Fixed-1000000-4 100 14294377 ns/op 61195010 B/op 42 allocs/op
BenchmarkToAndFrom/Proto-1000000-4 50 36688009 ns/op 63185665 B/op 42 allocs/op
BenchmarkToAndFrom/Flatb-10000000-4 5 324814643 ns/op 416909635 B/op 52 allocs/op
BenchmarkToAndFrom/Fixed-10000000-4 10 130693596 ns/op 583508736 B/op 52 allocs/op
BenchmarkToAndFrom/Proto-10000000-4 3 351247642 ns/op 603431680 B/op 52 allocs/op
PASS
ok github.com/dgraph-io/experiments/flats 41.644s
Given these results, it’s clear that PBs are better than FB. This result makes me very happy. We have been biting off the complicated API of Flatbuffers for a while, in the hope that it makes things faster for us - but given this benchmark, it’s clear that that’s not the case. The only place that FB helps is in terms of memory allocation per operation - in everything else PB-Fixed is a clear winner.
This is super exiciting, because this would make our code base a lot simpler, and allow mutability over the structures, which @jchiu has been looking for.