Curious about the switch from Protobuf -> Flatbuffers

I saw this PR, and there didn’t seem to be any explanation there. I’m guessing something performance related? I just don’t think I’ve ever seen a project use both protobuf and flatbuffers at the same time, so it made me curious!

@ibrahim , I’d love to know that too.

Gorgonia does use both protobuf and flatbuffers. There are benefits to either.

As the maintainer of that, I can give you some reasons why my project uses both:

Flatbuffers allow for much much faster deserialization (basically none at all!) but PB are more compressed, allowing for more efficient over-the-wire communications. In the case of Gorgonia, over-the-wire comms are done with PB, but storage is done either in FB or gob (which is slow but hey the community seems to love that idea…)

1 Like

Protocol buffers take a lot of memory in Go. And we have to pay the cost of unmarshal. So we use FB in critical paths.

FB APIs suck though.


We use flatbuffers for table index which comes from a SST file that’s memory mapped. This saves us a lot of memory (no memory copy required). If we were to use protobuf for those indices, we need to make a copy of the entire index into the protobuf.