Protocol Buffers are Great, Except…

2025/12/26

Intro

Serialization is one of those things that’s really tempting to build - it’s a constrained problem, has a simple abstraction with lots of depth behind it, is domain-specific so general-purpose solutions leave something to be desired, is relatively easy to test, and can be built and productionized within a reasonable amount of time by a solo engineer. As evidence, I can recall at least 10 different popular serialization formats off the top of my head: Protocol Buffers, JSON, BSON, MsgPack, Cap’n Proto, FlatBuffers, Simple Binary Encoding (SBE), cereal, Avro, Thrift, Arrow Flight, etc. And I’m sure this is missing a large number of alternatives.

In this post, I’d like to talk about serialization & schema design under a specific but rather widely applicable set of constraints:

That’s it. This is pretty much the basic requirements for communication between micro-services (whether through direct RPC or a message broker). It can also be appropriate for single-host multi-process communication, or logging structured events, or frontend-backend communication, etc.

The Landscape

Despite the common enough requirements, I don’t think there is a satisfactory, general-purpose solution to this problem (yet).

Let’s look at a few popular options and their problems:

What We Want

Going back to our original requirements, what do we really want in a serialization format, from an application’s point of view?

In terms of a balance of data size / (de)serialization speed / schema evolution support, I believe Protocol Buffer Encoding strikes quite a good balance here. It’s got some lightweight compression via varints, great schema evolution support, encoding/decoding can be quite fast, and finally has a large community. If we consider only the wire format, it’s difficult to beat it on all fronts. Optimizing for speed on top of Protocol Buffers Encoding usually means sacrificing data size & schema evolution a little. Optimizing for data size means more compression, which sacrifices speed.

Next let’s talk about what an application really wants from Protocol Buffers but doesn’t get automatically - fast (de)serialization, and ease of use.

A typical application trying to send/receive data will likely have native in-memory types for that data defined (let’s not worry about pure data forwarders here). With stock Protocol Buffers, the app has to maintain translation logic between the native types and the protobuf generated types, or be forced to use the generated types for business logic - neither of which is great. Instead, I’d like the Protocol Buffers schema and native in-memory type to be seamlessly integrated. For example if I have (examples in C++ as that’s what I work with mostly):

struct Order {
  double price;
  int qty;
  string order_id;
};

and then the following Protocol Buffers schema:

message Order {
  double price = 1;
  int32 qty = 2;
  string order_id = 3;
}

I’d like the C++ type and the protobuf wire format to “map” to each other. I’d like to be able to write:

Order order1;
Serialize(order1, buffer, size);

Order order2;
Deserialize(buffer, size, &order2);

without having to implement or maintain Serialize/Deserialize myself, or to update them when schemas change. Achieving this will require a lot of reflection - possibly in both the native language and the schema IDL. Protocol Buffers already provides excellent reflection, and popular languages either support reflection natively, or there’s usually some way to get around the lack of reflection to achieve it anyway.

Additionally, Serialize/Deserialize should be zero-allocation and minimal-copy, moving straight from native type to bytes, and vice versa. No intermediate generated types, and no memory allocation, hence the buffer+size API. This gets us close to Cap’n Proto level performance out of Protocol Buffers.

However, it’s unrealistic to always generate Serialize/Deserialize auto-magically when between native types and serialization schemas, because wire-format schemas are not expressive enough. Native in-memory types can be much richer - maps, sets, queues, “newtype” types, AoS vs SoA (array-of-structs vs struct-of-arrays), etc. Perhaps I’d like to translate the following schema:

message NamedPoints {
  repeated string names = 1;
  repeated double xs = 2;
  repeated double ys = 3;
}

to the following native type:

struct Point { int x; int y; };
DEFINE_NEW_TYPE(Name, std::string);
using NamedPoints = unordered_map<Name, Point>;

Trying to auto-generate Serialize/Deserialize directly becomes quite pointless in this case. Instead, a more realistic goal is to make it as easy as possible to implement a custom zero-allocation Serialize/Deserialize, based on auto-generated helpers from the message schema.

A Better Solution

Given the above, let’s consider what an ideal library for Protocol Buffers looks like. It should support:

I don’t think this library exists today. I expect there probably are already similar closed-source libraries out there, but none open-source that I know of. The one that comes closest to the above requirements I’ve seen is Pigweed Protobuf, but it is advertised as an embedded library and doesn’t seem intended for use in servers. I’m not sure why (except for lacking some minor feature support). It looks quite close to an ideal general purpose solution. Also it doesn’t support the visitor-style API mentioned above.

Caveats

I’ve glossed over some considerations to make some points more salient. Notably, serialization requirements differ a lot between problem domains, so the requirements mentioned above aren’t always applicable.

If your problem domain does not need much in terms of schema evolution, one can get true zero-copy deserialization in the vein of Cap’n Proto or something language-specific like rkyv, where the serialized format is also the in-memory format, even for complicated data structures like hash maps.

If you’d like to do selective reads over serialized data (e.g. via mmap) where the data being read is a small portion of the overall corpus, or if you’d like to selective in-place mutation of the data, then something like Cap’n Proto and FlatBuffers will be better. A downside of Protocol Buffers’ compact wire format is the lack of random access, which requires scanning the entire message even for a single field.

If you only work within a single language, then protocol buffers doesn’t buy much. You can use a language-specific serialization implementation which will always be more ergonomic than language-agnostic one.

Efficient Schema Design

Making schemas generate fast, easy-to-use code is only part of the story. It turns out that if you want good performance out of serialization, designing schemas approriately for efficiency is just as important.

As application programmers, our intuition may be to try as much as possible to have a one-to-one correspondence between our app’s native types and the wire format schema: an int for an int, an array for a repeated field, an optional for an optional, a hash map for a protobuf map, a sub-message for a protobuf sub-message, etc.

This intuition is not always the best use of protobuf schemas. While it may feel natural, ultimately native types and protobuf schemas are different things. Protobuf schemas are quite limited in their type selection and has its own quirks. For example:

I’m probably missing other gotchas. The overall point stands - designing protobuf schemas for performance is a very different game from designing schemas for native types.

My current advice when designing protobuf schemas is the following:

Conclusion?

You might expect this is where I will start shilling my own library, but there isn’t one! You’re welcome to build it if you agree this is a good idea. Or if you know a serialization format that readily solves all of the problems better than proposed, please do let me know.

Protocol Buffers are great. Let’s use them to their fullest potential.