Protocol Buffers
Protocol Buffers (protobuf) is Google's language-neutral, schema-first binary serialization format, ubiquitous in service APIs and increasingly present in event streams.
LicenseOpen source (permissive)· BSD-3-Clause
DeploymentSelf-hostedManaged
PricingFree
Operational complexityLow
WorkloadInteractiveStreaming
LanguageC++, C#, C, Java, Objective-C
Use it when
- Events originate in gRPC-native services and topics should share one set of definitions with the APIs that produce them.
- Registry support must be universal: every Kafka registry accepts it, and Buf's registry is built for it alone.
- Wire stability over decades matters; the encoding has not changed in twenty years, and editions evolve the language without touching serialized bytes.
- Compact, typed messages with rolling-upgrade safety are the requirement; unknown fields survive a parse-and-reserialize cycle.
Think twice when
- Consumers should read data without a compiler in the build; code generation is mandatory, where Avro reads dynamically against a fetched schema.
- Messages grow large or need random access; fields carry no offsets, parsing starts at the front, and the project's own guidance caps messages around a megabyte.
- Schema governance is loose: the evolution rules (never reuse a field number, reserve removed ones) are enforced by review, not by the runtime.
- Registry-checked evolution is the priority; Avro's specified resolution algorithm is the more flexible model there.
How it runs
protoc plus per-language plugins generate typed classes for eleven documented languages; gRPC generates its stubs from the same files. Editions set behaviors like field presence per file or field, released roughly annually with exact compatibility guarantees. Quarterly releases share one protoc number across runtimes.
Details
CompareHow Protocol Buffers answers the questions Serialization Formats turns on.
Serialization Formats
| How it works | |
| Encoding | Tag-length-value, and the tag is the design. Each field is preceded by a varint holding the field number shifted left three bits with a wire type in the low three (VARINT, I64, LEN, I32, and the deprecated SGROUP and EGROUP) so the parser knows how to advance without knowing the schema. Varints carry unsigned 64-bit values in one to ten bytes, small numbers costing least, with zigzag encoding available through sint32 and sint64 because a plain negative int32 otherwise costs the full ten. Strings, bytes, submessages and packed repeated fields share the LEN type. What the format deliberately does not do is delimit itself: the techniques guide is explicit that a parser cannot find message boundaries, and that a caller streaming several messages must write each one's size first |
| Random access | None, and the guidance goes further than absence. Fields carry no offsets and may arrive in any order, so a reader parses from the front, and the documentation states that serialization order is an implementation detail that may change. The project's own advice is to keep messages small enough that this never matters (past roughly a megabyte each "it may be time to consider an alternate strategy") which is the clearest statement in the capability of what this design is for. Cap'n Proto and FlatBuffers exist because that advice is unacceptable for a memory-mapped buffer; this row is the format they were reacting to |
| Self-describing | Structurally yes, semantically no, and it is the same shape as Thrift's answer. Every field is preceded by its number and a wire type, so any parser can walk a message and skip fields it has never heard of, proto3 preserves those unknown fields through a parse-and-reserialize cycle rather than dropping them. What the wire type cannot tell you is the type: VARINT covers int32, uint64, bool and enum alike, and LEN covers strings, bytes and submessages, so without the .proto file a reader sees structure and not meaning. Genuinely self-describing messages are possible but assembled by hand from a FileDescriptorSet plus an Any, and the documentation is candid that this is not in the core library because Google has not needed it |
| Schema evolution | The canonical rules of the capability, and the ones the other tagged formats copied. Adding fields is safe; removing one means reserving its number and ideally its name, because reuse is the failure that corrupts data or leaks it between versions; field numbers may never change. A defined set of type changes is wire-compatible (int32 and int64 and bool interchange, string and bytes when the bytes are valid UTF-8, singular and repeated for scalars) and each carries a caveat about data that may be silently truncated. Unknown-field preservation is what makes rolling upgrades safe: a proxy running old code round-trips a message containing new fields without losing them. Unlike Avro's resolution algorithm, nothing checks any of this at runtime; the guarantees hold if the rules were followed |
| Type system | Fifteen scalar types covering the width and signedness combinations (including the fixed-width and zigzag variants that exist for encoding reasons rather than semantic ones) plus message, enum, repeated, oneof where setting any member clears the others, and map with integral or string keys. Field presence is the subtle part and the thing editions exist to control: proto3 originally made presence implicit, optional was reintroduced to distinguish unset from default, and an edition now sets it as a feature. Above the base sit the well-known types (Any, Timestamp, Duration, Struct, FieldMask) which is where the semantics the wire lacks are conventionally recovered |
| RPC | Defined in the schema, delivered by someone else, and the split is deliberate. A .proto file declares services and methods with request and response message types, and the compiler emits interfaces and stubs for them; what it does not define is a transport, a framing or a call semantics. The documentation names the answer ("the most straightforward RPC system to use with protocol buffers is gRPC", generating its code through a compiler plugin) which is a different arrangement from every other row here: Thrift ships its own protocols and servers, Cap'n Proto builds a capability system into the format, Avro specifies an RPC nobody uses, and this one lets a separate project own the problem and win |
| Spec versions | The column's fourth distinct answer, and the most deliberate. The wire format has not changed in twenty years, while the language has changed twice (and the second change is the interesting one: editions replaced the proto2 and proto3 syntaxes with a feature model where behaviours like field presence are set per file, message or field, released roughly annually with 2023 as the baseline and 2024 current. The guarantee attached is exact: editions "won't break existing binaries and don't change a message's binary, text, or JSON serialization format", and a migration tool converts old files. Releases run quarterly on a shared protoc number with per-language majors. The one qualification worth making is that a default can still move even when the format does not: Edition 2023 made repeated primitive fields packed by default, so the same source now writes different bytes than it did) both encodings having always been valid, and any conformant parser reading either. So the specification moves constantly and the wire format does not, which is structurally the promise Cap'n Proto made for its 2.0, made by a much larger project with far more to break |
| Adoption and maturity | The default, and the row the other five are positioned against. About 71,700 stars and 16,200 forks, the largest in the capability by a wide margin, with commits landing the day this row was written and quarterly releases through the 35.x line. Its reach is structural rather than anecdotal: it is the payload format of gRPC and therefore of most modern service meshes and cloud APIs, the format Buf built an entire registry business around, and the one every schema registry supports. The qualifications are the ones the neighbouring rows exist to make (no random access, an explicit ceiling of about a megabyte a message, and a mandatory compiler) and none of them has dented it |
| Connections | |
| Registry support | The maximal answer, and the only format here with a registry built for it alone. Confluent, Karapace and AWS Glue accept it beside Avro and JSON Schema; Apicurio takes it among nine artifact types; and Buf's registry is Protobuf and nothing else, versioning modules of .proto files with their dependencies, which is the sharpest elimination in the schema-registry leaf and the strongest single argument for standardising on this format if a team's gRPC services and Kafka events share one set of definitions. Three of the six rows in this capability are accepted nowhere; this one is accepted everywhere |
| Generated code | Mandatory and industrial. protoc plus a plugin per target generates the classes, with reference documentation for eleven languages (C++, C#, Dart, Go, Java, Kotlin, Objective-C, PHP, Python, Ruby and Rust) though the boundary is worth knowing: Go and Dart live in their own repositories, and the version-support policy covers a narrower set of runtimes than the docs do. The plugin interface is also how gRPC generates its stubs from the same file. Dynamic use is possible through descriptors and DynamicMessage rather than being the default path, which is the inverse of Avro, where reading without generated classes is the headline feature. The compiler is the product in the way it is for Thrift, but with a versioning scheme unlike anything else here: one protoc release number shared across runtimes that each carry their own major, so release 35.1 is Java 4.35.1 and Python 7.35.1 |
Alternatives to Protocol Buffers
