Formats & Standards
Data architecture technologies for serialization formats, file formats, and open standards that data systems agree on.
Browse by subcategory
Serialization and RPC generated for fifteen languages at once
Serialization Formats
Apache Thrift is a cross-language services framework: an interface definition language and code generator that emit both the serialization code and the RPC client and server for more than a dozen languages, with pluggable protocols and transports underneath.
The wire format is the in-memory format — no decoding step
Serialization Formats
Cap'n Proto is an MIT-licensed binary interchange format and capability-based RPC system from the author of Protocol Buffers version 2, designed so that the encoded bytes are directly usable in memory — there is no parse step between receiving a message and reading a field.
Read fields straight out of the buffer, with no heap at all
Serialization Formats
FlatBuffers is Google's Apache-2.0 cross-platform serialization library, built for performance-critical code: data is accessed directly in the serialized buffer without unpacking, the only memory needed is the buffer itself, and schemas evolve with forwards and backwards compatibility.
The contract layer for the encoding everything already speaks
Serialization Formats
JSON Schema is the vocabulary for describing, validating, and documenting JSON data — the third schema type every major schema registry supports alongside Avro and Protocol Buffers, and the one that requires no new encoding on the wire.
The columnar format the Hive world standardised on
File Formats
Apache ORC is a columnar file format built for the Hadoop and Hive ecosystem: per-column min/max statistics and bloom filters for skipping data, support for ACID transactions and snapshot isolation, and native handling of Hive's compound types.
A columnar format built for random access, not just scans
File Formats
Vortex is an extensible columnar file format written in Rust and incubating at the Linux Foundation, claiming order-of-magnitude gains over Parquet on random access and scans at comparable compression, with an encoding system designed to be extended rather than frozen into the specification.
A columnar format for tables thousands of columns wide
File Formats
Nimble is Meta's Apache-2.0 columnar file format, built as a replacement for Parquet and ORC on workloads with thousands of columns: FlatBuffers metadata, block rather than stream encoding, and an encoding layer decoupled from the file layout so new schemes can be added and composed.
Row-oriented serialization with first-class schema evolution
Serialization Formats
Apache Avro is a compact, schema-based, row-oriented serialization format — the default encoding for Kafka pipelines and schema-registry workflows.
Compact, typed serialization from the gRPC ecosystem
Serialization Formats
Protocol Buffers (protobuf) is Google's language-neutral, schema-first binary serialization format, ubiquitous in service APIs and increasingly present in event streams.
Columnar file format for analytical workloads
File Formats
Apache Parquet is the standard columnar file format for analytics — compressed, splittable, and readable by effectively every engine, table format, and data tool.