Apache Parquet
Apache Parquet is the standard columnar file format for analytics — compressed, splittable, and readable by effectively every engine, table format, and data tool.
LicenseOpen source (permissive)· Apache-2.0
DeploymentSelf-hostedManaged
PricingFree
Operational complexityLow
WorkloadBatchInteractive
LanguageJava, C++, Rust, Go
Use it when
- You need the safe default: every engine reads it, every table format builds on it (Delta stores nothing else; Iceberg and Hudi default to it).
- Scans dominate and pruning pays: footer statistics, per-page column indexes, and bloom filters skip data before it is read.
- Files must stay readable for a decade; the specification deprecates but never breaks, and four independent implementations keep each other honest.
- Nested data is real: the Dremel model reads deeply nested fields without materializing parents, and VARIANT now covers semi-structured data.
Think twice when
- The workload is point lookups or random batches (feature serving, vector search); the floor is a page per column, which is what Lance and Vortex were built to beat.
- Tables run thousands of columns wide; the Thrift footer is deserialized whole before any data is read, a cost the project itself now has a proposal against.
- You need encodings the spec has not frozen in; adding one is a multi-year ecosystem negotiation.
- Records are in motion rather than at rest, where row-oriented Avro complements it.
How it runs
A specification with libraries inside the engines: Java, C++ (inside the Arrow repository), Rust, and Go. Files are immutable on object storage; anything transactional comes from a table format above. There is nothing to deploy and nothing to maintain beyond the files themselves.
Details
CompareHow Apache Parquet answers the questions File Formats turns on.
File Formats
| How it works | |
| Adoption and maturity | The default, and the word to avoid is legacy. The specification is at 2.13.0 from June 2026 and parquet-java at 1.18.0 from 11 August 2026, two days before this row was written; the format repository alone carries 2,531 stars and the Java implementation 3,072. The last two years of the specification are not maintenance either: Float16, size statistics for finer-grained filtering, geospatial types, a VARIANT type with a shredding specification for semi-structured data, floating-point statistics handling IEEE 754 total order and NaN counts, and a FILE logical type that resolves to inline bytes or an external URI, meant for images and audio in object storage. The format the AI-era challengers route around is chasing the same workload they are |
| Encodings and compression | A closed set, defined by the specification, and the closure is the whole argument the newer rows make. PLAIN, the RLE and bit-packing hybrid, dictionary encoding as RLE_DICTIONARY with PLAIN_DICTIONARY deprecated, DELTA_BINARY_PACKED for integers, DELTA_LENGTH_BYTE_ARRAY and DELTA_BYTE_ARRAY for strings, and BYTE_STREAM_SPLIT for floating point (under Snappy, GZIP, Brotli, ZSTD or LZ4_RAW. How slowly that list moves is in the changelog rather than in anyone's marketing: format 2.11.0 in March 2025 did not add an encoding, it extended BYTE_STREAM_SPLIT to more types. The project states the reason itself) adding an encoding or compression algorithm is not forward compatible, because older implementations cannot read the new files. What is new is the response: the 2.13.0 release added a formal proposal process, and FSST and ALP are both active proposals at Draft/PoC, which are two of the encodings Vortex already ships |
| Indexes | The reference implementation of statistics-based pruning, and the shape everything else on this page is a variation of. Column chunk statistics in the footer; a ColumnIndex holding per-page minima and maxima with an OffsetIndex mapping row indices to page locations, both stored near the footer so a reader fetches them in one go rather than deserialising every page header to find them; and split-block bloom filters in 256-bit blocks hashed with XXH64, for the high-cardinality columns where a dictionary would cost more than it saves. Size statistics arrived in format 2.10 for estimating unencoded sizes, and floating-point statistics with NaN counts in 2026. ORC's answer is the same shape with a coarser stride, Vortex has zone maps and no bloom filters, and Nimble has more index kinds and almost no users |
| Random access | The weakness the entire challenger generation was built around, and better than its reputation. The page index means, in the specification's own words, that a single-row lookup in a row group based on that row group's sort column will only read one data page per retrieved column, a binary search over per-page bounds rather than a scan. The qualifications are where the two orders of magnitude Vortex claims come from: the binary search needs the column to be the one the row group is sorted by, anything else is a linear pass over the min/max array, and the unit is always a page, so the floor for fetching one row is one page per projected column plus the index. Good for a format designed to scan, and not what a feature lookup or a vector-search sidecar wants |
| Wide schemas | The acknowledged weak point, and the acknowledgement is now in Parquet's own repository rather than in a competitor's README: "Flatbuf footer for wide table support" sits in the active proposals table at Draft/PoC. The problem is structural, the footer is Thrift, it carries metadata for every column chunk of every row group, and a reader deserialises it before touching any data, so the cost grows with columns multiplied by row groups and lands entirely before the first byte of a value is read. Nimble was built on exactly this critique, choosing FlatBuffers precisely to access large metadata sections without deserialising them whole. The incumbent has agreed and started work, which is a better answer than either denying it or losing the workload |
| Nested data | Dremel's model, and the one every other format's nesting is measured against: repetition and definition levels encode arbitrarily nested and repeated structures into flat columns, so a deeply nested field is read without materialising its parents. Lists and maps are annotated groups over that, and the specification carries explicit backward-compatibility rules for two decades of writers that got those annotations wrong, the sort of appendix only a format with a real installed base ever needs. There is no union type, which is ORC's one advantage in this column. The recent additions show the direction: VARIANT with a shredding specification for semi-structured data, GEOMETRY and GEOGRAPHY, FLOAT16, UUID, and a FILE type carrying a URI, offset, size, content type and checksum for unstructured data referenced rather than embedded |
| ACID primitives | None, and the only "none" here that is a decade-old decision rather than a young format's omission. No row IDs, no delete files, no transaction columns: everything transactional sits in a table format above, and the evidence that this was the right call is that all of them chose Parquet anyway, Delta Lake stores nothing else, Iceberg and Hudi default to it, and each built its own deletion mechanism over immutable files. The contrast with the row above is the page's cleanest: ORC put transactions in the file and Hive was the only engine that ever used them, while Parquet left them out and the whole lakehouse generation built on top |
| Spec versions | The strongest guarantee on the page, and it comes from conservatism rather than from a mechanism. The specification is versioned as a whole, now at 2.13.0, and it keeps its own history readable in place (BIT_PACKED and PLAIN_DICTIONARY are marked deprecated and still fully specified, because files written with them exist. The clearest evidence of the culture is a single constant in the reference implementation: parquet-java's default writer version is still PARQUET_1_0, so data page v2) in the specification for years, is not what the Java writer emits unless asked. The cost of that safety is stated by the project itself in its new proposal process, which exists for changes that are not forward compatible, like a new encoding or compression algorithm, because older implementations cannot read the resulting files. That sentence is the constraint Vortex's editions and planned WASM kernels are designed to escape, written down by the format it constrains |
| Connections | |
| Engine support | The answer that needs no list, and the reason the other three rows are arguments rather than replacements: every engine in this catalog reads it, and every table format is built on it, Delta Lake stores nothing else, Iceberg and Hudi default to it. Four independent implementations, and where they live matters: Java in parquet-java, C++ inside the Arrow repository, Rust as a crate in arrow-rs, Go in arrow-go. This is the column where a thirteen-year-old format beats formats that are better than it on almost every other row, and the reason a benchmark win does not settle anything here |
| Arrow interop | Closer than any other row, and structurally rather than by adapter: the C++ implementation lives inside the Arrow repository at cpp/src/parquet, the Rust one is a crate in arrow-rs beside parquet-variant and parquet-geospatial, and the Go one is a package in arrow-go. For three of the four language ecosystems, Parquet and Arrow are the same project's code, and PyArrow is how most of Python reads either. It is still a decode (Arrow's in-memory layout and Parquet's encodings are different things, and materialising one from the other is the step Vortex's compressed handoff exists to remove) but the boundary is thinner here than anywhere else on this page |
Alternatives to Apache Parquet
