Analytical File Formats
Data architecture technologies for columnar at-rest encodings optimized for scan-heavy analytical access.
The columnar format the Hive world standardised on
File Formats
Apache ORC is a columnar file format built for the Hadoop and Hive ecosystem: per-column min/max statistics and bloom filters for skipping data, support for ACID transactions and snapshot isolation, and native handling of Hive's compound types.
A columnar format built for random access, not just scans
File Formats
Vortex is an extensible columnar file format written in Rust and incubating at the Linux Foundation, claiming order-of-magnitude gains over Parquet on random access and scans at comparable compression, with an encoding system designed to be extended rather than frozen into the specification.
A columnar format for tables thousands of columns wide
File Formats
Nimble is Meta's Apache-2.0 columnar file format, built as a replacement for Parquet and ORC on workloads with thousands of columns: FlatBuffers metadata, block rather than stream encoding, and an encoding layer decoupled from the file layout so new schemes can be added and composed.
Columnar file format for analytical workloads
File Formats
Apache Parquet is the standard columnar file format for analytics — compressed, splittable, and readable by effectively every engine, table format, and data tool.