Vortex
Vortex is an extensible columnar file format written in Rust and incubating at the Linux Foundation, claiming order-of-magnitude gains over Parquet on random access and scans at comparable compression, with an encoding system designed to be extended rather than frozen into the specification.
LicenseOpen source (permissive)· Apache-2.0
DeploymentSelf-hostedManaged
PricingFree
Operational complexityLow
WorkloadBatchInteractive
LanguageRust
Use it when
- Random access over lake data is the workload: point lookups, feature fetches, vector-search sidecars, where the project's published numbers against Parquet are largest.
- Engines should work on compressed data: kernels run over the native encodings and hand compressed arrays straight to DuckDB or DataFusion.
- You want new compression research without a new format: encodings are plugins (FastLanes, FSST, ALP, PCodec) rather than spec-frozen.
- The integration path exists today: a DuckDB core extension, native DataFusion support, a Spark connector, and zero-copy Arrow both ways.
Think twice when
- You are standardizing a lake for years; the library is pre-1.0, releases land weekly, and the honest posture is evaluate rather than adopt.
- Pruning must go beyond statistics; there are zone maps but no bloom filters or value indexes.
- Your data needs maps, fixed-length binary, or variants, which are not supported yet.
- Table-format integration is the deciding factor; Iceberg support is still work in progress.
How it runs
A Rust library, incubating at LF AI & Data with Microsoft, Snowflake, and Palantir named as supporters. Files are readable forever from format version 0.36.0 on: encoding sets freeze into published editions, and writers default to a lagging edition so older deployments keep reading. Nothing to operate; read and write through DuckDB, DataFusion, Spark, or the Python bindings.
Details
CompareHow Vortex answers the questions File Formats turns on.
File Formats
| How it works | |
| Adoption and maturity | An Incubation-stage project at LF AI & Data since 6 August 2025, donated by SpiralDB to the Sandbox in June 2025, with Microsoft, Snowflake and Palantir named as supporters, the neutral home is real rather than a logo. About 3,100 stars, commits daily, and releases arriving several times a week through the 0.8x series, 0.84.0 on 7 August 2026. Adoption is early but no longer only the vendor's: DuckDB shipping it as a core extension is the strongest third-party signal, and Microsoft has demonstrated a 30% runtime reduction on Spark workloads with Vortex inside Iceberg. Library APIs are still pre-1.0 even though the format is not |
| Encodings and compression | The reason the format exists, and the axis Parquet cannot move on. Encodings are a plugin system decoupled from the layout (FastLanes SIMD bit-packing with delta, frame-of-reference and run-length, FSST for strings, ALP and ALPrd for floats, PCodec, ZigZag, RunEnd, sparse fill-plus-patches, dictionary and constant) and compute kernels run directly over the compressed representation, falling back to a canonical decode only when no kernel exists. Writers pick a strategy rather than a codec: BtrBlocks by default, cascading lightweight schemes for read speed, or Compact, which spends CPU on ZStd and PCodec to minimise size. Segments carry ordinary block compression too (LZ4, Zlib, ZStd). The project's own compression dashboard is honest about the trade: a geometric mean of 1.07x Parquet's file size (about 7% larger) bought with 3.01x the write throughput and 21.06x the scan throughput |
| Indexes | No bloom filters and no value indexes (nothing of the kind ORC ships) and pruning is entirely statistics-driven instead. A ZonedLayout stores zone maps for every 8,192 rows, and the file's postscript points at a statistics segment carrying per-field statistics for whole-file pruning before any column is touched. The stat set is wider than Parquet's: minimum, maximum, sum, null count, NaN count, is-constant, is-sorted, is-strict-sorted and uncompressed size. Pruning is a three-valued falsification proof, so a missing or inexact statistic lowers to null and can never skip data by accident, and at scan time the filter's conjuncts are reordered by measured selectivity as the scan learns which one cuts hardest |
| Random access | The row this format was built for, and the claim it leads with: roughly 100x faster point lookups than Parquet, take() on a file by row index rather than a scan with a predicate. The number is the project's own, and the benchmark behind it is at least specific (correlated lookups in clusters and uniform ones drawn from a Poisson process, over taxi, feature-vector, nested-list and nested-struct datasets, against Parquet and Lance, with the file handle cached and again reopened per lookup. The published dashboard currently reports 337 microseconds against Parquet's nine seconds, with Lance) the other format built for this, within 5% of Vortex rather than behind it. The mechanism is ordinary enough to believe: a FlatBuffers footer read from the file tail, a layout tree fetched lazily, a row-index layout that lets a filter address rows by position, and segment offsets that let a reader pull the bytes for one row without materialising a row group |
| Wide schemas | Designed for them, and the design is visible in the format rather than asserted in a benchmark. The postscript is capped at 65,527 bytes and the reader's default initial read is that cap plus the 8-byte end marker, so a single 64 KiB read of the file tail lands every metadata locator however wide the table; the footer is FlatBuffers with dictionary-encoded registries, giving O(1) access to a single column's specification without deserialising the whole thing; and the schema lives in its own segment separate from the footer precisely so a large one can be omitted from the file and fetched externally. The specification's own goal is "minimal overhead reading few columns or rows from wide or long arrays", and the LF AI & Data announcement puts wide schemas and multimodal data among the workloads it was built for. What is missing is a published number, Nimble is the row on this page that names a column count |
| Nested data | Struct, List and FixedSizeList as logical types, with nested structs and nested lists real enough to be two of the four datasets in the project's own random-access benchmark. Nullability sits on the dtype itself rather than on a field wrapper, and there is no schema concept at all, a file's root type need not be a struct, so a bare Float64 array is a valid Vortex file. Dates, times and timestamps are Extension types composed over primitives rather than built-ins. The gaps are stated in the documentation: maps, fixed-length binary and variants are not supported yet |
| ACID primitives | None, and the contrast is the point: where ORC put transactions in the file itself and Hive built ACID tables on that, Vortex has no row IDs, no delete files, no base-and-delta layering and no notion of a transaction. Everything of that kind belongs to a table format above it, which is what makes Apache Iceberg support (currently on the work-in-progress list, and the setting for Microsoft's Spark demonstration) the integration that decides whether this format reaches production lakehouses |
| Spec versions | Sharper than the 0.84 version number suggests, and worth reading before dismissing it as pre-1.0. The file format has been stable since 0.36.0, with every later library version able to read files written by 0.36.0 or later, and editions turn that into a read-forever guarantee: an edition's encoding list is frozen once published, core2025.05.0 being the first, new encodings stage in a draft edition, writers default to a core edition lagging the latest release so files stay readable by deployments a few versions behind, and encodings that fall out of use stop being written but never stop being read. What is missing is the other direction, a file using an encoding your build does not know means your build is too old, and forward compatibility, by embedding WASM decompression kernels for newer encodings, is planned before 1.0 rather than shipped. The 0.x number is the library API's, not the format's |
| Connections | |
| Engine support | DuckDB as a core extension since January 2026, built with DuckDB Labs, INSTALL vortex, then read_vortex() to read and COPY … TO … (FORMAT vortex) to write, with Linux and macOS builds and no Windows one. DataFusion natively, Spark through a DataSource V2 connector on Maven Central as dev.vortex:vortex-spark_2.13 for Spark 4 and _2.12 for Spark 3.5, and Python covering PyArrow, pandas, Polars and Ray Data. Trino is the caveat: the documentation home lists it among the native integrations, while the work-in-progress page puts the JNI connector under active development alongside Iceberg, Substrait and cuDF |
| Arrow interop | Zero-copy in both directions (vx.array() builds a Vortex array from an Arrow array without copies, to_arrow_array() and to_arrow_table() convert back, and reading a file returns a PyArrow RecordBatchReader. The more interesting half is where Arrow is deliberately bypassed: the Scan API hands compressed arrays straight to the engine in their native encoding, so DuckDB receives FSST-encoded strings into its own FSST representation with no decompression step, where the conventional integration would decompress into Arrow first. The type models differ on purpose too) nullability on the dtype, one Utf8 rather than string and large_string, encodings kept a separate concept from types rather than modelled as dictionary types |
Alternatives to Vortex
