Nimble
Nimble is Meta's Apache-2.0 columnar file format, built as a replacement for Parquet and ORC on workloads with thousands of columns: FlatBuffers metadata, block rather than stream encoding, and an encoding layer decoupled from the file layout so new schemes can be added and composed.
LicenseOpen source (permissive)· Apache-2.0
DeploymentSelf-hostedManaged
PricingFree
Operational complexityLow
WorkloadBatchInteractive
LanguageC++
Use it when
- Tables are thousands to tens of thousands of columns wide, the feature-engineering and ML-training shape Parquet's and ORC's footers choke on.
- Metadata cost is the bottleneck: FlatBuffers footers are read without whole deserialization, and even the metadata can be stream-encoded.
- Feature columns are huge maps; flat maps store one stream per key so a column with thousands of keys is read key by key.
- You are prepared to build and embed a C++ library and want Meta's actively developed answer to this workload.
Think twice when
- You expect an ecosystem: no open-source engine ships a Nimble reader, there is no Trino connector, DuckDB extension, Spark reader, Python binding, or Arrow interop.
- Stability matters now; the project states it provides no versioning guarantees yet, has no tagged releases, and says "use it at your own risk."
- Your types include decimals or dates; the reader supports only what ML needs.
- You want an open specification; the project is explicit that Nimble is a product with one library, and discourages independent implementations.
How it runs
A C++ library closely coupled to Velox, which it vendors as a submodule and registers into as a reader; using it outside Meta means linking this repository into your own engine build. Encodings cascade and are selected per block by pluggable policies; block compression is Zstd or LZ4 in open-source builds.
Details
CompareHow Nimble answers the questions File Formats turns on.
File Formats
| How it works | |
| Adoption and maturity | Meta's format, formerly Alpha, open-sourced in December 2023 and developed as an export from Meta's internal repository, 98 commits in the last 30 days, against 730 stars, 93 forks and 152 open issues. There are no releases and no tags, ever. Meta's own framing is ambition rather than deployment: the format has "the potential to supersede current mainstream analytic file formats within Meta and beyond", and the VeloxCon 2026 keynote positions the Nimble serializer as the near-term production path for training data and the cluster-key and dense-index work as the long-term one. External adoption is the missing half: no third-party engine ships it, and the anti-reimplementation stance is deliberate about keeping it that way |
| Encodings and compression | Cascading encodings, extensible by library users, and decoupled from the physical layout (an encoding's sub-streams are each encoded again by the same selection process with the parent's type excluded to stop the recursion, so a dictionary's alphabet and its indices take different schemes. The guide documents twelve (trivial, constant, mainly-constant, RLE, dictionary, fixed-bit-width, varint, delta, sentinel, nullable, sparse-bool, prefix) and the tree holds more than that) ALP, FSST, Huffman, PFOR, SIMD frame-of-reference bit-packing, sub-int splitting, frequency partitioning, shared dictionaries. Selection is a pluggable policy reading per-block statistics. Block compression underneath is Zstd, LZ4 or Meta's OpenZL, and there is a trap worth knowing: the default codec is MetaInternal inside Meta and Zstd in open-source builds, and MetaInternal is compiled out of those builds entirely, so a file written internally with the default has no decompressor outside |
| Indexes | The richest of the four, which is the opposite of what a format built for sequential ML reads suggests. Split-block bloom filters in 256-bit blocks, explicitly following Parquet's design; chunk stats that index every stream for O(1) chunk-level seeking by row ID, carrying per-chunk row counts, byte offsets and null counts; and a file-level index framework declaring two families (Cluster, a primary index over data ordered by its key columns, and Dense, a secondary index independent of physical ordering) with sorted and hash implementations behind a registry, each on a composite key. A sorted index stores encoded key plus row ID with per-chunk boundary keys for binary search. Column statistics carry min, max, value and null counts, and the whole apparatus is optional and threshold-driven, so a file only pays for what its workload uses |
| Random access | Real and unmeasured, which is the reverse of the Vortex row. The machinery is there in the source: point lookups and range scans on encoded keys through the cluster index, binary search over per-chunk keys down to a row ID, chunk-level seeking by row ID from the chunk stats, and a projector that batches many lookups and serialises just the projected columns for transport. What is absent is any published number (no random-access benchmark, no comparison against Parquet or anything else) and the work is recent enough that Meta's own keynote describes cluster keys and dense indexes as the long-term direction for training-data ingestion rather than as a finished feature |
| Wide schemas | The reason this format exists, and the only row on the page that names a number: tables with thousands of columns or streams, which the README puts at "thousands to tens of thousands", the shape feature-engineering and ML training tables actually take. Three decisions carry it. Metadata is FlatBuffers rather than Thrift or Protobuf, so a large metadata section is accessed without deserialising it whole. Stripes are grouped, and a stripe group's per-stream offsets and sizes can themselves be stored stream-major and Nimble-encoded, each stream's integers bit-packed to its own range, metadata compression for the case where the metadata is the problem. And flat maps turn a map column into one stream per key, with the writer able to order those streams explicitly, so a feature column with thousands of keys is read key by key instead of whole |
| Nested data | Rows, arrays and maps, and then the parts that give away what it was built for: thirteen flat-map kinds, one per key type from int8 through string and binary, plus ArrayWithOffsets and SlidingWindowMap, which exist to deduplicate repeated nested values rather than to model anything new. Nulls are an encoding choice rather than a fixed convention: a Nullable encoding pairs a null bitmap with a non-null value stream, a Sentinel encoding reserves a value to mean null, and the selection policy picks per stream. The type enum is also where the boundary shows: there is no decimal and no date type (timestamps are micro-nanosecond only) and the reader's own header says it "only supports the types needed for ML" |
| ACID primitives | None: no transactions, no row-level updates or deletes, no delete files and no row IDs of the kind Hive's ACID tables need from ORC. The file carries a key-value metadata section and checksums, and everything transactional is left to a layer above, which for this format does not exist yet: the Velox connector work to support Iceberg and Paimon, announced at VeloxCon 2026, is what would put a table format over it |
| Spec versions | The bluntest answer in the capability and the exact mirror of the Vortex row. The README states that Nimble "does not provide stability or versioning guarantees (yet)", that they will come with a future stable release, and closes the feature list with "Use it at your own risk" (and the repository backs that up by having no releases and no tags at all since it was opened in December 2023. What the format does practise, without promising it, is append-only compatibility: footer field slots frozen for pre-existing files, and new fields like per-chunk null counts appended so older readers treat them as unknown rather than failing. Behind this sits a deliberate governance position) "more than a specification, Nimble is a product", which discourages independent implementations of the spec in favour of bindings over the single library, and is the fact most likely to decide whether the format ever leaves Meta's orbit |
| Connections | |
| Engine support | The narrowest answer on the page, and narrower than "it runs on Velox" suggests. Nimble is a Velox client rather than a part of Velox (it vendors Velox as a submodule and registers its reader into Velox's DWIO registry, while Velox itself ships dwrf, orc, parquet and text and no nimble. So an engine reads Nimble only if its build links this repository, which no open-source distribution is known to do: no Trino connector, no DuckDB extension, no JVM Spark reader, no Python or Arrow entry point, and Prestissimo and Gluten are sibling Velox clients rather than shipping readers. Outside Meta, using it means building and embedding it yourself. The route outward being built is the table format) Velox's VeloxCon 2026 keynote put Iceberg and Paimon connector support beside the Nimble index work |
| Arrow interop | None, and this is the only row on the page where that is the whole answer. There is no Arrow dependency, no arrow module and no Python binding anywhere in the repository; the in-memory type it reads into is a Velox vector, and anything Arrow has to come from Velox's own Arrow bridge on the far side of that. For a format whose neighbours treat zero-copy Arrow as table stakes, this is the clearest expression of the project's position that Nimble is a product with one library rather than a specification with many readers |
Alternatives to Nimble
