Project Nessie
Project Nessie is an Apache-2.0 transactional catalog for data lakes with git-like semantics: Iceberg tables and views are versioned on branches and tags, changes across multiple tables commit atomically, and engines connect through the Iceberg REST protocol.
LicenseOpen source (permissive)· Apache-2.0
DeploymentSelf-hosted
PricingFree
Operational complexityMedium
WorkloadInteractiveBatch
LanguageJava
Use it when
- The lake should be branchable: rewrite several tables on a branch, validate, then merge or discard without consumers seeing partial state.
- Changes must land atomically across tables; a Nessie commit spans many, where every other catalog gives per-table atomicity only.
- Pinned states matter: tags and a commit log give reproducible reads at a commit, not just a timestamp.
- Access rules differ by branch, such as who may commit to prod while everyone branches freely.
Think twice when
- Your tables are not Iceberg; Nessie versions Iceberg tables and views, currently nothing else.
- Garbage collection cannot be an afterthought: Nessie GC is a mandatory separate job with its own database, and long-lived branches keep files alive.
- Choosing the version store is a real decision; relational backends bottleneck under concurrent commits to one branch, and several are experimental.
- You expect everything over Iceberg REST; that path is still labeled experimental, and the git-style operations arrive as Spark SQL extensions.
How it runs
A Java (Quarkus) server over a version store (BigTable, MongoDB, PostgreSQL, or single-node RocksDB in production), shipped as a small container image. Engines connect through the Nessie catalog that lives in Iceberg's own source tree, with Spark, Flink, Trino, Hive, and Dremio documented. Credential vending covers S3 session credentials and request signing. Releases are frequent and small; the 0.x version number signals cadence, not immaturity.
Details
CompareHow Project Nessie answers the questions Table Catalogs turns on.
Table Catalogs
| How it works | |
| Protocol | Two REST APIs from one server, Nessie's own, where branches, tags, merges and the commit log live, and the Iceberg REST API, which Nessie's own guide still labels experimental. The route engines have used for years is neither: a Nessie catalog implementation that ships inside Apache Iceberg itself, which is why the git-style operations arrive as Spark SQL extensions rather than as REST calls |
| What it catalogs | Namespaces, Iceberg tables and Iceberg views, plus the things that make it a repository rather than a directory, branches, tags and commits. Extending to Spark jobs, blobs or git submodules is explicitly still under discussion, so the catalogued surface is narrower than Unity Catalog's and the versioned surface is wider than anyone's |
| Branching and versioning | The reason this column exists. Branches, tags, merges and a commit log across the whole repository, driven from Spark SQL as CREATE BRANCH, MERGE BRANCH, SHOW LOG and time travel to a timestamp or a commit, with a single commit able to span many tables and three optimistic isolation levels exposed up to serializable. The honest qualification is that the server holds these primitives ahead of the tools: branch-then-merge works today, while the project says tools still need enhancing before most expose the multi-table commit directly |
| Running it | |
| Maintenance | Nessie GC, and it is not optional, keeping many versions of metadata alive means old files go away only when you run it. A separate uber-jar with its own JDBC database, in three phases: identify live content per named reference, expire by matching those references against a listing of the actual files, then delete, either immediately or from a recorded set of orphans. Iceberg only. This is where a long-lived branch shows up as a cost |
| What you run | A Quarkus server and a version store, and choosing the store is the real decision. BigTable, MongoDB, PostgreSQL and single-node RocksDB are production; DynamoDB is beta and explicitly not recommended with Iceberg REST because of its row-size limit; Cassandra, MariaDB, MySQL and CockroachDB are experimental with named failure modes; in-memory and H2 are development only. Relational stores are documented as a bottleneck under concurrent commits to one branch |
| Connections | |
| Table formats | Iceberg, and the documentation says "currently", Iceberg tables and Iceberg views, with other formats framed as something to add rather than something supported. Where the other rows widen across formats, this one takes a single format and adds a dimension to it |
| Engine support | Spark, Flink, Hive and Trino through the Nessie catalog that lives in Iceberg's own source tree, Dremio documented alongside them, and a feature matrix the project maintains for the rest. Anything reaching it over Iceberg REST instead is on the newer path, the one carrying the experimental label |
| Access | |
| Access control | Access control over a pair of coordinates, reference and path, so a rule can say who may commit to prod while leaving the same person free to branch and experiment, with an AccessChecker SPI for teams writing their own rules and OIDC behind authentication. The documentation is candid about the limit: Nessie governs metadata, not files, so anything reachable directly in the bucket is outside its reach |
| Storage credentials | Both mechanisms rather than one, short-lived, down-scoped S3 session credentials and S3 request signing, pushed to clients over Iceberg REST along with the FileIO configuration, with the configured access keys never reaching a client. Buckets are configured per warehouse with STS assume-role settings, and GCS and ADLS work the same way while both remain experimental |
| Cost | |
| Billing unit | Nothing to meter: Apache-2.0 with no commercial edition from the project. The cost is what you run: the server, the version store behind it and the GC database beside it, which is three pieces rather than one |
Alternatives to Project Nessie
