DuckLake
DuckLake is an integrated data lake and catalog format from the DuckDB Foundation that stores table metadata in an ACID SQL database rather than in files, with data in Parquet on object storage.
LicenseOpen source (permissive)· MIT
DeploymentSelf-hostedManaged
PricingFree
Operational complexityLow
LanguageC++
Use it when
- A small team wants lakehouse semantics (snapshots, time travel, schema evolution, ACID) without file-based metadata ceremony.
- You already run PostgreSQL and DuckDB; the catalog is a database you have, and commits are ordinary transactions.
- Writes are frequent and small; DuckLake inlines them in the catalog instead of scattering tiny Parquet files.
- Compaction fatigue is real: metadata never accumulates as files, so upkeep reduces to a few SQL calls or one CHECKPOINT.
Think twice when
- Every reader and writer must reach a running SQL database, which is easy for one team and more involved across an organization.
- Engine reach matters today: DuckDB is the reference implementation, with younger clients for Spark, Trino, DataFusion, and pandas.
- A long production track record is required; v1.0 arrived in April 2026, far younger than Iceberg, Delta, or Hudi.
- Interoperability must be in place rather than by copy; Iceberg exchange is a copy in or out, not a metadata bridge.
How it runs
DuckLake is an extension plus a catalog database: PostgreSQL, SQLite, or DuckDB itself holds all table metadata, while data stays in Parquet on local disk or object storage. A commit is a database transaction with snapshot isolation and automatic conflict retries. Maintenance is expire_snapshots, merge_adjacent_files, and rewrite_data_files, or CHECKPOINT to run everything at once.
Details
CompareHow DuckLake answers the questions Table Formats turns on.
Table Formats
| How it works | |
| Catalog | Required: a SQL database (PostgreSQL, SQLite, DuckDB) |
| Update model | Small writes inlined in the catalog; Parquet otherwise |
| Partitioning | Declared per table, murmur3 buckets included; file stats prune regardless |
| Schema evolution | Column IDs: add, drop and rename without rewriting files |
| Concurrent writers | The catalog database's transactions: snapshot isolation, conflicts retried automatically |
| Indexes | Per-file zone maps in the catalog tables; no secondary indexes |
| Change feed | table_changes() between two snapshots |
| Running it | |
| Maintenance | expire_snapshots, merge_adjacent_files, rewrite_data_files: or CHECKPOINT for all |
| Connections | |
| Engine support | DuckDB reference; clients for Spark, Trino, DataFusion and pandas |
| Cross-format reads | Copy to or from Iceberg since 0.3; Iceberg v3 deletion vectors experimental |
Alternatives to DuckLake
