# DuckLake

> Lakehouse format keeping all metadata in a SQL database

DuckLake is an integrated data lake and catalog format from the DuckDB Foundation that stores table metadata in an ACID SQL database rather than in files, with data in Parquet on object storage.

Website: https://ducklake.select

## Use it when

- A small team wants lakehouse semantics (snapshots, time travel, schema evolution, ACID) without file-based metadata ceremony.
- You already run PostgreSQL and DuckDB; the catalog is a database you have, and commits are ordinary transactions.
- Writes are frequent and small; DuckLake inlines them in the catalog instead of scattering tiny Parquet files.
- Compaction fatigue is real: metadata never accumulates as files, so upkeep reduces to a few SQL calls or one CHECKPOINT.

## Think twice when

- Every reader and writer must reach a running SQL database, which is easy for one team and more involved across an organization.
- Engine reach matters today: DuckDB is the reference implementation, with younger clients for Spark, Trino, DataFusion, and pandas.
- A long production track record is required; v1.0 arrived in April 2026, far younger than Iceberg, Delta, or Hudi.
- Interoperability must be in place rather than by copy; Iceberg exchange is a copy in or out, not a metadata bridge.

## How it runs

DuckLake is an extension plus a catalog database: PostgreSQL, SQLite, or DuckDB itself holds all table metadata, while data stays in Parquet on local disk or object storage. A commit is a database transaction with snapshot isolation and automatic conflict retries. Maintenance is expire_snapshots, merge_adjacent_files, and rewrite_data_files, or CHECKPOINT to run everything at once.

## Details

### General

| Attribute | Value |
| --- | --- |
| License | Open source (permissive) |
| SPDX identifier | MIT |
| Deployment | Self-hosted, Managed |
| Operational complexity | Low |
| Pricing | Free |
| Language | C++ |
| Repository | https://github.com/duckdb/ducklake |
| Documentation | https://ducklake.select/docs/stable/duckdb/introduction |
| Stars | 3k |

### Table Formats

| Attribute | Value |
| --- | --- |
| Engine support | DuckDB reference; clients for Spark, Trino, DataFusion and pandas |
| Catalog | Required: a SQL database (PostgreSQL, SQLite, DuckDB) |
| Update model | Small writes inlined in the catalog; Parquet otherwise |
| Partitioning | Declared per table, murmur3 buckets included; file stats prune regardless |
| Schema evolution | Column IDs: add, drop and rename without rewriting files |
| Concurrent writers | The catalog database's transactions: snapshot isolation, conflicts retried automatically |
| Indexes | Per-file zone maps in the catalog tables; no secondary indexes |
| Maintenance | expire_snapshots, merge_adjacent_files, rewrite_data_files: or CHECKPOINT for all |
| Change feed | table_changes() between two snapshots |
| Cross-format reads | Copy to or from Iceberg since 0.3; Iceberg v3 deletion vectors experimental |

## Capabilities

- [Table Formats](https://matca.io/capabilities/table-format)

---

Source: https://matca.io/ducklake
Last updated: 2026-08-31T11:40:47.169Z