# Apache Paimon

> Lake format with LSM-backed streaming updates and changelogs

Apache Paimon is a lake format unifying batch and streaming, using an LSM tree to absorb high-frequency updates and emitting changelogs so downstream jobs can consume what changed rather than rescanning.

Website: https://paimon.apache.org

## Use it when

- The pipeline is streaming-first and built on Flink; Paimon's deepest integration is there, including streaming reads, writes, and SQL.
- A table absorbs continuous high-frequency updates by primary key; the LSM structure is designed for exactly that write pattern.
- Downstream jobs need a real change stream: changelog producers emit before and after images instead of forcing snapshot diffs.
- You want one table serving both streaming consumers and petabyte-scale batch queries.

## Think twice when

- Your writers are not Flink or Spark; other engines mostly read, and the engine reach is narrower than Iceberg's or Delta's.
- Readers on the Iceberg side matter: the compatibility metadata needs full compaction unless deletion vectors are enabled.
- Compaction ownership is a concern; it runs in the writer or a dedicated job you operate.
- The workload is batch-mostly with occasional updates, where Iceberg or Delta Lake is the simpler fit.

## How it runs

Paimon is a library in the engine with tables on object storage or HDFS. Writes land in an LSM tree and flush as sorted runs; merge engines resolve duplicate keys in the background. The catalog is a filesystem directory by default, or Hive, JDBC, or REST. Snapshots expire automatically, orphan files are cleaned manually, and object stores need a catalog lock for concurrent commits.

## Details

### General

| Attribute | Value |
| --- | --- |
| License | Open source (permissive) |
| SPDX identifier | Apache-2.0 |
| Deployment | Self-hosted, Managed |
| Operational complexity | Medium |
| Pricing | Free |
| Language | Java |
| Repository | https://github.com/apache/paimon |
| Documentation | https://paimon.apache.org/docs/master/ |
| Stars | 3.4k |

### Table Formats

| Attribute | Value |
| --- | --- |
| Engine support | Flink-first; Spark and Hive native, read by others |
| Catalog | Filesystem by default; Hive, JDBC or REST metastore |
| Update model | LSM tree with merge engines per primary key; append tables have none |
| Partitioning | Hive-style partitions plus buckets: fixed, dynamic or postpone; nothing hidden |
| Schema evolution | Add, drop, rename and retype columns; primary and partition keys cannot be renamed |
| Concurrent writers | Optimistic snapshot commits, conflicts auto-resolved; object stores need a catalog lock |
| Indexes | LSM order plus file indexes, bloom filter, bitmap, range bitmap; deletion vectors |
| Maintenance | Compaction in the writer or a dedicated job; snapshots expire automatically, orphans manually |
| Change feed | Changelog producers emit before and after images |
| Cross-format reads | Iceberg metadata written alongside; readers need full compaction unless deletion vectors are on |

## Capabilities

- [Table Formats](https://matca.io/capabilities/table-format)

---

Source: https://matca.io/apache-paimon
Last updated: 2026-08-31T11:40:46.971Z