Apache Paimon

Use it when

• The pipeline is streaming-first and built on Flink; Paimon's deepest integration is there, including streaming reads, writes, and SQL.
• A table absorbs continuous high-frequency updates by primary key; the LSM structure is designed for exactly that write pattern.
• Downstream jobs need a real change stream: changelog producers emit before and after images instead of forcing snapshot diffs.
• You want one table serving both streaming consumers and petabyte-scale batch queries.

Think twice when

• Your writers are not Flink or Spark; other engines mostly read, and the engine reach is narrower than Iceberg's or Delta's.
• Readers on the Iceberg side matter: the compatibility metadata needs full compaction unless deletion vectors are enabled.
• Compaction ownership is a concern; it runs in the writer or a dedicated job you operate.
• The workload is batch-mostly with occasional updates, where Iceberg or Delta Lake is the simpler fit.

How it runs

Paimon is a library in the engine with tables on object storage or HDFS. Writes land in an LSM tree and flush as sorted runs; merge engines resolve duplicate keys in the background. The catalog is a filesystem directory by default, or Hive, JDBC, or REST. Snapshots expire automatically, orphan files are cleaned manually, and object stores need a catalog lock for concurrent commits.