Lance

Use it when

• The workload is machine learning over images, video, audio, text, or embeddings, where reads fetch scattered rows rather than scan columns.
• Queries mix vector similarity, full-text search, and SQL filters over one dataset; those indexes are part of the format itself.
• Feature engineering keeps adding derived columns to large datasets; schema evolution is zero-copy, metadata only.
• Training and serving pull random batches, the access pattern Parquet handles worst and Lance's headline claim (100x faster random access) targets.

Think twice when

• The job is conventional analytics over tabular data; Iceberg or Delta over Parquet is the established fit with the wider engine reach.
• Your stack expects deep Spark, Trino, or warehouse support; Lance is Python- and Rust-first, with those integrations younger.
• Interoperability matters: there is no metadata bridge, so catalogs can hold Lance beside Iceberg but the data stays Lance.
• You need a long-proven spec; the design was published at VLDB 2025 and the format is still moving quickly.

How it runs

Lance spans file format, table format, and catalog spec in one Rust core with Python bindings. Rows land in fragments on local disk or object storage; MVCC gives transactional commits, using conditional puts or DynamoDB on S3. You compact fragments, optimize indexes, and clean old versions, with configurable auto-cleanup. It plugs into pandas, Polars, Ray, PyTorch, DuckDB, and Spark, and LanceDB builds a vector database on top.