# Daft

> Rust data engine for multimodal and AI workloads

Daft is a Rust-based data engine with a Python interface, processing images, audio, video, embeddings, and structured data through one API, and scaling from a laptop to a cluster without code changes.

Website: https://www.daft.ai

## Use it when

- The dataset is multimodal: images, audio, video, embeddings, or text alongside tables, all in one Python API.
- Model inference belongs inside the pipeline; UDFs request GPUs (fractional shares included) and LLM prompting, embedding generation, and classification are built in.
- The same code must run on a laptop and on a Ray cluster without changes.
- You read broadly across lakehouse formats: Iceberg, Delta, Hudi, Paimon, and Lance, plus Unity Catalog and Glue.

## Think twice when

- The work is plain tabular SQL at scale; Spark, Trino, and the warehouses are the deeper, more settled route.
- You will not run Ray; scaling out has exactly one answer here.
- Memory-inflating operators and heavy UDFs can still exhaust a worker; the levers are batch size and UDF concurrency, and the project is younger than its peers.

## How it runs

Locally there is nothing to deploy: a Rust engine (Swordfish) streams morsels of work through your Python process. To scale out, the Flotilla runner spreads the same engine across a Ray cluster, deployable on Kubernetes via Helm. Checkpointing plus idempotent Iceberg and Delta commits let a rerun skip completed work.

## Details

### General

| Attribute | Value |
| --- | --- |
| License | Open source (permissive) |
| SPDX identifier | Apache-2.0 |
| Deployment | Self-hosted |
| Workload | Batch |
| Operational complexity | Medium |
| Pricing | Free |
| Language | Rust |
| Repository | https://github.com/Eventual-Inc/Daft |
| Documentation | https://docs.daft.ai |
| Stars | 5.8k |

### Batch Compute

| Attribute | Value |
| --- | --- |
| APIs | Python DataFrame and SQL, both first-class |
| Data model | Multimodal: images, audio, embeddings beside tables |
| Table formats | Iceberg, Delta Lake, Hudi, Paimon and Lance, the fullest here; Parquet, CSV, JSON and WARC underneath, plus Kafka and Hugging Face |
| Runtime | Rust core with Python bindings |
| Execution model | Streaming morsel-driven Rust engine locally, codenamed Swordfish; the Flotilla runner spreads that same engine across Ray actors |
| Larger than memory | Streams bounded batches and spills under pressure, but inflationary operators and UDFs still OOM, the levers are batch size and UDF concurrency |
| GPU support | UDFs request num_gpus and num_cpus, fractional GPUs included, with concurrency capping how many run at once, built for inference inside the pipeline |
| What you run | Nothing beyond your own machine locally; scaling out means a Ray cluster, and a Helm chart deploys either shape on Kubernetes |
| Scaling | The Ray autoscaler's, with Daft retiring idle Flotilla workers on scale-in; locally it just takes the cores it finds |
| Failure recovery | Checkpointing skips already-processed rows on a rerun through an anti-join, with idempotent commits to Iceberg and Delta Lake; in-flight recovery is the Ray runner's |

## Capabilities

- [Batch Compute](https://matca.io/capabilities/batch-compute)

---

Source: https://matca.io/daft
Last updated: 2026-08-31T11:40:47.491Z