Daft
Daft is a Rust-based data engine with a Python interface, processing images, audio, video, embeddings, and structured data through one API, and scaling from a laptop to a cluster without code changes.
LicenseOpen source (permissive)· Apache-2.0
DeploymentSelf-hosted
PricingFree
Operational complexityMedium
WorkloadBatch
LanguageRust
Use it when
- The dataset is multimodal: images, audio, video, embeddings, or text alongside tables, all in one Python API.
- Model inference belongs inside the pipeline; UDFs request GPUs (fractional shares included) and LLM prompting, embedding generation, and classification are built in.
- The same code must run on a laptop and on a Ray cluster without changes.
- You read broadly across lakehouse formats: Iceberg, Delta, Hudi, Paimon, and Lance, plus Unity Catalog and Glue.
Think twice when
- The work is plain tabular SQL at scale; Spark, Trino, and the warehouses are the deeper, more settled route.
- You will not run Ray; scaling out has exactly one answer here.
- Memory-inflating operators and heavy UDFs can still exhaust a worker; the levers are batch size and UDF concurrency, and the project is younger than its peers.
How it runs
Locally there is nothing to deploy: a Rust engine (Swordfish) streams morsels of work through your Python process. To scale out, the Flotilla runner spreads the same engine across a Ray cluster, deployable on Kubernetes via Helm. Checkpointing plus idempotent Iceberg and Delta commits let a rerun skip completed work.
Details
CompareHow Daft answers the questions Batch Compute turns on.
Batch Compute
| How it works | |
| APIs | Python DataFrame and SQL, both first-class |
| Data model | Multimodal: images, audio, embeddings beside tables |
| Execution model | Streaming morsel-driven Rust engine locally, codenamed Swordfish; the Flotilla runner spreads that same engine across Ray actors |
| Larger than memory | Streams bounded batches and spills under pressure, but inflationary operators and UDFs still OOM, the levers are batch size and UDF concurrency |
| Running it | |
| Runtime | Rust core with Python bindings |
| GPU support | UDFs request num_gpus and num_cpus, fractional GPUs included, with concurrency capping how many run at once, built for inference inside the pipeline |
| What you run | Nothing beyond your own machine locally; scaling out means a Ray cluster, and a Helm chart deploys either shape on Kubernetes |
| Scaling | The Ray autoscaler's, with Daft retiring idle Flotilla workers on scale-in; locally it just takes the cores it finds |
| Failure recovery | Checkpointing skips already-processed rows on a rerun through an anti-join, with idempotent commits to Iceberg and Delta Lake; in-flight recovery is the Ray runner's |
| Connections | |
| Table formats | Iceberg, Delta Lake, Hudi, Paimon and Lance, the fullest here; Parquet, CSV, JSON and WARC underneath, plus Kafka and Hugging Face |
Alternatives to Daft
