Apache DataFusion
Apache DataFusion is an extensible query engine written in Rust that uses Apache Arrow as its in-memory format, shipped as libraries for building database and analytic systems rather than as a server to run.
LicenseOpen source (permissive)· Apache-2.0
DeploymentSelf-hosted
PricingFree
Operational complexityLow
WorkloadInteractiveBatch
LanguageRust
Use it when
- You are building a database, query service, or analytic tool and want a production query engine as a library instead of writing one.
- Your system is Arrow-native; DataFusion keeps data in Arrow end to end and composes with that ecosystem.
- You need to customize deeply: data sources, functions, operators, and even the SQL front end are replaceable.
- You want in-process analytics in Rust or Python over Parquet, CSV, JSON, or Avro without a server.
Think twice when
- You want a queryable service for a team; DataFusion has no wire protocol, governance, or concurrency control, where DuckDB or Trino ship those.
- You need distributed execution out of the box; that lives in Ballista and other subprojects rather than the core.
- Iceberg or Delta are central to the design; both come from external crates, not the core engine.
How it runs
It does not run on its own; it is embedded. Add the Rust crate or the Python bindings to your application, register tables or a catalog in code, and execute SQL or DataFrame plans in-process on a Tokio thread pool. Comet accelerates Spark with it; Ballista distributes it.
Details
CompareHow Apache DataFusion answers the questions Query Engines turns on.
Query Engines
| How it works | |
| Engine shape | Embedded Rust library: a component, not a product |
| Catalog required | Nothing required: CatalogProvider is a trait the embedding application fills, and most just register tables directly |
| SQL dialect | Its own, parsed by datafusion-sqlparser-rs, generic with PostgreSQL features added on request, no compatibility target |
| Running it | |
| Concurrent users | Whatever the host process gives it, a Tokio thread pool, with no queueing, admission control or multi-tenancy |
| Acceleration | Every query scans again; the only cache is an opt-in session cache of file statistics and orderings, not of results |
| Failure recovery | None: a failed query fails; Ballista adds a distributed scheduler as a separate project |
| Connections | |
| Non-lake sources | None in core; datafusion-contrib crates add Postgres, MySQL, SQLite, DuckDB and Flight table providers, with pushdown through datafusion-federation |
| Table formats | CSV, Parquet, JSON, Avro and Arrow built in; Iceberg and Delta only through external crates |
| Client access | In-process from Rust or Python, through SQL or a DataFrame API; datafusion-cli for a shell. No wire protocol, so no JDBC or ODBC |
| Access | |
| Access control | None: whatever the embedding application enforces |
Alternatives to Apache DataFusion
