# Apache DataFusion

> Arrow-native query engine you embed rather than deploy

Apache DataFusion is an extensible query engine written in Rust that uses Apache Arrow as its in-memory format, shipped as libraries for building database and analytic systems rather than as a server to run.

Website: https://datafusion.apache.org

## Use it when

- You are building a database, query service, or analytic tool and want a production query engine as a library instead of writing one.
- Your system is Arrow-native; DataFusion keeps data in Arrow end to end and composes with that ecosystem.
- You need to customize deeply: data sources, functions, operators, and even the SQL front end are replaceable.
- You want in-process analytics in Rust or Python over Parquet, CSV, JSON, or Avro without a server.

## Think twice when

- You want a queryable service for a team; DataFusion has no wire protocol, governance, or concurrency control, where DuckDB or Trino ship those.
- You need distributed execution out of the box; that lives in Ballista and other subprojects rather than the core.
- Iceberg or Delta are central to the design; both come from external crates, not the core engine.

## How it runs

It does not run on its own; it is embedded. Add the Rust crate or the Python bindings to your application, register tables or a catalog in code, and execute SQL or DataFrame plans in-process on a Tokio thread pool. Comet accelerates Spark with it; Ballista distributes it.

## Details

### General

| Attribute | Value |
| --- | --- |
| License | Open source (permissive) |
| SPDX identifier | Apache-2.0 |
| Deployment | Self-hosted |
| Workload | Interactive, Batch |
| Operational complexity | Low |
| Pricing | Free |
| Language | Rust |
| Repository | https://github.com/apache/datafusion |
| Documentation | https://datafusion.apache.org/user-guide/introduction.html |
| Stars | 9.3k |

### Query Engines

| Attribute | Value |
| --- | --- |
| Engine shape | Embedded Rust library: a component, not a product |
| Non-lake sources | None in core; datafusion-contrib crates add Postgres, MySQL, SQLite, DuckDB and Flight table providers, with pushdown through datafusion-federation |
| Table formats | CSV, Parquet, JSON, Avro and Arrow built in; Iceberg and Delta only through external crates |
| Catalog required | Nothing required: CatalogProvider is a trait the embedding application fills, and most just register tables directly |
| SQL dialect | Its own, parsed by datafusion-sqlparser-rs, generic with PostgreSQL features added on request, no compatibility target |
| Client access | In-process from Rust or Python, through SQL or a DataFrame API; datafusion-cli for a shell. No wire protocol, so no JDBC or ODBC |
| Concurrent users | Whatever the host process gives it, a Tokio thread pool, with no queueing, admission control or multi-tenancy |
| Acceleration | Every query scans again; the only cache is an opt-in session cache of file statistics and orderings, not of results |
| Failure recovery | None: a failed query fails; Ballista adds a distributed scheduler as a separate project |
| Access control | None: whatever the embedding application enforces |

## Capabilities

- [Query Engines](https://matca.io/capabilities/query-engine)

---

Source: https://matca.io/apache-datafusion
Last updated: 2026-08-31T11:40:47.792Z