Polars
Polars is a DataFrame library written in Rust with a Python interface, built around a query optimiser and Arrow memory rather than row-by-row execution, and designed as a single-node engine that handles data most teams would otherwise put on a cluster.
LicenseOpen source (permissive)· MIT
DeploymentSelf-hostedHybrid
PricingFreeSubscription
Operational complexityLow
WorkloadInteractiveBatch
LanguageRust
Use it when
- pandas is the bottleneck: Polars keeps the DataFrame model but plans lazily, runs all cores, and streams datasets bigger than memory.
- Pipelines are Python or Rust code rather than SQL, and you want a query optimizer under a programmatic API.
- One capable machine can carry the workload and you would rather skip cluster operations entirely.
- You want optional GPU execution (cuDF under the same API) for heavy plans.
Think twice when
- The team works SQL-first; pl.sql covers a subset, and DuckDB is the fuller SQL engine on the same footprint.
- Workloads are genuinely multi-node today; the library is single-node by design, and the distributed engine lives in Polars Cloud, still in beta.
- You need catalog integration and governance; Unity Catalog support is partial and there is no access control.
How it runs
Install the package and it runs inside your process, multi-threaded over Arrow memory: no server, no configuration. It reads Parquet, CSV, JSON, IPC, Avro, Delta, and Iceberg, locally or on object storage. Polars Cloud, the commercial service on AWS, spins up compute in your own account for distributed and scheduled runs.
Details
CompareHow Polars answers the questions Query Engines turns on.
Query Engines
| How it works | |
| Engine shape | Single-node library; Polars Cloud distributes above it |
| Catalog required | None needed; pl.Catalog is a Unity Catalog client that lists and scans tables, though write_table covers only Delta and Delta Sharing |
| SQL dialect | DataFrame-first: pl.sql() and SQLContext translate a SQL subset onto the lazy engine rather than implementing a dialect |
| Running it | |
| Concurrent users | One process, multi-threaded across cores; no queueing or multi-tenancy, Polars Cloud is where more than one user's work gets scheduled |
| Acceleration | No result cache; the acceleration is a GPU engine, collect(engine='gpu') runs the plan on cuDF and falls back to CPU where unsupported, in open beta and single-GPU |
| Failure recovery | None: a failed query fails, in the process that ran it |
| Connections | |
| Non-lake sources | Files and object storage directly, plus read_database over ADBC or ConnectorX; a cross-source join materialises each side first, with no federated pushdown |
| Table formats | Parquet, CSV, JSON, IPC and Avro natively; Delta and Iceberg read and write, with merge on Delta |
| Client access | In-process from Python or Rust, with R and NodeJS bindings alongside; no wire protocol, so nothing connects over JDBC or ODBC |
| Access | |
| Access control | None: whatever the calling application enforces |
Alternatives to Polars
