Dask
Dask is a Python library for parallel and distributed computing that provides pandas-compatible DataFrames and NumPy-compatible Arrays over datasets larger than memory, running locally or across a cluster.
LicenseOpen source (permissive)· BSD-3-Clause
DeploymentSelf-hostedManagedHybrid
PricingFreeSubscription
Operational complexityMedium
WorkloadBatch
LanguagePython
Use it when
- The team lives in pandas and NumPy and needs the same APIs to survive past RAM and onto multiple cores or nodes.
- Work runs on HPC schedulers such as SLURM or PBS; no peer engine targets supercomputing environments the way Dask does.
- Custom parallel pipelines need a Futures and Delayed API rather than a fixed relational model.
- Array and scientific workloads matter as much as tables.
Think twice when
- The pipelines are relational at heart; Spark's optimizer and ecosystem go deeper, and DuckDB or Polars beat it on a single machine.
- The scheduler keeps no persistence and scattered data dies with its worker; if the scheduler goes, running computations go with it.
- pandas semantics at scale still surprise; partition-aware thinking is required despite the familiar API.
How it runs
A scheduler and workers: LocalCluster inside one machine, dask-jobqueue on SLURM or PBS, a Kubernetes operator, or Coiled as the managed option. Adaptive scaling sizes the cluster from pending work; workers spill to disk under memory pressure and pause before dying. RAPIDS (cuDF, CuPy) supplies the GPU execution.
Details
CompareHow Dask answers the questions Batch Compute turns on.
Batch Compute
| How it works | |
| APIs | pandas and NumPy APIs; Delayed and Futures for custom pipelines |
| Data model | DataFrames and arrays mirroring pandas and NumPy |
| Execution model | Dynamic task graph, scheduled per task; DataFrames have run through a query optimizer since 2024.03 |
| Larger than memory | Workers spill least-recently-used data to disk from 60% managed memory, pause at 80% and are terminated at 95% |
| Running it | |
| Runtime | Native Python |
| GPU support | Not Dask's own: it schedules Python functions and RAPIDS supplies the GPU ones, cuDF and CuPy, with Dask-CUDA managing devices |
| What you run | A scheduler and workers: a LocalCluster on one machine, the Kubernetes operator, dask-jobqueue on SLURM or PBS, or Coiled |
| Scaling | cluster.adapt() sizes from pending task runtime against a five-second target; a retiring worker moves its data elsewhere first |
| Failure recovery | Tasks rerun elsewhere and lost results rebuild from the graph, but scattered data dies with the worker and the scheduler has no persistence, if it goes, running computations go with it |
| Connections | |
| Table formats | Parquet, ORC, CSV, JSON, HDF5 and SQL; no lakehouse format natively, Delta needs the third-party dask-deltatable |
Alternatives to Dask
