Dagster
Dagster is an open-source orchestrator organized around software-defined assets — declaring the tables and models pipelines produce — with strong local development and testing.
LicenseOpen source (permissive)· Apache-2.0
DeploymentSelf-hostedManagedServerlessHybrid
PricingFreeSubscription
Operational complexityMedium
WorkloadBatch
LanguagePython, TypeScript
Use it when
- Orchestration should be organised around the tables, files, and models pipelines produce, with lineage and a catalog built in rather than bolted on.
- dbt is central; models appear as assets beside ingestion and ML steps.
- Local development and unit-testable pipelines matter to the engineering culture.
- Freshness-driven and event-driven materialisation (sensors, Declarative Automation) fits better than pure cron.
Think twice when
- The organisation already runs deep on Airflow, whose operator ecosystem and hiring pool remain larger.
- Workflows are generic task graphs with no data assets to model; the asset layer is overhead then.
- Dagster+ pricing is denominated in credits per materialisation and ops executed, which needs forecasting on large graphs.
How it runs
Open-source Dagster self-hosts with an executor per run (multiprocess by default, or Kubernetes, Docker, Celery, ECS). I/O managers persist asset outputs to filesystems, object stores, or warehouses, and Dagster Pipes hands heavy work to Spark, Databricks, or other runtimes while streaming logs back. Dagster+ is the managed control plane in serverless or hybrid form, billed in credits plus compute.
Details
CompareHow Dagster answers the questions Workflow Orchestration turns on.
Workflow Orchestration
| How it works | |
| Authoring model | Python: assets and ops, with the asset the primary unit |
| Unit of work | An asset (a named thing in storage that a function produces) with ops and jobs underneath it for work that produces nothing, plus asset checks and partitions. The loudest asset-first claim here, and the one the billing model is denominated in |
| Passing data | Return a value and an I/O manager persists it, then loads it as the downstream argument, pickles on the filesystem by default, or S3, GCS, ADLS2, Snowflake, BigQuery, DuckDB and ClickHouse, with pandas, PySpark and Polars variants. Skip it and every asset does its own reading and writing |
| Triggers | Cron schedules, sensors polling for custom events, asset sensors firing on a materialization, Declarative Automation evaluating conditions on assets and checks, freshness policies, and the GraphQL endpoint for anything external |
| Delivery guarantee | Retries via RetryPolicy: max attempts · delay · backoff and jitter |
| Caching and reruns | Staleness, not an input hash, a code_version on the asset and data versions on what it read tell Dagster whether re-materialising would change anything, so it can skip the work and keep the last value. The pre-1.0 memoization machinery was removed as superseded. Retries re-execute from failure by default, skipping successful ops and reusing their outputs |
| Running it | |
| Where tasks run | An executor per run: multiprocess by default, or Kubernetes, Docker, Celery or ECS |
| Connections | |
| Task languages | Python defines it; Dagster Pipes runs the real work elsewhere in another language and streams logs and metadata back, subprocess, Kubernetes, Databricks, PySpark, Scala on Spark, JavaScript, Azure ML and Dataproc |
| Connectors | 60+ integration libraries, some maintained by Dagster and some by the community, dbt, Airbyte, Fivetran, Sling, dlt and Meltano beside Snowflake, BigQuery, Databricks, Delta Lake and Iceberg, the three clouds, Kubernetes, Celery, Dask and Ray, Looker and Tableau, and an AI shelf of OpenAI, Anthropic and the vector stores |
| Cost | |
| Billing unit | Dagster+ bills credits plus serverless compute, $0.040 a credit on Solo at $10 a month, $0.035 on Starter at $100, and $0.010 a serverless minute, with hybrid deployments charged no compute at all. A credit is asset materializations plus ops executed, so the bill is denominated in the same unit as the programming model |
Alternatives to Dagster
