Apache Airflow
Apache Airflow is the widely adopted open-source orchestrator where pipelines are Python-defined DAGs, with a large provider ecosystem and multiple managed offerings.
LicenseOpen source (permissive)· Apache-2.0
DeploymentSelf-hostedManaged
PricingFree
Operational complexityHigh
WorkloadBatch
LanguagePython, TypeScript
Use it when
- You want the incumbent: the largest operator ecosystem (82 provider packages), the deepest hiring pool, and managed offerings from three vendors.
- Pipelines are Python-defined DAGs on schedules, with retries and backfills handled by the platform.
- Scheduling should react to data: asset-driven and event-driven triggers listen on Kafka, SQS, Pub/Sub, or Service Bus since 3.x.
- You may outgrow one executor; Local, Celery, Kubernetes, ECS, and Edge are pluggable.
Think twice when
- The team thinks in tables and models rather than tasks; Dagster's asset model carries that further.
- Nobody wants to run a scheduler, metadata database, and workers; operational complexity is high unless you buy MWAA, Cloud Composer, or Astronomer.
- Workflows are dynamic per-run Python rather than scheduled DAGs; Prefect is built for that shape.
- Tasks need result caching; Airflow has no memoization, only reruns.
How it runs
Self-hosted Airflow is a scheduler, webserver, metadata database, and workers under a pluggable executor. Tasks pass small values through XComs; real data goes to storage. Managed Airflow from Amazon MWAA, Google Cloud Composer, or Astronomer removes the ops, metered by each vendor; the project itself is Apache-2.0 with no paid tier.
Details
CompareHow Apache Airflow answers the questions Workflow Orchestration turns on.
Workflow Orchestration
| How it works | |
| Authoring model | Python: a DAG defined in code, from operators or @task functions |
| Unit of work | A task in a DAG, with the asset (renamed from dataset in 3.0) as the other first-class noun, and DAG versioning letting versions coexist. Task groups and dynamic task mapping replaced SubDAGs, which 3.0 removed outright |
| Passing data | XComs pushed and pulled by key through the metadata database, which the docs are explicit are for small values and not dataframes; swap in the object storage backend from common-io when they are not. 3.3 added task and asset state stores beside them |
| Triggers | Cron, timedeltas and custom timetables; asset-driven scheduling so a DAG runs when its inputs are produced, combinable with a time schedule; event-driven scheduling where an AssetWatcher listens on Kafka, SQS, Pub/Sub or Azure Service Bus; sensors and deferrable operators for waiting; plus the API, the CLI and backfills the scheduler manages itself since 3.0 |
| Delivery guarantee | Retries with idempotent tasks: the docs require it · since a task may re-run |
| Caching and reruns | None: no result cache and no memoization. Clearing a task increments its try number and resets its state so it simply runs again; catchup fills intervals that never ran and backfill replays a date range, managed by the scheduler itself since 3.0 rather than a separate command |
| Running it | |
| Where tasks run | A pluggable executor: Local by default, or Celery, Kubernetes, ECS or Edge |
| Connections | |
| Task languages | Python, and since 3.3 not only Python, the Task Execution API opened the door and Go and JVM task SDKs now ship in-tree, both experimental, the Java one at 1.0.0-beta1 and usable from any JVM language. Otherwise Bash, containers through the Docker and KubernetesPod operators, or whatever an operator can call |
| Connectors | 82 provider packages in the tree (the broadest integration surface here) spanning every major cloud, warehouse, database and SaaS the ecosystem has written an operator for, from Amazon, Google and Microsoft down to Anthropic, dbt, Databricks and Airbyte |
| Cost | |
| Billing unit | Nothing to pay for Airflow itself, an ASF project with no paid tier. The bill is whoever runs it: your own cluster, or a managed Airflow from Amazon MWAA, Google Cloud Composer or Astronomer, each metered on its own terms |
Alternatives to Apache Airflow
