Prefect
Prefect is an open-source orchestration framework where plain Python functions become flows and tasks, aimed at dynamic, failure-tolerant pipelines with minimal boilerplate.
LicenseOpen source (permissive)· Apache-2.0
DeploymentSelf-hostedManagedServerlessHybrid
PricingFreeSubscription
Operational complexityLow
WorkloadBatch
LanguagePython, TypeScript
Use it when
- Workflows are dynamic Python: branching, mapping, and runtime-generated work that a static DAG model fights.
- Adoption friction matters; @flow and @task on ordinary functions is the whole authoring model.
- Caching and retries should be first-class: cache policies compose over inputs, source, and parameters.
- Execution must span infrastructures; workers poll work pools on processes, Docker, Kubernetes, ECS, or Cloud Run.
Think twice when
- You want the biggest integration catalog; Prefect's twenty-odd libraries are the shortest list among the Python orchestrators, against Airflow's 82 provider packages.
- Asset-level lineage and a data catalog are the point; Dagster models that directly.
- Webhooks from outside systems must reach the free self-hosted server; they are Cloud-only.
- Seat-based Cloud tiers with bundled compute minutes fit heavy usage awkwardly.
How it runs
Self-host the Apache-2.0 server or use Prefect Cloud. Deployments bind a flow to a work pool and a schedule (cron, interval, or RRule), automations react to events and metric thresholds, and workers execute wherever they run; push pools need no worker at all. Results stay in memory unless persistence is enabled, which caching and retries require.
Details
CompareHow Prefect answers the questions Workflow Orchestration turns on.
Workflow Orchestration
| How it works | |
| Authoring model | Python: @flow and @task decorators on ordinary functions |
| Unit of work | A task inside a flow, with the deployment as the thing you actually schedule, a flow bound to a work pool, a schedule and its parameters. A subflow is one flow calling another, so there is no separate composition object to learn |
| Passing data | Return values stay in memory and pass straight to the next call, because a flow is an ordinary Python program. Persisting them is opt-in (persist_result, or naming a storage or serializer) writing pickle or JSON, compressed or not, to ~/.prefect/storage by default or to S3, GCS or Azure. Retries and caching both require it |
| Triggers | A deployment schedule as cron, an interval or an RRule, plus automations firing on events (flow run state changes, work pool or deployment status, metric thresholds like average duration or lateness, or a custom event) and on the absence of an expected event. Webhooks for outside events are Cloud-only |
| Delivery guarantee | Retries on both tasks and flows · with backoff and a custom retry condition |
| Caching and reruns | An algebra rather than a switch, cache policies keyed on INPUTS, TASK_SOURCE, RUN_ID or FLOW_PARAMETERS, added together for stricter invalidation or subtracted to ignore a parameter, with cache_key_fn for anything custom and cache_expiration as a timedelta. It needs result persistence on, and cache records sit beside the results unless given their own key storage |
| Running it | |
| Where tasks run | Workers poll a work pool, process, Docker, Kubernetes, ECS, Cloud Run; push pools need no worker |
| Connections | |
| Task languages | Python only: the decorators wrap Python functions and there is no second SDK. Anything else is shelled out through prefect-shell, or run as a container or Kubernetes job through prefect-docker and prefect-kubernetes |
| Connectors | About twenty integration libraries, seventeen maintained by Prefect (AWS, Azure, GCP, Databricks, Snowflake, dbt, Docker, Kubernetes, Dask, Ray, SQLAlchemy, Slack, Email, GitHub, GitLab, Bitbucket and Shell) with Coiled, Fivetran and Slurm from outside. The shortest catalogue of any row here that has one |
| Cost | |
| Billing unit | Prefect Cloud bills seats in tiers (Hobby free for 2 users and 5 deployments, Starter $100 a month for 3, Team $100 per user for 4 to 8, Enterprise custom) with managed serverless compute bundled per tier at 500 minutes, 75 hours and 225 hours a month rather than metered |
Alternatives to Prefect
