Workflow Orchestration

Kestra is an open-source orchestration platform whose workflows are declared in YAML rather than written in a programming language: an event-driven, language-agnostic engine with a large plugin catalogue, a JVM runtime backed by Postgres or MySQL, and Enterprise and Cloud editions above the Apache-2.0 core.

Argo Workflows is a CNCF-graduated workflow engine implemented as a Kubernetes custom resource: each step of a DAG runs in its own container, workflows are declared in YAML and submitted to the cluster, and the scheduler is the Kubernetes control plane rather than a separate application.

Temporal is an MIT-licensed durable execution platform: workflows are ordinary code in one of eight SDK languages, and the service persists every step so a function can keep running across crashes, restarts, and deploys for minutes or for months.

AWS Step Functions is a serverless orchestrator built on state machines: each step is a state that calls an AWS service or an external worker, execution history is retained and visually debuggable, and the bill is per state transition rather than per server.

Flyte is an LF AI & Data project for building production data and ML pipelines: tasks and workflows are Python functions with strongly typed interfaces, each execution is versioned and cacheable, and the whole thing runs on Kubernetes with a Go control plane underneath.

Apache DolphinScheduler is a distributed workflow orchestration platform with a drag-and-drop DAG editor, a decentralised multi-master and multi-worker architecture built for high availability, and built-in task types for the tools a data platform already runs.

Windmill is an open-source workflow engine and developer platform written in Rust: individual scripts in TypeScript, Python, Go, SQL, Bash and more become reusable runnables, which compose into flows through a low-code builder or YAML, and into internal apps on top.

Luigi is Spotify's Python library for building pipelines of batch jobs: tasks declare what they require and what they produce, a central scheduler resolves the graph and prevents duplicate work, and triggering is left to cron.

Apache Airflow is the widely adopted open-source orchestrator where pipelines are Python-defined DAGs, with a large provider ecosystem and multiple managed offerings.

Dagster is an open-source orchestrator organized around software-defined assets — declaring the tables and models pipelines produce — with strong local development and testing.

Prefect is an open-source orchestration framework where plain Python functions become flows and tasks, aimed at dynamic, failure-tolerant pipelines with minimal boilerplate.