Apache DolphinScheduler
Apache DolphinScheduler is a distributed workflow orchestration platform with a drag-and-drop DAG editor, a decentralised multi-master and multi-worker architecture built for high availability, and built-in task types for the tools a data platform already runs.
LicenseOpen source (permissive)· Apache-2.0
DeploymentSelf-hosted
PricingFree
Operational complexityHigh
WorkloadBatch
LanguageJava
Use it when
- Pipeline authors are not all engineers; workflows are dragged onto a canvas, with a Python SDK and an open API for those who prefer code.
- Scheduler availability is non-negotiable: the decentralised multi-master, multi-worker design has no single point of failure.
- The built-in task types (Spark, Flink, SeaTunnel, DataX, and 24 datasource types) match what your platform already runs.
- You want pause, resume, backfill, multi-tenancy, and permissions in the box, self-hosted for free.
Think twice when
- You want a managed service; there is none, and the JVM cluster is yours to operate.
- The team prefers code-first authoring, or leans on Western community gravity; adoption and documentation weight sit heavily in China.
- Data passing between tasks matters; parameters flow on only a subset of task types, and there is no result caching.
How it runs
A self-hosted Java platform deployed standalone, as a cluster, in Docker, or on Kubernetes: decentralised masters assign task nodes to worker thread pools, cron timings attached to workflow definitions drive runs, and failover resubmits tasks from lost workers. An Apache Software Foundation project, Apache-2.0, with no paid tier.
Details
CompareHow Apache DolphinScheduler answers the questions Workflow Orchestration turns on.
Workflow Orchestration
| How it works | |
| Authoring model | Drag tasks onto a visual canvas; a Python SDK and an open API alongside it |
| Unit of work | A task node on a workflow definition, which is versioned and has to be brought online before it can run; sub-workflow nodes nest one definition inside another, and a workflow instance is one run of it |
| Passing data | Parameters, not data: an upstream task declares an OUT parameter and the downstream one reads it, but only on eight task types out of thirty-six. FILE parameters are the exception that moves a real file or folder from an upstream working directory to a downstream task in the same instance. Priority rules settle name collisions, and the behaviour changed in 3.3.x |
| Triggers | A cron timing attached to a workflow definition, and brought online separately from the workflow, so a live workflow with an offline timing simply never runs; complement runs backfill a date range and can cascade to dependent downstream workflows; a Dependent node gates on another workflow having succeeded in a given period; otherwise manual runs and the open API |
| Delivery guarantee | Failover: on worker loss a master takes the task over and resubmits it |
| Caching and reruns | None: no result cache and no cache field on a task definition at all. Reruns are operational instead: rerun a finished instance whole, recover a failed one starting from the node that failed, or resume a paused one from where it stopped |
| Running it | |
| Where tasks run | WorkerServer thread pools, assigned by decentralised masters |
| Connections | |
| Task languages | Shell, Python, Java and SQL as first-class task types, plus stored procedures and a remote shell over SSH; everything else is a plugin that hands work to another engine rather than running code here |
| Connectors | 24 datasource types behind SQL tasks (MySQL, PostgreSQL, Oracle, SQL Server, DB2, Hana, Dameng, OceanBase, Vertica, Hive, Kyuubi, Spark, Presto, Trino, ClickHouse, Doris, StarRocks, Databend, DolphinDB, Redshift, Athena, Snowflake, Azure SQL and SSH) beside 36 task plugins covering Spark, Flink, SeaTunnel, DataX, Sqoop, EMR, DMS, DataSync, SageMaker, MLflow, Kubeflow, Dinky and Linkis |
| Cost | |
| Billing unit | Nothing to pay: an ASF project with no paid tier of its own. WhaleOps, founded by its original creators, sells WhaleStudio on top of it and Apache SeaTunnel, but that is a separate product rather than a hosted DolphinScheduler |
Alternatives to Apache DolphinScheduler
