Luigi
Luigi is Spotify's Python library for building pipelines of batch jobs: tasks declare what they require and what they produce, a central scheduler resolves the graph and prevents duplicate work, and triggering is left to cron.
LicenseOpen source (permissive)· Apache-2.0
DeploymentSelf-hosted
PricingFree
Operational complexityLow
WorkloadBatch
LanguagePython
Use it when
- Dependency-aware batch execution is all you need and cron already triggers everything.
- The idempotency model appeals: a task is complete when its output exists, so reruns skip finished work naturally.
- You want the smallest possible footprint: a Python package, no metadata database, no services beyond an optional coordinator.
- A decade of continuous maintenance with no vendor attached is a feature, not a gap.
Think twice when
- Anyone expects built-in triggering, retries with backoff, web-based authoring, or a managed edition; Luigi deliberately has none of that, and Airflow or Prefect do.
- Pipelines need event-driven runs, dynamic fan-out, or passing values between tasks; everything travels as files or tables.
- The team will grow into platform needs; there is no roadmap toward them.
How it runs
pip install luigi: tasks are Python classes declaring requires(), run(), and output(), and workers you start yourself (typically from cron) resolve the graph. The optional luigid coordinator prevents two workers running the same task and serves the graph UI. Outputs are Targets: files on local disk, S3, HDFS, or GCS, or database tables. Apache-2.0, no vendor, nothing to pay.
Details
CompareHow Luigi answers the questions Workflow Orchestration turns on.
Workflow Orchestration
| How it works | |
| Authoring model | Python: Task classes declaring requires(), run() and output() |
| Unit of work | A Task class with requires(), run() and output(). The DAG is implied by what requires() returns and never written down anywhere, so there is no workflow object to deploy, parameters are what make two instances of a task distinct |
| Passing data | None: tasks share Targets, not values. output() returns a Target, input() hands the upstream one to run(), and everything travels as a file on local disk, S3, HDFS or GCS, or as a database table; requires() explicitly cannot return a Target |
| Triggers | Nothing built in, deliberately: the docs state that "there is no central process that automatically triggers jobs" and point at cron or a long-running process. luigid neither executes nor parallelises; it stops two instances of a task running at once and draws the graph |
| Delivery guarantee | The scheduler stops two instances of the same task running at once · and a task whose output exists is already complete |
| Caching and reruns | No cache, and none wanted, a task is complete when its output Target exists, so a rerun skips whatever is already built and deleting the target is how you force one. It is the whole idempotency model rather than a feature added to it |
| Running it | |
| Where tasks run | Worker processes you start yourself; luigid only coordinates them |
| Connections | |
| Task languages | Python for the task itself and nothing else; everything else is shelled out, ExternalProgramTask runs a command, and contrib modules submit Hadoop jars, Spark and PySpark, Hive, Pig, Scalding, Docker containers, Kubernetes jobs and SSH commands |
| Connectors | Forty-eight contrib modules in the tree, S3, GCS, Azure Blob, HDFS and WebHDFS, Hadoop, Hive, Pig, Spark, Scalding, Presto, BigQuery, Redshift, Postgres, MySQL, MS SQL, SQLAlchemy, MongoDB, Redis, Elasticsearch, Salesforce, Dropbox, FTP, SSH, Kubernetes, Docker, AWS Batch and ECS, SGE and LSF, Datadog and Prometheus |
| Cost | |
| Billing unit | Nothing to pay and nobody to pay it to, Apache-2.0 with no vendor, no hosted edition and no paid tier of any kind; the cost is the machine you run it on |
Alternatives to Luigi
