Apache Spark is an open-source distributed compute engine for batch processing, SQL, machine learning, and stream processing across large clusters.

DuckDB is an open-source, in-process OLAP database — an embeddable columnar SQL engine that queries Parquet, CSV, and dataframes with no server to run.

Apache Airflow is the widely adopted open-source orchestrator where pipelines are Python-defined DAGs, with a large provider ecosystem and multiple managed offerings.

Dagster is an open-source orchestrator organized around software-defined assets — declaring the tables and models pipelines produce — with strong local development and testing.

Prefect is an open-source orchestration framework where plain Python functions become flows and tasks, aimed at dynamic, failure-tolerant pipelines with minimal boilerplate.

Great Expectations (GX) is an open-source Python framework for declaring, validating, and documenting data quality expectations inside pipelines.