Open-source data integration with a large connector catalog
Ingestion
Airbyte is a data integration platform that syncs data from APIs, databases, and files into warehouses and lakes, available self-hosted or as a managed cloud service.
Managed cloud data warehouse with separated storage and compute
Storage
Snowflake is a commercial cloud data platform offering elastic SQL warehouses, cross-account data sharing, and multi-cloud availability as a fully managed service.
BigQuery is Google Cloud's serverless, fully managed data warehouse with on-demand or capacity-based pricing, streaming ingestion, and built-in ML functions.
Column-oriented OLAP database for real-time analytics
Storage
ClickHouse is an open-source columnar database designed for low-latency analytical queries over large volumes, available self-hosted or as ClickHouse Cloud.
Kafka-compatible streaming platform in a single binary
Streaming
Redpanda is a Kafka API-compatible streaming platform written in C++, shipped as a single binary without ZooKeeper or a JVM, available self-hosted or as a managed cloud.
Apache Airflow is the widely adopted open-source orchestrator where pipelines are Python-defined DAGs, with a large provider ecosystem and multiple managed offerings.
Dagster is an open-source orchestrator organized around software-defined assets — declaring the tables and models pipelines produce — with strong local development and testing.
Pythonic workflow orchestration for dynamic pipelines
Orchestration
Prefect is an open-source orchestration framework where plain Python functions become flows and tasks, aimed at dynamic, failure-tolerant pipelines with minimal boilerplate.
DataHub is an open-source metadata platform providing search, discovery, and end-to-end lineage across warehouses, pipelines, and BI tools via push-based ingestion.