Processing & Transformation

SQLMesh is a data transformation framework from Tobiko Data that runs SQL and Python models with a plan-and-apply workflow, column-level lineage, unit tests, and virtual data environments that let teams preview changes without rebuilding tables.

Dataform is a service for developing, testing, version-controlling, and scheduling SQL transformation workflows in BigQuery, built on an Apache-2.0 meta-language and offered on Google Cloud at no charge beyond the BigQuery it runs.

Coalesce is a commercial transformation platform that builds warehouse pipelines from reusable metadata-driven nodes and templates rather than hand-written SQL, with column-level lineage, cataloging, and quality checks in one managed service.

Bruin is an open-source data pipeline tool written in Go that combines ingestion, SQL and Python transformations, automated quality checks, and column-level lineage in a single CLI, with a managed cloud layer above it.

Presto is a distributed SQL query engine for federated analytics, governed by the Presto Foundation under the Linux Foundation, whose members include Meta, IBM, Uber, and Nvidia — the original codebase that PrestoSQL forked from before renaming itself Trino.

Apache DataFusion is an extensible query engine written in Rust that uses Apache Arrow as its in-memory format, shipped as libraries for building database and analytic systems rather than as a server to run.

Polars is a DataFrame library written in Rust with a Python interface, built around a query optimiser and Arrow memory rather than row-by-row execution, and designed as a single-node engine that handles data most teams would otherwise put on a cluster.

Amazon Athena is a serverless interactive query service that runs SQL directly against data in Amazon S3, with no infrastructure to provision and a bill computed from the data each query scans.

Starburst is the commercial platform built on Trino, sold as Starburst Enterprise for self-managed clusters and Starburst Galaxy as a fully managed service, adding governance, performance work, and a supported Iceberg lakehouse on top of the open-source engine.

Dremio is a SQL engine for the lakehouse that federates queries across Iceberg, Delta Lake, databases, and warehouses, and accelerates them with materialised structures it calls Reflections — available as a free Community edition, a self-managed Enterprise edition, and a managed cloud service.

Apache Hive is a distributed, fault-tolerant data warehouse system that enables analytics at massive scale, running SQL over data in the Hadoop ecosystem on Apache Tez, with LLAP for sub-second queries and first-class Apache Iceberg support in the 4.x line.

Ray is an open-source framework for scaling Python and AI workloads, pairing a general-purpose distributed execution core with libraries for data, training, tuning, serving, and reinforcement learning.

Dask is a Python library for parallel and distributed computing that provides pandas-compatible DataFrames and NumPy-compatible Arrays over datasets larger than memory, running locally or across a cluster.

Daft is a Rust-based data engine with a Python interface, processing images, audio, video, embeddings, and structured data through one API, and scaling from a laptop to a cluster without code changes.

Apache Hadoop is the distributed storage and batch processing framework that started the category, comprising HDFS for storage, YARN for resource management and scheduling, and MapReduce as its batch execution engine.

Apache Spark is an open-source distributed compute engine for batch processing, SQL, machine learning, and stream processing across large clusters.

dbt is a transformation framework that compiles modular, version-controlled SQL models and runs them inside the warehouse, with testing, documentation, and lineage built in.

Trino is an open-source distributed SQL engine that queries data where it lives — object storage, lakehouse table formats, and operational databases — through a single interface.

DuckDB is an open-source, in-process OLAP database — an embeddable columnar SQL engine that queries Parquet, CSV, and dataframes with no server to run.