Batch Compute Engines
Data architecture technologies for distributed engines for heavy, general-purpose data processing beyond SQL.
Distributed execution for Python, from laptop to cluster
Batch Compute
Ray is an open-source framework for scaling Python and AI workloads, pairing a general-purpose distributed execution core with libraries for data, training, tuning, serving, and reinforcement learning.
Parallel pandas and NumPy, from one machine to a cluster
Batch Compute
Dask is a Python library for parallel and distributed computing that provides pandas-compatible DataFrames and NumPy-compatible Arrays over datasets larger than memory, running locally or across a cluster.
Rust data engine for multimodal and AI workloads
Batch Compute
Daft is a Rust-based data engine with a Python interface, processing images, audio, video, embeddings, and structured data through one API, and scaling from a laptop to a cluster without code changes.
The original distributed batch stack — HDFS, YARN, MapReduce
Batch Compute
Apache Hadoop is the distributed storage and batch processing framework that started the category, comprising HDFS for storage, YARN for resource management and scheduling, and MapReduce as its batch execution engine.
Unified engine for large-scale data processing
Batch ComputeStream Processing
Apache Spark is an open-source distributed compute engine for batch processing, SQL, machine learning, and stream processing across large clusters.