Use it when

You operate an existing Hadoop estate and the question is running it well rather than replacing it.
Data must stay on-premises on commodity hardware, with HDFS replication, snapshots, and erasure coding providing the durability.
YARN is your cluster's resource manager, and Spark, Hive, and the rest of the stack still schedule on it.
Compliance or latency rules out object storage, and a colocated storage-plus-compute cluster is the actual requirement.

Think twice when

You are starting anything new; object storage plus Spark, Trino, or a warehouse is the modern default, and even Hadoop's own documentation no longer presents MapReduce as the primary engine.
The team is small; a NameNode, DataNodes, ResourceManager, and NodeManagers are a full operations job before the first job runs.
Elasticity matters; nodes are commissioned and decommissioned by hand, and HDFS wants rebalancing afterwards.

How it runs

Self-hosted as a cluster: HDFS (NameNode plus DataNodes) for storage, YARN (ResourceManager plus NodeManagers) for scheduling, and MapReduce or, in practice, Spark for execution. High availability needs explicit configuration. Managed derivatives such as EMR and Dataproc run the same stack with the assembly done for you.

Details

Compare

How Apache Hadoop answers the questions Batch Compute turns on.

Batch Compute

How it works

APIs

MapReduce in Java; Streaming for other languages

Data model

Key-value records over HDFS files

Execution model

Map and reduce phases, materialised to disk between them

Larger than memory

Disk-based by design: map output spills through a sort buffer and merges on disk, so a job is bounded by disk rather than memory

Running it

Runtime

JVM

GPU support

YARN schedules NVIDIA GPUs and FPGAs, isolated with cgroups or nvidia-docker, but MapReduce itself has no use for them

What you run

A full cluster before a single job runs, HDFS NameNode and DataNodes, plus a YARN ResourceManager and NodeManagers

Scaling

NodeManagers are commissioned and decommissioned by hand; there is no autoscaler, and HDFS wants rebalancing afterwards

Failure recovery

The reason it existed: tasks retry four times, speculative execution is on by default, and HDFS keeps three replicas; the NameNode is a single point of failure until HA is configured

Connections

Table formats

Text and SequenceFile natively; Avro, Parquet and ORC through their own InputFormats; no lakehouse format

Alternatives in Batch Compute

View all technologies

Apache Spark

Unified engine for large-scale data processing

vs Apache Hadoop: Batch · Streaming · Scala, Python, Java

Apache Spark is an open-source distributed compute engine for batch processing, SQL, machine learning, and stream processing across large clusters.

Daft

Rust data engine for multimodal and AI workloads

vs Apache Hadoop: Self-hosted · Free · Operational complexity: Medium · Rust

Daft is a Rust-based data engine with a Python interface, processing images, audio, video, embeddings, and structured data through one API, and scaling from a laptop to a cluster without code changes.

Dask

Parallel pandas and NumPy, from one machine to a cluster

vs Apache Hadoop: Self-hosted · Managed · Hybrid · Operational complexity: Medium · Python

Dask is a Python library for parallel and distributed computing that provides pandas-compatible DataFrames and NumPy-compatible Arrays over datasets larger than memory, running locally or across a cluster.