# Apache Spark

> Unified engine for large-scale data processing

Apache Spark is an open-source distributed compute engine for batch processing, SQL, machine learning, and stream processing across large clusters.

Website: https://spark.apache.org

## Use it when

- Transformations run over data too large for one machine and need a mature, battle-tested distributed engine.
- One codebase should cover batch, SQL, micro-batch streaming, and ML across Python, Scala, Java, or R.
- You are building on Iceberg, Delta, or Hudi; Spark is the default write path in most lakehouse platforms.
- Long jobs must survive failures; lineage-based recovery and task retry are the model, not an add-on.

## Think twice when

- The data fits one node; DuckDB or Polars finish before a cluster has scheduled, with none of the operations.
- Event-at-a-time streaming with strict latency is the job; Flink's model fits where micro-batch does not.
- Nobody owns cluster tuning; executor sizing, shuffle, and memory knobs are real work, which is what EMR, Dataproc, and Databricks sell relief from.

## How it runs

A driver coordinates executors on Standalone, YARN, or Kubernetes; jobs form a DAG of stages with in-memory execution and disk spill. Run it self-managed, through EMR or Dataproc, or on Databricks. It reads and writes Parquet, ORC, Avro, CSV, and JSON natively, and the lakehouse formats through DataSource V2.

## Details

### General

| Attribute | Value |
| --- | --- |
| License | Open source (permissive) |
| SPDX identifier | Apache-2.0 |
| Deployment | Self-hosted, Managed |
| Workload | Batch, Streaming |
| Operational complexity | High |
| Pricing | Free, Subscription |
| Language | Scala, Python, Java |
| Repository | https://github.com/apache/spark |
| Documentation | https://spark.apache.org/docs/latest/ |
| Stars | 44k |

### Batch Compute

| Attribute | Value |
| --- | --- |
| APIs | RDDs, DataFrames and Datasets, SQL; Spark Connect client |
| Data model | Tabular DataFrames over rows; RDDs of objects |
| Table formats | Parquet, ORC, Avro, CSV and JSON natively; Iceberg, Delta Lake and Hudi through DataSource V2 |
| Runtime | JVM; Python over Arrow-based IPC |
| Execution model | DAG of stages separated by shuffles; in-memory |
| Larger than memory | Spills to disk: MEMORY_AND_DISK storage levels and shuffle spill, so a job is not bounded by RAM |
| GPU support | GPUs as a scheduled resource since 3.0; SQL and DataFrame offload needs the RAPIDS plugin |
| What you run | A Spark cluster: a driver and executors on Standalone, YARN or Kubernetes |
| Scaling | Dynamic allocation adds and removes executors, off by default and needing the external shuffle service; decommissioning migrates blocks first |
| Failure recovery | Lost partitions are recomputed from lineage; tasks and stages retry, and shuffle files are kept so a retry need not redo them |

### Stream Processing

| Attribute | Value |
| --- | --- |
| Processing model | Micro-batch by default; real-time mode since 4.1 runs a long-lived task per partition; continuous processing stays experimental |
| APIs | Structured Streaming (DataFrame/Dataset), SQL; declarative pipelines in SQL or Python |
| What you run | A Spark cluster: a driver and executors on Standalone, YARN or Kubernetes |
| Connectors | Kafka and files built in; everything else through foreachBatch or a DataSource V2 connector such as Delta, Iceberg or Kinesis |
| Windowing | Tumbling, sliding, session |
| Event time and lateness | Watermarks via withWatermark; rows later than the threshold are dropped, no side outputs |
| State backend | RocksDB, HDFS-backed (default) |
| Delivery guarantee | Exactly-once in micro-batch, sink-dependent; continuous processing is At-least-once |
| Reading results | Through sinks; the memory sink is for debugging only |
| Scaling | spark.sql.shuffle.partitions fixes the state partition count, changing it on a stateful query means discarding the checkpoint; stateless queries can change it since 4.1 |

## Capabilities

- [Batch Compute](https://matca.io/capabilities/batch-compute)
- [Stream Processing](https://matca.io/capabilities/stream-processing)

---

Source: https://matca.io/apache-spark
Last updated: 2026-08-31T11:40:47.967Z