Apache Spark
Apache Spark is an open-source distributed compute engine for batch processing, SQL, machine learning, and stream processing across large clusters.
LicenseOpen source (permissive)· Apache-2.0
DeploymentSelf-hostedManaged
PricingFreeSubscription
Operational complexityHigh
WorkloadBatchStreaming
LanguageScala, Python, Java
Use it when
- Transformations run over data too large for one machine and need a mature, battle-tested distributed engine.
- One codebase should cover batch, SQL, micro-batch streaming, and ML across Python, Scala, Java, or R.
- You are building on Iceberg, Delta, or Hudi; Spark is the default write path in most lakehouse platforms.
- Long jobs must survive failures; lineage-based recovery and task retry are the model, not an add-on.
Think twice when
- The data fits one node; DuckDB or Polars finish before a cluster has scheduled, with none of the operations.
- Event-at-a-time streaming with strict latency is the job; Flink's model fits where micro-batch does not.
- Nobody owns cluster tuning; executor sizing, shuffle, and memory knobs are real work, which is what EMR, Dataproc, and Databricks sell relief from.
How it runs
A driver coordinates executors on Standalone, YARN, or Kubernetes; jobs form a DAG of stages with in-memory execution and disk spill. Run it self-managed, through EMR or Dataproc, or on Databricks. It reads and writes Parquet, ORC, Avro, CSV, and JSON natively, and the lakehouse formats through DataSource V2.
Details
How Apache Spark answers the questions each capability turns on.
Batch Compute
| How it works | |
| APIs | RDDs, DataFrames and Datasets, SQL; Spark Connect client |
| Data model | Tabular DataFrames over rows; RDDs of objects |
| Execution model | DAG of stages separated by shuffles; in-memory |
| Larger than memory | Spills to disk: MEMORY_AND_DISK storage levels and shuffle spill, so a job is not bounded by RAM |
| Running it | |
| Runtime | JVM; Python over Arrow-based IPC |
| GPU support | GPUs as a scheduled resource since 3.0; SQL and DataFrame offload needs the RAPIDS plugin |
| What you run | A Spark cluster: a driver and executors on Standalone, YARN or Kubernetes |
| Scaling | Dynamic allocation adds and removes executors, off by default and needing the external shuffle service; decommissioning migrates blocks first |
| Failure recovery | Lost partitions are recomputed from lineage; tasks and stages retry, and shuffle files are kept so a retry need not redo them |
| Connections | |
| Table formats | Parquet, ORC, Avro, CSV and JSON natively; Iceberg, Delta Lake and Hudi through DataSource V2 |
Stream Processing
| How it works | |
| Processing model | Micro-batch by default; real-time mode since 4.1 runs a long-lived task per partition; continuous processing stays experimental |
| APIs | Structured Streaming (DataFrame/Dataset), SQL; declarative pipelines in SQL or Python |
| Windowing | Tumbling, sliding, session |
| Event time and lateness | Watermarks via withWatermark; rows later than the threshold are dropped, no side outputs |
| State backend | RocksDB, HDFS-backed (default) |
| Delivery guarantee | Exactly-once in micro-batch · sink-dependent; continuous processing is At-least-once |
| Reading results | Through sinks; the memory sink is for debugging only |
| Running it | |
| What you run | A Spark cluster: a driver and executors on Standalone, YARN or Kubernetes |
| Scaling | spark.sql.shuffle.partitions fixes the state partition count, changing it on a stateful query means discarding the checkpoint; stateless queries can change it since 4.1 |
| Connections | |
| Connectors | Kafka and files built in; everything else through foreachBatch or a DataSource V2 connector such as Delta, Iceberg or Kinesis |
Alternatives to Apache Spark
