# Apache Beam

> Portable pipeline model that runs on Flink, Spark, or Dataflow

Apache Beam is a unified programming model for batch and streaming pipelines, written once in Java, Python, or Go and executed by a runner — Flink, Spark, Dataflow, or others — rather than by an engine of its own.

Website: https://beam.apache.org

## Use it when

- The execution engine is genuinely undecided or expected to change; one pipeline runs on Flink, Spark, Dataflow, and others.
- Batch and streaming should share one codebase; the model treats batch as the bounded case of streaming.
- You need portable pipelines in Java, Python, or Go, with over 60 I/O connectors reachable cross-language.
- Fine-grained control matters: per-key state, event-time and processing-time timers, watermarks with triggers and allowed lateness.

## Think twice when

- The engine decision is already made and stable; the abstraction is then overhead over writing Flink or Spark directly.
- Guarantees must be uniform; exactly-once and performance belong to the runner, and a pipeline may behave differently across runners.
- You want a serving layer or SQL-first authoring; results leave through sinks, and the SQL and YAML surfaces are secondary to the SDKs.

## How it runs

Beam runs nothing itself: pipelines execute on a runner (Dataflow, Flink, Spark, and others, with Prism for local runs). Scaling, state backends, and delivery guarantees are the runner's, so Dataflow autoscales while Flink rescales from a savepoint. Free and Apache-2.0; the cost profile is whatever the chosen runner costs.

## Details

### General

| Attribute | Value |
| --- | --- |
| License | Open source (permissive) |
| SPDX identifier | Apache-2.0 |
| Deployment | Self-hosted, Managed |
| Workload | Streaming, Batch |
| Operational complexity | Medium |
| Pricing | Free |
| Language | Java, Python, Go |
| Repository | https://github.com/apache/beam |
| Documentation | https://beam.apache.org/documentation/ |
| Stars | 8.7k |

### Stream Processing

| Attribute | Value |
| --- | --- |
| Processing model | A portable model, not an engine, one pipeline covering batch and streaming, translated by a runner |
| APIs | Java, Python and Go SDKs, TypeScript under development; SQL and declarative YAML |
| What you run | Nothing of its own: a runner: Dataflow, Flink, Spark, Nemo, Hazelcast Jet or Twister2, with Prism for local runs |
| Connectors | Over 60 built-in I/O connectors, Kafka, Pub/Sub, Kinesis, JDBC, BigQuery, Iceberg, Delta Lake, files; Java has all of them, the other SDKs reach them cross-language |
| Windowing | Fixed, sliding, session, global; calendar in Java only |
| Event time and lateness | Watermarks with triggers, configurable allowed lateness and accumulation modes; late data fires the window again rather than being dropped |
| State backend | State API per key; the backend is the runner's |
| Delivery guarantee | The runner's: exactly-once on Dataflow and Flink |
| Reading results | Through sinks only: there is no serving layer and no engine to query |
| Scaling | The runner's: Dataflow autoscales, Flink rescales from a savepoint; Beam itself has no notion of capacity |

## Capabilities

- [Stream Processing](https://matca.io/capabilities/stream-processing)

---

Source: https://matca.io/apache-beam
Last updated: 2026-08-31T11:40:48.361Z