Apache Beam
Apache Beam is a unified programming model for batch and streaming pipelines, written once in Java, Python, or Go and executed by a runner — Flink, Spark, Dataflow, or others — rather than by an engine of its own.
LicenseOpen source (permissive)· Apache-2.0
DeploymentSelf-hostedManaged
PricingFree
Operational complexityMedium
WorkloadStreamingBatch
LanguageJava, Python, Go
Use it when
- The execution engine is genuinely undecided or expected to change; one pipeline runs on Flink, Spark, Dataflow, and others.
- Batch and streaming should share one codebase; the model treats batch as the bounded case of streaming.
- You need portable pipelines in Java, Python, or Go, with over 60 I/O connectors reachable cross-language.
- Fine-grained control matters: per-key state, event-time and processing-time timers, watermarks with triggers and allowed lateness.
Think twice when
- The engine decision is already made and stable; the abstraction is then overhead over writing Flink or Spark directly.
- Guarantees must be uniform; exactly-once and performance belong to the runner, and a pipeline may behave differently across runners.
- You want a serving layer or SQL-first authoring; results leave through sinks, and the SQL and YAML surfaces are secondary to the SDKs.
How it runs
Beam runs nothing itself: pipelines execute on a runner (Dataflow, Flink, Spark, and others, with Prism for local runs). Scaling, state backends, and delivery guarantees are the runner's, so Dataflow autoscales while Flink rescales from a savepoint. Free and Apache-2.0; the cost profile is whatever the chosen runner costs.
Details
CompareHow Apache Beam answers the questions Stream Processing turns on.
Stream Processing
| How it works | |
| Processing model | A portable model, not an engine, one pipeline covering batch and streaming, translated by a runner |
| APIs | Java, Python and Go SDKs, TypeScript under development; SQL and declarative YAML |
| Windowing | Fixed, sliding, session, global; calendar in Java only |
| Event time and lateness | Watermarks with triggers, configurable allowed lateness and accumulation modes; late data fires the window again rather than being dropped |
| State backend | State API per key; the backend is the runner's |
| Delivery guarantee | The runner's: exactly-once on Dataflow and Flink |
| Reading results | Through sinks only: there is no serving layer and no engine to query |
| Running it | |
| What you run | Nothing of its own: a runner: Dataflow, Flink, Spark, Nemo, Hazelcast Jet or Twister2, with Prism for local runs |
| Scaling | The runner's: Dataflow autoscales, Flink rescales from a savepoint; Beam itself has no notion of capacity |
| Connections | |
| Connectors | Over 60 built-in I/O connectors, Kafka, Pub/Sub, Kinesis, JDBC, BigQuery, Iceberg, Delta Lake, files; Java has all of them, the other SDKs reach them cross-language |
Alternatives to Apache Beam
