Google Cloud Dataflow
Google Cloud Dataflow is a fully managed service that executes Apache Beam pipelines for batch and streaming, provisioning and autoscaling worker VMs automatically and processing every record exactly once by default.
LicenseCommercial
DeploymentManagedServerless
PricingSubscription
Operational complexityLow
WorkloadStreamingBatch
Use it when
- You write Beam and want it run for you: workers are provisioned, autoscaled, rebalanced, and deleted per job, with nothing left running afterwards.
- Stragglers and spiky load are the pain; dynamic work rebalancing and horizontal autoscaling are on by default with Streaming Engine.
- Exactly-once processing should be the default, with an explicit at-least-once mode when cost and latency matter more.
- The pipeline lives on Google Cloud: Pub/Sub, BigQuery, Bigtable, Spanner, and Iceberg I/O are first-class.
Think twice when
- The pipeline must stay portable in practice; the code is Beam, but Streaming Engine, Prime, and templates tie operations to Google Cloud.
- Cost needs a ceiling; per-second billing of workers, plus Streaming Engine, needs the same attention any autoscaling service does.
- The team prefers SQL-first authoring; Dataflow SQL was shut down in 2025, leaving SDKs, YAML, and the job builder.
How it runs
Serverless in operation: submit a Beam pipeline (Java, Python, Go, or YAML) and the service allocates Compute Engine workers in your project, scales them, and tears them down. Streaming Engine moves execution state into the service backend; delivery is exactly-once by default.
Details
CompareHow Google Cloud Dataflow answers the questions Stream Processing turns on.
Stream Processing
| How it works | |
| Processing model | Beam's unified batch and streaming model, executed by the service rather than a cluster you run |
| APIs | Beam SDKs: Java, Python, Go; Beam YAML and the job builder UI; classic and Flex templates (Dataflow SQL shut down in 2025) |
| Windowing | Beam's: fixed, sliding, session, global; calendar in Java |
| Event time and lateness | Beam's: watermarks with triggers, configurable allowed lateness and accumulation modes |
| State backend | Managed by the service: on the worker VMs and their disks by default, in the service backend with Streaming Engine |
| Delivery guarantee | Exactly-once by default; at-least-once mode for streaming |
| Reading results | Through sinks only: there is no serving layer |
| Running it | |
| What you run | Nothing to install, but the worker VMs are Compute Engine instances in your own project; Streaming Engine moves execution into the service backend |
| Scaling | Horizontal autoscaling on by default with Streaming Engine, with min and max workers adjustable in flight; vertical autoscaling needs Dataflow Prime |
| Connections | |
| Connectors | Beam's I/O set: Pub/Sub, BigQuery, Bigtable, Spanner, Cloud Storage, Kafka, Iceberg; Managed I/O auto-upgrades the Kafka, Iceberg and BigQuery ones |
Alternatives to Google Cloud Dataflow
