Apache Flink
Apache Flink is an open-source framework for stateful computation over unbounded and bounded data streams, with event-time processing, exactly-once state consistency, and CDC ingestion via Flink CDC.
LicenseOpen source (permissive)· Apache-2.0
DeploymentSelf-hostedManaged
PricingFreeSubscription
Operational complexityHigh
WorkloadStreamingBatch
LanguageJava, Scala, Python
Use it when
- Correct low-latency results over large state are the requirement: event time, watermarks, and exactly-once state consistency are the reference implementation here.
- The job mixes paradigms; DataStream code, Table/SQL, and batch-as-bounded-stream share one engine.
- CDC belongs in the processor; Flink CDC snapshots and streams MySQL, PostgreSQL, MongoDB, and more into the same job that transforms them.
- You need ecosystem depth: Kafka, Pulsar, Kinesis, Iceberg, and Paimon connectors, plus managed offerings from every major cloud and vendor.
Think twice when
- The team is small and jobs are simple; a JobManager-and-TaskManagers cluster with checkpoints and savepoints is real operational weight (ops complexity here is High) against SQL-first engines like Arroyo or RisingWave.
- Rescaling must be online; changing parallelism means restoring from a checkpoint or savepoint.
- Results must be queryable in place; Flink emits to sinks and is not a serving layer.
How it runs
Self-host a cluster (JobManager plus TaskManagers) on Kubernetes or YARN, with state in HashMap or RocksDB backends checkpointed to durable storage. Applications use DataStream (Java/Scala), PyFlink, or Flink SQL. Managed Flink is sold by Confluent, Ververica, AWS, and others.
Details
How Apache Flink answers the questions each capability turns on.
Change Data Capture
| How it works | |
| Initial snapshot | Lock-free incremental snapshot, chunked and resumable |
| End-to-end lag | Sub-second; changes flow into the same job that processes them |
| Delivery guarantee | Exactly-once |
| Schema drift | Schema evolution modes: evolve, try_evolve, lenient, ignore, exception |
| Running it | |
| What you run | A Flink cluster: JobManager and TaskManagers |
| Connections | |
| Sources | MySQL, PostgreSQL, Oracle, MongoDB, SQL Server, Db2 |
| Source requirements | ROW binlog on MySQL, logical decoding on Postgres, per connector |
| Transformations | Full Flink processing in the same job; YAML transform for pipelines |
| Destinations | Any Flink sink: Kafka, warehouses, table formats |
| Cost | |
| Billing unit | None for the project; managed Flink billed by the vendor |
Stream Processing
| How it works | |
| Processing model | Record-at-a-time streaming; batch is a bounded stream |
| APIs | DataStream, Table/SQL, ProcessFunction |
| Windowing | Tumbling, sliding, session, global |
| Event time and lateness | Event time with watermarks; allowed lateness and side outputs |
| State backend | HashMap, RocksDB, ForSt (experimental, disaggregated) |
| Delivery guarantee | Exactly-once |
| Reading results | Through sinks; queryable state exists but is deprecated |
| Running it | |
| What you run | A Flink cluster: JobManager and TaskManagers |
| Scaling | Rescale from checkpoint; adaptive scheduler and reactive mode |
| Connections | |
| Connectors | Kafka, Pulsar, Kinesis, JDBC, filesystems, Iceberg and Paimon |
Alternatives to Apache Flink
