Ingestion & Integration

AWS Database Migration Service moves data between relational databases, warehouses, and NoSQL stores, either as a one-time migration or as ongoing replication that keeps a source and target in sync.

Datastream is Google Cloud's serverless change data capture and replication service, streaming changes from Oracle, MySQL, PostgreSQL, SQL Server, MongoDB, and Spanner into BigQuery and Cloud Storage with minimal latency.

Oracle GoldenGate is a real-time data replication and change data capture platform spanning Oracle and non-Oracle databases, NoSQL stores, warehouses, and messaging systems, available self-managed or as a managed OCI service.

PeerDB is an open-source change data capture engine built specifically for PostgreSQL sources, acquired by ClickHouse in 2024 and kept as a distinct component powering the managed Postgres CDC in ClickPipes.

Sequin is an open-source CDC tool that streams PostgreSQL changes to queues, streaming platforms, and webhooks with strict ordering and exactly-once processing, shipped as a Docker image rather than a Postgres extension.

Maxwell's Daemon reads the MySQL binary log and writes row-level changes as JSON to Kafka, Kinesis, and other streaming platforms — a single-purpose CDC process with no framework around it.

Striim is a commercial real-time data integration platform combining change data capture with streaming SQL, moving data from operational databases and SaaS applications into warehouses, lakehouses, and Kafka as it changes.

Qlik Replicate is a commercial data replication platform using agentless, log-based change data capture, spanning mainframe and mid-range systems, SAP estates, and modern cloud warehouses without installing software on the source.

dlt is an open-source Python library for the extract-and-load stage, inferring schemas, normalising nested data, and handling incremental loading — installed with pip and run wherever Python already runs.

Meltano is an open-source ELT orchestrator that assembles pipelines from Singer taps and targets declaratively in version-controlled YAML, with a hub cataloguing over 600 community-maintained connectors.

Estuary Flow is a data movement platform built on a streaming core, capturing from databases via CDC and from SaaS APIs, then materialising continuously into warehouses and lakes rather than on a schedule.

Hevo Data is a managed no-code data pipeline platform with over 150 connectors across SaaS applications, databases, and storage, billed by event — each row inserted, updated, or deleted at the destination.

Sling is a data movement tool distributed as one Go binary, moving data between databases, warehouses, and file storage from the command line or a pipe, with a managed platform available above it.

Matillion is a commercial data integration platform with a visual pipeline builder, loading data into cloud warehouses and pushing transformation work down to the warehouse's own compute rather than processing it separately.

Airbyte is a data integration platform that syncs data from APIs, databases, and files into warehouses and lakes, available self-hosted or as a managed cloud service.

Fivetran is a commercial, fully managed data movement platform with maintained connectors, automated schema handling, and consumption-based pricing.

Debezium is an open-source CDC platform that streams row-level changes from databases such as PostgreSQL, MySQL, and MongoDB, typically via Kafka Connect.

Apache Flink is an open-source framework for stateful computation over unbounded and bounded data streams, with event-time processing, exactly-once state consistency, and CDC ingestion via Flink CDC.