Technologies tagged with "Streaming"

Apache Thrift is a cross-language services framework: an interface definition language and code generator that emit both the serialization code and the RPC client and server for more than a dozen languages, with pluggable protocols and transports underneath.

Cap'n Proto is an MIT-licensed binary interchange format and capability-based RPC system from the author of Protocol Buffers version 2, designed so that the encoded bytes are directly usable in memory — there is no parse step between receiving a message and reading a field.

FlatBuffers is Google's Apache-2.0 cross-platform serialization library, built for performance-critical code: data is accessed directly in the serialized buffer without unpacking, the only memory needed is the buffer itself, and schemas evolve with forwards and backwards compatibility.

JSON Schema is the vocabulary for describing, validating, and documenting JSON data — the third schema type every major schema registry supports alongside Avro and Protocol Buffers, and the one that requires no new encoding on the wire.

Apicurio Registry is an Apache-2.0 registry that stores schemas and API definitions together — Avro, Protobuf, JSON Schema, OpenAPI, AsyncAPI, GraphQL, WSDL and XSD — behind a REST API, with a Confluent-compatible endpoint so existing Kafka clients work unchanged.

The Buf Schema Registry stores versioned Protobuf modules rather than individual topic schemas: pushes are linted and checked for breaking changes before they reach consumers, generated SDKs are published to native package managers, and a Confluent-compatible endpoint serves Kafka clients.

The AWS Glue Schema Registry is a serverless registry for streaming schemas — Avro, JSON Schema, and Protobuf — with eight compatibility modes, open-source serialisers, and integrations across MSK, Kinesis Data Streams, Managed Service for Apache Flink, and Lambda. AWS's documentation states it is free to use.

RisingWave is a PostgreSQL-compatible streaming database that maintains materialized views incrementally rather than recomputing them, storing state and tables in object storage so compute and storage scale separately.

Materialize incrementally updates SQL results as data arrives rather than recomputing them, built on Timely and Differential Dataflow and reachable over the PostgreSQL wire protocol, with strict serializability by default.

Apache Beam is a unified programming model for batch and streaming pipelines, written once in Java, Python, or Go and executed by a runner — Flink, Spark, Dataflow, or others — rather than by an engine of its own.

Google Cloud Dataflow is a fully managed service that executes Apache Beam pipelines for batch and streaming, provisioning and autoscaling worker VMs automatically and processing every record exactly once by default.

Azure Stream Analytics is a fully managed stream processing engine using a SQL dialect extended with temporal constraints, running in the cloud or on IoT Edge with the same tools and query language.

Arroyo is a distributed stream processing engine written in Rust that defines pipelines in SQL, performing stateful computation over bounded and unbounded sources with event-time processing and watermarks.

Apache Pulsar is an open-source messaging and streaming platform that splits brokers from storage, keeping segments in Apache BookKeeper so capacity and serving scale independently, with multi-tenancy and geo-replication built in.

Confluent is the commercial data streaming platform built around Apache Kafka by its original creators, sold as self-managed Confluent Platform and as the Confluent Cloud service, and acquired by IBM in March 2026.

Amazon Kinesis Data Streams is AWS's managed event streaming service, organising data into shards with provisioned or on-demand capacity and integrating directly with the rest of the AWS analytics stack.

Google Cloud Pub/Sub is an asynchronous, scalable messaging service that decouples producers from consumers, offering push and pull subscriptions, configurable exactly-once delivery, and roughly 100-millisecond latency.

Azure Event Hubs is Microsoft's fully managed event streaming platform, natively supporting the Apache Kafka protocol alongside AMQP and HTTPS so existing Kafka workloads run without code changes or cluster management.

NATS is a messaging system distributed as one small Go binary, offering at-most-once core messaging with JetStream adding a persistence layer for at-least-once delivery, replay, and server-side consumer state.

WarpStream is a Kafka-compatible streaming platform with no local disks, writing directly to S3-class object storage, and running its stateless agents inside the customer's own cloud account under a bring-your-own-cloud model.

AutoMQ keeps Apache Kafka's compute layer and replaces its storage with cloud object storage, claiming 100% Kafka API compatibility and substantially lower cost by removing multi-replica local disks and cross-zone transfer.

RabbitMQ is a widely deployed open-source message broker built on Erlang/OTP, combining rich queue-based routing with a Streams feature that adds append-only, replayable logs alongside traditional queues.

AWS Database Migration Service moves data between relational databases, warehouses, and NoSQL stores, either as a one-time migration or as ongoing replication that keeps a source and target in sync.

Datastream is Google Cloud's serverless change data capture and replication service, streaming changes from Oracle, MySQL, PostgreSQL, SQL Server, MongoDB, and Spanner into BigQuery and Cloud Storage with minimal latency.

Oracle GoldenGate is a real-time data replication and change data capture platform spanning Oracle and non-Oracle databases, NoSQL stores, warehouses, and messaging systems, available self-managed or as a managed OCI service.

PeerDB is an open-source change data capture engine built specifically for PostgreSQL sources, acquired by ClickHouse in 2024 and kept as a distinct component powering the managed Postgres CDC in ClickPipes.

Sequin is an open-source CDC tool that streams PostgreSQL changes to queues, streaming platforms, and webhooks with strict ordering and exactly-once processing, shipped as a Docker image rather than a Postgres extension.

Maxwell's Daemon reads the MySQL binary log and writes row-level changes as JSON to Kafka, Kinesis, and other streaming platforms — a single-purpose CDC process with no framework around it.

Striim is a commercial real-time data integration platform combining change data capture with streaming SQL, moving data from operational databases and SaaS applications into warehouses, lakehouses, and Kafka as it changes.

Qlik Replicate is a commercial data replication platform using agentless, log-based change data capture, spanning mainframe and mid-range systems, SAP estates, and modern cloud warehouses without installing software on the source.

Estuary Flow is a data movement platform built on a streaming core, capturing from databases via CDC and from SaaS APIs, then materialising continuously into warehouses and lakes rather than on a schedule.

Apache Hudi is an open lakehouse format that pioneered transactional data lakes, offering record-level upserts and deletes, incremental queries, and a choice between copy-on-write and merge-on-read table types.

Apache Paimon is a lake format unifying batch and streaming, using an LSM tree to absorb high-frequency updates and emitting changelogs so downstream jobs can consume what changed rather than rescanning.

Apache Druid is a real-time analytics database combining ideas from data warehouses, time-series databases, and log search systems, with columnar storage, time-based partitioning, and bitmap indexes serving sub-second queries over streaming data.

Apache Pinot is a distributed real-time OLAP store built to serve analytical queries at very high concurrency with millisecond latency, ingesting directly from streams.

Confluent Schema Registry stores versioned Avro, Protobuf, and JSON schemas for Kafka topics and enforces compatibility rules when producers register changes.