# Arroyo

> Rust stream processing engine driven entirely by SQL

Arroyo is a distributed stream processing engine written in Rust that defines pipelines in SQL, performing stateful computation over bounded and unbounded sources with event-time processing and watermarks.

Website: https://arroyo.dev

## Use it when

- Your pipelines are expressible in SQL and you want the smallest engine that runs them: a single Rust binary instead of a JVM cluster.
- Exactly-once matters; Chandy-Lamport checkpoints and transactional Kafka sinks provide it without extra machinery.
- Sources are varied: Kafka, Kinesis, NATS, MQTT, WebSocket, HTTP, Postgres, and MySQL in; Iceberg, Delta Lake, and Redis among the sinks.
- You want UDFs without leaving the model; Rust and Python functions, including async and aggregates, extend the SQL.

## Think twice when

- State exceeds memory; in the open-source release state lives in worker RAM, checkpointed to object storage but bounded by it.
- Jobs need code-level control or side outputs; there is one API, SQL, and late rows are dropped rather than rerouted.
- Rescaling must be online; changing parallelism means restarting from a checkpoint.

## How it runs

One binary carries the API, controller, and workers, scheduled as processes, on nodes, or on Kubernetes, with Postgres or SQLite for configuration and object storage for checkpoints as Parquet. Pipelines are SQL planned through DataFusion over Arrow batches. Apache-2.0 and free; Arroyo Cloud is the managed option.

## Details

### General

| Attribute | Value |
| --- | --- |
| License | Open source (permissive) |
| SPDX identifier | Apache-2.0 |
| Deployment | Self-hosted |
| Workload | Streaming, Batch |
| Operational complexity | Medium |
| Pricing | Free |
| Language | Rust |
| Repository | https://github.com/ArroyoSystems/arroyo |
| Documentation | https://doc.arroyo.dev |
| Stars | 5k |

### Stream Processing

| Attribute | Value |
| --- | --- |
| Processing model | Streaming dataflow DAG on Arrow batches; SQL planned with DataFusion |
| APIs | SQL; UDFs in Rust or Python, async and aggregate included; REST API and web UI |
| What you run | A single binary running the API, controller and workers; process, node or Kubernetes scheduler, plus Postgres or SQLite for config and object storage for checkpoints |
| Connectors | Kafka, Redpanda, Confluent, Kinesis, NATS, MQTT, RabbitMQ, Fluvio, WebSocket, SSE, polling HTTP, Postgres and MySQL; sinks add Iceberg, Delta Lake, filesystem, Redis and webhooks |
| Windowing | Tumbling, sliding (HOP), session |
| Event time and lateness | Watermarks emitted by sources as a fixed offset from the event timestamp; late rows are discarded, no side outputs |
| State backend | Worker memory in the open-source release, so state is bounded by RAM; checkpointed to object storage as Parquet |
| Delivery guarantee | Exactly-once: Chandy-Lamport checkpoints, transactional Kafka sinks |
| Reading results | Through sinks only: there is no serving layer |
| Scaling | Rescale by restarting from a checkpoint, not online; subtasks are placed by the process, node or Kubernetes scheduler |

## Capabilities

- [Stream Processing](https://matca.io/capabilities/stream-processing)

---

Source: https://matca.io/arroyo
Last updated: 2026-08-31T11:40:49.501Z