Arroyo
Arroyo is a distributed stream processing engine written in Rust that defines pipelines in SQL, performing stateful computation over bounded and unbounded sources with event-time processing and watermarks.
LicenseOpen source (permissive)· Apache-2.0
DeploymentSelf-hosted
PricingFree
Operational complexityMedium
WorkloadStreamingBatch
LanguageRust
Use it when
- Your pipelines are expressible in SQL and you want the smallest engine that runs them: a single Rust binary instead of a JVM cluster.
- Exactly-once matters; Chandy-Lamport checkpoints and transactional Kafka sinks provide it without extra machinery.
- Sources are varied: Kafka, Kinesis, NATS, MQTT, WebSocket, HTTP, Postgres, and MySQL in; Iceberg, Delta Lake, and Redis among the sinks.
- You want UDFs without leaving the model; Rust and Python functions, including async and aggregates, extend the SQL.
Think twice when
- State exceeds memory; in the open-source release state lives in worker RAM, checkpointed to object storage but bounded by it.
- Jobs need code-level control or side outputs; there is one API, SQL, and late rows are dropped rather than rerouted.
- Rescaling must be online; changing parallelism means restarting from a checkpoint.
How it runs
One binary carries the API, controller, and workers, scheduled as processes, on nodes, or on Kubernetes, with Postgres or SQLite for configuration and object storage for checkpoints as Parquet. Pipelines are SQL planned through DataFusion over Arrow batches. Apache-2.0 and free; Arroyo Cloud is the managed option.
Details
CompareHow Arroyo answers the questions Stream Processing turns on.
Stream Processing
| How it works | |
| Processing model | Streaming dataflow DAG on Arrow batches; SQL planned with DataFusion |
| APIs | SQL; UDFs in Rust or Python, async and aggregate included; REST API and web UI |
| Windowing | Tumbling, sliding (HOP), session |
| Event time and lateness | Watermarks emitted by sources as a fixed offset from the event timestamp; late rows are discarded, no side outputs |
| State backend | Worker memory in the open-source release, so state is bounded by RAM; checkpointed to object storage as Parquet |
| Delivery guarantee | Exactly-once: Chandy-Lamport checkpoints · transactional Kafka sinks |
| Reading results | Through sinks only: there is no serving layer |
| Running it | |
| What you run | A single binary running the API, controller and workers; process, node or Kubernetes scheduler, plus Postgres or SQLite for config and object storage for checkpoints |
| Scaling | Rescale by restarting from a checkpoint, not online; subtasks are placed by the process, node or Kubernetes scheduler |
| Connections | |
| Connectors | Kafka, Redpanda, Confluent, Kinesis, NATS, MQTT, RabbitMQ, Fluvio, WebSocket, SSE, polling HTTP, Postgres and MySQL; sinks add Iceberg, Delta Lake, filesystem, Redis and webhooks |
Alternatives to Arroyo
