Technologies tagged with "Batch"

Apache ORC is a columnar file format built for the Hadoop and Hive ecosystem: per-column min/max statistics and bloom filters for skipping data, support for ACID transactions and snapshot isolation, and native handling of Hive's compound types.

Vortex is an extensible columnar file format written in Rust and incubating at the Linux Foundation, claiming order-of-magnitude gains over Parquet on random access and scans at comparable compression, with an encoding system designed to be extended rather than frozen into the specification.

Nimble is Meta's Apache-2.0 columnar file format, built as a replacement for Parquet and ORC on workloads with thousands of columns: FlatBuffers metadata, block rather than stream encoding, and an encoding layer decoupled from the file layout so new schemes can be added and composed.

Deequ is an Apache-2.0 library from AWS Labs for defining "unit tests for data" on Apache Spark: constraints and metrics are computed as part of the Spark job itself, which makes quality checks scale with the data rather than sampling it.

Kestra is an open-source orchestration platform whose workflows are declared in YAML rather than written in a programming language: an event-driven, language-agnostic engine with a large plugin catalogue, a JVM runtime backed by Postgres or MySQL, and Enterprise and Cloud editions above the Apache-2.0 core.

Argo Workflows is a CNCF-graduated workflow engine implemented as a Kubernetes custom resource: each step of a DAG runs in its own container, workflows are declared in YAML and submitted to the cluster, and the scheduler is the Kubernetes control plane rather than a separate application.

Temporal is an MIT-licensed durable execution platform: workflows are ordinary code in one of eight SDK languages, and the service persists every step so a function can keep running across crashes, restarts, and deploys for minutes or for months.

AWS Step Functions is a serverless orchestrator built on state machines: each step is a state that calls an AWS service or an external worker, execution history is retained and visually debuggable, and the bill is per state transition rather than per server.

Flyte is an LF AI & Data project for building production data and ML pipelines: tasks and workflows are Python functions with strongly typed interfaces, each execution is versioned and cacheable, and the whole thing runs on Kubernetes with a Go control plane underneath.

Apache DolphinScheduler is a distributed workflow orchestration platform with a drag-and-drop DAG editor, a decentralised multi-master and multi-worker architecture built for high availability, and built-in task types for the tools a data platform already runs.

Windmill is an open-source workflow engine and developer platform written in Rust: individual scripts in TypeScript, Python, Go, SQL, Bash and more become reusable runnables, which compose into flows through a low-code builder or YAML, and into internal apps on top.

Luigi is Spotify's Python library for building pipelines of batch jobs: tasks declare what they require and what they produce, a central scheduler resolves the graph and prevents duplicate work, and triggering is left to cron.

SQLMesh is a data transformation framework from Tobiko Data that runs SQL and Python models with a plan-and-apply workflow, column-level lineage, unit tests, and virtual data environments that let teams preview changes without rebuilding tables.

Dataform is a service for developing, testing, version-controlling, and scheduling SQL transformation workflows in BigQuery, built on an Apache-2.0 meta-language and offered on Google Cloud at no charge beyond the BigQuery it runs.

Coalesce is a commercial transformation platform that builds warehouse pipelines from reusable metadata-driven nodes and templates rather than hand-written SQL, with column-level lineage, cataloging, and quality checks in one managed service.

Bruin is an open-source data pipeline tool written in Go that combines ingestion, SQL and Python transformations, automated quality checks, and column-level lineage in a single CLI, with a managed cloud layer above it.

Apache Hive is a distributed, fault-tolerant data warehouse system that enables analytics at massive scale, running SQL over data in the Hadoop ecosystem on Apache Tez, with LLAP for sub-second queries and first-class Apache Iceberg support in the 4.x line.

Ray is an open-source framework for scaling Python and AI workloads, pairing a general-purpose distributed execution core with libraries for data, training, tuning, serving, and reinforcement learning.

Dask is a Python library for parallel and distributed computing that provides pandas-compatible DataFrames and NumPy-compatible Arrays over datasets larger than memory, running locally or across a cluster.

Daft is a Rust-based data engine with a Python interface, processing images, audio, video, embeddings, and structured data through one API, and scaling from a laptop to a cluster without code changes.

Apache Hadoop is the distributed storage and batch processing framework that started the category, comprising HDFS for storage, YARN for resource management and scheduling, and MapReduce as its batch execution engine.

Apache Beam is a unified programming model for batch and streaming pipelines, written once in Java, Python, or Go and executed by a runner — Flink, Spark, Dataflow, or others — rather than by an engine of its own.

Google Cloud Dataflow is a fully managed service that executes Apache Beam pipelines for batch and streaming, provisioning and autoscaling worker VMs automatically and processing every record exactly once by default.

AWS Database Migration Service moves data between relational databases, warehouses, and NoSQL stores, either as a one-time migration or as ongoing replication that keeps a source and target in sync.

dlt is an open-source Python library for the extract-and-load stage, inferring schemas, normalising nested data, and handling incremental loading — installed with pip and run wherever Python already runs.

Meltano is an open-source ELT orchestrator that assembles pipelines from Singer taps and targets declaratively in version-controlled YAML, with a hub cataloguing over 600 community-maintained connectors.

Estuary Flow is a data movement platform built on a streaming core, capturing from databases via CDC and from SaaS APIs, then materialising continuously into warehouses and lakes rather than on a schedule.

Hevo Data is a managed no-code data pipeline platform with over 150 connectors across SaaS applications, databases, and storage, billed by event — each row inserted, updated, or deleted at the destination.

Sling is a data movement tool distributed as one Go binary, moving data between databases, warehouses, and file storage from the command line or a pipe, with a managed platform available above it.

Matillion is a commercial data integration platform with a visual pipeline builder, loading data into cloud warehouses and pushing transformation work down to the warehouse's own compute rather than processing it separately.

Airbyte is a data integration platform that syncs data from APIs, databases, and files into warehouses and lakes, available self-hosted or as a managed cloud service.

Fivetran is a commercial, fully managed data movement platform with maintained connectors, automated schema handling, and consumption-based pricing.

Apache Spark is an open-source distributed compute engine for batch processing, SQL, machine learning, and stream processing across large clusters.

dbt is a transformation framework that compiles modular, version-controlled SQL models and runs them inside the warehouse, with testing, documentation, and lineage built in.

DuckDB is an open-source, in-process OLAP database — an embeddable columnar SQL engine that queries Parquet, CSV, and dataframes with no server to run.

Apache Airflow is the widely adopted open-source orchestrator where pipelines are Python-defined DAGs, with a large provider ecosystem and multiple managed offerings.