Technologies tagged with "Open Source"

Apache Thrift is a cross-language services framework: an interface definition language and code generator that emit both the serialization code and the RPC client and server for more than a dozen languages, with pluggable protocols and transports underneath.

Cap'n Proto is an MIT-licensed binary interchange format and capability-based RPC system from the author of Protocol Buffers version 2, designed so that the encoded bytes are directly usable in memory — there is no parse step between receiving a message and reading a field.

FlatBuffers is Google's Apache-2.0 cross-platform serialization library, built for performance-critical code: data is accessed directly in the serialized buffer without unpacking, the only memory needed is the buffer itself, and schemas evolve with forwards and backwards compatibility.

JSON Schema is the vocabulary for describing, validating, and documenting JSON data — the third schema type every major schema registry supports alongside Avro and Protocol Buffers, and the one that requires no new encoding on the wire.

Apache ORC is a columnar file format built for the Hadoop and Hive ecosystem: per-column min/max statistics and bloom filters for skipping data, support for ACID transactions and snapshot isolation, and native handling of Hive's compound types.

Vortex is an extensible columnar file format written in Rust and incubating at the Linux Foundation, claiming order-of-magnitude gains over Parquet on random access and scans at comparable compression, with an encoding system designed to be extended rather than frozen into the specification.

Nimble is Meta's Apache-2.0 columnar file format, built as a replacement for Parquet and ORC on workloads with thousands of columns: FlatBuffers metadata, block rather than stream encoding, and an encoding layer decoupled from the file layout so new schemes can be added and composed.

Unity Catalog is an Apache-2.0 catalog hosted by the LF AI & Data Foundation that governs Delta, Iceberg, and Hudi tables alongside files, functions, and AI models, speaking both the Hive metastore API and the Iceberg REST catalog API.

Project Nessie is an Apache-2.0 transactional catalog for data lakes with git-like semantics: Iceberg tables and views are versioned on branches and tags, changes across multiple tables commit atomically, and engines connect through the Iceberg REST protocol.

Lakekeeper is an Apache-2.0 Apache Iceberg REST catalog written in Rust: a single binary with multi-tenant warehouses created at runtime, fine-grained authorization through OpenFGA or Cedar, vended credentials for S3, ADLS and GCS, and change events emitted to Kafka or NATS.

Apicurio Registry is an Apache-2.0 registry that stores schemas and API definitions together — Avro, Protobuf, JSON Schema, OpenAPI, AsyncAPI, GraphQL, WSDL and XSD — behind a REST API, with a Confluent-compatible endpoint so existing Kafka clients work unchanged.

Elementary is an Apache-2.0 data observability tool built for dbt projects: a dbt package and CLI collect test results and run anomaly detection on freshness, volume, and custom metrics, with Elementary Cloud adding column-level lineage, ML monitors, and incident workflows.

Deequ is an Apache-2.0 library from AWS Labs for defining "unit tests for data" on Apache Spark: constraints and metrics are computed as part of the Spark job itself, which makes quality checks scale with the data rather than sampling it.

pandera is an MIT-licensed Python library for validating dataframes: schemas are declared as objects or as typed classes and checked at runtime across pandas, polars, pyspark, dask, modin, ibis, and geopandas.

Apache Atlas is an Apache Software Foundation project providing metadata management, classification, and lineage for the Hadoop ecosystem, with an extensible type system and an authorization path through Apache Ranger.

Apache Gravitino is a federated metadata lake: rather than copying metadata into its own store, it manages metadata in place across Hive, JDBC databases, object storage, and message systems, exposing one model and one API for data and AI assets across regions and clouds.

Kestra is an open-source orchestration platform whose workflows are declared in YAML rather than written in a programming language: an event-driven, language-agnostic engine with a large plugin catalogue, a JVM runtime backed by Postgres or MySQL, and Enterprise and Cloud editions above the Apache-2.0 core.

Argo Workflows is a CNCF-graduated workflow engine implemented as a Kubernetes custom resource: each step of a DAG runs in its own container, workflows are declared in YAML and submitted to the cluster, and the scheduler is the Kubernetes control plane rather than a separate application.

Temporal is an MIT-licensed durable execution platform: workflows are ordinary code in one of eight SDK languages, and the service persists every step so a function can keep running across crashes, restarts, and deploys for minutes or for months.

Flyte is an LF AI & Data project for building production data and ML pipelines: tasks and workflows are Python functions with strongly typed interfaces, each execution is versioned and cacheable, and the whole thing runs on Kubernetes with a Go control plane underneath.

Apache DolphinScheduler is a distributed workflow orchestration platform with a drag-and-drop DAG editor, a decentralised multi-master and multi-worker architecture built for high availability, and built-in task types for the tools a data platform already runs.

Windmill is an open-source workflow engine and developer platform written in Rust: individual scripts in TypeScript, Python, Go, SQL, Bash and more become reusable runnables, which compose into flows through a low-code builder or YAML, and into internal apps on top.

Luigi is Spotify's Python library for building pipelines of batch jobs: tasks declare what they require and what they produce, a central scheduler resolves the graph and prevents duplicate work, and triggering is left to cron.

Lightdash is an open-source BI platform that builds its semantic layer directly from a dbt project — dimensions and metrics are declared in the dbt YAML an analytics team already maintains, and every chart and dashboard is generated from those definitions.

Evidence is an MIT-licensed framework that builds business intelligence from plain files: SQL queries and charts are written inside markdown, and the project compiles to a static website that can be deployed anywhere a static site can.

Rill is an Apache-2.0 BI tool distributed as a single Go binary: metrics and dashboards are defined in SQL and YAML in a local project, served by an embedded DuckDB or a connected ClickHouse, and deployed to Rill Cloud when they are ready to share.

Redash is a BSD-licensed query-and-dashboard tool: analysts write SQL against any connected source, turn results into visualisations, assemble them into dashboards, and share them — running entirely on the team's own infrastructure, with no commercial edition to upgrade to.

SQLMesh is a data transformation framework from Tobiko Data that runs SQL and Python models with a plan-and-apply workflow, column-level lineage, unit tests, and virtual data environments that let teams preview changes without rebuilding tables.

Dataform is a service for developing, testing, version-controlling, and scheduling SQL transformation workflows in BigQuery, built on an Apache-2.0 meta-language and offered on Google Cloud at no charge beyond the BigQuery it runs.

Bruin is an open-source data pipeline tool written in Go that combines ingestion, SQL and Python transformations, automated quality checks, and column-level lineage in a single CLI, with a managed cloud layer above it.

Presto is a distributed SQL query engine for federated analytics, governed by the Presto Foundation under the Linux Foundation, whose members include Meta, IBM, Uber, and Nvidia — the original codebase that PrestoSQL forked from before renaming itself Trino.

Apache DataFusion is an extensible query engine written in Rust that uses Apache Arrow as its in-memory format, shipped as libraries for building database and analytic systems rather than as a server to run.

Polars is a DataFrame library written in Rust with a Python interface, built around a query optimiser and Arrow memory rather than row-by-row execution, and designed as a single-node engine that handles data most teams would otherwise put on a cluster.

Dremio is a SQL engine for the lakehouse that federates queries across Iceberg, Delta Lake, databases, and warehouses, and accelerates them with materialised structures it calls Reflections — available as a free Community edition, a self-managed Enterprise edition, and a managed cloud service.

Apache Hive is a distributed, fault-tolerant data warehouse system that enables analytics at massive scale, running SQL over data in the Hadoop ecosystem on Apache Tez, with LLAP for sub-second queries and first-class Apache Iceberg support in the 4.x line.

Ray is an open-source framework for scaling Python and AI workloads, pairing a general-purpose distributed execution core with libraries for data, training, tuning, serving, and reinforcement learning.