Technologies tagged with "Self Hosted"

Unity Catalog is an Apache-2.0 catalog hosted by the LF AI & Data Foundation that governs Delta, Iceberg, and Hudi tables alongside files, functions, and AI models, speaking both the Hive metastore API and the Iceberg REST catalog API.

Project Nessie is an Apache-2.0 transactional catalog for data lakes with git-like semantics: Iceberg tables and views are versioned on branches and tags, changes across multiple tables commit atomically, and engines connect through the Iceberg REST protocol.

Lakekeeper is an Apache-2.0 Apache Iceberg REST catalog written in Rust: a single binary with multi-tenant warehouses created at runtime, fine-grained authorization through OpenFGA or Cedar, vended credentials for S3, ADLS and GCS, and change events emitted to Kafka or NATS.

Apicurio Registry is an Apache-2.0 registry that stores schemas and API definitions together — Avro, Protobuf, JSON Schema, OpenAPI, AsyncAPI, GraphQL, WSDL and XSD — behind a REST API, with a Confluent-compatible endpoint so existing Kafka clients work unchanged.

The Buf Schema Registry stores versioned Protobuf modules rather than individual topic schemas: pushes are linted and checked for breaking changes before they reach consumers, generated SDKs are published to native package managers, and a Confluent-compatible endpoint serves Kafka clients.

Soda is a data quality platform built on Soda Core, a Python library and CLI that runs checks and data contracts inside pipelines, with Soda Cloud for triage and collaboration and a hosted or self-hosted Runner for scheduled observability.

Elementary is an Apache-2.0 data observability tool built for dbt projects: a dbt package and CLI collect test results and run anomaly detection on freshness, volume, and custom metrics, with Elementary Cloud adding column-level lineage, ML monitors, and incident workflows.

Deequ is an Apache-2.0 library from AWS Labs for defining "unit tests for data" on Apache Spark: constraints and metrics are computed as part of the Spark job itself, which makes quality checks scale with the data rather than sampling it.

pandera is an MIT-licensed Python library for validating dataframes: schemas are declared as objects or as typed classes and checked at runtime across pandas, polars, pyspark, dask, modin, ibis, and geopandas.

Anomalo is a commercial data quality platform that applies machine learning to detect anomalies without hand-written rules, and can be deployed either as SaaS or entirely inside the customer's own VPC so that data never leaves their environment.

Datafold is a commercial platform built on value-level data diffing: it compares the output of a proposed change against production in CI, validates parity during warehouse migrations, and monitors metrics for deviation, deployable as SaaS or inside the customer's own cloud.

Apache Atlas is an Apache Software Foundation project providing metadata management, classification, and lineage for the Hadoop ecosystem, with an extensible type system and an authorization path through Apache Ranger.

Apache Gravitino is a federated metadata lake: rather than copying metadata into its own store, it manages metadata in place across Hive, JDBC databases, object storage, and message systems, exposing one model and one API for data and AI assets across regions and clouds.

Collibra is a commercial data intelligence platform whose centre of gravity is governance: stewardship workflows, policies, and a business glossary, with a data catalog, lineage, quality and observability, a data marketplace, and AI governance built around them.

Alation is a commercial data catalog and data intelligence platform combining search and discovery, lineage, stewardship, and data quality, deployable either as the Alation Cloud Service or on the customer's own infrastructure.

Kestra is an open-source orchestration platform whose workflows are declared in YAML rather than written in a programming language: an event-driven, language-agnostic engine with a large plugin catalogue, a JVM runtime backed by Postgres or MySQL, and Enterprise and Cloud editions above the Apache-2.0 core.

Argo Workflows is a CNCF-graduated workflow engine implemented as a Kubernetes custom resource: each step of a DAG runs in its own container, workflows are declared in YAML and submitted to the cluster, and the scheduler is the Kubernetes control plane rather than a separate application.

Temporal is an MIT-licensed durable execution platform: workflows are ordinary code in one of eight SDK languages, and the service persists every step so a function can keep running across crashes, restarts, and deploys for minutes or for months.

Flyte is an LF AI & Data project for building production data and ML pipelines: tasks and workflows are Python functions with strongly typed interfaces, each execution is versioned and cacheable, and the whole thing runs on Kubernetes with a Go control plane underneath.

Apache DolphinScheduler is a distributed workflow orchestration platform with a drag-and-drop DAG editor, a decentralised multi-master and multi-worker architecture built for high availability, and built-in task types for the tools a data platform already runs.

Windmill is an open-source workflow engine and developer platform written in Rust: individual scripts in TypeScript, Python, Go, SQL, Bash and more become reusable runnables, which compose into flows through a low-code builder or YAML, and into internal apps on top.

Luigi is Spotify's Python library for building pipelines of batch jobs: tasks declare what they require and what they produce, a central scheduler resolves the graph and prevents duplicate work, and triggering is left to cron.

Power BI is Microsoft's business intelligence platform — free Windows authoring in Power BI Desktop, publishing and sharing through the Power BI service at a per-user price, and an on-premises Report Server for organisations that cannot publish to the cloud.

Tableau is Salesforce's business intelligence platform, licensed by role — Creator, Explorer, Viewer — and deployable either as the hosted Tableau Cloud or as Tableau Server on the customer's own Windows or Linux machines.

Looker is Google Cloud's business intelligence platform, built around LookML — a version-controlled modelling language in which metrics and joins are defined once and every dashboard, explore, and API call is generated from that definition rather than from ad-hoc SQL.

Lightdash is an open-source BI platform that builds its semantic layer directly from a dbt project — dimensions and metrics are declared in the dbt YAML an analytics team already maintains, and every chart and dashboard is generated from those definitions.

Evidence is an MIT-licensed framework that builds business intelligence from plain files: SQL queries and charts are written inside markdown, and the project compiles to a static website that can be deployed anywhere a static site can.

Rill is an Apache-2.0 BI tool distributed as a single Go binary: metrics and dashboards are defined in SQL and YAML in a local project, served by an embedded DuckDB or a connected ClickHouse, and deployed to Rill Cloud when they are ready to share.

Redash is a BSD-licensed query-and-dashboard tool: analysts write SQL against any connected source, turn results into visualisations, assemble them into dashboards, and share them — running entirely on the team's own infrastructure, with no commercial edition to upgrade to.

SQLMesh is a data transformation framework from Tobiko Data that runs SQL and Python models with a plan-and-apply workflow, column-level lineage, unit tests, and virtual data environments that let teams preview changes without rebuilding tables.

Dataform is a service for developing, testing, version-controlling, and scheduling SQL transformation workflows in BigQuery, built on an Apache-2.0 meta-language and offered on Google Cloud at no charge beyond the BigQuery it runs.

Bruin is an open-source data pipeline tool written in Go that combines ingestion, SQL and Python transformations, automated quality checks, and column-level lineage in a single CLI, with a managed cloud layer above it.

Presto is a distributed SQL query engine for federated analytics, governed by the Presto Foundation under the Linux Foundation, whose members include Meta, IBM, Uber, and Nvidia — the original codebase that PrestoSQL forked from before renaming itself Trino.

Apache DataFusion is an extensible query engine written in Rust that uses Apache Arrow as its in-memory format, shipped as libraries for building database and analytic systems rather than as a server to run.

Polars is a DataFrame library written in Rust with a Python interface, built around a query optimiser and Arrow memory rather than row-by-row execution, and designed as a single-node engine that handles data most teams would otherwise put on a cluster.

Starburst is the commercial platform built on Trino, sold as Starburst Enterprise for self-managed clusters and Starburst Galaxy as a fully managed service, adding governance, performance work, and a supported Iceberg lakehouse on top of the open-source engine.