Governance & Metadata

Unity Catalog is an Apache-2.0 catalog hosted by the LF AI & Data Foundation that governs Delta, Iceberg, and Hudi tables alongside files, functions, and AI models, speaking both the Hive metastore API and the Iceberg REST catalog API.

Project Nessie is an Apache-2.0 transactional catalog for data lakes with git-like semantics: Iceberg tables and views are versioned on branches and tags, changes across multiple tables commit atomically, and engines connect through the Iceberg REST protocol.

Lakekeeper is an Apache-2.0 Apache Iceberg REST catalog written in Rust: a single binary with multi-tenant warehouses created at runtime, fine-grained authorization through OpenFGA or Cedar, vended credentials for S3, ADLS and GCS, and change events emitted to Kafka or NATS.

The Lakehouse runtime catalog — until April 2026, BigLake metastore — is Google Cloud's fully managed, serverless implementation of the Apache Iceberg REST Catalog API, letting Spark, Flink, Hive, Trino, and BigQuery share the same Iceberg tables without duplicating files.

Apicurio Registry is an Apache-2.0 registry that stores schemas and API definitions together — Avro, Protobuf, JSON Schema, OpenAPI, AsyncAPI, GraphQL, WSDL and XSD — behind a REST API, with a Confluent-compatible endpoint so existing Kafka clients work unchanged.

The Buf Schema Registry stores versioned Protobuf modules rather than individual topic schemas: pushes are linted and checked for breaking changes before they reach consumers, generated SDKs are published to native package managers, and a Confluent-compatible endpoint serves Kafka clients.

The AWS Glue Schema Registry is a serverless registry for streaming schemas — Avro, JSON Schema, and Protobuf — with eight compatibility modes, open-source serialisers, and integrations across MSK, Kinesis Data Streams, Managed Service for Apache Flink, and Lambda. AWS's documentation states it is free to use.

Soda is a data quality platform built on Soda Core, a Python library and CLI that runs checks and data contracts inside pipelines, with Soda Cloud for triage and collaboration and a hosted or self-hosted Runner for scheduled observability.

Elementary is an Apache-2.0 data observability tool built for dbt projects: a dbt package and CLI collect test results and run anomaly detection on freshness, volume, and custom metrics, with Elementary Cloud adding column-level lineage, ML monitors, and incident workflows.

Deequ is an Apache-2.0 library from AWS Labs for defining "unit tests for data" on Apache Spark: constraints and metrics are computed as part of the Spark job itself, which makes quality checks scale with the data rather than sampling it.

pandera is an MIT-licensed Python library for validating dataframes: schemas are declared as objects or as typed classes and checked at runtime across pandas, polars, pyspark, dask, modin, ibis, and geopandas.

Monte Carlo is a commercial data observability platform: it monitors tables automatically rather than waiting for hand-written tests, enriches alerts with lineage and root-cause context, and routes incidents through triage workflows.

Anomalo is a commercial data quality platform that applies machine learning to detect anomalies without hand-written rules, and can be deployed either as SaaS or entirely inside the customer's own VPC so that data never leaves their environment.

Datafold is a commercial platform built on value-level data diffing: it compares the output of a proposed change against production in CI, validates parity during warehouse migrations, and monitors metrics for deviation, deployable as SaaS or inside the customer's own cloud.

Apache Atlas is an Apache Software Foundation project providing metadata management, classification, and lineage for the Hadoop ecosystem, with an extensible type system and an authorization path through Apache Ranger.

Apache Gravitino is a federated metadata lake: rather than copying metadata into its own store, it manages metadata in place across Hive, JDBC databases, object storage, and message systems, exposing one model and one API for data and AI assets across regions and clouds.

Collibra is a commercial data intelligence platform whose centre of gravity is governance: stewardship workflows, policies, and a business glossary, with a data catalog, lineage, quality and observability, a data marketplace, and AI governance built around them.

Alation is a commercial data catalog and data intelligence platform combining search and discovery, lineage, stewardship, and data quality, deployable either as the Alation Cloud Service or on the customer's own infrastructure.

Atlan is a commercial metadata platform covering discovery, lineage, glossaries, contracts, and policies across 80-plus sources, delivered as SaaS with a self-deployed runtime that keeps metadata extraction inside the customer's own network.

Microsoft Purview combines data governance — a Data Map that scans and classifies assets and a Unified Catalog above it — with data security and compliance solutions, billed through Azure meters for capacity, scanning, and governed assets.

Knowledge Catalog — previously Dataplex Universal Catalog, and the replacement for the retired Data Catalog — harvests metadata from BigQuery, Vertex AI, Pub/Sub, Bigtable, Cloud SQL, and AlloyDB into entries described by aspects, with natural-language search, lineage, and glossaries over the result.

Amazon DataZone is AWS's data management service for cataloguing, discovering, sharing, and governing data across AWS, on-premises, and third-party sources, organised around projects that publish assets and subscription requests that are approved through a governed workflow.

Confluent Schema Registry stores versioned Avro, Protobuf, and JSON schemas for Kafka topics and enforces compatibility rules when producers register changes.

Karapace is an Apache-2.0 drop-in replacement for Confluent Schema Registry (plus a Kafka REST proxy), maintained by Aiven.

DataHub is an open-source metadata platform providing search, discovery, and end-to-end lineage across warehouses, pipelines, and BI tools via push-based ingestion.

OpenMetadata is an open-source metadata platform combining catalog, lineage, data quality tests, and collaboration on a single schema-first foundation with many connectors.

Great Expectations (GX) is an open-source Python framework for declaring, validating, and documenting data quality expectations inside pipelines.

The AWS Glue Data Catalog is a fully managed, Hive-compatible metastore that resolves table definitions — including Iceberg tables — for engines across the AWS analytics stack.

Apache Polaris is an open-source catalog implementing the Iceberg REST specification, enabling interoperable, vendor-neutral table access across engines and clouds.