Data Quality & Observability
Data architecture technologies for frameworks that declare, validate, and monitor expectations about data in production.
Data contracts in YAML, run from the pipeline or watched from the cloud
Data Quality
Soda is a data quality platform built on Soda Core, a Python library and CLI that runs checks and data contracts inside pipelines, with Soda Cloud for triage and collaboration and a hosted or self-hosted Runner for scheduled observability.
Data observability that installs as a dbt package
Data Quality
Elementary is an Apache-2.0 data observability tool built for dbt projects: a dbt package and CLI collect test results and run anomaly detection on freshness, volume, and custom metrics, with Elementary Cloud adding column-level lineage, ML monitors, and incident workflows.
Unit tests for data, computed inside the Spark job
Data Quality
Deequ is an Apache-2.0 library from AWS Labs for defining "unit tests for data" on Apache Spark: constraints and metrics are computed as part of the Spark job itself, which makes quality checks scale with the data rather than sampling it.
Schema validation for dataframes, in the code that builds them
Data Quality
pandera is an MIT-licensed Python library for validating dataframes: schemas are declared as objects or as typed classes and checked at runtime across pandas, polars, pyspark, dask, modin, ibis, and geopandas.
Data observability that writes its own monitors
Data Quality
Monte Carlo is a commercial data observability platform: it monitors tables automatically rather than waiting for hand-written tests, enriches alerts with lineage and root-cause context, and routes incidents through triage workflows.
Machine-learned monitoring that can run inside your own cloud
Data Quality
Anomalo is a commercial data quality platform that applies machine learning to detect anomalies without hand-written rules, and can be deployed either as SaaS or entirely inside the customer's own VPC so that data never leaves their environment.
Diff the data a change produces, before the change ships
Data Quality
Datafold is a commercial platform built on value-level data diffing: it compares the output of a proposed change against production in CI, validates parity during warehouse migrations, and monitors metrics for deviation, deployable as SaaS or inside the customer's own cloud.
Declarative data quality testing framework
Data Quality
Great Expectations (GX) is an open-source Python framework for declaring, validating, and documenting data quality expectations inside pipelines.