Deequ
Deequ is an Apache-2.0 library from AWS Labs for defining "unit tests for data" on Apache Spark: constraints and metrics are computed as part of the Spark job itself, which makes quality checks scale with the data rather than sampling it.
LicenseOpen source (permissive)· Apache-2.0
DeploymentSelf-hosted
PricingFree
Operational complexityLow
WorkloadBatch
LanguageScala
Use it when
- The data lives in Spark and is too large to sample: checks compile to Spark jobs and run inside the pipeline that already holds the data.
- You want metric histories, anomaly detection against them, and constraint suggestion from profiles, all from a library.
- Bad batches must be quarantined before applications read them: a run returns a status the surrounding job branches on.
- Declarative rules should drive the same engine: it speaks DQDL, the language behind AWS Glue Data Quality, and PyDeequ covers Python.
Think twice when
- There is no Spark in the picture; pandera fits Python dataframes and Great Expectations fits SQL pipelines better.
- You want a UI, alerting, or incident workflow; a library has none, by design.
- Version coupling hurts: Deequ lines pin to specific Spark and Scala versions.
How it runs
A JAR added to the Spark job: VerificationSuite chains constraints (completeness, uniqueness, ranges, quantiles) over a DataFrame, metrics persist to a repository for anomaly detection, and whatever wraps the job acts on the result. Apache-2.0 from AWS Labs with no commercial edition; the only cost is the cluster it already runs on.
Details
CompareHow Deequ answers the questions Data Quality turns on.
Data Quality
| How it works | |
| Authoring model | A Scala API inside the Spark job, VerificationSuite().onData(df).addCheck(Check(...)) chaining constraints like isComplete, isUnique, isContainedIn and hasApproxQuantile, each carrying its own assertion function. Or DQDL, AWS's declarative rule language, driving the same engine from configuration; PyDeequ wraps it for Python |
| Detection model | Declared constraints evaluated as Spark jobs, the suite computes metrics over the data and then applies your assertion functions to those metrics. A metrics repository persists the measures so anomaly detection can run against their history, and constraint suggestion proposes checks from a profile rather than waiting for anyone to write them |
| Built-in monitors | Constraints over computed metrics: completeness, uniqueness, distinctness, size, containment, ranges, approximate quantiles and pattern or URL matching from Scala; DQDL adds the named catalogue of around thirty, including entropy, correlation, skewness, kurtosis, variance, primary key, column length, freshness, custom SQL, row-count and schema match against a reference dataset, and composite rules joined by and or or |
| Prevention or detection | Prevention, and the README says why in its own words, state the assumptions as a unit test, and where the data has errors you can "quarantine and fix it, before we feed it to an application". A run returns a CheckStatus and Error-level checks are what the surrounding job branches on |
| Lineage | None: it sees a DataFrame and knows nothing about where it came from |
| Alerts and delivery | None. A run returns a result object, and whatever notices a failure is whatever wraps the Spark job |
| Incidents and ownership | None. No incident, no owner, no status, a library has no opinion about who fixes what, which is precisely the trade against the platforms in this capability |
| Running it | |
| What you run | Inside the Spark job as a JAR, checks translate into Spark jobs run by the same distributed engine that processes the data, so there is no extract, no separate query budget and no sampling compromise on tables that fit nowhere else. Deequ 2.x pins to Spark 3.1 on Java 8 and Scala 2.12, with legacy lines back to Spark 2.2 |
| Connections | |
| Connectors | None, and none needed: it takes a Spark DataFrame, so its reach is whatever Spark already reads. The only row in this capability with no connector list at all |
| Cost | |
| Billing unit | Nothing to pay: Apache-2.0 out of AWS Labs with no commercial edition, one of only two rows here with no paid tier at all. The cost is the Spark cluster, and that belongs to a different row |
Alternatives to Deequ
