verosynthea-validator 0.2.0__tar.gz

verosynthea_validator-0.2.0/PKG-INFO

@@ -0,0 +1,178 @@
+Metadata-Version: 2.4
+Name: verosynthea-validator
+Version: 0.2.0
+Summary: Fairness testing for ML models using Australian demographic data
+Author-email: Verosynthea <hello@verosynthea.com>
+License: MIT
+Project-URL: Homepage, https://verosynthea.com/for-ai-labs
+Project-URL: Repository, https://github.com/verosynthea/verosynthea-validator
+Project-URL: Dataset, https://huggingface.co/datasets/vero-synthea/ausynth-sample
+Project-URL: Documentation, https://verosynthea.com/about
+Keywords: fairness,ml,demographics,australia,synthetic-data,census
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: pandas>=1.5
+Requires-Dist: numpy>=1.23
+Requires-Dist: huggingface_hub>=0.20.0
+Requires-Dist: datasets>=2.14.0
+Provides-Extra: hf
+Requires-Dist: datasets>=2.0; extra == "hf"
+Provides-Extra: paid
+Requires-Dist: httpx>=0.24; extra == "paid"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: scikit-learn>=1.2; extra == "dev"
+
+# verosynthea-validator
+
+## Quick Start: Test a US-trained Model on Australian Data
+
+This is a real-world fairness scenario. We take a standard classifier trained on US Census data and test how it performs on Australian demographics.
+
+```bash
+pip install verosynthea-validator
+```
+
+```python
+from verosynthea_validator import FairnessReport
+from verosynthea_validator.demos import (
+    load_us_adult_baseline,
+    load_ausynth_test_set,
+)
+
+# Load a US-trained income classifier and Australian test data
+model = load_us_adult_baseline()
+au_data = load_ausynth_test_set()
+
+# Run fairness audit
+report = FairnessReport(
+    model=model,
+    target_column="income_above_threshold",
+    protected_attributes=["SEXP", "BPLP", "AGE5P"],
+)
+report.run(test_data=au_data)
+report.show()
+```
+
+### What you'll see
+
+- **Country-of-birth bias gap (~30%):** The US-trained model handles US-typical birth countries well, others poorly
+- **Income threshold miscalibration:** $50K USD doesn't map cleanly to Australian income distributions
+- **Occupation bias (~18%):** Australian occupation categories don't align with UCI Adult codes
+
+This is the standard fairness-testing scenario for Australian deployments: models trained on US data need to be validated against Australian populations before production use.
+
+### What just happened?
+
+The [UCI Adult Income dataset](https://archive.ics.uci.edu/dataset/2/adult) is the canonical fairness benchmark in ML, but it's US Census data from 1994. When you run a model trained on it against Australian population data, the validator surfaces the bias gaps that come from the distribution mismatch.
+
+For your own models, replace `load_us_adult_baseline()` with your model and `load_ausynth_test_set()` with your test data or an [AUSynth subset](https://huggingface.co/datasets/vero-synthea/ausynth-sample).
+
+---
+
+
+## CI/CD gate
+
+```python
+from verosynthea_validator import assert_fair
+
+# Fails the build if any group accuracy gap > 5%
+assert_fair(test_data, "label", "prediction", max_accuracy_gap=0.05)
+```
+
+In pytest:
+
+```python
+def test_model_fairness():
+    predictions = model.predict(test_data)
+    test_data["y_pred"] = predictions
+    assert_fair(
+        test_data, "y_true", "y_pred",
+        protected_columns=["SEXP", "BPLP", "profile_name"],
+        max_accuracy_gap=0.05,
+        max_demographic_parity_gap=0.10,
+    )
+```
+
+## What it measures
+
+For each protected column (e.g. sex, birthplace, demographic profile), the validator computes:
+
+| Metric | What it checks |
+|--------|---------------|
+| **Accuracy gap** | Max accuracy difference between any two groups |
+| **Demographic parity gap** | Max difference in selection rate (P(y_pred=1)) |
+| **Equalised odds gap** | Max difference in true positive rate or false positive rate |
+
+Groups smaller than 30 observations are excluded (configurable via `min_group_size`).
+
+## Why this instead of fairlearn or aif360?
+
+Those are general-purpose fairness frameworks. This package is purpose-built for Australian demographics:
+
+- **Pre-loaded demographic data.** The free tier includes 5,000 synthetic individuals from [AUSynth](https://huggingface.co/datasets/vero-synthea/ausynth-sample) with 25 Census-calibrated variables. No need to source your own protected attributes.
+- **8 demographic profiles.** AUSynth clusters every person into one of 8 profiles (High-earning professionals, Young singles, Retired, etc.) — a richer protected attribute than just age or sex.
+- **Australia-specific calibration.** Variables match ABS Census 2021 categories exactly. Income brackets, occupation codes, education levels, birthplace regions — all in Australian standard classifications.
+- **One-line CI gate.** `assert_fair()` drops into pytest with zero configuration.
+
+## Data tiers
+
+| Tier | Data | Cost |
+|------|------|------|
+| **Free** | 5,000-row Paddington 4064 sample from [Hugging Face](https://huggingface.co/datasets/vero-synthea/ausynth-sample) | $0 |
+| **Paid** | Full national dataset (32M individuals, 15,352 suburbs) via API | [verosynthea.com](https://verosynthea.com) |
+
+```python
+from verosynthea_validator import load_ausynth_sample
+
+# Free tier (downloads from HF on first call)
+df = load_ausynth_sample()
+
+# Paid tier
+df = load_ausynth_sample(api_key="vero_...", geography="bondi-2026-nsw")
+```
+
+## The 8 demographic profiles
+
+| ID | Name | Typical characteristics |
+|----|------|------------------------|
+| 0 | Labourers and operators | Blue-collar, lower income |
+| 1 | Young singles and non-workers | Under 25, students, NILF |
+| 2 | Children | Under 15 |
+| 3 | Non-earning dependants | Adults not in workforce |
+| 4 | Trades and technical workers | Certificate-qualified, mid income |
+| 5 | Established partnered households | Married, mid-career |
+| 6 | Retired and semi-retired | Over 60, pension income |
+| 7 | High-earning professionals | Degree-qualified, professional occupations |
+
+## Installation
+
+```bash
+pip install verosynthea-validator          # core (pandas + numpy)
+pip install verosynthea-validator[hf]     # + Hugging Face datasets loader
+pip install verosynthea-validator[paid]   # + httpx for API access
+pip install verosynthea-validator[dev]    # + pytest + sklearn for development
+```
+
+## Links
+
+- **Dataset:** [vero-synthea/ausynth-sample](https://huggingface.co/datasets/vero-synthea/ausynth-sample) on Hugging Face
+- **Full product:** [verosynthea.com](https://verosynthea.com)
+- **Methodology:** [verosynthea.com/about](https://verosynthea.com/about)
+
+## Citation
+
+```
+Verosynthea AUSynth (2026). Synthetic Australian Census Data.
+https://verosynthea.com
+```
+
+## License
+
+MIT

verosynthea_validator-0.2.0/README.md

@@ -0,0 +1,147 @@
+# verosynthea-validator
+
+## Quick Start: Test a US-trained Model on Australian Data
+
+This is a real-world fairness scenario. We take a standard classifier trained on US Census data and test how it performs on Australian demographics.
+
+```bash
+pip install verosynthea-validator
+```
+
+```python
+from verosynthea_validator import FairnessReport
+from verosynthea_validator.demos import (
+    load_us_adult_baseline,
+    load_ausynth_test_set,
+)
+
+# Load a US-trained income classifier and Australian test data
+model = load_us_adult_baseline()
+au_data = load_ausynth_test_set()
+
+# Run fairness audit
+report = FairnessReport(
+    model=model,
+    target_column="income_above_threshold",
+    protected_attributes=["SEXP", "BPLP", "AGE5P"],
+)
+report.run(test_data=au_data)
+report.show()
+```
+
+### What you'll see
+
+- **Country-of-birth bias gap (~30%):** The US-trained model handles US-typical birth countries well, others poorly
+- **Income threshold miscalibration:** $50K USD doesn't map cleanly to Australian income distributions
+- **Occupation bias (~18%):** Australian occupation categories don't align with UCI Adult codes
+
+This is the standard fairness-testing scenario for Australian deployments: models trained on US data need to be validated against Australian populations before production use.
+
+### What just happened?
+
+The [UCI Adult Income dataset](https://archive.ics.uci.edu/dataset/2/adult) is the canonical fairness benchmark in ML, but it's US Census data from 1994. When you run a model trained on it against Australian population data, the validator surfaces the bias gaps that come from the distribution mismatch.
+
+For your own models, replace `load_us_adult_baseline()` with your model and `load_ausynth_test_set()` with your test data or an [AUSynth subset](https://huggingface.co/datasets/vero-synthea/ausynth-sample).
+
+---
+
+
+## CI/CD gate
+
+```python
+from verosynthea_validator import assert_fair
+
+# Fails the build if any group accuracy gap > 5%
+assert_fair(test_data, "label", "prediction", max_accuracy_gap=0.05)
+```
+
+In pytest:
+
+```python
+def test_model_fairness():
+    predictions = model.predict(test_data)
+    test_data["y_pred"] = predictions
+    assert_fair(
+        test_data, "y_true", "y_pred",
+        protected_columns=["SEXP", "BPLP", "profile_name"],
+        max_accuracy_gap=0.05,
+        max_demographic_parity_gap=0.10,
+    )
+```
+
+## What it measures
+
+For each protected column (e.g. sex, birthplace, demographic profile), the validator computes:
+
+| Metric | What it checks |
+|--------|---------------|
+| **Accuracy gap** | Max accuracy difference between any two groups |
+| **Demographic parity gap** | Max difference in selection rate (P(y_pred=1)) |
+| **Equalised odds gap** | Max difference in true positive rate or false positive rate |
+
+Groups smaller than 30 observations are excluded (configurable via `min_group_size`).
+
+## Why this instead of fairlearn or aif360?
+
+Those are general-purpose fairness frameworks. This package is purpose-built for Australian demographics:
+
+- **Pre-loaded demographic data.** The free tier includes 5,000 synthetic individuals from [AUSynth](https://huggingface.co/datasets/vero-synthea/ausynth-sample) with 25 Census-calibrated variables. No need to source your own protected attributes.
+- **8 demographic profiles.** AUSynth clusters every person into one of 8 profiles (High-earning professionals, Young singles, Retired, etc.) — a richer protected attribute than just age or sex.
+- **Australia-specific calibration.** Variables match ABS Census 2021 categories exactly. Income brackets, occupation codes, education levels, birthplace regions — all in Australian standard classifications.
+- **One-line CI gate.** `assert_fair()` drops into pytest with zero configuration.
+
+## Data tiers
+
+| Tier | Data | Cost |
+|------|------|------|
+| **Free** | 5,000-row Paddington 4064 sample from [Hugging Face](https://huggingface.co/datasets/vero-synthea/ausynth-sample) | $0 |
+| **Paid** | Full national dataset (32M individuals, 15,352 suburbs) via API | [verosynthea.com](https://verosynthea.com) |
+
+```python
+from verosynthea_validator import load_ausynth_sample
+
+# Free tier (downloads from HF on first call)
+df = load_ausynth_sample()
+
+# Paid tier
+df = load_ausynth_sample(api_key="vero_...", geography="bondi-2026-nsw")
+```
+
+## The 8 demographic profiles
+
+| ID | Name | Typical characteristics |
+|----|------|------------------------|
+| 0 | Labourers and operators | Blue-collar, lower income |
+| 1 | Young singles and non-workers | Under 25, students, NILF |
+| 2 | Children | Under 15 |
+| 3 | Non-earning dependants | Adults not in workforce |
+| 4 | Trades and technical workers | Certificate-qualified, mid income |
+| 5 | Established partnered households | Married, mid-career |
+| 6 | Retired and semi-retired | Over 60, pension income |
+| 7 | High-earning professionals | Degree-qualified, professional occupations |
+
+## Installation
+
+```bash
+pip install verosynthea-validator          # core (pandas + numpy)
+pip install verosynthea-validator[hf]     # + Hugging Face datasets loader
+pip install verosynthea-validator[paid]   # + httpx for API access
+pip install verosynthea-validator[dev]    # + pytest + sklearn for development
+```
+
+## Links
+
+- **Dataset:** [vero-synthea/ausynth-sample](https://huggingface.co/datasets/vero-synthea/ausynth-sample) on Hugging Face
+- **Full product:** [verosynthea.com](https://verosynthea.com)
+- **Methodology:** [verosynthea.com/about](https://verosynthea.com/about)
+
+## Citation
+
+```
+Verosynthea AUSynth (2026). Synthetic Australian Census Data.
+https://verosynthea.com
+```
+
+## License
+
+MIT

verosynthea_validator-0.2.0/pyproject.toml

@@ -0,0 +1,40 @@
+[build-system]
+requires = ["setuptools>=68.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "verosynthea-validator"
+version = "0.2.0"
+description = "Fairness testing for ML models using Australian demographic data"
+readme = "README.md"
+license = {text = "MIT"}
+requires-python = ">=3.9"
+authors = [
+    {name = "Verosynthea", email = "hello@verosynthea.com"},
+]
+keywords = ["fairness", "ml", "demographics", "australia", "synthetic-data", "census"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+dependencies = [
+    "pandas>=1.5",
+    "numpy>=1.23",
+    "huggingface_hub>=0.20.0",
+    "datasets>=2.14.0",
+]
+
+[project.optional-dependencies]
+hf = ["datasets>=2.0"]
+paid = ["httpx>=0.24"]
+dev = ["pytest>=7.0", "scikit-learn>=1.2"]
+
+[project.urls]
+Homepage = "https://verosynthea.com/for-ai-labs"
+Repository = "https://github.com/verosynthea/verosynthea-validator"
+Dataset = "https://huggingface.co/datasets/vero-synthea/ausynth-sample"
+Documentation = "https://verosynthea.com/about"

verosynthea_validator-0.2.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

verosynthea_validator-0.2.0/tests/test_fairness.py

@@ -0,0 +1,187 @@
+"""
+Tests for verosynthea-validator fairness metrics.
+
+Covers:
+- FairnessReport with a fair model (should pass)
+- FairnessReport with a biased model (should fail)
+- assert_fair CI helper
+- Edge cases: small groups, single group, missing columns
+- All 8 demographic profiles as protected attribute
+"""
+import numpy as np
+import pandas as pd
+import pytest
+
+from verosynthea_validator import FairnessReport, assert_fair
+from verosynthea_validator.assertions import FairnessAssertionError
+
+
+@pytest.fixture
+def sample_data():
+    """Create a synthetic dataset mimicking AUSynth structure."""
+    np.random.seed(42)
+    n = 2000
+    profiles = np.random.choice(
+        ["High-earning professionals", "Young singles and non-workers",
+         "Established partnered households", "Trades and technical workers",
+         "Retired and semi-retired", "Labourers and operators",
+         "Non-earning dependants", "Children"],
+        size=n,
+        p=[0.35, 0.20, 0.15, 0.10, 0.08, 0.07, 0.03, 0.02],
+    )
+    return pd.DataFrame({
+        "SEXP": np.random.choice(["Male", "Female"], n),
+        "BPLP": np.random.choice(
+            ["Oceania and Antarctica", "North-West Europe", "South-East Asia"],
+            n, p=[0.6, 0.2, 0.2],
+        ),
+        "profile_name": profiles,
+        "profile_id": pd.Categorical(profiles).codes,
+        "y_true": np.random.binomial(1, 0.4, n),
+    })
+
+
+@pytest.fixture
+def fair_predictions(sample_data):
+    """A model with roughly equal accuracy across groups."""
+    np.random.seed(42)
+    noise = np.random.normal(0, 0.2, len(sample_data))
+    sample_data["y_pred"] = (
+        (sample_data["y_true"] + noise) > 0.5
+    ).astype(int)
+    return sample_data
+
+
+@pytest.fixture
+def biased_predictions(sample_data):
+    """A model that performs worse for South-East Asian birthplace."""
+    np.random.seed(42)
+    noise = np.random.normal(0, 0.2, len(sample_data))
+    bias = np.where(sample_data["BPLP"] == "South-East Asia", 0.4, 0)
+    sample_data["y_pred"] = (
+        (sample_data["y_true"] + noise + bias) > 0.5
+    ).astype(int)
+    return sample_data
+
+
+class TestFairnessReport:
+    def test_fair_model_passes(self, fair_predictions):
+        report = FairnessReport(
+            fair_predictions, "y_true", "y_pred",
+            protected_columns=["SEXP"],
+        )
+        results = report.run()
+        assert results.n_total == 2000
+        assert results.overall_accuracy > 0.5
+        assert len(results.results) == 1
+        assert results.results[0].accuracy_gap < 0.10
+
+    def test_biased_model_detected(self, biased_predictions):
+        report = FairnessReport(
+            biased_predictions, "y_true", "y_pred",
+            protected_columns=["BPLP"],
+        )
+        results = report.run()
+        bplp = results.results[0]
+        assert bplp.accuracy_gap > 0.05
+        assert bplp.column == "BPLP"
+
+    def test_multiple_protected_columns(self, fair_predictions):
+        report = FairnessReport(
+            fair_predictions, "y_true", "y_pred",
+            protected_columns=["SEXP", "BPLP", "profile_name"],
+        )
+        results = report.run()
+        assert len(results.results) == 3
+        columns = {r.column for r in results.results}
+        assert columns == {"SEXP", "BPLP", "profile_name"}
+
+    def test_profiles_as_protected(self, fair_predictions):
+        report = FairnessReport(
+            fair_predictions, "y_true", "y_pred",
+            protected_columns=["profile_name"],
+            min_group_size=20,
+        )
+        results = report.run()
+        profile_result = results.results[0]
+        assert profile_result.column == "profile_name"
+        assert len(profile_result.groups) >= 5
+
+    def test_summary_output(self, fair_predictions):
+        report = FairnessReport(
+            fair_predictions, "y_true", "y_pred",
+            protected_columns=["SEXP"],
+        )
+        results = report.run()
+        summary = results.summary()
+        assert "Fairness Report" in summary
+        assert "accuracy gap" in summary.lower()
+
+    def test_to_dataframe(self, fair_predictions):
+        report = FairnessReport(
+            fair_predictions, "y_true", "y_pred",
+            protected_columns=["SEXP", "BPLP"],
+        )
+        df = report.run().to_dataframe()
+        assert isinstance(df, pd.DataFrame)
+        assert "protected_column" in df.columns
+        assert "accuracy" in df.columns
+        assert len(df) >= 4
+
+    def test_min_group_size_filter(self, sample_data):
+        sample_data["y_pred"] = sample_data["y_true"]
+        sample_data["RARE"] = "common"
+        sample_data.loc[:4, "RARE"] = "rare"
+        report = FairnessReport(
+            sample_data, "y_true", "y_pred",
+            protected_columns=["RARE"],
+            min_group_size=30,
+        )
+        results = report.run()
+        assert len(results.results) == 0
+
+    def test_missing_column_raises(self, sample_data):
+        sample_data["y_pred"] = 0
+        with pytest.raises(ValueError, match="not found"):
+            FairnessReport(
+                sample_data, "y_true", "y_pred",
+                protected_columns=["NONEXISTENT"],
+            )
+
+
+class TestAssertFair:
+    def test_fair_model_passes(self, fair_predictions):
+        assert_fair(
+            fair_predictions, "y_true", "y_pred",
+            protected_columns=["SEXP"],
+            max_accuracy_gap=0.10,
+        )
+
+    def test_biased_model_fails(self, biased_predictions):
+        with pytest.raises(FairnessAssertionError) as exc_info:
+            assert_fair(
+                biased_predictions, "y_true", "y_pred",
+                protected_columns=["BPLP"],
+                max_accuracy_gap=0.05,
+            )
+        assert "failed fairness check" in str(exc_info.value)
+        assert hasattr(exc_info.value, "details")
+        assert len(exc_info.value.details["failures"]) > 0
+
+    def test_custom_thresholds(self, biased_predictions):
+        with pytest.raises(FairnessAssertionError):
+            assert_fair(
+                biased_predictions, "y_true", "y_pred",
+                protected_columns=["BPLP"],
+                max_accuracy_gap=0.01,
+                max_demographic_parity_gap=0.01,
+            )
+
+    def test_relaxed_thresholds_pass(self, biased_predictions):
+        assert_fair(
+            biased_predictions, "y_true", "y_pred",
+            protected_columns=["BPLP"],
+            max_accuracy_gap=0.50,
+            max_demographic_parity_gap=0.50,
+            max_equalised_odds_gap=0.50,
+        )

verosynthea_validator-0.2.0/verosynthea_validator/__init__.py

@@ -0,0 +1,22 @@
+"""
+verosynthea-validator — Fairness testing for ML models using Australian demographic data.
+
+Quick start:
+    from verosynthea_validator import FairnessReport, assert_fair
+
+    report = FairnessReport(data, y_true="label", y_pred="prediction",
+                            protected_columns=["SEXP", "BPLP"])
+    results = report.run()
+    print(results.summary())
+
+    # CI/CD gate:
+    assert_fair(data, "label", "prediction", max_accuracy_gap=0.05)
+"""
+
+from verosynthea_validator.fairness import FairnessReport, FairnessResults
+from verosynthea_validator.assertions import assert_fair
+from verosynthea_validator.data import load_ausynth_sample
+from .demos import load_us_adult_baseline, load_ausynth_test_set
+
+__version__ = "0.1.0"
+__all__ = ["FairnessReport", "FairnessResults", "assert_fair", "load_ausynth_sample"]

verosynthea_validator-0.2.0/verosynthea_validator/assertions.py

@@ -0,0 +1,100 @@
+"""
+CI/CD assertion helpers for fairness gating.
+
+Usage in pytest or CI:
+    from verosynthea_validator import assert_fair
+    assert_fair(test_data, "y_true", "y_pred", max_accuracy_gap=0.05)
+"""
+from __future__ import annotations
+
+from typing import Optional, Sequence
+
+import pandas as pd
+
+from verosynthea_validator.fairness import FairnessReport
+
+
+class FairnessAssertionError(AssertionError):
+    """Raised when a model fails a fairness check."""
+
+    def __init__(self, message: str, details: dict):
+        super().__init__(message)
+        self.details = details
+
+
+def assert_fair(
+    data: pd.DataFrame,
+    y_true: str,
+    y_pred: str,
+    protected_columns: Optional[Sequence[str]] = None,
+    max_accuracy_gap: float = 0.05,
+    max_demographic_parity_gap: float = 0.10,
+    max_equalised_odds_gap: float = 0.10,
+    min_group_size: int = 30,
+) -> None:
+    """Assert that a model's predictions are fair across demographic groups.
+
+    Raises FairnessAssertionError if any threshold is exceeded. Designed
+    for use in pytest tests and CI/CD pipelines.
+
+    Parameters
+    ----------
+    data : pd.DataFrame
+        Dataset with predictions and AUSynth demographic columns.
+    y_true : str
+        Ground-truth binary label column.
+    y_pred : str
+        Predicted binary label column.
+    protected_columns : list[str], optional
+        Columns to check. Default: ["SEXP", "BPLP", "profile_name"].
+    max_accuracy_gap : float
+        Maximum allowed accuracy difference between any two groups (default 0.05).
+    max_demographic_parity_gap : float
+        Maximum allowed selection rate difference (default 0.10).
+    max_equalised_odds_gap : float
+        Maximum allowed TPR or FPR difference (default 0.10).
+    min_group_size : int
+        Minimum group size to include in analysis (default 30).
+
+    Raises
+    ------
+    FairnessAssertionError
+        If any fairness threshold is exceeded. The error's `.details` dict
+        contains the full results for debugging.
+    """
+    report = FairnessReport(
+        data=data,
+        y_true=y_true,
+        y_pred=y_pred,
+        protected_columns=protected_columns,
+        min_group_size=min_group_size,
+    )
+    results = report.run()
+
+    failures = []
+    for r in results.results:
+        if r.accuracy_gap > max_accuracy_gap:
+            failures.append(
+                f"{r.column}: accuracy gap {r.accuracy_gap:.3f} > {max_accuracy_gap}"
+            )
+        if r.demographic_parity_gap > max_demographic_parity_gap:
+            failures.append(
+                f"{r.column}: demographic parity gap {r.demographic_parity_gap:.3f} "
+                f"> {max_demographic_parity_gap}"
+            )
+        if r.equalised_odds_gap > max_equalised_odds_gap:
+            failures.append(
+                f"{r.column}: equalised odds gap {r.equalised_odds_gap:.3f} "
+                f"> {max_equalised_odds_gap}"
+            )
+
+    if failures:
+        msg = (
+            f"Model failed fairness check on {len(failures)} metric(s):\n"
+            + "\n".join(f"  - {f}" for f in failures)
+            + "\n\nFull report:\n" + results.summary()
+        )
+        raise FairnessAssertionError(msg, {
+            "failures": failures,
+            "results": [r.to_dict() for r in results.results],
+        })