verosynthea-validator 0.1.0__tar.gz

verosynthea_validator-0.1.0/PKG-INFO

@@ -0,0 +1,169 @@
+Metadata-Version: 2.4
+Name: verosynthea-validator
+Version: 0.1.0
+Summary: Fairness testing for ML models using Australian demographic data
+Author-email: Verosynthea <hello@verosynthea.com>
+License: MIT
+Project-URL: Homepage, https://verosynthea.com/for-ai-labs
+Project-URL: Repository, https://github.com/verosynthea/verosynthea-validator
+Project-URL: Dataset, https://huggingface.co/datasets/vero-synthea/ausynth-sample
+Project-URL: Documentation, https://verosynthea.com/about
+Keywords: fairness,ml,demographics,australia,synthetic-data,census
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: pandas>=1.5
+Requires-Dist: numpy>=1.23
+Provides-Extra: hf
+Requires-Dist: datasets>=2.0; extra == "hf"
+Provides-Extra: paid
+Requires-Dist: httpx>=0.24; extra == "paid"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: scikit-learn>=1.2; extra == "dev"
+
+# verosynthea-validator
+
+Fairness testing for ML models using real Australian demographic data. One line to check whether your model treats demographic groups equally.
+
+```bash
+pip install verosynthea-validator
+```
+
+```python
+from verosynthea_validator import FairnessReport
+
+report = FairnessReport(
+    data=test_data,
+    y_true="label",
+    y_pred="prediction",
+    protected_columns=["SEXP", "BPLP", "profile_name"],
+)
+results = report.run()
+print(results.summary())
+```
+
+Output:
+
+```
+Fairness Report (n=5,000, overall accuracy=0.847)
+============================================================
+
+[PASS] SEXP (2 groups, smallest n=2,451)
+  Accuracy gap:           0.012
+  Demographic parity gap: 0.008
+  Equalised odds gap:     0.015
+
+[FAIL] BPLP (3 groups, smallest n=312)
+  Accuracy gap:           0.073
+  Demographic parity gap: 0.091
+  Equalised odds gap:     0.064
+
+============================================================
+Overall: FAIL (worst gap: 0.073 on BPLP)
+```
+
+## CI/CD gate
+
+```python
+from verosynthea_validator import assert_fair
+
+# Fails the build if any group accuracy gap > 5%
+assert_fair(test_data, "label", "prediction", max_accuracy_gap=0.05)
+```
+
+In pytest:
+
+```python
+def test_model_fairness():
+    predictions = model.predict(test_data)
+    test_data["y_pred"] = predictions
+    assert_fair(
+        test_data, "y_true", "y_pred",
+        protected_columns=["SEXP", "BPLP", "profile_name"],
+        max_accuracy_gap=0.05,
+        max_demographic_parity_gap=0.10,
+    )
+```
+
+## What it measures
+
+For each protected column (e.g. sex, birthplace, demographic profile), the validator computes:
+
+| Metric | What it checks |
+|--------|---------------|
+| **Accuracy gap** | Max accuracy difference between any two groups |
+| **Demographic parity gap** | Max difference in selection rate (P(y_pred=1)) |
+| **Equalised odds gap** | Max difference in true positive rate or false positive rate |
+
+Groups smaller than 30 observations are excluded (configurable via `min_group_size`).
+
+## Why this instead of fairlearn or aif360?
+
+Those are general-purpose fairness frameworks. This package is purpose-built for Australian demographics:
+
+- **Pre-loaded demographic data.** The free tier includes 5,000 synthetic individuals from [AUSynth](https://huggingface.co/datasets/vero-synthea/ausynth-sample) with 25 Census-calibrated variables. No need to source your own protected attributes.
+- **8 demographic profiles.** AUSynth clusters every person into one of 8 profiles (High-earning professionals, Young singles, Retired, etc.) — a richer protected attribute than just age or sex.
+- **Australia-specific calibration.** Variables match ABS Census 2021 categories exactly. Income brackets, occupation codes, education levels, birthplace regions — all in Australian standard classifications.
+- **One-line CI gate.** `assert_fair()` drops into pytest with zero configuration.
+
+## Data tiers
+
+| Tier | Data | Cost |
+|------|------|------|
+| **Free** | 5,000-row Paddington 4064 sample from [Hugging Face](https://huggingface.co/datasets/vero-synthea/ausynth-sample) | $0 |
+| **Paid** | Full national dataset (32M individuals, 15,352 suburbs) via API | [verosynthea.com](https://verosynthea.com) |
+
+```python
+from verosynthea_validator import load_ausynth_sample
+
+# Free tier (downloads from HF on first call)
+df = load_ausynth_sample()
+
+# Paid tier
+df = load_ausynth_sample(api_key="vero_...", geography="bondi-2026-nsw")
+```
+
+## The 8 demographic profiles
+
+| ID | Name | Typical characteristics |
+|----|------|------------------------|
+| 0 | Labourers and operators | Blue-collar, lower income |
+| 1 | Young singles and non-workers | Under 25, students, NILF |
+| 2 | Children | Under 15 |
+| 3 | Non-earning dependants | Adults not in workforce |
+| 4 | Trades and technical workers | Certificate-qualified, mid income |
+| 5 | Established partnered households | Married, mid-career |
+| 6 | Retired and semi-retired | Over 60, pension income |
+| 7 | High-earning professionals | Degree-qualified, professional occupations |
+
+## Installation
+
+```bash
+pip install verosynthea-validator          # core (pandas + numpy)
+pip install verosynthea-validator[hf]     # + Hugging Face datasets loader
+pip install verosynthea-validator[paid]   # + httpx for API access
+pip install verosynthea-validator[dev]    # + pytest + sklearn for development
+```
+
+## Links
+
+- **Dataset:** [vero-synthea/ausynth-sample](https://huggingface.co/datasets/vero-synthea/ausynth-sample) on Hugging Face
+- **Full product:** [verosynthea.com](https://verosynthea.com)
+- **Methodology:** [verosynthea.com/about](https://verosynthea.com/about)
+
+## Citation
+
+```
+Verosynthea AUSynth (2026). Synthetic Australian Census Data.
+https://verosynthea.com
+```
+
+## License
+
+MIT

verosynthea_validator-0.1.0/README.md

@@ -0,0 +1,140 @@
+# verosynthea-validator
+
+Fairness testing for ML models using real Australian demographic data. One line to check whether your model treats demographic groups equally.
+
+```bash
+pip install verosynthea-validator
+```
+
+```python
+from verosynthea_validator import FairnessReport
+
+report = FairnessReport(
+    data=test_data,
+    y_true="label",
+    y_pred="prediction",
+    protected_columns=["SEXP", "BPLP", "profile_name"],
+)
+results = report.run()
+print(results.summary())
+```
+
+Output:
+
+```
+Fairness Report (n=5,000, overall accuracy=0.847)
+============================================================
+
+[PASS] SEXP (2 groups, smallest n=2,451)
+  Accuracy gap:           0.012
+  Demographic parity gap: 0.008
+  Equalised odds gap:     0.015
+
+[FAIL] BPLP (3 groups, smallest n=312)
+  Accuracy gap:           0.073
+  Demographic parity gap: 0.091
+  Equalised odds gap:     0.064
+
+============================================================
+Overall: FAIL (worst gap: 0.073 on BPLP)
+```
+
+## CI/CD gate
+
+```python
+from verosynthea_validator import assert_fair
+
+# Fails the build if any group accuracy gap > 5%
+assert_fair(test_data, "label", "prediction", max_accuracy_gap=0.05)
+```
+
+In pytest:
+
+```python
+def test_model_fairness():
+    predictions = model.predict(test_data)
+    test_data["y_pred"] = predictions
+    assert_fair(
+        test_data, "y_true", "y_pred",
+        protected_columns=["SEXP", "BPLP", "profile_name"],
+        max_accuracy_gap=0.05,
+        max_demographic_parity_gap=0.10,
+    )
+```
+
+## What it measures
+
+For each protected column (e.g. sex, birthplace, demographic profile), the validator computes:
+
+| Metric | What it checks |
+|--------|---------------|
+| **Accuracy gap** | Max accuracy difference between any two groups |
+| **Demographic parity gap** | Max difference in selection rate (P(y_pred=1)) |
+| **Equalised odds gap** | Max difference in true positive rate or false positive rate |
+
+Groups smaller than 30 observations are excluded (configurable via `min_group_size`).
+
+## Why this instead of fairlearn or aif360?
+
+Those are general-purpose fairness frameworks. This package is purpose-built for Australian demographics:
+
+- **Pre-loaded demographic data.** The free tier includes 5,000 synthetic individuals from [AUSynth](https://huggingface.co/datasets/vero-synthea/ausynth-sample) with 25 Census-calibrated variables. No need to source your own protected attributes.
+- **8 demographic profiles.** AUSynth clusters every person into one of 8 profiles (High-earning professionals, Young singles, Retired, etc.) — a richer protected attribute than just age or sex.
+- **Australia-specific calibration.** Variables match ABS Census 2021 categories exactly. Income brackets, occupation codes, education levels, birthplace regions — all in Australian standard classifications.
+- **One-line CI gate.** `assert_fair()` drops into pytest with zero configuration.
+
+## Data tiers
+
+| Tier | Data | Cost |
+|------|------|------|
+| **Free** | 5,000-row Paddington 4064 sample from [Hugging Face](https://huggingface.co/datasets/vero-synthea/ausynth-sample) | $0 |
+| **Paid** | Full national dataset (32M individuals, 15,352 suburbs) via API | [verosynthea.com](https://verosynthea.com) |
+
+```python
+from verosynthea_validator import load_ausynth_sample
+
+# Free tier (downloads from HF on first call)
+df = load_ausynth_sample()
+
+# Paid tier
+df = load_ausynth_sample(api_key="vero_...", geography="bondi-2026-nsw")
+```
+
+## The 8 demographic profiles
+
+| ID | Name | Typical characteristics |
+|----|------|------------------------|
+| 0 | Labourers and operators | Blue-collar, lower income |
+| 1 | Young singles and non-workers | Under 25, students, NILF |
+| 2 | Children | Under 15 |
+| 3 | Non-earning dependants | Adults not in workforce |
+| 4 | Trades and technical workers | Certificate-qualified, mid income |
+| 5 | Established partnered households | Married, mid-career |
+| 6 | Retired and semi-retired | Over 60, pension income |
+| 7 | High-earning professionals | Degree-qualified, professional occupations |
+
+## Installation
+
+```bash
+pip install verosynthea-validator          # core (pandas + numpy)
+pip install verosynthea-validator[hf]     # + Hugging Face datasets loader
+pip install verosynthea-validator[paid]   # + httpx for API access
+pip install verosynthea-validator[dev]    # + pytest + sklearn for development
+```
+
+## Links
+
+- **Dataset:** [vero-synthea/ausynth-sample](https://huggingface.co/datasets/vero-synthea/ausynth-sample) on Hugging Face
+- **Full product:** [verosynthea.com](https://verosynthea.com)
+- **Methodology:** [verosynthea.com/about](https://verosynthea.com/about)
+
+## Citation
+
+```
+Verosynthea AUSynth (2026). Synthetic Australian Census Data.
+https://verosynthea.com
+```
+
+## License
+
+MIT

verosynthea_validator-0.1.0/pyproject.toml

@@ -0,0 +1,38 @@
+[build-system]
+requires = ["setuptools>=68.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "verosynthea-validator"
+version = "0.1.0"
+description = "Fairness testing for ML models using Australian demographic data"
+readme = "README.md"
+license = {text = "MIT"}
+requires-python = ">=3.9"
+authors = [
+    {name = "Verosynthea", email = "hello@verosynthea.com"},
+]
+keywords = ["fairness", "ml", "demographics", "australia", "synthetic-data", "census"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+dependencies = [
+    "pandas>=1.5",
+    "numpy>=1.23",
+]
+
+[project.optional-dependencies]
+hf = ["datasets>=2.0"]
+paid = ["httpx>=0.24"]
+dev = ["pytest>=7.0", "scikit-learn>=1.2"]
+
+[project.urls]
+Homepage = "https://verosynthea.com/for-ai-labs"
+Repository = "https://github.com/verosynthea/verosynthea-validator"
+Dataset = "https://huggingface.co/datasets/vero-synthea/ausynth-sample"
+Documentation = "https://verosynthea.com/about"

verosynthea_validator-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

verosynthea_validator-0.1.0/tests/test_fairness.py

@@ -0,0 +1,187 @@
+"""
+Tests for verosynthea-validator fairness metrics.
+
+Covers:
+- FairnessReport with a fair model (should pass)
+- FairnessReport with a biased model (should fail)
+- assert_fair CI helper
+- Edge cases: small groups, single group, missing columns
+- All 8 demographic profiles as protected attribute
+"""
+import numpy as np
+import pandas as pd
+import pytest
+
+from verosynthea_validator import FairnessReport, assert_fair
+from verosynthea_validator.assertions import FairnessAssertionError
+
+
+@pytest.fixture
+def sample_data():
+    """Create a synthetic dataset mimicking AUSynth structure."""
+    np.random.seed(42)
+    n = 2000
+    profiles = np.random.choice(
+        ["High-earning professionals", "Young singles and non-workers",
+         "Established partnered households", "Trades and technical workers",
+         "Retired and semi-retired", "Labourers and operators",
+         "Non-earning dependants", "Children"],
+        size=n,
+        p=[0.35, 0.20, 0.15, 0.10, 0.08, 0.07, 0.03, 0.02],
+    )
+    return pd.DataFrame({
+        "SEXP": np.random.choice(["Male", "Female"], n),
+        "BPLP": np.random.choice(
+            ["Oceania and Antarctica", "North-West Europe", "South-East Asia"],
+            n, p=[0.6, 0.2, 0.2],
+        ),
+        "profile_name": profiles,
+        "profile_id": pd.Categorical(profiles).codes,
+        "y_true": np.random.binomial(1, 0.4, n),
+    })
+
+
+@pytest.fixture
+def fair_predictions(sample_data):
+    """A model with roughly equal accuracy across groups."""
+    np.random.seed(42)
+    noise = np.random.normal(0, 0.2, len(sample_data))
+    sample_data["y_pred"] = (
+        (sample_data["y_true"] + noise) > 0.5
+    ).astype(int)
+    return sample_data
+
+
+@pytest.fixture
+def biased_predictions(sample_data):
+    """A model that performs worse for South-East Asian birthplace."""
+    np.random.seed(42)
+    noise = np.random.normal(0, 0.2, len(sample_data))
+    bias = np.where(sample_data["BPLP"] == "South-East Asia", 0.4, 0)
+    sample_data["y_pred"] = (
+        (sample_data["y_true"] + noise + bias) > 0.5
+    ).astype(int)
+    return sample_data
+
+
+class TestFairnessReport:
+    def test_fair_model_passes(self, fair_predictions):
+        report = FairnessReport(
+            fair_predictions, "y_true", "y_pred",
+            protected_columns=["SEXP"],
+        )
+        results = report.run()
+        assert results.n_total == 2000
+        assert results.overall_accuracy > 0.5
+        assert len(results.results) == 1
+        assert results.results[0].accuracy_gap < 0.10
+
+    def test_biased_model_detected(self, biased_predictions):
+        report = FairnessReport(
+            biased_predictions, "y_true", "y_pred",
+            protected_columns=["BPLP"],
+        )
+        results = report.run()
+        bplp = results.results[0]
+        assert bplp.accuracy_gap > 0.05
+        assert bplp.column == "BPLP"
+
+    def test_multiple_protected_columns(self, fair_predictions):
+        report = FairnessReport(
+            fair_predictions, "y_true", "y_pred",
+            protected_columns=["SEXP", "BPLP", "profile_name"],
+        )
+        results = report.run()
+        assert len(results.results) == 3
+        columns = {r.column for r in results.results}
+        assert columns == {"SEXP", "BPLP", "profile_name"}
+
+    def test_profiles_as_protected(self, fair_predictions):
+        report = FairnessReport(
+            fair_predictions, "y_true", "y_pred",
+            protected_columns=["profile_name"],
+            min_group_size=20,
+        )
+        results = report.run()
+        profile_result = results.results[0]
+        assert profile_result.column == "profile_name"
+        assert len(profile_result.groups) >= 5
+
+    def test_summary_output(self, fair_predictions):
+        report = FairnessReport(
+            fair_predictions, "y_true", "y_pred",
+            protected_columns=["SEXP"],
+        )
+        results = report.run()
+        summary = results.summary()
+        assert "Fairness Report" in summary
+        assert "accuracy gap" in summary.lower()
+
+    def test_to_dataframe(self, fair_predictions):
+        report = FairnessReport(
+            fair_predictions, "y_true", "y_pred",
+            protected_columns=["SEXP", "BPLP"],
+        )
+        df = report.run().to_dataframe()
+        assert isinstance(df, pd.DataFrame)
+        assert "protected_column" in df.columns
+        assert "accuracy" in df.columns
+        assert len(df) >= 4
+
+    def test_min_group_size_filter(self, sample_data):
+        sample_data["y_pred"] = sample_data["y_true"]
+        sample_data["RARE"] = "common"
+        sample_data.loc[:4, "RARE"] = "rare"
+        report = FairnessReport(
+            sample_data, "y_true", "y_pred",
+            protected_columns=["RARE"],
+            min_group_size=30,
+        )
+        results = report.run()
+        assert len(results.results) == 0
+
+    def test_missing_column_raises(self, sample_data):
+        sample_data["y_pred"] = 0
+        with pytest.raises(ValueError, match="not found"):
+            FairnessReport(
+                sample_data, "y_true", "y_pred",
+                protected_columns=["NONEXISTENT"],
+            )
+
+
+class TestAssertFair:
+    def test_fair_model_passes(self, fair_predictions):
+        assert_fair(
+            fair_predictions, "y_true", "y_pred",
+            protected_columns=["SEXP"],
+            max_accuracy_gap=0.10,
+        )
+
+    def test_biased_model_fails(self, biased_predictions):
+        with pytest.raises(FairnessAssertionError) as exc_info:
+            assert_fair(
+                biased_predictions, "y_true", "y_pred",
+                protected_columns=["BPLP"],
+                max_accuracy_gap=0.05,
+            )
+        assert "failed fairness check" in str(exc_info.value)
+        assert hasattr(exc_info.value, "details")
+        assert len(exc_info.value.details["failures"]) > 0
+
+    def test_custom_thresholds(self, biased_predictions):
+        with pytest.raises(FairnessAssertionError):
+            assert_fair(
+                biased_predictions, "y_true", "y_pred",
+                protected_columns=["BPLP"],
+                max_accuracy_gap=0.01,
+                max_demographic_parity_gap=0.01,
+            )
+
+    def test_relaxed_thresholds_pass(self, biased_predictions):
+        assert_fair(
+            biased_predictions, "y_true", "y_pred",
+            protected_columns=["BPLP"],
+            max_accuracy_gap=0.50,
+            max_demographic_parity_gap=0.50,
+            max_equalised_odds_gap=0.50,
+        )

verosynthea_validator-0.1.0/verosynthea_validator/__init__.py

@@ -0,0 +1,21 @@
+"""
+verosynthea-validator — Fairness testing for ML models using Australian demographic data.
+
+Quick start:
+    from verosynthea_validator import FairnessReport, assert_fair
+
+    report = FairnessReport(data, y_true="label", y_pred="prediction",
+                            protected_columns=["SEXP", "BPLP"])
+    results = report.run()
+    print(results.summary())
+
+    # CI/CD gate:
+    assert_fair(data, "label", "prediction", max_accuracy_gap=0.05)
+"""
+
+from verosynthea_validator.fairness import FairnessReport, FairnessResults
+from verosynthea_validator.assertions import assert_fair
+from verosynthea_validator.data import load_ausynth_sample
+
+__version__ = "0.1.0"
+__all__ = ["FairnessReport", "FairnessResults", "assert_fair", "load_ausynth_sample"]

verosynthea_validator-0.1.0/verosynthea_validator/assertions.py

@@ -0,0 +1,100 @@
+"""
+CI/CD assertion helpers for fairness gating.
+
+Usage in pytest or CI:
+    from verosynthea_validator import assert_fair
+    assert_fair(test_data, "y_true", "y_pred", max_accuracy_gap=0.05)
+"""
+from __future__ import annotations
+
+from typing import Optional, Sequence
+
+import pandas as pd
+
+from verosynthea_validator.fairness import FairnessReport
+
+
+class FairnessAssertionError(AssertionError):
+    """Raised when a model fails a fairness check."""
+
+    def __init__(self, message: str, details: dict):
+        super().__init__(message)
+        self.details = details
+
+
+def assert_fair(
+    data: pd.DataFrame,
+    y_true: str,
+    y_pred: str,
+    protected_columns: Optional[Sequence[str]] = None,
+    max_accuracy_gap: float = 0.05,
+    max_demographic_parity_gap: float = 0.10,
+    max_equalised_odds_gap: float = 0.10,
+    min_group_size: int = 30,
+) -> None:
+    """Assert that a model's predictions are fair across demographic groups.
+
+    Raises FairnessAssertionError if any threshold is exceeded. Designed
+    for use in pytest tests and CI/CD pipelines.
+
+    Parameters
+    ----------
+    data : pd.DataFrame
+        Dataset with predictions and AUSynth demographic columns.
+    y_true : str
+        Ground-truth binary label column.
+    y_pred : str
+        Predicted binary label column.
+    protected_columns : list[str], optional
+        Columns to check. Default: ["SEXP", "BPLP", "profile_name"].
+    max_accuracy_gap : float
+        Maximum allowed accuracy difference between any two groups (default 0.05).
+    max_demographic_parity_gap : float
+        Maximum allowed selection rate difference (default 0.10).
+    max_equalised_odds_gap : float
+        Maximum allowed TPR or FPR difference (default 0.10).
+    min_group_size : int
+        Minimum group size to include in analysis (default 30).
+
+    Raises
+    ------
+    FairnessAssertionError
+        If any fairness threshold is exceeded. The error's `.details` dict
+        contains the full results for debugging.
+    """
+    report = FairnessReport(
+        data=data,
+        y_true=y_true,
+        y_pred=y_pred,
+        protected_columns=protected_columns,
+        min_group_size=min_group_size,
+    )
+    results = report.run()
+
+    failures = []
+    for r in results.results:
+        if r.accuracy_gap > max_accuracy_gap:
+            failures.append(
+                f"{r.column}: accuracy gap {r.accuracy_gap:.3f} > {max_accuracy_gap}"
+            )
+        if r.demographic_parity_gap > max_demographic_parity_gap:
+            failures.append(
+                f"{r.column}: demographic parity gap {r.demographic_parity_gap:.3f} "
+                f"> {max_demographic_parity_gap}"
+            )
+        if r.equalised_odds_gap > max_equalised_odds_gap:
+            failures.append(
+                f"{r.column}: equalised odds gap {r.equalised_odds_gap:.3f} "
+                f"> {max_equalised_odds_gap}"
+            )
+
+    if failures:
+        msg = (
+            f"Model failed fairness check on {len(failures)} metric(s):\n"
+            + "\n".join(f"  - {f}" for f in failures)
+            + "\n\nFull report:\n" + results.summary()
+        )
+        raise FairnessAssertionError(msg, {
+            "failures": failures,
+            "results": [r.to_dict() for r in results.results],
+        })

verosynthea_validator-0.1.0/verosynthea_validator/data.py

@@ -0,0 +1,96 @@
+"""
+Data loading utilities for verosynthea-validator.
+
+Free tier: loads the 5,000-row AUSynth sample from Hugging Face.
+Paid tier: connects to verosynthea.com API for full national data.
+"""
+from __future__ import annotations
+
+import os
+from pathlib import Path
+from typing import Optional
+
+import pandas as pd
+
+
+_HF_DATASET = "vero-synthea/ausynth-sample"
+_HF_FILE = "ausynth_sample_paddington_4064.parquet"
+_CACHE_DIR = Path.home() / ".cache" / "verosynthea"
+
+
+def load_ausynth_sample(
+    api_key: Optional[str] = None,
+    geography: Optional[str] = None,
+) -> pd.DataFrame:
+    """Load AUSynth demographic data for fairness testing.
+
+    Parameters
+    ----------
+    api_key : str, optional
+        Verosynthea API key for the full national dataset. If None, loads
+        the free 5,000-row Paddington sample from Hugging Face.
+    geography : str, optional
+        Suburb slug (e.g. "paddington-4064-qld") for paid-tier queries.
+        Ignored when using the free sample.
+
+    Returns
+    -------
+    pd.DataFrame
+        Person-level demographic data with 25+ variables including
+        profile_id and profile_name.
+    """
+    if api_key:
+        return _load_paid(api_key, geography)
+    return _load_free_sample()
+
+
+def _load_free_sample() -> pd.DataFrame:
+    """Load the free HF sample. Downloads on first call, caches locally."""
+    cache_path = _CACHE_DIR / _HF_FILE
+    if cache_path.exists():
+        return pd.read_parquet(cache_path)
+
+    # Try loading from HF datasets library
+    try:
+        from datasets import load_dataset
+        ds = load_dataset(_HF_DATASET, split="train")
+        df = ds.to_pandas()
+        _CACHE_DIR.mkdir(parents=True, exist_ok=True)
+        df.to_parquet(cache_path, index=False)
+        return df
+    except ImportError:
+        pass
+
+    # Fallback: direct parquet download
+    try:
+        url = f"https://huggingface.co/datasets/{_HF_DATASET}/resolve/main/{_HF_FILE}"
+        df = pd.read_parquet(url)
+        _CACHE_DIR.mkdir(parents=True, exist_ok=True)
+        df.to_parquet(cache_path, index=False)
+        return df
+    except Exception as e:
+        raise RuntimeError(
+            f"Could not load AUSynth sample. Install the `datasets` library "
+            f"(`pip install datasets`) or check your internet connection. "
+            f"Original error: {e}"
+        ) from e
+
+
+def _load_paid(api_key: str, geography: Optional[str] = None) -> pd.DataFrame:
+    """Load data from the Verosynthea API (paid tier)."""
+    import httpx
+
+    base = os.environ.get("VEROSYNTHEA_API_URL", "https://api.verosynthea.com/v1")
+    params = {}
+    if geography:
+        params["geography"] = geography
+
+    resp = httpx.get(
+        f"{base}/data/persons",
+        headers={"Authorization": f"Bearer {api_key}"},
+        params=params,
+        timeout=60.0,
+    )
+    resp.raise_for_status()
+    import io
+    return pd.read_parquet(io.BytesIO(resp.content))

verosynthea_validator-0.1.0/verosynthea_validator/fairness.py

@@ -0,0 +1,244 @@
+"""
+Core fairness metrics and reporting for verosynthea-validator.
+
+Computes demographic parity, equalised odds, accuracy gap, and
+calibration gap across protected groups defined by AUSynth variables.
+"""
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Optional, Sequence
+
+import numpy as np
+import pandas as pd
+
+
+# AUSynth profile names for readable output
+PROFILE_NAMES = {
+    0: "Labourers and operators",
+    1: "Young singles and non-workers",
+    2: "Children",
+    3: "Non-earning dependants",
+    4: "Trades and technical workers",
+    5: "Established partnered households",
+    6: "Retired and semi-retired",
+    7: "High-earning professionals",
+}
+
+# Common protected columns in AUSynth data
+SUGGESTED_PROTECTED = ["SEXP", "BPLP", "GNGP", "AGE5P", "profile_name"]
+
+
+@dataclass
+class GroupMetrics:
+    """Fairness metrics for a single demographic group."""
+    group_name: str
+    group_value: str
+    n: int
+    accuracy: float
+    positive_rate: float        # P(y_pred=1) — selection rate
+    true_positive_rate: float   # P(y_pred=1 | y_true=1) — recall
+    false_positive_rate: float  # P(y_pred=1 | y_true=0) — false alarm
+    positive_predictive_value: float  # P(y_true=1 | y_pred=1) — precision
+
+
+@dataclass
+class ProtectedColumnResult:
+    """Fairness analysis for one protected attribute."""
+    column: str
+    groups: list[GroupMetrics]
+    accuracy_gap: float          # max - min accuracy across groups
+    demographic_parity_gap: float  # max - min positive_rate
+    equalised_odds_gap: float    # max gap in TPR or FPR
+    min_group_size: int
+
+    def to_dict(self) -> dict:
+        return {
+            "column": self.column,
+            "accuracy_gap": round(self.accuracy_gap, 4),
+            "demographic_parity_gap": round(self.demographic_parity_gap, 4),
+            "equalised_odds_gap": round(self.equalised_odds_gap, 4),
+            "n_groups": len(self.groups),
+            "min_group_size": self.min_group_size,
+            "groups": [
+                {
+                    "value": g.group_value,
+                    "n": g.n,
+                    "accuracy": round(g.accuracy, 4),
+                    "positive_rate": round(g.positive_rate, 4),
+                    "tpr": round(g.true_positive_rate, 4),
+                    "fpr": round(g.false_positive_rate, 4),
+                }
+                for g in self.groups
+            ],
+        }
+
+
+@dataclass
+class FairnessResults:
+    """Complete fairness report across all protected columns."""
+    results: list[ProtectedColumnResult]
+    overall_accuracy: float
+    n_total: int
+
+    def summary(self) -> str:
+        """Human-readable summary of fairness results."""
+        lines = [
+            f"Fairness Report (n={self.n_total:,}, overall accuracy={self.overall_accuracy:.3f})",
+            "=" * 60,
+        ]
+        for r in self.results:
+            status = "PASS" if r.accuracy_gap <= 0.05 else "FAIL"
+            lines.append(
+                f"\n[{status}] {r.column} ({len(r.groups)} groups, "
+                f"smallest n={r.min_group_size})"
+            )
+            lines.append(
+                f"  Accuracy gap:           {r.accuracy_gap:.3f}"
+            )
+            lines.append(
+                f"  Demographic parity gap: {r.demographic_parity_gap:.3f}"
+            )
+            lines.append(
+                f"  Equalised odds gap:     {r.equalised_odds_gap:.3f}"
+            )
+            for g in sorted(r.groups, key=lambda x: -x.accuracy):
+                lines.append(
+                    f"    {g.group_value:<40s} acc={g.accuracy:.3f}  "
+                    f"sel={g.positive_rate:.3f}  n={g.n}"
+                )
+        lines.append("\n" + "=" * 60)
+        max_gap = max(r.accuracy_gap for r in self.results) if self.results else 0
+        if max_gap <= 0.05:
+            lines.append("Overall: PASS (all accuracy gaps <= 0.05)")
+        else:
+            worst = max(self.results, key=lambda r: r.accuracy_gap)
+            lines.append(
+                f"Overall: FAIL (worst gap: {worst.accuracy_gap:.3f} "
+                f"on {worst.column})"
+            )
+        return "\n".join(lines)
+
+    def to_dataframe(self) -> pd.DataFrame:
+        """Flatten results to a DataFrame for further analysis."""
+        rows = []
+        for r in self.results:
+            for g in r.groups:
+                rows.append({
+                    "protected_column": r.column,
+                    "group": g.group_value,
+                    "n": g.n,
+                    "accuracy": g.accuracy,
+                    "positive_rate": g.positive_rate,
+                    "tpr": g.true_positive_rate,
+                    "fpr": g.false_positive_rate,
+                    "ppv": g.positive_predictive_value,
+                })
+        return pd.DataFrame(rows)
+
+    @property
+    def passed(self) -> bool:
+        """True if all accuracy gaps are <= 0.05."""
+        return all(r.accuracy_gap <= 0.05 for r in self.results)
+
+
+class FairnessReport:
+    """Run a fairness analysis across demographic groups.
+
+    Parameters
+    ----------
+    data : pd.DataFrame
+        Dataset with predictions and demographic columns.
+    y_true : str
+        Column name for ground-truth binary labels (0/1).
+    y_pred : str
+        Column name for predicted binary labels (0/1).
+    protected_columns : list[str], optional
+        Demographic columns to test. Defaults to ["SEXP", "BPLP", "profile_name"].
+    min_group_size : int
+        Minimum observations per group to include (default 30).
+    """
+
+    def __init__(
+        self,
+        data: pd.DataFrame,
+        y_true: str,
+        y_pred: str,
+        protected_columns: Optional[Sequence[str]] = None,
+        min_group_size: int = 30,
+    ):
+        self.data = data
+        self.y_true = y_true
+        self.y_pred = y_pred
+        self.protected_columns = list(
+            protected_columns or ["SEXP", "BPLP", "profile_name"]
+        )
+        self.min_group_size = min_group_size
+
+        # Validate inputs
+        for col in [y_true, y_pred] + self.protected_columns:
+            if col not in data.columns:
+                raise ValueError(f"Column '{col}' not found in data")
+
+    def run(self) -> FairnessResults:
+        """Compute fairness metrics for all protected columns."""
+        yt = self.data[self.y_true].values.astype(int)
+        yp = self.data[self.y_pred].values.astype(int)
+        overall_acc = (yt == yp).mean()
+
+        results = []
+        for col in self.protected_columns:
+            groups = []
+            for val, gdf in self.data.groupby(col):
+                if len(gdf) < self.min_group_size:
+                    continue
+                idx = gdf.index
+                gt = yt[idx]
+                pr = yp[idx]
+                n = len(gt)
+                acc = (gt == pr).mean()
+                pos_rate = pr.mean()
+
+                # TPR / FPR
+                pos_mask = gt == 1
+                neg_mask = gt == 0
+                tpr = pr[pos_mask].mean() if pos_mask.sum() > 0 else 0.0
+                fpr = pr[neg_mask].mean() if neg_mask.sum() > 0 else 0.0
+                ppv = gt[pr == 1].mean() if (pr == 1).sum() > 0 else 0.0
+
+                groups.append(GroupMetrics(
+                    group_name=col,
+                    group_value=str(val),
+                    n=n,
+                    accuracy=float(acc),
+                    positive_rate=float(pos_rate),
+                    true_positive_rate=float(tpr),
+                    false_positive_rate=float(fpr),
+                    positive_predictive_value=float(ppv),
+                ))
+
+            if len(groups) < 2:
+                continue
+
+            accs = [g.accuracy for g in groups]
+            prs = [g.positive_rate for g in groups]
+            tprs = [g.true_positive_rate for g in groups]
+            fprs = [g.false_positive_rate for g in groups]
+
+            results.append(ProtectedColumnResult(
+                column=col,
+                groups=groups,
+                accuracy_gap=max(accs) - min(accs),
+                demographic_parity_gap=max(prs) - min(prs),
+                equalised_odds_gap=max(
+                    max(tprs) - min(tprs),
+                    max(fprs) - min(fprs),
+                ),
+                min_group_size=min(g.n for g in groups),
+            ))
+
+        return FairnessResults(
+            results=results,
+            overall_accuracy=float(overall_acc),
+            n_total=len(self.data),
+        )

verosynthea_validator-0.1.0/verosynthea_validator.egg-info/PKG-INFO

@@ -0,0 +1,169 @@
+Metadata-Version: 2.4
+Name: verosynthea-validator
+Version: 0.1.0
+Summary: Fairness testing for ML models using Australian demographic data
+Author-email: Verosynthea <hello@verosynthea.com>
+License: MIT
+Project-URL: Homepage, https://verosynthea.com/for-ai-labs
+Project-URL: Repository, https://github.com/verosynthea/verosynthea-validator
+Project-URL: Dataset, https://huggingface.co/datasets/vero-synthea/ausynth-sample
+Project-URL: Documentation, https://verosynthea.com/about
+Keywords: fairness,ml,demographics,australia,synthetic-data,census
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: pandas>=1.5
+Requires-Dist: numpy>=1.23
+Provides-Extra: hf
+Requires-Dist: datasets>=2.0; extra == "hf"
+Provides-Extra: paid
+Requires-Dist: httpx>=0.24; extra == "paid"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: scikit-learn>=1.2; extra == "dev"
+
+# verosynthea-validator
+
+Fairness testing for ML models using real Australian demographic data. One line to check whether your model treats demographic groups equally.
+
+```bash
+pip install verosynthea-validator
+```
+
+```python
+from verosynthea_validator import FairnessReport
+
+report = FairnessReport(
+    data=test_data,
+    y_true="label",
+    y_pred="prediction",
+    protected_columns=["SEXP", "BPLP", "profile_name"],
+)
+results = report.run()
+print(results.summary())
+```
+
+Output:
+
+```
+Fairness Report (n=5,000, overall accuracy=0.847)
+============================================================
+
+[PASS] SEXP (2 groups, smallest n=2,451)
+  Accuracy gap:           0.012
+  Demographic parity gap: 0.008
+  Equalised odds gap:     0.015
+
+[FAIL] BPLP (3 groups, smallest n=312)
+  Accuracy gap:           0.073
+  Demographic parity gap: 0.091
+  Equalised odds gap:     0.064
+
+============================================================
+Overall: FAIL (worst gap: 0.073 on BPLP)
+```
+
+## CI/CD gate
+
+```python
+from verosynthea_validator import assert_fair
+
+# Fails the build if any group accuracy gap > 5%
+assert_fair(test_data, "label", "prediction", max_accuracy_gap=0.05)
+```
+
+In pytest:
+
+```python
+def test_model_fairness():
+    predictions = model.predict(test_data)
+    test_data["y_pred"] = predictions
+    assert_fair(
+        test_data, "y_true", "y_pred",
+        protected_columns=["SEXP", "BPLP", "profile_name"],
+        max_accuracy_gap=0.05,
+        max_demographic_parity_gap=0.10,
+    )
+```
+
+## What it measures
+
+For each protected column (e.g. sex, birthplace, demographic profile), the validator computes:
+
+| Metric | What it checks |
+|--------|---------------|
+| **Accuracy gap** | Max accuracy difference between any two groups |
+| **Demographic parity gap** | Max difference in selection rate (P(y_pred=1)) |
+| **Equalised odds gap** | Max difference in true positive rate or false positive rate |
+
+Groups smaller than 30 observations are excluded (configurable via `min_group_size`).
+
+## Why this instead of fairlearn or aif360?
+
+Those are general-purpose fairness frameworks. This package is purpose-built for Australian demographics:
+
+- **Pre-loaded demographic data.** The free tier includes 5,000 synthetic individuals from [AUSynth](https://huggingface.co/datasets/vero-synthea/ausynth-sample) with 25 Census-calibrated variables. No need to source your own protected attributes.
+- **8 demographic profiles.** AUSynth clusters every person into one of 8 profiles (High-earning professionals, Young singles, Retired, etc.) — a richer protected attribute than just age or sex.
+- **Australia-specific calibration.** Variables match ABS Census 2021 categories exactly. Income brackets, occupation codes, education levels, birthplace regions — all in Australian standard classifications.
+- **One-line CI gate.** `assert_fair()` drops into pytest with zero configuration.
+
+## Data tiers
+
+| Tier | Data | Cost |
+|------|------|------|
+| **Free** | 5,000-row Paddington 4064 sample from [Hugging Face](https://huggingface.co/datasets/vero-synthea/ausynth-sample) | $0 |
+| **Paid** | Full national dataset (32M individuals, 15,352 suburbs) via API | [verosynthea.com](https://verosynthea.com) |
+
+```python
+from verosynthea_validator import load_ausynth_sample
+
+# Free tier (downloads from HF on first call)
+df = load_ausynth_sample()
+
+# Paid tier
+df = load_ausynth_sample(api_key="vero_...", geography="bondi-2026-nsw")
+```
+
+## The 8 demographic profiles
+
+| ID | Name | Typical characteristics |
+|----|------|------------------------|
+| 0 | Labourers and operators | Blue-collar, lower income |
+| 1 | Young singles and non-workers | Under 25, students, NILF |
+| 2 | Children | Under 15 |
+| 3 | Non-earning dependants | Adults not in workforce |
+| 4 | Trades and technical workers | Certificate-qualified, mid income |
+| 5 | Established partnered households | Married, mid-career |
+| 6 | Retired and semi-retired | Over 60, pension income |
+| 7 | High-earning professionals | Degree-qualified, professional occupations |
+
+## Installation
+
+```bash
+pip install verosynthea-validator          # core (pandas + numpy)
+pip install verosynthea-validator[hf]     # + Hugging Face datasets loader
+pip install verosynthea-validator[paid]   # + httpx for API access
+pip install verosynthea-validator[dev]    # + pytest + sklearn for development
+```
+
+## Links
+
+- **Dataset:** [vero-synthea/ausynth-sample](https://huggingface.co/datasets/vero-synthea/ausynth-sample) on Hugging Face
+- **Full product:** [verosynthea.com](https://verosynthea.com)
+- **Methodology:** [verosynthea.com/about](https://verosynthea.com/about)
+
+## Citation
+
+```
+Verosynthea AUSynth (2026). Synthetic Australian Census Data.
+https://verosynthea.com
+```
+
+## License
+
+MIT

verosynthea_validator-0.1.0/verosynthea_validator.egg-info/SOURCES.txt

@@ -0,0 +1,12 @@
+README.md
+pyproject.toml
+tests/test_fairness.py
+verosynthea_validator/__init__.py
+verosynthea_validator/assertions.py
+verosynthea_validator/data.py
+verosynthea_validator/fairness.py
+verosynthea_validator.egg-info/PKG-INFO
+verosynthea_validator.egg-info/SOURCES.txt
+verosynthea_validator.egg-info/dependency_links.txt
+verosynthea_validator.egg-info/requires.txt
+verosynthea_validator.egg-info/top_level.txt

verosynthea_validator-0.1.0/verosynthea_validator.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

verosynthea_validator-0.1.0/verosynthea_validator.egg-info/requires.txt

@@ -0,0 +1,12 @@
+pandas>=1.5
+numpy>=1.23
+
+[dev]
+pytest>=7.0
+scikit-learn>=1.2
+
+[hf]
+datasets>=2.0
+
+[paid]
+httpx>=0.24

verosynthea_validator-0.1.0/verosynthea_validator.egg-info/top_level.txt

@@ -0,0 +1 @@
+verosynthea_validator