azoic 0.5.0__tar.gz

azoic-0.5.0/PKG-INFO

@@ -0,0 +1,108 @@
+Metadata-Version: 2.3
+Name: azoic
+Version: 0.5.0
+Summary: scikit-learn-compatible toolkit for non-life technical tariff (pure premium) modelling
+Author: thomas-bury
+Author-email: thomas-bury <thomas.bury@yuzzu.be>
+License: MIT
+Requires-Dist: numpy>=2.0
+Requires-Dist: pandas>=3.0
+Requires-Dist: scikit-learn>=1.6
+Requires-Dist: glum>=3.0
+Requires-Dist: lightgbm>=4.3
+Requires-Dist: pyarrow>=15.0
+Requires-Dist: pydantic>=2.7
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: typer>=0.12
+Requires-Dist: matplotlib>=3.8
+Requires-Dist: openpyxl>=3.1
+Requires-Dist: s3fs>=2024.6 ; extra == 'aws'
+Requires-Dist: mlflow>=2.14 ; extra == 'mlops'
+Requires-Dist: plotly>=6.9.0 ; extra == 'plot'
+Requires-Dist: optuna>=3.6 ; extra == 'tune'
+Requires-Python: >=3.12
+Project-URL: Repository, https://github.com/ThomasBury/azoic
+Project-URL: Issues, https://github.com/ThomasBury/azoic/issues
+Project-URL: Documentation, https://thomasbury.github.io/azoic/
+Provides-Extra: aws
+Provides-Extra: mlops
+Provides-Extra: plot
+Provides-Extra: tune
+Description-Content-Type: text/markdown
+
+# Azoic
+
+A scikit-learn-compatible Python toolkit for non-life technical tariff (pure
+premium) modelling: actuarial preprocessing, GLM (glum) + GBM (LightGBM),
+frequency-severity decomposition, actuarial diagnostics, reproducible runs,
+GBM-to-GLM distillation, and multiplicative tariff export.
+
+- Documentation: <https://thomasbury.github.io/azoic/>
+- Hosted tutorial series: <https://thomasbury.github.io/azoic/tutorial/index.html>
+- Full spec and conventions: [`PRD.md`](PRD.md)
+- Agent quick rules: [`AGENTS.md`](AGENTS.md)
+- Status: [`PROGRESS.md`](PROGRESS.md)
+
+## Install
+
+Requires Python 3.12 or newer. In your own uv project:
+
+```bash
+uv add azoic==0.5.0
+uv run azoic --help
+```
+
+For development or the executable tutorial, clone this repository and run:
+
+```bash
+uv sync --no-dev                    # runtime only
+uv sync                             # default development environment
+uv sync --all-extras --all-groups   # everything
+```
+
+## Correctness conventions
+
+Frequency models use `claim_count / exposure` with `sample_weight=exposure`.
+`deviance_test` is exposure-weighted mean Tweedie deviance with fixed
+`power=1.5`. Gini is a concentration Gini for ranking; equal prediction scores
+are evaluated as tied blocks.
+
+Temporal holdouts never split equal timestamps. Tuning selects parameters on an
+inner split of the outer training partition, then refits on outer training data
+and reports the untouched outer test. Exported tariffs reject unknown categorical
+levels and non-finite numeric inputs.
+
+Positive-objective GBMs can be exported through `azoic export-tariff
+--distill`; the workbook reproduces the distilled GLM student, not the teacher.
+
+## Executable tutorial series
+
+The numbered Quarto book under [`examples/`](examples/) runs the complete
+technical-tariff workflow on pinned OpenML data and then covers tuning,
+temporal stability, protected-group audit, and manual tariff recipes. Install
+the Jupyter kernel plus the MLflow, Plotly, and tuning extras; install Quarto
+separately and ensure its `quarto` executable is on `PATH`.
+
+```bash
+uv sync --group demo --extra mlops --extra plot --extra tune
+just demo
+```
+
+The render produces the ignored book under `examples/_book/`; fetched data,
+the tariff workbook, reports, and local MLflow files stay under the ignored
+`examples/_artifacts/`. To inspect the recorded run afterward:
+
+```bash
+uv run mlflow ui \
+  --backend-store-uri sqlite:///examples/_artifacts/fremtpl2/mlflow.db
+```
+
+## Dev
+
+```bash
+just check          # ruff + ty + pytest
+just test           # pytest -x
+just lint
+just format
+uv run ty check     # production source types
+```

azoic-0.5.0/README.md

@@ -0,0 +1,76 @@
+# Azoic
+
+A scikit-learn-compatible Python toolkit for non-life technical tariff (pure
+premium) modelling: actuarial preprocessing, GLM (glum) + GBM (LightGBM),
+frequency-severity decomposition, actuarial diagnostics, reproducible runs,
+GBM-to-GLM distillation, and multiplicative tariff export.
+
+- Documentation: <https://thomasbury.github.io/azoic/>
+- Hosted tutorial series: <https://thomasbury.github.io/azoic/tutorial/index.html>
+- Full spec and conventions: [`PRD.md`](PRD.md)
+- Agent quick rules: [`AGENTS.md`](AGENTS.md)
+- Status: [`PROGRESS.md`](PROGRESS.md)
+
+## Install
+
+Requires Python 3.12 or newer. In your own uv project:
+
+```bash
+uv add azoic==0.5.0
+uv run azoic --help
+```
+
+For development or the executable tutorial, clone this repository and run:
+
+```bash
+uv sync --no-dev                    # runtime only
+uv sync                             # default development environment
+uv sync --all-extras --all-groups   # everything
+```
+
+## Correctness conventions
+
+Frequency models use `claim_count / exposure` with `sample_weight=exposure`.
+`deviance_test` is exposure-weighted mean Tweedie deviance with fixed
+`power=1.5`. Gini is a concentration Gini for ranking; equal prediction scores
+are evaluated as tied blocks.
+
+Temporal holdouts never split equal timestamps. Tuning selects parameters on an
+inner split of the outer training partition, then refits on outer training data
+and reports the untouched outer test. Exported tariffs reject unknown categorical
+levels and non-finite numeric inputs.
+
+Positive-objective GBMs can be exported through `azoic export-tariff
+--distill`; the workbook reproduces the distilled GLM student, not the teacher.
+
+## Executable tutorial series
+
+The numbered Quarto book under [`examples/`](examples/) runs the complete
+technical-tariff workflow on pinned OpenML data and then covers tuning,
+temporal stability, protected-group audit, and manual tariff recipes. Install
+the Jupyter kernel plus the MLflow, Plotly, and tuning extras; install Quarto
+separately and ensure its `quarto` executable is on `PATH`.
+
+```bash
+uv sync --group demo --extra mlops --extra plot --extra tune
+just demo
+```
+
+The render produces the ignored book under `examples/_book/`; fetched data,
+the tariff workbook, reports, and local MLflow files stay under the ignored
+`examples/_artifacts/`. To inspect the recorded run afterward:
+
+```bash
+uv run mlflow ui \
+  --backend-store-uri sqlite:///examples/_artifacts/fremtpl2/mlflow.db
+```
+
+## Dev
+
+```bash
+just check          # ruff + ty + pytest
+just test           # pytest -x
+just lint
+just format
+uv run ty check     # production source types
+```

azoic-0.5.0/pyproject.toml

@@ -0,0 +1,92 @@
+[project]
+name = "azoic"
+version = "0.5.0"
+description = "scikit-learn-compatible toolkit for non-life technical tariff (pure premium) modelling"
+readme = "README.md"
+requires-python = ">=3.12"
+dependencies = [
+    "numpy>=2.0",
+    "pandas>=3.0",
+    "scikit-learn>=1.6",
+    "glum>=3.0",
+    "lightgbm>=4.3",
+    "pyarrow>=15.0",
+    "pydantic>=2.7",
+    "pyyaml>=6.0",
+    "typer>=0.12",
+    "matplotlib>=3.8",
+    "openpyxl>=3.1",
+]
+
+[[project.authors]]
+name = "thomas-bury"
+email = "thomas.bury@yuzzu.be"
+
+[project.license]
+text = "MIT"
+
+[project.urls]
+Repository = "https://github.com/ThomasBury/azoic"
+Issues = "https://github.com/ThomasBury/azoic/issues"
+Documentation = "https://thomasbury.github.io/azoic/"
+
+[project.optional-dependencies]
+aws = ["s3fs>=2024.6"]
+mlops = ["mlflow>=2.14"]
+tune = ["optuna>=3.6"]
+plot = ["plotly>=6.9.0"]
+
+[project.scripts]
+azoic = "azoic.cli:app"
+
+[build-system]
+requires = ["uv_build>=0.11.19,<0.12.0"]
+build-backend = "uv_build"
+
+[dependency-groups]
+demo = ["jupyter>=1.1"]
+dev = [
+    "pytest>=8.0",
+    "ruff>=0.5",
+    "pre-commit>=3.7",
+    "ty>=0.0.71",
+]
+docs = [
+    "mkdocstrings-python>=2.0.5",
+    "zensical>=0.0.54",
+]
+
+[tool.ruff]
+line-length = 100
+target-version = "py312"
+src = [
+    "src",
+    "tests",
+]
+
+[tool.ruff.lint]
+select = [
+    "E",
+    "F",
+    "W",
+    "I",
+    "B",
+    "UP",
+    "C4",
+    "SIM",
+]
+
+[tool.ruff.lint.per-file-ignores]
+"src/azoic/cli.py" = ["B008"]
+
+[tool.ruff.format]
+quote-style = "double"
+
+[tool.ty.src]
+include = ["src"]
+
+[tool.pytest.ini_options]
+minversion = "8.0"
+testpaths = ["tests"]
+addopts = "-ra --strict-markers"
+filterwarnings = ["ignore:Passing a non-Collection iterable to parametrize is deprecated"]

azoic-0.5.0/pyproject.toml.orig

@@ -0,0 +1,76 @@
+[project]
+name = "azoic"
+version = "0.5.0"
+description = "scikit-learn-compatible toolkit for non-life technical tariff (pure premium) modelling"
+readme = "README.md"
+authors = [{ name = "thomas-bury", email = "thomas.bury@yuzzu.be" }]
+license = { text = "MIT" }
+requires-python = ">=3.12"
+urls = { Repository = "https://github.com/ThomasBury/azoic", Issues = "https://github.com/ThomasBury/azoic/issues", Documentation = "https://thomasbury.github.io/azoic/" }
+dependencies = [
+    "numpy>=2.0",
+    "pandas>=3.0",
+    "scikit-learn>=1.6",
+    "glum>=3.0",
+    "lightgbm>=4.3",
+    "pyarrow>=15.0",
+    "pydantic>=2.7",
+    "pyyaml>=6.0",
+    "typer>=0.12",
+    "matplotlib>=3.8",
+    "openpyxl>=3.1",
+]
+
+[project.optional-dependencies]
+aws = ["s3fs>=2024.6"]
+mlops = ["mlflow>=2.14"]
+tune = ["optuna>=3.6"]
+plot = ["plotly>=6.9.0"]
+
+[project.scripts]
+azoic = "azoic.cli:app"
+
+[build-system]
+requires = ["uv_build>=0.11.19,<0.12.0"]
+build-backend = "uv_build"
+
+[dependency-groups]
+demo = ["jupyter>=1.1"]
+dev = [
+    "pytest>=8.0",
+    "ruff>=0.5",
+    "pre-commit>=3.7",
+    "ty>=0.0.71",
+]
+docs = [
+    "mkdocstrings-python>=2.0.5",
+    "zensical>=0.0.54",
+]
+
+[tool.ruff]
+line-length = 100
+target-version = "py312"
+src = ["src", "tests"]
+
+[tool.ruff.lint]
+select = ["E", "F", "W", "I", "B", "UP", "C4", "SIM"]
+# Typer's documented pattern uses `typer.Option(...)` / `typer.Argument(...)`
+# as parameter defaults; ruff's B008 misfires on it.
+[tool.ruff.lint.per-file-ignores]
+"src/azoic/cli.py" = ["B008"]
+
+[tool.ruff.format]
+quote-style = "double"
+
+[tool.ty.src]
+include = ["src"]
+
+[tool.pytest.ini_options]
+minversion = "8.0"
+testpaths = ["tests"]
+addopts = "-ra --strict-markers"
+filterwarnings = [
+    # sklearn.utils.estimator_checks.parametrize_with_checks still passes a
+    # generator to pytest.parametrize (upstream, not our control).
+    "ignore:Passing a non-Collection iterable to parametrize is deprecated",
+]

azoic-0.5.0/src/azoic/__init__.py

@@ -0,0 +1,7 @@
+"""Azoic: scikit-learn-compatible toolkit for non-life technical tariff modelling.
+
+Public API surfaces are added as modules land (see PROGRESS.md). Keep this file
+light so `import azoic` does not pull heavy backends.
+"""
+
+__version__ = "0.5.0"

azoic-0.5.0/src/azoic/cli.py

@@ -0,0 +1,208 @@
+"""Azoic CLI (Typer): ``profile`` / ``fit`` / ``compare`` / ``export-tariff`` /
+``tune``.
+
+Wraps the M5 + M6 + M7 pieces end-to-end:
+  * ``azoic profile`` -- runs ``profile_features`` + ``screen_features`` and
+    prints (or writes) the resulting screening table.
+  * ``azoic fit`` -- loads an ``ExperimentConfig`` YAML, ``run_experiment``s
+    it, prints a model card to stdout (and/or writes it to disk).
+  * ``azoic compare`` -- runs one or more configs and prints a side-by-side
+    per-model metrics table (CSV to disk if ``--out``).
+  * ``azoic export-tariff`` -- runs a config, exports a named GLM or, with
+    ``--distill``, an exportable GLM student of a positive-objective GBM.
+  * ``azoic tune`` -- optuna hyperparameter search per model
+    (``tune`` extra, M7 / v0.2 part 1) then a model card of the best fit.
+
+ponytail: the CLI is a thin shell over ``workflow`` + ``reporting`` +
+``tariff`` + ``tune``; no business logic lives here.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import typer
+
+from azoic.data import DatasetSpec, load_data
+from azoic.profile import profile_features, screen_features
+from azoic.reporting import comparison_table, model_card
+from azoic.tariff import distill_gbm as _distill_gbm
+from azoic.tariff import export_tariff as _export_tariff
+from azoic.workflow import ExperimentConfig, _data_fingerprint, run_experiment
+
+app = typer.Typer(
+    no_args_is_help=True,
+    add_completion=False,
+    help="Azoic: non-life technical tariff (pure premium) modelling.",
+)
+
+
+@app.command()
+def profile(
+    data: str = typer.Option(..., "--data", help="Parquet file (or s3:// path) to profile."),
+    target: str = typer.Option(..., "--target", help="Target column name (e.g. claim_amount)."),
+    exposure: str = typer.Option(..., "--exposure", help="Exposure column name."),
+    claim_count: str | None = typer.Option(None, "--claim-count", help="Claim-count column."),
+    time_col: str | None = typer.Option(None, "--time-col", help="Time column."),
+    out: Path | None = typer.Option(
+        None, "--out", help="Write the screening table to CSV (default: print to stdout)."
+    ),
+) -> None:
+    """Profile features in a portfolio and screen them for keep / bin / group / drop."""
+    spec = DatasetSpec(
+        target=target,
+        exposure=exposure,
+        claim_count=claim_count,
+        time_col=time_col,
+    )
+    df = load_data(data, spec=spec)
+    prof = profile_features(df)
+    screened = screen_features(prof)
+
+    if out is not None:
+        screened.to_csv(out, index=False)
+        typer.echo(f"Wrote {len(screened)} rows to {out}")
+        return
+
+    typer.echo(screened.to_string(index=False))
+
+
+@app.command()
+def fit(
+    config: Path = typer.Option(..., "--config", help="ExperimentConfig YAML."),
+    out: Path | None = typer.Option(None, "--out", help="Write the model card markdown here."),
+    quiet: bool = typer.Option(False, "--quiet", "-q", help="Do not print the card to stdout."),
+) -> None:
+    """Run all models in an ``ExperimentConfig`` YAML; produce a model card."""
+    cfg = ExperimentConfig.from_yaml(config)
+    run = run_experiment(cfg)
+    md = model_card(run)
+
+    if out is not None:
+        out.write_text(md, encoding="utf-8")
+        typer.echo(f"Wrote markdown card to {out}")
+    if not quiet:
+        typer.echo(md)
+
+
+@app.command()
+def compare(
+    configs: list[Path] = typer.Argument(
+        ..., help="One or more ExperimentConfig YAML files to run and compare."
+    ),
+    out: Path | None = typer.Option(
+        None, "--out", help="Write the comparison table to CSV (default: print to stdout)."
+    ),
+) -> None:
+    """Run multiple configs and print a side-by-side per-model metric table."""
+    runs = [run_experiment(ExperimentConfig.from_yaml(cfg_path)) for cfg_path in configs]
+    rows = comparison_table(runs)
+
+    if out is not None:
+        rows.to_csv(out, index=False)
+        typer.echo(f"Wrote {len(rows)} rows to {out}")
+        return
+
+    typer.echo(f"Azoic compare\n{rows.to_string(index=False)}")
+
+
+@app.command("export-tariff")
+def export_tariff(
+    config: Path = typer.Option(..., "--config", help="ExperimentConfig YAML."),
+    model: str = typer.Option(..., "--model", help="Name of the model to export."),
+    out: Path = typer.Option(..., "--out", help="Output xlsx path."),
+    distill: bool = typer.Option(
+        False,
+        "--distill",
+        help="Distill a positive-objective GBM to an exportable GLM on the existing split.",
+    ),
+    recalibrate: bool = typer.Option(
+        True,
+        "--recalibrate/--no-recalibrate",
+        help="Shift the base so the tariff reproduces the observed training "
+        "total claim amount (default: on).",
+    ),
+) -> None:
+    """Run a config and export a GLM or an explicitly distilled GBM tariff."""
+    from azoic.models import RiskGBM, RiskGLM
+
+    cfg = ExperimentConfig.from_yaml(config)
+    run, ests = run_experiment(cfg, return_estimators=True)
+    if model not in ests:
+        raise typer.BadParameter(
+            f"model {model!r} not in config models {list(ests)}; check the YAML `models:` keys."
+        )
+    est = ests[model]
+    final_est = est.steps[-1][1] if hasattr(est, "steps") else est
+    if isinstance(final_est, RiskGBM) and not distill:
+        raise typer.BadParameter(
+            f"model {model!r} ends in RiskGBM; pass --distill to export a GLM student."
+        )
+    if isinstance(final_est, RiskGLM) and distill:
+        raise typer.BadParameter("--distill is only valid for a RiskGBM model")
+    if not isinstance(final_est, (RiskGLM, RiskGBM)):
+        raise typer.BadParameter(
+            f"model {model!r} ends in {type(final_est).__name__}; export-tariff requires a RiskGLM."
+        )
+
+    df = load_data(cfg.data_path, spec=cfg.spec)
+    if _data_fingerprint(df) != run.data_fingerprint:
+        raise typer.BadParameter("dataset changed since fitting; rerun with an unchanged input")
+    X = df[list(est.feature_names_in_)]
+    y = df[cfg.spec.target]
+    train_indices = list(run.train_indices)
+    X_train = X.iloc[train_indices]
+    if isinstance(final_est, RiskGBM):
+        est = _distill_gbm(est, X_train, X.iloc[list(run.test_indices)])
+        metrics = vars(est)["distillation_metrics_"]
+        typer.echo(
+            "Distillation fidelity: "
+            f"teacher_student_deviance={metrics['teacher_student_deviance']:.6g}, "
+            f"student_teacher_total_ratio={metrics['student_teacher_total_ratio']:.6g}"
+        )
+
+    _export_tariff(
+        est,
+        out,
+        X=X_train,
+        y=y.iloc[train_indices],
+        exposure_col=cfg.spec.exposure,
+        recalibrate=recalibrate,
+    )
+    typer.echo(f"Wrote tariff for {model} to {out}")
+
+
+@app.command()
+def tune(
+    config: Path = typer.Option(..., "--config", help="ExperimentConfig YAML."),
+    trials: int | None = typer.Option(
+        None,
+        "--trials",
+        help="Optuna trials per model; overrides YAML (default: YAML or 20).",
+    ),
+    calibration_penalty: float | None = typer.Option(
+        None,
+        "--calibration-penalty",
+        help="O/P penalty weight; overrides YAML (default: YAML or 1.0).",
+    ),
+    out: Path | None = typer.Option(None, "--out", help="Write the model card markdown here."),
+    quiet: bool = typer.Option(False, "--quiet", "-q", help="Do not print the card to stdout."),
+) -> None:
+    """Tune model hyperparameters with optuna, then write a model card of the best fit."""
+    from azoic.tune import tune_experiment
+
+    cfg = ExperimentConfig.from_yaml(config)
+    result = tune_experiment(cfg, n_trials=trials, calibration_penalty=calibration_penalty)
+    md = model_card(result.run)
+
+    for name, params in result.best_params.items():
+        typer.echo(f"tuned {name}: " + ", ".join(f"{k}={v}" for k, v in params.items()))
+    if out is not None:
+        out.write_text(md, encoding="utf-8")
+        typer.echo(f"Wrote markdown card to {out}")
+    if not quiet:
+        typer.echo(md)
+
+
+if __name__ == "__main__":
+    app()

azoic-0.5.0/src/azoic/data.py

@@ -0,0 +1,95 @@
+"""Dataset specifications and loaders.
+
+pandas / pyarrow at boundaries. S3 paths work out of the box via fsspec when
+the optional ``aws`` extra is installed (`uv sync --extra aws`).
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Any
+
+import pandas as pd
+from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
+
+MIN_EXPOSURE = 1.0 / 366.0
+"""Minimum per-row exposure: one day in year fractions (leap-year denominator).
+
+Sub-day rows are data errors for pure-premium modelling: a tiny exposure
+inflates ``claim_amount / exposure`` rates and destabilizes GLM fits.
+"""
+EXPOSURE_FLOOR = MIN_EXPOSURE * (1.0 - 1e-3)
+"""Validation threshold: one day minus a 0.1% relative tolerance for day-count
+representations (freMTPL2 stores one-day policies ~163 ulps below 1/366)."""
+
+
+class DatasetSpec(BaseModel):
+    """Names of the special (non-feature) columns in a pricing dataset.
+
+    Feature columns are everything else. `.target` and `.exposure` are required;
+    the rest are optional. Unknown fields are rejected.
+    """
+
+    model_config = ConfigDict(extra="forbid")
+
+    target: str
+    exposure: str
+    claim_count: str | None = None
+    time_col: str | None = None
+    protected_cols: list[str] = Field(default_factory=list)
+
+    @field_validator("target", "exposure", "claim_count", "time_col")
+    @classmethod
+    def _non_empty(cls, v: str | None) -> str | None:
+        if v is not None and not v.strip():
+            raise ValueError("column name must be non-empty")
+        return v
+
+    @field_validator("protected_cols")
+    @classmethod
+    def _valid_protected_cols(cls, values: list[str]) -> list[str]:
+        if any(not name or not name.strip() for name in values):
+            raise ValueError("protected column names must be non-empty")
+        duplicates = sorted({name for name in values if values.count(name) > 1})
+        if duplicates:
+            raise ValueError(f"protected_cols must be unique; duplicates: {duplicates}")
+        return values
+
+    @model_validator(mode="after")
+    def _protected_cols_are_not_outcomes(self):
+        special = {self.target, self.exposure, self.claim_count, self.time_col}
+        overlap = sorted(set(self.protected_cols) & special)
+        if overlap:
+            raise ValueError(f"protected_cols overlaps other special columns: {overlap}")
+        return self
+
+    @model_validator(mode="after")
+    def _special_cols_are_distinct(self):
+        names = [self.target, self.exposure, self.claim_count, self.time_col]
+        present = [name for name in names if name is not None]
+        duplicates = sorted({name for name in present if present.count(name) > 1})
+        if duplicates:
+            raise ValueError(f"special column names must be distinct; duplicates: {duplicates}")
+        return self
+
+    def required_columns(self) -> list[str]:
+        cols = [self.target, self.exposure]
+        for opt in (self.claim_count, self.time_col):
+            if opt:
+                cols.append(opt)
+        cols.extend(self.protected_cols)
+        return cols
+
+
+def load_data(path: str | Path, spec: DatasetSpec | None = None, **kwargs: Any) -> pd.DataFrame:
+    """Read a Parquet dataset (local or s3://) into a pandas DataFrame.
+
+    When ``spec`` is given, required columns must be present or ValueError.
+    Extra kwargs forward to ``pandas.read_parquet``.
+    """
+    df = pd.read_parquet(path, **kwargs)
+    if spec is not None:
+        missing = [c for c in spec.required_columns() if c not in df.columns]
+        if missing:
+            raise ValueError(f"loaded dataset is missing required columns: {missing}")
+    return df