modeltest 0.2.0__tar.gz

modeltest-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 modeltest contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

modeltest-0.2.0/PKG-INFO

@@ -0,0 +1,252 @@
+Metadata-Version: 2.4
+Name: modeltest
+Version: 0.2.0
+Summary: If you test your code, why not your model? Unit tests for machine learning models.
+Author: modeltest contributors
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/Tzinny-dev/model-test
+Project-URL: Repository, https://github.com/Tzinny-dev/model-test
+Project-URL: Issues, https://github.com/Tzinny-dev/model-test/issues
+Keywords: machine-learning,mlops,testing,validation,ci
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Quality Assurance
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy
+Requires-Dist: pandas
+Requires-Dist: scikit-learn
+Requires-Dist: scipy
+Requires-Dist: PyYAML
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: ruff; extra == "dev"
+Requires-Dist: pre-commit; extra == "dev"
+Requires-Dist: shap; extra == "dev"
+Requires-Dist: mlflow; extra == "dev"
+Provides-Extra: explain
+Requires-Dist: shap; extra == "explain"
+Provides-Extra: mlflow
+Requires-Dist: mlflow; extra == "mlflow"
+Dynamic: license-file
+
+# modeltest
+
+> If you test your code, why not your model?
+
+`modeltest` is a unit-testing framework for machine learning models. It lets you
+define contracts for model quality, robustness, fairness, and data invariants,
+and run them automatically in your CI/CD pipeline — just like `pytest` for code.
+
+## Quick start
+
+```python
+from modeltest import ModelSuite
+from modeltest.scenarios import MinimumAccuracyTest, GroupPerformanceTest
+
+suite = ModelSuite(name="Fraud Detection")
+suite.add_test(MinimumAccuracyTest(threshold=0.85))
+suite.add_test(GroupPerformanceTest(metric="accuracy", threshold=0.8, group_col="gender"))
+
+result = suite.run(model, X_val, y_val, model_name="fraud_rf")
+print(result.report(style="table"))
+```
+
+## CLI
+
+After training, validate a model contract from the command line:
+
+```bash
+# Declarative suite (recommended)
+modeltest validate \
+  --suite suite.yaml \
+  --model model.pkl \
+  --data validation.csv \
+  --target target \
+  --train-data train.csv \
+  --output report.xml
+
+# Python suite (suite.py exposing `suite`)
+modeltest validate --suite suite.py --model model.pkl --data validation.csv --target target
+```
+
+Extras:
+- `--output report.xml` writes JUnit XML for CI reporters.
+- `--train-data` enables the drift tests (they compare train vs. validation distributions).
+- Exit code is `1` if any test fails, `0` otherwise.
+
+## Prediction caching
+
+Within a single `suite.run(...)`, predictions are computed once and reused
+across every test. `TestContext.predict()` caches by a content hash of the
+input, so tests predicting on the *same* data (`MinimumAccuracyTest`,
+`GroupPerformanceTest`, the fairness tests, ...) each reuse the result instead
+of re-running the model. Perturbed inputs (e.g. the robustness test's noisy
+copy) get their own cache entry, so caching never compromises correctness.
+
+Disable it if you need a fresh prediction every call:
+
+```python
+ctx = TestContext(model=model, X_val=X_val, y_val=y_val, cache_predictions=False)
+```
+
+## YAML suites
+
+Define your contract declaratively — no code needed:
+
+```yaml
+suite:
+  name: "Credit Scoring Model"
+  tests:
+    - type: minimum_accuracy
+      params: {threshold: 0.85}
+    - type: group_performance
+      params: {metric: accuracy, threshold: 0.8, group_col: "gender"}
+    - type: robustness
+      params: {noise_std: 0.01, max_drop: 0.03}
+    - type: data_drift
+      params: {features: [age, income], max_psi: 0.15}
+    - type: equal_opportunity
+      params: {protected: "gender", max_diff: 0.1}
+    - type: statistical_parity
+      params: {protected: "gender", max_diff: 0.1, min_ratio: 0.8}
+    - type: data_invariant
+      params: {expected_columns: [age, income], max_null_ratio: 0.02}
+```
+
+## Multi-framework support
+
+`TestContext` talks to models through a small adapter interface
+(`modeltest.wrappers`). Out of the box it normalizes:
+
+- **scikit-learn** estimators (`predict`, and `predict_proba` when available)
+- **PyTorch** `nn.Module` (`predict` = argmax over logits, `predict_proba` = softmax)
+- **Keras / TensorFlow** models (binary threshold or multiclass argmax)
+
+Pass any of these straight to `suite.run(model, ...)`; the right adapter is
+picked automatically. Custom framework? Implement a
+`ModelWrapper` subclass and pass an instance as the model.
+
+**scikit-learn `Pipeline`s** (feature engineering + model) work out of the box:
+the suite predicts through the whole pipeline, `model_features` filters
+validation data to the pipeline's raw input columns, and the SHAP-based
+explainability tests explain the final estimator against the engineered
+features (e.g. `num__age`, `cat__cat_a`).
+
+## Built-in test types
+
+| `type` (YAML) | Class | Checks |
+|---------------|-------|--------|
+| `minimum_accuracy` | `MinimumAccuracyTest` | Global metric above threshold |
+| `group_performance` | `GroupPerformanceTest` | Metric above threshold per subgroup |
+| `confidence_threshold` | `ConfidenceThresholdTest` | Metric floor via bootstrap CI (lower/upper bound) |
+| `robustness` | `RobustnessTest` | Performance under feature noise |
+| `data_invariant` | `DataInvariantTest` | Expected columns / null ratios |
+| `no_null` | `NoNullTest` | No missing values |
+| `data_drift` | `DataDriftTest` | PSI between train & validation |
+| `ks` | `KSTest` | KS p-value per column |
+| `equal_opportunity` | `EqualOpportunityTest` | Balanced TPR across protected groups |
+| `statistical_parity` | `StatisticalParityTest` | Balanced selection rate (+ 4/5ths rule) |
+| `feature_dominance` | `FeatureDominanceTest` | No single feature dominates attribution |
+| `top_features` | `TopFeaturesTest` | Top-K attributed features are expected |
+
+> Explainability tests use SHAP. Install with `pip install modeltest[explain]`
+> (or `modeltest`'s `explain` extra). You can also pass your own explainer
+> callable to any explainability test.
+
+## Custom tests
+
+Any class subclassing `ModelTest` can be referenced from a YAML suite
+directly, by dotted import path — no registration required:
+
+```yaml
+suite:
+  name: "Income Model"
+  tests:
+    - type: minimum_accuracy
+      params: {threshold: 0.85}
+    - type: myproject.custom_tests:ZeroPredictionShareTest
+      params: {min_positive_share: 0.01}
+```
+
+Both `module.path:Class` and `module.path.Class` work. The module is looked
+up on the import path (your current working directory is added automatically).
+Programmatic registration is also available for short, friendlier names:
+
+```python
+from modeltest import register
+register("zero_share", ZeroPredictionShareTest)
+# ...now use `type: zero_share` in YAML
+```
+
+## MLflow integration
+
+Log a finished validation run into [MLflow](https://mlflow.org/) as an
+experiment run — one param per test, one metric per numeric value, and the
+full JSON report saved as an artifact.
+
+```bash
+pip install modeltest[mlflow]
+```
+
+```python
+import mlflow
+from modeltest import ModelSuite
+from modeltest.integrations.mlflow import log_suite_result
+from modeltest.scenarios import MinimumAccuracyTest
+
+suite = ModelSuite(name="fraud-v2")
+suite.add_test(MinimumAccuracyTest(threshold=0.85))
+result = suite.run(model, X_val, y_val)
+
+with mlflow.start_run():
+    log_suite_result(result)
+```
+
+`log_suite_result` also supports a `run_id` parameter for logging into a
+specific (possibly already-finished) run, plus optional `param_prefix` /
+`metric_prefix` to namespace the logged names.
+
+## Development
+
+Install dev tools and run the quality gates:
+
+```bash
+make install          # pip install -e ".[dev]"
+make lint             # ruff check
+make format           # ruff format
+make test             # pytest (with 80% coverage gate)
+make precommit        # install git pre-commit hooks (lint+format)
+```
+
+Continuous integration mirrors these gates: `lint`, `test-library` (with
+coverage) and `validate-model` all run on every push / PR
+([.github/workflows/validate.yml](.github/workflows/validate.yml)).
+
+## CI/CD
+
+A ready-to-use [GitHub Actions workflow](.github/workflows/validate.yml) runs the
+library's own tests and validates your model contract on every push / PR. It
+publishes both reports (as JUnit) and fails the pipeline if the model doesn't
+meet its contract.
+
+The model job trains a sample model and validates it:
+
+```bash
+modeltest validate --suite examples/suite.yaml --model examples/model.pkl \
+  --data examples/validation.csv --target target --train-data examples/train.csv
+```
+
+To point it at your real artifacts, update the `validate-model` job's `run` step paths.
+

modeltest-0.2.0/README.md

@@ -0,0 +1,209 @@
+# modeltest
+
+> If you test your code, why not your model?
+
+`modeltest` is a unit-testing framework for machine learning models. It lets you
+define contracts for model quality, robustness, fairness, and data invariants,
+and run them automatically in your CI/CD pipeline — just like `pytest` for code.
+
+## Quick start
+
+```python
+from modeltest import ModelSuite
+from modeltest.scenarios import MinimumAccuracyTest, GroupPerformanceTest
+
+suite = ModelSuite(name="Fraud Detection")
+suite.add_test(MinimumAccuracyTest(threshold=0.85))
+suite.add_test(GroupPerformanceTest(metric="accuracy", threshold=0.8, group_col="gender"))
+
+result = suite.run(model, X_val, y_val, model_name="fraud_rf")
+print(result.report(style="table"))
+```
+
+## CLI
+
+After training, validate a model contract from the command line:
+
+```bash
+# Declarative suite (recommended)
+modeltest validate \
+  --suite suite.yaml \
+  --model model.pkl \
+  --data validation.csv \
+  --target target \
+  --train-data train.csv \
+  --output report.xml
+
+# Python suite (suite.py exposing `suite`)
+modeltest validate --suite suite.py --model model.pkl --data validation.csv --target target
+```
+
+Extras:
+- `--output report.xml` writes JUnit XML for CI reporters.
+- `--train-data` enables the drift tests (they compare train vs. validation distributions).
+- Exit code is `1` if any test fails, `0` otherwise.
+
+## Prediction caching
+
+Within a single `suite.run(...)`, predictions are computed once and reused
+across every test. `TestContext.predict()` caches by a content hash of the
+input, so tests predicting on the *same* data (`MinimumAccuracyTest`,
+`GroupPerformanceTest`, the fairness tests, ...) each reuse the result instead
+of re-running the model. Perturbed inputs (e.g. the robustness test's noisy
+copy) get their own cache entry, so caching never compromises correctness.
+
+Disable it if you need a fresh prediction every call:
+
+```python
+ctx = TestContext(model=model, X_val=X_val, y_val=y_val, cache_predictions=False)
+```
+
+## YAML suites
+
+Define your contract declaratively — no code needed:
+
+```yaml
+suite:
+  name: "Credit Scoring Model"
+  tests:
+    - type: minimum_accuracy
+      params: {threshold: 0.85}
+    - type: group_performance
+      params: {metric: accuracy, threshold: 0.8, group_col: "gender"}
+    - type: robustness
+      params: {noise_std: 0.01, max_drop: 0.03}
+    - type: data_drift
+      params: {features: [age, income], max_psi: 0.15}
+    - type: equal_opportunity
+      params: {protected: "gender", max_diff: 0.1}
+    - type: statistical_parity
+      params: {protected: "gender", max_diff: 0.1, min_ratio: 0.8}
+    - type: data_invariant
+      params: {expected_columns: [age, income], max_null_ratio: 0.02}
+```
+
+## Multi-framework support
+
+`TestContext` talks to models through a small adapter interface
+(`modeltest.wrappers`). Out of the box it normalizes:
+
+- **scikit-learn** estimators (`predict`, and `predict_proba` when available)
+- **PyTorch** `nn.Module` (`predict` = argmax over logits, `predict_proba` = softmax)
+- **Keras / TensorFlow** models (binary threshold or multiclass argmax)
+
+Pass any of these straight to `suite.run(model, ...)`; the right adapter is
+picked automatically. Custom framework? Implement a
+`ModelWrapper` subclass and pass an instance as the model.
+
+**scikit-learn `Pipeline`s** (feature engineering + model) work out of the box:
+the suite predicts through the whole pipeline, `model_features` filters
+validation data to the pipeline's raw input columns, and the SHAP-based
+explainability tests explain the final estimator against the engineered
+features (e.g. `num__age`, `cat__cat_a`).
+
+## Built-in test types
+
+| `type` (YAML) | Class | Checks |
+|---------------|-------|--------|
+| `minimum_accuracy` | `MinimumAccuracyTest` | Global metric above threshold |
+| `group_performance` | `GroupPerformanceTest` | Metric above threshold per subgroup |
+| `confidence_threshold` | `ConfidenceThresholdTest` | Metric floor via bootstrap CI (lower/upper bound) |
+| `robustness` | `RobustnessTest` | Performance under feature noise |
+| `data_invariant` | `DataInvariantTest` | Expected columns / null ratios |
+| `no_null` | `NoNullTest` | No missing values |
+| `data_drift` | `DataDriftTest` | PSI between train & validation |
+| `ks` | `KSTest` | KS p-value per column |
+| `equal_opportunity` | `EqualOpportunityTest` | Balanced TPR across protected groups |
+| `statistical_parity` | `StatisticalParityTest` | Balanced selection rate (+ 4/5ths rule) |
+| `feature_dominance` | `FeatureDominanceTest` | No single feature dominates attribution |
+| `top_features` | `TopFeaturesTest` | Top-K attributed features are expected |
+
+> Explainability tests use SHAP. Install with `pip install modeltest[explain]`
+> (or `modeltest`'s `explain` extra). You can also pass your own explainer
+> callable to any explainability test.
+
+## Custom tests
+
+Any class subclassing `ModelTest` can be referenced from a YAML suite
+directly, by dotted import path — no registration required:
+
+```yaml
+suite:
+  name: "Income Model"
+  tests:
+    - type: minimum_accuracy
+      params: {threshold: 0.85}
+    - type: myproject.custom_tests:ZeroPredictionShareTest
+      params: {min_positive_share: 0.01}
+```
+
+Both `module.path:Class` and `module.path.Class` work. The module is looked
+up on the import path (your current working directory is added automatically).
+Programmatic registration is also available for short, friendlier names:
+
+```python
+from modeltest import register
+register("zero_share", ZeroPredictionShareTest)
+# ...now use `type: zero_share` in YAML
+```
+
+## MLflow integration
+
+Log a finished validation run into [MLflow](https://mlflow.org/) as an
+experiment run — one param per test, one metric per numeric value, and the
+full JSON report saved as an artifact.
+
+```bash
+pip install modeltest[mlflow]
+```
+
+```python
+import mlflow
+from modeltest import ModelSuite
+from modeltest.integrations.mlflow import log_suite_result
+from modeltest.scenarios import MinimumAccuracyTest
+
+suite = ModelSuite(name="fraud-v2")
+suite.add_test(MinimumAccuracyTest(threshold=0.85))
+result = suite.run(model, X_val, y_val)
+
+with mlflow.start_run():
+    log_suite_result(result)
+```
+
+`log_suite_result` also supports a `run_id` parameter for logging into a
+specific (possibly already-finished) run, plus optional `param_prefix` /
+`metric_prefix` to namespace the logged names.
+
+## Development
+
+Install dev tools and run the quality gates:
+
+```bash
+make install          # pip install -e ".[dev]"
+make lint             # ruff check
+make format           # ruff format
+make test             # pytest (with 80% coverage gate)
+make precommit        # install git pre-commit hooks (lint+format)
+```
+
+Continuous integration mirrors these gates: `lint`, `test-library` (with
+coverage) and `validate-model` all run on every push / PR
+([.github/workflows/validate.yml](.github/workflows/validate.yml)).
+
+## CI/CD
+
+A ready-to-use [GitHub Actions workflow](.github/workflows/validate.yml) runs the
+library's own tests and validates your model contract on every push / PR. It
+publishes both reports (as JUnit) and fails the pipeline if the model doesn't
+meet its contract.
+
+The model job trains a sample model and validates it:
+
+```bash
+modeltest validate --suite examples/suite.yaml --model examples/model.pkl \
+  --data examples/validation.csv --target target --train-data examples/train.csv
+```
+
+To point it at your real artifacts, update the `validate-model` job's `run` step paths.
+

modeltest-0.2.0/modeltest/__init__.py

@@ -0,0 +1,28 @@
+from modeltest.config import register, unregister
+from modeltest.core.base import (
+    ModelSuite,
+    ModelTest,
+    SuiteResult,
+    TestContext,
+    TestResult,
+)
+from modeltest.core.runner import run_suite, run_test
+
+try:
+    from importlib.metadata import version as _metadata_version
+
+    __version__ = _metadata_version("modeltest")
+except Exception:  # noqa: BLE001 - not installed via pip
+    __version__ = "unknown"
+
+__all__ = [
+    "ModelTest",
+    "ModelSuite",
+    "SuiteResult",
+    "TestContext",
+    "TestResult",
+    "run_test",
+    "run_suite",
+    "register",
+    "unregister",
+]

modeltest-0.2.0/modeltest/cli.py

@@ -0,0 +1,92 @@
+"""Command-line interface: `modeltest validate`."""
+
+from __future__ import annotations
+
+import argparse
+import sys
+from typing import Optional
+
+from modeltest.core.base import ModelSuite
+
+
+def main(argv: Optional[list] = None) -> int:
+    parser = argparse.ArgumentParser(
+        prog="modeltest", description="Unit tests for machine learning models."
+    )
+    sub = parser.add_subparsers(dest="command", required=True)
+
+    validate = sub.add_parser("validate", help="Run a test suite against a model.")
+    validate.add_argument(
+        "--suite",
+        required=True,
+        help="Path to suite.py (defining `suite`) or suite.yaml (declarative).",
+    )
+    validate.add_argument("--model", required=True, help="Path to model (pickle).")
+    validate.add_argument("--data", required=True, help="Path to validation CSV.")
+    validate.add_argument(
+        "--target", default="target", help="Name of the target column."
+    )
+    validate.add_argument(
+        "--output", default=None, help="Write JUnit XML to this path."
+    )
+    validate.add_argument(
+        "--train-data", default=None, help="Optional training CSV (for drift tests)."
+    )
+    validate.set_defaults(func=_run_validate)
+
+    args = parser.parse_args(argv)
+    return args.func(args)
+
+
+def _load_suite(path: str) -> "ModelSuite":
+    import os
+
+    ext = os.path.splitext(path)[1].lower()
+    if ext in (".yaml", ".yml"):
+        from modeltest.config import load_suite_yaml
+
+        return load_suite_yaml(path)
+    # default: treat as a Python module exposing `suite`
+    import importlib.util
+
+    spec = importlib.util.spec_from_file_location("suite_module", os.path.abspath(path))
+    mod = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(mod)
+    return mod.suite
+
+
+def _run_validate(args: argparse.Namespace) -> int:
+    import os
+
+    import joblib
+    import pandas as pd
+
+    suite: ModelSuite = _load_suite(args.suite)
+    model = joblib.load(os.path.abspath(args.model))
+    df = pd.read_csv(args.data)
+    y = df[args.target]
+    X = df.drop(columns=[args.target])
+
+    X_train = None
+    if args.train_data:
+        train_df = pd.read_csv(args.train_data)
+        X_train = train_df.drop(columns=[args.target])
+
+    result = suite.run(
+        model, X, y, X_train=X_train, model_name=os.path.basename(args.model)
+    )
+
+    if args.output:
+        from modeltest.core.report import to_junit_xml
+
+        out_path = os.path.abspath(args.output)
+        os.makedirs(os.path.dirname(out_path), exist_ok=True)
+        with open(out_path, "w") as fh:
+            fh.write(to_junit_xml(result))
+
+    print(result.report(style="table"))
+    return 0 if result.passed else 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())