eval-toolkit 0.27.1__py3-none-any.whl

eval_toolkit-0.27.1.dist-info/METADATA

@@ -0,0 +1,314 @@
+Metadata-Version: 2.4
+Name: eval-toolkit
+Version: 0.27.1
+Summary: Reusable evaluation contracts for binary classification: metrics, bootstrap CIs, calibration, artifacts, and evidence gates.
+Project-URL: Homepage, https://github.com/brandon-behring/eval-toolkit
+Project-URL: Repository, https://github.com/brandon-behring/eval-toolkit.git
+Project-URL: Issues, https://github.com/brandon-behring/eval-toolkit/issues
+Project-URL: Changelog, https://github.com/brandon-behring/eval-toolkit/blob/main/CHANGELOG.md
+Author: Brandon Behring
+License-Expression: MIT
+License-File: LICENSE
+Keywords: binary-classification,bootstrap,calibration,evaluation,machine-learning,metrics
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Typing :: Typed
+Requires-Python: >=3.13
+Requires-Dist: jsonschema>=4.21
+Requires-Dist: numpy>=1.26
+Requires-Dist: scikit-learn>=1.4
+Requires-Dist: scipy>=1.11
+Provides-Extra: all
+Requires-Dist: hypothesis>=6.100; extra == 'all'
+Requires-Dist: matplotlib>=3.8; extra == 'all'
+Requires-Dist: pandas>=2.0; extra == 'all'
+Requires-Dist: pillow>=10.0; extra == 'all'
+Requires-Dist: pyarrow>=15.0; extra == 'all'
+Requires-Dist: pyyaml>=6.0; extra == 'all'
+Provides-Extra: dataframe
+Requires-Dist: pandas>=2.0; extra == 'dataframe'
+Provides-Extra: dev
+Requires-Dist: black>=24.0; extra == 'dev'
+Requires-Dist: hypothesis>=6.100; extra == 'dev'
+Requires-Dist: matplotlib>=3.8; extra == 'dev'
+Requires-Dist: mypy>=1.10; extra == 'dev'
+Requires-Dist: pandas>=2.0; extra == 'dev'
+Requires-Dist: pillow>=10.0; extra == 'dev'
+Requires-Dist: pre-commit>=3.7; extra == 'dev'
+Requires-Dist: pyarrow>=15.0; extra == 'dev'
+Requires-Dist: pytest-cov>=5.0; extra == 'dev'
+Requires-Dist: pytest-mpl>=0.16; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: pyyaml>=6.0; extra == 'dev'
+Requires-Dist: ruff>=0.5; extra == 'dev'
+Requires-Dist: sybil>=10.0; extra == 'dev'
+Provides-Extra: parquet
+Requires-Dist: pyarrow>=15.0; extra == 'parquet'
+Provides-Extra: plotting
+Requires-Dist: matplotlib>=3.8; extra == 'plotting'
+Requires-Dist: pillow>=10.0; extra == 'plotting'
+Provides-Extra: property
+Requires-Dist: hypothesis>=6.100; extra == 'property'
+Provides-Extra: validation
+Provides-Extra: yaml
+Requires-Dist: pyyaml>=6.0; extra == 'yaml'
+Description-Content-Type: text/markdown
+
+# eval-toolkit
+
+A **methodology-aware evaluation harness for binary classification**:
+metrics, bootstrap CIs, calibration, leakage detection, splitting,
+threshold selection, dataset loading, reproducibility manifests, and a
+slice-aware orchestrator that ties them together. Pure
+numpy/scipy/sklearn core; pandas/matplotlib/hypothesis are optional
+extras; PyTorch / HuggingFace / `datasets` are *consumer-side* (never
+required).
+
+Library-grade by design — every public function is type-annotated,
+every math kernel is documented with LaTeX + literature references,
+statistical validity (bootstrap CIs, MDE estimates, paired-difference
+tests) is built in, and the JSON outputs (`results.json` /
+`results_full.json` / `manifest.json`) ship with versioned [JSON
+Schemas](src/eval_toolkit/schemas/) so downstream parsers can gate on
+format changes.
+
+## Three-tier architecture
+
+```
+┌─ Tier 3 ─ Reproducibility scaffolding ─────────────────┐
+│  manifest.json + seeds + git_sha + data_hashes +       │
+│  gpu_info + leakage_report (NeurIPS-aligned)           │
+├─ Tier 2 ─ Protocol-based orchestration ────────────────┤
+│  Scorer / SliceAwareScorer / LeakageCheck / Splitter   │
+│  ThresholdSelector / DatasetLoader / SimilarityStrategy│
+│  Versioned (opt-in: per-object versions in manifest)   │
+├─ Tier 1 ─ Functional core ─────────────────────────────┤
+│  pr_auc / roc_auc / ECE variants / Brier / bootstrap_ci│
+│  paired_bootstrap_diff / cv_clt_ci / mde_from_ci       │
+│  reliability_curve / fit_temperature / fit_isotonic    │
+└────────────────────────────────────────────────────────┘
+```
+
+Pick the tier your task needs. Ad-hoc analysis: just call the
+functional core. Full eval pipelines: implement the Protocols. Every
+run: capture the manifest.
+
+## Documentation
+
+- **[Getting started](docs/getting-started.md)** — end-to-end
+  walkthrough for new users: install, define a Scorer, build slices,
+  run `evaluate()`, persist results, add a claim, render a plot.
+- **[Methodology curriculum](docs/methodology/README.md)** — 16
+  chapters on splits, metrics, calibration, evidence gates,
+  prediction artifacts, and more.
+- **[Schema reference](docs/schemas.md)** — field-by-field semantics
+  for `results.v1.json`, `results_full.v1.json`, `manifest.v1.json`.
+- **[Migration guides](docs/MIGRATION.md)** — v0.6→v0.7, v0.7→v0.8,
+  v0.8→v0.9.
+- **[Extending](docs/extending.md)** — Protocol-by-Protocol guide for
+  custom Scorers, Splitters, LeakageChecks, ThresholdSelectors,
+  DatasetLoaders, EvidenceGates.
+- **[Repo strategy](docs/repo-strategy.md)** — how the package is
+  organized, the 6-bucket target shape, and the checklist that
+  governs when to extract a sub-package into its own repo.
+
+## Methodology
+
+What good binary-classification evaluation looks like, with each
+concern mapped to the toolkit primitive that operationalizes it.
+
+- [`docs/methodology/`](docs/methodology/README.md) — the curriculum
+  (16 chapters). Recommended reading order:
+  [`leakage`](docs/methodology/leakage.md) →
+  [`splits`](docs/methodology/splits.md) →
+  [`thresholds`](docs/methodology/thresholds.md) →
+  [`calibration`](docs/methodology/calibration.md) →
+  [`comparison`](docs/methodology/comparison.md) →
+  [`bootstrap`](docs/methodology/bootstrap.md) →
+  [`length_stratification`](docs/methodology/length_stratification.md) →
+  [`text_dedup`](docs/methodology/text_dedup.md) →
+  [`versioning`](docs/methodology/versioning.md) →
+  [`fairness`](docs/methodology/fairness.md) →
+  [`reproducibility`](docs/methodology/reproducibility.md) →
+  [`testing`](docs/methodology/testing.md) →
+  [`reading_list`](docs/methodology/reading_list.md).
+- [`docs/MIGRATION.md`](docs/MIGRATION.md) — per-version migration
+  guides (v0.6→v0.7, v0.7→v0.8).
+- [`docs/roadmap.md`](docs/roadmap.md) — forward-looking tracker;
+  v1.0.0 path; consumer gap-doc cross-links.
+
+## Extending eval-toolkit
+
+How to plug your own scorers / leakage checks / splitters / loaders /
+threshold selectors into the harness.
+
+- [`docs/extending.md`](docs/extending.md) — Protocol-by-Protocol
+  guide, ~50-line full-harness recipe, project-layout pointer.
+
+## Worked examples
+
+- [`docs/examples/prompt_injection_walkthrough.md`](docs/examples/prompt_injection_walkthrough.md)
+  — End-to-end prompt-injection eval on a synthetic OWASP LLM01:2025
+  fixture; cross-links to the
+  [showcase repo](https://github.com/brandon-behring/prompt_injection_classifier_showcase)
+  for the real Lakera PINT walkthrough.
+- [`docs/examples/pytorch_scorer_example.md`](docs/examples/pytorch_scorer_example.md)
+  — HuggingFace transformer + LoRA `Scorer` adapter (batched inference,
+  GPU/CPU placement, deterministic-mode setup).
+
+## Install
+
+```bash
+uv venv
+uv pip install -e .[dev]
+```
+
+For consumers who only need the math kernels (no plotting, no pandas):
+
+```bash
+pip install eval-toolkit                        # core only: numpy/scipy/sklearn
+pip install "eval-toolkit[plotting]"            # adds matplotlib + pillow
+pip install "eval-toolkit[dataframe]"           # adds pandas
+pip install "eval-toolkit[all]"                 # everything
+```
+
+## Quick examples
+
+### Metrics
+
+```python
+import numpy as np
+from eval_toolkit import pr_auc, roc_auc, expected_calibration_error
+
+rng = np.random.default_rng(42)
+y = rng.integers(0, 2, size=200)
+# Clip to [0, 1] — ECE only meaningful on calibrated probabilities.
+s = np.clip(y + rng.normal(0, 0.3, size=200), 0, 1)
+
+print(f"PR-AUC: {pr_auc(y, s):.3f}")
+print(f"ROC-AUC: {roc_auc(y, s):.3f}")
+print(f"ECE (10 bins): {expected_calibration_error(y, s, n_bins=10):.3f}")
+```
+
+### Bootstrap confidence intervals
+
+```python
+from eval_toolkit import bootstrap_ci, paired_bootstrap_diff, pr_auc
+
+ci = bootstrap_ci(y, s, pr_auc, n_resamples=1000, seed=42)
+print(f"PR-AUC: {ci.point_estimate:.3f}  95% CI: [{ci.ci_low:.3f}, {ci.ci_high:.3f}]")
+
+# Paired bootstrap on the lift between two scorers (s_baseline must be in [0, 1] too).
+s_baseline = np.clip(rng.normal(0.5, 0.3, size=200), 0, 1)
+diff = paired_bootstrap_diff(y, s_baseline, s, pr_auc, n_resamples=1000, seed=42)
+print(f"Δ PR-AUC: {diff.delta:.3f}  overlaps zero: {diff.overlaps_zero}")
+```
+
+### Temperature scaling (Guo et al. 2017)
+
+```python
+from eval_toolkit import fit_temperature
+
+logits = rng.normal(size=(500, 2))
+labels = (logits[:, 1] > logits[:, 0]).astype(int)
+result = fit_temperature(logits, labels)
+print(f"Optimal T: {result['temperature']:.3f}")
+print(f"NLL: {result['nll_pre']:.3f} -> {result['nll_post']:.3f}")
+```
+
+### Reproducibility manifest (NeurIPS-aligned)
+
+```python
+import tempfile
+from pathlib import Path
+from eval_toolkit import build_manifest, write_manifest
+
+with tempfile.TemporaryDirectory() as run_dir:
+    # data_files: {name: path} → eval_toolkit hashes the files for you;
+    # versioned: any object with a `version` attribute (e.g. a scorer or
+    # leakage check) is captured by name → version in the manifest.
+    manifest = build_manifest(
+        run_id="quickstart-demo",
+        config={"threshold_criterion": "max_f1", "seed": 42},
+        seeds={"global": 42, "bootstrap": 42},
+    )
+    write_manifest(manifest, Path(run_dir))
+    # → run_dir/manifest.json: schema_version, git_sha, dirty_flag, code_versions,
+    #   env (python+platform), seeds, data_hashes, versioned_objects, gpu_info
+```
+
+## Modules
+
+| Module | Purpose |
+|---|---|
+| `eval_toolkit.metrics` | PR-AUC, ROC-AUC, ECE variants, Brier decomposition, prior-shift projection |
+| `eval_toolkit.thresholds` | `ThresholdSelector` Protocol + 6 reference impls (max-F1, target-recall/precision/FPR, Youden-J, cost-sensitive) |
+| `eval_toolkit.operating_points` | Fit thresholds on mixed-class slices and apply them to mixed or single-class target slices with provenance |
+| `eval_toolkit.bootstrap` | BCa + paired bootstrap, MDE estimates, two-level operating-point bootstrap, K-fold CLT-corrected CI |
+| `eval_toolkit.calibration` | Reliability curves, Bayes-optimal thresholds, isotonic/Platt/temperature scaling |
+| `eval_toolkit.harness` | `Scorer` Protocol + `evaluate(...)` + `evaluate_folded(...)` slice-aware orchestrators |
+| `eval_toolkit.leakage` | `LeakageCheck` Protocol + 7 reference impls (exact / near / encoding-obfuscated / cross-split / label-conflict / group / temporal); `Versioned` opt-in Protocol |
+| `eval_toolkit.splits` | `Splitter` Protocol + 5 reference impls (holdout / stratified / group / source-disjoint / time-series) |
+| `eval_toolkit.loaders` | `DatasetLoader` Protocol + 4 reference impls (DataFrame / SingleSlice / ParquetGlob / HF datasets) with Croissant-compatible `describe()` |
+| `eval_toolkit.manifest` | `RunManifest` (NeurIPS-aligned) + source-role / guardrail metadata + `build_manifest` / `write_manifest` |
+| `eval_toolkit.claims` | Generic evidence gates and `ClaimReport` for claim-mode vs exploratory-mode checks |
+| `eval_toolkit.text_dedup` | `SimilarityStrategy` Protocol + 5 strategies (TF-IDF / hash / embedding / Jaccard / MinHash-LSH); `near_dedup` / `cross_dedup` orchestrators |
+| `eval_toolkit.plotting` | PR curves, reliability diagrams, confusion matrices, score histograms, lift CIs |
+| `eval_toolkit.provenance` | File hashing, run-directory layout, figure metadata sidecar |
+| `eval_toolkit/schemas/` | Bundled JSON Schemas (`results.v1.json`, `results_full.v1.json`, `manifest.v1.json`) — load via `importlib.resources.files("eval_toolkit") / "schemas"` (not an importable Python module) |
+| `eval_toolkit.paths` | Repo-relative path normalization |
+| `eval_toolkit.seeds` | `set_global_seeds` (random + numpy + optional torch) |
+| `eval_toolkit.config` | `frozen_config` decorator + `from_yaml` loader |
+| `eval_toolkit.docs` | Anchor-based markdown rendering with formatter registry |
+
+## Fast iteration loop
+
+For development, skip slow tests with:
+
+```bash
+make fast              # or: nox -s fast
+# under the hood: uv run pytest -m "not slow" -q
+```
+
+CI runs the full suite (including `slow`) on every push. The `slow`
+marker is applied to tests exceeding ~2s (mostly Hypothesis property
+tests with large `max_examples` and a few bootstrap tests with
+`n_resamples >= 200`). `make fast` keeps the developer iteration loop
+under ~30 seconds.
+
+## Downstream contract testing (v4 sibling-smoke)
+
+A separate CI workflow (`.github/workflows/v4-smoke.yml`) checks out
+the downstream consumer `prompt-injection-v4` at `main`, installs it
+with this branch's eval-toolkit as an editable sibling dep (via v4's
+`[tool.uv.sources]`), and runs v4's fast `-m smoke` suite. This catches
+contract regressions at PR time rather than in v4's own CI post-merge.
+
+The workflow requires a `HF_TOKEN` repo secret (gated HuggingFace
+datasets used by v4's smoke fixtures). Set it at:
+`https://github.com/brandon-behring/eval-toolkit/settings/secrets/actions`
+
+The workflow runs with `continue-on-error: true` during a 2-3 week
+trial period; it'll be promoted to a required gate once the false-
+positive rate (from independent v4 main breakage or HF rate-limits)
+is characterized.
+
+## Standards
+
+See [`STYLE.md`](STYLE.md) for the full reconciled coding standards (formatting,
+naming, errors, docstrings, tests, packaging).
+
+## Versioning
+
+Semver from v0.1.0. See [`CHANGELOG.md`](CHANGELOG.md).
+
+## License
+
+MIT — see [`LICENSE`](LICENSE).

eval_toolkit-0.27.1.dist-info/RECORD

@@ -0,0 +1,36 @@
+eval_toolkit/__init__.py,sha256=jXtUumPAckGEUq6yKzouWeV6Ijm0VwS6CWc12tbpGpg,11202
+eval_toolkit/__main__.py,sha256=vvpb6rCQBjEJm1dTR0smtByrlJhx8RUvvIdrOm8mtvg,5807
+eval_toolkit/_version.py,sha256=Vn17cRk3n5FrF8Qk1_y919scG-HD6basX1ieIMZkVes,92
+eval_toolkit/analysis.py,sha256=LdO-SQNZTqD9OGBHX-HiLGTq89DvZ5j_Vp5K7Iy0t8Q,6798
+eval_toolkit/artifacts.py,sha256=s4ShmEiHj3vdb1t3medzh9nEav5QQEFfiXvDIZN3lfM,14107
+eval_toolkit/bootstrap.py,sha256=XrOzy1q6COHBGTlw3oSIiR3lEpy5KnXWssV0r_S-fLQ,47964
+eval_toolkit/calibration.py,sha256=zk8cW-q9vAyQZb3Rr-PdPuqhVhq29ITl0h7NDhMozF0,41960
+eval_toolkit/claims.py,sha256=CGVyltfCAs4PkyIp5R59qGCHpyEO1e7L0OsEt7vHAw8,23796
+eval_toolkit/config.py,sha256=GTRI5kY7manl4GDnfCzIsO8k90OiLBtt77SD1KEINC4,3253
+eval_toolkit/docs.py,sha256=stk1F_nyX3nBgdJN_Hwk88_mh9aCgHfGQdTduq1Qis4,8754
+eval_toolkit/evidence.py,sha256=uHRYrIp7YxadojRnz0jdznvySgJWtdOYFcnHp_d20xI,2475
+eval_toolkit/harness.py,sha256=F0lILnlVdSpJPUBNcOpkYfpf8i2AHUGOVrxR47DH_5I,46684
+eval_toolkit/leakage.py,sha256=xborrDqdf8hWHBjbcSXEcgKr2F8C3J0Itg5AZ7J4gBc,42271
+eval_toolkit/loaders.py,sha256=IPgapA97G3u7uOsG6JRyZ5HKaGf97LMl09MzdXIuSE4,15789
+eval_toolkit/manifest.py,sha256=FqhTvAedzKwq9bTU43KVLIhbKtX6bSyotK9jZGtqBxA,25385
+eval_toolkit/metrics.py,sha256=i0Jla2dolGQ3Dqor3OgvPrCEGt-sesC40eKjGE1podg,59625
+eval_toolkit/operating_points.py,sha256=IAOiOIMuGGE61q_z10VKAzPsjmGltzH0RKV2sYPpSag,6646
+eval_toolkit/paths.py,sha256=oS3nsqXLQJpyI6l73PKsCQe9vM2icw6gSEopgfuw35s,4156
+eval_toolkit/plotting.py,sha256=I8caaWUXM6G9FV3TiFDAs_zqrQkdWSEUzCkhodsQw4A,32867
+eval_toolkit/protocols.py,sha256=QobXqvqI3esfznwlMQZ7qoGN3fV-APMDcEvmaJPnzbU,2654
+eval_toolkit/provenance.py,sha256=mG5E7h1uBp1nQpgNuERC2OxUzTymkdYbn5QzKWzqFV8,7897
+eval_toolkit/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+eval_toolkit/seeds.py,sha256=54fhkO0V6mjTcy1f5nKWY3u4_NQamfW9LaGywPIlwO4,4180
+eval_toolkit/splits.py,sha256=rkU6XHFpntTU354SGe_ppSL7PSu7Qm1gaaSAFra5kjI,18168
+eval_toolkit/text_dedup.py,sha256=8_jYYdxL-_sXgKNLCglYNEmXiWCSRjpL3blVQOX1R6w,52902
+eval_toolkit/thresholds.py,sha256=uxnDYCPEQ2YxVHnM7IWtqsccLoyoHhOYdNqRgWWkYeM,30220
+eval_toolkit/schemas/manifest.v1.json,sha256=hNoyOqcsvxJMHm_fvNRc7hLAYe5UMy90-UVEZNDgV_4,5390
+eval_toolkit/schemas/manifest.v2.json,sha256=fW8PqIRHy3VCrfniiFz4DyJbz1B1Z0PCqr4nImhXJTs,7044
+eval_toolkit/schemas/manifest.v3.json,sha256=lca1u9PbSYpBAQ_8melv4OUhBqn57ZhRs6qLXSKWcWA,5924
+eval_toolkit/schemas/results.v1.json,sha256=jDK9bs4VECZwP2PEAAGhQYFGJANDglHkXsTjgxLPvGM,3182
+eval_toolkit/schemas/results_full.v1.json,sha256=gFQa5vG8lRJR2lJORojB-ltI5sCGaLIfwseoHEbNHOI,2760
+eval_toolkit-0.27.1.dist-info/METADATA,sha256=Z23WDxq8fg8c1uTPpMz6IdSz1LjVPHmJHaVkwy_jVHI,14312
+eval_toolkit-0.27.1.dist-info/WHEEL,sha256=QccIxa26bgl1E6uMy58deGWi-0aeIkkangHcxk2kWfw,87
+eval_toolkit-0.27.1.dist-info/entry_points.txt,sha256=sMt5QJp6AXbj6k1oMIFXNY5jqzmpzQK0loqqAWzamGo,60
+eval_toolkit-0.27.1.dist-info/licenses/LICENSE,sha256=ILekgpB5bjyOq3gi8MyG43Bd0fdQSoylFnWdeJgw5AA,1072
+eval_toolkit-0.27.1.dist-info/RECORD,,

eval_toolkit-0.27.1.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.29.0
+Root-Is-Purelib: true
+Tag: py3-none-any

eval_toolkit-0.27.1.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+eval-toolkit = eval_toolkit.__main__:main

eval_toolkit-0.27.1.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Brandon Behring
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.