eval-toolkit 0.41.0__tar.gz → 0.43.0__tar.gz

{eval_toolkit-0.41.0 → eval_toolkit-0.43.0}/CHANGELOG.md

@@ -5,7 +5,119 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [Unreleased]
+## [0.43.0] — 2026-05-19 — P1 batch: OOD manifest loader + character_injection sweep + ActivationDeltaProbe (closes #48, #49, #53)
+
+### Added
+
+- `ood_dataset_from_manifest(yaml_path, slices=..., cache_dir=...)` —
+  declarative loader for multiple OOD eval slates (BIPIA, AgentDojo,
+  InjecAgent, NotInject, PINT, LLMail-Inject, …) into a single
+  unified DataFrame with columns `text` / `label` / `source` /
+  `row_id` / `sha`. Bytes are downloaded once, sha256-verified
+  against the manifest, and cached on-disk keyed by content hash
+  (default `~/.cache/eval-toolkit/ood/`). Closes #48 — drops the
+  per-source loader boilerplate carried by
+  `prompt-injection-portfolio` and `prompt-injection-detection-submission`.
+- `OodManifestLoader` — `DatasetLoader`-Protocol-compliant wrapper
+  around the factory, returning `{"all": EvalSlice}` with
+  `source` as the default strata column for harness pipelines.
+- `src/eval_toolkit/schemas/ood_manifest.v1.json` — Draft 2020-12
+  JSON schema for the OOD manifest YAML; auto-validated by
+  `uv run eval-toolkit schemas check`.
+- `eval_toolkit.adversarial` — new module with character-injection
+  bypass suite (Microsoft Research 2024, arXiv 2404.13208).
+  Six core techniques shipped as frozen-dataclass strategies:
+  `ZeroWidthSpaceInjection`, `HomoglyphSubstitution`,
+  `DiacriticInjection`, `WhitespaceInjection`, `CaseRandomization`,
+  `PunctuationInjection`. All implement a `CharacterInjectionStrategy`
+  Protocol with `transform(text: str) -> str`. Six advanced techniques
+  (bidi RTL, tag stripping, synonym, token splitting, Unicode
+  normalization, invisible chars) scheduled for v0.43.1 — the sweep
+  API stabilizes in v0.43.0 so the v0.43.1 additions are pure
+  extensions. Closes #49 (core-6).
+- `adversarial.sweep(texts, scorer, techniques="all", threshold=0.5)`
+  — Scorer-Protocol-compliant adversarial-robustness sweep. Returns
+  a DataFrame with `(text_id, technique, original_score,
+  transformed_score, asr)` rows for matrix analysis.
+  Aggregate ASR with `df.groupby("technique")["asr"].mean()`.
+- `adversarial.character_injection` — `SimpleNamespace` exposing the
+  function-style API from the upstream issue spec
+  (`character_injection.zero_width_space(text)`,
+  `character_injection.sweep(...)`, etc.).
+- `eval_toolkit.probes` — new module with `ActivationDeltaProbe`:
+  TaskTracker-style linear probe over HuggingFace transformer
+  hidden-state activation deltas (Abdelnabi et al. 2024,
+  arXiv 2406.00799). Backbone-agnostic (encoder OR decoder).
+  Sklearn-compatible API: `.fit(clean_texts, injected_texts)`,
+  `.predict()` → `(n,)`, `.predict_proba()` → `(n, 2)`,
+  `.coef_`, `.classes_`. Activations cached to
+  `$XDG_CACHE_HOME/eval-toolkit/probes/` keyed by
+  `(backbone, layer_index, aggregate, sha256(text))` so re-runs are
+  near-instant. Aggregate modes: `mean`, `max`, `cls`. Closes #53.
+- `Probe` Protocol — minimal sklearn-shaped probe surface (`fit`,
+  `predict`, `predict_proba`, `coef_`, `classes_`). Distinct from
+  `Scorer` (which returns 1-D `P(positive)`); wrap with
+  `lambda p, X: p.predict_proba(X)[:, 1]` to adapt.
+- `ActivationExtractor` Protocol — pluggable hidden-state-extraction
+  contract for `ActivationDeltaProbe`; injectable for tests to avoid
+  loading a real backbone.
+- New optional extra `[probes] = torch>=2.0, transformers>=4.40`.
+  Follows the `[embeddings]` precedent — opt-in only, NOT in
+  `[all]` or `[dev]`, since the transitive install is ~600MB+.
+  Module is base-install-safe: a friendly `ImportError` fires only
+  if you try to use the default HF extractor without the extra.
+
+## [0.42.0] — 2026-05-19 — fit_isotonic_binary completes 4-calibrator family (closes #44)
+
+Final element of the binary scalar-prob calibrator family started by
+`fit_temperature_binary` (v0.35.0). All four now uniformly return
+`(params, apply)`:
+
+| Function | Params | Shipped |
+|---|---|---|
+| `fit_temperature_binary` | `(T,)` — single float | v0.35.0 |
+| `fit_isotonic_binary`    | `None` — non-parametric | **v0.42.0** |
+| `fit_platt_binary`       | `(a, b)` | v0.40.0 |
+| `fit_beta_binary`        | `(a, b, c)` | v0.40.0 |
+
+Consumer code can now iterate the family with a single shape, used
+to distinguish parametric from non-parametric via
+`if params is not None`:
+
+```text
+CALIBRATORS = {
+    "temperature": fit_temperature_binary,
+    "isotonic":    fit_isotonic_binary,
+    "platt":       fit_platt_binary,
+    "beta":        fit_beta_binary,
+}
+for name, fit_fn in CALIBRATORS.items():
+    params, apply = fit_fn(y_val, p_val)
+    calibrated = apply(p_test)
+    if params is not None:
+        manifest.record(f"{name}_params", params)
+```
+
+This matches the consumer's calibration-battery pattern in
+`prompt-injection-detection-prototype` (their ADR-056 supersedes
+ADR-023 to adopt the canonical `(params, apply)` shape across the
+full 4-calibrator audit battery).
+
+### Added
+
+- **`eval_toolkit.fit_isotonic_binary(y_true, y_score) -> (None,
+  apply)`** — thin wrapper over `fit_isotonic_calibrator`. The
+  `None` in the params slot encodes "non-parametric" (isotonic
+  regression is a monotone step function, no scalar params to log).
+- 6 new unit tests in `tests/test_calibration_binary_adapters.py`
+  including a 4-calibrator family-iteration integration test that
+  verifies the `None`-vs-tuple convention.
+
+### Protocol stability
+
+Additive only. No Tier-2 Protocol shape edits. v0.42 is minor 3 of
+consecutive-without-Protocol-changes (v0.40 + v0.41 + v0.42). Gate 2
+stays MET.
 
 ## [0.41.0] — 2026-05-18 — Croissant end-to-end (closes #42, v1.0 Gate 4 MET)
 

{eval_toolkit-0.41.0 → eval_toolkit-0.43.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: eval-toolkit
-Version: 0.41.0
+Version: 0.43.0
 Summary: Reusable evaluation contracts for binary classification: metrics, bootstrap CIs, calibration, artifacts, and evidence gates.
 Project-URL: Homepage, https://github.com/brandon-behring/eval-toolkit
 Project-URL: Documentation, https://brandon-behring.github.io/eval-toolkit/
@@ -67,6 +67,9 @@ Requires-Dist: pyarrow>=15.0; extra == 'parquet'
 Provides-Extra: plotting
 Requires-Dist: matplotlib>=3.8; extra == 'plotting'
 Requires-Dist: pillow>=10.0; extra == 'plotting'
+Provides-Extra: probes
+Requires-Dist: torch>=2.0; extra == 'probes'
+Requires-Dist: transformers>=4.40; extra == 'probes'
 Provides-Extra: property
 Requires-Dist: hypothesis>=6.100; extra == 'property'
 Provides-Extra: transformers
@@ -308,22 +311,6 @@ tests with large `max_examples` and a few bootstrap tests with
 `n_resamples >= 200`). `make fast` keeps the developer iteration loop
 under ~30 seconds.
 
-## Downstream contract testing (v4 sibling-smoke)
-
-A separate CI workflow (`.github/workflows/v4-smoke.yml`) checks out
-the downstream consumer `prompt-injection-v4` at `main`, installs it
-with this branch's eval-toolkit as an editable sibling dep (via v4's
-`[tool.uv.sources]`), and runs v4's fast `-m smoke` suite. This catches
-contract regressions at PR time rather than in v4's own CI post-merge.
-
-The workflow requires a `HF_TOKEN` repo secret (gated HuggingFace
-datasets used by v4's smoke fixtures). Set it at:
-`https://github.com/brandon-behring/eval-toolkit/settings/secrets/actions`
-
-The workflow runs with `continue-on-error: true` during a 2-3 week
-trial period; it'll be promoted to a required gate once the false-
-positive rate (from independent v4 main breakage or HF rate-limits)
-is characterized.
 
 ## Standards
 

{eval_toolkit-0.41.0 → eval_toolkit-0.43.0}/README.md

@@ -230,22 +230,6 @@ tests with large `max_examples` and a few bootstrap tests with
 `n_resamples >= 200`). `make fast` keeps the developer iteration loop
 under ~30 seconds.
 
-## Downstream contract testing (v4 sibling-smoke)
-
-A separate CI workflow (`.github/workflows/v4-smoke.yml`) checks out
-the downstream consumer `prompt-injection-v4` at `main`, installs it
-with this branch's eval-toolkit as an editable sibling dep (via v4's
-`[tool.uv.sources]`), and runs v4's fast `-m smoke` suite. This catches
-contract regressions at PR time rather than in v4's own CI post-merge.
-
-The workflow requires a `HF_TOKEN` repo secret (gated HuggingFace
-datasets used by v4's smoke fixtures). Set it at:
-`https://github.com/brandon-behring/eval-toolkit/settings/secrets/actions`
-
-The workflow runs with `continue-on-error: true` during a 2-3 week
-trial period; it'll be promoted to a required gate once the false-
-positive rate (from independent v4 main breakage or HF rate-limits)
-is characterized.
 
 ## Standards
 

{eval_toolkit-0.41.0 → eval_toolkit-0.43.0}/pyproject.toml

@@ -63,6 +63,12 @@ embeddings = ["sentence-transformers>=3.0"]
 # itself does not import transformers, so the optional install is
 # strictly for callers wanting AutoTokenizer.from_pretrained(...).
 transformers = ["transformers>=4.0"]
+# v0.43.0: ActivationDeltaProbe (TaskTracker-style linear activation probe;
+# closes #53). Pulls torch + transformers (~600MB+ transitive). Follows
+# the [embeddings] precedent: opt-in only, NOT in [all] / [dev]. Module
+# is base-install-safe (lazy imports inside ActivationDeltaProbe methods);
+# the extra is strictly for callers wanting to actually fit / predict.
+probes = ["torch>=2.0", "transformers>=4.40"]
 # DEPRECATED (announced v0.30.1, removal v0.33.0).
 #
 # Retained as a transitive no-op so `pip install eval-toolkit[validation]`

{eval_toolkit-0.41.0 → eval_toolkit-0.43.0}/src/eval_toolkit/__init__.py

@@ -30,6 +30,20 @@ _logging.getLogger("eval_toolkit").addHandler(_logging.NullHandler())
 # dividers below are informational only; the snapshot in
 # tests/golden/public_api/ reads dict keys + values, not comments.
 _EXPORTS: dict[str, str] = {
+    # --- adversarial ---
+    "CORE_TECHNIQUES": "eval_toolkit.adversarial",
+    "CaseRandomization": "eval_toolkit.adversarial",
+    "CharacterInjectionStrategy": "eval_toolkit.adversarial",
+    "DiacriticInjection": "eval_toolkit.adversarial",
+    "HomoglyphSubstitution": "eval_toolkit.adversarial",
+    "PunctuationInjection": "eval_toolkit.adversarial",
+    "WhitespaceInjection": "eval_toolkit.adversarial",
+    "ZeroWidthSpaceInjection": "eval_toolkit.adversarial",
+    "character_injection": "eval_toolkit.adversarial",
+    # --- probes ---
+    "ActivationDeltaProbe": "eval_toolkit.probes",
+    "ActivationExtractor": "eval_toolkit.probes",
+    "Probe": "eval_toolkit.probes",
     # --- analysis ---
     "CsvPredictionReader": "eval_toolkit.analysis",
     "JsonlPredictionReader": "eval_toolkit.analysis",
@@ -85,6 +99,7 @@ _EXPORTS: dict[str, str] = {
     "bayes_optimal_threshold": "eval_toolkit.calibration",
     "fit_beta_binary": "eval_toolkit.calibration",
     "fit_beta_calibrator": "eval_toolkit.calibration",
+    "fit_isotonic_binary": "eval_toolkit.calibration",
     "fit_isotonic_calibrator": "eval_toolkit.calibration",
     "fit_platt_binary": "eval_toolkit.calibration",
     "fit_platt_calibrator": "eval_toolkit.calibration",
@@ -155,8 +170,10 @@ _EXPORTS: dict[str, str] = {
     "DataFrameLoader": "eval_toolkit.loaders",
     "DatasetLoader": "eval_toolkit.loaders",
     "HFDatasetsLoader": "eval_toolkit.loaders",
+    "OodManifestLoader": "eval_toolkit.loaders",
     "ParquetGlobLoader": "eval_toolkit.loaders",
     "SingleSliceLoader": "eval_toolkit.loaders",
+    "ood_dataset_from_manifest": "eval_toolkit.loaders",
     # --- manifest ---
     "MANIFEST_SCHEMA_VERSION": "eval_toolkit.manifest",
     "RunManifest": "eval_toolkit.manifest",

{eval_toolkit-0.41.0 → eval_toolkit-0.43.0}/src/eval_toolkit/_version.py

@@ -2,4 +2,4 @@
 
 __all__ = ["__version__"]
 
-__version__ = "0.41.0"
+__version__ = "0.43.0"