eval-toolkit 1.5.0__tar.gz → 1.6.0__tar.gz

{eval_toolkit-1.5.0 → eval_toolkit-1.6.0}/CHANGELOG.md

@@ -5,6 +5,50 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.6.0] — 2026-05-29 — Tier-2 `eda` Job-2 + Job-3: shortcut + shift diagnostics (#86, #87)
+
+`eval_toolkit.eda.*` ADDITIVE per [ADR 0003](docs/source/adr/0003-stability-contract-and-gate3-methodology.md) — Tier-2, backward-compatible. Completes the EDA-first analytic layer above the v1.5.0 Job-1 integrity gate: **Job-2** lexical shortcut diagnostics (`lexical_association`, #86) and **Job-3** distribution-shift quantification (`distribution_shift`, #87). Both are dogfooded by the consumer portfolio's pre-modeling OOD-wall prediction (V5 + V9).
+
+### Added — Tier-2 `eda.lexical_association` shortcut diagnostics (Job-2: C1 + C2)
+
+`eval_toolkit.eda` ADDITIVE per [ADR 0003](docs/source/adr/0003-stability-contract-and-gate3-methodology.md) — Tier-2, torch-free (NumPy + scikit-learn). The analytic layer above the Job-1 integrity gate: *"is the label recoverable from a surface shortcut that will not transfer out-of-distribution?"*
+
+- **C1 — `weighted_log_odds` / `class_lexical_association`:** Monroe, Colaresi & Quinn (2008)
+  informative-Dirichlet weighted log-odds-ratio z-scores + smoothed PMI per token, with a
+  `min_count` rare-token floor (the V5 pitfall). Returns a `LexicalAssociationResult`
+  (`top_a` / `top_b` / `to_dict`); tokens ordered by descending z-score.
+- **C2 — `competency_baselines`:** partial-input baselines (length-only, char-n-gram, BoW)
+  fit on a train split and scored on a test split → `CompetencyResult` of per-baseline
+  average-precision vs the positive-prevalence floor (the *shortcut floor*; Feng, Wallace &
+  Boyd-Graber, ACL 2019 caveat documented). Vectorizers fit on train only (no test leakage);
+  empty or single-class train/test raises a diagnostic `ValueError`.
+- Exported via `from eval_toolkit.eda import ...`; 100% line+branch coverage; mypy-strict clean.
+
+### Added — Tier-2 `eda.distribution_shift` covariate-shift quantification (Job-3: E1)
+
+`eval_toolkit.eda` ADDITIVE per [ADR 0003](docs/source/adr/0003-stability-contract-and-gate3-methodology.md) — Tier-2. Public functions take **feature matrices**, so the module is base-install-safe (NumPy + SciPy + scikit-learn); embed text first with `eval_toolkit.embeddings.make_minilm_embedder` (`[embeddings]` extra) or any vectorizer.
+
+- **`proxy_a_distance`:** Ben-David et al. (2006/2010) PAD = `2(1 − 2ε)` from a **linear**
+  domain classifier's **k-fold CV** error, with **fixed strong regularization** (small `C`) —
+  *not* the high-`C` RBF-SVM-on-`predict_proba` recipe that overfits to `PAD ≈ 2` at small `n`.
+  Optional bootstrap CI.
+- **`maximum_mean_discrepancy`:** Gretton et al. (2012) **unbiased** RBF-kernel MMD² U-statistic +
+  **median-heuristic bandwidth** (freezable across folds) + **permutation-test** p-value
+  (Phipson & Smyth 2010, `(1+count)/(B+1)`, never zero). Optional bootstrap CI.
+- **`knn_purity`:** mean fraction of each point's k nearest neighbours sharing its domain label.
+- **`median_bandwidth`** helper + the **`distribution_shift`** orchestrator (all three) +
+  `PadResult` / `MmdResult` / `KnnPurityResult` / `DistributionShiftResult` dataclasses (`to_dict`).
+- Docstrings carry the pre-registered caveats: distance is **necessary-not-sufficient** for OOD
+  collapse (fuse with shortcut-exposure); a non-significant MMD p is not "no shift"; cross-dataset
+  distances are ordinal-only (covariate vs label-semantics conflation). 100% line+branch coverage.
+
+### Fixed
+
+- **Public-API golden `__version__` drift:** the `v1.5.0` release commit bumped
+  `_version.py` to `1.5.0` but did not regenerate `tests/golden/public_api/snapshot.json`,
+  which still pinned `'1.4.0'` — leaving `test_public_api_drift_guard` red on `main` (and on
+  every branch cut from it). Regenerated the golden (the diff is the `__version__` value only).
+
 ## [1.5.0] — 2026-05-29 — Tier-2 `eda` layer (#83) + schema-aware `HFDatasetsLoader` (#85)
 
 Tier-2 / `loaders` ADDITIVE per [ADR 0003](docs/source/adr/0003-stability-contract-and-gate3-methodology.md) — backward-compatible.

{eval_toolkit-1.5.0 → eval_toolkit-1.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: eval-toolkit
-Version: 1.5.0
+Version: 1.6.0
 Summary: Reusable evaluation contracts for binary classification: metrics, bootstrap CIs, calibration, artifacts, and evidence gates.
 Project-URL: Homepage, https://github.com/brandon-behring/eval-toolkit
 Project-URL: Documentation, https://brandon-behring.github.io/eval-toolkit/

{eval_toolkit-1.5.0 → eval_toolkit-1.6.0}/src/eval_toolkit/_version.py

@@ -2,4 +2,4 @@
 
 __all__ = ["__version__"]
 
-__version__ = "1.5.0"
+__version__ = "1.6.0"

eval_toolkit-1.6.0/src/eval_toolkit/eda/__init__.py

@@ -0,0 +1,144 @@
+"""``eval_toolkit.eda`` — EDA-first dataset integrity gating (Tier-2 surface).
+
+This subpackage is the **Job-1 integrity gate** of an EDA-first research
+program: thin, composable, torch-free per-split profiling + dataset-soundness
+gates, built by reusing the v1.4.0 :mod:`eval_toolkit.leakage`,
+:mod:`~eval_toolkit.text_dedup`, :mod:`~eval_toolkit.claims`, and
+:mod:`~eval_toolkit.artifacts` primitives.
+
+Stability tier
+--------------
+Public access is ``eval_toolkit.eda.*`` — **Tier-2** per ADR 0003. This layer
+is intentionally evolvable and is **not** part of the v2.0-frozen top-level
+:mod:`eval_toolkit` surface; nothing here is added to the package-root
+``_EXPORTS`` / ``__all__``. Import explicitly::
+
+    from eval_toolkit.eda import audit_dataset, DataAudit, SplitSummary
+
+Scope
+-----
+- **Job-1 integrity gate** (``data_audit`` + ``obfuscation``): row counts, class
+  balance, text-length quantiles, dedup / cross-split leakage, obfuscation
+  prevalence.
+- **Job-2 lexical shortcut diagnostics** (``lexical_association``): weighted
+  log-odds + PMI (C1) and partial-input / competency baselines (C2) — torch-free
+  (NumPy + scikit-learn).
+- **Job-3 distribution shift** (``distribution_shift``): proxy-A-distance, MMD
+  (permutation-tested), and kNN purity (E1) — operates on feature matrices, so
+  still base-install-safe (NumPy + SciPy + scikit-learn).
+
+The shift functions take **feature matrices**, not text — embed first with
+:func:`eval_toolkit.embeddings.make_minilm_embedder` (the optional
+``[embeddings]`` extra) or any vectorizer. UMAP / 2-D projections stay caller-side.
+"""
+
+from __future__ import annotations
+
+from eval_toolkit.eda.data_audit import (
+    DEFAULT_MAX_NEG_POS_RATIO,
+    DEFAULT_MIN_NEG_POS_RATIO,
+    DEFAULT_PCT_OVER_CONTEXT_THRESHOLD,
+    EDA_AUDIT_SCHEMA_VERSION,
+    DataAudit,
+    SplitSummary,
+    Tokenizer,
+    audit_dataset,
+    class_balance,
+    length_quantiles,
+    summarize_split,
+)
+from eval_toolkit.eda.distribution_shift import (
+    DEFAULT_KNN_K,
+    DEFAULT_MMD_PERMUTATIONS,
+    DEFAULT_PAD_C,
+    DEFAULT_PAD_FOLDS,
+    DistributionShiftResult,
+    KnnPurityResult,
+    MmdResult,
+    PadResult,
+    distribution_shift,
+    knn_purity,
+    maximum_mean_discrepancy,
+    median_bandwidth,
+    proxy_a_distance,
+)
+from eval_toolkit.eda.lexical_association import (
+    DEFAULT_CHAR_NGRAM_RANGE,
+    DEFAULT_MIN_COUNT,
+    DEFAULT_PRIOR_SCALE,
+    BaselineScore,
+    CompetencyResult,
+    LexicalAssociationResult,
+    StrTokenizer,
+    class_lexical_association,
+    competency_baselines,
+    default_tokenizer,
+    weighted_log_odds,
+)
+from eval_toolkit.eda.obfuscation import (
+    BASE64_ENTROPY_THRESHOLD,
+    HEX_ENTROPY_THRESHOLD,
+    ObfuscationProfile,
+    analyze_obfuscation,
+    count_invisible_chars,
+    has_high_entropy_alnum_run,
+    has_rot13_marker,
+    is_leeted_token,
+    leetspeak_counts,
+    nfkc_changed,
+    nfkc_char_delta,
+    shannon_entropy,
+)
+
+__all__ = [
+    # --- constants ---
+    "BASE64_ENTROPY_THRESHOLD",
+    "DEFAULT_CHAR_NGRAM_RANGE",
+    "DEFAULT_KNN_K",
+    "DEFAULT_MAX_NEG_POS_RATIO",
+    "DEFAULT_MIN_COUNT",
+    "DEFAULT_MIN_NEG_POS_RATIO",
+    "DEFAULT_MMD_PERMUTATIONS",
+    "DEFAULT_PAD_C",
+    "DEFAULT_PAD_FOLDS",
+    "DEFAULT_PCT_OVER_CONTEXT_THRESHOLD",
+    "DEFAULT_PRIOR_SCALE",
+    "EDA_AUDIT_SCHEMA_VERSION",
+    "HEX_ENTROPY_THRESHOLD",
+    # --- classes / type aliases ---
+    "BaselineScore",
+    "CompetencyResult",
+    "DataAudit",
+    "DistributionShiftResult",
+    "KnnPurityResult",
+    "LexicalAssociationResult",
+    "MmdResult",
+    "ObfuscationProfile",
+    "PadResult",
+    "SplitSummary",
+    "StrTokenizer",
+    "Tokenizer",
+    # --- functions ---
+    "analyze_obfuscation",
+    "audit_dataset",
+    "class_balance",
+    "class_lexical_association",
+    "competency_baselines",
+    "count_invisible_chars",
+    "default_tokenizer",
+    "distribution_shift",
+    "has_high_entropy_alnum_run",
+    "has_rot13_marker",
+    "is_leeted_token",
+    "knn_purity",
+    "leetspeak_counts",
+    "length_quantiles",
+    "maximum_mean_discrepancy",
+    "median_bandwidth",
+    "nfkc_char_delta",
+    "nfkc_changed",
+    "proxy_a_distance",
+    "shannon_entropy",
+    "summarize_split",
+    "weighted_log_odds",
+]