eval-toolkit 1.4.0__tar.gz → 1.6.0__tar.gz

{eval_toolkit-1.4.0 → eval_toolkit-1.6.0}/CHANGELOG.md

@@ -5,6 +5,63 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.6.0] — 2026-05-29 — Tier-2 `eda` Job-2 + Job-3: shortcut + shift diagnostics (#86, #87)
+
+`eval_toolkit.eda.*` ADDITIVE per [ADR 0003](docs/source/adr/0003-stability-contract-and-gate3-methodology.md) — Tier-2, backward-compatible. Completes the EDA-first analytic layer above the v1.5.0 Job-1 integrity gate: **Job-2** lexical shortcut diagnostics (`lexical_association`, #86) and **Job-3** distribution-shift quantification (`distribution_shift`, #87). Both are dogfooded by the consumer portfolio's pre-modeling OOD-wall prediction (V5 + V9).
+
+### Added — Tier-2 `eda.lexical_association` shortcut diagnostics (Job-2: C1 + C2)
+
+`eval_toolkit.eda` ADDITIVE per [ADR 0003](docs/source/adr/0003-stability-contract-and-gate3-methodology.md) — Tier-2, torch-free (NumPy + scikit-learn). The analytic layer above the Job-1 integrity gate: *"is the label recoverable from a surface shortcut that will not transfer out-of-distribution?"*
+
+- **C1 — `weighted_log_odds` / `class_lexical_association`:** Monroe, Colaresi & Quinn (2008)
+  informative-Dirichlet weighted log-odds-ratio z-scores + smoothed PMI per token, with a
+  `min_count` rare-token floor (the V5 pitfall). Returns a `LexicalAssociationResult`
+  (`top_a` / `top_b` / `to_dict`); tokens ordered by descending z-score.
+- **C2 — `competency_baselines`:** partial-input baselines (length-only, char-n-gram, BoW)
+  fit on a train split and scored on a test split → `CompetencyResult` of per-baseline
+  average-precision vs the positive-prevalence floor (the *shortcut floor*; Feng, Wallace &
+  Boyd-Graber, ACL 2019 caveat documented). Vectorizers fit on train only (no test leakage);
+  empty or single-class train/test raises a diagnostic `ValueError`.
+- Exported via `from eval_toolkit.eda import ...`; 100% line+branch coverage; mypy-strict clean.
+
+### Added — Tier-2 `eda.distribution_shift` covariate-shift quantification (Job-3: E1)
+
+`eval_toolkit.eda` ADDITIVE per [ADR 0003](docs/source/adr/0003-stability-contract-and-gate3-methodology.md) — Tier-2. Public functions take **feature matrices**, so the module is base-install-safe (NumPy + SciPy + scikit-learn); embed text first with `eval_toolkit.embeddings.make_minilm_embedder` (`[embeddings]` extra) or any vectorizer.
+
+- **`proxy_a_distance`:** Ben-David et al. (2006/2010) PAD = `2(1 − 2ε)` from a **linear**
+  domain classifier's **k-fold CV** error, with **fixed strong regularization** (small `C`) —
+  *not* the high-`C` RBF-SVM-on-`predict_proba` recipe that overfits to `PAD ≈ 2` at small `n`.
+  Optional bootstrap CI.
+- **`maximum_mean_discrepancy`:** Gretton et al. (2012) **unbiased** RBF-kernel MMD² U-statistic +
+  **median-heuristic bandwidth** (freezable across folds) + **permutation-test** p-value
+  (Phipson & Smyth 2010, `(1+count)/(B+1)`, never zero). Optional bootstrap CI.
+- **`knn_purity`:** mean fraction of each point's k nearest neighbours sharing its domain label.
+- **`median_bandwidth`** helper + the **`distribution_shift`** orchestrator (all three) +
+  `PadResult` / `MmdResult` / `KnnPurityResult` / `DistributionShiftResult` dataclasses (`to_dict`).
+- Docstrings carry the pre-registered caveats: distance is **necessary-not-sufficient** for OOD
+  collapse (fuse with shortcut-exposure); a non-significant MMD p is not "no shift"; cross-dataset
+  distances are ordinal-only (covariate vs label-semantics conflation). 100% line+branch coverage.
+
+### Fixed
+
+- **Public-API golden `__version__` drift:** the `v1.5.0` release commit bumped
+  `_version.py` to `1.5.0` but did not regenerate `tests/golden/public_api/snapshot.json`,
+  which still pinned `'1.4.0'` — leaving `test_public_api_drift_guard` red on `main` (and on
+  every branch cut from it). Regenerated the golden (the diff is the `__version__` value only).
+
+## [1.5.0] — 2026-05-29 — Tier-2 `eda` layer (#83) + schema-aware `HFDatasetsLoader` (#85)
+
+Tier-2 / `loaders` ADDITIVE per [ADR 0003](docs/source/adr/0003-stability-contract-and-gate3-methodology.md) — backward-compatible.
+
+- **`eda` Job-1 integrity gate (#83):** `audit_dataset` / `DataAudit` / `SplitSummary` + the
+  `class_balance` / `no_cross_split_leakage` / `context_window_fit` gates + the §B2 obfuscation
+  prevalence module.
+- **schema-aware `HFDatasetsLoader` (#85):** load real-world dataset schemas without column
+  guessing — `feature_cols` + `feature_join` (join multiple columns into one feature; NaN-safe),
+  `label_map` (remap raw labels → int; fail-fast `ValueError` lists unmapped values), `revision`
+  (pin the HF dataset SHA). All new params default to the prior behavior; a missing feature/label
+  column raises `KeyError` listing the observed columns.
+
 ## [1.4.0] — 2026-05-26 — `audit_citation_alignment` Layer 2 + Layer 3 (closes #82); shared `_narrative` helpers (ADR 0007)
 
 Tier-1 ADDITIVE per [ADR 0003](docs/source/adr/0003-stability-contract-and-gate3-methodology.md).

{eval_toolkit-1.4.0 → eval_toolkit-1.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: eval-toolkit
-Version: 1.4.0
+Version: 1.6.0
 Summary: Reusable evaluation contracts for binary classification: metrics, bootstrap CIs, calibration, artifacts, and evidence gates.
 Project-URL: Homepage, https://github.com/brandon-behring/eval-toolkit
 Project-URL: Documentation, https://brandon-behring.github.io/eval-toolkit/
@@ -60,6 +60,9 @@ Requires-Dist: sphinx-autodoc-typehints>=2.0; extra == 'docs'
 Requires-Dist: sphinx-copybutton>=0.5; extra == 'docs'
 Requires-Dist: sphinx-design>=0.6; extra == 'docs'
 Requires-Dist: sphinx>=7.3; extra == 'docs'
+Provides-Extra: eda
+Requires-Dist: matplotlib>=3.8; extra == 'eda'
+Requires-Dist: pandas>=2.0; extra == 'eda'
 Provides-Extra: embeddings
 Requires-Dist: sentence-transformers>=3.0; extra == 'embeddings'
 Provides-Extra: losses

{eval_toolkit-1.4.0 → eval_toolkit-1.6.0}/pyproject.toml

@@ -74,6 +74,14 @@ probes = ["torch>=2.0", "transformers>=4.40"]
 # (granular extras — losses callers should not have to install the larger
 # transformers stack). Shares the torch version pin with [probes].
 losses = ["torch>=2.0"]
+# v1.5.0 (feat/eda-data-audit): eval_toolkit.eda Job-1 integrity-gate layer.
+# Tier-2 surface (ADR 0003) — torch-free by design. pandas powers the
+# DataFrameLoader reuse path; matplotlib is reserved for the EDA layer's
+# future profiling plots. Intentionally NO sentence-transformers / torch:
+# the near-dup / cross-split checks use the lexical TfidfCosineStrategy and
+# token-length quantiles take a caller-supplied tokenizer (no transformers
+# import in this module). NOT folded into [all] / [dev] — opt-in only.
+eda = ["pandas>=2.0", "matplotlib>=3.8"]
 # NO-OP extra kept for backward compatibility (R3 at v0.49.0).
 #
 # jsonschema>=4.21 moved to base deps at v0.16.0; this extra has been a

{eval_toolkit-1.4.0 → eval_toolkit-1.6.0}/src/eval_toolkit/_version.py

@@ -2,4 +2,4 @@
 
 __all__ = ["__version__"]
 
-__version__ = "1.4.0"
+__version__ = "1.6.0"

eval_toolkit-1.6.0/src/eval_toolkit/eda/__init__.py

@@ -0,0 +1,144 @@
+"""``eval_toolkit.eda`` — EDA-first dataset integrity gating (Tier-2 surface).
+
+This subpackage is the **Job-1 integrity gate** of an EDA-first research
+program: thin, composable, torch-free per-split profiling + dataset-soundness
+gates, built by reusing the v1.4.0 :mod:`eval_toolkit.leakage`,
+:mod:`~eval_toolkit.text_dedup`, :mod:`~eval_toolkit.claims`, and
+:mod:`~eval_toolkit.artifacts` primitives.
+
+Stability tier
+--------------
+Public access is ``eval_toolkit.eda.*`` — **Tier-2** per ADR 0003. This layer
+is intentionally evolvable and is **not** part of the v2.0-frozen top-level
+:mod:`eval_toolkit` surface; nothing here is added to the package-root
+``_EXPORTS`` / ``__all__``. Import explicitly::
+
+    from eval_toolkit.eda import audit_dataset, DataAudit, SplitSummary
+
+Scope
+-----
+- **Job-1 integrity gate** (``data_audit`` + ``obfuscation``): row counts, class
+  balance, text-length quantiles, dedup / cross-split leakage, obfuscation
+  prevalence.
+- **Job-2 lexical shortcut diagnostics** (``lexical_association``): weighted
+  log-odds + PMI (C1) and partial-input / competency baselines (C2) — torch-free
+  (NumPy + scikit-learn).
+- **Job-3 distribution shift** (``distribution_shift``): proxy-A-distance, MMD
+  (permutation-tested), and kNN purity (E1) — operates on feature matrices, so
+  still base-install-safe (NumPy + SciPy + scikit-learn).
+
+The shift functions take **feature matrices**, not text — embed first with
+:func:`eval_toolkit.embeddings.make_minilm_embedder` (the optional
+``[embeddings]`` extra) or any vectorizer. UMAP / 2-D projections stay caller-side.
+"""
+
+from __future__ import annotations
+
+from eval_toolkit.eda.data_audit import (
+    DEFAULT_MAX_NEG_POS_RATIO,
+    DEFAULT_MIN_NEG_POS_RATIO,
+    DEFAULT_PCT_OVER_CONTEXT_THRESHOLD,
+    EDA_AUDIT_SCHEMA_VERSION,
+    DataAudit,
+    SplitSummary,
+    Tokenizer,
+    audit_dataset,
+    class_balance,
+    length_quantiles,
+    summarize_split,
+)
+from eval_toolkit.eda.distribution_shift import (
+    DEFAULT_KNN_K,
+    DEFAULT_MMD_PERMUTATIONS,
+    DEFAULT_PAD_C,
+    DEFAULT_PAD_FOLDS,
+    DistributionShiftResult,
+    KnnPurityResult,
+    MmdResult,
+    PadResult,
+    distribution_shift,
+    knn_purity,
+    maximum_mean_discrepancy,
+    median_bandwidth,
+    proxy_a_distance,
+)
+from eval_toolkit.eda.lexical_association import (
+    DEFAULT_CHAR_NGRAM_RANGE,
+    DEFAULT_MIN_COUNT,
+    DEFAULT_PRIOR_SCALE,
+    BaselineScore,
+    CompetencyResult,
+    LexicalAssociationResult,
+    StrTokenizer,
+    class_lexical_association,
+    competency_baselines,
+    default_tokenizer,
+    weighted_log_odds,
+)
+from eval_toolkit.eda.obfuscation import (
+    BASE64_ENTROPY_THRESHOLD,
+    HEX_ENTROPY_THRESHOLD,
+    ObfuscationProfile,
+    analyze_obfuscation,
+    count_invisible_chars,
+    has_high_entropy_alnum_run,
+    has_rot13_marker,
+    is_leeted_token,
+    leetspeak_counts,
+    nfkc_changed,
+    nfkc_char_delta,
+    shannon_entropy,
+)
+
+__all__ = [
+    # --- constants ---
+    "BASE64_ENTROPY_THRESHOLD",
+    "DEFAULT_CHAR_NGRAM_RANGE",
+    "DEFAULT_KNN_K",
+    "DEFAULT_MAX_NEG_POS_RATIO",
+    "DEFAULT_MIN_COUNT",
+    "DEFAULT_MIN_NEG_POS_RATIO",
+    "DEFAULT_MMD_PERMUTATIONS",
+    "DEFAULT_PAD_C",
+    "DEFAULT_PAD_FOLDS",
+    "DEFAULT_PCT_OVER_CONTEXT_THRESHOLD",
+    "DEFAULT_PRIOR_SCALE",
+    "EDA_AUDIT_SCHEMA_VERSION",
+    "HEX_ENTROPY_THRESHOLD",
+    # --- classes / type aliases ---
+    "BaselineScore",
+    "CompetencyResult",
+    "DataAudit",
+    "DistributionShiftResult",
+    "KnnPurityResult",
+    "LexicalAssociationResult",
+    "MmdResult",
+    "ObfuscationProfile",
+    "PadResult",
+    "SplitSummary",
+    "StrTokenizer",
+    "Tokenizer",
+    # --- functions ---
+    "analyze_obfuscation",
+    "audit_dataset",
+    "class_balance",
+    "class_lexical_association",
+    "competency_baselines",
+    "count_invisible_chars",
+    "default_tokenizer",
+    "distribution_shift",
+    "has_high_entropy_alnum_run",
+    "has_rot13_marker",
+    "is_leeted_token",
+    "knn_purity",
+    "leetspeak_counts",
+    "length_quantiles",
+    "maximum_mean_discrepancy",
+    "median_bandwidth",
+    "nfkc_char_delta",
+    "nfkc_changed",
+    "proxy_a_distance",
+    "shannon_entropy",
+    "summarize_split",
+    "weighted_log_odds",
+]