eval-toolkit 1.5.0__tar.gz → 1.7.0__tar.gz

{eval_toolkit-1.5.0 → eval_toolkit-1.7.0}/CHANGELOG.md

@@ -5,6 +5,65 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.7.0] — 2026-06-04 — label-stratified cluster bootstrap (#90, #91)
+
+### Added — `bootstrap.cluster_bootstrap_ci` (label-stratified cluster bootstrap)
+
+`eval_toolkit.bootstrap` ADDITIVE per [ADR 0003](docs/source/adr/0003-stability-contract-and-gate3-methodology.md) — backward-compatible. Closes the gap between the row-level (`bootstrap_ci`) and fold-level (`block_bootstrap_on_folds`) resamplers: **the missing middle — resampling clusters of rows.**
+
+- **`cluster_bootstrap_ci(y_true, y_score, groups, statistic, *, resample_labels=(0, 1), …)`** — percentile CI for a single-condition metric that resamples whole **clusters** (`groups`) with replacement, so the CI is honest under intra-cluster correlation (prompts sharing one attack payload; a document contributing both a poisoned and a benign row). The resample unit is `(label, group)`: by default positive- and negative-clusters are resampled **separately** (never a single-class draw); `resample_labels=(1,)` resamples only positive clusters with negatives held fixed (the payload-cluster convention). Returns a `BootstrapCI` with `method="cluster_percentile"`.
+- **Parallel + reproducible:** built on `parallel_map` + `spawn_seed_sequences`, so `n_jobs` gives bit-for-bit-identical CIs across worker counts (the v0.34.0 reproducibility contract). `n_jobs=-1` uses all cores.
+- Motivation: the analytic row-level AUROC-difference CI (`delong_roc_variance`) assumes row independence and under-covers on clustered eval data (LODO transfer gaps with payload/document/page clusters). Dogfooded by the consumer portfolio's attack-type / carrier / dialect leave-one-group-out bootstraps (Rule of Three).
+- Exported via `from eval_toolkit import cluster_bootstrap_ci`; `__all__` + `_EXPORTS` updated; doctest + n_jobs-reproducibility tests; mypy-strict clean.
+
+### Fixed — stale `seed=` kwarg in 2 bootstrap benchmarks (#91)
+
+`tests/benchmarks/test_kernel_benchmarks.py` passed `seed=` to `bootstrap_ci` / `paired_bootstrap_diff`, but those parameters migrated to `rng=` (SPEC 7) — the two bootstrap benchmark tests `TypeError`'d on the nightly-benchmarks workflow (excluded from PR CI via `-m "not benchmark"`, so it went unnoticed). 2-line `seed=`→`rng=` rename.
+
+## [1.6.0] — 2026-05-29 — Tier-2 `eda` Job-2 + Job-3: shortcut + shift diagnostics (#86, #87)
+
+`eval_toolkit.eda.*` ADDITIVE per [ADR 0003](docs/source/adr/0003-stability-contract-and-gate3-methodology.md) — Tier-2, backward-compatible. Completes the EDA-first analytic layer above the v1.5.0 Job-1 integrity gate: **Job-2** lexical shortcut diagnostics (`lexical_association`, #86) and **Job-3** distribution-shift quantification (`distribution_shift`, #87). Both are dogfooded by the consumer portfolio's pre-modeling OOD-wall prediction (V5 + V9).
+
+### Added — Tier-2 `eda.lexical_association` shortcut diagnostics (Job-2: C1 + C2)
+
+`eval_toolkit.eda` ADDITIVE per [ADR 0003](docs/source/adr/0003-stability-contract-and-gate3-methodology.md) — Tier-2, torch-free (NumPy + scikit-learn). The analytic layer above the Job-1 integrity gate: *"is the label recoverable from a surface shortcut that will not transfer out-of-distribution?"*
+
+- **C1 — `weighted_log_odds` / `class_lexical_association`:** Monroe, Colaresi & Quinn (2008)
+  informative-Dirichlet weighted log-odds-ratio z-scores + smoothed PMI per token, with a
+  `min_count` rare-token floor (the V5 pitfall). Returns a `LexicalAssociationResult`
+  (`top_a` / `top_b` / `to_dict`); tokens ordered by descending z-score.
+- **C2 — `competency_baselines`:** partial-input baselines (length-only, char-n-gram, BoW)
+  fit on a train split and scored on a test split → `CompetencyResult` of per-baseline
+  average-precision vs the positive-prevalence floor (the *shortcut floor*; Feng, Wallace &
+  Boyd-Graber, ACL 2019 caveat documented). Vectorizers fit on train only (no test leakage);
+  empty or single-class train/test raises a diagnostic `ValueError`.
+- Exported via `from eval_toolkit.eda import ...`; 100% line+branch coverage; mypy-strict clean.
+
+### Added — Tier-2 `eda.distribution_shift` covariate-shift quantification (Job-3: E1)
+
+`eval_toolkit.eda` ADDITIVE per [ADR 0003](docs/source/adr/0003-stability-contract-and-gate3-methodology.md) — Tier-2. Public functions take **feature matrices**, so the module is base-install-safe (NumPy + SciPy + scikit-learn); embed text first with `eval_toolkit.embeddings.make_minilm_embedder` (`[embeddings]` extra) or any vectorizer.
+
+- **`proxy_a_distance`:** Ben-David et al. (2006/2010) PAD = `2(1 − 2ε)` from a **linear**
+  domain classifier's **k-fold CV** error, with **fixed strong regularization** (small `C`) —
+  *not* the high-`C` RBF-SVM-on-`predict_proba` recipe that overfits to `PAD ≈ 2` at small `n`.
+  Optional bootstrap CI.
+- **`maximum_mean_discrepancy`:** Gretton et al. (2012) **unbiased** RBF-kernel MMD² U-statistic +
+  **median-heuristic bandwidth** (freezable across folds) + **permutation-test** p-value
+  (Phipson & Smyth 2010, `(1+count)/(B+1)`, never zero). Optional bootstrap CI.
+- **`knn_purity`:** mean fraction of each point's k nearest neighbours sharing its domain label.
+- **`median_bandwidth`** helper + the **`distribution_shift`** orchestrator (all three) +
+  `PadResult` / `MmdResult` / `KnnPurityResult` / `DistributionShiftResult` dataclasses (`to_dict`).
+- Docstrings carry the pre-registered caveats: distance is **necessary-not-sufficient** for OOD
+  collapse (fuse with shortcut-exposure); a non-significant MMD p is not "no shift"; cross-dataset
+  distances are ordinal-only (covariate vs label-semantics conflation). 100% line+branch coverage.
+
+### Fixed
+
+- **Public-API golden `__version__` drift:** the `v1.5.0` release commit bumped
+  `_version.py` to `1.5.0` but did not regenerate `tests/golden/public_api/snapshot.json`,
+  which still pinned `'1.4.0'` — leaving `test_public_api_drift_guard` red on `main` (and on
+  every branch cut from it). Regenerated the golden (the diff is the `__version__` value only).
+
 ## [1.5.0] — 2026-05-29 — Tier-2 `eda` layer (#83) + schema-aware `HFDatasetsLoader` (#85)
 
 Tier-2 / `loaders` ADDITIVE per [ADR 0003](docs/source/adr/0003-stability-contract-and-gate3-methodology.md) — backward-compatible.

{eval_toolkit-1.5.0 → eval_toolkit-1.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: eval-toolkit
-Version: 1.5.0
+Version: 1.7.0
 Summary: Reusable evaluation contracts for binary classification: metrics, bootstrap CIs, calibration, artifacts, and evidence gates.
 Project-URL: Homepage, https://github.com/brandon-behring/eval-toolkit
 Project-URL: Documentation, https://brandon-behring.github.io/eval-toolkit/

{eval_toolkit-1.5.0 → eval_toolkit-1.7.0}/src/eval_toolkit/__init__.py

@@ -129,6 +129,7 @@ _EXPORTS: dict[str, str] = {
     "block_bootstrap_on_folds": "eval_toolkit.bootstrap",
     "bonferroni_correct": "eval_toolkit.bootstrap",
     "bootstrap_ci": "eval_toolkit.bootstrap",
+    "cluster_bootstrap_ci": "eval_toolkit.bootstrap",
     "correct_p_values": "eval_toolkit.bootstrap",
     "cross_validate_metric": "eval_toolkit.bootstrap",
     "cv_clt_ci": "eval_toolkit.bootstrap",

{eval_toolkit-1.5.0 → eval_toolkit-1.7.0}/src/eval_toolkit/_version.py

@@ -2,4 +2,4 @@
 
 __all__ = ["__version__"]
 
-__version__ = "1.5.0"
+__version__ = "1.7.0"

{eval_toolkit-1.5.0 → eval_toolkit-1.7.0}/src/eval_toolkit/bootstrap.py

@@ -52,6 +52,7 @@ __all__ = [
     "block_bootstrap_on_folds",
     "bonferroni_correct",
     "bootstrap_ci",
+    "cluster_bootstrap_ci",
     "correct_p_values",
     "cross_validate_metric",
     "cv_clt_ci",
@@ -1447,6 +1448,205 @@ def block_bootstrap_on_folds(
     )
 
 
+def _label_cluster_units(y_true: np.ndarray, groups: np.ndarray) -> dict[int, list[np.ndarray]]:
+    """Index rows by ``(label, group)``: per label, a list of per-group row-index arrays.
+
+    A group that appears under both labels contributes a **separate** index array to each label's
+    list — the resample unit is ``(label, group)``, so a mixed-label group (e.g. a document with
+    both a poisoned and a benign variant sharing one id) splits into one positive unit and one
+    negative unit, resampled independently. Helper for :func:`cluster_bootstrap_ci`.
+    """
+    units: dict[int, list[np.ndarray]] = {}
+    for lab in np.unique(y_true):
+        lab_rows = np.flatnonzero(y_true == lab)
+        order = np.argsort(groups[lab_rows], kind="stable")
+        sorted_rows = lab_rows[order]
+        sorted_groups = groups[lab_rows][order]
+        # Split at the boundaries between consecutive distinct group ids (post-sort).
+        cut = np.flatnonzero(sorted_groups[1:] != sorted_groups[:-1]) + 1
+        units[int(lab)] = np.split(sorted_rows, cut)
+    return units
+
+
+def _cluster_bootstrap_step(
+    seed_seq: np.random.SeedSequence,
+    *,
+    y_true: np.ndarray,
+    y_score: np.ndarray,
+    units: dict[int, list[np.ndarray]],
+    statistic: MetricFn,
+    resample_labels: tuple[int, ...],
+) -> float | None:
+    """One cluster-resampled draw of ``statistic`` (module-level for parallel_map picklability).
+
+    For each label, its ``(label, group)`` units are resampled with replacement when the label is
+    in ``resample_labels``, else all its rows are held fixed. Returns the statistic on the gathered
+    rows, or ``None`` if the draw is degenerate (statistic raises — e.g. a single-class draw).
+    """
+    rng = np.random.default_rng(seed_seq)
+    parts: list[np.ndarray] = []
+    for lab, group_rows in units.items():
+        if lab in resample_labels:
+            chosen = rng.integers(0, len(group_rows), size=len(group_rows))
+            parts.extend(group_rows[c] for c in chosen)
+        else:
+            parts.extend(group_rows)
+    idx = np.concatenate(parts)
+    try:
+        return float(statistic(y_true[idx], y_score[idx]))
+    except (ValueError, RuntimeError):
+        return None
+
+
+def cluster_bootstrap_ci(
+    y_true: np.ndarray,
+    y_score: np.ndarray,
+    groups: np.ndarray,
+    statistic: MetricFn,
+    *,
+    resample_labels: tuple[int, ...] = (0, 1),
+    n_resamples: int = DEFAULT_N_RESAMPLES,
+    confidence: float = DEFAULT_CONFIDENCE,
+    rng: RNGLike | SeedLike | None = DEFAULT_SEED,
+    n_jobs: int = 1,
+) -> BootstrapCI:
+    r"""Label-stratified **cluster** (group) bootstrap percentile CI for a single-condition metric.
+
+    Resamples whole ``groups`` (clusters) with replacement rather than individual rows, so the CI
+    is honest under intra-cluster correlation (multiple prompts sharing one attack payload; a
+    document contributing both a poisoned and a benign row). The resample unit is ``(label,
+    group)``: by default (``resample_labels=(0, 1)``) positive-clusters and negative-clusters are
+    resampled **separately**, preserving the per-class cluster split so a draw is never
+    single-class. Pass ``resample_labels=(1,)`` to resample only the positive clusters while holding
+    all negatives fixed (the payload-cluster convention).
+
+    Where :func:`bootstrap_ci` resamples **rows** (i.i.d. assumption) and
+    :func:`block_bootstrap_on_folds` resamples **per-fold scalars**, this resamples **clusters of
+    rows** — the missing middle for grouped eval data. The analytic row-level AUROC-difference CI
+    (:func:`delong_roc_variance`) assumes row independence and so under-covers on clustered data,
+    which is the motivation for this estimator.
+
+    Parameters
+    ----------
+    y_true : np.ndarray, shape (n,)
+        Binary labels in ``{0, 1}``.
+    y_score : np.ndarray, shape (n,)
+        Scores aligned with ``y_true``.
+    groups : np.ndarray, shape (n,)
+        Cluster id per row (any sortable dtype — ints or strings).
+    statistic : callable ``(y_true, y_score) -> float``
+        Metric to bootstrap (e.g. ``roc_auc``). Must be **picklable** when ``n_jobs != 1`` (a named
+        top-level function — lambdas / closures are rejected).
+    resample_labels : tuple[int, ...], optional
+        Which label strata are cluster-resampled (default ``(0, 1)`` — both). Labels not listed are
+        held fixed (all their rows always included). Must be non-empty.
+    n_resamples, confidence, rng : standard bootstrap params (``rng`` per SPEC 7).
+    n_jobs : int, optional
+        Parallel workers (default 1 — sequential). ``n_jobs=-1`` uses all cores; ``n_jobs=0`` is
+        rejected. Per-resample seeding via :func:`spawn_seed_sequences` makes the CI **bit-for-bit
+        identical across ``n_jobs``** for a fixed ``rng``. See :ref:`methodology/parallelism`.
+
+    Returns
+    -------
+    BootstrapCI
+        ``method="cluster_percentile"``; ``point_estimate = statistic(y_true, y_score)`` on the full
+        data; ``[alpha/2, 1 - alpha/2]`` percentile CI over the cluster-resampled distribution.
+
+    Raises
+    ------
+    ValueError
+        On shape mismatch, non-1-D input, ``n < 10``, ``confidence`` outside (0, 1), empty
+        ``resample_labels``, a ``resample_labels`` entry absent from ``y_true``, ``n_jobs == 0``, or
+        > 5% degenerate resamples.
+    TypeError
+        If ``n_jobs != 1`` and ``statistic`` is not picklable.
+
+    Examples
+    --------
+    >>> import numpy as np
+    >>> from eval_toolkit.metrics import roc_auc
+    >>> rng = np.random.default_rng(0)
+    >>> groups = np.repeat(np.arange(40), 5)        # 40 clusters of 5 rows
+    >>> y = (groups % 2).astype(int)                # cluster-pure labels
+    >>> s = y + rng.normal(0, 0.3, size=y.size)
+    >>> ci = cluster_bootstrap_ci(y, s, groups, roc_auc, n_resamples=200, rng=0)
+    >>> ci.method
+    'cluster_percentile'
+    >>> bool(0.0 <= ci.ci_low <= ci.ci_high <= 1.0)
+    True
+
+    Notes
+    -----
+    For a *gap* statistic with a fixed offset (e.g. ``Gx = val_auc − test_auc`` with ``val_auc``
+    held fixed), bootstrap the variable term and shift the bounds: ``Gx_low = val_auc − ci.ci_high``,
+    ``Gx_high = val_auc − ci.ci_low``. For a one-sided 95% bound, pass ``confidence=0.90`` and read
+    the relevant bound.
+
+    References
+    ----------
+    .. [1] Efron, B. & Tibshirani, R. "An Introduction to the Bootstrap." Chapman & Hall, 1993.
+           (§8 — bootstrapping stratified / clustered data.)
+    .. [2] Field, C. A. & Welsh, A. H. "Bootstrapping clustered data." JRSS-B 69(3), 2007.
+    """
+    y_true_arr = np.asarray(y_true)
+    y_score_arr = np.asarray(y_score)
+    groups_arr = np.asarray(groups)
+    if not (y_true_arr.shape == y_score_arr.shape == groups_arr.shape):
+        raise ValueError(
+            f"shapes mismatch: y_true {y_true_arr.shape}, y_score {y_score_arr.shape}, "
+            f"groups {groups_arr.shape}"
+        )
+    if y_true_arr.ndim != 1:
+        raise ValueError(f"inputs must be 1-D; got shape {y_true_arr.shape}")
+    n = int(y_true_arr.size)
+    if n < 10:
+        raise ValueError(f"n={n} too small for cluster bootstrap; need ≥ 10")
+    if not 0.0 < confidence < 1.0:
+        raise ValueError(f"confidence must be in (0, 1), got {confidence}")
+    resample_labels = tuple(int(x) for x in resample_labels)
+    if not resample_labels:
+        raise ValueError("resample_labels must be non-empty (nothing would be resampled)")
+    present = {int(v) for v in np.unique(y_true_arr).tolist()}
+    missing = set(resample_labels) - present
+    if missing:
+        raise ValueError(
+            f"resample_labels {sorted(missing)} absent from y_true (present: {sorted(present)})"
+        )
+
+    point = float(statistic(y_true_arr, y_score_arr))
+    units = _label_cluster_units(y_true_arr, groups_arr)
+    seed_seqs = spawn_seed_sequences(rng, n_resamples)
+    step = functools.partial(
+        _cluster_bootstrap_step,
+        y_true=y_true_arr,
+        y_score=y_score_arr,
+        units=units,
+        statistic=statistic,
+        resample_labels=resample_labels,
+    )
+    raw = parallel_map(step, seed_seqs, n_jobs=n_jobs, description="cluster_bootstrap_ci")
+    failures = sum(1 for r in raw if r is None)
+    vals = [r for r in raw if r is not None]
+    if failures > 0.05 * n_resamples:
+        raise ValueError(
+            f"cluster_bootstrap_ci: {failures}/{n_resamples} resamples degenerate "
+            "(statistic raised — e.g. single-class draws); refusing to compute CI on > 5% degenerate"
+        )
+    if not vals:
+        raise ValueError("cluster_bootstrap_ci: no usable resamples")
+    arr = np.asarray(vals, dtype=np.float64)
+    alpha = 1.0 - confidence
+    ci_low, ci_high = np.quantile(arr, [alpha / 2.0, 1.0 - alpha / 2.0])
+    return BootstrapCI(
+        point_estimate=point,
+        ci_low=float(ci_low),
+        ci_high=float(ci_high),
+        confidence=confidence,
+        n_resamples=int(len(vals)),
+        method="cluster_percentile",
+    )
+
+
 def cross_validate_metric(
     y_true: np.ndarray,
     y_score: np.ndarray,

eval_toolkit-1.7.0/src/eval_toolkit/eda/__init__.py

@@ -0,0 +1,144 @@
+"""``eval_toolkit.eda`` — EDA-first dataset integrity gating (Tier-2 surface).
+
+This subpackage is the **Job-1 integrity gate** of an EDA-first research
+program: thin, composable, torch-free per-split profiling + dataset-soundness
+gates, built by reusing the v1.4.0 :mod:`eval_toolkit.leakage`,
+:mod:`~eval_toolkit.text_dedup`, :mod:`~eval_toolkit.claims`, and
+:mod:`~eval_toolkit.artifacts` primitives.
+
+Stability tier
+--------------
+Public access is ``eval_toolkit.eda.*`` — **Tier-2** per ADR 0003. This layer
+is intentionally evolvable and is **not** part of the v2.0-frozen top-level
+:mod:`eval_toolkit` surface; nothing here is added to the package-root
+``_EXPORTS`` / ``__all__``. Import explicitly::
+
+    from eval_toolkit.eda import audit_dataset, DataAudit, SplitSummary
+
+Scope
+-----
+- **Job-1 integrity gate** (``data_audit`` + ``obfuscation``): row counts, class
+  balance, text-length quantiles, dedup / cross-split leakage, obfuscation
+  prevalence.
+- **Job-2 lexical shortcut diagnostics** (``lexical_association``): weighted
+  log-odds + PMI (C1) and partial-input / competency baselines (C2) — torch-free
+  (NumPy + scikit-learn).
+- **Job-3 distribution shift** (``distribution_shift``): proxy-A-distance, MMD
+  (permutation-tested), and kNN purity (E1) — operates on feature matrices, so
+  still base-install-safe (NumPy + SciPy + scikit-learn).
+
+The shift functions take **feature matrices**, not text — embed first with
+:func:`eval_toolkit.embeddings.make_minilm_embedder` (the optional
+``[embeddings]`` extra) or any vectorizer. UMAP / 2-D projections stay caller-side.
+"""
+
+from __future__ import annotations
+
+from eval_toolkit.eda.data_audit import (
+    DEFAULT_MAX_NEG_POS_RATIO,
+    DEFAULT_MIN_NEG_POS_RATIO,
+    DEFAULT_PCT_OVER_CONTEXT_THRESHOLD,
+    EDA_AUDIT_SCHEMA_VERSION,
+    DataAudit,
+    SplitSummary,
+    Tokenizer,
+    audit_dataset,
+    class_balance,
+    length_quantiles,
+    summarize_split,
+)
+from eval_toolkit.eda.distribution_shift import (
+    DEFAULT_KNN_K,
+    DEFAULT_MMD_PERMUTATIONS,
+    DEFAULT_PAD_C,
+    DEFAULT_PAD_FOLDS,
+    DistributionShiftResult,
+    KnnPurityResult,
+    MmdResult,
+    PadResult,
+    distribution_shift,
+    knn_purity,
+    maximum_mean_discrepancy,
+    median_bandwidth,
+    proxy_a_distance,
+)
+from eval_toolkit.eda.lexical_association import (
+    DEFAULT_CHAR_NGRAM_RANGE,
+    DEFAULT_MIN_COUNT,
+    DEFAULT_PRIOR_SCALE,
+    BaselineScore,
+    CompetencyResult,
+    LexicalAssociationResult,
+    StrTokenizer,
+    class_lexical_association,
+    competency_baselines,
+    default_tokenizer,
+    weighted_log_odds,
+)
+from eval_toolkit.eda.obfuscation import (
+    BASE64_ENTROPY_THRESHOLD,
+    HEX_ENTROPY_THRESHOLD,
+    ObfuscationProfile,
+    analyze_obfuscation,
+    count_invisible_chars,
+    has_high_entropy_alnum_run,
+    has_rot13_marker,
+    is_leeted_token,
+    leetspeak_counts,
+    nfkc_changed,
+    nfkc_char_delta,
+    shannon_entropy,
+)
+
+__all__ = [
+    # --- constants ---
+    "BASE64_ENTROPY_THRESHOLD",
+    "DEFAULT_CHAR_NGRAM_RANGE",
+    "DEFAULT_KNN_K",
+    "DEFAULT_MAX_NEG_POS_RATIO",
+    "DEFAULT_MIN_COUNT",
+    "DEFAULT_MIN_NEG_POS_RATIO",
+    "DEFAULT_MMD_PERMUTATIONS",
+    "DEFAULT_PAD_C",
+    "DEFAULT_PAD_FOLDS",
+    "DEFAULT_PCT_OVER_CONTEXT_THRESHOLD",
+    "DEFAULT_PRIOR_SCALE",
+    "EDA_AUDIT_SCHEMA_VERSION",
+    "HEX_ENTROPY_THRESHOLD",
+    # --- classes / type aliases ---
+    "BaselineScore",
+    "CompetencyResult",
+    "DataAudit",
+    "DistributionShiftResult",
+    "KnnPurityResult",
+    "LexicalAssociationResult",
+    "MmdResult",
+    "ObfuscationProfile",
+    "PadResult",
+    "SplitSummary",
+    "StrTokenizer",
+    "Tokenizer",
+    # --- functions ---
+    "analyze_obfuscation",
+    "audit_dataset",
+    "class_balance",
+    "class_lexical_association",
+    "competency_baselines",
+    "count_invisible_chars",
+    "default_tokenizer",
+    "distribution_shift",
+    "has_high_entropy_alnum_run",
+    "has_rot13_marker",
+    "is_leeted_token",
+    "knn_purity",
+    "leetspeak_counts",
+    "length_quantiles",
+    "maximum_mean_discrepancy",
+    "median_bandwidth",
+    "nfkc_char_delta",
+    "nfkc_changed",
+    "proxy_a_distance",
+    "shannon_entropy",
+    "summarize_split",
+    "weighted_log_odds",
+]