shap-recommender 0.1.0__tar.gz

shap_recommender-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Kaylee
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

shap_recommender-0.1.0/PKG-INFO

@@ -0,0 +1,120 @@
+Metadata-Version: 2.4
+Name: shap-recommender
+Version: 0.1.0
+Summary: Exclusion / non-linearity / interaction recommendations from saved SHAP attribution files, and application of them to a design matrix.
+Author: Kaylee
+License-Expression: MIT
+Project-URL: Homepage, https://pypi.org/project/shap-recommender/
+Keywords: shap,feature-selection,interaction-detection,interpretability,machine-learning
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Intended Audience :: Science/Research
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.23
+Requires-Dist: pandas>=1.5
+Requires-Dist: scipy>=1.9
+Requires-Dist: statsmodels>=0.13
+Requires-Dist: patsy>=0.5
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Dynamic: license-file
+
+# shap-recommender
+
+Exclusion / non-linearity / interaction recommendations from saved SHAP
+attribution files, and application of them to a design matrix.
+
+One shared idea runs through two of the three rules. A feature's own
+contribution is represented flexibly (indicator columns when it takes few
+values, a restricted cubic spline when it is continuous), and a model built
+on that flexible basis is compared against a straight line in the feature.
+That comparison answers two different questions:
+
+- **non-linearity** -- does the attribution deviate from a linear function
+  of the feature? This directly tests the linear-trend assumption, rather
+  than relying on a raw correlation coefficient, which conflates "no effect"
+  with "non-linear effect".
+- **interaction** -- does the attribution vary among subjects who share the
+  same feature value? Under additivity the attribution is a deterministic
+  function of the feature, so residual dispersion implies effect
+  modification.
+
+Stratification for the interaction screen is always applied to the observed
+feature value at a pre-specified cut point, never to the attribution itself
+-- splitting on the attribution would condition on the candidate modifier.
+Within strata, the contrast `E[phi_y | y=1] - E[phi_y | y=0]` is compared,
+which (unlike the marginal attribution distribution) is invariant to
+stratum composition under additivity.
+
+## Install
+
+```bash
+pip install shap-recommender
+```
+
+## Expected input files
+
+For each dataset "tag" you want to load, `Recommender.load(tag)` (and the
+CLI's `--tags`) expects two tab-separated files in `res_dir`:
+
+- `shap_values_<tag>.tsv` -- SHAP values, one row per subject, one column
+  per feature, first column = row index.
+- `sel_data_<tag>.tsv` -- the corresponding feature values (design matrix),
+  same row index.
+
+## Command-line use
+
+```bash
+shap-recommender \
+    --res-dir ./shap_results \
+    --tags cohort_a cohort_b \
+    --nonlinear-candidates age bmi creatinine \
+    --out ./recommendations
+```
+
+This writes `exclusion_tests.tsv`, `exclusion_sensitivity.tsv`,
+`nonlinear_tests.tsv`, `interaction_tests.tsv`, `attribution_patterns.tsv`,
+and `recommendations.json` to `--out`. Run `shap-recommender --help` for all
+options (thresholds, bootstrap count, spline degrees of freedom, a
+`--cutpoints` JSON file for pre-specified stratification cut points, etc).
+
+## Library use
+
+```python
+from shap_recommender import Recommender
+
+rec = Recommender(res_dir="./shap_results")
+recommendations = rec.generate(
+    candidates_nonlinear=["age", "bmi", "creatinine"],
+    tags=["cohort_a", "cohort_b"],
+    out="./recommendations",
+)
+
+# apply the recommendations to a design matrix
+X_train_adj, X_test_adj = Recommender.apply(
+    X_train, X_test, recommendations, variant="all",
+)
+```
+
+`Recommender.apply(..., variant=...)` accepts `"baseline"`, `"exclusion"`,
+`"nonlinear"`, `"interaction"`, or `"all"`, so each rule's effect on
+downstream model performance can be evaluated separately.
+
+## Validating the interaction rule
+
+`Recommender.null_sim()` runs a small simulation under an additive null
+(no true interaction with the feature being tested) and reports the type-I
+error rate of the within-stratum contrast used by `stratified_screen`,
+compared against naively partitioning on the attribution itself:
+
+```python
+from shap_recommender import Recommender
+Recommender(res_dir=".").null_sim()
+```
+
+## License
+
+MIT

shap_recommender-0.1.0/README.md

@@ -0,0 +1,96 @@
+# shap-recommender
+
+Exclusion / non-linearity / interaction recommendations from saved SHAP
+attribution files, and application of them to a design matrix.
+
+One shared idea runs through two of the three rules. A feature's own
+contribution is represented flexibly (indicator columns when it takes few
+values, a restricted cubic spline when it is continuous), and a model built
+on that flexible basis is compared against a straight line in the feature.
+That comparison answers two different questions:
+
+- **non-linearity** -- does the attribution deviate from a linear function
+  of the feature? This directly tests the linear-trend assumption, rather
+  than relying on a raw correlation coefficient, which conflates "no effect"
+  with "non-linear effect".
+- **interaction** -- does the attribution vary among subjects who share the
+  same feature value? Under additivity the attribution is a deterministic
+  function of the feature, so residual dispersion implies effect
+  modification.
+
+Stratification for the interaction screen is always applied to the observed
+feature value at a pre-specified cut point, never to the attribution itself
+-- splitting on the attribution would condition on the candidate modifier.
+Within strata, the contrast `E[phi_y | y=1] - E[phi_y | y=0]` is compared,
+which (unlike the marginal attribution distribution) is invariant to
+stratum composition under additivity.
+
+## Install
+
+```bash
+pip install shap-recommender
+```
+
+## Expected input files
+
+For each dataset "tag" you want to load, `Recommender.load(tag)` (and the
+CLI's `--tags`) expects two tab-separated files in `res_dir`:
+
+- `shap_values_<tag>.tsv` -- SHAP values, one row per subject, one column
+  per feature, first column = row index.
+- `sel_data_<tag>.tsv` -- the corresponding feature values (design matrix),
+  same row index.
+
+## Command-line use
+
+```bash
+shap-recommender \
+    --res-dir ./shap_results \
+    --tags cohort_a cohort_b \
+    --nonlinear-candidates age bmi creatinine \
+    --out ./recommendations
+```
+
+This writes `exclusion_tests.tsv`, `exclusion_sensitivity.tsv`,
+`nonlinear_tests.tsv`, `interaction_tests.tsv`, `attribution_patterns.tsv`,
+and `recommendations.json` to `--out`. Run `shap-recommender --help` for all
+options (thresholds, bootstrap count, spline degrees of freedom, a
+`--cutpoints` JSON file for pre-specified stratification cut points, etc).
+
+## Library use
+
+```python
+from shap_recommender import Recommender
+
+rec = Recommender(res_dir="./shap_results")
+recommendations = rec.generate(
+    candidates_nonlinear=["age", "bmi", "creatinine"],
+    tags=["cohort_a", "cohort_b"],
+    out="./recommendations",
+)
+
+# apply the recommendations to a design matrix
+X_train_adj, X_test_adj = Recommender.apply(
+    X_train, X_test, recommendations, variant="all",
+)
+```
+
+`Recommender.apply(..., variant=...)` accepts `"baseline"`, `"exclusion"`,
+`"nonlinear"`, `"interaction"`, or `"all"`, so each rule's effect on
+downstream model performance can be evaluated separately.
+
+## Validating the interaction rule
+
+`Recommender.null_sim()` runs a small simulation under an additive null
+(no true interaction with the feature being tested) and reports the type-I
+error rate of the within-stratum contrast used by `stratified_screen`,
+compared against naively partitioning on the attribution itself:
+
+```python
+from shap_recommender import Recommender
+Recommender(res_dir=".").null_sim()
+```
+
+## License
+
+MIT

shap_recommender-0.1.0/pyproject.toml

@@ -0,0 +1,41 @@
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "shap-recommender"
+version = "0.1.0"
+description = "Exclusion / non-linearity / interaction recommendations from saved SHAP attribution files, and application of them to a design matrix."
+readme = "README.md"
+requires-python = ">=3.9"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [
+    { name = "Kaylee" },
+]
+keywords = ["shap", "feature-selection", "interaction-detection", "interpretability", "machine-learning"]
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "Operating System :: OS Independent",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+    "Intended Audience :: Science/Research",
+]
+dependencies = [
+    "numpy>=1.23",
+    "pandas>=1.5",
+    "scipy>=1.9",
+    "statsmodels>=0.13",
+    "patsy>=0.5",
+]
+
+[project.optional-dependencies]
+dev = ["pytest"]
+
+[project.scripts]
+shap-recommender = "shap_recommender.cli:main"
+
+[project.urls]
+Homepage = "https://pypi.org/project/shap-recommender/"
+
+[tool.setuptools.packages.find]
+where = ["src"]

shap_recommender-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

shap_recommender-0.1.0/src/shap_recommender/__init__.py

@@ -0,0 +1,13 @@
+"""shap_recommender: exclusion / non-linearity / interaction recommendations
+from saved SHAP attribution files, and application of them to a design
+matrix.
+
+See :class:`shap_recommender.core.Recommender` for the full API, and run
+``shap-recommender --help`` for the command-line interface.
+"""
+
+from .core import Recommender
+
+__version__ = "0.1.0"
+
+__all__ = ["Recommender", "__version__"]

shap_recommender-0.1.0/src/shap_recommender/cli.py

@@ -0,0 +1,84 @@
+"""
+Command-line entry point for :meth:`shap_recommender.core.Recommender.generate`.
+
+Usage
+-----
+    shap-recommender \\
+        --res-dir ./shap_results \\
+        --tags cohort_a cohort_b \\
+        --nonlinear-candidates age bmi creatinine \\
+        --out ./recommendations
+
+Expects, in --res-dir, a `shap_values_<tag>.tsv` and a `sel_data_<tag>.tsv`
+for every tag passed to --tags (see Recommender.load).
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+
+from .core import Recommender
+
+
+def build_parser() -> argparse.ArgumentParser:
+    p = argparse.ArgumentParser(
+        prog="shap-recommender",
+        description="Exclusion / non-linearity / interaction recommendations "
+                    "from saved SHAP attribution files.",
+    )
+    p.add_argument("--res-dir", required=True,
+                   help="Directory containing shap_values_<tag>.tsv and "
+                        "sel_data_<tag>.tsv files.")
+    p.add_argument("--tags", required=True, nargs="+",
+                   help="One or more dataset tags to load and pool.")
+    p.add_argument("--nonlinear-candidates", required=True, nargs="+",
+                   help="Feature names to test for non-linearity.")
+    p.add_argument("--candidates", default=None, nargs="+",
+                   help="Feature names eligible as interaction partners "
+                        "(default: all columns).")
+    p.add_argument("--cutpoints", default=None,
+                   help="Optional path to a JSON file of {feature: cut_point} "
+                        "pre-specified stratification cut points.")
+    p.add_argument("--out", default="recommendations",
+                   help="Output directory for test tables and recommendations.json.")
+    p.add_argument("--frac", type=float, default=0.05)
+    p.add_argument("--alpha", type=float, default=0.05)
+    p.add_argument("--n-boot", type=int, default=500)
+    p.add_argument("--min-delta-r2", type=float, default=0.01)
+    p.add_argument("--min-cell", type=int, default=100)
+    p.add_argument("--n-bins", type=int, default=10)
+    p.add_argument("--df-spline", type=int, default=4)
+    p.add_argument("--residual-thresh", type=float, default=0.05)
+    p.add_argument("--min-effect-ratio", type=float, default=0.10)
+    p.add_argument("--seed", type=int, default=1)
+    return p
+
+
+def main(argv=None) -> int:
+    args = build_parser().parse_args(argv)
+
+    cutpoints = None
+    if args.cutpoints:
+        with open(args.cutpoints) as fh:
+            cutpoints = json.load(fh)
+
+    rec = Recommender(
+        res_dir=args.res_dir, frac=args.frac, alpha=args.alpha,
+        n_boot=args.n_boot, min_delta_r2=args.min_delta_r2,
+        min_cell=args.min_cell, n_bins=args.n_bins, df_spline=args.df_spline,
+        residual_thresh=args.residual_thresh,
+        min_effect_ratio=args.min_effect_ratio, seed=args.seed,
+    )
+    rec.generate(
+        candidates_nonlinear=args.nonlinear_candidates,
+        tags=args.tags,
+        cutpoints=cutpoints,
+        candidates=args.candidates,
+        out=args.out,
+    )
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

shap_recommender-0.1.0/src/shap_recommender/core.py

@@ -0,0 +1,446 @@
+"""
+core.py -- exclusion / non-linearity / interaction recommendations from saved
+SHAP attribution files, and application of them to a design matrix.
+
+One shared idea runs through two of the three rules. ``_basis(x)`` represents
+a feature's own contribution flexibly: indicator columns when x takes few
+values, a restricted cubic spline when it is continuous. Comparing a model
+built on that basis against a straight line in x answers both questions:
+
+non-linearity
+    Does phi_x deviate from a linear function of x? This is a direct test of
+    the linear-trend assumption, rather than relying on a raw correlation
+    coefficient, which conflates "no effect" with "non-linear effect" --
+    a monotone non-linear relationship can have a small |r| while a
+    genuinely linear-but-noisy one can have a larger |r|.
+
+interaction
+    Does phi_x vary among patients who share a value of x? Under additivity
+    phi_x = h(x) - E[h(x)] is a deterministic function of x, so residual
+    dispersion implies effect modification; none implies a purely additive
+    (main-effects-only) relationship.
+
+Stratification is always applied to the observed feature value at a
+pre-specified cut point, never to the attribution itself: phi_x is a function
+of the whole feature vector, so splitting on it would condition on the
+candidate modifier itself. Within strata the contrast
+E[phi_y | y=1] - E[phi_y | y=0] is compared, which is invariant to stratum
+composition under additivity, unlike the marginal attribution distribution.
+"""
+
+import json
+import os
+
+import numpy as np
+import pandas as pd
+from scipy.stats import f as fdist, norm
+from statsmodels.stats.multitest import multipletests
+
+__all__ = ["Recommender"]
+
+
+class Recommender:
+    """
+    Parameters
+    ----------
+    res_dir : str
+        Directory containing ``shap_values_<tag>.tsv`` and
+        ``sel_data_<tag>.tsv`` files (see :meth:`load`).
+    frac : float
+        Exclusion threshold, as a fraction of the across-feature SD of mean
+        |attribution| (rule 1).
+    alpha : float
+        FDR-adjusted significance threshold used by the non-linearity and
+        interaction rules.
+    n_boot : int
+        Bootstrap resamples used for confidence intervals.
+    min_delta_r2 : float
+        Minimum incremental R^2 (flexible basis over a straight line) for a
+        feature to be flagged as non-linear.
+    min_cell : int
+        Minimum stratum size required to run the interaction screen.
+    n_bins : int
+        A feature with at most this many unique values is treated as
+        discrete (indicator basis); above it, a spline basis is used.
+    df_spline : int
+        Degrees of freedom for the restricted cubic spline basis.
+    residual_thresh : float
+        Residual-fraction threshold below which a feature's attribution is
+        treated as a deterministic (purely additive) function of the
+        feature itself.
+    min_effect_ratio : float
+        Minimum relative effect size for an interaction pair to be flagged.
+    seed : int
+        Seed for the bootstrap random number generator.
+    """
+
+    def __init__(self, res_dir, frac=0.05, alpha=0.05,
+                 n_boot=500, min_delta_r2=0.01, min_cell=100, n_bins=10,
+                 df_spline=4, residual_thresh=0.05, min_effect_ratio=0.10, seed=1):
+        self.res, self.frac, self.alpha = res_dir, frac, alpha
+        self.n_boot, self.min_delta_r2 = n_boot, min_delta_r2
+        self.min_cell, self.n_bins, self.df_spline = min_cell, n_bins, df_spline
+        self.residual_thresh, self.min_effect_ratio = residual_thresh, min_effect_ratio
+        self.rng = np.random.default_rng(seed)
+
+    def load(self, tag):
+        """Load a (shap_values, design_matrix) pair for `tag` from `res_dir`,
+        expecting files named ``shap_values_<tag>.tsv`` and
+        ``sel_data_<tag>.tsv`` (tab-separated, first column = row index)."""
+        s = pd.read_csv(f'{self.res}/shap_values_{tag}.tsv', sep='\t', index_col=0)
+        x = pd.read_csv(f'{self.res}/sel_data_{tag}.tsv', sep='\t', index_col=0)
+        return s, x.loc[s.index]
+
+    # ------------------------------------------------------ shared basis --
+    def _basis(self, x):
+        """Indicator columns for few-valued x, restricted cubic spline for
+        continuous x. Binning a continuous variable and using bin means
+        would leave h(x) varying inside each bin and get counted as
+        residual dispersion, so a spline basis is used instead once a
+        feature has more distinct values than `n_bins`."""
+        x = np.asarray(x, float)
+        u = np.unique(x[np.isfinite(x)])
+        if u.size < 2:
+            return np.empty((x.size, 0))
+        if u.size <= self.n_bins:
+            return np.column_stack([(x == v).astype(float) for v in u[1:]])
+        from patsy import dmatrix
+        return np.asarray(dmatrix(f'0 + cr(a, df={self.df_spline})', {'a': x},
+                                  return_type='dataframe'))
+
+    def _fit_r2(self, A, phi, tss):
+        beta = np.linalg.lstsq(A, phi, rcond=None)[0]
+        return 1 - float(((phi - A @ beta) ** 2).sum()) / tss
+
+    # ---------------------------------------------------- rule 1: exclude --
+    def exclusion(self, shap_df, frac=None, out=None):
+        """Exclude a feature when the upper bound of the 95% CI of its mean
+        absolute attribution falls below `frac` x SD(per-feature mean |attribution|)
+        -- the SD taken across features, not of the whole matrix."""
+        frac = self.frac if frac is None else frac
+        ma = shap_df.abs().mean(axis=0)
+        thr = frac * float(ma.std(ddof=1))
+        rows = []
+        for c in shap_df.columns:
+            v = shap_df[c].abs().to_numpy(float)
+            b = np.array([np.nanmean(v[self.rng.integers(0, v.size, v.size)])
+                          for _ in range(self.n_boot)])
+            hi = float(np.percentile(b, 97.5))
+            rows.append(dict(feature=c, mean_abs=float(ma[c]), ci_high=hi,
+                             threshold=thr, excluded=bool(hi < thr)))
+        t = pd.DataFrame(rows).sort_values('mean_abs')
+        if out:
+            t.to_csv(out, sep='\t', index=False)
+        excl = t.loc[t.excluded, 'feature'].tolist()
+        print(f'[exclusion] threshold={thr:.4g}; {len(excl)}/{len(t)} excluded')
+        return excl, t
+
+    def exclusion_sensitivity(self, shap_df, fracs=(0.01, 0.05, 0.10, 0.20), out=None):
+        """Re-run :meth:`exclusion` at several thresholds, to show how the
+        excluded-feature set changes with `frac`."""
+        rows = [dict(frac=f, n_excluded=len(e), features='; '.join(sorted(e))[:1000])
+                for f, e in ((f, self.exclusion(shap_df, frac=f)[0]) for f in fracs)]
+        t = pd.DataFrame(rows)
+        if out:
+            t.to_csv(out, sep='\t', index=False)
+        return t
+
+    # -------------------------------------------------- rule 2: nonlinear --
+    def nonlinear(self, shap_df, sel_df, candidates, out=None):
+        """Incremental R-squared of the flexible basis over a straight line.
+        Ordinal features are eligible and use the indicator basis, which is
+        the standard test of a linear-trend assumption for a variable whose
+        risk is not monotone in its coded levels."""
+        rows = []
+        for c in candidates:
+            if c not in shap_df.columns or c not in sel_df.columns:
+                continue
+            x, phi = sel_df[c].to_numpy(float), shap_df[c].to_numpy(float)
+            ok = np.isfinite(x) & np.isfinite(phi)
+            x, phi = x[ok], phi[ok]
+            tss = float(((phi - phi.mean()) ** 2).sum())
+            B = self._basis(x)
+            if x.size < 50 or B.shape[1] < 2 or tss == 0:
+                continue
+            r2_lin = self._fit_r2(np.c_[np.ones(x.size), x], phi, tss)
+            r2_flex = self._fit_r2(np.c_[np.ones(x.size), B], phi, tss)
+            k = B.shape[1]
+            den = (1 - r2_flex) / max(x.size - k - 1, 1)
+            F = ((r2_flex - r2_lin) / max(k - 1, 1) / den) if den > 0 else np.inf
+            rows.append(dict(feature=c, n=int(x.size), n_levels=int(np.unique(x).size),
+                             basis='indicator' if np.unique(x).size <= self.n_bins else 'spline',
+                             r=float(np.corrcoef(x, phi)[0, 1]),
+                             delta_r2=float(r2_flex - r2_lin),
+                             p=float(fdist.sf(F, max(k - 1, 1), max(x.size - k - 1, 1)))))
+        t = pd.DataFrame(rows)
+        if not len(t):
+            return [], t
+        t['p_fdr'] = multipletests(t['p'], method='fdr_bh')[1]
+        t['selected'] = (t.p_fdr < self.alpha) & (t.delta_r2 > self.min_delta_r2)
+        if out:
+            t.to_csv(out, sep='\t', index=False)
+        sel = t.loc[t.selected, 'feature'].tolist()
+        print(f'[non-linearity] {len(sel)}/{len(t)} flagged: {sel}')
+        return sel, t
+
+    # ------------------------------------------------ rule 3: interaction --
+    def residual_fraction(self, x, phi):
+        """Share of Var(phi_x) not explained by x itself. ~0 implies a
+        purely additive (main-effects-only) relationship."""
+        x, phi = np.asarray(x, float), np.asarray(phi, float)
+        tss = float(phi.var())
+        if tss == 0:
+            return 0.0
+        if np.unique(x[np.isfinite(x)]).size < 2:
+            return 1.0                                   # x constant: undefined, treat as flagged
+        A = np.c_[np.ones(x.size), self._basis(x)]
+        return float(np.var(phi - A @ np.linalg.lstsq(A, phi, rcond=None)[0]) / tss)
+
+    def _rf_ci(self, x, phi, b=200):
+        v = np.array([self.residual_fraction(x[i], phi[i]) for i in
+                      (self.rng.integers(0, len(x), len(x)) for _ in range(b))])
+        v = v[np.isfinite(v)]
+        return float(np.percentile(v, 2.5)), float(np.percentile(v, 97.5))
+
+    def _cutpoint(self, x, var, cutpoints):
+        """Continuous variables need a pre-specified cut point: a threshold
+        read off the attribution plot is not defensible, since the zero
+        crossing of phi_x depends on the cohort's covariate distribution and
+        is not a fixed property of the variable itself."""
+        u = np.unique(np.asarray(x, float)[np.isfinite(x)])
+        if cutpoints and var in cutpoints:
+            return float(cutpoints[var]), 'pre-specified'
+        if u.size == 2:
+            return float(u[0]), 'binary'
+        if u.size <= self.n_bins:
+            return float(np.median(u)), 'median of observed levels'
+        return None, 'continuous: no pre-specified cut point, skipped'
+
+    @staticmethod
+    def _slopes(Y, PHI):
+        """Column-wise OLS slope of each attribution on its own feature, with
+        a heteroskedasticity-robust variance. For a binary feature this
+        equals E[phi|y=1] - E[phi|y=0], the within-stratum contrast."""
+        Yc, Pc = Y - Y.mean(0), PHI - PHI.mean(0)
+        den = (Yc ** 2).sum(0)
+        safe = np.where(den > 0, den, 1.0)
+        beta = np.where(den > 0, (Yc * Pc).sum(0) / safe, np.nan)
+        resid = Pc - Yc * beta
+        var = np.where(den > 0, ((Yc ** 2) * (resid ** 2)).sum(0) / safe ** 2, np.nan)
+        return beta, var
+
+    def stratified_screen(self, x, cut, shap_df, X_df, strat_var, cols):
+        """Split on the observed feature value, compare within-stratum
+        contrasts rather than the marginal attribution distribution, since a
+        common reference point does not repair the marginal comparison when
+        the two strata have different prevalences of the partner feature."""
+        x = np.asarray(x, float)
+        m1 = x <= cut
+        m2 = ~m1
+        if m1.sum() < self.min_cell or m2.sum() < self.min_cell:
+            return pd.DataFrame()
+        d1, v1 = self._slopes(X_df.loc[m1, cols].to_numpy(float),
+                              shap_df.loc[m1, cols].to_numpy(float))
+        d2, v2 = self._slopes(X_df.loc[m2, cols].to_numpy(float),
+                              shap_df.loc[m2, cols].to_numpy(float))
+        diff, se = d1 - d2, np.sqrt(v1 + v2)
+        # when the attribution is a deterministic function of the feature the
+        # residual variance is zero and floating-point noise would read as
+        # infinite significance; require the SE to clear a relative floor
+        floor = 1e-8 * np.maximum(np.abs(d1) + np.abs(d2), 1e-12)
+        with np.errstate(divide='ignore', invalid='ignore'):
+            z = np.where(se > floor, diff / se, 0.0)
+            scale = (np.abs(d1) + np.abs(d2)) / 2
+            rel = np.where(scale > 0, np.abs(diff) / scale, 0.0)
+        ok = np.isfinite(diff) & np.isfinite(se)
+        return pd.DataFrame(dict(strat=strat_var, partner=np.asarray(cols)[ok],
+                                 cut_point=cut, n_group1=int(m1.sum()),
+                                 n_group2=int(m2.sum()),
+                                 contrast_group1=d1[ok], contrast_group2=d2[ok],
+                                 effect_size=diff[ok],
+                                 ci_low=diff[ok] - 1.96 * se[ok],
+                                 ci_high=diff[ok] + 1.96 * se[ok],
+                                 rel_effect=rel[ok],
+                                 p_raw=np.minimum(2 * norm.sf(np.abs(z))[ok], 1.0)))
+
+    def interaction(self, shap_df, X_df, cutpoints=None, candidates=None,
+                    out=None, diag_out=None):
+        """Screen all eligible feature pairs for effect modification. Returns
+        a `{stratifying_variable: [partner, ...]}` spec, the full test table,
+        and a per-variable diagnostic table (residual fraction and pattern)."""
+        X_df = X_df.loc[shap_df.index]
+        cand = [c for c in (candidates or X_df.columns) if c in X_df.columns]
+        diag, tabs = [], []
+        for v in [c for c in shap_df.columns if c in X_df.columns]:
+            x, phi = X_df[v].to_numpy(float), shap_df[v].to_numpy(float)
+            rf = self.residual_fraction(x, phi)
+            lo, hi = self._rf_ci(x, phi)
+            cut, basis = self._cutpoint(x, v, cutpoints)
+            pattern = ('A' if np.unique(x).size < 2 else
+                       'B' if rf < self.residual_thresh else 'C')
+            row = dict(variable=v, pattern=pattern, residual_fraction=rf,
+                       rf_ci_low=lo, rf_ci_high=hi, threshold=self.residual_thresh,
+                       cut_point=cut, cut_basis=basis, screened=False)
+            if lo > self.residual_thresh and cut is not None:
+                cols = [c for c in cand if c != v and c in shap_df.columns
+                        and X_df[c].nunique(dropna=True) > 1]
+                t = self.stratified_screen(x, cut, shap_df, X_df, v, cols)
+                if len(t):
+                    tabs.append(t)
+                    row['screened'] = True
+            diag.append(row)
+
+        diag_df = pd.DataFrame(diag).sort_values('residual_fraction', ascending=False)
+        if diag_out:
+            diag_df.to_csv(diag_out, sep='\t', index=False)
+        print(f"[interaction] {len(diag_df)} variables: "
+              f"{int((diag_df.pattern == 'B').sum())} purely additive (not "
+              f"stratified), {int(diag_df.screened.sum())} screened")
+        if not tabs:
+            return {}, pd.DataFrame(), diag_df
+
+        t = pd.concat(tabs, ignore_index=True)
+        t['n_tests_total'] = len(t)
+        t['p_fdr'] = multipletests(t['p_raw'], method='fdr_bh')[1]
+        t['selected'] = (t.p_fdr < self.alpha) & (t.rel_effect >= self.min_effect_ratio)
+        t['pair'] = ['||'.join(sorted([a, b])) for a, b in zip(t.strat, t.partner)]
+        if out:
+            t.to_csv(out, sep='\t', index=False)
+        sel = t[t.selected].sort_values('p_fdr').drop_duplicates('pair')
+        spec = {}
+        for _, r in sel.iterrows():
+            spec.setdefault(r.strat, []).append(r.partner)
+        print(f'[interaction] {len(t)} tests, {len(sel)} unique pairs after FDR')
+        return spec, t, diag_df
+
+    # ------------------------------------------------------------ generate --
+    def generate(self, candidates_nonlinear, tags, cutpoints=None, candidates=None,
+                 out='recs'):
+        """Run all three rules and write recommendations to `out`.
+
+        Parameters
+        ----------
+        candidates_nonlinear : list of str
+            Features to test for non-linearity.
+        tags : list of str
+            Dataset tags to load (via :meth:`load`) and concatenate before
+            screening -- e.g. separate tags for different subgroups or CV
+            folds that should be pooled for this analysis.
+        cutpoints : dict, optional
+            Pre-specified stratification cut points, keyed by variable name.
+        candidates : list of str, optional
+            Features eligible as interaction partners (default: all columns).
+        out : str
+            Output directory for the test tables and `recommendations.json`.
+        """
+        os.makedirs(out, exist_ok=True)
+        loaded = [self.load(tag) for tag in tags]
+        s_all = pd.concat([s for s, _ in loaded])
+        x_all = pd.concat([x for _, x in loaded])
+        s_uni = s_all[~s_all.index.duplicated()]
+
+        excl, _ = self.exclusion(s_uni, out=f'{out}/exclusion_tests.tsv')
+        self.exclusion_sensitivity(s_uni, out=f'{out}/exclusion_sensitivity.tsv')
+        nl, _ = self.nonlinear(s_all, x_all, candidates_nonlinear,
+                               out=f'{out}/nonlinear_tests.tsv')
+        spec, _, _ = self.interaction(s_all, x_all, cutpoints=cutpoints,
+                                      candidates=candidates,
+                                      out=f'{out}/interaction_tests.tsv',
+                                      diag_out=f'{out}/attribution_patterns.tsv')
+        rec = dict(exclusion=excl, nonlinear=nl, interaction=spec)
+        with open(f'{out}/recommendations.json', 'w') as fh:
+            json.dump(rec, fh, indent=2)
+        print(f'exclusion {len(excl)}, nonlinear {len(nl)}, '
+              f'interaction {sum(len(v) for v in spec.values())}')
+        return rec
+
+    # --------------------------------------------------------------- apply --
+    @staticmethod
+    def _cols(name, frame, groups):
+        """SHAP columns carry the parent one-hot name; the model matrix
+        carries the dummy columns."""
+        return [name] if name in frame.columns else \
+            [c for c in groups.get(name, []) if c in frame.columns]
+
+    @staticmethod
+    def apply(X_train, X_test, rec, variant='all', groups=None,
+              hierarchy='drop', log=None):
+        """variant: baseline | exclusion | nonlinear | interaction | all.
+        hierarchy='drop' removes an interaction whose main effect was
+        excluded; 'keep' restores the main effect instead."""
+        tr, te = X_train.copy(), X_test.copy()
+        groups = groups or {}
+        excl = rec['exclusion'] if variant in ('exclusion', 'all') else []
+        nl = rec['nonlinear'] if variant in ('nonlinear', 'all') else []
+        spec = rec['interaction'] if variant in ('interaction', 'all') else {}
+
+        if hierarchy == 'keep':
+            needed = set(spec) | {p for v in spec.values() for p in v}
+            excl = [c for c in excl if c not in needed]
+        drop = [c for f in excl for c in Recommender._cols(f, tr, groups)]
+        tr, te = tr.drop(columns=drop), te.drop(columns=drop)
+
+        for f in nl:
+            if f in tr.columns:
+                m = float(tr[f].mean())          # training mean, both frames
+                tr[f], te[f] = tr[f] - m, te[f] - m
+                tr[f + '_quad'], te[f + '_quad'] = tr[f] ** 2, te[f] ** 2
+
+        made, skipped = set(), []
+        for a, partners in spec.items():
+            for b in partners:
+                key = tuple(sorted((a, b)))
+                if key in made or a == b:
+                    continue
+                ca, cb = (Recommender._cols(a, tr, groups),
+                          Recommender._cols(b, tr, groups))
+                if not ca or not cb:
+                    skipped.append(dict(strat=a, partner=b, reason='main effect excluded'))
+                    continue
+                made.add(key)
+                for u in ca:
+                    for v in cb:
+                        if u != v:
+                            tr[f'{u}__x__{v}'] = tr[u] * tr[v]
+                            te[f'{u}__x__{v}'] = te[u] * te[v]
+        if skipped:
+            print(f'{len(skipped)} interaction(s) dropped: main effect excluded')
+            if log is not None:
+                log.extend(skipped)
+        print(f"{variant}: {tr.shape[1]} columns, {len(made)} pairs -> "
+              f"{sum('__x__' in c for c in tr.columns)} interaction terms, "
+              f"{len(drop)} dropped, {len(nl)} quadratics")
+        return tr, te
+
+    # ------------------------------------------------- validation simulation --
+    def null_sim(self, n=4000, corr=0.8, R=200, out=None):
+        """Type I error under an additive null where x truly interacts with z
+        only, and y is correlated with z but has no interaction with x.
+        Compares partitioning on the attribution against the within-stratum
+        contrast, to demonstrate why the latter is used by
+        :meth:`stratified_screen`."""
+        from scipy.stats import mannwhitneyu
+        rng = np.random.default_rng(0)
+        a, b, c = 0.8, 0.9, 1.5
+        rej_part = rej_contrast = 0
+        for _ in range(R):
+            z = rng.binomial(1, .4, n).astype(float)
+            y = np.where(rng.random(n) < corr, z, 1 - z).astype(float)
+            x = rng.binomial(1, .5, n).astype(float)
+            px, py, pz = x.mean(), y.mean(), z.mean()
+            phi_x = (x - px) * (a + c * pz / 2 + c * z / 2)
+            phi_y = (y - py) * b
+            m = x == 1
+            g1 = phi_x[m] <= np.median(phi_x[m])
+            rej_part += mannwhitneyu(phi_y[m][g1], phi_y[m][~g1],
+                                     alternative='two-sided').pvalue < 0.05
+            d1, v1 = self._slopes(y[x == 0][:, None], phi_y[x == 0][:, None])
+            d2, v2 = self._slopes(y[x == 1][:, None], phi_y[x == 1][:, None])
+            se = float(np.sqrt(v1 + v2)[0])
+            rej_contrast += (2 * norm.sf(abs(float((d1 - d2)[0]) / se)) < 0.05) if se > 0 else 0
+        res = pd.DataFrame([dict(rule='partition on attribution', type_I_error=rej_part / R),
+                            dict(rule='within-stratum contrast', type_I_error=rej_contrast / R)])
+        print(res.to_string(index=False))
+        if out:
+            res.to_csv(out, sep='\t', index=False)
+        return res

shap_recommender-0.1.0/src/shap_recommender.egg-info/PKG-INFO

@@ -0,0 +1,120 @@
+Metadata-Version: 2.4
+Name: shap-recommender
+Version: 0.1.0
+Summary: Exclusion / non-linearity / interaction recommendations from saved SHAP attribution files, and application of them to a design matrix.
+Author: Kaylee
+License-Expression: MIT
+Project-URL: Homepage, https://pypi.org/project/shap-recommender/
+Keywords: shap,feature-selection,interaction-detection,interpretability,machine-learning
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Intended Audience :: Science/Research
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.23
+Requires-Dist: pandas>=1.5
+Requires-Dist: scipy>=1.9
+Requires-Dist: statsmodels>=0.13
+Requires-Dist: patsy>=0.5
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Dynamic: license-file
+
+# shap-recommender
+
+Exclusion / non-linearity / interaction recommendations from saved SHAP
+attribution files, and application of them to a design matrix.
+
+One shared idea runs through two of the three rules. A feature's own
+contribution is represented flexibly (indicator columns when it takes few
+values, a restricted cubic spline when it is continuous), and a model built
+on that flexible basis is compared against a straight line in the feature.
+That comparison answers two different questions:
+
+- **non-linearity** -- does the attribution deviate from a linear function
+  of the feature? This directly tests the linear-trend assumption, rather
+  than relying on a raw correlation coefficient, which conflates "no effect"
+  with "non-linear effect".
+- **interaction** -- does the attribution vary among subjects who share the
+  same feature value? Under additivity the attribution is a deterministic
+  function of the feature, so residual dispersion implies effect
+  modification.
+
+Stratification for the interaction screen is always applied to the observed
+feature value at a pre-specified cut point, never to the attribution itself
+-- splitting on the attribution would condition on the candidate modifier.
+Within strata, the contrast `E[phi_y | y=1] - E[phi_y | y=0]` is compared,
+which (unlike the marginal attribution distribution) is invariant to
+stratum composition under additivity.
+
+## Install
+
+```bash
+pip install shap-recommender
+```
+
+## Expected input files
+
+For each dataset "tag" you want to load, `Recommender.load(tag)` (and the
+CLI's `--tags`) expects two tab-separated files in `res_dir`:
+
+- `shap_values_<tag>.tsv` -- SHAP values, one row per subject, one column
+  per feature, first column = row index.
+- `sel_data_<tag>.tsv` -- the corresponding feature values (design matrix),
+  same row index.
+
+## Command-line use
+
+```bash
+shap-recommender \
+    --res-dir ./shap_results \
+    --tags cohort_a cohort_b \
+    --nonlinear-candidates age bmi creatinine \
+    --out ./recommendations
+```
+
+This writes `exclusion_tests.tsv`, `exclusion_sensitivity.tsv`,
+`nonlinear_tests.tsv`, `interaction_tests.tsv`, `attribution_patterns.tsv`,
+and `recommendations.json` to `--out`. Run `shap-recommender --help` for all
+options (thresholds, bootstrap count, spline degrees of freedom, a
+`--cutpoints` JSON file for pre-specified stratification cut points, etc).
+
+## Library use
+
+```python
+from shap_recommender import Recommender
+
+rec = Recommender(res_dir="./shap_results")
+recommendations = rec.generate(
+    candidates_nonlinear=["age", "bmi", "creatinine"],
+    tags=["cohort_a", "cohort_b"],
+    out="./recommendations",
+)
+
+# apply the recommendations to a design matrix
+X_train_adj, X_test_adj = Recommender.apply(
+    X_train, X_test, recommendations, variant="all",
+)
+```
+
+`Recommender.apply(..., variant=...)` accepts `"baseline"`, `"exclusion"`,
+`"nonlinear"`, `"interaction"`, or `"all"`, so each rule's effect on
+downstream model performance can be evaluated separately.
+
+## Validating the interaction rule
+
+`Recommender.null_sim()` runs a small simulation under an additive null
+(no true interaction with the feature being tested) and reports the type-I
+error rate of the within-stratum contrast used by `stratified_screen`,
+compared against naively partitioning on the attribution itself:
+
+```python
+from shap_recommender import Recommender
+Recommender(res_dir=".").null_sim()
+```
+
+## License
+
+MIT

shap_recommender-0.1.0/src/shap_recommender.egg-info/SOURCES.txt

@@ -0,0 +1,13 @@
+LICENSE
+README.md
+pyproject.toml
+src/shap_recommender/__init__.py
+src/shap_recommender/cli.py
+src/shap_recommender/core.py
+src/shap_recommender.egg-info/PKG-INFO
+src/shap_recommender.egg-info/SOURCES.txt
+src/shap_recommender.egg-info/dependency_links.txt
+src/shap_recommender.egg-info/entry_points.txt
+src/shap_recommender.egg-info/requires.txt
+src/shap_recommender.egg-info/top_level.txt
+tests/test_recommender.py

shap_recommender-0.1.0/src/shap_recommender.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

shap_recommender-0.1.0/src/shap_recommender.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+shap-recommender = shap_recommender.cli:main

shap_recommender-0.1.0/src/shap_recommender.egg-info/requires.txt

@@ -0,0 +1,8 @@
+numpy>=1.23
+pandas>=1.5
+scipy>=1.9
+statsmodels>=0.13
+patsy>=0.5
+
+[dev]
+pytest

shap_recommender-0.1.0/src/shap_recommender.egg-info/top_level.txt

@@ -0,0 +1 @@
+shap_recommender

shap_recommender-0.1.0/tests/test_recommender.py

@@ -0,0 +1,92 @@
+import json
+
+import numpy as np
+import pandas as pd
+import pytest
+
+from shap_recommender import Recommender
+
+
+@pytest.fixture
+def synthetic(tmp_path):
+    rng = np.random.default_rng(0)
+    n = 600
+    age = rng.normal(60, 10, n)
+    sex = rng.binomial(1, 0.5, n)
+    noise_feat = rng.normal(0, 1, n)
+
+    # age: non-linear (quadratic) attribution
+    phi_age = 0.01 * (age - 60) ** 2 + rng.normal(0, 0.05, n)
+    # sex: interacts with age (effect modification)
+    phi_sex = np.where(age > 60, 0.5, -0.5) + rng.normal(0, 0.05, n)
+    # noise_feat: negligible attribution -> should be excluded
+    phi_noise = rng.normal(0, 0.001, n)
+
+    X = pd.DataFrame({"age": age, "sex": sex, "noise_feat": noise_feat})
+    S = pd.DataFrame({"age": phi_age, "sex": phi_sex, "noise_feat": phi_noise})
+
+    res_dir = tmp_path / "res"
+    res_dir.mkdir()
+    S.to_csv(res_dir / "shap_values_train.tsv", sep="\t")
+    X.to_csv(res_dir / "sel_data_train.tsv", sep="\t")
+
+    return str(res_dir), X, S
+
+
+def test_load_roundtrip(synthetic):
+    res_dir, X, S = synthetic
+    rec = Recommender(res_dir=res_dir, seed=0)
+    s, x = rec.load("train")
+    assert list(s.columns) == list(S.columns)
+    assert len(s) == len(x) == len(S)
+
+
+def test_exclusion_flags_negligible_feature(synthetic):
+    res_dir, X, S = synthetic
+    rec = Recommender(res_dir=res_dir, n_boot=100, seed=0)
+    excluded, table = rec.exclusion(S)
+    assert "noise_feat" in excluded
+    assert "age" not in excluded
+
+
+def test_nonlinear_flags_quadratic_feature(synthetic):
+    res_dir, X, S = synthetic
+    rec = Recommender(res_dir=res_dir, seed=0)
+    selected, table = rec.nonlinear(S, X, candidates=["age", "sex", "noise_feat"])
+    assert "age" in selected
+
+
+def test_interaction_flags_effect_modifier(synthetic):
+    res_dir, X, S = synthetic
+    rec = Recommender(res_dir=res_dir, min_cell=50, seed=0)
+    spec, table, diag = rec.interaction(S, X, cutpoints={"age": 60})
+    # sex's attribution should show up as varying with age (Pattern C / screened)
+    assert (diag.set_index("variable").loc["sex", "pattern"] in ("A", "C"))
+
+
+def test_generate_and_apply_end_to_end(synthetic, tmp_path):
+    res_dir, X, S = synthetic
+    rec = Recommender(res_dir=res_dir, n_boot=100, min_cell=50, seed=0)
+    out_dir = tmp_path / "out"
+    recs = rec.generate(
+        candidates_nonlinear=["age", "sex", "noise_feat"],
+        tags=["train"],
+        cutpoints={"age": 60},
+        out=str(out_dir),
+    )
+    assert (out_dir / "recommendations.json").exists()
+    with open(out_dir / "recommendations.json") as fh:
+        loaded = json.load(fh)
+    assert loaded == recs
+
+    X_train, X_test = X.iloc[:400], X.iloc[400:]
+    tr, te = Recommender.apply(X_train, X_test, recs, variant="all")
+    assert "noise_feat" not in tr.columns
+    assert set(tr.columns) == set(te.columns)
+
+
+def test_null_sim_runs():
+    rec = Recommender(res_dir=".", seed=0)
+    res = rec.null_sim(n=500, R=10)
+    assert set(res["rule"]) == {"partition on attribution", "within-stratum contrast"}
+    assert (res["type_I_error"] >= 0).all() and (res["type_I_error"] <= 1).all()