mirpy-lib 3.1.1__tar.gz → 3.2.0__tar.gz

{mirpy_lib-3.1.1 → mirpy_lib-3.2.0}/.gitignore

@@ -194,3 +194,6 @@ experiments/runs/
 
 # obsidian-memory-sync marker (machine-local absolute paths)
 .claude/obsidian-sync.json
+
+# git worktrees live inside the repo dir; never commit them as nested checkouts
+.claude/worktrees/

mirpy_lib-3.2.0/BENCHMARKS.md

@@ -0,0 +1,264 @@
+# Benchmark results — regression baseline
+
+Recorded results for the sample-level (repertoire) embedding benchmarks (`experiments/benchmark_repertoire_*.py`),
+so a re-run can be checked for regression. All numbers are **repeated 50-fold CV mean±std** unless noted, on
+arda-native coordinates. Theory/interpretation: `THEORY.md` T7. Data provenance: `SOURCES.md`.
+
+- **Env**: conda `mirpy` (Python 3.12), Apple M3 (32 GB). Seeds fixed (`seed=0`). `RUN_BENCHMARK` not required
+  for these (they are experiment scripts, not the guarded pytest tier).
+- **Recorded**: 2026-07-14. Re-run and compare; a drop beyond ~1 std on the headline metric is a regression.
+- **Reproducibility**: all load **local data first, HF fallback** — `aging`/`airr_hip` via `_hf.fetch`
+  (`hf_hub_download`), `airr_covid19` via `_covid.covid_path` (local `~/hf/airr_covid19` else `isalgo/airr_covid19`).
+
+## Sample-level embedding — established regime (airr_benchmark aging, airr_hip)
+
+| Benchmark | Command | Headline metric | Value |
+|---|---|---|---|
+| Depth-robustness (`prop:kme`) | `benchmark_repertoire_depth.py` | log–log slope `‖ΔΦ₁‖` vs `n_eff` (theory −0.5) | **−0.55** |
+| Age regression (n=79) | `benchmark_repertoire_aging.py` | \|Spearman(age,pred)\|: diversity / kernel-mean / k-mer | **0.76** / 0.58 / 0.28 |
+| CMV serostatus (n=240) | `benchmark_repertoire_cmvhla.py` | AUC: diversity / Φ blocks / learned | **0.83±0.05** / 0.59–0.63 / 0.49 |
+| CMV age-matched control | `benchmark_repertoire_cmvhla.py` | AUC: diversity / age-only (decade-matched) | **0.83** / 0.45 |
+| HLA-A\*02 (n=500, 25k) | `benchmark_repertoire_hla.py 250 25000` | AUC: second-moment / diversity | **0.623±0.048** / 0.460±0.048 |
+| HLA panel (airr_hip, n=300) | `benchmark_repertoire_hla*panel` | second-moment AUC: B\*08 / B\*07 / B\*44 | **0.82** / 0.76 / 0.66 |
+| Spike-in recovery | `benchmark_repertoire_spikein.py` | recall @N≤3k / FPR (clean P_gen null) | **~35–50%** / ~1.2% |
+
+## Aging divergence — `benchmark_repertoire_agediverge.py 500000 30` (58 donors @500k)
+
+Divergence rises with age at depth but is **diversity-coupled, not an independent axis**.
+
+| Metric | Value | Note |
+|---|---|---|
+| ρ(¹D, divergence) biased KME | −0.153 | biased V-stat `1/n_eff` artifact |
+| ρ(¹D, divergence) **unbiased** KME | **−0.052** | `mmd_matrix(unbiased=True)`; artifact removed |
+| ρ(¹D, divergence) overlap −logF | −0.680 | clonal-expansion driven |
+| ρ(age, divergence overlap F) | 0.704 | (0.24 at 250k — depth-dependent) |
+| **partial ρ(age, divergence \| ¹D)** | **0.067** (p=0.64) | n.s. — no independent axis (same at 40k/250k) |
+
+## COVID cohort (airr_covid19, 300 donors, ≤20k reads)
+
+### Batch cookbook — `benchmark_repertoire_covidbatch.py 300 20000` → **PASS**
+
+| signal | naive AUC | batch-residualized |
+|---|---|---|
+| batch identity (OvR) | 0.777 | **0.032** |
+| COVID status (⟂̸ batch) | **0.660** | 0.409 |
+| HLA-A\*02 (⟂ batch) | 0.596 | **0.609** |
+
+within-mixed-batch COVID (honest) **0.539**; MMD offset:biology ratio **1.05**.
+
+### HLA imprint class I+II — `benchmark_repertoire_covidhla.py 300 20000` (TRB)
+
+second-moment > diversity in direction **15/17** alleles; class-II present **8/9**.
+
+| allele | class | 2nd-moment AUC | Δ vs diversity |
+|---|---|---|---|
+| DRB1\*07:01 | II | **0.758** | +0.204 |
+| B\*07:02 | I | 0.702 | +0.154 |
+| DRB1\*15:01 | II | 0.702 | +0.110 |
+| DQB1\*05:01 | II | 0.619 | +0.152 |
+| A\*02:01 | I | 0.628 | +0.035 |
+
+### Paired α+β — `benchmark_repertoire_covidpaired.py 300 20000`
+
+**TRA carries the stronger HLA imprint**; paired concat sits between (noisier β dilutes).
+
+| allele | β (TRB) | α (TRA) | α+β paired |
+|---|---|---|---|
+| A\*02 | 0.538 | **0.636** | 0.612 |
+| B\*07 | 0.646 | **0.701** | 0.664 |
+| DRB1\*15 | 0.749 | **0.810** | 0.796 |
+
+### COVID biomarker — honest **negative** (`covidstatus.py` / `covidpaired.py`)
+
+| test | β | α | paired |
+|---|---|---|---|
+| COVID status, naive | 0.673 | 0.685 | **0.702** |
+| COVID status, batch-resid | 0.488 | 0.512 | **0.522** (chance) |
+| witness rediscovery of paper's clones | 0.373 | **0.450** | — |
+
+Convalescent (long-past) COVID leaves **no batch-robust bulk clonotype-identity signal** at RNA-seq depth; the
+naive 0.66–0.72 was batch confound. Ground truth is 87% α (4393 α vs 567 β).
+
+---
+
+## 2026-07-14 (pm) — spectral block, COVID witness, TCGA survival
+
+### Spectral interaction block — `benchmark_repertoire_spectral.py 300 20000`
+
+The opt-in top-`r` **eigenvalue** compaction of the second-moment block (new `n_eigs=` on `fit_repertoire_space`)
+is **lossy for the HLA imprint**: HLA-A\*02 carriage lives in *which* public clones co-occur (directional), and
+a rotation-invariant eigenvalue spectrum discards it. Default (`n_eigs=None`) keeps the full upper triangle,
+unchanged.
+
+| representation | block dim | HLA-A\*02 AUC |
+|---|---|---|
+| diversity (baseline) | 4 | 0.560±0.066 |
+| D₂=512 top-16 eigvals | 16 | 0.545±0.093 |
+| D₂=512 top-128 eigvals | 128 | 0.536±0.092 |
+| D₂=256 full upper-tri | 32896 | 0.562±0.078 |
+| **D₂=512 full upper-tri** | 131328 | **0.593±0.070** |
+
+### COVID witness — `benchmark_repertoire_covidwitness.py 300 20000`
+
+Why per-clonotype Fisher-significant clones "vanish" in the bulk embedding: **cohort breadth**, not read depth.
+
+| level | finding |
+|---|---|
+| Fisher genome-wide BH (150–300 donors) | **0** clones pass at any depth (20k or 120k reads) |
+| Fisher full cohort (~1137 donors, user tmp scan) | **39 β / 4 α** clones pass BH (min q≈5e-4) |
+| witness β: whole-cohort → **mixed-batch** | 0.510 → **0.752** (batch control is the lever) |
+| witness β: HLA-strat median / best | 0.532 / 0.712 (best = max-selection; median ≈ whole-cohort) |
+| witness α: whole / mixed / strat-best | 0.463 / 0.424 / 0.473 (no recovery — α witness fails) |
+
+Batch control (mixed-batch contrast, where COVID+healthy share runs) recovers the paper's β clones; per-allele
+HLA stratification adds more noise than signal at these donor counts. HLA+α+β is **not** the missing key — breadth is.
+
+### COVID motif recovery at full breadth — `benchmark_repertoire_covidmotif.py 0 60000`
+
+Using the tool that *works* at breadth (Emerson incidence Fisher) on the **full 1137-donor cohort** (761 COVID /
+376 healthy, 60 k reads), both chains, to actually *find COVID motifs*:
+
+| chain | genome-wide clones q<0.05 | GT-true recovered | + HLA-restricted |
+|---|---|---|---|
+| **α** | 7 | **4** | +0 |
+| β | 0 | 0 | +0 |
+
+The 4 recovered α clones are one coherent public family — **CAG·NYGGSQGNLIF** (single-residue variants), all
+members of the paper's COVID-associated α **cluster 31** (27 GT clones, all `has_covid_association`). **HLA
+restriction adds nothing** because these α clones are already *public* (present across HLA backgrounds) —
+per-allele restriction only loses power.
+
+**WS1 (2026-07-15) — the α/β loop closed, and it is NOT depth (refutes the earlier "β is depth-limited" note).**
+Running β at **full *native* depth** (no downsampling) over the full 1137-donor cohort
+(`benchmark_repertoire_covidmotif.py 0 0`) still passes **1** feature genome-wide at any `min_incidence∈{2,3,5}`
+and recovers **0** GT-β clones — full depth does not rescue β. Nor does the *restricting allele*: a per-stratum
+sweep (`benchmark_repertoire_covidmotif.py native_beta`, 12 class-II/B HLA carrier groups, 117–346 carriers each
+at **native depth** ~3.5 M reads/donor) recovers **0** GT-β in every stratum — refuting "β needs full depth +
+the restricting allele." And it is **not absence**: **253/256 GT-β clones
+are present** in the cohort (median incidence **9** donors, max **110**). The reason is that the paper's β
+"COVID-associated" clones are short **public bystanders** (`CASSx…YEQYF`, TRBJ2-7 — e.g. `CASSYSYEQYF` in 110
+donors) occurring in COVID **and** healthy donors alike, so they do not discriminate status by *incidence*.
+**The α/β asymmetry is which clones are status-enriched vs public bystanders — not depth or breadth:** α's
+recovered `CAG·NYGGSQGNLIF` family is a genuine status-enriched public expansion; β's GT set is dominated by
+non-discriminating convergent publics. *Caveat:* the incidence Fisher tests presence-breadth, not clonal-
+expansion magnitude — a frequency/abundance-based test could still recover β clones enriched in *size* not
+*breadth* (untested). Net: the recoverable "find-motifs" signal is the **public α compartment** (breadth-
+powered, no HLA needed); the breadth-starved bulk MMD witness recovered none of it.
+
+### TCGA — `benchmark_repertoire_tcga.py <chains> BRCA,LUAD,KIRC 50000`
+
+Tumor-type separation is **depth-dependent** (deep IG light chains ≫ shallow TR); **survival adds nothing** over
+clinical covariates. Base Cox C-index (age+sex+stage+log reads): BRCA 0.72, LUAD 0.66, KIRC 0.73.
+
+| chain | tumor-type macro-OvR AUC | survival ΔC (BRCA / LUAD / KIRC) |
+|---|---|---|
+| TRB (median ~24 clonotypes) | 0.523 | +0.003 / −0.003 / +0.002 |
+| IGH | 0.500 | +0.002 / +0.001 / −0.004 |
+| **IGK (deepest chain)** | **0.666** | −0.001 / −0.004 / −0.001 |
+| concat TRB+IGH | 0.538 | −0.002 / +0.001 / +0.001 |
+
+IGK's 0.67 shows the method works when clonotype depth suffices (the deep IG compartment carries tissue signal);
+the flat ΔC across every chain says the tumour-infiltrating repertoire adds **no prognostic value beyond clinical
+covariates** at TCGA RNA-seq depth.
+
+### TCGA survival — biology-grounded features — `benchmark_repertoire_tcga_survival.py`
+
+The clonotype *embedding* adds nothing, but interpretable **biology axes** (isotype from `c_call`, infiltration /
+hot-vs-cold, atypicality = gene-usage divergence from the tumour-type centroid, clonal expansion — modelled on an
+internal AIRR-tissue EDA) do, cancer-specifically. Base Cox = age+sex+stage+log reads; **bold** = best gain per row.
+
+C-index gain over clinical (5-fold CV):
+
+| cancer | C base | isotype | infiltration | atypicality | clonality | all-AIRR |
+|---|---|---|---|---|---|---|
+| SKCM (melanoma) | 0.609 | +0.008 | +0.034 | −0.001 | +0.031 | **+0.036** |
+| KIRP (renal pap.) | 0.717 | −0.008 | **+0.030** | −0.003 | +0.014 | +0.015 |
+| LGG (glioma) | 0.764 | −0.006 | **+0.019** | +0.008 | +0.016 | +0.011 |
+| KIRC (renal cc.) | 0.721 | +0.005 | **+0.009** | +0.002 | +0.002 | +0.006 |
+
+(LUAD / STAD / OV / BLCA: AIRR gain ≈ 0 or negative.)
+
+KM median-split log-rank p (the EDA's stratification test — surfaces threshold effects the linear C-index misses;
+**bold** = p<0.05):
+
+| cancer | infiltration | IgA fraction | atypicality |
+|---|---|---|---|
+| KIRC | **0.002** | 0.089 | 0.880 |
+| SKCM | **0.000** | **0.026** | 0.716 |
+| KIRP | **0.020** | 0.750 | 0.555 |
+| OV | **0.044** | 0.581 | 0.691 |
+| LGG | **0.022** | 0.076 | **0.001** |
+| BLCA | 0.676 | **0.010** | 0.243 |
+
+**Read:** infiltration (hot/cold) stratifies survival in **5/8** cancers (melanoma, both renal, ovarian, glioma)
+and adds C-index in melanoma/KIRP; **IgA** (isotype / mucosal) stratifies **bladder** + melanoma; **atypicality**
+stratifies **glioma** — each matching known immune biology. Prognosis lives in infiltration magnitude + isotype +
+typicality, **not** clonotype identity, and is cancer-specific — the reason the identity embedding's ΔC was flat.
+
+## Repertoire embedding for TME & survival — pan-cancer (33 TCGA types)
+
+The reframing: the prognostic axes above are **channels of one TME-aware, multi-chain repertoire embedding**
+Φ(S) (`_tcga_embedding.py`: per-chain identity ‖ diversity ‖ coverage/infiltration, + isotype + composition
++ atypicality). `benchmark_repertoire_tcga_pancancer.py ALL` fits Φ once over 9 425 OS-annotated samples
+(78-dim, 5 embeddable chains + all 7 in the composition channel) and, per cancer, reports the CV **ΔC-index**
+of clinical+Φ over clinical (age+sex+stage+log reads) and a **likelihood-ratio p** for the Φ block.
+
+Cancers where Φ is robustly prognostic (**both** LR p<0.05 **and** CV ΔC>0; bold):
+
+| cancer | n | events | C base | C+Φ | ΔC | LR p | top channel |
+|---|---|---|---|---|---|---|---|
+| **SKCM** (melanoma) | 460 | 219 | 0.609 | 0.647 | **+0.039** | **0.000** | coverage |
+| **BLCA** (bladder) | 411 | 182 | 0.650 | 0.675 | **+0.025** | **0.002** | identity |
+| **HNSC** (head & neck) | 500 | 218 | 0.617 | 0.639 | **+0.022** | **0.012** | coverage |
+| **LGG** (glioma) | 423 | 112 | 0.764 | 0.779 | **+0.016** | **0.000** | coverage |
+
+Strong effect-size positives (ΔC>0, LR n.s. — power-limited): SARC +0.038 (isotype), KIRP +0.034 (atypicality),
+LUAD +0.009, KIRC +0.008, BRCA +0.007. Immune-cold / small cohorts are flat-to-negative (overfit): LIHC −0.029,
+STAD −0.026, PAAD −0.029, UCEC −0.072 (n=181/35ev).
+
+**Pan-cancer (20 evaluable):** mean ΔC ≈ 0 (median +0.003), Φ significant in 5/20 — most-informative channel
+tally **coverage 5 · atypicality 5 · composition 4 · isotype 3 · identity 2 · diversity 1**. The signal is
+cancer-specific and lives in the **TME channels (infiltration, atypicality, composition), not clonotype
+identity**; the pan-cancer mean is ≈0 because immune-cold cohorts overfit and cancel the immune-hot wins —
+so the honest claim is *Φ adds significant prognostic value in immunologically active cancers* (melanoma,
+glioma, head&neck, bladder), not universally.
+
+### TME states (unsupervised) — `benchmark_repertoire_tcga_tme.py ALL 6`
+
+KMeans (k=6) on Φ's TME channels (9 425 samples) recovers coherent, interpretable microenvironment states
+(z-scored channel means; **bold** = defining extreme). The UMAP (`experiments/figures/umap_tcga_tme`) shows a
+clean **infiltration gradient** (hot → cold) organising the pan-cancer cohort.
+
+| state | n | infiltration | T-vs-B | switch | diversity | death | HR (vs ref) | enriched cancers |
+|---|---|---|---|---|---|---|---|---|
+| cold-humoral | 1565 | −0.53 | −0.36 B | +0.39 | −0.40 | 0.33 | **1.14** (p=0.011) | BRCA, BLCA, PRAD |
+| hot-diverse | 2090 | **+1.03** | −0.70 B | +0.35 | **+0.92** | 0.31 | 0.92 (p=0.072) | BRCA, LUAD, STAD |
+| T-balanced | 1277 | −0.10 | **+0.86 T** | +0.15 | +0.06 | 0.29 | 1.09 | KIRC, PRAD, BRCA |
+| cold-T unsw. | 883 | **−1.58** | **+1.72 T** | **−2.20** | −1.66 | 0.30 | 1.05 | LGG, LIHC, SARC |
+| cold | 871 | −1.27 | +0.74 T | −0.70 | −1.13 | 0.28 | 1.08 | THCA, LIHC, LGG |
+| warm-B (ref) | 2739 | +0.48 | −0.45 B | +0.37 | +0.40 | 0.31 | — | BRCA, LUAD, HNSC |
+
+Stratified multivariate log-rank across states (blocking tumour type) **p=0.0038** — the states are prognostic
+*beyond* cancer type. The **cold-humoral** low-infiltration state carries the **worst** outcome (HR 1.14, p=0.011);
+the **hot-diverse** state trends protective (HR 0.92). Glioma (LGG) falls in the cold, strongly T-skewed states,
+matching its known low-but-T-biased infiltrate. The repertoire embedding organises the TME unsupervised.
+
+### In-silico evolution — perturb infiltration, decode the coupled response — `benchmark_repertoire_tcga_insilico.py ALL`
+
+`mir.repertoire.sample_descriptor` makes every metric a **smooth derivable coordinate** (infiltration = log-mass,
+diversity = log n_eff, clonality = Σw², identity = kernel mean), so the cohort's joint distribution is a
+generative **manifold** and moving a sample along the infiltration axis (hot↔cold) while staying on it predicts
+how the other metrics respond. Gaussian-manifold conditional slopes d(metric)/d(infiltration), pan-cancer mean:
+
+| response as tumour gets hotter | slope | reading |
+|---|---|---|
+| diversity | **+0.84** | hot ⇒ more diverse (universal, all cancers) |
+| class-switch | +0.52 | hot ⇒ more class-switched B cells |
+| IgG | +0.49 | hot ⇒ more IgG (mature humoral) |
+| T-vs-B | −0.63 | hot ⇒ more B-skewed (TLS-like) |
+| atypicality | −0.23 | hot ⇒ less atypical / more convergent |
+
+**In-silico "make this tumour hotter"** (move cold→hot on-manifold, read the CoxPH Δlog-HR): predicted
+**protective in 12/20** cancers (SKCM −0.49, HNSC −0.63, LUAD −0.44, LAML −0.96 …) — TIL-is-good — but correctly
+**adverse in glioma** (LGG **+1.19**, GBM +0.47) and renal (KIRC/KIRP +0.3), matching the known "immune-hot glioma
+is worse" biology. The embedding *simulates* the hot↔cold survival axis; the couplings are learned, not imposed.

{mirpy_lib-3.1.1 → mirpy_lib-3.2.0}/CHANGELOG.md

@@ -3,6 +3,23 @@
 All notable changes to `mirpy-lib` (import `mir`). This project follows semantic versioning; the v3 line is a
 greenfield ML/embedding rewrite (the classical v1.x/v2 toolkit is frozen on branch `legacy-v2`).
 
+## 3.2.0 — 2026-07-17
+
+Minor: one new public module, nothing removed or changed.
+
+### Added
+
+- **`mir.explain`** (T7) — explainable readouts over any repertoire feature matrix.
+  `ChannelSpec` / `ChannelBuilder` / `stack_embeddings` attach the name→column map that `Φ.vector`
+  does not carry (`stack_embeddings` is exact: `X[i] == embs[i].vector`, names only, no transform);
+  `channel_report` ablates each named channel under a caller-supplied scorer (leave-one-in by
+  default; `mode="both"` adds the conditional half that exposes *redundant* channels — high `delta`,
+  `delta_out≈0`; optional row-permutation p-values); `channel_drivers` hops from a winning
+  **kernel-mean** channel to the clonotypes driving it via `class_witness`, and refuses channels with
+  no clonotype pre-image (a Hill number's "drivers" are a category error, not an open question).
+  Scorer-agnostic by design — the library never sees the labels and ships no scorers, so a Cox
+  C-index and a CV AUC both plug in. **No existing module changed.**
+
 ## 3.1.1 — 2026-07-14
 
 Maintenance re-release of 3.1.0 with no functional changes — 3.1.0 was withdrawn from PyPI, and

{mirpy_lib-3.1.1 → mirpy_lib-3.2.0}/CLAUDE.md

@@ -6,6 +6,28 @@ library** for immune receptors (TCR/BCR). A greenfield, slim rewrite: prototype
 (TCREMP) now, neural codecs + density methods next (Part 2). The classical v1.x/v2 toolkit is
 frozen on branch **`legacy-v2`** (`mirpy-lib` 2.x) — do not develop there.
 
+## Repo split (2026-07-16) — three homes
+This repo is now **the library + its CI/test data only**. Analysis and manuscript live elsewhere:
+- **`~/vcs/code/mirpy`** (here): `mir/` library, unit + fast/slow CI tests (`tests/`, on slim bundled
+  data), plans for new features, docs, and **slim tabular benchmark results** for README/docs. Benchmark
+  *scripts* and the theory *appendix* have moved out (below).
+- **`~/vcs/projects/2026-mirpy-analysis`**: all **benchmark scripts** (`benchmarks/` — local + aldan3),
+  full result docs (BENCHMARKS/THEORY/…), figures, run outputs, dataset catalog. Run the mirpy library
+  from here; refresh numbers-of-record here.
+- **`~/vcs/manuscripts/2026-mirpy-ms`**: the **theory appendix** (`appendix/tcremp_theory.tex`) + the paper
+  (`manuscript-latex/`). (Earlier this appendix lived here in the code repo; it migrated on 2026-07-16.)
+
+Compute: heavy benchmarks run on the **aldan3** cluster (env `mirpy`, `sbatch`); light jobs local. See the
+analysis repo.
+
+## Worktrees — REQUIRED for Claude sessions
+**Work in a git worktree for this repo, never directly on the main checkout.** Multiple Claude sessions
+edit mirpy concurrently (the 2026-07-16 split was done across parallel sessions and hit collisions), so at
+the start of any session that will modify this repo, call **`EnterWorktree`** (isolated worktree under
+`.claude/worktrees/`, fresh branch off `origin/master`); commit + push from there; merge via PR/fast-forward.
+This keeps concurrent sessions from clobbering each other's working trees. The same applies to the sibling
+repos (`2026-mirpy-analysis`, `2026-mirpy-ms`).
+
 ## Reuse, don't duplicate — the ecosystem
 mirpy v3 has **no AIRR data-model layer of its own**; it works on `vdjtools` polars frames and
 delegates:
@@ -46,13 +68,13 @@ fix were added to `vdjtools` under the owner's direction (this is that owner's e
   PCs; diverse chains (IGH/TR*) 2000 / ~65 PCs (95%), ~220–300 PCs (99%, for codec reconstruction).
 - `bench/` — `vdjdb.py` (loader), `metrics.py` (`cluster(method=…)` DBSCAN default | HDBSCAN | OPTICS,
   kneedle eps, F1/retention), `theory.py` (S1–S3 + T5 + T6 `tcrnet_convergence` + `codec_losslessness`).
-  Needs `[bench]`. Clustering is a precision/coverage trade-off (`experiments/benchmark_clustering.py`):
+  Needs `[bench]`. Clustering is a precision/coverage trade-off (`benchmarks/benchmark_clustering.py`):
   DBSCAN tightest/purest (paper regime), HDBSCAN ~3× coverage at lower F1 (variable-density), OPTICS
   dominated, KMeans no noise-rejection.
 - `density.py` — continuous-density TCRNET/ALICE (T6): `fit_density_space` (one shared PCA basis),
   `neighbor_enrichment` (balloon adaptive-radius Poisson/binomial + water-level calibration;
   `backend=` **exact** BallTree default | **kdtree** exact scipy cKDTree 5–9× faster | **ann**
-  pynndescent ~30× at ≥1e5, recall<1 conservative — `experiments/benchmark_ann.py`),
+  pynndescent ~30× at ≥1e5, recall<1 conservative — `benchmarks/benchmark_ann.py`),
   `enriched_mask`, `denoise_and_cluster`, `generate_background` (vdjtools P_gen, lazy). Torch-free
   (scipy/sklearn). Prefer a **biological control** as background (differential) over P_gen.
   **Abundance-aware** (T6 sec:dens-abund): pass `abundance=` (clone sizes) + `weight="log1p"`/`anscombe`
@@ -63,6 +85,18 @@ fix were added to `vdjtools` under the owner's direction (this is that owner's e
   clonotype-cloud PCA + RFF basis, prototype-hash-verified `RepertoireSpace`), `sample_embedding`
   (`Φ(S)` = RFF kernel mean ‖ coverage-standardized Hill ‖ second-moment Fisher; `n_eff=(Σw²)⁻¹`),
   `mmd_distance`/`mmd_matrix`/`hla_stratified_mmd`, `class_witness` (supervised MMD motif finder). Torch-free.
+  Opt-in `fit_repertoire_space(n_eigs=r)` swaps the second-moment block's full `D₂(D₂+1)/2` upper-triangle
+  for its top-`r` eigenvalues (rotation-invariant spectrum; default `None` = upper-tri, unchanged — but
+  lossy for the *directional* HLA imprint, so the full triangle stays the recommended block; `benchmark_repertoire_spectral.py`).
+- `explain.py` — **explainable readouts over any feature matrix** (T7). `ChannelSpec`/`ChannelBuilder`/
+  `stack_embeddings` attach the name→column map `Φ.vector` does not carry (per-chain blocks merge by
+  name); `channel_report(X, spec, scorer, base=, mode="in"|"out"|"both", n_permutations=)` ablates each
+  channel under a **caller-supplied** scorer — the library never sees `y` and ships no scorers, so a Cox
+  C-index and a CV AUC both plug in. `mode="in"` (default) is marginal, because the scorers reduce their
+  input and leave-one-*out* is near-blind to narrow channels; `"both"` adds the conditional half, and
+  high `delta` + `delta_out≈0` is the **redundancy** signature. `channel_drivers` hops channel→clonotypes
+  via `class_witness`, but **only** for a channel declared `attributable` (a kernel mean); a Hill number
+  has no clonotype pre-image and it raises. Depends one-way on `repertoire.py`; nothing there changed.
 - `ml/` — Part 2 (torch), neural codecs + `set_encoder.py` (learned repertoire track: Set-Transformer/DeepRC
   attention pooling + `SetEncoderBundle`).
 - `resources/` — `prototypes/` (TSVs + manifest), `gene_library/` (region_annotations.txt),
@@ -73,8 +107,8 @@ fix were added to `vdjtools` under the owner's direction (this is that owner's e
   (pure-Python hatchling; no C build). Extras: `[bench] [annotate] [build] [ml] [docs] [dev]`.
 - Tests: `python -m pytest tests/ -q` (78 pass; `-m "not integration"` for the ~5s fast tier —
   the pynndescent ANN parity test carries a one-time JIT cost). All self-contained on bundled resources.
-- Experiments: `python experiments/reproduce_supplementary.py` (theory S1–S3),
-  `python experiments/benchmark_vdjdb.py` (Table S1). Analyses: `analyze_prototype_counts.py`
+- Experiments: `python benchmarks/reproduce_supplementary.py` (theory S1–S3),
+  `python benchmarks/benchmark_vdjdb.py` (Table S1). Analyses: `analyze_prototype_counts.py`
   (geometry saturates by K≈100 — T.1/S4), `analyze_pc_decomposition.py` (V/J η² ≈0.44/0.49,
   CDR3-length η² 0.13 & R²=0.95; germline low-rank ~13 PCs — T.4). See `THEORY.md`.
 
@@ -100,7 +134,7 @@ fix were added to `vdjtools` under the owner's direction (this is that owner's e
   **PCA-compacted junction embedding** (T3: 1000-D junction → ~51 PCs @95% var on arda coords,
   99% for reconstruction), fit train-only.
   V/J stay exact germline lookups (nothing to learn); the codec's job is the junction part.
-  Results on M3 MPS (`experiments/train_{forward_encoder,inverse_decoder,pgen_regressor}.py`),
+  Results on M3 MPS (`benchmarks/train_{forward_encoder,inverse_decoder,pgen_regressor}.py`),
   **re-pinned on the arda-native coords** (2026-07-13):
   - **DONE forward codec** (`tokenize,encoder,train`): seq → compact code, reconstruction
     cosine **0.9984** (n=8k; paper 0.887). DNN inference is K-independent. Geometry is fine at
@@ -111,7 +145,7 @@ fix were added to `vdjtools` under the owner's direction (this is that owner's e
     (over-compacted); 99% (284 PCs) restores 0.40.** The T5 chain-adaptive lesson now bites TRB
     too, not just IGH/TRD — the decoder default is 99%.
   - **DONE Pgen regressor** (`train.train_pgen_regressor`): seq → log10 Pgen(1mm), r **0.967**,
-    **~190× faster** than the native DP. Breakdown (`experiments/benchmark_pgen_variants.py`):
+    **~190× faster** than the native DP. Breakdown (`benchmarks/benchmark_pgen_variants.py`):
     r ranks marginalized > J > V > V&J (a CDR3-only regressor best predicts the pure-CDR3
     marginalized Pgen; V&J-conditional depends on unseen genes) and 1mm > exact (smoother ball).
   - **DONE unified codec** (`codec.py`): jointly train encoder+decoder with a geometry-anchor
@@ -121,9 +155,9 @@ fix were added to `vdjtools` under the owner's direction (this is that owner's e
     rotation match.** `CodecBundle` serializes the PCA transform + a prototype hash + weights;
     `load` refuses a prototype-hash mismatch so incomparable embeddings can't be mixed. Any
     trained codec MUST be shipped as a bundle, never bare weights.
-  - Per-chain/species breakdown: `experiments/benchmark_codec_chains.py` (forward cos 0.997–0.999
+  - Per-chain/species breakdown: `benchmarks/benchmark_codec_chains.py` (forward cos 0.997–0.999
     universal; inverse chain-dependent — short κ/λ/γ/mouse easy, IGH/TRD hard).
-  - **DONE lossless-recon depth/K/data sweep** (`experiments/benchmark_lossless_{depth,kpc}.py`,
+  - **DONE lossless-recon depth/K/data sweep** (`benchmarks/benchmark_lossless_{depth,kpc}.py`,
     real held-out TRB from `isalgo/airr_control` vs arda landmarks): exact-match is **training-data-
     limited, not architecture-limited** — the old "~0.40 ceiling" was n≈8k starvation. Same one-shot
     decoder: n=20k→50k→100k drives exact **0.885→0.941→0.958** (token 0.996→0.998) at K=2000/PC=400,
@@ -140,7 +174,7 @@ fix were added to `vdjtools` under the owner's direction (this is that owner's e
     is V/J/CDR1/CDR2). Isotype is a low-cardinality categorical (~9 IGH classes ≈ 3 bits) *independent*
     of V/J/CDR3 ⇒ not reconstructable ⇒ **carry `c_call` as an exact stored column, never embed it**
     (same as v_call/j_call metadata). No codec change.
-  - **DONE T5 (SHM/IGH)** (`bench.theory.shm_embedding_drift`, `experiments/benchmark_igh_shm.py`):
+  - **DONE T5 (SHM/IGH)** (`bench.theory.shm_embedding_drift`, `benchmarks/benchmark_igh_shm.py`):
     SHM embedding drift is ~linear/sublinear in mutation load (bounded; IGH 104/mut < TRB 128/mut
     — IGH lowest slope, robust to SHM). IGH's hard reconstruction is **over-compaction, not the
     frame**: on arda coords 95% code (95 PCs) → exact 0.115, 99% (422 PCs) → 0.356 (> old 0.152;
@@ -151,7 +185,7 @@ fix were added to `vdjtools` under the owner's direction (this is that owner's e
     enrichment `E(z)=f_obs/f_gen` in embedding space; Poisson (ALICE, P_gen bg) or binomial
     (TCRNET, control bg) + BH q; water-level calibration for the naive regime. Theory
     `tcrnet_convergence` confirms the r→0 graph limit (ρ 0.37→−0.05 as radius grows). Benchmarks
-    `experiments/benchmark_density_{yfv,ankspond,tcrnet,vdjdb}.py` on HF
+    `benchmarks/benchmark_density_{yfv,ankspond,tcrnet,vdjdb}.py` on HF
     `isalgo/airr_{yfv19,ankspond,benchmark,control}` + VDJdb slim.
     **Key lesson**: real repertoires are pervasively convergent, so a P_gen background flags ~40% of
     clones — use a *biological control* (differential: day15-vs-day0, B27±, CMV-vs-control) for
@@ -167,7 +201,7 @@ fix were added to `vdjtools` under the owner's direction (this is that owner's e
   HLA-stratified; `class_witness` = supervised MMD motif finder; learned co-equal Set-Transformer/DeepRC in
   `mir.ml.set_encoder`. Reuse: `TCREmp.embed`, `density.{_WEIGHTS,fit_density_space,calibrate_radius}`,
   `vdjtools.stats.inext`, `preprocess.downsample`, `ml.bundle` hashing. Tests
-  `tests/test_{repertoire,set_encoder}.py`; benchmarks `experiments/benchmark_repertoire_{aging,depth,cmvhla,hla}.py`
+  `tests/test_{repertoire,set_encoder}.py`; benchmarks `benchmarks/benchmark_repertoire_{aging,depth,cmvhla,hla}.py`
   (shared `_cohort.py`; `airr_benchmark` aging 79, `airr_hip` Emerson 2017 786). **Key empirical lesson**
   (`THEORY.md` T7; **all adversarially verified — two over-claims caught & corrected**): depth-robustness holds
   (`prop:kme`, slope −0.55) but is a **generic KME/MMD Monte-Carlo rate**, not embedding-specific. **Age & CMV
@@ -197,4 +231,64 @@ fix were added to `vdjtools` under the owner's direction (this is that owner's e
     the paper's clones (0.37β/0.45α, GT is 87% α). Long-past exposure has no batch-robust bulk signal at RNA-seq depth.
   - **Parallelism/GPU documented** (README "Performance & parallelism"): `TCREmp(threads=0)` all-core; density
     `backend="kdtree"` multicore exact; `pick_device` **CUDA→MPS→CPU** (was MPS-only) + `MIR_DEVICE`.
+- **Sample-level refinements + TCGA — 2026-07-14 (pm)** (`BENCHMARKS.md` "2026-07-14 (pm)"; all numbers recorded):
+  - **WS1 spectral block** (`fit_repertoire_space(n_eigs=r)`, `benchmark_repertoire_spectral.py`): opt-in top-`r`
+    eigenvalues of the second-moment covariance. **Lossy for HLA** — HLA-A\*02 lives in *which* clones co-occur
+    (directional), eigenvalues are rotation-invariant → top-r ≤0.55 while full upper-tri reaches **0.593** (D₂=512).
+    Default `None` = upper-tri (unchanged); keep the full triangle. Test `test_spectral_second_block_top_r_eigvals`.
+  - **WS2 COVID witness** (`benchmark_repertoire_covidwitness.py`, vdjtools `biomarker.fisher_association`): answers
+    "Fisher passes but embedding vanishes — why?" → **breadth, not depth**. Genome-wide Fisher passes 0 clones at
+    150–300 donors (any depth 20k/120k); full ~1137-donor cohort passes 39β/4α (user tmp scan). Sample-level lever
+    is **batch control** (β witness whole 0.51 → mixed-batch **0.75**); per-allele HLA-stratification adds noise
+    (median≈whole). HLA+α+β is not the key — breadth is.
+  - **WS2b COVID motif recovery** (`benchmark_repertoire_covidmotif.py`, full 1137-donor incidence Fisher, both
+    chains): the *breadth-powered* way to find motifs. **α genome-wide recovers 4 GT-true clones = a coherent
+    public family CAG·NYGGSQGNLIF (paper cluster 31)**; β recovers 0 (rarer/HLA-restricted). **HLA restriction
+    adds 0** — the recoverable α clones are already public. **WS1 (2026-07-15) closed the α/β loop and REFUTED
+    the "β is depth/HLA-limited" guess**: β at full *native* depth genome-wide, and `native_beta` across 12
+    class-II/B carrier strata at native depth, both recover **0** GT-β. β's GT clones are short **public
+    bystanders** (`CASSx…YEQYF` TRBJ2-7, present in COVID *and* healthy; 253/256 present, median incidence 9),
+    so they don't discriminate status by *incidence* — the α/β split is **status-enriched vs public-bystander,
+    not depth/breadth/HLA**. (Untested caveat: an abundance/size-based test could still catch β clones enriched
+    in *magnitude* not *breadth*.)
+  - **WS3 UMAP** (`plot_sample_umap.py {covid,aging,hip}`): faceted UMAP of Φ₁ (=MMD geometry) by age/HLA/batch/
+    COVID/CMV → `benchmarks/figures/`. covid19 visibly clusters by **batch** in raw Φ₁, dissolves after
+    `residualize` (the `prop:batch` story made visual).
+  - **WS4 TCGA** (`_tcga.py` local-first loader, untars once; `benchmark_repertoire_tcga.py`; `lifelines` in
+    `[bench]`): 9591 samples, 7 chains, OS survival. **Tumor-type separation is depth-dependent** — deepest chain
+    **IGK 0.67** ≫ shallow TRB 0.52 / IGH 0.50 (IG light chains carry tissue signal). **Survival = honest negative**:
+    ΔC-index ≈0 for every chain over a clinical Cox (age+sex+stage+log reads, base C 0.66–0.73) — the tumour-
+    infiltrating repertoire adds no prognostic value at RNA-seq depth. No grade column (used stage); PFS empty.
+  - **WS4b TCGA survival via biology features** (`_tcga_features.py` isotype/infiltration/atypicality/clonality;
+    `benchmark_repertoire_tcga_survival.py`) — the prognosis the *identity embedding* misses lives in interpretable
+    axes (modelled on an internal AIRR-tissue EDA). **Infiltration (hot/cold Z-axis) stratifies survival in 5/8
+    cancers** (KIRC/SKCM/KIRP/OV/LGG log-rank p<0.05) and adds C-index in melanoma (+0.036) / KIRP (+0.030);
+    **IgA isotype** stratifies bladder (mucosal, p=0.010) + melanoma; **atypicality** stratifies glioma (p=0.001)
+    — all matching known immunobiology. Isotype/atypicality show via KM stratification, not linear C-index. Net:
+    tissue prognosis = infiltration magnitude + isotype + typicality, **not clonotype identity**.
+  - **REFRAME → TME-aware repertoire embedding, pan-cancer** (`_tcga_embedding.py` = per-chain identity ‖
+    diversity ‖ coverage/infiltration ‖ isotype ‖ composition ‖ atypicality — the biology axes recast as Φ(S)
+    channels; `benchmark_repertoire_tcga_{pancancer,tme}.py`; THEORY.md T7 "Repertoire embeddings for the TME",
+    BENCHMARKS.md "Repertoire embedding for TME & survival"). One Φ over 9425 OS samples (78-dim, 5 embeddable
+    chains + all 7 via composition). **Robustly prognostic** (LR p<0.05 & CV ΔC>0) in **SKCM +0.039 / BLCA
+    +0.025 / HNSC +0.022 / LGG +0.016**; effect-size positives SARC (isotype) / KIRP (atypicality). Top channel
+    = **coverage/atypicality/composition, ~never identity**; immune-cold cohorts overfit → pan-cancer mean ΔC≈0
+    masks the immune-hot wins. **Paradigm lesson (T7 #6): the same Φ(S) that reads infection/HLA in blood via
+    its identity channels stratifies the TME + survival in tissue via its non-identity channels.** Unsupervised
+    Φ clustering → TME states (`benchmark_repertoire_tcga_tme.py`, UMAP `benchmarks/figures/umap_tcga_tme`).
+  - **Derivable descriptor + in-silico evolution** (`mir.repertoire.sample_descriptor`/`RepertoireDescriptor`/
+    `decode_metrics`; `benchmark_repertoire_tcga_insilico.py`; THEORY.md T7 lesson 7): **mass-preserving** smooth
+    descriptor — infiltration=log-mass, diversity=log n_eff, clonality=Σw², identity=kernel mean are all smooth
+    coordinates (keeps the mass Φ normalises away). The cohort coordinate distribution = a generative manifold;
+    perturbing infiltration + conditioning (Gaussian) = **in-silico evolution**: hotter ⇒ diversity +0.84 /
+    switch +0.52 / T-vs-B −0.63; CoxPH "make hotter" protective 12/20 cancers, adverse in glioma (LGG +1.19) —
+    learned couplings match immunobiology. Test `test_descriptor_metrics_derivable_smooth_and_decodable`.
+    **TODO**: per-cancer n_pc; Thorsson immune-subtype validation; a learned (flow/VAE) generative manifold for
+    full sequence-level simulation. "Promote the multi-chain descriptor" is **partially done** (2026-07-17):
+    `build_embedding` was four things — (a) per-locus space fitting, (b) channel assembly + registry,
+    (c) TME feature engineering, (d) impute/z-score. `mir.explain` promotes **(b)+(d)**; **(c)** is correctly
+    analysis-local (isotype/composition/atypicality are tissue features, not library concerns — though
+    atypicality, being a Φ-geometry op, would belong in `repertoire.py` as `centroid_atypicality` if ever
+    wanted). **Remaining**: (a) — "fit one `RepertoireSpace` per locus over a cohort" is construction, so it
+    belongs in `mir.repertoire` as a separate, smaller follow-up.
 - Full plan: `~/.claude/plans/i-want-to-completely-crystalline-lake.md`.

{mirpy_lib-3.1.1 → mirpy_lib-3.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mirpy-lib
-Version: 3.1.1
+Version: 3.2.0
 Summary: ML-oriented embeddings for immune receptor repertoires (TCR/BCR)
 Project-URL: Homepage, https://github.com/antigenomics/mirpy
 Project-URL: Repository, https://github.com/antigenomics/mirpy
@@ -31,6 +31,7 @@ Requires-Dist: vdjmatch; extra == 'annotate'
 Provides-Extra: bench
 Requires-Dist: huggingface-hub; extra == 'bench'
 Requires-Dist: kneed; extra == 'bench'
+Requires-Dist: lifelines; extra == 'bench'
 Requires-Dist: matplotlib; extra == 'bench'
 Requires-Dist: pynndescent; extra == 'bench'
 Requires-Dist: seaborn; extra == 'bench'

{mirpy_lib-3.1.1 → mirpy_lib-3.2.0}/SOURCES.md

@@ -33,7 +33,7 @@ Phenotype-labelled repertoire cohorts for the §T.7 sample-level embedding bench
 | `airr_hip` | HF `isalgo/airr_hip` = **Emerson et al. 2017** (*Nat Genet*, PMID 28369038, doi:10.1038/ng.3822) | manifest `metadata.txt`: `file_name, sample_id, age, race, sex, cmv, hla`; **786 subjects**; TCRβ + **CMV serostatus** (`+`/`-`/`NA` = 340/421/25) + **HLA-A/B typing** (comma-sep `HLA-A*02,…`; A*02 most common, n=294). Sample files are the `file_name` path (`corr/HIP*.txt.gz`), full-depth (~10⁵ clonotypes) | `hf_hub_download(repo_id="isalgo/airr_hip", filename="metadata.txt", repo_type="dataset")` then per-sample `file_name`. Downsample per-sample; **age-match CMV±** (CMV rises with age, a diversity confound) |
 | SRA shallow RNA-seq | HF `isalgo/airr_benchmark/sra/` (`meta.tsv`, `samples.tar.gz`) | `PMID Run BioProject Sample`; 2993 rows; all PMID 30830871, BioProject PRJNA511467 | low-coverage bulk-RNA-seq stress set (the `10²–10⁴` clonotype/chain regime); accession manifest only |
 | `airr_covid19` | HF `isalgo/airr_covid19` = **Vlasova, Nekrasova, Komkov et al. 2026** (*Genome Med* 18, 20; DNA-multiplex FMBA cohort — cite per the dataset card, ⚠ verify before appendix use) | `metadata.tsv`: `file_name, reads, batch_id, sample_id, COVID_status` (COVID/healthy/precovid/unknown), `COVID_IgG/IgM/PCR`, full **4-digit HLA class I+II both alleles** (`HLA-{A,B,C}_{1,2}`, `HLA-{DPB1,DQB1,DRB1}_{1,2}`, `HLA-DRB{3,4,5}_1`), `donor_id`; **1258 donors**, **paired TRA+TRB**. **9 real sequencing batches** (`2020/09…2021/01_FMBA_NovaSeq*`, 103–185 donors each), partly status-confounded (NovaSeq4≈all-healthy, NovaSeq9=all-precovid; NovaSeq5/6/7 mixed). Ships `covid_associated_clonotypes.csv` (114 CDR3 clusters, `has_covid_association` T/F — COVID ground-truth motifs). **No `age`** (age+sex live in the sibling `isalgo/airr_covid19_vacc`). Files: **local git-LFS checkout** at `~/hf/airr_covid19/<file_name>` (vdjtools cols `count/freq/cdr3aa/v/j`), *not* the HF hub cache — load by local path, not `hf_hub_download` | local `~/hf/airr_covid19/`; `git clone git@hf.co:datasets/isalgo/airr_covid19` (LFS). Downsample per-sample. **Batch is a strong nuisance ⇒ prop:batch test bed**; HLA ⟂ batch (donor genetics) vs COVID-status ⟂̸ batch (confounded) |
-| `airr_tcga` | HF `isalgo/airr_tcga` = **TCGA** tumour bulk RNA-seq → AIRR (RNA-seq AIRR method Bolotin et al. *Nat Biotechnol* 2017, PMID 29020005) | `metadata.tsv`: `sample_id` (`TCGA-XX-XXXX.N`), `subject_id`, `cancer_type`/`disease`/`study_id` (**33 cancer types**), `sex/race/age`, `tumor_stage`, `therapy/response`, `OS/OS_event/PFS/PFS_event`, **`total_reads`** (raw FASTQ total, GDC realigned-BAM `Total_Reads`, verified ≠ aligned; 99.6% present) + `aligned_reads`; **9591 samples**, **TCR+BCR but IG-dominant** (~97% IG: IGK/IGL/IGH; ~3% TR), median **634** clonotypes/sample (RNA-seq regime). Caveats: 487 lack clinical, 42 lack `total_reads` | **local git-LFS checkout** at `~/hf/airr_tcga/` (`git clone git@hf.co:datasets/isalgo/airr_tcga`) or `hf_hub_download(repo_id="isalgo/airr_tcga", filename="samples.tar.gz"/"metadata.tsv", repo_type="dataset")` + `load.py`. Per-sample AIRR in `samples.tar.gz` (`samples/<sample_id>.tsv`, AIRR cols). Separate **`metadata.hla.tsv`** = donor-level **HLA class-I** keyed by `subject_id` (`HLA-{A,B,C}_{1,2}`, 4-digit, + `hla_source`), union of TCGA **PanImmune** (Thorsson et al. *Immunity* 2018, PMID 29628290; primary, 7649) + **OptiType** fill (Szolek et al. *Bioinformatics* 2014, PMID 25143287; 1474); **9123/9450 donors = 96.5%**, → 9263 samples. **Tissue depth↔infiltration test bed** (`sec:samp-norm`, `prop:infiltration`): `total_reads` is the technical denominator for the receptor read-fraction infiltration proxy; HLA enables the `prop:hla` stratification |
+| `airr_tcga` | HF `isalgo/airr_tcga` = **TCGA** tumour bulk RNA-seq → AIRR (RNA-seq AIRR extraction method Bolotin et al. *Nat Biotechnol* 2017, PMID 29020005) | `metadata.tsv`: `sample_id` (`TCGA-XX-XXXX.N`), `subject_id`, `cancer_type`/`disease`/`study_id` (**33 cancer types**), `sex/race/age`, **stage** (`cancer_stage` S1–S4 + `tumor_stage` i–iv; no grade column), `therapy/response`, `OS`+`OS_event` (days + 0/1 death, ~9510 usable; **`PFS`/`PFS_event` are present-but-empty — unusable**), **`total_reads`** (raw FASTQ total, GDC realigned-BAM `Total_Reads`, verified ≠ aligned; 99.6% present) + `aligned_reads`; **9591 samples**, **TCR+BCR but IG-dominant** (~97% IG: IGK/IGL/IGH; ~3% TR), median **634** clonotypes/sample (RNA-seq regime). Caveats: 487 lack clinical, 42 lack `total_reads` | **local git-LFS checkout** at `~/hf/airr_tcga/` (`git clone git@hf.co:datasets/isalgo/airr_tcga`) or `hf_hub_download(repo_id="isalgo/airr_tcga", filename="samples.tar.gz"/"metadata.tsv", repo_type="dataset")` + `load.py`. Per-sample AIRR in `samples.tar.gz` (`samples/<sample_id>.tsv`, AIRR cols). Separate **`metadata.hla.tsv`** = donor-level **HLA class-I** keyed by `subject_id` (`HLA-{A,B,C}_{1,2}`, 4-digit, + `hla_source`), union of TCGA **PanImmune** (Thorsson et al. *Immunity* 2018, PMID 29628290; primary, 7649) + **OptiType** fill (Szolek et al. *Bioinformatics* 2014, PMID 25143287; 1474); **9123/9450 donors = 96.5%**, → 9263 samples. **Tissue depth↔infiltration test bed** (`sec:samp-norm`, `prop:infiltration`): `total_reads` is the technical denominator for the receptor read-fraction infiltration proxy; HLA enables the `prop:hla` stratification |
 
 ## Theory appendix (`appendix/`)
 

{mirpy_lib-3.1.1 → mirpy_lib-3.2.0}/THEORY.md

@@ -241,6 +241,60 @@ against prototype k; `D_ij = ‖φ(i) − φ(j)‖₂` the embedding-space Eucli
   pairwise repertoire MMD must use the **unbiased** estimator (`mmd_matrix(unbiased=True)`) when samples differ
   in depth/diversity — the biased V-statistic's `1/n_eff` self-term otherwise fakes a divergence signal.
 
+- **2026-07-14 (pm) additions** (`BENCHMARKS.md` "2026-07-14 (pm)"; three more generalizable lessons):
+  **(3) the interaction block's information is directional, not spectral.** The second-moment block
+  `Σ w ψ₂ψ₂ᵀ` carries the HLA imprint in *which* public clones co-occur (its eigen-*vectors*), so an opt-in
+  top-`r` **eigenvalue** compaction (`fit_repertoire_space(n_eigs=r)`) — rotation-invariant — is lossy: HLA-A\*02
+  AUC ≤0.55 vs 0.593 for the full upper triangle. Keep the upper triangle; the spectral path is a compact
+  fingerprint only. **(4) clone-level significance is cohort-breadth-limited.** Per-clonotype Fisher (Emerson
+  incidence) needs the full cohort: 0 COVID clones clear BH FDR at 150–300 donors *at any read depth*, but ~1137
+  donors yield 39β/4α — so a bulk sample-level embedding cannot recover a signal that is not yet significant at
+  its own donor count; the tractable sample-level lever is **batch control** (mixed-batch witness β 0.51→0.75),
+  not per-allele HLA stratification. **(5) tissue signal is depth-gated.** On TCGA (bulk RNA-seq, IG-dominant),
+  the *deepest* chain IGK separates 33 tumour types at AUC 0.67 while shallow TR (~tens of clonotypes/sample)
+  sits at chance — but the embedding adds **no** survival value over clinical covariates (ΔC-index≈0, every
+  chain), a clean prognostic negative at RNA-seq depth (`benchmark_repertoire_tcga.py`).
+
+### Repertoire embeddings for the tumour microenvironment & survival (pan-cancer)
+
+The clonotype-*identity* embedding being flat for tumour survival is not the end of the paradigm — it is a
+statement about *which channel* carries the signal. A repertoire embedding is a **multi-channel object**, and
+different questions engage different channels:
+
+| channel | what it measures | question it answers |
+|---|---|---|
+| kernel-mean + second-moment (identity) | *which* clones / public structure | antigen specificity, HLA imprint, infection/vaccine response |
+| diversity (Hill) | clonality / richness | immune ageing, exhaustion |
+| coverage / infiltration | receptor read load (magnitude Φ₁ normalises away) | **tumour infiltration — hot vs cold** |
+| isotype / C-gene (IGH class-switch) | plasma / mucosal humoral state | **B-cell / TLS-driven TME** |
+| composition (T-vs-B, 7-chain balance) | TME cell-type mix | **immune contexture** |
+| atypicality (Φ-distance to tumour-type centroid) | selection / divergence from the typical repertoire | antigen-driven remodelling |
+
+Making these channels explicit (`_tcga_embedding.py`) turns the "biology features" into one **TME-aware,
+multi-chain repertoire embedding** Φ(S). Pan-cancer (33 TCGA types, 9 425 OS-annotated samples,
+`benchmark_repertoire_tcga_pancancer.py`): over a clinical Cox (age+sex+stage+log reads), Φ is **robustly
+prognostic** (LR p<0.05 *and* CV ΔC>0) in **SKCM +0.039, BLCA +0.025, HNSC +0.022, LGG +0.016** — the
+immunologically active cancers — with strong power-limited positives in SARC (isotype +0.038) and KIRP
+(atypicality +0.034). The most-informative channel is **coverage (infiltration) / atypicality / composition**,
+essentially never clonotype identity; immune-cold small cohorts overfit (LIHC/PAAD/UCEC negative), so the
+pan-cancer mean ΔC≈0 masks a clear immune-hot-cancer effect. **Lesson (6):** the same Φ(S) that fingerprints
+infection and HLA in blood also stratifies the tumour microenvironment and survival in tissue — one just reads
+the *non-identity* channels (infiltration magnitude + humoral class + contexture), which the depth-normalised,
+single-chain kernel mean discards by construction. Unsupervised clustering of Φ recovers interpretable TME
+states (hot-diverse / cold-humoral / T-skewed-unswitched), characterised by these channels, enriched in the
+expected cancers, and prognostic **beyond** tumour type (stratified multivariate log-rank p=0.004; the
+low-infiltration cold-humoral state carries the worst outcome, HR 1.14) — `benchmark_repertoire_tcga_tme.py`.
+
+**Lesson (7) — derivable metrics ⇒ a simulatable embedding.** Because the mass-preserving descriptor
+(`mir.repertoire.sample_descriptor`) makes infiltration (log-mass), diversity (log n_eff) and clonality (Σw²)
+**smooth analytic coordinates** — the "smoother form" of the Hill block, keeping the total mass Φ normalises
+away — the cohort's joint coordinate distribution is a generative **manifold**. Perturbing the infiltration
+coordinate and conditioning the rest (Gaussian manifold) is *in-silico evolution*: pan-cancer, a hotter tumour
+is predicted to become more diverse (+0.84), more class-switched (+0.52) and more B-skewed (−0.63 T-vs-B), and
+the CoxPH survival response is protective in 12/20 cancers but adverse in glioma (LGG Δlog-HR +1.19) — learned
+couplings that recover known immunobiology (`benchmark_repertoire_tcga_insilico.py`). The embedding is thus not
+only a descriptor but a *model* one can sample and evolve.
+
 ## Reproduced numbers (v3 pipeline)
 
 Run `python experiments/reproduce_supplementary.py` (S1–S3) and

{mirpy_lib-3.1.1 → mirpy_lib-3.2.0}/docs/api.rst

@@ -102,6 +102,16 @@ One vector per repertoire: RFF kernel mean ‖ Hill diversity ‖ second moment;
    :undoc-members:
    :show-inheritance:
 
+Explainable readouts (``mir.explain``)
+--------------------------------------
+
+Which named channel of ``Φ`` carries the signal, and which clonotypes drive it (Theory T7).
+
+.. automodule:: mir.explain
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
 Benchmark harness (``mir.bench``)
 ---------------------------------
 

{mirpy_lib-3.1.1 → mirpy_lib-3.2.0}/docs/index.rst

@@ -53,6 +53,9 @@ Capabilities (see the :doc:`API reference <api>`)
 - **Repertoire embedding** — one fixed vector per sample: RFF kernel mean, coverage-standardised
   Hill diversity, and a second-moment Fisher block; MMD / HLA-stratified distance and a motif
   witness (``mir.repertoire``, T7).
+- **Explainable readouts** — name ``Φ``'s channels and ablate them under *your* scorer to see which
+  one carries the signal (and whether it is merely redundant), then hop from the winning kernel-mean
+  channel to the clonotypes driving it (``mir.explain``, T7).
 - **Neural codecs** — forward / inverse / Pgen / unified codecs and a learned repertoire set
   encoder; CUDA → MPS → CPU device selection (``mir.ml``, ``[ml]`` extra).
 - **Benchmark harness** — VDJdb clustering + F1/retention and reproduced theory (``mir.bench``).