mirpy-lib 3.2.0__tar.gz → 3.4.0__tar.gz

{mirpy_lib-3.2.0 → mirpy_lib-3.4.0}/.gitignore

@@ -10,7 +10,7 @@ tests/assets/vdjdb.slim.txt.gz
 tests/assets/real_repertoires/
 tests/assets/srx_repertoires/
 *.airr.tsv
-notebooks/assets/
+examples/assets/
 
 ## VSCode
 
@@ -184,10 +184,10 @@ cython_debug/
 #  and can be added to the global gitignore or merged into this file.  For a more nuclear
 #  option (not recommended) you can uncomment the following to ignore the entire idea folder.
 #.idea/
-mir/docs
+src/mir/docs
 .idea
 public
-notebooks/assets/derived/
+examples/assets/derived/
 
 # ML run artifacts (checkpoints/metrics) — never committed
 experiments/runs/

mirpy_lib-3.4.0/CHANGELOG.md

@@ -0,0 +1,147 @@
+# Changelog
+
+All notable changes to `mirpy-lib` (import `mir`). This project follows semantic versioning; the v3 line is a
+greenfield ML/embedding rewrite (the classical v1.x/v2 toolkit is frozen on branch `legacy-v2`).
+
+## 3.4.0 — 2026-07-18
+
+Minor: a command-line interface, a uv-based dev setup, and a documentation overhaul. No public
+Python API removed; one optional-dependency group split out.
+
+### Added
+
+- **`mir` command-line interface** (`[project.scripts]`, also `python -m mir.cli`) — the two
+  embedding scales without writing Python:
+  - `mir embed clonotypes SAMPLE` → a per-clonotype TCREMP embedding table (`e0…`).
+  - `mir embed repertoires SAMPLE…` → one repertoire vector `Φ(S)` per sample **per chain** on one
+    shared basis (`phi0…`), with optional `--mmd` pairwise-distance output.
+
+  Inputs are any format `vdjtools.io` reads (AIRR/vdjtools/MiXCR/immunoSEQ/parquet); output is TSV
+  or Parquet. See `mir embed <cmd> -h`. Tests in `tests/test_cli.py`.
+- **`mir.repertoire.correct_batch`** — Harmony-like cluster-aware batch correction on a stacked
+  sample×feature Φ matrix. Removes the batch offset *per soft cluster* (batch-diversity-penalised),
+  so a batch confounded with a biological cluster is corrected without erasing that biology; reduces
+  exactly to `mir.cohort.residualize` at `n_clusters=1` / `theta=0` (`prop:batch`).
+- **`[ann]` optional-dependency group** for the approximate-NN density backend (`pynndescent`).
+
+### Changed
+
+- **Development now uses a repo-local `.venv` via [uv](https://docs.astral.sh/uv/)** instead of
+  conda. `setup.sh` is rewritten (bash/zsh portable; `--dev-parents`, `--docs`, `--tests`); the
+  conda `environment.yml` is removed. Runtime is unchanged — still a pure-Python `py3-none-any` wheel.
+- **`pynndescent` moved from `[bench]` to the new `[ann]` extra.** `[bench]` is now all pure-wheel
+  (no numba/llvmlite), so `pip install "mirpy-lib[bench]"` resolves cleanly on any Python. Users of
+  `density.neighbor_enrichment(backend="ann")` should install `"mirpy-lib[ann]"`.
+- **`vdjtools>=3.0.0`** (was `>=2.3.0`).
+- Documentation overhaul: a use-case-driven user guide, the two CLI commands documented, an
+  examples/notebooks page, `mir.cohort` and `mir.bench.eval` added to the API reference, a logo, and
+  the sample-embedding schematic + real depth-robustness figure. Zero-warning Sphinx build.
+- **Repo layout** (no effect on the installed package): adopted the **src-layout** (`mir/` →
+  `src/mir/`); renamed `notebooks/` → `examples/`; and moved the working result/plan markdown out of
+  the repo root — `THEORY.md` to the manuscript repo, `BENCHMARKS.md` / `REPERTOIRE_{EMBEDDING,LESSONS}.md`
+  / `SQRT_D_MIGRATION.md` / `ROADMAP.md` to `2026-mirpy-analysis/benchmarks/`. Root keeps
+  README / CHANGELOG / CLAUDE / SOURCES.
+
+### Fixed
+
+- `DEFAULT_GAP_POSITIONS = (3, 4, -4, -3)` had three independent definitions
+  (`distances/junction`, `embedding/tcremp`, `ml/bundle`); now defined once in `distances.junction`
+  and imported, so the coordinate constant cannot drift.
+- `cohort.cluster_samples` docstring described itself; `AntigenMetric` and `mir.bench.eval` gained
+  the docstrings/module-map entries they were missing.
+
+## 3.3.0 — 2026-07-17
+
+Minor: one new public parameter, nothing removed or changed.
+
+### Added
+
+- **`fit_density_space(chunk_size=)`** — embed and project in batches so the full raw matrix is never
+  resident. Peak memory becomes `max(pca_fit_cap, chunk_size) × n_features` instead of
+  `len(df) × n_features`: measured **10.60 GB → 1.81 GB** at 450k pooled clonotypes, and flat in `N`
+  (vs linear), at no wall-clock cost. This is what makes whole-cohort density arms runnable on a
+  laptop — the 4.2M-clonotype pooled arm is ~51 GB raw and ~102 GB once `scaler.transform` upcasts to
+  float64. Chunking is bit-exact at the embedding level (`_embed` of a slice == the slice of
+  `_embed`); the projected coordinates agree to float noise (~1e-7 relative), since BLAS summation
+  order depends on batch shape.
+
+### Fixed
+
+- `fit_density_space`'s `pca_fit_cap` docstring claimed it "lets whole repertoires be embedded without
+  a full-matrix PCA". It caps the **fit**, not the memory — both raw matrices were already
+  materialized before the PCA was fitted. Documented, and `chunk_size=` is the actual remedy.
+- **`mir.__version__` was stale** — it read `3.1.1` on the published 3.2.0, because the release bump
+  moved `pyproject.toml` but not `mir/__init__.py`, and `publish.yml` only validates *pyproject ==
+  tag*. Both now read 3.3.0. (`__version__` is still hand-maintained; deriving it from
+  `importlib.metadata` would retire this failure mode for good.)
+- `tests/assets/olga_humanTRB_1000.txt.gz` was a slice of the alphabetically sorted VDJdb TRB dump,
+  not OLGA output as its name and `SOURCES.md` claimed — so it was **not** an antigen-naive null (12%
+  of rows had a Hamming-1 neighbour vs 0.2% for real OLGA). Regenerated from `olga-generate_sequences`;
+  provenance and a byte-reproducible regenerate command recorded in `SOURCES.md`. No test was
+  invalidated (they use it only as a generic TRB junction pool), but external calibrations that treated
+  it as a synthetic negative control were comparing VDJdb against itself. Not shipped (tests are
+  excluded from the sdist); listed here because it invalidates results, not code.
+
+## 3.2.0 — 2026-07-17
+
+Minor: one new public module, nothing removed or changed.
+
+### Added
+
+- **`mir.explain`** (T7) — explainable readouts over any repertoire feature matrix.
+  `ChannelSpec` / `ChannelBuilder` / `stack_embeddings` attach the name→column map that `Φ.vector`
+  does not carry (`stack_embeddings` is exact: `X[i] == embs[i].vector`, names only, no transform);
+  `channel_report` ablates each named channel under a caller-supplied scorer (leave-one-in by
+  default; `mode="both"` adds the conditional half that exposes *redundant* channels — high `delta`,
+  `delta_out≈0`; optional row-permutation p-values); `channel_drivers` hops from a winning
+  **kernel-mean** channel to the clonotypes driving it via `class_witness`, and refuses channels with
+  no clonotype pre-image (a Hill number's "drivers" are a category error, not an open question).
+  Scorer-agnostic by design — the library never sees the labels and ships no scorers, so a Cox
+  C-index and a CV AUC both plug in. **No existing module changed.**
+
+## 3.1.1 — 2026-07-14
+
+Maintenance re-release of 3.1.0 with no functional changes — 3.1.0 was withdrawn from PyPI, and
+this version restores the package under a fresh, clean version. The API, coordinates, and behaviour
+are identical to 3.1.0 (see below).
+
+## 3.1.0 — 2026-07-14
+
+The Part-2 feature tier on top of the 3.0 consolidated embedding core: neural codecs, continuous-density
+methods, and the sample-level (repertoire) embedding, all on arda-native coordinates.
+
+### Added
+- **`mir.density`** (T6) — graph-free continuous-density TCRNET/ALICE: balloon adaptive-radius enrichment
+  (Poisson/binomial + BH q, water-level calibration), abundance-aware weighted mass, backends
+  `exact`(BallTree) / `kdtree`(multicore) / `ann`(pynndescent).
+- **`mir.repertoire`** (T7) — sample-level embedding `Φ(S)` = RFF kernel mean ‖ coverage-standardized Hill ‖
+  second-moment Fisher; `mmd_distance`/`mmd_matrix` (now with **`unbiased=True`** diagonal-removed MMD²),
+  `hla_stratified_mmd`, `class_witness` motif finder.
+- **`mir.ml`** (Part 2, `[ml]` extra) — forward/inverse/pgen/unified neural codecs + `CodecBundle`
+  (prototype-hash-verified shipping); learned repertoire track `set_encoder` (Set-Transformer/DeepRC).
+- **Bench** — clustering `method=` (dbscan/hdbscan/optics); `bench.theory` T6 `tcrnet_convergence`,
+  `codec_losslessness`.
+- **Repertoire benchmarks** — `experiments/benchmark_repertoire_{aging,depth,cmvhla,hla,yfv,spikein,
+  agediverge}.py` and the COVID cohort suite `{covidbatch,covidhla,covidstatus,covidpaired}.py`
+  (`airr_covid19`, local-first + HF fallback). Recorded baselines in `BENCHMARKS.md`.
+
+### Changed
+- **Coordinate system re-pinned to arda-native** germline + real-repertoire prototypes (versioned; any model
+  trained on the old coords must be retrained).
+- **`mir.ml` device selection** now **CUDA → MPS → CPU** (was MPS-only) with a `MIR_DEVICE` env override and
+  CUDA seeding — GPU support beyond Apple silicon.
+- Documented all parallelism knobs (README "Performance & parallelism"): `TCREmp(threads=0)` all-core,
+  density `backend="kdtree"` multicore-exact, `cluster(n_jobs=…)`, BLAS env.
+
+### Fixed
+- **Unbiased repertoire MMD** — the biased V-statistic's `1/n_eff` self-term inflated low-diversity samples and
+  faked a divergence signal; `unbiased=True` removes it. Aging-divergence re-evaluated at depth: real but
+  diversity-coupled, not an independent axis.
+
+### Notes
+- New lessons for the theory appendix in `REPERTOIRE_LESSONS.md`; full findings in `THEORY.md` T7.
+- Pure-Python `py3-none-any` wheel; native code comes from `seqtree`/`vdjtools` wheels.
+
+## 3.0.0 — greenfield v3 embedding core (unreleased)
+- Prototype (TCREMP) embedding on `seqtree.gapblock` + baked arda germline distances; vdjtools reuse (no AIRR
+  data-model layer of its own); bench harness (VDJdb Table S1); theory scaffold (`THEORY.md` S1–S3, T1–T5).

{mirpy_lib-3.2.0 → mirpy_lib-3.4.0}/CLAUDE.md

@@ -8,9 +8,12 @@ frozen on branch **`legacy-v2`** (`mirpy-lib` 2.x) — do not develop there.
 
 ## Repo split (2026-07-16) — three homes
 This repo is now **the library + its CI/test data only**. Analysis and manuscript live elsewhere:
-- **`~/vcs/code/mirpy`** (here): `mir/` library, unit + fast/slow CI tests (`tests/`, on slim bundled
-  data), plans for new features, docs, and **slim tabular benchmark results** for README/docs. Benchmark
-  *scripts* and the theory *appendix* have moved out (below).
+- **`~/vcs/code/mirpy`** (here): **src-layout** — the library is `src/mir/` (import `mir`); unit +
+  fast/slow CI tests (`tests/`, on slim bundled data), runnable `examples/` (marimo), docs. As of the
+  3.4.x cleanup the result/plan markdown moved out: **`THEORY.md` → the manuscript repo**;
+  **`BENCHMARKS.md`, `REPERTOIRE_{EMBEDDING,LESSONS}.md`, `SQRT_D_MIGRATION.md`, `ROADMAP.md` →
+  `2026-mirpy-analysis/benchmarks/`**. Only `README`/`CHANGELOG`/`CLAUDE`/`SOURCES` stay at the root.
+  Benchmark *scripts* and the theory *appendix* had already moved out (below).
 - **`~/vcs/projects/2026-mirpy-analysis`**: all **benchmark scripts** (`benchmarks/` — local + aldan3),
   full result docs (BENCHMARKS/THEORY/…), figures, run outputs, dataset catalog. Run the mirpy library
   from here; refresh numbers-of-record here.
@@ -53,7 +56,7 @@ negative S2) — so prototypes use real reads, not model generation.
 mirpy is normally read-only to the sibling repos; the `from_arda` builder + a tandem-D generation
 fix were added to `vdjtools` under the owner's direction (this is that owner's ecosystem).
 
-## Layout (`mir/`)
+## Layout (`src/mir/`)
 - `aliases.py`, `alleles.py` — species/locus + allele normalization.
 - `distances/junction.py` — `junction_distance_matrix` (seqtree.gapblock). Coordinate knobs (all
   default to the published space, threaded through `TCREmp`): `metric="squared"` (=`d`, default) |
@@ -88,6 +91,20 @@ fix were added to `vdjtools` under the owner's direction (this is that owner's e
   Opt-in `fit_repertoire_space(n_eigs=r)` swaps the second-moment block's full `D₂(D₂+1)/2` upper-triangle
   for its top-`r` eigenvalues (rotation-invariant spectrum; default `None` = upper-tri, unchanged — but
   lossy for the *directional* HLA imprint, so the full triangle stays the recommended block; `benchmark_repertoire_spectral.py`).
+  Also `fit_repertoire_spaces` (one basis **per locus** — the multi-chain fit), `centroid_atypicality`
+  (per-sample cosine distance to a group centroid — a Φ-geometry op feeding the digital-donor atypicality channel),
+  and `correct_batch` (Harmony-like cluster-aware batch correction on a stacked Φ matrix; reduces to
+  `cohort.residualize` at `n_clusters=1`/`theta=0` — `prop:batch`; test in `test_repertoire.py`).
+- `cohort.py` — **the digital donor** (T.7): `fit_donor_embeddings`/`DonorCohort` fuse per-chain identity
+  (kernel-mean, cross-sample PCA-reduced) ‖ diversity ‖ coverage across loci through **one `ChannelBuilder`**,
+  with an `extra_channels` hook for the analysis's own tissue/clinical blocks; comparability bites twice
+  (per-locus `prototype_hash` **and** the stored identity PCA) so `save`/`load` verify every hash and
+  `transform` is the only held-out path. Plus `residualize` (batch cookbook), `cluster_samples`
+  (MMD→precomputed-metric cluster), `incidence_biomarkers` (subject-incidence Fisher, delegating to
+  `vdjtools.biomarker.fisher`). Generalizes the analysis-repo `_tcga_embedding.build_embedding` glue.
+- `bench/eval.py` — the scorers `channel_report` consumes (kept out of `explain.py` so it stays scorer-free):
+  `cv_auc` / `held_out_auc` (classification), `cv_cindex` / `km_logrank` (Cox survival, `[bench]`→lifelines),
+  `kmer_matrix` (baseline).
 - `explain.py` — **explainable readouts over any feature matrix** (T7). `ChannelSpec`/`ChannelBuilder`/
   `stack_embeddings` attach the name→column map `Φ.vector` does not carry (per-chain blocks merge by
   name); `channel_report(X, spec, scorer, base=, mode="in"|"out"|"both", n_permutations=)` ablates each
@@ -99,14 +116,22 @@ fix were added to `vdjtools` under the owner's direction (this is that owner's e
   has no clonotype pre-image and it raises. Depends one-way on `repertoire.py`; nothing there changed.
 - `ml/` — Part 2 (torch), neural codecs + `set_encoder.py` (learned repertoire track: Set-Transformer/DeepRC
   attention pooling + `SetEncoderBundle`).
+- `cli.py` — the `mir` console script (argparse, stdlib): `embed clonotypes` (TCREmp table) /
+  `embed repertoires` (per-locus `fit_repertoire_space`→`sample_embedding` Φ(S), optional `--mmd`).
+  Reads via `vdjtools.io.read`; writes TSV/parquet.
 - `resources/` — `prototypes/` (TSVs + manifest), `gene_library/` (region_annotations.txt),
   `germline_dist/` (baked `.npz`, from `build_germline_dist.py`).
 
 ## Build / test / run
-- Conda env **`mirpy`** (Python 3.12; do NOT use `.venv` here). `pip install -e .`
-  (pure-Python hatchling; no C build). Extras: `[bench] [annotate] [build] [ml] [docs] [dev]`.
-- Tests: `python -m pytest tests/ -q` (78 pass; `-m "not integration"` for the ~5s fast tier —
-  the pynndescent ANN parity test carries a one-time JIT cost). All self-contained on bundled resources.
+- **Repo-local `.venv` via uv** (Python 3.12; conda retired 2026-07-18). `bash setup.sh`
+  (bash/zsh; `--dev-parents` editable-installs `../seqtree ../vdjtools ../vdjmatch`, `--docs`,
+  `--tests`), or `uv pip install -e ".[dev,bench]"`. Pure-Python hatchling; no C build for `mir`
+  itself. Extras: `[bench] [ann] [annotate] [build] [ml] [docs] [dev] [examples]` (`[ann]` =
+  pynndescent, split out of `[bench]` so `[bench]` stays numba-free / clean to resolve).
+- CLI: `mir embed clonotypes SAMPLE` (→ clonotype embedding table) / `mir embed repertoires
+  SAMPLE… ` (→ per-sample-per-chain Φ(S)); `mir/cli.py`, `tests/test_cli.py`.
+- Tests: `python -m pytest tests/ -q` (`-m "not integration and not benchmark"` for the fast tier;
+  torch/pynndescent tests skip unless `[ml]`/`[ann]` installed). All self-contained on bundled resources.
 - Experiments: `python benchmarks/reproduce_supplementary.py` (theory S1–S3),
   `python benchmarks/benchmark_vdjdb.py` (Table S1). Analyses: `analyze_prototype_counts.py`
   (geometry saturates by K≈100 — T.1/S4), `analyze_pc_decomposition.py` (V/J η² ≈0.44/0.49,
@@ -126,6 +151,16 @@ fix were added to `vdjtools` under the owner's direction (this is that owner's e
   GPU only in `mir.ml`: `pick_device()` = **CUDA → MPS → CPU** auto, override `device=`/`MIR_DEVICE`.
 
 ## Open loops / next steps
+- **Embedding-tier roadmap** (`2026-mirpy-analysis/benchmarks/ROADMAP.md`, moved out in the 3.4.x
+  cleanup) — the "vdjtools at the embedding level" audit +
+  plan (three verbs: make / measure / generate-decode). **Phase 0** (robustness + optimization quick wins)
+  and **Phase 1** (cohort tier: `bench/eval.py`, `repertoire.{fit_repertoire_spaces,centroid_atypicality}`,
+  `cohort.py` digital donor) are **DONE**. Next: **Phase 2** generative loop (`generate.py` `DescriptorDensity`
+  + `evolve`/`sample`, `CodecBundle.from_unified/from_decoder`), then multimodal encoders + embedding
+  trajectory. **Analysis-repo follow-up:** refactor `_tcga_embedding.build_embedding` onto
+  `cohort.fit_donor_embeddings` (+ `extra_channels` for isotype/composition/atypicality) and re-verify the
+  pan-cancer ΔC numbers. NB Phase 0 flipped the density default to `backend="kdtree"` — re-verify any
+  recorded balloon-mode baselines (±1 boundary counts).
 - **v3.0 remaining**: 10X paired benchmark; docs (Sphinx theory section + notebooks); CI; publish
   `py3-none-any` wheel; regenerate `generate_prototypes.py` via `vdjtools.model.generate`.
 - **Bench tuning**: raw kneedle eps over-merges; `cluster(eps_factor=0.4)` recovers the paper

{mirpy_lib-3.2.0 → mirpy_lib-3.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mirpy-lib
-Version: 3.2.0
+Version: 3.4.0
 Summary: ML-oriented embeddings for immune receptor repertoires (TCR/BCR)
 Project-URL: Homepage, https://github.com/antigenomics/mirpy
 Project-URL: Repository, https://github.com/antigenomics/mirpy
@@ -25,7 +25,9 @@ Requires-Dist: scikit-learn
 Requires-Dist: scipy
 Requires-Dist: seqtree>=0.3.0
 Requires-Dist: tqdm
-Requires-Dist: vdjtools>=2.3.0
+Requires-Dist: vdjtools>=3.0.0
+Provides-Extra: ann
+Requires-Dist: pynndescent; extra == 'ann'
 Provides-Extra: annotate
 Requires-Dist: vdjmatch; extra == 'annotate'
 Provides-Extra: bench
@@ -33,11 +35,10 @@ Requires-Dist: huggingface-hub; extra == 'bench'
 Requires-Dist: kneed; extra == 'bench'
 Requires-Dist: lifelines; extra == 'bench'
 Requires-Dist: matplotlib; extra == 'bench'
-Requires-Dist: pynndescent; extra == 'bench'
 Requires-Dist: seaborn; extra == 'bench'
 Provides-Extra: build
 Requires-Dist: biopython; extra == 'build'
-Requires-Dist: vdjtools[model]>=2.3.0; extra == 'build'
+Requires-Dist: vdjtools[model]>=3.0.0; extra == 'build'
 Provides-Extra: dev
 Requires-Dist: psutil>=5; extra == 'dev'
 Requires-Dist: pytest-cov; extra == 'dev'
@@ -56,6 +57,16 @@ Provides-Extra: ml
 Requires-Dist: torch; extra == 'ml'
 Description-Content-Type: text/markdown
 
+<p align="center">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="assets/mirpy_dark.svg">
+    <source media="(prefers-color-scheme: light)" srcset="assets/mirpy_light.svg">
+    <!-- Absolute PNG fallback: PyPI strips <picture>/<source> and cannot render a relative or
+         raw-served SVG, so the logo must be an absolute-URL raster here. GitHub uses the SVG sources. -->
+    <img alt="mirpy" src="https://raw.githubusercontent.com/antigenomics/mirpy/master/assets/mirpy_light.png" width="360">
+  </picture>
+</p>
+
 <h1 align="center">mirpy — ML embeddings for immune repertoires</h1>
 
 [![PyPI](https://img.shields.io/pypi/v/mirpy-lib.svg)](https://pypi.org/project/mirpy-lib/)
@@ -65,7 +76,7 @@ Description-Content-Type: text/markdown
 **mirpy v3** turns T-/B-cell receptor sequences into fixed-length numeric vectors you can
 cluster, visualize, and feed to ML models. It implements **TCREMP** — embedding each receptor
 by its alignment distances to a fixed set of *prototype* sequences — so that Euclidean distance
-in embedding space approximates pairwise alignment distance (see [`THEORY.md`](THEORY.md)).
+in embedding space approximates pairwise alignment distance (Theory T1).
 
 > v3 is a slim, embedding-focused rewrite. The classical repertoire toolkit (parsing, overlap,
 > diversity, TCRnet, GLIPH, …) lives on the **`legacy-v2`** branch (`mirpy-lib` 2.x) and in the
@@ -108,6 +119,24 @@ labels = cluster(pca_denoise(X, n_components=50))
 Paired chains concatenate per-chain embeddings via `PairedTCREmp`. Input/output are AIRR polars
 frames keyed by `vdjtools.io.schema` column names.
 
+## Command line
+
+`pip install mirpy-lib` also installs a `mir` command for the two embedding scales — no Python
+needed. Inputs are any format `vdjtools.io` reads (AIRR TSV, vdjtools, MiXCR, immunoSEQ, parquet).
+
+```bash
+# one repertoire  ->  per-clonotype embedding table (e0…), the input to clustering / ML
+mir embed clonotypes sample.tsv --pca 50 -o clonotypes.parquet
+
+# a dataset of repertoires  ->  one fingerprint Φ(S) per sample, per chain (phi0…), on one
+# shared basis so the rows are mutually comparable; --mmd also writes the pairwise MMD matrix
+mir embed repertoires cohort/*.tsv.gz -o phi.tsv --mmd mmd.tsv
+```
+
+`mir embed clonotypes -h` / `mir embed repertoires -h` list every flag (species, locus,
+prototype count, weight, Φ blocks, …). Sample id defaults to the filename stem; the locus is
+inferred per file (or restrict with `--locus`).
+
 ## Recommended presets
 
 `TCREmp.from_defaults(species, locus)` uses the per-chain preset when `n_prototypes` is
@@ -145,9 +174,11 @@ Xr = pca_denoise(X, n_components=p.n_components_recon)    # codec reconstruction
 | `mir.embedding.tcremp` | `TCREmp` / `PairedTCREmp` — the prototype embedding |
 | `mir.embedding.pca` | PCA denoising of embeddings |
 | `mir.distances` | junction distance (`seqtree.gapblock`; `metric`/`matrix`/`alignment` options) + baked germline distances |
-| `mir.bench` | VDJdb loader, clustering (`cluster(method=…)`: DBSCAN/HDBSCAN/OPTICS) + F1/retention, theory experiments (incl. `codec_losslessness`) |
+| `mir.bench` | VDJdb loader, clustering (`cluster(method=…)`: DBSCAN/HDBSCAN/OPTICS) + F1/retention, theory experiments (incl. `codec_losslessness`), cohort scorers (`bench.eval`: `cv_auc`/`cv_cindex`/`km_logrank`) |
 | `mir.density` | continuous-density TCRNET/ALICE — enrichment (+ clonal-abundance channel, `backend=` exact/kdtree/ann) + noise-filtering (Theory T6) |
-| `mir.repertoire` | sample-level (repertoire) embedding — RFF kernel mean ‖ Hill diversity ‖ second moment; MMD / HLA-stratified distance; motif witness (Theory §T.7) |
+| `mir.repertoire` | sample-level (repertoire) embedding — RFF kernel mean ‖ Hill diversity ‖ second moment; MMD / HLA-stratified distance; motif witness; `centroid_atypicality`, multi-locus `fit_repertoire_spaces` (Theory §T.7) |
+| `mir.explain` | named-channel fusion (`ChannelBuilder`) + scorer-agnostic ablation (`channel_report`/`channel_drivers`) — which part of Φ carries the signal (§T.7) |
+| `mir.cohort` | the **digital donor** — multi-chain `fit_donor_embeddings`/`DonorCohort` (+ `transform`/`save`/`load`) + `residualize` / `cluster_samples` / `incidence_biomarkers` (§T.7) |
 | `mir.ml` | neural codecs (forward/inverse/Pgen/unified) + learned repertoire `set_encoder` (Set-Transformer/DeepRC) — Part 2, experimental; `[ml]` extra |
 
 ## Background subtraction & clustering (`mir.density`)
@@ -174,11 +205,12 @@ Use a **biological control** as the background when you have one (e.g. pre- vs p
 patient vs healthy) — differential enrichment cancels generic public convergence and isolates the
 antigen-specific response. With no control, `generate_background(locus, n)` samples the vdjtools
 P_gen model (the ALICE regime); the "water level" of a naive repertoire is handled by the
-empirical-null calibration. See `experiments/benchmark_density_{yfv,ankspond,tcrnet}.py`.
+empirical-null calibration. The density benchmarks (YFV, ankylosing-spondylitis B27, TCRNET)
+live in the companion [`2026-mirpy-analysis`](https://github.com/antigenomics) repo.
 
 At whole-repertoire scale, pass `neighbor_enrichment(..., backend="kdtree")` (exact scipy cKDTree,
 5–9× faster than the default BallTree) or `backend="ann"` (approximate pynndescent, ~30× faster
-past ~10⁵ clones, trading a small conservative undercount); see `experiments/benchmark_ann.py`.
+past ~10⁵ clones, trading a small conservative undercount; `pip install "mirpy-lib[ann]"`).
 
 ## Sample-level (repertoire) embedding (`mir.repertoire`)
 
@@ -213,16 +245,24 @@ a batch offset is first-order and cancels, while a batch-orthogonal signal (e.g.
 empirical rule of thumb — **diversity for how-even, the embedding for which-clones**: clone-size
 phenotypes (age, CMV) are a diversity summary's turf, while clonotype identity (HLA — strongest in
 TRA and class II) lives in the second moment / witness. A learned co-equal set encoder
-(Set-Transformer / DeepRC) is in `mir.ml.set_encoder` (`[ml]` extra). See
-`experiments/benchmark_repertoire_*.py`, `BENCHMARKS.md`, and `THEORY.md` T7.
+(Set-Transformer / DeepRC) is in `mir.ml.set_encoder` (`[ml]` extra). Recorded results and theory
+(T7) live in the companion [`2026-mirpy-analysis`](https://github.com/antigenomics) repo
+(`benchmarks/{BENCHMARKS,THEORY}.md`) alongside the benchmark scripts.
 
 ## Reproduce the paper
 
+The self-contained theory notebooks run on bundled data:
+
 ```bash
-python experiments/reproduce_supplementary.py   # supplementary S1–S3
-python experiments/benchmark_vdjdb.py           # Table S1 (needs a VDJdb dump)
+pip install "mirpy-lib[examples]"
+marimo edit examples/theory.py          # supplementary S1–S3 (distance laws, D↔d, prototype robustness)
+marimo edit examples/quickstart.py      # embed + cluster VDJdb antigens
 ```
 
+The full benchmark suite (VDJdb Table S1, density, repertoire/TCGA) and result docs live in the
+companion analysis repo [`2026-mirpy-analysis`](https://github.com/antigenomics) — this repo is the
+library + CI tests only.
+
 Method: Kremlyakova *et al.*, *TCREMP: a bioinformatic pipeline for efficient embedding of
 T-cell receptor sequences*, **J Mol Biol** 437 (2025) 169205.
 
@@ -236,12 +276,14 @@ mirpy is CPU-parallel by default and uses the GPU for the neural codecs. Knobs,
 | Density kNN / balloon | `neighbor_enrichment(..., backend=…)` | `"exact"` (BallTree, **1 core**) | `backend="kdtree"` = exact scipy cKDTree, **all cores** (`workers=-1`), 5–9× faster; `backend="ann"` = pynndescent, auto all-core, ~30× at ≥1e5. Prefer `kdtree` for multicore exact. |
 | Clustering | `cluster(..., n_jobs=-1)` | sklearn default (1) | forwarded to DBSCAN/OPTICS/HDBSCAN via `**kwargs`; parallelizes the neighbour search. |
 | BLAS (PCA, RFF, matmul) | `OMP_NUM_THREADS` / `OPENBLAS_NUM_THREADS` env | all cores | numpy/sklearn use the platform BLAS; cap via env if oversubscribed. |
-| Neural codecs (`mir.ml`) | `pick_device()` / `device=` / `MIR_DEVICE` env | **CUDA → MPS → CPU**, auto | every `train_*` / codec / bundle takes `device=`; e.g. `MIR_DEVICE=cuda:1 python experiments/train_forward_encoder.py`. Torch-free paths (`density`, `repertoire`) never touch the GPU. |
+| Neural codecs (`mir.ml`) | `pick_device()` / `device=` / `MIR_DEVICE` env | **CUDA → MPS → CPU**, auto | every `train_*` / codec / bundle takes `device=`; e.g. `MIR_DEVICE=cuda:1` pins the second GPU. Torch-free paths (`density`, `repertoire`) never touch the GPU. |
 
 Rule of thumb: leave `threads=0` (all cores) for embedding; switch density to `backend="kdtree"`
 for exact multicore or `"ann"` at whole-repertoire scale; the GPU is used only by `mir.ml`.
 
 ## Development
 
-Conda env `mirpy` (Python 3.12): `bash setup.sh`. Tests: `python -m pytest tests/ -q`.
-See [`CLAUDE.md`](CLAUDE.md) for the architecture and reuse map.
+Repo-local `.venv` via [uv](https://docs.astral.sh/uv/) (bash/zsh): `bash setup.sh` — add
+`--dev-parents` to editable-install the sibling `seqtree` / `vdjtools` / `vdjmatch` checkouts and
+`--tests` to run the fast suite. Tests: `python -m pytest tests/ -q`. See [`CLAUDE.md`](CLAUDE.md)
+for the architecture and reuse map.

{mirpy_lib-3.2.0 → mirpy_lib-3.4.0}/README.md

@@ -1,3 +1,13 @@
+<p align="center">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="assets/mirpy_dark.svg">
+    <source media="(prefers-color-scheme: light)" srcset="assets/mirpy_light.svg">
+    <!-- Absolute PNG fallback: PyPI strips <picture>/<source> and cannot render a relative or
+         raw-served SVG, so the logo must be an absolute-URL raster here. GitHub uses the SVG sources. -->
+    <img alt="mirpy" src="https://raw.githubusercontent.com/antigenomics/mirpy/master/assets/mirpy_light.png" width="360">
+  </picture>
+</p>
+
 <h1 align="center">mirpy — ML embeddings for immune repertoires</h1>
 
 [![PyPI](https://img.shields.io/pypi/v/mirpy-lib.svg)](https://pypi.org/project/mirpy-lib/)
@@ -7,7 +17,7 @@
 **mirpy v3** turns T-/B-cell receptor sequences into fixed-length numeric vectors you can
 cluster, visualize, and feed to ML models. It implements **TCREMP** — embedding each receptor
 by its alignment distances to a fixed set of *prototype* sequences — so that Euclidean distance
-in embedding space approximates pairwise alignment distance (see [`THEORY.md`](THEORY.md)).
+in embedding space approximates pairwise alignment distance (Theory T1).
 
 > v3 is a slim, embedding-focused rewrite. The classical repertoire toolkit (parsing, overlap,
 > diversity, TCRnet, GLIPH, …) lives on the **`legacy-v2`** branch (`mirpy-lib` 2.x) and in the
@@ -50,6 +60,24 @@ labels = cluster(pca_denoise(X, n_components=50))
 Paired chains concatenate per-chain embeddings via `PairedTCREmp`. Input/output are AIRR polars
 frames keyed by `vdjtools.io.schema` column names.
 
+## Command line
+
+`pip install mirpy-lib` also installs a `mir` command for the two embedding scales — no Python
+needed. Inputs are any format `vdjtools.io` reads (AIRR TSV, vdjtools, MiXCR, immunoSEQ, parquet).
+
+```bash
+# one repertoire  ->  per-clonotype embedding table (e0…), the input to clustering / ML
+mir embed clonotypes sample.tsv --pca 50 -o clonotypes.parquet
+
+# a dataset of repertoires  ->  one fingerprint Φ(S) per sample, per chain (phi0…), on one
+# shared basis so the rows are mutually comparable; --mmd also writes the pairwise MMD matrix
+mir embed repertoires cohort/*.tsv.gz -o phi.tsv --mmd mmd.tsv
+```
+
+`mir embed clonotypes -h` / `mir embed repertoires -h` list every flag (species, locus,
+prototype count, weight, Φ blocks, …). Sample id defaults to the filename stem; the locus is
+inferred per file (or restrict with `--locus`).
+
 ## Recommended presets
 
 `TCREmp.from_defaults(species, locus)` uses the per-chain preset when `n_prototypes` is
@@ -87,9 +115,11 @@ Xr = pca_denoise(X, n_components=p.n_components_recon)    # codec reconstruction
 | `mir.embedding.tcremp` | `TCREmp` / `PairedTCREmp` — the prototype embedding |
 | `mir.embedding.pca` | PCA denoising of embeddings |
 | `mir.distances` | junction distance (`seqtree.gapblock`; `metric`/`matrix`/`alignment` options) + baked germline distances |
-| `mir.bench` | VDJdb loader, clustering (`cluster(method=…)`: DBSCAN/HDBSCAN/OPTICS) + F1/retention, theory experiments (incl. `codec_losslessness`) |
+| `mir.bench` | VDJdb loader, clustering (`cluster(method=…)`: DBSCAN/HDBSCAN/OPTICS) + F1/retention, theory experiments (incl. `codec_losslessness`), cohort scorers (`bench.eval`: `cv_auc`/`cv_cindex`/`km_logrank`) |
 | `mir.density` | continuous-density TCRNET/ALICE — enrichment (+ clonal-abundance channel, `backend=` exact/kdtree/ann) + noise-filtering (Theory T6) |
-| `mir.repertoire` | sample-level (repertoire) embedding — RFF kernel mean ‖ Hill diversity ‖ second moment; MMD / HLA-stratified distance; motif witness (Theory §T.7) |
+| `mir.repertoire` | sample-level (repertoire) embedding — RFF kernel mean ‖ Hill diversity ‖ second moment; MMD / HLA-stratified distance; motif witness; `centroid_atypicality`, multi-locus `fit_repertoire_spaces` (Theory §T.7) |
+| `mir.explain` | named-channel fusion (`ChannelBuilder`) + scorer-agnostic ablation (`channel_report`/`channel_drivers`) — which part of Φ carries the signal (§T.7) |
+| `mir.cohort` | the **digital donor** — multi-chain `fit_donor_embeddings`/`DonorCohort` (+ `transform`/`save`/`load`) + `residualize` / `cluster_samples` / `incidence_biomarkers` (§T.7) |
 | `mir.ml` | neural codecs (forward/inverse/Pgen/unified) + learned repertoire `set_encoder` (Set-Transformer/DeepRC) — Part 2, experimental; `[ml]` extra |
 
 ## Background subtraction & clustering (`mir.density`)
@@ -116,11 +146,12 @@ Use a **biological control** as the background when you have one (e.g. pre- vs p
 patient vs healthy) — differential enrichment cancels generic public convergence and isolates the
 antigen-specific response. With no control, `generate_background(locus, n)` samples the vdjtools
 P_gen model (the ALICE regime); the "water level" of a naive repertoire is handled by the
-empirical-null calibration. See `experiments/benchmark_density_{yfv,ankspond,tcrnet}.py`.
+empirical-null calibration. The density benchmarks (YFV, ankylosing-spondylitis B27, TCRNET)
+live in the companion [`2026-mirpy-analysis`](https://github.com/antigenomics) repo.
 
 At whole-repertoire scale, pass `neighbor_enrichment(..., backend="kdtree")` (exact scipy cKDTree,
 5–9× faster than the default BallTree) or `backend="ann"` (approximate pynndescent, ~30× faster
-past ~10⁵ clones, trading a small conservative undercount); see `experiments/benchmark_ann.py`.
+past ~10⁵ clones, trading a small conservative undercount; `pip install "mirpy-lib[ann]"`).
 
 ## Sample-level (repertoire) embedding (`mir.repertoire`)
 
@@ -155,16 +186,24 @@ a batch offset is first-order and cancels, while a batch-orthogonal signal (e.g.
 empirical rule of thumb — **diversity for how-even, the embedding for which-clones**: clone-size
 phenotypes (age, CMV) are a diversity summary's turf, while clonotype identity (HLA — strongest in
 TRA and class II) lives in the second moment / witness. A learned co-equal set encoder
-(Set-Transformer / DeepRC) is in `mir.ml.set_encoder` (`[ml]` extra). See
-`experiments/benchmark_repertoire_*.py`, `BENCHMARKS.md`, and `THEORY.md` T7.
+(Set-Transformer / DeepRC) is in `mir.ml.set_encoder` (`[ml]` extra). Recorded results and theory
+(T7) live in the companion [`2026-mirpy-analysis`](https://github.com/antigenomics) repo
+(`benchmarks/{BENCHMARKS,THEORY}.md`) alongside the benchmark scripts.
 
 ## Reproduce the paper
 
+The self-contained theory notebooks run on bundled data:
+
 ```bash
-python experiments/reproduce_supplementary.py   # supplementary S1–S3
-python experiments/benchmark_vdjdb.py           # Table S1 (needs a VDJdb dump)
+pip install "mirpy-lib[examples]"
+marimo edit examples/theory.py          # supplementary S1–S3 (distance laws, D↔d, prototype robustness)
+marimo edit examples/quickstart.py      # embed + cluster VDJdb antigens
 ```
 
+The full benchmark suite (VDJdb Table S1, density, repertoire/TCGA) and result docs live in the
+companion analysis repo [`2026-mirpy-analysis`](https://github.com/antigenomics) — this repo is the
+library + CI tests only.
+
 Method: Kremlyakova *et al.*, *TCREMP: a bioinformatic pipeline for efficient embedding of
 T-cell receptor sequences*, **J Mol Biol** 437 (2025) 169205.
 
@@ -178,12 +217,14 @@ mirpy is CPU-parallel by default and uses the GPU for the neural codecs. Knobs,
 | Density kNN / balloon | `neighbor_enrichment(..., backend=…)` | `"exact"` (BallTree, **1 core**) | `backend="kdtree"` = exact scipy cKDTree, **all cores** (`workers=-1`), 5–9× faster; `backend="ann"` = pynndescent, auto all-core, ~30× at ≥1e5. Prefer `kdtree` for multicore exact. |
 | Clustering | `cluster(..., n_jobs=-1)` | sklearn default (1) | forwarded to DBSCAN/OPTICS/HDBSCAN via `**kwargs`; parallelizes the neighbour search. |
 | BLAS (PCA, RFF, matmul) | `OMP_NUM_THREADS` / `OPENBLAS_NUM_THREADS` env | all cores | numpy/sklearn use the platform BLAS; cap via env if oversubscribed. |
-| Neural codecs (`mir.ml`) | `pick_device()` / `device=` / `MIR_DEVICE` env | **CUDA → MPS → CPU**, auto | every `train_*` / codec / bundle takes `device=`; e.g. `MIR_DEVICE=cuda:1 python experiments/train_forward_encoder.py`. Torch-free paths (`density`, `repertoire`) never touch the GPU. |
+| Neural codecs (`mir.ml`) | `pick_device()` / `device=` / `MIR_DEVICE` env | **CUDA → MPS → CPU**, auto | every `train_*` / codec / bundle takes `device=`; e.g. `MIR_DEVICE=cuda:1` pins the second GPU. Torch-free paths (`density`, `repertoire`) never touch the GPU. |
 
 Rule of thumb: leave `threads=0` (all cores) for embedding; switch density to `backend="kdtree"`
 for exact multicore or `"ann"` at whole-repertoire scale; the GPU is used only by `mir.ml`.
 
 ## Development
 
-Conda env `mirpy` (Python 3.12): `bash setup.sh`. Tests: `python -m pytest tests/ -q`.
-See [`CLAUDE.md`](CLAUDE.md) for the architecture and reuse map.
+Repo-local `.venv` via [uv](https://docs.astral.sh/uv/) (bash/zsh): `bash setup.sh` — add
+`--dev-parents` to editable-install the sibling `seqtree` / `vdjtools` / `vdjmatch` checkouts and
+`--tests` to run the fast suite. Tests: `python -m pytest tests/ -q`. See [`CLAUDE.md`](CLAUDE.md)
+for the architecture and reuse map.

{mirpy_lib-3.2.0 → mirpy_lib-3.4.0}/SOURCES.md

@@ -17,7 +17,29 @@ Every dataset used by mirpy v3, its origin, and how to regenerate it.
 |---|---|---|---|
 | VDJdb slim dump | `tests/assets/vdjdb.slim.txt.gz` | VDJdb (Goncharov et al., *Nat Methods* 2022) export, dotted-column slim format | Antigen labels for the Table S1 benchmark; a specific release — F1 differs from the paper's 2023 summer release |
 | Epitope-specific TRB sets | `tests/assets/{gilgfvftl,llwngpmav}_*.gz` | VDJdb subsets | Single-antigen fixtures |
-| OLGA TRB sample | `tests/assets/olga_humanTRB_1000.txt.gz` | OLGA-generated human TRB | Synthetic repertoire sample |
+| OLGA TRB sample | `tests/assets/olga_humanTRB_1000.txt.gz` | **Computed**, not experimental: 1 000 human TRB rearrangements drawn from the OLGA generative model (Sethna et al., *Bioinformatics* 2019), seed 42. Cols (no header): `junction_nt, junction_aa, v_call, j_call`; kept in **generation order — do not sort** (see note below) | `conda run -n mirpy --no-capture-output olga-generate_sequences --humanTRB -n 1000 -o /tmp/olga.tsv --seed 42` then `awk -F'\t' -v OFS='\t' '{print $1,$2,$3"*01",$4"*01"}' /tmp/olga.tsv \| gzip -n > tests/assets/olga_humanTRB_1000.txt.gz` (the `*01` suffix is appended because OLGA emits gene-level calls but `mir` expects allele-level, matching the bundled prototypes) |
+
+> **Provenance correction (2026-07-17).** Until this date `olga_humanTRB_1000.txt.gz` was **not**
+> OLGA output despite its name and this row's claim. It was a `head`-slice of the *alphabetically
+> sorted* VDJdb TRB dump: all 969 unique junctions matched `vdjdb.slim.txt.gz` TRB `cdr3` exactly
+> (100%, vs **0.96%** for genuine OLGA), the `junction_nt` column was a placeholder run of `N`, and
+> the first row (`ATSIRFTDTQYF`) lacked the Cys anchor OLGA always emits. Consequences of the old
+> file, for anyone re-reading results predating this fix:
+> 1. **Not an antigen-naive null.** 12% of its rows had a Hamming-1 neighbour within the file vs
+>    **0.2%** for real OLGA (166 vs 4 pairs per 1 000) — VDJdb is *selected for* the antigen-driven
+>    convergence such a background is supposed to lack. Any chance-rate / null calibration against it
+>    compared VDJdb to itself and is meaningless.
+> 2. **Sorted ⇒ slices are not exchangeable.** Because it was alphabetically sorted, the
+>    `_load_olga(n)` / `_load_olga(n, offset=400)` obs-vs-background slice pattern in
+>    `tests/test_density.py` drew two systematically different regions of sequence space (the
+>    background block was 58% `CAIS`-prefixed and shared only 20 of 54 V genes with obs). Genuine
+>    OLGA output is in generation order, which is why the regenerate command above must **not** sort.
+>
+> The `tests/` suite was **not** invalidated: all 58 tests pass identically on both files (they use
+> the asset only as a generic pool of TRB junctions, never as a null), and the Hamming-1 gate in
+> `test_continuous_matches_discrete_hamming1` scores rho 0.727 on the old file vs 0.809 on real OLGA.
+> The damage was confined to external calibrations that treated the file as a synthetic negative
+> control.
 
 Bundled model / Pgen data used at runtime lives in **vdjtools** (`vdjtools.model` bundled
 parquet marginals for 7 loci × {olga, learned}) — see the vdjtools `SOURCES.md`.

mirpy_lib-3.4.0/docs/_static/custom.css

@@ -0,0 +1,9 @@
+/* Figures with dark ink on a transparent background (schematics, plots exported from LaTeX/gnuplot)
+   stay legible in dark mode by rendering on a white card. Scoped to `.mir-fig` so the theme-aware
+   logo (only-light / only-dark) is untouched. */
+img.mir-fig {
+  background: #ffffff;
+  padding: 10px 14px;
+  border-radius: 8px;
+  max-width: 100%;
+}