mirpy-lib 3.2.0__tar.gz → 3.3.0__tar.gz

{mirpy_lib-3.2.0 → mirpy_lib-3.3.0}/CHANGELOG.md

@@ -3,6 +3,38 @@
 All notable changes to `mirpy-lib` (import `mir`). This project follows semantic versioning; the v3 line is a
 greenfield ML/embedding rewrite (the classical v1.x/v2 toolkit is frozen on branch `legacy-v2`).
 
+## 3.3.0 — 2026-07-17
+
+Minor: one new public parameter, nothing removed or changed.
+
+### Added
+
+- **`fit_density_space(chunk_size=)`** — embed and project in batches so the full raw matrix is never
+  resident. Peak memory becomes `max(pca_fit_cap, chunk_size) × n_features` instead of
+  `len(df) × n_features`: measured **10.60 GB → 1.81 GB** at 450k pooled clonotypes, and flat in `N`
+  (vs linear), at no wall-clock cost. This is what makes whole-cohort density arms runnable on a
+  laptop — the 4.2M-clonotype pooled arm is ~51 GB raw and ~102 GB once `scaler.transform` upcasts to
+  float64. Chunking is bit-exact at the embedding level (`_embed` of a slice == the slice of
+  `_embed`); the projected coordinates agree to float noise (~1e-7 relative), since BLAS summation
+  order depends on batch shape.
+
+### Fixed
+
+- `fit_density_space`'s `pca_fit_cap` docstring claimed it "lets whole repertoires be embedded without
+  a full-matrix PCA". It caps the **fit**, not the memory — both raw matrices were already
+  materialized before the PCA was fitted. Documented, and `chunk_size=` is the actual remedy.
+- **`mir.__version__` was stale** — it read `3.1.1` on the published 3.2.0, because the release bump
+  moved `pyproject.toml` but not `mir/__init__.py`, and `publish.yml` only validates *pyproject ==
+  tag*. Both now read 3.3.0. (`__version__` is still hand-maintained; deriving it from
+  `importlib.metadata` would retire this failure mode for good.)
+- `tests/assets/olga_humanTRB_1000.txt.gz` was a slice of the alphabetically sorted VDJdb TRB dump,
+  not OLGA output as its name and `SOURCES.md` claimed — so it was **not** an antigen-naive null (12%
+  of rows had a Hamming-1 neighbour vs 0.2% for real OLGA). Regenerated from `olga-generate_sequences`;
+  provenance and a byte-reproducible regenerate command recorded in `SOURCES.md`. No test was
+  invalidated (they use it only as a generic TRB junction pool), but external calibrations that treated
+  it as a synthetic negative control were comparing VDJdb against itself. Not shipped (tests are
+  excluded from the sdist); listed here because it invalidates results, not code.
+
 ## 3.2.0 — 2026-07-17
 
 Minor: one new public module, nothing removed or changed.

{mirpy_lib-3.2.0 → mirpy_lib-3.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mirpy-lib
-Version: 3.2.0
+Version: 3.3.0
 Summary: ML-oriented embeddings for immune receptor repertoires (TCR/BCR)
 Project-URL: Homepage, https://github.com/antigenomics/mirpy
 Project-URL: Repository, https://github.com/antigenomics/mirpy

{mirpy_lib-3.2.0 → mirpy_lib-3.3.0}/SOURCES.md

@@ -17,7 +17,29 @@ Every dataset used by mirpy v3, its origin, and how to regenerate it.
 |---|---|---|---|
 | VDJdb slim dump | `tests/assets/vdjdb.slim.txt.gz` | VDJdb (Goncharov et al., *Nat Methods* 2022) export, dotted-column slim format | Antigen labels for the Table S1 benchmark; a specific release — F1 differs from the paper's 2023 summer release |
 | Epitope-specific TRB sets | `tests/assets/{gilgfvftl,llwngpmav}_*.gz` | VDJdb subsets | Single-antigen fixtures |
-| OLGA TRB sample | `tests/assets/olga_humanTRB_1000.txt.gz` | OLGA-generated human TRB | Synthetic repertoire sample |
+| OLGA TRB sample | `tests/assets/olga_humanTRB_1000.txt.gz` | **Computed**, not experimental: 1 000 human TRB rearrangements drawn from the OLGA generative model (Sethna et al., *Bioinformatics* 2019), seed 42. Cols (no header): `junction_nt, junction_aa, v_call, j_call`; kept in **generation order — do not sort** (see note below) | `conda run -n mirpy --no-capture-output olga-generate_sequences --humanTRB -n 1000 -o /tmp/olga.tsv --seed 42` then `awk -F'\t' -v OFS='\t' '{print $1,$2,$3"*01",$4"*01"}' /tmp/olga.tsv \| gzip -n > tests/assets/olga_humanTRB_1000.txt.gz` (the `*01` suffix is appended because OLGA emits gene-level calls but `mir` expects allele-level, matching the bundled prototypes) |
+
+> **Provenance correction (2026-07-17).** Until this date `olga_humanTRB_1000.txt.gz` was **not**
+> OLGA output despite its name and this row's claim. It was a `head`-slice of the *alphabetically
+> sorted* VDJdb TRB dump: all 969 unique junctions matched `vdjdb.slim.txt.gz` TRB `cdr3` exactly
+> (100%, vs **0.96%** for genuine OLGA), the `junction_nt` column was a placeholder run of `N`, and
+> the first row (`ATSIRFTDTQYF`) lacked the Cys anchor OLGA always emits. Consequences of the old
+> file, for anyone re-reading results predating this fix:
+> 1. **Not an antigen-naive null.** 12% of its rows had a Hamming-1 neighbour within the file vs
+>    **0.2%** for real OLGA (166 vs 4 pairs per 1 000) — VDJdb is *selected for* the antigen-driven
+>    convergence such a background is supposed to lack. Any chance-rate / null calibration against it
+>    compared VDJdb to itself and is meaningless.
+> 2. **Sorted ⇒ slices are not exchangeable.** Because it was alphabetically sorted, the
+>    `_load_olga(n)` / `_load_olga(n, offset=400)` obs-vs-background slice pattern in
+>    `tests/test_density.py` drew two systematically different regions of sequence space (the
+>    background block was 58% `CAIS`-prefixed and shared only 20 of 54 V genes with obs). Genuine
+>    OLGA output is in generation order, which is why the regenerate command above must **not** sort.
+>
+> The `tests/` suite was **not** invalidated: all 58 tests pass identically on both files (they use
+> the asset only as a generic pool of TRB junctions, never as a null), and the Hamming-1 gate in
+> `test_continuous_matches_discrete_hamming1` scores rho 0.727 on the old file vs 0.809 on real OLGA.
+> The damage was confined to external calibrations that treated the file as a synthetic negative
+> control.
 
 Bundled model / Pgen data used at runtime lives in **vdjtools** (`vdjtools.model` bundled
 parquet marginals for 7 loci × {olga, learned}) — see the vdjtools `SOURCES.md`.

{mirpy_lib-3.2.0 → mirpy_lib-3.3.0}/mir/__init__.py

@@ -15,7 +15,7 @@ from __future__ import annotations
 
 import os
 
-__version__ = "3.1.1"
+__version__ = "3.3.0"
 
 __all__ = ["__version__", "get_resource_path", "TCREmp", "PairedTCREmp"]
 

{mirpy_lib-3.2.0 → mirpy_lib-3.3.0}/mir/density.py

@@ -99,6 +99,24 @@ def _embed(model, df: pl.DataFrame, space: str) -> np.ndarray:
     return _slice(model.embed(df), space).astype(np.float32, copy=False)
 
 
+def _embed_transform_chunked(model, df: pl.DataFrame, space: str, scaler, pca,
+                             chunk_size: int) -> np.ndarray:
+    """Embed *df* and project it into ``scaler``+``pca``, ``chunk_size`` rows at a time.
+
+    Only ``chunk_size × n_features`` of raw embedding is ever resident: each chunk is embedded,
+    standardized, projected down to ``n_components``, and dropped. Peak memory is set by the chunk,
+    not by ``len(df)`` — which is what makes whole-cohort clouds tractable (a 4.2M-clonotype arm is
+    ~51 GB raw, and ~102 GB again once ``scaler.transform`` upcasts it to float64).
+    """
+    out = np.empty((df.height, pca.n_components_), dtype=np.float64)
+    for start in range(0, df.height, chunk_size):
+        stop = min(start + chunk_size, df.height)
+        raw = _embed(model, df.slice(start, stop - start), space)
+        out[start:stop] = pca.transform(scaler.transform(raw))
+        del raw
+    return out
+
+
 @dataclass
 class DensitySpace:
     """A single fitted TCREMP → PCA coordinate system for density comparison.
@@ -158,6 +176,7 @@ def fit_density_space(
     space: str = "full",
     seed: int = 0,
     pca_fit_cap: int | None = None,
+    chunk_size: int | None = None,
 ) -> tuple[DensitySpace, np.ndarray, np.ndarray]:
     """Embed observed + background frames into one shared PCA coordinate system.
 
@@ -176,13 +195,27 @@ def fit_density_space(
             third of the memory at whole-repertoire scale).
         seed: RNG / PCA solver seed.
         pca_fit_cap: Fit the ``StandardScaler`` + PCA on at most this many randomly-sampled
-            pooled rows (then transform *all* rows). Lets whole repertoires be embedded
-            without a full-matrix PCA; ``None`` fits on everything.
+            pooled rows (then transform *all* rows). ``None`` fits on everything. NB on its own
+            this caps the **fit**, not the memory: without ``chunk_size`` the full raw matrix for
+            both frames is materialized before the PCA is fitted at all.
+        chunk_size: Embed and project in batches of this many rows, so the full raw matrix is
+            never resident (peak ≈ ``chunk_size × n_features`` instead of ``len(df) × n_features``).
+            Required for whole-cohort clouds: 4.2M clonotypes × 1000 prototypes is ~51 GB raw and
+            ~102 GB after ``scaler.transform`` upcasts to float64, versus ~2.4 GB at
+            ``chunk_size=200_000``. When set and ``pca_fit_cap`` is ``None``, the fit is capped at
+            200k pooled rows — fitting on everything would defeat the purpose. ``None`` (default)
+            keeps the original single-shot path exactly.
 
     Returns:
         ``(density_space, obs_emb, bg_emb)`` — the fitted :class:`DensitySpace` and the
         two reduced ``float`` arrays, row-aligned to ``obs_df`` / ``bg_df``.
     """
+    if chunk_size is not None:
+        return _fit_density_space_chunked(
+            model, obs_df, bg_df, n_components=n_components, space=space, seed=seed,
+            pca_fit_cap=pca_fit_cap if pca_fit_cap is not None else 200_000,
+            chunk_size=chunk_size,
+        )
     obs = _embed(model, obs_df, space)
     bg = _embed(model, bg_df, space)
     n_total, n_feat = obs.shape[0] + bg.shape[0], obs.shape[1]
@@ -203,6 +236,42 @@ def fit_density_space(
     return ds, pca.transform(scaler.transform(obs)), pca.transform(scaler.transform(bg))
 
 
+def _fit_density_space_chunked(
+    model, obs_df: pl.DataFrame, bg_df: pl.DataFrame, *, n_components: int, space: str,
+    seed: int, pca_fit_cap: int, chunk_size: int,
+) -> tuple[DensitySpace, np.ndarray, np.ndarray]:
+    """``fit_density_space`` without ever holding a full raw matrix.
+
+    Same coordinate system as the single-shot path — one scaler + one PCA fitted on a pooled
+    sample, applied to both frames — but the fit sample is drawn as *frame rows* and embedded on
+    its own, and each frame is then embedded/projected in chunks. Memory is set by
+    ``max(pca_fit_cap, chunk_size)``, not by the cohort.
+    """
+    rng = np.random.default_rng(seed)
+    n_o, n_b = obs_df.height, bg_df.height
+    n_total = n_o + n_b
+
+    if n_total > pca_fit_cap:
+        take_o = min(n_o, pca_fit_cap * n_o // n_total)
+        take_b = min(n_b, pca_fit_cap - take_o)
+        fit_rows = np.vstack([
+            _embed(model, obs_df[rng.choice(n_o, take_o, replace=False)], space),
+            _embed(model, bg_df[rng.choice(n_b, take_b, replace=False)], space),
+        ])
+    else:
+        fit_rows = np.vstack([_embed(model, obs_df, space), _embed(model, bg_df, space)])
+
+    scaler = StandardScaler().fit(fit_rows)
+    k = min(n_components, fit_rows.shape[0], fit_rows.shape[1])
+    pca = PCA(n_components=k, random_state=seed).fit(scaler.transform(fit_rows))
+    del fit_rows
+
+    ds = DensitySpace(model=model, space=space, scaler=scaler, pca=pca)
+    return (ds,
+            _embed_transform_chunked(model, obs_df, space, scaler, pca, chunk_size),
+            _embed_transform_chunked(model, bg_df, space, scaler, pca, chunk_size))
+
+
 def _mutate1(seq: str, rng) -> str:
     """Apply one interior amino-acid substitution to a *different* residue."""
     if len(seq) <= 2:

{mirpy_lib-3.2.0 → mirpy_lib-3.3.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "mirpy-lib"
-version = "3.2.0"
+version = "3.3.0"
 authors = [
   { name="ISALGO lab", email="mikhail.shugay@gmail.com"}
 ]