mirpy-lib 4.3.0__tar.gz → 4.4.0__tar.gz

{mirpy_lib-4.3.0 → mirpy_lib-4.4.0}/CHANGELOG.md

@@ -3,6 +3,99 @@
 All notable changes to `mirpy-lib` (import `mir`). This project follows semantic versioning; the v3 line is a
 greenfield ML/embedding rewrite (the classical v1.x/v2 toolkit is frozen on branch `legacy-v2`).
 
+## 4.4.0 — 2026-09-29
+
+Every module read for one pattern: Python iterating over *data* where a batched call, a polars
+expression or a table gather does the same work. Requires vdjtools >= 4.6.0, which is a **hard
+floor** — `strip_allele_values` does not exist before it.
+
+mirpy is pure Python by design (hatchling, no extension), so "make it native" here means reaching
+into seqtree's or vdjtools' C++ rather than adding a build. Both fixes below did the opposite of
+what that suggests, and the measurements say why.
+
+### Fixed: `shm_penalty_batch` said "vectorised" and was a per-row loop
+
+It called the scalar `shm_penalty` once per clonotype, which re-parsed the mutation string and
+made **one seqtree `penalty` call per substitution** — so a BCR repertoire of 10^5 clonotypes
+carrying ~15 mutations each made ~1.5 million calls to answer 676 distinct questions. It sits on
+`TCREmp.embed`, so it ran for every frame carrying `v_mutations` or `v_identity`.
+
+The parse is one polars pass and the penalties one gather from a `(26, 26)` table built once:
+
+| cohort | was | now | gain |
+|---|---|---|---|
+| 10,000 x 15 mutations | 32.7 ms | 10.0 ms | 3.3x |
+| 100,000 x 15 mutations | 324.3 ms | 83.2 ms | **3.9x** |
+| 200,000 x 18 mutations | 748.2 ms | 210.0 ms | 3.6x |
+| 200,000, identity only | 87.2 ms | 2.0 ms | **43.4x** |
+
+`shm_penalty` stays the scalar reference and `tests/test_shm.py` asserts the two agree with `==`,
+not a tolerance: every BLOSUM62 penalty is a small integer, so a float64 sum of them is exact
+however it is ordered, and a tolerance would hide a mis-indexed table — the one bug this rewrite
+could plausibly introduce. **This module had no tests at all**, which is the other half of why the
+docstring and the code disagreed for as long as they did.
+
+Three things the rewrite had to keep, each now pinned:
+
+- **J, O and U are letters and are not in BLOSUM62.** The scalar path raised because seqtree
+  rejected the pair; a lookup table with a silent default would return a finite, plausible penalty
+  instead. `nan` marks them and both paths raise together.
+- **A non-letter token is dropped in silence** (`23V`, `A23`, `AB`) because `parse_mutations`
+  requires the first and last characters to be letters. Telling that case from the one above is
+  the only reason the alpha check survives — and it now runs on the **null rows alone**, which is
+  worth 31.3 ms of a 113.5 ms pass, because a well-formed spec has none.
+- **Truthiness of the raw value, never of its string form.** The scalar path branches on
+  `if mutations:`, so an int `0` takes the identity path while `3` does not. Casting to string
+  first would make `"0"` truthy and score that row from the wrong branch — plausibly, and with
+  nothing about the number to show it had.
+
+### Fixed: the isotype channels spent their time stripping allele suffixes, not matching bands
+
+`isotype_masks` and `isotype_shares` each materialised the whole allele-stripped `c_call` column
+into a Python list and ran one `in` test per clonotype per band. **The membership tests were not
+the cost.** Profiled on 200,000 rows: `strip_allele` 100.7 ms, the three `is_in` tests 3.7 ms, the
+`to_list()` 5.0 ms, the Python `in` tests 25.9 ms. Replacing only the comprehensions moved
+132.2 ms to 134.8 — nothing, which is how the real cause surfaced.
+
+`strip_allele` is a polars *expression*, so it resolves the allele suffix once per row, and a
+`c_call` column carries a few dozen distinct values however deep the repertoire is. vdjtools 4.6.0
+adds `strip_allele_values`, which resolves it once per distinct call; both functions now share one
+pass through it:
+
+| clonotypes | `isotype_masks` | `isotype_shares` |
+|---|---|---|
+| 10,000 | 8.63 -> 1.20 ms (7.2x) | 7.66 -> 1.32 ms (5.8x) |
+| 50,000 | 30.85 -> 2.29 ms (13.5x) | 32.80 -> 2.88 ms (11.4x) |
+| 200,000 | 132.23 -> 6.10 ms (**21.7x**) | 124.89 -> 8.39 ms (**14.9x**) |
+
+Output is identical on plain calls, nulls, comma ambiguity, surrounding whitespace, a single
+occupied band, a band under the floor, and a frame with no `c_call`. `isotype_shares` takes its
+masks from `isotype_masks` rather than from a second copy of the same comprehension — two copies
+is how the two came to be able to disagree about a null.
+
+### Measured and deliberately left alone, with the numbers
+
+So nobody re-derives this. Every other module iterates over **structure** — loci, samples,
+donors, prototypes, PCs, channels, columns, epochs, minibatches — which is not the same thing as
+iterating over data, and each step is a real numpy or torch operation.
+
+| site | measured | why it stays |
+|---|---|---|
+| `signature.features.dispersion_pass` | chunked running sums, `(n, 3K)` never held | already the target shape: BLAS inside, linear in depth |
+| `embedding.TCREmp.embed` | preallocated output, one batched call per block | polars-vectorised validation, seqtree threads released |
+| `distances.germline.matrix` | resolves distinct alleles, `proto_idx` hoisted | previously audited; `np.unique` measured at 1.12x and rejected |
+| `density` neighbour lists | flatten + one `bincount` | the ragged lists are scipy/pynndescent's return shape |
+| `bench.metrics` purity map | 20.7 ms at 200,000 | one dict lookup per clonotype, once per benchmark |
+| `bench.eval` kmer matrix fill | 467.9 ms at 900 samples x 8,000 kmers | over the **vocabulary**, and the 900 `kmer_profile` calls dominate |
+| `bench.theory._hamming1_counts` | length-blocked around a batched `hamming_matrix` | already audited, and carries its own ceiling note |
+
+**Two paths iterate per pair because no batch exists upstream**, not because nobody looked:
+`distances.junction`'s `alignment="sw"` backend and `bench.theory.junction_dissimilarity_sw` both
+call BioPython's `PairwiseAligner` per pair, which has no batched API. Both are opt-in validation
+paths against the paper's exact Smith-Waterman; the default `gapblock` backend is one seqtree call
+with the GIL released. `resources/*/build_*.py` keep their `iter_rows` and their O(n^2) alignments
+because they are offline artifact builders run once.
+
 ## 4.3.0 — 2026-09-28
 
 The geometry half emits embedding **diversity**, not only Rao — and stops emitting a presence mask

{mirpy_lib-4.3.0 → mirpy_lib-4.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: mirpy-lib
-Version: 4.3.0
+Version: 4.4.0
 Summary: ML-oriented embeddings for immune receptor repertoires (TCR/BCR)
 Project-URL: Homepage, https://github.com/antigenomics/mirpy
 Project-URL: Repository, https://github.com/antigenomics/mirpy
@@ -24,7 +24,7 @@ Requires-Dist: polars
 Requires-Dist: scikit-learn
 Requires-Dist: scipy
 Requires-Dist: seqtree>=1.0.0
-Requires-Dist: vdjtools>=4.3.0
+Requires-Dist: vdjtools>=4.6.0
 Provides-Extra: ann
 Requires-Dist: numba>=0.60; extra == 'ann'
 Requires-Dist: pynndescent; extra == 'ann'
@@ -38,7 +38,7 @@ Requires-Dist: matplotlib; extra == 'bench'
 Requires-Dist: seaborn; extra == 'bench'
 Provides-Extra: build
 Requires-Dist: biopython; extra == 'build'
-Requires-Dist: vdjtools[model]>=4.2.0; extra == 'build'
+Requires-Dist: vdjtools[model]>=4.6.0; extra == 'build'
 Provides-Extra: dev
 Requires-Dist: psutil>=5; extra == 'dev'
 Requires-Dist: pytest-cov; extra == 'dev'

{mirpy_lib-4.3.0 → mirpy_lib-4.4.0}/docs/usage.rst

@@ -16,7 +16,7 @@ Install & the data model
    pip install "mirpy-lib[ml]"      # + neural codecs (torch)
    pip install "mirpy-lib[examples]"  # + marimo notebooks (marimo, matplotlib, umap-learn)
 
-Requires ``vdjtools>=4.3.0`` and ``seqtree>=1.0.0``, both installed for you; ``mir`` itself is a
+Requires ``vdjtools>=4.6.0`` and ``seqtree>=1.0.0``, both installed for you; ``mir`` itself is a
 pure-Python ``py3-none-any`` wheel. A clonotype table is a polars frame; clone sizes live in
 ``duplicate_count`` (repertoire-level embeddings weight by them):
 

{mirpy_lib-4.3.0 → mirpy_lib-4.4.0}/pyproject.toml

@@ -36,7 +36,7 @@ dependencies = [
   # `vdjtools.signature.corpus`, which does not exist before 4.0.0, and 4.0.0 never
   # reached PyPI. On the old `>=3.18.0` pip would have resolved vdjtools 3.18.1 beside
   # mirpy 4.0.0 and every signature entry point would have failed on import.
-  "vdjtools>=4.3.0",
+  "vdjtools>=4.6.0",
 ]
 license = { text = "GPL-3.0-or-later" }
 
@@ -46,7 +46,7 @@ annotate = ["vdjmatch"]
 # Build-time only: regenerate region_annotations.txt (arda) and the baked
 # germline distance matrices (BioPython). Runtime needs neither. arda comes
 # via vdjtools[model] (single arda version across the ecosystem), not pinned directly.
-build = ["vdjtools[model]>=4.2.0", "biopython"]
+build = ["vdjtools[model]>=4.6.0", "biopython"]
 # Part 2: neural codecs and density methods.
 ml = ["torch"]
 # Benchmark harness: DBSCAN eps selection + plotting + HuggingFace dataset fetch +
@@ -95,7 +95,17 @@ exclude = [
   "tests/**",
   "tmp/**",
   "*.log",
+  # Working notes: internal by design (cluster paths, private dataset names, cohort detail).
+  # .gitignore is what actually keeps them out -- hatchling builds from the VCS file list -- and
+  # these are the second line, for a tree where an untracked copy is sitting on disk.
   "SOURCES.md",
+  "CLAUDE.md",
+  "TODO.md",
+  "NOTES.md",
+  "ROADMAP.md",
+  "STATUS.md",
+  "ISSUES.md",
+  "ISSUES_ext.md",
 ]
 
 [tool.ruff]

{mirpy_lib-4.3.0 → mirpy_lib-4.4.0}/skills/mirpy/SKILL.md

@@ -149,6 +149,8 @@ with phenotype.
 
 ### `mir.signature` — the portable signature, geometry half
 
+**4.4.0 — the per-item-loop audit.** No API change. `shm_penalty_batch` was a per-row loop behind a docstring that said "vectorised" (3.3x-3.9x on the mutation path, 43.4x on the identity path), and the isotype channels spent 76% of their time in `strip_allele` rather than matching bands (`isotype_masks` 21.7x, `isotype_shares` 14.9x at 200,000 clonotypes). Requires **vdjtools >= 4.6.0** as a hard floor: `strip_allele_values` does not exist before it. Measurements and the seven sites deliberately left alone are in CLAUDE.md.
+
 **4.3.0 — the `div`/`disp`/`mask` channel families.** `rsig` emits embedding diversity, not only
 Rao: `q_v`/`q_j`/`q_c` (Rao per `Phi` stride — **`[0::3]`, `[1::3]`, `[2::3]`, never `[0:K]` blocks**;
 reading them as blocks is the likeliest single bug here and produces plausible numbers, caught only

{mirpy_lib-4.3.0 → mirpy_lib-4.4.0}/src/mir/__init__.py

@@ -15,7 +15,7 @@ from __future__ import annotations
 
 import os
 
-__version__ = "4.3.0"
+__version__ = "4.4.0"
 
 __all__ = ["__version__", "get_resource_path", "TCREmp", "PairedTCREmp"]
 

{mirpy_lib-4.3.0 → mirpy_lib-4.4.0}/src/mir/distances/shm.py

@@ -61,6 +61,7 @@ from dataclasses import dataclass
 from functools import lru_cache
 
 import numpy as np
+import polars as pl
 
 #: Typical IMGT V-region length in amino acids, used to turn a scalar identity into a mutation
 #: count. Real V regions run ~95-100 aa; the constant only sets the scale of the scalar path and
@@ -154,6 +155,33 @@ def shm_penalty(mutations: str | None = None, *, identity: float | None = None,
     return float(max(1.0 - identity, 0.0) * v_length * lam)
 
 
+@lru_cache(maxsize=1)
+def _penalty_table() -> np.ndarray:
+    """The whole BLOSUM62 penalty domain as a ``(26, 26)`` table, indexed by ``ord(aa) - 65``.
+
+    :func:`substitution_penalty` is a seqtree call per substitution, and the batch path made one
+    per mutation per clonotype: a real BCR repertoire carries ~15 mutations on each of ~10^5
+    clonotypes, so ~1.5 million calls to answer 676 distinct questions. ``nan`` marks the
+    upper-case codes BLOSUM62 does not define (J, O and U, and no others), so an unsupported
+    residue stays *detectable* rather than silently scoring 0 — the scalar path raises on
+    those, so this one must too.
+    """
+    m = _matrix()
+    t = np.full((26, 26), np.nan)
+    for i in range(26):
+        for j in range(26):
+            try:
+                t[i, j] = m.penalty(chr(65 + i), chr(65 + j))
+            except ValueError:
+                pass                                # BLOSUM62 does not define this pair
+    return t
+
+
+#: Upper-case residue to its row in :func:`_penalty_table`, built from the table's own index so
+#: the two cannot disagree about what is defined.
+_AA_INDEX = {chr(65 + i): i for i in range(26)}
+
+
 def shm_penalty_batch(mutations=None, identity=None, *, v_length: int = V_REGION_AA,
                       lambda_scalar: float | None = None) -> np.ndarray:
     """Vectorised :func:`shm_penalty` over a batch of clonotypes.
@@ -161,17 +189,91 @@ def shm_penalty_batch(mutations=None, identity=None, *, v_length: int = V_REGION
     Either argument may be ``None``; per row, a mutation spec wins over an identity. Rows with
     neither are ``nan``, which :class:`MutatedGermlineDistances` turns into "no SHM adjustment"
     rather than into a distance — see there for why those are different.
+
+    This was a ``for`` loop calling :func:`shm_penalty` per row while its own first line said
+    "vectorised", which put a string parse and one seqtree call per substitution on the embedding
+    path of every frame carrying mutation evidence: **3.10 microseconds per clonotype** at 15
+    mutations, 309.9 ms for 100,000. The parse is one polars pass now and the penalties one
+    gather from :func:`_penalty_table`. :func:`shm_penalty` stays the scalar reference, and
+    ``test_shm.py`` asserts the two agree — exactly, not within a tolerance, because every
+    penalty is a small integer and a float64 sum of those is exact however it is ordered.
     """
     n = len(mutations) if mutations is not None else len(identity)
-    out = np.empty(n, dtype=np.float64)
     lam = mean_shm_penalty() if lambda_scalar is None else lambda_scalar
-    for i in range(n):
-        m = mutations[i] if mutations is not None else None
-        d = identity[i] if identity is not None else None
-        out[i] = shm_penalty(m, identity=d, v_length=v_length, lambda_scalar=lam)
+    if mutations is not None and identity is not None and len(identity) != n:
+        raise ValueError(f"mutations has {n} entries and identity {len(identity)}")
+    out = np.full(n, np.nan, dtype=np.float64)
+
+    if identity is not None:
+        # `pl.Series` turns a None into a null and `to_numpy` a null into nan, which is the same
+        # answer the scalar path's `identity is None or not np.isfinite(identity)` gives.
+        idt = pl.Series(values=list(identity), dtype=pl.Float64, strict=False).to_numpy()
+        ok = np.isfinite(idt)
+        out[ok] = np.maximum(1.0 - idt[ok], 0.0) * v_length * lam
+
+    if mutations is not None:
+        # Truthiness of the RAW value, never of its string form: the scalar path branches on
+        # `if mutations:`, so an int 0 takes the identity path while an int 3 does not. Casting
+        # to string first would make "0" truthy and score that row from the wrong branch —
+        # plausibly, and with nothing about the number to show it had. One `bool()` per row is
+        # ~0.03 us against the 3.10 us this function used to spend on each of them.
+        has = np.fromiter((bool(m) for m in mutations), dtype=bool, count=n)
+        if has.any():
+            out[has] = _spec_penalties([m for m, h in zip(mutations, has) if h])
     return out
 
 
+def _spec_penalties(specs: list) -> np.ndarray:
+    """Summed BLOSUM62 penalty per mutation spec, for specs already known to be truthy.
+
+    Reproduces :func:`parse_mutations` as one polars pass: separators normalised, split, trimmed,
+    tokens under three characters dropped, then the first and last characters upper-cased and
+    required to be letters. A token that survives all that while naming a residue BLOSUM62 does
+    not define raises, exactly as the per-row path did when seqtree rejected it.
+    """
+    n = len(specs)
+    tok = (
+        pl.DataFrame({"s": pl.Series(values=specs, strict=False).cast(pl.String, strict=False)})
+        .with_row_index("row")
+        .with_columns(pl.col("s").str.replace_all("[; ]", ",").str.split(",").alias("t"))
+        .drop("s")
+        # `empty_as_null` is pinned, not inherited: polars 2.0 flips the default to False,
+        # and an empty token between two separators ("A23V,,S31N") is exactly what this
+        # parser meets. Either value is dropped by the length filter below, so pinning it
+        # keeps that true across the version change rather than relying on it.
+        .explode("t", empty_as_null=True)
+        .with_columns(pl.col("t").str.strip_chars())
+        .filter(pl.col("t").str.len_chars() >= 3)
+        .with_columns(a=pl.col("t").str.slice(0, 1).str.to_uppercase(),
+                      b=pl.col("t").str.slice(-1, 1).str.to_uppercase())
+        .with_columns(
+            ai=pl.col("a").replace_strict(_AA_INDEX, default=None, return_dtype=pl.Int64),
+            bi=pl.col("b").replace_strict(_AA_INDEX, default=None, return_dtype=pl.Int64))
+    )
+    if tok.height == 0:
+        return np.zeros(n, dtype=np.float64)
+    # A null index is either a first/last character that is not a letter, which
+    # `parse_mutations` skips in silence, or a letter outside A-Z, which seqtree rejected. Only
+    # the second is an error, and telling them apart needs `str.isalpha()`'s own domain
+    # (`\p{L}`) — so that regex runs on the null rows ALONE. It was 31.3 ms of a
+    # 113.5 ms pass when applied to all 1.5M tokens, and a well-formed spec has no null rows at
+    # all.
+    null = pl.col("ai").is_null() | pl.col("bi").is_null()
+    undefined = tok.filter(null).filter(
+        pl.col("a").str.contains(r"^\p{L}$") & pl.col("b").str.contains(r"^\p{L}$"))
+    if undefined.height == 0:
+        tok = tok.filter(~null) if tok.select(null.any()).item() else tok
+        pen = _penalty_table()[tok["ai"].to_numpy(), tok["bi"].to_numpy()]
+        if np.isfinite(pen).all():
+            return np.bincount(tok["row"].to_numpy(), weights=pen, minlength=n)
+        undefined = tok.filter(~np.isfinite(pen))
+    raise ValueError(
+        f"mutation spec names a residue BLOSUM62 does not define, e.g. "
+        f"{undefined['t'].unique().head(5).to_list()}. The defined upper-case codes are "
+        f"ABCDEFGHIKLMNPQRSTVWXYZ; J, O and U are not among them."
+    )
+
+
 @dataclass
 class MutatedGermlineDistances:
     """A :class:`~mir.distances.germline.GermlineDistances` that can see somatic mutation.

{mirpy_lib-4.3.0 → mirpy_lib-4.4.0}/src/mir/resources/prototypes/generate_prototypes.py

@@ -3,7 +3,8 @@
 
 Populates ``mir/resources/prototypes/`` with one TSV per (species, locus): ``N_PROTOTYPES`` rows
 of ``v_call, j_call, junction_aa`` sampled from **productive real reads annotated by arda**
-(``vdjtools.model.data.prepare(..., "functional")`` over the ``isalgo/airr_model_read`` FASTQs →
+(``vdjtools.model.data.prepare(..., "functional")`` over the FASTQs of the **private**
+HuggingFace dataset ``isalgo/airr_model_read`` →
 ``arda rnaseq map``). This gives arda IMGT allele names (one coordinate frame with arda-annotated
 query data → no germline-distance fallback) **and** the real CDR3 junction manifold, which embeds
 far better than synthetic P_gen junctions: EM-learned generative models produce degenerate

{mirpy_lib-4.3.0 → mirpy_lib-4.4.0}/src/mir/signature/features.py

@@ -416,15 +416,26 @@ def isotype_masks(df: pl.DataFrame) -> dict[str, np.ndarray]:
 
     Allele-stripped for the same reason :func:`isotype_shares` strips: the classes are gene names
     matched by equality, so ``IGHG1*01`` would match nothing and every read would come back
-    uncalled -- a composition rather than an error.
+    uncalled — a composition rather than an error.
+
+    One polars pass, three `is_in` expressions. It used to `to_list()` the whole column and run a
+    Python `in` test per clonotype per band: 8.63 ms at 10,000 clonotypes and **132.2 ms at
+    200,000**, to answer three questions about eight gene names. A null `c_call` is `False` in
+    every band, as `None in names` was.
     """
-    from vdjtools.io.schema import strip_allele
+    from vdjtools.io.schema import strip_allele_values
 
     if "c_call" not in df.columns:
         return {}
-    calls = df.select(strip_allele(pl.col("c_call").cast(pl.Utf8)).alias("c"))["c"].to_list()
-    return {name: np.array([c in names for c in calls], dtype=bool)
-            for name, names in ISOTYPE_BANDS.items()}
+    # `strip_allele_values` resolves the allele suffix once per DISTINCT call rather than once
+    # per row, which is where this function's time actually went: on 200,000 rows the three
+    # `is_in` tests are 3.7 ms and the stripping was 100.7 ms. A c_call column carries a few
+    # dozen distinct values however deep the repertoire is.
+    c = strip_allele_values(df["c_call"])
+    got = pl.DataFrame({"c": c}).select(
+        [pl.col("c").is_in(names).fill_null(False).alias(name)
+         for name, names in ISOTYPE_BANDS.items()])
+    return {name: got[name].to_numpy() for name in ISOTYPE_BANDS}
 
 
 def isotype_shares(df: pl.DataFrame, w: np.ndarray, *,
@@ -433,18 +444,16 @@ def isotype_shares(df: pl.DataFrame, w: np.ndarray, *,
 
     A *share of the geometry*, which is a different quantity from the read fraction the statistics
     half reports. The signature carries both rather than picking the flattering one.
-    """
-    from vdjtools.io.schema import strip_allele
 
-    if "c_call" not in df.columns:
+    The masks come from :func:`isotype_masks` rather than from a second copy of the same
+    allele-stripping comprehension — two copies is how the two came to disagree about a null.
+    """
+    masks = isotype_masks(df)
+    if not masks:
         return {}
-    # Allele-stripped: the classes are gene names matched by equality, so ``IGHG1*01`` would match
-    # nothing and the whole repertoire would come back uncalled -- a composition, not an error.
-    calls = df.select(strip_allele(pl.col("c_call").cast(pl.Utf8)).alias("c"))["c"].to_list()
     out: dict[str, float] = {}
     claimed = 0.0
-    for name, names in ISOTYPE_BANDS.items():
-        m = np.array([c in names for c in calls], dtype=bool)
+    for name, m in masks.items():
         if int(m.sum()) < min_clonotypes:
             continue
         out[name] = float(w[m].sum())