seqtree 0.2.0__tar.gz → 0.3.0__tar.gz

{seqtree-0.2.0 → seqtree-0.3.0}/.gitignore

@@ -11,3 +11,8 @@ _skbuild/
 docs/_build/
 bench/figures/
 bench/cache/
+
+# coverage artifacts
+.coverage
+.coverage.*
+htmlcov/

seqtree-0.3.0/CHANGELOG.md

@@ -0,0 +1,190 @@
+# Changelog
+
+All notable changes to `seqtree`. Dates are release dates; the project is pre-1.0, so a **minor**
+bump may carry breaking changes.
+
+## [0.3.0] — 2026-07-10
+
+Gap-block alignment, calibrated cutoffs, seed significance — the removal of several engine paths
+that returned confident wrong answers, and a corrected background control that changes every
+E-value.
+
+### Breaking
+
+- **The bundled control is a different set of sequences.** It was the *abundance head* of the
+  upstream repertoire — the 250,000 most expanded clonotypes — because both `gen_control.py` and
+  `_download` took the first `size` unique rows of a count-descending table. `appendix/evalue.tex`
+  (`ass:indep`) assumes the control's unique clonotypes are i.i.d. from `P₀`. Measured against a
+  uniform sample of the same size, the head is **25.8× more self-similar** (P(Hamming≤2 | equal
+  length) 3.11×10⁻³ vs 1.20×10⁻⁴) and carries **3.1× the ball mass** at a BLOSUM62 budget of 40
+  (mean n_C 110.1 vs 35.5). Both are now uniform reservoir samples over unique **productive**
+  clonotypes, seeded and shuffled so any prefix is itself a valid sub-sample.
+
+  **Every E-value moves.** Delete `~/.cache/seqtree/control_*.sqtree` after upgrading. Numbers
+  derived from the control are corrected throughout this file, `seeds.py`, `SKILL.md` and the
+  appendix.
+- **Controls are filtered to productive clonotypes.** VDJtools marks out-of-frame rearrangements
+  with `_` and in-frame stops with `*`; 13.7% of the mouse TRB table is out of frame. `_` cannot be
+  repaired at the amino-acid level — VDJtools collapses a *run* of untranslatable positions into one
+  character, so the residue count is already gone — and out-of-frame junctions escape thymic
+  selection, making them an estimator of `P_gen`, which `lem:hierarchy` says is not `P₀`.
+  `load_control("mouse_trb_aa")` previously raised on `_`; it now yields 694,241 clonotypes.
+- **`engine="auto"` now always resolves to `seqtm`.** It previously routed matrix-plus-indel
+  searches to `seqtrie`, whose budget defaults to `INT_MAX/4`, so
+  `SearchParams(max_subs=1, max_ins=1, matrix="BLOSUM62")` silently returned **every reference in
+  the index**. `seqtrie` ignores per-type edit caps and can never honour them; `auto` will not pick
+  it. Passing a matrix to `seqtrie` without an explicit `max_penalty` now raises.
+- **`Mode::Local` / `mode="local"` deleted.** It was a no-op: the field was stored and never read,
+  with zero call sites across `seqtree`, `vdjmatch` and `mhcmatch`.
+- **`GapPrior` takes `(block_start, block_length, longer_length)`** instead of
+  `(block_start, shorter_length)`. `central_prior`'s output is bit-identical
+  (`|2i − (m−d)| == |2i + d − m|`), and a frozen table pins that.
+- **`gap_extend` is now honoured.** `Index.align` is a real Gotoh affine alignment; previously
+  `gap_extend=1` and `gap_extend=99` produced identical scores and ops.
+
+### Fixed
+
+- `pairwise_batch` silently inverted `n_ins`/`n_dels` when transposing, but only when
+  `len(a) >= len(b)` — a size-dependent inversion.
+- `Index.align` did not validate the query alphabet (unlike `search_into`), and accepted negative
+  gap costs.
+- `Limits::max_hits` was set and never read; in `mode="all"` it truncated an **unsorted** list.
+- Stale docstring on `SubstitutionMatrix.from_similarity` (claimed `max(s_aa, s_bb) − s_ab`; the
+  code is the Gram form `s_aa + s_bb − 2·s_ab`).
+
+### Added
+
+- **`SubstitutionMatrix.scale()`** — the median mismatch penalty (BLOSUM62 → 14). Gap costs must
+  live on the matrix's scale; the old `gap_open=1` default made gaps ~14× cheaper than
+  substitutions, so `align()` would gap an equal-length pair rather than substitute. Use
+  `gap_open = 2 * matrix.scale()`.
+- **`seqtree.gapblock`** — single-contiguous-gap-block alignment for anchored junction loops.
+  `gapblock_score` is the exact `O(min(m,n))` optimum (0 mismatches in 55,727 pairs against
+  brute-force layout enumeration). `GapBlockIndex` reuses the existing Hamming engine over
+  deletion variants; no new C++.
+- **Gap priors** — `central_prior(lam)`, `profile_prior(lam, w)`, `frame_prior(lam, c)` and
+  `embed_in_frame(seq, width, c)`. A sequence score alone cannot place the block: a hard central
+  pin agrees with the flat, score-only choice on only 10.6% of pairs.
+- **`gapblock.score_matrix` and `ScoreMatrix`** — the dense `n × K` counterpart of
+  `GapBlockIndex.search`, for prototype-distance embeddings, where nothing can be pruned because
+  the distance to every reference *is* the output. C++, GIL released, one thread per core. On an
+  M3 against 3,000 prototypes: **51.3 M pairs/s** single-threaded, **532.7 M** on 16 cores, versus
+  0.41 M for pure-Python `gapblock_score` — while evaluating all `L+1` block positions, not a
+  fixed shortlist. The prior is flattened once into an `[m][d][i]` cube, so the kernel never
+  re-enters Python. `ScoreMatrix` carries the CPython buffer protocol: `numpy.asarray` wraps it
+  without copying, and seqtree keeps its zero runtime dependencies.
+- **`gapblock.positions_prior(starts)`** — restrict the block to a fixed set of starts, negative
+  values counting from the end, reproducing the `gap_positions=(3, 4, -4, -3)` convention that
+  other junction aligners hardcode. Shipped for interoperability, not as a recommendation: at a
+  matched false-positive rate on human TRB, candidate starts reach precision 0.156 against 0.414
+  for a single hard-pinned centre.
+- **`gapblock.IslandProfile`** — a per-island position weight matrix whose column penalty is
+  measured against the column's own consensus, `pen(j, a) = round(lam·log(p_max_j / p_j(a)))`. A
+  textbook log-odds score is signed and therefore not a ball; this one is `>= 0`, zero on the
+  consensus, and flows through `thetas_from_scores` unchanged. The frame column defaults to the
+  entropy-optimal one, which is modal at `c = 6` on real islands — where crystal structures put
+  the block.
+
+  Whether it beats scoring against every member **depends entirely on the cutoff**, which moves
+  with `N`: the E-value's `k = floor(e_target·M/N)` is how many control neighbours the cutoff may
+  admit, so the FPR is `k/M`. Over 108 calibrated VDJdb islands of ≥10 members (human TRB, three
+  held-out splits, paired bootstrap over islands, 250k control negatives):
+
+  | regime | FPR | min-over-members | `IslandProfile` | difference [95% CI] |
+  |---|---|---|---|---|
+  | loose reference | 1% | **99.5%** | 99.1% | −0.40 [−1.09, +0.14] |
+  | per-epitope islands (`N`= group, median 88) | 0.0568% | 88.3% | **89.3%** | +0.93 [−0.80, +2.79] |
+  | repertoire annotation (`N`≈20k) | 0.0012% | 37.6% | **48.5%** | +10.90 [+7.69, +14.21] |
+
+  So: **no significant difference while building the islands**, a large one when using them to
+  annotate a repertoire (on islands ≥50 members, 9.8% vs 22.6%). At `N≈20k` and `e_target=0.05`,
+  `k=0` and `thetas_from_scores` returns `-1` — the rule of three certifies no `E` below
+  `3N/M = 0.236`, and that is the cutoff the third row uses.
+
+  It does **not** generalise: same-epitope junctions in a different island are recovered by
+  neither representation. Nor is it a compression — 1,176 B against 182 B of member strings,
+  break-even at 84 members.
+- **`threshold_for_evalue` / `thetas_from_scores`** — invert `Ê = (N/M)·n_C` into the score cutoff
+  that achieves a target E, **per query**. Exact rather than a root-find, because scores are
+  integers. Returns `-1` where `e_target < 3N/M`, i.e. where the control is too small to certify
+  the bar.
+- **`seqtree.seeds`** — `core_kmers` and `SeedIndex` give control-calibrated E-values for shared
+  core k-mers. A shared rare central k-mer is ~4× enriched among co-specific pairs, but covers only
+  ~0.5% of them: seeds buy precision, not recall.
+- **`bench/bench_gapblock.py`** — the gap-freedom ladder, from a hard central pin through priors to
+  unrestricted affine.
+
+### Measured
+
+Numbers that constrain the API, all reproducible from `bench/` and the downstream repos:
+
+- **One gap block is enough.** Against a model-independent structural oracle (iterative
+  superposition + unrestricted affine DP) over 3,049 crystal junction pairs from 199 unique
+  sequences, the true correspondence is a single contiguous block in **95.2–100%** of cases for
+  every `d = 1..4`. Forcing one block costs no median CA-RMSD.
+- **The restriction is free where it applies, and protective where it does not.** At
+  `gap_open = 2*scale`, gap-block equals unrestricted affine on **98.8%** of related pairs (one
+  indel + 0–2 substitutions). On *unrelated* pairs affine undercuts it by a median of **106**
+  penalty units — affine inventing an alignment that does not exist. Extra gap freedom buys
+  manufactured similarity.
+- **A fixed score cutoff is not a calibrated cutoff.** Building islands on human TRB by union-find
+  at `gapblock_score ≤ 60`, **31.7%** of size-matched *random control* junctions land in a component
+  of ≥5 — structure invented by the threshold. Per-query E-value edges at `E* = 0.05` cut that to
+  **0.000** while raising the real signal: 2.334 edges per node against 0.021 for the control, which
+  forms 19,248 singletons, 223 pairs, three components of size 3–4, and nothing larger. The control
+  arm's realised edge rate lands on `E*`, which is the check that the calibration is honest.
+- **Mouse replicates it.** Against the mouse TRB control (694,241 productive clonotypes), 5 epitopes
+  and 1,692 TCRs give 5.856 calibrated edges per node against 0.019 for the control, which again
+  forms nothing larger than a pair. At a fixed θ=40 the control still lands 18.3% of its nodes in
+  components of ≥5.
+- **Constraining the block is what buys precision.** Compared at a *matched* false-positive rate —
+  each rung given the cutoff at which its own ball admits `E*` chance neighbours, since a freer rung
+  finds lower scores and a fixed budget would reward it for that — retrieval precision on the
+  length-different fraction of VDJdb human TRB same-epitope pairs (2,000 queries, `E* = 0.1`):
+
+  | rung | layouts | precision |
+  |---|---|---|
+  | fixed centre | 1 | **0.414** |
+  | central prior λ=21 | ~1–2 effective | 0.336 |
+  | flat (score alone) | L+1 | 0.176 |
+  | candidates (after 3–4, before last 3–4, centre) | 5 | 0.156 |
+
+  Trying several plausible positions and keeping the best score is worse than not trying: the score
+  picks the structurally correct layout about a tenth of the time, so each extra candidate is mostly
+  an opportunity to be wrong. Mouse replicates the ordering at `E* = 1.0` but its length-different
+  stratum holds only 24–79 true positives, too few to separate the rungs.
+- **Performance.** 91% of `GapBlockIndex.search` time is the query-deletion-variant branch, 9% the
+  9.8M-entry auxiliary indices. Netting the prior out of each variant's budget cuts that branch
+  from ~15 sub-searches to 2.5. Variant dedup (7–10% of variants before pruning, fewer after) and
+  length-bucketing are **not** built. At budget 40 over 250k references, `d_max=2` gap-block search
+  costs 2,562 µs/query — less than the plain Hamming ball at the same budget (3,051 µs/query).
+
+### Docs
+
+- `skills/seqtree/SKILL.md` — public API surface, invariants, and the gotchas that have bitten.
+- `docs/gapblock.rst` — a worked guide: why one gap block, how to choose its position, why a
+  placement rule is a column frame, and why a fixed score cutoff is not a calibrated one.
+- README corrected: `seqtrie` is a full-width DP that ignores per-type caps, not a banded one, and
+  `auto` does not choose between engines.
+- `appendix/evalue.tex` gains §"The score model: one gap block, placed by a prior" (the appendix
+  derived a theory of balls without ever saying what the score was) and a remark inverting the
+  E-value into a per-query cutoff. The pMHC section is compacted from ~110 lines of prose to ~50,
+  deferring to the `mhcmatch` appendix, which specialises this one rather than repeating it. Its
+  empirical tables stay: they are this repo's own `bench/bench_mhc_guess.py` output.
+- Test coverage: `gapblock.py`, `evalue.py` and `seeds.py` at **100%**; package total 88%. A new
+  `tests/python/test_doc_coverage.py` fails the build if a public symbol is undocumented, missing
+  from `__all__`, or unreachable from any docs page.
+
+## [0.2.0]
+
+- `structural` substitution matrix: Miyazawa–Jernigan interaction-strength similarity.
+- Built-in matrix list: `identity`, `BLOSUM62`, `PAM250`, `PAM100`, `structural` (dropped PAM50).
+
+## [0.1.0]
+
+- `SubstitutionMatrix.penalty(a, b)` exposed to Python.
+
+## [0.0.3]
+
+- Reproducible table→plot benchmark pipeline with oracle + perf regression.
+- pMHC non-binder E-value filter; class-II promiscuity notes.

{seqtree-0.2.0 → seqtree-0.3.0}/CMakeLists.txt

@@ -27,6 +27,7 @@ add_library(seqtree_core STATIC
   src/engine_seqtm.cpp
   src/engine_seqtrie.cpp
   src/searcher.cpp
+  src/gapblock.cpp
 )
 target_include_directories(seqtree_core PUBLIC include PRIVATE src)
 target_link_libraries(seqtree_core PUBLIC Threads::Threads)

{seqtree-0.2.0 → seqtree-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: seqtree
-Version: 0.2.0
+Version: 0.3.0
 Summary: Fast fuzzy search over biological sequences (C++ core, Python bindings)
 Keywords: sequence-search,fuzzy-matching,CDR3,immunology,bioinformatics,trie
 Author-Email: ISALGO laboratory <mikhail.shugay@gmail.com>
@@ -17,9 +17,9 @@ Project-URL: Documentation, https://antigenomics.github.io/seqtree/
 Requires-Python: >=3.10
 Provides-Extra: test
 Requires-Dist: pytest>=7.4; extra == "test"
+Requires-Dist: pytest-cov>=4.1; extra == "test"
 Provides-Extra: bench
 Requires-Dist: huggingface_hub; extra == "bench"
-Requires-Dist: pandas; extra == "bench"
 Requires-Dist: psutil; extra == "bench"
 Provides-Extra: docs
 Requires-Dist: sphinx; extra == "docs"
@@ -48,11 +48,12 @@ Two search engines over one trie:
 - **`seqtm`** — branch-and-bound enumeration. Exact per-type edit caps
   (`max_subs` / `max_ins` / `max_dels`) and a fast Hamming-only path. Best for
   small edit distances (UMI collapse, error correction, CDR3/epitope matching).
-- **`seqtrie`** — banded edit-distance DP. Matrix-weighted score budgets
-  (BLOSUM62 + gap costs), cost independent of edit count. Best for
-  similarity-scored searches.
+- **`seqtrie`** — full-width edit-distance DP carried down the trie. Honours the
+  `max_penalty` score budget only; it **ignores the per-type edit caps**. Use it
+  when the budget is the whole specification.
 
-`engine="auto"` picks one per query. Results are payload-agnostic:
+`engine="auto"` always picks `seqtm`, because it is the only engine that enforces
+the caps you asked for. Results are payload-agnostic:
 `(ref_id, score, n_subs, n_ins, n_dels)`. Downstream libraries map `ref_id` back
 to their own payloads (V gene, MHC, counts) and filter.
 
@@ -69,6 +70,23 @@ Beyond search, seqtree ships:
 - **E-values / significance** — calibrate hit counts against a background control repertoire
   (`load_control` + `evalues`), the TCRNET approach on a finite-sample footing. See the
   [E-value guide](https://antigenomics.github.io/seqtree/evalue.html).
+- **Calibrated cutoffs** — `threshold_for_evalue` inverts the E-value into the score cutoff that
+  achieves it, **per query**. A fixed cutoff is not a calibrated one: a control repertoire is
+  dense near germline and sparse among rare junctions, so the same threshold buys a common query
+  far more chance neighbours than a rare one.
+- **Gap-block alignment** — `gapblock` restricts alignment to one contiguous indel, which is the
+  right model for a V(D)J junction and, measured against unrestricted affine alignment, is
+  exactly optimal on **98.8%** of genuinely related pairs at a calibrated `gap_open`. A gap
+  prior (`central_prior`, `profile_prior`, `frame_prior`) chooses where the block goes — a
+  sequence score alone cannot. `score_matrix` scores a whole query set against a whole reference
+  set in one GIL-released C++ call (**532 M pairs/s** on 16 cores; `numpy.asarray` wraps the
+  result with no copy), the shape a prototype-distance embedding needs.
+- **Island profiles** — `IslandProfile.fit` builds a position weight matrix over a set of
+  frame-aligned junctions (an *island*) and scores a query column by column against the island
+  consensus, as a non-negative penalty that flows through `threshold_for_evalue` unchanged. At a
+  repertoire-scale cutoff it recovers **48.5%** of held-out members against **37.6%** for
+  min-over-members; at a loose cutoff the two are indistinguishable, so it earns its keep only
+  where the cutoff is strict.
 
 ## Install
 
@@ -85,7 +103,7 @@ needs a C++17 compiler and CMake (pulled in automatically by the build).
 ```fish
 bash setup.sh            # repo-local .venv + editable install
 bash setup.sh --tests    # + pytest
-bash setup.sh --bench     # + benchmark deps (huggingface_hub, pandas, psutil)
+bash setup.sh --bench     # + benchmark deps (huggingface_hub, psutil)
 ```
 
 ## Quickstart
@@ -119,6 +137,33 @@ target = seqtree.Index.build(vdjdb_cdr3s, alphabet="aa")
 for q, r in zip(queries, seqtree.evalues(target, control, queries, p)):
     if r["p_enrichment"] < 1e-3:
         print(q, r["E"], r["n_target"], r["n_control"])
+
+# ...and the cutoff that achieves a target E, per query (-1 = unreachable at this control size)
+ceiling = seqtree.SearchParams(max_subs=14, max_penalty=50, matrix="BLOSUM62", engine="seqtm")
+thetas = seqtree.threshold_for_evalue(target, control, queries, ceiling, e_target=0.05)
+
+# one contiguous gap block, placed by a prior rather than by the score alone
+from seqtree.gapblock import GapBlockIndex, central_prior, embed_in_frame
+
+gbi = GapBlockIndex(cdr3s, "aa", d_max=2)
+mat = seqtree.SubstitutionMatrix.blosum62()
+for ref_id, score, block_len, block_pos in gbi.search(
+        "CASSLGQAYEQYF", 40, mat, gap_open=2 * mat.scale(),
+        gap_prior=central_prior(int(1.5 * mat.scale()))):
+    ...
+
+# a fixed frame column makes gap placement transitive -- and a column index, hence a PWM, possible
+embed_in_frame("CASSGQAYEQYF", width=14, c=4)      # 'CASS--GQAYEQYF'
+
+# a whole query set vs a whole reference set, in one GIL-released C++ call
+from seqtree.gapblock import score_matrix, IslandProfile
+sm = score_matrix(clonotypes, prototypes, mat, gap_open=2 * mat.scale(), threads=0)
+import numpy as np
+distances = np.asarray(sm)                          # (len(clonotypes), len(prototypes)) int32, zero-copy
+
+# a position weight matrix over an island, still a non-negative penalty (feeds threshold_for_evalue)
+profile = IslandProfile.fit(island_members)
+profile.score("CASSLGQAYEQYF")                      # 0 on the consensus, > 0 for deviations
 ```
 
 ## Tests
@@ -138,6 +183,8 @@ python bench/bench.py                # recall vs ground truth (real VDJdb data)
 python bench/bench_evalue.py         # true E-value benchmark (target vs background control)
 python bench/bench_evalue_matrix.py  # significance across reference/control/query/scope grid
 python bench/bench_epitope.py        # epitope detection-complexity (GIL vs NLV)
+python bench/bench_gapblock.py       # the gap-freedom ladder: fixed centre → prior → flat → affine
+python bench/bench_score_matrix.py   # dense batch gap-block throughput (µs/pair, M pairs/s, RSS)
 ```
 
 Figures (throughput, scaling, matrix-scoring overhead, collisions, E-value matrix, epitope

{seqtree-0.2.0 → seqtree-0.3.0}/README.md

@@ -15,11 +15,12 @@ Two search engines over one trie:
 - **`seqtm`** — branch-and-bound enumeration. Exact per-type edit caps
   (`max_subs` / `max_ins` / `max_dels`) and a fast Hamming-only path. Best for
   small edit distances (UMI collapse, error correction, CDR3/epitope matching).
-- **`seqtrie`** — banded edit-distance DP. Matrix-weighted score budgets
-  (BLOSUM62 + gap costs), cost independent of edit count. Best for
-  similarity-scored searches.
+- **`seqtrie`** — full-width edit-distance DP carried down the trie. Honours the
+  `max_penalty` score budget only; it **ignores the per-type edit caps**. Use it
+  when the budget is the whole specification.
 
-`engine="auto"` picks one per query. Results are payload-agnostic:
+`engine="auto"` always picks `seqtm`, because it is the only engine that enforces
+the caps you asked for. Results are payload-agnostic:
 `(ref_id, score, n_subs, n_ins, n_dels)`. Downstream libraries map `ref_id` back
 to their own payloads (V gene, MHC, counts) and filter.
 
@@ -36,6 +37,23 @@ Beyond search, seqtree ships:
 - **E-values / significance** — calibrate hit counts against a background control repertoire
   (`load_control` + `evalues`), the TCRNET approach on a finite-sample footing. See the
   [E-value guide](https://antigenomics.github.io/seqtree/evalue.html).
+- **Calibrated cutoffs** — `threshold_for_evalue` inverts the E-value into the score cutoff that
+  achieves it, **per query**. A fixed cutoff is not a calibrated one: a control repertoire is
+  dense near germline and sparse among rare junctions, so the same threshold buys a common query
+  far more chance neighbours than a rare one.
+- **Gap-block alignment** — `gapblock` restricts alignment to one contiguous indel, which is the
+  right model for a V(D)J junction and, measured against unrestricted affine alignment, is
+  exactly optimal on **98.8%** of genuinely related pairs at a calibrated `gap_open`. A gap
+  prior (`central_prior`, `profile_prior`, `frame_prior`) chooses where the block goes — a
+  sequence score alone cannot. `score_matrix` scores a whole query set against a whole reference
+  set in one GIL-released C++ call (**532 M pairs/s** on 16 cores; `numpy.asarray` wraps the
+  result with no copy), the shape a prototype-distance embedding needs.
+- **Island profiles** — `IslandProfile.fit` builds a position weight matrix over a set of
+  frame-aligned junctions (an *island*) and scores a query column by column against the island
+  consensus, as a non-negative penalty that flows through `threshold_for_evalue` unchanged. At a
+  repertoire-scale cutoff it recovers **48.5%** of held-out members against **37.6%** for
+  min-over-members; at a loose cutoff the two are indistinguishable, so it earns its keep only
+  where the cutoff is strict.
 
 ## Install
 
@@ -52,7 +70,7 @@ needs a C++17 compiler and CMake (pulled in automatically by the build).
 ```fish
 bash setup.sh            # repo-local .venv + editable install
 bash setup.sh --tests    # + pytest
-bash setup.sh --bench     # + benchmark deps (huggingface_hub, pandas, psutil)
+bash setup.sh --bench     # + benchmark deps (huggingface_hub, psutil)
 ```
 
 ## Quickstart
@@ -86,6 +104,33 @@ target = seqtree.Index.build(vdjdb_cdr3s, alphabet="aa")
 for q, r in zip(queries, seqtree.evalues(target, control, queries, p)):
     if r["p_enrichment"] < 1e-3:
         print(q, r["E"], r["n_target"], r["n_control"])
+
+# ...and the cutoff that achieves a target E, per query (-1 = unreachable at this control size)
+ceiling = seqtree.SearchParams(max_subs=14, max_penalty=50, matrix="BLOSUM62", engine="seqtm")
+thetas = seqtree.threshold_for_evalue(target, control, queries, ceiling, e_target=0.05)
+
+# one contiguous gap block, placed by a prior rather than by the score alone
+from seqtree.gapblock import GapBlockIndex, central_prior, embed_in_frame
+
+gbi = GapBlockIndex(cdr3s, "aa", d_max=2)
+mat = seqtree.SubstitutionMatrix.blosum62()
+for ref_id, score, block_len, block_pos in gbi.search(
+        "CASSLGQAYEQYF", 40, mat, gap_open=2 * mat.scale(),
+        gap_prior=central_prior(int(1.5 * mat.scale()))):
+    ...
+
+# a fixed frame column makes gap placement transitive -- and a column index, hence a PWM, possible
+embed_in_frame("CASSGQAYEQYF", width=14, c=4)      # 'CASS--GQAYEQYF'
+
+# a whole query set vs a whole reference set, in one GIL-released C++ call
+from seqtree.gapblock import score_matrix, IslandProfile
+sm = score_matrix(clonotypes, prototypes, mat, gap_open=2 * mat.scale(), threads=0)
+import numpy as np
+distances = np.asarray(sm)                          # (len(clonotypes), len(prototypes)) int32, zero-copy
+
+# a position weight matrix over an island, still a non-negative penalty (feeds threshold_for_evalue)
+profile = IslandProfile.fit(island_members)
+profile.score("CASSLGQAYEQYF")                      # 0 on the consensus, > 0 for deviations
 ```
 
 ## Tests
@@ -105,6 +150,8 @@ python bench/bench.py                # recall vs ground truth (real VDJdb data)
 python bench/bench_evalue.py         # true E-value benchmark (target vs background control)
 python bench/bench_evalue_matrix.py  # significance across reference/control/query/scope grid
 python bench/bench_epitope.py        # epitope detection-complexity (GIL vs NLV)
+python bench/bench_gapblock.py       # the gap-freedom ladder: fixed centre → prior → flat → affine
+python bench/bench_score_matrix.py   # dense batch gap-block throughput (µs/pair, M pairs/s, RSS)
 ```
 
 Figures (throughput, scaling, matrix-scoring overhead, collisions, E-value matrix, epitope

{seqtree-0.2.0 → seqtree-0.3.0}/ROADMAP.md

@@ -24,12 +24,18 @@ between them, written so an agent developing either package can pick it up cold:
 
 | Capability | Module / symbol | Notes |
 |---|---|---|
-| Fuzzy fixed-length search | `seqtree.Index`, engines `seqtm` / `seqtrie` | per-type edit caps + Hamming fast path; or banded DP |
-| Local / best-window mode | `Mode::Local` (C++) | class-II register, general local match |
+| Fuzzy fixed-length search | `seqtree.Index`, engines `seqtm` / `seqtrie` | seqtm: per-type edit caps + Hamming fast path. seqtrie: budget-only full-width DP over the trie (not banded), ignores per-type caps |
+| Single-gap-block loop alignment | `seqtree.gapblock` (Python) | anchored CDR3/junction alignment: one contiguous indel + positional gap prior |
+| Dense batch gap-block scoring | `seqtree.gapblock.score_matrix` → `ScoreMatrix` | every query × every reference in one GIL-released C++ call (~532 M pairs/s on 16 cores); `numpy.asarray` is zero-copy. The shape a prototype-distance embedding needs, where nothing can be pruned |
+| Per-island profile (PWM) | `seqtree.gapblock.IslandProfile` | position weight matrix over a frame-aligned island; `pen(j,a) = round(lam·log(p_max_j/p_j(a)))` is `≥ 0` and `0` on the consensus, so it stays a ball and flows through `thetas_from_scores` unchanged. Beats min-over-members only at a strict cutoff |
 | Substitution scoring | `SubstitutionMatrix` (Gram → squared-distance penalty `s_aa+s_bb−2·s_ab`) | payload-agnostic |
 | Per-position scoring | `PositionalMatrix` (`penalty(pos,a,b)` = base × per-position weight; weight 0 = free/anchor) | the hook for PSSMs and anchor masking |
 | k-mer seed index | `KmerIndex` (C++): unique-k-mer trie + CSR postings + per-peptide allele tag; `seed_and_gather` (GIL-released, parallel) | million-scale candidate generation |
 | Control-calibrated E-values | `seqtree.evalues`, `seqtree.load_control` | `Ê = (N/M)·n_C`, Poisson tail, `exclude_exact` |
+| Calibrated score cutoffs | `seqtree.threshold_for_evalue`, `thetas_from_scores` | inverts `Ê` to a **per-query** θ. A fixed θ is not calibrated: at `gapblock_score ≤ 60`, 31.7% of *random control* junctions land in a component of ≥5 — structure invented by the threshold, which per-query cutoffs remove entirely |
+| Gap-placement priors | `central_prior`, `profile_prior`, `frame_prior`, `positions_prior`, `embed_in_frame` | `prior(i, d, m) ≥ 0` and `= 0` at `d = 0`. Only a **constant-`i`** rule (`frame_prior`) makes a frame transitive — a column index, hence a PWM, which `IslandProfile` now ships |
+| Seed significance | `seqtree.seeds`: `core_kmers`, `SeedIndex` | precision, not recall: ~0.5% cross-island coverage |
+| Background control | `seqtree.load_control`, `sanitize` | uniform reservoir sample over unique **productive** clonotypes. *Planned:* an out-of-frame control as an empirical `P_gen` sample for the `π̂_gen` fallback — needs the `*.ntvj` tables, since the `.aa` table's `_` has already collapsed a run of positions |
 | Anchor / layout model | `seqtree.layout`: `AnchorSpec`, `DEFAULTS`, `mask_anchors`, `kmers`, `presentation_features`, `weight_profile` | parametrized anchors; class-II register trick |
 | pMHC homology + reverse | `seqtree.pmhc`: `PMHCStore`, `search_homologs`, `assign_allele`, `find_mimics`, `build_kmer_index` | reference impl; mhcmatch productionizes |
 | Presentation-aware E-values | `seqtree.pmhc_evalue.homolog_evalue` | per-allele null |
@@ -37,7 +43,7 @@ between them, written so an agent developing either package can pick it up cold:
 **E-value theory** lives in `appendix/evalue.tex`: the empirical-control null (§Setup, §Null), the
 Poisson/Chen–Stein bound (§Poisson), multiple testing (§E-value), the closest-hit Gumbel law
 (§Gumbel), Karlin–Altschul as the i.i.d. special case (§KA), the pMHC presentation-aware extension
-(§Epitopes, §Reverse problem), and elementary applications — UMI/barcode birthday-collision and
+(§Epitopes), and elementary applications — UMI/barcode birthday-collision and
 CDR3-nt error clustering (§Related applications).
 
 **Datasets.** `isalgo/pmhc_data` ships two tiers (see `appendix` Table "Scope of the two pmhc_data
@@ -115,8 +121,8 @@ additions below.
   presentation-aware E-values via `pmhc_evalue.homolog_evalue`). Positive control: the Dolton et al.
   A\*02:01 cross-reactive trio.
 - Allele guessing (reverse problem): `assign_allele` + the vote-fraction ranking / register trick
-  validated in `bench/bench_mhc_guess.py` (appendix §Reverse problem). ROC-AUC 0.90–0.98.
-- **Non-binder filter** (appendix §Reverse problem, "Filtering non-binders"): _binds no MHC_ →
+  validated in `bench/bench_mhc_guess.py` (appendix §Epitopes). ROC-AUC 0.90–0.98.
+- **Non-binder filter** (appendix §Epitopes, "Non-binders and class-II promiscuity"): _binds no MHC_ →
   best-over-panel E-value high / no allele scores above background; _doesn't bind allele a_ →
   per-allele `E_a > α`. mhcmatch exposes both thresholds.
 - Tier choice (full vs shortlist, §1 Datasets).

seqtree-0.3.0/SOURCES.md

@@ -0,0 +1,97 @@
+# Data sources
+
+Every dataset seqtree ships, downloads, or reads in a benchmark. Provenance is stated as
+**experimental** (measured from a biological sample), **derived** (computed from experimental data),
+or **synthetic** (generated by a model). Never conflate the three.
+
+## Bundled with the wheel
+
+### `python/seqtree/data/control_human_trb_aa.txt`
+
+| | |
+|---|---|
+| what | 250,000 human TRB junction amino-acid sequences, the null distribution `P₀` |
+| provenance | **experimental**, then **derived** by uniform subsampling |
+| origin | HuggingFace `antigenomics/vdjdb-cdr3-frequencies`, human TRB pooled clonotype table |
+| format | one junction per line, productive only (20 standard amino acids) |
+| regenerate | `python bench/gen_control.py --species human --chain trb --size 250000 --seed 0` |
+
+The upstream table is sorted by clonotype count, descending. The sample is a **seeded reservoir
+sample over unique clonotypes**, then shuffled, so any prefix `bundled[:k]` is itself a uniform
+sub-sample. Taking the first *k* rows of the upstream table instead yields the abundance head, which
+is 25.8× more self-similar and carries 3.1× the ball mass — it violates `ass:indep` in
+`appendix/evalue.tex` and corrupts every E-value. See `CHANGELOG.md` [0.3.0] Breaking.
+
+Out-of-frame clonotypes (VDJtools `_`) and in-frame stops (`*`) are dropped. `_` marks a *collapsed
+run* of untranslatable positions, so the residue count is already lost and no amino-acid repair is
+possible; and out-of-frame junctions escape thymic selection, making them a sample of `P_gen`, which
+`lem:hierarchy` proves is not `P₀`.
+
+### Substitution matrices — `src/blosum62.inc`, `pam250.inc`, `pam100.inc`, `structural.inc`
+
+| | |
+|---|---|
+| provenance | **derived** (published log-odds / interaction-strength matrices) |
+| origin | BLOSUM62 (NCBI); EMBOSS EPAM250, EPAM100; Miyazawa–Jernigan interaction strengths |
+| regenerate | `python bench/gen_matrices.py` |
+
+Stored as non-negative penalties via the Gram transform `pen[a][b] = s[a][a] + s[b][b] − 2·s[a][b]`,
+clamped at zero, so `penalty(a, a) == 0` and the score defines a ball.
+
+## Downloaded on demand
+
+### `load_control(name)` → `~/.cache/seqtree/control_*.sqtree`
+
+| | |
+|---|---|
+| what | larger control repertoires, e.g. `mouse_trb_aa` (694,241 productive clonotypes) |
+| provenance | **experimental**, then **derived** by uniform subsampling |
+| origin | HuggingFace `antigenomics/vdjdb-cdr3-frequencies` |
+| fetch | `seqtree.load_control("mouse_trb_aa")` (uses `huggingface_hub`, an optional dependency) |
+
+The cached index is keyed by name only, **not** by content. Delete `~/.cache/seqtree/control_*.sqtree`
+after upgrading across a control change.
+
+## Benchmark inputs — not shipped, passed by path
+
+These live in sibling repositories. seqtree does not depend on either; the benchmarks that read them
+take `--prototypes` / `--vdjdb` arguments and parse the files with the standard library.
+
+### VDJdb slim dump
+
+| | |
+|---|---|
+| what | TCR↔epitope pairs with `v.end` / `j.start` templated-flank annotations |
+| provenance | **experimental** (curated from published studies) |
+| origin | `../mirpy/airr_benchmark/vdjdb/vdjdb-2025-12-29/vdjdb.slim.txt.gz` |
+| upstream | https://github.com/antigenomics/vdjdb-db — release 2025-12-29 |
+| covers | human TRA/TRB, mouse TRA/TRB, macaque TRB. Dotted column names (`v.segm`, `antigen.epitope`) |
+| used by | `bench/gap_rules_by_locus.py` (criterion A) |
+
+### Per-locus prototype sets
+
+| | |
+|---|---|
+| what | 10,000 clonotypes per (species, locus): `v_call`, `j_call`, `junction_aa` |
+| provenance | **experimental** where a real control repertoire exists, otherwise **synthetic** (OLGA) |
+| origin | `../mirpy/mir/resources/prototypes/{species}_{locus}.tsv` |
+| covers | human TRA TRB TRG TRD IGH IGK IGL; mouse TRA TRB |
+| regenerate | `python ../mirpy/mir/resources/prototypes/generate_prototypes.py` (committed; never rebuilt at test time) |
+| used by | `bench/gap_rules_by_locus.py` (criteria B and C) |
+
+Junctions, not IMGT CDR3: they include both anchors (`C…F` or `C…W`). Median lengths run from 12
+(IGK) to 20 (IGH), with IGH reaching 50 — which is why a fixed gap-position set calibrated on
+length-14 TRB does not transfer.
+
+### IMGT gene library
+
+| | |
+|---|---|
+| what | germline V/D/J nucleotide sequences, 2,710 (species, allele) entries |
+| provenance | **experimental** (IMGT reference directory) |
+| origin | `../mirpy/mir/resources/gene_library/imgt_gene_library.txt` |
+| caveat | **V genes translate in frame 0; J genes do not** (TRBJ2-7 is frame 1, TRBJ1-1 frame 2). The companion `region_annotations.txt` carries `jcdr3_aa`, which excludes the conserved F/W — it is CDR3 convention, not junction, and it disagrees with the translated library on some alleles (TRBV1\*01, IGHV1-2\*01). |
+
+Not currently read by any committed benchmark. Recorded because a germline-templated-flank prior was
+built against it and **falsified**: it recovers the structurally correct block position 0.4% of the
+time on TRA against 42.4% for a central prior.