seqtree 0.3.0__tar.gz → 0.4.0__tar.gz

{seqtree-0.3.0 → seqtree-0.4.0}/CHANGELOG.md

@@ -3,6 +3,101 @@
 All notable changes to `seqtree`. Dates are release dates; the project is pre-1.0, so a **minor**
 bump may carry breaking changes.
 
+## [0.4.0] — 2026-07-11
+
+### Added
+
+- **`seqtree.pairwise` — Needleman-Wunsch and Smith-Waterman, so ordinary protein alignment no
+  longer needs BioPython.** Everything else in seqtree *minimises a non-negative penalty*, which
+  is what a search ball and an E-value need. These **maximise a raw log-odds similarity**, the way
+  BLAST and BioPython do, because that is what a pairwise alignment means.
+
+  | | |
+  |---|---|
+  | `pairwise.score(q, r, matrix, mode=...)` | optimal score, `O(min(m,n))` memory |
+  | `pairwise.align(...)` | plus the aligned strings and ops |
+  | `pairwise.score_matrix(queries, refs, ...)` | dense `n × K`, GIL released, zero-copy numpy |
+  | `pairwise.dist_matrix(...)` | `d = s(a,a) + s(b,b) − 2·s(a,b)`: non-negative, zero on the diagonal |
+
+  `mode="global"` is Needleman-Wunsch, `mode="local"` Smith-Waterman, and **`gap_open == gap_extend`
+  gives linear gaps** — no separate mode. A gap run of length `L` costs `gap_open + (L-1)·gap_extend`,
+  and global charges end gaps (true NW, not semi-global).
+
+  **It is a drop-in.** `tests/python/test_pairwise.py` runs it against `Bio.Align.PairwiseAligner`
+  as an oracle over three matrices × ten gap/mode settings × sixty sequence shapes — **zero
+  disagreements**, including on real germline V genes. BioPython is a *test-only* dependency;
+  seqtree still has **zero required runtime dependencies** and never imports it.
+
+  Measured on an M3 (all-against-all, BLOSUM62, global, 11/1):
+
+  | sequence length | seqtree, 1 thread | seqtree, 16 threads | BioPython | speedup |
+  |---|---|---|---|---|
+  | 15 (a junction) | 1.7 M pairs/s | **20.1 M pairs/s** | 0.31 M pairs/s | **65×** |
+  | 90 (a germline V gene) | 72 k pairs/s | **893 k pairs/s** | 10 k pairs/s | **87×** |
+
+- **`SubstitutionMatrix.similarity(a, b)`** — the raw signed log-odds, alongside the existing
+  non-negative `penalty(a, b)`. The Gram transform `pen = s(a,a) + s(b,b) − 2·s(a,b)` is **lossy**:
+  it forces the diagonal to zero and destroys `s(a,a)`, so a similarity cannot be recovered from a
+  penalty. Both views are now stored.
+
+- **`SubstitutionMatrix.blosum45()` and `.blosum80()`**, and the names `"BLOSUM45"` / `"BLOSUM80"`
+  wherever a matrix name is accepted. Shallower and deeper than BLOSUM62 — for remote and close
+  homologs respectively.
+
+## [0.3.1] — 2026-07-10 (never published; folded into 0.4.0)
+
+> Tagged but not released to PyPI. Everything below ships in **0.4.0**, so upgrading from 0.3.0
+> straight to 0.4.0 picks it all up. Kept as its own section because it is a distinct set of fixes.
+
+
+### Fixed
+
+- **A cold cache shared by concurrent processes could hand back a half-written index.**
+  `Index::save` wrote straight into the destination, so for the whole duration of the write the
+  file existed but was truncated. A second process that checked `os.path.exists(cache)` in that
+  window loaded a stub and raised `RuntimeError: truncated or corrupt index`. On a 45 MB control
+  index the window is ~55 ms, and a reader racing a writer hit it **10 times out of 10**.
+
+  This is the first-use-only failure of any multi-process fan-out sharing `~/.cache`: pytest-xdist,
+  a Snakemake or Nextflow pipeline calling `load_control` in parallel, a `multiprocessing` pool.
+  Once the cache is warm it is read-only and was always safe. CI matrix jobs were never affected —
+  separate runners, separate caches.
+
+  `Index::save` and `KmerIndex::save` now serialize into a uniquely-named temporary beside the
+  destination and `rename` it into place. Rename is atomic on the same filesystem, on POSIX and
+  Windows alike, so a reader sees either the previous complete file or the new complete file and
+  never a partial one. A failed save cleans up its temporary and leaves any pre-existing index
+  intact.
+
+- **A corrupt or stale cache now rebuilds instead of raising.** A file truncated by a full disk,
+  left by a killed process, or written by an older seqtree sent `load_control` into an exception;
+  it now falls back to rebuilding. The cache was always best-effort and now behaves that way.
+
+- **The control cache is content-addressed, so a stale cache can no longer be served silently.**
+  The key was `control_{name}_{size}.sqtree`, which named neither the **alphabet**, nor the
+  **seed**, nor the **source data**. Three consequences, all live:
+
+  - Two calls differing only in `seed` — which must draw *different* reservoir samples — shared one
+    cache file, so the second silently received the first's sequences. Same for `alphabet`.
+  - An upgrade that changed the bundled control kept the same filename, so a warm cache served the
+    **previous release's** control. This is exactly how 0.3.0's corrected (uniform) control could be
+    masked by a stale 0.2.0 (abundance-head) cache — and why 0.3.0's notes had to ask people to
+    `rm ~/.cache/seqtree/control_*.sqtree` by hand.
+
+  The key now carries a fingerprint of the bundled asset's own bytes (or, on the download path, the
+  source and seed), so a control that changed simply misses the old cache. Superseded caches,
+  including pre-fingerprint ones from earlier releases, are deleted on the next build.
+
+  **You no longer need to clear `~/.cache/seqtree` when upgrading.** Doing so is harmless.
+
+### Added
+
+- **`load_control` takes an inter-process lock around build-and-save when `filelock` is available**
+  (it arrives with `huggingface_hub`). This is an optimisation, not the fix: correctness comes from
+  the atomic rename and holds with no lock at all. What the lock saves is work — without it, a cold
+  fan-out of N workers has every worker build the same 250k-clonotype index and discard N−1 of them.
+  seqtree still has **zero required runtime dependencies**; the import is guarded.
+
 ## [0.3.0] — 2026-07-10
 
 Gap-block alignment, calibrated cutoffs, seed significance — the removal of several engine paths
@@ -20,9 +115,11 @@ E-value.
   (mean n_C 110.1 vs 35.5). Both are now uniform reservoir samples over unique **productive**
   clonotypes, seeded and shuffled so any prefix is itself a valid sub-sample.
 
-  **Every E-value moves.** Delete `~/.cache/seqtree/control_*.sqtree` after upgrading. Numbers
-  derived from the control are corrected throughout this file, `seeds.py`, `SKILL.md` and the
-  appendix.
+  **Every E-value moves.** Delete `~/.cache/seqtree/control_*.sqtree` after upgrading — a warm
+  cache from 0.2.0 would otherwise be served in place of the corrected control. (Fixed in 0.3.1:
+  the cache is now content-addressed and a stale one simply misses. Upgrading straight from 0.2.0
+  to ≥0.3.1 needs no manual step.) Numbers derived from the control are corrected throughout this
+  file, `seeds.py`, `SKILL.md` and the appendix.
 - **Controls are filtered to productive clonotypes.** VDJtools marks out-of-frame rearrangements
   with `_` and in-frame stops with `*`; 13.7% of the mouse TRB table is out of frame. `_` cannot be
   repaired at the amino-acid level — VDJtools collapses a *run* of untranslatable positions into one

{seqtree-0.3.0 → seqtree-0.4.0}/CMakeLists.txt

@@ -28,6 +28,7 @@ add_library(seqtree_core STATIC
   src/engine_seqtrie.cpp
   src/searcher.cpp
   src/gapblock.cpp
+  src/pairwise.cpp
 )
 target_include_directories(seqtree_core PUBLIC include PRIVATE src)
 target_link_libraries(seqtree_core PUBLIC Threads::Threads)

{seqtree-0.3.0 → seqtree-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: seqtree
-Version: 0.3.0
+Version: 0.4.0
 Summary: Fast fuzzy search over biological sequences (C++ core, Python bindings)
 Keywords: sequence-search,fuzzy-matching,CDR3,immunology,bioinformatics,trie
 Author-Email: ISALGO laboratory <mikhail.shugay@gmail.com>
@@ -18,6 +18,7 @@ Requires-Python: >=3.10
 Provides-Extra: test
 Requires-Dist: pytest>=7.4; extra == "test"
 Requires-Dist: pytest-cov>=4.1; extra == "test"
+Requires-Dist: biopython>=1.81; extra == "test"
 Provides-Extra: bench
 Requires-Dist: huggingface_hub; extra == "bench"
 Requires-Dist: psutil; extra == "bench"
@@ -59,7 +60,7 @@ to their own payloads (V gene, MHC, counts) and filter.
 
 Beyond search, seqtree ships:
 
-- **Substitution matrices** — built-in `identity`, `BLOSUM62`, `PAM250`, `PAM100`, and `structural`
+- **Substitution matrices** — built-in `identity`, `BLOSUM45`, `BLOSUM62`, `BLOSUM80`, `PAM250`, `PAM100`, and `structural`
   — a **Miyazawa–Jernigan interaction-strength** matrix: each residue's strength `q(a)=mean_b e(a,b)`
   is read off the MJ contact potential, so substitutions between residues of like interaction strength
   are cheap. It separates strong (hydrophobic `F W C L Y M I V`) from weak (polar/charged
@@ -81,6 +82,13 @@ Beyond search, seqtree ships:
   sequence score alone cannot. `score_matrix` scores a whole query set against a whole reference
   set in one GIL-released C++ call (**532 M pairs/s** on 16 cores; `numpy.asarray` wraps the
   result with no copy), the shape a prototype-distance embedding needs.
+- **Pairwise alignment without BioPython** — `seqtree.pairwise` is Needleman–Wunsch
+  (`mode="global"`) and Smith–Waterman (`mode="local"`) with affine or linear gaps, on the raw
+  log-odds scale. It is a **drop-in for `Bio.Align.PairwiseAligner`** — verified against it as an
+  oracle across three matrices, ten gap/mode settings and sixty sequence shapes with **zero
+  disagreements** — and **65–87× faster**, since there is no Python in the per-pair loop.
+  `dist_matrix` gives `d = s(a,a) + s(b,b) − 2·s(a,b)` directly. BioPython is a *test-only*
+  dependency; seqtree still needs nothing at runtime.
 - **Island profiles** — `IslandProfile.fit` builds a position weight matrix over a set of
   frame-aligned junctions (an *island*) and scores a query column by column against the island
   consensus, as a non-negative penalty that flows through `threshold_for_evalue` unchanged. At a
@@ -164,6 +172,15 @@ distances = np.asarray(sm)                          # (len(clonotypes), len(prot
 # a position weight matrix over an island, still a non-negative penalty (feeds threshold_for_evalue)
 profile = IslandProfile.fit(island_members)
 profile.score("CASSLGQAYEQYF")                      # 0 on the consensus, > 0 for deviations
+
+# ordinary pairwise alignment -- Needleman-Wunsch / Smith-Waterman, no BioPython
+from seqtree.pairwise import align, score, dist_matrix
+score("CASSLGQAYEQYF", "CASSPGQAYEQF", mat)                    # global, BLAST defaults (11/1)
+score("WWWAAAWWW", "KKKAAAKKK", mat, mode="local")             # Smith-Waterman
+score("AAA", "AAAAA", mat, gap_open=5, gap_extend=5)           # linear gaps: open == extend
+aln = align("CASSLGQAYEQYF", "CASSPGQAYEQF", mat)              # + aligned strings and ops
+
+d = np.asarray(dist_matrix(v_genes, v_genes, mat, threads=0))  # s(a,a)+s(b,b)-2s(a,b), zero diagonal
 ```
 
 ## Tests

{seqtree-0.3.0 → seqtree-0.4.0}/README.md

@@ -26,7 +26,7 @@ to their own payloads (V gene, MHC, counts) and filter.
 
 Beyond search, seqtree ships:
 
-- **Substitution matrices** — built-in `identity`, `BLOSUM62`, `PAM250`, `PAM100`, and `structural`
+- **Substitution matrices** — built-in `identity`, `BLOSUM45`, `BLOSUM62`, `BLOSUM80`, `PAM250`, `PAM100`, and `structural`
   — a **Miyazawa–Jernigan interaction-strength** matrix: each residue's strength `q(a)=mean_b e(a,b)`
   is read off the MJ contact potential, so substitutions between residues of like interaction strength
   are cheap. It separates strong (hydrophobic `F W C L Y M I V`) from weak (polar/charged
@@ -48,6 +48,13 @@ Beyond search, seqtree ships:
   sequence score alone cannot. `score_matrix` scores a whole query set against a whole reference
   set in one GIL-released C++ call (**532 M pairs/s** on 16 cores; `numpy.asarray` wraps the
   result with no copy), the shape a prototype-distance embedding needs.
+- **Pairwise alignment without BioPython** — `seqtree.pairwise` is Needleman–Wunsch
+  (`mode="global"`) and Smith–Waterman (`mode="local"`) with affine or linear gaps, on the raw
+  log-odds scale. It is a **drop-in for `Bio.Align.PairwiseAligner`** — verified against it as an
+  oracle across three matrices, ten gap/mode settings and sixty sequence shapes with **zero
+  disagreements** — and **65–87× faster**, since there is no Python in the per-pair loop.
+  `dist_matrix` gives `d = s(a,a) + s(b,b) − 2·s(a,b)` directly. BioPython is a *test-only*
+  dependency; seqtree still needs nothing at runtime.
 - **Island profiles** — `IslandProfile.fit` builds a position weight matrix over a set of
   frame-aligned junctions (an *island*) and scores a query column by column against the island
   consensus, as a non-negative penalty that flows through `threshold_for_evalue` unchanged. At a
@@ -131,6 +138,15 @@ distances = np.asarray(sm)                          # (len(clonotypes), len(prot
 # a position weight matrix over an island, still a non-negative penalty (feeds threshold_for_evalue)
 profile = IslandProfile.fit(island_members)
 profile.score("CASSLGQAYEQYF")                      # 0 on the consensus, > 0 for deviations
+
+# ordinary pairwise alignment -- Needleman-Wunsch / Smith-Waterman, no BioPython
+from seqtree.pairwise import align, score, dist_matrix
+score("CASSLGQAYEQYF", "CASSPGQAYEQF", mat)                    # global, BLAST defaults (11/1)
+score("WWWAAAWWW", "KKKAAAKKK", mat, mode="local")             # Smith-Waterman
+score("AAA", "AAAAA", mat, gap_open=5, gap_extend=5)           # linear gaps: open == extend
+aln = align("CASSLGQAYEQYF", "CASSPGQAYEQF", mat)              # + aligned strings and ops
+
+d = np.asarray(dist_matrix(v_genes, v_genes, mat, threads=0))  # s(a,a)+s(b,b)-2s(a,b), zero diagonal
 ```
 
 ## Tests

{seqtree-0.3.0 → seqtree-0.4.0}/include/seqtree/seqtree.hpp

@@ -30,31 +30,48 @@ private:
     char     dec_[32];
 };
 
-// Non-negative substitution penalties indexed by alphabet code; penalty(a,a)==0.
+// A substitution matrix in two views.
+//
+//   penalty(a,b)     non-negative, penalty(a,a)==0. What search and the E-value ball need:
+//                    a distance-like cost that is minimised.
+//   similarity(a,b)  the raw log-odds, signed. What Needleman-Wunsch and Smith-Waterman need:
+//                    a score that is maximised.
+//
+// The penalty is the Gram / squared-distance transform of the similarity, which is LOSSY --
+// it forces the diagonal to zero, destroying s(a,a) -- so the raw grid is retained rather
+// than reconstructed.
 class SubstitutionMatrix {
 public:
     // Unit cost: 0 for a match, 1 for a mismatch (this is the "identity" matrix).
     static SubstitutionMatrix unit(uint8_t size);
     // Built-in amino-acid matrices (valid only for the AminoAcid codec order).
     static SubstitutionMatrix blosum62();
+    static SubstitutionMatrix blosum45();  // shallower: remote homologs
+    static SubstitutionMatrix blosum80();  // deeper: close homologs
     static SubstitutionMatrix pam250();   // EMBOSS EPAM250 (NCBI log-odds)
     static SubstitutionMatrix pam100();   // EMBOSS EPAM100 (NCBI log-odds)
     static SubstitutionMatrix structural();  // Miyazawa-Jernigan interaction-strength similarity
     // Convert a similarity matrix (row-major, size*size) to penalties via the Gram /
-    // squared-distance transform pen[a][b] = sim[a][a] + sim[b][b] - 2*sim[a][b].
+    // squared-distance transform pen[a][b] = sim[a][a] + sim[b][b] - 2*sim[a][b]. The raw
+    // similarity is kept too.
     static SubstitutionMatrix from_similarity(uint8_t size, const int32_t* sim);
 
     uint8_t size() const { return size_; }
     int32_t penalty(uint8_t a, uint8_t b) const { return pen_[a * size_ + b]; }
+    int32_t similarity(uint8_t a, uint8_t b) const { return sim_[a * size_ + b]; }
     // Median penalty over all mismatched symbol pairs -- the natural unit of this matrix.
     // Gap costs must be on this scale: the Gram transform makes a typical BLOSUM62 mismatch
     // cost ~15, so the default gap_open of 1 would make gaps ~15x cheaper than substitutions
     // and the aligner would gap rather than substitute. Use gap_open ~ 1-2 * scale().
+    //
+    // This is the PENALTY scale. A similarity-scoring aligner lives on the raw log-odds scale
+    // instead, where the usual BLAST defaults (gap_open 11, gap_extend 1) apply.
     int32_t scale() const;
 
 private:
     uint8_t size_ = 0;
     std::vector<int32_t> pen_;
+    std::vector<int32_t> sim_;
 };
 
 // Per-position substitution penalties pen(pos, a, b) over a fixed frame width W.
@@ -149,6 +166,48 @@ std::vector<std::vector<Hit>> pairwise_batch(const std::vector<std::string>& a,
                                              const std::vector<std::string>& b,
                                              Alphabet, const SearchParams&, int threads = 0);
 
+// ---------------------------------------------------------------------------------------
+// Pairwise similarity alignment: Needleman-Wunsch (global) and Smith-Waterman (local).
+//
+// These MAXIMISE the raw log-odds similarity, unlike everything else in this header, which
+// minimises a non-negative penalty. They exist so a caller does not need BioPython for an
+// ordinary protein alignment; the conventions match Bio.Align.PairwiseAligner exactly:
+//
+//   * a gap of length L costs  gap_open + (L-1)*gap_extend   (so gap_open is the cost of the
+//     first gap column, not an extra charge on top of it);
+//   * gap_open == gap_extend gives LINEAR gaps -- no separate mode is needed;
+//   * in global mode END gaps are charged like any other (true Needleman-Wunsch, not
+//     semi-global / overlap);
+//   * in local mode the score never drops below zero and the best local cell wins
+//     (Smith-Waterman).
+//
+// Gap costs are given as POSITIVE magnitudes and subtracted. BLAST's protein defaults are
+// gap_open = 11, gap_extend = 1.
+enum class AlignMode { Global, Local };
+
+// Optimal score only: O(min(m,n)) memory, no traceback. This is the hot path.
+int32_t align_score(std::string_view query, std::string_view ref, const SubstitutionMatrix&,
+                    Alphabet, AlignMode, int32_t gap_open, int32_t gap_extend);
+
+// Optimal score plus the aligned strings and ops. O(m*n) memory for the traceback.
+Alignment align_pair(std::string_view query, std::string_view ref, const SubstitutionMatrix&,
+                     Alphabet, AlignMode, int32_t gap_open, int32_t gap_extend);
+
+// Dense N*K similarity matrix, row-major (row i is queries[i] vs every ref). Parallel;
+// threads <= 0 => hardware_concurrency.
+std::vector<int32_t> align_score_matrix(const std::vector<std::string>& queries,
+                                        const std::vector<std::string>& refs,
+                                        const SubstitutionMatrix&, Alphabet, AlignMode,
+                                        int32_t gap_open, int32_t gap_extend, int threads = 0);
+
+// Dense N*K distance matrix d(a,b) = s(a,a) + s(b,b) - 2*s(a,b) >= 0, the Gram transform
+// applied at the SEQUENCE level to the alignment scores above. This is the distance a
+// prototype-embedding wants, and the one BioPython users hand-roll.
+std::vector<int32_t> align_dist_matrix(const std::vector<std::string>& queries,
+                                       const std::vector<std::string>& refs,
+                                       const SubstitutionMatrix&, Alphabet, AlignMode,
+                                       int32_t gap_open, int32_t gap_extend, int threads = 0);
+
 // Dense N*K single-gap-block penalty matrix, row-major (row i is queries[i] vs every ref).
 // Unlike pairwise_batch this is exhaustive: no budget, no trie, every cell scored.
 //

{seqtree-0.3.0 → seqtree-0.4.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "scikit_build_core.build"
 
 [project]
 name = "seqtree"
-version = "0.3.0"
+version = "0.4.0"
 description = "Fast fuzzy search over biological sequences (C++ core, Python bindings)"
 readme = "README.md"
 requires-python = ">=3.10"
@@ -22,7 +22,9 @@ classifiers = [
 dependencies = []
 
 [project.optional-dependencies]
-test = ["pytest>=7.4", "pytest-cov>=4.1"]
+# biopython is the ORACLE for seqtree.pairwise (NW/SW) -- a test-only dependency.
+# seqtree itself has no runtime dependencies and must never import it.
+test = ["pytest>=7.4", "pytest-cov>=4.1", "biopython>=1.81"]
 bench = ["huggingface_hub", "psutil"]
 docs = ["sphinx", "pydata-sphinx-theme", "nbsphinx"]
 control = ["huggingface_hub"]  # only needed to download controls larger than the bundled subset

{seqtree-0.3.0 → seqtree-0.4.0}/python/seqtree/__init__.py

@@ -24,7 +24,7 @@ from ._core import (
 )
 from .control import load_control
 from .evalue import evalues, thetas_from_scores, threshold_for_evalue
-from . import gapblock, layout, pmhc, seeds
+from . import gapblock, layout, pairwise, pmhc, seeds
 from .gapblock import (
     GapBlockIndex, IslandProfile, ScoreMatrix, central_prior, embed_in_frame, frame_prior,
     gapblock_score, positions_prior, profile_prior, score_matrix,
@@ -35,6 +35,7 @@ from .pmhc import PMHCStore, find_mimics
 __all__ = [
     "gapblock",
     "layout",
+    "pairwise",
     "pmhc",
     "seeds",
     "GapBlockIndex",