vdjtools 2.1.0__tar.gz → 2.2.0__tar.gz

{vdjtools-2.1.0 → vdjtools-2.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: vdjtools
-Version: 2.1.0
+Version: 2.2.0
 Summary: TCR/BCR repertoire analysis — Pgen/generation/inference, diversity, overlap, biomarkers (Python + C++)
 Keywords: immunology,tcr,bcr,airr,repertoire,pgen,olga,igor,bioinformatics
 Author-Email: Mikhail Shugay <mikhail.shugay@gmail.com>
@@ -86,10 +86,12 @@ Built on the antigenomics ecosystem:
 [vdjmatch](https://github.com/antigenomics/vdjmatch) (overlap + TCRnet),
 [arda](https://github.com/antigenomics/arda) (AIRR annotation + markup repair).
 
-> **Status: v2.0.0 pre-release under active development** (latest: `v2.0.0-alpha.2` — the native
-> V(D)J model engine). Install a pre-release with `pip install --pre vdjtools`. The legacy v1.x tool
-> lives on the [`legacy-1.x`](https://github.com/antigenomics/vdjtools/tree/legacy-1.x) branch and
-> its releases remain available under the repository tags (`v0.0.1` … `1.2.1`).
+> **Status: `v2.2.0`** — the native V(D)J model engine plus the full analytics suite (diversity,
+> overlap/TCRnet, preprocessing, biomarkers, single-cell), CDR features, and legacy-format ingestion
+> (MiXcr, MiGec, immunoSEQ, IMGT/HighV-QUEST, Vidjil, RTCR). Clonotype columns follow the AIRR
+> **junction** convention (`junction_nt` / `junction_aa`). The legacy v1.x tool lives on the
+> [`legacy-1.x`](https://github.com/antigenomics/vdjtools/tree/legacy-1.x) branch and its releases
+> remain available under the repository tags (`v0.0.1` … `1.2.1`).
 
 ## Install
 
@@ -129,6 +131,7 @@ model = load_bundled("TRB", source="olga")     # or source="learned" (fit to rea
 native.pgen_nt(model, "TGTGCCAGCAGC...")        # nucleotide generation probability (native C++)
 native.pgen_aa(model, "CASSLAPGATNEKLFF")       # amino-acid Pgen (codon-marginalised)
 native.pgen_aa(model, "CASSLAPGATNEKLFF", mismatches=1)   # + the whole Hamming-1 ball
+native.pgen_aa_batch(model, seqs, mismatches=1, threads=0)  # Pgen over many CDR3s, thread-parallel (~11×)
 generate(model, 1000)                           # sample a repertoire -> polars DataFrame
 ```
 
@@ -179,25 +182,31 @@ several times faster, and the built-in models keep the resident set small. Singl
 | sequence generation | **~32 000 seq/s** | — |
 
 Nucleotide Pgen (via the same transfer-matrix DP as the aa path — an in-frame CDR3 is an aa query with
-one codon fixed per position) is exact vs OLGA across all loci; the EM E-step parallelises over reads (~6.7× on 8
-threads); diversity/rarefaction run on a native iNEXT kernel (bootstrap + parallel batch). Memory
+one codon fixed per position) is exact vs OLGA across all loci. Batched Pgen / 1-mismatch over many
+CDR3s parallelises over sequences (`native.pgen_aa_batch`, **~11× on 16 cores**, bitwise-identical to
+the serial result); the EM E-step parallelises over reads (~6.7× on 8 threads); diversity/rarefaction
+run on a native iNEXT kernel (bootstrap + parallel batch). Memory
 stays light — **~63 MB** resident for `import vdjtools` plus one loaded model, **~123 MB** with all
 seven bundled models resident. Reproduce with `appendix/bench_pgen.py` and the `test_*_benchmark.py`
 suites (`RUN_BENCHMARK=1`).
 
-## Capabilities (rolling out by phase — see [ROADMAP.md](ROADMAP.md))
+## Capabilities (see [ROADMAP.md](ROADMAP.md) and the [API reference](https://docs.isalgo.dev/vdjtools/))
 
+- **IO** — canonical clonotype frame on AIRR **junction** columns (`junction_nt` / `junction_aa`);
+  readers for native vdjtools, AIRR Rearrangement TSV, and Parquet, plus format-detecting converters
+  for MiXcr (v1/2 + v3/4), MiGec, Adaptive immunoSEQ (v1/v2), IMGT/HighV-QUEST, Vidjil, and RTCR
+  ([`vdjtools.io.convert`](python/vdjtools/io/convert.py)); metadata-driven batch + hive-partitioned cohorts.
 - **Model** — native V(D)J recombination model: generation probability (Pgen — nt, aa,
-  1-mismatch, V/J-agnostic), sequence generation, and EM inference, all in a native (pybind11)
-  core. Supersedes OLGA and IGoR: arda-driven scenario enumeration, polars marginal tables,
-  read-parallelised EM, and **tandem-D (D-D)** support. Concordant with OLGA across all 7 loci;
+  1-mismatch, V/J-agnostic, **thread-parallel batch**), sequence generation, and EM inference, all in a
+  native (pybind11) core. Supersedes OLGA and IGoR: arda-driven scenario enumeration, polars marginal
+  tables, read-parallelised EM, and **tandem-D (D-D)** support. Concordant with OLGA across all 7 loci;
   precomputed OLGA + real-data-learned models bundled ([`load_bundled`](python/vdjtools/model/bundled.py)).
 - **Stats** — diversity (Chao1/Shannon/Simpson/…), spectratype, V/J/VJ usage.
 - **Features** — CDR physicochemical profiles, k-mer / V+k-mer summaries.
-- **Overlap** — sample overlap and TCRnet (via vdjmatch/seqtree).
+- **Overlap** — sample overlap and TCRnet (via vdjmatch/seqtree), similarity-aware overlap, clustering.
 - **Preprocess** — downsampling, error-correction, VJ-usage batch-effect correction, pooling/joining.
 - **Biomarker** — incidence-based association (Fisher) vs HLA / condition / chain-pairing; metaclonotypes.
-- **Single-cell** — AIRR Cell / 10x interoperability.
+- **Single-cell** — AIRR Cell / 10x interoperability, chain pairing + QC, and paired α/β Pgen.
 
 ## License
 

{vdjtools-2.1.0 → vdjtools-2.2.0}/README.md

@@ -27,10 +27,12 @@ Built on the antigenomics ecosystem:
 [vdjmatch](https://github.com/antigenomics/vdjmatch) (overlap + TCRnet),
 [arda](https://github.com/antigenomics/arda) (AIRR annotation + markup repair).
 
-> **Status: v2.0.0 pre-release under active development** (latest: `v2.0.0-alpha.2` — the native
-> V(D)J model engine). Install a pre-release with `pip install --pre vdjtools`. The legacy v1.x tool
-> lives on the [`legacy-1.x`](https://github.com/antigenomics/vdjtools/tree/legacy-1.x) branch and
-> its releases remain available under the repository tags (`v0.0.1` … `1.2.1`).
+> **Status: `v2.2.0`** — the native V(D)J model engine plus the full analytics suite (diversity,
+> overlap/TCRnet, preprocessing, biomarkers, single-cell), CDR features, and legacy-format ingestion
+> (MiXcr, MiGec, immunoSEQ, IMGT/HighV-QUEST, Vidjil, RTCR). Clonotype columns follow the AIRR
+> **junction** convention (`junction_nt` / `junction_aa`). The legacy v1.x tool lives on the
+> [`legacy-1.x`](https://github.com/antigenomics/vdjtools/tree/legacy-1.x) branch and its releases
+> remain available under the repository tags (`v0.0.1` … `1.2.1`).
 
 ## Install
 
@@ -70,6 +72,7 @@ model = load_bundled("TRB", source="olga")     # or source="learned" (fit to rea
 native.pgen_nt(model, "TGTGCCAGCAGC...")        # nucleotide generation probability (native C++)
 native.pgen_aa(model, "CASSLAPGATNEKLFF")       # amino-acid Pgen (codon-marginalised)
 native.pgen_aa(model, "CASSLAPGATNEKLFF", mismatches=1)   # + the whole Hamming-1 ball
+native.pgen_aa_batch(model, seqs, mismatches=1, threads=0)  # Pgen over many CDR3s, thread-parallel (~11×)
 generate(model, 1000)                           # sample a repertoire -> polars DataFrame
 ```
 
@@ -120,25 +123,31 @@ several times faster, and the built-in models keep the resident set small. Singl
 | sequence generation | **~32 000 seq/s** | — |
 
 Nucleotide Pgen (via the same transfer-matrix DP as the aa path — an in-frame CDR3 is an aa query with
-one codon fixed per position) is exact vs OLGA across all loci; the EM E-step parallelises over reads (~6.7× on 8
-threads); diversity/rarefaction run on a native iNEXT kernel (bootstrap + parallel batch). Memory
+one codon fixed per position) is exact vs OLGA across all loci. Batched Pgen / 1-mismatch over many
+CDR3s parallelises over sequences (`native.pgen_aa_batch`, **~11× on 16 cores**, bitwise-identical to
+the serial result); the EM E-step parallelises over reads (~6.7× on 8 threads); diversity/rarefaction
+run on a native iNEXT kernel (bootstrap + parallel batch). Memory
 stays light — **~63 MB** resident for `import vdjtools` plus one loaded model, **~123 MB** with all
 seven bundled models resident. Reproduce with `appendix/bench_pgen.py` and the `test_*_benchmark.py`
 suites (`RUN_BENCHMARK=1`).
 
-## Capabilities (rolling out by phase — see [ROADMAP.md](ROADMAP.md))
+## Capabilities (see [ROADMAP.md](ROADMAP.md) and the [API reference](https://docs.isalgo.dev/vdjtools/))
 
+- **IO** — canonical clonotype frame on AIRR **junction** columns (`junction_nt` / `junction_aa`);
+  readers for native vdjtools, AIRR Rearrangement TSV, and Parquet, plus format-detecting converters
+  for MiXcr (v1/2 + v3/4), MiGec, Adaptive immunoSEQ (v1/v2), IMGT/HighV-QUEST, Vidjil, and RTCR
+  ([`vdjtools.io.convert`](python/vdjtools/io/convert.py)); metadata-driven batch + hive-partitioned cohorts.
 - **Model** — native V(D)J recombination model: generation probability (Pgen — nt, aa,
-  1-mismatch, V/J-agnostic), sequence generation, and EM inference, all in a native (pybind11)
-  core. Supersedes OLGA and IGoR: arda-driven scenario enumeration, polars marginal tables,
-  read-parallelised EM, and **tandem-D (D-D)** support. Concordant with OLGA across all 7 loci;
+  1-mismatch, V/J-agnostic, **thread-parallel batch**), sequence generation, and EM inference, all in a
+  native (pybind11) core. Supersedes OLGA and IGoR: arda-driven scenario enumeration, polars marginal
+  tables, read-parallelised EM, and **tandem-D (D-D)** support. Concordant with OLGA across all 7 loci;
   precomputed OLGA + real-data-learned models bundled ([`load_bundled`](python/vdjtools/model/bundled.py)).
 - **Stats** — diversity (Chao1/Shannon/Simpson/…), spectratype, V/J/VJ usage.
 - **Features** — CDR physicochemical profiles, k-mer / V+k-mer summaries.
-- **Overlap** — sample overlap and TCRnet (via vdjmatch/seqtree).
+- **Overlap** — sample overlap and TCRnet (via vdjmatch/seqtree), similarity-aware overlap, clustering.
 - **Preprocess** — downsampling, error-correction, VJ-usage batch-effect correction, pooling/joining.
 - **Biomarker** — incidence-based association (Fisher) vs HLA / condition / chain-pairing; metaclonotypes.
-- **Single-cell** — AIRR Cell / 10x interoperability.
+- **Single-cell** — AIRR Cell / 10x interoperability, chain pairing + QC, and paired α/β Pgen.
 
 ## License
 

{vdjtools-2.1.0 → vdjtools-2.2.0}/include/vdjtools/model.hpp

@@ -64,6 +64,13 @@ double pgen_aa(const PackedModel& m, const std::string& aa, int v_idx, int j_idx
 // via the inclusion-exclusion identity over per-position wildcards. v_idx/j_idx as above.
 double pgen_aa_hamming1(const PackedModel& m, const std::string& aa, int v_idx, int j_idx);
 
+// Batch aa Pgen over many sequences, parallelized across sequences (mismatches: 0 = exact,
+// 1 = Hamming-1 ball). Bitwise-identical to the per-sequence calls. Per-sequence v/j indices;
+// empty v_idxs/j_idxs → all -1 (gene-agnostic). nthreads=0 → auto (hw-2).
+std::vector<double> pgen_aa_batch(const PackedModel& m, const std::vector<std::string>& seqs,
+                                  const std::vector<int>& v_idxs, const std::vector<int>& j_idxs,
+                                  int mismatches, int nthreads);
+
 // EM soft counts — one accumulator per event realization, laid out like the PackedModel prob
 // arrays so the Python M-step can renormalize them directly.
 struct Counts {

{vdjtools-2.1.0 → vdjtools-2.2.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "scikit_build_core.build"
 
 [project]
 name = "vdjtools"  # if the PyPI name is taken, fall back to a distinct dist name (cf. arda -> arda-mapper)
-version = "2.1.0"
+version = "2.2.0"
 description = "TCR/BCR repertoire analysis — Pgen/generation/inference, diversity, overlap, biomarkers (Python + C++)"
 readme = "README.md"
 requires-python = ">=3.10"

{vdjtools-2.1.0 → vdjtools-2.2.0}/python/vdjtools/__init__.py

@@ -13,5 +13,5 @@ vdjmatch/seqtree) until a feature that needs them is used.
 """
 from ._core import hamming
 
-__version__ = "2.1.0"
+__version__ = "2.2.0"
 __all__ = ["hamming", "__version__"]

{vdjtools-2.1.0 → vdjtools-2.2.0}/python/vdjtools/biomarker/fisher.py

@@ -15,8 +15,8 @@ doi:10.1038/ng.3822). A feature is public if it is present in ``>= min_incidence
 
 Two axes are exposed as first-class options:
 
-- **V/J match requirement** — the ``key`` tuple. ``(cdr3_aa,)`` matches on CDR3 alone;
-  ``(cdr3_aa, v_call)`` also requires the V gene; ``(cdr3_aa, v_call, j_call)`` (the
+- **V/J match requirement** — the ``key`` tuple. ``(junction_aa,)`` matches on CDR3 alone;
+  ``(junction_aa, v_call)`` also requires the V gene; ``(junction_aa, v_call, j_call)`` (the
   default, Emerson's definition) requires both.
 - **exact vs 1-mismatch CDR3 matching** — ``match``. ``"exact"`` groups on the literal
   key (pure polars, no extra deps). ``"1mm"`` first groups near-variant keys into
@@ -71,8 +71,8 @@ def fisher_association(
         phenotype: One row per subject with ``sample_id`` and the binary ``pheno_col``
             (bool / 0-1); subjects with a null label are dropped from both classes.
         pheno_col: Name of the binary phenotype column in ``phenotype``.
-        key: Feature key — a subset of ``(cdr3_aa, v_call, j_call)`` (must include
-            ``cdr3_aa``). This is the **V/J match requirement**.
+        key: Feature key — a subset of ``(junction_aa, v_call, j_call)`` (must include
+            ``junction_aa``). This is the **V/J match requirement**.
         match: ``"exact"`` (literal key) or ``"1mm"`` (group near-variants into
             metaclonotypes first; needs the ``[overlap]`` extra).
         min_incidence: Minimum number of subjects a feature must appear in to be tested.
@@ -90,7 +90,7 @@ def fisher_association(
         Hochberg) and ``direction`` (``"enriched"``/``"depleted"``), sorted by ``p_value``.
 
     Raises:
-        ValueError: If ``alternative``/``match`` is unrecognised, ``key`` omits ``cdr3_aa``,
+        ValueError: If ``alternative``/``match`` is unrecognised, ``key`` omits ``junction_aa``,
             or the phenotype has only one class after dropping unknowns.
     """
     if alternative not in _ALTERNATIVES:
@@ -98,8 +98,8 @@ def fisher_association(
     if match not in ("exact", "1mm"):
         raise ValueError(f"match must be 'exact' or '1mm'; got {match!r}")
     key = tuple(key)
-    if S.CDR3_AA not in key:
-        raise ValueError(f"key must include {S.CDR3_AA!r}; got {key}")
+    if S.JUNCTION_AA not in key:
+        raise ValueError(f"key must include {S.JUNCTION_AA!r}; got {key}")
 
     lf = cohort.lazy()
 
@@ -115,9 +115,9 @@ def fisher_association(
         raise ValueError("phenotype has only one class after dropping unknown labels")
 
     # Feature engineering: productive filter + allele stripping on the key's V/J columns.
-    lf = lf.filter(pl.col(S.CDR3_AA).is_not_null())
+    lf = lf.filter(pl.col(S.JUNCTION_AA).is_not_null())
     if productive_only:
-        lf = lf.filter(~pl.col(S.CDR3_AA).str.contains(r"[*_]"))
+        lf = lf.filter(~pl.col(S.JUNCTION_AA).str.contains(r"[*_]"))
     if strip_allele:
         for c in (S.V_CALL, S.J_CALL):
             if c in key:

{vdjtools-2.1.0 → vdjtools-2.2.0}/python/vdjtools/biomarker/metaclonotype.py

@@ -17,7 +17,7 @@ from __future__ import annotations
 
 import polars as pl
 
-from ..io.schema import CDR3_AA, J_CALL, V_CALL
+from ..io.schema import JUNCTION_AA, J_CALL, V_CALL
 
 _VDJMATCH_HINT = (
     "vdjmatch is required for vdjtools.biomarker.metaclonotype; install the extra with "
@@ -39,12 +39,12 @@ def metaclonotypes(clonotypes: pl.DataFrame, *, scope: str = "1,0,0,1",
                    threads: int = 0) -> pl.DataFrame:
     """Group unique clonotype keys into metaclonotypes by CDR3 edit-scope neighbourhood.
 
-    Two keys share a ``meta_id`` iff their ``cdr3_aa`` are within ``scope`` **and** they
+    Two keys share a ``meta_id`` iff their ``junction_aa`` are within ``scope`` **and** they
     share a V call (when ``match_v``) and a J call (when ``match_j``). Grouping is
     single-linkage (connected components of the neighbour graph).
 
     Args:
-        clonotypes: A clonotype frame; must carry ``cdr3_aa`` and, when the corresponding
+        clonotypes: A clonotype frame; must carry ``junction_aa`` and, when the corresponding
             ``match_*`` flag is set, ``v_call`` / ``j_call``.
         scope: vdjmatch edit-distance scope ``"subs,ins,dels,total"`` (default one
             substitution, length-preserving). ``"0,0,0,0"`` reduces to exact grouping.
@@ -53,7 +53,7 @@ def metaclonotypes(clonotypes: pl.DataFrame, *, scope: str = "1,0,0,1",
         threads: Worker threads for the native search (``0`` = all cores).
 
     Returns:
-        The distinct grouping keys (``cdr3_aa`` plus ``v_call``/``j_call`` as applicable)
+        The distinct grouping keys (``junction_aa`` plus ``v_call``/``j_call`` as applicable)
         with an added ``meta_id`` column (compact 0-based integer, singletons included).
 
     Raises:
@@ -67,8 +67,8 @@ def metaclonotypes(clonotypes: pl.DataFrame, *, scope: str = "1,0,0,1",
     if match_j and J_CALL in clonotypes.columns:
         group_cols.append(J_CALL)
 
-    uniq = (clonotypes.select([CDR3_AA, *group_cols])
-            .drop_nulls(CDR3_AA)
+    uniq = (clonotypes.select([JUNCTION_AA, *group_cols])
+            .drop_nulls(JUNCTION_AA)
             .unique(maintain_order=True)
             .with_row_index("_gid"))
     n = uniq.height
@@ -94,7 +94,7 @@ def metaclonotypes(clonotypes: pl.DataFrame, *, scope: str = "1,0,0,1",
     # union-find over the returned positional pairs. No partition (CDR3-only) => one pass.
     groups = uniq.group_by(group_cols, maintain_order=True) if group_cols else [((), uniq)]
     for _, sub in groups:
-        cdr3s = sub[CDR3_AA].to_list()
+        cdr3s = sub[JUNCTION_AA].to_list()
         if len(cdr3s) < 2:
             continue
         gids = sub["_gid"].to_list()

{vdjtools-2.1.0 → vdjtools-2.2.0}/python/vdjtools/cli/__init__.py

@@ -26,7 +26,7 @@ app = typer.Typer(
     help="vdjtools — TCR/BCR repertoire analysis: Pgen, sequence generation, diversity, overlap.",
 )
 
-_SEQ_COLS = ("cdr3_aa", "junction_aa", "cdr3aa", "cdr3_nt", "cdr3nt", "cdr3", "junction")
+_SEQ_COLS = ("junction_aa", "junction_nt", "cdr3_aa", "cdr3_nt", "cdr3aa", "cdr3nt", "cdr3", "junction")
 
 
 # ---------------------------------------------------------------------------- helpers
@@ -165,7 +165,7 @@ def generate(
 ) -> None:
     """Sample recombined sequences from a model — like ``olga-generate_sequences``.
 
-    Emits ``cdr3_nt, cdr3_aa, v_call, d_call, d2_call, j_call, productive`` (``d2_call`` is the
+    Emits ``junction_nt, junction_aa, v_call, d_call, d2_call, j_call, productive`` (``d2_call`` is the
     tandem D on the learned D-bearing loci; null otherwise).
     """
     from vdjtools.model.generate import generate as _generate

{vdjtools-2.1.0 → vdjtools-2.2.0}/python/vdjtools/features/kmer.py

@@ -1,6 +1,6 @@
 """CDR3 amino-acid k-mer profiles and joint V + k-mer + C feature summaries.
 
-K-mers are overlapping sliding windows over ``cdr3_aa``. Each k-mer occurrence
+K-mers are overlapping sliding windows over ``junction_aa``. Each k-mer occurrence
 carries its clonotype's weight (reads, unique, or frequency); weights are summed.
 """
 from __future__ import annotations
@@ -9,7 +9,7 @@ import polars as pl
 
 from ..io.schema import (
     C_CALL,
-    CDR3_AA,
+    JUNCTION_AA,
     LOCUS,
     V_CALL,
     add_locus,
@@ -19,17 +19,17 @@ from ..io.schema import (
 
 
 def _explode_kmers(df: pl.DataFrame, k: int) -> pl.DataFrame:
-    """Explode each clonotype's ``cdr3_aa`` into overlapping k-mers (column ``kmer``).
+    """Explode each clonotype's ``junction_aa`` into overlapping k-mers (column ``kmer``).
 
     Rows whose CDR3 is shorter than ``k`` (or null) contribute no k-mers.
     """
     if k < 1:
         raise ValueError(f"k must be >= 1; got {k}")
-    df = df.with_columns(pl.col(CDR3_AA).str.len_chars().alias("_len"))
+    df = df.with_columns(pl.col(JUNCTION_AA).str.len_chars().alias("_len"))
     df = df.filter(pl.col("_len") >= k)  # shorter/null CDR3s yield no k-mers
     df = df.with_columns(pl.int_ranges(0, pl.col("_len") - k + 1).alias("_pos"))
     df = df.explode("_pos", empty_as_null=True)
-    df = df.with_columns(pl.col(CDR3_AA).str.slice(pl.col("_pos"), k).alias("kmer"))
+    df = df.with_columns(pl.col(JUNCTION_AA).str.slice(pl.col("_pos"), k).alias("kmer"))
     return df.drop("_len", "_pos")
 
 

{vdjtools-2.1.0 → vdjtools-2.2.0}/python/vdjtools/features/physchem.py

@@ -4,12 +4,13 @@ For each clonotype the mean of a property over the residues of a chosen CDR3 reg
 is computed, then averaged (weighted by reads/frequency, or unweighted) within a
 group (e.g. per V-J pairing, or per locus).
 
-Region definitions (over ``cdr3_aa``, length ``L``):
-    ``all``     — the entire CDR3.
-    ``trimmed`` — ``cdr3_aa[3:-3]`` (conserved-anchor-trimmed core); clonotypes with
-        ``L <= 6`` have an empty core and are skipped for this region.
-    ``center``  — the middle five residues ``cdr3_aa[L//2-2 : L//2+3]``; clonotypes
-        with ``L < 5`` are skipped for this region.
+Region definitions (over ``junction_aa``, length ``L``):
+
+* ``all`` — the entire CDR3.
+* ``trimmed`` — ``junction_aa[3:-3]`` (conserved-anchor-trimmed core); clonotypes with
+  ``L <= 6`` have an empty core and are skipped for this region.
+* ``center`` — the middle five residues ``junction_aa[L//2-2 : L//2+3]``; clonotypes
+  with ``L < 5`` are skipped for this region.
 """
 from __future__ import annotations
 
@@ -19,7 +20,7 @@ from importlib import resources
 
 import polars as pl
 
-from ..io.schema import CDR3_AA, LOCUS, add_locus, weight_expr
+from ..io.schema import JUNCTION_AA, LOCUS, add_locus, weight_expr
 
 #: Default property subset: Kidera-factor-free physicochemistry + the 10 Kidera factors.
 DEFAULT_PROPERTIES = (
@@ -49,16 +50,16 @@ def load_property_table() -> pl.DataFrame:
 
 def _region_expr(region: str) -> pl.Expr:
     """Return the CDR3-region substring expression (null where the region is empty)."""
-    length = pl.col(CDR3_AA).str.len_chars()
+    length = pl.col(JUNCTION_AA).str.len_chars()
     if region == "all":
-        return pl.col(CDR3_AA)
+        return pl.col(JUNCTION_AA)
     if region == "trimmed":
         return (pl.when(length > 6)
-                  .then(pl.col(CDR3_AA).str.slice(3, length - 6))
+                  .then(pl.col(JUNCTION_AA).str.slice(3, length - 6))
                   .otherwise(None))
     if region == "center":
         return (pl.when(length >= 5)
-                  .then(pl.col(CDR3_AA).str.slice(length // 2 - 2, 5))
+                  .then(pl.col(JUNCTION_AA).str.slice(length // 2 - 2, 5))
                   .otherwise(None))
     raise ValueError(f"region must be 'all', 'trimmed' or 'center'; got {region!r}")
 

{vdjtools-2.1.0 → vdjtools-2.2.0}/python/vdjtools/io/__init__.py

@@ -5,11 +5,19 @@ Readers return the canonical clonotype frame defined in :mod:`vdjtools.io.schema
 from . import schema
 from .batch import iter_samples, read, read_metadata, read_samples, sniff_format
 from .cohort import ingest_cohort, scan_cohort
+from .convert import (
+    read_imgt,
+    read_immunoseq,
+    read_migec,
+    read_mixcr,
+    read_rtcr,
+    read_vidjil,
+)
 from .read import read_airr, read_parquet, read_vdjtools
 from .schema import (
     C_CALL,
-    CDR3_AA,
-    CDR3_NT,
+    JUNCTION_AA,
+    JUNCTION_NT,
     COLUMNS,
     COUNT,
     D_CALL,
@@ -30,6 +38,12 @@ __all__ = [
     "read_airr",
     "read_vdjtools",
     "read_parquet",
+    "read_mixcr",
+    "read_migec",
+    "read_immunoseq",
+    "read_imgt",
+    "read_vidjil",
+    "read_rtcr",
     "read_metadata",
     "read_samples",
     "iter_samples",
@@ -46,8 +60,8 @@ __all__ = [
     "D_CALL",
     "J_CALL",
     "C_CALL",
-    "CDR3_AA",
-    "CDR3_NT",
+    "JUNCTION_AA",
+    "JUNCTION_NT",
     "COUNT",
     "FREQ",
     "LOCUS",

{vdjtools-2.1.0 → vdjtools-2.2.0}/python/vdjtools/io/batch.py

@@ -6,14 +6,23 @@ metadata table into one long clonotype frame (or a per-sample dict).
 """
 from __future__ import annotations
 
+import gzip
 import os
 from pathlib import Path
 
 import polars as pl
 
+from . import convert
 from .read import read_airr, read_parquet, read_vdjtools
 from .schema import COLUMNS, LOCUS
 
+# Legacy third-party formats (vdjtools.io.convert), keyed by the fmt string used below.
+_CONVERTERS = {
+    "mixcr": convert.read_mixcr, "migec": convert.read_migec,
+    "immunoseq": convert.read_immunoseq, "imgt": convert.read_imgt,
+    "vidjil": convert.read_vidjil, "rtcr": convert.read_rtcr,
+}
+
 
 def _header_columns(path: str | os.PathLike) -> list[str]:
     """Return the (lower-cased) header column names of a TSV without reading rows."""
@@ -21,6 +30,15 @@ def _header_columns(path: str | os.PathLike) -> list[str]:
     return [c.lower() for c in cols]
 
 
+def _peek(path: str | os.PathLike, n: int = 8) -> str:
+    """Return the first ``n`` characters (gzip-transparent), for JSON detection."""
+    with open(path, "rb") as fb:
+        gz = fb.read(2) == b"\x1f\x8b"
+    opener = gzip.open if gz else open
+    with opener(path, "rt", errors="ignore") as f:
+        return f.read(n)
+
+
 def sniff_format(path: str | os.PathLike) -> str:
     """Detect a clonotype file's format from its header.
 
@@ -39,10 +57,26 @@ def sniff_format(path: str | os.PathLike) -> str:
     """
     if Path(path).suffix.lower() in (".parquet", ".pq"):
         return "parquet"
+    if Path(path).suffix.lower() in (".vidjil", ".json") or _peek(path).lstrip()[:1] == "{":
+        return "vidjil"  # Vidjil .vidjil JSON
     cols = set(_header_columns(path))
+    # Specific third-party formats first (their signature columns don't collide with each other).
+    if "v-gene and allele" in cols and "junction" in cols:
+        return "imgt"
+    if "cdr3 nucleotide sequence" in cols and "v segments" in cols:
+        return "migec"
+    if "number of reads" in cols and "junction nucleotide sequence" in cols:
+        return "rtcr"
+    if cols & {"all v hits", "allvhitswithscore"} and \
+            cols & {"n. seq. cdr3", "nseqcdr3", "nseqimputedcdr3"}:
+        return "mixcr"
+    if "count (templates/reads)" in cols or ({"rearrangement", "amino_acid"} <= cols
+                                             and "v_gene" in cols):
+        return "immunoseq"
+    # Native vdjtools / migmap (cdr3nt/cdr3aa headers) and AIRR Rearrangement.
     if "cdr3aa" in cols or {"count", "cdr3nt"} <= cols:
         return "vdjtools"
-    if cols & {"v_call", "junction_aa", "cdr3_aa"}:
+    if cols & {"v_call", "junction_aa", "junction_nt", "cdr3_aa"}:
         return "airr"
     raise ValueError(f"unrecognised clonotype format; header columns: {sorted(cols)}")
 
@@ -72,7 +106,14 @@ def read(path: str | os.PathLike, fmt: str = "auto",
         return read_airr(path, n_rows=n_rows)
     if fmt == "parquet":
         return read_parquet(path, n_rows=n_rows)
-    raise ValueError(f"fmt must be 'auto', 'vdjtools', 'airr' or 'parquet'; got {fmt!r}")
+    if fmt == "vidjil":
+        return convert.read_vidjil(path)  # whole-JSON reader (no row cap)
+    if fmt in _CONVERTERS:
+        return _CONVERTERS[fmt](path, n_rows=n_rows)
+    raise ValueError(
+        f"fmt must be 'auto', 'vdjtools', 'airr', 'parquet', or a legacy format "
+        f"({', '.join(sorted(_CONVERTERS))}); got {fmt!r}"
+    )
 
 
 def read_metadata(path: str | os.PathLike) -> pl.DataFrame: