migec 2.2.0__tar.gz → 2.2.1__tar.gz

{migec-2.2.0 → migec-2.2.1}/CHANGELOG.md

@@ -4,6 +4,88 @@ Hand-written and prose-heavy: each entry says what changed and, where it matters
 prevents. Releases before 2.0.0 are the Groovy MIGEC and are described by their git tags on the
 `legacy-v1` branch.
 
+## 2.2.1 — 2026-08-14
+
+Documentation, pipelines and tooling. Note: **the wheel is unchanged from 2.2.0** — no command,
+flag, output file or number that migec itself produces is different. Everything below is in the
+repository: the SLURM templates, the Nextflow downstream layer, two more notebooks, a new docs
+page, and the two scripts behind it.
+
+### Pipelines you can actually run
+
+`integrations/slurm/` is new: `migec_sample.sbatch` (one sample, environment-driven) and
+`migec_array.sbatch` (one array task per sample-sheet row), with an example sheet and a README that
+sizes the request per stage. Both run as **ordinary bash without SLURM** — every SLURM variable has
+a fallback — which is how they are tested here and how a layout should be checked before a cohort
+is queued. Note: array task 1 is the first *data* row; the header is skipped rather than counted,
+so the range is `1-(rows - 1)`, and an `--array=0-N` runs a task that reads the header as a sample.
+
+`integrations/nextflow/` gained a runnable entry point (`--mode consensus | ctdna | airr`) and a
+downstream layer: `modules/downstream/{align,callvariants,arda}` plus `subworkflows/migec_ctdna`
+and `subworkflows/migec_airr`. The align module **checks its own output** — if no `MI:Z:` tag
+reaches the BAM it exits non-zero and names the flag each aligner needs, because that failure is
+otherwise silent and surfaces much later as an inexplicably untagged BAM.
+
+### Notebooks
+
+`notebooks/README.md`, plus `exome_capture.py` (why coordinate deduplication undercounts a capture
+panel) and `airr_repertoire.py` (clonotype counts from reads against molecules). Both are
+simulated, so both run offline against a known answer. Never: the PEP 723 headers were
+**under-declared** — altair renders a polars frame through pyarrow and then pandas, so a clean
+machine failed three imports deep with an error naming neither. Every header is now what the
+notebook actually needs, and all of them were executed to prove it.
+
+### Which variant caller, and the count that decides it
+
+`docs/variants.rst` answers the question people arrive with — *rare variants out of ctDNA, what do
+I run after `assemble`?* — and extends `docs/downstream.rst`'s transport-vs-deduplicate rule one
+level: `Mutect2`, `LoFreq`, `FreeBayes`, `VarDict` and `bcftools` read a BAM and ignore `RX`, so
+they **compose**; `UMI-VarCal`, `UMIErrorCorrect`, `DREAMS-vc` and `Shearwater` group and consense
+themselves, so they **replace** a stage. Never: do not apply a UMI-aware caller's family-size
+filter to a consensus — after `assemble` every family has size 1 by construction, so
+`--min-family-size 3` discards the entire library and reports zero variants without erroring.
+
+The caller ranking is quoted from Maruzani et al. 2024 (BMC Genomics,
+[doi:10.1186/s12864-024-10737-w](https://doi.org/10.1186/s12864-024-10737-w)), which is the only
+independent comparison of the six. The page says which half of it transfers and why.
+
+### The ctDNA ground truth was found, not built
+
+`ROADMAP.md` item 7 said this had to be constructed because Maruzani's deposited runs carry no UMI.
+That is true of those two runs and not of the public record. Screening SRA read structure instead
+turned up **`PRJNA788522`** (72 runs, cell-free DNA reference material at certified 0 / 0.125 /
+0.25 / 1% VAF, crossed with 5/20/80 ng input and 3.3/10/30x depth, three replicates) and
+**`PRJNA507366`** (28 runs, six polymerases on the same material plus 0.031% and 0.0625% VAF).
+Both kept a real 12 nt inline UMI, which `migec suggest` recovers from base composition alone.
+
+Note: `PRJNA507366`'s design is in `library_name`, not `sample_alias` — every alias there reads
+`SeraCare_Reference_Material`, so reading the alias alone would have called the study undesigned.
+
+### `scripts/sra_fetch.py` — fetch on demand instead of mirroring
+
+`probe` reports reads-per-spot and their lengths from metadata alone, which is the cheap "did the
+UMI survive deposition" test; `url` gives the direct S3/HTTPS links with md5 and size; `peek`
+streams the first N spots for `migec suggest`; `get` downloads and converts. Never: the
+obvious-looking source is the slow one — ENA serves ready-made FASTQ and skips `fasterq-dump`, but
+caps at one connection. Measured on `SRR17220895`: **ENA 200 kB/s against NCBI S3 at 6.7 MB/s over
+8 connections**, so S3 is the default and `--prefer ena` is there for the studies that deposited a
+third file the `.sra` object folds away.
+
+This is what keeps `isalgo/umi_data` small: anything with a public accession is regenerated by a
+one-line command recorded in `SOURCES.md`, and only the CI fixtures and the semi-internal data that
+has nowhere else to live are mirrored.
+
+### `scripts/ctdna_titration.py` and `notebooks/ctdna_variants.py`
+
+The three stages over the titration, reporting molecules per site against the certified frequency.
+Never: **the molecule total of a multiplex panel is not the count at any one site** — a variant sits
+on one amplicon, so the total overstates the evidence by exactly the panel size. The amplicon count
+is measured from consensus prefixes rather than assumed, and its share threshold has to sit in the
+gap below the smallest real amplicon: at 1% the count moved with depth (5 amplicons on the deepest
+run, 10 on the shallowest) because a shallow consensus carries more payload error, which deflated
+the per-amplicon count on exactly the runs where the evidence was thinnest. The script warns when
+the count is not constant across runs, which is the check that the threshold is in the gap.
+
 ## 2.2.0 — 2026-08-13
 
 Throughput and one new measurement. No breaking change: every 2.1.0 command, flag and output file

{migec-2.2.0 → migec-2.2.1}/CLAUDE.md

@@ -465,12 +465,35 @@ correction is written up in `project/review-algorithms.md`.
   5. **i7xi5 contingency table** — needs nothing that is not already in the headers.
   6. **`2026-migec-benchmark`** and the published comparisons. This is what the version number is
      waiting on, not the code.
-  7. **ctDNA ground truth** (COSMIC at 0.005-0.075 VAF). Must be built: Maruzani's deposited runs
-     carry no UMI.
+  7. **Run the callers themselves** against the ctDNA ground truth below. It no longer has to be
+     built -- only the call sets are missing.
   8. **R1/R2 overlap merge**, as a case of `--contig`'s placement and never a second matcher.
   9. **Bit-parallel matcher**, last and deliberately: the scan is not the bottleneck.
+- **The ctDNA ground truth was FOUND, not built (2026-08-13).** `PRJNA788522` (72 runs, cfDNA
+  reference material at certified 0/0.125/0.25/1% VAF x 5/20/80 ng x 3.3/10/30x, 3 reps) and
+  `PRJNA507366` (28 runs, six polymerases plus 0.031%/0.0625% VAF) both kept a **real 12 nt inline
+  UMI**, and `migec suggest` recovers it from base composition alone. Never: "no public ctDNA data
+  has usable UMIs" is a statement about the two runs Maruzani picked, not about the archive --
+  check read structure (`scripts/sra_fetch.py probe`) before believing it. Note: `PRJNA507366`'s
+  design is in **`library_name`**, not `sample_alias`; every alias there is
+  `SeraCare_Reference_Material`, so the alias alone makes a designed study look undesigned.
+- **Never: the molecule total of a multiplex panel is not the count at a site.** A variant sits on
+  one amplicon, so the evidence a caller gets is `molecules / amplicons`. The amplicon count is
+  measured from consensus prefixes, and its share threshold must sit in the **gap** below the
+  smallest real amplicon (5%, where the gap is 7.6% to 1.1%): at 1% the count moved with depth --
+  5 on the deepest run, 10 on the shallowest, because a shallow consensus carries more payload
+  error -- which deflates the per-amplicon count on exactly the runs where evidence is thinnest.
+- **Note: `scripts/sra_fetch.py` is the fetch-on-demand path**, so `isalgo/umi_data` holds only CI
+  fixtures and semi-internal data. Never: ENA's ready-made FASTQ looks like the lazy source and is
+  33x slower -- **200 kB/s against NCBI S3's 6.7 MB/s** on 8 connections, measured on SRR17220895.
+  S3 plus `fasterq-dump` is the default; `--prefer ena` exists for studies that deposited a third
+  file the `.sra` folds away.
 - **Note: Britanova et al aging (bulk TCR, shallow) lives on aldan3** and is the real dataset for the
   1-3 read regime. Not pulled yet. aldan3 compute goes through SLURM, never the frontend.
+- **Note: the ctDNA titration runs on aldan3** (`~/migec-ctdna`, `migec_ctdna.sbatch`, partition
+  `medium`, 16 cores / 32 GB). The job builds its own micromamba env because the cluster's system
+  python is **3.8** and migec needs >= 3.10, and it installs from PyPI rather than from source,
+  which doubles as a check that the published linux wheels work.
 - The archive is pushed: `legacy-v1` + tag `v1-final`, and master is the rewrite. Recovery point
   `~/backup/migec-local-mirror-2026-08-13.git`. Note: The canonical repo is **antigenomics/migec**;
   `mikessh/migec` is a redirect, and `gh` commands must use the former.

{migec-2.2.0 → migec-2.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: migec
-Version: 2.2.0
+Version: 2.2.1
 Summary: UMI barcode extraction, correction and consensus assembly for barcoded sequencing
 Keywords: umi,barcode,consensus,airr,repseq,single-cell,sequencing
 Author-Email: Mikhail Shugay <mikhail.shugay@gmail.com>
@@ -51,10 +51,10 @@ Computational Biology* 2017).
 
 > **Version 2 is under construction.** All three stages work today — `checkout`, `refine` and
 > `assemble` — with cell barcodes, whitelists, dual-end and positional (10x) layouts, cell calling,
-> QC figures, and `suggest`/`subsample`/`plot`. Index hopping, `.mig` bucket output and the published benchmark
-> comparisons are what remain; see [`ROADMAP.md`](ROADMAP.md). The Groovy MIGEC 1.2.9 is archived on
-> branch [`legacy-v1`](../../tree/legacy-v1) and at tag `v1-final` — Java users want the jars on the
-> [1.2.9 release](../../releases/tag/1.2.9).
+> QC figures, and `suggest`/`subsample`/`plot`. Index hopping, `.mig` bucket output and the
+> published benchmark comparisons are what remain; see [`ROADMAP.md`](ROADMAP.md). The Groovy
+> MIGEC 1.2.9 is archived on branch [`legacy-v1`](../../tree/legacy-v1) and at tag `v1-final` —
+> Java users want the jars on the [1.2.9 release](../../releases/tag/1.2.9).
 
 ## Why
 
@@ -105,13 +105,49 @@ one buys.
 > **Never** run alevin, bustools or STARsolo on a consensus FASTQ. They read the barcode out of a
 > *raw* barcode read and deduplicate themselves; migec already did, and that read no longer exists.
 
+### Calling rare variants
+
+The same line decides which variant caller to use. `Mutect2`, `LoFreq`, `FreeBayes` and `VarDict`
+read a BAM and never look at `RX`, so they **compose**: one consensus is one molecule, so their
+depth already is a molecule count. `UMI-VarCal` and `UMIErrorCorrect` do their own grouping and
+consensus, so they **replace** `assemble` rather than following it — and a `--min-family-size 3`
+filter after `assemble` discards the whole library, because every family already has size 1.
+
+Which caller matters less than how many molecules it is handed, and that is fixed before any
+software runs: sequencing deeper buys more reads per molecule, only more input DNA buys molecules.
+[docs/variants.rst](docs/variants.rst) has the caller table, the published comparison, and the run
+over cell-free DNA reference material at known allele frequency that puts numbers on it.
+
 ## Install
 
 ```bash
-pip install migec
+uv tool install migec      # or: uv pip install migec, or: pip install migec
+migec info                 # prints the three version strings; they must agree
+```
+
+Wheels for CPython 3.10–3.13 on Linux x86-64 and macOS arm64, so nothing compiles. Python 3.10 or
+newer is required — on a cluster whose system Python is older, bring your own:
+
+```bash
+uv venv --python 3.12 ~/envs/migec && source ~/envs/migec/bin/activate && uv pip install migec
+```
+
+**From source**, for development:
+
+```bash
+git clone https://github.com/antigenomics/migec && cd migec
+bash setup.sh              # uv venv, editable install, and asserts the C++ extension imports
 ```
 
-Wheels for CPython 3.10–3.13 on Linux x86-64 and macOS arm64. From source: `bash setup.sh`.
+`setup.sh` needs [uv](https://docs.astral.sh/uv/getting-started/installation/) and a C++20
+compiler. It deliberately asserts that `migec._core` imports rather than only that the package
+does — without that check a failed C++ build looks like a successful install and only fails much
+later. Then:
+
+```bash
+cmake -S . -B build -DMIGEC_TESTS=ON && cmake --build build -j && ctest --test-dir build
+python -m pytest tests/unit tests/synthetic -q
+```
 
 ## Usage
 
@@ -535,6 +571,30 @@ It is also the memory-hostile case, because distinct barcodes are what everythin
 it is what the benchmarks use: 1,179,549 reads/s at 1.02 reads/UMI, 282 B resident per distinct
 barcode, still bounded by the bucket rather than the library.
 
+### For rare variants, the molecule count decides — not the caller
+
+Run over 100 runs of two SiMSen-Seq studies on cell-free DNA reference material at *certified*
+allele frequencies (`PRJNA788522`, `PRJNA507366`; `scripts/ctdna_titration.py`). At 0.125% VAF the
+DNA input alone settles the outcome:
+
+| input | depth | molecules at the site | variant molecules | P(at least 3 copies) |
+|---|---|---|---|---|
+| 5 ng | 3.3× | 2,699 | 3.4 | **0.65** |
+| 5 ng | 10× | 4,530 | 5.7 | **0.89** |
+| 20 ng | 3.3× | 7,162 | 9.0 | 0.99 |
+| 20 ng | 10× | 11,961 | 15.0 | 1.00 |
+
+At 5 ng a 0.125% variant is a coin flip and **no caller fixes it** — a third of replicates simply do
+not contain three copies. The decision that mattered was made at extraction.
+[docs/variants.rst](docs/variants.rst) has the caller table (standard callers *compose* with migec,
+UMI-aware ones *replace* a stage), the 33-run true-negative arm, and the rest of the numbers.
+
+The same data gives an independent check on the barcode-error estimate: `PRJNA507366` varies the
+polymerase while holding template and protocol fixed, and `refine` separates the high-fidelity
+enzymes from the standard ones by **3 Phred, a factor of two** — Accuprime/Platinum/Phusion at
+Q30.8–31.4 against Platinum HiFi at Q34.0 and Platinum SuperFi at Q34.8–35.1 — from the barcodes
+alone, with no reference, no alignment, and no knowledge of which enzyme it was looking at.
+
 ### Speed and memory are reported, not assumed
 
 `--threads` defaults to one per core on **all three stages**, and **the output is byte-identical
@@ -646,21 +706,67 @@ distinct molecules. Both are reported; only the collision form feeds any decisio
 
 ## Pipelines
 
-`integrations/nextflow/` is an nf-core-style local module set — four processes, a subworkflow,
-`meta.yml`, `nextflow.config`, `environment.yml` — that drops into
-[nf-core/airrflow](https://nf-co.re/airrflow) or anything else that hands you FASTQ pairs. SLURM is
-the pipeline's business, not the module's: it declares `label` and `task.cpus` and nothing more.
+Two, for two situations ([docs/nextflow.rst](docs/nextflow.rst) has both):
+
+**Nextflow** — `integrations/nextflow/` runs the three stages and then keeps going:
+
+```bash
+nextflow run integrations/nextflow --mode ctdna --input 'd/*_R{1,2}.fq.gz' --preset tso500 --fasta ref.fa
+nextflow run integrations/nextflow --mode airr  --input 'd/*_R{1,2}.fq.gz' --preset migec
+nextflow run integrations/nextflow --mode consensus --input 'd/*_R{1,2}.fq.gz' --preset 10x-v2
+```
+
+`ctdna` aligns and calls variants, `airr` calls clonotypes with arda, `consensus` stops at the
+consensus. It drops into [nf-core/airrflow](https://nf-co.re/airrflow) or anything else that hands
+you FASTQ pairs. The align module fails loudly if the `MI:Z:` tag does not reach the BAM, because
+that failure is otherwise silent.
+
+**SLURM** — `integrations/slurm/` is two sbatch templates and a sample sheet, for a cohort where
+the consensus *is* the deliverable:
+
+```bash
+sbatch --array=1-$(($(wc -l < samples.tsv) - 1)) integrations/slurm/migec_array.sbatch samples.tsv
+R1=s1.fq.gz SAMPLE=s1 BC_PATTERN='0:12' bash integrations/slurm/migec_sample.sbatch   # no SLURM
+```
+
+Both run as ordinary bash without SLURM — that is how they are tested, and how a layout should be
+checked before a cohort is queued.
 
 All three stages thread, and each is byte-identical at any `-t`, so a retry with different cores
-cannot change a result — which is what makes an escalating `errorStrategy 'retry'` safe here.
-Per-sample keys in `meta` beat the `params.migec_*` defaults, so one run can mix chemistries. See
-[docs/nextflow.rst](docs/nextflow.rst); nextflow is not installed on the machine these docs were
-measured on, so the modules are reviewed against the nf-core spec rather than verified by a run.
+cannot change a result — which is what makes an escalating `errorStrategy 'retry'` or `--requeue`
+safe. Per-sample keys in `meta` beat the `params.*` defaults, so one run can mix chemistries.
+
+> The SLURM templates have been run end to end here. Nextflow has not — it is not installed on this
+> machine, so the modules are reviewed against the nf-core spec rather than verified by a run.
+
+## Notebooks
+
+Six [marimo](https://marimo.io) notebooks, each a plain Python file with its own dependency header,
+so `uv` builds the environment and nothing needs installing first:
+
+```bash
+uv run marimo edit notebooks/ctdna_variants.py
+```
+
+`platforms.py` (every layout), `barcode_space.py` (collisions and the error budget),
+`refine_diagnostics.py` (where the molecules went), `exome_capture.py` (why coordinate
+deduplication undercounts), `airr_repertoire.py` (reads against molecules for clonotype counts),
+`ctdna_variants.py` (how many molecules a caller actually gets). See
+[notebooks/README.md](notebooks/README.md); four of the six need no network at all.
 
 ## Documentation
 
-<https://antigenomics.github.io/migec/> — see [`docs/formats.rst`](docs/formats.rst) for the on-disk
-format, and [`ROADMAP.md`](ROADMAP.md) for what is implemented.
+<https://antigenomics.github.io/migec/>. The pages worth knowing about:
+
+| | |
+|---|---|
+| [`docs/layouts.rst`](docs/layouts.rst) | where the barcode is: the grammar, every preset, fgbio read structures |
+| [`docs/variants.rst`](docs/variants.rst) | which variant caller, and the molecule count that decides whether any can see it |
+| [`docs/downstream.rst`](docs/downstream.rst) | what consumes the consensus, measured tool by tool |
+| [`docs/nextflow.rst`](docs/nextflow.rst) | the Nextflow modules and the SLURM templates |
+| [`docs/formats.rst`](docs/formats.rst) | the inter-stage contract: tags, `.mig` buckets, every TSV column |
+| [`SOURCES.md`](SOURCES.md) | every dataset, where it came from, and the command that re-fetches it |
+| [`ROADMAP.md`](ROADMAP.md) | what is implemented and what is not |
 
 ## Citing
 

{migec-2.2.0 → migec-2.2.1}/README.md

@@ -17,10 +17,10 @@ Computational Biology* 2017).
 
 > **Version 2 is under construction.** All three stages work today — `checkout`, `refine` and
 > `assemble` — with cell barcodes, whitelists, dual-end and positional (10x) layouts, cell calling,
-> QC figures, and `suggest`/`subsample`/`plot`. Index hopping, `.mig` bucket output and the published benchmark
-> comparisons are what remain; see [`ROADMAP.md`](ROADMAP.md). The Groovy MIGEC 1.2.9 is archived on
-> branch [`legacy-v1`](../../tree/legacy-v1) and at tag `v1-final` — Java users want the jars on the
-> [1.2.9 release](../../releases/tag/1.2.9).
+> QC figures, and `suggest`/`subsample`/`plot`. Index hopping, `.mig` bucket output and the
+> published benchmark comparisons are what remain; see [`ROADMAP.md`](ROADMAP.md). The Groovy
+> MIGEC 1.2.9 is archived on branch [`legacy-v1`](../../tree/legacy-v1) and at tag `v1-final` —
+> Java users want the jars on the [1.2.9 release](../../releases/tag/1.2.9).
 
 ## Why
 
@@ -71,13 +71,49 @@ one buys.
 > **Never** run alevin, bustools or STARsolo on a consensus FASTQ. They read the barcode out of a
 > *raw* barcode read and deduplicate themselves; migec already did, and that read no longer exists.
 
+### Calling rare variants
+
+The same line decides which variant caller to use. `Mutect2`, `LoFreq`, `FreeBayes` and `VarDict`
+read a BAM and never look at `RX`, so they **compose**: one consensus is one molecule, so their
+depth already is a molecule count. `UMI-VarCal` and `UMIErrorCorrect` do their own grouping and
+consensus, so they **replace** `assemble` rather than following it — and a `--min-family-size 3`
+filter after `assemble` discards the whole library, because every family already has size 1.
+
+Which caller matters less than how many molecules it is handed, and that is fixed before any
+software runs: sequencing deeper buys more reads per molecule, only more input DNA buys molecules.
+[docs/variants.rst](docs/variants.rst) has the caller table, the published comparison, and the run
+over cell-free DNA reference material at known allele frequency that puts numbers on it.
+
 ## Install
 
 ```bash
-pip install migec
+uv tool install migec      # or: uv pip install migec, or: pip install migec
+migec info                 # prints the three version strings; they must agree
+```
+
+Wheels for CPython 3.10–3.13 on Linux x86-64 and macOS arm64, so nothing compiles. Python 3.10 or
+newer is required — on a cluster whose system Python is older, bring your own:
+
+```bash
+uv venv --python 3.12 ~/envs/migec && source ~/envs/migec/bin/activate && uv pip install migec
+```
+
+**From source**, for development:
+
+```bash
+git clone https://github.com/antigenomics/migec && cd migec
+bash setup.sh              # uv venv, editable install, and asserts the C++ extension imports
 ```
 
-Wheels for CPython 3.10–3.13 on Linux x86-64 and macOS arm64. From source: `bash setup.sh`.
+`setup.sh` needs [uv](https://docs.astral.sh/uv/getting-started/installation/) and a C++20
+compiler. It deliberately asserts that `migec._core` imports rather than only that the package
+does — without that check a failed C++ build looks like a successful install and only fails much
+later. Then:
+
+```bash
+cmake -S . -B build -DMIGEC_TESTS=ON && cmake --build build -j && ctest --test-dir build
+python -m pytest tests/unit tests/synthetic -q
+```
 
 ## Usage
 
@@ -501,6 +537,30 @@ It is also the memory-hostile case, because distinct barcodes are what everythin
 it is what the benchmarks use: 1,179,549 reads/s at 1.02 reads/UMI, 282 B resident per distinct
 barcode, still bounded by the bucket rather than the library.
 
+### For rare variants, the molecule count decides — not the caller
+
+Run over 100 runs of two SiMSen-Seq studies on cell-free DNA reference material at *certified*
+allele frequencies (`PRJNA788522`, `PRJNA507366`; `scripts/ctdna_titration.py`). At 0.125% VAF the
+DNA input alone settles the outcome:
+
+| input | depth | molecules at the site | variant molecules | P(at least 3 copies) |
+|---|---|---|---|---|
+| 5 ng | 3.3× | 2,699 | 3.4 | **0.65** |
+| 5 ng | 10× | 4,530 | 5.7 | **0.89** |
+| 20 ng | 3.3× | 7,162 | 9.0 | 0.99 |
+| 20 ng | 10× | 11,961 | 15.0 | 1.00 |
+
+At 5 ng a 0.125% variant is a coin flip and **no caller fixes it** — a third of replicates simply do
+not contain three copies. The decision that mattered was made at extraction.
+[docs/variants.rst](docs/variants.rst) has the caller table (standard callers *compose* with migec,
+UMI-aware ones *replace* a stage), the 33-run true-negative arm, and the rest of the numbers.
+
+The same data gives an independent check on the barcode-error estimate: `PRJNA507366` varies the
+polymerase while holding template and protocol fixed, and `refine` separates the high-fidelity
+enzymes from the standard ones by **3 Phred, a factor of two** — Accuprime/Platinum/Phusion at
+Q30.8–31.4 against Platinum HiFi at Q34.0 and Platinum SuperFi at Q34.8–35.1 — from the barcodes
+alone, with no reference, no alignment, and no knowledge of which enzyme it was looking at.
+
 ### Speed and memory are reported, not assumed
 
 `--threads` defaults to one per core on **all three stages**, and **the output is byte-identical
@@ -612,21 +672,67 @@ distinct molecules. Both are reported; only the collision form feeds any decisio
 
 ## Pipelines
 
-`integrations/nextflow/` is an nf-core-style local module set — four processes, a subworkflow,
-`meta.yml`, `nextflow.config`, `environment.yml` — that drops into
-[nf-core/airrflow](https://nf-co.re/airrflow) or anything else that hands you FASTQ pairs. SLURM is
-the pipeline's business, not the module's: it declares `label` and `task.cpus` and nothing more.
+Two, for two situations ([docs/nextflow.rst](docs/nextflow.rst) has both):
+
+**Nextflow** — `integrations/nextflow/` runs the three stages and then keeps going:
+
+```bash
+nextflow run integrations/nextflow --mode ctdna --input 'd/*_R{1,2}.fq.gz' --preset tso500 --fasta ref.fa
+nextflow run integrations/nextflow --mode airr  --input 'd/*_R{1,2}.fq.gz' --preset migec
+nextflow run integrations/nextflow --mode consensus --input 'd/*_R{1,2}.fq.gz' --preset 10x-v2
+```
+
+`ctdna` aligns and calls variants, `airr` calls clonotypes with arda, `consensus` stops at the
+consensus. It drops into [nf-core/airrflow](https://nf-co.re/airrflow) or anything else that hands
+you FASTQ pairs. The align module fails loudly if the `MI:Z:` tag does not reach the BAM, because
+that failure is otherwise silent.
+
+**SLURM** — `integrations/slurm/` is two sbatch templates and a sample sheet, for a cohort where
+the consensus *is* the deliverable:
+
+```bash
+sbatch --array=1-$(($(wc -l < samples.tsv) - 1)) integrations/slurm/migec_array.sbatch samples.tsv
+R1=s1.fq.gz SAMPLE=s1 BC_PATTERN='0:12' bash integrations/slurm/migec_sample.sbatch   # no SLURM
+```
+
+Both run as ordinary bash without SLURM — that is how they are tested, and how a layout should be
+checked before a cohort is queued.
 
 All three stages thread, and each is byte-identical at any `-t`, so a retry with different cores
-cannot change a result — which is what makes an escalating `errorStrategy 'retry'` safe here.
-Per-sample keys in `meta` beat the `params.migec_*` defaults, so one run can mix chemistries. See
-[docs/nextflow.rst](docs/nextflow.rst); nextflow is not installed on the machine these docs were
-measured on, so the modules are reviewed against the nf-core spec rather than verified by a run.
+cannot change a result — which is what makes an escalating `errorStrategy 'retry'` or `--requeue`
+safe. Per-sample keys in `meta` beat the `params.*` defaults, so one run can mix chemistries.
+
+> The SLURM templates have been run end to end here. Nextflow has not — it is not installed on this
+> machine, so the modules are reviewed against the nf-core spec rather than verified by a run.
+
+## Notebooks
+
+Six [marimo](https://marimo.io) notebooks, each a plain Python file with its own dependency header,
+so `uv` builds the environment and nothing needs installing first:
+
+```bash
+uv run marimo edit notebooks/ctdna_variants.py
+```
+
+`platforms.py` (every layout), `barcode_space.py` (collisions and the error budget),
+`refine_diagnostics.py` (where the molecules went), `exome_capture.py` (why coordinate
+deduplication undercounts), `airr_repertoire.py` (reads against molecules for clonotype counts),
+`ctdna_variants.py` (how many molecules a caller actually gets). See
+[notebooks/README.md](notebooks/README.md); four of the six need no network at all.
 
 ## Documentation
 
-<https://antigenomics.github.io/migec/> — see [`docs/formats.rst`](docs/formats.rst) for the on-disk
-format, and [`ROADMAP.md`](ROADMAP.md) for what is implemented.
+<https://antigenomics.github.io/migec/>. The pages worth knowing about:
+
+| | |
+|---|---|
+| [`docs/layouts.rst`](docs/layouts.rst) | where the barcode is: the grammar, every preset, fgbio read structures |
+| [`docs/variants.rst`](docs/variants.rst) | which variant caller, and the molecule count that decides whether any can see it |
+| [`docs/downstream.rst`](docs/downstream.rst) | what consumes the consensus, measured tool by tool |
+| [`docs/nextflow.rst`](docs/nextflow.rst) | the Nextflow modules and the SLURM templates |
+| [`docs/formats.rst`](docs/formats.rst) | the inter-stage contract: tags, `.mig` buckets, every TSV column |
+| [`SOURCES.md`](SOURCES.md) | every dataset, where it came from, and the command that re-fetches it |
+| [`ROADMAP.md`](ROADMAP.md) | what is implemented and what is not |
 
 ## Citing
 

{migec-2.2.0 → migec-2.2.1}/ROADMAP.md

@@ -43,13 +43,14 @@ Everything below is either open or half-open. Nothing else on this page is.
 | 4 | `--rt-error auto` | M1 | falls straight out of (3). The floor is a property of the enzyme and the cycle count, so fitting it per dataset is the honest default once there is something to fit against |
 | 5 | i7xi5 contingency table | M2 | the only way index hopping is actually estimable, and it needs nothing that is not already in the headers |
 | 6 | `2026-migec-benchmark`, the published comparisons | M5 | MIGEC v1, MAGERI, UMI-tools, Calib, fgbio, Cell Ranger, UMI-VarCal, UMIErrorCorrect. This is what the version number is waiting on, not the code |
-| 7 | ctDNA ground truth, COSMIC at 0.005-0.075 VAF | M5 | the one claim with no ground truth yet. Never: Maruzani's deposited runs carry **no** UMI, so they have to be simulated -- and their own rule assumes co-terminal reads, which X1 falsified |
+| 7 | ctDNA ground truth | M5 | **no longer has to be built.** `PRJNA788522` and `PRJNA507366` are cfDNA reference material at certified VAF (0 / 0.031 / 0.0625 / 0.125 / 0.25 / 1%) with a **real 12 nt inline UMI that survived deposition**, and a true-negative `WT` arm. Maruzani's two runs carry no UMI and had to be simulated; these do not. `SOURCES.md`, `docs/variants.rst` |
 | 8 | R1/R2 overlap merge | M1 | a special case of `--contig`'s placement, never a second matcher in `checkout` |
 | 9 | Bit-parallel matcher | M2 | last, deliberately: the scan is O(offsets x pattern) and is **not** the bottleneck. It goes in when a benchmark says so |
 
-Two things are blocked on data rather than on work: **Britanova et al aging** (bulk TCR, shallow --
-the real 1-3 reads/UMI dataset) lives on aldan3 and has not been pulled, and the ctDNA ground truth
-above has to be built rather than downloaded.
+One thing is still blocked on data rather than on work: **Britanova et al aging** (bulk TCR,
+shallow -- the real 1-3 reads/UMI dataset) lives on aldan3 and has not been pulled. The ctDNA
+ground truth is no longer on that list: it was found rather than built, by screening SRA read
+structure instead of trusting the published claim that none exists (`scripts/sra_fetch.py probe`).
 
 ## M0 — skeleton, format, simulator
 
@@ -299,11 +300,16 @@ model has to use the evidence that survives at one read:
 - [ ] `2026-migec-benchmark` repo, `isalgo/umi_data`, comparisons against MIGEC v1, MAGERI,
       UMI-tools, Calib, fgbio, Cell Ranger, **UMI-VarCal and UMIErrorCorrect** (the two UMI-aware
       callers benchmarked by Maruzani et al. 2024 for low-frequency ctDNA)
-- [ ] ctDNA ground truth in the style of Maruzani et al.: COSMIC variants spiked at 0.005–0.075 VAF
-      into a real cfDNA background at 200x/450x/850x. Never: Their deposited runs carry **no** UMI
-      (aligned BAM submissions, `suggest` finds no pattern, `CMP_LINKAGE_GROUP` empty), so the UMIs
-      have to be simulated — and their rule assigns them to reads sharing start *and* end, which is
-      the co-terminal assumption X1 falsified
+- [x] ctDNA ground truth **located, not simulated** (2026-08-13): `PRJNA788522` (72 runs, cfDNA
+      reference material at 0 / 0.125 / 0.25 / 1% VAF x 5/20/80 ng x 3.3/10/30x, three replicates)
+      and `PRJNA507366` (28 runs, six polymerases plus 0.031% / 0.0625% VAF). Both carry a real
+      12 nt inline UMI. Never: Maruzani's runs carry **no** UMI (aligned BAM submissions, `suggest`
+      finds no pattern, `CMP_LINKAGE_GROUP` empty), so theirs had to be simulated — 9 nt, Phred
+      fixed at 37, assigned to reads sharing start *and* end, which is the co-terminal assumption
+      X1 falsified. That is a property of the two runs they picked, not of the public record
+- [ ] Variant calls on top of it: run the callers themselves (LoFreq, Mutect2, UMI-VarCal,
+      UMIErrorCorrect) against the `WT` true-negative arm and the certified dilutions. `docs/variants.rst`
+      reports the molecule counts and the detectability arithmetic; the call sets are the next step
 - Gate: grouping ARI ≥0.99; residual error ≤1e-5 on a clonal control; ≥3× MIGEC v1 wall-clock
 
 ## Deliberately not doing

{migec-2.2.0 → migec-2.2.1}/SOURCES.md

@@ -16,6 +16,7 @@ are never conflated in a table row.
 | `assets/benchmark_threads.tsv` + `.json` | this machine, 2026-08-13 | **experimental** (a timing measurement) | `python scripts/benchmark_threads.py --reads 2000000 -o assets/` |
 | `assets/SRR1763769.mig.tsv`, `assets/assemble.coverage.tsv` | `isalgo/umi_data` CI fixture | derived | `migec refine ci/SRR1763769_umi0.5pct.fq.gz -o r/ && migec assemble r/CTRL.fq.gz -o a/` |
 | `assets/*.svg`, `assets/*.gp` | the tables beside them | derived | `migec plot assets/ -o assets/` |
+| `assets/ctdna_titration.tsv` (+ `_runs.tsv`) | 100 runs of `PRJNA788522` + `PRJNA507366` | derived (migec over experimental data) | `python scripts/sra_fetch.py get <runs> -o simsen/` then `python scripts/ctdna_titration.py --reads simsen/ --out ctdna/ --design design.tsv` |
 
 Result tables and figures are output, not data, so they live here next to the script that made
 them rather than in `isalgo/umi_data`. Test corpora go the other way: HuggingFace, never git.
@@ -158,6 +159,52 @@ numbers inherits it.
 background at controlled depth is the ground truth a consensus-quality claim wants — and two more
 comparators for M5, `UMI-VarCal` and `UMIErrorCorrect`.
 
+### SiMSen-Seq — the ctDNA data that DID keep its UMIs (fetched 2026-08-13)
+
+Found by screening SRA rather than by assuming, after the Maruzani entry above established that
+their two accessions had not. Both studies are the Gothenburg SiMSen-Seq protocol on **commercial
+cell-free DNA reference material with certified mutant allele frequencies**, and both deposited
+the reads untrimmed, so the **12 nt inline UMI is still there**. `migec suggest` recovers it from
+base composition alone with no knowledge of the protocol — a 12 nt near-uniform run at cycles
+0-11, then the constant 16 nt spacer (`ATGGGAAAGAGTGTCC`), which is exactly UMIErrorCorrect's
+`-ul 12 -sl 16`.
+
+| Item | Value |
+|---|---|
+| Pattern | `--bc-pattern '0:12'`, single-end, positional (no `--max-offset`) |
+| Panel | multiplex amplicon, measured from the consensus prefixes: **5 amplicons** for `PRJNA788522`, **3** for `PRJNA507366` (whose own labels say `3plx`) |
+| Cluster copy | `/projects/tcr_bcr_rnaseq/migec_ctdna/fastq` on aldan3, 100 runs, 6.9 GB (2026-08-13). A convenience cache — the fetch command above regenerates it |
+| Provenance | experimental (SRA), commercial reference material; the certified VAFs are the vendor's |
+| Fetch | `python scripts/sra_fetch.py get <runs> -o data/` (NCBI S3, ~7 MB/s) |
+| Analyse | `python scripts/ctdna_titration.py --reads data/ --out out/ --design design.tsv` |
+| Designs | `curl -sG 'https://www.ebi.ac.uk/ena/portal/api/search' -d result=read_run -d query=study_accession=<PRJ> -d fields=run_accession,sample_alias -d format=tsv -d limit=0` |
+
+**`PRJNA788522` — the titration.** Österlund T, Filges S, Johansson G, Ståhlberg A.
+*UMIErrorCorrect and UMIAnalyzer: Software for Consensus Read Generation, Error Correction, and
+Visualization Using Unique Molecular Identifiers.* **Clin Chem** 2022;68(11):1425–1435.
+[doi:10.1093/clinchem/hvac136](https://doi.org/10.1093/clinchem/hvac136), PMID 36031761.
+
+72 runs, 41.8 M reads, Illumina MiniSeq, single-end 151 nt. The design is in `sample_alias` as
+`<input>ng_<vaf>_<depth>x_rep_<n>`: mutant allele frequency **0% (`WT`), 0.125%, 0.25%, 1%** plus
+an undiluted `cell_line` arm, crossed with DNA input **5 / 20 / 80 ng** and depth **3.3 / 10 / 30x
+reads per UMI**, three replicates each. Note: the `WT` arm is a **true negative** — its variant
+frequency is zero by construction, so anything a caller reports there is its own false-positive
+rate on real chemistry rather than on a simulation.
+
+**`PRJNA507366` — the polymerase panel.** Filges S, Yamada E, Ståhlberg A, Godfrey TE. *Impact of
+Polymerase Fidelity on Background Error Rates in Next-Generation Sequencing with Unique Molecular
+Identifiers/Barcodes.* **Sci Rep** 2019;9(1):3503.
+[doi:10.1038/s41598-019-39762-6](https://doi.org/10.1038/s41598-019-39762-6), PMID 30837525.
+
+28 runs, 81.4 M reads. Note: the design is **not** in `sample_alias` (every row reads
+`SeraCare_Reference_Material`) — it is in **`library_name`** (the enzyme: Phusion, Accuprime,
+Accuprime_hifi, Platinum, Platinum_Hifi, Platinum superfi, three replicates each) and in
+`experiment_title` (`Wildtype` against `0.031% VAF` / `0.0625% VAF` / `0.125% VAF`). Reading the
+alias alone would have called this study undesigned. It extends the frequency range **4x below**
+PRJNA788522's floor, and it is the only public dataset here that varies the polymerase while
+holding template and protocol fixed — which is the comparison `--rt-error auto` needs and the one
+McInerney 2014's published fidelities can be checked against.
+
 ### Calib (github.com/vpc-ccg/calib)
 
 | Item | Value |

migec-2.2.1/assets/ctdna_titration.tsv

@@ -0,0 +1,35 @@
+study	enzyme	input_ng	arm	depth	n	amplicons	reads	molecules_per_amplicon	reads_per_molecule	barcode_phred	variant_molecules	p_enough
+PRJNA507366	Accuprime		WT		3	3	1.95274e+06	79751.7	8.09	30.8233	0	0
+PRJNA507366	Accuprime_hifi		WT		3	3	2.25763e+06	93253.7	8.07	30.8367	0	0
+PRJNA507366	Phusion		WT		3	3	1.84092e+06	95455.7	6.43667	31.3967	0	0
+PRJNA507366	PlatRegular	80	WT		3	3	4.22022e+06	57728	24.3433	28.8667	0	0
+PRJNA507366	PlatSuperfi	80	WT		3	3	3.91793e+06	134430	9.73667	29.37	0	0
+PRJNA507366	Platinum		WT		3	3	2.15418e+06	86146.7	8.35667	31.0067	0	0
+PRJNA507366	Platinum superfi	50	0.125%		1	3	3.8665e+06	73286	17.59	34.78	91.61	1
+PRJNA507366	Platinum superfi	70	0.031%		3	3	4.07284e+06	89542	15.1367	35.1067	27.7567	1
+PRJNA507366	Platinum superfi	70	0.0625%		3	3	3.60725e+06	86139	13.89	34.9567	53.8367	1
+PRJNA507366	Platinum_Hifi		WT		3	3	1.81798e+06	80859.3	7.54	33.9867	0	0
+PRJNA788522		20	0.125	10.0	3	5	1.06487e+06	11960.7	17.7367	28.6467	14.9533	0.999933
+PRJNA788522		20	0.125	3.3	3	5	321953	7162.33	8.98667	27.65	8.95333	0.9935
+PRJNA788522		20	0.25	10.0	3	5	1.15595e+06	13349.7	17.1167	29.02	33.3733	1
+PRJNA788522		20	0.25	3.3	3	5	289023	8048.67	7.14333	27.7033	20.12	1
+PRJNA788522		20	1	10.0	3	5	445961	8630	10.3233	27.27	86.3	1
+PRJNA788522		20	1	3.3	3	5	131423	5970.67	4.40667	25.4833	59.7067	1
+PRJNA788522		20	WT	10.0	3	5	1.07623e+06	13988.7	15.3567	29.6467	0	0
+PRJNA788522		20	WT	3.3	3	5	334784	8479	7.89	27.3833	0	0
+PRJNA788522		20	cell_line	10.0	3	5	813932	10299.3	15.7467	29.9133		
+PRJNA788522		20	cell_line	3.3	3	5	232375	6310	7.39333	26.27		
+PRJNA788522		20	cell_line	30.0	3	5	1.9048e+06	16808.7	22.6567	30.9867		
+PRJNA788522		5	0.125	10.0	3	5	434820	4530	20.02	29.03	5.66333	0.894
+PRJNA788522		5	0.125	3.3	3	5	182150	2699	13.47	28.1	3.37	0.654933
+PRJNA788522		5	0.25	10.0	3	5	909770	5899.33	28.9033	29.14	14.75	0.9994
+PRJNA788522		5	0.25	3.3	3	5	182433	2945.67	12.32	28.17	7.36333	0.976367
+PRJNA788522		5	1	10.0	3	5	501350	4129	24.2467	29.0267	41.29	1
+PRJNA788522		5	1	3.3	3	5	167841	2888.33	11.5833	28.31	28.8833	1
+PRJNA788522		5	WT	10.0	3	5	533736	4428	24.04	29.21	0	0
+PRJNA788522		5	WT	3.3	3	5	146169	2868.67	10.1533	28.4567	0	0
+PRJNA788522		5	cell_line	10.0	3	5	304544	4559.33	13.5433	27.53		
+PRJNA788522		5	cell_line	3.3	3	5	76465.3	2958.33	5.18	20.6033		
+PRJNA788522		5	cell_line	30.0	3	5	635122	6664.33	19.0733	29.2467		
+PRJNA788522		80	cell_line	10.0	3	5	1.49859e+06	26793	11.2	31.46		
+PRJNA788522		80	cell_line	3.3	3	5	582020	19310	6.02667	29.8167