migec 2.1.0__tar.gz → 2.2.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Potentially problematic release.
This version of migec might be problematic. Click here for more details.
- {migec-2.1.0 → migec-2.2.0}/CHANGELOG.md +101 -0
- {migec-2.1.0 → migec-2.2.0}/CLAUDE.md +64 -4
- {migec-2.1.0 → migec-2.2.0}/PKG-INFO +53 -14
- {migec-2.1.0 → migec-2.2.0}/README.md +52 -13
- {migec-2.1.0 → migec-2.2.0}/ROADMAP.md +27 -2
- {migec-2.1.0 → migec-2.2.0}/SOURCES.md +3 -3
- migec-2.2.0/assets/PBMC.umi_errors.tsv +160 -0
- migec-2.2.0/assets/benchmark_threads.json +47 -0
- {migec-2.1.0 → migec-2.2.0}/assets/benchmark_threads.svg +15 -15
- migec-2.2.0/assets/benchmark_threads.tsv +6 -0
- {migec-2.1.0 → migec-2.2.0}/assets/mig_size_spectrum.gp +9 -1
- {migec-2.1.0 → migec-2.2.0}/assets/mig_size_spectrum.svg +163 -25
- migec-2.2.0/assets/umi_error_children.gp +32 -0
- migec-2.2.0/assets/umi_error_children.svg +578 -0
- migec-2.2.0/assets/umi_error_rate.gp +47 -0
- migec-2.2.0/assets/umi_error_rate.svg +532 -0
- {migec-2.1.0 → migec-2.2.0}/docs/assemble.rst +2 -2
- {migec-2.1.0 → migec-2.2.0}/docs/commands.rst +1 -1
- {migec-2.1.0 → migec-2.2.0}/docs/examples.rst +3 -0
- {migec-2.1.0 → migec-2.2.0}/docs/method.rst +3 -0
- {migec-2.1.0 → migec-2.2.0}/docs/nextflow.rst +3 -3
- {migec-2.1.0 → migec-2.2.0}/docs/performance.rst +82 -45
- {migec-2.1.0 → migec-2.2.0}/docs/plots.rst +10 -0
- {migec-2.1.0 → migec-2.2.0}/docs/refine.rst +9 -0
- migec-2.2.0/docs/umi_errors.rst +167 -0
- {migec-2.1.0 → migec-2.2.0}/include/migec/parallel.hpp +26 -10
- {migec-2.1.0 → migec-2.2.0}/include/migec/refine.hpp +12 -0
- {migec-2.1.0 → migec-2.2.0}/include/migec/umi_stats.hpp +5 -1
- {migec-2.1.0 → migec-2.2.0}/include/migec/version.hpp +1 -1
- {migec-2.1.0 → migec-2.2.0}/project/README.md +9 -0
- {migec-2.1.0 → migec-2.2.0}/pyproject.toml +1 -1
- {migec-2.1.0 → migec-2.2.0}/python/migec/__init__.py +1 -1
- {migec-2.1.0 → migec-2.2.0}/python/migec/checkout.py +5 -3
- {migec-2.1.0 → migec-2.2.0}/python/migec/cli.py +4 -3
- {migec-2.1.0 → migec-2.2.0}/python/migec/plot.py +76 -1
- {migec-2.1.0 → migec-2.2.0}/python/migec/refine.py +8 -0
- {migec-2.1.0 → migec-2.2.0}/scripts/example_figures.py +1 -1
- {migec-2.1.0 → migec-2.2.0}/skills/migec/SKILL.md +33 -2
- {migec-2.1.0 → migec-2.2.0}/src/_bindings.cpp +4 -0
- {migec-2.1.0 → migec-2.2.0}/src/assemble.cpp +19 -16
- {migec-2.1.0 → migec-2.2.0}/src/refine.cpp +169 -18
- {migec-2.1.0 → migec-2.2.0}/src/umi_stats.cpp +14 -7
- {migec-2.1.0 → migec-2.2.0}/tests/synthetic/test_refine.py +2 -0
- migec-2.2.0/tests/synthetic/test_umi_errors.py +135 -0
- migec-2.1.0/assets/benchmark_threads.json +0 -47
- migec-2.1.0/assets/benchmark_threads.tsv +0 -6
- {migec-2.1.0 → migec-2.2.0}/.github/workflows/ci.yml +0 -0
- {migec-2.1.0 → migec-2.2.0}/.github/workflows/docs.yml +0 -0
- {migec-2.1.0 → migec-2.2.0}/.github/workflows/publish.yml +0 -0
- {migec-2.1.0 → migec-2.2.0}/.github/workflows/testpypi.yml +0 -0
- {migec-2.1.0 → migec-2.2.0}/.gitignore +0 -0
- {migec-2.1.0 → migec-2.2.0}/CMakeLists.txt +0 -0
- {migec-2.1.0 → migec-2.2.0}/LICENSE +0 -0
- {migec-2.1.0 → migec-2.2.0}/assets/PBMC.cell_rank.tsv +0 -0
- {migec-2.1.0 → migec-2.2.0}/assets/PBMC.sizes.tsv +0 -0
- {migec-2.1.0 → migec-2.2.0}/assets/SRR1763769.mig.tsv +0 -0
- {migec-2.1.0 → migec-2.2.0}/assets/assemble.coverage.tsv +0 -0
- {migec-2.1.0 → migec-2.2.0}/assets/assemble.quality_by_depth.tsv +0 -0
- {migec-2.1.0 → migec-2.2.0}/assets/assemble_coverage.gp +0 -0
- {migec-2.1.0 → migec-2.2.0}/assets/assemble_coverage.svg +0 -0
- {migec-2.1.0 → migec-2.2.0}/assets/benchmark_threads.gp +0 -0
- {migec-2.1.0 → migec-2.2.0}/assets/cell_rank.gp +0 -0
- {migec-2.1.0 → migec-2.2.0}/assets/cell_rank.svg +0 -0
- {migec-2.1.0 → migec-2.2.0}/assets/consensus_error.gp +0 -0
- {migec-2.1.0 → migec-2.2.0}/assets/consensus_error.svg +0 -0
- {migec-2.1.0 → migec-2.2.0}/assets/consensus_layout.gp +0 -0
- {migec-2.1.0 → migec-2.2.0}/assets/consensus_layout.svg +0 -0
- {migec-2.1.0 → migec-2.2.0}/assets/consensus_quality.gp +0 -0
- {migec-2.1.0 → migec-2.2.0}/assets/consensus_quality.svg +0 -0
- {migec-2.1.0 → migec-2.2.0}/assets/mig_size_zipf.gp +0 -0
- {migec-2.1.0 → migec-2.2.0}/assets/mig_size_zipf.svg +0 -0
- {migec-2.1.0 → migec-2.2.0}/assets/pipeline.dot +0 -0
- {migec-2.1.0 → migec-2.2.0}/assets/pipeline.svg +0 -0
- {migec-2.1.0 → migec-2.2.0}/docs/Makefile +0 -0
- {migec-2.1.0 → migec-2.2.0}/docs/barcode_space.rst +0 -0
- {migec-2.1.0 → migec-2.2.0}/docs/checkout.rst +0 -0
- {migec-2.1.0 → migec-2.2.0}/docs/conf.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/docs/downstream.rst +0 -0
- {migec-2.1.0 → migec-2.2.0}/docs/formats.rst +0 -0
- {migec-2.1.0 → migec-2.2.0}/docs/fragmented.rst +0 -0
- {migec-2.1.0 → migec-2.2.0}/docs/grouping.rst +0 -0
- {migec-2.1.0 → migec-2.2.0}/docs/index.rst +0 -0
- {migec-2.1.0 → migec-2.2.0}/docs/installation.rst +0 -0
- {migec-2.1.0 → migec-2.2.0}/docs/layouts.rst +0 -0
- {migec-2.1.0 → migec-2.2.0}/docs/nulls.rst +0 -0
- {migec-2.1.0 → migec-2.2.0}/docs/quality_floor.rst +0 -0
- {migec-2.1.0 → migec-2.2.0}/docs/reference.rst +0 -0
- {migec-2.1.0 → migec-2.2.0}/docs/roadmap.rst +0 -0
- {migec-2.1.0 → migec-2.2.0}/docs/subsample.rst +0 -0
- {migec-2.1.0 → migec-2.2.0}/docs/suggest.rst +0 -0
- {migec-2.1.0 → migec-2.2.0}/docs/umi_statistics.rst +0 -0
- {migec-2.1.0 → migec-2.2.0}/docs/validation.rst +0 -0
- {migec-2.1.0 → migec-2.2.0}/include/migec/assemble.hpp +0 -0
- {migec-2.1.0 → migec-2.2.0}/include/migec/checkout.hpp +0 -0
- {migec-2.1.0 → migec-2.2.0}/include/migec/consensus.hpp +0 -0
- {migec-2.1.0 → migec-2.2.0}/include/migec/fastq.hpp +0 -0
- {migec-2.1.0 → migec-2.2.0}/include/migec/mig_record.hpp +0 -0
- {migec-2.1.0 → migec-2.2.0}/include/migec/pattern.hpp +0 -0
- {migec-2.1.0 → migec-2.2.0}/include/migec/resource.hpp +0 -0
- {migec-2.1.0 → migec-2.2.0}/include/migec/subsample.hpp +0 -0
- {migec-2.1.0 → migec-2.2.0}/include/migec/suggest.hpp +0 -0
- {migec-2.1.0 → migec-2.2.0}/include/migec/types.hpp +0 -0
- {migec-2.1.0 → migec-2.2.0}/include/migec/whitelist.hpp +0 -0
- {migec-2.1.0 → migec-2.2.0}/integrations/nextflow/README.md +0 -0
- {migec-2.1.0 → migec-2.2.0}/integrations/nextflow/modules/migec/assemble/main.nf +0 -0
- {migec-2.1.0 → migec-2.2.0}/integrations/nextflow/modules/migec/assemble/meta.yml +0 -0
- {migec-2.1.0 → migec-2.2.0}/integrations/nextflow/modules/migec/checkout/main.nf +0 -0
- {migec-2.1.0 → migec-2.2.0}/integrations/nextflow/modules/migec/checkout/meta.yml +0 -0
- {migec-2.1.0 → migec-2.2.0}/integrations/nextflow/modules/migec/environment.yml +0 -0
- {migec-2.1.0 → migec-2.2.0}/integrations/nextflow/modules/migec/plot/main.nf +0 -0
- {migec-2.1.0 → migec-2.2.0}/integrations/nextflow/modules/migec/refine/main.nf +0 -0
- {migec-2.1.0 → migec-2.2.0}/integrations/nextflow/modules/migec/refine/meta.yml +0 -0
- {migec-2.1.0 → migec-2.2.0}/integrations/nextflow/nextflow.config +0 -0
- {migec-2.1.0 → migec-2.2.0}/integrations/nextflow/subworkflows/migec/main.nf +0 -0
- {migec-2.1.0 → migec-2.2.0}/notebooks/barcode_space.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/notebooks/platforms.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/notebooks/refine_diagnostics.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/project/design-assemble.md +0 -0
- {migec-2.1.0 → migec-2.2.0}/project/design-benchmarks.md +0 -0
- {migec-2.1.0 → migec-2.2.0}/project/design-checkout.md +0 -0
- {migec-2.1.0 → migec-2.2.0}/project/design-io-interop.md +0 -0
- {migec-2.1.0 → migec-2.2.0}/project/design-refine.md +0 -0
- {migec-2.1.0 → migec-2.2.0}/project/design-repo.md +0 -0
- {migec-2.1.0 → migec-2.2.0}/project/review-algorithms.md +0 -0
- {migec-2.1.0 → migec-2.2.0}/project/review-scope.md +0 -0
- {migec-2.1.0 → migec-2.2.0}/python/migec/assemble.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/python/migec/sheet.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/python/migec/subsample.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/python/migec/suggest.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/scripts/benchmark_threads.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/scripts/collision_check.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/scripts/compare_calib.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/scripts/correction_accuracy.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/scripts/diagnose.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/scripts/permutation_nulls.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/scripts/quality_floor.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/scripts/read_start_dispersion.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/scripts/spikein_ratio.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/setup.sh +0 -0
- {migec-2.1.0 → migec-2.2.0}/src/checkout.cpp +0 -0
- {migec-2.1.0 → migec-2.2.0}/src/consensus.cpp +0 -0
- {migec-2.1.0 → migec-2.2.0}/src/fastq.cpp +0 -0
- {migec-2.1.0 → migec-2.2.0}/src/mig_record.cpp +0 -0
- {migec-2.1.0 → migec-2.2.0}/src/pattern.cpp +0 -0
- {migec-2.1.0 → migec-2.2.0}/src/resource.cpp +0 -0
- {migec-2.1.0 → migec-2.2.0}/src/subsample.cpp +0 -0
- {migec-2.1.0 → migec-2.2.0}/src/suggest.cpp +0 -0
- {migec-2.1.0 → migec-2.2.0}/src/types.cpp +0 -0
- {migec-2.1.0 → migec-2.2.0}/src/whitelist.cpp +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/__init__.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/benchmark/__init__.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/benchmark/test_assemble_speed.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/benchmark/test_checkout_speed.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/benchmark/test_refine_speed.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/conftest.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/cpp/doctest.h +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/cpp/test_checkout.cpp +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/cpp/test_checkout_run.cpp +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/cpp/test_consensus.cpp +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/cpp/test_fastq.cpp +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/cpp/test_mig_record.cpp +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/cpp/test_parallel_stages.cpp +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/cpp/test_pattern.cpp +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/cpp/test_types.cpp +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/cpp/test_umi_stats.cpp +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/cpp/test_whitelist.cpp +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/data/migec_spikein_truth.tsv +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/realworld/__init__.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/synthetic/__init__.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/synthetic/_sim.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/synthetic/test_assemble.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/synthetic/test_barcode_space.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/synthetic/test_checkout_end_to_end.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/synthetic/test_dual_end.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/synthetic/test_grouping_accuracy.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/synthetic/test_nulls.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/synthetic/test_quality_floor.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/synthetic/test_sim.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/synthetic/test_subsample.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/synthetic/test_suggest.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/synthetic/test_thread_invariance.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/unit/__init__.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/unit/test_bindings.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/unit/test_compare_calib.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/unit/test_core.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/unit/test_downstream.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/unit/test_plot.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/unit/test_sheet_and_cli.py +0 -0
- {migec-2.1.0 → migec-2.2.0}/tests/unit/test_spikein_ratio.py +0 -0
|
@@ -4,6 +4,107 @@ Hand-written and prose-heavy: each entry says what changed and, where it matters
|
|
|
4
4
|
prevents. Releases before 2.0.0 are the Groovy MIGEC and are described by their git tags on the
|
|
5
5
|
`legacy-v1` branch.
|
|
6
6
|
|
|
7
|
+
## 2.2.0 — 2026-08-13
|
|
8
|
+
|
|
9
|
+
Throughput and one new measurement. No breaking change: every 2.1.0 command, flag and output file
|
|
10
|
+
behaves as it did, and the additions are a new table, four new JSON fields and two new plot panels.
|
|
11
|
+
|
|
12
|
+
### The barcode error rate, measured at every depth
|
|
13
|
+
|
|
14
|
+
`refine` writes `<sample>.umi_errors.tsv`: one row per exact parent depth, carrying the distinct
|
|
15
|
+
error children that parent spawned, the reads in them, and the error rate each implies. A parent
|
|
16
|
+
carrying `c` reads offered `c*L` barcode bases to be miscalled, so the same eps falls out two ways:
|
|
17
|
+
|
|
18
|
+
distinct children u(c) = 3L (1 - exp(-c eps / 3)) -> eps = -(3/c) ln(1 - u/3L)
|
|
19
|
+
reads in children r(c) = c L eps -> eps = r / (c L)
|
|
20
|
+
|
|
21
|
+
A barcode has only `3L` neighbours one substitution away, so the first **saturates**; reads have no
|
|
22
|
+
ceiling, so the second does not. Where the two part company on the figure is where this library's
|
|
23
|
+
barcode neighbourhood filled up, read off the data instead of predicted. Past saturation the first
|
|
24
|
+
is left blank rather than reported as a small number: inverting a full neighbourhood returns "no
|
|
25
|
+
errors" for the most error-ridden library there can be.
|
|
26
|
+
|
|
27
|
+
New report lines and JSON fields `error_at_depth`, `error_phred`, `error_from_children`,
|
|
28
|
+
`error_depth`. The number to quote is `error_at_depth` — restricted to parents seen >= 10 times,
|
|
29
|
+
where correction is near-complete — and `error_phred` is the same thing as a Phred so it can be put
|
|
30
|
+
beside the barcode's own reported Q. On a diverse library sequenced 25 deep with 1e-3 injected, the
|
|
31
|
+
distance-1 excess gives 9.73e-04, the children give 9.89e-04 (Q30) and all depths give 9.98e-04.
|
|
32
|
+
Three routes to one number agreeing within 3% is what makes any of them believable.
|
|
33
|
+
|
|
34
|
+
Never: **both estimators are bounded by the merges correction actually made**, so neither is
|
|
35
|
+
saturation-free. Against a known injected rate, as a fraction of truth:
|
|
36
|
+
|
|
37
|
+
| occupancy | 0.2% | 2.3% | 9.8% | 33% | 100% |
|
|
38
|
+
|---|---|---|---|---|---|
|
|
39
|
+
| distance-1 excess | 0.97 | 0.96 | 0.76 | 0.45 | 0.001 |
|
|
40
|
+
| from the children | **0.99** | **0.95** | **0.88** | **0.62** | 0.00 |
|
|
41
|
+
|
|
42
|
+
The children estimate is the better of the two wherever either works, and at 100% they both go to
|
|
43
|
+
zero for the same reason: on a full barcode space `correct_umis` refuses to merge, correctly,
|
|
44
|
+
because a distance-1 neighbour there is more likely a real molecule than a child. The `saturated`
|
|
45
|
+
flag is what says the answer is a floor — read it, do not read this table instead of it. And read
|
|
46
|
+
the table at depth: a child whose parent was never sequenced cannot be counted, which at 1-3
|
|
47
|
+
reads/UMI is 80% of them. `tests/synthetic/test_umi_errors.py` holds all of it, `docs/umi_errors.rst`
|
|
48
|
+
works through it.
|
|
49
|
+
|
|
50
|
+
Two new `migec plot` panels over that table, `umi_error_children` and `umi_error_rate`, with the
|
|
51
|
+
`3L` ceiling and the reported estimate drawn as reference lines. Note: `neighbours` and `estimate`
|
|
52
|
+
are constant down their columns on purpose — the panels draw them, and a figure that needs a value
|
|
53
|
+
its own table does not carry will one day disagree with the report.
|
|
54
|
+
|
|
55
|
+
### `reads in them` is points, never a line
|
|
56
|
+
|
|
57
|
+
The `mig_size_spectrum` panel drew its reads series `with lines` over a table that has one row per
|
|
58
|
+
**exact** size. Past the head of the distribution almost every size holds exactly one molecule, so
|
|
59
|
+
reads == size, and the line rendered the `y = x` diagonal as the most prominent feature of the
|
|
60
|
+
figure — a tautology drawn as a second mode. Where two or three molecules shared a size the same
|
|
61
|
+
line sawtoothed between `size*1` and `size*2`, which is integer quantisation drawn as signal, and it
|
|
62
|
+
bridged gaps in the support where no size was observed at all. It is points now, and the new
|
|
63
|
+
barcode-error panels are points for the same reason.
|
|
64
|
+
|
|
65
|
+
### The thread helper was the bottleneck
|
|
66
|
+
|
|
67
|
+
With all three stages threaded, a sampling profile of
|
|
68
|
+
`assemble` put **21% of all CPU samples, across every thread, on a single instruction**: the atomic
|
|
69
|
+
`fetch_add` in `parallel_for` that handed out one item at a time. When an item is one read's tag
|
|
70
|
+
scan, sixteen cores serialise on one cache line and the counter costs more than the work it
|
|
71
|
+
distributes. Items are claimed in batches now, sized so each worker takes ~8 turns and capped so a
|
|
72
|
+
ten-million-item scan still hands out ten thousand batches. Never: the batch collapses to 1 when
|
|
73
|
+
there are few items, which is the uneven case -- one bucket per item, one holding ten molecules and
|
|
74
|
+
the next ten million -- that the counter exists for.
|
|
75
|
+
|
|
76
|
+
Two serial blocks fell out with it, both read-only `3L`-binary-search scans of the barcode table:
|
|
77
|
+
the **distance-1 census** in `estimate_umi_error` (checkout's per-sample statistics tail) and
|
|
78
|
+
refine's **residual-FDR scan**, which was 0.53 s of a 2.17 s run on one core after everything around
|
|
79
|
+
it had been parallelised. Both tally integers into per-worker counters summed afterwards, so `-t`
|
|
80
|
+
still changes nothing but the clock -- verified byte-for-byte at `-t` 1/3/16 on a clean library and
|
|
81
|
+
on one with injected barcode errors (residual 1,294 either way), and under the thread sanitizer
|
|
82
|
+
(104 cases, 224,116 assertions, no race).
|
|
83
|
+
|
|
84
|
+
Two allocation fixes on paths every read takes: refine's rewrite chunk is now **assigned into
|
|
85
|
+
rather than cleared** -- the same trade assemble's partition already made, since `clear()` destroys
|
|
86
|
+
four `std::string` per record -- and the rewrite's comment buffer and unpacked barcode are worker
|
|
87
|
+
scratch reused read after read, with the cell and UMI halves taken as `string_view` instead of
|
|
88
|
+
`substr`. That is four allocations a read removed; `rewrite_seconds` fell 0.51 s to 0.38 s.
|
|
89
|
+
`assemble` also unpacked each group's UMI twice, once for the composition tally and once for the
|
|
90
|
+
record name.
|
|
91
|
+
|
|
92
|
+
| stage | 1 thread | 16 threads | was, at 16 |
|
|
93
|
+
|---|---|---|---|
|
|
94
|
+
| `checkout` | 213,880 | **1,548,835** | 1,056,472 |
|
|
95
|
+
| `refine` | 617,802 | **1,554,156** | 1,012,368 |
|
|
96
|
+
| `assemble` | 554,106 | **2,470,928** | 2,051,937 |
|
|
97
|
+
|
|
98
|
+
reads/s, 500 k-read sample except checkout at 2 M; `assets/benchmark_threads.tsv` is regenerated
|
|
99
|
+
and the figure redrawn from it. checkout's gap between end-to-end and matching throughput fell from
|
|
100
|
+
20% to 9%.
|
|
101
|
+
|
|
102
|
+
Note: re-measured while it was in hand, a **64 k chunk in assemble's partition is still 32% faster**
|
|
103
|
+
(3,075,506 reads/s against 2,324,403 on 4 M reads) and is still not taken: it costs 16 MB of
|
|
104
|
+
resident chunk, which at NovaSeq scale makes pass 1 the memory peak and breaks the property that a
|
|
105
|
+
finer partition costs less rather than more. The benchmark tier does not fail at 64 k on a 500 k
|
|
106
|
+
corpus -- the objection is one of scale -- so the number is written down next to the constant.
|
|
107
|
+
|
|
7
108
|
## 2.1.0 — 2026-08-13
|
|
8
109
|
|
|
9
110
|
First non-alpha of the rewrite. The three stages, the eight commands and the on-disk formats have
|
|
@@ -261,6 +261,26 @@ correction is written up in `project/review-algorithms.md`.
|
|
|
261
261
|
is where the answer matters most. It caught me twice: once in the correction posterior, once in
|
|
262
262
|
the residual-FDR estimator. Both now use payload agreement as well. On a 1.23 reads/barcode
|
|
263
263
|
library the count-only residual is 0 and the full one is 1,294.
|
|
264
|
+
- **The barcode error rate is measured at every DEPTH, not once for the library.**
|
|
265
|
+
`<sample>.umi_errors.tsv`, one row per exact parent depth: a parent carrying `c` reads offered
|
|
266
|
+
`c*L` barcode bases to be miscalled, so `u(c) = 3L(1-exp(-c eps/3))` from the distinct children
|
|
267
|
+
and `r(c) = c L eps` from the reads in them invert to the same eps. Reported as
|
|
268
|
+
`error_at_depth` (parents seen >= 10 times, where correction is near-complete), `error_phred`,
|
|
269
|
+
and `error_from_children` (all depths, a LOWER bound). On 1e-3 injected: distance-1 9.73e-04,
|
|
270
|
+
children 9.89e-04 = Q30, all depths 9.98e-04.
|
|
271
|
+
- **Never: neither error estimator is saturation-free — both are bounded by the merges correction
|
|
272
|
+
MADE.** As a fraction of an injected truth: distance-1 0.97 / 0.96 / 0.76 / 0.45 / 0.001 and
|
|
273
|
+
children 0.99 / 0.95 / 0.88 / 0.62 / 0.00 at occupancy 0.2% / 2.3% / 9.8% / 33% / 100%. The
|
|
274
|
+
children estimate wins everywhere either works; at 100% BOTH are zero, because `correct_umis`
|
|
275
|
+
refuses to merge on a full space and is right to. `saturated` is the flag that says the answer
|
|
276
|
+
is a floor -- do not read the table instead of it. `docs/umi_errors.rst`,
|
|
277
|
+
`tests/synthetic/test_umi_errors.py`.
|
|
278
|
+
- **Never: a sparse spectrum is POINTS, never a line.** `mig_size_spectrum` drew "reads in them"
|
|
279
|
+
`with lines` over a table at EXACT sizes, where past the head almost every size holds one
|
|
280
|
+
molecule -- so reads == size and the line drew the `y = x` diagonal as the figure's most
|
|
281
|
+
prominent feature, a tautology that reads as a second mode. Two or three molecules on a size made
|
|
282
|
+
it sawtooth between `size*1` and `size*2`, and it bridged gaps where nothing was observed. Same
|
|
283
|
+
rule for the two `umi_error_*` panels.
|
|
264
284
|
- **The MIG-size FDR threshold is reported, never applied** (2026-08-13). Measured residual, not
|
|
265
285
|
derived. 5.25% of 1-read molecules at 1.23 reads/barcode, 0% at 4.62.
|
|
266
286
|
- **Dual-end barcodes work (2026-08-13)**: column 3 of the sheet is the slave pattern, on the
|
|
@@ -354,11 +374,37 @@ correction is written up in `project/review-algorithms.md`.
|
|
|
354
374
|
but the per-sample counters and the memory figure were one sample's. Fixed; the published table
|
|
355
375
|
is re-measured and now comes from `scripts/benchmark_threads.py`, which writes the TSV the
|
|
356
376
|
figure is drawn from.
|
|
357
|
-
- **All three stages thread (2026-08-13, 2.0.0a3).** refine 222k -> 1.01M reads/s, assemble
|
|
358
|
-
203k ->
|
|
377
|
+
- **All three stages thread (2026-08-13, 2.0.0a3-2.1.0).** refine 222k -> 1.01M reads/s, assemble
|
|
378
|
+
203k -> 2.05M (its partition threaded in a4), checkout unchanged at 1.06M. Never: the FIRST fix was not a thread -- zlib level 6
|
|
359
379
|
was 83% of refine's wall clock compressing an intermediate the next stage immediately
|
|
360
380
|
decompresses. Level 1 is now the default for every stage's output and gave 3x on its own.
|
|
361
381
|
Measure before parallelising.
|
|
382
|
+
- **Never: the thread HELPER was the bottleneck once everything else was threaded (unreleased).**
|
|
383
|
+
`parallel_for` claimed one item per atomic `fetch_add`, and a sampling profile of assemble put
|
|
384
|
+
**21% of all CPU samples across every thread on that one instruction** -- more than the parse it
|
|
385
|
+
was handing out. Sixteen cores serialise on one cache line. Batched now: ~8 turns per worker,
|
|
386
|
+
capped at 1024, collapsing to 1 when items are few, which is the uneven case (one bucket per
|
|
387
|
+
item) the counter exists for. Never: profile the helper, not only the work. Current numbers,
|
|
388
|
+
1 thread -> 16: checkout **213,880 -> 1,548,835** end to end (1,697,313 matching), refine
|
|
389
|
+
**617,802 -> 1,554,156**, assemble **554,106 -> 2,470,928**; `assets/benchmark_threads.tsv` is
|
|
390
|
+
the committed table and the figure is drawn from it.
|
|
391
|
+
- **Two serial scans went with it**: the distance-1 census in `estimate_umi_error` (which IS
|
|
392
|
+
checkout's per-sample statistics tail -- the end-to-end/matching gap fell from 20% to 9%) and
|
|
393
|
+
refine's residual-FDR scan, 0.53 s of a 2.17 s run on one core. Both are read-only `3L` binary
|
|
394
|
+
searches per barcode tallying integers, so per-worker counters summed afterwards keep `-t`
|
|
395
|
+
changing nothing but the clock. `estimate_umi_error` now takes a `threads` argument, defaulted
|
|
396
|
+
to 1 so the C++ test calls are unchanged.
|
|
397
|
+
- **Note: a 64 k chunk in assemble's partition is STILL 32% faster and still not taken**
|
|
398
|
+
(3,075,506 reads/s against 2,324,403 on 4 M, re-measured after the batching). It costs 16 MB of
|
|
399
|
+
resident chunk, which at NovaSeq scale makes pass 1 the memory peak. Never: the benchmark tier
|
|
400
|
+
does NOT fail at 64 k on a 500 k corpus -- the objection is scale, which no test here can see,
|
|
401
|
+
so the number lives next to the constant in `assemble.cpp`. Upgrade path is a persistent worker
|
|
402
|
+
pool, not a bigger chunk.
|
|
403
|
+
- **Never: `clear()` on a chunk of `FastqOwned` is four allocations a record.** assemble's
|
|
404
|
+
partition knew this; refine's rewrite still did it. Assigned into now, and the rewrite's comment
|
|
405
|
+
buffer and unpacked barcode are worker scratch with the cell/UMI halves as `string_view` rather
|
|
406
|
+
than `substr` -- `rewrite_seconds` 0.51 -> 0.38. Same class of bug in `assemble`: each group's
|
|
407
|
+
UMI was unpacked twice, once for the composition tally and once for the record name.
|
|
362
408
|
- **Never: `-t` changes nothing but the wall clock, on every stage.** refine parallelises the
|
|
363
409
|
neighbourhood SCAN (a pure function of the barcode table -- it reads no union-find state) and
|
|
364
410
|
applies the merges serially in the original order, so the result is identical, not equivalent.
|
|
@@ -407,8 +453,22 @@ correction is written up in `project/review-algorithms.md`.
|
|
|
407
453
|
- **Never: `containsKey`, not `?:`, for a nextflow boolean.** Groovy's elvis treats `false` as
|
|
408
454
|
absent, so a per-sample `contig: false` against `params.migec_contig = true` silently meant its
|
|
409
455
|
opposite -- the one direction a per-sample override exists to make possible.
|
|
410
|
-
- Next
|
|
411
|
-
|
|
456
|
+
- **Next, in order. `ROADMAP.md` has the same list with the reasoning; this is the short form.**
|
|
457
|
+
1. **`.mig` bucket output from `checkout`** — the only unbounded allocation left. The UMI
|
|
458
|
+
counters are 8.8 GB at NovaSeq scale held in one piece; checkout warns past 1 GB and a warning
|
|
459
|
+
is not a fix.
|
|
460
|
+
2. **Bucketed correction in `refine`**, two passes with the key rotated. Same problem one stage
|
|
461
|
+
on, and the fix is understood rather than still to be designed.
|
|
462
|
+
3. **The template's own error split** — the pattern's constant bases calibrate the PRIMER, not
|
|
463
|
+
the polymerase, so the RT/PCR separation needs a different standard.
|
|
464
|
+
4. **`--rt-error auto`**, which falls out of 3 and not before it.
|
|
465
|
+
5. **i7xi5 contingency table** — needs nothing that is not already in the headers.
|
|
466
|
+
6. **`2026-migec-benchmark`** and the published comparisons. This is what the version number is
|
|
467
|
+
waiting on, not the code.
|
|
468
|
+
7. **ctDNA ground truth** (COSMIC at 0.005-0.075 VAF). Must be built: Maruzani's deposited runs
|
|
469
|
+
carry no UMI.
|
|
470
|
+
8. **R1/R2 overlap merge**, as a case of `--contig`'s placement and never a second matcher.
|
|
471
|
+
9. **Bit-parallel matcher**, last and deliberately: the scan is not the bottleneck.
|
|
412
472
|
- **Note: Britanova et al aging (bulk TCR, shallow) lives on aldan3** and is the real dataset for the
|
|
413
473
|
1-3 read regime. Not pulled yet. aldan3 compute goes through SLURM, never the frontend.
|
|
414
474
|
- The archive is pushed: `legacy-v1` + tag `v1-final`, and master is the rewrite. Recovery point
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.2
|
|
2
2
|
Name: migec
|
|
3
|
-
Version: 2.
|
|
3
|
+
Version: 2.2.0
|
|
4
4
|
Summary: UMI barcode extraction, correction and consensus assembly for barcoded sequencing
|
|
5
5
|
Keywords: umi,barcode,consensus,airr,repseq,single-cell,sequencing
|
|
6
6
|
Author-Email: Mikhail Shugay <mikhail.shugay@gmail.com>
|
|
@@ -197,8 +197,8 @@ reads 2,000,000
|
|
|
197
197
|
unmatched 0 (0.0%)
|
|
198
198
|
ambiguous 0 (0.0%)
|
|
199
199
|
|
|
200
|
-
|
|
201
|
-
peak RSS
|
|
200
|
+
1.6 s (1,243,801 reads/s) = 1.5 s matching on 8 threads + 0.1 s UMI statistics
|
|
201
|
+
peak RSS 136.0 MB of which UMI counters 11.5 MB
|
|
202
202
|
|
|
203
203
|
sample reads UMIs reads/UMI UMI len eff len
|
|
204
204
|
S1 500,000 125,000 4.00 12 12.00
|
|
@@ -265,7 +265,7 @@ migec assemble out/S1.fq.gz -o cons/ --contig # random-primed reads that til
|
|
|
265
265
|
A molecule is **sample + cell barcode + UMI**, never the UMI alone — the same UMI in two cells is
|
|
266
266
|
two molecules, and that is the design rather than a defect. Reads are range partitioned on the
|
|
267
267
|
packed key into `.mig` buckets and one bucket is sorted at a time, so nothing scales with the
|
|
268
|
-
library:
|
|
268
|
+
library: 2,423,777 reads/s, and 123 MB at 16 buckets against 203 MB at one.
|
|
269
269
|
|
|
270
270
|
The per-column posterior is `LL[j][b] = Σ_i (r==b ? log(1−e) : log(e/3))`, and then the number that
|
|
271
271
|
matters:
|
|
@@ -368,6 +368,30 @@ bent one.
|
|
|
368
368
|
|
|
369
369
|
<p align="center"><img alt="molecule size against rank" src="assets/mig_size_zipf.svg" width="72%"></p>
|
|
370
370
|
|
|
371
|
+
**Barcode errors against the parent's depth** — how many distinct error children a molecule
|
|
372
|
+
spawned, and how many reads were in them. A parent seen *c* times had *c·L* barcode bases to
|
|
373
|
+
miscall, so both series climb with *c* — but only one of them can climb forever. There are exactly
|
|
374
|
+
*3L* barcodes one substitution away, so the distinct-children series bends over at the dashed line
|
|
375
|
+
while the reads series carries straight past it. Where they part is where this library's barcode
|
|
376
|
+
neighbourhood filled up, measured rather than predicted.
|
|
377
|
+
|
|
378
|
+
<p align="center"><img alt="barcode errors against depth" src="assets/umi_error_children.svg" width="72%"></p>
|
|
379
|
+
|
|
380
|
+
Invert both and you get two estimates of the same barcode error rate, per depth, with what `refine`
|
|
381
|
+
reports drawn across them. The y axis is a log error rate, so one decade is exactly ten Phred and
|
|
382
|
+
1e-3 is Q30 — which makes the number directly comparable with the barcode's own reported quality,
|
|
383
|
+
the only independent check on it there is. On a diverse library sequenced 25 deep it lands within
|
|
384
|
+
1% of a known injected rate.
|
|
385
|
+
|
|
386
|
+
<p align="center"><img alt="barcode error rate by depth" src="assets/umi_error_rate.svg" width="72%"></p>
|
|
387
|
+
|
|
388
|
+
> **Never** read this instead of the `saturated` flag. Both series count only the children
|
|
389
|
+
> correction actually merged, and on a *full* barcode space `refine` refuses to merge — rightly,
|
|
390
|
+
> because a distance-1 neighbour there is more likely a real molecule than an error child — so both
|
|
391
|
+
> estimates collapse to zero. Against a known rate they are 0.99 and 0.97 of the truth at 0.2%
|
|
392
|
+
> occupancy, 0.62 and 0.45 at 33%, and nothing at 100%. See
|
|
393
|
+
> [docs/umi_errors.rst](docs/umi_errors.rst).
|
|
394
|
+
|
|
371
395
|
**Consensus quality against depth**, as a box, never a thinned scatter. Emitted quality is discrete
|
|
372
396
|
and capped at the floor, so at any real depth every molecule sits on one or two integers: a cloud
|
|
373
397
|
of dots draws that as a flat line whether the bin holds ten molecules or ten million, and thinning
|
|
@@ -508,7 +532,7 @@ warning: 79.4% of molecules were seen once. A consensus over one read is that re
|
|
|
508
532
|
```
|
|
509
533
|
|
|
510
534
|
It is also the memory-hostile case, because distinct barcodes are what everything scales with, so
|
|
511
|
-
it is what the benchmarks use:
|
|
535
|
+
it is what the benchmarks use: 1,179,549 reads/s at 1.02 reads/UMI, 282 B resident per distinct
|
|
512
536
|
barcode, still bounded by the bucket rather than the library.
|
|
513
537
|
|
|
514
538
|
### Speed and memory are reported, not assumed
|
|
@@ -521,11 +545,12 @@ and nothing else, which is asserted per stage in C++, at the CLI, and under the
|
|
|
521
545
|
|
|
522
546
|
| stage | 1 thread | 16 threads | bound by |
|
|
523
547
|
|---|---|---|---|
|
|
524
|
-
| `checkout` |
|
|
525
|
-
| `refine` |
|
|
526
|
-
| `assemble` |
|
|
548
|
+
| `checkout` | 213,880 | **1,548,835** | reads |
|
|
549
|
+
| `refine` | 617,802 | **1,554,156** | distinct barcodes |
|
|
550
|
+
| `assemble` | 554,106 | **2,470,928** | reads, then the largest bucket |
|
|
527
551
|
|
|
528
|
-
reads/s
|
|
552
|
+
reads/s — `refine` and `assemble` on a 500 k-read sample, `checkout` on 2 M. Two of those used to
|
|
553
|
+
be 222,017 and 202,977, and the first fix was not a thread: **zlib at
|
|
529
554
|
its default level 6 was 83% of refine's wall clock**, compressing an intermediate the next stage
|
|
530
555
|
decompresses immediately. Level 1 costs 21% more bytes and gave 3x before a single thread was
|
|
531
556
|
added. Measure the stage before parallelising it.
|
|
@@ -543,18 +568,32 @@ malloc instead of inflate.
|
|
|
543
568
|
| reads/s | 1,481,946 | **2,051,937** |
|
|
544
569
|
| peak RSS | 1,479 MB | **789 MB** |
|
|
545
570
|
|
|
571
|
+
That is the record of *that* change. Batching the work-claiming atomic later took the same corpus
|
|
572
|
+
to **1.72 s and 2,324,403 reads/s**.
|
|
573
|
+
|
|
546
574
|
The memory fell with it, because the estimate deciding how finely to cut the input said a gzipped
|
|
547
575
|
FASTQ goes resident at 8x its on-disk size and it is really **19x** — a resident record is two heap
|
|
548
576
|
`std::string` with their allocator headers, not the 180 bytes of payload. Guessing low picks too
|
|
549
577
|
few buckets, and pass 2 holds sixteen of them at once.
|
|
550
578
|
|
|
579
|
+
And once every stage was threaded, the largest single cost in the whole pipeline was the
|
|
580
|
+
**thread helper itself**. `parallel_for` handed out one item per atomic `fetch_add`, and when an
|
|
581
|
+
item is one read's tag scan that instruction took **21% of all CPU samples across every thread** —
|
|
582
|
+
more than the work it was handing out. Sixteen cores serialise on one cache line. Items are claimed
|
|
583
|
+
in batches now, sized so each worker takes ~8 turns, collapsing to 1 when there are few items,
|
|
584
|
+
which is the uneven case (one bucket per item) the counter existed for. Two serial blocks went with
|
|
585
|
+
it: the distance-1 census inside `estimate_umi_error`, which is checkout's per-sample tail, and
|
|
586
|
+
refine's residual-FDR scan, which was 0.53 s of a 2.17 s run on one core after everything around it
|
|
587
|
+
had been parallelised. Both are read-only scans of the barcode table that tally integers, so they
|
|
588
|
+
thread exactly and `-t` still changes nothing but the clock.
|
|
589
|
+
|
|
551
590
|
| threads | reads/s | matching reads/s | peak RSS |
|
|
552
591
|
|---|---|---|---|
|
|
553
|
-
| 1 |
|
|
554
|
-
| 2 |
|
|
555
|
-
| 4 |
|
|
556
|
-
| 8 |
|
|
557
|
-
| 16 | **1,
|
|
592
|
+
| 1 | 213,880 | 216,584 | 59 MB |
|
|
593
|
+
| 2 | 394,471 | 403,393 | 79 MB |
|
|
594
|
+
| 4 | 737,777 | 768,810 | 101 MB |
|
|
595
|
+
| 8 | 1,256,838 | 1,349,533 | 139 MB |
|
|
596
|
+
| 16 | **1,548,835** | **1,697,313** | 215 MB |
|
|
558
597
|
|
|
559
598
|
<p align="center"><img alt="checkout thread scaling" src="assets/benchmark_threads.svg" width="72%"></p>
|
|
560
599
|
|
|
@@ -163,8 +163,8 @@ reads 2,000,000
|
|
|
163
163
|
unmatched 0 (0.0%)
|
|
164
164
|
ambiguous 0 (0.0%)
|
|
165
165
|
|
|
166
|
-
|
|
167
|
-
peak RSS
|
|
166
|
+
1.6 s (1,243,801 reads/s) = 1.5 s matching on 8 threads + 0.1 s UMI statistics
|
|
167
|
+
peak RSS 136.0 MB of which UMI counters 11.5 MB
|
|
168
168
|
|
|
169
169
|
sample reads UMIs reads/UMI UMI len eff len
|
|
170
170
|
S1 500,000 125,000 4.00 12 12.00
|
|
@@ -231,7 +231,7 @@ migec assemble out/S1.fq.gz -o cons/ --contig # random-primed reads that til
|
|
|
231
231
|
A molecule is **sample + cell barcode + UMI**, never the UMI alone — the same UMI in two cells is
|
|
232
232
|
two molecules, and that is the design rather than a defect. Reads are range partitioned on the
|
|
233
233
|
packed key into `.mig` buckets and one bucket is sorted at a time, so nothing scales with the
|
|
234
|
-
library:
|
|
234
|
+
library: 2,423,777 reads/s, and 123 MB at 16 buckets against 203 MB at one.
|
|
235
235
|
|
|
236
236
|
The per-column posterior is `LL[j][b] = Σ_i (r==b ? log(1−e) : log(e/3))`, and then the number that
|
|
237
237
|
matters:
|
|
@@ -334,6 +334,30 @@ bent one.
|
|
|
334
334
|
|
|
335
335
|
<p align="center"><img alt="molecule size against rank" src="assets/mig_size_zipf.svg" width="72%"></p>
|
|
336
336
|
|
|
337
|
+
**Barcode errors against the parent's depth** — how many distinct error children a molecule
|
|
338
|
+
spawned, and how many reads were in them. A parent seen *c* times had *c·L* barcode bases to
|
|
339
|
+
miscall, so both series climb with *c* — but only one of them can climb forever. There are exactly
|
|
340
|
+
*3L* barcodes one substitution away, so the distinct-children series bends over at the dashed line
|
|
341
|
+
while the reads series carries straight past it. Where they part is where this library's barcode
|
|
342
|
+
neighbourhood filled up, measured rather than predicted.
|
|
343
|
+
|
|
344
|
+
<p align="center"><img alt="barcode errors against depth" src="assets/umi_error_children.svg" width="72%"></p>
|
|
345
|
+
|
|
346
|
+
Invert both and you get two estimates of the same barcode error rate, per depth, with what `refine`
|
|
347
|
+
reports drawn across them. The y axis is a log error rate, so one decade is exactly ten Phred and
|
|
348
|
+
1e-3 is Q30 — which makes the number directly comparable with the barcode's own reported quality,
|
|
349
|
+
the only independent check on it there is. On a diverse library sequenced 25 deep it lands within
|
|
350
|
+
1% of a known injected rate.
|
|
351
|
+
|
|
352
|
+
<p align="center"><img alt="barcode error rate by depth" src="assets/umi_error_rate.svg" width="72%"></p>
|
|
353
|
+
|
|
354
|
+
> **Never** read this instead of the `saturated` flag. Both series count only the children
|
|
355
|
+
> correction actually merged, and on a *full* barcode space `refine` refuses to merge — rightly,
|
|
356
|
+
> because a distance-1 neighbour there is more likely a real molecule than an error child — so both
|
|
357
|
+
> estimates collapse to zero. Against a known rate they are 0.99 and 0.97 of the truth at 0.2%
|
|
358
|
+
> occupancy, 0.62 and 0.45 at 33%, and nothing at 100%. See
|
|
359
|
+
> [docs/umi_errors.rst](docs/umi_errors.rst).
|
|
360
|
+
|
|
337
361
|
**Consensus quality against depth**, as a box, never a thinned scatter. Emitted quality is discrete
|
|
338
362
|
and capped at the floor, so at any real depth every molecule sits on one or two integers: a cloud
|
|
339
363
|
of dots draws that as a flat line whether the bin holds ten molecules or ten million, and thinning
|
|
@@ -474,7 +498,7 @@ warning: 79.4% of molecules were seen once. A consensus over one read is that re
|
|
|
474
498
|
```
|
|
475
499
|
|
|
476
500
|
It is also the memory-hostile case, because distinct barcodes are what everything scales with, so
|
|
477
|
-
it is what the benchmarks use:
|
|
501
|
+
it is what the benchmarks use: 1,179,549 reads/s at 1.02 reads/UMI, 282 B resident per distinct
|
|
478
502
|
barcode, still bounded by the bucket rather than the library.
|
|
479
503
|
|
|
480
504
|
### Speed and memory are reported, not assumed
|
|
@@ -487,11 +511,12 @@ and nothing else, which is asserted per stage in C++, at the CLI, and under the
|
|
|
487
511
|
|
|
488
512
|
| stage | 1 thread | 16 threads | bound by |
|
|
489
513
|
|---|---|---|---|
|
|
490
|
-
| `checkout` |
|
|
491
|
-
| `refine` |
|
|
492
|
-
| `assemble` |
|
|
514
|
+
| `checkout` | 213,880 | **1,548,835** | reads |
|
|
515
|
+
| `refine` | 617,802 | **1,554,156** | distinct barcodes |
|
|
516
|
+
| `assemble` | 554,106 | **2,470,928** | reads, then the largest bucket |
|
|
493
517
|
|
|
494
|
-
reads/s
|
|
518
|
+
reads/s — `refine` and `assemble` on a 500 k-read sample, `checkout` on 2 M. Two of those used to
|
|
519
|
+
be 222,017 and 202,977, and the first fix was not a thread: **zlib at
|
|
495
520
|
its default level 6 was 83% of refine's wall clock**, compressing an intermediate the next stage
|
|
496
521
|
decompresses immediately. Level 1 costs 21% more bytes and gave 3x before a single thread was
|
|
497
522
|
added. Measure the stage before parallelising it.
|
|
@@ -509,18 +534,32 @@ malloc instead of inflate.
|
|
|
509
534
|
| reads/s | 1,481,946 | **2,051,937** |
|
|
510
535
|
| peak RSS | 1,479 MB | **789 MB** |
|
|
511
536
|
|
|
537
|
+
That is the record of *that* change. Batching the work-claiming atomic later took the same corpus
|
|
538
|
+
to **1.72 s and 2,324,403 reads/s**.
|
|
539
|
+
|
|
512
540
|
The memory fell with it, because the estimate deciding how finely to cut the input said a gzipped
|
|
513
541
|
FASTQ goes resident at 8x its on-disk size and it is really **19x** — a resident record is two heap
|
|
514
542
|
`std::string` with their allocator headers, not the 180 bytes of payload. Guessing low picks too
|
|
515
543
|
few buckets, and pass 2 holds sixteen of them at once.
|
|
516
544
|
|
|
545
|
+
And once every stage was threaded, the largest single cost in the whole pipeline was the
|
|
546
|
+
**thread helper itself**. `parallel_for` handed out one item per atomic `fetch_add`, and when an
|
|
547
|
+
item is one read's tag scan that instruction took **21% of all CPU samples across every thread** —
|
|
548
|
+
more than the work it was handing out. Sixteen cores serialise on one cache line. Items are claimed
|
|
549
|
+
in batches now, sized so each worker takes ~8 turns, collapsing to 1 when there are few items,
|
|
550
|
+
which is the uneven case (one bucket per item) the counter existed for. Two serial blocks went with
|
|
551
|
+
it: the distance-1 census inside `estimate_umi_error`, which is checkout's per-sample tail, and
|
|
552
|
+
refine's residual-FDR scan, which was 0.53 s of a 2.17 s run on one core after everything around it
|
|
553
|
+
had been parallelised. Both are read-only scans of the barcode table that tally integers, so they
|
|
554
|
+
thread exactly and `-t` still changes nothing but the clock.
|
|
555
|
+
|
|
517
556
|
| threads | reads/s | matching reads/s | peak RSS |
|
|
518
557
|
|---|---|---|---|
|
|
519
|
-
| 1 |
|
|
520
|
-
| 2 |
|
|
521
|
-
| 4 |
|
|
522
|
-
| 8 |
|
|
523
|
-
| 16 | **1,
|
|
558
|
+
| 1 | 213,880 | 216,584 | 59 MB |
|
|
559
|
+
| 2 | 394,471 | 403,393 | 79 MB |
|
|
560
|
+
| 4 | 737,777 | 768,810 | 101 MB |
|
|
561
|
+
| 8 | 1,256,838 | 1,349,533 | 139 MB |
|
|
562
|
+
| 16 | **1,548,835** | **1,697,313** | 215 MB |
|
|
524
563
|
|
|
525
564
|
<p align="center"><img alt="checkout thread scaling" src="assets/benchmark_threads.svg" width="72%"></p>
|
|
526
565
|
|
|
@@ -18,14 +18,39 @@ count. `scripts/spikein_ratio.py` computes the published spike-in validation met
|
|
|
18
18
|
and merging reported separately.
|
|
19
19
|
|
|
20
20
|
**Throughput and footprint**: every stage threads and every stage is byte-identical at any `-t` --
|
|
21
|
-
checkout 1.
|
|
21
|
+
checkout 1.55 M reads/s end to end (1.70 M matching), refine 1.55 M, assemble 2.47 M, all at 16
|
|
22
22
|
threads; ~22 bytes per distinct UMI against a hash map's ~48. Verified under the thread sanitizer
|
|
23
23
|
as well as by comparison at 1..16 threads.
|
|
24
24
|
The counters are not yet partitioned, which is the open memory item and lands with `.mig` buckets.
|
|
25
25
|
|
|
26
|
+
**QC**: twenty gnuplot panels over the tables the stages write, including the four figures a user
|
|
27
|
+
already knows how to read -- Cell Ranger's barcode rank plot on unique UMIs, the MIG size spectrum
|
|
28
|
+
with molecules *and* reads, the rank/Zipf curve, and consensus quality as a box over an exact
|
|
29
|
+
`(depth, quality)` grid.
|
|
30
|
+
|
|
26
31
|
Milestones are ordered by risk, not by pipeline order: the consensus quality model is the
|
|
27
32
|
scientific claim and is validated before any throughput work.
|
|
28
33
|
|
|
34
|
+
## What is left, in the order it should be done
|
|
35
|
+
|
|
36
|
+
Everything below is either open or half-open. Nothing else on this page is.
|
|
37
|
+
|
|
38
|
+
| # | item | milestone | why it is next |
|
|
39
|
+
|---|---|---|---|
|
|
40
|
+
| 1 | `.mig` bucket output from `checkout` | M2 | the **only** unbounded allocation left: the UMI counters are 8.8 GB at NovaSeq scale, held in one piece, and this is what bounds them. `checkout` warns past 1 GB and that is a warning, not a fix |
|
|
41
|
+
| 2 | Bucketed correction, two passes with the key rotated | M3 | same problem one stage on. `refine` holds the whole barcode table; a plain range partition splits a barcode from its neighbour for the top b/2 positions, so the rotation is the fix and it is understood, not designed |
|
|
42
|
+
| 3 | The template's own error split | M3 | the pattern's constant bases calibrate the **primer**, not the polymerase — the fitted intercept is a synthesis defect rate. Separating sequencing from RT/PCR error needs a different standard, and until it exists `--rt-error` is a named class rather than a measurement |
|
|
43
|
+
| 4 | `--rt-error auto` | M1 | falls straight out of (3). The floor is a property of the enzyme and the cycle count, so fitting it per dataset is the honest default once there is something to fit against |
|
|
44
|
+
| 5 | i7xi5 contingency table | M2 | the only way index hopping is actually estimable, and it needs nothing that is not already in the headers |
|
|
45
|
+
| 6 | `2026-migec-benchmark`, the published comparisons | M5 | MIGEC v1, MAGERI, UMI-tools, Calib, fgbio, Cell Ranger, UMI-VarCal, UMIErrorCorrect. This is what the version number is waiting on, not the code |
|
|
46
|
+
| 7 | ctDNA ground truth, COSMIC at 0.005-0.075 VAF | M5 | the one claim with no ground truth yet. Never: Maruzani's deposited runs carry **no** UMI, so they have to be simulated -- and their own rule assumes co-terminal reads, which X1 falsified |
|
|
47
|
+
| 8 | R1/R2 overlap merge | M1 | a special case of `--contig`'s placement, never a second matcher in `checkout` |
|
|
48
|
+
| 9 | Bit-parallel matcher | M2 | last, deliberately: the scan is O(offsets x pattern) and is **not** the bottleneck. It goes in when a benchmark says so |
|
|
49
|
+
|
|
50
|
+
Two things are blocked on data rather than on work: **Britanova et al aging** (bulk TCR, shallow --
|
|
51
|
+
the real 1-3 reads/UMI dataset) lives on aldan3 and has not been pulled, and the ctDNA ground truth
|
|
52
|
+
above has to be built rather than downloaded.
|
|
53
|
+
|
|
29
54
|
## M0 — skeleton, format, simulator
|
|
30
55
|
|
|
31
56
|
- [x] Archive the Groovy implementation on `legacy-v1` / `v1-final`, start master fresh
|
|
@@ -105,7 +130,7 @@ scientific claim and is validated before any throughput work.
|
|
|
105
130
|
when that makes contigs untrustworthy
|
|
106
131
|
- [x] Shallow libraries (1-3 reads/UMI) run, report the coverage histogram, threshold nothing, and
|
|
107
132
|
say that the UMI is buying counting rather than error correction. Benchmarked as the
|
|
108
|
-
memory-hostile shape:
|
|
133
|
+
memory-hostile shape: 1,179,549 reads/s at 1.02 reads/UMI, 282 B resident per distinct barcode
|
|
109
134
|
- [x] **`--rt-error` names the chemistry** rather than guessing: `rt` 1e-4 (default, Q40 -- 10x's
|
|
110
135
|
figure for the V(D)J RT and X2's own measurement), `medium` 1e-5, `high` 1e-6, or the rate.
|
|
111
136
|
Never: it is the ONE-MOLECULE floor; 10x's Q60 needs two UMIs to agree and that is arda's job
|
|
@@ -175,11 +175,11 @@ comparators for M5, `UMI-VarCal` and `UMIErrorCorrect`.
|
|
|
175
175
|
|
|
176
176
|
| Item | Value |
|
|
177
177
|
|---|---|
|
|
178
|
-
| Use | the
|
|
178
|
+
| Use | the panels the README embeds: barcode rank, MIG size spectrum, rank/Zipf, consensus quality, and the two barcode-error-against-depth panels |
|
|
179
179
|
| Regenerate | `python scripts/example_figures.py` -- simulates, runs all three stages, copies the tables, redraws |
|
|
180
|
-
| Tables | `assets/PBMC.cell_rank.tsv`, `PBMC.sizes.tsv`, `assemble.quality_by_depth.tsv`, `assemble.coverage.tsv` |
|
|
180
|
+
| Tables | `assets/PBMC.cell_rank.tsv`, `PBMC.sizes.tsv`, `PBMC.umi_errors.tsv`, `assemble.quality_by_depth.tsv`, `assemble.coverage.tsv` |
|
|
181
181
|
| Shape | 120 cells x 40-200 molecules, Pareto(1.1) depth, plus 4,000 ambient barcodes with 1-3 molecules each; 0.2% barcode and 1% payload error, seed 3 |
|
|
182
|
-
| Why that shape | a uniform simulator draws
|
|
182
|
+
| Why that shape | a uniform simulator draws every panel as a straight line. The Pareto tail reaching ~19,000 reads on one molecule is also what makes the barcode-error panels show anything: saturation of the `3L` neighbour shell is only visible on a molecule deep enough to have spawned all of it. The ambient population is what makes the knee; the Pareto tail is what gives the rank curve slope; the per-base error is what gives emitted quality a spread to draw quartiles over |
|
|
183
183
|
| Provenance | **derived (simulated)** -- never presented as a measurement. `assets/SRR1763769.mig.tsv` beside it is experimental; see the Primer ID entry |
|
|
184
184
|
| Storage | the TSVs are committed, the FASTQ is not (`--keep` writes it locally) |
|
|
185
185
|
|