migec 2.0.0a4__tar.gz → 2.2.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Potentially problematic release.
This version of migec might be problematic. Click here for more details.
- {migec-2.0.0a4 → migec-2.2.0}/CHANGELOG.md +146 -0
- {migec-2.0.0a4 → migec-2.2.0}/CLAUDE.md +82 -5
- {migec-2.0.0a4 → migec-2.2.0}/PKG-INFO +98 -27
- {migec-2.0.0a4 → migec-2.2.0}/README.md +97 -26
- {migec-2.0.0a4 → migec-2.2.0}/ROADMAP.md +45 -7
- {migec-2.0.0a4 → migec-2.2.0}/SOURCES.md +12 -0
- migec-2.2.0/assets/PBMC.cell_rank.tsv +144 -0
- migec-2.2.0/assets/PBMC.sizes.tsv +160 -0
- migec-2.2.0/assets/PBMC.umi_errors.tsv +160 -0
- migec-2.2.0/assets/assemble.coverage.tsv +16 -0
- migec-2.2.0/assets/assemble.quality_by_depth.tsv +14 -0
- migec-2.2.0/assets/assemble_coverage.gp +24 -0
- migec-2.2.0/assets/assemble_coverage.svg +230 -0
- {migec-2.0.0a4 → migec-2.2.0}/assets/benchmark_threads.gp +12 -6
- migec-2.2.0/assets/benchmark_threads.json +47 -0
- {migec-2.0.0a4 → migec-2.2.0}/assets/benchmark_threads.svg +87 -87
- migec-2.2.0/assets/benchmark_threads.tsv +6 -0
- migec-2.2.0/assets/cell_rank.gp +27 -0
- migec-2.2.0/assets/cell_rank.svg +234 -0
- migec-2.2.0/assets/consensus_error.gp +24 -0
- migec-2.2.0/assets/consensus_error.svg +287 -0
- migec-2.2.0/assets/consensus_layout.gp +24 -0
- {migec-2.0.0a4 → migec-2.2.0}/assets/consensus_layout.svg +69 -79
- migec-2.2.0/assets/consensus_quality.gp +29 -0
- migec-2.2.0/assets/consensus_quality.svg +325 -0
- migec-2.2.0/assets/mig_size_spectrum.gp +39 -0
- migec-2.2.0/assets/mig_size_spectrum.svg +511 -0
- migec-2.2.0/assets/mig_size_zipf.gp +27 -0
- migec-2.2.0/assets/mig_size_zipf.svg +240 -0
- {migec-2.0.0a4 → migec-2.2.0}/assets/pipeline.dot +18 -6
- migec-2.2.0/assets/pipeline.svg +163 -0
- migec-2.2.0/assets/umi_error_children.gp +32 -0
- migec-2.2.0/assets/umi_error_children.svg +578 -0
- migec-2.2.0/assets/umi_error_rate.gp +47 -0
- migec-2.2.0/assets/umi_error_rate.svg +532 -0
- {migec-2.0.0a4 → migec-2.2.0}/docs/assemble.rst +2 -2
- {migec-2.0.0a4 → migec-2.2.0}/docs/commands.rst +1 -1
- {migec-2.0.0a4 → migec-2.2.0}/docs/examples.rst +39 -0
- {migec-2.0.0a4 → migec-2.2.0}/docs/formats.rst +71 -0
- {migec-2.0.0a4 → migec-2.2.0}/docs/index.rst +2 -2
- {migec-2.0.0a4 → migec-2.2.0}/docs/method.rst +3 -0
- {migec-2.0.0a4 → migec-2.2.0}/docs/nextflow.rst +3 -3
- {migec-2.0.0a4 → migec-2.2.0}/docs/performance.rst +82 -45
- migec-2.2.0/docs/plots.rst +198 -0
- {migec-2.0.0a4 → migec-2.2.0}/docs/refine.rst +9 -0
- migec-2.2.0/docs/umi_errors.rst +167 -0
- {migec-2.0.0a4 → migec-2.2.0}/include/migec/assemble.hpp +11 -0
- {migec-2.0.0a4 → migec-2.2.0}/include/migec/parallel.hpp +26 -10
- {migec-2.0.0a4 → migec-2.2.0}/include/migec/refine.hpp +12 -0
- {migec-2.0.0a4 → migec-2.2.0}/include/migec/umi_stats.hpp +5 -1
- {migec-2.0.0a4 → migec-2.2.0}/include/migec/version.hpp +1 -1
- {migec-2.0.0a4 → migec-2.2.0}/integrations/nextflow/modules/migec/assemble/main.nf +1 -1
- {migec-2.0.0a4 → migec-2.2.0}/integrations/nextflow/modules/migec/checkout/main.nf +1 -1
- {migec-2.0.0a4 → migec-2.2.0}/integrations/nextflow/modules/migec/plot/main.nf +1 -1
- {migec-2.0.0a4 → migec-2.2.0}/integrations/nextflow/modules/migec/refine/main.nf +1 -1
- {migec-2.0.0a4 → migec-2.2.0}/project/README.md +9 -0
- {migec-2.0.0a4 → migec-2.2.0}/pyproject.toml +1 -1
- {migec-2.0.0a4 → migec-2.2.0}/python/migec/__init__.py +1 -1
- {migec-2.0.0a4 → migec-2.2.0}/python/migec/assemble.py +53 -0
- {migec-2.0.0a4 → migec-2.2.0}/python/migec/checkout.py +5 -3
- {migec-2.0.0a4 → migec-2.2.0}/python/migec/cli.py +4 -3
- {migec-2.0.0a4 → migec-2.2.0}/python/migec/plot.py +198 -28
- {migec-2.0.0a4 → migec-2.2.0}/python/migec/refine.py +8 -0
- migec-2.2.0/scripts/example_figures.py +113 -0
- {migec-2.0.0a4 → migec-2.2.0}/skills/migec/SKILL.md +33 -2
- {migec-2.0.0a4 → migec-2.2.0}/src/_bindings.cpp +19 -0
- {migec-2.0.0a4 → migec-2.2.0}/src/assemble.cpp +46 -16
- {migec-2.0.0a4 → migec-2.2.0}/src/refine.cpp +234 -18
- {migec-2.0.0a4 → migec-2.2.0}/src/umi_stats.cpp +14 -7
- {migec-2.0.0a4 → migec-2.2.0}/tests/synthetic/test_refine.py +2 -0
- migec-2.2.0/tests/synthetic/test_umi_errors.py +135 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/unit/test_plot.py +69 -0
- migec-2.0.0a4/assets/assemble.coverage.tsv +0 -10
- migec-2.0.0a4/assets/assemble_coverage.gp +0 -18
- migec-2.0.0a4/assets/assemble_coverage.svg +0 -212
- migec-2.0.0a4/assets/benchmark_threads.json +0 -47
- migec-2.0.0a4/assets/benchmark_threads.tsv +0 -6
- migec-2.0.0a4/assets/consensus_error.gp +0 -17
- migec-2.0.0a4/assets/consensus_error.svg +0 -335
- migec-2.0.0a4/assets/consensus_layout.gp +0 -18
- migec-2.0.0a4/assets/consensus_quality.gp +0 -18
- migec-2.0.0a4/assets/consensus_quality.svg +0 -374
- migec-2.0.0a4/assets/pipeline.svg +0 -164
- migec-2.0.0a4/docs/plots.rst +0 -113
- {migec-2.0.0a4 → migec-2.2.0}/.github/workflows/ci.yml +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/.github/workflows/docs.yml +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/.github/workflows/publish.yml +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/.github/workflows/testpypi.yml +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/.gitignore +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/CMakeLists.txt +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/LICENSE +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/assets/SRR1763769.mig.tsv +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/docs/Makefile +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/docs/barcode_space.rst +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/docs/checkout.rst +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/docs/conf.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/docs/downstream.rst +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/docs/fragmented.rst +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/docs/grouping.rst +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/docs/installation.rst +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/docs/layouts.rst +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/docs/nulls.rst +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/docs/quality_floor.rst +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/docs/reference.rst +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/docs/roadmap.rst +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/docs/subsample.rst +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/docs/suggest.rst +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/docs/umi_statistics.rst +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/docs/validation.rst +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/include/migec/checkout.hpp +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/include/migec/consensus.hpp +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/include/migec/fastq.hpp +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/include/migec/mig_record.hpp +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/include/migec/pattern.hpp +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/include/migec/resource.hpp +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/include/migec/subsample.hpp +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/include/migec/suggest.hpp +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/include/migec/types.hpp +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/include/migec/whitelist.hpp +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/integrations/nextflow/README.md +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/integrations/nextflow/modules/migec/assemble/meta.yml +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/integrations/nextflow/modules/migec/checkout/meta.yml +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/integrations/nextflow/modules/migec/environment.yml +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/integrations/nextflow/modules/migec/refine/meta.yml +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/integrations/nextflow/nextflow.config +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/integrations/nextflow/subworkflows/migec/main.nf +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/notebooks/barcode_space.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/notebooks/platforms.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/notebooks/refine_diagnostics.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/project/design-assemble.md +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/project/design-benchmarks.md +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/project/design-checkout.md +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/project/design-io-interop.md +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/project/design-refine.md +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/project/design-repo.md +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/project/review-algorithms.md +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/project/review-scope.md +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/python/migec/sheet.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/python/migec/subsample.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/python/migec/suggest.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/scripts/benchmark_threads.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/scripts/collision_check.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/scripts/compare_calib.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/scripts/correction_accuracy.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/scripts/diagnose.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/scripts/permutation_nulls.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/scripts/quality_floor.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/scripts/read_start_dispersion.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/scripts/spikein_ratio.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/setup.sh +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/src/checkout.cpp +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/src/consensus.cpp +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/src/fastq.cpp +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/src/mig_record.cpp +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/src/pattern.cpp +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/src/resource.cpp +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/src/subsample.cpp +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/src/suggest.cpp +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/src/types.cpp +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/src/whitelist.cpp +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/__init__.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/benchmark/__init__.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/benchmark/test_assemble_speed.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/benchmark/test_checkout_speed.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/benchmark/test_refine_speed.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/conftest.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/cpp/doctest.h +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/cpp/test_checkout.cpp +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/cpp/test_checkout_run.cpp +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/cpp/test_consensus.cpp +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/cpp/test_fastq.cpp +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/cpp/test_mig_record.cpp +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/cpp/test_parallel_stages.cpp +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/cpp/test_pattern.cpp +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/cpp/test_types.cpp +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/cpp/test_umi_stats.cpp +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/cpp/test_whitelist.cpp +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/data/migec_spikein_truth.tsv +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/realworld/__init__.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/synthetic/__init__.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/synthetic/_sim.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/synthetic/test_assemble.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/synthetic/test_barcode_space.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/synthetic/test_checkout_end_to_end.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/synthetic/test_dual_end.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/synthetic/test_grouping_accuracy.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/synthetic/test_nulls.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/synthetic/test_quality_floor.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/synthetic/test_sim.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/synthetic/test_subsample.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/synthetic/test_suggest.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/synthetic/test_thread_invariance.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/unit/__init__.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/unit/test_bindings.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/unit/test_compare_calib.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/unit/test_core.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/unit/test_downstream.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/unit/test_sheet_and_cli.py +0 -0
- {migec-2.0.0a4 → migec-2.2.0}/tests/unit/test_spikein_ratio.py +0 -0
|
@@ -4,6 +4,152 @@ Hand-written and prose-heavy: each entry says what changed and, where it matters
|
|
|
4
4
|
prevents. Releases before 2.0.0 are the Groovy MIGEC and are described by their git tags on the
|
|
5
5
|
`legacy-v1` branch.
|
|
6
6
|
|
|
7
|
+
## 2.2.0 — 2026-08-13
|
|
8
|
+
|
|
9
|
+
Throughput and one new measurement. No breaking change: every 2.1.0 command, flag and output file
|
|
10
|
+
behaves as it did, and the additions are a new table, four new JSON fields and two new plot panels.
|
|
11
|
+
|
|
12
|
+
### The barcode error rate, measured at every depth
|
|
13
|
+
|
|
14
|
+
`refine` writes `<sample>.umi_errors.tsv`: one row per exact parent depth, carrying the distinct
|
|
15
|
+
error children that parent spawned, the reads in them, and the error rate each implies. A parent
|
|
16
|
+
carrying `c` reads offered `c*L` barcode bases to be miscalled, so the same eps falls out two ways:
|
|
17
|
+
|
|
18
|
+
distinct children u(c) = 3L (1 - exp(-c eps / 3)) -> eps = -(3/c) ln(1 - u/3L)
|
|
19
|
+
reads in children r(c) = c L eps -> eps = r / (c L)
|
|
20
|
+
|
|
21
|
+
A barcode has only `3L` neighbours one substitution away, so the first **saturates**; reads have no
|
|
22
|
+
ceiling, so the second does not. Where the two part company on the figure is where this library's
|
|
23
|
+
barcode neighbourhood filled up, read off the data instead of predicted. Past saturation the first
|
|
24
|
+
is left blank rather than reported as a small number: inverting a full neighbourhood returns "no
|
|
25
|
+
errors" for the most error-ridden library there can be.
|
|
26
|
+
|
|
27
|
+
New report lines and JSON fields `error_at_depth`, `error_phred`, `error_from_children`,
|
|
28
|
+
`error_depth`. The number to quote is `error_at_depth` — restricted to parents seen >= 10 times,
|
|
29
|
+
where correction is near-complete — and `error_phred` is the same thing as a Phred so it can be put
|
|
30
|
+
beside the barcode's own reported Q. On a diverse library sequenced 25 deep with 1e-3 injected, the
|
|
31
|
+
distance-1 excess gives 9.73e-04, the children give 9.89e-04 (Q30) and all depths give 9.98e-04.
|
|
32
|
+
Three routes to one number agreeing within 3% is what makes any of them believable.
|
|
33
|
+
|
|
34
|
+
Never: **both estimators are bounded by the merges correction actually made**, so neither is
|
|
35
|
+
saturation-free. Against a known injected rate, as a fraction of truth:
|
|
36
|
+
|
|
37
|
+
| occupancy | 0.2% | 2.3% | 9.8% | 33% | 100% |
|
|
38
|
+
|---|---|---|---|---|---|
|
|
39
|
+
| distance-1 excess | 0.97 | 0.96 | 0.76 | 0.45 | 0.001 |
|
|
40
|
+
| from the children | **0.99** | **0.95** | **0.88** | **0.62** | 0.00 |
|
|
41
|
+
|
|
42
|
+
The children estimate is the better of the two wherever either works, and at 100% they both go to
|
|
43
|
+
zero for the same reason: on a full barcode space `correct_umis` refuses to merge, correctly,
|
|
44
|
+
because a distance-1 neighbour there is more likely a real molecule than a child. The `saturated`
|
|
45
|
+
flag is what says the answer is a floor — read it, do not read this table instead of it. And read
|
|
46
|
+
the table at depth: a child whose parent was never sequenced cannot be counted, which at 1-3
|
|
47
|
+
reads/UMI is 80% of them. `tests/synthetic/test_umi_errors.py` holds all of it, `docs/umi_errors.rst`
|
|
48
|
+
works through it.
|
|
49
|
+
|
|
50
|
+
Two new `migec plot` panels over that table, `umi_error_children` and `umi_error_rate`, with the
|
|
51
|
+
`3L` ceiling and the reported estimate drawn as reference lines. Note: `neighbours` and `estimate`
|
|
52
|
+
are constant down their columns on purpose — the panels draw them, and a figure that needs a value
|
|
53
|
+
its own table does not carry will one day disagree with the report.
|
|
54
|
+
|
|
55
|
+
### `reads in them` is points, never a line
|
|
56
|
+
|
|
57
|
+
The `mig_size_spectrum` panel drew its reads series `with lines` over a table that has one row per
|
|
58
|
+
**exact** size. Past the head of the distribution almost every size holds exactly one molecule, so
|
|
59
|
+
reads == size, and the line rendered the `y = x` diagonal as the most prominent feature of the
|
|
60
|
+
figure — a tautology drawn as a second mode. Where two or three molecules shared a size the same
|
|
61
|
+
line sawtoothed between `size*1` and `size*2`, which is integer quantisation drawn as signal, and it
|
|
62
|
+
bridged gaps in the support where no size was observed at all. It is points now, and the new
|
|
63
|
+
barcode-error panels are points for the same reason.
|
|
64
|
+
|
|
65
|
+
### The thread helper was the bottleneck
|
|
66
|
+
|
|
67
|
+
With all three stages threaded, a sampling profile of
|
|
68
|
+
`assemble` put **21% of all CPU samples, across every thread, on a single instruction**: the atomic
|
|
69
|
+
`fetch_add` in `parallel_for` that handed out one item at a time. When an item is one read's tag
|
|
70
|
+
scan, sixteen cores serialise on one cache line and the counter costs more than the work it
|
|
71
|
+
distributes. Items are claimed in batches now, sized so each worker takes ~8 turns and capped so a
|
|
72
|
+
ten-million-item scan still hands out ten thousand batches. Never: the batch collapses to 1 when
|
|
73
|
+
there are few items, which is the uneven case -- one bucket per item, one holding ten molecules and
|
|
74
|
+
the next ten million -- that the counter exists for.
|
|
75
|
+
|
|
76
|
+
Two serial blocks fell out with it, both read-only `3L`-binary-search scans of the barcode table:
|
|
77
|
+
the **distance-1 census** in `estimate_umi_error` (checkout's per-sample statistics tail) and
|
|
78
|
+
refine's **residual-FDR scan**, which was 0.53 s of a 2.17 s run on one core after everything around
|
|
79
|
+
it had been parallelised. Both tally integers into per-worker counters summed afterwards, so `-t`
|
|
80
|
+
still changes nothing but the clock -- verified byte-for-byte at `-t` 1/3/16 on a clean library and
|
|
81
|
+
on one with injected barcode errors (residual 1,294 either way), and under the thread sanitizer
|
|
82
|
+
(104 cases, 224,116 assertions, no race).
|
|
83
|
+
|
|
84
|
+
Two allocation fixes on paths every read takes: refine's rewrite chunk is now **assigned into
|
|
85
|
+
rather than cleared** -- the same trade assemble's partition already made, since `clear()` destroys
|
|
86
|
+
four `std::string` per record -- and the rewrite's comment buffer and unpacked barcode are worker
|
|
87
|
+
scratch reused read after read, with the cell and UMI halves taken as `string_view` instead of
|
|
88
|
+
`substr`. That is four allocations a read removed; `rewrite_seconds` fell 0.51 s to 0.38 s.
|
|
89
|
+
`assemble` also unpacked each group's UMI twice, once for the composition tally and once for the
|
|
90
|
+
record name.
|
|
91
|
+
|
|
92
|
+
| stage | 1 thread | 16 threads | was, at 16 |
|
|
93
|
+
|---|---|---|---|
|
|
94
|
+
| `checkout` | 213,880 | **1,548,835** | 1,056,472 |
|
|
95
|
+
| `refine` | 617,802 | **1,554,156** | 1,012,368 |
|
|
96
|
+
| `assemble` | 554,106 | **2,470,928** | 2,051,937 |
|
|
97
|
+
|
|
98
|
+
reads/s, 500 k-read sample except checkout at 2 M; `assets/benchmark_threads.tsv` is regenerated
|
|
99
|
+
and the figure redrawn from it. checkout's gap between end-to-end and matching throughput fell from
|
|
100
|
+
20% to 9%.
|
|
101
|
+
|
|
102
|
+
Note: re-measured while it was in hand, a **64 k chunk in assemble's partition is still 32% faster**
|
|
103
|
+
(3,075,506 reads/s against 2,324,403 on 4 M reads) and is still not taken: it costs 16 MB of
|
|
104
|
+
resident chunk, which at NovaSeq scale makes pass 1 the memory peak and breaks the property that a
|
|
105
|
+
finer partition costs less rather than more. The benchmark tier does not fail at 64 k on a 500 k
|
|
106
|
+
corpus -- the objection is one of scale -- so the number is written down next to the constant.
|
|
107
|
+
|
|
108
|
+
## 2.1.0 — 2026-08-13
|
|
109
|
+
|
|
110
|
+
First non-alpha of the rewrite. The three stages, the eight commands and the on-disk formats have
|
|
111
|
+
been stable across a3 and a4; what this release adds is the QC layer catching up with them.
|
|
112
|
+
|
|
113
|
+
**Four figures you already know how to read.** They come off new tables rather than out of a new
|
|
114
|
+
computation, so each is still redrawable from a committed TSV:
|
|
115
|
+
|
|
116
|
+
- **The barcode rank plot**, on [Cell Ranger's](https://www.10xgenomics.com/support/software/cell-ranger/latest/advanced/cr-ab-barcode-rank-plot)
|
|
117
|
+
axes, because it is the figure every user of a droplet protocol has seen: barcodes sorted by
|
|
118
|
+
content, log-log, knee where cells stop. Never: the y axis is **unique UMIs, never reads** — one
|
|
119
|
+
over-amplified molecule would otherwise put an empty droplet high on the curve, which is the
|
|
120
|
+
exact artefact the plot exists to show. The call is drawn on the curve, not described in a
|
|
121
|
+
caption. New table `<sample>.cell_rank.tsv`.
|
|
122
|
+
- **The MIG size spectrum**, molecules *and* the reads they account for, against `log(1 + size)`.
|
|
123
|
+
Both series, because they peak in different places the moment a library is over-sequenced: most
|
|
124
|
+
molecules are shallow, most reads are in the deep ones, and a figure with only one of them says
|
|
125
|
+
the opposite of a figure with only the other. `log1p`, so a molecule seen once has a place on
|
|
126
|
+
the axis.
|
|
127
|
+
- **The rank/Zipf curve**, molecule size against rank on log-log. Never: this is why the new
|
|
128
|
+
`<sample>.sizes.tsv` is written at **exact** sizes and not power-of-two bins — four bins make
|
|
129
|
+
four steps, and a straight line cannot be told from a bent one. It costs one row per distinct
|
|
130
|
+
depth, a few thousand on a real library, not one row per molecule.
|
|
131
|
+
- **Unique UMIs and reads per sample barcode**, off `checkout.summary.tsv`. The multiplexed
|
|
132
|
+
analogue of the same question.
|
|
133
|
+
|
|
134
|
+
**Consensus quality is a box, not a thinned scatter.** Emitted quality is discrete and capped at
|
|
135
|
+
the RT floor, so at any real depth every molecule sits on one or two integers: a cloud of dots
|
|
136
|
+
draws that as a flat line whether the bin holds ten molecules or ten million, and the `every 17`
|
|
137
|
+
that kept the SVG small threw away the tails that were the only thing the cloud could have shown.
|
|
138
|
+
`assemble` now accumulates the exact joint distribution of (depth bin, rounded Phred) — both are
|
|
139
|
+
small integers, so it is 61 counters per bin per bucket — and `assemble.quality_by_depth.tsv`
|
|
140
|
+
carries real order statistics over every molecule.
|
|
141
|
+
|
|
142
|
+
**Publication defaults on every panel.** Transparent background, so one SVG serves a light README,
|
|
143
|
+
a dark README and print. One ink colour (`#808080`), which reads on both. **The legend is inside
|
|
144
|
+
the plot box**: a key in the margin makes every figure wider than its data and is the first thing a
|
|
145
|
+
journal asks you to move. Frame is 760x520 rather than 900x560 wide.
|
|
146
|
+
|
|
147
|
+
**The pipeline figure is page-shaped.** `rankdir = TB` and three `rank = same` groups pinning each
|
|
148
|
+
side tool level with the data it reads; it was a 5:1 strip that filled the README column with air.
|
|
149
|
+
Transparent, and `minibwa` is in the downstream box.
|
|
150
|
+
|
|
151
|
+
Twenty panels now, from sixteen. `docs/formats.rst` documents every QC table and its columns.
|
|
152
|
+
|
|
7
153
|
## 2.0.0a4 — 2026-08-13
|
|
8
154
|
|
|
9
155
|
**The partition threads, and the memory estimate behind it was wrong.** With the consensus already
|
|
@@ -261,6 +261,26 @@ correction is written up in `project/review-algorithms.md`.
|
|
|
261
261
|
is where the answer matters most. It caught me twice: once in the correction posterior, once in
|
|
262
262
|
the residual-FDR estimator. Both now use payload agreement as well. On a 1.23 reads/barcode
|
|
263
263
|
library the count-only residual is 0 and the full one is 1,294.
|
|
264
|
+
- **The barcode error rate is measured at every DEPTH, not once for the library.**
|
|
265
|
+
`<sample>.umi_errors.tsv`, one row per exact parent depth: a parent carrying `c` reads offered
|
|
266
|
+
`c*L` barcode bases to be miscalled, so `u(c) = 3L(1-exp(-c eps/3))` from the distinct children
|
|
267
|
+
and `r(c) = c L eps` from the reads in them invert to the same eps. Reported as
|
|
268
|
+
`error_at_depth` (parents seen >= 10 times, where correction is near-complete), `error_phred`,
|
|
269
|
+
and `error_from_children` (all depths, a LOWER bound). On 1e-3 injected: distance-1 9.73e-04,
|
|
270
|
+
children 9.89e-04 = Q30, all depths 9.98e-04.
|
|
271
|
+
- **Never: neither error estimator is saturation-free — both are bounded by the merges correction
|
|
272
|
+
MADE.** As a fraction of an injected truth: distance-1 0.97 / 0.96 / 0.76 / 0.45 / 0.001 and
|
|
273
|
+
children 0.99 / 0.95 / 0.88 / 0.62 / 0.00 at occupancy 0.2% / 2.3% / 9.8% / 33% / 100%. The
|
|
274
|
+
children estimate wins everywhere either works; at 100% BOTH are zero, because `correct_umis`
|
|
275
|
+
refuses to merge on a full space and is right to. `saturated` is the flag that says the answer
|
|
276
|
+
is a floor -- do not read the table instead of it. `docs/umi_errors.rst`,
|
|
277
|
+
`tests/synthetic/test_umi_errors.py`.
|
|
278
|
+
- **Never: a sparse spectrum is POINTS, never a line.** `mig_size_spectrum` drew "reads in them"
|
|
279
|
+
`with lines` over a table at EXACT sizes, where past the head almost every size holds one
|
|
280
|
+
molecule -- so reads == size and the line drew the `y = x` diagonal as the figure's most
|
|
281
|
+
prominent feature, a tautology that reads as a second mode. Two or three molecules on a size made
|
|
282
|
+
it sawtooth between `size*1` and `size*2`, and it bridged gaps where nothing was observed. Same
|
|
283
|
+
rule for the two `umi_error_*` panels.
|
|
264
284
|
- **The MIG-size FDR threshold is reported, never applied** (2026-08-13). Measured residual, not
|
|
265
285
|
derived. 5.25% of 1-read molecules at 1.23 reads/barcode, 0% at 4.62.
|
|
266
286
|
- **Dual-end barcodes work (2026-08-13)**: column 3 of the sheet is the slave pattern, on the
|
|
@@ -311,7 +331,24 @@ correction is written up in `project/review-algorithms.md`.
|
|
|
311
331
|
moves the UMI into the FASTQ header and SRA rewrites headers. Confirmed on three runs by the
|
|
312
332
|
missing template-switch `GGG`. `suggest` reports it correctly. The `smarter-umi` preset is
|
|
313
333
|
sourced from the pipeline, not fitted to the data. In `SOURCES.md`.
|
|
314
|
-
-
|
|
334
|
+
- **The four familiar QC figures are in (2.1.0)**: Cell Ranger's barcode rank plot
|
|
335
|
+
(`<sample>.cell_rank.tsv`), the MIG size spectrum on log1p with molecules AND reads, the
|
|
336
|
+
rank/Zipf curve, and unique UMIs per sample barcode. Never: the knee plot's y axis is **unique
|
|
337
|
+
UMIs, never reads** -- one over-amplified molecule otherwise puts an empty droplet high on the
|
|
338
|
+
curve, which is the artefact the plot exists to show. Never: `<sample>.sizes.tsv` is at EXACT
|
|
339
|
+
sizes, never power-of-two bins -- the Zipf curve is its cumulative count and four bins make four
|
|
340
|
+
steps.
|
|
341
|
+
- **Never: a thinned scatter is not a distribution.** `consensus_quality` drew `every 17` molecules
|
|
342
|
+
as grey dots. Emitted quality is DISCRETE and capped at the floor, so at any real depth every
|
|
343
|
+
molecule sits on one or two integers: the cloud drew a flat line whether the bin held ten
|
|
344
|
+
molecules or ten million, and the thinning threw away the tails that were the only thing it could
|
|
345
|
+
have shown the line did not. `assemble` accumulates the exact (depth bin, rounded Phred) grid --
|
|
346
|
+
61 counters per bin -- and the panel is a candlestick over real order statistics.
|
|
347
|
+
- **Publication defaults on every panel (2.1.0)**: transparent background so one SVG serves a light
|
|
348
|
+
README, a dark README and print; one ink colour `#808080` that reads on both; **the key inside the
|
|
349
|
+
plot box**, because a legend gutter makes every figure wider than its data. Frame 760x520. The
|
|
350
|
+
pipeline figure is `rankdir = TB` with three `rank = same` groups, not a 5:1 strip.
|
|
351
|
+
- **`migec plot` works (2026-08-13, 2.0.0a3)**: twenty gnuplot panels over the TSVs the stages
|
|
315
352
|
already write, ColorBrewer Dark2, A/C/G/T always the same four colours. Never: it computes nothing
|
|
316
353
|
— a figure that cannot be redrawn from a committed table is a figure that will disagree with the
|
|
317
354
|
report. gnuplot is **not** a Python dependency; without it the `.gp` scripts are still written,
|
|
@@ -337,11 +374,37 @@ correction is written up in `project/review-algorithms.md`.
|
|
|
337
374
|
but the per-sample counters and the memory figure were one sample's. Fixed; the published table
|
|
338
375
|
is re-measured and now comes from `scripts/benchmark_threads.py`, which writes the TSV the
|
|
339
376
|
figure is drawn from.
|
|
340
|
-
- **All three stages thread (2026-08-13, 2.0.0a3).** refine 222k -> 1.01M reads/s, assemble
|
|
341
|
-
203k ->
|
|
377
|
+
- **All three stages thread (2026-08-13, 2.0.0a3-2.1.0).** refine 222k -> 1.01M reads/s, assemble
|
|
378
|
+
203k -> 2.05M (its partition threaded in a4), checkout unchanged at 1.06M. Never: the FIRST fix was not a thread -- zlib level 6
|
|
342
379
|
was 83% of refine's wall clock compressing an intermediate the next stage immediately
|
|
343
380
|
decompresses. Level 1 is now the default for every stage's output and gave 3x on its own.
|
|
344
381
|
Measure before parallelising.
|
|
382
|
+
- **Never: the thread HELPER was the bottleneck once everything else was threaded (unreleased).**
|
|
383
|
+
`parallel_for` claimed one item per atomic `fetch_add`, and a sampling profile of assemble put
|
|
384
|
+
**21% of all CPU samples across every thread on that one instruction** -- more than the parse it
|
|
385
|
+
was handing out. Sixteen cores serialise on one cache line. Batched now: ~8 turns per worker,
|
|
386
|
+
capped at 1024, collapsing to 1 when items are few, which is the uneven case (one bucket per
|
|
387
|
+
item) the counter exists for. Never: profile the helper, not only the work. Current numbers,
|
|
388
|
+
1 thread -> 16: checkout **213,880 -> 1,548,835** end to end (1,697,313 matching), refine
|
|
389
|
+
**617,802 -> 1,554,156**, assemble **554,106 -> 2,470,928**; `assets/benchmark_threads.tsv` is
|
|
390
|
+
the committed table and the figure is drawn from it.
|
|
391
|
+
- **Two serial scans went with it**: the distance-1 census in `estimate_umi_error` (which IS
|
|
392
|
+
checkout's per-sample statistics tail -- the end-to-end/matching gap fell from 20% to 9%) and
|
|
393
|
+
refine's residual-FDR scan, 0.53 s of a 2.17 s run on one core. Both are read-only `3L` binary
|
|
394
|
+
searches per barcode tallying integers, so per-worker counters summed afterwards keep `-t`
|
|
395
|
+
changing nothing but the clock. `estimate_umi_error` now takes a `threads` argument, defaulted
|
|
396
|
+
to 1 so the C++ test calls are unchanged.
|
|
397
|
+
- **Note: a 64 k chunk in assemble's partition is STILL 32% faster and still not taken**
|
|
398
|
+
(3,075,506 reads/s against 2,324,403 on 4 M, re-measured after the batching). It costs 16 MB of
|
|
399
|
+
resident chunk, which at NovaSeq scale makes pass 1 the memory peak. Never: the benchmark tier
|
|
400
|
+
does NOT fail at 64 k on a 500 k corpus -- the objection is scale, which no test here can see,
|
|
401
|
+
so the number lives next to the constant in `assemble.cpp`. Upgrade path is a persistent worker
|
|
402
|
+
pool, not a bigger chunk.
|
|
403
|
+
- **Never: `clear()` on a chunk of `FastqOwned` is four allocations a record.** assemble's
|
|
404
|
+
partition knew this; refine's rewrite still did it. Assigned into now, and the rewrite's comment
|
|
405
|
+
buffer and unpacked barcode are worker scratch with the cell/UMI halves as `string_view` rather
|
|
406
|
+
than `substr` -- `rewrite_seconds` 0.51 -> 0.38. Same class of bug in `assemble`: each group's
|
|
407
|
+
UMI was unpacked twice, once for the composition tally and once for the record name.
|
|
345
408
|
- **Never: `-t` changes nothing but the wall clock, on every stage.** refine parallelises the
|
|
346
409
|
neighbourhood SCAN (a pure function of the barcode table -- it reads no union-find state) and
|
|
347
410
|
applies the merges serially in the original order, so the result is identical, not equivalent.
|
|
@@ -390,8 +453,22 @@ correction is written up in `project/review-algorithms.md`.
|
|
|
390
453
|
- **Never: `containsKey`, not `?:`, for a nextflow boolean.** Groovy's elvis treats `false` as
|
|
391
454
|
absent, so a per-sample `contig: false` against `params.migec_contig = true` silently meant its
|
|
392
455
|
opposite -- the one direction a per-sample override exists to make possible.
|
|
393
|
-
- Next
|
|
394
|
-
|
|
456
|
+
- **Next, in order. `ROADMAP.md` has the same list with the reasoning; this is the short form.**
|
|
457
|
+
1. **`.mig` bucket output from `checkout`** — the only unbounded allocation left. The UMI
|
|
458
|
+
counters are 8.8 GB at NovaSeq scale held in one piece; checkout warns past 1 GB and a warning
|
|
459
|
+
is not a fix.
|
|
460
|
+
2. **Bucketed correction in `refine`**, two passes with the key rotated. Same problem one stage
|
|
461
|
+
on, and the fix is understood rather than still to be designed.
|
|
462
|
+
3. **The template's own error split** — the pattern's constant bases calibrate the PRIMER, not
|
|
463
|
+
the polymerase, so the RT/PCR separation needs a different standard.
|
|
464
|
+
4. **`--rt-error auto`**, which falls out of 3 and not before it.
|
|
465
|
+
5. **i7xi5 contingency table** — needs nothing that is not already in the headers.
|
|
466
|
+
6. **`2026-migec-benchmark`** and the published comparisons. This is what the version number is
|
|
467
|
+
waiting on, not the code.
|
|
468
|
+
7. **ctDNA ground truth** (COSMIC at 0.005-0.075 VAF). Must be built: Maruzani's deposited runs
|
|
469
|
+
carry no UMI.
|
|
470
|
+
8. **R1/R2 overlap merge**, as a case of `--contig`'s placement and never a second matcher.
|
|
471
|
+
9. **Bit-parallel matcher**, last and deliberately: the scan is not the bottleneck.
|
|
395
472
|
- **Note: Britanova et al aging (bulk TCR, shallow) lives on aldan3** and is the real dataset for the
|
|
396
473
|
1-3 read regime. Not pulled yet. aldan3 compute goes through SLURM, never the frontend.
|
|
397
474
|
- The archive is pushed: `legacy-v1` + tag `v1-final`, and master is the rewrite. Recovery point
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.2
|
|
2
2
|
Name: migec
|
|
3
|
-
Version: 2.0
|
|
3
|
+
Version: 2.2.0
|
|
4
4
|
Summary: UMI barcode extraction, correction and consensus assembly for barcoded sequencing
|
|
5
5
|
Keywords: umi,barcode,consensus,airr,repseq,single-cell,sequencing
|
|
6
6
|
Author-Email: Mikhail Shugay <mikhail.shugay@gmail.com>
|
|
@@ -76,7 +76,7 @@ possible. The difficulty is entirely in the details:
|
|
|
76
76
|
|
|
77
77
|
## Pipeline
|
|
78
78
|
|
|
79
|
-
<p align="center"><img alt="the migec pipeline" src="assets/pipeline.svg" width="
|
|
79
|
+
<p align="center"><img alt="the migec pipeline" src="assets/pipeline.svg" width="42%"></p>
|
|
80
80
|
|
|
81
81
|
Three stages, plus three tools that read no reads: `suggest` says where the barcode is,
|
|
82
82
|
`subsample` cuts a fixture that is still a library, and `plot` draws every QC panel with gnuplot
|
|
@@ -197,8 +197,8 @@ reads 2,000,000
|
|
|
197
197
|
unmatched 0 (0.0%)
|
|
198
198
|
ambiguous 0 (0.0%)
|
|
199
199
|
|
|
200
|
-
|
|
201
|
-
peak RSS
|
|
200
|
+
1.6 s (1,243,801 reads/s) = 1.5 s matching on 8 threads + 0.1 s UMI statistics
|
|
201
|
+
peak RSS 136.0 MB of which UMI counters 11.5 MB
|
|
202
202
|
|
|
203
203
|
sample reads UMIs reads/UMI UMI len eff len
|
|
204
204
|
S1 500,000 125,000 4.00 12 12.00
|
|
@@ -265,7 +265,7 @@ migec assemble out/S1.fq.gz -o cons/ --contig # random-primed reads that til
|
|
|
265
265
|
A molecule is **sample + cell barcode + UMI**, never the UMI alone — the same UMI in two cells is
|
|
266
266
|
two molecules, and that is the design rather than a defect. Reads are range partitioned on the
|
|
267
267
|
packed key into `.mig` buckets and one bucket is sorted at a time, so nothing scales with the
|
|
268
|
-
library:
|
|
268
|
+
library: 2,423,777 reads/s, and 123 MB at 16 buckets against 203 MB at one.
|
|
269
269
|
|
|
270
270
|
The per-column posterior is `LL[j][b] = Σ_i (r==b ? log(1−e) : log(e/3))`, and then the number that
|
|
271
271
|
matters:
|
|
@@ -295,10 +295,8 @@ independently (1.54e-4 on SRR1763769), and the polymerase classes come from
|
|
|
295
295
|
2.6e-6, Pwo 2.4e-6 per bp per duplication) — with the first cycle worth ~5x an ordinary one
|
|
296
296
|
([Shagin *et al.* 2017](https://doi.org/10.1038/s41598-017-02727-8)).
|
|
297
297
|
|
|
298
|
-
Which is exactly what comes out
|
|
299
|
-
|
|
300
|
-
|
|
301
|
-
<p align="center"><img alt="consensus quality against depth" src="assets/consensus_quality.svg" width="80%"></p>
|
|
298
|
+
Which is exactly what comes out: one read carries the payload's own error and the curve flattens
|
|
299
|
+
at the floor rather than at the instrument — drawn as a box per depth bin, [further down](#it-draws-its-own-qc).
|
|
302
300
|
|
|
303
301
|
Very deep barcodes are capped at **10,000 reads into the consensus** — past that the column
|
|
304
302
|
posterior has long since saturated while the group still costs time and memory, which is 10x's
|
|
@@ -339,13 +337,71 @@ migec plot out/ # every panel whose table is in out/
|
|
|
339
337
|
migec plot cons/ -o figs/ --format pdf
|
|
340
338
|
```
|
|
341
339
|
|
|
342
|
-
|
|
343
|
-
|
|
344
|
-
|
|
345
|
-
|
|
346
|
-
|
|
347
|
-
|
|
348
|
-
|
|
340
|
+
Twenty panels. Every one is a gnuplot script over a TSV a stage already wrote, so a figure can be
|
|
341
|
+
redrawn from the table next to it long after the FASTQ is gone, and a figure can never disagree
|
|
342
|
+
with the number in the report. gnuplot is not a Python dependency: without it the `.gp` scripts are
|
|
343
|
+
still written. Every SVG is **transparent and mid-grey**, so one file serves a light README, a dark
|
|
344
|
+
README and a printed page — and the legend sits inside the plot box, not in a gutter that makes
|
|
345
|
+
every figure wider than its data. See [docs/plots.rst](docs/plots.rst).
|
|
346
|
+
|
|
347
|
+
Four of them are the ones you already know how to read.
|
|
348
|
+
|
|
349
|
+
**The barcode rank plot**, on Cell Ranger's axes, because that is the figure every user of a
|
|
350
|
+
droplet protocol has seen. Barcodes sorted by how many **distinct UMIs** they carry — never by
|
|
351
|
+
reads, since one over-amplified molecule would otherwise put an empty droplet high on the curve,
|
|
352
|
+
which is the artefact the plot exists to show. The call is drawn *on* the curve.
|
|
353
|
+
|
|
354
|
+
<p align="center"><img alt="barcode rank plot" src="assets/cell_rank.svg" width="72%"></p>
|
|
355
|
+
|
|
356
|
+
**The MIG size spectrum**, molecules and the reads they account for, on `log(1 + size)`. Both
|
|
357
|
+
series, on their own axes, because they peak in different places the moment a library is
|
|
358
|
+
over-sequenced: most *molecules* are shallow, most *reads* are in the deep ones. A figure with only
|
|
359
|
+
the first says the library is fine; a figure with only the second says it is saturated. `log1p`, so
|
|
360
|
+
a molecule seen once still has a place on the axis.
|
|
361
|
+
|
|
362
|
+
<p align="center"><img alt="MIG size spectrum" src="assets/mig_size_spectrum.svg" width="72%"></p>
|
|
363
|
+
|
|
364
|
+
**The rank/Zipf curve** — molecule size against rank, log-log. A straight line is Zipf, and
|
|
365
|
+
amplification bias bends it. This is why `refine` writes the size spectrum at *exact* sizes rather
|
|
366
|
+
than in power-of-two bins: four bins make four steps, and a straight line cannot be told from a
|
|
367
|
+
bent one.
|
|
368
|
+
|
|
369
|
+
<p align="center"><img alt="molecule size against rank" src="assets/mig_size_zipf.svg" width="72%"></p>
|
|
370
|
+
|
|
371
|
+
**Barcode errors against the parent's depth** — how many distinct error children a molecule
|
|
372
|
+
spawned, and how many reads were in them. A parent seen *c* times had *c·L* barcode bases to
|
|
373
|
+
miscall, so both series climb with *c* — but only one of them can climb forever. There are exactly
|
|
374
|
+
*3L* barcodes one substitution away, so the distinct-children series bends over at the dashed line
|
|
375
|
+
while the reads series carries straight past it. Where they part is where this library's barcode
|
|
376
|
+
neighbourhood filled up, measured rather than predicted.
|
|
377
|
+
|
|
378
|
+
<p align="center"><img alt="barcode errors against depth" src="assets/umi_error_children.svg" width="72%"></p>
|
|
379
|
+
|
|
380
|
+
Invert both and you get two estimates of the same barcode error rate, per depth, with what `refine`
|
|
381
|
+
reports drawn across them. The y axis is a log error rate, so one decade is exactly ten Phred and
|
|
382
|
+
1e-3 is Q30 — which makes the number directly comparable with the barcode's own reported quality,
|
|
383
|
+
the only independent check on it there is. On a diverse library sequenced 25 deep it lands within
|
|
384
|
+
1% of a known injected rate.
|
|
385
|
+
|
|
386
|
+
<p align="center"><img alt="barcode error rate by depth" src="assets/umi_error_rate.svg" width="72%"></p>
|
|
387
|
+
|
|
388
|
+
> **Never** read this instead of the `saturated` flag. Both series count only the children
|
|
389
|
+
> correction actually merged, and on a *full* barcode space `refine` refuses to merge — rightly,
|
|
390
|
+
> because a distance-1 neighbour there is more likely a real molecule than an error child — so both
|
|
391
|
+
> estimates collapse to zero. Against a known rate they are 0.99 and 0.97 of the truth at 0.2%
|
|
392
|
+
> occupancy, 0.62 and 0.45 at 33%, and nothing at 100%. See
|
|
393
|
+
> [docs/umi_errors.rst](docs/umi_errors.rst).
|
|
394
|
+
|
|
395
|
+
**Consensus quality against depth**, as a box, never a thinned scatter. Emitted quality is discrete
|
|
396
|
+
and capped at the floor, so at any real depth every molecule sits on one or two integers: a cloud
|
|
397
|
+
of dots draws that as a flat line whether the bin holds ten molecules or ten million, and thinning
|
|
398
|
+
it throws away the tails that were the only thing the cloud could have shown. These are exact order
|
|
399
|
+
statistics over every molecule, read off a (depth, quality) count grid.
|
|
400
|
+
|
|
401
|
+
<p align="center"><img alt="consensus quality against depth" src="assets/consensus_quality.svg" width="72%"></p>
|
|
402
|
+
|
|
403
|
+
`checkout.summary.tsv` carries unique UMIs and reads per **sample** barcode, drawn the same way by
|
|
404
|
+
the `sample_umis` panel — the multiplexed analogue of the same question.
|
|
349
405
|
|
|
350
406
|
### It finds what the trim left behind
|
|
351
407
|
|
|
@@ -476,7 +532,7 @@ warning: 79.4% of molecules were seen once. A consensus over one read is that re
|
|
|
476
532
|
```
|
|
477
533
|
|
|
478
534
|
It is also the memory-hostile case, because distinct barcodes are what everything scales with, so
|
|
479
|
-
it is what the benchmarks use:
|
|
535
|
+
it is what the benchmarks use: 1,179,549 reads/s at 1.02 reads/UMI, 282 B resident per distinct
|
|
480
536
|
barcode, still bounded by the bucket rather than the library.
|
|
481
537
|
|
|
482
538
|
### Speed and memory are reported, not assumed
|
|
@@ -489,11 +545,12 @@ and nothing else, which is asserted per stage in C++, at the CLI, and under the
|
|
|
489
545
|
|
|
490
546
|
| stage | 1 thread | 16 threads | bound by |
|
|
491
547
|
|---|---|---|---|
|
|
492
|
-
| `checkout` |
|
|
493
|
-
| `refine` |
|
|
494
|
-
| `assemble` |
|
|
548
|
+
| `checkout` | 213,880 | **1,548,835** | reads |
|
|
549
|
+
| `refine` | 617,802 | **1,554,156** | distinct barcodes |
|
|
550
|
+
| `assemble` | 554,106 | **2,470,928** | reads, then the largest bucket |
|
|
495
551
|
|
|
496
|
-
reads/s
|
|
552
|
+
reads/s — `refine` and `assemble` on a 500 k-read sample, `checkout` on 2 M. Two of those used to
|
|
553
|
+
be 222,017 and 202,977, and the first fix was not a thread: **zlib at
|
|
497
554
|
its default level 6 was 83% of refine's wall clock**, compressing an intermediate the next stage
|
|
498
555
|
decompresses immediately. Level 1 costs 21% more bytes and gave 3x before a single thread was
|
|
499
556
|
added. Measure the stage before parallelising it.
|
|
@@ -511,20 +568,34 @@ malloc instead of inflate.
|
|
|
511
568
|
| reads/s | 1,481,946 | **2,051,937** |
|
|
512
569
|
| peak RSS | 1,479 MB | **789 MB** |
|
|
513
570
|
|
|
571
|
+
That is the record of *that* change. Batching the work-claiming atomic later took the same corpus
|
|
572
|
+
to **1.72 s and 2,324,403 reads/s**.
|
|
573
|
+
|
|
514
574
|
The memory fell with it, because the estimate deciding how finely to cut the input said a gzipped
|
|
515
575
|
FASTQ goes resident at 8x its on-disk size and it is really **19x** — a resident record is two heap
|
|
516
576
|
`std::string` with their allocator headers, not the 180 bytes of payload. Guessing low picks too
|
|
517
577
|
few buckets, and pass 2 holds sixteen of them at once.
|
|
518
578
|
|
|
579
|
+
And once every stage was threaded, the largest single cost in the whole pipeline was the
|
|
580
|
+
**thread helper itself**. `parallel_for` handed out one item per atomic `fetch_add`, and when an
|
|
581
|
+
item is one read's tag scan that instruction took **21% of all CPU samples across every thread** —
|
|
582
|
+
more than the work it was handing out. Sixteen cores serialise on one cache line. Items are claimed
|
|
583
|
+
in batches now, sized so each worker takes ~8 turns, collapsing to 1 when there are few items,
|
|
584
|
+
which is the uneven case (one bucket per item) the counter existed for. Two serial blocks went with
|
|
585
|
+
it: the distance-1 census inside `estimate_umi_error`, which is checkout's per-sample tail, and
|
|
586
|
+
refine's residual-FDR scan, which was 0.53 s of a 2.17 s run on one core after everything around it
|
|
587
|
+
had been parallelised. Both are read-only scans of the barcode table that tally integers, so they
|
|
588
|
+
thread exactly and `-t` still changes nothing but the clock.
|
|
589
|
+
|
|
519
590
|
| threads | reads/s | matching reads/s | peak RSS |
|
|
520
591
|
|---|---|---|---|
|
|
521
|
-
| 1 |
|
|
522
|
-
| 2 |
|
|
523
|
-
| 4 |
|
|
524
|
-
| 8 |
|
|
525
|
-
| 16 | **1,
|
|
592
|
+
| 1 | 213,880 | 216,584 | 59 MB |
|
|
593
|
+
| 2 | 394,471 | 403,393 | 79 MB |
|
|
594
|
+
| 4 | 737,777 | 768,810 | 101 MB |
|
|
595
|
+
| 8 | 1,256,838 | 1,349,533 | 139 MB |
|
|
596
|
+
| 16 | **1,548,835** | **1,697,313** | 215 MB |
|
|
526
597
|
|
|
527
|
-
<p align="center"><img alt="checkout thread scaling" src="assets/benchmark_threads.svg" width="
|
|
598
|
+
<p align="center"><img alt="checkout thread scaling" src="assets/benchmark_threads.svg" width="72%"></p>
|
|
528
599
|
|
|
529
600
|
2 M single-end 129 nt reads, four barcode patterns, 4 reads per molecule, M-series laptop.
|
|
530
601
|
`python scripts/benchmark_threads.py --reads 2000000 -o assets/` writes that table, and the figure
|