migec 2.1.0__tar.gz → 2.2.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Potentially problematic release.


This version of migec might be problematic. Click here for more details.

Files changed (189) hide show
  1. {migec-2.1.0 → migec-2.2.0}/CHANGELOG.md +101 -0
  2. {migec-2.1.0 → migec-2.2.0}/CLAUDE.md +64 -4
  3. {migec-2.1.0 → migec-2.2.0}/PKG-INFO +53 -14
  4. {migec-2.1.0 → migec-2.2.0}/README.md +52 -13
  5. {migec-2.1.0 → migec-2.2.0}/ROADMAP.md +27 -2
  6. {migec-2.1.0 → migec-2.2.0}/SOURCES.md +3 -3
  7. migec-2.2.0/assets/PBMC.umi_errors.tsv +160 -0
  8. migec-2.2.0/assets/benchmark_threads.json +47 -0
  9. {migec-2.1.0 → migec-2.2.0}/assets/benchmark_threads.svg +15 -15
  10. migec-2.2.0/assets/benchmark_threads.tsv +6 -0
  11. {migec-2.1.0 → migec-2.2.0}/assets/mig_size_spectrum.gp +9 -1
  12. {migec-2.1.0 → migec-2.2.0}/assets/mig_size_spectrum.svg +163 -25
  13. migec-2.2.0/assets/umi_error_children.gp +32 -0
  14. migec-2.2.0/assets/umi_error_children.svg +578 -0
  15. migec-2.2.0/assets/umi_error_rate.gp +47 -0
  16. migec-2.2.0/assets/umi_error_rate.svg +532 -0
  17. {migec-2.1.0 → migec-2.2.0}/docs/assemble.rst +2 -2
  18. {migec-2.1.0 → migec-2.2.0}/docs/commands.rst +1 -1
  19. {migec-2.1.0 → migec-2.2.0}/docs/examples.rst +3 -0
  20. {migec-2.1.0 → migec-2.2.0}/docs/method.rst +3 -0
  21. {migec-2.1.0 → migec-2.2.0}/docs/nextflow.rst +3 -3
  22. {migec-2.1.0 → migec-2.2.0}/docs/performance.rst +82 -45
  23. {migec-2.1.0 → migec-2.2.0}/docs/plots.rst +10 -0
  24. {migec-2.1.0 → migec-2.2.0}/docs/refine.rst +9 -0
  25. migec-2.2.0/docs/umi_errors.rst +167 -0
  26. {migec-2.1.0 → migec-2.2.0}/include/migec/parallel.hpp +26 -10
  27. {migec-2.1.0 → migec-2.2.0}/include/migec/refine.hpp +12 -0
  28. {migec-2.1.0 → migec-2.2.0}/include/migec/umi_stats.hpp +5 -1
  29. {migec-2.1.0 → migec-2.2.0}/include/migec/version.hpp +1 -1
  30. {migec-2.1.0 → migec-2.2.0}/project/README.md +9 -0
  31. {migec-2.1.0 → migec-2.2.0}/pyproject.toml +1 -1
  32. {migec-2.1.0 → migec-2.2.0}/python/migec/__init__.py +1 -1
  33. {migec-2.1.0 → migec-2.2.0}/python/migec/checkout.py +5 -3
  34. {migec-2.1.0 → migec-2.2.0}/python/migec/cli.py +4 -3
  35. {migec-2.1.0 → migec-2.2.0}/python/migec/plot.py +76 -1
  36. {migec-2.1.0 → migec-2.2.0}/python/migec/refine.py +8 -0
  37. {migec-2.1.0 → migec-2.2.0}/scripts/example_figures.py +1 -1
  38. {migec-2.1.0 → migec-2.2.0}/skills/migec/SKILL.md +33 -2
  39. {migec-2.1.0 → migec-2.2.0}/src/_bindings.cpp +4 -0
  40. {migec-2.1.0 → migec-2.2.0}/src/assemble.cpp +19 -16
  41. {migec-2.1.0 → migec-2.2.0}/src/refine.cpp +169 -18
  42. {migec-2.1.0 → migec-2.2.0}/src/umi_stats.cpp +14 -7
  43. {migec-2.1.0 → migec-2.2.0}/tests/synthetic/test_refine.py +2 -0
  44. migec-2.2.0/tests/synthetic/test_umi_errors.py +135 -0
  45. migec-2.1.0/assets/benchmark_threads.json +0 -47
  46. migec-2.1.0/assets/benchmark_threads.tsv +0 -6
  47. {migec-2.1.0 → migec-2.2.0}/.github/workflows/ci.yml +0 -0
  48. {migec-2.1.0 → migec-2.2.0}/.github/workflows/docs.yml +0 -0
  49. {migec-2.1.0 → migec-2.2.0}/.github/workflows/publish.yml +0 -0
  50. {migec-2.1.0 → migec-2.2.0}/.github/workflows/testpypi.yml +0 -0
  51. {migec-2.1.0 → migec-2.2.0}/.gitignore +0 -0
  52. {migec-2.1.0 → migec-2.2.0}/CMakeLists.txt +0 -0
  53. {migec-2.1.0 → migec-2.2.0}/LICENSE +0 -0
  54. {migec-2.1.0 → migec-2.2.0}/assets/PBMC.cell_rank.tsv +0 -0
  55. {migec-2.1.0 → migec-2.2.0}/assets/PBMC.sizes.tsv +0 -0
  56. {migec-2.1.0 → migec-2.2.0}/assets/SRR1763769.mig.tsv +0 -0
  57. {migec-2.1.0 → migec-2.2.0}/assets/assemble.coverage.tsv +0 -0
  58. {migec-2.1.0 → migec-2.2.0}/assets/assemble.quality_by_depth.tsv +0 -0
  59. {migec-2.1.0 → migec-2.2.0}/assets/assemble_coverage.gp +0 -0
  60. {migec-2.1.0 → migec-2.2.0}/assets/assemble_coverage.svg +0 -0
  61. {migec-2.1.0 → migec-2.2.0}/assets/benchmark_threads.gp +0 -0
  62. {migec-2.1.0 → migec-2.2.0}/assets/cell_rank.gp +0 -0
  63. {migec-2.1.0 → migec-2.2.0}/assets/cell_rank.svg +0 -0
  64. {migec-2.1.0 → migec-2.2.0}/assets/consensus_error.gp +0 -0
  65. {migec-2.1.0 → migec-2.2.0}/assets/consensus_error.svg +0 -0
  66. {migec-2.1.0 → migec-2.2.0}/assets/consensus_layout.gp +0 -0
  67. {migec-2.1.0 → migec-2.2.0}/assets/consensus_layout.svg +0 -0
  68. {migec-2.1.0 → migec-2.2.0}/assets/consensus_quality.gp +0 -0
  69. {migec-2.1.0 → migec-2.2.0}/assets/consensus_quality.svg +0 -0
  70. {migec-2.1.0 → migec-2.2.0}/assets/mig_size_zipf.gp +0 -0
  71. {migec-2.1.0 → migec-2.2.0}/assets/mig_size_zipf.svg +0 -0
  72. {migec-2.1.0 → migec-2.2.0}/assets/pipeline.dot +0 -0
  73. {migec-2.1.0 → migec-2.2.0}/assets/pipeline.svg +0 -0
  74. {migec-2.1.0 → migec-2.2.0}/docs/Makefile +0 -0
  75. {migec-2.1.0 → migec-2.2.0}/docs/barcode_space.rst +0 -0
  76. {migec-2.1.0 → migec-2.2.0}/docs/checkout.rst +0 -0
  77. {migec-2.1.0 → migec-2.2.0}/docs/conf.py +0 -0
  78. {migec-2.1.0 → migec-2.2.0}/docs/downstream.rst +0 -0
  79. {migec-2.1.0 → migec-2.2.0}/docs/formats.rst +0 -0
  80. {migec-2.1.0 → migec-2.2.0}/docs/fragmented.rst +0 -0
  81. {migec-2.1.0 → migec-2.2.0}/docs/grouping.rst +0 -0
  82. {migec-2.1.0 → migec-2.2.0}/docs/index.rst +0 -0
  83. {migec-2.1.0 → migec-2.2.0}/docs/installation.rst +0 -0
  84. {migec-2.1.0 → migec-2.2.0}/docs/layouts.rst +0 -0
  85. {migec-2.1.0 → migec-2.2.0}/docs/nulls.rst +0 -0
  86. {migec-2.1.0 → migec-2.2.0}/docs/quality_floor.rst +0 -0
  87. {migec-2.1.0 → migec-2.2.0}/docs/reference.rst +0 -0
  88. {migec-2.1.0 → migec-2.2.0}/docs/roadmap.rst +0 -0
  89. {migec-2.1.0 → migec-2.2.0}/docs/subsample.rst +0 -0
  90. {migec-2.1.0 → migec-2.2.0}/docs/suggest.rst +0 -0
  91. {migec-2.1.0 → migec-2.2.0}/docs/umi_statistics.rst +0 -0
  92. {migec-2.1.0 → migec-2.2.0}/docs/validation.rst +0 -0
  93. {migec-2.1.0 → migec-2.2.0}/include/migec/assemble.hpp +0 -0
  94. {migec-2.1.0 → migec-2.2.0}/include/migec/checkout.hpp +0 -0
  95. {migec-2.1.0 → migec-2.2.0}/include/migec/consensus.hpp +0 -0
  96. {migec-2.1.0 → migec-2.2.0}/include/migec/fastq.hpp +0 -0
  97. {migec-2.1.0 → migec-2.2.0}/include/migec/mig_record.hpp +0 -0
  98. {migec-2.1.0 → migec-2.2.0}/include/migec/pattern.hpp +0 -0
  99. {migec-2.1.0 → migec-2.2.0}/include/migec/resource.hpp +0 -0
  100. {migec-2.1.0 → migec-2.2.0}/include/migec/subsample.hpp +0 -0
  101. {migec-2.1.0 → migec-2.2.0}/include/migec/suggest.hpp +0 -0
  102. {migec-2.1.0 → migec-2.2.0}/include/migec/types.hpp +0 -0
  103. {migec-2.1.0 → migec-2.2.0}/include/migec/whitelist.hpp +0 -0
  104. {migec-2.1.0 → migec-2.2.0}/integrations/nextflow/README.md +0 -0
  105. {migec-2.1.0 → migec-2.2.0}/integrations/nextflow/modules/migec/assemble/main.nf +0 -0
  106. {migec-2.1.0 → migec-2.2.0}/integrations/nextflow/modules/migec/assemble/meta.yml +0 -0
  107. {migec-2.1.0 → migec-2.2.0}/integrations/nextflow/modules/migec/checkout/main.nf +0 -0
  108. {migec-2.1.0 → migec-2.2.0}/integrations/nextflow/modules/migec/checkout/meta.yml +0 -0
  109. {migec-2.1.0 → migec-2.2.0}/integrations/nextflow/modules/migec/environment.yml +0 -0
  110. {migec-2.1.0 → migec-2.2.0}/integrations/nextflow/modules/migec/plot/main.nf +0 -0
  111. {migec-2.1.0 → migec-2.2.0}/integrations/nextflow/modules/migec/refine/main.nf +0 -0
  112. {migec-2.1.0 → migec-2.2.0}/integrations/nextflow/modules/migec/refine/meta.yml +0 -0
  113. {migec-2.1.0 → migec-2.2.0}/integrations/nextflow/nextflow.config +0 -0
  114. {migec-2.1.0 → migec-2.2.0}/integrations/nextflow/subworkflows/migec/main.nf +0 -0
  115. {migec-2.1.0 → migec-2.2.0}/notebooks/barcode_space.py +0 -0
  116. {migec-2.1.0 → migec-2.2.0}/notebooks/platforms.py +0 -0
  117. {migec-2.1.0 → migec-2.2.0}/notebooks/refine_diagnostics.py +0 -0
  118. {migec-2.1.0 → migec-2.2.0}/project/design-assemble.md +0 -0
  119. {migec-2.1.0 → migec-2.2.0}/project/design-benchmarks.md +0 -0
  120. {migec-2.1.0 → migec-2.2.0}/project/design-checkout.md +0 -0
  121. {migec-2.1.0 → migec-2.2.0}/project/design-io-interop.md +0 -0
  122. {migec-2.1.0 → migec-2.2.0}/project/design-refine.md +0 -0
  123. {migec-2.1.0 → migec-2.2.0}/project/design-repo.md +0 -0
  124. {migec-2.1.0 → migec-2.2.0}/project/review-algorithms.md +0 -0
  125. {migec-2.1.0 → migec-2.2.0}/project/review-scope.md +0 -0
  126. {migec-2.1.0 → migec-2.2.0}/python/migec/assemble.py +0 -0
  127. {migec-2.1.0 → migec-2.2.0}/python/migec/sheet.py +0 -0
  128. {migec-2.1.0 → migec-2.2.0}/python/migec/subsample.py +0 -0
  129. {migec-2.1.0 → migec-2.2.0}/python/migec/suggest.py +0 -0
  130. {migec-2.1.0 → migec-2.2.0}/scripts/benchmark_threads.py +0 -0
  131. {migec-2.1.0 → migec-2.2.0}/scripts/collision_check.py +0 -0
  132. {migec-2.1.0 → migec-2.2.0}/scripts/compare_calib.py +0 -0
  133. {migec-2.1.0 → migec-2.2.0}/scripts/correction_accuracy.py +0 -0
  134. {migec-2.1.0 → migec-2.2.0}/scripts/diagnose.py +0 -0
  135. {migec-2.1.0 → migec-2.2.0}/scripts/permutation_nulls.py +0 -0
  136. {migec-2.1.0 → migec-2.2.0}/scripts/quality_floor.py +0 -0
  137. {migec-2.1.0 → migec-2.2.0}/scripts/read_start_dispersion.py +0 -0
  138. {migec-2.1.0 → migec-2.2.0}/scripts/spikein_ratio.py +0 -0
  139. {migec-2.1.0 → migec-2.2.0}/setup.sh +0 -0
  140. {migec-2.1.0 → migec-2.2.0}/src/checkout.cpp +0 -0
  141. {migec-2.1.0 → migec-2.2.0}/src/consensus.cpp +0 -0
  142. {migec-2.1.0 → migec-2.2.0}/src/fastq.cpp +0 -0
  143. {migec-2.1.0 → migec-2.2.0}/src/mig_record.cpp +0 -0
  144. {migec-2.1.0 → migec-2.2.0}/src/pattern.cpp +0 -0
  145. {migec-2.1.0 → migec-2.2.0}/src/resource.cpp +0 -0
  146. {migec-2.1.0 → migec-2.2.0}/src/subsample.cpp +0 -0
  147. {migec-2.1.0 → migec-2.2.0}/src/suggest.cpp +0 -0
  148. {migec-2.1.0 → migec-2.2.0}/src/types.cpp +0 -0
  149. {migec-2.1.0 → migec-2.2.0}/src/whitelist.cpp +0 -0
  150. {migec-2.1.0 → migec-2.2.0}/tests/__init__.py +0 -0
  151. {migec-2.1.0 → migec-2.2.0}/tests/benchmark/__init__.py +0 -0
  152. {migec-2.1.0 → migec-2.2.0}/tests/benchmark/test_assemble_speed.py +0 -0
  153. {migec-2.1.0 → migec-2.2.0}/tests/benchmark/test_checkout_speed.py +0 -0
  154. {migec-2.1.0 → migec-2.2.0}/tests/benchmark/test_refine_speed.py +0 -0
  155. {migec-2.1.0 → migec-2.2.0}/tests/conftest.py +0 -0
  156. {migec-2.1.0 → migec-2.2.0}/tests/cpp/doctest.h +0 -0
  157. {migec-2.1.0 → migec-2.2.0}/tests/cpp/test_checkout.cpp +0 -0
  158. {migec-2.1.0 → migec-2.2.0}/tests/cpp/test_checkout_run.cpp +0 -0
  159. {migec-2.1.0 → migec-2.2.0}/tests/cpp/test_consensus.cpp +0 -0
  160. {migec-2.1.0 → migec-2.2.0}/tests/cpp/test_fastq.cpp +0 -0
  161. {migec-2.1.0 → migec-2.2.0}/tests/cpp/test_mig_record.cpp +0 -0
  162. {migec-2.1.0 → migec-2.2.0}/tests/cpp/test_parallel_stages.cpp +0 -0
  163. {migec-2.1.0 → migec-2.2.0}/tests/cpp/test_pattern.cpp +0 -0
  164. {migec-2.1.0 → migec-2.2.0}/tests/cpp/test_types.cpp +0 -0
  165. {migec-2.1.0 → migec-2.2.0}/tests/cpp/test_umi_stats.cpp +0 -0
  166. {migec-2.1.0 → migec-2.2.0}/tests/cpp/test_whitelist.cpp +0 -0
  167. {migec-2.1.0 → migec-2.2.0}/tests/data/migec_spikein_truth.tsv +0 -0
  168. {migec-2.1.0 → migec-2.2.0}/tests/realworld/__init__.py +0 -0
  169. {migec-2.1.0 → migec-2.2.0}/tests/synthetic/__init__.py +0 -0
  170. {migec-2.1.0 → migec-2.2.0}/tests/synthetic/_sim.py +0 -0
  171. {migec-2.1.0 → migec-2.2.0}/tests/synthetic/test_assemble.py +0 -0
  172. {migec-2.1.0 → migec-2.2.0}/tests/synthetic/test_barcode_space.py +0 -0
  173. {migec-2.1.0 → migec-2.2.0}/tests/synthetic/test_checkout_end_to_end.py +0 -0
  174. {migec-2.1.0 → migec-2.2.0}/tests/synthetic/test_dual_end.py +0 -0
  175. {migec-2.1.0 → migec-2.2.0}/tests/synthetic/test_grouping_accuracy.py +0 -0
  176. {migec-2.1.0 → migec-2.2.0}/tests/synthetic/test_nulls.py +0 -0
  177. {migec-2.1.0 → migec-2.2.0}/tests/synthetic/test_quality_floor.py +0 -0
  178. {migec-2.1.0 → migec-2.2.0}/tests/synthetic/test_sim.py +0 -0
  179. {migec-2.1.0 → migec-2.2.0}/tests/synthetic/test_subsample.py +0 -0
  180. {migec-2.1.0 → migec-2.2.0}/tests/synthetic/test_suggest.py +0 -0
  181. {migec-2.1.0 → migec-2.2.0}/tests/synthetic/test_thread_invariance.py +0 -0
  182. {migec-2.1.0 → migec-2.2.0}/tests/unit/__init__.py +0 -0
  183. {migec-2.1.0 → migec-2.2.0}/tests/unit/test_bindings.py +0 -0
  184. {migec-2.1.0 → migec-2.2.0}/tests/unit/test_compare_calib.py +0 -0
  185. {migec-2.1.0 → migec-2.2.0}/tests/unit/test_core.py +0 -0
  186. {migec-2.1.0 → migec-2.2.0}/tests/unit/test_downstream.py +0 -0
  187. {migec-2.1.0 → migec-2.2.0}/tests/unit/test_plot.py +0 -0
  188. {migec-2.1.0 → migec-2.2.0}/tests/unit/test_sheet_and_cli.py +0 -0
  189. {migec-2.1.0 → migec-2.2.0}/tests/unit/test_spikein_ratio.py +0 -0
@@ -4,6 +4,107 @@ Hand-written and prose-heavy: each entry says what changed and, where it matters
4
4
  prevents. Releases before 2.0.0 are the Groovy MIGEC and are described by their git tags on the
5
5
  `legacy-v1` branch.
6
6
 
7
+ ## 2.2.0 — 2026-08-13
8
+
9
+ Throughput and one new measurement. No breaking change: every 2.1.0 command, flag and output file
10
+ behaves as it did, and the additions are a new table, four new JSON fields and two new plot panels.
11
+
12
+ ### The barcode error rate, measured at every depth
13
+
14
+ `refine` writes `<sample>.umi_errors.tsv`: one row per exact parent depth, carrying the distinct
15
+ error children that parent spawned, the reads in them, and the error rate each implies. A parent
16
+ carrying `c` reads offered `c*L` barcode bases to be miscalled, so the same eps falls out two ways:
17
+
18
+ distinct children u(c) = 3L (1 - exp(-c eps / 3)) -> eps = -(3/c) ln(1 - u/3L)
19
+ reads in children r(c) = c L eps -> eps = r / (c L)
20
+
21
+ A barcode has only `3L` neighbours one substitution away, so the first **saturates**; reads have no
22
+ ceiling, so the second does not. Where the two part company on the figure is where this library's
23
+ barcode neighbourhood filled up, read off the data instead of predicted. Past saturation the first
24
+ is left blank rather than reported as a small number: inverting a full neighbourhood returns "no
25
+ errors" for the most error-ridden library there can be.
26
+
27
+ New report lines and JSON fields `error_at_depth`, `error_phred`, `error_from_children`,
28
+ `error_depth`. The number to quote is `error_at_depth` — restricted to parents seen >= 10 times,
29
+ where correction is near-complete — and `error_phred` is the same thing as a Phred so it can be put
30
+ beside the barcode's own reported Q. On a diverse library sequenced 25 deep with 1e-3 injected, the
31
+ distance-1 excess gives 9.73e-04, the children give 9.89e-04 (Q30) and all depths give 9.98e-04.
32
+ Three routes to one number agreeing within 3% is what makes any of them believable.
33
+
34
+ Never: **both estimators are bounded by the merges correction actually made**, so neither is
35
+ saturation-free. Against a known injected rate, as a fraction of truth:
36
+
37
+ | occupancy | 0.2% | 2.3% | 9.8% | 33% | 100% |
38
+ |---|---|---|---|---|---|
39
+ | distance-1 excess | 0.97 | 0.96 | 0.76 | 0.45 | 0.001 |
40
+ | from the children | **0.99** | **0.95** | **0.88** | **0.62** | 0.00 |
41
+
42
+ The children estimate is the better of the two wherever either works, and at 100% they both go to
43
+ zero for the same reason: on a full barcode space `correct_umis` refuses to merge, correctly,
44
+ because a distance-1 neighbour there is more likely a real molecule than a child. The `saturated`
45
+ flag is what says the answer is a floor — read it, do not read this table instead of it. And read
46
+ the table at depth: a child whose parent was never sequenced cannot be counted, which at 1-3
47
+ reads/UMI is 80% of them. `tests/synthetic/test_umi_errors.py` holds all of it, `docs/umi_errors.rst`
48
+ works through it.
49
+
50
+ Two new `migec plot` panels over that table, `umi_error_children` and `umi_error_rate`, with the
51
+ `3L` ceiling and the reported estimate drawn as reference lines. Note: `neighbours` and `estimate`
52
+ are constant down their columns on purpose — the panels draw them, and a figure that needs a value
53
+ its own table does not carry will one day disagree with the report.
54
+
55
+ ### `reads in them` is points, never a line
56
+
57
+ The `mig_size_spectrum` panel drew its reads series `with lines` over a table that has one row per
58
+ **exact** size. Past the head of the distribution almost every size holds exactly one molecule, so
59
+ reads == size, and the line rendered the `y = x` diagonal as the most prominent feature of the
60
+ figure — a tautology drawn as a second mode. Where two or three molecules shared a size the same
61
+ line sawtoothed between `size*1` and `size*2`, which is integer quantisation drawn as signal, and it
62
+ bridged gaps in the support where no size was observed at all. It is points now, and the new
63
+ barcode-error panels are points for the same reason.
64
+
65
+ ### The thread helper was the bottleneck
66
+
67
+ With all three stages threaded, a sampling profile of
68
+ `assemble` put **21% of all CPU samples, across every thread, on a single instruction**: the atomic
69
+ `fetch_add` in `parallel_for` that handed out one item at a time. When an item is one read's tag
70
+ scan, sixteen cores serialise on one cache line and the counter costs more than the work it
71
+ distributes. Items are claimed in batches now, sized so each worker takes ~8 turns and capped so a
72
+ ten-million-item scan still hands out ten thousand batches. Never: the batch collapses to 1 when
73
+ there are few items, which is the uneven case -- one bucket per item, one holding ten molecules and
74
+ the next ten million -- that the counter exists for.
75
+
76
+ Two serial blocks fell out with it, both read-only `3L`-binary-search scans of the barcode table:
77
+ the **distance-1 census** in `estimate_umi_error` (checkout's per-sample statistics tail) and
78
+ refine's **residual-FDR scan**, which was 0.53 s of a 2.17 s run on one core after everything around
79
+ it had been parallelised. Both tally integers into per-worker counters summed afterwards, so `-t`
80
+ still changes nothing but the clock -- verified byte-for-byte at `-t` 1/3/16 on a clean library and
81
+ on one with injected barcode errors (residual 1,294 either way), and under the thread sanitizer
82
+ (104 cases, 224,116 assertions, no race).
83
+
84
+ Two allocation fixes on paths every read takes: refine's rewrite chunk is now **assigned into
85
+ rather than cleared** -- the same trade assemble's partition already made, since `clear()` destroys
86
+ four `std::string` per record -- and the rewrite's comment buffer and unpacked barcode are worker
87
+ scratch reused read after read, with the cell and UMI halves taken as `string_view` instead of
88
+ `substr`. That is four allocations a read removed; `rewrite_seconds` fell 0.51 s to 0.38 s.
89
+ `assemble` also unpacked each group's UMI twice, once for the composition tally and once for the
90
+ record name.
91
+
92
+ | stage | 1 thread | 16 threads | was, at 16 |
93
+ |---|---|---|---|
94
+ | `checkout` | 213,880 | **1,548,835** | 1,056,472 |
95
+ | `refine` | 617,802 | **1,554,156** | 1,012,368 |
96
+ | `assemble` | 554,106 | **2,470,928** | 2,051,937 |
97
+
98
+ reads/s, 500 k-read sample except checkout at 2 M; `assets/benchmark_threads.tsv` is regenerated
99
+ and the figure redrawn from it. checkout's gap between end-to-end and matching throughput fell from
100
+ 20% to 9%.
101
+
102
+ Note: re-measured while it was in hand, a **64 k chunk in assemble's partition is still 32% faster**
103
+ (3,075,506 reads/s against 2,324,403 on 4 M reads) and is still not taken: it costs 16 MB of
104
+ resident chunk, which at NovaSeq scale makes pass 1 the memory peak and breaks the property that a
105
+ finer partition costs less rather than more. The benchmark tier does not fail at 64 k on a 500 k
106
+ corpus -- the objection is one of scale -- so the number is written down next to the constant.
107
+
7
108
  ## 2.1.0 — 2026-08-13
8
109
 
9
110
  First non-alpha of the rewrite. The three stages, the eight commands and the on-disk formats have
@@ -261,6 +261,26 @@ correction is written up in `project/review-algorithms.md`.
261
261
  is where the answer matters most. It caught me twice: once in the correction posterior, once in
262
262
  the residual-FDR estimator. Both now use payload agreement as well. On a 1.23 reads/barcode
263
263
  library the count-only residual is 0 and the full one is 1,294.
264
+ - **The barcode error rate is measured at every DEPTH, not once for the library.**
265
+ `<sample>.umi_errors.tsv`, one row per exact parent depth: a parent carrying `c` reads offered
266
+ `c*L` barcode bases to be miscalled, so `u(c) = 3L(1-exp(-c eps/3))` from the distinct children
267
+ and `r(c) = c L eps` from the reads in them invert to the same eps. Reported as
268
+ `error_at_depth` (parents seen >= 10 times, where correction is near-complete), `error_phred`,
269
+ and `error_from_children` (all depths, a LOWER bound). On 1e-3 injected: distance-1 9.73e-04,
270
+ children 9.89e-04 = Q30, all depths 9.98e-04.
271
+ - **Never: neither error estimator is saturation-free — both are bounded by the merges correction
272
+ MADE.** As a fraction of an injected truth: distance-1 0.97 / 0.96 / 0.76 / 0.45 / 0.001 and
273
+ children 0.99 / 0.95 / 0.88 / 0.62 / 0.00 at occupancy 0.2% / 2.3% / 9.8% / 33% / 100%. The
274
+ children estimate wins everywhere either works; at 100% BOTH are zero, because `correct_umis`
275
+ refuses to merge on a full space and is right to. `saturated` is the flag that says the answer
276
+ is a floor -- do not read the table instead of it. `docs/umi_errors.rst`,
277
+ `tests/synthetic/test_umi_errors.py`.
278
+ - **Never: a sparse spectrum is POINTS, never a line.** `mig_size_spectrum` drew "reads in them"
279
+ `with lines` over a table at EXACT sizes, where past the head almost every size holds one
280
+ molecule -- so reads == size and the line drew the `y = x` diagonal as the figure's most
281
+ prominent feature, a tautology that reads as a second mode. Two or three molecules on a size made
282
+ it sawtooth between `size*1` and `size*2`, and it bridged gaps where nothing was observed. Same
283
+ rule for the two `umi_error_*` panels.
264
284
  - **The MIG-size FDR threshold is reported, never applied** (2026-08-13). Measured residual, not
265
285
  derived. 5.25% of 1-read molecules at 1.23 reads/barcode, 0% at 4.62.
266
286
  - **Dual-end barcodes work (2026-08-13)**: column 3 of the sheet is the slave pattern, on the
@@ -354,11 +374,37 @@ correction is written up in `project/review-algorithms.md`.
354
374
  but the per-sample counters and the memory figure were one sample's. Fixed; the published table
355
375
  is re-measured and now comes from `scripts/benchmark_threads.py`, which writes the TSV the
356
376
  figure is drawn from.
357
- - **All three stages thread (2026-08-13, 2.0.0a3).** refine 222k -> 1.01M reads/s, assemble
358
- 203k -> 1.43M, checkout unchanged at 1.06M. Never: the FIRST fix was not a thread -- zlib level 6
377
+ - **All three stages thread (2026-08-13, 2.0.0a3-2.1.0).** refine 222k -> 1.01M reads/s, assemble
378
+ 203k -> 2.05M (its partition threaded in a4), checkout unchanged at 1.06M. Never: the FIRST fix was not a thread -- zlib level 6
359
379
  was 83% of refine's wall clock compressing an intermediate the next stage immediately
360
380
  decompresses. Level 1 is now the default for every stage's output and gave 3x on its own.
361
381
  Measure before parallelising.
382
+ - **Never: the thread HELPER was the bottleneck once everything else was threaded (unreleased).**
383
+ `parallel_for` claimed one item per atomic `fetch_add`, and a sampling profile of assemble put
384
+ **21% of all CPU samples across every thread on that one instruction** -- more than the parse it
385
+ was handing out. Sixteen cores serialise on one cache line. Batched now: ~8 turns per worker,
386
+ capped at 1024, collapsing to 1 when items are few, which is the uneven case (one bucket per
387
+ item) the counter exists for. Never: profile the helper, not only the work. Current numbers,
388
+ 1 thread -> 16: checkout **213,880 -> 1,548,835** end to end (1,697,313 matching), refine
389
+ **617,802 -> 1,554,156**, assemble **554,106 -> 2,470,928**; `assets/benchmark_threads.tsv` is
390
+ the committed table and the figure is drawn from it.
391
+ - **Two serial scans went with it**: the distance-1 census in `estimate_umi_error` (which IS
392
+ checkout's per-sample statistics tail -- the end-to-end/matching gap fell from 20% to 9%) and
393
+ refine's residual-FDR scan, 0.53 s of a 2.17 s run on one core. Both are read-only `3L` binary
394
+ searches per barcode tallying integers, so per-worker counters summed afterwards keep `-t`
395
+ changing nothing but the clock. `estimate_umi_error` now takes a `threads` argument, defaulted
396
+ to 1 so the C++ test calls are unchanged.
397
+ - **Note: a 64 k chunk in assemble's partition is STILL 32% faster and still not taken**
398
+ (3,075,506 reads/s against 2,324,403 on 4 M, re-measured after the batching). It costs 16 MB of
399
+ resident chunk, which at NovaSeq scale makes pass 1 the memory peak. Never: the benchmark tier
400
+ does NOT fail at 64 k on a 500 k corpus -- the objection is scale, which no test here can see,
401
+ so the number lives next to the constant in `assemble.cpp`. Upgrade path is a persistent worker
402
+ pool, not a bigger chunk.
403
+ - **Never: `clear()` on a chunk of `FastqOwned` is four allocations a record.** assemble's
404
+ partition knew this; refine's rewrite still did it. Assigned into now, and the rewrite's comment
405
+ buffer and unpacked barcode are worker scratch with the cell/UMI halves as `string_view` rather
406
+ than `substr` -- `rewrite_seconds` 0.51 -> 0.38. Same class of bug in `assemble`: each group's
407
+ UMI was unpacked twice, once for the composition tally and once for the record name.
362
408
  - **Never: `-t` changes nothing but the wall clock, on every stage.** refine parallelises the
363
409
  neighbourhood SCAN (a pure function of the barcode table -- it reads no union-find state) and
364
410
  applies the merges serially in the original order, so the result is identical, not equivalent.
@@ -407,8 +453,22 @@ correction is written up in `project/review-algorithms.md`.
407
453
  - **Never: `containsKey`, not `?:`, for a nextflow boolean.** Groovy's elvis treats `false` as
408
454
  absent, so a per-sample `contig: false` against `params.migec_contig = true` silently meant its
409
455
  opposite -- the one direction a per-sample override exists to make possible.
410
- - Next: M3's remainder (the template's own error split), then the M2 remainder
411
- (`.mig` bucket output from checkout, i7xi5, bit-parallel matcher), then `--rt-error auto`.
456
+ - **Next, in order. `ROADMAP.md` has the same list with the reasoning; this is the short form.**
457
+ 1. **`.mig` bucket output from `checkout`** — the only unbounded allocation left. The UMI
458
+ counters are 8.8 GB at NovaSeq scale held in one piece; checkout warns past 1 GB and a warning
459
+ is not a fix.
460
+ 2. **Bucketed correction in `refine`**, two passes with the key rotated. Same problem one stage
461
+ on, and the fix is understood rather than still to be designed.
462
+ 3. **The template's own error split** — the pattern's constant bases calibrate the PRIMER, not
463
+ the polymerase, so the RT/PCR separation needs a different standard.
464
+ 4. **`--rt-error auto`**, which falls out of 3 and not before it.
465
+ 5. **i7xi5 contingency table** — needs nothing that is not already in the headers.
466
+ 6. **`2026-migec-benchmark`** and the published comparisons. This is what the version number is
467
+ waiting on, not the code.
468
+ 7. **ctDNA ground truth** (COSMIC at 0.005-0.075 VAF). Must be built: Maruzani's deposited runs
469
+ carry no UMI.
470
+ 8. **R1/R2 overlap merge**, as a case of `--contig`'s placement and never a second matcher.
471
+ 9. **Bit-parallel matcher**, last and deliberately: the scan is not the bottleneck.
412
472
  - **Note: Britanova et al aging (bulk TCR, shallow) lives on aldan3** and is the real dataset for the
413
473
  1-3 read regime. Not pulled yet. aldan3 compute goes through SLURM, never the frontend.
414
474
  - The archive is pushed: `legacy-v1` + tag `v1-final`, and master is the rewrite. Recovery point
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.2
2
2
  Name: migec
3
- Version: 2.1.0
3
+ Version: 2.2.0
4
4
  Summary: UMI barcode extraction, correction and consensus assembly for barcoded sequencing
5
5
  Keywords: umi,barcode,consensus,airr,repseq,single-cell,sequencing
6
6
  Author-Email: Mikhail Shugay <mikhail.shugay@gmail.com>
@@ -197,8 +197,8 @@ reads 2,000,000
197
197
  unmatched 0 (0.0%)
198
198
  ambiguous 0 (0.0%)
199
199
 
200
- 2.2 s (903,599 reads/s) = 1.5 s matching on 8 threads + 0.7 s UMI statistics, serial
201
- peak RSS 131.0 MB of which UMI counters 21.2 MB
200
+ 1.6 s (1,243,801 reads/s) = 1.5 s matching on 8 threads + 0.1 s UMI statistics
201
+ peak RSS 136.0 MB of which UMI counters 11.5 MB
202
202
 
203
203
  sample reads UMIs reads/UMI UMI len eff len
204
204
  S1 500,000 125,000 4.00 12 12.00
@@ -265,7 +265,7 @@ migec assemble out/S1.fq.gz -o cons/ --contig # random-primed reads that til
265
265
  A molecule is **sample + cell barcode + UMI**, never the UMI alone — the same UMI in two cells is
266
266
  two molecules, and that is the design rather than a defect. Reads are range partitioned on the
267
267
  packed key into `.mig` buckets and one bucket is sorted at a time, so nothing scales with the
268
- library: 531,365 reads/s, and 121 MB at 16 buckets against 203 MB at one.
268
+ library: 2,423,777 reads/s, and 123 MB at 16 buckets against 203 MB at one.
269
269
 
270
270
  The per-column posterior is `LL[j][b] = Σ_i (r==b ? log(1−e) : log(e/3))`, and then the number that
271
271
  matters:
@@ -368,6 +368,30 @@ bent one.
368
368
 
369
369
  <p align="center"><img alt="molecule size against rank" src="assets/mig_size_zipf.svg" width="72%"></p>
370
370
 
371
+ **Barcode errors against the parent's depth** — how many distinct error children a molecule
372
+ spawned, and how many reads were in them. A parent seen *c* times had *c·L* barcode bases to
373
+ miscall, so both series climb with *c* — but only one of them can climb forever. There are exactly
374
+ *3L* barcodes one substitution away, so the distinct-children series bends over at the dashed line
375
+ while the reads series carries straight past it. Where they part is where this library's barcode
376
+ neighbourhood filled up, measured rather than predicted.
377
+
378
+ <p align="center"><img alt="barcode errors against depth" src="assets/umi_error_children.svg" width="72%"></p>
379
+
380
+ Invert both and you get two estimates of the same barcode error rate, per depth, with what `refine`
381
+ reports drawn across them. The y axis is a log error rate, so one decade is exactly ten Phred and
382
+ 1e-3 is Q30 — which makes the number directly comparable with the barcode's own reported quality,
383
+ the only independent check on it there is. On a diverse library sequenced 25 deep it lands within
384
+ 1% of a known injected rate.
385
+
386
+ <p align="center"><img alt="barcode error rate by depth" src="assets/umi_error_rate.svg" width="72%"></p>
387
+
388
+ > **Never** read this instead of the `saturated` flag. Both series count only the children
389
+ > correction actually merged, and on a *full* barcode space `refine` refuses to merge — rightly,
390
+ > because a distance-1 neighbour there is more likely a real molecule than an error child — so both
391
+ > estimates collapse to zero. Against a known rate they are 0.99 and 0.97 of the truth at 0.2%
392
+ > occupancy, 0.62 and 0.45 at 33%, and nothing at 100%. See
393
+ > [docs/umi_errors.rst](docs/umi_errors.rst).
394
+
371
395
  **Consensus quality against depth**, as a box, never a thinned scatter. Emitted quality is discrete
372
396
  and capped at the floor, so at any real depth every molecule sits on one or two integers: a cloud
373
397
  of dots draws that as a flat line whether the bin holds ten molecules or ten million, and thinning
@@ -508,7 +532,7 @@ warning: 79.4% of molecules were seen once. A consensus over one read is that re
508
532
  ```
509
533
 
510
534
  It is also the memory-hostile case, because distinct barcodes are what everything scales with, so
511
- it is what the benchmarks use: 190,595 reads/s at 1.02 reads/UMI, 259 B resident per distinct
535
+ it is what the benchmarks use: 1,179,549 reads/s at 1.02 reads/UMI, 282 B resident per distinct
512
536
  barcode, still bounded by the bucket rather than the library.
513
537
 
514
538
  ### Speed and memory are reported, not assumed
@@ -521,11 +545,12 @@ and nothing else, which is asserted per stage in C++, at the CLI, and under the
521
545
 
522
546
  | stage | 1 thread | 16 threads | bound by |
523
547
  |---|---|---|---|
524
- | `checkout` | 202,717 | **1,056,472** | reads |
525
- | `refine` | 605,611 | **1,012,368** | distinct barcodes |
526
- | `assemble` | 549,745 | **2,051,937** | reads, then the largest bucket |
548
+ | `checkout` | 213,880 | **1,548,835** | reads |
549
+ | `refine` | 617,802 | **1,554,156** | distinct barcodes |
550
+ | `assemble` | 554,106 | **2,470,928** | reads, then the largest bucket |
527
551
 
528
- reads/s. Two of those used to be 222,017 and 202,977, and the first fix was not a thread: **zlib at
552
+ reads/s — `refine` and `assemble` on a 500 k-read sample, `checkout` on 2 M. Two of those used to
553
+ be 222,017 and 202,977, and the first fix was not a thread: **zlib at
529
554
  its default level 6 was 83% of refine's wall clock**, compressing an intermediate the next stage
530
555
  decompresses immediately. Level 1 costs 21% more bytes and gave 3x before a single thread was
531
556
  added. Measure the stage before parallelising it.
@@ -543,18 +568,32 @@ malloc instead of inflate.
543
568
  | reads/s | 1,481,946 | **2,051,937** |
544
569
  | peak RSS | 1,479 MB | **789 MB** |
545
570
 
571
+ That is the record of *that* change. Batching the work-claiming atomic later took the same corpus
572
+ to **1.72 s and 2,324,403 reads/s**.
573
+
546
574
  The memory fell with it, because the estimate deciding how finely to cut the input said a gzipped
547
575
  FASTQ goes resident at 8x its on-disk size and it is really **19x** — a resident record is two heap
548
576
  `std::string` with their allocator headers, not the 180 bytes of payload. Guessing low picks too
549
577
  few buckets, and pass 2 holds sixteen of them at once.
550
578
 
579
+ And once every stage was threaded, the largest single cost in the whole pipeline was the
580
+ **thread helper itself**. `parallel_for` handed out one item per atomic `fetch_add`, and when an
581
+ item is one read's tag scan that instruction took **21% of all CPU samples across every thread** —
582
+ more than the work it was handing out. Sixteen cores serialise on one cache line. Items are claimed
583
+ in batches now, sized so each worker takes ~8 turns, collapsing to 1 when there are few items,
584
+ which is the uneven case (one bucket per item) the counter existed for. Two serial blocks went with
585
+ it: the distance-1 census inside `estimate_umi_error`, which is checkout's per-sample tail, and
586
+ refine's residual-FDR scan, which was 0.53 s of a 2.17 s run on one core after everything around it
587
+ had been parallelised. Both are read-only scans of the barcode table that tally integers, so they
588
+ thread exactly and `-t` still changes nothing but the clock.
589
+
551
590
  | threads | reads/s | matching reads/s | peak RSS |
552
591
  |---|---|---|---|
553
- | 1 | 202,717 | 217,506 | 60 MB |
554
- | 2 | 362,486 | 412,855 | 78 MB |
555
- | 4 | 603,773 | 758,801 | 97 MB |
556
- | 8 | 910,500 | 1,335,791 | 136 MB |
557
- | 16 | **1,056,472** | **1,684,654** | 217 MB |
592
+ | 1 | 213,880 | 216,584 | 59 MB |
593
+ | 2 | 394,471 | 403,393 | 79 MB |
594
+ | 4 | 737,777 | 768,810 | 101 MB |
595
+ | 8 | 1,256,838 | 1,349,533 | 139 MB |
596
+ | 16 | **1,548,835** | **1,697,313** | 215 MB |
558
597
 
559
598
  <p align="center"><img alt="checkout thread scaling" src="assets/benchmark_threads.svg" width="72%"></p>
560
599
 
@@ -163,8 +163,8 @@ reads 2,000,000
163
163
  unmatched 0 (0.0%)
164
164
  ambiguous 0 (0.0%)
165
165
 
166
- 2.2 s (903,599 reads/s) = 1.5 s matching on 8 threads + 0.7 s UMI statistics, serial
167
- peak RSS 131.0 MB of which UMI counters 21.2 MB
166
+ 1.6 s (1,243,801 reads/s) = 1.5 s matching on 8 threads + 0.1 s UMI statistics
167
+ peak RSS 136.0 MB of which UMI counters 11.5 MB
168
168
 
169
169
  sample reads UMIs reads/UMI UMI len eff len
170
170
  S1 500,000 125,000 4.00 12 12.00
@@ -231,7 +231,7 @@ migec assemble out/S1.fq.gz -o cons/ --contig # random-primed reads that til
231
231
  A molecule is **sample + cell barcode + UMI**, never the UMI alone — the same UMI in two cells is
232
232
  two molecules, and that is the design rather than a defect. Reads are range partitioned on the
233
233
  packed key into `.mig` buckets and one bucket is sorted at a time, so nothing scales with the
234
- library: 531,365 reads/s, and 121 MB at 16 buckets against 203 MB at one.
234
+ library: 2,423,777 reads/s, and 123 MB at 16 buckets against 203 MB at one.
235
235
 
236
236
  The per-column posterior is `LL[j][b] = Σ_i (r==b ? log(1−e) : log(e/3))`, and then the number that
237
237
  matters:
@@ -334,6 +334,30 @@ bent one.
334
334
 
335
335
  <p align="center"><img alt="molecule size against rank" src="assets/mig_size_zipf.svg" width="72%"></p>
336
336
 
337
+ **Barcode errors against the parent's depth** — how many distinct error children a molecule
338
+ spawned, and how many reads were in them. A parent seen *c* times had *c·L* barcode bases to
339
+ miscall, so both series climb with *c* — but only one of them can climb forever. There are exactly
340
+ *3L* barcodes one substitution away, so the distinct-children series bends over at the dashed line
341
+ while the reads series carries straight past it. Where they part is where this library's barcode
342
+ neighbourhood filled up, measured rather than predicted.
343
+
344
+ <p align="center"><img alt="barcode errors against depth" src="assets/umi_error_children.svg" width="72%"></p>
345
+
346
+ Invert both and you get two estimates of the same barcode error rate, per depth, with what `refine`
347
+ reports drawn across them. The y axis is a log error rate, so one decade is exactly ten Phred and
348
+ 1e-3 is Q30 — which makes the number directly comparable with the barcode's own reported quality,
349
+ the only independent check on it there is. On a diverse library sequenced 25 deep it lands within
350
+ 1% of a known injected rate.
351
+
352
+ <p align="center"><img alt="barcode error rate by depth" src="assets/umi_error_rate.svg" width="72%"></p>
353
+
354
+ > **Never** read this instead of the `saturated` flag. Both series count only the children
355
+ > correction actually merged, and on a *full* barcode space `refine` refuses to merge — rightly,
356
+ > because a distance-1 neighbour there is more likely a real molecule than an error child — so both
357
+ > estimates collapse to zero. Against a known rate they are 0.99 and 0.97 of the truth at 0.2%
358
+ > occupancy, 0.62 and 0.45 at 33%, and nothing at 100%. See
359
+ > [docs/umi_errors.rst](docs/umi_errors.rst).
360
+
337
361
  **Consensus quality against depth**, as a box, never a thinned scatter. Emitted quality is discrete
338
362
  and capped at the floor, so at any real depth every molecule sits on one or two integers: a cloud
339
363
  of dots draws that as a flat line whether the bin holds ten molecules or ten million, and thinning
@@ -474,7 +498,7 @@ warning: 79.4% of molecules were seen once. A consensus over one read is that re
474
498
  ```
475
499
 
476
500
  It is also the memory-hostile case, because distinct barcodes are what everything scales with, so
477
- it is what the benchmarks use: 190,595 reads/s at 1.02 reads/UMI, 259 B resident per distinct
501
+ it is what the benchmarks use: 1,179,549 reads/s at 1.02 reads/UMI, 282 B resident per distinct
478
502
  barcode, still bounded by the bucket rather than the library.
479
503
 
480
504
  ### Speed and memory are reported, not assumed
@@ -487,11 +511,12 @@ and nothing else, which is asserted per stage in C++, at the CLI, and under the
487
511
 
488
512
  | stage | 1 thread | 16 threads | bound by |
489
513
  |---|---|---|---|
490
- | `checkout` | 202,717 | **1,056,472** | reads |
491
- | `refine` | 605,611 | **1,012,368** | distinct barcodes |
492
- | `assemble` | 549,745 | **2,051,937** | reads, then the largest bucket |
514
+ | `checkout` | 213,880 | **1,548,835** | reads |
515
+ | `refine` | 617,802 | **1,554,156** | distinct barcodes |
516
+ | `assemble` | 554,106 | **2,470,928** | reads, then the largest bucket |
493
517
 
494
- reads/s. Two of those used to be 222,017 and 202,977, and the first fix was not a thread: **zlib at
518
+ reads/s — `refine` and `assemble` on a 500 k-read sample, `checkout` on 2 M. Two of those used to
519
+ be 222,017 and 202,977, and the first fix was not a thread: **zlib at
495
520
  its default level 6 was 83% of refine's wall clock**, compressing an intermediate the next stage
496
521
  decompresses immediately. Level 1 costs 21% more bytes and gave 3x before a single thread was
497
522
  added. Measure the stage before parallelising it.
@@ -509,18 +534,32 @@ malloc instead of inflate.
509
534
  | reads/s | 1,481,946 | **2,051,937** |
510
535
  | peak RSS | 1,479 MB | **789 MB** |
511
536
 
537
+ That is the record of *that* change. Batching the work-claiming atomic later took the same corpus
538
+ to **1.72 s and 2,324,403 reads/s**.
539
+
512
540
  The memory fell with it, because the estimate deciding how finely to cut the input said a gzipped
513
541
  FASTQ goes resident at 8x its on-disk size and it is really **19x** — a resident record is two heap
514
542
  `std::string` with their allocator headers, not the 180 bytes of payload. Guessing low picks too
515
543
  few buckets, and pass 2 holds sixteen of them at once.
516
544
 
545
+ And once every stage was threaded, the largest single cost in the whole pipeline was the
546
+ **thread helper itself**. `parallel_for` handed out one item per atomic `fetch_add`, and when an
547
+ item is one read's tag scan that instruction took **21% of all CPU samples across every thread** —
548
+ more than the work it was handing out. Sixteen cores serialise on one cache line. Items are claimed
549
+ in batches now, sized so each worker takes ~8 turns, collapsing to 1 when there are few items,
550
+ which is the uneven case (one bucket per item) the counter existed for. Two serial blocks went with
551
+ it: the distance-1 census inside `estimate_umi_error`, which is checkout's per-sample tail, and
552
+ refine's residual-FDR scan, which was 0.53 s of a 2.17 s run on one core after everything around it
553
+ had been parallelised. Both are read-only scans of the barcode table that tally integers, so they
554
+ thread exactly and `-t` still changes nothing but the clock.
555
+
517
556
  | threads | reads/s | matching reads/s | peak RSS |
518
557
  |---|---|---|---|
519
- | 1 | 202,717 | 217,506 | 60 MB |
520
- | 2 | 362,486 | 412,855 | 78 MB |
521
- | 4 | 603,773 | 758,801 | 97 MB |
522
- | 8 | 910,500 | 1,335,791 | 136 MB |
523
- | 16 | **1,056,472** | **1,684,654** | 217 MB |
558
+ | 1 | 213,880 | 216,584 | 59 MB |
559
+ | 2 | 394,471 | 403,393 | 79 MB |
560
+ | 4 | 737,777 | 768,810 | 101 MB |
561
+ | 8 | 1,256,838 | 1,349,533 | 139 MB |
562
+ | 16 | **1,548,835** | **1,697,313** | 215 MB |
524
563
 
525
564
  <p align="center"><img alt="checkout thread scaling" src="assets/benchmark_threads.svg" width="72%"></p>
526
565
 
@@ -18,14 +18,39 @@ count. `scripts/spikein_ratio.py` computes the published spike-in validation met
18
18
  and merging reported separately.
19
19
 
20
20
  **Throughput and footprint**: every stage threads and every stage is byte-identical at any `-t` --
21
- checkout 1.06 M reads/s end to end (1.68 M matching), refine 1.01 M, assemble 1.43 M, all at 16
21
+ checkout 1.55 M reads/s end to end (1.70 M matching), refine 1.55 M, assemble 2.47 M, all at 16
22
22
  threads; ~22 bytes per distinct UMI against a hash map's ~48. Verified under the thread sanitizer
23
23
  as well as by comparison at 1..16 threads.
24
24
  The counters are not yet partitioned, which is the open memory item and lands with `.mig` buckets.
25
25
 
26
+ **QC**: twenty gnuplot panels over the tables the stages write, including the four figures a user
27
+ already knows how to read -- Cell Ranger's barcode rank plot on unique UMIs, the MIG size spectrum
28
+ with molecules *and* reads, the rank/Zipf curve, and consensus quality as a box over an exact
29
+ `(depth, quality)` grid.
30
+
26
31
  Milestones are ordered by risk, not by pipeline order: the consensus quality model is the
27
32
  scientific claim and is validated before any throughput work.
28
33
 
34
+ ## What is left, in the order it should be done
35
+
36
+ Everything below is either open or half-open. Nothing else on this page is.
37
+
38
+ | # | item | milestone | why it is next |
39
+ |---|---|---|---|
40
+ | 1 | `.mig` bucket output from `checkout` | M2 | the **only** unbounded allocation left: the UMI counters are 8.8 GB at NovaSeq scale, held in one piece, and this is what bounds them. `checkout` warns past 1 GB and that is a warning, not a fix |
41
+ | 2 | Bucketed correction, two passes with the key rotated | M3 | same problem one stage on. `refine` holds the whole barcode table; a plain range partition splits a barcode from its neighbour for the top b/2 positions, so the rotation is the fix and it is understood, not designed |
42
+ | 3 | The template's own error split | M3 | the pattern's constant bases calibrate the **primer**, not the polymerase — the fitted intercept is a synthesis defect rate. Separating sequencing from RT/PCR error needs a different standard, and until it exists `--rt-error` is a named class rather than a measurement |
43
+ | 4 | `--rt-error auto` | M1 | falls straight out of (3). The floor is a property of the enzyme and the cycle count, so fitting it per dataset is the honest default once there is something to fit against |
44
+ | 5 | i7xi5 contingency table | M2 | the only way index hopping is actually estimable, and it needs nothing that is not already in the headers |
45
+ | 6 | `2026-migec-benchmark`, the published comparisons | M5 | MIGEC v1, MAGERI, UMI-tools, Calib, fgbio, Cell Ranger, UMI-VarCal, UMIErrorCorrect. This is what the version number is waiting on, not the code |
46
+ | 7 | ctDNA ground truth, COSMIC at 0.005-0.075 VAF | M5 | the one claim with no ground truth yet. Never: Maruzani's deposited runs carry **no** UMI, so they have to be simulated -- and their own rule assumes co-terminal reads, which X1 falsified |
47
+ | 8 | R1/R2 overlap merge | M1 | a special case of `--contig`'s placement, never a second matcher in `checkout` |
48
+ | 9 | Bit-parallel matcher | M2 | last, deliberately: the scan is O(offsets x pattern) and is **not** the bottleneck. It goes in when a benchmark says so |
49
+
50
+ Two things are blocked on data rather than on work: **Britanova et al aging** (bulk TCR, shallow --
51
+ the real 1-3 reads/UMI dataset) lives on aldan3 and has not been pulled, and the ctDNA ground truth
52
+ above has to be built rather than downloaded.
53
+
29
54
  ## M0 — skeleton, format, simulator
30
55
 
31
56
  - [x] Archive the Groovy implementation on `legacy-v1` / `v1-final`, start master fresh
@@ -105,7 +130,7 @@ scientific claim and is validated before any throughput work.
105
130
  when that makes contigs untrustworthy
106
131
  - [x] Shallow libraries (1-3 reads/UMI) run, report the coverage histogram, threshold nothing, and
107
132
  say that the UMI is buying counting rather than error correction. Benchmarked as the
108
- memory-hostile shape: 190,595 reads/s at 1.02 reads/UMI, 259 B resident per distinct barcode
133
+ memory-hostile shape: 1,179,549 reads/s at 1.02 reads/UMI, 282 B resident per distinct barcode
109
134
  - [x] **`--rt-error` names the chemistry** rather than guessing: `rt` 1e-4 (default, Q40 -- 10x's
110
135
  figure for the V(D)J RT and X2's own measurement), `medium` 1e-5, `high` 1e-6, or the rate.
111
136
  Never: it is the ONE-MOLECULE floor; 10x's Q60 needs two UMIs to agree and that is arda's job
@@ -175,11 +175,11 @@ comparators for M5, `UMI-VarCal` and `UMIErrorCorrect`.
175
175
 
176
176
  | Item | Value |
177
177
  |---|---|
178
- | Use | the four panels the README embeds: barcode rank, MIG size spectrum, rank/Zipf, consensus quality |
178
+ | Use | the panels the README embeds: barcode rank, MIG size spectrum, rank/Zipf, consensus quality, and the two barcode-error-against-depth panels |
179
179
  | Regenerate | `python scripts/example_figures.py` -- simulates, runs all three stages, copies the tables, redraws |
180
- | Tables | `assets/PBMC.cell_rank.tsv`, `PBMC.sizes.tsv`, `assemble.quality_by_depth.tsv`, `assemble.coverage.tsv` |
180
+ | Tables | `assets/PBMC.cell_rank.tsv`, `PBMC.sizes.tsv`, `PBMC.umi_errors.tsv`, `assemble.quality_by_depth.tsv`, `assemble.coverage.tsv` |
181
181
  | Shape | 120 cells x 40-200 molecules, Pareto(1.1) depth, plus 4,000 ambient barcodes with 1-3 molecules each; 0.2% barcode and 1% payload error, seed 3 |
182
- | Why that shape | a uniform simulator draws all four panels as straight lines. The ambient population is what makes the knee; the Pareto tail is what gives the rank curve slope; the per-base error is what gives emitted quality a spread to draw quartiles over |
182
+ | Why that shape | a uniform simulator draws every panel as a straight line. The Pareto tail reaching ~19,000 reads on one molecule is also what makes the barcode-error panels show anything: saturation of the `3L` neighbour shell is only visible on a molecule deep enough to have spawned all of it. The ambient population is what makes the knee; the Pareto tail is what gives the rank curve slope; the per-base error is what gives emitted quality a spread to draw quartiles over |
183
183
  | Provenance | **derived (simulated)** -- never presented as a measurement. `assets/SRR1763769.mig.tsv` beside it is experimental; see the Primer ID entry |
184
184
  | Storage | the TSVs are committed, the FASTQ is not (`--keep` writes it locally) |
185
185