migec 2.0.0a4__tar.gz → 2.2.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Potentially problematic release.


This version of migec might be problematic. Click here for more details.

Files changed (199) hide show
  1. {migec-2.0.0a4 → migec-2.2.0}/CHANGELOG.md +146 -0
  2. {migec-2.0.0a4 → migec-2.2.0}/CLAUDE.md +82 -5
  3. {migec-2.0.0a4 → migec-2.2.0}/PKG-INFO +98 -27
  4. {migec-2.0.0a4 → migec-2.2.0}/README.md +97 -26
  5. {migec-2.0.0a4 → migec-2.2.0}/ROADMAP.md +45 -7
  6. {migec-2.0.0a4 → migec-2.2.0}/SOURCES.md +12 -0
  7. migec-2.2.0/assets/PBMC.cell_rank.tsv +144 -0
  8. migec-2.2.0/assets/PBMC.sizes.tsv +160 -0
  9. migec-2.2.0/assets/PBMC.umi_errors.tsv +160 -0
  10. migec-2.2.0/assets/assemble.coverage.tsv +16 -0
  11. migec-2.2.0/assets/assemble.quality_by_depth.tsv +14 -0
  12. migec-2.2.0/assets/assemble_coverage.gp +24 -0
  13. migec-2.2.0/assets/assemble_coverage.svg +230 -0
  14. {migec-2.0.0a4 → migec-2.2.0}/assets/benchmark_threads.gp +12 -6
  15. migec-2.2.0/assets/benchmark_threads.json +47 -0
  16. {migec-2.0.0a4 → migec-2.2.0}/assets/benchmark_threads.svg +87 -87
  17. migec-2.2.0/assets/benchmark_threads.tsv +6 -0
  18. migec-2.2.0/assets/cell_rank.gp +27 -0
  19. migec-2.2.0/assets/cell_rank.svg +234 -0
  20. migec-2.2.0/assets/consensus_error.gp +24 -0
  21. migec-2.2.0/assets/consensus_error.svg +287 -0
  22. migec-2.2.0/assets/consensus_layout.gp +24 -0
  23. {migec-2.0.0a4 → migec-2.2.0}/assets/consensus_layout.svg +69 -79
  24. migec-2.2.0/assets/consensus_quality.gp +29 -0
  25. migec-2.2.0/assets/consensus_quality.svg +325 -0
  26. migec-2.2.0/assets/mig_size_spectrum.gp +39 -0
  27. migec-2.2.0/assets/mig_size_spectrum.svg +511 -0
  28. migec-2.2.0/assets/mig_size_zipf.gp +27 -0
  29. migec-2.2.0/assets/mig_size_zipf.svg +240 -0
  30. {migec-2.0.0a4 → migec-2.2.0}/assets/pipeline.dot +18 -6
  31. migec-2.2.0/assets/pipeline.svg +163 -0
  32. migec-2.2.0/assets/umi_error_children.gp +32 -0
  33. migec-2.2.0/assets/umi_error_children.svg +578 -0
  34. migec-2.2.0/assets/umi_error_rate.gp +47 -0
  35. migec-2.2.0/assets/umi_error_rate.svg +532 -0
  36. {migec-2.0.0a4 → migec-2.2.0}/docs/assemble.rst +2 -2
  37. {migec-2.0.0a4 → migec-2.2.0}/docs/commands.rst +1 -1
  38. {migec-2.0.0a4 → migec-2.2.0}/docs/examples.rst +39 -0
  39. {migec-2.0.0a4 → migec-2.2.0}/docs/formats.rst +71 -0
  40. {migec-2.0.0a4 → migec-2.2.0}/docs/index.rst +2 -2
  41. {migec-2.0.0a4 → migec-2.2.0}/docs/method.rst +3 -0
  42. {migec-2.0.0a4 → migec-2.2.0}/docs/nextflow.rst +3 -3
  43. {migec-2.0.0a4 → migec-2.2.0}/docs/performance.rst +82 -45
  44. migec-2.2.0/docs/plots.rst +198 -0
  45. {migec-2.0.0a4 → migec-2.2.0}/docs/refine.rst +9 -0
  46. migec-2.2.0/docs/umi_errors.rst +167 -0
  47. {migec-2.0.0a4 → migec-2.2.0}/include/migec/assemble.hpp +11 -0
  48. {migec-2.0.0a4 → migec-2.2.0}/include/migec/parallel.hpp +26 -10
  49. {migec-2.0.0a4 → migec-2.2.0}/include/migec/refine.hpp +12 -0
  50. {migec-2.0.0a4 → migec-2.2.0}/include/migec/umi_stats.hpp +5 -1
  51. {migec-2.0.0a4 → migec-2.2.0}/include/migec/version.hpp +1 -1
  52. {migec-2.0.0a4 → migec-2.2.0}/integrations/nextflow/modules/migec/assemble/main.nf +1 -1
  53. {migec-2.0.0a4 → migec-2.2.0}/integrations/nextflow/modules/migec/checkout/main.nf +1 -1
  54. {migec-2.0.0a4 → migec-2.2.0}/integrations/nextflow/modules/migec/plot/main.nf +1 -1
  55. {migec-2.0.0a4 → migec-2.2.0}/integrations/nextflow/modules/migec/refine/main.nf +1 -1
  56. {migec-2.0.0a4 → migec-2.2.0}/project/README.md +9 -0
  57. {migec-2.0.0a4 → migec-2.2.0}/pyproject.toml +1 -1
  58. {migec-2.0.0a4 → migec-2.2.0}/python/migec/__init__.py +1 -1
  59. {migec-2.0.0a4 → migec-2.2.0}/python/migec/assemble.py +53 -0
  60. {migec-2.0.0a4 → migec-2.2.0}/python/migec/checkout.py +5 -3
  61. {migec-2.0.0a4 → migec-2.2.0}/python/migec/cli.py +4 -3
  62. {migec-2.0.0a4 → migec-2.2.0}/python/migec/plot.py +198 -28
  63. {migec-2.0.0a4 → migec-2.2.0}/python/migec/refine.py +8 -0
  64. migec-2.2.0/scripts/example_figures.py +113 -0
  65. {migec-2.0.0a4 → migec-2.2.0}/skills/migec/SKILL.md +33 -2
  66. {migec-2.0.0a4 → migec-2.2.0}/src/_bindings.cpp +19 -0
  67. {migec-2.0.0a4 → migec-2.2.0}/src/assemble.cpp +46 -16
  68. {migec-2.0.0a4 → migec-2.2.0}/src/refine.cpp +234 -18
  69. {migec-2.0.0a4 → migec-2.2.0}/src/umi_stats.cpp +14 -7
  70. {migec-2.0.0a4 → migec-2.2.0}/tests/synthetic/test_refine.py +2 -0
  71. migec-2.2.0/tests/synthetic/test_umi_errors.py +135 -0
  72. {migec-2.0.0a4 → migec-2.2.0}/tests/unit/test_plot.py +69 -0
  73. migec-2.0.0a4/assets/assemble.coverage.tsv +0 -10
  74. migec-2.0.0a4/assets/assemble_coverage.gp +0 -18
  75. migec-2.0.0a4/assets/assemble_coverage.svg +0 -212
  76. migec-2.0.0a4/assets/benchmark_threads.json +0 -47
  77. migec-2.0.0a4/assets/benchmark_threads.tsv +0 -6
  78. migec-2.0.0a4/assets/consensus_error.gp +0 -17
  79. migec-2.0.0a4/assets/consensus_error.svg +0 -335
  80. migec-2.0.0a4/assets/consensus_layout.gp +0 -18
  81. migec-2.0.0a4/assets/consensus_quality.gp +0 -18
  82. migec-2.0.0a4/assets/consensus_quality.svg +0 -374
  83. migec-2.0.0a4/assets/pipeline.svg +0 -164
  84. migec-2.0.0a4/docs/plots.rst +0 -113
  85. {migec-2.0.0a4 → migec-2.2.0}/.github/workflows/ci.yml +0 -0
  86. {migec-2.0.0a4 → migec-2.2.0}/.github/workflows/docs.yml +0 -0
  87. {migec-2.0.0a4 → migec-2.2.0}/.github/workflows/publish.yml +0 -0
  88. {migec-2.0.0a4 → migec-2.2.0}/.github/workflows/testpypi.yml +0 -0
  89. {migec-2.0.0a4 → migec-2.2.0}/.gitignore +0 -0
  90. {migec-2.0.0a4 → migec-2.2.0}/CMakeLists.txt +0 -0
  91. {migec-2.0.0a4 → migec-2.2.0}/LICENSE +0 -0
  92. {migec-2.0.0a4 → migec-2.2.0}/assets/SRR1763769.mig.tsv +0 -0
  93. {migec-2.0.0a4 → migec-2.2.0}/docs/Makefile +0 -0
  94. {migec-2.0.0a4 → migec-2.2.0}/docs/barcode_space.rst +0 -0
  95. {migec-2.0.0a4 → migec-2.2.0}/docs/checkout.rst +0 -0
  96. {migec-2.0.0a4 → migec-2.2.0}/docs/conf.py +0 -0
  97. {migec-2.0.0a4 → migec-2.2.0}/docs/downstream.rst +0 -0
  98. {migec-2.0.0a4 → migec-2.2.0}/docs/fragmented.rst +0 -0
  99. {migec-2.0.0a4 → migec-2.2.0}/docs/grouping.rst +0 -0
  100. {migec-2.0.0a4 → migec-2.2.0}/docs/installation.rst +0 -0
  101. {migec-2.0.0a4 → migec-2.2.0}/docs/layouts.rst +0 -0
  102. {migec-2.0.0a4 → migec-2.2.0}/docs/nulls.rst +0 -0
  103. {migec-2.0.0a4 → migec-2.2.0}/docs/quality_floor.rst +0 -0
  104. {migec-2.0.0a4 → migec-2.2.0}/docs/reference.rst +0 -0
  105. {migec-2.0.0a4 → migec-2.2.0}/docs/roadmap.rst +0 -0
  106. {migec-2.0.0a4 → migec-2.2.0}/docs/subsample.rst +0 -0
  107. {migec-2.0.0a4 → migec-2.2.0}/docs/suggest.rst +0 -0
  108. {migec-2.0.0a4 → migec-2.2.0}/docs/umi_statistics.rst +0 -0
  109. {migec-2.0.0a4 → migec-2.2.0}/docs/validation.rst +0 -0
  110. {migec-2.0.0a4 → migec-2.2.0}/include/migec/checkout.hpp +0 -0
  111. {migec-2.0.0a4 → migec-2.2.0}/include/migec/consensus.hpp +0 -0
  112. {migec-2.0.0a4 → migec-2.2.0}/include/migec/fastq.hpp +0 -0
  113. {migec-2.0.0a4 → migec-2.2.0}/include/migec/mig_record.hpp +0 -0
  114. {migec-2.0.0a4 → migec-2.2.0}/include/migec/pattern.hpp +0 -0
  115. {migec-2.0.0a4 → migec-2.2.0}/include/migec/resource.hpp +0 -0
  116. {migec-2.0.0a4 → migec-2.2.0}/include/migec/subsample.hpp +0 -0
  117. {migec-2.0.0a4 → migec-2.2.0}/include/migec/suggest.hpp +0 -0
  118. {migec-2.0.0a4 → migec-2.2.0}/include/migec/types.hpp +0 -0
  119. {migec-2.0.0a4 → migec-2.2.0}/include/migec/whitelist.hpp +0 -0
  120. {migec-2.0.0a4 → migec-2.2.0}/integrations/nextflow/README.md +0 -0
  121. {migec-2.0.0a4 → migec-2.2.0}/integrations/nextflow/modules/migec/assemble/meta.yml +0 -0
  122. {migec-2.0.0a4 → migec-2.2.0}/integrations/nextflow/modules/migec/checkout/meta.yml +0 -0
  123. {migec-2.0.0a4 → migec-2.2.0}/integrations/nextflow/modules/migec/environment.yml +0 -0
  124. {migec-2.0.0a4 → migec-2.2.0}/integrations/nextflow/modules/migec/refine/meta.yml +0 -0
  125. {migec-2.0.0a4 → migec-2.2.0}/integrations/nextflow/nextflow.config +0 -0
  126. {migec-2.0.0a4 → migec-2.2.0}/integrations/nextflow/subworkflows/migec/main.nf +0 -0
  127. {migec-2.0.0a4 → migec-2.2.0}/notebooks/barcode_space.py +0 -0
  128. {migec-2.0.0a4 → migec-2.2.0}/notebooks/platforms.py +0 -0
  129. {migec-2.0.0a4 → migec-2.2.0}/notebooks/refine_diagnostics.py +0 -0
  130. {migec-2.0.0a4 → migec-2.2.0}/project/design-assemble.md +0 -0
  131. {migec-2.0.0a4 → migec-2.2.0}/project/design-benchmarks.md +0 -0
  132. {migec-2.0.0a4 → migec-2.2.0}/project/design-checkout.md +0 -0
  133. {migec-2.0.0a4 → migec-2.2.0}/project/design-io-interop.md +0 -0
  134. {migec-2.0.0a4 → migec-2.2.0}/project/design-refine.md +0 -0
  135. {migec-2.0.0a4 → migec-2.2.0}/project/design-repo.md +0 -0
  136. {migec-2.0.0a4 → migec-2.2.0}/project/review-algorithms.md +0 -0
  137. {migec-2.0.0a4 → migec-2.2.0}/project/review-scope.md +0 -0
  138. {migec-2.0.0a4 → migec-2.2.0}/python/migec/sheet.py +0 -0
  139. {migec-2.0.0a4 → migec-2.2.0}/python/migec/subsample.py +0 -0
  140. {migec-2.0.0a4 → migec-2.2.0}/python/migec/suggest.py +0 -0
  141. {migec-2.0.0a4 → migec-2.2.0}/scripts/benchmark_threads.py +0 -0
  142. {migec-2.0.0a4 → migec-2.2.0}/scripts/collision_check.py +0 -0
  143. {migec-2.0.0a4 → migec-2.2.0}/scripts/compare_calib.py +0 -0
  144. {migec-2.0.0a4 → migec-2.2.0}/scripts/correction_accuracy.py +0 -0
  145. {migec-2.0.0a4 → migec-2.2.0}/scripts/diagnose.py +0 -0
  146. {migec-2.0.0a4 → migec-2.2.0}/scripts/permutation_nulls.py +0 -0
  147. {migec-2.0.0a4 → migec-2.2.0}/scripts/quality_floor.py +0 -0
  148. {migec-2.0.0a4 → migec-2.2.0}/scripts/read_start_dispersion.py +0 -0
  149. {migec-2.0.0a4 → migec-2.2.0}/scripts/spikein_ratio.py +0 -0
  150. {migec-2.0.0a4 → migec-2.2.0}/setup.sh +0 -0
  151. {migec-2.0.0a4 → migec-2.2.0}/src/checkout.cpp +0 -0
  152. {migec-2.0.0a4 → migec-2.2.0}/src/consensus.cpp +0 -0
  153. {migec-2.0.0a4 → migec-2.2.0}/src/fastq.cpp +0 -0
  154. {migec-2.0.0a4 → migec-2.2.0}/src/mig_record.cpp +0 -0
  155. {migec-2.0.0a4 → migec-2.2.0}/src/pattern.cpp +0 -0
  156. {migec-2.0.0a4 → migec-2.2.0}/src/resource.cpp +0 -0
  157. {migec-2.0.0a4 → migec-2.2.0}/src/subsample.cpp +0 -0
  158. {migec-2.0.0a4 → migec-2.2.0}/src/suggest.cpp +0 -0
  159. {migec-2.0.0a4 → migec-2.2.0}/src/types.cpp +0 -0
  160. {migec-2.0.0a4 → migec-2.2.0}/src/whitelist.cpp +0 -0
  161. {migec-2.0.0a4 → migec-2.2.0}/tests/__init__.py +0 -0
  162. {migec-2.0.0a4 → migec-2.2.0}/tests/benchmark/__init__.py +0 -0
  163. {migec-2.0.0a4 → migec-2.2.0}/tests/benchmark/test_assemble_speed.py +0 -0
  164. {migec-2.0.0a4 → migec-2.2.0}/tests/benchmark/test_checkout_speed.py +0 -0
  165. {migec-2.0.0a4 → migec-2.2.0}/tests/benchmark/test_refine_speed.py +0 -0
  166. {migec-2.0.0a4 → migec-2.2.0}/tests/conftest.py +0 -0
  167. {migec-2.0.0a4 → migec-2.2.0}/tests/cpp/doctest.h +0 -0
  168. {migec-2.0.0a4 → migec-2.2.0}/tests/cpp/test_checkout.cpp +0 -0
  169. {migec-2.0.0a4 → migec-2.2.0}/tests/cpp/test_checkout_run.cpp +0 -0
  170. {migec-2.0.0a4 → migec-2.2.0}/tests/cpp/test_consensus.cpp +0 -0
  171. {migec-2.0.0a4 → migec-2.2.0}/tests/cpp/test_fastq.cpp +0 -0
  172. {migec-2.0.0a4 → migec-2.2.0}/tests/cpp/test_mig_record.cpp +0 -0
  173. {migec-2.0.0a4 → migec-2.2.0}/tests/cpp/test_parallel_stages.cpp +0 -0
  174. {migec-2.0.0a4 → migec-2.2.0}/tests/cpp/test_pattern.cpp +0 -0
  175. {migec-2.0.0a4 → migec-2.2.0}/tests/cpp/test_types.cpp +0 -0
  176. {migec-2.0.0a4 → migec-2.2.0}/tests/cpp/test_umi_stats.cpp +0 -0
  177. {migec-2.0.0a4 → migec-2.2.0}/tests/cpp/test_whitelist.cpp +0 -0
  178. {migec-2.0.0a4 → migec-2.2.0}/tests/data/migec_spikein_truth.tsv +0 -0
  179. {migec-2.0.0a4 → migec-2.2.0}/tests/realworld/__init__.py +0 -0
  180. {migec-2.0.0a4 → migec-2.2.0}/tests/synthetic/__init__.py +0 -0
  181. {migec-2.0.0a4 → migec-2.2.0}/tests/synthetic/_sim.py +0 -0
  182. {migec-2.0.0a4 → migec-2.2.0}/tests/synthetic/test_assemble.py +0 -0
  183. {migec-2.0.0a4 → migec-2.2.0}/tests/synthetic/test_barcode_space.py +0 -0
  184. {migec-2.0.0a4 → migec-2.2.0}/tests/synthetic/test_checkout_end_to_end.py +0 -0
  185. {migec-2.0.0a4 → migec-2.2.0}/tests/synthetic/test_dual_end.py +0 -0
  186. {migec-2.0.0a4 → migec-2.2.0}/tests/synthetic/test_grouping_accuracy.py +0 -0
  187. {migec-2.0.0a4 → migec-2.2.0}/tests/synthetic/test_nulls.py +0 -0
  188. {migec-2.0.0a4 → migec-2.2.0}/tests/synthetic/test_quality_floor.py +0 -0
  189. {migec-2.0.0a4 → migec-2.2.0}/tests/synthetic/test_sim.py +0 -0
  190. {migec-2.0.0a4 → migec-2.2.0}/tests/synthetic/test_subsample.py +0 -0
  191. {migec-2.0.0a4 → migec-2.2.0}/tests/synthetic/test_suggest.py +0 -0
  192. {migec-2.0.0a4 → migec-2.2.0}/tests/synthetic/test_thread_invariance.py +0 -0
  193. {migec-2.0.0a4 → migec-2.2.0}/tests/unit/__init__.py +0 -0
  194. {migec-2.0.0a4 → migec-2.2.0}/tests/unit/test_bindings.py +0 -0
  195. {migec-2.0.0a4 → migec-2.2.0}/tests/unit/test_compare_calib.py +0 -0
  196. {migec-2.0.0a4 → migec-2.2.0}/tests/unit/test_core.py +0 -0
  197. {migec-2.0.0a4 → migec-2.2.0}/tests/unit/test_downstream.py +0 -0
  198. {migec-2.0.0a4 → migec-2.2.0}/tests/unit/test_sheet_and_cli.py +0 -0
  199. {migec-2.0.0a4 → migec-2.2.0}/tests/unit/test_spikein_ratio.py +0 -0
@@ -4,6 +4,152 @@ Hand-written and prose-heavy: each entry says what changed and, where it matters
4
4
  prevents. Releases before 2.0.0 are the Groovy MIGEC and are described by their git tags on the
5
5
  `legacy-v1` branch.
6
6
 
7
+ ## 2.2.0 — 2026-08-13
8
+
9
+ Throughput and one new measurement. No breaking change: every 2.1.0 command, flag and output file
10
+ behaves as it did, and the additions are a new table, four new JSON fields and two new plot panels.
11
+
12
+ ### The barcode error rate, measured at every depth
13
+
14
+ `refine` writes `<sample>.umi_errors.tsv`: one row per exact parent depth, carrying the distinct
15
+ error children that parent spawned, the reads in them, and the error rate each implies. A parent
16
+ carrying `c` reads offered `c*L` barcode bases to be miscalled, so the same eps falls out two ways:
17
+
18
+ distinct children u(c) = 3L (1 - exp(-c eps / 3)) -> eps = -(3/c) ln(1 - u/3L)
19
+ reads in children r(c) = c L eps -> eps = r / (c L)
20
+
21
+ A barcode has only `3L` neighbours one substitution away, so the first **saturates**; reads have no
22
+ ceiling, so the second does not. Where the two part company on the figure is where this library's
23
+ barcode neighbourhood filled up, read off the data instead of predicted. Past saturation the first
24
+ is left blank rather than reported as a small number: inverting a full neighbourhood returns "no
25
+ errors" for the most error-ridden library there can be.
26
+
27
+ New report lines and JSON fields `error_at_depth`, `error_phred`, `error_from_children`,
28
+ `error_depth`. The number to quote is `error_at_depth` — restricted to parents seen >= 10 times,
29
+ where correction is near-complete — and `error_phred` is the same thing as a Phred so it can be put
30
+ beside the barcode's own reported Q. On a diverse library sequenced 25 deep with 1e-3 injected, the
31
+ distance-1 excess gives 9.73e-04, the children give 9.89e-04 (Q30) and all depths give 9.98e-04.
32
+ Three routes to one number agreeing within 3% is what makes any of them believable.
33
+
34
+ Never: **both estimators are bounded by the merges correction actually made**, so neither is
35
+ saturation-free. Against a known injected rate, as a fraction of truth:
36
+
37
+ | occupancy | 0.2% | 2.3% | 9.8% | 33% | 100% |
38
+ |---|---|---|---|---|---|
39
+ | distance-1 excess | 0.97 | 0.96 | 0.76 | 0.45 | 0.001 |
40
+ | from the children | **0.99** | **0.95** | **0.88** | **0.62** | 0.00 |
41
+
42
+ The children estimate is the better of the two wherever either works, and at 100% they both go to
43
+ zero for the same reason: on a full barcode space `correct_umis` refuses to merge, correctly,
44
+ because a distance-1 neighbour there is more likely a real molecule than a child. The `saturated`
45
+ flag is what says the answer is a floor — read it, do not read this table instead of it. And read
46
+ the table at depth: a child whose parent was never sequenced cannot be counted, which at 1-3
47
+ reads/UMI is 80% of them. `tests/synthetic/test_umi_errors.py` holds all of it, `docs/umi_errors.rst`
48
+ works through it.
49
+
50
+ Two new `migec plot` panels over that table, `umi_error_children` and `umi_error_rate`, with the
51
+ `3L` ceiling and the reported estimate drawn as reference lines. Note: `neighbours` and `estimate`
52
+ are constant down their columns on purpose — the panels draw them, and a figure that needs a value
53
+ its own table does not carry will one day disagree with the report.
54
+
55
+ ### `reads in them` is points, never a line
56
+
57
+ The `mig_size_spectrum` panel drew its reads series `with lines` over a table that has one row per
58
+ **exact** size. Past the head of the distribution almost every size holds exactly one molecule, so
59
+ reads == size, and the line rendered the `y = x` diagonal as the most prominent feature of the
60
+ figure — a tautology drawn as a second mode. Where two or three molecules shared a size the same
61
+ line sawtoothed between `size*1` and `size*2`, which is integer quantisation drawn as signal, and it
62
+ bridged gaps in the support where no size was observed at all. It is points now, and the new
63
+ barcode-error panels are points for the same reason.
64
+
65
+ ### The thread helper was the bottleneck
66
+
67
+ With all three stages threaded, a sampling profile of
68
+ `assemble` put **21% of all CPU samples, across every thread, on a single instruction**: the atomic
69
+ `fetch_add` in `parallel_for` that handed out one item at a time. When an item is one read's tag
70
+ scan, sixteen cores serialise on one cache line and the counter costs more than the work it
71
+ distributes. Items are claimed in batches now, sized so each worker takes ~8 turns and capped so a
72
+ ten-million-item scan still hands out ten thousand batches. Never: the batch collapses to 1 when
73
+ there are few items, which is the uneven case -- one bucket per item, one holding ten molecules and
74
+ the next ten million -- that the counter exists for.
75
+
76
+ Two serial blocks fell out with it, both read-only `3L`-binary-search scans of the barcode table:
77
+ the **distance-1 census** in `estimate_umi_error` (checkout's per-sample statistics tail) and
78
+ refine's **residual-FDR scan**, which was 0.53 s of a 2.17 s run on one core after everything around
79
+ it had been parallelised. Both tally integers into per-worker counters summed afterwards, so `-t`
80
+ still changes nothing but the clock -- verified byte-for-byte at `-t` 1/3/16 on a clean library and
81
+ on one with injected barcode errors (residual 1,294 either way), and under the thread sanitizer
82
+ (104 cases, 224,116 assertions, no race).
83
+
84
+ Two allocation fixes on paths every read takes: refine's rewrite chunk is now **assigned into
85
+ rather than cleared** -- the same trade assemble's partition already made, since `clear()` destroys
86
+ four `std::string` per record -- and the rewrite's comment buffer and unpacked barcode are worker
87
+ scratch reused read after read, with the cell and UMI halves taken as `string_view` instead of
88
+ `substr`. That is four allocations a read removed; `rewrite_seconds` fell 0.51 s to 0.38 s.
89
+ `assemble` also unpacked each group's UMI twice, once for the composition tally and once for the
90
+ record name.
91
+
92
+ | stage | 1 thread | 16 threads | was, at 16 |
93
+ |---|---|---|---|
94
+ | `checkout` | 213,880 | **1,548,835** | 1,056,472 |
95
+ | `refine` | 617,802 | **1,554,156** | 1,012,368 |
96
+ | `assemble` | 554,106 | **2,470,928** | 2,051,937 |
97
+
98
+ reads/s, 500 k-read sample except checkout at 2 M; `assets/benchmark_threads.tsv` is regenerated
99
+ and the figure redrawn from it. checkout's gap between end-to-end and matching throughput fell from
100
+ 20% to 9%.
101
+
102
+ Note: re-measured while it was in hand, a **64 k chunk in assemble's partition is still 32% faster**
103
+ (3,075,506 reads/s against 2,324,403 on 4 M reads) and is still not taken: it costs 16 MB of
104
+ resident chunk, which at NovaSeq scale makes pass 1 the memory peak and breaks the property that a
105
+ finer partition costs less rather than more. The benchmark tier does not fail at 64 k on a 500 k
106
+ corpus -- the objection is one of scale -- so the number is written down next to the constant.
107
+
108
+ ## 2.1.0 — 2026-08-13
109
+
110
+ First non-alpha of the rewrite. The three stages, the eight commands and the on-disk formats have
111
+ been stable across a3 and a4; what this release adds is the QC layer catching up with them.
112
+
113
+ **Four figures you already know how to read.** They come off new tables rather than out of a new
114
+ computation, so each is still redrawable from a committed TSV:
115
+
116
+ - **The barcode rank plot**, on [Cell Ranger's](https://www.10xgenomics.com/support/software/cell-ranger/latest/advanced/cr-ab-barcode-rank-plot)
117
+ axes, because it is the figure every user of a droplet protocol has seen: barcodes sorted by
118
+ content, log-log, knee where cells stop. Never: the y axis is **unique UMIs, never reads** — one
119
+ over-amplified molecule would otherwise put an empty droplet high on the curve, which is the
120
+ exact artefact the plot exists to show. The call is drawn on the curve, not described in a
121
+ caption. New table `<sample>.cell_rank.tsv`.
122
+ - **The MIG size spectrum**, molecules *and* the reads they account for, against `log(1 + size)`.
123
+ Both series, because they peak in different places the moment a library is over-sequenced: most
124
+ molecules are shallow, most reads are in the deep ones, and a figure with only one of them says
125
+ the opposite of a figure with only the other. `log1p`, so a molecule seen once has a place on
126
+ the axis.
127
+ - **The rank/Zipf curve**, molecule size against rank on log-log. Never: this is why the new
128
+ `<sample>.sizes.tsv` is written at **exact** sizes and not power-of-two bins — four bins make
129
+ four steps, and a straight line cannot be told from a bent one. It costs one row per distinct
130
+ depth, a few thousand on a real library, not one row per molecule.
131
+ - **Unique UMIs and reads per sample barcode**, off `checkout.summary.tsv`. The multiplexed
132
+ analogue of the same question.
133
+
134
+ **Consensus quality is a box, not a thinned scatter.** Emitted quality is discrete and capped at
135
+ the RT floor, so at any real depth every molecule sits on one or two integers: a cloud of dots
136
+ draws that as a flat line whether the bin holds ten molecules or ten million, and the `every 17`
137
+ that kept the SVG small threw away the tails that were the only thing the cloud could have shown.
138
+ `assemble` now accumulates the exact joint distribution of (depth bin, rounded Phred) — both are
139
+ small integers, so it is 61 counters per bin per bucket — and `assemble.quality_by_depth.tsv`
140
+ carries real order statistics over every molecule.
141
+
142
+ **Publication defaults on every panel.** Transparent background, so one SVG serves a light README,
143
+ a dark README and print. One ink colour (`#808080`), which reads on both. **The legend is inside
144
+ the plot box**: a key in the margin makes every figure wider than its data and is the first thing a
145
+ journal asks you to move. Frame is 760x520 rather than 900x560 wide.
146
+
147
+ **The pipeline figure is page-shaped.** `rankdir = TB` and three `rank = same` groups pinning each
148
+ side tool level with the data it reads; it was a 5:1 strip that filled the README column with air.
149
+ Transparent, and `minibwa` is in the downstream box.
150
+
151
+ Twenty panels now, from sixteen. `docs/formats.rst` documents every QC table and its columns.
152
+
7
153
  ## 2.0.0a4 — 2026-08-13
8
154
 
9
155
  **The partition threads, and the memory estimate behind it was wrong.** With the consensus already
@@ -261,6 +261,26 @@ correction is written up in `project/review-algorithms.md`.
261
261
  is where the answer matters most. It caught me twice: once in the correction posterior, once in
262
262
  the residual-FDR estimator. Both now use payload agreement as well. On a 1.23 reads/barcode
263
263
  library the count-only residual is 0 and the full one is 1,294.
264
+ - **The barcode error rate is measured at every DEPTH, not once for the library.**
265
+ `<sample>.umi_errors.tsv`, one row per exact parent depth: a parent carrying `c` reads offered
266
+ `c*L` barcode bases to be miscalled, so `u(c) = 3L(1-exp(-c eps/3))` from the distinct children
267
+ and `r(c) = c L eps` from the reads in them invert to the same eps. Reported as
268
+ `error_at_depth` (parents seen >= 10 times, where correction is near-complete), `error_phred`,
269
+ and `error_from_children` (all depths, a LOWER bound). On 1e-3 injected: distance-1 9.73e-04,
270
+ children 9.89e-04 = Q30, all depths 9.98e-04.
271
+ - **Never: neither error estimator is saturation-free — both are bounded by the merges correction
272
+ MADE.** As a fraction of an injected truth: distance-1 0.97 / 0.96 / 0.76 / 0.45 / 0.001 and
273
+ children 0.99 / 0.95 / 0.88 / 0.62 / 0.00 at occupancy 0.2% / 2.3% / 9.8% / 33% / 100%. The
274
+ children estimate wins everywhere either works; at 100% BOTH are zero, because `correct_umis`
275
+ refuses to merge on a full space and is right to. `saturated` is the flag that says the answer
276
+ is a floor -- do not read the table instead of it. `docs/umi_errors.rst`,
277
+ `tests/synthetic/test_umi_errors.py`.
278
+ - **Never: a sparse spectrum is POINTS, never a line.** `mig_size_spectrum` drew "reads in them"
279
+ `with lines` over a table at EXACT sizes, where past the head almost every size holds one
280
+ molecule -- so reads == size and the line drew the `y = x` diagonal as the figure's most
281
+ prominent feature, a tautology that reads as a second mode. Two or three molecules on a size made
282
+ it sawtooth between `size*1` and `size*2`, and it bridged gaps where nothing was observed. Same
283
+ rule for the two `umi_error_*` panels.
264
284
  - **The MIG-size FDR threshold is reported, never applied** (2026-08-13). Measured residual, not
265
285
  derived. 5.25% of 1-read molecules at 1.23 reads/barcode, 0% at 4.62.
266
286
  - **Dual-end barcodes work (2026-08-13)**: column 3 of the sheet is the slave pattern, on the
@@ -311,7 +331,24 @@ correction is written up in `project/review-algorithms.md`.
311
331
  moves the UMI into the FASTQ header and SRA rewrites headers. Confirmed on three runs by the
312
332
  missing template-switch `GGG`. `suggest` reports it correctly. The `smarter-umi` preset is
313
333
  sourced from the pipeline, not fitted to the data. In `SOURCES.md`.
314
- - **`migec plot` works (2026-08-13, 2.0.0a3)**: sixteen gnuplot panels over the TSVs the stages
334
+ - **The four familiar QC figures are in (2.1.0)**: Cell Ranger's barcode rank plot
335
+ (`<sample>.cell_rank.tsv`), the MIG size spectrum on log1p with molecules AND reads, the
336
+ rank/Zipf curve, and unique UMIs per sample barcode. Never: the knee plot's y axis is **unique
337
+ UMIs, never reads** -- one over-amplified molecule otherwise puts an empty droplet high on the
338
+ curve, which is the artefact the plot exists to show. Never: `<sample>.sizes.tsv` is at EXACT
339
+ sizes, never power-of-two bins -- the Zipf curve is its cumulative count and four bins make four
340
+ steps.
341
+ - **Never: a thinned scatter is not a distribution.** `consensus_quality` drew `every 17` molecules
342
+ as grey dots. Emitted quality is DISCRETE and capped at the floor, so at any real depth every
343
+ molecule sits on one or two integers: the cloud drew a flat line whether the bin held ten
344
+ molecules or ten million, and the thinning threw away the tails that were the only thing it could
345
+ have shown the line did not. `assemble` accumulates the exact (depth bin, rounded Phred) grid --
346
+ 61 counters per bin -- and the panel is a candlestick over real order statistics.
347
+ - **Publication defaults on every panel (2.1.0)**: transparent background so one SVG serves a light
348
+ README, a dark README and print; one ink colour `#808080` that reads on both; **the key inside the
349
+ plot box**, because a legend gutter makes every figure wider than its data. Frame 760x520. The
350
+ pipeline figure is `rankdir = TB` with three `rank = same` groups, not a 5:1 strip.
351
+ - **`migec plot` works (2026-08-13, 2.0.0a3)**: twenty gnuplot panels over the TSVs the stages
315
352
  already write, ColorBrewer Dark2, A/C/G/T always the same four colours. Never: it computes nothing
316
353
  — a figure that cannot be redrawn from a committed table is a figure that will disagree with the
317
354
  report. gnuplot is **not** a Python dependency; without it the `.gp` scripts are still written,
@@ -337,11 +374,37 @@ correction is written up in `project/review-algorithms.md`.
337
374
  but the per-sample counters and the memory figure were one sample's. Fixed; the published table
338
375
  is re-measured and now comes from `scripts/benchmark_threads.py`, which writes the TSV the
339
376
  figure is drawn from.
340
- - **All three stages thread (2026-08-13, 2.0.0a3).** refine 222k -> 1.01M reads/s, assemble
341
- 203k -> 1.43M, checkout unchanged at 1.06M. Never: the FIRST fix was not a thread -- zlib level 6
377
+ - **All three stages thread (2026-08-13, 2.0.0a3-2.1.0).** refine 222k -> 1.01M reads/s, assemble
378
+ 203k -> 2.05M (its partition threaded in a4), checkout unchanged at 1.06M. Never: the FIRST fix was not a thread -- zlib level 6
342
379
  was 83% of refine's wall clock compressing an intermediate the next stage immediately
343
380
  decompresses. Level 1 is now the default for every stage's output and gave 3x on its own.
344
381
  Measure before parallelising.
382
+ - **Never: the thread HELPER was the bottleneck once everything else was threaded (unreleased).**
383
+ `parallel_for` claimed one item per atomic `fetch_add`, and a sampling profile of assemble put
384
+ **21% of all CPU samples across every thread on that one instruction** -- more than the parse it
385
+ was handing out. Sixteen cores serialise on one cache line. Batched now: ~8 turns per worker,
386
+ capped at 1024, collapsing to 1 when items are few, which is the uneven case (one bucket per
387
+ item) the counter exists for. Never: profile the helper, not only the work. Current numbers,
388
+ 1 thread -> 16: checkout **213,880 -> 1,548,835** end to end (1,697,313 matching), refine
389
+ **617,802 -> 1,554,156**, assemble **554,106 -> 2,470,928**; `assets/benchmark_threads.tsv` is
390
+ the committed table and the figure is drawn from it.
391
+ - **Two serial scans went with it**: the distance-1 census in `estimate_umi_error` (which IS
392
+ checkout's per-sample statistics tail -- the end-to-end/matching gap fell from 20% to 9%) and
393
+ refine's residual-FDR scan, 0.53 s of a 2.17 s run on one core. Both are read-only `3L` binary
394
+ searches per barcode tallying integers, so per-worker counters summed afterwards keep `-t`
395
+ changing nothing but the clock. `estimate_umi_error` now takes a `threads` argument, defaulted
396
+ to 1 so the C++ test calls are unchanged.
397
+ - **Note: a 64 k chunk in assemble's partition is STILL 32% faster and still not taken**
398
+ (3,075,506 reads/s against 2,324,403 on 4 M, re-measured after the batching). It costs 16 MB of
399
+ resident chunk, which at NovaSeq scale makes pass 1 the memory peak. Never: the benchmark tier
400
+ does NOT fail at 64 k on a 500 k corpus -- the objection is scale, which no test here can see,
401
+ so the number lives next to the constant in `assemble.cpp`. Upgrade path is a persistent worker
402
+ pool, not a bigger chunk.
403
+ - **Never: `clear()` on a chunk of `FastqOwned` is four allocations a record.** assemble's
404
+ partition knew this; refine's rewrite still did it. Assigned into now, and the rewrite's comment
405
+ buffer and unpacked barcode are worker scratch with the cell/UMI halves as `string_view` rather
406
+ than `substr` -- `rewrite_seconds` 0.51 -> 0.38. Same class of bug in `assemble`: each group's
407
+ UMI was unpacked twice, once for the composition tally and once for the record name.
345
408
  - **Never: `-t` changes nothing but the wall clock, on every stage.** refine parallelises the
346
409
  neighbourhood SCAN (a pure function of the barcode table -- it reads no union-find state) and
347
410
  applies the merges serially in the original order, so the result is identical, not equivalent.
@@ -390,8 +453,22 @@ correction is written up in `project/review-algorithms.md`.
390
453
  - **Never: `containsKey`, not `?:`, for a nextflow boolean.** Groovy's elvis treats `false` as
391
454
  absent, so a per-sample `contig: false` against `params.migec_contig = true` silently meant its
392
455
  opposite -- the one direction a per-sample override exists to make possible.
393
- - Next: M3's remainder (the template's own error split), then the M2 remainder
394
- (`.mig` bucket output from checkout, i7xi5, bit-parallel matcher), then `--rt-error auto`.
456
+ - **Next, in order. `ROADMAP.md` has the same list with the reasoning; this is the short form.**
457
+ 1. **`.mig` bucket output from `checkout`** — the only unbounded allocation left. The UMI
458
+ counters are 8.8 GB at NovaSeq scale held in one piece; checkout warns past 1 GB and a warning
459
+ is not a fix.
460
+ 2. **Bucketed correction in `refine`**, two passes with the key rotated. Same problem one stage
461
+ on, and the fix is understood rather than still to be designed.
462
+ 3. **The template's own error split** — the pattern's constant bases calibrate the PRIMER, not
463
+ the polymerase, so the RT/PCR separation needs a different standard.
464
+ 4. **`--rt-error auto`**, which falls out of 3 and not before it.
465
+ 5. **i7xi5 contingency table** — needs nothing that is not already in the headers.
466
+ 6. **`2026-migec-benchmark`** and the published comparisons. This is what the version number is
467
+ waiting on, not the code.
468
+ 7. **ctDNA ground truth** (COSMIC at 0.005-0.075 VAF). Must be built: Maruzani's deposited runs
469
+ carry no UMI.
470
+ 8. **R1/R2 overlap merge**, as a case of `--contig`'s placement and never a second matcher.
471
+ 9. **Bit-parallel matcher**, last and deliberately: the scan is not the bottleneck.
395
472
  - **Note: Britanova et al aging (bulk TCR, shallow) lives on aldan3** and is the real dataset for the
396
473
  1-3 read regime. Not pulled yet. aldan3 compute goes through SLURM, never the frontend.
397
474
  - The archive is pushed: `legacy-v1` + tag `v1-final`, and master is the rewrite. Recovery point
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.2
2
2
  Name: migec
3
- Version: 2.0.0a4
3
+ Version: 2.2.0
4
4
  Summary: UMI barcode extraction, correction and consensus assembly for barcoded sequencing
5
5
  Keywords: umi,barcode,consensus,airr,repseq,single-cell,sequencing
6
6
  Author-Email: Mikhail Shugay <mikhail.shugay@gmail.com>
@@ -76,7 +76,7 @@ possible. The difficulty is entirely in the details:
76
76
 
77
77
  ## Pipeline
78
78
 
79
- <p align="center"><img alt="the migec pipeline" src="assets/pipeline.svg" width="100%"></p>
79
+ <p align="center"><img alt="the migec pipeline" src="assets/pipeline.svg" width="42%"></p>
80
80
 
81
81
  Three stages, plus three tools that read no reads: `suggest` says where the barcode is,
82
82
  `subsample` cuts a fixture that is still a library, and `plot` draws every QC panel with gnuplot
@@ -197,8 +197,8 @@ reads 2,000,000
197
197
  unmatched 0 (0.0%)
198
198
  ambiguous 0 (0.0%)
199
199
 
200
- 2.2 s (903,599 reads/s) = 1.5 s matching on 8 threads + 0.7 s UMI statistics, serial
201
- peak RSS 131.0 MB of which UMI counters 21.2 MB
200
+ 1.6 s (1,243,801 reads/s) = 1.5 s matching on 8 threads + 0.1 s UMI statistics
201
+ peak RSS 136.0 MB of which UMI counters 11.5 MB
202
202
 
203
203
  sample reads UMIs reads/UMI UMI len eff len
204
204
  S1 500,000 125,000 4.00 12 12.00
@@ -265,7 +265,7 @@ migec assemble out/S1.fq.gz -o cons/ --contig # random-primed reads that til
265
265
  A molecule is **sample + cell barcode + UMI**, never the UMI alone — the same UMI in two cells is
266
266
  two molecules, and that is the design rather than a defect. Reads are range partitioned on the
267
267
  packed key into `.mig` buckets and one bucket is sorted at a time, so nothing scales with the
268
- library: 531,365 reads/s, and 121 MB at 16 buckets against 203 MB at one.
268
+ library: 2,423,777 reads/s, and 123 MB at 16 buckets against 203 MB at one.
269
269
 
270
270
  The per-column posterior is `LL[j][b] = Σ_i (r==b ? log(1−e) : log(e/3))`, and then the number that
271
271
  matters:
@@ -295,10 +295,8 @@ independently (1.54e-4 on SRR1763769), and the polymerase classes come from
295
295
  2.6e-6, Pwo 2.4e-6 per bp per duplication) — with the first cycle worth ~5x an ordinary one
296
296
  ([Shagin *et al.* 2017](https://doi.org/10.1038/s41598-017-02727-8)).
297
297
 
298
- Which is exactly what comes out, on a real HIV-1 Primer ID library: one read gives Q30, and the
299
- curve flattens at the floor rather than at the instrument.
300
-
301
- <p align="center"><img alt="consensus quality against depth" src="assets/consensus_quality.svg" width="80%"></p>
298
+ Which is exactly what comes out: one read carries the payload's own error and the curve flattens
299
+ at the floor rather than at the instrument — drawn as a box per depth bin, [further down](#it-draws-its-own-qc).
302
300
 
303
301
  Very deep barcodes are capped at **10,000 reads into the consensus** — past that the column
304
302
  posterior has long since saturated while the group still costs time and memory, which is 10x's
@@ -339,13 +337,71 @@ migec plot out/ # every panel whose table is in out/
339
337
  migec plot cons/ -o figs/ --format pdf
340
338
  ```
341
339
 
342
- Sixteen panels — UMI PWM and information content, barcode quality and its calibration, MIG size
343
- coverage, trimming, barcode space, the `suggest` cycle trace, overrepresented k-mers, the cell
344
- rank curve, consensus quality, error, layout, and thread scaling. Every one is a gnuplot script
345
- over a TSV a stage already wrote, so a figure can be redrawn from the table next to it long after
346
- the FASTQ is gone, and a figure can never disagree with the number in the report. gnuplot is not a
347
- Python dependency: without it the `.gp` scripts are still written. See
348
- [docs/plots.rst](docs/plots.rst).
340
+ Twenty panels. Every one is a gnuplot script over a TSV a stage already wrote, so a figure can be
341
+ redrawn from the table next to it long after the FASTQ is gone, and a figure can never disagree
342
+ with the number in the report. gnuplot is not a Python dependency: without it the `.gp` scripts are
343
+ still written. Every SVG is **transparent and mid-grey**, so one file serves a light README, a dark
344
+ README and a printed page — and the legend sits inside the plot box, not in a gutter that makes
345
+ every figure wider than its data. See [docs/plots.rst](docs/plots.rst).
346
+
347
+ Four of them are the ones you already know how to read.
348
+
349
+ **The barcode rank plot**, on Cell Ranger's axes, because that is the figure every user of a
350
+ droplet protocol has seen. Barcodes sorted by how many **distinct UMIs** they carry — never by
351
+ reads, since one over-amplified molecule would otherwise put an empty droplet high on the curve,
352
+ which is the artefact the plot exists to show. The call is drawn *on* the curve.
353
+
354
+ <p align="center"><img alt="barcode rank plot" src="assets/cell_rank.svg" width="72%"></p>
355
+
356
+ **The MIG size spectrum**, molecules and the reads they account for, on `log(1 + size)`. Both
357
+ series, on their own axes, because they peak in different places the moment a library is
358
+ over-sequenced: most *molecules* are shallow, most *reads* are in the deep ones. A figure with only
359
+ the first says the library is fine; a figure with only the second says it is saturated. `log1p`, so
360
+ a molecule seen once still has a place on the axis.
361
+
362
+ <p align="center"><img alt="MIG size spectrum" src="assets/mig_size_spectrum.svg" width="72%"></p>
363
+
364
+ **The rank/Zipf curve** — molecule size against rank, log-log. A straight line is Zipf, and
365
+ amplification bias bends it. This is why `refine` writes the size spectrum at *exact* sizes rather
366
+ than in power-of-two bins: four bins make four steps, and a straight line cannot be told from a
367
+ bent one.
368
+
369
+ <p align="center"><img alt="molecule size against rank" src="assets/mig_size_zipf.svg" width="72%"></p>
370
+
371
+ **Barcode errors against the parent's depth** — how many distinct error children a molecule
372
+ spawned, and how many reads were in them. A parent seen *c* times had *c·L* barcode bases to
373
+ miscall, so both series climb with *c* — but only one of them can climb forever. There are exactly
374
+ *3L* barcodes one substitution away, so the distinct-children series bends over at the dashed line
375
+ while the reads series carries straight past it. Where they part is where this library's barcode
376
+ neighbourhood filled up, measured rather than predicted.
377
+
378
+ <p align="center"><img alt="barcode errors against depth" src="assets/umi_error_children.svg" width="72%"></p>
379
+
380
+ Invert both and you get two estimates of the same barcode error rate, per depth, with what `refine`
381
+ reports drawn across them. The y axis is a log error rate, so one decade is exactly ten Phred and
382
+ 1e-3 is Q30 — which makes the number directly comparable with the barcode's own reported quality,
383
+ the only independent check on it there is. On a diverse library sequenced 25 deep it lands within
384
+ 1% of a known injected rate.
385
+
386
+ <p align="center"><img alt="barcode error rate by depth" src="assets/umi_error_rate.svg" width="72%"></p>
387
+
388
+ > **Never** read this instead of the `saturated` flag. Both series count only the children
389
+ > correction actually merged, and on a *full* barcode space `refine` refuses to merge — rightly,
390
+ > because a distance-1 neighbour there is more likely a real molecule than an error child — so both
391
+ > estimates collapse to zero. Against a known rate they are 0.99 and 0.97 of the truth at 0.2%
392
+ > occupancy, 0.62 and 0.45 at 33%, and nothing at 100%. See
393
+ > [docs/umi_errors.rst](docs/umi_errors.rst).
394
+
395
+ **Consensus quality against depth**, as a box, never a thinned scatter. Emitted quality is discrete
396
+ and capped at the floor, so at any real depth every molecule sits on one or two integers: a cloud
397
+ of dots draws that as a flat line whether the bin holds ten molecules or ten million, and thinning
398
+ it throws away the tails that were the only thing the cloud could have shown. These are exact order
399
+ statistics over every molecule, read off a (depth, quality) count grid.
400
+
401
+ <p align="center"><img alt="consensus quality against depth" src="assets/consensus_quality.svg" width="72%"></p>
402
+
403
+ `checkout.summary.tsv` carries unique UMIs and reads per **sample** barcode, drawn the same way by
404
+ the `sample_umis` panel — the multiplexed analogue of the same question.
349
405
 
350
406
  ### It finds what the trim left behind
351
407
 
@@ -476,7 +532,7 @@ warning: 79.4% of molecules were seen once. A consensus over one read is that re
476
532
  ```
477
533
 
478
534
  It is also the memory-hostile case, because distinct barcodes are what everything scales with, so
479
- it is what the benchmarks use: 190,595 reads/s at 1.02 reads/UMI, 259 B resident per distinct
535
+ it is what the benchmarks use: 1,179,549 reads/s at 1.02 reads/UMI, 282 B resident per distinct
480
536
  barcode, still bounded by the bucket rather than the library.
481
537
 
482
538
  ### Speed and memory are reported, not assumed
@@ -489,11 +545,12 @@ and nothing else, which is asserted per stage in C++, at the CLI, and under the
489
545
 
490
546
  | stage | 1 thread | 16 threads | bound by |
491
547
  |---|---|---|---|
492
- | `checkout` | 202,717 | **1,056,472** | reads |
493
- | `refine` | 605,611 | **1,012,368** | distinct barcodes |
494
- | `assemble` | 549,745 | **2,051,937** | reads, then the largest bucket |
548
+ | `checkout` | 213,880 | **1,548,835** | reads |
549
+ | `refine` | 617,802 | **1,554,156** | distinct barcodes |
550
+ | `assemble` | 554,106 | **2,470,928** | reads, then the largest bucket |
495
551
 
496
- reads/s. Two of those used to be 222,017 and 202,977, and the first fix was not a thread: **zlib at
552
+ reads/s — `refine` and `assemble` on a 500 k-read sample, `checkout` on 2 M. Two of those used to
553
+ be 222,017 and 202,977, and the first fix was not a thread: **zlib at
497
554
  its default level 6 was 83% of refine's wall clock**, compressing an intermediate the next stage
498
555
  decompresses immediately. Level 1 costs 21% more bytes and gave 3x before a single thread was
499
556
  added. Measure the stage before parallelising it.
@@ -511,20 +568,34 @@ malloc instead of inflate.
511
568
  | reads/s | 1,481,946 | **2,051,937** |
512
569
  | peak RSS | 1,479 MB | **789 MB** |
513
570
 
571
+ That is the record of *that* change. Batching the work-claiming atomic later took the same corpus
572
+ to **1.72 s and 2,324,403 reads/s**.
573
+
514
574
  The memory fell with it, because the estimate deciding how finely to cut the input said a gzipped
515
575
  FASTQ goes resident at 8x its on-disk size and it is really **19x** — a resident record is two heap
516
576
  `std::string` with their allocator headers, not the 180 bytes of payload. Guessing low picks too
517
577
  few buckets, and pass 2 holds sixteen of them at once.
518
578
 
579
+ And once every stage was threaded, the largest single cost in the whole pipeline was the
580
+ **thread helper itself**. `parallel_for` handed out one item per atomic `fetch_add`, and when an
581
+ item is one read's tag scan that instruction took **21% of all CPU samples across every thread** —
582
+ more than the work it was handing out. Sixteen cores serialise on one cache line. Items are claimed
583
+ in batches now, sized so each worker takes ~8 turns, collapsing to 1 when there are few items,
584
+ which is the uneven case (one bucket per item) the counter existed for. Two serial blocks went with
585
+ it: the distance-1 census inside `estimate_umi_error`, which is checkout's per-sample tail, and
586
+ refine's residual-FDR scan, which was 0.53 s of a 2.17 s run on one core after everything around it
587
+ had been parallelised. Both are read-only scans of the barcode table that tally integers, so they
588
+ thread exactly and `-t` still changes nothing but the clock.
589
+
519
590
  | threads | reads/s | matching reads/s | peak RSS |
520
591
  |---|---|---|---|
521
- | 1 | 202,717 | 217,506 | 60 MB |
522
- | 2 | 362,486 | 412,855 | 78 MB |
523
- | 4 | 603,773 | 758,801 | 97 MB |
524
- | 8 | 910,500 | 1,335,791 | 136 MB |
525
- | 16 | **1,056,472** | **1,684,654** | 217 MB |
592
+ | 1 | 213,880 | 216,584 | 59 MB |
593
+ | 2 | 394,471 | 403,393 | 79 MB |
594
+ | 4 | 737,777 | 768,810 | 101 MB |
595
+ | 8 | 1,256,838 | 1,349,533 | 139 MB |
596
+ | 16 | **1,548,835** | **1,697,313** | 215 MB |
526
597
 
527
- <p align="center"><img alt="checkout thread scaling" src="assets/benchmark_threads.svg" width="80%"></p>
598
+ <p align="center"><img alt="checkout thread scaling" src="assets/benchmark_threads.svg" width="72%"></p>
528
599
 
529
600
  2 M single-end 129 nt reads, four barcode patterns, 4 reads per molecule, M-series laptop.
530
601
  `python scripts/benchmark_threads.py --reads 2000000 -o assets/` writes that table, and the figure