stenograf 0.2.2__tar.gz → 0.3.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (234) hide show
  1. {stenograf-0.2.2 → stenograf-0.3.0}/PKG-INFO +31 -6
  2. {stenograf-0.2.2 → stenograf-0.3.0}/PLAN.md +148 -0
  3. {stenograf-0.2.2 → stenograf-0.3.0}/README.md +29 -5
  4. stenograf-0.3.0/eval/README.md +306 -0
  5. stenograf-0.3.0/eval/bias.py +858 -0
  6. stenograf-0.3.0/eval/bias_confidence.py +190 -0
  7. stenograf-0.3.0/eval/bias_data.py +230 -0
  8. stenograf-0.3.0/eval/bias_fluid.py +346 -0
  9. stenograf-0.3.0/eval/bias_score.py +452 -0
  10. stenograf-0.3.0/eval/bias_tts.py +314 -0
  11. {stenograf-0.2.2 → stenograf-0.3.0}/pyproject.toml +21 -1
  12. stenograf-0.3.0/src/stenograf/asr/biasing.py +368 -0
  13. stenograf-0.3.0/src/stenograf/asr/parakeet.py +255 -0
  14. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/asr/parakeet_onnx.py +100 -2
  15. stenograf-0.3.0/src/stenograf/asr/tokens.py +146 -0
  16. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/capture/linux.py +25 -5
  17. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/capture/macos.py +41 -8
  18. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/capture/streaming.py +38 -0
  19. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/capture/windows.py +11 -7
  20. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/cli/run.py +4 -3
  21. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/cli/start.py +42 -6
  22. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/cli/transcribe.py +8 -0
  23. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/config.py +3 -3
  24. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/glossary.py +76 -11
  25. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/loaders.py +81 -9
  26. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/settings.py +38 -4
  27. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/ui/flow.py +10 -1
  28. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/ui/setup.py +6 -6
  29. {stenograf-0.2.2 → stenograf-0.3.0}/tests/fake_parec.py +6 -1
  30. {stenograf-0.2.2 → stenograf-0.3.0}/tests/fake_stenocap.py +21 -3
  31. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_asr_tokens.py +15 -0
  32. stenograf-0.3.0/tests/test_biasing.py +157 -0
  33. stenograf-0.3.0/tests/test_biasing_loops.py +207 -0
  34. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_capture_linux.py +25 -0
  35. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_capture_macos.py +32 -0
  36. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_capture_windows.py +14 -0
  37. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_cli.py +106 -17
  38. stenograf-0.3.0/tests/test_eval_bias.py +312 -0
  39. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_glossary.py +60 -5
  40. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_settings.py +2 -0
  41. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_ui.py +6 -1
  42. {stenograf-0.2.2 → stenograf-0.3.0}/uv.lock +87 -33
  43. stenograf-0.2.2/eval/README.md +0 -142
  44. stenograf-0.2.2/src/stenograf/asr/parakeet.py +0 -128
  45. stenograf-0.2.2/src/stenograf/asr/tokens.py +0 -47
  46. {stenograf-0.2.2 → stenograf-0.3.0}/.claude/skills/verify/SKILL.md +0 -0
  47. {stenograf-0.2.2 → stenograf-0.3.0}/.github/workflows/ci.yml +0 -0
  48. {stenograf-0.2.2 → stenograf-0.3.0}/.github/workflows/release.yml +0 -0
  49. {stenograf-0.2.2 → stenograf-0.3.0}/.gitignore +0 -0
  50. {stenograf-0.2.2 → stenograf-0.3.0}/CLAUDE.md +0 -0
  51. {stenograf-0.2.2 → stenograf-0.3.0}/LICENSE +0 -0
  52. {stenograf-0.2.2 → stenograf-0.3.0}/PLAN-AEC.md +0 -0
  53. {stenograf-0.2.2 → stenograf-0.3.0}/PLAN-CLEANUP.md +0 -0
  54. {stenograf-0.2.2 → stenograf-0.3.0}/eval/adjudicate.py +0 -0
  55. {stenograf-0.2.2 → stenograf-0.3.0}/eval/aec_rig.py +0 -0
  56. {stenograf-0.2.2 → stenograf-0.3.0}/eval/aec_score.py +0 -0
  57. {stenograf-0.2.2 → stenograf-0.3.0}/eval/backends.py +0 -0
  58. {stenograf-0.2.2 → stenograf-0.3.0}/eval/common.py +0 -0
  59. {stenograf-0.2.2 → stenograf-0.3.0}/eval/der.py +0 -0
  60. {stenograf-0.2.2 → stenograf-0.3.0}/eval/diarize.py +0 -0
  61. {stenograf-0.2.2 → stenograf-0.3.0}/eval/extract.py +0 -0
  62. {stenograf-0.2.2 → stenograf-0.3.0}/eval/live.py +0 -0
  63. {stenograf-0.2.2 → stenograf-0.3.0}/eval/manifest.example.json +0 -0
  64. {stenograf-0.2.2 → stenograf-0.3.0}/eval/parity.py +0 -0
  65. {stenograf-0.2.2 → stenograf-0.3.0}/eval/rttm.py +0 -0
  66. {stenograf-0.2.2 → stenograf-0.3.0}/eval/scan_languages.py +0 -0
  67. {stenograf-0.2.2 → stenograf-0.3.0}/eval/score.py +0 -0
  68. {stenograf-0.2.2 → stenograf-0.3.0}/eval/transcribe.py +0 -0
  69. {stenograf-0.2.2 → stenograf-0.3.0}/hatch_build.py +0 -0
  70. {stenograf-0.2.2 → stenograf-0.3.0}/install.ps1 +0 -0
  71. {stenograf-0.2.2 → stenograf-0.3.0}/install.sh +0 -0
  72. {stenograf-0.2.2 → stenograf-0.3.0}/native/README.md +0 -0
  73. {stenograf-0.2.2 → stenograf-0.3.0}/native/helper/Info.plist +0 -0
  74. {stenograf-0.2.2 → stenograf-0.3.0}/native/helper/build.sh +0 -0
  75. {stenograf-0.2.2 → stenograf-0.3.0}/native/helper/main.swift +0 -0
  76. {stenograf-0.2.2 → stenograf-0.3.0}/native/spike/Info.plist +0 -0
  77. {stenograf-0.2.2 → stenograf-0.3.0}/native/spike/build.sh +0 -0
  78. {stenograf-0.2.2 → stenograf-0.3.0}/native/spike/main.swift +0 -0
  79. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/.gitignore +0 -0
  80. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/Cargo.lock +0 -0
  81. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/Cargo.toml +0 -0
  82. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/build.ps1 +0 -0
  83. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/build.sh +0 -0
  84. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/src/main.rs +0 -0
  85. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/VENDOR.md +0 -0
  86. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/Cargo.toml +0 -0
  87. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/LICENSE +0 -0
  88. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/README.md +0 -0
  89. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/binarize.rs +0 -0
  90. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/clustering/ahc.rs +0 -0
  91. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/clustering/plda.rs +0 -0
  92. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/clustering/vbx.rs +0 -0
  93. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/clustering.rs +0 -0
  94. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/inference/coreml/array.rs +0 -0
  95. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/inference/coreml/path.rs +0 -0
  96. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/inference/coreml/runtime.rs +0 -0
  97. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/inference/coreml.rs +0 -0
  98. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/inference/embedding/batch.rs +0 -0
  99. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/inference/embedding/chunk.rs +0 -0
  100. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/inference/embedding/fbank.rs +0 -0
  101. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/inference/embedding/load/sessions.rs +0 -0
  102. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/inference/embedding/load.rs +0 -0
  103. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/inference/embedding/native/loaders.rs +0 -0
  104. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/inference/embedding/native.rs +0 -0
  105. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/inference/embedding/paths.rs +0 -0
  106. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/inference/embedding/run.rs +0 -0
  107. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/inference/embedding/session.rs +0 -0
  108. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/inference/embedding/tail.rs +0 -0
  109. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/inference/embedding/tensor.rs +0 -0
  110. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/inference/embedding.rs +0 -0
  111. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/inference/segmentation/native.rs +0 -0
  112. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/inference/segmentation/parallel/batch.rs +0 -0
  113. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/inference/segmentation/parallel/single.rs +0 -0
  114. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/inference/segmentation/parallel.rs +0 -0
  115. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/inference/segmentation/run.rs +0 -0
  116. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/inference/segmentation/tensor.rs +0 -0
  117. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/inference/segmentation.rs +0 -0
  118. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/inference.rs +0 -0
  119. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/lib.rs +0 -0
  120. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/linalg.rs +0 -0
  121. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/metrics.rs +0 -0
  122. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/models.rs +0 -0
  123. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/pipeline/builder.rs +0 -0
  124. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/pipeline/chunk_embedding/collect.rs +0 -0
  125. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/pipeline/chunk_embedding/error.rs +0 -0
  126. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/pipeline/chunk_embedding/gpu.rs +0 -0
  127. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/pipeline/chunk_embedding/orchestrate.rs +0 -0
  128. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/pipeline/chunk_embedding/prep.rs +0 -0
  129. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/pipeline/chunk_embedding.rs +0 -0
  130. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/pipeline/clustering.rs +0 -0
  131. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/pipeline/concurrent.rs +0 -0
  132. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/pipeline/config.rs +0 -0
  133. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/pipeline/post_inference.rs +0 -0
  134. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/pipeline/queued.rs +0 -0
  135. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/pipeline/tests.rs +0 -0
  136. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/pipeline/types/data.rs +0 -0
  137. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/pipeline/types/error.rs +0 -0
  138. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/pipeline/types/extract.rs +0 -0
  139. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/pipeline/types/layout.rs +0 -0
  140. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/pipeline/types.rs +0 -0
  141. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/pipeline.rs +0 -0
  142. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/powerset.rs +0 -0
  143. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/reconstruct.rs +0 -0
  144. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/segment.rs +0 -0
  145. {stenograf-0.2.2 → stenograf-0.3.0}/native/stenodiar/vendor/speakrs/src/utils.rs +0 -0
  146. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/__init__.py +0 -0
  147. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/__main__.py +0 -0
  148. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/aec.py +0 -0
  149. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/asr/__init__.py +0 -0
  150. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/asr/base.py +0 -0
  151. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/asr/providers.py +0 -0
  152. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/asr/registry.py +0 -0
  153. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/audio.py +0 -0
  154. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/capture/__init__.py +0 -0
  155. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/capture/base.py +0 -0
  156. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/capture/file.py +0 -0
  157. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/cli/__init__.py +0 -0
  158. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/cli/doctor_cmd.py +0 -0
  159. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/cli/format.py +0 -0
  160. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/cli/notes.py +0 -0
  161. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/cli/profiles.py +0 -0
  162. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/cli/settings_cmd.py +0 -0
  163. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/diarization/__init__.py +0 -0
  164. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/diarization/base.py +0 -0
  165. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/diarization/sherpa.py +0 -0
  166. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/diarization/speakrs.py +0 -0
  167. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/doctor.py +0 -0
  168. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/lid.py +0 -0
  169. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/live.py +0 -0
  170. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/models.py +0 -0
  171. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/notes/__init__.py +0 -0
  172. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/notes/backend.py +0 -0
  173. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/notes/command.py +0 -0
  174. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/notes/export.py +0 -0
  175. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/notes/generate.py +0 -0
  176. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/notes/mlx.py +0 -0
  177. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/notes/model.py +0 -0
  178. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/notes/ollama.py +0 -0
  179. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/notes/prompt.py +0 -0
  180. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/output.py +0 -0
  181. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/pipeline.py +0 -0
  182. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/profiles.py +0 -0
  183. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/recording.py +0 -0
  184. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/session.py +0 -0
  185. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/shortcut.py +0 -0
  186. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/transcript.py +0 -0
  187. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/ui/__init__.py +0 -0
  188. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/ui/_fps.py +0 -0
  189. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/ui/app.py +0 -0
  190. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/ui/doctor.py +0 -0
  191. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/ui/home.py +0 -0
  192. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/ui/meeting.py +0 -0
  193. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/ui/notes.py +0 -0
  194. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/ui/settings.py +0 -0
  195. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/ui/transcribe.py +0 -0
  196. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/ui/widgets.py +0 -0
  197. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/vad.py +0 -0
  198. {stenograf-0.2.2 → stenograf-0.3.0}/src/stenograf/view.py +0 -0
  199. {stenograf-0.2.2 → stenograf-0.3.0}/tests/conftest.py +0 -0
  200. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_aec.py +0 -0
  201. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_asr_parakeet.py +0 -0
  202. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_asr_parakeet_onnx.py +0 -0
  203. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_asr_providers.py +0 -0
  204. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_asr_registry.py +0 -0
  205. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_audio.py +0 -0
  206. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_build_hook.py +0 -0
  207. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_capture_file.py +0 -0
  208. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_capture_streaming.py +0 -0
  209. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_cli_notes.py +0 -0
  210. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_config.py +0 -0
  211. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_diarization_sherpa.py +0 -0
  212. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_diarization_sherpa_unit.py +0 -0
  213. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_diarization_speakrs.py +0 -0
  214. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_doctor.py +0 -0
  215. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_echo_dedup.py +0 -0
  216. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_eval_der.py +0 -0
  217. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_lid.py +0 -0
  218. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_live.py +0 -0
  219. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_live_orchestration.py +0 -0
  220. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_meeting.py +0 -0
  221. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_models.py +0 -0
  222. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_notes_backends.py +0 -0
  223. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_notes_export.py +0 -0
  224. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_notes_generate.py +0 -0
  225. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_notes_model.py +0 -0
  226. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_output.py +0 -0
  227. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_pipeline.py +0 -0
  228. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_profiles.py +0 -0
  229. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_recording.py +0 -0
  230. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_session.py +0 -0
  231. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_shortcut.py +0 -0
  232. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_transcript.py +0 -0
  233. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_vad.py +0 -0
  234. {stenograf-0.2.2 → stenograf-0.3.0}/tests/test_view.py +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: stenograf
3
- Version: 0.2.2
3
+ Version: 0.3.0
4
4
  Summary: Accuracy-first local meeting transcription for German and English. Audio stays in RAM; only text is written.
5
5
  Project-URL: Repository, https://github.com/daniel-om-weber/stenograf
6
6
  Project-URL: Issues, https://github.com/daniel-om-weber/stenograf/issues
@@ -26,6 +26,7 @@ Requires-Dist: onnx-asr[hub]>=0.11
26
26
  Requires-Dist: onnxruntime-directml>=1.24; sys_platform == 'win32'
27
27
  Requires-Dist: onnxruntime>=1.18.1; sys_platform != 'win32'
28
28
  Requires-Dist: parakeet-mlx>=0.5.2; sys_platform == 'darwin' and platform_machine == 'arm64'
29
+ Requires-Dist: sentencepiece>=0.2
29
30
  Requires-Dist: sherpa-onnx-core<1.13,>=1.12.40
30
31
  Requires-Dist: sherpa-onnx<1.13,>=1.12.40
31
32
  Requires-Dist: soundcard>=0.4.4; sys_platform == 'win32'
@@ -287,16 +288,40 @@ dir (not the model cache) and are never uploaded.
287
288
 
288
289
  ## Vocabulary
289
290
 
290
- Domain terms and attendee names are corrected in the finalized transcript
291
- (the ASR has no decode-time biasing, so this is a post-correction pass):
291
+ Domain terms and attendee names steer the transcript twice: they boost the
292
+ decoder toward those spellings *while* it transcribes, and then correct the
293
+ near-misses it still got wrong in the finalized text.
292
294
 
293
295
  ```sh
294
296
  steno transcribe rec.mov --attendee "Anja Müller" --glossary Kubernetes,gRPC
295
297
  steno transcribe rec.mov --glossary-file terms.txt
296
298
  ```
297
299
 
298
- A term and its transcription must share a word count — `gRPC` can fix `G R P C`
299
- spoken as one word, but not a term split across word boundaries.
300
+ Write a term the way it appears in a sentence. A capitalized term (`Kubernetes`,
301
+ `iOS`) is imposed verbatim; an all-lowercase one asserts the spelling but leaves
302
+ the model's capitalization alone. In the correction pass a term and its
303
+ transcription must share a word count — `gRPC` can fix `G R P C` spoken as one
304
+ word, but not a term split across word boundaries.
305
+
306
+ **An acronym you pronounce as a word needs both spellings.** Biasing rewards the
307
+ exact token path you write, so `NIRS` only ever rewards `N-I-R-S` — a spelling
308
+ the acoustics never take when you say "nirs". List `NIRS` *and* `Nirs`: the
309
+ decoder can reach the second, and the correction pass snaps it back to the first,
310
+ since the two normalize to the same word. Only case variants of the same letters
311
+ work — a respelling (`Ekmo` for `ECMO`) is reachable but never snaps back.
312
+
313
+ Both layers are on whenever there are terms. `[asr] boost` scales the decoder
314
+ biasing (default 1.0; `0` turns it off, and much above ~3 it starts rewriting
315
+ words that were never in your list), and `[vocab] glossary_threshold` is the
316
+ similarity a word must reach before the correction pass touches it (default
317
+ 0.95 — deliberately strict, because a loose threshold corrupts words the model
318
+ already had right).
319
+
320
+ Reach for the spellings before the knobs. On an 85-minute German meeting, adding
321
+ the pronounceable twins fixed three more terms for four extra changed words in
322
+ the rest of the transcript, where raising `boost` to 1.5 bought two and disturbed
323
+ forty-six — at 2.0 the correct compound "Medizintechnikgruppe" decayed into
324
+ "Medizinischechnikgruppe", pulled apart by a listed term it half-matched.
300
325
 
301
326
  ## Settings
302
327
 
@@ -321,7 +346,7 @@ formats = ["md", "json", "txt"] # default --format list (srt/vtt for subtitles
321
346
  [vocab] # standing vocabulary — MERGED with per-run
322
347
  glossary_file = "~/steno/glossary.txt" # --glossary/--attendee flags
323
348
  attendees = ["Anja Müller"]
324
- glossary_threshold = 0.82
349
+ glossary_threshold = 0.95
325
350
 
326
351
  [output]
327
352
  dir = "~/Documents/Meetings" # where meeting folders are created
@@ -1284,6 +1284,154 @@ rest. Two naming landmines for future screens: never define `_running` or
1284
1284
  `_render` on a Screen — both shadow Textual internals (`MessagePump._running`,
1285
1285
  `Widget._render`) and break the screen silently.
1286
1286
 
1287
+ ### Contextual-biasing evaluation harness (built 2026-07-13)
1288
+
1289
+ Decode-time biasing (`stenograf/asr/biasing.py`, a from-scratch port of NVIDIA's
1290
+ TurboBias/GPU-PB) shipped with a tree verified against NeMo's golden vectors but
1291
+ an *effect* resting on one TTS clip and three meeting WAVs. That is enough to
1292
+ prove the mechanism fires and not enough to set `[asr] boost` or to defend the
1293
+ two places we deliberately diverge from NeMo. `eval/bias*.py` replaces the
1294
+ anecdotes with numbers, with **zero hand labeling** — every reference and word
1295
+ list comes from corpora that already ship them. Usage in `eval/README.md`.
1296
+
1297
+ **The harness.** `bias_score.py` is a pure scorer (B-WER / U-WER / entity
1298
+ recall+precision / false insertions / surface damage), a faithful port of is21's
1299
+ own alignment — which turns the **44 hypothesis+result file pairs they publish
1300
+ into a free correctness oracle**: `tests/test_eval_bias.py` reproduces every one
1301
+ of them to the digit (skips until `bias_data.py --fetch is21` has run).
1302
+ `bias_data.py` downloads the English lists and **derives the German benchmark**
1303
+ (none exists) from MLS's free train transcripts by the identical recipe — 3,394
1304
+ utts, a 299,699-word distractor pool, 16.4 % rare-token density (vs 11 % in
1305
+ English: German compounding fattens the tail, which makes it a *better* biasing
1306
+ benchmark). `bias.py` drives the real shipped backend through `create_backend`.
1307
+
1308
+ **What it found** (500-utt pinned subsample, `parakeet` MLX, N=100 unless noted):
1309
+
1310
+ - **English correctness gate — PASS.** Baseline B-WER 12.73 % vs U-WER 1.03 %
1311
+ (rare words 12× harder); biasing → **B-WER −34.9 % relative, U-WER exactly
1312
+ flat**. The is21 authors report ~30 % relative with U-WER flat, so the port
1313
+ reproduces the literature on a far stronger model. B-WER also degrades
1314
+ monotonically with list size (12.22 → 12.84 % for N=100 → 2000), which is the
1315
+ bug detector: it proves the distractors are really in the tree and competing.
1316
+ - **`boost = 1.0` CONFIRMED as the default.** German B-WER by boost: 0.5 →
1317
+ −16.5 %, **1.0 → −27.0 % (U-WER −3.2 %, i.e. it *improves*)**, 2.0 → −39.4 %
1318
+ but U-WER **+6.7 %** and false insertions 3 → 27, 3.0 → no further B-WER gain
1319
+ (−39.2 %) with U-WER **+38.4 %** and 105 false insertions. The knee is exactly
1320
+ between 1 and 2; 3 is the catastrophe NVIDIA warns about, measured.
1321
+ - **`unk_score = 1.0` (our divergence from NeMo's code) CONFIRMED.** NeMo's
1322
+ `unk_score=0` wins on B-WER (−33.7 % vs −27.0 %) but rewrites **33 words'
1323
+ surface forms vs our 6** — and the rewrites are exactly the predicted failure:
1324
+ spurious capitalization (`ihrer`→`Ihrer`, `war`→`War`, `fertig`→`Fertig`),
1325
+ because with `unk_score < context_score` the root's arcs stop being uniform and
1326
+ the tree yanks the decoder toward glossary-ish spellings at *every word start*.
1327
+ The `DEFAULT_UNK_SCORE` docstring predicted this mechanism; the sweep measured it.
1328
+ - **Compound-tail tokenization — NULL on this benchmark, and that is a fact about
1329
+ the corpus, not the feature.** Identical B-WER and recall with it on and off
1330
+ (12.22 %, 87.8 %). MLS German is 19th-century audiobook prose, where a glossary
1331
+ term almost never appears *inside* a compound — the case the feature exists for.
1332
+ It is confirmed elsewhere (the real-model measurement that motivated it, and the
1333
+ TTS reachability tier) and shown here to do **no harm**. Tier 2 is simply the
1334
+ wrong corpus to price it; a German *technical-speech* set would be the right one,
1335
+ and none exists.
1336
+ - **The post-correction layer was over-correcting, and `glossary_threshold` moved
1337
+ 0.82 → 0.95** (`--post`, added 2026-07-13; costs no decoding — post-correction is
1338
+ a pure text transform over the other arms' cached hypotheses). We ship *two*
1339
+ glossary layers and had measured only one. At 0.82, post-correction posts a B-WER
1340
+ drop twice biasing's (−52.8 % German, −55.5 % English) **and damages the
1341
+ transcript**: U-WER +9.9 % German / **+86.3 % English**, 85 false insertions vs
1342
+ biasing's 3. The shipped stack (both layers) was worse than the `boost = 2.0`
1343
+ config we had already rejected for exactly this failure (+6.5 % vs +6.7 % U-WER;
1344
+ 84 vs 27 insertions). Fuzzy matching answers to no acoustics, so it cannot
1345
+ distinguish a misrecognition from a correct word that merely looks like a term —
1346
+ and **B-WER alone cannot see the difference**. At 0.95 the layer is a free win:
1347
+ B-WER −30.3 % German / −36.3 % English (*better* than biasing alone: −27.0 % /
1348
+ −34.9 %), U-WER flat or improving, false insertions at biasing's own level (4/3),
1349
+ and zero added insertions on real meeting audio at a realistic 30-term glossary.
1350
+ The surface-damage column then caught a **second** bug, and was the only metric
1351
+ that could: it sat at ~1190 for the post arms at *every* threshold, and a quantity
1352
+ that ignores the knob gating it is not being produced by that knob. The word lists
1353
+ are 100 % lowercase and `glossary.py` imposed a term's spelling verbatim, so it
1354
+ de-capitalized German nouns wholesale. It now applies `biasing.surface_forms`' rule
1355
+ — capitals in a term are a deliberate spelling and win; an all-lowercase term
1356
+ asserts nothing about case and leaves the model's alone (`_recase`). The column
1357
+ fell **1190 → 2 with B-WER, U-WER and false insertions bit-identical**: every
1358
+ WER-shaped metric normalizes case away and would have shipped it.
1359
+ - **The fuzzy layer is not redundant with a better-tuned decoder — it is the
1360
+ *cheaper* knob** (`--boosts 0.5 1.0 1.25 1.5 1.75 2.0 --post 0.95`, the question
1361
+ "can we tune `boost` well enough to delete `glossary.py`?"). Answer: no. Matched on
1362
+ B-WER, lower-boost+post dominates higher-boost-alone on **both** damage metrics —
1363
+ at −37.6 % B-WER, `boost=1.5 + post` costs 8 false insertions and U-WER −2.3 %
1364
+ where `boost=1.75` alone costs **16** and −0.6 %; at −40 %, `boost=1.75 + post`
1365
+ (−40.7 %, U-WER −1.0 %, 16 FI) beats `boost=2.0` alone, which *fails* the U-WER
1366
+ gate (−39.4 %, **+6.7 %**, 27 FI) to get less. The layers fail differently, which
1367
+ is the whole point: boost makes the decoder credulous *everywhere*, so its damage
1368
+ is diffuse (U-WER, across the transcript); post@0.95 only fires on near-exact
1369
+ strings, so its damage is narrow. Buying a targeted fix with a diffuse knob is a
1370
+ bad trade. On **real meeting audio** the asymmetry is starker still: post@0.95 adds
1371
+ **0** false insertions at every boost level, while boost 1.0 → 1.75 takes them from
1372
+ 2 → 11. The fuzzy layer is free; boost is not. Only at ≈ −30 % B-WER is it a tie
1373
+ (`boost=1.25` alone ≈ `boost=1.0 + post`), so the layer is discardable only if we
1374
+ are content to stop there. **Shipped defaults stay at `boost = 1.0` + threshold
1375
+ 0.95** — the conservative end of the frontier, consistent with over-correction
1376
+ being worse than a missed correction. The rest of the frontier, if we ever want
1377
+ recall: `boost=1.5 + post` = B-WER −37.6 % for 7 false insertions on real audio.
1378
+ - **Head-to-head vs TypeWhisper's engine (`eval/bias_fluid.py`, English only).** Its
1379
+ Parakeet plugin calls FluidAudio, which runs *the same TDT 0.6b v3 we do*, so model,
1380
+ audio, lists and scorer are all held constant and the biasing **mechanism** is the
1381
+ only variable: our tree over token logits *inside* the greedy loop vs their second
1382
+ CTC model rescoring the *finished* transcript. **German is not runnable at all** —
1383
+ their spotter (`parakeet-ctc-110m`) has 1024 tokens and *zero* non-ASCII ones, so
1384
+ German terms tokenize to `<unk>` and are silently kept; TypeWhisper pairs it with the
1385
+ multilingual transcriber with no language check. **As TypeWhisper ships it, the
1386
+ vocabulary destroys the transcript**: B-WER −75.3 % bought with U-WER **+305.6 %** and
1387
+ **375 false insertions** (287/500 utterances altered; `glowing`→`unloving`,
1388
+ `pound`→`compound` — correct words snapped onto *distractors*). Not tuning: their own
1389
+ defaults give 374, their documented `cbw 3.0` gives 362. **But the mechanism is fine —
1390
+ the shipped config is the bug.** At `minSimilarity 0.85` the same engine reaches B-WER
1391
+ **−32.2 %, U-WER +2.4 %, 8 false insertions**, i.e. *parity* with our −34.9 % / +0.0 %
1392
+ / 2. On accuracy this benchmark does not separate the two approaches; what separates
1393
+ them is one pass vs two models, streaming vs batch-only, and German. Two defaults do
1394
+ the damage and both are worst where real users live: `rescorerConfig(forVocabSize:)`
1395
+ gives **≤10 terms the *loosest* threshold (0.50)** — a real glossary is 10–30 terms —
1396
+ and the **spotter rescue** (on by default) wrecks small lists outright (762 false
1397
+ insertions on an oracle list of only-spoken words; `--vocab-disable-spotter-rescue`
1398
+ drops it to 104, and minSimilarity does not gate it). Caveat, stated in the README:
1399
+ is21's "rare words" are ordinary English words, not the entities FluidAudio is built
1400
+ for, so this benchmark is harsher on their design than their target domain is.
1401
+ - **Glossary size is a risk knob in its own right** (`--tier distractor`, biasing
1402
+ with words known to be absent, so any change at all is a false insertion): 10
1403
+ bogus terms → **0 edits**, 50 → 2, 100 → 5, 500 → 17. Damage scales with list
1404
+ length because more phrases mean more prefixes for the decoder to fall into
1405
+ mid-word. Real meeting glossaries are 10–30 terms, where it is clean — but this
1406
+ is the argument for keeping them to the terms that actually come up.
1407
+ - **TTS (Tier 5) is a reachability diagnostic and cannot be more.** All four error
1408
+ classes are reachable (casing, compound, glued, multi-token name) and the
1409
+ absent-term probes stay clean. But the tier had to be *redesigned mid-flight*:
1410
+ it originally demanded the term be recovered under both German and English
1411
+ pronunciation, and measurement killed that. A German acoustic model fed English
1412
+ phonemes can lose the word entirely ("Kafka-Consumer" → heard as "Kafketon
1413
+ Zoomer") or produce a *different real word* — the English rendering of "Ada" is
1414
+ heard as "Ede", so the audio genuinely says "Ede", and demanding recovery there
1415
+ would be demanding the decoder overwrite what was said. **Requiring the fix would
1416
+ be requiring a false insertion.** Two voices of the *same* engine also landed on
1417
+ opposite sides of that line for one sentence. Verdict is therefore per error
1418
+ *class*, and every non-recovery prints what the engine actually said.
1419
+
1420
+ **Landmines paid for here.** `load_dataset(repo, config, split="test")` downloads
1421
+ the whole config — for LibriSpeech "clean" that is train-clean-360 too, 30+ GB to
1422
+ read a 350 MB test split; name the parquet instead. `datasets` ≥ 4 decodes audio
1423
+ through **torchcodec**, i.e. it would drag torch into a torch-free project — so the
1424
+ harness reads parquet with pyarrow + soundfile and decodes only the sampled rows.
1425
+ Piper's VITS decoder is **stochastic**: regenerating fixtures produced different
1426
+ audio and flipped a verdict, so synthesis noise is zeroed and the fixture set is
1427
+ pinned. `de_DE-kerstin-low` ships a 130-phoneme map with no combining cedilla — it
1428
+ cannot say the German ich-laut and Piper drops it with a log line, silently
1429
+ mangling "Ich"/"Küche"; the generator now refuses any voice that cannot render a
1430
+ fixture.
1431
+
1432
+ **Open.** The sweep did not re-run the winning config over the *full* test set (the
1433
+ tables say so); `[asr] boost` keeps its 1.0 default, now with evidence behind it.
1434
+
1287
1435
  ---
1288
1436
 
1289
1437
  ## 6. Key sources
@@ -251,16 +251,40 @@ dir (not the model cache) and are never uploaded.
251
251
 
252
252
  ## Vocabulary
253
253
 
254
- Domain terms and attendee names are corrected in the finalized transcript
255
- (the ASR has no decode-time biasing, so this is a post-correction pass):
254
+ Domain terms and attendee names steer the transcript twice: they boost the
255
+ decoder toward those spellings *while* it transcribes, and then correct the
256
+ near-misses it still got wrong in the finalized text.
256
257
 
257
258
  ```sh
258
259
  steno transcribe rec.mov --attendee "Anja Müller" --glossary Kubernetes,gRPC
259
260
  steno transcribe rec.mov --glossary-file terms.txt
260
261
  ```
261
262
 
262
- A term and its transcription must share a word count — `gRPC` can fix `G R P C`
263
- spoken as one word, but not a term split across word boundaries.
263
+ Write a term the way it appears in a sentence. A capitalized term (`Kubernetes`,
264
+ `iOS`) is imposed verbatim; an all-lowercase one asserts the spelling but leaves
265
+ the model's capitalization alone. In the correction pass a term and its
266
+ transcription must share a word count — `gRPC` can fix `G R P C` spoken as one
267
+ word, but not a term split across word boundaries.
268
+
269
+ **An acronym you pronounce as a word needs both spellings.** Biasing rewards the
270
+ exact token path you write, so `NIRS` only ever rewards `N-I-R-S` — a spelling
271
+ the acoustics never take when you say "nirs". List `NIRS` *and* `Nirs`: the
272
+ decoder can reach the second, and the correction pass snaps it back to the first,
273
+ since the two normalize to the same word. Only case variants of the same letters
274
+ work — a respelling (`Ekmo` for `ECMO`) is reachable but never snaps back.
275
+
276
+ Both layers are on whenever there are terms. `[asr] boost` scales the decoder
277
+ biasing (default 1.0; `0` turns it off, and much above ~3 it starts rewriting
278
+ words that were never in your list), and `[vocab] glossary_threshold` is the
279
+ similarity a word must reach before the correction pass touches it (default
280
+ 0.95 — deliberately strict, because a loose threshold corrupts words the model
281
+ already had right).
282
+
283
+ Reach for the spellings before the knobs. On an 85-minute German meeting, adding
284
+ the pronounceable twins fixed three more terms for four extra changed words in
285
+ the rest of the transcript, where raising `boost` to 1.5 bought two and disturbed
286
+ forty-six — at 2.0 the correct compound "Medizintechnikgruppe" decayed into
287
+ "Medizinischechnikgruppe", pulled apart by a listed term it half-matched.
264
288
 
265
289
  ## Settings
266
290
 
@@ -285,7 +309,7 @@ formats = ["md", "json", "txt"] # default --format list (srt/vtt for subtitles
285
309
  [vocab] # standing vocabulary — MERGED with per-run
286
310
  glossary_file = "~/steno/glossary.txt" # --glossary/--attendee flags
287
311
  attendees = ["Anja Müller"]
288
- glossary_threshold = 0.82
312
+ glossary_threshold = 0.95
289
313
 
290
314
  [output]
291
315
  dir = "~/Documents/Meetings" # where meeting folders are created
@@ -0,0 +1,306 @@
1
+ # Phase 0 — model evaluation harness
2
+
3
+ **Result (2026-07-06): Parakeet-TDT-0.6B-v3 is the default for finalize + live.**
4
+ Blind adjudication of 161 disagreement sites: Parakeet tied Whisper 42:42,
5
+ Voxtral lost to both, Canary lost every pairing ~1:2. Details in PLAN.md §2;
6
+ raw judgments in out/adjudication-results-2026-07-06.json (gitignored).
7
+
8
+ Goal: decide the default finalize-pass ASR model on *real* meeting audio, not
9
+ read-speech leaderboards. Candidates:
10
+
11
+ | Model | Runtime | Role | Status |
12
+ |---|---|---|---|
13
+ | Parakeet-TDT-0.6B-v3 | parakeet-mlx | **default candidate** (finalize *and* live) | wired up |
14
+ | Voxtral Small 24B (4-bit) | mlx-voxtral | max-accuracy challenger | wired up (text only — no timestamps), ~14 GB download |
15
+ | Whisper large-v3 | mlx-whisper | mature fallback | wired up |
16
+ | Canary-1B-v2 | NeMo on MPS | accuracy-ceiling reference only | wired up, needs `uv sync --group eval-canary` (slow; never shippable) |
17
+
18
+ Canary was demoted from expected default (July 2026 research): no MLX/CoreML
19
+ runtime emits its word timestamps — PyPI `canary-mlx` is an abandoned template,
20
+ mlx-audio's Canary returns hardcoded 0.0 timestamps, and onnx-asr's timestamp
21
+ support covers TDT/CTC/RNNT only. NeMo-on-MPS is the sole real path and is too
22
+ slow/heavy to ship, so it serves purely as the accuracy ceiling in this eval.
23
+
24
+ ## Workflow
25
+
26
+ Everything under `eval/audio/`, `eval/out/`, and `eval/refs/` contains private
27
+ meeting content and is gitignored.
28
+
29
+ ```sh
30
+ # 0. See what language/content each recording holds (Whisper-tiny probes)
31
+ uv run --group eval eval/scan_languages.py
32
+
33
+ # 1. Define eval segments in manifest.json (id, source file, start/end seconds,
34
+ # language), then cut them to mono 16 kHz WAV:
35
+ uv run --group eval eval/extract.py
36
+ # To pick bounds by listening, first extract a full recording:
37
+ uv run --group eval eval/extract.py --full "my-meeting-recording.mov"
38
+
39
+ # 2. Judge the models. Primary path — blind disagreement adjudication:
40
+ # align all hypotheses, review only the spots where models disagree
41
+ # (audio snippet + shuffled unlabeled variants, ~4s each):
42
+ uv run --group eval eval/adjudicate.py # → eval/out/adjudication.html
43
+ # Open in a browser, judge (keys 1–9, 0 = unsure), download results, then:
44
+ uv run --group eval eval/adjudicate.py --score ~/Downloads/adjudication-results.json
45
+ #
46
+ # Secondary path — full hand-corrected references (slow, and beware anchor
47
+ # bias: a reference corrected from model X's draft flatters model X; the
48
+ # de-1 attempt measured Whisper at 1.3% WER for exactly that reason).
49
+ # Fix eval/refs/<id>.draft.txt while listening, rename to <id>.txt;
50
+ # only *.txt files are scored by score.py.
51
+
52
+ # 3. Run every candidate (one process per backend, for clean peak-memory numbers):
53
+ uv run --group eval eval/transcribe.py --backend parakeet
54
+ uv run --group eval eval/transcribe.py --backend whisper
55
+ uv run --group eval eval/transcribe.py --backend voxtral
56
+
57
+ # 4. Score:
58
+ uv run --group eval eval/score.py # → eval/out/report.md
59
+ ```
60
+
61
+ Target coverage: ~10 min hand-corrected reference per language (German +
62
+ English), including one in-room far-field sample, per PLAN.md Phase 0.
63
+
64
+ ## Diarization scoring (Phase 3, Task 0d)
65
+
66
+ Measures the *diarizer*, not the ASR: **DER** (Diarization Error Rate) and
67
+ **word attribution** (of the finalized words, the fraction placed on the right
68
+ speaker). Nothing speaker-centric — re-ID threshold tuning, clustering/embedding
69
+ upgrades — is measurable without this, so it is the gating prerequisite.
70
+
71
+ References are **hand-labelled** speaker turns in NIST RTTM. The audio is a mono
72
+ downmix of every speaker (`extract.py`'s `-ac 1`), so one RTTM per segment labels
73
+ all distinct voices in it.
74
+
75
+ ```sh
76
+ # 1. Seed a draft from the current diarizer to correct while listening (much
77
+ # faster than labelling boundaries from scratch). Pass the real speaker count
78
+ # you remember — unconstrained estimation over-clusters badly (de-1 → 13
79
+ # speakers), which is exactly the problem this measures.
80
+ uv run eval/diarize.py --bootstrap --segments de-1,de-2,en-1 --num-speakers 3
81
+ # → eval/refs/<id>.draft.rttm (each line: SPEAKER <id> 1 <onset> <dur> <NA> <NA> <spk> <NA> <NA>)
82
+
83
+ # 2. Fix boundaries + merge/rename speakers against the audio, then rename to
84
+ # the scored name (only <id>.rttm is scored; never score a .draft — it
85
+ # flatters the model that produced it):
86
+ mv eval/refs/de-1.draft.rttm eval/refs/de-1.rttm # after correcting
87
+
88
+ # 3. Produce hypotheses (raw diarizer turns + finalized word labels):
89
+ uv run eval/diarize.py --segments de-1,de-2,en-1 # → eval/out/diar/<id>.{rttm,words.json}
90
+
91
+ # 4. Score (DER + word attribution, optimal speaker mapping, 0.25 s collar):
92
+ uv run eval/der.py # → eval/out/diar-report.md
93
+ ```
94
+
95
+ `der.py`/`rttm.py` are pure (numpy + scipy) and unit-tested in
96
+ `tests/test_eval_der.py` against hand-computed cases; `diarize.py` drives the
97
+ real stenograf backends. Everything under `eval/refs/` and `eval/out/` stays
98
+ gitignored (private content).
99
+
100
+ ## Echo cancellation (PLAN-AEC.md)
101
+
102
+ Layer-0 signal scoring of the AEC path. A meeting run with `--aec-dump DIR`
103
+ writes the clock-aligned `mic.wav`/`lpb.wav`/`enh.wav` triple (near end as
104
+ captured, far-end reference, near end as the ASR receives it); score it with:
105
+
106
+ ```sh
107
+ uv run --group eval eval/aec_score.py DIR --scenario st # far-end single-talk
108
+ uv run --group eval eval/aec_score.py DIR --scenario dt # double-talk
109
+ ```
110
+
111
+ Reports ERLE + residual level (energy over 10 ms frames during far-end
112
+ activity) and AECMOS (`speechmos`, the AEC-Challenge metric) — `echo_mos` for
113
+ "is the echo gone", `deg_mos` for "did we damage the local speaker". The
114
+ `--no-aec --aec-dump DIR` combination records the uncancelled baseline.
115
+
116
+ `aec_rig.py` runs a whole scenario on real hardware — plays a speech WAV out
117
+ the speakers while the real pipeline captures — and scores both layers (signal
118
+ metrics + leaked `Local-N` lines in the transcript):
119
+
120
+ ```sh
121
+ uv run --group eval eval/aec_rig.py far-only --seconds 60 # pass = 0 leaked lines
122
+ uv run --group eval eval/aec_rig.py far-only --no-aec # uncancelled baseline
123
+ uv run --group eval eval/aec_rig.py double-talk # talk over it yourself
124
+ ```
125
+
126
+ Runs land in `eval/out/aec/<scenario>-<stamp>/` with the meeting output, the
127
+ dump triple, and `rig.json`. Keep volume, lid angle, and source clip fixed
128
+ across runs you compare. Measured 2026-07-10 (MacBook speakers, volume 63):
129
+ AEC on → 37.6 dB ERLE, −65 dBFS residual, AECMOS echo 4.73, **0 leaked lines
130
+ before any text backstop**; AEC off → −27 dBFS raw echo, AECMOS echo 1.49.
131
+
132
+ ## Contextual-biasing evaluation (Phase 5)
133
+
134
+ Decode-time biasing (`stenograf.asr.biasing`) ships with a tree verified against
135
+ NeMo's golden vectors — but its *effect* rested on one TTS clip and three meeting
136
+ WAVs, which is enough to prove the mechanism fires and not enough to set `[asr]
137
+ boost` or to defend our two deliberate divergences from NeMo (`unk_score=1.0`, and
138
+ the German compound-tail tokenization). This harness replaces the anecdotes with
139
+ numbers, and needs **zero hand labeling**: every reference and every word list is
140
+ derived from corpora that already ship them.
141
+
142
+ ```sh
143
+ # 0. Fetch/derive the benchmarks (is21 English lists; German built from MLS)
144
+ uv run --group eval eval/bias_data.py --fetch all --sizes 100 500 1000 2000
145
+
146
+ # 1. Correctness gate — the only language with published numbers to check against
147
+ uv run --group eval eval/bias.py --tier english --n 100
148
+
149
+ # 2. The benchmark that sets the shipped defaults (ablates boost/unk/compound-tail)
150
+ uv run --group eval eval/bias.py --tier german --sweep
151
+
152
+ # 3. False insertions with ground truth: bias with words known to be ABSENT,
153
+ # so any change at all is a false insertion. Runs on real meeting audio.
154
+ uv run --group eval eval/bias.py --tier distractor --wav eval/audio/*.wav
155
+
156
+ # 3b. BOTH glossary layers — decode-time biasing vs post-hoc fuzzy correction vs
157
+ # the stack. Works on any tier and re-decodes nothing (post-correction is a
158
+ # pure text transform over the arms' cached hypotheses).
159
+ uv run --group eval eval/bias.py --tier german --post 0.88 0.92 0.95
160
+
161
+ # 4. Reachability probes (synthetic; a diagnostic, never a quality metric)
162
+ uv run --group eval eval/bias_tts.py && uv run --group eval eval/bias.py --tier tts
163
+
164
+ # 5. Head-to-head vs TypeWhisper's engine (FluidAudio), English only — see below
165
+ uv run --group eval eval/bias_fluid.py --cli /path/to/.build/release/fluidaudiocli
166
+ ```
167
+
168
+ ## The head-to-head: FluidAudio (TypeWhisper's Parakeet engine)
169
+
170
+ `bias_fluid.py` runs the *same acoustic model we do* (Parakeet TDT 0.6b v3) through
171
+ FluidAudio's context-biasing path — the engine behind TypeWhisper's Parakeet plugin —
172
+ on the same pinned 500 utterances, the same per-utterance 100-term lists, scored by
173
+ the same `bias_score`. Held constant: model, audio, lists, scorer. The only variable is
174
+ the **mechanism**: our boosting tree over token logits *inside* the greedy TDT loop, vs
175
+ their second CTC model rescoring the *finished* transcript. Each system is reported
176
+ against **its own** unbiased baseline (their encoder is CoreML int8, ours MLX fp32 — the
177
+ absolute WERs are not comparable; the deltas are).
178
+
179
+ **English only, and that is their limit, not our choice.** Their spotter
180
+ (`parakeet-ctc-110m`) has a 1024-token vocabulary with **zero** non-ASCII tokens, so
181
+ German terms tokenize into `<unk>` holes and are silently kept. TypeWhisper pairs it
182
+ with the multilingual transcriber with no language check. Our German tier has no
183
+ opponent — that is a capability they lack, not a number they lose.
184
+
185
+ **Result (2026-07-13).** As TypeWhisper ships it, the vocabulary posts a spectacular
186
+ B-WER −75.3 % and **destroys the transcript doing it**: U-WER **+305.6 %**, **375 false
187
+ insertions**, 287 of 500 utterances altered, with rewrites like `glowing`→`unloving`
188
+ and `pound`→`compound` — correct common words snapped onto *distractors*, terms the
189
+ benchmark included precisely because they are absent. Not a tuning accident: their own
190
+ defaults give 374, their documented `cbw 3.0` gives 362.
191
+
192
+ **But the mechanism is sound and the configuration is the bug.** Forced to
193
+ `minSimilarity 0.85`, the same engine reaches **B-WER −32.2 %, U-WER +2.4 %, 8 false
194
+ insertions** — parity with our −34.9 % / +0.0 % / 2. On accuracy alone this benchmark
195
+ does not separate in-loop boosting from post-decode rescoring. What separates them is
196
+ the rest of the bill: one pass vs two models, streaming vs batch-only, and German.
197
+
198
+ Two defaults do the damage, both worst where real users live:
199
+
200
+ | | |
201
+ |---|---|
202
+ | `rescorerConfig(forVocabSize:)` | minSimilarity **0.60** above 100 terms, **0.55** at 11–100, **0.50** at ≤10 — *smaller glossary, looser matching*. A real meeting glossary is 10–30 terms. |
203
+ | **spotter rescue** (on by default) | wrecks small lists: an oracle list of only genuinely-spoken words yields **762** false insertions and U-WER +1407 %; `--vocab-disable-spotter-rescue` alone drops it to 104. minSimilarity does not gate it — 0.85 leaves the collapse intact. |
204
+
205
+ **Read it fairly.** is21's "rare words" are ordinary English words (`frail`, `idly`,
206
+ `holiness`), not the entities and jargon FluidAudio is designed for; their published
207
+ 99.3 % precision on earnings calls is plausibly true in that domain. The bounded claim:
208
+ *on the standard biasing benchmark, at its standard list size, the configuration
209
+ TypeWhisper ships is destructive — and the mechanism underneath it is not.*
210
+
211
+ ## The confidence gate we did NOT build (`bias_confidence.py`)
212
+
213
+ The idea worth stealing from FluidAudio is the principle, not the code: **a post-hoc
214
+ correction must answer to the audio**, which is exactly what our fuzzy layer cannot do
215
+ and why over-correction forced the threshold to 0.95. The cheap version of that idea:
216
+ our TDT already emits per-token confidence from the *unbiased* distribution, so gate
217
+ corrections on it — refuse to rewrite a word the model was sure about. No second model,
218
+ no extra compute, works in every language.
219
+
220
+ `bias_confidence.py` tested the claim before anyone implemented it, and **killed it**:
221
+
222
+ | | n | median conf | p10 |
223
+ |---|---|---|---|
224
+ | false insertions | 53 | **0.999** | 0.938 |
225
+ | true fixes | 165 | **0.951** | 0.858 |
226
+
227
+ Directionally right, practically useless. The best gate (c = 0.95) blocks 45 of 53 false
228
+ insertions and destroys **84 of 165 true fixes**; looser points are worse. `der` → `deri`
229
+ (false insertion) scored 1.000 and `finde` → `find` (real fix) scored 0.997 — a greedy
230
+ RNN-T's entropy-normalized confidence saturates at ~1.0 for nearly everything, so it
231
+ cannot carry a gate. One decode pass, no refactor, question closed.
232
+
233
+ What survives is the *other* question. The gate asked "how sure was the model about the
234
+ word it wrote?" (always: certain). The discriminative one is "how much does the model
235
+ dislike the word we want to put there?" — scoring the **candidate** over the span,
236
+ against the tokens actually emitted. That is FluidAudio's idea done with the right model
237
+ (our 600M TDT, not a weak English-only 110M CTC), and it needs forced alignment through
238
+ the joint. Unproven, unbuilt; `bias_confidence.py` is what would price it.
239
+
240
+ **Metrics** (`bias_score.py`, pure — pinned by `tests/test_eval_bias.py`): B-WER
241
+ (WER over reference words in the biasing list — must fall), U-WER (every other word
242
+ — must **not** rise; over-boosting is visible here and nowhere else), entity
243
+ recall/precision/F, false insertions, and surface damage (`Ada` → `ADA`, which every
244
+ WER-shaped metric normalizes away before it can see it). Entity numbers are reported
245
+ strict *and* prefix-tolerant, because in German a term survives inside an inflected
246
+ or compounded word (`Europa` in `Europas`).
247
+
248
+ The scorer is a faithful port of is21's own alignment, so their **44 published
249
+ hypothesis/result file pairs are a free correctness oracle** — `tests/test_eval_bias.py`
250
+ reproduces every one of them to the digit (it skips until `--fetch is21` has run).
251
+
252
+ **What `--post` found (2026-07-13), and why the threshold moved to 0.95.** We ship
253
+ two glossary layers and had only ever measured one. Scored on the same terms, the
254
+ post-correction layer at its old 0.82 default looked spectacular on B-WER (−52.8 %
255
+ German, −55.5 % English — *twice* what decode-time biasing gets) and was in fact
256
+ damaging the transcript: U-WER **+9.9 % German / +86.3 % English**, 85–86 false
257
+ insertions against biasing's 3. The shipped stack (both layers, 0.82) came in at
258
+ U-WER +6.5 % and 84 false insertions — **worse than the `boost = 2.0` config we had
259
+ already rejected** (+6.7 %, 27 insertions). We had a bar; we had just never pointed
260
+ it at the second layer.
261
+
262
+ The cause is structural and worth remembering when adding any correction layer:
263
+ fuzzy text matching answers to no acoustics, so nothing prevents it from snapping a
264
+ correct common word onto a glossary term that merely looks like it, and **B-WER
265
+ alone cannot see the difference** — only U-WER and false insertions can. At 0.95 the
266
+ layer becomes a free win (B-WER −30.3 % German / −36.3 % English, i.e. *better* than
267
+ biasing alone, with U-WER flat and false insertions at biasing's own level), and on
268
+ real meeting audio with a realistic 30-term glossary it inserts nothing that biasing
269
+ did not already insert. One caveat on the tables: damage scales with list length, so
270
+ the N=100 lists overstate the risk for the 10–30-term glossaries real meetings use.
271
+
272
+ **The surface-damage column paid for itself here.** It was the only metric that could
273
+ see a second bug, and it sat at ~1190 for the post arms *at every threshold* — a
274
+ quantity that ignores the knob gating it is not being produced by that knob. Cause:
275
+ these word lists are 100 % lowercase, and `glossary.py` imposed a term's spelling
276
+ verbatim, so it dutifully de-capitalized German nouns. Fixed by teaching it what
277
+ `asr.biasing.surface_forms` already knew — capitals in a term are a deliberate
278
+ spelling and win; an all-lowercase term asserts nothing about case and must not
279
+ overwrite the model's. Re-running the tier drove the column **1190 → 2 with B-WER,
280
+ U-WER and false insertions bit-identical** (they normalize case, so they were blind
281
+ to the whole thing). Every WER-shaped metric would have shipped this.
282
+
283
+ Landmines, each verified and each worth a day: Parakeet emits punctuation and case
284
+ while LibriSpeech/MLS references do not (normalize both sides, and case-match the
285
+ boost phrases); every German noun is capitalized, so rare-by-frequency is the only
286
+ usable definition of a rare word; Common Voice is a dead stub on HuggingFace since
287
+ Mozilla moved it behind their Data Collective; AMI is uppercase, unpunctuated and
288
+ proper-noun-*sparse*, i.e. near-worthless for biasing despite being the closest
289
+ thing to meeting audio.
290
+
291
+ Sweeps run on a **pinned 500-utterance subsample** (the full grid is 5–10 h of
292
+ decoding); only the winning config is re-run over the full test set, and every table
293
+ states which it was. Hypotheses are cached per config, so an interrupted sweep only
294
+ costs the configs it had not reached.
295
+
296
+ ## Side quests
297
+
298
+ - ~~Canary-1B-v2 runtime~~ — resolved, see above: no accelerated runtime with
299
+ word timestamps exists; NeMo-on-MPS reference backend wired up instead.
300
+ - **speakrs diarization sanity check** on the same audio (not wired up yet).
301
+
302
+ ## Metrics
303
+
304
+ `transcribe.py` records per segment: wall time, speed (×RT), model load time,
305
+ peak RSS, MLX peak memory, detected language. `score.py` adds WER/CER (jiwer)
306
+ after normalization (lowercase, punctuation stripped, umlauts kept).