kotoshu 1.0.2-aarch64-linux

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (458) hide show
  1. checksums.yaml +7 -0
  2. data/.kotoshu-baseline.json +1 -0
  3. data/.pre-commit-hooks.yaml +18 -0
  4. data/.rspec +3 -0
  5. data/.rubocop.yml +351 -0
  6. data/.yardopts +8 -0
  7. data/CHANGELOG.md +813 -0
  8. data/CLAUDE.md +176 -0
  9. data/CODE_OF_CONDUCT.md +132 -0
  10. data/Cargo.lock +359 -0
  11. data/Cargo.toml +15 -0
  12. data/LICENSE +31 -0
  13. data/README.adoc +1292 -0
  14. data/Rakefile +150 -0
  15. data/SECURITY.md +93 -0
  16. data/TODO.impl/00-cut-0.1.md +121 -0
  17. data/TODO.impl/00-cut-0.2.md +175 -0
  18. data/TODO.impl/00-vision.md +281 -0
  19. data/TODO.impl/01-hunspell-correctness.md +111 -0
  20. data/TODO.impl/02-cli-unification.md +74 -0
  21. data/TODO.impl/03-dynamic-download.md +107 -0
  22. data/TODO.impl/04-language-modules.md +89 -0
  23. data/TODO.impl/05-semantic-path.md +83 -0
  24. data/TODO.impl/06-cjk-support.md +105 -0
  25. data/TODO.impl/07-rtl-support.md +71 -0
  26. data/TODO.impl/08-grammar-engine.md +106 -0
  27. data/TODO.impl/09-integrity-security.md +90 -0
  28. data/TODO.impl/10-testing-ci.md +98 -0
  29. data/TODO.impl/100-full-feature-batch3.md +28 -0
  30. data/TODO.impl/101-int4-tier-experiment.md +32 -0
  31. data/TODO.impl/102-lid-chain.md +28 -0
  32. data/TODO.impl/103-oov-bucket-tables.md +27 -0
  33. data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
  34. data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
  35. data/TODO.impl/106-native-lid-detection.md +36 -0
  36. data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
  37. data/TODO.impl/108-ko-ne-full-feature.md +37 -0
  38. data/TODO.impl/109-ten-audit.md +26 -0
  39. data/TODO.impl/11-release-v1.md +95 -0
  40. data/TODO.impl/110-sr-latn-and-polish.md +33 -0
  41. data/TODO.impl/111-embedding-bakeoff.md +33 -0
  42. data/TODO.impl/112-deployability-and-gating.md +26 -0
  43. data/TODO.impl/113-language-packs.md +26 -0
  44. data/TODO.impl/114-typo-biencoder-thread.md +33 -0
  45. data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
  46. data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
  47. data/TODO.impl/117-deterministic-frequency-tiers.md +110 -0
  48. data/TODO.impl/118-server-gem-floor.md +33 -0
  49. data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
  50. data/TODO.impl/120-setup-metadata-refresh.md +35 -0
  51. data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
  52. data/TODO.impl/122-pack-compression-kpk2.md +66 -0
  53. data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
  54. data/TODO.impl/124-owner-residue.md +19 -0
  55. data/TODO.impl/125-consumer-floor-guards.md +43 -0
  56. data/TODO.impl/126-tier-error-budgets.md +33 -0
  57. data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
  58. data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
  59. data/TODO.impl/129-pt-fr-corpora.md +23 -0
  60. data/TODO.impl/130-news-truth-0912.md +23 -0
  61. data/TODO.impl/131-hybrid-productization.md +42 -0
  62. data/TODO.impl/132-techniques-page-and-prose.md +27 -0
  63. data/TODO.impl/133-rust-first-distribution.md +68 -0
  64. data/TODO.impl/20-encapsulation-cleanup.md +47 -0
  65. data/TODO.impl/21-autoload-migration.md +64 -0
  66. data/TODO.impl/22-url-pin-single-source-of-truth.md +120 -0
  67. data/TODO.impl/23-fix-url-defaults.md +57 -0
  68. data/TODO.impl/24-cli-auto-setup-prompt.md +76 -0
  69. data/TODO.impl/25-soft-onnxruntime-require.md +75 -0
  70. data/TODO.impl/26-onnx-vocab-json-generation.md +96 -0
  71. data/TODO.impl/27-end-to-end-smoke-spec.md +113 -0
  72. data/TODO.impl/28-readme-quickstart-verification.md +68 -0
  73. data/TODO.impl/29-status-command.md +84 -0
  74. data/TODO.impl/30-language-auto-detection.md +74 -0
  75. data/TODO.impl/31-download-progress.md +58 -0
  76. data/TODO.impl/32-sarif-output-verification.md +72 -0
  77. data/TODO.impl/33-audit-log-rotation.md +54 -0
  78. data/TODO.impl/34-cache-eviction.md +59 -0
  79. data/TODO.impl/35-shell-completion.md +61 -0
  80. data/TODO.impl/36-cleanup-regressions-0.3.1.md +134 -0
  81. data/TODO.impl/37-hunspell-correctness-tier2.md +203 -0
  82. data/TODO.impl/38-onnx-semantic-gating.md +218 -0
  83. data/TODO.impl/39-tier3-and-beyond.md +232 -0
  84. data/TODO.impl/40-spec-drift-cleanup.md +134 -0
  85. data/TODO.impl/50-structure-aware-document-api.md +194 -0
  86. data/TODO.impl/51-grammar-engine-expansion.md +75 -0
  87. data/TODO.impl/52-multi-language-documents.md +95 -0
  88. data/TODO.impl/53-document-plugin-architecture.md +94 -0
  89. data/TODO.impl/54-cjk-support.md +117 -0
  90. data/TODO.impl/55-rtl-support.md +109 -0
  91. data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
  92. data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
  93. data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
  94. data/TODO.impl/59-yard-api-documentation.md +75 -0
  95. data/TODO.impl/60-lsp-server.md +174 -0
  96. data/TODO.impl/61-editor-ecosystem.md +167 -0
  97. data/TODO.impl/62-framework-integrations.md +159 -0
  98. data/TODO.impl/63-web-playground-wasm.md +186 -0
  99. data/TODO.impl/64-http-api-and-sdks.md +191 -0
  100. data/TODO.impl/65-universal-kotoshu.md +251 -0
  101. data/TODO.impl/66-kotoshu-core.md +169 -0
  102. data/TODO.impl/67-kotoshu-rs-and-access-libraries.md +137 -0
  103. data/TODO.impl/68-sota-adoption.md +152 -0
  104. data/TODO.impl/69-sota-typo-eval.md +74 -0
  105. data/TODO.impl/70-sota-confidence-cascade.md +59 -0
  106. data/TODO.impl/71-sota-nested-model.md +63 -0
  107. data/TODO.impl/72-model2vec-evaluation.md +49 -0
  108. data/TODO.impl/73-rl-reranker.md +42 -0
  109. data/TODO.impl/74-distilled-reranker.md +36 -0
  110. data/TODO.impl/75-llm-evaluator.md +35 -0
  111. data/TODO.impl/76-optional-llm-backend.md +37 -0
  112. data/TODO.impl/77-model-coverage-expansion.md +96 -0
  113. data/TODO.impl/78-in-browser-playground.md +69 -0
  114. data/TODO.impl/79-news-and-releases.md +60 -0
  115. data/TODO.impl/80-ecosystem-docs.md +56 -0
  116. data/TODO.impl/81-editor-reach.md +68 -0
  117. data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
  118. data/TODO.impl/83-models-batch-2.md +56 -0
  119. data/TODO.impl/84-language-modules-wave2.md +75 -0
  120. data/TODO.impl/85-wasm-model-api.md +48 -0
  121. data/TODO.impl/86-site-truth-wave2.md +45 -0
  122. data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
  123. data/TODO.impl/88-cli-directory-mode.md +54 -0
  124. data/TODO.impl/89-framework-integrations-exec.md +59 -0
  125. data/TODO.impl/90-gem-correctness-residues.md +44 -0
  126. data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
  127. data/TODO.impl/92-tier-mirror-cors.md +88 -0
  128. data/TODO.impl/93-python-wheels.md +32 -0
  129. data/TODO.impl/94-site-integrations-docs.md +35 -0
  130. data/TODO.impl/95-editor-action-polish.md +27 -0
  131. data/TODO.impl/96-server-semantic-models.md +50 -0
  132. data/TODO.impl/97-yard-execution.md +73 -0
  133. data/TODO.impl/98-campaign-cleanup.md +78 -0
  134. data/TODO.impl/99-site-truth-performance-docs.md +31 -0
  135. data/conformance/vectors.jsonl +2630 -0
  136. data/data/en/grammar/rules.yaml +851 -0
  137. data/docs/STABILITY-1.0-AUDIT.md +251 -0
  138. data/docs/STABILITY-1.0.md +137 -0
  139. data/docs/cache-architecture.md +502 -0
  140. data/docs/verification-evidence/correct3/pr_body.md +37 -0
  141. data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
  142. data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
  143. data/docs/verification-evidence/correct3/sv.aff +856 -0
  144. data/docs/verification-evidence/correct3/sv.dic +153717 -0
  145. data/examples/01_basic_word_checking.rb +38 -0
  146. data/examples/02_text_document_checking.rb +77 -0
  147. data/examples/03_dictionary_backends.rb +137 -0
  148. data/examples/04_trie_data_structure.rb +146 -0
  149. data/examples/05_suggestion_algorithms.rb +239 -0
  150. data/examples/06_configuration_advanced.rb +287 -0
  151. data/examples/07_multi_language_dictionaries.rb +278 -0
  152. data/exe/kotoshu +6 -0
  153. data/lib/kotoshu/algorithms/capitalization.rb +276 -0
  154. data/lib/kotoshu/algorithms/edit_distance.rb +128 -0
  155. data/lib/kotoshu/algorithms/lookup/affix_form.rb +132 -0
  156. data/lib/kotoshu/algorithms/lookup/compound_checks.rb +209 -0
  157. data/lib/kotoshu/algorithms/lookup/compound_form.rb +50 -0
  158. data/lib/kotoshu/algorithms/lookup.rb +841 -0
  159. data/lib/kotoshu/algorithms/ngram_suggest.rb +367 -0
  160. data/lib/kotoshu/algorithms/permutations.rb +283 -0
  161. data/lib/kotoshu/algorithms/phonet_suggest.rb +162 -0
  162. data/lib/kotoshu/algorithms/soundex.rb +92 -0
  163. data/lib/kotoshu/algorithms/suggest.rb +794 -0
  164. data/lib/kotoshu/algorithms.rb +23 -0
  165. data/lib/kotoshu/analyzers/semantic_analyzer.rb +405 -0
  166. data/lib/kotoshu/analyzers.rb +8 -0
  167. data/lib/kotoshu/baseline/application.rb +18 -0
  168. data/lib/kotoshu/baseline/entry.rb +33 -0
  169. data/lib/kotoshu/baseline/store.rb +175 -0
  170. data/lib/kotoshu/baseline.rb +20 -0
  171. data/lib/kotoshu/cache/base_cache.rb +778 -0
  172. data/lib/kotoshu/cache/cache.rb +91 -0
  173. data/lib/kotoshu/cache/eviction_policy.rb +66 -0
  174. data/lib/kotoshu/cache/frequency_cache.rb +283 -0
  175. data/lib/kotoshu/cache/language_cache.rb +511 -0
  176. data/lib/kotoshu/cache/lookup_cache.rb +166 -0
  177. data/lib/kotoshu/cache/model_cache.rb +1147 -0
  178. data/lib/kotoshu/cache/model_registry.rb +109 -0
  179. data/lib/kotoshu/cache/staged_languages.rb +45 -0
  180. data/lib/kotoshu/cache/suggestion_cache.rb +113 -0
  181. data/lib/kotoshu/cache.rb +49 -0
  182. data/lib/kotoshu/cli/auto_setup.rb +71 -0
  183. data/lib/kotoshu/cli/baseline_command.rb +119 -0
  184. data/lib/kotoshu/cli/batch_reporter.rb +321 -0
  185. data/lib/kotoshu/cli/cache_command.rb +282 -0
  186. data/lib/kotoshu/cli/completions_command.rb +210 -0
  187. data/lib/kotoshu/cli/dict_command.rb +64 -0
  188. data/lib/kotoshu/cli/directory_check.rb +146 -0
  189. data/lib/kotoshu/cli/directory_walker.rb +132 -0
  190. data/lib/kotoshu/cli/display_formatter.rb +429 -0
  191. data/lib/kotoshu/cli/errors.rb +36 -0
  192. data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
  193. data/lib/kotoshu/cli/interactive_reviewer.rb +314 -0
  194. data/lib/kotoshu/cli/language_resolver.rb +91 -0
  195. data/lib/kotoshu/cli/navigation_manager.rb +273 -0
  196. data/lib/kotoshu/cli/personal_command.rb +113 -0
  197. data/lib/kotoshu/cli/progress_reporter.rb +114 -0
  198. data/lib/kotoshu/cli/status_report.rb +130 -0
  199. data/lib/kotoshu/cli.rb +880 -0
  200. data/lib/kotoshu/components/passthrough_spell_checker.rb +70 -0
  201. data/lib/kotoshu/components/pos_tagger.rb +98 -0
  202. data/lib/kotoshu/components/spell_checker.rb +73 -0
  203. data/lib/kotoshu/components/synthesizer.rb +60 -0
  204. data/lib/kotoshu/components/tokenizer.rb +58 -0
  205. data/lib/kotoshu/components/whitespace_tokenizer.rb +94 -0
  206. data/lib/kotoshu/components.rb +13 -0
  207. data/lib/kotoshu/configuration/builder.rb +207 -0
  208. data/lib/kotoshu/configuration/resolver.rb +124 -0
  209. data/lib/kotoshu/configuration.rb +829 -0
  210. data/lib/kotoshu/conformance_exporter.rb +256 -0
  211. data/lib/kotoshu/conformance_runner.rb +176 -0
  212. data/lib/kotoshu/core/exceptions.rb +186 -0
  213. data/lib/kotoshu/core/indexed_dictionary.rb +293 -0
  214. data/lib/kotoshu/core/models/affix_rule.rb +262 -0
  215. data/lib/kotoshu/core/models/result/document_result.rb +132 -0
  216. data/lib/kotoshu/core/models/result/word_result.rb +131 -0
  217. data/lib/kotoshu/core/models/word.rb +131 -0
  218. data/lib/kotoshu/core/trie/builder.rb +119 -0
  219. data/lib/kotoshu/core/trie/node.rb +94 -0
  220. data/lib/kotoshu/core/trie/trie.rb +249 -0
  221. data/lib/kotoshu/core.rb +28 -0
  222. data/lib/kotoshu/data/common_words/de.yml +1800 -0
  223. data/lib/kotoshu/data/common_words/en.yml +1215 -0
  224. data/lib/kotoshu/data/common_words/es.yml +750 -0
  225. data/lib/kotoshu/data/common_words/fr.yml +1015 -0
  226. data/lib/kotoshu/data/common_words/pt.yml +870 -0
  227. data/lib/kotoshu/data/common_words/ru.yml +484 -0
  228. data/lib/kotoshu/data/common_words_loader.rb +141 -0
  229. data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
  230. data/lib/kotoshu/data/frozen_kelly.rb +38 -0
  231. data/lib/kotoshu/data.rb +8 -0
  232. data/lib/kotoshu/data_structures/bloom_filter.rb +188 -0
  233. data/lib/kotoshu/data_structures.rb +8 -0
  234. data/lib/kotoshu/debug_logger.rb +146 -0
  235. data/lib/kotoshu/debug_mode.rb +136 -0
  236. data/lib/kotoshu/defaults.rb +84 -0
  237. data/lib/kotoshu/dictionaries/catalog.rb +814 -0
  238. data/lib/kotoshu/dictionaries.rb +8 -0
  239. data/lib/kotoshu/dictionary/base.rb +324 -0
  240. data/lib/kotoshu/dictionary/cspell.rb +243 -0
  241. data/lib/kotoshu/dictionary/custom.rb +225 -0
  242. data/lib/kotoshu/dictionary/hunspell.rb +520 -0
  243. data/lib/kotoshu/dictionary/plain_text.rb +302 -0
  244. data/lib/kotoshu/dictionary/repository.rb +246 -0
  245. data/lib/kotoshu/dictionary/unified.rb +236 -0
  246. data/lib/kotoshu/dictionary/unix_words.rb +218 -0
  247. data/lib/kotoshu/dictionary.rb +15 -0
  248. data/lib/kotoshu/documents/document.rb +120 -0
  249. data/lib/kotoshu/documents/plain_text_document.rb +73 -0
  250. data/lib/kotoshu/documents/source_position.rb +45 -0
  251. data/lib/kotoshu/documents/source_range.rb +76 -0
  252. data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
  253. data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
  254. data/lib/kotoshu/documents/suppressions.rb +44 -0
  255. data/lib/kotoshu/documents/text_node.rb +49 -0
  256. data/lib/kotoshu/documents.rb +178 -0
  257. data/lib/kotoshu/embeddings/embedding_pipeline.rb +238 -0
  258. data/lib/kotoshu/embeddings/lru_cache.rb +244 -0
  259. data/lib/kotoshu/embeddings/onnx_runtime_model.rb +411 -0
  260. data/lib/kotoshu/embeddings/protocol.rb +96 -0
  261. data/lib/kotoshu/embeddings/protocols.rb +23 -0
  262. data/lib/kotoshu/embeddings/registry.rb +184 -0
  263. data/lib/kotoshu/embeddings/search.rb +204 -0
  264. data/lib/kotoshu/embeddings/similarity_engine.rb +278 -0
  265. data/lib/kotoshu/embeddings/similarity_search.rb +328 -0
  266. data/lib/kotoshu/embeddings/vocabulary.rb +306 -0
  267. data/lib/kotoshu/embeddings.rb +87 -0
  268. data/lib/kotoshu/fluent_checker.rb +94 -0
  269. data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +48 -0
  270. data/lib/kotoshu/grammar/pattern_matchers/double_negative_matcher.rb +179 -0
  271. data/lib/kotoshu/grammar/pattern_matchers/phrase_matcher.rb +86 -0
  272. data/lib/kotoshu/grammar/pattern_matchers/possessive_context_matcher.rb +75 -0
  273. data/lib/kotoshu/grammar/pattern_matchers/possessive_contraction_matcher.rb +87 -0
  274. data/lib/kotoshu/grammar/pattern_matchers/sentence_start_matcher.rb +84 -0
  275. data/lib/kotoshu/grammar/pattern_matchers/vowel_sound_matcher.rb +81 -0
  276. data/lib/kotoshu/grammar/pattern_matchers/word_list_matcher.rb +66 -0
  277. data/lib/kotoshu/grammar/rule.rb +99 -0
  278. data/lib/kotoshu/grammar/rule_engine.rb +96 -0
  279. data/lib/kotoshu/grammar/rule_loader.rb +30 -0
  280. data/lib/kotoshu/grammar.rb +24 -0
  281. data/lib/kotoshu/integrity/audit_log.rb +167 -0
  282. data/lib/kotoshu/integrity/manifest.rb +116 -0
  283. data/lib/kotoshu/integrity/net_http.rb +51 -0
  284. data/lib/kotoshu/integrity/rotation_policy.rb +88 -0
  285. data/lib/kotoshu/integrity.rb +26 -0
  286. data/lib/kotoshu/jekyll.rb +101 -0
  287. data/lib/kotoshu/keyboard/layout.rb +116 -0
  288. data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
  289. data/lib/kotoshu/keyboard/layouts/azerty.rb +55 -0
  290. data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
  291. data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
  292. data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
  293. data/lib/kotoshu/keyboard/layouts/dvorak.rb +54 -0
  294. data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
  295. data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
  296. data/lib/kotoshu/keyboard/layouts/hrsl.rb +94 -0
  297. data/lib/kotoshu/keyboard/layouts/jcuken.rb +57 -0
  298. data/lib/kotoshu/keyboard/layouts/latin.rb +266 -0
  299. data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
  300. data/lib/kotoshu/keyboard/layouts/qwerty.rb +52 -0
  301. data/lib/kotoshu/keyboard/layouts/qwertz.rb +55 -0
  302. data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
  303. data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
  304. data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
  305. data/lib/kotoshu/keyboard/registry.rb +201 -0
  306. data/lib/kotoshu/keyboard.rb +96 -0
  307. data/lib/kotoshu/kotoshu_native.so +0 -0
  308. data/lib/kotoshu/language/detector.rb +242 -0
  309. data/lib/kotoshu/language/identifier.rb +376 -0
  310. data/lib/kotoshu/language/languages/base.rb +271 -0
  311. data/lib/kotoshu/language/lid_detector.rb +188 -0
  312. data/lib/kotoshu/language/normalizer/arabic.rb +113 -0
  313. data/lib/kotoshu/language/normalizer/base.rb +133 -0
  314. data/lib/kotoshu/language/normalizer/greek.rb +63 -0
  315. data/lib/kotoshu/language/normalizer/hebrew.rb +72 -0
  316. data/lib/kotoshu/language/normalizer/persian.rb +36 -0
  317. data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
  318. data/lib/kotoshu/language/registry.rb +209 -0
  319. data/lib/kotoshu/language/resources/ar/common_words.txt +6753 -0
  320. data/lib/kotoshu/language/resources/ar/confusion_sets.txt +11 -0
  321. data/lib/kotoshu/language/resources/de/common_words.txt +10003 -0
  322. data/lib/kotoshu/language/resources/de/confusion_sets.txt +246 -0
  323. data/lib/kotoshu/language/resources/en/common_words.txt +9979 -0
  324. data/lib/kotoshu/language/resources/en/confusion_sets.txt +871 -0
  325. data/lib/kotoshu/language/resources/es/common_words.txt +9992 -0
  326. data/lib/kotoshu/language/resources/es/confusion_sets.txt +17 -0
  327. data/lib/kotoshu/language/resources/fr/common_words.txt +9993 -0
  328. data/lib/kotoshu/language/resources/fr/confusion_sets.txt +76 -0
  329. data/lib/kotoshu/language/resources/pt/common_words.txt +9977 -0
  330. data/lib/kotoshu/language/resources/pt/confusion_sets.txt +18 -0
  331. data/lib/kotoshu/language/resources/ru/common_words.txt +9951 -0
  332. data/lib/kotoshu/language/resources/ru/confusion_sets.txt +5 -0
  333. data/lib/kotoshu/language/script.rb +128 -0
  334. data/lib/kotoshu/language/segmenter.rb +122 -0
  335. data/lib/kotoshu/language/suika.rb +38 -0
  336. data/lib/kotoshu/language/tokenizer/base.rb +187 -0
  337. data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
  338. data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
  339. data/lib/kotoshu/language/tokenizer/french_tokenizer.rb +172 -0
  340. data/lib/kotoshu/language/tokenizer/german_tokenizer.rb +41 -0
  341. data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
  342. data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
  343. data/lib/kotoshu/language/tokenizer/japanese_tokenizer.rb +55 -0
  344. data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +153 -0
  345. data/lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb +160 -0
  346. data/lib/kotoshu/language/tokenizer/russian_tokenizer.rb +93 -0
  347. data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
  348. data/lib/kotoshu/language/tokenizer/spanish_tokenizer.rb +122 -0
  349. data/lib/kotoshu/language.rb +127 -0
  350. data/lib/kotoshu/languages/ar/language.rb +98 -0
  351. data/lib/kotoshu/languages/bg/language.rb +37 -0
  352. data/lib/kotoshu/languages/ca/language.rb +22 -0
  353. data/lib/kotoshu/languages/cs/language.rb +22 -0
  354. data/lib/kotoshu/languages/da/language.rb +22 -0
  355. data/lib/kotoshu/languages/de/language.rb +554 -0
  356. data/lib/kotoshu/languages/el/language.rb +50 -0
  357. data/lib/kotoshu/languages/en/language.rb +470 -0
  358. data/lib/kotoshu/languages/es/language.rb +476 -0
  359. data/lib/kotoshu/languages/et/language.rb +24 -0
  360. data/lib/kotoshu/languages/fa/language.rb +91 -0
  361. data/lib/kotoshu/languages/fr/language.rb +505 -0
  362. data/lib/kotoshu/languages/he/language.rb +98 -0
  363. data/lib/kotoshu/languages/hr/language.rb +24 -0
  364. data/lib/kotoshu/languages/hu/language.rb +22 -0
  365. data/lib/kotoshu/languages/id/language.rb +25 -0
  366. data/lib/kotoshu/languages/it/language.rb +22 -0
  367. data/lib/kotoshu/languages/ja/language.rb +476 -0
  368. data/lib/kotoshu/languages/ko/language.rb +56 -0
  369. data/lib/kotoshu/languages/latin_base.rb +53 -0
  370. data/lib/kotoshu/languages/lt/language.rb +24 -0
  371. data/lib/kotoshu/languages/lv/language.rb +24 -0
  372. data/lib/kotoshu/languages/nb/language.rb +25 -0
  373. data/lib/kotoshu/languages/ne/language.rb +60 -0
  374. data/lib/kotoshu/languages/nl/language.rb +22 -0
  375. data/lib/kotoshu/languages/nn/language.rb +26 -0
  376. data/lib/kotoshu/languages/pl/language.rb +22 -0
  377. data/lib/kotoshu/languages/pt/language.rb +439 -0
  378. data/lib/kotoshu/languages/ro/language.rb +22 -0
  379. data/lib/kotoshu/languages/ru/language.rb +420 -0
  380. data/lib/kotoshu/languages/sk/language.rb +25 -0
  381. data/lib/kotoshu/languages/sl/language.rb +25 -0
  382. data/lib/kotoshu/languages/sr/language.rb +39 -0
  383. data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
  384. data/lib/kotoshu/languages/sv/language.rb +22 -0
  385. data/lib/kotoshu/languages/tr/language.rb +36 -0
  386. data/lib/kotoshu/languages/uk/language.rb +38 -0
  387. data/lib/kotoshu/languages/vi/language.rb +22 -0
  388. data/lib/kotoshu/languages.rb +64 -0
  389. data/lib/kotoshu/metrics_collector.rb +238 -0
  390. data/lib/kotoshu/metrics_module.rb +121 -0
  391. data/lib/kotoshu/models/context.rb +119 -0
  392. data/lib/kotoshu/models/embedding_model.rb +181 -0
  393. data/lib/kotoshu/models/fasttext_model.rb +212 -0
  394. data/lib/kotoshu/models/nearest_neighbor.rb +96 -0
  395. data/lib/kotoshu/models/onnx_model.rb +330 -0
  396. data/lib/kotoshu/models/semantic_error.rb +204 -0
  397. data/lib/kotoshu/models/suggestion.rb +106 -0
  398. data/lib/kotoshu/models/word_embedding.rb +97 -0
  399. data/lib/kotoshu/models.rb +27 -0
  400. data/lib/kotoshu/multi_language_checker.rb +127 -0
  401. data/lib/kotoshu/native.rb +83 -0
  402. data/lib/kotoshu/native_backend.rb +131 -0
  403. data/lib/kotoshu/paths.rb +79 -0
  404. data/lib/kotoshu/personal_dictionary.rb +105 -0
  405. data/lib/kotoshu/plugins/plugin.rb +61 -0
  406. data/lib/kotoshu/plugins/registry.rb +118 -0
  407. data/lib/kotoshu/plugins.rb +9 -0
  408. data/lib/kotoshu/project_config.rb +76 -0
  409. data/lib/kotoshu/readers/aff_data.rb +530 -0
  410. data/lib/kotoshu/readers/aff_reader.rb +393 -0
  411. data/lib/kotoshu/readers/condition_checker.rb +125 -0
  412. data/lib/kotoshu/readers/dic_reader.rb +208 -0
  413. data/lib/kotoshu/readers/file_reader.rb +377 -0
  414. data/lib/kotoshu/readers/lookup_builder.rb +584 -0
  415. data/lib/kotoshu/readers/readers.rb +17 -0
  416. data/lib/kotoshu/readers.rb +25 -0
  417. data/lib/kotoshu/resource_bundle.rb +55 -0
  418. data/lib/kotoshu/resource_manager.rb +600 -0
  419. data/lib/kotoshu/results/result.rb +165 -0
  420. data/lib/kotoshu/results.rb +8 -0
  421. data/lib/kotoshu/rspec.rb +163 -0
  422. data/lib/kotoshu/scripts/fasttext_to_onnx.py +275 -0
  423. data/lib/kotoshu/source_registry.rb +104 -0
  424. data/lib/kotoshu/spellchecker/parallel_checker.rb +90 -0
  425. data/lib/kotoshu/spellchecker.rb +406 -0
  426. data/lib/kotoshu/string_metrics.rb +153 -0
  427. data/lib/kotoshu/suggestions/context.rb +43 -0
  428. data/lib/kotoshu/suggestions/frequency_provider.rb +80 -0
  429. data/lib/kotoshu/suggestions/generator.rb +169 -0
  430. data/lib/kotoshu/suggestions/pipeline.rb +135 -0
  431. data/lib/kotoshu/suggestions/semantic_cascade.rb +73 -0
  432. data/lib/kotoshu/suggestions/strategies/base_strategy.rb +315 -0
  433. data/lib/kotoshu/suggestions/strategies/composite_strategy.rb +197 -0
  434. data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +631 -0
  435. data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +207 -0
  436. data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +165 -0
  437. data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +284 -0
  438. data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +317 -0
  439. data/lib/kotoshu/suggestions/strategies/symspell_strategy.rb +270 -0
  440. data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
  441. data/lib/kotoshu/suggestions/suggestion.rb +154 -0
  442. data/lib/kotoshu/suggestions/suggestion_set.rb +270 -0
  443. data/lib/kotoshu/suggestions/sweep_index.rb +137 -0
  444. data/lib/kotoshu/suggestions.rb +29 -0
  445. data/lib/kotoshu/tasks/check_task.rb +104 -0
  446. data/lib/kotoshu/tasks.rb +11 -0
  447. data/lib/kotoshu/validators/spelling_validator.rb +101 -0
  448. data/lib/kotoshu/validators.rb +9 -0
  449. data/lib/kotoshu/version.rb +5 -0
  450. data/lib/kotoshu.rb +538 -0
  451. data/rakelib/compile.rake +10 -0
  452. data/script/validate_all_dictionaries.rb +444 -0
  453. data/scripts/build_native_gem.rb +45 -0
  454. data/scripts/extract_eval_grids.rb +37 -0
  455. data/scripts/generate_frozen_tiers.rb +44 -0
  456. data/sig/kotoshu.rbs +435 -0
  457. data/test_oop.rb +79 -0
  458. metadata +546 -0
@@ -0,0 +1,134 @@
1
+ # 36 — Cleanup Regressions + 0.3.1 Bug-Fix Release
2
+
3
+ ## Goal
4
+
5
+ Ship `kotoshu-0.3.1` — a focused bug-fix release that closes the
6
+ regressions introduced by the 0.3.0 lutaml-model migration and a small
7
+ set of pre-existing spec failures. This is the **only** Tier-1 work;
8
+ its acceptance criterion is a clean spec run on the non-network,
9
+ non-ONNX subset.
10
+
11
+ ## Scope (locked)
12
+
13
+ | # | Failure cluster | Root cause | Fix shape |
14
+ |---|---|---|---|
15
+ | 1 | `spec/integration/walking_skeleton_spec.rb:22` — `result.suggestions.words` | Lutaml migration changed `WordResult#suggestions` from `SuggestionSet` to `Array<Suggestion>`. | Update spec to `result.suggestions.map(&:word)`. |
16
+ | 2 | `spec/integration/walking_skeleton_spec.rb:265` — `result.to_h` | Lutaml-model replaced hand-rolled `to_h`. The instance method is gone; framework supplies `to_hash`. | Update spec to call `result.to_hash` (lutaml-supplied). |
17
+ | 3 | `spec/kotoshu/suggestions/strategies/semantic_strategy_spec.rb` (22 failures) | Specs assume `onnxruntime` is installed and a model is loaded. Without ONNX, every example errors. | Tag the spec file `:onnx` and exclude by default; document the opt-in env (`ONNX_TESTS=1`) in CLAUDE.md. Mirror the `:network` opt-in pattern. |
18
+ | 4 | `spec/unit/hunspell/read_aff_spec.rb` (7 failures) | Real parser bugs in REP, MAP, PFX directive parsing and UTF-8 / numeric / long flag formats. | Fix each parser branch with a targeted spec; do **not** rewrite the whole reader. |
19
+ | 5 | `spec/kotoshu/language/detector_spec.rb:56` — Arabic detection | Pre-existing FastText LID gap. | Mark `pending` with a TODO referencing the upstream model; do not block the release on it. |
20
+ | 6 | `spec/benchmark/symspell_benchmark_spec.rb` (1 failure) | Likely environment-dependent timing. | Audit; if real bug, fix; if env noise, mark `:slow`. |
21
+
22
+ **Out of scope:** the 90+ `spec/integrational/{lookup,suggest}_spec.rb`
23
+ Hunspell-fixture failures. Those are T2 (`TODO.impl/37-hunspell-correctness-tier2.md`).
24
+
25
+ ## Phase A — Lutaml regression cleanups
26
+
27
+ ### A1. walking_skeleton_spec
28
+
29
+ - Line 28: `expect(result.suggestions.words).to include("hello")`
30
+ → `expect(result.suggestions.map(&:word)).to include("hello")`
31
+ - Line 268: `hash = result.to_h`
32
+ → `hash = result.to_hash` (lutaml-model supplies this)
33
+
34
+ ### A2. semantic_strategy_spec gating
35
+
36
+ - Add `describe ..., :onnx do` or `RSpec.describe ... do; before { skip "set ONNX_TESTS=1" unless ENV["ONNX_TESTS"] } end`.
37
+ - Update `spec/spec_helper.rb` to exclude `onnx: true` unless `ONNX_TESTS=1`.
38
+ - Update `CLAUDE.md` "Notes" section to mention the opt-in.
39
+
40
+ ### A3. Audit other lutaml fallout
41
+
42
+ - `grep -rn "\.suggestions\.words\|\.suggestions\.to_a\|\.to_h\b" spec/`
43
+ (only the two walking_skeleton hits — confirmed.)
44
+ - `grep -rn "\.as_json\b" spec/` — already replaced in cli.rb; verify
45
+ no spec calls the removed instance method.
46
+
47
+ **Acceptance:** `bundle exec rspec --tag ~network --tag ~onnx` shows
48
+ **0 failures** attributable to lutaml migration.
49
+
50
+ ## Phase B — Hunspell .aff parser fixes
51
+
52
+ For each failing directive, add a focused spec covering the exact
53
+ shape from the failing fixture, then fix the parser branch.
54
+
55
+ ### B1. REP directive (read_aff_spec:12)
56
+
57
+ Repairs — substitution pairs used by the suggester. Currently parsed
58
+ incorrectly.
59
+
60
+ ### B2. MAP directive (read_aff_spec:31)
61
+
62
+ Character equivalence classes (e.g. `MAP uü`, `MAP aàáâã`). Sets up
63
+ related-character substitution in suggestions.
64
+
65
+ ### B3. PFX directive (read_aff_spec:47)
66
+
67
+ Prefix rules. Currently fails to parse multi-rule PFX blocks. Likely
68
+ the same bug as SFX since they share the parser.
69
+
70
+ ### B4. Long flag format (read_aff_spec:62)
71
+
72
+ `FLAG long` mode — two-character flags.
73
+
74
+ ### B5. UTF-8 flag format (read_aff_spec:106)
75
+
76
+ `FLAG UTF-8` mode — flags are single Unicode codepoints.
77
+
78
+ ### B6. Numeric flag format (read_aff_spec:125)
79
+
80
+ `FLAG num` mode — integer flags.
81
+
82
+ ### B7. AF directive / flag aliases (read_aff_spec:125)
83
+
84
+ `AF` directive defines aliases used to keep `.dic` files compact.
85
+
86
+ **Acceptance:** all 7 `read_aff_spec` examples green; no regression in
87
+ existing `algorithms/` specs.
88
+
89
+ ## Phase C — Pre-existing pending/skips
90
+
91
+ - Arabic detection: `pending "FastText LID missing Arabic vector — see TODO.impl/30-language-auto-detection.md"`.
92
+ - Symspell benchmark: audit; if real, fix; if env noise, `:slow`.
93
+
94
+ ## Phase D — Cut 0.3.1
95
+
96
+ 1. Bump `lib/kotoshu/version.rb` → `0.3.1`.
97
+ 2. Update CHANGELOG (or create if missing).
98
+ 3. Update README quickstart if it references any 0.3.0-broken API.
99
+ 4. Run full suite (`bundle exec rspec --tag ~network --tag ~onnx`)
100
+ green; `bundle exec rubocop` clean.
101
+ 5. PR (not direct-to-main — see global rule).
102
+ 6. After merge: tag `v0.3.1` (the **user** does this; never the agent).
103
+
104
+ ## Release-level acceptance criteria
105
+
106
+ - `bundle exec rspec --tag ~network --tag ~onnx` → 0 failures
107
+ (excluding the explicitly-pending FastText Arabic case).
108
+ - `bundle exec rubocop` → clean.
109
+ - Lutaml migration regressions: 0.
110
+ - Pre-existing Hunspell `.aff` parser bugs: 0.
111
+ - No new public API changes — 0.3.1 is strictly a bug-fix release.
112
+
113
+ ## Dependencies
114
+
115
+ - **Blocked by:** nothing.
116
+ - **Blocks:** `TODO.impl/37-hunspell-correctness-tier2.md` (the next
117
+ tier of Hunspell-fixture work).
118
+
119
+ ## Status (audited 2026-09-03, gem is 0.6.x)
120
+
121
+ All clusters were closed long ago at 0.3.1 (commit `364a8fa`) or since;
122
+ re-audited against main today — nothing in this plan still reproduced.
123
+
124
+ | Cluster | State |
125
+ |---|---|
126
+ | 1 walking_skeleton `suggestions.words` | Done at 0.3.1 (`map(&:word)`); spec green today. |
127
+ | 2 walking_skeleton `to_h` → `to_hash` | Done at 0.3.1; spec green today. |
128
+ | 3 semantic_strategy_spec gating | Gating wired in `spec_helper.rb`; done at 0.3.1. (A different, later failure in this file — vocab.json format drift — was fixed today under plan 38.) |
129
+ | 4 read_aff REP/MAP/PFX/flag bugs | Fixed long ago; 7/7 examples green today. kotoshu-rs subsequently ported these parsers against 1315 conformance vectors exported from this gem. |
130
+ | 5 Arabic detection | Still `pending` as planned (upstream LID gap); suite green. |
131
+ | 6 symspell benchmark | Green today; no `:slow` change needed. |
132
+ | Phase D 0.3.1 release | Shipped long ago (gem is now 0.6.x). Versioning/release decisions remain owner-only. |
133
+
134
+ Nothing remains open in this plan.
@@ -0,0 +1,203 @@
1
+ # 37 — Hunspell Correctness (Tier 2)
2
+
3
+ ## Goal
4
+
5
+ Drive the integrational Hunspell-fixture suite (`spec/integrational/
6
+ {lookup,suggest}_spec.rb`) from its current ~90-failure state to a
7
+ **bounded, categorized, steadily green** state. T2 is *correctness
8
+ work on the reader/lookup/suggest core*, not a rewrite.
9
+
10
+ ## Why this is T2
11
+
12
+ The fixtures are ported from
13
+ [`spylls`](https://github.com/neolithos/spylls) (Python) which itself
14
+ mirrors upstream Hunspell's `tests/` directory. Each fixture exercises
15
+ a single directive (`FLAG long`, `COMPOUNDRULE`, `CHECKCOMPOUNDPATTERN`,
16
+ …). The current Ruby port passes the easy ones and fails the ones that
17
+ exercise less-trafficked code paths.
18
+
19
+ This work is **not** a release blocker for 0.3.1 (see
20
+ `TODO.impl/36-cleanup-regressions-0.3.1.md`). It is the next tier.
21
+
22
+ ## Source of truth
23
+
24
+ - Reference: `/Users/mulgogi/src/external/hunspell/` (C++ upstream).
25
+ - Reference: `spylls` (Python port) — `algorithms/` is a Ruby line-by-line
26
+ port of `spylls/spylls/hunspell/`.
27
+ - Fixtures live in `spec/integrational/fixtures/` and are loaded via
28
+ `SpyllsTestHelper` (mixed into every spec through `spec_helper.rb`).
29
+
30
+ ## Phase structure
31
+
32
+ Each phase ends with a **green run on the fixtures it targets** plus a
33
+ commit on the T2 branch. The phases are independent — partial completion
34
+ still ships value.
35
+
36
+ ### Phase 2A — Flag-format correctness
37
+
38
+ **Fixtures affected:** `flaglong`, `flagnum`, `flagutf8`, `alias`,
39
+ `alias2`, `alias3`.
40
+
41
+ **Root causes (hypothesized, verify before fixing):**
42
+
43
+ 1. `FLAG long` mode: `parse_flags` in `aff_reader.rb` uses
44
+ `string.scan(/../)` which fails on **odd-length** flag runs (the last
45
+ char gets dropped). Real Hunspell pads/aligns differently per context
46
+ (single flag = 2 chars, multi-flag = even run).
47
+ 2. `FLAG num` mode: integer flags are parsed as strings, but downstream
48
+ `Set` operations sometimes compare against integers.
49
+ 3. `FLAG UTF-8` mode: surrogate-pair / multi-byte handling is incorrect
50
+ on non-BMP characters (see also `utf8_nonbmp`).
51
+ 4. `AF` flag aliases: aliases are looked up *per-rule* but should be
52
+ resolved once at load time and stashed on the `Affix` instance.
53
+
54
+ **Acceptance:** the 10 fixtures above pass in both `lookup_spec` and
55
+ `suggest_spec` (where a `.sug` fixture exists).
56
+
57
+ ### Phase 2B — Compounding
58
+
59
+ **Fixtures affected:** `compoundflag`, `onlyincompound`, `compoundaffix`,
60
+ `compoundrule{1..8}`, `checkcompoundcase{,2,utf}`,
61
+ `checkcompounddup`, `checkcompoundpattern{,2,3,4}`,
62
+ `checkcompoundrep`, `checkcompoundtriple`, `compoundforbid`,
63
+ `simplifiedtriple`, `wordpair`, `forceucase`, `utfcompound`,
64
+ `fogemorpheme`, `opentaal_cpdpat{,2}`, `opentaal_forbiddenword{1,2}`,
65
+ `germancompounding{,old}`.
66
+
67
+ This is the largest single cluster — roughly 30 fixtures.
68
+
69
+ **Approach:**
70
+
71
+ - Bring the Ruby `algorithms/suggest.rb` / `algorithms/lookup.rb` back
72
+ in line with `spylls` commit-by-commit. Each fixture failure has a
73
+ corresponding code path in spylls; find the divergence.
74
+ - The `COMPOUNDRULE` regex matcher (`CompoundRule#fullmatch` in
75
+ `aff_data.rb`) has known issues with parenthesized flag groups —
76
+ audit against spylls' `algorithms/compoundrule.py`.
77
+ - `CHECKCOMPOUNDPATTERN` with replacement (`/`-separated third field)
78
+ is `pending` in `lookup_spec.rb` already; **finish** that work.
79
+
80
+ **Acceptance:** all 30 fixtures green. The three already-`pending`
81
+ `replacement in pattern` cases move from pending to passing.
82
+
83
+ ### Phase 2C — Suggestion-quality
84
+
85
+ **Fixtures affected:** every fixture that has a `.sug` file in
86
+ `suggest_spec.rb`.
87
+
88
+ **Note:** suggestion output is order-sensitive and Hunspell's actual
89
+ output is the reference. We don't need *identical* output to upstream;
90
+ we need **the same set** with reasonable ordering. Where spylls asserts
91
+ ordering, mirror it; where Hunspell docs are silent, accept any
92
+ permutation.
93
+
94
+ **Approach:**
95
+
96
+ - `algorithms/ngram_suggest.rb`: port the latest ngram ranking from
97
+ spylls.
98
+ - `algorithms/phonet_suggest.rb`: audit against spylls; the PhonetTable
99
+ rule parser in `aff_data.rb` is the likely source of phantom
100
+ suggestions.
101
+ - `REP` patterns are applied too eagerly in the suggester (fixed in
102
+ T1 at the *reader* level; here we fix the *suggester* application).
103
+
104
+ **Acceptance:** every `suggest_spec` example either passes or carries
105
+ a documented `pending` with a concrete hypothesis.
106
+
107
+ ### Phase 2D — Edge cases & language-specific
108
+
109
+ **Fixtures affected:** `1463589`, `1592880`, `1695964`, `1706659`,
110
+ `1975530`, `2970240`, `2970242`, `2999225`, `i35725`, `i53643`,
111
+ `i54633`, `i54980`, `i58202`, `slash`, `morph`, `ngram_utf_fix`,
112
+ `ph2`, `warn`, `korean`, `utf8_nonbmp`, `checksharps{,utf}`,
113
+ `dotless_i`, `IJ`, `nepali`, `allcaps{,2,3,_utf}`, `forbiddenword`,
114
+ `keepcase`, `nosuggest`, `iconv{,2}`, `oconv{,2}`,
115
+ `breakdefault`, `break`, `breakoff`, `fullstrip`, `zeroaffix`,
116
+ `needaffix{,2,3,4,5}`, `circumfix`, `condition{,_utf}`,
117
+ `conditionalprefix`, `complexprefixes{,2,utf}`, `affixes`,
118
+ `base{,_utf}`, `encoding`, `utf8{,_bom,_bom2}`,
119
+ `right_to_left_mark`, `ignore{,sug,_utf}`, `opentaal_keepcase`.
120
+
121
+ These are the bug-tracker cases (numbers are Hunspell GitHub issue
122
+ numbers) plus a long tail of single-directive edge cases.
123
+
124
+ **Approach:** bucket by directive; fix one bucket per PR.
125
+
126
+ **Acceptance:** `bundle exec rspec spec/integrational --tag ~network`
127
+ shows ≤ 5 failing examples (each with a documented `pending` and
128
+ an upstream link).
129
+
130
+ ## Cross-cutting work
131
+
132
+ ### C1. `SpyllsTestHelper` audit
133
+
134
+ The helper currently `include`s in every spec. The `read_dictionary`
135
+ helper loads fixtures by name and constructs a `Hunspell` dictionary.
136
+ Audit:
137
+
138
+ - Does it use `double()` anywhere? (Forbidden per global rule.)
139
+ - Does it bypass the public `Dictionary::Hunspell` API to poke at
140
+ internals? If so, the API is the wrong shape — fix the API.
141
+ - Are the fixture paths resolved correctly under `bundle exec` versus
142
+ direct invocation?
143
+
144
+ ### C2. Reader/lookup OO refactor
145
+
146
+ The current `AffReader` returns a raw `Hash` with string keys. As T2
147
+ progresses, consider:
148
+
149
+ - A `Readers::AffData` value object (not a `Hash`) with typed accessors.
150
+ - A `Readers::DicData` parallel.
151
+ - The `Dictionary::Hunspell` constructor consumes `AffData`/`DicData`,
152
+ not hashes.
153
+
154
+ This is **optional** for T2 correctness — only do it if a fixture
155
+ demands it. If pursued, do it as the **last** step so the bulk of T2
156
+ remains "fix the bug, don't reshape the code".
157
+
158
+ ## Acceptance criteria for T2 release
159
+
160
+ - `bundle exec rspec spec/integrational --tag ~network` → ≤ 5 failing
161
+ examples, each `pending` with an upstream issue link.
162
+ - `bundle exec rspec --tag ~network --tag ~onnx` → all green (T1 still
163
+ green).
164
+ - `bundle exec rubocop` → clean.
165
+ - **No new public API changes.** T2 is internal correctness.
166
+
167
+ ## Dependencies
168
+
169
+ - **Blocked by:** `TODO.impl/36-cleanup-regressions-0.3.1.md`. The
170
+ integrational suite needs to be runnable on a clean baseline before
171
+ we can track fixture-level regressions.
172
+ - **Blocks:** nothing in T3. T3 features (CJK, RTL, grammar) build on a
173
+ correct core, but they can be developed in parallel branches against
174
+ the current (pre-T2) state if needed.
175
+
176
+ ## Out of scope
177
+
178
+ - Performance optimization (`spec/benchmark/`, `spec/performance/`).
179
+ That's T4.
180
+ - New languages beyond what the fixtures cover. That's T3.
181
+ - Grammar engine. That's T3 (`TODO.impl/39-tier3-and-beyond.md`).
182
+ - ONNX semantic path correctness. That's `TODO.impl/38-onnx-semantic-gating.md`.
183
+
184
+ ## Phase log (2026-09-04)
185
+
186
+ - **2A** — gem PR #104: AF alias values decode with the active FLAG
187
+ format (real FLAG-num bug; other hypotheses audited stale). Rust
188
+ re-synced same day (kotoshu-rs PR #4); vectors unchanged.
189
+ - **2B** — resolved via the owner collaborator PR #94 (HassanAkbar):
190
+ CHECKCOMPOUNDPATTERN replacement support ported from Hunspell +
191
+ compound checks module refactor + dedicated fixtures + perf spec;
192
+ merged by rebase after its stale-base failures cleared against
193
+ post-#106 main. A parallel independent implementation remains on
194
+ branch fix/t2-phase2b-compounding for reference.
195
+ - **2C** — gem PR #111: dot-split casing + INITCAP ngram-root skip
196
+ ported from hunspell suggestmgr (spylls never had them); the three
197
+ silently-excused pendings now pass for real. Full fixture dump:
198
+ exactly 3 vectors changed, zero collateral.
199
+ - Conformance vectors regenerated on post-#94 main: 2630 lines,
200
+ ~13 vectors changed (replacement fixtures now produce corrected
201
+ outputs). kotoshu-rs re-sync (this pack + the behavior ports) is
202
+ the follow-up once 2D lands.
203
+
@@ -0,0 +1,218 @@
1
+ # 38 — ONNX Semantic Path Gating & Hardening
2
+
3
+ ## Goal
4
+
5
+ Make the semantic path (ONNX FastText embeddings) a **first-class opt-in
6
+ feature** with a clear contract: when ONNX is absent, the traditional
7
+ path is unaffected; when ONNX is present, the semantic path is
8
+ testable, debuggable, and measurable.
9
+
10
+ ## Why this is its own tier
11
+
12
+ The semantic path is the differentiator — it's the "semantic" in
13
+ "kotoshu is a semantic spell checker." But it depends on
14
+ `onnxruntime` (a native extension), a multi-megabyte model download per
15
+ language, and a non-trivial embedding pipeline. None of those can be
16
+ assumed present in CI or on a fresh `gem install`.
17
+
18
+ Right now the path is half-gated: specs that need ONNX fail noisily
19
+ rather than skipping; the runtime gating is correct but the test
20
+ gating are missing.
21
+
22
+ ## Phase structure
23
+
24
+ ### Phase A — Spec gating (the T1 deliverable)
25
+
26
+ This is also referenced from `TODO.impl/36-cleanup-regressions-0.3.1.md`
27
+ Phase A2; details here.
28
+
29
+ **Actions:**
30
+
31
+ 1. Add `:onnx` to the `:integration`-tagged contexts in
32
+ `spec/kotoshu/suggestions/strategies/semantic_strategy_spec.rb`
33
+ (currently tagged `:integration` only — repurpose to `:onnx` since
34
+ the file is the sole user).
35
+ 2. Update `spec/spec_helper.rb`:
36
+ ```ruby
37
+ config.filter_run_excluding :onnx unless ENV.fetch("ONNX_TESTS", nil)
38
+ ```
39
+ 3. Document `ONNX_TESTS=1` in `CLAUDE.md` Notes section, mirroring the
40
+ `NETWORK_TESTS=1` pattern.
41
+ 4. Add a "smoke" spec (always-run, no `:onnx` tag) that asserts
42
+ `Kotoshu::Models::OnnxModel::ONNX_LOADED` matches
43
+ `Gem::Specification.find_all_by_name("onnxruntime").any?`. This
44
+ catches the silent regression where the soft-require path breaks.
45
+
46
+ **Acceptance:**
47
+
48
+ - `bundle exec rspec` (default) → 0 ONNX-related failures.
49
+ - `ONNX_TESTS=1 bundle exec rspec spec/kotoshu/suggestions/strategies/semantic_strategy_spec.rb`
50
+ → either all green (if onnxruntime installed + model cached) or all
51
+ skipped with a clear "model not cached; run `kotoshu setup :en --model`"
52
+ message.
53
+
54
+ ### Phase B — Runtime gating hardening
55
+
56
+ **Audit each entry point into the semantic path:**
57
+
58
+ 1. `Kotoshu::Analyzers::SemanticAnalyzer` — already raises
59
+ `Models::OnnxModel::OnnxUnavailable`. Verify the message is
60
+ actionable.
61
+ 2. `Suggestions::Strategies::SemanticStrategy#generate` — returns an
62
+ empty `SuggestionSet` when ONNX is unavailable. This is silent;
63
+ consider logging at debug level.
64
+ 3. `Kotoshu.correct?` / `Kotoshu.suggest` / `Kotoshu.check` — these
65
+ should *never* hit the semantic path unless explicitly requested
66
+ (e.g. via `Kotoshu.suggest(word, semantic: true)`). Verify.
67
+ 4. `KOTOSHU_NO_ONNX=1` is documented; verify it short-circuits all
68
+ semantic entry points.
69
+
70
+ **Acceptance:** with `onnxruntime` *installed* but
71
+ `KOTOSHU_NO_ONNX=1` set, `bundle exec rspec --tag ~network --tag ~onnx`
72
+ passes. With `onnxruntime` *uninstalled*, same suite passes.
73
+
74
+ ### Phase C — Model cache correctness
75
+
76
+ The semantic path depends on `Cache::ModelCache` finding a model file
77
+ under `~/.cache/kotoshu/models/{lang}/`. Issues to address:
78
+
79
+ 1. **Manifest validation.** `kotoshu/models-fasttext-onnx` ships a
80
+ `manifest.json` (18 entries per `kotoshu-content-repos-state.md`
81
+ memory). Verify `ModelCache` reads the manifest, not a glob, and
82
+ rejects unknown shapes.
83
+ 2. **Version pinning.** Each manifest entry should have a content hash
84
+ or version; verify the cache checks it.
85
+ 3. **Stale-model recovery.** If the model file is truncated or the
86
+ onnxruntime session can't deserialize, the error message should
87
+ point to `kotoshu cache download :en --model` (or equivalent).
88
+
89
+ **Acceptance:** deliberately corrupt a cached model file; the next
90
+ `Kotoshu.suggest(word, semantic: true)` raises a clear, actionable
91
+ error and points the user to the cache subcommand.
92
+
93
+ ### Phase D — Vocabulary / embedding pipeline tests
94
+
95
+ The `Embeddings::*` namespace (vocabulary, similarity engine, search,
96
+ LRU cache) is undertested. Add specs for:
97
+
98
+ 1. `Embeddings::Vocabulary` — token → id, id → token round-trip; OOV
99
+ behavior; boundary indices.
100
+ 2. `Embeddings::SimilarityEngine` — cosine similarity of orthogonal
101
+ vs identical vectors; dimension mismatch handling.
102
+ 3. `Embeddings::LruCache` — eviction at capacity; hit/miss accounting.
103
+
104
+ **These specs do not require ONNX** — they test the math, not the
105
+ runtime. They should be in the always-run suite.
106
+
107
+ **Acceptance:** the three classes above have direct unit specs (no
108
+ `:onnx` tag), all green in default `bundle exec rspec`.
109
+
110
+ ### Phase E — End-to-end semantic smoke test
111
+
112
+ A single `:onnx`-tagged spec that:
113
+
114
+ 1. Resolves the English model from cache (skip if missing).
115
+ 2. Builds a `SemanticAnalyzer` for English.
116
+ 3. Asserts that `analyze("their")` vs `analyze("there")` returns
117
+ distinguishable contexts.
118
+ 4. Asserts that `Kotoshu.suggest("helo", semantic: true)` returns
119
+ "hello" in the top 5.
120
+
121
+ This is the "does the semantic path actually work end-to-end" guard.
122
+
123
+ **Acceptance:** spec passes when ONNX is available; skips with a clear
124
+ message otherwise.
125
+
126
+ ## Cross-cutting: OCP/MECE audit
127
+
128
+ While doing the above, audit the embeddings namespace for:
129
+
130
+ - **OCP violations:** places where adding a new model type (e.g.
131
+ Word2Vec, GloVe) requires editing a switch instead of registering.
132
+ - **MECE violations:** overlap between `Embeddings::SimilarityEngine`
133
+ and `Embeddings::SimilaritySearch` — if they have overlapping
134
+ responsibilities, consolidate or split cleanly.
135
+ - **DRY:** the embedding-pipeline code is the prime suspect for
136
+ copy-paste between FastText and ONNX paths.
137
+
138
+ Document findings; address in T3 if scope creep risks the release.
139
+
140
+ ## Acceptance criteria
141
+
142
+ - Default `bundle exec rspec` → 0 ONNX-related failures, 0 ONNX-related
143
+ skips leaking into the non-`:onnx` suite.
144
+ - `ONNX_TESTS=1 bundle exec rspec --tag onnx` → all green (with model
145
+ cached) or all gracefully skipped (without).
146
+ - `KOTOSHU_NO_ONNX=1 bundle exec rspec` → identical pass rate to the
147
+ default run.
148
+ - Semantic path is documented end-to-end in a single place (probably
149
+ `docs/semantic-path.md`).
150
+
151
+ ## Dependencies
152
+
153
+ - **Blocked by:** nothing.
154
+ - **Blocks:** `TODO.impl/39-tier3-and-beyond.md` (the grammar / CJK
155
+ work uses the embedding pipeline; the gating must be solid first).
156
+
157
+ ## Status (re-audited 2026-09-03)
158
+
159
+ Done long ago: Phase A gating (`spec_helper.rb` `:onnx` exclusion,
160
+ `ONNX_TESTS=1` opt-in documented in CLAUDE.md), Phase D unit specs
161
+ (`embeddings/vocabulary_spec`, `similarity_engine_spec`, `lru_cache_spec`,
162
+ `models/models_spec`), Phase E e2e spec (`spec/kotoshu/semantic_e2e_spec.rb`,
163
+ tagged `:onnx` with skip guards).
164
+
165
+ Verified today:
166
+
167
+ - Phase B: `KOTOSHU_NO_ONNX=1 bundle exec rspec --tag ~network --tag ~onnx`
168
+ → 3285 examples, 0 failures (one extra skip: the new load-failure spec
169
+ correctly skips when the runtime is off). `SemanticStrategy` empty-return
170
+ still only warns at construction (`warn … if $VERBOSE`); there is no
171
+ debug-log infrastructure in the gem — left as-is (building one is out of
172
+ scope for this residue pass).
173
+
174
+ Fixed today (branch `fix/plan36-38-residue`):
175
+
176
+ - **vocab.json format drift** (the live 5 failures in
177
+ `semantic_strategy_spec`): the models repo / `fasttext_to_onnx.py` ship
178
+ vocab.json as `{"vocab_size", "word_to_idx"}`; `Embeddings::Vocabulary`
179
+ and `Models::OnnxModel.from_file` only parsed flat maps, so
180
+ `SemanticStrategy.new(language_code: "en")` crashed in
181
+ `Vocabulary#initialize` (`TypeError` via `Hash#<`). Both loaders now
182
+ share `Vocabulary.word_to_index_from_document` (flat, wrapped, and array
183
+ shapes); `Vocabulary#initialize` raises a clear ArgumentError for
184
+ non-Integer index values instead of the cryptic TypeError.
185
+ - **LFS pointer corruption** (Phase C root cause):
186
+ `raw.githubusercontent.com` serves 134-byte LFS pointer stubs for the
187
+ LFS-tracked `*.onnx`; the legacy full-tier download cached a stub and
188
+ recorded the stub's own sha256, so checksum verification passed forever
189
+ and the runtime died with a raw `OnnxRuntime::Error: Protobuf parsing
190
+ failed`. Fixes: `SourceRegistry` fetches `:model` from
191
+ `media.githubusercontent.com/media` (LFS host; custom mirrors untouched),
192
+ `ModelCache` refuses pointer stubs at download time and at
193
+ `verify_cached_integrity!` (purges stub + metadata, raises an error
194
+ pointing at `kotoshu cache download :{lang} --model`), and
195
+ `Embeddings::OnnxRuntimeModel#load!` wraps session-creation failures in
196
+ the same actionable `Kotoshu::Error`.
197
+ - **`Models::OnnxModel` session rot** (previously masked by the vocab
198
+ crash): `OnnxRuntime::Session` (nonexistent class) →
199
+ `OnnxRuntime::InferenceSession` with graph-detected IO names
200
+ (`word_index`/`embedding`); tensor I/O now uses real values instead of
201
+ `pack('q<')`/`unpack('e*')` byte surgery; batching executes sequential
202
+ single lookups (the converted graph's input is fixed shape [1]).
203
+ - **Spec drift**: strategy config examples no longer construct against the
204
+ developer's real `en` cache (use `xx`, per the file's own convention —
205
+ construction could otherwise trigger a 120MB download mid-spec);
206
+ identical-word similarity compared with float tolerance; removed the
207
+ stale `#embedding_for` describe (that API lives on the model, not the
208
+ strategy). New direct specs: `models/onnx_model_spec` (both vocab shapes),
209
+ `embeddings/onnx_runtime_model_spec` (actionable corrupt-model error).
210
+
211
+ `ONNX_TESTS=1 bundle exec rspec` on the semantic specs: 32 examples,
212
+ 0 failures against the real cached English model.
213
+
214
+ Remains (noted, not blocking): `Embeddings::OnnxRuntimeModel`'s
215
+ `run_batch_inference` feeds >1 indices to a fixed-shape-[1] graph, so
216
+ multi-index `get_embeddings` / `preload_embeddings!` fail at runtime; the
217
+ live single-lookup path is unaffected. Belongs with the tiered-model work
218
+ (plans 39/67).