kotoshu 1.0.2-aarch64-linux

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (458) hide show
  1. checksums.yaml +7 -0
  2. data/.kotoshu-baseline.json +1 -0
  3. data/.pre-commit-hooks.yaml +18 -0
  4. data/.rspec +3 -0
  5. data/.rubocop.yml +351 -0
  6. data/.yardopts +8 -0
  7. data/CHANGELOG.md +813 -0
  8. data/CLAUDE.md +176 -0
  9. data/CODE_OF_CONDUCT.md +132 -0
  10. data/Cargo.lock +359 -0
  11. data/Cargo.toml +15 -0
  12. data/LICENSE +31 -0
  13. data/README.adoc +1292 -0
  14. data/Rakefile +150 -0
  15. data/SECURITY.md +93 -0
  16. data/TODO.impl/00-cut-0.1.md +121 -0
  17. data/TODO.impl/00-cut-0.2.md +175 -0
  18. data/TODO.impl/00-vision.md +281 -0
  19. data/TODO.impl/01-hunspell-correctness.md +111 -0
  20. data/TODO.impl/02-cli-unification.md +74 -0
  21. data/TODO.impl/03-dynamic-download.md +107 -0
  22. data/TODO.impl/04-language-modules.md +89 -0
  23. data/TODO.impl/05-semantic-path.md +83 -0
  24. data/TODO.impl/06-cjk-support.md +105 -0
  25. data/TODO.impl/07-rtl-support.md +71 -0
  26. data/TODO.impl/08-grammar-engine.md +106 -0
  27. data/TODO.impl/09-integrity-security.md +90 -0
  28. data/TODO.impl/10-testing-ci.md +98 -0
  29. data/TODO.impl/100-full-feature-batch3.md +28 -0
  30. data/TODO.impl/101-int4-tier-experiment.md +32 -0
  31. data/TODO.impl/102-lid-chain.md +28 -0
  32. data/TODO.impl/103-oov-bucket-tables.md +27 -0
  33. data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
  34. data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
  35. data/TODO.impl/106-native-lid-detection.md +36 -0
  36. data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
  37. data/TODO.impl/108-ko-ne-full-feature.md +37 -0
  38. data/TODO.impl/109-ten-audit.md +26 -0
  39. data/TODO.impl/11-release-v1.md +95 -0
  40. data/TODO.impl/110-sr-latn-and-polish.md +33 -0
  41. data/TODO.impl/111-embedding-bakeoff.md +33 -0
  42. data/TODO.impl/112-deployability-and-gating.md +26 -0
  43. data/TODO.impl/113-language-packs.md +26 -0
  44. data/TODO.impl/114-typo-biencoder-thread.md +33 -0
  45. data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
  46. data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
  47. data/TODO.impl/117-deterministic-frequency-tiers.md +110 -0
  48. data/TODO.impl/118-server-gem-floor.md +33 -0
  49. data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
  50. data/TODO.impl/120-setup-metadata-refresh.md +35 -0
  51. data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
  52. data/TODO.impl/122-pack-compression-kpk2.md +66 -0
  53. data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
  54. data/TODO.impl/124-owner-residue.md +19 -0
  55. data/TODO.impl/125-consumer-floor-guards.md +43 -0
  56. data/TODO.impl/126-tier-error-budgets.md +33 -0
  57. data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
  58. data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
  59. data/TODO.impl/129-pt-fr-corpora.md +23 -0
  60. data/TODO.impl/130-news-truth-0912.md +23 -0
  61. data/TODO.impl/131-hybrid-productization.md +42 -0
  62. data/TODO.impl/132-techniques-page-and-prose.md +27 -0
  63. data/TODO.impl/133-rust-first-distribution.md +68 -0
  64. data/TODO.impl/20-encapsulation-cleanup.md +47 -0
  65. data/TODO.impl/21-autoload-migration.md +64 -0
  66. data/TODO.impl/22-url-pin-single-source-of-truth.md +120 -0
  67. data/TODO.impl/23-fix-url-defaults.md +57 -0
  68. data/TODO.impl/24-cli-auto-setup-prompt.md +76 -0
  69. data/TODO.impl/25-soft-onnxruntime-require.md +75 -0
  70. data/TODO.impl/26-onnx-vocab-json-generation.md +96 -0
  71. data/TODO.impl/27-end-to-end-smoke-spec.md +113 -0
  72. data/TODO.impl/28-readme-quickstart-verification.md +68 -0
  73. data/TODO.impl/29-status-command.md +84 -0
  74. data/TODO.impl/30-language-auto-detection.md +74 -0
  75. data/TODO.impl/31-download-progress.md +58 -0
  76. data/TODO.impl/32-sarif-output-verification.md +72 -0
  77. data/TODO.impl/33-audit-log-rotation.md +54 -0
  78. data/TODO.impl/34-cache-eviction.md +59 -0
  79. data/TODO.impl/35-shell-completion.md +61 -0
  80. data/TODO.impl/36-cleanup-regressions-0.3.1.md +134 -0
  81. data/TODO.impl/37-hunspell-correctness-tier2.md +203 -0
  82. data/TODO.impl/38-onnx-semantic-gating.md +218 -0
  83. data/TODO.impl/39-tier3-and-beyond.md +232 -0
  84. data/TODO.impl/40-spec-drift-cleanup.md +134 -0
  85. data/TODO.impl/50-structure-aware-document-api.md +194 -0
  86. data/TODO.impl/51-grammar-engine-expansion.md +75 -0
  87. data/TODO.impl/52-multi-language-documents.md +95 -0
  88. data/TODO.impl/53-document-plugin-architecture.md +94 -0
  89. data/TODO.impl/54-cjk-support.md +117 -0
  90. data/TODO.impl/55-rtl-support.md +109 -0
  91. data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
  92. data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
  93. data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
  94. data/TODO.impl/59-yard-api-documentation.md +75 -0
  95. data/TODO.impl/60-lsp-server.md +174 -0
  96. data/TODO.impl/61-editor-ecosystem.md +167 -0
  97. data/TODO.impl/62-framework-integrations.md +159 -0
  98. data/TODO.impl/63-web-playground-wasm.md +186 -0
  99. data/TODO.impl/64-http-api-and-sdks.md +191 -0
  100. data/TODO.impl/65-universal-kotoshu.md +251 -0
  101. data/TODO.impl/66-kotoshu-core.md +169 -0
  102. data/TODO.impl/67-kotoshu-rs-and-access-libraries.md +137 -0
  103. data/TODO.impl/68-sota-adoption.md +152 -0
  104. data/TODO.impl/69-sota-typo-eval.md +74 -0
  105. data/TODO.impl/70-sota-confidence-cascade.md +59 -0
  106. data/TODO.impl/71-sota-nested-model.md +63 -0
  107. data/TODO.impl/72-model2vec-evaluation.md +49 -0
  108. data/TODO.impl/73-rl-reranker.md +42 -0
  109. data/TODO.impl/74-distilled-reranker.md +36 -0
  110. data/TODO.impl/75-llm-evaluator.md +35 -0
  111. data/TODO.impl/76-optional-llm-backend.md +37 -0
  112. data/TODO.impl/77-model-coverage-expansion.md +96 -0
  113. data/TODO.impl/78-in-browser-playground.md +69 -0
  114. data/TODO.impl/79-news-and-releases.md +60 -0
  115. data/TODO.impl/80-ecosystem-docs.md +56 -0
  116. data/TODO.impl/81-editor-reach.md +68 -0
  117. data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
  118. data/TODO.impl/83-models-batch-2.md +56 -0
  119. data/TODO.impl/84-language-modules-wave2.md +75 -0
  120. data/TODO.impl/85-wasm-model-api.md +48 -0
  121. data/TODO.impl/86-site-truth-wave2.md +45 -0
  122. data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
  123. data/TODO.impl/88-cli-directory-mode.md +54 -0
  124. data/TODO.impl/89-framework-integrations-exec.md +59 -0
  125. data/TODO.impl/90-gem-correctness-residues.md +44 -0
  126. data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
  127. data/TODO.impl/92-tier-mirror-cors.md +88 -0
  128. data/TODO.impl/93-python-wheels.md +32 -0
  129. data/TODO.impl/94-site-integrations-docs.md +35 -0
  130. data/TODO.impl/95-editor-action-polish.md +27 -0
  131. data/TODO.impl/96-server-semantic-models.md +50 -0
  132. data/TODO.impl/97-yard-execution.md +73 -0
  133. data/TODO.impl/98-campaign-cleanup.md +78 -0
  134. data/TODO.impl/99-site-truth-performance-docs.md +31 -0
  135. data/conformance/vectors.jsonl +2630 -0
  136. data/data/en/grammar/rules.yaml +851 -0
  137. data/docs/STABILITY-1.0-AUDIT.md +251 -0
  138. data/docs/STABILITY-1.0.md +137 -0
  139. data/docs/cache-architecture.md +502 -0
  140. data/docs/verification-evidence/correct3/pr_body.md +37 -0
  141. data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
  142. data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
  143. data/docs/verification-evidence/correct3/sv.aff +856 -0
  144. data/docs/verification-evidence/correct3/sv.dic +153717 -0
  145. data/examples/01_basic_word_checking.rb +38 -0
  146. data/examples/02_text_document_checking.rb +77 -0
  147. data/examples/03_dictionary_backends.rb +137 -0
  148. data/examples/04_trie_data_structure.rb +146 -0
  149. data/examples/05_suggestion_algorithms.rb +239 -0
  150. data/examples/06_configuration_advanced.rb +287 -0
  151. data/examples/07_multi_language_dictionaries.rb +278 -0
  152. data/exe/kotoshu +6 -0
  153. data/lib/kotoshu/algorithms/capitalization.rb +276 -0
  154. data/lib/kotoshu/algorithms/edit_distance.rb +128 -0
  155. data/lib/kotoshu/algorithms/lookup/affix_form.rb +132 -0
  156. data/lib/kotoshu/algorithms/lookup/compound_checks.rb +209 -0
  157. data/lib/kotoshu/algorithms/lookup/compound_form.rb +50 -0
  158. data/lib/kotoshu/algorithms/lookup.rb +841 -0
  159. data/lib/kotoshu/algorithms/ngram_suggest.rb +367 -0
  160. data/lib/kotoshu/algorithms/permutations.rb +283 -0
  161. data/lib/kotoshu/algorithms/phonet_suggest.rb +162 -0
  162. data/lib/kotoshu/algorithms/soundex.rb +92 -0
  163. data/lib/kotoshu/algorithms/suggest.rb +794 -0
  164. data/lib/kotoshu/algorithms.rb +23 -0
  165. data/lib/kotoshu/analyzers/semantic_analyzer.rb +405 -0
  166. data/lib/kotoshu/analyzers.rb +8 -0
  167. data/lib/kotoshu/baseline/application.rb +18 -0
  168. data/lib/kotoshu/baseline/entry.rb +33 -0
  169. data/lib/kotoshu/baseline/store.rb +175 -0
  170. data/lib/kotoshu/baseline.rb +20 -0
  171. data/lib/kotoshu/cache/base_cache.rb +778 -0
  172. data/lib/kotoshu/cache/cache.rb +91 -0
  173. data/lib/kotoshu/cache/eviction_policy.rb +66 -0
  174. data/lib/kotoshu/cache/frequency_cache.rb +283 -0
  175. data/lib/kotoshu/cache/language_cache.rb +511 -0
  176. data/lib/kotoshu/cache/lookup_cache.rb +166 -0
  177. data/lib/kotoshu/cache/model_cache.rb +1147 -0
  178. data/lib/kotoshu/cache/model_registry.rb +109 -0
  179. data/lib/kotoshu/cache/staged_languages.rb +45 -0
  180. data/lib/kotoshu/cache/suggestion_cache.rb +113 -0
  181. data/lib/kotoshu/cache.rb +49 -0
  182. data/lib/kotoshu/cli/auto_setup.rb +71 -0
  183. data/lib/kotoshu/cli/baseline_command.rb +119 -0
  184. data/lib/kotoshu/cli/batch_reporter.rb +321 -0
  185. data/lib/kotoshu/cli/cache_command.rb +282 -0
  186. data/lib/kotoshu/cli/completions_command.rb +210 -0
  187. data/lib/kotoshu/cli/dict_command.rb +64 -0
  188. data/lib/kotoshu/cli/directory_check.rb +146 -0
  189. data/lib/kotoshu/cli/directory_walker.rb +132 -0
  190. data/lib/kotoshu/cli/display_formatter.rb +429 -0
  191. data/lib/kotoshu/cli/errors.rb +36 -0
  192. data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
  193. data/lib/kotoshu/cli/interactive_reviewer.rb +314 -0
  194. data/lib/kotoshu/cli/language_resolver.rb +91 -0
  195. data/lib/kotoshu/cli/navigation_manager.rb +273 -0
  196. data/lib/kotoshu/cli/personal_command.rb +113 -0
  197. data/lib/kotoshu/cli/progress_reporter.rb +114 -0
  198. data/lib/kotoshu/cli/status_report.rb +130 -0
  199. data/lib/kotoshu/cli.rb +880 -0
  200. data/lib/kotoshu/components/passthrough_spell_checker.rb +70 -0
  201. data/lib/kotoshu/components/pos_tagger.rb +98 -0
  202. data/lib/kotoshu/components/spell_checker.rb +73 -0
  203. data/lib/kotoshu/components/synthesizer.rb +60 -0
  204. data/lib/kotoshu/components/tokenizer.rb +58 -0
  205. data/lib/kotoshu/components/whitespace_tokenizer.rb +94 -0
  206. data/lib/kotoshu/components.rb +13 -0
  207. data/lib/kotoshu/configuration/builder.rb +207 -0
  208. data/lib/kotoshu/configuration/resolver.rb +124 -0
  209. data/lib/kotoshu/configuration.rb +829 -0
  210. data/lib/kotoshu/conformance_exporter.rb +256 -0
  211. data/lib/kotoshu/conformance_runner.rb +176 -0
  212. data/lib/kotoshu/core/exceptions.rb +186 -0
  213. data/lib/kotoshu/core/indexed_dictionary.rb +293 -0
  214. data/lib/kotoshu/core/models/affix_rule.rb +262 -0
  215. data/lib/kotoshu/core/models/result/document_result.rb +132 -0
  216. data/lib/kotoshu/core/models/result/word_result.rb +131 -0
  217. data/lib/kotoshu/core/models/word.rb +131 -0
  218. data/lib/kotoshu/core/trie/builder.rb +119 -0
  219. data/lib/kotoshu/core/trie/node.rb +94 -0
  220. data/lib/kotoshu/core/trie/trie.rb +249 -0
  221. data/lib/kotoshu/core.rb +28 -0
  222. data/lib/kotoshu/data/common_words/de.yml +1800 -0
  223. data/lib/kotoshu/data/common_words/en.yml +1215 -0
  224. data/lib/kotoshu/data/common_words/es.yml +750 -0
  225. data/lib/kotoshu/data/common_words/fr.yml +1015 -0
  226. data/lib/kotoshu/data/common_words/pt.yml +870 -0
  227. data/lib/kotoshu/data/common_words/ru.yml +484 -0
  228. data/lib/kotoshu/data/common_words_loader.rb +141 -0
  229. data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
  230. data/lib/kotoshu/data/frozen_kelly.rb +38 -0
  231. data/lib/kotoshu/data.rb +8 -0
  232. data/lib/kotoshu/data_structures/bloom_filter.rb +188 -0
  233. data/lib/kotoshu/data_structures.rb +8 -0
  234. data/lib/kotoshu/debug_logger.rb +146 -0
  235. data/lib/kotoshu/debug_mode.rb +136 -0
  236. data/lib/kotoshu/defaults.rb +84 -0
  237. data/lib/kotoshu/dictionaries/catalog.rb +814 -0
  238. data/lib/kotoshu/dictionaries.rb +8 -0
  239. data/lib/kotoshu/dictionary/base.rb +324 -0
  240. data/lib/kotoshu/dictionary/cspell.rb +243 -0
  241. data/lib/kotoshu/dictionary/custom.rb +225 -0
  242. data/lib/kotoshu/dictionary/hunspell.rb +520 -0
  243. data/lib/kotoshu/dictionary/plain_text.rb +302 -0
  244. data/lib/kotoshu/dictionary/repository.rb +246 -0
  245. data/lib/kotoshu/dictionary/unified.rb +236 -0
  246. data/lib/kotoshu/dictionary/unix_words.rb +218 -0
  247. data/lib/kotoshu/dictionary.rb +15 -0
  248. data/lib/kotoshu/documents/document.rb +120 -0
  249. data/lib/kotoshu/documents/plain_text_document.rb +73 -0
  250. data/lib/kotoshu/documents/source_position.rb +45 -0
  251. data/lib/kotoshu/documents/source_range.rb +76 -0
  252. data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
  253. data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
  254. data/lib/kotoshu/documents/suppressions.rb +44 -0
  255. data/lib/kotoshu/documents/text_node.rb +49 -0
  256. data/lib/kotoshu/documents.rb +178 -0
  257. data/lib/kotoshu/embeddings/embedding_pipeline.rb +238 -0
  258. data/lib/kotoshu/embeddings/lru_cache.rb +244 -0
  259. data/lib/kotoshu/embeddings/onnx_runtime_model.rb +411 -0
  260. data/lib/kotoshu/embeddings/protocol.rb +96 -0
  261. data/lib/kotoshu/embeddings/protocols.rb +23 -0
  262. data/lib/kotoshu/embeddings/registry.rb +184 -0
  263. data/lib/kotoshu/embeddings/search.rb +204 -0
  264. data/lib/kotoshu/embeddings/similarity_engine.rb +278 -0
  265. data/lib/kotoshu/embeddings/similarity_search.rb +328 -0
  266. data/lib/kotoshu/embeddings/vocabulary.rb +306 -0
  267. data/lib/kotoshu/embeddings.rb +87 -0
  268. data/lib/kotoshu/fluent_checker.rb +94 -0
  269. data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +48 -0
  270. data/lib/kotoshu/grammar/pattern_matchers/double_negative_matcher.rb +179 -0
  271. data/lib/kotoshu/grammar/pattern_matchers/phrase_matcher.rb +86 -0
  272. data/lib/kotoshu/grammar/pattern_matchers/possessive_context_matcher.rb +75 -0
  273. data/lib/kotoshu/grammar/pattern_matchers/possessive_contraction_matcher.rb +87 -0
  274. data/lib/kotoshu/grammar/pattern_matchers/sentence_start_matcher.rb +84 -0
  275. data/lib/kotoshu/grammar/pattern_matchers/vowel_sound_matcher.rb +81 -0
  276. data/lib/kotoshu/grammar/pattern_matchers/word_list_matcher.rb +66 -0
  277. data/lib/kotoshu/grammar/rule.rb +99 -0
  278. data/lib/kotoshu/grammar/rule_engine.rb +96 -0
  279. data/lib/kotoshu/grammar/rule_loader.rb +30 -0
  280. data/lib/kotoshu/grammar.rb +24 -0
  281. data/lib/kotoshu/integrity/audit_log.rb +167 -0
  282. data/lib/kotoshu/integrity/manifest.rb +116 -0
  283. data/lib/kotoshu/integrity/net_http.rb +51 -0
  284. data/lib/kotoshu/integrity/rotation_policy.rb +88 -0
  285. data/lib/kotoshu/integrity.rb +26 -0
  286. data/lib/kotoshu/jekyll.rb +101 -0
  287. data/lib/kotoshu/keyboard/layout.rb +116 -0
  288. data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
  289. data/lib/kotoshu/keyboard/layouts/azerty.rb +55 -0
  290. data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
  291. data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
  292. data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
  293. data/lib/kotoshu/keyboard/layouts/dvorak.rb +54 -0
  294. data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
  295. data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
  296. data/lib/kotoshu/keyboard/layouts/hrsl.rb +94 -0
  297. data/lib/kotoshu/keyboard/layouts/jcuken.rb +57 -0
  298. data/lib/kotoshu/keyboard/layouts/latin.rb +266 -0
  299. data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
  300. data/lib/kotoshu/keyboard/layouts/qwerty.rb +52 -0
  301. data/lib/kotoshu/keyboard/layouts/qwertz.rb +55 -0
  302. data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
  303. data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
  304. data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
  305. data/lib/kotoshu/keyboard/registry.rb +201 -0
  306. data/lib/kotoshu/keyboard.rb +96 -0
  307. data/lib/kotoshu/kotoshu_native.so +0 -0
  308. data/lib/kotoshu/language/detector.rb +242 -0
  309. data/lib/kotoshu/language/identifier.rb +376 -0
  310. data/lib/kotoshu/language/languages/base.rb +271 -0
  311. data/lib/kotoshu/language/lid_detector.rb +188 -0
  312. data/lib/kotoshu/language/normalizer/arabic.rb +113 -0
  313. data/lib/kotoshu/language/normalizer/base.rb +133 -0
  314. data/lib/kotoshu/language/normalizer/greek.rb +63 -0
  315. data/lib/kotoshu/language/normalizer/hebrew.rb +72 -0
  316. data/lib/kotoshu/language/normalizer/persian.rb +36 -0
  317. data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
  318. data/lib/kotoshu/language/registry.rb +209 -0
  319. data/lib/kotoshu/language/resources/ar/common_words.txt +6753 -0
  320. data/lib/kotoshu/language/resources/ar/confusion_sets.txt +11 -0
  321. data/lib/kotoshu/language/resources/de/common_words.txt +10003 -0
  322. data/lib/kotoshu/language/resources/de/confusion_sets.txt +246 -0
  323. data/lib/kotoshu/language/resources/en/common_words.txt +9979 -0
  324. data/lib/kotoshu/language/resources/en/confusion_sets.txt +871 -0
  325. data/lib/kotoshu/language/resources/es/common_words.txt +9992 -0
  326. data/lib/kotoshu/language/resources/es/confusion_sets.txt +17 -0
  327. data/lib/kotoshu/language/resources/fr/common_words.txt +9993 -0
  328. data/lib/kotoshu/language/resources/fr/confusion_sets.txt +76 -0
  329. data/lib/kotoshu/language/resources/pt/common_words.txt +9977 -0
  330. data/lib/kotoshu/language/resources/pt/confusion_sets.txt +18 -0
  331. data/lib/kotoshu/language/resources/ru/common_words.txt +9951 -0
  332. data/lib/kotoshu/language/resources/ru/confusion_sets.txt +5 -0
  333. data/lib/kotoshu/language/script.rb +128 -0
  334. data/lib/kotoshu/language/segmenter.rb +122 -0
  335. data/lib/kotoshu/language/suika.rb +38 -0
  336. data/lib/kotoshu/language/tokenizer/base.rb +187 -0
  337. data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
  338. data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
  339. data/lib/kotoshu/language/tokenizer/french_tokenizer.rb +172 -0
  340. data/lib/kotoshu/language/tokenizer/german_tokenizer.rb +41 -0
  341. data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
  342. data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
  343. data/lib/kotoshu/language/tokenizer/japanese_tokenizer.rb +55 -0
  344. data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +153 -0
  345. data/lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb +160 -0
  346. data/lib/kotoshu/language/tokenizer/russian_tokenizer.rb +93 -0
  347. data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
  348. data/lib/kotoshu/language/tokenizer/spanish_tokenizer.rb +122 -0
  349. data/lib/kotoshu/language.rb +127 -0
  350. data/lib/kotoshu/languages/ar/language.rb +98 -0
  351. data/lib/kotoshu/languages/bg/language.rb +37 -0
  352. data/lib/kotoshu/languages/ca/language.rb +22 -0
  353. data/lib/kotoshu/languages/cs/language.rb +22 -0
  354. data/lib/kotoshu/languages/da/language.rb +22 -0
  355. data/lib/kotoshu/languages/de/language.rb +554 -0
  356. data/lib/kotoshu/languages/el/language.rb +50 -0
  357. data/lib/kotoshu/languages/en/language.rb +470 -0
  358. data/lib/kotoshu/languages/es/language.rb +476 -0
  359. data/lib/kotoshu/languages/et/language.rb +24 -0
  360. data/lib/kotoshu/languages/fa/language.rb +91 -0
  361. data/lib/kotoshu/languages/fr/language.rb +505 -0
  362. data/lib/kotoshu/languages/he/language.rb +98 -0
  363. data/lib/kotoshu/languages/hr/language.rb +24 -0
  364. data/lib/kotoshu/languages/hu/language.rb +22 -0
  365. data/lib/kotoshu/languages/id/language.rb +25 -0
  366. data/lib/kotoshu/languages/it/language.rb +22 -0
  367. data/lib/kotoshu/languages/ja/language.rb +476 -0
  368. data/lib/kotoshu/languages/ko/language.rb +56 -0
  369. data/lib/kotoshu/languages/latin_base.rb +53 -0
  370. data/lib/kotoshu/languages/lt/language.rb +24 -0
  371. data/lib/kotoshu/languages/lv/language.rb +24 -0
  372. data/lib/kotoshu/languages/nb/language.rb +25 -0
  373. data/lib/kotoshu/languages/ne/language.rb +60 -0
  374. data/lib/kotoshu/languages/nl/language.rb +22 -0
  375. data/lib/kotoshu/languages/nn/language.rb +26 -0
  376. data/lib/kotoshu/languages/pl/language.rb +22 -0
  377. data/lib/kotoshu/languages/pt/language.rb +439 -0
  378. data/lib/kotoshu/languages/ro/language.rb +22 -0
  379. data/lib/kotoshu/languages/ru/language.rb +420 -0
  380. data/lib/kotoshu/languages/sk/language.rb +25 -0
  381. data/lib/kotoshu/languages/sl/language.rb +25 -0
  382. data/lib/kotoshu/languages/sr/language.rb +39 -0
  383. data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
  384. data/lib/kotoshu/languages/sv/language.rb +22 -0
  385. data/lib/kotoshu/languages/tr/language.rb +36 -0
  386. data/lib/kotoshu/languages/uk/language.rb +38 -0
  387. data/lib/kotoshu/languages/vi/language.rb +22 -0
  388. data/lib/kotoshu/languages.rb +64 -0
  389. data/lib/kotoshu/metrics_collector.rb +238 -0
  390. data/lib/kotoshu/metrics_module.rb +121 -0
  391. data/lib/kotoshu/models/context.rb +119 -0
  392. data/lib/kotoshu/models/embedding_model.rb +181 -0
  393. data/lib/kotoshu/models/fasttext_model.rb +212 -0
  394. data/lib/kotoshu/models/nearest_neighbor.rb +96 -0
  395. data/lib/kotoshu/models/onnx_model.rb +330 -0
  396. data/lib/kotoshu/models/semantic_error.rb +204 -0
  397. data/lib/kotoshu/models/suggestion.rb +106 -0
  398. data/lib/kotoshu/models/word_embedding.rb +97 -0
  399. data/lib/kotoshu/models.rb +27 -0
  400. data/lib/kotoshu/multi_language_checker.rb +127 -0
  401. data/lib/kotoshu/native.rb +83 -0
  402. data/lib/kotoshu/native_backend.rb +131 -0
  403. data/lib/kotoshu/paths.rb +79 -0
  404. data/lib/kotoshu/personal_dictionary.rb +105 -0
  405. data/lib/kotoshu/plugins/plugin.rb +61 -0
  406. data/lib/kotoshu/plugins/registry.rb +118 -0
  407. data/lib/kotoshu/plugins.rb +9 -0
  408. data/lib/kotoshu/project_config.rb +76 -0
  409. data/lib/kotoshu/readers/aff_data.rb +530 -0
  410. data/lib/kotoshu/readers/aff_reader.rb +393 -0
  411. data/lib/kotoshu/readers/condition_checker.rb +125 -0
  412. data/lib/kotoshu/readers/dic_reader.rb +208 -0
  413. data/lib/kotoshu/readers/file_reader.rb +377 -0
  414. data/lib/kotoshu/readers/lookup_builder.rb +584 -0
  415. data/lib/kotoshu/readers/readers.rb +17 -0
  416. data/lib/kotoshu/readers.rb +25 -0
  417. data/lib/kotoshu/resource_bundle.rb +55 -0
  418. data/lib/kotoshu/resource_manager.rb +600 -0
  419. data/lib/kotoshu/results/result.rb +165 -0
  420. data/lib/kotoshu/results.rb +8 -0
  421. data/lib/kotoshu/rspec.rb +163 -0
  422. data/lib/kotoshu/scripts/fasttext_to_onnx.py +275 -0
  423. data/lib/kotoshu/source_registry.rb +104 -0
  424. data/lib/kotoshu/spellchecker/parallel_checker.rb +90 -0
  425. data/lib/kotoshu/spellchecker.rb +406 -0
  426. data/lib/kotoshu/string_metrics.rb +153 -0
  427. data/lib/kotoshu/suggestions/context.rb +43 -0
  428. data/lib/kotoshu/suggestions/frequency_provider.rb +80 -0
  429. data/lib/kotoshu/suggestions/generator.rb +169 -0
  430. data/lib/kotoshu/suggestions/pipeline.rb +135 -0
  431. data/lib/kotoshu/suggestions/semantic_cascade.rb +73 -0
  432. data/lib/kotoshu/suggestions/strategies/base_strategy.rb +315 -0
  433. data/lib/kotoshu/suggestions/strategies/composite_strategy.rb +197 -0
  434. data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +631 -0
  435. data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +207 -0
  436. data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +165 -0
  437. data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +284 -0
  438. data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +317 -0
  439. data/lib/kotoshu/suggestions/strategies/symspell_strategy.rb +270 -0
  440. data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
  441. data/lib/kotoshu/suggestions/suggestion.rb +154 -0
  442. data/lib/kotoshu/suggestions/suggestion_set.rb +270 -0
  443. data/lib/kotoshu/suggestions/sweep_index.rb +137 -0
  444. data/lib/kotoshu/suggestions.rb +29 -0
  445. data/lib/kotoshu/tasks/check_task.rb +104 -0
  446. data/lib/kotoshu/tasks.rb +11 -0
  447. data/lib/kotoshu/validators/spelling_validator.rb +101 -0
  448. data/lib/kotoshu/validators.rb +9 -0
  449. data/lib/kotoshu/version.rb +5 -0
  450. data/lib/kotoshu.rb +538 -0
  451. data/rakelib/compile.rake +10 -0
  452. data/script/validate_all_dictionaries.rb +444 -0
  453. data/scripts/build_native_gem.rb +45 -0
  454. data/scripts/extract_eval_grids.rb +37 -0
  455. data/scripts/generate_frozen_tiers.rb +44 -0
  456. data/sig/kotoshu.rbs +435 -0
  457. data/test_oop.rb +79 -0
  458. metadata +546 -0
data/CHANGELOG.md ADDED
@@ -0,0 +1,813 @@
1
+ # Changelog
2
+
3
+ All notable changes to this project will be documented in this file.
4
+
5
+ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
6
+ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7
+
8
+ ## [Unreleased]
9
+
10
+ ### Added
11
+ - **Korean and Nepali full-feature modules** (plan 108) - the last
12
+ big-population languages without modules move from the plan 107
13
+ basic tier to full feature. `ko` ships the Hangul tokenizer
14
+ (an eojeol is a run of Hangul syllables U+AC00-D7A3 plus bare jamo
15
+ U+1100-11FF/U+3130-318F that escaped IME composition) and the
16
+ Dubeolsik keyboard grid keyed on jamo per the KS X 5002 standard 2-set
17
+ layout, so typo proximity lands on the unit vowel mergers and
18
+ fortis/lenis confusions operate on. `ne` ships the Devanagari
19
+ tokenizer (matras U+093E-094C, the virama U+094D and vowel signs
20
+ stay attached to the base consonant; the danda and Devanagari
21
+ digits are separators) and the Devanagari InScript grid (the Indian
22
+ national standard, mirrored from the models repo eval harness and
23
+ drift-checked against it; the Nepali Traditional Romanized layout
24
+ is more common on the ground but is not a documented standard).
25
+ Both dictionaries were already staged, so `kotoshu setup ko ne`
26
+ serves them; full-feature languages grow 33 to 35.
27
+ - **Basic support for every staged dictionary language** (plan 107) -
28
+ `kotoshu setup` now serves all 95 languages staged in the
29
+ kotoshu/dictionaries manifest, not just the ones with gem modules.
30
+ `LanguageCache::AVAILABLE_LANGUAGES` derives from a vendored
31
+ snapshot of the manifest (Cache::StagedLanguages, refreshed by
32
+ `rake kotoshu:staged_languages:sync`), keeping the hot path
33
+ network-free. Languages without a module run on the basic tier:
34
+ script classification (Language::Script) picks the tokenizer whose
35
+ word regex drives check/suggest word extraction (shared Latin /
36
+ Cyrillic / Greek tokenizers plus a generic regex tokenizer for
37
+ Arabic, Hebrew, Armenian, Georgian, Devanagari and Hangul), and the
38
+ keyboard registry falls back to the script's standard grid (JCUKEN
39
+ for Cyrillic, Arabic 101, Hebrew SI-1452, QWERTY otherwise) instead
40
+ of QWERTY for everything. Two documented tiers: full feature =
41
+ module (registered keyboard + verified specimens,
42
+ `LanguageCache.full_feature_languages`), basic = manifest-present.
43
+ `nn` (Norwegian Nynorsk) is wired as a full-feature module while
44
+ here - dictionary and model are both staged. Modules keep winning
45
+ over the fallbacks; the 32 existing modules behave identically
46
+ (conformance vectors untouched).
47
+ - **Personal dictionary in the check path** (plan 105) -
48
+ `Spellchecker#check` (and therefore `Kotoshu.check`,
49
+ `Kotoshu.check_file`, and `kotoshu check`) no longer flags words
50
+ present in the user personal dictionary
51
+ (~/.config/kotoshu/personal.dic, KOTOSHU_PERSONAL_DIC override),
52
+ matching the semantics the LSP already applies to diagnostics:
53
+ case-insensitive, filtered at result assembly without suppression
54
+ metadata, words still counted, and `suggest`/`correct?` untouched so
55
+ personal words stay spellcheckable. The dictionary is loaded once
56
+ per process (no hot reload; the LSP keeps its mtime-based reload).
57
+ Opt out with `Kotoshu.configure { |c| c.personal_dictionary = false }`
58
+ / KOTOSHU_PERSONAL_DICTIONARY=false / `kotoshu check --no-personal`.
59
+ - **Native language detection** (plan 106) - `Kotoshu.detect_language(text)`
60
+ now returns a `Kotoshu::Language::Detection` (`code` + `score`) scored by
61
+ the Rust extension over the `kotoshu://models/lid/lid-176` registry
62
+ artifact pair (the ONNX-converted lid.176 model, 176 languages,
63
+ bit-faithful to the fastText bindings - parity 55/55 against the frozen
64
+ corpus shared with kotoshu-rs). Fetch the pair once with
65
+ `Kotoshu.setup_lid` (registry-resolved, sha-verified, cached under
66
+ `models-cache/lid/`); detect resolves cache-only and never downloads.
67
+ The 7-language `Language::Detector` heuristic stays the fallback when
68
+ the extension is absent, the model is not set up, or
69
+ `KOTOSHU_BACKEND=ruby` opts out - so pure-Ruby installs are unaffected
70
+ (no new gemspec dependency, no onnxruntime). The magnus shim
71
+ (ext/kotoshu_native) gains `Kotoshu::Native::LidModel.load` +
72
+ `#detect` behind the kotoshu-rs `model` feature; the workspace
73
+ Cargo.lock moves to the kotoshu-rs revision carrying the reader.
74
+ `Kotoshu.detect_language_with_confidence` keeps its `[code, score]`
75
+ shape and now routes through the same engine.
76
+ - **Full-feature language batch 3** (plan 100) - twelve languages gain the
77
+ downloadable spelling dictionary, a keyboard layout, and a gem module:
78
+ `ar` `id` `fa` `he` `bg` `sr` `hr` `sk` `sl` `lt` `lv` `et` (the RTL
79
+ ar/fa/he modules are promoted from tokenizer-only wiring; the other nine
80
+ are new thin modules - id hr sl sk et lt lv on `LatinBase`, bg sr with
81
+ the Cyrillic tokenizer). Six national key grids join the layout registry
82
+ (Arabic 101, Persian standard ISIRI 9147, Hebrew SI-1452, Bulgarian BDS,
83
+ Serbian Cyrillic, and the shared Croatian/Slovenian QWERTZ), all mirrored
84
+ from the models repo eval harness and drift-checked against it; five new
85
+ Latin family members (Indonesian Slovak Estonian Lithuanian Latvian)
86
+ type on the US QWERTY grid like the eval model. Bulgarian and Serbian
87
+ register ahead of JCUKEN so their Cyrillic grids win the lookup. The
88
+ eval-grid drift fixture now covers all nine national grids. Specimen
89
+ misspelling -> correction pairs for every language are engine-verified
90
+ against the staged dictionaries in a `:network` spec.
91
+
92
+ ### Fixed
93
+ - **RTL word extraction on the check path** - the plan-91 script-aware
94
+ word regex never reached the Arabic, Persian and Hebrew tokenizers, so
95
+ `Kotoshu.check` / `kotoshu check -l ar|fa|he` extracted zero words and
96
+ silently passed every RTL document. The three tokenizers now declare
97
+ their script sets (`\p{Arabic}`, `\p{Arabic}` + ZWNJ for Persian
98
+ compounds like می‌روم, `\p{Hebrew}` + geresh/gershayim), and RTL
99
+ misspellings surface with ranked suggestions.
100
+ - **Suggestion candidate sweep missed transpositions, substitutions, and
101
+ dictionary forms** - `EditDistanceStrategy` only scored raw dictionary
102
+ stems, so valid dictionary forms could never be suggested no matter how
103
+ close: an adjacent transposition paired with a case difference ("Teh" ->
104
+ "The") was charged as two operations, and affixed forms ("definately" ->
105
+ "definitely", a stem + `-ly` suffix never present as a stem) were absent
106
+ from the candidate set entirely. The strategy now also sweeps the
107
+ distance-1 edits of the misspelling itself - adjacent transposition
108
+ (restricted Damerau operation, cost 1), substitution and insertion over
109
+ the dictionary TRY string, and deletion - validating each against the
110
+ full affix-aware dictionary lookup, and case-variant duplicates of the
111
+ same word are deduplicated keeping the best-scoring form. The phonetic
112
+ and keyboard-proximity strategies now measure candidates with
113
+ `Algorithms::EditDistance` (Damerau-Levenshtein; previously each carried
114
+ a private plain-Levenshtein copy charging transpositions 2), and the
115
+ keyboard variant generator enumerates adjacent transpositions as a
116
+ single edit. `Dictionary::Base#try_string` / `Dictionary::Hunspell#try_string`
117
+ expose the Hunspell TRY directive for the sweep alphabet. Conformance
118
+ suggest vectors regenerated (84 of 1315 rows improved - affixed, compound,
119
+ and correctly-cased forms now appear; all 1315 correct vectors unchanged).
120
+ - **Dictionary mutation correctness** (from suleman-uzair, PR rescue of #93) -
121
+ `remove_word` deleted the wrong word after any prior removal because the
122
+ word set stored array indices that went stale; `Custom`, `PlainText`, and
123
+ `UnixWords` now track membership (word => true) and remove by value, so
124
+ every copy of a duplicate word is gone and `words` / `include?` stay
125
+ consistent across interleaved add/remove sequences. `PlainText#add_word` /
126
+ `#remove_word` now also keep the length-bucketed index in sync, so
127
+ `find_by_length_range` (the edit-distance candidate filter) reflects
128
+ post-construction mutations instead of returning stale buckets.
129
+ - **Edit-distance threshold on empty strings** (from suleman-uzair, PR rescue
130
+ of #93) - `Algorithms::EditDistance.distance_with_threshold` returned the
131
+ other string's length for empty inputs even when that exceeded the
132
+ threshold; the length pre-filter now runs before the empty-string
133
+ shortcuts, so the nil-above-threshold contract holds for empty strings.
134
+ - **Offline mode enforcement and cache-only suggestion hot path** (from
135
+ suleman-uzair, PR rescue of #93) - `Kotoshu::Configuration.offline` /
136
+ `KOTOSHU_OFFLINE=1` previously did not stop downloads. Every cache
137
+ download path now refuses in offline mode (`BaseCache#download` and the
138
+ `download_url` / `download_file` transports raise
139
+ `ResourceNotCachedError`; `Integrity::NetHTTP.get` refuses before opening
140
+ a connection; `LanguageCache#download_spelling` / `#download_grammar`
141
+ raise). The resolve paths (`ResourceManager`, `FrequencyProvider`) read
142
+ via the public cache-only `load_cached` instead of the download-on-miss
143
+ `get`, so the suggestion hot path never triggers a download.
144
+ `Configuration.default` no longer re-applies `DEFAULTS` as explicit
145
+ settings, which clobbered every `KOTOSHU_*` ENV-derived value including
146
+ `KOTOSHU_OFFLINE`. `kotoshu cache download` surfaces offline and
147
+ integrity failures as `ResourceUnavailable` instead of a stack trace.
148
+ - **Kelly frequency loader dead branch** (from suleman-uzair, PR rescue of
149
+ #93) - `CommonWordsLoader.load_from_frequency_file` kept a legacy
150
+ array-format branch that crashed on its own probe and had no producer;
151
+ it now parses the Kelly shape only and degrades unknown shapes to empty
152
+ tiers.
153
+ - Remote `kotoshu setup` for staged languages: the dictionaries repo ships only
154
+ `en` under the `{lang}/spelling/` layout while staged languages sit flat at
155
+ `{lang}/index.*`; downloads now try the sublayout first and fall back to the
156
+ flat layout (verified end-to-end for `it`).
157
+ - `require "kotoshu/tasks"` / `require "kotoshu/jekyll"` standalone now load
158
+ without needing `require "kotoshu"` first.
159
+ - Baseline paths compare canonically: entries recorded as `docs/a.md` match a
160
+ directory-walk check over `./docs/a.md`, and recorded entries are stored
161
+ canonical.
162
+ ### Fixed
163
+ - **Unicode word detection** (plan 91 Track A) - `Spellchecker` word
164
+ extraction accepted only ASCII letters, so Greek and Ukrainian users
165
+ could not check any text through the CLI. Extraction now follows the
166
+ configured language's tokenizer: Greek words for `el`, Cyrillic for
167
+ `uk` (in-word apostrophe included, for names like Мар'яна), and
168
+ Latin-script letters including capitals such as Å and Ä for Latin
169
+ languages. Wrong-script words stay uncheckable; languages without a
170
+ script tokenizer (en, de, ...) keep the historical ASCII behavior;
171
+ the frozen conformance vectors are unchanged.
172
+ - **Swedish aff parsing** (plan 91 Track B) - `Readers::AffReader`
173
+ raised `RegexpError: unmatched close parenthesis` on the Swedish
174
+ dictionary (dictionaries pin 1829a3e). `COMPOUNDRULE )k` uses `)`
175
+ itself as a flag character, but the rule compiler escaped it only in
176
+ some positions. Every flag character is now escaped at compile time,
177
+ so the dictionary loads and its compound rules match.
178
+ ### Added
179
+ - **Framework integrations** (plan 89) - four opt-in integration
180
+ layers, all inside the gem with NO new runtime dependencies
181
+ (soft-dependency pattern, like onnxruntime/suika):
182
+ - `Kotoshu::Validators::SpellingValidator`, an ActiveModel
183
+ EachValidator (`validates :body, spelling: true` after the
184
+ documented one-line alias, or `validates_with` directly): one
185
+ validation error per misspelling with the top suggestion in the
186
+ message; `language:` and `personal_words:` options. Loads the
187
+ real ActiveModel standalone - Rails is not required.
188
+ - RSpec matchers (`require "kotoshu/rspec"` +
189
+ `include Kotoshu::Rspec::Matchers`):
190
+ `expect_words("helo").to all_be_spelled_correctly`, and
191
+ `expect(text) / expect_document(path).to be_spelled_correctly`
192
+ for words or whole documents, with failure messages that list
193
+ each misspelling and its suggestions.
194
+ - Rake task (`require "kotoshu/tasks"`): `rake kotoshu:check`
195
+ over the repository text files using the plan-88 file selection
196
+ (known extensions, .gitignore/.ignore, default skips);
197
+ `Kotoshu::Tasks::CheckTask` exposes files/language/fail_on_error
198
+ /baseline configuration.
199
+ - Jekyll generator (`Kotoshu::Jekyll::Generator`, `safe`):
200
+ checks posts and drafts and fails the build on new spelling
201
+ errors; a `.kotoshu-baseline.json` in the site source is
202
+ respected so baselined errors do not block builds.
203
+ - **CLI directory mode** (plan 88) - `kotoshu check DIR [DIR ...]`
204
+ walks trees and checks every file with a known text extension
205
+ (md markdown asciidoc adoc txt rst mdx) unless `--include` /
206
+ `--exclude` globs say otherwise. `.gitignore` and `.ignore` files
207
+ are honored through a standard glob subset implemented in Ruby
208
+ (`*`, `?`, `**` as a whole segment, `!` negation with last-match
209
+ wins, trailing `/` for directory-only patterns, anchoring, nested
210
+ scoping; files inside ignored directories cannot be re-included).
211
+ Hidden files and `.git` / `node_modules` / `vendor` / `target` are
212
+ skipped by default. Text output prints one section per file plus a
213
+ summary; JSON and SARIF emit one combined document with per-file
214
+ entries/runs; `--baseline` applies per file as shipped; exit codes
215
+ are unchanged; interactive mode stays file-only with an explicit
216
+ notice for directory targets. Single-file and stdin behavior is
217
+ byte-identical to before.
218
+ - **Wave-1 language modules and keyboard layouts** (plan 84) -
219
+ language modules for the thirteen model-registry newcomers (ca cs
220
+ da el hu it nl pl ro sv tr uk vi): thin compositions over the shared
221
+ Latin tokenizer/normalizer plus registry entries, with real care
222
+ where the script needs it — Turkish dotless-i case folding
223
+ (`İZMİR` -> `izmir`, `ISTANBUL` -> `ıstanbul`), Greek-aware
224
+ normalization (final sigma `ΕΛΛΑΣ` -> `ελλας`, opt-in accent
225
+ stripping) and Greek/Cyrillic script tokenizers that keep
226
+ non-Latin words. Keyboard layouts grow from 5 to 19: Turkish-Q,
227
+ Ukrainian-JCUKEN and Greek-Phonetic national grids mirrored from
228
+ the models repo eval harness (a drift spec asserts the gem grids
229
+ cover the eval grids; `scripts/extract_eval_grids.rb` refreshes the
230
+ committed snapshot), plus one parameterized Latin family
231
+ (`Keyboard::Layouts::Latin` and members) over the qwerty/qwertz
232
+ base grids with real Nordic å/æ/ø and å/ä/ö keys. The pre-existing
233
+ five layouts and ten language modules are unchanged.
234
+ - **Inline ignore directives** (plan 82) - `kotoshu:disable-line`,
235
+ `kotoshu:disable-next-line [WORDS]`, and the nestable
236
+ `kotoshu:disable-file` / `kotoshu:enable-file` block, recognized in
237
+ each format's comment syntax: HTML comments in Markdown, `//` line
238
+ comments in AsciiDoc, bare lines in plain text (trailing directives
239
+ in Markdown/AsciiDoc). One shared scanner in `Documents::Suppressions`
240
+ attaches them to every `Document`, and the `Spellchecker` facade
241
+ filters them out of `check`: suppressed words move to
242
+ `DocumentResult#suppressed_errors` (marked `suppressed: true`,
243
+ `suppressed_by: "inline"`), and `kotoshu check --show-suppressed`
244
+ lists them. Result JSON gains `suppressed` / `suppressed_by` on
245
+ errors and `suppressedCount` / `suppressedErrors` at the top level.
246
+ - **CI baselines** (plan 82) - `kotoshu baseline init FILE ...` writes
247
+ `.kotoshu-baseline.json` (canonical `{version, entries: [{file, line,
248
+ word, count}]}` via lutaml-model; count-based, so it survives
249
+ reformatting); `kotoshu check --baseline FILE` lets covered errors
250
+ pass, fails on new ones, and reports stale entries. SARIF output
251
+ marks baseline-suppressed results with a `suppressions` entry
252
+ (`kind: external`, `justification: "baseline"`), and JSON output
253
+ adds a `baseline` block.
254
+ - **pre-commit hook** (plan 81) - `.pre-commit-hooks.yaml` ships hook
255
+ id `kotoshu` (`language: system`, entry `kotoshu check`, text-format
256
+ file pattern); requires Ruby plus the gem, documented honestly in the
257
+ README.
258
+
259
+ ## [0.7.0] — 2026-09-05
260
+
261
+ The universal-kotoshu cut (plan 65's roadmap milestone): tiered
262
+ models, the Rust core, and the native extension.
263
+
264
+ ### Added
265
+
266
+ - **Model tiers** — `full` (~120 MB) / `fluency` (~15 MB, the new
267
+ default) / `mini` (~3 MB), resolved through the models repo's
268
+ registry (`kotoshu://models/{lang}/{tier}`) with sha256-verified
269
+ primary→mirror→vocab downloads; `KOTOSHU_MODEL_TIER`, `setup
270
+ LANG --model --tier`, and a tier-less legacy-cache bridge.
271
+ - **Native extension** (`ext/kotoshu_native`, optional Rust
272
+ accelerator): `KOTOSHU_BACKEND=ruby|native|auto` (default ruby;
273
+ pure-Ruby installs unchanged), `Kotoshu::Native.available?`, and
274
+ `rake kotoshu:conformance:compare` — 2630 vectors, zero divergences
275
+ between the Ruby and Rust engines.
276
+ - **Conformance export** — `rake kotoshu:conformance:export` freezes
277
+ engine behavior as golden vectors (2630 lines) consumed by
278
+ kotoshu-rs CI.
279
+ - **Confidence cascade** — `semantic_cascade_threshold`
280
+ (`KOTOSHU_SEMANTIC_CASCADE_THRESHOLD`): skip the ONNX rerank when
281
+ the composite strategies are already confident.
282
+ - Hunspell correctness: CHECKCOMPOUNDPATTERN replacement support,
283
+ dot-split casing, INITCAP ngram-root skip, AF alias flag-format
284
+ parsing, and a long tail of fixture fixes (integrational suite
285
+ 176 examples, 0 failures, 1 documented pending).
286
+
287
+ ### Fixed
288
+
289
+ - Model downloads cached 134-byte LFS pointer stubs whose own sha256
290
+ verified forever — the model source now fetches from the media host
291
+ and rejects stubs at download and verify.
292
+ - `Embeddings::Vocabulary` crashed on the shipped wrapped vocab
293
+ format; shared normalizer added.
294
+ - `Models::OnnxModel` used a nonexistent ONNX Runtime API and wrong
295
+ IO names — rewritten against `InferenceSession`.
296
+ - CLI JSON output restored to the documented camelCase contract;
297
+ `DictionaryNotFoundError` mapping restored at the cache boundary.
298
+
299
+ ## [0.5.0] — 2026-06-30
300
+
301
+ Tier 2.5 release. Adds the personal-dictionary CLI, completes direct
302
+ spec coverage for every previously-untested namespace (~370 new
303
+ examples), eliminates every `respond_to?` / `send`-to-private /
304
+ `instance_variable_*` violation from `lib/`, and fixes a long list of
305
+ production bugs the refactors surfaced.
306
+
307
+ ### Added
308
+
309
+ - **Cache eviction** (`Kotoshu::Cache::EvictionPolicy` +
310
+ `BaseCache#evict` + `kotoshu cache evict`). A pure value object
311
+ decides which entries to evict (LRU by `cached_at`, oldest first)
312
+ to fit a configured size cap; `BaseCache#evict` collects on-disk
313
+ entries (one record per discovered `metadata.json`) and executes
314
+ the plan. `--dry-run` returns the plan without touching disk. The
315
+ cap defaults to `max_cache_size` (1 GB, `KOTOSHU_MAX_CACHE_SIZE`)
316
+ and is now wired through `Configuration` instead of a hardcoded
317
+ constant. Corrupt metadata (missing `cached_at`) sorts oldest so it
318
+ is evicted first.
319
+ - **Audit log rotation** (`Kotoshu::Integrity::RotationPolicy`). When the
320
+ current `audit.log` exceeds `audit_max_bytes` (default 10 MB,
321
+ `KOTOSHU_AUDIT_MAX_BYTES`), the log rotates through `audit_rotations`
322
+ historical files (default 5, `KOTOSHU_AUDIT_ROTATIONS`). Total on-disk
323
+ footprint is bounded at `max_bytes * (rotations + 1)`. The rotation
324
+ policy is a pure value object — `AuditLog#record` consults it on every
325
+ write and executes the returned rename plan under an exclusive flock
326
+ on a sibling lockfile (`audit.log.lock`) so concurrent writers cannot
327
+ race the rename chain. `AuditLog#entries` now walks both the current
328
+ log and all rotations, newest-first.
329
+ - **Shell completion** (`kotoshu completions bash|zsh|fish`). Emits a
330
+ shell-specific completion script that completes top-level subcommands
331
+ and dynamically completes language codes for `setup` / `fetch` by
332
+ shelling out to `kotoshu completions languages`. README documents
333
+ install paths for bash, zsh, and fish.
334
+ - **Load-time model cache integrity** (`Cache::ModelCache`). A cached
335
+ ONNX model file is re-verified against the SHA-256 recorded in its
336
+ `metadata.json` every time it is loaded — not just at download time.
337
+ Tampering, truncation, or partial writes are caught before the file
338
+ reaches the ONNX runtime, and the resulting `Kotoshu::IntegrityError`
339
+ includes a `remediation:` hint pointing the user at
340
+ `kotoshu cache download :LANG --model`. Legacy caches whose metadata
341
+ predates the checksum field continue to load (graceful degradation).
342
+ - **`IntegrityError#remediation`** — the error now accepts an optional
343
+ `remediation:` keyword whose text is appended to the message and
344
+ exposed via a reader, so callers can surface the user-facing fix
345
+ (e.g. "Run `kotoshu cache download :en --model` to re-download").
346
+ Existing callers that omit it see no change in the message format.
347
+ - **Personal dictionary CLI** (`kotoshu personal SUBCOMMAND`). Thor
348
+ subcommand wrapping `Kotoshu::PersonalDictionary`: `add`, `remove`,
349
+ `list`, `import` (with `--dry-run`), `path`, and `clear` (with
350
+ `--yes`). Errors raise `Kotoshu::Cli::Errors::UsageError` (exit 2),
351
+ matching the existing CLI error contract.
352
+ - **Direct specs for previously-untested namespaces.** ~370 new
353
+ examples pinning the public contract of `Integrity`, `Embeddings`,
354
+ `Language`, `Dictionary`, and `Cache` namespaces — each spec uses
355
+ real instances and tmpdir/fixture IO, never `double()`.
356
+ - **`Kotoshu::Dictionary::Unified`** is now loadable. The class
357
+ was previously declared as `class Kotoshu::Dictionary`, colliding
358
+ with the existing `module Kotoshu::Dictionary` namespace and
359
+ raising `TypeError` on autoload. Renamed to `Unified` to match the
360
+ autoload entry. Two follow-on bugs (mismatched `Suggester.new`
361
+ kwargs and a custom `build_dic_structure` that produced the wrong
362
+ shape) fixed in the same change.
363
+
364
+ ### Changed
365
+
366
+ - **`Embeddings::LruCache#fetch`** now yields the *key* to the provided
367
+ block (matching Ruby's `Hash#fetch` convention) instead of invoking
368
+ the block with no arguments. Callers that compute the missing value
369
+ from the key no longer need a closure over an outer variable.
370
+ - **`Embeddings::SimilarityEngine`** exposes a `pre_normalize?`
371
+ predicate (replacing a dead `attr_reader :pre_normalize` that was
372
+ silently shadowed by a private method of the same name) and now
373
+ raises `ArgumentError` with a clear dimension-mismatch message
374
+ instead of crashing with a `TypeError` from `Array#zip` when given
375
+ vectors of differing lengths. `nil` and empty inputs still return
376
+ `0.0` (early-return path).
377
+ - **Eliminated `respond_to?` across `lib/`.** Replaced every
378
+ occurrence with `is_a?` / `case-when` / `method_defined?` / direct
379
+ calls. Duck-typing via `respond_to?` hid real type errors — the
380
+ production bugs surfaced by this refactor (see Fixed) had been
381
+ silently no-op'ing for as long as the code existed.
382
+ - **Eliminated `send` to private/protected methods** in `lib/`. The
383
+ `cache` CLI subcommand's `cache.send(:read_metadata, ...)` is now
384
+ `cache.read_metadata(...)` after promoting the method to public
385
+ (`public def`). `SemanticStrategy#embedding_for` was dead code that
386
+ called `@search.send(:get_embedding, ...)` on a method that didn't
387
+ exist — removed entirely.
388
+ - **Eliminated `instance_variable_set/get` across `lib/`.**
389
+ `BloomFilter#merge` now reads `other.bits` (public `attr_reader`)
390
+ instead of `other.instance_variable_get(:@bits)`.
391
+ `Dictionary::PlainText` and `Dictionary::CSpell` constructors now
392
+ accept a `words:` keyword instead of `from_words` poking ivars via
393
+ `allocate` + `instance_variable_set`. `Readers::LookupBuilder`
394
+ gained `aff_data:` / `words:` keyword args to replace the same
395
+ pattern in `from_data`.
396
+ - **Configuration dispatch is type-checked, not duck-typed.**
397
+ `Configuration#apply_*` previously called
398
+ `send("#{key}=", value) if respond_to?("#{key}=")`; now uses
399
+ `public_send(setter, value) if self.class.method_defined?(setter)`.
400
+ - **`Dictionary::Base` exposes a default `path` method** returning
401
+ `nil` (overridden by file-backed subclasses). `Spellchecker`
402
+ previously guarded with `dictionary.respond_to?(:path)`; now calls
403
+ unconditionally.
404
+ - **`Language::Registry.info` checks the type bound.** Was
405
+ `klass.respond_to?(:instance)`; now
406
+ `klass.is_a?(Class) && klass <= Kotoshu::Language::Base`.
407
+
408
+ ### Fixed
409
+
410
+ - **`AuditLog#each` now iterates newest-first across current and
411
+ rotations.** The docstring promised newest-first but the
412
+ implementation yielded each file in write order (oldest line first)
413
+ then walked current → oldest file, producing a jumbled order. Each
414
+ file's lines are now read into memory and yielded in reverse before
415
+ walking to the next older rotation.
416
+ - **`Embeddings::Protocol` is no longer broken.** The module declared
417
+ `required_methods` and `required_methods(*names)` (a reader + a
418
+ writer with the same name). Ruby's last-def-wins shadowed the
419
+ reader, so every protocol registration silently dropped its
420
+ arguments and `compliance_errors` returned `[]` unconditionally —
421
+ every class appeared to conform to every protocol. Renamed the
422
+ writer to `required` (and `optional_methods` → `optional`); also
423
+ fixed `compliance_errors` to use `klass.method_defined?` instead of
424
+ `klass.respond_to?` (which checks class methods, not instance
425
+ methods).
426
+ - **`Embeddings::OnnxRuntimeModel` defines `loaded?`.** The protocol
427
+ requires `loaded?` but the class declared only `attr_reader :loaded`
428
+ (no question mark). `EmbeddingPipeline#stats` and `#to_s` called
429
+ `@model.loaded?` — would have raised `NoMethodError` on the real
430
+ model.
431
+ - **`Dictionary::Base` aliases dispatch dynamically.** Ruby's `alias`
432
+ keyword binds to the parent method body at definition time,
433
+ bypassing subclass overrides. `has_word?`/`include?`/`contains?`/
434
+ `lookup?`/`<<`/`all_words` on a concrete subclass (e.g.,
435
+ `UnixWords`) would invoke the abstract `Base#lookup` /
436
+ `Base#add_word` and raise `NotImplementedError`. Replaced with
437
+ one-line method definitions that call the abstract method (which
438
+ then dispatches dynamically to the subclass override).
439
+ - **`Dictionary::CSpell#add_word` no longer crashes.** Tried to
440
+ `@trie.insert` on a frozen `Trie` (frozen by `Builder#build`),
441
+ raising `FrozenError`. Aligned with `remove_word`'s existing
442
+ "always returns false (immutable after load)" contract.
443
+ - **`Cache::FrequencyCache#get_frequency` no longer raises
444
+ `ArgumentError`.** Called `get(language_code, force_download)`
445
+ positionally, but `BaseCache#get` declares `force_download:` as
446
+ keyword. Fixed to use the keyword form.
447
+ - **`Cache::FrequencyCache#cached_resources` no longer reports the
448
+ `tmp/` scratch slot.** `BaseCache#initialize` creates
449
+ `<cache_path>/tmp/` as a working directory; the directory-walking
450
+ implementation in `FrequencyCache` listed it as a cached language.
451
+ Added an explicit `basename != "tmp"` exclusion.
452
+ - **`Cache::FrequencyCache#install_local` exists.** `ResourceManager`
453
+ called `freq_cache.install_local(...)` behind a `respond_to?` guard
454
+ that always returned `false`, so local frequency installs were
455
+ silently skipped — `setup(from:)` reported `frequency_status: :local`
456
+ even though no install happened. Added the method, mirroring
457
+ `LanguageCache#install_local` but taking a single `path:` kwarg.
458
+ - **`PhonetTable::Rule#match?` is no longer dead code.**
459
+ `Algorithms::PhonetSuggest` had its own `match_rule` that
460
+ duplicated the rule-matching logic, ignoring `Rule#match?`
461
+ entirely. Consolidated into `Rule#match_length` (returning
462
+ `Integer|nil` — the metaphone walker needs the length to advance
463
+ its position) with `Rule#match?` as a boolean wrapper;
464
+ `PhonetSuggest#match_rule` now delegates.
465
+ - **`DoubleNegativeMatcher` recognizes the "not only...but also"
466
+ correlative.** The previous check fired only on the narrow shape
467
+ `not [current_negative] only`, missing the actual idiom (where
468
+ "not" and "only" are adjacent). Replaced with a region-based
469
+ matcher that finds every exception phrase in the token stream and
470
+ suppresses any negative whose index falls inside a region. Supports
471
+ both `"prefix...suffix"` (open-ended) and `"literal n-gram"` (no
472
+ ellipsis) phrase shapes, driven by the rule's `exceptions.phrases`
473
+ config.
474
+ - **`SemanticAnalyzer` no longer returns `[]` for OOV words.**
475
+ `suggest_corrections` called `@model.nearest_neighbors(word)`,
476
+ which returns `[]` for OOV inputs — but OOV is precisely what the
477
+ analyzer exists to handle. Added an edit-distance fallback: when
478
+ the model returns `[]`, walk the vocabulary and return words within
479
+ Levenshtein distance 2, scored by `1/(1+distance)`. Drive-by fix:
480
+ `Models::Suggestion.new` was called with `word:` as keyword but
481
+ the constructor takes `word` positional — exposed once the OOV
482
+ fallback actually reached the `Suggestion.new` call.
483
+
484
+ ### Removed
485
+
486
+ - **Orphan files emptied to no-op stubs.**
487
+ `lib/kotoshu/embeddings/protocols.rb` (plural) had
488
+ `require_relative 'protocols/embedding_model'` pointing at
489
+ nonexistent files (loading it raised `LoadError`).
490
+ `lib/kotoshu/readers/readers.rb` (nested) duplicated the autoload
491
+ setup in `lib/kotoshu/readers.rb` with `require_relative` calls
492
+ (violating the project rule against `require_relative` in lib/).
493
+ Both reduced to documentation-only stubs; no callers were affected.
494
+ The real protocol definitions live in
495
+ `lib/kotoshu/embeddings/protocol.rb`; the real reader autoloads
496
+ live in `lib/kotoshu/readers.rb`.
497
+
498
+ ## [0.4.0] — 2026-06-29
499
+
500
+ Tier 2 release. Completes the Hunspell correctness work (morphological
501
+ rules, REP/phonet/ngram ranking, compound handling), migrates the entire
502
+ library from `require_relative` to Ruby `autoload`, and tightens the
503
+ cache interface alignment that 0.3.1 sketched.
504
+
505
+ ### Added
506
+
507
+ - **Hunspell morphological correctness** (T2 Phases 2A–2D):
508
+ - `FLAG long|num|utf-8` parsing and `AF` alias resolution.
509
+ - `COMPOUNDRULE`, `CHECKCOMPOUNDPATTERN` (with empty-flag fix),
510
+ `CIRCUMFIX` handling.
511
+ - `ICONV`/`OCONV` `ConvTable` with stable sort — preserves Nepali
512
+ `ZWNJ`/`ZWJ` ordering.
513
+ - `KEEPCASE` suggestion flag with `CHECKSHARPS` exception, plus the
514
+ German eszet rule (`ss` ↔ `SS`).
515
+ - `REP` table, phonet ordering, ngram ranking wired end-to-end.
516
+ - Edge cases: `IJ` digraph, `i58202` case preservation,
517
+ `opentaal_forbiddenword2`, `breakdefault` `BREAK` pattern.
518
+ - **Hunspell Suggester** is now wired into the Hunspell dictionary; the
519
+ default affix directives are honoured when generating suggestions.
520
+ - **`BaseCache#max_cache_size`** attribute (1 GB default) — the
521
+ forward-looking hook for cache eviction work (see
522
+ `TODO.impl/34-cache-eviction.md`).
523
+ - **`LanguageCache#language_path(lang, type)`** public helper; resource
524
+ and metadata path resolution is now DRY'd through it.
525
+ - **`Kotoshu::Language` suika soft-load module** with specs — keeps
526
+ tokenization lazy and optional.
527
+ - **Multi-language EditDistanceStrategy spec coverage** — pins
528
+ per-language keyboard selection (QWERTZ/AZERTY/QWERTY/JCUKEN) and
529
+ verifies German and French typo correction end-to-end. The previous
530
+ `skip 'Multi-language support not yet implemented'` hook is removed;
531
+ the strategy has been language-aware via `Keyboard::Registry` +
532
+ `Cache::FrequencyCache` for several releases.
533
+
534
+ ### Changed
535
+
536
+ - **Library-wide `autoload` migration.** Every `require_relative` (and
537
+ in-library `require`) under `lib/kotoshu/` is replaced with a Ruby
538
+ `autoload` declared in the immediate parent namespace's file. This
539
+ fixes load-order races, makes the eager/opt-in split explicit, and
540
+ matches the contributor guidance in `CLAUDE.md`. Behaviour is
541
+ unchanged for callers who only `require "kotoshu"`, but extenders who
542
+ reached past the public facade may need to update their load entry
543
+ points.
544
+ - **`Embeddings` namespace consolidation.** All embedding classes are
545
+ nested under `Kotoshu::Embeddings` (was scattered with several
546
+ `require_relative` shims).
547
+ - **Grammar rules bundled in the gem.** English `rules.yaml` ships
548
+ inside the gem package so a `gem install kotoshu` is self-contained
549
+ for grammar checking without a separate download.
550
+ - **Plain-text edit distance is now case-insensitive**, matching the
551
+ Hunspell backend's behaviour for `PlainText` dictionaries.
552
+
553
+ ### Fixed
554
+
555
+ - **`Cli::CacheCommand` wired to the real `LanguageCache` API.** The
556
+ previous implementation called nonexistent methods (`cache_status`,
557
+ `get_frequency_data`, `get_language_info`, `purge_all`); `kotoshu cache`
558
+ now correctly routes through `available?`, `cached_resources`,
559
+ `clear_all`, `clean`, `stats`, `get_spelling`, `get_grammar`, and the
560
+ standalone `Cache::FrequencyCache` for frequency downloads.
561
+ - **`Keyboard::Registry` lazy reload.** `clear` now marks
562
+ `@languages_loaded = true` to prevent an infinite lazy-reload loop
563
+ when the registry is reset at runtime.
564
+ - **German common-words YAML.** `data/common_words/de.yml` line 466 had
565
+ `-ecke` (missing space after dash), a syntax error that prevented the
566
+ file from loading and silently broke German strategy instantiation.
567
+ - **`Readers::AffReader` Hunspell edge cases** — encoding fallback for
568
+ Latin-1 fixtures, `REP` table splitting, `MultiWord` dash variants,
569
+ default affix directives honoured when the `.aff` file omits them.
570
+ - **RuboCop 3.x / plugins migration.** `rubocop-rspec` is bumped to
571
+ 3.x and the config uses the new plugins syntax; safe auto-corrections
572
+ applied across `lib/` and `spec/` (`Layout/ExtraSpacing`,
573
+ `Layout/EmptyLine`).
574
+ - **Cross-platform CI.** `bundle` is invoked via `Gem.ruby -S` for
575
+ Windows compatibility; Ruby 4.0 compatibility fixes applied.
576
+
577
+ ### Internal
578
+
579
+ - `TODO.impl/` planning documents (00–40) are the tiered execution
580
+ plan. Tier 2 is complete; Tier 3 (`audit-log-rotation`,
581
+ `cache-eviction`, `shell-completion`) and content-repo tasks
582
+ (`onnx-vocab-json-generation`) remain deferred past 0.4.0.
583
+
584
+ ## [0.3.1] — 2026-06-28
585
+
586
+ Bug-fix release. Closes the regressions introduced by the 0.3.0
587
+ lutaml-model migration and tightens up spec gating so the default suite
588
+ is green on the non-network, non-ONNX subset.
589
+
590
+ ### Fixed
591
+
592
+ - `WordResult#suggestions` is `Array<Suggestion>` after the lutaml
593
+ migration; the integration spec now reads it as
594
+ `suggestions.map(&:word)` instead of the removed `SuggestionSet#words`.
595
+ - `WordResult#to_h` was removed by lutaml; the integration spec now uses
596
+ the framework-supplied `to_hash`.
597
+ - `Readers::AffReader#detect_encoding` now handles a nil path so the
598
+ reader can be constructed from an in-memory `StringReader` (used by
599
+ the unit tests and by programmatic callers that build affix data
600
+ without a file on disk).
601
+ - `Embeddings::SimilaritySearch` is now eagerly loaded alongside the
602
+ rest of the embeddings namespace; previously the constant was
603
+ uninitialized when `SemanticStrategy` tried to use it.
604
+ - `Embeddings::Vocabulary.from_cache` is implemented (was missing). It
605
+ resolves the `vocab.json` sibling of the cached ONNX model and
606
+ returns nil when the model or vocab is unavailable, so callers can
607
+ degrade gracefully.
608
+ - `Embeddings::OnnxRuntimeModel.from_cache` now accepts the `cache:`
609
+ keyword (matching `Vocabulary.from_cache` and
610
+ `SimilaritySearch.from_cache`); the previous positional signature
611
+ silently swallowed the cache as a Hash.
612
+ - `Suggestions::Strategies::SemanticStrategy` now surfaces its named
613
+ constructor kwargs (`min_semantic_similarity`,
614
+ `semantic_boost_weight`, `max_context_window`) through `get_config`,
615
+ so the strategy's configurable knobs are queryable the same way as
616
+ the base strategy's `**config` bag.
617
+
618
+ ### Changed
619
+
620
+ - `spec/spec_helper.rb` excludes `:onnx`-tagged examples unless
621
+ `ONNX_TESTS=1` is set, mirroring the existing `:network` and `:slow`
622
+ opt-in pattern. `spec/kotoshu/suggestions/strategies/semantic_strategy_spec.rb`
623
+ is the sole consumer; its `:integration` tags are now `:onnx`.
624
+ - The SymSpell benchmark spec and the edit-distance / symspell
625
+ performance describe blocks are tagged `:slow`, removing
626
+ environment-dependent timing flakes from the default run.
627
+ - Arabic language detection is marked `pending` with a pointer to
628
+ `TODO.impl/30-language-auto-detection.md` (FastText LID missing the
629
+ Arabic vector).
630
+ - `edit_distance_strategy_spec` now uses the `from_source?` predicate
631
+ instead of asserting `source == :edit_distance`, matching the
632
+ lutaml-era contract that `Suggestion#source` is a string.
633
+
634
+ ### Internal
635
+
636
+ - Added `TODO.impl/36-cleanup-regressions-0.3.1.md` (this release),
637
+ `TODO.impl/37-hunspell-correctness-tier2.md`,
638
+ `TODO.impl/38-onnx-semantic-gating.md`, and
639
+ `TODO.impl/39-tier3-and-beyond.md` as the tiered execution plan.
640
+
641
+ ## [0.3.0] — 2026-06-27
642
+
643
+ The two-stage release. Resources are now downloaded explicitly via
644
+ `Kotoshu.setup(:en)`; the hot path (`correct?`, `suggest`, `check`) reads
645
+ only from cache and raises a typed error when a language is missing instead
646
+ of triggering a network download. The CLI adds `setup`, `status`, language
647
+ auto-detection, SARIF/JSON output, and an interactive auto-setup prompt.
648
+ `onnxruntime` is now a soft dependency, so `gem install kotoshu` succeeds on
649
+ hosts that can't load native ONNX runtime.
650
+
651
+ ### Added
652
+
653
+ - **Two-stage resource model** (`Kotoshu::ResourceManager`):
654
+ `Kotoshu.setup(:en, want: %i[spelling frequency model])` writes into the
655
+ cache; `Kotoshu::ResourceManager.resolve(language:, want:)` is instant and
656
+ cache-only, raising `ResourceNotSetupError` on miss. `Kotoshu.setup?` is
657
+ the predicate for "is this language already cached?". The library never
658
+ triggers a surprise download; the CLI prompts the user via `AutoSetup`.
659
+ - **`SourceRegistry`** — single source of truth for the three content repos'
660
+ URLs and per-repo pins. `kotoshu/dictionaries` is pinned to the `v1`
661
+ branch; `frequency-list-kelly` and `models-fasttext-onnx` are on `main`.
662
+ Override at runtime via `KOTOSHU_REPOS_BASE_URL`, `KOTOSHU_DICTIONARIES_PIN`,
663
+ `KOTOSHU_FREQUENCY_PIN`, `KOTOSHU_MODELS_PIN`.
664
+ - **XDG Base Directory layout** (`Kotoshu::Paths`): dictionaries, frequency
665
+ lists, ONNX models under `$XDG_CACHE_HOME/kotoshu/`; personal dictionary
666
+ and `kotoshu.cfg` under `$XDG_CONFIG_HOME/kotoshu/`; audit log under
667
+ `$XDG_DATA_HOME/kotoshu/audit.log`. Override per-axis with
668
+ `KOTOSHU_CACHE_PATH`, `KOTOSHU_CONFIG_PATH`, `KOTOSHU_DATA_PATH`.
669
+ - **Integrity verification** — `Kotoshu::Integrity::Manifest` (SHA-256) is
670
+ fetched per content repo and matched against every download. Mismatches
671
+ raise `Kotoshu::IntegrityError`. Outcomes (verified / unverified / mismatch)
672
+ are written to the audit log. Missing manifests degrade gracefully.
673
+ - **CLI `setup` command** — `kotoshu setup LANG [--force] [--no-frequency]
674
+ [--no-model]` writes the requested resources into the cache with progress
675
+ reporting.
676
+ - **CLI `status` command** — `kotoshu status [--json]` summarises installed
677
+ resources, sizes, mtimes, and ONNX runtime availability.
678
+ - **CLI `check --language auto`** — auto-detects document language via
679
+ FastText LID; falls back to the configured default language when detection
680
+ is unavailable or the detected language is not set up.
681
+ - **CLI `check --format json|sarif`** — machine-readable output. SARIF
682
+ follows v2.1.0 with `kotoshu/spelling` rule id, JSON exposes
683
+ `success`/`wordCount`/`errorCount`/`uniqueErrorCount`/`errors`/`source`.
684
+ - **CLI auto-setup prompt** — when the hot path raises
685
+ `ResourceNotSetupError` in an interactive session, the user is prompted to
686
+ run setup now and the original command is retried on success. Non-TTY,
687
+ offline (`--offline`), and `--no-prompt` invocations skip the prompt and
688
+ surface the error as before.
689
+ - **Download progress reporting** (`Kotoshu::Cli::ProgressReporter`) — TTY
690
+ mode renders a determinate/indeterminate progress bar; non-TTY mode prints
691
+ a periodic line every 10 MiB. `Kotoshu.configuration.download_reporter=`
692
+ exposes the reporter for programmatic use.
693
+ - **End-to-end smoke spec** (`spec/integration/end_to_end_spec.rb`) covers
694
+ install → setup → `correct?` → `suggest.to_words` → `check` →
695
+ `setup?` predicate → `ResourceNotSetupError` → idempotent re-setup.
696
+ Tagged `:network`, opted into via `NETWORK_TESTS=1`.
697
+ - **CLI format spec** (`spec/kotoshu/cli/check_format_spec.rb`) shells out to
698
+ the real `kotoshu` CLI and asserts JSON / SARIF structure and exit codes.
699
+
700
+ ### Changed
701
+
702
+ - **`onnxruntime` is a soft dependency.** Removed from `kotoshu.gemspec`.
703
+ `Kotoshu::Models::OnnxModel` soft-requires it at load time and exposes
704
+ `ONNX_LOADED`. When false, semantic methods raise
705
+ `Kotoshu::Models::OnnxModel::OnnxUnavailable` with a caller-friendly
706
+ message. `KOTOSHU_NO_ONNX=1` forces semantic off even when the gem is
707
+ present. The traditional spell-checking path never touches `onnxruntime`.
708
+ - **Loading strategy** — `lib/kotoshu.rb` eagerly loads only the facade
709
+ dependencies; heavier or optional pieces (ONNX models, interactive CLI,
710
+ caches, language detection) are wired through Ruby `autoload` registered
711
+ in their immediate parent namespace.
712
+ - **Public API** — `suggest` returns a `SuggestionSet`; call `.to_words` for
713
+ an `Array<String>`. `Kotoshu.check` returns a `DocumentResult`; iterate
714
+ `errors` for `WordResult` instances with `word`, `position`, `line`,
715
+ `column`, `suggestions`.
716
+ - **README quickstart** — reflects the two-stage API; documents XDG paths;
717
+ marks `onnxruntime` as optional.
718
+
719
+ ### Fixed
720
+
721
+ - `gem install kotoshu` no longer requires `onnxruntime` or its native
722
+ toolchain.
723
+ - Resource resolution no longer triggers downloads from inside the hot path.
724
+ - Per-repo pins are honoured — the `v1` branch of `kotoshu/dictionaries` is
725
+ fetched instead of `main`.
726
+
727
+ ### Known limitations (carried from 0.1.0, scope reduced)
728
+
729
+ - **Hunspell correctness**: compound rules, circumfix, ICONV/OCONV, German ß,
730
+ Turkish dotless-i remain partial. See `TODO.impl/01-hunspell-correctness.md`.
731
+ - **CJK and RTL**: tokenizer, normalizer, and keyboard layouts exist for
732
+ supported languages; full CJK/RTL support deferred past 0.3.
733
+ See `TODO.impl/06-cjk-support.md` and `TODO.impl/07-rtl-support.md`.
734
+ - **Grammar rules**: the rule engine exists; no rule packs are shipped.
735
+ See `TODO.impl/08-grammar-engine.md`.
736
+ - **Audit log rotation, cache eviction policy, and shell completion** are
737
+ deferred past 0.3 (T3 TODOs).
738
+
739
+ ### Internal
740
+
741
+ - 9 logical commits on `release-0.3` cover the T1 (architectural) and T2
742
+ (user-facing) work for this release.
743
+ - `SourceRegistry`, `Paths`, `ResourceManager`, `ResourceBundle`,
744
+ `SetupResult`, `Integrity::Manifest`, `Integrity::AuditLog`,
745
+ `Cli::AutoSetup`, `Cli::StatusReport`, `Cli::LanguageResolver`,
746
+ `Cli::ProgressReporter` are new model-driven types.
747
+ - 73 new specs added (source_registry, end_to_end, check_format,
748
+ progress_reporter, language_resolver, status_report, auto_setup).
749
+
750
+ ### Contributors
751
+
752
+ - Ribose Inc.
753
+
754
+ ## [0.1.0] — 2026-06-25
755
+
756
+ First public release. Kotoshu is a pure-Ruby spellchecker that combines a
757
+ Ruby port of the Hunspell algorithm with optional FastText ONNX embeddings
758
+ for semantic reranking. This release establishes the public Ruby API, the
759
+ basic CLI, and the cache layer.
760
+
761
+ ### Working
762
+
763
+ - **Ruby API**: `Kotoshu.correct?`, `Kotoshu.suggest`, `Kotoshu.check`,
764
+ `Kotoshu.check_file`, `Kotoshu.detect_language`
765
+ - **CLI**: `kotoshu check TARGET`, `kotoshu dict SUBCOMMAND`, `kotoshu cache
766
+ SUBCOMMAND`, `kotoshu version`
767
+ - **Dictionary backends**: Hunspell (`.aff`/`.dic`), CSpell, UnixWords
768
+ (`/usr/share/dict/words`), PlainText, Custom
769
+ - **Suggestion strategies**: edit distance, phonetic (Phonet), keyboard
770
+ proximity, n-gram, symspell, composite pipeline
771
+ - **Configuration**: `Kotoshu.configure`, CLI > ENV (`KOTOSHU_*`) >
772
+ programmatic > defaults via `Configuration::Resolver`
773
+ - **Cache layer**: `LanguageCache`, `FrequencyCache`, `ModelCache` with TTLs
774
+ and download from `kotoshu/dictionaries`, `kotoshu/frequency-list-kelly`,
775
+ `kotoshu/models-fasttext-onnx`
776
+ - **Language detection**: FastText LID, 127 languages
777
+ - **Documents**: Plain text, Markdown (Kramdown), AsciiDoc (Asciidoctor)
778
+ - **Test suite**: 803 of 866 examples passing (92.7%), 6 pending
779
+
780
+ ### Known limitations (not blocking 0.1)
781
+
782
+ - **Hunspell correctness**: compound rules, circumfix, ICONV/OCONV, German
783
+ ß, Turkish dotless-i are not fully implemented. Single-word lookup and
784
+ basic affixes work. See `TODO.impl/01-hunspell-correctness.md`.
785
+ - **CLI surface**: `--interactive`, `--format sarif|json|yaml|csv`,
786
+ `--model fasttext|hybrid`, `--language auto` exist in
787
+ `lib/kotoshu/commands/check_command.rb` but are not wired through
788
+ `exe/kotoshu`. See `TODO.impl/02-cli-unification.md`.
789
+ - **Semantic path**: gated behind `ENV['KOTOSHU_REQUIRE_ONNX']` because
790
+ `onnxruntime` loads eagerly otherwise. Hybrid mode is not the default.
791
+ See `TODO.impl/05-semantic-path.md`.
792
+ - **Dynamic resolution**: the three caches exist independently; there is
793
+ no unified `ResourceManager` that takes arbitrary text and yields the
794
+ full resource bundle. See `TODO.impl/03-dynamic-download.md`.
795
+ - **Languages**: code is wired for English by default. The
796
+ `dictionaries` repo has 98 language directories but the gem's
797
+ `lib/kotoshu/languages/` has only 7 modules (de, en, es, fr, ja, pt,
798
+ ru). See `TODO.impl/04-language-modules.md`.
799
+ - **CJK, RTL**: not implemented. See `TODO.impl/06-cjk-support.md`
800
+ and `TODO.impl/07-rtl-support.md`.
801
+ - **Grammar rules**: the rule engine exists; no rule packs are shipped.
802
+ See `TODO.impl/08-grammar-engine.md`.
803
+ - **Integrity verification**: downloaded resources are not currently
804
+ checksummed. See `TODO.impl/09-integrity-security.md`.
805
+
806
+ ### Internal
807
+
808
+ - 12 plans under `TODO.impl/` define the path to 1.0
809
+ - Architecture documentation consolidated under `docs/`
810
+
811
+ ### Contributors
812
+
813
+ - Ribose Inc.