kotoshu 1.0.2-x86_64-linux

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (458) hide show
  1. checksums.yaml +7 -0
  2. data/.kotoshu-baseline.json +1 -0
  3. data/.pre-commit-hooks.yaml +18 -0
  4. data/.rspec +3 -0
  5. data/.rubocop.yml +351 -0
  6. data/.yardopts +8 -0
  7. data/CHANGELOG.md +813 -0
  8. data/CLAUDE.md +176 -0
  9. data/CODE_OF_CONDUCT.md +132 -0
  10. data/Cargo.lock +359 -0
  11. data/Cargo.toml +15 -0
  12. data/LICENSE +31 -0
  13. data/README.adoc +1292 -0
  14. data/Rakefile +150 -0
  15. data/SECURITY.md +93 -0
  16. data/TODO.impl/00-cut-0.1.md +121 -0
  17. data/TODO.impl/00-cut-0.2.md +175 -0
  18. data/TODO.impl/00-vision.md +281 -0
  19. data/TODO.impl/01-hunspell-correctness.md +111 -0
  20. data/TODO.impl/02-cli-unification.md +74 -0
  21. data/TODO.impl/03-dynamic-download.md +107 -0
  22. data/TODO.impl/04-language-modules.md +89 -0
  23. data/TODO.impl/05-semantic-path.md +83 -0
  24. data/TODO.impl/06-cjk-support.md +105 -0
  25. data/TODO.impl/07-rtl-support.md +71 -0
  26. data/TODO.impl/08-grammar-engine.md +106 -0
  27. data/TODO.impl/09-integrity-security.md +90 -0
  28. data/TODO.impl/10-testing-ci.md +98 -0
  29. data/TODO.impl/100-full-feature-batch3.md +28 -0
  30. data/TODO.impl/101-int4-tier-experiment.md +32 -0
  31. data/TODO.impl/102-lid-chain.md +28 -0
  32. data/TODO.impl/103-oov-bucket-tables.md +27 -0
  33. data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
  34. data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
  35. data/TODO.impl/106-native-lid-detection.md +36 -0
  36. data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
  37. data/TODO.impl/108-ko-ne-full-feature.md +37 -0
  38. data/TODO.impl/109-ten-audit.md +26 -0
  39. data/TODO.impl/11-release-v1.md +95 -0
  40. data/TODO.impl/110-sr-latn-and-polish.md +33 -0
  41. data/TODO.impl/111-embedding-bakeoff.md +33 -0
  42. data/TODO.impl/112-deployability-and-gating.md +26 -0
  43. data/TODO.impl/113-language-packs.md +26 -0
  44. data/TODO.impl/114-typo-biencoder-thread.md +33 -0
  45. data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
  46. data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
  47. data/TODO.impl/117-deterministic-frequency-tiers.md +110 -0
  48. data/TODO.impl/118-server-gem-floor.md +33 -0
  49. data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
  50. data/TODO.impl/120-setup-metadata-refresh.md +35 -0
  51. data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
  52. data/TODO.impl/122-pack-compression-kpk2.md +66 -0
  53. data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
  54. data/TODO.impl/124-owner-residue.md +19 -0
  55. data/TODO.impl/125-consumer-floor-guards.md +43 -0
  56. data/TODO.impl/126-tier-error-budgets.md +33 -0
  57. data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
  58. data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
  59. data/TODO.impl/129-pt-fr-corpora.md +23 -0
  60. data/TODO.impl/130-news-truth-0912.md +23 -0
  61. data/TODO.impl/131-hybrid-productization.md +42 -0
  62. data/TODO.impl/132-techniques-page-and-prose.md +27 -0
  63. data/TODO.impl/133-rust-first-distribution.md +68 -0
  64. data/TODO.impl/20-encapsulation-cleanup.md +47 -0
  65. data/TODO.impl/21-autoload-migration.md +64 -0
  66. data/TODO.impl/22-url-pin-single-source-of-truth.md +120 -0
  67. data/TODO.impl/23-fix-url-defaults.md +57 -0
  68. data/TODO.impl/24-cli-auto-setup-prompt.md +76 -0
  69. data/TODO.impl/25-soft-onnxruntime-require.md +75 -0
  70. data/TODO.impl/26-onnx-vocab-json-generation.md +96 -0
  71. data/TODO.impl/27-end-to-end-smoke-spec.md +113 -0
  72. data/TODO.impl/28-readme-quickstart-verification.md +68 -0
  73. data/TODO.impl/29-status-command.md +84 -0
  74. data/TODO.impl/30-language-auto-detection.md +74 -0
  75. data/TODO.impl/31-download-progress.md +58 -0
  76. data/TODO.impl/32-sarif-output-verification.md +72 -0
  77. data/TODO.impl/33-audit-log-rotation.md +54 -0
  78. data/TODO.impl/34-cache-eviction.md +59 -0
  79. data/TODO.impl/35-shell-completion.md +61 -0
  80. data/TODO.impl/36-cleanup-regressions-0.3.1.md +134 -0
  81. data/TODO.impl/37-hunspell-correctness-tier2.md +203 -0
  82. data/TODO.impl/38-onnx-semantic-gating.md +218 -0
  83. data/TODO.impl/39-tier3-and-beyond.md +232 -0
  84. data/TODO.impl/40-spec-drift-cleanup.md +134 -0
  85. data/TODO.impl/50-structure-aware-document-api.md +194 -0
  86. data/TODO.impl/51-grammar-engine-expansion.md +75 -0
  87. data/TODO.impl/52-multi-language-documents.md +95 -0
  88. data/TODO.impl/53-document-plugin-architecture.md +94 -0
  89. data/TODO.impl/54-cjk-support.md +117 -0
  90. data/TODO.impl/55-rtl-support.md +109 -0
  91. data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
  92. data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
  93. data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
  94. data/TODO.impl/59-yard-api-documentation.md +75 -0
  95. data/TODO.impl/60-lsp-server.md +174 -0
  96. data/TODO.impl/61-editor-ecosystem.md +167 -0
  97. data/TODO.impl/62-framework-integrations.md +159 -0
  98. data/TODO.impl/63-web-playground-wasm.md +186 -0
  99. data/TODO.impl/64-http-api-and-sdks.md +191 -0
  100. data/TODO.impl/65-universal-kotoshu.md +251 -0
  101. data/TODO.impl/66-kotoshu-core.md +169 -0
  102. data/TODO.impl/67-kotoshu-rs-and-access-libraries.md +137 -0
  103. data/TODO.impl/68-sota-adoption.md +152 -0
  104. data/TODO.impl/69-sota-typo-eval.md +74 -0
  105. data/TODO.impl/70-sota-confidence-cascade.md +59 -0
  106. data/TODO.impl/71-sota-nested-model.md +63 -0
  107. data/TODO.impl/72-model2vec-evaluation.md +49 -0
  108. data/TODO.impl/73-rl-reranker.md +42 -0
  109. data/TODO.impl/74-distilled-reranker.md +36 -0
  110. data/TODO.impl/75-llm-evaluator.md +35 -0
  111. data/TODO.impl/76-optional-llm-backend.md +37 -0
  112. data/TODO.impl/77-model-coverage-expansion.md +96 -0
  113. data/TODO.impl/78-in-browser-playground.md +69 -0
  114. data/TODO.impl/79-news-and-releases.md +60 -0
  115. data/TODO.impl/80-ecosystem-docs.md +56 -0
  116. data/TODO.impl/81-editor-reach.md +68 -0
  117. data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
  118. data/TODO.impl/83-models-batch-2.md +56 -0
  119. data/TODO.impl/84-language-modules-wave2.md +75 -0
  120. data/TODO.impl/85-wasm-model-api.md +48 -0
  121. data/TODO.impl/86-site-truth-wave2.md +45 -0
  122. data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
  123. data/TODO.impl/88-cli-directory-mode.md +54 -0
  124. data/TODO.impl/89-framework-integrations-exec.md +59 -0
  125. data/TODO.impl/90-gem-correctness-residues.md +44 -0
  126. data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
  127. data/TODO.impl/92-tier-mirror-cors.md +88 -0
  128. data/TODO.impl/93-python-wheels.md +32 -0
  129. data/TODO.impl/94-site-integrations-docs.md +35 -0
  130. data/TODO.impl/95-editor-action-polish.md +27 -0
  131. data/TODO.impl/96-server-semantic-models.md +50 -0
  132. data/TODO.impl/97-yard-execution.md +73 -0
  133. data/TODO.impl/98-campaign-cleanup.md +78 -0
  134. data/TODO.impl/99-site-truth-performance-docs.md +31 -0
  135. data/conformance/vectors.jsonl +2630 -0
  136. data/data/en/grammar/rules.yaml +851 -0
  137. data/docs/STABILITY-1.0-AUDIT.md +251 -0
  138. data/docs/STABILITY-1.0.md +137 -0
  139. data/docs/cache-architecture.md +502 -0
  140. data/docs/verification-evidence/correct3/pr_body.md +37 -0
  141. data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
  142. data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
  143. data/docs/verification-evidence/correct3/sv.aff +856 -0
  144. data/docs/verification-evidence/correct3/sv.dic +153717 -0
  145. data/examples/01_basic_word_checking.rb +38 -0
  146. data/examples/02_text_document_checking.rb +77 -0
  147. data/examples/03_dictionary_backends.rb +137 -0
  148. data/examples/04_trie_data_structure.rb +146 -0
  149. data/examples/05_suggestion_algorithms.rb +239 -0
  150. data/examples/06_configuration_advanced.rb +287 -0
  151. data/examples/07_multi_language_dictionaries.rb +278 -0
  152. data/exe/kotoshu +6 -0
  153. data/lib/kotoshu/algorithms/capitalization.rb +276 -0
  154. data/lib/kotoshu/algorithms/edit_distance.rb +128 -0
  155. data/lib/kotoshu/algorithms/lookup/affix_form.rb +132 -0
  156. data/lib/kotoshu/algorithms/lookup/compound_checks.rb +209 -0
  157. data/lib/kotoshu/algorithms/lookup/compound_form.rb +50 -0
  158. data/lib/kotoshu/algorithms/lookup.rb +841 -0
  159. data/lib/kotoshu/algorithms/ngram_suggest.rb +367 -0
  160. data/lib/kotoshu/algorithms/permutations.rb +283 -0
  161. data/lib/kotoshu/algorithms/phonet_suggest.rb +162 -0
  162. data/lib/kotoshu/algorithms/soundex.rb +92 -0
  163. data/lib/kotoshu/algorithms/suggest.rb +794 -0
  164. data/lib/kotoshu/algorithms.rb +23 -0
  165. data/lib/kotoshu/analyzers/semantic_analyzer.rb +405 -0
  166. data/lib/kotoshu/analyzers.rb +8 -0
  167. data/lib/kotoshu/baseline/application.rb +18 -0
  168. data/lib/kotoshu/baseline/entry.rb +33 -0
  169. data/lib/kotoshu/baseline/store.rb +175 -0
  170. data/lib/kotoshu/baseline.rb +20 -0
  171. data/lib/kotoshu/cache/base_cache.rb +778 -0
  172. data/lib/kotoshu/cache/cache.rb +91 -0
  173. data/lib/kotoshu/cache/eviction_policy.rb +66 -0
  174. data/lib/kotoshu/cache/frequency_cache.rb +283 -0
  175. data/lib/kotoshu/cache/language_cache.rb +511 -0
  176. data/lib/kotoshu/cache/lookup_cache.rb +166 -0
  177. data/lib/kotoshu/cache/model_cache.rb +1147 -0
  178. data/lib/kotoshu/cache/model_registry.rb +109 -0
  179. data/lib/kotoshu/cache/staged_languages.rb +45 -0
  180. data/lib/kotoshu/cache/suggestion_cache.rb +113 -0
  181. data/lib/kotoshu/cache.rb +49 -0
  182. data/lib/kotoshu/cli/auto_setup.rb +71 -0
  183. data/lib/kotoshu/cli/baseline_command.rb +119 -0
  184. data/lib/kotoshu/cli/batch_reporter.rb +321 -0
  185. data/lib/kotoshu/cli/cache_command.rb +282 -0
  186. data/lib/kotoshu/cli/completions_command.rb +210 -0
  187. data/lib/kotoshu/cli/dict_command.rb +64 -0
  188. data/lib/kotoshu/cli/directory_check.rb +146 -0
  189. data/lib/kotoshu/cli/directory_walker.rb +132 -0
  190. data/lib/kotoshu/cli/display_formatter.rb +429 -0
  191. data/lib/kotoshu/cli/errors.rb +36 -0
  192. data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
  193. data/lib/kotoshu/cli/interactive_reviewer.rb +314 -0
  194. data/lib/kotoshu/cli/language_resolver.rb +91 -0
  195. data/lib/kotoshu/cli/navigation_manager.rb +273 -0
  196. data/lib/kotoshu/cli/personal_command.rb +113 -0
  197. data/lib/kotoshu/cli/progress_reporter.rb +114 -0
  198. data/lib/kotoshu/cli/status_report.rb +130 -0
  199. data/lib/kotoshu/cli.rb +880 -0
  200. data/lib/kotoshu/components/passthrough_spell_checker.rb +70 -0
  201. data/lib/kotoshu/components/pos_tagger.rb +98 -0
  202. data/lib/kotoshu/components/spell_checker.rb +73 -0
  203. data/lib/kotoshu/components/synthesizer.rb +60 -0
  204. data/lib/kotoshu/components/tokenizer.rb +58 -0
  205. data/lib/kotoshu/components/whitespace_tokenizer.rb +94 -0
  206. data/lib/kotoshu/components.rb +13 -0
  207. data/lib/kotoshu/configuration/builder.rb +207 -0
  208. data/lib/kotoshu/configuration/resolver.rb +124 -0
  209. data/lib/kotoshu/configuration.rb +829 -0
  210. data/lib/kotoshu/conformance_exporter.rb +256 -0
  211. data/lib/kotoshu/conformance_runner.rb +176 -0
  212. data/lib/kotoshu/core/exceptions.rb +186 -0
  213. data/lib/kotoshu/core/indexed_dictionary.rb +293 -0
  214. data/lib/kotoshu/core/models/affix_rule.rb +262 -0
  215. data/lib/kotoshu/core/models/result/document_result.rb +132 -0
  216. data/lib/kotoshu/core/models/result/word_result.rb +131 -0
  217. data/lib/kotoshu/core/models/word.rb +131 -0
  218. data/lib/kotoshu/core/trie/builder.rb +119 -0
  219. data/lib/kotoshu/core/trie/node.rb +94 -0
  220. data/lib/kotoshu/core/trie/trie.rb +249 -0
  221. data/lib/kotoshu/core.rb +28 -0
  222. data/lib/kotoshu/data/common_words/de.yml +1800 -0
  223. data/lib/kotoshu/data/common_words/en.yml +1215 -0
  224. data/lib/kotoshu/data/common_words/es.yml +750 -0
  225. data/lib/kotoshu/data/common_words/fr.yml +1015 -0
  226. data/lib/kotoshu/data/common_words/pt.yml +870 -0
  227. data/lib/kotoshu/data/common_words/ru.yml +484 -0
  228. data/lib/kotoshu/data/common_words_loader.rb +141 -0
  229. data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
  230. data/lib/kotoshu/data/frozen_kelly.rb +38 -0
  231. data/lib/kotoshu/data.rb +8 -0
  232. data/lib/kotoshu/data_structures/bloom_filter.rb +188 -0
  233. data/lib/kotoshu/data_structures.rb +8 -0
  234. data/lib/kotoshu/debug_logger.rb +146 -0
  235. data/lib/kotoshu/debug_mode.rb +136 -0
  236. data/lib/kotoshu/defaults.rb +84 -0
  237. data/lib/kotoshu/dictionaries/catalog.rb +814 -0
  238. data/lib/kotoshu/dictionaries.rb +8 -0
  239. data/lib/kotoshu/dictionary/base.rb +324 -0
  240. data/lib/kotoshu/dictionary/cspell.rb +243 -0
  241. data/lib/kotoshu/dictionary/custom.rb +225 -0
  242. data/lib/kotoshu/dictionary/hunspell.rb +520 -0
  243. data/lib/kotoshu/dictionary/plain_text.rb +302 -0
  244. data/lib/kotoshu/dictionary/repository.rb +246 -0
  245. data/lib/kotoshu/dictionary/unified.rb +236 -0
  246. data/lib/kotoshu/dictionary/unix_words.rb +218 -0
  247. data/lib/kotoshu/dictionary.rb +15 -0
  248. data/lib/kotoshu/documents/document.rb +120 -0
  249. data/lib/kotoshu/documents/plain_text_document.rb +73 -0
  250. data/lib/kotoshu/documents/source_position.rb +45 -0
  251. data/lib/kotoshu/documents/source_range.rb +76 -0
  252. data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
  253. data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
  254. data/lib/kotoshu/documents/suppressions.rb +44 -0
  255. data/lib/kotoshu/documents/text_node.rb +49 -0
  256. data/lib/kotoshu/documents.rb +178 -0
  257. data/lib/kotoshu/embeddings/embedding_pipeline.rb +238 -0
  258. data/lib/kotoshu/embeddings/lru_cache.rb +244 -0
  259. data/lib/kotoshu/embeddings/onnx_runtime_model.rb +411 -0
  260. data/lib/kotoshu/embeddings/protocol.rb +96 -0
  261. data/lib/kotoshu/embeddings/protocols.rb +23 -0
  262. data/lib/kotoshu/embeddings/registry.rb +184 -0
  263. data/lib/kotoshu/embeddings/search.rb +204 -0
  264. data/lib/kotoshu/embeddings/similarity_engine.rb +278 -0
  265. data/lib/kotoshu/embeddings/similarity_search.rb +328 -0
  266. data/lib/kotoshu/embeddings/vocabulary.rb +306 -0
  267. data/lib/kotoshu/embeddings.rb +87 -0
  268. data/lib/kotoshu/fluent_checker.rb +94 -0
  269. data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +48 -0
  270. data/lib/kotoshu/grammar/pattern_matchers/double_negative_matcher.rb +179 -0
  271. data/lib/kotoshu/grammar/pattern_matchers/phrase_matcher.rb +86 -0
  272. data/lib/kotoshu/grammar/pattern_matchers/possessive_context_matcher.rb +75 -0
  273. data/lib/kotoshu/grammar/pattern_matchers/possessive_contraction_matcher.rb +87 -0
  274. data/lib/kotoshu/grammar/pattern_matchers/sentence_start_matcher.rb +84 -0
  275. data/lib/kotoshu/grammar/pattern_matchers/vowel_sound_matcher.rb +81 -0
  276. data/lib/kotoshu/grammar/pattern_matchers/word_list_matcher.rb +66 -0
  277. data/lib/kotoshu/grammar/rule.rb +99 -0
  278. data/lib/kotoshu/grammar/rule_engine.rb +96 -0
  279. data/lib/kotoshu/grammar/rule_loader.rb +30 -0
  280. data/lib/kotoshu/grammar.rb +24 -0
  281. data/lib/kotoshu/integrity/audit_log.rb +167 -0
  282. data/lib/kotoshu/integrity/manifest.rb +116 -0
  283. data/lib/kotoshu/integrity/net_http.rb +51 -0
  284. data/lib/kotoshu/integrity/rotation_policy.rb +88 -0
  285. data/lib/kotoshu/integrity.rb +26 -0
  286. data/lib/kotoshu/jekyll.rb +101 -0
  287. data/lib/kotoshu/keyboard/layout.rb +116 -0
  288. data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
  289. data/lib/kotoshu/keyboard/layouts/azerty.rb +55 -0
  290. data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
  291. data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
  292. data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
  293. data/lib/kotoshu/keyboard/layouts/dvorak.rb +54 -0
  294. data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
  295. data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
  296. data/lib/kotoshu/keyboard/layouts/hrsl.rb +94 -0
  297. data/lib/kotoshu/keyboard/layouts/jcuken.rb +57 -0
  298. data/lib/kotoshu/keyboard/layouts/latin.rb +266 -0
  299. data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
  300. data/lib/kotoshu/keyboard/layouts/qwerty.rb +52 -0
  301. data/lib/kotoshu/keyboard/layouts/qwertz.rb +55 -0
  302. data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
  303. data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
  304. data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
  305. data/lib/kotoshu/keyboard/registry.rb +201 -0
  306. data/lib/kotoshu/keyboard.rb +96 -0
  307. data/lib/kotoshu/kotoshu_native.so +0 -0
  308. data/lib/kotoshu/language/detector.rb +242 -0
  309. data/lib/kotoshu/language/identifier.rb +376 -0
  310. data/lib/kotoshu/language/languages/base.rb +271 -0
  311. data/lib/kotoshu/language/lid_detector.rb +188 -0
  312. data/lib/kotoshu/language/normalizer/arabic.rb +113 -0
  313. data/lib/kotoshu/language/normalizer/base.rb +133 -0
  314. data/lib/kotoshu/language/normalizer/greek.rb +63 -0
  315. data/lib/kotoshu/language/normalizer/hebrew.rb +72 -0
  316. data/lib/kotoshu/language/normalizer/persian.rb +36 -0
  317. data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
  318. data/lib/kotoshu/language/registry.rb +209 -0
  319. data/lib/kotoshu/language/resources/ar/common_words.txt +6753 -0
  320. data/lib/kotoshu/language/resources/ar/confusion_sets.txt +11 -0
  321. data/lib/kotoshu/language/resources/de/common_words.txt +10003 -0
  322. data/lib/kotoshu/language/resources/de/confusion_sets.txt +246 -0
  323. data/lib/kotoshu/language/resources/en/common_words.txt +9979 -0
  324. data/lib/kotoshu/language/resources/en/confusion_sets.txt +871 -0
  325. data/lib/kotoshu/language/resources/es/common_words.txt +9992 -0
  326. data/lib/kotoshu/language/resources/es/confusion_sets.txt +17 -0
  327. data/lib/kotoshu/language/resources/fr/common_words.txt +9993 -0
  328. data/lib/kotoshu/language/resources/fr/confusion_sets.txt +76 -0
  329. data/lib/kotoshu/language/resources/pt/common_words.txt +9977 -0
  330. data/lib/kotoshu/language/resources/pt/confusion_sets.txt +18 -0
  331. data/lib/kotoshu/language/resources/ru/common_words.txt +9951 -0
  332. data/lib/kotoshu/language/resources/ru/confusion_sets.txt +5 -0
  333. data/lib/kotoshu/language/script.rb +128 -0
  334. data/lib/kotoshu/language/segmenter.rb +122 -0
  335. data/lib/kotoshu/language/suika.rb +38 -0
  336. data/lib/kotoshu/language/tokenizer/base.rb +187 -0
  337. data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
  338. data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
  339. data/lib/kotoshu/language/tokenizer/french_tokenizer.rb +172 -0
  340. data/lib/kotoshu/language/tokenizer/german_tokenizer.rb +41 -0
  341. data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
  342. data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
  343. data/lib/kotoshu/language/tokenizer/japanese_tokenizer.rb +55 -0
  344. data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +153 -0
  345. data/lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb +160 -0
  346. data/lib/kotoshu/language/tokenizer/russian_tokenizer.rb +93 -0
  347. data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
  348. data/lib/kotoshu/language/tokenizer/spanish_tokenizer.rb +122 -0
  349. data/lib/kotoshu/language.rb +127 -0
  350. data/lib/kotoshu/languages/ar/language.rb +98 -0
  351. data/lib/kotoshu/languages/bg/language.rb +37 -0
  352. data/lib/kotoshu/languages/ca/language.rb +22 -0
  353. data/lib/kotoshu/languages/cs/language.rb +22 -0
  354. data/lib/kotoshu/languages/da/language.rb +22 -0
  355. data/lib/kotoshu/languages/de/language.rb +554 -0
  356. data/lib/kotoshu/languages/el/language.rb +50 -0
  357. data/lib/kotoshu/languages/en/language.rb +470 -0
  358. data/lib/kotoshu/languages/es/language.rb +476 -0
  359. data/lib/kotoshu/languages/et/language.rb +24 -0
  360. data/lib/kotoshu/languages/fa/language.rb +91 -0
  361. data/lib/kotoshu/languages/fr/language.rb +505 -0
  362. data/lib/kotoshu/languages/he/language.rb +98 -0
  363. data/lib/kotoshu/languages/hr/language.rb +24 -0
  364. data/lib/kotoshu/languages/hu/language.rb +22 -0
  365. data/lib/kotoshu/languages/id/language.rb +25 -0
  366. data/lib/kotoshu/languages/it/language.rb +22 -0
  367. data/lib/kotoshu/languages/ja/language.rb +476 -0
  368. data/lib/kotoshu/languages/ko/language.rb +56 -0
  369. data/lib/kotoshu/languages/latin_base.rb +53 -0
  370. data/lib/kotoshu/languages/lt/language.rb +24 -0
  371. data/lib/kotoshu/languages/lv/language.rb +24 -0
  372. data/lib/kotoshu/languages/nb/language.rb +25 -0
  373. data/lib/kotoshu/languages/ne/language.rb +60 -0
  374. data/lib/kotoshu/languages/nl/language.rb +22 -0
  375. data/lib/kotoshu/languages/nn/language.rb +26 -0
  376. data/lib/kotoshu/languages/pl/language.rb +22 -0
  377. data/lib/kotoshu/languages/pt/language.rb +439 -0
  378. data/lib/kotoshu/languages/ro/language.rb +22 -0
  379. data/lib/kotoshu/languages/ru/language.rb +420 -0
  380. data/lib/kotoshu/languages/sk/language.rb +25 -0
  381. data/lib/kotoshu/languages/sl/language.rb +25 -0
  382. data/lib/kotoshu/languages/sr/language.rb +39 -0
  383. data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
  384. data/lib/kotoshu/languages/sv/language.rb +22 -0
  385. data/lib/kotoshu/languages/tr/language.rb +36 -0
  386. data/lib/kotoshu/languages/uk/language.rb +38 -0
  387. data/lib/kotoshu/languages/vi/language.rb +22 -0
  388. data/lib/kotoshu/languages.rb +64 -0
  389. data/lib/kotoshu/metrics_collector.rb +238 -0
  390. data/lib/kotoshu/metrics_module.rb +121 -0
  391. data/lib/kotoshu/models/context.rb +119 -0
  392. data/lib/kotoshu/models/embedding_model.rb +181 -0
  393. data/lib/kotoshu/models/fasttext_model.rb +212 -0
  394. data/lib/kotoshu/models/nearest_neighbor.rb +96 -0
  395. data/lib/kotoshu/models/onnx_model.rb +330 -0
  396. data/lib/kotoshu/models/semantic_error.rb +204 -0
  397. data/lib/kotoshu/models/suggestion.rb +106 -0
  398. data/lib/kotoshu/models/word_embedding.rb +97 -0
  399. data/lib/kotoshu/models.rb +27 -0
  400. data/lib/kotoshu/multi_language_checker.rb +127 -0
  401. data/lib/kotoshu/native.rb +83 -0
  402. data/lib/kotoshu/native_backend.rb +131 -0
  403. data/lib/kotoshu/paths.rb +79 -0
  404. data/lib/kotoshu/personal_dictionary.rb +105 -0
  405. data/lib/kotoshu/plugins/plugin.rb +61 -0
  406. data/lib/kotoshu/plugins/registry.rb +118 -0
  407. data/lib/kotoshu/plugins.rb +9 -0
  408. data/lib/kotoshu/project_config.rb +76 -0
  409. data/lib/kotoshu/readers/aff_data.rb +530 -0
  410. data/lib/kotoshu/readers/aff_reader.rb +393 -0
  411. data/lib/kotoshu/readers/condition_checker.rb +125 -0
  412. data/lib/kotoshu/readers/dic_reader.rb +208 -0
  413. data/lib/kotoshu/readers/file_reader.rb +377 -0
  414. data/lib/kotoshu/readers/lookup_builder.rb +584 -0
  415. data/lib/kotoshu/readers/readers.rb +17 -0
  416. data/lib/kotoshu/readers.rb +25 -0
  417. data/lib/kotoshu/resource_bundle.rb +55 -0
  418. data/lib/kotoshu/resource_manager.rb +600 -0
  419. data/lib/kotoshu/results/result.rb +165 -0
  420. data/lib/kotoshu/results.rb +8 -0
  421. data/lib/kotoshu/rspec.rb +163 -0
  422. data/lib/kotoshu/scripts/fasttext_to_onnx.py +275 -0
  423. data/lib/kotoshu/source_registry.rb +104 -0
  424. data/lib/kotoshu/spellchecker/parallel_checker.rb +90 -0
  425. data/lib/kotoshu/spellchecker.rb +406 -0
  426. data/lib/kotoshu/string_metrics.rb +153 -0
  427. data/lib/kotoshu/suggestions/context.rb +43 -0
  428. data/lib/kotoshu/suggestions/frequency_provider.rb +80 -0
  429. data/lib/kotoshu/suggestions/generator.rb +169 -0
  430. data/lib/kotoshu/suggestions/pipeline.rb +135 -0
  431. data/lib/kotoshu/suggestions/semantic_cascade.rb +73 -0
  432. data/lib/kotoshu/suggestions/strategies/base_strategy.rb +315 -0
  433. data/lib/kotoshu/suggestions/strategies/composite_strategy.rb +197 -0
  434. data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +631 -0
  435. data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +207 -0
  436. data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +165 -0
  437. data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +284 -0
  438. data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +317 -0
  439. data/lib/kotoshu/suggestions/strategies/symspell_strategy.rb +270 -0
  440. data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
  441. data/lib/kotoshu/suggestions/suggestion.rb +154 -0
  442. data/lib/kotoshu/suggestions/suggestion_set.rb +270 -0
  443. data/lib/kotoshu/suggestions/sweep_index.rb +137 -0
  444. data/lib/kotoshu/suggestions.rb +29 -0
  445. data/lib/kotoshu/tasks/check_task.rb +104 -0
  446. data/lib/kotoshu/tasks.rb +11 -0
  447. data/lib/kotoshu/validators/spelling_validator.rb +101 -0
  448. data/lib/kotoshu/validators.rb +9 -0
  449. data/lib/kotoshu/version.rb +5 -0
  450. data/lib/kotoshu.rb +538 -0
  451. data/rakelib/compile.rake +10 -0
  452. data/script/validate_all_dictionaries.rb +444 -0
  453. data/scripts/build_native_gem.rb +45 -0
  454. data/scripts/extract_eval_grids.rb +37 -0
  455. data/scripts/generate_frozen_tiers.rb +44 -0
  456. data/sig/kotoshu.rbs +435 -0
  457. data/test_oop.rb +79 -0
  458. metadata +546 -0
@@ -0,0 +1,27 @@
1
+ # Plan 132 — The techniques page; professional prose across the site
2
+
3
+ Status: executed (2026-09-12, site push; this plan)
4
+
5
+ ## What shipped
6
+
7
+ 1. /docs/techniques — a complete catalog of the stack's techniques in
8
+ four groups (engines, models, delivery, reliability), each entry
9
+ naming the mechanism and its measured effect in full sentences.
10
+ Written as the site's first MDX page: @astrojs/mdx installed and
11
+ wired, DocsLayout applied through an imported component wrapper.
12
+ 2. A professional rewrite of the published copy: 34 fragment-style
13
+ news titles became declarative sentences, and 69 fragment-stacked
14
+ summaries and senses across all 36 entries were rewritten as full
15
+ sentences with clear subjects. The worst patterns eliminated:
16
+ em-dash fragment leads ("What 1.0 freezes:"), colon-list openers
17
+ ("Also in the patch:"), trailing fragment appends ("— no download,
18
+ no CI seeding"), and lowercase clause starts.
19
+ 3. The docs pages received the same treatment where fragments
20
+ appeared: the performance error-budget section, the caching
21
+ language-packs section, and three headlines.
22
+
23
+ ## Conventions established
24
+
25
+ - Text-heavy documentation pages are written in MDX, not .astro HTML.
26
+ - News titles are complete clauses; summaries and senses are full
27
+ sentences. Release titles use the "package X.Y.Z: description" form.
@@ -0,0 +1,68 @@
1
+ # Plan 133 — Rust-first distribution: precompiled native gems and complete wheels
2
+
3
+ Status: proposed (execution-ready)
4
+ Depends on: the 1.0 conformance contract (identical outputs make the
5
+ backend default flip correctness-free); plan 93 (the python wheel matrix)
6
+
7
+ ## Problem
8
+
9
+ The Rust engine is the faster implementation (~10-15x on the sweep) and
10
+ the strategic default, but today reaching it requires a build toolchain:
11
+ the gem compiles its native extension at install time, and the python
12
+ native wheel matrix is incomplete on PyPI. Users who cannot run a
13
+ toolchain get pure Ruby whether they wanted it or not — and users who
14
+ wanted Rust pay a compile for it. The architecture makes both engines
15
+ byte-identical, so distribution — not capability — is the only gap.
16
+
17
+ ## Design
18
+
19
+ ### Ruby: precompiled platform gems (the rb_sys flow)
20
+
21
+ 1. New gem workflow `release-native.yml`: build matrix over
22
+ {x86_64-linux, aarch64-linux, x86_64-linux-musl, arm64-darwin,
23
+ x86_64-darwin, x64-mingw-ucrt} x {ruby 3.1..4.0} using
24
+ rb-sys-dock images for the Linux targets and native runners for
25
+ macOS/Windows; each job builds and pushes
26
+ `kotoshu-X.Y.Z-<platform>.gem` through trusted publishing. The
27
+ pure-Ruby platform gem continues to publish exactly as today, so
28
+ every unmatched platform resolves to it.
29
+ 2. Backend default `ruby` -> `auto`: use the native engine when the
30
+ extension loads, fall back silently to pure Ruby when it does not.
31
+ Auto never downloads and never fails a check; specs pin both
32
+ branches; the conformance job already runs both engines.
33
+ 3. The docker CI image drops its build-tool layer (it installs the
34
+ platform gem instead) — smaller image, faster builds, and its
35
+ version guard gains a native-backend assertion.
36
+
37
+ ### Python: complete the kotoshu-native wheel matrix
38
+
39
+ 1. Audit PyPI's published wheels for kotoshu-native against the
40
+ plan-93 matrix: manylinux x86_64/aarch64, macOS x86_64/arm64,
41
+ windows x86_64 (+ arm64 if tractable), sdist.
42
+ 2. Fill the gaps in release-pypi.yml (maturin + the standard target
43
+ matrix), publish a patch release, and verify pip resolves a wheel
44
+ (no source build) on each platform.
45
+ 3. Add the trove classifiers for every published target.
46
+
47
+ ### Site
48
+
49
+ Install page's engines-and-bindings section gains the resolution
50
+ sentence: the native engine installs automatically wherever a prebuilt
51
+ artifact exists for your platform, and pure Ruby is always the
52
+ fallback; python's entry notes the wheel coverage.
53
+
54
+ ## Gates
55
+
56
+ - A clean-machine `gem install kotoshu` on each published platform
57
+ loads the native backend with no toolchain present (CI-verifiable
58
+ on the matrix runners).
59
+ - The 2,630-vector conformance compare passes under `auto` on both a
60
+ native and a pure-Ruby install.
61
+ - pip on each target platform resolves a wheel, not the sdist.
62
+ - The pure-Ruby path is unchanged: same gem, same outputs, same specs.
63
+
64
+ ## Non-goals
65
+
66
+ - No behavior change of any kind — outputs are frozen; only which
67
+ engine computes them changes.
68
+ - No deprecation of anything: pure Ruby is a feature, not a legacy.
@@ -0,0 +1,47 @@
1
+ # 20 — Encapsulation cleanup (T3, deferred past 0.3)
2
+
3
+ ## Status
4
+ Deferred past 0.3 release. Documented for 0.4.
5
+
6
+ ## Problem
7
+ The codebase has three categories of encapsulation violations prohibited
8
+ by `~/.claude/CLAUDE.md`:
9
+
10
+ | Violation | Count | Source |
11
+ |---|---|---|
12
+ | `obj.send(:private_method, ...)` | 4 | breaks private boundary |
13
+ | `instance_variable_set` / `instance_variable_get` | 21 | breaks encapsulation of another object's state |
14
+ | `respond_to?(:method)` for type checks | 24 | hides type errors until runtime |
15
+
16
+ New code in 0.3 follows the rules; this file tracks migrating the
17
+ existing violations.
18
+
19
+ ## Plan
20
+
21
+ ### send to private (4 occurrences)
22
+ Each call site is a design smell — the API boundary is wrong. Fix per
23
+ location by either:
24
+ - Making the method public (if callers outside genuinely need it).
25
+ - Adding a public method that does what the caller wants (rename intent).
26
+ - Inlining the call at the (usually single) call site.
27
+
28
+ ### instance_variable_set / instance_variable_get (21)
29
+ Replace each with a public accessor or a constructor argument. If the
30
+ target object genuinely needs to be mutated externally, expose a public
31
+ setter that maintains invariants.
32
+
33
+ ### respond_to? for type checks (24)
34
+ Replace with `is_a?(SpecificType)` or — better — redesign so the type
35
+ hierarchy makes the check unnecessary (polymorphism).
36
+
37
+ ## Acceptance
38
+
39
+ - [ ] `grep -rn "\.send(" lib/ | wc -l` is 0 (or all remaining are
40
+ legitimate metaprogramming, with an inline comment justifying each).
41
+ - [ ] `grep -rn "instance_variable_set\|instance_variable_get" lib/` is 0.
42
+ - [ ] `grep -rn "respond_to?" lib/` returns only duck-typing cases that
43
+ have been audited and have an inline comment.
44
+
45
+ ## Why deferred
46
+ Touching 49 call sites risks regressions in working code. 0.3 priority
47
+ is user-facing usability. Sweep this in a dedicated 0.4 PR.
@@ -0,0 +1,64 @@
1
+ # 21 — Migrate require_relative to autoload (T3, deferred past 0.3)
2
+
3
+ ## Status
4
+ Deferred past 0.3 release. Documented for 0.4.
5
+
6
+ ## Problem
7
+ 288 `require_relative` calls in `lib/`. Per `~/.claude/CLAUDE.md`:
8
+
9
+ > NEVER use `require_relative` for internal library code. Never use
10
+ > `require` with a path to code within your own library. Use Ruby
11
+ > `autoload` instead. Define autoload entries in the immediate parent
12
+ > namespace's file — create that file if it doesn't exist.
13
+
14
+ Current state:
15
+ - `lib/kotoshu.rb` eagerly `require_relative`s the traditional path
16
+ (core models, dictionaries, strategies, configuration, spellchecker)
17
+ and uses `autoload` for the rest.
18
+ - Many subdirs lack a parent-namespace file (e.g. `lib/kotoshu/models.rb`
19
+ does not exist).
20
+
21
+ ## Plan
22
+
23
+ ### Phase 1 — Create missing parent namespace files
24
+ Create each of:
25
+ ```
26
+ lib/kotoshu/models.rb
27
+ lib/kotoshu/dictionaries.rb
28
+ lib/kotoshu/dictionary.rb
29
+ lib/kotoshu/suggestions.rb
30
+ lib/kotoshu/analyzers.rb
31
+ lib/kotoshu/documents.rb
32
+ lib/kotoshu/components.rb
33
+ lib/kotoshu/plugins.rb
34
+ lib/kotoshu/data_structures.rb
35
+ lib/kotoshu/results.rb
36
+ lib/kotoshu/commands.rb
37
+ ```
38
+ Each declares `module Kotoshu::<Ns>` with one `autoload :ClassName,
39
+ "kotoshu/<ns>/<class_name_in_snake_case>"` per file in the subdir.
40
+
41
+ ### Phase 2 — Replace require_relative inside lib/
42
+ Inside `lib/kotoshu/**/*.rb`:
43
+ - Replace `require_relative "foo"` with `autoload :Foo, "kotoshu/<ns>/foo"`.
44
+ - The autoload declaration lives in the immediate parent namespace file
45
+ (created in Phase 1), not in the file that needs the class.
46
+ - The top-level `lib/kotoshu.rb` only `require_relative`s the namespace
47
+ files (`kotoshu/cache`, `kotoshu/models`, etc.), not individual classes.
48
+
49
+ ### Phase 3 — Verify eager paths still work
50
+ Some classes must still load eagerly (the public facade in `lib/kotoshu.rb`).
51
+ Keep those as `require_relative "kotoshu/<ns>"` (which loads the parent
52
+ namespace file, which can either autoload or eager-require its children).
53
+
54
+ ## Acceptance
55
+
56
+ - [ ] `grep -rn "require_relative" lib/ | wc -l` is ≤ ~10 (only top-level
57
+ `lib/kotoshu.rb` loading namespace files).
58
+ - [ ] `bundle exec rspec` passes with no regressions.
59
+ - [ ] Boot time (`bundle exec ruby -e 'require "kotoshu"; puts Kotoshu'`)
60
+ is no slower than today (autoload should make it faster).
61
+
62
+ ## Why deferred
63
+ Touches every file in `lib/`. High regression surface. New 0.3 code
64
+ follows the autoload rule; full migration is a dedicated 0.4 effort.
@@ -0,0 +1,120 @@
1
+ # 22 — Single source of truth for URL + pin config (T1, 0.3-blocker)
2
+
3
+ ## Status
4
+ Implemented (shipped in the 0.3 line; re-verified 2026-09-03 against 0.6.0).
5
+
6
+ Evidence:
7
+
8
+ - `lib/kotoshu/source_registry.rb` — `SourceRegistry` with a frozen
9
+ `SOURCES` map (spelling/grammar/dict_manifest/frequency/freq_manifest/
10
+ model/model_vocab/model_manifest/model_registry), per-repo
11
+ `default_pin` (`dictionaries` → `v1`, the other repos → `main`),
12
+ `url_for` / `pin_for_source`. Zero `require_relative`; autoloaded from
13
+ `lib/kotoshu.rb:46`.
14
+ - `Configuration` exposes `repos_base_url` (`KOTOSHU_REPOS_BASE_URL`),
15
+ `dictionaries_pin` (`KOTOSHU_DICTIONARIES_PIN`, default `v1`),
16
+ `frequency_pin`, `models_pin` (`lib/kotoshu/configuration.rb:75-98`)
17
+ and builds the registry via `#source_registry` (ibid., line 478).
18
+ - All three caches build URLs through the registry — no inline URL
19
+ strings: `language_cache.rb:195,204,233,261,295`,
20
+ `frequency_cache.rb:232,239`, `model_cache.rb` (now registry-driven,
21
+ see below).
22
+ - Spec: `spec/kotoshu/source_registry_spec.rb` — run 2026-09-03,
23
+ 19 examples, 0 failures.
24
+ - Live check (2026-09-03):
25
+ `Kotoshu::SourceRegistry.new.url_for(:spelling, lang: "en", ext: "aff")`
26
+ → `https://raw.githubusercontent.com/kotoshu/dictionaries/v1/en/spelling/index.aff`
27
+ (exactly the acceptance URL).
28
+
29
+ Deviations from the plan text:
30
+
31
+ - `dictionaries_url` / `models_url` were not removed — they remain in
32
+ `SCHEMA` marked "Deprecated" and are consumed nowhere outside
33
+ `configuration.rb`.
34
+ - The model source later evolved past URL templates into a tiered
35
+ `registry.json` (`lib/kotoshu/cache/model_registry.rb`,
36
+ TODO.impl/67 M1: mini/fluency/full tiers).
37
+
38
+ ## Problem
39
+ Today there are three overlapping configuration knobs for where to fetch
40
+ resources from:
41
+
42
+ ```ruby
43
+ dictionaries_url: "https://raw.githubusercontent.com/kotoshu/dictionaries/main"
44
+ resource_pin: "main" # used by language_cache, frequency_cache
45
+ models_url: "https://github.com/kotoshu/models-fasttext-onnx/raw/main"
46
+ models_pin: "main" # used by model_cache
47
+ ```
48
+
49
+ Three problems:
50
+
51
+ 1. **`/main` is baked into `dictionaries_url` even though `resource_pin`
52
+ is meant to be the branch selector.** The URL builder in
53
+ `lib/kotoshu/cache/language_cache.rb:181` is:
54
+ ```ruby
55
+ "#{@url_base}/dictionaries/#{@resource_pin}/#{language}/spelling/index.aff"
56
+ ```
57
+ So the URL is `<url_base>/dictionaries/<pin>/<lang>/...` — and the
58
+ `dictionaries/main` in the default URL is dead weight, never substituted.
59
+
60
+ 2. **Each repo has its own default branch.** `kotoshu/dictionaries` is
61
+ on `v1`; `frequency-list-kelly` and `models-fasttext-onnx` are on
62
+ `main`. A single `resource_pin` cannot honor both.
63
+
64
+ 3. **URL templates are scattered.** `language_cache.rb`, `frequency_cache.rb`,
65
+ `model_cache.rb` each build their own URL strings. No single source.
66
+
67
+ ## Plan
68
+
69
+ ### Introduce `Kotoshu::SourceRegistry`
70
+
71
+ A new model class — single source of truth for "where does each resource
72
+ type live, and at what pin."
73
+
74
+ ```ruby
75
+ # lib/kotoshu/source_registry.rb
76
+ module Kotoshu
77
+ class SourceRegistry
78
+ Source = Struct.new(:repo, :default_pin, :url_template, keyword_init: true)
79
+
80
+ SOURCES = {
81
+ spelling: Source.new(repo: "dictionaries", default_pin: "v1",
82
+ url_template: "%{base}/dictionaries/%{repo}/%{pin}/%{lang}/spelling/index.%{ext}"),
83
+ frequency: Source.new(repo: "frequency-list-kelly", default_pin: "main",
84
+ url_template: "%{base}/frequency-list-kelly/%{pin}/data/%{lang}.json"),
85
+ model: Source.new(repo: "models-fasttext-onnx", default_pin: "main",
86
+ url_template: "%{base}/models-fasttext-onnx/%{pin}/models/%{lang}/fasttext.%{lang}.onnx"),
87
+ manifest: Source.new(repo: "dictionaries", default_pin: "v1",
88
+ url_template: "%{base}/dictionaries/%{pin}/manifest.json")
89
+ }.freeze
90
+
91
+ def self.url_for(resource_kind, lang: nil, ext: nil, base: nil, pin: nil)
92
+ # ...
93
+ end
94
+ end
95
+ end
96
+ ```
97
+
98
+ ### Replace `dictionaries_url`, `resource_pin`, `models_url`, `models_pin`
99
+
100
+ In `Configuration`:
101
+ - Keep `repos_base_url` (one config knob — the GitHub raw root).
102
+ - Keep per-source pin overrides: `dictionaries_pin`, `frequency_pin`,
103
+ `models_pin` — but with correct defaults (`v1` for dictionaries).
104
+ - Remove `dictionaries_url`, `models_url`, `resource_pin` (drift-prone).
105
+
106
+ Caches call `SourceRegistry.url_for(:spelling, lang: "en", ext: "aff")`
107
+ instead of building strings.
108
+
109
+ ## Acceptance
110
+
111
+ - [ ] `SourceRegistry` exists, is fully tested, has zero `require_relative`.
112
+ - [ ] `Configuration` exposes only `repos_base_url`, `dictionaries_pin`,
113
+ `frequency_pin`, `models_pin`.
114
+ - [ ] Each cache class no longer builds URL strings inline — calls
115
+ `SourceRegistry.url_for`.
116
+ - [ ] Default URL for `kotoshu setup en` resolves to a real file
117
+ (i.e. `https://raw.githubusercontent.com/kotoshu/dictionaries/v1/en/spelling/index.aff`).
118
+
119
+ ## Dependency
120
+ - Blocks #23 (URL defaults fix), #29 (status command needs to show URLs).
@@ -0,0 +1,57 @@
1
+ # 23 — Fix URL + pin defaults so first-use doesn't 404 (T1, 0.3-blocker)
2
+
3
+ ## Status
4
+ Implemented (shipped in the 0.3 line; re-verified 2026-09-03 against 0.6.0).
5
+
6
+ Evidence (all run 2026-09-03):
7
+
8
+ - `NETWORK_TESTS=1 bundle exec rspec spec/integration/end_to_end_spec.rb`
9
+ → 12 examples, 0 failures — real download of `en` spelling from
10
+ `kotoshu/dictionaries` into an isolated tmp XDG home; setup returns
11
+ `:downloaded`, `correct?("hello")` true, `suggest("teh")` includes
12
+ `"the"`. No 404s.
13
+ - Live German acceptance, isolated tmp XDG home:
14
+ `Kotoshu.setup(:de, want: %i[spelling]).spelling` → `:downloaded`.
15
+ - Root cause fixed structurally: dictionaries default pin is `v1` in
16
+ `SourceRegistry::SOURCES[:spelling]` (`lib/kotoshu/source_registry.rb:22`).
17
+
18
+ The `en`/`de` correct?/suggest acceptance items were also exercised
19
+ through the real dev cache via the README quickstart (see #28).
20
+
21
+ ## Problem
22
+ Today `Kotoshu.setup(:en)` fails:
23
+
24
+ ```
25
+ $ KOTOSHU_OFFLINE=0 bundle exec ruby -e 'Kotoshu.setup(:en)'
26
+ # => 404 from raw.githubusercontent.com/kotoshu/dictionaries/main/...
27
+ ```
28
+
29
+ Two causes:
30
+ 1. `dictionaries_url` default ends in `/main` but the dictionaries repo
31
+ default branch is `v1`.
32
+ 2. `resource_pin` default is `main` regardless of repo.
33
+
34
+ ## Plan
35
+
36
+ ### Step 1 — Implement TODO #22 (`SourceRegistry`)
37
+ Done first.
38
+
39
+ ### Step 2 — Update default pin for dictionaries to `v1`
40
+ Via `SourceRegistry::SOURCES[:spelling].default_pin = "v1"`.
41
+
42
+ ### Step 3 — Verify end-to-end
43
+ ```ruby
44
+ Kotoshu.setup(:en, want: %i[spelling]).spelling # => :downloaded
45
+ Kotoshu.correct?("hello") # => true
46
+ Kotoshu.suggest("teh").include?("the") # => true
47
+ ```
48
+
49
+ ## Acceptance
50
+
51
+ - [ ] Live run of `Kotoshu.setup(:en)` returns `:downloaded` (not 404).
52
+ - [ ] Live run of `Kotoshu.setup(:de)` returns `:downloaded`.
53
+ - [ ] `Kotoshu.correct?("hello")` returns `true` after setup.
54
+ - [ ] `Kotoshu.suggest("teh")` returns a list that includes `"the"`.
55
+
56
+ ## Dependencies
57
+ - #22 (SourceRegistry)
@@ -0,0 +1,76 @@
1
+ # 24 — CLI auto-setup prompt (T1, 0.3-blocker)
2
+
3
+ ## Status
4
+ Implemented (shipped in the 0.3 line; re-verified 2026-09-03 against 0.6.0).
5
+
6
+ Evidence:
7
+
8
+ - `lib/kotoshu/cli/auto_setup.rb` — `Kotoshu::Cli::AutoSetup` exactly as
9
+ planned: non-TTY / offline → returns nil (no prompt); TTY → prints
10
+ `Language 'en' is not set up (missing spelling). Download now (~5 MB
11
+ from github.com/kotoshu/dictionaries)? [Y/n]`; on yes runs
12
+ `Kotoshu.setup(language, want:)`.
13
+ - Wired in the CLI dispatcher: `lib/kotoshu/cli.rb:307-308` rescues
14
+ `Kotoshu::ResourceNotSetupError`, delegates to `AutoSetup`, and
15
+ re-raises as `Errors::ResourceUnavailable` (exit 3) when declined.
16
+ - Library API stays strict — only the CLI layer prompts.
17
+ - Spec: `spec/kotoshu/cli/auto_setup_spec.rb` — run 2026-09-03,
18
+ 0 failures. Exit code 3 is documented in the `check` long_desc and
19
+ README.adoc ("language not set up").
20
+
21
+ Not re-verified interactively today (requires a TTY); the non-TTY and
22
+ prompt logic are spec-covered.
23
+
24
+ ## Problem
25
+ Today:
26
+
27
+ ```
28
+ $ kotoshu check README.md
29
+ # (stack trace from ResourceNotSetupError)
30
+ ```
31
+
32
+ The library raises `ResourceNotSetupError` with a clear message — but
33
+ the CLI surfaces it as a stack trace. Users who install the gem and run
34
+ `kotoshu check` on day one don't know to run `kotoshu setup` first.
35
+
36
+ The two-stage setup/resolve split is correct for the **library** API
37
+ (see `lib/kotoshu/resource_manager.rb` — explicit setup, cache-only hot
38
+ path, no surprise downloads on metered networks). But the **CLI** should
39
+ be the friendly face.
40
+
41
+ ## Plan
42
+
43
+ ### Add `Kotoshu::Cli::AutoSetup` middleware
44
+
45
+ Catch `ResourceNotSetupError` in `Cli.start` (the existing dispatch
46
+ override, per project memory `kotoshu-cli-semantic-errors`). When caught:
47
+
48
+ 1. Detect TTY. If non-TTY (piped output, CI), re-raise with the existing
49
+ message — no surprise interactive prompts in scripts.
50
+ 2. If TTY, prompt:
51
+ ```
52
+ Language 'en' is not set up (missing spelling).
53
+ Download now (~5 MB from github.com/kotoshu/dictionaries)? [Y/n]
54
+ ```
55
+ 3. On yes: call `Kotoshu.setup(lang, want: %i[spelling])`, then retry
56
+ the original command.
57
+ 4. On no: exit 3 with the original message.
58
+ 5. Honor `--offline` flag and `KOTOSHU_OFFLINE=1` env var — never prompt
59
+ in offline mode, just exit with the error.
60
+
61
+ ### Why this respects the two-stage model
62
+ The library API (`Kotoshu.correct?`, `Kotoshu.suggest`) still raises
63
+ strictly. Only the human-facing CLI adds the prompt layer. Programmatic
64
+ users get the strict behavior they need.
65
+
66
+ ## Acceptance
67
+
68
+ - [ ] `kotoshu check file.txt` on a fresh cache prompts interactively.
69
+ - [ ] `echo "text" | kotoshu check -` (non-TTY) does NOT prompt — exits 3
70
+ with the original error message.
71
+ - [ ] `KOTOSHU_OFFLINE=1 kotoshu check file.txt` does NOT prompt — exits 3.
72
+ - [ ] After "y" at the prompt, the check completes successfully.
73
+ - [ ] Existing CLI tests still pass; new spec covers the prompt path.
74
+
75
+ ## Dependencies
76
+ - #23 (setup must actually work first)
@@ -0,0 +1,75 @@
1
+ # 25 — Soft ONNX runtime require (T1, 0.3-blocker)
2
+
3
+ ## Status
4
+ Implemented (shipped in the 0.3 line; re-verified 2026-09-03 against 0.6.0).
5
+
6
+ Evidence:
7
+
8
+ - `kotoshu.gemspec` runtime deps are only `thor`, `rubyzip`,
9
+ `lutaml-model` — no `onnxruntime` (nor `suika`; both documented as
10
+ soft deps in comments).
11
+ - `lib/kotoshu/models/onnx_model.rb:25-34` — load-time
12
+ `begin require "onnxruntime" ... rescue LoadError` sets
13
+ `ONNX_LOADED = false`; `KOTOSHU_NO_ONNX=1` forces it off (opt-out,
14
+ replacing the old opt-in).
15
+ - Actionable error: `OnnxModel::OnnxUnavailable` (ibid., lines 38-46)
16
+ — "Install with: gem install onnxruntime".
17
+ - `KOTOSHU_REQUIRE_ONNX` is gone from `lib/` and the docs; the only
18
+ remaining mentions are historical (CHANGELOG 0.1.0 entry) and in the
19
+ plan files themselves.
20
+ - Live: `bundle exec exe/kotoshu status` reports
21
+ `onnxruntime loaded` on this machine.
22
+
23
+ ## Problem
24
+ `onnxruntime` is a runtime dependency in `kotoshu.gemspec`:
25
+ ```ruby
26
+ spec.add_dependency "onnxruntime", "~> 0.10"
27
+ ```
28
+
29
+ But `Models::OnnxModel` only does `require "onnxruntime"` when
30
+ `ENV['KOTOSHU_REQUIRE_ONNX']` is set. Worst of both worlds:
31
+
32
+ 1. `bundle install` can fail on platforms where `onnxruntime` doesn't
33
+ build — the whole gem becomes unusable even for non-semantic use.
34
+ 2. Even when installed, semantic analysis silently stays off until the
35
+ user sets an env var they have no way to discover.
36
+
37
+ ## Plan
38
+
39
+ ### Step 1 — Move onnxruntime to soft dependency
40
+ Remove from `spec.add_dependency`. Document in README that semantic
41
+ features require `gem install onnxruntime` (or `bundle add onnxruntime`).
42
+
43
+ ### Step 2 — Replace env-var gate with load-time rescue
44
+ In `lib/kotoshu/models/onnx_model.rb`:
45
+ ```ruby
46
+ begin
47
+ require "onnxruntime"
48
+ ONNX_LOADED = true
49
+ rescue LoadError => e
50
+ ONNX_LOADED = false
51
+ warn "kotoshu: onnxruntime not available (#{e.message}); semantic analysis disabled"
52
+ end
53
+ ```
54
+
55
+ ### Step 3 — Fail with actionable error when semantic is requested but unavailable
56
+ `OnnxModel.from_file` raises `SemanticError` (already exists) with:
57
+ ```
58
+ onnxruntime gem not installed. Install with: gem install onnxruntime
59
+ ```
60
+
61
+ ### Step 4 — Drop `KOTOSHU_REQUIRE_ONNX` env var
62
+ If users want to disable ONNX explicitly (e.g. to avoid the load warning),
63
+ support `KOTOSHU_NO_ONNX=1` as opt-OUT (inverted from current opt-IN).
64
+
65
+ ## Acceptance
66
+
67
+ - [ ] `gem install kotoshu` succeeds without `onnxruntime` installed.
68
+ - [ ] `Kotoshu.correct?("hello")` works without `onnxruntime` (traditional path).
69
+ - [ ] Calling semantic methods without onnxruntime raises a clear `SemanticError`.
70
+ - [ ] After `gem install onnxruntime`, semantic methods work without any env var.
71
+ - [ ] `KOTOSHU_NO_ONNX=1` disables semantic even when onnxruntime is installed.
72
+ - [ ] No mention of `KOTOSHU_REQUIRE_ONNX` in code or docs.
73
+
74
+ ## Dependencies
75
+ - None (independent).
@@ -0,0 +1,96 @@
1
+ # 26 — Generate .vocab.json for 9 active ONNX models (T1, 0.3-blocker)
2
+
3
+ ## Status
4
+ Implemented — and superseded in scope (re-verified 2026-09-03 against 0.6.0).
5
+
6
+ Evidence:
7
+
8
+ - `kotoshu/models-fasttext-onnx` now ships 27 `.vocab.json` files on
9
+ disk (9 languages × 3 tiers: full/fluency/mini), e.g.
10
+ `models/en/fasttext.en.vocab.json`, `models/en/fasttext.en.mini.vocab.json`;
11
+ `manifest.json` carries them with `"type": "vocab"` entries.
12
+ - Gem side consumes them: `lib/kotoshu/embeddings/vocabulary.rb`
13
+ (`Embeddings::Vocabulary.from_file` line 137, `.from_cache` line 269)
14
+ and `Models::OnnxModel.from_file` requires the `.vocab.json` sibling
15
+ (`lib/kotoshu/models/onnx_model.rb:86-93`).
16
+ - Cache side: `SourceRegistry` has a `model_vocab` source and
17
+ `ModelCache` fetches vocab siblings with SHA-256 verification against
18
+ the registry (`lib/kotoshu/cache/model_cache.rb`).
19
+
20
+ Superseded: the "9 flat models" world was replaced by the tiered
21
+ registry (registry.json, mini/fluency/full — TODO.impl/67 M1), so the
22
+ plan's manifest-with-18-entries shape no longer applies.
23
+
24
+ Not re-verified live today: `Kotoshu.setup(:en, want: %i[model])`
25
+ (114 MB download; `kotoshu status` shows model ✗ in the dev cache).
26
+
27
+ ## Problem
28
+ The 9 active ONNX models in `kotoshu/models-fasttext-onnx` have no
29
+ `.vocab.json` siblings:
30
+ ```
31
+ models/de/fasttext.de.onnx # 114 MB
32
+ models/de/fasttext.de.vocab.json # MISSING
33
+ ```
34
+
35
+ `OnnxModel.from_file` in the gem requires both files — the `.onnx` holds
36
+ only the embedding matrix; word → row index lives in `.vocab.json`.
37
+ Without it, every semantic lookup crashes.
38
+
39
+ The conversion script changes I made earlier (always emit vocab.json)
40
+ were post-hoc — they apply to FUTURE conversions, not the existing 158
41
+ files on disk.
42
+
43
+ ## Plan
44
+
45
+ ### Step 1 — Add a vocab-only extraction script
46
+ New script `scripts/extract_vocab_from_vec.py`:
47
+ - Input: FastText `.vec` file path + vocab size (default 100K)
48
+ - Output: `<stem>.vocab.json` matching the format emitted by
49
+ `fasttext_to_onnx.py`'s `save_vocabulary()`.
50
+
51
+ This is faster than re-running the full ONNX conversion (which would
52
+ also re-emit the 114 MB .onnx unnecessarily).
53
+
54
+ ### Step 2 — Download the 9 source .vec files from FastText CDN
55
+ ~7 GB total (compressed .vec.gz, ~700 MB per language). Cache under
56
+ `/tmp/fasttext-vec/` so re-runs don't re-download.
57
+
58
+ For each active language (de, en, es, fr, pt, ru, zh, ja, ko):
59
+ ```
60
+ curl -O https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/cc.<lang>.300.vec.gz
61
+ gunzip cc.<lang>.300.vec.gz
62
+ python3 scripts/extract_vocab_from_vec.py cc.<lang>.300.vec \
63
+ --vocab-size 100000 \
64
+ --output models/<lang>/fasttext.<lang>.vocab.json
65
+ ```
66
+
67
+ ### Step 3 — Verify vocab.json matches the existing .onnx
68
+ Each `.onnx` has `vocab_size` in its metadata. The `.vocab.json` must
69
+ have the same `vocab_size` AND the word order must match what was used
70
+ to build the embedding matrix. This is the load-bearing correctness
71
+ check — a mismatch makes embeddings return wrong vectors.
72
+
73
+ Verify by spot-checking a known word (e.g. "the" in en) and confirming
74
+ the embedding matches a reference value.
75
+
76
+ ### Step 4 — Regenerate manifest + commit + push
77
+ The 9 new `.vocab.json` files get added to `manifest.json` via the
78
+ existing `generate_manifest.rb` script.
79
+
80
+ ## Acceptance
81
+
82
+ - [ ] 9 `.vocab.json` files committed under `models/<lang>/`.
83
+ - [ ] `manifest.json` includes 18 entries (9 .onnx + 9 .vocab.json).
84
+ - [ ] Gem's `AVAILABLE_MODELS[:onnx]` unchanged (no code change).
85
+ - [ ] Live: `Kotoshu.setup(:en, want: %i[model])` succeeds.
86
+ - [ ] Live: `Kotoshu::Models::OnnxModel.from_file(...)` returns a working
87
+ model that can produce an embedding for "the".
88
+
89
+ ## Dependencies
90
+ - None (independent of gem code).
91
+
92
+ ## Risks
93
+ - 7 GB download may saturate a metered connection. Run on a connection
94
+ without caps.
95
+ - Vocab ordering mismatch with original conversion would silently
96
+ corrupt semantic results. Must verify by embedding spot-check.