kotoshu 1.0.2-x86_64-linux

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (458) hide show
  1. checksums.yaml +7 -0
  2. data/.kotoshu-baseline.json +1 -0
  3. data/.pre-commit-hooks.yaml +18 -0
  4. data/.rspec +3 -0
  5. data/.rubocop.yml +351 -0
  6. data/.yardopts +8 -0
  7. data/CHANGELOG.md +813 -0
  8. data/CLAUDE.md +176 -0
  9. data/CODE_OF_CONDUCT.md +132 -0
  10. data/Cargo.lock +359 -0
  11. data/Cargo.toml +15 -0
  12. data/LICENSE +31 -0
  13. data/README.adoc +1292 -0
  14. data/Rakefile +150 -0
  15. data/SECURITY.md +93 -0
  16. data/TODO.impl/00-cut-0.1.md +121 -0
  17. data/TODO.impl/00-cut-0.2.md +175 -0
  18. data/TODO.impl/00-vision.md +281 -0
  19. data/TODO.impl/01-hunspell-correctness.md +111 -0
  20. data/TODO.impl/02-cli-unification.md +74 -0
  21. data/TODO.impl/03-dynamic-download.md +107 -0
  22. data/TODO.impl/04-language-modules.md +89 -0
  23. data/TODO.impl/05-semantic-path.md +83 -0
  24. data/TODO.impl/06-cjk-support.md +105 -0
  25. data/TODO.impl/07-rtl-support.md +71 -0
  26. data/TODO.impl/08-grammar-engine.md +106 -0
  27. data/TODO.impl/09-integrity-security.md +90 -0
  28. data/TODO.impl/10-testing-ci.md +98 -0
  29. data/TODO.impl/100-full-feature-batch3.md +28 -0
  30. data/TODO.impl/101-int4-tier-experiment.md +32 -0
  31. data/TODO.impl/102-lid-chain.md +28 -0
  32. data/TODO.impl/103-oov-bucket-tables.md +27 -0
  33. data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
  34. data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
  35. data/TODO.impl/106-native-lid-detection.md +36 -0
  36. data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
  37. data/TODO.impl/108-ko-ne-full-feature.md +37 -0
  38. data/TODO.impl/109-ten-audit.md +26 -0
  39. data/TODO.impl/11-release-v1.md +95 -0
  40. data/TODO.impl/110-sr-latn-and-polish.md +33 -0
  41. data/TODO.impl/111-embedding-bakeoff.md +33 -0
  42. data/TODO.impl/112-deployability-and-gating.md +26 -0
  43. data/TODO.impl/113-language-packs.md +26 -0
  44. data/TODO.impl/114-typo-biencoder-thread.md +33 -0
  45. data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
  46. data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
  47. data/TODO.impl/117-deterministic-frequency-tiers.md +110 -0
  48. data/TODO.impl/118-server-gem-floor.md +33 -0
  49. data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
  50. data/TODO.impl/120-setup-metadata-refresh.md +35 -0
  51. data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
  52. data/TODO.impl/122-pack-compression-kpk2.md +66 -0
  53. data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
  54. data/TODO.impl/124-owner-residue.md +19 -0
  55. data/TODO.impl/125-consumer-floor-guards.md +43 -0
  56. data/TODO.impl/126-tier-error-budgets.md +33 -0
  57. data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
  58. data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
  59. data/TODO.impl/129-pt-fr-corpora.md +23 -0
  60. data/TODO.impl/130-news-truth-0912.md +23 -0
  61. data/TODO.impl/131-hybrid-productization.md +42 -0
  62. data/TODO.impl/132-techniques-page-and-prose.md +27 -0
  63. data/TODO.impl/133-rust-first-distribution.md +68 -0
  64. data/TODO.impl/20-encapsulation-cleanup.md +47 -0
  65. data/TODO.impl/21-autoload-migration.md +64 -0
  66. data/TODO.impl/22-url-pin-single-source-of-truth.md +120 -0
  67. data/TODO.impl/23-fix-url-defaults.md +57 -0
  68. data/TODO.impl/24-cli-auto-setup-prompt.md +76 -0
  69. data/TODO.impl/25-soft-onnxruntime-require.md +75 -0
  70. data/TODO.impl/26-onnx-vocab-json-generation.md +96 -0
  71. data/TODO.impl/27-end-to-end-smoke-spec.md +113 -0
  72. data/TODO.impl/28-readme-quickstart-verification.md +68 -0
  73. data/TODO.impl/29-status-command.md +84 -0
  74. data/TODO.impl/30-language-auto-detection.md +74 -0
  75. data/TODO.impl/31-download-progress.md +58 -0
  76. data/TODO.impl/32-sarif-output-verification.md +72 -0
  77. data/TODO.impl/33-audit-log-rotation.md +54 -0
  78. data/TODO.impl/34-cache-eviction.md +59 -0
  79. data/TODO.impl/35-shell-completion.md +61 -0
  80. data/TODO.impl/36-cleanup-regressions-0.3.1.md +134 -0
  81. data/TODO.impl/37-hunspell-correctness-tier2.md +203 -0
  82. data/TODO.impl/38-onnx-semantic-gating.md +218 -0
  83. data/TODO.impl/39-tier3-and-beyond.md +232 -0
  84. data/TODO.impl/40-spec-drift-cleanup.md +134 -0
  85. data/TODO.impl/50-structure-aware-document-api.md +194 -0
  86. data/TODO.impl/51-grammar-engine-expansion.md +75 -0
  87. data/TODO.impl/52-multi-language-documents.md +95 -0
  88. data/TODO.impl/53-document-plugin-architecture.md +94 -0
  89. data/TODO.impl/54-cjk-support.md +117 -0
  90. data/TODO.impl/55-rtl-support.md +109 -0
  91. data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
  92. data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
  93. data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
  94. data/TODO.impl/59-yard-api-documentation.md +75 -0
  95. data/TODO.impl/60-lsp-server.md +174 -0
  96. data/TODO.impl/61-editor-ecosystem.md +167 -0
  97. data/TODO.impl/62-framework-integrations.md +159 -0
  98. data/TODO.impl/63-web-playground-wasm.md +186 -0
  99. data/TODO.impl/64-http-api-and-sdks.md +191 -0
  100. data/TODO.impl/65-universal-kotoshu.md +251 -0
  101. data/TODO.impl/66-kotoshu-core.md +169 -0
  102. data/TODO.impl/67-kotoshu-rs-and-access-libraries.md +137 -0
  103. data/TODO.impl/68-sota-adoption.md +152 -0
  104. data/TODO.impl/69-sota-typo-eval.md +74 -0
  105. data/TODO.impl/70-sota-confidence-cascade.md +59 -0
  106. data/TODO.impl/71-sota-nested-model.md +63 -0
  107. data/TODO.impl/72-model2vec-evaluation.md +49 -0
  108. data/TODO.impl/73-rl-reranker.md +42 -0
  109. data/TODO.impl/74-distilled-reranker.md +36 -0
  110. data/TODO.impl/75-llm-evaluator.md +35 -0
  111. data/TODO.impl/76-optional-llm-backend.md +37 -0
  112. data/TODO.impl/77-model-coverage-expansion.md +96 -0
  113. data/TODO.impl/78-in-browser-playground.md +69 -0
  114. data/TODO.impl/79-news-and-releases.md +60 -0
  115. data/TODO.impl/80-ecosystem-docs.md +56 -0
  116. data/TODO.impl/81-editor-reach.md +68 -0
  117. data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
  118. data/TODO.impl/83-models-batch-2.md +56 -0
  119. data/TODO.impl/84-language-modules-wave2.md +75 -0
  120. data/TODO.impl/85-wasm-model-api.md +48 -0
  121. data/TODO.impl/86-site-truth-wave2.md +45 -0
  122. data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
  123. data/TODO.impl/88-cli-directory-mode.md +54 -0
  124. data/TODO.impl/89-framework-integrations-exec.md +59 -0
  125. data/TODO.impl/90-gem-correctness-residues.md +44 -0
  126. data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
  127. data/TODO.impl/92-tier-mirror-cors.md +88 -0
  128. data/TODO.impl/93-python-wheels.md +32 -0
  129. data/TODO.impl/94-site-integrations-docs.md +35 -0
  130. data/TODO.impl/95-editor-action-polish.md +27 -0
  131. data/TODO.impl/96-server-semantic-models.md +50 -0
  132. data/TODO.impl/97-yard-execution.md +73 -0
  133. data/TODO.impl/98-campaign-cleanup.md +78 -0
  134. data/TODO.impl/99-site-truth-performance-docs.md +31 -0
  135. data/conformance/vectors.jsonl +2630 -0
  136. data/data/en/grammar/rules.yaml +851 -0
  137. data/docs/STABILITY-1.0-AUDIT.md +251 -0
  138. data/docs/STABILITY-1.0.md +137 -0
  139. data/docs/cache-architecture.md +502 -0
  140. data/docs/verification-evidence/correct3/pr_body.md +37 -0
  141. data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
  142. data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
  143. data/docs/verification-evidence/correct3/sv.aff +856 -0
  144. data/docs/verification-evidence/correct3/sv.dic +153717 -0
  145. data/examples/01_basic_word_checking.rb +38 -0
  146. data/examples/02_text_document_checking.rb +77 -0
  147. data/examples/03_dictionary_backends.rb +137 -0
  148. data/examples/04_trie_data_structure.rb +146 -0
  149. data/examples/05_suggestion_algorithms.rb +239 -0
  150. data/examples/06_configuration_advanced.rb +287 -0
  151. data/examples/07_multi_language_dictionaries.rb +278 -0
  152. data/exe/kotoshu +6 -0
  153. data/lib/kotoshu/algorithms/capitalization.rb +276 -0
  154. data/lib/kotoshu/algorithms/edit_distance.rb +128 -0
  155. data/lib/kotoshu/algorithms/lookup/affix_form.rb +132 -0
  156. data/lib/kotoshu/algorithms/lookup/compound_checks.rb +209 -0
  157. data/lib/kotoshu/algorithms/lookup/compound_form.rb +50 -0
  158. data/lib/kotoshu/algorithms/lookup.rb +841 -0
  159. data/lib/kotoshu/algorithms/ngram_suggest.rb +367 -0
  160. data/lib/kotoshu/algorithms/permutations.rb +283 -0
  161. data/lib/kotoshu/algorithms/phonet_suggest.rb +162 -0
  162. data/lib/kotoshu/algorithms/soundex.rb +92 -0
  163. data/lib/kotoshu/algorithms/suggest.rb +794 -0
  164. data/lib/kotoshu/algorithms.rb +23 -0
  165. data/lib/kotoshu/analyzers/semantic_analyzer.rb +405 -0
  166. data/lib/kotoshu/analyzers.rb +8 -0
  167. data/lib/kotoshu/baseline/application.rb +18 -0
  168. data/lib/kotoshu/baseline/entry.rb +33 -0
  169. data/lib/kotoshu/baseline/store.rb +175 -0
  170. data/lib/kotoshu/baseline.rb +20 -0
  171. data/lib/kotoshu/cache/base_cache.rb +778 -0
  172. data/lib/kotoshu/cache/cache.rb +91 -0
  173. data/lib/kotoshu/cache/eviction_policy.rb +66 -0
  174. data/lib/kotoshu/cache/frequency_cache.rb +283 -0
  175. data/lib/kotoshu/cache/language_cache.rb +511 -0
  176. data/lib/kotoshu/cache/lookup_cache.rb +166 -0
  177. data/lib/kotoshu/cache/model_cache.rb +1147 -0
  178. data/lib/kotoshu/cache/model_registry.rb +109 -0
  179. data/lib/kotoshu/cache/staged_languages.rb +45 -0
  180. data/lib/kotoshu/cache/suggestion_cache.rb +113 -0
  181. data/lib/kotoshu/cache.rb +49 -0
  182. data/lib/kotoshu/cli/auto_setup.rb +71 -0
  183. data/lib/kotoshu/cli/baseline_command.rb +119 -0
  184. data/lib/kotoshu/cli/batch_reporter.rb +321 -0
  185. data/lib/kotoshu/cli/cache_command.rb +282 -0
  186. data/lib/kotoshu/cli/completions_command.rb +210 -0
  187. data/lib/kotoshu/cli/dict_command.rb +64 -0
  188. data/lib/kotoshu/cli/directory_check.rb +146 -0
  189. data/lib/kotoshu/cli/directory_walker.rb +132 -0
  190. data/lib/kotoshu/cli/display_formatter.rb +429 -0
  191. data/lib/kotoshu/cli/errors.rb +36 -0
  192. data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
  193. data/lib/kotoshu/cli/interactive_reviewer.rb +314 -0
  194. data/lib/kotoshu/cli/language_resolver.rb +91 -0
  195. data/lib/kotoshu/cli/navigation_manager.rb +273 -0
  196. data/lib/kotoshu/cli/personal_command.rb +113 -0
  197. data/lib/kotoshu/cli/progress_reporter.rb +114 -0
  198. data/lib/kotoshu/cli/status_report.rb +130 -0
  199. data/lib/kotoshu/cli.rb +880 -0
  200. data/lib/kotoshu/components/passthrough_spell_checker.rb +70 -0
  201. data/lib/kotoshu/components/pos_tagger.rb +98 -0
  202. data/lib/kotoshu/components/spell_checker.rb +73 -0
  203. data/lib/kotoshu/components/synthesizer.rb +60 -0
  204. data/lib/kotoshu/components/tokenizer.rb +58 -0
  205. data/lib/kotoshu/components/whitespace_tokenizer.rb +94 -0
  206. data/lib/kotoshu/components.rb +13 -0
  207. data/lib/kotoshu/configuration/builder.rb +207 -0
  208. data/lib/kotoshu/configuration/resolver.rb +124 -0
  209. data/lib/kotoshu/configuration.rb +829 -0
  210. data/lib/kotoshu/conformance_exporter.rb +256 -0
  211. data/lib/kotoshu/conformance_runner.rb +176 -0
  212. data/lib/kotoshu/core/exceptions.rb +186 -0
  213. data/lib/kotoshu/core/indexed_dictionary.rb +293 -0
  214. data/lib/kotoshu/core/models/affix_rule.rb +262 -0
  215. data/lib/kotoshu/core/models/result/document_result.rb +132 -0
  216. data/lib/kotoshu/core/models/result/word_result.rb +131 -0
  217. data/lib/kotoshu/core/models/word.rb +131 -0
  218. data/lib/kotoshu/core/trie/builder.rb +119 -0
  219. data/lib/kotoshu/core/trie/node.rb +94 -0
  220. data/lib/kotoshu/core/trie/trie.rb +249 -0
  221. data/lib/kotoshu/core.rb +28 -0
  222. data/lib/kotoshu/data/common_words/de.yml +1800 -0
  223. data/lib/kotoshu/data/common_words/en.yml +1215 -0
  224. data/lib/kotoshu/data/common_words/es.yml +750 -0
  225. data/lib/kotoshu/data/common_words/fr.yml +1015 -0
  226. data/lib/kotoshu/data/common_words/pt.yml +870 -0
  227. data/lib/kotoshu/data/common_words/ru.yml +484 -0
  228. data/lib/kotoshu/data/common_words_loader.rb +141 -0
  229. data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
  230. data/lib/kotoshu/data/frozen_kelly.rb +38 -0
  231. data/lib/kotoshu/data.rb +8 -0
  232. data/lib/kotoshu/data_structures/bloom_filter.rb +188 -0
  233. data/lib/kotoshu/data_structures.rb +8 -0
  234. data/lib/kotoshu/debug_logger.rb +146 -0
  235. data/lib/kotoshu/debug_mode.rb +136 -0
  236. data/lib/kotoshu/defaults.rb +84 -0
  237. data/lib/kotoshu/dictionaries/catalog.rb +814 -0
  238. data/lib/kotoshu/dictionaries.rb +8 -0
  239. data/lib/kotoshu/dictionary/base.rb +324 -0
  240. data/lib/kotoshu/dictionary/cspell.rb +243 -0
  241. data/lib/kotoshu/dictionary/custom.rb +225 -0
  242. data/lib/kotoshu/dictionary/hunspell.rb +520 -0
  243. data/lib/kotoshu/dictionary/plain_text.rb +302 -0
  244. data/lib/kotoshu/dictionary/repository.rb +246 -0
  245. data/lib/kotoshu/dictionary/unified.rb +236 -0
  246. data/lib/kotoshu/dictionary/unix_words.rb +218 -0
  247. data/lib/kotoshu/dictionary.rb +15 -0
  248. data/lib/kotoshu/documents/document.rb +120 -0
  249. data/lib/kotoshu/documents/plain_text_document.rb +73 -0
  250. data/lib/kotoshu/documents/source_position.rb +45 -0
  251. data/lib/kotoshu/documents/source_range.rb +76 -0
  252. data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
  253. data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
  254. data/lib/kotoshu/documents/suppressions.rb +44 -0
  255. data/lib/kotoshu/documents/text_node.rb +49 -0
  256. data/lib/kotoshu/documents.rb +178 -0
  257. data/lib/kotoshu/embeddings/embedding_pipeline.rb +238 -0
  258. data/lib/kotoshu/embeddings/lru_cache.rb +244 -0
  259. data/lib/kotoshu/embeddings/onnx_runtime_model.rb +411 -0
  260. data/lib/kotoshu/embeddings/protocol.rb +96 -0
  261. data/lib/kotoshu/embeddings/protocols.rb +23 -0
  262. data/lib/kotoshu/embeddings/registry.rb +184 -0
  263. data/lib/kotoshu/embeddings/search.rb +204 -0
  264. data/lib/kotoshu/embeddings/similarity_engine.rb +278 -0
  265. data/lib/kotoshu/embeddings/similarity_search.rb +328 -0
  266. data/lib/kotoshu/embeddings/vocabulary.rb +306 -0
  267. data/lib/kotoshu/embeddings.rb +87 -0
  268. data/lib/kotoshu/fluent_checker.rb +94 -0
  269. data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +48 -0
  270. data/lib/kotoshu/grammar/pattern_matchers/double_negative_matcher.rb +179 -0
  271. data/lib/kotoshu/grammar/pattern_matchers/phrase_matcher.rb +86 -0
  272. data/lib/kotoshu/grammar/pattern_matchers/possessive_context_matcher.rb +75 -0
  273. data/lib/kotoshu/grammar/pattern_matchers/possessive_contraction_matcher.rb +87 -0
  274. data/lib/kotoshu/grammar/pattern_matchers/sentence_start_matcher.rb +84 -0
  275. data/lib/kotoshu/grammar/pattern_matchers/vowel_sound_matcher.rb +81 -0
  276. data/lib/kotoshu/grammar/pattern_matchers/word_list_matcher.rb +66 -0
  277. data/lib/kotoshu/grammar/rule.rb +99 -0
  278. data/lib/kotoshu/grammar/rule_engine.rb +96 -0
  279. data/lib/kotoshu/grammar/rule_loader.rb +30 -0
  280. data/lib/kotoshu/grammar.rb +24 -0
  281. data/lib/kotoshu/integrity/audit_log.rb +167 -0
  282. data/lib/kotoshu/integrity/manifest.rb +116 -0
  283. data/lib/kotoshu/integrity/net_http.rb +51 -0
  284. data/lib/kotoshu/integrity/rotation_policy.rb +88 -0
  285. data/lib/kotoshu/integrity.rb +26 -0
  286. data/lib/kotoshu/jekyll.rb +101 -0
  287. data/lib/kotoshu/keyboard/layout.rb +116 -0
  288. data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
  289. data/lib/kotoshu/keyboard/layouts/azerty.rb +55 -0
  290. data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
  291. data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
  292. data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
  293. data/lib/kotoshu/keyboard/layouts/dvorak.rb +54 -0
  294. data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
  295. data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
  296. data/lib/kotoshu/keyboard/layouts/hrsl.rb +94 -0
  297. data/lib/kotoshu/keyboard/layouts/jcuken.rb +57 -0
  298. data/lib/kotoshu/keyboard/layouts/latin.rb +266 -0
  299. data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
  300. data/lib/kotoshu/keyboard/layouts/qwerty.rb +52 -0
  301. data/lib/kotoshu/keyboard/layouts/qwertz.rb +55 -0
  302. data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
  303. data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
  304. data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
  305. data/lib/kotoshu/keyboard/registry.rb +201 -0
  306. data/lib/kotoshu/keyboard.rb +96 -0
  307. data/lib/kotoshu/kotoshu_native.so +0 -0
  308. data/lib/kotoshu/language/detector.rb +242 -0
  309. data/lib/kotoshu/language/identifier.rb +376 -0
  310. data/lib/kotoshu/language/languages/base.rb +271 -0
  311. data/lib/kotoshu/language/lid_detector.rb +188 -0
  312. data/lib/kotoshu/language/normalizer/arabic.rb +113 -0
  313. data/lib/kotoshu/language/normalizer/base.rb +133 -0
  314. data/lib/kotoshu/language/normalizer/greek.rb +63 -0
  315. data/lib/kotoshu/language/normalizer/hebrew.rb +72 -0
  316. data/lib/kotoshu/language/normalizer/persian.rb +36 -0
  317. data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
  318. data/lib/kotoshu/language/registry.rb +209 -0
  319. data/lib/kotoshu/language/resources/ar/common_words.txt +6753 -0
  320. data/lib/kotoshu/language/resources/ar/confusion_sets.txt +11 -0
  321. data/lib/kotoshu/language/resources/de/common_words.txt +10003 -0
  322. data/lib/kotoshu/language/resources/de/confusion_sets.txt +246 -0
  323. data/lib/kotoshu/language/resources/en/common_words.txt +9979 -0
  324. data/lib/kotoshu/language/resources/en/confusion_sets.txt +871 -0
  325. data/lib/kotoshu/language/resources/es/common_words.txt +9992 -0
  326. data/lib/kotoshu/language/resources/es/confusion_sets.txt +17 -0
  327. data/lib/kotoshu/language/resources/fr/common_words.txt +9993 -0
  328. data/lib/kotoshu/language/resources/fr/confusion_sets.txt +76 -0
  329. data/lib/kotoshu/language/resources/pt/common_words.txt +9977 -0
  330. data/lib/kotoshu/language/resources/pt/confusion_sets.txt +18 -0
  331. data/lib/kotoshu/language/resources/ru/common_words.txt +9951 -0
  332. data/lib/kotoshu/language/resources/ru/confusion_sets.txt +5 -0
  333. data/lib/kotoshu/language/script.rb +128 -0
  334. data/lib/kotoshu/language/segmenter.rb +122 -0
  335. data/lib/kotoshu/language/suika.rb +38 -0
  336. data/lib/kotoshu/language/tokenizer/base.rb +187 -0
  337. data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
  338. data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
  339. data/lib/kotoshu/language/tokenizer/french_tokenizer.rb +172 -0
  340. data/lib/kotoshu/language/tokenizer/german_tokenizer.rb +41 -0
  341. data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
  342. data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
  343. data/lib/kotoshu/language/tokenizer/japanese_tokenizer.rb +55 -0
  344. data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +153 -0
  345. data/lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb +160 -0
  346. data/lib/kotoshu/language/tokenizer/russian_tokenizer.rb +93 -0
  347. data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
  348. data/lib/kotoshu/language/tokenizer/spanish_tokenizer.rb +122 -0
  349. data/lib/kotoshu/language.rb +127 -0
  350. data/lib/kotoshu/languages/ar/language.rb +98 -0
  351. data/lib/kotoshu/languages/bg/language.rb +37 -0
  352. data/lib/kotoshu/languages/ca/language.rb +22 -0
  353. data/lib/kotoshu/languages/cs/language.rb +22 -0
  354. data/lib/kotoshu/languages/da/language.rb +22 -0
  355. data/lib/kotoshu/languages/de/language.rb +554 -0
  356. data/lib/kotoshu/languages/el/language.rb +50 -0
  357. data/lib/kotoshu/languages/en/language.rb +470 -0
  358. data/lib/kotoshu/languages/es/language.rb +476 -0
  359. data/lib/kotoshu/languages/et/language.rb +24 -0
  360. data/lib/kotoshu/languages/fa/language.rb +91 -0
  361. data/lib/kotoshu/languages/fr/language.rb +505 -0
  362. data/lib/kotoshu/languages/he/language.rb +98 -0
  363. data/lib/kotoshu/languages/hr/language.rb +24 -0
  364. data/lib/kotoshu/languages/hu/language.rb +22 -0
  365. data/lib/kotoshu/languages/id/language.rb +25 -0
  366. data/lib/kotoshu/languages/it/language.rb +22 -0
  367. data/lib/kotoshu/languages/ja/language.rb +476 -0
  368. data/lib/kotoshu/languages/ko/language.rb +56 -0
  369. data/lib/kotoshu/languages/latin_base.rb +53 -0
  370. data/lib/kotoshu/languages/lt/language.rb +24 -0
  371. data/lib/kotoshu/languages/lv/language.rb +24 -0
  372. data/lib/kotoshu/languages/nb/language.rb +25 -0
  373. data/lib/kotoshu/languages/ne/language.rb +60 -0
  374. data/lib/kotoshu/languages/nl/language.rb +22 -0
  375. data/lib/kotoshu/languages/nn/language.rb +26 -0
  376. data/lib/kotoshu/languages/pl/language.rb +22 -0
  377. data/lib/kotoshu/languages/pt/language.rb +439 -0
  378. data/lib/kotoshu/languages/ro/language.rb +22 -0
  379. data/lib/kotoshu/languages/ru/language.rb +420 -0
  380. data/lib/kotoshu/languages/sk/language.rb +25 -0
  381. data/lib/kotoshu/languages/sl/language.rb +25 -0
  382. data/lib/kotoshu/languages/sr/language.rb +39 -0
  383. data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
  384. data/lib/kotoshu/languages/sv/language.rb +22 -0
  385. data/lib/kotoshu/languages/tr/language.rb +36 -0
  386. data/lib/kotoshu/languages/uk/language.rb +38 -0
  387. data/lib/kotoshu/languages/vi/language.rb +22 -0
  388. data/lib/kotoshu/languages.rb +64 -0
  389. data/lib/kotoshu/metrics_collector.rb +238 -0
  390. data/lib/kotoshu/metrics_module.rb +121 -0
  391. data/lib/kotoshu/models/context.rb +119 -0
  392. data/lib/kotoshu/models/embedding_model.rb +181 -0
  393. data/lib/kotoshu/models/fasttext_model.rb +212 -0
  394. data/lib/kotoshu/models/nearest_neighbor.rb +96 -0
  395. data/lib/kotoshu/models/onnx_model.rb +330 -0
  396. data/lib/kotoshu/models/semantic_error.rb +204 -0
  397. data/lib/kotoshu/models/suggestion.rb +106 -0
  398. data/lib/kotoshu/models/word_embedding.rb +97 -0
  399. data/lib/kotoshu/models.rb +27 -0
  400. data/lib/kotoshu/multi_language_checker.rb +127 -0
  401. data/lib/kotoshu/native.rb +83 -0
  402. data/lib/kotoshu/native_backend.rb +131 -0
  403. data/lib/kotoshu/paths.rb +79 -0
  404. data/lib/kotoshu/personal_dictionary.rb +105 -0
  405. data/lib/kotoshu/plugins/plugin.rb +61 -0
  406. data/lib/kotoshu/plugins/registry.rb +118 -0
  407. data/lib/kotoshu/plugins.rb +9 -0
  408. data/lib/kotoshu/project_config.rb +76 -0
  409. data/lib/kotoshu/readers/aff_data.rb +530 -0
  410. data/lib/kotoshu/readers/aff_reader.rb +393 -0
  411. data/lib/kotoshu/readers/condition_checker.rb +125 -0
  412. data/lib/kotoshu/readers/dic_reader.rb +208 -0
  413. data/lib/kotoshu/readers/file_reader.rb +377 -0
  414. data/lib/kotoshu/readers/lookup_builder.rb +584 -0
  415. data/lib/kotoshu/readers/readers.rb +17 -0
  416. data/lib/kotoshu/readers.rb +25 -0
  417. data/lib/kotoshu/resource_bundle.rb +55 -0
  418. data/lib/kotoshu/resource_manager.rb +600 -0
  419. data/lib/kotoshu/results/result.rb +165 -0
  420. data/lib/kotoshu/results.rb +8 -0
  421. data/lib/kotoshu/rspec.rb +163 -0
  422. data/lib/kotoshu/scripts/fasttext_to_onnx.py +275 -0
  423. data/lib/kotoshu/source_registry.rb +104 -0
  424. data/lib/kotoshu/spellchecker/parallel_checker.rb +90 -0
  425. data/lib/kotoshu/spellchecker.rb +406 -0
  426. data/lib/kotoshu/string_metrics.rb +153 -0
  427. data/lib/kotoshu/suggestions/context.rb +43 -0
  428. data/lib/kotoshu/suggestions/frequency_provider.rb +80 -0
  429. data/lib/kotoshu/suggestions/generator.rb +169 -0
  430. data/lib/kotoshu/suggestions/pipeline.rb +135 -0
  431. data/lib/kotoshu/suggestions/semantic_cascade.rb +73 -0
  432. data/lib/kotoshu/suggestions/strategies/base_strategy.rb +315 -0
  433. data/lib/kotoshu/suggestions/strategies/composite_strategy.rb +197 -0
  434. data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +631 -0
  435. data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +207 -0
  436. data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +165 -0
  437. data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +284 -0
  438. data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +317 -0
  439. data/lib/kotoshu/suggestions/strategies/symspell_strategy.rb +270 -0
  440. data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
  441. data/lib/kotoshu/suggestions/suggestion.rb +154 -0
  442. data/lib/kotoshu/suggestions/suggestion_set.rb +270 -0
  443. data/lib/kotoshu/suggestions/sweep_index.rb +137 -0
  444. data/lib/kotoshu/suggestions.rb +29 -0
  445. data/lib/kotoshu/tasks/check_task.rb +104 -0
  446. data/lib/kotoshu/tasks.rb +11 -0
  447. data/lib/kotoshu/validators/spelling_validator.rb +101 -0
  448. data/lib/kotoshu/validators.rb +9 -0
  449. data/lib/kotoshu/version.rb +5 -0
  450. data/lib/kotoshu.rb +538 -0
  451. data/rakelib/compile.rake +10 -0
  452. data/script/validate_all_dictionaries.rb +444 -0
  453. data/scripts/build_native_gem.rb +45 -0
  454. data/scripts/extract_eval_grids.rb +37 -0
  455. data/scripts/generate_frozen_tiers.rb +44 -0
  456. data/sig/kotoshu.rbs +435 -0
  457. data/test_oop.rb +79 -0
  458. metadata +546 -0
@@ -0,0 +1,32 @@
1
+ # Plan 104 — VS Code marketplace + Open VSX readiness
2
+
3
+ ## Why
4
+ The extension is built, CI-green, with a .vsix artifact — but it is not
5
+ installable without the marketplace listing, which is the single largest
6
+ adoption blocker for the editor audience. Publishing needs the owner's
7
+ publisher PAT (a credential we never hold); everything else can be ready
8
+ so publishing is one secret + one click.
9
+
10
+ ## Work (vscode repo)
11
+ 1. Open VSX publish workflow (release-ovsx.yml, secret-gated) beside the
12
+ existing CI — Open VSX serves VSCodium/Code-OSS users without the
13
+ Microsoft marketplace.
14
+ 2. Metadata polish: gallery banner/colors, keywords, categories, repository
15
+ links, the kotoshu-vscode README (quickstart with kotoshu-lsp install,
16
+ settings reference: log level, personal dictionary command).
17
+ 3. PREPUBLISH.md: exact owner steps — create publisher, add PAT as secret,
18
+ run the workflow; nothing else.
19
+ 4. Site follow-up line (rides plan 99's pass): /docs/clients/lsp gains the
20
+ marketplace link placeholder to fill on publish day.
21
+
22
+ ## Verification
23
+ `vsce package` clean with no warnings; workflow lint-valid; PREPUBLISH
24
+ steps verified against the actual GitHub UI labels.
25
+
26
+ ## Status
27
+ Executed 2026-09-07 — vscode PR #3, merged. release-marketplace.yml (VSCE_PAT)
28
+ + release-ovsx.yml (OPEN_VSX_TOKEN), both version-gated; PREPUBLISH.md with
29
+ verified owner steps; package.json publisher/categories/keywords/banner;
30
+ README quickstart against gem 0.1.1; vsce package warning-clean. Publishing
31
+ itself is one PAT secret + one tag — owner action. The /docs/clients/lsp
32
+ marketplace link rides the coordinated site pass.
@@ -0,0 +1,34 @@
1
+ # Plan 105 — The personal dictionary reaches the check path
2
+
3
+ ## Why
4
+ `kotoshu personal add recieve` then `kotoshu check` still flags the word —
5
+ PersonalDictionary is wired only into the `personal` subcommand and the LSP
6
+ (resolved server-side), not into `Spellchecker#check`. Users who add words
7
+ see them flagged again on the very next run; the README story ("add words
8
+ you accept") is currently false on the CLI/API path.
9
+
10
+ ## Work (gem repo)
11
+ 1. `Spellchecker#check` consults the personal dictionary: a word present in
12
+ personal.dic is not an error (mirror the LSP's semantics: filter at
13
+ result assembly, keep suppression metadata out of the error rows).
14
+ 2. Reload semantics: the LSP reloads on mtime change; the CLI/one-shot path
15
+ loads once per process. Do not add hot reload here.
16
+ 3. Opt-out: `Kotoshu.configure { personal_dictionary false }` (or the
17
+ existing config seam — find it) and `--no-personal` on `kotoshu check`.
18
+ 4. Specs: add-then-check round trip on a real dictionary; opt-out flags
19
+ again; personal words still spellcheckable via suggest.
20
+ 5. Docs truth: /docs/ignores gains the personal dictionary section (the
21
+ action PR #2 README noted this gap) — deliver the copy in the PR body
22
+ for the site follow-up, do not touch the site repo.
23
+
24
+ ## Verification
25
+ Suite green (3,900+ examples); rubocop clean; `kotoshu personal add x &&
26
+ kotoshu check` live round trip recorded in the PR body.
27
+
28
+ ## Status
29
+ Executed 2026-09-07 — gem PR (branch plan-105-personal-dict-in-check):
30
+ check-path filter mirroring the LSP, one load per process,
31
+ Configuration#personal_dictionary + KOTOSHU_PERSONAL_DICTIONARY +
32
+ `kotoshu check --no-personal`; suite pins KOTOSHU_PERSONAL_DIC for
33
+ hermeticity. /docs/ignores copy delivered in the PR body for the site
34
+ follow-up.
@@ -0,0 +1,36 @@
1
+ # Plan 106 — Document language detection through the pure-Rust LID reader
2
+
3
+ ## Why
4
+ The gem's LanguageIdentifier needs the Python fastText bindings (and breaks
5
+ under numpy 2), so /v1/detect ships a 7-language heuristic. kotoshu-rs now
6
+ carries a pure-Rust lid.176 reader (kotoshu/src/lid, parity 55/55 vs the
7
+ gem) behind the same `model` feature the native ext already builds.
8
+
9
+ ## Work (gem repo)
10
+ 1. Expose `Kotoshu.detect_language(text)` through the native backend:
11
+ download/resolve lid.176.onnx + vocab via the models registry
12
+ (kotoshu://models/lid/lid-176, v1.4.0), score with the ext, return
13
+ { code, score }. Heuristic Language::Detector stays the fallback when
14
+ the native ext is absent or the model is not set up.
15
+ 2. FFI seam: extend the magnus shim (ext/kotoshu_native) with the call —
16
+ mirror how suggest/check already cross; keep it behind the existing
17
+ feature so pure-Ruby installs are unaffected.
18
+ 3. Server follow-up (kotoshu-server repo, separate PR riding this):
19
+ /v1/detect prefers Kotoshu.detect_language when available. Note it in
20
+ the PR body; do not touch the server repo yourself.
21
+ 4. Specs: native-vs-fixture parity (reuse the rs corpus JSON shape),
22
+ fallback behavior, registry miss error.
23
+
24
+ ## Verification
25
+ Suite green; a live detect over en/de/ru/ja/ar samples recorded in the PR
26
+ body; pure-Ruby suite still green (no new hard dependency).
27
+
28
+ ## Status
29
+ Executed 2026-09-07 — gem PR (branch plan-106-native-lid-detection):
30
+ Kotoshu.detect_language -> Language::Detection through the ext
31
+ (LidModel.load/#detect in the magnus shim, kotoshu-rs model feature,
32
+ Cargo.lock bumped to the reader revision); ModelCache gained the
33
+ lid pair (download_lid/load_cached_lid, top-level lid/ dir);
34
+ Kotoshu.setup_lid two-stage; heuristic fallback incl.
35
+ KOTOSHU_BACKEND=ruby; parity 55/55 vs the shared frozen corpus.
36
+ Server follow-up (/v1/detect) noted in the PR body.
@@ -0,0 +1,42 @@
1
+ # Plan 107 — Basic support for every staged dictionary language
2
+
3
+ ## Why
4
+ `kotoshu setup` only serves the 32 languages with modules. Sixty-plus staged
5
+ dictionaries (from gd to ia) are unreachable: a Welsh or Icelandic user
6
+ cannot use the gem at all, even though the dictionary and model are
7
+ downloadable. Modules should be the UPGRADE (keyboard + verified specimens),
8
+ not the gate.
9
+
10
+ ## Work (gem repo)
11
+ 1. `LanguageCache::AVAILABLE_LANGUAGES` derives from the dictionaries
12
+ manifest (the source the registry/staging uses) instead of the hardcoded
13
+ module list; modules register themselves as full-feature upgrades.
14
+ 2. Default behavior for module-less languages: script-appropriate tokenizer
15
+ (reuse the script-regex machinery from plan 91), generic keyboard
16
+ fallback (QWERTY for Latin, JCUKEN Cyrillic, Arabic 101 Arabic-script),
17
+ setup/check/suggest all functional.
18
+ 3. Wire nn (model + dictionary exist) as a full-feature module while here;
19
+ note sr-Latn as follow-up if the Latin mapping needs product sign-off.
20
+ 4. Specs: setup + specimen round trip on 3 module-less staged languages
21
+ (e.g. is, cy, gd); full-feature languages behave identically to today;
22
+ conformance untouched.
23
+
24
+ ## Verification
25
+ Live `kotoshu setup is && check -l is` round trip; suite green; the 32
26
+ modules' outputs byte-identical.
27
+
28
+ ## Status
29
+ Executed 2026-09-07 — AVAILABLE_LANGUAGES derives from a vendored
30
+ snapshot of the dictionaries manifest (Cache::StagedLanguages, 95
31
+ languages, refreshed by `rake kotoshu:staged_languages:sync`);
32
+ module-less languages ride Language::Script (script tokenizer for
33
+ word extraction: shared Latin/Cyrillic/Greek classes plus a generic
34
+ regex tokenizer for Arabic/Hebrew/Armenian/Georgian/Devanagari/
35
+ Hangul) and the script keyboard default (JCUKEN Cyrillic, Arabic 101,
36
+ Hebrew SI-1452, QWERTY Latin). Tiers documented: full feature =
37
+ module (LanguageCache.full_feature_languages, 33 with nn); basic =
38
+ manifest-present. nn wired as a full-feature module. Specimens
39
+ engine-verified for is cy gd (basic) + nn; live round trips
40
+ setup+check -l for is cy gd nn; conformance 2630 vectors 0 failures;
41
+ suite 3948 examples 0 failures. sr-Latn stays follow-up (normalize
42
+ maps it to sr; needs product sign-off).
@@ -0,0 +1,37 @@
1
+ # Plan 108 — Korean and Nepali: the last big-population languages
2
+
3
+ ## Why
4
+ ko (~82M) and ne (~16M) were deferred from batch 3 because Hangul jamo
5
+ input and Devanagari conjuncts need ja-style tokenizer treatment. They are
6
+ the largest populations without full-feature support; both have staged
7
+ dictionaries and models.
8
+
9
+ ## Work (gem repo; sequenced AFTER plan 107 merges — same area)
10
+ 1. ko: Hangul tokenizer (syllable-block aware, jamo-level word regex so
11
+ check extracts real eojeol), Korean keyboard grid (Dubeolsik 2-set),
12
+ module + specimens engine-verified.
13
+ 2. ne: Devanagari tokenizer (matras and conjuncts stay attached to the
14
+ base consonant — the Unicode grapheme rules the script needs), Roman
15
+ keyboard fallback or a Nepali traditional grid if defensibly sourced,
16
+ module + specimens.
17
+ 3. AVAILABLE_LANGUAGES + site-flip deliverables (specimen table, sizes).
18
+
19
+ ## Verification
20
+ Same discipline as batch 3: every specimen engine-verified through
21
+ correct?/suggest; RTL/complex-script sentence round trips through
22
+ Kotoshu.check; suite green.
23
+
24
+ ## Status
25
+ Executed 2026-09-07 — ko: HangulTokenizer (eojeol = run of syllables
26
+ U+AC00-D7A3 plus bare jamo U+1100-11FF/U+3130-318F), Dubeolsik grid
27
+ keyed on jamo per KS X 5002 (the eval harness models ko slips as
28
+ jamo confusion pairs, not a key grid, so no drift check applies),
29
+ module + 7 engine-verified specimens. ne: DevanagariTokenizer
30
+ (matras and virama conjuncts attached to the base consonant; danda
31
+ and Devanagari digits are separators), Devanagari InScript grid
32
+ mirrored from the eval harness _NE_INSCRIPT and drift-checked (the
33
+ defensible standard — Traditional Romanized is common but
34
+ undocumented), module + 3 specimens. Full-feature languages 33 -> 35.
35
+ Live round trips setup + check -l ko ne recorded; conformance 2630
36
+ vectors 0 failures; suite 3985 examples 0 failures. Specimen table
37
+ and dictionary sizes in the PR body for the site flip.
@@ -0,0 +1,26 @@
1
+ # Plan 109 — The 1.0 readiness audit
2
+
3
+ ## Why
4
+ Adopters hesitate at 0.x. A 1.0 cut needs a frozen public surface and a
5
+ stated policy. The audit inventories what 1.0 would freeze and what it
6
+ would deprecate — it does NOT cut versions (owner decision).
7
+
8
+ ## Work (gem repo docs/, analysis only)
9
+ 1. Inventory the public surface: Ruby API (Kotoshu module functions,
10
+ Spellchecker, Configuration, Tasks, validators), CLI flags, wasm JS
11
+ surface (KotoshuWasm, loadModel/rerank/semanticSuggest/loadLid/
12
+ detectLanguage), HTTP contract (openapi.yaml), action inputs, registry
13
+ schema.
14
+ 2. For each: stable / needs-deprecation-notice / internal-accidentally-
15
+ public. Flag anything accidentally public that 1.0 should privatize.
16
+ 3. Draft the stability policy (semver commitments, deprecation window,
17
+ conformance vectors as the behavioral contract) as docs/STABILITY.md.
18
+ 4. A 1.0 checklist: what must land first (nothing mechanical is known to
19
+ block), what the release train looks like (gem+wasm+server together).
20
+
21
+ ## Verification
22
+ Every inventory line cites the file that defines it; the deprecation list
23
+ reviewed against gemspec/exported constants.
24
+
25
+ ## Status
26
+ Pending
@@ -0,0 +1,95 @@
1
+ # 11 — Release v1.0
2
+
3
+ ## Goal
4
+
5
+ Cut a v1.0.0 release of the `kotoshu` gem with everything in plans 01-10
6
+ landed (or explicitly descoped), and a published v1.0.0 tag of each
7
+ content repo.
8
+
9
+ ## Why
10
+
11
+ No gem release exists. `git tag` shows nothing on `main`. The gemspec
12
+ already references `CHANGELOG.md` which doesn't exist. v1 is the
13
+ commitment that the public API is stable and the supported-language
14
+ matrix is real.
15
+
16
+ ## Tasks
17
+
18
+ 1. **API freeze audit.** Walk every public method on the `Kotoshu`
19
+ module (`lib/kotoshu.rb`) and the `Configuration` class. Ensure
20
+ every signature is documented, every option is in `SCHEMA`, and the
21
+ YARD docs are complete. After v1, breaking changes require v2.
22
+ 2. **CHANGELOG.md.** Write the v1.0.0 entry following Keep a Changelog
23
+ format. The gemspec metadata points at this file.
24
+ 3. **README accuracy pass.** Fix:
25
+ - License (says MIT, actually BSD-2-Clause)
26
+ - ONNX model count (says 6, will be ~30+ at v1)
27
+ - Language matrix (must match `Kotoshu.supported_languages`)
28
+ - CLI examples (every example must work verbatim)
29
+ 4. **Deprecation policy.** Add `SECURITY.md` (from plan 09) and
30
+ `POLICY.md` documenting semver, supported Ruby versions, EOL
31
+ cadence.
32
+ 5. **Sign the gem** with a project rubygems.org account. Document
33
+ who has release privileges.
34
+ 6. **Content repo tags.** Tag each content repo in lockstep:
35
+ - `dictionaries` → `v1.0.0`
36
+ - `frequency-list-kelly` → `v1.0.0`
37
+ - `models-fasttext-onnx` → `v1.0.0`
38
+ Pin the gem's default `resource_sources` URLs to these tags (via
39
+ the manifest system from plan 09).
40
+ 7. **Release automation.** GitHub Action that, on tag push:
41
+ - Builds the gem
42
+ - Publishes to rubygems.org
43
+ - Creates a GitHub Release with auto-generated notes
44
+ 8. **Post-release smoke test.** A script that, on a fresh Docker
45
+ container with only Ruby installed:
46
+ - `gem install kotoshu`
47
+ - `ruby -e 'require "kotoshu"; puts Kotoshu.correct?("hello")'`
48
+ - Exercises each documented CLI example from the README
49
+ 9. **Announcement.** Blog post on kotoshu.github.io (once it exists,
50
+ plan depends on `kotoshu.github.io/TODO.impl/`), Ruby Weekly
51
+ submission, Reddit /r/ruby.
52
+
53
+ ## v1 scope (locked)
54
+
55
+ In scope:
56
+ - Traditional Hunspell path (Latin scripts) — plans 01, 04
57
+ - Semantic ONNX reranking — plan 05
58
+ - Dynamic download — plan 03
59
+ - Integrity verification — plan 09
60
+ - Grammar rules (starter pack for the 6 fully-supported languages) —
61
+ plan 08
62
+ - CLI with all documented flags — plan 02
63
+ - ≥ 30 supported languages with end-to-end coverage
64
+
65
+ Out of scope (post-v1):
66
+ - CJK full support — plan 06 (may ship experimental in v1.1)
67
+ - RTL full support — plan 07 (same)
68
+ - LSP / HTTP server
69
+ - Editor plugins
70
+
71
+ ## Acceptance criteria
72
+
73
+ - `gem install kotoshu` works on a clean Ruby 3.1+ machine
74
+ - `Kotoshu::VERSION == "1.0.0"`
75
+ - `git tag v1.0.0` exists on `kotoshu/kotoshu`, `kotoshu/dictionaries`,
76
+ `kotoshu/frequency-list-kelly`, `kotoshu/models-fasttext-onnx`
77
+ - The post-release smoke test passes
78
+ - rubygems.org page shows the gem with correct metadata
79
+ - README's first 100 lines are accurate (no broken examples)
80
+
81
+ ## Dependencies
82
+
83
+ - Blocked by: all of plans 01-10
84
+ - Cross-repo: each content repo's `TODO.impl/03-*-releases.md` (or
85
+ equivalent)
86
+
87
+ ## Out of scope
88
+
89
+ - v1.1, v2 planning (post-release)
90
+ - Marketing beyond the announcement
91
+ - Conference talks (delightful but not blocking)
92
+
93
+ ## Status
94
+
95
+ _Pending._
@@ -0,0 +1,33 @@
1
+ # Plan 110 — sr-Latn wiring and the long-tail polish list
2
+
3
+ ## Why
4
+ Plan 107 wired nn and deferred sr-Latn (the Latin script variant of Serbian)
5
+ as follow-up. With the basic tier live, this is the last named language gap
6
+ in the corpus; everything after it is small-polish.
7
+
8
+ ## Work (gem repo)
9
+ 1. sr-Latn: register as a full-feature module reusing the Serbian dictionary
10
+ with the Latin tokenizer + the Croatian/Slovenian QWERTZ grid (the
11
+ scripts share the Latin layout), or map sr-Latn -> the staged sr-Latn
12
+ dictionary if one exists in the manifest — follow the data, document the
13
+ choice. Specimens engine-verified.
14
+ 2. Polish sweep: any spellcheck_word_regex script set lacking a staged
15
+ language (inventory + one-line table in the PR body); README language
16
+ table refresh to the 35+ tiers.
17
+ 3. Frequency-list inventory (no sourcing): list which staged languages have
18
+ published frequency data wired vs not — record only, sourcing is a
19
+ separate owner decision.
20
+
21
+ ## Verification
22
+ Live setup + check -l sr-Latn round trip; suite green; conformance
23
+ untouched.
24
+
25
+ ## Status
26
+ Executed 2026-09-07 — the data chose the distinct staged dictionary:
27
+ sr-Latn is a full-feature module (Languages::SerbianLatin, Latin
28
+ tokenizer, Serbian-Latin QWERTZ grid sharing the hrsl key table), and
29
+ the sr-Latn code now survives ResourceManager normalization
30
+ (script subtags kept, region subtags still fold) so the staged Latin
31
+ files are reachable — previously sr-Latn collapsed onto the Cyrillic
32
+ sr dictionary. Script-set and frequency inventories recorded in the
33
+ PR body (record-only). Suite green, conformance untouched.
@@ -0,0 +1,33 @@
1
+ # Plan 111 — The embedding bake-off: is fastText still the right model?
2
+
3
+ ## Why
4
+ Our gates prove int8 near-lossless RELATIVE to fastText — but fastText is
5
+ 2017-era, context-blind, and our own numbers cap intended-top-5 near 0.21
6
+ on hard English typos even with buckets. The eval harness (keyboard-aware
7
+ grids, typo corpora, frozen gates) is the asset; use it to test successors.
8
+
9
+ ## Work (models repo; eval only — no registry release without a verdict)
10
+ 1. Baseline: current mini/fluency+buckets through the existing gates.
11
+ 2. Candidate B: fastText + small cross-encoder rerank over (context,
12
+ candidate) pairs — ONNX-exportable distill class, 15-30 MB int8.
13
+ 3. Candidate C: purpose-trained typo bi-encoder over the vendored
14
+ github-typo-corpus (train/eval split strictly by repo).
15
+ 4. Candidate D: ModernBERT-class small encoder, same size budget.
16
+ 5. Report: gates table per candidate per language (en de ru es minimum),
17
+ size, latency-per-suggest, and a ship/reject verdict with the same
18
+ discipline as int4 (reject on data is a complete outcome).
19
+
20
+ ## Verification
21
+ Every number from the existing harness, unweakened; frozen corpora; no
22
+ registry changes in this plan.
23
+
24
+ ## Status
25
+ Executed 2026-09-09 — models PR #25 merged. VERDICT: fastText stays.
26
+ B (MiniLM cross-encoder rerank) REJECT — loses 4/4 languages (en 0.194 vs
27
+ fluency 0.252 top-5; slate recall caps en at 0.43; context ablation
28
+ negative). D (ModernBERT-base) REJECT — representation failure (top-5
29
+ 0.03-0.24), 5x size cap. C (0.5 MB char-BiGRU typo bi-encoder, 5.2
30
+ ms/suggest) REJECT as drop-in but PROMISING — beats the full tier on the
31
+ strict repo-clean subset for en and de; open thread needing non-en data,
32
+ a bigger clean bench, and a top-1 story. Full ladders in
33
+ eval/reports/bakeoff-v1.md.
@@ -0,0 +1,26 @@
1
+ # Plan 112 — Deployability and CI gating
2
+
3
+ ## Work (kotoshu-rs; site rides a follow-up)
4
+ 1. Prebuilt worker on npm: @kotoshu/worker exposing the playground's
5
+ engine worker (load/check/suggest-batch/semantic/detect protocol) so
6
+ JS users embed the engine without reimplementing the worker; versioned
7
+ with the wasm package.
8
+ 2. CI latency gate: the multi-language sweep bench (rs PR #23 harness)
9
+ runs on CI with budgets (en < 120 ms, pt < 700 ms avg) failing red.
10
+ 3. Wasm memory ceiling test: dict + mini tier + buckets resident under a
11
+ stated budget per language class.
12
+ 4. Language pack (design note, owner decision): one fetch for
13
+ dict+tier+buckets — additive registry field sketch in the PR body.
14
+
15
+ ## Verification
16
+ Node smoke for the worker package; budgets tuned to measured p50s, not
17
+ aspirations; all existing CI green.
18
+
19
+ ## Status
20
+ Executed 2026-09-09 — kotoshu-rs PR #28 merged. @kotoshu/worker package
21
+ (createEngine full protocol, ./engine-worker entry, semantic-merge pure
22
+ half; publish pending owner npm registration); latency.yml gate with
23
+ measured budgets (en 56ms avg vs 120 gate, pt 346 vs 700 — 2x headroom);
24
+ wasm32 memory ceilings frozen (en 46.9/64 MB, pt 165/192 MB) with
25
+ behavior probes; language-pack kotoshu://packs/{lang} sketch in the PR
26
+ body awaiting owner decision.
@@ -0,0 +1,26 @@
1
+ # Plan 113 — Language packs: one fetch for a whole language
2
+
3
+ ## Why
4
+ Loading a language today is 3+ round trips (dict aff+dic, tier, buckets,
5
+ sometimes registry) - the rs PR #28 sketch showed a single length-prefixed
6
+ pack is simpler for SDK users and cuts playground cold loads.
7
+
8
+ ## Work
9
+ 1. models repo: pack builder (dict sources + tier + buckets concatenated,
10
+ length-prefixed sections; kotoshu://packs/{lang} additive registry
11
+ entries, versioned with their parts); v1.6.0 when validated.
12
+ 2. kotoshu-rs: wasm loadPack(bytes) -> {dictionary, model} handle; worker
13
+ package gains pack mode; keep per-artifact loading as fallback.
14
+ 3. Site: playground opt-in pack path with one progress bar.
15
+
16
+ ## Verification
17
+ Pack bytes == concatenation of parts (checksums per section); parity
18
+ specs per language; playground CDP cold-load comparison (requests and ms).
19
+
20
+ ## Status
21
+ Executed 2026-09-09 — models PR #26 + kotoshu-rs PR #32, both merged.
22
+ KPK1 format (length+tag framed sections, sha256 footers, golden-sha parity
23
+ across implementations); en/de/pt packs LFS-committed with live mirrors;
24
+ additive kotoshu://packs/{lang} entries; wasm loadPack + worker pack mode
25
+ with per-artifact fallback. Registry v1.6.0 cut awaiting owner; site
26
+ playground flip optional (worker package already serves pack mode).
@@ -0,0 +1,33 @@
1
+ # Plan 114 — The typo bi-encoder thread (bake-off follow-up)
2
+
3
+ ## Why
4
+ Bake-off candidate C (0.5 MB char-BiGRU, 5.2 ms/suggest) beat the FULL
5
+ tier on repo-clean en/de pairs - the only successor that did. It needs:
6
+ non-en training data, a bigger clean bench, and a top-1 story.
7
+
8
+ ## Work (models repo)
9
+ 1. Extend training data beyond en (de ru es typo corpora; synth via the
10
+ existing generator where corpora are thin).
11
+ 2. Grow the repo-clean bench (target 2000+ pairs/lang); freeze as the
12
+ C-benchmark.
13
+ 3. Train v2; measure against fastText tiers on the same bench; ship as
14
+ an OPT-IN registry resource only if it wins en+de clearly and degrades
15
+ nowhere - reject honestly otherwise.
16
+
17
+ ## Status
18
+ Executed 2026-09-09 — models PR #27 merged. VERDICT: REJECT, honest on
19
+ data; fastText tiers stay, no registry change, no tag. Frozen C-benchmark:
20
+ en 2509 real repo-clean pairs (target 2000+ met with real data only),
21
+ de/ru/es real clean pairs are 14/27/20 — the corpus holds only 212/551/186
22
+ unique pairs for those languages — plus a labeled 2000-pair noise.py synth
23
+ component per language, disjoint from all training sources. C v2 (same
24
+ 0.481 MB architecture, training data the one changed variable) beats the
25
+ full tier on 2509 real unseen en pairs (+7.5 pp top-5, CI +5.9 to +9.1)
26
+ and fixed the top-1 gap (0.128 vs 0.103), but fails the pre-declared rule:
27
+ de real clears no paired CI at n=14, and es real regresses 10 pp top-5 at
28
+ n=20 (three independent es slices agree). Synth dominance (+40 to +48 pp)
29
+ is generator-domain and does not transfer to real es. Surviving thread for
30
+ a future plan: the C-retrieve + fastText-rescore hybrid beats the full
31
+ tier top-1 and top-5 on all four real components, but needs the
32
+ per-language vocab matrix cost priced and a real es/de typo corpus.
33
+ Full ladder: models eval/reports/biencoder-v2.md.
@@ -0,0 +1,22 @@
1
+ # Plan 115 — The hybrid retrieval thread: price it before believing it
2
+
3
+ ## Why
4
+ Plan 114's reject left one survivor on record: C-retrieve (0.5 MB bi-encoder)
5
+ + fastText-rescore beat the FULL tier on top-1 AND top-5 across all four
6
+ real bench components. Before any ship decision, its costs and its
7
+ dependence on synthetic-heavy slices must be priced honestly.
8
+
9
+ ## Work (models repo)
10
+ 1. Price the retrieval half: per-language 100k x 256 vocab matrix (fp16 +
11
+ int8 variants) — build for en, measure size, build time, top-k
12
+ retrieval latency, and quality vs the brute-force sweep on the frozen
13
+ C-benchmark.
14
+ 2. Quality stress: score the hybrid ONLY on real-pair components
15
+ (en 2509; de/ru/es real n are tiny — report with CIs, no synth pooling
16
+ for the verdict).
17
+ 3. Verdict per the plan-114 rule: ship as opt-in registry resource only
18
+ on clear wins with no regression; honest reject otherwise. No registry
19
+ changes on reject.
20
+
21
+ ## Status
22
+ Pending
@@ -0,0 +1,81 @@
1
+ # Plan 116 — Baseline-aware suggestion skip in `kotoshu check`
2
+
3
+ Status: executed (gem PR #175, 2026-09-10)
4
+
5
+ Measured on this repo's own dogfood gate (prose baseline, 640 entries /
6
+ 1,905 covered occurrences): check wall time 13m+ -> 13.7s, identical
7
+ output. Full default suite 4003/0.
8
+
9
+ Depends on: plan 82 (baselines), plan 113 (packs — unrelated but same file surfaces)
10
+ Blocks: nothing; targets the 0.11.x line before 1.0
11
+
12
+ ## Problem
13
+
14
+ `kotoshu check --baseline` in CI regenerates the suggestion sweep for
15
+ every error it is about to suppress. On the kotoshu repo itself the
16
+ prose baseline holds 1,905 occurrences across 7 files; the check phase
17
+ of the dogfood gate spends ~10 CPU-minutes producing suggestions that
18
+ the baseline then throws away. The site repo's spellcheck job runs
19
+ 6m48s for the same reason. Suggestions only matter for errors that
20
+ SURVIVE the baseline — those are the rows a human reads.
21
+
22
+ `kotoshu baseline init` already fixed this for itself
23
+ (`Spellchecker#check(text, suggestions: false)`, plan-executed in
24
+ "fix(cli): make baseline init usable on real repositories"). `check`
25
+ still pays full price.
26
+
27
+ ## Design
28
+
29
+ The budget is knowable before checking: the baseline entry list for the
30
+ file under check says exactly which words will be absorbed (and how
31
+ many times). So:
32
+
33
+ 1. `Cli::DirectoryCheck` (and the single-file check path) computes
34
+ `covered = baseline.entries_for(path)` → a word→count budget hash,
35
+ before invoking the spellcheck callback.
36
+ 2. `Spellchecker#check` gains an optional `suggestions_filter:` hook —
37
+ a callable receiving the word, returning whether to generate
38
+ suggestions. Default nil = generate for everything (current
39
+ behavior). The loop already has the `suggestions:` branch; this
40
+ generalizes it without touching result shape.
41
+ - Occurrence counting must be honored: the budget for `wrold: 3`
42
+ means the first 3 occurrences skip, the 4th (a NEW error)
43
+ generates. The filter therefore closes over a mutable budget
44
+ hash in the caller, decremented per consumed occurrence.
45
+ 3. Errors suppressed by the baseline keep their (empty) suggestion set
46
+ — unchanged from today, since suppressed rows render without
47
+ suggestions anyway.
48
+ 4. New errors — beyond budget or unknown words — generate suggestions
49
+ exactly as today. SARIF, text, and JSON output shapes unchanged.
50
+ 5. Inline-suppressed words: keep generating suggestions today (they
51
+ are dropped at assembly regardless) — fold into the same skip by
52
+ treating `kotoshu:disable` regions as covered when the suppression
53
+ scan already knows the word/line. Out of scope if the scan order
54
+ makes this awkward; the baseline case is the measured cost.
55
+
56
+ ## Surfaces touched
57
+
58
+ - `lib/kotoshu/spellchecker.rb` — `check(text, suggestions:,
59
+ suggestions_filter:)`
60
+ - `lib/kotoshu/cli/directory_check.rb` — budget construction from the
61
+ loaded baseline, filter closure
62
+ - `lib/kotoshu/cli.rb` — single-file `--baseline` path, same wiring
63
+ - `spec/kotoshu/cli/baseline_cli_spec.rb` — new cases: covered word
64
+ beyond budget generates suggestions; covered occurrence does not;
65
+ no baseline = unchanged
66
+
67
+ ## Non-goals
68
+
69
+ - No output format changes; suppressed rows stay suggestion-less.
70
+ - No change to `Baseline::Store` semantics (apply stays the authority
71
+ on pass/fail); the filter is purely a compute optimization whose
72
+ visible effect is empty suggestions on rows that were going to be
73
+ suppressed.
74
+
75
+ ## Acceptance
76
+
77
+ - Dogfood gate on the kotoshu repo: check phase wall time drops from
78
+ ~10 min to under ~2 min (the sweep itself, minus 1,905 suggestion
79
+ generations).
80
+ - All existing baseline specs green; new filter specs green.
81
+ - Site repo spellcheck job measurably faster after its gem bump.