kotoshu 1.0.2-aarch64-linux

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (458) hide show
  1. checksums.yaml +7 -0
  2. data/.kotoshu-baseline.json +1 -0
  3. data/.pre-commit-hooks.yaml +18 -0
  4. data/.rspec +3 -0
  5. data/.rubocop.yml +351 -0
  6. data/.yardopts +8 -0
  7. data/CHANGELOG.md +813 -0
  8. data/CLAUDE.md +176 -0
  9. data/CODE_OF_CONDUCT.md +132 -0
  10. data/Cargo.lock +359 -0
  11. data/Cargo.toml +15 -0
  12. data/LICENSE +31 -0
  13. data/README.adoc +1292 -0
  14. data/Rakefile +150 -0
  15. data/SECURITY.md +93 -0
  16. data/TODO.impl/00-cut-0.1.md +121 -0
  17. data/TODO.impl/00-cut-0.2.md +175 -0
  18. data/TODO.impl/00-vision.md +281 -0
  19. data/TODO.impl/01-hunspell-correctness.md +111 -0
  20. data/TODO.impl/02-cli-unification.md +74 -0
  21. data/TODO.impl/03-dynamic-download.md +107 -0
  22. data/TODO.impl/04-language-modules.md +89 -0
  23. data/TODO.impl/05-semantic-path.md +83 -0
  24. data/TODO.impl/06-cjk-support.md +105 -0
  25. data/TODO.impl/07-rtl-support.md +71 -0
  26. data/TODO.impl/08-grammar-engine.md +106 -0
  27. data/TODO.impl/09-integrity-security.md +90 -0
  28. data/TODO.impl/10-testing-ci.md +98 -0
  29. data/TODO.impl/100-full-feature-batch3.md +28 -0
  30. data/TODO.impl/101-int4-tier-experiment.md +32 -0
  31. data/TODO.impl/102-lid-chain.md +28 -0
  32. data/TODO.impl/103-oov-bucket-tables.md +27 -0
  33. data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
  34. data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
  35. data/TODO.impl/106-native-lid-detection.md +36 -0
  36. data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
  37. data/TODO.impl/108-ko-ne-full-feature.md +37 -0
  38. data/TODO.impl/109-ten-audit.md +26 -0
  39. data/TODO.impl/11-release-v1.md +95 -0
  40. data/TODO.impl/110-sr-latn-and-polish.md +33 -0
  41. data/TODO.impl/111-embedding-bakeoff.md +33 -0
  42. data/TODO.impl/112-deployability-and-gating.md +26 -0
  43. data/TODO.impl/113-language-packs.md +26 -0
  44. data/TODO.impl/114-typo-biencoder-thread.md +33 -0
  45. data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
  46. data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
  47. data/TODO.impl/117-deterministic-frequency-tiers.md +110 -0
  48. data/TODO.impl/118-server-gem-floor.md +33 -0
  49. data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
  50. data/TODO.impl/120-setup-metadata-refresh.md +35 -0
  51. data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
  52. data/TODO.impl/122-pack-compression-kpk2.md +66 -0
  53. data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
  54. data/TODO.impl/124-owner-residue.md +19 -0
  55. data/TODO.impl/125-consumer-floor-guards.md +43 -0
  56. data/TODO.impl/126-tier-error-budgets.md +33 -0
  57. data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
  58. data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
  59. data/TODO.impl/129-pt-fr-corpora.md +23 -0
  60. data/TODO.impl/130-news-truth-0912.md +23 -0
  61. data/TODO.impl/131-hybrid-productization.md +42 -0
  62. data/TODO.impl/132-techniques-page-and-prose.md +27 -0
  63. data/TODO.impl/133-rust-first-distribution.md +68 -0
  64. data/TODO.impl/20-encapsulation-cleanup.md +47 -0
  65. data/TODO.impl/21-autoload-migration.md +64 -0
  66. data/TODO.impl/22-url-pin-single-source-of-truth.md +120 -0
  67. data/TODO.impl/23-fix-url-defaults.md +57 -0
  68. data/TODO.impl/24-cli-auto-setup-prompt.md +76 -0
  69. data/TODO.impl/25-soft-onnxruntime-require.md +75 -0
  70. data/TODO.impl/26-onnx-vocab-json-generation.md +96 -0
  71. data/TODO.impl/27-end-to-end-smoke-spec.md +113 -0
  72. data/TODO.impl/28-readme-quickstart-verification.md +68 -0
  73. data/TODO.impl/29-status-command.md +84 -0
  74. data/TODO.impl/30-language-auto-detection.md +74 -0
  75. data/TODO.impl/31-download-progress.md +58 -0
  76. data/TODO.impl/32-sarif-output-verification.md +72 -0
  77. data/TODO.impl/33-audit-log-rotation.md +54 -0
  78. data/TODO.impl/34-cache-eviction.md +59 -0
  79. data/TODO.impl/35-shell-completion.md +61 -0
  80. data/TODO.impl/36-cleanup-regressions-0.3.1.md +134 -0
  81. data/TODO.impl/37-hunspell-correctness-tier2.md +203 -0
  82. data/TODO.impl/38-onnx-semantic-gating.md +218 -0
  83. data/TODO.impl/39-tier3-and-beyond.md +232 -0
  84. data/TODO.impl/40-spec-drift-cleanup.md +134 -0
  85. data/TODO.impl/50-structure-aware-document-api.md +194 -0
  86. data/TODO.impl/51-grammar-engine-expansion.md +75 -0
  87. data/TODO.impl/52-multi-language-documents.md +95 -0
  88. data/TODO.impl/53-document-plugin-architecture.md +94 -0
  89. data/TODO.impl/54-cjk-support.md +117 -0
  90. data/TODO.impl/55-rtl-support.md +109 -0
  91. data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
  92. data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
  93. data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
  94. data/TODO.impl/59-yard-api-documentation.md +75 -0
  95. data/TODO.impl/60-lsp-server.md +174 -0
  96. data/TODO.impl/61-editor-ecosystem.md +167 -0
  97. data/TODO.impl/62-framework-integrations.md +159 -0
  98. data/TODO.impl/63-web-playground-wasm.md +186 -0
  99. data/TODO.impl/64-http-api-and-sdks.md +191 -0
  100. data/TODO.impl/65-universal-kotoshu.md +251 -0
  101. data/TODO.impl/66-kotoshu-core.md +169 -0
  102. data/TODO.impl/67-kotoshu-rs-and-access-libraries.md +137 -0
  103. data/TODO.impl/68-sota-adoption.md +152 -0
  104. data/TODO.impl/69-sota-typo-eval.md +74 -0
  105. data/TODO.impl/70-sota-confidence-cascade.md +59 -0
  106. data/TODO.impl/71-sota-nested-model.md +63 -0
  107. data/TODO.impl/72-model2vec-evaluation.md +49 -0
  108. data/TODO.impl/73-rl-reranker.md +42 -0
  109. data/TODO.impl/74-distilled-reranker.md +36 -0
  110. data/TODO.impl/75-llm-evaluator.md +35 -0
  111. data/TODO.impl/76-optional-llm-backend.md +37 -0
  112. data/TODO.impl/77-model-coverage-expansion.md +96 -0
  113. data/TODO.impl/78-in-browser-playground.md +69 -0
  114. data/TODO.impl/79-news-and-releases.md +60 -0
  115. data/TODO.impl/80-ecosystem-docs.md +56 -0
  116. data/TODO.impl/81-editor-reach.md +68 -0
  117. data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
  118. data/TODO.impl/83-models-batch-2.md +56 -0
  119. data/TODO.impl/84-language-modules-wave2.md +75 -0
  120. data/TODO.impl/85-wasm-model-api.md +48 -0
  121. data/TODO.impl/86-site-truth-wave2.md +45 -0
  122. data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
  123. data/TODO.impl/88-cli-directory-mode.md +54 -0
  124. data/TODO.impl/89-framework-integrations-exec.md +59 -0
  125. data/TODO.impl/90-gem-correctness-residues.md +44 -0
  126. data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
  127. data/TODO.impl/92-tier-mirror-cors.md +88 -0
  128. data/TODO.impl/93-python-wheels.md +32 -0
  129. data/TODO.impl/94-site-integrations-docs.md +35 -0
  130. data/TODO.impl/95-editor-action-polish.md +27 -0
  131. data/TODO.impl/96-server-semantic-models.md +50 -0
  132. data/TODO.impl/97-yard-execution.md +73 -0
  133. data/TODO.impl/98-campaign-cleanup.md +78 -0
  134. data/TODO.impl/99-site-truth-performance-docs.md +31 -0
  135. data/conformance/vectors.jsonl +2630 -0
  136. data/data/en/grammar/rules.yaml +851 -0
  137. data/docs/STABILITY-1.0-AUDIT.md +251 -0
  138. data/docs/STABILITY-1.0.md +137 -0
  139. data/docs/cache-architecture.md +502 -0
  140. data/docs/verification-evidence/correct3/pr_body.md +37 -0
  141. data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
  142. data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
  143. data/docs/verification-evidence/correct3/sv.aff +856 -0
  144. data/docs/verification-evidence/correct3/sv.dic +153717 -0
  145. data/examples/01_basic_word_checking.rb +38 -0
  146. data/examples/02_text_document_checking.rb +77 -0
  147. data/examples/03_dictionary_backends.rb +137 -0
  148. data/examples/04_trie_data_structure.rb +146 -0
  149. data/examples/05_suggestion_algorithms.rb +239 -0
  150. data/examples/06_configuration_advanced.rb +287 -0
  151. data/examples/07_multi_language_dictionaries.rb +278 -0
  152. data/exe/kotoshu +6 -0
  153. data/lib/kotoshu/algorithms/capitalization.rb +276 -0
  154. data/lib/kotoshu/algorithms/edit_distance.rb +128 -0
  155. data/lib/kotoshu/algorithms/lookup/affix_form.rb +132 -0
  156. data/lib/kotoshu/algorithms/lookup/compound_checks.rb +209 -0
  157. data/lib/kotoshu/algorithms/lookup/compound_form.rb +50 -0
  158. data/lib/kotoshu/algorithms/lookup.rb +841 -0
  159. data/lib/kotoshu/algorithms/ngram_suggest.rb +367 -0
  160. data/lib/kotoshu/algorithms/permutations.rb +283 -0
  161. data/lib/kotoshu/algorithms/phonet_suggest.rb +162 -0
  162. data/lib/kotoshu/algorithms/soundex.rb +92 -0
  163. data/lib/kotoshu/algorithms/suggest.rb +794 -0
  164. data/lib/kotoshu/algorithms.rb +23 -0
  165. data/lib/kotoshu/analyzers/semantic_analyzer.rb +405 -0
  166. data/lib/kotoshu/analyzers.rb +8 -0
  167. data/lib/kotoshu/baseline/application.rb +18 -0
  168. data/lib/kotoshu/baseline/entry.rb +33 -0
  169. data/lib/kotoshu/baseline/store.rb +175 -0
  170. data/lib/kotoshu/baseline.rb +20 -0
  171. data/lib/kotoshu/cache/base_cache.rb +778 -0
  172. data/lib/kotoshu/cache/cache.rb +91 -0
  173. data/lib/kotoshu/cache/eviction_policy.rb +66 -0
  174. data/lib/kotoshu/cache/frequency_cache.rb +283 -0
  175. data/lib/kotoshu/cache/language_cache.rb +511 -0
  176. data/lib/kotoshu/cache/lookup_cache.rb +166 -0
  177. data/lib/kotoshu/cache/model_cache.rb +1147 -0
  178. data/lib/kotoshu/cache/model_registry.rb +109 -0
  179. data/lib/kotoshu/cache/staged_languages.rb +45 -0
  180. data/lib/kotoshu/cache/suggestion_cache.rb +113 -0
  181. data/lib/kotoshu/cache.rb +49 -0
  182. data/lib/kotoshu/cli/auto_setup.rb +71 -0
  183. data/lib/kotoshu/cli/baseline_command.rb +119 -0
  184. data/lib/kotoshu/cli/batch_reporter.rb +321 -0
  185. data/lib/kotoshu/cli/cache_command.rb +282 -0
  186. data/lib/kotoshu/cli/completions_command.rb +210 -0
  187. data/lib/kotoshu/cli/dict_command.rb +64 -0
  188. data/lib/kotoshu/cli/directory_check.rb +146 -0
  189. data/lib/kotoshu/cli/directory_walker.rb +132 -0
  190. data/lib/kotoshu/cli/display_formatter.rb +429 -0
  191. data/lib/kotoshu/cli/errors.rb +36 -0
  192. data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
  193. data/lib/kotoshu/cli/interactive_reviewer.rb +314 -0
  194. data/lib/kotoshu/cli/language_resolver.rb +91 -0
  195. data/lib/kotoshu/cli/navigation_manager.rb +273 -0
  196. data/lib/kotoshu/cli/personal_command.rb +113 -0
  197. data/lib/kotoshu/cli/progress_reporter.rb +114 -0
  198. data/lib/kotoshu/cli/status_report.rb +130 -0
  199. data/lib/kotoshu/cli.rb +880 -0
  200. data/lib/kotoshu/components/passthrough_spell_checker.rb +70 -0
  201. data/lib/kotoshu/components/pos_tagger.rb +98 -0
  202. data/lib/kotoshu/components/spell_checker.rb +73 -0
  203. data/lib/kotoshu/components/synthesizer.rb +60 -0
  204. data/lib/kotoshu/components/tokenizer.rb +58 -0
  205. data/lib/kotoshu/components/whitespace_tokenizer.rb +94 -0
  206. data/lib/kotoshu/components.rb +13 -0
  207. data/lib/kotoshu/configuration/builder.rb +207 -0
  208. data/lib/kotoshu/configuration/resolver.rb +124 -0
  209. data/lib/kotoshu/configuration.rb +829 -0
  210. data/lib/kotoshu/conformance_exporter.rb +256 -0
  211. data/lib/kotoshu/conformance_runner.rb +176 -0
  212. data/lib/kotoshu/core/exceptions.rb +186 -0
  213. data/lib/kotoshu/core/indexed_dictionary.rb +293 -0
  214. data/lib/kotoshu/core/models/affix_rule.rb +262 -0
  215. data/lib/kotoshu/core/models/result/document_result.rb +132 -0
  216. data/lib/kotoshu/core/models/result/word_result.rb +131 -0
  217. data/lib/kotoshu/core/models/word.rb +131 -0
  218. data/lib/kotoshu/core/trie/builder.rb +119 -0
  219. data/lib/kotoshu/core/trie/node.rb +94 -0
  220. data/lib/kotoshu/core/trie/trie.rb +249 -0
  221. data/lib/kotoshu/core.rb +28 -0
  222. data/lib/kotoshu/data/common_words/de.yml +1800 -0
  223. data/lib/kotoshu/data/common_words/en.yml +1215 -0
  224. data/lib/kotoshu/data/common_words/es.yml +750 -0
  225. data/lib/kotoshu/data/common_words/fr.yml +1015 -0
  226. data/lib/kotoshu/data/common_words/pt.yml +870 -0
  227. data/lib/kotoshu/data/common_words/ru.yml +484 -0
  228. data/lib/kotoshu/data/common_words_loader.rb +141 -0
  229. data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
  230. data/lib/kotoshu/data/frozen_kelly.rb +38 -0
  231. data/lib/kotoshu/data.rb +8 -0
  232. data/lib/kotoshu/data_structures/bloom_filter.rb +188 -0
  233. data/lib/kotoshu/data_structures.rb +8 -0
  234. data/lib/kotoshu/debug_logger.rb +146 -0
  235. data/lib/kotoshu/debug_mode.rb +136 -0
  236. data/lib/kotoshu/defaults.rb +84 -0
  237. data/lib/kotoshu/dictionaries/catalog.rb +814 -0
  238. data/lib/kotoshu/dictionaries.rb +8 -0
  239. data/lib/kotoshu/dictionary/base.rb +324 -0
  240. data/lib/kotoshu/dictionary/cspell.rb +243 -0
  241. data/lib/kotoshu/dictionary/custom.rb +225 -0
  242. data/lib/kotoshu/dictionary/hunspell.rb +520 -0
  243. data/lib/kotoshu/dictionary/plain_text.rb +302 -0
  244. data/lib/kotoshu/dictionary/repository.rb +246 -0
  245. data/lib/kotoshu/dictionary/unified.rb +236 -0
  246. data/lib/kotoshu/dictionary/unix_words.rb +218 -0
  247. data/lib/kotoshu/dictionary.rb +15 -0
  248. data/lib/kotoshu/documents/document.rb +120 -0
  249. data/lib/kotoshu/documents/plain_text_document.rb +73 -0
  250. data/lib/kotoshu/documents/source_position.rb +45 -0
  251. data/lib/kotoshu/documents/source_range.rb +76 -0
  252. data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
  253. data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
  254. data/lib/kotoshu/documents/suppressions.rb +44 -0
  255. data/lib/kotoshu/documents/text_node.rb +49 -0
  256. data/lib/kotoshu/documents.rb +178 -0
  257. data/lib/kotoshu/embeddings/embedding_pipeline.rb +238 -0
  258. data/lib/kotoshu/embeddings/lru_cache.rb +244 -0
  259. data/lib/kotoshu/embeddings/onnx_runtime_model.rb +411 -0
  260. data/lib/kotoshu/embeddings/protocol.rb +96 -0
  261. data/lib/kotoshu/embeddings/protocols.rb +23 -0
  262. data/lib/kotoshu/embeddings/registry.rb +184 -0
  263. data/lib/kotoshu/embeddings/search.rb +204 -0
  264. data/lib/kotoshu/embeddings/similarity_engine.rb +278 -0
  265. data/lib/kotoshu/embeddings/similarity_search.rb +328 -0
  266. data/lib/kotoshu/embeddings/vocabulary.rb +306 -0
  267. data/lib/kotoshu/embeddings.rb +87 -0
  268. data/lib/kotoshu/fluent_checker.rb +94 -0
  269. data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +48 -0
  270. data/lib/kotoshu/grammar/pattern_matchers/double_negative_matcher.rb +179 -0
  271. data/lib/kotoshu/grammar/pattern_matchers/phrase_matcher.rb +86 -0
  272. data/lib/kotoshu/grammar/pattern_matchers/possessive_context_matcher.rb +75 -0
  273. data/lib/kotoshu/grammar/pattern_matchers/possessive_contraction_matcher.rb +87 -0
  274. data/lib/kotoshu/grammar/pattern_matchers/sentence_start_matcher.rb +84 -0
  275. data/lib/kotoshu/grammar/pattern_matchers/vowel_sound_matcher.rb +81 -0
  276. data/lib/kotoshu/grammar/pattern_matchers/word_list_matcher.rb +66 -0
  277. data/lib/kotoshu/grammar/rule.rb +99 -0
  278. data/lib/kotoshu/grammar/rule_engine.rb +96 -0
  279. data/lib/kotoshu/grammar/rule_loader.rb +30 -0
  280. data/lib/kotoshu/grammar.rb +24 -0
  281. data/lib/kotoshu/integrity/audit_log.rb +167 -0
  282. data/lib/kotoshu/integrity/manifest.rb +116 -0
  283. data/lib/kotoshu/integrity/net_http.rb +51 -0
  284. data/lib/kotoshu/integrity/rotation_policy.rb +88 -0
  285. data/lib/kotoshu/integrity.rb +26 -0
  286. data/lib/kotoshu/jekyll.rb +101 -0
  287. data/lib/kotoshu/keyboard/layout.rb +116 -0
  288. data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
  289. data/lib/kotoshu/keyboard/layouts/azerty.rb +55 -0
  290. data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
  291. data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
  292. data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
  293. data/lib/kotoshu/keyboard/layouts/dvorak.rb +54 -0
  294. data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
  295. data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
  296. data/lib/kotoshu/keyboard/layouts/hrsl.rb +94 -0
  297. data/lib/kotoshu/keyboard/layouts/jcuken.rb +57 -0
  298. data/lib/kotoshu/keyboard/layouts/latin.rb +266 -0
  299. data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
  300. data/lib/kotoshu/keyboard/layouts/qwerty.rb +52 -0
  301. data/lib/kotoshu/keyboard/layouts/qwertz.rb +55 -0
  302. data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
  303. data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
  304. data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
  305. data/lib/kotoshu/keyboard/registry.rb +201 -0
  306. data/lib/kotoshu/keyboard.rb +96 -0
  307. data/lib/kotoshu/kotoshu_native.so +0 -0
  308. data/lib/kotoshu/language/detector.rb +242 -0
  309. data/lib/kotoshu/language/identifier.rb +376 -0
  310. data/lib/kotoshu/language/languages/base.rb +271 -0
  311. data/lib/kotoshu/language/lid_detector.rb +188 -0
  312. data/lib/kotoshu/language/normalizer/arabic.rb +113 -0
  313. data/lib/kotoshu/language/normalizer/base.rb +133 -0
  314. data/lib/kotoshu/language/normalizer/greek.rb +63 -0
  315. data/lib/kotoshu/language/normalizer/hebrew.rb +72 -0
  316. data/lib/kotoshu/language/normalizer/persian.rb +36 -0
  317. data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
  318. data/lib/kotoshu/language/registry.rb +209 -0
  319. data/lib/kotoshu/language/resources/ar/common_words.txt +6753 -0
  320. data/lib/kotoshu/language/resources/ar/confusion_sets.txt +11 -0
  321. data/lib/kotoshu/language/resources/de/common_words.txt +10003 -0
  322. data/lib/kotoshu/language/resources/de/confusion_sets.txt +246 -0
  323. data/lib/kotoshu/language/resources/en/common_words.txt +9979 -0
  324. data/lib/kotoshu/language/resources/en/confusion_sets.txt +871 -0
  325. data/lib/kotoshu/language/resources/es/common_words.txt +9992 -0
  326. data/lib/kotoshu/language/resources/es/confusion_sets.txt +17 -0
  327. data/lib/kotoshu/language/resources/fr/common_words.txt +9993 -0
  328. data/lib/kotoshu/language/resources/fr/confusion_sets.txt +76 -0
  329. data/lib/kotoshu/language/resources/pt/common_words.txt +9977 -0
  330. data/lib/kotoshu/language/resources/pt/confusion_sets.txt +18 -0
  331. data/lib/kotoshu/language/resources/ru/common_words.txt +9951 -0
  332. data/lib/kotoshu/language/resources/ru/confusion_sets.txt +5 -0
  333. data/lib/kotoshu/language/script.rb +128 -0
  334. data/lib/kotoshu/language/segmenter.rb +122 -0
  335. data/lib/kotoshu/language/suika.rb +38 -0
  336. data/lib/kotoshu/language/tokenizer/base.rb +187 -0
  337. data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
  338. data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
  339. data/lib/kotoshu/language/tokenizer/french_tokenizer.rb +172 -0
  340. data/lib/kotoshu/language/tokenizer/german_tokenizer.rb +41 -0
  341. data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
  342. data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
  343. data/lib/kotoshu/language/tokenizer/japanese_tokenizer.rb +55 -0
  344. data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +153 -0
  345. data/lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb +160 -0
  346. data/lib/kotoshu/language/tokenizer/russian_tokenizer.rb +93 -0
  347. data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
  348. data/lib/kotoshu/language/tokenizer/spanish_tokenizer.rb +122 -0
  349. data/lib/kotoshu/language.rb +127 -0
  350. data/lib/kotoshu/languages/ar/language.rb +98 -0
  351. data/lib/kotoshu/languages/bg/language.rb +37 -0
  352. data/lib/kotoshu/languages/ca/language.rb +22 -0
  353. data/lib/kotoshu/languages/cs/language.rb +22 -0
  354. data/lib/kotoshu/languages/da/language.rb +22 -0
  355. data/lib/kotoshu/languages/de/language.rb +554 -0
  356. data/lib/kotoshu/languages/el/language.rb +50 -0
  357. data/lib/kotoshu/languages/en/language.rb +470 -0
  358. data/lib/kotoshu/languages/es/language.rb +476 -0
  359. data/lib/kotoshu/languages/et/language.rb +24 -0
  360. data/lib/kotoshu/languages/fa/language.rb +91 -0
  361. data/lib/kotoshu/languages/fr/language.rb +505 -0
  362. data/lib/kotoshu/languages/he/language.rb +98 -0
  363. data/lib/kotoshu/languages/hr/language.rb +24 -0
  364. data/lib/kotoshu/languages/hu/language.rb +22 -0
  365. data/lib/kotoshu/languages/id/language.rb +25 -0
  366. data/lib/kotoshu/languages/it/language.rb +22 -0
  367. data/lib/kotoshu/languages/ja/language.rb +476 -0
  368. data/lib/kotoshu/languages/ko/language.rb +56 -0
  369. data/lib/kotoshu/languages/latin_base.rb +53 -0
  370. data/lib/kotoshu/languages/lt/language.rb +24 -0
  371. data/lib/kotoshu/languages/lv/language.rb +24 -0
  372. data/lib/kotoshu/languages/nb/language.rb +25 -0
  373. data/lib/kotoshu/languages/ne/language.rb +60 -0
  374. data/lib/kotoshu/languages/nl/language.rb +22 -0
  375. data/lib/kotoshu/languages/nn/language.rb +26 -0
  376. data/lib/kotoshu/languages/pl/language.rb +22 -0
  377. data/lib/kotoshu/languages/pt/language.rb +439 -0
  378. data/lib/kotoshu/languages/ro/language.rb +22 -0
  379. data/lib/kotoshu/languages/ru/language.rb +420 -0
  380. data/lib/kotoshu/languages/sk/language.rb +25 -0
  381. data/lib/kotoshu/languages/sl/language.rb +25 -0
  382. data/lib/kotoshu/languages/sr/language.rb +39 -0
  383. data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
  384. data/lib/kotoshu/languages/sv/language.rb +22 -0
  385. data/lib/kotoshu/languages/tr/language.rb +36 -0
  386. data/lib/kotoshu/languages/uk/language.rb +38 -0
  387. data/lib/kotoshu/languages/vi/language.rb +22 -0
  388. data/lib/kotoshu/languages.rb +64 -0
  389. data/lib/kotoshu/metrics_collector.rb +238 -0
  390. data/lib/kotoshu/metrics_module.rb +121 -0
  391. data/lib/kotoshu/models/context.rb +119 -0
  392. data/lib/kotoshu/models/embedding_model.rb +181 -0
  393. data/lib/kotoshu/models/fasttext_model.rb +212 -0
  394. data/lib/kotoshu/models/nearest_neighbor.rb +96 -0
  395. data/lib/kotoshu/models/onnx_model.rb +330 -0
  396. data/lib/kotoshu/models/semantic_error.rb +204 -0
  397. data/lib/kotoshu/models/suggestion.rb +106 -0
  398. data/lib/kotoshu/models/word_embedding.rb +97 -0
  399. data/lib/kotoshu/models.rb +27 -0
  400. data/lib/kotoshu/multi_language_checker.rb +127 -0
  401. data/lib/kotoshu/native.rb +83 -0
  402. data/lib/kotoshu/native_backend.rb +131 -0
  403. data/lib/kotoshu/paths.rb +79 -0
  404. data/lib/kotoshu/personal_dictionary.rb +105 -0
  405. data/lib/kotoshu/plugins/plugin.rb +61 -0
  406. data/lib/kotoshu/plugins/registry.rb +118 -0
  407. data/lib/kotoshu/plugins.rb +9 -0
  408. data/lib/kotoshu/project_config.rb +76 -0
  409. data/lib/kotoshu/readers/aff_data.rb +530 -0
  410. data/lib/kotoshu/readers/aff_reader.rb +393 -0
  411. data/lib/kotoshu/readers/condition_checker.rb +125 -0
  412. data/lib/kotoshu/readers/dic_reader.rb +208 -0
  413. data/lib/kotoshu/readers/file_reader.rb +377 -0
  414. data/lib/kotoshu/readers/lookup_builder.rb +584 -0
  415. data/lib/kotoshu/readers/readers.rb +17 -0
  416. data/lib/kotoshu/readers.rb +25 -0
  417. data/lib/kotoshu/resource_bundle.rb +55 -0
  418. data/lib/kotoshu/resource_manager.rb +600 -0
  419. data/lib/kotoshu/results/result.rb +165 -0
  420. data/lib/kotoshu/results.rb +8 -0
  421. data/lib/kotoshu/rspec.rb +163 -0
  422. data/lib/kotoshu/scripts/fasttext_to_onnx.py +275 -0
  423. data/lib/kotoshu/source_registry.rb +104 -0
  424. data/lib/kotoshu/spellchecker/parallel_checker.rb +90 -0
  425. data/lib/kotoshu/spellchecker.rb +406 -0
  426. data/lib/kotoshu/string_metrics.rb +153 -0
  427. data/lib/kotoshu/suggestions/context.rb +43 -0
  428. data/lib/kotoshu/suggestions/frequency_provider.rb +80 -0
  429. data/lib/kotoshu/suggestions/generator.rb +169 -0
  430. data/lib/kotoshu/suggestions/pipeline.rb +135 -0
  431. data/lib/kotoshu/suggestions/semantic_cascade.rb +73 -0
  432. data/lib/kotoshu/suggestions/strategies/base_strategy.rb +315 -0
  433. data/lib/kotoshu/suggestions/strategies/composite_strategy.rb +197 -0
  434. data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +631 -0
  435. data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +207 -0
  436. data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +165 -0
  437. data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +284 -0
  438. data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +317 -0
  439. data/lib/kotoshu/suggestions/strategies/symspell_strategy.rb +270 -0
  440. data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
  441. data/lib/kotoshu/suggestions/suggestion.rb +154 -0
  442. data/lib/kotoshu/suggestions/suggestion_set.rb +270 -0
  443. data/lib/kotoshu/suggestions/sweep_index.rb +137 -0
  444. data/lib/kotoshu/suggestions.rb +29 -0
  445. data/lib/kotoshu/tasks/check_task.rb +104 -0
  446. data/lib/kotoshu/tasks.rb +11 -0
  447. data/lib/kotoshu/validators/spelling_validator.rb +101 -0
  448. data/lib/kotoshu/validators.rb +9 -0
  449. data/lib/kotoshu/version.rb +5 -0
  450. data/lib/kotoshu.rb +538 -0
  451. data/rakelib/compile.rake +10 -0
  452. data/script/validate_all_dictionaries.rb +444 -0
  453. data/scripts/build_native_gem.rb +45 -0
  454. data/scripts/extract_eval_grids.rb +37 -0
  455. data/scripts/generate_frozen_tiers.rb +44 -0
  456. data/sig/kotoshu.rbs +435 -0
  457. data/test_oop.rb +79 -0
  458. metadata +546 -0
@@ -0,0 +1,281 @@
1
+ # Kotoshu Vision & Master Plan
2
+
3
+ > **Execution roadmap**: [65-universal-kotoshu.md](65-universal-kotoshu.md)
4
+ > is the master plan to "full success" — one Rust core with TS/Ruby/
5
+ > Python/Rust interfaces ([66-kotoshu-core.md](66-kotoshu-core.md),
6
+ > the parsanol blueprint: feature-gated FFI in the core, thin cdylib
7
+ > shims per language, `ort` for ONNX), tiered micro-models (mini ≈2 MB
8
+ > / fluency ≈14 MB / full ≈120 MB, benchmarked and labeled), and
9
+ > distribution on every channel (registries, editor marketplaces, OS
10
+ > packages, containers, signed offline bundles). Grounded in Microsoft
11
+ > Word's proof that a 14 MB signed ONNX fluency model is world-class.
12
+ > This file remains the philosophy; 65 is the road.
13
+
14
+ ## What Kotoshu is becoming
15
+
16
+ **A pure-Ruby spellchecker that works for every language by dynamically
17
+ downloading the right dictionary + frequency data + embedding model on
18
+ demand.**
19
+
20
+ Three load-bearing promises:
21
+
22
+ 1. **Pure Ruby workflow.** No shelling out to `hunspell`, `aspell`, or any
23
+ external binary. The only native dependency is `onnxruntime` (for embedding
24
+ inference). Every other piece — affix parsing, compounding, suggestion
25
+ algorithms, language detection, document parsing, interactive UI, output
26
+ formatting — is Ruby that ships in the gem.
27
+ 2. **Dynamic resource resolution.** Given any text, Kotoshu detects the
28
+ language, resolves the dictionary + frequency list + embedding model +
29
+ grammar rules that language needs, downloads them on first use into
30
+ `~/.kotoshu/`, verifies their integrity, and caches with TTL. The user
31
+ never installs dictionaries manually.
32
+ 3. **All languages.** Latin scripts (Hunspell morphological rules), CJK
33
+ (morphological tokenizers + confusion rules — no traditional spellcheck),
34
+ RTL (shaping-aware affix rules). A pluggable per-language module system
35
+ decides what each language uses.
36
+
37
+ ## Three analysis dimensions, composed per language
38
+
39
+ | Dimension | Source | Role |
40
+ |---|---|---|
41
+ | Traditional | Hunspell `.aff`/`.dic` | Authoritative correctness, morphological generation |
42
+ | Semantic | FastText ONNX embeddings | Context-aware reranking, OOV handling |
43
+ | Grammar | `rules.yaml` per language | Style/usage rules LanguageTool-style |
44
+
45
+ Each language module declares which dimensions it supports; the pipeline
46
+ composes whatever is available.
47
+
48
+ ## The five repos and what they're for
49
+
50
+ | Repo | Role |
51
+ |---|---|
52
+ | `kotoshu/kotoshu` | The gem (this one). Ruby API + CLI + analysis pipeline |
53
+ | `kotoshu/dictionaries` | Hunspell dictionaries, 98 language dirs |
54
+ | `kotoshu/frequency-list-kelly` | Kelly Project frequency tiers for ranking |
55
+ | `kotoshu/models-fasttext-onnx` | FastText embeddings converted to ONNX |
56
+ | `kotoshu/kotoshu.github.io` | Marketing/docs site |
57
+
58
+ The gem depends on the other four as **content repos** — never vendored, always
59
+ fetched on demand by the cache layer.
60
+
61
+ ## Plan index for this repo
62
+
63
+ | # | Plan | Why |
64
+ |---|---|---|
65
+ | 01 | [Hunspell correctness](01-hunspell-correctness.md) | Core lookup is ~50% on Spylls fixtures — must be fixed before anything trusts it |
66
+ | 02 | [CLI unification](02-cli-unification.md) | Two competing `check` commands; only the basic one is wired |
67
+ | 03 | [Dynamic resource download](03-dynamic-download.md) | The promise — unified resolver for any language |
68
+ | 04 | [Language modules](04-language-modules.md) | Wire 93+ dictionaries into the language module system |
69
+ | 05 | [Semantic path productionization](05-semantic-path.md) | Make ONNX reranking always-available, memory-bounded |
70
+ | 06 | [CJK support](06-cjk-support.md) | Japanese + Chinese (different paradigm — tokenizers + confusion) |
71
+ | 07 | [RTL support](07-rtl-support.md) | Arabic, Hebrew, Persian, Urdu |
72
+ | 08 | [Grammar engine](08-grammar-engine.md) | LanguageTool-style rule loader and pattern matchers |
73
+ | 09 | [Integrity & security](09-integrity-security.md) | Checksums, signed releases, audit log for downloads |
74
+ | 10 | [Testing & CI](10-testing-ci.md) | Coverage targets, CI matrix, perf benchmarks |
75
+ | 11 | [Release v1.0](11-release-v1.md) | Tag, publish, CHANGELOG, supported-language matrix |
76
+
77
+ ## Cross-repo dependency order
78
+
79
+ ```
80
+ 09-integrity ─┐
81
+ 03-download ──┼─► 04-language-modules ─► 06-cjk ─┐
82
+ │ ├─► 08-grammar ─► 11-release
83
+ ├─► 01-hunspell ──────────────────┤
84
+ └─► 05-semantic ───────────────────┘
85
+
86
+ 02-cli, 07-rtl, 10-testing can run in parallel with the above
87
+ ```
88
+
89
+ | Plan 03 (dynamic download) is the unlock — it depends on 09 (integrity) and
90
+ unblocks both 04 (language modules) and the user-facing "any language"
91
+ promise. Plans 01 (Hunspell) and 05 (semantic) can proceed independently.
92
+
93
+ Plans 60–64 are the **ecosystem / reach** track — they turn the working
94
+ gem into something users actually adopt. They live in their own repos
95
+ under `kotoshu/` per the five-repo workspace model:
96
+
97
+ | # | Plan | Why |
98
+ |---|---|---|
99
+ | 60 | [LSP server](60-lsp-server.md) | One server, every editor. Highest-leverage reach. |
100
+ | 61 | [Editor & CI ecosystem](61-editor-ecosystem.md) | VS Code, Vim, JetBrains, GitHub Action, pre-commit hooks |
101
+ | 62 | [Framework integrations](62-framework-integrations.md) | Rails validators, Jekyll rake tasks, RSpec matchers — inside the library gem |
102
+ | 63 | [Web playground (WASM)](63-web-playground-wasm.md) | Browser playground at `kotoshu.github.io/playground/` |
103
+ | 64 | [HTTP API & SDKs](64-http-api-and-sdks.md) | Self-hostable server + Python / JS / Go SDKs for non-Ruby stacks |
104
+
105
+ Plan 67 is the **delivery train** that publishes 65/66: tiered models
106
+ (mini/fluency/full) through the models repo's registry + releases, the
107
+ kotoshu-rs bootstrap and port, and the Ruby / WASM / Python access
108
+ libraries — [67-kotoshu-rs-and-access-libraries.md](67-kotoshu-rs-and-access-libraries.md).
109
+
110
+ Plan 68 is the **research adoption plan** (2026-09-02 SOTA survey):
111
+ what modern GEC/embedding/quantization research and frontier-model
112
+ training techniques Kotoshu adopts, adapts, or rejects —
113
+ [68-sota-adoption.md](68-sota-adoption.md). Its adopt-now items are
114
+ executed by plans
115
+ [69 (keyboard typo eval + corpus benchmark)](69-sota-typo-eval.md),
116
+ [70 (confidence cascade)](70-sota-confidence-cascade.md), and
117
+ [71 (nested single-artifact experiment)](71-sota-nested-model.md).
118
+ The remaining SOTA items are planned as
119
+ [72 (Model2Vec evaluation)](72-model2vec-evaluation.md),
120
+ [73 (RL-trained reranker, gated)](73-rl-reranker.md),
121
+ [74 (distilled reranker, gated)](74-distilled-reranker.md),
122
+ [75 (LLM-as-evaluator)](75-llm-evaluator.md), and
123
+ [76 (optional external LLM backend, gated on demand)](76-optional-llm-backend.md).
124
+
125
+ Plans 77–82 are the **adoption wave** (owner ask 2026-09-05: are the
126
+ models comprehensive, how do we cover more places, docs + news):
127
+ closing the verified user gaps after all channels went live.
128
+
129
+ | # | Plan | Gap it closes |
130
+ |---|---|---|
131
+ | 77 | [Model coverage expansion](77-model-coverage-expansion.md) | Semantic models cover 9 languages vs 98 dictionaries — expand to 25+ |
132
+ | 78 | [In-browser playground](78-in-browser-playground.md) | /playground needs a running server; @kotoshu/wasm enables zero-install |
133
+ | 79 | [News + releases](79-news-and-releases.md) | No news surface on the site; seven registries shipped invisibly |
134
+ | 80 | [Ecosystem docs](80-ecosystem-docs.md) | Docs are gem-only; pip/npm/cargo/go/HTTP/LSP/Action users have no path in |
135
+ | 81 | [Editor reach](81-editor-reach.md) | kotoshu-lsp shipped but no VS Code package; no pre-commit hook (supersedes 61 remainder) |
136
+ | 82 | [Document ignores + baselines](82-doc-formats-and-ignores.md) | No inline suppressions; no CI baseline freeze |
137
+
138
+ Plans 83–89 are **wave 2** (2026-09-05, owner re-ask after the wave-1
139
+ ship): driving models toward full intersection coverage, making the
140
+ 22 model languages full-feature, semantic reranking in the browser,
141
+ and the site/docs truth that follows.
142
+
143
+ | # | Plan | Gap it closes |
144
+ |---|---|---|
145
+ | 83 | [Models batch 2](83-models-batch-2.md) | ar/fa/he have modules but no models; ~50 more intersection languages; fi/id dictionary sourcing |
146
+ | 84 | [Language modules + keyboards wave 2](84-language-modules-wave2.md) | 13 model languages lack gem modules; 11 keyboard layouts missing |
147
+ | 85 | [Wasm model API](85-wasm-model-api.md) | Browser semantic reranking — no model API on the wasm surface, CORS distribution gap |
148
+ | 86 | [Site truth wave 2](86-site-truth-wave2.md) | Ignores/baselines undocumented; no Windows page; wrong personal.dic path in CLI docs; playground breadth |
149
+ | 87 | [LSP personal dictionary](87-lsp-personal-dictionary.md) | Server never reads personal.dic — add-to-dictionary appears broken |
150
+ | 88 | [CLI directory mode](88-cli-directory-mode.md) | No tree checking — the actual CI workflow |
151
+ | 89 | [Framework integrations](89-framework-integrations-exec.md) | Executes plan 62: Rails validator, RSpec matcher, Rake/Jekyll tasks |
152
+
153
+ Plans 91–95 are **wave 3** (2026-09-05 late): the correctness core the
154
+ site flip exposed, the browser-models distribution fix, Python wheels,
155
+ the docs for wave-2 features, and editor/action truth.
156
+
157
+ | # | Plan | Gap it closes |
158
+ |---|---|---|
159
+ | 91 | [Unicode + sv aff](91-unicode-and-sv-aff.md) | Executes plan 90: ASCII-only word detection hides non-Latin scripts; AffReader crashes on Swedish |
160
+ | 92 | [Tier mirror CORS](92-tier-mirror-cors.md) | Media host already sends ACAO — mirror the tiers, finish the browser-models path |
161
+ | 93 | [Python wheels](93-python-wheels.md) | kotoshu-native is macOS-only on PyPI; build the platform matrix + keyless publish |
162
+ | 94 | [Site integrations docs](94-site-integrations-docs.md) | Rails/RSpec/Rake/Jekyll + directory mode shipped undocumented; action page stale |
163
+ | 95 | [Editor + action polish](95-editor-action-polish.md) | Extension delegates the now-existing server command; action gains baseline + 19 languages |
164
+
165
+ Plans 96–97 are **wave 4** (2026-09-06): the audit found the wave-1–3
166
+ surface complete except two honest items — the HTTP server never
167
+ loads models, and plan 59's YARD reference is still unpublished. (The
168
+ Rust engine was audited too: its AST-based aff parser is immune to
169
+ the Swedish COMPOUNDRULE bug by design.)
170
+
171
+ | # | Plan | Gap it closes |
172
+ |---|---|---|
173
+ | 96 | [Server semantic models](96-server-semantic-models.md) | /v1/check is dictionary-only — SDK users get no reranking through HTTP |
174
+ | 97 | [YARD execution](97-yard-execution.md) | Executes plan 59: the Ruby API reference still does not exist |
175
+
176
+ Plan 98 is **wave 5, the cleanup and closeout** (2026-09-06): after the
177
+ release-day cuts (gem 0.8.0 → 0.9.0, @kotoshu/wasm 0.2.0, registry
178
+ v1.3.0 with nb, lsp/server 0.1.1 via trusted publishing) the workspace
179
+ was swept — 214 GB of models worktrees reclaimed, the plan corpus
180
+ 01–98 fully tracked, verification evidence preserved.
181
+
182
+ | # | Plan | What it records |
183
+ |---|---|---|
184
+ | 98 | [Campaign cleanup](98-campaign-cleanup.md) | The inventory, the executed sweep, the keep list, and the deliberately-unforced items |
185
+
186
+ Campaign state at close: every channel live and populated; the
187
+ convertible model pool exhausted at 55 languages; owner-side
188
+ remnants (PyPI registrations, marketplace PAT, nds/fi calls) recorded
189
+ in the plans, not pending work.
190
+
191
+ Plans 99-104 are **wave 6, the live-quality follow-through**
192
+ (2026-09-07), written after the sweep-performance arc (gem 0.9.1 ->
193
+ 0.9.2, @kotoshu/wasm 0.3.0 -> 0.3.2) and a fresh state audit: 55 model
194
+ languages x 3 tiers confirmed (ja/ko/zh rerank-only by design), no LID
195
+ resource mirrored, lsp/server `~> 0.6` constraints already resolving
196
+ current engines, site docs carrying stale version references.
197
+
198
+ | # | Plan | Gap it closes |
199
+ |---|---|---|
200
+ | 99 | [Site truth + performance docs](99-site-truth-performance-docs.md) | Docs misstate current versions; the measured per-language latency is unpublished |
201
+ | 100 | [Full-feature batch 3](100-full-feature-batch3.md) | ~35 languages have dictionaries + models but no gem module - the big-audience remainder (ar id fa he ...) |
202
+ | 101 | [int4 tier experiment](101-int4-tier-experiment.md) | Halving tier bytes was named (B1) but never run through the gates |
203
+ | 102 | [LID chain](102-lid-chain.md) | No detection in the registry, wasm, or playground - users pick the language blind |
204
+ | 103 | [OOV bucket tables](103-oov-bucket-tables.md) | Model candidates miss n-grams absent from vocab (the Teh gap's model half) |
205
+ | 104 | [VS Code marketplace readiness](104-vscode-marketplace-readiness.md) | The extension is built but not installable; everything short of the owner PAT |
206
+
207
+ Execution notes: plan 103 is sequenced after 101 (same repo, one
208
+ worktree at a time); all site-facing flips (99, 100, 102's playground
209
+ half, 104's link line) land through one coordinated site pass to keep
210
+ the repo conflict-free; lsp/server 0.1.2 bumps are deliberately NOT
211
+ planned - the `~> 0.6` floor already resolves 0.9.2 for fresh installs.
212
+
213
+ Dependency order across the ecosystem track:
214
+
215
+ ```
216
+ 60-lsp ─┬─► 61-editor-ecosystem
217
+ ├─► 62-framework-integrations (also depends on 02)
218
+ └─► 64-http-api (also depends on 02, 03, 05, 09)
219
+ 63-web-playground is independent (depends only on 02, 04)
220
+ ```
221
+
222
+ ## Cross-repo plan files
223
+
224
+ Each content repo has its own `TODO.impl/` directory:
225
+
226
+ - `dictionaries/TODO.impl/` — manifest, coverage matrix, releases, metadata, licenses
227
+ - `frequency-list-kelly/TODO.impl/` — coverage extension, validation, releases
228
+ - `models-fasttext-onnx/TODO.impl/` — publishing, storage, manifest, conversion CI
229
+ - `kotoshu.github.io/TODO.impl/` — site skeleton, content, examples landing
230
+
231
+ ## Source-of-truth note
232
+
233
+ Older planning docs now live in `docs/`
234
+ (`KOTOSHU_FULL_PLAN.md`, `KOTOSHU_SOLIDIFICATION_PLAN.md`,
235
+ `KOTOSHU_LANGUAGETOOL_GAPS.md`, `ARCHITECTURE_IMPROVEMENTS.md`,
236
+ `ARCHITECTURE_IMPROVEMENTS_PLAN.md`, `TDD_ITERATION_STRATEGY.md`,
237
+ `ONNX_FUNCTIONAL_VERIFICATION.md`). Actionable items have been
238
+ integrated into the relevant execution plans below — pointers in each
239
+ plan name the source.
240
+
241
+ | Source | Integrated into |
242
+ |---|---|
243
+ | `docs/KOTOSHU_FULL_PLAN.md` | Phases 0–8 spread across `01`–`11` per phase scope |
244
+ | `docs/KOTOSHU_SOLIDIFICATION_PLAN.md` (49 OOP items) | `01` (Parts 1–2 Hunspell/CSpell), `04`/`06`/`07` (Part 3 LanguageTool), `10` (Part 5 testing); Part 4 perf items live in the relevant feature plan |
245
+ | `docs/KOTOSHU_LANGUAGETOOL_GAPS.md` | `08` (rules, confusion), `02` (CLI surface, output formats), `04` (target languages) |
246
+ | `docs/ARCHITECTURE_IMPROVEMENTS.md` (15 patterns) | Most patterns are architectural guidance applied across plans; load-bearing rules captured in `CLAUDE.md` |
247
+ | `docs/ARCHITECTURE_IMPROVEMENTS_PLAN.md` (5 phases) | Phase 1 perf items in `01`/`05`; Phase 2 architecture in `00-vision` (this doc); Phase 3 UX mostly implemented (`PersonalDictionary`, `ProjectConfig`, `FluentChecker` exist); Phase 4 quality in `10`; Phase 5 polish spread |
248
+ | `docs/TDD_ITERATION_STRATEGY.md` | `10` (test methodology + CI); project-rule overrides (no doubles, behavior-not-implementation) in `CLAUDE.md` |
249
+ | `docs/ONNX_FUNCTIONAL_VERIFICATION.md` | `05` (semantic path productionization) |
250
+
251
+
252
+ Plans 105-107 plus models 09 and site 11 are **wave 7, detection and
253
+ coverage depth** (2026-09-07): after the wave-6 releases (gem 0.9.3,
254
+ wasm 0.4.0, registry v1.4.0, action v2), the remaining gaps are the
255
+ personal dictionary never reaching the check path, detection stuck on
256
+ a 7-language heuristic, buckets only for en/de, and the playground not
257
+ yet passing bucket bytes to loadModel.
258
+
259
+ | # | Plan | Gap it closes |
260
+ |---|---|---|
261
+ | 105 | [Personal dictionary in check](105-personal-dictionary-in-check.md) | kotoshu personal add then check still flags the word |
262
+ | 106 | [Native LID detection](106-native-lid-detection.md) | detect is a 7-language heuristic; the pure-Rust lid reader exists unused |
263
+ | 107 | models [09 bucket expansion](../../models-fasttext-onnx/TODO.impl/09-bucket-table-expansion.md) | bucket OOV siblings exist only for en and de |
264
+ | — | site 11 playground buckets | wasm 0.4.0 accepts bucketsBytes; the worker never fetches the sibling |
265
+
266
+
267
+ Plans 107-109 are **wave 8, reachability and the road to 1.0**
268
+ (2026-09-07): after wave 7 (personal dict in check, 176-language
269
+ detection, buckets rolling to all languages), the remaining user gaps
270
+ are the sixty-plus staged dictionaries unreachable without a module,
271
+ the last big-population languages (ko, ne), and the absence of a
272
+ stability commitment adopters can build on.
273
+
274
+ | # | Plan | Gap it closes |
275
+ |---|---|---|
276
+ | 107 | [Basic support for every staged language](107-basic-support-every-staged-language.md) | setup rejects 60+ staged languages; modules become upgrades, not gates |
277
+ | 108 | [ko and ne full-feature](108-ko-ne-full-feature.md) | Hangul and Devanagari - the last ~100M speakers without support |
278
+ | 109 | [The 1.0 readiness audit](109-ten-audit.md) | no frozen surface or stability policy for adopters |
279
+
280
+ Sequencing: 108 after 107 (same area); 109 is analysis-only and runs in
281
+ parallel. The 1.0 cut itself remains the owner's version call.
@@ -0,0 +1,111 @@
1
+ # 01 — Hunspell Correctness
2
+
3
+ ## Goal
4
+
5
+ Pass 100% of the Spylls-ported fixture suite in `spec/hunspell_tests/` and
6
+ the Hunspell reference fixtures under
7
+ `/Users/mulgogi/src/external/hunspell/tests/`. Baseline at the start of
8
+ this plan was **48/107 (45 %)** passing; current state per
9
+ `CHANGELOG.md` is **803/866 (92.7 %)** — close but not done.
10
+
11
+ ## Why
12
+
13
+ The lookup/affix/compound machinery is the foundation of the
14
+ traditional path; nothing downstream (suggestions, grammar,
15
+ multi-language) can be trusted until this is correct.
16
+
17
+ ## Current blockers
18
+
19
+ - `lib/kotoshu/algorithms/lookup.rb` is a procedural ~877-line Python
20
+ port; hash/array passing instead of domain models.
21
+ - Overlap between `Lookup`, `LookupBuilder`, `Lookuper` (not MECE).
22
+ - Affix data split between `AffReader`, `aff_data.rb`, `lookup.rb`.
23
+ - Missing: complex compounding, circumfix, ICONV/OCONV, German ß,
24
+ Turkish dotless-i, special-character handling.
25
+
26
+ ## Tasks
27
+
28
+ 1. **Audit current pass rate.** Run `bundle exec rspec spec/hunspell_tests/`
29
+ and `bundle exec rspec spec/integration/`. Record the actual number
30
+ and failing fixture names in "Status" below.
31
+ 2. **Compound rules.** Implement `COMPOUNDRULE`, `COMPOUNDFLAG`,
32
+ `COMPOUNDBEGIN/MIDDLE/END`, `COMPOUNDPERMFLAG`, `COMPOUNDMIN`,
33
+ `CHECKCOMPOUNDREP`, `CHECKCOMPOUNDCASE`, `CHECKCOMPOUNDTRIPLE`,
34
+ `CHECKCOMPOUNDDUP`, `FORBIDDENWORD` inside compounds.
35
+ 3. **Circumfix.** Bidirectional prefix+suffix via `CIRCUMFIX` flag.
36
+ 4. **ICONV / OCONV.** Input/output character conversion tables applied
37
+ at the right point in the pipeline.
38
+ 5. **Special characters.** German ß uppercasing, Turkish dotless-i,
39
+ Dutch IJ, Break patterns (`BREAK`), `IGNORE` chars, `CHECKSHARPS`,
40
+ Arabic character ignoring, UTF-8 BOM, legacy encodings
41
+ (Windows-1251).
42
+ 6. **MECE refactor of the lookup pipeline.** Replace the procedural
43
+ `lookup.rb` with a four-layer separation (per the reimplementation
44
+ plan):
45
+ - **Dictionary layer** (data structures): `WordRepository`,
46
+ `AffixRepository`, `CompoundRuleRepository`,
47
+ `MorphologicalAnalyzer`.
48
+ - **Lookup layer** (algorithms): `WordLookup`, `AffixLookup`,
49
+ `CompoundLookup`, `MorphologicalLookup`.
50
+ - **Validation layer** (business rules): `ConditionValidator`,
51
+ `FlagValidator`, `CompoundValidator`,
52
+ `CapitalizationValidator`.
53
+ - **Transformation layer** (operations): `AffixApplier`,
54
+ `AffixStripper`, `WordFormer`, `CompoundFormer`.
55
+ Collapse `Lookup`/`LookupBuilder`/`Lookuper` into one responsibility
56
+ boundary or document why three classes exist.
57
+ 7. **Domain-model completion.** Verify `Word`, `AffixRule`,
58
+ `CompoundWord`, `Condition` carry the behavior, not the services
59
+ (per the global "models first" rule). Query methods required:
60
+ `word.stem`, `word.has_flag?(:verb)`, `word.forms`,
61
+ `affix_rule.valid_for?(stem)`, `affix_rule.apply(stem)`,
62
+ `affix_rule.can_combine?(other)`, `compound.valid_at?(position)`,
63
+ `condition.matches?(stem)`.
64
+ 8. **Strangler-Fig migration.** Build the new pipeline alongside the
65
+ old; expose a configuration switch (e.g.
66
+ `Configuration.use_modern_lookup` defaulting to `false`) so the
67
+ cutover is per-language and reversible. Remove the old
68
+ `algorithms/lookup.rb` only after the new path passes every fixture
69
+ the old one did.
70
+ 9. **Port the failing Spylls integrational fixtures** as named spec
71
+ examples: `dotless_i`, `IJ`, `ignore`, `checksharps`, `break`,
72
+ `needaffix`, `onlyincompound`, `alias`, `complexprefixes`,
73
+ `germancompounding`, `checkcompoundpattern`, plus the Hunspell
74
+ reference corpus fixtures (`1463589` Turkish UTF-8, `1592880`,
75
+ `1695964`, `1706659`, `1748408`, `breakdefault`, `breakoff`,
76
+ `encoding`, `utf8`, `utf8_bom`, `utf8_bom2`, `ignoreutf`,
77
+ `ignoresug`, `checksharpsutf`).
78
+ 10. **Performance baseline.** Profile lookup, affix application, and
79
+ compound check; add `spec/performance/lookup_bench.rb` with
80
+ assertions.
81
+
82
+ ## Acceptance criteria
83
+
84
+ - `bundle exec rspec spec/hunspell_tests/` is green (100 % of Spylls
85
+ fixtures).
86
+ - New specs added for every behavior fixed (no fixture passes
87
+ silently).
88
+ - `lib/kotoshu/algorithms/lookup.rb` is under 400 lines, or its
89
+ responsibilities have been redistributed across the four layers
90
+ above with the boundary documented.
91
+ - **Performance:** < 100 ms average lookup time per 1000 words;
92
+ < 100 MB memory for the en_US dictionary.
93
+ - **Code quality:** ≤ 50 lines per method, ≤ 300 lines per file, no
94
+ primitive obsession (domain models, not raw hashes), SRP followed,
95
+ dependencies on abstractions.
96
+ - No `double()` in any new spec (project rule — see `CLAUDE.md`).
97
+
98
+ ## Dependencies
99
+
100
+ - None inside this repo. Blocks `04-language-modules` (which assumes
101
+ a trustworthy Hunspell path) and `11-release`.
102
+
103
+ ## Out of scope
104
+
105
+ - Suggestion quality tuning (covered in `05-semantic-path`).
106
+ - Grammar rules (covered in `08-grammar-engine`).
107
+ - New languages beyond what the existing fixture suite covers.
108
+
109
+ ## Status
110
+
111
+ _Pending — fill in actual pass rate after task 1._
@@ -0,0 +1,74 @@
1
+ # 02 — CLI Unification
2
+
3
+ ## Goal
4
+
5
+ One canonical CLI surface that exposes every feature the gem supports, with
6
+ flags that match the README.
7
+
8
+ ## Why
9
+
10
+ There are two competing implementations:
11
+
12
+ - `lib/kotoshu/cli.rb` — `Kotoshu::Cli::Cli < Thor`. This is what
13
+ `exe/kotoshu` actually starts. Has only a basic `check TARGET` command
14
+ and `dict`/`cache` subcommands.
15
+ - `lib/kotoshu/commands/check_command.rb` — `Kotoshu::CheckCommand < Thor`
16
+ with `namespace :check`, exposes `--language auto`, `--interactive`,
17
+ `--output`, `--format text|json|yaml|csv|sarif`, `--model fasttext|hunspell`,
18
+ `--download`. **Not registered as a subcommand anywhere — dead code.**
19
+
20
+ The README advertises `--interactive`, `--format sarif`, `--model`,
21
+ `--language auto`. Users running `bundle exec exe/kotoshu check README.md
22
+ --interactive` get an "unknown option" error.
23
+
24
+ ## Tasks
25
+
26
+ 1. **Decide canonical location.** Recommendation: keep the rich
27
+ `Kotoshu::CheckCommand` (it has all the features users want) and delete
28
+ the basic `check` from `cli.rb`. Wire `cli.rb` to register
29
+ `CheckCommand` via `subcommand "check", Kotoshu::CheckCommand` (or
30
+ register at top level if `check` is the only command).
31
+ 2. **Move `DictCommand` and `CacheCommand`** into `lib/kotoshu/commands/`
32
+ alongside `CheckCommand` so all CLI commands live in one directory.
33
+ 3. **Add a `model` subcommand** so users can manage ONNX models
34
+ (`kotoshu model download en`, `kotoshu model list`,
35
+ `kotoshu model validate`, `kotoshu model convert`). The README
36
+ references these — they live in `lib/kotoshu/commands/model_command.rb`
37
+ already; verify it's wired.
38
+ 4. **Add `--from-stdin` mode** so the CLI can be used in shell pipes:
39
+ `echo "wrold" | kotoshu check --from-stdin`.
40
+ 5. **Reconcile flag names** between README and code. Specifically:
41
+ - README says `--dictionary=unix_words`; `cli.rb` has both `--dictionary`
42
+ and `--dictionary-path` — pick one.
43
+ - README says `--output=json`; `CheckCommand` uses `--format=json
44
+ --output=FILE` — clarify the two-flag split.
45
+ 6. **Exit codes.** Define and document: 0 = clean, 1 = errors found, 2 =
46
+ usage error, 3 = dictionary/model fetch failure. SARIF consumers
47
+ (GitHub Code Scanning) require this contract.
48
+ 7. **Help text audit.** Every Thor `desc` should match what's in the
49
+ README, and the README should match what's in code.
50
+
51
+ ## Acceptance criteria
52
+
53
+ - `bundle exec exe/kotoshu check README.md --interactive` works as
54
+ documented
55
+ - `bundle exec exe/kotoshu check README.md --format sarif --output
56
+ results.sarif` produces valid SARIF 2.1.0 that GitHub accepts
57
+ - `bundle exec exe/kotoshu --help` lists every command the README mentions
58
+ - No duplicate command classes; one canonical path per command
59
+ - A spec under `spec/kotoshu/cli/` exercises each subcommand end-to-end
60
+ (using a temp dir, not real `~/.kotoshu`)
61
+
62
+ ## Dependencies
63
+
64
+ - Light dependency on `03-dynamic-download` (CLI should let users
65
+ pre-download resources), but can proceed independently.
66
+
67
+ ## Out of scope
68
+
69
+ - LSP server, HTTP server (future work, not v1)
70
+ - GUI / TUI redesign — keep the existing Thor UX
71
+
72
+ ## Status
73
+
74
+ _Pending._
@@ -0,0 +1,107 @@
1
+ # 03 — Dynamic Resource Download
2
+
3
+ ## Goal
4
+
5
+ One call — `Kotoshu.check(text)` — works for any of the supported
6
+ languages without the user pre-installing anything. The library detects
7
+ language, resolves what's needed, downloads it, verifies integrity, and
8
+ caches.
9
+
10
+ This is the heart of the project's promise.
11
+
12
+ ## Why
13
+
14
+ Today the three caches (`LanguageCache`, `FrequencyCache`, `ModelCache`)
15
+ exist independently. The user (or the spellchecker) has to know which to
16
+ call. Configuration is keyed on `dictionary_type` and `language`, but
17
+ there's no unified resolver that takes arbitrary text and yields the full
18
+ resource bundle.
19
+
20
+ ## The missing piece: `ResourceManager`
21
+
22
+ A new top-level component (autoloaded from
23
+ `lib/kotoshu/resource_manager.rb`) that owns the contract:
24
+
25
+ ```ruby
26
+ bundle = Kotoshu::ResourceManager.resolve(
27
+ text: "Guten Tag, wie geht's?",
28
+ language: :auto, # or "de", "en-US", nil
29
+ want: %i[spelling frequency embeddings grammar],
30
+ force_download: false
31
+ )
32
+ # => #<ResourceBundle
33
+ # language: "de",
34
+ # dictionary: #<Dictionary::Hunspell ...>,
35
+ # frequency: #<FrequencyData ...>,
36
+ # model: #<OnnxModel ...>, # nil if unavailable
37
+ # rules: #<Grammar::RuleSet ...>, # nil if not present
38
+ # cached: true,
39
+ # source_urls: [...]>
40
+ ```
41
+
42
+ The `Spellchecker` is rewritten to take a `ResourceBundle` instead of a
43
+ single `Dictionary::Base`.
44
+
45
+ ## Tasks
46
+
47
+ 1. **Define `ResourceBundle`** as a value object (Struct or plain class —
48
+ no hand-rolled `to_h`; project rule). Fields: `language`, `dictionary`,
49
+ `frequency`, `model`, `rules`, `cached`, `source_urls`.
50
+ 2. **Implement `ResourceManager.resolve`.** Pseudocode:
51
+ ```
52
+ detect language if :auto (Language::Identifier)
53
+ parallel-fetch each wanted resource (cache → download → verify)
54
+ construct ResourceBundle
55
+ ```
56
+ Use existing caches — don't reimplement.
57
+ 3. **Manifest verification** (depends on `09-integrity-security` and on
58
+ each content repo shipping a `manifest.json` with SHA-256s). Every
59
+ download is rejected if the checksum doesn't match.
60
+ 4. **Offline mode.** `Kotoshu.check(text, offline: true)` only uses cached
61
+ resources; raises `ResourceNotCachedError` if missing instead of
62
+ hitting the network. Required for CI/CD and air-gapped use.
63
+ 5. **Strict mode.** `Kotoshu.check(text, strict: true)` raises if any
64
+ wanted resource can't be obtained (vs. default graceful degradation:
65
+ missing model → skip semantic rerank; missing frequency → no bonus).
66
+ 6. **Cache pre-warm CLI**. `kotoshu fetch de en es fr pt ru` resolves and
67
+ downloads all resource types for those languages. Useful before
68
+ air travel / CI.
69
+ 7. **Configuration knobs**. Add to `Configuration::SCHEMA`:
70
+ - `resource_sources` — hash mapping resource type → URL base
71
+ (defaults to current `dictionaries_url`)
72
+ - `offline` (boolean, default false)
73
+ - `strict_resources` (boolean, default false)
74
+ - `parallel_downloads` (integer, default 4)
75
+ 8. **Progress reporting.** A `Progress` callback hooks into the cache
76
+ downloads so the CLI can show `[en] downloading dictionary (3.2MB)…`
77
+ 9. **Spec coverage.** A new `spec/integration/dynamic_resolution_spec.rb`
78
+ with a fake HTTP server (Sinatra::Base, no VCR — project rule from
79
+ commit `90aa886`) that serves fixtures and verifies the full resolve →
80
+ cache → re-use cycle. Tag `:network` so it's skippable.
81
+
82
+ ## Acceptance criteria
83
+
84
+ - `Kotoshu.check("Bonjour le monde")` works on a fresh machine after
85
+ installing only the gem (auto-downloads fr dictionary + Kelly + ONNX)
86
+ - `Kotoshu.check("こんにちは", offline: true)` raises a useful error
87
+ explaining what to pre-fetch
88
+ - All downloads verify checksums; mismatch raises
89
+ `IntegrityError` with the expected vs. actual hash
90
+ - CLI `kotoshu fetch de` warms every resource type for German
91
+ - No network call when the cache is hot
92
+
93
+ ## Dependencies
94
+
95
+ - Blocks: `04-language-modules`, `05-semantic`, `06-cjk`, `07-rtl`
96
+ - Blocked by: `09-integrity-security`, and each content repo's
97
+ `01-manifest-checksums.md`
98
+
99
+ ## Out of scope
100
+
101
+ - P2P distribution, torrent mirrors
102
+ - Streaming/incremental dictionary load (load whole file then index)
103
+ - WebAssembly / in-browser use
104
+
105
+ ## Status
106
+
107
+ _Pending._
@@ -0,0 +1,89 @@
1
+ # 04 — Language Modules
2
+
3
+ ## Goal
4
+
5
+ Every language in `kotoshu/dictionaries` (98 dirs) is reachable through
6
+ the same `Language::*` module system, with the right tokenizer,
7
+ normalizer, keyboard layout, and resource bindings.
8
+
9
+ ## Why
10
+
11
+ The current code ships 7 language modules under `lib/kotoshu/languages/`
12
+ (de, en, es, fr, ja, pt, ru) and 8 tokenizers under
13
+ `lib/kotoshu/language/tokenizer/`. The `dictionaries` repo exposes 98
14
+ language codes. The gap is the user-visible "all languages" promise.
15
+
16
+ ## Tasks
17
+
18
+ 1. **Language module contract.** Codify what
19
+ `Kotoshu::Language::Base` (in `language/languages/base.rb`) requires:
20
+ - `code` (ISO 639-1, e.g. `"de"`)
21
+ - `locale` (default, e.g. `"de-DE"`)
22
+ - `script` (`:latin | :cyrillic | :cjk | :rtl`)
23
+ - `tokenizer_class`
24
+ - `normalizer_class`
25
+ - `keyboard_layout` (symbol)
26
+ - `dictionary_source` (URL/path within `dictionaries` repo)
27
+ - `frequency_source` (URL/path within `frequency-list-kelly` repo, or
28
+ nil)
29
+ - `model_source` (URL/path within `models-fasttext-onnx` repo, or
30
+ nil)
31
+ - `grammar_rules_path` (within `dictionaries/{code}/grammar/`, or
32
+ nil)
33
+ 2. **Audit the dictionaries repo** (cross-references
34
+ `dictionaries/TODO.impl/02-coverage-matrix.md`) and register a module
35
+ for every code that has at least a `spelling/` dir. Generate the
36
+ module files from a template rather than hand-writing 90+ files.
37
+ 3. **Tokenizer coverage.** Latin languages share `LatinTokenizer`; map
38
+ each Latin language to it with locale-specific overrides
39
+ (contractions for fr/it/es, ß handling for de, etc.). Cyrillic uses
40
+ the same family. CJK and RTL get dedicated tokenizers (covered in
41
+ plans `06` and `07`).
42
+ 4. **Normalizer coverage.** Per-language normalization rules (case
43
+ folding, accent stripping where appropriate, Unicode NFC).
44
+ 5. **Keyboard layouts.** Currently `keyboard/layouts/` has qwerty,
45
+ qwertz, azerty, jcuken, dvorak. Add layouts needed by the rest of
46
+ the matrix (e.g. Turkish F, Arabic, Hebrew).
47
+ 6. **Language auto-detection contract.** `Language::Identifier` returns
48
+ ISO 639-1 codes; ensure every code it can return has a registered
49
+ module. If detection returns an unsupported code, fall back to
50
+ English with a warning.
51
+ 7. **Language-specific resource URL templates.** Centralize the URL
52
+ patterns in `LanguageCache`/`FrequencyCache`/`ModelCache` so a new
53
+ language module doesn't need to repeat them.
54
+ 8. **`Kotoshu.supported_languages` accuracy.** Currently returns
55
+ `Language.supported_codes`; ensure this is the source of truth for
56
+ the README's language matrix and the CLI's `--language` enum.
57
+
58
+ ## Acceptance criteria
59
+
60
+ - Running `Kotoshu.supported_languages.size` returns ≥ 90 (matching
61
+ dictionaries repo coverage)
62
+ - For every language code in
63
+ `dictionaries/*/spelling/index.dic`, a corresponding
64
+ `lib/kotoshu/languages/{code}/language.rb` exists and is registered
65
+ - A new integration spec `spec/integration/all_languages_spec.rb`
66
+ iterates every supported language and asserts that
67
+ `ResourceManager.resolve(text: sample_text, language: code)` returns
68
+ a valid `ResourceBundle` with a non-nil `dictionary`. Tag `:network`.
69
+ - README's "Language Support Matrix" is generated from code, not
70
+ hand-maintained
71
+
72
+ ## Dependencies
73
+
74
+ - Blocks: nothing directly, but enables v1 release
75
+ - Blocked by: `01-hunspell-correctness` (need correct morphological
76
+ path), `03-dynamic-download` (need resource resolution)
77
+ - Cross-repo: depends on `dictionaries/TODO.impl/02-coverage-matrix.md`
78
+ being current
79
+
80
+ ## Out of scope
81
+
82
+ - CJK and RTL tokenizers (separate plans `06`, `07`)
83
+ - New grammar rules per language (plan `08`)
84
+ - Custom dictionaries users might supply (already supported via
85
+ `Dictionary::Custom`)
86
+
87
+ ## Status
88
+
89
+ _Pending._