kotoshu 1.0.2-aarch64-linux

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (458) hide show
  1. checksums.yaml +7 -0
  2. data/.kotoshu-baseline.json +1 -0
  3. data/.pre-commit-hooks.yaml +18 -0
  4. data/.rspec +3 -0
  5. data/.rubocop.yml +351 -0
  6. data/.yardopts +8 -0
  7. data/CHANGELOG.md +813 -0
  8. data/CLAUDE.md +176 -0
  9. data/CODE_OF_CONDUCT.md +132 -0
  10. data/Cargo.lock +359 -0
  11. data/Cargo.toml +15 -0
  12. data/LICENSE +31 -0
  13. data/README.adoc +1292 -0
  14. data/Rakefile +150 -0
  15. data/SECURITY.md +93 -0
  16. data/TODO.impl/00-cut-0.1.md +121 -0
  17. data/TODO.impl/00-cut-0.2.md +175 -0
  18. data/TODO.impl/00-vision.md +281 -0
  19. data/TODO.impl/01-hunspell-correctness.md +111 -0
  20. data/TODO.impl/02-cli-unification.md +74 -0
  21. data/TODO.impl/03-dynamic-download.md +107 -0
  22. data/TODO.impl/04-language-modules.md +89 -0
  23. data/TODO.impl/05-semantic-path.md +83 -0
  24. data/TODO.impl/06-cjk-support.md +105 -0
  25. data/TODO.impl/07-rtl-support.md +71 -0
  26. data/TODO.impl/08-grammar-engine.md +106 -0
  27. data/TODO.impl/09-integrity-security.md +90 -0
  28. data/TODO.impl/10-testing-ci.md +98 -0
  29. data/TODO.impl/100-full-feature-batch3.md +28 -0
  30. data/TODO.impl/101-int4-tier-experiment.md +32 -0
  31. data/TODO.impl/102-lid-chain.md +28 -0
  32. data/TODO.impl/103-oov-bucket-tables.md +27 -0
  33. data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
  34. data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
  35. data/TODO.impl/106-native-lid-detection.md +36 -0
  36. data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
  37. data/TODO.impl/108-ko-ne-full-feature.md +37 -0
  38. data/TODO.impl/109-ten-audit.md +26 -0
  39. data/TODO.impl/11-release-v1.md +95 -0
  40. data/TODO.impl/110-sr-latn-and-polish.md +33 -0
  41. data/TODO.impl/111-embedding-bakeoff.md +33 -0
  42. data/TODO.impl/112-deployability-and-gating.md +26 -0
  43. data/TODO.impl/113-language-packs.md +26 -0
  44. data/TODO.impl/114-typo-biencoder-thread.md +33 -0
  45. data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
  46. data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
  47. data/TODO.impl/117-deterministic-frequency-tiers.md +110 -0
  48. data/TODO.impl/118-server-gem-floor.md +33 -0
  49. data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
  50. data/TODO.impl/120-setup-metadata-refresh.md +35 -0
  51. data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
  52. data/TODO.impl/122-pack-compression-kpk2.md +66 -0
  53. data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
  54. data/TODO.impl/124-owner-residue.md +19 -0
  55. data/TODO.impl/125-consumer-floor-guards.md +43 -0
  56. data/TODO.impl/126-tier-error-budgets.md +33 -0
  57. data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
  58. data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
  59. data/TODO.impl/129-pt-fr-corpora.md +23 -0
  60. data/TODO.impl/130-news-truth-0912.md +23 -0
  61. data/TODO.impl/131-hybrid-productization.md +42 -0
  62. data/TODO.impl/132-techniques-page-and-prose.md +27 -0
  63. data/TODO.impl/133-rust-first-distribution.md +68 -0
  64. data/TODO.impl/20-encapsulation-cleanup.md +47 -0
  65. data/TODO.impl/21-autoload-migration.md +64 -0
  66. data/TODO.impl/22-url-pin-single-source-of-truth.md +120 -0
  67. data/TODO.impl/23-fix-url-defaults.md +57 -0
  68. data/TODO.impl/24-cli-auto-setup-prompt.md +76 -0
  69. data/TODO.impl/25-soft-onnxruntime-require.md +75 -0
  70. data/TODO.impl/26-onnx-vocab-json-generation.md +96 -0
  71. data/TODO.impl/27-end-to-end-smoke-spec.md +113 -0
  72. data/TODO.impl/28-readme-quickstart-verification.md +68 -0
  73. data/TODO.impl/29-status-command.md +84 -0
  74. data/TODO.impl/30-language-auto-detection.md +74 -0
  75. data/TODO.impl/31-download-progress.md +58 -0
  76. data/TODO.impl/32-sarif-output-verification.md +72 -0
  77. data/TODO.impl/33-audit-log-rotation.md +54 -0
  78. data/TODO.impl/34-cache-eviction.md +59 -0
  79. data/TODO.impl/35-shell-completion.md +61 -0
  80. data/TODO.impl/36-cleanup-regressions-0.3.1.md +134 -0
  81. data/TODO.impl/37-hunspell-correctness-tier2.md +203 -0
  82. data/TODO.impl/38-onnx-semantic-gating.md +218 -0
  83. data/TODO.impl/39-tier3-and-beyond.md +232 -0
  84. data/TODO.impl/40-spec-drift-cleanup.md +134 -0
  85. data/TODO.impl/50-structure-aware-document-api.md +194 -0
  86. data/TODO.impl/51-grammar-engine-expansion.md +75 -0
  87. data/TODO.impl/52-multi-language-documents.md +95 -0
  88. data/TODO.impl/53-document-plugin-architecture.md +94 -0
  89. data/TODO.impl/54-cjk-support.md +117 -0
  90. data/TODO.impl/55-rtl-support.md +109 -0
  91. data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
  92. data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
  93. data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
  94. data/TODO.impl/59-yard-api-documentation.md +75 -0
  95. data/TODO.impl/60-lsp-server.md +174 -0
  96. data/TODO.impl/61-editor-ecosystem.md +167 -0
  97. data/TODO.impl/62-framework-integrations.md +159 -0
  98. data/TODO.impl/63-web-playground-wasm.md +186 -0
  99. data/TODO.impl/64-http-api-and-sdks.md +191 -0
  100. data/TODO.impl/65-universal-kotoshu.md +251 -0
  101. data/TODO.impl/66-kotoshu-core.md +169 -0
  102. data/TODO.impl/67-kotoshu-rs-and-access-libraries.md +137 -0
  103. data/TODO.impl/68-sota-adoption.md +152 -0
  104. data/TODO.impl/69-sota-typo-eval.md +74 -0
  105. data/TODO.impl/70-sota-confidence-cascade.md +59 -0
  106. data/TODO.impl/71-sota-nested-model.md +63 -0
  107. data/TODO.impl/72-model2vec-evaluation.md +49 -0
  108. data/TODO.impl/73-rl-reranker.md +42 -0
  109. data/TODO.impl/74-distilled-reranker.md +36 -0
  110. data/TODO.impl/75-llm-evaluator.md +35 -0
  111. data/TODO.impl/76-optional-llm-backend.md +37 -0
  112. data/TODO.impl/77-model-coverage-expansion.md +96 -0
  113. data/TODO.impl/78-in-browser-playground.md +69 -0
  114. data/TODO.impl/79-news-and-releases.md +60 -0
  115. data/TODO.impl/80-ecosystem-docs.md +56 -0
  116. data/TODO.impl/81-editor-reach.md +68 -0
  117. data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
  118. data/TODO.impl/83-models-batch-2.md +56 -0
  119. data/TODO.impl/84-language-modules-wave2.md +75 -0
  120. data/TODO.impl/85-wasm-model-api.md +48 -0
  121. data/TODO.impl/86-site-truth-wave2.md +45 -0
  122. data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
  123. data/TODO.impl/88-cli-directory-mode.md +54 -0
  124. data/TODO.impl/89-framework-integrations-exec.md +59 -0
  125. data/TODO.impl/90-gem-correctness-residues.md +44 -0
  126. data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
  127. data/TODO.impl/92-tier-mirror-cors.md +88 -0
  128. data/TODO.impl/93-python-wheels.md +32 -0
  129. data/TODO.impl/94-site-integrations-docs.md +35 -0
  130. data/TODO.impl/95-editor-action-polish.md +27 -0
  131. data/TODO.impl/96-server-semantic-models.md +50 -0
  132. data/TODO.impl/97-yard-execution.md +73 -0
  133. data/TODO.impl/98-campaign-cleanup.md +78 -0
  134. data/TODO.impl/99-site-truth-performance-docs.md +31 -0
  135. data/conformance/vectors.jsonl +2630 -0
  136. data/data/en/grammar/rules.yaml +851 -0
  137. data/docs/STABILITY-1.0-AUDIT.md +251 -0
  138. data/docs/STABILITY-1.0.md +137 -0
  139. data/docs/cache-architecture.md +502 -0
  140. data/docs/verification-evidence/correct3/pr_body.md +37 -0
  141. data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
  142. data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
  143. data/docs/verification-evidence/correct3/sv.aff +856 -0
  144. data/docs/verification-evidence/correct3/sv.dic +153717 -0
  145. data/examples/01_basic_word_checking.rb +38 -0
  146. data/examples/02_text_document_checking.rb +77 -0
  147. data/examples/03_dictionary_backends.rb +137 -0
  148. data/examples/04_trie_data_structure.rb +146 -0
  149. data/examples/05_suggestion_algorithms.rb +239 -0
  150. data/examples/06_configuration_advanced.rb +287 -0
  151. data/examples/07_multi_language_dictionaries.rb +278 -0
  152. data/exe/kotoshu +6 -0
  153. data/lib/kotoshu/algorithms/capitalization.rb +276 -0
  154. data/lib/kotoshu/algorithms/edit_distance.rb +128 -0
  155. data/lib/kotoshu/algorithms/lookup/affix_form.rb +132 -0
  156. data/lib/kotoshu/algorithms/lookup/compound_checks.rb +209 -0
  157. data/lib/kotoshu/algorithms/lookup/compound_form.rb +50 -0
  158. data/lib/kotoshu/algorithms/lookup.rb +841 -0
  159. data/lib/kotoshu/algorithms/ngram_suggest.rb +367 -0
  160. data/lib/kotoshu/algorithms/permutations.rb +283 -0
  161. data/lib/kotoshu/algorithms/phonet_suggest.rb +162 -0
  162. data/lib/kotoshu/algorithms/soundex.rb +92 -0
  163. data/lib/kotoshu/algorithms/suggest.rb +794 -0
  164. data/lib/kotoshu/algorithms.rb +23 -0
  165. data/lib/kotoshu/analyzers/semantic_analyzer.rb +405 -0
  166. data/lib/kotoshu/analyzers.rb +8 -0
  167. data/lib/kotoshu/baseline/application.rb +18 -0
  168. data/lib/kotoshu/baseline/entry.rb +33 -0
  169. data/lib/kotoshu/baseline/store.rb +175 -0
  170. data/lib/kotoshu/baseline.rb +20 -0
  171. data/lib/kotoshu/cache/base_cache.rb +778 -0
  172. data/lib/kotoshu/cache/cache.rb +91 -0
  173. data/lib/kotoshu/cache/eviction_policy.rb +66 -0
  174. data/lib/kotoshu/cache/frequency_cache.rb +283 -0
  175. data/lib/kotoshu/cache/language_cache.rb +511 -0
  176. data/lib/kotoshu/cache/lookup_cache.rb +166 -0
  177. data/lib/kotoshu/cache/model_cache.rb +1147 -0
  178. data/lib/kotoshu/cache/model_registry.rb +109 -0
  179. data/lib/kotoshu/cache/staged_languages.rb +45 -0
  180. data/lib/kotoshu/cache/suggestion_cache.rb +113 -0
  181. data/lib/kotoshu/cache.rb +49 -0
  182. data/lib/kotoshu/cli/auto_setup.rb +71 -0
  183. data/lib/kotoshu/cli/baseline_command.rb +119 -0
  184. data/lib/kotoshu/cli/batch_reporter.rb +321 -0
  185. data/lib/kotoshu/cli/cache_command.rb +282 -0
  186. data/lib/kotoshu/cli/completions_command.rb +210 -0
  187. data/lib/kotoshu/cli/dict_command.rb +64 -0
  188. data/lib/kotoshu/cli/directory_check.rb +146 -0
  189. data/lib/kotoshu/cli/directory_walker.rb +132 -0
  190. data/lib/kotoshu/cli/display_formatter.rb +429 -0
  191. data/lib/kotoshu/cli/errors.rb +36 -0
  192. data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
  193. data/lib/kotoshu/cli/interactive_reviewer.rb +314 -0
  194. data/lib/kotoshu/cli/language_resolver.rb +91 -0
  195. data/lib/kotoshu/cli/navigation_manager.rb +273 -0
  196. data/lib/kotoshu/cli/personal_command.rb +113 -0
  197. data/lib/kotoshu/cli/progress_reporter.rb +114 -0
  198. data/lib/kotoshu/cli/status_report.rb +130 -0
  199. data/lib/kotoshu/cli.rb +880 -0
  200. data/lib/kotoshu/components/passthrough_spell_checker.rb +70 -0
  201. data/lib/kotoshu/components/pos_tagger.rb +98 -0
  202. data/lib/kotoshu/components/spell_checker.rb +73 -0
  203. data/lib/kotoshu/components/synthesizer.rb +60 -0
  204. data/lib/kotoshu/components/tokenizer.rb +58 -0
  205. data/lib/kotoshu/components/whitespace_tokenizer.rb +94 -0
  206. data/lib/kotoshu/components.rb +13 -0
  207. data/lib/kotoshu/configuration/builder.rb +207 -0
  208. data/lib/kotoshu/configuration/resolver.rb +124 -0
  209. data/lib/kotoshu/configuration.rb +829 -0
  210. data/lib/kotoshu/conformance_exporter.rb +256 -0
  211. data/lib/kotoshu/conformance_runner.rb +176 -0
  212. data/lib/kotoshu/core/exceptions.rb +186 -0
  213. data/lib/kotoshu/core/indexed_dictionary.rb +293 -0
  214. data/lib/kotoshu/core/models/affix_rule.rb +262 -0
  215. data/lib/kotoshu/core/models/result/document_result.rb +132 -0
  216. data/lib/kotoshu/core/models/result/word_result.rb +131 -0
  217. data/lib/kotoshu/core/models/word.rb +131 -0
  218. data/lib/kotoshu/core/trie/builder.rb +119 -0
  219. data/lib/kotoshu/core/trie/node.rb +94 -0
  220. data/lib/kotoshu/core/trie/trie.rb +249 -0
  221. data/lib/kotoshu/core.rb +28 -0
  222. data/lib/kotoshu/data/common_words/de.yml +1800 -0
  223. data/lib/kotoshu/data/common_words/en.yml +1215 -0
  224. data/lib/kotoshu/data/common_words/es.yml +750 -0
  225. data/lib/kotoshu/data/common_words/fr.yml +1015 -0
  226. data/lib/kotoshu/data/common_words/pt.yml +870 -0
  227. data/lib/kotoshu/data/common_words/ru.yml +484 -0
  228. data/lib/kotoshu/data/common_words_loader.rb +141 -0
  229. data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
  230. data/lib/kotoshu/data/frozen_kelly.rb +38 -0
  231. data/lib/kotoshu/data.rb +8 -0
  232. data/lib/kotoshu/data_structures/bloom_filter.rb +188 -0
  233. data/lib/kotoshu/data_structures.rb +8 -0
  234. data/lib/kotoshu/debug_logger.rb +146 -0
  235. data/lib/kotoshu/debug_mode.rb +136 -0
  236. data/lib/kotoshu/defaults.rb +84 -0
  237. data/lib/kotoshu/dictionaries/catalog.rb +814 -0
  238. data/lib/kotoshu/dictionaries.rb +8 -0
  239. data/lib/kotoshu/dictionary/base.rb +324 -0
  240. data/lib/kotoshu/dictionary/cspell.rb +243 -0
  241. data/lib/kotoshu/dictionary/custom.rb +225 -0
  242. data/lib/kotoshu/dictionary/hunspell.rb +520 -0
  243. data/lib/kotoshu/dictionary/plain_text.rb +302 -0
  244. data/lib/kotoshu/dictionary/repository.rb +246 -0
  245. data/lib/kotoshu/dictionary/unified.rb +236 -0
  246. data/lib/kotoshu/dictionary/unix_words.rb +218 -0
  247. data/lib/kotoshu/dictionary.rb +15 -0
  248. data/lib/kotoshu/documents/document.rb +120 -0
  249. data/lib/kotoshu/documents/plain_text_document.rb +73 -0
  250. data/lib/kotoshu/documents/source_position.rb +45 -0
  251. data/lib/kotoshu/documents/source_range.rb +76 -0
  252. data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
  253. data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
  254. data/lib/kotoshu/documents/suppressions.rb +44 -0
  255. data/lib/kotoshu/documents/text_node.rb +49 -0
  256. data/lib/kotoshu/documents.rb +178 -0
  257. data/lib/kotoshu/embeddings/embedding_pipeline.rb +238 -0
  258. data/lib/kotoshu/embeddings/lru_cache.rb +244 -0
  259. data/lib/kotoshu/embeddings/onnx_runtime_model.rb +411 -0
  260. data/lib/kotoshu/embeddings/protocol.rb +96 -0
  261. data/lib/kotoshu/embeddings/protocols.rb +23 -0
  262. data/lib/kotoshu/embeddings/registry.rb +184 -0
  263. data/lib/kotoshu/embeddings/search.rb +204 -0
  264. data/lib/kotoshu/embeddings/similarity_engine.rb +278 -0
  265. data/lib/kotoshu/embeddings/similarity_search.rb +328 -0
  266. data/lib/kotoshu/embeddings/vocabulary.rb +306 -0
  267. data/lib/kotoshu/embeddings.rb +87 -0
  268. data/lib/kotoshu/fluent_checker.rb +94 -0
  269. data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +48 -0
  270. data/lib/kotoshu/grammar/pattern_matchers/double_negative_matcher.rb +179 -0
  271. data/lib/kotoshu/grammar/pattern_matchers/phrase_matcher.rb +86 -0
  272. data/lib/kotoshu/grammar/pattern_matchers/possessive_context_matcher.rb +75 -0
  273. data/lib/kotoshu/grammar/pattern_matchers/possessive_contraction_matcher.rb +87 -0
  274. data/lib/kotoshu/grammar/pattern_matchers/sentence_start_matcher.rb +84 -0
  275. data/lib/kotoshu/grammar/pattern_matchers/vowel_sound_matcher.rb +81 -0
  276. data/lib/kotoshu/grammar/pattern_matchers/word_list_matcher.rb +66 -0
  277. data/lib/kotoshu/grammar/rule.rb +99 -0
  278. data/lib/kotoshu/grammar/rule_engine.rb +96 -0
  279. data/lib/kotoshu/grammar/rule_loader.rb +30 -0
  280. data/lib/kotoshu/grammar.rb +24 -0
  281. data/lib/kotoshu/integrity/audit_log.rb +167 -0
  282. data/lib/kotoshu/integrity/manifest.rb +116 -0
  283. data/lib/kotoshu/integrity/net_http.rb +51 -0
  284. data/lib/kotoshu/integrity/rotation_policy.rb +88 -0
  285. data/lib/kotoshu/integrity.rb +26 -0
  286. data/lib/kotoshu/jekyll.rb +101 -0
  287. data/lib/kotoshu/keyboard/layout.rb +116 -0
  288. data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
  289. data/lib/kotoshu/keyboard/layouts/azerty.rb +55 -0
  290. data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
  291. data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
  292. data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
  293. data/lib/kotoshu/keyboard/layouts/dvorak.rb +54 -0
  294. data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
  295. data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
  296. data/lib/kotoshu/keyboard/layouts/hrsl.rb +94 -0
  297. data/lib/kotoshu/keyboard/layouts/jcuken.rb +57 -0
  298. data/lib/kotoshu/keyboard/layouts/latin.rb +266 -0
  299. data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
  300. data/lib/kotoshu/keyboard/layouts/qwerty.rb +52 -0
  301. data/lib/kotoshu/keyboard/layouts/qwertz.rb +55 -0
  302. data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
  303. data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
  304. data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
  305. data/lib/kotoshu/keyboard/registry.rb +201 -0
  306. data/lib/kotoshu/keyboard.rb +96 -0
  307. data/lib/kotoshu/kotoshu_native.so +0 -0
  308. data/lib/kotoshu/language/detector.rb +242 -0
  309. data/lib/kotoshu/language/identifier.rb +376 -0
  310. data/lib/kotoshu/language/languages/base.rb +271 -0
  311. data/lib/kotoshu/language/lid_detector.rb +188 -0
  312. data/lib/kotoshu/language/normalizer/arabic.rb +113 -0
  313. data/lib/kotoshu/language/normalizer/base.rb +133 -0
  314. data/lib/kotoshu/language/normalizer/greek.rb +63 -0
  315. data/lib/kotoshu/language/normalizer/hebrew.rb +72 -0
  316. data/lib/kotoshu/language/normalizer/persian.rb +36 -0
  317. data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
  318. data/lib/kotoshu/language/registry.rb +209 -0
  319. data/lib/kotoshu/language/resources/ar/common_words.txt +6753 -0
  320. data/lib/kotoshu/language/resources/ar/confusion_sets.txt +11 -0
  321. data/lib/kotoshu/language/resources/de/common_words.txt +10003 -0
  322. data/lib/kotoshu/language/resources/de/confusion_sets.txt +246 -0
  323. data/lib/kotoshu/language/resources/en/common_words.txt +9979 -0
  324. data/lib/kotoshu/language/resources/en/confusion_sets.txt +871 -0
  325. data/lib/kotoshu/language/resources/es/common_words.txt +9992 -0
  326. data/lib/kotoshu/language/resources/es/confusion_sets.txt +17 -0
  327. data/lib/kotoshu/language/resources/fr/common_words.txt +9993 -0
  328. data/lib/kotoshu/language/resources/fr/confusion_sets.txt +76 -0
  329. data/lib/kotoshu/language/resources/pt/common_words.txt +9977 -0
  330. data/lib/kotoshu/language/resources/pt/confusion_sets.txt +18 -0
  331. data/lib/kotoshu/language/resources/ru/common_words.txt +9951 -0
  332. data/lib/kotoshu/language/resources/ru/confusion_sets.txt +5 -0
  333. data/lib/kotoshu/language/script.rb +128 -0
  334. data/lib/kotoshu/language/segmenter.rb +122 -0
  335. data/lib/kotoshu/language/suika.rb +38 -0
  336. data/lib/kotoshu/language/tokenizer/base.rb +187 -0
  337. data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
  338. data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
  339. data/lib/kotoshu/language/tokenizer/french_tokenizer.rb +172 -0
  340. data/lib/kotoshu/language/tokenizer/german_tokenizer.rb +41 -0
  341. data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
  342. data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
  343. data/lib/kotoshu/language/tokenizer/japanese_tokenizer.rb +55 -0
  344. data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +153 -0
  345. data/lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb +160 -0
  346. data/lib/kotoshu/language/tokenizer/russian_tokenizer.rb +93 -0
  347. data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
  348. data/lib/kotoshu/language/tokenizer/spanish_tokenizer.rb +122 -0
  349. data/lib/kotoshu/language.rb +127 -0
  350. data/lib/kotoshu/languages/ar/language.rb +98 -0
  351. data/lib/kotoshu/languages/bg/language.rb +37 -0
  352. data/lib/kotoshu/languages/ca/language.rb +22 -0
  353. data/lib/kotoshu/languages/cs/language.rb +22 -0
  354. data/lib/kotoshu/languages/da/language.rb +22 -0
  355. data/lib/kotoshu/languages/de/language.rb +554 -0
  356. data/lib/kotoshu/languages/el/language.rb +50 -0
  357. data/lib/kotoshu/languages/en/language.rb +470 -0
  358. data/lib/kotoshu/languages/es/language.rb +476 -0
  359. data/lib/kotoshu/languages/et/language.rb +24 -0
  360. data/lib/kotoshu/languages/fa/language.rb +91 -0
  361. data/lib/kotoshu/languages/fr/language.rb +505 -0
  362. data/lib/kotoshu/languages/he/language.rb +98 -0
  363. data/lib/kotoshu/languages/hr/language.rb +24 -0
  364. data/lib/kotoshu/languages/hu/language.rb +22 -0
  365. data/lib/kotoshu/languages/id/language.rb +25 -0
  366. data/lib/kotoshu/languages/it/language.rb +22 -0
  367. data/lib/kotoshu/languages/ja/language.rb +476 -0
  368. data/lib/kotoshu/languages/ko/language.rb +56 -0
  369. data/lib/kotoshu/languages/latin_base.rb +53 -0
  370. data/lib/kotoshu/languages/lt/language.rb +24 -0
  371. data/lib/kotoshu/languages/lv/language.rb +24 -0
  372. data/lib/kotoshu/languages/nb/language.rb +25 -0
  373. data/lib/kotoshu/languages/ne/language.rb +60 -0
  374. data/lib/kotoshu/languages/nl/language.rb +22 -0
  375. data/lib/kotoshu/languages/nn/language.rb +26 -0
  376. data/lib/kotoshu/languages/pl/language.rb +22 -0
  377. data/lib/kotoshu/languages/pt/language.rb +439 -0
  378. data/lib/kotoshu/languages/ro/language.rb +22 -0
  379. data/lib/kotoshu/languages/ru/language.rb +420 -0
  380. data/lib/kotoshu/languages/sk/language.rb +25 -0
  381. data/lib/kotoshu/languages/sl/language.rb +25 -0
  382. data/lib/kotoshu/languages/sr/language.rb +39 -0
  383. data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
  384. data/lib/kotoshu/languages/sv/language.rb +22 -0
  385. data/lib/kotoshu/languages/tr/language.rb +36 -0
  386. data/lib/kotoshu/languages/uk/language.rb +38 -0
  387. data/lib/kotoshu/languages/vi/language.rb +22 -0
  388. data/lib/kotoshu/languages.rb +64 -0
  389. data/lib/kotoshu/metrics_collector.rb +238 -0
  390. data/lib/kotoshu/metrics_module.rb +121 -0
  391. data/lib/kotoshu/models/context.rb +119 -0
  392. data/lib/kotoshu/models/embedding_model.rb +181 -0
  393. data/lib/kotoshu/models/fasttext_model.rb +212 -0
  394. data/lib/kotoshu/models/nearest_neighbor.rb +96 -0
  395. data/lib/kotoshu/models/onnx_model.rb +330 -0
  396. data/lib/kotoshu/models/semantic_error.rb +204 -0
  397. data/lib/kotoshu/models/suggestion.rb +106 -0
  398. data/lib/kotoshu/models/word_embedding.rb +97 -0
  399. data/lib/kotoshu/models.rb +27 -0
  400. data/lib/kotoshu/multi_language_checker.rb +127 -0
  401. data/lib/kotoshu/native.rb +83 -0
  402. data/lib/kotoshu/native_backend.rb +131 -0
  403. data/lib/kotoshu/paths.rb +79 -0
  404. data/lib/kotoshu/personal_dictionary.rb +105 -0
  405. data/lib/kotoshu/plugins/plugin.rb +61 -0
  406. data/lib/kotoshu/plugins/registry.rb +118 -0
  407. data/lib/kotoshu/plugins.rb +9 -0
  408. data/lib/kotoshu/project_config.rb +76 -0
  409. data/lib/kotoshu/readers/aff_data.rb +530 -0
  410. data/lib/kotoshu/readers/aff_reader.rb +393 -0
  411. data/lib/kotoshu/readers/condition_checker.rb +125 -0
  412. data/lib/kotoshu/readers/dic_reader.rb +208 -0
  413. data/lib/kotoshu/readers/file_reader.rb +377 -0
  414. data/lib/kotoshu/readers/lookup_builder.rb +584 -0
  415. data/lib/kotoshu/readers/readers.rb +17 -0
  416. data/lib/kotoshu/readers.rb +25 -0
  417. data/lib/kotoshu/resource_bundle.rb +55 -0
  418. data/lib/kotoshu/resource_manager.rb +600 -0
  419. data/lib/kotoshu/results/result.rb +165 -0
  420. data/lib/kotoshu/results.rb +8 -0
  421. data/lib/kotoshu/rspec.rb +163 -0
  422. data/lib/kotoshu/scripts/fasttext_to_onnx.py +275 -0
  423. data/lib/kotoshu/source_registry.rb +104 -0
  424. data/lib/kotoshu/spellchecker/parallel_checker.rb +90 -0
  425. data/lib/kotoshu/spellchecker.rb +406 -0
  426. data/lib/kotoshu/string_metrics.rb +153 -0
  427. data/lib/kotoshu/suggestions/context.rb +43 -0
  428. data/lib/kotoshu/suggestions/frequency_provider.rb +80 -0
  429. data/lib/kotoshu/suggestions/generator.rb +169 -0
  430. data/lib/kotoshu/suggestions/pipeline.rb +135 -0
  431. data/lib/kotoshu/suggestions/semantic_cascade.rb +73 -0
  432. data/lib/kotoshu/suggestions/strategies/base_strategy.rb +315 -0
  433. data/lib/kotoshu/suggestions/strategies/composite_strategy.rb +197 -0
  434. data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +631 -0
  435. data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +207 -0
  436. data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +165 -0
  437. data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +284 -0
  438. data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +317 -0
  439. data/lib/kotoshu/suggestions/strategies/symspell_strategy.rb +270 -0
  440. data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
  441. data/lib/kotoshu/suggestions/suggestion.rb +154 -0
  442. data/lib/kotoshu/suggestions/suggestion_set.rb +270 -0
  443. data/lib/kotoshu/suggestions/sweep_index.rb +137 -0
  444. data/lib/kotoshu/suggestions.rb +29 -0
  445. data/lib/kotoshu/tasks/check_task.rb +104 -0
  446. data/lib/kotoshu/tasks.rb +11 -0
  447. data/lib/kotoshu/validators/spelling_validator.rb +101 -0
  448. data/lib/kotoshu/validators.rb +9 -0
  449. data/lib/kotoshu/version.rb +5 -0
  450. data/lib/kotoshu.rb +538 -0
  451. data/rakelib/compile.rake +10 -0
  452. data/script/validate_all_dictionaries.rb +444 -0
  453. data/scripts/build_native_gem.rb +45 -0
  454. data/scripts/extract_eval_grids.rb +37 -0
  455. data/scripts/generate_frozen_tiers.rb +44 -0
  456. data/sig/kotoshu.rbs +435 -0
  457. data/test_oop.rb +79 -0
  458. metadata +546 -0
@@ -0,0 +1,186 @@
1
+ # 63 — Web playground (WASM)
2
+
3
+ Promotes T6.2 in `TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md`
4
+ into a promotable plan.
5
+
6
+ ## Goal
7
+
8
+ A browser-only Kotoshu playground at `play.kotoshu.dev` (or under
9
+ `kotoshu.github.io/playground/`) that:
10
+
11
+ - Loads instantly, no signup, no backend.
12
+ - Spell-checks user-pasted text in real time.
13
+ - Demonstrates the library's capabilities (suggestions, language
14
+ detection, document-format awareness) as a marketing surface for the
15
+ gem.
16
+
17
+ Built on [ruby.wasm](https://github.com/ruby/ruby.wasm), so the actual
18
+ Kotoshu Ruby code runs in the browser — not a JS port, not a server
19
+ round-trip.
20
+
21
+ ## Why
22
+
23
+ A playground is the canonical "try before you install" for developer
24
+ tools. Without one:
25
+
26
+ - Curious users have to `gem install`, set up a language, and write a
27
+ Ruby script just to see what Kotoshu does. Most bounce.
28
+ - The semantic path (the gem's standout feature) is invisible until
29
+ you've installed onnxruntime and a 4 GB model — a non-starter for
30
+ casual evaluation.
31
+
32
+ A WASM playground flips that: paste, see, decide. It also doubles as:
33
+
34
+ - **A bug-report reproducer** — link a playground URL with the failing
35
+ text in issues.
36
+ - **A feature showcase** for the marketing site (plan in
37
+ `kotoshu.github.io/TODO.impl/`).
38
+ - **A regression test surface** — snapshot outputs across releases.
39
+
40
+ ## Constraints (these shape every task)
41
+
42
+ - **No native extensions in the browser.** WASM Ruby cannot load C
43
+ extensions. This means:
44
+ - No `onnxruntime` — the semantic path is not available in-browser.
45
+ - No `suika` — Japanese tokenization (the dartsclone native ext) is
46
+ unavailable. Japanese either ships with a pre-tokenized dictionary
47
+ or is omitted from the playground.
48
+ - No `rubyzip` — bundle dictionaries unpacked.
49
+ - **Size budget.** ruby.wasm is ~3 MB. A full Hunspell dictionary is
50
+ ~5 MB compressed. A full FastText `.vec` is 2 GB. The playground
51
+ cannot ship the full corpus.
52
+ - **Cold start.** WASM Ruby instantiation takes ~1–2 s on a fast
53
+ machine, more on mobile. The UI must hide that behind a loading
54
+ state.
55
+
56
+ ## Tasks
57
+
58
+ ### Phase 1 — Minimum viable playground
59
+
60
+ 1. **Pick the dictionary subset.** Ship a top-50K-words-per-language
61
+ subset of the Hunspell `.dic` plus the full `.aff` (affixes are
62
+ small). 50K covers >99% of typical prose. Subset lives in the
63
+ `kotoshu.github.io` repo under `playground/dictionaries/<code>/`.
64
+ 2. **Language support.** Start with the six full-feature languages
65
+ (de, en, es, fr, pt, ru). Document that Japanese is omitted (no
66
+ suika in WASM); other languages follow as language modules land
67
+ (plan 04).
68
+ 3. **ruby.wasm packaging.** Build pipeline (Rakefile in the
69
+ `kotoshu.github.io` repo) that:
70
+ - Compiles a `kotoshu.wasm` with the gem baked in.
71
+ - Strips the gem's `onnxruntime`, `suika`, `rubyzip` require paths
72
+ via conditional loading (the library already soft-requires them —
73
+ verify WASM mode is treated like the "neither soft-dep installed"
74
+ case).
75
+ - Bundles the 50K-word dictionaries as gzipped JSON.
76
+ 4. **UI shell.** Single page:
77
+ - A textarea (or Monaco editor for syntax-highlighting) on the left.
78
+ - A diagnostics panel on the right, mirroring what the LSP server
79
+ emits (plan 60).
80
+ - A language selector (defaults to auto-detect).
81
+ - A "Format" toggle (plain / markdown / asciidoc) to demo document
82
+ parsing.
83
+ 5. **In-browser caching.** IndexedDB stores the WASM blob, the
84
+ dictionaries, and the user's personal dictionary. Subsequent visits
85
+ load from cache.
86
+
87
+ ### Phase 2 — Showcase features
88
+
89
+ 6. **Suggestion panel.** Clicking a flagged word shows ranked
90
+ suggestions with frequency-rank badges (from the Kelly lists, if
91
+ the playground ships them).
92
+ 7. **Language detection demo.** A visible "Detected: fr (95%
93
+ confidence)" badge using the pure-Ruby heuristic path
94
+ (`Language::Identifier` falls back to n-gram detection when the
95
+ FastText LID model is unavailable — verify this is what happens in
96
+ WASM, or ship the LID model if feasible).
97
+ 8. **Permalinks.** The textarea content is encoded in the URL hash
98
+ (compressed with `lz-string`) so users can share a reproduction
99
+ case.
100
+ 9. **"Get this in your editor" CTA.** A footer banner linking to the
101
+ install instructions (plan 61) for users who want it locally.
102
+
103
+ ### Phase 3 — Performance and reliability
104
+
105
+ 10. **Lazy dictionary load.** Don't ship all six dictionaries upfront;
106
+ fetch the relevant one on first language detection. Cache for
107
+ subsequent visits.
108
+ 11. **Web Worker.** Run Kotoshu in a Web Worker so spell-checking
109
+ doesn't block the UI thread. Debounce by 500 ms.
110
+ 12. **Offline support.** Service Worker caches the WASM + dictionaries
111
+ so the playground works without network after first load.
112
+ 13. **Memory ceiling.** Browsers tab-dismiss pages that exceed memory
113
+ budgets. Cap resident memory at ~100 MB; refuse to load a second
114
+ dictionary until the first is released.
115
+
116
+ ### Phase 4 — Operations
117
+
118
+ 14. **Hosting.** GitHub Pages on `kotoshu.github.io`. Subpath
119
+ `/playground/` or a custom subdomain via CNAME.
120
+ 15. **CI build.** On release of the library gem, a workflow rebuilds
121
+ the WASM and deploys. Version the playground to match the gem so
122
+ bug reports reference a known state.
123
+ 16. **Telemetry.** Privacy-respecting: count unique visits, top
124
+ languages used, top error patterns. No user content logged. Use
125
+ Plausible or self-hosted; never Google Analytics.
126
+
127
+ ## Acceptance criteria
128
+
129
+ - `https://kotoshu.github.io/playground/` loads in under 5 s on a
130
+ cold cache (3G simulation), under 1 s on a warm cache.
131
+ - Pasting English prose flags misspellings within 500 ms of typing
132
+ pause.
133
+ - Selecting a suggestion replaces the word in the editor.
134
+ - Sharing a permalink reproduces the exact state.
135
+ - The page works offline after first visit.
136
+ - Memory stays under 100 MB with one language loaded; under 200 MB
137
+ with three.
138
+
139
+ ## Dependencies
140
+
141
+ - **Blocked by:**
142
+ - `02-cli-unification` — the playground wraps the library API; a
143
+ clean surface is required
144
+ - `04-language-modules` — for the languages available in the
145
+ playground (only languages whose tokenizer + normalizer is pure
146
+ Ruby can ship in WASM)
147
+ - **Blocks:** nothing — this is a marketing/UX surface, not a
148
+ prerequisite for other features.
149
+ - **Cross-repo:** ships in `kotoshu/kotoshu.github.io`; consumes the
150
+ library gem's WASM build (which the library repo must produce — see
151
+ task 3).
152
+ - **Promotes from:** T6.2 in `56-t4-t5-t6-quality-architecture-ecosystem.md`
153
+
154
+ ## Out of scope
155
+
156
+ - The semantic path in-browser. No `onnxruntime` in WASM; revisit if
157
+ ONNX-runtime-on-WASM matures (it's nascent in 2026).
158
+ - A full FastText model (2 GB per language is not browser-feasible).
159
+ - User accounts or saved documents. The playground is for trying, not
160
+ for storing.
161
+ - A full dictionary (every word, every language). The 50K subset is a
162
+ deliberate tradeoff for size; users who need full coverage install
163
+ the gem.
164
+
165
+ ## Risks
166
+
167
+ - **ruby.wasm maturity.** ruby.wasm is actively developed; some stdlib
168
+ pieces behave differently (notably `IO`, `File`, `Thread`). Audit
169
+ the Kotoshu codepath for stdlib calls that don't translate and add
170
+ WASM-specific stubs.
171
+ - **Bundle size creep.** Easy to accidentally ship the full gem
172
+ (with onnxruntime glue code) — the build pipeline must tree-shake
173
+ aggressively. Set a CI gate on the WASM blob size.
174
+ - **Browser memory limits.** Mobile Safari is aggressive. Test on a
175
+ real iPhone before claiming "works on mobile."
176
+ - **Dictionary licensing.** Each Hunspell dictionary has its own
177
+ license. Verify the subset bundle is redist-compatible. The
178
+ `dictionaries/` repo already tracks per-language licenses — reuse
179
+ that metadata.
180
+
181
+ ## Status
182
+
183
+ _Pending._ Phase 1 is a coherent MVP — could ship before v1.0 of the
184
+ library as part of marketing buildup. The single biggest risk is
185
+ ruby.wasm compatibility; spike task (verify Kotoshu loads in ruby.wasm
186
+ at all) before committing to a deadline.
@@ -0,0 +1,191 @@
1
+ # 64 — HTTP API server & multi-language SDKs
2
+
3
+ Promotes the cross-ecosystem reach idea into a promotable plan. This is
4
+ the "make it big" path for users who don't run Ruby.
5
+
6
+ ## Goal
7
+
8
+ A self-hostable HTTP API that wraps the Kotoshu library, plus thin SDK
9
+ clients in Python, JavaScript/TypeScript, and Go so non-Ruby projects
10
+ can use Kotoshu without shelling out to the CLI.
11
+
12
+ ```
13
+ ┌────────────────┐ ┌─────────────────┐
14
+ │ Python app │ HTTPS │ │
15
+ │ JS app ├────────►│ kotoshu server │──► Kotoshu.check / .suggest
16
+ │ Go app │ │ (Rack/Sinatra) │
17
+ │ Rust app │ │ │
18
+ └────────────────┘ └─────────────────┘
19
+ ```
20
+
21
+ The server is a **separate gem** (`kotoshu-server`); SDKs are separate
22
+ repos under `kotoshu/`. The library gem stays focused on
23
+ spell-checking; the server gem holds the HTTP surface.
24
+
25
+ ## Why
26
+
27
+ Kotoshu is a Ruby library. The Ruby ecosystem is one of many. Without
28
+ an HTTP entry point:
29
+
30
+ - Python shops (Django, Flask) can't use Kotoshu without a Ruby
31
+ subprocess.
32
+ - JavaScript shops (Node, Deno, Bun) reach for `spellchecker` (node)
33
+ or `cspell` and never hear about Kotoshu.
34
+ - Go shops shell out and parse SARIF — works, but no one's first
35
+ choice.
36
+ - Enterprise stacks (Java/Kotlin, .NET) are simply unreachable from a
37
+ Ruby library.
38
+
39
+ An HTTP API + thin SDKs puts Kotoshu in the same shape as LanguageTool
40
+ (which ships a public HTTP API and is the de-facto standard for this)
41
+ without giving up the "self-hostable" guarantee that the
42
+ privacy-conscious require.
43
+
44
+ This is also the answer to "more use cases": with the API in place,
45
+ the use cases multiply without further Kotoshu work — a browser
46
+ extension, a Slack bot, a WordPress plugin, an Obsidian plugin all
47
+ become one HTTP call away.
48
+
49
+ ## Tasks
50
+
51
+ ### Phase 1 — HTTP server (`kotoshu-server` gem)
52
+
53
+ 1. **New repo `kotoshu/kotoshu-server`.** Rack-based (Sinatra for the
54
+ routing, or pure Rack for minimalism). Depends on `kotoshu`.
55
+ 2. **Endpoints.**
56
+ - `POST /v1/check` — body: `{ text, language?, format? }`. Returns
57
+ a `DocumentResult`-shaped JSON (mirrors the CLI's `--format json`).
58
+ - `POST /v1/suggest` — body: `{ word, language?, max? }`. Returns
59
+ a `SuggestionSet`-shaped JSON.
60
+ - `POST /v1/detect` — body: `{ text }`. Returns detected language
61
+ + confidence.
62
+ - `GET /v1/languages` — lists supported + cached languages.
63
+ - `GET /v1/health` — liveness probe; returns cache state.
64
+ - `GET /v1/version` — library + server versions.
65
+ 3. **Resource management.** On startup, the server runs
66
+ `Kotoshu.setup(*configured_languages, want: %i[spelling frequency])`
67
+ so the cache is warm before serving requests. New languages can be
68
+ added at runtime via `POST /v1/admin/setup` (gated behind an admin
69
+ token).
70
+ 4. **Semantic mode.** Optional: when ONNX is enabled (plan 05), a
71
+ `model: "hybrid"` query param turns on reranking for that request.
72
+ Default `model: "hunspell"` to bound latency.
73
+ 5. **Concurrency model.** Puma or Falcon (Falcon for async I/O —
74
+ better fit for the long-running suggestion pipeline). Multiple
75
+ workers; each worker owns its own `Spellchecker` instance (or
76
+ shares a thread-safe one — verify in `02-cli-unification`).
77
+ 6. **Rate limiting + auth.** Optional API key middleware (operator's
78
+ choice). Ship disabled by default; document how to enable.
79
+ 7. **Observability.** Structured JSON logs (one line per request);
80
+ Prometheus metrics endpoint at `/v1/metrics`. Latency histograms
81
+ for `check` and `suggest`.
82
+
83
+ ### Phase 2 — Docker & ops
84
+
85
+ 8. **`kotoshu/server` Docker image.** Multi-arch (amd64, arm64).
86
+ Pre-warmed with the six full-feature languages. Configurable via
87
+ `KOTOSHU_*` env vars.
88
+ 9. **Helm chart.** For Kubernetes deployments. Configurable replicas,
89
+ resource limits, language pre-warm list.
90
+ 10. **`docker-compose.yml`.** For local dev: one command (`docker
91
+ compose up`) gives a running server on `:9292`.
92
+ 11. **Benchmark suite.** `spec/performance/server_bench.rb`:
93
+ - Throughput: requests/sec on a 4-core machine (target: > 200 rps
94
+ for `check` with short text).
95
+ - Latency p50/p95/p99 for `check` and `suggest`.
96
+ - Memory under sustained load (target: < 2 GB resident with
97
+ `hunspell` mode, < 5 GB with `hybrid`).
98
+
99
+ ### Phase 3 — SDKs
100
+
101
+ 12. **`kotoshu/kotoshu-python`.** PyPI package `kotoshu`.
102
+ - Sync client (`requests`) and async client (`httpx`).
103
+ - Type stubs for IDE autocomplete.
104
+ - Methods: `check(text, language=None)`, `suggest(word, language=None)`,
105
+ `detect(text)`.
106
+ - Streaming client for `POST /v1/check` if/when streaming lands.
107
+ 13. **`kotoshu/kotoshu-js`.** npm package `@kotoshu/client`.
108
+ - Works in Node, Deno, Bun, and modern browsers (via `fetch`).
109
+ - TypeScript types.
110
+ - Same method surface as the Python SDK.
111
+ 14. **`kotoshu/kotoshu-go`.** Go module.
112
+ - `kotoshu.NewClient(url)` → `.Check(ctx, text, opts)` etc.
113
+ - Idiomatic Go (context.Context everywhere, typed errors).
114
+ 15. **SDK contract tests.** A shared OpenAPI spec drives generated
115
+ fixtures; each SDK's test suite runs against the fixtures so the
116
+ SDKs stay in sync with the server.
117
+
118
+ ### Phase 4 — Documentation & reach
119
+
120
+ 16. **OpenAPI spec** in the server repo, published to
121
+ [`openapis.org`](https://openapis.org) cataloging. Auto-generates
122
+ SDK clients in other languages (Rust, .NET, Java) via
123
+ openapi-generator — best-effort, community-supported.
124
+ 17. **Self-hosting guide.** Step-by-step for Docker, Kubernetes, bare
125
+ Ruby. Covers sizing, language pre-warm, and monitoring.
126
+ 18. **Privacy guarantee.** Document that self-hosted = your data stays
127
+ in your network. Contrast with LanguageTool's public API.
128
+ 19. **Migrating from LanguageTool API.** Side-by-side: same input,
129
+ different response shape; provide a shim if there's demand.
130
+
131
+ ## Acceptance criteria
132
+
133
+ - `docker run -p 9292:9292 kotoshu/server` serves `POST /v1/check`
134
+ within 5 s of container start.
135
+ - Throughput > 200 rps for `check` (short text) on a 4-core machine.
136
+ - Each SDK has a one-page quickstart that works against a running
137
+ server.
138
+ - All SDKs pass contract tests against the latest server release.
139
+ - OpenAPI spec validates; auto-generated Rust client compiles.
140
+ - Helm chart deploys to a test cluster and serves requests.
141
+
142
+ ## Dependencies
143
+
144
+ - **Blocked by:**
145
+ - `02-cli-unification` — clean library API to wrap
146
+ - `03-dynamic-download` — resource management for the server
147
+ - `05-semantic-path` — for the `hybrid` mode option
148
+ - `09-integrity-security` — for safe download + verification in
149
+ server-managed cache
150
+ - **Blocks:** `11-release-v1` (the server + SDKs are part of the v1
151
+ reach story).
152
+ - **Cross-repo:** new repos `kotoshu/kotoshu-server`,
153
+ `kotoshu/kotoshu-python`, `kotoshu/kotoshu-js`, `kotoshu/kotoshu-go`.
154
+ - **Promotes from:** implied by "every project, every language" in
155
+ `00-vision.md`.
156
+
157
+ ## Out of scope
158
+
159
+ - A managed / hosted API (operator concern; document self-hosting
160
+ instead — privacy is the differentiator).
161
+ - Multi-tenant features (per-tenant dictionaries, per-tenant rate
162
+ limits). Operators build that on top if needed.
163
+ - Real-time streaming of partial check results (defer to a v2 if
164
+ there's demand).
165
+ - Auth provider integrations (OAuth, OIDC). Ship an API-key
166
+ middleware; operators wire their own auth in front.
167
+ - A Rust SDK in-repo (the OpenAPI-generated client is best-effort;
168
+ accept community contributions if a maintainer steps up).
169
+
170
+ ## Risks
171
+
172
+ - **Operational burden.** A Docker image + Helm chart is a maintenance
173
+ surface. Decide upfront whether the project commits to LTS-style
174
+ releases of the server, or treats it as community-maintained.
175
+ - **API stability.** v1 endpoints must be stable. Freeze the JSON
176
+ shapes before tagging `kotoshu-server` v1.0; thereafter breaking
177
+ changes go to `/v2/*`.
178
+ - **Cold-start cost.** Pre-warming six languages on container start
179
+ means slow boot. Tradeoff: eager warm (fast requests, slow boot) vs
180
+ lazy (fast boot, slow first request). Default eager; document
181
+ `KOTOSHU_SERVER_LAZY=1` for the other mode.
182
+ - **SDK drift.** Three SDKs × ongoing server changes is real work.
183
+ The OpenAPI-generated contract tests (task 15) are the only sane
184
+ way to keep them aligned; budget for maintaining the spec.
185
+
186
+ ## Status
187
+
188
+ _Pending._ Phase 1 + 2 is a coherent v0.1; phase 3 SDKs are v0.2; v1.0
189
+ is the API-freeze milestone. The single biggest risk is the operational
190
+ commitment — confirm with maintainers that the project is willing to
191
+ ship and support a server image before starting phase 2.
@@ -0,0 +1,251 @@
1
+ # 65 — Universal Kotoshu: master plan & roadmap
2
+
3
+ Polyglot engine · tiered micro-models · distribution everywhere.
4
+ This is the master plan for "full success": a spell-checking system
5
+ **every user can install in under two commands, on every platform,
6
+ for every language they write** — developers, writers, CI, enterprise,
7
+ and eventually office-suite users.
8
+
9
+ Supersedes the scale of `00-vision.md` (which remains the statement of
10
+ philosophy); this file is the execution roadmap that reaches it.
11
+
12
+ ---
13
+
14
+ ## 1. Grounding — what we know today
15
+
16
+ | Fact | Source | Implication |
17
+ |---|---|---|
18
+ | Microsoft Word ships **14 MB** of fluency models: one `word_fluency_v2.onnx` + a 168-byte runtime-config JSON, code-signed, on onnxruntime | `/Applications/Microsoft Word.app/Contents/Resources/WordModels.bundle` | Proof that a world-class fluency model is a **single-digit-MB ONNX artifact**. We already run onnxruntime. Same distribution pattern applies 1:1. |
19
+ | Our converted models are **~120 MB/lang** (`.onnx` + vocab JSON), 9 languages converted, sha256 manifest exists | `models-fasttext-onnx/manifest.json` | 10× too heavy for bundling. Needs vocab pruning + dim reduction + int8 quantization → tiers. |
20
+ | ~98 Hunspell dictionaries staged, each dir already has `license` + `package.json` | `dictionaries/` | License metadata foundation exists; must become machine-readable + redistribution-classified. |
21
+ | Gem already downloads → sha256-verifies → caches with TTL, cache-only hot path | `LanguageCache` / `FrequencyCache` / `ModelCache` | The resource *client* is done. What's missing is the resource *contract* (tiers, licenses, lockfiles) and the *small artifacts*. |
22
+ | 12 repos live: gem 0.6.0, lsp/server 0.1.0, go v0.1.0, action v1, 3 SDKs, docker, 3 content repos, site | RubyGems / Go proxy / GitHub | The surfaces exist. The gap is engine portability + model tiers + packaging depth. |
23
+
24
+ **North star.** *Any user, any platform, any language: install in ≤2
25
+ commands, work fully offline, and get tiered accuracy they can choose —
26
+ from a 1 MB dictionary to a 15 MB fluency model to a research-grade
27
+ full model.*
28
+
29
+ ### Success metrics (measurable, per cut)
30
+
31
+ - **Languages**: 6 full-feature today → 30 wired at v1 → 98 at v2;
32
+ mini+fluency tiers published for ≥30 languages.
33
+ - **Accuracy**: fluency tier within 2 % top-1 of the full tier on the
34
+ benchmark harness; scores published per language × tier.
35
+ - **Size/time**: editor bundle < 25 MB; cold start < 5 s; CI cache hit
36
+ = zero downloads; air-gapped install = one tarball + one command.
37
+ - **Portability**: conformance suite green on every engine
38
+ implementation, on linux/amd64, linux/arm64, macOS universal,
39
+ windows x64/arm64, and wasm32.
40
+
41
+ ---
42
+
43
+ ## 2. Five pillars
44
+
45
+ ### Pillar 1 — Resource Spec v1 + registry (the contract)
46
+
47
+ One versioned format that every repo, cache, and SDK speaks:
48
+
49
+ ```yaml
50
+ # kotoshu-resource-spec v1 (per language package)
51
+ language: en
52
+ version: 2026.09.1
53
+ license: { name: BSD-3-Clause, redistribution: allowed } # or download-only
54
+ resources:
55
+ spelling: { path: spelling/index.dic, sha256: …, size: 551_762 }
56
+ frequency: { path: frequency/tiers.json, sha256: … }
57
+ model:
58
+ mini: { path: models/mini.onnx, sha256: …, size: ~2_000_000,
59
+ metrics: { top1: 0.91, recall5: 0.98 } }
60
+ fluency: { path: models/fluency.onnx, sha256: …, size: ~14_000_000,
61
+ metrics: { top1: 0.95, recall5: 0.99 } }
62
+ full: { path: models/full.onnx, sha256: …, size: ~120_000_000 }
63
+ signature: cosign …
64
+ ```
65
+
66
+ - **New repo `kotoshu/registry`**: one signed index aggregating
67
+ dictionaries + frequency + models manifests; lockfile support
68
+ (`kotoshu.lock`) for reproducible offline installs.
69
+ - License field is load-bearing: `redistribution: allowed` gates what
70
+ may be **bundled** into images/extensions vs download-only.
71
+ - Gem change: `ResourceManager` resolves **tiers** (`want: [:spelling,
72
+ model: :mini]`); `kotoshu export en --tiers mini --out en.tar.zst`
73
+ and `kotoshu install ./en.tar.zst` close the air-gap loop.
74
+
75
+ ### Pillar 2 — Tiered micro-models + model factory
76
+
77
+ | Tier | Size target | Contents | Purpose |
78
+ |---|---|---|---|
79
+ | **0 · dict** | 0.5–5 MB | Hunspell dic/aff + frequency tiers | Correctness; all 98 languages; no runtime |
80
+ | **1 · mini** | 1–3 MB | char-ngram-only embeddings, 32–64 d, int8 | OOV vectors + reranking where no model ships today |
81
+ | **2 · fluency** | 5–15 MB | pruned-vocab (top ~200 k) + dim-reduced + int8 embeddings, optional tiny context head | The Word-class tier: context-aware rerank, desert/dessert disambiguation |
82
+ | **3 · full** | 100–120 MB+ | today's `models-fasttext-onnx` artifacts | Research-grade; opt-in |
83
+
84
+ - **New repo `kotoshu/model-factory`**: pipeline `sources → prune →
85
+ reduce → quantize → eval → sign → publish`. Sources: cc/wiki fastText
86
+ vectors, Kelly + OPUS frequencies, dictionaries wordlists.
87
+ Reuses `scripts/convert_fasttext_to_onnx.py` from
88
+ models-fasttext-onnx as the `full` tier path.
89
+ - **Eval harness inside model-factory** (eval/): per-language benchmark
90
+ suites seeded from the Spylls/Hunspell fixtures already ported in the
91
+ gem, plus Wikipedia-typo corpora where available. Metrics: top-1,
92
+ recall@5, MRR, latency, size. **Gate**: a tier only publishes with
93
+ metrics attached; fluency must be within 2 % top-1 of full.
94
+ - Site gains an **accuracy page**: language × tier matrix with sizes
95
+ and scores — honesty as a feature.
96
+ - `auto` tier selection: engine picks the best tier fitting a
97
+ caller-supplied size budget (editor default 15 MB; server default full).
98
+
99
+ ### Pillar 3 — Polyglot engine: one Rust core, feature-gated bindings (decision made)
100
+
101
+ Two meanings of "language" — both are goals:
102
+ - **Checked natural languages**: 98 staged (Pillar 1/2 supply them).
103
+ - **Implementation languages**: TS, Ruby, Python, Rust — decided; this is
104
+ the **parsanol blueprint**, copied wholesale because it works in
105
+ production at Ribose today (`parsanol-rs` + `parsanol-ruby`).
106
+
107
+ **Architecture (parsanol model):**
108
+
109
+ - **New repo `kotoshu/kotoshu-rs`** — Cargo workspace; core crate is a
110
+ pure `rlib` (dic/aff parsing, affix expansion, trie/DAFSA lookup,
111
+ suggestion strategies, rerank math, tokenization, resource verify).
112
+ All FFI lives **inside the core** under `ffi/`, behind features:
113
+ `ruby` (magnus), `wasm` (wasm-bindgen + js-sys + panic hook), and a
114
+ **C ABI (`ffi/c.rs`) always available** — Python and any C-calling
115
+ host bind through it. `ffi/shared.rs` defines the batch serialization
116
+ every binding shares (parsanol's 3–5× FFI speedup pattern).
117
+ - **Language packages are thin cdylib shims in their own repos**:
118
+ `kotoshu` gem gains `ext/kotoshu_native` (≈10 lines:
119
+ `#[magnus::init]` calling `kotoshu::ffi::ruby::init`), depending on
120
+ the core via git `features=["ruby"]`. Python adds a PyO3 shim wheel
121
+ (maturin) over the same core; TS gets `@kotoshu/wasm` built from the
122
+ core's `wasm` feature (Node *and* browser); Rust consumes the core
123
+ directly. Each package versions independently (parsanol ships gem
124
+ 1.3.0 over core 0.5.1 — same policy).
125
+ - **Pure Ruby stays as the fallback and the semantic reference**: the
126
+ gem works without the ext (`Kotoshu::Native.available?` guards), and
127
+ the existing rspec suite runs against **both backends**
128
+ (`rake compat:ruby` / `compat:native`, parsanol's compat-namespace
129
+ pattern) — the gem's suite *is* the conformance suite.
130
+ - **ONNX: `ort` (pyke/ort v2), not the abandoned `onnxruntime` crate.**
131
+ Policy: inference stays host-injected via an `EmbeddingProvider`
132
+ trait; the core's own optional provider (`feature = "onnx"`) uses
133
+ **`ort` with `load-dynamic`** so it dlopens the libonnxruntime the
134
+ host already ships (Ruby's `onnxruntime` gem, pip's onnxruntime,
135
+ onnxruntime-node) — one runtime per process, no double linking, tiny
136
+ artifacts. Standalone Rust builds (CLI/LSP binaries) use ort's
137
+ bundled binaries.
138
+ - **Distribution (parsanol's low-maintenance path)**: source gem that
139
+ compiles on `gem install` via rb-sys (`create_rust_makefile`,
140
+ stable-API fallback for ruby-head, optional auto toolchain install);
141
+ prebuilt platform gems via rake-compiler-dock only if source builds
142
+ hurt adoption. Core releases automated with release-plz.
143
+ - Copy list from parsanol, as policy: git-rev `[patch.crates-io]` for
144
+ magnus/rb-sys when crates.io lags; `[net] git-fetch-with-cli`; wasm
145
+ rustflags for getrandom; cargo-deny + typos + pre-commit; core
146
+ profile `lto="fat", codegen-units=1`, ext profile `lto="off"`; core
147
+ CI includes `ruby-ffi.yml` and `wasm.yml` so every binding is tested
148
+ from the core repo.
149
+
150
+ ### Pillar 4 — Distribution matrix (install in ≤2 commands, everywhere)
151
+
152
+ | Channel | Package | Phase |
153
+ |---|---|---|
154
+ | RubyGems | `kotoshu`, `-lsp`, `-server` | live |
155
+ | Go proxy | `kotoshu-go` | live |
156
+ | GitHub Action / Docker | `action-kotoshu`, CI image → GHCR | live → GHCR at 0.7 |
157
+ | PyPI / npm (SDKs) | `kotoshu`, `@kotoshu/client` | 0.7 (tokens pending) |
158
+ | npm `@kotoshu/wasm` + jsDelivr CDN | browser engine, plays into playground & embeds | 0.8 |
159
+ | VS Code Marketplace + Open VSX, JetBrains | editor extensions bundling fluency tiers (self-contained, <25 MB, no gem install) | 0.9–1.0 |
160
+ | Homebrew tap / winget / choco / scoop / apt via OBS | `kotoshu` CLI + standalone LSP binaries | 1.0–1.1 |
161
+ | OCI (GHCR) | server image + resource bundles as artifacts | 1.0 |
162
+ | Offline bundles | `kotoshu export/install` tarballs, cosign-signed, sbom (CycloneDX), SLSA provenance | 0.7 → 1.0 |
163
+
164
+ Standalone LSP at 0.9 via traveling-ruby/ruby-packer (single binary,
165
+ 3 platforms) so editor users never need Ruby; native core later makes
166
+ this trivially static.
167
+
168
+ ### Pillar 5 — Audiences & end-to-end journeys
169
+
170
+ | Audience | Need | Surface (→ phase) |
171
+ |---|---|---|
172
+ | Ruby devs | library API | gem (live) |
173
+ | Python/JS/Go devs | typed client | SDKs + server (live → 1.0 at 0.9) |
174
+ | Any-stack devs | no SDK needed | HTTP API + OpenAPI (live) |
175
+ | Editor writers | instant offline squiggles | LSP (live) → **bundled-fluency extension** (0.9–1.0) |
176
+ | CI maintainers | fail builds nicely | Action + Docker (live, mini-tier bake at 0.7) |
177
+ | Enterprise / air-gapped | offline, signed, auditable | export/install bundles + SBOM (0.7 → 1.0) |
178
+ | Multi-lingual writers | every language on demand | 30 wired (1.0) → 98 (2.0) |
179
+ | Office-suite users | proofing like Word | Word add-in / LibreOffice ext exploration (2.0 gate) |
180
+ | Low-resource language communities | contribute a language | dictionaries + model-factory recipes (1.0+) |
181
+
182
+ **Golden journeys** (each must be testable e2e): A) VS Code user
183
+ installs extension → squiggles offline in <25 MB. B) `gem install
184
+ kotoshu && kotoshu setup en` → check. C) CI run with cache hit → SARIF
185
+ annotations, zero downloads. D) air-gapped host: `kotoshu install
186
+ en.tar.zst` after `cosign verify`. E) browser playground running the
187
+ WASM engine on mini/fluency tiers. F) polyglot team: `docker run
188
+ kotoshu/server` + `pip install kotoshu`.
189
+
190
+ **Platform/arch matrix**: linux amd64/arm64, macOS universal2,
191
+ windows x64/arm64, wasm32-wasi (browser); CI builds all; native core
192
+ adds riscv64/mobile for free.
193
+
194
+ ---
195
+
196
+ ## 3. Repo-by-repo deltas
197
+
198
+ | Repo | Change |
199
+ |---|---|
200
+ | **new** `kotoshu/kotoshu-rs` | the Rust core workspace (see plan 66): pure-rlib engine, `ffi/{c,ruby,wasm}` feature-gated, conformance CI (`ruby-ffi.yml`, `wasm.yml`), release-plz, builds `@kotoshu/wasm` |
201
+ | `kotoshu` (gem) | gains `ext/kotoshu_native` cdylib shim + rb-sys extconf; pure-Ruby stays fallback; `rake compat:{ruby,native}` runs the suite on both backends; plus tier-aware ResourceManager, export/install bundles, lockfile, `auto` tier |
202
+ | `dictionaries` | machine-readable license manifest (name, redistribution flag) — already has per-lang files; aggregate to registry |
203
+ | `frequency-list-kelly` | no change; feed model-factory |
204
+ | `models-fasttext-onnx` | becomes the **full tier** source feeding model-factory; keep publishing |
205
+ | **new** `kotoshu/registry` | signed index, lockfiles, tier metadata, accuracy scores |
206
+ | **new** `kotoshu/model-factory` | prune/reduce/quantize/eval/sign/publish pipeline + benchmark suites |
207
+ | **new** `kotoshu/conformance` | shared golden-vector pack (extracted from the gem's Spylls fixtures); consumed by kotoshu-rs CI and any future implementation |
208
+ | `kotoshu-python` | adds native wheel (PyO3 shim over the core, maturin) alongside the HTTP client |
209
+ | `kotoshu-js` | adds `@kotoshu/wasm` re-export + types (built in kotoshu-rs); HTTP client stays |
210
+ | `kotoshu-server` | tier param on endpoints; resource listing; auth/metrics/Helm at GA |
211
+ | `kotoshu-lsp` | tier selection config; standalone single-binary distribution from the Rust core at 0.9 |
212
+ | `docker-kotoshu-ci` | publish to GHCR; bake mini tier (image stays <300 MB) |
213
+ | `action-kotoshu` | prewarm-mini default |
214
+ | `kotoshu.github.io` | accuracy page, downloads page, journey docs |
215
+
216
+ ---
217
+
218
+ ## 4. Roadmap (cuts)
219
+
220
+ | Cut | Theme | Key deliverables | Exit criteria |
221
+ |---|---|---|---|
222
+ | **0.7** | Resource Spec + mini tier + core scaffold | spec v1; `registry` repo; license manifest; model-factory alpha producing **mini** for 6 langs; gem tier-aware cache + export/install; GHCR image; SDK registry pushes; **`kotoshu-rs` workspace scaffolded with C ABI + conformance-vector pack v0** | mini tier installable for 6 langs; bundle verified offline; Docker <300 MB; core passes conformance v0 vectors |
223
+ | **0.8** | Fluency tier + core engine | **fluency** (≈14 MB) for 6 langs with eval gates; accuracy page; **kotoshu-rs ports dic/aff + lookup + suggest strategies, conformance green**; `@kotoshu/wasm` published from the core (resolves plan 63 — playground goes client-side) | fluency within 2 % top-1 of full; core matches Ruby backend on the whole suite; playground runs on wasm |
224
+ | **0.9** | Polyglot GA + editors | server 1.0 (auth/metrics/Helm); SDKs 1.0 with contract tests; **gem ships `ext/kotoshu_native` (both backends green); Python native wheel**; **VS Code extension** bundling fluency; **standalone single-binary LSP/CLI from the Rust core** (3 platforms) | dual-backend suite green; extension works offline <25 MB; single binary passes conformance |
225
+ | **1.0** | Kotoshu for developers | 30 languages wired; Neovim/JetBrains distributions; Action GA; cosign+SBOM on all artifacts; docs complete; Homebrew tap + winget | every journey A–F green; 5-signature quality bar met |
226
+ | **1.1–2.0** | Everyone | prebuilt platform gems (rake-compiler-dock) if source installs hurt; 98 languages; **gate:** office-suite add-ins exploration; mobile via wasm/native; auto-tier under size budgets | any user, any platform, any language, ≤2 commands |
227
+
228
+ **Decision gates** (explicit, not defaults): prebuilt-vs-source gems
229
+ (post-1.0, adoption-driven); office add-ins (2.0, only after a design
230
+ spike proves the API story). The native-core question is **decided**
231
+ (plan 66) — WASM and Ruby ext both come from the same `kotoshu-rs`.
232
+
233
+ ---
234
+
235
+ ## 5. Risks
236
+
237
+ | Risk | Mitigation |
238
+ |---|---|
239
+ | License heterogeneity blocks bundling | redistribution flag in spec; bundle only `allowed`; download-only path stays |
240
+ | Small models lose accuracy | eval gates with published metrics; tiers never silently replace full |
241
+ | Two engines (Ruby + Rust) drift | the gem's suite runs on **both backends** (`rake compat:*`); shared conformance-vector pack in kotoshu-rs CI; pure Ruby remains the reference |
242
+ | Source gem needs a Rust toolchain to install | rb-sys optional auto-install; pure-Ruby fallback always works; prebuilt platform gems gated at post-1.0 |
243
+ | magnus/rb-sys rev churn (Ruby 4.0 prep) | parsanol's patch policy copied: pin git revs in `[patch.crates-io]`, relax when crates.io catches up |
244
+ | ort dynamic-lib version skew with host onnxruntime | `load-dynamic` against onnxruntime 1.x C API (stable); document minimum; standalone Rust uses bundled ort |
245
+ | Registry uptime | GitHub Releases + CDN + lockfiles; caches already tolerate outage |
246
+ | Scope explosion | cuts with exit criteria; gates before big bets; 30-before-98 languages |
247
+
248
+ ## Status
249
+
250
+ _Planning._ This document is the master index; per-cut plans get their
251
+ own `TODO.impl/{n}-*.md` when promoted, per house convention.