kotoshu 1.0.2-aarch64-linux

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (458) hide show
  1. checksums.yaml +7 -0
  2. data/.kotoshu-baseline.json +1 -0
  3. data/.pre-commit-hooks.yaml +18 -0
  4. data/.rspec +3 -0
  5. data/.rubocop.yml +351 -0
  6. data/.yardopts +8 -0
  7. data/CHANGELOG.md +813 -0
  8. data/CLAUDE.md +176 -0
  9. data/CODE_OF_CONDUCT.md +132 -0
  10. data/Cargo.lock +359 -0
  11. data/Cargo.toml +15 -0
  12. data/LICENSE +31 -0
  13. data/README.adoc +1292 -0
  14. data/Rakefile +150 -0
  15. data/SECURITY.md +93 -0
  16. data/TODO.impl/00-cut-0.1.md +121 -0
  17. data/TODO.impl/00-cut-0.2.md +175 -0
  18. data/TODO.impl/00-vision.md +281 -0
  19. data/TODO.impl/01-hunspell-correctness.md +111 -0
  20. data/TODO.impl/02-cli-unification.md +74 -0
  21. data/TODO.impl/03-dynamic-download.md +107 -0
  22. data/TODO.impl/04-language-modules.md +89 -0
  23. data/TODO.impl/05-semantic-path.md +83 -0
  24. data/TODO.impl/06-cjk-support.md +105 -0
  25. data/TODO.impl/07-rtl-support.md +71 -0
  26. data/TODO.impl/08-grammar-engine.md +106 -0
  27. data/TODO.impl/09-integrity-security.md +90 -0
  28. data/TODO.impl/10-testing-ci.md +98 -0
  29. data/TODO.impl/100-full-feature-batch3.md +28 -0
  30. data/TODO.impl/101-int4-tier-experiment.md +32 -0
  31. data/TODO.impl/102-lid-chain.md +28 -0
  32. data/TODO.impl/103-oov-bucket-tables.md +27 -0
  33. data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
  34. data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
  35. data/TODO.impl/106-native-lid-detection.md +36 -0
  36. data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
  37. data/TODO.impl/108-ko-ne-full-feature.md +37 -0
  38. data/TODO.impl/109-ten-audit.md +26 -0
  39. data/TODO.impl/11-release-v1.md +95 -0
  40. data/TODO.impl/110-sr-latn-and-polish.md +33 -0
  41. data/TODO.impl/111-embedding-bakeoff.md +33 -0
  42. data/TODO.impl/112-deployability-and-gating.md +26 -0
  43. data/TODO.impl/113-language-packs.md +26 -0
  44. data/TODO.impl/114-typo-biencoder-thread.md +33 -0
  45. data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
  46. data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
  47. data/TODO.impl/117-deterministic-frequency-tiers.md +110 -0
  48. data/TODO.impl/118-server-gem-floor.md +33 -0
  49. data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
  50. data/TODO.impl/120-setup-metadata-refresh.md +35 -0
  51. data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
  52. data/TODO.impl/122-pack-compression-kpk2.md +66 -0
  53. data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
  54. data/TODO.impl/124-owner-residue.md +19 -0
  55. data/TODO.impl/125-consumer-floor-guards.md +43 -0
  56. data/TODO.impl/126-tier-error-budgets.md +33 -0
  57. data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
  58. data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
  59. data/TODO.impl/129-pt-fr-corpora.md +23 -0
  60. data/TODO.impl/130-news-truth-0912.md +23 -0
  61. data/TODO.impl/131-hybrid-productization.md +42 -0
  62. data/TODO.impl/132-techniques-page-and-prose.md +27 -0
  63. data/TODO.impl/133-rust-first-distribution.md +68 -0
  64. data/TODO.impl/20-encapsulation-cleanup.md +47 -0
  65. data/TODO.impl/21-autoload-migration.md +64 -0
  66. data/TODO.impl/22-url-pin-single-source-of-truth.md +120 -0
  67. data/TODO.impl/23-fix-url-defaults.md +57 -0
  68. data/TODO.impl/24-cli-auto-setup-prompt.md +76 -0
  69. data/TODO.impl/25-soft-onnxruntime-require.md +75 -0
  70. data/TODO.impl/26-onnx-vocab-json-generation.md +96 -0
  71. data/TODO.impl/27-end-to-end-smoke-spec.md +113 -0
  72. data/TODO.impl/28-readme-quickstart-verification.md +68 -0
  73. data/TODO.impl/29-status-command.md +84 -0
  74. data/TODO.impl/30-language-auto-detection.md +74 -0
  75. data/TODO.impl/31-download-progress.md +58 -0
  76. data/TODO.impl/32-sarif-output-verification.md +72 -0
  77. data/TODO.impl/33-audit-log-rotation.md +54 -0
  78. data/TODO.impl/34-cache-eviction.md +59 -0
  79. data/TODO.impl/35-shell-completion.md +61 -0
  80. data/TODO.impl/36-cleanup-regressions-0.3.1.md +134 -0
  81. data/TODO.impl/37-hunspell-correctness-tier2.md +203 -0
  82. data/TODO.impl/38-onnx-semantic-gating.md +218 -0
  83. data/TODO.impl/39-tier3-and-beyond.md +232 -0
  84. data/TODO.impl/40-spec-drift-cleanup.md +134 -0
  85. data/TODO.impl/50-structure-aware-document-api.md +194 -0
  86. data/TODO.impl/51-grammar-engine-expansion.md +75 -0
  87. data/TODO.impl/52-multi-language-documents.md +95 -0
  88. data/TODO.impl/53-document-plugin-architecture.md +94 -0
  89. data/TODO.impl/54-cjk-support.md +117 -0
  90. data/TODO.impl/55-rtl-support.md +109 -0
  91. data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
  92. data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
  93. data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
  94. data/TODO.impl/59-yard-api-documentation.md +75 -0
  95. data/TODO.impl/60-lsp-server.md +174 -0
  96. data/TODO.impl/61-editor-ecosystem.md +167 -0
  97. data/TODO.impl/62-framework-integrations.md +159 -0
  98. data/TODO.impl/63-web-playground-wasm.md +186 -0
  99. data/TODO.impl/64-http-api-and-sdks.md +191 -0
  100. data/TODO.impl/65-universal-kotoshu.md +251 -0
  101. data/TODO.impl/66-kotoshu-core.md +169 -0
  102. data/TODO.impl/67-kotoshu-rs-and-access-libraries.md +137 -0
  103. data/TODO.impl/68-sota-adoption.md +152 -0
  104. data/TODO.impl/69-sota-typo-eval.md +74 -0
  105. data/TODO.impl/70-sota-confidence-cascade.md +59 -0
  106. data/TODO.impl/71-sota-nested-model.md +63 -0
  107. data/TODO.impl/72-model2vec-evaluation.md +49 -0
  108. data/TODO.impl/73-rl-reranker.md +42 -0
  109. data/TODO.impl/74-distilled-reranker.md +36 -0
  110. data/TODO.impl/75-llm-evaluator.md +35 -0
  111. data/TODO.impl/76-optional-llm-backend.md +37 -0
  112. data/TODO.impl/77-model-coverage-expansion.md +96 -0
  113. data/TODO.impl/78-in-browser-playground.md +69 -0
  114. data/TODO.impl/79-news-and-releases.md +60 -0
  115. data/TODO.impl/80-ecosystem-docs.md +56 -0
  116. data/TODO.impl/81-editor-reach.md +68 -0
  117. data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
  118. data/TODO.impl/83-models-batch-2.md +56 -0
  119. data/TODO.impl/84-language-modules-wave2.md +75 -0
  120. data/TODO.impl/85-wasm-model-api.md +48 -0
  121. data/TODO.impl/86-site-truth-wave2.md +45 -0
  122. data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
  123. data/TODO.impl/88-cli-directory-mode.md +54 -0
  124. data/TODO.impl/89-framework-integrations-exec.md +59 -0
  125. data/TODO.impl/90-gem-correctness-residues.md +44 -0
  126. data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
  127. data/TODO.impl/92-tier-mirror-cors.md +88 -0
  128. data/TODO.impl/93-python-wheels.md +32 -0
  129. data/TODO.impl/94-site-integrations-docs.md +35 -0
  130. data/TODO.impl/95-editor-action-polish.md +27 -0
  131. data/TODO.impl/96-server-semantic-models.md +50 -0
  132. data/TODO.impl/97-yard-execution.md +73 -0
  133. data/TODO.impl/98-campaign-cleanup.md +78 -0
  134. data/TODO.impl/99-site-truth-performance-docs.md +31 -0
  135. data/conformance/vectors.jsonl +2630 -0
  136. data/data/en/grammar/rules.yaml +851 -0
  137. data/docs/STABILITY-1.0-AUDIT.md +251 -0
  138. data/docs/STABILITY-1.0.md +137 -0
  139. data/docs/cache-architecture.md +502 -0
  140. data/docs/verification-evidence/correct3/pr_body.md +37 -0
  141. data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
  142. data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
  143. data/docs/verification-evidence/correct3/sv.aff +856 -0
  144. data/docs/verification-evidence/correct3/sv.dic +153717 -0
  145. data/examples/01_basic_word_checking.rb +38 -0
  146. data/examples/02_text_document_checking.rb +77 -0
  147. data/examples/03_dictionary_backends.rb +137 -0
  148. data/examples/04_trie_data_structure.rb +146 -0
  149. data/examples/05_suggestion_algorithms.rb +239 -0
  150. data/examples/06_configuration_advanced.rb +287 -0
  151. data/examples/07_multi_language_dictionaries.rb +278 -0
  152. data/exe/kotoshu +6 -0
  153. data/lib/kotoshu/algorithms/capitalization.rb +276 -0
  154. data/lib/kotoshu/algorithms/edit_distance.rb +128 -0
  155. data/lib/kotoshu/algorithms/lookup/affix_form.rb +132 -0
  156. data/lib/kotoshu/algorithms/lookup/compound_checks.rb +209 -0
  157. data/lib/kotoshu/algorithms/lookup/compound_form.rb +50 -0
  158. data/lib/kotoshu/algorithms/lookup.rb +841 -0
  159. data/lib/kotoshu/algorithms/ngram_suggest.rb +367 -0
  160. data/lib/kotoshu/algorithms/permutations.rb +283 -0
  161. data/lib/kotoshu/algorithms/phonet_suggest.rb +162 -0
  162. data/lib/kotoshu/algorithms/soundex.rb +92 -0
  163. data/lib/kotoshu/algorithms/suggest.rb +794 -0
  164. data/lib/kotoshu/algorithms.rb +23 -0
  165. data/lib/kotoshu/analyzers/semantic_analyzer.rb +405 -0
  166. data/lib/kotoshu/analyzers.rb +8 -0
  167. data/lib/kotoshu/baseline/application.rb +18 -0
  168. data/lib/kotoshu/baseline/entry.rb +33 -0
  169. data/lib/kotoshu/baseline/store.rb +175 -0
  170. data/lib/kotoshu/baseline.rb +20 -0
  171. data/lib/kotoshu/cache/base_cache.rb +778 -0
  172. data/lib/kotoshu/cache/cache.rb +91 -0
  173. data/lib/kotoshu/cache/eviction_policy.rb +66 -0
  174. data/lib/kotoshu/cache/frequency_cache.rb +283 -0
  175. data/lib/kotoshu/cache/language_cache.rb +511 -0
  176. data/lib/kotoshu/cache/lookup_cache.rb +166 -0
  177. data/lib/kotoshu/cache/model_cache.rb +1147 -0
  178. data/lib/kotoshu/cache/model_registry.rb +109 -0
  179. data/lib/kotoshu/cache/staged_languages.rb +45 -0
  180. data/lib/kotoshu/cache/suggestion_cache.rb +113 -0
  181. data/lib/kotoshu/cache.rb +49 -0
  182. data/lib/kotoshu/cli/auto_setup.rb +71 -0
  183. data/lib/kotoshu/cli/baseline_command.rb +119 -0
  184. data/lib/kotoshu/cli/batch_reporter.rb +321 -0
  185. data/lib/kotoshu/cli/cache_command.rb +282 -0
  186. data/lib/kotoshu/cli/completions_command.rb +210 -0
  187. data/lib/kotoshu/cli/dict_command.rb +64 -0
  188. data/lib/kotoshu/cli/directory_check.rb +146 -0
  189. data/lib/kotoshu/cli/directory_walker.rb +132 -0
  190. data/lib/kotoshu/cli/display_formatter.rb +429 -0
  191. data/lib/kotoshu/cli/errors.rb +36 -0
  192. data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
  193. data/lib/kotoshu/cli/interactive_reviewer.rb +314 -0
  194. data/lib/kotoshu/cli/language_resolver.rb +91 -0
  195. data/lib/kotoshu/cli/navigation_manager.rb +273 -0
  196. data/lib/kotoshu/cli/personal_command.rb +113 -0
  197. data/lib/kotoshu/cli/progress_reporter.rb +114 -0
  198. data/lib/kotoshu/cli/status_report.rb +130 -0
  199. data/lib/kotoshu/cli.rb +880 -0
  200. data/lib/kotoshu/components/passthrough_spell_checker.rb +70 -0
  201. data/lib/kotoshu/components/pos_tagger.rb +98 -0
  202. data/lib/kotoshu/components/spell_checker.rb +73 -0
  203. data/lib/kotoshu/components/synthesizer.rb +60 -0
  204. data/lib/kotoshu/components/tokenizer.rb +58 -0
  205. data/lib/kotoshu/components/whitespace_tokenizer.rb +94 -0
  206. data/lib/kotoshu/components.rb +13 -0
  207. data/lib/kotoshu/configuration/builder.rb +207 -0
  208. data/lib/kotoshu/configuration/resolver.rb +124 -0
  209. data/lib/kotoshu/configuration.rb +829 -0
  210. data/lib/kotoshu/conformance_exporter.rb +256 -0
  211. data/lib/kotoshu/conformance_runner.rb +176 -0
  212. data/lib/kotoshu/core/exceptions.rb +186 -0
  213. data/lib/kotoshu/core/indexed_dictionary.rb +293 -0
  214. data/lib/kotoshu/core/models/affix_rule.rb +262 -0
  215. data/lib/kotoshu/core/models/result/document_result.rb +132 -0
  216. data/lib/kotoshu/core/models/result/word_result.rb +131 -0
  217. data/lib/kotoshu/core/models/word.rb +131 -0
  218. data/lib/kotoshu/core/trie/builder.rb +119 -0
  219. data/lib/kotoshu/core/trie/node.rb +94 -0
  220. data/lib/kotoshu/core/trie/trie.rb +249 -0
  221. data/lib/kotoshu/core.rb +28 -0
  222. data/lib/kotoshu/data/common_words/de.yml +1800 -0
  223. data/lib/kotoshu/data/common_words/en.yml +1215 -0
  224. data/lib/kotoshu/data/common_words/es.yml +750 -0
  225. data/lib/kotoshu/data/common_words/fr.yml +1015 -0
  226. data/lib/kotoshu/data/common_words/pt.yml +870 -0
  227. data/lib/kotoshu/data/common_words/ru.yml +484 -0
  228. data/lib/kotoshu/data/common_words_loader.rb +141 -0
  229. data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
  230. data/lib/kotoshu/data/frozen_kelly.rb +38 -0
  231. data/lib/kotoshu/data.rb +8 -0
  232. data/lib/kotoshu/data_structures/bloom_filter.rb +188 -0
  233. data/lib/kotoshu/data_structures.rb +8 -0
  234. data/lib/kotoshu/debug_logger.rb +146 -0
  235. data/lib/kotoshu/debug_mode.rb +136 -0
  236. data/lib/kotoshu/defaults.rb +84 -0
  237. data/lib/kotoshu/dictionaries/catalog.rb +814 -0
  238. data/lib/kotoshu/dictionaries.rb +8 -0
  239. data/lib/kotoshu/dictionary/base.rb +324 -0
  240. data/lib/kotoshu/dictionary/cspell.rb +243 -0
  241. data/lib/kotoshu/dictionary/custom.rb +225 -0
  242. data/lib/kotoshu/dictionary/hunspell.rb +520 -0
  243. data/lib/kotoshu/dictionary/plain_text.rb +302 -0
  244. data/lib/kotoshu/dictionary/repository.rb +246 -0
  245. data/lib/kotoshu/dictionary/unified.rb +236 -0
  246. data/lib/kotoshu/dictionary/unix_words.rb +218 -0
  247. data/lib/kotoshu/dictionary.rb +15 -0
  248. data/lib/kotoshu/documents/document.rb +120 -0
  249. data/lib/kotoshu/documents/plain_text_document.rb +73 -0
  250. data/lib/kotoshu/documents/source_position.rb +45 -0
  251. data/lib/kotoshu/documents/source_range.rb +76 -0
  252. data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
  253. data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
  254. data/lib/kotoshu/documents/suppressions.rb +44 -0
  255. data/lib/kotoshu/documents/text_node.rb +49 -0
  256. data/lib/kotoshu/documents.rb +178 -0
  257. data/lib/kotoshu/embeddings/embedding_pipeline.rb +238 -0
  258. data/lib/kotoshu/embeddings/lru_cache.rb +244 -0
  259. data/lib/kotoshu/embeddings/onnx_runtime_model.rb +411 -0
  260. data/lib/kotoshu/embeddings/protocol.rb +96 -0
  261. data/lib/kotoshu/embeddings/protocols.rb +23 -0
  262. data/lib/kotoshu/embeddings/registry.rb +184 -0
  263. data/lib/kotoshu/embeddings/search.rb +204 -0
  264. data/lib/kotoshu/embeddings/similarity_engine.rb +278 -0
  265. data/lib/kotoshu/embeddings/similarity_search.rb +328 -0
  266. data/lib/kotoshu/embeddings/vocabulary.rb +306 -0
  267. data/lib/kotoshu/embeddings.rb +87 -0
  268. data/lib/kotoshu/fluent_checker.rb +94 -0
  269. data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +48 -0
  270. data/lib/kotoshu/grammar/pattern_matchers/double_negative_matcher.rb +179 -0
  271. data/lib/kotoshu/grammar/pattern_matchers/phrase_matcher.rb +86 -0
  272. data/lib/kotoshu/grammar/pattern_matchers/possessive_context_matcher.rb +75 -0
  273. data/lib/kotoshu/grammar/pattern_matchers/possessive_contraction_matcher.rb +87 -0
  274. data/lib/kotoshu/grammar/pattern_matchers/sentence_start_matcher.rb +84 -0
  275. data/lib/kotoshu/grammar/pattern_matchers/vowel_sound_matcher.rb +81 -0
  276. data/lib/kotoshu/grammar/pattern_matchers/word_list_matcher.rb +66 -0
  277. data/lib/kotoshu/grammar/rule.rb +99 -0
  278. data/lib/kotoshu/grammar/rule_engine.rb +96 -0
  279. data/lib/kotoshu/grammar/rule_loader.rb +30 -0
  280. data/lib/kotoshu/grammar.rb +24 -0
  281. data/lib/kotoshu/integrity/audit_log.rb +167 -0
  282. data/lib/kotoshu/integrity/manifest.rb +116 -0
  283. data/lib/kotoshu/integrity/net_http.rb +51 -0
  284. data/lib/kotoshu/integrity/rotation_policy.rb +88 -0
  285. data/lib/kotoshu/integrity.rb +26 -0
  286. data/lib/kotoshu/jekyll.rb +101 -0
  287. data/lib/kotoshu/keyboard/layout.rb +116 -0
  288. data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
  289. data/lib/kotoshu/keyboard/layouts/azerty.rb +55 -0
  290. data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
  291. data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
  292. data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
  293. data/lib/kotoshu/keyboard/layouts/dvorak.rb +54 -0
  294. data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
  295. data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
  296. data/lib/kotoshu/keyboard/layouts/hrsl.rb +94 -0
  297. data/lib/kotoshu/keyboard/layouts/jcuken.rb +57 -0
  298. data/lib/kotoshu/keyboard/layouts/latin.rb +266 -0
  299. data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
  300. data/lib/kotoshu/keyboard/layouts/qwerty.rb +52 -0
  301. data/lib/kotoshu/keyboard/layouts/qwertz.rb +55 -0
  302. data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
  303. data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
  304. data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
  305. data/lib/kotoshu/keyboard/registry.rb +201 -0
  306. data/lib/kotoshu/keyboard.rb +96 -0
  307. data/lib/kotoshu/kotoshu_native.so +0 -0
  308. data/lib/kotoshu/language/detector.rb +242 -0
  309. data/lib/kotoshu/language/identifier.rb +376 -0
  310. data/lib/kotoshu/language/languages/base.rb +271 -0
  311. data/lib/kotoshu/language/lid_detector.rb +188 -0
  312. data/lib/kotoshu/language/normalizer/arabic.rb +113 -0
  313. data/lib/kotoshu/language/normalizer/base.rb +133 -0
  314. data/lib/kotoshu/language/normalizer/greek.rb +63 -0
  315. data/lib/kotoshu/language/normalizer/hebrew.rb +72 -0
  316. data/lib/kotoshu/language/normalizer/persian.rb +36 -0
  317. data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
  318. data/lib/kotoshu/language/registry.rb +209 -0
  319. data/lib/kotoshu/language/resources/ar/common_words.txt +6753 -0
  320. data/lib/kotoshu/language/resources/ar/confusion_sets.txt +11 -0
  321. data/lib/kotoshu/language/resources/de/common_words.txt +10003 -0
  322. data/lib/kotoshu/language/resources/de/confusion_sets.txt +246 -0
  323. data/lib/kotoshu/language/resources/en/common_words.txt +9979 -0
  324. data/lib/kotoshu/language/resources/en/confusion_sets.txt +871 -0
  325. data/lib/kotoshu/language/resources/es/common_words.txt +9992 -0
  326. data/lib/kotoshu/language/resources/es/confusion_sets.txt +17 -0
  327. data/lib/kotoshu/language/resources/fr/common_words.txt +9993 -0
  328. data/lib/kotoshu/language/resources/fr/confusion_sets.txt +76 -0
  329. data/lib/kotoshu/language/resources/pt/common_words.txt +9977 -0
  330. data/lib/kotoshu/language/resources/pt/confusion_sets.txt +18 -0
  331. data/lib/kotoshu/language/resources/ru/common_words.txt +9951 -0
  332. data/lib/kotoshu/language/resources/ru/confusion_sets.txt +5 -0
  333. data/lib/kotoshu/language/script.rb +128 -0
  334. data/lib/kotoshu/language/segmenter.rb +122 -0
  335. data/lib/kotoshu/language/suika.rb +38 -0
  336. data/lib/kotoshu/language/tokenizer/base.rb +187 -0
  337. data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
  338. data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
  339. data/lib/kotoshu/language/tokenizer/french_tokenizer.rb +172 -0
  340. data/lib/kotoshu/language/tokenizer/german_tokenizer.rb +41 -0
  341. data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
  342. data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
  343. data/lib/kotoshu/language/tokenizer/japanese_tokenizer.rb +55 -0
  344. data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +153 -0
  345. data/lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb +160 -0
  346. data/lib/kotoshu/language/tokenizer/russian_tokenizer.rb +93 -0
  347. data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
  348. data/lib/kotoshu/language/tokenizer/spanish_tokenizer.rb +122 -0
  349. data/lib/kotoshu/language.rb +127 -0
  350. data/lib/kotoshu/languages/ar/language.rb +98 -0
  351. data/lib/kotoshu/languages/bg/language.rb +37 -0
  352. data/lib/kotoshu/languages/ca/language.rb +22 -0
  353. data/lib/kotoshu/languages/cs/language.rb +22 -0
  354. data/lib/kotoshu/languages/da/language.rb +22 -0
  355. data/lib/kotoshu/languages/de/language.rb +554 -0
  356. data/lib/kotoshu/languages/el/language.rb +50 -0
  357. data/lib/kotoshu/languages/en/language.rb +470 -0
  358. data/lib/kotoshu/languages/es/language.rb +476 -0
  359. data/lib/kotoshu/languages/et/language.rb +24 -0
  360. data/lib/kotoshu/languages/fa/language.rb +91 -0
  361. data/lib/kotoshu/languages/fr/language.rb +505 -0
  362. data/lib/kotoshu/languages/he/language.rb +98 -0
  363. data/lib/kotoshu/languages/hr/language.rb +24 -0
  364. data/lib/kotoshu/languages/hu/language.rb +22 -0
  365. data/lib/kotoshu/languages/id/language.rb +25 -0
  366. data/lib/kotoshu/languages/it/language.rb +22 -0
  367. data/lib/kotoshu/languages/ja/language.rb +476 -0
  368. data/lib/kotoshu/languages/ko/language.rb +56 -0
  369. data/lib/kotoshu/languages/latin_base.rb +53 -0
  370. data/lib/kotoshu/languages/lt/language.rb +24 -0
  371. data/lib/kotoshu/languages/lv/language.rb +24 -0
  372. data/lib/kotoshu/languages/nb/language.rb +25 -0
  373. data/lib/kotoshu/languages/ne/language.rb +60 -0
  374. data/lib/kotoshu/languages/nl/language.rb +22 -0
  375. data/lib/kotoshu/languages/nn/language.rb +26 -0
  376. data/lib/kotoshu/languages/pl/language.rb +22 -0
  377. data/lib/kotoshu/languages/pt/language.rb +439 -0
  378. data/lib/kotoshu/languages/ro/language.rb +22 -0
  379. data/lib/kotoshu/languages/ru/language.rb +420 -0
  380. data/lib/kotoshu/languages/sk/language.rb +25 -0
  381. data/lib/kotoshu/languages/sl/language.rb +25 -0
  382. data/lib/kotoshu/languages/sr/language.rb +39 -0
  383. data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
  384. data/lib/kotoshu/languages/sv/language.rb +22 -0
  385. data/lib/kotoshu/languages/tr/language.rb +36 -0
  386. data/lib/kotoshu/languages/uk/language.rb +38 -0
  387. data/lib/kotoshu/languages/vi/language.rb +22 -0
  388. data/lib/kotoshu/languages.rb +64 -0
  389. data/lib/kotoshu/metrics_collector.rb +238 -0
  390. data/lib/kotoshu/metrics_module.rb +121 -0
  391. data/lib/kotoshu/models/context.rb +119 -0
  392. data/lib/kotoshu/models/embedding_model.rb +181 -0
  393. data/lib/kotoshu/models/fasttext_model.rb +212 -0
  394. data/lib/kotoshu/models/nearest_neighbor.rb +96 -0
  395. data/lib/kotoshu/models/onnx_model.rb +330 -0
  396. data/lib/kotoshu/models/semantic_error.rb +204 -0
  397. data/lib/kotoshu/models/suggestion.rb +106 -0
  398. data/lib/kotoshu/models/word_embedding.rb +97 -0
  399. data/lib/kotoshu/models.rb +27 -0
  400. data/lib/kotoshu/multi_language_checker.rb +127 -0
  401. data/lib/kotoshu/native.rb +83 -0
  402. data/lib/kotoshu/native_backend.rb +131 -0
  403. data/lib/kotoshu/paths.rb +79 -0
  404. data/lib/kotoshu/personal_dictionary.rb +105 -0
  405. data/lib/kotoshu/plugins/plugin.rb +61 -0
  406. data/lib/kotoshu/plugins/registry.rb +118 -0
  407. data/lib/kotoshu/plugins.rb +9 -0
  408. data/lib/kotoshu/project_config.rb +76 -0
  409. data/lib/kotoshu/readers/aff_data.rb +530 -0
  410. data/lib/kotoshu/readers/aff_reader.rb +393 -0
  411. data/lib/kotoshu/readers/condition_checker.rb +125 -0
  412. data/lib/kotoshu/readers/dic_reader.rb +208 -0
  413. data/lib/kotoshu/readers/file_reader.rb +377 -0
  414. data/lib/kotoshu/readers/lookup_builder.rb +584 -0
  415. data/lib/kotoshu/readers/readers.rb +17 -0
  416. data/lib/kotoshu/readers.rb +25 -0
  417. data/lib/kotoshu/resource_bundle.rb +55 -0
  418. data/lib/kotoshu/resource_manager.rb +600 -0
  419. data/lib/kotoshu/results/result.rb +165 -0
  420. data/lib/kotoshu/results.rb +8 -0
  421. data/lib/kotoshu/rspec.rb +163 -0
  422. data/lib/kotoshu/scripts/fasttext_to_onnx.py +275 -0
  423. data/lib/kotoshu/source_registry.rb +104 -0
  424. data/lib/kotoshu/spellchecker/parallel_checker.rb +90 -0
  425. data/lib/kotoshu/spellchecker.rb +406 -0
  426. data/lib/kotoshu/string_metrics.rb +153 -0
  427. data/lib/kotoshu/suggestions/context.rb +43 -0
  428. data/lib/kotoshu/suggestions/frequency_provider.rb +80 -0
  429. data/lib/kotoshu/suggestions/generator.rb +169 -0
  430. data/lib/kotoshu/suggestions/pipeline.rb +135 -0
  431. data/lib/kotoshu/suggestions/semantic_cascade.rb +73 -0
  432. data/lib/kotoshu/suggestions/strategies/base_strategy.rb +315 -0
  433. data/lib/kotoshu/suggestions/strategies/composite_strategy.rb +197 -0
  434. data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +631 -0
  435. data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +207 -0
  436. data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +165 -0
  437. data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +284 -0
  438. data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +317 -0
  439. data/lib/kotoshu/suggestions/strategies/symspell_strategy.rb +270 -0
  440. data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
  441. data/lib/kotoshu/suggestions/suggestion.rb +154 -0
  442. data/lib/kotoshu/suggestions/suggestion_set.rb +270 -0
  443. data/lib/kotoshu/suggestions/sweep_index.rb +137 -0
  444. data/lib/kotoshu/suggestions.rb +29 -0
  445. data/lib/kotoshu/tasks/check_task.rb +104 -0
  446. data/lib/kotoshu/tasks.rb +11 -0
  447. data/lib/kotoshu/validators/spelling_validator.rb +101 -0
  448. data/lib/kotoshu/validators.rb +9 -0
  449. data/lib/kotoshu/version.rb +5 -0
  450. data/lib/kotoshu.rb +538 -0
  451. data/rakelib/compile.rake +10 -0
  452. data/script/validate_all_dictionaries.rb +444 -0
  453. data/scripts/build_native_gem.rb +45 -0
  454. data/scripts/extract_eval_grids.rb +37 -0
  455. data/scripts/generate_frozen_tiers.rb +44 -0
  456. data/sig/kotoshu.rbs +435 -0
  457. data/test_oop.rb +79 -0
  458. metadata +546 -0
@@ -0,0 +1,49 @@
1
+ # 72 — Model2Vec Evaluation (adopt-later C1 decision)
2
+
3
+ From [68-sota-adoption.md](68-sota-adoption.md) item C1. Model2Vec /
4
+ Potion distilled static embeddings: ~30 MB, ~500× faster than fastText,
5
+ beats other statics on MTEB
6
+ ([repo](https://github.com/MinishLab/model2vec),
7
+ [fastText comparison](https://minish.ai/blog/2025-07-28-fasttext/)).
8
+ Word-keyed like ours — same OOV gap.
9
+
10
+ ## Goal
11
+
12
+ Decide adopt/reject on DATA, using the corpus benchmark plan 69 built:
13
+ can a Potion-class static embedding beat our shipped full/fluency tiers
14
+ on the GitHub Typo Corpus top-1/5/20 hit rates, at comparable or
15
+ smaller size? License per model verified for redistribution (Potion
16
+ series is MIT, models distill from different teachers — record each
17
+ teacher's license).
18
+
19
+ ## Tasks
20
+
21
+ 1. `eval/model2vec_bench.py` — for each candidate model
22
+ (potion-base-8M, potion-base-32M, potion-mini-8M and one
23
+ fastText-comparable MTEB-strong variant): load via HF hub (network,
24
+ never in CI), map our corpus pair words, compute the same top-k hit
25
+ rates as `corpus_bench.py`, per language.
26
+ 2. Report `eval/reports/model2vec.{lang}.json`: candidate vs
27
+ full/fluency/mini hit rates + model size + license chain.
28
+ 3. Decision rule (pre-declared): adopt only if a candidate beats
29
+ **fluency** top-5 on ≥ 5 of 9 languages at ≤ 20 MB with a clean
30
+ license chain; otherwise record reject with numbers.
31
+ 4. If adopt: a separate plan supersedes this one for integration
32
+ (registry entries, gem resolver) — this plan only decides.
33
+
34
+ ## Acceptance
35
+
36
+ - Reports for all 9 languages × candidates, committed; corpora never
37
+ committed; decisions traceable to numbers.
38
+
39
+ ## Status
40
+
41
+ **Evaluated (2026-09-03): REJECT on data** (models PR #7). The only
42
+ size-compliant candidate (potion-base-2M, 7.6 MB) wins 0/9 languages
43
+ (en top-5 0.241 vs fluency 0.252); the 129 MB potion-base-32M manages
44
+ one marginal English edge and loses everywhere else. Structural
45
+ blocker: Potion is English-only WordPiece - eight of nine languages
46
+ are not coverable. Licenses were clean; the rejection is on merit.
47
+ Side-result: our fastText tiers beat the current static-embedding SOTA
48
+ on our own task. C1 closed - revisit only if a genuinely multilingual
49
+ static model appears under the size cap.
@@ -0,0 +1,42 @@
1
+ # 73 — RL-Trained Reranker (adopt-later C2)
2
+
3
+ From [68-sota-adoption.md](68-sota-adoption.md) item C2. The GLM-5.3
4
+ recipe scaled down: RL on **binary verifiable rewards** — spelling
5
+ correction is verifiable by construction (output == reference)
6
+ ([GLM tech report](https://arxiv.org/html/2602.15763v1)).
7
+
8
+ ## Goal
9
+
10
+ Train tier-sized reranker weights (not LLMs — the existing embedding +
11
+ a small scorer head) with RL where the reward is exact-match on typo
12
+ corrections, replacing heuristic rerank scoring with a learned policy
13
+ that keeps our latency/memory envelope.
14
+
15
+ ## Preconditions (all three, no exceptions)
16
+
17
+ 1. Conformance vectors exist and are stable (plan 67 M3 output) — the
18
+ reward must be measurable against golden fixtures.
19
+ 2. A training-pipeline owner is named (compute, seeds, checkpoints).
20
+ 3. The corpus bench (plan 69) provides the train/eval split hygiene:
21
+ train on synthetic keyboard typos, eval on GitHub Typo Corpus.
22
+
23
+ ## Tasks (when unblocked)
24
+
25
+ 1. `training/` workspace (likely models repo or a new kotoshu-training
26
+ repo — owner decision): policy = current cosine scorer + small MLP
27
+ head over (candidate, context) features; GRPO-style or simple REINFORCE
28
+ with exact-match reward; baselines from the corpus bench.
29
+ 2. Gate: beat the shipped rerank on corpus top-1 by ≥ 5 points without
30
+ regressing rank_corr gates; same ONNX exportability; int8-able.
31
+ 3. Distribution through the existing registry as new model versions —
32
+ never a silent replacement.
33
+
34
+ ## Acceptance
35
+
36
+ - Trained artifact passes all existing eval gates + corpus bench lift;
37
+ training reproducible from committed seeds and data manifests.
38
+
39
+ ## Status
40
+
41
+ _Planning — gated_ on plan 67 M3 conformance vectors + named training
42
+ owner. Do not start before both exist.
@@ -0,0 +1,36 @@
1
+ # 74 — Distilled Reranker (adopt-later C3)
2
+
3
+ From [68-sota-adoption.md](68-sota-adoption.md) item C3. The DeepSeek
4
+ V4 recipe: consolidate specialists into a student via **on-policy
5
+ distillation (reverse KL)** — the exact recipe for distilling a large
6
+ teacher reranker into tier-sized weights
7
+ ([report](https://arxiv.org/html/2606.19348v1)).
8
+
9
+ ## Goal
10
+
11
+ Where RL (plan 73) learns from binary rewards alone, distillation
12
+ transfers a strong teacher's **soft judgments** — a large embedding or
13
+ LLM scoring the same candidate lists — into the small scorer, then the
14
+ RL stage (73) fine-tunes on top. Teacher runs offline during training
15
+ only; nothing large ships.
16
+
17
+ ## Preconditions
18
+
19
+ Same as plan 73, plus a teacher selection (open-weights embedding
20
+ model or API LLM with a cost cap — owner decision; API usage must be
21
+ budgeted and never in CI).
22
+
23
+ ## Tasks (when unblocked)
24
+
25
+ 1. Teacher scoring pass over the training corpus (offline, one-off,
26
+ committed scores) — candidate lists identical to what the engine
27
+ produces at runtime.
28
+ 2. Student = same architecture as plan 73's policy; loss = reverse KL
29
+ to teacher distribution over candidates; then plan 73's RL stage.
30
+ 3. Gate: student ≥ teacher-within-1-point on corpus top-1 while fitting
31
+ the tier envelope; all existing gates pass.
32
+
33
+ ## Status
34
+
35
+ _Planning — gated_ behind plan 73's preconditions + teacher decision.
36
+ Sequence: 74 distill → 73 RL fine-tune, one pipeline.
@@ -0,0 +1,35 @@
1
+ # 75 — LLM-as-Evaluator (adopt-later C4)
2
+
3
+ From [68-sota-adoption.md](68-sota-adoption.md) item C4. LLM-as-GEC-
4
+ evaluator is standard practice in the literature
5
+ ([BEA 2024](https://aclanthology.org/2024.bea-1.6/)).
6
+
7
+ ## Goal
8
+
9
+ Augment the eval harness with an optional LLM judge that scores tier
10
+ outputs where our proxy metrics are weakest: fluency of the corrected
11
+ sentence and overcorrection risk — dimensions cosine hit rates cannot
12
+ see. Strictly offline tooling for model selection; never a release
13
+ gate (no API dependency in CI), never runtime.
14
+
15
+ ## Tasks
16
+
17
+ 1. `eval/llm_judge.py` — takes candidate corrections + references,
18
+ prompts a judge model (API, user-provided key via env; never
19
+ committed), produces per-language fluency/overcorrection scores
20
+ appended to a SEPARATE report file (eval/reports/llm-judge.*.json),
21
+ clearly marked non-deterministic.
22
+ 2. Prompt + rubric documented in-repo; judge model + version recorded
23
+ in every report.
24
+ 3. Cost cap per run (pair count × languages), hard default small
25
+ (e.g. 100 pairs/language) — a full sweep needs explicit flags.
26
+
27
+ ## Acceptance
28
+
29
+ - Runs only with an explicit env key + opt-in flag; reports carry
30
+ model/version/cost metadata; zero effect on gates or CI.
31
+
32
+ ## Status
33
+
34
+ _Planning — low priority._ Proxy metrics suffice until corpora-driven
35
+ work (72–74) lands.
@@ -0,0 +1,37 @@
1
+ # 76 — Optional External LLM Backend (adopt-later C5)
2
+
3
+ From [68-sota-adoption.md](68-sota-adoption.md) item C5 and its
4
+ rejection note: LLMs never become the default engine (violates small/
5
+ offline/CPU), but an explicit opt-in "heavy" backend can exist for
6
+ users who want maximum fluency polishing and accept the cost.
7
+
8
+ ## Goal
9
+
10
+ A `Kotoshu::Backends::Llm` adapter behind the same interfaces the
11
+ engine already exposes (checker/reranker seams), delegating
12
+ sentence-level polishing to a local (1–8B via ONNX Runtime GenAI /
13
+ llama.cpp) or API model. Default OFF; absent config = today's behavior
14
+ exactly.
15
+
16
+ ## Design constraints
17
+
18
+ 1. Adapter pattern, no engine changes: register like any plugin;
19
+ capability flags mark it slow/networked so callers can decline.
20
+ 2. Two transports: local GGUF/ORT-GenAI (user-supplied model path) and
21
+ HTTP API (user-supplied endpoint + key via env). No keys in code,
22
+ no telemetry.
23
+ 3. Interactive surfaces (LSP, CLI `--interactive`) surface a visible
24
+ "heavy mode" indicator; CI surfaces refuse it by default.
25
+ 4. Output contract: same suggestion objects; LLM refinements only
26
+ applied on explicit accept in interactive mode.
27
+
28
+ ## Acceptance
29
+
30
+ - Engine untouched with backend disabled (existing suite proves it);
31
+ adapter specs use a stub server (no real API in tests); docs state
32
+ latency/memory expectations honestly.
33
+
34
+ ## Status
35
+
36
+ _Planning — gated_ on user demand. Not scheduled until someone asks
37
+ for it.
@@ -0,0 +1,96 @@
1
+ # 77 — Model coverage expansion: 9 → 25+ languages
2
+
3
+ > Asked by the owner 2026-09-05: "Are our models fully complete and
4
+ > comprehensive with no user gaps?" They are not — this plan closes the gap.
5
+
6
+ ## Context
7
+
8
+ The registry (models-fasttext-onnx `registry.json`, v1.0.1) carries
9
+ semantic models for exactly **9 languages** (de en es fr ja ko pt ru zh)
10
+ × 3 tiers = 27 entries. Meanwhile:
11
+
12
+ - `dictionaries/` repo: **98** languages with Hunspell `.aff`/`.dic`
13
+ - FastText CC vectors exist for **157+** languages
14
+ - The site language matrix and gem language registry advertise the
15
+ dictionary breadth, but semantic reranking silently stops at 9
16
+
17
+ A user who spells in Italian, Polish, Turkish, or Dutch gets the
18
+ traditional path only. That is the largest user gap in the ecosystem.
19
+
20
+ ## What is NOT changing
21
+
22
+ - The tier recipe (full 300d / fluency int8 top-50k / mini int8
23
+ top-10k) — settled by plan 67's measured gates; no SVD, no Model2Vec.
24
+ - The eval gates from plan 67's gate table: fluency
25
+ `rank_corr ≥ 0.97 / top1 ≥ 0.95`, mini `rank_corr ≥ 0.90 /
26
+ top1 ≥ 0.85` (keyboard-aware eval from plan 69). A language that
27
+ fails gates ships **no** tiers rather than bad tiers.
28
+
29
+ ## Phases
30
+
31
+ ### Phase 1 — candidate selection (deterministic)
32
+
33
+ Candidates = dictionaries repo dirs ∩ FastText `cc.{lang}.300.vec`
34
+ availability ∩ eval-gate pass. Rank by speaker base × dictionary
35
+ quality (dictionary repo manifest). Batch 1 target (16):
36
+
37
+ ```
38
+ it nl pl uk tr cs sv el hu ro da fi no id vi ca
39
+ ```
40
+
41
+ Batch 2 (stretch, if compute allows): ar fa he th vi-tok hi bn ms tl
42
+ sk sl lt lv et is hr sr bg.
43
+
44
+ ### Phase 2 — pipeline run per language
45
+
46
+ Use the existing scripts (see models repo `TODO.impl/01`, `04`, `06`):
47
+ download `cc.{lang}.300.vec.gz` → convert to ONNX full → int8 +
48
+ vocab-cut to fluency/mini → keyboard-aware eval → gates. Record
49
+ per-language eval JSON into `eval/` like the existing nine.
50
+
51
+ ### Phase 3 — registry + release
52
+
53
+ Regenerate `registry.json` (+ manifest, schemas untouched), bump
54
+ registry_version, tag `v1.1.0` (policy from models plan 05: pre-declared
55
+ minor for coverage expansion). Release assets: 3 tiers × new languages,
56
+ mirrored to the media host per plan 07's distribution map.
57
+
58
+ ### Phase 4 — downstream truth updates
59
+
60
+ - Gem `Language` registry: full-feature languages list gains the
61
+ new entries (spelling + model available).
62
+ - Site language matrix + per-language pages: flip `hasModel` for
63
+ the new set; copy updated by the site plan (79/80 reuse).
64
+ - kotoshu-rs / SDKs: no code change (registry-driven).
65
+
66
+ ## Owner gates
67
+
68
+ - None for batch 1 execution (standing directive 2026-09-05).
69
+ - Version numbers: v1.1.0 follows plan 05's pre-declared policy —
70
+ registry major/minor tracks coverage, patch tracks regeneration.
71
+ If the owner objects, yank per policy is NOT possible for release
72
+ assets — ask first if in doubt.
73
+
74
+ ## Success criteria
75
+
76
+ - ≥ 8 new languages pass gates and appear in registry + release.
77
+ - Every shipped tier has eval JSON proving the gates.
78
+ - Registry schema version unchanged; gem + site consume it without
79
+ code changes.
80
+
81
+ ## Status
82
+
83
+ **Implemented (2026-09-05, models PR #13 + tag v1.1.0).** 13 of 16
84
+ batch-one candidates shipped: ca cs da el hu it nl pl ro sv tr uk vi
85
+ — 22 languages x 3 tiers, 66 registry resources, 134 release assets.
86
+ All tiers passed the gates unweakened (fluency rank_corr 0.9999,
87
+ top1 0.9583-1.0000). noise.py grew QWERTY/QWERTZ supplements plus
88
+ curated Turkish-Q / Ukrainian JCUKEN / Greek-phonetic grids; a
89
+ pre-existing release_notes.py crash on null tier mirrors was fixed
90
+ in passing. Released registry byte-identical to committed (local
91
+ rebuild reproduces CI sha256).
92
+
93
+ Dropped from batch 1 (no dictionaries-repo aff/dic): fi, no, id —
94
+ `no` exists only as nb/nn; an owner mapping decision for a v1.2.
95
+ Site truth updated (MODEL_LANGUAGES + news entry, site PR #4); the
96
+ gem needed no change (registry-driven).
@@ -0,0 +1,69 @@
1
+ # 78 — Zero-install wasm playground on the site
2
+
3
+ ## Context
4
+
5
+ The site playground (`/playground`) currently requires a locally
6
+ running `kotoshu-server` — a real barrier for the curious visitor.
7
+ Since `@kotoshu/wasm` 0.1.0 is live on npm (291 KiB gz engine), a
8
+ zero-install in-browser demo is now possible: load the engine from a
9
+ CDN, fetch `.aff`/`.dic` + a mini-tier model, check text live.
10
+
11
+ This is the single highest-leverage marketing surface: every visitor
12
+ can feel the product in five seconds with no install.
13
+
14
+ ## Design
15
+
16
+ - **Route**: keep `/playground` as the home of the in-browser demo;
17
+ the server-based playground moves to `/playground/server` (linked
18
+ as "run it against your own instance").
19
+ - **Engine**: `import { init } from "@kotoshu/wasm"` via a pinned
20
+ CDN URL (esm.run / esm.sh, version-pinned to 0.1.x). Never
21
+ `@latest`.
22
+ - **Dictionaries**: `kotoshu/dictionaries` repo files via jsDelivr
23
+ `/gh/kotoshu/dictionaries@<tag>/<lang>.aff|.dic` — verify the
24
+ tag layout first (dictionaries repo manifest). Files are < 20 MiB
25
+ so jsDelivr gh CDN is legal.
26
+ - **Model**: mini tier from the models registry. Registry URLs point
27
+ at release assets — verify browser CORS on
28
+ `objects.githubusercontent.com` before committing to that source;
29
+ fallback = jsDelivr gh for tag-tree files, else document the
30
+ limitation and ship dictionary-only mode with a note that
31
+ reranking needs the full app.
32
+ - **UX**: language select (defaults to browser language if one of
33
+ the supported set), textarea with misspelling underlines,
34
+ suggestion popover on click, tier badge (mini), first-load size
35
+ readout. Reduced-motion respected; no layout shift.
36
+ - **Aesthetic**: follows the site's dictionary-catalog design
37
+ system (site plans 05/09) — the playground is a "specimen" page,
38
+ not a generic SaaS demo.
39
+
40
+ ## Phases
41
+
42
+ 1. **Fetch spike** (an hour): prove CORS + CDN loads for engine,
43
+ one dictionary, mini model. Decide model source. If models CORS
44
+ fails → dictionary-only v1, models behind the server playground.
45
+ 2. **Component**: `PlaygroundLab.astro` + island
46
+ (`client:load`), module-scope caching of loaded languages,
47
+ Web Worker for the engine to keep typing at 60fps (the engine
48
+ API is sync; worker keeps the main thread free).
49
+ 3. **Wire routes**, update copy on `/` hero to link the demo
50
+ ("try it now — nothing installs").
51
+ 4. **Verify in a browser** before merge: golden path (load en,
52
+ type a typo, get suggestions), CJK (ja), offline (graceful
53
+ error), reduced-motion, mobile width.
54
+
55
+ ## Owner gates
56
+
57
+ None. CDN pinning note: record the chosen CDN + version in this
58
+ file's Status when shipped.
59
+
60
+ ## Status
61
+
62
+ **Implemented (2026-09-05, site PRs #1-#3, live at kotoshu.org/playground).**
63
+ Zero-install wasm demo: @kotoshu/wasm 0.1.0 in a worker, underlines +
64
+ click-to-mend popover, six languages; server playground moved to
65
+ /playground/server. CDN pin: jsDelivr npm raw files @0.1.0 with manual
66
+ instantiate (esm.sh and esm.run both broken for the bundler-target
67
+ package); dictionaries via jsDelivr gh @commit 1829a3e. Models verdict:
68
+ GitHub release assets send no CORS headers, so the demo is honestly
69
+ dictionary-only; the model story lives on the server playground page.
@@ -0,0 +1,60 @@
1
+ # 79 — News + releases section on the site
2
+
3
+ ## Context
4
+
5
+ The site has docs, languages, audiences, install — but no news.
6
+ Releases across seven registries shipped in 2026-09 and a visitor
7
+ cannot tell the project is alive. The owner asked for this
8
+ explicitly ("think also about documentation and our website
9
+ (e.g. docs, news)").
10
+
11
+ ## Design
12
+
13
+ - `/news` — index, reverse-chronological, one entry per release or
14
+ event, each rendered as a dictionary entry (date as the headword's
15
+ etymology line; the entry body is the note).
16
+ - Entry pages `/news/<slug>` for anything longer than a paragraph.
17
+ - **RSS + Atom** at `/news/rss.xml` + `/news/atom.xml` (static).
18
+ - Design follows the site system (eyebrows, `辞書` vertical accent,
19
+ numbered senses for changelog bullets). No blog-engine machinery —
20
+ entries are typed content collections.
21
+
22
+ ## Seed content (real history, verified 2026-09-05)
23
+
24
+ 1. **2026-09-05 — Trusted publishing everywhere** — npm, RubyGems,
25
+ crates.io now publish keyless via OIDC (RFC 3691 on crates.io,
26
+ provenance on npm).
27
+ 2. **2026-09-05 — First releases on all channels** — PyPI
28
+ `kotoshu` + `kotoshu-native` 0.1.0; npm `@kotoshu/wasm` +
29
+ `@kotoshu/client` 0.1.0; crates.io `kotoshu` 0.1.0.
30
+ 3. **2026-09-05 — Ruby gem 0.7.0** — the universal-kotoshu cut:
31
+ model tiers + registry, native extension (dual-backend parity,
32
+ 2630 conformance vectors, 0 divergences), legacy-cache bridge,
33
+ tier default = fluency.
34
+ 4. **2026-09-04 — Engine correctness wave** — compound/suggest
35
+ parity with hunspell semantics (T2), 1315+1315 conformance green
36
+ in Rust and Ruby.
37
+ 5. **2026-09-03 — Models v1.0.x registry + tiers** — 9 languages ×
38
+ 3 tiers, keyboard-aware eval, media-host distribution.
39
+ 6. **Earlier** — kotoshu-lsp 0.1.0, kotoshu-server 0.1.0,
40
+ kotoshu-go v0.1.0, action-kotoshu v1, site launch.
41
+
42
+ Pull dates from the registries/gem CHANGELOG while writing — verify,
43
+ don't guess.
44
+
45
+ ## Phases
46
+
47
+ 1. Content collection schema + seed entries above.
48
+ 2. `/news` index + slugs + feeds; nav + footer links; sitemap.
49
+ 3. Cross-link: gem docs page → news; news → install.
50
+
51
+ ## Owner gates
52
+
53
+ None.
54
+
55
+ ## Status
56
+
57
+ **Implemented (2026-09-05, site PR #1).** /news with 8 entries as
58
+ dictionary entries, RSS 2.0 + Atom 1.0 static feeds with autodiscovery,
59
+ nav/footer/sitemap wired. Dates verified against registry timestamps
60
+ (two plan guesses corrected: PyPI shipped 09-04; channels entry split).
@@ -0,0 +1,56 @@
1
+ # 80 — Ecosystem docs: every channel documented
2
+
3
+ ## Context
4
+
5
+ The site documents the gem well (cli, caching, configuration, api,
6
+ comparison, migration, plugins). But the ecosystem now ships on
7
+ eight channels, and a Python/JS/Go/Rust user arriving at
8
+ kotoshu.org finds no path in. Audience: non-Ruby developers —
9
+ exactly who the access libraries (plan 67) were built for.
10
+
11
+ ## Channels to document (verified live 2026-09-05)
12
+
13
+ | Channel | Package | What a user needs |
14
+ |---|---|---|
15
+ | PyPI | `kotoshu`, `kotoshu-native` | pip install, Python API, maturin wheel note |
16
+ | npm | `@kotoshu/client`, `@kotoshu/wasm` | npm install, JS API, browser/Node |
17
+ | crates.io | `kotoshu` | cargo add, Rust API, KOSH-v1 batch format |
18
+ | Go | `kotoshu-go` | go get, HTTP-mode usage |
19
+ | RubyGems | `kotoshu` | already documented (link) |
20
+ | HTTP | `kotoshu-server` | /v1/check contract, docker run |
21
+ | LSP | `kotoshu-lsp` | gem install + editor config (Neovim lspconfig, generic LSP) |
22
+ | CI | `action-kotoshu` | workflow yaml, SARIF upload |
23
+
24
+ ## Structure
25
+
26
+ - `/install` becomes a channel-picker (tabs or a table: language →
27
+ command), each linking to a per-channel page under
28
+ `/docs/clients/…`.
29
+ - `/docs/clients/python|javascript|rust|go|http|lsp|action` pages:
30
+ install, minimal example, capability notes (which engine tier,
31
+ what is native vs client), link to package registry page.
32
+ - Per-channel quick starts **verified by running them** (pip install
33
+ + 5-line script; npm install + node -e; cargo add + cargo run;
34
+ docker run + curl) — the docs/quickstart-verification precedent
35
+ (gem plan 28) applies: no example ships unexecuted.
36
+ - The audiences page cross-links each persona to their channel.
37
+
38
+ ## Phases
39
+
40
+ 1. `/install` restructure (channel picker).
41
+ 2. Seven per-channel pages + examples verified.
42
+ 3. Nav/sitemap/search-index updates.
43
+
44
+ ## Owner gates
45
+
46
+ None.
47
+
48
+ ## Status
49
+
50
+ **Implemented (2026-09-05, site PR #2).** /install is a channel catalog
51
+ (8 channels); seven guides under /docs/clients/ (python javascript rust
52
+ go http lsp action). Every quick start executed for real (pip, npm http
53
+ + wasm, cargo, go, server-from-source + curl, LSP stdio handshake);
54
+ the action YAML parse-validated only. Each page carries a verified-date
55
+ annotation. kotoshu-server docs intentionally show run-from-source
56
+ until the empty 0.1.0 gem is republished.
@@ -0,0 +1,68 @@
1
+ # 81 — Editor reach: VS Code extension + pre-commit
2
+
3
+ > Supersedes the unshipped parts of
4
+ > [61-editor-ecosystem.md](61-editor-ecosystem.md). Status
5
+ > correction: kotoshu-lsp 0.1.0 (plan 60), kotoshu-server 0.1.0 +
6
+ > SDKs (plan 64, partial), and action-kotoshu v1 shipped during the
7
+ > 2026-09 campaign; what remains is the editor package and the
8
+ > pre-commit hook.
9
+
10
+ ## Context
11
+
12
+ The LSP server exists (`kotoshu-lsp` gem) but nothing delivers it
13
+ into editors. The generic-LSP path requires manual config most
14
+ users never do. VS Code via a marketplace extension is the single
15
+ highest-leverage editor surface. pre-commit is where CI-minded
16
+ Python/Go shops hook spellchecking.
17
+
18
+ ## Track A — `kotoshu-vscode` (new repo)
19
+
20
+ - Scaffold: TypeScript extension, `vscode-languageclient`, no
21
+ bundling surprises (esbuild), `package.json` contributes
22
+ `kotoshu.languageTool` (document selectors: markdown, asciidoc,
23
+ plaintext, code comments opt-in later).
24
+ - Server discovery order: `kotoshu-lsp.serverPath` setting →
25
+ `bundle exec kotoshu-lsp` in workspace → global `kotoshu-lsp` →
26
+ actionable error message with install command (no silent
27
+ failure).
28
+ - Features v0.1: publish diagnostics on open/change/save;
29
+ suggestions as code actions; personal-dictionary add action
30
+ (server command if supported, else documented).
31
+ - Repo: `kotoshu/kotoshu-vscode`, CI (build + `vscode-test` smoke
32
+ against a fixture workspace with a deliberate misspelling),
33
+ LICENSE BSD-2-Clause, README with GIF slot.
34
+ - Packaging: `vsce package` produces a .vsix in CI artifacts.
35
+ **Marketplace publish is owner-gated** (needs publisher PAT);
36
+ Open VSX after that.
37
+
38
+ ## Track B — pre-commit hook
39
+
40
+ - `.pre-commit-hooks.yaml` living where the hook can `gem install`
41
+ or reuse a bundled environment. Evaluate: hook id
42
+ `kotoshu-spell` running `kotoshu check` (requires Ruby+gem —
43
+ document honestly), vs a container hook (`language: docker_ruby`
44
+ via the published image). Ship the honest one with clear docs;
45
+ note the wasm client as the future zero-Ruby path.
46
+ - Placement: `action-kotoshu` repo (CI-adjacent) or the gem repo
47
+ (ships with the tool). Prefer the **gem repo** — the hook file
48
+ documents itself beside the CLI it calls.
49
+
50
+ ## Phases
51
+
52
+ A1 scaffold → A2 discovery + diagnostics → A3 code actions + smoke
53
+ CI → B1 hook + docs → B2 (owner) marketplace publish.
54
+
55
+ ## Owner gates
56
+
57
+ - Marketplace publish (Track A final step).
58
+ - Open VSX mirror.
59
+
60
+ ## Status
61
+
62
+ **Track A done (2026-09-05): kotoshu/kotoshu-vscode exists** - TS
63
+ extension wrapping kotoshu-lsp over stdio, diagnostics + quickfix code
64
+ actions + hover, server discovery (setting -> bundler -> PATH) with
65
+ actionable install error, CI green including a vscode-test smoke
66
+ against a misspelling fixture, vsce package artifact gated by a manifest
67
+ check. Marketplace publish remains owner-gated (publisher PAT).
68
+ **Track B done: .pre-commit-hooks.yaml shipped in gem PR #118.**
@@ -0,0 +1,81 @@
1
+ # 82 — Document ignores + CI baselines
2
+
3
+ ## Context
4
+
5
+ Two long-standing user gaps in the gem (verified 2026-09-05 — no
6
+ `disable` handling exists anywhere in `lib/kotoshu/documents/`):
7
+
8
+ 1. **Inline ignores** — a word that is intentionally "wrong"
9
+ (product names, code identifiers in prose, dialect) cannot be
10
+ suppressed per-line; users fall back to polluting the personal
11
+ dictionary.
12
+ 2. **CI baselines** — `kotoshu check` in CI fails on ALL known
13
+ errors forever; there is no way to freeze existing debt and fail
14
+ only on new errors. Every serious linter has this (ESLint
15
+ `--baseline`, RuboCop `--exclude-limit` culture, cspell
16
+ `--no-cache` + wordlists).
17
+
18
+ ## Track A — ignore directives
19
+
20
+ Syntax (comments recognized by each document format's parser):
21
+
22
+ ```
23
+ kotoshu:disable-line — suppress everything on this line
24
+ kotoshu:disable-next-line — suppress on the following line
25
+ kotoshu:disable-next-line foo bar — suppress only these words
26
+ kotoshu:disable-file / kotoshu:enable-file — block suppression
27
+ ```
28
+
29
+ - Markdown: HTML comments `<!-- kotoshu:disable-next-line -->`;
30
+ AsciiDoc: `// kotoshu:disable-next-line`; plain text: a line whose
31
+ trimmed content is the directive. Trailing directives
32
+ (`text <!-- kotoshu:disable-line -->`) supported in MD/ADOC.
33
+ - Implementation: document layer (`documents/`) attaches
34
+ `suppressions` (source ranges) during parse; the checker filters
35
+ results whose range/word matches. Suppression data flows through
36
+ the public result objects (`results/*`) so SARIF/JSON consumers
37
+ see suppressed entries either omitted or marked (omitted by
38
+ default; `--show-suppressed` lists them).
39
+ - Also recognized: the spellchecker facade (`.check`) so the gem
40
+ API gets it for free.
41
+
42
+ ## Track B — baseline files
43
+
44
+ - `kotoshu check --baseline .kotoshu-baseline.json`:
45
+ - Baseline format: canonical JSON `{file, line, word, count}` —
46
+ stable across reformatting (count-based, not position-based).
47
+ - `kotoshu baseline init` generates it from current findings.
48
+ - Semantics: an error passes if it matches the baseline
49
+ (same file+word, count not exceeded); new errors fail; a
50
+ summary line reports how many baseline entries are now stale
51
+ (file no longer has the error) so debt shrinks visibly.
52
+ - SARIF output marks suppressed entries with the baseline tag.
53
+
54
+ ## Specs
55
+
56
+ Behavior specs throughout (no doubles; real documents). Cover:
57
+ directive forms per format, word-scoped suppression, block
58
+ disable/enable nesting, baseline count semantics, stale-entry
59
+ reporting, SARIF marking.
60
+
61
+ ## Phases
62
+
63
+ A1 directive parsing + filtering → A2 per-format specs + docs page
64
+ section → B1 baseline format + commands → B2 CI docs example.
65
+
66
+ ## Owner gates
67
+
68
+ None. CHANGELOG entry under Unreleased (version = owner).
69
+
70
+ ## Status
71
+
72
+ **Implemented (2026-09-05, gem PR #118).** Shared
73
+ Documents::Suppressions scanner (disable-line, disable-next-line with
74
+ optional word list, nestable disable-file/enable-file blocks; per-format
75
+ comment detection; directive lines never spellchecked); check filters
76
+ via the :auto profile; suppressed errors ride DocumentResult with
77
+ suppressed/suppressed_by flags; --show-suppressed. Baselines: baseline
78
+ init + check --baseline with count-based budgets, stale-entry
79
+ reporting, SARIF suppression marking; serialized via lutaml-model.
80
+ 55 new examples; default suite 3410/0/26. Release cut pending owner
81
+ version decision.