kotoshu 1.0.2-aarch64-linux

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (458) hide show
  1. checksums.yaml +7 -0
  2. data/.kotoshu-baseline.json +1 -0
  3. data/.pre-commit-hooks.yaml +18 -0
  4. data/.rspec +3 -0
  5. data/.rubocop.yml +351 -0
  6. data/.yardopts +8 -0
  7. data/CHANGELOG.md +813 -0
  8. data/CLAUDE.md +176 -0
  9. data/CODE_OF_CONDUCT.md +132 -0
  10. data/Cargo.lock +359 -0
  11. data/Cargo.toml +15 -0
  12. data/LICENSE +31 -0
  13. data/README.adoc +1292 -0
  14. data/Rakefile +150 -0
  15. data/SECURITY.md +93 -0
  16. data/TODO.impl/00-cut-0.1.md +121 -0
  17. data/TODO.impl/00-cut-0.2.md +175 -0
  18. data/TODO.impl/00-vision.md +281 -0
  19. data/TODO.impl/01-hunspell-correctness.md +111 -0
  20. data/TODO.impl/02-cli-unification.md +74 -0
  21. data/TODO.impl/03-dynamic-download.md +107 -0
  22. data/TODO.impl/04-language-modules.md +89 -0
  23. data/TODO.impl/05-semantic-path.md +83 -0
  24. data/TODO.impl/06-cjk-support.md +105 -0
  25. data/TODO.impl/07-rtl-support.md +71 -0
  26. data/TODO.impl/08-grammar-engine.md +106 -0
  27. data/TODO.impl/09-integrity-security.md +90 -0
  28. data/TODO.impl/10-testing-ci.md +98 -0
  29. data/TODO.impl/100-full-feature-batch3.md +28 -0
  30. data/TODO.impl/101-int4-tier-experiment.md +32 -0
  31. data/TODO.impl/102-lid-chain.md +28 -0
  32. data/TODO.impl/103-oov-bucket-tables.md +27 -0
  33. data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
  34. data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
  35. data/TODO.impl/106-native-lid-detection.md +36 -0
  36. data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
  37. data/TODO.impl/108-ko-ne-full-feature.md +37 -0
  38. data/TODO.impl/109-ten-audit.md +26 -0
  39. data/TODO.impl/11-release-v1.md +95 -0
  40. data/TODO.impl/110-sr-latn-and-polish.md +33 -0
  41. data/TODO.impl/111-embedding-bakeoff.md +33 -0
  42. data/TODO.impl/112-deployability-and-gating.md +26 -0
  43. data/TODO.impl/113-language-packs.md +26 -0
  44. data/TODO.impl/114-typo-biencoder-thread.md +33 -0
  45. data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
  46. data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
  47. data/TODO.impl/117-deterministic-frequency-tiers.md +110 -0
  48. data/TODO.impl/118-server-gem-floor.md +33 -0
  49. data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
  50. data/TODO.impl/120-setup-metadata-refresh.md +35 -0
  51. data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
  52. data/TODO.impl/122-pack-compression-kpk2.md +66 -0
  53. data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
  54. data/TODO.impl/124-owner-residue.md +19 -0
  55. data/TODO.impl/125-consumer-floor-guards.md +43 -0
  56. data/TODO.impl/126-tier-error-budgets.md +33 -0
  57. data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
  58. data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
  59. data/TODO.impl/129-pt-fr-corpora.md +23 -0
  60. data/TODO.impl/130-news-truth-0912.md +23 -0
  61. data/TODO.impl/131-hybrid-productization.md +42 -0
  62. data/TODO.impl/132-techniques-page-and-prose.md +27 -0
  63. data/TODO.impl/133-rust-first-distribution.md +68 -0
  64. data/TODO.impl/20-encapsulation-cleanup.md +47 -0
  65. data/TODO.impl/21-autoload-migration.md +64 -0
  66. data/TODO.impl/22-url-pin-single-source-of-truth.md +120 -0
  67. data/TODO.impl/23-fix-url-defaults.md +57 -0
  68. data/TODO.impl/24-cli-auto-setup-prompt.md +76 -0
  69. data/TODO.impl/25-soft-onnxruntime-require.md +75 -0
  70. data/TODO.impl/26-onnx-vocab-json-generation.md +96 -0
  71. data/TODO.impl/27-end-to-end-smoke-spec.md +113 -0
  72. data/TODO.impl/28-readme-quickstart-verification.md +68 -0
  73. data/TODO.impl/29-status-command.md +84 -0
  74. data/TODO.impl/30-language-auto-detection.md +74 -0
  75. data/TODO.impl/31-download-progress.md +58 -0
  76. data/TODO.impl/32-sarif-output-verification.md +72 -0
  77. data/TODO.impl/33-audit-log-rotation.md +54 -0
  78. data/TODO.impl/34-cache-eviction.md +59 -0
  79. data/TODO.impl/35-shell-completion.md +61 -0
  80. data/TODO.impl/36-cleanup-regressions-0.3.1.md +134 -0
  81. data/TODO.impl/37-hunspell-correctness-tier2.md +203 -0
  82. data/TODO.impl/38-onnx-semantic-gating.md +218 -0
  83. data/TODO.impl/39-tier3-and-beyond.md +232 -0
  84. data/TODO.impl/40-spec-drift-cleanup.md +134 -0
  85. data/TODO.impl/50-structure-aware-document-api.md +194 -0
  86. data/TODO.impl/51-grammar-engine-expansion.md +75 -0
  87. data/TODO.impl/52-multi-language-documents.md +95 -0
  88. data/TODO.impl/53-document-plugin-architecture.md +94 -0
  89. data/TODO.impl/54-cjk-support.md +117 -0
  90. data/TODO.impl/55-rtl-support.md +109 -0
  91. data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
  92. data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
  93. data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
  94. data/TODO.impl/59-yard-api-documentation.md +75 -0
  95. data/TODO.impl/60-lsp-server.md +174 -0
  96. data/TODO.impl/61-editor-ecosystem.md +167 -0
  97. data/TODO.impl/62-framework-integrations.md +159 -0
  98. data/TODO.impl/63-web-playground-wasm.md +186 -0
  99. data/TODO.impl/64-http-api-and-sdks.md +191 -0
  100. data/TODO.impl/65-universal-kotoshu.md +251 -0
  101. data/TODO.impl/66-kotoshu-core.md +169 -0
  102. data/TODO.impl/67-kotoshu-rs-and-access-libraries.md +137 -0
  103. data/TODO.impl/68-sota-adoption.md +152 -0
  104. data/TODO.impl/69-sota-typo-eval.md +74 -0
  105. data/TODO.impl/70-sota-confidence-cascade.md +59 -0
  106. data/TODO.impl/71-sota-nested-model.md +63 -0
  107. data/TODO.impl/72-model2vec-evaluation.md +49 -0
  108. data/TODO.impl/73-rl-reranker.md +42 -0
  109. data/TODO.impl/74-distilled-reranker.md +36 -0
  110. data/TODO.impl/75-llm-evaluator.md +35 -0
  111. data/TODO.impl/76-optional-llm-backend.md +37 -0
  112. data/TODO.impl/77-model-coverage-expansion.md +96 -0
  113. data/TODO.impl/78-in-browser-playground.md +69 -0
  114. data/TODO.impl/79-news-and-releases.md +60 -0
  115. data/TODO.impl/80-ecosystem-docs.md +56 -0
  116. data/TODO.impl/81-editor-reach.md +68 -0
  117. data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
  118. data/TODO.impl/83-models-batch-2.md +56 -0
  119. data/TODO.impl/84-language-modules-wave2.md +75 -0
  120. data/TODO.impl/85-wasm-model-api.md +48 -0
  121. data/TODO.impl/86-site-truth-wave2.md +45 -0
  122. data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
  123. data/TODO.impl/88-cli-directory-mode.md +54 -0
  124. data/TODO.impl/89-framework-integrations-exec.md +59 -0
  125. data/TODO.impl/90-gem-correctness-residues.md +44 -0
  126. data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
  127. data/TODO.impl/92-tier-mirror-cors.md +88 -0
  128. data/TODO.impl/93-python-wheels.md +32 -0
  129. data/TODO.impl/94-site-integrations-docs.md +35 -0
  130. data/TODO.impl/95-editor-action-polish.md +27 -0
  131. data/TODO.impl/96-server-semantic-models.md +50 -0
  132. data/TODO.impl/97-yard-execution.md +73 -0
  133. data/TODO.impl/98-campaign-cleanup.md +78 -0
  134. data/TODO.impl/99-site-truth-performance-docs.md +31 -0
  135. data/conformance/vectors.jsonl +2630 -0
  136. data/data/en/grammar/rules.yaml +851 -0
  137. data/docs/STABILITY-1.0-AUDIT.md +251 -0
  138. data/docs/STABILITY-1.0.md +137 -0
  139. data/docs/cache-architecture.md +502 -0
  140. data/docs/verification-evidence/correct3/pr_body.md +37 -0
  141. data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
  142. data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
  143. data/docs/verification-evidence/correct3/sv.aff +856 -0
  144. data/docs/verification-evidence/correct3/sv.dic +153717 -0
  145. data/examples/01_basic_word_checking.rb +38 -0
  146. data/examples/02_text_document_checking.rb +77 -0
  147. data/examples/03_dictionary_backends.rb +137 -0
  148. data/examples/04_trie_data_structure.rb +146 -0
  149. data/examples/05_suggestion_algorithms.rb +239 -0
  150. data/examples/06_configuration_advanced.rb +287 -0
  151. data/examples/07_multi_language_dictionaries.rb +278 -0
  152. data/exe/kotoshu +6 -0
  153. data/lib/kotoshu/algorithms/capitalization.rb +276 -0
  154. data/lib/kotoshu/algorithms/edit_distance.rb +128 -0
  155. data/lib/kotoshu/algorithms/lookup/affix_form.rb +132 -0
  156. data/lib/kotoshu/algorithms/lookup/compound_checks.rb +209 -0
  157. data/lib/kotoshu/algorithms/lookup/compound_form.rb +50 -0
  158. data/lib/kotoshu/algorithms/lookup.rb +841 -0
  159. data/lib/kotoshu/algorithms/ngram_suggest.rb +367 -0
  160. data/lib/kotoshu/algorithms/permutations.rb +283 -0
  161. data/lib/kotoshu/algorithms/phonet_suggest.rb +162 -0
  162. data/lib/kotoshu/algorithms/soundex.rb +92 -0
  163. data/lib/kotoshu/algorithms/suggest.rb +794 -0
  164. data/lib/kotoshu/algorithms.rb +23 -0
  165. data/lib/kotoshu/analyzers/semantic_analyzer.rb +405 -0
  166. data/lib/kotoshu/analyzers.rb +8 -0
  167. data/lib/kotoshu/baseline/application.rb +18 -0
  168. data/lib/kotoshu/baseline/entry.rb +33 -0
  169. data/lib/kotoshu/baseline/store.rb +175 -0
  170. data/lib/kotoshu/baseline.rb +20 -0
  171. data/lib/kotoshu/cache/base_cache.rb +778 -0
  172. data/lib/kotoshu/cache/cache.rb +91 -0
  173. data/lib/kotoshu/cache/eviction_policy.rb +66 -0
  174. data/lib/kotoshu/cache/frequency_cache.rb +283 -0
  175. data/lib/kotoshu/cache/language_cache.rb +511 -0
  176. data/lib/kotoshu/cache/lookup_cache.rb +166 -0
  177. data/lib/kotoshu/cache/model_cache.rb +1147 -0
  178. data/lib/kotoshu/cache/model_registry.rb +109 -0
  179. data/lib/kotoshu/cache/staged_languages.rb +45 -0
  180. data/lib/kotoshu/cache/suggestion_cache.rb +113 -0
  181. data/lib/kotoshu/cache.rb +49 -0
  182. data/lib/kotoshu/cli/auto_setup.rb +71 -0
  183. data/lib/kotoshu/cli/baseline_command.rb +119 -0
  184. data/lib/kotoshu/cli/batch_reporter.rb +321 -0
  185. data/lib/kotoshu/cli/cache_command.rb +282 -0
  186. data/lib/kotoshu/cli/completions_command.rb +210 -0
  187. data/lib/kotoshu/cli/dict_command.rb +64 -0
  188. data/lib/kotoshu/cli/directory_check.rb +146 -0
  189. data/lib/kotoshu/cli/directory_walker.rb +132 -0
  190. data/lib/kotoshu/cli/display_formatter.rb +429 -0
  191. data/lib/kotoshu/cli/errors.rb +36 -0
  192. data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
  193. data/lib/kotoshu/cli/interactive_reviewer.rb +314 -0
  194. data/lib/kotoshu/cli/language_resolver.rb +91 -0
  195. data/lib/kotoshu/cli/navigation_manager.rb +273 -0
  196. data/lib/kotoshu/cli/personal_command.rb +113 -0
  197. data/lib/kotoshu/cli/progress_reporter.rb +114 -0
  198. data/lib/kotoshu/cli/status_report.rb +130 -0
  199. data/lib/kotoshu/cli.rb +880 -0
  200. data/lib/kotoshu/components/passthrough_spell_checker.rb +70 -0
  201. data/lib/kotoshu/components/pos_tagger.rb +98 -0
  202. data/lib/kotoshu/components/spell_checker.rb +73 -0
  203. data/lib/kotoshu/components/synthesizer.rb +60 -0
  204. data/lib/kotoshu/components/tokenizer.rb +58 -0
  205. data/lib/kotoshu/components/whitespace_tokenizer.rb +94 -0
  206. data/lib/kotoshu/components.rb +13 -0
  207. data/lib/kotoshu/configuration/builder.rb +207 -0
  208. data/lib/kotoshu/configuration/resolver.rb +124 -0
  209. data/lib/kotoshu/configuration.rb +829 -0
  210. data/lib/kotoshu/conformance_exporter.rb +256 -0
  211. data/lib/kotoshu/conformance_runner.rb +176 -0
  212. data/lib/kotoshu/core/exceptions.rb +186 -0
  213. data/lib/kotoshu/core/indexed_dictionary.rb +293 -0
  214. data/lib/kotoshu/core/models/affix_rule.rb +262 -0
  215. data/lib/kotoshu/core/models/result/document_result.rb +132 -0
  216. data/lib/kotoshu/core/models/result/word_result.rb +131 -0
  217. data/lib/kotoshu/core/models/word.rb +131 -0
  218. data/lib/kotoshu/core/trie/builder.rb +119 -0
  219. data/lib/kotoshu/core/trie/node.rb +94 -0
  220. data/lib/kotoshu/core/trie/trie.rb +249 -0
  221. data/lib/kotoshu/core.rb +28 -0
  222. data/lib/kotoshu/data/common_words/de.yml +1800 -0
  223. data/lib/kotoshu/data/common_words/en.yml +1215 -0
  224. data/lib/kotoshu/data/common_words/es.yml +750 -0
  225. data/lib/kotoshu/data/common_words/fr.yml +1015 -0
  226. data/lib/kotoshu/data/common_words/pt.yml +870 -0
  227. data/lib/kotoshu/data/common_words/ru.yml +484 -0
  228. data/lib/kotoshu/data/common_words_loader.rb +141 -0
  229. data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
  230. data/lib/kotoshu/data/frozen_kelly.rb +38 -0
  231. data/lib/kotoshu/data.rb +8 -0
  232. data/lib/kotoshu/data_structures/bloom_filter.rb +188 -0
  233. data/lib/kotoshu/data_structures.rb +8 -0
  234. data/lib/kotoshu/debug_logger.rb +146 -0
  235. data/lib/kotoshu/debug_mode.rb +136 -0
  236. data/lib/kotoshu/defaults.rb +84 -0
  237. data/lib/kotoshu/dictionaries/catalog.rb +814 -0
  238. data/lib/kotoshu/dictionaries.rb +8 -0
  239. data/lib/kotoshu/dictionary/base.rb +324 -0
  240. data/lib/kotoshu/dictionary/cspell.rb +243 -0
  241. data/lib/kotoshu/dictionary/custom.rb +225 -0
  242. data/lib/kotoshu/dictionary/hunspell.rb +520 -0
  243. data/lib/kotoshu/dictionary/plain_text.rb +302 -0
  244. data/lib/kotoshu/dictionary/repository.rb +246 -0
  245. data/lib/kotoshu/dictionary/unified.rb +236 -0
  246. data/lib/kotoshu/dictionary/unix_words.rb +218 -0
  247. data/lib/kotoshu/dictionary.rb +15 -0
  248. data/lib/kotoshu/documents/document.rb +120 -0
  249. data/lib/kotoshu/documents/plain_text_document.rb +73 -0
  250. data/lib/kotoshu/documents/source_position.rb +45 -0
  251. data/lib/kotoshu/documents/source_range.rb +76 -0
  252. data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
  253. data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
  254. data/lib/kotoshu/documents/suppressions.rb +44 -0
  255. data/lib/kotoshu/documents/text_node.rb +49 -0
  256. data/lib/kotoshu/documents.rb +178 -0
  257. data/lib/kotoshu/embeddings/embedding_pipeline.rb +238 -0
  258. data/lib/kotoshu/embeddings/lru_cache.rb +244 -0
  259. data/lib/kotoshu/embeddings/onnx_runtime_model.rb +411 -0
  260. data/lib/kotoshu/embeddings/protocol.rb +96 -0
  261. data/lib/kotoshu/embeddings/protocols.rb +23 -0
  262. data/lib/kotoshu/embeddings/registry.rb +184 -0
  263. data/lib/kotoshu/embeddings/search.rb +204 -0
  264. data/lib/kotoshu/embeddings/similarity_engine.rb +278 -0
  265. data/lib/kotoshu/embeddings/similarity_search.rb +328 -0
  266. data/lib/kotoshu/embeddings/vocabulary.rb +306 -0
  267. data/lib/kotoshu/embeddings.rb +87 -0
  268. data/lib/kotoshu/fluent_checker.rb +94 -0
  269. data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +48 -0
  270. data/lib/kotoshu/grammar/pattern_matchers/double_negative_matcher.rb +179 -0
  271. data/lib/kotoshu/grammar/pattern_matchers/phrase_matcher.rb +86 -0
  272. data/lib/kotoshu/grammar/pattern_matchers/possessive_context_matcher.rb +75 -0
  273. data/lib/kotoshu/grammar/pattern_matchers/possessive_contraction_matcher.rb +87 -0
  274. data/lib/kotoshu/grammar/pattern_matchers/sentence_start_matcher.rb +84 -0
  275. data/lib/kotoshu/grammar/pattern_matchers/vowel_sound_matcher.rb +81 -0
  276. data/lib/kotoshu/grammar/pattern_matchers/word_list_matcher.rb +66 -0
  277. data/lib/kotoshu/grammar/rule.rb +99 -0
  278. data/lib/kotoshu/grammar/rule_engine.rb +96 -0
  279. data/lib/kotoshu/grammar/rule_loader.rb +30 -0
  280. data/lib/kotoshu/grammar.rb +24 -0
  281. data/lib/kotoshu/integrity/audit_log.rb +167 -0
  282. data/lib/kotoshu/integrity/manifest.rb +116 -0
  283. data/lib/kotoshu/integrity/net_http.rb +51 -0
  284. data/lib/kotoshu/integrity/rotation_policy.rb +88 -0
  285. data/lib/kotoshu/integrity.rb +26 -0
  286. data/lib/kotoshu/jekyll.rb +101 -0
  287. data/lib/kotoshu/keyboard/layout.rb +116 -0
  288. data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
  289. data/lib/kotoshu/keyboard/layouts/azerty.rb +55 -0
  290. data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
  291. data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
  292. data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
  293. data/lib/kotoshu/keyboard/layouts/dvorak.rb +54 -0
  294. data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
  295. data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
  296. data/lib/kotoshu/keyboard/layouts/hrsl.rb +94 -0
  297. data/lib/kotoshu/keyboard/layouts/jcuken.rb +57 -0
  298. data/lib/kotoshu/keyboard/layouts/latin.rb +266 -0
  299. data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
  300. data/lib/kotoshu/keyboard/layouts/qwerty.rb +52 -0
  301. data/lib/kotoshu/keyboard/layouts/qwertz.rb +55 -0
  302. data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
  303. data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
  304. data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
  305. data/lib/kotoshu/keyboard/registry.rb +201 -0
  306. data/lib/kotoshu/keyboard.rb +96 -0
  307. data/lib/kotoshu/kotoshu_native.so +0 -0
  308. data/lib/kotoshu/language/detector.rb +242 -0
  309. data/lib/kotoshu/language/identifier.rb +376 -0
  310. data/lib/kotoshu/language/languages/base.rb +271 -0
  311. data/lib/kotoshu/language/lid_detector.rb +188 -0
  312. data/lib/kotoshu/language/normalizer/arabic.rb +113 -0
  313. data/lib/kotoshu/language/normalizer/base.rb +133 -0
  314. data/lib/kotoshu/language/normalizer/greek.rb +63 -0
  315. data/lib/kotoshu/language/normalizer/hebrew.rb +72 -0
  316. data/lib/kotoshu/language/normalizer/persian.rb +36 -0
  317. data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
  318. data/lib/kotoshu/language/registry.rb +209 -0
  319. data/lib/kotoshu/language/resources/ar/common_words.txt +6753 -0
  320. data/lib/kotoshu/language/resources/ar/confusion_sets.txt +11 -0
  321. data/lib/kotoshu/language/resources/de/common_words.txt +10003 -0
  322. data/lib/kotoshu/language/resources/de/confusion_sets.txt +246 -0
  323. data/lib/kotoshu/language/resources/en/common_words.txt +9979 -0
  324. data/lib/kotoshu/language/resources/en/confusion_sets.txt +871 -0
  325. data/lib/kotoshu/language/resources/es/common_words.txt +9992 -0
  326. data/lib/kotoshu/language/resources/es/confusion_sets.txt +17 -0
  327. data/lib/kotoshu/language/resources/fr/common_words.txt +9993 -0
  328. data/lib/kotoshu/language/resources/fr/confusion_sets.txt +76 -0
  329. data/lib/kotoshu/language/resources/pt/common_words.txt +9977 -0
  330. data/lib/kotoshu/language/resources/pt/confusion_sets.txt +18 -0
  331. data/lib/kotoshu/language/resources/ru/common_words.txt +9951 -0
  332. data/lib/kotoshu/language/resources/ru/confusion_sets.txt +5 -0
  333. data/lib/kotoshu/language/script.rb +128 -0
  334. data/lib/kotoshu/language/segmenter.rb +122 -0
  335. data/lib/kotoshu/language/suika.rb +38 -0
  336. data/lib/kotoshu/language/tokenizer/base.rb +187 -0
  337. data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
  338. data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
  339. data/lib/kotoshu/language/tokenizer/french_tokenizer.rb +172 -0
  340. data/lib/kotoshu/language/tokenizer/german_tokenizer.rb +41 -0
  341. data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
  342. data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
  343. data/lib/kotoshu/language/tokenizer/japanese_tokenizer.rb +55 -0
  344. data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +153 -0
  345. data/lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb +160 -0
  346. data/lib/kotoshu/language/tokenizer/russian_tokenizer.rb +93 -0
  347. data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
  348. data/lib/kotoshu/language/tokenizer/spanish_tokenizer.rb +122 -0
  349. data/lib/kotoshu/language.rb +127 -0
  350. data/lib/kotoshu/languages/ar/language.rb +98 -0
  351. data/lib/kotoshu/languages/bg/language.rb +37 -0
  352. data/lib/kotoshu/languages/ca/language.rb +22 -0
  353. data/lib/kotoshu/languages/cs/language.rb +22 -0
  354. data/lib/kotoshu/languages/da/language.rb +22 -0
  355. data/lib/kotoshu/languages/de/language.rb +554 -0
  356. data/lib/kotoshu/languages/el/language.rb +50 -0
  357. data/lib/kotoshu/languages/en/language.rb +470 -0
  358. data/lib/kotoshu/languages/es/language.rb +476 -0
  359. data/lib/kotoshu/languages/et/language.rb +24 -0
  360. data/lib/kotoshu/languages/fa/language.rb +91 -0
  361. data/lib/kotoshu/languages/fr/language.rb +505 -0
  362. data/lib/kotoshu/languages/he/language.rb +98 -0
  363. data/lib/kotoshu/languages/hr/language.rb +24 -0
  364. data/lib/kotoshu/languages/hu/language.rb +22 -0
  365. data/lib/kotoshu/languages/id/language.rb +25 -0
  366. data/lib/kotoshu/languages/it/language.rb +22 -0
  367. data/lib/kotoshu/languages/ja/language.rb +476 -0
  368. data/lib/kotoshu/languages/ko/language.rb +56 -0
  369. data/lib/kotoshu/languages/latin_base.rb +53 -0
  370. data/lib/kotoshu/languages/lt/language.rb +24 -0
  371. data/lib/kotoshu/languages/lv/language.rb +24 -0
  372. data/lib/kotoshu/languages/nb/language.rb +25 -0
  373. data/lib/kotoshu/languages/ne/language.rb +60 -0
  374. data/lib/kotoshu/languages/nl/language.rb +22 -0
  375. data/lib/kotoshu/languages/nn/language.rb +26 -0
  376. data/lib/kotoshu/languages/pl/language.rb +22 -0
  377. data/lib/kotoshu/languages/pt/language.rb +439 -0
  378. data/lib/kotoshu/languages/ro/language.rb +22 -0
  379. data/lib/kotoshu/languages/ru/language.rb +420 -0
  380. data/lib/kotoshu/languages/sk/language.rb +25 -0
  381. data/lib/kotoshu/languages/sl/language.rb +25 -0
  382. data/lib/kotoshu/languages/sr/language.rb +39 -0
  383. data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
  384. data/lib/kotoshu/languages/sv/language.rb +22 -0
  385. data/lib/kotoshu/languages/tr/language.rb +36 -0
  386. data/lib/kotoshu/languages/uk/language.rb +38 -0
  387. data/lib/kotoshu/languages/vi/language.rb +22 -0
  388. data/lib/kotoshu/languages.rb +64 -0
  389. data/lib/kotoshu/metrics_collector.rb +238 -0
  390. data/lib/kotoshu/metrics_module.rb +121 -0
  391. data/lib/kotoshu/models/context.rb +119 -0
  392. data/lib/kotoshu/models/embedding_model.rb +181 -0
  393. data/lib/kotoshu/models/fasttext_model.rb +212 -0
  394. data/lib/kotoshu/models/nearest_neighbor.rb +96 -0
  395. data/lib/kotoshu/models/onnx_model.rb +330 -0
  396. data/lib/kotoshu/models/semantic_error.rb +204 -0
  397. data/lib/kotoshu/models/suggestion.rb +106 -0
  398. data/lib/kotoshu/models/word_embedding.rb +97 -0
  399. data/lib/kotoshu/models.rb +27 -0
  400. data/lib/kotoshu/multi_language_checker.rb +127 -0
  401. data/lib/kotoshu/native.rb +83 -0
  402. data/lib/kotoshu/native_backend.rb +131 -0
  403. data/lib/kotoshu/paths.rb +79 -0
  404. data/lib/kotoshu/personal_dictionary.rb +105 -0
  405. data/lib/kotoshu/plugins/plugin.rb +61 -0
  406. data/lib/kotoshu/plugins/registry.rb +118 -0
  407. data/lib/kotoshu/plugins.rb +9 -0
  408. data/lib/kotoshu/project_config.rb +76 -0
  409. data/lib/kotoshu/readers/aff_data.rb +530 -0
  410. data/lib/kotoshu/readers/aff_reader.rb +393 -0
  411. data/lib/kotoshu/readers/condition_checker.rb +125 -0
  412. data/lib/kotoshu/readers/dic_reader.rb +208 -0
  413. data/lib/kotoshu/readers/file_reader.rb +377 -0
  414. data/lib/kotoshu/readers/lookup_builder.rb +584 -0
  415. data/lib/kotoshu/readers/readers.rb +17 -0
  416. data/lib/kotoshu/readers.rb +25 -0
  417. data/lib/kotoshu/resource_bundle.rb +55 -0
  418. data/lib/kotoshu/resource_manager.rb +600 -0
  419. data/lib/kotoshu/results/result.rb +165 -0
  420. data/lib/kotoshu/results.rb +8 -0
  421. data/lib/kotoshu/rspec.rb +163 -0
  422. data/lib/kotoshu/scripts/fasttext_to_onnx.py +275 -0
  423. data/lib/kotoshu/source_registry.rb +104 -0
  424. data/lib/kotoshu/spellchecker/parallel_checker.rb +90 -0
  425. data/lib/kotoshu/spellchecker.rb +406 -0
  426. data/lib/kotoshu/string_metrics.rb +153 -0
  427. data/lib/kotoshu/suggestions/context.rb +43 -0
  428. data/lib/kotoshu/suggestions/frequency_provider.rb +80 -0
  429. data/lib/kotoshu/suggestions/generator.rb +169 -0
  430. data/lib/kotoshu/suggestions/pipeline.rb +135 -0
  431. data/lib/kotoshu/suggestions/semantic_cascade.rb +73 -0
  432. data/lib/kotoshu/suggestions/strategies/base_strategy.rb +315 -0
  433. data/lib/kotoshu/suggestions/strategies/composite_strategy.rb +197 -0
  434. data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +631 -0
  435. data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +207 -0
  436. data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +165 -0
  437. data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +284 -0
  438. data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +317 -0
  439. data/lib/kotoshu/suggestions/strategies/symspell_strategy.rb +270 -0
  440. data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
  441. data/lib/kotoshu/suggestions/suggestion.rb +154 -0
  442. data/lib/kotoshu/suggestions/suggestion_set.rb +270 -0
  443. data/lib/kotoshu/suggestions/sweep_index.rb +137 -0
  444. data/lib/kotoshu/suggestions.rb +29 -0
  445. data/lib/kotoshu/tasks/check_task.rb +104 -0
  446. data/lib/kotoshu/tasks.rb +11 -0
  447. data/lib/kotoshu/validators/spelling_validator.rb +101 -0
  448. data/lib/kotoshu/validators.rb +9 -0
  449. data/lib/kotoshu/version.rb +5 -0
  450. data/lib/kotoshu.rb +538 -0
  451. data/rakelib/compile.rake +10 -0
  452. data/script/validate_all_dictionaries.rb +444 -0
  453. data/scripts/build_native_gem.rb +45 -0
  454. data/scripts/extract_eval_grids.rb +37 -0
  455. data/scripts/generate_frozen_tiers.rb +44 -0
  456. data/sig/kotoshu.rbs +435 -0
  457. data/test_oop.rb +79 -0
  458. metadata +546 -0
@@ -0,0 +1,113 @@
1
+ # 58 — T5.2: Configuration as data, not singleton
2
+
3
+ ## Goal
4
+
5
+ Drop the `Configuration.instance` process-wide singleton. The
6
+ singleton forces every test to call `Configuration.reset` in
7
+ before/after hooks, prevents per-document configuration, and makes
8
+ parallel test execution unsafe.
9
+
10
+ ## Current state
11
+
12
+ `lib/kotoshu/configuration.rb` defines:
13
+
14
+ ```ruby
15
+ def self.instance
16
+ @instance ||= new
17
+ end
18
+
19
+ def self.reset
20
+ @instance = nil
21
+ end
22
+ ```
23
+
24
+ Every caller does:
25
+
26
+ ```ruby
27
+ config = Kotoshu::Configuration.instance
28
+ config.cache_path = "/tmp/foo"
29
+ ```
30
+
31
+ This is referenced from:
32
+ - `lib/kotoshu/spellchecker.rb`
33
+ - `lib/kotoshu/resource_manager.rb`
34
+ - `lib/kotoshu/cache/*.rb` (all cache classes)
35
+ - `lib/kotoshu/cli.rb`
36
+ - `lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb` (was; now goes through FrequencyProvider)
37
+
38
+ ## Target design
39
+
40
+ ```ruby
41
+ # Configuration is a plain class. Callers create instances.
42
+ config = Kotoshu::Configuration.new(cache_path: "/tmp/foo")
43
+
44
+ # Spellchecker accepts config explicitly.
45
+ speller = Kotoshu::Spellchecker.new(config: config, resource_bundle: bundle)
46
+
47
+ # The Kotoshu module facade retains a process-default for ergonomics.
48
+ Kotoshu.configure { |c| c.cache_path = "/tmp/foo" }
49
+ Kotoshu.correct?("hello") # uses process-default config
50
+ ```
51
+
52
+ The facade keeps the simple API. Under the hood, it manages a
53
+ process-default `Configuration` instance instead of a global
54
+ singleton via `Configuration.instance`.
55
+
56
+ ## Phases
57
+
58
+ ### Phase A — Spellchecker accepts config kwarg
59
+
60
+ `Spellchecker.new(resource_bundle:, config:)` accepts a Configuration
61
+ instance. Default: `Kotoshu.configuration` (process-default managed
62
+ by the facade, not by `Configuration.instance`).
63
+
64
+ This is a non-breaking change: existing callers that omit `config:`
65
+ get the process default.
66
+
67
+ ### Phase B — Migrate callers off Configuration.instance
68
+
69
+ Walk through `resource_manager.rb`, `cache/*.rb`, etc. Replace
70
+ `Configuration.instance` with config passed in via constructor or
71
+ method parameter. Each file becomes a separate PR (small, focused).
72
+
73
+ ### Phase C — Deprecate Configuration.instance
74
+
75
+ Mark `Configuration.instance` as deprecated with a warning. Keep
76
+ the method for one release cycle, then remove.
77
+
78
+ ## Acceptance criteria
79
+
80
+ - [ ] `Spellchecker.new` accepts `config:` kwarg.
81
+ - [ ] No production code calls `Configuration.instance` directly
82
+ (only the facade does, for the process default).
83
+ - [ ] Tests no longer call `Configuration.reset` — they construct
84
+ their own instance and pass it explicitly.
85
+ - [ ] `parallelize_me!` can be added to the spec suite without
86
+ flaky failures.
87
+
88
+ ## Why this matters
89
+
90
+ - **Test isolation.** Today, every test that touches configuration
91
+ has to reset the singleton. Forgetting the reset causes cascade
92
+ failures.
93
+ - **Per-document config.** Multi-document checking (TODO 52) wants
94
+ per-document config (different personal dicts, different ignore
95
+ lists). The singleton blocks this.
96
+ - **Parallelism.** `ParallelChecker` shares strategy instances
97
+ across threads today; if strategies read `Configuration.instance`,
98
+ that's a race waiting to happen once we add threads that use
99
+ different configs.
100
+
101
+ ## Dependencies
102
+
103
+ - **Blocked by:** none.
104
+ - **Blocks:** TODO 52 (multi-language document checking) benefits
105
+ from per-segment config. TODO 56 T4.2 (CI matrix) — parallel
106
+ specs become safe.
107
+
108
+ ## Out of scope
109
+
110
+ - Changing the Configuration SCHEMA (already data-driven).
111
+ - Adding new configuration options.
112
+ - Removing the `Kotoshu.configure` facade method (kept for
113
+ ergonomics).
@@ -0,0 +1,75 @@
1
+ # 59 — YARD API documentation
2
+
3
+ ## Goal
4
+
5
+ Public kotoshu APIs are essentially undocumented. The codebase has
6
+ YARD-style docstrings on many methods but no published API docs,
7
+ and many public methods lack docstrings entirely.
8
+
9
+ ## Current state
10
+
11
+ - `bundle exec yard` works locally (if yard is installed)
12
+ - No `docs/` build pipeline in CI
13
+ - No gh-pages deployment
14
+ - Many public classes/methods lack `@param` / `@return` / `@example`
15
+ tags.
16
+
17
+ The README.adoc is user-facing; YARD docs are the contributor/dev
18
+ surface.
19
+
20
+ ## What needs doing
21
+
22
+ ### Phase 1 — Inventory public API
23
+
24
+ Identify which classes/methods are part of the public API (vs
25
+ internal). Heuristic:
26
+ - `Kotoshu.*` module methods are public.
27
+ - Classes in `lib/kotoshu/` whose methods are documented as
28
+ public-facing (`Kotoshu.correct?`, `Spellchecker#check`, etc.)
29
+ are public.
30
+ - Classes marked `@api private` or in clearly internal namespaces
31
+ (e.g., `Grammar::PatternMatchers::*`) are internal.
32
+
33
+ Output: a list of public API methods that need full YARD docs.
34
+
35
+ ### Phase 2 — Write docstrings
36
+
37
+ For each public method:
38
+ - Add `@param` for every parameter.
39
+ - Add `@return` describing the return value.
40
+ - Add `@example` for non-trivial methods.
41
+ - Add `@raise` for any raised exceptions.
42
+ - Mark internal helpers `@api private`.
43
+
44
+ ### Phase 3 — CI integration
45
+
46
+ - Add a `docs` job to `.github/workflows/ci.yml` that runs
47
+ `bundle exec yard --no-output`. Fails on warnings.
48
+ - Optional: deploy to gh-pages on tag pushes.
49
+
50
+ ## Acceptance criteria
51
+
52
+ - [ ] Every public method has YARD docstring with `@param` and
53
+ `@return`.
54
+ - [ ] `bundle exec yard --no-output` exits 0 (no warnings).
55
+ - [ ] CI runs YARD check on every PR.
56
+ - [ ] (Optional) gh-pages site at `kotoshu.github.io/kotoshu` or
57
+ similar.
58
+
59
+ ## Why this matters
60
+
61
+ - **Discoverability.** New users can't easily learn the API from
62
+ source alone.
63
+ - **Editoring.** IDE intellisense uses YARD types when available.
64
+ - **API contract.** Docstrings force authors to think about what
65
+ the method promises.
66
+
67
+ ## Dependencies
68
+
69
+ - **Blocked by:** none (independent of code changes).
70
+ - **Blocks:** nothing (long-running cleanup).
71
+
72
+ ## Out of scope
73
+
74
+ - Tutorial-style documentation (lives in README.adoc and `docs/`).
75
+ - Architecture decision records (lives in `docs/adr/` if added).
@@ -0,0 +1,174 @@
1
+ # 60 — LSP server (`kotoshu-lsp`)
2
+
3
+ Expands T6.1 in `TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md`
4
+ into a promotable plan with concrete acceptance criteria.
5
+
6
+ ## Goal
7
+
8
+ A Language Server Protocol implementation that ships Kotoshu diagnostics
9
+ into every editor with LSP support — VS Code, Neovim, Emacs (`lsp-mode` /
10
+ `eglot`), JetBrains (via the LSP connector), Sublime Text (LSP package),
11
+ VSCodium, Helix, Zed. One server, every editor.
12
+
13
+ The server is a **separate gem** (`kotoshu-lsp`), per the five-repo
14
+ workspace model — it depends on `kotoshu` but ships its own release
15
+ cadence. The library gem stays focused on spell-checking; the LSP gem
16
+ holds the editor-facing surface.
17
+
18
+ ## Why
19
+
20
+ A spellchecker reaches daily users through their editor, not through a
21
+ CLI they remember to run. Today Kotoshu's reach is bounded by:
22
+
23
+ - The CLI is a manual, batch-style invocation. Real users want inline
24
+ squiggly underlines and one-keystroke fixes.
25
+ - SARIF / JSON output targets CI/CD pipelines, not interactive editing.
26
+ - The Hunspell lineage has no editor story; LanguageTool's LSP server
27
+ (`ltex-ls`) is the de-facto incumbent. Kotoshu needs parity.
28
+
29
+ LSP is the single highest-leverage move because:
30
+
31
+ - **One server, N editors.** Implement once, every LSP-capable editor
32
+ picks it up.
33
+ - **Reusable infrastructure.** All the heavy lifting (document parsing,
34
+ suggestion generation, language detection, semantic reranking) is
35
+ already in the library. The LSP server is a thin protocol adapter.
36
+ - **Compound value with semantic path.** ONNX reranking (plan 05) shows
37
+ its worth when suggestions are surfaced inline, where users actually
38
+ pick them.
39
+
40
+ ## Tasks
41
+
42
+ ### Phase 1 — Minimum viable diagnostics
43
+
44
+ 1. **New repo `kotoshu/kotoshu-lsp`.** Skeleton gemspec depending on
45
+ `kotoshu` and an LSP library. Candidate LSP libraries to evaluate:
46
+ - `language_server-protocol` (TypeScript-style typed protocol in Ruby)
47
+ - Roll a minimal JSON-RPC + LSP framing layer ourselves
48
+ Pick one and document the choice in the gem README.
49
+ 2. **stdio transport.** LSP servers communicate over stdio JSON-RPC.
50
+ Implement the framing (`Content-Length: N\r\n\r\n{json}`), the
51
+ `initialize` / `initialized` / `shutdown` / `exit` handshake, and
52
+ `textDocument/didOpen`, `textDocument/didChange`,
53
+ `textDocument/didClose`.
54
+ 3. **Diagnostics on save (`textDocument/publishDiagnostics`).** Wire
55
+ `Kotoshu.check_file` (or `Kotoshu.check(text, language:)`) into a
56
+ document-changed handler. Map `DocumentResult#errors` to
57
+ `LSP::Diagnostic[]` with `range`, `message`, `severity = warning`,
58
+ `source = "kotoshu"`, and `codeActions` listing the top-N
59
+ suggestions.
60
+ 4. **Per-language resource resolution.** Each open document resolves
61
+ its language via `Kotoshu.detect_language` (or `languageId` from the
62
+ editor). The server calls `Kotoshu.resolve(language: lang)` — on
63
+ `ResourceNotSetupError`, send a `window/showMessage` request asking
64
+ the user to run `kotoshu setup <lang>` (or, with user consent,
65
+ trigger it server-side via `Kotoshu.setup`).
66
+ 5. **Document-format awareness.** Use `Kotoshu::Documents::*`
67
+ (`plain_text_document`, `markdown_document`, `asciidoc_document`)
68
+ based on the file extension or `languageId`. Errors carry
69
+ AST-derived positions; map those back to LSP `Position`s.
70
+
71
+ ### Phase 2 — Interactive fixes
72
+
73
+ 6. **Code actions (`textDocument/codeAction`).** For each diagnostic,
74
+ emit one `CodeAction` per suggestion: "Change to 'hello'", "Add
75
+ 'helo' to personal dictionary", "Ignore in this project".
76
+ Selecting a fix applies a `WorkspaceEdit` with a single
77
+ `textDocument/edits` entry.
78
+ 7. **Hover (`textDocument/hover`).** On a flagged word, hover shows the
79
+ full suggestion list with semantic-similarity scores (when the
80
+ ONNX path is enabled) and frequency rank.
81
+ 8. **Personal dictionary integration.** "Add to personal dictionary"
82
+ calls `Kotoshu::PersonalDictionary#add(word)`; subsequent
83
+ diagnostics drop the word for the current user. Stored under
84
+ `$XDG_CONFIG_HOME/kotoshu/personal-dict.txt` (see `Kotoshu::Paths`).
85
+ 9. **Project-level configuration.** Read `.kotoshu.yml` from the
86
+ workspace root (already supported via `Kotoshu::ProjectConfig`):
87
+ ignored words, file globs, default language, hint language list.
88
+
89
+ ### Phase 3 — Performance and UX
90
+
91
+ 10. **On-type diagnostics with debounce.** Default 500ms after the last
92
+ keystroke; configurable. Long documents must not re-check
93
+ unmodified regions — use `textDocument/didChange`'s
94
+ `contentChanges` range to scope re-analysis where the document
95
+ format supports it (plain text: re-check the line; Markdown /
96
+ AsciiDoc: re-check the enclosing block).
97
+ 11. **Background model warm.** When semantic mode is enabled, call
98
+ `OnnxModel#preload` on `initialize` so the first diagnostic push
99
+ isn't delayed by the 4 GB model load.
100
+ 12. **Configuration via LSP `workspace/didChangeConfiguration`.** Mirror
101
+ the CLI flags: `language`, `model` (`hunspell` | `fasttext` |
102
+ `hybrid`), `offline`, `personalDictionary`, `ignoredWords`.
103
+ 13. **Trace logging.** LSP servers are notoriously hard to debug.
104
+ Implement `$/setTrace` and write a per-session log under
105
+ `$XDG_CACHE_HOME/kotoshu-lsp/<pid>.log` with rotation.
106
+
107
+ ### Phase 4 — Packaging
108
+
109
+ 14. **Executable `kotoshu-lsp`.** Installs with `gem install kotoshu-lsp`
110
+ (pulls in `kotoshu` transitively). Editor config snippets in the
111
+ README point at this binary.
112
+ 15. **Server capabilities advertisement.** Declare exactly the
113
+ capabilities implemented in phases 1–3 so editors don't send
114
+ unsupported requests.
115
+ 16. **Smoke-test harness.** Spin up the server as a subprocess, drive
116
+ it with a scripted LSP client over stdio, assert diagnostics
117
+ arrive. Add to CI.
118
+
119
+ ## Acceptance criteria
120
+
121
+ - `gem install kotoshu-lsp && kotoshu-lsp` starts a server that
122
+ answers `initialize` within 200 ms.
123
+ - Opening a Markdown file with a misspelling in VS Code (Neovim,
124
+ Emacs, JetBrains) shows a yellow squiggle within 500 ms of save.
125
+ - Code actions appear in the editor's quick-fix menu; applying the
126
+ top suggestion rewrites the document.
127
+ - After `Add to personal dictionary`, the same word no longer
128
+ flags on subsequent saves (without restarting the server).
129
+ - Memory ceiling: a single open document does not exceed 250 MB
130
+ resident (without ONNX) / 1.5 GB (with ONNX hybrid).
131
+ - Smoke-test harness runs in CI and fails on regression.
132
+
133
+ ## Dependencies
134
+
135
+ - **Blocks:** `11-release-v1` (LSP is the headline v1 feature)
136
+ - **Blocked by:**
137
+ - `02-cli-unification` — clean library API surface to wrap
138
+ - `05-semantic-path` — for hover scores and reranking value
139
+ - `04-language-modules` — for the language detection path
140
+ - **Cross-repo:** new repo `kotoshu/kotoshu-lsp`; depends on
141
+ `kotoshu/dictionaries` and (optionally) `kotoshu/models-fasttext-onnx`
142
+ at runtime via the cache layer.
143
+ - **Promotes from:** T6.1 in `56-t4-t5-t6-quality-architecture-ecosystem.md`
144
+
145
+ ## Out of scope
146
+
147
+ - A VS Code extension (that's `61-editor-ecosystem.md` — the extension
148
+ just bundles the LSP server; it doesn't reimplement it).
149
+ - Grammar diagnostics beyond what `Kotoshu::Grammar` already produces.
150
+ Grammar rule packs land via `08-grammar-engine` /
151
+ `51-grammar-engine-expansion`.
152
+ - Multi-file workspace-wide analysis (defer to a future
153
+ `workspace/diagnostic` refresh pull).
154
+ - Translation of suggestion text to the user's UI locale — suggestions
155
+ are dictionary-language-scoped by design.
156
+
157
+ ## Risks
158
+
159
+ - **Ruby startup latency.** `ruby` cold start is ~100–200 ms, fine for a
160
+ long-running LSP server but worth a baseline benchmark. If startup is
161
+ a problem, evaluate `ruby --enable-frozen-string-literal` and
162
+ `bootsnap`.
163
+ - **Single-threaded model.** MRI's GIL means long `Kotoshu.check` calls
164
+ block JSON-RPC reads. Run heavy analysis on a worker thread; the
165
+ server thread only handles protocol I/O.
166
+ - **Editor coverage.** Each editor has quirks (JetBrains' LSP connector
167
+ lags Neovim's). Document per-editor caveats in the gem README; don't
168
+ let one editor's bug block the release.
169
+
170
+ ## Status
171
+
172
+ _Pending._ New repo `kotoshu/kotoshu-lsp` to be created as the first
173
+ task. Phase 1 + Phase 2 (steps 1–9) is a coherent v0.1 release; phases
174
+ 3–4 are v0.2.
@@ -0,0 +1,167 @@
1
+ # 61 — Editor & CI ecosystem
2
+
3
+ Promotes the editor-integration portions of T6.1 / T6.3 in
4
+ `TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md` into a
5
+ promotable plan.
6
+
7
+ ## Goal
8
+
9
+ Kotoshu available in every major editor and every major CI system
10
+ without users writing their own glue. Each integration is a **thin
11
+ adapter** over either:
12
+
13
+ - The LSP server (plan 60) — for interactive editing.
14
+ - The CLI (`kotoshu check`) — for batch / CI use.
15
+
16
+ The integrations live in **separate repos** under `kotoshu/`, per the
17
+ five-repo workspace model. The library gem stays focused on
18
+ spell-checking; the integrations ship their own release cadence.
19
+
20
+ ## Why
21
+
22
+ The CLI is shipped. The LSP server (plan 60) is the editor surface.
23
+ What's missing is the **adapter packaging** that makes a user say "I
24
+ installed it in 30 seconds":
25
+
26
+ - VS Code users install an extension, not a binary + JSON config.
27
+ - Vim users want a one-line plugin manager entry, not a 50-line `lsp`
28
+ config.
29
+ - CI users want a one-step action / template, not a multi-line script.
30
+ - Pre-commit framework users want a single repo entry, not a wrapper.
31
+
32
+ This is the difference between "available" and "adopted". Each adapter
33
+ is small — the heavy lifting is in the library — but the user-facing
34
+ convenience is the leverage point.
35
+
36
+ ## Tasks
37
+
38
+ ### Track A — Editor extensions (LSP-backed)
39
+
40
+ 1. **`kotoshu/vscode-kotoshu`.** VS Code / VSCodium extension.
41
+ - Bundles the LSP server as a recommended dependency (auto-install
42
+ on first activation if missing, with user consent).
43
+ - Adds a single `kotoshu.enabled` setting (default true) plus
44
+ pass-through settings mirroring the LSP server's configuration
45
+ keys (plan 60, step 12).
46
+ - Contributes command palette actions: `Kotoshu: Set Up Language`,
47
+ `Kotoshu: Add Word to Personal Dictionary`, `Kotoshu: Check
48
+ Workspace`.
49
+ - Activation on `onLanguage:*` for the supported `languageId`s.
50
+ 2. **`kotoshu/kotoshu-nvim`.** Neovim plugin (Lua).
51
+ - Built-in LSP client config — `require('kotoshu').setup({})`
52
+ wires `vim.lsp.start({ name = 'kotoshu', cmd = { 'kotoshu-lsp' } })`.
53
+ - `personal_dict` user command wraps the LSP code action.
54
+ - Defaults that work without configuration; override hook for
55
+ power users.
56
+ 3. **`kotoshu/vim-kotoshu`.** Vim 8+ / Neovim (legacy) plugin for
57
+ users not on Neovim's built-in LSP.
58
+ - Wraps [ALE](https://github.com/dense-analysis/ale) as a linter
59
+ definition; ALE handles async + location list.
60
+ - Ships a fallback synchronous `:KotoshuCheck` that shells out to
61
+ the CLI for non-LSP users.
62
+ 4. **Emacs.** Recipe for `lsp-mode` and `eglot` documented in the LSP
63
+ server's README; no separate repo needed (Emacs users expect to add
64
+ entries to `lsp-language-id-configuration` themselves).
65
+ 5. **JetBrains.** Document the LSP connector path (Settings → Language
66
+ Servers → add `kotoshu-lsp`). No separate plugin unless we want
67
+ custom intention UI; defer until LSP connector gaps are observed.
68
+ 6. **Sublime Text, Helix, Zed.** Document LSP connector config in the
69
+ README; no separate repo.
70
+
71
+ ### Track B — Git pre-commit hooks
72
+
73
+ 7. **`kotoshu/pre-commit-kotoshu`.** Hook for the
74
+ [pre-commit](https://pre-commit.com) framework.
75
+ - Entry in the central `pre-commit` hooks catalog so users add three
76
+ lines to `.pre-commit-config.yaml`.
77
+ - Runs `kotoshu check --format sarif` on staged hunks; fails the
78
+ commit when errors are found.
79
+ - Honors `.kotoshu.yml` for ignored words and file globs.
80
+ 8. **Plain git hook script.** Ship a `scripts/install-git-hook.sh` in
81
+ the gem (or LSP repo) that writes `.git/hooks/pre-commit` calling
82
+ `kotoshu check`. Documented in the main README.
83
+
84
+ ### Track C — CI integrations
85
+
86
+ 9. **`kotoshu/action-kotoshu`.** GitHub Action published on the
87
+ Marketplace.
88
+ - `uses: kotoshu/action-kotoshu@v1` with `files:` and `language:`
89
+ inputs.
90
+ - Caches `$XDG_CACHE_HOME/kotoshu/` across runs via
91
+ `actions/cache`.
92
+ - Runs `kotoshu setup` on first run; `kotoshu check --format sarif`
93
+ thereafter.
94
+ - Uploads the SARIF report via `github/codeql-action/upload-sarif`
95
+ so results surface in the repo's Security tab.
96
+ 10. **GitLab CI template.** Published in the GitLab CI templates
97
+ catalog; one-line `include:` for GitLab users.
98
+ 11. **CircleCI / Jenkins / Drone.** Documented snippets in the README;
99
+ no separate repo unless demand warrants.
100
+ 12. **Docker image.** `kotoshu/ci` image with the gem + a
101
+ pre-warmed en/de/es/fr/pt/ru dictionary cache baked in. Used by
102
+ CI integrations to avoid cold downloads.
103
+
104
+ ### Track D — Documentation
105
+
106
+ 13. **Editor install matrix** on `kotoshu.github.io`. One row per
107
+ editor, with install instructions and a screenshot / GIF.
108
+ 14. **CI integration matrix.** One row per CI system, with snippet and
109
+ expected output.
110
+ 15. **Migrating-from-LTeX / -LanguageTool guide.** Direct comparison:
111
+ features, performance, configuration parity.
112
+
113
+ ## Acceptance criteria
114
+
115
+ Per adapter:
116
+
117
+ - The README's install instructions work end-to-end on a clean machine.
118
+ - A misspelled word in a sample repo triggers a diagnostic in < 1 s
119
+ (editor adapters) or fails the CI run (CI adapters).
120
+ - The adapter honors `.kotoshu.yml` (ignored words, file globs).
121
+ - Smoke test in the adapter's own CI spins up the editor / CI scenario
122
+ and asserts the expected behavior.
123
+
124
+ Across adapters:
125
+
126
+ - All adapters reference the LSP server or the CLI; **none** duplicate
127
+ spell-checking logic.
128
+ - All adapters live in their own repo under `kotoshu/`.
129
+ - `kotoshu.github.io` editor & CI matrices cover every adapter that
130
+ ships.
131
+
132
+ ## Dependencies
133
+
134
+ - **Blocked by:**
135
+ - `60-lsp-server` — every interactive editor adapter depends on it
136
+ - `02-cli-unification` — every CI adapter depends on a stable CLI
137
+ - **Blocks:** `11-release-v1` (ecosystem is what makes v1 land softly)
138
+ - **Cross-repo:** many new repos under `kotoshu/` (see tasks).
139
+ - **Promotes from:** T6.1, T6.3 in `56-t4-t5-t6-quality-architecture-ecosystem.md`
140
+
141
+ ## Out of scope
142
+
143
+ - Reimplementing spell-checking inside any adapter. Adapters are thin.
144
+ - A desktop GUI app for dictionary curation (defer to T6.3 in plan 56
145
+ — needs its own plan if pursued).
146
+ - Mobile editor integrations (Working Copy, Textastic) — low demand;
147
+ revisit if asked.
148
+ - AI assistant integrations (Copilot, Cursor) — those read LSP
149
+ diagnostics natively, so once the LSP server is up, they get
150
+ Kotoshu for free. Document but don't ship.
151
+
152
+ ## Risks
153
+
154
+ - **Editor churn.** VS Code breaks APIs every release; pin a Node
155
+ version range and test on the latest two VS Code releases.
156
+ - **Pre-commit framework politics.** The central hook catalog has
157
+ review latency. Mirror the hook in `kotoshu/pre-commit-kotoshu` and
158
+ document the local-repo path as the primary install until catalog
159
+ inclusion lands.
160
+ - **CI cold-start cost.** Downloading dictionaries on every CI run is
161
+ slow. The `kotoshu/ci` Docker image (task 12) is the fix; without
162
+ it, CI integrations feel sluggish.
163
+
164
+ ## Status
165
+
166
+ _Pending._ Track A and Track B can ship in parallel; Track C depends on
167
+ the SARIF output being stable (plan 32 — shipped in 0.3).
@@ -0,0 +1,159 @@
1
+ # 62 — Framework integrations (Rails, static-site generators, test suites)
2
+
3
+ Promotes the Ruby-ecosystem integration idea into a promotable plan.
4
+
5
+ ## Goal
6
+
7
+ Kotoshu is a Ruby library first. Make it idiomatic inside the Ruby
8
+ ecosystem so a Rails app, a Jekyll site, or an RSpec suite can opt into
9
+ spell-checking in one line.
10
+
11
+ Three integration surfaces, all shipped **inside the library gem** (not
12
+ in separate repos — these are pure Ruby, no packaging overhead):
13
+
14
+ 1. **Validation layer** — ActiveModel validators and equivalents that
15
+ fail on misspelled content at model save time.
16
+ 2. **Build-time check** — Rake tasks for static-site generators
17
+ (Jekyll, Bridgetown, Middleman, Hugo via shell-out) so a misspelled
18
+ post fails the build.
19
+ 3. **Test helpers** — RSpec custom matchers and a Capybara integration
20
+ for system-test assertions on rendered text.
21
+
22
+ ## Why
23
+
24
+ The library API (`Kotoshu.correct?`, `.check`) is correct but generic.
25
+ Each Ruby framework has its own idioms — Rails has `ActiveModel::Validator`,
26
+ Jekyll has `:site, :post_write` hooks, RSpec has its matcher protocol.
27
+ Without framework-native entry points, users either skip spell-checking
28
+ or reinvent the wrapper.
29
+
30
+ A one-line `validates :body, kotoshu: true` in a Rails model is the
31
+ difference between "everyone on the team checks spelling" and "nobody
32
+ does." Same for `rake kotoshu:check` in a Jekyll site's deploy pipeline.
33
+
34
+ ## Tasks
35
+
36
+ ### Phase 1 — ActiveModel validator (Rails / Hanami / AnyModel)
37
+
38
+ 1. **`Kotoshu::Rails::EachValidator`.** Register as
39
+ `validates :body, kotoshu: true`. Options:
40
+ - `language:` — overrides auto-detection (default: auto).
41
+ - `personal_dictionary:` — path or `false` to disable.
42
+ - `ignored:` — array of words to skip.
43
+ - `suggestions:` — include suggestions in the validation error
44
+ message (default true).
45
+ 2. **Railtie.** Auto-register the validator on Rails boot when
46
+ `defined?(::Rails)`. Lazy — non-Rails apps pay nothing.
47
+ 3. **Generator.** `rails g kotoshu:install` writes a `.kotoshu.yml`
48
+ scaffold and an initializer that pre-warms the default language on
49
+ boot (configurable, off by default to avoid blocking boot).
50
+ 4. **Hanami / Dry-Validation contract.** Document the Dry-Validation
51
+ macro pattern; ship a `Kotoshu::Dry::Predicate` users import into
52
+ their contracts.
53
+ 5. **Sinatra / Roda** — a `before` filter recipe in the README; no
54
+ framework-specific code needed.
55
+
56
+ ### Phase 2 — Static-site generator rake tasks
57
+
58
+ 6. **`Kotoshu::RakeTask`.** Reusable rake task with config DSL:
59
+ ```ruby
60
+ Kotoshu::RakeTask.new do |t|
61
+ t.files = FileList["_posts/**/*.md", "_pages/**/*.md"]
62
+ t.language = "en" # or :auto
63
+ t.fail_on_error = true
64
+ t.format = :text # or :sarif, :json
65
+ end
66
+ ```
67
+ Generates `rake kotoshu:check`.
68
+ 7. **Jekyll integration.** Auto-detect `_config.yml` and register the
69
+ rake task. Optionally a Jekyll `:site, :pre_render` hook that
70
+ spell-checks each rendered page in development (off in production
71
+ to avoid slowing the build).
72
+ 8. **Bridgetown / Middleman.** Document the rake task invocation; no
73
+ gem-level integration unless there's a plugin API worth hooking.
74
+ 9. **Hugo / Eleventy / Zola.** Shell-out recipe in the README; the
75
+ rake task is language-agnostic, the site generator is irrelevant.
76
+
77
+ ### Phase 3 — Test helpers
78
+
79
+ 10. **RSpec custom matcher `be_well_spelled`.**
80
+ ```ruby
81
+ expect(rendered_page).to be_well_spelled
82
+ expect(user.bio).to be_well_spelled(in: "en")
83
+ ```
84
+ Failure message lists the misspelled words with suggestions.
85
+ 11. **Minitest matcher** (mirrors RSpec) so Rails default tests work.
86
+ 12. **Capybara system-test integration.** `page.assert_no_spelling_errors`
87
+ runs Kotoshu against the rendered DOM text (excluding `<script>`,
88
+ `<style>`, `<input>` values). Options for allowlist selectors
89
+ (e.g. skip `[contenteditable]` blocks).
90
+ 13. **ActionMailer / ActionView preview check.** A test helper that
91
+ spell-checks rendered email previews; catches typos in transactional
92
+ copy.
93
+
94
+ ### Phase 4 — Packaging
95
+
96
+ 14. **All of the above inside `lib/kotoshu/rails/`, `lib/kotoshu/rake/`,
97
+ `lib/kotoshu/rspec/`, `lib/kotoshu/minitest/`,
98
+ `lib/kotoshu/capybara/`.** Lazy-loaded via autoload; users who
99
+ don't `require 'kotoshu/rails'` pay no cost.
100
+ 15. **`kotoshu.gemspec` soft-deps.** Rails, Jekyll, RSpec, Minitest,
101
+ Capybara are NOT runtime deps. Code paths guard on
102
+ `defined?(::Rails)` etc. so the gem still installs on slim
103
+ environments (consistent with the soft-dep policy for onnxruntime
104
+ and suika — see CLAUDE.md).
105
+ 16. **Examples** in `examples/` for each integration
106
+ (`examples/30_rails_validator.rb`,
107
+ `examples/31_jekyll_rake_task.rb`, etc.).
108
+
109
+ ## Acceptance criteria
110
+
111
+ - A fresh Rails 7 app with `gem "kotoshu"` and
112
+ `validates :body, kotoshu: true` rejects a misspelled `body` on save.
113
+ - A fresh Jekyll site with the rake task fails `rake kotoshu:check` on
114
+ a misspelled post.
115
+ - An RSpec spec with `expect(text).to be_well_spelled` fails on
116
+ misspelled text and the failure message includes suggestions.
117
+ - All framework code is autoloaded; `Kotoshu.correct?` latency is
118
+ unchanged in apps that don't use the framework integrations.
119
+ - Each integration has its own spec under `spec/kotoshu/rails/`,
120
+ `spec/kotoshu/rake/`, etc. Specs use real framework instances
121
+ (no doubles — global rule).
122
+
123
+ ## Dependencies
124
+
125
+ - **Blocked by:**
126
+ - `02-cli-unification` — the rake task wraps the CLI; clean library
127
+ API is required
128
+ - `50-structure-aware-document-api` — for accurate positions in
129
+ Markdown content
130
+ - **Blocks:** nothing directly; ecosystem multiplier for adoption.
131
+ - **Cross-repo:** none — these live inside the library gem.
132
+ - **Promotes from:** implied by the "make it work for every Ruby
133
+ project" thread of the vision (`00-vision.md`).
134
+
135
+ ## Out of scope
136
+
137
+ - Non-Ruby frameworks (Django plugins, Express middleware, Laravel
138
+ validators). Those need HTTP API clients — see `64-http-api-and-sdks.md`.
139
+ - Spelling fixes as part of the Rails request cycle (in-request
140
+ rewriting of submitted content). That's a content-sanitization
141
+ decision the application should own.
142
+ - Auto-rewriting static-site posts. Surfaces the suggestion; the author
143
+ decides. Same policy as the CLI's `--interactive` mode.
144
+
145
+ ## Risks
146
+
147
+ - **Framework version drift.** Rails / RSpec / Minitest APIs change.
148
+ Pin tested versions in `Appraisals` (or equivalent); run integration
149
+ tests against the matrix.
150
+ - **Boot-time cost.** A Railtie that pre-warms dictionaries on boot is
151
+ a footgun. Default off; document the tradeoff loudly.
152
+ - **Test-suite speed.** Running Kotoshu on every Capybara screenshot is
153
+ slow. Make it opt-in per-test (`scenario "renders correctly", :spell
154
+ do`) rather than blanket-on.
155
+
156
+ ## Status
157
+
158
+ _Pending._ Phase 1 + 2 is a coherent 0.4-shippable unit; phase 3 is
159
+ 0.4+ depending on community demand signal.