kotoshu 1.0.2-aarch64-linux

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (458) hide show
  1. checksums.yaml +7 -0
  2. data/.kotoshu-baseline.json +1 -0
  3. data/.pre-commit-hooks.yaml +18 -0
  4. data/.rspec +3 -0
  5. data/.rubocop.yml +351 -0
  6. data/.yardopts +8 -0
  7. data/CHANGELOG.md +813 -0
  8. data/CLAUDE.md +176 -0
  9. data/CODE_OF_CONDUCT.md +132 -0
  10. data/Cargo.lock +359 -0
  11. data/Cargo.toml +15 -0
  12. data/LICENSE +31 -0
  13. data/README.adoc +1292 -0
  14. data/Rakefile +150 -0
  15. data/SECURITY.md +93 -0
  16. data/TODO.impl/00-cut-0.1.md +121 -0
  17. data/TODO.impl/00-cut-0.2.md +175 -0
  18. data/TODO.impl/00-vision.md +281 -0
  19. data/TODO.impl/01-hunspell-correctness.md +111 -0
  20. data/TODO.impl/02-cli-unification.md +74 -0
  21. data/TODO.impl/03-dynamic-download.md +107 -0
  22. data/TODO.impl/04-language-modules.md +89 -0
  23. data/TODO.impl/05-semantic-path.md +83 -0
  24. data/TODO.impl/06-cjk-support.md +105 -0
  25. data/TODO.impl/07-rtl-support.md +71 -0
  26. data/TODO.impl/08-grammar-engine.md +106 -0
  27. data/TODO.impl/09-integrity-security.md +90 -0
  28. data/TODO.impl/10-testing-ci.md +98 -0
  29. data/TODO.impl/100-full-feature-batch3.md +28 -0
  30. data/TODO.impl/101-int4-tier-experiment.md +32 -0
  31. data/TODO.impl/102-lid-chain.md +28 -0
  32. data/TODO.impl/103-oov-bucket-tables.md +27 -0
  33. data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
  34. data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
  35. data/TODO.impl/106-native-lid-detection.md +36 -0
  36. data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
  37. data/TODO.impl/108-ko-ne-full-feature.md +37 -0
  38. data/TODO.impl/109-ten-audit.md +26 -0
  39. data/TODO.impl/11-release-v1.md +95 -0
  40. data/TODO.impl/110-sr-latn-and-polish.md +33 -0
  41. data/TODO.impl/111-embedding-bakeoff.md +33 -0
  42. data/TODO.impl/112-deployability-and-gating.md +26 -0
  43. data/TODO.impl/113-language-packs.md +26 -0
  44. data/TODO.impl/114-typo-biencoder-thread.md +33 -0
  45. data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
  46. data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
  47. data/TODO.impl/117-deterministic-frequency-tiers.md +110 -0
  48. data/TODO.impl/118-server-gem-floor.md +33 -0
  49. data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
  50. data/TODO.impl/120-setup-metadata-refresh.md +35 -0
  51. data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
  52. data/TODO.impl/122-pack-compression-kpk2.md +66 -0
  53. data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
  54. data/TODO.impl/124-owner-residue.md +19 -0
  55. data/TODO.impl/125-consumer-floor-guards.md +43 -0
  56. data/TODO.impl/126-tier-error-budgets.md +33 -0
  57. data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
  58. data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
  59. data/TODO.impl/129-pt-fr-corpora.md +23 -0
  60. data/TODO.impl/130-news-truth-0912.md +23 -0
  61. data/TODO.impl/131-hybrid-productization.md +42 -0
  62. data/TODO.impl/132-techniques-page-and-prose.md +27 -0
  63. data/TODO.impl/133-rust-first-distribution.md +68 -0
  64. data/TODO.impl/20-encapsulation-cleanup.md +47 -0
  65. data/TODO.impl/21-autoload-migration.md +64 -0
  66. data/TODO.impl/22-url-pin-single-source-of-truth.md +120 -0
  67. data/TODO.impl/23-fix-url-defaults.md +57 -0
  68. data/TODO.impl/24-cli-auto-setup-prompt.md +76 -0
  69. data/TODO.impl/25-soft-onnxruntime-require.md +75 -0
  70. data/TODO.impl/26-onnx-vocab-json-generation.md +96 -0
  71. data/TODO.impl/27-end-to-end-smoke-spec.md +113 -0
  72. data/TODO.impl/28-readme-quickstart-verification.md +68 -0
  73. data/TODO.impl/29-status-command.md +84 -0
  74. data/TODO.impl/30-language-auto-detection.md +74 -0
  75. data/TODO.impl/31-download-progress.md +58 -0
  76. data/TODO.impl/32-sarif-output-verification.md +72 -0
  77. data/TODO.impl/33-audit-log-rotation.md +54 -0
  78. data/TODO.impl/34-cache-eviction.md +59 -0
  79. data/TODO.impl/35-shell-completion.md +61 -0
  80. data/TODO.impl/36-cleanup-regressions-0.3.1.md +134 -0
  81. data/TODO.impl/37-hunspell-correctness-tier2.md +203 -0
  82. data/TODO.impl/38-onnx-semantic-gating.md +218 -0
  83. data/TODO.impl/39-tier3-and-beyond.md +232 -0
  84. data/TODO.impl/40-spec-drift-cleanup.md +134 -0
  85. data/TODO.impl/50-structure-aware-document-api.md +194 -0
  86. data/TODO.impl/51-grammar-engine-expansion.md +75 -0
  87. data/TODO.impl/52-multi-language-documents.md +95 -0
  88. data/TODO.impl/53-document-plugin-architecture.md +94 -0
  89. data/TODO.impl/54-cjk-support.md +117 -0
  90. data/TODO.impl/55-rtl-support.md +109 -0
  91. data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
  92. data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
  93. data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
  94. data/TODO.impl/59-yard-api-documentation.md +75 -0
  95. data/TODO.impl/60-lsp-server.md +174 -0
  96. data/TODO.impl/61-editor-ecosystem.md +167 -0
  97. data/TODO.impl/62-framework-integrations.md +159 -0
  98. data/TODO.impl/63-web-playground-wasm.md +186 -0
  99. data/TODO.impl/64-http-api-and-sdks.md +191 -0
  100. data/TODO.impl/65-universal-kotoshu.md +251 -0
  101. data/TODO.impl/66-kotoshu-core.md +169 -0
  102. data/TODO.impl/67-kotoshu-rs-and-access-libraries.md +137 -0
  103. data/TODO.impl/68-sota-adoption.md +152 -0
  104. data/TODO.impl/69-sota-typo-eval.md +74 -0
  105. data/TODO.impl/70-sota-confidence-cascade.md +59 -0
  106. data/TODO.impl/71-sota-nested-model.md +63 -0
  107. data/TODO.impl/72-model2vec-evaluation.md +49 -0
  108. data/TODO.impl/73-rl-reranker.md +42 -0
  109. data/TODO.impl/74-distilled-reranker.md +36 -0
  110. data/TODO.impl/75-llm-evaluator.md +35 -0
  111. data/TODO.impl/76-optional-llm-backend.md +37 -0
  112. data/TODO.impl/77-model-coverage-expansion.md +96 -0
  113. data/TODO.impl/78-in-browser-playground.md +69 -0
  114. data/TODO.impl/79-news-and-releases.md +60 -0
  115. data/TODO.impl/80-ecosystem-docs.md +56 -0
  116. data/TODO.impl/81-editor-reach.md +68 -0
  117. data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
  118. data/TODO.impl/83-models-batch-2.md +56 -0
  119. data/TODO.impl/84-language-modules-wave2.md +75 -0
  120. data/TODO.impl/85-wasm-model-api.md +48 -0
  121. data/TODO.impl/86-site-truth-wave2.md +45 -0
  122. data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
  123. data/TODO.impl/88-cli-directory-mode.md +54 -0
  124. data/TODO.impl/89-framework-integrations-exec.md +59 -0
  125. data/TODO.impl/90-gem-correctness-residues.md +44 -0
  126. data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
  127. data/TODO.impl/92-tier-mirror-cors.md +88 -0
  128. data/TODO.impl/93-python-wheels.md +32 -0
  129. data/TODO.impl/94-site-integrations-docs.md +35 -0
  130. data/TODO.impl/95-editor-action-polish.md +27 -0
  131. data/TODO.impl/96-server-semantic-models.md +50 -0
  132. data/TODO.impl/97-yard-execution.md +73 -0
  133. data/TODO.impl/98-campaign-cleanup.md +78 -0
  134. data/TODO.impl/99-site-truth-performance-docs.md +31 -0
  135. data/conformance/vectors.jsonl +2630 -0
  136. data/data/en/grammar/rules.yaml +851 -0
  137. data/docs/STABILITY-1.0-AUDIT.md +251 -0
  138. data/docs/STABILITY-1.0.md +137 -0
  139. data/docs/cache-architecture.md +502 -0
  140. data/docs/verification-evidence/correct3/pr_body.md +37 -0
  141. data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
  142. data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
  143. data/docs/verification-evidence/correct3/sv.aff +856 -0
  144. data/docs/verification-evidence/correct3/sv.dic +153717 -0
  145. data/examples/01_basic_word_checking.rb +38 -0
  146. data/examples/02_text_document_checking.rb +77 -0
  147. data/examples/03_dictionary_backends.rb +137 -0
  148. data/examples/04_trie_data_structure.rb +146 -0
  149. data/examples/05_suggestion_algorithms.rb +239 -0
  150. data/examples/06_configuration_advanced.rb +287 -0
  151. data/examples/07_multi_language_dictionaries.rb +278 -0
  152. data/exe/kotoshu +6 -0
  153. data/lib/kotoshu/algorithms/capitalization.rb +276 -0
  154. data/lib/kotoshu/algorithms/edit_distance.rb +128 -0
  155. data/lib/kotoshu/algorithms/lookup/affix_form.rb +132 -0
  156. data/lib/kotoshu/algorithms/lookup/compound_checks.rb +209 -0
  157. data/lib/kotoshu/algorithms/lookup/compound_form.rb +50 -0
  158. data/lib/kotoshu/algorithms/lookup.rb +841 -0
  159. data/lib/kotoshu/algorithms/ngram_suggest.rb +367 -0
  160. data/lib/kotoshu/algorithms/permutations.rb +283 -0
  161. data/lib/kotoshu/algorithms/phonet_suggest.rb +162 -0
  162. data/lib/kotoshu/algorithms/soundex.rb +92 -0
  163. data/lib/kotoshu/algorithms/suggest.rb +794 -0
  164. data/lib/kotoshu/algorithms.rb +23 -0
  165. data/lib/kotoshu/analyzers/semantic_analyzer.rb +405 -0
  166. data/lib/kotoshu/analyzers.rb +8 -0
  167. data/lib/kotoshu/baseline/application.rb +18 -0
  168. data/lib/kotoshu/baseline/entry.rb +33 -0
  169. data/lib/kotoshu/baseline/store.rb +175 -0
  170. data/lib/kotoshu/baseline.rb +20 -0
  171. data/lib/kotoshu/cache/base_cache.rb +778 -0
  172. data/lib/kotoshu/cache/cache.rb +91 -0
  173. data/lib/kotoshu/cache/eviction_policy.rb +66 -0
  174. data/lib/kotoshu/cache/frequency_cache.rb +283 -0
  175. data/lib/kotoshu/cache/language_cache.rb +511 -0
  176. data/lib/kotoshu/cache/lookup_cache.rb +166 -0
  177. data/lib/kotoshu/cache/model_cache.rb +1147 -0
  178. data/lib/kotoshu/cache/model_registry.rb +109 -0
  179. data/lib/kotoshu/cache/staged_languages.rb +45 -0
  180. data/lib/kotoshu/cache/suggestion_cache.rb +113 -0
  181. data/lib/kotoshu/cache.rb +49 -0
  182. data/lib/kotoshu/cli/auto_setup.rb +71 -0
  183. data/lib/kotoshu/cli/baseline_command.rb +119 -0
  184. data/lib/kotoshu/cli/batch_reporter.rb +321 -0
  185. data/lib/kotoshu/cli/cache_command.rb +282 -0
  186. data/lib/kotoshu/cli/completions_command.rb +210 -0
  187. data/lib/kotoshu/cli/dict_command.rb +64 -0
  188. data/lib/kotoshu/cli/directory_check.rb +146 -0
  189. data/lib/kotoshu/cli/directory_walker.rb +132 -0
  190. data/lib/kotoshu/cli/display_formatter.rb +429 -0
  191. data/lib/kotoshu/cli/errors.rb +36 -0
  192. data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
  193. data/lib/kotoshu/cli/interactive_reviewer.rb +314 -0
  194. data/lib/kotoshu/cli/language_resolver.rb +91 -0
  195. data/lib/kotoshu/cli/navigation_manager.rb +273 -0
  196. data/lib/kotoshu/cli/personal_command.rb +113 -0
  197. data/lib/kotoshu/cli/progress_reporter.rb +114 -0
  198. data/lib/kotoshu/cli/status_report.rb +130 -0
  199. data/lib/kotoshu/cli.rb +880 -0
  200. data/lib/kotoshu/components/passthrough_spell_checker.rb +70 -0
  201. data/lib/kotoshu/components/pos_tagger.rb +98 -0
  202. data/lib/kotoshu/components/spell_checker.rb +73 -0
  203. data/lib/kotoshu/components/synthesizer.rb +60 -0
  204. data/lib/kotoshu/components/tokenizer.rb +58 -0
  205. data/lib/kotoshu/components/whitespace_tokenizer.rb +94 -0
  206. data/lib/kotoshu/components.rb +13 -0
  207. data/lib/kotoshu/configuration/builder.rb +207 -0
  208. data/lib/kotoshu/configuration/resolver.rb +124 -0
  209. data/lib/kotoshu/configuration.rb +829 -0
  210. data/lib/kotoshu/conformance_exporter.rb +256 -0
  211. data/lib/kotoshu/conformance_runner.rb +176 -0
  212. data/lib/kotoshu/core/exceptions.rb +186 -0
  213. data/lib/kotoshu/core/indexed_dictionary.rb +293 -0
  214. data/lib/kotoshu/core/models/affix_rule.rb +262 -0
  215. data/lib/kotoshu/core/models/result/document_result.rb +132 -0
  216. data/lib/kotoshu/core/models/result/word_result.rb +131 -0
  217. data/lib/kotoshu/core/models/word.rb +131 -0
  218. data/lib/kotoshu/core/trie/builder.rb +119 -0
  219. data/lib/kotoshu/core/trie/node.rb +94 -0
  220. data/lib/kotoshu/core/trie/trie.rb +249 -0
  221. data/lib/kotoshu/core.rb +28 -0
  222. data/lib/kotoshu/data/common_words/de.yml +1800 -0
  223. data/lib/kotoshu/data/common_words/en.yml +1215 -0
  224. data/lib/kotoshu/data/common_words/es.yml +750 -0
  225. data/lib/kotoshu/data/common_words/fr.yml +1015 -0
  226. data/lib/kotoshu/data/common_words/pt.yml +870 -0
  227. data/lib/kotoshu/data/common_words/ru.yml +484 -0
  228. data/lib/kotoshu/data/common_words_loader.rb +141 -0
  229. data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
  230. data/lib/kotoshu/data/frozen_kelly.rb +38 -0
  231. data/lib/kotoshu/data.rb +8 -0
  232. data/lib/kotoshu/data_structures/bloom_filter.rb +188 -0
  233. data/lib/kotoshu/data_structures.rb +8 -0
  234. data/lib/kotoshu/debug_logger.rb +146 -0
  235. data/lib/kotoshu/debug_mode.rb +136 -0
  236. data/lib/kotoshu/defaults.rb +84 -0
  237. data/lib/kotoshu/dictionaries/catalog.rb +814 -0
  238. data/lib/kotoshu/dictionaries.rb +8 -0
  239. data/lib/kotoshu/dictionary/base.rb +324 -0
  240. data/lib/kotoshu/dictionary/cspell.rb +243 -0
  241. data/lib/kotoshu/dictionary/custom.rb +225 -0
  242. data/lib/kotoshu/dictionary/hunspell.rb +520 -0
  243. data/lib/kotoshu/dictionary/plain_text.rb +302 -0
  244. data/lib/kotoshu/dictionary/repository.rb +246 -0
  245. data/lib/kotoshu/dictionary/unified.rb +236 -0
  246. data/lib/kotoshu/dictionary/unix_words.rb +218 -0
  247. data/lib/kotoshu/dictionary.rb +15 -0
  248. data/lib/kotoshu/documents/document.rb +120 -0
  249. data/lib/kotoshu/documents/plain_text_document.rb +73 -0
  250. data/lib/kotoshu/documents/source_position.rb +45 -0
  251. data/lib/kotoshu/documents/source_range.rb +76 -0
  252. data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
  253. data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
  254. data/lib/kotoshu/documents/suppressions.rb +44 -0
  255. data/lib/kotoshu/documents/text_node.rb +49 -0
  256. data/lib/kotoshu/documents.rb +178 -0
  257. data/lib/kotoshu/embeddings/embedding_pipeline.rb +238 -0
  258. data/lib/kotoshu/embeddings/lru_cache.rb +244 -0
  259. data/lib/kotoshu/embeddings/onnx_runtime_model.rb +411 -0
  260. data/lib/kotoshu/embeddings/protocol.rb +96 -0
  261. data/lib/kotoshu/embeddings/protocols.rb +23 -0
  262. data/lib/kotoshu/embeddings/registry.rb +184 -0
  263. data/lib/kotoshu/embeddings/search.rb +204 -0
  264. data/lib/kotoshu/embeddings/similarity_engine.rb +278 -0
  265. data/lib/kotoshu/embeddings/similarity_search.rb +328 -0
  266. data/lib/kotoshu/embeddings/vocabulary.rb +306 -0
  267. data/lib/kotoshu/embeddings.rb +87 -0
  268. data/lib/kotoshu/fluent_checker.rb +94 -0
  269. data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +48 -0
  270. data/lib/kotoshu/grammar/pattern_matchers/double_negative_matcher.rb +179 -0
  271. data/lib/kotoshu/grammar/pattern_matchers/phrase_matcher.rb +86 -0
  272. data/lib/kotoshu/grammar/pattern_matchers/possessive_context_matcher.rb +75 -0
  273. data/lib/kotoshu/grammar/pattern_matchers/possessive_contraction_matcher.rb +87 -0
  274. data/lib/kotoshu/grammar/pattern_matchers/sentence_start_matcher.rb +84 -0
  275. data/lib/kotoshu/grammar/pattern_matchers/vowel_sound_matcher.rb +81 -0
  276. data/lib/kotoshu/grammar/pattern_matchers/word_list_matcher.rb +66 -0
  277. data/lib/kotoshu/grammar/rule.rb +99 -0
  278. data/lib/kotoshu/grammar/rule_engine.rb +96 -0
  279. data/lib/kotoshu/grammar/rule_loader.rb +30 -0
  280. data/lib/kotoshu/grammar.rb +24 -0
  281. data/lib/kotoshu/integrity/audit_log.rb +167 -0
  282. data/lib/kotoshu/integrity/manifest.rb +116 -0
  283. data/lib/kotoshu/integrity/net_http.rb +51 -0
  284. data/lib/kotoshu/integrity/rotation_policy.rb +88 -0
  285. data/lib/kotoshu/integrity.rb +26 -0
  286. data/lib/kotoshu/jekyll.rb +101 -0
  287. data/lib/kotoshu/keyboard/layout.rb +116 -0
  288. data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
  289. data/lib/kotoshu/keyboard/layouts/azerty.rb +55 -0
  290. data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
  291. data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
  292. data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
  293. data/lib/kotoshu/keyboard/layouts/dvorak.rb +54 -0
  294. data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
  295. data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
  296. data/lib/kotoshu/keyboard/layouts/hrsl.rb +94 -0
  297. data/lib/kotoshu/keyboard/layouts/jcuken.rb +57 -0
  298. data/lib/kotoshu/keyboard/layouts/latin.rb +266 -0
  299. data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
  300. data/lib/kotoshu/keyboard/layouts/qwerty.rb +52 -0
  301. data/lib/kotoshu/keyboard/layouts/qwertz.rb +55 -0
  302. data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
  303. data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
  304. data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
  305. data/lib/kotoshu/keyboard/registry.rb +201 -0
  306. data/lib/kotoshu/keyboard.rb +96 -0
  307. data/lib/kotoshu/kotoshu_native.so +0 -0
  308. data/lib/kotoshu/language/detector.rb +242 -0
  309. data/lib/kotoshu/language/identifier.rb +376 -0
  310. data/lib/kotoshu/language/languages/base.rb +271 -0
  311. data/lib/kotoshu/language/lid_detector.rb +188 -0
  312. data/lib/kotoshu/language/normalizer/arabic.rb +113 -0
  313. data/lib/kotoshu/language/normalizer/base.rb +133 -0
  314. data/lib/kotoshu/language/normalizer/greek.rb +63 -0
  315. data/lib/kotoshu/language/normalizer/hebrew.rb +72 -0
  316. data/lib/kotoshu/language/normalizer/persian.rb +36 -0
  317. data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
  318. data/lib/kotoshu/language/registry.rb +209 -0
  319. data/lib/kotoshu/language/resources/ar/common_words.txt +6753 -0
  320. data/lib/kotoshu/language/resources/ar/confusion_sets.txt +11 -0
  321. data/lib/kotoshu/language/resources/de/common_words.txt +10003 -0
  322. data/lib/kotoshu/language/resources/de/confusion_sets.txt +246 -0
  323. data/lib/kotoshu/language/resources/en/common_words.txt +9979 -0
  324. data/lib/kotoshu/language/resources/en/confusion_sets.txt +871 -0
  325. data/lib/kotoshu/language/resources/es/common_words.txt +9992 -0
  326. data/lib/kotoshu/language/resources/es/confusion_sets.txt +17 -0
  327. data/lib/kotoshu/language/resources/fr/common_words.txt +9993 -0
  328. data/lib/kotoshu/language/resources/fr/confusion_sets.txt +76 -0
  329. data/lib/kotoshu/language/resources/pt/common_words.txt +9977 -0
  330. data/lib/kotoshu/language/resources/pt/confusion_sets.txt +18 -0
  331. data/lib/kotoshu/language/resources/ru/common_words.txt +9951 -0
  332. data/lib/kotoshu/language/resources/ru/confusion_sets.txt +5 -0
  333. data/lib/kotoshu/language/script.rb +128 -0
  334. data/lib/kotoshu/language/segmenter.rb +122 -0
  335. data/lib/kotoshu/language/suika.rb +38 -0
  336. data/lib/kotoshu/language/tokenizer/base.rb +187 -0
  337. data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
  338. data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
  339. data/lib/kotoshu/language/tokenizer/french_tokenizer.rb +172 -0
  340. data/lib/kotoshu/language/tokenizer/german_tokenizer.rb +41 -0
  341. data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
  342. data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
  343. data/lib/kotoshu/language/tokenizer/japanese_tokenizer.rb +55 -0
  344. data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +153 -0
  345. data/lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb +160 -0
  346. data/lib/kotoshu/language/tokenizer/russian_tokenizer.rb +93 -0
  347. data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
  348. data/lib/kotoshu/language/tokenizer/spanish_tokenizer.rb +122 -0
  349. data/lib/kotoshu/language.rb +127 -0
  350. data/lib/kotoshu/languages/ar/language.rb +98 -0
  351. data/lib/kotoshu/languages/bg/language.rb +37 -0
  352. data/lib/kotoshu/languages/ca/language.rb +22 -0
  353. data/lib/kotoshu/languages/cs/language.rb +22 -0
  354. data/lib/kotoshu/languages/da/language.rb +22 -0
  355. data/lib/kotoshu/languages/de/language.rb +554 -0
  356. data/lib/kotoshu/languages/el/language.rb +50 -0
  357. data/lib/kotoshu/languages/en/language.rb +470 -0
  358. data/lib/kotoshu/languages/es/language.rb +476 -0
  359. data/lib/kotoshu/languages/et/language.rb +24 -0
  360. data/lib/kotoshu/languages/fa/language.rb +91 -0
  361. data/lib/kotoshu/languages/fr/language.rb +505 -0
  362. data/lib/kotoshu/languages/he/language.rb +98 -0
  363. data/lib/kotoshu/languages/hr/language.rb +24 -0
  364. data/lib/kotoshu/languages/hu/language.rb +22 -0
  365. data/lib/kotoshu/languages/id/language.rb +25 -0
  366. data/lib/kotoshu/languages/it/language.rb +22 -0
  367. data/lib/kotoshu/languages/ja/language.rb +476 -0
  368. data/lib/kotoshu/languages/ko/language.rb +56 -0
  369. data/lib/kotoshu/languages/latin_base.rb +53 -0
  370. data/lib/kotoshu/languages/lt/language.rb +24 -0
  371. data/lib/kotoshu/languages/lv/language.rb +24 -0
  372. data/lib/kotoshu/languages/nb/language.rb +25 -0
  373. data/lib/kotoshu/languages/ne/language.rb +60 -0
  374. data/lib/kotoshu/languages/nl/language.rb +22 -0
  375. data/lib/kotoshu/languages/nn/language.rb +26 -0
  376. data/lib/kotoshu/languages/pl/language.rb +22 -0
  377. data/lib/kotoshu/languages/pt/language.rb +439 -0
  378. data/lib/kotoshu/languages/ro/language.rb +22 -0
  379. data/lib/kotoshu/languages/ru/language.rb +420 -0
  380. data/lib/kotoshu/languages/sk/language.rb +25 -0
  381. data/lib/kotoshu/languages/sl/language.rb +25 -0
  382. data/lib/kotoshu/languages/sr/language.rb +39 -0
  383. data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
  384. data/lib/kotoshu/languages/sv/language.rb +22 -0
  385. data/lib/kotoshu/languages/tr/language.rb +36 -0
  386. data/lib/kotoshu/languages/uk/language.rb +38 -0
  387. data/lib/kotoshu/languages/vi/language.rb +22 -0
  388. data/lib/kotoshu/languages.rb +64 -0
  389. data/lib/kotoshu/metrics_collector.rb +238 -0
  390. data/lib/kotoshu/metrics_module.rb +121 -0
  391. data/lib/kotoshu/models/context.rb +119 -0
  392. data/lib/kotoshu/models/embedding_model.rb +181 -0
  393. data/lib/kotoshu/models/fasttext_model.rb +212 -0
  394. data/lib/kotoshu/models/nearest_neighbor.rb +96 -0
  395. data/lib/kotoshu/models/onnx_model.rb +330 -0
  396. data/lib/kotoshu/models/semantic_error.rb +204 -0
  397. data/lib/kotoshu/models/suggestion.rb +106 -0
  398. data/lib/kotoshu/models/word_embedding.rb +97 -0
  399. data/lib/kotoshu/models.rb +27 -0
  400. data/lib/kotoshu/multi_language_checker.rb +127 -0
  401. data/lib/kotoshu/native.rb +83 -0
  402. data/lib/kotoshu/native_backend.rb +131 -0
  403. data/lib/kotoshu/paths.rb +79 -0
  404. data/lib/kotoshu/personal_dictionary.rb +105 -0
  405. data/lib/kotoshu/plugins/plugin.rb +61 -0
  406. data/lib/kotoshu/plugins/registry.rb +118 -0
  407. data/lib/kotoshu/plugins.rb +9 -0
  408. data/lib/kotoshu/project_config.rb +76 -0
  409. data/lib/kotoshu/readers/aff_data.rb +530 -0
  410. data/lib/kotoshu/readers/aff_reader.rb +393 -0
  411. data/lib/kotoshu/readers/condition_checker.rb +125 -0
  412. data/lib/kotoshu/readers/dic_reader.rb +208 -0
  413. data/lib/kotoshu/readers/file_reader.rb +377 -0
  414. data/lib/kotoshu/readers/lookup_builder.rb +584 -0
  415. data/lib/kotoshu/readers/readers.rb +17 -0
  416. data/lib/kotoshu/readers.rb +25 -0
  417. data/lib/kotoshu/resource_bundle.rb +55 -0
  418. data/lib/kotoshu/resource_manager.rb +600 -0
  419. data/lib/kotoshu/results/result.rb +165 -0
  420. data/lib/kotoshu/results.rb +8 -0
  421. data/lib/kotoshu/rspec.rb +163 -0
  422. data/lib/kotoshu/scripts/fasttext_to_onnx.py +275 -0
  423. data/lib/kotoshu/source_registry.rb +104 -0
  424. data/lib/kotoshu/spellchecker/parallel_checker.rb +90 -0
  425. data/lib/kotoshu/spellchecker.rb +406 -0
  426. data/lib/kotoshu/string_metrics.rb +153 -0
  427. data/lib/kotoshu/suggestions/context.rb +43 -0
  428. data/lib/kotoshu/suggestions/frequency_provider.rb +80 -0
  429. data/lib/kotoshu/suggestions/generator.rb +169 -0
  430. data/lib/kotoshu/suggestions/pipeline.rb +135 -0
  431. data/lib/kotoshu/suggestions/semantic_cascade.rb +73 -0
  432. data/lib/kotoshu/suggestions/strategies/base_strategy.rb +315 -0
  433. data/lib/kotoshu/suggestions/strategies/composite_strategy.rb +197 -0
  434. data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +631 -0
  435. data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +207 -0
  436. data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +165 -0
  437. data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +284 -0
  438. data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +317 -0
  439. data/lib/kotoshu/suggestions/strategies/symspell_strategy.rb +270 -0
  440. data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
  441. data/lib/kotoshu/suggestions/suggestion.rb +154 -0
  442. data/lib/kotoshu/suggestions/suggestion_set.rb +270 -0
  443. data/lib/kotoshu/suggestions/sweep_index.rb +137 -0
  444. data/lib/kotoshu/suggestions.rb +29 -0
  445. data/lib/kotoshu/tasks/check_task.rb +104 -0
  446. data/lib/kotoshu/tasks.rb +11 -0
  447. data/lib/kotoshu/validators/spelling_validator.rb +101 -0
  448. data/lib/kotoshu/validators.rb +9 -0
  449. data/lib/kotoshu/version.rb +5 -0
  450. data/lib/kotoshu.rb +538 -0
  451. data/rakelib/compile.rake +10 -0
  452. data/script/validate_all_dictionaries.rb +444 -0
  453. data/scripts/build_native_gem.rb +45 -0
  454. data/scripts/extract_eval_grids.rb +37 -0
  455. data/scripts/generate_frozen_tiers.rb +44 -0
  456. data/sig/kotoshu.rbs +435 -0
  457. data/test_oop.rb +79 -0
  458. metadata +546 -0
@@ -0,0 +1,56 @@
1
+ # 83 — Models batch 2: every dictionary language FastText can serve
2
+
3
+ > Wave-2 continuation of [77](77-model-coverage-expansion.md).
4
+ > v1.1.0 shipped 22 languages; the user asks again: are the models
5
+ > comprehensive? Not yet — this plan drives toward full intersection
6
+ > coverage.
7
+
8
+ ## Context
9
+
10
+ - Dictionaries repo: 98 languages. Models registry v1.1.0: 22.
11
+ - FastText CC vectors exist for 157+ languages.
12
+ - The intersection (dictionaries ∩ fastText, minus the 22 shipped) is
13
+ the candidate pool — enumerate it exactly; expect ~50-60.
14
+ - Notable misses with gem language modules already wired: **ar, fa,
15
+ he** (RTL modules exist; no models). These are the highest-priority
16
+ additions — semantic reranking for RTL users currently does nothing.
17
+
18
+ ## Phases
19
+
20
+ 1. **Enumerate** the intersection; rank by (existing gem module,
21
+ speaker base, dictionary size). ar, fa, he lead.
22
+ 2. **Ship in priority order under a compute budget** — same pipeline,
23
+ same gates (fluency rank_corr ≥ 0.97 / top1 ≥ 0.95, mini ≥ 0.90 /
24
+ 0.85), same keyboard-aware eval. Extend `eval/noise.py` per
25
+ language as needed (Hebrew/Arabic grids; reuse wave-1 patterns).
26
+ Target ≥ 15 shipped in this batch (ar fa he + a dozen more);
27
+ record the remaining tail in the registry README as the backlog.
28
+ 3. **fi / id dictionaries** — not in the dictionaries repo. Source
29
+ from upstream (LibreOffice/hunspell releases) INTO the
30
+ dictionaries repo with license metadata per its manifest schema;
31
+ then run them through the same pipeline. LICENSE VERIFICATION IS
32
+ A HARD GATE — if provenance is unclear, skip and report.
33
+ 4. **`no` → nb/nn mapping** — proposal only: registry aliases or gem
34
+ resolution mapping `no` to `nb` (Bokmål). Owner decision; do not
35
+ implement unilaterally.
36
+ 5. **Release v1.2.0** (plan 05 policy: coverage = minor), 3 tiers per
37
+ new language, media-host mirrors, byte-identical registry proof.
38
+
39
+ ## Owner gates
40
+
41
+ - `no` mapping decision (proposal delivered, not executed).
42
+ - Version follows the plan-05 pre-declared policy as v1.1.0 did.
43
+
44
+ ## Status
45
+
46
+ **Implemented (2026-09-05, models PR #14 + tag v1.2.0).** 22 -> 54
47
+ languages (32 shipped: fa he ar id hy hr bg sr sk mn lt ka ne nn sl
48
+ mk lv et tk la gl eu cy eo is lb fy br ga gd oc ia), 162 registry
49
+ resources rev 3, 326 assets, byte-identity proven. ar/id
50
+ dictionaries sourced with verified licenses (dictionaries PRs #1-#3,
51
+ that repo's default branch is v1). Gates never weakened. DROPS:
52
+ nds (ISO 639-2-only vs the two-letter schema contract - owner spec
53
+ decision), fi (no license-clear upstream - hard gate), fo rw ie fur
54
+ tlh (no fastText CC vectors). BACKLOG EMPTY: every dictionaries
55
+ language FastText can serve ships. Owner proposal pending: no->nb/nn
56
+ (cc.no as Bokmal-labeled nb + gem resolves no->nb).
@@ -0,0 +1,75 @@
1
+ # 84 — Language modules + keyboards for the 22 model languages
2
+
3
+ ## Context
4
+
5
+ The models registry now serves 22 languages, but gem language modules
6
+ exist for only 10 (ar de en es fa fr he ja pt ru) and keyboard layouts
7
+ for 5 (azerty dvorak jcuken qwerty qwertz). A Turkish user gets a
8
+ model but no Turkish-Q keyboard proximity; Greek, Ukrainian, Polish,
9
+ Czech, Hungarian, Nordic users likewise. The site honestly labels this
10
+ gap ("staged dictionary + model until the module is wired") — close it.
11
+
12
+ ## Track A — keyboard layouts
13
+
14
+ New layouts in `lib/kotoshu/keyboard/layouts/`: tr (Turkish-Q), uk
15
+ (ЙЦУКЕН Ukrainian — extends jcuken with ҐЄІЇ), el (Greek), pl, cs,
16
+ hu, da/nb/sv (Nordic shared layout with per-language diacritics), ro,
17
+ ca, it, nl (Latin layouts parameterized over the qwerty/qwertz base —
18
+ prefer ONE parameterized Latin layout family + per-language key maps
19
+ over 11 near-duplicate files; keep the existing five as-is).
20
+
21
+ **Single source of truth**: the models repo's `eval/noise.py` grids
22
+ (wave-1 added Turkish-Q / Ukrainian-JCUKEN / Greek-phonetic) encode
23
+ the same adjacency data — derive the gem layouts to match them and
24
+ record the sync note in both repos' plans (drift check: a spec
25
+ asserting the gem layout keys ⊇ the eval grid keys for tr/uk/el).
26
+
27
+ ## Track B — language modules
28
+
29
+ `lib/kotoshu/languages/` modules for the 13 wave-1 Latin newcomers
30
+ (ca cs da el hu it nl pl ro sv tr uk vi): most are thin — Latin
31
+ tokenizer/normalizer composition + layout + registry entry. Follow the
32
+ existing module pattern (see `de`, `pt`); el gets Greek-aware
33
+ normalization; tr gets case-folding care (dotless i); vi may reuse
34
+ Latin defaults. Each module ships with specs against real dictionary
35
+ fixtures (no doubles).
36
+
37
+ ## Track C — site truth (executed by the site agent, not here)
38
+
39
+ Full-feature list grows; the "staged + model" note shrinks. Recorded
40
+ here as the dependency so the site agent can sequence on this PR.
41
+
42
+ ## Constraints
43
+
44
+ - No behavior change for the existing 10 modules.
45
+ - `register_suggestion_algorithm`/layout registry patterns unchanged
46
+ (OCP: layouts register, nothing edits a switch).
47
+ - Suite stays green; add specs per layout + module.
48
+
49
+ ## Sync note (the drift check)
50
+
51
+ The gem layouts for tr/uk/el are derived from
52
+ models-fasttext-onnx `eval/noise.py` (`_TR_Q`, `_UK_JCUKEN`,
53
+ `_EL_PHONETIC`, wave 1 / registry v1.1.0) and must stay byte-equal.
54
+ `spec/kotoshu/keyboard/layouts/eval_grid_drift_spec.rb` enforces it
55
+ both ways: against the live models-repo checkout when one is present
56
+ (workspace sibling or `KOTOSHU_MODELS_REPO`; skipped when the
57
+ checkout predates the wave-1 grids) and against the committed
58
+ snapshot `spec/fixtures/models_eval_grids.json` otherwise (CI).
59
+ After changing the eval grids, refresh the snapshot with
60
+ `ruby scripts/extract_eval_grids.rb PATH/eval/noise.py`. The models
61
+ repo carries the mirror-direction note in its plan 77 status.
62
+
63
+ ## Status
64
+
65
+ **Implemented 2026-09-05 (gem PR feat/language-modules-wave2).**
66
+ Track A: Turkish-Q, Ukrainian-JCUKEN, Greek-Phonetic national
67
+ layouts plus one parameterized Latin family (11 members over the
68
+ qwerty/qwertz base grids; Nordic members carry real å/æ/ø and å/ä/ö
69
+ keys); registry 5 -> 19 layouts; the pre-existing five layout files
70
+ are untouched (Ukrainian registers before JCUKEN so `uk` resolves to
71
+ its dedicated grid). Track B: 13 thin modules on a shared
72
+ `Languages::LatinBase` (tokenizer/normalizer composition), with
73
+ Turkish dotless-i folding, Greek final-sigma/accent normalizer,
74
+ Greek/Cyrillic script tokenizers; the existing ten modules are
75
+ unchanged. Track C (site) stays with the site agent.
@@ -0,0 +1,48 @@
1
+ # 85 — Browser semantic reranking: wasm model API
2
+
3
+ ## Context
4
+
5
+ The playground is honestly dictionary-only for two reasons found in
6
+ wave 1: GitHub release assets send no CORS headers, and the wasm
7
+ surface exposes no model API. The first is a distribution problem;
8
+ the second is the real gap — this plan closes both.
9
+
10
+ ## Track A — CORS spike (decides the distribution path)
11
+
12
+ Check `Access-Control-Allow-Origin` on: the media-host mirror URLs in
13
+ the registry, the models repo's raw.githubusercontent paths, and the
14
+ registry.json itself (which the browser must read to discover URLs).
15
+ Outcomes:
16
+ - Media host serves ACAO → use registry mirror URLs directly.
17
+ - Nothing serves ACAO → tier artifacts move into the jsDelivr-servable
18
+ git tree (tiers are LFS-free and ~3-15 MB; jsDelivr gh caps at
19
+ 20 MB) via a registry "browser_url" field — additive schema, old
20
+ consumers unaffected.
21
+ Record the verdict + evidence in this file.
22
+
23
+ ## Track B — wasm model API
24
+
25
+ Expose on `@kotoshu/wasm` (wasm-bindgen surface, kotoshu-wasm crate):
26
+ - `createModelDictionary`-adjacent: `loadModel(bytes, vocabBytes)`
27
+ parsing the int8 tier (quantized matrix + vocab) into memory.
28
+ - `rerank(word, context) -> f32 score` (dot-product against the
29
+ quantized embeddings, dequantized on the fly) — enough to drive the
30
+ same reranking the gem does. Pure Rust, NO ort in the wasm build
31
+ (load-dynamic cannot work in a browser); the int8 tiers exist
32
+ precisely so this is cheap.
33
+ - Memory discipline: document peak memory (vocab 10k-50k × int8 ×
34
+ dim ≈ 3-15 MB) and free on `drop`.
35
+ - Node smoke in CI exercises load + rerank against a real mini-tier
36
+ fixture (checked-in, small) — the same pattern as the dictionary
37
+ smoke.
38
+
39
+ ## Track C — release readiness (owner-gated)
40
+
41
+ npm version 0.2.0 candidate: PR + CI only. Publishing @kotoshu/wasm
42
+ is the owner's version call (tag `@kotoshu/wasm-v*` publishes keyless
43
+ once decided). Playground wiring happens after the API ships, as a
44
+ site follow-up.
45
+
46
+ ## Status
47
+
48
+ **Pending.**
@@ -0,0 +1,45 @@
1
+ # 86 — Site truth wave 2: ignores, Windows, playground breadth
2
+
3
+ ## Context
4
+
5
+ Wave 1 shipped features the site does not document yet, and one docs
6
+ bug: the CLI page claims the personal dictionary lives at
7
+ `~/.config/kotoshu/personal_dictionary` — the gem actually uses
8
+ `personal.dic` (Paths.personal_dictionary_path, KOTOSHU_PERSONAL_DIC
9
+ override). Wrong paths in docs are worse than no docs.
10
+
11
+ ## Phases
12
+
13
+ 1. **CLI docs fixes + ignoring & baselines guide**: correct the
14
+ personal-dictionary path everywhere it appears; new docs page
15
+ covering `kotoshu:disable-line` / `-next-line [WORDS]` /
16
+ `-file` blocks per format, `kotoshu baseline init` + `check
17
+ --baseline` semantics (count budgets, stale entries, SARIF
18
+ suppression marking), and the pre-commit hook — matching gem PR
19
+ #118 exactly (read its README sections + specs as the source of
20
+ truth, not this plan).
21
+ 2. **Windows page**: the CI matrix now runs 3.3/3.4/4.0 on
22
+ windows-latest green — document first-class Windows support:
23
+ `gem install kotoshu` on Windows, onnxruntime soft-dep behavior,
24
+ path handling, and the native extension's Windows status (it is
25
+ NOT built in CI — pending; say so honestly). Link from /install.
26
+ 3. **Playground breadth**: offer every language whose dictionary the
27
+ CDN can serve well — at minimum the 22 model languages plus the
28
+ staged-dictionary set with sane sizes (cap the list at languages
29
+ whose .aff+.dic < ~5 MB on the CDN; largest dictionaries get a
30
+ size note). Dictionary-only behavior unchanged until plan 85
31
+ ships.
32
+ 4. **Matrix flip** (sequences on plan 84): when the gem modules land,
33
+ FULL_LANGUAGES grows per the merged module list; the "staged +
34
+ model" note shrinks. If 84 has not merged when you reach this
35
+ phase, skip it and note it — do not block the other phases.
36
+
37
+ ## Constraints
38
+
39
+ House rules: verify against the gem source (paths, flags), run the
40
+ build, CDP-check the playground still works after the language list
41
+ change. No new dependencies.
42
+
43
+ ## Status
44
+
45
+ **Pending.**
@@ -0,0 +1,36 @@
1
+ # 87 — kotoshu-lsp: read the personal dictionary
2
+
3
+ ## Context
4
+
5
+ The VS Code extension's "add to personal dictionary" writes
6
+ `~/.config/kotoshu/personal.dic` (same path + format as the gem CLI —
7
+ `Kotoshu::Paths.personal_dictionary_path`), but the LSP server never
8
+ READS it. The user adds a word, the diagnostic stays, the feature
9
+ feels broken. Found by the vscode agent (wave 1) and confirmed by
10
+ grep: server.rb only *writes* suggestions for the action.
11
+
12
+ ## Design
13
+
14
+ - On server start AND on dictionary-file change, load the personal
15
+ dictionary (gem API: `Kotoshu::PersonalDictionary` — reuse, do not
16
+ re-parse) into the spellcheck path so its words are correct.
17
+ - File watching: simplest honest approach that works everywhere —
18
+ mtime check on didOpen/didChange re-checks (cheap stat), plus a
19
+ documented limitation note in README (no OS file watcher needed
20
+ for v0.1.x).
21
+ - After an add-to-dictionary code action (server executes the write
22
+ itself, or the client notifies via workspace/didChangeConfiguration
23
+ / custom notification), re-publish diagnostics for open documents.
24
+ Prefer: the SERVER performs the write (single writer, no
25
+ client/server filename drift) — the extension's client-side write
26
+ moves to the server command; keep the extension working with both
27
+ (it may still write for its own UI; server write wins when present).
28
+ - Specs: real temp personal.dic files (no doubles); add → diagnostic
29
+ clears; KOTOSHU_PERSONAL_DIC env honored (gem Paths already does).
30
+
31
+ Leaves the repo 0.1.1-ready (owner dispatches the release; RubyGems
32
+ 0.1.0 is empty — see the wave-1 fixes).
33
+
34
+ ## Status
35
+
36
+ **Pending.**
@@ -0,0 +1,54 @@
1
+ # 88 — CLI directory mode: check a whole tree
2
+
3
+ ## Context
4
+
5
+ `kotoshu check` handles one file or stdin. Every serious checker
6
+ (cspell, codespell, hunspell wrappers) checks a directory tree — that
7
+ is the actual CI workflow (`action-kotoshu` users fake it with
8
+ find/xargs today). Baselines (gem PR #118) already key on file paths,
9
+ so directory mode + baselines is the complete CI story.
10
+
11
+ ## Design
12
+
13
+ - `kotoshu check DIR [DIR...]` — walk the tree; check files whose
14
+ extension is a known text format (markdown, asciidoc, txt, rst,
15
+ adoc, mdx) unless `--include`/`--exclude` globs say otherwise.
16
+ - Respect `.gitignore` (and `.ignore`) semantics: implement the
17
+ standard glob subset (directories, `*`, `**`, `?`, negation `!`,
18
+ trailing `/`); document the subset honestly. NO shelling out to
19
+ git — the vision forbids external binaries.
20
+ - Hidden files/dirs and `node_modules`/`.git`/`vendor`/`target`
21
+ skipped by default.
22
+ - Output: per-file sections in text mode; one combined SARIF/JSON
23
+ document with per-file runs. `--baseline` applies per file+word as
24
+ shipped. Exit codes unchanged (1 = errors found).
25
+ - Interactive mode stays file-only (documented no-op with a message
26
+ for directories).
27
+
28
+ ## Specs
29
+
30
+ Real directory trees in specs (tmpdirs with fixtures + .gitignore
31
+ cases: negation, nested ignores, globstar). Behavior specs, no
32
+ doubles.
33
+
34
+ ## Status
35
+
36
+ **Implemented 2026-09-05 (gem PR feat/cli-directory-mode).**
37
+ `kotoshu check FILE [DIR ...]`: DirectoryWalker selects known text
38
+ extensions by default (md markdown asciidoc adoc txt rst mdx) with
39
+ --include/--exclude globs replacing/overriding the default
40
+ (slashless globs match basenames, slashed globs match the path from
41
+ the root); hidden entries and .git/node_modules/vendor/target are
42
+ always skipped. IgnoreMatcher implements the .gitignore/.ignore
43
+ glob subset in Ruby (no shelling out): * ? ** anchoring trailing-/
44
+ directory patterns ! negation with last-match-wins, nested
45
+ scoping, and git-style no-re-include under ignored directories.
46
+ Output: per-file text sections + summary; one combined JSON
47
+ document (fileCount/wordCount/errorCount/files[] with the
48
+ single-file payload keys); one SARIF run per file. --baseline
49
+ applies per file unchanged; exit codes unchanged; interactive mode
50
+ prints a file-only notice for directory targets. Single-file and
51
+ stdin paths are byte-identical (the single-file code path is
52
+ preserved verbatim). Behavior specs run the real CLI against real
53
+ tmpdir trees with the real en dictionary from committed fixtures -
54
+ no network, no doubles.
@@ -0,0 +1,59 @@
1
+ # 89 — Framework integrations (executes plan 62)
2
+
3
+ > [62-framework-integrations.md](62-framework-integrations.md) is the
4
+ > original plan — still Pending. This file is the wave-2 execution
5
+ > order; 62's design stands.
6
+
7
+ ## What ships (inside the gem, soft-dependency style)
8
+
9
+ 1. **Rails / ActiveModel validator** — `validates :body,
10
+ spelling: true` (an EachValidator): checks the attribute with
11
+ Kotoshu, adds one error per misspelling with the top suggestion.
12
+ Config per-attribute: language, personal words. NO Rails
13
+ dependency in the gemspec — the validator autoloads only when
14
+ ActiveModel is defined (the onnxruntime/suika soft-dep pattern).
15
+ 2. **RSpec matcher** — `expect_words("helo").to all_be_spelled_correctly`
16
+ style plus a block matcher over a document; ships with proper
17
+ failure messages.
18
+ 3. **Rake task** — `kotoshu/check.rake`: checks changed markdown/
19
+ asciidoc in the repo (uses the same file-selection as plan 88),
20
+ wired via `gem 'kotoshu'` + `require 'kotoshu/tasks'`.
21
+ 4. **Jekyll** — a generator plugin class checking posts/drafts,
22
+ failing the build on new errors (respects baseline if present).
23
+
24
+ Each integration has specs that load the real framework pieces where
25
+ feasible (ActiveModel is loadable standalone in specs without Rails).
26
+
27
+ ## What does NOT ship
28
+
29
+ Engine packaging, `kotoshu-rails` separate gem, Sidekiq batch
30
+ checkers — plan 62 phase 3, demand-gated. (Also not shipped from 62
31
+ phases 1-2: the railtie, rails generator, dry-validation macro,
32
+ Capybara/Minitest helpers, ActionMailer preview checks.)
33
+
34
+ ## Status
35
+
36
+ **Implemented 2026-09-05 (gem PR feat/framework-integrations).**
37
+ All four items shipped inside the gem on the soft-dependency
38
+ pattern (nothing new in kotoshu.gemspec; activemodel and jekyll are
39
+ dev-group test deps only):
40
+
41
+ 1. `Kotoshu::Validators::SpellingValidator` — ActiveModel
42
+ EachValidator; `validates :body, spelling: true` works after the
43
+ documented one-line top-level alias (ActiveModel resolves helper
44
+ validators from Object), `validates_with` works directly.
45
+ language/personal_words options; one error per misspelling with
46
+ the top suggestion. Specs load real ActiveModel standalone.
47
+ 2. `Kotoshu::Rspec::Matchers` — expect_words + all_be_spelled_correctly,
48
+ be_spelled_correctly (word or document), expect_document, with
49
+ suggestion-bearing failure messages.
50
+ 3. `Kotoshu::Tasks::CheckTask` — `require "kotoshu/tasks"` gives
51
+ `rake kotoshu:check` over the repo text files via the plan-88
52
+ selection (known extensions, .gitignore/.ignore, default skips);
53
+ DSL: files/language/fail_on_error/baseline/root.
54
+ 4. `Kotoshu::Jekyll::Generator` — safe generator checking posts and
55
+ drafts, failing the build on new errors, respecting
56
+ .kotoshu-baseline.json in the site source.
57
+
58
+ The rake task is require-only (defining a task on autoload would be
59
+ a surprise side effect); the others autoload from Kotoshu.
@@ -0,0 +1,44 @@
1
+ # 90 — Gem correctness residues found by the site flip
2
+
3
+ > Found 2026-09-05 while engine-verifying the 13 new specimen pages.
4
+ > The available-languages gate (the worst of the three) was fixed the
5
+ > same night in passing; the two below remain.
6
+
7
+ ## Fixed same-night (context)
8
+
9
+ `LanguageCache::AVAILABLE_LANGUAGES` hardcoded the original six, so
10
+ `kotoshu setup it` (and every other new module language) 404'd at the
11
+ pin — blocking the entire wave-2 module work at the last step. Now the
12
+ 19 module languages, with a comment explaining the module-boundary
13
+ rule (staged dictionaries join when their module lands).
14
+
15
+ ## Remaining
16
+
17
+ ### A — AffReader crashes on the Swedish aff
18
+
19
+ `Readers::AffReader` raises `RegexpError: unmatched close parenthesis`
20
+ on the staged Swedish `.aff` (dictionaries pin 1829a3e). Verified by
21
+ the site agent: the file loads fine under hunspell 1.7.2, so this is
22
+ our parser, not the data. Likely an unescaped `)` inside a condition
23
+ or MAP/BREAK pattern reaching `Regexp.new` without escaping. Fix the
24
+ escaping site (not the fixture); spec with the real sv aff slice.
25
+
26
+ ### B — Spellchecker word detection is ASCII-only
27
+
28
+ `Spellchecker#word_char?` (and whatever tokenizer feeds it) treats
29
+ only ASCII as word characters, so el/uk (and any non-Latin script)
30
+ words are not extracted on the CLI path — the site agent verified the
31
+ new Greek/Ukrainian specimens via dictionary lookup because the CLI
32
+ path could not see the words at all. Extend word-char detection to
33
+ Unicode letters (per-language script knowledge already exists in the
34
+ tokenizers); guard behavior with specs on el/uk/ja text and confirm no
35
+ regression on the ASCII languages (the whole conformance corpus is
36
+ ASCII — it must stay byte-identical).
37
+
38
+ ## Owner gates
39
+
40
+ None.
41
+
42
+ ## Status
43
+
44
+ **Pending.**
@@ -0,0 +1,47 @@
1
+ # 91 — Executes plan 90: Unicode word detection + the Swedish aff parser
2
+
3
+ ## Track A — Unicode word detection (the bigger one)
4
+
5
+ `Spellchecker#word_char?` (spellchecker.rb:313) accepts only ASCII
6
+ letters, so Greek/Ukrainian/any non-Latin script words are never
7
+ extracted on the CLI path — el/uk users cannot use the checker at all
8
+ (found by the site-flip agent; specimens could only be verified via
9
+ raw dictionary lookup).
10
+
11
+ - Extend word-char detection to Unicode letters. The per-language
12
+ tokenizers (wave-2 added Greek/Cyrillic ones) already know their
13
+ scripts — the fix should respect the language's tokenizer/script
14
+ rather than widening a global ASCII set (wrong-script words must
15
+ not suddenly become checkable).
16
+ - **Non-negotiable**: the frozen conformance corpus is ASCII — every
17
+ vector must stay byte-identical. Guard with the full conformance
18
+ run plus new el/uk/ja extraction specs.
19
+ - End-to-end verify: `echo "<greek typo>" | bundle exec exe/kotoshu
20
+ check -l el` flags it; same for uk; en/de regression-clean.
21
+
22
+ ## Track B — AffReader RegexpError on the Swedish aff
23
+
24
+ `Readers::AffReader` raises `RegexpError: unmatched close parenthesis`
25
+ on the staged Swedish .aff (pin 1829a3e); hunspell 1.7.2 loads it
26
+ fine. Find the unescaped-`)` site (likely condition/MAP/BREAK pattern
27
+ compilation) and fix the escaping — never the fixture. Spec with a
28
+ minimal slice of the real sv aff.
29
+
30
+ ## Track C — setup end-to-end spot-check
31
+
32
+ After PR #127's gate widening, run `kotoshu setup` + check for it,
33
+ el, tr, sv (sv will hit Track B until fixed — fix first). Record
34
+ which languages verified end-to-end through the CLI.
35
+
36
+ ## Status
37
+
38
+ **Implemented (2026-09-05, gem PR #130 + residue PR #131).** Track B:
39
+ COMPOUNDRULE )k - the sv aff uses ) as a flag char; flag atoms now
40
+ Regexp.escape'd (full 152k-word sv dictionary loads). Track A:
41
+ Tokenizer::Base#spellcheck_word_regex with Greek/Cyrillic/Latin
42
+ overrides, digits excluded, ASCII default byte-identical (2630 frozen
43
+ vectors 0 failures); plus the bundle-language-priority fix that made
44
+ `check -l el` extract zero words. Track C: remote setup for staged
45
+ languages was the LAST gap - PR #131 added the flat-layout fallback
46
+ (verified live: setup it + check -l it, probva -> prova). el/tr/sv/it
47
+ all verified end-to-end. Suite 3701/0/27.
@@ -0,0 +1,88 @@
1
+ # 92 — Tier mirror CORS: finish the browser-models distribution
2
+
3
+ ## Context
4
+
5
+ The wasm model API shipped (kotoshu-rs PR #12: `loadModel` +
6
+ `rerank`), but the CORS spike found NO browser-usable source for tier
7
+ bytes: release assets send no ACAO; jsDelivr gh serves only 134-byte
8
+ LFS pointers. The ORIGINAL proposal (tier blobs in the git tree +
9
+ `browser_url`) would add ~1 GB to the repo — rejected on reflection.
10
+
11
+ The right fix is already half-built: **the media-host mirror sends
12
+ `Access-Control-Allow-Origin: *`** (proven by the spike) but today
13
+ mirrors only the 120 MB full tier (`mirror: null` for mini/fluency).
14
+
15
+ ## Job (models repo)
16
+
17
+ 1. Extend media-host mirroring to the mini and fluency tiers for all
18
+ 54 languages (the mirror upload path exists for full; tiers are
19
+ 3-15 MB, cheaper to mirror than full).
20
+ 2. Populate `mirror` for tier entries in registry.json; schema
21
+ unchanged.
22
+ 3. Verify from a browser-like context: `curl -I -H "Origin:
23
+ https://www.kotoshu.org" <mirror-url>` shows ACAO on tier files
24
+ AND the registry itself (the registry must stay fetchable too —
25
+ raw.githubusercontent already sends ACAO:*; document both paths).
26
+ 4. Release v1.2.1 (patch = distribution fix per plan 05 policy),
27
+ byte-identity + asset checks as usual.
28
+ 5. Record in this file's Status the exact fetch recipe for the
29
+ playground (registry URL + tier mirror URLs + the wasm API) so the
30
+ site wiring is mechanical once @kotoshu/wasm 0.2.0 is published
31
+ (that publish remains owner-gated).
32
+
33
+ ## Status
34
+
35
+ **DONE (2026-09-05, models PR #15 + tag v1.2.1).**
36
+
37
+ - All 54 languages x 2 tiers mirrored: every fasttext.{lang}.{mini,fluency}.onnx
38
+ and its .vocab.json entered LFS as media-host objects; urls.mirror
39
+ populated for all 162 resources (data rev 3 -> 4, schema unchanged).
40
+ Binaries are the v1.2.0 release bytes - all 216 files sha256-verified
41
+ against registry/tiers.json ground truth before staging, reused from the
42
+ expand/batch2 worktrees, nothing rebuilt locally. The v1.2.1 release
43
+ workflow rebuilt everything in CI and regenerated a registry whose 162
44
+ sha256 values are identical to the committed one (only generated_at
45
+ differs) - independent proof the mirror bytes equal the release bytes.
46
+ - Verification: 108/108 tier mirror URLs return HTTP 200/206 with
47
+ access-control-allow-origin: * and content-length exactly equal to
48
+ registry size_bytes (GET; note HEAD does NOT show ACAO on these hosts).
49
+ 16/16 vocab mirrors sha256-verified + ACAO. Release v1.2.1: 326 assets
50
+ (162 onnx + 162 vocab + registry.json + manifest-v1.2.1.json).
51
+ - Docs: README browser/wasm section + plan-92 addendum in
52
+ docs/storage-decision.md (rejected alternative: plain git blobs ~1 GB
53
+ per clone).
54
+
55
+ ### Browser fetch recipe (playground wiring, mechanical)
56
+
57
+ ```js
58
+ // 1. Registry - plain git file, raw host sends ACAO:* (real JSON, never
59
+ // an LFS pointer):
60
+ const REGISTRY_URL = (tag) =>
61
+ `https://raw.githubusercontent.com/kotoshu/models-fasttext-onnx/${tag}/registry.json`;
62
+
63
+ // 2. Model bytes - release assets send NO ACAO; use the registry mirror
64
+ // (media host = LFS content host, sends ACAO:*):
65
+ // https://media.githubusercontent.com/media/kotoshu/models-fasttext-onnx/main/models/{lang}/fasttext.{lang}.{tier}.onnx
66
+ // Vocab = same URL with .onnx -> .vocab.json (also LFS, also ACAO:*).
67
+
68
+ // 3. Engine - @kotoshu/wasm 0.2.0 (owner-gated publish), wasm-pack
69
+ // bundler target; jsDelivr npm raw + manual init works (esm.sh/esm.run
70
+ // do not for the bundler target - see plan 78 CDN pin):
71
+ import init, { loadModel, rerank } from "@kotoshu/wasm";
72
+
73
+ await init(/* fetch the .wasm yourself from the pinned CDN */);
74
+ const registry = await (await fetch(REGISTRY_URL("v1.2.1"))).json();
75
+ const entry = registry.resources[`kotoshu://models/${lang}/mini`]; // or fluency
76
+ const modelBytes = new Uint8Array(await (await fetch(entry.urls.mirror)).arrayBuffer());
77
+ const vocabBytes = new Uint8Array(await (await fetch(
78
+ entry.urls.mirror.replace(/\.onnx$/, ".vocab.json"))).arrayBuffer());
79
+
80
+ const model = loadModel(modelBytes, vocabBytes); // -> KotoshuModel
81
+ const score = rerank(model, word, context); // -> f32 in [-1, 1]
82
+ model.free(); // optional, before GC
83
+ ```
84
+
85
+ Sizes per language: mini ~3 MB + vocab ~0.25 MB; fluency ~15 MB +
86
+ vocab ~1.3 MB. sha256/size_bytes in each entry verify the mirror
87
+ bytes. Remaining owner gate: publish @kotoshu/wasm 0.2.0, then the
88
+ /playground can flip the matrix on.
@@ -0,0 +1,32 @@
1
+ # 93 — Python wheels for every platform + PyPI trusted publishing
2
+
3
+ ## Context
4
+
5
+ `kotoshu-native` 0.1.0 on PyPI ships an sdist + one cp310 macOS
6
+ arm64 wheel — every other Python user compiles from source (needs
7
+ Rust). maturin + cibuildwheel in CI produces the matrix mechanically.
8
+
9
+ ## Job (kotoshu-python repo)
10
+
11
+ 1. CI workflow building wheels: linux x86_64 + aarch64 (manylinux),
12
+ macOS x86_64 + arm64, windows; Python 3.10-3.13; maturin
13
+ sdist+wheel; artifact upload; a pip-install-and-import smoke per
14
+ platform.
15
+ 2. PyPI trusted publishing workflow (pypa/gh-action-pypi-publish,
16
+ id-token: write) gated on a tag — consistent with the ecosystem's
17
+ keyless posture. The OWNER registers trusted publishers at
18
+ pypi.org/manage/account/publishing for kotoshu + kotoshu-native
19
+ (workflow filename, no environment); publishing itself waits for
20
+ the owner's version call.
21
+ 3. Docs: README wheel-matrix table + the owner registration steps.
22
+
23
+ ## Status
24
+
25
+ **Implemented (2026-09-05, kotoshu-rs PR #13 + kotoshu-python PR #2).**
26
+ 20 wheels + sdist per run (manylinux x86_64/aarch64, macOS x86_64/
27
+ arm64, windows x64; py3.10-3.13), 17-assertion smoke per wheel;
28
+ keyless release-pypi.yml on both repos behind tags kotoshu-native-v* /
29
+ kotoshu-v*. Owner registrations ready (project-level: kotoshu ->
30
+ kotoshu-python/release-pypi.yml; kotoshu-native -> kotoshu-rs/
31
+ release-pypi.yml; environment blank). windows-arm64 dropped,
32
+ documented.
@@ -0,0 +1,35 @@
1
+ # 94 — Site: integrations docs + directory mode + action truth
2
+
3
+ ## Context
4
+
5
+ Wave 2 shipped features with zero site documentation: the Rails
6
+ validator, RSpec matchers, Rake task, Jekyll generator (gem PR #125)
7
+ and CLI directory mode (gem PR #124). The action page still says the
8
+ gem supports six languages.
9
+
10
+ ## Job (site repo)
11
+
12
+ 1. `/docs/integrations` — Rails (`validates :body, spelling: true`),
13
+ RSpec matchers, `rake kotoshu:check`, Jekyll generator, pre-commit
14
+ hook, GitHub Action — each example verified against the gem source
15
+ (read lib/kotoshu/integrations* or wherever PR #125 landed them;
16
+ run the rake task in a scratch dir if cheap).
17
+ 2. CLI page: directory mode — `kotoshu check .`, known extensions,
18
+ --include/--exclude, the gitignore subset (negation, globstar,
19
+ anchoring), baselines interaction. Flags verified against
20
+ `bundle exec exe/kotoshu check --help` in the gem repo.
21
+ 3. Action page + install page: language lists updated to 19
22
+ full-feature; action docs note directory mode (files: "." now
23
+ walks the tree) and baseline usage.
24
+ 4. News entry if the integrations wave was not already covered by a
25
+ prior entry (check news.ts — the full-feature-19 entry exists;
26
+ extend it or add "Framework integrations + directory mode").
27
+
28
+ ## Status
29
+
30
+ **Implemented (2026-09-05, site PR #7).** /docs/integrations with
31
+ every snippet EXECUTED (validator on ActiveModel 8.1.3.1, matchers,
32
+ rake default+DSL, Jekyll 4.4.1, walk rules, include/exclude, SARIF,
33
+ baseline); CLI directory-mode section with flags from `kotoshu help
34
+ check`; action/install truth. Two gem bugs found in the process were
35
+ fixed in PR #131 (standalone requires, canonical baseline paths).
@@ -0,0 +1,27 @@
1
+ # 95 — Editor + action polish
2
+
3
+ ## A — kotoshu-vscode: delegate add-to-dictionary
4
+
5
+ The extension's client-side write + "diagnostics stay until the
6
+ server adds support" message are now stale: kotoshu-lsp PR #3 added
7
+ the server-side `kotoshu.addToPersonalDictionary` command WITH
8
+ republish. Update the extension to execute the server command (via
9
+ the language client's executeCommand), falling back to the local
10
+ write if the server is older; fix the message; refresh the README
11
+ note. Smoke test in CI still passes (server built from source there).
12
+
13
+ ## B — action-kotoshu truth
14
+
15
+ - action.yml language description says six languages — now 19.
16
+ - Add an optional `baseline` input passing `--baseline` through
17
+ (directory mode makes the action a whole-repo checker; baselines
18
+ are its natural companion). README examples updated.
19
+
20
+ ## Status
21
+
22
+ **Implemented (2026-09-05, action PR #1 + kotoshu-vscode PR #2).**
23
+ Action: language input truth + next-release docs (the published 0.7.0
24
+ gem still gates six languages - directory mode/baselines/19 langs
25
+ ride the next gem cut). VS Code: add-to-dictionary delegates to the
26
+ server command (with republish) when advertised, local-write
27
+ fallback otherwise.