kotoshu 1.0.2-aarch64-linux

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (458) hide show
  1. checksums.yaml +7 -0
  2. data/.kotoshu-baseline.json +1 -0
  3. data/.pre-commit-hooks.yaml +18 -0
  4. data/.rspec +3 -0
  5. data/.rubocop.yml +351 -0
  6. data/.yardopts +8 -0
  7. data/CHANGELOG.md +813 -0
  8. data/CLAUDE.md +176 -0
  9. data/CODE_OF_CONDUCT.md +132 -0
  10. data/Cargo.lock +359 -0
  11. data/Cargo.toml +15 -0
  12. data/LICENSE +31 -0
  13. data/README.adoc +1292 -0
  14. data/Rakefile +150 -0
  15. data/SECURITY.md +93 -0
  16. data/TODO.impl/00-cut-0.1.md +121 -0
  17. data/TODO.impl/00-cut-0.2.md +175 -0
  18. data/TODO.impl/00-vision.md +281 -0
  19. data/TODO.impl/01-hunspell-correctness.md +111 -0
  20. data/TODO.impl/02-cli-unification.md +74 -0
  21. data/TODO.impl/03-dynamic-download.md +107 -0
  22. data/TODO.impl/04-language-modules.md +89 -0
  23. data/TODO.impl/05-semantic-path.md +83 -0
  24. data/TODO.impl/06-cjk-support.md +105 -0
  25. data/TODO.impl/07-rtl-support.md +71 -0
  26. data/TODO.impl/08-grammar-engine.md +106 -0
  27. data/TODO.impl/09-integrity-security.md +90 -0
  28. data/TODO.impl/10-testing-ci.md +98 -0
  29. data/TODO.impl/100-full-feature-batch3.md +28 -0
  30. data/TODO.impl/101-int4-tier-experiment.md +32 -0
  31. data/TODO.impl/102-lid-chain.md +28 -0
  32. data/TODO.impl/103-oov-bucket-tables.md +27 -0
  33. data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
  34. data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
  35. data/TODO.impl/106-native-lid-detection.md +36 -0
  36. data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
  37. data/TODO.impl/108-ko-ne-full-feature.md +37 -0
  38. data/TODO.impl/109-ten-audit.md +26 -0
  39. data/TODO.impl/11-release-v1.md +95 -0
  40. data/TODO.impl/110-sr-latn-and-polish.md +33 -0
  41. data/TODO.impl/111-embedding-bakeoff.md +33 -0
  42. data/TODO.impl/112-deployability-and-gating.md +26 -0
  43. data/TODO.impl/113-language-packs.md +26 -0
  44. data/TODO.impl/114-typo-biencoder-thread.md +33 -0
  45. data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
  46. data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
  47. data/TODO.impl/117-deterministic-frequency-tiers.md +110 -0
  48. data/TODO.impl/118-server-gem-floor.md +33 -0
  49. data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
  50. data/TODO.impl/120-setup-metadata-refresh.md +35 -0
  51. data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
  52. data/TODO.impl/122-pack-compression-kpk2.md +66 -0
  53. data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
  54. data/TODO.impl/124-owner-residue.md +19 -0
  55. data/TODO.impl/125-consumer-floor-guards.md +43 -0
  56. data/TODO.impl/126-tier-error-budgets.md +33 -0
  57. data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
  58. data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
  59. data/TODO.impl/129-pt-fr-corpora.md +23 -0
  60. data/TODO.impl/130-news-truth-0912.md +23 -0
  61. data/TODO.impl/131-hybrid-productization.md +42 -0
  62. data/TODO.impl/132-techniques-page-and-prose.md +27 -0
  63. data/TODO.impl/133-rust-first-distribution.md +68 -0
  64. data/TODO.impl/20-encapsulation-cleanup.md +47 -0
  65. data/TODO.impl/21-autoload-migration.md +64 -0
  66. data/TODO.impl/22-url-pin-single-source-of-truth.md +120 -0
  67. data/TODO.impl/23-fix-url-defaults.md +57 -0
  68. data/TODO.impl/24-cli-auto-setup-prompt.md +76 -0
  69. data/TODO.impl/25-soft-onnxruntime-require.md +75 -0
  70. data/TODO.impl/26-onnx-vocab-json-generation.md +96 -0
  71. data/TODO.impl/27-end-to-end-smoke-spec.md +113 -0
  72. data/TODO.impl/28-readme-quickstart-verification.md +68 -0
  73. data/TODO.impl/29-status-command.md +84 -0
  74. data/TODO.impl/30-language-auto-detection.md +74 -0
  75. data/TODO.impl/31-download-progress.md +58 -0
  76. data/TODO.impl/32-sarif-output-verification.md +72 -0
  77. data/TODO.impl/33-audit-log-rotation.md +54 -0
  78. data/TODO.impl/34-cache-eviction.md +59 -0
  79. data/TODO.impl/35-shell-completion.md +61 -0
  80. data/TODO.impl/36-cleanup-regressions-0.3.1.md +134 -0
  81. data/TODO.impl/37-hunspell-correctness-tier2.md +203 -0
  82. data/TODO.impl/38-onnx-semantic-gating.md +218 -0
  83. data/TODO.impl/39-tier3-and-beyond.md +232 -0
  84. data/TODO.impl/40-spec-drift-cleanup.md +134 -0
  85. data/TODO.impl/50-structure-aware-document-api.md +194 -0
  86. data/TODO.impl/51-grammar-engine-expansion.md +75 -0
  87. data/TODO.impl/52-multi-language-documents.md +95 -0
  88. data/TODO.impl/53-document-plugin-architecture.md +94 -0
  89. data/TODO.impl/54-cjk-support.md +117 -0
  90. data/TODO.impl/55-rtl-support.md +109 -0
  91. data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
  92. data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
  93. data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
  94. data/TODO.impl/59-yard-api-documentation.md +75 -0
  95. data/TODO.impl/60-lsp-server.md +174 -0
  96. data/TODO.impl/61-editor-ecosystem.md +167 -0
  97. data/TODO.impl/62-framework-integrations.md +159 -0
  98. data/TODO.impl/63-web-playground-wasm.md +186 -0
  99. data/TODO.impl/64-http-api-and-sdks.md +191 -0
  100. data/TODO.impl/65-universal-kotoshu.md +251 -0
  101. data/TODO.impl/66-kotoshu-core.md +169 -0
  102. data/TODO.impl/67-kotoshu-rs-and-access-libraries.md +137 -0
  103. data/TODO.impl/68-sota-adoption.md +152 -0
  104. data/TODO.impl/69-sota-typo-eval.md +74 -0
  105. data/TODO.impl/70-sota-confidence-cascade.md +59 -0
  106. data/TODO.impl/71-sota-nested-model.md +63 -0
  107. data/TODO.impl/72-model2vec-evaluation.md +49 -0
  108. data/TODO.impl/73-rl-reranker.md +42 -0
  109. data/TODO.impl/74-distilled-reranker.md +36 -0
  110. data/TODO.impl/75-llm-evaluator.md +35 -0
  111. data/TODO.impl/76-optional-llm-backend.md +37 -0
  112. data/TODO.impl/77-model-coverage-expansion.md +96 -0
  113. data/TODO.impl/78-in-browser-playground.md +69 -0
  114. data/TODO.impl/79-news-and-releases.md +60 -0
  115. data/TODO.impl/80-ecosystem-docs.md +56 -0
  116. data/TODO.impl/81-editor-reach.md +68 -0
  117. data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
  118. data/TODO.impl/83-models-batch-2.md +56 -0
  119. data/TODO.impl/84-language-modules-wave2.md +75 -0
  120. data/TODO.impl/85-wasm-model-api.md +48 -0
  121. data/TODO.impl/86-site-truth-wave2.md +45 -0
  122. data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
  123. data/TODO.impl/88-cli-directory-mode.md +54 -0
  124. data/TODO.impl/89-framework-integrations-exec.md +59 -0
  125. data/TODO.impl/90-gem-correctness-residues.md +44 -0
  126. data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
  127. data/TODO.impl/92-tier-mirror-cors.md +88 -0
  128. data/TODO.impl/93-python-wheels.md +32 -0
  129. data/TODO.impl/94-site-integrations-docs.md +35 -0
  130. data/TODO.impl/95-editor-action-polish.md +27 -0
  131. data/TODO.impl/96-server-semantic-models.md +50 -0
  132. data/TODO.impl/97-yard-execution.md +73 -0
  133. data/TODO.impl/98-campaign-cleanup.md +78 -0
  134. data/TODO.impl/99-site-truth-performance-docs.md +31 -0
  135. data/conformance/vectors.jsonl +2630 -0
  136. data/data/en/grammar/rules.yaml +851 -0
  137. data/docs/STABILITY-1.0-AUDIT.md +251 -0
  138. data/docs/STABILITY-1.0.md +137 -0
  139. data/docs/cache-architecture.md +502 -0
  140. data/docs/verification-evidence/correct3/pr_body.md +37 -0
  141. data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
  142. data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
  143. data/docs/verification-evidence/correct3/sv.aff +856 -0
  144. data/docs/verification-evidence/correct3/sv.dic +153717 -0
  145. data/examples/01_basic_word_checking.rb +38 -0
  146. data/examples/02_text_document_checking.rb +77 -0
  147. data/examples/03_dictionary_backends.rb +137 -0
  148. data/examples/04_trie_data_structure.rb +146 -0
  149. data/examples/05_suggestion_algorithms.rb +239 -0
  150. data/examples/06_configuration_advanced.rb +287 -0
  151. data/examples/07_multi_language_dictionaries.rb +278 -0
  152. data/exe/kotoshu +6 -0
  153. data/lib/kotoshu/algorithms/capitalization.rb +276 -0
  154. data/lib/kotoshu/algorithms/edit_distance.rb +128 -0
  155. data/lib/kotoshu/algorithms/lookup/affix_form.rb +132 -0
  156. data/lib/kotoshu/algorithms/lookup/compound_checks.rb +209 -0
  157. data/lib/kotoshu/algorithms/lookup/compound_form.rb +50 -0
  158. data/lib/kotoshu/algorithms/lookup.rb +841 -0
  159. data/lib/kotoshu/algorithms/ngram_suggest.rb +367 -0
  160. data/lib/kotoshu/algorithms/permutations.rb +283 -0
  161. data/lib/kotoshu/algorithms/phonet_suggest.rb +162 -0
  162. data/lib/kotoshu/algorithms/soundex.rb +92 -0
  163. data/lib/kotoshu/algorithms/suggest.rb +794 -0
  164. data/lib/kotoshu/algorithms.rb +23 -0
  165. data/lib/kotoshu/analyzers/semantic_analyzer.rb +405 -0
  166. data/lib/kotoshu/analyzers.rb +8 -0
  167. data/lib/kotoshu/baseline/application.rb +18 -0
  168. data/lib/kotoshu/baseline/entry.rb +33 -0
  169. data/lib/kotoshu/baseline/store.rb +175 -0
  170. data/lib/kotoshu/baseline.rb +20 -0
  171. data/lib/kotoshu/cache/base_cache.rb +778 -0
  172. data/lib/kotoshu/cache/cache.rb +91 -0
  173. data/lib/kotoshu/cache/eviction_policy.rb +66 -0
  174. data/lib/kotoshu/cache/frequency_cache.rb +283 -0
  175. data/lib/kotoshu/cache/language_cache.rb +511 -0
  176. data/lib/kotoshu/cache/lookup_cache.rb +166 -0
  177. data/lib/kotoshu/cache/model_cache.rb +1147 -0
  178. data/lib/kotoshu/cache/model_registry.rb +109 -0
  179. data/lib/kotoshu/cache/staged_languages.rb +45 -0
  180. data/lib/kotoshu/cache/suggestion_cache.rb +113 -0
  181. data/lib/kotoshu/cache.rb +49 -0
  182. data/lib/kotoshu/cli/auto_setup.rb +71 -0
  183. data/lib/kotoshu/cli/baseline_command.rb +119 -0
  184. data/lib/kotoshu/cli/batch_reporter.rb +321 -0
  185. data/lib/kotoshu/cli/cache_command.rb +282 -0
  186. data/lib/kotoshu/cli/completions_command.rb +210 -0
  187. data/lib/kotoshu/cli/dict_command.rb +64 -0
  188. data/lib/kotoshu/cli/directory_check.rb +146 -0
  189. data/lib/kotoshu/cli/directory_walker.rb +132 -0
  190. data/lib/kotoshu/cli/display_formatter.rb +429 -0
  191. data/lib/kotoshu/cli/errors.rb +36 -0
  192. data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
  193. data/lib/kotoshu/cli/interactive_reviewer.rb +314 -0
  194. data/lib/kotoshu/cli/language_resolver.rb +91 -0
  195. data/lib/kotoshu/cli/navigation_manager.rb +273 -0
  196. data/lib/kotoshu/cli/personal_command.rb +113 -0
  197. data/lib/kotoshu/cli/progress_reporter.rb +114 -0
  198. data/lib/kotoshu/cli/status_report.rb +130 -0
  199. data/lib/kotoshu/cli.rb +880 -0
  200. data/lib/kotoshu/components/passthrough_spell_checker.rb +70 -0
  201. data/lib/kotoshu/components/pos_tagger.rb +98 -0
  202. data/lib/kotoshu/components/spell_checker.rb +73 -0
  203. data/lib/kotoshu/components/synthesizer.rb +60 -0
  204. data/lib/kotoshu/components/tokenizer.rb +58 -0
  205. data/lib/kotoshu/components/whitespace_tokenizer.rb +94 -0
  206. data/lib/kotoshu/components.rb +13 -0
  207. data/lib/kotoshu/configuration/builder.rb +207 -0
  208. data/lib/kotoshu/configuration/resolver.rb +124 -0
  209. data/lib/kotoshu/configuration.rb +829 -0
  210. data/lib/kotoshu/conformance_exporter.rb +256 -0
  211. data/lib/kotoshu/conformance_runner.rb +176 -0
  212. data/lib/kotoshu/core/exceptions.rb +186 -0
  213. data/lib/kotoshu/core/indexed_dictionary.rb +293 -0
  214. data/lib/kotoshu/core/models/affix_rule.rb +262 -0
  215. data/lib/kotoshu/core/models/result/document_result.rb +132 -0
  216. data/lib/kotoshu/core/models/result/word_result.rb +131 -0
  217. data/lib/kotoshu/core/models/word.rb +131 -0
  218. data/lib/kotoshu/core/trie/builder.rb +119 -0
  219. data/lib/kotoshu/core/trie/node.rb +94 -0
  220. data/lib/kotoshu/core/trie/trie.rb +249 -0
  221. data/lib/kotoshu/core.rb +28 -0
  222. data/lib/kotoshu/data/common_words/de.yml +1800 -0
  223. data/lib/kotoshu/data/common_words/en.yml +1215 -0
  224. data/lib/kotoshu/data/common_words/es.yml +750 -0
  225. data/lib/kotoshu/data/common_words/fr.yml +1015 -0
  226. data/lib/kotoshu/data/common_words/pt.yml +870 -0
  227. data/lib/kotoshu/data/common_words/ru.yml +484 -0
  228. data/lib/kotoshu/data/common_words_loader.rb +141 -0
  229. data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
  230. data/lib/kotoshu/data/frozen_kelly.rb +38 -0
  231. data/lib/kotoshu/data.rb +8 -0
  232. data/lib/kotoshu/data_structures/bloom_filter.rb +188 -0
  233. data/lib/kotoshu/data_structures.rb +8 -0
  234. data/lib/kotoshu/debug_logger.rb +146 -0
  235. data/lib/kotoshu/debug_mode.rb +136 -0
  236. data/lib/kotoshu/defaults.rb +84 -0
  237. data/lib/kotoshu/dictionaries/catalog.rb +814 -0
  238. data/lib/kotoshu/dictionaries.rb +8 -0
  239. data/lib/kotoshu/dictionary/base.rb +324 -0
  240. data/lib/kotoshu/dictionary/cspell.rb +243 -0
  241. data/lib/kotoshu/dictionary/custom.rb +225 -0
  242. data/lib/kotoshu/dictionary/hunspell.rb +520 -0
  243. data/lib/kotoshu/dictionary/plain_text.rb +302 -0
  244. data/lib/kotoshu/dictionary/repository.rb +246 -0
  245. data/lib/kotoshu/dictionary/unified.rb +236 -0
  246. data/lib/kotoshu/dictionary/unix_words.rb +218 -0
  247. data/lib/kotoshu/dictionary.rb +15 -0
  248. data/lib/kotoshu/documents/document.rb +120 -0
  249. data/lib/kotoshu/documents/plain_text_document.rb +73 -0
  250. data/lib/kotoshu/documents/source_position.rb +45 -0
  251. data/lib/kotoshu/documents/source_range.rb +76 -0
  252. data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
  253. data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
  254. data/lib/kotoshu/documents/suppressions.rb +44 -0
  255. data/lib/kotoshu/documents/text_node.rb +49 -0
  256. data/lib/kotoshu/documents.rb +178 -0
  257. data/lib/kotoshu/embeddings/embedding_pipeline.rb +238 -0
  258. data/lib/kotoshu/embeddings/lru_cache.rb +244 -0
  259. data/lib/kotoshu/embeddings/onnx_runtime_model.rb +411 -0
  260. data/lib/kotoshu/embeddings/protocol.rb +96 -0
  261. data/lib/kotoshu/embeddings/protocols.rb +23 -0
  262. data/lib/kotoshu/embeddings/registry.rb +184 -0
  263. data/lib/kotoshu/embeddings/search.rb +204 -0
  264. data/lib/kotoshu/embeddings/similarity_engine.rb +278 -0
  265. data/lib/kotoshu/embeddings/similarity_search.rb +328 -0
  266. data/lib/kotoshu/embeddings/vocabulary.rb +306 -0
  267. data/lib/kotoshu/embeddings.rb +87 -0
  268. data/lib/kotoshu/fluent_checker.rb +94 -0
  269. data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +48 -0
  270. data/lib/kotoshu/grammar/pattern_matchers/double_negative_matcher.rb +179 -0
  271. data/lib/kotoshu/grammar/pattern_matchers/phrase_matcher.rb +86 -0
  272. data/lib/kotoshu/grammar/pattern_matchers/possessive_context_matcher.rb +75 -0
  273. data/lib/kotoshu/grammar/pattern_matchers/possessive_contraction_matcher.rb +87 -0
  274. data/lib/kotoshu/grammar/pattern_matchers/sentence_start_matcher.rb +84 -0
  275. data/lib/kotoshu/grammar/pattern_matchers/vowel_sound_matcher.rb +81 -0
  276. data/lib/kotoshu/grammar/pattern_matchers/word_list_matcher.rb +66 -0
  277. data/lib/kotoshu/grammar/rule.rb +99 -0
  278. data/lib/kotoshu/grammar/rule_engine.rb +96 -0
  279. data/lib/kotoshu/grammar/rule_loader.rb +30 -0
  280. data/lib/kotoshu/grammar.rb +24 -0
  281. data/lib/kotoshu/integrity/audit_log.rb +167 -0
  282. data/lib/kotoshu/integrity/manifest.rb +116 -0
  283. data/lib/kotoshu/integrity/net_http.rb +51 -0
  284. data/lib/kotoshu/integrity/rotation_policy.rb +88 -0
  285. data/lib/kotoshu/integrity.rb +26 -0
  286. data/lib/kotoshu/jekyll.rb +101 -0
  287. data/lib/kotoshu/keyboard/layout.rb +116 -0
  288. data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
  289. data/lib/kotoshu/keyboard/layouts/azerty.rb +55 -0
  290. data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
  291. data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
  292. data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
  293. data/lib/kotoshu/keyboard/layouts/dvorak.rb +54 -0
  294. data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
  295. data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
  296. data/lib/kotoshu/keyboard/layouts/hrsl.rb +94 -0
  297. data/lib/kotoshu/keyboard/layouts/jcuken.rb +57 -0
  298. data/lib/kotoshu/keyboard/layouts/latin.rb +266 -0
  299. data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
  300. data/lib/kotoshu/keyboard/layouts/qwerty.rb +52 -0
  301. data/lib/kotoshu/keyboard/layouts/qwertz.rb +55 -0
  302. data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
  303. data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
  304. data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
  305. data/lib/kotoshu/keyboard/registry.rb +201 -0
  306. data/lib/kotoshu/keyboard.rb +96 -0
  307. data/lib/kotoshu/kotoshu_native.so +0 -0
  308. data/lib/kotoshu/language/detector.rb +242 -0
  309. data/lib/kotoshu/language/identifier.rb +376 -0
  310. data/lib/kotoshu/language/languages/base.rb +271 -0
  311. data/lib/kotoshu/language/lid_detector.rb +188 -0
  312. data/lib/kotoshu/language/normalizer/arabic.rb +113 -0
  313. data/lib/kotoshu/language/normalizer/base.rb +133 -0
  314. data/lib/kotoshu/language/normalizer/greek.rb +63 -0
  315. data/lib/kotoshu/language/normalizer/hebrew.rb +72 -0
  316. data/lib/kotoshu/language/normalizer/persian.rb +36 -0
  317. data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
  318. data/lib/kotoshu/language/registry.rb +209 -0
  319. data/lib/kotoshu/language/resources/ar/common_words.txt +6753 -0
  320. data/lib/kotoshu/language/resources/ar/confusion_sets.txt +11 -0
  321. data/lib/kotoshu/language/resources/de/common_words.txt +10003 -0
  322. data/lib/kotoshu/language/resources/de/confusion_sets.txt +246 -0
  323. data/lib/kotoshu/language/resources/en/common_words.txt +9979 -0
  324. data/lib/kotoshu/language/resources/en/confusion_sets.txt +871 -0
  325. data/lib/kotoshu/language/resources/es/common_words.txt +9992 -0
  326. data/lib/kotoshu/language/resources/es/confusion_sets.txt +17 -0
  327. data/lib/kotoshu/language/resources/fr/common_words.txt +9993 -0
  328. data/lib/kotoshu/language/resources/fr/confusion_sets.txt +76 -0
  329. data/lib/kotoshu/language/resources/pt/common_words.txt +9977 -0
  330. data/lib/kotoshu/language/resources/pt/confusion_sets.txt +18 -0
  331. data/lib/kotoshu/language/resources/ru/common_words.txt +9951 -0
  332. data/lib/kotoshu/language/resources/ru/confusion_sets.txt +5 -0
  333. data/lib/kotoshu/language/script.rb +128 -0
  334. data/lib/kotoshu/language/segmenter.rb +122 -0
  335. data/lib/kotoshu/language/suika.rb +38 -0
  336. data/lib/kotoshu/language/tokenizer/base.rb +187 -0
  337. data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
  338. data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
  339. data/lib/kotoshu/language/tokenizer/french_tokenizer.rb +172 -0
  340. data/lib/kotoshu/language/tokenizer/german_tokenizer.rb +41 -0
  341. data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
  342. data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
  343. data/lib/kotoshu/language/tokenizer/japanese_tokenizer.rb +55 -0
  344. data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +153 -0
  345. data/lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb +160 -0
  346. data/lib/kotoshu/language/tokenizer/russian_tokenizer.rb +93 -0
  347. data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
  348. data/lib/kotoshu/language/tokenizer/spanish_tokenizer.rb +122 -0
  349. data/lib/kotoshu/language.rb +127 -0
  350. data/lib/kotoshu/languages/ar/language.rb +98 -0
  351. data/lib/kotoshu/languages/bg/language.rb +37 -0
  352. data/lib/kotoshu/languages/ca/language.rb +22 -0
  353. data/lib/kotoshu/languages/cs/language.rb +22 -0
  354. data/lib/kotoshu/languages/da/language.rb +22 -0
  355. data/lib/kotoshu/languages/de/language.rb +554 -0
  356. data/lib/kotoshu/languages/el/language.rb +50 -0
  357. data/lib/kotoshu/languages/en/language.rb +470 -0
  358. data/lib/kotoshu/languages/es/language.rb +476 -0
  359. data/lib/kotoshu/languages/et/language.rb +24 -0
  360. data/lib/kotoshu/languages/fa/language.rb +91 -0
  361. data/lib/kotoshu/languages/fr/language.rb +505 -0
  362. data/lib/kotoshu/languages/he/language.rb +98 -0
  363. data/lib/kotoshu/languages/hr/language.rb +24 -0
  364. data/lib/kotoshu/languages/hu/language.rb +22 -0
  365. data/lib/kotoshu/languages/id/language.rb +25 -0
  366. data/lib/kotoshu/languages/it/language.rb +22 -0
  367. data/lib/kotoshu/languages/ja/language.rb +476 -0
  368. data/lib/kotoshu/languages/ko/language.rb +56 -0
  369. data/lib/kotoshu/languages/latin_base.rb +53 -0
  370. data/lib/kotoshu/languages/lt/language.rb +24 -0
  371. data/lib/kotoshu/languages/lv/language.rb +24 -0
  372. data/lib/kotoshu/languages/nb/language.rb +25 -0
  373. data/lib/kotoshu/languages/ne/language.rb +60 -0
  374. data/lib/kotoshu/languages/nl/language.rb +22 -0
  375. data/lib/kotoshu/languages/nn/language.rb +26 -0
  376. data/lib/kotoshu/languages/pl/language.rb +22 -0
  377. data/lib/kotoshu/languages/pt/language.rb +439 -0
  378. data/lib/kotoshu/languages/ro/language.rb +22 -0
  379. data/lib/kotoshu/languages/ru/language.rb +420 -0
  380. data/lib/kotoshu/languages/sk/language.rb +25 -0
  381. data/lib/kotoshu/languages/sl/language.rb +25 -0
  382. data/lib/kotoshu/languages/sr/language.rb +39 -0
  383. data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
  384. data/lib/kotoshu/languages/sv/language.rb +22 -0
  385. data/lib/kotoshu/languages/tr/language.rb +36 -0
  386. data/lib/kotoshu/languages/uk/language.rb +38 -0
  387. data/lib/kotoshu/languages/vi/language.rb +22 -0
  388. data/lib/kotoshu/languages.rb +64 -0
  389. data/lib/kotoshu/metrics_collector.rb +238 -0
  390. data/lib/kotoshu/metrics_module.rb +121 -0
  391. data/lib/kotoshu/models/context.rb +119 -0
  392. data/lib/kotoshu/models/embedding_model.rb +181 -0
  393. data/lib/kotoshu/models/fasttext_model.rb +212 -0
  394. data/lib/kotoshu/models/nearest_neighbor.rb +96 -0
  395. data/lib/kotoshu/models/onnx_model.rb +330 -0
  396. data/lib/kotoshu/models/semantic_error.rb +204 -0
  397. data/lib/kotoshu/models/suggestion.rb +106 -0
  398. data/lib/kotoshu/models/word_embedding.rb +97 -0
  399. data/lib/kotoshu/models.rb +27 -0
  400. data/lib/kotoshu/multi_language_checker.rb +127 -0
  401. data/lib/kotoshu/native.rb +83 -0
  402. data/lib/kotoshu/native_backend.rb +131 -0
  403. data/lib/kotoshu/paths.rb +79 -0
  404. data/lib/kotoshu/personal_dictionary.rb +105 -0
  405. data/lib/kotoshu/plugins/plugin.rb +61 -0
  406. data/lib/kotoshu/plugins/registry.rb +118 -0
  407. data/lib/kotoshu/plugins.rb +9 -0
  408. data/lib/kotoshu/project_config.rb +76 -0
  409. data/lib/kotoshu/readers/aff_data.rb +530 -0
  410. data/lib/kotoshu/readers/aff_reader.rb +393 -0
  411. data/lib/kotoshu/readers/condition_checker.rb +125 -0
  412. data/lib/kotoshu/readers/dic_reader.rb +208 -0
  413. data/lib/kotoshu/readers/file_reader.rb +377 -0
  414. data/lib/kotoshu/readers/lookup_builder.rb +584 -0
  415. data/lib/kotoshu/readers/readers.rb +17 -0
  416. data/lib/kotoshu/readers.rb +25 -0
  417. data/lib/kotoshu/resource_bundle.rb +55 -0
  418. data/lib/kotoshu/resource_manager.rb +600 -0
  419. data/lib/kotoshu/results/result.rb +165 -0
  420. data/lib/kotoshu/results.rb +8 -0
  421. data/lib/kotoshu/rspec.rb +163 -0
  422. data/lib/kotoshu/scripts/fasttext_to_onnx.py +275 -0
  423. data/lib/kotoshu/source_registry.rb +104 -0
  424. data/lib/kotoshu/spellchecker/parallel_checker.rb +90 -0
  425. data/lib/kotoshu/spellchecker.rb +406 -0
  426. data/lib/kotoshu/string_metrics.rb +153 -0
  427. data/lib/kotoshu/suggestions/context.rb +43 -0
  428. data/lib/kotoshu/suggestions/frequency_provider.rb +80 -0
  429. data/lib/kotoshu/suggestions/generator.rb +169 -0
  430. data/lib/kotoshu/suggestions/pipeline.rb +135 -0
  431. data/lib/kotoshu/suggestions/semantic_cascade.rb +73 -0
  432. data/lib/kotoshu/suggestions/strategies/base_strategy.rb +315 -0
  433. data/lib/kotoshu/suggestions/strategies/composite_strategy.rb +197 -0
  434. data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +631 -0
  435. data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +207 -0
  436. data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +165 -0
  437. data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +284 -0
  438. data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +317 -0
  439. data/lib/kotoshu/suggestions/strategies/symspell_strategy.rb +270 -0
  440. data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
  441. data/lib/kotoshu/suggestions/suggestion.rb +154 -0
  442. data/lib/kotoshu/suggestions/suggestion_set.rb +270 -0
  443. data/lib/kotoshu/suggestions/sweep_index.rb +137 -0
  444. data/lib/kotoshu/suggestions.rb +29 -0
  445. data/lib/kotoshu/tasks/check_task.rb +104 -0
  446. data/lib/kotoshu/tasks.rb +11 -0
  447. data/lib/kotoshu/validators/spelling_validator.rb +101 -0
  448. data/lib/kotoshu/validators.rb +9 -0
  449. data/lib/kotoshu/version.rb +5 -0
  450. data/lib/kotoshu.rb +538 -0
  451. data/rakelib/compile.rake +10 -0
  452. data/script/validate_all_dictionaries.rb +444 -0
  453. data/scripts/build_native_gem.rb +45 -0
  454. data/scripts/extract_eval_grids.rb +37 -0
  455. data/scripts/generate_frozen_tiers.rb +44 -0
  456. data/sig/kotoshu.rbs +435 -0
  457. data/test_oop.rb +79 -0
  458. metadata +546 -0
data/README.adoc ADDED
@@ -0,0 +1,1292 @@
1
+ = Kotoshu: Semantic Spell Checking for Ruby
2
+
3
+ image:https://img.shields.io/gem/v/kotoshu.svg[RubyGems Version]
4
+ image:https://img.shields.io/github/license/kotoshu/kotoshu.svg[License]
5
+ image:https://github.com/kotoshu/kotoshu/actions/workflows/main.yml/badge.svg["CI", link="https://github.com/kotoshu/kotoshu/actions/workflows/main.yml"]
6
+
7
+ == Status
8
+
9
+ Kotoshu is **v0.3.0** — building on the 0.2 cut, this release adds a strict
10
+ two-stage resource model (explicit `setup`, cache-only hot path), XDG base
11
+ directory layout, SHA-256 integrity verification, SARIF output, an
12
+ `--interactive` review loop, and ONNX model pipeline wiring.
13
+
14
+ === What works in 0.3
15
+
16
+ * Two-stage resource model — `Kotoshu.setup(:en)` then `Kotoshu.correct?("hello")`. The hot path is cache-only and raises `ResourceNotSetupError` on miss; downloads are never implicit.
17
+ * `Kotoshu.check(text, language: "en")` / `Kotoshu.suggest("helo")` — full document check and suggestions
18
+ * `Kotoshu.spellchecker_for(lang)` — per-language checker resolved from the cache (`strict:` is a `Kotoshu.setup` option)
19
+ * `kotoshu check FILE` CLI with these flags:
20
+ ** `--language LANG|auto` — any set-up language code, script variants included (`sr-Latn`) (default: `auto`)
21
+ ** `--format text|json|sarif`
22
+ ** `--offline` — use only cached resources, never download
23
+ ** `--strict` — exit 3 if any optional resource (frequency, model) can't load
24
+ ** `--interactive` — review each error after the check
25
+ ** `--verbose`
26
+ * `kotoshu setup LANGUAGE [LANGUAGE ...]` — pre-warm spelling + frequency + ONNX caches for offline use (`fetch` is kept as a hidden deprecated alias)
27
+ * Local-source setup: `kotoshu setup en --aff path/to.en.aff --dic path/to.en.dic` or `kotoshu setup en --from /path/to/dict/dir/`
28
+ * Exit codes: `0` clean, `1` errors found, `2` usage error, `3` resource setup failed
29
+ * SHA-256 integrity verification (manifest-based, with graceful degradation when manifest is absent)
30
+ * Offline mode via `KOTOSHU_OFFLINE=1` or `--offline`
31
+ * XDG base directory layout — caches in `$XDG_CACHE_HOME/kotoshu/`, config in `$XDG_CONFIG_HOME/kotoshu/`, data in `$XDG_LOCAL_HOME/kotoshu/` (overridable via `KOTOSHU_CACHE_PATH`, `KOTOSHU_CONFIG_PATH`, `KOTOSHU_DATA_PATH`)
32
+
33
+ === Planned for 0.4+
34
+
35
+ * `--output` (file output redirection)
36
+ * ONNX semantic reranking as default path
37
+ * ≥30 language modules wired
38
+ * Grammar rule packs
39
+ * CJK and RTL language support
40
+
41
+ See link:TODO.impl/00-cut-0.2.md[the 0.2 cut plan], the 0.3 tasks under
42
+ `TODO.impl/`, and link:TODO.impl/00-vision.md[the vision] for the path to 1.0.
43
+
44
+ == Purpose
45
+
46
+ Kotoshu 「言修」 is a pure-Ruby spell checker that aims to work for every
47
+ language by dynamically downloading the right combination of dictionary,
48
+ frequency data, and embedding model on demand.
49
+
50
+ The current release pairs a Ruby port of the Hunspell algorithm
51
+ (traditional morphological lookup + affix rules) with optional FastText
52
+ word embeddings converted to ONNX for context-aware reranking.
53
+
54
+ NOTE: The semantic (ONNX) path is an optional feature. `gem install kotoshu`
55
+ works without `onnxruntime`; install it separately (`gem install onnxruntime`)
56
+ to enable context-aware reranking. Set `KOTOSHU_NO_ONNX=1` to opt back out.
57
+
58
+ == Features
59
+
60
+ NOTE: The list below describes the design vision. See <<status>> for
61
+ exactly what works in 0.2 and what is planned for 0.3+.
62
+
63
+ * <<multi-language, Multi-language support with automatic detection>>
64
+ * <<semantic-analysis, Semantic error detection using word embeddings>> (opt-in via Ruby API in 0.2)
65
+ * <<interactive-mode, Interactive review mode with full navigation>> (planned for 0.3)
66
+ * <<batch-processing, Batch processing for CI/CD>> (JSON in 0.2; SARIF planned for 0.3)
67
+ * <<onnx-models, Fast ONNX inference via ONNX Runtime>>
68
+ * <<document-formats, Support for Markdown, AsciiDoc, and plain text>>
69
+ * <<multiple-models, Multiple analysis models (Hunspell, FastText, Hybrid)>> (Hunspell path only in 0.2)
70
+
71
+ == Architecture
72
+
73
+ Kotoshu is built on a modern, semantic architecture:
74
+
75
+ .Architecture overview
76
+ [source]
77
+ ----
78
+ ╔═══════════════════════════════════════════════════════════════════╗
79
+ ║ Kotoshu Semantic Architecture ║
80
+ ╠═══════════════════════════════════════════════════════════════════╣
81
+ ║ ║
82
+ ║ ┌─────────────────────────────────────────────────────────────┐ ║
83
+ ║ │ Interface Layer │ ║
84
+ ║ │ ┌─────────────────────┐ ┌─────────────────────────────┐ │ ║
85
+ ║ │ │ CLI (Thor) │ │ Ruby API │ │ ║
86
+ ║ │ │ lib/kotoshu/cli/ │ │ Kotoshu module methods │ │ ║
87
+ ║ │ └──────────┬──────────┘ └───────────┬─────────────────┘ │ ║
88
+ ║ │ │ Auto Language Detect │ │ ║
89
+ ║ └─────────────┼──────────────────────────┼─────────────────────┘ ║
90
+ ║ │ │ ║
91
+ ║ ▼ ▼ ║
92
+ ║ ┌─────────────────────────────────────────────────────────────┐ ║
93
+ ║ │ Analysis Layer │ ║
94
+ ║ │ ┌──────────────┐ ┌─────────────┐ ┌───────────────────┐ │ ║
95
+ ║ │ │ Hunspell │ │ FastText │ │ Hybrid (Best!) │ │ ║
96
+ ║ │ │ Dictionary │ │ Embeddings │ │ Combined │ │ ║
97
+ ║ │ │ (Traditional)│ │ (ONNX) │ │ Approach │ │ ║
98
+ ║ │ └──────────────┘ └─────────────┘ └───────────────────┘ │ ║
99
+ ║ └───────────────────────────┬─────────────────────────────────┘ ║
100
+ ║ │ ║
101
+ ║ ┌───────────────────────────▼─────────────────────────────────┐ ║
102
+ ║ │ Model Layer (ONNX) │ ║
103
+ ║ │ ┌──────────────────────────────────────────────────────┐ │ ║
104
+ ║ │ │ ONNX Runtime → Fast Embedding Lookup │ │ ║
105
+ ║ │ │ Semantic Similarity → Context-Aware Suggestions │ │ ║
106
+ ║ │ │ Nearest Neighbor Search → Smart Corrections │ │ ║
107
+ ║ │ └──────────────────────────────────────────────────────┘ │ ║
108
+ ║ └─────────────────────────────────────────────────────────────┘ ║
109
+ ║ ║
110
+ ╚═══════════════════════════════════════════════════════════════════╝
111
+ ----
112
+
113
+ === Key Components
114
+
115
+ * `Kotoshu::Models::OnnxModel`: ONNX-based word embedding model for fast
116
+ semantic similarity and nearest neighbor search.
117
+
118
+ * `Kotoshu::Analyzers::SemanticAnalyzer`: Unified semantic error detection
119
+ using word embeddings (no artificial spelling/grammar split).
120
+
121
+ * `Kotoshu::Language::LanguageIdentifier`: Automatic language detection
122
+ using FastText LID model (127 languages).
123
+
124
+ * `Kotoshu::Cli::InteractiveReviewer`: Interactive CLI for error review
125
+ with full navigation (forward, backward, jump, skip, accept).
126
+
127
+ * `Kotoshu::Dictionary::Hunspell`: Traditional Hunspell dictionary backend
128
+ for morphological analysis and affix rules.
129
+
130
+ == Why ONNX?
131
+
132
+ ONNX Runtime provides:
133
+
134
+ * **Performance**: C++ implementation, 10-100x faster than pure Ruby
135
+ * **Portability**: Works on CPU, GPU, TPU, mobile devices
136
+ * **Optimization**: Automatic graph optimization and quantization
137
+ * **Interoperability**: Models can be trained in Python, deployed in Ruby
138
+
139
+ Kotoshu uses FastText models converted to ONNX format for semantic spell checking.
140
+
141
+ [[semantic-analysis]]
142
+ == Semantic Analysis
143
+
144
+ Unlike traditional spell checkers that only check dictionary membership and
145
+ edit distance, Kotoshu uses semantic similarity to:
146
+
147
+ * Detect contextually appropriate corrections ("desert" vs "dessert")
148
+ * Handle out-of-vocabulary words via subword embeddings
149
+ * Provide ranked suggestions based on semantic similarity
150
+ * Support compound words and morphological variations
151
+
152
+ .Usage example
153
+ [example]
154
+ ====
155
+ [source,ruby]
156
+ ----
157
+ Kotoshu.setup(:en, want: %i[spelling model]) # one-time per language
158
+
159
+ # Traditional: knows "helo" is wrong and lists edit-distance candidates
160
+ Kotoshu.suggest("helo").to_words
161
+ # => ["hello", "help", "held", "hell", "hole"]
162
+
163
+ # Semantic: reranks candidates by context similarity
164
+ model = Kotoshu::Models::OnnxModel.from_github("en")
165
+ analyzer = Kotoshu::Analyzers::SemanticAnalyzer.new(model)
166
+ analyzer.suggest_corrections("helo", context: "I said helo to the world").map(&:word)
167
+ # => ["hello"] # "hello" makes more sense in greeting context
168
+ ----
169
+ ====
170
+
171
+ NOTE: The semantic path requires the optional `onnxruntime` gem. See
172
+ <<_requirements,Requirements>>.
173
+
174
+ [[confidence-cascade]]
175
+ === Confidence cascade
176
+
177
+ When the traditional (dictionary + edit distance) strategies are already
178
+ confident about a word, the semantic rerank adds latency and model load
179
+ without changing the answer. The confidence cascade formalizes the
180
+ hybrid split: dictionary verdict first, ONNX rerank only for uncertain
181
+ candidates.
182
+
183
+ The `semantic_cascade_threshold` configuration option controls it:
184
+
185
+ [cols="1,3"]
186
+ |===
187
+ |Threshold |Behavior
188
+
189
+ |`1.0` (default)
190
+ |Always rerank — identical to the classic pipeline. This is the sentinel:
191
+ the composite can legitimately score exactly `1.0`, so `1.0` never skips.
192
+
193
+ |`0.9`
194
+ |Rerank only when the top traditional candidate scores below `0.9`;
195
+ otherwise skip the ONNX rerank for that word.
196
+
197
+ |`0.0`
198
+ |Never rerank — the semantic path is bypassed entirely.
199
+ |===
200
+
201
+ [source,bash]
202
+ ----
203
+ # Tune via environment variable
204
+ KOTOSHU_SEMANTIC_CASCADE_THRESHOLD=0.9 kotoshu check document.txt
205
+
206
+ # Or via the Ruby API
207
+ Kotoshu.configuration.semantic_cascade_threshold = 0.9
208
+ ----
209
+
210
+ Each skip is counted in the metrics registry
211
+ (`Kotoshu::Metrics.stats[:semantic_cascade_skips]`, also exported in
212
+ StatsD and Prometheus formats) and logged under debug/verbose output.
213
+
214
+ Tuning guidance will come from the eval harness once the corpus
215
+ benchmark lands; until then, `0.8`–`0.9` is a reasonable range for
216
+ typo-heavy text.
217
+
218
+ [[multi-language]]
219
+ == Multi-Language Support
220
+
221
+ Kotoshu serves every language staged in the dictionaries manifest —
222
+ 36 full-feature codes with per-language modules (script-aware
223
+ tokenizer, national keyboard grid, engine-verified specimens) and the
224
+ script-fallback basic tier for the rest; see the
225
+ <<language-matrix, Language Support Matrix>> for the breakdown.
226
+ Semantic reranking adds an ONNX model where the models registry pins
227
+ one (50+ languages), for example:
228
+
229
+ * *de* - German (Deutsch)
230
+ * *en* - English
231
+ * *es* - Spanish (Español)
232
+ * *fr* - French (Français)
233
+ * *pt* - Portuguese (Português)
234
+ * *ru* - Russian (Русский)
235
+
236
+ Automatic language detection is enabled by default:
237
+
238
+ .Usage example
239
+ [example]
240
+ ====
241
+ [source,bash]
242
+ ----
243
+ # Language auto-detected from document content
244
+ kotoshu check document.txt
245
+ # Detected: en (95% confidence)
246
+ # Analyzing document.txt (language: en)...
247
+
248
+ # Explicit language specification
249
+ kotoshu check document.txt --language de
250
+ ----
251
+ ====
252
+
253
+ [[onnx-models]]
254
+ == ONNX Models
255
+
256
+ Kotoshu uses FastText crawl vectors converted to ONNX format:
257
+
258
+ * Source: https://fasttext.cc/docs/en/crawl-vectors.html[FastText Crawl Vectors]
259
+ * Format: ONNX with optimized runtime
260
+ * Vocabulary: 2 million words per language (full coverage)
261
+ * Dimension: 300-dimensional word vectors
262
+ * Size: ~2.4GB per language
263
+
264
+ === FastText File Formats
265
+
266
+ FastText provides two file formats. Kotoshu uses the `.vec` format for ONNX conversion.
267
+
268
+ [cols="1,1,2"]
269
+ |===
270
+ |Aspect |`.vec` (Text) |`.bin` (Binary)
271
+
272
+ |Content
273
+ |Word vectors only (pre-computed embeddings)
274
+ |Full FastText model (trained model)
275
+
276
+ |Structure
277
+ |Text: one word + 300 floats per line
278
+ |Binary: complete model with matrices
279
+
280
+ |File Size
281
+ |~1.3GB compressed (~2.4GB uncompressed)
282
+ |~1.8GB compressed (~4.8GB uncompressed)
283
+
284
+ |Train New Words
285
+ |✗ No (static lookup only)
286
+ |✓ Yes (can train/OOV with subword info)
287
+
288
+ |Subword Embeddings
289
+ |✗ No
290
+ |✓ Yes (n-gram character embeddings)
291
+
292
+ |ONNX Converter
293
+ |✓ Supported (what we use)
294
+ |✗ Not supported
295
+
296
+ |Use Case
297
+ |Simple word vector lookup for spell checking
298
+ |Full FastText functionality (training, OOV)
299
+ |===
300
+
301
+ Kotoshu uses `.vec` files because:
302
+
303
+ * Simpler extraction: Just word → vector mapping
304
+ * No subword complexity needed: Dictionary-based spell checking doesn't require OOV generation
305
+ * Smaller ONNX models: ~2.4GB vs ~4.8GB
306
+ * Faster conversion: Direct serialization to ONNX
307
+
308
+ .Model management
309
+ [example]
310
+ ====
311
+ [source,bash]
312
+ ----
313
+ # Set up a language with spelling + ONNX semantic model
314
+ kotoshu setup en --want spelling,model
315
+
316
+ # List what's set up in the cache
317
+ kotoshu setup --list
318
+
319
+ # Re-validate cached resources
320
+ kotoshu cache validate
321
+ ----
322
+ ====
323
+
324
+ NOTE: FastText `.vec` → ONNX conversion is done upstream in the
325
+ https://github.com/kotoshu/models-fasttext-onnx[`kotoshu/models-fasttext-onnx']
326
+ repo. The CLI downloads pre-converted artifacts; users do not run
327
+ conversion locally.
328
+
329
+ [[model-tiers]]
330
+ === Model tiers
331
+
332
+ ONNX models come in three tiers, published by
333
+ https://github.com/kotoshu/models-fasttext-onnx[models-fasttext-onnx]
334
+ in a resource registry under stable ids (`kotoshu://models/{lang}/{tier}`):
335
+
336
+ [cols="1,2,3"]
337
+ |===
338
+ |Tier |Approx. size |What it is
339
+
340
+ |`full`
341
+ |~120 MB
342
+ |Unquantized, ~100k-word vocabulary. The same artifact and cache layout
343
+ as before tiers existed.
344
+
345
+ |`fluency`
346
+ |~15-18 MB
347
+ |int8-quantized, reduced vocabulary (~50k-60k words).
348
+
349
+ |`mini`
350
+ |~3 MB
351
+ |int8-quantized, 10k-word vocabulary.
352
+ |===
353
+
354
+ Tiered downloads are verified: the model bytes are fetched from the
355
+ registry entry's primary URL, falling back to its mirror, with the vocab
356
+ sibling downloaded alongside; the SHA-256 is checked against the
357
+ registry entry before the model is accepted into the cache.
358
+
359
+ .Picking a tier
360
+ [example]
361
+ ====
362
+ [source,ruby]
363
+ ----
364
+ # One-time setup at a specific tier
365
+ Kotoshu.setup(:en, want: %i[model], tier: :fluency)
366
+
367
+ # Resolve stays cache-only: the requested tier must already be cached,
368
+ # otherwise ResourceNotSetupError is raised (no download, no fallback
369
+ # to another tier)
370
+ Kotoshu.resolve(language: :en, want: %i[spelling model], tier: :fluency)
371
+
372
+ # "Is this tier cached?" (tier: :any asks whether any tier is cached)
373
+ Kotoshu.setup?(:en, resource: :model, tier: :mini)
374
+ ----
375
+
376
+ [source,bash]
377
+ ----
378
+ kotoshu setup en --model --tier mini
379
+ ----
380
+ ====
381
+
382
+ `tier: :any` on resolve maps to the *single* cached tier. It resolves
383
+ when exactly one tier is cached and raises otherwise: with no tier
384
+ cached it raises `ResourceNotSetupError`, and with several it raises
385
+ `Kotoshu::ResourceResolutionError` listing them. That listing uses the
386
+ fixed order `mini`, `fluency`, `full` — a reporting order for
387
+ deterministic error messages, not a fallback chain; tiers are never
388
+ silently substituted for one another.
389
+
390
+ The default tier is `fluency` (~15 MB, near-lossless; owner decision
391
+ 2026-09-04) — caches written under the old `full` default keep working
392
+ via the tier-less legacy bridge. Override it per call with `tier:`, or set
393
+ it globally via the `model_tier` configuration option (environment
394
+ variable `KOTOSHU_MODEL_TIER`; one of `full`, `fluency`, `mini`):
395
+
396
+ [source,bash]
397
+ ----
398
+ KOTOSHU_MODEL_TIER=mini kotoshu setup en --model
399
+ ----
400
+
401
+ [[interactive-mode]]
402
+ [[interactive-mode]]
403
+ == Interactive Mode
404
+
405
+ NOTE: Interactive mode shipped in 0.3.0. It is navigation-only — the
406
+ session records which suggestions the user accepted but does not rewrite
407
+ the source file yet.
408
+
409
+ [source,bash]
410
+ ----
411
+ kotoshu check README.md --interactive
412
+ ----
413
+
414
+ Features in 0.3:
415
+
416
+ * **Navigate**: [n] / Enter next, [p] previous, [l] list
417
+ * **Accept**: [1-9] record suggestion N for the current error
418
+ * **Skip**: [s] skip the current error
419
+ * **Quit**: [q] exit the review loop
420
+
421
+ [[batch-processing]]
422
+ == Batch Processing
423
+
424
+ For CI/CD and automation, Kotoshu supports JSON and SARIF output in 0.3;
425
+ `--output` file redirection is planned for 0.4+.
426
+
427
+ .JSON output for CI/CD
428
+ [example]
429
+ ====
430
+ [source,bash]
431
+ ----
432
+ # JSON output to stdout (supported in 0.3)
433
+ kotoshu check README.md --format json
434
+
435
+ # SARIF 2.1.0 output (supported in 0.3)
436
+ kotoshu check README.md --format sarif
437
+
438
+ # Exit code for CI
439
+ kotoshu check README.md
440
+ echo $? # 0 if no errors, 1 if errors found
441
+ ----
442
+ ====
443
+
444
+ [[ignoring-words]]
445
+ == Ignoring Words Inline
446
+ Words that are intentionally "wrong" - product names, code identifiers
447
+ quoted in prose, dialect - can be suppressed in place instead of being
448
+ added to the personal dictionary:
449
+ ----
450
+ kotoshu:disable-line suppress everything on this line
451
+ kotoshu:disable-next-line suppress the following line
452
+ kotoshu:disable-next-line foo bar suppress only these words
453
+ kotoshu:disable-file / enable-file block suppression (nestable)
454
+ ----
455
+ Each format recognizes the directive in its own comment syntax, and the
456
+ directive line itself is never spellchecked:
457
+ [options="header"]
458
+ |===
459
+ |Format |Form |Trailing directive
460
+
461
+ |Plain text
462
+ |a bare line: `kotoshu:disable-line`
463
+ |not supported
464
+
465
+ |Markdown
466
+ |`<!-- kotoshu:disable-next-line -->`
467
+ |supported
468
+
469
+ |AsciiDoc
470
+ |`// kotoshu:disable-next-line`
471
+ |supported
472
+
473
+ |===
474
+
475
+ The Ruby API picks this up automatically:
476
+ `Kotoshu.spellchecker_for("en").check(text)` moves suppressed words to
477
+ `result.suppressed_errors` (each marked `suppressed: true` and
478
+ `suppressed_by: "inline"`) so only real errors stay in `result.errors`.
479
+ Pass `--show-suppressed` to `kotoshu check` to list them.
480
+
481
+ [[ci-baselines]]
482
+ == CI Baselines
483
+ Freeze existing spelling debt so CI fails only on NEW errors:
484
+ [source,bash]
485
+ ----
486
+ # Record current findings: one entry per file+word with a count,
487
+ # stable across reformatting
488
+ kotoshu baseline init README.md docs/*.adoc
489
+
490
+ # CI: errors covered by the baseline pass, new errors fail
491
+ kotoshu check README.md --baseline .kotoshu-baseline.json
492
+ ----
493
+ A summary line reports baseline entries that have gone stale (the error
494
+ no longer exists), so the debt shrinks visibly; refresh by re-running
495
+ `kotoshu baseline init`. JSON output adds `suppressedCount` /
496
+ `suppressedErrors` plus a `baseline` block, and SARIF marks
497
+ baseline-suppressed results with a `suppressions` entry
498
+ (`justification: "baseline"`).
499
+
500
+ [[pre-commit]]
501
+ == Git pre-commit Hook
502
+ The gem repository ships a https://pre-commit.com[pre-commit] hook
503
+ definition (`.pre-commit-hooks.yaml`):
504
+ [source,yaml]
505
+ ----
506
+ repos:
507
+ - repo: https://github.com/kotoshu/kotoshu
508
+ rev: v0.7.0 # pin to a released tag
509
+ hooks:
510
+ - id: kotoshu
511
+ ----
512
+ NOTE: The hook uses `language: system`: it runs the `kotoshu` executable
513
+ from PATH, so it honestly requires a Ruby environment with the gem
514
+ installed (`gem install kotoshu`, then `kotoshu setup en`). A
515
+ WebAssembly client (in development) is the future zero-Ruby path.
516
+
517
+ [[document-formats]]
518
+ == Document Formats
519
+
520
+ Kotoshu supports structured documents with AST parsing:
521
+
522
+ * *Plain text*: Line-based error detection
523
+ * *Markdown*: AST-based using Kramdown parser
524
+ * *AsciiDoc*: AST-based using Asciidoctor parser
525
+
526
+ Structured documents preserve node paths for precise error location.
527
+
528
+ [[multiple-models]]
529
+ == Analysis Models
530
+
531
+ NOTE: In 0.2, the CLI runs the Hunspell traditional path only.
532
+ The `--model` flag and FastText/Hybrid paths are planned for 0.3+.
533
+ The Ruby API can opt into the semantic path today via
534
+ `Kotoshu::Models::OnnxModel` (auto-available when `onnxruntime` is installed).
535
+
536
+ Kotoshu is designed to support three analysis models:
537
+
538
+ .Dictionary backend comparison
539
+ [cols="2,2,2"]
540
+ |===
541
+ | Model | Description | Best For
542
+
543
+ | *hunspell* | Traditional dictionary-based with morphological rules | Fast checking, compound words, languages with complex morphology
544
+
545
+ | *fasttext* | Pure semantic embeddings via ONNX | Context awareness, out-of-vocabulary words, semantic similarity
546
+
547
+ | *hybrid* | Hunspell candidates + FastText reranking (recommended) | Maximum accuracy, best of both worlds
548
+ |===
549
+
550
+ .Intended usage (0.3+)
551
+ [example]
552
+ ====
553
+ [source,bash]
554
+ ----
555
+ # Fast dictionary-based checking (default in 0.2)
556
+ kotoshu check document.txt # 0.2: Hunspell path
557
+
558
+ # Semantic / hybrid paths: planned for 0.3
559
+ # kotoshu check document.txt --model fasttext
560
+ # kotoshu check document.txt --model hybrid
561
+ ----
562
+ ====
563
+
564
+ == Installation
565
+
566
+ Add this line to your application's Gemfile:
567
+
568
+ [source,ruby]
569
+ ----
570
+ gem 'kotoshu'
571
+ ----
572
+
573
+ And then execute:
574
+
575
+ [source,sh]
576
+ ----
577
+ bundle install
578
+ ----
579
+
580
+ Or install it yourself as:
581
+
582
+ [source,sh]
583
+ ----
584
+ gem install kotoshu
585
+ ----
586
+
587
+ NOTE: `onnxruntime` is an optional dependency. Install it separately
588
+ (`gem install onnxruntime`) to enable semantic analysis; the
589
+ traditional Hunspell path works without it.
590
+
591
+ == Quick Start
592
+
593
+ [source,sh]
594
+ ----
595
+ # One-time per language: download spelling dictionary from
596
+ # github.com/kotoshu/dictionaries (idempotent, ~5 MB)
597
+ kotoshu setup en
598
+
599
+ # Then check files instantly, cache-only
600
+ kotoshu check README.md
601
+ ----
602
+
603
+ Or skip the explicit setup — the CLI will prompt interactively the
604
+ first time you check a file in a non-cached language (TTY only; in
605
+ non-TTY or `KOTOSHU_OFFLINE=1` mode it exits with code 3).
606
+
607
+ .Command-line usage
608
+ [source,bash]
609
+ ----
610
+ # Check a file (uses --language, or auto-detects from content)
611
+ kotoshu check README.md
612
+
613
+ # Explicit language
614
+ kotoshu check README.md --language en
615
+
616
+ # JSON output for programmatic use
617
+ kotoshu check README.md --format json
618
+
619
+ # Offline mode — use only cached dictionaries, never download
620
+ kotoshu check README.md --offline
621
+
622
+ # Check stdin
623
+ echo "helo wrld" | kotoshu check
624
+ ----
625
+
626
+ Exit codes: `0` (no errors), `1` (errors found), `2` (usage error),
627
+ `3` (language not set up — run `kotoshu setup LANG`, or run `kotoshu check`
628
+ in a TTY to be prompted).
629
+
630
+ .Ruby API usage
631
+ [source,ruby]
632
+ ----
633
+ require 'kotoshu'
634
+
635
+ # Stage 1: set up the language once (downloads from github.com/kotoshu/dictionaries)
636
+ Kotoshu.setup(:en)
637
+
638
+ # Stage 2: hot-path checks are cache-only and never touch the network
639
+ Kotoshu.correct?("hello") # => true
640
+ Kotoshu.correct?("helo") # => false
641
+
642
+ # Suggestions return a SuggestionSet; call #to_words for an Array
643
+ Kotoshu.suggest("helo").to_words # => ["hello", "help", "held", ...]
644
+
645
+ # Check a document
646
+ result = Kotoshu.check("Hello wrold")
647
+ result.errors.map(&:word) # => ["wrold"]
648
+
649
+ # Each error carries position + suggestions
650
+ result = Kotoshu.check_file("README.md")
651
+ result.errors.each do |error|
652
+ puts "#{error.word} at offset #{error.position}: #{error.top_suggestions(3).join(', ')}"
653
+ end
654
+
655
+ # Semantic analysis is optional — requires the onnxruntime gem
656
+ # (gem install onnxruntime). Skip this block if you only want Hunspell.
657
+ if Kotoshu::Models::OnnxModel::ONNX_LOADED
658
+ Kotoshu.setup(:en, want: %i[spelling model])
659
+ model = Kotoshu::Models::OnnxModel.from_github('en')
660
+ analyzer = Kotoshu::Analyzers::SemanticAnalyzer.new(model)
661
+ analyzer.analyze(Kotoshu.check("Hello wrold"))
662
+ end
663
+ ----
664
+
665
+ NOTE: The library API is strict: calls like `Kotoshu.correct?` raise
666
+ `Kotoshu::ResourceNotSetupError` until you've run `Kotoshu.setup`. This
667
+ prevents surprise downloads on metered networks. The CLI (`kotoshu check`)
668
+ intercepts the error and prompts to download interactively.
669
+
670
+ == Requirements
671
+
672
+ * Ruby 3.1+
673
+ * `onnxruntime` gem (optional — enables semantic spell checking; install separately with `gem install onnxruntime`)
674
+ * Python 3 + fasttext (optional, only if you want to convert `.vec` → `.onnx` upstream)
675
+
676
+ == Resource Caching and Language Support
677
+
678
+ Kotoshu uses a sophisticated multi-layer caching system to manage dictionaries,
679
+ frequency lists, and embedding models. Resources are downloaded explicitly via
680
+ `Kotoshu.setup` (or `kotoshu setup`) and cached under the XDG base directory
681
+ layout (`~/.cache/kotoshu/` by default; override via `KOTOSHU_CACHE_PATH`,
682
+ `KOTOSHU_CONFIG_PATH`, `KOTOSHU_DATA_PATH`, or the `XDG_*_HOME` vars).
683
+
684
+ === Cache Architecture
685
+
686
+ .Cache System Class Diagram
687
+ [source]
688
+ ----
689
+ ┌────────────────────────────────────────────────────────────────────────────┐
690
+ │ BaseCache (Abstract) │
691
+ │ ┌────────────────────────────────────────────────────────────────────┐ │
692
+ │ │ Common: download, metadata, validation, stats, TTL management │ │
693
+ │ └────────────────────────────────────────────────────────────────────┘ │
694
+ └────────────────────┬───────────────────┬────────────────────┬──────────────┘
695
+ │ │ │
696
+ ┌────────────▼────────┐ ┌──────▼──────┐ ┌───────▼─────────┐
697
+ │ LanguageCache │ │ModelCache │ │ FrequencyCache │
698
+ │ (Dictionaries) │ │ (Embeddings)│ │ (Kelly Lists) │
699
+ └─────────────────────┘ └─────────────┘ └─────────────────┘
700
+ │ │ │
701
+ ┌────────────▼────────┐ ┌──────▼──────┐ ┌───────▼─────────┐
702
+ │ ~/.cache/kotoshu/ │ │~/.cache/ │ │ ~/.cache/kotoshu/│
703
+ │ languages/ │ │ kotoshu/ │ │frequency-lists/ │
704
+ │ │ │ models/ │ │ │
705
+ └─────────────────────┘ └─────────────┘ └─────────────────┘
706
+ ----
707
+
708
+ === Cache Types
709
+
710
+ ==== LanguageCache (Dictionaries)
711
+
712
+ Manages Hunspell dictionaries and grammar rules for spell checking.
713
+
714
+ * **Cache Path**: `~/.cache/kotoshu/languages/{code}/`
715
+ * **TTL**: 7 days (604,800 seconds)
716
+ * **Source**: https://github.com/kotoshu/dictionaries[kotoshu/dictionaries]
717
+ * **Resources per language**:
718
+ ** `spelling/`**: Hunspell dictionary (`index.dic`, `index.aff`)
719
+ ** `grammar/`**: Grammar rules (`rules.yaml`) - *future*
720
+ ** `frequency/`**: Frequency data - *deprecated, use FrequencyCache*
721
+
722
+ .Usage
723
+ [source,ruby]
724
+ ----
725
+ # Access via cache
726
+ cache = Kotoshu::Cache::LanguageCache.new
727
+ dict = cache.get_spelling('en')
728
+
729
+ # Result:
730
+ # {
731
+ # aff_path: "~/.cache/kotoshu/languages/en/spelling/index.aff",
732
+ # dic_path: "~/.cache/kotoshu/languages/en/spelling/index.dic",
733
+ # cached: true,
734
+ # metadata: { ... }
735
+ # }
736
+ ----
737
+
738
+ ==== FrequencyCache (Kelly Project)
739
+
740
+ Manages Kelly Project frequency lists for intelligent suggestion ranking.
741
+
742
+ * **Cache Path**: `~/.cache/kotoshu/frequency-lists/{code}/`
743
+ * **TTL**: 7 days (604,800 seconds)
744
+ * **Source**: https://github.com/kotoshu/frequency-list-kelly[kotoshu/frequency-list-kelly]
745
+ * **Format**: JSON with tiered word frequency data
746
+
747
+ .Kelly Frequency Data Structure
748
+ [source,json]
749
+ ----
750
+ {
751
+ "metadata": {
752
+ "language": "en",
753
+ "source": "Kelly Project (University of Leeds)",
754
+ "total_words_analyzed": 1500000
755
+ },
756
+ "tiers": {
757
+ "top_50": {
758
+ "words": ["the", "be", "to", "of", "and", ...],
759
+ "info": "Most common 50 words"
760
+ },
761
+ "top_200": {
762
+ "words": ["will", "my", "one", "all", ...],
763
+ "info": "Most common 200 words"
764
+ },
765
+ "top_1000": {
766
+ "words": ["however", "although", ...],
767
+ "info": "Most common 1000 words"
768
+ }
769
+ }
770
+ }
771
+ ----
772
+
773
+ .Usage
774
+ [source,ruby]
775
+ ----
776
+ # Access via cache
777
+ cache = Kotoshu::Cache::FrequencyCache.new
778
+ freq_data = cache.get('en', force_download: true)
779
+
780
+ # Result:
781
+ # {
782
+ # frequency_path: "~/.cache/kotoshu/frequency-lists/en/frequency.json",
783
+ # tiers: {
784
+ # top_50: Set<...>,
785
+ # top_200: Set<...>,
786
+ # top_1000: Set<...>
787
+ # },
788
+ # metadata: { ... }
789
+ # }
790
+
791
+ # Integrated into EditDistanceStrategy
792
+ strategy = Kotoshu::Suggestions::Strategies::EditDistanceStrategy.new(
793
+ language_code: 'en'
794
+ )
795
+ strategy.frequency_bonus('the') # => 200 (top 50)
796
+ strategy.frequency_bonus('hello') # => 100 (top 200)
797
+ strategy.frequency_bonus('xyz') # => 0 (not in lists)
798
+ ----
799
+
800
+ ==== ModelCache (Embedding Models)
801
+
802
+ Manages FastText and ONNX embedding models for semantic spell checking.
803
+
804
+ * **Cache Path**: `~/.cache/kotoshu/models/{code}/models/{type}/`
805
+ * **TTL**: 30 days (2,592,000 seconds)
806
+ * **Sources**:
807
+ ** FastText (.vec): Facebook CDN (https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/[dl.fbaipublicfiles.com])
808
+ ** ONNX (.onnx): Converted locally from FastText models
809
+ * **Supported Types**:
810
+ ** `fasttext`: FastText word vectors (.vec files, 300D) - Downloaded from Facebook CDN
811
+ ** `onnx`: ONNX-converted models (.onnx files) - Auto-converted from FastText
812
+
813
+ NOTE: ONNX models are automatically converted from FastText models on first use.
814
+ The conversion uses `lib/kotoshu/scripts/fasttext_to_onnx.py` and requires Python 3 with
815
+ `numpy` and `onnx` packages installed.
816
+
817
+ .Model Files by Language
818
+ [width="100%",cols="3,m"]
819
+ |=========================================
820
+ | Language | FastText File | ONNX File |
821
+ |=========================================
822
+ | de (German) | cc.de.300.vec | fasttext.de.onnx |
823
+ | en (English) | cc.en.300.vec | fasttext.en.onnx |
824
+ | es (Spanish) | cc.es.300.vec | fasttext.es.onnx |
825
+ | fr (French) | cc.fr.300.vec | fasttext.fr.onnx |
826
+ | pt (Portuguese) | cc.pt.300.vec | fasttext.pt.onnx |
827
+ | ru (Russian) | cc.ru.300.vec | fasttext.ru.onnx |
828
+ |=========================================|
829
+
830
+ === CLI Cache Management
831
+
832
+ Kotoshu provides CLI commands for managing cached resources:
833
+
834
+ [source,bash]
835
+ ----
836
+ # List all cached resources
837
+ kotoshu cache list
838
+
839
+ # List specific cache type
840
+ kotoshu cache list language
841
+ kotoshu cache list model
842
+ kotoshu cache list frequency
843
+
844
+ # Show cache statistics
845
+ kotoshu cache status
846
+
847
+ # Show detailed status (verbose)
848
+ kotoshu cache status --verbose
849
+
850
+ # Download a resource
851
+ kotoshu cache download language en
852
+ kotoshu cache download model en:fasttext
853
+ kotoshu cache download frequency en
854
+
855
+ # Get information about a resource
856
+ kotoshu cache info language en
857
+ kotoshu cache info model en:fasttext
858
+ kotoshu cache info frequency en
859
+
860
+ # Purge cached data
861
+ kotoshu cache purge all
862
+ kotoshu cache purge language en
863
+ kotoshu cache purge frequency
864
+
865
+ # Clean expired entries
866
+ kotoshu cache clean
867
+ ----
868
+
869
+ === Cache Statistics
870
+
871
+ Each cache type tracks statistics:
872
+
873
+ * **Hits**: Number of cache hits (resource found locally)
874
+ * **Misses**: Number of cache misses (had to download)
875
+ * **Hit Rate**: Percentage of cache hits
876
+ * **Size**: Total disk space used
877
+ * **Cached Resources**: Number of resources cached
878
+
879
+ [source,bash]
880
+ ----
881
+ $ kotoshu cache status
882
+ ======================================================================
883
+ Kotoshu Cache Status
884
+ ======================================================================
885
+
886
+ Language Cache:
887
+ Directory: /Users/username/.cache/kotoshu/languages
888
+ Resources cached: 2
889
+ Size: 2.45 MB
890
+ Hits: 15, Misses: 2
891
+ Hit rate: 88.2%
892
+
893
+ Frequency Cache:
894
+ Directory: /Users/username/.cache/kotoshu/frequency-lists
895
+ Resources cached: 1
896
+ Size: 815.84 KB
897
+ Hits: 42, Misses: 1
898
+ Hit rate: 97.7%
899
+
900
+ Model Cache:
901
+ Directory: /Users/username/.cache/kotoshu/models
902
+ Resources cached: 0
903
+ Size: 0 B
904
+ Hits: 0, Misses: 0
905
+ Hit rate: 0.0%
906
+
907
+ Total:
908
+ Total size: 3.26 MB
909
+ Overall hit rate: 93.5%
910
+ ======================================================================
911
+ ----
912
+
913
+ === Shell Completion
914
+
915
+ Kotoshu ships tab-completion scripts for bash, zsh, and fish. The scripts
916
+ complete subcommands for `kotoshu <TAB>` and language codes for
917
+ `kotoshu setup <TAB>` (the language list is read dynamically via
918
+ `kotoshu completions languages`, so newly registered languages appear
919
+ without re-running the install step).
920
+
921
+ Install for your shell:
922
+
923
+ [source,bash]
924
+ ----
925
+ # bash (system-wide)
926
+ kotoshu completions bash > /etc/bash_completion.d/kotoshu
927
+
928
+ # bash (single user)
929
+ kotoshu completions bash > ~/.local/share/bash-completion/completions/kotoshu
930
+
931
+ # zsh — drop into any directory on your $fpath
932
+ kotoshu completions zsh > "${fpath[1]}/_kotoshu"
933
+
934
+ # fish
935
+ kotoshu completions fish > ~/.config/fish/completions/kotoshu.fish
936
+ ----
937
+
938
+ To see the supported language codes without installing a completion
939
+ script:
940
+
941
+ [source,bash]
942
+ ----
943
+ kotoshu completions languages
944
+ ----
945
+
946
+ [[language-matrix]]
947
+ === Language Support Matrix
948
+
949
+ Kotoshu serves languages in tiers. Every language staged in the
950
+ kotoshu/dictionaries manifest downloads through `kotoshu setup` and
951
+ checks through the Hunspell engine. Region variants fold onto their
952
+ base dictionary (`en-GB` resolves `en`, `es-MX` resolves `es`);
953
+ script variants keep a dictionary of their own (`sr-Latn` stages the
954
+ Latin Serbian dictionary alongside the Cyrillic `sr`).
955
+
956
+ .Language Support Matrix
957
+ [width="100%",cols="^2,^3,^3",options="header"]
958
+ |=========================================
959
+ | Tier | Languages | What it carries |
960
+ |=========================================
961
+ | Full feature (35 languages, 36 codes) | ar bg ca cs da de el en es et fa fr he hr hu id it ko lt lv nb ne nl nn pl pt ro ru sk sl sr sr-Latn sv tr uk vi | Per-language module: staged dictionary + Hunspell affix rules, script-aware tokenizer, national keyboard grid for typo-proximity suggestions, engine-verified specimens |
962
+ | Basic (26 languages) | br cy eo eu fo fur fy ga gd gl hy hyw ia ie is ka la lb ltg mk mn nds oc rw tk tlh | Staged dictionary + script-fallback word extraction and the generic keyboard |
963
+ | Kelly frequency (8 languages) | ar el en it no ru sv zh | Top-1000 frequency list wired into suggestion ranking, on top of either tier |
964
+ |=========================================+
965
+
966
+ ONNX semantic models (opt-in, `--model`) resolve per the registry the
967
+ link:https://github.com/kotoshu/models-fasttext-onnx[models repo]
968
+ pins — 50+ languages across the tiers, in full/fluency/mini sizes.
969
+
970
+ .Dictionary Sources
971
+ [width="100%",cols="^1,^1,^1,^1",options="header"]
972
+ |=========================================
973
+ | Language | Word Count | License | Source |
974
+ |=========================================
975
+ | de (German) | 75,873 | GPL | igerman98 |
976
+ | en (English) | 49,568 | LGPL/MPL/GPL | SCOWL |
977
+ | es (Spanish) | 57,344 | GPL | LibreOffice |
978
+ | fr (French) | 84,310 | MPL 2.0 | Grammalecte |
979
+ | pt (Portuguese) | 312,368 | LGPLv3 + MPL | VERO |
980
+ | ru (Russian) | 146,269 | BSD-style | Alexander Lebedev |
981
+ |=========================================+
982
+
983
+ Staged dictionaries for the other tiers carry their provenance in the
984
+ dictionaries repo manifest (`LICENSE` and source per file; LibreOffice
985
+ / wooorm-dictionaries derivations dominate).
986
+
987
+ .Kelly Frequency Lists
988
+ [width="100%",cols="^1,^1,^1",options="header"]
989
+ |=========================================
990
+ | Language | Size | Coverage |
991
+ |=========================================
992
+ | ar (Arabic) | ~750 KB | Top 1000 words |
993
+ | zh (Chinese) | ~800 KB | Top 1000 words |
994
+ | en (English) | 815 KB | Top 1000 words |
995
+ | el (Greek) | ~780 KB | Top 1000 words |
996
+ | it (Italian) | ~790 KB | Top 1000 words |
997
+ | no (Norwegian) | ~770 KB | Top 1000 words |
998
+ | ru (Russian) | 780 KB | Top 1000 words |
999
+ | sv (Swedish) | ~775 KB | Top 1000 words |
1000
+ |=========================================+
1001
+
1002
+ NOTE: Kelly frequency lists provide the top 1000 most common words from
1003
+ the Kelly Project (University of Leeds & University of Gothenburg).
1004
+ Languages not listed here require external frequency data sources.
1005
+
1006
+ === Programmatic Usage
1007
+
1008
+ ==== Using Language Cache
1009
+
1010
+ [source,ruby]
1011
+ ----
1012
+ require 'kotoshu/cache/language_cache'
1013
+
1014
+ cache = Kotoshu::Cache::LanguageCache.new
1015
+
1016
+ # Get spelling dictionary
1017
+ dict = cache.get_spelling('en')
1018
+ puts "Dictionary: #{dict[:dic_path]}"
1019
+ puts "Words: #{File.readlines(dict[:dic_path]).count}"
1020
+
1021
+ # Get available languages
1022
+ cache.available_languages # => ["de", "en", "es", "fr", "pt", "ru"]
1023
+
1024
+ # Check if resource is cached
1025
+ cache.available?('en:spelling') # => true
1026
+
1027
+ # Get language info
1028
+ info = cache.language_info('en')
1029
+ puts "Language: #{info[:name]}"
1030
+ puts "Words: #{info[:word_count]}"
1031
+ puts "License: #{info[:license]}"
1032
+ ----
1033
+
1034
+ ==== Using Frequency Cache
1035
+
1036
+ [source,ruby]
1037
+ ----
1038
+ require 'kotoshu/cache/frequency_cache'
1039
+
1040
+ cache = Kotoshu::Cache::FrequencyCache.new
1041
+
1042
+ # Get frequency data
1043
+ freq_data = cache.get('en')
1044
+
1045
+ # Access frequency tiers
1046
+ top_50 = freq_data[:tiers][:top_50]
1047
+ top_50.include?('the') # => true
1048
+ top_50.include?('hello') # => true (in top 200)
1049
+
1050
+ # Get available languages
1051
+ cache.available_languages # => ["ar", "zh", "en", "el", "it", "no", "ru", "sv"]
1052
+ ----
1053
+
1054
+ ==== Integration with Suggestion Strategies
1055
+
1056
+ [source,ruby]
1057
+ ----
1058
+ require 'kotoshu/suggestions/strategies/edit_distance_strategy'
1059
+
1060
+ # Frequency bonuses automatically applied
1061
+ strategy = Kotoshu::Suggestions::Strategies::EditDistanceStrategy.new(
1062
+ language_code: 'en'
1063
+ )
1064
+
1065
+ # Suggestions are ranked by frequency
1066
+ suggestions = strategy.suggest('helo', max_results: 5)
1067
+ # => [
1068
+ # { word: "hello", score: 1200 }, # High frequency word
1069
+ # { word: "help", score: 1150 }, # Medium frequency word
1070
+ # ...
1071
+ # ]
1072
+ ----
1073
+
1074
+ === Cache TTL and Expiration
1075
+
1076
+ All cached resources have a Time-To-Live (TTL) and automatically expire:
1077
+
1078
+ * **LanguageCache**: 7 days (dictionaries change infrequently)
1079
+ * **FrequencyCache**: 7 days (frequency lists are stable)
1080
+ * **ModelCache**: 30 days (models are large and change rarely)
1081
+
1082
+ Expired resources are automatically re-downloaded on next access.
1083
+
1084
+ [source,ruby]
1085
+ ----
1086
+ cache = Kotoshu::Cache::FrequencyCache.new
1087
+
1088
+ # Force re-download (ignores cache)
1089
+ freq_data = cache.get('en', force_download: true)
1090
+
1091
+ # Clean expired entries manually
1092
+ cache.clean
1093
+ ----
1094
+
1095
+ === Manual Cache Management
1096
+
1097
+ [source,ruby]
1098
+ ----
1099
+ cache = Kotoshu::Cache::LanguageCache.new
1100
+
1101
+ # Clear specific resource
1102
+ cache.clear('en:spelling')
1103
+
1104
+ # Clear all resources
1105
+ cache.clear_all
1106
+
1107
+ # Check if resource exists
1108
+ cache.available?('en:spelling') # => true after download
1109
+
1110
+ # Get statistics
1111
+ stats = cache.stats
1112
+ puts "Hit rate: #{stats[:hit_rate] * 100}%"
1113
+ puts "Size: #{stats[:size_bytes]} bytes"
1114
+ ----
1115
+
1116
+ === GitHub Repository Structure
1117
+
1118
+ The kotoshu/dictionaries repository follows this structure:
1119
+
1120
+ [source]
1121
+ ----
1122
+ kotoshu/dictionaries/
1123
+ ├── en/
1124
+ │ ├── spelling/
1125
+ │ │ ├── index.dic # Hunspell dictionary
1126
+ │ │ ├── index.aff # Hunspell affix rules
1127
+ │ │ └── metadata.json # Version info
1128
+ │ ├── grammar/
1129
+ │ │ └── rules.yaml # Grammar rules (future)
1130
+ │ └── models/
1131
+ │ ├── fasttext/
1132
+ │ │ └── cc.en.300.vec # FastText vectors
1133
+ │ └── onnx/
1134
+ │ └── fasttext.en.onnx # ONNX model
1135
+ ├── de/
1136
+ │ └── ... (same structure)
1137
+ └── README.md
1138
+
1139
+ kotoshu/frequency-list-kelly/
1140
+ ├── data/
1141
+ │ ├── en.json # Kelly frequency data
1142
+ │ ├── ru.json
1143
+ │ └── ...
1144
+ └── README.md
1145
+ ----
1146
+
1147
+ === Adding New Languages
1148
+
1149
+ To add support for a new language:
1150
+
1151
+ 1. **Dictionary**: Add Hunspell dictionary to `kotoshu/dictionaries/{code}/spelling/`
1152
+ 2. **Frequency**: Add Kelly frequency data to `kotoshu/frequency-list-kelly/data/{code}.json`
1153
+ 3. **Register**: Add to `AVAILABLE_LANGUAGES` in `LanguageCache`
1154
+ 4. **Test**: Run integration tests to verify
1155
+
1156
+ See CONTRIBUTING.adoc for detailed guidelines.
1157
+
1158
+ == Model Repository
1159
+
1160
+ ONNX models are hosted at: https://github.com/kotoshu/dictionaries[kotoshu/dictionaries]
1161
+
1162
+ Download and setup:
1163
+
1164
+ [source,bash]
1165
+ ----
1166
+ # Preferred: let kotoshu fetch and verify the model
1167
+ kotoshu setup en --want spelling,model
1168
+
1169
+ # Manual clone (advanced; bypasses manifest verification)
1170
+ git clone https://github.com/kotoshu/dictionaries.git ~/src/kotoshu/dictionaries
1171
+ ----
1172
+
1173
+ == Writing a Document Plugin
1174
+
1175
+ Kotoshu ships only `Kotoshu::Documents::PlainTextDocument` for the
1176
+ plain-text case. Format-specific parsers (Markdown, AsciiDoc,
1177
+ reStructuredText, LaTeX, …) live in plugins — see the
1178
+ `kotoshu-document-plugin-boundary` design note. Kotoshu auto-discovers
1179
+ plugin files via `Gem.find_files("kotoshu_plugin/document/*.rb")` — install
1180
+ the gem and the parser registers itself on first use.
1181
+
1182
+ A plugin has three pieces:
1183
+
1184
+ 1. *Parser class* — responds to
1185
+ `.from_string(text, language_code:)` (and optionally
1186
+ `.from_file(path, language_code:)`). Returns a
1187
+ `Kotoshu::Documents::Document` whose `text_nodes` carry proper
1188
+ `Documents::SourceRange`s.
1189
+ 2. *Plugin file* — placed at
1190
+ `lib/kotoshu_plugin/document/<format>.rb` in the gem. The file body
1191
+ calls `Kotoshu::Documents.register(format, ParserClass)` at load
1192
+ time.
1193
+ 3. *Gem dependency* — `kotoshu` (this gem) and whatever
1194
+ format-parsing library the parser wraps (e.g. `kramdown` for
1195
+ Markdown, `asciidoctor` for AsciiDoc).
1196
+
1197
+ Example skeleton (`my_markdown_parser` gem):
1198
+
1199
+ [source,ruby]
1200
+ ----
1201
+ # lib/my_markdown_parser/document_parser.rb
1202
+ module MyMarkdownParser
1203
+ class DocumentParser
1204
+ # Build one or more TextNodes from the Markdown AST, each
1205
+ # carrying a SourceRange that points at the original source.
1206
+ def self.from_string(text, language_code: nil)
1207
+ # ... walk the AST, collect (text, source_offset) per text run,
1208
+ # build TextNodes with SourceRanges ...
1209
+ Kotoshu::Documents::Document.new(
1210
+ text_nodes: nodes,
1211
+ source: text,
1212
+ format: :markdown,
1213
+ language_code: language_code
1214
+ )
1215
+ end
1216
+ end
1217
+ end
1218
+ ----
1219
+
1220
+ [source,ruby]
1221
+ ----
1222
+ # lib/kotoshu_plugin/document/markdown.rb
1223
+ require "kotoshu"
1224
+ require "my_markdown_parser/document_parser"
1225
+ Kotoshu::Documents.register(:markdown, MyMarkdownParser::DocumentParser)
1226
+ ----
1227
+
1228
+ After installing the gem, callers get Markdown parsing for free:
1229
+
1230
+ [source,ruby]
1231
+ ----
1232
+ require "kotoshu"
1233
+
1234
+ doc = Kotoshu::Documents.parse(source, format: :markdown,
1235
+ language_code: "en")
1236
+ errors = Kotoshu::Spellchecker.new(
1237
+ resource_bundle: Kotoshu::ResourceManager.resolve(language: "en")
1238
+ ).check(doc.flattened_text)
1239
+ errors.each do |error|
1240
+ puts "#{error.source_range}: #{error.original} → #{error.recommended_suggestion}"
1241
+ end
1242
+ ----
1243
+
1244
+ The structure-aware contract: the analyzer scans
1245
+ `doc.flattened_text` (concatenation of every node's `text`), but every
1246
+ error carries a `source_range` from the original source. An editor can
1247
+ use that range to highlight the user's actual markup, not the
1248
+ stripped text. For example, on the source
1249
+
1250
+ I'm an **friend** of Tom
1251
+
1252
+ a grammar error "an friend" → "a friend" carries a source_range
1253
+ pointing at the original `an **friend**` range, not the flattened
1254
+ "an friend" range.
1255
+
1256
+ Plugin gems should publish to RubyGems with a runtime dependency on
1257
+ `kotoshu >= 0.5.0`. Format-specific parsers are the plugin's
1258
+ responsibility — kotoshu never owns document parsing.
1259
+
1260
+ == API Documentation
1261
+
1262
+ The YARD API reference for the Ruby API builds in CI on every change
1263
+ to `lib/` (the `docs` workflow); the generated HTML is attached to each
1264
+ run as the `yard-docs` artifact. A hosted copy is pending — it will
1265
+ live either on this repository's GitHub Pages or as an API subpath on
1266
+ https://kotoshu.github.io[kotoshu.github.io].
1267
+
1268
+ To build and browse it locally:
1269
+
1270
+ [source,bash]
1271
+ ----
1272
+ bundle install
1273
+ bundle exec yard doc
1274
+ open doc/index.html
1275
+ ----
1276
+
1277
+ == License
1278
+
1279
+ BSD 2-Clause — see the link:LICENSE[LICENSE] file for details.
1280
+
1281
+ Bundled dictionaries and frequency lists carry their own licenses; see the
1282
+ per-language `license` files in https://github.com/kotoshu/dictionaries[kotoshu/dictionaries].
1283
+
1284
+ == Contributing
1285
+
1286
+ Contributions are welcome! Please see CONTRIBUTING.adoc for guidelines.
1287
+
1288
+ == Acknowledgments
1289
+
1290
+ * FastText: https://fasttext.cc/[Facebook Research]
1291
+ * ONNX Runtime: https://onnxruntime.ai/[Microsoft]
1292
+ * Hunspell: https://hunspell.github.io/[László Németh]