kotoshu 1.0.2-x86_64-linux

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (458) hide show
  1. checksums.yaml +7 -0
  2. data/.kotoshu-baseline.json +1 -0
  3. data/.pre-commit-hooks.yaml +18 -0
  4. data/.rspec +3 -0
  5. data/.rubocop.yml +351 -0
  6. data/.yardopts +8 -0
  7. data/CHANGELOG.md +813 -0
  8. data/CLAUDE.md +176 -0
  9. data/CODE_OF_CONDUCT.md +132 -0
  10. data/Cargo.lock +359 -0
  11. data/Cargo.toml +15 -0
  12. data/LICENSE +31 -0
  13. data/README.adoc +1292 -0
  14. data/Rakefile +150 -0
  15. data/SECURITY.md +93 -0
  16. data/TODO.impl/00-cut-0.1.md +121 -0
  17. data/TODO.impl/00-cut-0.2.md +175 -0
  18. data/TODO.impl/00-vision.md +281 -0
  19. data/TODO.impl/01-hunspell-correctness.md +111 -0
  20. data/TODO.impl/02-cli-unification.md +74 -0
  21. data/TODO.impl/03-dynamic-download.md +107 -0
  22. data/TODO.impl/04-language-modules.md +89 -0
  23. data/TODO.impl/05-semantic-path.md +83 -0
  24. data/TODO.impl/06-cjk-support.md +105 -0
  25. data/TODO.impl/07-rtl-support.md +71 -0
  26. data/TODO.impl/08-grammar-engine.md +106 -0
  27. data/TODO.impl/09-integrity-security.md +90 -0
  28. data/TODO.impl/10-testing-ci.md +98 -0
  29. data/TODO.impl/100-full-feature-batch3.md +28 -0
  30. data/TODO.impl/101-int4-tier-experiment.md +32 -0
  31. data/TODO.impl/102-lid-chain.md +28 -0
  32. data/TODO.impl/103-oov-bucket-tables.md +27 -0
  33. data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
  34. data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
  35. data/TODO.impl/106-native-lid-detection.md +36 -0
  36. data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
  37. data/TODO.impl/108-ko-ne-full-feature.md +37 -0
  38. data/TODO.impl/109-ten-audit.md +26 -0
  39. data/TODO.impl/11-release-v1.md +95 -0
  40. data/TODO.impl/110-sr-latn-and-polish.md +33 -0
  41. data/TODO.impl/111-embedding-bakeoff.md +33 -0
  42. data/TODO.impl/112-deployability-and-gating.md +26 -0
  43. data/TODO.impl/113-language-packs.md +26 -0
  44. data/TODO.impl/114-typo-biencoder-thread.md +33 -0
  45. data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
  46. data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
  47. data/TODO.impl/117-deterministic-frequency-tiers.md +110 -0
  48. data/TODO.impl/118-server-gem-floor.md +33 -0
  49. data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
  50. data/TODO.impl/120-setup-metadata-refresh.md +35 -0
  51. data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
  52. data/TODO.impl/122-pack-compression-kpk2.md +66 -0
  53. data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
  54. data/TODO.impl/124-owner-residue.md +19 -0
  55. data/TODO.impl/125-consumer-floor-guards.md +43 -0
  56. data/TODO.impl/126-tier-error-budgets.md +33 -0
  57. data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
  58. data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
  59. data/TODO.impl/129-pt-fr-corpora.md +23 -0
  60. data/TODO.impl/130-news-truth-0912.md +23 -0
  61. data/TODO.impl/131-hybrid-productization.md +42 -0
  62. data/TODO.impl/132-techniques-page-and-prose.md +27 -0
  63. data/TODO.impl/133-rust-first-distribution.md +68 -0
  64. data/TODO.impl/20-encapsulation-cleanup.md +47 -0
  65. data/TODO.impl/21-autoload-migration.md +64 -0
  66. data/TODO.impl/22-url-pin-single-source-of-truth.md +120 -0
  67. data/TODO.impl/23-fix-url-defaults.md +57 -0
  68. data/TODO.impl/24-cli-auto-setup-prompt.md +76 -0
  69. data/TODO.impl/25-soft-onnxruntime-require.md +75 -0
  70. data/TODO.impl/26-onnx-vocab-json-generation.md +96 -0
  71. data/TODO.impl/27-end-to-end-smoke-spec.md +113 -0
  72. data/TODO.impl/28-readme-quickstart-verification.md +68 -0
  73. data/TODO.impl/29-status-command.md +84 -0
  74. data/TODO.impl/30-language-auto-detection.md +74 -0
  75. data/TODO.impl/31-download-progress.md +58 -0
  76. data/TODO.impl/32-sarif-output-verification.md +72 -0
  77. data/TODO.impl/33-audit-log-rotation.md +54 -0
  78. data/TODO.impl/34-cache-eviction.md +59 -0
  79. data/TODO.impl/35-shell-completion.md +61 -0
  80. data/TODO.impl/36-cleanup-regressions-0.3.1.md +134 -0
  81. data/TODO.impl/37-hunspell-correctness-tier2.md +203 -0
  82. data/TODO.impl/38-onnx-semantic-gating.md +218 -0
  83. data/TODO.impl/39-tier3-and-beyond.md +232 -0
  84. data/TODO.impl/40-spec-drift-cleanup.md +134 -0
  85. data/TODO.impl/50-structure-aware-document-api.md +194 -0
  86. data/TODO.impl/51-grammar-engine-expansion.md +75 -0
  87. data/TODO.impl/52-multi-language-documents.md +95 -0
  88. data/TODO.impl/53-document-plugin-architecture.md +94 -0
  89. data/TODO.impl/54-cjk-support.md +117 -0
  90. data/TODO.impl/55-rtl-support.md +109 -0
  91. data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
  92. data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
  93. data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
  94. data/TODO.impl/59-yard-api-documentation.md +75 -0
  95. data/TODO.impl/60-lsp-server.md +174 -0
  96. data/TODO.impl/61-editor-ecosystem.md +167 -0
  97. data/TODO.impl/62-framework-integrations.md +159 -0
  98. data/TODO.impl/63-web-playground-wasm.md +186 -0
  99. data/TODO.impl/64-http-api-and-sdks.md +191 -0
  100. data/TODO.impl/65-universal-kotoshu.md +251 -0
  101. data/TODO.impl/66-kotoshu-core.md +169 -0
  102. data/TODO.impl/67-kotoshu-rs-and-access-libraries.md +137 -0
  103. data/TODO.impl/68-sota-adoption.md +152 -0
  104. data/TODO.impl/69-sota-typo-eval.md +74 -0
  105. data/TODO.impl/70-sota-confidence-cascade.md +59 -0
  106. data/TODO.impl/71-sota-nested-model.md +63 -0
  107. data/TODO.impl/72-model2vec-evaluation.md +49 -0
  108. data/TODO.impl/73-rl-reranker.md +42 -0
  109. data/TODO.impl/74-distilled-reranker.md +36 -0
  110. data/TODO.impl/75-llm-evaluator.md +35 -0
  111. data/TODO.impl/76-optional-llm-backend.md +37 -0
  112. data/TODO.impl/77-model-coverage-expansion.md +96 -0
  113. data/TODO.impl/78-in-browser-playground.md +69 -0
  114. data/TODO.impl/79-news-and-releases.md +60 -0
  115. data/TODO.impl/80-ecosystem-docs.md +56 -0
  116. data/TODO.impl/81-editor-reach.md +68 -0
  117. data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
  118. data/TODO.impl/83-models-batch-2.md +56 -0
  119. data/TODO.impl/84-language-modules-wave2.md +75 -0
  120. data/TODO.impl/85-wasm-model-api.md +48 -0
  121. data/TODO.impl/86-site-truth-wave2.md +45 -0
  122. data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
  123. data/TODO.impl/88-cli-directory-mode.md +54 -0
  124. data/TODO.impl/89-framework-integrations-exec.md +59 -0
  125. data/TODO.impl/90-gem-correctness-residues.md +44 -0
  126. data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
  127. data/TODO.impl/92-tier-mirror-cors.md +88 -0
  128. data/TODO.impl/93-python-wheels.md +32 -0
  129. data/TODO.impl/94-site-integrations-docs.md +35 -0
  130. data/TODO.impl/95-editor-action-polish.md +27 -0
  131. data/TODO.impl/96-server-semantic-models.md +50 -0
  132. data/TODO.impl/97-yard-execution.md +73 -0
  133. data/TODO.impl/98-campaign-cleanup.md +78 -0
  134. data/TODO.impl/99-site-truth-performance-docs.md +31 -0
  135. data/conformance/vectors.jsonl +2630 -0
  136. data/data/en/grammar/rules.yaml +851 -0
  137. data/docs/STABILITY-1.0-AUDIT.md +251 -0
  138. data/docs/STABILITY-1.0.md +137 -0
  139. data/docs/cache-architecture.md +502 -0
  140. data/docs/verification-evidence/correct3/pr_body.md +37 -0
  141. data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
  142. data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
  143. data/docs/verification-evidence/correct3/sv.aff +856 -0
  144. data/docs/verification-evidence/correct3/sv.dic +153717 -0
  145. data/examples/01_basic_word_checking.rb +38 -0
  146. data/examples/02_text_document_checking.rb +77 -0
  147. data/examples/03_dictionary_backends.rb +137 -0
  148. data/examples/04_trie_data_structure.rb +146 -0
  149. data/examples/05_suggestion_algorithms.rb +239 -0
  150. data/examples/06_configuration_advanced.rb +287 -0
  151. data/examples/07_multi_language_dictionaries.rb +278 -0
  152. data/exe/kotoshu +6 -0
  153. data/lib/kotoshu/algorithms/capitalization.rb +276 -0
  154. data/lib/kotoshu/algorithms/edit_distance.rb +128 -0
  155. data/lib/kotoshu/algorithms/lookup/affix_form.rb +132 -0
  156. data/lib/kotoshu/algorithms/lookup/compound_checks.rb +209 -0
  157. data/lib/kotoshu/algorithms/lookup/compound_form.rb +50 -0
  158. data/lib/kotoshu/algorithms/lookup.rb +841 -0
  159. data/lib/kotoshu/algorithms/ngram_suggest.rb +367 -0
  160. data/lib/kotoshu/algorithms/permutations.rb +283 -0
  161. data/lib/kotoshu/algorithms/phonet_suggest.rb +162 -0
  162. data/lib/kotoshu/algorithms/soundex.rb +92 -0
  163. data/lib/kotoshu/algorithms/suggest.rb +794 -0
  164. data/lib/kotoshu/algorithms.rb +23 -0
  165. data/lib/kotoshu/analyzers/semantic_analyzer.rb +405 -0
  166. data/lib/kotoshu/analyzers.rb +8 -0
  167. data/lib/kotoshu/baseline/application.rb +18 -0
  168. data/lib/kotoshu/baseline/entry.rb +33 -0
  169. data/lib/kotoshu/baseline/store.rb +175 -0
  170. data/lib/kotoshu/baseline.rb +20 -0
  171. data/lib/kotoshu/cache/base_cache.rb +778 -0
  172. data/lib/kotoshu/cache/cache.rb +91 -0
  173. data/lib/kotoshu/cache/eviction_policy.rb +66 -0
  174. data/lib/kotoshu/cache/frequency_cache.rb +283 -0
  175. data/lib/kotoshu/cache/language_cache.rb +511 -0
  176. data/lib/kotoshu/cache/lookup_cache.rb +166 -0
  177. data/lib/kotoshu/cache/model_cache.rb +1147 -0
  178. data/lib/kotoshu/cache/model_registry.rb +109 -0
  179. data/lib/kotoshu/cache/staged_languages.rb +45 -0
  180. data/lib/kotoshu/cache/suggestion_cache.rb +113 -0
  181. data/lib/kotoshu/cache.rb +49 -0
  182. data/lib/kotoshu/cli/auto_setup.rb +71 -0
  183. data/lib/kotoshu/cli/baseline_command.rb +119 -0
  184. data/lib/kotoshu/cli/batch_reporter.rb +321 -0
  185. data/lib/kotoshu/cli/cache_command.rb +282 -0
  186. data/lib/kotoshu/cli/completions_command.rb +210 -0
  187. data/lib/kotoshu/cli/dict_command.rb +64 -0
  188. data/lib/kotoshu/cli/directory_check.rb +146 -0
  189. data/lib/kotoshu/cli/directory_walker.rb +132 -0
  190. data/lib/kotoshu/cli/display_formatter.rb +429 -0
  191. data/lib/kotoshu/cli/errors.rb +36 -0
  192. data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
  193. data/lib/kotoshu/cli/interactive_reviewer.rb +314 -0
  194. data/lib/kotoshu/cli/language_resolver.rb +91 -0
  195. data/lib/kotoshu/cli/navigation_manager.rb +273 -0
  196. data/lib/kotoshu/cli/personal_command.rb +113 -0
  197. data/lib/kotoshu/cli/progress_reporter.rb +114 -0
  198. data/lib/kotoshu/cli/status_report.rb +130 -0
  199. data/lib/kotoshu/cli.rb +880 -0
  200. data/lib/kotoshu/components/passthrough_spell_checker.rb +70 -0
  201. data/lib/kotoshu/components/pos_tagger.rb +98 -0
  202. data/lib/kotoshu/components/spell_checker.rb +73 -0
  203. data/lib/kotoshu/components/synthesizer.rb +60 -0
  204. data/lib/kotoshu/components/tokenizer.rb +58 -0
  205. data/lib/kotoshu/components/whitespace_tokenizer.rb +94 -0
  206. data/lib/kotoshu/components.rb +13 -0
  207. data/lib/kotoshu/configuration/builder.rb +207 -0
  208. data/lib/kotoshu/configuration/resolver.rb +124 -0
  209. data/lib/kotoshu/configuration.rb +829 -0
  210. data/lib/kotoshu/conformance_exporter.rb +256 -0
  211. data/lib/kotoshu/conformance_runner.rb +176 -0
  212. data/lib/kotoshu/core/exceptions.rb +186 -0
  213. data/lib/kotoshu/core/indexed_dictionary.rb +293 -0
  214. data/lib/kotoshu/core/models/affix_rule.rb +262 -0
  215. data/lib/kotoshu/core/models/result/document_result.rb +132 -0
  216. data/lib/kotoshu/core/models/result/word_result.rb +131 -0
  217. data/lib/kotoshu/core/models/word.rb +131 -0
  218. data/lib/kotoshu/core/trie/builder.rb +119 -0
  219. data/lib/kotoshu/core/trie/node.rb +94 -0
  220. data/lib/kotoshu/core/trie/trie.rb +249 -0
  221. data/lib/kotoshu/core.rb +28 -0
  222. data/lib/kotoshu/data/common_words/de.yml +1800 -0
  223. data/lib/kotoshu/data/common_words/en.yml +1215 -0
  224. data/lib/kotoshu/data/common_words/es.yml +750 -0
  225. data/lib/kotoshu/data/common_words/fr.yml +1015 -0
  226. data/lib/kotoshu/data/common_words/pt.yml +870 -0
  227. data/lib/kotoshu/data/common_words/ru.yml +484 -0
  228. data/lib/kotoshu/data/common_words_loader.rb +141 -0
  229. data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
  230. data/lib/kotoshu/data/frozen_kelly.rb +38 -0
  231. data/lib/kotoshu/data.rb +8 -0
  232. data/lib/kotoshu/data_structures/bloom_filter.rb +188 -0
  233. data/lib/kotoshu/data_structures.rb +8 -0
  234. data/lib/kotoshu/debug_logger.rb +146 -0
  235. data/lib/kotoshu/debug_mode.rb +136 -0
  236. data/lib/kotoshu/defaults.rb +84 -0
  237. data/lib/kotoshu/dictionaries/catalog.rb +814 -0
  238. data/lib/kotoshu/dictionaries.rb +8 -0
  239. data/lib/kotoshu/dictionary/base.rb +324 -0
  240. data/lib/kotoshu/dictionary/cspell.rb +243 -0
  241. data/lib/kotoshu/dictionary/custom.rb +225 -0
  242. data/lib/kotoshu/dictionary/hunspell.rb +520 -0
  243. data/lib/kotoshu/dictionary/plain_text.rb +302 -0
  244. data/lib/kotoshu/dictionary/repository.rb +246 -0
  245. data/lib/kotoshu/dictionary/unified.rb +236 -0
  246. data/lib/kotoshu/dictionary/unix_words.rb +218 -0
  247. data/lib/kotoshu/dictionary.rb +15 -0
  248. data/lib/kotoshu/documents/document.rb +120 -0
  249. data/lib/kotoshu/documents/plain_text_document.rb +73 -0
  250. data/lib/kotoshu/documents/source_position.rb +45 -0
  251. data/lib/kotoshu/documents/source_range.rb +76 -0
  252. data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
  253. data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
  254. data/lib/kotoshu/documents/suppressions.rb +44 -0
  255. data/lib/kotoshu/documents/text_node.rb +49 -0
  256. data/lib/kotoshu/documents.rb +178 -0
  257. data/lib/kotoshu/embeddings/embedding_pipeline.rb +238 -0
  258. data/lib/kotoshu/embeddings/lru_cache.rb +244 -0
  259. data/lib/kotoshu/embeddings/onnx_runtime_model.rb +411 -0
  260. data/lib/kotoshu/embeddings/protocol.rb +96 -0
  261. data/lib/kotoshu/embeddings/protocols.rb +23 -0
  262. data/lib/kotoshu/embeddings/registry.rb +184 -0
  263. data/lib/kotoshu/embeddings/search.rb +204 -0
  264. data/lib/kotoshu/embeddings/similarity_engine.rb +278 -0
  265. data/lib/kotoshu/embeddings/similarity_search.rb +328 -0
  266. data/lib/kotoshu/embeddings/vocabulary.rb +306 -0
  267. data/lib/kotoshu/embeddings.rb +87 -0
  268. data/lib/kotoshu/fluent_checker.rb +94 -0
  269. data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +48 -0
  270. data/lib/kotoshu/grammar/pattern_matchers/double_negative_matcher.rb +179 -0
  271. data/lib/kotoshu/grammar/pattern_matchers/phrase_matcher.rb +86 -0
  272. data/lib/kotoshu/grammar/pattern_matchers/possessive_context_matcher.rb +75 -0
  273. data/lib/kotoshu/grammar/pattern_matchers/possessive_contraction_matcher.rb +87 -0
  274. data/lib/kotoshu/grammar/pattern_matchers/sentence_start_matcher.rb +84 -0
  275. data/lib/kotoshu/grammar/pattern_matchers/vowel_sound_matcher.rb +81 -0
  276. data/lib/kotoshu/grammar/pattern_matchers/word_list_matcher.rb +66 -0
  277. data/lib/kotoshu/grammar/rule.rb +99 -0
  278. data/lib/kotoshu/grammar/rule_engine.rb +96 -0
  279. data/lib/kotoshu/grammar/rule_loader.rb +30 -0
  280. data/lib/kotoshu/grammar.rb +24 -0
  281. data/lib/kotoshu/integrity/audit_log.rb +167 -0
  282. data/lib/kotoshu/integrity/manifest.rb +116 -0
  283. data/lib/kotoshu/integrity/net_http.rb +51 -0
  284. data/lib/kotoshu/integrity/rotation_policy.rb +88 -0
  285. data/lib/kotoshu/integrity.rb +26 -0
  286. data/lib/kotoshu/jekyll.rb +101 -0
  287. data/lib/kotoshu/keyboard/layout.rb +116 -0
  288. data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
  289. data/lib/kotoshu/keyboard/layouts/azerty.rb +55 -0
  290. data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
  291. data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
  292. data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
  293. data/lib/kotoshu/keyboard/layouts/dvorak.rb +54 -0
  294. data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
  295. data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
  296. data/lib/kotoshu/keyboard/layouts/hrsl.rb +94 -0
  297. data/lib/kotoshu/keyboard/layouts/jcuken.rb +57 -0
  298. data/lib/kotoshu/keyboard/layouts/latin.rb +266 -0
  299. data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
  300. data/lib/kotoshu/keyboard/layouts/qwerty.rb +52 -0
  301. data/lib/kotoshu/keyboard/layouts/qwertz.rb +55 -0
  302. data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
  303. data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
  304. data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
  305. data/lib/kotoshu/keyboard/registry.rb +201 -0
  306. data/lib/kotoshu/keyboard.rb +96 -0
  307. data/lib/kotoshu/kotoshu_native.so +0 -0
  308. data/lib/kotoshu/language/detector.rb +242 -0
  309. data/lib/kotoshu/language/identifier.rb +376 -0
  310. data/lib/kotoshu/language/languages/base.rb +271 -0
  311. data/lib/kotoshu/language/lid_detector.rb +188 -0
  312. data/lib/kotoshu/language/normalizer/arabic.rb +113 -0
  313. data/lib/kotoshu/language/normalizer/base.rb +133 -0
  314. data/lib/kotoshu/language/normalizer/greek.rb +63 -0
  315. data/lib/kotoshu/language/normalizer/hebrew.rb +72 -0
  316. data/lib/kotoshu/language/normalizer/persian.rb +36 -0
  317. data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
  318. data/lib/kotoshu/language/registry.rb +209 -0
  319. data/lib/kotoshu/language/resources/ar/common_words.txt +6753 -0
  320. data/lib/kotoshu/language/resources/ar/confusion_sets.txt +11 -0
  321. data/lib/kotoshu/language/resources/de/common_words.txt +10003 -0
  322. data/lib/kotoshu/language/resources/de/confusion_sets.txt +246 -0
  323. data/lib/kotoshu/language/resources/en/common_words.txt +9979 -0
  324. data/lib/kotoshu/language/resources/en/confusion_sets.txt +871 -0
  325. data/lib/kotoshu/language/resources/es/common_words.txt +9992 -0
  326. data/lib/kotoshu/language/resources/es/confusion_sets.txt +17 -0
  327. data/lib/kotoshu/language/resources/fr/common_words.txt +9993 -0
  328. data/lib/kotoshu/language/resources/fr/confusion_sets.txt +76 -0
  329. data/lib/kotoshu/language/resources/pt/common_words.txt +9977 -0
  330. data/lib/kotoshu/language/resources/pt/confusion_sets.txt +18 -0
  331. data/lib/kotoshu/language/resources/ru/common_words.txt +9951 -0
  332. data/lib/kotoshu/language/resources/ru/confusion_sets.txt +5 -0
  333. data/lib/kotoshu/language/script.rb +128 -0
  334. data/lib/kotoshu/language/segmenter.rb +122 -0
  335. data/lib/kotoshu/language/suika.rb +38 -0
  336. data/lib/kotoshu/language/tokenizer/base.rb +187 -0
  337. data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
  338. data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
  339. data/lib/kotoshu/language/tokenizer/french_tokenizer.rb +172 -0
  340. data/lib/kotoshu/language/tokenizer/german_tokenizer.rb +41 -0
  341. data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
  342. data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
  343. data/lib/kotoshu/language/tokenizer/japanese_tokenizer.rb +55 -0
  344. data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +153 -0
  345. data/lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb +160 -0
  346. data/lib/kotoshu/language/tokenizer/russian_tokenizer.rb +93 -0
  347. data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
  348. data/lib/kotoshu/language/tokenizer/spanish_tokenizer.rb +122 -0
  349. data/lib/kotoshu/language.rb +127 -0
  350. data/lib/kotoshu/languages/ar/language.rb +98 -0
  351. data/lib/kotoshu/languages/bg/language.rb +37 -0
  352. data/lib/kotoshu/languages/ca/language.rb +22 -0
  353. data/lib/kotoshu/languages/cs/language.rb +22 -0
  354. data/lib/kotoshu/languages/da/language.rb +22 -0
  355. data/lib/kotoshu/languages/de/language.rb +554 -0
  356. data/lib/kotoshu/languages/el/language.rb +50 -0
  357. data/lib/kotoshu/languages/en/language.rb +470 -0
  358. data/lib/kotoshu/languages/es/language.rb +476 -0
  359. data/lib/kotoshu/languages/et/language.rb +24 -0
  360. data/lib/kotoshu/languages/fa/language.rb +91 -0
  361. data/lib/kotoshu/languages/fr/language.rb +505 -0
  362. data/lib/kotoshu/languages/he/language.rb +98 -0
  363. data/lib/kotoshu/languages/hr/language.rb +24 -0
  364. data/lib/kotoshu/languages/hu/language.rb +22 -0
  365. data/lib/kotoshu/languages/id/language.rb +25 -0
  366. data/lib/kotoshu/languages/it/language.rb +22 -0
  367. data/lib/kotoshu/languages/ja/language.rb +476 -0
  368. data/lib/kotoshu/languages/ko/language.rb +56 -0
  369. data/lib/kotoshu/languages/latin_base.rb +53 -0
  370. data/lib/kotoshu/languages/lt/language.rb +24 -0
  371. data/lib/kotoshu/languages/lv/language.rb +24 -0
  372. data/lib/kotoshu/languages/nb/language.rb +25 -0
  373. data/lib/kotoshu/languages/ne/language.rb +60 -0
  374. data/lib/kotoshu/languages/nl/language.rb +22 -0
  375. data/lib/kotoshu/languages/nn/language.rb +26 -0
  376. data/lib/kotoshu/languages/pl/language.rb +22 -0
  377. data/lib/kotoshu/languages/pt/language.rb +439 -0
  378. data/lib/kotoshu/languages/ro/language.rb +22 -0
  379. data/lib/kotoshu/languages/ru/language.rb +420 -0
  380. data/lib/kotoshu/languages/sk/language.rb +25 -0
  381. data/lib/kotoshu/languages/sl/language.rb +25 -0
  382. data/lib/kotoshu/languages/sr/language.rb +39 -0
  383. data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
  384. data/lib/kotoshu/languages/sv/language.rb +22 -0
  385. data/lib/kotoshu/languages/tr/language.rb +36 -0
  386. data/lib/kotoshu/languages/uk/language.rb +38 -0
  387. data/lib/kotoshu/languages/vi/language.rb +22 -0
  388. data/lib/kotoshu/languages.rb +64 -0
  389. data/lib/kotoshu/metrics_collector.rb +238 -0
  390. data/lib/kotoshu/metrics_module.rb +121 -0
  391. data/lib/kotoshu/models/context.rb +119 -0
  392. data/lib/kotoshu/models/embedding_model.rb +181 -0
  393. data/lib/kotoshu/models/fasttext_model.rb +212 -0
  394. data/lib/kotoshu/models/nearest_neighbor.rb +96 -0
  395. data/lib/kotoshu/models/onnx_model.rb +330 -0
  396. data/lib/kotoshu/models/semantic_error.rb +204 -0
  397. data/lib/kotoshu/models/suggestion.rb +106 -0
  398. data/lib/kotoshu/models/word_embedding.rb +97 -0
  399. data/lib/kotoshu/models.rb +27 -0
  400. data/lib/kotoshu/multi_language_checker.rb +127 -0
  401. data/lib/kotoshu/native.rb +83 -0
  402. data/lib/kotoshu/native_backend.rb +131 -0
  403. data/lib/kotoshu/paths.rb +79 -0
  404. data/lib/kotoshu/personal_dictionary.rb +105 -0
  405. data/lib/kotoshu/plugins/plugin.rb +61 -0
  406. data/lib/kotoshu/plugins/registry.rb +118 -0
  407. data/lib/kotoshu/plugins.rb +9 -0
  408. data/lib/kotoshu/project_config.rb +76 -0
  409. data/lib/kotoshu/readers/aff_data.rb +530 -0
  410. data/lib/kotoshu/readers/aff_reader.rb +393 -0
  411. data/lib/kotoshu/readers/condition_checker.rb +125 -0
  412. data/lib/kotoshu/readers/dic_reader.rb +208 -0
  413. data/lib/kotoshu/readers/file_reader.rb +377 -0
  414. data/lib/kotoshu/readers/lookup_builder.rb +584 -0
  415. data/lib/kotoshu/readers/readers.rb +17 -0
  416. data/lib/kotoshu/readers.rb +25 -0
  417. data/lib/kotoshu/resource_bundle.rb +55 -0
  418. data/lib/kotoshu/resource_manager.rb +600 -0
  419. data/lib/kotoshu/results/result.rb +165 -0
  420. data/lib/kotoshu/results.rb +8 -0
  421. data/lib/kotoshu/rspec.rb +163 -0
  422. data/lib/kotoshu/scripts/fasttext_to_onnx.py +275 -0
  423. data/lib/kotoshu/source_registry.rb +104 -0
  424. data/lib/kotoshu/spellchecker/parallel_checker.rb +90 -0
  425. data/lib/kotoshu/spellchecker.rb +406 -0
  426. data/lib/kotoshu/string_metrics.rb +153 -0
  427. data/lib/kotoshu/suggestions/context.rb +43 -0
  428. data/lib/kotoshu/suggestions/frequency_provider.rb +80 -0
  429. data/lib/kotoshu/suggestions/generator.rb +169 -0
  430. data/lib/kotoshu/suggestions/pipeline.rb +135 -0
  431. data/lib/kotoshu/suggestions/semantic_cascade.rb +73 -0
  432. data/lib/kotoshu/suggestions/strategies/base_strategy.rb +315 -0
  433. data/lib/kotoshu/suggestions/strategies/composite_strategy.rb +197 -0
  434. data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +631 -0
  435. data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +207 -0
  436. data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +165 -0
  437. data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +284 -0
  438. data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +317 -0
  439. data/lib/kotoshu/suggestions/strategies/symspell_strategy.rb +270 -0
  440. data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
  441. data/lib/kotoshu/suggestions/suggestion.rb +154 -0
  442. data/lib/kotoshu/suggestions/suggestion_set.rb +270 -0
  443. data/lib/kotoshu/suggestions/sweep_index.rb +137 -0
  444. data/lib/kotoshu/suggestions.rb +29 -0
  445. data/lib/kotoshu/tasks/check_task.rb +104 -0
  446. data/lib/kotoshu/tasks.rb +11 -0
  447. data/lib/kotoshu/validators/spelling_validator.rb +101 -0
  448. data/lib/kotoshu/validators.rb +9 -0
  449. data/lib/kotoshu/version.rb +5 -0
  450. data/lib/kotoshu.rb +538 -0
  451. data/rakelib/compile.rake +10 -0
  452. data/script/validate_all_dictionaries.rb +444 -0
  453. data/scripts/build_native_gem.rb +45 -0
  454. data/scripts/extract_eval_grids.rb +37 -0
  455. data/scripts/generate_frozen_tiers.rb +44 -0
  456. data/sig/kotoshu.rbs +435 -0
  457. data/test_oop.rb +79 -0
  458. metadata +546 -0
@@ -0,0 +1,83 @@
1
+ # 05 — Semantic Path Productionization
2
+
3
+ ## Goal
4
+
5
+ The ONNX/FastText semantic path is a first-class, always-available
6
+ dimension of analysis for every language that has an embedding model —
7
+ not an opt-in afterthought.
8
+
9
+ ## Why
10
+
11
+ Today the semantic path is gated behind `ENV['KOTOSHU_REQUIRE_ONNX']`
12
+ (see `lib/kotoshu/models/onnx_model.rb` line 22). Without that env var,
13
+ the onnxruntime gem isn't loaded and semantic features silently
14
+ disappear. The hybrid model (`README.adoc` advertises "Hunspell
15
+ candidates + FastText reranking") is the recommended mode but isn't
16
+ wired as a default.
17
+
18
+ ## Tasks
19
+
20
+ 1. **Drop the env gate.** Require `onnxruntime` lazily on first
21
+ `OnnxModel` instantiation, with a clear error if the gem isn't
22
+ installed (`RuntimeError` with install instructions). Remove the
23
+ `KOTOSHU_REQUIRE_ONNX` env var — it's an undocumented footgun.
24
+ 2. **Make `onnxruntime` a hard runtime dependency** in
25
+ `kotoshu.gemspec` (it already is, but the lazy require contradicts
26
+ this). Verify `gem install kotoshu` pulls it in.
27
+ 3. **Default to hybrid mode.** The README says hybrid is "recommended"
28
+ but the code defaults to `hunspell` only (see
29
+ `commands/check_command.rb` line 23). Make `:hybrid` the default;
30
+ document when to choose pure `:hunspell` (speed) or pure
31
+ `:fasttext` (semantic-only, no dictionary).
32
+ 4. **Memory budget.** ONNX models are 200MB-4GB each. Only one model
33
+ per language should be resident at a time. Wire
34
+ `Embeddings::LruCache` so the `ResourceManager` owns the cache and
35
+ evicts when memory pressure exceeds a configurable ceiling
36
+ (`Configuration.max_embedding_memory_mb`, default 1024).
37
+ 5. **Cold-start latency.** Loading a 4GB ONNX model takes seconds.
38
+ Show a one-time progress message in the CLI; expose a `preload`
39
+ method on `OnnxModel` so background warming is possible.
40
+ 6. **Vocabulary coverage fallback.** If a word is OOV for the
41
+ embedding model, fall back to subword FastText if available;
42
+ otherwise skip semantic reranking for that word (don't crash).
43
+ 7. **Hybrid scoring formula.** Document and tune the actual formula:
44
+ `final_score = α·hunspell_distance_score + β·semantic_similarity +
45
+ γ·frequency_bonus`. Make α/β/γ configurable per language; defaults
46
+ per language live in the language module.
47
+ 8. **Benchmark suite.** Add `spec/performance/semantic_bench.rb`:
48
+ - Time to load model (baseline: < 1 s for the 114 MB models)
49
+ - Time per 1000 suggestions with vs. without semantic rerank
50
+ (baseline: ~1–2 ms/query inference, ~115 MB resident per language)
51
+ - Memory ceiling under load
52
+ Track these in `PERFORMANCE.md` and gate against regressions in CI.
53
+ 9. **ONNX Runtime version compatibility.** README claims
54
+ `onnxruntime 1.23.2+`. Verify across 1.22, 1.23, 1.24. Lock the gem
55
+ version range in gemspec and document.
56
+
57
+ ## Acceptance criteria
58
+
59
+ - `Kotoshu.suggest("helo", language: "en")` uses hybrid mode by
60
+ default, no env var needed
61
+ - Loading two languages sequentially does not double memory (LRU
62
+ evicts)
63
+ - `spec/performance/semantic_bench.rb` runs in CI and fails on >20%
64
+ regression vs. baseline
65
+ - README's "hybrid" example works as documented
66
+ - `KOTOSHU_REQUIRE_ONNX` is removed from the codebase and from any
67
+ docs that mention it
68
+
69
+ ## Dependencies
70
+
71
+ - Blocks: `11-release`
72
+ - Blocked by: `03-dynamic-download` (model resolution), `09-integrity-security`
73
+ - Cross-repo: depends on `models-fasttext-onnx/TODO.impl/01-publish-all-models.md`
74
+
75
+ ## Out of scope
76
+
77
+ - Training new FastText models (use upstream Facebook vectors)
78
+ - GPU acceleration (CPU onnxruntime is the v1 target)
79
+ - Quantization / INT8 (already 37x compressed vs. raw .vec — defer)
80
+
81
+ ## Status
82
+
83
+ _Pending._
@@ -0,0 +1,105 @@
1
+ # 06 — CJK Support (Japanese, Chinese)
2
+
3
+ ## Goal
4
+
5
+ Japanese and Chinese work end-to-end. CJK doesn't have "spelling" in the
6
+ alphabetic sense — the paradigm is morphological tokenization +
7
+ confusion-pair rules. Kotoshu should expose this honestly rather than
8
+ pretend a dictionary lookup works.
9
+
10
+ ## Why
11
+
12
+ ROADMAP plan `004-japanese.md` and `005-chinese.md` are 🔲 Not Started.
13
+ The gem ships a `JapaneseTokenizer` skeleton under
14
+ `lib/kotoshu/language/tokenizer/` but no morphological backend, and a
15
+ `ja/language.rb` module with no real wiring. The `dictionaries` repo
16
+ has `ja/` and `zh/` dirs but only with `grammar/` subdirs (no
17
+ `spelling/`) — correct, since CJK doesn't use spelling dicts.
18
+
19
+ ## Paradigm
20
+
21
+ | Concern | Latin | CJK |
22
+ |---|---|---|
23
+ | Spellcheck | Dictionary membership + affixes | N/A |
24
+ | Tokenization | Whitespace/regex | Morphological analyzer |
25
+ | Errors | Out-of-dict words | Confusion-pair rules (e.g. 辨/辯/辮) |
26
+ | Suggestions | Edit distance + phonetic | Confusion probability tables |
27
+
28
+ ## Tasks
29
+
30
+ ### Japanese
31
+
32
+ 1. **Pick a morphological backend.** Options:
33
+ - **Sen** (Java, requires JRuby — likely unacceptable for pure-Ruby
34
+ promise)
35
+ - **Suika** (already in the gemspec — pure Ruby MeCab-IPADic port,
36
+ the right choice)
37
+ - FFI to MeCab (breaks pure-Ruby promise)
38
+ Decision: **use suika**. Add `Kotoshu::Language::Tokenizer::SuikaTokenizer`
39
+ that wraps `Suika::Tagger`.
40
+ 2. **Wire `JapaneseTokenizer`** to delegate to suika for segmentation +
41
+ POS. Keep the existing class name for backward compatibility.
42
+ 3. **Confusion rules.** Source a Japanese confusion-pair list (homophone
43
+ kanji 食べる/搗ける etc.) and ship as
44
+ `dictionaries/ja/grammar/confusion.yaml`. The grammar engine
45
+ (plan `08`) reads it.
46
+ 4. **Reading-check rules.** Kotoshi-specific (言修 = ことしゅ): verify
47
+ furigana/reading correctness against jumbo dictionaries. Defer if
48
+ out of scope for v1.
49
+ 5. **No embedding model.** Japanese FastText exists upstream but the
50
+ `models-fasttext-onnx` repo doesn't have `ja/`. Either convert it
51
+ (cross-ref `models-fasttext-onnx/TODO.impl/01-publish-all-models.md`)
52
+ or document Japanese as traditional-rules-only.
53
+
54
+ ### Chinese
55
+
56
+ 6. **Pick a segmenter.** Pure-Ruby options are limited. Options:
57
+ - Hand-rolled maximum-matching against a word list (acceptable for
58
+ v1, fast, deterministic)
59
+ - FFI to HanLP (breaks pure-Ruby promise)
60
+ - Port a small CRF model to Ruby (large effort)
61
+ Decision: **ship max-matching with the existing dictionaries/zh word
62
+ list for v1; note the accuracy ceiling in docs**.
63
+ 7. **Simplified/Traditional** detection and cross-checking.
64
+ 8. **Confusion rules.** Common confusions (的/得/地, 在/再, 做/作) ship
65
+ as `dictionaries/zh/grammar/confusion.yaml`.
66
+
67
+ ### Shared
68
+
69
+ 9. **Update `Language::Base`** so `script: :cjk` languages declare
70
+ `spell_checker: :passthrough` (use existing
71
+ `Components::PassthroughSpellChecker`) and require a tokenizer.
72
+ 10. **CLI flag `--model`** for CJK should reject `:hunspell` and
73
+ `:hybrid` with a clear error explaining why.
74
+ 11. **CJK-aware SARIF output.** Character offsets must be in Unicode
75
+ codepoints, not bytes; verify the `Location` class in `documents/`
76
+ handles this.
77
+
78
+ ## Acceptance criteria
79
+
80
+ - `Kotoshu.check("私は 学生 です。")` tokenizes correctly and reports
81
+ no errors (with default rules)
82
+ - `Kotoshu.check("私はがくせいです。")` flags the use of hiragana where
83
+ kanji is conventional, with a confusion-rule suggestion
84
+ - `Kotoshu.check("他是在做好了。")` flags 的/得 confusion
85
+ - README documents that CJK is "rules + tokenization, no spellcheck"
86
+ rather than silently producing nonsense
87
+
88
+ ## Dependencies
89
+
90
+ - Blocks: nothing
91
+ - Blocked by: `03-dynamic-download` (need to fetch the suika dict or
92
+ bundle it), `08-grammar-engine` (confusion rules need the engine)
93
+ - Cross-repo: `dictionaries/ja/grammar/` and `dictionaries/zh/grammar/`
94
+ need content; coordinate with
95
+ `dictionaries/TODO.impl/02-coverage-matrix.md`
96
+
97
+ ## Out of scope
98
+
99
+ - Korean (similar paradigm but no plan written — add later)
100
+ - Full furigana generation
101
+ - Translation / ROMAJI conversion
102
+
103
+ ## Status
104
+
105
+ _Pending._
@@ -0,0 +1,71 @@
1
+ # 07 — RTL Script Support (Arabic, Hebrew, Persian, Urdu)
2
+
3
+ ## Goal
4
+
5
+ Right-to-left scripts work end-to-end with correct shaping, normalization,
6
+ and bidirectional handling.
7
+
8
+ ## Why
9
+
10
+ ROADMAP plan `007-rtl-languages.md` is 🔲 Not Started. The
11
+ `dictionaries` repo has `ar`, `he`, `fa` (likely; verify) dirs.
12
+ Hunspell handles RTL fine in principle, but Ruby's default string
13
+ handling needs explicit care.
14
+
15
+ ## Tasks
16
+
17
+ 1. **Verify dictionary coverage** in the `dictionaries` repo for `ar`,
18
+ `he`, `fa`, `ur`. If any are missing, file under
19
+ `dictionaries/TODO.impl/02-coverage-matrix.md`.
20
+ 2. **Unicode normalization.** Arabic especially has presentation forms
21
+ (U+FExx range) that must be normalized to canonical forms (U+06xx)
22
+ before lookup. Add a `RtlNormalizer` under
23
+ `lib/kotoshu/language/normalizer/`.
24
+ 3. **Bidi handling in output.** When SARIF/text output contains mixed
25
+ RTL/LTR (e.g. error message in English pointing at Arabic text),
26
+ apply explicit Unicode bidi marks (LRM/RLM) so terminal and editor
27
+ render correctly. Verify the `DisplayFormatter` handles this.
28
+ 4. **Tokenizer.** Arabic/Hebrew script tokenizers differ from Latin
29
+ (clitics, the definite article "ال"). Add
30
+ `ArabicTokenizer`/`HebrewTokenizer` under
31
+ `lib/kotoshu/language/tokenizer/`.
32
+ 5. **Keyboard layouts.** Add Arabic (101/102), Hebrew standard. Map to
33
+ the existing `KeyboardProximityStrategy`.
34
+ 6. **FastText models.** All four languages have upstream FastText
35
+ crawl vectors. Verify they're in `models-fasttext-onnx` (the local
36
+ repo has 158 models; spot-check ar/he/fa/ur are present).
37
+ 7. **Grammar rules (light).** Common confusions: Arabic alif variants
38
+ (ا/أ/إ/آ), Hebrew similar-final letter pairs (כ/ך, מ/ם). Ship as
39
+ `dictionaries/{code}/grammar/confusion.yaml`.
40
+ 8. **Kelly frequency.** Arabic has Kelly data; verify Hebrew/Persian/Urdu
41
+ coverage in `frequency-list-kelly` and file extension tasks if
42
+ missing.
43
+ 9. **Regression tests.** Each RTL language gets a spec with a known
44
+ error and assertion that the suggestion is in the correct script
45
+ and the position offset is correct in codepoints.
46
+
47
+ ## Acceptance criteria
48
+
49
+ - `Kotoshu.check("السلام عليكم")` (correct) → no errors
50
+ - `Kotoshu.check("السلم عليكم")` (missing alif) → flagged with
51
+ suggestion "السلام"
52
+ - SARIF output for an RTL error renders correctly on GitHub
53
+ - Terminal output doesn't corrupt cursor position from bidi reordering
54
+
55
+ ## Dependencies
56
+
57
+ - Blocks: nothing
58
+ - Blocked by: `03-dynamic-download`, `04-language-modules`
59
+ - Cross-repo: `dictionaries/TODO.impl/02-coverage-matrix.md`,
60
+ `frequency-list-kelly/TODO.impl/01-extend-coverage.md`,
61
+ `models-fasttext-onnx/TODO.impl/01-publish-all-models.md`
62
+
63
+ ## Out of scope
64
+
65
+ - Diacritics-only checking (tashkeel) for Arabic — future
66
+ - Ligature-aware suggestions
67
+ - Calligraphy-style variants
68
+
69
+ ## Status
70
+
71
+ _Pending._
@@ -0,0 +1,106 @@
1
+ # 08 — Grammar Engine
2
+
3
+ ## Goal
4
+
5
+ Ship a LanguageTool-style rule engine that loads per-language rule packs
6
+ from `dictionaries/{code}/grammar/rules.yaml`, runs pattern matchers
7
+ during analysis, and produces the same `WordResult`-style findings as
8
+ spellcheck errors.
9
+
10
+ ## Why
11
+
12
+ The `grammar/` module exists (`lib/kotoshu/grammar/` — `rule.rb`,
13
+ `rule_engine.rb`, `rule_loader.rb`, `pattern_matchers/`) and there are
14
+ matchers for double_negative, possessive_context, vowel_sound. But:
15
+
16
+ - No rules are shipped in `dictionaries/*/grammar/rules.yaml` (README
17
+ marks it as "future")
18
+ - The engine isn't wired into the `Spellchecker.check` pipeline
19
+ - The CLI doesn't expose `--grammar on|off`
20
+ - No language has a meaningful rule pack
21
+
22
+ ## Tasks
23
+
24
+ 1. **Rule schema.** Finalize the YAML schema
25
+ `dictionaries/{code}/grammar/rules.yaml`:
26
+ ```yaml
27
+ version: 1
28
+ language: en
29
+ rules:
30
+ - id: EN_A_VS_AN
31
+ category: GRAMMAR
32
+ description: "Use 'an' before vowel sounds"
33
+ pattern:
34
+ type: regex
35
+ before: '\b([aA])\s+([aeiouAEIOU]\w*)'
36
+ suggestion: 'an \2'
37
+ examples:
38
+ - incorrect: "a apple"
39
+ correct: "an apple"
40
+ ```
41
+ 2. **Wire the engine into Spellchecker.** After dictionary lookup,
42
+ `Grammar::RuleEngine.new(rules).apply(tokens)` runs and appends
43
+ findings to the `DocumentResult`. Each finding has `category`
44
+ (`GRAMMAR`, `STYLE`, `TYPOS`, `CONFUSION_WORDS`) so output can
45
+ filter.
46
+ 3. **Loader integration.** `ResourceManager` (plan `03`) loads
47
+ `grammar/rules.yaml` as part of the resource bundle for any language
48
+ that ships one.
49
+ 4. **Pattern matcher coverage.** Beyond the three current matchers,
50
+ add:
51
+ - `regex_matcher` — generic regex-based pattern
52
+ - `confusion_matcher` — pair-based confusion (的/得, à/a)
53
+ - `repetition_matcher` — doubled words ("the the")
54
+ - `capitalization_matcher` — sentence-start capitalization
55
+ 5. **Rule disable/enable.** Configuration:
56
+ - `Kotoshu.configure { |c| c.disabled_rules = %w[EN_A_VS_AN] }`
57
+ - CLI: `--disable-rule ID`, `--enable-only-category GRAMMAR`
58
+ 6. **Per-language rule packs.** For each of the 6 fully-supported
59
+ languages, ship a starter pack of 10-20 high-confidence rules
60
+ sourced from LanguageTool's open-source rule sets (license: LGPL).
61
+ Coordinate cross-repo with `dictionaries/TODO.impl/02-coverage-matrix.md`.
62
+ 7. **Rule test fixtures.** Every rule has positive and negative
63
+ examples in its YAML; the spec suite auto-discovers and tests them.
64
+ 8. **Performance.** Rule matching must add < 20% to check latency for
65
+ typical documents. Benchmark in `spec/performance/grammar_bench.rb`.
66
+ 9. **Auto-correction.** `Kotoshu.correct(text)` / `check --apply` applies
67
+ the top suggestion for each finding when confidence is above a
68
+ threshold. Strategies: `:top` (always top suggestion), `:interactive`
69
+ (prompt per finding — reuse `cli/interactive_reviewer`), `:smart`
70
+ (skip when multiple equally-ranked suggestions exist). Default:
71
+ `:smart`.
72
+
73
+ ## Acceptance criteria
74
+
75
+ - `Kotoshu.check("I want to a apple.")` flags the a→an error
76
+ - `Kotoshu.check("the the cat")` flags the doubled word
77
+ - `kotoshu check file.txt --disable-rule EN_A_VS_AN` skips that rule
78
+ - Every shipped rule has passing examples in CI
79
+ - `dictionaries/en/grammar/rules.yaml` has ≥ 15 rules
80
+
81
+ ## Dependencies
82
+
83
+ - Blocks: `06-cjk` (needs confusion rules), `07-rtl` (needs confusion
84
+ rules), `11-release`
85
+ - Blocked by: `03-dynamic-download` (rule loading)
86
+ - Cross-repo: `dictionaries/TODO.impl/02-coverage-matrix.md` (rule
87
+ packs must exist)
88
+
89
+ ## Out of scope
90
+
91
+ - ML-based grammar models (n-gram error detection is in ROADMAP plan
92
+ 008 but defer from v1)
93
+ - Style scoring / "tone" suggestions
94
+ - Premium/paid rule packs
95
+
96
+ ## Status
97
+
98
+ _Pending._
99
+
100
+ ## Source
101
+
102
+ `docs/KOTOSHU_LANGUAGETOOL_GAPS.md` (dated 2026-01-30) is the reference for
103
+ the 11-language target matrix (en, fr, de, es, ru, ar, pt, zh-Hans,
104
+ zh-Hant, ja, ko) and per-language tokenization/normalization rules.
105
+ Tokenization and language-specific requirements feed `04-language-modules`;
106
+ CLI surface and output formats feed `02-cli-unification`.
@@ -0,0 +1,90 @@
1
+ # 09 — Integrity & Security
2
+
3
+ ## Goal
4
+
5
+ Every resource Kotoshu downloads from the network is verified against a
6
+ cryptographic checksum before use. Releases are signed. Downloaded data
7
+ cannot silently change without detection.
8
+
9
+ ## Why
10
+
11
+ `Configuration::SCHEMA` has `auto_download: true` by default. The three
12
+ caches pull arbitrary bytes over HTTPS with zero integrity verification.
13
+ For CI/CD use (SARIF pipelines), supply-chain attacks (a compromised
14
+ GitHub repo swapping in a malicious dictionary), and reproducibility,
15
+ this is unacceptable. SLSA / supply-chain-attestation norms now expect
16
+ this baseline.
17
+
18
+ ## Tasks
19
+
20
+ 1. **Manifest format.** Define `manifest.json` at the root of each
21
+ content repo:
22
+ ```json
23
+ {
24
+ "version": 1,
25
+ "generated_at": "2026-06-25T10:00:00Z",
26
+ "resources": {
27
+ "en/spelling/index.dic": {
28
+ "size": 49568,
29
+ "sha256": "ab12...",
30
+ "language": "en",
31
+ "type": "spelling",
32
+ "license": "LGPL/MPL/GPL",
33
+ "source": "SCOWL"
34
+ },
35
+ }
36
+ }
37
+ }
38
+ ```
39
+ 2. **Manifest fetching.** The caches fetch `manifest.json` first, then
40
+ individual resources. Resources without a manifest entry are refused.
41
+ 3. **SHA-256 verification.** After download, recompute SHA-256 and
42
+ compare. Mismatch raises `Kotoshu::IntegrityError` with expected vs.
43
+ actual in the message.
44
+ 4. **Signing (stretch).** Sign `manifest.json` with a Kotoshu project
45
+ key (minisign or sigstore). The gem bundles the public key; CI
46
+ verifies the signature. Stretch goal — checksum-only is the v1
47
+ baseline.
48
+ 5. **Audit log.** Every download is logged to
49
+ `~/.kotoshu/audit.log` with timestamp, URL, size, SHA-256, source
50
+ manifest hash. Lets users inspect what was fetched.
51
+ 6. **Pin/lockfile support.** `Kotoshu.configure { |c| c.resource_pin =
52
+ "v1.2.0" }` resolves against a specific content repo tag instead of
53
+ `main`. Required for reproducible CI.
54
+ 7. **Rate limiting.** Respect GitHub's rate limits (60/hr unauthenticated,
55
+ 5000/hr with token). Add exponential backoff and a clear error when
56
+ rate-limited. Support `KOTOSHU_GITHUB_TOKEN` env var.
57
+ 8. **TLS pinning (optional).** Document that GitHub's CDN cert pin is
58
+ out of scope for v1 but the gem always verifies TLS chain.
59
+ 9. **Threat model doc.** Add `SECURITY.md` documenting: what we verify
60
+ (checksums), what we don't (signing is stretch), how to report a
61
+ compromised resource.
62
+
63
+ ## Acceptance criteria
64
+
65
+ - Tampering with a single byte in a cached file causes the next
66
+ `Kotoshu.check` to refuse to load it
67
+ - `Kotoshu.check(text, resource_pin: "v1.0.0")` resolves against
68
+ `dictionaries` repo tag `v1.0.0`
69
+ - `~/.kotoshu/audit.log` records every download
70
+ - CI runs without hitting GitHub rate limits on a fresh runner
71
+ - `SECURITY.md` exists and is linked from the README
72
+
73
+ ## Dependencies
74
+
75
+ - Blocks: `03-dynamic-download` (ResourceManager needs this to verify),
76
+ `11-release`
77
+ - Cross-repo: each content repo must ship `manifest.json`:
78
+ - `dictionaries/TODO.impl/01-manifest-checksums.md`
79
+ - `frequency-list-kelly/TODO.impl/02-data-validation.md`
80
+ - `models-fasttext-onnx/TODO.impl/03-manifest-checksums.md`
81
+
82
+ ## Out of scope
83
+
84
+ - Full SLSA Level 3 attestation (overkill for v1)
85
+ - Mirroring/CDN strategy (GitHub raw content is fine for v1)
86
+ - Encrypting cached files at rest (user's filesystem is their boundary)
87
+
88
+ ## Status
89
+
90
+ _Pending._
@@ -0,0 +1,98 @@
1
+ # 10 — Testing & CI
2
+
3
+ ## Goal
4
+
5
+ A test suite that meaningfully verifies behavior (not just that methods
6
+ are called) and a CI pipeline that prevents regressions across the
7
+ supported Ruby version and language matrix.
8
+
9
+ ## Why
10
+
11
+ Current state:
12
+ - `:network` specs are gated behind `NETWORK_TESTS=1`, meaning default
13
+ `bundle exec rspec` skips large chunks of behavior
14
+ - No CI workflow visible in the repo for the gem itself
15
+ - SimpleCov is configured but `minimum_coverage` is commented out
16
+ (`# TODO: Re-enable when coverage increases`)
17
+ - No performance regression gates
18
+ - Many root-level `test_*.rb` files bypass RSpec entirely
19
+
20
+ ## Tasks
21
+
22
+ 1. **CI workflow.** Add `.github/workflows/ci.yml`:
23
+ - Matrix: Ruby 3.1, 3.2, 3.3
24
+ - Job 1: `bundle exec rake` (default = spec + rubocop)
25
+ - Job 2 (nightly, separate): `NETWORK_TESTS=1 bundle exec rspec` —
26
+ uses cached resources to avoid re-downloading
27
+ - Job 3: `bundle exec rspec spec/performance/` — runs benchmarks,
28
+ fails on >20% regression vs. baseline committed in
29
+ `spec/performance/baseline.json`
30
+ 2. **Coverage gate.** Re-enable `minimum_coverage` once it's at a
31
+ meaningful number. Target: 80% line coverage of `lib/` excluding
32
+ `lib/kotoshu/cli/` (interactive UI is hard to test).
33
+ 3. **Spec hygiene.**
34
+ - No `double()` anywhere (project rule). Audit and replace with real
35
+ instances or `Struct`.
36
+ - No VCR / mocks of HTTP (commit `90aa886` already removed these;
37
+ ensure they don't come back). Network specs use a Sinatra fake
38
+ server fixture.
39
+ - Every spec under `spec/kotoshu/` mirrors `lib/kotoshu/` 1:1.
40
+ 4. **Property-based tests.** Expand `spec/properties/`:
41
+ - Tokenizer round-trip: every word a tokenizer extracts can be
42
+ re-inserted at its offset to reconstruct the original
43
+ - Suggestion idempotency: a correct word never appears in
44
+ suggestions for itself
45
+ - Hunspell fixture port: every Spylls test scenario is covered
46
+ 5. **Integration test matrix.** A single
47
+ `spec/integration/matrix_spec.rb` iterates each supported language
48
+ and runs 5 sample documents through the full pipeline (tokenize →
49
+ resolve → analyze → suggest → format). Tags: `:network` (downloads),
50
+ `:slow` (>5s). Lets us see at a glance which languages are healthy.
51
+ 6. **SARIF validation.** Spec that runs `check` and feeds the SARIF
52
+ output through the official SARIF validator gem; fail CI if invalid.
53
+ 7. **Move root scratch files.** `test_affix*.rb`, `test_onnx_*.rb`,
54
+ `test_vocab_setup.rb`, `test_combined_cache.rb`, `verify_onnx_all.rb`,
55
+ `deploy_onnx_all_languages.rb`, `monitor_onnx_progress.sh` — move
56
+ useful ones into `scripts/`, delete pure scratch (ask first per
57
+ global rule).
58
+ 8. **Conformance harness.** A script that runs Kotoshu against the
59
+ Hunspell reference test corpus and reports pass rate. Becomes the
60
+ source of truth for plan `01-hunspell-correctness` status.
61
+ 9. **Flake detection.** CI runs the full suite 3 times; any example
62
+ that fails non-deterministically is flagged.
63
+
64
+ ## Acceptance criteria
65
+
66
+ - PRs to `main` cannot merge if `bundle exec rake` fails
67
+ - Coverage report shows ≥ 80% on `lib/` (excluding CLI)
68
+ - `NETWORK_TESTS=1` job runs nightly, posts a summary of
69
+ language-matrix results to the repo's Actions tab
70
+ - No `double()` calls in the codebase (`grep -rn 'double(' spec/` is
71
+ empty)
72
+ - No `test_*.rb` files in the repo root
73
+
74
+ ## Dependencies
75
+
76
+ - None — can run in parallel with everything else
77
+ - Unlocks: confidence for `11-release`
78
+
79
+ ## Out of scope
80
+
81
+ - Fuzzing (out of scope for v1)
82
+ - Mutation testing (`mutant` gem) — stretch goal
83
+ - Stress/load testing of a future HTTP server
84
+
85
+ ## Status
86
+
87
+ _Pending._
88
+
89
+ ## Source
90
+
91
+ `docs/TDD_ITERATION_STRATEGY.md` (original dated 2025-01-29) is the
92
+ methodology reference: TDD + MDD cycle, walking skeleton, 10-week
93
+ build-out, Definition of Done per vertical slice. The build-out is
94
+ substantially executed (walking skeleton, core models, all dictionary
95
+ backends, suggestion strategies, Hunspell port all exist with specs).
96
+ Load-bearing rules from that doc — no doubles, test-behavior-not-
97
+ implementation, Strangler Fig for large refactors — live in
98
+ `~/.claude/CLAUDE.md` and `CLAUDE.md`, not duplicated here.
@@ -0,0 +1,28 @@
1
+ # Plan 100 — Full-feature language batch 3: the highest-audience remainder
2
+
3
+ ## Why
4
+ 20 languages are full-feature (gem module + keyboard layout + verified
5
+ specimens); the dictionaries manifest and the model registry carry ~35 more
6
+ with both resources. Every remaining language without a module leaves
7
+ speakers on the degraded path. The gaps are the big-audience ones: ar (~400M),
8
+ id (~200M), fa (~80M), he, bg, sr, hr, sk, sl, and more.
9
+
10
+ ## Work (gem repo only — the site flip rides a follow-up)
11
+ 1. Enumerate: languages present in BOTH the dictionaries manifest and the
12
+ models registry that lack a gem module. Rank by speaker count.
13
+ 2. For the top ~12 (expected: ar id fa he bg sr hr sk sl lt lv et — confirm
14
+ against data): create the language module (pattern of the 19 from PR
15
+ #123), keyboard layout (Arabic 101 for ar; Persian for fa; Hebrew for he;
16
+ Cyrillic/YUSI for bg/sr; QWERTZ/Slovak for the Latinate), AVAILABLE_LANGUAGES
17
+ extension, and specimen pairs verified against the STAGED dictionary via
18
+ the real engine (correct()/suggest() round trips) — the PR #123 discipline.
19
+ 3. RTL already wired (ar/id from batch 2); no tokenizer work needed.
20
+ 4. Suite green; rubocop clean.
21
+
22
+ ## Verification
23
+ One spec per module (specimen corrections engine-verified, layout smoke);
24
+ `kotoshu setup <lang>` + `check -l <lang>` exercised live for at least ar,
25
+ id, fa, he. Deliver the verified specimen table + sizes for the site flip.
26
+
27
+ ## Status
28
+ Executed 2026-09-07 — gem PR #151 merged. Twelve languages (ar id fa he bg sr hr sk sl lt lv et) full-feature; AVAILABLE_LANGUAGES 20→32; 49 specimens; RTL tokenizer fix for ar/fa/he. Site flip PR #27.
@@ -0,0 +1,32 @@
1
+ # Plan 101 — int4 tier experiment (models repo), eval-gated
2
+
3
+ ## Why
4
+ The tier recipe settled at int8 (rank_corr 0.9999, near-lossless) after SVD
5
+ failed its gates. int4 halves the bytes again — mini ~1.5 MB, fluency ~8 MB —
6
+ which changes browser adoption math for the wasm playground. It was named in
7
+ the original kotoshu-rs TODO (B1) but never run through the gates.
8
+
9
+ ## Work (models repo)
10
+ 1. Quantization pass: per-row int4 (packed nibbles) + fp16 scale, over the
11
+ same vocab cuts as fluency/mini.
12
+ 2. The existing eval gates unchanged and unweakened: rank correlation vs
13
+ full float32 on the keyboard-aware eval; accept only if rank_corr within
14
+ 0.001 of int8's and top-1 agreement holds on the probe-limited languages.
15
+ 3. If gates pass: registry v1.4.0 adding the int4 variants as NEW resources
16
+ (`{lang}/mini4`? or a field) — additive, never replacing; mirror with
17
+ ACAO:* like v1.2.1+. If gates fail: record the numbers as the verdict
18
+ (the SVD precedent — measured, not assumed).
19
+ 4. ONNX reader side (kotoshu-rs) only if the gates pass.
20
+
21
+ ## Verification
22
+ Gates table per language; byte sizes; the release registry byte-identical to
23
+ the committed one; no existing tier URLs change.
24
+
25
+ ## Status
26
+ Executed 2026-09-07 — REJECTED on data (models PR #17, merged). int4-per-row
27
+ (one fp16 scale, packed nibbles) over 8 languages: rank_corr deltas -0.0146
28
+ to -0.0199 vs int8 (15-20x outside the 0.001 window), top-1 fails on 8/8
29
+ fluency4 and 2/8 mini4. Sizes halve (mini4 1.52 MB, fluency4 7.60 MB) but
30
+ accuracy does not hold. With the earlier group-128/64/32 sweep the scale
31
+ ladder is measured end to end — int8-per-row stands final. Registry
32
+ untouched; evidence in models eval/reports/.
@@ -0,0 +1,28 @@
1
+ # Plan 102 — Language detection end to end: registry mirror, wasm, playground
2
+
3
+ ## Why
4
+ The gem detects document language (server /v1/detect) using its own detection
5
+ model — but the registry mirrors none of it, the wasm surface has no detect
6
+ API, and the playground makes a user pick a language dropdown blind. A
7
+ writer pasting text should be told what it is. The FastText LID model is
8
+ tiny (~1 MB) and the mirror pattern is proven (v1.2.1+).
9
+
10
+ ## Work
11
+ 1. Models repo: add the LID model as a registry resource (converted like the
12
+ tiers, int8-quantized if gates hold vs the float32 labels), mirrored with
13
+ ACAO:*; registry v1.4.x alongside plan 101 (coordinate the version).
14
+ 2. kotoshu-rs: `detect_language(bytes, vocab_bytes) -> {code, score}` on the
15
+ model surface beside loadModel/rerank; wasm export `detectLanguage`;
16
+ tests against the gem's detection outputs on a fixed multilingual corpus.
17
+ 3. Playground (rides the site follow-up with plans 99/100 flips): on load
18
+ and on demand ("detect" affordance beside the lang select), load the LID
19
+ model (~1 MB, cached like the semantic tier) and set/select the language,
20
+ showing the detected code + confidence. Never auto-switch a user's
21
+ explicit choice silently — propose it.
22
+
23
+ ## Verification
24
+ Registry mirror ACAO + size checks; wasm parity vs gem detect on the corpus;
25
+ playground verified with multilingual paste (en/de/ru/ja/ar).
26
+
27
+ ## Status
28
+ Executed 2026-09-07 (core) — models PR #18 (lid-176, registry v1.4.0) + kotoshu-rs PR #25 (loadLid/detectLanguage). Playground propose-UI still pending a wasm cut that ships the export.
@@ -0,0 +1,27 @@
1
+ # Plan 103 — OOV bucket-table artifacts: model-generated candidates for the last mile
2
+
3
+ ## Why
4
+ semanticSuggest embeds an OOV word via subword n-grams PRESENT IN THE VOCAB;
5
+ a typo whose n-grams never appear (Teh — only n-gram "teh") embeds nothing
6
+ and returns empty. fastText's real OOV path hashes n-grams into buckets the
7
+ training wrote to; exporting the bucket table closes that gap so the model,
8
+ not just the Damerau sweep, can reach the intended word. Recorded as the
9
+ honest gap in kotoshu-rs oov.rs docs and PR #18's report.
10
+
11
+ ## Work (models repo, then kotoshu-rs)
12
+ 1. Export the bucket-table rows the FastText training produced (hash bucket
13
+ → vector) for each tier alongside the vocab; size audit first — the
14
+ bucket table may exceed the tier itself; if so, gate to the top-K buckets
15
+ by usage and measure the quality delta honestly.
16
+ 2. kotoshu-rs: extend the int8 reader to consult bucket rows for n-grams
17
+ absent from vocab (feature-gated like `model`); semantic_neighbors then
18
+ covers bucket-backed OOV.
19
+ 3. Gates: existing rerank conformance must not move (frozen); new specs for
20
+ bucket-backed neighbors (Teh→the expected); eval on the rerank probe set.
21
+
22
+ ## Verification
23
+ Bucket-backed neighbor specs; size deltas recorded; rerank conformance 0
24
+ change; a 0.4.x wasm release only after both engines agree.
25
+
26
+ ## Status
27
+ Executed 2026-09-07 — models PR #20 (en/de buckets) + kotoshu-rs PR #26 (bucket-backed OOV). Teh→the via buckets. Release assets + wasm 0.4.0 pin still owner/coordinated cut. (sequenced AFTER plan 101 — same repo, one worktree at a time)