kotoshu 1.0.2-x86_64-linux

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (458) hide show
  1. checksums.yaml +7 -0
  2. data/.kotoshu-baseline.json +1 -0
  3. data/.pre-commit-hooks.yaml +18 -0
  4. data/.rspec +3 -0
  5. data/.rubocop.yml +351 -0
  6. data/.yardopts +8 -0
  7. data/CHANGELOG.md +813 -0
  8. data/CLAUDE.md +176 -0
  9. data/CODE_OF_CONDUCT.md +132 -0
  10. data/Cargo.lock +359 -0
  11. data/Cargo.toml +15 -0
  12. data/LICENSE +31 -0
  13. data/README.adoc +1292 -0
  14. data/Rakefile +150 -0
  15. data/SECURITY.md +93 -0
  16. data/TODO.impl/00-cut-0.1.md +121 -0
  17. data/TODO.impl/00-cut-0.2.md +175 -0
  18. data/TODO.impl/00-vision.md +281 -0
  19. data/TODO.impl/01-hunspell-correctness.md +111 -0
  20. data/TODO.impl/02-cli-unification.md +74 -0
  21. data/TODO.impl/03-dynamic-download.md +107 -0
  22. data/TODO.impl/04-language-modules.md +89 -0
  23. data/TODO.impl/05-semantic-path.md +83 -0
  24. data/TODO.impl/06-cjk-support.md +105 -0
  25. data/TODO.impl/07-rtl-support.md +71 -0
  26. data/TODO.impl/08-grammar-engine.md +106 -0
  27. data/TODO.impl/09-integrity-security.md +90 -0
  28. data/TODO.impl/10-testing-ci.md +98 -0
  29. data/TODO.impl/100-full-feature-batch3.md +28 -0
  30. data/TODO.impl/101-int4-tier-experiment.md +32 -0
  31. data/TODO.impl/102-lid-chain.md +28 -0
  32. data/TODO.impl/103-oov-bucket-tables.md +27 -0
  33. data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
  34. data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
  35. data/TODO.impl/106-native-lid-detection.md +36 -0
  36. data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
  37. data/TODO.impl/108-ko-ne-full-feature.md +37 -0
  38. data/TODO.impl/109-ten-audit.md +26 -0
  39. data/TODO.impl/11-release-v1.md +95 -0
  40. data/TODO.impl/110-sr-latn-and-polish.md +33 -0
  41. data/TODO.impl/111-embedding-bakeoff.md +33 -0
  42. data/TODO.impl/112-deployability-and-gating.md +26 -0
  43. data/TODO.impl/113-language-packs.md +26 -0
  44. data/TODO.impl/114-typo-biencoder-thread.md +33 -0
  45. data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
  46. data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
  47. data/TODO.impl/117-deterministic-frequency-tiers.md +110 -0
  48. data/TODO.impl/118-server-gem-floor.md +33 -0
  49. data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
  50. data/TODO.impl/120-setup-metadata-refresh.md +35 -0
  51. data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
  52. data/TODO.impl/122-pack-compression-kpk2.md +66 -0
  53. data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
  54. data/TODO.impl/124-owner-residue.md +19 -0
  55. data/TODO.impl/125-consumer-floor-guards.md +43 -0
  56. data/TODO.impl/126-tier-error-budgets.md +33 -0
  57. data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
  58. data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
  59. data/TODO.impl/129-pt-fr-corpora.md +23 -0
  60. data/TODO.impl/130-news-truth-0912.md +23 -0
  61. data/TODO.impl/131-hybrid-productization.md +42 -0
  62. data/TODO.impl/132-techniques-page-and-prose.md +27 -0
  63. data/TODO.impl/133-rust-first-distribution.md +68 -0
  64. data/TODO.impl/20-encapsulation-cleanup.md +47 -0
  65. data/TODO.impl/21-autoload-migration.md +64 -0
  66. data/TODO.impl/22-url-pin-single-source-of-truth.md +120 -0
  67. data/TODO.impl/23-fix-url-defaults.md +57 -0
  68. data/TODO.impl/24-cli-auto-setup-prompt.md +76 -0
  69. data/TODO.impl/25-soft-onnxruntime-require.md +75 -0
  70. data/TODO.impl/26-onnx-vocab-json-generation.md +96 -0
  71. data/TODO.impl/27-end-to-end-smoke-spec.md +113 -0
  72. data/TODO.impl/28-readme-quickstart-verification.md +68 -0
  73. data/TODO.impl/29-status-command.md +84 -0
  74. data/TODO.impl/30-language-auto-detection.md +74 -0
  75. data/TODO.impl/31-download-progress.md +58 -0
  76. data/TODO.impl/32-sarif-output-verification.md +72 -0
  77. data/TODO.impl/33-audit-log-rotation.md +54 -0
  78. data/TODO.impl/34-cache-eviction.md +59 -0
  79. data/TODO.impl/35-shell-completion.md +61 -0
  80. data/TODO.impl/36-cleanup-regressions-0.3.1.md +134 -0
  81. data/TODO.impl/37-hunspell-correctness-tier2.md +203 -0
  82. data/TODO.impl/38-onnx-semantic-gating.md +218 -0
  83. data/TODO.impl/39-tier3-and-beyond.md +232 -0
  84. data/TODO.impl/40-spec-drift-cleanup.md +134 -0
  85. data/TODO.impl/50-structure-aware-document-api.md +194 -0
  86. data/TODO.impl/51-grammar-engine-expansion.md +75 -0
  87. data/TODO.impl/52-multi-language-documents.md +95 -0
  88. data/TODO.impl/53-document-plugin-architecture.md +94 -0
  89. data/TODO.impl/54-cjk-support.md +117 -0
  90. data/TODO.impl/55-rtl-support.md +109 -0
  91. data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
  92. data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
  93. data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
  94. data/TODO.impl/59-yard-api-documentation.md +75 -0
  95. data/TODO.impl/60-lsp-server.md +174 -0
  96. data/TODO.impl/61-editor-ecosystem.md +167 -0
  97. data/TODO.impl/62-framework-integrations.md +159 -0
  98. data/TODO.impl/63-web-playground-wasm.md +186 -0
  99. data/TODO.impl/64-http-api-and-sdks.md +191 -0
  100. data/TODO.impl/65-universal-kotoshu.md +251 -0
  101. data/TODO.impl/66-kotoshu-core.md +169 -0
  102. data/TODO.impl/67-kotoshu-rs-and-access-libraries.md +137 -0
  103. data/TODO.impl/68-sota-adoption.md +152 -0
  104. data/TODO.impl/69-sota-typo-eval.md +74 -0
  105. data/TODO.impl/70-sota-confidence-cascade.md +59 -0
  106. data/TODO.impl/71-sota-nested-model.md +63 -0
  107. data/TODO.impl/72-model2vec-evaluation.md +49 -0
  108. data/TODO.impl/73-rl-reranker.md +42 -0
  109. data/TODO.impl/74-distilled-reranker.md +36 -0
  110. data/TODO.impl/75-llm-evaluator.md +35 -0
  111. data/TODO.impl/76-optional-llm-backend.md +37 -0
  112. data/TODO.impl/77-model-coverage-expansion.md +96 -0
  113. data/TODO.impl/78-in-browser-playground.md +69 -0
  114. data/TODO.impl/79-news-and-releases.md +60 -0
  115. data/TODO.impl/80-ecosystem-docs.md +56 -0
  116. data/TODO.impl/81-editor-reach.md +68 -0
  117. data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
  118. data/TODO.impl/83-models-batch-2.md +56 -0
  119. data/TODO.impl/84-language-modules-wave2.md +75 -0
  120. data/TODO.impl/85-wasm-model-api.md +48 -0
  121. data/TODO.impl/86-site-truth-wave2.md +45 -0
  122. data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
  123. data/TODO.impl/88-cli-directory-mode.md +54 -0
  124. data/TODO.impl/89-framework-integrations-exec.md +59 -0
  125. data/TODO.impl/90-gem-correctness-residues.md +44 -0
  126. data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
  127. data/TODO.impl/92-tier-mirror-cors.md +88 -0
  128. data/TODO.impl/93-python-wheels.md +32 -0
  129. data/TODO.impl/94-site-integrations-docs.md +35 -0
  130. data/TODO.impl/95-editor-action-polish.md +27 -0
  131. data/TODO.impl/96-server-semantic-models.md +50 -0
  132. data/TODO.impl/97-yard-execution.md +73 -0
  133. data/TODO.impl/98-campaign-cleanup.md +78 -0
  134. data/TODO.impl/99-site-truth-performance-docs.md +31 -0
  135. data/conformance/vectors.jsonl +2630 -0
  136. data/data/en/grammar/rules.yaml +851 -0
  137. data/docs/STABILITY-1.0-AUDIT.md +251 -0
  138. data/docs/STABILITY-1.0.md +137 -0
  139. data/docs/cache-architecture.md +502 -0
  140. data/docs/verification-evidence/correct3/pr_body.md +37 -0
  141. data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
  142. data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
  143. data/docs/verification-evidence/correct3/sv.aff +856 -0
  144. data/docs/verification-evidence/correct3/sv.dic +153717 -0
  145. data/examples/01_basic_word_checking.rb +38 -0
  146. data/examples/02_text_document_checking.rb +77 -0
  147. data/examples/03_dictionary_backends.rb +137 -0
  148. data/examples/04_trie_data_structure.rb +146 -0
  149. data/examples/05_suggestion_algorithms.rb +239 -0
  150. data/examples/06_configuration_advanced.rb +287 -0
  151. data/examples/07_multi_language_dictionaries.rb +278 -0
  152. data/exe/kotoshu +6 -0
  153. data/lib/kotoshu/algorithms/capitalization.rb +276 -0
  154. data/lib/kotoshu/algorithms/edit_distance.rb +128 -0
  155. data/lib/kotoshu/algorithms/lookup/affix_form.rb +132 -0
  156. data/lib/kotoshu/algorithms/lookup/compound_checks.rb +209 -0
  157. data/lib/kotoshu/algorithms/lookup/compound_form.rb +50 -0
  158. data/lib/kotoshu/algorithms/lookup.rb +841 -0
  159. data/lib/kotoshu/algorithms/ngram_suggest.rb +367 -0
  160. data/lib/kotoshu/algorithms/permutations.rb +283 -0
  161. data/lib/kotoshu/algorithms/phonet_suggest.rb +162 -0
  162. data/lib/kotoshu/algorithms/soundex.rb +92 -0
  163. data/lib/kotoshu/algorithms/suggest.rb +794 -0
  164. data/lib/kotoshu/algorithms.rb +23 -0
  165. data/lib/kotoshu/analyzers/semantic_analyzer.rb +405 -0
  166. data/lib/kotoshu/analyzers.rb +8 -0
  167. data/lib/kotoshu/baseline/application.rb +18 -0
  168. data/lib/kotoshu/baseline/entry.rb +33 -0
  169. data/lib/kotoshu/baseline/store.rb +175 -0
  170. data/lib/kotoshu/baseline.rb +20 -0
  171. data/lib/kotoshu/cache/base_cache.rb +778 -0
  172. data/lib/kotoshu/cache/cache.rb +91 -0
  173. data/lib/kotoshu/cache/eviction_policy.rb +66 -0
  174. data/lib/kotoshu/cache/frequency_cache.rb +283 -0
  175. data/lib/kotoshu/cache/language_cache.rb +511 -0
  176. data/lib/kotoshu/cache/lookup_cache.rb +166 -0
  177. data/lib/kotoshu/cache/model_cache.rb +1147 -0
  178. data/lib/kotoshu/cache/model_registry.rb +109 -0
  179. data/lib/kotoshu/cache/staged_languages.rb +45 -0
  180. data/lib/kotoshu/cache/suggestion_cache.rb +113 -0
  181. data/lib/kotoshu/cache.rb +49 -0
  182. data/lib/kotoshu/cli/auto_setup.rb +71 -0
  183. data/lib/kotoshu/cli/baseline_command.rb +119 -0
  184. data/lib/kotoshu/cli/batch_reporter.rb +321 -0
  185. data/lib/kotoshu/cli/cache_command.rb +282 -0
  186. data/lib/kotoshu/cli/completions_command.rb +210 -0
  187. data/lib/kotoshu/cli/dict_command.rb +64 -0
  188. data/lib/kotoshu/cli/directory_check.rb +146 -0
  189. data/lib/kotoshu/cli/directory_walker.rb +132 -0
  190. data/lib/kotoshu/cli/display_formatter.rb +429 -0
  191. data/lib/kotoshu/cli/errors.rb +36 -0
  192. data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
  193. data/lib/kotoshu/cli/interactive_reviewer.rb +314 -0
  194. data/lib/kotoshu/cli/language_resolver.rb +91 -0
  195. data/lib/kotoshu/cli/navigation_manager.rb +273 -0
  196. data/lib/kotoshu/cli/personal_command.rb +113 -0
  197. data/lib/kotoshu/cli/progress_reporter.rb +114 -0
  198. data/lib/kotoshu/cli/status_report.rb +130 -0
  199. data/lib/kotoshu/cli.rb +880 -0
  200. data/lib/kotoshu/components/passthrough_spell_checker.rb +70 -0
  201. data/lib/kotoshu/components/pos_tagger.rb +98 -0
  202. data/lib/kotoshu/components/spell_checker.rb +73 -0
  203. data/lib/kotoshu/components/synthesizer.rb +60 -0
  204. data/lib/kotoshu/components/tokenizer.rb +58 -0
  205. data/lib/kotoshu/components/whitespace_tokenizer.rb +94 -0
  206. data/lib/kotoshu/components.rb +13 -0
  207. data/lib/kotoshu/configuration/builder.rb +207 -0
  208. data/lib/kotoshu/configuration/resolver.rb +124 -0
  209. data/lib/kotoshu/configuration.rb +829 -0
  210. data/lib/kotoshu/conformance_exporter.rb +256 -0
  211. data/lib/kotoshu/conformance_runner.rb +176 -0
  212. data/lib/kotoshu/core/exceptions.rb +186 -0
  213. data/lib/kotoshu/core/indexed_dictionary.rb +293 -0
  214. data/lib/kotoshu/core/models/affix_rule.rb +262 -0
  215. data/lib/kotoshu/core/models/result/document_result.rb +132 -0
  216. data/lib/kotoshu/core/models/result/word_result.rb +131 -0
  217. data/lib/kotoshu/core/models/word.rb +131 -0
  218. data/lib/kotoshu/core/trie/builder.rb +119 -0
  219. data/lib/kotoshu/core/trie/node.rb +94 -0
  220. data/lib/kotoshu/core/trie/trie.rb +249 -0
  221. data/lib/kotoshu/core.rb +28 -0
  222. data/lib/kotoshu/data/common_words/de.yml +1800 -0
  223. data/lib/kotoshu/data/common_words/en.yml +1215 -0
  224. data/lib/kotoshu/data/common_words/es.yml +750 -0
  225. data/lib/kotoshu/data/common_words/fr.yml +1015 -0
  226. data/lib/kotoshu/data/common_words/pt.yml +870 -0
  227. data/lib/kotoshu/data/common_words/ru.yml +484 -0
  228. data/lib/kotoshu/data/common_words_loader.rb +141 -0
  229. data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
  230. data/lib/kotoshu/data/frozen_kelly.rb +38 -0
  231. data/lib/kotoshu/data.rb +8 -0
  232. data/lib/kotoshu/data_structures/bloom_filter.rb +188 -0
  233. data/lib/kotoshu/data_structures.rb +8 -0
  234. data/lib/kotoshu/debug_logger.rb +146 -0
  235. data/lib/kotoshu/debug_mode.rb +136 -0
  236. data/lib/kotoshu/defaults.rb +84 -0
  237. data/lib/kotoshu/dictionaries/catalog.rb +814 -0
  238. data/lib/kotoshu/dictionaries.rb +8 -0
  239. data/lib/kotoshu/dictionary/base.rb +324 -0
  240. data/lib/kotoshu/dictionary/cspell.rb +243 -0
  241. data/lib/kotoshu/dictionary/custom.rb +225 -0
  242. data/lib/kotoshu/dictionary/hunspell.rb +520 -0
  243. data/lib/kotoshu/dictionary/plain_text.rb +302 -0
  244. data/lib/kotoshu/dictionary/repository.rb +246 -0
  245. data/lib/kotoshu/dictionary/unified.rb +236 -0
  246. data/lib/kotoshu/dictionary/unix_words.rb +218 -0
  247. data/lib/kotoshu/dictionary.rb +15 -0
  248. data/lib/kotoshu/documents/document.rb +120 -0
  249. data/lib/kotoshu/documents/plain_text_document.rb +73 -0
  250. data/lib/kotoshu/documents/source_position.rb +45 -0
  251. data/lib/kotoshu/documents/source_range.rb +76 -0
  252. data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
  253. data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
  254. data/lib/kotoshu/documents/suppressions.rb +44 -0
  255. data/lib/kotoshu/documents/text_node.rb +49 -0
  256. data/lib/kotoshu/documents.rb +178 -0
  257. data/lib/kotoshu/embeddings/embedding_pipeline.rb +238 -0
  258. data/lib/kotoshu/embeddings/lru_cache.rb +244 -0
  259. data/lib/kotoshu/embeddings/onnx_runtime_model.rb +411 -0
  260. data/lib/kotoshu/embeddings/protocol.rb +96 -0
  261. data/lib/kotoshu/embeddings/protocols.rb +23 -0
  262. data/lib/kotoshu/embeddings/registry.rb +184 -0
  263. data/lib/kotoshu/embeddings/search.rb +204 -0
  264. data/lib/kotoshu/embeddings/similarity_engine.rb +278 -0
  265. data/lib/kotoshu/embeddings/similarity_search.rb +328 -0
  266. data/lib/kotoshu/embeddings/vocabulary.rb +306 -0
  267. data/lib/kotoshu/embeddings.rb +87 -0
  268. data/lib/kotoshu/fluent_checker.rb +94 -0
  269. data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +48 -0
  270. data/lib/kotoshu/grammar/pattern_matchers/double_negative_matcher.rb +179 -0
  271. data/lib/kotoshu/grammar/pattern_matchers/phrase_matcher.rb +86 -0
  272. data/lib/kotoshu/grammar/pattern_matchers/possessive_context_matcher.rb +75 -0
  273. data/lib/kotoshu/grammar/pattern_matchers/possessive_contraction_matcher.rb +87 -0
  274. data/lib/kotoshu/grammar/pattern_matchers/sentence_start_matcher.rb +84 -0
  275. data/lib/kotoshu/grammar/pattern_matchers/vowel_sound_matcher.rb +81 -0
  276. data/lib/kotoshu/grammar/pattern_matchers/word_list_matcher.rb +66 -0
  277. data/lib/kotoshu/grammar/rule.rb +99 -0
  278. data/lib/kotoshu/grammar/rule_engine.rb +96 -0
  279. data/lib/kotoshu/grammar/rule_loader.rb +30 -0
  280. data/lib/kotoshu/grammar.rb +24 -0
  281. data/lib/kotoshu/integrity/audit_log.rb +167 -0
  282. data/lib/kotoshu/integrity/manifest.rb +116 -0
  283. data/lib/kotoshu/integrity/net_http.rb +51 -0
  284. data/lib/kotoshu/integrity/rotation_policy.rb +88 -0
  285. data/lib/kotoshu/integrity.rb +26 -0
  286. data/lib/kotoshu/jekyll.rb +101 -0
  287. data/lib/kotoshu/keyboard/layout.rb +116 -0
  288. data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
  289. data/lib/kotoshu/keyboard/layouts/azerty.rb +55 -0
  290. data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
  291. data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
  292. data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
  293. data/lib/kotoshu/keyboard/layouts/dvorak.rb +54 -0
  294. data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
  295. data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
  296. data/lib/kotoshu/keyboard/layouts/hrsl.rb +94 -0
  297. data/lib/kotoshu/keyboard/layouts/jcuken.rb +57 -0
  298. data/lib/kotoshu/keyboard/layouts/latin.rb +266 -0
  299. data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
  300. data/lib/kotoshu/keyboard/layouts/qwerty.rb +52 -0
  301. data/lib/kotoshu/keyboard/layouts/qwertz.rb +55 -0
  302. data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
  303. data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
  304. data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
  305. data/lib/kotoshu/keyboard/registry.rb +201 -0
  306. data/lib/kotoshu/keyboard.rb +96 -0
  307. data/lib/kotoshu/kotoshu_native.so +0 -0
  308. data/lib/kotoshu/language/detector.rb +242 -0
  309. data/lib/kotoshu/language/identifier.rb +376 -0
  310. data/lib/kotoshu/language/languages/base.rb +271 -0
  311. data/lib/kotoshu/language/lid_detector.rb +188 -0
  312. data/lib/kotoshu/language/normalizer/arabic.rb +113 -0
  313. data/lib/kotoshu/language/normalizer/base.rb +133 -0
  314. data/lib/kotoshu/language/normalizer/greek.rb +63 -0
  315. data/lib/kotoshu/language/normalizer/hebrew.rb +72 -0
  316. data/lib/kotoshu/language/normalizer/persian.rb +36 -0
  317. data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
  318. data/lib/kotoshu/language/registry.rb +209 -0
  319. data/lib/kotoshu/language/resources/ar/common_words.txt +6753 -0
  320. data/lib/kotoshu/language/resources/ar/confusion_sets.txt +11 -0
  321. data/lib/kotoshu/language/resources/de/common_words.txt +10003 -0
  322. data/lib/kotoshu/language/resources/de/confusion_sets.txt +246 -0
  323. data/lib/kotoshu/language/resources/en/common_words.txt +9979 -0
  324. data/lib/kotoshu/language/resources/en/confusion_sets.txt +871 -0
  325. data/lib/kotoshu/language/resources/es/common_words.txt +9992 -0
  326. data/lib/kotoshu/language/resources/es/confusion_sets.txt +17 -0
  327. data/lib/kotoshu/language/resources/fr/common_words.txt +9993 -0
  328. data/lib/kotoshu/language/resources/fr/confusion_sets.txt +76 -0
  329. data/lib/kotoshu/language/resources/pt/common_words.txt +9977 -0
  330. data/lib/kotoshu/language/resources/pt/confusion_sets.txt +18 -0
  331. data/lib/kotoshu/language/resources/ru/common_words.txt +9951 -0
  332. data/lib/kotoshu/language/resources/ru/confusion_sets.txt +5 -0
  333. data/lib/kotoshu/language/script.rb +128 -0
  334. data/lib/kotoshu/language/segmenter.rb +122 -0
  335. data/lib/kotoshu/language/suika.rb +38 -0
  336. data/lib/kotoshu/language/tokenizer/base.rb +187 -0
  337. data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
  338. data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
  339. data/lib/kotoshu/language/tokenizer/french_tokenizer.rb +172 -0
  340. data/lib/kotoshu/language/tokenizer/german_tokenizer.rb +41 -0
  341. data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
  342. data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
  343. data/lib/kotoshu/language/tokenizer/japanese_tokenizer.rb +55 -0
  344. data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +153 -0
  345. data/lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb +160 -0
  346. data/lib/kotoshu/language/tokenizer/russian_tokenizer.rb +93 -0
  347. data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
  348. data/lib/kotoshu/language/tokenizer/spanish_tokenizer.rb +122 -0
  349. data/lib/kotoshu/language.rb +127 -0
  350. data/lib/kotoshu/languages/ar/language.rb +98 -0
  351. data/lib/kotoshu/languages/bg/language.rb +37 -0
  352. data/lib/kotoshu/languages/ca/language.rb +22 -0
  353. data/lib/kotoshu/languages/cs/language.rb +22 -0
  354. data/lib/kotoshu/languages/da/language.rb +22 -0
  355. data/lib/kotoshu/languages/de/language.rb +554 -0
  356. data/lib/kotoshu/languages/el/language.rb +50 -0
  357. data/lib/kotoshu/languages/en/language.rb +470 -0
  358. data/lib/kotoshu/languages/es/language.rb +476 -0
  359. data/lib/kotoshu/languages/et/language.rb +24 -0
  360. data/lib/kotoshu/languages/fa/language.rb +91 -0
  361. data/lib/kotoshu/languages/fr/language.rb +505 -0
  362. data/lib/kotoshu/languages/he/language.rb +98 -0
  363. data/lib/kotoshu/languages/hr/language.rb +24 -0
  364. data/lib/kotoshu/languages/hu/language.rb +22 -0
  365. data/lib/kotoshu/languages/id/language.rb +25 -0
  366. data/lib/kotoshu/languages/it/language.rb +22 -0
  367. data/lib/kotoshu/languages/ja/language.rb +476 -0
  368. data/lib/kotoshu/languages/ko/language.rb +56 -0
  369. data/lib/kotoshu/languages/latin_base.rb +53 -0
  370. data/lib/kotoshu/languages/lt/language.rb +24 -0
  371. data/lib/kotoshu/languages/lv/language.rb +24 -0
  372. data/lib/kotoshu/languages/nb/language.rb +25 -0
  373. data/lib/kotoshu/languages/ne/language.rb +60 -0
  374. data/lib/kotoshu/languages/nl/language.rb +22 -0
  375. data/lib/kotoshu/languages/nn/language.rb +26 -0
  376. data/lib/kotoshu/languages/pl/language.rb +22 -0
  377. data/lib/kotoshu/languages/pt/language.rb +439 -0
  378. data/lib/kotoshu/languages/ro/language.rb +22 -0
  379. data/lib/kotoshu/languages/ru/language.rb +420 -0
  380. data/lib/kotoshu/languages/sk/language.rb +25 -0
  381. data/lib/kotoshu/languages/sl/language.rb +25 -0
  382. data/lib/kotoshu/languages/sr/language.rb +39 -0
  383. data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
  384. data/lib/kotoshu/languages/sv/language.rb +22 -0
  385. data/lib/kotoshu/languages/tr/language.rb +36 -0
  386. data/lib/kotoshu/languages/uk/language.rb +38 -0
  387. data/lib/kotoshu/languages/vi/language.rb +22 -0
  388. data/lib/kotoshu/languages.rb +64 -0
  389. data/lib/kotoshu/metrics_collector.rb +238 -0
  390. data/lib/kotoshu/metrics_module.rb +121 -0
  391. data/lib/kotoshu/models/context.rb +119 -0
  392. data/lib/kotoshu/models/embedding_model.rb +181 -0
  393. data/lib/kotoshu/models/fasttext_model.rb +212 -0
  394. data/lib/kotoshu/models/nearest_neighbor.rb +96 -0
  395. data/lib/kotoshu/models/onnx_model.rb +330 -0
  396. data/lib/kotoshu/models/semantic_error.rb +204 -0
  397. data/lib/kotoshu/models/suggestion.rb +106 -0
  398. data/lib/kotoshu/models/word_embedding.rb +97 -0
  399. data/lib/kotoshu/models.rb +27 -0
  400. data/lib/kotoshu/multi_language_checker.rb +127 -0
  401. data/lib/kotoshu/native.rb +83 -0
  402. data/lib/kotoshu/native_backend.rb +131 -0
  403. data/lib/kotoshu/paths.rb +79 -0
  404. data/lib/kotoshu/personal_dictionary.rb +105 -0
  405. data/lib/kotoshu/plugins/plugin.rb +61 -0
  406. data/lib/kotoshu/plugins/registry.rb +118 -0
  407. data/lib/kotoshu/plugins.rb +9 -0
  408. data/lib/kotoshu/project_config.rb +76 -0
  409. data/lib/kotoshu/readers/aff_data.rb +530 -0
  410. data/lib/kotoshu/readers/aff_reader.rb +393 -0
  411. data/lib/kotoshu/readers/condition_checker.rb +125 -0
  412. data/lib/kotoshu/readers/dic_reader.rb +208 -0
  413. data/lib/kotoshu/readers/file_reader.rb +377 -0
  414. data/lib/kotoshu/readers/lookup_builder.rb +584 -0
  415. data/lib/kotoshu/readers/readers.rb +17 -0
  416. data/lib/kotoshu/readers.rb +25 -0
  417. data/lib/kotoshu/resource_bundle.rb +55 -0
  418. data/lib/kotoshu/resource_manager.rb +600 -0
  419. data/lib/kotoshu/results/result.rb +165 -0
  420. data/lib/kotoshu/results.rb +8 -0
  421. data/lib/kotoshu/rspec.rb +163 -0
  422. data/lib/kotoshu/scripts/fasttext_to_onnx.py +275 -0
  423. data/lib/kotoshu/source_registry.rb +104 -0
  424. data/lib/kotoshu/spellchecker/parallel_checker.rb +90 -0
  425. data/lib/kotoshu/spellchecker.rb +406 -0
  426. data/lib/kotoshu/string_metrics.rb +153 -0
  427. data/lib/kotoshu/suggestions/context.rb +43 -0
  428. data/lib/kotoshu/suggestions/frequency_provider.rb +80 -0
  429. data/lib/kotoshu/suggestions/generator.rb +169 -0
  430. data/lib/kotoshu/suggestions/pipeline.rb +135 -0
  431. data/lib/kotoshu/suggestions/semantic_cascade.rb +73 -0
  432. data/lib/kotoshu/suggestions/strategies/base_strategy.rb +315 -0
  433. data/lib/kotoshu/suggestions/strategies/composite_strategy.rb +197 -0
  434. data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +631 -0
  435. data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +207 -0
  436. data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +165 -0
  437. data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +284 -0
  438. data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +317 -0
  439. data/lib/kotoshu/suggestions/strategies/symspell_strategy.rb +270 -0
  440. data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
  441. data/lib/kotoshu/suggestions/suggestion.rb +154 -0
  442. data/lib/kotoshu/suggestions/suggestion_set.rb +270 -0
  443. data/lib/kotoshu/suggestions/sweep_index.rb +137 -0
  444. data/lib/kotoshu/suggestions.rb +29 -0
  445. data/lib/kotoshu/tasks/check_task.rb +104 -0
  446. data/lib/kotoshu/tasks.rb +11 -0
  447. data/lib/kotoshu/validators/spelling_validator.rb +101 -0
  448. data/lib/kotoshu/validators.rb +9 -0
  449. data/lib/kotoshu/version.rb +5 -0
  450. data/lib/kotoshu.rb +538 -0
  451. data/rakelib/compile.rake +10 -0
  452. data/script/validate_all_dictionaries.rb +444 -0
  453. data/scripts/build_native_gem.rb +45 -0
  454. data/scripts/extract_eval_grids.rb +37 -0
  455. data/scripts/generate_frozen_tiers.rb +44 -0
  456. data/sig/kotoshu.rbs +435 -0
  457. data/test_oop.rb +79 -0
  458. metadata +546 -0
@@ -0,0 +1,232 @@
1
+ # 39 — Tier 3 and Beyond
2
+
3
+ ## Goal
4
+
5
+ The long-form backlog. Each item here is a *capability* rather than a
6
+ bug fix. Items are roughly prioritized; the order is not a commitment.
7
+
8
+ This file exists so we don't lose institutional memory between
9
+ releases. When an item becomes the next priority, promote it to its
10
+ own `TODO.impl/{n}-{name}.md` file with phases and acceptance criteria,
11
+ then delete the entry here.
12
+
13
+ ## T3 — Capabilities
14
+
15
+ ### T3.1 — CJK support (promote from `TODO.impl/06-cjk-support.md`)
16
+
17
+ Status: outline only. Real work needs:
18
+
19
+ - Tokenizer that handles Chinese (no spaces), Japanese (mixed kana /
20
+ kanji), Korean (spaces + Hangul).
21
+ - Dictionary format: CJK dictionaries don't have affixes; they're
22
+ frequency-sorted word lists.
23
+ - Suggestion strategy: edit-distance on Unicode codepoints is wrong;
24
+ need pinyin (zh) / romaji (ja) / romanization (ko) aware edit distance.
25
+ - suika (current tokenizer) has segmentation; verify CJK coverage.
26
+
27
+ Reference: `/Users/mulgogi/src/external/cspell/` has a working CJK
28
+ implementation in TypeScript; port the algorithms, not the code.
29
+
30
+ ### T3.2 — RTL support (promote from `TODO.impl/07-rtl-support.md`)
31
+
32
+ Status: outline only. Real work needs:
33
+
34
+ - Arabic / Hebrew normalizers (NFC, ligature handling, diacritic stripping).
35
+ - RTL-aware display in the CLI batch reporter and interactive reviewer.
36
+ - Right-to-left mark / left-to-right mark handling in the tokenizer
37
+ (see `spec/integrational/fixtures/right_to_left_mark.*`).
38
+
39
+ ### T3.3 — Grammar engine (promote from `TODO.impl/08-grammar-engine.md`)
40
+
41
+ Status: skeleton in `lib/kotoshu/grammar/`. Real work needs:
42
+
43
+ - Rule format stabilized (currently `Rule` + `PatternMatchers::*`).
44
+ - A first useful rule set (probably article/confusion pairs: a/an,
45
+ its/it's, their/there/they're).
46
+ - Integration with the semantic analyzer for context-aware rules.
47
+ - A way to load rule packs per language.
48
+
49
+ Reference: `/Users/mulgogi/src/external/languagetool/` (Java) for rule
50
+ shape and category taxonomy. Don't reimplement LT's category model
51
+ verbatim — port the *shape*, not the 5000-rule corpus.
52
+
53
+ ### T3.4 — Document plugins
54
+
55
+ The boundary is decided (see `kotoshu-document-plugin-boundary.md`
56
+ memory): kotoshu never owns document parsing. Existing plugins:
57
+
58
+ - `coradoc-plugin-kotoshu` (AsciiDoc via coradoc).
59
+ - Markdown: kotoshu has a built-in `documents/markdown_document.rb`.
60
+ This violates the boundary; migrate to a plugin or formalize the
61
+ exception.
62
+ - Plain text: built-in, keep.
63
+ - reStructuredText, Org, HTML, LaTeX, etc.: community plugins.
64
+
65
+ Work: define the `Kotoshu::Plugin::Document` interface and the
66
+ registration mechanism, then audit existing built-ins against it.
67
+
68
+ ### T3.5 — Multi-language document checking
69
+
70
+ Right now `Kotoshu.check_file` is single-language. Real documents mix
71
+ languages (code comments in English + prose in French, etc.).
72
+
73
+ Work:
74
+
75
+ - Per-paragraph language detection via `Language::Identifier`.
76
+ - Per-language dictionary resolution.
77
+ - Result aggregation across languages.
78
+
79
+ ### T3.6 — Personal dictionary management
80
+
81
+ The personal dictionary (`~/.config/kotoshu/personal_dictionary.txt`
82
+ or similar) exists but is undertested. Work:
83
+
84
+ - CLI for add/remove/list/import.
85
+ - Per-project personal dictionaries (`.kotoshu/` in project root).
86
+ - Sync with Hunspell-style personal dict format for interop.
87
+
88
+ ## T4 — Quality
89
+
90
+ ### T4.1 — Performance pass
91
+
92
+ `spec/benchmark/` and `spec/performance/` exist but are skipped by
93
+ default. Work:
94
+
95
+ - Establish baselines for: cold-start, lookup throughput,
96
+ suggest latency at 1k/10k/100k dictionary size.
97
+ - Profile the hot path (`Kotoshu.correct?`); attack the biggest
98
+ offender.
99
+ - Consider Memoization, Trie compression, Bloom filter tuning.
100
+
101
+ ### T4.2 — CI
102
+
103
+ GitHub Actions workflow that runs:
104
+
105
+ - `bundle exec rspec --tag ~network --tag ~onnx` on every push.
106
+ - `bundle exec rubocop` on every push.
107
+ - `NETWORK_TESTS=1 bundle exec rspec` nightly on `main`.
108
+ - `ONNX_TESTS=1 bundle exec rspec` nightly on `main` (separate job;
109
+ cache the model).
110
+
111
+ ### T4.3 — Property-based testing
112
+
113
+ `spec/properties/` exists but is thin. Expand:
114
+
115
+ - Trie: insertion is order-independent; lookup is total.
116
+ - Suggestion generator: `suggestions.include?(word)` implies
117
+ `dictionary.lookup?(word)` is true (no out-of-dictionary suggestions).
118
+ - Hunspell affix application: affixed forms round-trip through the
119
+ stemmer.
120
+
121
+ ### T4.4 — Documentation pass
122
+
123
+ - README.adoc: audit against actual CLI shape; remove 0.2-era examples.
124
+ - `docs/`: prune superseded planning docs; mark historical ones as
125
+ historical.
126
+ - YARD: every public API has a docstring; private APIs are marked
127
+ `@private`.
128
+
129
+ ## T5 — Architecture
130
+
131
+ ### T5.1 — MECE refactor of `suggestions/strategies/`
132
+
133
+ Current strategies overlap (e.g. `symspell` and `edit_distance` both
134
+ generate edit-distance-1 candidates). Refactor:
135
+
136
+ - Each strategy owns a *unique* candidate-generation mechanism.
137
+ - `CompositeStrategy` merges and ranks, doesn't dedupe by side effect.
138
+ - Strategies are pure functions of `(word, dictionary) → candidates`;
139
+ no shared state, no global config reads.
140
+
141
+ ### T5.2 — Configuration as data, not singleton
142
+
143
+ `Configuration.instance` is a process-wide singleton. Hard to test,
144
+ hard to scope per-document. Refactor:
145
+
146
+ - Pass a `Configuration` instance explicitly to `Spellchecker.new`.
147
+ - Default the facade (`Kotoshu.correct?`) to a process-default instance.
148
+ - Drop the singleton.
149
+
150
+ ### T5.3 — Result model consistency
151
+
152
+ `WordResult` / `DocumentResult` / `SuggestionSet` migrated to
153
+ `lutaml-model` in 0.3.0. Audit:
154
+
155
+ - All three have consistent `to_hash` / `from_hash` shapes.
156
+ - No hand-rolled `to_h` anywhere (per global rule).
157
+ - CLI consumers use the framework-supplied serialization.
158
+
159
+ ### T5.4 — Plugin architecture formalization
160
+
161
+ `Kotoshu::Plugin::*` exists but is informal. Define:
162
+
163
+ - Plugin discovery (`Gem.find_files` or explicit `Kotoshu::Plugin.register`).
164
+ - Plugin lifecycle (load, configure, start, stop).
165
+ - Plugin capabilities (document parser, suggestion strategy, language
166
+ module, output format).
167
+
168
+ ## T6 — Ecosystem
169
+
170
+ ### T6.1 — VSCode / LSP integration
171
+
172
+ A language server that runs kotoshu on save, reports diagnostics via
173
+ LSP. Probably a separate gem (`kotoshu-lsp`).
174
+
175
+ ### T6.2 — Web playground
176
+
177
+ A WASM build of kotoshu running in the browser. Requires a pure-Ruby
178
+ path (no native extensions) and a reasonable dictionary size budget.
179
+
180
+ ### T6.3 — Dictionary editor GUI
181
+
182
+ A desktop app (Tauri / Electron) for curating personal dictionaries and
183
+ custom affix files. Reads/writes Hunspell format.
184
+
185
+ ## Acceptance criteria for T3+ releases
186
+
187
+ Each item, when promoted, gets its own acceptance criteria. The
188
+ overarching rule: **never break T1/T2 correctness for a T3+ feature**.
189
+ Re-run the full `--tag ~network --tag ~onnx` suite before tagging any
190
+ T3+ release.
191
+
192
+ ## Dependencies
193
+
194
+ - **Blocked by:** T1 (`TODO.impl/36`) and T2 (`TODO.impl/37`) — the
195
+ baseline must be green before adding features.
196
+ - **Blocks:** nothing. This is the long horizon.
197
+
198
+ ## Status (reconciled 2026-09-03)
199
+
200
+ Verified shipped, with evidence:
201
+
202
+ - **T3.6 Personal dictionary** — `kotoshu personal add/remove/list/
203
+ import/path/clear` all exist (lib/kotoshu/cli/personal_command.rb,
204
+ wired in cli.rb) and are documented on the site (docs/cli).
205
+ - **T5.3 Result models on lutaml-model** — shipped in the 0.3.0
206
+ migration (per plan 36 history); `to_hash`/`from_hash` framework
207
+ paths in use.
208
+ - **T4.2 CI** — workflows exist (rake matrix on 3.3/3.4/4.0 ×
209
+ ubuntu/macOS/windows); the network-test job is red on main
210
+ (pre-existing, tracked separately). Do not treat CI-green as done.
211
+ - **T6.1 LSP** — shipped as the separate gem kotoshu-lsp 0.1.0
212
+ (github.com/kotoshu/kotoshu-lsp); superseded by the ecosystem track
213
+ (plans 60-64, all implemented).
214
+ - **T6.2 Web playground** — live at kotoshu.org/playground against
215
+ kotoshu-server; the WASM variant in this item is superseded by
216
+ kotoshu-rs P4 (gem plan 66) — in flight 2026-09-03.
217
+
218
+ Still backlog (unverified or not started):
219
+
220
+ - T3.1 CJK (superseded by TODO.impl/54-cjk-support.md and the 9-model
221
+ corpus now shipped for ja/ko/zh), T3.2 RTL (superseded by 55), T3.3
222
+ grammar expansion (superseded by 51), T3.4/T3.5 document plugins and
223
+ multi-language documents (50/52/53).
224
+ - T4.1 performance pass (benchmark baselines), T4.3 property tests,
225
+ T4.4 documentation pass (partially superseded by the 2026-09 docs
226
+ work on the site and README tiers).
227
+ - T5.1 strategy MECE refactor, T5.2 configuration-as-data
228
+ (58 tracks the design), T5.4 plugin formalization.
229
+ - T6.3 dictionary editor GUI — untouched.
230
+
231
+ This file remains the long-horizon index; promote items to their own
232
+ plans (as 50-59 already did) when they become priority.
@@ -0,0 +1,134 @@
1
+ # 40 — Cache spec / CLI drift cleanup
2
+
3
+ ## Status
4
+ Discovered while running the full suite after T2 Phase 2B completion.
5
+ Pre-existing — not introduced by T2 work.
6
+
7
+ ## Problem
8
+
9
+ Two distinct spec/impl drifts cause **39 failures** in the full suite
10
+ that are unrelated to T2 correctness:
11
+
12
+ ### Drift A — `LanguageCache` KV interface mismatch (25 failures)
13
+
14
+ `spec/kotoshu/cache/language_cache_spec.rb` expects a generic key-value
15
+ cache API on `Kotoshu::Cache::LanguageCache`:
16
+
17
+ - `read(key)` / `write(key, value)` / `fetch(key) { ... }`
18
+ - `delete(key)` / `key?(key)` / `clear` (no arg) / `size`
19
+ - `stats` returning `:hits, :misses, :size, :hit_rate,
20
+ :total_size_bytes, :cached_languages, :oldest_entry`
21
+ - `max_cache_size` attribute (1 GB default)
22
+ - `language_path(lang, type)` public method
23
+
24
+ But `LanguageCache < BaseCache` is a **disk-backed resource downloader**,
25
+ not a KV cache. Its actual API is `get / available? / clear(id) /
26
+ clear_all / stats / clean / get_spelling / get_grammar /
27
+ frequency_available? / cached_resources / language_info / install_local`.
28
+
29
+ The KV interface already lives on `LookupCache` and `SuggestionCache`
30
+ (in-memory LRU), which have their own dedicated specs at
31
+ `spec/kotoshu/cache/{lookup,suggestion}_cache_spec.rb`. Mixing the two
32
+ abstractions would violate the MECE principle.
33
+
34
+ The spec was written TDD-style before the implementation crystallized.
35
+ The implementation went the MECE route (KV → `LookupCache`, resources →
36
+ `LanguageCache`), and the spec was never reconciled.
37
+
38
+ ### Drift B — Wired CLI uses nonexistent `LanguageCache` methods (12 failures)
39
+
40
+ `lib/kotoshu/cli.rb` registers `subcommand "cache", CacheCommand` and
41
+ loads `lib/kotoshu/cli/cache_command.rb` (the `Kotoshu::Cli::CacheCommand`
42
+ class). That class calls **methods that do not exist** on
43
+ `LanguageCache`:
44
+
45
+ | Call in `cli/cache_command.rb` | Actual `LanguageCache` API |
46
+ |--------------------------------------|----------------------------|
47
+ | `cache.cache_status` | (none — must construct) |
48
+ | `cache.get_frequency_data(lang)` | (private `download_frequency`) |
49
+ | `cache.get_language_info(lang)` | `cache.language_info(lang)`|
50
+ | `cache.purge_all` | `cache.clear_all` |
51
+
52
+ A parallel implementation exists at `lib/kotoshu/commands/cache_command.rb`
53
+ (`Kotoshu::CacheCommand` — no `Cli` namespace) that **uses the correct
54
+ API** but is **not wired up** in `cli.rb`. So `kotoshu cache list` and
55
+ friends crash on every invocation in the current build.
56
+
57
+ `spec/kotoshu/cli/cache_command_spec.rb` exercises the broken
58
+ `Kotoshu::Cli::CacheCommand` and therefore fails 12 examples.
59
+
60
+ ### Drift C — Performance regression (2 failures)
61
+
62
+ `spec/performance/performance_regression_spec.rb:40,49` assert
63
+ suggestion latency under fixed thresholds (10ms, "with cache faster than
64
+ without"). These are timing-sensitive, machine-load-dependent, and not
65
+ related to T2.
66
+
67
+ ## Plan
68
+
69
+ ### Phase 1 — Realign `language_cache_spec.rb` to actual API
70
+
71
+ Rewrite the spec to test what `LanguageCache` actually does:
72
+
73
+ - `initialize` (path, ttl, registry wiring) — keep, drop
74
+ `max_cache_size` assertion unless we add the attribute (see below).
75
+ - `available?(resource_id)` — replace the KV "key?" tests.
76
+ - `cached_resources` — replace the KV "size" tests.
77
+ - `clear_all` / `clear(resource_id)` — replace the KV "clear" tests.
78
+ - `stats` — assert the **actual** shape (`hits, misses, total,
79
+ hit_rate, cached_resources, size_bytes, oldest_entry`).
80
+ - `clean` — keep, already correct.
81
+ - `get_spelling` / `get_grammar` — keep, already correct.
82
+ - Drop the `read/write/fetch/delete` KV tests entirely. They duplicate
83
+ `spec/kotoshu/cache/lookup_cache_spec.rb` and test the wrong class.
84
+
85
+ **Add** a `language_path(lang, type)` public helper on `LanguageCache`
86
+ as a useful, well-named accessor — it makes test setup and downstream
87
+ code (CLI status displays, audit log relative paths) clearer than
88
+ reaching into `cache_path` and joining paths by hand.
89
+
90
+ **Optional:** add `max_cache_size` attribute (default 1 GB) as
91
+ forward-looking work for `TODO.impl/34-cache-eviction.md`. Just an
92
+ `attr_reader` with a constructor kwarg — no eviction logic yet.
93
+
94
+ ### Phase 2 — Document and quarantine the broken CLI
95
+
96
+ Do **not** delete `lib/kotoshu/cli/cache_command.rb` (global rule:
97
+ never delete source). Instead:
98
+
99
+ 1. Mark the 12 failing examples in `spec/kotoshu/cli/cache_command_spec.rb`
100
+ as `pending` with a clear pointer to this file and the underlying
101
+ API gap.
102
+ 2. Leave it to a follow-up task to either:
103
+ - Consolidate the two CLIs (delete `cli/cache_command.rb`, rewire
104
+ `cli.rb` to `commands/cache_command.rb`); OR
105
+ - Fix `cli/cache_command.rb` to call the real `LanguageCache` API.
106
+
107
+ That decision is bigger than spec cleanup — it touches the public CLI
108
+ surface and needs design judgment. Defer.
109
+
110
+ ### Phase 3 — Quarantine performance regression specs
111
+
112
+ Mark `spec/performance/performance_regression_spec.rb:40,49` as
113
+ `:slow` (they already should be) or `pending` with reason
114
+ "timing-sensitive; revisit after T4.1 performance pass".
115
+
116
+ ## Acceptance
117
+
118
+ - [ ] `bundle exec rspec spec/kotoshu/cache/language_cache_spec.rb`
119
+ is green and tests the actual `LanguageCache` API.
120
+ - [ ] `bundle exec rspec spec/kotoshu/cli/cache_command_spec.rb`
121
+ has no unexpected failures (all pending with documented reasons).
122
+ - [ ] `bundle exec rspec spec/performance/performance_regression_spec.rb`
123
+ has no unexpected failures.
124
+ - [ ] `lib/kotoshu/cache/language_cache.rb` adds `language_path` and
125
+ (optionally) `max_cache_size`.
126
+ - [ ] No production code regressions — `bundle exec rspec spec/kotoshu`
127
+ minus the three quarantined files is fully green.
128
+
129
+ ## Out of scope for this file
130
+
131
+ - Consolidating the two `cache_command.rb` implementations.
132
+ - Fixing the wired-up CLI (`kotoshu cache ...`) end-to-end.
133
+ - Cache eviction (see `TODO.impl/34-cache-eviction.md`).
134
+ - Performance baseline work (see `TODO.impl/39-tier3-and-beyond.md` T4.1).
@@ -0,0 +1,194 @@
1
+ # 50 — Structure-aware document checking API
2
+
3
+ ## Goal
4
+
5
+ Define the value-object + parser-contract layer that lets document
6
+ plugins (coradoc-plugin-kotoshu, markdown-plugin-kotoshu, etc.) perform
7
+ **structure-aware** spell/grammar checks: the checker scans the
8
+ flattened text, but every error carries a position that maps back to
9
+ the original (markup-bearing) source.
10
+
11
+ ## Motivation
12
+
13
+ Concrete example from the user:
14
+
15
+ > Source: `I'm an **friend** of Tom`
16
+ > Flattened: `I'm an friend of Tom`
17
+ > Error: "an friend" should be "a friend"
18
+
19
+ The grammar checker sees the flattened sentence and flags "an friend".
20
+ But the error report must reference the original source — specifically
21
+ the range `an **friend**` — so an editor can highlight the actual
22
+ markdown/AsciiDoc the user wrote, not the stripped text.
23
+
24
+ This is the **plugin's** job (kotoshu never owns document parsing —
25
+ see `kotoshu-document-plugin-boundary.md` memory). Kotoshu's job is to
26
+ provide:
27
+
28
+ 1. Value objects for source position + range.
29
+ 2. A TextNode model that pairs flattened text with its source range.
30
+ 3. A Document interface with bidirectional offset mapping.
31
+ 4. An error model that carries source positions.
32
+ 5. A registry where plugins register format-specific parsers.
33
+
34
+ ## Design
35
+
36
+ ### Value objects
37
+
38
+ Kotoshu::Documents::SourcePosition
39
+ offset : Integer # 0-based char offset in the original source
40
+ line : Integer # 1-based line number
41
+ column : Integer # 1-based column (chars from line start)
42
+
43
+ Kotoshu::Documents::SourceRange
44
+ start : SourcePosition # inclusive
45
+ end : SourcePosition # exclusive
46
+
47
+ Both are `Struct.new(..., keyword_init: true)` with `freeze` on
48
+ construct. Comparable via `include Comparable` so ranges can be sorted
49
+ and tested for containment.
50
+
51
+ ### Text node
52
+
53
+ Kotoshu::Documents::TextNode
54
+ text : String # flattened text (no markup)
55
+ source_range : SourceRange # where this node lives in source
56
+ flattened_offset : Integer # 0-based offset in concatenated flattened text
57
+ format : Symbol # :plain, :bold, :italic, :code, :heading, ...
58
+ metadata : Hash # format-specific ({level: 2} for h2, etc.)
59
+
60
+ A document is a list of text nodes. Concatenating `node.text` for
61
+ every node yields the flattened text the checker scans.
62
+
63
+ ### Document interface
64
+
65
+ class Kotoshu::Documents::Document
66
+ attr_reader :text_nodes, :source, :format
67
+
68
+ def flattened_text
69
+ text_nodes.map(&:text).join
70
+ end
71
+
72
+ def flattened_length
73
+ text_nodes.sum { |n| n.text.length }
74
+ end
75
+
76
+ # Map a flattened-text offset to the source range containing it.
77
+ def source_range_at(flattened_offset)
78
+ # Walk nodes; find the one whose flattened window contains the
79
+ # offset. Return that node's source_range.
80
+ end
81
+
82
+ # Map a flattened-text range to the source range spanning it.
83
+ def source_range_for(flattened_start, flattened_end)
84
+ # Combine source_range_at(start).start with
85
+ # source_range_at(end - 1).end.
86
+ end
87
+ end
88
+
89
+ ### Built-in: PlainTextDocument
90
+
91
+ class Kotoshu::Documents::PlainTextDocument < Document
92
+ def self.from_string(text, language_code: nil)
93
+ nodes = [TextNode.new(text: text, source_range: full_range(text),
94
+ flattened_offset: 0, format: :plain, metadata: {})]
95
+ new(text_nodes: nodes, source: text, format: :plain)
96
+ end
97
+
98
+ def self.from_file(path, language_code: nil)
99
+ from_string(File.read(path), language_code: language_code)
100
+ end
101
+ end
102
+
103
+ PlainTextDocument is the only parser kotoshu ships. Markdown, AsciiDoc,
104
+ reStructuredText, etc. are plugin territory.
105
+
106
+ ### Plugin parser registry
107
+
108
+ module Kotoshu::Documents
109
+ def self.register(format, parser_class)
110
+ # parser_class responds to .from_string(text, language_code:) and
111
+ # optionally .from_file(path, language_code:).
112
+ end
113
+
114
+ def self.parser_for(format)
115
+ # Returns the registered parser class or nil.
116
+ end
117
+
118
+ def self.parse(source, format:, language_code: nil)
119
+ parser = parser_for(format) || PlainTextDocument
120
+ parser.from_string(source, language_code: language_code)
121
+ end
122
+ end
123
+
124
+ A plugin (`coradoc-plugin-kotoshu`) does:
125
+
126
+ Kotoshu::Documents.register(:asciidoc, CoradocPluginKotoshu::AsciidocParser)
127
+
128
+ on load. The parser produces a `Kotoshu::Documents::Document` whose
129
+ text_nodes carry proper source_ranges (coradoc knows the offsets).
130
+
131
+ ### Error model
132
+
133
+ Errors carry the source range directly. The checker resolves it from
134
+ the document before constructing the error:
135
+
136
+ class Kotoshu::Models::SemanticError
137
+ attr_reader :source_range, :original, :suggestions, ...
138
+ def initialize(source_range:, original:, suggestions:, ...)
139
+ end
140
+
141
+ For checkers that work on plain strings (no Document), `source_range`
142
+ is nil and the flattened position is the only positional info.
143
+
144
+ ### Check pipeline
145
+
146
+ analyzer.analyze(document) do |error|
147
+ # error.source_range is already resolved
148
+ plugin_or_editor.highlight(error.source_range)
149
+ end
150
+
151
+ Existing analyzer code stays the same — it iterates text_nodes and
152
+ emits errors. The change is that errors now carry source_range from
153
+ the text_node they originated in.
154
+
155
+ ## Phases
156
+
157
+ ### Phase 1 — Core abstractions (this PR)
158
+ - SourcePosition, SourceRange, TextNode, Document, PlainTextDocument.
159
+ - Documents::Document.register / parser_for / parse registry.
160
+ - Specs for each.
161
+
162
+ ### Phase 2 — Pipeline integration (this PR)
163
+ - SemanticAnalyzer accepts a Document, emits errors with source_range.
164
+ - WordResult / DocumentResult carry source_range.
165
+ - CheckCommand (currently dead code referencing non-existent
166
+ `Documents::Document`) is either removed or rewritten to use the new
167
+ API.
168
+
169
+ ### Phase 3 — Plugin contract docs (this PR)
170
+ - README section "Writing a document plugin".
171
+ - YARD on `Kotoshu::Documents.register` documenting the parser class
172
+ contract.
173
+
174
+ ### Phase 4 — coradoc-plugin-kotoshu (separate repo)
175
+ - coradoc-plugin-kotoshu implements the parser for AsciiDoc.
176
+ - Out of scope for kotoshu proper.
177
+
178
+ ## Acceptance criteria
179
+
180
+ - [ ] SourcePosition, SourceRange, TextNode, Document, PlainTextDocument
181
+ all have direct specs (real instances, no doubles).
182
+ - [ ] `Document#source_range_for(flattened_start, flattened_end)` returns
183
+ the correct SourceRange for the example "I'm an **friend** of Tom".
184
+ - [ ] SemanticAnalyzer emits errors carrying `source_range`.
185
+ - [ ] A plugin can register via `Kotoshu::Documents.register(:fmt, klass)`.
186
+ - [ ] No `respond_to?`, `send`-to-private, `instance_variable_*`,
187
+ `require_relative` in the new code.
188
+ - [ ] Full suite stays green (2901+ examples, 0 failures).
189
+
190
+ ## Dependencies
191
+
192
+ - **Blocked by:** none.
193
+ - **Blocks:** coradoc-plugin-kotoshu's structure-aware checking (that
194
+ plugin consumes this API).
@@ -0,0 +1,75 @@
1
+ # 51 — Grammar engine expansion (T3.3)
2
+
3
+ ## Goal
4
+
5
+ Ship useful grammar rule packs for English, building on the existing
6
+ `Kotoshu::Grammar` skeleton. Currently we have:
7
+
8
+ - `EN_A_VS_AN` (article usage before vowel/consonant sounds)
9
+ - `EN_THERE_THEIR` (their vs there)
10
+ - `EN_DOUBLE_NEGATIVE` (with the correlative-idiom exception fix
11
+ from PR #52)
12
+
13
+ Each pack adds one or more rules for a common confusion. The intent
14
+ is to demonstrate the engine works for real-world cases — not to
15
+ match LanguageTool's 5000-rule corpus.
16
+
17
+ ## Phases
18
+
19
+ ### Phase 1 — Possessive / contraction pairs
20
+
21
+ | Rule ID | Pattern | Why |
22
+ |-----------------|-----------------------------------------------|------------------------------|
23
+ | EN_ITS_IT_S | `its` vs `it's` from POS + context | Possessive vs contraction |
24
+ | EN_YOUR_YOURE | `your` vs `you're` | Possessive vs contraction |
25
+ | EN_WHOSE_WHO_IS | `whose` vs `who's` | Possessive vs contraction |
26
+
27
+ These all need a context-check matcher that looks at surrounding
28
+ verbs. Reuse the existing `PossessiveContextMatcher` pattern.
29
+
30
+ ### Phase 2 — Common confusion pairs
31
+
32
+ | Rule ID | From → To | Detection heuristic |
33
+ |--------------------|----------------------------|------------------------------------|
34
+ | EN_THEN_THAN | `then` → `than` (and vice versa) | Pre-comparison ("bigger then") |
35
+ | EN_AFFECT_EFFECT | `affect` ↔ `effect` | POS (verb vs noun) |
36
+ | EN_ACCEPT_EXCEPT | `accept` ↔ `except` | Context (preposition vs verb) |
37
+ | EN_LOSE_LOOSE | `lose` ↔ `loose` | POS |
38
+
39
+ ### Phase 3 — Capitalization
40
+
41
+ | Rule ID | Detection |
42
+ |------------------------|-------------------------------------------|
43
+ | EN_SENTENCE_START_CAP | First letter after `.`, `!`, `?` |
44
+ | EN_PROPER_NOUN_CAP | Common proper nouns ("monday", "english") |
45
+
46
+ Capitalization rules need a SentenceStartMatcher — new matcher type.
47
+
48
+ ### Phase 4 — Rule packs per language
49
+
50
+ - Move `data/en/grammar/rules.yaml` rules into a `RulePack` abstraction.
51
+ - Each language can have multiple packs (base, formal, technical).
52
+ - Loader picks the right pack based on configuration.
53
+
54
+ ## Design
55
+
56
+ - Each rule lives in `data/{lang}/grammar/rules.yaml` (data-driven,
57
+ same as today).
58
+ - New matcher types go in `lib/kotoshu/grammar/pattern_matchers/`.
59
+ - Rule engine itself doesn't change — Phase 4 just adds a RulePack
60
+ layer over the existing loader.
61
+
62
+ ## Acceptance criteria
63
+
64
+ - [ ] At least 4 new rule packs shipped for English (Phase 1 + Phase 2).
65
+ - [ ] Each rule has direct specs in `spec/kotoshu/languages/en/`.
66
+ - [ ] No false positives on the example sentences in the rule
67
+ descriptions.
68
+ - [ ] Full suite stays green (2945+ examples, 0 failures).
69
+
70
+ ## Dependencies
71
+
72
+ - **Blocked by:** none.
73
+ - **Blocks:** T3.4 multi-language document checking (the same engine
74
+ is used; per-paragraph language detection picks the right rule
75
+ pack).