kotoshu 1.0.2-aarch64-linux

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (458) hide show
  1. checksums.yaml +7 -0
  2. data/.kotoshu-baseline.json +1 -0
  3. data/.pre-commit-hooks.yaml +18 -0
  4. data/.rspec +3 -0
  5. data/.rubocop.yml +351 -0
  6. data/.yardopts +8 -0
  7. data/CHANGELOG.md +813 -0
  8. data/CLAUDE.md +176 -0
  9. data/CODE_OF_CONDUCT.md +132 -0
  10. data/Cargo.lock +359 -0
  11. data/Cargo.toml +15 -0
  12. data/LICENSE +31 -0
  13. data/README.adoc +1292 -0
  14. data/Rakefile +150 -0
  15. data/SECURITY.md +93 -0
  16. data/TODO.impl/00-cut-0.1.md +121 -0
  17. data/TODO.impl/00-cut-0.2.md +175 -0
  18. data/TODO.impl/00-vision.md +281 -0
  19. data/TODO.impl/01-hunspell-correctness.md +111 -0
  20. data/TODO.impl/02-cli-unification.md +74 -0
  21. data/TODO.impl/03-dynamic-download.md +107 -0
  22. data/TODO.impl/04-language-modules.md +89 -0
  23. data/TODO.impl/05-semantic-path.md +83 -0
  24. data/TODO.impl/06-cjk-support.md +105 -0
  25. data/TODO.impl/07-rtl-support.md +71 -0
  26. data/TODO.impl/08-grammar-engine.md +106 -0
  27. data/TODO.impl/09-integrity-security.md +90 -0
  28. data/TODO.impl/10-testing-ci.md +98 -0
  29. data/TODO.impl/100-full-feature-batch3.md +28 -0
  30. data/TODO.impl/101-int4-tier-experiment.md +32 -0
  31. data/TODO.impl/102-lid-chain.md +28 -0
  32. data/TODO.impl/103-oov-bucket-tables.md +27 -0
  33. data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
  34. data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
  35. data/TODO.impl/106-native-lid-detection.md +36 -0
  36. data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
  37. data/TODO.impl/108-ko-ne-full-feature.md +37 -0
  38. data/TODO.impl/109-ten-audit.md +26 -0
  39. data/TODO.impl/11-release-v1.md +95 -0
  40. data/TODO.impl/110-sr-latn-and-polish.md +33 -0
  41. data/TODO.impl/111-embedding-bakeoff.md +33 -0
  42. data/TODO.impl/112-deployability-and-gating.md +26 -0
  43. data/TODO.impl/113-language-packs.md +26 -0
  44. data/TODO.impl/114-typo-biencoder-thread.md +33 -0
  45. data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
  46. data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
  47. data/TODO.impl/117-deterministic-frequency-tiers.md +110 -0
  48. data/TODO.impl/118-server-gem-floor.md +33 -0
  49. data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
  50. data/TODO.impl/120-setup-metadata-refresh.md +35 -0
  51. data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
  52. data/TODO.impl/122-pack-compression-kpk2.md +66 -0
  53. data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
  54. data/TODO.impl/124-owner-residue.md +19 -0
  55. data/TODO.impl/125-consumer-floor-guards.md +43 -0
  56. data/TODO.impl/126-tier-error-budgets.md +33 -0
  57. data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
  58. data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
  59. data/TODO.impl/129-pt-fr-corpora.md +23 -0
  60. data/TODO.impl/130-news-truth-0912.md +23 -0
  61. data/TODO.impl/131-hybrid-productization.md +42 -0
  62. data/TODO.impl/132-techniques-page-and-prose.md +27 -0
  63. data/TODO.impl/133-rust-first-distribution.md +68 -0
  64. data/TODO.impl/20-encapsulation-cleanup.md +47 -0
  65. data/TODO.impl/21-autoload-migration.md +64 -0
  66. data/TODO.impl/22-url-pin-single-source-of-truth.md +120 -0
  67. data/TODO.impl/23-fix-url-defaults.md +57 -0
  68. data/TODO.impl/24-cli-auto-setup-prompt.md +76 -0
  69. data/TODO.impl/25-soft-onnxruntime-require.md +75 -0
  70. data/TODO.impl/26-onnx-vocab-json-generation.md +96 -0
  71. data/TODO.impl/27-end-to-end-smoke-spec.md +113 -0
  72. data/TODO.impl/28-readme-quickstart-verification.md +68 -0
  73. data/TODO.impl/29-status-command.md +84 -0
  74. data/TODO.impl/30-language-auto-detection.md +74 -0
  75. data/TODO.impl/31-download-progress.md +58 -0
  76. data/TODO.impl/32-sarif-output-verification.md +72 -0
  77. data/TODO.impl/33-audit-log-rotation.md +54 -0
  78. data/TODO.impl/34-cache-eviction.md +59 -0
  79. data/TODO.impl/35-shell-completion.md +61 -0
  80. data/TODO.impl/36-cleanup-regressions-0.3.1.md +134 -0
  81. data/TODO.impl/37-hunspell-correctness-tier2.md +203 -0
  82. data/TODO.impl/38-onnx-semantic-gating.md +218 -0
  83. data/TODO.impl/39-tier3-and-beyond.md +232 -0
  84. data/TODO.impl/40-spec-drift-cleanup.md +134 -0
  85. data/TODO.impl/50-structure-aware-document-api.md +194 -0
  86. data/TODO.impl/51-grammar-engine-expansion.md +75 -0
  87. data/TODO.impl/52-multi-language-documents.md +95 -0
  88. data/TODO.impl/53-document-plugin-architecture.md +94 -0
  89. data/TODO.impl/54-cjk-support.md +117 -0
  90. data/TODO.impl/55-rtl-support.md +109 -0
  91. data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
  92. data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
  93. data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
  94. data/TODO.impl/59-yard-api-documentation.md +75 -0
  95. data/TODO.impl/60-lsp-server.md +174 -0
  96. data/TODO.impl/61-editor-ecosystem.md +167 -0
  97. data/TODO.impl/62-framework-integrations.md +159 -0
  98. data/TODO.impl/63-web-playground-wasm.md +186 -0
  99. data/TODO.impl/64-http-api-and-sdks.md +191 -0
  100. data/TODO.impl/65-universal-kotoshu.md +251 -0
  101. data/TODO.impl/66-kotoshu-core.md +169 -0
  102. data/TODO.impl/67-kotoshu-rs-and-access-libraries.md +137 -0
  103. data/TODO.impl/68-sota-adoption.md +152 -0
  104. data/TODO.impl/69-sota-typo-eval.md +74 -0
  105. data/TODO.impl/70-sota-confidence-cascade.md +59 -0
  106. data/TODO.impl/71-sota-nested-model.md +63 -0
  107. data/TODO.impl/72-model2vec-evaluation.md +49 -0
  108. data/TODO.impl/73-rl-reranker.md +42 -0
  109. data/TODO.impl/74-distilled-reranker.md +36 -0
  110. data/TODO.impl/75-llm-evaluator.md +35 -0
  111. data/TODO.impl/76-optional-llm-backend.md +37 -0
  112. data/TODO.impl/77-model-coverage-expansion.md +96 -0
  113. data/TODO.impl/78-in-browser-playground.md +69 -0
  114. data/TODO.impl/79-news-and-releases.md +60 -0
  115. data/TODO.impl/80-ecosystem-docs.md +56 -0
  116. data/TODO.impl/81-editor-reach.md +68 -0
  117. data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
  118. data/TODO.impl/83-models-batch-2.md +56 -0
  119. data/TODO.impl/84-language-modules-wave2.md +75 -0
  120. data/TODO.impl/85-wasm-model-api.md +48 -0
  121. data/TODO.impl/86-site-truth-wave2.md +45 -0
  122. data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
  123. data/TODO.impl/88-cli-directory-mode.md +54 -0
  124. data/TODO.impl/89-framework-integrations-exec.md +59 -0
  125. data/TODO.impl/90-gem-correctness-residues.md +44 -0
  126. data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
  127. data/TODO.impl/92-tier-mirror-cors.md +88 -0
  128. data/TODO.impl/93-python-wheels.md +32 -0
  129. data/TODO.impl/94-site-integrations-docs.md +35 -0
  130. data/TODO.impl/95-editor-action-polish.md +27 -0
  131. data/TODO.impl/96-server-semantic-models.md +50 -0
  132. data/TODO.impl/97-yard-execution.md +73 -0
  133. data/TODO.impl/98-campaign-cleanup.md +78 -0
  134. data/TODO.impl/99-site-truth-performance-docs.md +31 -0
  135. data/conformance/vectors.jsonl +2630 -0
  136. data/data/en/grammar/rules.yaml +851 -0
  137. data/docs/STABILITY-1.0-AUDIT.md +251 -0
  138. data/docs/STABILITY-1.0.md +137 -0
  139. data/docs/cache-architecture.md +502 -0
  140. data/docs/verification-evidence/correct3/pr_body.md +37 -0
  141. data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
  142. data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
  143. data/docs/verification-evidence/correct3/sv.aff +856 -0
  144. data/docs/verification-evidence/correct3/sv.dic +153717 -0
  145. data/examples/01_basic_word_checking.rb +38 -0
  146. data/examples/02_text_document_checking.rb +77 -0
  147. data/examples/03_dictionary_backends.rb +137 -0
  148. data/examples/04_trie_data_structure.rb +146 -0
  149. data/examples/05_suggestion_algorithms.rb +239 -0
  150. data/examples/06_configuration_advanced.rb +287 -0
  151. data/examples/07_multi_language_dictionaries.rb +278 -0
  152. data/exe/kotoshu +6 -0
  153. data/lib/kotoshu/algorithms/capitalization.rb +276 -0
  154. data/lib/kotoshu/algorithms/edit_distance.rb +128 -0
  155. data/lib/kotoshu/algorithms/lookup/affix_form.rb +132 -0
  156. data/lib/kotoshu/algorithms/lookup/compound_checks.rb +209 -0
  157. data/lib/kotoshu/algorithms/lookup/compound_form.rb +50 -0
  158. data/lib/kotoshu/algorithms/lookup.rb +841 -0
  159. data/lib/kotoshu/algorithms/ngram_suggest.rb +367 -0
  160. data/lib/kotoshu/algorithms/permutations.rb +283 -0
  161. data/lib/kotoshu/algorithms/phonet_suggest.rb +162 -0
  162. data/lib/kotoshu/algorithms/soundex.rb +92 -0
  163. data/lib/kotoshu/algorithms/suggest.rb +794 -0
  164. data/lib/kotoshu/algorithms.rb +23 -0
  165. data/lib/kotoshu/analyzers/semantic_analyzer.rb +405 -0
  166. data/lib/kotoshu/analyzers.rb +8 -0
  167. data/lib/kotoshu/baseline/application.rb +18 -0
  168. data/lib/kotoshu/baseline/entry.rb +33 -0
  169. data/lib/kotoshu/baseline/store.rb +175 -0
  170. data/lib/kotoshu/baseline.rb +20 -0
  171. data/lib/kotoshu/cache/base_cache.rb +778 -0
  172. data/lib/kotoshu/cache/cache.rb +91 -0
  173. data/lib/kotoshu/cache/eviction_policy.rb +66 -0
  174. data/lib/kotoshu/cache/frequency_cache.rb +283 -0
  175. data/lib/kotoshu/cache/language_cache.rb +511 -0
  176. data/lib/kotoshu/cache/lookup_cache.rb +166 -0
  177. data/lib/kotoshu/cache/model_cache.rb +1147 -0
  178. data/lib/kotoshu/cache/model_registry.rb +109 -0
  179. data/lib/kotoshu/cache/staged_languages.rb +45 -0
  180. data/lib/kotoshu/cache/suggestion_cache.rb +113 -0
  181. data/lib/kotoshu/cache.rb +49 -0
  182. data/lib/kotoshu/cli/auto_setup.rb +71 -0
  183. data/lib/kotoshu/cli/baseline_command.rb +119 -0
  184. data/lib/kotoshu/cli/batch_reporter.rb +321 -0
  185. data/lib/kotoshu/cli/cache_command.rb +282 -0
  186. data/lib/kotoshu/cli/completions_command.rb +210 -0
  187. data/lib/kotoshu/cli/dict_command.rb +64 -0
  188. data/lib/kotoshu/cli/directory_check.rb +146 -0
  189. data/lib/kotoshu/cli/directory_walker.rb +132 -0
  190. data/lib/kotoshu/cli/display_formatter.rb +429 -0
  191. data/lib/kotoshu/cli/errors.rb +36 -0
  192. data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
  193. data/lib/kotoshu/cli/interactive_reviewer.rb +314 -0
  194. data/lib/kotoshu/cli/language_resolver.rb +91 -0
  195. data/lib/kotoshu/cli/navigation_manager.rb +273 -0
  196. data/lib/kotoshu/cli/personal_command.rb +113 -0
  197. data/lib/kotoshu/cli/progress_reporter.rb +114 -0
  198. data/lib/kotoshu/cli/status_report.rb +130 -0
  199. data/lib/kotoshu/cli.rb +880 -0
  200. data/lib/kotoshu/components/passthrough_spell_checker.rb +70 -0
  201. data/lib/kotoshu/components/pos_tagger.rb +98 -0
  202. data/lib/kotoshu/components/spell_checker.rb +73 -0
  203. data/lib/kotoshu/components/synthesizer.rb +60 -0
  204. data/lib/kotoshu/components/tokenizer.rb +58 -0
  205. data/lib/kotoshu/components/whitespace_tokenizer.rb +94 -0
  206. data/lib/kotoshu/components.rb +13 -0
  207. data/lib/kotoshu/configuration/builder.rb +207 -0
  208. data/lib/kotoshu/configuration/resolver.rb +124 -0
  209. data/lib/kotoshu/configuration.rb +829 -0
  210. data/lib/kotoshu/conformance_exporter.rb +256 -0
  211. data/lib/kotoshu/conformance_runner.rb +176 -0
  212. data/lib/kotoshu/core/exceptions.rb +186 -0
  213. data/lib/kotoshu/core/indexed_dictionary.rb +293 -0
  214. data/lib/kotoshu/core/models/affix_rule.rb +262 -0
  215. data/lib/kotoshu/core/models/result/document_result.rb +132 -0
  216. data/lib/kotoshu/core/models/result/word_result.rb +131 -0
  217. data/lib/kotoshu/core/models/word.rb +131 -0
  218. data/lib/kotoshu/core/trie/builder.rb +119 -0
  219. data/lib/kotoshu/core/trie/node.rb +94 -0
  220. data/lib/kotoshu/core/trie/trie.rb +249 -0
  221. data/lib/kotoshu/core.rb +28 -0
  222. data/lib/kotoshu/data/common_words/de.yml +1800 -0
  223. data/lib/kotoshu/data/common_words/en.yml +1215 -0
  224. data/lib/kotoshu/data/common_words/es.yml +750 -0
  225. data/lib/kotoshu/data/common_words/fr.yml +1015 -0
  226. data/lib/kotoshu/data/common_words/pt.yml +870 -0
  227. data/lib/kotoshu/data/common_words/ru.yml +484 -0
  228. data/lib/kotoshu/data/common_words_loader.rb +141 -0
  229. data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
  230. data/lib/kotoshu/data/frozen_kelly.rb +38 -0
  231. data/lib/kotoshu/data.rb +8 -0
  232. data/lib/kotoshu/data_structures/bloom_filter.rb +188 -0
  233. data/lib/kotoshu/data_structures.rb +8 -0
  234. data/lib/kotoshu/debug_logger.rb +146 -0
  235. data/lib/kotoshu/debug_mode.rb +136 -0
  236. data/lib/kotoshu/defaults.rb +84 -0
  237. data/lib/kotoshu/dictionaries/catalog.rb +814 -0
  238. data/lib/kotoshu/dictionaries.rb +8 -0
  239. data/lib/kotoshu/dictionary/base.rb +324 -0
  240. data/lib/kotoshu/dictionary/cspell.rb +243 -0
  241. data/lib/kotoshu/dictionary/custom.rb +225 -0
  242. data/lib/kotoshu/dictionary/hunspell.rb +520 -0
  243. data/lib/kotoshu/dictionary/plain_text.rb +302 -0
  244. data/lib/kotoshu/dictionary/repository.rb +246 -0
  245. data/lib/kotoshu/dictionary/unified.rb +236 -0
  246. data/lib/kotoshu/dictionary/unix_words.rb +218 -0
  247. data/lib/kotoshu/dictionary.rb +15 -0
  248. data/lib/kotoshu/documents/document.rb +120 -0
  249. data/lib/kotoshu/documents/plain_text_document.rb +73 -0
  250. data/lib/kotoshu/documents/source_position.rb +45 -0
  251. data/lib/kotoshu/documents/source_range.rb +76 -0
  252. data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
  253. data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
  254. data/lib/kotoshu/documents/suppressions.rb +44 -0
  255. data/lib/kotoshu/documents/text_node.rb +49 -0
  256. data/lib/kotoshu/documents.rb +178 -0
  257. data/lib/kotoshu/embeddings/embedding_pipeline.rb +238 -0
  258. data/lib/kotoshu/embeddings/lru_cache.rb +244 -0
  259. data/lib/kotoshu/embeddings/onnx_runtime_model.rb +411 -0
  260. data/lib/kotoshu/embeddings/protocol.rb +96 -0
  261. data/lib/kotoshu/embeddings/protocols.rb +23 -0
  262. data/lib/kotoshu/embeddings/registry.rb +184 -0
  263. data/lib/kotoshu/embeddings/search.rb +204 -0
  264. data/lib/kotoshu/embeddings/similarity_engine.rb +278 -0
  265. data/lib/kotoshu/embeddings/similarity_search.rb +328 -0
  266. data/lib/kotoshu/embeddings/vocabulary.rb +306 -0
  267. data/lib/kotoshu/embeddings.rb +87 -0
  268. data/lib/kotoshu/fluent_checker.rb +94 -0
  269. data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +48 -0
  270. data/lib/kotoshu/grammar/pattern_matchers/double_negative_matcher.rb +179 -0
  271. data/lib/kotoshu/grammar/pattern_matchers/phrase_matcher.rb +86 -0
  272. data/lib/kotoshu/grammar/pattern_matchers/possessive_context_matcher.rb +75 -0
  273. data/lib/kotoshu/grammar/pattern_matchers/possessive_contraction_matcher.rb +87 -0
  274. data/lib/kotoshu/grammar/pattern_matchers/sentence_start_matcher.rb +84 -0
  275. data/lib/kotoshu/grammar/pattern_matchers/vowel_sound_matcher.rb +81 -0
  276. data/lib/kotoshu/grammar/pattern_matchers/word_list_matcher.rb +66 -0
  277. data/lib/kotoshu/grammar/rule.rb +99 -0
  278. data/lib/kotoshu/grammar/rule_engine.rb +96 -0
  279. data/lib/kotoshu/grammar/rule_loader.rb +30 -0
  280. data/lib/kotoshu/grammar.rb +24 -0
  281. data/lib/kotoshu/integrity/audit_log.rb +167 -0
  282. data/lib/kotoshu/integrity/manifest.rb +116 -0
  283. data/lib/kotoshu/integrity/net_http.rb +51 -0
  284. data/lib/kotoshu/integrity/rotation_policy.rb +88 -0
  285. data/lib/kotoshu/integrity.rb +26 -0
  286. data/lib/kotoshu/jekyll.rb +101 -0
  287. data/lib/kotoshu/keyboard/layout.rb +116 -0
  288. data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
  289. data/lib/kotoshu/keyboard/layouts/azerty.rb +55 -0
  290. data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
  291. data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
  292. data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
  293. data/lib/kotoshu/keyboard/layouts/dvorak.rb +54 -0
  294. data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
  295. data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
  296. data/lib/kotoshu/keyboard/layouts/hrsl.rb +94 -0
  297. data/lib/kotoshu/keyboard/layouts/jcuken.rb +57 -0
  298. data/lib/kotoshu/keyboard/layouts/latin.rb +266 -0
  299. data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
  300. data/lib/kotoshu/keyboard/layouts/qwerty.rb +52 -0
  301. data/lib/kotoshu/keyboard/layouts/qwertz.rb +55 -0
  302. data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
  303. data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
  304. data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
  305. data/lib/kotoshu/keyboard/registry.rb +201 -0
  306. data/lib/kotoshu/keyboard.rb +96 -0
  307. data/lib/kotoshu/kotoshu_native.so +0 -0
  308. data/lib/kotoshu/language/detector.rb +242 -0
  309. data/lib/kotoshu/language/identifier.rb +376 -0
  310. data/lib/kotoshu/language/languages/base.rb +271 -0
  311. data/lib/kotoshu/language/lid_detector.rb +188 -0
  312. data/lib/kotoshu/language/normalizer/arabic.rb +113 -0
  313. data/lib/kotoshu/language/normalizer/base.rb +133 -0
  314. data/lib/kotoshu/language/normalizer/greek.rb +63 -0
  315. data/lib/kotoshu/language/normalizer/hebrew.rb +72 -0
  316. data/lib/kotoshu/language/normalizer/persian.rb +36 -0
  317. data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
  318. data/lib/kotoshu/language/registry.rb +209 -0
  319. data/lib/kotoshu/language/resources/ar/common_words.txt +6753 -0
  320. data/lib/kotoshu/language/resources/ar/confusion_sets.txt +11 -0
  321. data/lib/kotoshu/language/resources/de/common_words.txt +10003 -0
  322. data/lib/kotoshu/language/resources/de/confusion_sets.txt +246 -0
  323. data/lib/kotoshu/language/resources/en/common_words.txt +9979 -0
  324. data/lib/kotoshu/language/resources/en/confusion_sets.txt +871 -0
  325. data/lib/kotoshu/language/resources/es/common_words.txt +9992 -0
  326. data/lib/kotoshu/language/resources/es/confusion_sets.txt +17 -0
  327. data/lib/kotoshu/language/resources/fr/common_words.txt +9993 -0
  328. data/lib/kotoshu/language/resources/fr/confusion_sets.txt +76 -0
  329. data/lib/kotoshu/language/resources/pt/common_words.txt +9977 -0
  330. data/lib/kotoshu/language/resources/pt/confusion_sets.txt +18 -0
  331. data/lib/kotoshu/language/resources/ru/common_words.txt +9951 -0
  332. data/lib/kotoshu/language/resources/ru/confusion_sets.txt +5 -0
  333. data/lib/kotoshu/language/script.rb +128 -0
  334. data/lib/kotoshu/language/segmenter.rb +122 -0
  335. data/lib/kotoshu/language/suika.rb +38 -0
  336. data/lib/kotoshu/language/tokenizer/base.rb +187 -0
  337. data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
  338. data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
  339. data/lib/kotoshu/language/tokenizer/french_tokenizer.rb +172 -0
  340. data/lib/kotoshu/language/tokenizer/german_tokenizer.rb +41 -0
  341. data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
  342. data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
  343. data/lib/kotoshu/language/tokenizer/japanese_tokenizer.rb +55 -0
  344. data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +153 -0
  345. data/lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb +160 -0
  346. data/lib/kotoshu/language/tokenizer/russian_tokenizer.rb +93 -0
  347. data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
  348. data/lib/kotoshu/language/tokenizer/spanish_tokenizer.rb +122 -0
  349. data/lib/kotoshu/language.rb +127 -0
  350. data/lib/kotoshu/languages/ar/language.rb +98 -0
  351. data/lib/kotoshu/languages/bg/language.rb +37 -0
  352. data/lib/kotoshu/languages/ca/language.rb +22 -0
  353. data/lib/kotoshu/languages/cs/language.rb +22 -0
  354. data/lib/kotoshu/languages/da/language.rb +22 -0
  355. data/lib/kotoshu/languages/de/language.rb +554 -0
  356. data/lib/kotoshu/languages/el/language.rb +50 -0
  357. data/lib/kotoshu/languages/en/language.rb +470 -0
  358. data/lib/kotoshu/languages/es/language.rb +476 -0
  359. data/lib/kotoshu/languages/et/language.rb +24 -0
  360. data/lib/kotoshu/languages/fa/language.rb +91 -0
  361. data/lib/kotoshu/languages/fr/language.rb +505 -0
  362. data/lib/kotoshu/languages/he/language.rb +98 -0
  363. data/lib/kotoshu/languages/hr/language.rb +24 -0
  364. data/lib/kotoshu/languages/hu/language.rb +22 -0
  365. data/lib/kotoshu/languages/id/language.rb +25 -0
  366. data/lib/kotoshu/languages/it/language.rb +22 -0
  367. data/lib/kotoshu/languages/ja/language.rb +476 -0
  368. data/lib/kotoshu/languages/ko/language.rb +56 -0
  369. data/lib/kotoshu/languages/latin_base.rb +53 -0
  370. data/lib/kotoshu/languages/lt/language.rb +24 -0
  371. data/lib/kotoshu/languages/lv/language.rb +24 -0
  372. data/lib/kotoshu/languages/nb/language.rb +25 -0
  373. data/lib/kotoshu/languages/ne/language.rb +60 -0
  374. data/lib/kotoshu/languages/nl/language.rb +22 -0
  375. data/lib/kotoshu/languages/nn/language.rb +26 -0
  376. data/lib/kotoshu/languages/pl/language.rb +22 -0
  377. data/lib/kotoshu/languages/pt/language.rb +439 -0
  378. data/lib/kotoshu/languages/ro/language.rb +22 -0
  379. data/lib/kotoshu/languages/ru/language.rb +420 -0
  380. data/lib/kotoshu/languages/sk/language.rb +25 -0
  381. data/lib/kotoshu/languages/sl/language.rb +25 -0
  382. data/lib/kotoshu/languages/sr/language.rb +39 -0
  383. data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
  384. data/lib/kotoshu/languages/sv/language.rb +22 -0
  385. data/lib/kotoshu/languages/tr/language.rb +36 -0
  386. data/lib/kotoshu/languages/uk/language.rb +38 -0
  387. data/lib/kotoshu/languages/vi/language.rb +22 -0
  388. data/lib/kotoshu/languages.rb +64 -0
  389. data/lib/kotoshu/metrics_collector.rb +238 -0
  390. data/lib/kotoshu/metrics_module.rb +121 -0
  391. data/lib/kotoshu/models/context.rb +119 -0
  392. data/lib/kotoshu/models/embedding_model.rb +181 -0
  393. data/lib/kotoshu/models/fasttext_model.rb +212 -0
  394. data/lib/kotoshu/models/nearest_neighbor.rb +96 -0
  395. data/lib/kotoshu/models/onnx_model.rb +330 -0
  396. data/lib/kotoshu/models/semantic_error.rb +204 -0
  397. data/lib/kotoshu/models/suggestion.rb +106 -0
  398. data/lib/kotoshu/models/word_embedding.rb +97 -0
  399. data/lib/kotoshu/models.rb +27 -0
  400. data/lib/kotoshu/multi_language_checker.rb +127 -0
  401. data/lib/kotoshu/native.rb +83 -0
  402. data/lib/kotoshu/native_backend.rb +131 -0
  403. data/lib/kotoshu/paths.rb +79 -0
  404. data/lib/kotoshu/personal_dictionary.rb +105 -0
  405. data/lib/kotoshu/plugins/plugin.rb +61 -0
  406. data/lib/kotoshu/plugins/registry.rb +118 -0
  407. data/lib/kotoshu/plugins.rb +9 -0
  408. data/lib/kotoshu/project_config.rb +76 -0
  409. data/lib/kotoshu/readers/aff_data.rb +530 -0
  410. data/lib/kotoshu/readers/aff_reader.rb +393 -0
  411. data/lib/kotoshu/readers/condition_checker.rb +125 -0
  412. data/lib/kotoshu/readers/dic_reader.rb +208 -0
  413. data/lib/kotoshu/readers/file_reader.rb +377 -0
  414. data/lib/kotoshu/readers/lookup_builder.rb +584 -0
  415. data/lib/kotoshu/readers/readers.rb +17 -0
  416. data/lib/kotoshu/readers.rb +25 -0
  417. data/lib/kotoshu/resource_bundle.rb +55 -0
  418. data/lib/kotoshu/resource_manager.rb +600 -0
  419. data/lib/kotoshu/results/result.rb +165 -0
  420. data/lib/kotoshu/results.rb +8 -0
  421. data/lib/kotoshu/rspec.rb +163 -0
  422. data/lib/kotoshu/scripts/fasttext_to_onnx.py +275 -0
  423. data/lib/kotoshu/source_registry.rb +104 -0
  424. data/lib/kotoshu/spellchecker/parallel_checker.rb +90 -0
  425. data/lib/kotoshu/spellchecker.rb +406 -0
  426. data/lib/kotoshu/string_metrics.rb +153 -0
  427. data/lib/kotoshu/suggestions/context.rb +43 -0
  428. data/lib/kotoshu/suggestions/frequency_provider.rb +80 -0
  429. data/lib/kotoshu/suggestions/generator.rb +169 -0
  430. data/lib/kotoshu/suggestions/pipeline.rb +135 -0
  431. data/lib/kotoshu/suggestions/semantic_cascade.rb +73 -0
  432. data/lib/kotoshu/suggestions/strategies/base_strategy.rb +315 -0
  433. data/lib/kotoshu/suggestions/strategies/composite_strategy.rb +197 -0
  434. data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +631 -0
  435. data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +207 -0
  436. data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +165 -0
  437. data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +284 -0
  438. data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +317 -0
  439. data/lib/kotoshu/suggestions/strategies/symspell_strategy.rb +270 -0
  440. data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
  441. data/lib/kotoshu/suggestions/suggestion.rb +154 -0
  442. data/lib/kotoshu/suggestions/suggestion_set.rb +270 -0
  443. data/lib/kotoshu/suggestions/sweep_index.rb +137 -0
  444. data/lib/kotoshu/suggestions.rb +29 -0
  445. data/lib/kotoshu/tasks/check_task.rb +104 -0
  446. data/lib/kotoshu/tasks.rb +11 -0
  447. data/lib/kotoshu/validators/spelling_validator.rb +101 -0
  448. data/lib/kotoshu/validators.rb +9 -0
  449. data/lib/kotoshu/version.rb +5 -0
  450. data/lib/kotoshu.rb +538 -0
  451. data/rakelib/compile.rake +10 -0
  452. data/script/validate_all_dictionaries.rb +444 -0
  453. data/scripts/build_native_gem.rb +45 -0
  454. data/scripts/extract_eval_grids.rb +37 -0
  455. data/scripts/generate_frozen_tiers.rb +44 -0
  456. data/sig/kotoshu.rbs +435 -0
  457. data/test_oop.rb +79 -0
  458. metadata +546 -0
@@ -0,0 +1,95 @@
1
+ # 52 — Multi-language document checking (T3.5)
2
+
3
+ ## Goal
4
+
5
+ Right now `Kotoshu.check_file` and `Kotoshu::Spellchecker` are
6
+ single-language. Real documents mix languages: code comments in
7
+ English embedded in French prose, a Japanese paper with English
8
+ abstract, etc. This TODO adds per-paragraph language detection and
9
+ per-language resolution so a single document is checked with the
10
+ right dictionary + rules for each segment.
11
+
12
+ ## Design
13
+
14
+ ### Layer 1: Per-segment detection
15
+
16
+ `Kotoshu::Language::Identifier` (FastText LID) already exists. Wire
17
+ it into a `DocumentSegmenter` that splits a `Kotoshu::Documents::Document`
18
+ into language-tagged segments.
19
+
20
+ class Kotoshu::Language::Segmenter
21
+ # @param document [Documents::Document]
22
+ # @param threshold [Float] detection confidence threshold (default 0.7)
23
+ # @return [Array<Segment>] language-tagged segments
24
+ def segment(document, threshold: 0.7)
25
+ # Walk document.text_nodes.
26
+ # Group consecutive nodes by detected language.
27
+ # Detect per node (or per paragraph break).
28
+ end
29
+ end
30
+
31
+ class Kotoshu::Language::Segment
32
+ # @param language_code [String]
33
+ # @param text_nodes [Array<Documents::TextNode>]
34
+ # @param confidence [Float]
35
+ end
36
+
37
+ ### Layer 2: Per-segment resource resolution
38
+
39
+ For each segment, resolve the right dictionary + rules via the
40
+ existing two-stage `ResourceManager`:
41
+
42
+ Kotoshu::ResourceManager.resolve(language: segment.language_code, want: %i[spelling grammar])
43
+
44
+ ### Layer 3: Multi-language check pipeline
45
+
46
+ class Kotoshu::MultiLanguageChecker
47
+ def check(document)
48
+ segments = segmenter.segment(document)
49
+ segments.flat_map do |segment|
50
+ bundle = ResourceManager.resolve(language: segment.language_code)
51
+ spellchecker = Spellchecker.new(resource_bundle: bundle)
52
+ segment.text_nodes.flat_map { |node|
53
+ spellchecker.check_text(node.text)
54
+ }
55
+ end
56
+ end
57
+ end
58
+
59
+ ### Layer 4: Result aggregation
60
+
61
+ Errors carry `source_range` from TODO 50. Per-segment errors already
62
+ have positions; aggregation is just a concatenation sorted by
63
+ source_range.start.
64
+
65
+ ## Phases
66
+
67
+ ### Phase 1 — Segmenter (this PR series)
68
+ - `Language::Segmenter` with paragraph-level detection.
69
+ - Specs with mixed-language fixtures (en + de, en + ja).
70
+
71
+ ### Phase 2 — MultiLanguageChecker (this PR series)
72
+ - Composes segmenter + spellchecker.
73
+ - Carries `source_range` through.
74
+
75
+ ### Phase 3 — CLI integration
76
+ - `kotoshu check --language auto FILE` picks the multi-language path.
77
+ - Default for `auto` becomes multi-language; single-language mode is
78
+ opt-in via `--language en`.
79
+
80
+ ## Acceptance criteria
81
+
82
+ - [ ] A document with English prose and French quotes is checked with
83
+ both dictionaries, errors from both languages surface in the
84
+ output with correct source positions.
85
+ - [ ] A document with English code comments and Japanese prose is
86
+ checked with the right language per segment.
87
+ - [ ] Performance: < 2x the single-language cost for typical mixed
88
+ documents.
89
+ - [ ] Full suite stays green.
90
+
91
+ ## Dependencies
92
+
93
+ - **Blocked by:** TODO 50 (document API with source positions).
94
+ - **Blocks:** T3.4 document plugin architecture (plugins want
95
+ multi-language support to come for free).
@@ -0,0 +1,94 @@
1
+ # 53 — Document plugin architecture (T3.4)
2
+
3
+ ## Goal
4
+
5
+ Formalize the boundary between kotoshu-core and format-specific
6
+ document parsers. The boundary is already decided (see
7
+ `kotoshu-document-plugin-boundary` memory): kotoshu never owns
8
+ document parsing. Plugins like `coradoc-plugin-kotoshu` provide
9
+ parsers that produce `Kotoshu::Documents::Document` instances.
10
+
11
+ This TODO formalizes the plugin contract, the registration mechanism,
12
+ and the audit that moves the existing built-in Markdown parser out of
13
+ core kotoshu (if/when a `markdown-plugin-kotoshu` exists).
14
+
15
+ ## Design
16
+
17
+ ### Plugin contract
18
+
19
+ A document plugin is a Ruby gem that:
20
+
21
+ 1. Depends on `kotoshu`.
22
+ 2. On load, calls `Kotoshu::Documents.register(format, ParserClass)`.
23
+ 3. The parser class responds to:
24
+ - `.from_string(text, language_code:)` → `Kotoshu::Documents::Document`
25
+ - `.from_file(path, language_code:)` → `Kotoshu::Documents::Document` (optional)
26
+ 4. The Document's `text_nodes` carry proper `SourceRange`s pointing
27
+ into the original markup-bearing source.
28
+
29
+ ### Discovery
30
+
31
+ Two discovery mechanisms:
32
+
33
+ 1. **Explicit**: user code calls `Kotoshu::Documents.register` after
34
+ requiring the plugin.
35
+ 2. **Gem find-files**: a plugin ships a `kotoshu_plugin/document/*.rb`
36
+ file. Kotoshu loads every such file at startup via
37
+ `Gem.find_files("kotoshu_plugin/document/*.rb")`.
38
+
39
+ Both are supported. Explicit is the default; gem find-files is the
40
+ "just install the gem and it works" path.
41
+
42
+ ### Built-in parsers
43
+
44
+ - **PlainTextDocument**: built-in, never moves. Plain text needs no
45
+ parser.
46
+ - **MarkdownDocument**: currently lives in `lib/kotoshu/documents/`
47
+ (was removed in TODO 50; verify and re-add as plugin if needed).
48
+ - **AsciiDocDocument**: same — was removed; should come from
49
+ `coradoc-plugin-kotoshu`.
50
+
51
+ ### Audit checklist
52
+
53
+ For each currently-shipped document parser, decide:
54
+
55
+ 1. **Keep in core** (e.g. PlainTextDocument) — generic enough.
56
+ 2. **Move to a plugin** (e.g. MarkdownDocument) — needs format-specific
57
+ parsing logic that violates the boundary.
58
+
59
+ ## Phases
60
+
61
+ ### Phase 1 — Plugin API (DONE in TODO 50)
62
+ - `Kotoshu::Documents.register`, `parser_for`, `parse`,
63
+ `registered_formats`, `reset!`.
64
+
65
+ ### Phase 2 — Gem find-files discovery
66
+ - On kotoshu load, walk `Gem.find_files("kotoshu_plugin/document/*.rb")`.
67
+ - Each file is responsible for calling `Documents.register`.
68
+ - Add `Kotoshu::Documents.discovered_plugins` reader.
69
+
70
+ ### Phase 3 — Audit existing parsers
71
+ - Confirm `lib/kotoshu/documents/` only contains core abstractions
72
+ (Location, TextNode, Document, PlainTextDocument).
73
+ - If a Markdown or AsciiDoc parser exists in core, document it as a
74
+ temporary exception OR migrate.
75
+
76
+ ### Phase 4 — Plugin authoring docs
77
+ - README section "Writing a document plugin".
78
+ - Example plugin skeleton.
79
+ - YARD on the contract.
80
+
81
+ ## Acceptance criteria
82
+
83
+ - [ ] A plugin installed as a gem is auto-discovered without user code.
84
+ - [ ] `Kotoshu::Documents.discovered_plugins` lists discovered plugins.
85
+ - [ ] No format-specific parsing code in `lib/kotoshu/documents/`
86
+ beyond PlainTextDocument.
87
+ - [ ] README has a "Writing a document plugin" section.
88
+ - [ ] Full suite stays green.
89
+
90
+ ## Dependencies
91
+
92
+ - **Blocked by:** TODO 50 (document API).
93
+ - **Blocks:** coradoc-plugin-kotoshu's structure-aware checking
94
+ (that plugin is the canonical consumer).
@@ -0,0 +1,117 @@
1
+ # 54 — CJK support (T3.1)
2
+
3
+ ## Goal
4
+
5
+ Real CJK (Chinese, Japanese, Korean) spell and grammar checking.
6
+ This is the largest tier 3 item: CJK languages need word-segmentation
7
+ (no spaces in zh/ja), different dictionary formats, and
8
+ romanization-aware suggestion strategies.
9
+
10
+ Status: outline only. This TODO documents the design space; real work
11
+ needs upstream library choices and per-language fixture data.
12
+
13
+ ## Per-language needs
14
+
15
+ ### Chinese (zh)
16
+
17
+ - **Tokenizer**: needs word segmentation. Options:
18
+ - `jieba-ruby` (port of the Python jieba) — HMM + dictionary-based.
19
+ - `suika` (already a kotoshu dependency for Japanese) has some CJK
20
+ support — verify coverage.
21
+ - **Dictionary format**: frequency-sorted word lists, not Hunspell.
22
+ CJK dictionaries don't have affixes.
23
+ - **Suggestions**: edit distance on Unicode codepoints is wrong (zh
24
+ characters are logographic). Need pinyin-aware edit distance:
25
+ - Map each character to its pinyin romanization.
26
+ - Edit-distance on pinyin, then map back to characters.
27
+ - Requires a `pinyin-ruby` gem or port of `pypinyin`.
28
+
29
+ ### Japanese (ja)
30
+
31
+ - **Tokenizer**: `suika` already integrated. Works for kana + kanji.
32
+ - **Dictionary format**: frequency-sorted word lists.
33
+ - **Suggestions**: romaji-aware edit distance. Map kana → romaji via
34
+ `romaji` gem, then edit-distance on romaji.
35
+
36
+ ### Korean (ko)
37
+
38
+ - **Tokenizer**: Korean HAS spaces but morphological analysis is
39
+ needed for accurate word boundaries. Options:
40
+ - `korean-stemmer` (pure Ruby port of mecab-ko).
41
+ - Or treat Korean like Latin-family languages (use space-split).
42
+ - **Dictionary format**: word lists.
43
+ - **Suggestions**: romanization via `romaja` or similar.
44
+
45
+ ## Design space
46
+
47
+ ### New tokenizer base classes
48
+
49
+ class Kotoshu::Language::Tokenizer::CJKTokenizer < Base
50
+ # Subclasses: ChineseTokenizer, JapaneseTokenizer, KoreanTokenizer
51
+ end
52
+
53
+ Each owns its segmentation algorithm and dictionary interaction.
54
+
55
+ ### New suggestion strategy
56
+
57
+ class Kotoshu::Suggestions::Strategies::RomanizationStrategy < BaseStrategy
58
+ # Edit distance on romanized form, then map candidates back to
59
+ # the original script.
60
+ end
61
+
62
+ Registered alongside the existing edit_distance, phonetic, etc.
63
+
64
+ ### New dictionary backend
65
+
66
+ class Kotoshu::Dictionary::FrequencySorted < Base
67
+ # CJK dictionaries don't have affixes. Just a frequency-sorted
68
+ # word list, optionally with POS tags.
69
+ end
70
+
71
+ ## Phases
72
+
73
+ ### Phase 0 — Survey (this TODO)
74
+ - Pick segmentation libraries (suika / jieba-ruby / etc.).
75
+ - Confirm license compatibility.
76
+ - Identify fixture sources (CC-CEDICT, JMDict, KRDict).
77
+
78
+ ### Phase 1 — Japanese (smallest scope)
79
+ - Tokenizer already exists (suika-backed `JapaneseTokenizer`).
80
+ - Add frequency-sorted dictionary backend.
81
+ - Add romaji-aware suggestion strategy.
82
+ - Specs with a small JMDict extract.
83
+
84
+ ### Phase 2 — Chinese
85
+ - Add `jieba-ruby` as a soft dependency.
86
+ - New `ChineseTokenizer`.
87
+ - Pinyin mapping.
88
+ - Specs with a small CC-CEDICT extract.
89
+
90
+ ### Phase 3 — Korean
91
+ - Decision on morphological analysis (mecab-ko vs simple split).
92
+ - Specs.
93
+
94
+ ## Acceptance criteria
95
+
96
+ - [ ] `kotoshu setup :ja` produces a working Japanese checker that
97
+ detects and suggests corrections for non-words.
98
+ - [ ] Same for Chinese and Korean.
99
+ - [ ] Suggestion quality on CJK typos is reasonable (top-3 contains
100
+ the intended word in > 70% of test cases).
101
+ - [ ] No CJK-specific code in core kotoshu outside `languages/ja`,
102
+ `languages/zh`, `languages/ko`, and the new strategy/backend.
103
+
104
+ ## Dependencies
105
+
106
+ - **Blocked by:** none directly, but Phases 1-3 each need their
107
+ upstream library chosen and bundled as soft deps.
108
+ - **Blocks:** nothing.
109
+
110
+ ## Open questions
111
+
112
+ - Should we ship CJK dictionaries in `data/{lang}/` or only via
113
+ dynamic download? Current direction: dynamic download (matches
114
+ Latin languages).
115
+ - Is suika's Korean support real? Needs verification.
116
+ - For Chinese, jieba vs jieba-ruby vs python-subprocess? Performance
117
+ and packaging tradeoffs.
@@ -0,0 +1,109 @@
1
+ # 55 — RTL support (T3.2)
2
+
3
+ ## Goal
4
+
5
+ Right-to-left script support: Arabic, Hebrew, Persian, Urdu. Covers
6
+ normalization, display, and tokenizer handling.
7
+
8
+ ## Per-language needs
9
+
10
+ ### Arabic (ar)
11
+
12
+ - **Normalizer**: NFC + ligature handling + diacritic stripping.
13
+ Arabic has many presentation forms (U+FE70..U+FEFF) that should be
14
+ canonicalized to base forms (U+0627..U+0645).
15
+ - **Tokenizer**: Arabic HAS spaces but morphological prefixes
16
+ (ال, و, ب, ف, ل) attach to words. Need light stemming for lookup.
17
+ - **Suggestions**: edit distance on Unicode codepoints works (unlike
18
+ CJK). Phonetic similarity via Soundex-like algorithm possible.
19
+
20
+ ### Hebrew (he)
21
+
22
+ - **Normalizer**: NFC + dagesh + niqqud handling. Modern Hebrew is
23
+ typically written without niqqud; for spell-checking, strip niqqud
24
+ before lookup.
25
+ - **Tokenizer**: Hebrew HAS spaces. No prefix issues as severe as
26
+ Arabic.
27
+ - **Suggestions**: edit distance on Hebrew codepoints.
28
+
29
+ ### Persian (fa)
30
+
31
+ - Like Arabic but with Persian-specific letters (گ چ پ ژ ک ی).
32
+ - Normalizer should map Arabic Yeh (ي) to Persian Yeh (ی).
33
+
34
+ ### Urdu (ur)
35
+
36
+ - Like Persian but written in Nastaliq style (calligraphic).
37
+ - Tokenization is the same as Persian; the visual rendering is a
38
+ display concern, not a spell-check concern.
39
+
40
+ ## Design
41
+
42
+ ### New normalizer base class
43
+
44
+ class Kotoshu::Language::Normalizer::Rtl < Base
45
+ # Subclasses: ArabicNormalizer, HebrewNormalizer, PersianNormalizer.
46
+ # Each overrides normalize_word(word) to do NFC + script-specific
47
+ # canonicalization.
48
+ end
49
+
50
+ ### Tokenizer flag
51
+
52
+ `Kotoshu::Language::Base#rtl?` already returns false for Latin and
53
+ true for Arabic (verify). Use this flag to:
54
+
55
+ 1. Pick the right normalizer.
56
+ 2. Tell the CLI batch reporter to align output RTL.
57
+ 3. Tell the tokenizer to handle RTL marks (U+200E, U+200F, U+202A-E).
58
+
59
+ ### RTL mark handling
60
+
61
+ RTL/LTR marks are zero-width characters that affect display. The
62
+ tokenizer should:
63
+
64
+ 1. Strip marks before tokenizing (so they don't become "characters"
65
+ in a word).
66
+ 2. Preserve them in the source mapping (so the editor shows them
67
+ correctly).
68
+
69
+ The document API (TODO 50) already supports this naturally: marks
70
+ live in the source between text nodes; text nodes themselves are
71
+ mark-free.
72
+
73
+ ## Phases
74
+
75
+ ### Phase 1 — Arabic normalizer
76
+ - New `lib/kotoshu/language/normalizers/arabic.rb`.
77
+ - NFC + presentation-form canonicalization + optional diacritic strip.
78
+ - Specs with the existing Arabic fixtures in `data/ar/`.
79
+
80
+ ### Phase 2 — Arabic tokenizer + dictionary integration
81
+ - Verify `languages/ar/language.rb` uses the new normalizer.
82
+ - Wire Arabic into the `Language::Registry` and the spell-checker
83
+ pipeline.
84
+ - Specs with a small Arabic word list.
85
+
86
+ ### Phase 3 — Hebrew + Persian normalizers
87
+ - Same shape as Arabic; different canonicalization rules.
88
+ - Specs per language.
89
+
90
+ ### Phase 4 — RTL-aware display in CLI
91
+ - `cli/display_formatter.rb` aligns error context RTL when the
92
+ detected language is RTL.
93
+ - `cli/batch_reporter.rb` aligns suggestions RTL.
94
+
95
+ ## Acceptance criteria
96
+
97
+ - [ ] Arabic text is normalized identically regardless of input form
98
+ (presentation form vs base form).
99
+ - [ ] `kotoshu check ARABIC_FILE` reports errors with correct source
100
+ positions, RTL-aligned context.
101
+ - [ ] Same for Hebrew and Persian.
102
+ - [ ] The pre-existing `spec/integrational/fixtures/right_to_left_mark.*`
103
+ fixtures pass.
104
+ - [ ] Full suite stays green.
105
+
106
+ ## Dependencies
107
+
108
+ - **Blocked by:** TODO 50 (document API for source positions).
109
+ - **Blocks:** nothing.
@@ -0,0 +1,126 @@
1
+ # 56 — T4+ Quality, Architecture, and Ecosystem backlog
2
+
3
+ This file rolls up the T4 (Quality), T5 (Architecture), and T6
4
+ (Ecosystem) items from TODO.impl/39-tier3-and-beyond.md into
5
+ promotable units with concrete acceptance criteria.
6
+
7
+ ## T4 — Quality
8
+
9
+ ### T4.1 — Performance pass
10
+
11
+ Status: outline. Baseline measurements needed before optimization.
12
+
13
+ - `spec/benchmark/` and `spec/performance/` exist but are skipped by
14
+ default (SLOW_TESTS=1).
15
+ - Establish baselines for:
16
+ - Cold-start time (time to first `correct?` call).
17
+ - Lookup throughput (correct? calls/sec on a 100k-word dictionary).
18
+ - Suggest latency at 1k / 10k / 100k dictionary sizes.
19
+ - Profile the hot path (`Kotoshu.correct?`); attack the biggest
20
+ offender. Likely candidates:
21
+ - `IndexedDictionary` lookup: is it O(1)?
22
+ - `Suggestions::Generator#generate`: is the strategy composition
23
+ optimal?
24
+ - `Trie` traversal: any obvious wins?
25
+ - Consider Memoization, Trie compression, Bloom filter tuning.
26
+
27
+ ### T4.2 — CI matrix
28
+
29
+ - Workflow that runs `bundle exec rspec --tag ~network --tag ~onnx`
30
+ on every push (already done).
31
+ - Workflow that runs `bundle exec rubocop` on every push (already done).
32
+ - Nightly workflow on main that runs `NETWORK_TESTS=1` (already done).
33
+ - Nightly workflow on main that runs `ONNX_TESTS=1` with cached models
34
+ (already done).
35
+ - Add: weekly workflow that runs `SLOW_TESTS=1` benchmarks and
36
+ publishes deltas to a dashboard.
37
+
38
+ ### T4.3 — Property-based testing
39
+
40
+ Status: skeleton exists in `spec/properties/`.
41
+
42
+ - Trie: insertion is order-independent; lookup is total.
43
+ - Suggestion generator: `suggestions.include?(word)` implies
44
+ `dictionary.lookup?(word)` is true (no out-of-dictionary
45
+ suggestions).
46
+ - Hunspell affix application: affixed forms round-trip through the
47
+ stemmer.
48
+
49
+ Expand each with at least 5 properties.
50
+
51
+ ### T4.4 — Documentation pass
52
+
53
+ - README.adoc: audit against actual CLI shape; remove 0.2-era
54
+ examples.
55
+ - `docs/`: prune superseded planning docs; mark historical ones as
56
+ historical.
57
+ - YARD: every public API has a docstring; private APIs are marked
58
+ `@private`.
59
+ - Add API docs publishing to CI (gh-pages).
60
+
61
+ ## T5 — Architecture
62
+
63
+ ### T5.1 — MECE refactor of `suggestions/strategies/`
64
+
65
+ Current strategies overlap (e.g. `symspell` and `edit_distance` both
66
+ generate edit-distance-1 candidates). Refactor:
67
+
68
+ - Each strategy owns a *unique* candidate-generation mechanism.
69
+ - `CompositeStrategy` merges and ranks, doesn't dedupe by side
70
+ effect.
71
+ - Strategies are pure functions of `(word, dictionary) → candidates`;
72
+ no shared state, no global config reads.
73
+
74
+ ### T5.2 — Configuration as data, not singleton
75
+
76
+ `Configuration.instance` is a process-wide singleton. Hard to test,
77
+ hard to scope per-document. Refactor:
78
+
79
+ - Pass a `Configuration` instance explicitly to `Spellchecker.new`.
80
+ - Default the facade (`Kotoshu.correct?`) to a process-default
81
+ instance.
82
+ - Drop the singleton.
83
+
84
+ ### T5.3 — Result model consistency
85
+
86
+ `WordResult` / `DocumentResult` / `SuggestionSet` migrated to
87
+ `lutaml-model` in 0.3.0. Audit:
88
+
89
+ - All three have consistent `to_hash` / `from_hash` shapes.
90
+ - No hand-rolled `to_h` anywhere (per global rule).
91
+ - CLI consumers use the framework-supplied serialization.
92
+
93
+ ### T5.4 — Plugin architecture formalization
94
+
95
+ See TODO.impl/53.
96
+
97
+ ## T6 — Ecosystem
98
+
99
+ ### T6.1 — VSCode / LSP integration
100
+
101
+ A language server that runs kotoshu on save, reports diagnostics via
102
+ LSP. Probably a separate gem (`kotoshu-lsp`).
103
+
104
+ ### T6.2 — Web playground
105
+
106
+ A WASM build of kotoshu running in the browser. Requires a pure-Ruby
107
+ path (no native extensions) and a reasonable dictionary size budget.
108
+
109
+ ### T6.3 — Dictionary editor GUI
110
+
111
+ A desktop app (Tauri / Electron) for curating personal dictionaries
112
+ and custom affix files. Reads/writes Hunspell format.
113
+
114
+ ## Acceptance criteria per promotable item
115
+
116
+ Each item, when promoted from this file to its own
117
+ `TODO.impl/{n}-{name}.md`, gets its own acceptance criteria. The
118
+ overarching rule: **never break T1/T2 correctness for a T3+ feature**.
119
+ Re-run the full `--tag ~network --tag ~onnx` suite before tagging any
120
+ T3+ release.
121
+
122
+ ## Dependencies
123
+
124
+ - **Blocked by:** T1 (`TODO.impl/36`) and T2 (`TODO.impl/37`) — the
125
+ baseline must be green before adding features.
126
+ - **Blocks:** nothing. This is the long horizon.
@@ -0,0 +1,138 @@
1
+ # 57 — T5.1 step 3: Strategies as pure functions
2
+
3
+ ## Goal
4
+
5
+ Finish the MECE refactor of `suggestions/strategies/` started in
6
+ TODO 56 T5.1. Steps 1 and 2 are done (PRs #75, #76):
7
+
8
+ - Step 1 (PR #75): Extracted `SuggestionSet#deduplicate!` as a named
9
+ method with a `duplicates_removed` counter. Was previously buried
10
+ as a side effect inside `sort_and_limit!`.
11
+ - Step 2 (PR #76): `CompositeStrategy#generate` now collects all
12
+ strategy outputs via `flat_map` and builds one `SuggestionSet` at
13
+ the end — was running sort/dedup/limit N times per merge.
14
+
15
+ Step 3 remains: make strategies **pure functions** of
16
+ `(word, dictionary) → candidates`, with no shared state and no
17
+ global config reads.
18
+
19
+ ## Current state (what's wrong)
20
+
21
+ `EditDistanceStrategy` is the worst offender. Its constructor:
22
+
23
+ ```ruby
24
+ def initialize(name: :edit_distance, language_code: 'en',
25
+ keyboard_layout: nil, **config)
26
+ super(name: name, **config)
27
+ @language_code = language_code
28
+ @keyboard_layout = keyboard_layout
29
+ @frequency_data = load_frequency_data(language_code) # disk IO at construction
30
+ @keyboard_layout = resolve_keyboard_layout(keyboard_layout)
31
+ end
32
+ ```
33
+
34
+ Consequences:
35
+
36
+ - Strategy is **bound to one language** at construction. Switching
37
+ languages requires a new strategy instance (and re-loads frequency
38
+ data from disk).
39
+ - Strategy reads `Kotoshu::Cache::FrequencyCache` (global state) via
40
+ `load_frequency_data`. Hard to test without network/disk fixtures.
41
+ - Strategy is **stateful** — same instance returns different scores
42
+ for the same `(word, dictionary)` pair if the underlying cache
43
+ changes between calls.
44
+ - Per-document configuration (different frequency list, different
45
+ keyboard layout) requires reconstructing the strategy pipeline.
46
+
47
+ `SymspellStrategy`, `NgramStrategy`, `PhoneticStrategy` have the
48
+ same shape; `SemanticStrategy` already takes its model via context.
49
+
50
+ ## Target design
51
+
52
+ Strategies become stateless. Per-call state (language, keyboard,
53
+ frequency data) flows through `Suggestions::Context`.
54
+
55
+ ```ruby
56
+ class EditDistanceStrategy < BaseStrategy
57
+ def generate(context)
58
+ keyboard = context.keyboard_layout # was @keyboard_layout
59
+ frequency = context.frequency_data # was @frequency_data
60
+ # ... pure function of (context.word, context.dictionary, keyboard, frequency)
61
+ end
62
+ end
63
+ ```
64
+
65
+ `Suggestions::Context` gains accessors for the data the strategies
66
+ need. A `ContextBuilder` (new) is responsible for loading the
67
+ language-specific data lazily, keyed by `language_code` and
68
+ `dictionary.language_code`. The builder is the only place that
69
+ touches `Cache::FrequencyCache` and `Keyboard::Registry`.
70
+
71
+ ## Phases
72
+
73
+ ### Phase A — Move keyboard_layout to Context
74
+
75
+ `KeyboardProximityStrategy` and `EditDistanceStrategy` both currently
76
+ hold `@keyboard_layout`. Move to Context. Strategies read
77
+ `context.keyboard_layout`.
78
+
79
+ ### Phase B — Move frequency_data to Context
80
+
81
+ `EditDistanceStrategy` currently calls `load_frequency_data` in
82
+ `initialize`. Move the loader into a `Suggestions::ContextBuilder`
83
+ that produces a Context with `frequency_data` populated.
84
+
85
+ ### Phase C — Drop strategy configuration coupling
86
+
87
+ Stop passing `language_code` to strategy constructors. The language
88
+ is already on `context.dictionary.language_code`. Strategies read
89
+ language from context, not from their own configuration.
90
+
91
+ ### Phase D — Verify strategies are pure
92
+
93
+ Add a property test:
94
+
95
+ ```ruby
96
+ it "strategy.generate(ctx) returns the same SuggestionSet across calls" do
97
+ # ... generate twice with same context, expect identical results
98
+ end
99
+ ```
100
+
101
+ ## Acceptance criteria
102
+
103
+ - [ ] No strategy reads `Cache::FrequencyCache` or
104
+ `Keyboard::Registry` directly.
105
+ - [ ] No strategy accepts `language_code` or `keyboard_layout` in
106
+ its constructor.
107
+ - [ ] `Suggestions::Context` carries all per-call state.
108
+ - [ ] Strategies pass a purity property test (same context → same
109
+ result, no global-state-dependent variation).
110
+ - [ ] Existing strategy specs continue to pass (or are updated to
111
+ build context with the new state).
112
+ - [ ] Generator (orchestrator) constructs the ContextBuilder once
113
+ per pipeline run, not once per strategy.
114
+
115
+ ## Why this matters
116
+
117
+ - **Per-document configuration.** A multi-language document needs
118
+ different strategies per segment. Today, the strategy instances
119
+ are bound to one language at construction — `MultiLanguageChecker`
120
+ has to rebuild the pipeline per segment.
121
+ - **Testing.** Stateful strategies that hit disk in their
122
+ constructor are slow to test and require network/disk fixtures.
123
+ Pure-function strategies only need a Context value object.
124
+ - **Parallelism.** `ParallelChecker` can't safely share strategy
125
+ instances across threads today because the strategies hold mutable
126
+ state. Pure functions are trivially thread-safe.
127
+
128
+ ## Dependencies
129
+
130
+ - **Blocked by:** none.
131
+ - **Blocks:** TODO 52 (multi-language document checking) becomes
132
+ much simpler once strategies are pure.
133
+
134
+ ## Out of scope
135
+
136
+ - Changing the suggestion algorithm itself.
137
+ - Removing the `BaseStrategy` inheritance hierarchy.
138
+ - Renaming or restructuring the existing strategy classes.