kotoshu 1.0.2-x86_64-linux

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (458) hide show
  1. checksums.yaml +7 -0
  2. data/.kotoshu-baseline.json +1 -0
  3. data/.pre-commit-hooks.yaml +18 -0
  4. data/.rspec +3 -0
  5. data/.rubocop.yml +351 -0
  6. data/.yardopts +8 -0
  7. data/CHANGELOG.md +813 -0
  8. data/CLAUDE.md +176 -0
  9. data/CODE_OF_CONDUCT.md +132 -0
  10. data/Cargo.lock +359 -0
  11. data/Cargo.toml +15 -0
  12. data/LICENSE +31 -0
  13. data/README.adoc +1292 -0
  14. data/Rakefile +150 -0
  15. data/SECURITY.md +93 -0
  16. data/TODO.impl/00-cut-0.1.md +121 -0
  17. data/TODO.impl/00-cut-0.2.md +175 -0
  18. data/TODO.impl/00-vision.md +281 -0
  19. data/TODO.impl/01-hunspell-correctness.md +111 -0
  20. data/TODO.impl/02-cli-unification.md +74 -0
  21. data/TODO.impl/03-dynamic-download.md +107 -0
  22. data/TODO.impl/04-language-modules.md +89 -0
  23. data/TODO.impl/05-semantic-path.md +83 -0
  24. data/TODO.impl/06-cjk-support.md +105 -0
  25. data/TODO.impl/07-rtl-support.md +71 -0
  26. data/TODO.impl/08-grammar-engine.md +106 -0
  27. data/TODO.impl/09-integrity-security.md +90 -0
  28. data/TODO.impl/10-testing-ci.md +98 -0
  29. data/TODO.impl/100-full-feature-batch3.md +28 -0
  30. data/TODO.impl/101-int4-tier-experiment.md +32 -0
  31. data/TODO.impl/102-lid-chain.md +28 -0
  32. data/TODO.impl/103-oov-bucket-tables.md +27 -0
  33. data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
  34. data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
  35. data/TODO.impl/106-native-lid-detection.md +36 -0
  36. data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
  37. data/TODO.impl/108-ko-ne-full-feature.md +37 -0
  38. data/TODO.impl/109-ten-audit.md +26 -0
  39. data/TODO.impl/11-release-v1.md +95 -0
  40. data/TODO.impl/110-sr-latn-and-polish.md +33 -0
  41. data/TODO.impl/111-embedding-bakeoff.md +33 -0
  42. data/TODO.impl/112-deployability-and-gating.md +26 -0
  43. data/TODO.impl/113-language-packs.md +26 -0
  44. data/TODO.impl/114-typo-biencoder-thread.md +33 -0
  45. data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
  46. data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
  47. data/TODO.impl/117-deterministic-frequency-tiers.md +110 -0
  48. data/TODO.impl/118-server-gem-floor.md +33 -0
  49. data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
  50. data/TODO.impl/120-setup-metadata-refresh.md +35 -0
  51. data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
  52. data/TODO.impl/122-pack-compression-kpk2.md +66 -0
  53. data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
  54. data/TODO.impl/124-owner-residue.md +19 -0
  55. data/TODO.impl/125-consumer-floor-guards.md +43 -0
  56. data/TODO.impl/126-tier-error-budgets.md +33 -0
  57. data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
  58. data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
  59. data/TODO.impl/129-pt-fr-corpora.md +23 -0
  60. data/TODO.impl/130-news-truth-0912.md +23 -0
  61. data/TODO.impl/131-hybrid-productization.md +42 -0
  62. data/TODO.impl/132-techniques-page-and-prose.md +27 -0
  63. data/TODO.impl/133-rust-first-distribution.md +68 -0
  64. data/TODO.impl/20-encapsulation-cleanup.md +47 -0
  65. data/TODO.impl/21-autoload-migration.md +64 -0
  66. data/TODO.impl/22-url-pin-single-source-of-truth.md +120 -0
  67. data/TODO.impl/23-fix-url-defaults.md +57 -0
  68. data/TODO.impl/24-cli-auto-setup-prompt.md +76 -0
  69. data/TODO.impl/25-soft-onnxruntime-require.md +75 -0
  70. data/TODO.impl/26-onnx-vocab-json-generation.md +96 -0
  71. data/TODO.impl/27-end-to-end-smoke-spec.md +113 -0
  72. data/TODO.impl/28-readme-quickstart-verification.md +68 -0
  73. data/TODO.impl/29-status-command.md +84 -0
  74. data/TODO.impl/30-language-auto-detection.md +74 -0
  75. data/TODO.impl/31-download-progress.md +58 -0
  76. data/TODO.impl/32-sarif-output-verification.md +72 -0
  77. data/TODO.impl/33-audit-log-rotation.md +54 -0
  78. data/TODO.impl/34-cache-eviction.md +59 -0
  79. data/TODO.impl/35-shell-completion.md +61 -0
  80. data/TODO.impl/36-cleanup-regressions-0.3.1.md +134 -0
  81. data/TODO.impl/37-hunspell-correctness-tier2.md +203 -0
  82. data/TODO.impl/38-onnx-semantic-gating.md +218 -0
  83. data/TODO.impl/39-tier3-and-beyond.md +232 -0
  84. data/TODO.impl/40-spec-drift-cleanup.md +134 -0
  85. data/TODO.impl/50-structure-aware-document-api.md +194 -0
  86. data/TODO.impl/51-grammar-engine-expansion.md +75 -0
  87. data/TODO.impl/52-multi-language-documents.md +95 -0
  88. data/TODO.impl/53-document-plugin-architecture.md +94 -0
  89. data/TODO.impl/54-cjk-support.md +117 -0
  90. data/TODO.impl/55-rtl-support.md +109 -0
  91. data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
  92. data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
  93. data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
  94. data/TODO.impl/59-yard-api-documentation.md +75 -0
  95. data/TODO.impl/60-lsp-server.md +174 -0
  96. data/TODO.impl/61-editor-ecosystem.md +167 -0
  97. data/TODO.impl/62-framework-integrations.md +159 -0
  98. data/TODO.impl/63-web-playground-wasm.md +186 -0
  99. data/TODO.impl/64-http-api-and-sdks.md +191 -0
  100. data/TODO.impl/65-universal-kotoshu.md +251 -0
  101. data/TODO.impl/66-kotoshu-core.md +169 -0
  102. data/TODO.impl/67-kotoshu-rs-and-access-libraries.md +137 -0
  103. data/TODO.impl/68-sota-adoption.md +152 -0
  104. data/TODO.impl/69-sota-typo-eval.md +74 -0
  105. data/TODO.impl/70-sota-confidence-cascade.md +59 -0
  106. data/TODO.impl/71-sota-nested-model.md +63 -0
  107. data/TODO.impl/72-model2vec-evaluation.md +49 -0
  108. data/TODO.impl/73-rl-reranker.md +42 -0
  109. data/TODO.impl/74-distilled-reranker.md +36 -0
  110. data/TODO.impl/75-llm-evaluator.md +35 -0
  111. data/TODO.impl/76-optional-llm-backend.md +37 -0
  112. data/TODO.impl/77-model-coverage-expansion.md +96 -0
  113. data/TODO.impl/78-in-browser-playground.md +69 -0
  114. data/TODO.impl/79-news-and-releases.md +60 -0
  115. data/TODO.impl/80-ecosystem-docs.md +56 -0
  116. data/TODO.impl/81-editor-reach.md +68 -0
  117. data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
  118. data/TODO.impl/83-models-batch-2.md +56 -0
  119. data/TODO.impl/84-language-modules-wave2.md +75 -0
  120. data/TODO.impl/85-wasm-model-api.md +48 -0
  121. data/TODO.impl/86-site-truth-wave2.md +45 -0
  122. data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
  123. data/TODO.impl/88-cli-directory-mode.md +54 -0
  124. data/TODO.impl/89-framework-integrations-exec.md +59 -0
  125. data/TODO.impl/90-gem-correctness-residues.md +44 -0
  126. data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
  127. data/TODO.impl/92-tier-mirror-cors.md +88 -0
  128. data/TODO.impl/93-python-wheels.md +32 -0
  129. data/TODO.impl/94-site-integrations-docs.md +35 -0
  130. data/TODO.impl/95-editor-action-polish.md +27 -0
  131. data/TODO.impl/96-server-semantic-models.md +50 -0
  132. data/TODO.impl/97-yard-execution.md +73 -0
  133. data/TODO.impl/98-campaign-cleanup.md +78 -0
  134. data/TODO.impl/99-site-truth-performance-docs.md +31 -0
  135. data/conformance/vectors.jsonl +2630 -0
  136. data/data/en/grammar/rules.yaml +851 -0
  137. data/docs/STABILITY-1.0-AUDIT.md +251 -0
  138. data/docs/STABILITY-1.0.md +137 -0
  139. data/docs/cache-architecture.md +502 -0
  140. data/docs/verification-evidence/correct3/pr_body.md +37 -0
  141. data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
  142. data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
  143. data/docs/verification-evidence/correct3/sv.aff +856 -0
  144. data/docs/verification-evidence/correct3/sv.dic +153717 -0
  145. data/examples/01_basic_word_checking.rb +38 -0
  146. data/examples/02_text_document_checking.rb +77 -0
  147. data/examples/03_dictionary_backends.rb +137 -0
  148. data/examples/04_trie_data_structure.rb +146 -0
  149. data/examples/05_suggestion_algorithms.rb +239 -0
  150. data/examples/06_configuration_advanced.rb +287 -0
  151. data/examples/07_multi_language_dictionaries.rb +278 -0
  152. data/exe/kotoshu +6 -0
  153. data/lib/kotoshu/algorithms/capitalization.rb +276 -0
  154. data/lib/kotoshu/algorithms/edit_distance.rb +128 -0
  155. data/lib/kotoshu/algorithms/lookup/affix_form.rb +132 -0
  156. data/lib/kotoshu/algorithms/lookup/compound_checks.rb +209 -0
  157. data/lib/kotoshu/algorithms/lookup/compound_form.rb +50 -0
  158. data/lib/kotoshu/algorithms/lookup.rb +841 -0
  159. data/lib/kotoshu/algorithms/ngram_suggest.rb +367 -0
  160. data/lib/kotoshu/algorithms/permutations.rb +283 -0
  161. data/lib/kotoshu/algorithms/phonet_suggest.rb +162 -0
  162. data/lib/kotoshu/algorithms/soundex.rb +92 -0
  163. data/lib/kotoshu/algorithms/suggest.rb +794 -0
  164. data/lib/kotoshu/algorithms.rb +23 -0
  165. data/lib/kotoshu/analyzers/semantic_analyzer.rb +405 -0
  166. data/lib/kotoshu/analyzers.rb +8 -0
  167. data/lib/kotoshu/baseline/application.rb +18 -0
  168. data/lib/kotoshu/baseline/entry.rb +33 -0
  169. data/lib/kotoshu/baseline/store.rb +175 -0
  170. data/lib/kotoshu/baseline.rb +20 -0
  171. data/lib/kotoshu/cache/base_cache.rb +778 -0
  172. data/lib/kotoshu/cache/cache.rb +91 -0
  173. data/lib/kotoshu/cache/eviction_policy.rb +66 -0
  174. data/lib/kotoshu/cache/frequency_cache.rb +283 -0
  175. data/lib/kotoshu/cache/language_cache.rb +511 -0
  176. data/lib/kotoshu/cache/lookup_cache.rb +166 -0
  177. data/lib/kotoshu/cache/model_cache.rb +1147 -0
  178. data/lib/kotoshu/cache/model_registry.rb +109 -0
  179. data/lib/kotoshu/cache/staged_languages.rb +45 -0
  180. data/lib/kotoshu/cache/suggestion_cache.rb +113 -0
  181. data/lib/kotoshu/cache.rb +49 -0
  182. data/lib/kotoshu/cli/auto_setup.rb +71 -0
  183. data/lib/kotoshu/cli/baseline_command.rb +119 -0
  184. data/lib/kotoshu/cli/batch_reporter.rb +321 -0
  185. data/lib/kotoshu/cli/cache_command.rb +282 -0
  186. data/lib/kotoshu/cli/completions_command.rb +210 -0
  187. data/lib/kotoshu/cli/dict_command.rb +64 -0
  188. data/lib/kotoshu/cli/directory_check.rb +146 -0
  189. data/lib/kotoshu/cli/directory_walker.rb +132 -0
  190. data/lib/kotoshu/cli/display_formatter.rb +429 -0
  191. data/lib/kotoshu/cli/errors.rb +36 -0
  192. data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
  193. data/lib/kotoshu/cli/interactive_reviewer.rb +314 -0
  194. data/lib/kotoshu/cli/language_resolver.rb +91 -0
  195. data/lib/kotoshu/cli/navigation_manager.rb +273 -0
  196. data/lib/kotoshu/cli/personal_command.rb +113 -0
  197. data/lib/kotoshu/cli/progress_reporter.rb +114 -0
  198. data/lib/kotoshu/cli/status_report.rb +130 -0
  199. data/lib/kotoshu/cli.rb +880 -0
  200. data/lib/kotoshu/components/passthrough_spell_checker.rb +70 -0
  201. data/lib/kotoshu/components/pos_tagger.rb +98 -0
  202. data/lib/kotoshu/components/spell_checker.rb +73 -0
  203. data/lib/kotoshu/components/synthesizer.rb +60 -0
  204. data/lib/kotoshu/components/tokenizer.rb +58 -0
  205. data/lib/kotoshu/components/whitespace_tokenizer.rb +94 -0
  206. data/lib/kotoshu/components.rb +13 -0
  207. data/lib/kotoshu/configuration/builder.rb +207 -0
  208. data/lib/kotoshu/configuration/resolver.rb +124 -0
  209. data/lib/kotoshu/configuration.rb +829 -0
  210. data/lib/kotoshu/conformance_exporter.rb +256 -0
  211. data/lib/kotoshu/conformance_runner.rb +176 -0
  212. data/lib/kotoshu/core/exceptions.rb +186 -0
  213. data/lib/kotoshu/core/indexed_dictionary.rb +293 -0
  214. data/lib/kotoshu/core/models/affix_rule.rb +262 -0
  215. data/lib/kotoshu/core/models/result/document_result.rb +132 -0
  216. data/lib/kotoshu/core/models/result/word_result.rb +131 -0
  217. data/lib/kotoshu/core/models/word.rb +131 -0
  218. data/lib/kotoshu/core/trie/builder.rb +119 -0
  219. data/lib/kotoshu/core/trie/node.rb +94 -0
  220. data/lib/kotoshu/core/trie/trie.rb +249 -0
  221. data/lib/kotoshu/core.rb +28 -0
  222. data/lib/kotoshu/data/common_words/de.yml +1800 -0
  223. data/lib/kotoshu/data/common_words/en.yml +1215 -0
  224. data/lib/kotoshu/data/common_words/es.yml +750 -0
  225. data/lib/kotoshu/data/common_words/fr.yml +1015 -0
  226. data/lib/kotoshu/data/common_words/pt.yml +870 -0
  227. data/lib/kotoshu/data/common_words/ru.yml +484 -0
  228. data/lib/kotoshu/data/common_words_loader.rb +141 -0
  229. data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
  230. data/lib/kotoshu/data/frozen_kelly.rb +38 -0
  231. data/lib/kotoshu/data.rb +8 -0
  232. data/lib/kotoshu/data_structures/bloom_filter.rb +188 -0
  233. data/lib/kotoshu/data_structures.rb +8 -0
  234. data/lib/kotoshu/debug_logger.rb +146 -0
  235. data/lib/kotoshu/debug_mode.rb +136 -0
  236. data/lib/kotoshu/defaults.rb +84 -0
  237. data/lib/kotoshu/dictionaries/catalog.rb +814 -0
  238. data/lib/kotoshu/dictionaries.rb +8 -0
  239. data/lib/kotoshu/dictionary/base.rb +324 -0
  240. data/lib/kotoshu/dictionary/cspell.rb +243 -0
  241. data/lib/kotoshu/dictionary/custom.rb +225 -0
  242. data/lib/kotoshu/dictionary/hunspell.rb +520 -0
  243. data/lib/kotoshu/dictionary/plain_text.rb +302 -0
  244. data/lib/kotoshu/dictionary/repository.rb +246 -0
  245. data/lib/kotoshu/dictionary/unified.rb +236 -0
  246. data/lib/kotoshu/dictionary/unix_words.rb +218 -0
  247. data/lib/kotoshu/dictionary.rb +15 -0
  248. data/lib/kotoshu/documents/document.rb +120 -0
  249. data/lib/kotoshu/documents/plain_text_document.rb +73 -0
  250. data/lib/kotoshu/documents/source_position.rb +45 -0
  251. data/lib/kotoshu/documents/source_range.rb +76 -0
  252. data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
  253. data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
  254. data/lib/kotoshu/documents/suppressions.rb +44 -0
  255. data/lib/kotoshu/documents/text_node.rb +49 -0
  256. data/lib/kotoshu/documents.rb +178 -0
  257. data/lib/kotoshu/embeddings/embedding_pipeline.rb +238 -0
  258. data/lib/kotoshu/embeddings/lru_cache.rb +244 -0
  259. data/lib/kotoshu/embeddings/onnx_runtime_model.rb +411 -0
  260. data/lib/kotoshu/embeddings/protocol.rb +96 -0
  261. data/lib/kotoshu/embeddings/protocols.rb +23 -0
  262. data/lib/kotoshu/embeddings/registry.rb +184 -0
  263. data/lib/kotoshu/embeddings/search.rb +204 -0
  264. data/lib/kotoshu/embeddings/similarity_engine.rb +278 -0
  265. data/lib/kotoshu/embeddings/similarity_search.rb +328 -0
  266. data/lib/kotoshu/embeddings/vocabulary.rb +306 -0
  267. data/lib/kotoshu/embeddings.rb +87 -0
  268. data/lib/kotoshu/fluent_checker.rb +94 -0
  269. data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +48 -0
  270. data/lib/kotoshu/grammar/pattern_matchers/double_negative_matcher.rb +179 -0
  271. data/lib/kotoshu/grammar/pattern_matchers/phrase_matcher.rb +86 -0
  272. data/lib/kotoshu/grammar/pattern_matchers/possessive_context_matcher.rb +75 -0
  273. data/lib/kotoshu/grammar/pattern_matchers/possessive_contraction_matcher.rb +87 -0
  274. data/lib/kotoshu/grammar/pattern_matchers/sentence_start_matcher.rb +84 -0
  275. data/lib/kotoshu/grammar/pattern_matchers/vowel_sound_matcher.rb +81 -0
  276. data/lib/kotoshu/grammar/pattern_matchers/word_list_matcher.rb +66 -0
  277. data/lib/kotoshu/grammar/rule.rb +99 -0
  278. data/lib/kotoshu/grammar/rule_engine.rb +96 -0
  279. data/lib/kotoshu/grammar/rule_loader.rb +30 -0
  280. data/lib/kotoshu/grammar.rb +24 -0
  281. data/lib/kotoshu/integrity/audit_log.rb +167 -0
  282. data/lib/kotoshu/integrity/manifest.rb +116 -0
  283. data/lib/kotoshu/integrity/net_http.rb +51 -0
  284. data/lib/kotoshu/integrity/rotation_policy.rb +88 -0
  285. data/lib/kotoshu/integrity.rb +26 -0
  286. data/lib/kotoshu/jekyll.rb +101 -0
  287. data/lib/kotoshu/keyboard/layout.rb +116 -0
  288. data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
  289. data/lib/kotoshu/keyboard/layouts/azerty.rb +55 -0
  290. data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
  291. data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
  292. data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
  293. data/lib/kotoshu/keyboard/layouts/dvorak.rb +54 -0
  294. data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
  295. data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
  296. data/lib/kotoshu/keyboard/layouts/hrsl.rb +94 -0
  297. data/lib/kotoshu/keyboard/layouts/jcuken.rb +57 -0
  298. data/lib/kotoshu/keyboard/layouts/latin.rb +266 -0
  299. data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
  300. data/lib/kotoshu/keyboard/layouts/qwerty.rb +52 -0
  301. data/lib/kotoshu/keyboard/layouts/qwertz.rb +55 -0
  302. data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
  303. data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
  304. data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
  305. data/lib/kotoshu/keyboard/registry.rb +201 -0
  306. data/lib/kotoshu/keyboard.rb +96 -0
  307. data/lib/kotoshu/kotoshu_native.so +0 -0
  308. data/lib/kotoshu/language/detector.rb +242 -0
  309. data/lib/kotoshu/language/identifier.rb +376 -0
  310. data/lib/kotoshu/language/languages/base.rb +271 -0
  311. data/lib/kotoshu/language/lid_detector.rb +188 -0
  312. data/lib/kotoshu/language/normalizer/arabic.rb +113 -0
  313. data/lib/kotoshu/language/normalizer/base.rb +133 -0
  314. data/lib/kotoshu/language/normalizer/greek.rb +63 -0
  315. data/lib/kotoshu/language/normalizer/hebrew.rb +72 -0
  316. data/lib/kotoshu/language/normalizer/persian.rb +36 -0
  317. data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
  318. data/lib/kotoshu/language/registry.rb +209 -0
  319. data/lib/kotoshu/language/resources/ar/common_words.txt +6753 -0
  320. data/lib/kotoshu/language/resources/ar/confusion_sets.txt +11 -0
  321. data/lib/kotoshu/language/resources/de/common_words.txt +10003 -0
  322. data/lib/kotoshu/language/resources/de/confusion_sets.txt +246 -0
  323. data/lib/kotoshu/language/resources/en/common_words.txt +9979 -0
  324. data/lib/kotoshu/language/resources/en/confusion_sets.txt +871 -0
  325. data/lib/kotoshu/language/resources/es/common_words.txt +9992 -0
  326. data/lib/kotoshu/language/resources/es/confusion_sets.txt +17 -0
  327. data/lib/kotoshu/language/resources/fr/common_words.txt +9993 -0
  328. data/lib/kotoshu/language/resources/fr/confusion_sets.txt +76 -0
  329. data/lib/kotoshu/language/resources/pt/common_words.txt +9977 -0
  330. data/lib/kotoshu/language/resources/pt/confusion_sets.txt +18 -0
  331. data/lib/kotoshu/language/resources/ru/common_words.txt +9951 -0
  332. data/lib/kotoshu/language/resources/ru/confusion_sets.txt +5 -0
  333. data/lib/kotoshu/language/script.rb +128 -0
  334. data/lib/kotoshu/language/segmenter.rb +122 -0
  335. data/lib/kotoshu/language/suika.rb +38 -0
  336. data/lib/kotoshu/language/tokenizer/base.rb +187 -0
  337. data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
  338. data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
  339. data/lib/kotoshu/language/tokenizer/french_tokenizer.rb +172 -0
  340. data/lib/kotoshu/language/tokenizer/german_tokenizer.rb +41 -0
  341. data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
  342. data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
  343. data/lib/kotoshu/language/tokenizer/japanese_tokenizer.rb +55 -0
  344. data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +153 -0
  345. data/lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb +160 -0
  346. data/lib/kotoshu/language/tokenizer/russian_tokenizer.rb +93 -0
  347. data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
  348. data/lib/kotoshu/language/tokenizer/spanish_tokenizer.rb +122 -0
  349. data/lib/kotoshu/language.rb +127 -0
  350. data/lib/kotoshu/languages/ar/language.rb +98 -0
  351. data/lib/kotoshu/languages/bg/language.rb +37 -0
  352. data/lib/kotoshu/languages/ca/language.rb +22 -0
  353. data/lib/kotoshu/languages/cs/language.rb +22 -0
  354. data/lib/kotoshu/languages/da/language.rb +22 -0
  355. data/lib/kotoshu/languages/de/language.rb +554 -0
  356. data/lib/kotoshu/languages/el/language.rb +50 -0
  357. data/lib/kotoshu/languages/en/language.rb +470 -0
  358. data/lib/kotoshu/languages/es/language.rb +476 -0
  359. data/lib/kotoshu/languages/et/language.rb +24 -0
  360. data/lib/kotoshu/languages/fa/language.rb +91 -0
  361. data/lib/kotoshu/languages/fr/language.rb +505 -0
  362. data/lib/kotoshu/languages/he/language.rb +98 -0
  363. data/lib/kotoshu/languages/hr/language.rb +24 -0
  364. data/lib/kotoshu/languages/hu/language.rb +22 -0
  365. data/lib/kotoshu/languages/id/language.rb +25 -0
  366. data/lib/kotoshu/languages/it/language.rb +22 -0
  367. data/lib/kotoshu/languages/ja/language.rb +476 -0
  368. data/lib/kotoshu/languages/ko/language.rb +56 -0
  369. data/lib/kotoshu/languages/latin_base.rb +53 -0
  370. data/lib/kotoshu/languages/lt/language.rb +24 -0
  371. data/lib/kotoshu/languages/lv/language.rb +24 -0
  372. data/lib/kotoshu/languages/nb/language.rb +25 -0
  373. data/lib/kotoshu/languages/ne/language.rb +60 -0
  374. data/lib/kotoshu/languages/nl/language.rb +22 -0
  375. data/lib/kotoshu/languages/nn/language.rb +26 -0
  376. data/lib/kotoshu/languages/pl/language.rb +22 -0
  377. data/lib/kotoshu/languages/pt/language.rb +439 -0
  378. data/lib/kotoshu/languages/ro/language.rb +22 -0
  379. data/lib/kotoshu/languages/ru/language.rb +420 -0
  380. data/lib/kotoshu/languages/sk/language.rb +25 -0
  381. data/lib/kotoshu/languages/sl/language.rb +25 -0
  382. data/lib/kotoshu/languages/sr/language.rb +39 -0
  383. data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
  384. data/lib/kotoshu/languages/sv/language.rb +22 -0
  385. data/lib/kotoshu/languages/tr/language.rb +36 -0
  386. data/lib/kotoshu/languages/uk/language.rb +38 -0
  387. data/lib/kotoshu/languages/vi/language.rb +22 -0
  388. data/lib/kotoshu/languages.rb +64 -0
  389. data/lib/kotoshu/metrics_collector.rb +238 -0
  390. data/lib/kotoshu/metrics_module.rb +121 -0
  391. data/lib/kotoshu/models/context.rb +119 -0
  392. data/lib/kotoshu/models/embedding_model.rb +181 -0
  393. data/lib/kotoshu/models/fasttext_model.rb +212 -0
  394. data/lib/kotoshu/models/nearest_neighbor.rb +96 -0
  395. data/lib/kotoshu/models/onnx_model.rb +330 -0
  396. data/lib/kotoshu/models/semantic_error.rb +204 -0
  397. data/lib/kotoshu/models/suggestion.rb +106 -0
  398. data/lib/kotoshu/models/word_embedding.rb +97 -0
  399. data/lib/kotoshu/models.rb +27 -0
  400. data/lib/kotoshu/multi_language_checker.rb +127 -0
  401. data/lib/kotoshu/native.rb +83 -0
  402. data/lib/kotoshu/native_backend.rb +131 -0
  403. data/lib/kotoshu/paths.rb +79 -0
  404. data/lib/kotoshu/personal_dictionary.rb +105 -0
  405. data/lib/kotoshu/plugins/plugin.rb +61 -0
  406. data/lib/kotoshu/plugins/registry.rb +118 -0
  407. data/lib/kotoshu/plugins.rb +9 -0
  408. data/lib/kotoshu/project_config.rb +76 -0
  409. data/lib/kotoshu/readers/aff_data.rb +530 -0
  410. data/lib/kotoshu/readers/aff_reader.rb +393 -0
  411. data/lib/kotoshu/readers/condition_checker.rb +125 -0
  412. data/lib/kotoshu/readers/dic_reader.rb +208 -0
  413. data/lib/kotoshu/readers/file_reader.rb +377 -0
  414. data/lib/kotoshu/readers/lookup_builder.rb +584 -0
  415. data/lib/kotoshu/readers/readers.rb +17 -0
  416. data/lib/kotoshu/readers.rb +25 -0
  417. data/lib/kotoshu/resource_bundle.rb +55 -0
  418. data/lib/kotoshu/resource_manager.rb +600 -0
  419. data/lib/kotoshu/results/result.rb +165 -0
  420. data/lib/kotoshu/results.rb +8 -0
  421. data/lib/kotoshu/rspec.rb +163 -0
  422. data/lib/kotoshu/scripts/fasttext_to_onnx.py +275 -0
  423. data/lib/kotoshu/source_registry.rb +104 -0
  424. data/lib/kotoshu/spellchecker/parallel_checker.rb +90 -0
  425. data/lib/kotoshu/spellchecker.rb +406 -0
  426. data/lib/kotoshu/string_metrics.rb +153 -0
  427. data/lib/kotoshu/suggestions/context.rb +43 -0
  428. data/lib/kotoshu/suggestions/frequency_provider.rb +80 -0
  429. data/lib/kotoshu/suggestions/generator.rb +169 -0
  430. data/lib/kotoshu/suggestions/pipeline.rb +135 -0
  431. data/lib/kotoshu/suggestions/semantic_cascade.rb +73 -0
  432. data/lib/kotoshu/suggestions/strategies/base_strategy.rb +315 -0
  433. data/lib/kotoshu/suggestions/strategies/composite_strategy.rb +197 -0
  434. data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +631 -0
  435. data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +207 -0
  436. data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +165 -0
  437. data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +284 -0
  438. data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +317 -0
  439. data/lib/kotoshu/suggestions/strategies/symspell_strategy.rb +270 -0
  440. data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
  441. data/lib/kotoshu/suggestions/suggestion.rb +154 -0
  442. data/lib/kotoshu/suggestions/suggestion_set.rb +270 -0
  443. data/lib/kotoshu/suggestions/sweep_index.rb +137 -0
  444. data/lib/kotoshu/suggestions.rb +29 -0
  445. data/lib/kotoshu/tasks/check_task.rb +104 -0
  446. data/lib/kotoshu/tasks.rb +11 -0
  447. data/lib/kotoshu/validators/spelling_validator.rb +101 -0
  448. data/lib/kotoshu/validators.rb +9 -0
  449. data/lib/kotoshu/version.rb +5 -0
  450. data/lib/kotoshu.rb +538 -0
  451. data/rakelib/compile.rake +10 -0
  452. data/script/validate_all_dictionaries.rb +444 -0
  453. data/scripts/build_native_gem.rb +45 -0
  454. data/scripts/extract_eval_grids.rb +37 -0
  455. data/scripts/generate_frozen_tiers.rb +44 -0
  456. data/sig/kotoshu.rbs +435 -0
  457. data/test_oop.rb +79 -0
  458. metadata +546 -0
@@ -0,0 +1,502 @@
1
+ # Kotoshu Cache System Architecture
2
+
3
+ ## Overview
4
+
5
+ The Kotoshu cache system provides a unified, object-oriented architecture for managing linguistic resources (dictionaries, frequency lists, and embedding models). The system implements automatic download, local caching, metadata management, and TTL-based expiration.
6
+
7
+ ## Design Principles
8
+
9
+ ### 1. DRY (Don't Repeat Yourself)
10
+ All cache types inherit from `BaseCache`, eliminating code duplication for:
11
+ - HTTP downloads
12
+ - Metadata serialization/deserialization
13
+ - Cache validation (existence, expiration)
14
+ - Statistics tracking (hits, misses, hit rate)
15
+ - TTL management
16
+
17
+ ### 2. OOP (Object-Oriented Programming)
18
+ - Abstract base class defines the interface
19
+ - Concrete implementations provide type-specific behavior
20
+ - Template Method pattern for extensible operations
21
+ - Strategy pattern for resource type handling
22
+
23
+ ### 3. Separation of Concerns
24
+ - `BaseCache`: Common cache operations
25
+ - `LanguageCache`: Dictionary and grammar resources
26
+ - `ModelCache`: Embedding model resources
27
+ - `FrequencyCache`: Frequency list resources
28
+
29
+ ### 4. Single Responsibility
30
+ Each cache class has one responsibility:
31
+ - LanguageCache manages dictionaries
32
+ - ModelCache manages models
33
+ - FrequencyCache manages frequency lists
34
+
35
+ ## Architecture Diagram
36
+
37
+ ```
38
+ ┌─────────────────────────────────────────────────────────────────────────────┐
39
+ │ Application Layer │
40
+ │ (Kotoshu::SpellChecker, Kotoshu::Suggestions::Strategies::EditDistance) │
41
+ └─────────────────────────────────────────────────────────────────────────────┘
42
+ │
43
+ ▼
44
+ ┌─────────────────────────────────────────────────────────────────────────────┐
45
+ │ Cache Interface Layer │
46
+ │ (CLI: kotoshu cache ...) │
47
+ └─────────────────────────────────────────────────────────────────────────────┘
48
+ │
49
+ ┌─────────────────────┼─────────────────────┐
50
+ │ │ │
51
+ ▼ ▼ ▼
52
+ ┌───────────────────────────┐ ┌──────────────────┐ ┌─────────────────────────┐
53
+ │ LanguageCache │ │ ModelCache │ │ FrequencyCache │
54
+ │ (Dictionaries) │ │ (Embeddings) │ │ (Kelly Lists) │
55
+ ├───────────────────────────┤ ├──────────────────┤ ├─────────────────────────┤
56
+ │ + get_spelling() │ │ + get_fasttext() │ │ + get_frequency() │
57
+ │ + get_grammar() │ │ + get_onnx() │ │ + available_languages() │
58
+ │ + available_languages() │ │ + available_... │ │ │
59
+ └───────────┬───────────────┘ └──────┬───────────┘ └───────────┬─────────────┘
60
+ │ │ │
61
+ │ │ │
62
+ ▼ ▼ ▼
63
+ ┌─────────────────────────────────────────────────────────────────────────────┐
64
+ │ BaseCache (Abstract) │
65
+ │ ┌─────────────────────────────────────────────────────────────────────┐ │
66
+ │ │ Public API │ │
67
+ │ │ + get(resource_id, force_download: false) │ │
68
+ │ │ + available?(resource_id) │ │
69
+ │ │ + clear(resource_id) │ │
70
+ │ │ + clear_all() │ │
71
+ │ │ + stats │ │
72
+ │ │ + clean() │ │
73
+ │ └─────────────────────────────────────────────────────────────────────┘ │
74
+ │ ┌─────────────────────────────────────────────────────────────────────┐ │
75
+ │ │ Protected Abstract Methods (subclasses must implement) │ │
76
+ │ │ + download_resource(resource_id, dest_path) │ │
77
+ │ │ + load_cached(resource_id) │ │
78
+ │ │ + metadata_path_for(resource_id) │ │
79
+ │ │ + resource_dir_for(resource_id) │ │
80
+ │ │ + resource_files_exist?(resource_id) │ │
81
+ │ │ + supports_resource?(resource_id) │ │
82
+ │ │ + cached_resources │ │
83
+ │ └─────────────────────────────────────────────────────────────────────┘ │
84
+ │ ┌─────────────────────────────────────────────────────────────────────┐ │
85
+ │ │ Protected Helper Methods (available to subclasses) │ │
86
+ │ │ + download_url(url) │ │
87
+ │ │ + download_file(url, dest_path) │ │
88
+ │ │ + write_metadata(path, metadata) │ │
89
+ │ │ + read_metadata(path) │ │
90
+ │ │ + cached?(metadata_path) │ │
91
+ │ │ + expired?(metadata_path) │ │
92
+ │ │ + checksum(content) │ │
93
+ │ │ + parse_resource_id(resource_id) │ │
94
+ │ │ + extract_language(resource_id) │ │
95
+ │ │ + extract_type(resource_id) │ │
96
+ │ └─────────────────────────────────────────────────────────────────────┘ │
97
+ │ ┌─────────────────────────────────────────────────────────────────────┐ │
98
+ │ │ Private Methods │ │
99
+ │ │ + default_cache_path │ │
100
+ │ │ + default_url_base │ │
101
+ │ │ + default_cache_ttl │ │
102
+ │ └─────────────────────────────────────────────────────────────────────┘ │
103
+ └─────────────────────────────────────────────────────────────────────────────┘
104
+ │
105
+ ┌─────────────────────┼─────────────────────┐
106
+ │ │ │
107
+ ▼ ▼ ▼
108
+ ┌─────────────────────┐ ┌──────────────┐ ┌─────────────────────────┐
109
+ │ ~/.kotoshu/ │ │ ~/.kotoshu/ │ │ ~/.kotoshu/ │
110
+ │ languages/ │ │ models/ │ │ frequency-lists/ │
111
+ │ ┌─────────────────┐ │ │ ┌──────────┐ │ │ ┌─────────────────────┐ │
112
+ │ │ en/ │ │ │ │ en/ │ │ │ │ en/ │ │
113
+ │ │ ├── spelling/ │ │ │ │ └──models/│ │ │ │ ├── frequency.json │ │
114
+ │ │ │ ├── index.dic│ │ │ │ fasttext│ │ │ │ └── metadata.json │ │
115
+ │ │ │ └── index.aff│ │ │ │ onnx/ │ │ │ └─────────────────────┘ │
116
+ │ │ └── metadata.json│ │ │ │ fasttext│ │ │ │
117
+ │ └─────────────────┘ │ │ │ onnx │ │ │ ru/, ar/, etc. │
118
+ │ de/, es/, fr/, etc. │ │ └──────────┘ │ │ │
119
+ └─────────────────────┘ └──────────────┘ └─────────────────────────┘
120
+ ```
121
+
122
+ ## Resource ID Format
123
+
124
+ All resources use a colon-separated identifier format: `{language}:{type}`
125
+
126
+ ### Examples
127
+ - `en:spelling` - English spelling dictionary
128
+ - `en:fasttext` - English FastText model
129
+ - `en:frequency` - English frequency data (deprecated, use FrequencyCache)
130
+ - `ru:spelling` - Russian spelling dictionary
131
+ - `de:onnx` - German ONNX model
132
+ - `en:onnx:mini` - English mini-tier ONNX model (tiered form `{language}:onnx:{tier}`; tiers are `full`, `fluency`, `mini`; `full` uses the legacy two-part id)
133
+
134
+ ## Model Tiers and the Registry
135
+
136
+ `ModelCache` supports tiered ONNX model resources (`full`, `fluency`, `mini`), resolved through the resource registry (`registry.json`) published by `kotoshu/models-fasttext-onnx` under stable ids (`kotoshu://models/{lang}/{tier}`). Approximate sizes: `full` ~120 MB, `fluency` ~15-18 MB, `mini` ~3 MB.
137
+
138
+ ### Tiered on-disk layout
139
+
140
+ Tiered resource ids carry a third segment (`{language}:onnx:{tier}`). The `full` tier keeps the legacy two-part id (`{language}:onnx`) and the legacy on-disk layout, preserved byte-compatibly; `fluency` and `mini` nest under the onnx directory as sibling directories so tiers coexist:
141
+
142
+ ```
143
+ {model cache}/
144
+ └── en/
145
+ └── models/
146
+ ├── fasttext/
147
+ └── onnx/ # full tier (legacy layout, unchanged)
148
+ ├── fasttext.en.onnx
149
+ ├── fasttext.en.vocab.json
150
+ ├── metadata.json
151
+ ├── mini/ # tiered siblings
152
+ │ ├── fasttext.en.mini.onnx
153
+ │ ├── fasttext.en.mini.vocab.json
154
+ │ └── metadata.json
155
+ └── fluency/
156
+ └── ...
157
+ ```
158
+
159
+ `ModelCache#cached_tiers(language)` lists the tiers currently cached for a language with a disk-only lookup (it never touches the network, so it is resolve-safe), in the fixed preference order `mini`, `fluency`, `full`. That order exists so ambiguity errors are deterministic across machines and runs — it is a reporting order, not a fallback chain; tiers are never silently substituted.
160
+
161
+ ### The cached registry
162
+
163
+ `registry.json` is fetched at most once per cache instance and stored under `{model cache}/registry/`:
164
+
165
+ - `registry.json` — the raw registry bytes
166
+ - `metadata.json` — records the bytes' own SHA-256, source URL, and fetch time
167
+
168
+ The cached copy is reused while fresh (subject to the cache TTL). A copy that is absent, unparsable, or fails its recorded SHA-256 is treated as unusable and re-fetched rather than trusted.
169
+
170
+ `KOTOSHU_OFFLINE=1`: the registry is never fetched. A cached copy — even a stale one — is reused; when none is cached, `Kotoshu::Error` is raised (run once online, or copy `registry.json` there first).
171
+
172
+ The resolve path never consults the registry at all. It is cache-only, so a missing tier raises `ResourceNotSetupError` exactly like any other unset resource — resolve never downloads and never falls back to another tier.
173
+
174
+ ### Download and verification
175
+
176
+ Tiered downloads (`ModelCache#download_tiered_model`) raise rather than degrade to `nil`, so setup callers can report `:unavailable`:
177
+
178
+ 1. Look up the `(language, tier)` entry in the registry. No entry means the language publishes no such tier.
179
+ 2. Download the model from `urls.primary`, falling back to `urls.mirror` (partial bytes from a failed attempt are removed first).
180
+ 3. Verify the downloaded bytes' SHA-256 against the registry entry. On mismatch the corrupt file is deleted, the mismatch is recorded in the audit log, and `Kotoshu::IntegrityError` is raised with a remediation hint.
181
+ 4. Download the vocab sibling (a missing vocab is a warning, not a failure).
182
+
183
+ ## Cache Lifecycle
184
+
185
+ ### 1. Initial Access (Cache Miss)
186
+
187
+ ```
188
+ User calls cache.get('en:spelling')
189
+ │
190
+ ▼
191
+ BaseCache.get() checks:
192
+ - supports_resource?('en:spelling') → true
193
+ - cached?(metadata_path) → false (first access)
194
+ │
195
+ ▼
196
+ Download triggered:
197
+ - download_resource('en:spelling', dest_path)
198
+ - download_file(url, dest_path)
199
+ - write_metadata(metadata_path, metadata)
200
+ - track_miss
201
+ │
202
+ ▼
203
+ Return: { aff_path, dic_path, cached: false, metadata }
204
+ ```
205
+
206
+ ### 2. Subsequent Access (Cache Hit)
207
+
208
+ ```
209
+ User calls cache.get('en:spelling')
210
+ │
211
+ ▼
212
+ BaseCache.get() checks:
213
+ - supports_resource?('en:spelling') → true
214
+ - cached?(metadata_path) → true
215
+ - expired?(metadata_path) → false (within TTL)
216
+ │
217
+ ▼
218
+ Load from cache:
219
+ - load_cached('en:spelling')
220
+ - read_metadata(metadata_path)
221
+ - track_hit
222
+ │
223
+ ▼
224
+ Return: { aff_path, dic_path, cached: true, metadata }
225
+ ```
226
+
227
+ ### 3. Expired Cache Access
228
+
229
+ ```
230
+ User calls cache.get('en:spelling')
231
+ │
232
+ ▼
233
+ BaseCache.get() checks:
234
+ - supports_resource?('en:spelling') → true
235
+ - cached?(metadata_path) → true
236
+ - expired?(metadata_path) → true (older than 7 days)
237
+ │
238
+ ▼
239
+ Re-download triggered:
240
+ - download_resource('en:spelling', dest_path)
241
+ - (same as initial access)
242
+ ```
243
+
244
+ ## Metadata Format
245
+
246
+ All cached resources include metadata in JSON format:
247
+
248
+ ```json
249
+ {
250
+ "version": "2025-01-15T10:30:00Z",
251
+ "url": "https://raw.githubusercontent.com/...",
252
+ "language": "en",
253
+ "type": "spelling",
254
+ "checksum": "a1b2c3d4e5f6...",
255
+ "cached_at": "2025-01-15T10:30:00Z"
256
+ }
257
+ ```
258
+
259
+ ### Metadata Fields
260
+ - `version`: Timestamp when resource was cached
261
+ - `url`: Source URL for the resource
262
+ - `language`: ISO 639-1 language code
263
+ - `type`: Resource type (spelling, grammar, fasttext, onnx, kelly_frequency)
264
+ - `checksum`: SHA256 checksum for integrity verification
265
+ - `cached_at`: Timestamp of caching
266
+
267
+ ## Thread Safety
268
+
269
+ The cache system is designed for single-threaded use. For multi-threaded environments:
270
+
271
+ 1. **Process-based parallelism**: Each process has its own cache
272
+ 2. **File locking**: Not implemented (future enhancement)
273
+ 3. **Recommended**: Use separate cache instances per thread
274
+
275
+ ## Error Handling
276
+
277
+ ### Download Failures
278
+ - HTTP errors raise `RuntimeError` with details
279
+ - Invalid JSON raises `JSON::ParserError`
280
+ - File I/O errors propagate to caller
281
+
282
+ ### Cache Validation
283
+ - Missing metadata returns `nil`
284
+ - Expired cache triggers re-download
285
+ - Invalid checksum triggers re-download (future)
286
+
287
+ ## Performance Considerations
288
+
289
+ ### Cache Hit Rates
290
+ Target hit rates by cache type:
291
+ - **LanguageCache**: 85-95% (dictionaries rarely change)
292
+ - **FrequencyCache**: 95-99% (frequency lists very stable)
293
+ - **ModelCache**: 98-99% (models rarely change)
294
+
295
+ ### Disk Usage
296
+ Typical disk usage per language:
297
+ - **Dictionary**: ~2-4 MB (Hunspell files)
298
+ - **Frequency**: ~800 KB (Kelly JSON)
299
+ - **Models**: ~600-2000 MB (FastText vectors, varies by language)
300
+
301
+ ### Download Speeds
302
+ Typical download times (broadband):
303
+ - **Dictionary**: 2-5 seconds
304
+ - **Frequency**: 1-2 seconds
305
+ - **Models**: 30-120 seconds (large files)
306
+
307
+ ## Extending the Cache System
308
+
309
+ ### Adding a New Cache Type
310
+
311
+ 1. **Create subclass of BaseCache**:
312
+
313
+ ```ruby
314
+ class MyCache < Kotoshu::Cache::BaseCache
315
+ def initialize(cache_path: nil, url_base: nil, cache_ttl: nil)
316
+ super
317
+ end
318
+
319
+ protected
320
+
321
+ def download_resource(resource_id, dest_path)
322
+ # Download logic
323
+ end
324
+
325
+ def load_cached(resource_id)
326
+ # Load logic
327
+ end
328
+
329
+ def metadata_path_for(resource_id)
330
+ # Return path to metadata.json
331
+ end
332
+
333
+ def resource_dir_for(resource_id)
334
+ # Return directory path
335
+ end
336
+
337
+ def resource_files_exist?(resource_id)
338
+ # Check if files exist
339
+ end
340
+
341
+ def supports_resource?(resource_id)
342
+ # Check if resource is supported
343
+ end
344
+
345
+ def cached_resources
346
+ # Return list of cached resource IDs
347
+ end
348
+ end
349
+ ```
350
+
351
+ 2. **Implement abstract methods**
352
+ 3. **Set appropriate TTL** (override `default_cache_ttl`)
353
+ 4. **Set cache path** (override `default_cache_path`)
354
+
355
+ ### Downloading from External Sources
356
+
357
+ For resources hosted outside GitHub (like FastText models), override the
358
+ download logic and URL generation:
359
+
360
+ ```ruby
361
+ def model_url(language, type, filename)
362
+ case type
363
+ when "fasttext"
364
+ # Download from FastText CDN (Facebook Research)
365
+ # https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/
366
+ "https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/#{filename}"
367
+ else
368
+ "#{@url_base}/default/path/#{filename}"
369
+ end
370
+ end
371
+
372
+ def download_resource(resource_id, dest_path)
373
+ # ... existing code ...
374
+
375
+ filename = model_info[:file]
376
+
377
+ # Handle gzip compression
378
+ if filename.end_with?('.gz')
379
+ download_and_decompress(url, dest_path)
380
+ else
381
+ download_file(url, dest_path)
382
+ end
383
+ end
384
+
385
+ def download_and_decompress(url, dest_path)
386
+ # Download to temporary .gz file
387
+ temp_gz = "#{dest_path}.gz"
388
+ URI.open(url, open_timeout: 30, read_timeout: 300) do |uri|
389
+ File.write(temp_gz, uri.read, mode: 'wb')
390
+ end
391
+
392
+ # Decompress gzip
393
+ Zlib::GzipReader.open(temp_gz) do |gz|
394
+ File.write(dest_path, gz.read)
395
+ end
396
+
397
+ File.delete(temp_gz)
398
+ end
399
+ ```
400
+
401
+ ### FastText Model Sources
402
+
403
+ FastText models are downloaded from Facebook's public CDN:
404
+
405
+ - **Base URL**: `https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/`
406
+ - **Files**: `cc.{lang}.300.vec.gz` (gzipped FastText vectors)
407
+ - **Decompressed**: `cc.{lang}.300.vec` (text format, 300 dimensions)
408
+ - **Sizes**: ~1.2-2.7 GB per language (decompressed)
409
+
410
+ .File sizes by language
411
+ [width="100%",cols="^1,^1,^1",options="header"]
412
+ |=========================================
413
+ | Language | Compressed | Decompressed | Word Count |
414
+ |=========================================
415
+ | de (German) | 1.2 GB | ~2.5 GB | 1,000,000 |
416
+ | en (English) | 2.7 GB | ~5.0 GB | 2,000,000 |
417
+ | es (Spanish) | 1.2 GB | ~2.5 GB | 1,000,000 |
418
+ | fr (French) | 1.2 GB | ~2.5 GB | 1,000,000 |
419
+ | pt (Portuguese) | 1.2 GB | ~2.5 GB | 1,000,000 |
420
+ | ru (Russian) | 1.2 GB | ~2.5 GB | 1,000,000 |
421
+ |=========================================]
422
+
423
+ ### Adding CLI Commands
424
+
425
+ To add CLI commands for a new cache type, edit `lib/kotoshu/cli/cache_command.rb`:
426
+
427
+ ```ruby
428
+ desc 'download TYPE RESOURCE', 'Download a resource from GitHub'
429
+ def download(type, resource)
430
+ cache = cache_for_type(type)
431
+
432
+ case type
433
+ when 'mycache', 'mc'
434
+ download_mycache(cache, resource)
435
+ end
436
+ end
437
+
438
+ private
439
+
440
+ def download_mycache(cache, resource)
441
+ result = cache.get(resource, force_download: options[:force])
442
+ puts "Downloaded: #{result[:path]}"
443
+ end
444
+
445
+ def cache_for_type(type)
446
+ case type
447
+ when 'mycache', 'mc'
448
+ MyCache.new
449
+ # ... existing types
450
+ end
451
+ end
452
+ ```
453
+
454
+ ## Testing
455
+
456
+ ### Unit Tests
457
+ Each cache type has comprehensive RSpec tests:
458
+ - `spec/kotoshu/cache/language_cache_spec.rb`
459
+ - `spec/kotoshu/cache/model_cache_spec.rb`
460
+ - `spec/kotoshu/cache/frequency_cache_spec.rb`
461
+
462
+ ### Integration Tests
463
+ Run combined cache test:
464
+ ```bash
465
+ ruby test_combined_cache.rb
466
+ ```
467
+
468
+ ## Troubleshooting
469
+
470
+ ### Cache Not Downloading
471
+ Check:
472
+ 1. Internet connection
473
+ 2. GitHub repository accessibility
474
+ 3. Available disk space
475
+ 4. File permissions on `~/.kotoshu/`
476
+
477
+ ### Stale Cache Data
478
+ Force re-download:
479
+ ```ruby
480
+ cache.get('en:spelling', force_download: true)
481
+ ```
482
+
483
+ Or use CLI:
484
+ ```bash
485
+ kotoshu cache download language en
486
+ kotoshu cache purge language en
487
+ kotoshu cache download language en
488
+ ```
489
+
490
+ ### High Cache Miss Rate
491
+ - Check TTL settings
492
+ - Verify cache directory exists
493
+ - Check metadata files are valid JSON
494
+
495
+ ## Future Enhancements
496
+
497
+ 1. **File Locking**: Support multi-threaded access
498
+ 2. **Compression**: Compress cached models to save disk space
499
+ 3. **Partial Downloads**: Resume interrupted downloads
500
+ 4. **Cache Warming**: Pre-download frequently used resources
501
+ 5. **Distributed Caching**: Support Redis/memcached backends
502
+ 6. **Incremental Updates**: Update only changed resources
@@ -0,0 +1,37 @@
1
+ ## Summary
2
+
3
+ Plan 91: two correctness fixes for non-ASCII users, found by the site-flip agent (plan 90 residues).
4
+
5
+ ### Track A — Unicode word detection
6
+
7
+ `Spellchecker#word_char?` accepted only ASCII letters, so Greek and Ukrainian users could not check any text: every non-Latin word was silently invisible and `kotoshu check -l el` reported `0 words`.
8
+
9
+ Word extraction now follows the language being checked:
10
+
11
+ - `Tokenizer::Base#spellcheck_word_regex` — single-character regex for spell-check extraction. Base default is the historical ASCII set (letters + apostrophe), so tokenizers without an override keep their exact behavior.
12
+ - Overrides: Greek `[\p{Greek}']` (apostrophe stays in-word for elisions like `απ’`), Cyrillic `[\p{Cyrillic}']` (`Мар'яна` stays one word), Latin `[\p{Latin}']` (covers capitals like `Å`/`Ä` that the old `à-ÿ` range missed, plus Turkish `ı ğ ş İ`).
13
+ - Digits are never word characters, so number-adjacent splitting is unchanged.
14
+ - `Spellchecker` resolves the tokenizer from the resource bundle language first (the facade passes the global `Configuration` next to the resolved bundle, so the config default is the wrong source) and falls back to the `Language::Registry` language, then to ASCII. Wrong-script words stay invisible: Greek text under `-l en` extracts nothing.
15
+ - Languages without a script tokenizer (en uses the components whitespace tokenizer, de has no override yet) keep the ASCII behavior — guarded by specs.
16
+
17
+ ### Track B — Swedish aff `RegexpError`
18
+
19
+ `Readers::AffReader` raised `RegexpError: unmatched close parenthesis` on the Swedish dictionary (dictionaries pin 1829a3e). Root cause: `COMPOUNDRULE )k` uses `)` itself as a flag character ("for adjectives: )k", "for verbs: >j" per the aff comments), but `CompoundRule` only escaped `)` when preceded by a non-operator character. A leading or post-operator `)` survived raw into `Regexp.new` and raised. The fix escapes every flag atom (`Regexp.escape`) while keeping `*`/`?` operators and `(...)` groups intact. The full sv dictionary (152,719 words) now loads and checks.
20
+
21
+ ## Verification
22
+
23
+ - Full suite: `3696 examples, 0 failures, 27 pending` (13 new examples)
24
+ - Frozen conformance replay: `rake kotoshu:conformance:ruby` — 2630 vectors, 0 failures, byte-identical
25
+ - rubocop clean on all changed files
26
+ - End-to-end through the real CLI (setup + check, isolated cache, dictionaries from the pin):
27
+
28
+ | lang | correct sentence | bogus word |
29
+ |---|---|---|
30
+ | el | OK, 5 words, no errors | `θθξκζψθ` flagged |
31
+ | tr | OK, 5 words, no errors | `ğşxqzz` flagged |
32
+ | sv | OK, 7 words, no errors | `ÅÅxxqqz` flagged |
33
+ | it | OK, 5 words, no errors | `zzqqxvv` flagged |
34
+
35
+ Note: remote `kotoshu setup it/el/tr/sv` currently 404s because upstream `dictionaries` only ships `en/` under the `spelling/` URL layout — verified instead with `setup LANG --from` against dictionaries fetched from pin 1829a3e. That URL-layout gap is upstream content, not this PR.
36
+
37
+ RBS: added `spellcheck_word_regex` to the declared Greek/Cyrillic tokenizers. No version changes.