kotoshu 1.0.2-x86_64-linux

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (458) hide show
  1. checksums.yaml +7 -0
  2. data/.kotoshu-baseline.json +1 -0
  3. data/.pre-commit-hooks.yaml +18 -0
  4. data/.rspec +3 -0
  5. data/.rubocop.yml +351 -0
  6. data/.yardopts +8 -0
  7. data/CHANGELOG.md +813 -0
  8. data/CLAUDE.md +176 -0
  9. data/CODE_OF_CONDUCT.md +132 -0
  10. data/Cargo.lock +359 -0
  11. data/Cargo.toml +15 -0
  12. data/LICENSE +31 -0
  13. data/README.adoc +1292 -0
  14. data/Rakefile +150 -0
  15. data/SECURITY.md +93 -0
  16. data/TODO.impl/00-cut-0.1.md +121 -0
  17. data/TODO.impl/00-cut-0.2.md +175 -0
  18. data/TODO.impl/00-vision.md +281 -0
  19. data/TODO.impl/01-hunspell-correctness.md +111 -0
  20. data/TODO.impl/02-cli-unification.md +74 -0
  21. data/TODO.impl/03-dynamic-download.md +107 -0
  22. data/TODO.impl/04-language-modules.md +89 -0
  23. data/TODO.impl/05-semantic-path.md +83 -0
  24. data/TODO.impl/06-cjk-support.md +105 -0
  25. data/TODO.impl/07-rtl-support.md +71 -0
  26. data/TODO.impl/08-grammar-engine.md +106 -0
  27. data/TODO.impl/09-integrity-security.md +90 -0
  28. data/TODO.impl/10-testing-ci.md +98 -0
  29. data/TODO.impl/100-full-feature-batch3.md +28 -0
  30. data/TODO.impl/101-int4-tier-experiment.md +32 -0
  31. data/TODO.impl/102-lid-chain.md +28 -0
  32. data/TODO.impl/103-oov-bucket-tables.md +27 -0
  33. data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
  34. data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
  35. data/TODO.impl/106-native-lid-detection.md +36 -0
  36. data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
  37. data/TODO.impl/108-ko-ne-full-feature.md +37 -0
  38. data/TODO.impl/109-ten-audit.md +26 -0
  39. data/TODO.impl/11-release-v1.md +95 -0
  40. data/TODO.impl/110-sr-latn-and-polish.md +33 -0
  41. data/TODO.impl/111-embedding-bakeoff.md +33 -0
  42. data/TODO.impl/112-deployability-and-gating.md +26 -0
  43. data/TODO.impl/113-language-packs.md +26 -0
  44. data/TODO.impl/114-typo-biencoder-thread.md +33 -0
  45. data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
  46. data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
  47. data/TODO.impl/117-deterministic-frequency-tiers.md +110 -0
  48. data/TODO.impl/118-server-gem-floor.md +33 -0
  49. data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
  50. data/TODO.impl/120-setup-metadata-refresh.md +35 -0
  51. data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
  52. data/TODO.impl/122-pack-compression-kpk2.md +66 -0
  53. data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
  54. data/TODO.impl/124-owner-residue.md +19 -0
  55. data/TODO.impl/125-consumer-floor-guards.md +43 -0
  56. data/TODO.impl/126-tier-error-budgets.md +33 -0
  57. data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
  58. data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
  59. data/TODO.impl/129-pt-fr-corpora.md +23 -0
  60. data/TODO.impl/130-news-truth-0912.md +23 -0
  61. data/TODO.impl/131-hybrid-productization.md +42 -0
  62. data/TODO.impl/132-techniques-page-and-prose.md +27 -0
  63. data/TODO.impl/133-rust-first-distribution.md +68 -0
  64. data/TODO.impl/20-encapsulation-cleanup.md +47 -0
  65. data/TODO.impl/21-autoload-migration.md +64 -0
  66. data/TODO.impl/22-url-pin-single-source-of-truth.md +120 -0
  67. data/TODO.impl/23-fix-url-defaults.md +57 -0
  68. data/TODO.impl/24-cli-auto-setup-prompt.md +76 -0
  69. data/TODO.impl/25-soft-onnxruntime-require.md +75 -0
  70. data/TODO.impl/26-onnx-vocab-json-generation.md +96 -0
  71. data/TODO.impl/27-end-to-end-smoke-spec.md +113 -0
  72. data/TODO.impl/28-readme-quickstart-verification.md +68 -0
  73. data/TODO.impl/29-status-command.md +84 -0
  74. data/TODO.impl/30-language-auto-detection.md +74 -0
  75. data/TODO.impl/31-download-progress.md +58 -0
  76. data/TODO.impl/32-sarif-output-verification.md +72 -0
  77. data/TODO.impl/33-audit-log-rotation.md +54 -0
  78. data/TODO.impl/34-cache-eviction.md +59 -0
  79. data/TODO.impl/35-shell-completion.md +61 -0
  80. data/TODO.impl/36-cleanup-regressions-0.3.1.md +134 -0
  81. data/TODO.impl/37-hunspell-correctness-tier2.md +203 -0
  82. data/TODO.impl/38-onnx-semantic-gating.md +218 -0
  83. data/TODO.impl/39-tier3-and-beyond.md +232 -0
  84. data/TODO.impl/40-spec-drift-cleanup.md +134 -0
  85. data/TODO.impl/50-structure-aware-document-api.md +194 -0
  86. data/TODO.impl/51-grammar-engine-expansion.md +75 -0
  87. data/TODO.impl/52-multi-language-documents.md +95 -0
  88. data/TODO.impl/53-document-plugin-architecture.md +94 -0
  89. data/TODO.impl/54-cjk-support.md +117 -0
  90. data/TODO.impl/55-rtl-support.md +109 -0
  91. data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
  92. data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
  93. data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
  94. data/TODO.impl/59-yard-api-documentation.md +75 -0
  95. data/TODO.impl/60-lsp-server.md +174 -0
  96. data/TODO.impl/61-editor-ecosystem.md +167 -0
  97. data/TODO.impl/62-framework-integrations.md +159 -0
  98. data/TODO.impl/63-web-playground-wasm.md +186 -0
  99. data/TODO.impl/64-http-api-and-sdks.md +191 -0
  100. data/TODO.impl/65-universal-kotoshu.md +251 -0
  101. data/TODO.impl/66-kotoshu-core.md +169 -0
  102. data/TODO.impl/67-kotoshu-rs-and-access-libraries.md +137 -0
  103. data/TODO.impl/68-sota-adoption.md +152 -0
  104. data/TODO.impl/69-sota-typo-eval.md +74 -0
  105. data/TODO.impl/70-sota-confidence-cascade.md +59 -0
  106. data/TODO.impl/71-sota-nested-model.md +63 -0
  107. data/TODO.impl/72-model2vec-evaluation.md +49 -0
  108. data/TODO.impl/73-rl-reranker.md +42 -0
  109. data/TODO.impl/74-distilled-reranker.md +36 -0
  110. data/TODO.impl/75-llm-evaluator.md +35 -0
  111. data/TODO.impl/76-optional-llm-backend.md +37 -0
  112. data/TODO.impl/77-model-coverage-expansion.md +96 -0
  113. data/TODO.impl/78-in-browser-playground.md +69 -0
  114. data/TODO.impl/79-news-and-releases.md +60 -0
  115. data/TODO.impl/80-ecosystem-docs.md +56 -0
  116. data/TODO.impl/81-editor-reach.md +68 -0
  117. data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
  118. data/TODO.impl/83-models-batch-2.md +56 -0
  119. data/TODO.impl/84-language-modules-wave2.md +75 -0
  120. data/TODO.impl/85-wasm-model-api.md +48 -0
  121. data/TODO.impl/86-site-truth-wave2.md +45 -0
  122. data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
  123. data/TODO.impl/88-cli-directory-mode.md +54 -0
  124. data/TODO.impl/89-framework-integrations-exec.md +59 -0
  125. data/TODO.impl/90-gem-correctness-residues.md +44 -0
  126. data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
  127. data/TODO.impl/92-tier-mirror-cors.md +88 -0
  128. data/TODO.impl/93-python-wheels.md +32 -0
  129. data/TODO.impl/94-site-integrations-docs.md +35 -0
  130. data/TODO.impl/95-editor-action-polish.md +27 -0
  131. data/TODO.impl/96-server-semantic-models.md +50 -0
  132. data/TODO.impl/97-yard-execution.md +73 -0
  133. data/TODO.impl/98-campaign-cleanup.md +78 -0
  134. data/TODO.impl/99-site-truth-performance-docs.md +31 -0
  135. data/conformance/vectors.jsonl +2630 -0
  136. data/data/en/grammar/rules.yaml +851 -0
  137. data/docs/STABILITY-1.0-AUDIT.md +251 -0
  138. data/docs/STABILITY-1.0.md +137 -0
  139. data/docs/cache-architecture.md +502 -0
  140. data/docs/verification-evidence/correct3/pr_body.md +37 -0
  141. data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
  142. data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
  143. data/docs/verification-evidence/correct3/sv.aff +856 -0
  144. data/docs/verification-evidence/correct3/sv.dic +153717 -0
  145. data/examples/01_basic_word_checking.rb +38 -0
  146. data/examples/02_text_document_checking.rb +77 -0
  147. data/examples/03_dictionary_backends.rb +137 -0
  148. data/examples/04_trie_data_structure.rb +146 -0
  149. data/examples/05_suggestion_algorithms.rb +239 -0
  150. data/examples/06_configuration_advanced.rb +287 -0
  151. data/examples/07_multi_language_dictionaries.rb +278 -0
  152. data/exe/kotoshu +6 -0
  153. data/lib/kotoshu/algorithms/capitalization.rb +276 -0
  154. data/lib/kotoshu/algorithms/edit_distance.rb +128 -0
  155. data/lib/kotoshu/algorithms/lookup/affix_form.rb +132 -0
  156. data/lib/kotoshu/algorithms/lookup/compound_checks.rb +209 -0
  157. data/lib/kotoshu/algorithms/lookup/compound_form.rb +50 -0
  158. data/lib/kotoshu/algorithms/lookup.rb +841 -0
  159. data/lib/kotoshu/algorithms/ngram_suggest.rb +367 -0
  160. data/lib/kotoshu/algorithms/permutations.rb +283 -0
  161. data/lib/kotoshu/algorithms/phonet_suggest.rb +162 -0
  162. data/lib/kotoshu/algorithms/soundex.rb +92 -0
  163. data/lib/kotoshu/algorithms/suggest.rb +794 -0
  164. data/lib/kotoshu/algorithms.rb +23 -0
  165. data/lib/kotoshu/analyzers/semantic_analyzer.rb +405 -0
  166. data/lib/kotoshu/analyzers.rb +8 -0
  167. data/lib/kotoshu/baseline/application.rb +18 -0
  168. data/lib/kotoshu/baseline/entry.rb +33 -0
  169. data/lib/kotoshu/baseline/store.rb +175 -0
  170. data/lib/kotoshu/baseline.rb +20 -0
  171. data/lib/kotoshu/cache/base_cache.rb +778 -0
  172. data/lib/kotoshu/cache/cache.rb +91 -0
  173. data/lib/kotoshu/cache/eviction_policy.rb +66 -0
  174. data/lib/kotoshu/cache/frequency_cache.rb +283 -0
  175. data/lib/kotoshu/cache/language_cache.rb +511 -0
  176. data/lib/kotoshu/cache/lookup_cache.rb +166 -0
  177. data/lib/kotoshu/cache/model_cache.rb +1147 -0
  178. data/lib/kotoshu/cache/model_registry.rb +109 -0
  179. data/lib/kotoshu/cache/staged_languages.rb +45 -0
  180. data/lib/kotoshu/cache/suggestion_cache.rb +113 -0
  181. data/lib/kotoshu/cache.rb +49 -0
  182. data/lib/kotoshu/cli/auto_setup.rb +71 -0
  183. data/lib/kotoshu/cli/baseline_command.rb +119 -0
  184. data/lib/kotoshu/cli/batch_reporter.rb +321 -0
  185. data/lib/kotoshu/cli/cache_command.rb +282 -0
  186. data/lib/kotoshu/cli/completions_command.rb +210 -0
  187. data/lib/kotoshu/cli/dict_command.rb +64 -0
  188. data/lib/kotoshu/cli/directory_check.rb +146 -0
  189. data/lib/kotoshu/cli/directory_walker.rb +132 -0
  190. data/lib/kotoshu/cli/display_formatter.rb +429 -0
  191. data/lib/kotoshu/cli/errors.rb +36 -0
  192. data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
  193. data/lib/kotoshu/cli/interactive_reviewer.rb +314 -0
  194. data/lib/kotoshu/cli/language_resolver.rb +91 -0
  195. data/lib/kotoshu/cli/navigation_manager.rb +273 -0
  196. data/lib/kotoshu/cli/personal_command.rb +113 -0
  197. data/lib/kotoshu/cli/progress_reporter.rb +114 -0
  198. data/lib/kotoshu/cli/status_report.rb +130 -0
  199. data/lib/kotoshu/cli.rb +880 -0
  200. data/lib/kotoshu/components/passthrough_spell_checker.rb +70 -0
  201. data/lib/kotoshu/components/pos_tagger.rb +98 -0
  202. data/lib/kotoshu/components/spell_checker.rb +73 -0
  203. data/lib/kotoshu/components/synthesizer.rb +60 -0
  204. data/lib/kotoshu/components/tokenizer.rb +58 -0
  205. data/lib/kotoshu/components/whitespace_tokenizer.rb +94 -0
  206. data/lib/kotoshu/components.rb +13 -0
  207. data/lib/kotoshu/configuration/builder.rb +207 -0
  208. data/lib/kotoshu/configuration/resolver.rb +124 -0
  209. data/lib/kotoshu/configuration.rb +829 -0
  210. data/lib/kotoshu/conformance_exporter.rb +256 -0
  211. data/lib/kotoshu/conformance_runner.rb +176 -0
  212. data/lib/kotoshu/core/exceptions.rb +186 -0
  213. data/lib/kotoshu/core/indexed_dictionary.rb +293 -0
  214. data/lib/kotoshu/core/models/affix_rule.rb +262 -0
  215. data/lib/kotoshu/core/models/result/document_result.rb +132 -0
  216. data/lib/kotoshu/core/models/result/word_result.rb +131 -0
  217. data/lib/kotoshu/core/models/word.rb +131 -0
  218. data/lib/kotoshu/core/trie/builder.rb +119 -0
  219. data/lib/kotoshu/core/trie/node.rb +94 -0
  220. data/lib/kotoshu/core/trie/trie.rb +249 -0
  221. data/lib/kotoshu/core.rb +28 -0
  222. data/lib/kotoshu/data/common_words/de.yml +1800 -0
  223. data/lib/kotoshu/data/common_words/en.yml +1215 -0
  224. data/lib/kotoshu/data/common_words/es.yml +750 -0
  225. data/lib/kotoshu/data/common_words/fr.yml +1015 -0
  226. data/lib/kotoshu/data/common_words/pt.yml +870 -0
  227. data/lib/kotoshu/data/common_words/ru.yml +484 -0
  228. data/lib/kotoshu/data/common_words_loader.rb +141 -0
  229. data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
  230. data/lib/kotoshu/data/frozen_kelly.rb +38 -0
  231. data/lib/kotoshu/data.rb +8 -0
  232. data/lib/kotoshu/data_structures/bloom_filter.rb +188 -0
  233. data/lib/kotoshu/data_structures.rb +8 -0
  234. data/lib/kotoshu/debug_logger.rb +146 -0
  235. data/lib/kotoshu/debug_mode.rb +136 -0
  236. data/lib/kotoshu/defaults.rb +84 -0
  237. data/lib/kotoshu/dictionaries/catalog.rb +814 -0
  238. data/lib/kotoshu/dictionaries.rb +8 -0
  239. data/lib/kotoshu/dictionary/base.rb +324 -0
  240. data/lib/kotoshu/dictionary/cspell.rb +243 -0
  241. data/lib/kotoshu/dictionary/custom.rb +225 -0
  242. data/lib/kotoshu/dictionary/hunspell.rb +520 -0
  243. data/lib/kotoshu/dictionary/plain_text.rb +302 -0
  244. data/lib/kotoshu/dictionary/repository.rb +246 -0
  245. data/lib/kotoshu/dictionary/unified.rb +236 -0
  246. data/lib/kotoshu/dictionary/unix_words.rb +218 -0
  247. data/lib/kotoshu/dictionary.rb +15 -0
  248. data/lib/kotoshu/documents/document.rb +120 -0
  249. data/lib/kotoshu/documents/plain_text_document.rb +73 -0
  250. data/lib/kotoshu/documents/source_position.rb +45 -0
  251. data/lib/kotoshu/documents/source_range.rb +76 -0
  252. data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
  253. data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
  254. data/lib/kotoshu/documents/suppressions.rb +44 -0
  255. data/lib/kotoshu/documents/text_node.rb +49 -0
  256. data/lib/kotoshu/documents.rb +178 -0
  257. data/lib/kotoshu/embeddings/embedding_pipeline.rb +238 -0
  258. data/lib/kotoshu/embeddings/lru_cache.rb +244 -0
  259. data/lib/kotoshu/embeddings/onnx_runtime_model.rb +411 -0
  260. data/lib/kotoshu/embeddings/protocol.rb +96 -0
  261. data/lib/kotoshu/embeddings/protocols.rb +23 -0
  262. data/lib/kotoshu/embeddings/registry.rb +184 -0
  263. data/lib/kotoshu/embeddings/search.rb +204 -0
  264. data/lib/kotoshu/embeddings/similarity_engine.rb +278 -0
  265. data/lib/kotoshu/embeddings/similarity_search.rb +328 -0
  266. data/lib/kotoshu/embeddings/vocabulary.rb +306 -0
  267. data/lib/kotoshu/embeddings.rb +87 -0
  268. data/lib/kotoshu/fluent_checker.rb +94 -0
  269. data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +48 -0
  270. data/lib/kotoshu/grammar/pattern_matchers/double_negative_matcher.rb +179 -0
  271. data/lib/kotoshu/grammar/pattern_matchers/phrase_matcher.rb +86 -0
  272. data/lib/kotoshu/grammar/pattern_matchers/possessive_context_matcher.rb +75 -0
  273. data/lib/kotoshu/grammar/pattern_matchers/possessive_contraction_matcher.rb +87 -0
  274. data/lib/kotoshu/grammar/pattern_matchers/sentence_start_matcher.rb +84 -0
  275. data/lib/kotoshu/grammar/pattern_matchers/vowel_sound_matcher.rb +81 -0
  276. data/lib/kotoshu/grammar/pattern_matchers/word_list_matcher.rb +66 -0
  277. data/lib/kotoshu/grammar/rule.rb +99 -0
  278. data/lib/kotoshu/grammar/rule_engine.rb +96 -0
  279. data/lib/kotoshu/grammar/rule_loader.rb +30 -0
  280. data/lib/kotoshu/grammar.rb +24 -0
  281. data/lib/kotoshu/integrity/audit_log.rb +167 -0
  282. data/lib/kotoshu/integrity/manifest.rb +116 -0
  283. data/lib/kotoshu/integrity/net_http.rb +51 -0
  284. data/lib/kotoshu/integrity/rotation_policy.rb +88 -0
  285. data/lib/kotoshu/integrity.rb +26 -0
  286. data/lib/kotoshu/jekyll.rb +101 -0
  287. data/lib/kotoshu/keyboard/layout.rb +116 -0
  288. data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
  289. data/lib/kotoshu/keyboard/layouts/azerty.rb +55 -0
  290. data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
  291. data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
  292. data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
  293. data/lib/kotoshu/keyboard/layouts/dvorak.rb +54 -0
  294. data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
  295. data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
  296. data/lib/kotoshu/keyboard/layouts/hrsl.rb +94 -0
  297. data/lib/kotoshu/keyboard/layouts/jcuken.rb +57 -0
  298. data/lib/kotoshu/keyboard/layouts/latin.rb +266 -0
  299. data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
  300. data/lib/kotoshu/keyboard/layouts/qwerty.rb +52 -0
  301. data/lib/kotoshu/keyboard/layouts/qwertz.rb +55 -0
  302. data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
  303. data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
  304. data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
  305. data/lib/kotoshu/keyboard/registry.rb +201 -0
  306. data/lib/kotoshu/keyboard.rb +96 -0
  307. data/lib/kotoshu/kotoshu_native.so +0 -0
  308. data/lib/kotoshu/language/detector.rb +242 -0
  309. data/lib/kotoshu/language/identifier.rb +376 -0
  310. data/lib/kotoshu/language/languages/base.rb +271 -0
  311. data/lib/kotoshu/language/lid_detector.rb +188 -0
  312. data/lib/kotoshu/language/normalizer/arabic.rb +113 -0
  313. data/lib/kotoshu/language/normalizer/base.rb +133 -0
  314. data/lib/kotoshu/language/normalizer/greek.rb +63 -0
  315. data/lib/kotoshu/language/normalizer/hebrew.rb +72 -0
  316. data/lib/kotoshu/language/normalizer/persian.rb +36 -0
  317. data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
  318. data/lib/kotoshu/language/registry.rb +209 -0
  319. data/lib/kotoshu/language/resources/ar/common_words.txt +6753 -0
  320. data/lib/kotoshu/language/resources/ar/confusion_sets.txt +11 -0
  321. data/lib/kotoshu/language/resources/de/common_words.txt +10003 -0
  322. data/lib/kotoshu/language/resources/de/confusion_sets.txt +246 -0
  323. data/lib/kotoshu/language/resources/en/common_words.txt +9979 -0
  324. data/lib/kotoshu/language/resources/en/confusion_sets.txt +871 -0
  325. data/lib/kotoshu/language/resources/es/common_words.txt +9992 -0
  326. data/lib/kotoshu/language/resources/es/confusion_sets.txt +17 -0
  327. data/lib/kotoshu/language/resources/fr/common_words.txt +9993 -0
  328. data/lib/kotoshu/language/resources/fr/confusion_sets.txt +76 -0
  329. data/lib/kotoshu/language/resources/pt/common_words.txt +9977 -0
  330. data/lib/kotoshu/language/resources/pt/confusion_sets.txt +18 -0
  331. data/lib/kotoshu/language/resources/ru/common_words.txt +9951 -0
  332. data/lib/kotoshu/language/resources/ru/confusion_sets.txt +5 -0
  333. data/lib/kotoshu/language/script.rb +128 -0
  334. data/lib/kotoshu/language/segmenter.rb +122 -0
  335. data/lib/kotoshu/language/suika.rb +38 -0
  336. data/lib/kotoshu/language/tokenizer/base.rb +187 -0
  337. data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
  338. data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
  339. data/lib/kotoshu/language/tokenizer/french_tokenizer.rb +172 -0
  340. data/lib/kotoshu/language/tokenizer/german_tokenizer.rb +41 -0
  341. data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
  342. data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
  343. data/lib/kotoshu/language/tokenizer/japanese_tokenizer.rb +55 -0
  344. data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +153 -0
  345. data/lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb +160 -0
  346. data/lib/kotoshu/language/tokenizer/russian_tokenizer.rb +93 -0
  347. data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
  348. data/lib/kotoshu/language/tokenizer/spanish_tokenizer.rb +122 -0
  349. data/lib/kotoshu/language.rb +127 -0
  350. data/lib/kotoshu/languages/ar/language.rb +98 -0
  351. data/lib/kotoshu/languages/bg/language.rb +37 -0
  352. data/lib/kotoshu/languages/ca/language.rb +22 -0
  353. data/lib/kotoshu/languages/cs/language.rb +22 -0
  354. data/lib/kotoshu/languages/da/language.rb +22 -0
  355. data/lib/kotoshu/languages/de/language.rb +554 -0
  356. data/lib/kotoshu/languages/el/language.rb +50 -0
  357. data/lib/kotoshu/languages/en/language.rb +470 -0
  358. data/lib/kotoshu/languages/es/language.rb +476 -0
  359. data/lib/kotoshu/languages/et/language.rb +24 -0
  360. data/lib/kotoshu/languages/fa/language.rb +91 -0
  361. data/lib/kotoshu/languages/fr/language.rb +505 -0
  362. data/lib/kotoshu/languages/he/language.rb +98 -0
  363. data/lib/kotoshu/languages/hr/language.rb +24 -0
  364. data/lib/kotoshu/languages/hu/language.rb +22 -0
  365. data/lib/kotoshu/languages/id/language.rb +25 -0
  366. data/lib/kotoshu/languages/it/language.rb +22 -0
  367. data/lib/kotoshu/languages/ja/language.rb +476 -0
  368. data/lib/kotoshu/languages/ko/language.rb +56 -0
  369. data/lib/kotoshu/languages/latin_base.rb +53 -0
  370. data/lib/kotoshu/languages/lt/language.rb +24 -0
  371. data/lib/kotoshu/languages/lv/language.rb +24 -0
  372. data/lib/kotoshu/languages/nb/language.rb +25 -0
  373. data/lib/kotoshu/languages/ne/language.rb +60 -0
  374. data/lib/kotoshu/languages/nl/language.rb +22 -0
  375. data/lib/kotoshu/languages/nn/language.rb +26 -0
  376. data/lib/kotoshu/languages/pl/language.rb +22 -0
  377. data/lib/kotoshu/languages/pt/language.rb +439 -0
  378. data/lib/kotoshu/languages/ro/language.rb +22 -0
  379. data/lib/kotoshu/languages/ru/language.rb +420 -0
  380. data/lib/kotoshu/languages/sk/language.rb +25 -0
  381. data/lib/kotoshu/languages/sl/language.rb +25 -0
  382. data/lib/kotoshu/languages/sr/language.rb +39 -0
  383. data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
  384. data/lib/kotoshu/languages/sv/language.rb +22 -0
  385. data/lib/kotoshu/languages/tr/language.rb +36 -0
  386. data/lib/kotoshu/languages/uk/language.rb +38 -0
  387. data/lib/kotoshu/languages/vi/language.rb +22 -0
  388. data/lib/kotoshu/languages.rb +64 -0
  389. data/lib/kotoshu/metrics_collector.rb +238 -0
  390. data/lib/kotoshu/metrics_module.rb +121 -0
  391. data/lib/kotoshu/models/context.rb +119 -0
  392. data/lib/kotoshu/models/embedding_model.rb +181 -0
  393. data/lib/kotoshu/models/fasttext_model.rb +212 -0
  394. data/lib/kotoshu/models/nearest_neighbor.rb +96 -0
  395. data/lib/kotoshu/models/onnx_model.rb +330 -0
  396. data/lib/kotoshu/models/semantic_error.rb +204 -0
  397. data/lib/kotoshu/models/suggestion.rb +106 -0
  398. data/lib/kotoshu/models/word_embedding.rb +97 -0
  399. data/lib/kotoshu/models.rb +27 -0
  400. data/lib/kotoshu/multi_language_checker.rb +127 -0
  401. data/lib/kotoshu/native.rb +83 -0
  402. data/lib/kotoshu/native_backend.rb +131 -0
  403. data/lib/kotoshu/paths.rb +79 -0
  404. data/lib/kotoshu/personal_dictionary.rb +105 -0
  405. data/lib/kotoshu/plugins/plugin.rb +61 -0
  406. data/lib/kotoshu/plugins/registry.rb +118 -0
  407. data/lib/kotoshu/plugins.rb +9 -0
  408. data/lib/kotoshu/project_config.rb +76 -0
  409. data/lib/kotoshu/readers/aff_data.rb +530 -0
  410. data/lib/kotoshu/readers/aff_reader.rb +393 -0
  411. data/lib/kotoshu/readers/condition_checker.rb +125 -0
  412. data/lib/kotoshu/readers/dic_reader.rb +208 -0
  413. data/lib/kotoshu/readers/file_reader.rb +377 -0
  414. data/lib/kotoshu/readers/lookup_builder.rb +584 -0
  415. data/lib/kotoshu/readers/readers.rb +17 -0
  416. data/lib/kotoshu/readers.rb +25 -0
  417. data/lib/kotoshu/resource_bundle.rb +55 -0
  418. data/lib/kotoshu/resource_manager.rb +600 -0
  419. data/lib/kotoshu/results/result.rb +165 -0
  420. data/lib/kotoshu/results.rb +8 -0
  421. data/lib/kotoshu/rspec.rb +163 -0
  422. data/lib/kotoshu/scripts/fasttext_to_onnx.py +275 -0
  423. data/lib/kotoshu/source_registry.rb +104 -0
  424. data/lib/kotoshu/spellchecker/parallel_checker.rb +90 -0
  425. data/lib/kotoshu/spellchecker.rb +406 -0
  426. data/lib/kotoshu/string_metrics.rb +153 -0
  427. data/lib/kotoshu/suggestions/context.rb +43 -0
  428. data/lib/kotoshu/suggestions/frequency_provider.rb +80 -0
  429. data/lib/kotoshu/suggestions/generator.rb +169 -0
  430. data/lib/kotoshu/suggestions/pipeline.rb +135 -0
  431. data/lib/kotoshu/suggestions/semantic_cascade.rb +73 -0
  432. data/lib/kotoshu/suggestions/strategies/base_strategy.rb +315 -0
  433. data/lib/kotoshu/suggestions/strategies/composite_strategy.rb +197 -0
  434. data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +631 -0
  435. data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +207 -0
  436. data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +165 -0
  437. data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +284 -0
  438. data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +317 -0
  439. data/lib/kotoshu/suggestions/strategies/symspell_strategy.rb +270 -0
  440. data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
  441. data/lib/kotoshu/suggestions/suggestion.rb +154 -0
  442. data/lib/kotoshu/suggestions/suggestion_set.rb +270 -0
  443. data/lib/kotoshu/suggestions/sweep_index.rb +137 -0
  444. data/lib/kotoshu/suggestions.rb +29 -0
  445. data/lib/kotoshu/tasks/check_task.rb +104 -0
  446. data/lib/kotoshu/tasks.rb +11 -0
  447. data/lib/kotoshu/validators/spelling_validator.rb +101 -0
  448. data/lib/kotoshu/validators.rb +9 -0
  449. data/lib/kotoshu/version.rb +5 -0
  450. data/lib/kotoshu.rb +538 -0
  451. data/rakelib/compile.rake +10 -0
  452. data/script/validate_all_dictionaries.rb +444 -0
  453. data/scripts/build_native_gem.rb +45 -0
  454. data/scripts/extract_eval_grids.rb +37 -0
  455. data/scripts/generate_frozen_tiers.rb +44 -0
  456. data/sig/kotoshu.rbs +435 -0
  457. data/test_oop.rb +79 -0
  458. metadata +546 -0
@@ -0,0 +1,113 @@
1
+ # 27 — End-to-end smoke spec (T1, 0.3-blocker)
2
+
3
+ ## Status
4
+ Implemented (shipped in the 0.3 line; re-verified 2026-09-03 against 0.6.0).
5
+
6
+ Evidence:
7
+
8
+ - `spec/integration/end_to_end_spec.rb` exists with the planned shape:
9
+ fresh install → setup → use (setup `:downloaded`, `correct?`,
10
+ misspelling rejection, `suggest` includes "the", document check),
11
+ setup predicate, and two-stage `ResourceNotSetupError` enforcement —
12
+ 12 examples, tagged `:network`, isolated via tmp XDG dirs in an
13
+ `around` hook.
14
+ - Run 2026-09-03: `NETWORK_TESTS=1 bundle exec rspec
15
+ spec/integration/end_to_end_spec.rb` → 12 examples, 0 failures
16
+ (real downloads from kotoshu/dictionaries).
17
+ - Real instances throughout; no `double()`. Excluded from the default
18
+ run via the `:network` tag as planned.
19
+
20
+ ## Problem
21
+ There is no spec that walks the actual install → setup → use path a real
22
+ user takes. Unit tests verify pieces; integration tests like
23
+ `walking_skeleton_spec.rb` use mocked or fixture data. Nothing catches
24
+ "this works end-to-end starting from an empty cache."
25
+
26
+ ## Plan
27
+
28
+ ### New spec: `spec/integration/end_to_end_spec.rb`
29
+
30
+ A single spec file with multiple examples covering the full path:
31
+
32
+ ```ruby
33
+ RSpec.describe "Kotoshu end-to-end", :network do
34
+ # Use temp XDG dir for full isolation
35
+ around do |ex|
36
+ Dir.mktmpdir do |dir|
37
+ ENV["XDG_CACHE_HOME"] = "#{dir}/cache"
38
+ ENV["XDG_CONFIG_HOME"] = "#{dir}/config"
39
+ ENV["XDG_DATA_HOME"] = "#{dir}/local"
40
+ Kotoshu::Configuration.reset
41
+ ex.run
42
+ end
43
+ end
44
+
45
+ describe "fresh install → setup → use" do
46
+ it "sets up English spelling" do
47
+ result = Kotoshu.setup(:en, want: %i[spelling])
48
+ expect(result.spelling).to eq(:downloaded)
49
+ end
50
+
51
+ it "accepts a correctly-spelled word" do
52
+ Kotoshu.setup(:en, want: %i[spelling])
53
+ expect(Kotoshu.correct?("hello")).to be(true)
54
+ end
55
+
56
+ it "rejects a misspelling" do
57
+ Kotoshu.setup(:en, want: %i[spelling])
58
+ expect(Kotoshu.correct?("xyzzq")).to be(false)
59
+ end
60
+
61
+ it "suggests corrections" do
62
+ Kotoshu.setup(:en, want: %i[spelling])
63
+ expect(Kotoshu.suggest("teh")).to include("the")
64
+ end
65
+
66
+ it "checks a document and flags misspellings" do
67
+ Kotoshu.setup(:en, want: %i[spelling])
68
+ result = Kotoshu.check("this is a teh test")
69
+ expect(result.misspelled_words.map(&:word)).to include("teh")
70
+ end
71
+ end
72
+
73
+ describe "setup predicate" do
74
+ it "returns false before setup" do
75
+ expect(Kotoshu.setup?(:en)).to be(false)
76
+ end
77
+
78
+ it "returns true after setup" do
79
+ Kotoshu.setup(:en, want: %i[spelling])
80
+ expect(Kotoshu.setup?(:en)).to be(true)
81
+ end
82
+ end
83
+
84
+ describe "two-stage model enforcement" do
85
+ it "raises ResourceNotSetupError on cold correct?" do
86
+ expect { Kotoshu.correct?("hello") }
87
+ .to raise_error(Kotoshu::ResourceNotSetupError)
88
+ end
89
+ end
90
+ end
91
+ ```
92
+
93
+ ### Use `:network` tag
94
+ Already configured in `spec/spec_helper.rb` to only run when
95
+ `NETWORK_TESTS=1`. Run locally before release:
96
+ ```
97
+ NETWORK_TESTS=1 bundle exec rspec spec/integration/end_to_end_spec.rb
98
+ ```
99
+
100
+ ### Real instances, no doubles
101
+ Per `~/.claude/CLAUDE.md`: no `double()` in specs. Use real cache, real
102
+ HTTP, real dictionaries.
103
+
104
+ ## Acceptance
105
+
106
+ - [ ] `spec/integration/end_to_end_spec.rb` exists, all examples pass
107
+ with `NETWORK_TESTS=1`.
108
+ - [ ] Spec isolates itself via temp XDG dirs (no pollution of dev cache).
109
+ - [ ] Spec uses real model instances, no `double()`.
110
+ - [ ] CI can run this on demand (network tag excludes from default run).
111
+
112
+ ## Dependencies
113
+ - #23 (URL fix must land first so setup actually works).
@@ -0,0 +1,68 @@
1
+ # 28 — Verify README quickstart (T1, 0.3-blocker)
2
+
3
+ ## Status
4
+ Implemented — quickstart re-verified 2026-09-03 against 0.6.0 (via
5
+ `bundle exec` in the repo, not a sandbox gem install).
6
+
7
+ Commands actually run, all matching README.adoc output:
8
+
9
+ - `bundle exec exe/kotoshu setup en` →
10
+ `Setup en... OK (spelling: cached, frequency: skipped, model: skipped, source: kotoshu)`.
11
+ - `printf 'helo wrld\n' | bundle exec exe/kotoshu check` →
12
+ `FAIL <stdin> (2 errors)` with `helo -> hello, help, hell` /
13
+ `wrld -> world, weld, wald`, exit 1.
14
+ - Ruby API: `Kotoshu.correct?("hello")` → `true`;
15
+ `Kotoshu.correct?("helo")` → `false`;
16
+ `Kotoshu.suggest("helo").to_words.first(5)` →
17
+ `["hello", "help", "hell", "helot", "halo"]`;
18
+ `Kotoshu.check("Hello wrold").errors.map(&:word)` → `["wrold"]`.
19
+ - `kotoshu status`, `--format json/sarif`, exit codes 0/1/2/3 — all
20
+ behave as documented (see #29, #32).
21
+
22
+ Known drift found during verification (worth a fix): Thor emits
23
+ `[WARNING] Attempted to create command ...` lines from
24
+ `lib/kotoshu/cli/cache_command.rb:191,201,213` to **stdout** at class
25
+ load, which pollutes `--format json` / `--format sarif` output.
26
+ Consumers must filter them until the Thor descs are added.
27
+
28
+ ## Problem
29
+ The README's quickstart examples may not actually work against the
30
+ shipped 0.3 code. The README has been rewritten multiple times; commands
31
+ and outputs may have drifted.
32
+
33
+ ## Plan
34
+
35
+ ### Step 1 — Build the gem locally
36
+ ```
37
+ gem build kotoshu.gemspec
38
+ ```
39
+
40
+ ### Step 2 — Install into a fresh sandbox
41
+ ```
42
+ rvm use 3.1@kotoshu-sandbox --create
43
+ gem install ./kotoshu-0.3.0.gem
44
+ ```
45
+
46
+ ### Step 3 — Walk every code block in README.adoc
47
+ For each example, paste into irb and verify output matches.
48
+
49
+ Specifically:
50
+ - `Kotoshu.correct?("hello")` → `true`
51
+ - `Kotoshu.correct?("xyzzq")` → `false`
52
+ - `Kotoshu.suggest("recieve")` → `["receive", ...]`
53
+ - `Kotoshu.check("this is a speeling test")` → document result
54
+ - CLI: `kotoshu setup en` → success
55
+ - CLI: `kotoshu suggest teh` → `["the", ...]`
56
+ - CLI: `kotoshu check README.md` → flags misspellings
57
+
58
+ ### Step 4 — Fix any drift
59
+ Either fix the code or update the README to match.
60
+
61
+ ## Acceptance
62
+
63
+ - [ ] Every code block in README.adoc runs without error against 0.3 gem.
64
+ - [ ] Outputs shown in README match observed outputs (within reason).
65
+ - [ ] `kotoshu --help` shows all subcommands documented in README.
66
+
67
+ ## Dependencies
68
+ - #23, #24, #25 (must be fixed before quickstart can work).
@@ -0,0 +1,84 @@
1
+ # 29 — `kotoshu status` command (T2, 0.3-blocker)
2
+
3
+ ## Status
4
+ Implemented (shipped in the 0.3 line; re-verified 2026-09-03 against 0.6.0).
5
+
6
+ Evidence:
7
+
8
+ - `bundle exec exe/kotoshu status` run 2026-09-03 — output matches the
9
+ planned layout: version, Setup table (per-language × resource with
10
+ size + cached date), Cache path/size/language count, Semantic
11
+ (`onnxruntime loaded`, active models), Other (audit log path+size,
12
+ default lang, offline mode). Exit 0.
13
+ - `kotoshu status --json` emits valid JSON (same StatusReport, second
14
+ presenter).
15
+ - Code: `desc "status"` in `lib/kotoshu/cli.rb:267-287`; pure model at
16
+ `lib/kotoshu/cli/status_report.rb` (no presentation logic).
17
+ - Spec: `spec/kotoshu/cli/status_report_spec.rb`.
18
+
19
+ Cosmetic caveat: the Thor `[WARNING]` stdout pollution noted in #28
20
+ also precedes status output.
21
+
22
+ ## Problem
23
+ Users have no single command to see:
24
+ - What languages are set up (spelling, frequency, model)
25
+ - Cache disk usage
26
+ - Audit log location and size
27
+ - Whether semantic is available (onnxruntime loaded)
28
+ - Default language
29
+
30
+ `kotoshu setup --list` exists but is narrow. A dedicated `status` command
31
+ gives users a single thing to run when something goes wrong.
32
+
33
+ ## Plan
34
+
35
+ ### New command: `Kotoshu::Commands::StatusCommand`
36
+
37
+ Subclass of `Thor::Group` (or registered via `register` in the CLI).
38
+ Output:
39
+
40
+ ```
41
+ Kotoshu 0.3.0
42
+
43
+ Setup:
44
+ en spelling ✓ (~12 MB, cached 2026-06-27)
45
+ en frequency ✓ (~250 KB, cached 2026-06-27)
46
+ en model ✓ (~114 MB, cached 2026-06-27)
47
+ de spelling ✓ (~8 MB, cached 2026-06-25)
48
+
49
+ Cache:
50
+ Path /Users/user/.cache/kotoshu
51
+ Size 142 MB
52
+ Languages 2
53
+
54
+ Semantic:
55
+ onnxruntime loaded
56
+ Active models 1 (en)
57
+
58
+ Other:
59
+ Audit log /Users/user/.local/share/kotoshu/audit.log (12 KB)
60
+ Default lang en
61
+ Offline mode no
62
+ ```
63
+
64
+ ### Implementation
65
+ - Reuse `ResourceManager.languages_setup`, `setup?`, etc.
66
+ - New helper `Kotoshu::Paths.cache_size` (sums `File.size` of all cached files).
67
+ - Honor `--json` flag for machine-readable output (single source of
68
+ truth: same StatusReport model, two presenters).
69
+
70
+ ### Status report model
71
+ `Kotoshu::Cli::StatusReport` — pure model. CLI formats to text; JSON
72
+ presenter emits structured output. MECE: model has no presentation.
73
+
74
+ ## Acceptance
75
+
76
+ - [ ] `kotoshu status` works on a fresh install (shows empty setup).
77
+ - [ ] `kotoshu status` after `kotoshu setup en` shows the entry.
78
+ - [ ] `kotoshu status --json` emits valid JSON.
79
+ - [ ] Cache size is accurate to within 1 MB.
80
+ - [ ] Audit log path matches `Kotoshu::Paths.audit_log_path`.
81
+
82
+ ## Dependencies
83
+ - #22 (SourceRegistry) — to show URLs.
84
+ - #25 (soft onnxruntime) — to report semantic availability.
@@ -0,0 +1,74 @@
1
+ # 30 — Default `--language auto` for `kotoshu check` (T2, 0.3-blocker)
2
+
3
+ ## Status
4
+ Implemented (shipped in the 0.3 line; re-verified 2026-09-03 against 0.6.0).
5
+
6
+ Evidence:
7
+
8
+ - `lib/kotoshu/cli/language_resolver.rb` implements the exact flag
9
+ matrix from this plan: omitted/`auto` → detect with fallback;
10
+ `default` → `Configuration.default_language`; any code → as-is.
11
+ A detection only "sticks" if the language is set up
12
+ (`Kotoshu.setup?`), otherwise the default is used and the result
13
+ carries the note `Detected: X (fallback: en)` — printed to stderr by
14
+ `cli.rb#resolve_language` (`warn "# #{result.note}"`).
15
+ - `--language` defaults to `auto` on the wired CLI check
16
+ (`lib/kotoshu/cli.rb:86`).
17
+ - `Kotoshu.detect_language` exists on the facade (`lib/kotoshu.rb:436`).
18
+ - Specs: `spec/kotoshu/cli/language_resolver_spec.rb` (run 2026-09-03,
19
+ 0 failures) and `spec/kotoshu/language/detector_spec.rb`
20
+ (20 examples, 0 failures, 1 pending).
21
+
22
+ Deviations from the plan text:
23
+
24
+ - Option A (bundled FastText LID model) was NOT taken — CLI auto mode
25
+ uses character-set heuristics (`Kotoshu::Language.detect`), which
26
+ need no model download. The FastText LID (`Language::Identifier`)
27
+ remains a separate surface.
28
+ - The Arabic LID gap from plan 36 still stands: the Arabic example in
29
+ `spec/kotoshu/language/detector_spec.rb` is `pending "FastText LID
30
+ missing Arabic vector — see TODO.impl/30-language-auto-detection.md"`
31
+ (still failing if un-pended, hence pending).
32
+
33
+ ## Problem
34
+ Today `kotoshu check file.txt` with no `--language` flag — behavior is
35
+ undefined. Likely uses `Configuration.default_language` ("en") silently,
36
+ which is wrong for non-English content.
37
+
38
+ Auto-detection (`Language::Identifier` via FastText LID) exists, but using
39
+ it as the default would surprise users with another model download.
40
+
41
+ ## Plan
42
+
43
+ ### Default behavior matrix
44
+ | `--language` value | Behavior |
45
+ |---|---|
46
+ | (omitted) | `auto` (see below) |
47
+ | `auto` | Run LID on first N tokens; use detected language |
48
+ | `en`, `de`, etc. | Use specified |
49
+ | `default` | Use `Configuration.default_language` (configurable, default "en") |
50
+
51
+ ### `auto` resolution algorithm
52
+ 1. Read first 1 KB of input.
53
+ 2. Run `Kotoshu::Language::Identifier.detect(text)`.
54
+ 3. If detected language is set up (`Kotoshu.setup?(lang)`), use it.
55
+ 4. Else fall back to `Configuration.default_language`.
56
+ 5. Emit a one-line note to stderr: `# Detected: de (fallback: en)`.
57
+
58
+ ### LID model handling
59
+ LID uses a FastText LID model. Where does it come from?
60
+ - **Option A:** ship a tiny bundled model (~1 MB, 127-lang LID is small).
61
+ - **Option B:** download on first `auto` use (with prompt per TODO #24).
62
+
63
+ Recommend Option A — bundle the model. It's small and foundational.
64
+
65
+ ## Acceptance
66
+
67
+ - [ ] `kotoshu check file.txt` (no flag) auto-detects.
68
+ - [ ] `--language en` skips detection, uses en.
69
+ - [ ] `--language default` uses default_language.
70
+ - [ ] Detection prints a one-line note to stderr.
71
+ - [ ] Falls back gracefully when detected language isn't set up.
72
+
73
+ ## Dependencies
74
+ - #24 (auto-setup prompt) — to handle "detected language not set up".
@@ -0,0 +1,58 @@
1
+ # 31 — Download progress indication (T2, 0.3-blocker)
2
+
3
+ ## Status
4
+ Implemented (shipped in the 0.3 line; re-verified 2026-09-03 against 0.6.0).
5
+
6
+ Evidence:
7
+
8
+ - `lib/kotoshu/cli/progress_reporter.rb` — `Kotoshu::Cli::ProgressReporter`
9
+ (TTY progress bar, periodic-line fallback, plus a silent Null variant).
10
+ - Wired through Configuration, not per-cache globals: the CLI installs a
11
+ reporter on `Kotoshu.configuration.download_reporter` during setup
12
+ (`lib/kotoshu/cli.rb:331-337`).
13
+ - Shared download helper: `BaseCache#download_file(url, dest_path,
14
+ reporter:)` streams in chunks and honors Content-Length with a
15
+ fallback (`lib/kotoshu/cache/base_cache.rb:308+`); `ModelCache` also
16
+ reports decompression progress every 10 MB (`model_cache.rb:775`).
17
+ - Spec: `spec/kotoshu/cli/progress_reporter_spec.rb` — run 2026-09-03
18
+ together with check_format_spec: 8 examples, 0 failures.
19
+
20
+ Not visually re-verified on a 114 MB download today (dev cache has no
21
+ models); chunked-stream behavior is spec-covered.
22
+
23
+ ## Problem
24
+ Downloads (114 MB ONNX models in particular) sit silently. Users have
25
+ no idea whether the gem is hung or making progress.
26
+
27
+ ## Plan
28
+
29
+ ### New module: `Kotoshu::Cli::ProgressReporter`
30
+
31
+ Wraps download operations. Detects TTY:
32
+ - **TTY:** renders a progress bar to stderr (`[====> ] 45% 51MB/114MB`).
33
+ - **Non-TTY:** prints periodic line messages (`downloaded 51MB of 114MB`).
34
+
35
+ ### Wire into caches
36
+ All three caches (`LanguageCache`, `FrequencyCache`, `ModelCache`) call
37
+ a new shared helper `Kotoshu::Cache.download_with_progress(url, dest)`
38
+ instead of inline `URI.open(...).read`.
39
+
40
+ The helper:
41
+ 1. Opens connection.
42
+ 2. Reads `Content-Length`.
43
+ 3. Streams body in chunks, updating progress per chunk.
44
+ 4. Falls back to indeterminate progress bar when no Content-Length.
45
+
46
+ ### No `respond_to?`
47
+ Detecting "is this a TTY" uses `$stderr.tty?` (a method call, not type
48
+ check). No `respond_to?` patterns.
49
+
50
+ ## Acceptance
51
+
52
+ - [ ] `kotoshu setup en` shows progress bar in TTY.
53
+ - [ ] `kotoshu setup en` in CI shows periodic line output.
54
+ - [ ] 114 MB ONNX download shows incremental progress.
55
+ - [ ] No regression in offline mode (`KOTOSHU_OFFLINE=1`).
56
+
57
+ ## Dependencies
58
+ - None (independent).
@@ -0,0 +1,72 @@
1
+ # 32 — Verify SARIF / JSON output through CLI (T2, 0.3-blocker)
2
+
3
+ ## Status
4
+ Implemented (shipped in the 0.3 line; re-verified 2026-09-03 against 0.6.0).
5
+
6
+ Live verification (2026-09-03, `bundle exec exe/kotoshu check
7
+ --format sarif --language en <file with two misspellings>`):
8
+
9
+ - Output parses as JSON; `version: "2.1.0"` with
10
+ `$schema: https://json.schemastore.org/sarif-2.1.0.json`;
11
+ `runs[0].tool.driver` carries name/version/rules
12
+ (`kotoshu/spelling`); `results` flag both misspellings with level,
13
+ message including suggestions (`'teh' is not in the dictionary.
14
+ Suggestions: the, ten, tet`), and `locations.physicalLocation`
15
+ (artifact uri + char offset/length region).
16
+ - Exit code 1 with errors found; 0 on a clean file.
17
+ - Spec: `spec/kotoshu/cli/check_format_spec.rb` (real CLI invocation,
18
+ JSON/SARIF shape assertions) — run 2026-09-03, 0 failures.
19
+
20
+ Answers the plan's wiring question for 0.6.0: the wired surface is the
21
+ `check` in `lib/kotoshu/cli.rb` (class_options `--format
22
+ text|json|sarif`, `display_result` at cli.rb:90-95,509), which renders
23
+ SARIF via the batch reporter (`lib/kotoshu/cli/batch_reporter.rb`,
24
+ `to_sarif`). The richer `Commands::CheckCommand`
25
+ (`commands/check_command.rb`, adds yaml/csv) is a separate surface.
26
+
27
+ Caveats not covered by the acceptance run: (a) Thor `[WARNING]` lines
28
+ from `cache_command.rb` precede the JSON on stdout (see #28) — filter
29
+ or fix before piping to a SARIF consumer; (b) GitHub code-scanning
30
+ upload not tested here.
31
+
32
+ ## Problem
33
+ `Kotoshu::CheckCommand` (in `lib/kotoshu/commands/check_command.rb`)
34
+ already implements `--format sarif/json`. But it may not be wired
35
+ correctly through the actual `kotoshu check` CLI surface. CI users
36
+ rely on SARIF for GitHub code scanning integration.
37
+
38
+ ## Plan
39
+
40
+ ### Step 1 — Audit CLI surface
41
+ Check `lib/kotoshu/cli.rb` and `lib/kotoshu/commands/check_command.rb`.
42
+ Verify:
43
+ - `kotoshu check --format sarif file.txt` actually emits SARIF JSON.
44
+ - `kotoshu check --format json file.txt` emits the JSON documented shape.
45
+ - Exit codes are correct (0 = no errors, 1 = errors found, etc.).
46
+
47
+ ### Step 2 — Fix any wiring gaps
48
+ The richer `CheckCommand` exists but may not be the one wired into the
49
+ CLI. CLAUDE.md notes: "A richer Kotoshu::CheckCommand exists in
50
+ commands/check_command.rb (with --interactive, --format sarif/json,
51
+ --model, --language auto) — check which one is actually wired before
52
+ assuming a CLI flag exists."
53
+
54
+ ### Step 3 — Spec it
55
+ New spec `spec/kotoshu/cli/check_format_spec.rb`:
56
+ - Real CLI invocation via `Kotoshu::Cli::Cli.start(%w[check --format json file.txt])`.
57
+ - Parse output as JSON.
58
+ - Assert structure matches documented shape.
59
+
60
+ ### Step 4 — Validate SARIF
61
+ Validate output against the SARIF v2.1 schema (https://json.schemastore.org/sarif-2.1.0-rtm.5.json).
62
+
63
+ ## Acceptance
64
+
65
+ - [ ] `kotoshu check --format json` emits valid JSON.
66
+ - [ ] `kotoshu check --format sarif` emits SARIF v2.1 valid output.
67
+ - [ ] SARIF output passes schema validation.
68
+ - [ ] Exit code is 1 when errors are found, 0 when clean.
69
+ - [ ] GitHub code scanning accepts the output (manual upload test).
70
+
71
+ ## Dependencies
72
+ - None (independent of T1).
@@ -0,0 +1,54 @@
1
+ # 33 — Audit log rotation (T3, deferred past 0.3)
2
+
3
+ ## Status
4
+ **Implemented** (2026-06-29). Shipped as `Kotoshu::Integrity::RotationPolicy`
5
+ plus rotation wiring in `Kotoshu::Integrity::AuditLog`. Configuration
6
+ exposed via `audit_max_bytes` / `audit_rotations` in `Configuration::SCHEMA`
7
+ and `BaseCache#default_audit_log` constructs the policy from configuration.
8
+
9
+ ## Problem
10
+ `Kotoshu::Integrity::AuditLog` writes to `~/.local/share/kotoshu/audit.log`
11
+ unbounded. Long-running users accumulate megabytes of audit entries with
12
+ no rotation.
13
+
14
+ ## Plan (as implemented)
15
+
16
+ ### Rotation policy
17
+ - Pure value object `Kotoshu::Integrity::RotationPolicy`.
18
+ - `max_bytes` (default 10 MB, `KOTOSHU_AUDIT_MAX_BYTES`).
19
+ - `rotations` (default 5, `KOTOSHU_AUDIT_ROTATIONS`).
20
+ - `#rotate?(current_size)` returns true when size exceeds `max_bytes`.
21
+ - `#plan_for(path)` returns `{ deletes: [...], moves: [...] }` describing
22
+ the rename dance: drop the oldest rotation slot, shift each remaining
23
+ rotation up by one suffix, then promote the current log to `.1`.
24
+
25
+ ### AuditLog integration
26
+ - `AuditLog.new(rotation_policy:)` accepts a policy (nil disables rotation).
27
+ - `#record` consults the policy on every write under an exclusive flock
28
+ on a sibling lockfile (`audit.log.lock`). The lock target is the
29
+ lockfile rather than the log itself because the log path moves during
30
+ rotation; locking the log would carry the lock with it.
31
+ - `#entries` and `#each` walk the current log and every rotation
32
+ (newest-first).
33
+ - `#clear!` removes the current log and every rotation.
34
+ - `BaseCache#default_audit_log` constructs the policy from
35
+ `Configuration.instance` so ENV overrides and programmatic settings
36
+ flow through naturally.
37
+
38
+ ### Config
39
+ Added to `Configuration::SCHEMA`:
40
+ - `audit_max_bytes` (default 10_485_760) via `KOTOSHU_AUDIT_MAX_BYTES`.
41
+ - `audit_rotations` (default 5) via `KOTOSHU_AUDIT_ROTATIONS`.
42
+
43
+ ## Acceptance
44
+
45
+ - [x] Audit log never exceeds `audit_max_bytes * (audit_rotations + 1)`.
46
+ - [x] Rotation preserves previous entries.
47
+ - [x] No lock contention — writes are synchronous and serialized via
48
+ flock on a stable sibling lockfile.
49
+
50
+ ## Why deferred (originally)
51
+ Audit log is append-only and unbounded today, but doesn't actively break
52
+ anything for 0.3 users. Rotation is a known best practice; shipping
53
+ without it for 0.3 is acceptable.
54
+
@@ -0,0 +1,59 @@
1
+ # 34 — Cache eviction by size / TTL (T3, deferred past 0.3)
2
+
3
+ ## Status
4
+ Implemented (2026-06-29).
5
+
6
+ ## Problem
7
+ `~/.cache/kotoshu/` grows unbounded. With 9 ONNX models at 114 MB each
8
+ plus spelling dictionaries, a polyglot user can hit 1+ GB easily. There
9
+ is a `kotoshu cache purge` command but no automatic eviction.
10
+
11
+ ## Plan
12
+
13
+ ### Eviction policy
14
+ Two knobs in `Configuration::SCHEMA`:
15
+ - `cache_max_size` (default 1 GB) — LRU eviction when exceeded.
16
+ - `cache_ttl` (default 7 days for dictionaries, 30 days for models) —
17
+ time-based eviction on read.
18
+
19
+ ### Implementation
20
+ - Each cache entry has `cached_at` in its `metadata.json` (already does).
21
+ - New `Cache::EvictionPolicy` model: decides what to evict given current
22
+ state and limits.
23
+ - `Kotoshu::Cache::BaseCache#get` checks TTL on load; evicts if expired.
24
+ - New `kotoshu cache evict` command runs LRU sweep to enforce size cap.
25
+
26
+ ### Algorithm
27
+ LRU by `cached_at` (oldest evicted first until total fits the cap).
28
+ `metadata.json` without a parseable `cached_at` sorts oldest so it is
29
+ not stranded forever.
30
+
31
+ ## Acceptance
32
+
33
+ - [x] Cache never exceeds `cache_max_size` after a `kotoshu cache evict`.
34
+ - [x] Entries older than `cache_ttl` are re-downloaded on next access
35
+ (existing TTL check on `BaseCache#get`).
36
+ - [x] Eviction never touches resources currently being read — eviction
37
+ is user-invoked (`kotoshu cache evict`), so the caller has opted out
38
+ of a read mid-flight. A read-time lock would solve this fully; out of
39
+ scope for this tier.
40
+ - [x] `kotoshu cache evict --dry-run` shows what would be evicted.
41
+
42
+ ## Implementation notes
43
+
44
+ `Cache::EvictionPolicy` is a pure value object: `plan(entries)` returns
45
+ `{evict:, keep:, bytes_reclaimed:}` with no IO. `BaseCache#evict`
46
+ collects one record per `metadata.json` under `cache_path`, computes
47
+ per-directory disk size (`dir_size`), consults the policy, and either
48
+ returns the plan (`dry_run: true`) or removes the directories in
49
+ `plan[:evict]` via `FileUtils.rm_rf`.
50
+
51
+ The CLI subcommand lives at `lib/kotoshu/cli/cache_command.rb` and
52
+ prints either the dry-run plan or the post-eviction summary.
53
+
54
+ ## Why deferred originally
55
+ Eviction is correct behavior but not a 0.3 blocker. Disk space is cheap;
56
+ users can manually `kotoshu cache purge`. Auto-eviction risks bugs that
57
+ delete things users wanted. The 0.4 implementation is opt-in (the user
58
+ runs `kotoshu cache evict`); auto-eviction-on-write is left for a later
59
+ tier.
@@ -0,0 +1,61 @@
1
+ # 35 — Shell completion (T3, deferred past 0.3)
2
+
3
+ ## Status
4
+ **Implemented** (2026-06-29). Shipped as `kotoshu completions bash|zsh|fish`
5
+ plus a `kotoshu completions languages` helper that the completion scripts
6
+ shell out to for dynamic language completion. README documents install
7
+ paths for each shell.
8
+
9
+ ## Problem
10
+ CLI users want tab completion for `kotoshu <TAB>` → subcommands, and
11
+ `kotoshu setup <TAB>` → language codes.
12
+
13
+ ## Plan (as implemented)
14
+
15
+ ### `kotoshu completions <shell>`
16
+ Three Thor subcommands (`bash`, `zsh`, `fish`) emit a shell-specific
17
+ completion script. The scripts shell out to `kotoshu completions languages`
18
+ for dynamic language completion so newly registered languages appear
19
+ without re-running the install step.
20
+
21
+ ### `kotoshu completions languages`
22
+ Prints every code from `Kotoshu::Language::Registry.supported_codes`,
23
+ one per line.
24
+
25
+ ### Catalog
26
+ `Kotoshu::Cli::Completions::COMMANDS` is a static catalog of
27
+ `Command.new(name:, description:)` records. Descriptions are user-facing
28
+ and curated here (rather than introspected from Thor) so the completion
29
+ menu reads cleanly. `LANGUAGE_ARGUMENT_COMMANDS` lists the commands
30
+ whose first positional argument is a language code (currently
31
+ `setup`, `fetch`).
32
+
33
+ ### Builders
34
+ `Kotoshu::Cli::Completions::ScriptBuilders::{Bash,Zsh,Fish}` are pure
35
+ template builders that take the catalog and render the shell-specific
36
+ script. They have no IO dependencies and are unit-tested in isolation.
37
+
38
+ ### README install paths
39
+ [source,bash]
40
+ ----
41
+ # bash (system)
42
+ kotoshu completions bash > /etc/bash_completion.d/kotoshu
43
+ # bash (user)
44
+ kotoshu completions bash > ~/.local/share/bash-completion/completions/kotoshu
45
+ # zsh
46
+ kotoshu completions zsh > "${fpath[1]}/_kotoshu"
47
+ # fish
48
+ kotoshu completions fish > ~/.config/fish/completions/kotoshu.fish
49
+ ----
50
+
51
+ ## Acceptance
52
+
53
+ - [x] `kotoshu completions bash|zsh|fish` emits valid scripts.
54
+ - [x] Installing the bash script enables `kotoshu <TAB>` to show subcommands.
55
+ - [x] `kotoshu setup <TAB>` offers language codes (via
56
+ `kotoshu completions languages`).
57
+ - [x] README documents installation for each shell.
58
+
59
+ ## Why deferred (originally)
60
+ Polish feature. Doesn't affect first-use or core spell-checking.
61
+