kotoshu 1.0.2-x86_64-linux

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (458) hide show
  1. checksums.yaml +7 -0
  2. data/.kotoshu-baseline.json +1 -0
  3. data/.pre-commit-hooks.yaml +18 -0
  4. data/.rspec +3 -0
  5. data/.rubocop.yml +351 -0
  6. data/.yardopts +8 -0
  7. data/CHANGELOG.md +813 -0
  8. data/CLAUDE.md +176 -0
  9. data/CODE_OF_CONDUCT.md +132 -0
  10. data/Cargo.lock +359 -0
  11. data/Cargo.toml +15 -0
  12. data/LICENSE +31 -0
  13. data/README.adoc +1292 -0
  14. data/Rakefile +150 -0
  15. data/SECURITY.md +93 -0
  16. data/TODO.impl/00-cut-0.1.md +121 -0
  17. data/TODO.impl/00-cut-0.2.md +175 -0
  18. data/TODO.impl/00-vision.md +281 -0
  19. data/TODO.impl/01-hunspell-correctness.md +111 -0
  20. data/TODO.impl/02-cli-unification.md +74 -0
  21. data/TODO.impl/03-dynamic-download.md +107 -0
  22. data/TODO.impl/04-language-modules.md +89 -0
  23. data/TODO.impl/05-semantic-path.md +83 -0
  24. data/TODO.impl/06-cjk-support.md +105 -0
  25. data/TODO.impl/07-rtl-support.md +71 -0
  26. data/TODO.impl/08-grammar-engine.md +106 -0
  27. data/TODO.impl/09-integrity-security.md +90 -0
  28. data/TODO.impl/10-testing-ci.md +98 -0
  29. data/TODO.impl/100-full-feature-batch3.md +28 -0
  30. data/TODO.impl/101-int4-tier-experiment.md +32 -0
  31. data/TODO.impl/102-lid-chain.md +28 -0
  32. data/TODO.impl/103-oov-bucket-tables.md +27 -0
  33. data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
  34. data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
  35. data/TODO.impl/106-native-lid-detection.md +36 -0
  36. data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
  37. data/TODO.impl/108-ko-ne-full-feature.md +37 -0
  38. data/TODO.impl/109-ten-audit.md +26 -0
  39. data/TODO.impl/11-release-v1.md +95 -0
  40. data/TODO.impl/110-sr-latn-and-polish.md +33 -0
  41. data/TODO.impl/111-embedding-bakeoff.md +33 -0
  42. data/TODO.impl/112-deployability-and-gating.md +26 -0
  43. data/TODO.impl/113-language-packs.md +26 -0
  44. data/TODO.impl/114-typo-biencoder-thread.md +33 -0
  45. data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
  46. data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
  47. data/TODO.impl/117-deterministic-frequency-tiers.md +110 -0
  48. data/TODO.impl/118-server-gem-floor.md +33 -0
  49. data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
  50. data/TODO.impl/120-setup-metadata-refresh.md +35 -0
  51. data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
  52. data/TODO.impl/122-pack-compression-kpk2.md +66 -0
  53. data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
  54. data/TODO.impl/124-owner-residue.md +19 -0
  55. data/TODO.impl/125-consumer-floor-guards.md +43 -0
  56. data/TODO.impl/126-tier-error-budgets.md +33 -0
  57. data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
  58. data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
  59. data/TODO.impl/129-pt-fr-corpora.md +23 -0
  60. data/TODO.impl/130-news-truth-0912.md +23 -0
  61. data/TODO.impl/131-hybrid-productization.md +42 -0
  62. data/TODO.impl/132-techniques-page-and-prose.md +27 -0
  63. data/TODO.impl/133-rust-first-distribution.md +68 -0
  64. data/TODO.impl/20-encapsulation-cleanup.md +47 -0
  65. data/TODO.impl/21-autoload-migration.md +64 -0
  66. data/TODO.impl/22-url-pin-single-source-of-truth.md +120 -0
  67. data/TODO.impl/23-fix-url-defaults.md +57 -0
  68. data/TODO.impl/24-cli-auto-setup-prompt.md +76 -0
  69. data/TODO.impl/25-soft-onnxruntime-require.md +75 -0
  70. data/TODO.impl/26-onnx-vocab-json-generation.md +96 -0
  71. data/TODO.impl/27-end-to-end-smoke-spec.md +113 -0
  72. data/TODO.impl/28-readme-quickstart-verification.md +68 -0
  73. data/TODO.impl/29-status-command.md +84 -0
  74. data/TODO.impl/30-language-auto-detection.md +74 -0
  75. data/TODO.impl/31-download-progress.md +58 -0
  76. data/TODO.impl/32-sarif-output-verification.md +72 -0
  77. data/TODO.impl/33-audit-log-rotation.md +54 -0
  78. data/TODO.impl/34-cache-eviction.md +59 -0
  79. data/TODO.impl/35-shell-completion.md +61 -0
  80. data/TODO.impl/36-cleanup-regressions-0.3.1.md +134 -0
  81. data/TODO.impl/37-hunspell-correctness-tier2.md +203 -0
  82. data/TODO.impl/38-onnx-semantic-gating.md +218 -0
  83. data/TODO.impl/39-tier3-and-beyond.md +232 -0
  84. data/TODO.impl/40-spec-drift-cleanup.md +134 -0
  85. data/TODO.impl/50-structure-aware-document-api.md +194 -0
  86. data/TODO.impl/51-grammar-engine-expansion.md +75 -0
  87. data/TODO.impl/52-multi-language-documents.md +95 -0
  88. data/TODO.impl/53-document-plugin-architecture.md +94 -0
  89. data/TODO.impl/54-cjk-support.md +117 -0
  90. data/TODO.impl/55-rtl-support.md +109 -0
  91. data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
  92. data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
  93. data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
  94. data/TODO.impl/59-yard-api-documentation.md +75 -0
  95. data/TODO.impl/60-lsp-server.md +174 -0
  96. data/TODO.impl/61-editor-ecosystem.md +167 -0
  97. data/TODO.impl/62-framework-integrations.md +159 -0
  98. data/TODO.impl/63-web-playground-wasm.md +186 -0
  99. data/TODO.impl/64-http-api-and-sdks.md +191 -0
  100. data/TODO.impl/65-universal-kotoshu.md +251 -0
  101. data/TODO.impl/66-kotoshu-core.md +169 -0
  102. data/TODO.impl/67-kotoshu-rs-and-access-libraries.md +137 -0
  103. data/TODO.impl/68-sota-adoption.md +152 -0
  104. data/TODO.impl/69-sota-typo-eval.md +74 -0
  105. data/TODO.impl/70-sota-confidence-cascade.md +59 -0
  106. data/TODO.impl/71-sota-nested-model.md +63 -0
  107. data/TODO.impl/72-model2vec-evaluation.md +49 -0
  108. data/TODO.impl/73-rl-reranker.md +42 -0
  109. data/TODO.impl/74-distilled-reranker.md +36 -0
  110. data/TODO.impl/75-llm-evaluator.md +35 -0
  111. data/TODO.impl/76-optional-llm-backend.md +37 -0
  112. data/TODO.impl/77-model-coverage-expansion.md +96 -0
  113. data/TODO.impl/78-in-browser-playground.md +69 -0
  114. data/TODO.impl/79-news-and-releases.md +60 -0
  115. data/TODO.impl/80-ecosystem-docs.md +56 -0
  116. data/TODO.impl/81-editor-reach.md +68 -0
  117. data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
  118. data/TODO.impl/83-models-batch-2.md +56 -0
  119. data/TODO.impl/84-language-modules-wave2.md +75 -0
  120. data/TODO.impl/85-wasm-model-api.md +48 -0
  121. data/TODO.impl/86-site-truth-wave2.md +45 -0
  122. data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
  123. data/TODO.impl/88-cli-directory-mode.md +54 -0
  124. data/TODO.impl/89-framework-integrations-exec.md +59 -0
  125. data/TODO.impl/90-gem-correctness-residues.md +44 -0
  126. data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
  127. data/TODO.impl/92-tier-mirror-cors.md +88 -0
  128. data/TODO.impl/93-python-wheels.md +32 -0
  129. data/TODO.impl/94-site-integrations-docs.md +35 -0
  130. data/TODO.impl/95-editor-action-polish.md +27 -0
  131. data/TODO.impl/96-server-semantic-models.md +50 -0
  132. data/TODO.impl/97-yard-execution.md +73 -0
  133. data/TODO.impl/98-campaign-cleanup.md +78 -0
  134. data/TODO.impl/99-site-truth-performance-docs.md +31 -0
  135. data/conformance/vectors.jsonl +2630 -0
  136. data/data/en/grammar/rules.yaml +851 -0
  137. data/docs/STABILITY-1.0-AUDIT.md +251 -0
  138. data/docs/STABILITY-1.0.md +137 -0
  139. data/docs/cache-architecture.md +502 -0
  140. data/docs/verification-evidence/correct3/pr_body.md +37 -0
  141. data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
  142. data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
  143. data/docs/verification-evidence/correct3/sv.aff +856 -0
  144. data/docs/verification-evidence/correct3/sv.dic +153717 -0
  145. data/examples/01_basic_word_checking.rb +38 -0
  146. data/examples/02_text_document_checking.rb +77 -0
  147. data/examples/03_dictionary_backends.rb +137 -0
  148. data/examples/04_trie_data_structure.rb +146 -0
  149. data/examples/05_suggestion_algorithms.rb +239 -0
  150. data/examples/06_configuration_advanced.rb +287 -0
  151. data/examples/07_multi_language_dictionaries.rb +278 -0
  152. data/exe/kotoshu +6 -0
  153. data/lib/kotoshu/algorithms/capitalization.rb +276 -0
  154. data/lib/kotoshu/algorithms/edit_distance.rb +128 -0
  155. data/lib/kotoshu/algorithms/lookup/affix_form.rb +132 -0
  156. data/lib/kotoshu/algorithms/lookup/compound_checks.rb +209 -0
  157. data/lib/kotoshu/algorithms/lookup/compound_form.rb +50 -0
  158. data/lib/kotoshu/algorithms/lookup.rb +841 -0
  159. data/lib/kotoshu/algorithms/ngram_suggest.rb +367 -0
  160. data/lib/kotoshu/algorithms/permutations.rb +283 -0
  161. data/lib/kotoshu/algorithms/phonet_suggest.rb +162 -0
  162. data/lib/kotoshu/algorithms/soundex.rb +92 -0
  163. data/lib/kotoshu/algorithms/suggest.rb +794 -0
  164. data/lib/kotoshu/algorithms.rb +23 -0
  165. data/lib/kotoshu/analyzers/semantic_analyzer.rb +405 -0
  166. data/lib/kotoshu/analyzers.rb +8 -0
  167. data/lib/kotoshu/baseline/application.rb +18 -0
  168. data/lib/kotoshu/baseline/entry.rb +33 -0
  169. data/lib/kotoshu/baseline/store.rb +175 -0
  170. data/lib/kotoshu/baseline.rb +20 -0
  171. data/lib/kotoshu/cache/base_cache.rb +778 -0
  172. data/lib/kotoshu/cache/cache.rb +91 -0
  173. data/lib/kotoshu/cache/eviction_policy.rb +66 -0
  174. data/lib/kotoshu/cache/frequency_cache.rb +283 -0
  175. data/lib/kotoshu/cache/language_cache.rb +511 -0
  176. data/lib/kotoshu/cache/lookup_cache.rb +166 -0
  177. data/lib/kotoshu/cache/model_cache.rb +1147 -0
  178. data/lib/kotoshu/cache/model_registry.rb +109 -0
  179. data/lib/kotoshu/cache/staged_languages.rb +45 -0
  180. data/lib/kotoshu/cache/suggestion_cache.rb +113 -0
  181. data/lib/kotoshu/cache.rb +49 -0
  182. data/lib/kotoshu/cli/auto_setup.rb +71 -0
  183. data/lib/kotoshu/cli/baseline_command.rb +119 -0
  184. data/lib/kotoshu/cli/batch_reporter.rb +321 -0
  185. data/lib/kotoshu/cli/cache_command.rb +282 -0
  186. data/lib/kotoshu/cli/completions_command.rb +210 -0
  187. data/lib/kotoshu/cli/dict_command.rb +64 -0
  188. data/lib/kotoshu/cli/directory_check.rb +146 -0
  189. data/lib/kotoshu/cli/directory_walker.rb +132 -0
  190. data/lib/kotoshu/cli/display_formatter.rb +429 -0
  191. data/lib/kotoshu/cli/errors.rb +36 -0
  192. data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
  193. data/lib/kotoshu/cli/interactive_reviewer.rb +314 -0
  194. data/lib/kotoshu/cli/language_resolver.rb +91 -0
  195. data/lib/kotoshu/cli/navigation_manager.rb +273 -0
  196. data/lib/kotoshu/cli/personal_command.rb +113 -0
  197. data/lib/kotoshu/cli/progress_reporter.rb +114 -0
  198. data/lib/kotoshu/cli/status_report.rb +130 -0
  199. data/lib/kotoshu/cli.rb +880 -0
  200. data/lib/kotoshu/components/passthrough_spell_checker.rb +70 -0
  201. data/lib/kotoshu/components/pos_tagger.rb +98 -0
  202. data/lib/kotoshu/components/spell_checker.rb +73 -0
  203. data/lib/kotoshu/components/synthesizer.rb +60 -0
  204. data/lib/kotoshu/components/tokenizer.rb +58 -0
  205. data/lib/kotoshu/components/whitespace_tokenizer.rb +94 -0
  206. data/lib/kotoshu/components.rb +13 -0
  207. data/lib/kotoshu/configuration/builder.rb +207 -0
  208. data/lib/kotoshu/configuration/resolver.rb +124 -0
  209. data/lib/kotoshu/configuration.rb +829 -0
  210. data/lib/kotoshu/conformance_exporter.rb +256 -0
  211. data/lib/kotoshu/conformance_runner.rb +176 -0
  212. data/lib/kotoshu/core/exceptions.rb +186 -0
  213. data/lib/kotoshu/core/indexed_dictionary.rb +293 -0
  214. data/lib/kotoshu/core/models/affix_rule.rb +262 -0
  215. data/lib/kotoshu/core/models/result/document_result.rb +132 -0
  216. data/lib/kotoshu/core/models/result/word_result.rb +131 -0
  217. data/lib/kotoshu/core/models/word.rb +131 -0
  218. data/lib/kotoshu/core/trie/builder.rb +119 -0
  219. data/lib/kotoshu/core/trie/node.rb +94 -0
  220. data/lib/kotoshu/core/trie/trie.rb +249 -0
  221. data/lib/kotoshu/core.rb +28 -0
  222. data/lib/kotoshu/data/common_words/de.yml +1800 -0
  223. data/lib/kotoshu/data/common_words/en.yml +1215 -0
  224. data/lib/kotoshu/data/common_words/es.yml +750 -0
  225. data/lib/kotoshu/data/common_words/fr.yml +1015 -0
  226. data/lib/kotoshu/data/common_words/pt.yml +870 -0
  227. data/lib/kotoshu/data/common_words/ru.yml +484 -0
  228. data/lib/kotoshu/data/common_words_loader.rb +141 -0
  229. data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
  230. data/lib/kotoshu/data/frozen_kelly.rb +38 -0
  231. data/lib/kotoshu/data.rb +8 -0
  232. data/lib/kotoshu/data_structures/bloom_filter.rb +188 -0
  233. data/lib/kotoshu/data_structures.rb +8 -0
  234. data/lib/kotoshu/debug_logger.rb +146 -0
  235. data/lib/kotoshu/debug_mode.rb +136 -0
  236. data/lib/kotoshu/defaults.rb +84 -0
  237. data/lib/kotoshu/dictionaries/catalog.rb +814 -0
  238. data/lib/kotoshu/dictionaries.rb +8 -0
  239. data/lib/kotoshu/dictionary/base.rb +324 -0
  240. data/lib/kotoshu/dictionary/cspell.rb +243 -0
  241. data/lib/kotoshu/dictionary/custom.rb +225 -0
  242. data/lib/kotoshu/dictionary/hunspell.rb +520 -0
  243. data/lib/kotoshu/dictionary/plain_text.rb +302 -0
  244. data/lib/kotoshu/dictionary/repository.rb +246 -0
  245. data/lib/kotoshu/dictionary/unified.rb +236 -0
  246. data/lib/kotoshu/dictionary/unix_words.rb +218 -0
  247. data/lib/kotoshu/dictionary.rb +15 -0
  248. data/lib/kotoshu/documents/document.rb +120 -0
  249. data/lib/kotoshu/documents/plain_text_document.rb +73 -0
  250. data/lib/kotoshu/documents/source_position.rb +45 -0
  251. data/lib/kotoshu/documents/source_range.rb +76 -0
  252. data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
  253. data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
  254. data/lib/kotoshu/documents/suppressions.rb +44 -0
  255. data/lib/kotoshu/documents/text_node.rb +49 -0
  256. data/lib/kotoshu/documents.rb +178 -0
  257. data/lib/kotoshu/embeddings/embedding_pipeline.rb +238 -0
  258. data/lib/kotoshu/embeddings/lru_cache.rb +244 -0
  259. data/lib/kotoshu/embeddings/onnx_runtime_model.rb +411 -0
  260. data/lib/kotoshu/embeddings/protocol.rb +96 -0
  261. data/lib/kotoshu/embeddings/protocols.rb +23 -0
  262. data/lib/kotoshu/embeddings/registry.rb +184 -0
  263. data/lib/kotoshu/embeddings/search.rb +204 -0
  264. data/lib/kotoshu/embeddings/similarity_engine.rb +278 -0
  265. data/lib/kotoshu/embeddings/similarity_search.rb +328 -0
  266. data/lib/kotoshu/embeddings/vocabulary.rb +306 -0
  267. data/lib/kotoshu/embeddings.rb +87 -0
  268. data/lib/kotoshu/fluent_checker.rb +94 -0
  269. data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +48 -0
  270. data/lib/kotoshu/grammar/pattern_matchers/double_negative_matcher.rb +179 -0
  271. data/lib/kotoshu/grammar/pattern_matchers/phrase_matcher.rb +86 -0
  272. data/lib/kotoshu/grammar/pattern_matchers/possessive_context_matcher.rb +75 -0
  273. data/lib/kotoshu/grammar/pattern_matchers/possessive_contraction_matcher.rb +87 -0
  274. data/lib/kotoshu/grammar/pattern_matchers/sentence_start_matcher.rb +84 -0
  275. data/lib/kotoshu/grammar/pattern_matchers/vowel_sound_matcher.rb +81 -0
  276. data/lib/kotoshu/grammar/pattern_matchers/word_list_matcher.rb +66 -0
  277. data/lib/kotoshu/grammar/rule.rb +99 -0
  278. data/lib/kotoshu/grammar/rule_engine.rb +96 -0
  279. data/lib/kotoshu/grammar/rule_loader.rb +30 -0
  280. data/lib/kotoshu/grammar.rb +24 -0
  281. data/lib/kotoshu/integrity/audit_log.rb +167 -0
  282. data/lib/kotoshu/integrity/manifest.rb +116 -0
  283. data/lib/kotoshu/integrity/net_http.rb +51 -0
  284. data/lib/kotoshu/integrity/rotation_policy.rb +88 -0
  285. data/lib/kotoshu/integrity.rb +26 -0
  286. data/lib/kotoshu/jekyll.rb +101 -0
  287. data/lib/kotoshu/keyboard/layout.rb +116 -0
  288. data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
  289. data/lib/kotoshu/keyboard/layouts/azerty.rb +55 -0
  290. data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
  291. data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
  292. data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
  293. data/lib/kotoshu/keyboard/layouts/dvorak.rb +54 -0
  294. data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
  295. data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
  296. data/lib/kotoshu/keyboard/layouts/hrsl.rb +94 -0
  297. data/lib/kotoshu/keyboard/layouts/jcuken.rb +57 -0
  298. data/lib/kotoshu/keyboard/layouts/latin.rb +266 -0
  299. data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
  300. data/lib/kotoshu/keyboard/layouts/qwerty.rb +52 -0
  301. data/lib/kotoshu/keyboard/layouts/qwertz.rb +55 -0
  302. data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
  303. data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
  304. data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
  305. data/lib/kotoshu/keyboard/registry.rb +201 -0
  306. data/lib/kotoshu/keyboard.rb +96 -0
  307. data/lib/kotoshu/kotoshu_native.so +0 -0
  308. data/lib/kotoshu/language/detector.rb +242 -0
  309. data/lib/kotoshu/language/identifier.rb +376 -0
  310. data/lib/kotoshu/language/languages/base.rb +271 -0
  311. data/lib/kotoshu/language/lid_detector.rb +188 -0
  312. data/lib/kotoshu/language/normalizer/arabic.rb +113 -0
  313. data/lib/kotoshu/language/normalizer/base.rb +133 -0
  314. data/lib/kotoshu/language/normalizer/greek.rb +63 -0
  315. data/lib/kotoshu/language/normalizer/hebrew.rb +72 -0
  316. data/lib/kotoshu/language/normalizer/persian.rb +36 -0
  317. data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
  318. data/lib/kotoshu/language/registry.rb +209 -0
  319. data/lib/kotoshu/language/resources/ar/common_words.txt +6753 -0
  320. data/lib/kotoshu/language/resources/ar/confusion_sets.txt +11 -0
  321. data/lib/kotoshu/language/resources/de/common_words.txt +10003 -0
  322. data/lib/kotoshu/language/resources/de/confusion_sets.txt +246 -0
  323. data/lib/kotoshu/language/resources/en/common_words.txt +9979 -0
  324. data/lib/kotoshu/language/resources/en/confusion_sets.txt +871 -0
  325. data/lib/kotoshu/language/resources/es/common_words.txt +9992 -0
  326. data/lib/kotoshu/language/resources/es/confusion_sets.txt +17 -0
  327. data/lib/kotoshu/language/resources/fr/common_words.txt +9993 -0
  328. data/lib/kotoshu/language/resources/fr/confusion_sets.txt +76 -0
  329. data/lib/kotoshu/language/resources/pt/common_words.txt +9977 -0
  330. data/lib/kotoshu/language/resources/pt/confusion_sets.txt +18 -0
  331. data/lib/kotoshu/language/resources/ru/common_words.txt +9951 -0
  332. data/lib/kotoshu/language/resources/ru/confusion_sets.txt +5 -0
  333. data/lib/kotoshu/language/script.rb +128 -0
  334. data/lib/kotoshu/language/segmenter.rb +122 -0
  335. data/lib/kotoshu/language/suika.rb +38 -0
  336. data/lib/kotoshu/language/tokenizer/base.rb +187 -0
  337. data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
  338. data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
  339. data/lib/kotoshu/language/tokenizer/french_tokenizer.rb +172 -0
  340. data/lib/kotoshu/language/tokenizer/german_tokenizer.rb +41 -0
  341. data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
  342. data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
  343. data/lib/kotoshu/language/tokenizer/japanese_tokenizer.rb +55 -0
  344. data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +153 -0
  345. data/lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb +160 -0
  346. data/lib/kotoshu/language/tokenizer/russian_tokenizer.rb +93 -0
  347. data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
  348. data/lib/kotoshu/language/tokenizer/spanish_tokenizer.rb +122 -0
  349. data/lib/kotoshu/language.rb +127 -0
  350. data/lib/kotoshu/languages/ar/language.rb +98 -0
  351. data/lib/kotoshu/languages/bg/language.rb +37 -0
  352. data/lib/kotoshu/languages/ca/language.rb +22 -0
  353. data/lib/kotoshu/languages/cs/language.rb +22 -0
  354. data/lib/kotoshu/languages/da/language.rb +22 -0
  355. data/lib/kotoshu/languages/de/language.rb +554 -0
  356. data/lib/kotoshu/languages/el/language.rb +50 -0
  357. data/lib/kotoshu/languages/en/language.rb +470 -0
  358. data/lib/kotoshu/languages/es/language.rb +476 -0
  359. data/lib/kotoshu/languages/et/language.rb +24 -0
  360. data/lib/kotoshu/languages/fa/language.rb +91 -0
  361. data/lib/kotoshu/languages/fr/language.rb +505 -0
  362. data/lib/kotoshu/languages/he/language.rb +98 -0
  363. data/lib/kotoshu/languages/hr/language.rb +24 -0
  364. data/lib/kotoshu/languages/hu/language.rb +22 -0
  365. data/lib/kotoshu/languages/id/language.rb +25 -0
  366. data/lib/kotoshu/languages/it/language.rb +22 -0
  367. data/lib/kotoshu/languages/ja/language.rb +476 -0
  368. data/lib/kotoshu/languages/ko/language.rb +56 -0
  369. data/lib/kotoshu/languages/latin_base.rb +53 -0
  370. data/lib/kotoshu/languages/lt/language.rb +24 -0
  371. data/lib/kotoshu/languages/lv/language.rb +24 -0
  372. data/lib/kotoshu/languages/nb/language.rb +25 -0
  373. data/lib/kotoshu/languages/ne/language.rb +60 -0
  374. data/lib/kotoshu/languages/nl/language.rb +22 -0
  375. data/lib/kotoshu/languages/nn/language.rb +26 -0
  376. data/lib/kotoshu/languages/pl/language.rb +22 -0
  377. data/lib/kotoshu/languages/pt/language.rb +439 -0
  378. data/lib/kotoshu/languages/ro/language.rb +22 -0
  379. data/lib/kotoshu/languages/ru/language.rb +420 -0
  380. data/lib/kotoshu/languages/sk/language.rb +25 -0
  381. data/lib/kotoshu/languages/sl/language.rb +25 -0
  382. data/lib/kotoshu/languages/sr/language.rb +39 -0
  383. data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
  384. data/lib/kotoshu/languages/sv/language.rb +22 -0
  385. data/lib/kotoshu/languages/tr/language.rb +36 -0
  386. data/lib/kotoshu/languages/uk/language.rb +38 -0
  387. data/lib/kotoshu/languages/vi/language.rb +22 -0
  388. data/lib/kotoshu/languages.rb +64 -0
  389. data/lib/kotoshu/metrics_collector.rb +238 -0
  390. data/lib/kotoshu/metrics_module.rb +121 -0
  391. data/lib/kotoshu/models/context.rb +119 -0
  392. data/lib/kotoshu/models/embedding_model.rb +181 -0
  393. data/lib/kotoshu/models/fasttext_model.rb +212 -0
  394. data/lib/kotoshu/models/nearest_neighbor.rb +96 -0
  395. data/lib/kotoshu/models/onnx_model.rb +330 -0
  396. data/lib/kotoshu/models/semantic_error.rb +204 -0
  397. data/lib/kotoshu/models/suggestion.rb +106 -0
  398. data/lib/kotoshu/models/word_embedding.rb +97 -0
  399. data/lib/kotoshu/models.rb +27 -0
  400. data/lib/kotoshu/multi_language_checker.rb +127 -0
  401. data/lib/kotoshu/native.rb +83 -0
  402. data/lib/kotoshu/native_backend.rb +131 -0
  403. data/lib/kotoshu/paths.rb +79 -0
  404. data/lib/kotoshu/personal_dictionary.rb +105 -0
  405. data/lib/kotoshu/plugins/plugin.rb +61 -0
  406. data/lib/kotoshu/plugins/registry.rb +118 -0
  407. data/lib/kotoshu/plugins.rb +9 -0
  408. data/lib/kotoshu/project_config.rb +76 -0
  409. data/lib/kotoshu/readers/aff_data.rb +530 -0
  410. data/lib/kotoshu/readers/aff_reader.rb +393 -0
  411. data/lib/kotoshu/readers/condition_checker.rb +125 -0
  412. data/lib/kotoshu/readers/dic_reader.rb +208 -0
  413. data/lib/kotoshu/readers/file_reader.rb +377 -0
  414. data/lib/kotoshu/readers/lookup_builder.rb +584 -0
  415. data/lib/kotoshu/readers/readers.rb +17 -0
  416. data/lib/kotoshu/readers.rb +25 -0
  417. data/lib/kotoshu/resource_bundle.rb +55 -0
  418. data/lib/kotoshu/resource_manager.rb +600 -0
  419. data/lib/kotoshu/results/result.rb +165 -0
  420. data/lib/kotoshu/results.rb +8 -0
  421. data/lib/kotoshu/rspec.rb +163 -0
  422. data/lib/kotoshu/scripts/fasttext_to_onnx.py +275 -0
  423. data/lib/kotoshu/source_registry.rb +104 -0
  424. data/lib/kotoshu/spellchecker/parallel_checker.rb +90 -0
  425. data/lib/kotoshu/spellchecker.rb +406 -0
  426. data/lib/kotoshu/string_metrics.rb +153 -0
  427. data/lib/kotoshu/suggestions/context.rb +43 -0
  428. data/lib/kotoshu/suggestions/frequency_provider.rb +80 -0
  429. data/lib/kotoshu/suggestions/generator.rb +169 -0
  430. data/lib/kotoshu/suggestions/pipeline.rb +135 -0
  431. data/lib/kotoshu/suggestions/semantic_cascade.rb +73 -0
  432. data/lib/kotoshu/suggestions/strategies/base_strategy.rb +315 -0
  433. data/lib/kotoshu/suggestions/strategies/composite_strategy.rb +197 -0
  434. data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +631 -0
  435. data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +207 -0
  436. data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +165 -0
  437. data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +284 -0
  438. data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +317 -0
  439. data/lib/kotoshu/suggestions/strategies/symspell_strategy.rb +270 -0
  440. data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
  441. data/lib/kotoshu/suggestions/suggestion.rb +154 -0
  442. data/lib/kotoshu/suggestions/suggestion_set.rb +270 -0
  443. data/lib/kotoshu/suggestions/sweep_index.rb +137 -0
  444. data/lib/kotoshu/suggestions.rb +29 -0
  445. data/lib/kotoshu/tasks/check_task.rb +104 -0
  446. data/lib/kotoshu/tasks.rb +11 -0
  447. data/lib/kotoshu/validators/spelling_validator.rb +101 -0
  448. data/lib/kotoshu/validators.rb +9 -0
  449. data/lib/kotoshu/version.rb +5 -0
  450. data/lib/kotoshu.rb +538 -0
  451. data/rakelib/compile.rake +10 -0
  452. data/script/validate_all_dictionaries.rb +444 -0
  453. data/scripts/build_native_gem.rb +45 -0
  454. data/scripts/extract_eval_grids.rb +37 -0
  455. data/scripts/generate_frozen_tiers.rb +44 -0
  456. data/sig/kotoshu.rbs +435 -0
  457. data/test_oop.rb +79 -0
  458. metadata +546 -0
data/Rakefile ADDED
@@ -0,0 +1,150 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "bundler/gem_tasks"
4
+ require "rspec/core/rake_task"
5
+
6
+ RSpec::Core::RakeTask.new(:spec)
7
+
8
+ require "rubocop/rake_task"
9
+
10
+ RuboCop::RakeTask.new
11
+
12
+ task default: %i[spec rubocop]
13
+
14
+ # Conformance failure reporters (kept as lambdas so nothing leaks into
15
+ # Object from the Rakefile).
16
+ report_failures = lambda do |failures, label = nil|
17
+ failures.first(10).each do |failure|
18
+ prefix = label ? "[#{label}] " : ""
19
+ puts "#{prefix}##{failure.index} #{failure.kind} #{failure.input.inspect} " \
20
+ "on #{failure.dictionary}: expected #{failure.expected.inspect}, got #{failure.actual.inspect}"
21
+ end
22
+ puts "... and #{failures.size - 10} more" if failures.size > 10
23
+ end
24
+
25
+ report_divergences = lambda do |divergences|
26
+ divergences.first(10).each do |divergence|
27
+ puts "[divergence] ##{divergence.index} #{divergence.kind} #{divergence.input.inspect} " \
28
+ "on #{divergence.dictionary}: ruby #{divergence.expected.inspect}, " \
29
+ "native #{divergence.actual.inspect}"
30
+ end
31
+ puts "... and #{divergences.size - 10} more" if divergences.size > 10
32
+ end
33
+
34
+ namespace :kotoshu do
35
+ namespace :staged_languages do
36
+ desc <<~DESC
37
+ Regenerate the vendored staged-language list (plan 107). Fetches \
38
+ the dictionaries manifest at the SourceRegistry pin, extracts the \
39
+ distinct language codes from its resources, and rewrites \
40
+ lib/kotoshu/cache/staged_languages.rb in place so \
41
+ LanguageCache::AVAILABLE_LANGUAGES tracks the manifest. Run after \
42
+ the dictionaries repo stages new languages.
43
+ DESC
44
+ task :sync do
45
+ require "kotoshu"
46
+ require "json"
47
+
48
+ url = Kotoshu::SourceRegistry.new.url_for(:dict_manifest)
49
+ puts "Fetching #{url}"
50
+ bytes = Kotoshu::Integrity::NetHTTP.get(url)
51
+ abort "manifest not found at #{url}" if bytes.nil?
52
+
53
+ manifest = JSON.parse(bytes)
54
+ codes = manifest.fetch("resources").values.map { |r| r.fetch("language") }.uniq.sort
55
+ abort "manifest carried no languages" if codes.empty?
56
+
57
+ target = File.expand_path("lib/kotoshu/cache/staged_languages.rb", __dir__)
58
+ source = File.read(target)
59
+ updated = source.sub(/CODES = %w\[\n(?: {8}\S+(?: \S+)*\n)+ {6}\]\.freeze/) do
60
+ wrapped = codes.each_slice(10).map { |slice| " #{slice.join(' ')}" }.join("\n")
61
+ "CODES = %w[\n#{wrapped}\n ].freeze"
62
+ end
63
+ abort "CODES block not found in #{target}" if updated == source
64
+
65
+ File.write(target, updated)
66
+ puts "Rewrote #{target}: #{codes.size} languages (was #{Kotoshu::Cache::StagedLanguages::CODES.size})"
67
+ end
68
+ end
69
+
70
+ namespace :conformance do
71
+ desc <<~DESC
72
+ Export conformance vectors for kotoshu-rs (plan 67 M3). Walks the Spylls \
73
+ fixtures (spec/integrational/fixtures) plus the spec fixture corpus, runs \
74
+ the real engine over every fixture word, and writes one JSON object per \
75
+ line. `expected` values are FROZEN engine behavior (byte-reproducible \
76
+ cross-implementation), not claims about linguistic correctness. \
77
+ Output path: ENV["CONFORMANCE_VECTORS_PATH"] or conformance/vectors.jsonl \
78
+ (repo root). The generated file IS committed -- conformance/ is \
79
+ deliberately NOT gitignored -- and kotoshu-rs copies it into \
80
+ tests/conformance/ via its documented sync step so its CI needs no gem.
81
+ DESC
82
+ task :export do
83
+ path = ENV.fetch("CONFORMANCE_VECTORS_PATH", nil) || "conformance/vectors.jsonl"
84
+ require "kotoshu"
85
+ result = Kotoshu::ConformanceExporter.export(path: path)
86
+ puts "Wrote #{result.row_count} vectors from #{result.corpus_count} corpora to #{result.path}"
87
+ unless result.skipped_corpora.empty?
88
+ puts "Skipped corpora:"
89
+ result.skipped_corpora.each { |id, reason| puts " #{id}: #{reason}" }
90
+ end
91
+ result.skipped_words.each { |(id, word), reason| puts " Skipped word #{id} #{word.inspect}: #{reason}" }
92
+ end
93
+
94
+ desc <<~DESC
95
+ Replay conformance/vectors.jsonl through the pure-Ruby engine \
96
+ (plan 66 dual-backend suite). Every vector must reproduce its frozen \
97
+ expectation; exits nonzero on any mismatch.
98
+ DESC
99
+ task :ruby do
100
+ require "kotoshu"
101
+ result = Kotoshu::ConformanceRunner.new.run(backend: :ruby)
102
+ puts "ruby backend: #{result.row_count} vectors, #{result.failures.size} failures"
103
+ report_failures.call(result.failures)
104
+ abort "kotoshu:conformance:ruby: #{result.failures.size} of #{result.row_count} vectors diverged" unless result.ok?
105
+ end
106
+
107
+ desc <<~DESC
108
+ Replay conformance/vectors.jsonl through the native (Rust) engine \
109
+ (plan 66 dual-backend suite). Requires the extension to be built \
110
+ (rake compile); exits nonzero on any mismatch.
111
+ DESC
112
+ task :native do
113
+ require "kotoshu"
114
+ unless Kotoshu::Native.available?
115
+ abort "kotoshu:conformance:native requires the native extension -- run `rake compile` first"
116
+ end
117
+
118
+ result = Kotoshu::ConformanceRunner.new.run(backend: :native)
119
+ puts "native backend: #{result.row_count} vectors, #{result.failures.size} failures"
120
+ report_failures.call(result.failures)
121
+ abort "kotoshu:conformance:native: #{result.failures.size} of #{result.row_count} vectors diverged" unless result.ok?
122
+ end
123
+
124
+ desc <<~DESC
125
+ Run the conformance vectors through BOTH backends and diff every row \
126
+ (plan 66 acceptance: zero diffs). Also fails when either backend \
127
+ diverges from the frozen expectations. Requires the extension \
128
+ (rake compile).
129
+ DESC
130
+ task :compare do
131
+ require "kotoshu"
132
+ unless Kotoshu::Native.available?
133
+ abort "kotoshu:conformance:compare requires the native extension -- run `rake compile` first"
134
+ end
135
+
136
+ result = Kotoshu::ConformanceRunner.new.compare
137
+ puts "compare: #{result.ruby.row_count} vectors -- ruby failures: " \
138
+ "#{result.ruby.failures.size}, native failures: #{result.native.failures.size}, " \
139
+ "backend divergences: #{result.divergences.size}"
140
+ report_failures.call(result.ruby.failures, "ruby")
141
+ report_failures.call(result.native.failures, "native")
142
+ report_divergences.call(result.divergences)
143
+ unless result.ok?
144
+ abort "kotoshu:conformance:compare: backends diverged " \
145
+ "(ruby #{result.ruby.failures.size}, native #{result.native.failures.size}, " \
146
+ "cross #{result.divergences.size} of #{result.ruby.row_count})"
147
+ end
148
+ end
149
+ end
150
+ end
data/SECURITY.md ADDED
@@ -0,0 +1,93 @@
1
+ # Security Policy
2
+
3
+ ## Reporting a Vulnerability
4
+
5
+ Email security@kotoshu.org with details and a reproduction. You will
6
+ receive an acknowledgement within 72 hours. Please do not file public
7
+ issues for security reports.
8
+
9
+ ## Supported Versions
10
+
11
+ The latest 0.x minor is supported. Older minors are not maintained.
12
+
13
+ ## Threat Model
14
+
15
+ Kotoshu downloads dictionaries, frequency lists, and (optionally)
16
+ embedding models from public GitHub repositories on first use:
17
+
18
+ | Resource | Source repo | Cached at |
19
+ |---|---|---|
20
+ | Spelling dictionaries | `kotoshu/dictionaries` | `~/.kotoshu/languages/{code}/` |
21
+ | Kelly frequency lists | `kotoshu/frequency-list-kelly` | `~/.kotoshu/frequency-lists/{code}/` |
22
+ | FastText / ONNX models | `kotoshu/models-fasttext-onnx` | `~/.kotoshu/models/{code}/` |
23
+
24
+ Downloads flow over HTTPS from `raw.githubusercontent.com`. The threat
25
+ model assumes GitHub serves the bytes the repository owner committed,
26
+ and treats anyone with push access to those repos as trusted.
27
+
28
+ ## Integrity Verification
29
+
30
+ Each content repo may ship a `manifest.json` at its root listing every
31
+ file with its SHA-256 hash, size, and language/type tags. When a
32
+ manifest is present, every download is verified against it:
33
+
34
+ 1. The manifest is fetched once per cache session.
35
+ 2. Each downloaded file's SHA-256 is computed locally and compared to
36
+ the manifest entry.
37
+ 3. On mismatch, `Kotoshu::IntegrityError` is raised with the expected
38
+ and actual hashes. The download is rejected and the cache is left
39
+ untouched.
40
+ 4. Every verification outcome (verified / unverified / mismatch /
41
+ missing) is appended to `~/.kotoshu/audit.log` as one JSON object
42
+ per line.
43
+
44
+ ### Graceful Degradation
45
+
46
+ When a manifest is **absent** (HTTP 404), verification silently
47
+ downgrades to `"unverified"` status and the download proceeds. This
48
+ preserves forward compatibility with repos that have not yet shipped a
49
+ manifest. The audit log records the difference.
50
+
51
+ ### Strict Mode
52
+
53
+ `Kotoshu.spellchecker_for(lang, strict: true)` (and the CLI's
54
+ `--strict` flag) re-raise on any optional-resource failure — including
55
+ integrity mismatches on frequency data — instead of silently
56
+ degrading. Spelling-dictionary integrity is always enforced.
57
+
58
+ ## Cache Layout
59
+
60
+ The cache is written under `$KOTOSHU_HOME` (default `~/.kotoshu/`).
61
+ Files are created with the user's default umask. Cache contents are
62
+ not encrypted at rest.
63
+
64
+ ## Audit Log
65
+
66
+ `~/.kotoshu/audit.log` is append-only, JSON-per-line, and never
67
+ auto-rotated. Operators in multi-user environments should rotate it
68
+ via logrotate or equivalent. To inspect:
69
+
70
+ [source,bash]
71
+ ----
72
+ cat ~/.kotoshu/audit.log | jq .
73
+ ----
74
+
75
+ To clear the audit log:
76
+
77
+ [source,ruby]
78
+ ----
79
+ Kotoshu::Integrity::AuditLog.new(path: "#{ENV['HOME']}/.kotoshu/audit.log").clear!
80
+ ----
81
+
82
+ ## Network Egress
83
+
84
+ `offline: true` (or `KOTOSHU_OFFLINE=1` or `--offline`) disables all
85
+ network egress and only reads from the on-disk cache. If a required
86
+ resource is not cached, the call raises `Kotoshu::ResourceNotCachedError`
87
+ (CLI exits 3). Use `kotoshu fetch LANGUAGE` to pre-warm the cache in
88
+ environments without outbound network access.
89
+
90
+ ## Scope
91
+
92
+ This policy covers the kotoshu gem itself. Vulnerabilities in
93
+ dependencies (Thor, suika, onnxruntime) should be reported upstream.
@@ -0,0 +1,121 @@
1
+ # 00-cut-0.1 — The 0.1.0 Release Cut
2
+
3
+ ## Goal
4
+
5
+ Ship `kotoshu-0.1.0.gem` to rubygems.org with a clear, honest scope: the
6
+ pure-Ruby API and CLI work for English via system dictionaries; everything
7
+ else is documented as in-progress.
8
+
9
+ This is **not** v1.0. The point of 0.1 is to be installable, runnable,
10
+ and discoverable — to claim the name on rubygems.org and establish the
11
+ public API shape.
12
+
13
+ ## 0.1.0 scope (locked)
14
+
15
+ In scope for 0.1.0:
16
+
17
+ | Capability | Status |
18
+ |---|---|
19
+ | `Kotoshu.correct?(word)` / `suggest` / `check` / `check_file` | ✅ Working |
20
+ | `Kotoshu.detect_language(text)` (FastText LID) | ✅ Working |
21
+ | `kotoshu check TARGET` basic CLI | ✅ Working |
22
+ | `kotoshu dict list/info`, `kotoshu cache list/info/...` | ✅ Working |
23
+ | Hunspell dictionary backend (Ruby port) | ✅ Working for common cases |
24
+ | CSpell, UnixWords, PlainText, Custom backends | ✅ Working |
25
+ | Edit distance / phonetic / keyboard / n-gram / symspell strategies | ✅ Working |
26
+ | Configuration via `Kotoshu.configure` + ENV (`KOTOSHU_*`) + CLI | ✅ Working |
27
+ | 3-layer cache (LanguageCache, FrequencyCache, ModelCache) | ✅ Working |
28
+ | Document parsing (Plain text, Markdown, AsciiDoc) | ✅ Working |
29
+ | LICENSE (BSD-2-Clause), CHANGELOG.md, accurate README | ✅ Shipped |
30
+
31
+ Explicitly **deferred** to later 0.x or 1.0:
32
+
33
+ | Capability | Target | Plan |
34
+ |---|---|---|
35
+ | Unified `ResourceManager` for "any language, just works" | 0.3 | `03-dynamic-download.md` |
36
+ | `--interactive`, `--format sarif`, `--model`, `--language auto` wired | 0.2 | `02-cli-unification.md` |
37
+ | Hybrid (Hunspell + ONNX) as default | 0.4 | `05-semantic-path.md` |
38
+ | Hunspell compound rules, circumfix, ICONV/OCONV, ß, dotless-i | 0.5 | `01-hunspell-correctness.md` |
39
+ | ≥30 language modules wired | 0.6 | `04-language-modules.md` |
40
+ | Grammar rule packs | 0.7 | `08-grammar-engine.md` |
41
+ | CJK (Japanese, Chinese) | 0.8 | `06-cjk-support.md` |
42
+ | RTL (Arabic, Hebrew, Persian, Urdu) | 0.9 | `07-rtl-support.md` |
43
+ | Integrity verification (checksums, signed manifests) | 0.4 | `09-integrity-security.md` |
44
+ | v1.0 | — | `11-release-v1.md` |
45
+
46
+ ## Tasks executed for 0.1.0
47
+
48
+ 1. ✅ Audit current state — `bundle install`, smoke-test `Kotoshu.correct?`,
49
+ run rspec. Result: 803/866 (92.7%) passing; VERSION already `0.1.0`.
50
+ 2. ✅ Add `LICENSE` (BSD-2-Clause — was missing despite gemspec claim).
51
+ 3. ✅ Add `CHANGELOG.md` with full 0.1.0 scope and known-limitation list.
52
+ 4. ✅ Fix README accuracy: license, status section at top, ONNX env gate.
53
+ 5. ✅ Tighten CI workflow — rubocop no longer `continue-on-error`.
54
+ 6. ✅ Prepare release checklist (see "Release execution" below).
55
+
56
+ ## Release execution (manual — requires maintainer)
57
+
58
+ The global project rule is "AI never pushes tags, never pushes to main,
59
+ never publishes." The following must be run by a human:
60
+
61
+ ```bash
62
+ # 1. On a clean main, verify everything passes
63
+ git checkout main
64
+ git pull
65
+ bundle install
66
+ bundle exec rspec
67
+ bundle exec rubocop
68
+
69
+ # 2. Verify VERSION
70
+ grep VERSION lib/kotoshu/version.rb
71
+ # Should read: VERSION = "0.1.0"
72
+
73
+ # 3. Build the gem
74
+ gem build kotoshu.gemspec
75
+ # Produces kotoshu-0.1.0.gem
76
+
77
+ # 4. Sanity-check the gem contents
78
+ gem contents kotoshu-0.1.0.gem --tarball
79
+
80
+ # 5. Tag (the human decides when)
81
+ git tag -a v0.1.0 -m "kotoshu 0.1.0 — first public release"
82
+ git push origin v0.1.0
83
+
84
+ # 6. Publish to rubygems.org
85
+ gem push kotoshu-0.1.0.gem
86
+
87
+ # 7. Cut a GitHub Release
88
+ gh release create v0.1.0 --notes-from-tag
89
+
90
+ # 8. Smoke test from a clean environment
91
+ docker run --rm -it ruby:3.3 bash -c '
92
+ gem install kotoshu
93
+ ruby -e "require \"kotoshu\"; puts Kotoshu.correct?(\"hello\")"
94
+ echo "Hello wrold" | ruby -e "require \"kotoshu\"; puts Kotoshu.check(STDIN.read).errors.map(&:word)"
95
+ '
96
+ ```
97
+
98
+ ## Post-release
99
+
100
+ - Update `TODO.impl/00-vision.md` to reflect that 0.1 shipped; the
101
+ critical path now runs through `03-dynamic-download` →
102
+ `04-language-modules` → `11-release-v1`.
103
+ - Start `02-cli-unification.md` (next-highest user-visible value).
104
+ - Start `01-hunspell-correctness.md` task 1 (audit) to get an honest
105
+ fixture-pass number.
106
+
107
+ ## Acceptance criteria
108
+
109
+ - `gem install kotoshu` works on Ruby 3.1+
110
+ - `Kotoshu.correct?("hello")` returns `true` on a fresh install with
111
+ no env vars set
112
+ - `Kotoshu.correct?("helo")` returns `false`
113
+ - `kotoshu check README.adoc` runs without crashing
114
+ - README has no false claims (license, model count, CLI flags)
115
+ - LICENSE and CHANGELOG.md exist
116
+ - rubygems.org page shows v0.1.0 with correct metadata
117
+
118
+ ## Status
119
+
120
+ v0.1.0 release-prep complete on this branch (2026-06-25). Release
121
+ execution pending maintainer sign-off.
@@ -0,0 +1,175 @@
1
+ # 00-cut-0.2 — The First Functional Library
2
+
3
+ ## Goal
4
+
5
+ Ship `kotoshu-0.2.0` — the first release where `gem install kotoshu`
6
+ works on any Ruby 3.1+ system with **no system dictionary dependency**.
7
+ One call, `Kotoshu.check("Hello wrold")`, auto-downloads the English
8
+ Hunspell dictionary on first use and returns correct results. The CLI
9
+ works end-to-end with proper exit codes.
10
+
11
+ 0.1.0 proved the API shape. 0.2.0 delivers the promise: "just works,
12
+ no setup." For 0.2.0 the resolution machinery is English-only; the
13
+ code is language-agnostic so adding more languages is a content task
14
+ (shipping dictionaries in `kotoshu/dictionaries`), not a code change.
15
+
16
+ ## 0.2.0 scope (locked)
17
+
18
+ | Capability | Source plan |
19
+ |---|---|
20
+ | `ResourceManager.resolve` (spelling + frequency) | `03` tasks 1–2 |
21
+ | `ResourceBundle` value object | `03` task 1 |
22
+ | en_US Hunspell auto-download on first use | `03` task 2 |
23
+ | Spellchecker accepts `ResourceBundle` (Strangler-Fig) | `03` |
24
+ | One canonical `CheckCommand` wired into CLI | `02` task 1 |
25
+ | Exit codes 0/1/2/3 | `02` task 6 |
26
+ | `--language en\|auto`, `--format text\|json` | `02` task 5 |
27
+ | en_US Hunspell fixtures at 100 % | `01` tasks 1–2 |
28
+ | Offline mode (`offline: true`, `KOTOSHU_OFFLINE=1`) | `03` task 4 |
29
+ | Fake-server integration spec | `03` task 9 |
30
+
31
+ Explicitly **deferred** to 0.3+:
32
+
33
+ | Capability | Target | Plan |
34
+ |---|---|---|
35
+ | Checksum / integrity verification | 0.3 | `09-integrity-security` |
36
+ | `--interactive`, `--format sarif` | 0.3 | `02` tasks 4, 5 |
37
+ | `kotoshu fetch` pre-warm command | 0.3 | `03` task 6 |
38
+ | Strict mode | 0.3 | `03` task 5 |
39
+ | ONNX model auto-download (semantic rerank) | 0.4 | `05-semantic-path` |
40
+ | ≥30 language modules wired | 0.6 | `04-language-modules` |
41
+ | Grammar rule packs | 0.7 | `08-grammar-engine` |
42
+ | CJK (Japanese, Chinese) | 0.8 | `06-cjk-support` |
43
+ | RTL (Arabic, Hebrew, Persian, Urdu) | 0.9 | `07-rtl-support` |
44
+ | Hunspell MECE 4-layer refactor | 1.0 | `01` task 6 |
45
+
46
+ ## Critical path — three phases
47
+
48
+ ### Phase A — ResourceManager (foundation)
49
+
50
+ The Spellchecker today takes a single `Dictionary::Base` the user must
51
+ configure manually. Phase A introduces `ResourceManager` as the thing
52
+ that produces that dictionary (plus frequency data) from a language
53
+ code, using the existing `LanguageCache`.
54
+
55
+ 1. **`ResourceBundle`** (`lib/kotoshu/resource_bundle.rb`). Plain
56
+ value object — Struct or class with `language`, `dictionary`,
57
+ `frequency`, `model` (nil for 0.2), `rules` (nil for 0.2),
58
+ `cached`, `source_urls`. No hand-rolled `to_h` (project rule).
59
+ 2. **`ResourceManager`** (`lib/kotoshu/resource_manager.rb`,
60
+ autoloaded from `lib/kotoshu.rb`). API:
61
+ ```ruby
62
+ Kotoshu::ResourceManager.resolve(
63
+ text: "Hello wrold",
64
+ language: :auto, # or "en", "de", nil
65
+ want: %i[spelling frequency],
66
+ offline: false
67
+ ) # => ResourceBundle
68
+ ```
69
+ Internals: detect language via `Language::Identifier` when `:auto`;
70
+ fetch dictionary via `LanguageCache.get(language)`; fetch frequency
71
+ via `FrequencyCache.get(language)` when wanted; construct
72
+ `ResourceBundle`. No new network code — the caches already
73
+ download from `kotoshu/dictionaries` and `kotoshu/frequency-list-kelly`.
74
+ 3. **Spellchecker accepts `ResourceBundle`.** Modify
75
+ `lib/kotoshu/spellchecker.rb` to accept either a `ResourceBundle`
76
+ (new path) or a `Dictionary::Base` (legacy path). Strangler-Fig —
77
+ don't break the 0.1 API. Extract `dictionary` and `frequency` from
78
+ the bundle; ignore `model`/`rules` (nil for 0.2).
79
+ 4. **First-use UX.** When `LanguageCache` downloads, print
80
+ `[en] downloading dictionary (X.X MB)…` to STDERR. One line per
81
+ resource, not per byte. Suppress when `$stdout.tty?` is false and
82
+ `KOTOSHU_QUIET` is set.
83
+ 5. **Configuration.** Add to `Configuration::SCHEMA`:
84
+ - `offline` (boolean, default `false`) — also via `KOTOSHU_OFFLINE=1`
85
+ - `default_language` (string, default `"en"`) — used when
86
+ `language: nil` and detection is inconclusive
87
+ 6. **Offline mode.** `ResourceManager.resolve(..., offline: true)`
88
+ only uses cached resources; raises `ResourceNotCachedError` with a
89
+ message naming what's missing and how to pre-fetch.
90
+
91
+ **Phase A acceptance:** `Kotoshu.correct?("hello")` works on a Docker
92
+ `ruby:3.3` container with no `/usr/share/dict/words`, auto-downloading
93
+ en_US on first call.
94
+
95
+ ### Phase B — English correctness
96
+
97
+ 7. **Audit en_US fixtures.** Run
98
+ `bundle exec rspec spec/hunspell_tests/ --tag lang:en` (or filter
99
+ by fixture path). Record which examples fail and categorize:
100
+ compound words, capitalization, affix combinations, special
101
+ characters. This is `01` task 1 scoped to en_US only.
102
+ 8. **Fix en_US failures.** Target the categories from the audit. The
103
+ full MECE 4-layer refactor (`01` task 6) is explicitly **not**
104
+ required for 0.2 — patch the procedural `algorithms/lookup.rb`
105
+ where needed. Each fix gets a named spec example.
106
+ 9. **Suggestion sanity.** Verify `Kotoshu.suggest("helo").first.word`
107
+ == `"hello"` for the top-20 common misspellings. Tune
108
+ `EditDistanceStrategy` weight or `frequency_bonus` if the top
109
+ suggestion is wrong.
110
+
111
+ **Phase B acceptance:** 100 % of en_US Hunspell fixtures pass.
112
+ `suggest("helo")[0]` == `"hello"`.
113
+
114
+ ### Phase C — CLI wired
115
+
116
+ 10. **Wire `CheckCommand` as canonical `check`.** Remove the basic
117
+ `check` from `lib/kotoshu/cli.rb`; register
118
+ `Kotoshu::CheckCommand` (from `commands/check_command.rb`) via
119
+ `subcommand "check", Kotoshu::CheckCommand`. This is `02` task 1.
120
+ 11. **Exit codes** (`02` task 6):
121
+ - `0` — no errors found
122
+ - `1` — errors found
123
+ - `2` — usage error
124
+ - `3` — dictionary fetch failure (network down, not cached + offline)
125
+ 12. **Flag reconciliation** (`02` task 5). For 0.2, support exactly:
126
+ - `kotoshu check FILE` — default check, text output
127
+ - `--language en|de|...|auto` — language selection (`auto` =
128
+ detect via `Language::Identifier`)
129
+ - `--format text|json` — output format (sarif deferred to 0.3)
130
+ - `--offline` — use cached resources only
131
+ Remove or hide every other flag the README mentions but code
132
+ doesn't support yet. Update README to match.
133
+ 13. **End-to-end CLI spec.** Under `spec/kotoshu/cli/`, using a temp
134
+ dir (not real `~/.kotoshu`): write a file with known misspellings,
135
+ run `kotoshu check`, assert exit code + output content for both
136
+ `text` and `json` formats.
137
+
138
+ **Phase C acceptance:** `kotoshu check README.adoc` exits 1 with
139
+ readable error output; `kotoshu check clean_file.txt` exits 0.
140
+
141
+ ## Release-level acceptance criteria
142
+
143
+ - Fresh `docker run --rm ruby:3.3` → `gem install kotoshu` →
144
+ `ruby -e "require 'kotoshu'; puts Kotoshu.correct?('hello')"`
145
+ prints `true` (auto-downloads en_US on first call)
146
+ - `Kotoshu.correct?("helo")` returns `false`
147
+ - `Kotoshu.suggest("helo").first.word` == `"hello"`
148
+ - `kotoshu check README.adoc` exits 1, reports errors with suggestions
149
+ - `kotoshu check clean.txt` exits 0
150
+ - en_US Hunspell fixtures: 100 % pass
151
+ - No system dictionary dependency (no `/usr/share/dict/words` read)
152
+ - `Kotoshu.check(text, offline: true)` raises clear error if not cached
153
+ - `ENV['KOTOSHU_OFFLINE']` == `"1"` respected throughout
154
+ - README accurately reflects 0.2 capabilities (no false claims)
155
+ - `bundle exec rspec` green; `bundle exec rubocop` clean
156
+
157
+ ## Dependencies
158
+
159
+ - **Blocked by:** nothing — 0.1.0 is shipped.
160
+ - **Blocks:** `04-language-modules` (needs `ResourceManager`),
161
+ `05-semantic-path` (needs `ResourceManager` for model resolution).
162
+
163
+ ## Out of scope for 0.2.0
164
+
165
+ Everything in the "deferred" table above. Additionally:
166
+ - The Hunspell MECE 4-layer refactor (`01` task 6) — patch, don't
167
+ rewrite, for 0.2.
168
+ - `DictCommand` / `CacheCommand` relocation to `commands/` (`02`
169
+ task 2) — cosmetic, defer.
170
+ - `model` subcommand (`02` task 3) — needs ONNX, defer to 0.4.
171
+ - `--from-stdin` (`02` task 4) — nice-to-have, defer to 0.3.
172
+
173
+ ## Status
174
+
175
+ Starting now. Phase A (`ResourceManager`) is the first task.