kotoshu 1.0.2-x86_64-linux

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (458) hide show
  1. checksums.yaml +7 -0
  2. data/.kotoshu-baseline.json +1 -0
  3. data/.pre-commit-hooks.yaml +18 -0
  4. data/.rspec +3 -0
  5. data/.rubocop.yml +351 -0
  6. data/.yardopts +8 -0
  7. data/CHANGELOG.md +813 -0
  8. data/CLAUDE.md +176 -0
  9. data/CODE_OF_CONDUCT.md +132 -0
  10. data/Cargo.lock +359 -0
  11. data/Cargo.toml +15 -0
  12. data/LICENSE +31 -0
  13. data/README.adoc +1292 -0
  14. data/Rakefile +150 -0
  15. data/SECURITY.md +93 -0
  16. data/TODO.impl/00-cut-0.1.md +121 -0
  17. data/TODO.impl/00-cut-0.2.md +175 -0
  18. data/TODO.impl/00-vision.md +281 -0
  19. data/TODO.impl/01-hunspell-correctness.md +111 -0
  20. data/TODO.impl/02-cli-unification.md +74 -0
  21. data/TODO.impl/03-dynamic-download.md +107 -0
  22. data/TODO.impl/04-language-modules.md +89 -0
  23. data/TODO.impl/05-semantic-path.md +83 -0
  24. data/TODO.impl/06-cjk-support.md +105 -0
  25. data/TODO.impl/07-rtl-support.md +71 -0
  26. data/TODO.impl/08-grammar-engine.md +106 -0
  27. data/TODO.impl/09-integrity-security.md +90 -0
  28. data/TODO.impl/10-testing-ci.md +98 -0
  29. data/TODO.impl/100-full-feature-batch3.md +28 -0
  30. data/TODO.impl/101-int4-tier-experiment.md +32 -0
  31. data/TODO.impl/102-lid-chain.md +28 -0
  32. data/TODO.impl/103-oov-bucket-tables.md +27 -0
  33. data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
  34. data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
  35. data/TODO.impl/106-native-lid-detection.md +36 -0
  36. data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
  37. data/TODO.impl/108-ko-ne-full-feature.md +37 -0
  38. data/TODO.impl/109-ten-audit.md +26 -0
  39. data/TODO.impl/11-release-v1.md +95 -0
  40. data/TODO.impl/110-sr-latn-and-polish.md +33 -0
  41. data/TODO.impl/111-embedding-bakeoff.md +33 -0
  42. data/TODO.impl/112-deployability-and-gating.md +26 -0
  43. data/TODO.impl/113-language-packs.md +26 -0
  44. data/TODO.impl/114-typo-biencoder-thread.md +33 -0
  45. data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
  46. data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
  47. data/TODO.impl/117-deterministic-frequency-tiers.md +110 -0
  48. data/TODO.impl/118-server-gem-floor.md +33 -0
  49. data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
  50. data/TODO.impl/120-setup-metadata-refresh.md +35 -0
  51. data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
  52. data/TODO.impl/122-pack-compression-kpk2.md +66 -0
  53. data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
  54. data/TODO.impl/124-owner-residue.md +19 -0
  55. data/TODO.impl/125-consumer-floor-guards.md +43 -0
  56. data/TODO.impl/126-tier-error-budgets.md +33 -0
  57. data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
  58. data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
  59. data/TODO.impl/129-pt-fr-corpora.md +23 -0
  60. data/TODO.impl/130-news-truth-0912.md +23 -0
  61. data/TODO.impl/131-hybrid-productization.md +42 -0
  62. data/TODO.impl/132-techniques-page-and-prose.md +27 -0
  63. data/TODO.impl/133-rust-first-distribution.md +68 -0
  64. data/TODO.impl/20-encapsulation-cleanup.md +47 -0
  65. data/TODO.impl/21-autoload-migration.md +64 -0
  66. data/TODO.impl/22-url-pin-single-source-of-truth.md +120 -0
  67. data/TODO.impl/23-fix-url-defaults.md +57 -0
  68. data/TODO.impl/24-cli-auto-setup-prompt.md +76 -0
  69. data/TODO.impl/25-soft-onnxruntime-require.md +75 -0
  70. data/TODO.impl/26-onnx-vocab-json-generation.md +96 -0
  71. data/TODO.impl/27-end-to-end-smoke-spec.md +113 -0
  72. data/TODO.impl/28-readme-quickstart-verification.md +68 -0
  73. data/TODO.impl/29-status-command.md +84 -0
  74. data/TODO.impl/30-language-auto-detection.md +74 -0
  75. data/TODO.impl/31-download-progress.md +58 -0
  76. data/TODO.impl/32-sarif-output-verification.md +72 -0
  77. data/TODO.impl/33-audit-log-rotation.md +54 -0
  78. data/TODO.impl/34-cache-eviction.md +59 -0
  79. data/TODO.impl/35-shell-completion.md +61 -0
  80. data/TODO.impl/36-cleanup-regressions-0.3.1.md +134 -0
  81. data/TODO.impl/37-hunspell-correctness-tier2.md +203 -0
  82. data/TODO.impl/38-onnx-semantic-gating.md +218 -0
  83. data/TODO.impl/39-tier3-and-beyond.md +232 -0
  84. data/TODO.impl/40-spec-drift-cleanup.md +134 -0
  85. data/TODO.impl/50-structure-aware-document-api.md +194 -0
  86. data/TODO.impl/51-grammar-engine-expansion.md +75 -0
  87. data/TODO.impl/52-multi-language-documents.md +95 -0
  88. data/TODO.impl/53-document-plugin-architecture.md +94 -0
  89. data/TODO.impl/54-cjk-support.md +117 -0
  90. data/TODO.impl/55-rtl-support.md +109 -0
  91. data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
  92. data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
  93. data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
  94. data/TODO.impl/59-yard-api-documentation.md +75 -0
  95. data/TODO.impl/60-lsp-server.md +174 -0
  96. data/TODO.impl/61-editor-ecosystem.md +167 -0
  97. data/TODO.impl/62-framework-integrations.md +159 -0
  98. data/TODO.impl/63-web-playground-wasm.md +186 -0
  99. data/TODO.impl/64-http-api-and-sdks.md +191 -0
  100. data/TODO.impl/65-universal-kotoshu.md +251 -0
  101. data/TODO.impl/66-kotoshu-core.md +169 -0
  102. data/TODO.impl/67-kotoshu-rs-and-access-libraries.md +137 -0
  103. data/TODO.impl/68-sota-adoption.md +152 -0
  104. data/TODO.impl/69-sota-typo-eval.md +74 -0
  105. data/TODO.impl/70-sota-confidence-cascade.md +59 -0
  106. data/TODO.impl/71-sota-nested-model.md +63 -0
  107. data/TODO.impl/72-model2vec-evaluation.md +49 -0
  108. data/TODO.impl/73-rl-reranker.md +42 -0
  109. data/TODO.impl/74-distilled-reranker.md +36 -0
  110. data/TODO.impl/75-llm-evaluator.md +35 -0
  111. data/TODO.impl/76-optional-llm-backend.md +37 -0
  112. data/TODO.impl/77-model-coverage-expansion.md +96 -0
  113. data/TODO.impl/78-in-browser-playground.md +69 -0
  114. data/TODO.impl/79-news-and-releases.md +60 -0
  115. data/TODO.impl/80-ecosystem-docs.md +56 -0
  116. data/TODO.impl/81-editor-reach.md +68 -0
  117. data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
  118. data/TODO.impl/83-models-batch-2.md +56 -0
  119. data/TODO.impl/84-language-modules-wave2.md +75 -0
  120. data/TODO.impl/85-wasm-model-api.md +48 -0
  121. data/TODO.impl/86-site-truth-wave2.md +45 -0
  122. data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
  123. data/TODO.impl/88-cli-directory-mode.md +54 -0
  124. data/TODO.impl/89-framework-integrations-exec.md +59 -0
  125. data/TODO.impl/90-gem-correctness-residues.md +44 -0
  126. data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
  127. data/TODO.impl/92-tier-mirror-cors.md +88 -0
  128. data/TODO.impl/93-python-wheels.md +32 -0
  129. data/TODO.impl/94-site-integrations-docs.md +35 -0
  130. data/TODO.impl/95-editor-action-polish.md +27 -0
  131. data/TODO.impl/96-server-semantic-models.md +50 -0
  132. data/TODO.impl/97-yard-execution.md +73 -0
  133. data/TODO.impl/98-campaign-cleanup.md +78 -0
  134. data/TODO.impl/99-site-truth-performance-docs.md +31 -0
  135. data/conformance/vectors.jsonl +2630 -0
  136. data/data/en/grammar/rules.yaml +851 -0
  137. data/docs/STABILITY-1.0-AUDIT.md +251 -0
  138. data/docs/STABILITY-1.0.md +137 -0
  139. data/docs/cache-architecture.md +502 -0
  140. data/docs/verification-evidence/correct3/pr_body.md +37 -0
  141. data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
  142. data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
  143. data/docs/verification-evidence/correct3/sv.aff +856 -0
  144. data/docs/verification-evidence/correct3/sv.dic +153717 -0
  145. data/examples/01_basic_word_checking.rb +38 -0
  146. data/examples/02_text_document_checking.rb +77 -0
  147. data/examples/03_dictionary_backends.rb +137 -0
  148. data/examples/04_trie_data_structure.rb +146 -0
  149. data/examples/05_suggestion_algorithms.rb +239 -0
  150. data/examples/06_configuration_advanced.rb +287 -0
  151. data/examples/07_multi_language_dictionaries.rb +278 -0
  152. data/exe/kotoshu +6 -0
  153. data/lib/kotoshu/algorithms/capitalization.rb +276 -0
  154. data/lib/kotoshu/algorithms/edit_distance.rb +128 -0
  155. data/lib/kotoshu/algorithms/lookup/affix_form.rb +132 -0
  156. data/lib/kotoshu/algorithms/lookup/compound_checks.rb +209 -0
  157. data/lib/kotoshu/algorithms/lookup/compound_form.rb +50 -0
  158. data/lib/kotoshu/algorithms/lookup.rb +841 -0
  159. data/lib/kotoshu/algorithms/ngram_suggest.rb +367 -0
  160. data/lib/kotoshu/algorithms/permutations.rb +283 -0
  161. data/lib/kotoshu/algorithms/phonet_suggest.rb +162 -0
  162. data/lib/kotoshu/algorithms/soundex.rb +92 -0
  163. data/lib/kotoshu/algorithms/suggest.rb +794 -0
  164. data/lib/kotoshu/algorithms.rb +23 -0
  165. data/lib/kotoshu/analyzers/semantic_analyzer.rb +405 -0
  166. data/lib/kotoshu/analyzers.rb +8 -0
  167. data/lib/kotoshu/baseline/application.rb +18 -0
  168. data/lib/kotoshu/baseline/entry.rb +33 -0
  169. data/lib/kotoshu/baseline/store.rb +175 -0
  170. data/lib/kotoshu/baseline.rb +20 -0
  171. data/lib/kotoshu/cache/base_cache.rb +778 -0
  172. data/lib/kotoshu/cache/cache.rb +91 -0
  173. data/lib/kotoshu/cache/eviction_policy.rb +66 -0
  174. data/lib/kotoshu/cache/frequency_cache.rb +283 -0
  175. data/lib/kotoshu/cache/language_cache.rb +511 -0
  176. data/lib/kotoshu/cache/lookup_cache.rb +166 -0
  177. data/lib/kotoshu/cache/model_cache.rb +1147 -0
  178. data/lib/kotoshu/cache/model_registry.rb +109 -0
  179. data/lib/kotoshu/cache/staged_languages.rb +45 -0
  180. data/lib/kotoshu/cache/suggestion_cache.rb +113 -0
  181. data/lib/kotoshu/cache.rb +49 -0
  182. data/lib/kotoshu/cli/auto_setup.rb +71 -0
  183. data/lib/kotoshu/cli/baseline_command.rb +119 -0
  184. data/lib/kotoshu/cli/batch_reporter.rb +321 -0
  185. data/lib/kotoshu/cli/cache_command.rb +282 -0
  186. data/lib/kotoshu/cli/completions_command.rb +210 -0
  187. data/lib/kotoshu/cli/dict_command.rb +64 -0
  188. data/lib/kotoshu/cli/directory_check.rb +146 -0
  189. data/lib/kotoshu/cli/directory_walker.rb +132 -0
  190. data/lib/kotoshu/cli/display_formatter.rb +429 -0
  191. data/lib/kotoshu/cli/errors.rb +36 -0
  192. data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
  193. data/lib/kotoshu/cli/interactive_reviewer.rb +314 -0
  194. data/lib/kotoshu/cli/language_resolver.rb +91 -0
  195. data/lib/kotoshu/cli/navigation_manager.rb +273 -0
  196. data/lib/kotoshu/cli/personal_command.rb +113 -0
  197. data/lib/kotoshu/cli/progress_reporter.rb +114 -0
  198. data/lib/kotoshu/cli/status_report.rb +130 -0
  199. data/lib/kotoshu/cli.rb +880 -0
  200. data/lib/kotoshu/components/passthrough_spell_checker.rb +70 -0
  201. data/lib/kotoshu/components/pos_tagger.rb +98 -0
  202. data/lib/kotoshu/components/spell_checker.rb +73 -0
  203. data/lib/kotoshu/components/synthesizer.rb +60 -0
  204. data/lib/kotoshu/components/tokenizer.rb +58 -0
  205. data/lib/kotoshu/components/whitespace_tokenizer.rb +94 -0
  206. data/lib/kotoshu/components.rb +13 -0
  207. data/lib/kotoshu/configuration/builder.rb +207 -0
  208. data/lib/kotoshu/configuration/resolver.rb +124 -0
  209. data/lib/kotoshu/configuration.rb +829 -0
  210. data/lib/kotoshu/conformance_exporter.rb +256 -0
  211. data/lib/kotoshu/conformance_runner.rb +176 -0
  212. data/lib/kotoshu/core/exceptions.rb +186 -0
  213. data/lib/kotoshu/core/indexed_dictionary.rb +293 -0
  214. data/lib/kotoshu/core/models/affix_rule.rb +262 -0
  215. data/lib/kotoshu/core/models/result/document_result.rb +132 -0
  216. data/lib/kotoshu/core/models/result/word_result.rb +131 -0
  217. data/lib/kotoshu/core/models/word.rb +131 -0
  218. data/lib/kotoshu/core/trie/builder.rb +119 -0
  219. data/lib/kotoshu/core/trie/node.rb +94 -0
  220. data/lib/kotoshu/core/trie/trie.rb +249 -0
  221. data/lib/kotoshu/core.rb +28 -0
  222. data/lib/kotoshu/data/common_words/de.yml +1800 -0
  223. data/lib/kotoshu/data/common_words/en.yml +1215 -0
  224. data/lib/kotoshu/data/common_words/es.yml +750 -0
  225. data/lib/kotoshu/data/common_words/fr.yml +1015 -0
  226. data/lib/kotoshu/data/common_words/pt.yml +870 -0
  227. data/lib/kotoshu/data/common_words/ru.yml +484 -0
  228. data/lib/kotoshu/data/common_words_loader.rb +141 -0
  229. data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
  230. data/lib/kotoshu/data/frozen_kelly.rb +38 -0
  231. data/lib/kotoshu/data.rb +8 -0
  232. data/lib/kotoshu/data_structures/bloom_filter.rb +188 -0
  233. data/lib/kotoshu/data_structures.rb +8 -0
  234. data/lib/kotoshu/debug_logger.rb +146 -0
  235. data/lib/kotoshu/debug_mode.rb +136 -0
  236. data/lib/kotoshu/defaults.rb +84 -0
  237. data/lib/kotoshu/dictionaries/catalog.rb +814 -0
  238. data/lib/kotoshu/dictionaries.rb +8 -0
  239. data/lib/kotoshu/dictionary/base.rb +324 -0
  240. data/lib/kotoshu/dictionary/cspell.rb +243 -0
  241. data/lib/kotoshu/dictionary/custom.rb +225 -0
  242. data/lib/kotoshu/dictionary/hunspell.rb +520 -0
  243. data/lib/kotoshu/dictionary/plain_text.rb +302 -0
  244. data/lib/kotoshu/dictionary/repository.rb +246 -0
  245. data/lib/kotoshu/dictionary/unified.rb +236 -0
  246. data/lib/kotoshu/dictionary/unix_words.rb +218 -0
  247. data/lib/kotoshu/dictionary.rb +15 -0
  248. data/lib/kotoshu/documents/document.rb +120 -0
  249. data/lib/kotoshu/documents/plain_text_document.rb +73 -0
  250. data/lib/kotoshu/documents/source_position.rb +45 -0
  251. data/lib/kotoshu/documents/source_range.rb +76 -0
  252. data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
  253. data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
  254. data/lib/kotoshu/documents/suppressions.rb +44 -0
  255. data/lib/kotoshu/documents/text_node.rb +49 -0
  256. data/lib/kotoshu/documents.rb +178 -0
  257. data/lib/kotoshu/embeddings/embedding_pipeline.rb +238 -0
  258. data/lib/kotoshu/embeddings/lru_cache.rb +244 -0
  259. data/lib/kotoshu/embeddings/onnx_runtime_model.rb +411 -0
  260. data/lib/kotoshu/embeddings/protocol.rb +96 -0
  261. data/lib/kotoshu/embeddings/protocols.rb +23 -0
  262. data/lib/kotoshu/embeddings/registry.rb +184 -0
  263. data/lib/kotoshu/embeddings/search.rb +204 -0
  264. data/lib/kotoshu/embeddings/similarity_engine.rb +278 -0
  265. data/lib/kotoshu/embeddings/similarity_search.rb +328 -0
  266. data/lib/kotoshu/embeddings/vocabulary.rb +306 -0
  267. data/lib/kotoshu/embeddings.rb +87 -0
  268. data/lib/kotoshu/fluent_checker.rb +94 -0
  269. data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +48 -0
  270. data/lib/kotoshu/grammar/pattern_matchers/double_negative_matcher.rb +179 -0
  271. data/lib/kotoshu/grammar/pattern_matchers/phrase_matcher.rb +86 -0
  272. data/lib/kotoshu/grammar/pattern_matchers/possessive_context_matcher.rb +75 -0
  273. data/lib/kotoshu/grammar/pattern_matchers/possessive_contraction_matcher.rb +87 -0
  274. data/lib/kotoshu/grammar/pattern_matchers/sentence_start_matcher.rb +84 -0
  275. data/lib/kotoshu/grammar/pattern_matchers/vowel_sound_matcher.rb +81 -0
  276. data/lib/kotoshu/grammar/pattern_matchers/word_list_matcher.rb +66 -0
  277. data/lib/kotoshu/grammar/rule.rb +99 -0
  278. data/lib/kotoshu/grammar/rule_engine.rb +96 -0
  279. data/lib/kotoshu/grammar/rule_loader.rb +30 -0
  280. data/lib/kotoshu/grammar.rb +24 -0
  281. data/lib/kotoshu/integrity/audit_log.rb +167 -0
  282. data/lib/kotoshu/integrity/manifest.rb +116 -0
  283. data/lib/kotoshu/integrity/net_http.rb +51 -0
  284. data/lib/kotoshu/integrity/rotation_policy.rb +88 -0
  285. data/lib/kotoshu/integrity.rb +26 -0
  286. data/lib/kotoshu/jekyll.rb +101 -0
  287. data/lib/kotoshu/keyboard/layout.rb +116 -0
  288. data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
  289. data/lib/kotoshu/keyboard/layouts/azerty.rb +55 -0
  290. data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
  291. data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
  292. data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
  293. data/lib/kotoshu/keyboard/layouts/dvorak.rb +54 -0
  294. data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
  295. data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
  296. data/lib/kotoshu/keyboard/layouts/hrsl.rb +94 -0
  297. data/lib/kotoshu/keyboard/layouts/jcuken.rb +57 -0
  298. data/lib/kotoshu/keyboard/layouts/latin.rb +266 -0
  299. data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
  300. data/lib/kotoshu/keyboard/layouts/qwerty.rb +52 -0
  301. data/lib/kotoshu/keyboard/layouts/qwertz.rb +55 -0
  302. data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
  303. data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
  304. data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
  305. data/lib/kotoshu/keyboard/registry.rb +201 -0
  306. data/lib/kotoshu/keyboard.rb +96 -0
  307. data/lib/kotoshu/kotoshu_native.so +0 -0
  308. data/lib/kotoshu/language/detector.rb +242 -0
  309. data/lib/kotoshu/language/identifier.rb +376 -0
  310. data/lib/kotoshu/language/languages/base.rb +271 -0
  311. data/lib/kotoshu/language/lid_detector.rb +188 -0
  312. data/lib/kotoshu/language/normalizer/arabic.rb +113 -0
  313. data/lib/kotoshu/language/normalizer/base.rb +133 -0
  314. data/lib/kotoshu/language/normalizer/greek.rb +63 -0
  315. data/lib/kotoshu/language/normalizer/hebrew.rb +72 -0
  316. data/lib/kotoshu/language/normalizer/persian.rb +36 -0
  317. data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
  318. data/lib/kotoshu/language/registry.rb +209 -0
  319. data/lib/kotoshu/language/resources/ar/common_words.txt +6753 -0
  320. data/lib/kotoshu/language/resources/ar/confusion_sets.txt +11 -0
  321. data/lib/kotoshu/language/resources/de/common_words.txt +10003 -0
  322. data/lib/kotoshu/language/resources/de/confusion_sets.txt +246 -0
  323. data/lib/kotoshu/language/resources/en/common_words.txt +9979 -0
  324. data/lib/kotoshu/language/resources/en/confusion_sets.txt +871 -0
  325. data/lib/kotoshu/language/resources/es/common_words.txt +9992 -0
  326. data/lib/kotoshu/language/resources/es/confusion_sets.txt +17 -0
  327. data/lib/kotoshu/language/resources/fr/common_words.txt +9993 -0
  328. data/lib/kotoshu/language/resources/fr/confusion_sets.txt +76 -0
  329. data/lib/kotoshu/language/resources/pt/common_words.txt +9977 -0
  330. data/lib/kotoshu/language/resources/pt/confusion_sets.txt +18 -0
  331. data/lib/kotoshu/language/resources/ru/common_words.txt +9951 -0
  332. data/lib/kotoshu/language/resources/ru/confusion_sets.txt +5 -0
  333. data/lib/kotoshu/language/script.rb +128 -0
  334. data/lib/kotoshu/language/segmenter.rb +122 -0
  335. data/lib/kotoshu/language/suika.rb +38 -0
  336. data/lib/kotoshu/language/tokenizer/base.rb +187 -0
  337. data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
  338. data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
  339. data/lib/kotoshu/language/tokenizer/french_tokenizer.rb +172 -0
  340. data/lib/kotoshu/language/tokenizer/german_tokenizer.rb +41 -0
  341. data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
  342. data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
  343. data/lib/kotoshu/language/tokenizer/japanese_tokenizer.rb +55 -0
  344. data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +153 -0
  345. data/lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb +160 -0
  346. data/lib/kotoshu/language/tokenizer/russian_tokenizer.rb +93 -0
  347. data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
  348. data/lib/kotoshu/language/tokenizer/spanish_tokenizer.rb +122 -0
  349. data/lib/kotoshu/language.rb +127 -0
  350. data/lib/kotoshu/languages/ar/language.rb +98 -0
  351. data/lib/kotoshu/languages/bg/language.rb +37 -0
  352. data/lib/kotoshu/languages/ca/language.rb +22 -0
  353. data/lib/kotoshu/languages/cs/language.rb +22 -0
  354. data/lib/kotoshu/languages/da/language.rb +22 -0
  355. data/lib/kotoshu/languages/de/language.rb +554 -0
  356. data/lib/kotoshu/languages/el/language.rb +50 -0
  357. data/lib/kotoshu/languages/en/language.rb +470 -0
  358. data/lib/kotoshu/languages/es/language.rb +476 -0
  359. data/lib/kotoshu/languages/et/language.rb +24 -0
  360. data/lib/kotoshu/languages/fa/language.rb +91 -0
  361. data/lib/kotoshu/languages/fr/language.rb +505 -0
  362. data/lib/kotoshu/languages/he/language.rb +98 -0
  363. data/lib/kotoshu/languages/hr/language.rb +24 -0
  364. data/lib/kotoshu/languages/hu/language.rb +22 -0
  365. data/lib/kotoshu/languages/id/language.rb +25 -0
  366. data/lib/kotoshu/languages/it/language.rb +22 -0
  367. data/lib/kotoshu/languages/ja/language.rb +476 -0
  368. data/lib/kotoshu/languages/ko/language.rb +56 -0
  369. data/lib/kotoshu/languages/latin_base.rb +53 -0
  370. data/lib/kotoshu/languages/lt/language.rb +24 -0
  371. data/lib/kotoshu/languages/lv/language.rb +24 -0
  372. data/lib/kotoshu/languages/nb/language.rb +25 -0
  373. data/lib/kotoshu/languages/ne/language.rb +60 -0
  374. data/lib/kotoshu/languages/nl/language.rb +22 -0
  375. data/lib/kotoshu/languages/nn/language.rb +26 -0
  376. data/lib/kotoshu/languages/pl/language.rb +22 -0
  377. data/lib/kotoshu/languages/pt/language.rb +439 -0
  378. data/lib/kotoshu/languages/ro/language.rb +22 -0
  379. data/lib/kotoshu/languages/ru/language.rb +420 -0
  380. data/lib/kotoshu/languages/sk/language.rb +25 -0
  381. data/lib/kotoshu/languages/sl/language.rb +25 -0
  382. data/lib/kotoshu/languages/sr/language.rb +39 -0
  383. data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
  384. data/lib/kotoshu/languages/sv/language.rb +22 -0
  385. data/lib/kotoshu/languages/tr/language.rb +36 -0
  386. data/lib/kotoshu/languages/uk/language.rb +38 -0
  387. data/lib/kotoshu/languages/vi/language.rb +22 -0
  388. data/lib/kotoshu/languages.rb +64 -0
  389. data/lib/kotoshu/metrics_collector.rb +238 -0
  390. data/lib/kotoshu/metrics_module.rb +121 -0
  391. data/lib/kotoshu/models/context.rb +119 -0
  392. data/lib/kotoshu/models/embedding_model.rb +181 -0
  393. data/lib/kotoshu/models/fasttext_model.rb +212 -0
  394. data/lib/kotoshu/models/nearest_neighbor.rb +96 -0
  395. data/lib/kotoshu/models/onnx_model.rb +330 -0
  396. data/lib/kotoshu/models/semantic_error.rb +204 -0
  397. data/lib/kotoshu/models/suggestion.rb +106 -0
  398. data/lib/kotoshu/models/word_embedding.rb +97 -0
  399. data/lib/kotoshu/models.rb +27 -0
  400. data/lib/kotoshu/multi_language_checker.rb +127 -0
  401. data/lib/kotoshu/native.rb +83 -0
  402. data/lib/kotoshu/native_backend.rb +131 -0
  403. data/lib/kotoshu/paths.rb +79 -0
  404. data/lib/kotoshu/personal_dictionary.rb +105 -0
  405. data/lib/kotoshu/plugins/plugin.rb +61 -0
  406. data/lib/kotoshu/plugins/registry.rb +118 -0
  407. data/lib/kotoshu/plugins.rb +9 -0
  408. data/lib/kotoshu/project_config.rb +76 -0
  409. data/lib/kotoshu/readers/aff_data.rb +530 -0
  410. data/lib/kotoshu/readers/aff_reader.rb +393 -0
  411. data/lib/kotoshu/readers/condition_checker.rb +125 -0
  412. data/lib/kotoshu/readers/dic_reader.rb +208 -0
  413. data/lib/kotoshu/readers/file_reader.rb +377 -0
  414. data/lib/kotoshu/readers/lookup_builder.rb +584 -0
  415. data/lib/kotoshu/readers/readers.rb +17 -0
  416. data/lib/kotoshu/readers.rb +25 -0
  417. data/lib/kotoshu/resource_bundle.rb +55 -0
  418. data/lib/kotoshu/resource_manager.rb +600 -0
  419. data/lib/kotoshu/results/result.rb +165 -0
  420. data/lib/kotoshu/results.rb +8 -0
  421. data/lib/kotoshu/rspec.rb +163 -0
  422. data/lib/kotoshu/scripts/fasttext_to_onnx.py +275 -0
  423. data/lib/kotoshu/source_registry.rb +104 -0
  424. data/lib/kotoshu/spellchecker/parallel_checker.rb +90 -0
  425. data/lib/kotoshu/spellchecker.rb +406 -0
  426. data/lib/kotoshu/string_metrics.rb +153 -0
  427. data/lib/kotoshu/suggestions/context.rb +43 -0
  428. data/lib/kotoshu/suggestions/frequency_provider.rb +80 -0
  429. data/lib/kotoshu/suggestions/generator.rb +169 -0
  430. data/lib/kotoshu/suggestions/pipeline.rb +135 -0
  431. data/lib/kotoshu/suggestions/semantic_cascade.rb +73 -0
  432. data/lib/kotoshu/suggestions/strategies/base_strategy.rb +315 -0
  433. data/lib/kotoshu/suggestions/strategies/composite_strategy.rb +197 -0
  434. data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +631 -0
  435. data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +207 -0
  436. data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +165 -0
  437. data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +284 -0
  438. data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +317 -0
  439. data/lib/kotoshu/suggestions/strategies/symspell_strategy.rb +270 -0
  440. data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
  441. data/lib/kotoshu/suggestions/suggestion.rb +154 -0
  442. data/lib/kotoshu/suggestions/suggestion_set.rb +270 -0
  443. data/lib/kotoshu/suggestions/sweep_index.rb +137 -0
  444. data/lib/kotoshu/suggestions.rb +29 -0
  445. data/lib/kotoshu/tasks/check_task.rb +104 -0
  446. data/lib/kotoshu/tasks.rb +11 -0
  447. data/lib/kotoshu/validators/spelling_validator.rb +101 -0
  448. data/lib/kotoshu/validators.rb +9 -0
  449. data/lib/kotoshu/version.rb +5 -0
  450. data/lib/kotoshu.rb +538 -0
  451. data/rakelib/compile.rake +10 -0
  452. data/script/validate_all_dictionaries.rb +444 -0
  453. data/scripts/build_native_gem.rb +45 -0
  454. data/scripts/extract_eval_grids.rb +37 -0
  455. data/scripts/generate_frozen_tiers.rb +44 -0
  456. data/sig/kotoshu.rbs +435 -0
  457. data/test_oop.rb +79 -0
  458. metadata +546 -0
@@ -0,0 +1,110 @@
1
+ # Plan 117 — Deterministic frequency tiers: stop the silent YAML fallback
2
+
3
+ Status: executed (gem PR #178, 2026-09-10)
4
+
5
+ Acceptance met: with the cache backdated past the TTL the compare is
6
+ 2630/0/0 and suggest("ba") on the compoundrule fixtures returns the
7
+ frozen expectations exactly. conformance.yml gates the compare on
8
+ every push and PR; its first green run landed on the PR itself.
9
+ Suite 4007/0, rubocop clean. The checked-in-fixture variant of the
10
+ hermetic runner (design item 2) was not needed — the CI job seeds the
11
+ pinned cache instead; revisit only if the job proves flaky.
12
+
13
+ Depends on: plan 82 (conformance runner), the 2,630-vector contract (plan 66+)
14
+ Blocks: nothing — audit checklist item 7 is satisfied and CI-gated
15
+
16
+ ## What happened (forensics, 2026-09-10)
17
+
18
+ Preparing the 1.0 train (checklist item 7) ran `rake
19
+ kotoshu:conformance:compare` on main:
20
+
21
+ compare: 2630 vectors -- ruby failures: 16, native failures: 0, divergences: 16
22
+
23
+ All 16 failures are `suggest` vectors on the compoundrule{,2,3} fixtures:
24
+ same candidate words, same distances, different `confidence` values
25
+ (e.g. expected `b` = 0.952380952…, ruby = 0.8333333…).
26
+
27
+ Ruled out, with evidence:
28
+
29
+ - **Not plan 116 / not recent main.** v0.11.0 shows the identical 16
30
+ failures; the suggestions/, dictionary/, core/ trees have zero commits
31
+ between v0.9.2 and main.
32
+ - **Not the machine cache contents.** The downloaded Kelly `en.json`
33
+ sha256 `97535823f41a3f0…` matches the pin in kotoshu-rs
34
+ `suggest/frequency_data.rs` byte-for-byte; `load_cached` returns the
35
+ same 47/185/907 tier sizes rust embeds.
36
+ - **Not an engine disagreement.** With a live (unexpired) cache:
37
+
38
+ compare: 2630 vectors -- ruby failures: 0, native failures: 0,
39
+ backend divergences: 0
40
+
41
+ ### The actual mechanism
42
+
43
+ `Suggestions::FrequencyProvider` falls back
44
+ `FrequencyCache` → local YAML (`lib/kotoshu/data/common_words/en.yml`)
45
+ → empty tiers. `BaseCache#available?` returns false when the metadata's
46
+ `cached_at` is older than the 7-day TTL. **The two datasets differ**:
47
+ the local YAML's top_50 contains `a` (bonus 200); Kelly's top_1000 is
48
+ where `a` lives (bonus 50). The bonus shifts one candidate's enhanced
49
+ score, the min–max normalization re-scales every confidence on the
50
+ tiny fixtures, and 16 vectors "fail".
51
+
52
+ So any machine whose frequency cache is absent or >7 days old silently
53
+ runs ruby on the YAML dataset — different suggestion ranking from the
54
+ frozen contract and from the Rust engine. This is a **user-facing
55
+ determinism bug**, not conformance bookkeeping: suggestion output
56
+ depends on cache age.
57
+
58
+ Two aggravators found on the way:
59
+
60
+ - `Kotoshu.setup(:en, want: [:frequency])` reported
61
+ `frequency: :downloaded` but left `cached_at` at 2026-09-02 — either
62
+ the re-download was skipped on checksum match (fine) and the
63
+ timestamp was not refreshed (bug: guarantees later expiry), or the
64
+ report is wrong. Verify and fix.
65
+ - Nothing CI-gates `conformance:compare` on the gem side (the rs repo
66
+ gates its half only), which is why a two-week-old dataset switch was
67
+ invisible.
68
+
69
+ ## Fix design
70
+
71
+ 1. **Expired ≠ unusable.** In `FrequencyProvider#load`, when
72
+ `available?` is false but the on-disk cached data exists, load it
73
+ anyway (`load_cached` does no network). TTL governs *refresh*
74
+ (setup/`get` paths), never a silent switch to a different dataset.
75
+ The YAML fallback then only serves machines with NO frequency data
76
+ at all.
77
+ 2. **Hermetic conformance.** The runner must not depend on machine
78
+ cache state: check in the frozen Kelly tiers (the three sets +
79
+ sha256, mirroring rs `frequency_data.rs`) as a conformance fixture
80
+ and construct the engines with that provider data pinned. Then
81
+ `conformance:compare` is deterministic on any machine, cache or no
82
+ cache, and cross-verifies the fixture against the rs tables.
83
+ 3. **CI gate.** Add an Ubuntu job: `rake compile && rake
84
+ kotoshu:conformance:compare` on every push/PR. This invariant is
85
+ the product's core correctness claim; it must not rely on someone
86
+ remembering to run it.
87
+ 4. **Setup timestamp.** When setup skips a download because the
88
+ checksum matches, refresh `cached_at` (and report `:cached`).
89
+
90
+ ## Acceptance
91
+
92
+ - `XDG_CACHE_HOME=<empty> rake kotoshu:conformance:compare` → 0/0/0
93
+ (today it is 16/0/16; with a warm cache it is 0/0/0).
94
+ - Same on a machine with an expired cache (mock by backdating
95
+ `cached_at`).
96
+ - New CI job green on main; suite unchanged otherwise; suggestion
97
+ output for cache-cold machines now byte-identical to cache-warm
98
+ ones on the fixture corpus.
99
+ - Explicitly out of scope: regenerating any vector. The 2,630 are
100
+ correct; the data path was not.
101
+
102
+ ## Surfaces
103
+
104
+ - `lib/kotoshu/suggestions/frequency_provider.rb`
105
+ - `lib/kotoshu/cache/base_cache.rb` (a load-explicit accessor if
106
+ needed) and the setup refresh path
107
+ - `lib/kotoshu/conformance_runner.rb` (pinned provider)
108
+ - `conformance/fixtures/` (frozen tier fixture), `.github/workflows/`
109
+ (compare job)
110
+ - specs for each behavior above
@@ -0,0 +1,33 @@
1
+ # Plan 118 — kotoshu-server 1.0 must ride kotoshu 1.0
2
+
3
+ Status: executed (kotoshu-server PR #8 + release 1.0.1, 2026-09-11)
4
+
5
+ gemspec + Gemfile + lock floor ~> 1.0; guard spec added; server 1.0.1
6
+ live on RubyGems (verified). The suite ran green on kotoshu 1.0.0
7
+ (25/0) - the 0.6-era engine had been serving this repo's own CI.
8
+ Priority: P0 (a shipped 1.0.0 release cannot see the 1.0 gem)
9
+
10
+ ## Problem
11
+
12
+ `kotoshu-server.gemspec` declares `spec.add_runtime_dependency "kotoshu",
13
+ "~> 0.6"`. Ruby's pessimistic operator makes that `>= 0.6, < 1.0` — so
14
+ the server 1.0.0 released in the 1.0 train resolves **kotoshu 0.11.1**,
15
+ not 1.0.0. Every fresh `gem install kotoshu-server` today deploys the
16
+ pre-freeze engine under a 1.0 label. The floor was never revisited as
17
+ the gem moved 0.6 → 1.0.
18
+
19
+ ## Fix
20
+
21
+ - `spec.add_runtime_dependency "kotoshu", "~> 1.0"` — the server's
22
+ public surface consumes stable gem API only (two-stage setup, check
23
+ facade, semantic cascade), all frozen at 1.0.
24
+ - Release kotoshu-server 1.0.1 carrying the floor bump.
25
+ - Guard: a spec that parses the gemspec and asserts the runtime floor
26
+ does not exclude the current kotoshu major (the class of bug is
27
+ silent by construction).
28
+
29
+ ## Acceptance
30
+
31
+ - `gem dependency kotoshu-server -v 1.0.1` shows kotoshu ~> 1.0.
32
+ - A fresh resolve (`gem install kotoshu-server --version 1.0.1` in a
33
+ scratch GEM_HOME) picks kotoshu 1.0.x.
@@ -0,0 +1,56 @@
1
+ # Plan 119 — Embed the frozen Kelly tiers in the gem
2
+
3
+ Status: executed (gem PR #181, 2026-09-11)
4
+
5
+ Acceptance exceeded: XDG_CACHE_HOME=<empty> compare is 2630/0/0 -
6
+ determinism by construction, no CI seeding needed. Embedded tiers are
7
+ Set-equal to the checksummed cache tiers; data ships as JSON generated
8
+ by scripts/generate_frozen_tiers.rb (provenance + sha inside the
9
+ file); other languages unchanged. Full suite 4010/0, rubocop clean.
10
+ Depends on: plan 117 (the dataset-divergence forensics)
11
+
12
+ ## Problem
13
+
14
+ Plan 117 made suggestion ranking deterministic for machines with
15
+ frequency data, but a cache-cold machine still falls back to
16
+ `lib/kotoshu/data/common_words/en.yml` — a *differently curated*
17
+ dataset (top_50 holds `a`; Kelly holds `a` in top_1000). The Rust
18
+ engine embeds the frozen Kelly tables (`kotoshu-rs
19
+ kotoshu/src/suggest/frequency_data.rs`, en.json sha256 97535823f41aa3f0…)
20
+ as its *only* source; the Ruby engine downloads them. Two engines, two
21
+ data paths, one contract — the fallback path is still
22
+ contract-divergent by construction.
23
+
24
+ ## Design (single source of truth, model-driven)
25
+
26
+ - Generate `lib/kotoshu/data/frozen_kelly/en.rb` FROM the rs tables
27
+ (script: `scripts/generate_frozen_tiers.rb`, documented provenance:
28
+ the rs file path + the en.json sha). Content: three sorted word
29
+ arrays, sizes 47/185/907, byte-identical membership to the rs
30
+ `TOP_50`/`TOP_200`/`TOP_1000`.
31
+ - `Suggestions::FrozenTiers.tiers_for(language_code)` — returns the
32
+ Set-shaped tiers for `en`, nil otherwise. Pure, no IO.
33
+ - `FrequencyProvider#load` fallback chain becomes:
34
+ cache (any present data, plan 117) → **frozen embedded (en)** →
35
+ local YAML → empty. For `en` the YAML is thereby unreachable; for
36
+ every other language nothing changes (rs embeds en only — MECE with
37
+ the Rust side).
38
+ - Cache-cold machines now produce contract-exact ranking for `en`
39
+ with zero network, zero setup — determinism by construction instead
40
+ of by CI seeding.
41
+
42
+ ## Acceptance
43
+
44
+ - `XDG_CACHE_HOME=<empty>` conformance compare: 2630/0/0 (today the
45
+ empty-cache run fails 16 — the CI seed papers over it).
46
+ - Parity spec: embedded tiers == the tiers `FrequencyCache` returns
47
+ for a checksummed en download (structural: sizes + sampled words +
48
+ `a` in top_1000 / `the` in top_50).
49
+ - Full suite green; no behavior change for cache-present machines.
50
+
51
+ ## Non-goals
52
+
53
+ - No YAML deletion (other languages still use it; `en.yml` stays for
54
+ now as a documented dead fallback — removal is a 1.x deprecation
55
+ decision).
56
+ - No registry or format changes.
@@ -0,0 +1,35 @@
1
+ # Plan 120 — setup refreshes cached_at when a download is skipped
2
+
3
+ Status: resolved as obsolete (2026-09-11, code inspection during execution)
4
+
5
+ `setup_frequency_remote` (resource_manager.rb) has no checksum-skip
6
+ path: `return :cached if cache.available? && !force` — expiry makes
7
+ `available?` false — `else cache.get` → `download` → `download_resource`
8
+ ALWAYS writes fresh metadata (cached_at = now). The observed
9
+ "downloaded but stale timestamp" was plan 117's writer/reader path
10
+ split in full; no second mechanism exists. Verified by walkthrough of
11
+ get/setup_frequency_remote/download_resource; no code change needed.
12
+
13
+ ## Problem
14
+
15
+ `Kotoshu.setup(:en, want: [:frequency])` can skip the download when
16
+ the checksum of the fetched bytes matches the cached file, but the
17
+ skip path historically left `cached_at` untouched (observed 2026-09-10:
18
+ setup reported `frequency: :downloaded` while the reader-side metadata
19
+ still said 2026-09-02). An unrefreshed timestamp guarantees the next
20
+ TTL check fails and the resource is re-fetched for nothing — and any
21
+ reader still consulting TTL sees stale.
22
+
23
+ ## Fix
24
+
25
+ - On a checksum-matching skip: rewrite the metadata with a fresh
26
+ `cached_at` (same checksum, same url) and report `:cached`, not
27
+ `:downloaded`. The report must describe what actually happened.
28
+ - Spec: backdate metadata, run setup with a stubbed fetcher (no
29
+ network), assert `cached_at` moved and the result says `:cached`.
30
+
31
+ ## Acceptance
32
+
33
+ - Second consecutive `setup` run performs no re-download (checksum
34
+ match) yet leaves a TTL-valid metadata file.
35
+ - Honest SetupResult reporting.
@@ -0,0 +1,33 @@
1
+ # Plan 121 — Jekyll generator skips the sweep for baseline-covered words
2
+
3
+ Status: executed (gem PR #181, 2026-09-11)
4
+
5
+ Third call site wired: the generator builds the per-file budget from
6
+ its own baseline store and threads suggestions_filter through
7
+ Kotoshu.check. Generator spec proves a beyond-budget occurrence
8
+ surfaces with suggestions while covered words never pay for the
9
+ sweep.
10
+ Depends on: plan 116 (the suggestions_filter mechanism)
11
+
12
+ ## Problem
13
+
14
+ The rake task and the CLI thread `suggestions_filter:` through their
15
+ check callables; the Jekyll generator (plan 89) has its own baseline
16
+ store (`lib/kotoshu/jekyll.rb`) but still calls the checker without
17
+ the filter — Jekyll builds pay the full suggestion sweep for entries
18
+ their baseline then suppresses. Same repository, same mechanism,
19
+ three call sites, two wired.
20
+
21
+ ## Fix
22
+
23
+ - Build the per-file budget filter via `store.suggestions_filter_for`
24
+ exactly as `Cli::DirectoryCheck` does, pass it through the check
25
+ call. The generator's baseline is a `Baseline::Store` already — the
26
+ filter comes from the same object (DRY by construction).
27
+ - Spec: a Jekyll site fixture with a baseline; assert suppressed
28
+ entries carry no suggestions and a new error does.
29
+
30
+ ## Acceptance
31
+
32
+ - Generator output unchanged; sweep cost on baselined sites drops as
33
+ it did for the rake task (plan 116 evidence).
@@ -0,0 +1,66 @@
1
+ # Plan 122 — KPK2: compressed pack sections
2
+
3
+ Status: executed as REJECTED on the pre-declared gate (2026-09-12 measurement)
4
+
5
+ zstd-19 per-section, en-1.5.0.bin (15,173,525 bytes):
6
+
7
+ aff 3,086 -> 908 (3.40x)
8
+ dic 551,762 -> 169,700 (3.25x)
9
+ model 3,040,752 -> 2,769,252 (1.10x)
10
+ vocab 202,597 -> 59,342 (3.41x)
11
+ buckets 11,375,135 -> 9,854,872 (1.15x)
12
+ total -> 12,854,074 (84.7% of KPK1)
13
+
14
+ The acceptance gate was <= 60%. The pack is 75% int8 bucket table and
15
+ 20% int8 ONNX weights — nearly incompressible; the sections that win
16
+ 3.4x are 5% of the bytes. A 15% wire saving does not pay for a format
17
+ generation, a decompressor on the wasm surface, and a registry
18
+ re-cut. Revisit only if buckets shrink or sub-int8 quantization lands
19
+ (the thing that would actually move the 84.7%).
20
+ Depends on: plan 113 (KPK1), the 2026-09-10 model-efficiency research
21
+
22
+ ## Problem
23
+
24
+ KPK1 packs ship raw sections: en-1.5.0.bin is 15.2 MB over the wire
25
+ (aff+dic text ~1.2 MB, int8 ONNX ~12 MB, vocab JSON, buckets). The
26
+ text/JSON sections compress ~5-10x; the int8 weights ~1.2-1.5x. The
27
+ playground's first-load cost is dominated by bytes that decompress in
28
+ microseconds — costs are being paid in the expensive dimension
29
+ (bandwidth) instead of the cheap one (CPU). Same pattern as
30
+ DeepSeek-V4.1-Flash storing KV in FP4 and dequantizing before
31
+ attention: compress storage, not compute.
32
+
33
+ ## Design
34
+
35
+ - KPK2 container: same section framing as KPK1 (magic+count, framed
36
+ sections, sha256 footers) with (a) a new magic version byte, (b)
37
+ each section optionally zstd-frame-compressed, flagged by a new tag
38
+ bit; uncompressed sections remain valid — readers negotiate on the
39
+ magic, never on heuristics.
40
+ - Engine side (`kotoshu-rs`): `ruzstd` (pure-Rust decoder, no C
41
+ dependency — wasm-safe). Decompress per section before sha verify?
42
+ **No**: keep the footer sha over the COMPRESSED bytes so corruption
43
+ detection happens before decompression; the KPK1 golden-sha parity
44
+ test gains KPK2 twins.
45
+ - Builder side (`models-fasttext-onnx`): pack builder emits KPK2 with
46
+ zstd level 19 for text/JSON sections, raw-or-zstd for int8 weights
47
+ (measure; ship whichever wins per section).
48
+ - Registry: `kotoshu://packs/{lang}` entries gain the new artifacts;
49
+ bump registry to v1.7.0 per plan 05 policy; mirrors must serve both
50
+ generations during the rollout window.
51
+ - Consumers: wasm `loadPack` (versioned by magic), worker pack mode,
52
+ playground pin bump, site caching-docs truth.
53
+
54
+ ## Acceptance
55
+
56
+ - en pack wire size measurably smaller (target: ≤ 60% of KPK1;
57
+ report actual), loadPack wall time within +10% of KPK1 (143 ms
58
+ baseline on M-series).
59
+ - Conformance: KPK1 packs still load unchanged (back-compat gate).
60
+ - Registry v1.7.0 released with both generations mirrored.
61
+
62
+ ## Execution order (next session)
63
+
64
+ models builder → rs engine + wasm 1.1.0 → registry v1.7.0 →
65
+ worker/playground flip → site docs+news. The LFS upload pipeline is
66
+ the long pole (historically ~1h with CDN throttles) — start it first.
@@ -0,0 +1,55 @@
1
+ # Plan 123 — Verifiable typo corpora, synthesized at scale
2
+
3
+ Status: executed (models PR #32, 2026-09-12)
4
+
5
+ de/es: 5,000 pairs each, dictionary-grounded, deterministic. The
6
+ admission rule was SHAPED BY MEASUREMENT: the first draft sampled raw
7
+ dictionary stems and 77% of corrections landed outside the model
8
+ vocab - the corpus would have measured the vocabulary cut, not
9
+ ranking; the checked-in rule samples the rank-ordered dict-vocab
10
+ intersection (correction-OOV now 0). make_typo_traced added to
11
+ noise.py (op provenance, weighting unchanged). Headline tier finding:
12
+ full-tier typo-OOV on these typos is 98.5% de / 98.6% es - the
13
+ quantified gap the hybrid thread claims to close. Tests in the repo's
14
+ unittest convention (admission invariants, histogram consistency,
15
+ dictionary verification, byte-deterministic rebuild). The hybrid
16
+ pricing rerun (plan 115 ship decision) still needs the plan-114
17
+ training environment reassembled; the data blocker is closed.
18
+ Depends on: plan 115 (hybrid pricing blocked on real de/es corpora);
19
+ validated direction: DeepSeek-V4.1-Flash §5.1 — data/environment
20
+ synthesis exceeds algorithmic novelty
21
+
22
+ ## Problem
23
+
24
+ Plan 114's surviving thread (C-retrieve + fastText-rescore hybrid)
25
+ beat the full tier on all four real components but was parked for lack
26
+ of real de/es typo corpora; plan 115 priced the thread and named the
27
+ corpus as the blocker. Meanwhile the eval harness already owns the
28
+ generators: `eval/noise.py` (keyboard-aware grids, Turkish-Q/JCUKEN/
29
+ Greek, parameterized Latin family) and the per-language dictionaries.
30
+
31
+ ## Design
32
+
33
+ - Formalize a corpus entry as (typo, correction, provenance) — the
34
+ verification system is the dictionary itself plus the noise model:
35
+ a pair is admitted iff the correction is dictionary-valid, the typo
36
+ is not, and the typo is reachable from the correction by the
37
+ declared noise operation(s). Difficulty calibration = the noise
38
+ op mix and edit distance distribution, reported per corpus.
39
+ - Scale: ≥ 5k pairs per language for de and es (matching the en
40
+ frozen benchmark's 2,509 real pairs at minimum, target 2x), plus
41
+ pt/fr as stretch; dedup by (typo, correction); stratify by distance
42
+ 1/2 and by keyboard-adjacency vs random substitution.
43
+ - Honest split: synthesis trains/reprices, it must NOT be presented
44
+ as "real user" data — corpora carry `synthetic: true` provenance
45
+ and the plan-115 pricing reruns on both the frozen real components
46
+ (unchanged) and the synthetic corpora (new).
47
+ - Gates: the hybrid must still win on real components; synthetic
48
+ corpora may only ADD evidence, never replace the real gates
49
+ (pre-declared rule, plan 114 discipline).
50
+
51
+ ## Acceptance
52
+
53
+ - de/es corpora checked in with provenance + difficulty histograms.
54
+ - Plan 115 pricing rerun: hybrid vs full tier on real (unchanged) +
55
+ synthetic (new) — ship-only-if-wins applies to the union.
@@ -0,0 +1,19 @@
1
+ # Plan 124 — Owner-gated residue (inventory, not executable by agents)
2
+
3
+ Status: standing inventory (updated 2026-09-10)
4
+
5
+ - VS Code Marketplace + Open VSX PATs → publish kotoshu-vscode
6
+ (extension is publish-ready; owner: "later").
7
+ - Worktree cleanup — ASK OWNER before removing anything:
8
+ gem `.claude/worktrees/*` (10+ campaign worktrees), kotoshu-rs
9
+ `.claude/worktrees/packs` (content already merged), and /tmp
10
+ worktrees from the 1.0 day: fix-baseline, gem-dogfood, plan-116,
11
+ impl-116, impl-117, conf-0110, conf-0100, conf-092, conf-4597,
12
+ wasm-10, plans-118.
13
+ - kotoshu-lsp: still 0.1.1 — joins a future train if the owner wants
14
+ LSP on the 1.x line (audit train deliberately excluded it).
15
+ - Docker/kotoshu-ci + kotoshu-go/kotoshu-python floors: verify their
16
+ gem/base pins resolve 1.x on their next cuts (same class of bug as
17
+ plan 118 — check before, not after).
18
+ - rubygems 0.1.0 empties (lsp/server): support@rubygems.org deletion
19
+ remains the only path; cosmetic, owner call.
@@ -0,0 +1,43 @@
1
+ # Plan 125 — Dependency-floor guards in every consumer repo
2
+
3
+ Status: executed (docker PR #1; site push 70df393; 2026-09-12)
4
+
5
+ The guard found a REAL break on its first run: the docker CI image
6
+ could not install kotoshu at all since the native extension shipped
7
+ (ruby:slim lacks make; the 'no native ext' comment was stale). Fix:
8
+ build-essential + clang/libclang + rustup in the builder stage.
9
+ Dispositions: kotoshu-go has no artifact pin to guard (pure /v1 HTTP
10
+ client); kotoshu-python's native extra rides the deliberately-0.x
11
+ kotoshu-native line - a floor lands when that line crosses 1.0
12
+ (tracked in plan 124).
13
+ Depends on: plan 118 (the bug class: silent floor drift)
14
+
15
+ ## Problem
16
+
17
+ Plan 118 found kotoshu-server shipping 1.0.0 with a runtime floor
18
+ (`~> 0.6`) that excluded the gem it was releasing alongside — and its
19
+ own CI had been bundling kotoshu 0.6.0 the whole time. The same
20
+ silent-drift class lives wherever a consumer pins a kotoshu artifact:
21
+ docker-kotoshu-ci (base image gem), kotoshu-go (embedded wasm/native
22
+ pins or docs), kotoshu-python (wheel cargo/git pin of kotoshu-rs).
23
+ Nothing reads those pins after they are written.
24
+
25
+ ## Fix
26
+
27
+ Each repo gains the same guard shape the server got: a spec/test that
28
+ loads the pin (gemspec/Dockerfile/dependency manifest) and asserts it
29
+ admits the artifact version the repo actually builds/tests against.
30
+ A constraint nobody reads is guarded by a test that reads it.
31
+
32
+ - docker-kotoshu-ci: a CI step that installs the pinned gem and
33
+ asserts `kotoshu --version` >= the floor the Dockerfile names.
34
+ - kotoshu-python / kotoshu-go: assert the pinned kotoshu-rs rev/tag
35
+ resolves to the crate the wheels/binaries were built from (build
36
+ metadata comparison), or at minimum that the pin admits the
37
+ workspace version.
38
+
39
+ ## Acceptance
40
+
41
+ - Each repo's CI fails if its pin excludes the version it builds.
42
+ - 124's inventory item "floors to check on next cuts" shrinks to
43
+ "guards exist".
@@ -0,0 +1,33 @@
1
+ # Plan 126 — Publish the measured error budget per tier
2
+
3
+ Status: executed (site 70df393, 2026-09-12)
4
+
5
+ /performance carries the table: 55-language worst-case rank_corr /
6
+ top-1 agreement per tier against the gates, the vocabulary-cut
7
+ caveat, and the plan-123 finding that even the full tier embeds only
8
+ ~1.5% of dictionary-grounded typos.
9
+ Depends on: the 2026-09-10 model-efficiency research (DeepSeek-V4.1-Flash:
10
+ every "cheaper" claim carries its measured number)
11
+
12
+ ## Problem
13
+
14
+ The site documents tier sizes and the near-lossless headline
15
+ (fluency rank_corr 0.9999) but not a per-tier, per-language error
16
+ budget a user can reason about: what quality do I trade for mini's
17
+ 3 MB vs fluency's 15 MB vs full? The numbers exist (gates.json,
18
+ corpus benches); they are just not published as a table.
19
+
20
+ ## Fix
21
+
22
+ - /docs/performance gains an "Error budget" section: per tier, the
23
+ frozen gate values (fluency rank_corr, top-1/top-5 agreement deltas
24
+ vs full on the real corpora) with corpus sizes stated, plus the
25
+ dictionary-only baseline for contrast.
26
+ - The table cites its source (eval gates at registry tag) and its
27
+ date; numbers regenerate with each registry release (script optional
28
+ — a hand-truthed table with provenance beats a stale pipeline).
29
+
30
+ ## Acceptance
31
+
32
+ - A user can answer "what do I lose on mini?" with one number pair.
33
+ - The section states corpora sizes and OOV caveats honestly.
@@ -0,0 +1,34 @@
1
+ # Plan 127 — Report-a-wrong-suggestion: backendless feedback loop
2
+
3
+ Status: executed (site 70df393, 2026-09-12)
4
+
5
+ The popover carries a 'wrong suggestions? report' link opening a
6
+ pre-filled issue (word, top-5 rows with source+confidence, language,
7
+ engine version - no page text).
8
+ Depends on: the 2026-09-11 recommendation (real failure reports feed
9
+ plan 123's corpora; DeepSeek's environments-from-real-failures lesson)
10
+
11
+ ## Problem
12
+
13
+ The playground is our only surface where real users meet real
14
+ suggestions, and when a suggestion is wrong the feedback evaporates.
15
+ Plan 123 synthesizes pairs; the higher-value stream is REAL wrong-
16
+ suggestion reports, and capturing them needs no backend: a GitHub
17
+ issue prefill carries the diagnostic.
18
+
19
+ ## Fix
20
+
21
+ - Playground suggestion popover gains a small "report" affordance on
22
+ each row: opens github.com/kotoshu/kotoshu/issues/new?title=... with
23
+ a body template containing the word, the offered suggestions, the
24
+ language, the engine version, and (already public) tier info — no
25
+ page content, no user text beyond what the user chooses to add.
26
+ - The issue template lands in the gem repo (.github/ISSUE_TEMPLATE/
27
+ wrong-suggestion.md) so reports arrive structured; a triage note
28
+ points verified pairs at eval/corpora for the next reprice.
29
+
30
+ ## Acceptance
31
+
32
+ - One click from a wrong suggestion to a structured, pre-filled issue.
33
+ - No PII beyond the single word the user is already looking at; the
34
+ template says so.
@@ -0,0 +1,47 @@
1
+ # Plan 128 — The hybrid ship decision: price C-retrieve + fastText-rescore on the closed data blocker
2
+
3
+ Status: executed — VERDICT SHIP=True (models PR #34, 2026-09-12)
4
+
5
+ The full arc: frozen cbench rebuilt byte-faithfully (every sha256
6
+ reproduced; only the receipt wall-clock differed, reverted), c_typo_v2
7
+ retrained (0.481 MB, probe_hit5 0.410), pricing verdict per the
8
+ pre-declared rule: hybrid beats full on top-1 AND top-5 in all four
9
+ REAL components (en +6.3pp top5 CI [+4.9,+7.7] decisive; de +28.6,
10
+ ru +11.1, es +10.0), degrades nowhere, 25.8 MB int8 at ~3-4 ms.
11
+ Densified: on ~4,700 synth pairs/lang the hybrid evaluates essentially
12
+ all (top5 0.467-0.478) where full embeds only 69-85 pairs. Engine
13
+ productization = the follow-on plan (131).
14
+ Depends on: plan 123 (corpora shipped; the data blocker that parked
15
+ plan 115), plan 114 (the frozen C-benchmark + pre-declared verdict rule)
16
+
17
+ ## Problem
18
+
19
+ Plan 115 priced the hybrid retrieval thread and parked on missing de/es
20
+ typo data. Plan 123 closed that blocker (5,000 dictionary-grounded
21
+ pairs each). The plan-114 artifacts (frozen cbench files, c_typo_v2
22
+ encoder) were gitignored and wiped in the September cleanup — but the
23
+ corpus is vendored, the builder and trainer are committed, and the
24
+ whole chain is deterministic. The verdict arc is executable end to end.
25
+
26
+ ## Execution
27
+
28
+ 1. Rebuild the frozen C-benchmark: `scripts/build_cbench.py` — the
29
+ receipt (eval/reports/cbench.frozen.json) carries per-file sha256s;
30
+ the rebuild must reproduce them byte for byte or the freeze is
31
+ broken and the run stops (integrity before evidence).
32
+ 2. Retrain `c_typo_v2`: `scripts/train_typo_biencoder_v2.py` (char
33
+ bi-GRU, CPU, minutes; artifacts gitignored as before).
34
+ 3. Price: `eval/hybrid_pricing_bench.py` over en/de/ru/es — the frozen
35
+ REAL components are the ship gate (unchanged), the plan-123 corpora
36
+ are additional labeled evidence for the de/es gap.
37
+ 4. Verdict per the plan-114 pre-declared rule (wins on real, degrades
38
+ nowhere with n>=20) — ship, or reject on data like 114 did. Either
39
+ outcome closes the thread honestly.
40
+
41
+ ## Acceptance
42
+
43
+ - The cbench rebuild reproduces every frozen sha256.
44
+ - The pricing report lands (eval/reports/hybrid-pricing.json refresh)
45
+ with real + synth components labeled, and the verdict is recorded in
46
+ the plan status with numbers.
47
+ - No frozen artifact changes; new evidence only.
@@ -0,0 +1,23 @@
1
+ # Plan 129 — pt/fr synthetic corpora (plan 123 stretch targets)
2
+
3
+ Status: executed (models PR #34, 2026-09-12) — 5,000 pairs each,
4
+ same rule, tests at four languages; tier numbers in the refreshed
5
+ synth-corpus summary.
6
+ Depends on: plan 123 (the generator)
7
+
8
+ ## Problem
9
+
10
+ Plan 123 shipped de/es; pt and fr were declared stretch targets. The
11
+ generator, tests, and bench all generalize by construction — extending
12
+ is a parameter, not new code.
13
+
14
+ ## Fix
15
+
16
+ - Generate eval/corpora/synth/{pt,fr}.json at the same 5,000-pair
17
+ target with the same admission rule (pt: ABNT qwerty + accents; fr:
18
+ azerty alternates — both already in noise.py's layout registry).
19
+ - Extend the tier baseline bench + tests to the new languages.
20
+
21
+ ## Acceptance
22
+
23
+ - Same invariants as 123, same test coverage, tier numbers recorded.
@@ -0,0 +1,23 @@
1
+ # Plan 130 — News truth: the 1.0.1 patch and the 09-12 wave
2
+
3
+ Status: executed (site 6daa72d, 2026-09-12) — gem-1-0-1 and
4
+ evidence-2026-09 entries live; NewsKind widened with 'docs'.
5
+ Depends on: plans 119/121 (1.0.1 content), 123/125/126/127
6
+
7
+ ## Problem
8
+
9
+ Two shipped changes have no news entries: gem 1.0.1 (embedded frozen
10
+ tiers, Jekyll sweep skip) and the 09-12 wave (corpora, error budgets,
11
+ the feedback loop, the docker image fix). The site's discipline is one
12
+ entry per user-visible change.
13
+
14
+ ## Fix
15
+
16
+ - One entry for 1.0.1 (determinism by construction: cache-cold == warm
17
+ == frozen vectors).
18
+ - One entry for the wave: corpora + the 1.5% typo-embedding finding +
19
+ error-budget table + report-a-suggestion link.
20
+
21
+ ## Acceptance
22
+
23
+ - Entries live, dates and version references truthful.
@@ -0,0 +1,42 @@
1
+ # Plan 131 — Productize the hybrid: C-retrieve + fastText-rescore in the engines
2
+
3
+ Status: proposed (the decision evidence is complete; plan 128 verdict SHIP)
4
+ Depends on: plan 128 (verdict + artifacts), plan 113 (packs — the
5
+ delivery vehicle), registry v1.6.0 (typo-biencoder assets already opt-in)
6
+
7
+ ## What ships
8
+
9
+ The hybrid becomes the opt-in "typo retrieval" layer, the same shape
10
+ as the semantic tier: per-language int8 C-matrix (25.8 MB) + the char
11
+ bi-encoder (0.48 MB) fetched from the registry, C-retrieve top-20 over
12
+ the vocabulary, fastText-rescore of the candidates, merged into the
13
+ suggestion pipeline ahead of frequency ranking.
14
+
15
+ ## Surfaces
16
+
17
+ - kotoshu-rs: char-bi-encoder ONNX inference (the pure-Rust ONNX
18
+ reader already exists for the tiers), the retrieval+rescore path,
19
+ wasm surface (`loadTypo` + `typoSuggest`), KPK1 pack extension (the
20
+ C-matrix rides the pack for pack languages).
21
+ - gem: native-ext accelerated path where present; the merge into
22
+ Suggestions::Pipeline as a strategy (OCP: a new strategy class, no
23
+ pipeline surgery).
24
+ - Conformance: new vectors for the hybrid path are NOT frozen
25
+ initially (it is opt-in, tier-like, gated by eval not conformance —
26
+ the tier precedent).
27
+ - Registry: promote the typo-biencoder entries from opt-in to exposed;
28
+ server/playground/worker flips follow the semantic-tier pattern.
29
+
30
+ ## Gates (pre-declared)
31
+
32
+ - The frozen real components + synth corpora re-run against the
33
+ SHIPPED engines must reproduce the PR-34 numbers within noise.
34
+ - Latency: per-suggest budget ≤ 10 ms amortized (the plan-112 gates
35
+ tighten for the shipped path).
36
+ - Degrades nowhere vs dictionary-only when disabled; zero behavior
37
+ change at default settings.
38
+
39
+ ## Non-goals
40
+
41
+ - No retraining; the v2 artifacts are frozen (probe_hit5 0.410).
42
+ - No default-on: opt-in per language, like the semantic tier.