kotoshu 1.0.2-aarch64-linux

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (458) hide show
  1. checksums.yaml +7 -0
  2. data/.kotoshu-baseline.json +1 -0
  3. data/.pre-commit-hooks.yaml +18 -0
  4. data/.rspec +3 -0
  5. data/.rubocop.yml +351 -0
  6. data/.yardopts +8 -0
  7. data/CHANGELOG.md +813 -0
  8. data/CLAUDE.md +176 -0
  9. data/CODE_OF_CONDUCT.md +132 -0
  10. data/Cargo.lock +359 -0
  11. data/Cargo.toml +15 -0
  12. data/LICENSE +31 -0
  13. data/README.adoc +1292 -0
  14. data/Rakefile +150 -0
  15. data/SECURITY.md +93 -0
  16. data/TODO.impl/00-cut-0.1.md +121 -0
  17. data/TODO.impl/00-cut-0.2.md +175 -0
  18. data/TODO.impl/00-vision.md +281 -0
  19. data/TODO.impl/01-hunspell-correctness.md +111 -0
  20. data/TODO.impl/02-cli-unification.md +74 -0
  21. data/TODO.impl/03-dynamic-download.md +107 -0
  22. data/TODO.impl/04-language-modules.md +89 -0
  23. data/TODO.impl/05-semantic-path.md +83 -0
  24. data/TODO.impl/06-cjk-support.md +105 -0
  25. data/TODO.impl/07-rtl-support.md +71 -0
  26. data/TODO.impl/08-grammar-engine.md +106 -0
  27. data/TODO.impl/09-integrity-security.md +90 -0
  28. data/TODO.impl/10-testing-ci.md +98 -0
  29. data/TODO.impl/100-full-feature-batch3.md +28 -0
  30. data/TODO.impl/101-int4-tier-experiment.md +32 -0
  31. data/TODO.impl/102-lid-chain.md +28 -0
  32. data/TODO.impl/103-oov-bucket-tables.md +27 -0
  33. data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
  34. data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
  35. data/TODO.impl/106-native-lid-detection.md +36 -0
  36. data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
  37. data/TODO.impl/108-ko-ne-full-feature.md +37 -0
  38. data/TODO.impl/109-ten-audit.md +26 -0
  39. data/TODO.impl/11-release-v1.md +95 -0
  40. data/TODO.impl/110-sr-latn-and-polish.md +33 -0
  41. data/TODO.impl/111-embedding-bakeoff.md +33 -0
  42. data/TODO.impl/112-deployability-and-gating.md +26 -0
  43. data/TODO.impl/113-language-packs.md +26 -0
  44. data/TODO.impl/114-typo-biencoder-thread.md +33 -0
  45. data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
  46. data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
  47. data/TODO.impl/117-deterministic-frequency-tiers.md +110 -0
  48. data/TODO.impl/118-server-gem-floor.md +33 -0
  49. data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
  50. data/TODO.impl/120-setup-metadata-refresh.md +35 -0
  51. data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
  52. data/TODO.impl/122-pack-compression-kpk2.md +66 -0
  53. data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
  54. data/TODO.impl/124-owner-residue.md +19 -0
  55. data/TODO.impl/125-consumer-floor-guards.md +43 -0
  56. data/TODO.impl/126-tier-error-budgets.md +33 -0
  57. data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
  58. data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
  59. data/TODO.impl/129-pt-fr-corpora.md +23 -0
  60. data/TODO.impl/130-news-truth-0912.md +23 -0
  61. data/TODO.impl/131-hybrid-productization.md +42 -0
  62. data/TODO.impl/132-techniques-page-and-prose.md +27 -0
  63. data/TODO.impl/133-rust-first-distribution.md +68 -0
  64. data/TODO.impl/20-encapsulation-cleanup.md +47 -0
  65. data/TODO.impl/21-autoload-migration.md +64 -0
  66. data/TODO.impl/22-url-pin-single-source-of-truth.md +120 -0
  67. data/TODO.impl/23-fix-url-defaults.md +57 -0
  68. data/TODO.impl/24-cli-auto-setup-prompt.md +76 -0
  69. data/TODO.impl/25-soft-onnxruntime-require.md +75 -0
  70. data/TODO.impl/26-onnx-vocab-json-generation.md +96 -0
  71. data/TODO.impl/27-end-to-end-smoke-spec.md +113 -0
  72. data/TODO.impl/28-readme-quickstart-verification.md +68 -0
  73. data/TODO.impl/29-status-command.md +84 -0
  74. data/TODO.impl/30-language-auto-detection.md +74 -0
  75. data/TODO.impl/31-download-progress.md +58 -0
  76. data/TODO.impl/32-sarif-output-verification.md +72 -0
  77. data/TODO.impl/33-audit-log-rotation.md +54 -0
  78. data/TODO.impl/34-cache-eviction.md +59 -0
  79. data/TODO.impl/35-shell-completion.md +61 -0
  80. data/TODO.impl/36-cleanup-regressions-0.3.1.md +134 -0
  81. data/TODO.impl/37-hunspell-correctness-tier2.md +203 -0
  82. data/TODO.impl/38-onnx-semantic-gating.md +218 -0
  83. data/TODO.impl/39-tier3-and-beyond.md +232 -0
  84. data/TODO.impl/40-spec-drift-cleanup.md +134 -0
  85. data/TODO.impl/50-structure-aware-document-api.md +194 -0
  86. data/TODO.impl/51-grammar-engine-expansion.md +75 -0
  87. data/TODO.impl/52-multi-language-documents.md +95 -0
  88. data/TODO.impl/53-document-plugin-architecture.md +94 -0
  89. data/TODO.impl/54-cjk-support.md +117 -0
  90. data/TODO.impl/55-rtl-support.md +109 -0
  91. data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
  92. data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
  93. data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
  94. data/TODO.impl/59-yard-api-documentation.md +75 -0
  95. data/TODO.impl/60-lsp-server.md +174 -0
  96. data/TODO.impl/61-editor-ecosystem.md +167 -0
  97. data/TODO.impl/62-framework-integrations.md +159 -0
  98. data/TODO.impl/63-web-playground-wasm.md +186 -0
  99. data/TODO.impl/64-http-api-and-sdks.md +191 -0
  100. data/TODO.impl/65-universal-kotoshu.md +251 -0
  101. data/TODO.impl/66-kotoshu-core.md +169 -0
  102. data/TODO.impl/67-kotoshu-rs-and-access-libraries.md +137 -0
  103. data/TODO.impl/68-sota-adoption.md +152 -0
  104. data/TODO.impl/69-sota-typo-eval.md +74 -0
  105. data/TODO.impl/70-sota-confidence-cascade.md +59 -0
  106. data/TODO.impl/71-sota-nested-model.md +63 -0
  107. data/TODO.impl/72-model2vec-evaluation.md +49 -0
  108. data/TODO.impl/73-rl-reranker.md +42 -0
  109. data/TODO.impl/74-distilled-reranker.md +36 -0
  110. data/TODO.impl/75-llm-evaluator.md +35 -0
  111. data/TODO.impl/76-optional-llm-backend.md +37 -0
  112. data/TODO.impl/77-model-coverage-expansion.md +96 -0
  113. data/TODO.impl/78-in-browser-playground.md +69 -0
  114. data/TODO.impl/79-news-and-releases.md +60 -0
  115. data/TODO.impl/80-ecosystem-docs.md +56 -0
  116. data/TODO.impl/81-editor-reach.md +68 -0
  117. data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
  118. data/TODO.impl/83-models-batch-2.md +56 -0
  119. data/TODO.impl/84-language-modules-wave2.md +75 -0
  120. data/TODO.impl/85-wasm-model-api.md +48 -0
  121. data/TODO.impl/86-site-truth-wave2.md +45 -0
  122. data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
  123. data/TODO.impl/88-cli-directory-mode.md +54 -0
  124. data/TODO.impl/89-framework-integrations-exec.md +59 -0
  125. data/TODO.impl/90-gem-correctness-residues.md +44 -0
  126. data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
  127. data/TODO.impl/92-tier-mirror-cors.md +88 -0
  128. data/TODO.impl/93-python-wheels.md +32 -0
  129. data/TODO.impl/94-site-integrations-docs.md +35 -0
  130. data/TODO.impl/95-editor-action-polish.md +27 -0
  131. data/TODO.impl/96-server-semantic-models.md +50 -0
  132. data/TODO.impl/97-yard-execution.md +73 -0
  133. data/TODO.impl/98-campaign-cleanup.md +78 -0
  134. data/TODO.impl/99-site-truth-performance-docs.md +31 -0
  135. data/conformance/vectors.jsonl +2630 -0
  136. data/data/en/grammar/rules.yaml +851 -0
  137. data/docs/STABILITY-1.0-AUDIT.md +251 -0
  138. data/docs/STABILITY-1.0.md +137 -0
  139. data/docs/cache-architecture.md +502 -0
  140. data/docs/verification-evidence/correct3/pr_body.md +37 -0
  141. data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
  142. data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
  143. data/docs/verification-evidence/correct3/sv.aff +856 -0
  144. data/docs/verification-evidence/correct3/sv.dic +153717 -0
  145. data/examples/01_basic_word_checking.rb +38 -0
  146. data/examples/02_text_document_checking.rb +77 -0
  147. data/examples/03_dictionary_backends.rb +137 -0
  148. data/examples/04_trie_data_structure.rb +146 -0
  149. data/examples/05_suggestion_algorithms.rb +239 -0
  150. data/examples/06_configuration_advanced.rb +287 -0
  151. data/examples/07_multi_language_dictionaries.rb +278 -0
  152. data/exe/kotoshu +6 -0
  153. data/lib/kotoshu/algorithms/capitalization.rb +276 -0
  154. data/lib/kotoshu/algorithms/edit_distance.rb +128 -0
  155. data/lib/kotoshu/algorithms/lookup/affix_form.rb +132 -0
  156. data/lib/kotoshu/algorithms/lookup/compound_checks.rb +209 -0
  157. data/lib/kotoshu/algorithms/lookup/compound_form.rb +50 -0
  158. data/lib/kotoshu/algorithms/lookup.rb +841 -0
  159. data/lib/kotoshu/algorithms/ngram_suggest.rb +367 -0
  160. data/lib/kotoshu/algorithms/permutations.rb +283 -0
  161. data/lib/kotoshu/algorithms/phonet_suggest.rb +162 -0
  162. data/lib/kotoshu/algorithms/soundex.rb +92 -0
  163. data/lib/kotoshu/algorithms/suggest.rb +794 -0
  164. data/lib/kotoshu/algorithms.rb +23 -0
  165. data/lib/kotoshu/analyzers/semantic_analyzer.rb +405 -0
  166. data/lib/kotoshu/analyzers.rb +8 -0
  167. data/lib/kotoshu/baseline/application.rb +18 -0
  168. data/lib/kotoshu/baseline/entry.rb +33 -0
  169. data/lib/kotoshu/baseline/store.rb +175 -0
  170. data/lib/kotoshu/baseline.rb +20 -0
  171. data/lib/kotoshu/cache/base_cache.rb +778 -0
  172. data/lib/kotoshu/cache/cache.rb +91 -0
  173. data/lib/kotoshu/cache/eviction_policy.rb +66 -0
  174. data/lib/kotoshu/cache/frequency_cache.rb +283 -0
  175. data/lib/kotoshu/cache/language_cache.rb +511 -0
  176. data/lib/kotoshu/cache/lookup_cache.rb +166 -0
  177. data/lib/kotoshu/cache/model_cache.rb +1147 -0
  178. data/lib/kotoshu/cache/model_registry.rb +109 -0
  179. data/lib/kotoshu/cache/staged_languages.rb +45 -0
  180. data/lib/kotoshu/cache/suggestion_cache.rb +113 -0
  181. data/lib/kotoshu/cache.rb +49 -0
  182. data/lib/kotoshu/cli/auto_setup.rb +71 -0
  183. data/lib/kotoshu/cli/baseline_command.rb +119 -0
  184. data/lib/kotoshu/cli/batch_reporter.rb +321 -0
  185. data/lib/kotoshu/cli/cache_command.rb +282 -0
  186. data/lib/kotoshu/cli/completions_command.rb +210 -0
  187. data/lib/kotoshu/cli/dict_command.rb +64 -0
  188. data/lib/kotoshu/cli/directory_check.rb +146 -0
  189. data/lib/kotoshu/cli/directory_walker.rb +132 -0
  190. data/lib/kotoshu/cli/display_formatter.rb +429 -0
  191. data/lib/kotoshu/cli/errors.rb +36 -0
  192. data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
  193. data/lib/kotoshu/cli/interactive_reviewer.rb +314 -0
  194. data/lib/kotoshu/cli/language_resolver.rb +91 -0
  195. data/lib/kotoshu/cli/navigation_manager.rb +273 -0
  196. data/lib/kotoshu/cli/personal_command.rb +113 -0
  197. data/lib/kotoshu/cli/progress_reporter.rb +114 -0
  198. data/lib/kotoshu/cli/status_report.rb +130 -0
  199. data/lib/kotoshu/cli.rb +880 -0
  200. data/lib/kotoshu/components/passthrough_spell_checker.rb +70 -0
  201. data/lib/kotoshu/components/pos_tagger.rb +98 -0
  202. data/lib/kotoshu/components/spell_checker.rb +73 -0
  203. data/lib/kotoshu/components/synthesizer.rb +60 -0
  204. data/lib/kotoshu/components/tokenizer.rb +58 -0
  205. data/lib/kotoshu/components/whitespace_tokenizer.rb +94 -0
  206. data/lib/kotoshu/components.rb +13 -0
  207. data/lib/kotoshu/configuration/builder.rb +207 -0
  208. data/lib/kotoshu/configuration/resolver.rb +124 -0
  209. data/lib/kotoshu/configuration.rb +829 -0
  210. data/lib/kotoshu/conformance_exporter.rb +256 -0
  211. data/lib/kotoshu/conformance_runner.rb +176 -0
  212. data/lib/kotoshu/core/exceptions.rb +186 -0
  213. data/lib/kotoshu/core/indexed_dictionary.rb +293 -0
  214. data/lib/kotoshu/core/models/affix_rule.rb +262 -0
  215. data/lib/kotoshu/core/models/result/document_result.rb +132 -0
  216. data/lib/kotoshu/core/models/result/word_result.rb +131 -0
  217. data/lib/kotoshu/core/models/word.rb +131 -0
  218. data/lib/kotoshu/core/trie/builder.rb +119 -0
  219. data/lib/kotoshu/core/trie/node.rb +94 -0
  220. data/lib/kotoshu/core/trie/trie.rb +249 -0
  221. data/lib/kotoshu/core.rb +28 -0
  222. data/lib/kotoshu/data/common_words/de.yml +1800 -0
  223. data/lib/kotoshu/data/common_words/en.yml +1215 -0
  224. data/lib/kotoshu/data/common_words/es.yml +750 -0
  225. data/lib/kotoshu/data/common_words/fr.yml +1015 -0
  226. data/lib/kotoshu/data/common_words/pt.yml +870 -0
  227. data/lib/kotoshu/data/common_words/ru.yml +484 -0
  228. data/lib/kotoshu/data/common_words_loader.rb +141 -0
  229. data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
  230. data/lib/kotoshu/data/frozen_kelly.rb +38 -0
  231. data/lib/kotoshu/data.rb +8 -0
  232. data/lib/kotoshu/data_structures/bloom_filter.rb +188 -0
  233. data/lib/kotoshu/data_structures.rb +8 -0
  234. data/lib/kotoshu/debug_logger.rb +146 -0
  235. data/lib/kotoshu/debug_mode.rb +136 -0
  236. data/lib/kotoshu/defaults.rb +84 -0
  237. data/lib/kotoshu/dictionaries/catalog.rb +814 -0
  238. data/lib/kotoshu/dictionaries.rb +8 -0
  239. data/lib/kotoshu/dictionary/base.rb +324 -0
  240. data/lib/kotoshu/dictionary/cspell.rb +243 -0
  241. data/lib/kotoshu/dictionary/custom.rb +225 -0
  242. data/lib/kotoshu/dictionary/hunspell.rb +520 -0
  243. data/lib/kotoshu/dictionary/plain_text.rb +302 -0
  244. data/lib/kotoshu/dictionary/repository.rb +246 -0
  245. data/lib/kotoshu/dictionary/unified.rb +236 -0
  246. data/lib/kotoshu/dictionary/unix_words.rb +218 -0
  247. data/lib/kotoshu/dictionary.rb +15 -0
  248. data/lib/kotoshu/documents/document.rb +120 -0
  249. data/lib/kotoshu/documents/plain_text_document.rb +73 -0
  250. data/lib/kotoshu/documents/source_position.rb +45 -0
  251. data/lib/kotoshu/documents/source_range.rb +76 -0
  252. data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
  253. data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
  254. data/lib/kotoshu/documents/suppressions.rb +44 -0
  255. data/lib/kotoshu/documents/text_node.rb +49 -0
  256. data/lib/kotoshu/documents.rb +178 -0
  257. data/lib/kotoshu/embeddings/embedding_pipeline.rb +238 -0
  258. data/lib/kotoshu/embeddings/lru_cache.rb +244 -0
  259. data/lib/kotoshu/embeddings/onnx_runtime_model.rb +411 -0
  260. data/lib/kotoshu/embeddings/protocol.rb +96 -0
  261. data/lib/kotoshu/embeddings/protocols.rb +23 -0
  262. data/lib/kotoshu/embeddings/registry.rb +184 -0
  263. data/lib/kotoshu/embeddings/search.rb +204 -0
  264. data/lib/kotoshu/embeddings/similarity_engine.rb +278 -0
  265. data/lib/kotoshu/embeddings/similarity_search.rb +328 -0
  266. data/lib/kotoshu/embeddings/vocabulary.rb +306 -0
  267. data/lib/kotoshu/embeddings.rb +87 -0
  268. data/lib/kotoshu/fluent_checker.rb +94 -0
  269. data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +48 -0
  270. data/lib/kotoshu/grammar/pattern_matchers/double_negative_matcher.rb +179 -0
  271. data/lib/kotoshu/grammar/pattern_matchers/phrase_matcher.rb +86 -0
  272. data/lib/kotoshu/grammar/pattern_matchers/possessive_context_matcher.rb +75 -0
  273. data/lib/kotoshu/grammar/pattern_matchers/possessive_contraction_matcher.rb +87 -0
  274. data/lib/kotoshu/grammar/pattern_matchers/sentence_start_matcher.rb +84 -0
  275. data/lib/kotoshu/grammar/pattern_matchers/vowel_sound_matcher.rb +81 -0
  276. data/lib/kotoshu/grammar/pattern_matchers/word_list_matcher.rb +66 -0
  277. data/lib/kotoshu/grammar/rule.rb +99 -0
  278. data/lib/kotoshu/grammar/rule_engine.rb +96 -0
  279. data/lib/kotoshu/grammar/rule_loader.rb +30 -0
  280. data/lib/kotoshu/grammar.rb +24 -0
  281. data/lib/kotoshu/integrity/audit_log.rb +167 -0
  282. data/lib/kotoshu/integrity/manifest.rb +116 -0
  283. data/lib/kotoshu/integrity/net_http.rb +51 -0
  284. data/lib/kotoshu/integrity/rotation_policy.rb +88 -0
  285. data/lib/kotoshu/integrity.rb +26 -0
  286. data/lib/kotoshu/jekyll.rb +101 -0
  287. data/lib/kotoshu/keyboard/layout.rb +116 -0
  288. data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
  289. data/lib/kotoshu/keyboard/layouts/azerty.rb +55 -0
  290. data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
  291. data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
  292. data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
  293. data/lib/kotoshu/keyboard/layouts/dvorak.rb +54 -0
  294. data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
  295. data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
  296. data/lib/kotoshu/keyboard/layouts/hrsl.rb +94 -0
  297. data/lib/kotoshu/keyboard/layouts/jcuken.rb +57 -0
  298. data/lib/kotoshu/keyboard/layouts/latin.rb +266 -0
  299. data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
  300. data/lib/kotoshu/keyboard/layouts/qwerty.rb +52 -0
  301. data/lib/kotoshu/keyboard/layouts/qwertz.rb +55 -0
  302. data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
  303. data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
  304. data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
  305. data/lib/kotoshu/keyboard/registry.rb +201 -0
  306. data/lib/kotoshu/keyboard.rb +96 -0
  307. data/lib/kotoshu/kotoshu_native.so +0 -0
  308. data/lib/kotoshu/language/detector.rb +242 -0
  309. data/lib/kotoshu/language/identifier.rb +376 -0
  310. data/lib/kotoshu/language/languages/base.rb +271 -0
  311. data/lib/kotoshu/language/lid_detector.rb +188 -0
  312. data/lib/kotoshu/language/normalizer/arabic.rb +113 -0
  313. data/lib/kotoshu/language/normalizer/base.rb +133 -0
  314. data/lib/kotoshu/language/normalizer/greek.rb +63 -0
  315. data/lib/kotoshu/language/normalizer/hebrew.rb +72 -0
  316. data/lib/kotoshu/language/normalizer/persian.rb +36 -0
  317. data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
  318. data/lib/kotoshu/language/registry.rb +209 -0
  319. data/lib/kotoshu/language/resources/ar/common_words.txt +6753 -0
  320. data/lib/kotoshu/language/resources/ar/confusion_sets.txt +11 -0
  321. data/lib/kotoshu/language/resources/de/common_words.txt +10003 -0
  322. data/lib/kotoshu/language/resources/de/confusion_sets.txt +246 -0
  323. data/lib/kotoshu/language/resources/en/common_words.txt +9979 -0
  324. data/lib/kotoshu/language/resources/en/confusion_sets.txt +871 -0
  325. data/lib/kotoshu/language/resources/es/common_words.txt +9992 -0
  326. data/lib/kotoshu/language/resources/es/confusion_sets.txt +17 -0
  327. data/lib/kotoshu/language/resources/fr/common_words.txt +9993 -0
  328. data/lib/kotoshu/language/resources/fr/confusion_sets.txt +76 -0
  329. data/lib/kotoshu/language/resources/pt/common_words.txt +9977 -0
  330. data/lib/kotoshu/language/resources/pt/confusion_sets.txt +18 -0
  331. data/lib/kotoshu/language/resources/ru/common_words.txt +9951 -0
  332. data/lib/kotoshu/language/resources/ru/confusion_sets.txt +5 -0
  333. data/lib/kotoshu/language/script.rb +128 -0
  334. data/lib/kotoshu/language/segmenter.rb +122 -0
  335. data/lib/kotoshu/language/suika.rb +38 -0
  336. data/lib/kotoshu/language/tokenizer/base.rb +187 -0
  337. data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
  338. data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
  339. data/lib/kotoshu/language/tokenizer/french_tokenizer.rb +172 -0
  340. data/lib/kotoshu/language/tokenizer/german_tokenizer.rb +41 -0
  341. data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
  342. data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
  343. data/lib/kotoshu/language/tokenizer/japanese_tokenizer.rb +55 -0
  344. data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +153 -0
  345. data/lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb +160 -0
  346. data/lib/kotoshu/language/tokenizer/russian_tokenizer.rb +93 -0
  347. data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
  348. data/lib/kotoshu/language/tokenizer/spanish_tokenizer.rb +122 -0
  349. data/lib/kotoshu/language.rb +127 -0
  350. data/lib/kotoshu/languages/ar/language.rb +98 -0
  351. data/lib/kotoshu/languages/bg/language.rb +37 -0
  352. data/lib/kotoshu/languages/ca/language.rb +22 -0
  353. data/lib/kotoshu/languages/cs/language.rb +22 -0
  354. data/lib/kotoshu/languages/da/language.rb +22 -0
  355. data/lib/kotoshu/languages/de/language.rb +554 -0
  356. data/lib/kotoshu/languages/el/language.rb +50 -0
  357. data/lib/kotoshu/languages/en/language.rb +470 -0
  358. data/lib/kotoshu/languages/es/language.rb +476 -0
  359. data/lib/kotoshu/languages/et/language.rb +24 -0
  360. data/lib/kotoshu/languages/fa/language.rb +91 -0
  361. data/lib/kotoshu/languages/fr/language.rb +505 -0
  362. data/lib/kotoshu/languages/he/language.rb +98 -0
  363. data/lib/kotoshu/languages/hr/language.rb +24 -0
  364. data/lib/kotoshu/languages/hu/language.rb +22 -0
  365. data/lib/kotoshu/languages/id/language.rb +25 -0
  366. data/lib/kotoshu/languages/it/language.rb +22 -0
  367. data/lib/kotoshu/languages/ja/language.rb +476 -0
  368. data/lib/kotoshu/languages/ko/language.rb +56 -0
  369. data/lib/kotoshu/languages/latin_base.rb +53 -0
  370. data/lib/kotoshu/languages/lt/language.rb +24 -0
  371. data/lib/kotoshu/languages/lv/language.rb +24 -0
  372. data/lib/kotoshu/languages/nb/language.rb +25 -0
  373. data/lib/kotoshu/languages/ne/language.rb +60 -0
  374. data/lib/kotoshu/languages/nl/language.rb +22 -0
  375. data/lib/kotoshu/languages/nn/language.rb +26 -0
  376. data/lib/kotoshu/languages/pl/language.rb +22 -0
  377. data/lib/kotoshu/languages/pt/language.rb +439 -0
  378. data/lib/kotoshu/languages/ro/language.rb +22 -0
  379. data/lib/kotoshu/languages/ru/language.rb +420 -0
  380. data/lib/kotoshu/languages/sk/language.rb +25 -0
  381. data/lib/kotoshu/languages/sl/language.rb +25 -0
  382. data/lib/kotoshu/languages/sr/language.rb +39 -0
  383. data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
  384. data/lib/kotoshu/languages/sv/language.rb +22 -0
  385. data/lib/kotoshu/languages/tr/language.rb +36 -0
  386. data/lib/kotoshu/languages/uk/language.rb +38 -0
  387. data/lib/kotoshu/languages/vi/language.rb +22 -0
  388. data/lib/kotoshu/languages.rb +64 -0
  389. data/lib/kotoshu/metrics_collector.rb +238 -0
  390. data/lib/kotoshu/metrics_module.rb +121 -0
  391. data/lib/kotoshu/models/context.rb +119 -0
  392. data/lib/kotoshu/models/embedding_model.rb +181 -0
  393. data/lib/kotoshu/models/fasttext_model.rb +212 -0
  394. data/lib/kotoshu/models/nearest_neighbor.rb +96 -0
  395. data/lib/kotoshu/models/onnx_model.rb +330 -0
  396. data/lib/kotoshu/models/semantic_error.rb +204 -0
  397. data/lib/kotoshu/models/suggestion.rb +106 -0
  398. data/lib/kotoshu/models/word_embedding.rb +97 -0
  399. data/lib/kotoshu/models.rb +27 -0
  400. data/lib/kotoshu/multi_language_checker.rb +127 -0
  401. data/lib/kotoshu/native.rb +83 -0
  402. data/lib/kotoshu/native_backend.rb +131 -0
  403. data/lib/kotoshu/paths.rb +79 -0
  404. data/lib/kotoshu/personal_dictionary.rb +105 -0
  405. data/lib/kotoshu/plugins/plugin.rb +61 -0
  406. data/lib/kotoshu/plugins/registry.rb +118 -0
  407. data/lib/kotoshu/plugins.rb +9 -0
  408. data/lib/kotoshu/project_config.rb +76 -0
  409. data/lib/kotoshu/readers/aff_data.rb +530 -0
  410. data/lib/kotoshu/readers/aff_reader.rb +393 -0
  411. data/lib/kotoshu/readers/condition_checker.rb +125 -0
  412. data/lib/kotoshu/readers/dic_reader.rb +208 -0
  413. data/lib/kotoshu/readers/file_reader.rb +377 -0
  414. data/lib/kotoshu/readers/lookup_builder.rb +584 -0
  415. data/lib/kotoshu/readers/readers.rb +17 -0
  416. data/lib/kotoshu/readers.rb +25 -0
  417. data/lib/kotoshu/resource_bundle.rb +55 -0
  418. data/lib/kotoshu/resource_manager.rb +600 -0
  419. data/lib/kotoshu/results/result.rb +165 -0
  420. data/lib/kotoshu/results.rb +8 -0
  421. data/lib/kotoshu/rspec.rb +163 -0
  422. data/lib/kotoshu/scripts/fasttext_to_onnx.py +275 -0
  423. data/lib/kotoshu/source_registry.rb +104 -0
  424. data/lib/kotoshu/spellchecker/parallel_checker.rb +90 -0
  425. data/lib/kotoshu/spellchecker.rb +406 -0
  426. data/lib/kotoshu/string_metrics.rb +153 -0
  427. data/lib/kotoshu/suggestions/context.rb +43 -0
  428. data/lib/kotoshu/suggestions/frequency_provider.rb +80 -0
  429. data/lib/kotoshu/suggestions/generator.rb +169 -0
  430. data/lib/kotoshu/suggestions/pipeline.rb +135 -0
  431. data/lib/kotoshu/suggestions/semantic_cascade.rb +73 -0
  432. data/lib/kotoshu/suggestions/strategies/base_strategy.rb +315 -0
  433. data/lib/kotoshu/suggestions/strategies/composite_strategy.rb +197 -0
  434. data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +631 -0
  435. data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +207 -0
  436. data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +165 -0
  437. data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +284 -0
  438. data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +317 -0
  439. data/lib/kotoshu/suggestions/strategies/symspell_strategy.rb +270 -0
  440. data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
  441. data/lib/kotoshu/suggestions/suggestion.rb +154 -0
  442. data/lib/kotoshu/suggestions/suggestion_set.rb +270 -0
  443. data/lib/kotoshu/suggestions/sweep_index.rb +137 -0
  444. data/lib/kotoshu/suggestions.rb +29 -0
  445. data/lib/kotoshu/tasks/check_task.rb +104 -0
  446. data/lib/kotoshu/tasks.rb +11 -0
  447. data/lib/kotoshu/validators/spelling_validator.rb +101 -0
  448. data/lib/kotoshu/validators.rb +9 -0
  449. data/lib/kotoshu/version.rb +5 -0
  450. data/lib/kotoshu.rb +538 -0
  451. data/rakelib/compile.rake +10 -0
  452. data/script/validate_all_dictionaries.rb +444 -0
  453. data/scripts/build_native_gem.rb +45 -0
  454. data/scripts/extract_eval_grids.rb +37 -0
  455. data/scripts/generate_frozen_tiers.rb +44 -0
  456. data/sig/kotoshu.rbs +435 -0
  457. data/test_oop.rb +79 -0
  458. metadata +546 -0
@@ -0,0 +1,251 @@
1
+ # The 1.0 Readiness Audit
2
+
3
+ Plan 109. Analysis and documentation only — no behavior change, no version
4
+ bump, no release. The version cut itself is an owner decision.
5
+
6
+ - Audited at: gem 0.10.0 (`lib/kotoshu/version.rb:4`), main @ 73b7fac.
7
+ - Companion policy draft: `docs/STABILITY-1.0.md`.
8
+ - Method: every inventory line cites the file that defines the surface.
9
+ Reachability was verified by grep, not by assumption: a class being
10
+ present under `lib/` is not the same as being wired to `exe/kotoshu`.
11
+
12
+ ## Inventory
13
+
14
+ Verdicts: **stable** (freeze at 1.0), **deprecate-before-1.0** (needs a
15
+ deprecation cycle first), **accidentally-public** (internal or dead, 1.0
16
+ should privatize or delete).
17
+
18
+ ### Ruby gem — module functions (`lib/kotoshu.rb`)
19
+
20
+ | Surface | Where defined | Verdict | Note |
21
+ |---|---|---|---|
22
+ | `configure` / `configuration` / `configuration=` | `lib/kotoshu.rb:98,109,127` | stable | Type-checked global swap (127-131). |
23
+ | `spellchecker` / `spellchecker_for` | `lib/kotoshu.rb:139,157` | stable | Cache-only; README-documented (README.adoc:18). |
24
+ | `resolve` | `lib/kotoshu.rb:179` | stable | Stage-2 resolve; README.adoc:364. |
25
+ | `setup` / `setup?` / `languages_setup` | `lib/kotoshu.rb:220,245,257` | stable | Stage-1 entry; idempotent; `want:`/`tier:`/local-source opts frozen. |
26
+ | `reset_spellchecker` | `lib/kotoshu.rb:268` | stable | Test/embedding seam. |
27
+ | `correct?` / `misspelled?` / `suggest` / `check` / `check_file` / `check_files` | `lib/kotoshu.rb:286,297,313,331,349,363` | stable | The hot path; strict two-stage contract is the product identity. |
28
+ | `detect_language` / `detect_language_with_confidence` / `setup_lid` / `setup_lid?` | `lib/kotoshu.rb:465,481,501,508` | stable | Native LID with heuristic fallback (plan 106). |
29
+ | `get_language` / `language_registered?` / `supported_languages` / `language` | `lib/kotoshu.rb:519,530,540,443` | stable | `language` is a trivial namespace accessor; keep. |
30
+ | `dictionary` / `trie` / `suggestion_pipeline` | `lib/kotoshu.rb:371,388,409` | stable | Convenience factories; exercised by `examples/04..06`. |
31
+ | `register_dictionary_type` / `register_suggestion_algorithm` | `lib/kotoshu.rb:422,433` | stable | The extension points; 1.0 must not narrow them. |
32
+ | `VERSION` | `lib/kotoshu/version.rb:4` | stable | |
33
+
34
+ ### Ruby gem — core classes
35
+
36
+ | Surface | Where defined | Verdict | Note |
37
+ |---|---|---|---|
38
+ | `Kotoshu::Spellchecker` public methods: `correct?`, `incorrect?`, `suggest`, `check_word`, `check`, `check_file`, `check_directory`, `tokenize`, `dictionary`, `reload_dictionary` | `lib/kotoshu/spellchecker.rb:118,130,143,164,196,242,265,280,308,315` | stable | Constructor kwargs (`dictionary:`, `config:`, `resource_bundle:`) frozen at 65. |
39
+ | `Kotoshu::Spellchecker::ASCII_WORD_REGEX` | `lib/kotoshu/spellchecker.rb:39` | accidentally-public | Internal fallback word-char set exposed as a constant. Freeze + document, or privatize (see deprecations). |
40
+ | `Kotoshu::Configuration`: `SCHEMA`, `DEFAULTS`, accessors, `get`, `key?`, `load_dictionary`, `reset_dictionary`, `source_registry`, `self.default/instance/instance=/reset` | `lib/kotoshu/configuration.rb:38,222,423,431,447,474,517,541,565,576,585` | stable | ENV contract (`KOTOSHU_*`) rides on SCHEMA — every key there is public surface. |
41
+ | `Configuration#dictionaries_url` / `#models_url` | `lib/kotoshu/configuration.rb:63-74` (SCHEMA), `:318-321` (attrs) | deprecate-before-1.0 | SCHEMA descriptions already say "Deprecated: use repos_base_url + pins via SourceRegistry". |
42
+ | `ResourceManager.setup/resolve/setup?/languages_setup`, `DEFAULT_WANT` | `lib/kotoshu/resource_manager.rb:69,97,21` | stable | Facade delegates here; two-stage model frozen. |
43
+ | `ResourceBundle` (`dictionary`/`frequency`/`model`/`rules`) | `lib/kotoshu/resource_bundle.rb` | stable | Consumed by `Kotoshu.resolve` (kotoshu.rb:179). |
44
+ | Exception hierarchy: `Error` + `DictionaryNotFoundError`, `InvalidDictionaryFormatError`, `ConfigurationError`, `SpellcheckError`, `AffixRuleError`, `ResourceNotCachedError`, `ResourceNotSetupError`, `ResourceResolutionError`, `IntegrityError`, `SuikaUnavailable` | `lib/kotoshu/core/exceptions.rb:8,14,32,54,72,90,106,122,136,156,177`; autoloaded `lib/kotoshu.rb:65-75` | stable | Message text is not frozen; class names are. |
45
+ | `Kotoshu::Paths` (XDG + `KOTOSHU_{CACHE,CONFIG,DATA}_PATH`) | `lib/kotoshu/paths.rb` | stable | On-disk layout is user-visible contract. |
46
+ | Autoloaded namespaces behind `Kotoshu::*` | `lib/kotoshu.rb:14-62` | stable (as namespaces) | Namespaces stay; member-by-member stability is YARD-documented per class. 1.0 freezes namespace existence, not every leaf. |
47
+ | Top-level singleton aliases `Kotoshu::LanguageCache`, `Kotoshu::LanguageIdentifier`, `Kotoshu::ModelCache`, `Kotoshu::SemanticAnalyzer` | `lib/kotoshu.rb:80,81,84,85` | accidentally-public | Zero hits in lib/, spec/, README; canonical homes are `Kotoshu::Cache::LanguageCache`, `Kotoshu::Language::*`, `Kotoshu::Cache::ModelCache`, `Kotoshu::Analyzers::SemanticAnalyzer`. |
48
+ | `Kotoshu::Debug`, `DebugLogger`, `Metrics`, `MetricsCollector` | `lib/kotoshu.rb:78-83`; `lib/kotoshu/debug_mode.rb`, `metrics_module.rb`, `metrics_collector.rb` | stable | Documented in their doc comments; `Kotoshu::Metrics.stats` is README-documented (README.adoc:211). |
49
+ | `Kotoshu::FluentChecker` / `Kotoshu::MultiLanguageChecker` | `lib/kotoshu/fluent_checker.rb`, `lib/kotoshu/multi_language_checker.rb` | deprecate-before-1.0 (or document) | Spec-covered (`spec/kotoshu/{fluent,multi_language}_checker_spec.rb`) but absent from README.adoc — decide: document or deprecate. |
50
+
51
+ ### Ruby gem — framework integrations (all soft deps, plan 89)
52
+
53
+ | Surface | Where defined | Verdict | Note |
54
+ |---|---|---|---|
55
+ | `require "kotoshu/validators"` → `Kotoshu::Validators::SpellingValidator` (+ `ACTIVE_MODEL_LOADED`, `ActiveModelUnavailable`) | `lib/kotoshu/validators/spelling_validator.rb:10-99`; autoload `lib/kotoshu.rb:60` | stable | Placeholder class keeps constant resolvable without Rails (91-99). |
56
+ | `require "kotoshu/rspec"` → `Kotoshu::Rspec::Matchers` (`expect_words`, `expect_document`, `all_be_spelled_correctly`, `be_spelled_correctly(in:)`) | `lib/kotoshu/rspec.rb:26-160`; autoload `lib/kotoshu.rb:61` | stable | Matcher names are the contract. |
57
+ | `require "kotoshu/tasks"` → default `rake kotoshu:check`; `Kotoshu::Tasks::CheckTask` | `lib/kotoshu/tasks.rb:11`, `lib/kotoshu/tasks/check_task.rb:22` | stable | require-only (defining a task on load is the point — kotoshu.rb:56-59). |
58
+ | `require "kotoshu/jekyll"` → `Kotoshu::Jekyll::Generator` (+ `JEKYLL_LOADED`, `JekyllUnavailable`) | `lib/kotoshu/jekyll.rb:9-96`; autoload `lib/kotoshu.rb:62` | stable | Baseline honored via `Baseline::Store::DEFAULT_FILENAME`. |
59
+
60
+ ### Ruby gem — conformance surface (the behavioral contract)
61
+
62
+ | Surface | Where defined | Verdict | Note |
63
+ |---|---|---|---|
64
+ | `conformance/vectors.jsonl` (2630 rows, committed) | `conformance/vectors.jsonl`; sync contract documented in `lib/kotoshu/conformance_exporter.rb:54-60` | stable — frozen expectations | `expected` records what the Ruby engine ACTUALLY returns, not linguistic truth (conformance_exporter.rb:31-37). |
65
+ | Row shape v0: `kind/language/dictionary/input/limit/expected`; suggestion keys `word,distance,confidence,source` | `lib/kotoshu/conformance_exporter.rb:16-29,80` | stable | Four-key shape shared with wasm and `ffi::ruby`. |
66
+ | `Kotoshu::ConformanceRunner` (`run`, `compare`, rake tasks) | `lib/kotoshu/conformance_runner.rb:26,72,97` | stable | Guards Ruby-vs-vectors and native-vs-Ruby equality. |
67
+ | `Kotoshu::ConformanceExporter` (`export`, corpora selection, `EXCLUDED_FIXTURES`) | `lib/kotoshu/conformance_exporter.rb:61-254` | stable | Deterministic re-export (44-53). |
68
+
69
+ ### CLI (`exe/kotoshu` → `Kotoshu::Cli::Cli`)
70
+
71
+ | Surface | Where defined | Verdict | Note |
72
+ |---|---|---|---|
73
+ | Commands: `check`, `setup`, `dict`, `cache`, `personal`, `completions`, `baseline`, `status`, `version` (`--version`/`-V`) | `lib/kotoshu/cli.rb:178,241,317-329,349,358-367` | stable | |
74
+ | Class options `--language/-l`, `--format/-f` (text,json,sarif), `--interactive/-i`, `--verbose/-v` | `lib/kotoshu/cli.rb:102-125` | stable | |
75
+ | `check` options `--baseline`, `--show-suppressed`, `--include`, `--exclude`, `--personal` | `lib/kotoshu/cli.rb:151-168` | stable | |
76
+ | `setup` options `--aff --dic --from --frequency --want --model --tier --force --strict --list` | `lib/kotoshu/cli.rb:214-240` | stable | |
77
+ | Exit codes 0/1/2/3 | `lib/kotoshu/cli.rb:92-97,145-149` | stable | Consumed by action-kotoshu (action.yml:178-179). |
78
+ | JSON output top-level keys; SARIF 2.1.0 shape | `lib/kotoshu/cli.rb:624-651,686-715` | stable | Machine-readable contract; additive keys only. |
79
+ | `cache` subcommands `list/info/clean/evict/download/purge/validate` | `lib/kotoshu/cli/cache_command.rb:27,50,79,101,133,166,184` | stable | |
80
+ | `personal` subcommands `add/remove/list/import/path/clear` | `lib/kotoshu/cli/personal_command.rb:28-89` | stable | |
81
+ | `baseline init` (`--output`, `--language`) | `lib/kotoshu/cli/baseline_command.rb:15-36` | stable | |
82
+ | `completions bash/zsh` | `lib/kotoshu/cli/completions_command.rb:13-26` | stable | Catalog is hand-maintained — must track command changes. |
83
+ | `fetch` (hidden alias of `setup`) | `lib/kotoshu/cli.rb:280-315` | deprecate-before-1.0 | Already hidden and labeled deprecated; remove at 1.0. |
84
+
85
+ ### Ruby gem — accidentally-public legacy commands (the hazard block)
86
+
87
+ | Surface | Where defined | Verdict | Note |
88
+ |---|---|---|---|
89
+ | `::DictCommand` (top-level constant!) | `lib/kotoshu/cli.rb:10` | accidentally-public | Defined on `Object`, outside `module Kotoshu`. Pollutes the global namespace of every process that requires the gem; collides with any other gem defining `DictCommand`. Move under `Kotoshu::Cli` before 1.0. |
90
+ | `Kotoshu::Commands::CheckCommand` | `lib/kotoshu/commands/check_command.rb:4`; autoload `lib/kotoshu/commands.rb:9` | accidentally-public | Not wired into `Cli` (no `subcommand "check"` at cli.rb:317-329 — `check` is a method on `Cli` itself, cli.rb:178). Half-placeholder implementation; its hunspell path calls `SpellChecker.new` (check_command.rb:198) which resolves to nonexistent `Kotoshu::SpellChecker` (the real class is `Kotoshu::Spellchecker`) → `NameError` at runtime. Declares flags that do not exist on the real CLI (`--model`, `-o`, `csv/yaml` formats), contradicting cli.rb:108-113. Delete or privatize. |
91
+ | `Kotoshu::CacheCommand` | `lib/kotoshu/commands/cache_command.rb:4` | accidentally-public | Dead duplicate of the wired `Kotoshu::Cli::CacheCommand` (autoload cli.rb:70; subcommand cli.rb:321). Nothing requires this file (verified by grep); acknowledged as historical in `lib/kotoshu/commands.rb:7-8`. Still shipped in the gem (`spec.files` = `git ls-files`, kotoshu.gemspec:30-38). Delete. |
92
+ | `Kotoshu::Commands::ModelCommand` | `lib/kotoshu/commands/model_command.rb:4`; autoload `lib/kotoshu/commands.rb:10` | accidentally-public | No `kotoshu model` subcommand is registered anywhere. `convert` shells out to `scripts/convert_fasttext_to_onnx.py` (model_command.rb:33-36) — untracked in git, therefore NOT shipped in the gem, so the command cannot work as packaged. Delete. |
93
+
94
+ ### Ruby gem — native extension
95
+
96
+ | Surface | Where defined | Verdict | Note |
97
+ |---|---|---|---|
98
+ | Optional Rust ext build (`extconf.rb`) | `kotoshu.gemspec:53`; `ext/kotoshu_native/extconf.rb` | stable | Soft: pure-Ruby is the complete fallback (gemspec:44-52). |
99
+ | `Kotoshu::Native.available?`, `Native.suggestion(row)`, `Native::Unavailable`, `Native::EXTENSION` | `lib/kotoshu/native.rb:49,65,35,39` | stable | Extension defines `Native::VERSION`, `Native::Error`, `Native::Dictionary.load/correct?/suggest` (native.rb:9-14). |
100
+ | `Kotoshu::NativeBackend.resolve` boundary (ruby/auto/native) | `lib/kotoshu/native_backend.rb:45-73` | stable | Accelerator-only boundary is documented and frozen. |
101
+
102
+ ### wasm package (kotoshu-rs)
103
+
104
+ | Surface | Where defined | Verdict | Note |
105
+ |---|---|---|---|
106
+ | `KotoshuWasm` constructor `(affSource, dicSource)`, `correct(word)`, `suggest(word, limit?)`, static `VERSION` | `kotoshu-rs/kotoshu/src/ffi/wasm/mod.rs:165-174,178,185,152-155` | stable | Contents-not-paths loading; errors reject with the Rust message. |
107
+ | `loadModel(modelBytes, vocabBytes, bucketsBytes?)` → `KotoshuModel`; `.free()` | mod.rs:220-235,207-209 | stable | Optional third arg added by plan 103 — additive pattern to keep. |
108
+ | `rerank(model, word, context)` → number in [-1,1] | mod.rs:241-244 | stable | Mean-cosine contract documented (mod.rs:73-82). |
109
+ | `semanticSuggest(model, word, k?)` → `{word, score}[]` | mod.rs:252-266 | stable | |
110
+ | `loadLid(onnxBytes, vocabBytes)` → `KotoshuLid`; `detectLanguage(lid, text)` → `{code, score}` | mod.rs:281-287,293-300 | stable | Parity contract with the gem documented (mod.rs:84-94). |
111
+ | Suggestion row shape: exactly `word/distance/confidence/source` | mod.rs:48-54,190-198 | stable — frozen | Same four keys as the conformance vectors (`conformance_exporter.rb:80`). |
112
+ | npm identity | `kotoshu-rs/kotoshu-wasm/pkg/package.json:2,5` | needs a decision | Package name is `kotoshu-wasm` but every doc says `@kotoshu/wasm` (mod.rs:2-3); publication still blocked on npm org credentials. Decide the final name before 1.0 — npm names are sticky. |
113
+
114
+ ### HTTP server (kotoshu-server)
115
+
116
+ | Surface | Where defined | Verdict | Note |
117
+ |---|---|---|---|
118
+ | `GET /` (metadata), `GET /v1/health`, `GET /v1/version`, `GET /v1/languages` | `kotoshu-server/openapi.yaml:15,25,35,45`; routes `lib/kotoshu/server/app.rb:290,300,306,311` | stable | |
119
+ | `POST /v1/check` (`text`, `language`, `format`, `model`), `POST /v1/suggest` (`word`, `language`, `max` 1-50), `POST /v1/detect` (`text`) | openapi.yaml:55-117,179-203; routes app.rb:318,350,366 | stable | Errors: 400/422/503 shapes (openapi.yaml:118-137). |
120
+ | Response schemas `DocumentResult/WordError/Suggestions/Suggestion/Detection` (incl. `engine: lid-176|heuristic`) | openapi.yaml:204-245 | stable | openapi.yaml is the contract source of truth; app.rb matches it 1:1 (verified). |
121
+ | `/v1` prefix as the contract version | openapi.yaml:25-100 | stable | Breaking changes require `/v2`, not field mutations. |
122
+
123
+ ### GitHub action (action-kotoshu)
124
+
125
+ | Surface | Where defined | Verdict | Note |
126
+ |---|---|---|---|
127
+ | Inputs: `files, language, format, baseline, include, exclude, show_suppressed, fail_on_error, category, version, prewarm_languages, offline, output_path` | `action-kotoshu/action.yml:8-59` | stable — additive-only | New inputs must default to current behavior. |
128
+ | Outputs: `sarif-path`, `error-count` | action.yml:61-67 | stable | |
129
+ | Gem floor `>= 0.8.0` (v2 of the action) | action.yml:45,98-105 | stable | Floor bumps are breaking for the action and need a major action version. |
130
+
131
+ ### Registry schema (models-fasttext-onnx)
132
+
133
+ | Surface | Where defined | Verdict | Note |
134
+ |---|---|---|---|
135
+ | `spec: "kotoshu.resources/v1"` (const) | `models-fasttext-onnx/schemas/registry.schema.json:15-17` | stable — frozen | Change requires a new spec string, never an edit. |
136
+ | Resource id pattern `kotoshu://models/(lid/lid-176|{lang}/(full|fluency|mini|buckets))` | registry.schema.json:35 | stable | Ids are never reused; content changes bump `release_tag` (schema description, registry.schema.json:5). |
137
+ | Resource object fields incl. `min_engine_version`, `sha256`, `license` | registry.schema.json:43-165 | stable | `additionalProperties: false` — additive fields need a schema change = coordinated release. |
138
+
139
+ ## Deprecation list for the 1.0 cut
140
+
141
+ Concrete, with defining locations. All need small code PRs — deliberately
142
+ NOT implemented by this docs-only change.
143
+
144
+ 1. `::DictCommand` — `lib/kotoshu/cli.rb:10`. Global-namespace leak from a
145
+ file that otherwise lives in `Kotoshu::Cli`. Rename to
146
+ `Kotoshu::Cli::DictCommand` (internal; `subcommand "dict"` wiring at
147
+ cli.rb:317-318 is the only consumer). Highest-priority item: it can
148
+ break other gems in the same process, which is exactly the class of
149
+ thing 1.0 exists to prevent.
150
+ Handled in this PR: the class moved to
151
+ `lib/kotoshu/cli/dict_command.rb` as `Kotoshu::Cli::DictCommand`
152
+ (autoloaded like its sibling subcommands); `::DictCommand` is gone.
153
+ 2. `Kotoshu::Commands::CheckCommand` — `lib/kotoshu/commands/check_command.rb:4`
154
+ (autoload `lib/kotoshu/commands.rb:9`). Unwired, placeholder-laden, and
155
+ broken at check_command.rb:198 (`SpellChecker` → `NameError`; real class
156
+ is `Kotoshu::Spellchecker`, spellchecker.rb:21). Its flag surface
157
+ (`--model`, `-o/--output`, `csv|yaml` formats) contradicts the real CLI
158
+ and will mislead users who find it via constants. Delete; keep
159
+ `lib/kotoshu/commands.rb` as the namespace or remove both.
160
+ Handled in this PR: deleted, along with the now-empty `Commands`
161
+ namespace (file and autoload). No spec referenced it.
162
+ 3. `Kotoshu::CacheCommand` — `lib/kotoshu/commands/cache_command.rb:4`.
163
+ Dead duplicate of `Kotoshu::Cli::CacheCommand`; unreachable by any
164
+ require in the repo; still shipped by the gemspec file list. Delete.
165
+ Handled in this PR: deleted. No spec referenced it.
166
+ 4. `Kotoshu::Commands::ModelCommand` — `lib/kotoshu/commands/model_command.rb:4`
167
+ (autoload `lib/kotoshu/commands.rb:10`). No `model` subcommand exists;
168
+ `convert` depends on an untracked Python script not shipped in the gem
169
+ (model_command.rb:33-36 vs gemspec:30-38). Delete.
170
+ Handled in this PR: deleted. No spec referenced it.
171
+ 5. Alias constants `Kotoshu::LanguageCache`, `Kotoshu::LanguageIdentifier`,
172
+ `Kotoshu::ModelCache`, `Kotoshu::SemanticAnalyzer` — `lib/kotoshu.rb:80,81,84,85`.
173
+ No usage anywhere in lib/, spec/, or README. Add a deprecation `warn`
174
+ in a 0.x minor and drop the autoloads at 1.0 (canonical names stay).
175
+ Handled in this PR: grep confirmed zero references to the aliases
176
+ anywhere (lib, spec, exe, docs, README, sig) — the short-name hits all
177
+ resolve to the canonical constants lexically — so the autoloads were
178
+ dropped straight away, ahead of the suggested warn cycle. Canonical
179
+ names are untouched.
180
+ 6. CLI `fetch` — `lib/kotoshu/cli.rb:280-315`. Already hidden and marked
181
+ deprecated. Remove at 1.0.
182
+ 7. `Configuration#dictionaries_url` / `#models_url` —
183
+ `lib/kotoshu/configuration.rb:63-74,318-321`. Marked deprecated in SCHEMA
184
+ since the SourceRegistry pins landed. Emit a warning on use, remove the
185
+ SCHEMA entries at 1.0 (ENV vars `KOTOSHU_DICTIONARIES_URL` /
186
+ `KOTOSHU_MODELS_URL` go with them — call that out in the changelog).
187
+ 8. `Kotoshu::Spellchecker::ASCII_WORD_REGEX` — `lib/kotoshu/spellchecker.rb:39`.
188
+ Internal fallback set posing as API. Either document + freeze or make it
189
+ private before 1.0.
190
+ Handled in this PR: zero external references confirmed by grep, so the
191
+ constant was deleted and its literal inlined at the single internal use
192
+ site (constructor fallback). Behavior unchanged.
193
+ 9. `Kotoshu::FluentChecker` / `Kotoshu::MultiLanguageChecker` —
194
+ `lib/kotoshu/fluent_checker.rb`, `lib/kotoshu/multi_language_checker.rb`.
195
+ Spec-covered but undocumented in README.adoc. Decide per class: document
196
+ as stable, or deprecate. Do not freeze silently.
197
+ Handled in this PR: both class docstrings now state "Experimental and
198
+ not public API: outside the 1.0 stability freeze, may change or be
199
+ removed without a deprecation cycle." Specs stay.
200
+ 10. wasm npm identity — `kotoshu-rs/kotoshu-wasm/pkg/package.json:2` says
201
+ `kotoshu-wasm` while `kotoshu-rs/kotoshu/src/ffi/wasm/mod.rs:2-3` and
202
+ all docs say `@kotoshu/wasm`. Resolve before first publication; the
203
+ audit treats the documented scoped name as intended.
204
+ Handled in this PR: verified, no code change needed.
205
+ `kotoshu-rs/scripts/wasm_build.sh` rewrites the generated
206
+ `package.json` name to `@kotoshu/wasm` on every build (the source
207
+ manifest name is a wasm-pack crate-name artifact; `pkg/` is a
208
+ gitignored build output). The shipped identity is already
209
+ `@kotoshu/wasm`; the owner decision that remains is publication
210
+ credentials, not the name.
211
+
212
+ ## 1.0 checklist
213
+
214
+ Known blockers — none are mechanical code fixes; all are owner decisions
215
+ or ecosystem plumbing, stated honestly:
216
+
217
+ 1. Dispositions for the ten deprecation items above (small code PRs; this
218
+ audit only records them).
219
+ 2. npm org + publish credentials for `@kotoshu/wasm` (and the name
220
+ decision, item 10). The wasm build exists; publication is blocked.
221
+ 3. The JS SDK PR (kotoshu-js #1) still pending merge — the HTTP contract
222
+ it consumes must be the frozen one.
223
+ 4. PyPI token for kotoshu-python publication (same train).
224
+ 5. kotoshu-server is at 0.1.0 with an unexplained prewarm hang noted in
225
+ ecosystem state — the 1.0 train should not ship a server cut with a
226
+ known hang.
227
+ 6. RBS coverage: `sig/kotoshu.rbs` should be checked against the frozen
228
+ surface so the types do not contradict the freeze.
229
+ 7. Conformance vectors green on both engines
230
+ (`rake kotoshu:conformance:compare`) at the cut commit.
231
+
232
+ Release-train shape (gem + wasm + server together, per plan):
233
+
234
+ - Cut order: registry release_tag (if models moved) → gem 1.0 → wasm
235
+ package (independent version line, same day) → server (depends on the
236
+ gem floor) → action (no code change; floor stays 0.8.0-compatible, or
237
+ bump with an action major).
238
+ - The conformance vectors must be byte-identical across the train: they
239
+ are re-exported only when engine behavior intentionally changes, and
240
+ any such change is exactly what the deprecation window is for.
241
+ - Docs: this audit + `docs/STABILITY-1.0.md` land first (this PR); the
242
+ policy takes effect the day 1.0 ships.
243
+
244
+ What stays post-1.0 (policy constants, see the policy draft):
245
+
246
+ - The strict two-stage model — `setup` is never implicit; the hot path
247
+ never downloads.
248
+ - Soft-dependency policy for onnxruntime, suika, activemodel, jekyll.
249
+ - XDG paths + `KOTOSHU_*` envs as documented configuration surface.
250
+ - CLI exit codes 0/1/2/3 and the JSON/SARIF output shapes.
251
+ - `kotoshu.resources/v1` and the `/v1` HTTP prefix.
@@ -0,0 +1,137 @@
1
+ # Kotoshu Stability Policy — 1.0 draft
2
+
3
+ Status: DRAFT (plan 109). Takes effect the day 1.0 ships; until then the
4
+ usual 0.x rules apply (anything may change). The surface this policy
5
+ protects is itemized in `docs/STABILITY-1.0-AUDIT.md`.
6
+
7
+ ## Principles
8
+
9
+ 1. Semver, per delivery vehicle, cut together on one release train.
10
+ 2. Additive changes only, within a versioned boundary (`/v1`,
11
+ `kotoshu.resources/v1`, the four-key suggestion row).
12
+ 3. Behavior is frozen where the conformance vectors freeze it — the
13
+ vectors ARE the behavioral contract.
14
+ 4. No removal without a deprecation window (below).
15
+ 5. The strict two-stage resource model is permanent: `setup` is explicit,
16
+ the hot path never downloads. No feature may weaken it.
17
+
18
+ ## Per-surface commitments
19
+
20
+ ### Ruby gem (`kotoshu`)
21
+
22
+ - **Patch** (1.0.x): bug fixes only. No public API, CLI, ENV, output, or
23
+ conformance-vector change. `conformance/vectors.jsonl` must replay
24
+ byte-identically (`lib/kotoshu/conformance_runner.rb:72`).
25
+ - **Minor** (1.x): additive only — new methods, new config keys (must
26
+ default to current behavior), new CLI flags with defaults, new
27
+ exception subclasses of existing roots, new suggestion strategies.
28
+ Frozen in place: existing method signatures, the exception class list
29
+ at `lib/kotoshu/core/exceptions.rb`, `SCHEMA` key semantics
30
+ (`lib/kotoshu/configuration.rb:38`), exit codes 0/1/2/3
31
+ (`lib/kotoshu/cli.rb:92-97`), and the JSON/SARIF output shapes
32
+ (`lib/kotoshu/cli.rb:624-651,686-715`) — additions to JSON output are
33
+ allowed (consumers must ignore unknown keys), renames and removals are
34
+ not.
35
+ - **Major** (2.0): anything else, including conformance-vector re-exports
36
+ that change expected results.
37
+ - Ruby floor moves only at a major (`kotoshu.gemspec:17`).
38
+ - Requirements and dependency floors are owner decisions, never inferred
39
+ from code findings.
40
+
41
+ ### wasm package (`@kotoshu/wasm` / kotoshu-rs)
42
+
43
+ - Independent version line (its `package.json` governs, not the crate
44
+ constant — `kotoshu-rs/kotoshu/src/ffi/wasm/mod.rs:149-155`), released
45
+ on the same train as the gem.
46
+ - The JS surface is the module doc contract:
47
+ `KotoshuWasm` (constructor / `correct` / `suggest` / `VERSION`),
48
+ `loadModel`, `rerank`, `semanticSuggest`, `loadLid`, `detectLanguage`,
49
+ and the `KotoshuModel` / `KotoshuLid` handles
50
+ (`kotoshu-rs/kotoshu/src/ffi/wasm/mod.rs:19-46`).
51
+ - The suggestion row is frozen at exactly `word`, `distance`,
52
+ `confidence`, `source` — the conformance `SUGGESTION_KEYS`
53
+ (`lib/kotoshu/conformance_exporter.rb:80`). Additive optional
54
+ parameters are allowed (the `bucketsBytes` third argument of
55
+ `loadModel` is the precedent, mod.rs:215-219); changing existing
56
+ parameters or return shapes is a major.
57
+ - `KotoshuWasm.VERSION` tracks the crate version; it is informational,
58
+ not the package version.
59
+
60
+ ### HTTP server (kotoshu-server)
61
+
62
+ - `openapi.yaml` is the source of truth; `lib/kotoshu/server/app.rb`
63
+ must never diverge from it (today they match route for route).
64
+ - The `/v1` prefix IS the contract version. Within `/v1`: new endpoints,
65
+ new optional request fields, and new optional response fields are
66
+ minor. Changing a field type, removing a field, altering status-code
67
+ semantics (400/422/503), or mutating a frozen schema is a major —
68
+ breaking changes ship as `/v2`, with `/v1` kept for the deprecation
69
+ window.
70
+ - Error bodies keep the `{error, message, hint}` shape
71
+ (`kotoshu-server/openapi.yaml:246-251`).
72
+
73
+ ### GitHub action (action-kotoshu)
74
+
75
+ - Inputs are additive-only and every new input defaults to the previous
76
+ behavior (`action-kotoshu/action.yml:8-59`). Outputs
77
+ (`sarif-path`, `error-count`) are frozen.
78
+ - Bumping the gem floor (`version` input minimum, action.yml:98-105) or
79
+ dropping an input is a new action major. Exit-code mapping 0/1/2/3 is
80
+ part of the contract (action.yml:178-189).
81
+
82
+ ### Resource registry (models-fasttext-onnx)
83
+
84
+ - The spec string `kotoshu.resources/v1` is frozen
85
+ (`schemas/registry.schema.json:15-17`). Any schema change to the
86
+ registry shape means `kotoshu.resources/v2`, consumed side by side.
87
+ - Resource ids are permanent and never reused; artifact content changes
88
+ bump `release_tag` and `registry_version`, never the id (schema
89
+ description, registry.schema.json:5).
90
+ - `min_engine_version` is the compatibility lever for old gem versions
91
+ reading a newer registry — raising it requires a coordinated release
92
+ note.
93
+
94
+ ### Conformance vectors (the behavioral contract)
95
+
96
+ - `conformance/vectors.jsonl` (2630 rows) freezes what the engines
97
+ ACTUALLY return over the fixture corpora — Ruby and native must both
98
+ replay it identically (`lib/kotoshu/conformance_runner.rb:97-112`).
99
+ - A patch release MUST NOT change any vector. A minor may add vectors
100
+ (new corpora/words) but not change existing expectations. Changed
101
+ expectations = changed user-visible suggestions = major, with a
102
+ CHANGELOG entry listing affected inputs.
103
+ - Re-export procedure stays as documented: deterministic export
104
+ (`lib/kotoshu/conformance_exporter.rb:44-53`), commit to this repo,
105
+ kotoshu-rs syncs from it (conformance_exporter.rb:54-60).
106
+
107
+ ## Deprecation window
108
+
109
+ Proposal: **two minor releases** (the standard the ecosystem already
110
+ drifted toward — `fetch` and `dictionaries_url` both wait indefinitely;
111
+ 1.0 is the moment to make it finite).
112
+
113
+ - Ruby API: YARD `@deprecated` tag + a `Kernel#warn` (once per process)
114
+ from the moment of deprecation; removal at the second subsequent
115
+ minor, or at the next major, whichever comes first.
116
+ - Constants/autoloads (e.g. the alias constants at `lib/kotoshu.rb:80-85`):
117
+ same window; `autoload` removal is the deletion.
118
+ - CLI: the command/flag stays functional but hidden from help, printing a
119
+ stderr deprecation notice with the replacement; removal after two
120
+ minors. Exit codes do not change during the window.
121
+ - ENV/config keys (`KOTOSHU_*`): warn on read for two minors, then remove.
122
+ ENV removals are called out in the changelog header — they break CI
123
+ setups silently otherwise.
124
+ - HTTP: a deprecated field/endpoint returns `Deprecation`/`Sunset` style
125
+ headers during the window; `/v1` keeps serving it until `/v2` has been
126
+ stable for one minor.
127
+ - Exception: anything already deprecated throughout late 0.x (the `fetch`
128
+ alias, `dictionaries_url`/`models_url`) may be removed AT 1.0 without a
129
+ new window — the 0.x line was the window.
130
+
131
+ ## The 1.0 gate
132
+
133
+ 1.0 ships when the audit checklist in `docs/STABILITY-1.0-AUDIT.md` is
134
+ cleared: deprecation dispositions landed, the release train (gem + wasm +
135
+ server) coordinated, conformance compare green on both engines, and this
136
+ policy approved by the owner. Version numbers, floors, and the cut date
137
+ are owner decisions — this document proposes, the owner disposes.