kotoshu 1.0.2-x86_64-linux

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (458) hide show
  1. checksums.yaml +7 -0
  2. data/.kotoshu-baseline.json +1 -0
  3. data/.pre-commit-hooks.yaml +18 -0
  4. data/.rspec +3 -0
  5. data/.rubocop.yml +351 -0
  6. data/.yardopts +8 -0
  7. data/CHANGELOG.md +813 -0
  8. data/CLAUDE.md +176 -0
  9. data/CODE_OF_CONDUCT.md +132 -0
  10. data/Cargo.lock +359 -0
  11. data/Cargo.toml +15 -0
  12. data/LICENSE +31 -0
  13. data/README.adoc +1292 -0
  14. data/Rakefile +150 -0
  15. data/SECURITY.md +93 -0
  16. data/TODO.impl/00-cut-0.1.md +121 -0
  17. data/TODO.impl/00-cut-0.2.md +175 -0
  18. data/TODO.impl/00-vision.md +281 -0
  19. data/TODO.impl/01-hunspell-correctness.md +111 -0
  20. data/TODO.impl/02-cli-unification.md +74 -0
  21. data/TODO.impl/03-dynamic-download.md +107 -0
  22. data/TODO.impl/04-language-modules.md +89 -0
  23. data/TODO.impl/05-semantic-path.md +83 -0
  24. data/TODO.impl/06-cjk-support.md +105 -0
  25. data/TODO.impl/07-rtl-support.md +71 -0
  26. data/TODO.impl/08-grammar-engine.md +106 -0
  27. data/TODO.impl/09-integrity-security.md +90 -0
  28. data/TODO.impl/10-testing-ci.md +98 -0
  29. data/TODO.impl/100-full-feature-batch3.md +28 -0
  30. data/TODO.impl/101-int4-tier-experiment.md +32 -0
  31. data/TODO.impl/102-lid-chain.md +28 -0
  32. data/TODO.impl/103-oov-bucket-tables.md +27 -0
  33. data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
  34. data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
  35. data/TODO.impl/106-native-lid-detection.md +36 -0
  36. data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
  37. data/TODO.impl/108-ko-ne-full-feature.md +37 -0
  38. data/TODO.impl/109-ten-audit.md +26 -0
  39. data/TODO.impl/11-release-v1.md +95 -0
  40. data/TODO.impl/110-sr-latn-and-polish.md +33 -0
  41. data/TODO.impl/111-embedding-bakeoff.md +33 -0
  42. data/TODO.impl/112-deployability-and-gating.md +26 -0
  43. data/TODO.impl/113-language-packs.md +26 -0
  44. data/TODO.impl/114-typo-biencoder-thread.md +33 -0
  45. data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
  46. data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
  47. data/TODO.impl/117-deterministic-frequency-tiers.md +110 -0
  48. data/TODO.impl/118-server-gem-floor.md +33 -0
  49. data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
  50. data/TODO.impl/120-setup-metadata-refresh.md +35 -0
  51. data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
  52. data/TODO.impl/122-pack-compression-kpk2.md +66 -0
  53. data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
  54. data/TODO.impl/124-owner-residue.md +19 -0
  55. data/TODO.impl/125-consumer-floor-guards.md +43 -0
  56. data/TODO.impl/126-tier-error-budgets.md +33 -0
  57. data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
  58. data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
  59. data/TODO.impl/129-pt-fr-corpora.md +23 -0
  60. data/TODO.impl/130-news-truth-0912.md +23 -0
  61. data/TODO.impl/131-hybrid-productization.md +42 -0
  62. data/TODO.impl/132-techniques-page-and-prose.md +27 -0
  63. data/TODO.impl/133-rust-first-distribution.md +68 -0
  64. data/TODO.impl/20-encapsulation-cleanup.md +47 -0
  65. data/TODO.impl/21-autoload-migration.md +64 -0
  66. data/TODO.impl/22-url-pin-single-source-of-truth.md +120 -0
  67. data/TODO.impl/23-fix-url-defaults.md +57 -0
  68. data/TODO.impl/24-cli-auto-setup-prompt.md +76 -0
  69. data/TODO.impl/25-soft-onnxruntime-require.md +75 -0
  70. data/TODO.impl/26-onnx-vocab-json-generation.md +96 -0
  71. data/TODO.impl/27-end-to-end-smoke-spec.md +113 -0
  72. data/TODO.impl/28-readme-quickstart-verification.md +68 -0
  73. data/TODO.impl/29-status-command.md +84 -0
  74. data/TODO.impl/30-language-auto-detection.md +74 -0
  75. data/TODO.impl/31-download-progress.md +58 -0
  76. data/TODO.impl/32-sarif-output-verification.md +72 -0
  77. data/TODO.impl/33-audit-log-rotation.md +54 -0
  78. data/TODO.impl/34-cache-eviction.md +59 -0
  79. data/TODO.impl/35-shell-completion.md +61 -0
  80. data/TODO.impl/36-cleanup-regressions-0.3.1.md +134 -0
  81. data/TODO.impl/37-hunspell-correctness-tier2.md +203 -0
  82. data/TODO.impl/38-onnx-semantic-gating.md +218 -0
  83. data/TODO.impl/39-tier3-and-beyond.md +232 -0
  84. data/TODO.impl/40-spec-drift-cleanup.md +134 -0
  85. data/TODO.impl/50-structure-aware-document-api.md +194 -0
  86. data/TODO.impl/51-grammar-engine-expansion.md +75 -0
  87. data/TODO.impl/52-multi-language-documents.md +95 -0
  88. data/TODO.impl/53-document-plugin-architecture.md +94 -0
  89. data/TODO.impl/54-cjk-support.md +117 -0
  90. data/TODO.impl/55-rtl-support.md +109 -0
  91. data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
  92. data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
  93. data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
  94. data/TODO.impl/59-yard-api-documentation.md +75 -0
  95. data/TODO.impl/60-lsp-server.md +174 -0
  96. data/TODO.impl/61-editor-ecosystem.md +167 -0
  97. data/TODO.impl/62-framework-integrations.md +159 -0
  98. data/TODO.impl/63-web-playground-wasm.md +186 -0
  99. data/TODO.impl/64-http-api-and-sdks.md +191 -0
  100. data/TODO.impl/65-universal-kotoshu.md +251 -0
  101. data/TODO.impl/66-kotoshu-core.md +169 -0
  102. data/TODO.impl/67-kotoshu-rs-and-access-libraries.md +137 -0
  103. data/TODO.impl/68-sota-adoption.md +152 -0
  104. data/TODO.impl/69-sota-typo-eval.md +74 -0
  105. data/TODO.impl/70-sota-confidence-cascade.md +59 -0
  106. data/TODO.impl/71-sota-nested-model.md +63 -0
  107. data/TODO.impl/72-model2vec-evaluation.md +49 -0
  108. data/TODO.impl/73-rl-reranker.md +42 -0
  109. data/TODO.impl/74-distilled-reranker.md +36 -0
  110. data/TODO.impl/75-llm-evaluator.md +35 -0
  111. data/TODO.impl/76-optional-llm-backend.md +37 -0
  112. data/TODO.impl/77-model-coverage-expansion.md +96 -0
  113. data/TODO.impl/78-in-browser-playground.md +69 -0
  114. data/TODO.impl/79-news-and-releases.md +60 -0
  115. data/TODO.impl/80-ecosystem-docs.md +56 -0
  116. data/TODO.impl/81-editor-reach.md +68 -0
  117. data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
  118. data/TODO.impl/83-models-batch-2.md +56 -0
  119. data/TODO.impl/84-language-modules-wave2.md +75 -0
  120. data/TODO.impl/85-wasm-model-api.md +48 -0
  121. data/TODO.impl/86-site-truth-wave2.md +45 -0
  122. data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
  123. data/TODO.impl/88-cli-directory-mode.md +54 -0
  124. data/TODO.impl/89-framework-integrations-exec.md +59 -0
  125. data/TODO.impl/90-gem-correctness-residues.md +44 -0
  126. data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
  127. data/TODO.impl/92-tier-mirror-cors.md +88 -0
  128. data/TODO.impl/93-python-wheels.md +32 -0
  129. data/TODO.impl/94-site-integrations-docs.md +35 -0
  130. data/TODO.impl/95-editor-action-polish.md +27 -0
  131. data/TODO.impl/96-server-semantic-models.md +50 -0
  132. data/TODO.impl/97-yard-execution.md +73 -0
  133. data/TODO.impl/98-campaign-cleanup.md +78 -0
  134. data/TODO.impl/99-site-truth-performance-docs.md +31 -0
  135. data/conformance/vectors.jsonl +2630 -0
  136. data/data/en/grammar/rules.yaml +851 -0
  137. data/docs/STABILITY-1.0-AUDIT.md +251 -0
  138. data/docs/STABILITY-1.0.md +137 -0
  139. data/docs/cache-architecture.md +502 -0
  140. data/docs/verification-evidence/correct3/pr_body.md +37 -0
  141. data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
  142. data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
  143. data/docs/verification-evidence/correct3/sv.aff +856 -0
  144. data/docs/verification-evidence/correct3/sv.dic +153717 -0
  145. data/examples/01_basic_word_checking.rb +38 -0
  146. data/examples/02_text_document_checking.rb +77 -0
  147. data/examples/03_dictionary_backends.rb +137 -0
  148. data/examples/04_trie_data_structure.rb +146 -0
  149. data/examples/05_suggestion_algorithms.rb +239 -0
  150. data/examples/06_configuration_advanced.rb +287 -0
  151. data/examples/07_multi_language_dictionaries.rb +278 -0
  152. data/exe/kotoshu +6 -0
  153. data/lib/kotoshu/algorithms/capitalization.rb +276 -0
  154. data/lib/kotoshu/algorithms/edit_distance.rb +128 -0
  155. data/lib/kotoshu/algorithms/lookup/affix_form.rb +132 -0
  156. data/lib/kotoshu/algorithms/lookup/compound_checks.rb +209 -0
  157. data/lib/kotoshu/algorithms/lookup/compound_form.rb +50 -0
  158. data/lib/kotoshu/algorithms/lookup.rb +841 -0
  159. data/lib/kotoshu/algorithms/ngram_suggest.rb +367 -0
  160. data/lib/kotoshu/algorithms/permutations.rb +283 -0
  161. data/lib/kotoshu/algorithms/phonet_suggest.rb +162 -0
  162. data/lib/kotoshu/algorithms/soundex.rb +92 -0
  163. data/lib/kotoshu/algorithms/suggest.rb +794 -0
  164. data/lib/kotoshu/algorithms.rb +23 -0
  165. data/lib/kotoshu/analyzers/semantic_analyzer.rb +405 -0
  166. data/lib/kotoshu/analyzers.rb +8 -0
  167. data/lib/kotoshu/baseline/application.rb +18 -0
  168. data/lib/kotoshu/baseline/entry.rb +33 -0
  169. data/lib/kotoshu/baseline/store.rb +175 -0
  170. data/lib/kotoshu/baseline.rb +20 -0
  171. data/lib/kotoshu/cache/base_cache.rb +778 -0
  172. data/lib/kotoshu/cache/cache.rb +91 -0
  173. data/lib/kotoshu/cache/eviction_policy.rb +66 -0
  174. data/lib/kotoshu/cache/frequency_cache.rb +283 -0
  175. data/lib/kotoshu/cache/language_cache.rb +511 -0
  176. data/lib/kotoshu/cache/lookup_cache.rb +166 -0
  177. data/lib/kotoshu/cache/model_cache.rb +1147 -0
  178. data/lib/kotoshu/cache/model_registry.rb +109 -0
  179. data/lib/kotoshu/cache/staged_languages.rb +45 -0
  180. data/lib/kotoshu/cache/suggestion_cache.rb +113 -0
  181. data/lib/kotoshu/cache.rb +49 -0
  182. data/lib/kotoshu/cli/auto_setup.rb +71 -0
  183. data/lib/kotoshu/cli/baseline_command.rb +119 -0
  184. data/lib/kotoshu/cli/batch_reporter.rb +321 -0
  185. data/lib/kotoshu/cli/cache_command.rb +282 -0
  186. data/lib/kotoshu/cli/completions_command.rb +210 -0
  187. data/lib/kotoshu/cli/dict_command.rb +64 -0
  188. data/lib/kotoshu/cli/directory_check.rb +146 -0
  189. data/lib/kotoshu/cli/directory_walker.rb +132 -0
  190. data/lib/kotoshu/cli/display_formatter.rb +429 -0
  191. data/lib/kotoshu/cli/errors.rb +36 -0
  192. data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
  193. data/lib/kotoshu/cli/interactive_reviewer.rb +314 -0
  194. data/lib/kotoshu/cli/language_resolver.rb +91 -0
  195. data/lib/kotoshu/cli/navigation_manager.rb +273 -0
  196. data/lib/kotoshu/cli/personal_command.rb +113 -0
  197. data/lib/kotoshu/cli/progress_reporter.rb +114 -0
  198. data/lib/kotoshu/cli/status_report.rb +130 -0
  199. data/lib/kotoshu/cli.rb +880 -0
  200. data/lib/kotoshu/components/passthrough_spell_checker.rb +70 -0
  201. data/lib/kotoshu/components/pos_tagger.rb +98 -0
  202. data/lib/kotoshu/components/spell_checker.rb +73 -0
  203. data/lib/kotoshu/components/synthesizer.rb +60 -0
  204. data/lib/kotoshu/components/tokenizer.rb +58 -0
  205. data/lib/kotoshu/components/whitespace_tokenizer.rb +94 -0
  206. data/lib/kotoshu/components.rb +13 -0
  207. data/lib/kotoshu/configuration/builder.rb +207 -0
  208. data/lib/kotoshu/configuration/resolver.rb +124 -0
  209. data/lib/kotoshu/configuration.rb +829 -0
  210. data/lib/kotoshu/conformance_exporter.rb +256 -0
  211. data/lib/kotoshu/conformance_runner.rb +176 -0
  212. data/lib/kotoshu/core/exceptions.rb +186 -0
  213. data/lib/kotoshu/core/indexed_dictionary.rb +293 -0
  214. data/lib/kotoshu/core/models/affix_rule.rb +262 -0
  215. data/lib/kotoshu/core/models/result/document_result.rb +132 -0
  216. data/lib/kotoshu/core/models/result/word_result.rb +131 -0
  217. data/lib/kotoshu/core/models/word.rb +131 -0
  218. data/lib/kotoshu/core/trie/builder.rb +119 -0
  219. data/lib/kotoshu/core/trie/node.rb +94 -0
  220. data/lib/kotoshu/core/trie/trie.rb +249 -0
  221. data/lib/kotoshu/core.rb +28 -0
  222. data/lib/kotoshu/data/common_words/de.yml +1800 -0
  223. data/lib/kotoshu/data/common_words/en.yml +1215 -0
  224. data/lib/kotoshu/data/common_words/es.yml +750 -0
  225. data/lib/kotoshu/data/common_words/fr.yml +1015 -0
  226. data/lib/kotoshu/data/common_words/pt.yml +870 -0
  227. data/lib/kotoshu/data/common_words/ru.yml +484 -0
  228. data/lib/kotoshu/data/common_words_loader.rb +141 -0
  229. data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
  230. data/lib/kotoshu/data/frozen_kelly.rb +38 -0
  231. data/lib/kotoshu/data.rb +8 -0
  232. data/lib/kotoshu/data_structures/bloom_filter.rb +188 -0
  233. data/lib/kotoshu/data_structures.rb +8 -0
  234. data/lib/kotoshu/debug_logger.rb +146 -0
  235. data/lib/kotoshu/debug_mode.rb +136 -0
  236. data/lib/kotoshu/defaults.rb +84 -0
  237. data/lib/kotoshu/dictionaries/catalog.rb +814 -0
  238. data/lib/kotoshu/dictionaries.rb +8 -0
  239. data/lib/kotoshu/dictionary/base.rb +324 -0
  240. data/lib/kotoshu/dictionary/cspell.rb +243 -0
  241. data/lib/kotoshu/dictionary/custom.rb +225 -0
  242. data/lib/kotoshu/dictionary/hunspell.rb +520 -0
  243. data/lib/kotoshu/dictionary/plain_text.rb +302 -0
  244. data/lib/kotoshu/dictionary/repository.rb +246 -0
  245. data/lib/kotoshu/dictionary/unified.rb +236 -0
  246. data/lib/kotoshu/dictionary/unix_words.rb +218 -0
  247. data/lib/kotoshu/dictionary.rb +15 -0
  248. data/lib/kotoshu/documents/document.rb +120 -0
  249. data/lib/kotoshu/documents/plain_text_document.rb +73 -0
  250. data/lib/kotoshu/documents/source_position.rb +45 -0
  251. data/lib/kotoshu/documents/source_range.rb +76 -0
  252. data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
  253. data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
  254. data/lib/kotoshu/documents/suppressions.rb +44 -0
  255. data/lib/kotoshu/documents/text_node.rb +49 -0
  256. data/lib/kotoshu/documents.rb +178 -0
  257. data/lib/kotoshu/embeddings/embedding_pipeline.rb +238 -0
  258. data/lib/kotoshu/embeddings/lru_cache.rb +244 -0
  259. data/lib/kotoshu/embeddings/onnx_runtime_model.rb +411 -0
  260. data/lib/kotoshu/embeddings/protocol.rb +96 -0
  261. data/lib/kotoshu/embeddings/protocols.rb +23 -0
  262. data/lib/kotoshu/embeddings/registry.rb +184 -0
  263. data/lib/kotoshu/embeddings/search.rb +204 -0
  264. data/lib/kotoshu/embeddings/similarity_engine.rb +278 -0
  265. data/lib/kotoshu/embeddings/similarity_search.rb +328 -0
  266. data/lib/kotoshu/embeddings/vocabulary.rb +306 -0
  267. data/lib/kotoshu/embeddings.rb +87 -0
  268. data/lib/kotoshu/fluent_checker.rb +94 -0
  269. data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +48 -0
  270. data/lib/kotoshu/grammar/pattern_matchers/double_negative_matcher.rb +179 -0
  271. data/lib/kotoshu/grammar/pattern_matchers/phrase_matcher.rb +86 -0
  272. data/lib/kotoshu/grammar/pattern_matchers/possessive_context_matcher.rb +75 -0
  273. data/lib/kotoshu/grammar/pattern_matchers/possessive_contraction_matcher.rb +87 -0
  274. data/lib/kotoshu/grammar/pattern_matchers/sentence_start_matcher.rb +84 -0
  275. data/lib/kotoshu/grammar/pattern_matchers/vowel_sound_matcher.rb +81 -0
  276. data/lib/kotoshu/grammar/pattern_matchers/word_list_matcher.rb +66 -0
  277. data/lib/kotoshu/grammar/rule.rb +99 -0
  278. data/lib/kotoshu/grammar/rule_engine.rb +96 -0
  279. data/lib/kotoshu/grammar/rule_loader.rb +30 -0
  280. data/lib/kotoshu/grammar.rb +24 -0
  281. data/lib/kotoshu/integrity/audit_log.rb +167 -0
  282. data/lib/kotoshu/integrity/manifest.rb +116 -0
  283. data/lib/kotoshu/integrity/net_http.rb +51 -0
  284. data/lib/kotoshu/integrity/rotation_policy.rb +88 -0
  285. data/lib/kotoshu/integrity.rb +26 -0
  286. data/lib/kotoshu/jekyll.rb +101 -0
  287. data/lib/kotoshu/keyboard/layout.rb +116 -0
  288. data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
  289. data/lib/kotoshu/keyboard/layouts/azerty.rb +55 -0
  290. data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
  291. data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
  292. data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
  293. data/lib/kotoshu/keyboard/layouts/dvorak.rb +54 -0
  294. data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
  295. data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
  296. data/lib/kotoshu/keyboard/layouts/hrsl.rb +94 -0
  297. data/lib/kotoshu/keyboard/layouts/jcuken.rb +57 -0
  298. data/lib/kotoshu/keyboard/layouts/latin.rb +266 -0
  299. data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
  300. data/lib/kotoshu/keyboard/layouts/qwerty.rb +52 -0
  301. data/lib/kotoshu/keyboard/layouts/qwertz.rb +55 -0
  302. data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
  303. data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
  304. data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
  305. data/lib/kotoshu/keyboard/registry.rb +201 -0
  306. data/lib/kotoshu/keyboard.rb +96 -0
  307. data/lib/kotoshu/kotoshu_native.so +0 -0
  308. data/lib/kotoshu/language/detector.rb +242 -0
  309. data/lib/kotoshu/language/identifier.rb +376 -0
  310. data/lib/kotoshu/language/languages/base.rb +271 -0
  311. data/lib/kotoshu/language/lid_detector.rb +188 -0
  312. data/lib/kotoshu/language/normalizer/arabic.rb +113 -0
  313. data/lib/kotoshu/language/normalizer/base.rb +133 -0
  314. data/lib/kotoshu/language/normalizer/greek.rb +63 -0
  315. data/lib/kotoshu/language/normalizer/hebrew.rb +72 -0
  316. data/lib/kotoshu/language/normalizer/persian.rb +36 -0
  317. data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
  318. data/lib/kotoshu/language/registry.rb +209 -0
  319. data/lib/kotoshu/language/resources/ar/common_words.txt +6753 -0
  320. data/lib/kotoshu/language/resources/ar/confusion_sets.txt +11 -0
  321. data/lib/kotoshu/language/resources/de/common_words.txt +10003 -0
  322. data/lib/kotoshu/language/resources/de/confusion_sets.txt +246 -0
  323. data/lib/kotoshu/language/resources/en/common_words.txt +9979 -0
  324. data/lib/kotoshu/language/resources/en/confusion_sets.txt +871 -0
  325. data/lib/kotoshu/language/resources/es/common_words.txt +9992 -0
  326. data/lib/kotoshu/language/resources/es/confusion_sets.txt +17 -0
  327. data/lib/kotoshu/language/resources/fr/common_words.txt +9993 -0
  328. data/lib/kotoshu/language/resources/fr/confusion_sets.txt +76 -0
  329. data/lib/kotoshu/language/resources/pt/common_words.txt +9977 -0
  330. data/lib/kotoshu/language/resources/pt/confusion_sets.txt +18 -0
  331. data/lib/kotoshu/language/resources/ru/common_words.txt +9951 -0
  332. data/lib/kotoshu/language/resources/ru/confusion_sets.txt +5 -0
  333. data/lib/kotoshu/language/script.rb +128 -0
  334. data/lib/kotoshu/language/segmenter.rb +122 -0
  335. data/lib/kotoshu/language/suika.rb +38 -0
  336. data/lib/kotoshu/language/tokenizer/base.rb +187 -0
  337. data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
  338. data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
  339. data/lib/kotoshu/language/tokenizer/french_tokenizer.rb +172 -0
  340. data/lib/kotoshu/language/tokenizer/german_tokenizer.rb +41 -0
  341. data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
  342. data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
  343. data/lib/kotoshu/language/tokenizer/japanese_tokenizer.rb +55 -0
  344. data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +153 -0
  345. data/lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb +160 -0
  346. data/lib/kotoshu/language/tokenizer/russian_tokenizer.rb +93 -0
  347. data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
  348. data/lib/kotoshu/language/tokenizer/spanish_tokenizer.rb +122 -0
  349. data/lib/kotoshu/language.rb +127 -0
  350. data/lib/kotoshu/languages/ar/language.rb +98 -0
  351. data/lib/kotoshu/languages/bg/language.rb +37 -0
  352. data/lib/kotoshu/languages/ca/language.rb +22 -0
  353. data/lib/kotoshu/languages/cs/language.rb +22 -0
  354. data/lib/kotoshu/languages/da/language.rb +22 -0
  355. data/lib/kotoshu/languages/de/language.rb +554 -0
  356. data/lib/kotoshu/languages/el/language.rb +50 -0
  357. data/lib/kotoshu/languages/en/language.rb +470 -0
  358. data/lib/kotoshu/languages/es/language.rb +476 -0
  359. data/lib/kotoshu/languages/et/language.rb +24 -0
  360. data/lib/kotoshu/languages/fa/language.rb +91 -0
  361. data/lib/kotoshu/languages/fr/language.rb +505 -0
  362. data/lib/kotoshu/languages/he/language.rb +98 -0
  363. data/lib/kotoshu/languages/hr/language.rb +24 -0
  364. data/lib/kotoshu/languages/hu/language.rb +22 -0
  365. data/lib/kotoshu/languages/id/language.rb +25 -0
  366. data/lib/kotoshu/languages/it/language.rb +22 -0
  367. data/lib/kotoshu/languages/ja/language.rb +476 -0
  368. data/lib/kotoshu/languages/ko/language.rb +56 -0
  369. data/lib/kotoshu/languages/latin_base.rb +53 -0
  370. data/lib/kotoshu/languages/lt/language.rb +24 -0
  371. data/lib/kotoshu/languages/lv/language.rb +24 -0
  372. data/lib/kotoshu/languages/nb/language.rb +25 -0
  373. data/lib/kotoshu/languages/ne/language.rb +60 -0
  374. data/lib/kotoshu/languages/nl/language.rb +22 -0
  375. data/lib/kotoshu/languages/nn/language.rb +26 -0
  376. data/lib/kotoshu/languages/pl/language.rb +22 -0
  377. data/lib/kotoshu/languages/pt/language.rb +439 -0
  378. data/lib/kotoshu/languages/ro/language.rb +22 -0
  379. data/lib/kotoshu/languages/ru/language.rb +420 -0
  380. data/lib/kotoshu/languages/sk/language.rb +25 -0
  381. data/lib/kotoshu/languages/sl/language.rb +25 -0
  382. data/lib/kotoshu/languages/sr/language.rb +39 -0
  383. data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
  384. data/lib/kotoshu/languages/sv/language.rb +22 -0
  385. data/lib/kotoshu/languages/tr/language.rb +36 -0
  386. data/lib/kotoshu/languages/uk/language.rb +38 -0
  387. data/lib/kotoshu/languages/vi/language.rb +22 -0
  388. data/lib/kotoshu/languages.rb +64 -0
  389. data/lib/kotoshu/metrics_collector.rb +238 -0
  390. data/lib/kotoshu/metrics_module.rb +121 -0
  391. data/lib/kotoshu/models/context.rb +119 -0
  392. data/lib/kotoshu/models/embedding_model.rb +181 -0
  393. data/lib/kotoshu/models/fasttext_model.rb +212 -0
  394. data/lib/kotoshu/models/nearest_neighbor.rb +96 -0
  395. data/lib/kotoshu/models/onnx_model.rb +330 -0
  396. data/lib/kotoshu/models/semantic_error.rb +204 -0
  397. data/lib/kotoshu/models/suggestion.rb +106 -0
  398. data/lib/kotoshu/models/word_embedding.rb +97 -0
  399. data/lib/kotoshu/models.rb +27 -0
  400. data/lib/kotoshu/multi_language_checker.rb +127 -0
  401. data/lib/kotoshu/native.rb +83 -0
  402. data/lib/kotoshu/native_backend.rb +131 -0
  403. data/lib/kotoshu/paths.rb +79 -0
  404. data/lib/kotoshu/personal_dictionary.rb +105 -0
  405. data/lib/kotoshu/plugins/plugin.rb +61 -0
  406. data/lib/kotoshu/plugins/registry.rb +118 -0
  407. data/lib/kotoshu/plugins.rb +9 -0
  408. data/lib/kotoshu/project_config.rb +76 -0
  409. data/lib/kotoshu/readers/aff_data.rb +530 -0
  410. data/lib/kotoshu/readers/aff_reader.rb +393 -0
  411. data/lib/kotoshu/readers/condition_checker.rb +125 -0
  412. data/lib/kotoshu/readers/dic_reader.rb +208 -0
  413. data/lib/kotoshu/readers/file_reader.rb +377 -0
  414. data/lib/kotoshu/readers/lookup_builder.rb +584 -0
  415. data/lib/kotoshu/readers/readers.rb +17 -0
  416. data/lib/kotoshu/readers.rb +25 -0
  417. data/lib/kotoshu/resource_bundle.rb +55 -0
  418. data/lib/kotoshu/resource_manager.rb +600 -0
  419. data/lib/kotoshu/results/result.rb +165 -0
  420. data/lib/kotoshu/results.rb +8 -0
  421. data/lib/kotoshu/rspec.rb +163 -0
  422. data/lib/kotoshu/scripts/fasttext_to_onnx.py +275 -0
  423. data/lib/kotoshu/source_registry.rb +104 -0
  424. data/lib/kotoshu/spellchecker/parallel_checker.rb +90 -0
  425. data/lib/kotoshu/spellchecker.rb +406 -0
  426. data/lib/kotoshu/string_metrics.rb +153 -0
  427. data/lib/kotoshu/suggestions/context.rb +43 -0
  428. data/lib/kotoshu/suggestions/frequency_provider.rb +80 -0
  429. data/lib/kotoshu/suggestions/generator.rb +169 -0
  430. data/lib/kotoshu/suggestions/pipeline.rb +135 -0
  431. data/lib/kotoshu/suggestions/semantic_cascade.rb +73 -0
  432. data/lib/kotoshu/suggestions/strategies/base_strategy.rb +315 -0
  433. data/lib/kotoshu/suggestions/strategies/composite_strategy.rb +197 -0
  434. data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +631 -0
  435. data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +207 -0
  436. data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +165 -0
  437. data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +284 -0
  438. data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +317 -0
  439. data/lib/kotoshu/suggestions/strategies/symspell_strategy.rb +270 -0
  440. data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
  441. data/lib/kotoshu/suggestions/suggestion.rb +154 -0
  442. data/lib/kotoshu/suggestions/suggestion_set.rb +270 -0
  443. data/lib/kotoshu/suggestions/sweep_index.rb +137 -0
  444. data/lib/kotoshu/suggestions.rb +29 -0
  445. data/lib/kotoshu/tasks/check_task.rb +104 -0
  446. data/lib/kotoshu/tasks.rb +11 -0
  447. data/lib/kotoshu/validators/spelling_validator.rb +101 -0
  448. data/lib/kotoshu/validators.rb +9 -0
  449. data/lib/kotoshu/version.rb +5 -0
  450. data/lib/kotoshu.rb +538 -0
  451. data/rakelib/compile.rake +10 -0
  452. data/script/validate_all_dictionaries.rb +444 -0
  453. data/scripts/build_native_gem.rb +45 -0
  454. data/scripts/extract_eval_grids.rb +37 -0
  455. data/scripts/generate_frozen_tiers.rb +44 -0
  456. data/sig/kotoshu.rbs +435 -0
  457. data/test_oop.rb +79 -0
  458. metadata +546 -0
@@ -0,0 +1,169 @@
1
+ # 66 — kotoshu-rs: the Rust core (parsanol blueprint)
2
+
3
+ Implements Pillar 3 of [65-universal-kotoshu.md](65-universal-kotoshu.md).
4
+ **The approach is copied wholesale from `parsanol-rs` + `parsanol-ruby`**
5
+ (Ribose, in production) because it demonstrably works: one pure-Rust
6
+ core, all FFI feature-gated *inside* the core, per-language packages as
7
+ thin cdylib shims, pure-Ruby fallback retained, dual-backend test suite.
8
+
9
+ ## Decision: `ort`, not `onnxruntime`
10
+
11
+ - **`ort`** (pykeio/ort, v2.x): actively maintained, tracks current
12
+ onnxruntime, execution providers, and — decisive for us — a
13
+ **`load-dynamic`** mode that dlopens a host-provided
14
+ libonnxruntime instead of linking one in.
15
+ - The `onnxruntime` crate (nbigaouette) is effectively unmaintained and
16
+ pinned to old runtime versions. Rejected.
17
+ - **Policy**: inference is host-injected via an `EmbeddingProvider`
18
+ trait (rerank math in the core works on vectors the host supplies).
19
+ The core's own provider is `feature = "onnx"` →
20
+ `ort = { version = "2", default-features = false, features =
21
+ ["load-dynamic"] }`, so Ruby/Python/Node hosts share the lib they
22
+ already ship (the `onnxruntime` gem / pip / onnxruntime-node).
23
+ Standalone Rust builds (CLI, LSP binary) enable ort's bundled
24
+ binaries instead. Minimum onnxruntime 1.x documented per release.
25
+
26
+ ### Addendum (2026-09-02): runtime validation + `onnx-ir` assessment
27
+
28
+ - **ort under real load** — Weirich, "Rust, ORT, ONNX: Real-Time YOLO on
29
+ a Live Webcam" (Nov 2025): YOLOv11 at 30 FPS webcam inference in Rust
30
+ via a four-thread pipeline (capture → resize/pack → inference →
31
+ render), ~22% CPU on a MacBook; same expected on a Jetson with
32
+ hardware acceleration. Our workload (single-token embedding lookups,
33
+ rerank of ≤25 candidates) is orders of magnitude lighter — ort
34
+ headroom is not a risk. The article's pipeline shape is also the
35
+ reference pattern for the LSP/server: inference isolated on its own
36
+ worker thread, results serialized back; the editor loop never blocks
37
+ on a model call. ORT's WebAssembly support is confirmed for the
38
+ browser path.
39
+ - **`onnx-ir`** (tracel-ai, Burn ecosystem) is a pure-Rust ONNX
40
+ **parser → IR**, not a runtime: executing the graph means burn-import
41
+ compiling it to a Burn backend — and therefore **zero native
42
+ dependencies**. That matters exactly where `load-dynamic` cannot
43
+ work: static musl CLI binaries and wasm32. FastText-style graphs
44
+ (embedding gather + matmul + normalization) sit comfortably inside
45
+ Burn's operator coverage.
46
+ - **Policy consequence — the trait already anticipated this.** `ort`
47
+ stays THE provider (decided above). At P3, if ort's wasm story proves
48
+ heavy, evaluate a `burn` provider (`onnx-ir`/burn-import) behind the
49
+ same `EmbeddingProvider` trait, gated to `wasm32`/static targets
50
+ only. `tract` (sonos — pure-Rust inference, no C deps) is the noted
51
+ third option if Burn's op coverage ever falls short. One trait,
52
+ three candidate providers, no core changes.
53
+
54
+ ## Workspace layout (mirrors parsanol-rs)
55
+
56
+ ```
57
+ kotoshu-rs/
58
+ ├── Cargo.toml # workspace: lto="fat", codegen-units=1
59
+ ├── deny.toml typos.toml release-plz.toml .pre-commit-config.yaml
60
+ ├── .cargo/config.toml # [net] git-fetch-with-cli; wasm32 getrandom flags
61
+ ├── kotoshu/ # core crate, crate-type = ["rlib"]
62
+ │ └── src/
63
+ │ ├── lib.rs # pub mod ffi; pub use ffi::ruby as ruby_ffi
64
+ │ ├── portable/ # portability layer (parsanol pattern)
65
+ │ ├── dict/ # aff/dic parsing, affix expansion, DAFSA/trie
66
+ │ ├── suggest/ # edit-distance(banded), phonetic, keyboard, ngram
67
+ │ ├── rerank.rs # pure vector math + EmbeddingProvider trait
68
+ │ ├── resource/ # manifest parse, sha256, tier metadata
69
+ │ ├── tokenize.rs
70
+ │ └── ffi/
71
+ │ ├── mod.rs # feature gates + re-exports
72
+ │ ├── shared.rs # batch serialization shared by ALL bindings
73
+ │ ├── c.rs # C ABI — always available (parsanol: c/ always)
74
+ │ ├── ruby/ # feature = "ruby" → magnus
75
+ │ └── wasm/ # feature = "wasm" → wasm-bindgen, js-sys,
76
+ │ # console_error_panic_hook
77
+ ├── benches/ examples/ tests/ # criterion, proptest, conformance-vector runner
78
+ └── .github/workflows/
79
+ ├── _rust-setup.yml _rust-build-test.yml ci.yml
80
+ ├── release.yml # release-plz → crates.io
81
+ ├── release-binary.yml # kotoshu CLI / kotoshu-lsp single binaries
82
+ ├── ruby-ffi.yml # builds core with feature "ruby", runs magnus tests
83
+ ├── wasm.yml # wasm-pack build → publishes @kotoshu/wasm
84
+ └── docs.yml
85
+ ```
86
+
87
+ **Features**: `default = []`, `ruby = ["magnus"]`,
88
+ `wasm = ["wasm-bindgen", "js-sys", "console_error_panic_hook"]`,
89
+ `onnx = ["ort/load-dynamic"]`, `parallel = ["rayon"]`, `logging = ["log"]`.
90
+ Core profile `opt-level 3, lto true, codegen-units 1`; dep crates
91
+ `opt-level "s"` — parsanol's exact settings.
92
+
93
+ ## Language packages (mirrors parsanol-ruby)
94
+
95
+ | Package | Shim | Notes |
96
+ |---|---|---|
97
+ | `kotoshu` gem | `ext/kotoshu_native` cdylib: `#[magnus::init] fn init(r) { kotoshu::ffi::ruby::init(r) }` | gemspec: `spec.extensions`, runtime dep `rb_sys "~> 0.9"`, files include `Cargo.toml`+`Cargo.lock`, reject `*.so|dylib|bundle|dll`; versions independently of the core (parsanol: gem 1.3.x over core 0.5.x) |
98
+ | `kotoshu-python` | PyO3 shim crate in-repo, maturin wheels | over the same core; HTTP client remains the pure-Python path |
99
+ | `kotoshu-js` | none — re-exports `@kotoshu/wasm` | wasm package built+published from kotoshu-rs (`wasm.yml`); napi-rs native later only if wasm latency hurts |
100
+ | Rust CLI/LSP | direct dep | `release-binary.yml` ships static binaries (linux amd64/arm64, macOS universal, windows x64/arm64) |
101
+
102
+ **extconf.rb (copied verbatim in spirit)**: `require "rb_sys/mkmf"`;
103
+ `create_rust_makefile("kotoshu/kotoshu_native")` with
104
+ `r.profile = ENV.fetch("RB_SYS_CARGO_PROFILE", :dev).to_sym`,
105
+ `r.use_stable_api_compiled_fallback = true`,
106
+ `r.force_install_rust_toolchain = false`. Source gem compiles on
107
+ install — **no cross-compile matrix to maintain**; prebuilt platform
108
+ gems (rake-compiler-dock) only if adoption demands (gate, post-1.0).
109
+
110
+ **Ruby loader/fallback**: `Kotoshu::Native.available?` guard; every
111
+ public API keeps its pure-Ruby path; the ext is a pure accelerator —
112
+ identical results, faster. Env `KOTOSHU_BACKEND=native|ruby|auto`
113
+ selects explicitly for debugging/benching.
114
+
115
+ **Dual-backend conformance**: parsanol's `compat:` namespace, applied to
116
+ the suite we already own —
117
+ `rake compat:ruby` / `compat:native` / `compat:compare` run the gem's
118
+ full rspec suite (Splylls/Hunspell fixtures included) against each
119
+ backend and diff the outputs. CI runs both. Golden vectors from the
120
+ same fixtures are exported into `kotoshu/conformance` and consumed by
121
+ kotoshu-rs CI so the C ABI, wasm build, and future ports are held to
122
+ identical outputs — **one source of truth, three enforcement points.**
123
+
124
+ ## Porting order (each step lands with conformance green)
125
+
126
+ 1. **P0 scaffold**: workspace, `ffi/shared.rs` batch format, C ABI
127
+ skeleton, conformance-vector runner, CI (incl. `ruby-ffi.yml`,
128
+ `wasm.yml`), release-plz wired.
129
+ 2. **P1 dictionary**: aff parser → dic loader → affix expansion →
130
+ lookup (`correct?`). Vectors: the gem's affix/lookup specs.
131
+ 3. **P2 suggest**: banded edit-distance with threshold (port the gem's
132
+ optimized DP), phonetic, keyboard proximity, n-gram, composite
133
+ ranking, frequency bonus. Vectors: suggestion specs (ranked lists).
134
+ 4. **P3 rerank + resources**: `EmbeddingProvider` trait + ort provider
135
+ (load-dynamic); manifest/sha256/tier parse. Vectors: rerank specs.
136
+ 5. **P4 shims**: gem ext (0.9), Python wheel, wasm publish, single-binary
137
+ LSP/CLI.
138
+
139
+ ## Parsanol details adopted as policy
140
+
141
+ - `[patch.crates-io]` git-revs for magnus/magnus-macros/rb-sys when
142
+ crates.io lags Ruby (current parsanol revs noted; relax on release).
143
+ - `ffi/shared.rs` batch FFI format — one serialization, all bindings;
144
+ parsanol measured 3–5× over object-by-object FFI.
145
+ - Quality kit: cargo-deny (`deny.toml`), typos, pre-commit, criterion
146
+ benches, proptest, `rust-version` floor per crate.
147
+ - Core repo tests the Ruby feature itself (`ruby-ffi.yml`) so binding
148
+ breakage is caught in the core's CI, not downstream.
149
+ - Separate version lines per package; release-plz automates the core's.
150
+
151
+ ## Acceptance
152
+
153
+ - `kotoshu-rs` passes the full conformance-vector pack on the C ABI,
154
+ with `--features ruby` under MRI 3.2/3.3/3.4/4.0-head, and as wasm32.
155
+ - `rake compat:compare` in the gem shows zero diffs between backends.
156
+ - `gem install kotoshu` works WITHOUT a Rust toolchain (pure Ruby) and
157
+ WITH one (native build), same results either way.
158
+ - `@kotoshu/wasm` loads in Node 18+ and browsers; playground runs on it.
159
+
160
+ ## Risks
161
+
162
+ Two engines during port (mitigated: dual-backend suite gates every
163
+ merge); Rust toolchain requirement for the native path (mitigated:
164
+ fallback + prebuilt gate); magnus/rb-sys rev churn (patch policy);
165
+ ort dynamic-lib skew (documented minimum, C API stable across 1.x).
166
+
167
+ ## Status
168
+
169
+ _Planning._ Promotes into P0–P4 execution plans when cut 0.7 opens.
@@ -0,0 +1,137 @@
1
+ # 67 — Publishing the Stack: Tiered Models, kotoshu-rs, Access Libraries
2
+
3
+ The delivery train for [65-universal-kotoshu.md](65-universal-kotoshu.md):
4
+ turns plans 66 (Rust core), the models-repo plans 06/07 (tiers,
5
+ registry, releases) into a sequenced rollout of **what gets published,
6
+ from which repo, in what order**. Execution-level: each milestone has
7
+ one owner repo, a concrete deliverable, and acceptance.
8
+
9
+ ## The train
10
+
11
+ ```
12
+ M0 spec freeze ─► M1 gem tier client ─► M2 models release ─┐
13
+ (registry v1) (owner │
14
+ tags) ├─► M5 shims
15
+ M3 kotoshu-rs P0 (bootstrap) ─► M4 P1–P2 port ────────────┘ (ruby/wasm/python)
16
+ │
17
+ ▼
18
+ M6 polish: platform gems gate,
19
+ signed bundles, site/docs
20
+ ```
21
+
22
+ M0–M2 (models + gem) and M3–M4 (Rust) are **parallel tracks**; they
23
+ join at M5.
24
+
25
+ ### M0 — Resource Spec v1 freeze
26
+
27
+ - Deliverable: `schemas/registry.schema.json` merged in
28
+ `models-fasttext-onnx` (plan 07). The registry shape is a cross-repo
29
+ API — freeze it before both the gem client and kotoshu-rs parse it.
30
+ - Acceptance: schema merged; a sample registry validates; id scheme
31
+ `kotoshu://models/{lang}/{tier}` locked.
32
+
33
+ ### M1 — Gem: tier-aware ModelCache (PR to this repo)
34
+
35
+ - `Kotoshu.setup(:en, want: :model, tier: :fluency|:mini|:full)`;
36
+ `ModelCache` resolves tier from the registry; fallback chain
37
+ full→fluency→mini only when explicitly requested.
38
+ - Config: `model_tier` option (SCHEMA entry → env `KOTOSHU_MODEL_TIER`
39
+ automatic), CLI `kotoshu setup LANG --model --tier mini`.
40
+ - Resolve stays cache-only; a missing tier raises
41
+ `ResourceNotSetupError` exactly like today. No implicit downloads.
42
+ - Acceptance: specs for tier resolution + fallback; existing specs
43
+ untouched (default tier = owner decision, see gates below).
44
+
45
+ ### M2 — First models release (owner action required)
46
+
47
+ - Plan 07 workflow builds the release; **the tag name is the owner's
48
+ decision** — the workflow refuses to invent versions. Default pin in
49
+ the gem follows only after the owner picks.
50
+
51
+ ### M3 — kotoshu-rs bootstrap (new repo `kotoshu/kotoshu-rs`, P0 of plan 66)
52
+
53
+ - Repo creation under the org (owner or delegated `gh repo create`).
54
+ - Scaffold per plan 66: workspace (lto fat / codegen-units 1),
55
+ `kotoshu` core crate with `ffi/{mod,shared,c}.rs`, features
56
+ `default=[] ruby wasm onnx parallel logging`, deny/typos/pre-commit,
57
+ CI trio (`ci.yml`, `ruby-ffi.yml`, `wasm.yml`), release-plz config.
58
+ - **Conformance export first**: `rake kotoshu:conformance:export` in
59
+ this repo writes golden JSONL (fixtures + suggestion outputs) to
60
+ `conformance/`; kotoshu-rs CI consumes it — one source of truth,
61
+ three enforcement points (C ABI, ruby feature, wasm).
62
+ - Acceptance: green CI incl. `--features ruby` under one MRI and a
63
+ wasm32 build; C ABI round-trips the batch format on exported vectors.
64
+
65
+ ### M4 — Port P1–P2: dictionary + suggest
66
+
67
+ - aff/dic parsing → affix expansion → lookup (P1); banded
68
+ edit-distance, phonetic, keyboard, n-gram, composite ranking,
69
+ frequency bonus (P2). Each lands with conformance green against the
70
+ golden vectors — no "port now, verify later".
71
+ - Acceptance: full conformance pack passes on the C ABI; criterion
72
+ benchmarks published in CI artifacts.
73
+
74
+ ### M5 — Access libraries (the publish wave)
75
+
76
+ Order = blast-radius ascending, value descending:
77
+
78
+ 1. **Ruby** — `ext/kotoshu_native` in this repo (thin magnus shim per
79
+ plan 66), `KOTOSHU_BACKEND=native|ruby|auto`, `rake compat:ruby|
80
+ native|compare` in CI. The gem keeps installing pure-Ruby with no
81
+ toolchain. PR-gated like every change to this repo.
82
+ 2. **JS/WASM** — `@kotoshu/wasm` built+published from kotoshu-rs
83
+ (`wasm.yml`, wasm-pack); `kotoshu-js` re-exports it alongside the
84
+ HTTP client. **npm publish needs the org's npm credentials —
85
+ currently blocked** (same blocker as kotoshu-js itself).
86
+ 3. **Python** — maturin wheel over the same core in `kotoshu-python`,
87
+ optional extra alongside the HTTP client. **PyPI publish needs the
88
+ org's token — currently blocked (403)**.
89
+ 4. **Go** — stays on the HTTP path (kotoshu-server); cgo over the C
90
+ ABI is a deliberate non-goal until demand appears.
91
+
92
+ - Acceptance per shim: conformance vectors pass through the binding;
93
+ perf report vs pure-Ruby/HTTP path; version lines independent
94
+ (parsanol policy).
95
+
96
+ ### M6 — Polish
97
+
98
+ - Prebuilt platform gems via rake-compiler-dock **only if** adoption
99
+ demands (gate, post-adoption — plan 66 policy).
100
+ - Signed offline bundles (plan 65 pillar: minisign/cosign over the
101
+ registry + assets) — signature verification in gem and kotoshu-rs.
102
+ - Site: tier badges on language pages, `docs/caching` tier section,
103
+ playground option to run `@kotoshu/wasm` locally once published.
104
+
105
+ ## Owner-decision gates (never taken by tooling)
106
+
107
+ | Decision | Milestone |
108
+ |---|---|
109
+ | Registry/tier **default tier** (bandwidth vs accuracy) | M1 — **DECIDED 2026-09-04: `fluency`** (owner instruction: light/medium default; legacy caches bridged) |
110
+ | Models **release tag & versions** | M2, every release |
111
+ | kotoshu-rs **crate versions, MSRV**, release-plz first release | M3+ |
112
+ | Ruby ext gem version; python/js package versions | M5 |
113
+ | License attribution wording (CC BY-SA inheritance) | M2 |
114
+ | gem required_ruby_version changes (if any) | never tooling's call |
115
+
116
+ ## Publishing blockers that exist today
117
+
118
+ - PyPI token invalid/absent; npm org auth absent. The M5 publish wave
119
+ for python/js cannot complete until the owner provides them — flag
120
+ early, publish the code (repos, CI) regardless.
121
+
122
+ ## Acceptance (whole train)
123
+
124
+ - `gem install kotoshu` → `kotoshu setup en` pulls the fluency tier by
125
+ default (or the owner's chosen default), verified by sha256 from the
126
+ registry, cache-only afterwards.
127
+ - `kotoshu-rs` passes the conformance pack on C ABI, `--features ruby`
128
+ under MRI 3.2–4.0-head, and wasm32.
129
+ - `rake compat:compare` shows zero diffs between Ruby and native
130
+ backends.
131
+ - `@kotoshu/wasm` loads in Node 18+ and browsers; python wheel
132
+ installs without a Rust toolchain (or is clearly marked source-build).
133
+
134
+ ## Status
135
+
136
+ _Planning._ Milestones promote to individual execution plans
137
+ (66 P0–P4 for M3–M4; per-shim plans for M5) as each opens.
@@ -0,0 +1,152 @@
1
+ # 68 — SOTA Techniques: Research Adoption Plan
2
+
3
+ Mandate: survey 2023–2026 ML/AI and spellcheck research (arxiv, ACL,
4
+ lab reports) and the current frontier model lines, and decide what
5
+ Kotoshu adopts, adapts, or rejects. Research conducted 2026-09-02; the
6
+ full sourced brief lives in this plan's "Findings" section. Companion
7
+ execution context: [65](65-universal-kotoshu.md) (pillars),
8
+ [66](66-kotoshu-core.md) (Rust core, `ort`), [67](67-kotoshu-rs-and-access-libraries.md)
9
+ (delivery train); models-repo plans 06/07 (tiers, registry) already
10
+ shipped eval-gated tiers.
11
+
12
+ ## Findings (compressed; URLs are the evidence)
13
+
14
+ ### Spelling / GEC state of the art
15
+
16
+ | Approach | Class | BEA-2019 F0.5 | Note |
17
+ |---|---|---|---|
18
+ | GECToR-style tagging + distill | 100–250M | ~74–75 | Still beats prompting on precision-weighted benches — small supervised > LLM prompting for correction ([NLP-progress](http://nlpprogress.com/english/grammatical_error_correction.html), [Grammarly GECToR](https://www.grammarly.com/blog/engineering/experimenting-with-gector/)) |
19
+ | T5/BART seq2seq | 250M–3B | ~73 | Linear-decoding tricks close the gap ([gec-papers](https://github.com/Chunngai/gec-papers)) |
20
+ | Prompted GPT-4-class | API-scale | high-50s–60s | Win on JFLEG *fluency*, lose on precision ([OpenReview](https://openreview.net/forum?id=yAMayChodt)) |
21
+ | Fine-tuned 1–8B LLM | edge-LLM | closing | Overcorrection risk ([arxiv 2509.20811](https://arxiv.org/html/2509.20811v1)) |
22
+ | Masked-LM real-word ranking | small | ~85% acc | The closest published twin to our pipeline: dictionary candidates + neural ranker, with **training-data construction called "the crucial component"** ([arxiv 2410.23514](https://arxiv.org/html/2410.23514v1)) |
23
+
24
+ LLM-as-GEC-evaluator is now standard practice
25
+ ([BEA 2024](https://aclanthology.org/2024.bea-1.6/)) — relevant to our
26
+ eval harness, not our runtime.
27
+
28
+ ### Embeddings / tiering (directly about our shipped tiers)
29
+
30
+ - **Matryoshka / 2D-MRL**: plain truncation of non-MRL embeddings is
31
+ safe until deep reduction; MRL training only pays beyond that
32
+ ([arxiv 2205.13147](https://arxiv.org/abs/2205.13147),
33
+ [arxiv 2605.16608](https://arxiv.org/html/2605.16608v2)). Our tiers
34
+ already are 2D truncation (vocab axis prefix + kept dims) —
35
+ independently validated by 2D-MRL work
36
+ ([arxiv 2410.13230](https://arxiv.org/html/2410.13230v1),
37
+ [HF guide](https://huggingface.co/blog/matryoshka)). **A single
38
+ frequency-ordered file with declared truncation points can replace
39
+ three artifacts with no retraining.**
40
+ - **Quantization**: int8 near-lossless (matches our 0.9999 rank_corr);
41
+ int4 with group-wise scales (≤128) still beats ANN-approximation
42
+ error ([arxiv 2501.10534](https://arxiv.org/html/2501.10534v1)).
43
+ Full 120 MB → int4 ≈ 15–20 MB near-lossless.
44
+ - **Static embeddings**: Model2Vec/Potion — distilled static
45
+ embeddings, ~30 MB, 500× faster, beats other statics on MTEB
46
+ ([MinishLab](https://github.com/MinishLab/model2vec),
47
+ [fastText comparison](https://minish.ai/blog/2025-07-28-fasttext/)).
48
+ Word-keyed like ours — same OOV gap.
49
+ - **OOV (our biggest functional gap — models are word-keyed)**:
50
+ fastText hashed char 3–6-gram buckets remain the standard answer;
51
+ collision-aware hash embeddings improve them
52
+ ([arxiv 1709.03933](https://arxiv.org/abs/1709.03933)).
53
+
54
+ ### Named model lines — verified, techniques extracted
55
+
56
+ - **Claude Fable 5 / 5.1** — VERIFIED
57
+ ([announce](https://www.anthropic.com/news/claude-fable-5-mythos-5),
58
+ [5.1](https://www.anthropic.com/claude-fable-and-mythos-5-1),
59
+ [Willison](https://simonwillison.net/2026/Jun/9/claude-fable-5/));
60
+ "Fable 5.0" is not a named release. Techniques: same-weights
61
+ multi-safeguard releases, 5-level reasoning-effort control, cheap
62
+ cache reads. **Relevance: low** — effort-tiering is already our
63
+ tier/backend model.
64
+ - **GLM 5.1/5.2/5.3/5.3-Flash** — VERIFIED
65
+ ([tech report](https://arxiv.org/html/2602.15763v1), z.ai blogs).
66
+ Key adoptable idea from 5.3: **RL on synthesized verifiable
67
+ environments with binary rewards** — all gains from post-training on
68
+ the same base. Spelling correction is verifiable by construction
69
+ (output == reference), making it an ideal small-scale RL domain.
70
+ - **DeepSeek V4 Pro/Flash** — VERIFIED
71
+ ([report](https://arxiv.org/html/2606.19348v1)). Adoptable:
72
+ specialist→student **on-policy distillation (reverse KL)** — the
73
+ exact recipe for distilling a big reranker into our tier sizes;
74
+ FP4/QAT embedding-table practice. MoE/CSA/1M-context: scale-mismatched.
75
+ - **Hy4** — IDENTIFIED as Tencent Hunyuan Hy4 preview (770B/49B MoE,
76
+ Apache-2.0, [HF](https://huggingface.co/tencent/Hy4-preview)).
77
+ **Relevance: low** — no training details published.
78
+
79
+ ## Adoption matrix
80
+
81
+ ### Adopt now (Ruby today, no new deps)
82
+
83
+ | # | What | Where it lands | Why now |
84
+ |---|---|---|---|
85
+ | A1 | **Multilingual keyboard-aware typo generator** (MULTYPO-style) for the eval harness — our current `run_eval.py` typos are ASCII-lowercase-only, nearly meaningless for de/ru/ja/zh | models repo, `eval/` | Biggest correctness hole in our own gates; pure code; keyboard layouts already exist in the gem ([MULTYPO](https://github.com/cisnlp/multypo)) |
86
+ | A2 | **GitHub Typo Corpus** as a real-world multilingual eval set (350k+ edits, 15+ languages) alongside the synthetic proxy | models repo, `eval/corpora/` (download-once, cache) | Grounds the proxy metric in real human typos ([corpus](https://github.com/mhagiwara/github-typo-corpus), [LREC paper](https://aclanthology.org/2020.lrec-1.835/)) |
87
+ | A3 | **Confidence cascade, formalized**: dictionary/affix verdict first; ONNX rerank only for uncertain candidates; document thresholds | gem, semantic path (plan 05) | Published hybrid twin does exactly this; we half-do it today ([arxiv 2410.23514](https://arxiv.org/html/2410.23514v1)) |
88
+ | A4 | **Single nested (2D-Matryoshka) artifact**: one frequency-ordered file + declared truncation points replacing mini/fluency downloads; separate files stay available | models repo, post-07 experiment | Truncation safety is literature-validated; one download serves all tiers ([arxiv 2605.16608](https://arxiv.org/html/2605.16608v2)) |
89
+
90
+ ### Adopt with kotoshu-rs (ort / Rust core)
91
+
92
+ | # | What | Lands in | Why then |
93
+ |---|---|---|---|
94
+ | B1 | **int4 group-128 quantized full tier** (120 MB → ~15–20 MB near-lossless); possibly a `nano` tier | kotoshu-rs P3 + models repo tiers | Needs the core's own dequant math / ort support; near-lossless per [2501.10534](https://arxiv.org/html/2501.10534v1) |
95
+ | B2 | **Hashed char n-gram OOV fallback** (fastText buckets or collision-aware hashing) so unseen words still embed | kotoshu-rs P3; converter in models repo | Fixes the word-keyed OOV gap at last ([1709.03933](https://arxiv.org/abs/1709.03933)) |
96
+ | B3 | **Tiny cross-encoder reranker option** (~22M distilled MiniLM class, ONNX) behind the provider trait | kotoshu-rs P3+ | Better rerank than cosine bi-encoder for hard cases; our ≤25-candidate scale keeps it cheap ([sbert efficiency](https://sbert.net/docs/cross_encoder/usage/efficiency.html)) |
97
+
98
+ ### Adopt later (deliberate, data-gated)
99
+
100
+ | # | What | Gate |
101
+ |---|---|---|
102
+ | C1 | **Model2Vec/Potion static embeddings** as reranker base (~30 MB, 500× faster) | per-model license verified for redistribution; win over fastText shown on our eval |
103
+ | C2 | **RL with binary verifiable rewards** to train reranker weights (GLM-5.3 recipe, scaled down ~10⁶) | after conformance vectors exist (67 M3) so reward is measurable; needs a training pipeline owner |
104
+ | C3 | **On-policy distillation** of a large teacher into tier-sized rerankers (DeepSeek V4 recipe) | same gate as C2 |
105
+ | C4 | **LLM-as-evaluator** for eval reports (offline, not CI) | optional; proxy metrics suffice until corpora land |
106
+ | C5 | **Optional external LLM mode** (1–8B fine-tuned GEC or API) as an explicit opt-in "heavy" backend | user demand; never default (latency/RAM violate our constraints) |
107
+
108
+ ### Reject
109
+
110
+ - **Shipping any LLM as the default engine** — violates small/offline/CPU.
111
+ - **MoE, MLA/CSA attention, 1M-context techniques** — meaningful only at
112
+ 10⁹+ parameter scale; a 300-dim embedding table has nothing to sparse-ify.
113
+ - **Distillation-prevention / watermarking / cache-economics patterns**
114
+ from the frontier labs — business-side, not engine-side.
115
+
116
+ ## Near-term execution (promotes from this plan)
117
+
118
+ 1. **A1 typo generator** — extend `eval/` with per-language keyboard
119
+ noise models driven by the gem's existing layout data; regenerate
120
+ all 18 reports; gates unchanged (they will get *harder*, honestly).
121
+ 2. **A2 corpus benchmark** — fetch script + cache + report format;
122
+ license (MIT) recorded in the registry-style metadata.
123
+ 3. **A4 nested-artifact experiment** — build `fasttext.{lang}.matryoshka.onnx`
124
+ (full matrix, frequency-ordered, truncation manifest), eval
125
+ truncated views against the shipped tiers; adopt only if equal or
126
+ better; registry ids unchanged.
127
+ 4. Record B1/B2 as named milestones inside plan 66 P3 (no separate
128
+ plan files until the core exists).
129
+
130
+ ## Owner gates
131
+
132
+ New tier names (`nano`), changing the default tier, any new model
133
+ licenses (Model2Vec per-model, GitHub Typo Corpus MIT), and anything
134
+ requiring network in eval CI — all owner decisions.
135
+
136
+ ## Status
137
+
138
+ **Adopted 2026-09-02.** Items A1–A4 promote to models-repo execution;
139
+ B1–B3 fold into plan 66 P3; C/D items remain gated. Research brief
140
+ preserved above for provenance.
141
+
142
+ **B1 evaluated (2026-09-03, models PR #11): DO NOT SHIP.** int4-group128
143
+ hits 16.2 MB (7.4x under full) and clears rank_corr on all 9 languages,
144
+ but top1_agreement fails the 0.95 fluency gate on 8 of 9 (ru only
145
+ passer) - 4-bit noise flips 1-2 typo probes per language. Implementation
146
+ verified exact (ORT round-trip 0.0), so the format itself loses;
147
+ int8-per-row stays. The levers were then exhausted (models PR #12): half-away rounding is
148
+ a provable no-op, group-size shrink helps monotonically on ja but
149
+ asymptotes below the gate (en pinned at 0.90), and the mixed
150
+ int4-mini fallback fails on de. B1 closed - int8-per-row is the
151
+ final answer. The third eval-driven rejection of the campaign (after
152
+ SVD dim-reduction and Model2Vec).
@@ -0,0 +1,74 @@
1
+ # 69 — Keyboard-Aware Typo Eval + Real-Typo Corpus Benchmark
2
+
3
+ Executes the "adopt now" items A1 + A2 of
4
+ [68-sota-adoption.md](68-sota-adoption.md). Lands in
5
+ `models-fasttext-onnx` (`eval/`), feeding the same gate machinery as
6
+ plan 06.
7
+
8
+ ## Goal
9
+
10
+ 1. Replace the eval harness's ASCII-lowercase-only typo generator with
11
+ a **keyboard-layout-aware multilingual noise model** (MULTYPO-style,
12
+ [repo](https://github.com/cisnlp/multypo),
13
+ [arxiv 2005.01158](https://arxiv.org/html/2005.01158v1)):
14
+ QWERTY/QWERTZ/AZERTY/JCUKEN neighbor-weighted slips for
15
+ de/en/es/fr/pt/ru, umlaut/eszett keys for de, Cyrillic neighbors for
16
+ ru; for ja/ko/zh a visual/phonetic character-confusion model
17
+ (homoglyph + stroke-similar substitutions) replaces key-adjacency.
18
+ All deterministic (seeded), in-vocab typos only (the metric requires
19
+ both models to embed the typo).
20
+ 2. Add the **GitHub Typo Corpus**
21
+ ([repo](https://github.com/mhagiwara/github-typo-corpus),
22
+ [LREC 2020](https://aclanthology.org/2020.lrec-1.835/); 350k+ real
23
+ human typo corrections, 15+ languages, MIT) as a real-world
24
+ embedding-side benchmark: for corpus (typo → correction) pairs in
25
+ our 9 languages where both words are in-vocab, measure how the
26
+ cosine-to-typo ranking places the correction, per tier. Report-only
27
+ (no gate) in v1 — it measures the full correction pipeline only
28
+ after kotoshu-rs conformance vectors exist (plan 67 M3).
29
+
30
+ ## Tasks
31
+
32
+ 1. `eval/noise.py` — per-language deterministic typo generator:
33
+ layout-driven for Latin/Cyrillic scripts (the gem's own
34
+ `keyboard/layouts` are the layout source of truth — mirror them),
35
+ character-confusion for CJK. Same RNG discipline as `run_eval.py`
36
+ (`default_rng([seed, crc32(lang)])`).
37
+ 2. Wire into `run_eval.top1_agreement_metric` as the typo source;
38
+ **gates unchanged** — they get honestly harder. Ladders climb if a
39
+ language now misses (record attempts as before).
40
+ 3. `scripts/fetch_corpus.py` — download-once the GitHub Typo Corpus
41
+ release into `eval/corpora/` (gitignored), verify checksum, extract
42
+ per-language pairs.
43
+ 4. `eval/corpus_bench.py` — the benchmark runner →
44
+ `eval/reports/corpus.{lang}.json` (committed): pairs evaluated,
45
+ correction-in-top-1/5/20 rates per tier vs full.
46
+ 5. Regenerate all 18 tier reports with the new typo model; commit the
47
+ new reports + report the gate deltas in the PR.
48
+
49
+ ## Acceptance
50
+
51
+ - `run_eval.py` produces identical results for identical inputs
52
+ (determinism test); non-Latin languages get realistic typos
53
+ (spot-check list in the PR).
54
+ - All 9×2 tier reports regenerated; any ladder climbs recorded.
55
+ - `corpus_bench.py` runs offline after `fetch_corpus.py`; reports
56
+ committed; corpus itself never committed (size + license header
57
+ attribution in the report JSON).
58
+
59
+ ## Dependencies
60
+
61
+ - Feeds plan 06's gate machinery; blocks nothing.
62
+ - Owner gate: none (corpus is MIT; no new model distribution).
63
+
64
+ ## Status
65
+
66
+ **Implemented (2026-09-03, models PR #5 merged).** eval/noise.py ships
67
+ layout-neighbor slips (QWERTY/QWERTZ/AZERTY/JCUKEN mirrored from the
68
+ gem layouts, diacritics included), Hangul jamo decomposition and
69
+ curated CJK confusion sets, transposition fallback; deterministic. All
70
+ 18 reports regenerated under the harder typos - gates pass, no ladder
71
+ climbs; de/fluency corrected to 50k (the 60k was small-n noise).
72
+ GitHub Typo Corpus bench live (official S3 dead; Wayback snapshot with
73
+ self-pinned sha256); top-5 full-tier baselines 0.13 (ja) - 0.60 (ko).
74
+ Part of release v1.0.1.
@@ -0,0 +1,59 @@
1
+ # 70 — Confidence Cascade in the Semantic Path
2
+
3
+ Executes adopt-now item A3 of
4
+ [68-sota-adoption.md](68-sota-adoption.md) in THIS repo (the gem),
5
+ following the published hybrid dictionary+neural architecture
6
+ ([arxiv 2410.23514](https://arxiv.org/html/2410.23514v1): dictionary
7
+ first, neural rerank only for uncertain candidates — and their finding
8
+ that the confidence boundary is where the quality lives).
9
+
10
+ ## Goal
11
+
12
+ Formalize what the semantic path already half-does: the ONNX rerank
13
+ runs **only when the traditional composite strategy is not already
14
+ confident**, with an explicit, configurable threshold — cutting model
15
+ loads and latency on the happy path without changing results where the
16
+ dictionary is certain.
17
+
18
+ ## Design
19
+
20
+ 1. `Configuration::SCHEMA` gains `semantic_cascade_threshold`
21
+ (float, default `1.0` — preserving today's always-rerank behavior;
22
+ ENV `KOTOSHU_SEMANTIC_CASCADE_THRESHOLD` automatic via the schema).
23
+ 2. In `Suggestions::Generator` (or `SemanticStrategy`), when the
24
+ composite strategy's top candidate confidence ≥ threshold, skip the
25
+ semantic rerank for that suggestion set; log the skip under debug
26
+ metrics.
27
+ 3. `docs/` note + README section: what the threshold means, how to
28
+ tune it (0.0 = never rerank, 1.0 = always), and that measured
29
+ guidance comes from the eval harness once plan 69's corpus bench
30
+ lands.
31
+ 4. Spec coverage: behavior with threshold 1.0 (identical to today),
32
+ 0.0, and a mid value — real model instances per the global no-doubles
33
+ rule; onnx-dependent specs tagged `:onnx` per the existing opt-in
34
+ convention.
35
+
36
+ ## Acceptance
37
+
38
+ - Default behavior is byte-identical to today (existing suite green).
39
+ - A mid-threshold spec proves the skip happens exactly when confidence
40
+ ≥ threshold, and never otherwise.
41
+ - `kotoshu check --verbose` surfaces cascade-skip counts.
42
+
43
+ ## Dependencies
44
+
45
+ - Independent of 69/71; tuning guidance arrives from 69's corpus bench.
46
+ - Performance numbers for the README: measure rerank skip rate on the
47
+ existing example corpus at thresholds {1.0, 0.9, 0.8}.
48
+
49
+ ## Status
50
+
51
+ **Implemented (2026-09-03, gem PR #98 merged).** semantic_cascade_
52
+ threshold in the SCHEMA (env automatic), Suggestions::SemanticCascade
53
+ decision object with the 1.0 never-skip sentinel (composite confidence
54
+ legitimately reaches exactly 1.0 - documented in ALWAYS_RERANK),
55
+ semantic_cascade_skips counters in metrics + verbose. Drive-by fixes:
56
+ dead convert_schema_value Integer/Float branches (ENV never converted),
57
+ missing Metrics::Collector / Debug::Logger autoloads. 97 examples
58
+ green, rubocop clean. Tuning guidance pending plan 69 corpus numbers
59
+ now committed in the models repo.
@@ -0,0 +1,63 @@
1
+ # 71 — Nested (2D-Matryoshka) Single-Artifact Experiment
2
+
3
+ Executes adopt-now item A4 of
4
+ [68-sota-adoption.md](68-sota-adoption.md) in `models-fasttext-onnx`.
5
+ Literature: plain prefix truncation of embeddings is safe at our
6
+ ratios ([arxiv 2605.16608](https://arxiv.org/html/2605.16608v2));
7
+ 2D-MRL nesting along vocab+dims is validated practice
8
+ ([arxiv 2410.13230](https://arxiv.org/html/2410.13230v1),
9
+ [HF Matryoshka guide](https://huggingface.co/blog/matryoshka)).
10
+
11
+ ## Goal
12
+
13
+ One int8 file per language, frequency-ordered, whose **row-prefix
14
+ views ARE the tiers**: a `fasttext.{lang}.nested.onnx` (100k × 300d,
15
+ int8-per-row, ~30 MB) from which `fluency` = first-50k view (60k de)
16
+ and `mini` = first-10k view. If the views eval identically to the
17
+ shipped separate tiers, consumers download one artifact per language
18
+ instead of up to three, and tier "upgrades" are free (the bytes are
19
+ already on disk).
20
+
21
+ ## Tasks
22
+
23
+ 1. `scripts/build_nested.py` — build the nested artifact from the
24
+ full model (same int8-per-row math as `build_tiers.py`; import, do
25
+ not duplicate). Emit a truncation manifest (per-language prefix
26
+ lengths) into the artifact's metadata_props and a sibling
27
+ `fasttext.{lang}.nested.json`.
28
+ 2. `eval/` view-eval: run the existing gate metrics for the fluency
29
+ and mini views of the nested file against the shipped separate
30
+ tier files — assert metric parity (same seeds, same probe
31
+ sequences ⇒ identical numbers, since the construction is
32
+ identical; any delta is a bug, not noise).
33
+ 3. Size/bandwidth report: per language, {3 separate files} vs {full +
34
+ nested}; which consumers win under which access pattern.
35
+ 4. **Recommendation, not adoption**: registry ids and the release
36
+ workflow stay unchanged in this plan. Adoption = a follow-up
37
+ registry v2 field (`view_of`) + gem/kotoshu-rs resolver support,
38
+ proposed only if the parity + size report supports it.
39
+
40
+ ## Acceptance
41
+
42
+ - Nested views eval **identically** (bit-level metric parity) to the
43
+ shipped tiers for all 9 languages.
44
+ - Size report committed (`eval/reports/nested.{lang}.json` summary).
45
+ - No change to registry.json, manifest.json, or release.yml in this
46
+ plan.
47
+
48
+ ## Dependencies
49
+
50
+ - After 69 if report regeneration is in flight (same eval files) —
51
+ sequence to avoid collisions; otherwise independent.
52
+ - Owner gate: adoption itself (new registry field + default artifact
53
+ strategy) is an owner decision.
54
+
55
+ ## Status
56
+
57
+ **Experiment complete (2026-09-03, models PR #6 merged).** build_
58
+ nested.py + check_nested_parity.py; 18/18 EXACT array + metric parity
59
+ against the keyboard-aware reports. Recommendation recorded in
60
+ nested.summary.json: separate tiers stay the default (mini-only users
61
+ would pay ~10x); nested ships as an optional all-in-one if adopted.
62
+ Adoption (registry view_of field + resolver support) remains an owner
63
+ decision - deliberately not taken.