kotoshu 0.9.2 → 0.10.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (48) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +53 -0
  3. data/Cargo.lock +72 -4
  4. data/TODO.impl/00-vision.md +37 -0
  5. data/TODO.impl/100-full-feature-batch3.md +28 -0
  6. data/TODO.impl/101-int4-tier-experiment.md +32 -0
  7. data/TODO.impl/102-lid-chain.md +28 -0
  8. data/TODO.impl/103-oov-bucket-tables.md +27 -0
  9. data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
  10. data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
  11. data/TODO.impl/106-native-lid-detection.md +36 -0
  12. data/TODO.impl/99-site-truth-performance-docs.md +31 -0
  13. data/ext/kotoshu_native/Cargo.toml +5 -2
  14. data/ext/kotoshu_native/src/lib.rs +134 -6
  15. data/lib/kotoshu/cache/language_cache.rb +4 -1
  16. data/lib/kotoshu/cache/model_cache.rb +150 -2
  17. data/lib/kotoshu/cli.rb +8 -0
  18. data/lib/kotoshu/configuration.rb +20 -2
  19. data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
  20. data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
  21. data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
  22. data/lib/kotoshu/keyboard/layouts/hrsl.rb +78 -0
  23. data/lib/kotoshu/keyboard/layouts/latin.rb +47 -1
  24. data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
  25. data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
  26. data/lib/kotoshu/keyboard/registry.rb +20 -4
  27. data/lib/kotoshu/keyboard.rb +9 -0
  28. data/lib/kotoshu/language/lid_detector.rb +188 -0
  29. data/lib/kotoshu/language.rb +4 -0
  30. data/lib/kotoshu/languages/ar/language.rb +14 -0
  31. data/lib/kotoshu/languages/bg/language.rb +37 -0
  32. data/lib/kotoshu/languages/et/language.rb +24 -0
  33. data/lib/kotoshu/languages/fa/language.rb +16 -0
  34. data/lib/kotoshu/languages/he/language.rb +15 -0
  35. data/lib/kotoshu/languages/hr/language.rb +24 -0
  36. data/lib/kotoshu/languages/id/language.rb +25 -0
  37. data/lib/kotoshu/languages/lt/language.rb +24 -0
  38. data/lib/kotoshu/languages/lv/language.rb +24 -0
  39. data/lib/kotoshu/languages/sk/language.rb +25 -0
  40. data/lib/kotoshu/languages/sl/language.rb +25 -0
  41. data/lib/kotoshu/languages/sr/language.rb +38 -0
  42. data/lib/kotoshu/languages.rb +12 -0
  43. data/lib/kotoshu/spellchecker.rb +35 -0
  44. data/lib/kotoshu/version.rb +1 -1
  45. data/lib/kotoshu.rb +45 -5
  46. data/scripts/extract_eval_grids.rb +4 -1
  47. data/sig/kotoshu.rbs +45 -0
  48. metadata +25 -1
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 44cb12b37d8bf72fa6febd503bdcd713aa569d64e4c648f02c422e4d362ab891
4
- data.tar.gz: b42724adb7b58ca46c5494770cff1fb6c821b68272a545cd24e1f1ac152b3799
3
+ metadata.gz: a0093eb0f194f55beb4950a039c37e4b3abe6b33d073e7d7a8de0958e22a32cc
4
+ data.tar.gz: 795382646b0e0fdee0010b76d8fcd9afefa55447f513494ec3e832fe225f6110
5
5
  SHA512:
6
- metadata.gz: 17486ce07cb10da6c26bfd643aa7fc0d7d2ec7fb28cb8a406a6e3eb0a3612931a1db33badc577624e9e566902ab3e3f4a104a1ef10c174b7629651c3aeb11522
7
- data.tar.gz: 66f0dba2a4a727d700ba8c4c7d386380c2d77380b2b8cd7f4943dae377bff1db751bd1ca7f5dcb2075a65139a453ccdf4395c2331874d1b2e783fcd7400be53f
6
+ metadata.gz: 59fb8d99d938013b349ca779593a72b2fa8390cf725950c07acd9e117fde7ff138710fb9dfa1ba3d7581947d261dea0f990f555e1456421156316c746e8d66a8
7
+ data.tar.gz: 18c2713e916171510e45028f08968d443248f5426c614e56ab21a32078543bdad69dd7f6aed4d44ced5ee00f7d18554f9050f19e84a3fb936798d3846bb00838
data/CHANGELOG.md CHANGED
@@ -7,7 +7,60 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
7
7
 
8
8
  ## [Unreleased]
9
9
 
10
+ ### Added
11
+ - **Personal dictionary in the check path** (plan 105) -
12
+ `Spellchecker#check` (and therefore `Kotoshu.check`,
13
+ `Kotoshu.check_file`, and `kotoshu check`) no longer flags words
14
+ present in the user personal dictionary
15
+ (~/.config/kotoshu/personal.dic, KOTOSHU_PERSONAL_DIC override),
16
+ matching the semantics the LSP already applies to diagnostics:
17
+ case-insensitive, filtered at result assembly without suppression
18
+ metadata, words still counted, and `suggest`/`correct?` untouched so
19
+ personal words stay spellcheckable. The dictionary is loaded once
20
+ per process (no hot reload; the LSP keeps its mtime-based reload).
21
+ Opt out with `Kotoshu.configure { |c| c.personal_dictionary = false }`
22
+ / KOTOSHU_PERSONAL_DICTIONARY=false / `kotoshu check --no-personal`.
23
+ - **Native language detection** (plan 106) - `Kotoshu.detect_language(text)`
24
+ now returns a `Kotoshu::Language::Detection` (`code` + `score`) scored by
25
+ the Rust extension over the `kotoshu://models/lid/lid-176` registry
26
+ artifact pair (the ONNX-converted lid.176 model, 176 languages,
27
+ bit-faithful to the fastText bindings - parity 55/55 against the frozen
28
+ corpus shared with kotoshu-rs). Fetch the pair once with
29
+ `Kotoshu.setup_lid` (registry-resolved, sha-verified, cached under
30
+ `models-cache/lid/`); detect resolves cache-only and never downloads.
31
+ The 7-language `Language::Detector` heuristic stays the fallback when
32
+ the extension is absent, the model is not set up, or
33
+ `KOTOSHU_BACKEND=ruby` opts out - so pure-Ruby installs are unaffected
34
+ (no new gemspec dependency, no onnxruntime). The magnus shim
35
+ (ext/kotoshu_native) gains `Kotoshu::Native::LidModel.load` +
36
+ `#detect` behind the kotoshu-rs `model` feature; the workspace
37
+ Cargo.lock moves to the kotoshu-rs revision carrying the reader.
38
+ `Kotoshu.detect_language_with_confidence` keeps its `[code, score]`
39
+ shape and now routes through the same engine.
40
+ - **Full-feature language batch 3** (plan 100) - twelve languages gain the
41
+ downloadable spelling dictionary, a keyboard layout, and a gem module:
42
+ `ar` `id` `fa` `he` `bg` `sr` `hr` `sk` `sl` `lt` `lv` `et` (the RTL
43
+ ar/fa/he modules are promoted from tokenizer-only wiring; the other nine
44
+ are new thin modules - id hr sl sk et lt lv on `LatinBase`, bg sr with
45
+ the Cyrillic tokenizer). Six national key grids join the layout registry
46
+ (Arabic 101, Persian standard ISIRI 9147, Hebrew SI-1452, Bulgarian BDS,
47
+ Serbian Cyrillic, and the shared Croatian/Slovenian QWERTZ), all mirrored
48
+ from the models repo eval harness and drift-checked against it; five new
49
+ Latin family members (Indonesian Slovak Estonian Lithuanian Latvian)
50
+ type on the US QWERTY grid like the eval model. Bulgarian and Serbian
51
+ register ahead of JCUKEN so their Cyrillic grids win the lookup. The
52
+ eval-grid drift fixture now covers all nine national grids. Specimen
53
+ misspelling -> correction pairs for every language are engine-verified
54
+ against the staged dictionaries in a `:network` spec.
55
+
10
56
  ### Fixed
57
+ - **RTL word extraction on the check path** - the plan-91 script-aware
58
+ word regex never reached the Arabic, Persian and Hebrew tokenizers, so
59
+ `Kotoshu.check` / `kotoshu check -l ar|fa|he` extracted zero words and
60
+ silently passed every RTL document. The three tokenizers now declare
61
+ their script sets (`\p{Arabic}`, `\p{Arabic}` + ZWNJ for Persian
62
+ compounds like می‌روم, `\p{Hebrew}` + geresh/gershayim), and RTL
63
+ misspellings surface with ranked suggestions.
11
64
  - **Suggestion candidate sweep missed transpositions, substitutions, and
12
65
  dictionary forms** - `EditDistanceStrategy` only scored raw dictionary
13
66
  stems, so valid dictionary forms could never be suggested no matter how
data/Cargo.lock CHANGED
@@ -26,7 +26,7 @@ dependencies = [
26
26
  "regex",
27
27
  "rustc-hash",
28
28
  "shlex",
29
- "syn",
29
+ "syn 2.0.119",
30
30
  ]
31
31
 
32
32
  [[package]]
@@ -82,12 +82,20 @@ dependencies = [
82
82
  "either",
83
83
  ]
84
84
 
85
+ [[package]]
86
+ name = "itoa"
87
+ version = "1.0.18"
88
+ source = "registry+https://github.com/rust-lang/crates.io-index"
89
+ checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682"
90
+
85
91
  [[package]]
86
92
  name = "kotoshu"
87
93
  version = "0.1.0"
88
- source = "git+https://github.com/kotoshu/kotoshu-rs?branch=main#9941f1ea1a6157c57659e428f475959508d4d0b3"
94
+ source = "git+https://github.com/kotoshu/kotoshu-rs?branch=main#e574c445df5f6eee8e978f1f4d26dc48cfb1947b"
89
95
  dependencies = [
90
96
  "magnus",
97
+ "serde",
98
+ "serde_json",
91
99
  ]
92
100
 
93
101
  [[package]]
@@ -141,7 +149,7 @@ checksum = "47607461fd8e1513cb4f2076c197d8092d921a1ea75bd08af97398f593751892"
141
149
  dependencies = [
142
150
  "proc-macro2",
143
151
  "quote",
144
- "syn",
152
+ "syn 2.0.119",
145
153
  ]
146
154
 
147
155
  [[package]]
@@ -205,7 +213,7 @@ dependencies = [
205
213
  "quote",
206
214
  "regex",
207
215
  "shell-words",
208
- "syn",
216
+ "syn 2.0.119",
209
217
  ]
210
218
 
211
219
  [[package]]
@@ -255,6 +263,49 @@ version = "0.3.6"
255
263
  source = "registry+https://github.com/rust-lang/crates.io-index"
256
264
  checksum = "1bc711410fbe7399f390ca1c3b60ad0f53f80e95c5eb935e52268a0e2cd49acc"
257
265
 
266
+ [[package]]
267
+ name = "serde"
268
+ version = "1.0.229"
269
+ source = "registry+https://github.com/rust-lang/crates.io-index"
270
+ checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba"
271
+ dependencies = [
272
+ "serde_core",
273
+ "serde_derive",
274
+ ]
275
+
276
+ [[package]]
277
+ name = "serde_core"
278
+ version = "1.0.229"
279
+ source = "registry+https://github.com/rust-lang/crates.io-index"
280
+ checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48"
281
+ dependencies = [
282
+ "serde_derive",
283
+ ]
284
+
285
+ [[package]]
286
+ name = "serde_derive"
287
+ version = "1.0.229"
288
+ source = "registry+https://github.com/rust-lang/crates.io-index"
289
+ checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348"
290
+ dependencies = [
291
+ "proc-macro2",
292
+ "quote",
293
+ "syn 3.0.5",
294
+ ]
295
+
296
+ [[package]]
297
+ name = "serde_json"
298
+ version = "1.0.151"
299
+ source = "registry+https://github.com/rust-lang/crates.io-index"
300
+ checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14"
301
+ dependencies = [
302
+ "itoa",
303
+ "memchr",
304
+ "serde",
305
+ "serde_core",
306
+ "zmij",
307
+ ]
308
+
258
309
  [[package]]
259
310
  name = "shell-words"
260
311
  version = "1.1.1"
@@ -278,6 +329,17 @@ dependencies = [
278
329
  "unicode-ident",
279
330
  ]
280
331
 
332
+ [[package]]
333
+ name = "syn"
334
+ version = "3.0.5"
335
+ source = "registry+https://github.com/rust-lang/crates.io-index"
336
+ checksum = "12df2e0110f65b775f769bb17ef989067a1d931b2eb822bd4346631eeada89f9"
337
+ dependencies = [
338
+ "proc-macro2",
339
+ "quote",
340
+ "unicode-ident",
341
+ ]
342
+
281
343
  [[package]]
282
344
  name = "unicode-ident"
283
345
  version = "1.0.24"
@@ -289,3 +351,9 @@ name = "windows-link"
289
351
  version = "0.2.1"
290
352
  source = "registry+https://github.com/rust-lang/crates.io-index"
291
353
  checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"
354
+
355
+ [[package]]
356
+ name = "zmij"
357
+ version = "1.0.23"
358
+ source = "registry+https://github.com/rust-lang/crates.io-index"
359
+ checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b"
@@ -188,6 +188,28 @@ convertible model pool exhausted at 55 languages; owner-side
188
188
  remnants (PyPI registrations, marketplace PAT, nds/fi calls) recorded
189
189
  in the plans, not pending work.
190
190
 
191
+ Plans 99-104 are **wave 6, the live-quality follow-through**
192
+ (2026-09-07), written after the sweep-performance arc (gem 0.9.1 ->
193
+ 0.9.2, @kotoshu/wasm 0.3.0 -> 0.3.2) and a fresh state audit: 55 model
194
+ languages x 3 tiers confirmed (ja/ko/zh rerank-only by design), no LID
195
+ resource mirrored, lsp/server `~> 0.6` constraints already resolving
196
+ current engines, site docs carrying stale version references.
197
+
198
+ | # | Plan | Gap it closes |
199
+ |---|---|---|
200
+ | 99 | [Site truth + performance docs](99-site-truth-performance-docs.md) | Docs misstate current versions; the measured per-language latency is unpublished |
201
+ | 100 | [Full-feature batch 3](100-full-feature-batch3.md) | ~35 languages have dictionaries + models but no gem module - the big-audience remainder (ar id fa he ...) |
202
+ | 101 | [int4 tier experiment](101-int4-tier-experiment.md) | Halving tier bytes was named (B1) but never run through the gates |
203
+ | 102 | [LID chain](102-lid-chain.md) | No detection in the registry, wasm, or playground - users pick the language blind |
204
+ | 103 | [OOV bucket tables](103-oov-bucket-tables.md) | Model candidates miss n-grams absent from vocab (the Teh gap's model half) |
205
+ | 104 | [VS Code marketplace readiness](104-vscode-marketplace-readiness.md) | The extension is built but not installable; everything short of the owner PAT |
206
+
207
+ Execution notes: plan 103 is sequenced after 101 (same repo, one
208
+ worktree at a time); all site-facing flips (99, 100, 102's playground
209
+ half, 104's link line) land through one coordinated site pass to keep
210
+ the repo conflict-free; lsp/server 0.1.2 bumps are deliberately NOT
211
+ planned - the `~> 0.6` floor already resolves 0.9.2 for fresh installs.
212
+
191
213
  Dependency order across the ecosystem track:
192
214
 
193
215
  ```
@@ -225,3 +247,18 @@ plan name the source.
225
247
  | `docs/ARCHITECTURE_IMPROVEMENTS_PLAN.md` (5 phases) | Phase 1 perf items in `01`/`05`; Phase 2 architecture in `00-vision` (this doc); Phase 3 UX mostly implemented (`PersonalDictionary`, `ProjectConfig`, `FluentChecker` exist); Phase 4 quality in `10`; Phase 5 polish spread |
226
248
  | `docs/TDD_ITERATION_STRATEGY.md` | `10` (test methodology + CI); project-rule overrides (no doubles, behavior-not-implementation) in `CLAUDE.md` |
227
249
  | `docs/ONNX_FUNCTIONAL_VERIFICATION.md` | `05` (semantic path productionization) |
250
+
251
+
252
+ Plans 105-107 plus models 09 and site 11 are **wave 7, detection and
253
+ coverage depth** (2026-09-07): after the wave-6 releases (gem 0.9.3,
254
+ wasm 0.4.0, registry v1.4.0, action v2), the remaining gaps are the
255
+ personal dictionary never reaching the check path, detection stuck on
256
+ a 7-language heuristic, buckets only for en/de, and the playground not
257
+ yet passing bucket bytes to loadModel.
258
+
259
+ | # | Plan | Gap it closes |
260
+ |---|---|---|
261
+ | 105 | [Personal dictionary in check](105-personal-dictionary-in-check.md) | kotoshu personal add then check still flags the word |
262
+ | 106 | [Native LID detection](106-native-lid-detection.md) | detect is a 7-language heuristic; the pure-Rust lid reader exists unused |
263
+ | 107 | models [09 bucket expansion](../../models-fasttext-onnx/TODO.impl/09-bucket-table-expansion.md) | bucket OOV siblings exist only for en and de |
264
+ | — | site 11 playground buckets | wasm 0.4.0 accepts bucketsBytes; the worker never fetches the sibling |
@@ -0,0 +1,28 @@
1
+ # Plan 100 — Full-feature language batch 3: the highest-audience remainder
2
+
3
+ ## Why
4
+ 20 languages are full-feature (gem module + keyboard layout + verified
5
+ specimens); the dictionaries manifest and the model registry carry ~35 more
6
+ with both resources. Every remaining language without a module leaves
7
+ speakers on the degraded path. The gaps are the big-audience ones: ar (~400M),
8
+ id (~200M), fa (~80M), he, bg, sr, hr, sk, sl, and more.
9
+
10
+ ## Work (gem repo only — the site flip rides a follow-up)
11
+ 1. Enumerate: languages present in BOTH the dictionaries manifest and the
12
+ models registry that lack a gem module. Rank by speaker count.
13
+ 2. For the top ~12 (expected: ar id fa he bg sr hr sk sl lt lv et — confirm
14
+ against data): create the language module (pattern of the 19 from PR
15
+ #123), keyboard layout (Arabic 101 for ar; Persian for fa; Hebrew for he;
16
+ Cyrillic/YUSI for bg/sr; QWERTZ/Slovak for the Latinate), AVAILABLE_LANGUAGES
17
+ extension, and specimen pairs verified against the STAGED dictionary via
18
+ the real engine (correct()/suggest() round trips) — the PR #123 discipline.
19
+ 3. RTL already wired (ar/id from batch 2); no tokenizer work needed.
20
+ 4. Suite green; rubocop clean.
21
+
22
+ ## Verification
23
+ One spec per module (specimen corrections engine-verified, layout smoke);
24
+ `kotoshu setup <lang>` + `check -l <lang>` exercised live for at least ar,
25
+ id, fa, he. Deliver the verified specimen table + sizes for the site flip.
26
+
27
+ ## Status
28
+ Executed 2026-09-07 — gem PR #151 merged. Twelve languages (ar id fa he bg sr hr sk sl lt lv et) full-feature; AVAILABLE_LANGUAGES 20→32; 49 specimens; RTL tokenizer fix for ar/fa/he. Site flip PR #27.
@@ -0,0 +1,32 @@
1
+ # Plan 101 — int4 tier experiment (models repo), eval-gated
2
+
3
+ ## Why
4
+ The tier recipe settled at int8 (rank_corr 0.9999, near-lossless) after SVD
5
+ failed its gates. int4 halves the bytes again — mini ~1.5 MB, fluency ~8 MB —
6
+ which changes browser adoption math for the wasm playground. It was named in
7
+ the original kotoshu-rs TODO (B1) but never run through the gates.
8
+
9
+ ## Work (models repo)
10
+ 1. Quantization pass: per-row int4 (packed nibbles) + fp16 scale, over the
11
+ same vocab cuts as fluency/mini.
12
+ 2. The existing eval gates unchanged and unweakened: rank correlation vs
13
+ full float32 on the keyboard-aware eval; accept only if rank_corr within
14
+ 0.001 of int8's and top-1 agreement holds on the probe-limited languages.
15
+ 3. If gates pass: registry v1.4.0 adding the int4 variants as NEW resources
16
+ (`{lang}/mini4`? or a field) — additive, never replacing; mirror with
17
+ ACAO:* like v1.2.1+. If gates fail: record the numbers as the verdict
18
+ (the SVD precedent — measured, not assumed).
19
+ 4. ONNX reader side (kotoshu-rs) only if the gates pass.
20
+
21
+ ## Verification
22
+ Gates table per language; byte sizes; the release registry byte-identical to
23
+ the committed one; no existing tier URLs change.
24
+
25
+ ## Status
26
+ Executed 2026-09-07 — REJECTED on data (models PR #17, merged). int4-per-row
27
+ (one fp16 scale, packed nibbles) over 8 languages: rank_corr deltas -0.0146
28
+ to -0.0199 vs int8 (15-20x outside the 0.001 window), top-1 fails on 8/8
29
+ fluency4 and 2/8 mini4. Sizes halve (mini4 1.52 MB, fluency4 7.60 MB) but
30
+ accuracy does not hold. With the earlier group-128/64/32 sweep the scale
31
+ ladder is measured end to end — int8-per-row stands final. Registry
32
+ untouched; evidence in models eval/reports/.
@@ -0,0 +1,28 @@
1
+ # Plan 102 — Language detection end to end: registry mirror, wasm, playground
2
+
3
+ ## Why
4
+ The gem detects document language (server /v1/detect) using its own detection
5
+ model — but the registry mirrors none of it, the wasm surface has no detect
6
+ API, and the playground makes a user pick a language dropdown blind. A
7
+ writer pasting text should be told what it is. The FastText LID model is
8
+ tiny (~1 MB) and the mirror pattern is proven (v1.2.1+).
9
+
10
+ ## Work
11
+ 1. Models repo: add the LID model as a registry resource (converted like the
12
+ tiers, int8-quantized if gates hold vs the float32 labels), mirrored with
13
+ ACAO:*; registry v1.4.x alongside plan 101 (coordinate the version).
14
+ 2. kotoshu-rs: `detect_language(bytes, vocab_bytes) -> {code, score}` on the
15
+ model surface beside loadModel/rerank; wasm export `detectLanguage`;
16
+ tests against the gem's detection outputs on a fixed multilingual corpus.
17
+ 3. Playground (rides the site follow-up with plans 99/100 flips): on load
18
+ and on demand ("detect" affordance beside the lang select), load the LID
19
+ model (~1 MB, cached like the semantic tier) and set/select the language,
20
+ showing the detected code + confidence. Never auto-switch a user's
21
+ explicit choice silently — propose it.
22
+
23
+ ## Verification
24
+ Registry mirror ACAO + size checks; wasm parity vs gem detect on the corpus;
25
+ playground verified with multilingual paste (en/de/ru/ja/ar).
26
+
27
+ ## Status
28
+ Executed 2026-09-07 (core) — models PR #18 (lid-176, registry v1.4.0) + kotoshu-rs PR #25 (loadLid/detectLanguage). Playground propose-UI still pending a wasm cut that ships the export.
@@ -0,0 +1,27 @@
1
+ # Plan 103 — OOV bucket-table artifacts: model-generated candidates for the last mile
2
+
3
+ ## Why
4
+ semanticSuggest embeds an OOV word via subword n-grams PRESENT IN THE VOCAB;
5
+ a typo whose n-grams never appear (Teh — only n-gram "teh") embeds nothing
6
+ and returns empty. fastText's real OOV path hashes n-grams into buckets the
7
+ training wrote to; exporting the bucket table closes that gap so the model,
8
+ not just the Damerau sweep, can reach the intended word. Recorded as the
9
+ honest gap in kotoshu-rs oov.rs docs and PR #18's report.
10
+
11
+ ## Work (models repo, then kotoshu-rs)
12
+ 1. Export the bucket-table rows the FastText training produced (hash bucket
13
+ → vector) for each tier alongside the vocab; size audit first — the
14
+ bucket table may exceed the tier itself; if so, gate to the top-K buckets
15
+ by usage and measure the quality delta honestly.
16
+ 2. kotoshu-rs: extend the int8 reader to consult bucket rows for n-grams
17
+ absent from vocab (feature-gated like `model`); semantic_neighbors then
18
+ covers bucket-backed OOV.
19
+ 3. Gates: existing rerank conformance must not move (frozen); new specs for
20
+ bucket-backed neighbors (Teh→the expected); eval on the rerank probe set.
21
+
22
+ ## Verification
23
+ Bucket-backed neighbor specs; size deltas recorded; rerank conformance 0
24
+ change; a 0.4.x wasm release only after both engines agree.
25
+
26
+ ## Status
27
+ Executed 2026-09-07 — models PR #20 (en/de buckets) + kotoshu-rs PR #26 (bucket-backed OOV). Teh→the via buckets. Release assets + wasm 0.4.0 pin still owner/coordinated cut. (sequenced AFTER plan 101 — same repo, one worktree at a time)
@@ -0,0 +1,32 @@
1
+ # Plan 104 — VS Code marketplace + Open VSX readiness
2
+
3
+ ## Why
4
+ The extension is built, CI-green, with a .vsix artifact — but it is not
5
+ installable without the marketplace listing, which is the single largest
6
+ adoption blocker for the editor audience. Publishing needs the owner's
7
+ publisher PAT (a credential we never hold); everything else can be ready
8
+ so publishing is one secret + one click.
9
+
10
+ ## Work (vscode repo)
11
+ 1. Open VSX publish workflow (release-ovsx.yml, secret-gated) beside the
12
+ existing CI — Open VSX serves VSCodium/Code-OSS users without the
13
+ Microsoft marketplace.
14
+ 2. Metadata polish: gallery banner/colors, keywords, categories, repository
15
+ links, the kotoshu-vscode README (quickstart with kotoshu-lsp install,
16
+ settings reference: log level, personal dictionary command).
17
+ 3. PREPUBLISH.md: exact owner steps — create publisher, add PAT as secret,
18
+ run the workflow; nothing else.
19
+ 4. Site follow-up line (rides plan 99's pass): /docs/clients/lsp gains the
20
+ marketplace link placeholder to fill on publish day.
21
+
22
+ ## Verification
23
+ `vsce package` clean with no warnings; workflow lint-valid; PREPUBLISH
24
+ steps verified against the actual GitHub UI labels.
25
+
26
+ ## Status
27
+ Executed 2026-09-07 — vscode PR #3, merged. release-marketplace.yml (VSCE_PAT)
28
+ + release-ovsx.yml (OPEN_VSX_TOKEN), both version-gated; PREPUBLISH.md with
29
+ verified owner steps; package.json publisher/categories/keywords/banner;
30
+ README quickstart against gem 0.1.1; vsce package warning-clean. Publishing
31
+ itself is one PAT secret + one tag — owner action. The /docs/clients/lsp
32
+ marketplace link rides the coordinated site pass.
@@ -0,0 +1,34 @@
1
+ # Plan 105 — The personal dictionary reaches the check path
2
+
3
+ ## Why
4
+ `kotoshu personal add recieve` then `kotoshu check` still flags the word —
5
+ PersonalDictionary is wired only into the `personal` subcommand and the LSP
6
+ (resolved server-side), not into `Spellchecker#check`. Users who add words
7
+ see them flagged again on the very next run; the README story ("add words
8
+ you accept") is currently false on the CLI/API path.
9
+
10
+ ## Work (gem repo)
11
+ 1. `Spellchecker#check` consults the personal dictionary: a word present in
12
+ personal.dic is not an error (mirror the LSP's semantics: filter at
13
+ result assembly, keep suppression metadata out of the error rows).
14
+ 2. Reload semantics: the LSP reloads on mtime change; the CLI/one-shot path
15
+ loads once per process. Do not add hot reload here.
16
+ 3. Opt-out: `Kotoshu.configure { personal_dictionary false }` (or the
17
+ existing config seam — find it) and `--no-personal` on `kotoshu check`.
18
+ 4. Specs: add-then-check round trip on a real dictionary; opt-out flags
19
+ again; personal words still spellcheckable via suggest.
20
+ 5. Docs truth: /docs/ignores gains the personal dictionary section (the
21
+ action PR #2 README noted this gap) — deliver the copy in the PR body
22
+ for the site follow-up, do not touch the site repo.
23
+
24
+ ## Verification
25
+ Suite green (3,900+ examples); rubocop clean; `kotoshu personal add x &&
26
+ kotoshu check` live round trip recorded in the PR body.
27
+
28
+ ## Status
29
+ Executed 2026-09-07 — gem PR (branch plan-105-personal-dict-in-check):
30
+ check-path filter mirroring the LSP, one load per process,
31
+ Configuration#personal_dictionary + KOTOSHU_PERSONAL_DICTIONARY +
32
+ `kotoshu check --no-personal`; suite pins KOTOSHU_PERSONAL_DIC for
33
+ hermeticity. /docs/ignores copy delivered in the PR body for the site
34
+ follow-up.
@@ -0,0 +1,36 @@
1
+ # Plan 106 — Document language detection through the pure-Rust LID reader
2
+
3
+ ## Why
4
+ The gem's LanguageIdentifier needs the Python fastText bindings (and breaks
5
+ under numpy 2), so /v1/detect ships a 7-language heuristic. kotoshu-rs now
6
+ carries a pure-Rust lid.176 reader (kotoshu/src/lid, parity 55/55 vs the
7
+ gem) behind the same `model` feature the native ext already builds.
8
+
9
+ ## Work (gem repo)
10
+ 1. Expose `Kotoshu.detect_language(text)` through the native backend:
11
+ download/resolve lid.176.onnx + vocab via the models registry
12
+ (kotoshu://models/lid/lid-176, v1.4.0), score with the ext, return
13
+ { code, score }. Heuristic Language::Detector stays the fallback when
14
+ the native ext is absent or the model is not set up.
15
+ 2. FFI seam: extend the magnus shim (ext/kotoshu_native) with the call —
16
+ mirror how suggest/check already cross; keep it behind the existing
17
+ feature so pure-Ruby installs are unaffected.
18
+ 3. Server follow-up (kotoshu-server repo, separate PR riding this):
19
+ /v1/detect prefers Kotoshu.detect_language when available. Note it in
20
+ the PR body; do not touch the server repo yourself.
21
+ 4. Specs: native-vs-fixture parity (reuse the rs corpus JSON shape),
22
+ fallback behavior, registry miss error.
23
+
24
+ ## Verification
25
+ Suite green; a live detect over en/de/ru/ja/ar samples recorded in the PR
26
+ body; pure-Ruby suite still green (no new hard dependency).
27
+
28
+ ## Status
29
+ Executed 2026-09-07 — gem PR (branch plan-106-native-lid-detection):
30
+ Kotoshu.detect_language -> Language::Detection through the ext
31
+ (LidModel.load/#detect in the magnus shim, kotoshu-rs model feature,
32
+ Cargo.lock bumped to the reader revision); ModelCache gained the
33
+ lid pair (download_lid/load_cached_lid, top-level lid/ dir);
34
+ Kotoshu.setup_lid two-stage; heuristic fallback incl.
35
+ KOTOSHU_BACKEND=ruby; parity 55/55 vs the shared frozen corpus.
36
+ Server follow-up (/v1/detect) noted in the PR body.
@@ -0,0 +1,31 @@
1
+ # Plan 99 — Site truth pass + measured performance documentation
2
+
3
+ ## Why
4
+ The docs were written against the versions live at the time; three releases
5
+ later some pages misstate reality. Users deserve the measured numbers the
6
+ perf work produced — no spellchecker site publishes per-language sweep
7
+ latency; we have it, verified, for eight languages.
8
+
9
+ ## Work (site repo)
10
+ 1. `/docs/performance` (new page): the measured tables — whole-text check
11
+ latency (1-33 ms per 100 words), per-sweep averages post-0.3.2
12
+ (en 45 ms ... pt 423 ms, fr worst-case note), tier sizes (full/fluency/mini),
13
+ and tuning: personal dictionaries, baselines, tier choice guidance
14
+ (`--tier`), backend choice (ruby vs native vs wasm). Link from the
15
+ playground footnote, /languages, and /install.
16
+ 2. Truth pass: /docs/clients/action still says the action "installs the gem
17
+ from RubyGems where 0.7.0 gates six languages" — stale; the action now
18
+ rides 0.9.2 (20 full-feature languages, baselines, directory mode).
19
+ Audit every page for version references (0.7.0/0.8.0/0.1.1/wasm 0.2.0)
20
+ and flip to current-with-dates. /playground/server: verify the hosted
21
+ demo state and the run-your-own instructions.
22
+ 3. Footer/readout: confirm every registry badge resolves (RubyGems 0.9.2,
23
+ npm 0.3.2, PyPI, crates.io).
24
+
25
+ ## Verification
26
+ Rendered text extraction on the built site for stale version strings;
27
+ every measured number in /docs/performance traceable to the 0.3.2 bench
28
+ harness (kotoshu-rs PR #23 table) — no inferred numbers.
29
+
30
+ ## Status
31
+ Executed 2026-09-07 — site PR #26: /docs/performance + truth pass. Action gap found and closed via action-kotoshu PR #2 + site PR #27 docs flip.
@@ -33,5 +33,8 @@ rb-sys = { version = "0.9.124", features = ["global-allocator"] }
33
33
  # and are a documented relax trigger for the P4 gate, not adopted here).
34
34
  magnus = "0.8"
35
35
 
36
- # The engine. Feature "ruby" turns on ffi::ruby (magnus bindings).
37
- kotoshu = { git = "https://github.com/kotoshu/kotoshu-rs", branch = "main", features = ["ruby"] }
36
+ # The engine. Feature "ruby" turns on ffi::ruby (magnus bindings);
37
+ # "model" turns on the pure-Rust LID reader (kotoshu::lid, plan 106) —
38
+ # the same feature the core crate ships it under, so pure-Ruby installs
39
+ # that never build this extension are unaffected.
40
+ kotoshu = { git = "https://github.com/kotoshu/kotoshu-rs", branch = "main", features = ["ruby", "model"] }