kotoshu 0.9.3 → 0.11.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (48) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +65 -0
  3. data/Cargo.lock +72 -4
  4. data/Rakefile +35 -0
  5. data/TODO.impl/00-vision.md +32 -0
  6. data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
  7. data/TODO.impl/106-native-lid-detection.md +36 -0
  8. data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
  9. data/TODO.impl/108-ko-ne-full-feature.md +37 -0
  10. data/TODO.impl/109-ten-audit.md +26 -0
  11. data/docs/STABILITY-1.0-AUDIT.md +251 -0
  12. data/docs/STABILITY-1.0.md +137 -0
  13. data/docs/cache-architecture.md +1 -1
  14. data/ext/kotoshu_native/Cargo.toml +5 -2
  15. data/ext/kotoshu_native/src/lib.rs +134 -6
  16. data/lib/kotoshu/cache/language_cache.rb +26 -11
  17. data/lib/kotoshu/cache/model_cache.rb +150 -2
  18. data/lib/kotoshu/cache/staged_languages.rb +45 -0
  19. data/lib/kotoshu/cache.rb +3 -0
  20. data/lib/kotoshu/cli/dict_command.rb +64 -0
  21. data/lib/kotoshu/cli.rb +9 -57
  22. data/lib/kotoshu/configuration.rb +20 -2
  23. data/lib/kotoshu/fluent_checker.rb +3 -0
  24. data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
  25. data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
  26. data/lib/kotoshu/keyboard/registry.rb +24 -2
  27. data/lib/kotoshu/keyboard.rb +6 -0
  28. data/lib/kotoshu/language/lid_detector.rb +188 -0
  29. data/lib/kotoshu/language/script.rb +128 -0
  30. data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
  31. data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
  32. data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
  33. data/lib/kotoshu/language.rb +9 -0
  34. data/lib/kotoshu/languages/ko/language.rb +56 -0
  35. data/lib/kotoshu/languages/ne/language.rb +60 -0
  36. data/lib/kotoshu/languages/nn/language.rb +26 -0
  37. data/lib/kotoshu/languages.rb +8 -0
  38. data/lib/kotoshu/multi_language_checker.rb +3 -0
  39. data/lib/kotoshu/spellchecker.rb +45 -8
  40. data/lib/kotoshu/version.rb +1 -1
  41. data/lib/kotoshu.rb +45 -10
  42. data/scripts/extract_eval_grids.rb +1 -1
  43. data/sig/kotoshu.rbs +77 -0
  44. metadata +20 -5
  45. data/lib/kotoshu/commands/cache_command.rb +0 -421
  46. data/lib/kotoshu/commands/check_command.rb +0 -301
  47. data/lib/kotoshu/commands/model_command.rb +0 -291
  48. data/lib/kotoshu/commands.rb +0 -12
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 4891dc67bd85b5c342fdbc7453085990a1e9ff9bea5b394a67f87e11b4bbd6c7
4
- data.tar.gz: 378d1b77bea8113d155774bc07e481afaa9229d6e1edcd7d5fda2805d28a963c
3
+ metadata.gz: a193e09b8fdaaf3e05a6b988e47ca460d8f7797b30e1719d109fdb9d28a89794
4
+ data.tar.gz: 377667004f6fb82be98470efa6e667a623afd96c01c61872db8b4c0048329c54
5
5
  SHA512:
6
- metadata.gz: bebc7b8be31a4f9f0127bddf2cb4a8de23c9908b05ddf91f51a43af5927af4608a0398090cce27deefce0e5c86f4a7d9edb0ffdf6b122f849bb065cfcee44e2d
7
- data.tar.gz: 1240bb5e7d513c4d3b3dd936042c3d177cbad6aebc402fc6ef91f86e3163bc0ba2dcf516f93d7658171d2c952c1f2fed6162928c7a50bdef3e5703233acce1ce
6
+ metadata.gz: 4c90da5e79e5b5f5a8e91a3a3b734874cc30699c9d95462f8cdae9160389601de3eea63b7231e19ba887e41d4b1e99ce12ec0ed07351868bb48c2ac8ba852007
7
+ data.tar.gz: de5adbf779c0c384b91186a73aaa571fe02bd49c84e7cad3a8bf62b366b76da4b7a853745c68b50eae6c5482bfe95fb1e55ca3f4798eb73b4194ea48c358ef62
data/CHANGELOG.md CHANGED
@@ -8,6 +8,71 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
8
8
  ## [Unreleased]
9
9
 
10
10
  ### Added
11
+ - **Korean and Nepali full-feature modules** (plan 108) - the last
12
+ big-population languages without modules move from the plan 107
13
+ basic tier to full feature. `ko` ships the Hangul tokenizer
14
+ (an eojeol is a run of Hangul syllables U+AC00-D7A3 plus bare jamo
15
+ U+1100-11FF/U+3130-318F that escaped IME composition) and the
16
+ Dubeolsik keyboard grid keyed on jamo per the KS X 5002 standard 2-set
17
+ layout, so typo proximity lands on the unit vowel mergers and
18
+ fortis/lenis confusions operate on. `ne` ships the Devanagari
19
+ tokenizer (matras U+093E-094C, the virama U+094D and vowel signs
20
+ stay attached to the base consonant; the danda and Devanagari
21
+ digits are separators) and the Devanagari InScript grid (the Indian
22
+ national standard, mirrored from the models repo eval harness and
23
+ drift-checked against it; the Nepali Traditional Romanized layout
24
+ is more common on the ground but is not a documented standard).
25
+ Both dictionaries were already staged, so `kotoshu setup ko ne`
26
+ serves them; full-feature languages grow 33 to 35.
27
+ - **Basic support for every staged dictionary language** (plan 107) -
28
+ `kotoshu setup` now serves all 95 languages staged in the
29
+ kotoshu/dictionaries manifest, not just the ones with gem modules.
30
+ `LanguageCache::AVAILABLE_LANGUAGES` derives from a vendored
31
+ snapshot of the manifest (Cache::StagedLanguages, refreshed by
32
+ `rake kotoshu:staged_languages:sync`), keeping the hot path
33
+ network-free. Languages without a module run on the basic tier:
34
+ script classification (Language::Script) picks the tokenizer whose
35
+ word regex drives check/suggest word extraction (shared Latin /
36
+ Cyrillic / Greek tokenizers plus a generic regex tokenizer for
37
+ Arabic, Hebrew, Armenian, Georgian, Devanagari and Hangul), and the
38
+ keyboard registry falls back to the script's standard grid (JCUKEN
39
+ for Cyrillic, Arabic 101, Hebrew SI-1452, QWERTY otherwise) instead
40
+ of QWERTY for everything. Two documented tiers: full feature =
41
+ module (registered keyboard + verified specimens,
42
+ `LanguageCache.full_feature_languages`), basic = manifest-present.
43
+ `nn` (Norwegian Nynorsk) is wired as a full-feature module while
44
+ here - dictionary and model are both staged. Modules keep winning
45
+ over the fallbacks; the 32 existing modules behave identically
46
+ (conformance vectors untouched).
47
+ - **Personal dictionary in the check path** (plan 105) -
48
+ `Spellchecker#check` (and therefore `Kotoshu.check`,
49
+ `Kotoshu.check_file`, and `kotoshu check`) no longer flags words
50
+ present in the user personal dictionary
51
+ (~/.config/kotoshu/personal.dic, KOTOSHU_PERSONAL_DIC override),
52
+ matching the semantics the LSP already applies to diagnostics:
53
+ case-insensitive, filtered at result assembly without suppression
54
+ metadata, words still counted, and `suggest`/`correct?` untouched so
55
+ personal words stay spellcheckable. The dictionary is loaded once
56
+ per process (no hot reload; the LSP keeps its mtime-based reload).
57
+ Opt out with `Kotoshu.configure { |c| c.personal_dictionary = false }`
58
+ / KOTOSHU_PERSONAL_DICTIONARY=false / `kotoshu check --no-personal`.
59
+ - **Native language detection** (plan 106) - `Kotoshu.detect_language(text)`
60
+ now returns a `Kotoshu::Language::Detection` (`code` + `score`) scored by
61
+ the Rust extension over the `kotoshu://models/lid/lid-176` registry
62
+ artifact pair (the ONNX-converted lid.176 model, 176 languages,
63
+ bit-faithful to the fastText bindings - parity 55/55 against the frozen
64
+ corpus shared with kotoshu-rs). Fetch the pair once with
65
+ `Kotoshu.setup_lid` (registry-resolved, sha-verified, cached under
66
+ `models-cache/lid/`); detect resolves cache-only and never downloads.
67
+ The 7-language `Language::Detector` heuristic stays the fallback when
68
+ the extension is absent, the model is not set up, or
69
+ `KOTOSHU_BACKEND=ruby` opts out - so pure-Ruby installs are unaffected
70
+ (no new gemspec dependency, no onnxruntime). The magnus shim
71
+ (ext/kotoshu_native) gains `Kotoshu::Native::LidModel.load` +
72
+ `#detect` behind the kotoshu-rs `model` feature; the workspace
73
+ Cargo.lock moves to the kotoshu-rs revision carrying the reader.
74
+ `Kotoshu.detect_language_with_confidence` keeps its `[code, score]`
75
+ shape and now routes through the same engine.
11
76
  - **Full-feature language batch 3** (plan 100) - twelve languages gain the
12
77
  downloadable spelling dictionary, a keyboard layout, and a gem module:
13
78
  `ar` `id` `fa` `he` `bg` `sr` `hr` `sk` `sl` `lt` `lv` `et` (the RTL
data/Cargo.lock CHANGED
@@ -26,7 +26,7 @@ dependencies = [
26
26
  "regex",
27
27
  "rustc-hash",
28
28
  "shlex",
29
- "syn",
29
+ "syn 2.0.119",
30
30
  ]
31
31
 
32
32
  [[package]]
@@ -82,12 +82,20 @@ dependencies = [
82
82
  "either",
83
83
  ]
84
84
 
85
+ [[package]]
86
+ name = "itoa"
87
+ version = "1.0.18"
88
+ source = "registry+https://github.com/rust-lang/crates.io-index"
89
+ checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682"
90
+
85
91
  [[package]]
86
92
  name = "kotoshu"
87
93
  version = "0.1.0"
88
- source = "git+https://github.com/kotoshu/kotoshu-rs?branch=main#9941f1ea1a6157c57659e428f475959508d4d0b3"
94
+ source = "git+https://github.com/kotoshu/kotoshu-rs?branch=main#e574c445df5f6eee8e978f1f4d26dc48cfb1947b"
89
95
  dependencies = [
90
96
  "magnus",
97
+ "serde",
98
+ "serde_json",
91
99
  ]
92
100
 
93
101
  [[package]]
@@ -141,7 +149,7 @@ checksum = "47607461fd8e1513cb4f2076c197d8092d921a1ea75bd08af97398f593751892"
141
149
  dependencies = [
142
150
  "proc-macro2",
143
151
  "quote",
144
- "syn",
152
+ "syn 2.0.119",
145
153
  ]
146
154
 
147
155
  [[package]]
@@ -205,7 +213,7 @@ dependencies = [
205
213
  "quote",
206
214
  "regex",
207
215
  "shell-words",
208
- "syn",
216
+ "syn 2.0.119",
209
217
  ]
210
218
 
211
219
  [[package]]
@@ -255,6 +263,49 @@ version = "0.3.6"
255
263
  source = "registry+https://github.com/rust-lang/crates.io-index"
256
264
  checksum = "1bc711410fbe7399f390ca1c3b60ad0f53f80e95c5eb935e52268a0e2cd49acc"
257
265
 
266
+ [[package]]
267
+ name = "serde"
268
+ version = "1.0.229"
269
+ source = "registry+https://github.com/rust-lang/crates.io-index"
270
+ checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba"
271
+ dependencies = [
272
+ "serde_core",
273
+ "serde_derive",
274
+ ]
275
+
276
+ [[package]]
277
+ name = "serde_core"
278
+ version = "1.0.229"
279
+ source = "registry+https://github.com/rust-lang/crates.io-index"
280
+ checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48"
281
+ dependencies = [
282
+ "serde_derive",
283
+ ]
284
+
285
+ [[package]]
286
+ name = "serde_derive"
287
+ version = "1.0.229"
288
+ source = "registry+https://github.com/rust-lang/crates.io-index"
289
+ checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348"
290
+ dependencies = [
291
+ "proc-macro2",
292
+ "quote",
293
+ "syn 3.0.5",
294
+ ]
295
+
296
+ [[package]]
297
+ name = "serde_json"
298
+ version = "1.0.151"
299
+ source = "registry+https://github.com/rust-lang/crates.io-index"
300
+ checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14"
301
+ dependencies = [
302
+ "itoa",
303
+ "memchr",
304
+ "serde",
305
+ "serde_core",
306
+ "zmij",
307
+ ]
308
+
258
309
  [[package]]
259
310
  name = "shell-words"
260
311
  version = "1.1.1"
@@ -278,6 +329,17 @@ dependencies = [
278
329
  "unicode-ident",
279
330
  ]
280
331
 
332
+ [[package]]
333
+ name = "syn"
334
+ version = "3.0.5"
335
+ source = "registry+https://github.com/rust-lang/crates.io-index"
336
+ checksum = "12df2e0110f65b775f769bb17ef989067a1d931b2eb822bd4346631eeada89f9"
337
+ dependencies = [
338
+ "proc-macro2",
339
+ "quote",
340
+ "unicode-ident",
341
+ ]
342
+
281
343
  [[package]]
282
344
  name = "unicode-ident"
283
345
  version = "1.0.24"
@@ -289,3 +351,9 @@ name = "windows-link"
289
351
  version = "0.2.1"
290
352
  source = "registry+https://github.com/rust-lang/crates.io-index"
291
353
  checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"
354
+
355
+ [[package]]
356
+ name = "zmij"
357
+ version = "1.0.23"
358
+ source = "registry+https://github.com/rust-lang/crates.io-index"
359
+ checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b"
data/Rakefile CHANGED
@@ -32,6 +32,41 @@ report_divergences = lambda do |divergences|
32
32
  end
33
33
 
34
34
  namespace :kotoshu do
35
+ namespace :staged_languages do
36
+ desc <<~DESC
37
+ Regenerate the vendored staged-language list (plan 107). Fetches \
38
+ the dictionaries manifest at the SourceRegistry pin, extracts the \
39
+ distinct language codes from its resources, and rewrites \
40
+ lib/kotoshu/cache/staged_languages.rb in place so \
41
+ LanguageCache::AVAILABLE_LANGUAGES tracks the manifest. Run after \
42
+ the dictionaries repo stages new languages.
43
+ DESC
44
+ task :sync do
45
+ require "kotoshu"
46
+ require "json"
47
+
48
+ url = Kotoshu::SourceRegistry.new.url_for(:dict_manifest)
49
+ puts "Fetching #{url}"
50
+ bytes = Kotoshu::Integrity::NetHTTP.get(url)
51
+ abort "manifest not found at #{url}" if bytes.nil?
52
+
53
+ manifest = JSON.parse(bytes)
54
+ codes = manifest.fetch("resources").values.map { |r| r.fetch("language") }.uniq.sort
55
+ abort "manifest carried no languages" if codes.empty?
56
+
57
+ target = File.expand_path("lib/kotoshu/cache/staged_languages.rb", __dir__)
58
+ source = File.read(target)
59
+ updated = source.sub(/CODES = %w\[\n(?: {8}\S+(?: \S+)*\n)+ {6}\]\.freeze/) do
60
+ wrapped = codes.each_slice(10).map { |slice| " #{slice.join(' ')}" }.join("\n")
61
+ "CODES = %w[\n#{wrapped}\n ].freeze"
62
+ end
63
+ abort "CODES block not found in #{target}" if updated == source
64
+
65
+ File.write(target, updated)
66
+ puts "Rewrote #{target}: #{codes.size} languages (was #{Kotoshu::Cache::StagedLanguages::CODES.size})"
67
+ end
68
+ end
69
+
35
70
  namespace :conformance do
36
71
  desc <<~DESC
37
72
  Export conformance vectors for kotoshu-rs (plan 67 M3). Walks the Spylls \
@@ -247,3 +247,35 @@ plan name the source.
247
247
  | `docs/ARCHITECTURE_IMPROVEMENTS_PLAN.md` (5 phases) | Phase 1 perf items in `01`/`05`; Phase 2 architecture in `00-vision` (this doc); Phase 3 UX mostly implemented (`PersonalDictionary`, `ProjectConfig`, `FluentChecker` exist); Phase 4 quality in `10`; Phase 5 polish spread |
248
248
  | `docs/TDD_ITERATION_STRATEGY.md` | `10` (test methodology + CI); project-rule overrides (no doubles, behavior-not-implementation) in `CLAUDE.md` |
249
249
  | `docs/ONNX_FUNCTIONAL_VERIFICATION.md` | `05` (semantic path productionization) |
250
+
251
+
252
+ Plans 105-107 plus models 09 and site 11 are **wave 7, detection and
253
+ coverage depth** (2026-09-07): after the wave-6 releases (gem 0.9.3,
254
+ wasm 0.4.0, registry v1.4.0, action v2), the remaining gaps are the
255
+ personal dictionary never reaching the check path, detection stuck on
256
+ a 7-language heuristic, buckets only for en/de, and the playground not
257
+ yet passing bucket bytes to loadModel.
258
+
259
+ | # | Plan | Gap it closes |
260
+ |---|---|---|
261
+ | 105 | [Personal dictionary in check](105-personal-dictionary-in-check.md) | kotoshu personal add then check still flags the word |
262
+ | 106 | [Native LID detection](106-native-lid-detection.md) | detect is a 7-language heuristic; the pure-Rust lid reader exists unused |
263
+ | 107 | models [09 bucket expansion](../../models-fasttext-onnx/TODO.impl/09-bucket-table-expansion.md) | bucket OOV siblings exist only for en and de |
264
+ | — | site 11 playground buckets | wasm 0.4.0 accepts bucketsBytes; the worker never fetches the sibling |
265
+
266
+
267
+ Plans 107-109 are **wave 8, reachability and the road to 1.0**
268
+ (2026-09-07): after wave 7 (personal dict in check, 176-language
269
+ detection, buckets rolling to all languages), the remaining user gaps
270
+ are the sixty-plus staged dictionaries unreachable without a module,
271
+ the last big-population languages (ko, ne), and the absence of a
272
+ stability commitment adopters can build on.
273
+
274
+ | # | Plan | Gap it closes |
275
+ |---|---|---|
276
+ | 107 | [Basic support for every staged language](107-basic-support-every-staged-language.md) | setup rejects 60+ staged languages; modules become upgrades, not gates |
277
+ | 108 | [ko and ne full-feature](108-ko-ne-full-feature.md) | Hangul and Devanagari - the last ~100M speakers without support |
278
+ | 109 | [The 1.0 readiness audit](109-ten-audit.md) | no frozen surface or stability policy for adopters |
279
+
280
+ Sequencing: 108 after 107 (same area); 109 is analysis-only and runs in
281
+ parallel. The 1.0 cut itself remains the owner's version call.
@@ -0,0 +1,34 @@
1
+ # Plan 105 — The personal dictionary reaches the check path
2
+
3
+ ## Why
4
+ `kotoshu personal add recieve` then `kotoshu check` still flags the word —
5
+ PersonalDictionary is wired only into the `personal` subcommand and the LSP
6
+ (resolved server-side), not into `Spellchecker#check`. Users who add words
7
+ see them flagged again on the very next run; the README story ("add words
8
+ you accept") is currently false on the CLI/API path.
9
+
10
+ ## Work (gem repo)
11
+ 1. `Spellchecker#check` consults the personal dictionary: a word present in
12
+ personal.dic is not an error (mirror the LSP's semantics: filter at
13
+ result assembly, keep suppression metadata out of the error rows).
14
+ 2. Reload semantics: the LSP reloads on mtime change; the CLI/one-shot path
15
+ loads once per process. Do not add hot reload here.
16
+ 3. Opt-out: `Kotoshu.configure { personal_dictionary false }` (or the
17
+ existing config seam — find it) and `--no-personal` on `kotoshu check`.
18
+ 4. Specs: add-then-check round trip on a real dictionary; opt-out flags
19
+ again; personal words still spellcheckable via suggest.
20
+ 5. Docs truth: /docs/ignores gains the personal dictionary section (the
21
+ action PR #2 README noted this gap) — deliver the copy in the PR body
22
+ for the site follow-up, do not touch the site repo.
23
+
24
+ ## Verification
25
+ Suite green (3,900+ examples); rubocop clean; `kotoshu personal add x &&
26
+ kotoshu check` live round trip recorded in the PR body.
27
+
28
+ ## Status
29
+ Executed 2026-09-07 — gem PR (branch plan-105-personal-dict-in-check):
30
+ check-path filter mirroring the LSP, one load per process,
31
+ Configuration#personal_dictionary + KOTOSHU_PERSONAL_DICTIONARY +
32
+ `kotoshu check --no-personal`; suite pins KOTOSHU_PERSONAL_DIC for
33
+ hermeticity. /docs/ignores copy delivered in the PR body for the site
34
+ follow-up.
@@ -0,0 +1,36 @@
1
+ # Plan 106 — Document language detection through the pure-Rust LID reader
2
+
3
+ ## Why
4
+ The gem's LanguageIdentifier needs the Python fastText bindings (and breaks
5
+ under numpy 2), so /v1/detect ships a 7-language heuristic. kotoshu-rs now
6
+ carries a pure-Rust lid.176 reader (kotoshu/src/lid, parity 55/55 vs the
7
+ gem) behind the same `model` feature the native ext already builds.
8
+
9
+ ## Work (gem repo)
10
+ 1. Expose `Kotoshu.detect_language(text)` through the native backend:
11
+ download/resolve lid.176.onnx + vocab via the models registry
12
+ (kotoshu://models/lid/lid-176, v1.4.0), score with the ext, return
13
+ { code, score }. Heuristic Language::Detector stays the fallback when
14
+ the native ext is absent or the model is not set up.
15
+ 2. FFI seam: extend the magnus shim (ext/kotoshu_native) with the call —
16
+ mirror how suggest/check already cross; keep it behind the existing
17
+ feature so pure-Ruby installs are unaffected.
18
+ 3. Server follow-up (kotoshu-server repo, separate PR riding this):
19
+ /v1/detect prefers Kotoshu.detect_language when available. Note it in
20
+ the PR body; do not touch the server repo yourself.
21
+ 4. Specs: native-vs-fixture parity (reuse the rs corpus JSON shape),
22
+ fallback behavior, registry miss error.
23
+
24
+ ## Verification
25
+ Suite green; a live detect over en/de/ru/ja/ar samples recorded in the PR
26
+ body; pure-Ruby suite still green (no new hard dependency).
27
+
28
+ ## Status
29
+ Executed 2026-09-07 — gem PR (branch plan-106-native-lid-detection):
30
+ Kotoshu.detect_language -> Language::Detection through the ext
31
+ (LidModel.load/#detect in the magnus shim, kotoshu-rs model feature,
32
+ Cargo.lock bumped to the reader revision); ModelCache gained the
33
+ lid pair (download_lid/load_cached_lid, top-level lid/ dir);
34
+ Kotoshu.setup_lid two-stage; heuristic fallback incl.
35
+ KOTOSHU_BACKEND=ruby; parity 55/55 vs the shared frozen corpus.
36
+ Server follow-up (/v1/detect) noted in the PR body.
@@ -0,0 +1,42 @@
1
+ # Plan 107 — Basic support for every staged dictionary language
2
+
3
+ ## Why
4
+ `kotoshu setup` only serves the 32 languages with modules. Sixty-plus staged
5
+ dictionaries (from gd to ia) are unreachable: a Welsh or Icelandic user
6
+ cannot use the gem at all, even though the dictionary and model are
7
+ downloadable. Modules should be the UPGRADE (keyboard + verified specimens),
8
+ not the gate.
9
+
10
+ ## Work (gem repo)
11
+ 1. `LanguageCache::AVAILABLE_LANGUAGES` derives from the dictionaries
12
+ manifest (the source the registry/staging uses) instead of the hardcoded
13
+ module list; modules register themselves as full-feature upgrades.
14
+ 2. Default behavior for module-less languages: script-appropriate tokenizer
15
+ (reuse the script-regex machinery from plan 91), generic keyboard
16
+ fallback (QWERTY for Latin, JCUKEN Cyrillic, Arabic 101 Arabic-script),
17
+ setup/check/suggest all functional.
18
+ 3. Wire nn (model + dictionary exist) as a full-feature module while here;
19
+ note sr-Latn as follow-up if the Latin mapping needs product sign-off.
20
+ 4. Specs: setup + specimen round trip on 3 module-less staged languages
21
+ (e.g. is, cy, gd); full-feature languages behave identically to today;
22
+ conformance untouched.
23
+
24
+ ## Verification
25
+ Live `kotoshu setup is && check -l is` round trip; suite green; the 32
26
+ modules' outputs byte-identical.
27
+
28
+ ## Status
29
+ Executed 2026-09-07 — AVAILABLE_LANGUAGES derives from a vendored
30
+ snapshot of the dictionaries manifest (Cache::StagedLanguages, 95
31
+ languages, refreshed by `rake kotoshu:staged_languages:sync`);
32
+ module-less languages ride Language::Script (script tokenizer for
33
+ word extraction: shared Latin/Cyrillic/Greek classes plus a generic
34
+ regex tokenizer for Arabic/Hebrew/Armenian/Georgian/Devanagari/
35
+ Hangul) and the script keyboard default (JCUKEN Cyrillic, Arabic 101,
36
+ Hebrew SI-1452, QWERTY Latin). Tiers documented: full feature =
37
+ module (LanguageCache.full_feature_languages, 33 with nn); basic =
38
+ manifest-present. nn wired as a full-feature module. Specimens
39
+ engine-verified for is cy gd (basic) + nn; live round trips
40
+ setup+check -l for is cy gd nn; conformance 2630 vectors 0 failures;
41
+ suite 3948 examples 0 failures. sr-Latn stays follow-up (normalize
42
+ maps it to sr; needs product sign-off).
@@ -0,0 +1,37 @@
1
+ # Plan 108 — Korean and Nepali: the last big-population languages
2
+
3
+ ## Why
4
+ ko (~82M) and ne (~16M) were deferred from batch 3 because Hangul jamo
5
+ input and Devanagari conjuncts need ja-style tokenizer treatment. They are
6
+ the largest populations without full-feature support; both have staged
7
+ dictionaries and models.
8
+
9
+ ## Work (gem repo; sequenced AFTER plan 107 merges — same area)
10
+ 1. ko: Hangul tokenizer (syllable-block aware, jamo-level word regex so
11
+ check extracts real eojeol), Korean keyboard grid (Dubeolsik 2-set),
12
+ module + specimens engine-verified.
13
+ 2. ne: Devanagari tokenizer (matras and conjuncts stay attached to the
14
+ base consonant — the Unicode grapheme rules the script needs), Roman
15
+ keyboard fallback or a Nepali traditional grid if defensibly sourced,
16
+ module + specimens.
17
+ 3. AVAILABLE_LANGUAGES + site-flip deliverables (specimen table, sizes).
18
+
19
+ ## Verification
20
+ Same discipline as batch 3: every specimen engine-verified through
21
+ correct?/suggest; RTL/complex-script sentence round trips through
22
+ Kotoshu.check; suite green.
23
+
24
+ ## Status
25
+ Executed 2026-09-07 — ko: HangulTokenizer (eojeol = run of syllables
26
+ U+AC00-D7A3 plus bare jamo U+1100-11FF/U+3130-318F), Dubeolsik grid
27
+ keyed on jamo per KS X 5002 (the eval harness models ko slips as
28
+ jamo confusion pairs, not a key grid, so no drift check applies),
29
+ module + 7 engine-verified specimens. ne: DevanagariTokenizer
30
+ (matras and virama conjuncts attached to the base consonant; danda
31
+ and Devanagari digits are separators), Devanagari InScript grid
32
+ mirrored from the eval harness _NE_INSCRIPT and drift-checked (the
33
+ defensible standard — Traditional Romanized is common but
34
+ undocumented), module + 3 specimens. Full-feature languages 33 -> 35.
35
+ Live round trips setup + check -l ko ne recorded; conformance 2630
36
+ vectors 0 failures; suite 3985 examples 0 failures. Specimen table
37
+ and dictionary sizes in the PR body for the site flip.
@@ -0,0 +1,26 @@
1
+ # Plan 109 — The 1.0 readiness audit
2
+
3
+ ## Why
4
+ Adopters hesitate at 0.x. A 1.0 cut needs a frozen public surface and a
5
+ stated policy. The audit inventories what 1.0 would freeze and what it
6
+ would deprecate — it does NOT cut versions (owner decision).
7
+
8
+ ## Work (gem repo docs/, analysis only)
9
+ 1. Inventory the public surface: Ruby API (Kotoshu module functions,
10
+ Spellchecker, Configuration, Tasks, validators), CLI flags, wasm JS
11
+ surface (KotoshuWasm, loadModel/rerank/semanticSuggest/loadLid/
12
+ detectLanguage), HTTP contract (openapi.yaml), action inputs, registry
13
+ schema.
14
+ 2. For each: stable / needs-deprecation-notice / internal-accidentally-
15
+ public. Flag anything accidentally public that 1.0 should privatize.
16
+ 3. Draft the stability policy (semver commitments, deprecation window,
17
+ conformance vectors as the behavioral contract) as docs/STABILITY.md.
18
+ 4. A 1.0 checklist: what must land first (nothing mechanical is known to
19
+ block), what the release train looks like (gem+wasm+server together).
20
+
21
+ ## Verification
22
+ Every inventory line cites the file that defines it; the deprecation list
23
+ reviewed against gemspec/exported constants.
24
+
25
+ ## Status
26
+ Pending