kotoshu 0.8.0 → 0.9.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (71) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +53 -0
  3. data/README.adoc +1 -1
  4. data/TODO.impl/00-cut-0.1.md +121 -0
  5. data/TODO.impl/00-cut-0.2.md +175 -0
  6. data/TODO.impl/00-vision.md +15 -0
  7. data/TODO.impl/01-hunspell-correctness.md +111 -0
  8. data/TODO.impl/02-cli-unification.md +74 -0
  9. data/TODO.impl/03-dynamic-download.md +107 -0
  10. data/TODO.impl/04-language-modules.md +89 -0
  11. data/TODO.impl/05-semantic-path.md +83 -0
  12. data/TODO.impl/06-cjk-support.md +105 -0
  13. data/TODO.impl/07-rtl-support.md +71 -0
  14. data/TODO.impl/08-grammar-engine.md +106 -0
  15. data/TODO.impl/09-integrity-security.md +90 -0
  16. data/TODO.impl/10-testing-ci.md +98 -0
  17. data/TODO.impl/11-release-v1.md +95 -0
  18. data/TODO.impl/20-encapsulation-cleanup.md +47 -0
  19. data/TODO.impl/21-autoload-migration.md +64 -0
  20. data/TODO.impl/33-audit-log-rotation.md +54 -0
  21. data/TODO.impl/34-cache-eviction.md +59 -0
  22. data/TODO.impl/35-shell-completion.md +61 -0
  23. data/TODO.impl/40-spec-drift-cleanup.md +134 -0
  24. data/TODO.impl/50-structure-aware-document-api.md +194 -0
  25. data/TODO.impl/51-grammar-engine-expansion.md +75 -0
  26. data/TODO.impl/52-multi-language-documents.md +95 -0
  27. data/TODO.impl/53-document-plugin-architecture.md +94 -0
  28. data/TODO.impl/54-cjk-support.md +117 -0
  29. data/TODO.impl/55-rtl-support.md +109 -0
  30. data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
  31. data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
  32. data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
  33. data/TODO.impl/59-yard-api-documentation.md +75 -0
  34. data/TODO.impl/60-lsp-server.md +174 -0
  35. data/TODO.impl/61-editor-ecosystem.md +167 -0
  36. data/TODO.impl/62-framework-integrations.md +159 -0
  37. data/TODO.impl/63-web-playground-wasm.md +186 -0
  38. data/TODO.impl/64-http-api-and-sdks.md +191 -0
  39. data/TODO.impl/98-campaign-cleanup.md +78 -0
  40. data/conformance/vectors.jsonl +84 -84
  41. data/docs/verification-evidence/correct3/pr_body.md +37 -0
  42. data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
  43. data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
  44. data/docs/verification-evidence/correct3/sv.aff +856 -0
  45. data/docs/verification-evidence/correct3/sv.dic +153717 -0
  46. data/lib/kotoshu/algorithms/edit_distance.rb +6 -3
  47. data/lib/kotoshu/cache/base_cache.rb +33 -0
  48. data/lib/kotoshu/cache/frequency_cache.rb +28 -24
  49. data/lib/kotoshu/cache/language_cache.rb +23 -16
  50. data/lib/kotoshu/cache/model_cache.rb +44 -41
  51. data/lib/kotoshu/cli/cache_command.rb +4 -0
  52. data/lib/kotoshu/configuration.rb +5 -1
  53. data/lib/kotoshu/data/common_words_loader.rb +21 -31
  54. data/lib/kotoshu/dictionary/base.rb +12 -0
  55. data/lib/kotoshu/dictionary/custom.rb +6 -6
  56. data/lib/kotoshu/dictionary/hunspell.rb +8 -0
  57. data/lib/kotoshu/dictionary/plain_text.rb +8 -5
  58. data/lib/kotoshu/dictionary/unix_words.rb +5 -5
  59. data/lib/kotoshu/integrity/net_http.rb +5 -0
  60. data/lib/kotoshu/languages/nb/language.rb +25 -0
  61. data/lib/kotoshu/languages.rb +1 -0
  62. data/lib/kotoshu/resource_manager.rb +5 -3
  63. data/lib/kotoshu/suggestions/frequency_provider.rb +13 -4
  64. data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +115 -0
  65. data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +26 -47
  66. data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +25 -9
  67. data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +20 -38
  68. data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
  69. data/lib/kotoshu/suggestions.rb +1 -0
  70. data/lib/kotoshu/version.rb +1 -1
  71. metadata +44 -2
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: f90290a36de788d46588e6883b0619f9fb1dff8e841f629fd021c07e156256ae
4
- data.tar.gz: 113cbf58fb1b9c297cfb8551a9076875f4882be9e7b17bbebe330ecc53aac94a
3
+ metadata.gz: c1082e88abf75329c969089181472e5a1e511bbb48b7f1c7ab4c113b5e6fc310
4
+ data.tar.gz: 92cb046340c93ffdf72f7c9c69eb776a683757723020a96b3c4805416c43d2f7
5
5
  SHA512:
6
- metadata.gz: 1520bf0ce2667b31872c137b55099834a93954f821c93903b193306d537703173ad37e84446dfd41e082e25db50d4a54c694d1623b43875fd5b61294b4cf67f4
7
- data.tar.gz: f22ca5507daa129090a2d292d4e526200f197f32520b864c67e3ffbab82862a431d155303613e670aab97e48d213bad32ad0ba4b4cdf1178a8de1dff940a1cbd
6
+ metadata.gz: f5aee6be1e7c35c0bb8f3e8bcac0aa089e1caac0bd8cb0e05925c74f5e4a925fd5e0206ba22759421140e96fcfc5610596bd3bdfe14fde489b1117cc46969c15
7
+ data.tar.gz: a62aa1ec6ddbd3cc499958ad45b07fd281c5f1b919b7692960a1b59602fd3bbc8352f2d769946e91518de29e7e2cc620c3e2cba818e19fcd4eb295a48af0ddd7
data/CHANGELOG.md CHANGED
@@ -8,6 +8,59 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
8
8
  ## [Unreleased]
9
9
 
10
10
  ### Fixed
11
+ - **Suggestion candidate sweep missed transpositions, substitutions, and
12
+ dictionary forms** - `EditDistanceStrategy` only scored raw dictionary
13
+ stems, so valid dictionary forms could never be suggested no matter how
14
+ close: an adjacent transposition paired with a case difference ("Teh" ->
15
+ "The") was charged as two operations, and affixed forms ("definately" ->
16
+ "definitely", a stem + `-ly` suffix never present as a stem) were absent
17
+ from the candidate set entirely. The strategy now also sweeps the
18
+ distance-1 edits of the misspelling itself - adjacent transposition
19
+ (restricted Damerau operation, cost 1), substitution and insertion over
20
+ the dictionary TRY string, and deletion - validating each against the
21
+ full affix-aware dictionary lookup, and case-variant duplicates of the
22
+ same word are deduplicated keeping the best-scoring form. The phonetic
23
+ and keyboard-proximity strategies now measure candidates with
24
+ `Algorithms::EditDistance` (Damerau-Levenshtein; previously each carried
25
+ a private plain-Levenshtein copy charging transpositions 2), and the
26
+ keyboard variant generator enumerates adjacent transpositions as a
27
+ single edit. `Dictionary::Base#try_string` / `Dictionary::Hunspell#try_string`
28
+ expose the Hunspell TRY directive for the sweep alphabet. Conformance
29
+ suggest vectors regenerated (84 of 1315 rows improved - affixed, compound,
30
+ and correctly-cased forms now appear; all 1315 correct vectors unchanged).
31
+ - **Dictionary mutation correctness** (from suleman-uzair, PR rescue of #93) -
32
+ `remove_word` deleted the wrong word after any prior removal because the
33
+ word set stored array indices that went stale; `Custom`, `PlainText`, and
34
+ `UnixWords` now track membership (word => true) and remove by value, so
35
+ every copy of a duplicate word is gone and `words` / `include?` stay
36
+ consistent across interleaved add/remove sequences. `PlainText#add_word` /
37
+ `#remove_word` now also keep the length-bucketed index in sync, so
38
+ `find_by_length_range` (the edit-distance candidate filter) reflects
39
+ post-construction mutations instead of returning stale buckets.
40
+ - **Edit-distance threshold on empty strings** (from suleman-uzair, PR rescue
41
+ of #93) - `Algorithms::EditDistance.distance_with_threshold` returned the
42
+ other string's length for empty inputs even when that exceeded the
43
+ threshold; the length pre-filter now runs before the empty-string
44
+ shortcuts, so the nil-above-threshold contract holds for empty strings.
45
+ - **Offline mode enforcement and cache-only suggestion hot path** (from
46
+ suleman-uzair, PR rescue of #93) - `Kotoshu::Configuration.offline` /
47
+ `KOTOSHU_OFFLINE=1` previously did not stop downloads. Every cache
48
+ download path now refuses in offline mode (`BaseCache#download` and the
49
+ `download_url` / `download_file` transports raise
50
+ `ResourceNotCachedError`; `Integrity::NetHTTP.get` refuses before opening
51
+ a connection; `LanguageCache#download_spelling` / `#download_grammar`
52
+ raise). The resolve paths (`ResourceManager`, `FrequencyProvider`) read
53
+ via the public cache-only `load_cached` instead of the download-on-miss
54
+ `get`, so the suggestion hot path never triggers a download.
55
+ `Configuration.default` no longer re-applies `DEFAULTS` as explicit
56
+ settings, which clobbered every `KOTOSHU_*` ENV-derived value including
57
+ `KOTOSHU_OFFLINE`. `kotoshu cache download` surfaces offline and
58
+ integrity failures as `ResourceUnavailable` instead of a stack trace.
59
+ - **Kelly frequency loader dead branch** (from suleman-uzair, PR rescue of
60
+ #93) - `CommonWordsLoader.load_from_frequency_file` kept a legacy
61
+ array-format branch that crashed on its own probe and had no producer;
62
+ it now parses the Kelly shape only and degrades unknown shapes to empty
63
+ tiers.
11
64
  - Remote `kotoshu setup` for staged languages: the dictionaries repo ships only
12
65
  `en` under the `{lang}/spelling/` layout while staged languages sit flat at
13
66
  `{lang}/index.*`; downloads now try the sublayout first and fall back to the
data/README.adoc CHANGED
@@ -15,7 +15,7 @@ directory layout, SHA-256 integrity verification, SARIF output, an
15
15
 
16
16
  * Two-stage resource model — `Kotoshu.setup(:en)` then `Kotoshu.correct?("hello")`. The hot path is cache-only and raises `ResourceNotSetupError` on miss; downloads are never implicit.
17
17
  * `Kotoshu.check(text, language: "en")` / `Kotoshu.suggest("helo")` — full document check and suggestions
18
- * `Kotoshu.spellchecker_for(lang, strict: true)` — re-raise on optional-resource failures
18
+ * `Kotoshu.spellchecker_for(lang)` — per-language checker resolved from the cache (`strict:` is a `Kotoshu.setup` option)
19
19
  * `kotoshu check FILE` CLI with these flags:
20
20
  ** `--language en|de|es|fr|pt|ru|auto` (default: `auto`)
21
21
  ** `--format text|json|sarif`
@@ -0,0 +1,121 @@
1
+ # 00-cut-0.1 — The 0.1.0 Release Cut
2
+
3
+ ## Goal
4
+
5
+ Ship `kotoshu-0.1.0.gem` to rubygems.org with a clear, honest scope: the
6
+ pure-Ruby API and CLI work for English via system dictionaries; everything
7
+ else is documented as in-progress.
8
+
9
+ This is **not** v1.0. The point of 0.1 is to be installable, runnable,
10
+ and discoverable — to claim the name on rubygems.org and establish the
11
+ public API shape.
12
+
13
+ ## 0.1.0 scope (locked)
14
+
15
+ In scope for 0.1.0:
16
+
17
+ | Capability | Status |
18
+ |---|---|
19
+ | `Kotoshu.correct?(word)` / `suggest` / `check` / `check_file` | ✅ Working |
20
+ | `Kotoshu.detect_language(text)` (FastText LID) | ✅ Working |
21
+ | `kotoshu check TARGET` basic CLI | ✅ Working |
22
+ | `kotoshu dict list/info`, `kotoshu cache list/info/...` | ✅ Working |
23
+ | Hunspell dictionary backend (Ruby port) | ✅ Working for common cases |
24
+ | CSpell, UnixWords, PlainText, Custom backends | ✅ Working |
25
+ | Edit distance / phonetic / keyboard / n-gram / symspell strategies | ✅ Working |
26
+ | Configuration via `Kotoshu.configure` + ENV (`KOTOSHU_*`) + CLI | ✅ Working |
27
+ | 3-layer cache (LanguageCache, FrequencyCache, ModelCache) | ✅ Working |
28
+ | Document parsing (Plain text, Markdown, AsciiDoc) | ✅ Working |
29
+ | LICENSE (BSD-2-Clause), CHANGELOG.md, accurate README | ✅ Shipped |
30
+
31
+ Explicitly **deferred** to later 0.x or 1.0:
32
+
33
+ | Capability | Target | Plan |
34
+ |---|---|---|
35
+ | Unified `ResourceManager` for "any language, just works" | 0.3 | `03-dynamic-download.md` |
36
+ | `--interactive`, `--format sarif`, `--model`, `--language auto` wired | 0.2 | `02-cli-unification.md` |
37
+ | Hybrid (Hunspell + ONNX) as default | 0.4 | `05-semantic-path.md` |
38
+ | Hunspell compound rules, circumfix, ICONV/OCONV, ß, dotless-i | 0.5 | `01-hunspell-correctness.md` |
39
+ | ≥30 language modules wired | 0.6 | `04-language-modules.md` |
40
+ | Grammar rule packs | 0.7 | `08-grammar-engine.md` |
41
+ | CJK (Japanese, Chinese) | 0.8 | `06-cjk-support.md` |
42
+ | RTL (Arabic, Hebrew, Persian, Urdu) | 0.9 | `07-rtl-support.md` |
43
+ | Integrity verification (checksums, signed manifests) | 0.4 | `09-integrity-security.md` |
44
+ | v1.0 | — | `11-release-v1.md` |
45
+
46
+ ## Tasks executed for 0.1.0
47
+
48
+ 1. ✅ Audit current state — `bundle install`, smoke-test `Kotoshu.correct?`,
49
+ run rspec. Result: 803/866 (92.7%) passing; VERSION already `0.1.0`.
50
+ 2. ✅ Add `LICENSE` (BSD-2-Clause — was missing despite gemspec claim).
51
+ 3. ✅ Add `CHANGELOG.md` with full 0.1.0 scope and known-limitation list.
52
+ 4. ✅ Fix README accuracy: license, status section at top, ONNX env gate.
53
+ 5. ✅ Tighten CI workflow — rubocop no longer `continue-on-error`.
54
+ 6. ✅ Prepare release checklist (see "Release execution" below).
55
+
56
+ ## Release execution (manual — requires maintainer)
57
+
58
+ The global project rule is "AI never pushes tags, never pushes to main,
59
+ never publishes." The following must be run by a human:
60
+
61
+ ```bash
62
+ # 1. On a clean main, verify everything passes
63
+ git checkout main
64
+ git pull
65
+ bundle install
66
+ bundle exec rspec
67
+ bundle exec rubocop
68
+
69
+ # 2. Verify VERSION
70
+ grep VERSION lib/kotoshu/version.rb
71
+ # Should read: VERSION = "0.1.0"
72
+
73
+ # 3. Build the gem
74
+ gem build kotoshu.gemspec
75
+ # Produces kotoshu-0.1.0.gem
76
+
77
+ # 4. Sanity-check the gem contents
78
+ gem contents kotoshu-0.1.0.gem --tarball
79
+
80
+ # 5. Tag (the human decides when)
81
+ git tag -a v0.1.0 -m "kotoshu 0.1.0 — first public release"
82
+ git push origin v0.1.0
83
+
84
+ # 6. Publish to rubygems.org
85
+ gem push kotoshu-0.1.0.gem
86
+
87
+ # 7. Cut a GitHub Release
88
+ gh release create v0.1.0 --notes-from-tag
89
+
90
+ # 8. Smoke test from a clean environment
91
+ docker run --rm -it ruby:3.3 bash -c '
92
+ gem install kotoshu
93
+ ruby -e "require \"kotoshu\"; puts Kotoshu.correct?(\"hello\")"
94
+ echo "Hello wrold" | ruby -e "require \"kotoshu\"; puts Kotoshu.check(STDIN.read).errors.map(&:word)"
95
+ '
96
+ ```
97
+
98
+ ## Post-release
99
+
100
+ - Update `TODO.impl/00-vision.md` to reflect that 0.1 shipped; the
101
+ critical path now runs through `03-dynamic-download` →
102
+ `04-language-modules` → `11-release-v1`.
103
+ - Start `02-cli-unification.md` (next-highest user-visible value).
104
+ - Start `01-hunspell-correctness.md` task 1 (audit) to get an honest
105
+ fixture-pass number.
106
+
107
+ ## Acceptance criteria
108
+
109
+ - `gem install kotoshu` works on Ruby 3.1+
110
+ - `Kotoshu.correct?("hello")` returns `true` on a fresh install with
111
+ no env vars set
112
+ - `Kotoshu.correct?("helo")` returns `false`
113
+ - `kotoshu check README.adoc` runs without crashing
114
+ - README has no false claims (license, model count, CLI flags)
115
+ - LICENSE and CHANGELOG.md exist
116
+ - rubygems.org page shows v0.1.0 with correct metadata
117
+
118
+ ## Status
119
+
120
+ v0.1.0 release-prep complete on this branch (2026-06-25). Release
121
+ execution pending maintainer sign-off.
@@ -0,0 +1,175 @@
1
+ # 00-cut-0.2 — The First Functional Library
2
+
3
+ ## Goal
4
+
5
+ Ship `kotoshu-0.2.0` — the first release where `gem install kotoshu`
6
+ works on any Ruby 3.1+ system with **no system dictionary dependency**.
7
+ One call, `Kotoshu.check("Hello wrold")`, auto-downloads the English
8
+ Hunspell dictionary on first use and returns correct results. The CLI
9
+ works end-to-end with proper exit codes.
10
+
11
+ 0.1.0 proved the API shape. 0.2.0 delivers the promise: "just works,
12
+ no setup." For 0.2.0 the resolution machinery is English-only; the
13
+ code is language-agnostic so adding more languages is a content task
14
+ (shipping dictionaries in `kotoshu/dictionaries`), not a code change.
15
+
16
+ ## 0.2.0 scope (locked)
17
+
18
+ | Capability | Source plan |
19
+ |---|---|
20
+ | `ResourceManager.resolve` (spelling + frequency) | `03` tasks 1–2 |
21
+ | `ResourceBundle` value object | `03` task 1 |
22
+ | en_US Hunspell auto-download on first use | `03` task 2 |
23
+ | Spellchecker accepts `ResourceBundle` (Strangler-Fig) | `03` |
24
+ | One canonical `CheckCommand` wired into CLI | `02` task 1 |
25
+ | Exit codes 0/1/2/3 | `02` task 6 |
26
+ | `--language en\|auto`, `--format text\|json` | `02` task 5 |
27
+ | en_US Hunspell fixtures at 100 % | `01` tasks 1–2 |
28
+ | Offline mode (`offline: true`, `KOTOSHU_OFFLINE=1`) | `03` task 4 |
29
+ | Fake-server integration spec | `03` task 9 |
30
+
31
+ Explicitly **deferred** to 0.3+:
32
+
33
+ | Capability | Target | Plan |
34
+ |---|---|---|
35
+ | Checksum / integrity verification | 0.3 | `09-integrity-security` |
36
+ | `--interactive`, `--format sarif` | 0.3 | `02` tasks 4, 5 |
37
+ | `kotoshu fetch` pre-warm command | 0.3 | `03` task 6 |
38
+ | Strict mode | 0.3 | `03` task 5 |
39
+ | ONNX model auto-download (semantic rerank) | 0.4 | `05-semantic-path` |
40
+ | ≥30 language modules wired | 0.6 | `04-language-modules` |
41
+ | Grammar rule packs | 0.7 | `08-grammar-engine` |
42
+ | CJK (Japanese, Chinese) | 0.8 | `06-cjk-support` |
43
+ | RTL (Arabic, Hebrew, Persian, Urdu) | 0.9 | `07-rtl-support` |
44
+ | Hunspell MECE 4-layer refactor | 1.0 | `01` task 6 |
45
+
46
+ ## Critical path — three phases
47
+
48
+ ### Phase A — ResourceManager (foundation)
49
+
50
+ The Spellchecker today takes a single `Dictionary::Base` the user must
51
+ configure manually. Phase A introduces `ResourceManager` as the thing
52
+ that produces that dictionary (plus frequency data) from a language
53
+ code, using the existing `LanguageCache`.
54
+
55
+ 1. **`ResourceBundle`** (`lib/kotoshu/resource_bundle.rb`). Plain
56
+ value object — Struct or class with `language`, `dictionary`,
57
+ `frequency`, `model` (nil for 0.2), `rules` (nil for 0.2),
58
+ `cached`, `source_urls`. No hand-rolled `to_h` (project rule).
59
+ 2. **`ResourceManager`** (`lib/kotoshu/resource_manager.rb`,
60
+ autoloaded from `lib/kotoshu.rb`). API:
61
+ ```ruby
62
+ Kotoshu::ResourceManager.resolve(
63
+ text: "Hello wrold",
64
+ language: :auto, # or "en", "de", nil
65
+ want: %i[spelling frequency],
66
+ offline: false
67
+ ) # => ResourceBundle
68
+ ```
69
+ Internals: detect language via `Language::Identifier` when `:auto`;
70
+ fetch dictionary via `LanguageCache.get(language)`; fetch frequency
71
+ via `FrequencyCache.get(language)` when wanted; construct
72
+ `ResourceBundle`. No new network code — the caches already
73
+ download from `kotoshu/dictionaries` and `kotoshu/frequency-list-kelly`.
74
+ 3. **Spellchecker accepts `ResourceBundle`.** Modify
75
+ `lib/kotoshu/spellchecker.rb` to accept either a `ResourceBundle`
76
+ (new path) or a `Dictionary::Base` (legacy path). Strangler-Fig —
77
+ don't break the 0.1 API. Extract `dictionary` and `frequency` from
78
+ the bundle; ignore `model`/`rules` (nil for 0.2).
79
+ 4. **First-use UX.** When `LanguageCache` downloads, print
80
+ `[en] downloading dictionary (X.X MB)…` to STDERR. One line per
81
+ resource, not per byte. Suppress when `$stdout.tty?` is false and
82
+ `KOTOSHU_QUIET` is set.
83
+ 5. **Configuration.** Add to `Configuration::SCHEMA`:
84
+ - `offline` (boolean, default `false`) — also via `KOTOSHU_OFFLINE=1`
85
+ - `default_language` (string, default `"en"`) — used when
86
+ `language: nil` and detection is inconclusive
87
+ 6. **Offline mode.** `ResourceManager.resolve(..., offline: true)`
88
+ only uses cached resources; raises `ResourceNotCachedError` with a
89
+ message naming what's missing and how to pre-fetch.
90
+
91
+ **Phase A acceptance:** `Kotoshu.correct?("hello")` works on a Docker
92
+ `ruby:3.3` container with no `/usr/share/dict/words`, auto-downloading
93
+ en_US on first call.
94
+
95
+ ### Phase B — English correctness
96
+
97
+ 7. **Audit en_US fixtures.** Run
98
+ `bundle exec rspec spec/hunspell_tests/ --tag lang:en` (or filter
99
+ by fixture path). Record which examples fail and categorize:
100
+ compound words, capitalization, affix combinations, special
101
+ characters. This is `01` task 1 scoped to en_US only.
102
+ 8. **Fix en_US failures.** Target the categories from the audit. The
103
+ full MECE 4-layer refactor (`01` task 6) is explicitly **not**
104
+ required for 0.2 — patch the procedural `algorithms/lookup.rb`
105
+ where needed. Each fix gets a named spec example.
106
+ 9. **Suggestion sanity.** Verify `Kotoshu.suggest("helo").first.word`
107
+ == `"hello"` for the top-20 common misspellings. Tune
108
+ `EditDistanceStrategy` weight or `frequency_bonus` if the top
109
+ suggestion is wrong.
110
+
111
+ **Phase B acceptance:** 100 % of en_US Hunspell fixtures pass.
112
+ `suggest("helo")[0]` == `"hello"`.
113
+
114
+ ### Phase C — CLI wired
115
+
116
+ 10. **Wire `CheckCommand` as canonical `check`.** Remove the basic
117
+ `check` from `lib/kotoshu/cli.rb`; register
118
+ `Kotoshu::CheckCommand` (from `commands/check_command.rb`) via
119
+ `subcommand "check", Kotoshu::CheckCommand`. This is `02` task 1.
120
+ 11. **Exit codes** (`02` task 6):
121
+ - `0` — no errors found
122
+ - `1` — errors found
123
+ - `2` — usage error
124
+ - `3` — dictionary fetch failure (network down, not cached + offline)
125
+ 12. **Flag reconciliation** (`02` task 5). For 0.2, support exactly:
126
+ - `kotoshu check FILE` — default check, text output
127
+ - `--language en|de|...|auto` — language selection (`auto` =
128
+ detect via `Language::Identifier`)
129
+ - `--format text|json` — output format (sarif deferred to 0.3)
130
+ - `--offline` — use cached resources only
131
+ Remove or hide every other flag the README mentions but code
132
+ doesn't support yet. Update README to match.
133
+ 13. **End-to-end CLI spec.** Under `spec/kotoshu/cli/`, using a temp
134
+ dir (not real `~/.kotoshu`): write a file with known misspellings,
135
+ run `kotoshu check`, assert exit code + output content for both
136
+ `text` and `json` formats.
137
+
138
+ **Phase C acceptance:** `kotoshu check README.adoc` exits 1 with
139
+ readable error output; `kotoshu check clean_file.txt` exits 0.
140
+
141
+ ## Release-level acceptance criteria
142
+
143
+ - Fresh `docker run --rm ruby:3.3` → `gem install kotoshu` →
144
+ `ruby -e "require 'kotoshu'; puts Kotoshu.correct?('hello')"`
145
+ prints `true` (auto-downloads en_US on first call)
146
+ - `Kotoshu.correct?("helo")` returns `false`
147
+ - `Kotoshu.suggest("helo").first.word` == `"hello"`
148
+ - `kotoshu check README.adoc` exits 1, reports errors with suggestions
149
+ - `kotoshu check clean.txt` exits 0
150
+ - en_US Hunspell fixtures: 100 % pass
151
+ - No system dictionary dependency (no `/usr/share/dict/words` read)
152
+ - `Kotoshu.check(text, offline: true)` raises clear error if not cached
153
+ - `ENV['KOTOSHU_OFFLINE']` == `"1"` respected throughout
154
+ - README accurately reflects 0.2 capabilities (no false claims)
155
+ - `bundle exec rspec` green; `bundle exec rubocop` clean
156
+
157
+ ## Dependencies
158
+
159
+ - **Blocked by:** nothing — 0.1.0 is shipped.
160
+ - **Blocks:** `04-language-modules` (needs `ResourceManager`),
161
+ `05-semantic-path` (needs `ResourceManager` for model resolution).
162
+
163
+ ## Out of scope for 0.2.0
164
+
165
+ Everything in the "deferred" table above. Additionally:
166
+ - The Hunspell MECE 4-layer refactor (`01` task 6) — patch, don't
167
+ rewrite, for 0.2.
168
+ - `DictCommand` / `CacheCommand` relocation to `commands/` (`02`
169
+ task 2) — cosmetic, defer.
170
+ - `model` subcommand (`02` task 3) — needs ONNX, defer to 0.4.
171
+ - `--from-stdin` (`02` task 4) — nice-to-have, defer to 0.3.
172
+
173
+ ## Status
174
+
175
+ Starting now. Phase A (`ResourceManager`) is the first task.
@@ -173,6 +173,21 @@ the Swedish COMPOUNDRULE bug by design.)
173
173
  | 96 | [Server semantic models](96-server-semantic-models.md) | /v1/check is dictionary-only — SDK users get no reranking through HTTP |
174
174
  | 97 | [YARD execution](97-yard-execution.md) | Executes plan 59: the Ruby API reference still does not exist |
175
175
 
176
+ Plan 98 is **wave 5, the cleanup and closeout** (2026-09-06): after the
177
+ release-day cuts (gem 0.8.0 → 0.9.0, @kotoshu/wasm 0.2.0, registry
178
+ v1.3.0 with nb, lsp/server 0.1.1 via trusted publishing) the workspace
179
+ was swept — 214 GB of models worktrees reclaimed, the plan corpus
180
+ 01–98 fully tracked, verification evidence preserved.
181
+
182
+ | # | Plan | What it records |
183
+ |---|---|---|
184
+ | 98 | [Campaign cleanup](98-campaign-cleanup.md) | The inventory, the executed sweep, the keep list, and the deliberately-unforced items |
185
+
186
+ Campaign state at close: every channel live and populated; the
187
+ convertible model pool exhausted at 55 languages; owner-side
188
+ remnants (PyPI registrations, marketplace PAT, nds/fi calls) recorded
189
+ in the plans, not pending work.
190
+
176
191
  Dependency order across the ecosystem track:
177
192
 
178
193
  ```
@@ -0,0 +1,111 @@
1
+ # 01 — Hunspell Correctness
2
+
3
+ ## Goal
4
+
5
+ Pass 100% of the Spylls-ported fixture suite in `spec/hunspell_tests/` and
6
+ the Hunspell reference fixtures under
7
+ `/Users/mulgogi/src/external/hunspell/tests/`. Baseline at the start of
8
+ this plan was **48/107 (45 %)** passing; current state per
9
+ `CHANGELOG.md` is **803/866 (92.7 %)** — close but not done.
10
+
11
+ ## Why
12
+
13
+ The lookup/affix/compound machinery is the foundation of the
14
+ traditional path; nothing downstream (suggestions, grammar,
15
+ multi-language) can be trusted until this is correct.
16
+
17
+ ## Current blockers
18
+
19
+ - `lib/kotoshu/algorithms/lookup.rb` is a procedural ~877-line Python
20
+ port; hash/array passing instead of domain models.
21
+ - Overlap between `Lookup`, `LookupBuilder`, `Lookuper` (not MECE).
22
+ - Affix data split between `AffReader`, `aff_data.rb`, `lookup.rb`.
23
+ - Missing: complex compounding, circumfix, ICONV/OCONV, German ß,
24
+ Turkish dotless-i, special-character handling.
25
+
26
+ ## Tasks
27
+
28
+ 1. **Audit current pass rate.** Run `bundle exec rspec spec/hunspell_tests/`
29
+ and `bundle exec rspec spec/integration/`. Record the actual number
30
+ and failing fixture names in "Status" below.
31
+ 2. **Compound rules.** Implement `COMPOUNDRULE`, `COMPOUNDFLAG`,
32
+ `COMPOUNDBEGIN/MIDDLE/END`, `COMPOUNDPERMFLAG`, `COMPOUNDMIN`,
33
+ `CHECKCOMPOUNDREP`, `CHECKCOMPOUNDCASE`, `CHECKCOMPOUNDTRIPLE`,
34
+ `CHECKCOMPOUNDDUP`, `FORBIDDENWORD` inside compounds.
35
+ 3. **Circumfix.** Bidirectional prefix+suffix via `CIRCUMFIX` flag.
36
+ 4. **ICONV / OCONV.** Input/output character conversion tables applied
37
+ at the right point in the pipeline.
38
+ 5. **Special characters.** German ß uppercasing, Turkish dotless-i,
39
+ Dutch IJ, Break patterns (`BREAK`), `IGNORE` chars, `CHECKSHARPS`,
40
+ Arabic character ignoring, UTF-8 BOM, legacy encodings
41
+ (Windows-1251).
42
+ 6. **MECE refactor of the lookup pipeline.** Replace the procedural
43
+ `lookup.rb` with a four-layer separation (per the reimplementation
44
+ plan):
45
+ - **Dictionary layer** (data structures): `WordRepository`,
46
+ `AffixRepository`, `CompoundRuleRepository`,
47
+ `MorphologicalAnalyzer`.
48
+ - **Lookup layer** (algorithms): `WordLookup`, `AffixLookup`,
49
+ `CompoundLookup`, `MorphologicalLookup`.
50
+ - **Validation layer** (business rules): `ConditionValidator`,
51
+ `FlagValidator`, `CompoundValidator`,
52
+ `CapitalizationValidator`.
53
+ - **Transformation layer** (operations): `AffixApplier`,
54
+ `AffixStripper`, `WordFormer`, `CompoundFormer`.
55
+ Collapse `Lookup`/`LookupBuilder`/`Lookuper` into one responsibility
56
+ boundary or document why three classes exist.
57
+ 7. **Domain-model completion.** Verify `Word`, `AffixRule`,
58
+ `CompoundWord`, `Condition` carry the behavior, not the services
59
+ (per the global "models first" rule). Query methods required:
60
+ `word.stem`, `word.has_flag?(:verb)`, `word.forms`,
61
+ `affix_rule.valid_for?(stem)`, `affix_rule.apply(stem)`,
62
+ `affix_rule.can_combine?(other)`, `compound.valid_at?(position)`,
63
+ `condition.matches?(stem)`.
64
+ 8. **Strangler-Fig migration.** Build the new pipeline alongside the
65
+ old; expose a configuration switch (e.g.
66
+ `Configuration.use_modern_lookup` defaulting to `false`) so the
67
+ cutover is per-language and reversible. Remove the old
68
+ `algorithms/lookup.rb` only after the new path passes every fixture
69
+ the old one did.
70
+ 9. **Port the failing Spylls integrational fixtures** as named spec
71
+ examples: `dotless_i`, `IJ`, `ignore`, `checksharps`, `break`,
72
+ `needaffix`, `onlyincompound`, `alias`, `complexprefixes`,
73
+ `germancompounding`, `checkcompoundpattern`, plus the Hunspell
74
+ reference corpus fixtures (`1463589` Turkish UTF-8, `1592880`,
75
+ `1695964`, `1706659`, `1748408`, `breakdefault`, `breakoff`,
76
+ `encoding`, `utf8`, `utf8_bom`, `utf8_bom2`, `ignoreutf`,
77
+ `ignoresug`, `checksharpsutf`).
78
+ 10. **Performance baseline.** Profile lookup, affix application, and
79
+ compound check; add `spec/performance/lookup_bench.rb` with
80
+ assertions.
81
+
82
+ ## Acceptance criteria
83
+
84
+ - `bundle exec rspec spec/hunspell_tests/` is green (100 % of Spylls
85
+ fixtures).
86
+ - New specs added for every behavior fixed (no fixture passes
87
+ silently).
88
+ - `lib/kotoshu/algorithms/lookup.rb` is under 400 lines, or its
89
+ responsibilities have been redistributed across the four layers
90
+ above with the boundary documented.
91
+ - **Performance:** < 100 ms average lookup time per 1000 words;
92
+ < 100 MB memory for the en_US dictionary.
93
+ - **Code quality:** ≤ 50 lines per method, ≤ 300 lines per file, no
94
+ primitive obsession (domain models, not raw hashes), SRP followed,
95
+ dependencies on abstractions.
96
+ - No `double()` in any new spec (project rule — see `CLAUDE.md`).
97
+
98
+ ## Dependencies
99
+
100
+ - None inside this repo. Blocks `04-language-modules` (which assumes
101
+ a trustworthy Hunspell path) and `11-release`.
102
+
103
+ ## Out of scope
104
+
105
+ - Suggestion quality tuning (covered in `05-semantic-path`).
106
+ - Grammar rules (covered in `08-grammar-engine`).
107
+ - New languages beyond what the existing fixture suite covers.
108
+
109
+ ## Status
110
+
111
+ _Pending — fill in actual pass rate after task 1._
@@ -0,0 +1,74 @@
1
+ # 02 — CLI Unification
2
+
3
+ ## Goal
4
+
5
+ One canonical CLI surface that exposes every feature the gem supports, with
6
+ flags that match the README.
7
+
8
+ ## Why
9
+
10
+ There are two competing implementations:
11
+
12
+ - `lib/kotoshu/cli.rb` — `Kotoshu::Cli::Cli < Thor`. This is what
13
+ `exe/kotoshu` actually starts. Has only a basic `check TARGET` command
14
+ and `dict`/`cache` subcommands.
15
+ - `lib/kotoshu/commands/check_command.rb` — `Kotoshu::CheckCommand < Thor`
16
+ with `namespace :check`, exposes `--language auto`, `--interactive`,
17
+ `--output`, `--format text|json|yaml|csv|sarif`, `--model fasttext|hunspell`,
18
+ `--download`. **Not registered as a subcommand anywhere — dead code.**
19
+
20
+ The README advertises `--interactive`, `--format sarif`, `--model`,
21
+ `--language auto`. Users running `bundle exec exe/kotoshu check README.md
22
+ --interactive` get an "unknown option" error.
23
+
24
+ ## Tasks
25
+
26
+ 1. **Decide canonical location.** Recommendation: keep the rich
27
+ `Kotoshu::CheckCommand` (it has all the features users want) and delete
28
+ the basic `check` from `cli.rb`. Wire `cli.rb` to register
29
+ `CheckCommand` via `subcommand "check", Kotoshu::CheckCommand` (or
30
+ register at top level if `check` is the only command).
31
+ 2. **Move `DictCommand` and `CacheCommand`** into `lib/kotoshu/commands/`
32
+ alongside `CheckCommand` so all CLI commands live in one directory.
33
+ 3. **Add a `model` subcommand** so users can manage ONNX models
34
+ (`kotoshu model download en`, `kotoshu model list`,
35
+ `kotoshu model validate`, `kotoshu model convert`). The README
36
+ references these — they live in `lib/kotoshu/commands/model_command.rb`
37
+ already; verify it's wired.
38
+ 4. **Add `--from-stdin` mode** so the CLI can be used in shell pipes:
39
+ `echo "wrold" | kotoshu check --from-stdin`.
40
+ 5. **Reconcile flag names** between README and code. Specifically:
41
+ - README says `--dictionary=unix_words`; `cli.rb` has both `--dictionary`
42
+ and `--dictionary-path` — pick one.
43
+ - README says `--output=json`; `CheckCommand` uses `--format=json
44
+ --output=FILE` — clarify the two-flag split.
45
+ 6. **Exit codes.** Define and document: 0 = clean, 1 = errors found, 2 =
46
+ usage error, 3 = dictionary/model fetch failure. SARIF consumers
47
+ (GitHub Code Scanning) require this contract.
48
+ 7. **Help text audit.** Every Thor `desc` should match what's in the
49
+ README, and the README should match what's in code.
50
+
51
+ ## Acceptance criteria
52
+
53
+ - `bundle exec exe/kotoshu check README.md --interactive` works as
54
+ documented
55
+ - `bundle exec exe/kotoshu check README.md --format sarif --output
56
+ results.sarif` produces valid SARIF 2.1.0 that GitHub accepts
57
+ - `bundle exec exe/kotoshu --help` lists every command the README mentions
58
+ - No duplicate command classes; one canonical path per command
59
+ - A spec under `spec/kotoshu/cli/` exercises each subcommand end-to-end
60
+ (using a temp dir, not real `~/.kotoshu`)
61
+
62
+ ## Dependencies
63
+
64
+ - Light dependency on `03-dynamic-download` (CLI should let users
65
+ pre-download resources), but can proceed independently.
66
+
67
+ ## Out of scope
68
+
69
+ - LSP server, HTTP server (future work, not v1)
70
+ - GUI / TUI redesign — keep the existing Thor UX
71
+
72
+ ## Status
73
+
74
+ _Pending._