kotoshu 0.8.0 → 0.9.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +53 -0
- data/README.adoc +1 -1
- data/TODO.impl/00-cut-0.1.md +121 -0
- data/TODO.impl/00-cut-0.2.md +175 -0
- data/TODO.impl/00-vision.md +15 -0
- data/TODO.impl/01-hunspell-correctness.md +111 -0
- data/TODO.impl/02-cli-unification.md +74 -0
- data/TODO.impl/03-dynamic-download.md +107 -0
- data/TODO.impl/04-language-modules.md +89 -0
- data/TODO.impl/05-semantic-path.md +83 -0
- data/TODO.impl/06-cjk-support.md +105 -0
- data/TODO.impl/07-rtl-support.md +71 -0
- data/TODO.impl/08-grammar-engine.md +106 -0
- data/TODO.impl/09-integrity-security.md +90 -0
- data/TODO.impl/10-testing-ci.md +98 -0
- data/TODO.impl/11-release-v1.md +95 -0
- data/TODO.impl/20-encapsulation-cleanup.md +47 -0
- data/TODO.impl/21-autoload-migration.md +64 -0
- data/TODO.impl/33-audit-log-rotation.md +54 -0
- data/TODO.impl/34-cache-eviction.md +59 -0
- data/TODO.impl/35-shell-completion.md +61 -0
- data/TODO.impl/40-spec-drift-cleanup.md +134 -0
- data/TODO.impl/50-structure-aware-document-api.md +194 -0
- data/TODO.impl/51-grammar-engine-expansion.md +75 -0
- data/TODO.impl/52-multi-language-documents.md +95 -0
- data/TODO.impl/53-document-plugin-architecture.md +94 -0
- data/TODO.impl/54-cjk-support.md +117 -0
- data/TODO.impl/55-rtl-support.md +109 -0
- data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
- data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
- data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
- data/TODO.impl/59-yard-api-documentation.md +75 -0
- data/TODO.impl/60-lsp-server.md +174 -0
- data/TODO.impl/61-editor-ecosystem.md +167 -0
- data/TODO.impl/62-framework-integrations.md +159 -0
- data/TODO.impl/63-web-playground-wasm.md +186 -0
- data/TODO.impl/64-http-api-and-sdks.md +191 -0
- data/TODO.impl/98-campaign-cleanup.md +78 -0
- data/conformance/vectors.jsonl +84 -84
- data/docs/verification-evidence/correct3/pr_body.md +37 -0
- data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
- data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
- data/docs/verification-evidence/correct3/sv.aff +856 -0
- data/docs/verification-evidence/correct3/sv.dic +153717 -0
- data/lib/kotoshu/algorithms/edit_distance.rb +6 -3
- data/lib/kotoshu/cache/base_cache.rb +33 -0
- data/lib/kotoshu/cache/frequency_cache.rb +28 -24
- data/lib/kotoshu/cache/language_cache.rb +23 -16
- data/lib/kotoshu/cache/model_cache.rb +44 -41
- data/lib/kotoshu/cli/cache_command.rb +4 -0
- data/lib/kotoshu/configuration.rb +5 -1
- data/lib/kotoshu/data/common_words_loader.rb +21 -31
- data/lib/kotoshu/dictionary/base.rb +12 -0
- data/lib/kotoshu/dictionary/custom.rb +6 -6
- data/lib/kotoshu/dictionary/hunspell.rb +8 -0
- data/lib/kotoshu/dictionary/plain_text.rb +8 -5
- data/lib/kotoshu/dictionary/unix_words.rb +5 -5
- data/lib/kotoshu/integrity/net_http.rb +5 -0
- data/lib/kotoshu/languages/nb/language.rb +25 -0
- data/lib/kotoshu/languages.rb +1 -0
- data/lib/kotoshu/resource_manager.rb +5 -3
- data/lib/kotoshu/suggestions/frequency_provider.rb +13 -4
- data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +115 -0
- data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +26 -47
- data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +25 -9
- data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +20 -38
- data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
- data/lib/kotoshu/suggestions.rb +1 -0
- data/lib/kotoshu/version.rb +1 -1
- metadata +44 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: c1082e88abf75329c969089181472e5a1e511bbb48b7f1c7ab4c113b5e6fc310
|
|
4
|
+
data.tar.gz: 92cb046340c93ffdf72f7c9c69eb776a683757723020a96b3c4805416c43d2f7
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: f5aee6be1e7c35c0bb8f3e8bcac0aa089e1caac0bd8cb0e05925c74f5e4a925fd5e0206ba22759421140e96fcfc5610596bd3bdfe14fde489b1117cc46969c15
|
|
7
|
+
data.tar.gz: a62aa1ec6ddbd3cc499958ad45b07fd281c5f1b919b7692960a1b59602fd3bbc8352f2d769946e91518de29e7e2cc620c3e2cba818e19fcd4eb295a48af0ddd7
|
data/CHANGELOG.md
CHANGED
|
@@ -8,6 +8,59 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
10
|
### Fixed
|
|
11
|
+
- **Suggestion candidate sweep missed transpositions, substitutions, and
|
|
12
|
+
dictionary forms** - `EditDistanceStrategy` only scored raw dictionary
|
|
13
|
+
stems, so valid dictionary forms could never be suggested no matter how
|
|
14
|
+
close: an adjacent transposition paired with a case difference ("Teh" ->
|
|
15
|
+
"The") was charged as two operations, and affixed forms ("definately" ->
|
|
16
|
+
"definitely", a stem + `-ly` suffix never present as a stem) were absent
|
|
17
|
+
from the candidate set entirely. The strategy now also sweeps the
|
|
18
|
+
distance-1 edits of the misspelling itself - adjacent transposition
|
|
19
|
+
(restricted Damerau operation, cost 1), substitution and insertion over
|
|
20
|
+
the dictionary TRY string, and deletion - validating each against the
|
|
21
|
+
full affix-aware dictionary lookup, and case-variant duplicates of the
|
|
22
|
+
same word are deduplicated keeping the best-scoring form. The phonetic
|
|
23
|
+
and keyboard-proximity strategies now measure candidates with
|
|
24
|
+
`Algorithms::EditDistance` (Damerau-Levenshtein; previously each carried
|
|
25
|
+
a private plain-Levenshtein copy charging transpositions 2), and the
|
|
26
|
+
keyboard variant generator enumerates adjacent transpositions as a
|
|
27
|
+
single edit. `Dictionary::Base#try_string` / `Dictionary::Hunspell#try_string`
|
|
28
|
+
expose the Hunspell TRY directive for the sweep alphabet. Conformance
|
|
29
|
+
suggest vectors regenerated (84 of 1315 rows improved - affixed, compound,
|
|
30
|
+
and correctly-cased forms now appear; all 1315 correct vectors unchanged).
|
|
31
|
+
- **Dictionary mutation correctness** (from suleman-uzair, PR rescue of #93) -
|
|
32
|
+
`remove_word` deleted the wrong word after any prior removal because the
|
|
33
|
+
word set stored array indices that went stale; `Custom`, `PlainText`, and
|
|
34
|
+
`UnixWords` now track membership (word => true) and remove by value, so
|
|
35
|
+
every copy of a duplicate word is gone and `words` / `include?` stay
|
|
36
|
+
consistent across interleaved add/remove sequences. `PlainText#add_word` /
|
|
37
|
+
`#remove_word` now also keep the length-bucketed index in sync, so
|
|
38
|
+
`find_by_length_range` (the edit-distance candidate filter) reflects
|
|
39
|
+
post-construction mutations instead of returning stale buckets.
|
|
40
|
+
- **Edit-distance threshold on empty strings** (from suleman-uzair, PR rescue
|
|
41
|
+
of #93) - `Algorithms::EditDistance.distance_with_threshold` returned the
|
|
42
|
+
other string's length for empty inputs even when that exceeded the
|
|
43
|
+
threshold; the length pre-filter now runs before the empty-string
|
|
44
|
+
shortcuts, so the nil-above-threshold contract holds for empty strings.
|
|
45
|
+
- **Offline mode enforcement and cache-only suggestion hot path** (from
|
|
46
|
+
suleman-uzair, PR rescue of #93) - `Kotoshu::Configuration.offline` /
|
|
47
|
+
`KOTOSHU_OFFLINE=1` previously did not stop downloads. Every cache
|
|
48
|
+
download path now refuses in offline mode (`BaseCache#download` and the
|
|
49
|
+
`download_url` / `download_file` transports raise
|
|
50
|
+
`ResourceNotCachedError`; `Integrity::NetHTTP.get` refuses before opening
|
|
51
|
+
a connection; `LanguageCache#download_spelling` / `#download_grammar`
|
|
52
|
+
raise). The resolve paths (`ResourceManager`, `FrequencyProvider`) read
|
|
53
|
+
via the public cache-only `load_cached` instead of the download-on-miss
|
|
54
|
+
`get`, so the suggestion hot path never triggers a download.
|
|
55
|
+
`Configuration.default` no longer re-applies `DEFAULTS` as explicit
|
|
56
|
+
settings, which clobbered every `KOTOSHU_*` ENV-derived value including
|
|
57
|
+
`KOTOSHU_OFFLINE`. `kotoshu cache download` surfaces offline and
|
|
58
|
+
integrity failures as `ResourceUnavailable` instead of a stack trace.
|
|
59
|
+
- **Kelly frequency loader dead branch** (from suleman-uzair, PR rescue of
|
|
60
|
+
#93) - `CommonWordsLoader.load_from_frequency_file` kept a legacy
|
|
61
|
+
array-format branch that crashed on its own probe and had no producer;
|
|
62
|
+
it now parses the Kelly shape only and degrades unknown shapes to empty
|
|
63
|
+
tiers.
|
|
11
64
|
- Remote `kotoshu setup` for staged languages: the dictionaries repo ships only
|
|
12
65
|
`en` under the `{lang}/spelling/` layout while staged languages sit flat at
|
|
13
66
|
`{lang}/index.*`; downloads now try the sublayout first and fall back to the
|
data/README.adoc
CHANGED
|
@@ -15,7 +15,7 @@ directory layout, SHA-256 integrity verification, SARIF output, an
|
|
|
15
15
|
|
|
16
16
|
* Two-stage resource model — `Kotoshu.setup(:en)` then `Kotoshu.correct?("hello")`. The hot path is cache-only and raises `ResourceNotSetupError` on miss; downloads are never implicit.
|
|
17
17
|
* `Kotoshu.check(text, language: "en")` / `Kotoshu.suggest("helo")` — full document check and suggestions
|
|
18
|
-
* `Kotoshu.spellchecker_for(lang
|
|
18
|
+
* `Kotoshu.spellchecker_for(lang)` — per-language checker resolved from the cache (`strict:` is a `Kotoshu.setup` option)
|
|
19
19
|
* `kotoshu check FILE` CLI with these flags:
|
|
20
20
|
** `--language en|de|es|fr|pt|ru|auto` (default: `auto`)
|
|
21
21
|
** `--format text|json|sarif`
|
|
@@ -0,0 +1,121 @@
|
|
|
1
|
+
# 00-cut-0.1 — The 0.1.0 Release Cut
|
|
2
|
+
|
|
3
|
+
## Goal
|
|
4
|
+
|
|
5
|
+
Ship `kotoshu-0.1.0.gem` to rubygems.org with a clear, honest scope: the
|
|
6
|
+
pure-Ruby API and CLI work for English via system dictionaries; everything
|
|
7
|
+
else is documented as in-progress.
|
|
8
|
+
|
|
9
|
+
This is **not** v1.0. The point of 0.1 is to be installable, runnable,
|
|
10
|
+
and discoverable — to claim the name on rubygems.org and establish the
|
|
11
|
+
public API shape.
|
|
12
|
+
|
|
13
|
+
## 0.1.0 scope (locked)
|
|
14
|
+
|
|
15
|
+
In scope for 0.1.0:
|
|
16
|
+
|
|
17
|
+
| Capability | Status |
|
|
18
|
+
|---|---|
|
|
19
|
+
| `Kotoshu.correct?(word)` / `suggest` / `check` / `check_file` | ✅ Working |
|
|
20
|
+
| `Kotoshu.detect_language(text)` (FastText LID) | ✅ Working |
|
|
21
|
+
| `kotoshu check TARGET` basic CLI | ✅ Working |
|
|
22
|
+
| `kotoshu dict list/info`, `kotoshu cache list/info/...` | ✅ Working |
|
|
23
|
+
| Hunspell dictionary backend (Ruby port) | ✅ Working for common cases |
|
|
24
|
+
| CSpell, UnixWords, PlainText, Custom backends | ✅ Working |
|
|
25
|
+
| Edit distance / phonetic / keyboard / n-gram / symspell strategies | ✅ Working |
|
|
26
|
+
| Configuration via `Kotoshu.configure` + ENV (`KOTOSHU_*`) + CLI | ✅ Working |
|
|
27
|
+
| 3-layer cache (LanguageCache, FrequencyCache, ModelCache) | ✅ Working |
|
|
28
|
+
| Document parsing (Plain text, Markdown, AsciiDoc) | ✅ Working |
|
|
29
|
+
| LICENSE (BSD-2-Clause), CHANGELOG.md, accurate README | ✅ Shipped |
|
|
30
|
+
|
|
31
|
+
Explicitly **deferred** to later 0.x or 1.0:
|
|
32
|
+
|
|
33
|
+
| Capability | Target | Plan |
|
|
34
|
+
|---|---|---|
|
|
35
|
+
| Unified `ResourceManager` for "any language, just works" | 0.3 | `03-dynamic-download.md` |
|
|
36
|
+
| `--interactive`, `--format sarif`, `--model`, `--language auto` wired | 0.2 | `02-cli-unification.md` |
|
|
37
|
+
| Hybrid (Hunspell + ONNX) as default | 0.4 | `05-semantic-path.md` |
|
|
38
|
+
| Hunspell compound rules, circumfix, ICONV/OCONV, ß, dotless-i | 0.5 | `01-hunspell-correctness.md` |
|
|
39
|
+
| ≥30 language modules wired | 0.6 | `04-language-modules.md` |
|
|
40
|
+
| Grammar rule packs | 0.7 | `08-grammar-engine.md` |
|
|
41
|
+
| CJK (Japanese, Chinese) | 0.8 | `06-cjk-support.md` |
|
|
42
|
+
| RTL (Arabic, Hebrew, Persian, Urdu) | 0.9 | `07-rtl-support.md` |
|
|
43
|
+
| Integrity verification (checksums, signed manifests) | 0.4 | `09-integrity-security.md` |
|
|
44
|
+
| v1.0 | — | `11-release-v1.md` |
|
|
45
|
+
|
|
46
|
+
## Tasks executed for 0.1.0
|
|
47
|
+
|
|
48
|
+
1. ✅ Audit current state — `bundle install`, smoke-test `Kotoshu.correct?`,
|
|
49
|
+
run rspec. Result: 803/866 (92.7%) passing; VERSION already `0.1.0`.
|
|
50
|
+
2. ✅ Add `LICENSE` (BSD-2-Clause — was missing despite gemspec claim).
|
|
51
|
+
3. ✅ Add `CHANGELOG.md` with full 0.1.0 scope and known-limitation list.
|
|
52
|
+
4. ✅ Fix README accuracy: license, status section at top, ONNX env gate.
|
|
53
|
+
5. ✅ Tighten CI workflow — rubocop no longer `continue-on-error`.
|
|
54
|
+
6. ✅ Prepare release checklist (see "Release execution" below).
|
|
55
|
+
|
|
56
|
+
## Release execution (manual — requires maintainer)
|
|
57
|
+
|
|
58
|
+
The global project rule is "AI never pushes tags, never pushes to main,
|
|
59
|
+
never publishes." The following must be run by a human:
|
|
60
|
+
|
|
61
|
+
```bash
|
|
62
|
+
# 1. On a clean main, verify everything passes
|
|
63
|
+
git checkout main
|
|
64
|
+
git pull
|
|
65
|
+
bundle install
|
|
66
|
+
bundle exec rspec
|
|
67
|
+
bundle exec rubocop
|
|
68
|
+
|
|
69
|
+
# 2. Verify VERSION
|
|
70
|
+
grep VERSION lib/kotoshu/version.rb
|
|
71
|
+
# Should read: VERSION = "0.1.0"
|
|
72
|
+
|
|
73
|
+
# 3. Build the gem
|
|
74
|
+
gem build kotoshu.gemspec
|
|
75
|
+
# Produces kotoshu-0.1.0.gem
|
|
76
|
+
|
|
77
|
+
# 4. Sanity-check the gem contents
|
|
78
|
+
gem contents kotoshu-0.1.0.gem --tarball
|
|
79
|
+
|
|
80
|
+
# 5. Tag (the human decides when)
|
|
81
|
+
git tag -a v0.1.0 -m "kotoshu 0.1.0 — first public release"
|
|
82
|
+
git push origin v0.1.0
|
|
83
|
+
|
|
84
|
+
# 6. Publish to rubygems.org
|
|
85
|
+
gem push kotoshu-0.1.0.gem
|
|
86
|
+
|
|
87
|
+
# 7. Cut a GitHub Release
|
|
88
|
+
gh release create v0.1.0 --notes-from-tag
|
|
89
|
+
|
|
90
|
+
# 8. Smoke test from a clean environment
|
|
91
|
+
docker run --rm -it ruby:3.3 bash -c '
|
|
92
|
+
gem install kotoshu
|
|
93
|
+
ruby -e "require \"kotoshu\"; puts Kotoshu.correct?(\"hello\")"
|
|
94
|
+
echo "Hello wrold" | ruby -e "require \"kotoshu\"; puts Kotoshu.check(STDIN.read).errors.map(&:word)"
|
|
95
|
+
'
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
## Post-release
|
|
99
|
+
|
|
100
|
+
- Update `TODO.impl/00-vision.md` to reflect that 0.1 shipped; the
|
|
101
|
+
critical path now runs through `03-dynamic-download` →
|
|
102
|
+
`04-language-modules` → `11-release-v1`.
|
|
103
|
+
- Start `02-cli-unification.md` (next-highest user-visible value).
|
|
104
|
+
- Start `01-hunspell-correctness.md` task 1 (audit) to get an honest
|
|
105
|
+
fixture-pass number.
|
|
106
|
+
|
|
107
|
+
## Acceptance criteria
|
|
108
|
+
|
|
109
|
+
- `gem install kotoshu` works on Ruby 3.1+
|
|
110
|
+
- `Kotoshu.correct?("hello")` returns `true` on a fresh install with
|
|
111
|
+
no env vars set
|
|
112
|
+
- `Kotoshu.correct?("helo")` returns `false`
|
|
113
|
+
- `kotoshu check README.adoc` runs without crashing
|
|
114
|
+
- README has no false claims (license, model count, CLI flags)
|
|
115
|
+
- LICENSE and CHANGELOG.md exist
|
|
116
|
+
- rubygems.org page shows v0.1.0 with correct metadata
|
|
117
|
+
|
|
118
|
+
## Status
|
|
119
|
+
|
|
120
|
+
v0.1.0 release-prep complete on this branch (2026-06-25). Release
|
|
121
|
+
execution pending maintainer sign-off.
|
|
@@ -0,0 +1,175 @@
|
|
|
1
|
+
# 00-cut-0.2 — The First Functional Library
|
|
2
|
+
|
|
3
|
+
## Goal
|
|
4
|
+
|
|
5
|
+
Ship `kotoshu-0.2.0` — the first release where `gem install kotoshu`
|
|
6
|
+
works on any Ruby 3.1+ system with **no system dictionary dependency**.
|
|
7
|
+
One call, `Kotoshu.check("Hello wrold")`, auto-downloads the English
|
|
8
|
+
Hunspell dictionary on first use and returns correct results. The CLI
|
|
9
|
+
works end-to-end with proper exit codes.
|
|
10
|
+
|
|
11
|
+
0.1.0 proved the API shape. 0.2.0 delivers the promise: "just works,
|
|
12
|
+
no setup." For 0.2.0 the resolution machinery is English-only; the
|
|
13
|
+
code is language-agnostic so adding more languages is a content task
|
|
14
|
+
(shipping dictionaries in `kotoshu/dictionaries`), not a code change.
|
|
15
|
+
|
|
16
|
+
## 0.2.0 scope (locked)
|
|
17
|
+
|
|
18
|
+
| Capability | Source plan |
|
|
19
|
+
|---|---|
|
|
20
|
+
| `ResourceManager.resolve` (spelling + frequency) | `03` tasks 1–2 |
|
|
21
|
+
| `ResourceBundle` value object | `03` task 1 |
|
|
22
|
+
| en_US Hunspell auto-download on first use | `03` task 2 |
|
|
23
|
+
| Spellchecker accepts `ResourceBundle` (Strangler-Fig) | `03` |
|
|
24
|
+
| One canonical `CheckCommand` wired into CLI | `02` task 1 |
|
|
25
|
+
| Exit codes 0/1/2/3 | `02` task 6 |
|
|
26
|
+
| `--language en\|auto`, `--format text\|json` | `02` task 5 |
|
|
27
|
+
| en_US Hunspell fixtures at 100 % | `01` tasks 1–2 |
|
|
28
|
+
| Offline mode (`offline: true`, `KOTOSHU_OFFLINE=1`) | `03` task 4 |
|
|
29
|
+
| Fake-server integration spec | `03` task 9 |
|
|
30
|
+
|
|
31
|
+
Explicitly **deferred** to 0.3+:
|
|
32
|
+
|
|
33
|
+
| Capability | Target | Plan |
|
|
34
|
+
|---|---|---|
|
|
35
|
+
| Checksum / integrity verification | 0.3 | `09-integrity-security` |
|
|
36
|
+
| `--interactive`, `--format sarif` | 0.3 | `02` tasks 4, 5 |
|
|
37
|
+
| `kotoshu fetch` pre-warm command | 0.3 | `03` task 6 |
|
|
38
|
+
| Strict mode | 0.3 | `03` task 5 |
|
|
39
|
+
| ONNX model auto-download (semantic rerank) | 0.4 | `05-semantic-path` |
|
|
40
|
+
| ≥30 language modules wired | 0.6 | `04-language-modules` |
|
|
41
|
+
| Grammar rule packs | 0.7 | `08-grammar-engine` |
|
|
42
|
+
| CJK (Japanese, Chinese) | 0.8 | `06-cjk-support` |
|
|
43
|
+
| RTL (Arabic, Hebrew, Persian, Urdu) | 0.9 | `07-rtl-support` |
|
|
44
|
+
| Hunspell MECE 4-layer refactor | 1.0 | `01` task 6 |
|
|
45
|
+
|
|
46
|
+
## Critical path — three phases
|
|
47
|
+
|
|
48
|
+
### Phase A — ResourceManager (foundation)
|
|
49
|
+
|
|
50
|
+
The Spellchecker today takes a single `Dictionary::Base` the user must
|
|
51
|
+
configure manually. Phase A introduces `ResourceManager` as the thing
|
|
52
|
+
that produces that dictionary (plus frequency data) from a language
|
|
53
|
+
code, using the existing `LanguageCache`.
|
|
54
|
+
|
|
55
|
+
1. **`ResourceBundle`** (`lib/kotoshu/resource_bundle.rb`). Plain
|
|
56
|
+
value object — Struct or class with `language`, `dictionary`,
|
|
57
|
+
`frequency`, `model` (nil for 0.2), `rules` (nil for 0.2),
|
|
58
|
+
`cached`, `source_urls`. No hand-rolled `to_h` (project rule).
|
|
59
|
+
2. **`ResourceManager`** (`lib/kotoshu/resource_manager.rb`,
|
|
60
|
+
autoloaded from `lib/kotoshu.rb`). API:
|
|
61
|
+
```ruby
|
|
62
|
+
Kotoshu::ResourceManager.resolve(
|
|
63
|
+
text: "Hello wrold",
|
|
64
|
+
language: :auto, # or "en", "de", nil
|
|
65
|
+
want: %i[spelling frequency],
|
|
66
|
+
offline: false
|
|
67
|
+
) # => ResourceBundle
|
|
68
|
+
```
|
|
69
|
+
Internals: detect language via `Language::Identifier` when `:auto`;
|
|
70
|
+
fetch dictionary via `LanguageCache.get(language)`; fetch frequency
|
|
71
|
+
via `FrequencyCache.get(language)` when wanted; construct
|
|
72
|
+
`ResourceBundle`. No new network code — the caches already
|
|
73
|
+
download from `kotoshu/dictionaries` and `kotoshu/frequency-list-kelly`.
|
|
74
|
+
3. **Spellchecker accepts `ResourceBundle`.** Modify
|
|
75
|
+
`lib/kotoshu/spellchecker.rb` to accept either a `ResourceBundle`
|
|
76
|
+
(new path) or a `Dictionary::Base` (legacy path). Strangler-Fig —
|
|
77
|
+
don't break the 0.1 API. Extract `dictionary` and `frequency` from
|
|
78
|
+
the bundle; ignore `model`/`rules` (nil for 0.2).
|
|
79
|
+
4. **First-use UX.** When `LanguageCache` downloads, print
|
|
80
|
+
`[en] downloading dictionary (X.X MB)…` to STDERR. One line per
|
|
81
|
+
resource, not per byte. Suppress when `$stdout.tty?` is false and
|
|
82
|
+
`KOTOSHU_QUIET` is set.
|
|
83
|
+
5. **Configuration.** Add to `Configuration::SCHEMA`:
|
|
84
|
+
- `offline` (boolean, default `false`) — also via `KOTOSHU_OFFLINE=1`
|
|
85
|
+
- `default_language` (string, default `"en"`) — used when
|
|
86
|
+
`language: nil` and detection is inconclusive
|
|
87
|
+
6. **Offline mode.** `ResourceManager.resolve(..., offline: true)`
|
|
88
|
+
only uses cached resources; raises `ResourceNotCachedError` with a
|
|
89
|
+
message naming what's missing and how to pre-fetch.
|
|
90
|
+
|
|
91
|
+
**Phase A acceptance:** `Kotoshu.correct?("hello")` works on a Docker
|
|
92
|
+
`ruby:3.3` container with no `/usr/share/dict/words`, auto-downloading
|
|
93
|
+
en_US on first call.
|
|
94
|
+
|
|
95
|
+
### Phase B — English correctness
|
|
96
|
+
|
|
97
|
+
7. **Audit en_US fixtures.** Run
|
|
98
|
+
`bundle exec rspec spec/hunspell_tests/ --tag lang:en` (or filter
|
|
99
|
+
by fixture path). Record which examples fail and categorize:
|
|
100
|
+
compound words, capitalization, affix combinations, special
|
|
101
|
+
characters. This is `01` task 1 scoped to en_US only.
|
|
102
|
+
8. **Fix en_US failures.** Target the categories from the audit. The
|
|
103
|
+
full MECE 4-layer refactor (`01` task 6) is explicitly **not**
|
|
104
|
+
required for 0.2 — patch the procedural `algorithms/lookup.rb`
|
|
105
|
+
where needed. Each fix gets a named spec example.
|
|
106
|
+
9. **Suggestion sanity.** Verify `Kotoshu.suggest("helo").first.word`
|
|
107
|
+
== `"hello"` for the top-20 common misspellings. Tune
|
|
108
|
+
`EditDistanceStrategy` weight or `frequency_bonus` if the top
|
|
109
|
+
suggestion is wrong.
|
|
110
|
+
|
|
111
|
+
**Phase B acceptance:** 100 % of en_US Hunspell fixtures pass.
|
|
112
|
+
`suggest("helo")[0]` == `"hello"`.
|
|
113
|
+
|
|
114
|
+
### Phase C — CLI wired
|
|
115
|
+
|
|
116
|
+
10. **Wire `CheckCommand` as canonical `check`.** Remove the basic
|
|
117
|
+
`check` from `lib/kotoshu/cli.rb`; register
|
|
118
|
+
`Kotoshu::CheckCommand` (from `commands/check_command.rb`) via
|
|
119
|
+
`subcommand "check", Kotoshu::CheckCommand`. This is `02` task 1.
|
|
120
|
+
11. **Exit codes** (`02` task 6):
|
|
121
|
+
- `0` — no errors found
|
|
122
|
+
- `1` — errors found
|
|
123
|
+
- `2` — usage error
|
|
124
|
+
- `3` — dictionary fetch failure (network down, not cached + offline)
|
|
125
|
+
12. **Flag reconciliation** (`02` task 5). For 0.2, support exactly:
|
|
126
|
+
- `kotoshu check FILE` — default check, text output
|
|
127
|
+
- `--language en|de|...|auto` — language selection (`auto` =
|
|
128
|
+
detect via `Language::Identifier`)
|
|
129
|
+
- `--format text|json` — output format (sarif deferred to 0.3)
|
|
130
|
+
- `--offline` — use cached resources only
|
|
131
|
+
Remove or hide every other flag the README mentions but code
|
|
132
|
+
doesn't support yet. Update README to match.
|
|
133
|
+
13. **End-to-end CLI spec.** Under `spec/kotoshu/cli/`, using a temp
|
|
134
|
+
dir (not real `~/.kotoshu`): write a file with known misspellings,
|
|
135
|
+
run `kotoshu check`, assert exit code + output content for both
|
|
136
|
+
`text` and `json` formats.
|
|
137
|
+
|
|
138
|
+
**Phase C acceptance:** `kotoshu check README.adoc` exits 1 with
|
|
139
|
+
readable error output; `kotoshu check clean_file.txt` exits 0.
|
|
140
|
+
|
|
141
|
+
## Release-level acceptance criteria
|
|
142
|
+
|
|
143
|
+
- Fresh `docker run --rm ruby:3.3` → `gem install kotoshu` →
|
|
144
|
+
`ruby -e "require 'kotoshu'; puts Kotoshu.correct?('hello')"`
|
|
145
|
+
prints `true` (auto-downloads en_US on first call)
|
|
146
|
+
- `Kotoshu.correct?("helo")` returns `false`
|
|
147
|
+
- `Kotoshu.suggest("helo").first.word` == `"hello"`
|
|
148
|
+
- `kotoshu check README.adoc` exits 1, reports errors with suggestions
|
|
149
|
+
- `kotoshu check clean.txt` exits 0
|
|
150
|
+
- en_US Hunspell fixtures: 100 % pass
|
|
151
|
+
- No system dictionary dependency (no `/usr/share/dict/words` read)
|
|
152
|
+
- `Kotoshu.check(text, offline: true)` raises clear error if not cached
|
|
153
|
+
- `ENV['KOTOSHU_OFFLINE']` == `"1"` respected throughout
|
|
154
|
+
- README accurately reflects 0.2 capabilities (no false claims)
|
|
155
|
+
- `bundle exec rspec` green; `bundle exec rubocop` clean
|
|
156
|
+
|
|
157
|
+
## Dependencies
|
|
158
|
+
|
|
159
|
+
- **Blocked by:** nothing — 0.1.0 is shipped.
|
|
160
|
+
- **Blocks:** `04-language-modules` (needs `ResourceManager`),
|
|
161
|
+
`05-semantic-path` (needs `ResourceManager` for model resolution).
|
|
162
|
+
|
|
163
|
+
## Out of scope for 0.2.0
|
|
164
|
+
|
|
165
|
+
Everything in the "deferred" table above. Additionally:
|
|
166
|
+
- The Hunspell MECE 4-layer refactor (`01` task 6) — patch, don't
|
|
167
|
+
rewrite, for 0.2.
|
|
168
|
+
- `DictCommand` / `CacheCommand` relocation to `commands/` (`02`
|
|
169
|
+
task 2) — cosmetic, defer.
|
|
170
|
+
- `model` subcommand (`02` task 3) — needs ONNX, defer to 0.4.
|
|
171
|
+
- `--from-stdin` (`02` task 4) — nice-to-have, defer to 0.3.
|
|
172
|
+
|
|
173
|
+
## Status
|
|
174
|
+
|
|
175
|
+
Starting now. Phase A (`ResourceManager`) is the first task.
|
data/TODO.impl/00-vision.md
CHANGED
|
@@ -173,6 +173,21 @@ the Swedish COMPOUNDRULE bug by design.)
|
|
|
173
173
|
| 96 | [Server semantic models](96-server-semantic-models.md) | /v1/check is dictionary-only — SDK users get no reranking through HTTP |
|
|
174
174
|
| 97 | [YARD execution](97-yard-execution.md) | Executes plan 59: the Ruby API reference still does not exist |
|
|
175
175
|
|
|
176
|
+
Plan 98 is **wave 5, the cleanup and closeout** (2026-09-06): after the
|
|
177
|
+
release-day cuts (gem 0.8.0 → 0.9.0, @kotoshu/wasm 0.2.0, registry
|
|
178
|
+
v1.3.0 with nb, lsp/server 0.1.1 via trusted publishing) the workspace
|
|
179
|
+
was swept — 214 GB of models worktrees reclaimed, the plan corpus
|
|
180
|
+
01–98 fully tracked, verification evidence preserved.
|
|
181
|
+
|
|
182
|
+
| # | Plan | What it records |
|
|
183
|
+
|---|---|---|
|
|
184
|
+
| 98 | [Campaign cleanup](98-campaign-cleanup.md) | The inventory, the executed sweep, the keep list, and the deliberately-unforced items |
|
|
185
|
+
|
|
186
|
+
Campaign state at close: every channel live and populated; the
|
|
187
|
+
convertible model pool exhausted at 55 languages; owner-side
|
|
188
|
+
remnants (PyPI registrations, marketplace PAT, nds/fi calls) recorded
|
|
189
|
+
in the plans, not pending work.
|
|
190
|
+
|
|
176
191
|
Dependency order across the ecosystem track:
|
|
177
192
|
|
|
178
193
|
```
|
|
@@ -0,0 +1,111 @@
|
|
|
1
|
+
# 01 — Hunspell Correctness
|
|
2
|
+
|
|
3
|
+
## Goal
|
|
4
|
+
|
|
5
|
+
Pass 100% of the Spylls-ported fixture suite in `spec/hunspell_tests/` and
|
|
6
|
+
the Hunspell reference fixtures under
|
|
7
|
+
`/Users/mulgogi/src/external/hunspell/tests/`. Baseline at the start of
|
|
8
|
+
this plan was **48/107 (45 %)** passing; current state per
|
|
9
|
+
`CHANGELOG.md` is **803/866 (92.7 %)** — close but not done.
|
|
10
|
+
|
|
11
|
+
## Why
|
|
12
|
+
|
|
13
|
+
The lookup/affix/compound machinery is the foundation of the
|
|
14
|
+
traditional path; nothing downstream (suggestions, grammar,
|
|
15
|
+
multi-language) can be trusted until this is correct.
|
|
16
|
+
|
|
17
|
+
## Current blockers
|
|
18
|
+
|
|
19
|
+
- `lib/kotoshu/algorithms/lookup.rb` is a procedural ~877-line Python
|
|
20
|
+
port; hash/array passing instead of domain models.
|
|
21
|
+
- Overlap between `Lookup`, `LookupBuilder`, `Lookuper` (not MECE).
|
|
22
|
+
- Affix data split between `AffReader`, `aff_data.rb`, `lookup.rb`.
|
|
23
|
+
- Missing: complex compounding, circumfix, ICONV/OCONV, German ß,
|
|
24
|
+
Turkish dotless-i, special-character handling.
|
|
25
|
+
|
|
26
|
+
## Tasks
|
|
27
|
+
|
|
28
|
+
1. **Audit current pass rate.** Run `bundle exec rspec spec/hunspell_tests/`
|
|
29
|
+
and `bundle exec rspec spec/integration/`. Record the actual number
|
|
30
|
+
and failing fixture names in "Status" below.
|
|
31
|
+
2. **Compound rules.** Implement `COMPOUNDRULE`, `COMPOUNDFLAG`,
|
|
32
|
+
`COMPOUNDBEGIN/MIDDLE/END`, `COMPOUNDPERMFLAG`, `COMPOUNDMIN`,
|
|
33
|
+
`CHECKCOMPOUNDREP`, `CHECKCOMPOUNDCASE`, `CHECKCOMPOUNDTRIPLE`,
|
|
34
|
+
`CHECKCOMPOUNDDUP`, `FORBIDDENWORD` inside compounds.
|
|
35
|
+
3. **Circumfix.** Bidirectional prefix+suffix via `CIRCUMFIX` flag.
|
|
36
|
+
4. **ICONV / OCONV.** Input/output character conversion tables applied
|
|
37
|
+
at the right point in the pipeline.
|
|
38
|
+
5. **Special characters.** German ß uppercasing, Turkish dotless-i,
|
|
39
|
+
Dutch IJ, Break patterns (`BREAK`), `IGNORE` chars, `CHECKSHARPS`,
|
|
40
|
+
Arabic character ignoring, UTF-8 BOM, legacy encodings
|
|
41
|
+
(Windows-1251).
|
|
42
|
+
6. **MECE refactor of the lookup pipeline.** Replace the procedural
|
|
43
|
+
`lookup.rb` with a four-layer separation (per the reimplementation
|
|
44
|
+
plan):
|
|
45
|
+
- **Dictionary layer** (data structures): `WordRepository`,
|
|
46
|
+
`AffixRepository`, `CompoundRuleRepository`,
|
|
47
|
+
`MorphologicalAnalyzer`.
|
|
48
|
+
- **Lookup layer** (algorithms): `WordLookup`, `AffixLookup`,
|
|
49
|
+
`CompoundLookup`, `MorphologicalLookup`.
|
|
50
|
+
- **Validation layer** (business rules): `ConditionValidator`,
|
|
51
|
+
`FlagValidator`, `CompoundValidator`,
|
|
52
|
+
`CapitalizationValidator`.
|
|
53
|
+
- **Transformation layer** (operations): `AffixApplier`,
|
|
54
|
+
`AffixStripper`, `WordFormer`, `CompoundFormer`.
|
|
55
|
+
Collapse `Lookup`/`LookupBuilder`/`Lookuper` into one responsibility
|
|
56
|
+
boundary or document why three classes exist.
|
|
57
|
+
7. **Domain-model completion.** Verify `Word`, `AffixRule`,
|
|
58
|
+
`CompoundWord`, `Condition` carry the behavior, not the services
|
|
59
|
+
(per the global "models first" rule). Query methods required:
|
|
60
|
+
`word.stem`, `word.has_flag?(:verb)`, `word.forms`,
|
|
61
|
+
`affix_rule.valid_for?(stem)`, `affix_rule.apply(stem)`,
|
|
62
|
+
`affix_rule.can_combine?(other)`, `compound.valid_at?(position)`,
|
|
63
|
+
`condition.matches?(stem)`.
|
|
64
|
+
8. **Strangler-Fig migration.** Build the new pipeline alongside the
|
|
65
|
+
old; expose a configuration switch (e.g.
|
|
66
|
+
`Configuration.use_modern_lookup` defaulting to `false`) so the
|
|
67
|
+
cutover is per-language and reversible. Remove the old
|
|
68
|
+
`algorithms/lookup.rb` only after the new path passes every fixture
|
|
69
|
+
the old one did.
|
|
70
|
+
9. **Port the failing Spylls integrational fixtures** as named spec
|
|
71
|
+
examples: `dotless_i`, `IJ`, `ignore`, `checksharps`, `break`,
|
|
72
|
+
`needaffix`, `onlyincompound`, `alias`, `complexprefixes`,
|
|
73
|
+
`germancompounding`, `checkcompoundpattern`, plus the Hunspell
|
|
74
|
+
reference corpus fixtures (`1463589` Turkish UTF-8, `1592880`,
|
|
75
|
+
`1695964`, `1706659`, `1748408`, `breakdefault`, `breakoff`,
|
|
76
|
+
`encoding`, `utf8`, `utf8_bom`, `utf8_bom2`, `ignoreutf`,
|
|
77
|
+
`ignoresug`, `checksharpsutf`).
|
|
78
|
+
10. **Performance baseline.** Profile lookup, affix application, and
|
|
79
|
+
compound check; add `spec/performance/lookup_bench.rb` with
|
|
80
|
+
assertions.
|
|
81
|
+
|
|
82
|
+
## Acceptance criteria
|
|
83
|
+
|
|
84
|
+
- `bundle exec rspec spec/hunspell_tests/` is green (100 % of Spylls
|
|
85
|
+
fixtures).
|
|
86
|
+
- New specs added for every behavior fixed (no fixture passes
|
|
87
|
+
silently).
|
|
88
|
+
- `lib/kotoshu/algorithms/lookup.rb` is under 400 lines, or its
|
|
89
|
+
responsibilities have been redistributed across the four layers
|
|
90
|
+
above with the boundary documented.
|
|
91
|
+
- **Performance:** < 100 ms average lookup time per 1000 words;
|
|
92
|
+
< 100 MB memory for the en_US dictionary.
|
|
93
|
+
- **Code quality:** ≤ 50 lines per method, ≤ 300 lines per file, no
|
|
94
|
+
primitive obsession (domain models, not raw hashes), SRP followed,
|
|
95
|
+
dependencies on abstractions.
|
|
96
|
+
- No `double()` in any new spec (project rule — see `CLAUDE.md`).
|
|
97
|
+
|
|
98
|
+
## Dependencies
|
|
99
|
+
|
|
100
|
+
- None inside this repo. Blocks `04-language-modules` (which assumes
|
|
101
|
+
a trustworthy Hunspell path) and `11-release`.
|
|
102
|
+
|
|
103
|
+
## Out of scope
|
|
104
|
+
|
|
105
|
+
- Suggestion quality tuning (covered in `05-semantic-path`).
|
|
106
|
+
- Grammar rules (covered in `08-grammar-engine`).
|
|
107
|
+
- New languages beyond what the existing fixture suite covers.
|
|
108
|
+
|
|
109
|
+
## Status
|
|
110
|
+
|
|
111
|
+
_Pending — fill in actual pass rate after task 1._
|
|
@@ -0,0 +1,74 @@
|
|
|
1
|
+
# 02 — CLI Unification
|
|
2
|
+
|
|
3
|
+
## Goal
|
|
4
|
+
|
|
5
|
+
One canonical CLI surface that exposes every feature the gem supports, with
|
|
6
|
+
flags that match the README.
|
|
7
|
+
|
|
8
|
+
## Why
|
|
9
|
+
|
|
10
|
+
There are two competing implementations:
|
|
11
|
+
|
|
12
|
+
- `lib/kotoshu/cli.rb` — `Kotoshu::Cli::Cli < Thor`. This is what
|
|
13
|
+
`exe/kotoshu` actually starts. Has only a basic `check TARGET` command
|
|
14
|
+
and `dict`/`cache` subcommands.
|
|
15
|
+
- `lib/kotoshu/commands/check_command.rb` — `Kotoshu::CheckCommand < Thor`
|
|
16
|
+
with `namespace :check`, exposes `--language auto`, `--interactive`,
|
|
17
|
+
`--output`, `--format text|json|yaml|csv|sarif`, `--model fasttext|hunspell`,
|
|
18
|
+
`--download`. **Not registered as a subcommand anywhere — dead code.**
|
|
19
|
+
|
|
20
|
+
The README advertises `--interactive`, `--format sarif`, `--model`,
|
|
21
|
+
`--language auto`. Users running `bundle exec exe/kotoshu check README.md
|
|
22
|
+
--interactive` get an "unknown option" error.
|
|
23
|
+
|
|
24
|
+
## Tasks
|
|
25
|
+
|
|
26
|
+
1. **Decide canonical location.** Recommendation: keep the rich
|
|
27
|
+
`Kotoshu::CheckCommand` (it has all the features users want) and delete
|
|
28
|
+
the basic `check` from `cli.rb`. Wire `cli.rb` to register
|
|
29
|
+
`CheckCommand` via `subcommand "check", Kotoshu::CheckCommand` (or
|
|
30
|
+
register at top level if `check` is the only command).
|
|
31
|
+
2. **Move `DictCommand` and `CacheCommand`** into `lib/kotoshu/commands/`
|
|
32
|
+
alongside `CheckCommand` so all CLI commands live in one directory.
|
|
33
|
+
3. **Add a `model` subcommand** so users can manage ONNX models
|
|
34
|
+
(`kotoshu model download en`, `kotoshu model list`,
|
|
35
|
+
`kotoshu model validate`, `kotoshu model convert`). The README
|
|
36
|
+
references these — they live in `lib/kotoshu/commands/model_command.rb`
|
|
37
|
+
already; verify it's wired.
|
|
38
|
+
4. **Add `--from-stdin` mode** so the CLI can be used in shell pipes:
|
|
39
|
+
`echo "wrold" | kotoshu check --from-stdin`.
|
|
40
|
+
5. **Reconcile flag names** between README and code. Specifically:
|
|
41
|
+
- README says `--dictionary=unix_words`; `cli.rb` has both `--dictionary`
|
|
42
|
+
and `--dictionary-path` — pick one.
|
|
43
|
+
- README says `--output=json`; `CheckCommand` uses `--format=json
|
|
44
|
+
--output=FILE` — clarify the two-flag split.
|
|
45
|
+
6. **Exit codes.** Define and document: 0 = clean, 1 = errors found, 2 =
|
|
46
|
+
usage error, 3 = dictionary/model fetch failure. SARIF consumers
|
|
47
|
+
(GitHub Code Scanning) require this contract.
|
|
48
|
+
7. **Help text audit.** Every Thor `desc` should match what's in the
|
|
49
|
+
README, and the README should match what's in code.
|
|
50
|
+
|
|
51
|
+
## Acceptance criteria
|
|
52
|
+
|
|
53
|
+
- `bundle exec exe/kotoshu check README.md --interactive` works as
|
|
54
|
+
documented
|
|
55
|
+
- `bundle exec exe/kotoshu check README.md --format sarif --output
|
|
56
|
+
results.sarif` produces valid SARIF 2.1.0 that GitHub accepts
|
|
57
|
+
- `bundle exec exe/kotoshu --help` lists every command the README mentions
|
|
58
|
+
- No duplicate command classes; one canonical path per command
|
|
59
|
+
- A spec under `spec/kotoshu/cli/` exercises each subcommand end-to-end
|
|
60
|
+
(using a temp dir, not real `~/.kotoshu`)
|
|
61
|
+
|
|
62
|
+
## Dependencies
|
|
63
|
+
|
|
64
|
+
- Light dependency on `03-dynamic-download` (CLI should let users
|
|
65
|
+
pre-download resources), but can proceed independently.
|
|
66
|
+
|
|
67
|
+
## Out of scope
|
|
68
|
+
|
|
69
|
+
- LSP server, HTTP server (future work, not v1)
|
|
70
|
+
- GUI / TUI redesign — keep the existing Thor UX
|
|
71
|
+
|
|
72
|
+
## Status
|
|
73
|
+
|
|
74
|
+
_Pending._
|