kotoshu 0.10.0 → 0.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +36 -0
- data/Rakefile +35 -0
- data/TODO.impl/00-vision.md +17 -0
- data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
- data/TODO.impl/108-ko-ne-full-feature.md +37 -0
- data/TODO.impl/109-ten-audit.md +26 -0
- data/docs/STABILITY-1.0-AUDIT.md +251 -0
- data/docs/STABILITY-1.0.md +137 -0
- data/docs/cache-architecture.md +1 -1
- data/lib/kotoshu/cache/language_cache.rb +26 -11
- data/lib/kotoshu/cache/staged_languages.rb +45 -0
- data/lib/kotoshu/cache.rb +3 -0
- data/lib/kotoshu/cli/dict_command.rb +64 -0
- data/lib/kotoshu/cli.rb +1 -57
- data/lib/kotoshu/fluent_checker.rb +3 -0
- data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
- data/lib/kotoshu/keyboard/registry.rb +24 -2
- data/lib/kotoshu/keyboard.rb +6 -0
- data/lib/kotoshu/language/script.rb +128 -0
- data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
- data/lib/kotoshu/language.rb +5 -0
- data/lib/kotoshu/languages/ko/language.rb +56 -0
- data/lib/kotoshu/languages/ne/language.rb +60 -0
- data/lib/kotoshu/languages/nn/language.rb +26 -0
- data/lib/kotoshu/languages.rb +8 -0
- data/lib/kotoshu/multi_language_checker.rb +3 -0
- data/lib/kotoshu/spellchecker.rb +10 -8
- data/lib/kotoshu/version.rb +1 -1
- data/lib/kotoshu.rb +0 -5
- data/scripts/extract_eval_grids.rb +1 -1
- data/sig/kotoshu.rbs +56 -0
- metadata +17 -5
- data/lib/kotoshu/commands/cache_command.rb +0 -421
- data/lib/kotoshu/commands/check_command.rb +0 -301
- data/lib/kotoshu/commands/model_command.rb +0 -291
- data/lib/kotoshu/commands.rb +0 -12
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: a193e09b8fdaaf3e05a6b988e47ca460d8f7797b30e1719d109fdb9d28a89794
|
|
4
|
+
data.tar.gz: 377667004f6fb82be98470efa6e667a623afd96c01c61872db8b4c0048329c54
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 4c90da5e79e5b5f5a8e91a3a3b734874cc30699c9d95462f8cdae9160389601de3eea63b7231e19ba887e41d4b1e99ce12ec0ed07351868bb48c2ac8ba852007
|
|
7
|
+
data.tar.gz: de5adbf779c0c384b91186a73aaa571fe02bd49c84e7cad3a8bf62b366b76da4b7a853745c68b50eae6c5482bfe95fb1e55ca3f4798eb73b4194ea48c358ef62
|
data/CHANGELOG.md
CHANGED
|
@@ -8,6 +8,42 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
10
|
### Added
|
|
11
|
+
- **Korean and Nepali full-feature modules** (plan 108) - the last
|
|
12
|
+
big-population languages without modules move from the plan 107
|
|
13
|
+
basic tier to full feature. `ko` ships the Hangul tokenizer
|
|
14
|
+
(an eojeol is a run of Hangul syllables U+AC00-D7A3 plus bare jamo
|
|
15
|
+
U+1100-11FF/U+3130-318F that escaped IME composition) and the
|
|
16
|
+
Dubeolsik keyboard grid keyed on jamo per the KS X 5002 standard 2-set
|
|
17
|
+
layout, so typo proximity lands on the unit vowel mergers and
|
|
18
|
+
fortis/lenis confusions operate on. `ne` ships the Devanagari
|
|
19
|
+
tokenizer (matras U+093E-094C, the virama U+094D and vowel signs
|
|
20
|
+
stay attached to the base consonant; the danda and Devanagari
|
|
21
|
+
digits are separators) and the Devanagari InScript grid (the Indian
|
|
22
|
+
national standard, mirrored from the models repo eval harness and
|
|
23
|
+
drift-checked against it; the Nepali Traditional Romanized layout
|
|
24
|
+
is more common on the ground but is not a documented standard).
|
|
25
|
+
Both dictionaries were already staged, so `kotoshu setup ko ne`
|
|
26
|
+
serves them; full-feature languages grow 33 to 35.
|
|
27
|
+
- **Basic support for every staged dictionary language** (plan 107) -
|
|
28
|
+
`kotoshu setup` now serves all 95 languages staged in the
|
|
29
|
+
kotoshu/dictionaries manifest, not just the ones with gem modules.
|
|
30
|
+
`LanguageCache::AVAILABLE_LANGUAGES` derives from a vendored
|
|
31
|
+
snapshot of the manifest (Cache::StagedLanguages, refreshed by
|
|
32
|
+
`rake kotoshu:staged_languages:sync`), keeping the hot path
|
|
33
|
+
network-free. Languages without a module run on the basic tier:
|
|
34
|
+
script classification (Language::Script) picks the tokenizer whose
|
|
35
|
+
word regex drives check/suggest word extraction (shared Latin /
|
|
36
|
+
Cyrillic / Greek tokenizers plus a generic regex tokenizer for
|
|
37
|
+
Arabic, Hebrew, Armenian, Georgian, Devanagari and Hangul), and the
|
|
38
|
+
keyboard registry falls back to the script's standard grid (JCUKEN
|
|
39
|
+
for Cyrillic, Arabic 101, Hebrew SI-1452, QWERTY otherwise) instead
|
|
40
|
+
of QWERTY for everything. Two documented tiers: full feature =
|
|
41
|
+
module (registered keyboard + verified specimens,
|
|
42
|
+
`LanguageCache.full_feature_languages`), basic = manifest-present.
|
|
43
|
+
`nn` (Norwegian Nynorsk) is wired as a full-feature module while
|
|
44
|
+
here - dictionary and model are both staged. Modules keep winning
|
|
45
|
+
over the fallbacks; the 32 existing modules behave identically
|
|
46
|
+
(conformance vectors untouched).
|
|
11
47
|
- **Personal dictionary in the check path** (plan 105) -
|
|
12
48
|
`Spellchecker#check` (and therefore `Kotoshu.check`,
|
|
13
49
|
`Kotoshu.check_file`, and `kotoshu check`) no longer flags words
|
data/Rakefile
CHANGED
|
@@ -32,6 +32,41 @@ report_divergences = lambda do |divergences|
|
|
|
32
32
|
end
|
|
33
33
|
|
|
34
34
|
namespace :kotoshu do
|
|
35
|
+
namespace :staged_languages do
|
|
36
|
+
desc <<~DESC
|
|
37
|
+
Regenerate the vendored staged-language list (plan 107). Fetches \
|
|
38
|
+
the dictionaries manifest at the SourceRegistry pin, extracts the \
|
|
39
|
+
distinct language codes from its resources, and rewrites \
|
|
40
|
+
lib/kotoshu/cache/staged_languages.rb in place so \
|
|
41
|
+
LanguageCache::AVAILABLE_LANGUAGES tracks the manifest. Run after \
|
|
42
|
+
the dictionaries repo stages new languages.
|
|
43
|
+
DESC
|
|
44
|
+
task :sync do
|
|
45
|
+
require "kotoshu"
|
|
46
|
+
require "json"
|
|
47
|
+
|
|
48
|
+
url = Kotoshu::SourceRegistry.new.url_for(:dict_manifest)
|
|
49
|
+
puts "Fetching #{url}"
|
|
50
|
+
bytes = Kotoshu::Integrity::NetHTTP.get(url)
|
|
51
|
+
abort "manifest not found at #{url}" if bytes.nil?
|
|
52
|
+
|
|
53
|
+
manifest = JSON.parse(bytes)
|
|
54
|
+
codes = manifest.fetch("resources").values.map { |r| r.fetch("language") }.uniq.sort
|
|
55
|
+
abort "manifest carried no languages" if codes.empty?
|
|
56
|
+
|
|
57
|
+
target = File.expand_path("lib/kotoshu/cache/staged_languages.rb", __dir__)
|
|
58
|
+
source = File.read(target)
|
|
59
|
+
updated = source.sub(/CODES = %w\[\n(?: {8}\S+(?: \S+)*\n)+ {6}\]\.freeze/) do
|
|
60
|
+
wrapped = codes.each_slice(10).map { |slice| " #{slice.join(' ')}" }.join("\n")
|
|
61
|
+
"CODES = %w[\n#{wrapped}\n ].freeze"
|
|
62
|
+
end
|
|
63
|
+
abort "CODES block not found in #{target}" if updated == source
|
|
64
|
+
|
|
65
|
+
File.write(target, updated)
|
|
66
|
+
puts "Rewrote #{target}: #{codes.size} languages (was #{Kotoshu::Cache::StagedLanguages::CODES.size})"
|
|
67
|
+
end
|
|
68
|
+
end
|
|
69
|
+
|
|
35
70
|
namespace :conformance do
|
|
36
71
|
desc <<~DESC
|
|
37
72
|
Export conformance vectors for kotoshu-rs (plan 67 M3). Walks the Spylls \
|
data/TODO.impl/00-vision.md
CHANGED
|
@@ -262,3 +262,20 @@ yet passing bucket bytes to loadModel.
|
|
|
262
262
|
| 106 | [Native LID detection](106-native-lid-detection.md) | detect is a 7-language heuristic; the pure-Rust lid reader exists unused |
|
|
263
263
|
| 107 | models [09 bucket expansion](../../models-fasttext-onnx/TODO.impl/09-bucket-table-expansion.md) | bucket OOV siblings exist only for en and de |
|
|
264
264
|
| — | site 11 playground buckets | wasm 0.4.0 accepts bucketsBytes; the worker never fetches the sibling |
|
|
265
|
+
|
|
266
|
+
|
|
267
|
+
Plans 107-109 are **wave 8, reachability and the road to 1.0**
|
|
268
|
+
(2026-09-07): after wave 7 (personal dict in check, 176-language
|
|
269
|
+
detection, buckets rolling to all languages), the remaining user gaps
|
|
270
|
+
are the sixty-plus staged dictionaries unreachable without a module,
|
|
271
|
+
the last big-population languages (ko, ne), and the absence of a
|
|
272
|
+
stability commitment adopters can build on.
|
|
273
|
+
|
|
274
|
+
| # | Plan | Gap it closes |
|
|
275
|
+
|---|---|---|
|
|
276
|
+
| 107 | [Basic support for every staged language](107-basic-support-every-staged-language.md) | setup rejects 60+ staged languages; modules become upgrades, not gates |
|
|
277
|
+
| 108 | [ko and ne full-feature](108-ko-ne-full-feature.md) | Hangul and Devanagari - the last ~100M speakers without support |
|
|
278
|
+
| 109 | [The 1.0 readiness audit](109-ten-audit.md) | no frozen surface or stability policy for adopters |
|
|
279
|
+
|
|
280
|
+
Sequencing: 108 after 107 (same area); 109 is analysis-only and runs in
|
|
281
|
+
parallel. The 1.0 cut itself remains the owner's version call.
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
# Plan 107 — Basic support for every staged dictionary language
|
|
2
|
+
|
|
3
|
+
## Why
|
|
4
|
+
`kotoshu setup` only serves the 32 languages with modules. Sixty-plus staged
|
|
5
|
+
dictionaries (from gd to ia) are unreachable: a Welsh or Icelandic user
|
|
6
|
+
cannot use the gem at all, even though the dictionary and model are
|
|
7
|
+
downloadable. Modules should be the UPGRADE (keyboard + verified specimens),
|
|
8
|
+
not the gate.
|
|
9
|
+
|
|
10
|
+
## Work (gem repo)
|
|
11
|
+
1. `LanguageCache::AVAILABLE_LANGUAGES` derives from the dictionaries
|
|
12
|
+
manifest (the source the registry/staging uses) instead of the hardcoded
|
|
13
|
+
module list; modules register themselves as full-feature upgrades.
|
|
14
|
+
2. Default behavior for module-less languages: script-appropriate tokenizer
|
|
15
|
+
(reuse the script-regex machinery from plan 91), generic keyboard
|
|
16
|
+
fallback (QWERTY for Latin, JCUKEN Cyrillic, Arabic 101 Arabic-script),
|
|
17
|
+
setup/check/suggest all functional.
|
|
18
|
+
3. Wire nn (model + dictionary exist) as a full-feature module while here;
|
|
19
|
+
note sr-Latn as follow-up if the Latin mapping needs product sign-off.
|
|
20
|
+
4. Specs: setup + specimen round trip on 3 module-less staged languages
|
|
21
|
+
(e.g. is, cy, gd); full-feature languages behave identically to today;
|
|
22
|
+
conformance untouched.
|
|
23
|
+
|
|
24
|
+
## Verification
|
|
25
|
+
Live `kotoshu setup is && check -l is` round trip; suite green; the 32
|
|
26
|
+
modules' outputs byte-identical.
|
|
27
|
+
|
|
28
|
+
## Status
|
|
29
|
+
Executed 2026-09-07 — AVAILABLE_LANGUAGES derives from a vendored
|
|
30
|
+
snapshot of the dictionaries manifest (Cache::StagedLanguages, 95
|
|
31
|
+
languages, refreshed by `rake kotoshu:staged_languages:sync`);
|
|
32
|
+
module-less languages ride Language::Script (script tokenizer for
|
|
33
|
+
word extraction: shared Latin/Cyrillic/Greek classes plus a generic
|
|
34
|
+
regex tokenizer for Arabic/Hebrew/Armenian/Georgian/Devanagari/
|
|
35
|
+
Hangul) and the script keyboard default (JCUKEN Cyrillic, Arabic 101,
|
|
36
|
+
Hebrew SI-1452, QWERTY Latin). Tiers documented: full feature =
|
|
37
|
+
module (LanguageCache.full_feature_languages, 33 with nn); basic =
|
|
38
|
+
manifest-present. nn wired as a full-feature module. Specimens
|
|
39
|
+
engine-verified for is cy gd (basic) + nn; live round trips
|
|
40
|
+
setup+check -l for is cy gd nn; conformance 2630 vectors 0 failures;
|
|
41
|
+
suite 3948 examples 0 failures. sr-Latn stays follow-up (normalize
|
|
42
|
+
maps it to sr; needs product sign-off).
|
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
# Plan 108 — Korean and Nepali: the last big-population languages
|
|
2
|
+
|
|
3
|
+
## Why
|
|
4
|
+
ko (~82M) and ne (~16M) were deferred from batch 3 because Hangul jamo
|
|
5
|
+
input and Devanagari conjuncts need ja-style tokenizer treatment. They are
|
|
6
|
+
the largest populations without full-feature support; both have staged
|
|
7
|
+
dictionaries and models.
|
|
8
|
+
|
|
9
|
+
## Work (gem repo; sequenced AFTER plan 107 merges — same area)
|
|
10
|
+
1. ko: Hangul tokenizer (syllable-block aware, jamo-level word regex so
|
|
11
|
+
check extracts real eojeol), Korean keyboard grid (Dubeolsik 2-set),
|
|
12
|
+
module + specimens engine-verified.
|
|
13
|
+
2. ne: Devanagari tokenizer (matras and conjuncts stay attached to the
|
|
14
|
+
base consonant — the Unicode grapheme rules the script needs), Roman
|
|
15
|
+
keyboard fallback or a Nepali traditional grid if defensibly sourced,
|
|
16
|
+
module + specimens.
|
|
17
|
+
3. AVAILABLE_LANGUAGES + site-flip deliverables (specimen table, sizes).
|
|
18
|
+
|
|
19
|
+
## Verification
|
|
20
|
+
Same discipline as batch 3: every specimen engine-verified through
|
|
21
|
+
correct?/suggest; RTL/complex-script sentence round trips through
|
|
22
|
+
Kotoshu.check; suite green.
|
|
23
|
+
|
|
24
|
+
## Status
|
|
25
|
+
Executed 2026-09-07 — ko: HangulTokenizer (eojeol = run of syllables
|
|
26
|
+
U+AC00-D7A3 plus bare jamo U+1100-11FF/U+3130-318F), Dubeolsik grid
|
|
27
|
+
keyed on jamo per KS X 5002 (the eval harness models ko slips as
|
|
28
|
+
jamo confusion pairs, not a key grid, so no drift check applies),
|
|
29
|
+
module + 7 engine-verified specimens. ne: DevanagariTokenizer
|
|
30
|
+
(matras and virama conjuncts attached to the base consonant; danda
|
|
31
|
+
and Devanagari digits are separators), Devanagari InScript grid
|
|
32
|
+
mirrored from the eval harness _NE_INSCRIPT and drift-checked (the
|
|
33
|
+
defensible standard — Traditional Romanized is common but
|
|
34
|
+
undocumented), module + 3 specimens. Full-feature languages 33 -> 35.
|
|
35
|
+
Live round trips setup + check -l ko ne recorded; conformance 2630
|
|
36
|
+
vectors 0 failures; suite 3985 examples 0 failures. Specimen table
|
|
37
|
+
and dictionary sizes in the PR body for the site flip.
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
# Plan 109 — The 1.0 readiness audit
|
|
2
|
+
|
|
3
|
+
## Why
|
|
4
|
+
Adopters hesitate at 0.x. A 1.0 cut needs a frozen public surface and a
|
|
5
|
+
stated policy. The audit inventories what 1.0 would freeze and what it
|
|
6
|
+
would deprecate — it does NOT cut versions (owner decision).
|
|
7
|
+
|
|
8
|
+
## Work (gem repo docs/, analysis only)
|
|
9
|
+
1. Inventory the public surface: Ruby API (Kotoshu module functions,
|
|
10
|
+
Spellchecker, Configuration, Tasks, validators), CLI flags, wasm JS
|
|
11
|
+
surface (KotoshuWasm, loadModel/rerank/semanticSuggest/loadLid/
|
|
12
|
+
detectLanguage), HTTP contract (openapi.yaml), action inputs, registry
|
|
13
|
+
schema.
|
|
14
|
+
2. For each: stable / needs-deprecation-notice / internal-accidentally-
|
|
15
|
+
public. Flag anything accidentally public that 1.0 should privatize.
|
|
16
|
+
3. Draft the stability policy (semver commitments, deprecation window,
|
|
17
|
+
conformance vectors as the behavioral contract) as docs/STABILITY.md.
|
|
18
|
+
4. A 1.0 checklist: what must land first (nothing mechanical is known to
|
|
19
|
+
block), what the release train looks like (gem+wasm+server together).
|
|
20
|
+
|
|
21
|
+
## Verification
|
|
22
|
+
Every inventory line cites the file that defines it; the deprecation list
|
|
23
|
+
reviewed against gemspec/exported constants.
|
|
24
|
+
|
|
25
|
+
## Status
|
|
26
|
+
Pending
|
|
@@ -0,0 +1,251 @@
|
|
|
1
|
+
# The 1.0 Readiness Audit
|
|
2
|
+
|
|
3
|
+
Plan 109. Analysis and documentation only — no behavior change, no version
|
|
4
|
+
bump, no release. The version cut itself is an owner decision.
|
|
5
|
+
|
|
6
|
+
- Audited at: gem 0.10.0 (`lib/kotoshu/version.rb:4`), main @ 73b7fac.
|
|
7
|
+
- Companion policy draft: `docs/STABILITY-1.0.md`.
|
|
8
|
+
- Method: every inventory line cites the file that defines the surface.
|
|
9
|
+
Reachability was verified by grep, not by assumption: a class being
|
|
10
|
+
present under `lib/` is not the same as being wired to `exe/kotoshu`.
|
|
11
|
+
|
|
12
|
+
## Inventory
|
|
13
|
+
|
|
14
|
+
Verdicts: **stable** (freeze at 1.0), **deprecate-before-1.0** (needs a
|
|
15
|
+
deprecation cycle first), **accidentally-public** (internal or dead, 1.0
|
|
16
|
+
should privatize or delete).
|
|
17
|
+
|
|
18
|
+
### Ruby gem — module functions (`lib/kotoshu.rb`)
|
|
19
|
+
|
|
20
|
+
| Surface | Where defined | Verdict | Note |
|
|
21
|
+
|---|---|---|---|
|
|
22
|
+
| `configure` / `configuration` / `configuration=` | `lib/kotoshu.rb:98,109,127` | stable | Type-checked global swap (127-131). |
|
|
23
|
+
| `spellchecker` / `spellchecker_for` | `lib/kotoshu.rb:139,157` | stable | Cache-only; README-documented (README.adoc:18). |
|
|
24
|
+
| `resolve` | `lib/kotoshu.rb:179` | stable | Stage-2 resolve; README.adoc:364. |
|
|
25
|
+
| `setup` / `setup?` / `languages_setup` | `lib/kotoshu.rb:220,245,257` | stable | Stage-1 entry; idempotent; `want:`/`tier:`/local-source opts frozen. |
|
|
26
|
+
| `reset_spellchecker` | `lib/kotoshu.rb:268` | stable | Test/embedding seam. |
|
|
27
|
+
| `correct?` / `misspelled?` / `suggest` / `check` / `check_file` / `check_files` | `lib/kotoshu.rb:286,297,313,331,349,363` | stable | The hot path; strict two-stage contract is the product identity. |
|
|
28
|
+
| `detect_language` / `detect_language_with_confidence` / `setup_lid` / `setup_lid?` | `lib/kotoshu.rb:465,481,501,508` | stable | Native LID with heuristic fallback (plan 106). |
|
|
29
|
+
| `get_language` / `language_registered?` / `supported_languages` / `language` | `lib/kotoshu.rb:519,530,540,443` | stable | `language` is a trivial namespace accessor; keep. |
|
|
30
|
+
| `dictionary` / `trie` / `suggestion_pipeline` | `lib/kotoshu.rb:371,388,409` | stable | Convenience factories; exercised by `examples/04..06`. |
|
|
31
|
+
| `register_dictionary_type` / `register_suggestion_algorithm` | `lib/kotoshu.rb:422,433` | stable | The extension points; 1.0 must not narrow them. |
|
|
32
|
+
| `VERSION` | `lib/kotoshu/version.rb:4` | stable | |
|
|
33
|
+
|
|
34
|
+
### Ruby gem — core classes
|
|
35
|
+
|
|
36
|
+
| Surface | Where defined | Verdict | Note |
|
|
37
|
+
|---|---|---|---|
|
|
38
|
+
| `Kotoshu::Spellchecker` public methods: `correct?`, `incorrect?`, `suggest`, `check_word`, `check`, `check_file`, `check_directory`, `tokenize`, `dictionary`, `reload_dictionary` | `lib/kotoshu/spellchecker.rb:118,130,143,164,196,242,265,280,308,315` | stable | Constructor kwargs (`dictionary:`, `config:`, `resource_bundle:`) frozen at 65. |
|
|
39
|
+
| `Kotoshu::Spellchecker::ASCII_WORD_REGEX` | `lib/kotoshu/spellchecker.rb:39` | accidentally-public | Internal fallback word-char set exposed as a constant. Freeze + document, or privatize (see deprecations). |
|
|
40
|
+
| `Kotoshu::Configuration`: `SCHEMA`, `DEFAULTS`, accessors, `get`, `key?`, `load_dictionary`, `reset_dictionary`, `source_registry`, `self.default/instance/instance=/reset` | `lib/kotoshu/configuration.rb:38,222,423,431,447,474,517,541,565,576,585` | stable | ENV contract (`KOTOSHU_*`) rides on SCHEMA — every key there is public surface. |
|
|
41
|
+
| `Configuration#dictionaries_url` / `#models_url` | `lib/kotoshu/configuration.rb:63-74` (SCHEMA), `:318-321` (attrs) | deprecate-before-1.0 | SCHEMA descriptions already say "Deprecated: use repos_base_url + pins via SourceRegistry". |
|
|
42
|
+
| `ResourceManager.setup/resolve/setup?/languages_setup`, `DEFAULT_WANT` | `lib/kotoshu/resource_manager.rb:69,97,21` | stable | Facade delegates here; two-stage model frozen. |
|
|
43
|
+
| `ResourceBundle` (`dictionary`/`frequency`/`model`/`rules`) | `lib/kotoshu/resource_bundle.rb` | stable | Consumed by `Kotoshu.resolve` (kotoshu.rb:179). |
|
|
44
|
+
| Exception hierarchy: `Error` + `DictionaryNotFoundError`, `InvalidDictionaryFormatError`, `ConfigurationError`, `SpellcheckError`, `AffixRuleError`, `ResourceNotCachedError`, `ResourceNotSetupError`, `ResourceResolutionError`, `IntegrityError`, `SuikaUnavailable` | `lib/kotoshu/core/exceptions.rb:8,14,32,54,72,90,106,122,136,156,177`; autoloaded `lib/kotoshu.rb:65-75` | stable | Message text is not frozen; class names are. |
|
|
45
|
+
| `Kotoshu::Paths` (XDG + `KOTOSHU_{CACHE,CONFIG,DATA}_PATH`) | `lib/kotoshu/paths.rb` | stable | On-disk layout is user-visible contract. |
|
|
46
|
+
| Autoloaded namespaces behind `Kotoshu::*` | `lib/kotoshu.rb:14-62` | stable (as namespaces) | Namespaces stay; member-by-member stability is YARD-documented per class. 1.0 freezes namespace existence, not every leaf. |
|
|
47
|
+
| Top-level singleton aliases `Kotoshu::LanguageCache`, `Kotoshu::LanguageIdentifier`, `Kotoshu::ModelCache`, `Kotoshu::SemanticAnalyzer` | `lib/kotoshu.rb:80,81,84,85` | accidentally-public | Zero hits in lib/, spec/, README; canonical homes are `Kotoshu::Cache::LanguageCache`, `Kotoshu::Language::*`, `Kotoshu::Cache::ModelCache`, `Kotoshu::Analyzers::SemanticAnalyzer`. |
|
|
48
|
+
| `Kotoshu::Debug`, `DebugLogger`, `Metrics`, `MetricsCollector` | `lib/kotoshu.rb:78-83`; `lib/kotoshu/debug_mode.rb`, `metrics_module.rb`, `metrics_collector.rb` | stable | Documented in their doc comments; `Kotoshu::Metrics.stats` is README-documented (README.adoc:211). |
|
|
49
|
+
| `Kotoshu::FluentChecker` / `Kotoshu::MultiLanguageChecker` | `lib/kotoshu/fluent_checker.rb`, `lib/kotoshu/multi_language_checker.rb` | deprecate-before-1.0 (or document) | Spec-covered (`spec/kotoshu/{fluent,multi_language}_checker_spec.rb`) but absent from README.adoc — decide: document or deprecate. |
|
|
50
|
+
|
|
51
|
+
### Ruby gem — framework integrations (all soft deps, plan 89)
|
|
52
|
+
|
|
53
|
+
| Surface | Where defined | Verdict | Note |
|
|
54
|
+
|---|---|---|---|
|
|
55
|
+
| `require "kotoshu/validators"` → `Kotoshu::Validators::SpellingValidator` (+ `ACTIVE_MODEL_LOADED`, `ActiveModelUnavailable`) | `lib/kotoshu/validators/spelling_validator.rb:10-99`; autoload `lib/kotoshu.rb:60` | stable | Placeholder class keeps constant resolvable without Rails (91-99). |
|
|
56
|
+
| `require "kotoshu/rspec"` → `Kotoshu::Rspec::Matchers` (`expect_words`, `expect_document`, `all_be_spelled_correctly`, `be_spelled_correctly(in:)`) | `lib/kotoshu/rspec.rb:26-160`; autoload `lib/kotoshu.rb:61` | stable | Matcher names are the contract. |
|
|
57
|
+
| `require "kotoshu/tasks"` → default `rake kotoshu:check`; `Kotoshu::Tasks::CheckTask` | `lib/kotoshu/tasks.rb:11`, `lib/kotoshu/tasks/check_task.rb:22` | stable | require-only (defining a task on load is the point — kotoshu.rb:56-59). |
|
|
58
|
+
| `require "kotoshu/jekyll"` → `Kotoshu::Jekyll::Generator` (+ `JEKYLL_LOADED`, `JekyllUnavailable`) | `lib/kotoshu/jekyll.rb:9-96`; autoload `lib/kotoshu.rb:62` | stable | Baseline honored via `Baseline::Store::DEFAULT_FILENAME`. |
|
|
59
|
+
|
|
60
|
+
### Ruby gem — conformance surface (the behavioral contract)
|
|
61
|
+
|
|
62
|
+
| Surface | Where defined | Verdict | Note |
|
|
63
|
+
|---|---|---|---|
|
|
64
|
+
| `conformance/vectors.jsonl` (2630 rows, committed) | `conformance/vectors.jsonl`; sync contract documented in `lib/kotoshu/conformance_exporter.rb:54-60` | stable — frozen expectations | `expected` records what the Ruby engine ACTUALLY returns, not linguistic truth (conformance_exporter.rb:31-37). |
|
|
65
|
+
| Row shape v0: `kind/language/dictionary/input/limit/expected`; suggestion keys `word,distance,confidence,source` | `lib/kotoshu/conformance_exporter.rb:16-29,80` | stable | Four-key shape shared with wasm and `ffi::ruby`. |
|
|
66
|
+
| `Kotoshu::ConformanceRunner` (`run`, `compare`, rake tasks) | `lib/kotoshu/conformance_runner.rb:26,72,97` | stable | Guards Ruby-vs-vectors and native-vs-Ruby equality. |
|
|
67
|
+
| `Kotoshu::ConformanceExporter` (`export`, corpora selection, `EXCLUDED_FIXTURES`) | `lib/kotoshu/conformance_exporter.rb:61-254` | stable | Deterministic re-export (44-53). |
|
|
68
|
+
|
|
69
|
+
### CLI (`exe/kotoshu` → `Kotoshu::Cli::Cli`)
|
|
70
|
+
|
|
71
|
+
| Surface | Where defined | Verdict | Note |
|
|
72
|
+
|---|---|---|---|
|
|
73
|
+
| Commands: `check`, `setup`, `dict`, `cache`, `personal`, `completions`, `baseline`, `status`, `version` (`--version`/`-V`) | `lib/kotoshu/cli.rb:178,241,317-329,349,358-367` | stable | |
|
|
74
|
+
| Class options `--language/-l`, `--format/-f` (text,json,sarif), `--interactive/-i`, `--verbose/-v` | `lib/kotoshu/cli.rb:102-125` | stable | |
|
|
75
|
+
| `check` options `--baseline`, `--show-suppressed`, `--include`, `--exclude`, `--personal` | `lib/kotoshu/cli.rb:151-168` | stable | |
|
|
76
|
+
| `setup` options `--aff --dic --from --frequency --want --model --tier --force --strict --list` | `lib/kotoshu/cli.rb:214-240` | stable | |
|
|
77
|
+
| Exit codes 0/1/2/3 | `lib/kotoshu/cli.rb:92-97,145-149` | stable | Consumed by action-kotoshu (action.yml:178-179). |
|
|
78
|
+
| JSON output top-level keys; SARIF 2.1.0 shape | `lib/kotoshu/cli.rb:624-651,686-715` | stable | Machine-readable contract; additive keys only. |
|
|
79
|
+
| `cache` subcommands `list/info/clean/evict/download/purge/validate` | `lib/kotoshu/cli/cache_command.rb:27,50,79,101,133,166,184` | stable | |
|
|
80
|
+
| `personal` subcommands `add/remove/list/import/path/clear` | `lib/kotoshu/cli/personal_command.rb:28-89` | stable | |
|
|
81
|
+
| `baseline init` (`--output`, `--language`) | `lib/kotoshu/cli/baseline_command.rb:15-36` | stable | |
|
|
82
|
+
| `completions bash/zsh` | `lib/kotoshu/cli/completions_command.rb:13-26` | stable | Catalog is hand-maintained — must track command changes. |
|
|
83
|
+
| `fetch` (hidden alias of `setup`) | `lib/kotoshu/cli.rb:280-315` | deprecate-before-1.0 | Already hidden and labeled deprecated; remove at 1.0. |
|
|
84
|
+
|
|
85
|
+
### Ruby gem — accidentally-public legacy commands (the hazard block)
|
|
86
|
+
|
|
87
|
+
| Surface | Where defined | Verdict | Note |
|
|
88
|
+
|---|---|---|---|
|
|
89
|
+
| `::DictCommand` (top-level constant!) | `lib/kotoshu/cli.rb:10` | accidentally-public | Defined on `Object`, outside `module Kotoshu`. Pollutes the global namespace of every process that requires the gem; collides with any other gem defining `DictCommand`. Move under `Kotoshu::Cli` before 1.0. |
|
|
90
|
+
| `Kotoshu::Commands::CheckCommand` | `lib/kotoshu/commands/check_command.rb:4`; autoload `lib/kotoshu/commands.rb:9` | accidentally-public | Not wired into `Cli` (no `subcommand "check"` at cli.rb:317-329 — `check` is a method on `Cli` itself, cli.rb:178). Half-placeholder implementation; its hunspell path calls `SpellChecker.new` (check_command.rb:198) which resolves to nonexistent `Kotoshu::SpellChecker` (the real class is `Kotoshu::Spellchecker`) → `NameError` at runtime. Declares flags that do not exist on the real CLI (`--model`, `-o`, `csv/yaml` formats), contradicting cli.rb:108-113. Delete or privatize. |
|
|
91
|
+
| `Kotoshu::CacheCommand` | `lib/kotoshu/commands/cache_command.rb:4` | accidentally-public | Dead duplicate of the wired `Kotoshu::Cli::CacheCommand` (autoload cli.rb:70; subcommand cli.rb:321). Nothing requires this file (verified by grep); acknowledged as historical in `lib/kotoshu/commands.rb:7-8`. Still shipped in the gem (`spec.files` = `git ls-files`, kotoshu.gemspec:30-38). Delete. |
|
|
92
|
+
| `Kotoshu::Commands::ModelCommand` | `lib/kotoshu/commands/model_command.rb:4`; autoload `lib/kotoshu/commands.rb:10` | accidentally-public | No `kotoshu model` subcommand is registered anywhere. `convert` shells out to `scripts/convert_fasttext_to_onnx.py` (model_command.rb:33-36) — untracked in git, therefore NOT shipped in the gem, so the command cannot work as packaged. Delete. |
|
|
93
|
+
|
|
94
|
+
### Ruby gem — native extension
|
|
95
|
+
|
|
96
|
+
| Surface | Where defined | Verdict | Note |
|
|
97
|
+
|---|---|---|---|
|
|
98
|
+
| Optional Rust ext build (`extconf.rb`) | `kotoshu.gemspec:53`; `ext/kotoshu_native/extconf.rb` | stable | Soft: pure-Ruby is the complete fallback (gemspec:44-52). |
|
|
99
|
+
| `Kotoshu::Native.available?`, `Native.suggestion(row)`, `Native::Unavailable`, `Native::EXTENSION` | `lib/kotoshu/native.rb:49,65,35,39` | stable | Extension defines `Native::VERSION`, `Native::Error`, `Native::Dictionary.load/correct?/suggest` (native.rb:9-14). |
|
|
100
|
+
| `Kotoshu::NativeBackend.resolve` boundary (ruby/auto/native) | `lib/kotoshu/native_backend.rb:45-73` | stable | Accelerator-only boundary is documented and frozen. |
|
|
101
|
+
|
|
102
|
+
### wasm package (kotoshu-rs)
|
|
103
|
+
|
|
104
|
+
| Surface | Where defined | Verdict | Note |
|
|
105
|
+
|---|---|---|---|
|
|
106
|
+
| `KotoshuWasm` constructor `(affSource, dicSource)`, `correct(word)`, `suggest(word, limit?)`, static `VERSION` | `kotoshu-rs/kotoshu/src/ffi/wasm/mod.rs:165-174,178,185,152-155` | stable | Contents-not-paths loading; errors reject with the Rust message. |
|
|
107
|
+
| `loadModel(modelBytes, vocabBytes, bucketsBytes?)` → `KotoshuModel`; `.free()` | mod.rs:220-235,207-209 | stable | Optional third arg added by plan 103 — additive pattern to keep. |
|
|
108
|
+
| `rerank(model, word, context)` → number in [-1,1] | mod.rs:241-244 | stable | Mean-cosine contract documented (mod.rs:73-82). |
|
|
109
|
+
| `semanticSuggest(model, word, k?)` → `{word, score}[]` | mod.rs:252-266 | stable | |
|
|
110
|
+
| `loadLid(onnxBytes, vocabBytes)` → `KotoshuLid`; `detectLanguage(lid, text)` → `{code, score}` | mod.rs:281-287,293-300 | stable | Parity contract with the gem documented (mod.rs:84-94). |
|
|
111
|
+
| Suggestion row shape: exactly `word/distance/confidence/source` | mod.rs:48-54,190-198 | stable — frozen | Same four keys as the conformance vectors (`conformance_exporter.rb:80`). |
|
|
112
|
+
| npm identity | `kotoshu-rs/kotoshu-wasm/pkg/package.json:2,5` | needs a decision | Package name is `kotoshu-wasm` but every doc says `@kotoshu/wasm` (mod.rs:2-3); publication still blocked on npm org credentials. Decide the final name before 1.0 — npm names are sticky. |
|
|
113
|
+
|
|
114
|
+
### HTTP server (kotoshu-server)
|
|
115
|
+
|
|
116
|
+
| Surface | Where defined | Verdict | Note |
|
|
117
|
+
|---|---|---|---|
|
|
118
|
+
| `GET /` (metadata), `GET /v1/health`, `GET /v1/version`, `GET /v1/languages` | `kotoshu-server/openapi.yaml:15,25,35,45`; routes `lib/kotoshu/server/app.rb:290,300,306,311` | stable | |
|
|
119
|
+
| `POST /v1/check` (`text`, `language`, `format`, `model`), `POST /v1/suggest` (`word`, `language`, `max` 1-50), `POST /v1/detect` (`text`) | openapi.yaml:55-117,179-203; routes app.rb:318,350,366 | stable | Errors: 400/422/503 shapes (openapi.yaml:118-137). |
|
|
120
|
+
| Response schemas `DocumentResult/WordError/Suggestions/Suggestion/Detection` (incl. `engine: lid-176|heuristic`) | openapi.yaml:204-245 | stable | openapi.yaml is the contract source of truth; app.rb matches it 1:1 (verified). |
|
|
121
|
+
| `/v1` prefix as the contract version | openapi.yaml:25-100 | stable | Breaking changes require `/v2`, not field mutations. |
|
|
122
|
+
|
|
123
|
+
### GitHub action (action-kotoshu)
|
|
124
|
+
|
|
125
|
+
| Surface | Where defined | Verdict | Note |
|
|
126
|
+
|---|---|---|---|
|
|
127
|
+
| Inputs: `files, language, format, baseline, include, exclude, show_suppressed, fail_on_error, category, version, prewarm_languages, offline, output_path` | `action-kotoshu/action.yml:8-59` | stable — additive-only | New inputs must default to current behavior. |
|
|
128
|
+
| Outputs: `sarif-path`, `error-count` | action.yml:61-67 | stable | |
|
|
129
|
+
| Gem floor `>= 0.8.0` (v2 of the action) | action.yml:45,98-105 | stable | Floor bumps are breaking for the action and need a major action version. |
|
|
130
|
+
|
|
131
|
+
### Registry schema (models-fasttext-onnx)
|
|
132
|
+
|
|
133
|
+
| Surface | Where defined | Verdict | Note |
|
|
134
|
+
|---|---|---|---|
|
|
135
|
+
| `spec: "kotoshu.resources/v1"` (const) | `models-fasttext-onnx/schemas/registry.schema.json:15-17` | stable — frozen | Change requires a new spec string, never an edit. |
|
|
136
|
+
| Resource id pattern `kotoshu://models/(lid/lid-176|{lang}/(full|fluency|mini|buckets))` | registry.schema.json:35 | stable | Ids are never reused; content changes bump `release_tag` (schema description, registry.schema.json:5). |
|
|
137
|
+
| Resource object fields incl. `min_engine_version`, `sha256`, `license` | registry.schema.json:43-165 | stable | `additionalProperties: false` — additive fields need a schema change = coordinated release. |
|
|
138
|
+
|
|
139
|
+
## Deprecation list for the 1.0 cut
|
|
140
|
+
|
|
141
|
+
Concrete, with defining locations. All need small code PRs — deliberately
|
|
142
|
+
NOT implemented by this docs-only change.
|
|
143
|
+
|
|
144
|
+
1. `::DictCommand` — `lib/kotoshu/cli.rb:10`. Global-namespace leak from a
|
|
145
|
+
file that otherwise lives in `Kotoshu::Cli`. Rename to
|
|
146
|
+
`Kotoshu::Cli::DictCommand` (internal; `subcommand "dict"` wiring at
|
|
147
|
+
cli.rb:317-318 is the only consumer). Highest-priority item: it can
|
|
148
|
+
break other gems in the same process, which is exactly the class of
|
|
149
|
+
thing 1.0 exists to prevent.
|
|
150
|
+
Handled in this PR: the class moved to
|
|
151
|
+
`lib/kotoshu/cli/dict_command.rb` as `Kotoshu::Cli::DictCommand`
|
|
152
|
+
(autoloaded like its sibling subcommands); `::DictCommand` is gone.
|
|
153
|
+
2. `Kotoshu::Commands::CheckCommand` — `lib/kotoshu/commands/check_command.rb:4`
|
|
154
|
+
(autoload `lib/kotoshu/commands.rb:9`). Unwired, placeholder-laden, and
|
|
155
|
+
broken at check_command.rb:198 (`SpellChecker` → `NameError`; real class
|
|
156
|
+
is `Kotoshu::Spellchecker`, spellchecker.rb:21). Its flag surface
|
|
157
|
+
(`--model`, `-o/--output`, `csv|yaml` formats) contradicts the real CLI
|
|
158
|
+
and will mislead users who find it via constants. Delete; keep
|
|
159
|
+
`lib/kotoshu/commands.rb` as the namespace or remove both.
|
|
160
|
+
Handled in this PR: deleted, along with the now-empty `Commands`
|
|
161
|
+
namespace (file and autoload). No spec referenced it.
|
|
162
|
+
3. `Kotoshu::CacheCommand` — `lib/kotoshu/commands/cache_command.rb:4`.
|
|
163
|
+
Dead duplicate of `Kotoshu::Cli::CacheCommand`; unreachable by any
|
|
164
|
+
require in the repo; still shipped by the gemspec file list. Delete.
|
|
165
|
+
Handled in this PR: deleted. No spec referenced it.
|
|
166
|
+
4. `Kotoshu::Commands::ModelCommand` — `lib/kotoshu/commands/model_command.rb:4`
|
|
167
|
+
(autoload `lib/kotoshu/commands.rb:10`). No `model` subcommand exists;
|
|
168
|
+
`convert` depends on an untracked Python script not shipped in the gem
|
|
169
|
+
(model_command.rb:33-36 vs gemspec:30-38). Delete.
|
|
170
|
+
Handled in this PR: deleted. No spec referenced it.
|
|
171
|
+
5. Alias constants `Kotoshu::LanguageCache`, `Kotoshu::LanguageIdentifier`,
|
|
172
|
+
`Kotoshu::ModelCache`, `Kotoshu::SemanticAnalyzer` — `lib/kotoshu.rb:80,81,84,85`.
|
|
173
|
+
No usage anywhere in lib/, spec/, or README. Add a deprecation `warn`
|
|
174
|
+
in a 0.x minor and drop the autoloads at 1.0 (canonical names stay).
|
|
175
|
+
Handled in this PR: grep confirmed zero references to the aliases
|
|
176
|
+
anywhere (lib, spec, exe, docs, README, sig) — the short-name hits all
|
|
177
|
+
resolve to the canonical constants lexically — so the autoloads were
|
|
178
|
+
dropped straight away, ahead of the suggested warn cycle. Canonical
|
|
179
|
+
names are untouched.
|
|
180
|
+
6. CLI `fetch` — `lib/kotoshu/cli.rb:280-315`. Already hidden and marked
|
|
181
|
+
deprecated. Remove at 1.0.
|
|
182
|
+
7. `Configuration#dictionaries_url` / `#models_url` —
|
|
183
|
+
`lib/kotoshu/configuration.rb:63-74,318-321`. Marked deprecated in SCHEMA
|
|
184
|
+
since the SourceRegistry pins landed. Emit a warning on use, remove the
|
|
185
|
+
SCHEMA entries at 1.0 (ENV vars `KOTOSHU_DICTIONARIES_URL` /
|
|
186
|
+
`KOTOSHU_MODELS_URL` go with them — call that out in the changelog).
|
|
187
|
+
8. `Kotoshu::Spellchecker::ASCII_WORD_REGEX` — `lib/kotoshu/spellchecker.rb:39`.
|
|
188
|
+
Internal fallback set posing as API. Either document + freeze or make it
|
|
189
|
+
private before 1.0.
|
|
190
|
+
Handled in this PR: zero external references confirmed by grep, so the
|
|
191
|
+
constant was deleted and its literal inlined at the single internal use
|
|
192
|
+
site (constructor fallback). Behavior unchanged.
|
|
193
|
+
9. `Kotoshu::FluentChecker` / `Kotoshu::MultiLanguageChecker` —
|
|
194
|
+
`lib/kotoshu/fluent_checker.rb`, `lib/kotoshu/multi_language_checker.rb`.
|
|
195
|
+
Spec-covered but undocumented in README.adoc. Decide per class: document
|
|
196
|
+
as stable, or deprecate. Do not freeze silently.
|
|
197
|
+
Handled in this PR: both class docstrings now state "Experimental and
|
|
198
|
+
not public API: outside the 1.0 stability freeze, may change or be
|
|
199
|
+
removed without a deprecation cycle." Specs stay.
|
|
200
|
+
10. wasm npm identity — `kotoshu-rs/kotoshu-wasm/pkg/package.json:2` says
|
|
201
|
+
`kotoshu-wasm` while `kotoshu-rs/kotoshu/src/ffi/wasm/mod.rs:2-3` and
|
|
202
|
+
all docs say `@kotoshu/wasm`. Resolve before first publication; the
|
|
203
|
+
audit treats the documented scoped name as intended.
|
|
204
|
+
Handled in this PR: verified, no code change needed.
|
|
205
|
+
`kotoshu-rs/scripts/wasm_build.sh` rewrites the generated
|
|
206
|
+
`package.json` name to `@kotoshu/wasm` on every build (the source
|
|
207
|
+
manifest name is a wasm-pack crate-name artifact; `pkg/` is a
|
|
208
|
+
gitignored build output). The shipped identity is already
|
|
209
|
+
`@kotoshu/wasm`; the owner decision that remains is publication
|
|
210
|
+
credentials, not the name.
|
|
211
|
+
|
|
212
|
+
## 1.0 checklist
|
|
213
|
+
|
|
214
|
+
Known blockers — none are mechanical code fixes; all are owner decisions
|
|
215
|
+
or ecosystem plumbing, stated honestly:
|
|
216
|
+
|
|
217
|
+
1. Dispositions for the ten deprecation items above (small code PRs; this
|
|
218
|
+
audit only records them).
|
|
219
|
+
2. npm org + publish credentials for `@kotoshu/wasm` (and the name
|
|
220
|
+
decision, item 10). The wasm build exists; publication is blocked.
|
|
221
|
+
3. The JS SDK PR (kotoshu-js #1) still pending merge — the HTTP contract
|
|
222
|
+
it consumes must be the frozen one.
|
|
223
|
+
4. PyPI token for kotoshu-python publication (same train).
|
|
224
|
+
5. kotoshu-server is at 0.1.0 with an unexplained prewarm hang noted in
|
|
225
|
+
ecosystem state — the 1.0 train should not ship a server cut with a
|
|
226
|
+
known hang.
|
|
227
|
+
6. RBS coverage: `sig/kotoshu.rbs` should be checked against the frozen
|
|
228
|
+
surface so the types do not contradict the freeze.
|
|
229
|
+
7. Conformance vectors green on both engines
|
|
230
|
+
(`rake kotoshu:conformance:compare`) at the cut commit.
|
|
231
|
+
|
|
232
|
+
Release-train shape (gem + wasm + server together, per plan):
|
|
233
|
+
|
|
234
|
+
- Cut order: registry release_tag (if models moved) → gem 1.0 → wasm
|
|
235
|
+
package (independent version line, same day) → server (depends on the
|
|
236
|
+
gem floor) → action (no code change; floor stays 0.8.0-compatible, or
|
|
237
|
+
bump with an action major).
|
|
238
|
+
- The conformance vectors must be byte-identical across the train: they
|
|
239
|
+
are re-exported only when engine behavior intentionally changes, and
|
|
240
|
+
any such change is exactly what the deprecation window is for.
|
|
241
|
+
- Docs: this audit + `docs/STABILITY-1.0.md` land first (this PR); the
|
|
242
|
+
policy takes effect the day 1.0 ships.
|
|
243
|
+
|
|
244
|
+
What stays post-1.0 (policy constants, see the policy draft):
|
|
245
|
+
|
|
246
|
+
- The strict two-stage model — `setup` is never implicit; the hot path
|
|
247
|
+
never downloads.
|
|
248
|
+
- Soft-dependency policy for onnxruntime, suika, activemodel, jekyll.
|
|
249
|
+
- XDG paths + `KOTOSHU_*` envs as documented configuration surface.
|
|
250
|
+
- CLI exit codes 0/1/2/3 and the JSON/SARIF output shapes.
|
|
251
|
+
- `kotoshu.resources/v1` and the `/v1` HTTP prefix.
|
|
@@ -0,0 +1,137 @@
|
|
|
1
|
+
# Kotoshu Stability Policy — 1.0 draft
|
|
2
|
+
|
|
3
|
+
Status: DRAFT (plan 109). Takes effect the day 1.0 ships; until then the
|
|
4
|
+
usual 0.x rules apply (anything may change). The surface this policy
|
|
5
|
+
protects is itemized in `docs/STABILITY-1.0-AUDIT.md`.
|
|
6
|
+
|
|
7
|
+
## Principles
|
|
8
|
+
|
|
9
|
+
1. Semver, per delivery vehicle, cut together on one release train.
|
|
10
|
+
2. Additive changes only, within a versioned boundary (`/v1`,
|
|
11
|
+
`kotoshu.resources/v1`, the four-key suggestion row).
|
|
12
|
+
3. Behavior is frozen where the conformance vectors freeze it — the
|
|
13
|
+
vectors ARE the behavioral contract.
|
|
14
|
+
4. No removal without a deprecation window (below).
|
|
15
|
+
5. The strict two-stage resource model is permanent: `setup` is explicit,
|
|
16
|
+
the hot path never downloads. No feature may weaken it.
|
|
17
|
+
|
|
18
|
+
## Per-surface commitments
|
|
19
|
+
|
|
20
|
+
### Ruby gem (`kotoshu`)
|
|
21
|
+
|
|
22
|
+
- **Patch** (1.0.x): bug fixes only. No public API, CLI, ENV, output, or
|
|
23
|
+
conformance-vector change. `conformance/vectors.jsonl` must replay
|
|
24
|
+
byte-identically (`lib/kotoshu/conformance_runner.rb:72`).
|
|
25
|
+
- **Minor** (1.x): additive only — new methods, new config keys (must
|
|
26
|
+
default to current behavior), new CLI flags with defaults, new
|
|
27
|
+
exception subclasses of existing roots, new suggestion strategies.
|
|
28
|
+
Frozen in place: existing method signatures, the exception class list
|
|
29
|
+
at `lib/kotoshu/core/exceptions.rb`, `SCHEMA` key semantics
|
|
30
|
+
(`lib/kotoshu/configuration.rb:38`), exit codes 0/1/2/3
|
|
31
|
+
(`lib/kotoshu/cli.rb:92-97`), and the JSON/SARIF output shapes
|
|
32
|
+
(`lib/kotoshu/cli.rb:624-651,686-715`) — additions to JSON output are
|
|
33
|
+
allowed (consumers must ignore unknown keys), renames and removals are
|
|
34
|
+
not.
|
|
35
|
+
- **Major** (2.0): anything else, including conformance-vector re-exports
|
|
36
|
+
that change expected results.
|
|
37
|
+
- Ruby floor moves only at a major (`kotoshu.gemspec:17`).
|
|
38
|
+
- Requirements and dependency floors are owner decisions, never inferred
|
|
39
|
+
from code findings.
|
|
40
|
+
|
|
41
|
+
### wasm package (`@kotoshu/wasm` / kotoshu-rs)
|
|
42
|
+
|
|
43
|
+
- Independent version line (its `package.json` governs, not the crate
|
|
44
|
+
constant — `kotoshu-rs/kotoshu/src/ffi/wasm/mod.rs:149-155`), released
|
|
45
|
+
on the same train as the gem.
|
|
46
|
+
- The JS surface is the module doc contract:
|
|
47
|
+
`KotoshuWasm` (constructor / `correct` / `suggest` / `VERSION`),
|
|
48
|
+
`loadModel`, `rerank`, `semanticSuggest`, `loadLid`, `detectLanguage`,
|
|
49
|
+
and the `KotoshuModel` / `KotoshuLid` handles
|
|
50
|
+
(`kotoshu-rs/kotoshu/src/ffi/wasm/mod.rs:19-46`).
|
|
51
|
+
- The suggestion row is frozen at exactly `word`, `distance`,
|
|
52
|
+
`confidence`, `source` — the conformance `SUGGESTION_KEYS`
|
|
53
|
+
(`lib/kotoshu/conformance_exporter.rb:80`). Additive optional
|
|
54
|
+
parameters are allowed (the `bucketsBytes` third argument of
|
|
55
|
+
`loadModel` is the precedent, mod.rs:215-219); changing existing
|
|
56
|
+
parameters or return shapes is a major.
|
|
57
|
+
- `KotoshuWasm.VERSION` tracks the crate version; it is informational,
|
|
58
|
+
not the package version.
|
|
59
|
+
|
|
60
|
+
### HTTP server (kotoshu-server)
|
|
61
|
+
|
|
62
|
+
- `openapi.yaml` is the source of truth; `lib/kotoshu/server/app.rb`
|
|
63
|
+
must never diverge from it (today they match route for route).
|
|
64
|
+
- The `/v1` prefix IS the contract version. Within `/v1`: new endpoints,
|
|
65
|
+
new optional request fields, and new optional response fields are
|
|
66
|
+
minor. Changing a field type, removing a field, altering status-code
|
|
67
|
+
semantics (400/422/503), or mutating a frozen schema is a major —
|
|
68
|
+
breaking changes ship as `/v2`, with `/v1` kept for the deprecation
|
|
69
|
+
window.
|
|
70
|
+
- Error bodies keep the `{error, message, hint}` shape
|
|
71
|
+
(`kotoshu-server/openapi.yaml:246-251`).
|
|
72
|
+
|
|
73
|
+
### GitHub action (action-kotoshu)
|
|
74
|
+
|
|
75
|
+
- Inputs are additive-only and every new input defaults to the previous
|
|
76
|
+
behavior (`action-kotoshu/action.yml:8-59`). Outputs
|
|
77
|
+
(`sarif-path`, `error-count`) are frozen.
|
|
78
|
+
- Bumping the gem floor (`version` input minimum, action.yml:98-105) or
|
|
79
|
+
dropping an input is a new action major. Exit-code mapping 0/1/2/3 is
|
|
80
|
+
part of the contract (action.yml:178-189).
|
|
81
|
+
|
|
82
|
+
### Resource registry (models-fasttext-onnx)
|
|
83
|
+
|
|
84
|
+
- The spec string `kotoshu.resources/v1` is frozen
|
|
85
|
+
(`schemas/registry.schema.json:15-17`). Any schema change to the
|
|
86
|
+
registry shape means `kotoshu.resources/v2`, consumed side by side.
|
|
87
|
+
- Resource ids are permanent and never reused; artifact content changes
|
|
88
|
+
bump `release_tag` and `registry_version`, never the id (schema
|
|
89
|
+
description, registry.schema.json:5).
|
|
90
|
+
- `min_engine_version` is the compatibility lever for old gem versions
|
|
91
|
+
reading a newer registry — raising it requires a coordinated release
|
|
92
|
+
note.
|
|
93
|
+
|
|
94
|
+
### Conformance vectors (the behavioral contract)
|
|
95
|
+
|
|
96
|
+
- `conformance/vectors.jsonl` (2630 rows) freezes what the engines
|
|
97
|
+
ACTUALLY return over the fixture corpora — Ruby and native must both
|
|
98
|
+
replay it identically (`lib/kotoshu/conformance_runner.rb:97-112`).
|
|
99
|
+
- A patch release MUST NOT change any vector. A minor may add vectors
|
|
100
|
+
(new corpora/words) but not change existing expectations. Changed
|
|
101
|
+
expectations = changed user-visible suggestions = major, with a
|
|
102
|
+
CHANGELOG entry listing affected inputs.
|
|
103
|
+
- Re-export procedure stays as documented: deterministic export
|
|
104
|
+
(`lib/kotoshu/conformance_exporter.rb:44-53`), commit to this repo,
|
|
105
|
+
kotoshu-rs syncs from it (conformance_exporter.rb:54-60).
|
|
106
|
+
|
|
107
|
+
## Deprecation window
|
|
108
|
+
|
|
109
|
+
Proposal: **two minor releases** (the standard the ecosystem already
|
|
110
|
+
drifted toward — `fetch` and `dictionaries_url` both wait indefinitely;
|
|
111
|
+
1.0 is the moment to make it finite).
|
|
112
|
+
|
|
113
|
+
- Ruby API: YARD `@deprecated` tag + a `Kernel#warn` (once per process)
|
|
114
|
+
from the moment of deprecation; removal at the second subsequent
|
|
115
|
+
minor, or at the next major, whichever comes first.
|
|
116
|
+
- Constants/autoloads (e.g. the alias constants at `lib/kotoshu.rb:80-85`):
|
|
117
|
+
same window; `autoload` removal is the deletion.
|
|
118
|
+
- CLI: the command/flag stays functional but hidden from help, printing a
|
|
119
|
+
stderr deprecation notice with the replacement; removal after two
|
|
120
|
+
minors. Exit codes do not change during the window.
|
|
121
|
+
- ENV/config keys (`KOTOSHU_*`): warn on read for two minors, then remove.
|
|
122
|
+
ENV removals are called out in the changelog header — they break CI
|
|
123
|
+
setups silently otherwise.
|
|
124
|
+
- HTTP: a deprecated field/endpoint returns `Deprecation`/`Sunset` style
|
|
125
|
+
headers during the window; `/v1` keeps serving it until `/v2` has been
|
|
126
|
+
stable for one minor.
|
|
127
|
+
- Exception: anything already deprecated throughout late 0.x (the `fetch`
|
|
128
|
+
alias, `dictionaries_url`/`models_url`) may be removed AT 1.0 without a
|
|
129
|
+
new window — the 0.x line was the window.
|
|
130
|
+
|
|
131
|
+
## The 1.0 gate
|
|
132
|
+
|
|
133
|
+
1.0 ships when the audit checklist in `docs/STABILITY-1.0-AUDIT.md` is
|
|
134
|
+
cleared: deprecation dispositions landed, the release train (gem + wasm +
|
|
135
|
+
server) coordinated, conformance compare green on both engines, and this
|
|
136
|
+
policy approved by the owner. Version numbers, floors, and the cut date
|
|
137
|
+
are owner decisions — this document proposes, the owner disposes.
|
data/docs/cache-architecture.md
CHANGED
|
@@ -422,7 +422,7 @@ FastText models are downloaded from Facebook's public CDN:
|
|
|
422
422
|
|
|
423
423
|
### Adding CLI Commands
|
|
424
424
|
|
|
425
|
-
To add CLI commands for a new cache type, edit `lib/kotoshu/
|
|
425
|
+
To add CLI commands for a new cache type, edit `lib/kotoshu/cli/cache_command.rb`:
|
|
426
426
|
|
|
427
427
|
```ruby
|
|
428
428
|
desc 'download TYPE RESOURCE', 'Download a resource from GitHub'
|