kotoshu 0.10.0 → 0.11.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (40) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +36 -0
  3. data/Rakefile +35 -0
  4. data/TODO.impl/00-vision.md +17 -0
  5. data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
  6. data/TODO.impl/108-ko-ne-full-feature.md +37 -0
  7. data/TODO.impl/109-ten-audit.md +26 -0
  8. data/docs/STABILITY-1.0-AUDIT.md +251 -0
  9. data/docs/STABILITY-1.0.md +137 -0
  10. data/docs/cache-architecture.md +1 -1
  11. data/lib/kotoshu/cache/language_cache.rb +26 -11
  12. data/lib/kotoshu/cache/staged_languages.rb +45 -0
  13. data/lib/kotoshu/cache.rb +3 -0
  14. data/lib/kotoshu/cli/dict_command.rb +64 -0
  15. data/lib/kotoshu/cli.rb +1 -57
  16. data/lib/kotoshu/fluent_checker.rb +3 -0
  17. data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
  18. data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
  19. data/lib/kotoshu/keyboard/registry.rb +24 -2
  20. data/lib/kotoshu/keyboard.rb +6 -0
  21. data/lib/kotoshu/language/script.rb +128 -0
  22. data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
  23. data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
  24. data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
  25. data/lib/kotoshu/language.rb +5 -0
  26. data/lib/kotoshu/languages/ko/language.rb +56 -0
  27. data/lib/kotoshu/languages/ne/language.rb +60 -0
  28. data/lib/kotoshu/languages/nn/language.rb +26 -0
  29. data/lib/kotoshu/languages.rb +8 -0
  30. data/lib/kotoshu/multi_language_checker.rb +3 -0
  31. data/lib/kotoshu/spellchecker.rb +10 -8
  32. data/lib/kotoshu/version.rb +1 -1
  33. data/lib/kotoshu.rb +0 -5
  34. data/scripts/extract_eval_grids.rb +1 -1
  35. data/sig/kotoshu.rbs +56 -0
  36. metadata +17 -5
  37. data/lib/kotoshu/commands/cache_command.rb +0 -421
  38. data/lib/kotoshu/commands/check_command.rb +0 -301
  39. data/lib/kotoshu/commands/model_command.rb +0 -291
  40. data/lib/kotoshu/commands.rb +0 -12
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: a0093eb0f194f55beb4950a039c37e4b3abe6b33d073e7d7a8de0958e22a32cc
4
- data.tar.gz: 795382646b0e0fdee0010b76d8fcd9afefa55447f513494ec3e832fe225f6110
3
+ metadata.gz: a193e09b8fdaaf3e05a6b988e47ca460d8f7797b30e1719d109fdb9d28a89794
4
+ data.tar.gz: 377667004f6fb82be98470efa6e667a623afd96c01c61872db8b4c0048329c54
5
5
  SHA512:
6
- metadata.gz: 59fb8d99d938013b349ca779593a72b2fa8390cf725950c07acd9e117fde7ff138710fb9dfa1ba3d7581947d261dea0f990f555e1456421156316c746e8d66a8
7
- data.tar.gz: 18c2713e916171510e45028f08968d443248f5426c614e56ab21a32078543bdad69dd7f6aed4d44ced5ee00f7d18554f9050f19e84a3fb936798d3846bb00838
6
+ metadata.gz: 4c90da5e79e5b5f5a8e91a3a3b734874cc30699c9d95462f8cdae9160389601de3eea63b7231e19ba887e41d4b1e99ce12ec0ed07351868bb48c2ac8ba852007
7
+ data.tar.gz: de5adbf779c0c384b91186a73aaa571fe02bd49c84e7cad3a8bf62b366b76da4b7a853745c68b50eae6c5482bfe95fb1e55ca3f4798eb73b4194ea48c358ef62
data/CHANGELOG.md CHANGED
@@ -8,6 +8,42 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
8
8
  ## [Unreleased]
9
9
 
10
10
  ### Added
11
+ - **Korean and Nepali full-feature modules** (plan 108) - the last
12
+ big-population languages without modules move from the plan 107
13
+ basic tier to full feature. `ko` ships the Hangul tokenizer
14
+ (an eojeol is a run of Hangul syllables U+AC00-D7A3 plus bare jamo
15
+ U+1100-11FF/U+3130-318F that escaped IME composition) and the
16
+ Dubeolsik keyboard grid keyed on jamo per the KS X 5002 standard 2-set
17
+ layout, so typo proximity lands on the unit vowel mergers and
18
+ fortis/lenis confusions operate on. `ne` ships the Devanagari
19
+ tokenizer (matras U+093E-094C, the virama U+094D and vowel signs
20
+ stay attached to the base consonant; the danda and Devanagari
21
+ digits are separators) and the Devanagari InScript grid (the Indian
22
+ national standard, mirrored from the models repo eval harness and
23
+ drift-checked against it; the Nepali Traditional Romanized layout
24
+ is more common on the ground but is not a documented standard).
25
+ Both dictionaries were already staged, so `kotoshu setup ko ne`
26
+ serves them; full-feature languages grow 33 to 35.
27
+ - **Basic support for every staged dictionary language** (plan 107) -
28
+ `kotoshu setup` now serves all 95 languages staged in the
29
+ kotoshu/dictionaries manifest, not just the ones with gem modules.
30
+ `LanguageCache::AVAILABLE_LANGUAGES` derives from a vendored
31
+ snapshot of the manifest (Cache::StagedLanguages, refreshed by
32
+ `rake kotoshu:staged_languages:sync`), keeping the hot path
33
+ network-free. Languages without a module run on the basic tier:
34
+ script classification (Language::Script) picks the tokenizer whose
35
+ word regex drives check/suggest word extraction (shared Latin /
36
+ Cyrillic / Greek tokenizers plus a generic regex tokenizer for
37
+ Arabic, Hebrew, Armenian, Georgian, Devanagari and Hangul), and the
38
+ keyboard registry falls back to the script's standard grid (JCUKEN
39
+ for Cyrillic, Arabic 101, Hebrew SI-1452, QWERTY otherwise) instead
40
+ of QWERTY for everything. Two documented tiers: full feature =
41
+ module (registered keyboard + verified specimens,
42
+ `LanguageCache.full_feature_languages`), basic = manifest-present.
43
+ `nn` (Norwegian Nynorsk) is wired as a full-feature module while
44
+ here - dictionary and model are both staged. Modules keep winning
45
+ over the fallbacks; the 32 existing modules behave identically
46
+ (conformance vectors untouched).
11
47
  - **Personal dictionary in the check path** (plan 105) -
12
48
  `Spellchecker#check` (and therefore `Kotoshu.check`,
13
49
  `Kotoshu.check_file`, and `kotoshu check`) no longer flags words
data/Rakefile CHANGED
@@ -32,6 +32,41 @@ report_divergences = lambda do |divergences|
32
32
  end
33
33
 
34
34
  namespace :kotoshu do
35
+ namespace :staged_languages do
36
+ desc <<~DESC
37
+ Regenerate the vendored staged-language list (plan 107). Fetches \
38
+ the dictionaries manifest at the SourceRegistry pin, extracts the \
39
+ distinct language codes from its resources, and rewrites \
40
+ lib/kotoshu/cache/staged_languages.rb in place so \
41
+ LanguageCache::AVAILABLE_LANGUAGES tracks the manifest. Run after \
42
+ the dictionaries repo stages new languages.
43
+ DESC
44
+ task :sync do
45
+ require "kotoshu"
46
+ require "json"
47
+
48
+ url = Kotoshu::SourceRegistry.new.url_for(:dict_manifest)
49
+ puts "Fetching #{url}"
50
+ bytes = Kotoshu::Integrity::NetHTTP.get(url)
51
+ abort "manifest not found at #{url}" if bytes.nil?
52
+
53
+ manifest = JSON.parse(bytes)
54
+ codes = manifest.fetch("resources").values.map { |r| r.fetch("language") }.uniq.sort
55
+ abort "manifest carried no languages" if codes.empty?
56
+
57
+ target = File.expand_path("lib/kotoshu/cache/staged_languages.rb", __dir__)
58
+ source = File.read(target)
59
+ updated = source.sub(/CODES = %w\[\n(?: {8}\S+(?: \S+)*\n)+ {6}\]\.freeze/) do
60
+ wrapped = codes.each_slice(10).map { |slice| " #{slice.join(' ')}" }.join("\n")
61
+ "CODES = %w[\n#{wrapped}\n ].freeze"
62
+ end
63
+ abort "CODES block not found in #{target}" if updated == source
64
+
65
+ File.write(target, updated)
66
+ puts "Rewrote #{target}: #{codes.size} languages (was #{Kotoshu::Cache::StagedLanguages::CODES.size})"
67
+ end
68
+ end
69
+
35
70
  namespace :conformance do
36
71
  desc <<~DESC
37
72
  Export conformance vectors for kotoshu-rs (plan 67 M3). Walks the Spylls \
@@ -262,3 +262,20 @@ yet passing bucket bytes to loadModel.
262
262
  | 106 | [Native LID detection](106-native-lid-detection.md) | detect is a 7-language heuristic; the pure-Rust lid reader exists unused |
263
263
  | 107 | models [09 bucket expansion](../../models-fasttext-onnx/TODO.impl/09-bucket-table-expansion.md) | bucket OOV siblings exist only for en and de |
264
264
  | — | site 11 playground buckets | wasm 0.4.0 accepts bucketsBytes; the worker never fetches the sibling |
265
+
266
+
267
+ Plans 107-109 are **wave 8, reachability and the road to 1.0**
268
+ (2026-09-07): after wave 7 (personal dict in check, 176-language
269
+ detection, buckets rolling to all languages), the remaining user gaps
270
+ are the sixty-plus staged dictionaries unreachable without a module,
271
+ the last big-population languages (ko, ne), and the absence of a
272
+ stability commitment adopters can build on.
273
+
274
+ | # | Plan | Gap it closes |
275
+ |---|---|---|
276
+ | 107 | [Basic support for every staged language](107-basic-support-every-staged-language.md) | setup rejects 60+ staged languages; modules become upgrades, not gates |
277
+ | 108 | [ko and ne full-feature](108-ko-ne-full-feature.md) | Hangul and Devanagari - the last ~100M speakers without support |
278
+ | 109 | [The 1.0 readiness audit](109-ten-audit.md) | no frozen surface or stability policy for adopters |
279
+
280
+ Sequencing: 108 after 107 (same area); 109 is analysis-only and runs in
281
+ parallel. The 1.0 cut itself remains the owner's version call.
@@ -0,0 +1,42 @@
1
+ # Plan 107 — Basic support for every staged dictionary language
2
+
3
+ ## Why
4
+ `kotoshu setup` only serves the 32 languages with modules. Sixty-plus staged
5
+ dictionaries (from gd to ia) are unreachable: a Welsh or Icelandic user
6
+ cannot use the gem at all, even though the dictionary and model are
7
+ downloadable. Modules should be the UPGRADE (keyboard + verified specimens),
8
+ not the gate.
9
+
10
+ ## Work (gem repo)
11
+ 1. `LanguageCache::AVAILABLE_LANGUAGES` derives from the dictionaries
12
+ manifest (the source the registry/staging uses) instead of the hardcoded
13
+ module list; modules register themselves as full-feature upgrades.
14
+ 2. Default behavior for module-less languages: script-appropriate tokenizer
15
+ (reuse the script-regex machinery from plan 91), generic keyboard
16
+ fallback (QWERTY for Latin, JCUKEN Cyrillic, Arabic 101 Arabic-script),
17
+ setup/check/suggest all functional.
18
+ 3. Wire nn (model + dictionary exist) as a full-feature module while here;
19
+ note sr-Latn as follow-up if the Latin mapping needs product sign-off.
20
+ 4. Specs: setup + specimen round trip on 3 module-less staged languages
21
+ (e.g. is, cy, gd); full-feature languages behave identically to today;
22
+ conformance untouched.
23
+
24
+ ## Verification
25
+ Live `kotoshu setup is && check -l is` round trip; suite green; the 32
26
+ modules' outputs byte-identical.
27
+
28
+ ## Status
29
+ Executed 2026-09-07 — AVAILABLE_LANGUAGES derives from a vendored
30
+ snapshot of the dictionaries manifest (Cache::StagedLanguages, 95
31
+ languages, refreshed by `rake kotoshu:staged_languages:sync`);
32
+ module-less languages ride Language::Script (script tokenizer for
33
+ word extraction: shared Latin/Cyrillic/Greek classes plus a generic
34
+ regex tokenizer for Arabic/Hebrew/Armenian/Georgian/Devanagari/
35
+ Hangul) and the script keyboard default (JCUKEN Cyrillic, Arabic 101,
36
+ Hebrew SI-1452, QWERTY Latin). Tiers documented: full feature =
37
+ module (LanguageCache.full_feature_languages, 33 with nn); basic =
38
+ manifest-present. nn wired as a full-feature module. Specimens
39
+ engine-verified for is cy gd (basic) + nn; live round trips
40
+ setup+check -l for is cy gd nn; conformance 2630 vectors 0 failures;
41
+ suite 3948 examples 0 failures. sr-Latn stays follow-up (normalize
42
+ maps it to sr; needs product sign-off).
@@ -0,0 +1,37 @@
1
+ # Plan 108 — Korean and Nepali: the last big-population languages
2
+
3
+ ## Why
4
+ ko (~82M) and ne (~16M) were deferred from batch 3 because Hangul jamo
5
+ input and Devanagari conjuncts need ja-style tokenizer treatment. They are
6
+ the largest populations without full-feature support; both have staged
7
+ dictionaries and models.
8
+
9
+ ## Work (gem repo; sequenced AFTER plan 107 merges — same area)
10
+ 1. ko: Hangul tokenizer (syllable-block aware, jamo-level word regex so
11
+ check extracts real eojeol), Korean keyboard grid (Dubeolsik 2-set),
12
+ module + specimens engine-verified.
13
+ 2. ne: Devanagari tokenizer (matras and conjuncts stay attached to the
14
+ base consonant — the Unicode grapheme rules the script needs), Roman
15
+ keyboard fallback or a Nepali traditional grid if defensibly sourced,
16
+ module + specimens.
17
+ 3. AVAILABLE_LANGUAGES + site-flip deliverables (specimen table, sizes).
18
+
19
+ ## Verification
20
+ Same discipline as batch 3: every specimen engine-verified through
21
+ correct?/suggest; RTL/complex-script sentence round trips through
22
+ Kotoshu.check; suite green.
23
+
24
+ ## Status
25
+ Executed 2026-09-07 — ko: HangulTokenizer (eojeol = run of syllables
26
+ U+AC00-D7A3 plus bare jamo U+1100-11FF/U+3130-318F), Dubeolsik grid
27
+ keyed on jamo per KS X 5002 (the eval harness models ko slips as
28
+ jamo confusion pairs, not a key grid, so no drift check applies),
29
+ module + 7 engine-verified specimens. ne: DevanagariTokenizer
30
+ (matras and virama conjuncts attached to the base consonant; danda
31
+ and Devanagari digits are separators), Devanagari InScript grid
32
+ mirrored from the eval harness _NE_INSCRIPT and drift-checked (the
33
+ defensible standard — Traditional Romanized is common but
34
+ undocumented), module + 3 specimens. Full-feature languages 33 -> 35.
35
+ Live round trips setup + check -l ko ne recorded; conformance 2630
36
+ vectors 0 failures; suite 3985 examples 0 failures. Specimen table
37
+ and dictionary sizes in the PR body for the site flip.
@@ -0,0 +1,26 @@
1
+ # Plan 109 — The 1.0 readiness audit
2
+
3
+ ## Why
4
+ Adopters hesitate at 0.x. A 1.0 cut needs a frozen public surface and a
5
+ stated policy. The audit inventories what 1.0 would freeze and what it
6
+ would deprecate — it does NOT cut versions (owner decision).
7
+
8
+ ## Work (gem repo docs/, analysis only)
9
+ 1. Inventory the public surface: Ruby API (Kotoshu module functions,
10
+ Spellchecker, Configuration, Tasks, validators), CLI flags, wasm JS
11
+ surface (KotoshuWasm, loadModel/rerank/semanticSuggest/loadLid/
12
+ detectLanguage), HTTP contract (openapi.yaml), action inputs, registry
13
+ schema.
14
+ 2. For each: stable / needs-deprecation-notice / internal-accidentally-
15
+ public. Flag anything accidentally public that 1.0 should privatize.
16
+ 3. Draft the stability policy (semver commitments, deprecation window,
17
+ conformance vectors as the behavioral contract) as docs/STABILITY.md.
18
+ 4. A 1.0 checklist: what must land first (nothing mechanical is known to
19
+ block), what the release train looks like (gem+wasm+server together).
20
+
21
+ ## Verification
22
+ Every inventory line cites the file that defines it; the deprecation list
23
+ reviewed against gemspec/exported constants.
24
+
25
+ ## Status
26
+ Pending
@@ -0,0 +1,251 @@
1
+ # The 1.0 Readiness Audit
2
+
3
+ Plan 109. Analysis and documentation only — no behavior change, no version
4
+ bump, no release. The version cut itself is an owner decision.
5
+
6
+ - Audited at: gem 0.10.0 (`lib/kotoshu/version.rb:4`), main @ 73b7fac.
7
+ - Companion policy draft: `docs/STABILITY-1.0.md`.
8
+ - Method: every inventory line cites the file that defines the surface.
9
+ Reachability was verified by grep, not by assumption: a class being
10
+ present under `lib/` is not the same as being wired to `exe/kotoshu`.
11
+
12
+ ## Inventory
13
+
14
+ Verdicts: **stable** (freeze at 1.0), **deprecate-before-1.0** (needs a
15
+ deprecation cycle first), **accidentally-public** (internal or dead, 1.0
16
+ should privatize or delete).
17
+
18
+ ### Ruby gem — module functions (`lib/kotoshu.rb`)
19
+
20
+ | Surface | Where defined | Verdict | Note |
21
+ |---|---|---|---|
22
+ | `configure` / `configuration` / `configuration=` | `lib/kotoshu.rb:98,109,127` | stable | Type-checked global swap (127-131). |
23
+ | `spellchecker` / `spellchecker_for` | `lib/kotoshu.rb:139,157` | stable | Cache-only; README-documented (README.adoc:18). |
24
+ | `resolve` | `lib/kotoshu.rb:179` | stable | Stage-2 resolve; README.adoc:364. |
25
+ | `setup` / `setup?` / `languages_setup` | `lib/kotoshu.rb:220,245,257` | stable | Stage-1 entry; idempotent; `want:`/`tier:`/local-source opts frozen. |
26
+ | `reset_spellchecker` | `lib/kotoshu.rb:268` | stable | Test/embedding seam. |
27
+ | `correct?` / `misspelled?` / `suggest` / `check` / `check_file` / `check_files` | `lib/kotoshu.rb:286,297,313,331,349,363` | stable | The hot path; strict two-stage contract is the product identity. |
28
+ | `detect_language` / `detect_language_with_confidence` / `setup_lid` / `setup_lid?` | `lib/kotoshu.rb:465,481,501,508` | stable | Native LID with heuristic fallback (plan 106). |
29
+ | `get_language` / `language_registered?` / `supported_languages` / `language` | `lib/kotoshu.rb:519,530,540,443` | stable | `language` is a trivial namespace accessor; keep. |
30
+ | `dictionary` / `trie` / `suggestion_pipeline` | `lib/kotoshu.rb:371,388,409` | stable | Convenience factories; exercised by `examples/04..06`. |
31
+ | `register_dictionary_type` / `register_suggestion_algorithm` | `lib/kotoshu.rb:422,433` | stable | The extension points; 1.0 must not narrow them. |
32
+ | `VERSION` | `lib/kotoshu/version.rb:4` | stable | |
33
+
34
+ ### Ruby gem — core classes
35
+
36
+ | Surface | Where defined | Verdict | Note |
37
+ |---|---|---|---|
38
+ | `Kotoshu::Spellchecker` public methods: `correct?`, `incorrect?`, `suggest`, `check_word`, `check`, `check_file`, `check_directory`, `tokenize`, `dictionary`, `reload_dictionary` | `lib/kotoshu/spellchecker.rb:118,130,143,164,196,242,265,280,308,315` | stable | Constructor kwargs (`dictionary:`, `config:`, `resource_bundle:`) frozen at 65. |
39
+ | `Kotoshu::Spellchecker::ASCII_WORD_REGEX` | `lib/kotoshu/spellchecker.rb:39` | accidentally-public | Internal fallback word-char set exposed as a constant. Freeze + document, or privatize (see deprecations). |
40
+ | `Kotoshu::Configuration`: `SCHEMA`, `DEFAULTS`, accessors, `get`, `key?`, `load_dictionary`, `reset_dictionary`, `source_registry`, `self.default/instance/instance=/reset` | `lib/kotoshu/configuration.rb:38,222,423,431,447,474,517,541,565,576,585` | stable | ENV contract (`KOTOSHU_*`) rides on SCHEMA — every key there is public surface. |
41
+ | `Configuration#dictionaries_url` / `#models_url` | `lib/kotoshu/configuration.rb:63-74` (SCHEMA), `:318-321` (attrs) | deprecate-before-1.0 | SCHEMA descriptions already say "Deprecated: use repos_base_url + pins via SourceRegistry". |
42
+ | `ResourceManager.setup/resolve/setup?/languages_setup`, `DEFAULT_WANT` | `lib/kotoshu/resource_manager.rb:69,97,21` | stable | Facade delegates here; two-stage model frozen. |
43
+ | `ResourceBundle` (`dictionary`/`frequency`/`model`/`rules`) | `lib/kotoshu/resource_bundle.rb` | stable | Consumed by `Kotoshu.resolve` (kotoshu.rb:179). |
44
+ | Exception hierarchy: `Error` + `DictionaryNotFoundError`, `InvalidDictionaryFormatError`, `ConfigurationError`, `SpellcheckError`, `AffixRuleError`, `ResourceNotCachedError`, `ResourceNotSetupError`, `ResourceResolutionError`, `IntegrityError`, `SuikaUnavailable` | `lib/kotoshu/core/exceptions.rb:8,14,32,54,72,90,106,122,136,156,177`; autoloaded `lib/kotoshu.rb:65-75` | stable | Message text is not frozen; class names are. |
45
+ | `Kotoshu::Paths` (XDG + `KOTOSHU_{CACHE,CONFIG,DATA}_PATH`) | `lib/kotoshu/paths.rb` | stable | On-disk layout is user-visible contract. |
46
+ | Autoloaded namespaces behind `Kotoshu::*` | `lib/kotoshu.rb:14-62` | stable (as namespaces) | Namespaces stay; member-by-member stability is YARD-documented per class. 1.0 freezes namespace existence, not every leaf. |
47
+ | Top-level singleton aliases `Kotoshu::LanguageCache`, `Kotoshu::LanguageIdentifier`, `Kotoshu::ModelCache`, `Kotoshu::SemanticAnalyzer` | `lib/kotoshu.rb:80,81,84,85` | accidentally-public | Zero hits in lib/, spec/, README; canonical homes are `Kotoshu::Cache::LanguageCache`, `Kotoshu::Language::*`, `Kotoshu::Cache::ModelCache`, `Kotoshu::Analyzers::SemanticAnalyzer`. |
48
+ | `Kotoshu::Debug`, `DebugLogger`, `Metrics`, `MetricsCollector` | `lib/kotoshu.rb:78-83`; `lib/kotoshu/debug_mode.rb`, `metrics_module.rb`, `metrics_collector.rb` | stable | Documented in their doc comments; `Kotoshu::Metrics.stats` is README-documented (README.adoc:211). |
49
+ | `Kotoshu::FluentChecker` / `Kotoshu::MultiLanguageChecker` | `lib/kotoshu/fluent_checker.rb`, `lib/kotoshu/multi_language_checker.rb` | deprecate-before-1.0 (or document) | Spec-covered (`spec/kotoshu/{fluent,multi_language}_checker_spec.rb`) but absent from README.adoc — decide: document or deprecate. |
50
+
51
+ ### Ruby gem — framework integrations (all soft deps, plan 89)
52
+
53
+ | Surface | Where defined | Verdict | Note |
54
+ |---|---|---|---|
55
+ | `require "kotoshu/validators"` → `Kotoshu::Validators::SpellingValidator` (+ `ACTIVE_MODEL_LOADED`, `ActiveModelUnavailable`) | `lib/kotoshu/validators/spelling_validator.rb:10-99`; autoload `lib/kotoshu.rb:60` | stable | Placeholder class keeps constant resolvable without Rails (91-99). |
56
+ | `require "kotoshu/rspec"` → `Kotoshu::Rspec::Matchers` (`expect_words`, `expect_document`, `all_be_spelled_correctly`, `be_spelled_correctly(in:)`) | `lib/kotoshu/rspec.rb:26-160`; autoload `lib/kotoshu.rb:61` | stable | Matcher names are the contract. |
57
+ | `require "kotoshu/tasks"` → default `rake kotoshu:check`; `Kotoshu::Tasks::CheckTask` | `lib/kotoshu/tasks.rb:11`, `lib/kotoshu/tasks/check_task.rb:22` | stable | require-only (defining a task on load is the point — kotoshu.rb:56-59). |
58
+ | `require "kotoshu/jekyll"` → `Kotoshu::Jekyll::Generator` (+ `JEKYLL_LOADED`, `JekyllUnavailable`) | `lib/kotoshu/jekyll.rb:9-96`; autoload `lib/kotoshu.rb:62` | stable | Baseline honored via `Baseline::Store::DEFAULT_FILENAME`. |
59
+
60
+ ### Ruby gem — conformance surface (the behavioral contract)
61
+
62
+ | Surface | Where defined | Verdict | Note |
63
+ |---|---|---|---|
64
+ | `conformance/vectors.jsonl` (2630 rows, committed) | `conformance/vectors.jsonl`; sync contract documented in `lib/kotoshu/conformance_exporter.rb:54-60` | stable — frozen expectations | `expected` records what the Ruby engine ACTUALLY returns, not linguistic truth (conformance_exporter.rb:31-37). |
65
+ | Row shape v0: `kind/language/dictionary/input/limit/expected`; suggestion keys `word,distance,confidence,source` | `lib/kotoshu/conformance_exporter.rb:16-29,80` | stable | Four-key shape shared with wasm and `ffi::ruby`. |
66
+ | `Kotoshu::ConformanceRunner` (`run`, `compare`, rake tasks) | `lib/kotoshu/conformance_runner.rb:26,72,97` | stable | Guards Ruby-vs-vectors and native-vs-Ruby equality. |
67
+ | `Kotoshu::ConformanceExporter` (`export`, corpora selection, `EXCLUDED_FIXTURES`) | `lib/kotoshu/conformance_exporter.rb:61-254` | stable | Deterministic re-export (44-53). |
68
+
69
+ ### CLI (`exe/kotoshu` → `Kotoshu::Cli::Cli`)
70
+
71
+ | Surface | Where defined | Verdict | Note |
72
+ |---|---|---|---|
73
+ | Commands: `check`, `setup`, `dict`, `cache`, `personal`, `completions`, `baseline`, `status`, `version` (`--version`/`-V`) | `lib/kotoshu/cli.rb:178,241,317-329,349,358-367` | stable | |
74
+ | Class options `--language/-l`, `--format/-f` (text,json,sarif), `--interactive/-i`, `--verbose/-v` | `lib/kotoshu/cli.rb:102-125` | stable | |
75
+ | `check` options `--baseline`, `--show-suppressed`, `--include`, `--exclude`, `--personal` | `lib/kotoshu/cli.rb:151-168` | stable | |
76
+ | `setup` options `--aff --dic --from --frequency --want --model --tier --force --strict --list` | `lib/kotoshu/cli.rb:214-240` | stable | |
77
+ | Exit codes 0/1/2/3 | `lib/kotoshu/cli.rb:92-97,145-149` | stable | Consumed by action-kotoshu (action.yml:178-179). |
78
+ | JSON output top-level keys; SARIF 2.1.0 shape | `lib/kotoshu/cli.rb:624-651,686-715` | stable | Machine-readable contract; additive keys only. |
79
+ | `cache` subcommands `list/info/clean/evict/download/purge/validate` | `lib/kotoshu/cli/cache_command.rb:27,50,79,101,133,166,184` | stable | |
80
+ | `personal` subcommands `add/remove/list/import/path/clear` | `lib/kotoshu/cli/personal_command.rb:28-89` | stable | |
81
+ | `baseline init` (`--output`, `--language`) | `lib/kotoshu/cli/baseline_command.rb:15-36` | stable | |
82
+ | `completions bash/zsh` | `lib/kotoshu/cli/completions_command.rb:13-26` | stable | Catalog is hand-maintained — must track command changes. |
83
+ | `fetch` (hidden alias of `setup`) | `lib/kotoshu/cli.rb:280-315` | deprecate-before-1.0 | Already hidden and labeled deprecated; remove at 1.0. |
84
+
85
+ ### Ruby gem — accidentally-public legacy commands (the hazard block)
86
+
87
+ | Surface | Where defined | Verdict | Note |
88
+ |---|---|---|---|
89
+ | `::DictCommand` (top-level constant!) | `lib/kotoshu/cli.rb:10` | accidentally-public | Defined on `Object`, outside `module Kotoshu`. Pollutes the global namespace of every process that requires the gem; collides with any other gem defining `DictCommand`. Move under `Kotoshu::Cli` before 1.0. |
90
+ | `Kotoshu::Commands::CheckCommand` | `lib/kotoshu/commands/check_command.rb:4`; autoload `lib/kotoshu/commands.rb:9` | accidentally-public | Not wired into `Cli` (no `subcommand "check"` at cli.rb:317-329 — `check` is a method on `Cli` itself, cli.rb:178). Half-placeholder implementation; its hunspell path calls `SpellChecker.new` (check_command.rb:198) which resolves to nonexistent `Kotoshu::SpellChecker` (the real class is `Kotoshu::Spellchecker`) → `NameError` at runtime. Declares flags that do not exist on the real CLI (`--model`, `-o`, `csv/yaml` formats), contradicting cli.rb:108-113. Delete or privatize. |
91
+ | `Kotoshu::CacheCommand` | `lib/kotoshu/commands/cache_command.rb:4` | accidentally-public | Dead duplicate of the wired `Kotoshu::Cli::CacheCommand` (autoload cli.rb:70; subcommand cli.rb:321). Nothing requires this file (verified by grep); acknowledged as historical in `lib/kotoshu/commands.rb:7-8`. Still shipped in the gem (`spec.files` = `git ls-files`, kotoshu.gemspec:30-38). Delete. |
92
+ | `Kotoshu::Commands::ModelCommand` | `lib/kotoshu/commands/model_command.rb:4`; autoload `lib/kotoshu/commands.rb:10` | accidentally-public | No `kotoshu model` subcommand is registered anywhere. `convert` shells out to `scripts/convert_fasttext_to_onnx.py` (model_command.rb:33-36) — untracked in git, therefore NOT shipped in the gem, so the command cannot work as packaged. Delete. |
93
+
94
+ ### Ruby gem — native extension
95
+
96
+ | Surface | Where defined | Verdict | Note |
97
+ |---|---|---|---|
98
+ | Optional Rust ext build (`extconf.rb`) | `kotoshu.gemspec:53`; `ext/kotoshu_native/extconf.rb` | stable | Soft: pure-Ruby is the complete fallback (gemspec:44-52). |
99
+ | `Kotoshu::Native.available?`, `Native.suggestion(row)`, `Native::Unavailable`, `Native::EXTENSION` | `lib/kotoshu/native.rb:49,65,35,39` | stable | Extension defines `Native::VERSION`, `Native::Error`, `Native::Dictionary.load/correct?/suggest` (native.rb:9-14). |
100
+ | `Kotoshu::NativeBackend.resolve` boundary (ruby/auto/native) | `lib/kotoshu/native_backend.rb:45-73` | stable | Accelerator-only boundary is documented and frozen. |
101
+
102
+ ### wasm package (kotoshu-rs)
103
+
104
+ | Surface | Where defined | Verdict | Note |
105
+ |---|---|---|---|
106
+ | `KotoshuWasm` constructor `(affSource, dicSource)`, `correct(word)`, `suggest(word, limit?)`, static `VERSION` | `kotoshu-rs/kotoshu/src/ffi/wasm/mod.rs:165-174,178,185,152-155` | stable | Contents-not-paths loading; errors reject with the Rust message. |
107
+ | `loadModel(modelBytes, vocabBytes, bucketsBytes?)` → `KotoshuModel`; `.free()` | mod.rs:220-235,207-209 | stable | Optional third arg added by plan 103 — additive pattern to keep. |
108
+ | `rerank(model, word, context)` → number in [-1,1] | mod.rs:241-244 | stable | Mean-cosine contract documented (mod.rs:73-82). |
109
+ | `semanticSuggest(model, word, k?)` → `{word, score}[]` | mod.rs:252-266 | stable | |
110
+ | `loadLid(onnxBytes, vocabBytes)` → `KotoshuLid`; `detectLanguage(lid, text)` → `{code, score}` | mod.rs:281-287,293-300 | stable | Parity contract with the gem documented (mod.rs:84-94). |
111
+ | Suggestion row shape: exactly `word/distance/confidence/source` | mod.rs:48-54,190-198 | stable — frozen | Same four keys as the conformance vectors (`conformance_exporter.rb:80`). |
112
+ | npm identity | `kotoshu-rs/kotoshu-wasm/pkg/package.json:2,5` | needs a decision | Package name is `kotoshu-wasm` but every doc says `@kotoshu/wasm` (mod.rs:2-3); publication still blocked on npm org credentials. Decide the final name before 1.0 — npm names are sticky. |
113
+
114
+ ### HTTP server (kotoshu-server)
115
+
116
+ | Surface | Where defined | Verdict | Note |
117
+ |---|---|---|---|
118
+ | `GET /` (metadata), `GET /v1/health`, `GET /v1/version`, `GET /v1/languages` | `kotoshu-server/openapi.yaml:15,25,35,45`; routes `lib/kotoshu/server/app.rb:290,300,306,311` | stable | |
119
+ | `POST /v1/check` (`text`, `language`, `format`, `model`), `POST /v1/suggest` (`word`, `language`, `max` 1-50), `POST /v1/detect` (`text`) | openapi.yaml:55-117,179-203; routes app.rb:318,350,366 | stable | Errors: 400/422/503 shapes (openapi.yaml:118-137). |
120
+ | Response schemas `DocumentResult/WordError/Suggestions/Suggestion/Detection` (incl. `engine: lid-176|heuristic`) | openapi.yaml:204-245 | stable | openapi.yaml is the contract source of truth; app.rb matches it 1:1 (verified). |
121
+ | `/v1` prefix as the contract version | openapi.yaml:25-100 | stable | Breaking changes require `/v2`, not field mutations. |
122
+
123
+ ### GitHub action (action-kotoshu)
124
+
125
+ | Surface | Where defined | Verdict | Note |
126
+ |---|---|---|---|
127
+ | Inputs: `files, language, format, baseline, include, exclude, show_suppressed, fail_on_error, category, version, prewarm_languages, offline, output_path` | `action-kotoshu/action.yml:8-59` | stable — additive-only | New inputs must default to current behavior. |
128
+ | Outputs: `sarif-path`, `error-count` | action.yml:61-67 | stable | |
129
+ | Gem floor `>= 0.8.0` (v2 of the action) | action.yml:45,98-105 | stable | Floor bumps are breaking for the action and need a major action version. |
130
+
131
+ ### Registry schema (models-fasttext-onnx)
132
+
133
+ | Surface | Where defined | Verdict | Note |
134
+ |---|---|---|---|
135
+ | `spec: "kotoshu.resources/v1"` (const) | `models-fasttext-onnx/schemas/registry.schema.json:15-17` | stable — frozen | Change requires a new spec string, never an edit. |
136
+ | Resource id pattern `kotoshu://models/(lid/lid-176|{lang}/(full|fluency|mini|buckets))` | registry.schema.json:35 | stable | Ids are never reused; content changes bump `release_tag` (schema description, registry.schema.json:5). |
137
+ | Resource object fields incl. `min_engine_version`, `sha256`, `license` | registry.schema.json:43-165 | stable | `additionalProperties: false` — additive fields need a schema change = coordinated release. |
138
+
139
+ ## Deprecation list for the 1.0 cut
140
+
141
+ Concrete, with defining locations. All need small code PRs — deliberately
142
+ NOT implemented by this docs-only change.
143
+
144
+ 1. `::DictCommand` — `lib/kotoshu/cli.rb:10`. Global-namespace leak from a
145
+ file that otherwise lives in `Kotoshu::Cli`. Rename to
146
+ `Kotoshu::Cli::DictCommand` (internal; `subcommand "dict"` wiring at
147
+ cli.rb:317-318 is the only consumer). Highest-priority item: it can
148
+ break other gems in the same process, which is exactly the class of
149
+ thing 1.0 exists to prevent.
150
+ Handled in this PR: the class moved to
151
+ `lib/kotoshu/cli/dict_command.rb` as `Kotoshu::Cli::DictCommand`
152
+ (autoloaded like its sibling subcommands); `::DictCommand` is gone.
153
+ 2. `Kotoshu::Commands::CheckCommand` — `lib/kotoshu/commands/check_command.rb:4`
154
+ (autoload `lib/kotoshu/commands.rb:9`). Unwired, placeholder-laden, and
155
+ broken at check_command.rb:198 (`SpellChecker` → `NameError`; real class
156
+ is `Kotoshu::Spellchecker`, spellchecker.rb:21). Its flag surface
157
+ (`--model`, `-o/--output`, `csv|yaml` formats) contradicts the real CLI
158
+ and will mislead users who find it via constants. Delete; keep
159
+ `lib/kotoshu/commands.rb` as the namespace or remove both.
160
+ Handled in this PR: deleted, along with the now-empty `Commands`
161
+ namespace (file and autoload). No spec referenced it.
162
+ 3. `Kotoshu::CacheCommand` — `lib/kotoshu/commands/cache_command.rb:4`.
163
+ Dead duplicate of `Kotoshu::Cli::CacheCommand`; unreachable by any
164
+ require in the repo; still shipped by the gemspec file list. Delete.
165
+ Handled in this PR: deleted. No spec referenced it.
166
+ 4. `Kotoshu::Commands::ModelCommand` — `lib/kotoshu/commands/model_command.rb:4`
167
+ (autoload `lib/kotoshu/commands.rb:10`). No `model` subcommand exists;
168
+ `convert` depends on an untracked Python script not shipped in the gem
169
+ (model_command.rb:33-36 vs gemspec:30-38). Delete.
170
+ Handled in this PR: deleted. No spec referenced it.
171
+ 5. Alias constants `Kotoshu::LanguageCache`, `Kotoshu::LanguageIdentifier`,
172
+ `Kotoshu::ModelCache`, `Kotoshu::SemanticAnalyzer` — `lib/kotoshu.rb:80,81,84,85`.
173
+ No usage anywhere in lib/, spec/, or README. Add a deprecation `warn`
174
+ in a 0.x minor and drop the autoloads at 1.0 (canonical names stay).
175
+ Handled in this PR: grep confirmed zero references to the aliases
176
+ anywhere (lib, spec, exe, docs, README, sig) — the short-name hits all
177
+ resolve to the canonical constants lexically — so the autoloads were
178
+ dropped straight away, ahead of the suggested warn cycle. Canonical
179
+ names are untouched.
180
+ 6. CLI `fetch` — `lib/kotoshu/cli.rb:280-315`. Already hidden and marked
181
+ deprecated. Remove at 1.0.
182
+ 7. `Configuration#dictionaries_url` / `#models_url` —
183
+ `lib/kotoshu/configuration.rb:63-74,318-321`. Marked deprecated in SCHEMA
184
+ since the SourceRegistry pins landed. Emit a warning on use, remove the
185
+ SCHEMA entries at 1.0 (ENV vars `KOTOSHU_DICTIONARIES_URL` /
186
+ `KOTOSHU_MODELS_URL` go with them — call that out in the changelog).
187
+ 8. `Kotoshu::Spellchecker::ASCII_WORD_REGEX` — `lib/kotoshu/spellchecker.rb:39`.
188
+ Internal fallback set posing as API. Either document + freeze or make it
189
+ private before 1.0.
190
+ Handled in this PR: zero external references confirmed by grep, so the
191
+ constant was deleted and its literal inlined at the single internal use
192
+ site (constructor fallback). Behavior unchanged.
193
+ 9. `Kotoshu::FluentChecker` / `Kotoshu::MultiLanguageChecker` —
194
+ `lib/kotoshu/fluent_checker.rb`, `lib/kotoshu/multi_language_checker.rb`.
195
+ Spec-covered but undocumented in README.adoc. Decide per class: document
196
+ as stable, or deprecate. Do not freeze silently.
197
+ Handled in this PR: both class docstrings now state "Experimental and
198
+ not public API: outside the 1.0 stability freeze, may change or be
199
+ removed without a deprecation cycle." Specs stay.
200
+ 10. wasm npm identity — `kotoshu-rs/kotoshu-wasm/pkg/package.json:2` says
201
+ `kotoshu-wasm` while `kotoshu-rs/kotoshu/src/ffi/wasm/mod.rs:2-3` and
202
+ all docs say `@kotoshu/wasm`. Resolve before first publication; the
203
+ audit treats the documented scoped name as intended.
204
+ Handled in this PR: verified, no code change needed.
205
+ `kotoshu-rs/scripts/wasm_build.sh` rewrites the generated
206
+ `package.json` name to `@kotoshu/wasm` on every build (the source
207
+ manifest name is a wasm-pack crate-name artifact; `pkg/` is a
208
+ gitignored build output). The shipped identity is already
209
+ `@kotoshu/wasm`; the owner decision that remains is publication
210
+ credentials, not the name.
211
+
212
+ ## 1.0 checklist
213
+
214
+ Known blockers — none are mechanical code fixes; all are owner decisions
215
+ or ecosystem plumbing, stated honestly:
216
+
217
+ 1. Dispositions for the ten deprecation items above (small code PRs; this
218
+ audit only records them).
219
+ 2. npm org + publish credentials for `@kotoshu/wasm` (and the name
220
+ decision, item 10). The wasm build exists; publication is blocked.
221
+ 3. The JS SDK PR (kotoshu-js #1) still pending merge — the HTTP contract
222
+ it consumes must be the frozen one.
223
+ 4. PyPI token for kotoshu-python publication (same train).
224
+ 5. kotoshu-server is at 0.1.0 with an unexplained prewarm hang noted in
225
+ ecosystem state — the 1.0 train should not ship a server cut with a
226
+ known hang.
227
+ 6. RBS coverage: `sig/kotoshu.rbs` should be checked against the frozen
228
+ surface so the types do not contradict the freeze.
229
+ 7. Conformance vectors green on both engines
230
+ (`rake kotoshu:conformance:compare`) at the cut commit.
231
+
232
+ Release-train shape (gem + wasm + server together, per plan):
233
+
234
+ - Cut order: registry release_tag (if models moved) → gem 1.0 → wasm
235
+ package (independent version line, same day) → server (depends on the
236
+ gem floor) → action (no code change; floor stays 0.8.0-compatible, or
237
+ bump with an action major).
238
+ - The conformance vectors must be byte-identical across the train: they
239
+ are re-exported only when engine behavior intentionally changes, and
240
+ any such change is exactly what the deprecation window is for.
241
+ - Docs: this audit + `docs/STABILITY-1.0.md` land first (this PR); the
242
+ policy takes effect the day 1.0 ships.
243
+
244
+ What stays post-1.0 (policy constants, see the policy draft):
245
+
246
+ - The strict two-stage model — `setup` is never implicit; the hot path
247
+ never downloads.
248
+ - Soft-dependency policy for onnxruntime, suika, activemodel, jekyll.
249
+ - XDG paths + `KOTOSHU_*` envs as documented configuration surface.
250
+ - CLI exit codes 0/1/2/3 and the JSON/SARIF output shapes.
251
+ - `kotoshu.resources/v1` and the `/v1` HTTP prefix.
@@ -0,0 +1,137 @@
1
+ # Kotoshu Stability Policy — 1.0 draft
2
+
3
+ Status: DRAFT (plan 109). Takes effect the day 1.0 ships; until then the
4
+ usual 0.x rules apply (anything may change). The surface this policy
5
+ protects is itemized in `docs/STABILITY-1.0-AUDIT.md`.
6
+
7
+ ## Principles
8
+
9
+ 1. Semver, per delivery vehicle, cut together on one release train.
10
+ 2. Additive changes only, within a versioned boundary (`/v1`,
11
+ `kotoshu.resources/v1`, the four-key suggestion row).
12
+ 3. Behavior is frozen where the conformance vectors freeze it — the
13
+ vectors ARE the behavioral contract.
14
+ 4. No removal without a deprecation window (below).
15
+ 5. The strict two-stage resource model is permanent: `setup` is explicit,
16
+ the hot path never downloads. No feature may weaken it.
17
+
18
+ ## Per-surface commitments
19
+
20
+ ### Ruby gem (`kotoshu`)
21
+
22
+ - **Patch** (1.0.x): bug fixes only. No public API, CLI, ENV, output, or
23
+ conformance-vector change. `conformance/vectors.jsonl` must replay
24
+ byte-identically (`lib/kotoshu/conformance_runner.rb:72`).
25
+ - **Minor** (1.x): additive only — new methods, new config keys (must
26
+ default to current behavior), new CLI flags with defaults, new
27
+ exception subclasses of existing roots, new suggestion strategies.
28
+ Frozen in place: existing method signatures, the exception class list
29
+ at `lib/kotoshu/core/exceptions.rb`, `SCHEMA` key semantics
30
+ (`lib/kotoshu/configuration.rb:38`), exit codes 0/1/2/3
31
+ (`lib/kotoshu/cli.rb:92-97`), and the JSON/SARIF output shapes
32
+ (`lib/kotoshu/cli.rb:624-651,686-715`) — additions to JSON output are
33
+ allowed (consumers must ignore unknown keys), renames and removals are
34
+ not.
35
+ - **Major** (2.0): anything else, including conformance-vector re-exports
36
+ that change expected results.
37
+ - Ruby floor moves only at a major (`kotoshu.gemspec:17`).
38
+ - Requirements and dependency floors are owner decisions, never inferred
39
+ from code findings.
40
+
41
+ ### wasm package (`@kotoshu/wasm` / kotoshu-rs)
42
+
43
+ - Independent version line (its `package.json` governs, not the crate
44
+ constant — `kotoshu-rs/kotoshu/src/ffi/wasm/mod.rs:149-155`), released
45
+ on the same train as the gem.
46
+ - The JS surface is the module doc contract:
47
+ `KotoshuWasm` (constructor / `correct` / `suggest` / `VERSION`),
48
+ `loadModel`, `rerank`, `semanticSuggest`, `loadLid`, `detectLanguage`,
49
+ and the `KotoshuModel` / `KotoshuLid` handles
50
+ (`kotoshu-rs/kotoshu/src/ffi/wasm/mod.rs:19-46`).
51
+ - The suggestion row is frozen at exactly `word`, `distance`,
52
+ `confidence`, `source` — the conformance `SUGGESTION_KEYS`
53
+ (`lib/kotoshu/conformance_exporter.rb:80`). Additive optional
54
+ parameters are allowed (the `bucketsBytes` third argument of
55
+ `loadModel` is the precedent, mod.rs:215-219); changing existing
56
+ parameters or return shapes is a major.
57
+ - `KotoshuWasm.VERSION` tracks the crate version; it is informational,
58
+ not the package version.
59
+
60
+ ### HTTP server (kotoshu-server)
61
+
62
+ - `openapi.yaml` is the source of truth; `lib/kotoshu/server/app.rb`
63
+ must never diverge from it (today they match route for route).
64
+ - The `/v1` prefix IS the contract version. Within `/v1`: new endpoints,
65
+ new optional request fields, and new optional response fields are
66
+ minor. Changing a field type, removing a field, altering status-code
67
+ semantics (400/422/503), or mutating a frozen schema is a major —
68
+ breaking changes ship as `/v2`, with `/v1` kept for the deprecation
69
+ window.
70
+ - Error bodies keep the `{error, message, hint}` shape
71
+ (`kotoshu-server/openapi.yaml:246-251`).
72
+
73
+ ### GitHub action (action-kotoshu)
74
+
75
+ - Inputs are additive-only and every new input defaults to the previous
76
+ behavior (`action-kotoshu/action.yml:8-59`). Outputs
77
+ (`sarif-path`, `error-count`) are frozen.
78
+ - Bumping the gem floor (`version` input minimum, action.yml:98-105) or
79
+ dropping an input is a new action major. Exit-code mapping 0/1/2/3 is
80
+ part of the contract (action.yml:178-189).
81
+
82
+ ### Resource registry (models-fasttext-onnx)
83
+
84
+ - The spec string `kotoshu.resources/v1` is frozen
85
+ (`schemas/registry.schema.json:15-17`). Any schema change to the
86
+ registry shape means `kotoshu.resources/v2`, consumed side by side.
87
+ - Resource ids are permanent and never reused; artifact content changes
88
+ bump `release_tag` and `registry_version`, never the id (schema
89
+ description, registry.schema.json:5).
90
+ - `min_engine_version` is the compatibility lever for old gem versions
91
+ reading a newer registry — raising it requires a coordinated release
92
+ note.
93
+
94
+ ### Conformance vectors (the behavioral contract)
95
+
96
+ - `conformance/vectors.jsonl` (2630 rows) freezes what the engines
97
+ ACTUALLY return over the fixture corpora — Ruby and native must both
98
+ replay it identically (`lib/kotoshu/conformance_runner.rb:97-112`).
99
+ - A patch release MUST NOT change any vector. A minor may add vectors
100
+ (new corpora/words) but not change existing expectations. Changed
101
+ expectations = changed user-visible suggestions = major, with a
102
+ CHANGELOG entry listing affected inputs.
103
+ - Re-export procedure stays as documented: deterministic export
104
+ (`lib/kotoshu/conformance_exporter.rb:44-53`), commit to this repo,
105
+ kotoshu-rs syncs from it (conformance_exporter.rb:54-60).
106
+
107
+ ## Deprecation window
108
+
109
+ Proposal: **two minor releases** (the standard the ecosystem already
110
+ drifted toward — `fetch` and `dictionaries_url` both wait indefinitely;
111
+ 1.0 is the moment to make it finite).
112
+
113
+ - Ruby API: YARD `@deprecated` tag + a `Kernel#warn` (once per process)
114
+ from the moment of deprecation; removal at the second subsequent
115
+ minor, or at the next major, whichever comes first.
116
+ - Constants/autoloads (e.g. the alias constants at `lib/kotoshu.rb:80-85`):
117
+ same window; `autoload` removal is the deletion.
118
+ - CLI: the command/flag stays functional but hidden from help, printing a
119
+ stderr deprecation notice with the replacement; removal after two
120
+ minors. Exit codes do not change during the window.
121
+ - ENV/config keys (`KOTOSHU_*`): warn on read for two minors, then remove.
122
+ ENV removals are called out in the changelog header — they break CI
123
+ setups silently otherwise.
124
+ - HTTP: a deprecated field/endpoint returns `Deprecation`/`Sunset` style
125
+ headers during the window; `/v1` keeps serving it until `/v2` has been
126
+ stable for one minor.
127
+ - Exception: anything already deprecated throughout late 0.x (the `fetch`
128
+ alias, `dictionaries_url`/`models_url`) may be removed AT 1.0 without a
129
+ new window — the 0.x line was the window.
130
+
131
+ ## The 1.0 gate
132
+
133
+ 1.0 ships when the audit checklist in `docs/STABILITY-1.0-AUDIT.md` is
134
+ cleared: deprecation dispositions landed, the release train (gem + wasm +
135
+ server) coordinated, conformance compare green on both engines, and this
136
+ policy approved by the owner. Version numbers, floors, and the cut date
137
+ are owner decisions — this document proposes, the owner disposes.
@@ -422,7 +422,7 @@ FastText models are downloaded from Facebook's public CDN:
422
422
 
423
423
  ### Adding CLI Commands
424
424
 
425
- To add CLI commands for a new cache type, edit `lib/kotoshu/commands/cache_command.rb`:
425
+ To add CLI commands for a new cache type, edit `lib/kotoshu/cli/cache_command.rb`:
426
426
 
427
427
  ```ruby
428
428
  desc 'download TYPE RESOURCE', 'Download a resource from GitHub'