kotoshu 0.7.0 → 0.8.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (107) hide show
  1. checksums.yaml +4 -4
  2. data/.pre-commit-hooks.yaml +18 -0
  3. data/.yardopts +8 -0
  4. data/CHANGELOG.md +109 -0
  5. data/CLAUDE.md +5 -22
  6. data/README.adoc +90 -0
  7. data/TODO.impl/00-vision.md +51 -0
  8. data/TODO.impl/77-model-coverage-expansion.md +96 -0
  9. data/TODO.impl/78-in-browser-playground.md +69 -0
  10. data/TODO.impl/79-news-and-releases.md +60 -0
  11. data/TODO.impl/80-ecosystem-docs.md +56 -0
  12. data/TODO.impl/81-editor-reach.md +68 -0
  13. data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
  14. data/TODO.impl/83-models-batch-2.md +56 -0
  15. data/TODO.impl/84-language-modules-wave2.md +75 -0
  16. data/TODO.impl/85-wasm-model-api.md +48 -0
  17. data/TODO.impl/86-site-truth-wave2.md +45 -0
  18. data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
  19. data/TODO.impl/88-cli-directory-mode.md +54 -0
  20. data/TODO.impl/89-framework-integrations-exec.md +59 -0
  21. data/TODO.impl/90-gem-correctness-residues.md +44 -0
  22. data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
  23. data/TODO.impl/92-tier-mirror-cors.md +88 -0
  24. data/TODO.impl/93-python-wheels.md +32 -0
  25. data/TODO.impl/94-site-integrations-docs.md +35 -0
  26. data/TODO.impl/95-editor-action-polish.md +27 -0
  27. data/TODO.impl/96-server-semantic-models.md +50 -0
  28. data/TODO.impl/97-yard-execution.md +73 -0
  29. data/lib/kotoshu/analyzers/semantic_analyzer.rb +1 -1
  30. data/lib/kotoshu/baseline/application.rb +18 -0
  31. data/lib/kotoshu/baseline/entry.rb +33 -0
  32. data/lib/kotoshu/baseline/store.rb +152 -0
  33. data/lib/kotoshu/baseline.rb +20 -0
  34. data/lib/kotoshu/cache/frequency_cache.rb +2 -2
  35. data/lib/kotoshu/cache/language_cache.rb +33 -11
  36. data/lib/kotoshu/cache/model_cache.rb +2 -2
  37. data/lib/kotoshu/cli/baseline_command.rb +81 -0
  38. data/lib/kotoshu/cli/completions_command.rb +1 -1
  39. data/lib/kotoshu/cli/directory_check.rb +141 -0
  40. data/lib/kotoshu/cli/directory_walker.rb +132 -0
  41. data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
  42. data/lib/kotoshu/cli.rb +304 -59
  43. data/lib/kotoshu/configuration.rb +8 -2
  44. data/lib/kotoshu/core/models/result/document_result.rb +13 -3
  45. data/lib/kotoshu/core/models/result/word_result.rb +19 -1
  46. data/lib/kotoshu/data/common_words_loader.rb +0 -1
  47. data/lib/kotoshu/dictionary/cspell.rb +3 -8
  48. data/lib/kotoshu/documents/document.rb +12 -2
  49. data/lib/kotoshu/documents/plain_text_document.rb +5 -2
  50. data/lib/kotoshu/documents/source_position.rb +8 -0
  51. data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
  52. data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
  53. data/lib/kotoshu/documents/suppressions.rb +44 -0
  54. data/lib/kotoshu/documents.rb +1 -0
  55. data/lib/kotoshu/embeddings/embedding_pipeline.rb +3 -5
  56. data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +2 -2
  57. data/lib/kotoshu/jekyll.rb +98 -0
  58. data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
  59. data/lib/kotoshu/keyboard/layouts/latin.rb +220 -0
  60. data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
  61. data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
  62. data/lib/kotoshu/keyboard/registry.rb +20 -0
  63. data/lib/kotoshu/keyboard.rb +8 -0
  64. data/lib/kotoshu/language/normalizer/greek.rb +63 -0
  65. data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
  66. data/lib/kotoshu/language/tokenizer/base.rb +17 -0
  67. data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
  68. data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
  69. data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +12 -0
  70. data/lib/kotoshu/language.rb +4 -0
  71. data/lib/kotoshu/languages/ca/language.rb +22 -0
  72. data/lib/kotoshu/languages/cs/language.rb +22 -0
  73. data/lib/kotoshu/languages/da/language.rb +22 -0
  74. data/lib/kotoshu/languages/el/language.rb +50 -0
  75. data/lib/kotoshu/languages/hu/language.rb +22 -0
  76. data/lib/kotoshu/languages/it/language.rb +22 -0
  77. data/lib/kotoshu/languages/latin_base.rb +53 -0
  78. data/lib/kotoshu/languages/nl/language.rb +22 -0
  79. data/lib/kotoshu/languages/pl/language.rb +22 -0
  80. data/lib/kotoshu/languages/ro/language.rb +22 -0
  81. data/lib/kotoshu/languages/sv/language.rb +22 -0
  82. data/lib/kotoshu/languages/tr/language.rb +36 -0
  83. data/lib/kotoshu/languages/uk/language.rb +38 -0
  84. data/lib/kotoshu/languages/vi/language.rb +22 -0
  85. data/lib/kotoshu/languages.rb +18 -0
  86. data/lib/kotoshu/metrics_collector.rb +2 -2
  87. data/lib/kotoshu/models/nearest_neighbor.rb +11 -3
  88. data/lib/kotoshu/paths.rb +26 -0
  89. data/lib/kotoshu/readers/aff_data.rb +7 -3
  90. data/lib/kotoshu/readers/aff_reader.rb +39 -31
  91. data/lib/kotoshu/readers/lookup_builder.rb +1 -1
  92. data/lib/kotoshu/resource_bundle.rb +32 -7
  93. data/lib/kotoshu/resource_manager.rb +137 -1
  94. data/lib/kotoshu/rspec.rb +163 -0
  95. data/lib/kotoshu/source_registry.rb +5 -0
  96. data/lib/kotoshu/spellchecker.rb +64 -22
  97. data/lib/kotoshu/suggestions/generator.rb +4 -5
  98. data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +2 -2
  99. data/lib/kotoshu/tasks/check_task.rb +98 -0
  100. data/lib/kotoshu/tasks.rb +11 -0
  101. data/lib/kotoshu/validators/spelling_validator.rb +101 -0
  102. data/lib/kotoshu/validators.rb +9 -0
  103. data/lib/kotoshu/version.rb +1 -1
  104. data/lib/kotoshu.rb +23 -9
  105. data/scripts/extract_eval_grids.rb +34 -0
  106. data/sig/kotoshu.rbs +331 -1
  107. metadata +69 -3
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 48277511d3fd0b5244b98f54a43c1867991fd943092c9b78f657885addfa940e
4
- data.tar.gz: 0434b01a14db2d334fceb53ced9003ac1378abf75db4386aa5397dfc5d853367
3
+ metadata.gz: f90290a36de788d46588e6883b0619f9fb1dff8e841f629fd021c07e156256ae
4
+ data.tar.gz: 113cbf58fb1b9c297cfb8551a9076875f4882be9e7b17bbebe330ecc53aac94a
5
5
  SHA512:
6
- metadata.gz: adb238648d03837cb4b0fc326c4038d3a40bb5cea968f6ffc49bd2a4bc2ef2614a42a1b79f4cd69e5efc279c0730878aa64885a2384f5ab9cca61c8ade7f096e
7
- data.tar.gz: 3c1f20fd74572d1ae069767c78faad2fb37caffa3a051d8f6ca2355816f3e6d96fee23de275166b8307cd2622ac80d6982d4aaab35db2617c675e4766d705514
6
+ metadata.gz: 1520bf0ce2667b31872c137b55099834a93954f821c93903b193306d537703173ad37e84446dfd41e082e25db50d4a54c694d1623b43875fd5b61294b4cf67f4
7
+ data.tar.gz: f22ca5507daa129090a2d292d4e526200f197f32520b864c67e3ffbab82862a431d155303613e670aab97e48d213bad32ad0ba4b4cdf1178a8de1dff940a1cbd
@@ -0,0 +1,18 @@
1
+ # pre-commit hook definitions (https://pre-commit.com).
2
+ #
3
+ # Honest requirements: this hook is `language: system` — it runs the
4
+ # `kotoshu` executable from PATH, so the machine needs a Ruby
5
+ # environment with the gem installed:
6
+ #
7
+ # gem install kotoshu
8
+ # kotoshu setup en # or the languages you write in
9
+ #
10
+ # pre-commit passes the matched filenames to the entry; `kotoshu check`
11
+ # exits 1 when a file has unsuppressed misspellings. The WebAssembly
12
+ # client (in development) is the future zero-Ruby path.
13
+ - id: kotoshu
14
+ name: kotoshu semantic spell check
15
+ description: Semantic spell checking for prose and documentation
16
+ entry: kotoshu check
17
+ language: system
18
+ files: \.(txt|text|md|markdown|mdown|mkd|adoc|asciidoc|ad|rst|rest|creole|rdoc|org|pod|textile|wiki|mail|article|log|notes)$
data/.yardopts ADDED
@@ -0,0 +1,8 @@
1
+ --title "Kotoshu API Reference"
2
+ --markup markdown
3
+ --readme README.adoc
4
+ --exclude ^spec/
5
+ --exclude ^tasks/
6
+ --exclude ^ext/
7
+ --exclude ^conformance/
8
+ --hide-void-return
data/CHANGELOG.md CHANGED
@@ -5,6 +5,115 @@ All notable changes to this project will be documented in this file.
5
5
  The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
6
6
  and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7
7
 
8
+ ## [Unreleased]
9
+
10
+ ### Fixed
11
+ - Remote `kotoshu setup` for staged languages: the dictionaries repo ships only
12
+ `en` under the `{lang}/spelling/` layout while staged languages sit flat at
13
+ `{lang}/index.*`; downloads now try the sublayout first and fall back to the
14
+ flat layout (verified end-to-end for `it`).
15
+ - `require "kotoshu/tasks"` / `require "kotoshu/jekyll"` standalone now load
16
+ without needing `require "kotoshu"` first.
17
+ - Baseline paths compare canonically: entries recorded as `docs/a.md` match a
18
+ directory-walk check over `./docs/a.md`, and recorded entries are stored
19
+ canonical.
20
+ ### Fixed
21
+ - **Unicode word detection** (plan 91 Track A) - `Spellchecker` word
22
+ extraction accepted only ASCII letters, so Greek and Ukrainian users
23
+ could not check any text through the CLI. Extraction now follows the
24
+ configured language's tokenizer: Greek words for `el`, Cyrillic for
25
+ `uk` (in-word apostrophe included, for names like Мар'яна), and
26
+ Latin-script letters including capitals such as Å and Ä for Latin
27
+ languages. Wrong-script words stay uncheckable; languages without a
28
+ script tokenizer (en, de, ...) keep the historical ASCII behavior;
29
+ the frozen conformance vectors are unchanged.
30
+ - **Swedish aff parsing** (plan 91 Track B) - `Readers::AffReader`
31
+ raised `RegexpError: unmatched close parenthesis` on the Swedish
32
+ dictionary (dictionaries pin 1829a3e). `COMPOUNDRULE )k` uses `)`
33
+ itself as a flag character, but the rule compiler escaped it only in
34
+ some positions. Every flag character is now escaped at compile time,
35
+ so the dictionary loads and its compound rules match.
36
+ ### Added
37
+ - **Framework integrations** (plan 89) - four opt-in integration
38
+ layers, all inside the gem with NO new runtime dependencies
39
+ (soft-dependency pattern, like onnxruntime/suika):
40
+ - `Kotoshu::Validators::SpellingValidator`, an ActiveModel
41
+ EachValidator (`validates :body, spelling: true` after the
42
+ documented one-line alias, or `validates_with` directly): one
43
+ validation error per misspelling with the top suggestion in the
44
+ message; `language:` and `personal_words:` options. Loads the
45
+ real ActiveModel standalone - Rails is not required.
46
+ - RSpec matchers (`require "kotoshu/rspec"` +
47
+ `include Kotoshu::Rspec::Matchers`):
48
+ `expect_words("helo").to all_be_spelled_correctly`, and
49
+ `expect(text) / expect_document(path).to be_spelled_correctly`
50
+ for words or whole documents, with failure messages that list
51
+ each misspelling and its suggestions.
52
+ - Rake task (`require "kotoshu/tasks"`): `rake kotoshu:check`
53
+ over the repository text files using the plan-88 file selection
54
+ (known extensions, .gitignore/.ignore, default skips);
55
+ `Kotoshu::Tasks::CheckTask` exposes files/language/fail_on_error
56
+ /baseline configuration.
57
+ - Jekyll generator (`Kotoshu::Jekyll::Generator`, `safe`):
58
+ checks posts and drafts and fails the build on new spelling
59
+ errors; a `.kotoshu-baseline.json` in the site source is
60
+ respected so baselined errors do not block builds.
61
+ - **CLI directory mode** (plan 88) - `kotoshu check DIR [DIR ...]`
62
+ walks trees and checks every file with a known text extension
63
+ (md markdown asciidoc adoc txt rst mdx) unless `--include` /
64
+ `--exclude` globs say otherwise. `.gitignore` and `.ignore` files
65
+ are honored through a standard glob subset implemented in Ruby
66
+ (`*`, `?`, `**` as a whole segment, `!` negation with last-match
67
+ wins, trailing `/` for directory-only patterns, anchoring, nested
68
+ scoping; files inside ignored directories cannot be re-included).
69
+ Hidden files and `.git` / `node_modules` / `vendor` / `target` are
70
+ skipped by default. Text output prints one section per file plus a
71
+ summary; JSON and SARIF emit one combined document with per-file
72
+ entries/runs; `--baseline` applies per file as shipped; exit codes
73
+ are unchanged; interactive mode stays file-only with an explicit
74
+ notice for directory targets. Single-file and stdin behavior is
75
+ byte-identical to before.
76
+ - **Wave-1 language modules and keyboard layouts** (plan 84) -
77
+ language modules for the thirteen model-registry newcomers (ca cs
78
+ da el hu it nl pl ro sv tr uk vi): thin compositions over the shared
79
+ Latin tokenizer/normalizer plus registry entries, with real care
80
+ where the script needs it — Turkish dotless-i case folding
81
+ (`İZMİR` -> `izmir`, `ISTANBUL` -> `ıstanbul`), Greek-aware
82
+ normalization (final sigma `ΕΛΛΑΣ` -> `ελλας`, opt-in accent
83
+ stripping) and Greek/Cyrillic script tokenizers that keep
84
+ non-Latin words. Keyboard layouts grow from 5 to 19: Turkish-Q,
85
+ Ukrainian-JCUKEN and Greek-Phonetic national grids mirrored from
86
+ the models repo eval harness (a drift spec asserts the gem grids
87
+ cover the eval grids; `scripts/extract_eval_grids.rb` refreshes the
88
+ committed snapshot), plus one parameterized Latin family
89
+ (`Keyboard::Layouts::Latin` and members) over the qwerty/qwertz
90
+ base grids with real Nordic å/æ/ø and å/ä/ö keys. The pre-existing
91
+ five layouts and ten language modules are unchanged.
92
+ - **Inline ignore directives** (plan 82) - `kotoshu:disable-line`,
93
+ `kotoshu:disable-next-line [WORDS]`, and the nestable
94
+ `kotoshu:disable-file` / `kotoshu:enable-file` block, recognized in
95
+ each format's comment syntax: HTML comments in Markdown, `//` line
96
+ comments in AsciiDoc, bare lines in plain text (trailing directives
97
+ in Markdown/AsciiDoc). One shared scanner in `Documents::Suppressions`
98
+ attaches them to every `Document`, and the `Spellchecker` facade
99
+ filters them out of `check`: suppressed words move to
100
+ `DocumentResult#suppressed_errors` (marked `suppressed: true`,
101
+ `suppressed_by: "inline"`), and `kotoshu check --show-suppressed`
102
+ lists them. Result JSON gains `suppressed` / `suppressed_by` on
103
+ errors and `suppressedCount` / `suppressedErrors` at the top level.
104
+ - **CI baselines** (plan 82) - `kotoshu baseline init FILE ...` writes
105
+ `.kotoshu-baseline.json` (canonical `{version, entries: [{file, line,
106
+ word, count}]}` via lutaml-model; count-based, so it survives
107
+ reformatting); `kotoshu check --baseline FILE` lets covered errors
108
+ pass, fails on new ones, and reports stale entries. SARIF output
109
+ marks baseline-suppressed results with a `suppressions` entry
110
+ (`kind: external`, `justification: "baseline"`), and JSON output
111
+ adds a `baseline` block.
112
+ - **pre-commit hook** (plan 81) - `.pre-commit-hooks.yaml` ships hook
113
+ id `kotoshu` (`language: system`, entry `kotoshu check`, text-format
114
+ file pattern); requires Ruby plus the gem, documented honestly in the
115
+ README.
116
+
8
117
  ## [0.7.0] — 2026-09-05
9
118
 
10
119
  The universal-kotoshu cut (plan 65's roadmap milestone): tiered
data/CLAUDE.md CHANGED
@@ -9,25 +9,8 @@ dictionary/affix backend (Hunspell-style) with ONNX-converted FastText word
9
9
  embeddings for context-aware suggestions. The README.adoc is the authoritative
10
10
  user-facing description; this file is the contributor-facing map.
11
11
 
12
- Runtime deps (`kotoshu.gemspec`): `thor` (CLI), `rubyzip` (archive extraction during setup), `lutaml-model` (all model (de)serialization — see below). **Soft deps** (NOT in gemspec; `gem install kotoshu` works without them): `onnxruntime` (semantic / ONNX inference) and `suika` (Japanese morphological tokenization; its native ext `dartsclone` would otherwise break slim installs). Ruby 3.1+.
13
-
14
- ## Serialization rule (load-bearing)
15
-
16
- All model classes use `lutaml-model` for serialization — declare attributes via `attribute :name, :type` and wire names via `mapping do ... end`. **Never hand-roll `to_h` / `from_h` / `to_json` / `from_json` / `serialize` / `deserialize`** on a model. Renames go in `mapping`, not in method bodies. This is a global rule (~/.claude/CLAUDE.md) — existing hand-rolled serialization is debt to migrate, not a pattern to extend.
17
-
18
- ## Workspace: five repos under `kotoshu/`
19
-
20
- The parent directory `/Users/mulgogi/src/kotoshu/` is a multi-repo workspace — the gem is one piece. The other four are **content repos** fetched on demand by the cache layer, never vendored:
21
-
22
- | Sibling dir | Repo | Role |
23
- |---|---|---|
24
- | `kotoshu/` (this one) | `kotoshu/kotoshu` | The gem — Ruby API + CLI + analysis pipeline |
25
- | `dictionaries/` | `kotoshu/dictionaries` | Hunspell `.aff`/`.dic` for ~98 languages (download source for `LanguageCache`) |
26
- | `frequency-list-kelly/` | `kotoshu/frequency-list-kelly` | Kelly Project frequency tiers (download source for `FrequencyCache`) |
27
- | `models-fasttext-onnx/` | `kotoshu/models-fasttext-onnx` | FastText `.vec` → ONNX (download source for `ModelCache`) |
28
- | `kotoshu.github.io/` | `kotoshu/kotoshu.github.io` | Marketing/docs site |
29
-
30
- Each content repo has its own `TODO.impl/`. The vision and cross-repo plan index live in `TODO.impl/00-vision.md`.
12
+ Key dependencies (`kotoshu.gemspec`): `thor` (CLI), `suika` (tokenizer),
13
+ `onnxruntime` (semantic inference). Ruby 3.1+.
31
14
 
32
15
  ## Development Commands
33
16
 
@@ -122,7 +105,7 @@ Three caches under `~/.cache/kotoshu/` (see `CACHE_ARCHITECTURE.md` for detail,
122
105
  |---|---|---|---|
123
106
  | `Cache::LanguageCache` | `~/.cache/kotoshu/languages/{code}/spelling/` | `github.com/kotoshu/dictionaries` | 7 days |
124
107
  | `Cache::FrequencyCache` | `~/.cache/kotoshu/frequency-lists/{code}/` | `github.com/kotoshu/frequency-list-kelly` | 7 days |
125
- | `Cache::ModelCache` | `~/.cache/kotoshu/models/{code}/...` | `github.com/kotoshu/models-fasttext-onnx` (FastText `.vec` → ONNX converted upstream; tiers: full/fluency/mini, registry-resolved) | 30 days |
108
+ | `Cache::ModelCache` | `~/.cache/kotoshu/models/{code}/...` | `github.com/kotoshu/models-fasttext-onnx` (FastText `.vec` → ONNX converted upstream) | 30 days |
126
109
 
127
110
  `FrequencyCache` feeds `frequency_bonus` in `Suggestions::Strategies::EditDistanceStrategy` — high-frequency words get a ranking boost. The `kotoshu cache` subcommand exposes list/status/download/info/purge/clean operations.
128
111
 
@@ -187,7 +170,7 @@ When implementing features, study these alongside Kotoshu:
187
170
 
188
171
  ## Other Notes
189
172
 
190
- - License is **BSD-2-Clause** (per `kotoshu.gemspec` and `LICENSE`).
173
+ - License is **BSD-2-Clause** (not MIT — the README's "License" section is wrong).
191
174
  - RBS signatures live in `sig/kotoshu.rbs` (the `sig/kotoshu/` subdirectory is empty). Update signatures when changing public APIs.
192
- - `scripts/` contains one-off utilities (FastText→ONNX conversion in Python, Kelly frequency parsing, diagnostics). `examples/` has numbered walkthrough scripts (`01_*.rb` … `14_*.rb`) plus per-language demo scripts and Spylls-comparison scripts.
175
+ - `scripts/` contains one-off utilities (FastText→ONNX conversion in Python, Kelly frequency parsing, diagnostics). `examples/` has numbered walkthrough scripts (`01_*.rb` … `07_*.rb`).
193
176
  - Design history and superseded planning docs live in `docs/` (`architecture.md`, `cache-architecture.md`, `performance.md`, `plugins.md`, `getting-started.md`, plus integrated planning docs like `KOTOSHU_SOLIDIFICATION_PLAN.md`, `ARCHITECTURE_IMPROVEMENTS.md`, `TDD_ITERATION_STRATEGY.md`). Treat them as historical context, verify against current code before relying on them. `TODO.impl/` is the current source of truth for execution plans.
data/README.adoc CHANGED
@@ -435,6 +435,79 @@ echo $? # 0 if no errors, 1 if errors found
435
435
  ----
436
436
  ====
437
437
 
438
+ [[ignoring-words]]
439
+ == Ignoring Words Inline
440
+ Words that are intentionally "wrong" - product names, code identifiers
441
+ quoted in prose, dialect - can be suppressed in place instead of being
442
+ added to the personal dictionary:
443
+ ----
444
+ kotoshu:disable-line suppress everything on this line
445
+ kotoshu:disable-next-line suppress the following line
446
+ kotoshu:disable-next-line foo bar suppress only these words
447
+ kotoshu:disable-file / enable-file block suppression (nestable)
448
+ ----
449
+ Each format recognizes the directive in its own comment syntax, and the
450
+ directive line itself is never spellchecked:
451
+ [options="header"]
452
+ |===
453
+ |Format |Form |Trailing directive
454
+
455
+ |Plain text
456
+ |a bare line: `kotoshu:disable-line`
457
+ |not supported
458
+
459
+ |Markdown
460
+ |`<!-- kotoshu:disable-next-line -->`
461
+ |supported
462
+
463
+ |AsciiDoc
464
+ |`// kotoshu:disable-next-line`
465
+ |supported
466
+
467
+ |===
468
+
469
+ The Ruby API picks this up automatically:
470
+ `Kotoshu.spellchecker_for("en").check(text)` moves suppressed words to
471
+ `result.suppressed_errors` (each marked `suppressed: true` and
472
+ `suppressed_by: "inline"`) so only real errors stay in `result.errors`.
473
+ Pass `--show-suppressed` to `kotoshu check` to list them.
474
+
475
+ [[ci-baselines]]
476
+ == CI Baselines
477
+ Freeze existing spelling debt so CI fails only on NEW errors:
478
+ [source,bash]
479
+ ----
480
+ # Record current findings: one entry per file+word with a count,
481
+ # stable across reformatting
482
+ kotoshu baseline init README.md docs/*.adoc
483
+
484
+ # CI: errors covered by the baseline pass, new errors fail
485
+ kotoshu check README.md --baseline .kotoshu-baseline.json
486
+ ----
487
+ A summary line reports baseline entries that have gone stale (the error
488
+ no longer exists), so the debt shrinks visibly; refresh by re-running
489
+ `kotoshu baseline init`. JSON output adds `suppressedCount` /
490
+ `suppressedErrors` plus a `baseline` block, and SARIF marks
491
+ baseline-suppressed results with a `suppressions` entry
492
+ (`justification: "baseline"`).
493
+
494
+ [[pre-commit]]
495
+ == Git pre-commit Hook
496
+ The gem repository ships a https://pre-commit.com[pre-commit] hook
497
+ definition (`.pre-commit-hooks.yaml`):
498
+ [source,yaml]
499
+ ----
500
+ repos:
501
+ - repo: https://github.com/kotoshu/kotoshu
502
+ rev: v0.7.0 # pin to a released tag
503
+ hooks:
504
+ - id: kotoshu
505
+ ----
506
+ NOTE: The hook uses `language: system`: it runs the `kotoshu` executable
507
+ from PATH, so it honestly requires a Ruby environment with the gem
508
+ installed (`gem install kotoshu`, then `kotoshu setup en`). A
509
+ WebAssembly client (in development) is the future zero-Ruby path.
510
+
438
511
  [[document-formats]]
439
512
  == Document Formats
440
513
 
@@ -1173,6 +1246,23 @@ Plugin gems should publish to RubyGems with a runtime dependency on
1173
1246
  `kotoshu >= 0.5.0`. Format-specific parsers are the plugin's
1174
1247
  responsibility — kotoshu never owns document parsing.
1175
1248
 
1249
+ == API Documentation
1250
+
1251
+ The YARD API reference for the Ruby API builds in CI on every change
1252
+ to `lib/` (the `docs` workflow); the generated HTML is attached to each
1253
+ run as the `yard-docs` artifact. A hosted copy is pending — it will
1254
+ live either on this repository's GitHub Pages or as an API subpath on
1255
+ https://kotoshu.github.io[kotoshu.github.io].
1256
+
1257
+ To build and browse it locally:
1258
+
1259
+ [source,bash]
1260
+ ----
1261
+ bundle install
1262
+ bundle exec yard doc
1263
+ open doc/index.html
1264
+ ----
1265
+
1176
1266
  == License
1177
1267
 
1178
1268
  BSD 2-Clause — see the link:LICENSE[LICENSE] file for details.
@@ -122,6 +122,57 @@ The remaining SOTA items are planned as
122
122
  [75 (LLM-as-evaluator)](75-llm-evaluator.md), and
123
123
  [76 (optional external LLM backend, gated on demand)](76-optional-llm-backend.md).
124
124
 
125
+ Plans 77–82 are the **adoption wave** (owner ask 2026-09-05: are the
126
+ models comprehensive, how do we cover more places, docs + news):
127
+ closing the verified user gaps after all channels went live.
128
+
129
+ | # | Plan | Gap it closes |
130
+ |---|---|---|
131
+ | 77 | [Model coverage expansion](77-model-coverage-expansion.md) | Semantic models cover 9 languages vs 98 dictionaries — expand to 25+ |
132
+ | 78 | [In-browser playground](78-in-browser-playground.md) | /playground needs a running server; @kotoshu/wasm enables zero-install |
133
+ | 79 | [News + releases](79-news-and-releases.md) | No news surface on the site; seven registries shipped invisibly |
134
+ | 80 | [Ecosystem docs](80-ecosystem-docs.md) | Docs are gem-only; pip/npm/cargo/go/HTTP/LSP/Action users have no path in |
135
+ | 81 | [Editor reach](81-editor-reach.md) | kotoshu-lsp shipped but no VS Code package; no pre-commit hook (supersedes 61 remainder) |
136
+ | 82 | [Document ignores + baselines](82-doc-formats-and-ignores.md) | No inline suppressions; no CI baseline freeze |
137
+
138
+ Plans 83–89 are **wave 2** (2026-09-05, owner re-ask after the wave-1
139
+ ship): driving models toward full intersection coverage, making the
140
+ 22 model languages full-feature, semantic reranking in the browser,
141
+ and the site/docs truth that follows.
142
+
143
+ | # | Plan | Gap it closes |
144
+ |---|---|---|
145
+ | 83 | [Models batch 2](83-models-batch-2.md) | ar/fa/he have modules but no models; ~50 more intersection languages; fi/id dictionary sourcing |
146
+ | 84 | [Language modules + keyboards wave 2](84-language-modules-wave2.md) | 13 model languages lack gem modules; 11 keyboard layouts missing |
147
+ | 85 | [Wasm model API](85-wasm-model-api.md) | Browser semantic reranking — no model API on the wasm surface, CORS distribution gap |
148
+ | 86 | [Site truth wave 2](86-site-truth-wave2.md) | Ignores/baselines undocumented; no Windows page; wrong personal.dic path in CLI docs; playground breadth |
149
+ | 87 | [LSP personal dictionary](87-lsp-personal-dictionary.md) | Server never reads personal.dic — add-to-dictionary appears broken |
150
+ | 88 | [CLI directory mode](88-cli-directory-mode.md) | No tree checking — the actual CI workflow |
151
+ | 89 | [Framework integrations](89-framework-integrations-exec.md) | Executes plan 62: Rails validator, RSpec matcher, Rake/Jekyll tasks |
152
+
153
+ Plans 91–95 are **wave 3** (2026-09-05 late): the correctness core the
154
+ site flip exposed, the browser-models distribution fix, Python wheels,
155
+ the docs for wave-2 features, and editor/action truth.
156
+
157
+ | # | Plan | Gap it closes |
158
+ |---|---|---|
159
+ | 91 | [Unicode + sv aff](91-unicode-and-sv-aff.md) | Executes plan 90: ASCII-only word detection hides non-Latin scripts; AffReader crashes on Swedish |
160
+ | 92 | [Tier mirror CORS](92-tier-mirror-cors.md) | Media host already sends ACAO — mirror the tiers, finish the browser-models path |
161
+ | 93 | [Python wheels](93-python-wheels.md) | kotoshu-native is macOS-only on PyPI; build the platform matrix + keyless publish |
162
+ | 94 | [Site integrations docs](94-site-integrations-docs.md) | Rails/RSpec/Rake/Jekyll + directory mode shipped undocumented; action page stale |
163
+ | 95 | [Editor + action polish](95-editor-action-polish.md) | Extension delegates the now-existing server command; action gains baseline + 19 languages |
164
+
165
+ Plans 96–97 are **wave 4** (2026-09-06): the audit found the wave-1–3
166
+ surface complete except two honest items — the HTTP server never
167
+ loads models, and plan 59's YARD reference is still unpublished. (The
168
+ Rust engine was audited too: its AST-based aff parser is immune to
169
+ the Swedish COMPOUNDRULE bug by design.)
170
+
171
+ | # | Plan | Gap it closes |
172
+ |---|---|---|
173
+ | 96 | [Server semantic models](96-server-semantic-models.md) | /v1/check is dictionary-only — SDK users get no reranking through HTTP |
174
+ | 97 | [YARD execution](97-yard-execution.md) | Executes plan 59: the Ruby API reference still does not exist |
175
+
125
176
  Dependency order across the ecosystem track:
126
177
 
127
178
  ```
@@ -0,0 +1,96 @@
1
+ # 77 — Model coverage expansion: 9 → 25+ languages
2
+
3
+ > Asked by the owner 2026-09-05: "Are our models fully complete and
4
+ > comprehensive with no user gaps?" They are not — this plan closes the gap.
5
+
6
+ ## Context
7
+
8
+ The registry (models-fasttext-onnx `registry.json`, v1.0.1) carries
9
+ semantic models for exactly **9 languages** (de en es fr ja ko pt ru zh)
10
+ × 3 tiers = 27 entries. Meanwhile:
11
+
12
+ - `dictionaries/` repo: **98** languages with Hunspell `.aff`/`.dic`
13
+ - FastText CC vectors exist for **157+** languages
14
+ - The site language matrix and gem language registry advertise the
15
+ dictionary breadth, but semantic reranking silently stops at 9
16
+
17
+ A user who spells in Italian, Polish, Turkish, or Dutch gets the
18
+ traditional path only. That is the largest user gap in the ecosystem.
19
+
20
+ ## What is NOT changing
21
+
22
+ - The tier recipe (full 300d / fluency int8 top-50k / mini int8
23
+ top-10k) — settled by plan 67's measured gates; no SVD, no Model2Vec.
24
+ - The eval gates from plan 67's gate table: fluency
25
+ `rank_corr ≥ 0.97 / top1 ≥ 0.95`, mini `rank_corr ≥ 0.90 /
26
+ top1 ≥ 0.85` (keyboard-aware eval from plan 69). A language that
27
+ fails gates ships **no** tiers rather than bad tiers.
28
+
29
+ ## Phases
30
+
31
+ ### Phase 1 — candidate selection (deterministic)
32
+
33
+ Candidates = dictionaries repo dirs ∩ FastText `cc.{lang}.300.vec`
34
+ availability ∩ eval-gate pass. Rank by speaker base × dictionary
35
+ quality (dictionary repo manifest). Batch 1 target (16):
36
+
37
+ ```
38
+ it nl pl uk tr cs sv el hu ro da fi no id vi ca
39
+ ```
40
+
41
+ Batch 2 (stretch, if compute allows): ar fa he th vi-tok hi bn ms tl
42
+ sk sl lt lv et is hr sr bg.
43
+
44
+ ### Phase 2 — pipeline run per language
45
+
46
+ Use the existing scripts (see models repo `TODO.impl/01`, `04`, `06`):
47
+ download `cc.{lang}.300.vec.gz` → convert to ONNX full → int8 +
48
+ vocab-cut to fluency/mini → keyboard-aware eval → gates. Record
49
+ per-language eval JSON into `eval/` like the existing nine.
50
+
51
+ ### Phase 3 — registry + release
52
+
53
+ Regenerate `registry.json` (+ manifest, schemas untouched), bump
54
+ registry_version, tag `v1.1.0` (policy from models plan 05: pre-declared
55
+ minor for coverage expansion). Release assets: 3 tiers × new languages,
56
+ mirrored to the media host per plan 07's distribution map.
57
+
58
+ ### Phase 4 — downstream truth updates
59
+
60
+ - Gem `Language` registry: full-feature languages list gains the
61
+ new entries (spelling + model available).
62
+ - Site language matrix + per-language pages: flip `hasModel` for
63
+ the new set; copy updated by the site plan (79/80 reuse).
64
+ - kotoshu-rs / SDKs: no code change (registry-driven).
65
+
66
+ ## Owner gates
67
+
68
+ - None for batch 1 execution (standing directive 2026-09-05).
69
+ - Version numbers: v1.1.0 follows plan 05's pre-declared policy —
70
+ registry major/minor tracks coverage, patch tracks regeneration.
71
+ If the owner objects, yank per policy is NOT possible for release
72
+ assets — ask first if in doubt.
73
+
74
+ ## Success criteria
75
+
76
+ - ≥ 8 new languages pass gates and appear in registry + release.
77
+ - Every shipped tier has eval JSON proving the gates.
78
+ - Registry schema version unchanged; gem + site consume it without
79
+ code changes.
80
+
81
+ ## Status
82
+
83
+ **Implemented (2026-09-05, models PR #13 + tag v1.1.0).** 13 of 16
84
+ batch-one candidates shipped: ca cs da el hu it nl pl ro sv tr uk vi
85
+ — 22 languages x 3 tiers, 66 registry resources, 134 release assets.
86
+ All tiers passed the gates unweakened (fluency rank_corr 0.9999,
87
+ top1 0.9583-1.0000). noise.py grew QWERTY/QWERTZ supplements plus
88
+ curated Turkish-Q / Ukrainian JCUKEN / Greek-phonetic grids; a
89
+ pre-existing release_notes.py crash on null tier mirrors was fixed
90
+ in passing. Released registry byte-identical to committed (local
91
+ rebuild reproduces CI sha256).
92
+
93
+ Dropped from batch 1 (no dictionaries-repo aff/dic): fi, no, id —
94
+ `no` exists only as nb/nn; an owner mapping decision for a v1.2.
95
+ Site truth updated (MODEL_LANGUAGES + news entry, site PR #4); the
96
+ gem needed no change (registry-driven).
@@ -0,0 +1,69 @@
1
+ # 78 — Zero-install wasm playground on the site
2
+
3
+ ## Context
4
+
5
+ The site playground (`/playground`) currently requires a locally
6
+ running `kotoshu-server` — a real barrier for the curious visitor.
7
+ Since `@kotoshu/wasm` 0.1.0 is live on npm (291 KiB gz engine), a
8
+ zero-install in-browser demo is now possible: load the engine from a
9
+ CDN, fetch `.aff`/`.dic` + a mini-tier model, check text live.
10
+
11
+ This is the single highest-leverage marketing surface: every visitor
12
+ can feel the product in five seconds with no install.
13
+
14
+ ## Design
15
+
16
+ - **Route**: keep `/playground` as the home of the in-browser demo;
17
+ the server-based playground moves to `/playground/server` (linked
18
+ as "run it against your own instance").
19
+ - **Engine**: `import { init } from "@kotoshu/wasm"` via a pinned
20
+ CDN URL (esm.run / esm.sh, version-pinned to 0.1.x). Never
21
+ `@latest`.
22
+ - **Dictionaries**: `kotoshu/dictionaries` repo files via jsDelivr
23
+ `/gh/kotoshu/dictionaries@<tag>/<lang>.aff|.dic` — verify the
24
+ tag layout first (dictionaries repo manifest). Files are < 20 MiB
25
+ so jsDelivr gh CDN is legal.
26
+ - **Model**: mini tier from the models registry. Registry URLs point
27
+ at release assets — verify browser CORS on
28
+ `objects.githubusercontent.com` before committing to that source;
29
+ fallback = jsDelivr gh for tag-tree files, else document the
30
+ limitation and ship dictionary-only mode with a note that
31
+ reranking needs the full app.
32
+ - **UX**: language select (defaults to browser language if one of
33
+ the supported set), textarea with misspelling underlines,
34
+ suggestion popover on click, tier badge (mini), first-load size
35
+ readout. Reduced-motion respected; no layout shift.
36
+ - **Aesthetic**: follows the site's dictionary-catalog design
37
+ system (site plans 05/09) — the playground is a "specimen" page,
38
+ not a generic SaaS demo.
39
+
40
+ ## Phases
41
+
42
+ 1. **Fetch spike** (an hour): prove CORS + CDN loads for engine,
43
+ one dictionary, mini model. Decide model source. If models CORS
44
+ fails → dictionary-only v1, models behind the server playground.
45
+ 2. **Component**: `PlaygroundLab.astro` + island
46
+ (`client:load`), module-scope caching of loaded languages,
47
+ Web Worker for the engine to keep typing at 60fps (the engine
48
+ API is sync; worker keeps the main thread free).
49
+ 3. **Wire routes**, update copy on `/` hero to link the demo
50
+ ("try it now — nothing installs").
51
+ 4. **Verify in a browser** before merge: golden path (load en,
52
+ type a typo, get suggestions), CJK (ja), offline (graceful
53
+ error), reduced-motion, mobile width.
54
+
55
+ ## Owner gates
56
+
57
+ None. CDN pinning note: record the chosen CDN + version in this
58
+ file's Status when shipped.
59
+
60
+ ## Status
61
+
62
+ **Implemented (2026-09-05, site PRs #1-#3, live at kotoshu.org/playground).**
63
+ Zero-install wasm demo: @kotoshu/wasm 0.1.0 in a worker, underlines +
64
+ click-to-mend popover, six languages; server playground moved to
65
+ /playground/server. CDN pin: jsDelivr npm raw files @0.1.0 with manual
66
+ instantiate (esm.sh and esm.run both broken for the bundler-target
67
+ package); dictionaries via jsDelivr gh @commit 1829a3e. Models verdict:
68
+ GitHub release assets send no CORS headers, so the demo is honestly
69
+ dictionary-only; the model story lives on the server playground page.
@@ -0,0 +1,60 @@
1
+ # 79 — News + releases section on the site
2
+
3
+ ## Context
4
+
5
+ The site has docs, languages, audiences, install — but no news.
6
+ Releases across seven registries shipped in 2026-09 and a visitor
7
+ cannot tell the project is alive. The owner asked for this
8
+ explicitly ("think also about documentation and our website
9
+ (e.g. docs, news)").
10
+
11
+ ## Design
12
+
13
+ - `/news` — index, reverse-chronological, one entry per release or
14
+ event, each rendered as a dictionary entry (date as the headword's
15
+ etymology line; the entry body is the note).
16
+ - Entry pages `/news/<slug>` for anything longer than a paragraph.
17
+ - **RSS + Atom** at `/news/rss.xml` + `/news/atom.xml` (static).
18
+ - Design follows the site system (eyebrows, `辞書` vertical accent,
19
+ numbered senses for changelog bullets). No blog-engine machinery —
20
+ entries are typed content collections.
21
+
22
+ ## Seed content (real history, verified 2026-09-05)
23
+
24
+ 1. **2026-09-05 — Trusted publishing everywhere** — npm, RubyGems,
25
+ crates.io now publish keyless via OIDC (RFC 3691 on crates.io,
26
+ provenance on npm).
27
+ 2. **2026-09-05 — First releases on all channels** — PyPI
28
+ `kotoshu` + `kotoshu-native` 0.1.0; npm `@kotoshu/wasm` +
29
+ `@kotoshu/client` 0.1.0; crates.io `kotoshu` 0.1.0.
30
+ 3. **2026-09-05 — Ruby gem 0.7.0** — the universal-kotoshu cut:
31
+ model tiers + registry, native extension (dual-backend parity,
32
+ 2630 conformance vectors, 0 divergences), legacy-cache bridge,
33
+ tier default = fluency.
34
+ 4. **2026-09-04 — Engine correctness wave** — compound/suggest
35
+ parity with hunspell semantics (T2), 1315+1315 conformance green
36
+ in Rust and Ruby.
37
+ 5. **2026-09-03 — Models v1.0.x registry + tiers** — 9 languages ×
38
+ 3 tiers, keyboard-aware eval, media-host distribution.
39
+ 6. **Earlier** — kotoshu-lsp 0.1.0, kotoshu-server 0.1.0,
40
+ kotoshu-go v0.1.0, action-kotoshu v1, site launch.
41
+
42
+ Pull dates from the registries/gem CHANGELOG while writing — verify,
43
+ don't guess.
44
+
45
+ ## Phases
46
+
47
+ 1. Content collection schema + seed entries above.
48
+ 2. `/news` index + slugs + feeds; nav + footer links; sitemap.
49
+ 3. Cross-link: gem docs page → news; news → install.
50
+
51
+ ## Owner gates
52
+
53
+ None.
54
+
55
+ ## Status
56
+
57
+ **Implemented (2026-09-05, site PR #1).** /news with 8 entries as
58
+ dictionary entries, RSS 2.0 + Atom 1.0 static feeds with autodiscovery,
59
+ nav/footer/sitemap wired. Dates verified against registry timestamps
60
+ (two plan guesses corrected: PyPI shipped 09-04; channels entry split).
@@ -0,0 +1,56 @@
1
+ # 80 — Ecosystem docs: every channel documented
2
+
3
+ ## Context
4
+
5
+ The site documents the gem well (cli, caching, configuration, api,
6
+ comparison, migration, plugins). But the ecosystem now ships on
7
+ eight channels, and a Python/JS/Go/Rust user arriving at
8
+ kotoshu.org finds no path in. Audience: non-Ruby developers —
9
+ exactly who the access libraries (plan 67) were built for.
10
+
11
+ ## Channels to document (verified live 2026-09-05)
12
+
13
+ | Channel | Package | What a user needs |
14
+ |---|---|---|
15
+ | PyPI | `kotoshu`, `kotoshu-native` | pip install, Python API, maturin wheel note |
16
+ | npm | `@kotoshu/client`, `@kotoshu/wasm` | npm install, JS API, browser/Node |
17
+ | crates.io | `kotoshu` | cargo add, Rust API, KOSH-v1 batch format |
18
+ | Go | `kotoshu-go` | go get, HTTP-mode usage |
19
+ | RubyGems | `kotoshu` | already documented (link) |
20
+ | HTTP | `kotoshu-server` | /v1/check contract, docker run |
21
+ | LSP | `kotoshu-lsp` | gem install + editor config (Neovim lspconfig, generic LSP) |
22
+ | CI | `action-kotoshu` | workflow yaml, SARIF upload |
23
+
24
+ ## Structure
25
+
26
+ - `/install` becomes a channel-picker (tabs or a table: language →
27
+ command), each linking to a per-channel page under
28
+ `/docs/clients/…`.
29
+ - `/docs/clients/python|javascript|rust|go|http|lsp|action` pages:
30
+ install, minimal example, capability notes (which engine tier,
31
+ what is native vs client), link to package registry page.
32
+ - Per-channel quick starts **verified by running them** (pip install
33
+ + 5-line script; npm install + node -e; cargo add + cargo run;
34
+ docker run + curl) — the docs/quickstart-verification precedent
35
+ (gem plan 28) applies: no example ships unexecuted.
36
+ - The audiences page cross-links each persona to their channel.
37
+
38
+ ## Phases
39
+
40
+ 1. `/install` restructure (channel picker).
41
+ 2. Seven per-channel pages + examples verified.
42
+ 3. Nav/sitemap/search-index updates.
43
+
44
+ ## Owner gates
45
+
46
+ None.
47
+
48
+ ## Status
49
+
50
+ **Implemented (2026-09-05, site PR #2).** /install is a channel catalog
51
+ (8 channels); seven guides under /docs/clients/ (python javascript rust
52
+ go http lsp action). Every quick start executed for real (pip, npm http
53
+ + wasm, cargo, go, server-from-source + curl, LSP stdio handshake);
54
+ the action YAML parse-validated only. Each page carries a verified-date
55
+ annotation. kotoshu-server docs intentionally show run-from-source
56
+ until the empty 0.1.0 gem is republished.