kotoshu 0.3.0 → 0.4.0

data/CHANGELOG.md

@@ -5,7 +5,347 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [Unreleased]
+## [0.5.0] — 2026-06-30
+
+Tier 2.5 release. Adds the personal-dictionary CLI, completes direct
+spec coverage for every previously-untested namespace (~370 new
+examples), eliminates every `respond_to?` / `send`-to-private /
+`instance_variable_*` violation from `lib/`, and fixes a long list of
+production bugs the refactors surfaced.
+
+### Added
+
+- **Cache eviction** (`Kotoshu::Cache::EvictionPolicy` +
+  `BaseCache#evict` + `kotoshu cache evict`). A pure value object
+  decides which entries to evict (LRU by `cached_at`, oldest first)
+  to fit a configured size cap; `BaseCache#evict` collects on-disk
+  entries (one record per discovered `metadata.json`) and executes
+  the plan. `--dry-run` returns the plan without touching disk. The
+  cap defaults to `max_cache_size` (1 GB, `KOTOSHU_MAX_CACHE_SIZE`)
+  and is now wired through `Configuration` instead of a hardcoded
+  constant. Corrupt metadata (missing `cached_at`) sorts oldest so it
+  is evicted first.
+- **Audit log rotation** (`Kotoshu::Integrity::RotationPolicy`). When the
+  current `audit.log` exceeds `audit_max_bytes` (default 10 MB,
+  `KOTOSHU_AUDIT_MAX_BYTES`), the log rotates through `audit_rotations`
+  historical files (default 5, `KOTOSHU_AUDIT_ROTATIONS`). Total on-disk
+  footprint is bounded at `max_bytes * (rotations + 1)`. The rotation
+  policy is a pure value object — `AuditLog#record` consults it on every
+  write and executes the returned rename plan under an exclusive flock
+  on a sibling lockfile (`audit.log.lock`) so concurrent writers cannot
+  race the rename chain. `AuditLog#entries` now walks both the current
+  log and all rotations, newest-first.
+- **Shell completion** (`kotoshu completions bash|zsh|fish`). Emits a
+  shell-specific completion script that completes top-level subcommands
+  and dynamically completes language codes for `setup` / `fetch` by
+  shelling out to `kotoshu completions languages`. README documents
+  install paths for bash, zsh, and fish.
+- **Load-time model cache integrity** (`Cache::ModelCache`). A cached
+  ONNX model file is re-verified against the SHA-256 recorded in its
+  `metadata.json` every time it is loaded — not just at download time.
+  Tampering, truncation, or partial writes are caught before the file
+  reaches the ONNX runtime, and the resulting `Kotoshu::IntegrityError`
+  includes a `remediation:` hint pointing the user at
+  `kotoshu cache download :LANG --model`. Legacy caches whose metadata
+  predates the checksum field continue to load (graceful degradation).
+- **`IntegrityError#remediation`** — the error now accepts an optional
+  `remediation:` keyword whose text is appended to the message and
+  exposed via a reader, so callers can surface the user-facing fix
+  (e.g. "Run `kotoshu cache download :en --model` to re-download").
+  Existing callers that omit it see no change in the message format.
+- **Personal dictionary CLI** (`kotoshu personal SUBCOMMAND`). Thor
+  subcommand wrapping `Kotoshu::PersonalDictionary`: `add`, `remove`,
+  `list`, `import` (with `--dry-run`), `path`, and `clear` (with
+  `--yes`). Errors raise `Kotoshu::Cli::Errors::UsageError` (exit 2),
+  matching the existing CLI error contract.
+- **Direct specs for previously-untested namespaces.** ~370 new
+  examples pinning the public contract of `Integrity`, `Embeddings`,
+  `Language`, `Dictionary`, and `Cache` namespaces — each spec uses
+  real instances and tmpdir/fixture IO, never `double()`.
+- **`Kotoshu::Dictionary::Unified`** is now loadable. The class
+  was previously declared as `class Kotoshu::Dictionary`, colliding
+  with the existing `module Kotoshu::Dictionary` namespace and
+  raising `TypeError` on autoload. Renamed to `Unified` to match the
+  autoload entry. Two follow-on bugs (mismatched `Suggester.new`
+  kwargs and a custom `build_dic_structure` that produced the wrong
+  shape) fixed in the same change.
+
+### Changed
+
+- **`Embeddings::LruCache#fetch`** now yields the *key* to the provided
+  block (matching Ruby's `Hash#fetch` convention) instead of invoking
+  the block with no arguments. Callers that compute the missing value
+  from the key no longer need a closure over an outer variable.
+- **`Embeddings::SimilarityEngine`** exposes a `pre_normalize?`
+  predicate (replacing a dead `attr_reader :pre_normalize` that was
+  silently shadowed by a private method of the same name) and now
+  raises `ArgumentError` with a clear dimension-mismatch message
+  instead of crashing with a `TypeError` from `Array#zip` when given
+  vectors of differing lengths. `nil` and empty inputs still return
+  `0.0` (early-return path).
+- **Eliminated `respond_to?` across `lib/`.** Replaced every
+  occurrence with `is_a?` / `case-when` / `method_defined?` / direct
+  calls. Duck-typing via `respond_to?` hid real type errors — the
+  production bugs surfaced by this refactor (see Fixed) had been
+  silently no-op'ing for as long as the code existed.
+- **Eliminated `send` to private/protected methods** in `lib/`. The
+  `cache` CLI subcommand's `cache.send(:read_metadata, ...)` is now
+  `cache.read_metadata(...)` after promoting the method to public
+  (`public def`). `SemanticStrategy#embedding_for` was dead code that
+  called `@search.send(:get_embedding, ...)` on a method that didn't
+  exist — removed entirely.
+- **Eliminated `instance_variable_set/get` across `lib/`.**
+  `BloomFilter#merge` now reads `other.bits` (public `attr_reader`)
+  instead of `other.instance_variable_get(:@bits)`.
+  `Dictionary::PlainText` and `Dictionary::CSpell` constructors now
+  accept a `words:` keyword instead of `from_words` poking ivars via
+  `allocate` + `instance_variable_set`. `Readers::LookupBuilder`
+  gained `aff_data:` / `words:` keyword args to replace the same
+  pattern in `from_data`.
+- **Configuration dispatch is type-checked, not duck-typed.**
+  `Configuration#apply_*` previously called
+  `send("#{key}=", value) if respond_to?("#{key}=")`; now uses
+  `public_send(setter, value) if self.class.method_defined?(setter)`.
+- **`Dictionary::Base` exposes a default `path` method** returning
+  `nil` (overridden by file-backed subclasses). `Spellchecker`
+  previously guarded with `dictionary.respond_to?(:path)`; now calls
+  unconditionally.
+- **`Language::Registry.info` checks the type bound.** Was
+  `klass.respond_to?(:instance)`; now
+  `klass.is_a?(Class) && klass <= Kotoshu::Language::Base`.
+
+### Fixed
+
+- **`AuditLog#each` now iterates newest-first across current and
+  rotations.** The docstring promised newest-first but the
+  implementation yielded each file in write order (oldest line first)
+  then walked current → oldest file, producing a jumbled order. Each
+  file's lines are now read into memory and yielded in reverse before
+  walking to the next older rotation.
+- **`Embeddings::Protocol` is no longer broken.** The module declared
+  `required_methods` and `required_methods(*names)` (a reader + a
+  writer with the same name). Ruby's last-def-wins shadowed the
+  reader, so every protocol registration silently dropped its
+  arguments and `compliance_errors` returned `[]` unconditionally —
+  every class appeared to conform to every protocol. Renamed the
+  writer to `required` (and `optional_methods` → `optional`); also
+  fixed `compliance_errors` to use `klass.method_defined?` instead of
+  `klass.respond_to?` (which checks class methods, not instance
+  methods).
+- **`Embeddings::OnnxRuntimeModel` defines `loaded?`.** The protocol
+  requires `loaded?` but the class declared only `attr_reader :loaded`
+  (no question mark). `EmbeddingPipeline#stats` and `#to_s` called
+  `@model.loaded?` — would have raised `NoMethodError` on the real
+  model.
+- **`Dictionary::Base` aliases dispatch dynamically.** Ruby's `alias`
+  keyword binds to the parent method body at definition time,
+  bypassing subclass overrides. `has_word?`/`include?`/`contains?`/
+  `lookup?`/`<<`/`all_words` on a concrete subclass (e.g.,
+  `UnixWords`) would invoke the abstract `Base#lookup` /
+  `Base#add_word` and raise `NotImplementedError`. Replaced with
+  one-line method definitions that call the abstract method (which
+  then dispatches dynamically to the subclass override).
+- **`Dictionary::CSpell#add_word` no longer crashes.** Tried to
+  `@trie.insert` on a frozen `Trie` (frozen by `Builder#build`),
+  raising `FrozenError`. Aligned with `remove_word`'s existing
+  "always returns false (immutable after load)" contract.
+- **`Cache::FrequencyCache#get_frequency` no longer raises
+  `ArgumentError`.** Called `get(language_code, force_download)`
+  positionally, but `BaseCache#get` declares `force_download:` as
+  keyword. Fixed to use the keyword form.
+- **`Cache::FrequencyCache#cached_resources` no longer reports the
+  `tmp/` scratch slot.** `BaseCache#initialize` creates
+  `<cache_path>/tmp/` as a working directory; the directory-walking
+  implementation in `FrequencyCache` listed it as a cached language.
+  Added an explicit `basename != "tmp"` exclusion.
+- **`Cache::FrequencyCache#install_local` exists.** `ResourceManager`
+  called `freq_cache.install_local(...)` behind a `respond_to?` guard
+  that always returned `false`, so local frequency installs were
+  silently skipped — `setup(from:)` reported `frequency_status: :local`
+  even though no install happened. Added the method, mirroring
+  `LanguageCache#install_local` but taking a single `path:` kwarg.
+- **`PhonetTable::Rule#match?` is no longer dead code.**
+  `Algorithms::PhonetSuggest` had its own `match_rule` that
+  duplicated the rule-matching logic, ignoring `Rule#match?`
+  entirely. Consolidated into `Rule#match_length` (returning
+  `Integer|nil` — the metaphone walker needs the length to advance
+  its position) with `Rule#match?` as a boolean wrapper;
+  `PhonetSuggest#match_rule` now delegates.
+- **`DoubleNegativeMatcher` recognizes the "not only...but also"
+  correlative.** The previous check fired only on the narrow shape
+  `not [current_negative] only`, missing the actual idiom (where
+  "not" and "only" are adjacent). Replaced with a region-based
+  matcher that finds every exception phrase in the token stream and
+  suppresses any negative whose index falls inside a region. Supports
+  both `"prefix...suffix"` (open-ended) and `"literal n-gram"` (no
+  ellipsis) phrase shapes, driven by the rule's `exceptions.phrases`
+  config.
+- **`SemanticAnalyzer` no longer returns `[]` for OOV words.**
+  `suggest_corrections` called `@model.nearest_neighbors(word)`,
+  which returns `[]` for OOV inputs — but OOV is precisely what the
+  analyzer exists to handle. Added an edit-distance fallback: when
+  the model returns `[]`, walk the vocabulary and return words within
+  Levenshtein distance 2, scored by `1/(1+distance)`. Drive-by fix:
+  `Models::Suggestion.new` was called with `word:` as keyword but
+  the constructor takes `word` positional — exposed once the OOV
+  fallback actually reached the `Suggestion.new` call.
+
+### Removed
+
+- **Orphan files emptied to no-op stubs.**
+  `lib/kotoshu/embeddings/protocols.rb` (plural) had
+  `require_relative 'protocols/embedding_model'` pointing at
+  nonexistent files (loading it raised `LoadError`).
+  `lib/kotoshu/readers/readers.rb` (nested) duplicated the autoload
+  setup in `lib/kotoshu/readers.rb` with `require_relative` calls
+  (violating the project rule against `require_relative` in lib/).
+  Both reduced to documentation-only stubs; no callers were affected.
+  The real protocol definitions live in
+  `lib/kotoshu/embeddings/protocol.rb`; the real reader autoloads
+  live in `lib/kotoshu/readers.rb`.
+
+## [0.4.0] — 2026-06-29
+
+Tier 2 release. Completes the Hunspell correctness work (morphological
+rules, REP/phonet/ngram ranking, compound handling), migrates the entire
+library from `require_relative` to Ruby `autoload`, and tightens the
+cache interface alignment that 0.3.1 sketched.
+
+### Added
+
+- **Hunspell morphological correctness** (T2 Phases 2A–2D):
+  - `FLAG long|num|utf-8` parsing and `AF` alias resolution.
+  - `COMPOUNDRULE`, `CHECKCOMPOUNDPATTERN` (with empty-flag fix),
+    `CIRCUMFIX` handling.
+  - `ICONV`/`OCONV` `ConvTable` with stable sort — preserves Nepali
+    `ZWNJ`/`ZWJ` ordering.
+  - `KEEPCASE` suggestion flag with `CHECKSHARPS` exception, plus the
+    German eszet rule (`ss` ↔ `SS`).
+  - `REP` table, phonet ordering, ngram ranking wired end-to-end.
+  - Edge cases: `IJ` digraph, `i58202` case preservation,
+    `opentaal_forbiddenword2`, `breakdefault` `BREAK` pattern.
+- **Hunspell Suggester** is now wired into the Hunspell dictionary; the
+  default affix directives are honoured when generating suggestions.
+- **`BaseCache#max_cache_size`** attribute (1 GB default) — the
+  forward-looking hook for cache eviction work (see
+  `TODO.impl/34-cache-eviction.md`).
+- **`LanguageCache#language_path(lang, type)`** public helper; resource
+  and metadata path resolution is now DRY'd through it.
+- **`Kotoshu::Language` suika soft-load module** with specs — keeps
+  tokenization lazy and optional.
+- **Multi-language EditDistanceStrategy spec coverage** — pins
+  per-language keyboard selection (QWERTZ/AZERTY/QWERTY/JCUKEN) and
+  verifies German and French typo correction end-to-end. The previous
+  `skip 'Multi-language support not yet implemented'` hook is removed;
+  the strategy has been language-aware via `Keyboard::Registry` +
+  `Cache::FrequencyCache` for several releases.
+
+### Changed
+
+- **Library-wide `autoload` migration.** Every `require_relative` (and
+  in-library `require`) under `lib/kotoshu/` is replaced with a Ruby
+  `autoload` declared in the immediate parent namespace's file. This
+  fixes load-order races, makes the eager/opt-in split explicit, and
+  matches the contributor guidance in `CLAUDE.md`. Behaviour is
+  unchanged for callers who only `require "kotoshu"`, but extenders who
+  reached past the public facade may need to update their load entry
+  points.
+- **`Embeddings` namespace consolidation.** All embedding classes are
+  nested under `Kotoshu::Embeddings` (was scattered with several
+  `require_relative` shims).
+- **Grammar rules bundled in the gem.** English `rules.yaml` ships
+  inside the gem package so a `gem install kotoshu` is self-contained
+  for grammar checking without a separate download.
+- **Plain-text edit distance is now case-insensitive**, matching the
+  Hunspell backend's behaviour for `PlainText` dictionaries.
+
+### Fixed
+
+- **`Cli::CacheCommand` wired to the real `LanguageCache` API.** The
+  previous implementation called nonexistent methods (`cache_status`,
+  `get_frequency_data`, `get_language_info`, `purge_all`); `kotoshu cache`
+  now correctly routes through `available?`, `cached_resources`,
+  `clear_all`, `clean`, `stats`, `get_spelling`, `get_grammar`, and the
+  standalone `Cache::FrequencyCache` for frequency downloads.
+- **`Keyboard::Registry` lazy reload.** `clear` now marks
+  `@languages_loaded = true` to prevent an infinite lazy-reload loop
+  when the registry is reset at runtime.
+- **German common-words YAML.** `data/common_words/de.yml` line 466 had
+  `-ecke` (missing space after dash), a syntax error that prevented the
+  file from loading and silently broke German strategy instantiation.
+- **`Readers::AffReader` Hunspell edge cases** — encoding fallback for
+  Latin-1 fixtures, `REP` table splitting, `MultiWord` dash variants,
+  default affix directives honoured when the `.aff` file omits them.
+- **RuboCop 3.x / plugins migration.** `rubocop-rspec` is bumped to
+  3.x and the config uses the new plugins syntax; safe auto-corrections
+  applied across `lib/` and `spec/` (`Layout/ExtraSpacing`,
+  `Layout/EmptyLine`).
+- **Cross-platform CI.** `bundle` is invoked via `Gem.ruby -S` for
+  Windows compatibility; Ruby 4.0 compatibility fixes applied.
+
+### Internal
+
+- `TODO.impl/` planning documents (00–40) are the tiered execution
+  plan. Tier 2 is complete; Tier 3 (`audit-log-rotation`,
+  `cache-eviction`, `shell-completion`) and content-repo tasks
+  (`onnx-vocab-json-generation`) remain deferred past 0.4.0.
+
+## [0.3.1] — 2026-06-28
+
+Bug-fix release. Closes the regressions introduced by the 0.3.0
+lutaml-model migration and tightens up spec gating so the default suite
+is green on the non-network, non-ONNX subset.
+
+### Fixed
+
+- `WordResult#suggestions` is `Array<Suggestion>` after the lutaml
+  migration; the integration spec now reads it as
+  `suggestions.map(&:word)` instead of the removed `SuggestionSet#words`.
+- `WordResult#to_h` was removed by lutaml; the integration spec now uses
+  the framework-supplied `to_hash`.
+- `Readers::AffReader#detect_encoding` now handles a nil path so the
+  reader can be constructed from an in-memory `StringReader` (used by
+  the unit tests and by programmatic callers that build affix data
+  without a file on disk).
+- `Embeddings::SimilaritySearch` is now eagerly loaded alongside the
+  rest of the embeddings namespace; previously the constant was
+  uninitialized when `SemanticStrategy` tried to use it.
+- `Embeddings::Vocabulary.from_cache` is implemented (was missing). It
+  resolves the `vocab.json` sibling of the cached ONNX model and
+  returns nil when the model or vocab is unavailable, so callers can
+  degrade gracefully.
+- `Embeddings::OnnxRuntimeModel.from_cache` now accepts the `cache:`
+  keyword (matching `Vocabulary.from_cache` and
+  `SimilaritySearch.from_cache`); the previous positional signature
+  silently swallowed the cache as a Hash.
+- `Suggestions::Strategies::SemanticStrategy` now surfaces its named
+  constructor kwargs (`min_semantic_similarity`,
+  `semantic_boost_weight`, `max_context_window`) through `get_config`,
+  so the strategy's configurable knobs are queryable the same way as
+  the base strategy's `**config` bag.
+
+### Changed
+
+- `spec/spec_helper.rb` excludes `:onnx`-tagged examples unless
+  `ONNX_TESTS=1` is set, mirroring the existing `:network` and `:slow`
+  opt-in pattern. `spec/kotoshu/suggestions/strategies/semantic_strategy_spec.rb`
+  is the sole consumer; its `:integration` tags are now `:onnx`.
+- The SymSpell benchmark spec and the edit-distance / symspell
+  performance describe blocks are tagged `:slow`, removing
+  environment-dependent timing flakes from the default run.
+- Arabic language detection is marked `pending` with a pointer to
+  `TODO.impl/30-language-auto-detection.md` (FastText LID missing the
+  Arabic vector).
+- `edit_distance_strategy_spec` now uses the `from_source?` predicate
+  instead of asserting `source == :edit_distance`, matching the
+  lutaml-era contract that `Suggestion#source` is a string.
+
+### Internal
+
+- Added `TODO.impl/36-cleanup-regressions-0.3.1.md` (this release),
+  `TODO.impl/37-hunspell-correctness-tier2.md`,
+  `TODO.impl/38-onnx-semantic-gating.md`, and
+  `TODO.impl/39-tier3-and-beyond.md` as the tiered execution plan.
 
 ## [0.3.0] — 2026-06-27
 

data/CLAUDE.md

@@ -21,6 +21,8 @@ bundle exec rspec -e "matches a word"   # Run examples matching a name
 bundle exec rspec --only-failures       # Rerun just failing examples (uses .rspec_status)
 
 NETWORK_TESTS=1 bundle exec rspec       # Opt INTO tests that download dictionaries
+ONNX_TESTS=1 bundle exec rspec          # Opt INTO tests that need onnxruntime + cached models
+SLOW_TESTS=1 bundle exec rspec          # Opt INTO benchmarks and full-dictionary sweeps
 bundle exec rubocop                     # Lint
 bundle exec rubocop -A                  # Lint with safe auto-fix
 bundle exec rake                        # default task = spec + rubocop
@@ -32,6 +34,8 @@ gem build kotoshu.gemspec && gem install kotoshu-*.gem
 
 Notes that aren't obvious from the Rakefile:
 - `spec/spec_helper.rb` excludes anything tagged `:network` unless `NETWORK_TESTS=1` is set — those specs download dictionaries from GitHub and are slow/flaky.
+- `spec/spec_helper.rb` excludes anything tagged `:onnx` unless `ONNX_TESTS=1` is set — those specs need `onnxruntime` installed **and** the language model cached via `kotoshu setup :en --model`.
+- `spec/spec_helper.rb` excludes anything tagged `:slow` unless `SLOW_TESTS=1` is set — those are timing-sensitive benchmarks and full-dictionary sweeps.
 - SimpleCov runs on every `rspec` invocation (configured in `spec_helper.rb`).
 - `spec/spylls_test_helper.rb` is mixed into every spec. It ports Hunspell's reference test fixtures from [Splylls](https://github.com/neolithos/spylls) (the Python Hunspell port); many specs assert behavior against those fixtures.
 

data/README.adoc

@@ -715,6 +715,39 @@ Total:
 ======================================================================
 ----
 
+=== Shell Completion
+
+Kotoshu ships tab-completion scripts for bash, zsh, and fish. The scripts
+complete subcommands for `kotoshu <TAB>` and language codes for
+`kotoshu setup <TAB>` (the language list is read dynamically via
+`kotoshu completions languages`, so newly registered languages appear
+without re-running the install step).
+
+Install for your shell:
+
+[source,bash]
+----
+# bash (system-wide)
+kotoshu completions bash > /etc/bash_completion.d/kotoshu
+
+# bash (single user)
+kotoshu completions bash > ~/.local/share/bash-completion/completions/kotoshu
+
+# zsh — drop into any directory on your $fpath
+kotoshu completions zsh > "${fpath[1]}/_kotoshu"
+
+# fish
+kotoshu completions fish > ~/.config/fish/completions/kotoshu.fish
+----
+
+To see the supported language codes without installing a completion
+script:
+
+[source,bash]
+----
+kotoshu completions languages
+----
+
 === Language Support Matrix
 
 Kotoshu provides multi-language support with varying feature availability.
@@ -937,6 +970,93 @@ kotoshu setup en --want spelling,model
 git clone https://github.com/kotoshu/dictionaries.git ~/src/kotoshu/dictionaries
 ----
 
+== Writing a Document Plugin
+
+Kotoshu ships only `Kotoshu::Documents::PlainTextDocument` for the
+plain-text case. Format-specific parsers (Markdown, AsciiDoc,
+reStructuredText, LaTeX, …) live in plugins — see the
+`kotoshu-document-plugin-boundary` design note. Kotoshu auto-discovers
+plugin files via `Gem.find_files("kotoshu_plugin/document/*.rb")` — install
+the gem and the parser registers itself on first use.
+
+A plugin has three pieces:
+
+1. *Parser class* — responds to
+   `.from_string(text, language_code:)` (and optionally
+   `.from_file(path, language_code:)`). Returns a
+   `Kotoshu::Documents::Document` whose `text_nodes` carry proper
+   `Documents::SourceRange`s.
+2. *Plugin file* — placed at
+   `lib/kotoshu_plugin/document/<format>.rb` in the gem. The file body
+   calls `Kotoshu::Documents.register(format, ParserClass)` at load
+   time.
+3. *Gem dependency* — `kotoshu` (this gem) and whatever
+   format-parsing library the parser wraps (e.g. `kramdown` for
+   Markdown, `asciidoctor` for AsciiDoc).
+
+Example skeleton (`my_markdown_parser` gem):
+
+[source,ruby]
+----
+# lib/my_markdown_parser/document_parser.rb
+module MyMarkdownParser
+  class DocumentParser
+    # Build one or more TextNodes from the Markdown AST, each
+    # carrying a SourceRange that points at the original source.
+    def self.from_string(text, language_code: nil)
+      # ... walk the AST, collect (text, source_offset) per text run,
+      # build TextNodes with SourceRanges ...
+      Kotoshu::Documents::Document.new(
+        text_nodes: nodes,
+        source: text,
+        format: :markdown,
+        language_code: language_code
+      )
+    end
+  end
+end
+----
+
+[source,ruby]
+----
+# lib/kotoshu_plugin/document/markdown.rb
+require "kotoshu"
+require "my_markdown_parser/document_parser"
+Kotoshu::Documents.register(:markdown, MyMarkdownParser::DocumentParser)
+----
+
+After installing the gem, callers get Markdown parsing for free:
+
+[source,ruby]
+----
+require "kotoshu"
+
+doc = Kotoshu::Documents.parse(source, format: :markdown,
+                              language_code: "en")
+errors = Kotoshu::Spellchecker.new(
+  resource_bundle: Kotoshu::ResourceManager.resolve(language: "en")
+).check(doc.flattened_text)
+errors.each do |error|
+  puts "#{error.source_range}: #{error.original} → #{error.recommended_suggestion}"
+end
+----
+
+The structure-aware contract: the analyzer scans
+`doc.flattened_text` (concatenation of every node's `text`), but every
+error carries a `source_range` from the original source. An editor can
+use that range to highlight the user's actual markup, not the
+stripped text. For example, on the source
+
+  I'm an **friend** of Tom
+
+a grammar error "an friend" → "a friend" carries a source_range
+pointing at the original `an **friend**` range, not the flattened
+"an friend" range.
+
+Plugin gems should publish to RubyGems with a runtime dependency on
+`kotoshu >= 0.5.0`. Format-specific parsers are the plugin's
+responsibility — kotoshu never owns document parsing.
+
 == License
 
 BSD 2-Clause — see the link:LICENSE[LICENSE] file for details.

data/TODO.impl/36-cleanup-regressions-0.3.1.md

@@ -0,0 +1,117 @@
+# 36 — Cleanup Regressions + 0.3.1 Bug-Fix Release
+
+## Goal
+
+Ship `kotoshu-0.3.1` — a focused bug-fix release that closes the
+regressions introduced by the 0.3.0 lutaml-model migration and a small
+set of pre-existing spec failures. This is the **only** Tier-1 work;
+its acceptance criterion is a clean spec run on the non-network,
+non-ONNX subset.
+
+## Scope (locked)
+
+| # | Failure cluster | Root cause | Fix shape |
+|---|---|---|---|
+| 1 | `spec/integration/walking_skeleton_spec.rb:22` — `result.suggestions.words` | Lutaml migration changed `WordResult#suggestions` from `SuggestionSet` to `Array<Suggestion>`. | Update spec to `result.suggestions.map(&:word)`. |
+| 2 | `spec/integration/walking_skeleton_spec.rb:265` — `result.to_h` | Lutaml-model replaced hand-rolled `to_h`. The instance method is gone; framework supplies `to_hash`. | Update spec to call `result.to_hash` (lutaml-supplied). |
+| 3 | `spec/kotoshu/suggestions/strategies/semantic_strategy_spec.rb` (22 failures) | Specs assume `onnxruntime` is installed and a model is loaded. Without ONNX, every example errors. | Tag the spec file `:onnx` and exclude by default; document the opt-in env (`ONNX_TESTS=1`) in CLAUDE.md. Mirror the `:network` opt-in pattern. |
+| 4 | `spec/unit/hunspell/read_aff_spec.rb` (7 failures) | Real parser bugs in REP, MAP, PFX directive parsing and UTF-8 / numeric / long flag formats. | Fix each parser branch with a targeted spec; do **not** rewrite the whole reader. |
+| 5 | `spec/kotoshu/language/detector_spec.rb:56` — Arabic detection | Pre-existing FastText LID gap. | Mark `pending` with a TODO referencing the upstream model; do not block the release on it. |
+| 6 | `spec/benchmark/symspell_benchmark_spec.rb` (1 failure) | Likely environment-dependent timing. | Audit; if real bug, fix; if env noise, mark `:slow`. |
+
+**Out of scope:** the 90+ `spec/integrational/{lookup,suggest}_spec.rb`
+Hunspell-fixture failures. Those are T2 (`TODO.impl/37-hunspell-correctness-tier2.md`).
+
+## Phase A — Lutaml regression cleanups
+
+### A1. walking_skeleton_spec
+
+- Line 28: `expect(result.suggestions.words).to include("hello")`
+  → `expect(result.suggestions.map(&:word)).to include("hello")`
+- Line 268: `hash = result.to_h`
+  → `hash = result.to_hash` (lutaml-model supplies this)
+
+### A2. semantic_strategy_spec gating
+
+- Add `describe ..., :onnx do` or `RSpec.describe ... do; before { skip "set ONNX_TESTS=1" unless ENV["ONNX_TESTS"] } end`.
+- Update `spec/spec_helper.rb` to exclude `onnx: true` unless `ONNX_TESTS=1`.
+- Update `CLAUDE.md` "Notes" section to mention the opt-in.
+
+### A3. Audit other lutaml fallout
+
+- `grep -rn "\.suggestions\.words\|\.suggestions\.to_a\|\.to_h\b" spec/`
+  (only the two walking_skeleton hits — confirmed.)
+- `grep -rn "\.as_json\b" spec/` — already replaced in cli.rb; verify
+  no spec calls the removed instance method.
+
+**Acceptance:** `bundle exec rspec --tag ~network --tag ~onnx` shows
+**0 failures** attributable to lutaml migration.
+
+## Phase B — Hunspell .aff parser fixes
+
+For each failing directive, add a focused spec covering the exact
+shape from the failing fixture, then fix the parser branch.
+
+### B1. REP directive (read_aff_spec:12)
+
+Repairs — substitution pairs used by the suggester. Currently parsed
+incorrectly.
+
+### B2. MAP directive (read_aff_spec:31)
+
+Character equivalence classes (e.g. `MAP uü`, `MAP aàáâã`). Sets up
+related-character substitution in suggestions.
+
+### B3. PFX directive (read_aff_spec:47)
+
+Prefix rules. Currently fails to parse multi-rule PFX blocks. Likely
+the same bug as SFX since they share the parser.
+
+### B4. Long flag format (read_aff_spec:62)
+
+`FLAG long` mode — two-character flags.
+
+### B5. UTF-8 flag format (read_aff_spec:106)
+
+`FLAG UTF-8` mode — flags are single Unicode codepoints.
+
+### B6. Numeric flag format (read_aff_spec:125)
+
+`FLAG num` mode — integer flags.
+
+### B7. AF directive / flag aliases (read_aff_spec:125)
+
+`AF` directive defines aliases used to keep `.dic` files compact.
+
+**Acceptance:** all 7 `read_aff_spec` examples green; no regression in
+existing `algorithms/` specs.
+
+## Phase C — Pre-existing pending/skips
+
+- Arabic detection: `pending "FastText LID missing Arabic vector — see TODO.impl/30-language-auto-detection.md"`.
+- Symspell benchmark: audit; if real, fix; if env noise, `:slow`.
+
+## Phase D — Cut 0.3.1
+
+1. Bump `lib/kotoshu/version.rb` → `0.3.1`.
+2. Update CHANGELOG (or create if missing).
+3. Update README quickstart if it references any 0.3.0-broken API.
+4. Run full suite (`bundle exec rspec --tag ~network --tag ~onnx`)
+   green; `bundle exec rubocop` clean.
+5. PR (not direct-to-main — see global rule).
+6. After merge: tag `v0.3.1` (the **user** does this; never the agent).
+
+## Release-level acceptance criteria
+
+- `bundle exec rspec --tag ~network --tag ~onnx` → 0 failures
+  (excluding the explicitly-pending FastText Arabic case).
+- `bundle exec rubocop` → clean.
+- Lutaml migration regressions: 0.
+- Pre-existing Hunspell `.aff` parser bugs: 0.
+- No new public API changes — 0.3.1 is strictly a bug-fix release.
+
+## Dependencies
+
+- **Blocked by:** nothing.
+- **Blocks:** `TODO.impl/37-hunspell-correctness-tier2.md` (the next
+  tier of Hunspell-fixture work).