kotoshu 0.3.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 496e00727251ee935375c0a8013132917384d1847fe99ed3752a414f3a73e5d6
+  data.tar.gz: 6998e4c7879ecca1888a1bedbfe41af5e5516c8a2497457f887c06f310f53678
+SHA512:
+  metadata.gz: 9bafd7e06458c93a4a3b817fdddd50beac4a74cbcc22c3cf0de805dd1f2211d023c5f7dea2e187b1c0fdbe1772901c7e9c22d2956c3b9d887a2f20dbdb267b35
+  data.tar.gz: 4e37cee72f98cf5a171c1f167c52720213ab93d99919f50268bba998836233d09a3e4fb8b0d1756bacc52fb554e42541a7ba8918dba99067dc2da146dd5e3e51

data/.rspec

@@ -0,0 +1,3 @@
+--format documentation
+--color
+--require spec_helper

data/.rubocop.yml

@@ -0,0 +1,18 @@
+inherit_from:
+  - https://raw.githubusercontent.com/riboseinc/oss-guides/main/ci/rubocop.yml
+  - .rubocop_todo.yml
+
+plugins:
+- rubocop-performance
+- rubocop-rake
+- rubocop-rspec
+
+AllCops:
+  TargetRubyVersion: 3.0
+  NewCops: enable
+  SuggestExtensions: false
+  Exclude:
+    - 'debug_*.rb'
+    - 'test_*.rb'
+    - 'vendor/**/*'
+    - 'tmp/**/*'

data/CHANGELOG.md

@@ -0,0 +1,182 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.3.0] — 2026-06-27
+
+The two-stage release. Resources are now downloaded explicitly via
+`Kotoshu.setup(:en)`; the hot path (`correct?`, `suggest`, `check`) reads
+only from cache and raises a typed error when a language is missing instead
+of triggering a network download. The CLI adds `setup`, `status`, language
+auto-detection, SARIF/JSON output, and an interactive auto-setup prompt.
+`onnxruntime` is now a soft dependency, so `gem install kotoshu` succeeds on
+hosts that can't load native ONNX runtime.
+
+### Added
+
+- **Two-stage resource model** (`Kotoshu::ResourceManager`):
+  `Kotoshu.setup(:en, want: %i[spelling frequency model])` writes into the
+  cache; `Kotoshu::ResourceManager.resolve(language:, want:)` is instant and
+  cache-only, raising `ResourceNotSetupError` on miss. `Kotoshu.setup?` is
+  the predicate for "is this language already cached?". The library never
+  triggers a surprise download; the CLI prompts the user via `AutoSetup`.
+- **`SourceRegistry`** — single source of truth for the three content repos'
+  URLs and per-repo pins. `kotoshu/dictionaries` is pinned to the `v1`
+  branch; `frequency-list-kelly` and `models-fasttext-onnx` are on `main`.
+  Override at runtime via `KOTOSHU_REPOS_BASE_URL`, `KOTOSHU_DICTIONARIES_PIN`,
+  `KOTOSHU_FREQUENCY_PIN`, `KOTOSHU_MODELS_PIN`.
+- **XDG Base Directory layout** (`Kotoshu::Paths`): dictionaries, frequency
+  lists, ONNX models under `$XDG_CACHE_HOME/kotoshu/`; personal dictionary
+  and `kotoshu.cfg` under `$XDG_CONFIG_HOME/kotoshu/`; audit log under
+  `$XDG_DATA_HOME/kotoshu/audit.log`. Override per-axis with
+  `KOTOSHU_CACHE_PATH`, `KOTOSHU_CONFIG_PATH`, `KOTOSHU_DATA_PATH`.
+- **Integrity verification** — `Kotoshu::Integrity::Manifest` (SHA-256) is
+  fetched per content repo and matched against every download. Mismatches
+  raise `Kotoshu::IntegrityError`. Outcomes (verified / unverified / mismatch)
+  are written to the audit log. Missing manifests degrade gracefully.
+- **CLI `setup` command** — `kotoshu setup LANG [--force] [--no-frequency]
+  [--no-model]` writes the requested resources into the cache with progress
+  reporting.
+- **CLI `status` command** — `kotoshu status [--json]` summarises installed
+  resources, sizes, mtimes, and ONNX runtime availability.
+- **CLI `check --language auto`** — auto-detects document language via
+  FastText LID; falls back to the configured default language when detection
+  is unavailable or the detected language is not set up.
+- **CLI `check --format json|sarif`** — machine-readable output. SARIF
+  follows v2.1.0 with `kotoshu/spelling` rule id, JSON exposes
+  `success`/`wordCount`/`errorCount`/`uniqueErrorCount`/`errors`/`source`.
+- **CLI auto-setup prompt** — when the hot path raises
+  `ResourceNotSetupError` in an interactive session, the user is prompted to
+  run setup now and the original command is retried on success. Non-TTY,
+  offline (`--offline`), and `--no-prompt` invocations skip the prompt and
+  surface the error as before.
+- **Download progress reporting** (`Kotoshu::Cli::ProgressReporter`) — TTY
+  mode renders a determinate/indeterminate progress bar; non-TTY mode prints
+  a periodic line every 10 MiB. `Kotoshu.configuration.download_reporter=`
+  exposes the reporter for programmatic use.
+- **End-to-end smoke spec** (`spec/integration/end_to_end_spec.rb`) covers
+  install → setup → `correct?` → `suggest.to_words` → `check` →
+  `setup?` predicate → `ResourceNotSetupError` → idempotent re-setup.
+  Tagged `:network`, opted into via `NETWORK_TESTS=1`.
+- **CLI format spec** (`spec/kotoshu/cli/check_format_spec.rb`) shells out to
+  the real `kotoshu` CLI and asserts JSON / SARIF structure and exit codes.
+
+### Changed
+
+- **`onnxruntime` is a soft dependency.** Removed from `kotoshu.gemspec`.
+  `Kotoshu::Models::OnnxModel` soft-requires it at load time and exposes
+  `ONNX_LOADED`. When false, semantic methods raise
+  `Kotoshu::Models::OnnxModel::OnnxUnavailable` with a caller-friendly
+  message. `KOTOSHU_NO_ONNX=1` forces semantic off even when the gem is
+  present. The traditional spell-checking path never touches `onnxruntime`.
+- **Loading strategy** — `lib/kotoshu.rb` eagerly loads only the facade
+  dependencies; heavier or optional pieces (ONNX models, interactive CLI,
+  caches, language detection) are wired through Ruby `autoload` registered
+  in their immediate parent namespace.
+- **Public API** — `suggest` returns a `SuggestionSet`; call `.to_words` for
+  an `Array<String>`. `Kotoshu.check` returns a `DocumentResult`; iterate
+  `errors` for `WordResult` instances with `word`, `position`, `line`,
+  `column`, `suggestions`.
+- **README quickstart** — reflects the two-stage API; documents XDG paths;
+  marks `onnxruntime` as optional.
+
+### Fixed
+
+- `gem install kotoshu` no longer requires `onnxruntime` or its native
+  toolchain.
+- Resource resolution no longer triggers downloads from inside the hot path.
+- Per-repo pins are honoured — the `v1` branch of `kotoshu/dictionaries` is
+  fetched instead of `main`.
+
+### Known limitations (carried from 0.1.0, scope reduced)
+
+- **Hunspell correctness**: compound rules, circumfix, ICONV/OCONV, German ß,
+  Turkish dotless-i remain partial. See `TODO.impl/01-hunspell-correctness.md`.
+- **CJK and RTL**: tokenizer, normalizer, and keyboard layouts exist for
+  supported languages; full CJK/RTL support deferred past 0.3.
+  See `TODO.impl/06-cjk-support.md` and `TODO.impl/07-rtl-support.md`.
+- **Grammar rules**: the rule engine exists; no rule packs are shipped.
+  See `TODO.impl/08-grammar-engine.md`.
+- **Audit log rotation, cache eviction policy, and shell completion** are
+  deferred past 0.3 (T3 TODOs).
+
+### Internal
+
+- 9 logical commits on `release-0.3` cover the T1 (architectural) and T2
+  (user-facing) work for this release.
+- `SourceRegistry`, `Paths`, `ResourceManager`, `ResourceBundle`,
+  `SetupResult`, `Integrity::Manifest`, `Integrity::AuditLog`,
+  `Cli::AutoSetup`, `Cli::StatusReport`, `Cli::LanguageResolver`,
+  `Cli::ProgressReporter` are new model-driven types.
+- 73 new specs added (source_registry, end_to_end, check_format,
+  progress_reporter, language_resolver, status_report, auto_setup).
+
+### Contributors
+
+- Ribose Inc.
+
+## [0.1.0] — 2026-06-25
+
+First public release. Kotoshu is a pure-Ruby spellchecker that combines a
+Ruby port of the Hunspell algorithm with optional FastText ONNX embeddings
+for semantic reranking. This release establishes the public Ruby API, the
+basic CLI, and the cache layer.
+
+### Working
+
+- **Ruby API**: `Kotoshu.correct?`, `Kotoshu.suggest`, `Kotoshu.check`,
+  `Kotoshu.check_file`, `Kotoshu.detect_language`
+- **CLI**: `kotoshu check TARGET`, `kotoshu dict SUBCOMMAND`, `kotoshu cache
+  SUBCOMMAND`, `kotoshu version`
+- **Dictionary backends**: Hunspell (`.aff`/`.dic`), CSpell, UnixWords
+  (`/usr/share/dict/words`), PlainText, Custom
+- **Suggestion strategies**: edit distance, phonetic (Phonet), keyboard
+  proximity, n-gram, symspell, composite pipeline
+- **Configuration**: `Kotoshu.configure`, CLI > ENV (`KOTOSHU_*`) >
+  programmatic > defaults via `Configuration::Resolver`
+- **Cache layer**: `LanguageCache`, `FrequencyCache`, `ModelCache` with TTLs
+  and download from `kotoshu/dictionaries`, `kotoshu/frequency-list-kelly`,
+  `kotoshu/models-fasttext-onnx`
+- **Language detection**: FastText LID, 127 languages
+- **Documents**: Plain text, Markdown (Kramdown), AsciiDoc (Asciidoctor)
+- **Test suite**: 803 of 866 examples passing (92.7%), 6 pending
+
+### Known limitations (not blocking 0.1)
+
+- **Hunspell correctness**: compound rules, circumfix, ICONV/OCONV, German
+  ß, Turkish dotless-i are not fully implemented. Single-word lookup and
+  basic affixes work. See `TODO.impl/01-hunspell-correctness.md`.
+- **CLI surface**: `--interactive`, `--format sarif|json|yaml|csv`,
+  `--model fasttext|hybrid`, `--language auto` exist in
+  `lib/kotoshu/commands/check_command.rb` but are not wired through
+  `exe/kotoshu`. See `TODO.impl/02-cli-unification.md`.
+- **Semantic path**: gated behind `ENV['KOTOSHU_REQUIRE_ONNX']` because
+  `onnxruntime` loads eagerly otherwise. Hybrid mode is not the default.
+  See `TODO.impl/05-semantic-path.md`.
+- **Dynamic resolution**: the three caches exist independently; there is
+  no unified `ResourceManager` that takes arbitrary text and yields the
+  full resource bundle. See `TODO.impl/03-dynamic-download.md`.
+- **Languages**: code is wired for English by default. The
+  `dictionaries` repo has 98 language directories but the gem's
+  `lib/kotoshu/languages/` has only 7 modules (de, en, es, fr, ja, pt,
+  ru). See `TODO.impl/04-language-modules.md`.
+- **CJK, RTL**: not implemented. See `TODO.impl/06-cjk-support.md`
+  and `TODO.impl/07-rtl-support.md`.
+- **Grammar rules**: the rule engine exists; no rule packs are shipped.
+  See `TODO.impl/08-grammar-engine.md`.
+- **Integrity verification**: downloaded resources are not currently
+  checksummed. See `TODO.impl/09-integrity-security.md`.
+
+### Internal
+
+- 12 plans under `TODO.impl/` define the path to 1.0
+- Architecture documentation consolidated under `docs/`
+
+### Contributors
+
+- Ribose Inc.

data/CLAUDE.md

@@ -0,0 +1,172 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## What Kotoshu Is
+
+Kotoshu 「言修」 is a **semantic** spell checker for Ruby. It pairs a traditional
+dictionary/affix backend (Hunspell-style) with ONNX-converted FastText word
+embeddings for context-aware suggestions. The README.adoc is the authoritative
+user-facing description; this file is the contributor-facing map.
+
+Key dependencies (`kotoshu.gemspec`): `thor` (CLI), `suika` (tokenizer),
+`onnxruntime` (semantic inference). Ruby 3.1+.
+
+## Development Commands
+
+```bash
+bundle exec rspec                       # Run the full test suite
+bundle exec rspec spec/path/to_spec.rb  # Run one file
+bundle exec rspec -e "matches a word"   # Run examples matching a name
+bundle exec rspec --only-failures       # Rerun just failing examples (uses .rspec_status)
+
+NETWORK_TESTS=1 bundle exec rspec       # Opt INTO tests that download dictionaries
+bundle exec rubocop                     # Lint
+bundle exec rubocop -A                  # Lint with safe auto-fix
+bundle exec rake                        # default task = spec + rubocop
+
+bundle exec bin/console                 # IRB with Kotoshu loaded
+bundle exec exe/kotoshu check FILE      # Run the CLI locally
+gem build kotoshu.gemspec && gem install kotoshu-*.gem
+```
+
+Notes that aren't obvious from the Rakefile:
+- `spec/spec_helper.rb` excludes anything tagged `:network` unless `NETWORK_TESTS=1` is set — those specs download dictionaries from GitHub and are slow/flaky.
+- SimpleCov runs on every `rspec` invocation (configured in `spec_helper.rb`).
+- `spec/spylls_test_helper.rb` is mixed into every spec. It ports Hunspell's reference test fixtures from [Splylls](https://github.com/neolithos/spylls) (the Python Hunspell port); many specs assert behavior against those fixtures.
+
+## Architecture
+
+Kotoshu has **two parallel checking paths** that share infrastructure:
+
+1. **Traditional path** — `Kotoshu::Spellchecker` (facade) → `Suggestions::Generator` → pluggable `Dictionary::*` backends + `Suggestions::Strategies::*` algorithms. This is what `Kotoshu.correct?` / `Kotoshu.suggest` / `Kotoshu.check` use.
+2. **Semantic path** — `Analyzers::SemanticAnalyzer` driven by an `Models::EmbeddingModel` (`FastTextModel` or `OnnxModel`). Used for context-aware reranking and OOV handling. This path is **opt-in** and only loads when needed.
+
+### Layer map
+
+```
+exe/kotoshu ─► lib/kotoshu/cli.rb (Kotoshu::Cli::Cli < Thor)
+                    subcommands: check, dict (DictCommand), cache (CacheCommand)
+                    helpers: cli/interactive_reviewer, cli/batch_reporter,
+                             cli/navigation_manager, cli/display_formatter
+
+Kotoshu module (lib/kotoshu.rb) ─► public facade methods
+                    .correct? .suggest .check .check_file .detect_language ...
+                    all delegate to a singleton Spellchecker
+
+Spellchecker ─► Configuration ─► Dictionary::Repository ─► Dictionary::*
+                                              │
+                                              └─► Suggestions::Generator
+                                                       └─► Strategies::CompositeStrategy
+                                                                (edit_distance, phonetic,
+                                                                 keyboard_proximity, ngram,
+                                                                 symspell, semantic)
+
+SemanticAnalyzer ─► Models::OnnxModel | Models::FastTextModel
+                       └─► Embeddings::* (vocabulary, similarity search, LRU cache)
+```
+
+### Loading strategy
+
+`lib/kotoshu.rb` eagerly `require_relative`s the traditional path (core models, dictionaries, strategies, configuration, spellchecker) and `autoload`s the heavier / optional pieces (ONNX models, documents, interactive CLI, caches, language detection, debug/metrics). When adding a new top-level component, follow the existing split: eager-load only what the facade needs at boot; autoload the rest.
+
+**ONNX is a soft dependency.** `onnxruntime` is NOT in `kotoshu.gemspec` — `gem install kotoshu` succeeds without it. `Models::OnnxModel` soft-requires it at load time and exposes `ONNX_LOADED` (true/false). When false, semantic methods raise `Models::OnnxModel::OnnxUnavailable` with a caller-friendly message. `KOTOSHU_NO_ONNX=1` forces semantic off even when the gem is present. The traditional spell-checking path never touches onnxruntime.
+
+### Resource lifecycle — two-stage model
+
+Resources (dictionaries, frequency lists, ONNX models) flow through a strict two-stage API in `ResourceManager`:
+
+1. **Setup** (`Kotoshu.setup(:en, want: %i[spelling frequency model])`, or `Kotoshu.setup(:en, aff:, dic:)` / `from:` for local sources). Slow, network-required, explicit. Writes into the cache.
+2. **Resolve** (`Kotoshu::ResourceManager.resolve(language:, want:)`). Instant, cache-only, raises `ResourceNotSetupError` on miss.
+
+The hot path (`Kotoshu.correct?`, `.check`, `.suggest`, `.spellchecker_for`) calls `resolve` and lets the error propagate — **setup is never implicit**. This is intentional: users on metered networks or air-gapped hosts must not get a surprise download. `Kotoshu.setup?(:en, resource: :spelling|:frequency|:model)` is the predicate for "is this already in cache?".
+
+`ResourceBundle` (the resolve result) carries `dictionary`, `frequency`, `model`, and `rules`. `SetupResult` (the setup result) reports per-resource status (`:downloaded | :local | :cached | :unavailable`).
+
+### Paths — XDG Base Directory
+
+All on-disk locations are resolved through `Kotoshu::Paths`, which honors `XDG_CACHE_HOME`, `XDG_CONFIG_HOME`, `XDG_LOCAL_HOME` and the override envs `KOTOSHU_CACHE_PATH`, `KOTOSHU_CONFIG_PATH`, `KOTOSHU_DATA_PATH`. Defaults:
+
+| Concern | Default path |
+|---|---|
+| Language dictionaries, frequency lists, ONNX models | `~/.cache/kotoshu/` |
+| Personal dictionary, kotoshu.cfg | `~/.config/kotoshu/` |
+| Audit log | `~/.local/share/kotoshu/audit.log` |
+
+### Resource caching
+
+Three caches under `~/.cache/kotoshu/` (see `CACHE_ARCHITECTURE.md` for detail, README.adoc for the user-facing version):
+
+| Cache | Path | Source | TTL |
+|---|---|---|---|
+| `Cache::LanguageCache` | `~/.cache/kotoshu/languages/{code}/spelling/` | `github.com/kotoshu/dictionaries` | 7 days |
+| `Cache::FrequencyCache` | `~/.cache/kotoshu/frequency-lists/{code}/` | `github.com/kotoshu/frequency-list-kelly` | 7 days |
+| `Cache::ModelCache` | `~/.cache/kotoshu/models/{code}/...` | `github.com/kotoshu/models-fasttext-onnx` (FastText `.vec` → ONNX converted upstream) | 30 days |
+
+`FrequencyCache` feeds `frequency_bonus` in `Suggestions::Strategies::EditDistanceStrategy` — high-frequency words get a ranking boost. The `kotoshu cache` subcommand exposes list/status/download/info/purge/clean operations.
+
+### Configuration
+
+`Configuration` (singleton via `.instance`) is built from a `SCHEMA` hash that declares each option's ENV var, default, type, and description. The `Configuration::Resolver` enforces the priority chain: **CLI flags > ENV (`KOTOSHU_*`) > programmatic > defaults**. When adding a config option, add it to `SCHEMA` (and probably `DEFAULTS`) rather than sprinkling `attr_accessor`s — that's how it picks up ENV support automatically.
+
+`dictionary_type` selects the backend: `:unix_words | :plain_text | :custom | :hunspell | :cspell`. The dictionary is lazy-loaded through `Configuration#dictionary` (cached on the instance; call `reset_dictionary` to reload).
+
+### Language support
+
+Full features (dictionary + affixes + FastText + ONNX + keyboard layout): `de, en, es, fr, pt, ru`.
+Kelly frequency only: `ar, zh, el, it, no, sv` (and `ru`).
+`Language::Identifier` does automatic detection (FastText LID model, 127 languages). Per-language behavior (tokenizer, normalizer) lives in `languages/{code}/language.rb` and `language/tokenizer/*`. Keyboard layouts (`keyboard/layouts/*`) feed `KeyboardProximityStrategy`.
+
+### Suggestion strategies
+
+`Suggestions::Generator::DEFAULT_ALGORITHMS` = `[EditDistanceStrategy, PhoneticStrategy, KeyboardProximityStrategy, NgramStrategy]`, composed via `Strategies::CompositeStrategy`. Also available: `SymspellStrategy`, `SemanticStrategy`. Register new algorithms via `Kotoshu.register_suggestion_algorithm(:name, Klass)` (uses `BaseStrategy.register_type`).
+
+## Code Layout (lib/kotoshu/)
+
+| Path | Responsibility |
+|---|---|
+| `kotoshu.rb` | Public facade + eager/autoload wiring |
+| `spellchecker.rb`, `spellchecker/parallel_checker.rb` | Traditional check facade |
+| `paths.rb` | XDG path resolution (cache, config, data, audit log, personal dict) |
+| `resource_manager.rb`, `resource_bundle.rb` | Two-stage setup/resolve flow + result structs |
+| `configuration.rb`, `configuration/{builder,resolver}.rb` | Config + priority resolution |
+| `core/` | Domain models (`Word`, `AffixRule`, `result/*`), `IndexedDictionary`, `Trie/*`, `exceptions` |
+| `dictionary/` | Backends: `base`, `hunspell`, `cspell`, `unix_words`, `plain_text`, `custom`, `unified`, `repository` |
+| `readers/` | Parsers for Hunspell `.aff` / `.dic` (aff_data, aff_reader, dic_reader, condition_checker, lookup_builder) |
+| `suggestions/` | `generator`, `context`, `suggestion{,_set}`, `pipeline`, `strategies/*` |
+| `algorithms/` | Lower-level Hunspell-style suggestion primitives (ported from Spylls): `ngram_suggest`, `phonet_suggest`, `suggest`, `lookup`, `permutations`, `capitalization` |
+| `analyzers/` | `semantic_analyzer` — the embedding-based checker |
+| `models/` | `embedding_model` (abstract), `fasttext_model`, `onnx_model`, `word_embedding`, `nearest_neighbor`, `semantic_error`, `context`, `suggestion` |
+| `embeddings/` | ONNX runtime glue: `onnx_runtime_model`, `vocabulary`, `similarity_engine`, `similarity_search`, `search`, `embedding_pipeline`, `protocols{,_registry}`, `lru_cache` |
+| `cache/` | `base_cache`, `language_cache`, `model_cache`, `frequency_cache`, plus `lookup_cache` / `suggestion_cache` runtime caches |
+| `language/`, `languages/` | Detection (`identifier`, `detector`), registry, per-language modules, tokenizers, normalizers |
+| `documents/` | Document abstraction: `plain_text_document`, `markdown_document`, `asciidoc_document`, `location` |
+| `cli/` | CLI helpers (interactive reviewer, batch reporter, navigation, display) |
+| `commands/` | Thor subcommands: `check_command`, `cache_command`, `model_command` |
+| `grammar/` | Rule engine + pattern matchers (`rule`, `rule_engine`, `rule_loader`, `pattern_matchers/*`) |
+| `keyboard/` | Layout registry + per-layout files (qwerty, qwertz, azerty, jcuken, dvorak) |
+| `components/`, `plugins/`, `data_structures/`, `results/`, `data/` | Tokenizer/POS/synthesizer components, plugin registry, bloom filter, result base, common-words loader |
+
+The exe uses `Kotoshu::Cli::Cli` (in `cli.rb`), which registers `dict` → `DictCommand` and `cache` → `CacheCommand` as subcommands. A richer `Kotoshu::CheckCommand` exists in `commands/check_command.rb` (with `--interactive`, `--format sarif/json`, `--model`, `--language auto`) — check which one is actually wired before assuming a CLI flag exists.
+
+## Specs
+
+Spec layout mirrors lib: `spec/kotoshu/...`, plus `spec/integration/`, `spec/integrational/`, `spec/performance/`, `spec/benchmark/`, `spec/properties/`, `spec/unit/`, `spec/hunspell_tests/` (Splylls-ported fixtures), `spec/fixtures/`, `spec/support/`.
+
+Global rules that apply here (see `~/.claude/CLAUDE.md`): **no `double()` in specs** — use real instances or `Struct.new`; **no hand-rolled serialization** (`to_h`/`from_h` on models).
+
+## Reference Implementations (read-only, on disk)
+
+When implementing features, study these alongside Kotoshu:
+
+- `/Users/mulgogi/src/external/hunspell/` — morphological rules, affix processing, suggestions (C++ reference).
+- `/Users/mulgogi/src/external/cspell/` — trie/DAFSA dictionaries, code-aware checking (TypeScript reference).
+- `/Users/mulgogi/src/external/languagetool/` — rule-based grammar, multi-interface (library + HTTP), caching patterns (Java reference).
+- Spylls (Python Hunspell port) — the algorithms in `algorithms/` and the fixtures in `spec/hunspell_tests/` derive from here.
+
+## Other Notes
+
+- License is **BSD-2-Clause** (not MIT — the README's "License" section is wrong).
+- RBS signatures live in `sig/kotoshu.rbs` (the `sig/kotoshu/` subdirectory is empty). Update signatures when changing public APIs.
+- `scripts/` contains one-off utilities (FastText→ONNX conversion in Python, Kelly frequency parsing, diagnostics). `examples/` has numbered walkthrough scripts (`01_*.rb` … `07_*.rb`).
+- Design history and superseded planning docs live in `docs/` (`architecture.md`, `cache-architecture.md`, `performance.md`, `plugins.md`, `getting-started.md`, plus integrated planning docs like `KOTOSHU_SOLIDIFICATION_PLAN.md`, `ARCHITECTURE_IMPROVEMENTS.md`, `TDD_ITERATION_STRATEGY.md`). Treat them as historical context, verify against current code before relying on them. `TODO.impl/` is the current source of truth for execution plans.

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,132 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, caste, color, religion, or sexual
+identity and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming,
+diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment for our
+community include:
+
+* Demonstrating empathy and kindness toward other people
+* Being respectful of differing opinions, viewpoints, and experiences
+* Giving and gracefully accepting constructive feedback
+* Accepting responsibility and apologizing to those affected by our mistakes,
+  and learning from the experience
+* Focusing on what is best not just for us as individuals, but for the overall
+  community
+
+Examples of unacceptable behavior include:
+
+* The use of sexualized language or imagery, and sexual attention or advances of
+  any kind
+* Trolling, insulting or derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or email address,
+  without their explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Enforcement Responsibilities
+
+Community leaders are responsible for clarifying and enforcing our standards of
+acceptable behavior and will take appropriate and fair corrective action in
+response to any behavior that they deem inappropriate, threatening, offensive,
+or harmful.
+
+Community leaders have the right and responsibility to remove, edit, or reject
+comments, commits, code, wiki edits, issues, and other contributions that are
+not aligned to this Code of Conduct, and will communicate reasons for moderation
+decisions when appropriate.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when
+an individual is officially representing the community in public spaces.
+Examples of representing our community include using an official email address,
+posting via an official social media account, or acting as an appointed
+representative at an online or offline event.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the community leaders responsible for enforcement at
+[INSERT CONTACT METHOD].
+All complaints will be reviewed and investigated promptly and fairly.
+
+All community leaders are obligated to respect the privacy and security of the
+reporter of any incident.
+
+## Enforcement Guidelines
+
+Community leaders will follow these Community Impact Guidelines in determining
+the consequences for any action they deem in violation of this Code of Conduct:
+
+### 1. Correction
+
+**Community Impact**: Use of inappropriate language or other behavior deemed
+unprofessional or unwelcome in the community.
+
+**Consequence**: A private, written warning from community leaders, providing
+clarity around the nature of the violation and an explanation of why the
+behavior was inappropriate. A public apology may be requested.
+
+### 2. Warning
+
+**Community Impact**: A violation through a single incident or series of
+actions.
+
+**Consequence**: A warning with consequences for continued behavior. No
+interaction with the people involved, including unsolicited interaction with
+those enforcing the Code of Conduct, for a specified period of time. This
+includes avoiding interactions in community spaces as well as external channels
+like social media. Violating these terms may lead to a temporary or permanent
+ban.
+
+### 3. Temporary Ban
+
+**Community Impact**: A serious violation of community standards, including
+sustained inappropriate behavior.
+
+**Consequence**: A temporary ban from any sort of interaction or public
+communication with the community for a specified period of time. No public or
+private interaction with the people involved, including unsolicited interaction
+with those enforcing the Code of Conduct, is allowed during this period.
+Violating these terms may lead to a permanent ban.
+
+### 4. Permanent Ban
+
+**Community Impact**: Demonstrating a pattern of violation of community
+standards, including sustained inappropriate behavior, harassment of an
+individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence**: A permanent ban from any sort of public interaction within the
+community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 2.1, available at
+[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
+
+Community Impact Guidelines were inspired by
+[Mozilla's code of conduct enforcement ladder][Mozilla CoC].
+
+For answers to common questions about this code of conduct, see the FAQ at
+[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at
+[https://www.contributor-covenant.org/translations][translations].
+
+[homepage]: https://www.contributor-covenant.org
+[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
+[Mozilla CoC]: https://github.com/mozilla/diversity
+[FAQ]: https://www.contributor-covenant.org/faq
+[translations]: https://www.contributor-covenant.org/translations

data/LICENSE

@@ -0,0 +1,31 @@
+BSD 2-Clause License
+
+Copyright (c) 2025-2026, Kotoshu contributors
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+------------------------------------------------------------------------------
+
+Bundled dictionaries and frequency lists carry their own licenses — see the
+per-language `license` files in https://github.com/kotoshu/dictionaries and
+the attribution file at the root of that repository.