kotoshu 0.6.0 → 0.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 503b4db65b4971cdc8854123771d9c1855af0de6f408f625d63219d5470c848c
-  data.tar.gz: caa57938fc7a76f3c0f7d026346db8e7f247db34b1a9b51d79ab2e9962668f57
+  metadata.gz: 48277511d3fd0b5244b98f54a43c1867991fd943092c9b78f657885addfa940e
+  data.tar.gz: 0434b01a14db2d334fceb53ced9003ac1378abf75db4386aa5397dfc5d853367
 SHA512:
-  metadata.gz: 9bb38cd338777586b8bf3c4700d36c5bd083b81f002fc9df0ee3ab667bde36a2a31b0dd8ebd9096543b57de8e151c18b9265df900306c7c71ae06cdc8e33825f
-  data.tar.gz: 6a11f2cbc34632d9533af0665308943063f06d952816e277fd1e995adfbbe84a56915693a6b9a9a53a4c64589ff8d09399c66ace9c7f7c657f865e4b0ef7adc4
+  metadata.gz: adb238648d03837cb4b0fc326c4038d3a40bb5cea968f6ffc49bd2a4bc2ef2614a42a1b79f4cd69e5efc279c0730878aa64885a2384f5ab9cca61c8ade7f096e
+  data.tar.gz: 3c1f20fd74572d1ae069767c78faad2fb37caffa3a051d8f6ca2355816f3e6d96fee23de275166b8307cd2622ac80d6982d4aaab35db2617c675e4766d705514

data/CHANGELOG.md

@@ -5,6 +5,46 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.7.0] — 2026-09-05
+
+The universal-kotoshu cut (plan 65's roadmap milestone): tiered
+models, the Rust core, and the native extension.
+
+### Added
+
+- **Model tiers** — `full` (~120 MB) / `fluency` (~15 MB, the new
+  default) / `mini` (~3 MB), resolved through the models repo's
+  registry (`kotoshu://models/{lang}/{tier}`) with sha256-verified
+  primary→mirror→vocab downloads; `KOTOSHU_MODEL_TIER`, `setup
+  LANG --model --tier`, and a tier-less legacy-cache bridge.
+- **Native extension** (`ext/kotoshu_native`, optional Rust
+  accelerator): `KOTOSHU_BACKEND=ruby|native|auto` (default ruby;
+  pure-Ruby installs unchanged), `Kotoshu::Native.available?`, and
+  `rake kotoshu:conformance:compare` — 2630 vectors, zero divergences
+  between the Ruby and Rust engines.
+- **Conformance export** — `rake kotoshu:conformance:export` freezes
+  engine behavior as golden vectors (2630 lines) consumed by
+  kotoshu-rs CI.
+- **Confidence cascade** — `semantic_cascade_threshold`
+  (`KOTOSHU_SEMANTIC_CASCADE_THRESHOLD`): skip the ONNX rerank when
+  the composite strategies are already confident.
+- Hunspell correctness: CHECKCOMPOUNDPATTERN replacement support,
+  dot-split casing, INITCAP ngram-root skip, AF alias flag-format
+  parsing, and a long tail of fixture fixes (integrational suite
+  176 examples, 0 failures, 1 documented pending).
+
+### Fixed
+
+- Model downloads cached 134-byte LFS pointer stubs whose own sha256
+  verified forever — the model source now fetches from the media host
+  and rejects stubs at download and verify.
+- `Embeddings::Vocabulary` crashed on the shipped wrapped vocab
+  format; shared normalizer added.
+- `Models::OnnxModel` used a nonexistent ONNX Runtime API and wrong
+  IO names — rewritten against `InferenceSession`.
+- CLI JSON output restored to the documented camelCase contract;
+  `DictionaryNotFoundError` mapping restored at the cache boundary.
+
 ## [0.5.0] — 2026-06-30
 
 Tier 2.5 release. Adds the personal-dictionary CLI, completes direct

data/CLAUDE.md

@@ -9,8 +9,25 @@ dictionary/affix backend (Hunspell-style) with ONNX-converted FastText word
 embeddings for context-aware suggestions. The README.adoc is the authoritative
 user-facing description; this file is the contributor-facing map.
 
-Key dependencies (`kotoshu.gemspec`): `thor` (CLI), `suika` (tokenizer),
-`onnxruntime` (semantic inference). Ruby 3.1+.
+Runtime deps (`kotoshu.gemspec`): `thor` (CLI), `rubyzip` (archive extraction during setup), `lutaml-model` (all model (de)serialization — see below). **Soft deps** (NOT in gemspec; `gem install kotoshu` works without them): `onnxruntime` (semantic / ONNX inference) and `suika` (Japanese morphological tokenization; its native ext `dartsclone` would otherwise break slim installs). Ruby 3.1+.
+
+## Serialization rule (load-bearing)
+
+All model classes use `lutaml-model` for serialization — declare attributes via `attribute :name, :type` and wire names via `mapping do ... end`. **Never hand-roll `to_h` / `from_h` / `to_json` / `from_json` / `serialize` / `deserialize`** on a model. Renames go in `mapping`, not in method bodies. This is a global rule (~/.claude/CLAUDE.md) — existing hand-rolled serialization is debt to migrate, not a pattern to extend.
+
+## Workspace: five repos under `kotoshu/`
+
+The parent directory `/Users/mulgogi/src/kotoshu/` is a multi-repo workspace — the gem is one piece. The other four are **content repos** fetched on demand by the cache layer, never vendored:
+
+| Sibling dir | Repo | Role |
+|---|---|---|
+| `kotoshu/` (this one) | `kotoshu/kotoshu` | The gem — Ruby API + CLI + analysis pipeline |
+| `dictionaries/` | `kotoshu/dictionaries` | Hunspell `.aff`/`.dic` for ~98 languages (download source for `LanguageCache`) |
+| `frequency-list-kelly/` | `kotoshu/frequency-list-kelly` | Kelly Project frequency tiers (download source for `FrequencyCache`) |
+| `models-fasttext-onnx/` | `kotoshu/models-fasttext-onnx` | FastText `.vec` → ONNX (download source for `ModelCache`) |
+| `kotoshu.github.io/` | `kotoshu/kotoshu.github.io` | Marketing/docs site |
+
+Each content repo has its own `TODO.impl/`. The vision and cross-repo plan index live in `TODO.impl/00-vision.md`.
 
 ## Development Commands
 
@@ -105,7 +122,7 @@ Three caches under `~/.cache/kotoshu/` (see `CACHE_ARCHITECTURE.md` for detail,
 |---|---|---|---|
 | `Cache::LanguageCache` | `~/.cache/kotoshu/languages/{code}/spelling/` | `github.com/kotoshu/dictionaries` | 7 days |
 | `Cache::FrequencyCache` | `~/.cache/kotoshu/frequency-lists/{code}/` | `github.com/kotoshu/frequency-list-kelly` | 7 days |
-| `Cache::ModelCache` | `~/.cache/kotoshu/models/{code}/...` | `github.com/kotoshu/models-fasttext-onnx` (FastText `.vec` → ONNX converted upstream) | 30 days |
+| `Cache::ModelCache` | `~/.cache/kotoshu/models/{code}/...` | `github.com/kotoshu/models-fasttext-onnx` (FastText `.vec` → ONNX converted upstream; tiers: full/fluency/mini, registry-resolved) | 30 days |
 
 `FrequencyCache` feeds `frequency_bonus` in `Suggestions::Strategies::EditDistanceStrategy` — high-frequency words get a ranking boost. The `kotoshu cache` subcommand exposes list/status/download/info/purge/clean operations.
 
@@ -170,7 +187,7 @@ When implementing features, study these alongside Kotoshu:
 
 ## Other Notes
 
-- License is **BSD-2-Clause** (not MIT — the README's "License" section is wrong).
+- License is **BSD-2-Clause** (per `kotoshu.gemspec` and `LICENSE`).
 - RBS signatures live in `sig/kotoshu.rbs` (the `sig/kotoshu/` subdirectory is empty). Update signatures when changing public APIs.
-- `scripts/` contains one-off utilities (FastText→ONNX conversion in Python, Kelly frequency parsing, diagnostics). `examples/` has numbered walkthrough scripts (`01_*.rb` … `07_*.rb`).
+- `scripts/` contains one-off utilities (FastText→ONNX conversion in Python, Kelly frequency parsing, diagnostics). `examples/` has numbered walkthrough scripts (`01_*.rb` … `14_*.rb`) plus per-language demo scripts and Spylls-comparison scripts.
 - Design history and superseded planning docs live in `docs/` (`architecture.md`, `cache-architecture.md`, `performance.md`, `plugins.md`, `getting-started.md`, plus integrated planning docs like `KOTOSHU_SOLIDIFICATION_PLAN.md`, `ARCHITECTURE_IMPROVEMENTS.md`, `TDD_ITERATION_STRATEGY.md`). Treat them as historical context, verify against current code before relying on them. `TODO.impl/` is the current source of truth for execution plans.

data/Cargo.lock

@@ -0,0 +1,291 @@
+# This file is automatically @generated by Cargo.
+# It is not intended for manual editing.
+version = 4
+
+[[package]]
+name = "aho-corasick"
+version = "1.1.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c982642fa9e8606056828ee9a8505737230110bb1099153c79efe865c59d12ba"
+dependencies = [
+ "memchr",
+]
+
+[[package]]
+name = "bindgen"
+version = "0.72.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "993776b509cfb49c750f11b8f07a46fa23e0a1386ffc01fb1e7d343efc387895"
+dependencies = [
+ "bitflags",
+ "cexpr",
+ "clang-sys",
+ "itertools",
+ "proc-macro2",
+ "quote",
+ "regex",
+ "rustc-hash",
+ "shlex",
+ "syn",
+]
+
+[[package]]
+name = "bitflags"
+version = "2.13.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da"
+
+[[package]]
+name = "cexpr"
+version = "0.6.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6fac387a98bb7c37292057cffc56d62ecb629900026402633ae9160df93a8766"
+dependencies = [
+ "nom",
+]
+
+[[package]]
+name = "cfg-if"
+version = "1.0.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801"
+
+[[package]]
+name = "clang-sys"
+version = "1.9.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "157a8ba7b480713b56f4c09fd13fc3e0a22a5dfab8097ba61cbc5feef950788a"
+dependencies = [
+ "glob",
+ "libc",
+ "libloading",
+]
+
+[[package]]
+name = "either"
+version = "1.18.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "252afb9ae5eaa683babdc6a068b3f5726eb19e05070c731f9b2a23a7c3e8ed34"
+
+[[package]]
+name = "glob"
+version = "0.3.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e4eba85ea1d0a966a983acd07deee566e67395d2d96b6fb39e62b5a833f1eb0b"
+
+[[package]]
+name = "itertools"
+version = "0.13.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "413ee7dfc52ee1a4949ceeb7dbc8a33f2d6c088194d9f922fb8318faf1f01186"
+dependencies = [
+ "either",
+]
+
+[[package]]
+name = "kotoshu"
+version = "0.1.0"
+source = "git+https://github.com/kotoshu/kotoshu-rs?branch=main#9941f1ea1a6157c57659e428f475959508d4d0b3"
+dependencies = [
+ "magnus",
+]
+
+[[package]]
+name = "kotoshu_native"
+version = "0.0.0"
+dependencies = [
+ "kotoshu",
+ "magnus",
+ "rb-sys",
+]
+
+[[package]]
+name = "lazy_static"
+version = "1.5.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe"
+
+[[package]]
+name = "libc"
+version = "0.2.189"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2"
+
+[[package]]
+name = "libloading"
+version = "0.8.9"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d7c4b02199fee7c5d21a5ae7d8cfa79a6ef5bb2fc834d6e9058e89c825efdc55"
+dependencies = [
+ "cfg-if",
+ "windows-link",
+]
+
+[[package]]
+name = "magnus"
+version = "0.8.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3b36a5b126bbe97eb0d02d07acfeb327036c6319fd816139a49824a83b7f9012"
+dependencies = [
+ "magnus-macros",
+ "rb-sys",
+ "rb-sys-env",
+ "seq-macro",
+]
+
+[[package]]
+name = "magnus-macros"
+version = "0.8.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "47607461fd8e1513cb4f2076c197d8092d921a1ea75bd08af97398f593751892"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "syn",
+]
+
+[[package]]
+name = "memchr"
+version = "2.8.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98"
+
+[[package]]
+name = "minimal-lexical"
+version = "0.2.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "68354c5c6bd36d73ff3feceb05efa59b6acb7626617f4962be322a825e61f79a"
+
+[[package]]
+name = "nom"
+version = "7.1.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d273983c5a657a70a3e8f2a01329822f3b8c8172b73826411a55751e404a0a4a"
+dependencies = [
+ "memchr",
+ "minimal-lexical",
+]
+
+[[package]]
+name = "proc-macro2"
+version = "1.0.107"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9"
+dependencies = [
+ "unicode-ident",
+]
+
+[[package]]
+name = "quote"
+version = "1.0.47"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001"
+dependencies = [
+ "proc-macro2",
+]
+
+[[package]]
+name = "rb-sys"
+version = "0.9.130"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "02faf625bb10ba893e3ae620f19c9fb1b5f8fcae0fe4eb86bb3f2230fad75edb"
+dependencies = [
+ "rb-sys-build",
+]
+
+[[package]]
+name = "rb-sys-build"
+version = "0.9.130"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "05be6f9c86fe5482808162826f6c047d093b3670582296c770de1d94f8066694"
+dependencies = [
+ "bindgen",
+ "lazy_static",
+ "proc-macro2",
+ "quote",
+ "regex",
+ "shell-words",
+ "syn",
+]
+
+[[package]]
+name = "rb-sys-env"
+version = "0.2.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "cca7ad6a7e21e72151d56fe2495a259b5670e204c3adac41ee7ef676ea08117a"
+
+[[package]]
+name = "regex"
+version = "1.13.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f020237b6c8eed93db2e2cb53c00c60a8e1bc73da7d073199a1180401450218d"
+dependencies = [
+ "aho-corasick",
+ "memchr",
+ "regex-automata",
+ "regex-syntax",
+]
+
+[[package]]
+name = "regex-automata"
+version = "0.4.18"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ad8553b9b26413251cbf30e620595c7a41b3887f03da04579c0e6b0d6a06b4b2"
+dependencies = [
+ "aho-corasick",
+ "memchr",
+ "regex-syntax",
+]
+
+[[package]]
+name = "regex-syntax"
+version = "0.8.11"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4"
+
+[[package]]
+name = "rustc-hash"
+version = "2.1.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6b1e7f9a428571be2dc5bc0505c13fb6bf936822b894ec87abf8a08a4e51742d"
+
+[[package]]
+name = "seq-macro"
+version = "0.3.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1bc711410fbe7399f390ca1c3b60ad0f53f80e95c5eb935e52268a0e2cd49acc"
+
+[[package]]
+name = "shell-words"
+version = "1.1.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "dc6fe69c597f9c37bfeeeeeb33da3530379845f10be461a66d16d03eca2ded77"
+
+[[package]]
+name = "shlex"
+version = "1.3.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0fda2ff0d084019ba4d7c6f371c95d8fd75ce3524c3cb8fb653a3023f6323e64"
+
+[[package]]
+name = "syn"
+version = "2.0.119"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "unicode-ident",
+]
+
+[[package]]
+name = "unicode-ident"
+version = "1.0.24"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75"
+
+[[package]]
+name = "windows-link"
+version = "0.2.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"

data/Cargo.toml

@@ -0,0 +1,15 @@
+# Kotoshu gem Rust workspace (plan 66 / P4b) — mirrors parsanol-ruby's
+# root Cargo.toml: rb_sys's compile task runs `cargo metadata` from the gem
+# root, so the extension crate must be reachable through a root manifest.
+[workspace]
+resolver = "2"
+members = ["ext/kotoshu_native"]
+
+# Release profile follows the kotoshu-rs workspace policy (plan 66): fat
+# LTO and single codegen unit so the shipped accelerator is as fast as the
+# core's own optimized builds. Local development builds use the dev profile
+# via RB_SYS_CARGO_PROFILE (see ext/kotoshu_native/extconf.rb).
+[profile.release]
+opt-level = 3
+lto = "fat"
+codegen-units = 1

data/README.adoc

@@ -171,6 +171,50 @@ analyzer.suggest_corrections("helo", context: "I said helo to the world").map(&:
 NOTE: The semantic path requires the optional `onnxruntime` gem. See
 <<_requirements,Requirements>>.
 
+[[confidence-cascade]]
+=== Confidence cascade
+
+When the traditional (dictionary + edit distance) strategies are already
+confident about a word, the semantic rerank adds latency and model load
+without changing the answer. The confidence cascade formalizes the
+hybrid split: dictionary verdict first, ONNX rerank only for uncertain
+candidates.
+
+The `semantic_cascade_threshold` configuration option controls it:
+
+[cols="1,3"]
+|===
+|Threshold |Behavior
+
+|`1.0` (default)
+|Always rerank — identical to the classic pipeline. This is the sentinel:
+the composite can legitimately score exactly `1.0`, so `1.0` never skips.
+
+|`0.9`
+|Rerank only when the top traditional candidate scores below `0.9`;
+otherwise skip the ONNX rerank for that word.
+
+|`0.0`
+|Never rerank — the semantic path is bypassed entirely.
+|===
+
+[source,bash]
+----
+# Tune via environment variable
+KOTOSHU_SEMANTIC_CASCADE_THRESHOLD=0.9 kotoshu check document.txt
+
+# Or via the Ruby API
+Kotoshu.configuration.semantic_cascade_threshold = 0.9
+----
+
+Each skip is counted in the metrics registry
+(`Kotoshu::Metrics.stats[:semantic_cascade_skips]`, also exported in
+StatsD and Prometheus formats) and logged under debug/verbose output.
+
+Tuning guidance will come from the eval harness once the corpus
+benchmark lands; until then, `0.8`–`0.9` is a reasonable range for
+typo-heavy text.
+
 [[multi-language]]
 == Multi-Language Support
 
@@ -276,6 +320,78 @@ https://github.com/kotoshu/models-fasttext-onnx[`kotoshu/models-fasttext-onnx']
 repo. The CLI downloads pre-converted artifacts; users do not run
 conversion locally.
 
+[[model-tiers]]
+=== Model tiers
+
+ONNX models come in three tiers, published by
+https://github.com/kotoshu/models-fasttext-onnx[models-fasttext-onnx]
+in a resource registry under stable ids (`kotoshu://models/{lang}/{tier}`):
+
+[cols="1,2,3"]
+|===
+|Tier |Approx. size |What it is
+
+|`full`
+|~120 MB
+|Unquantized, ~100k-word vocabulary. The same artifact and cache layout
+as before tiers existed.
+
+|`fluency`
+|~15-18 MB
+|int8-quantized, reduced vocabulary (~50k-60k words).
+
+|`mini`
+|~3 MB
+|int8-quantized, 10k-word vocabulary.
+|===
+
+Tiered downloads are verified: the model bytes are fetched from the
+registry entry's primary URL, falling back to its mirror, with the vocab
+sibling downloaded alongside; the SHA-256 is checked against the
+registry entry before the model is accepted into the cache.
+
+.Picking a tier
+[example]
+====
+[source,ruby]
+----
+# One-time setup at a specific tier
+Kotoshu.setup(:en, want: %i[model], tier: :fluency)
+
+# Resolve stays cache-only: the requested tier must already be cached,
+# otherwise ResourceNotSetupError is raised (no download, no fallback
+# to another tier)
+Kotoshu.resolve(language: :en, want: %i[spelling model], tier: :fluency)
+
+# "Is this tier cached?" (tier: :any asks whether any tier is cached)
+Kotoshu.setup?(:en, resource: :model, tier: :mini)
+----
+
+[source,bash]
+----
+kotoshu setup en --model --tier mini
+----
+====
+
+`tier: :any` on resolve maps to the *single* cached tier. It resolves
+when exactly one tier is cached and raises otherwise: with no tier
+cached it raises `ResourceNotSetupError`, and with several it raises
+`Kotoshu::ResourceResolutionError` listing them. That listing uses the
+fixed order `mini`, `fluency`, `full` — a reporting order for
+deterministic error messages, not a fallback chain; tiers are never
+silently substituted for one another.
+
+The default tier is `fluency` (~15 MB, near-lossless; owner decision
+2026-09-04) — caches written under the old `full` default keep working
+via the tier-less legacy bridge. Override it per call with `tier:`, or set
+it globally via the `model_tier` configuration option (environment
+variable `KOTOSHU_MODEL_TIER`; one of `full`, `fluency`, `mini`):
+
+[source,bash]
+----
+KOTOSHU_MODEL_TIER=mini kotoshu setup en --model
+----
+
 [[interactive-mode]]
 [[interactive-mode]]
 == Interactive Mode

data/Rakefile

@@ -10,3 +10,106 @@ require "rubocop/rake_task"
 RuboCop::RakeTask.new
 
 task default: %i[spec rubocop]
+
+# Conformance failure reporters (kept as lambdas so nothing leaks into
+# Object from the Rakefile).
+report_failures = lambda do |failures, label = nil|
+  failures.first(10).each do |failure|
+    prefix = label ? "[#{label}] " : ""
+    puts "#{prefix}##{failure.index} #{failure.kind} #{failure.input.inspect} " \
+         "on #{failure.dictionary}: expected #{failure.expected.inspect}, got #{failure.actual.inspect}"
+  end
+  puts "... and #{failures.size - 10} more" if failures.size > 10
+end
+
+report_divergences = lambda do |divergences|
+  divergences.first(10).each do |divergence|
+    puts "[divergence] ##{divergence.index} #{divergence.kind} #{divergence.input.inspect} " \
+         "on #{divergence.dictionary}: ruby #{divergence.expected.inspect}, " \
+         "native #{divergence.actual.inspect}"
+  end
+  puts "... and #{divergences.size - 10} more" if divergences.size > 10
+end
+
+namespace :kotoshu do
+  namespace :conformance do
+    desc <<~DESC
+      Export conformance vectors for kotoshu-rs (plan 67 M3). Walks the Spylls \
+      fixtures (spec/integrational/fixtures) plus the spec fixture corpus, runs \
+      the real engine over every fixture word, and writes one JSON object per \
+      line. `expected` values are FROZEN engine behavior (byte-reproducible \
+      cross-implementation), not claims about linguistic correctness. \
+      Output path: ENV["CONFORMANCE_VECTORS_PATH"] or conformance/vectors.jsonl \
+      (repo root). The generated file IS committed -- conformance/ is \
+      deliberately NOT gitignored -- and kotoshu-rs copies it into \
+      tests/conformance/ via its documented sync step so its CI needs no gem.
+    DESC
+    task :export do
+      path = ENV.fetch("CONFORMANCE_VECTORS_PATH", nil) || "conformance/vectors.jsonl"
+      require "kotoshu"
+      result = Kotoshu::ConformanceExporter.export(path: path)
+      puts "Wrote #{result.row_count} vectors from #{result.corpus_count} corpora to #{result.path}"
+      unless result.skipped_corpora.empty?
+        puts "Skipped corpora:"
+        result.skipped_corpora.each { |id, reason| puts "  #{id}: #{reason}" }
+      end
+      result.skipped_words.each { |(id, word), reason| puts "  Skipped word #{id} #{word.inspect}: #{reason}" }
+    end
+
+    desc <<~DESC
+      Replay conformance/vectors.jsonl through the pure-Ruby engine \
+      (plan 66 dual-backend suite). Every vector must reproduce its frozen \
+      expectation; exits nonzero on any mismatch.
+    DESC
+    task :ruby do
+      require "kotoshu"
+      result = Kotoshu::ConformanceRunner.new.run(backend: :ruby)
+      puts "ruby backend: #{result.row_count} vectors, #{result.failures.size} failures"
+      report_failures.call(result.failures)
+      abort "kotoshu:conformance:ruby: #{result.failures.size} of #{result.row_count} vectors diverged" unless result.ok?
+    end
+
+    desc <<~DESC
+      Replay conformance/vectors.jsonl through the native (Rust) engine \
+      (plan 66 dual-backend suite). Requires the extension to be built \
+      (rake compile); exits nonzero on any mismatch.
+    DESC
+    task :native do
+      require "kotoshu"
+      unless Kotoshu::Native.available?
+        abort "kotoshu:conformance:native requires the native extension -- run `rake compile` first"
+      end
+
+      result = Kotoshu::ConformanceRunner.new.run(backend: :native)
+      puts "native backend: #{result.row_count} vectors, #{result.failures.size} failures"
+      report_failures.call(result.failures)
+      abort "kotoshu:conformance:native: #{result.failures.size} of #{result.row_count} vectors diverged" unless result.ok?
+    end
+
+    desc <<~DESC
+      Run the conformance vectors through BOTH backends and diff every row \
+      (plan 66 acceptance: zero diffs). Also fails when either backend \
+      diverges from the frozen expectations. Requires the extension \
+      (rake compile).
+    DESC
+    task :compare do
+      require "kotoshu"
+      unless Kotoshu::Native.available?
+        abort "kotoshu:conformance:compare requires the native extension -- run `rake compile` first"
+      end
+
+      result = Kotoshu::ConformanceRunner.new.compare
+      puts "compare: #{result.ruby.row_count} vectors -- ruby failures: " \
+           "#{result.ruby.failures.size}, native failures: #{result.native.failures.size}, " \
+           "backend divergences: #{result.divergences.size}"
+      report_failures.call(result.ruby.failures, "ruby")
+      report_failures.call(result.native.failures, "native")
+      report_divergences.call(result.divergences)
+      unless result.ok?
+        abort "kotoshu:conformance:compare: backends diverged " \
+              "(ruby #{result.ruby.failures.size}, native #{result.native.failures.size}, " \
+              "cross #{result.divergences.size} of #{result.ruby.row_count})"
+      end
+    end
+  end
+end