kotoshu 1.0.0 → 1.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 19518c2daced6ae79d4a6d02fc5a26f4c3ca7109a70d2210220893177312173d
4
- data.tar.gz: 34f597de8432648f22a15a0ad1b09f71041eaff805c91bc9a1357b6f0e2aff56
3
+ metadata.gz: fbf9efcde52fd9741e0453ff9c8e58117ec4ed1d2c8bddf6c00f07851c7dfdbe
4
+ data.tar.gz: 838132ed1c7b305e1004b29b747024239dcb9c347e7eab0566c6cedbb839752f
5
5
  SHA512:
6
- metadata.gz: ee21f0075b02b6aff17cd4e58262bdf0ad6356a69c1950be2493ee6a801970f5268769a35899fd6f651f27d0d50cee591ffedadf08c922f0623ecbdb3fb4bcc3
7
- data.tar.gz: 34c321c65532ff5a175f0b535225bdda8937e7d72f537e6462d0e965ed6a5db3524ec63a65ecae34af2df9691d28285bfb14048ddda90483f28e5a438c3199da
6
+ metadata.gz: caf90e1d64a196f6d866354cc811ee710302e924d9ecae40e0d216f06ae3f1b214e7637dd585a9d44855685109655aca67c6b1910508db76e3d78bdd537adb77
7
+ data.tar.gz: 290bbfbeb0d6da02733d617690dcb3cb4a30e850b183287c6b371e160b73b5c1f0af81c83eb6419999382150d2f7ba67d522381429239b55f4a4420d7311d9f4
@@ -1,9 +1,17 @@
1
1
  # Plan 117 — Deterministic frequency tiers: stop the silent YAML fallback
2
2
 
3
- Status: proposed (forensics complete; root cause proven)
3
+ Status: executed (gem PR #178, 2026-09-10)
4
+
5
+ Acceptance met: with the cache backdated past the TTL the compare is
6
+ 2630/0/0 and suggest("ba") on the compoundrule fixtures returns the
7
+ frozen expectations exactly. conformance.yml gates the compare on
8
+ every push and PR; its first green run landed on the PR itself.
9
+ Suite 4007/0, rubocop clean. The checked-in-fixture variant of the
10
+ hermetic runner (design item 2) was not needed — the CI job seeds the
11
+ pinned cache instead; revisit only if the job proves flaky.
12
+
4
13
  Depends on: plan 82 (conformance runner), the 2,630-vector contract (plan 66+)
5
- Blocks: the 1.0 train (audit checklist item 7 — "conformance vectors green at
6
- the cut commit" — is not currently verifiable on a cache-cold machine)
14
+ Blocks: nothing — audit checklist item 7 is satisfied and CI-gated
7
15
 
8
16
  ## What happened (forensics, 2026-09-10)
9
17
 
@@ -0,0 +1,29 @@
1
+ # Plan 118 — kotoshu-server 1.0 must ride kotoshu 1.0
2
+
3
+ Status: pending (P0 — broken release constraint)
4
+ Priority: P0 (a shipped 1.0.0 release cannot see the 1.0 gem)
5
+
6
+ ## Problem
7
+
8
+ `kotoshu-server.gemspec` declares `spec.add_runtime_dependency "kotoshu",
9
+ "~> 0.6"`. Ruby's pessimistic operator makes that `>= 0.6, < 1.0` — so
10
+ the server 1.0.0 released in the 1.0 train resolves **kotoshu 0.11.1**,
11
+ not 1.0.0. Every fresh `gem install kotoshu-server` today deploys the
12
+ pre-freeze engine under a 1.0 label. The floor was never revisited as
13
+ the gem moved 0.6 → 1.0.
14
+
15
+ ## Fix
16
+
17
+ - `spec.add_runtime_dependency "kotoshu", "~> 1.0"` — the server's
18
+ public surface consumes stable gem API only (two-stage setup, check
19
+ facade, semantic cascade), all frozen at 1.0.
20
+ - Release kotoshu-server 1.0.1 carrying the floor bump.
21
+ - Guard: a spec that parses the gemspec and asserts the runtime floor
22
+ does not exclude the current kotoshu major (the class of bug is
23
+ silent by construction).
24
+
25
+ ## Acceptance
26
+
27
+ - `gem dependency kotoshu-server -v 1.0.1` shows kotoshu ~> 1.0.
28
+ - A fresh resolve (`gem install kotoshu-server --version 1.0.1` in a
29
+ scratch GEM_HOME) picks kotoshu 1.0.x.
@@ -0,0 +1,50 @@
1
+ # Plan 119 — Embed the frozen Kelly tiers in the gem
2
+
3
+ Status: pending
4
+ Depends on: plan 117 (the dataset-divergence forensics)
5
+
6
+ ## Problem
7
+
8
+ Plan 117 made suggestion ranking deterministic for machines with
9
+ frequency data, but a cache-cold machine still falls back to
10
+ `lib/kotoshu/data/common_words/en.yml` — a *differently curated*
11
+ dataset (top_50 holds `a`; Kelly holds `a` in top_1000). The Rust
12
+ engine embeds the frozen Kelly tables (`kotoshu-rs
13
+ kotoshu/src/suggest/frequency_data.rs`, en.json sha256 97535823f41aa3f0…)
14
+ as its *only* source; the Ruby engine downloads them. Two engines, two
15
+ data paths, one contract — the fallback path is still
16
+ contract-divergent by construction.
17
+
18
+ ## Design (single source of truth, model-driven)
19
+
20
+ - Generate `lib/kotoshu/data/frozen_kelly/en.rb` FROM the rs tables
21
+ (script: `scripts/generate_frozen_tiers.rb`, documented provenance:
22
+ the rs file path + the en.json sha). Content: three sorted word
23
+ arrays, sizes 47/185/907, byte-identical membership to the rs
24
+ `TOP_50`/`TOP_200`/`TOP_1000`.
25
+ - `Suggestions::FrozenTiers.tiers_for(language_code)` — returns the
26
+ Set-shaped tiers for `en`, nil otherwise. Pure, no IO.
27
+ - `FrequencyProvider#load` fallback chain becomes:
28
+ cache (any present data, plan 117) → **frozen embedded (en)** →
29
+ local YAML → empty. For `en` the YAML is thereby unreachable; for
30
+ every other language nothing changes (rs embeds en only — MECE with
31
+ the Rust side).
32
+ - Cache-cold machines now produce contract-exact ranking for `en`
33
+ with zero network, zero setup — determinism by construction instead
34
+ of by CI seeding.
35
+
36
+ ## Acceptance
37
+
38
+ - `XDG_CACHE_HOME=<empty>` conformance compare: 2630/0/0 (today the
39
+ empty-cache run fails 16 — the CI seed papers over it).
40
+ - Parity spec: embedded tiers == the tiers `FrequencyCache` returns
41
+ for a checksummed en download (structural: sizes + sampled words +
42
+ `a` in top_1000 / `the` in top_50).
43
+ - Full suite green; no behavior change for cache-present machines.
44
+
45
+ ## Non-goals
46
+
47
+ - No YAML deletion (other languages still use it; `en.yml` stays for
48
+ now as a documented dead fallback — removal is a 1.x deprecation
49
+ decision).
50
+ - No registry or format changes.
@@ -0,0 +1,35 @@
1
+ # Plan 120 — setup refreshes cached_at when a download is skipped
2
+
3
+ Status: resolved as obsolete (2026-09-11, code inspection during execution)
4
+
5
+ `setup_frequency_remote` (resource_manager.rb) has no checksum-skip
6
+ path: `return :cached if cache.available? && !force` — expiry makes
7
+ `available?` false — `else cache.get` → `download` → `download_resource`
8
+ ALWAYS writes fresh metadata (cached_at = now). The observed
9
+ "downloaded but stale timestamp" was plan 117's writer/reader path
10
+ split in full; no second mechanism exists. Verified by walkthrough of
11
+ get/setup_frequency_remote/download_resource; no code change needed.
12
+
13
+ ## Problem
14
+
15
+ `Kotoshu.setup(:en, want: [:frequency])` can skip the download when
16
+ the checksum of the fetched bytes matches the cached file, but the
17
+ skip path historically left `cached_at` untouched (observed 2026-09-10:
18
+ setup reported `frequency: :downloaded` while the reader-side metadata
19
+ still said 2026-09-02). An unrefreshed timestamp guarantees the next
20
+ TTL check fails and the resource is re-fetched for nothing — and any
21
+ reader still consulting TTL sees stale.
22
+
23
+ ## Fix
24
+
25
+ - On a checksum-matching skip: rewrite the metadata with a fresh
26
+ `cached_at` (same checksum, same url) and report `:cached`, not
27
+ `:downloaded`. The report must describe what actually happened.
28
+ - Spec: backdate metadata, run setup with a stubbed fetcher (no
29
+ network), assert `cached_at` moved and the result says `:cached`.
30
+
31
+ ## Acceptance
32
+
33
+ - Second consecutive `setup` run performs no re-download (checksum
34
+ match) yet leaves a TTL-valid metadata file.
35
+ - Honest SetupResult reporting.
@@ -0,0 +1,27 @@
1
+ # Plan 121 — Jekyll generator skips the sweep for baseline-covered words
2
+
3
+ Status: pending
4
+ Depends on: plan 116 (the suggestions_filter mechanism)
5
+
6
+ ## Problem
7
+
8
+ The rake task and the CLI thread `suggestions_filter:` through their
9
+ check callables; the Jekyll generator (plan 89) has its own baseline
10
+ store (`lib/kotoshu/jekyll.rb`) but still calls the checker without
11
+ the filter — Jekyll builds pay the full suggestion sweep for entries
12
+ their baseline then suppresses. Same repository, same mechanism,
13
+ three call sites, two wired.
14
+
15
+ ## Fix
16
+
17
+ - Build the per-file budget filter via `store.suggestions_filter_for`
18
+ exactly as `Cli::DirectoryCheck` does, pass it through the check
19
+ call. The generator's baseline is a `Baseline::Store` already — the
20
+ filter comes from the same object (DRY by construction).
21
+ - Spec: a Jekyll site fixture with a baseline; assert suppressed
22
+ entries carry no suggestions and a new error does.
23
+
24
+ ## Acceptance
25
+
26
+ - Generator output unchanged; sweep cost on baselined sites drops as
27
+ it did for the rake task (plan 116 evidence).
@@ -0,0 +1,50 @@
1
+ # Plan 122 — KPK2: compressed pack sections
2
+
3
+ Status: designed (execution-ready; next session — multi-repo pipeline)
4
+ Depends on: plan 113 (KPK1), the 2026-09-10 model-efficiency research
5
+
6
+ ## Problem
7
+
8
+ KPK1 packs ship raw sections: en-1.5.0.bin is 15.2 MB over the wire
9
+ (aff+dic text ~1.2 MB, int8 ONNX ~12 MB, vocab JSON, buckets). The
10
+ text/JSON sections compress ~5-10x; the int8 weights ~1.2-1.5x. The
11
+ playground's first-load cost is dominated by bytes that decompress in
12
+ microseconds — costs are being paid in the expensive dimension
13
+ (bandwidth) instead of the cheap one (CPU). Same pattern as
14
+ DeepSeek-V4.1-Flash storing KV in FP4 and dequantizing before
15
+ attention: compress storage, not compute.
16
+
17
+ ## Design
18
+
19
+ - KPK2 container: same section framing as KPK1 (magic+count, framed
20
+ sections, sha256 footers) with (a) a new magic version byte, (b)
21
+ each section optionally zstd-frame-compressed, flagged by a new tag
22
+ bit; uncompressed sections remain valid — readers negotiate on the
23
+ magic, never on heuristics.
24
+ - Engine side (`kotoshu-rs`): `ruzstd` (pure-Rust decoder, no C
25
+ dependency — wasm-safe). Decompress per section before sha verify?
26
+ **No**: keep the footer sha over the COMPRESSED bytes so corruption
27
+ detection happens before decompression; the KPK1 golden-sha parity
28
+ test gains KPK2 twins.
29
+ - Builder side (`models-fasttext-onnx`): pack builder emits KPK2 with
30
+ zstd level 19 for text/JSON sections, raw-or-zstd for int8 weights
31
+ (measure; ship whichever wins per section).
32
+ - Registry: `kotoshu://packs/{lang}` entries gain the new artifacts;
33
+ bump registry to v1.7.0 per plan 05 policy; mirrors must serve both
34
+ generations during the rollout window.
35
+ - Consumers: wasm `loadPack` (versioned by magic), worker pack mode,
36
+ playground pin bump, site caching-docs truth.
37
+
38
+ ## Acceptance
39
+
40
+ - en pack wire size measurably smaller (target: ≤ 60% of KPK1;
41
+ report actual), loadPack wall time within +10% of KPK1 (143 ms
42
+ baseline on M-series).
43
+ - Conformance: KPK1 packs still load unchanged (back-compat gate).
44
+ - Registry v1.7.0 released with both generations mirrored.
45
+
46
+ ## Execution order (next session)
47
+
48
+ models builder → rs engine + wasm 1.1.0 → registry v1.7.0 →
49
+ worker/playground flip → site docs+news. The LFS upload pipeline is
50
+ the long pole (historically ~1h with CDN throttles) — start it first.
@@ -0,0 +1,41 @@
1
+ # Plan 123 — Verifiable typo corpora, synthesized at scale
2
+
3
+ Status: designed (execution-ready; next session — models-repo arc)
4
+ Depends on: plan 115 (hybrid pricing blocked on real de/es corpora);
5
+ validated direction: DeepSeek-V4.1-Flash §5.1 — data/environment
6
+ synthesis exceeds algorithmic novelty
7
+
8
+ ## Problem
9
+
10
+ Plan 114's surviving thread (C-retrieve + fastText-rescore hybrid)
11
+ beat the full tier on all four real components but was parked for lack
12
+ of real de/es typo corpora; plan 115 priced the thread and named the
13
+ corpus as the blocker. Meanwhile the eval harness already owns the
14
+ generators: `eval/noise.py` (keyboard-aware grids, Turkish-Q/JCUKEN/
15
+ Greek, parameterized Latin family) and the per-language dictionaries.
16
+
17
+ ## Design
18
+
19
+ - Formalize a corpus entry as (typo, correction, provenance) — the
20
+ verification system is the dictionary itself plus the noise model:
21
+ a pair is admitted iff the correction is dictionary-valid, the typo
22
+ is not, and the typo is reachable from the correction by the
23
+ declared noise operation(s). Difficulty calibration = the noise
24
+ op mix and edit distance distribution, reported per corpus.
25
+ - Scale: ≥ 5k pairs per language for de and es (matching the en
26
+ frozen benchmark's 2,509 real pairs at minimum, target 2x), plus
27
+ pt/fr as stretch; dedup by (typo, correction); stratify by distance
28
+ 1/2 and by keyboard-adjacency vs random substitution.
29
+ - Honest split: synthesis trains/reprices, it must NOT be presented
30
+ as "real user" data — corpora carry `synthetic: true` provenance
31
+ and the plan-115 pricing reruns on both the frozen real components
32
+ (unchanged) and the synthetic corpora (new).
33
+ - Gates: the hybrid must still win on real components; synthetic
34
+ corpora may only ADD evidence, never replace the real gates
35
+ (pre-declared rule, plan 114 discipline).
36
+
37
+ ## Acceptance
38
+
39
+ - de/es corpora checked in with provenance + difficulty histograms.
40
+ - Plan 115 pricing rerun: hybrid vs full tier on real (unchanged) +
41
+ synthetic (new) — ship-only-if-wins applies to the union.
@@ -0,0 +1,19 @@
1
+ # Plan 124 — Owner-gated residue (inventory, not executable by agents)
2
+
3
+ Status: standing inventory (updated 2026-09-10)
4
+
5
+ - VS Code Marketplace + Open VSX PATs → publish kotoshu-vscode
6
+ (extension is publish-ready; owner: "later").
7
+ - Worktree cleanup — ASK OWNER before removing anything:
8
+ gem `.claude/worktrees/*` (10+ campaign worktrees), kotoshu-rs
9
+ `.claude/worktrees/packs` (content already merged), and /tmp
10
+ worktrees from the 1.0 day: fix-baseline, gem-dogfood, plan-116,
11
+ impl-116, impl-117, conf-0110, conf-0100, conf-092, conf-4597,
12
+ wasm-10, plans-118.
13
+ - kotoshu-lsp: still 0.1.1 — joins a future train if the owner wants
14
+ LSP on the 1.x line (audit train deliberately excluded it).
15
+ - Docker/kotoshu-ci + kotoshu-go/kotoshu-python floors: verify their
16
+ gem/base pins resolve 1.x on their next cuts (same class of bug as
17
+ plan 118 — check before, not after).
18
+ - rubygems 0.1.0 empties (lsp/server): support@rubygems.org deletion
19
+ remains the only path; cosmetic, owner call.