kotoshu 1.0.1 → 1.0.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/TODO.impl/118-server-gem-floor.md +5 -1
- data/TODO.impl/119-embedded-kelly-tiers.md +7 -1
- data/TODO.impl/121-jekyll-baseline-skip.md +7 -1
- data/TODO.impl/122-pack-compression-kpk2.md +17 -1
- data/TODO.impl/123-typo-corpus-synthesis.md +15 -1
- data/TODO.impl/124-owner-residue.md +15 -6
- data/TODO.impl/125-consumer-floor-guards.md +43 -0
- data/TODO.impl/126-tier-error-budgets.md +33 -0
- data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
- data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
- data/TODO.impl/129-pt-fr-corpora.md +23 -0
- data/TODO.impl/130-news-truth-0912.md +23 -0
- data/TODO.impl/131-hybrid-productization.md +57 -0
- data/TODO.impl/132-techniques-page-and-prose.md +27 -0
- data/TODO.impl/133-rust-first-distribution.md +90 -0
- data/TODO.impl/134-post-131-improvements.md +67 -0
- data/TODO.impl/54-cjk-support.md +5 -2
- data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +29 -0
- data/lib/kotoshu/cache/model_cache.rb +159 -10
- data/lib/kotoshu/cli.rb +13 -1
- data/lib/kotoshu/configuration.rb +23 -3
- data/lib/kotoshu/resource_manager.rb +47 -0
- data/lib/kotoshu/spellchecker.rb +32 -2
- data/lib/kotoshu/suggestions/strategies/typo_retrieval_strategy.rb +55 -0
- data/lib/kotoshu/suggestions/typo_merge.rb +26 -0
- data/lib/kotoshu/suggestions.rb +2 -0
- data/lib/kotoshu/typo/engine.rb +125 -0
- data/lib/kotoshu/typo.rb +9 -0
- data/lib/kotoshu/version.rb +1 -1
- data/lib/kotoshu.rb +1 -0
- data/scripts/build_native_gem.rb +45 -0
- metadata +17 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 3722ff8a55645f8a8a6f08bc7c3a34427dcd1167f94c4c07831065db01684e5c
|
|
4
|
+
data.tar.gz: 659ac873d5f35c26a3f9a8775860c5c6f31dfcade3015703d1d4eadc824f712a
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 1821c5433b271031b199d70b8b748affb4332f12dba6b27c23d412abf8958cfbf6d71b9a74950ca09c52a3f8ef299eca2238aee3d86ae37e1ca6f595c3306d2e
|
|
7
|
+
data.tar.gz: b0fc00ab0fef49a8c1d3d46b81daa14c9ea896988f006d2c583a512f50bc2a5a81ce3d99cdcfd82bf060f6deabaf981e814140a994c9273b21eddb6c20aff54e
|
|
@@ -1,6 +1,10 @@
|
|
|
1
1
|
# Plan 118 — kotoshu-server 1.0 must ride kotoshu 1.0
|
|
2
2
|
|
|
3
|
-
Status:
|
|
3
|
+
Status: executed (kotoshu-server PR #8 + release 1.0.1, 2026-09-11)
|
|
4
|
+
|
|
5
|
+
gemspec + Gemfile + lock floor ~> 1.0; guard spec added; server 1.0.1
|
|
6
|
+
live on RubyGems (verified). The suite ran green on kotoshu 1.0.0
|
|
7
|
+
(25/0) - the 0.6-era engine had been serving this repo's own CI.
|
|
4
8
|
Priority: P0 (a shipped 1.0.0 release cannot see the 1.0 gem)
|
|
5
9
|
|
|
6
10
|
## Problem
|
|
@@ -1,6 +1,12 @@
|
|
|
1
1
|
# Plan 119 — Embed the frozen Kelly tiers in the gem
|
|
2
2
|
|
|
3
|
-
Status:
|
|
3
|
+
Status: executed (gem PR #181, 2026-09-11)
|
|
4
|
+
|
|
5
|
+
Acceptance exceeded: XDG_CACHE_HOME=<empty> compare is 2630/0/0 -
|
|
6
|
+
determinism by construction, no CI seeding needed. Embedded tiers are
|
|
7
|
+
Set-equal to the checksummed cache tiers; data ships as JSON generated
|
|
8
|
+
by scripts/generate_frozen_tiers.rb (provenance + sha inside the
|
|
9
|
+
file); other languages unchanged. Full suite 4010/0, rubocop clean.
|
|
4
10
|
Depends on: plan 117 (the dataset-divergence forensics)
|
|
5
11
|
|
|
6
12
|
## Problem
|
|
@@ -1,6 +1,12 @@
|
|
|
1
1
|
# Plan 121 — Jekyll generator skips the sweep for baseline-covered words
|
|
2
2
|
|
|
3
|
-
Status:
|
|
3
|
+
Status: executed (gem PR #181, 2026-09-11)
|
|
4
|
+
|
|
5
|
+
Third call site wired: the generator builds the per-file budget from
|
|
6
|
+
its own baseline store and threads suggestions_filter through
|
|
7
|
+
Kotoshu.check. Generator spec proves a beyond-budget occurrence
|
|
8
|
+
surfaces with suggestions while covered words never pay for the
|
|
9
|
+
sweep.
|
|
4
10
|
Depends on: plan 116 (the suggestions_filter mechanism)
|
|
5
11
|
|
|
6
12
|
## Problem
|
|
@@ -1,6 +1,22 @@
|
|
|
1
1
|
# Plan 122 — KPK2: compressed pack sections
|
|
2
2
|
|
|
3
|
-
Status:
|
|
3
|
+
Status: executed as REJECTED on the pre-declared gate (2026-09-12 measurement)
|
|
4
|
+
|
|
5
|
+
zstd-19 per-section, en-1.5.0.bin (15,173,525 bytes):
|
|
6
|
+
|
|
7
|
+
aff 3,086 -> 908 (3.40x)
|
|
8
|
+
dic 551,762 -> 169,700 (3.25x)
|
|
9
|
+
model 3,040,752 -> 2,769,252 (1.10x)
|
|
10
|
+
vocab 202,597 -> 59,342 (3.41x)
|
|
11
|
+
buckets 11,375,135 -> 9,854,872 (1.15x)
|
|
12
|
+
total -> 12,854,074 (84.7% of KPK1)
|
|
13
|
+
|
|
14
|
+
The acceptance gate was <= 60%. The pack is 75% int8 bucket table and
|
|
15
|
+
20% int8 ONNX weights — nearly incompressible; the sections that win
|
|
16
|
+
3.4x are 5% of the bytes. A 15% wire saving does not pay for a format
|
|
17
|
+
generation, a decompressor on the wasm surface, and a registry
|
|
18
|
+
re-cut. Revisit only if buckets shrink or sub-int8 quantization lands
|
|
19
|
+
(the thing that would actually move the 84.7%).
|
|
4
20
|
Depends on: plan 113 (KPK1), the 2026-09-10 model-efficiency research
|
|
5
21
|
|
|
6
22
|
## Problem
|
|
@@ -1,6 +1,20 @@
|
|
|
1
1
|
# Plan 123 — Verifiable typo corpora, synthesized at scale
|
|
2
2
|
|
|
3
|
-
Status:
|
|
3
|
+
Status: executed (models PR #32, 2026-09-12)
|
|
4
|
+
|
|
5
|
+
de/es: 5,000 pairs each, dictionary-grounded, deterministic. The
|
|
6
|
+
admission rule was SHAPED BY MEASUREMENT: the first draft sampled raw
|
|
7
|
+
dictionary stems and 77% of corrections landed outside the model
|
|
8
|
+
vocab - the corpus would have measured the vocabulary cut, not
|
|
9
|
+
ranking; the checked-in rule samples the rank-ordered dict-vocab
|
|
10
|
+
intersection (correction-OOV now 0). make_typo_traced added to
|
|
11
|
+
noise.py (op provenance, weighting unchanged). Headline tier finding:
|
|
12
|
+
full-tier typo-OOV on these typos is 98.5% de / 98.6% es - the
|
|
13
|
+
quantified gap the hybrid thread claims to close. Tests in the repo's
|
|
14
|
+
unittest convention (admission invariants, histogram consistency,
|
|
15
|
+
dictionary verification, byte-deterministic rebuild). The hybrid
|
|
16
|
+
pricing rerun (plan 115 ship decision) still needs the plan-114
|
|
17
|
+
training environment reassembled; the data blocker is closed.
|
|
4
18
|
Depends on: plan 115 (hybrid pricing blocked on real de/es corpora);
|
|
5
19
|
validated direction: DeepSeek-V4.1-Flash §5.1 — data/environment
|
|
6
20
|
synthesis exceeds algorithmic novelty
|
|
@@ -4,12 +4,21 @@ Status: standing inventory (updated 2026-09-10)
|
|
|
4
4
|
|
|
5
5
|
- VS Code Marketplace + Open VSX PATs → publish kotoshu-vscode
|
|
6
6
|
(extension is publish-ready; owner: "later").
|
|
7
|
-
- Worktree cleanup —
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
7
|
+
- ~~Worktree cleanup~~ — DONE 2026-09-12 under the owner's "make the
|
|
8
|
+
needed cleanups now": every repo down to its single main worktree,
|
|
9
|
+
merged branches deleted local+remote (see the memory entry).
|
|
10
|
+
- PyPI trusted publisher for kotoshu-native — owner registration
|
|
11
|
+
(project kotoshu-native, owner kotoshu, repo kotoshu-rs, workflow
|
|
12
|
+
release-pypi.yml, env blank), then rerun run 34681280341: the full
|
|
13
|
+
16-wheel matrix + sdist are BUILT and preserved as artifacts; the
|
|
14
|
+
publish step failed only on the missing registration.
|
|
15
|
+
- Typo layer registry cut — owner's three mechanical steps, runbooked
|
|
16
|
+
in models-fasttext-onnx docs/RELEASING.md; the moment the pair
|
|
17
|
+
carries vocab_url, `kotoshu setup LANG --typo` arms the layer.
|
|
18
|
+
- Gem 1.0.3 cut decision — main is ahead of 1.0.2 (opt-in typo layer,
|
|
19
|
+
--typo setup, Windows strict-wrap fix, eager GVL-released build);
|
|
20
|
+
nothing changes for default installs, so the cut timing and number
|
|
21
|
+
are the owner's call.
|
|
13
22
|
- kotoshu-lsp: still 0.1.1 — joins a future train if the owner wants
|
|
14
23
|
LSP on the 1.x line (audit train deliberately excluded it).
|
|
15
24
|
- Docker/kotoshu-ci + kotoshu-go/kotoshu-python floors: verify their
|
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
# Plan 125 — Dependency-floor guards in every consumer repo
|
|
2
|
+
|
|
3
|
+
Status: executed (docker PR #1; site push 70df393; 2026-09-12)
|
|
4
|
+
|
|
5
|
+
The guard found a REAL break on its first run: the docker CI image
|
|
6
|
+
could not install kotoshu at all since the native extension shipped
|
|
7
|
+
(ruby:slim lacks make; the 'no native ext' comment was stale). Fix:
|
|
8
|
+
build-essential + clang/libclang + rustup in the builder stage.
|
|
9
|
+
Dispositions: kotoshu-go has no artifact pin to guard (pure /v1 HTTP
|
|
10
|
+
client); kotoshu-python's native extra rides the deliberately-0.x
|
|
11
|
+
kotoshu-native line - a floor lands when that line crosses 1.0
|
|
12
|
+
(tracked in plan 124).
|
|
13
|
+
Depends on: plan 118 (the bug class: silent floor drift)
|
|
14
|
+
|
|
15
|
+
## Problem
|
|
16
|
+
|
|
17
|
+
Plan 118 found kotoshu-server shipping 1.0.0 with a runtime floor
|
|
18
|
+
(`~> 0.6`) that excluded the gem it was releasing alongside — and its
|
|
19
|
+
own CI had been bundling kotoshu 0.6.0 the whole time. The same
|
|
20
|
+
silent-drift class lives wherever a consumer pins a kotoshu artifact:
|
|
21
|
+
docker-kotoshu-ci (base image gem), kotoshu-go (embedded wasm/native
|
|
22
|
+
pins or docs), kotoshu-python (wheel cargo/git pin of kotoshu-rs).
|
|
23
|
+
Nothing reads those pins after they are written.
|
|
24
|
+
|
|
25
|
+
## Fix
|
|
26
|
+
|
|
27
|
+
Each repo gains the same guard shape the server got: a spec/test that
|
|
28
|
+
loads the pin (gemspec/Dockerfile/dependency manifest) and asserts it
|
|
29
|
+
admits the artifact version the repo actually builds/tests against.
|
|
30
|
+
A constraint nobody reads is guarded by a test that reads it.
|
|
31
|
+
|
|
32
|
+
- docker-kotoshu-ci: a CI step that installs the pinned gem and
|
|
33
|
+
asserts `kotoshu --version` >= the floor the Dockerfile names.
|
|
34
|
+
- kotoshu-python / kotoshu-go: assert the pinned kotoshu-rs rev/tag
|
|
35
|
+
resolves to the crate the wheels/binaries were built from (build
|
|
36
|
+
metadata comparison), or at minimum that the pin admits the
|
|
37
|
+
workspace version.
|
|
38
|
+
|
|
39
|
+
## Acceptance
|
|
40
|
+
|
|
41
|
+
- Each repo's CI fails if its pin excludes the version it builds.
|
|
42
|
+
- 124's inventory item "floors to check on next cuts" shrinks to
|
|
43
|
+
"guards exist".
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
# Plan 126 — Publish the measured error budget per tier
|
|
2
|
+
|
|
3
|
+
Status: executed (site 70df393, 2026-09-12)
|
|
4
|
+
|
|
5
|
+
/performance carries the table: 55-language worst-case rank_corr /
|
|
6
|
+
top-1 agreement per tier against the gates, the vocabulary-cut
|
|
7
|
+
caveat, and the plan-123 finding that even the full tier embeds only
|
|
8
|
+
~1.5% of dictionary-grounded typos.
|
|
9
|
+
Depends on: the 2026-09-10 model-efficiency research (DeepSeek-V4.1-Flash:
|
|
10
|
+
every "cheaper" claim carries its measured number)
|
|
11
|
+
|
|
12
|
+
## Problem
|
|
13
|
+
|
|
14
|
+
The site documents tier sizes and the near-lossless headline
|
|
15
|
+
(fluency rank_corr 0.9999) but not a per-tier, per-language error
|
|
16
|
+
budget a user can reason about: what quality do I trade for mini's
|
|
17
|
+
3 MB vs fluency's 15 MB vs full? The numbers exist (gates.json,
|
|
18
|
+
corpus benches); they are just not published as a table.
|
|
19
|
+
|
|
20
|
+
## Fix
|
|
21
|
+
|
|
22
|
+
- /docs/performance gains an "Error budget" section: per tier, the
|
|
23
|
+
frozen gate values (fluency rank_corr, top-1/top-5 agreement deltas
|
|
24
|
+
vs full on the real corpora) with corpus sizes stated, plus the
|
|
25
|
+
dictionary-only baseline for contrast.
|
|
26
|
+
- The table cites its source (eval gates at registry tag) and its
|
|
27
|
+
date; numbers regenerate with each registry release (script optional
|
|
28
|
+
— a hand-truthed table with provenance beats a stale pipeline).
|
|
29
|
+
|
|
30
|
+
## Acceptance
|
|
31
|
+
|
|
32
|
+
- A user can answer "what do I lose on mini?" with one number pair.
|
|
33
|
+
- The section states corpora sizes and OOV caveats honestly.
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
# Plan 127 — Report-a-wrong-suggestion: backendless feedback loop
|
|
2
|
+
|
|
3
|
+
Status: executed (site 70df393, 2026-09-12)
|
|
4
|
+
|
|
5
|
+
The popover carries a 'wrong suggestions? report' link opening a
|
|
6
|
+
pre-filled issue (word, top-5 rows with source+confidence, language,
|
|
7
|
+
engine version - no page text).
|
|
8
|
+
Depends on: the 2026-09-11 recommendation (real failure reports feed
|
|
9
|
+
plan 123's corpora; DeepSeek's environments-from-real-failures lesson)
|
|
10
|
+
|
|
11
|
+
## Problem
|
|
12
|
+
|
|
13
|
+
The playground is our only surface where real users meet real
|
|
14
|
+
suggestions, and when a suggestion is wrong the feedback evaporates.
|
|
15
|
+
Plan 123 synthesizes pairs; the higher-value stream is REAL wrong-
|
|
16
|
+
suggestion reports, and capturing them needs no backend: a GitHub
|
|
17
|
+
issue prefill carries the diagnostic.
|
|
18
|
+
|
|
19
|
+
## Fix
|
|
20
|
+
|
|
21
|
+
- Playground suggestion popover gains a small "report" affordance on
|
|
22
|
+
each row: opens github.com/kotoshu/kotoshu/issues/new?title=... with
|
|
23
|
+
a body template containing the word, the offered suggestions, the
|
|
24
|
+
language, the engine version, and (already public) tier info — no
|
|
25
|
+
page content, no user text beyond what the user chooses to add.
|
|
26
|
+
- The issue template lands in the gem repo (.github/ISSUE_TEMPLATE/
|
|
27
|
+
wrong-suggestion.md) so reports arrive structured; a triage note
|
|
28
|
+
points verified pairs at eval/corpora for the next reprice.
|
|
29
|
+
|
|
30
|
+
## Acceptance
|
|
31
|
+
|
|
32
|
+
- One click from a wrong suggestion to a structured, pre-filled issue.
|
|
33
|
+
- No PII beyond the single word the user is already looking at; the
|
|
34
|
+
template says so.
|
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
# Plan 128 — The hybrid ship decision: price C-retrieve + fastText-rescore on the closed data blocker
|
|
2
|
+
|
|
3
|
+
Status: executed — VERDICT SHIP=True (models PR #34, 2026-09-12)
|
|
4
|
+
|
|
5
|
+
The full arc: frozen cbench rebuilt byte-faithfully (every sha256
|
|
6
|
+
reproduced; only the receipt wall-clock differed, reverted), c_typo_v2
|
|
7
|
+
retrained (0.481 MB, probe_hit5 0.410), pricing verdict per the
|
|
8
|
+
pre-declared rule: hybrid beats full on top-1 AND top-5 in all four
|
|
9
|
+
REAL components (en +6.3pp top5 CI [+4.9,+7.7] decisive; de +28.6,
|
|
10
|
+
ru +11.1, es +10.0), degrades nowhere, 25.8 MB int8 at ~3-4 ms.
|
|
11
|
+
Densified: on ~4,700 synth pairs/lang the hybrid evaluates essentially
|
|
12
|
+
all (top5 0.467-0.478) where full embeds only 69-85 pairs. Engine
|
|
13
|
+
productization = the follow-on plan (131).
|
|
14
|
+
Depends on: plan 123 (corpora shipped; the data blocker that parked
|
|
15
|
+
plan 115), plan 114 (the frozen C-benchmark + pre-declared verdict rule)
|
|
16
|
+
|
|
17
|
+
## Problem
|
|
18
|
+
|
|
19
|
+
Plan 115 priced the hybrid retrieval thread and parked on missing de/es
|
|
20
|
+
typo data. Plan 123 closed that blocker (5,000 dictionary-grounded
|
|
21
|
+
pairs each). The plan-114 artifacts (frozen cbench files, c_typo_v2
|
|
22
|
+
encoder) were gitignored and wiped in the September cleanup — but the
|
|
23
|
+
corpus is vendored, the builder and trainer are committed, and the
|
|
24
|
+
whole chain is deterministic. The verdict arc is executable end to end.
|
|
25
|
+
|
|
26
|
+
## Execution
|
|
27
|
+
|
|
28
|
+
1. Rebuild the frozen C-benchmark: `scripts/build_cbench.py` — the
|
|
29
|
+
receipt (eval/reports/cbench.frozen.json) carries per-file sha256s;
|
|
30
|
+
the rebuild must reproduce them byte for byte or the freeze is
|
|
31
|
+
broken and the run stops (integrity before evidence).
|
|
32
|
+
2. Retrain `c_typo_v2`: `scripts/train_typo_biencoder_v2.py` (char
|
|
33
|
+
bi-GRU, CPU, minutes; artifacts gitignored as before).
|
|
34
|
+
3. Price: `eval/hybrid_pricing_bench.py` over en/de/ru/es — the frozen
|
|
35
|
+
REAL components are the ship gate (unchanged), the plan-123 corpora
|
|
36
|
+
are additional labeled evidence for the de/es gap.
|
|
37
|
+
4. Verdict per the plan-114 pre-declared rule (wins on real, degrades
|
|
38
|
+
nowhere with n>=20) — ship, or reject on data like 114 did. Either
|
|
39
|
+
outcome closes the thread honestly.
|
|
40
|
+
|
|
41
|
+
## Acceptance
|
|
42
|
+
|
|
43
|
+
- The cbench rebuild reproduces every frozen sha256.
|
|
44
|
+
- The pricing report lands (eval/reports/hybrid-pricing.json refresh)
|
|
45
|
+
with real + synth components labeled, and the verdict is recorded in
|
|
46
|
+
the plan status with numbers.
|
|
47
|
+
- No frozen artifact changes; new evidence only.
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
# Plan 129 — pt/fr synthetic corpora (plan 123 stretch targets)
|
|
2
|
+
|
|
3
|
+
Status: executed (models PR #34, 2026-09-12) — 5,000 pairs each,
|
|
4
|
+
same rule, tests at four languages; tier numbers in the refreshed
|
|
5
|
+
synth-corpus summary.
|
|
6
|
+
Depends on: plan 123 (the generator)
|
|
7
|
+
|
|
8
|
+
## Problem
|
|
9
|
+
|
|
10
|
+
Plan 123 shipped de/es; pt and fr were declared stretch targets. The
|
|
11
|
+
generator, tests, and bench all generalize by construction — extending
|
|
12
|
+
is a parameter, not new code.
|
|
13
|
+
|
|
14
|
+
## Fix
|
|
15
|
+
|
|
16
|
+
- Generate eval/corpora/synth/{pt,fr}.json at the same 5,000-pair
|
|
17
|
+
target with the same admission rule (pt: ABNT qwerty + accents; fr:
|
|
18
|
+
azerty alternates — both already in noise.py's layout registry).
|
|
19
|
+
- Extend the tier baseline bench + tests to the new languages.
|
|
20
|
+
|
|
21
|
+
## Acceptance
|
|
22
|
+
|
|
23
|
+
- Same invariants as 123, same test coverage, tier numbers recorded.
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
# Plan 130 — News truth: the 1.0.1 patch and the 09-12 wave
|
|
2
|
+
|
|
3
|
+
Status: executed (site 6daa72d, 2026-09-12) — gem-1-0-1 and
|
|
4
|
+
evidence-2026-09 entries live; NewsKind widened with 'docs'.
|
|
5
|
+
Depends on: plans 119/121 (1.0.1 content), 123/125/126/127
|
|
6
|
+
|
|
7
|
+
## Problem
|
|
8
|
+
|
|
9
|
+
Two shipped changes have no news entries: gem 1.0.1 (embedded frozen
|
|
10
|
+
tiers, Jekyll sweep skip) and the 09-12 wave (corpora, error budgets,
|
|
11
|
+
the feedback loop, the docker image fix). The site's discipline is one
|
|
12
|
+
entry per user-visible change.
|
|
13
|
+
|
|
14
|
+
## Fix
|
|
15
|
+
|
|
16
|
+
- One entry for 1.0.1 (determinism by construction: cache-cold == warm
|
|
17
|
+
== frozen vectors).
|
|
18
|
+
- One entry for the wave: corpora + the 1.5% typo-embedding finding +
|
|
19
|
+
error-budget table + report-a-suggestion link.
|
|
20
|
+
|
|
21
|
+
## Acceptance
|
|
22
|
+
|
|
23
|
+
- Entries live, dates and version references truthful.
|
|
@@ -0,0 +1,57 @@
|
|
|
1
|
+
# Plan 131 — Productize the hybrid: C-retrieve + fastText-rescore in the engines
|
|
2
|
+
|
|
3
|
+
Status: executed on the engine and gem sides (2026-09-12; kotoshu-rs
|
|
4
|
+
PR #38 merged — the pure-Rust bi-encoder at parity cosine 1.00000 vs
|
|
5
|
+
onnxruntime, the int8 retrieval scan at 1.81 ms per suggest, the
|
|
6
|
+
frozen C-benchmark numbers reproduced EXACTLY through the shipped
|
|
7
|
+
engine path: top1 0.118772, top5 0.269829 on en real n=2509; wasm
|
|
8
|
+
loadTypo/typoSuggest + KPK1 typo sections + fp32 full-tier RowStore +
|
|
9
|
+
ruby bindings with a 28-assertion smoke; gem PR #191 — Typo::Engine
|
|
10
|
+
loader, TypoRetrievalStrategy, TypoMerge, KOTOSHU_TYPO_RETRIEVAL
|
|
11
|
+
opt-in with a byte-identical disabled passthrough, 27 specs, full
|
|
12
|
+
suite 4030/0). Gates: all three pre-declared clauses pass (numbers
|
|
13
|
+
within noise = exact on the decision metric; latency 1.8 ms against
|
|
14
|
+
the 10 ms budget; zero default change by construction and by spec).
|
|
15
|
+
REMAINING (owner-coupled): the registry promotion — the typo entry's
|
|
16
|
+
primary URL and vocab_url stay null until the owner cuts a registry
|
|
17
|
+
release carrying them; the server/playground/worker flips follow that
|
|
18
|
+
cut on the semantic-tier pattern
|
|
19
|
+
Depends on: plan 128 (verdict + artifacts), plan 113 (packs — the
|
|
20
|
+
delivery vehicle), registry v1.6.0 (typo-biencoder assets already opt-in)
|
|
21
|
+
|
|
22
|
+
## What ships
|
|
23
|
+
|
|
24
|
+
The hybrid becomes the opt-in "typo retrieval" layer, the same shape
|
|
25
|
+
as the semantic tier: per-language int8 C-matrix (25.8 MB) + the char
|
|
26
|
+
bi-encoder (0.48 MB) fetched from the registry, C-retrieve top-20 over
|
|
27
|
+
the vocabulary, fastText-rescore of the candidates, merged into the
|
|
28
|
+
suggestion pipeline ahead of frequency ranking.
|
|
29
|
+
|
|
30
|
+
## Surfaces
|
|
31
|
+
|
|
32
|
+
- kotoshu-rs: char-bi-encoder ONNX inference (the pure-Rust ONNX
|
|
33
|
+
reader already exists for the tiers), the retrieval+rescore path,
|
|
34
|
+
wasm surface (`loadTypo` + `typoSuggest`), KPK1 pack extension (the
|
|
35
|
+
C-matrix rides the pack for pack languages).
|
|
36
|
+
- gem: native-ext accelerated path where present; the merge into
|
|
37
|
+
Suggestions::Pipeline as a strategy (OCP: a new strategy class, no
|
|
38
|
+
pipeline surgery).
|
|
39
|
+
- Conformance: new vectors for the hybrid path are NOT frozen
|
|
40
|
+
initially (it is opt-in, tier-like, gated by eval not conformance —
|
|
41
|
+
the tier precedent).
|
|
42
|
+
- Registry: promote the typo-biencoder entries from opt-in to exposed;
|
|
43
|
+
server/playground/worker flips follow the semantic-tier pattern.
|
|
44
|
+
|
|
45
|
+
## Gates (pre-declared)
|
|
46
|
+
|
|
47
|
+
- The frozen real components + synth corpora re-run against the
|
|
48
|
+
SHIPPED engines must reproduce the PR-34 numbers within noise.
|
|
49
|
+
- Latency: per-suggest budget ≤ 10 ms amortized (the plan-112 gates
|
|
50
|
+
tighten for the shipped path).
|
|
51
|
+
- Degrades nowhere vs dictionary-only when disabled; zero behavior
|
|
52
|
+
change at default settings.
|
|
53
|
+
|
|
54
|
+
## Non-goals
|
|
55
|
+
|
|
56
|
+
- No retraining; the v2 artifacts are frozen (probe_hit5 0.410).
|
|
57
|
+
- No default-on: opt-in per language, like the semantic tier.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
# Plan 132 — The techniques page; professional prose across the site
|
|
2
|
+
|
|
3
|
+
Status: executed (2026-09-12, site push; this plan)
|
|
4
|
+
|
|
5
|
+
## What shipped
|
|
6
|
+
|
|
7
|
+
1. /docs/techniques — a complete catalog of the stack's techniques in
|
|
8
|
+
four groups (engines, models, delivery, reliability), each entry
|
|
9
|
+
naming the mechanism and its measured effect in full sentences.
|
|
10
|
+
Written as the site's first MDX page: @astrojs/mdx installed and
|
|
11
|
+
wired, DocsLayout applied through an imported component wrapper.
|
|
12
|
+
2. A professional rewrite of the published copy: 34 fragment-style
|
|
13
|
+
news titles became declarative sentences, and 69 fragment-stacked
|
|
14
|
+
summaries and senses across all 36 entries were rewritten as full
|
|
15
|
+
sentences with clear subjects. The worst patterns eliminated:
|
|
16
|
+
em-dash fragment leads ("What 1.0 freezes:"), colon-list openers
|
|
17
|
+
("Also in the patch:"), trailing fragment appends ("— no download,
|
|
18
|
+
no CI seeding"), and lowercase clause starts.
|
|
19
|
+
3. The docs pages received the same treatment where fragments
|
|
20
|
+
appeared: the performance error-budget section, the caching
|
|
21
|
+
language-packs section, and three headlines.
|
|
22
|
+
|
|
23
|
+
## Conventions established
|
|
24
|
+
|
|
25
|
+
- Text-heavy documentation pages are written in MDX, not .astro HTML.
|
|
26
|
+
- News titles are complete clauses; summaries and senses are full
|
|
27
|
+
sentences. Release titles use the "package X.Y.Z: description" form.
|
|
@@ -0,0 +1,90 @@
|
|
|
1
|
+
# Plan 133 — Rust-first distribution: precompiled native gems and complete wheels
|
|
2
|
+
|
|
3
|
+
Status: executed (2026-09-12) — PR #189 (7-iteration CI chain: the
|
|
4
|
+
MinGW Rust toolchain for the ABI match, bindgen pointed at the
|
|
5
|
+
RubyInstaller devkit's ucrt64 headers through BINDGEN_EXTRA_CLANG_ARGS
|
|
6
|
+
with the windows-gnu target, macos-13 -> macos-15-intel for the retired
|
|
7
|
+
x86_64 macOS runner, and the release race fix so the native legs build
|
|
8
|
+
from the v<version> tag). Released as kotoshu 1.0.2: six artifacts on
|
|
9
|
+
RubyGems (ruby + x86_64-linux, aarch64-linux, arm64-darwin, x86_64-darwin,
|
|
10
|
+
x64-mingw-ucrt), verified by a toolchain-free scratch-GEM_HOME install
|
|
11
|
+
that resolved kotoshu-1.0.2-arm64-darwin, engaged the native engine
|
|
12
|
+
under the auto default, and passed the frozen vectors. Docker: both
|
|
13
|
+
images (docker-kotoshu-ci PR #2, kotoshu-server PR #9) dropped their
|
|
14
|
+
build-tool layers / gained the native-backend build assertion, with the
|
|
15
|
+
guard proving the platform gem resolves inside the slim builder. Site:
|
|
16
|
+
engines section resolution paragraph + the 1.0.2 news entry (092fc03).
|
|
17
|
+
Design deviation: no x86_64-linux-musl platform gem — there is no
|
|
18
|
+
hosted musl Ruby runner and cross needs a musl Ruby; Alpine resolves
|
|
19
|
+
the pure-Ruby gem (source-build of the extension remains possible
|
|
20
|
+
there). Python: the 16-wheel + sdist kotoshu-native 0.1.1 matrix is
|
|
21
|
+
built and smoke-green (run 34681280341) but the PyPI publish is gated
|
|
22
|
+
on the owner registering the trusted publisher (pypi.org, project
|
|
23
|
+
kotoshu-native, repo kotoshu-rs, workflow release-pypi.yml, env blank);
|
|
24
|
+
rerun `gh run rerun 34681280341 --repo kotoshu/kotoshu-rs --failed`
|
|
25
|
+
afterward — the wheel artifacts persist, no rebuild needed.
|
|
26
|
+
Depends on: the 1.0 conformance contract (identical outputs make the
|
|
27
|
+
backend default flip correctness-free); plan 93 (the python wheel matrix)
|
|
28
|
+
|
|
29
|
+
## Problem
|
|
30
|
+
|
|
31
|
+
The Rust engine is the faster implementation (~10-15x on the sweep) and
|
|
32
|
+
the strategic default, but today reaching it requires a build toolchain:
|
|
33
|
+
the gem compiles its native extension at install time, and the python
|
|
34
|
+
native wheel matrix is incomplete on PyPI. Users who cannot run a
|
|
35
|
+
toolchain get pure Ruby whether they wanted it or not — and users who
|
|
36
|
+
wanted Rust pay a compile for it. The architecture makes both engines
|
|
37
|
+
byte-identical, so distribution — not capability — is the only gap.
|
|
38
|
+
|
|
39
|
+
## Design
|
|
40
|
+
|
|
41
|
+
### Ruby: precompiled platform gems (the rb_sys flow)
|
|
42
|
+
|
|
43
|
+
1. New gem workflow `release-native.yml`: build matrix over
|
|
44
|
+
{x86_64-linux, aarch64-linux, x86_64-linux-musl, arm64-darwin,
|
|
45
|
+
x86_64-darwin, x64-mingw-ucrt} x {ruby 3.1..4.0} using
|
|
46
|
+
rb-sys-dock images for the Linux targets and native runners for
|
|
47
|
+
macOS/Windows; each job builds and pushes
|
|
48
|
+
`kotoshu-X.Y.Z-<platform>.gem` through trusted publishing. The
|
|
49
|
+
pure-Ruby platform gem continues to publish exactly as today, so
|
|
50
|
+
every unmatched platform resolves to it.
|
|
51
|
+
2. Backend default `ruby` -> `auto`: use the native engine when the
|
|
52
|
+
extension loads, fall back silently to pure Ruby when it does not.
|
|
53
|
+
Auto never downloads and never fails a check; specs pin both
|
|
54
|
+
branches; the conformance job already runs both engines.
|
|
55
|
+
3. The docker CI image drops its build-tool layer (it installs the
|
|
56
|
+
platform gem instead) — smaller image, faster builds, and its
|
|
57
|
+
version guard gains a native-backend assertion.
|
|
58
|
+
|
|
59
|
+
### Python: complete the kotoshu-native wheel matrix
|
|
60
|
+
|
|
61
|
+
1. Audit PyPI's published wheels for kotoshu-native against the
|
|
62
|
+
plan-93 matrix: manylinux x86_64/aarch64, macOS x86_64/arm64,
|
|
63
|
+
windows x86_64 (+ arm64 if tractable), sdist.
|
|
64
|
+
2. Fill the gaps in release-pypi.yml (maturin + the standard target
|
|
65
|
+
matrix), publish a patch release, and verify pip resolves a wheel
|
|
66
|
+
(no source build) on each platform.
|
|
67
|
+
3. Add the trove classifiers for every published target.
|
|
68
|
+
|
|
69
|
+
### Site
|
|
70
|
+
|
|
71
|
+
Install page's engines-and-bindings section gains the resolution
|
|
72
|
+
sentence: the native engine installs automatically wherever a prebuilt
|
|
73
|
+
artifact exists for your platform, and pure Ruby is always the
|
|
74
|
+
fallback; python's entry notes the wheel coverage.
|
|
75
|
+
|
|
76
|
+
## Gates
|
|
77
|
+
|
|
78
|
+
- A clean-machine `gem install kotoshu` on each published platform
|
|
79
|
+
loads the native backend with no toolchain present (CI-verifiable
|
|
80
|
+
on the matrix runners).
|
|
81
|
+
- The 2,630-vector conformance compare passes under `auto` on both a
|
|
82
|
+
native and a pure-Ruby install.
|
|
83
|
+
- pip on each target platform resolves a wheel, not the sdist.
|
|
84
|
+
- The pure-Ruby path is unchanged: same gem, same outputs, same specs.
|
|
85
|
+
|
|
86
|
+
## Non-goals
|
|
87
|
+
|
|
88
|
+
- No behavior change of any kind — outputs are frozen; only which
|
|
89
|
+
engine computes them changes.
|
|
90
|
+
- No deprecation of anything: pure Ruby is a feature, not a legacy.
|
|
@@ -0,0 +1,67 @@
|
|
|
1
|
+
# Plan 134 — Post-131 improvements: the audit's honest gaps
|
|
2
|
+
|
|
3
|
+
Status: executed (2026-09-13). (1) GVL release rs PR #39 — the
|
|
4
|
+
without_gvl helper runs the typo compute through
|
|
5
|
+
rb_thread_call_without_gvl, PROVEN LIVE: a 5 ms ticker thread logged
|
|
6
|
+
4,118 ticks during the 24.8 s / 100k-word index build (a held GVL
|
|
7
|
+
starves it to ~0), suggest answers 2.2 ms after the eager build; gem
|
|
8
|
+
PR #195 makes Engine.for build eagerly. (2) native-suite CI leg (gem
|
|
9
|
+
PR #195): the conformance workflow compiles the extension and runs
|
|
10
|
+
the full suite — NativeBackend can no longer regress matrix-green.
|
|
11
|
+
(3) release verify (gem PR #195): the release asserts all six
|
|
12
|
+
RubyGems artifacts before reporting success. (4) runbook models PR
|
|
13
|
+
#37: docs/RELEASING.md carries the standard cut and the typo
|
|
14
|
+
promotion steps. (5) perf-weekly (gem PR #195): the weekly slow-suite
|
|
15
|
+
run whose first execution caught the suites rotted — repaired with
|
|
16
|
+
dated 2x-headroom bounds. Found and fixed en route (gem PR #196,
|
|
17
|
+
merged): setup_typo_remote rescued only Kotoshu::Error, leaking raw
|
|
18
|
+
Errno::ECONNREFUSED on Windows (caught by merge-preview CI on main).
|
|
19
|
+
|
|
20
|
+
Depends on: plan 131 (the typo layer), plan 133 (the platform gems)
|
|
21
|
+
|
|
22
|
+
## The items
|
|
23
|
+
|
|
24
|
+
### 1. The typo layer's first-suggest stall (production defect class)
|
|
25
|
+
|
|
26
|
+
The 100k-word index build (~15 s) happens lazily inside the first
|
|
27
|
+
`typo_suggest`, and the Ruby binding holds the GVL for the whole Rust
|
|
28
|
+
call — on the server, one arming request blocks every other request
|
|
29
|
+
for the build's duration. Fixes, both:
|
|
30
|
+
|
|
31
|
+
- The binding releases the GVL around the pure-Rust compute (index
|
|
32
|
+
build and scan touch no Ruby objects; results re-enter GVL scope to
|
|
33
|
+
build hashes), so a build never stops the world.
|
|
34
|
+
- The gem resolves EAGERLY: `Engine.for` builds the index at load —
|
|
35
|
+
the cost moves to opt-in setup where it belongs, never mid-check.
|
|
36
|
+
|
|
37
|
+
### 2. The native path is invisible to the main rake matrix
|
|
38
|
+
|
|
39
|
+
The nine rake legs report the native specs PENDING ("native extension
|
|
40
|
+
not built"); only the separate compare job compiles the ext. A
|
|
41
|
+
`NativeBackend` regression could land matrix-green. One dedicated
|
|
42
|
+
matrix job compiles the extension and runs the suite un-pended.
|
|
43
|
+
|
|
44
|
+
### 3. The release workflow trusts its own pushes
|
|
45
|
+
|
|
46
|
+
Nothing asserts the release is COMPLETE — a silently failed platform
|
|
47
|
+
push ships a partial release green. A `verify` job after the native
|
|
48
|
+
legs queries the RubyGems API for version x platform coverage (ruby +
|
|
49
|
+
the five native platforms) and fails on anything missing.
|
|
50
|
+
|
|
51
|
+
### 4. The owner's cut-day procedure lives in session memory
|
|
52
|
+
|
|
53
|
+
The three-step typo release checklist is recorded in memory and PR
|
|
54
|
+
bodies. The models repo carries the committed runbook instead — the
|
|
55
|
+
owner cuts the release without needing this session.
|
|
56
|
+
|
|
57
|
+
### 5. The weekly SLOW_TESTS dashboard, owner-free form
|
|
58
|
+
|
|
59
|
+
The T4.2 remainder as a workflow: weekly run of the slow suites, the
|
|
60
|
+
delta summary posted on the run page (a job summary + artifact — no
|
|
61
|
+
external dashboard to own).
|
|
62
|
+
|
|
63
|
+
## Gates
|
|
64
|
+
|
|
65
|
+
- Every fix ships with its spec or CI proof; no behavior change to
|
|
66
|
+
any frozen output (the GVL release is invisible; eager build moves
|
|
67
|
+
WHEN the same index is built, not what it computes).
|
data/TODO.impl/54-cjk-support.md
CHANGED
|
@@ -7,8 +7,11 @@ This is the largest tier 3 item: CJK languages need word-segmentation
|
|
|
7
7
|
(no spaces in zh/ja), different dictionary formats, and
|
|
8
8
|
romanization-aware suggestion strategies.
|
|
9
9
|
|
|
10
|
-
Status: outline only
|
|
11
|
-
|
|
10
|
+
Status: outline only, partially superseded. Japanese tokenization
|
|
11
|
+
shipped (the ja module with the suika tokenizer, live since the
|
|
12
|
+
language-modules wave); real zh/ko spell checking remains design
|
|
13
|
+
space — this TODO documents it, and real work still needs upstream
|
|
14
|
+
segmentation library choices and per-language fixture data.
|
|
12
15
|
|
|
13
16
|
## Per-language needs
|
|
14
17
|
|
|
@@ -1,5 +1,34 @@
|
|
|
1
1
|
# 56 — T4+ Quality, Architecture, and Ecosystem backlog
|
|
2
2
|
|
|
3
|
+
Status: largely superseded — most items below were promoted and
|
|
4
|
+
executed by the numbered plans; the remaining open ones are marked.
|
|
5
|
+
Snapshot annotations (2026-09-13): T4.1 shipped as the plan-147/148
|
|
6
|
+
perf waves (WordIndex + SweepIndex: suggest 20-140 s -> sub-second,
|
|
7
|
+
sweep 1472 -> 661 ms en; the Rust engine then took the sweep 10-15x
|
|
8
|
+
further, and plan 131's hybrid retrieval added 1.8 ms retrieval) with
|
|
9
|
+
published error budgets on /performance. T4.4 shipped as plan 97 (YARD
|
|
10
|
+
100% public surface, docs CI, hosted reference) plus the repeated
|
|
11
|
+
README truth passes. T5.2 and T5.3 shipped (Configuration as data
|
|
12
|
+
with the Resolver; WordResult/DocumentResult/SuggestionSet/Suggestion
|
|
13
|
+
on lutaml-model, hand-rolled to_h banned by the global rule). T6.1
|
|
14
|
+
shipped (kotoshu-lsp, live on RubyGems) and T6.2 shipped (the wasm
|
|
15
|
+
playground, @kotoshu/wasm, language packs). Still open: T4.2's weekly
|
|
16
|
+
SLOW_TESTS dashboard, T5.4 (= plan 53), and T6.3 (desktop dictionary
|
|
17
|
+
editor — future). CLOSED SINCE: T4.3's named properties shipped
|
|
18
|
+
(spec/properties/data_structure_properties_spec.rb — Trie
|
|
19
|
+
order-independence and totality, prefix agreement, set recovery, and
|
|
20
|
+
the every-suggestion-is-a-dictionary-word invariant; the affix
|
|
21
|
+
round-trip property remains the one open sub-item, gated on a
|
|
22
|
+
morphological API surface). T5.1 audited clean (2026-09-13): the
|
|
23
|
+
dedupe-once merge CompositeStrategy already performs is exactly the
|
|
24
|
+
fix T5.1 proposed (the code cites it); strategy purity holds with one
|
|
25
|
+
documented exception — the semantic-cascade threshold falls back to
|
|
26
|
+
the process Configuration when the composite was not given an
|
|
27
|
+
explicit one, by design and overridable; and the symspell/edit-1
|
|
28
|
+
overlap is moot — neither is wired into any default flow (the frozen
|
|
29
|
+
Generator owns the real path; the strategy classes are the composable
|
|
30
|
+
API, and the single-set merge makes overlapping compositions safe).
|
|
31
|
+
|
|
3
32
|
This file rolls up the T4 (Quality), T5 (Architecture), and T6
|
|
4
33
|
(Ecosystem) items from TODO.impl/39-tier3-and-beyond.md into
|
|
5
34
|
promotable units with concrete acceptance criteria.
|