kotoshu 0.11.1 → 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 19518c2daced6ae79d4a6d02fc5a26f4c3ca7109a70d2210220893177312173d
|
|
4
|
+
data.tar.gz: 34f597de8432648f22a15a0ad1b09f71041eaff805c91bc9a1357b6f0e2aff56
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: ee21f0075b02b6aff17cd4e58262bdf0ad6356a69c1950be2493ee6a801970f5268769a35899fd6f651f27d0d50cee591ffedadf08c922f0623ecbdb3fb4bcc3
|
|
7
|
+
data.tar.gz: 34c321c65532ff5a175f0b535225bdda8937e7d72f537e6462d0e965ed6a5db3524ec63a65ecae34af2df9691d28285bfb14048ddda90483f28e5a438c3199da
|
|
@@ -0,0 +1,102 @@
|
|
|
1
|
+
# Plan 117 — Deterministic frequency tiers: stop the silent YAML fallback
|
|
2
|
+
|
|
3
|
+
Status: proposed (forensics complete; root cause proven)
|
|
4
|
+
Depends on: plan 82 (conformance runner), the 2,630-vector contract (plan 66+)
|
|
5
|
+
Blocks: the 1.0 train (audit checklist item 7 — "conformance vectors green at
|
|
6
|
+
the cut commit" — is not currently verifiable on a cache-cold machine)
|
|
7
|
+
|
|
8
|
+
## What happened (forensics, 2026-09-10)
|
|
9
|
+
|
|
10
|
+
Preparing the 1.0 train (checklist item 7) ran `rake
|
|
11
|
+
kotoshu:conformance:compare` on main:
|
|
12
|
+
|
|
13
|
+
compare: 2630 vectors -- ruby failures: 16, native failures: 0, divergences: 16
|
|
14
|
+
|
|
15
|
+
All 16 failures are `suggest` vectors on the compoundrule{,2,3} fixtures:
|
|
16
|
+
same candidate words, same distances, different `confidence` values
|
|
17
|
+
(e.g. expected `b` = 0.952380952…, ruby = 0.8333333…).
|
|
18
|
+
|
|
19
|
+
Ruled out, with evidence:
|
|
20
|
+
|
|
21
|
+
- **Not plan 116 / not recent main.** v0.11.0 shows the identical 16
|
|
22
|
+
failures; the suggestions/, dictionary/, core/ trees have zero commits
|
|
23
|
+
between v0.9.2 and main.
|
|
24
|
+
- **Not the machine cache contents.** The downloaded Kelly `en.json`
|
|
25
|
+
sha256 `97535823f41a3f0…` matches the pin in kotoshu-rs
|
|
26
|
+
`suggest/frequency_data.rs` byte-for-byte; `load_cached` returns the
|
|
27
|
+
same 47/185/907 tier sizes rust embeds.
|
|
28
|
+
- **Not an engine disagreement.** With a live (unexpired) cache:
|
|
29
|
+
|
|
30
|
+
compare: 2630 vectors -- ruby failures: 0, native failures: 0,
|
|
31
|
+
backend divergences: 0
|
|
32
|
+
|
|
33
|
+
### The actual mechanism
|
|
34
|
+
|
|
35
|
+
`Suggestions::FrequencyProvider` falls back
|
|
36
|
+
`FrequencyCache` → local YAML (`lib/kotoshu/data/common_words/en.yml`)
|
|
37
|
+
→ empty tiers. `BaseCache#available?` returns false when the metadata's
|
|
38
|
+
`cached_at` is older than the 7-day TTL. **The two datasets differ**:
|
|
39
|
+
the local YAML's top_50 contains `a` (bonus 200); Kelly's top_1000 is
|
|
40
|
+
where `a` lives (bonus 50). The bonus shifts one candidate's enhanced
|
|
41
|
+
score, the min–max normalization re-scales every confidence on the
|
|
42
|
+
tiny fixtures, and 16 vectors "fail".
|
|
43
|
+
|
|
44
|
+
So any machine whose frequency cache is absent or >7 days old silently
|
|
45
|
+
runs ruby on the YAML dataset — different suggestion ranking from the
|
|
46
|
+
frozen contract and from the Rust engine. This is a **user-facing
|
|
47
|
+
determinism bug**, not conformance bookkeeping: suggestion output
|
|
48
|
+
depends on cache age.
|
|
49
|
+
|
|
50
|
+
Two aggravators found on the way:
|
|
51
|
+
|
|
52
|
+
- `Kotoshu.setup(:en, want: [:frequency])` reported
|
|
53
|
+
`frequency: :downloaded` but left `cached_at` at 2026-09-02 — either
|
|
54
|
+
the re-download was skipped on checksum match (fine) and the
|
|
55
|
+
timestamp was not refreshed (bug: guarantees later expiry), or the
|
|
56
|
+
report is wrong. Verify and fix.
|
|
57
|
+
- Nothing CI-gates `conformance:compare` on the gem side (the rs repo
|
|
58
|
+
gates its half only), which is why a two-week-old dataset switch was
|
|
59
|
+
invisible.
|
|
60
|
+
|
|
61
|
+
## Fix design
|
|
62
|
+
|
|
63
|
+
1. **Expired ≠ unusable.** In `FrequencyProvider#load`, when
|
|
64
|
+
`available?` is false but the on-disk cached data exists, load it
|
|
65
|
+
anyway (`load_cached` does no network). TTL governs *refresh*
|
|
66
|
+
(setup/`get` paths), never a silent switch to a different dataset.
|
|
67
|
+
The YAML fallback then only serves machines with NO frequency data
|
|
68
|
+
at all.
|
|
69
|
+
2. **Hermetic conformance.** The runner must not depend on machine
|
|
70
|
+
cache state: check in the frozen Kelly tiers (the three sets +
|
|
71
|
+
sha256, mirroring rs `frequency_data.rs`) as a conformance fixture
|
|
72
|
+
and construct the engines with that provider data pinned. Then
|
|
73
|
+
`conformance:compare` is deterministic on any machine, cache or no
|
|
74
|
+
cache, and cross-verifies the fixture against the rs tables.
|
|
75
|
+
3. **CI gate.** Add an Ubuntu job: `rake compile && rake
|
|
76
|
+
kotoshu:conformance:compare` on every push/PR. This invariant is
|
|
77
|
+
the product's core correctness claim; it must not rely on someone
|
|
78
|
+
remembering to run it.
|
|
79
|
+
4. **Setup timestamp.** When setup skips a download because the
|
|
80
|
+
checksum matches, refresh `cached_at` (and report `:cached`).
|
|
81
|
+
|
|
82
|
+
## Acceptance
|
|
83
|
+
|
|
84
|
+
- `XDG_CACHE_HOME=<empty> rake kotoshu:conformance:compare` → 0/0/0
|
|
85
|
+
(today it is 16/0/16; with a warm cache it is 0/0/0).
|
|
86
|
+
- Same on a machine with an expired cache (mock by backdating
|
|
87
|
+
`cached_at`).
|
|
88
|
+
- New CI job green on main; suite unchanged otherwise; suggestion
|
|
89
|
+
output for cache-cold machines now byte-identical to cache-warm
|
|
90
|
+
ones on the fixture corpus.
|
|
91
|
+
- Explicitly out of scope: regenerating any vector. The 2,630 are
|
|
92
|
+
correct; the data path was not.
|
|
93
|
+
|
|
94
|
+
## Surfaces
|
|
95
|
+
|
|
96
|
+
- `lib/kotoshu/suggestions/frequency_provider.rb`
|
|
97
|
+
- `lib/kotoshu/cache/base_cache.rb` (a load-explicit accessor if
|
|
98
|
+
needed) and the setup refresh path
|
|
99
|
+
- `lib/kotoshu/conformance_runner.rb` (pinned provider)
|
|
100
|
+
- `conformance/fixtures/` (frozen tier fixture), `.github/workflows/`
|
|
101
|
+
(compare job)
|
|
102
|
+
- specs for each behavior above
|
|
@@ -94,6 +94,20 @@ module Kotoshu
|
|
|
94
94
|
resource_files_exist?(resource_id)
|
|
95
95
|
end
|
|
96
96
|
|
|
97
|
+
# Whether readable cached data exists for +resource_id+, TTL
|
|
98
|
+
# ignored (plan 117). Expiry is a refresh signal for the setup /
|
|
99
|
+
# download paths, never a reason for readers to behave as if the
|
|
100
|
+
# data were absent: the bytes are checksummed at write time, so
|
|
101
|
+
# present-but-expired data is exactly the dataset it was.
|
|
102
|
+
#
|
|
103
|
+
# @param resource_id [String] The resource identifier
|
|
104
|
+
# @return [Boolean] True if metadata and data files exist
|
|
105
|
+
def cached_data?(resource_id)
|
|
106
|
+
return false unless supports_resource?(resource_id)
|
|
107
|
+
|
|
108
|
+
File.exist?(metadata_path_for(resource_id)) && resource_files_exist?(resource_id)
|
|
109
|
+
end
|
|
110
|
+
|
|
97
111
|
# Get a resource from cache or download it.
|
|
98
112
|
#
|
|
99
113
|
# @param resource_id [String] The resource identifier
|
|
@@ -55,6 +55,21 @@ module Kotoshu
|
|
|
55
55
|
KELLY_LANGUAGES.include?(resource_id)
|
|
56
56
|
end
|
|
57
57
|
|
|
58
|
+
# The frequency cache always roots under .../frequency-lists,
|
|
59
|
+
# including when a base cache path is injected (ResourceManager
|
|
60
|
+
# passes the shared cfg.cache_path, the CLI passes its root):
|
|
61
|
+
# without the suffix, writers and readers split across two
|
|
62
|
+
# layouts and the metadata goes permanently stale (plan 117).
|
|
63
|
+
#
|
|
64
|
+
# @param cache_path [String, nil] base cache directory
|
|
65
|
+
def initialize(cache_path: nil, **kwargs)
|
|
66
|
+
base = cache_path || default_cache_path
|
|
67
|
+
unless base.end_with?("frequency-lists")
|
|
68
|
+
base = File.join(base, "frequency-lists")
|
|
69
|
+
end
|
|
70
|
+
super(cache_path: base, **kwargs)
|
|
71
|
+
end
|
|
72
|
+
|
|
58
73
|
# Install a local frequency file into the cache without going
|
|
59
74
|
# through the network. Mirrors {LanguageCache#install_local} for
|
|
60
75
|
# the frequency resource type — symlinks the user's file into the
|
|
@@ -49,12 +49,17 @@ module Kotoshu
|
|
|
49
49
|
|
|
50
50
|
def try_load_from_frequency_cache(language_code)
|
|
51
51
|
cache = @frequency_cache || Cache::FrequencyCache.new
|
|
52
|
-
#
|
|
53
|
-
#
|
|
54
|
-
#
|
|
55
|
-
#
|
|
56
|
-
#
|
|
57
|
-
|
|
52
|
+
# Read cache-only: the suggestion hot path must never trigger a
|
|
53
|
+
# download (downloads happen only through explicit setup /
|
|
54
|
+
# `kotoshu cache download`). TTL expiry is a refresh signal,
|
|
55
|
+
# NOT a reason to switch datasets: expired-but-present bytes
|
|
56
|
+
# are still the checksummed Kelly tiers the conformance
|
|
57
|
+
# vectors freeze, while the YAML fallback below is a different,
|
|
58
|
+
# more coarsely curated set whose bonus differences re-scale
|
|
59
|
+
# every suggestion confidence (plan 117). So: any readable
|
|
60
|
+
# cached data wins; the YAML only serves machines with no
|
|
61
|
+
# frequency data at all.
|
|
62
|
+
return nil unless cache.cached_data?(language_code)
|
|
58
63
|
|
|
59
64
|
begin
|
|
60
65
|
cache.load_cached(language_code)
|
data/lib/kotoshu/version.rb
CHANGED
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: kotoshu
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 1.0.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Ribose Inc.
|
|
@@ -122,6 +122,7 @@ files:
|
|
|
122
122
|
- TODO.impl/114-typo-biencoder-thread.md
|
|
123
123
|
- TODO.impl/115-hybrid-retrieval-pricing.md
|
|
124
124
|
- TODO.impl/116-baseline-suggestion-skip.md
|
|
125
|
+
- TODO.impl/117-deterministic-frequency-tiers.md
|
|
125
126
|
- TODO.impl/20-encapsulation-cleanup.md
|
|
126
127
|
- TODO.impl/21-autoload-migration.md
|
|
127
128
|
- TODO.impl/22-url-pin-single-source-of-truth.md
|