kotoshu 0.11.1 → 1.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: b024cec75cd870973f2993b7eac7b4dba110867803896bff14588513b9a75cb4
4
- data.tar.gz: aa4a76db758b89a7ae14ac24f59905216be4a3fdc5e523a97c4c3275013da706
3
+ metadata.gz: 19518c2daced6ae79d4a6d02fc5a26f4c3ca7109a70d2210220893177312173d
4
+ data.tar.gz: 34f597de8432648f22a15a0ad1b09f71041eaff805c91bc9a1357b6f0e2aff56
5
5
  SHA512:
6
- metadata.gz: e882b3899dedf60569ca598741be502d6606964616186de31e57ab3c2150e0886f1b445be7147167fc69c880eb603b9c5d9a1b67e20d1f5c801884a0f6083f19
7
- data.tar.gz: 0ebadb70fffa35c2847268871ce0b030b9626989ad980ff8de02ab02bc53f62ee9f395b98c2e37d9ec4caab13bc91d7e1c5b69120cdc901c861c04d9dd02c91a
6
+ metadata.gz: ee21f0075b02b6aff17cd4e58262bdf0ad6356a69c1950be2493ee6a801970f5268769a35899fd6f651f27d0d50cee591ffedadf08c922f0623ecbdb3fb4bcc3
7
+ data.tar.gz: 34c321c65532ff5a175f0b535225bdda8937e7d72f537e6462d0e965ed6a5db3524ec63a65ecae34af2df9691d28285bfb14048ddda90483f28e5a438c3199da
@@ -0,0 +1,102 @@
1
+ # Plan 117 — Deterministic frequency tiers: stop the silent YAML fallback
2
+
3
+ Status: proposed (forensics complete; root cause proven)
4
+ Depends on: plan 82 (conformance runner), the 2,630-vector contract (plan 66+)
5
+ Blocks: the 1.0 train (audit checklist item 7 — "conformance vectors green at
6
+ the cut commit" — is not currently verifiable on a cache-cold machine)
7
+
8
+ ## What happened (forensics, 2026-09-10)
9
+
10
+ Preparing the 1.0 train (checklist item 7) ran `rake
11
+ kotoshu:conformance:compare` on main:
12
+
13
+ compare: 2630 vectors -- ruby failures: 16, native failures: 0, divergences: 16
14
+
15
+ All 16 failures are `suggest` vectors on the compoundrule{,2,3} fixtures:
16
+ same candidate words, same distances, different `confidence` values
17
+ (e.g. expected `b` = 0.952380952…, ruby = 0.8333333…).
18
+
19
+ Ruled out, with evidence:
20
+
21
+ - **Not plan 116 / not recent main.** v0.11.0 shows the identical 16
22
+ failures; the suggestions/, dictionary/, core/ trees have zero commits
23
+ between v0.9.2 and main.
24
+ - **Not the machine cache contents.** The downloaded Kelly `en.json`
25
+ sha256 `97535823f41a3f0…` matches the pin in kotoshu-rs
26
+ `suggest/frequency_data.rs` byte-for-byte; `load_cached` returns the
27
+ same 47/185/907 tier sizes rust embeds.
28
+ - **Not an engine disagreement.** With a live (unexpired) cache:
29
+
30
+ compare: 2630 vectors -- ruby failures: 0, native failures: 0,
31
+ backend divergences: 0
32
+
33
+ ### The actual mechanism
34
+
35
+ `Suggestions::FrequencyProvider` falls back
36
+ `FrequencyCache` → local YAML (`lib/kotoshu/data/common_words/en.yml`)
37
+ → empty tiers. `BaseCache#available?` returns false when the metadata's
38
+ `cached_at` is older than the 7-day TTL. **The two datasets differ**:
39
+ the local YAML's top_50 contains `a` (bonus 200); Kelly's top_1000 is
40
+ where `a` lives (bonus 50). The bonus shifts one candidate's enhanced
41
+ score, the min–max normalization re-scales every confidence on the
42
+ tiny fixtures, and 16 vectors "fail".
43
+
44
+ So any machine whose frequency cache is absent or >7 days old silently
45
+ runs ruby on the YAML dataset — different suggestion ranking from the
46
+ frozen contract and from the Rust engine. This is a **user-facing
47
+ determinism bug**, not conformance bookkeeping: suggestion output
48
+ depends on cache age.
49
+
50
+ Two aggravators found on the way:
51
+
52
+ - `Kotoshu.setup(:en, want: [:frequency])` reported
53
+ `frequency: :downloaded` but left `cached_at` at 2026-09-02 — either
54
+ the re-download was skipped on checksum match (fine) and the
55
+ timestamp was not refreshed (bug: guarantees later expiry), or the
56
+ report is wrong. Verify and fix.
57
+ - Nothing CI-gates `conformance:compare` on the gem side (the rs repo
58
+ gates its half only), which is why a two-week-old dataset switch was
59
+ invisible.
60
+
61
+ ## Fix design
62
+
63
+ 1. **Expired ≠ unusable.** In `FrequencyProvider#load`, when
64
+ `available?` is false but the on-disk cached data exists, load it
65
+ anyway (`load_cached` does no network). TTL governs *refresh*
66
+ (setup/`get` paths), never a silent switch to a different dataset.
67
+ The YAML fallback then only serves machines with NO frequency data
68
+ at all.
69
+ 2. **Hermetic conformance.** The runner must not depend on machine
70
+ cache state: check in the frozen Kelly tiers (the three sets +
71
+ sha256, mirroring rs `frequency_data.rs`) as a conformance fixture
72
+ and construct the engines with that provider data pinned. Then
73
+ `conformance:compare` is deterministic on any machine, cache or no
74
+ cache, and cross-verifies the fixture against the rs tables.
75
+ 3. **CI gate.** Add an Ubuntu job: `rake compile && rake
76
+ kotoshu:conformance:compare` on every push/PR. This invariant is
77
+ the product's core correctness claim; it must not rely on someone
78
+ remembering to run it.
79
+ 4. **Setup timestamp.** When setup skips a download because the
80
+ checksum matches, refresh `cached_at` (and report `:cached`).
81
+
82
+ ## Acceptance
83
+
84
+ - `XDG_CACHE_HOME=<empty> rake kotoshu:conformance:compare` → 0/0/0
85
+ (today it is 16/0/16; with a warm cache it is 0/0/0).
86
+ - Same on a machine with an expired cache (mock by backdating
87
+ `cached_at`).
88
+ - New CI job green on main; suite unchanged otherwise; suggestion
89
+ output for cache-cold machines now byte-identical to cache-warm
90
+ ones on the fixture corpus.
91
+ - Explicitly out of scope: regenerating any vector. The 2,630 are
92
+ correct; the data path was not.
93
+
94
+ ## Surfaces
95
+
96
+ - `lib/kotoshu/suggestions/frequency_provider.rb`
97
+ - `lib/kotoshu/cache/base_cache.rb` (a load-explicit accessor if
98
+ needed) and the setup refresh path
99
+ - `lib/kotoshu/conformance_runner.rb` (pinned provider)
100
+ - `conformance/fixtures/` (frozen tier fixture), `.github/workflows/`
101
+ (compare job)
102
+ - specs for each behavior above
@@ -94,6 +94,20 @@ module Kotoshu
94
94
  resource_files_exist?(resource_id)
95
95
  end
96
96
 
97
+ # Whether readable cached data exists for +resource_id+, TTL
98
+ # ignored (plan 117). Expiry is a refresh signal for the setup /
99
+ # download paths, never a reason for readers to behave as if the
100
+ # data were absent: the bytes are checksummed at write time, so
101
+ # present-but-expired data is exactly the dataset it was.
102
+ #
103
+ # @param resource_id [String] The resource identifier
104
+ # @return [Boolean] True if metadata and data files exist
105
+ def cached_data?(resource_id)
106
+ return false unless supports_resource?(resource_id)
107
+
108
+ File.exist?(metadata_path_for(resource_id)) && resource_files_exist?(resource_id)
109
+ end
110
+
97
111
  # Get a resource from cache or download it.
98
112
  #
99
113
  # @param resource_id [String] The resource identifier
@@ -55,6 +55,21 @@ module Kotoshu
55
55
  KELLY_LANGUAGES.include?(resource_id)
56
56
  end
57
57
 
58
+ # The frequency cache always roots under .../frequency-lists,
59
+ # including when a base cache path is injected (ResourceManager
60
+ # passes the shared cfg.cache_path, the CLI passes its root):
61
+ # without the suffix, writers and readers split across two
62
+ # layouts and the metadata goes permanently stale (plan 117).
63
+ #
64
+ # @param cache_path [String, nil] base cache directory
65
+ def initialize(cache_path: nil, **kwargs)
66
+ base = cache_path || default_cache_path
67
+ unless base.end_with?("frequency-lists")
68
+ base = File.join(base, "frequency-lists")
69
+ end
70
+ super(cache_path: base, **kwargs)
71
+ end
72
+
58
73
  # Install a local frequency file into the cache without going
59
74
  # through the network. Mirrors {LanguageCache#install_local} for
60
75
  # the frequency resource type — symlinks the user's file into the
@@ -49,12 +49,17 @@ module Kotoshu
49
49
 
50
50
  def try_load_from_frequency_cache(language_code)
51
51
  cache = @frequency_cache || Cache::FrequencyCache.new
52
- # Ask about actual cache state (TTL-aware available?), not the
53
- # static supported-language list — and read cache-only. The
54
- # suggestion hot path must never trigger a download; downloads
55
- # happen only through explicit setup (Kotoshu.setup /
56
- # kotoshu cache download).
57
- return nil unless cache.available?(language_code)
52
+ # Read cache-only: the suggestion hot path must never trigger a
53
+ # download (downloads happen only through explicit setup /
54
+ # `kotoshu cache download`). TTL expiry is a refresh signal,
55
+ # NOT a reason to switch datasets: expired-but-present bytes
56
+ # are still the checksummed Kelly tiers the conformance
57
+ # vectors freeze, while the YAML fallback below is a different,
58
+ # more coarsely curated set whose bonus differences re-scale
59
+ # every suggestion confidence (plan 117). So: any readable
60
+ # cached data wins; the YAML only serves machines with no
61
+ # frequency data at all.
62
+ return nil unless cache.cached_data?(language_code)
58
63
 
59
64
  begin
60
65
  cache.load_cached(language_code)
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Kotoshu
4
- VERSION = '0.11.1'
4
+ VERSION = '1.0.0'
5
5
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: kotoshu
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.11.1
4
+ version: 1.0.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Ribose Inc.
@@ -122,6 +122,7 @@ files:
122
122
  - TODO.impl/114-typo-biencoder-thread.md
123
123
  - TODO.impl/115-hybrid-retrieval-pricing.md
124
124
  - TODO.impl/116-baseline-suggestion-skip.md
125
+ - TODO.impl/117-deterministic-frequency-tiers.md
125
126
  - TODO.impl/20-encapsulation-cleanup.md
126
127
  - TODO.impl/21-autoload-migration.md
127
128
  - TODO.impl/22-url-pin-single-source-of-truth.md