kotoshu 1.0.2-aarch64-linux → 1.0.3-aarch64-linux
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/TODO.impl/124-owner-residue.md +15 -6
- data/TODO.impl/131-hybrid-productization.md +16 -1
- data/TODO.impl/133-rust-first-distribution.md +23 -1
- data/TODO.impl/134-post-131-improvements.md +67 -0
- data/TODO.impl/54-cjk-support.md +5 -2
- data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +29 -0
- data/lib/kotoshu/cache/model_cache.rb +159 -10
- data/lib/kotoshu/cli.rb +13 -1
- data/lib/kotoshu/configuration.rb +17 -2
- data/lib/kotoshu/resource_manager.rb +47 -0
- data/lib/kotoshu/spellchecker.rb +32 -2
- data/lib/kotoshu/suggestions/strategies/typo_retrieval_strategy.rb +55 -0
- data/lib/kotoshu/suggestions/typo_merge.rb +26 -0
- data/lib/kotoshu/suggestions.rb +2 -0
- data/lib/kotoshu/typo/engine.rb +125 -0
- data/lib/kotoshu/typo.rb +9 -0
- data/lib/kotoshu/version.rb +1 -1
- data/lib/kotoshu.rb +1 -0
- metadata +6 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: ee5adc450313e1bcb119db5e2241c83166cec21650e84fab745d2f118763983c
|
|
4
|
+
data.tar.gz: b2439602b868c0ac5851d502ef08bdb01e1a78883beeceec27511fe4ad996609
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 82f9569d119b24e47516dcf5fd3e5e102c3742c77c9ecafced3908303e6e0b30ba3a1b95589a1a31411e1dec15aabb2248c28fb52912478038268e40b5d87194
|
|
7
|
+
data.tar.gz: bb21d49ce30ae56f90309b4eb90c83aa6ccbc7f5e51d38d99181a12a9d00b363b735f91476aad454504d2505a15b052a909c2f3bf80feae96f10edc2a23f78c5
|
|
@@ -4,12 +4,21 @@ Status: standing inventory (updated 2026-09-10)
|
|
|
4
4
|
|
|
5
5
|
- VS Code Marketplace + Open VSX PATs → publish kotoshu-vscode
|
|
6
6
|
(extension is publish-ready; owner: "later").
|
|
7
|
-
- Worktree cleanup —
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
7
|
+
- ~~Worktree cleanup~~ — DONE 2026-09-12 under the owner's "make the
|
|
8
|
+
needed cleanups now": every repo down to its single main worktree,
|
|
9
|
+
merged branches deleted local+remote (see the memory entry).
|
|
10
|
+
- PyPI trusted publisher for kotoshu-native — owner registration
|
|
11
|
+
(project kotoshu-native, owner kotoshu, repo kotoshu-rs, workflow
|
|
12
|
+
release-pypi.yml, env blank), then rerun run 34681280341: the full
|
|
13
|
+
16-wheel matrix + sdist are BUILT and preserved as artifacts; the
|
|
14
|
+
publish step failed only on the missing registration.
|
|
15
|
+
- Typo layer registry cut — owner's three mechanical steps, runbooked
|
|
16
|
+
in models-fasttext-onnx docs/RELEASING.md; the moment the pair
|
|
17
|
+
carries vocab_url, `kotoshu setup LANG --typo` arms the layer.
|
|
18
|
+
- Gem 1.0.3 cut decision — main is ahead of 1.0.2 (opt-in typo layer,
|
|
19
|
+
--typo setup, Windows strict-wrap fix, eager GVL-released build);
|
|
20
|
+
nothing changes for default installs, so the cut timing and number
|
|
21
|
+
are the owner's call.
|
|
13
22
|
- kotoshu-lsp: still 0.1.1 — joins a future train if the owner wants
|
|
14
23
|
LSP on the 1.x line (audit train deliberately excluded it).
|
|
15
24
|
- Docker/kotoshu-ci + kotoshu-go/kotoshu-python floors: verify their
|
|
@@ -1,6 +1,21 @@
|
|
|
1
1
|
# Plan 131 — Productize the hybrid: C-retrieve + fastText-rescore in the engines
|
|
2
2
|
|
|
3
|
-
Status:
|
|
3
|
+
Status: executed on the engine and gem sides (2026-09-12; kotoshu-rs
|
|
4
|
+
PR #38 merged — the pure-Rust bi-encoder at parity cosine 1.00000 vs
|
|
5
|
+
onnxruntime, the int8 retrieval scan at 1.81 ms per suggest, the
|
|
6
|
+
frozen C-benchmark numbers reproduced EXACTLY through the shipped
|
|
7
|
+
engine path: top1 0.118772, top5 0.269829 on en real n=2509; wasm
|
|
8
|
+
loadTypo/typoSuggest + KPK1 typo sections + fp32 full-tier RowStore +
|
|
9
|
+
ruby bindings with a 28-assertion smoke; gem PR #191 — Typo::Engine
|
|
10
|
+
loader, TypoRetrievalStrategy, TypoMerge, KOTOSHU_TYPO_RETRIEVAL
|
|
11
|
+
opt-in with a byte-identical disabled passthrough, 27 specs, full
|
|
12
|
+
suite 4030/0). Gates: all three pre-declared clauses pass (numbers
|
|
13
|
+
within noise = exact on the decision metric; latency 1.8 ms against
|
|
14
|
+
the 10 ms budget; zero default change by construction and by spec).
|
|
15
|
+
REMAINING (owner-coupled): the registry promotion — the typo entry's
|
|
16
|
+
primary URL and vocab_url stay null until the owner cuts a registry
|
|
17
|
+
release carrying them; the server/playground/worker flips follow that
|
|
18
|
+
cut on the semantic-tier pattern
|
|
4
19
|
Depends on: plan 128 (verdict + artifacts), plan 113 (packs — the
|
|
5
20
|
delivery vehicle), registry v1.6.0 (typo-biencoder assets already opt-in)
|
|
6
21
|
|
|
@@ -1,6 +1,28 @@
|
|
|
1
1
|
# Plan 133 — Rust-first distribution: precompiled native gems and complete wheels
|
|
2
2
|
|
|
3
|
-
Status:
|
|
3
|
+
Status: executed (2026-09-12) — PR #189 (7-iteration CI chain: the
|
|
4
|
+
MinGW Rust toolchain for the ABI match, bindgen pointed at the
|
|
5
|
+
RubyInstaller devkit's ucrt64 headers through BINDGEN_EXTRA_CLANG_ARGS
|
|
6
|
+
with the windows-gnu target, macos-13 -> macos-15-intel for the retired
|
|
7
|
+
x86_64 macOS runner, and the release race fix so the native legs build
|
|
8
|
+
from the v<version> tag). Released as kotoshu 1.0.2: six artifacts on
|
|
9
|
+
RubyGems (ruby + x86_64-linux, aarch64-linux, arm64-darwin, x86_64-darwin,
|
|
10
|
+
x64-mingw-ucrt), verified by a toolchain-free scratch-GEM_HOME install
|
|
11
|
+
that resolved kotoshu-1.0.2-arm64-darwin, engaged the native engine
|
|
12
|
+
under the auto default, and passed the frozen vectors. Docker: both
|
|
13
|
+
images (docker-kotoshu-ci PR #2, kotoshu-server PR #9) dropped their
|
|
14
|
+
build-tool layers / gained the native-backend build assertion, with the
|
|
15
|
+
guard proving the platform gem resolves inside the slim builder. Site:
|
|
16
|
+
engines section resolution paragraph + the 1.0.2 news entry (092fc03).
|
|
17
|
+
Design deviation: no x86_64-linux-musl platform gem — there is no
|
|
18
|
+
hosted musl Ruby runner and cross needs a musl Ruby; Alpine resolves
|
|
19
|
+
the pure-Ruby gem (source-build of the extension remains possible
|
|
20
|
+
there). Python: the 16-wheel + sdist kotoshu-native 0.1.1 matrix is
|
|
21
|
+
built and smoke-green (run 34681280341) but the PyPI publish is gated
|
|
22
|
+
on the owner registering the trusted publisher (pypi.org, project
|
|
23
|
+
kotoshu-native, repo kotoshu-rs, workflow release-pypi.yml, env blank);
|
|
24
|
+
rerun `gh run rerun 34681280341 --repo kotoshu/kotoshu-rs --failed`
|
|
25
|
+
afterward — the wheel artifacts persist, no rebuild needed.
|
|
4
26
|
Depends on: the 1.0 conformance contract (identical outputs make the
|
|
5
27
|
backend default flip correctness-free); plan 93 (the python wheel matrix)
|
|
6
28
|
|
|
@@ -0,0 +1,67 @@
|
|
|
1
|
+
# Plan 134 — Post-131 improvements: the audit's honest gaps
|
|
2
|
+
|
|
3
|
+
Status: executed (2026-09-13). (1) GVL release rs PR #39 — the
|
|
4
|
+
without_gvl helper runs the typo compute through
|
|
5
|
+
rb_thread_call_without_gvl, PROVEN LIVE: a 5 ms ticker thread logged
|
|
6
|
+
4,118 ticks during the 24.8 s / 100k-word index build (a held GVL
|
|
7
|
+
starves it to ~0), suggest answers 2.2 ms after the eager build; gem
|
|
8
|
+
PR #195 makes Engine.for build eagerly. (2) native-suite CI leg (gem
|
|
9
|
+
PR #195): the conformance workflow compiles the extension and runs
|
|
10
|
+
the full suite — NativeBackend can no longer regress matrix-green.
|
|
11
|
+
(3) release verify (gem PR #195): the release asserts all six
|
|
12
|
+
RubyGems artifacts before reporting success. (4) runbook models PR
|
|
13
|
+
#37: docs/RELEASING.md carries the standard cut and the typo
|
|
14
|
+
promotion steps. (5) perf-weekly (gem PR #195): the weekly slow-suite
|
|
15
|
+
run whose first execution caught the suites rotted — repaired with
|
|
16
|
+
dated 2x-headroom bounds. Found and fixed en route (gem PR #196,
|
|
17
|
+
merged): setup_typo_remote rescued only Kotoshu::Error, leaking raw
|
|
18
|
+
Errno::ECONNREFUSED on Windows (caught by merge-preview CI on main).
|
|
19
|
+
|
|
20
|
+
Depends on: plan 131 (the typo layer), plan 133 (the platform gems)
|
|
21
|
+
|
|
22
|
+
## The items
|
|
23
|
+
|
|
24
|
+
### 1. The typo layer's first-suggest stall (production defect class)
|
|
25
|
+
|
|
26
|
+
The 100k-word index build (~15 s) happens lazily inside the first
|
|
27
|
+
`typo_suggest`, and the Ruby binding holds the GVL for the whole Rust
|
|
28
|
+
call — on the server, one arming request blocks every other request
|
|
29
|
+
for the build's duration. Fixes, both:
|
|
30
|
+
|
|
31
|
+
- The binding releases the GVL around the pure-Rust compute (index
|
|
32
|
+
build and scan touch no Ruby objects; results re-enter GVL scope to
|
|
33
|
+
build hashes), so a build never stops the world.
|
|
34
|
+
- The gem resolves EAGERLY: `Engine.for` builds the index at load —
|
|
35
|
+
the cost moves to opt-in setup where it belongs, never mid-check.
|
|
36
|
+
|
|
37
|
+
### 2. The native path is invisible to the main rake matrix
|
|
38
|
+
|
|
39
|
+
The nine rake legs report the native specs PENDING ("native extension
|
|
40
|
+
not built"); only the separate compare job compiles the ext. A
|
|
41
|
+
`NativeBackend` regression could land matrix-green. One dedicated
|
|
42
|
+
matrix job compiles the extension and runs the suite un-pended.
|
|
43
|
+
|
|
44
|
+
### 3. The release workflow trusts its own pushes
|
|
45
|
+
|
|
46
|
+
Nothing asserts the release is COMPLETE — a silently failed platform
|
|
47
|
+
push ships a partial release green. A `verify` job after the native
|
|
48
|
+
legs queries the RubyGems API for version x platform coverage (ruby +
|
|
49
|
+
the five native platforms) and fails on anything missing.
|
|
50
|
+
|
|
51
|
+
### 4. The owner's cut-day procedure lives in session memory
|
|
52
|
+
|
|
53
|
+
The three-step typo release checklist is recorded in memory and PR
|
|
54
|
+
bodies. The models repo carries the committed runbook instead — the
|
|
55
|
+
owner cuts the release without needing this session.
|
|
56
|
+
|
|
57
|
+
### 5. The weekly SLOW_TESTS dashboard, owner-free form
|
|
58
|
+
|
|
59
|
+
The T4.2 remainder as a workflow: weekly run of the slow suites, the
|
|
60
|
+
delta summary posted on the run page (a job summary + artifact — no
|
|
61
|
+
external dashboard to own).
|
|
62
|
+
|
|
63
|
+
## Gates
|
|
64
|
+
|
|
65
|
+
- Every fix ships with its spec or CI proof; no behavior change to
|
|
66
|
+
any frozen output (the GVL release is invisible; eager build moves
|
|
67
|
+
WHEN the same index is built, not what it computes).
|
data/TODO.impl/54-cjk-support.md
CHANGED
|
@@ -7,8 +7,11 @@ This is the largest tier 3 item: CJK languages need word-segmentation
|
|
|
7
7
|
(no spaces in zh/ja), different dictionary formats, and
|
|
8
8
|
romanization-aware suggestion strategies.
|
|
9
9
|
|
|
10
|
-
Status: outline only
|
|
11
|
-
|
|
10
|
+
Status: outline only, partially superseded. Japanese tokenization
|
|
11
|
+
shipped (the ja module with the suika tokenizer, live since the
|
|
12
|
+
language-modules wave); real zh/ko spell checking remains design
|
|
13
|
+
space — this TODO documents it, and real work still needs upstream
|
|
14
|
+
segmentation library choices and per-language fixture data.
|
|
12
15
|
|
|
13
16
|
## Per-language needs
|
|
14
17
|
|
|
@@ -1,5 +1,34 @@
|
|
|
1
1
|
# 56 — T4+ Quality, Architecture, and Ecosystem backlog
|
|
2
2
|
|
|
3
|
+
Status: largely superseded — most items below were promoted and
|
|
4
|
+
executed by the numbered plans; the remaining open ones are marked.
|
|
5
|
+
Snapshot annotations (2026-09-13): T4.1 shipped as the plan-147/148
|
|
6
|
+
perf waves (WordIndex + SweepIndex: suggest 20-140 s -> sub-second,
|
|
7
|
+
sweep 1472 -> 661 ms en; the Rust engine then took the sweep 10-15x
|
|
8
|
+
further, and plan 131's hybrid retrieval added 1.8 ms retrieval) with
|
|
9
|
+
published error budgets on /performance. T4.4 shipped as plan 97 (YARD
|
|
10
|
+
100% public surface, docs CI, hosted reference) plus the repeated
|
|
11
|
+
README truth passes. T5.2 and T5.3 shipped (Configuration as data
|
|
12
|
+
with the Resolver; WordResult/DocumentResult/SuggestionSet/Suggestion
|
|
13
|
+
on lutaml-model, hand-rolled to_h banned by the global rule). T6.1
|
|
14
|
+
shipped (kotoshu-lsp, live on RubyGems) and T6.2 shipped (the wasm
|
|
15
|
+
playground, @kotoshu/wasm, language packs). Still open: T4.2's weekly
|
|
16
|
+
SLOW_TESTS dashboard, T5.4 (= plan 53), and T6.3 (desktop dictionary
|
|
17
|
+
editor — future). CLOSED SINCE: T4.3's named properties shipped
|
|
18
|
+
(spec/properties/data_structure_properties_spec.rb — Trie
|
|
19
|
+
order-independence and totality, prefix agreement, set recovery, and
|
|
20
|
+
the every-suggestion-is-a-dictionary-word invariant; the affix
|
|
21
|
+
round-trip property remains the one open sub-item, gated on a
|
|
22
|
+
morphological API surface). T5.1 audited clean (2026-09-13): the
|
|
23
|
+
dedupe-once merge CompositeStrategy already performs is exactly the
|
|
24
|
+
fix T5.1 proposed (the code cites it); strategy purity holds with one
|
|
25
|
+
documented exception — the semantic-cascade threshold falls back to
|
|
26
|
+
the process Configuration when the composite was not given an
|
|
27
|
+
explicit one, by design and overridable; and the symspell/edit-1
|
|
28
|
+
overlap is moot — neither is wired into any default flow (the frozen
|
|
29
|
+
Generator owns the real path; the strategy classes are the composable
|
|
30
|
+
API, and the single-set merge makes overlapping compositions safe).
|
|
31
|
+
|
|
3
32
|
This file rolls up the T4 (Quality), T5 (Architecture), and T6
|
|
4
33
|
(Ecosystem) items from TODO.impl/39-tier3-and-beyond.md into
|
|
5
34
|
promotable units with concrete acceptance criteria.
|
|
@@ -88,6 +88,16 @@ module Kotoshu
|
|
|
88
88
|
# Directory (under the cache root) holding the LID artifact pair.
|
|
89
89
|
LID_DIRECTORY = "lid"
|
|
90
90
|
|
|
91
|
+
# The hybrid typo bi-encoder (plan 131): a language-less
|
|
92
|
+
# registry resource (kotoshu://models/typo/typo-biencoder) with
|
|
93
|
+
# a REQUIRED vocab sidecar, resolved exactly like the LID pair.
|
|
94
|
+
TYPO_LANGUAGE = "typo"
|
|
95
|
+
TYPO_TIER = "typo-biencoder"
|
|
96
|
+
TYPO_REGISTRY_ID = ModelRegistry::Resource.id_for(TYPO_LANGUAGE, TYPO_TIER)
|
|
97
|
+
|
|
98
|
+
# Cache directory (under the models root, beside the languages).
|
|
99
|
+
TYPO_DIRECTORY = File.join("models", "typo")
|
|
100
|
+
|
|
91
101
|
# Normalize and validate a tier name.
|
|
92
102
|
#
|
|
93
103
|
# @param tier [String, Symbol] "full", "fluency", or "mini"
|
|
@@ -224,9 +234,17 @@ module Kotoshu
|
|
|
224
234
|
def download_tiered_model(language_code, tier:, force_download: false)
|
|
225
235
|
tier = self.class.normalize_tier(tier)
|
|
226
236
|
lang = language_code.to_s
|
|
227
|
-
return get("#{lang}:onnx", force_download: force_download) if tier == :full
|
|
228
237
|
|
|
238
|
+
# The full tier rides the registry when the cached registry
|
|
239
|
+
# knows it (primary URL, vocab sibling, sha256 — the path the
|
|
240
|
+
# typo layer's rescore requires, since the legacy download
|
|
241
|
+
# never fetched the vocab). Offline without a cached registry
|
|
242
|
+
# the entry is nil and the legacy CDN path serves, preserving
|
|
243
|
+
# the pre-registry behavior exactly.
|
|
229
244
|
entry = registry_entry_for(lang, tier, force: force_download)
|
|
245
|
+
if tier == :full && entry.nil?
|
|
246
|
+
return get("#{lang}:onnx", force_download: force_download)
|
|
247
|
+
end
|
|
230
248
|
unless entry
|
|
231
249
|
raise Kotoshu::Error,
|
|
232
250
|
"no registry entry for #{ModelRegistry::Resource.id_for(lang, tier)}"
|
|
@@ -257,6 +275,7 @@ module Kotoshu
|
|
|
257
275
|
cached_at: Time.now.utc.iso8601,
|
|
258
276
|
source: "registry"
|
|
259
277
|
}
|
|
278
|
+
metadata[:vocab_file] = File.basename(vocab_path) if vocab_path
|
|
260
279
|
write_metadata(File.join(dest_path, "metadata.json"), metadata)
|
|
261
280
|
|
|
262
281
|
result = { model_path: model_file, metadata: metadata }
|
|
@@ -293,7 +312,7 @@ module Kotoshu
|
|
|
293
312
|
return nil unless File.exist?(onnx_file) && File.size(onnx_file).positive?
|
|
294
313
|
return nil unless File.exist?(vocab_file) && File.size(vocab_file).positive?
|
|
295
314
|
|
|
296
|
-
|
|
315
|
+
verify_model_pair_integrity!(metadata, onnx_file, vocab_file)
|
|
297
316
|
|
|
298
317
|
{ onnx_path: onnx_file, vocab_path: vocab_file, metadata: metadata }
|
|
299
318
|
end
|
|
@@ -354,6 +373,133 @@ module Kotoshu
|
|
|
354
373
|
{ onnx_path: onnx_file, vocab_path: vocab_file, metadata: metadata }
|
|
355
374
|
end
|
|
356
375
|
|
|
376
|
+
# ---- typo bi-encoder pair (plan 131) ----
|
|
377
|
+
|
|
378
|
+
# Whether the typo bi-encoder pair is cached and complete
|
|
379
|
+
# (both files, checksum-verified).
|
|
380
|
+
#
|
|
381
|
+
# @return [Boolean]
|
|
382
|
+
def typo_biencoder_cached?
|
|
383
|
+
!load_cached_typo_biencoder.nil?
|
|
384
|
+
end
|
|
385
|
+
|
|
386
|
+
# Resolve the typo bi-encoder pair from the cache only — never
|
|
387
|
+
# touches the network (the suggest path resolves; setup
|
|
388
|
+
# downloads, the same two-stage split as the LID pair).
|
|
389
|
+
#
|
|
390
|
+
# @return [Hash, nil] { onnx_path:, vocab_path:, metadata } when
|
|
391
|
+
# cached, nil when not set up
|
|
392
|
+
# @raise [Kotoshu::IntegrityError] cached bytes fail their
|
|
393
|
+
# recorded checksum
|
|
394
|
+
def load_cached_typo_biencoder
|
|
395
|
+
dir = File.join(@cache_path, TYPO_DIRECTORY)
|
|
396
|
+
metadata_path = File.join(dir, "metadata.json")
|
|
397
|
+
return nil unless File.exist?(metadata_path)
|
|
398
|
+
|
|
399
|
+
metadata = read_metadata(metadata_path)
|
|
400
|
+
return nil unless metadata
|
|
401
|
+
|
|
402
|
+
onnx_file = File.join(dir, metadata["file"].to_s)
|
|
403
|
+
vocab_file = File.join(dir, metadata["vocab_file"].to_s)
|
|
404
|
+
return nil unless File.exist?(onnx_file) && File.size(onnx_file).positive?
|
|
405
|
+
return nil unless File.exist?(vocab_file) && File.size(vocab_file).positive?
|
|
406
|
+
|
|
407
|
+
verify_model_pair_integrity!(
|
|
408
|
+
metadata, onnx_file, vocab_file,
|
|
409
|
+
directory: dir,
|
|
410
|
+
registry_id: TYPO_REGISTRY_ID,
|
|
411
|
+
remediation: "Run Kotoshu.setup_typo(language, force: true) to re-download."
|
|
412
|
+
)
|
|
413
|
+
|
|
414
|
+
{ onnx_path: onnx_file, vocab_path: vocab_file, metadata: metadata }
|
|
415
|
+
end
|
|
416
|
+
|
|
417
|
+
# Download the typo bi-encoder pair from the models registry,
|
|
418
|
+
# verifying the ONNX bytes against the registry sha256 (the
|
|
419
|
+
# setup half of the two-stage model). The vocab sidecar is
|
|
420
|
+
# required, not optional — the encoder cannot tokenize without
|
|
421
|
+
# its char vocabulary — so a registry entry without vocab_url
|
|
422
|
+
# (the pre-cut state) raises the honest error instead of
|
|
423
|
+
# caching a half-usable artifact.
|
|
424
|
+
#
|
|
425
|
+
# @param force [Boolean] re-fetch the registry first
|
|
426
|
+
# @return [Hash] { onnx_path:, vocab_path:, metadata }
|
|
427
|
+
# @raise [Kotoshu::Error] no entry, entry without vocab_url, or
|
|
428
|
+
# offline without a cached registry
|
|
429
|
+
# @raise [Kotoshu::IntegrityError] sha256 mismatch
|
|
430
|
+
def download_typo_biencoder(force: false)
|
|
431
|
+
entry = registry(force: force)&.find(TYPO_LANGUAGE, TYPO_TIER)
|
|
432
|
+
unless entry
|
|
433
|
+
raise Kotoshu::Error, "no registry entry for #{TYPO_REGISTRY_ID}"
|
|
434
|
+
end
|
|
435
|
+
if entry.vocab_url.nil? || entry.vocab_url.to_s.strip.empty?
|
|
436
|
+
raise Kotoshu::Error,
|
|
437
|
+
"#{TYPO_REGISTRY_ID} carries no vocab_url yet — the typo pair ships with a models registry release"
|
|
438
|
+
end
|
|
439
|
+
|
|
440
|
+
dir = File.join(@cache_path, TYPO_DIRECTORY)
|
|
441
|
+
FileUtils.mkdir_p(dir)
|
|
442
|
+
|
|
443
|
+
onnx_file = File.join(dir, filename_from_url(entry.urls.primary))
|
|
444
|
+
used_url = download_primary_or_mirror!(entry, onnx_file)
|
|
445
|
+
verify_registry_sha256!(entry, onnx_file, TYPO_REGISTRY_ID, used_url,
|
|
446
|
+
remediation: "Run Kotoshu::Typo.setup(force: true) to re-download.")
|
|
447
|
+
|
|
448
|
+
vocab_file = File.join(dir, filename_from_url(entry.vocab_url))
|
|
449
|
+
begin
|
|
450
|
+
download_file(entry.vocab_url, vocab_file)
|
|
451
|
+
rescue StandardError => e
|
|
452
|
+
File.delete(onnx_file) if File.exist?(onnx_file)
|
|
453
|
+
raise Kotoshu::Error, "typo vocab unavailable at #{entry.vocab_url}: #{e.message}"
|
|
454
|
+
end
|
|
455
|
+
|
|
456
|
+
metadata = {
|
|
457
|
+
version: entry.version,
|
|
458
|
+
url: used_url,
|
|
459
|
+
language: TYPO_LANGUAGE,
|
|
460
|
+
type: "typo-biencoder",
|
|
461
|
+
tier: TYPO_TIER,
|
|
462
|
+
file: File.basename(onnx_file),
|
|
463
|
+
vocab_file: File.basename(vocab_file),
|
|
464
|
+
checksum: entry.sha256,
|
|
465
|
+
registry_id: TYPO_REGISTRY_ID,
|
|
466
|
+
size_bytes: entry.size_bytes,
|
|
467
|
+
cached_at: Time.now.utc.iso8601,
|
|
468
|
+
source: "registry"
|
|
469
|
+
}
|
|
470
|
+
write_metadata(File.join(dir, "metadata.json"), metadata)
|
|
471
|
+
|
|
472
|
+
{ onnx_path: onnx_file, vocab_path: vocab_file, metadata: metadata }
|
|
473
|
+
end
|
|
474
|
+
|
|
475
|
+
# Resolve a tiered model from the cache only — the cache half of
|
|
476
|
+
# the two-stage model for tiers, symmetric with
|
|
477
|
+
# {#load_cached_lid} / {#load_cached_typo_biencoder}. Returns
|
|
478
|
+
# the vocab sibling when the registry-served download landed it.
|
|
479
|
+
#
|
|
480
|
+
# @param language_code [String]
|
|
481
|
+
# @param tier [String, Symbol]
|
|
482
|
+
# @return [Hash, nil] { model_path:, vocab_path?, metadata } or nil
|
|
483
|
+
def load_cached_tier(language_code, tier)
|
|
484
|
+
tier = self.class.normalize_tier(tier)
|
|
485
|
+
resource_id = tier_resource_id(language_code.to_s, tier)
|
|
486
|
+
dir = resource_dir_for(resource_id)
|
|
487
|
+
metadata_path = File.join(dir, "metadata.json")
|
|
488
|
+
return nil unless File.exist?(metadata_path)
|
|
489
|
+
|
|
490
|
+
metadata = read_metadata(metadata_path)
|
|
491
|
+
return nil unless metadata
|
|
492
|
+
|
|
493
|
+
model_file = File.join(dir, metadata["file"].to_s)
|
|
494
|
+
return nil unless File.exist?(model_file) && File.size(model_file).positive?
|
|
495
|
+
|
|
496
|
+
result = { model_path: model_file, metadata: metadata }
|
|
497
|
+
vocab = metadata["vocab_file"].to_s
|
|
498
|
+
vocab_path = File.join(dir, vocab)
|
|
499
|
+
result[:vocab_path] = vocab_path if !vocab.empty? && File.exist?(vocab_path)
|
|
500
|
+
result
|
|
501
|
+
end
|
|
502
|
+
|
|
357
503
|
# Get available model types for a language.
|
|
358
504
|
#
|
|
359
505
|
# @param language_code [String] ISO 639-1 language code
|
|
@@ -801,16 +947,19 @@ module Kotoshu
|
|
|
801
947
|
# @return [void]
|
|
802
948
|
# @raise [Kotoshu::IntegrityError, Kotoshu::Error] on mismatch;
|
|
803
949
|
# corrupt bytes are removed so the next setup re-downloads
|
|
804
|
-
def
|
|
805
|
-
|
|
950
|
+
def verify_model_pair_integrity!(metadata, onnx_file, vocab_file, directory: nil,
|
|
951
|
+
registry_id: nil, remediation: nil)
|
|
952
|
+
registry_id ||= LID_REGISTRY_ID
|
|
953
|
+
directory ||= File.join(@cache_path, LID_DIRECTORY)
|
|
954
|
+
remediation ||= "Run Kotoshu.setup_lid(force: true) to re-download."
|
|
806
955
|
if lfs_pointer?(onnx_file) || lfs_pointer?(vocab_file)
|
|
807
956
|
File.delete(onnx_file)
|
|
808
957
|
File.delete(vocab_file)
|
|
809
|
-
FileUtils.rm_f(File.join(
|
|
958
|
+
FileUtils.rm_f(File.join(directory, "metadata.json"))
|
|
810
959
|
raise Kotoshu::Error,
|
|
811
|
-
"Integrity verification failed for #{
|
|
812
|
-
"artifact is a Git LFS pointer stub, not model content.
|
|
813
|
-
"
|
|
960
|
+
"Integrity verification failed for #{registry_id}: cached " \
|
|
961
|
+
"artifact is a Git LFS pointer stub, not model content. " \
|
|
962
|
+
"#{remediation}"
|
|
814
963
|
end
|
|
815
964
|
|
|
816
965
|
expected = metadata["checksum"]
|
|
@@ -820,11 +969,11 @@ module Kotoshu
|
|
|
820
969
|
return if actual == expected
|
|
821
970
|
|
|
822
971
|
raise Kotoshu::IntegrityError.new(
|
|
823
|
-
|
|
972
|
+
registry_id,
|
|
824
973
|
expected: expected,
|
|
825
974
|
actual: actual,
|
|
826
975
|
url: metadata["url"],
|
|
827
|
-
remediation:
|
|
976
|
+
remediation: remediation
|
|
828
977
|
)
|
|
829
978
|
end
|
|
830
979
|
|
data/lib/kotoshu/cli.rb
CHANGED
|
@@ -146,6 +146,7 @@ module Kotoshu
|
|
|
146
146
|
kotoshu setup en de fr # download from GitHub
|
|
147
147
|
kotoshu setup en --want spelling,frequency # also fetch Kelly list
|
|
148
148
|
kotoshu setup en --model --tier mini # model at the mini tier
|
|
149
|
+
kotoshu setup en --typo # typo bi-encoder + full tier
|
|
149
150
|
kotoshu setup en --aff /p/en.aff --dic /p/en.dic
|
|
150
151
|
kotoshu setup en --from /usr/share/hunspell/
|
|
151
152
|
kotoshu setup --force en # re-download
|
|
@@ -167,6 +168,10 @@ module Kotoshu
|
|
|
167
168
|
type: :boolean,
|
|
168
169
|
default: false,
|
|
169
170
|
desc: "Fetch the ONNX model (shorthand for adding model to --want)"
|
|
171
|
+
method_option :typo,
|
|
172
|
+
type: :boolean,
|
|
173
|
+
default: false,
|
|
174
|
+
desc: "Fetch the typo-retrieval pair (bi-encoder + full tier; needs a registry release carrying it)"
|
|
170
175
|
method_option :tier,
|
|
171
176
|
type: :string,
|
|
172
177
|
desc: "Model tier: full, fluency, or mini (default: full)"
|
|
@@ -192,6 +197,7 @@ module Kotoshu
|
|
|
192
197
|
|
|
193
198
|
want = (options[:want] || "spelling").split(",").map(&:strip).map(&:to_sym)
|
|
194
199
|
want << :model if options[:model] && !want.include?(:model)
|
|
200
|
+
want << :typo if options[:typo] && !want.include?(:typo)
|
|
195
201
|
opts = setup_source_options(languages)
|
|
196
202
|
opts[:want] = want
|
|
197
203
|
opts[:tier] = options[:tier].to_sym if options[:tier]
|
|
@@ -234,6 +240,10 @@ module Kotoshu
|
|
|
234
240
|
type: :boolean,
|
|
235
241
|
default: false,
|
|
236
242
|
desc: "Fetch the ONNX model (shorthand for adding model to --want)"
|
|
243
|
+
method_option :typo,
|
|
244
|
+
type: :boolean,
|
|
245
|
+
default: false,
|
|
246
|
+
desc: "Fetch the typo-retrieval pair (bi-encoder + full tier; needs a registry release carrying it)"
|
|
237
247
|
method_option :tier,
|
|
238
248
|
type: :string,
|
|
239
249
|
desc: "Model tier: full, fluency, or mini (default: full)"
|
|
@@ -525,8 +535,10 @@ module Kotoshu
|
|
|
525
535
|
frequency = result.frequency || "skipped"
|
|
526
536
|
model = result.model || "skipped"
|
|
527
537
|
model += " (#{result.model_tier} tier)" if result.model && result.model_tier
|
|
538
|
+
typo = result.typo || "skipped"
|
|
528
539
|
source = result.source
|
|
529
|
-
puts "OK (spelling: #{spelling}, frequency: #{frequency}, model: #{model},
|
|
540
|
+
puts "OK (spelling: #{spelling}, frequency: #{frequency}, model: #{model}, " \
|
|
541
|
+
"typo: #{typo}, source: #{source})"
|
|
530
542
|
end
|
|
531
543
|
|
|
532
544
|
def list_setup
|
|
@@ -206,10 +206,16 @@ module Kotoshu
|
|
|
206
206
|
},
|
|
207
207
|
backend: {
|
|
208
208
|
env: "KOTOSHU_BACKEND",
|
|
209
|
-
default: "
|
|
210
|
-
description: "Engine backend for correct?/suggest (auto
|
|
209
|
+
default: "auto",
|
|
210
|
+
description: "Engine backend for correct?/suggest (auto: native when the extension is available, ruby otherwise; or force either)",
|
|
211
211
|
type: String
|
|
212
212
|
},
|
|
213
|
+
typo_retrieval: {
|
|
214
|
+
env: "KOTOSHU_TYPO_RETRIEVAL",
|
|
215
|
+
default: false,
|
|
216
|
+
description: "Opt-in hybrid typo retrieval (plan 131): the native bi-encoder slate merges ahead of frequency ranking; needs the native backend",
|
|
217
|
+
type: :boolean
|
|
218
|
+
},
|
|
213
219
|
personal_dictionary: {
|
|
214
220
|
env: "KOTOSHU_PERSONAL_DICTIONARY",
|
|
215
221
|
default: true,
|
|
@@ -241,6 +247,7 @@ module Kotoshu
|
|
|
241
247
|
# conformance contract), so the default is the faster engine
|
|
242
248
|
# with the pure fallback, never a failure.
|
|
243
249
|
backend: "auto",
|
|
250
|
+
typo_retrieval: false,
|
|
244
251
|
personal_dictionary: true
|
|
245
252
|
}.freeze
|
|
246
253
|
|
|
@@ -301,6 +308,12 @@ module Kotoshu
|
|
|
301
308
|
# NativeBackend for the boundary — the semantic path stays Ruby.
|
|
302
309
|
attr_accessor :backend
|
|
303
310
|
|
|
311
|
+
# @return [Boolean] Opt-in hybrid typo retrieval (plan 131): the
|
|
312
|
+
# native bi-encoder's rescored slate merges ahead of frequency
|
|
313
|
+
# ranking when the native backend serves the language. Env:
|
|
314
|
+
# KOTOSHU_TYPO_RETRIEVAL.
|
|
315
|
+
attr_accessor :typo_retrieval
|
|
316
|
+
|
|
304
317
|
# @return [Boolean] Whether Spellchecker#check accepts words from
|
|
305
318
|
# the user personal dictionary (~/.config/kotoshu/personal.dic,
|
|
306
319
|
# KOTOSHU_PERSONAL_DIC override) instead of flagging them as
|
|
@@ -511,6 +524,7 @@ module Kotoshu
|
|
|
511
524
|
max_cache_size: @max_cache_size,
|
|
512
525
|
semantic_cascade_threshold: @semantic_cascade_threshold,
|
|
513
526
|
backend: @backend,
|
|
527
|
+
typo_retrieval: @typo_retrieval,
|
|
514
528
|
personal_dictionary: @personal_dictionary
|
|
515
529
|
}
|
|
516
530
|
end
|
|
@@ -629,6 +643,7 @@ module Kotoshu
|
|
|
629
643
|
@resource_pin = DEFAULTS[:resource_pin]
|
|
630
644
|
@semantic_cascade_threshold = DEFAULTS[:semantic_cascade_threshold]
|
|
631
645
|
@backend = DEFAULTS[:backend]
|
|
646
|
+
@typo_retrieval = DEFAULTS[:typo_retrieval]
|
|
632
647
|
@personal_dictionary = DEFAULTS[:personal_dictionary]
|
|
633
648
|
end
|
|
634
649
|
|
|
@@ -41,6 +41,7 @@ module Kotoshu
|
|
|
41
41
|
:frequency, # :downloaded | :local | :cached | :unavailable | nil
|
|
42
42
|
:model, # :downloaded | :cached | :unavailable | nil
|
|
43
43
|
:model_tier, # :full | :fluency | :mini | nil (tier actually set up)
|
|
44
|
+
:typo, # :downloaded | :cached | :unavailable | nil (plan 131 pair)
|
|
44
45
|
:source, # :kotoshu | :local
|
|
45
46
|
keyword_init: true
|
|
46
47
|
) do
|
|
@@ -280,6 +281,11 @@ module Kotoshu
|
|
|
280
281
|
when :frequency
|
|
281
282
|
fc = frequency_cache_for
|
|
282
283
|
fc.supports_resource?(lang) && fc.available?(lang)
|
|
284
|
+
when :typo
|
|
285
|
+
# Both halves: the language-less pair and the language's full
|
|
286
|
+
# tier (the rescore's exactness contract).
|
|
287
|
+
cache = model_cache_for
|
|
288
|
+
cache.typo_biencoder_cached? && cache.available?(cache.tier_resource_id(lang, :full))
|
|
283
289
|
when :model
|
|
284
290
|
cache = model_cache_for
|
|
285
291
|
if tier.nil?
|
|
@@ -343,16 +349,57 @@ module Kotoshu
|
|
|
343
349
|
end
|
|
344
350
|
end
|
|
345
351
|
|
|
352
|
+
typo_status = if want.include?(:typo)
|
|
353
|
+
setup_typo_remote(lang, force: force, strict: strict, config: config,
|
|
354
|
+
tier: model_tier)
|
|
355
|
+
end
|
|
356
|
+
|
|
346
357
|
SetupResult.new(
|
|
347
358
|
language: lang,
|
|
348
359
|
spelling: spelling_status,
|
|
349
360
|
frequency: frequency_status,
|
|
350
361
|
model: model_status,
|
|
351
362
|
model_tier: model_tier,
|
|
363
|
+
typo: typo_status,
|
|
352
364
|
source: :kotoshu
|
|
353
365
|
)
|
|
354
366
|
end
|
|
355
367
|
|
|
368
|
+
# The typo layer's setup half (plan 131): the language-less
|
|
369
|
+
# bi-encoder pair, plus the language's FULL tier when this run did
|
|
370
|
+
# not already fetch it (the hybrid's rescore is full-tier-exact by
|
|
371
|
+
# contract — a mini tier would rank wrong, so full is fetched
|
|
372
|
+
# explicitly, not defaulted).
|
|
373
|
+
#
|
|
374
|
+
# Returns :cached when both halves were already present,
|
|
375
|
+
# :downloaded when anything was fetched, :unavailable when the
|
|
376
|
+
# registry cannot serve the pair yet (strict: raises instead —
|
|
377
|
+
# the pre-cut state).
|
|
378
|
+
def setup_typo_remote(lang, force:, strict:, config:, tier: nil)
|
|
379
|
+
cache = model_cache_for(config: config)
|
|
380
|
+
unless force
|
|
381
|
+
full_id = cache.tier_resource_id(lang, :full)
|
|
382
|
+
return :cached if cache.typo_biencoder_cached? && cache.available?(full_id)
|
|
383
|
+
end
|
|
384
|
+
begin
|
|
385
|
+
cache.download_typo_biencoder(force: force)
|
|
386
|
+
cache.download_tiered_model(lang, tier: :full, force_download: force) unless tier == :full
|
|
387
|
+
rescue Kotoshu::Error
|
|
388
|
+
raise if strict
|
|
389
|
+
|
|
390
|
+
return :unavailable
|
|
391
|
+
rescue StandardError => e
|
|
392
|
+
# Windows surfaces closed-port fetches as raw Errno classes the
|
|
393
|
+
# cache does not always wrap; the two-stage contract is
|
|
394
|
+
# setup-degrades unless strict, and strict reports through the
|
|
395
|
+
# typed error like every setup path.
|
|
396
|
+
raise Kotoshu::Error, "typo setup failed: #{e.class}: #{e.message}" if strict
|
|
397
|
+
|
|
398
|
+
return :unavailable
|
|
399
|
+
end
|
|
400
|
+
:downloaded
|
|
401
|
+
end
|
|
402
|
+
|
|
356
403
|
def setup_frequency_remote(lang, force:, strict:, config:)
|
|
357
404
|
cache = frequency_cache_for(config: config)
|
|
358
405
|
return :unavailable unless cache.supports_resource?(lang)
|
data/lib/kotoshu/spellchecker.rb
CHANGED
|
@@ -140,9 +140,39 @@ module Kotoshu
|
|
|
140
140
|
def suggest(word, max_suggestions: nil)
|
|
141
141
|
return Suggestions::SuggestionSet.empty if word.nil? || word.empty?
|
|
142
142
|
|
|
143
|
-
|
|
143
|
+
base = if @native_backend
|
|
144
|
+
@native_backend.suggest(word, max_suggestions: max_suggestions)
|
|
145
|
+
else
|
|
146
|
+
@generator.generate(word, max_suggestions: max_suggestions)
|
|
147
|
+
end
|
|
148
|
+
merge_typo_retrieval(base, word, max_suggestions: max_suggestions)
|
|
149
|
+
end
|
|
150
|
+
|
|
151
|
+
# Merge the opt-in typo-retrieval slate (plan 131) AHEAD of the
|
|
152
|
+
# base suggestions: the bi-encoder surfaces corrections a
|
|
153
|
+
# frequency-ranked sweep misses, so its rows lead. Inert unless
|
|
154
|
+
# typo_retrieval is configured AND the native engine loaded for
|
|
155
|
+
# this language — a disabled layer is a passthrough, never an
|
|
156
|
+
# error, never a different order.
|
|
157
|
+
def merge_typo_retrieval(base, word, max_suggestions: nil)
|
|
158
|
+
return base unless @config.typo_retrieval
|
|
159
|
+
|
|
160
|
+
engine = typo_retrieval_engine
|
|
161
|
+
return base unless engine
|
|
162
|
+
|
|
163
|
+
typo = engine.suggest(word, max_suggestions: max_suggestions)
|
|
164
|
+
limit = max_suggestions || @config.max_suggestions
|
|
165
|
+
Suggestions::TypoMerge.call(base: base, typo: typo, limit: limit)
|
|
166
|
+
end
|
|
167
|
+
|
|
168
|
+
# The typo engine for this language, resolved once. A nil result
|
|
169
|
+
# is memoized too — an unavailable layer must not retry its
|
|
170
|
+
# artifact resolution on every suggestion.
|
|
171
|
+
def typo_retrieval_engine
|
|
172
|
+
return @typo_retrieval_engine if defined?(@typo_retrieval_engine)
|
|
144
173
|
|
|
145
|
-
@
|
|
174
|
+
code = (@resource_bundle&.language || @config.language).to_s.split("-").first
|
|
175
|
+
@typo_retrieval_engine = Kotoshu::Typo::Engine.for(code, configuration: @config)
|
|
146
176
|
end
|
|
147
177
|
|
|
148
178
|
# Check a word and return a result object.
|
|
@@ -0,0 +1,55 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Kotoshu
|
|
4
|
+
module Suggestions
|
|
5
|
+
module Strategies
|
|
6
|
+
# The hybrid typo-retrieval strategy (plan 131): a frozen char
|
|
7
|
+
# bi-encoder retrieves dictionary candidates a frequency-ranked
|
|
8
|
+
# sweep misses, the fastText full tier rescores them, and the
|
|
9
|
+
# rows join the suggestion pipeline AHEAD of frequency ranking
|
|
10
|
+
# — the layer measured at +6.3 pp top-5 on the frozen
|
|
11
|
+
# benchmark. Opt-in and native-only; the engine is injected so
|
|
12
|
+
# the strategy itself stays pure orchestration.
|
|
13
|
+
class TypoRetrievalStrategy < BaseStrategy
|
|
14
|
+
# @return [#suggest] the typo engine ({Typo::Engine} in
|
|
15
|
+
# production; anything answering #suggest(word) in specs)
|
|
16
|
+
attr_reader :engine
|
|
17
|
+
|
|
18
|
+
# @param engine [#suggest] the typo engine
|
|
19
|
+
# @param config [Hash] additional configuration
|
|
20
|
+
def initialize(engine:, **config)
|
|
21
|
+
super(name: :typo_retrieval, **config)
|
|
22
|
+
@engine = engine
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
# Generate the typo-retrieval slate for the context's word.
|
|
26
|
+
#
|
|
27
|
+
# @param context [Context] the suggestion context
|
|
28
|
+
# @return [SuggestionSet]
|
|
29
|
+
def generate(context)
|
|
30
|
+
return SuggestionSet.empty unless enabled? && @engine
|
|
31
|
+
|
|
32
|
+
@engine.suggest(context.word, max_suggestions: max_results)
|
|
33
|
+
end
|
|
34
|
+
|
|
35
|
+
# This strategy is a candidate generator for misspellings; it
|
|
36
|
+
# has no real-word-error story, which the semantic strategy
|
|
37
|
+
# owns.
|
|
38
|
+
#
|
|
39
|
+
# @param context [Context]
|
|
40
|
+
# @return [Boolean]
|
|
41
|
+
def handles?(context)
|
|
42
|
+
enabled? && @engine && !context.word.empty?
|
|
43
|
+
end
|
|
44
|
+
|
|
45
|
+
# The strategy's rows already carry their own order (the
|
|
46
|
+
# rescored ranking); do not let a confident base set skip it.
|
|
47
|
+
#
|
|
48
|
+
# @return [Boolean]
|
|
49
|
+
def skip_when_confident?
|
|
50
|
+
false
|
|
51
|
+
end
|
|
52
|
+
end
|
|
53
|
+
end
|
|
54
|
+
end
|
|
55
|
+
end
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Kotoshu
|
|
4
|
+
module Suggestions
|
|
5
|
+
# The typo-retrieval merge (plan 131): the bi-encoder's rescored
|
|
6
|
+
# slate joins AHEAD of the base suggestions — it surfaces
|
|
7
|
+
# corrections a frequency-ranked sweep misses — with duplicates
|
|
8
|
+
# (same word) dropped from the base side and the set capped at the
|
|
9
|
+
# caller's limit. A pure function over real SuggestionSets; the
|
|
10
|
+
# spellchecker calls it only when the opt-in layer is armed.
|
|
11
|
+
module TypoMerge
|
|
12
|
+
# @param base [SuggestionSet] the engine's own suggestions
|
|
13
|
+
# @param typo [SuggestionSet] the typo-retrieval slate
|
|
14
|
+
# @param limit [Integer] final cap
|
|
15
|
+
# @return [SuggestionSet] merged set, order adopted verbatim
|
|
16
|
+
def self.call(base:, typo:, limit:)
|
|
17
|
+
return base if typo.empty?
|
|
18
|
+
|
|
19
|
+
typo_words = typo.map(&:word)
|
|
20
|
+
merged = typo.suggestions +
|
|
21
|
+
base.suggestions.reject { |suggestion| typo_words.include?(suggestion.word) }
|
|
22
|
+
SuggestionSet.new(merged.first(limit), max_size: limit, ranked: true)
|
|
23
|
+
end
|
|
24
|
+
end
|
|
25
|
+
end
|
|
26
|
+
end
|
data/lib/kotoshu/suggestions.rb
CHANGED
|
@@ -12,6 +12,7 @@ module Kotoshu
|
|
|
12
12
|
autoload :Suggestion, "kotoshu/suggestions/suggestion"
|
|
13
13
|
autoload :SuggestionSet, "kotoshu/suggestions/suggestion_set"
|
|
14
14
|
autoload :SweepIndex, "kotoshu/suggestions/sweep_index"
|
|
15
|
+
autoload :TypoMerge, "kotoshu/suggestions/typo_merge"
|
|
15
16
|
|
|
16
17
|
# Strategies sub-namespace.
|
|
17
18
|
module Strategies
|
|
@@ -23,6 +24,7 @@ module Kotoshu
|
|
|
23
24
|
autoload :PhoneticStrategy, "kotoshu/suggestions/strategies/phonetic_strategy"
|
|
24
25
|
autoload :SemanticStrategy, "kotoshu/suggestions/strategies/semantic_strategy"
|
|
25
26
|
autoload :SymspellStrategy, "kotoshu/suggestions/strategies/symspell_strategy"
|
|
27
|
+
autoload :TypoRetrievalStrategy, "kotoshu/suggestions/strategies/typo_retrieval_strategy"
|
|
26
28
|
autoload :WordIndex, "kotoshu/suggestions/strategies/word_index"
|
|
27
29
|
end
|
|
28
30
|
end
|
|
@@ -0,0 +1,125 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Kotoshu
|
|
4
|
+
module Typo
|
|
5
|
+
# The hybrid typo-retrieval engine (plan 131): the frozen char
|
|
6
|
+
# bi-encoder retrieves a top-20 vocabulary slate for a
|
|
7
|
+
# misspelling and the fastText full tier rescores it — the layer
|
|
8
|
+
# the models repo measured at +6.3 pp top-5 over frequency
|
|
9
|
+
# ranking on the frozen benchmark.
|
|
10
|
+
#
|
|
11
|
+
# The engine itself lives in the native extension
|
|
12
|
+
# ({Kotoshu::Native::TypoEngine}); this class is the Ruby-side
|
|
13
|
+
# loader: it resolves the artifact pairs through the model cache
|
|
14
|
+
# (the registry resources `kotoshu://models/typo/typo-biencoder`
|
|
15
|
+
# and the language's full tier), constructs the native engine
|
|
16
|
+
# once, and answers {#suggest} as a SuggestionSet the pipeline
|
|
17
|
+
# merges ahead of frequency ranking.
|
|
18
|
+
#
|
|
19
|
+
# The layer is opt-in and native-only: {for} returns nil — never
|
|
20
|
+
# raises — when the extension is absent or the artifacts cannot
|
|
21
|
+
# resolve, the same silent-degrade contract the semantic tier and
|
|
22
|
+
# the LID detector follow.
|
|
23
|
+
class Engine
|
|
24
|
+
# @return [Kotoshu::Native::TypoEngine, nil] the loaded native engine
|
|
25
|
+
attr_reader :native_engine
|
|
26
|
+
|
|
27
|
+
# @return [String] the language the tier was loaded for
|
|
28
|
+
attr_reader :language
|
|
29
|
+
|
|
30
|
+
# Load the engine for a language. Yielded block receives the
|
|
31
|
+
# cache to use (defaults to the configured model cache) — specs
|
|
32
|
+
# inject a stub cache with local artifact paths.
|
|
33
|
+
#
|
|
34
|
+
# @param language [String] ISO 639-1 code
|
|
35
|
+
# @param configuration [Configuration]
|
|
36
|
+
# @param cache [Cache::ModelCache, nil] explicit cache
|
|
37
|
+
# @return [Engine, nil] nil when anything in the chain is absent
|
|
38
|
+
def self.for(language, configuration: Kotoshu.configuration, cache: nil)
|
|
39
|
+
return nil unless defined?(Kotoshu::Native::TypoModel)
|
|
40
|
+
return nil unless Kotoshu::Native.available?
|
|
41
|
+
|
|
42
|
+
cache ||= Cache::ModelCache.new(cache_path: configuration.cache_path)
|
|
43
|
+
typo = cache.load_cached_typo_biencoder
|
|
44
|
+
tier = cache.load_cached_tier(language, :full)
|
|
45
|
+
return nil unless typo && tier && tier[:vocab_path]
|
|
46
|
+
|
|
47
|
+
native = Kotoshu::Native::TypoModel.load(typo[:onnx_path], typo[:vocab_path])
|
|
48
|
+
native_tier = Kotoshu::Native::TypoTier.load(tier[:model_path], tier[:vocab_path])
|
|
49
|
+
engine = new(Kotoshu::Native::TypoEngine.new(native, native_tier),
|
|
50
|
+
native_tier: native_tier, language: language)
|
|
51
|
+
# EAGER index build (plan 134): the multi-second derivation
|
|
52
|
+
# runs HERE - with the GVL released by the binding - instead
|
|
53
|
+
# of stalling the first suggest (and every Ruby thread under a
|
|
54
|
+
# held GVL). The cost lands at opt-in setup where it belongs.
|
|
55
|
+
engine.build_index
|
|
56
|
+
engine
|
|
57
|
+
rescue StandardError
|
|
58
|
+
# The opt-in layer never breaks a check: any load failure
|
|
59
|
+
# (corrupt artifact, engine error) degrades to "absent".
|
|
60
|
+
nil
|
|
61
|
+
end
|
|
62
|
+
|
|
63
|
+
# Derive the per-language index now (the eager half; the
|
|
64
|
+
# binding releases the GVL and memoizes - building again is a
|
|
65
|
+
# no-op). Returns the indexed vocabulary size.
|
|
66
|
+
#
|
|
67
|
+
# @return [Integer]
|
|
68
|
+
def build_index
|
|
69
|
+
@native_engine.build_index(@native_tier)
|
|
70
|
+
end
|
|
71
|
+
|
|
72
|
+
# @param native_engine [Kotoshu::Native::TypoEngine]
|
|
73
|
+
# @param native_tier [Kotoshu::Native::TypoTier] the rescore
|
|
74
|
+
# tier (kept for the eager index build)
|
|
75
|
+
# @param language [String]
|
|
76
|
+
def initialize(native_engine, language:, native_tier: nil)
|
|
77
|
+
@native_engine = native_engine
|
|
78
|
+
@native_tier = native_tier
|
|
79
|
+
@language = language
|
|
80
|
+
end
|
|
81
|
+
|
|
82
|
+
# The rescored slate for a word as a SuggestionSet, source
|
|
83
|
+
# :typo_retrieval, cosine confidences clamped to [0, 1]. Empty
|
|
84
|
+
# for a word outside the tier vocabulary (the hybrid's honest
|
|
85
|
+
# in-vocab rule — there is nothing the layer can say).
|
|
86
|
+
#
|
|
87
|
+
# @param word [String]
|
|
88
|
+
# @param max_suggestions [Integer, nil]
|
|
89
|
+
# @return [Suggestions::SuggestionSet]
|
|
90
|
+
def suggest(word, max_suggestions: nil)
|
|
91
|
+
limit = max_suggestions || DEFAULT_SLATE
|
|
92
|
+
rows = @native_engine.typo_suggest(word)
|
|
93
|
+
suggestions = rows.first(limit).map do |row|
|
|
94
|
+
Suggestions::Suggestion.new(
|
|
95
|
+
word: row["word"],
|
|
96
|
+
distance: 0,
|
|
97
|
+
confidence: row["score"].clamp(0.0, 1.0),
|
|
98
|
+
source: :typo_retrieval
|
|
99
|
+
)
|
|
100
|
+
end
|
|
101
|
+
Suggestions::SuggestionSet.new(suggestions, max_size: limit, ranked: true)
|
|
102
|
+
end
|
|
103
|
+
|
|
104
|
+
DEFAULT_SLATE = 20
|
|
105
|
+
end
|
|
106
|
+
end
|
|
107
|
+
|
|
108
|
+
# The setup half of the two-stage model (plan 131): download the
|
|
109
|
+
# typo bi-encoder pair and the language's full tier with its vocab
|
|
110
|
+
# sibling, so {Engine.for} can resolve cache-only at suggest time.
|
|
111
|
+
# Raises the same errors {Cache::ModelCache#download_typo_biencoder}
|
|
112
|
+
# and {Cache::ModelCache#download_tiered_model} raise — setup
|
|
113
|
+
# callers report, they do not degrade.
|
|
114
|
+
#
|
|
115
|
+
# @param language [String] ISO 639-1 code
|
|
116
|
+
# @param configuration [Configuration]
|
|
117
|
+
# @param force [Boolean] re-fetch the registry first
|
|
118
|
+
# @return [Hash] { typo: {onnx_path, vocab_path}, tier: {model_path, vocab_path?} }
|
|
119
|
+
def self.setup_typo(language, configuration: Kotoshu.configuration, force: false)
|
|
120
|
+
cache = Cache::ModelCache.new(cache_path: configuration.cache_path)
|
|
121
|
+
typo = cache.download_typo_biencoder(force: force)
|
|
122
|
+
tier = cache.download_tiered_model(language, tier: :full, force_download: force)
|
|
123
|
+
{ typo: typo, tier: tier }
|
|
124
|
+
end
|
|
125
|
+
end
|
data/lib/kotoshu/typo.rb
ADDED
data/lib/kotoshu/version.rb
CHANGED
data/lib/kotoshu.rb
CHANGED
|
@@ -44,6 +44,7 @@ module Kotoshu
|
|
|
44
44
|
# autoload so a missing extension can never break `require "kotoshu"`.
|
|
45
45
|
autoload :Native, "kotoshu/native"
|
|
46
46
|
autoload :NativeBackend, "kotoshu/native_backend"
|
|
47
|
+
autoload :Typo, "kotoshu/typo"
|
|
47
48
|
autoload :Readers, "kotoshu/readers"
|
|
48
49
|
autoload :ResourceBundle, "kotoshu/resource_bundle"
|
|
49
50
|
autoload :ResourceManager, "kotoshu/resource_manager"
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: kotoshu
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.0.
|
|
4
|
+
version: 1.0.3
|
|
5
5
|
platform: aarch64-linux
|
|
6
6
|
authors:
|
|
7
7
|
- Ribose Inc.
|
|
@@ -123,6 +123,7 @@ files:
|
|
|
123
123
|
- TODO.impl/131-hybrid-productization.md
|
|
124
124
|
- TODO.impl/132-techniques-page-and-prose.md
|
|
125
125
|
- TODO.impl/133-rust-first-distribution.md
|
|
126
|
+
- TODO.impl/134-post-131-improvements.md
|
|
126
127
|
- TODO.impl/20-encapsulation-cleanup.md
|
|
127
128
|
- TODO.impl/21-autoload-migration.md
|
|
128
129
|
- TODO.impl/22-url-pin-single-source-of-truth.md
|
|
@@ -501,12 +502,16 @@ files:
|
|
|
501
502
|
- lib/kotoshu/suggestions/strategies/phonetic_strategy.rb
|
|
502
503
|
- lib/kotoshu/suggestions/strategies/semantic_strategy.rb
|
|
503
504
|
- lib/kotoshu/suggestions/strategies/symspell_strategy.rb
|
|
505
|
+
- lib/kotoshu/suggestions/strategies/typo_retrieval_strategy.rb
|
|
504
506
|
- lib/kotoshu/suggestions/strategies/word_index.rb
|
|
505
507
|
- lib/kotoshu/suggestions/suggestion.rb
|
|
506
508
|
- lib/kotoshu/suggestions/suggestion_set.rb
|
|
507
509
|
- lib/kotoshu/suggestions/sweep_index.rb
|
|
510
|
+
- lib/kotoshu/suggestions/typo_merge.rb
|
|
508
511
|
- lib/kotoshu/tasks.rb
|
|
509
512
|
- lib/kotoshu/tasks/check_task.rb
|
|
513
|
+
- lib/kotoshu/typo.rb
|
|
514
|
+
- lib/kotoshu/typo/engine.rb
|
|
510
515
|
- lib/kotoshu/validators.rb
|
|
511
516
|
- lib/kotoshu/validators/spelling_validator.rb
|
|
512
517
|
- lib/kotoshu/version.rb
|