kotoshu 1.0.2 → 1.0.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 15c72e0b9eecd7ad94734eb9f8cab33e693ceff9642898fb7227f68121759425
4
- data.tar.gz: 7da7d2b1d3b8bc5bdfa5651455c47ef602d0bbac53516795a51888b573adf5e8
3
+ metadata.gz: 3722ff8a55645f8a8a6f08bc7c3a34427dcd1167f94c4c07831065db01684e5c
4
+ data.tar.gz: 659ac873d5f35c26a3f9a8775860c5c6f31dfcade3015703d1d4eadc824f712a
5
5
  SHA512:
6
- metadata.gz: 9c14c361ee5c5cb94e3db292cd801e3af0993797643de17bf5a40b2fe0e08f2d27e8060d92290103b1db038766cddf3ee0ff8a5b9c0afc621fb4f3373f50e81e
7
- data.tar.gz: 43186beae6c49a91cde7bdd66674906eb2936f75cf25c6df118c2682a26aa578a453031a903b02b08f6c8dc83b295b889da56308ffcf89ce39c5873be4dce541
6
+ metadata.gz: 1821c5433b271031b199d70b8b748affb4332f12dba6b27c23d412abf8958cfbf6d71b9a74950ca09c52a3f8ef299eca2238aee3d86ae37e1ca6f595c3306d2e
7
+ data.tar.gz: b0fc00ab0fef49a8c1d3d46b81daa14c9ea896988f006d2c583a512f50bc2a5a81ce3d99cdcfd82bf060f6deabaf981e814140a994c9273b21eddb6c20aff54e
@@ -4,12 +4,21 @@ Status: standing inventory (updated 2026-09-10)
4
4
 
5
5
  - VS Code Marketplace + Open VSX PATs → publish kotoshu-vscode
6
6
  (extension is publish-ready; owner: "later").
7
- - Worktree cleanup — ASK OWNER before removing anything:
8
- gem `.claude/worktrees/*` (10+ campaign worktrees), kotoshu-rs
9
- `.claude/worktrees/packs` (content already merged), and /tmp
10
- worktrees from the 1.0 day: fix-baseline, gem-dogfood, plan-116,
11
- impl-116, impl-117, conf-0110, conf-0100, conf-092, conf-4597,
12
- wasm-10, plans-118.
7
+ - ~~Worktree cleanup~~ — DONE 2026-09-12 under the owner's "make the
8
+ needed cleanups now": every repo down to its single main worktree,
9
+ merged branches deleted local+remote (see the memory entry).
10
+ - PyPI trusted publisher for kotoshu-native — owner registration
11
+ (project kotoshu-native, owner kotoshu, repo kotoshu-rs, workflow
12
+ release-pypi.yml, env blank), then rerun run 34681280341: the full
13
+ 16-wheel matrix + sdist are BUILT and preserved as artifacts; the
14
+ publish step failed only on the missing registration.
15
+ - Typo layer registry cut — owner's three mechanical steps, runbooked
16
+ in models-fasttext-onnx docs/RELEASING.md; the moment the pair
17
+ carries vocab_url, `kotoshu setup LANG --typo` arms the layer.
18
+ - Gem 1.0.3 cut decision — main is ahead of 1.0.2 (opt-in typo layer,
19
+ --typo setup, Windows strict-wrap fix, eager GVL-released build);
20
+ nothing changes for default installs, so the cut timing and number
21
+ are the owner's call.
13
22
  - kotoshu-lsp: still 0.1.1 — joins a future train if the owner wants
14
23
  LSP on the 1.x line (audit train deliberately excluded it).
15
24
  - Docker/kotoshu-ci + kotoshu-go/kotoshu-python floors: verify their
@@ -1,6 +1,21 @@
1
1
  # Plan 131 — Productize the hybrid: C-retrieve + fastText-rescore in the engines
2
2
 
3
- Status: proposed (the decision evidence is complete; plan 128 verdict SHIP)
3
+ Status: executed on the engine and gem sides (2026-09-12; kotoshu-rs
4
+ PR #38 merged — the pure-Rust bi-encoder at parity cosine 1.00000 vs
5
+ onnxruntime, the int8 retrieval scan at 1.81 ms per suggest, the
6
+ frozen C-benchmark numbers reproduced EXACTLY through the shipped
7
+ engine path: top1 0.118772, top5 0.269829 on en real n=2509; wasm
8
+ loadTypo/typoSuggest + KPK1 typo sections + fp32 full-tier RowStore +
9
+ ruby bindings with a 28-assertion smoke; gem PR #191 — Typo::Engine
10
+ loader, TypoRetrievalStrategy, TypoMerge, KOTOSHU_TYPO_RETRIEVAL
11
+ opt-in with a byte-identical disabled passthrough, 27 specs, full
12
+ suite 4030/0). Gates: all three pre-declared clauses pass (numbers
13
+ within noise = exact on the decision metric; latency 1.8 ms against
14
+ the 10 ms budget; zero default change by construction and by spec).
15
+ REMAINING (owner-coupled): the registry promotion — the typo entry's
16
+ primary URL and vocab_url stay null until the owner cuts a registry
17
+ release carrying them; the server/playground/worker flips follow that
18
+ cut on the semantic-tier pattern
4
19
  Depends on: plan 128 (verdict + artifacts), plan 113 (packs — the
5
20
  delivery vehicle), registry v1.6.0 (typo-biencoder assets already opt-in)
6
21
 
@@ -1,6 +1,28 @@
1
1
  # Plan 133 — Rust-first distribution: precompiled native gems and complete wheels
2
2
 
3
- Status: proposed (execution-ready)
3
+ Status: executed (2026-09-12) — PR #189 (7-iteration CI chain: the
4
+ MinGW Rust toolchain for the ABI match, bindgen pointed at the
5
+ RubyInstaller devkit's ucrt64 headers through BINDGEN_EXTRA_CLANG_ARGS
6
+ with the windows-gnu target, macos-13 -> macos-15-intel for the retired
7
+ x86_64 macOS runner, and the release race fix so the native legs build
8
+ from the v<version> tag). Released as kotoshu 1.0.2: six artifacts on
9
+ RubyGems (ruby + x86_64-linux, aarch64-linux, arm64-darwin, x86_64-darwin,
10
+ x64-mingw-ucrt), verified by a toolchain-free scratch-GEM_HOME install
11
+ that resolved kotoshu-1.0.2-arm64-darwin, engaged the native engine
12
+ under the auto default, and passed the frozen vectors. Docker: both
13
+ images (docker-kotoshu-ci PR #2, kotoshu-server PR #9) dropped their
14
+ build-tool layers / gained the native-backend build assertion, with the
15
+ guard proving the platform gem resolves inside the slim builder. Site:
16
+ engines section resolution paragraph + the 1.0.2 news entry (092fc03).
17
+ Design deviation: no x86_64-linux-musl platform gem — there is no
18
+ hosted musl Ruby runner and cross needs a musl Ruby; Alpine resolves
19
+ the pure-Ruby gem (source-build of the extension remains possible
20
+ there). Python: the 16-wheel + sdist kotoshu-native 0.1.1 matrix is
21
+ built and smoke-green (run 34681280341) but the PyPI publish is gated
22
+ on the owner registering the trusted publisher (pypi.org, project
23
+ kotoshu-native, repo kotoshu-rs, workflow release-pypi.yml, env blank);
24
+ rerun `gh run rerun 34681280341 --repo kotoshu/kotoshu-rs --failed`
25
+ afterward — the wheel artifacts persist, no rebuild needed.
4
26
  Depends on: the 1.0 conformance contract (identical outputs make the
5
27
  backend default flip correctness-free); plan 93 (the python wheel matrix)
6
28
 
@@ -0,0 +1,67 @@
1
+ # Plan 134 — Post-131 improvements: the audit's honest gaps
2
+
3
+ Status: executed (2026-09-13). (1) GVL release rs PR #39 — the
4
+ without_gvl helper runs the typo compute through
5
+ rb_thread_call_without_gvl, PROVEN LIVE: a 5 ms ticker thread logged
6
+ 4,118 ticks during the 24.8 s / 100k-word index build (a held GVL
7
+ starves it to ~0), suggest answers 2.2 ms after the eager build; gem
8
+ PR #195 makes Engine.for build eagerly. (2) native-suite CI leg (gem
9
+ PR #195): the conformance workflow compiles the extension and runs
10
+ the full suite — NativeBackend can no longer regress matrix-green.
11
+ (3) release verify (gem PR #195): the release asserts all six
12
+ RubyGems artifacts before reporting success. (4) runbook models PR
13
+ #37: docs/RELEASING.md carries the standard cut and the typo
14
+ promotion steps. (5) perf-weekly (gem PR #195): the weekly slow-suite
15
+ run whose first execution caught the suites rotted — repaired with
16
+ dated 2x-headroom bounds. Found and fixed en route (gem PR #196,
17
+ merged): setup_typo_remote rescued only Kotoshu::Error, leaking raw
18
+ Errno::ECONNREFUSED on Windows (caught by merge-preview CI on main).
19
+
20
+ Depends on: plan 131 (the typo layer), plan 133 (the platform gems)
21
+
22
+ ## The items
23
+
24
+ ### 1. The typo layer's first-suggest stall (production defect class)
25
+
26
+ The 100k-word index build (~15 s) happens lazily inside the first
27
+ `typo_suggest`, and the Ruby binding holds the GVL for the whole Rust
28
+ call — on the server, one arming request blocks every other request
29
+ for the build's duration. Fixes, both:
30
+
31
+ - The binding releases the GVL around the pure-Rust compute (index
32
+ build and scan touch no Ruby objects; results re-enter GVL scope to
33
+ build hashes), so a build never stops the world.
34
+ - The gem resolves EAGERLY: `Engine.for` builds the index at load —
35
+ the cost moves to opt-in setup where it belongs, never mid-check.
36
+
37
+ ### 2. The native path is invisible to the main rake matrix
38
+
39
+ The nine rake legs report the native specs PENDING ("native extension
40
+ not built"); only the separate compare job compiles the ext. A
41
+ `NativeBackend` regression could land matrix-green. One dedicated
42
+ matrix job compiles the extension and runs the suite un-pended.
43
+
44
+ ### 3. The release workflow trusts its own pushes
45
+
46
+ Nothing asserts the release is COMPLETE — a silently failed platform
47
+ push ships a partial release green. A `verify` job after the native
48
+ legs queries the RubyGems API for version x platform coverage (ruby +
49
+ the five native platforms) and fails on anything missing.
50
+
51
+ ### 4. The owner's cut-day procedure lives in session memory
52
+
53
+ The three-step typo release checklist is recorded in memory and PR
54
+ bodies. The models repo carries the committed runbook instead — the
55
+ owner cuts the release without needing this session.
56
+
57
+ ### 5. The weekly SLOW_TESTS dashboard, owner-free form
58
+
59
+ The T4.2 remainder as a workflow: weekly run of the slow suites, the
60
+ delta summary posted on the run page (a job summary + artifact — no
61
+ external dashboard to own).
62
+
63
+ ## Gates
64
+
65
+ - Every fix ships with its spec or CI proof; no behavior change to
66
+ any frozen output (the GVL release is invisible; eager build moves
67
+ WHEN the same index is built, not what it computes).
@@ -7,8 +7,11 @@ This is the largest tier 3 item: CJK languages need word-segmentation
7
7
  (no spaces in zh/ja), different dictionary formats, and
8
8
  romanization-aware suggestion strategies.
9
9
 
10
- Status: outline only. This TODO documents the design space; real work
11
- needs upstream library choices and per-language fixture data.
10
+ Status: outline only, partially superseded. Japanese tokenization
11
+ shipped (the ja module with the suika tokenizer, live since the
12
+ language-modules wave); real zh/ko spell checking remains design
13
+ space — this TODO documents it, and real work still needs upstream
14
+ segmentation library choices and per-language fixture data.
12
15
 
13
16
  ## Per-language needs
14
17
 
@@ -1,5 +1,34 @@
1
1
  # 56 — T4+ Quality, Architecture, and Ecosystem backlog
2
2
 
3
+ Status: largely superseded — most items below were promoted and
4
+ executed by the numbered plans; the remaining open ones are marked.
5
+ Snapshot annotations (2026-09-13): T4.1 shipped as the plan-147/148
6
+ perf waves (WordIndex + SweepIndex: suggest 20-140 s -> sub-second,
7
+ sweep 1472 -> 661 ms en; the Rust engine then took the sweep 10-15x
8
+ further, and plan 131's hybrid retrieval added 1.8 ms retrieval) with
9
+ published error budgets on /performance. T4.4 shipped as plan 97 (YARD
10
+ 100% public surface, docs CI, hosted reference) plus the repeated
11
+ README truth passes. T5.2 and T5.3 shipped (Configuration as data
12
+ with the Resolver; WordResult/DocumentResult/SuggestionSet/Suggestion
13
+ on lutaml-model, hand-rolled to_h banned by the global rule). T6.1
14
+ shipped (kotoshu-lsp, live on RubyGems) and T6.2 shipped (the wasm
15
+ playground, @kotoshu/wasm, language packs). Still open: T4.2's weekly
16
+ SLOW_TESTS dashboard, T5.4 (= plan 53), and T6.3 (desktop dictionary
17
+ editor — future). CLOSED SINCE: T4.3's named properties shipped
18
+ (spec/properties/data_structure_properties_spec.rb — Trie
19
+ order-independence and totality, prefix agreement, set recovery, and
20
+ the every-suggestion-is-a-dictionary-word invariant; the affix
21
+ round-trip property remains the one open sub-item, gated on a
22
+ morphological API surface). T5.1 audited clean (2026-09-13): the
23
+ dedupe-once merge CompositeStrategy already performs is exactly the
24
+ fix T5.1 proposed (the code cites it); strategy purity holds with one
25
+ documented exception — the semantic-cascade threshold falls back to
26
+ the process Configuration when the composite was not given an
27
+ explicit one, by design and overridable; and the symspell/edit-1
28
+ overlap is moot — neither is wired into any default flow (the frozen
29
+ Generator owns the real path; the strategy classes are the composable
30
+ API, and the single-set merge makes overlapping compositions safe).
31
+
3
32
  This file rolls up the T4 (Quality), T5 (Architecture), and T6
4
33
  (Ecosystem) items from TODO.impl/39-tier3-and-beyond.md into
5
34
  promotable units with concrete acceptance criteria.
@@ -88,6 +88,16 @@ module Kotoshu
88
88
  # Directory (under the cache root) holding the LID artifact pair.
89
89
  LID_DIRECTORY = "lid"
90
90
 
91
+ # The hybrid typo bi-encoder (plan 131): a language-less
92
+ # registry resource (kotoshu://models/typo/typo-biencoder) with
93
+ # a REQUIRED vocab sidecar, resolved exactly like the LID pair.
94
+ TYPO_LANGUAGE = "typo"
95
+ TYPO_TIER = "typo-biencoder"
96
+ TYPO_REGISTRY_ID = ModelRegistry::Resource.id_for(TYPO_LANGUAGE, TYPO_TIER)
97
+
98
+ # Cache directory (under the models root, beside the languages).
99
+ TYPO_DIRECTORY = File.join("models", "typo")
100
+
91
101
  # Normalize and validate a tier name.
92
102
  #
93
103
  # @param tier [String, Symbol] "full", "fluency", or "mini"
@@ -224,9 +234,17 @@ module Kotoshu
224
234
  def download_tiered_model(language_code, tier:, force_download: false)
225
235
  tier = self.class.normalize_tier(tier)
226
236
  lang = language_code.to_s
227
- return get("#{lang}:onnx", force_download: force_download) if tier == :full
228
237
 
238
+ # The full tier rides the registry when the cached registry
239
+ # knows it (primary URL, vocab sibling, sha256 — the path the
240
+ # typo layer's rescore requires, since the legacy download
241
+ # never fetched the vocab). Offline without a cached registry
242
+ # the entry is nil and the legacy CDN path serves, preserving
243
+ # the pre-registry behavior exactly.
229
244
  entry = registry_entry_for(lang, tier, force: force_download)
245
+ if tier == :full && entry.nil?
246
+ return get("#{lang}:onnx", force_download: force_download)
247
+ end
230
248
  unless entry
231
249
  raise Kotoshu::Error,
232
250
  "no registry entry for #{ModelRegistry::Resource.id_for(lang, tier)}"
@@ -257,6 +275,7 @@ module Kotoshu
257
275
  cached_at: Time.now.utc.iso8601,
258
276
  source: "registry"
259
277
  }
278
+ metadata[:vocab_file] = File.basename(vocab_path) if vocab_path
260
279
  write_metadata(File.join(dest_path, "metadata.json"), metadata)
261
280
 
262
281
  result = { model_path: model_file, metadata: metadata }
@@ -293,7 +312,7 @@ module Kotoshu
293
312
  return nil unless File.exist?(onnx_file) && File.size(onnx_file).positive?
294
313
  return nil unless File.exist?(vocab_file) && File.size(vocab_file).positive?
295
314
 
296
- verify_lid_integrity!(metadata, onnx_file, vocab_file)
315
+ verify_model_pair_integrity!(metadata, onnx_file, vocab_file)
297
316
 
298
317
  { onnx_path: onnx_file, vocab_path: vocab_file, metadata: metadata }
299
318
  end
@@ -354,6 +373,133 @@ module Kotoshu
354
373
  { onnx_path: onnx_file, vocab_path: vocab_file, metadata: metadata }
355
374
  end
356
375
 
376
+ # ---- typo bi-encoder pair (plan 131) ----
377
+
378
+ # Whether the typo bi-encoder pair is cached and complete
379
+ # (both files, checksum-verified).
380
+ #
381
+ # @return [Boolean]
382
+ def typo_biencoder_cached?
383
+ !load_cached_typo_biencoder.nil?
384
+ end
385
+
386
+ # Resolve the typo bi-encoder pair from the cache only — never
387
+ # touches the network (the suggest path resolves; setup
388
+ # downloads, the same two-stage split as the LID pair).
389
+ #
390
+ # @return [Hash, nil] { onnx_path:, vocab_path:, metadata } when
391
+ # cached, nil when not set up
392
+ # @raise [Kotoshu::IntegrityError] cached bytes fail their
393
+ # recorded checksum
394
+ def load_cached_typo_biencoder
395
+ dir = File.join(@cache_path, TYPO_DIRECTORY)
396
+ metadata_path = File.join(dir, "metadata.json")
397
+ return nil unless File.exist?(metadata_path)
398
+
399
+ metadata = read_metadata(metadata_path)
400
+ return nil unless metadata
401
+
402
+ onnx_file = File.join(dir, metadata["file"].to_s)
403
+ vocab_file = File.join(dir, metadata["vocab_file"].to_s)
404
+ return nil unless File.exist?(onnx_file) && File.size(onnx_file).positive?
405
+ return nil unless File.exist?(vocab_file) && File.size(vocab_file).positive?
406
+
407
+ verify_model_pair_integrity!(
408
+ metadata, onnx_file, vocab_file,
409
+ directory: dir,
410
+ registry_id: TYPO_REGISTRY_ID,
411
+ remediation: "Run Kotoshu.setup_typo(language, force: true) to re-download."
412
+ )
413
+
414
+ { onnx_path: onnx_file, vocab_path: vocab_file, metadata: metadata }
415
+ end
416
+
417
+ # Download the typo bi-encoder pair from the models registry,
418
+ # verifying the ONNX bytes against the registry sha256 (the
419
+ # setup half of the two-stage model). The vocab sidecar is
420
+ # required, not optional — the encoder cannot tokenize without
421
+ # its char vocabulary — so a registry entry without vocab_url
422
+ # (the pre-cut state) raises the honest error instead of
423
+ # caching a half-usable artifact.
424
+ #
425
+ # @param force [Boolean] re-fetch the registry first
426
+ # @return [Hash] { onnx_path:, vocab_path:, metadata }
427
+ # @raise [Kotoshu::Error] no entry, entry without vocab_url, or
428
+ # offline without a cached registry
429
+ # @raise [Kotoshu::IntegrityError] sha256 mismatch
430
+ def download_typo_biencoder(force: false)
431
+ entry = registry(force: force)&.find(TYPO_LANGUAGE, TYPO_TIER)
432
+ unless entry
433
+ raise Kotoshu::Error, "no registry entry for #{TYPO_REGISTRY_ID}"
434
+ end
435
+ if entry.vocab_url.nil? || entry.vocab_url.to_s.strip.empty?
436
+ raise Kotoshu::Error,
437
+ "#{TYPO_REGISTRY_ID} carries no vocab_url yet — the typo pair ships with a models registry release"
438
+ end
439
+
440
+ dir = File.join(@cache_path, TYPO_DIRECTORY)
441
+ FileUtils.mkdir_p(dir)
442
+
443
+ onnx_file = File.join(dir, filename_from_url(entry.urls.primary))
444
+ used_url = download_primary_or_mirror!(entry, onnx_file)
445
+ verify_registry_sha256!(entry, onnx_file, TYPO_REGISTRY_ID, used_url,
446
+ remediation: "Run Kotoshu::Typo.setup(force: true) to re-download.")
447
+
448
+ vocab_file = File.join(dir, filename_from_url(entry.vocab_url))
449
+ begin
450
+ download_file(entry.vocab_url, vocab_file)
451
+ rescue StandardError => e
452
+ File.delete(onnx_file) if File.exist?(onnx_file)
453
+ raise Kotoshu::Error, "typo vocab unavailable at #{entry.vocab_url}: #{e.message}"
454
+ end
455
+
456
+ metadata = {
457
+ version: entry.version,
458
+ url: used_url,
459
+ language: TYPO_LANGUAGE,
460
+ type: "typo-biencoder",
461
+ tier: TYPO_TIER,
462
+ file: File.basename(onnx_file),
463
+ vocab_file: File.basename(vocab_file),
464
+ checksum: entry.sha256,
465
+ registry_id: TYPO_REGISTRY_ID,
466
+ size_bytes: entry.size_bytes,
467
+ cached_at: Time.now.utc.iso8601,
468
+ source: "registry"
469
+ }
470
+ write_metadata(File.join(dir, "metadata.json"), metadata)
471
+
472
+ { onnx_path: onnx_file, vocab_path: vocab_file, metadata: metadata }
473
+ end
474
+
475
+ # Resolve a tiered model from the cache only — the cache half of
476
+ # the two-stage model for tiers, symmetric with
477
+ # {#load_cached_lid} / {#load_cached_typo_biencoder}. Returns
478
+ # the vocab sibling when the registry-served download landed it.
479
+ #
480
+ # @param language_code [String]
481
+ # @param tier [String, Symbol]
482
+ # @return [Hash, nil] { model_path:, vocab_path?, metadata } or nil
483
+ def load_cached_tier(language_code, tier)
484
+ tier = self.class.normalize_tier(tier)
485
+ resource_id = tier_resource_id(language_code.to_s, tier)
486
+ dir = resource_dir_for(resource_id)
487
+ metadata_path = File.join(dir, "metadata.json")
488
+ return nil unless File.exist?(metadata_path)
489
+
490
+ metadata = read_metadata(metadata_path)
491
+ return nil unless metadata
492
+
493
+ model_file = File.join(dir, metadata["file"].to_s)
494
+ return nil unless File.exist?(model_file) && File.size(model_file).positive?
495
+
496
+ result = { model_path: model_file, metadata: metadata }
497
+ vocab = metadata["vocab_file"].to_s
498
+ vocab_path = File.join(dir, vocab)
499
+ result[:vocab_path] = vocab_path if !vocab.empty? && File.exist?(vocab_path)
500
+ result
501
+ end
502
+
357
503
  # Get available model types for a language.
358
504
  #
359
505
  # @param language_code [String] ISO 639-1 language code
@@ -801,16 +947,19 @@ module Kotoshu
801
947
  # @return [void]
802
948
  # @raise [Kotoshu::IntegrityError, Kotoshu::Error] on mismatch;
803
949
  # corrupt bytes are removed so the next setup re-downloads
804
- def verify_lid_integrity!(metadata, onnx_file, vocab_file)
805
- dir = File.join(@cache_path, LID_DIRECTORY)
950
+ def verify_model_pair_integrity!(metadata, onnx_file, vocab_file, directory: nil,
951
+ registry_id: nil, remediation: nil)
952
+ registry_id ||= LID_REGISTRY_ID
953
+ directory ||= File.join(@cache_path, LID_DIRECTORY)
954
+ remediation ||= "Run Kotoshu.setup_lid(force: true) to re-download."
806
955
  if lfs_pointer?(onnx_file) || lfs_pointer?(vocab_file)
807
956
  File.delete(onnx_file)
808
957
  File.delete(vocab_file)
809
- FileUtils.rm_f(File.join(dir, "metadata.json"))
958
+ FileUtils.rm_f(File.join(directory, "metadata.json"))
810
959
  raise Kotoshu::Error,
811
- "Integrity verification failed for #{LID_REGISTRY_ID}: cached LID " \
812
- "artifact is a Git LFS pointer stub, not model content. Run " \
813
- "Kotoshu.setup_lid(force: true) to re-download."
960
+ "Integrity verification failed for #{registry_id}: cached " \
961
+ "artifact is a Git LFS pointer stub, not model content. " \
962
+ "#{remediation}"
814
963
  end
815
964
 
816
965
  expected = metadata["checksum"]
@@ -820,11 +969,11 @@ module Kotoshu
820
969
  return if actual == expected
821
970
 
822
971
  raise Kotoshu::IntegrityError.new(
823
- LID_REGISTRY_ID,
972
+ registry_id,
824
973
  expected: expected,
825
974
  actual: actual,
826
975
  url: metadata["url"],
827
- remediation: "Run Kotoshu.setup_lid(force: true) to re-download."
976
+ remediation: remediation
828
977
  )
829
978
  end
830
979
 
data/lib/kotoshu/cli.rb CHANGED
@@ -146,6 +146,7 @@ module Kotoshu
146
146
  kotoshu setup en de fr # download from GitHub
147
147
  kotoshu setup en --want spelling,frequency # also fetch Kelly list
148
148
  kotoshu setup en --model --tier mini # model at the mini tier
149
+ kotoshu setup en --typo # typo bi-encoder + full tier
149
150
  kotoshu setup en --aff /p/en.aff --dic /p/en.dic
150
151
  kotoshu setup en --from /usr/share/hunspell/
151
152
  kotoshu setup --force en # re-download
@@ -167,6 +168,10 @@ module Kotoshu
167
168
  type: :boolean,
168
169
  default: false,
169
170
  desc: "Fetch the ONNX model (shorthand for adding model to --want)"
171
+ method_option :typo,
172
+ type: :boolean,
173
+ default: false,
174
+ desc: "Fetch the typo-retrieval pair (bi-encoder + full tier; needs a registry release carrying it)"
170
175
  method_option :tier,
171
176
  type: :string,
172
177
  desc: "Model tier: full, fluency, or mini (default: full)"
@@ -192,6 +197,7 @@ module Kotoshu
192
197
 
193
198
  want = (options[:want] || "spelling").split(",").map(&:strip).map(&:to_sym)
194
199
  want << :model if options[:model] && !want.include?(:model)
200
+ want << :typo if options[:typo] && !want.include?(:typo)
195
201
  opts = setup_source_options(languages)
196
202
  opts[:want] = want
197
203
  opts[:tier] = options[:tier].to_sym if options[:tier]
@@ -234,6 +240,10 @@ module Kotoshu
234
240
  type: :boolean,
235
241
  default: false,
236
242
  desc: "Fetch the ONNX model (shorthand for adding model to --want)"
243
+ method_option :typo,
244
+ type: :boolean,
245
+ default: false,
246
+ desc: "Fetch the typo-retrieval pair (bi-encoder + full tier; needs a registry release carrying it)"
237
247
  method_option :tier,
238
248
  type: :string,
239
249
  desc: "Model tier: full, fluency, or mini (default: full)"
@@ -525,8 +535,10 @@ module Kotoshu
525
535
  frequency = result.frequency || "skipped"
526
536
  model = result.model || "skipped"
527
537
  model += " (#{result.model_tier} tier)" if result.model && result.model_tier
538
+ typo = result.typo || "skipped"
528
539
  source = result.source
529
- puts "OK (spelling: #{spelling}, frequency: #{frequency}, model: #{model}, source: #{source})"
540
+ puts "OK (spelling: #{spelling}, frequency: #{frequency}, model: #{model}, " \
541
+ "typo: #{typo}, source: #{source})"
530
542
  end
531
543
 
532
544
  def list_setup
@@ -206,10 +206,16 @@ module Kotoshu
206
206
  },
207
207
  backend: {
208
208
  env: "KOTOSHU_BACKEND",
209
- default: "ruby",
210
- description: "Engine backend for correct?/suggest (auto by default: native when the extension is available, ruby otherwise; or force either)",
209
+ default: "auto",
210
+ description: "Engine backend for correct?/suggest (auto: native when the extension is available, ruby otherwise; or force either)",
211
211
  type: String
212
212
  },
213
+ typo_retrieval: {
214
+ env: "KOTOSHU_TYPO_RETRIEVAL",
215
+ default: false,
216
+ description: "Opt-in hybrid typo retrieval (plan 131): the native bi-encoder slate merges ahead of frequency ranking; needs the native backend",
217
+ type: :boolean
218
+ },
213
219
  personal_dictionary: {
214
220
  env: "KOTOSHU_PERSONAL_DICTIONARY",
215
221
  default: true,
@@ -241,6 +247,7 @@ module Kotoshu
241
247
  # conformance contract), so the default is the faster engine
242
248
  # with the pure fallback, never a failure.
243
249
  backend: "auto",
250
+ typo_retrieval: false,
244
251
  personal_dictionary: true
245
252
  }.freeze
246
253
 
@@ -301,6 +308,12 @@ module Kotoshu
301
308
  # NativeBackend for the boundary — the semantic path stays Ruby.
302
309
  attr_accessor :backend
303
310
 
311
+ # @return [Boolean] Opt-in hybrid typo retrieval (plan 131): the
312
+ # native bi-encoder's rescored slate merges ahead of frequency
313
+ # ranking when the native backend serves the language. Env:
314
+ # KOTOSHU_TYPO_RETRIEVAL.
315
+ attr_accessor :typo_retrieval
316
+
304
317
  # @return [Boolean] Whether Spellchecker#check accepts words from
305
318
  # the user personal dictionary (~/.config/kotoshu/personal.dic,
306
319
  # KOTOSHU_PERSONAL_DIC override) instead of flagging them as
@@ -511,6 +524,7 @@ module Kotoshu
511
524
  max_cache_size: @max_cache_size,
512
525
  semantic_cascade_threshold: @semantic_cascade_threshold,
513
526
  backend: @backend,
527
+ typo_retrieval: @typo_retrieval,
514
528
  personal_dictionary: @personal_dictionary
515
529
  }
516
530
  end
@@ -629,6 +643,7 @@ module Kotoshu
629
643
  @resource_pin = DEFAULTS[:resource_pin]
630
644
  @semantic_cascade_threshold = DEFAULTS[:semantic_cascade_threshold]
631
645
  @backend = DEFAULTS[:backend]
646
+ @typo_retrieval = DEFAULTS[:typo_retrieval]
632
647
  @personal_dictionary = DEFAULTS[:personal_dictionary]
633
648
  end
634
649
 
@@ -41,6 +41,7 @@ module Kotoshu
41
41
  :frequency, # :downloaded | :local | :cached | :unavailable | nil
42
42
  :model, # :downloaded | :cached | :unavailable | nil
43
43
  :model_tier, # :full | :fluency | :mini | nil (tier actually set up)
44
+ :typo, # :downloaded | :cached | :unavailable | nil (plan 131 pair)
44
45
  :source, # :kotoshu | :local
45
46
  keyword_init: true
46
47
  ) do
@@ -280,6 +281,11 @@ module Kotoshu
280
281
  when :frequency
281
282
  fc = frequency_cache_for
282
283
  fc.supports_resource?(lang) && fc.available?(lang)
284
+ when :typo
285
+ # Both halves: the language-less pair and the language's full
286
+ # tier (the rescore's exactness contract).
287
+ cache = model_cache_for
288
+ cache.typo_biencoder_cached? && cache.available?(cache.tier_resource_id(lang, :full))
283
289
  when :model
284
290
  cache = model_cache_for
285
291
  if tier.nil?
@@ -343,16 +349,57 @@ module Kotoshu
343
349
  end
344
350
  end
345
351
 
352
+ typo_status = if want.include?(:typo)
353
+ setup_typo_remote(lang, force: force, strict: strict, config: config,
354
+ tier: model_tier)
355
+ end
356
+
346
357
  SetupResult.new(
347
358
  language: lang,
348
359
  spelling: spelling_status,
349
360
  frequency: frequency_status,
350
361
  model: model_status,
351
362
  model_tier: model_tier,
363
+ typo: typo_status,
352
364
  source: :kotoshu
353
365
  )
354
366
  end
355
367
 
368
+ # The typo layer's setup half (plan 131): the language-less
369
+ # bi-encoder pair, plus the language's FULL tier when this run did
370
+ # not already fetch it (the hybrid's rescore is full-tier-exact by
371
+ # contract — a mini tier would rank wrong, so full is fetched
372
+ # explicitly, not defaulted).
373
+ #
374
+ # Returns :cached when both halves were already present,
375
+ # :downloaded when anything was fetched, :unavailable when the
376
+ # registry cannot serve the pair yet (strict: raises instead —
377
+ # the pre-cut state).
378
+ def setup_typo_remote(lang, force:, strict:, config:, tier: nil)
379
+ cache = model_cache_for(config: config)
380
+ unless force
381
+ full_id = cache.tier_resource_id(lang, :full)
382
+ return :cached if cache.typo_biencoder_cached? && cache.available?(full_id)
383
+ end
384
+ begin
385
+ cache.download_typo_biencoder(force: force)
386
+ cache.download_tiered_model(lang, tier: :full, force_download: force) unless tier == :full
387
+ rescue Kotoshu::Error
388
+ raise if strict
389
+
390
+ return :unavailable
391
+ rescue StandardError => e
392
+ # Windows surfaces closed-port fetches as raw Errno classes the
393
+ # cache does not always wrap; the two-stage contract is
394
+ # setup-degrades unless strict, and strict reports through the
395
+ # typed error like every setup path.
396
+ raise Kotoshu::Error, "typo setup failed: #{e.class}: #{e.message}" if strict
397
+
398
+ return :unavailable
399
+ end
400
+ :downloaded
401
+ end
402
+
356
403
  def setup_frequency_remote(lang, force:, strict:, config:)
357
404
  cache = frequency_cache_for(config: config)
358
405
  return :unavailable unless cache.supports_resource?(lang)
@@ -140,9 +140,39 @@ module Kotoshu
140
140
  def suggest(word, max_suggestions: nil)
141
141
  return Suggestions::SuggestionSet.empty if word.nil? || word.empty?
142
142
 
143
- return @native_backend.suggest(word, max_suggestions: max_suggestions) if @native_backend
143
+ base = if @native_backend
144
+ @native_backend.suggest(word, max_suggestions: max_suggestions)
145
+ else
146
+ @generator.generate(word, max_suggestions: max_suggestions)
147
+ end
148
+ merge_typo_retrieval(base, word, max_suggestions: max_suggestions)
149
+ end
150
+
151
+ # Merge the opt-in typo-retrieval slate (plan 131) AHEAD of the
152
+ # base suggestions: the bi-encoder surfaces corrections a
153
+ # frequency-ranked sweep misses, so its rows lead. Inert unless
154
+ # typo_retrieval is configured AND the native engine loaded for
155
+ # this language — a disabled layer is a passthrough, never an
156
+ # error, never a different order.
157
+ def merge_typo_retrieval(base, word, max_suggestions: nil)
158
+ return base unless @config.typo_retrieval
159
+
160
+ engine = typo_retrieval_engine
161
+ return base unless engine
162
+
163
+ typo = engine.suggest(word, max_suggestions: max_suggestions)
164
+ limit = max_suggestions || @config.max_suggestions
165
+ Suggestions::TypoMerge.call(base: base, typo: typo, limit: limit)
166
+ end
167
+
168
+ # The typo engine for this language, resolved once. A nil result
169
+ # is memoized too — an unavailable layer must not retry its
170
+ # artifact resolution on every suggestion.
171
+ def typo_retrieval_engine
172
+ return @typo_retrieval_engine if defined?(@typo_retrieval_engine)
144
173
 
145
- @generator.generate(word, max_suggestions: max_suggestions)
174
+ code = (@resource_bundle&.language || @config.language).to_s.split("-").first
175
+ @typo_retrieval_engine = Kotoshu::Typo::Engine.for(code, configuration: @config)
146
176
  end
147
177
 
148
178
  # Check a word and return a result object.
@@ -0,0 +1,55 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Kotoshu
4
+ module Suggestions
5
+ module Strategies
6
+ # The hybrid typo-retrieval strategy (plan 131): a frozen char
7
+ # bi-encoder retrieves dictionary candidates a frequency-ranked
8
+ # sweep misses, the fastText full tier rescores them, and the
9
+ # rows join the suggestion pipeline AHEAD of frequency ranking
10
+ # — the layer measured at +6.3 pp top-5 on the frozen
11
+ # benchmark. Opt-in and native-only; the engine is injected so
12
+ # the strategy itself stays pure orchestration.
13
+ class TypoRetrievalStrategy < BaseStrategy
14
+ # @return [#suggest] the typo engine ({Typo::Engine} in
15
+ # production; anything answering #suggest(word) in specs)
16
+ attr_reader :engine
17
+
18
+ # @param engine [#suggest] the typo engine
19
+ # @param config [Hash] additional configuration
20
+ def initialize(engine:, **config)
21
+ super(name: :typo_retrieval, **config)
22
+ @engine = engine
23
+ end
24
+
25
+ # Generate the typo-retrieval slate for the context's word.
26
+ #
27
+ # @param context [Context] the suggestion context
28
+ # @return [SuggestionSet]
29
+ def generate(context)
30
+ return SuggestionSet.empty unless enabled? && @engine
31
+
32
+ @engine.suggest(context.word, max_suggestions: max_results)
33
+ end
34
+
35
+ # This strategy is a candidate generator for misspellings; it
36
+ # has no real-word-error story, which the semantic strategy
37
+ # owns.
38
+ #
39
+ # @param context [Context]
40
+ # @return [Boolean]
41
+ def handles?(context)
42
+ enabled? && @engine && !context.word.empty?
43
+ end
44
+
45
+ # The strategy's rows already carry their own order (the
46
+ # rescored ranking); do not let a confident base set skip it.
47
+ #
48
+ # @return [Boolean]
49
+ def skip_when_confident?
50
+ false
51
+ end
52
+ end
53
+ end
54
+ end
55
+ end
@@ -0,0 +1,26 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Kotoshu
4
+ module Suggestions
5
+ # The typo-retrieval merge (plan 131): the bi-encoder's rescored
6
+ # slate joins AHEAD of the base suggestions — it surfaces
7
+ # corrections a frequency-ranked sweep misses — with duplicates
8
+ # (same word) dropped from the base side and the set capped at the
9
+ # caller's limit. A pure function over real SuggestionSets; the
10
+ # spellchecker calls it only when the opt-in layer is armed.
11
+ module TypoMerge
12
+ # @param base [SuggestionSet] the engine's own suggestions
13
+ # @param typo [SuggestionSet] the typo-retrieval slate
14
+ # @param limit [Integer] final cap
15
+ # @return [SuggestionSet] merged set, order adopted verbatim
16
+ def self.call(base:, typo:, limit:)
17
+ return base if typo.empty?
18
+
19
+ typo_words = typo.map(&:word)
20
+ merged = typo.suggestions +
21
+ base.suggestions.reject { |suggestion| typo_words.include?(suggestion.word) }
22
+ SuggestionSet.new(merged.first(limit), max_size: limit, ranked: true)
23
+ end
24
+ end
25
+ end
26
+ end
@@ -12,6 +12,7 @@ module Kotoshu
12
12
  autoload :Suggestion, "kotoshu/suggestions/suggestion"
13
13
  autoload :SuggestionSet, "kotoshu/suggestions/suggestion_set"
14
14
  autoload :SweepIndex, "kotoshu/suggestions/sweep_index"
15
+ autoload :TypoMerge, "kotoshu/suggestions/typo_merge"
15
16
 
16
17
  # Strategies sub-namespace.
17
18
  module Strategies
@@ -23,6 +24,7 @@ module Kotoshu
23
24
  autoload :PhoneticStrategy, "kotoshu/suggestions/strategies/phonetic_strategy"
24
25
  autoload :SemanticStrategy, "kotoshu/suggestions/strategies/semantic_strategy"
25
26
  autoload :SymspellStrategy, "kotoshu/suggestions/strategies/symspell_strategy"
27
+ autoload :TypoRetrievalStrategy, "kotoshu/suggestions/strategies/typo_retrieval_strategy"
26
28
  autoload :WordIndex, "kotoshu/suggestions/strategies/word_index"
27
29
  end
28
30
  end
@@ -0,0 +1,125 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Kotoshu
4
+ module Typo
5
+ # The hybrid typo-retrieval engine (plan 131): the frozen char
6
+ # bi-encoder retrieves a top-20 vocabulary slate for a
7
+ # misspelling and the fastText full tier rescores it — the layer
8
+ # the models repo measured at +6.3 pp top-5 over frequency
9
+ # ranking on the frozen benchmark.
10
+ #
11
+ # The engine itself lives in the native extension
12
+ # ({Kotoshu::Native::TypoEngine}); this class is the Ruby-side
13
+ # loader: it resolves the artifact pairs through the model cache
14
+ # (the registry resources `kotoshu://models/typo/typo-biencoder`
15
+ # and the language's full tier), constructs the native engine
16
+ # once, and answers {#suggest} as a SuggestionSet the pipeline
17
+ # merges ahead of frequency ranking.
18
+ #
19
+ # The layer is opt-in and native-only: {for} returns nil — never
20
+ # raises — when the extension is absent or the artifacts cannot
21
+ # resolve, the same silent-degrade contract the semantic tier and
22
+ # the LID detector follow.
23
+ class Engine
24
+ # @return [Kotoshu::Native::TypoEngine, nil] the loaded native engine
25
+ attr_reader :native_engine
26
+
27
+ # @return [String] the language the tier was loaded for
28
+ attr_reader :language
29
+
30
+ # Load the engine for a language. Yielded block receives the
31
+ # cache to use (defaults to the configured model cache) — specs
32
+ # inject a stub cache with local artifact paths.
33
+ #
34
+ # @param language [String] ISO 639-1 code
35
+ # @param configuration [Configuration]
36
+ # @param cache [Cache::ModelCache, nil] explicit cache
37
+ # @return [Engine, nil] nil when anything in the chain is absent
38
+ def self.for(language, configuration: Kotoshu.configuration, cache: nil)
39
+ return nil unless defined?(Kotoshu::Native::TypoModel)
40
+ return nil unless Kotoshu::Native.available?
41
+
42
+ cache ||= Cache::ModelCache.new(cache_path: configuration.cache_path)
43
+ typo = cache.load_cached_typo_biencoder
44
+ tier = cache.load_cached_tier(language, :full)
45
+ return nil unless typo && tier && tier[:vocab_path]
46
+
47
+ native = Kotoshu::Native::TypoModel.load(typo[:onnx_path], typo[:vocab_path])
48
+ native_tier = Kotoshu::Native::TypoTier.load(tier[:model_path], tier[:vocab_path])
49
+ engine = new(Kotoshu::Native::TypoEngine.new(native, native_tier),
50
+ native_tier: native_tier, language: language)
51
+ # EAGER index build (plan 134): the multi-second derivation
52
+ # runs HERE - with the GVL released by the binding - instead
53
+ # of stalling the first suggest (and every Ruby thread under a
54
+ # held GVL). The cost lands at opt-in setup where it belongs.
55
+ engine.build_index
56
+ engine
57
+ rescue StandardError
58
+ # The opt-in layer never breaks a check: any load failure
59
+ # (corrupt artifact, engine error) degrades to "absent".
60
+ nil
61
+ end
62
+
63
+ # Derive the per-language index now (the eager half; the
64
+ # binding releases the GVL and memoizes - building again is a
65
+ # no-op). Returns the indexed vocabulary size.
66
+ #
67
+ # @return [Integer]
68
+ def build_index
69
+ @native_engine.build_index(@native_tier)
70
+ end
71
+
72
+ # @param native_engine [Kotoshu::Native::TypoEngine]
73
+ # @param native_tier [Kotoshu::Native::TypoTier] the rescore
74
+ # tier (kept for the eager index build)
75
+ # @param language [String]
76
+ def initialize(native_engine, language:, native_tier: nil)
77
+ @native_engine = native_engine
78
+ @native_tier = native_tier
79
+ @language = language
80
+ end
81
+
82
+ # The rescored slate for a word as a SuggestionSet, source
83
+ # :typo_retrieval, cosine confidences clamped to [0, 1]. Empty
84
+ # for a word outside the tier vocabulary (the hybrid's honest
85
+ # in-vocab rule — there is nothing the layer can say).
86
+ #
87
+ # @param word [String]
88
+ # @param max_suggestions [Integer, nil]
89
+ # @return [Suggestions::SuggestionSet]
90
+ def suggest(word, max_suggestions: nil)
91
+ limit = max_suggestions || DEFAULT_SLATE
92
+ rows = @native_engine.typo_suggest(word)
93
+ suggestions = rows.first(limit).map do |row|
94
+ Suggestions::Suggestion.new(
95
+ word: row["word"],
96
+ distance: 0,
97
+ confidence: row["score"].clamp(0.0, 1.0),
98
+ source: :typo_retrieval
99
+ )
100
+ end
101
+ Suggestions::SuggestionSet.new(suggestions, max_size: limit, ranked: true)
102
+ end
103
+
104
+ DEFAULT_SLATE = 20
105
+ end
106
+ end
107
+
108
+ # The setup half of the two-stage model (plan 131): download the
109
+ # typo bi-encoder pair and the language's full tier with its vocab
110
+ # sibling, so {Engine.for} can resolve cache-only at suggest time.
111
+ # Raises the same errors {Cache::ModelCache#download_typo_biencoder}
112
+ # and {Cache::ModelCache#download_tiered_model} raise — setup
113
+ # callers report, they do not degrade.
114
+ #
115
+ # @param language [String] ISO 639-1 code
116
+ # @param configuration [Configuration]
117
+ # @param force [Boolean] re-fetch the registry first
118
+ # @return [Hash] { typo: {onnx_path, vocab_path}, tier: {model_path, vocab_path?} }
119
+ def self.setup_typo(language, configuration: Kotoshu.configuration, force: false)
120
+ cache = Cache::ModelCache.new(cache_path: configuration.cache_path)
121
+ typo = cache.download_typo_biencoder(force: force)
122
+ tier = cache.download_tiered_model(language, tier: :full, force_download: force)
123
+ { typo: typo, tier: tier }
124
+ end
125
+ end
@@ -0,0 +1,9 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Kotoshu
4
+ # The hybrid typo-retrieval layer (plan 131): opt-in, native-only,
5
+ # silent-degrade.
6
+ module Typo
7
+ autoload :Engine, "kotoshu/typo/engine"
8
+ end
9
+ end
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Kotoshu
4
- VERSION = '1.0.2'
4
+ VERSION = '1.0.3'
5
5
  end
data/lib/kotoshu.rb CHANGED
@@ -44,6 +44,7 @@ module Kotoshu
44
44
  # autoload so a missing extension can never break `require "kotoshu"`.
45
45
  autoload :Native, "kotoshu/native"
46
46
  autoload :NativeBackend, "kotoshu/native_backend"
47
+ autoload :Typo, "kotoshu/typo"
47
48
  autoload :Readers, "kotoshu/readers"
48
49
  autoload :ResourceBundle, "kotoshu/resource_bundle"
49
50
  autoload :ResourceManager, "kotoshu/resource_manager"
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: kotoshu
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.0.2
4
+ version: 1.0.3
5
5
  platform: ruby
6
6
  authors:
7
7
  - Ribose Inc.
8
8
  autorequire:
9
9
  bindir: exe
10
10
  cert_chain: []
11
- date: 2026-09-12 00:00:00.000000000 Z
11
+ date: 2026-09-14 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: thor
@@ -139,6 +139,7 @@ files:
139
139
  - TODO.impl/131-hybrid-productization.md
140
140
  - TODO.impl/132-techniques-page-and-prose.md
141
141
  - TODO.impl/133-rust-first-distribution.md
142
+ - TODO.impl/134-post-131-improvements.md
142
143
  - TODO.impl/20-encapsulation-cleanup.md
143
144
  - TODO.impl/21-autoload-migration.md
144
145
  - TODO.impl/22-url-pin-single-source-of-truth.md
@@ -519,12 +520,16 @@ files:
519
520
  - lib/kotoshu/suggestions/strategies/phonetic_strategy.rb
520
521
  - lib/kotoshu/suggestions/strategies/semantic_strategy.rb
521
522
  - lib/kotoshu/suggestions/strategies/symspell_strategy.rb
523
+ - lib/kotoshu/suggestions/strategies/typo_retrieval_strategy.rb
522
524
  - lib/kotoshu/suggestions/strategies/word_index.rb
523
525
  - lib/kotoshu/suggestions/suggestion.rb
524
526
  - lib/kotoshu/suggestions/suggestion_set.rb
525
527
  - lib/kotoshu/suggestions/sweep_index.rb
528
+ - lib/kotoshu/suggestions/typo_merge.rb
526
529
  - lib/kotoshu/tasks.rb
527
530
  - lib/kotoshu/tasks/check_task.rb
531
+ - lib/kotoshu/typo.rb
532
+ - lib/kotoshu/typo/engine.rb
528
533
  - lib/kotoshu/validators.rb
529
534
  - lib/kotoshu/validators/spelling_validator.rb
530
535
  - lib/kotoshu/version.rb