kotoshu 1.0.0 → 1.0.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/TODO.impl/117-deterministic-frequency-tiers.md +11 -3
- data/TODO.impl/118-server-gem-floor.md +33 -0
- data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
- data/TODO.impl/120-setup-metadata-refresh.md +35 -0
- data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
- data/TODO.impl/122-pack-compression-kpk2.md +66 -0
- data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
- data/TODO.impl/124-owner-residue.md +19 -0
- data/TODO.impl/125-consumer-floor-guards.md +43 -0
- data/TODO.impl/126-tier-error-budgets.md +33 -0
- data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
- data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
- data/TODO.impl/129-pt-fr-corpora.md +23 -0
- data/TODO.impl/130-news-truth-0912.md +23 -0
- data/TODO.impl/131-hybrid-productization.md +42 -0
- data/TODO.impl/132-techniques-page-and-prose.md +27 -0
- data/TODO.impl/133-rust-first-distribution.md +68 -0
- data/lib/kotoshu/configuration.rb +7 -2
- data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
- data/lib/kotoshu/data/frozen_kelly.rb +38 -0
- data/lib/kotoshu/jekyll.rb +4 -1
- data/lib/kotoshu/suggestions/frequency_provider.rb +8 -1
- data/lib/kotoshu/suggestions.rb +1 -0
- data/lib/kotoshu/version.rb +1 -1
- data/scripts/build_native_gem.rb +45 -0
- data/scripts/generate_frozen_tiers.rb +44 -0
- metadata +22 -2
|
@@ -0,0 +1,68 @@
|
|
|
1
|
+
# Plan 133 — Rust-first distribution: precompiled native gems and complete wheels
|
|
2
|
+
|
|
3
|
+
Status: proposed (execution-ready)
|
|
4
|
+
Depends on: the 1.0 conformance contract (identical outputs make the
|
|
5
|
+
backend default flip correctness-free); plan 93 (the python wheel matrix)
|
|
6
|
+
|
|
7
|
+
## Problem
|
|
8
|
+
|
|
9
|
+
The Rust engine is the faster implementation (~10-15x on the sweep) and
|
|
10
|
+
the strategic default, but today reaching it requires a build toolchain:
|
|
11
|
+
the gem compiles its native extension at install time, and the python
|
|
12
|
+
native wheel matrix is incomplete on PyPI. Users who cannot run a
|
|
13
|
+
toolchain get pure Ruby whether they wanted it or not — and users who
|
|
14
|
+
wanted Rust pay a compile for it. The architecture makes both engines
|
|
15
|
+
byte-identical, so distribution — not capability — is the only gap.
|
|
16
|
+
|
|
17
|
+
## Design
|
|
18
|
+
|
|
19
|
+
### Ruby: precompiled platform gems (the rb_sys flow)
|
|
20
|
+
|
|
21
|
+
1. New gem workflow `release-native.yml`: build matrix over
|
|
22
|
+
{x86_64-linux, aarch64-linux, x86_64-linux-musl, arm64-darwin,
|
|
23
|
+
x86_64-darwin, x64-mingw-ucrt} x {ruby 3.1..4.0} using
|
|
24
|
+
rb-sys-dock images for the Linux targets and native runners for
|
|
25
|
+
macOS/Windows; each job builds and pushes
|
|
26
|
+
`kotoshu-X.Y.Z-<platform>.gem` through trusted publishing. The
|
|
27
|
+
pure-Ruby platform gem continues to publish exactly as today, so
|
|
28
|
+
every unmatched platform resolves to it.
|
|
29
|
+
2. Backend default `ruby` -> `auto`: use the native engine when the
|
|
30
|
+
extension loads, fall back silently to pure Ruby when it does not.
|
|
31
|
+
Auto never downloads and never fails a check; specs pin both
|
|
32
|
+
branches; the conformance job already runs both engines.
|
|
33
|
+
3. The docker CI image drops its build-tool layer (it installs the
|
|
34
|
+
platform gem instead) — smaller image, faster builds, and its
|
|
35
|
+
version guard gains a native-backend assertion.
|
|
36
|
+
|
|
37
|
+
### Python: complete the kotoshu-native wheel matrix
|
|
38
|
+
|
|
39
|
+
1. Audit PyPI's published wheels for kotoshu-native against the
|
|
40
|
+
plan-93 matrix: manylinux x86_64/aarch64, macOS x86_64/arm64,
|
|
41
|
+
windows x86_64 (+ arm64 if tractable), sdist.
|
|
42
|
+
2. Fill the gaps in release-pypi.yml (maturin + the standard target
|
|
43
|
+
matrix), publish a patch release, and verify pip resolves a wheel
|
|
44
|
+
(no source build) on each platform.
|
|
45
|
+
3. Add the trove classifiers for every published target.
|
|
46
|
+
|
|
47
|
+
### Site
|
|
48
|
+
|
|
49
|
+
Install page's engines-and-bindings section gains the resolution
|
|
50
|
+
sentence: the native engine installs automatically wherever a prebuilt
|
|
51
|
+
artifact exists for your platform, and pure Ruby is always the
|
|
52
|
+
fallback; python's entry notes the wheel coverage.
|
|
53
|
+
|
|
54
|
+
## Gates
|
|
55
|
+
|
|
56
|
+
- A clean-machine `gem install kotoshu` on each published platform
|
|
57
|
+
loads the native backend with no toolchain present (CI-verifiable
|
|
58
|
+
on the matrix runners).
|
|
59
|
+
- The 2,630-vector conformance compare passes under `auto` on both a
|
|
60
|
+
native and a pure-Ruby install.
|
|
61
|
+
- pip on each target platform resolves a wheel, not the sdist.
|
|
62
|
+
- The pure-Ruby path is unchanged: same gem, same outputs, same specs.
|
|
63
|
+
|
|
64
|
+
## Non-goals
|
|
65
|
+
|
|
66
|
+
- No behavior change of any kind — outputs are frozen; only which
|
|
67
|
+
engine computes them changes.
|
|
68
|
+
- No deprecation of anything: pure Ruby is a feature, not a legacy.
|
|
@@ -207,7 +207,7 @@ module Kotoshu
|
|
|
207
207
|
backend: {
|
|
208
208
|
env: "KOTOSHU_BACKEND",
|
|
209
209
|
default: "ruby",
|
|
210
|
-
description: "Engine backend for correct?/suggest (
|
|
210
|
+
description: "Engine backend for correct?/suggest (auto by default: native when the extension is available, ruby otherwise; or force either)",
|
|
211
211
|
type: String
|
|
212
212
|
},
|
|
213
213
|
personal_dictionary: {
|
|
@@ -235,7 +235,12 @@ module Kotoshu
|
|
|
235
235
|
default_language: "en",
|
|
236
236
|
resource_pin: "main",
|
|
237
237
|
semantic_cascade_threshold: 1.0,
|
|
238
|
-
|
|
238
|
+
# "auto" (plan 133): the Rust engine whenever the native
|
|
239
|
+
# extension loaded and the dictionary is a file-backed Hunspell,
|
|
240
|
+
# pure Ruby otherwise — identical outputs either way (the
|
|
241
|
+
# conformance contract), so the default is the faster engine
|
|
242
|
+
# with the pure fallback, never a failure.
|
|
243
|
+
backend: "auto",
|
|
239
244
|
personal_dictionary: true
|
|
240
245
|
}.freeze
|
|
241
246
|
|