kotoshu 0.9.3 → 0.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +65 -0
- data/Cargo.lock +72 -4
- data/Rakefile +35 -0
- data/TODO.impl/00-vision.md +32 -0
- data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
- data/TODO.impl/106-native-lid-detection.md +36 -0
- data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
- data/TODO.impl/108-ko-ne-full-feature.md +37 -0
- data/TODO.impl/109-ten-audit.md +26 -0
- data/docs/STABILITY-1.0-AUDIT.md +251 -0
- data/docs/STABILITY-1.0.md +137 -0
- data/docs/cache-architecture.md +1 -1
- data/ext/kotoshu_native/Cargo.toml +5 -2
- data/ext/kotoshu_native/src/lib.rs +134 -6
- data/lib/kotoshu/cache/language_cache.rb +26 -11
- data/lib/kotoshu/cache/model_cache.rb +150 -2
- data/lib/kotoshu/cache/staged_languages.rb +45 -0
- data/lib/kotoshu/cache.rb +3 -0
- data/lib/kotoshu/cli/dict_command.rb +64 -0
- data/lib/kotoshu/cli.rb +9 -57
- data/lib/kotoshu/configuration.rb +20 -2
- data/lib/kotoshu/fluent_checker.rb +3 -0
- data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
- data/lib/kotoshu/keyboard/registry.rb +24 -2
- data/lib/kotoshu/keyboard.rb +6 -0
- data/lib/kotoshu/language/lid_detector.rb +188 -0
- data/lib/kotoshu/language/script.rb +128 -0
- data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
- data/lib/kotoshu/language.rb +9 -0
- data/lib/kotoshu/languages/ko/language.rb +56 -0
- data/lib/kotoshu/languages/ne/language.rb +60 -0
- data/lib/kotoshu/languages/nn/language.rb +26 -0
- data/lib/kotoshu/languages.rb +8 -0
- data/lib/kotoshu/multi_language_checker.rb +3 -0
- data/lib/kotoshu/spellchecker.rb +45 -8
- data/lib/kotoshu/version.rb +1 -1
- data/lib/kotoshu.rb +45 -10
- data/scripts/extract_eval_grids.rb +1 -1
- data/sig/kotoshu.rbs +77 -0
- metadata +20 -5
- data/lib/kotoshu/commands/cache_command.rb +0 -421
- data/lib/kotoshu/commands/check_command.rb +0 -301
- data/lib/kotoshu/commands/model_command.rb +0 -291
- data/lib/kotoshu/commands.rb +0 -12
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: a193e09b8fdaaf3e05a6b988e47ca460d8f7797b30e1719d109fdb9d28a89794
|
|
4
|
+
data.tar.gz: 377667004f6fb82be98470efa6e667a623afd96c01c61872db8b4c0048329c54
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 4c90da5e79e5b5f5a8e91a3a3b734874cc30699c9d95462f8cdae9160389601de3eea63b7231e19ba887e41d4b1e99ce12ec0ed07351868bb48c2ac8ba852007
|
|
7
|
+
data.tar.gz: de5adbf779c0c384b91186a73aaa571fe02bd49c84e7cad3a8bf62b366b76da4b7a853745c68b50eae6c5482bfe95fb1e55ca3f4798eb73b4194ea48c358ef62
|
data/CHANGELOG.md
CHANGED
|
@@ -8,6 +8,71 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
10
|
### Added
|
|
11
|
+
- **Korean and Nepali full-feature modules** (plan 108) - the last
|
|
12
|
+
big-population languages without modules move from the plan 107
|
|
13
|
+
basic tier to full feature. `ko` ships the Hangul tokenizer
|
|
14
|
+
(an eojeol is a run of Hangul syllables U+AC00-D7A3 plus bare jamo
|
|
15
|
+
U+1100-11FF/U+3130-318F that escaped IME composition) and the
|
|
16
|
+
Dubeolsik keyboard grid keyed on jamo per the KS X 5002 standard 2-set
|
|
17
|
+
layout, so typo proximity lands on the unit vowel mergers and
|
|
18
|
+
fortis/lenis confusions operate on. `ne` ships the Devanagari
|
|
19
|
+
tokenizer (matras U+093E-094C, the virama U+094D and vowel signs
|
|
20
|
+
stay attached to the base consonant; the danda and Devanagari
|
|
21
|
+
digits are separators) and the Devanagari InScript grid (the Indian
|
|
22
|
+
national standard, mirrored from the models repo eval harness and
|
|
23
|
+
drift-checked against it; the Nepali Traditional Romanized layout
|
|
24
|
+
is more common on the ground but is not a documented standard).
|
|
25
|
+
Both dictionaries were already staged, so `kotoshu setup ko ne`
|
|
26
|
+
serves them; full-feature languages grow 33 to 35.
|
|
27
|
+
- **Basic support for every staged dictionary language** (plan 107) -
|
|
28
|
+
`kotoshu setup` now serves all 95 languages staged in the
|
|
29
|
+
kotoshu/dictionaries manifest, not just the ones with gem modules.
|
|
30
|
+
`LanguageCache::AVAILABLE_LANGUAGES` derives from a vendored
|
|
31
|
+
snapshot of the manifest (Cache::StagedLanguages, refreshed by
|
|
32
|
+
`rake kotoshu:staged_languages:sync`), keeping the hot path
|
|
33
|
+
network-free. Languages without a module run on the basic tier:
|
|
34
|
+
script classification (Language::Script) picks the tokenizer whose
|
|
35
|
+
word regex drives check/suggest word extraction (shared Latin /
|
|
36
|
+
Cyrillic / Greek tokenizers plus a generic regex tokenizer for
|
|
37
|
+
Arabic, Hebrew, Armenian, Georgian, Devanagari and Hangul), and the
|
|
38
|
+
keyboard registry falls back to the script's standard grid (JCUKEN
|
|
39
|
+
for Cyrillic, Arabic 101, Hebrew SI-1452, QWERTY otherwise) instead
|
|
40
|
+
of QWERTY for everything. Two documented tiers: full feature =
|
|
41
|
+
module (registered keyboard + verified specimens,
|
|
42
|
+
`LanguageCache.full_feature_languages`), basic = manifest-present.
|
|
43
|
+
`nn` (Norwegian Nynorsk) is wired as a full-feature module while
|
|
44
|
+
here - dictionary and model are both staged. Modules keep winning
|
|
45
|
+
over the fallbacks; the 32 existing modules behave identically
|
|
46
|
+
(conformance vectors untouched).
|
|
47
|
+
- **Personal dictionary in the check path** (plan 105) -
|
|
48
|
+
`Spellchecker#check` (and therefore `Kotoshu.check`,
|
|
49
|
+
`Kotoshu.check_file`, and `kotoshu check`) no longer flags words
|
|
50
|
+
present in the user personal dictionary
|
|
51
|
+
(~/.config/kotoshu/personal.dic, KOTOSHU_PERSONAL_DIC override),
|
|
52
|
+
matching the semantics the LSP already applies to diagnostics:
|
|
53
|
+
case-insensitive, filtered at result assembly without suppression
|
|
54
|
+
metadata, words still counted, and `suggest`/`correct?` untouched so
|
|
55
|
+
personal words stay spellcheckable. The dictionary is loaded once
|
|
56
|
+
per process (no hot reload; the LSP keeps its mtime-based reload).
|
|
57
|
+
Opt out with `Kotoshu.configure { |c| c.personal_dictionary = false }`
|
|
58
|
+
/ KOTOSHU_PERSONAL_DICTIONARY=false / `kotoshu check --no-personal`.
|
|
59
|
+
- **Native language detection** (plan 106) - `Kotoshu.detect_language(text)`
|
|
60
|
+
now returns a `Kotoshu::Language::Detection` (`code` + `score`) scored by
|
|
61
|
+
the Rust extension over the `kotoshu://models/lid/lid-176` registry
|
|
62
|
+
artifact pair (the ONNX-converted lid.176 model, 176 languages,
|
|
63
|
+
bit-faithful to the fastText bindings - parity 55/55 against the frozen
|
|
64
|
+
corpus shared with kotoshu-rs). Fetch the pair once with
|
|
65
|
+
`Kotoshu.setup_lid` (registry-resolved, sha-verified, cached under
|
|
66
|
+
`models-cache/lid/`); detect resolves cache-only and never downloads.
|
|
67
|
+
The 7-language `Language::Detector` heuristic stays the fallback when
|
|
68
|
+
the extension is absent, the model is not set up, or
|
|
69
|
+
`KOTOSHU_BACKEND=ruby` opts out - so pure-Ruby installs are unaffected
|
|
70
|
+
(no new gemspec dependency, no onnxruntime). The magnus shim
|
|
71
|
+
(ext/kotoshu_native) gains `Kotoshu::Native::LidModel.load` +
|
|
72
|
+
`#detect` behind the kotoshu-rs `model` feature; the workspace
|
|
73
|
+
Cargo.lock moves to the kotoshu-rs revision carrying the reader.
|
|
74
|
+
`Kotoshu.detect_language_with_confidence` keeps its `[code, score]`
|
|
75
|
+
shape and now routes through the same engine.
|
|
11
76
|
- **Full-feature language batch 3** (plan 100) - twelve languages gain the
|
|
12
77
|
downloadable spelling dictionary, a keyboard layout, and a gem module:
|
|
13
78
|
`ar` `id` `fa` `he` `bg` `sr` `hr` `sk` `sl` `lt` `lv` `et` (the RTL
|
data/Cargo.lock
CHANGED
|
@@ -26,7 +26,7 @@ dependencies = [
|
|
|
26
26
|
"regex",
|
|
27
27
|
"rustc-hash",
|
|
28
28
|
"shlex",
|
|
29
|
-
"syn",
|
|
29
|
+
"syn 2.0.119",
|
|
30
30
|
]
|
|
31
31
|
|
|
32
32
|
[[package]]
|
|
@@ -82,12 +82,20 @@ dependencies = [
|
|
|
82
82
|
"either",
|
|
83
83
|
]
|
|
84
84
|
|
|
85
|
+
[[package]]
|
|
86
|
+
name = "itoa"
|
|
87
|
+
version = "1.0.18"
|
|
88
|
+
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
89
|
+
checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682"
|
|
90
|
+
|
|
85
91
|
[[package]]
|
|
86
92
|
name = "kotoshu"
|
|
87
93
|
version = "0.1.0"
|
|
88
|
-
source = "git+https://github.com/kotoshu/kotoshu-rs?branch=main#
|
|
94
|
+
source = "git+https://github.com/kotoshu/kotoshu-rs?branch=main#e574c445df5f6eee8e978f1f4d26dc48cfb1947b"
|
|
89
95
|
dependencies = [
|
|
90
96
|
"magnus",
|
|
97
|
+
"serde",
|
|
98
|
+
"serde_json",
|
|
91
99
|
]
|
|
92
100
|
|
|
93
101
|
[[package]]
|
|
@@ -141,7 +149,7 @@ checksum = "47607461fd8e1513cb4f2076c197d8092d921a1ea75bd08af97398f593751892"
|
|
|
141
149
|
dependencies = [
|
|
142
150
|
"proc-macro2",
|
|
143
151
|
"quote",
|
|
144
|
-
"syn",
|
|
152
|
+
"syn 2.0.119",
|
|
145
153
|
]
|
|
146
154
|
|
|
147
155
|
[[package]]
|
|
@@ -205,7 +213,7 @@ dependencies = [
|
|
|
205
213
|
"quote",
|
|
206
214
|
"regex",
|
|
207
215
|
"shell-words",
|
|
208
|
-
"syn",
|
|
216
|
+
"syn 2.0.119",
|
|
209
217
|
]
|
|
210
218
|
|
|
211
219
|
[[package]]
|
|
@@ -255,6 +263,49 @@ version = "0.3.6"
|
|
|
255
263
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
256
264
|
checksum = "1bc711410fbe7399f390ca1c3b60ad0f53f80e95c5eb935e52268a0e2cd49acc"
|
|
257
265
|
|
|
266
|
+
[[package]]
|
|
267
|
+
name = "serde"
|
|
268
|
+
version = "1.0.229"
|
|
269
|
+
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
270
|
+
checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba"
|
|
271
|
+
dependencies = [
|
|
272
|
+
"serde_core",
|
|
273
|
+
"serde_derive",
|
|
274
|
+
]
|
|
275
|
+
|
|
276
|
+
[[package]]
|
|
277
|
+
name = "serde_core"
|
|
278
|
+
version = "1.0.229"
|
|
279
|
+
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
280
|
+
checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48"
|
|
281
|
+
dependencies = [
|
|
282
|
+
"serde_derive",
|
|
283
|
+
]
|
|
284
|
+
|
|
285
|
+
[[package]]
|
|
286
|
+
name = "serde_derive"
|
|
287
|
+
version = "1.0.229"
|
|
288
|
+
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
289
|
+
checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348"
|
|
290
|
+
dependencies = [
|
|
291
|
+
"proc-macro2",
|
|
292
|
+
"quote",
|
|
293
|
+
"syn 3.0.5",
|
|
294
|
+
]
|
|
295
|
+
|
|
296
|
+
[[package]]
|
|
297
|
+
name = "serde_json"
|
|
298
|
+
version = "1.0.151"
|
|
299
|
+
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
300
|
+
checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14"
|
|
301
|
+
dependencies = [
|
|
302
|
+
"itoa",
|
|
303
|
+
"memchr",
|
|
304
|
+
"serde",
|
|
305
|
+
"serde_core",
|
|
306
|
+
"zmij",
|
|
307
|
+
]
|
|
308
|
+
|
|
258
309
|
[[package]]
|
|
259
310
|
name = "shell-words"
|
|
260
311
|
version = "1.1.1"
|
|
@@ -278,6 +329,17 @@ dependencies = [
|
|
|
278
329
|
"unicode-ident",
|
|
279
330
|
]
|
|
280
331
|
|
|
332
|
+
[[package]]
|
|
333
|
+
name = "syn"
|
|
334
|
+
version = "3.0.5"
|
|
335
|
+
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
336
|
+
checksum = "12df2e0110f65b775f769bb17ef989067a1d931b2eb822bd4346631eeada89f9"
|
|
337
|
+
dependencies = [
|
|
338
|
+
"proc-macro2",
|
|
339
|
+
"quote",
|
|
340
|
+
"unicode-ident",
|
|
341
|
+
]
|
|
342
|
+
|
|
281
343
|
[[package]]
|
|
282
344
|
name = "unicode-ident"
|
|
283
345
|
version = "1.0.24"
|
|
@@ -289,3 +351,9 @@ name = "windows-link"
|
|
|
289
351
|
version = "0.2.1"
|
|
290
352
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
291
353
|
checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"
|
|
354
|
+
|
|
355
|
+
[[package]]
|
|
356
|
+
name = "zmij"
|
|
357
|
+
version = "1.0.23"
|
|
358
|
+
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
359
|
+
checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b"
|
data/Rakefile
CHANGED
|
@@ -32,6 +32,41 @@ report_divergences = lambda do |divergences|
|
|
|
32
32
|
end
|
|
33
33
|
|
|
34
34
|
namespace :kotoshu do
|
|
35
|
+
namespace :staged_languages do
|
|
36
|
+
desc <<~DESC
|
|
37
|
+
Regenerate the vendored staged-language list (plan 107). Fetches \
|
|
38
|
+
the dictionaries manifest at the SourceRegistry pin, extracts the \
|
|
39
|
+
distinct language codes from its resources, and rewrites \
|
|
40
|
+
lib/kotoshu/cache/staged_languages.rb in place so \
|
|
41
|
+
LanguageCache::AVAILABLE_LANGUAGES tracks the manifest. Run after \
|
|
42
|
+
the dictionaries repo stages new languages.
|
|
43
|
+
DESC
|
|
44
|
+
task :sync do
|
|
45
|
+
require "kotoshu"
|
|
46
|
+
require "json"
|
|
47
|
+
|
|
48
|
+
url = Kotoshu::SourceRegistry.new.url_for(:dict_manifest)
|
|
49
|
+
puts "Fetching #{url}"
|
|
50
|
+
bytes = Kotoshu::Integrity::NetHTTP.get(url)
|
|
51
|
+
abort "manifest not found at #{url}" if bytes.nil?
|
|
52
|
+
|
|
53
|
+
manifest = JSON.parse(bytes)
|
|
54
|
+
codes = manifest.fetch("resources").values.map { |r| r.fetch("language") }.uniq.sort
|
|
55
|
+
abort "manifest carried no languages" if codes.empty?
|
|
56
|
+
|
|
57
|
+
target = File.expand_path("lib/kotoshu/cache/staged_languages.rb", __dir__)
|
|
58
|
+
source = File.read(target)
|
|
59
|
+
updated = source.sub(/CODES = %w\[\n(?: {8}\S+(?: \S+)*\n)+ {6}\]\.freeze/) do
|
|
60
|
+
wrapped = codes.each_slice(10).map { |slice| " #{slice.join(' ')}" }.join("\n")
|
|
61
|
+
"CODES = %w[\n#{wrapped}\n ].freeze"
|
|
62
|
+
end
|
|
63
|
+
abort "CODES block not found in #{target}" if updated == source
|
|
64
|
+
|
|
65
|
+
File.write(target, updated)
|
|
66
|
+
puts "Rewrote #{target}: #{codes.size} languages (was #{Kotoshu::Cache::StagedLanguages::CODES.size})"
|
|
67
|
+
end
|
|
68
|
+
end
|
|
69
|
+
|
|
35
70
|
namespace :conformance do
|
|
36
71
|
desc <<~DESC
|
|
37
72
|
Export conformance vectors for kotoshu-rs (plan 67 M3). Walks the Spylls \
|
data/TODO.impl/00-vision.md
CHANGED
|
@@ -247,3 +247,35 @@ plan name the source.
|
|
|
247
247
|
| `docs/ARCHITECTURE_IMPROVEMENTS_PLAN.md` (5 phases) | Phase 1 perf items in `01`/`05`; Phase 2 architecture in `00-vision` (this doc); Phase 3 UX mostly implemented (`PersonalDictionary`, `ProjectConfig`, `FluentChecker` exist); Phase 4 quality in `10`; Phase 5 polish spread |
|
|
248
248
|
| `docs/TDD_ITERATION_STRATEGY.md` | `10` (test methodology + CI); project-rule overrides (no doubles, behavior-not-implementation) in `CLAUDE.md` |
|
|
249
249
|
| `docs/ONNX_FUNCTIONAL_VERIFICATION.md` | `05` (semantic path productionization) |
|
|
250
|
+
|
|
251
|
+
|
|
252
|
+
Plans 105-107 plus models 09 and site 11 are **wave 7, detection and
|
|
253
|
+
coverage depth** (2026-09-07): after the wave-6 releases (gem 0.9.3,
|
|
254
|
+
wasm 0.4.0, registry v1.4.0, action v2), the remaining gaps are the
|
|
255
|
+
personal dictionary never reaching the check path, detection stuck on
|
|
256
|
+
a 7-language heuristic, buckets only for en/de, and the playground not
|
|
257
|
+
yet passing bucket bytes to loadModel.
|
|
258
|
+
|
|
259
|
+
| # | Plan | Gap it closes |
|
|
260
|
+
|---|---|---|
|
|
261
|
+
| 105 | [Personal dictionary in check](105-personal-dictionary-in-check.md) | kotoshu personal add then check still flags the word |
|
|
262
|
+
| 106 | [Native LID detection](106-native-lid-detection.md) | detect is a 7-language heuristic; the pure-Rust lid reader exists unused |
|
|
263
|
+
| 107 | models [09 bucket expansion](../../models-fasttext-onnx/TODO.impl/09-bucket-table-expansion.md) | bucket OOV siblings exist only for en and de |
|
|
264
|
+
| — | site 11 playground buckets | wasm 0.4.0 accepts bucketsBytes; the worker never fetches the sibling |
|
|
265
|
+
|
|
266
|
+
|
|
267
|
+
Plans 107-109 are **wave 8, reachability and the road to 1.0**
|
|
268
|
+
(2026-09-07): after wave 7 (personal dict in check, 176-language
|
|
269
|
+
detection, buckets rolling to all languages), the remaining user gaps
|
|
270
|
+
are the sixty-plus staged dictionaries unreachable without a module,
|
|
271
|
+
the last big-population languages (ko, ne), and the absence of a
|
|
272
|
+
stability commitment adopters can build on.
|
|
273
|
+
|
|
274
|
+
| # | Plan | Gap it closes |
|
|
275
|
+
|---|---|---|
|
|
276
|
+
| 107 | [Basic support for every staged language](107-basic-support-every-staged-language.md) | setup rejects 60+ staged languages; modules become upgrades, not gates |
|
|
277
|
+
| 108 | [ko and ne full-feature](108-ko-ne-full-feature.md) | Hangul and Devanagari - the last ~100M speakers without support |
|
|
278
|
+
| 109 | [The 1.0 readiness audit](109-ten-audit.md) | no frozen surface or stability policy for adopters |
|
|
279
|
+
|
|
280
|
+
Sequencing: 108 after 107 (same area); 109 is analysis-only and runs in
|
|
281
|
+
parallel. The 1.0 cut itself remains the owner's version call.
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
# Plan 105 — The personal dictionary reaches the check path
|
|
2
|
+
|
|
3
|
+
## Why
|
|
4
|
+
`kotoshu personal add recieve` then `kotoshu check` still flags the word —
|
|
5
|
+
PersonalDictionary is wired only into the `personal` subcommand and the LSP
|
|
6
|
+
(resolved server-side), not into `Spellchecker#check`. Users who add words
|
|
7
|
+
see them flagged again on the very next run; the README story ("add words
|
|
8
|
+
you accept") is currently false on the CLI/API path.
|
|
9
|
+
|
|
10
|
+
## Work (gem repo)
|
|
11
|
+
1. `Spellchecker#check` consults the personal dictionary: a word present in
|
|
12
|
+
personal.dic is not an error (mirror the LSP's semantics: filter at
|
|
13
|
+
result assembly, keep suppression metadata out of the error rows).
|
|
14
|
+
2. Reload semantics: the LSP reloads on mtime change; the CLI/one-shot path
|
|
15
|
+
loads once per process. Do not add hot reload here.
|
|
16
|
+
3. Opt-out: `Kotoshu.configure { personal_dictionary false }` (or the
|
|
17
|
+
existing config seam — find it) and `--no-personal` on `kotoshu check`.
|
|
18
|
+
4. Specs: add-then-check round trip on a real dictionary; opt-out flags
|
|
19
|
+
again; personal words still spellcheckable via suggest.
|
|
20
|
+
5. Docs truth: /docs/ignores gains the personal dictionary section (the
|
|
21
|
+
action PR #2 README noted this gap) — deliver the copy in the PR body
|
|
22
|
+
for the site follow-up, do not touch the site repo.
|
|
23
|
+
|
|
24
|
+
## Verification
|
|
25
|
+
Suite green (3,900+ examples); rubocop clean; `kotoshu personal add x &&
|
|
26
|
+
kotoshu check` live round trip recorded in the PR body.
|
|
27
|
+
|
|
28
|
+
## Status
|
|
29
|
+
Executed 2026-09-07 — gem PR (branch plan-105-personal-dict-in-check):
|
|
30
|
+
check-path filter mirroring the LSP, one load per process,
|
|
31
|
+
Configuration#personal_dictionary + KOTOSHU_PERSONAL_DICTIONARY +
|
|
32
|
+
`kotoshu check --no-personal`; suite pins KOTOSHU_PERSONAL_DIC for
|
|
33
|
+
hermeticity. /docs/ignores copy delivered in the PR body for the site
|
|
34
|
+
follow-up.
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
# Plan 106 — Document language detection through the pure-Rust LID reader
|
|
2
|
+
|
|
3
|
+
## Why
|
|
4
|
+
The gem's LanguageIdentifier needs the Python fastText bindings (and breaks
|
|
5
|
+
under numpy 2), so /v1/detect ships a 7-language heuristic. kotoshu-rs now
|
|
6
|
+
carries a pure-Rust lid.176 reader (kotoshu/src/lid, parity 55/55 vs the
|
|
7
|
+
gem) behind the same `model` feature the native ext already builds.
|
|
8
|
+
|
|
9
|
+
## Work (gem repo)
|
|
10
|
+
1. Expose `Kotoshu.detect_language(text)` through the native backend:
|
|
11
|
+
download/resolve lid.176.onnx + vocab via the models registry
|
|
12
|
+
(kotoshu://models/lid/lid-176, v1.4.0), score with the ext, return
|
|
13
|
+
{ code, score }. Heuristic Language::Detector stays the fallback when
|
|
14
|
+
the native ext is absent or the model is not set up.
|
|
15
|
+
2. FFI seam: extend the magnus shim (ext/kotoshu_native) with the call —
|
|
16
|
+
mirror how suggest/check already cross; keep it behind the existing
|
|
17
|
+
feature so pure-Ruby installs are unaffected.
|
|
18
|
+
3. Server follow-up (kotoshu-server repo, separate PR riding this):
|
|
19
|
+
/v1/detect prefers Kotoshu.detect_language when available. Note it in
|
|
20
|
+
the PR body; do not touch the server repo yourself.
|
|
21
|
+
4. Specs: native-vs-fixture parity (reuse the rs corpus JSON shape),
|
|
22
|
+
fallback behavior, registry miss error.
|
|
23
|
+
|
|
24
|
+
## Verification
|
|
25
|
+
Suite green; a live detect over en/de/ru/ja/ar samples recorded in the PR
|
|
26
|
+
body; pure-Ruby suite still green (no new hard dependency).
|
|
27
|
+
|
|
28
|
+
## Status
|
|
29
|
+
Executed 2026-09-07 — gem PR (branch plan-106-native-lid-detection):
|
|
30
|
+
Kotoshu.detect_language -> Language::Detection through the ext
|
|
31
|
+
(LidModel.load/#detect in the magnus shim, kotoshu-rs model feature,
|
|
32
|
+
Cargo.lock bumped to the reader revision); ModelCache gained the
|
|
33
|
+
lid pair (download_lid/load_cached_lid, top-level lid/ dir);
|
|
34
|
+
Kotoshu.setup_lid two-stage; heuristic fallback incl.
|
|
35
|
+
KOTOSHU_BACKEND=ruby; parity 55/55 vs the shared frozen corpus.
|
|
36
|
+
Server follow-up (/v1/detect) noted in the PR body.
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
# Plan 107 — Basic support for every staged dictionary language
|
|
2
|
+
|
|
3
|
+
## Why
|
|
4
|
+
`kotoshu setup` only serves the 32 languages with modules. Sixty-plus staged
|
|
5
|
+
dictionaries (from gd to ia) are unreachable: a Welsh or Icelandic user
|
|
6
|
+
cannot use the gem at all, even though the dictionary and model are
|
|
7
|
+
downloadable. Modules should be the UPGRADE (keyboard + verified specimens),
|
|
8
|
+
not the gate.
|
|
9
|
+
|
|
10
|
+
## Work (gem repo)
|
|
11
|
+
1. `LanguageCache::AVAILABLE_LANGUAGES` derives from the dictionaries
|
|
12
|
+
manifest (the source the registry/staging uses) instead of the hardcoded
|
|
13
|
+
module list; modules register themselves as full-feature upgrades.
|
|
14
|
+
2. Default behavior for module-less languages: script-appropriate tokenizer
|
|
15
|
+
(reuse the script-regex machinery from plan 91), generic keyboard
|
|
16
|
+
fallback (QWERTY for Latin, JCUKEN Cyrillic, Arabic 101 Arabic-script),
|
|
17
|
+
setup/check/suggest all functional.
|
|
18
|
+
3. Wire nn (model + dictionary exist) as a full-feature module while here;
|
|
19
|
+
note sr-Latn as follow-up if the Latin mapping needs product sign-off.
|
|
20
|
+
4. Specs: setup + specimen round trip on 3 module-less staged languages
|
|
21
|
+
(e.g. is, cy, gd); full-feature languages behave identically to today;
|
|
22
|
+
conformance untouched.
|
|
23
|
+
|
|
24
|
+
## Verification
|
|
25
|
+
Live `kotoshu setup is && check -l is` round trip; suite green; the 32
|
|
26
|
+
modules' outputs byte-identical.
|
|
27
|
+
|
|
28
|
+
## Status
|
|
29
|
+
Executed 2026-09-07 — AVAILABLE_LANGUAGES derives from a vendored
|
|
30
|
+
snapshot of the dictionaries manifest (Cache::StagedLanguages, 95
|
|
31
|
+
languages, refreshed by `rake kotoshu:staged_languages:sync`);
|
|
32
|
+
module-less languages ride Language::Script (script tokenizer for
|
|
33
|
+
word extraction: shared Latin/Cyrillic/Greek classes plus a generic
|
|
34
|
+
regex tokenizer for Arabic/Hebrew/Armenian/Georgian/Devanagari/
|
|
35
|
+
Hangul) and the script keyboard default (JCUKEN Cyrillic, Arabic 101,
|
|
36
|
+
Hebrew SI-1452, QWERTY Latin). Tiers documented: full feature =
|
|
37
|
+
module (LanguageCache.full_feature_languages, 33 with nn); basic =
|
|
38
|
+
manifest-present. nn wired as a full-feature module. Specimens
|
|
39
|
+
engine-verified for is cy gd (basic) + nn; live round trips
|
|
40
|
+
setup+check -l for is cy gd nn; conformance 2630 vectors 0 failures;
|
|
41
|
+
suite 3948 examples 0 failures. sr-Latn stays follow-up (normalize
|
|
42
|
+
maps it to sr; needs product sign-off).
|
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
# Plan 108 — Korean and Nepali: the last big-population languages
|
|
2
|
+
|
|
3
|
+
## Why
|
|
4
|
+
ko (~82M) and ne (~16M) were deferred from batch 3 because Hangul jamo
|
|
5
|
+
input and Devanagari conjuncts need ja-style tokenizer treatment. They are
|
|
6
|
+
the largest populations without full-feature support; both have staged
|
|
7
|
+
dictionaries and models.
|
|
8
|
+
|
|
9
|
+
## Work (gem repo; sequenced AFTER plan 107 merges — same area)
|
|
10
|
+
1. ko: Hangul tokenizer (syllable-block aware, jamo-level word regex so
|
|
11
|
+
check extracts real eojeol), Korean keyboard grid (Dubeolsik 2-set),
|
|
12
|
+
module + specimens engine-verified.
|
|
13
|
+
2. ne: Devanagari tokenizer (matras and conjuncts stay attached to the
|
|
14
|
+
base consonant — the Unicode grapheme rules the script needs), Roman
|
|
15
|
+
keyboard fallback or a Nepali traditional grid if defensibly sourced,
|
|
16
|
+
module + specimens.
|
|
17
|
+
3. AVAILABLE_LANGUAGES + site-flip deliverables (specimen table, sizes).
|
|
18
|
+
|
|
19
|
+
## Verification
|
|
20
|
+
Same discipline as batch 3: every specimen engine-verified through
|
|
21
|
+
correct?/suggest; RTL/complex-script sentence round trips through
|
|
22
|
+
Kotoshu.check; suite green.
|
|
23
|
+
|
|
24
|
+
## Status
|
|
25
|
+
Executed 2026-09-07 — ko: HangulTokenizer (eojeol = run of syllables
|
|
26
|
+
U+AC00-D7A3 plus bare jamo U+1100-11FF/U+3130-318F), Dubeolsik grid
|
|
27
|
+
keyed on jamo per KS X 5002 (the eval harness models ko slips as
|
|
28
|
+
jamo confusion pairs, not a key grid, so no drift check applies),
|
|
29
|
+
module + 7 engine-verified specimens. ne: DevanagariTokenizer
|
|
30
|
+
(matras and virama conjuncts attached to the base consonant; danda
|
|
31
|
+
and Devanagari digits are separators), Devanagari InScript grid
|
|
32
|
+
mirrored from the eval harness _NE_INSCRIPT and drift-checked (the
|
|
33
|
+
defensible standard — Traditional Romanized is common but
|
|
34
|
+
undocumented), module + 3 specimens. Full-feature languages 33 -> 35.
|
|
35
|
+
Live round trips setup + check -l ko ne recorded; conformance 2630
|
|
36
|
+
vectors 0 failures; suite 3985 examples 0 failures. Specimen table
|
|
37
|
+
and dictionary sizes in the PR body for the site flip.
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
# Plan 109 — The 1.0 readiness audit
|
|
2
|
+
|
|
3
|
+
## Why
|
|
4
|
+
Adopters hesitate at 0.x. A 1.0 cut needs a frozen public surface and a
|
|
5
|
+
stated policy. The audit inventories what 1.0 would freeze and what it
|
|
6
|
+
would deprecate — it does NOT cut versions (owner decision).
|
|
7
|
+
|
|
8
|
+
## Work (gem repo docs/, analysis only)
|
|
9
|
+
1. Inventory the public surface: Ruby API (Kotoshu module functions,
|
|
10
|
+
Spellchecker, Configuration, Tasks, validators), CLI flags, wasm JS
|
|
11
|
+
surface (KotoshuWasm, loadModel/rerank/semanticSuggest/loadLid/
|
|
12
|
+
detectLanguage), HTTP contract (openapi.yaml), action inputs, registry
|
|
13
|
+
schema.
|
|
14
|
+
2. For each: stable / needs-deprecation-notice / internal-accidentally-
|
|
15
|
+
public. Flag anything accidentally public that 1.0 should privatize.
|
|
16
|
+
3. Draft the stability policy (semver commitments, deprecation window,
|
|
17
|
+
conformance vectors as the behavioral contract) as docs/STABILITY.md.
|
|
18
|
+
4. A 1.0 checklist: what must land first (nothing mechanical is known to
|
|
19
|
+
block), what the release train looks like (gem+wasm+server together).
|
|
20
|
+
|
|
21
|
+
## Verification
|
|
22
|
+
Every inventory line cites the file that defines it; the deprecation list
|
|
23
|
+
reviewed against gemspec/exported constants.
|
|
24
|
+
|
|
25
|
+
## Status
|
|
26
|
+
Pending
|