kotoshu 0.9.2 → 0.9.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (36) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +24 -0
  3. data/TODO.impl/00-vision.md +22 -0
  4. data/TODO.impl/100-full-feature-batch3.md +28 -0
  5. data/TODO.impl/101-int4-tier-experiment.md +32 -0
  6. data/TODO.impl/102-lid-chain.md +28 -0
  7. data/TODO.impl/103-oov-bucket-tables.md +27 -0
  8. data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
  9. data/TODO.impl/99-site-truth-performance-docs.md +31 -0
  10. data/lib/kotoshu/cache/language_cache.rb +4 -1
  11. data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
  12. data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
  13. data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
  14. data/lib/kotoshu/keyboard/layouts/hrsl.rb +78 -0
  15. data/lib/kotoshu/keyboard/layouts/latin.rb +47 -1
  16. data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
  17. data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
  18. data/lib/kotoshu/keyboard/registry.rb +20 -4
  19. data/lib/kotoshu/keyboard.rb +9 -0
  20. data/lib/kotoshu/languages/ar/language.rb +14 -0
  21. data/lib/kotoshu/languages/bg/language.rb +37 -0
  22. data/lib/kotoshu/languages/et/language.rb +24 -0
  23. data/lib/kotoshu/languages/fa/language.rb +16 -0
  24. data/lib/kotoshu/languages/he/language.rb +15 -0
  25. data/lib/kotoshu/languages/hr/language.rb +24 -0
  26. data/lib/kotoshu/languages/id/language.rb +25 -0
  27. data/lib/kotoshu/languages/lt/language.rb +24 -0
  28. data/lib/kotoshu/languages/lv/language.rb +24 -0
  29. data/lib/kotoshu/languages/sk/language.rb +25 -0
  30. data/lib/kotoshu/languages/sl/language.rb +25 -0
  31. data/lib/kotoshu/languages/sr/language.rb +38 -0
  32. data/lib/kotoshu/languages.rb +12 -0
  33. data/lib/kotoshu/version.rb +1 -1
  34. data/scripts/extract_eval_grids.rb +4 -1
  35. data/sig/kotoshu.rbs +24 -0
  36. metadata +22 -1
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 44cb12b37d8bf72fa6febd503bdcd713aa569d64e4c648f02c422e4d362ab891
4
- data.tar.gz: b42724adb7b58ca46c5494770cff1fb6c821b68272a545cd24e1f1ac152b3799
3
+ metadata.gz: 4891dc67bd85b5c342fdbc7453085990a1e9ff9bea5b394a67f87e11b4bbd6c7
4
+ data.tar.gz: 378d1b77bea8113d155774bc07e481afaa9229d6e1edcd7d5fda2805d28a963c
5
5
  SHA512:
6
- metadata.gz: 17486ce07cb10da6c26bfd643aa7fc0d7d2ec7fb28cb8a406a6e3eb0a3612931a1db33badc577624e9e566902ab3e3f4a104a1ef10c174b7629651c3aeb11522
7
- data.tar.gz: 66f0dba2a4a727d700ba8c4c7d386380c2d77380b2b8cd7f4943dae377bff1db751bd1ca7f5dcb2075a65139a453ccdf4395c2331874d1b2e783fcd7400be53f
6
+ metadata.gz: bebc7b8be31a4f9f0127bddf2cb4a8de23c9908b05ddf91f51a43af5927af4608a0398090cce27deefce0e5c86f4a7d9edb0ffdf6b122f849bb065cfcee44e2d
7
+ data.tar.gz: 1240bb5e7d513c4d3b3dd936042c3d177cbad6aebc402fc6ef91f86e3163bc0ba2dcf516f93d7658171d2c952c1f2fed6162928c7a50bdef3e5703233acce1ce
data/CHANGELOG.md CHANGED
@@ -7,7 +7,31 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
7
7
 
8
8
  ## [Unreleased]
9
9
 
10
+ ### Added
11
+ - **Full-feature language batch 3** (plan 100) - twelve languages gain the
12
+ downloadable spelling dictionary, a keyboard layout, and a gem module:
13
+ `ar` `id` `fa` `he` `bg` `sr` `hr` `sk` `sl` `lt` `lv` `et` (the RTL
14
+ ar/fa/he modules are promoted from tokenizer-only wiring; the other nine
15
+ are new thin modules - id hr sl sk et lt lv on `LatinBase`, bg sr with
16
+ the Cyrillic tokenizer). Six national key grids join the layout registry
17
+ (Arabic 101, Persian standard ISIRI 9147, Hebrew SI-1452, Bulgarian BDS,
18
+ Serbian Cyrillic, and the shared Croatian/Slovenian QWERTZ), all mirrored
19
+ from the models repo eval harness and drift-checked against it; five new
20
+ Latin family members (Indonesian Slovak Estonian Lithuanian Latvian)
21
+ type on the US QWERTY grid like the eval model. Bulgarian and Serbian
22
+ register ahead of JCUKEN so their Cyrillic grids win the lookup. The
23
+ eval-grid drift fixture now covers all nine national grids. Specimen
24
+ misspelling -> correction pairs for every language are engine-verified
25
+ against the staged dictionaries in a `:network` spec.
26
+
10
27
  ### Fixed
28
+ - **RTL word extraction on the check path** - the plan-91 script-aware
29
+ word regex never reached the Arabic, Persian and Hebrew tokenizers, so
30
+ `Kotoshu.check` / `kotoshu check -l ar|fa|he` extracted zero words and
31
+ silently passed every RTL document. The three tokenizers now declare
32
+ their script sets (`\p{Arabic}`, `\p{Arabic}` + ZWNJ for Persian
33
+ compounds like می‌روم, `\p{Hebrew}` + geresh/gershayim), and RTL
34
+ misspellings surface with ranked suggestions.
11
35
  - **Suggestion candidate sweep missed transpositions, substitutions, and
12
36
  dictionary forms** - `EditDistanceStrategy` only scored raw dictionary
13
37
  stems, so valid dictionary forms could never be suggested no matter how
@@ -188,6 +188,28 @@ convertible model pool exhausted at 55 languages; owner-side
188
188
  remnants (PyPI registrations, marketplace PAT, nds/fi calls) recorded
189
189
  in the plans, not pending work.
190
190
 
191
+ Plans 99-104 are **wave 6, the live-quality follow-through**
192
+ (2026-09-07), written after the sweep-performance arc (gem 0.9.1 ->
193
+ 0.9.2, @kotoshu/wasm 0.3.0 -> 0.3.2) and a fresh state audit: 55 model
194
+ languages x 3 tiers confirmed (ja/ko/zh rerank-only by design), no LID
195
+ resource mirrored, lsp/server `~> 0.6` constraints already resolving
196
+ current engines, site docs carrying stale version references.
197
+
198
+ | # | Plan | Gap it closes |
199
+ |---|---|---|
200
+ | 99 | [Site truth + performance docs](99-site-truth-performance-docs.md) | Docs misstate current versions; the measured per-language latency is unpublished |
201
+ | 100 | [Full-feature batch 3](100-full-feature-batch3.md) | ~35 languages have dictionaries + models but no gem module - the big-audience remainder (ar id fa he ...) |
202
+ | 101 | [int4 tier experiment](101-int4-tier-experiment.md) | Halving tier bytes was named (B1) but never run through the gates |
203
+ | 102 | [LID chain](102-lid-chain.md) | No detection in the registry, wasm, or playground - users pick the language blind |
204
+ | 103 | [OOV bucket tables](103-oov-bucket-tables.md) | Model candidates miss n-grams absent from vocab (the Teh gap's model half) |
205
+ | 104 | [VS Code marketplace readiness](104-vscode-marketplace-readiness.md) | The extension is built but not installable; everything short of the owner PAT |
206
+
207
+ Execution notes: plan 103 is sequenced after 101 (same repo, one
208
+ worktree at a time); all site-facing flips (99, 100, 102's playground
209
+ half, 104's link line) land through one coordinated site pass to keep
210
+ the repo conflict-free; lsp/server 0.1.2 bumps are deliberately NOT
211
+ planned - the `~> 0.6` floor already resolves 0.9.2 for fresh installs.
212
+
191
213
  Dependency order across the ecosystem track:
192
214
 
193
215
  ```
@@ -0,0 +1,28 @@
1
+ # Plan 100 — Full-feature language batch 3: the highest-audience remainder
2
+
3
+ ## Why
4
+ 20 languages are full-feature (gem module + keyboard layout + verified
5
+ specimens); the dictionaries manifest and the model registry carry ~35 more
6
+ with both resources. Every remaining language without a module leaves
7
+ speakers on the degraded path. The gaps are the big-audience ones: ar (~400M),
8
+ id (~200M), fa (~80M), he, bg, sr, hr, sk, sl, and more.
9
+
10
+ ## Work (gem repo only — the site flip rides a follow-up)
11
+ 1. Enumerate: languages present in BOTH the dictionaries manifest and the
12
+ models registry that lack a gem module. Rank by speaker count.
13
+ 2. For the top ~12 (expected: ar id fa he bg sr hr sk sl lt lv et — confirm
14
+ against data): create the language module (pattern of the 19 from PR
15
+ #123), keyboard layout (Arabic 101 for ar; Persian for fa; Hebrew for he;
16
+ Cyrillic/YUSI for bg/sr; QWERTZ/Slovak for the Latinate), AVAILABLE_LANGUAGES
17
+ extension, and specimen pairs verified against the STAGED dictionary via
18
+ the real engine (correct()/suggest() round trips) — the PR #123 discipline.
19
+ 3. RTL already wired (ar/id from batch 2); no tokenizer work needed.
20
+ 4. Suite green; rubocop clean.
21
+
22
+ ## Verification
23
+ One spec per module (specimen corrections engine-verified, layout smoke);
24
+ `kotoshu setup <lang>` + `check -l <lang>` exercised live for at least ar,
25
+ id, fa, he. Deliver the verified specimen table + sizes for the site flip.
26
+
27
+ ## Status
28
+ Executed 2026-09-07 — gem PR #151 merged. Twelve languages (ar id fa he bg sr hr sk sl lt lv et) full-feature; AVAILABLE_LANGUAGES 20→32; 49 specimens; RTL tokenizer fix for ar/fa/he. Site flip PR #27.
@@ -0,0 +1,32 @@
1
+ # Plan 101 — int4 tier experiment (models repo), eval-gated
2
+
3
+ ## Why
4
+ The tier recipe settled at int8 (rank_corr 0.9999, near-lossless) after SVD
5
+ failed its gates. int4 halves the bytes again — mini ~1.5 MB, fluency ~8 MB —
6
+ which changes browser adoption math for the wasm playground. It was named in
7
+ the original kotoshu-rs TODO (B1) but never run through the gates.
8
+
9
+ ## Work (models repo)
10
+ 1. Quantization pass: per-row int4 (packed nibbles) + fp16 scale, over the
11
+ same vocab cuts as fluency/mini.
12
+ 2. The existing eval gates unchanged and unweakened: rank correlation vs
13
+ full float32 on the keyboard-aware eval; accept only if rank_corr within
14
+ 0.001 of int8's and top-1 agreement holds on the probe-limited languages.
15
+ 3. If gates pass: registry v1.4.0 adding the int4 variants as NEW resources
16
+ (`{lang}/mini4`? or a field) — additive, never replacing; mirror with
17
+ ACAO:* like v1.2.1+. If gates fail: record the numbers as the verdict
18
+ (the SVD precedent — measured, not assumed).
19
+ 4. ONNX reader side (kotoshu-rs) only if the gates pass.
20
+
21
+ ## Verification
22
+ Gates table per language; byte sizes; the release registry byte-identical to
23
+ the committed one; no existing tier URLs change.
24
+
25
+ ## Status
26
+ Executed 2026-09-07 — REJECTED on data (models PR #17, merged). int4-per-row
27
+ (one fp16 scale, packed nibbles) over 8 languages: rank_corr deltas -0.0146
28
+ to -0.0199 vs int8 (15-20x outside the 0.001 window), top-1 fails on 8/8
29
+ fluency4 and 2/8 mini4. Sizes halve (mini4 1.52 MB, fluency4 7.60 MB) but
30
+ accuracy does not hold. With the earlier group-128/64/32 sweep the scale
31
+ ladder is measured end to end — int8-per-row stands final. Registry
32
+ untouched; evidence in models eval/reports/.
@@ -0,0 +1,28 @@
1
+ # Plan 102 — Language detection end to end: registry mirror, wasm, playground
2
+
3
+ ## Why
4
+ The gem detects document language (server /v1/detect) using its own detection
5
+ model — but the registry mirrors none of it, the wasm surface has no detect
6
+ API, and the playground makes a user pick a language dropdown blind. A
7
+ writer pasting text should be told what it is. The FastText LID model is
8
+ tiny (~1 MB) and the mirror pattern is proven (v1.2.1+).
9
+
10
+ ## Work
11
+ 1. Models repo: add the LID model as a registry resource (converted like the
12
+ tiers, int8-quantized if gates hold vs the float32 labels), mirrored with
13
+ ACAO:*; registry v1.4.x alongside plan 101 (coordinate the version).
14
+ 2. kotoshu-rs: `detect_language(bytes, vocab_bytes) -> {code, score}` on the
15
+ model surface beside loadModel/rerank; wasm export `detectLanguage`;
16
+ tests against the gem's detection outputs on a fixed multilingual corpus.
17
+ 3. Playground (rides the site follow-up with plans 99/100 flips): on load
18
+ and on demand ("detect" affordance beside the lang select), load the LID
19
+ model (~1 MB, cached like the semantic tier) and set/select the language,
20
+ showing the detected code + confidence. Never auto-switch a user's
21
+ explicit choice silently — propose it.
22
+
23
+ ## Verification
24
+ Registry mirror ACAO + size checks; wasm parity vs gem detect on the corpus;
25
+ playground verified with multilingual paste (en/de/ru/ja/ar).
26
+
27
+ ## Status
28
+ Executed 2026-09-07 (core) — models PR #18 (lid-176, registry v1.4.0) + kotoshu-rs PR #25 (loadLid/detectLanguage). Playground propose-UI still pending a wasm cut that ships the export.
@@ -0,0 +1,27 @@
1
+ # Plan 103 — OOV bucket-table artifacts: model-generated candidates for the last mile
2
+
3
+ ## Why
4
+ semanticSuggest embeds an OOV word via subword n-grams PRESENT IN THE VOCAB;
5
+ a typo whose n-grams never appear (Teh — only n-gram "teh") embeds nothing
6
+ and returns empty. fastText's real OOV path hashes n-grams into buckets the
7
+ training wrote to; exporting the bucket table closes that gap so the model,
8
+ not just the Damerau sweep, can reach the intended word. Recorded as the
9
+ honest gap in kotoshu-rs oov.rs docs and PR #18's report.
10
+
11
+ ## Work (models repo, then kotoshu-rs)
12
+ 1. Export the bucket-table rows the FastText training produced (hash bucket
13
+ → vector) for each tier alongside the vocab; size audit first — the
14
+ bucket table may exceed the tier itself; if so, gate to the top-K buckets
15
+ by usage and measure the quality delta honestly.
16
+ 2. kotoshu-rs: extend the int8 reader to consult bucket rows for n-grams
17
+ absent from vocab (feature-gated like `model`); semantic_neighbors then
18
+ covers bucket-backed OOV.
19
+ 3. Gates: existing rerank conformance must not move (frozen); new specs for
20
+ bucket-backed neighbors (Teh→the expected); eval on the rerank probe set.
21
+
22
+ ## Verification
23
+ Bucket-backed neighbor specs; size deltas recorded; rerank conformance 0
24
+ change; a 0.4.x wasm release only after both engines agree.
25
+
26
+ ## Status
27
+ Executed 2026-09-07 — models PR #20 (en/de buckets) + kotoshu-rs PR #26 (bucket-backed OOV). Teh→the via buckets. Release assets + wasm 0.4.0 pin still owner/coordinated cut. (sequenced AFTER plan 101 — same repo, one worktree at a time)
@@ -0,0 +1,32 @@
1
+ # Plan 104 — VS Code marketplace + Open VSX readiness
2
+
3
+ ## Why
4
+ The extension is built, CI-green, with a .vsix artifact — but it is not
5
+ installable without the marketplace listing, which is the single largest
6
+ adoption blocker for the editor audience. Publishing needs the owner's
7
+ publisher PAT (a credential we never hold); everything else can be ready
8
+ so publishing is one secret + one click.
9
+
10
+ ## Work (vscode repo)
11
+ 1. Open VSX publish workflow (release-ovsx.yml, secret-gated) beside the
12
+ existing CI — Open VSX serves VSCodium/Code-OSS users without the
13
+ Microsoft marketplace.
14
+ 2. Metadata polish: gallery banner/colors, keywords, categories, repository
15
+ links, the kotoshu-vscode README (quickstart with kotoshu-lsp install,
16
+ settings reference: log level, personal dictionary command).
17
+ 3. PREPUBLISH.md: exact owner steps — create publisher, add PAT as secret,
18
+ run the workflow; nothing else.
19
+ 4. Site follow-up line (rides plan 99's pass): /docs/clients/lsp gains the
20
+ marketplace link placeholder to fill on publish day.
21
+
22
+ ## Verification
23
+ `vsce package` clean with no warnings; workflow lint-valid; PREPUBLISH
24
+ steps verified against the actual GitHub UI labels.
25
+
26
+ ## Status
27
+ Executed 2026-09-07 — vscode PR #3, merged. release-marketplace.yml (VSCE_PAT)
28
+ + release-ovsx.yml (OPEN_VSX_TOKEN), both version-gated; PREPUBLISH.md with
29
+ verified owner steps; package.json publisher/categories/keywords/banner;
30
+ README quickstart against gem 0.1.1; vsce package warning-clean. Publishing
31
+ itself is one PAT secret + one tag — owner action. The /docs/clients/lsp
32
+ marketplace link rides the coordinated site pass.
@@ -0,0 +1,31 @@
1
+ # Plan 99 — Site truth pass + measured performance documentation
2
+
3
+ ## Why
4
+ The docs were written against the versions live at the time; three releases
5
+ later some pages misstate reality. Users deserve the measured numbers the
6
+ perf work produced — no spellchecker site publishes per-language sweep
7
+ latency; we have it, verified, for eight languages.
8
+
9
+ ## Work (site repo)
10
+ 1. `/docs/performance` (new page): the measured tables — whole-text check
11
+ latency (1-33 ms per 100 words), per-sweep averages post-0.3.2
12
+ (en 45 ms ... pt 423 ms, fr worst-case note), tier sizes (full/fluency/mini),
13
+ and tuning: personal dictionaries, baselines, tier choice guidance
14
+ (`--tier`), backend choice (ruby vs native vs wasm). Link from the
15
+ playground footnote, /languages, and /install.
16
+ 2. Truth pass: /docs/clients/action still says the action "installs the gem
17
+ from RubyGems where 0.7.0 gates six languages" — stale; the action now
18
+ rides 0.9.2 (20 full-feature languages, baselines, directory mode).
19
+ Audit every page for version references (0.7.0/0.8.0/0.1.1/wasm 0.2.0)
20
+ and flip to current-with-dates. /playground/server: verify the hosted
21
+ demo state and the run-your-own instructions.
22
+ 3. Footer/readout: confirm every registry badge resolves (RubyGems 0.9.2,
23
+ npm 0.3.2, PyPI, crates.io).
24
+
25
+ ## Verification
26
+ Rendered text extraction on the built site for stale version strings;
27
+ every measured number in /docs/performance traceable to the 0.3.2 bench
28
+ harness (kotoshu-rs PR #23 table) — no inferred numbers.
29
+
30
+ ## Status
31
+ Executed 2026-09-07 — site PR #26: /docs/performance + truth pass. Action gap found and closed via action-kotoshu PR #2 + site PR #27 docs flip.
@@ -31,8 +31,11 @@ module Kotoshu
31
31
  # modules (lib/kotoshu/languages/). Staged-only dictionaries in
32
32
  # the dictionaries repo stay outside this list until their
33
33
  # module lands; local files come via setup --aff/--dic instead.
34
+ # Batch 3 (plan 100) promotes the RTL wiring ar/fa/he and adds
35
+ # bg et hr id lt lv sk sl sr.
34
36
  AVAILABLE_LANGUAGES = %w[
35
- ca cs da de el en es fr hu it nb nl pl pt ro ru sv tr uk vi
37
+ ar bg ca cs da de el en es et fa fr he hr hu id it lt lv nb nl
38
+ pl pt ro ru sk sl sr sv tr uk vi
36
39
  ].freeze
37
40
 
38
41
  # Get or download spelling dictionary for a language.
@@ -0,0 +1,59 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Kotoshu
4
+ module Keyboard
5
+ module Layouts
6
+ # Arabic 101 keyboard layout (standard IBM PC Arabic layout).
7
+ #
8
+ # The Arabic letters sit on the QWERTY key faces: the number row
9
+ # is shared with ASCII, and ض ص ث ق ف غ ع ه خ ح ج د / ش س ي ب ل ا
10
+ # ت ن م ك ط / ئ ء ؤ ر لا ى ة و ز ظ fill the three letter rows.
11
+ # Diacritics (tashkeel) are dead keys and are not in the grid;
12
+ # the hamza carriers أ إ آ and the ه/ة ي/ى spelling slips are
13
+ # handled at the normalizer level, not as physical keys.
14
+ #
15
+ # The key grid is mirrored from the models repo eval harness
16
+ # (kotoshu/models-fasttext-onnx eval/noise.py _AR_101, plan 83
17
+ # batch 2 of the model coverage expansion, verified there
18
+ # against the kbdlayout.info Arabic 101 driver table) so
19
+ # typo-adjacency used for suggestion ranking matches the
20
+ # adjacency used to evaluate the models.
21
+ # spec/kotoshu/keyboard/layouts/eval_grid_drift_spec.rb enforces
22
+ # the sync.
23
+ #
24
+ # Languages: ar, ar-SA
25
+ class Arabic101 < Layout
26
+ # Key positions for the Arabic 101 layout.
27
+ # Each key maps to [row, column] coordinates.
28
+ KEY_POSITIONS = {
29
+ # Number row
30
+ 'ذ' => [0, 0], '1' => [0, 1], '2' => [0, 2], '3' => [0, 3],
31
+ '4' => [0, 4], '5' => [0, 5], '6' => [0, 6], '7' => [0, 7],
32
+ '8' => [0, 8], '9' => [0, 9], '0' => [0, 10], '-' => [0, 11],
33
+ '=' => [0, 12],
34
+ # Top row (ض ص ث ق ف غ ع ه خ ح ج د)
35
+ 'ض' => [1, 0], 'ص' => [1, 1], 'ث' => [1, 2], 'ق' => [1, 3],
36
+ 'ف' => [1, 4], 'غ' => [1, 5], 'ع' => [1, 6], 'ه' => [1, 7],
37
+ 'خ' => [1, 8], 'ح' => [1, 9], 'ج' => [1, 10], 'د' => [1, 11],
38
+ # Home row (ش س ي ب ل ا ت ن م ك ط)
39
+ 'ش' => [2, 0], 'س' => [2, 1], 'ي' => [2, 2], 'ب' => [2, 3],
40
+ 'ل' => [2, 4], 'ا' => [2, 5], 'ت' => [2, 6], 'ن' => [2, 7],
41
+ 'م' => [2, 8], 'ك' => [2, 9], 'ط' => [2, 10],
42
+ # Bottom row (ئ ء ؤ ر لا ى ة و ز ظ)
43
+ 'ئ' => [3, 0], 'ء' => [3, 1], 'ؤ' => [3, 2], 'ر' => [3, 3],
44
+ 'لا' => [3, 4], 'ى' => [3, 5], 'ة' => [3, 6], 'و' => [3, 7],
45
+ 'ز' => [3, 8], 'ظ' => [3, 9]
46
+ }.freeze
47
+
48
+ # Initialize the Arabic 101 layout.
49
+ def initialize
50
+ super(
51
+ name: 'Arabic-101',
52
+ language_codes: %w[ar ar-SA],
53
+ key_positions: KEY_POSITIONS
54
+ )
55
+ end
56
+ end
57
+ end
58
+ end
59
+ end
@@ -0,0 +1,57 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Kotoshu
4
+ module Keyboard
5
+ module Layouts
6
+ # Bulgarian BDS keyboard layout (БДС 5237:1978, KBDBUL.DLL).
7
+ #
8
+ # The national Bulgarian phonetic-adjacent ordering: ы у е и ш щ
9
+ # к с д з ц / ь я а о ж г т н в м ч / ю й ъ э ф х п р л б. The
10
+ # Bulgarian-only letters ъ ь are real keys (unlike JCUKEN, where
11
+ # ъ sits top-right and ы is the home-row opener).
12
+ #
13
+ # The key grid is mirrored from the models repo eval harness
14
+ # (kotoshu/models-fasttext-onnx eval/noise.py _BG_BDS, plan 83
15
+ # batch 2 of the model coverage expansion, verified there
16
+ # against the kbdlayout.info KBDBUL driver table) so
17
+ # typo-adjacency used for suggestion ranking matches the
18
+ # adjacency used to evaluate the models.
19
+ # spec/kotoshu/keyboard/layouts/eval_grid_drift_spec.rb enforces
20
+ # the sync.
21
+ #
22
+ # Languages: bg, bg-BG
23
+ class BulgarianBds < Layout
24
+ # Key positions for the Bulgarian BDS layout.
25
+ # Each key maps to [row, column] coordinates.
26
+ KEY_POSITIONS = {
27
+ # Number row
28
+ ')' => [0, 0], '1' => [0, 1], '2' => [0, 2], '3' => [0, 3],
29
+ '4' => [0, 4], '5' => [0, 5], '6' => [0, 6], '7' => [0, 7],
30
+ '8' => [0, 8], '9' => [0, 9], '0' => [0, 10], '$' => [0, 11],
31
+ '€' => [0, 12],
32
+ # Top row (ы у е и ш щ к с д з ц)
33
+ 'ы' => [1, 0], 'у' => [1, 1], 'е' => [1, 2], 'и' => [1, 3],
34
+ 'ш' => [1, 4], 'щ' => [1, 5], 'к' => [1, 6], 'с' => [1, 7],
35
+ 'д' => [1, 8], 'з' => [1, 9], 'ц' => [1, 10],
36
+ # Home row (ь я а о ж г т н в м ч)
37
+ 'ь' => [2, 0], 'я' => [2, 1], 'а' => [2, 2], 'о' => [2, 3],
38
+ 'ж' => [2, 4], 'г' => [2, 5], 'т' => [2, 6], 'н' => [2, 7],
39
+ 'в' => [2, 8], 'м' => [2, 9], 'ч' => [2, 10],
40
+ # Bottom row (ю й ъ э ф х п р л б)
41
+ 'ю' => [3, 0], 'й' => [3, 1], 'ъ' => [3, 2], 'э' => [3, 3],
42
+ 'ф' => [3, 4], 'х' => [3, 5], 'п' => [3, 6], 'р' => [3, 7],
43
+ 'л' => [3, 8], 'б' => [3, 9]
44
+ }.freeze
45
+
46
+ # Initialize the Bulgarian BDS layout.
47
+ def initialize
48
+ super(
49
+ name: 'Bulgarian-BDS',
50
+ language_codes: %w[bg bg-BG],
51
+ key_positions: KEY_POSITIONS
52
+ )
53
+ end
54
+ end
55
+ end
56
+ end
57
+ end
@@ -0,0 +1,57 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Kotoshu
4
+ module Keyboard
5
+ module Layouts
6
+ # Hebrew standard keyboard layout (SI-1452).
7
+ #
8
+ # The 27 Hebrew letters all sit on key faces in the standard
9
+ # ordering, including the five final-letter forms ם ן ך ף ץ as
10
+ # real keys. Niqqud (vowel points) are dead keys and are not in
11
+ # the grid; geresh/gershayim are shift states, not base keys.
12
+ #
13
+ # The key grid is mirrored from the models repo eval harness
14
+ # (kotoshu/models-fasttext-onnx eval/noise.py _HE_SI1452, plan 83
15
+ # batch 2 of the model coverage expansion, verified there
16
+ # against the Culmus SI-1452 layout table) so typo-adjacency
17
+ # used for suggestion ranking matches the adjacency used to
18
+ # evaluate the models.
19
+ # spec/kotoshu/keyboard/layouts/eval_grid_drift_spec.rb enforces
20
+ # the sync.
21
+ #
22
+ # Languages: he, he-IL
23
+ class HebrewSI1452 < Layout
24
+ # Key positions for the Hebrew SI-1452 layout.
25
+ # Each key maps to [row, column] coordinates.
26
+ KEY_POSITIONS = {
27
+ # Number row
28
+ '`' => [0, 0], '1' => [0, 1], '2' => [0, 2], '3' => [0, 3],
29
+ '4' => [0, 4], '5' => [0, 5], '6' => [0, 6], '7' => [0, 7],
30
+ '8' => [0, 8], '9' => [0, 9], '0' => [0, 10], '-' => [0, 11],
31
+ '=' => [0, 12],
32
+ # Top row (ק ר א ט ו ן ם פ)
33
+ '/' => [1, 0], "'" => [1, 1], 'ק' => [1, 2], 'ר' => [1, 3],
34
+ 'א' => [1, 4], 'ט' => [1, 5], 'ו' => [1, 6], 'ן' => [1, 7],
35
+ 'ם' => [1, 8], 'פ' => [1, 9], '[' => [1, 10], ']' => [1, 11],
36
+ # Home row (ש ד ג כ ע י ח ל ך ף)
37
+ 'ש' => [2, 0], 'ד' => [2, 1], 'ג' => [2, 2], 'כ' => [2, 3],
38
+ 'ע' => [2, 4], 'י' => [2, 5], 'ח' => [2, 6], 'ל' => [2, 7],
39
+ 'ך' => [2, 8], 'ף' => [2, 9], ',' => [2, 10],
40
+ # Bottom row (ז ס ב ה נ מ צ ת ץ)
41
+ 'ז' => [3, 0], 'ס' => [3, 1], 'ב' => [3, 2], 'ה' => [3, 3],
42
+ 'נ' => [3, 4], 'מ' => [3, 5], 'צ' => [3, 6], 'ת' => [3, 7],
43
+ 'ץ' => [3, 8], '.' => [3, 9]
44
+ }.freeze
45
+
46
+ # Initialize the Hebrew SI-1452 layout.
47
+ def initialize
48
+ super(
49
+ name: 'Hebrew-SI-1452',
50
+ language_codes: %w[he he-IL],
51
+ key_positions: KEY_POSITIONS
52
+ )
53
+ end
54
+ end
55
+ end
56
+ end
57
+ end
@@ -0,0 +1,78 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Kotoshu
4
+ module Keyboard
5
+ module Layouts
6
+ # Croatian and Slovenian QWERTZ keyboard layouts (KBDCRO /
7
+ # KBDYCL driver tables — the two national standards share one
8
+ # physical grid).
9
+ #
10
+ # The five South-Slavic Latin letters are real keys: š đ take
11
+ # the top-row bracket faces, č ć ž the home-row umlaut faces
12
+ # and the key right of them. Everything else is the QWERTZ
13
+ # physical arrangement (z/y swapped), so diacritic slips are not
14
+ # possible for these five — they are key slips.
15
+ #
16
+ # The shared key grid is mirrored from the models repo eval
17
+ # harness (kotoshu/models-fasttext-onnx eval/noise.py _HRSL,
18
+ # plan 83 batch 2 of the model coverage expansion, verified
19
+ # there against the kbdlayout.info driver tables) so
20
+ # typo-adjacency used for suggestion ranking matches the
21
+ # adjacency used to evaluate the models.
22
+ # spec/kotoshu/keyboard/layouts/eval_grid_drift_spec.rb enforces
23
+ # the sync.
24
+ module HrSlQwertz
25
+ # Key positions for the Croatian/Slovenian QWERTZ grid.
26
+ # Each key maps to [row, column] coordinates.
27
+ KEY_POSITIONS = {
28
+ # Number row
29
+ '‚' => [0, 0], '1' => [0, 1], '2' => [0, 2], '3' => [0, 3],
30
+ '4' => [0, 4], '5' => [0, 5], '6' => [0, 6], '7' => [0, 7],
31
+ '8' => [0, 8], '9' => [0, 9], '0' => [0, 10],
32
+ "'" => [0, 11], '+' => [0, 12],
33
+ # Top row (QWERTZ + š đ)
34
+ 'q' => [1, 0], 'w' => [1, 1], 'e' => [1, 2], 'r' => [1, 3],
35
+ 't' => [1, 4], 'z' => [1, 5], 'u' => [1, 6], 'i' => [1, 7],
36
+ 'o' => [1, 8], 'p' => [1, 9], 'š' => [1, 10], 'đ' => [1, 11],
37
+ # Home row (ASDFG + č ć ž)
38
+ 'a' => [2, 0], 's' => [2, 1], 'd' => [2, 2], 'f' => [2, 3],
39
+ 'g' => [2, 4], 'h' => [2, 5], 'j' => [2, 6], 'k' => [2, 7],
40
+ 'l' => [2, 8], 'č' => [2, 9], 'ć' => [2, 10], 'ž' => [2, 11],
41
+ # Bottom row (YXCVB)
42
+ 'y' => [3, 0], 'x' => [3, 1], 'c' => [3, 2], 'v' => [3, 3],
43
+ 'b' => [3, 4], 'n' => [3, 5], 'm' => [3, 6],
44
+ ',' => [3, 7], '.' => [3, 8], '-' => [3, 9]
45
+ }.freeze
46
+ end
47
+
48
+ # Croatian QWERTZ (KBDCRO / Croatian standard grid).
49
+ #
50
+ # Languages: hr, hr-HR
51
+ class Croatian < Layout
52
+ # Initialize the Croatian layout.
53
+ def initialize
54
+ super(
55
+ name: 'Croatian-QWERTZ',
56
+ language_codes: %w[hr hr-HR],
57
+ key_positions: HrSlQwertz::KEY_POSITIONS
58
+ )
59
+ end
60
+ end
61
+
62
+ # Slovenian QWERTZ (KBDYCL / Slovenian standard grid — the same
63
+ # physical arrangement as Croatian).
64
+ #
65
+ # Languages: sl, sl-SI
66
+ class Slovenian < Layout
67
+ # Initialize the Slovenian layout.
68
+ def initialize
69
+ super(
70
+ name: 'Slovenian-QWERTZ',
71
+ language_codes: %w[sl sl-SI],
72
+ key_positions: HrSlQwertz::KEY_POSITIONS
73
+ )
74
+ end
75
+ end
76
+ end
77
+ end
78
+ end
@@ -15,7 +15,8 @@ module Kotoshu
15
15
  # (kotoshu/models-fasttext-onnx eval/noise.py: LANG_LAYOUT maps
16
16
  # the Latin newcomers onto the qwerty/qwertz grids and handles
17
17
  # diacritics as _LANG_ALTERNATES slips), keeping gem suggestion
18
- # adjacency and eval adjacency in sync.
18
+ # adjacency and eval adjacency in sync. Batch 3 (plan 100) adds
19
+ # id sk et lt lv on the same qwerty base grid.
19
20
  #
20
21
  # One family, per-language declarations — no near-duplicate
21
22
  # layout files. The five pre-existing layouts (qwerty qwertz
@@ -214,6 +215,51 @@ module Kotoshu
214
215
  removed_keys %w([ ] ; ')
215
216
  extra_positions({ 'å' => [1, 10], 'ä' => [2, 9], 'ö' => [2, 10] })
216
217
  end
218
+
219
+ # Indonesian: US QWERTY physical layout — the Indonesian
220
+ # national standard types on the unmodified US grid, and the
221
+ # eval harness maps id the same way (plan 83 batch 2).
222
+ class Indonesian < Latin
223
+ layout_name 'Indonesian-QWERTY'
224
+ languages %w[id id-ID]
225
+ base_grid :qwerty
226
+ end
227
+
228
+ # Slovak: US QWERTY physical layout; the eval harness maps sk
229
+ # to the qwerty grid (plan 83 batch 2) with the háčik
230
+ # diacritics as dead-key slips, exactly the Polish
231
+ # programmers' pattern. The Slovak QWERTZ national variant
232
+ # exists, but Slovak QWERTY is the widely-typed form.
233
+ class Slovak < Latin
234
+ layout_name 'Slovak-QWERTY'
235
+ languages %w[sk sk-SK]
236
+ base_grid :qwerty
237
+ end
238
+
239
+ # Estonian: US QWERTY physical layout; ä ö õ ü š ž are dead-key
240
+ # or AltGr sequences on the eval model grid (plan 83 batch 2).
241
+ class Estonian < Latin
242
+ layout_name 'Estonian-QWERTY'
243
+ languages %w[et et-EE]
244
+ base_grid :qwerty
245
+ end
246
+
247
+ # Lithuanian: US QWERTY physical layout; ą č ę ė į š ų ū ž are
248
+ # dead-key or AltGr sequences (plan 83 batch 2 eval mapping).
249
+ class Lithuanian < Latin
250
+ layout_name 'Lithuanian-QWERTY'
251
+ languages %w[lt lt-LT]
252
+ base_grid :qwerty
253
+ end
254
+
255
+ # Latvian: US QWERTY physical layout; ā č ē ģ ī ķ ļ ņ š ū ž
256
+ # are dead-key or AltGr sequences (plan 83 batch 2 eval
257
+ # mapping).
258
+ class Latvian < Latin
259
+ layout_name 'Latvian-QWERTY'
260
+ languages %w[lv lv-LV]
261
+ base_grid :qwerty
262
+ end
217
263
  end
218
264
  end
219
265
  end
@@ -0,0 +1,58 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Kotoshu
4
+ module Keyboard
5
+ module Layouts
6
+ # Persian standard keyboard layout (ISIRI 9147).
7
+ #
8
+ # Shares the Arabic letter skeleton but carries the four Persian
9
+ # letters as real keys: پ چ ژ گ replace their Arabic equivalents
10
+ # at standard positions, and ک ی are the Persian keheh/yeh
11
+ # keycaps. The ZWNJ and diacritics are dead keys and stay out of
12
+ # the grid; the Arabic→Persian letter slips (ك→ک ي→ی) are
13
+ # handled at the normalizer level.
14
+ #
15
+ # The key grid is mirrored from the models repo eval harness
16
+ # (kotoshu/models-fasttext-onnx eval/noise.py _FA_STD, plan 83
17
+ # batch 2 of the model coverage expansion, verified there
18
+ # against ISIRI 9147 renderings) so typo-adjacency used for
19
+ # suggestion ranking matches the adjacency used to evaluate the
20
+ # models. spec/kotoshu/keyboard/layouts/eval_grid_drift_spec.rb
21
+ # enforces the sync.
22
+ #
23
+ # Languages: fa, fa-IR
24
+ class PersianStandard < Layout
25
+ # Key positions for the Persian standard layout.
26
+ # Each key maps to [row, column] coordinates.
27
+ KEY_POSITIONS = {
28
+ # Number row
29
+ '`' => [0, 0], '1' => [0, 1], '2' => [0, 2], '3' => [0, 3],
30
+ '4' => [0, 4], '5' => [0, 5], '6' => [0, 6], '7' => [0, 7],
31
+ '8' => [0, 8], '9' => [0, 9], '0' => [0, 10], '-' => [0, 11],
32
+ '=' => [0, 12],
33
+ # Top row (ض ص ث ق ف غ ع ه خ ح ج چ)
34
+ 'ض' => [1, 0], 'ص' => [1, 1], 'ث' => [1, 2], 'ق' => [1, 3],
35
+ 'ف' => [1, 4], 'غ' => [1, 5], 'ع' => [1, 6], 'ه' => [1, 7],
36
+ 'خ' => [1, 8], 'ح' => [1, 9], 'ج' => [1, 10], 'چ' => [1, 11],
37
+ # Home row (ش س ی ب ل ا ت ن م ک گ)
38
+ 'ش' => [2, 0], 'س' => [2, 1], 'ی' => [2, 2], 'ب' => [2, 3],
39
+ 'ل' => [2, 4], 'ا' => [2, 5], 'ت' => [2, 6], 'ن' => [2, 7],
40
+ 'م' => [2, 8], 'ک' => [2, 9], 'گ' => [2, 10],
41
+ # Bottom row (ظ ط ز ر ذ د پ و ۀ .)
42
+ 'ظ' => [3, 0], 'ط' => [3, 1], 'ز' => [3, 2], 'ر' => [3, 3],
43
+ 'ذ' => [3, 4], 'د' => [3, 5], 'پ' => [3, 6], 'و' => [3, 7],
44
+ 'ۀ' => [3, 8], '.' => [3, 9]
45
+ }.freeze
46
+
47
+ # Initialize the Persian standard layout.
48
+ def initialize
49
+ super(
50
+ name: 'Persian-Standard',
51
+ language_codes: %w[fa fa-IR],
52
+ key_positions: KEY_POSITIONS
53
+ )
54
+ end
55
+ end
56
+ end
57
+ end
58
+ end
@@ -0,0 +1,59 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Kotoshu
4
+ module Keyboard
5
+ module Layouts
6
+ # Serbian Cyrillic keyboard layout (ЉЊЕРТЗУИОПШ /
7
+ # АСДФГХЈКЛЧЋ / ЏЦВБНМ, KBDSR driver table).
8
+ #
9
+ # The Serbian-specific letters љ њ ђ ћ џ ј are real keys: љ њ
10
+ # take the q/w faces, ђ sits where QWERTY carries the right
11
+ # bracket, and џ ц в б н м fill the bottom row. Serbian Latin
12
+ # (sr-Latn) types on a QWERTZ physical grid but is not claimed
13
+ # by this layout — the gem wires Cyrillic as the primary Serbian
14
+ # script, matching the dictionaries repo dictionary.
15
+ #
16
+ # The key grid is mirrored from the models repo eval harness
17
+ # (kotoshu/models-fasttext-onnx eval/noise.py _SR_CYR, plan 83
18
+ # batch 2 of the model coverage expansion) so typo-adjacency
19
+ # used for suggestion ranking matches the adjacency used to
20
+ # evaluate the models.
21
+ # spec/kotoshu/keyboard/layouts/eval_grid_drift_spec.rb enforces
22
+ # the sync.
23
+ #
24
+ # Languages: sr, sr-RS
25
+ class SerbianCyrillic < Layout
26
+ # Key positions for the Serbian Cyrillic layout.
27
+ # Each key maps to [row, column] coordinates.
28
+ KEY_POSITIONS = {
29
+ # Number row
30
+ '`' => [0, 0], '1' => [0, 1], '2' => [0, 2], '3' => [0, 3],
31
+ '4' => [0, 4], '5' => [0, 5], '6' => [0, 6], '7' => [0, 7],
32
+ '8' => [0, 8], '9' => [0, 9], '0' => [0, 10],
33
+ "'" => [0, 11], '+' => [0, 12],
34
+ # Top row (љ њ е р т з у и о п ш ђ)
35
+ 'љ' => [1, 0], 'њ' => [1, 1], 'е' => [1, 2], 'р' => [1, 3],
36
+ 'т' => [1, 4], 'з' => [1, 5], 'у' => [1, 6], 'и' => [1, 7],
37
+ 'о' => [1, 8], 'п' => [1, 9], 'ш' => [1, 10], 'ђ' => [1, 11],
38
+ # Home row (а с д ф г х ј к л ч ћ ж)
39
+ 'а' => [2, 0], 'с' => [2, 1], 'д' => [2, 2], 'ф' => [2, 3],
40
+ 'г' => [2, 4], 'х' => [2, 5], 'ј' => [2, 6], 'к' => [2, 7],
41
+ 'л' => [2, 8], 'ч' => [2, 9], 'ћ' => [2, 10], 'ж' => [2, 11],
42
+ # Bottom row (џ ц в б н м , . -)
43
+ 'џ' => [3, 0], 'ц' => [3, 1], 'в' => [3, 2], 'б' => [3, 3],
44
+ 'н' => [3, 4], 'м' => [3, 5], ',' => [3, 6], '.' => [3, 7],
45
+ '-' => [3, 8]
46
+ }.freeze
47
+
48
+ # Initialize the Serbian Cyrillic layout.
49
+ def initialize
50
+ super(
51
+ name: 'Serbian-Cyrillic',
52
+ language_codes: %w[sr sr-RS],
53
+ key_positions: KEY_POSITIONS
54
+ )
55
+ end
56
+ end
57
+ end
58
+ end
59
+ end
@@ -132,11 +132,14 @@ module Kotoshu
132
132
  register(Layouts::QWERTY)
133
133
  register(Layouts::QWERTZ)
134
134
  register(Layouts::AZERTY)
135
- # Ukrainian registers before JCUKEN: the base JCUKEN layout
136
- # blanket-claims uk, but Ukrainian typists get the dedicated
137
- # grid with the ґ є і ї keys (plan 84). Lookup is
138
- # first-registered-wins, so order here is load-bearing for uk.
135
+ # Ukrainian, Bulgarian and Serbian register before JCUKEN: the
136
+ # base JCUKEN layout blanket-claims uk bg, but typists of those
137
+ # languages get their dedicated grids (plan 84 uk, plan 100
138
+ # bg/sr). Lookup is first-registered-wins, so order here is
139
+ # load-bearing for uk bg sr.
139
140
  register(Layouts::Ukrainian)
141
+ register(Layouts::BulgarianBds)
142
+ register(Layouts::SerbianCyrillic)
140
143
  register(Layouts::JCUKEN)
141
144
  register(Layouts::Dvorak)
142
145
  # Wave-1 national layouts (plan 84)
@@ -154,6 +157,19 @@ module Kotoshu
154
157
  register(Layouts::Latin::Danish)
155
158
  register(Layouts::Latin::Norwegian)
156
159
  register(Layouts::Latin::Swedish)
160
+ # Batch-3 national layouts (plan 100): ar fa he type on their
161
+ # national-script grids; hr and sl share the South-Slavic QWERTZ
162
+ # grid; the id sk et lt lv Latin members follow.
163
+ register(Layouts::Arabic101)
164
+ register(Layouts::PersianStandard)
165
+ register(Layouts::HebrewSI1452)
166
+ register(Layouts::Croatian)
167
+ register(Layouts::Slovenian)
168
+ register(Layouts::Latin::Indonesian)
169
+ register(Layouts::Latin::Slovak)
170
+ register(Layouts::Latin::Estonian)
171
+ register(Layouts::Latin::Lithuanian)
172
+ register(Layouts::Latin::Latvian)
157
173
  end
158
174
  end
159
175
  end
@@ -32,6 +32,15 @@ module Kotoshu
32
32
  # Parameterized Latin family (plan 84): one file, per-language
33
33
  # declarations over the qwerty/qwertz base grids.
34
34
  autoload :Latin, "kotoshu/keyboard/layouts/latin"
35
+ # Batch-3 national layouts (plan 100): grids mirrored from the
36
+ # models repo eval harness (plan 83 batch-2 grids).
37
+ autoload :Arabic101, "kotoshu/keyboard/layouts/arabic"
38
+ autoload :PersianStandard, "kotoshu/keyboard/layouts/persian"
39
+ autoload :HebrewSI1452, "kotoshu/keyboard/layouts/hebrew"
40
+ autoload :BulgarianBds, "kotoshu/keyboard/layouts/bulgarian"
41
+ autoload :SerbianCyrillic, "kotoshu/keyboard/layouts/serbian"
42
+ autoload :Croatian, "kotoshu/keyboard/layouts/hrsl"
43
+ autoload :Slovenian, "kotoshu/keyboard/layouts/hrsl"
35
44
  end
36
45
 
37
46
  class << self
@@ -14,6 +14,11 @@ module Kotoshu
14
14
  # lang = Kotoshu::Languages::Arabic.new
15
15
  # lang.script_type # => :arabic
16
16
  # lang.rtl? # => true
17
+ #
18
+ # Full-feature since plan 100 (batch 3): the spelling dictionary
19
+ # downloads from the dictionaries repo (AVAILABLE_LANGUAGES) and
20
+ # keyboard proximity uses the Arabic 101 grid
21
+ # (Keyboard::Layouts::Arabic101).
17
22
  class Arabic < Language::Base
18
23
  register "ar"
19
24
  register "ar-SA"
@@ -69,6 +74,15 @@ module Kotoshu
69
74
  .reject { |token| skip_token?(token) }
70
75
  end
71
76
 
77
+ # Spell-check word characters (plan 91 pattern, added by
78
+ # plan 100): Arabic-script letters. Arabic needs no ZWNJ (the
79
+ # staged dictionary carries none) and no apostrophe.
80
+ #
81
+ # @return [Regexp] Regex matching a single word character
82
+ def spellcheck_word_regex
83
+ /\p{Arabic}/
84
+ end
85
+
72
86
  protected
73
87
 
74
88
  def normalize(token)
@@ -0,0 +1,37 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Kotoshu
4
+ module Languages
5
+ # Bulgarian language implementation (plan 100, batch 3).
6
+ #
7
+ # Cyrillic composition like Ukrainian: the Cyrillic tokenizer
8
+ # keeps Bulgarian-script words, the base normalizer suffices —
9
+ # Bulgarian case folding has no special pairs. The Bulgarian-only
10
+ # letter ъ (and ь) are real keys on the Bulgarian BDS grid
11
+ # (Keyboard::Layouts::BulgarianBds), not the Russian JCUKEN the
12
+ # pre-existing Cyrillic layout models.
13
+ class Bulgarian < LatinBase
14
+ register "bg"
15
+ register "bg-BG"
16
+
17
+ def initialize(code: "bg", name: "Bulgarian")
18
+ super
19
+ end
20
+
21
+ # Cyrillic-script tokenizer.
22
+ #
23
+ # @return [Language::Tokenizer::CyrillicTokenizer]
24
+ def tokenizer
25
+ @tokenizer ||= Language::Tokenizer::CyrillicTokenizer.new
26
+ end
27
+
28
+ def script_type
29
+ :cyrillic
30
+ end
31
+
32
+ def default_dictionary_paths
33
+ ["/usr/share/dict/bulgarian"]
34
+ end
35
+ end
36
+ end
37
+ end
@@ -0,0 +1,24 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Kotoshu
4
+ module Languages
5
+ # Estonian language implementation (plan 100, batch 3).
6
+ #
7
+ # Thin composition: shared Latin tokenizer, base normalizer,
8
+ # registry entry. The Estonian letters ä ö õ ü š ž are dead-key /
9
+ # AltGr sequences over the US QWERTY grid
10
+ # (Keyboard::Layouts::Latin::Estonian), like the eval model.
11
+ class Estonian < LatinBase
12
+ register "et"
13
+ register "et-EE"
14
+
15
+ def initialize(code: "et", name: "Estonian")
16
+ super
17
+ end
18
+
19
+ def default_dictionary_paths
20
+ ["/usr/share/dict/estonian"]
21
+ end
22
+ end
23
+ end
24
+ end
@@ -13,6 +13,11 @@ module Kotoshu
13
13
  # lang = Kotoshu::Languages::Persian.new
14
14
  # lang.script_type # => :arabic
15
15
  # lang.rtl? # => true
16
+ #
17
+ # Full-feature since plan 100 (batch 3): the spelling dictionary
18
+ # downloads from the dictionaries repo (AVAILABLE_LANGUAGES) and
19
+ # keyboard proximity uses the Persian standard grid
20
+ # (Keyboard::Layouts::PersianStandard).
16
21
  class Persian < Language::Base
17
22
  register "fa"
18
23
  register "fa-IR"
@@ -60,6 +65,17 @@ module Kotoshu
60
65
  .reject { |token| skip_token?(token) }
61
66
  end
62
67
 
68
+ # Spell-check word characters (plan 91 pattern, added by
69
+ # plan 100): Arabic-script letters plus ZWNJ (U+200C) —
70
+ # Persian compounds like می‌روم keep their nimb-fāsele inside
71
+ # the word, and the staged dictionary carries tens of
72
+ # thousands of ZWNJ forms.
73
+ #
74
+ # @return [Regexp] Regex matching a single word character
75
+ def spellcheck_word_regex
76
+ /[\p{Arabic}\u200C]/
77
+ end
78
+
63
79
  protected
64
80
 
65
81
  def normalize(token)
@@ -14,6 +14,11 @@ module Kotoshu
14
14
  # lang = Kotoshu::Languages::Hebrew.new
15
15
  # lang.script_type # => :hebrew
16
16
  # lang.rtl? # => true
17
+ #
18
+ # Full-feature since plan 100 (batch 3): the spelling dictionary
19
+ # downloads from the dictionaries repo (AVAILABLE_LANGUAGES) and
20
+ # keyboard proximity uses the Hebrew SI-1452 grid
21
+ # (Keyboard::Layouts::HebrewSI1452).
17
22
  class Hebrew < Language::Base
18
23
  register "he"
19
24
  register "he-IL"
@@ -68,6 +73,16 @@ module Kotoshu
68
73
  .reject { |token| skip_token?(token) }
69
74
  end
70
75
 
76
+ # Spell-check word characters (plan 91 pattern, added by
77
+ # plan 100): Hebrew letters plus the geresh/gershayim used
78
+ # inside acronyms. Niqqud are combining marks, not letters —
79
+ # the Hebrew normalizer strips them from input before lookup.
80
+ #
81
+ # @return [Regexp] Regex matching a single word character
82
+ def spellcheck_word_regex
83
+ /[\p{Hebrew}\u05F3\u05F4]/
84
+ end
85
+
71
86
  protected
72
87
 
73
88
  def normalize(token)
@@ -0,0 +1,24 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Kotoshu
4
+ module Languages
5
+ # Croatian language implementation (plan 100, batch 3).
6
+ #
7
+ # Thin composition: shared Latin tokenizer, base normalizer,
8
+ # registry entry. The five Croatian digraph letters š đ č ć ž are
9
+ # real keys on the South-Slavic QWERTZ physical layout
10
+ # (Keyboard::Layouts::Croatian), shared with Slovenian.
11
+ class Croatian < LatinBase
12
+ register "hr"
13
+ register "hr-HR"
14
+
15
+ def initialize(code: "hr", name: "Croatian")
16
+ super
17
+ end
18
+
19
+ def default_dictionary_paths
20
+ ["/usr/share/dict/croatian"]
21
+ end
22
+ end
23
+ end
24
+ end
@@ -0,0 +1,25 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Kotoshu
4
+ module Languages
5
+ # Indonesian language implementation (plan 100, batch 3).
6
+ #
7
+ # Thin composition like the wave-1 members: shared Latin
8
+ # tokenizer, base normalizer, registry entry. Indonesian is typed
9
+ # on the unmodified US QWERTY grid (Keyboard::Layouts::Latin::Indonesian)
10
+ # — the language has no diacritics, so the Latin base covers it
11
+ # without any per-language care.
12
+ class Indonesian < LatinBase
13
+ register "id"
14
+ register "id-ID"
15
+
16
+ def initialize(code: "id", name: "Indonesian")
17
+ super
18
+ end
19
+
20
+ def default_dictionary_paths
21
+ ["/usr/share/dict/indonesian"]
22
+ end
23
+ end
24
+ end
25
+ end
@@ -0,0 +1,24 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Kotoshu
4
+ module Languages
5
+ # Lithuanian language implementation (plan 100, batch 3).
6
+ #
7
+ # Thin composition: shared Latin tokenizer, base normalizer,
8
+ # registry entry. The Lithuanian letters ą č ę ė į š ų ū ž are
9
+ # dead-key / AltGr sequences over the US QWERTY grid
10
+ # (Keyboard::Layouts::Latin::Lithuanian), like the eval model.
11
+ class Lithuanian < LatinBase
12
+ register "lt"
13
+ register "lt-LT"
14
+
15
+ def initialize(code: "lt", name: "Lithuanian")
16
+ super
17
+ end
18
+
19
+ def default_dictionary_paths
20
+ ["/usr/share/dict/lithuanian"]
21
+ end
22
+ end
23
+ end
24
+ end
@@ -0,0 +1,24 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Kotoshu
4
+ module Languages
5
+ # Latvian language implementation (plan 100, batch 3).
6
+ #
7
+ # Thin composition: shared Latin tokenizer, base normalizer,
8
+ # registry entry. The Latvian letters ā č ē ģ ī ķ ļ ņ š ū ž are
9
+ # dead-key / AltGr sequences over the US QWERTY grid
10
+ # (Keyboard::Layouts::Latin::Latvian), like the eval model.
11
+ class Latvian < LatinBase
12
+ register "lv"
13
+ register "lv-LV"
14
+
15
+ def initialize(code: "lv", name: "Latvian")
16
+ super
17
+ end
18
+
19
+ def default_dictionary_paths
20
+ ["/usr/share/dict/latvian"]
21
+ end
22
+ end
23
+ end
24
+ end
@@ -0,0 +1,25 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Kotoshu
4
+ module Languages
5
+ # Slovak language implementation (plan 100, batch 3).
6
+ #
7
+ # Thin composition: shared Latin tokenizer, base normalizer,
8
+ # registry entry. Slovak diacritics (á ä č ď é í ľ ĺ ň ó ô ŕ š ť
9
+ # ú ý ž) are dead keys over the US QWERTY grid
10
+ # (Keyboard::Layouts::Latin::Slovak), the Polish programmers'
11
+ # pattern the eval harness also models.
12
+ class Slovak < LatinBase
13
+ register "sk"
14
+ register "sk-SK"
15
+
16
+ def initialize(code: "sk", name: "Slovak")
17
+ super
18
+ end
19
+
20
+ def default_dictionary_paths
21
+ ["/usr/share/dict/slovak"]
22
+ end
23
+ end
24
+ end
25
+ end
@@ -0,0 +1,25 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Kotoshu
4
+ module Languages
5
+ # Slovenian language implementation (plan 100, batch 3).
6
+ #
7
+ # Thin composition: shared Latin tokenizer, base normalizer,
8
+ # registry entry. The Slovenian letters č š ž (plus the Croatian
9
+ # đ ć the grid also carries) are real keys on the South-Slavic
10
+ # QWERTZ physical layout (Keyboard::Layouts::Slovenian), shared
11
+ # with Croatian.
12
+ class Slovenian < LatinBase
13
+ register "sl"
14
+ register "sl-SI"
15
+
16
+ def initialize(code: "sl", name: "Slovenian")
17
+ super
18
+ end
19
+
20
+ def default_dictionary_paths
21
+ ["/usr/share/dict/slovenian"]
22
+ end
23
+ end
24
+ end
25
+ end
@@ -0,0 +1,38 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Kotoshu
4
+ module Languages
5
+ # Serbian language implementation (plan 100, batch 3).
6
+ #
7
+ # Cyrillic composition like Bulgarian/Ukrainian: Serbian is
8
+ # written in both scripts, but Cyrillic is the constitutional
9
+ # primary and the script the staged dictionary carries — the
10
+ # module registers sr for the Cyrillic grid
11
+ # (Keyboard::Layouts::SerbianCyrillic); Serbian Latin (sr-Latn)
12
+ # stays unwired like nn until its own module lands. The Serbian
13
+ # letters љ њ ђ ћ џ ј are real keys on that grid.
14
+ class Serbian < LatinBase
15
+ register "sr"
16
+ register "sr-RS"
17
+
18
+ def initialize(code: "sr", name: "Serbian")
19
+ super
20
+ end
21
+
22
+ # Cyrillic-script tokenizer.
23
+ #
24
+ # @return [Language::Tokenizer::CyrillicTokenizer]
25
+ def tokenizer
26
+ @tokenizer ||= Language::Tokenizer::CyrillicTokenizer.new
27
+ end
28
+
29
+ def script_type
30
+ :cyrillic
31
+ end
32
+
33
+ def default_dictionary_paths
34
+ ["/usr/share/dict/serbian"]
35
+ end
36
+ end
37
+ end
38
+ end
@@ -36,5 +36,17 @@ module Kotoshu
36
36
  autoload :Turkish, "kotoshu/languages/tr/language"
37
37
  autoload :Ukrainian, "kotoshu/languages/uk/language"
38
38
  autoload :Vietnamese, "kotoshu/languages/vi/language"
39
+ # Batch-3 modules (plan 100): id hr sl sk et lt lv (Latin) and
40
+ # bg sr (Cyrillic). The pre-existing ar fa he modules become
41
+ # full-feature in this batch (layout + downloadable dictionary).
42
+ autoload :Bulgarian, "kotoshu/languages/bg/language"
43
+ autoload :Croatian, "kotoshu/languages/hr/language"
44
+ autoload :Estonian, "kotoshu/languages/et/language"
45
+ autoload :Indonesian, "kotoshu/languages/id/language"
46
+ autoload :Latvian, "kotoshu/languages/lv/language"
47
+ autoload :Lithuanian, "kotoshu/languages/lt/language"
48
+ autoload :Serbian, "kotoshu/languages/sr/language"
49
+ autoload :Slovak, "kotoshu/languages/sk/language"
50
+ autoload :Slovenian, "kotoshu/languages/sl/language"
39
51
  end
40
52
  end
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Kotoshu
4
- VERSION = '0.9.2'
4
+ VERSION = '0.9.3'
5
5
  end
@@ -16,7 +16,10 @@ abort "noise.py not found: #{source}" unless File.exist?(source)
16
16
 
17
17
  src = File.read(source)
18
18
  grids = {}
19
- %w[_TR_Q _UK_JCUKEN _EL_PHONETIC].each do |name|
19
+ %w[
20
+ _TR_Q _UK_JCUKEN _EL_PHONETIC _AR_101 _FA_STD _HE_SI1452 _BG_BDS
21
+ _SR_CYR _HRSL
22
+ ].each do |name|
20
23
  body = src[/^#{name} = \{(.*?)^\}/m, 1]
21
24
  abort "grid #{name} not found in #{source}" unless body
22
25
  grid = {}
data/sig/kotoshu.rbs CHANGED
@@ -152,6 +152,30 @@ def spellcheck_word_regex: () -> Regexp
152
152
  end
153
153
  class Vietnamese < LatinBase
154
154
  end
155
+ # Batch-3 modules (plan 100): id hr sl sk et lt lv (Latin) and
156
+ # bg sr (Cyrillic).
157
+ class Bulgarian < LatinBase
158
+ def tokenizer: () -> Language::Tokenizer::CyrillicTokenizer
159
+ def script_type: () -> :cyrillic
160
+ end
161
+ class Croatian < LatinBase
162
+ end
163
+ class Estonian < LatinBase
164
+ end
165
+ class Indonesian < LatinBase
166
+ end
167
+ class Latvian < LatinBase
168
+ end
169
+ class Lithuanian < LatinBase
170
+ end
171
+ class Serbian < LatinBase
172
+ def tokenizer: () -> Language::Tokenizer::CyrillicTokenizer
173
+ def script_type: () -> :cyrillic
174
+ end
175
+ class Slovak < LatinBase
176
+ end
177
+ class Slovenian < LatinBase
178
+ end
155
179
  end
156
180
 
157
181
  module Cli
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: kotoshu
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.9.2
4
+ version: 0.9.3
5
5
  platform: ruby
6
6
  authors:
7
7
  - Ribose Inc.
@@ -103,6 +103,11 @@ files:
103
103
  - TODO.impl/08-grammar-engine.md
104
104
  - TODO.impl/09-integrity-security.md
105
105
  - TODO.impl/10-testing-ci.md
106
+ - TODO.impl/100-full-feature-batch3.md
107
+ - TODO.impl/101-int4-tier-experiment.md
108
+ - TODO.impl/102-lid-chain.md
109
+ - TODO.impl/103-oov-bucket-tables.md
110
+ - TODO.impl/104-vscode-marketplace-readiness.md
106
111
  - TODO.impl/11-release-v1.md
107
112
  - TODO.impl/20-encapsulation-cleanup.md
108
113
  - TODO.impl/21-autoload-migration.md
@@ -174,6 +179,7 @@ files:
174
179
  - TODO.impl/96-server-semantic-models.md
175
180
  - TODO.impl/97-yard-execution.md
176
181
  - TODO.impl/98-campaign-cleanup.md
182
+ - TODO.impl/99-site-truth-performance-docs.md
177
183
  - conformance/vectors.jsonl
178
184
  - data/en/grammar/rules.yaml
179
185
  - docs/cache-architecture.md
@@ -330,13 +336,19 @@ files:
330
336
  - lib/kotoshu/jekyll.rb
331
337
  - lib/kotoshu/keyboard.rb
332
338
  - lib/kotoshu/keyboard/layout.rb
339
+ - lib/kotoshu/keyboard/layouts/arabic.rb
333
340
  - lib/kotoshu/keyboard/layouts/azerty.rb
341
+ - lib/kotoshu/keyboard/layouts/bulgarian.rb
334
342
  - lib/kotoshu/keyboard/layouts/dvorak.rb
335
343
  - lib/kotoshu/keyboard/layouts/greek.rb
344
+ - lib/kotoshu/keyboard/layouts/hebrew.rb
345
+ - lib/kotoshu/keyboard/layouts/hrsl.rb
336
346
  - lib/kotoshu/keyboard/layouts/jcuken.rb
337
347
  - lib/kotoshu/keyboard/layouts/latin.rb
348
+ - lib/kotoshu/keyboard/layouts/persian.rb
338
349
  - lib/kotoshu/keyboard/layouts/qwerty.rb
339
350
  - lib/kotoshu/keyboard/layouts/qwertz.rb
351
+ - lib/kotoshu/keyboard/layouts/serbian.rb
340
352
  - lib/kotoshu/keyboard/layouts/turkish_q.rb
341
353
  - lib/kotoshu/keyboard/layouts/ukrainian.rb
342
354
  - lib/kotoshu/keyboard/registry.rb
@@ -379,6 +391,7 @@ files:
379
391
  - lib/kotoshu/language/tokenizer/spanish_tokenizer.rb
380
392
  - lib/kotoshu/languages.rb
381
393
  - lib/kotoshu/languages/ar/language.rb
394
+ - lib/kotoshu/languages/bg/language.rb
382
395
  - lib/kotoshu/languages/ca/language.rb
383
396
  - lib/kotoshu/languages/cs/language.rb
384
397
  - lib/kotoshu/languages/da/language.rb
@@ -386,19 +399,27 @@ files:
386
399
  - lib/kotoshu/languages/el/language.rb
387
400
  - lib/kotoshu/languages/en/language.rb
388
401
  - lib/kotoshu/languages/es/language.rb
402
+ - lib/kotoshu/languages/et/language.rb
389
403
  - lib/kotoshu/languages/fa/language.rb
390
404
  - lib/kotoshu/languages/fr/language.rb
391
405
  - lib/kotoshu/languages/he/language.rb
406
+ - lib/kotoshu/languages/hr/language.rb
392
407
  - lib/kotoshu/languages/hu/language.rb
408
+ - lib/kotoshu/languages/id/language.rb
393
409
  - lib/kotoshu/languages/it/language.rb
394
410
  - lib/kotoshu/languages/ja/language.rb
395
411
  - lib/kotoshu/languages/latin_base.rb
412
+ - lib/kotoshu/languages/lt/language.rb
413
+ - lib/kotoshu/languages/lv/language.rb
396
414
  - lib/kotoshu/languages/nb/language.rb
397
415
  - lib/kotoshu/languages/nl/language.rb
398
416
  - lib/kotoshu/languages/pl/language.rb
399
417
  - lib/kotoshu/languages/pt/language.rb
400
418
  - lib/kotoshu/languages/ro/language.rb
401
419
  - lib/kotoshu/languages/ru/language.rb
420
+ - lib/kotoshu/languages/sk/language.rb
421
+ - lib/kotoshu/languages/sl/language.rb
422
+ - lib/kotoshu/languages/sr/language.rb
402
423
  - lib/kotoshu/languages/sv/language.rb
403
424
  - lib/kotoshu/languages/tr/language.rb
404
425
  - lib/kotoshu/languages/uk/language.rb