indickit 0.12.0 → 0.13.0

package/CHANGELOG.md

@@ -8,6 +8,75 @@ version, and compare `RULES_VERSION` (or `SEARCH_VERSION`) to know which
 rows. A version whose rules line equals the one before it gives every
 input the same output.
 
+## 0.13.0 (2026-10-13)
+
+Rules: phonetic 2026-10-12, phonetic-search 2026-10-12, normalize 2026-10-12.1, segment 2026-10-11, stem 2026-10-12, romanize 2026-10-13, deromanize 2026-10-12.1
+
+- Changed output: phonetic 2026-10-11.8 → 2026-10-12, phonetic-search
+  follows (thresholds unchanged). Recompute stored keys. Six folds from the
+  key's misses on village names: a Latin final -on is also -o (Pimpalgaon =
+  पिंपळगाव), Bk and Kh are also budruk and khurd, a Latin rh is also r
+  (Aligarh = अलीगढ, a known miss before), Bengali-script ওয় is also ov
+  (Dewan = দেওয়ান), a Latin initial ye is also e (Yekodi = एकोडी), Tamil ா்
+  is also ர் (நாவலூா் = Navallur). The native forms of the titles Sardar,
+  Maulana, Janab and Begum are dropped like the Latin ones:
+  match("Sardar Vallabhbhai Patel", "સરદાર વલ્લભભાઈ પટેલ") is now true.
+  On OpenStreetMap places held out to this read (1,506 places), mean recall
+  rises 0.34 points (95%: +0.14 to +0.72), Marathi 2.4, Hindi 2.0, wrong
+  merges +1.2%; on the development half of LGD villages 79.4% → 83.2%.
+  Every key of 0.12.0 is kept; 1,334 of the 199,992 test words gain one.
+  rules.json gains two fields both ports read: `abbreviations` and
+  `second_readings`. The browser file grows to 9 KB gzipped.
+- New option, default output unchanged: romanize rules 2026-10-12 →
+  2026-10-13 adds a route to Hindi's names mode. Every call as before gives
+  the same output as 2026-10-12. With the route, a word in the list of Hindi
+  names the tables were trained on (7,604 words,
+  `romanize/lang/hi/hi.known.bin`, 26 KB brotli) keeps its names list, and
+  any other word gets the words mode's list. On names in news held out (PIB,
+  August 2023) Hindi's first spelling goes from 55.0% to 62.2% (+7.2
+  points, 95% interval +4.1 to +10.0; IndicXlit 59.1%). Only Hindi has a
+  route: in the other languages the words tables spell Wikidata's unseen
+  names worse.
+- API: TypeScript `load("hi", "names", { route: true })` turns the route on;
+  it also fetches the Hindi words tables and the list (26 KB more on a page
+  that has the words tables, 2.5 MB in all otherwise). `fromBytes` takes an
+  optional sixth argument, `{ known, other }` (the list's bytes and the Hindi
+  words Romanizer), and routes only when it is given. Go:
+  `romanize.WordWith` and `romanize.TextWith` take `romanize.Options{Route:
+  true}` and return an error when the list or the words tables are not
+  imported (`lang/hi` imports both). `Word`, `Text` and `Languages` are
+  unchanged. The browser file grows to 9 KB gzipped.
+- Changed output: deromanize rules 2026-10-09.2 → 2026-10-12.1. Recompute stored
+  output. Sanskrit's words model learns from all of Aksharantar's Sanskrit pairs (+2.8
+  points on development words, +1.4 on running text). A spelling a renderer would draw
+  broken (a dotted circle) or with punctuation inside (अभवत्।, ري،) goes after every
+  whole spelling, in both modes. In words mode, a spelling that writes nothing for a
+  consonant is no longer ranked first in Kashmiri, Manipuri, Sanskrit and Sindhi. The
+  word lists are keyed with normalize 2026-10-12.1 (217 keys in as bn kn ml or ta, each a
+  two-part vowel sign typed in the other order, merge into the frequent word's key). Of the
+  conformance file's 3,028 shared inputs, 297 change, 33 at the first answer (sa 17,
+  sd 5, ks 3, ur 2, as 2, bn brx ne ta 1). Held out, read once: Dakshina's typed
+  sentences (11 languages) 81.1% first against IndicXlit's 77.1%; Dakshina's test
+  lexicons 69.4% against 72.6%. The names key changed: a typed name now accepts every
+  attested spelling (development names 69.2% → 74.5%; IndicXlit 66.2% → 71.4%).
+  deromanize output is always normalized text: a spelling that normalize rewrites is
+  returned in its normalized form, and a duplicate that results is dropped (74 of 3,114
+  conformance inputs change against the unreleased 2026-10-12, 6 at the first answer, 5
+  of them in bytes only).
+  The browser file grows from 28 KB to 34 KB gzipped: it now holds `wellformed`.
+- Changed output: normalize rules 2026-10-12 → 2026-10-12.1. Recompute stored
+  text. One rule added, nothing else changes: a two-part vowel sign typed as
+  its parts in the other order (Odia କଠାେର, ା then େ, drawn as କଠୋର) becomes
+  the sign, after a consonant or a nukta, in the 16 orders of Bengali, Odia,
+  Tamil, Kannada and Malayalam that every font we hold draws the same (four
+  Kannada orders stay). On the other half of the Odia web text (IndicCorp v2,
+  read once): look-alike groups made equal 80.5% → 81.2%; of 215 groups holding
+  such a spelling, 1 → 172 joined; no word changed its look by this rule.
+  `fold` is unchanged; no search key of the conformance sample moves.
+  rules.json gains `order`, which both ports read. The browser file is 22.5 KB
+  gzipped (README budget 23 KB). deromanize's word lists are keyed by
+  normalize and are re-exported against this version.
+
 ## 0.12.0 (2026-10-12)
 
 Rules: phonetic 2026-10-11.8, normalize 2026-10-12, segment 2026-10-11, stem 2026-10-12, romanize 2026-10-12, deromanize 2026-10-09.2, phonetic-search 2026-10-11.8

package/README.md

@@ -99,14 +99,14 @@ TypeScript, set `moduleResolution` to `nodenext` or `bundler`; the old
 In a browser, with no build step:
 
 ```js
-import { normalize } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.12.0/dist/normalize.js";
-import { stem } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.12.0/dist/stem.js";
-import { match } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.12.0/dist/phonetic.js";
-import { segment } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.12.0/dist/segment.js";
-import { load } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.12.0/dist/romanize.js";
-import { loadSearch } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.12.0/dist/phonetic-search.js";
-import { load as loadDeromanize } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.12.0/dist/deromanize.js";
-import { isWellFormed } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.12.0/dist/wellformed.js";
+import { normalize } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.13.0/dist/normalize.js";
+import { stem } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.13.0/dist/stem.js";
+import { match } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.13.0/dist/phonetic.js";
+import { segment } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.13.0/dist/segment.js";
+import { load } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.13.0/dist/romanize.js";
+import { loadSearch } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.13.0/dist/phonetic-search.js";
+import { load as loadDeromanize } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.13.0/dist/deromanize.js";
+import { isWellFormed } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.13.0/dist/wellformed.js";
 ```
 
 Each utility is its own file, so a page loads only the rules it uses.
@@ -383,12 +383,14 @@ key("bahut") === key("bohot");     // true: both are बहुत
 key("kahaan") === key("kahan");    // false: कहाँ and कहा
 ```
 
-On words `deromanize` never trained on (Dakshina's lexicons, Bengali,
-Hindi, Marathi, Tamil, Telugu and Urdu), this key matches 30–67% of the
-pairs of spellings of one word and gives 3–20% of spellings the key of
-another word. Fuzzy matching (one edit) matches 28–65% and gives 28–51% a
-wrong match. Keys from all four spellings match 52–88%, at 16–46% wrong.
-Tamil is the weakest language. The `phonetic` key is not for this: it drops
+On words `deromanize` never trained on (Dakshina's test lexicons, read
+once, 11 languages: Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi,
+Punjabi, Sindhi, Tamil, Telugu and Urdu), this key matches 40–69% of the
+pairs of spellings of one word and gives 0–13% of spellings the key of
+another word. Fuzzy matching (one edit) matches 33–57% and gives 10–39% a
+wrong match. Keys from all four spellings match 54–84%, at 4–33% wrong.
+Malayalam is the weakest language: level with fuzzy matching, at a tenth of
+its wrong matches. The `phonetic` key is not for this: it drops
 vowels, which carry a Latin-typed word.
 
 ### The same in Go
@@ -458,12 +460,12 @@ other tools tried and each weak spot are in `docs/`.
 
 | Utility | Result | Best other tool | Browser file | Details |
 |---|---|---|---|---|
-| `normalize` | makes 88.1% of look-alike spellings equal, and changes the look of no word | Indic NLP Library: 82.4%, and changes 0.68% of words | 22 KB | [docs/normalize.md](docs/normalize.md) |
+| `normalize` | makes 88.1% of look-alike spellings equal, and changes the look of no word | Indic NLP Library: 82.4%, and changes 0.68% of words | 23 KB | [docs/normalize.md](docs/normalize.md) |
 | `stem` | finds 3.7–15 points more of the right sentences than exact search, in 22 languages | ahead in 5 languages, level in 2, a trade in 4, behind in Nepali; no other stemmer for the other 10 | 4 KB | [docs/stem.md](docs/stem.md) |
-| `phonetic` | finds 89% of place-name pairs across scripts (OpenStreetMap, held out, rules 2026-10-11.7; 91% on a second slice, rules 2026-10-11.8); 84% of people on clean names in 21 languages (rules 2026-10-05, a lower bound for the rules that ship); 88% of a second roster as written (titles, initials; rules 2026-10-11); with `phonetic-search`, 93% of places (search 2026-10-11.7; 92% on the second slice, search 2026-10-11.8) and 86% of a roster as written (search 2026-10-09) | romanize + Soundex: 71% of place pairs, 23% of people in Tamil; cannot read Urdu; ICU + fuzzy match: 74% of places | 8 KB | [docs/phonetic.md](docs/phonetic.md) |
+| `phonetic` | finds 89% of place-name pairs across scripts (OpenStreetMap, held out, rules 2026-10-11.7; 91% on a second slice, rules 2026-10-11.8; rules 2026-10-12 add 0.34 points on a third); 84% of people on clean names in 21 languages (rules 2026-10-05, a lower bound for the rules that ship); 88% of a second roster as written (titles, initials; rules 2026-10-11); with `phonetic-search`, 93% of places (search 2026-10-11.7; 92% on the second slice, search 2026-10-11.8) and 86% of a roster as written (search 2026-10-09) | romanize + Soundex: 71% of place pairs, 23% of people in Tamil; cannot read Urdu; ICU + fuzzy match: 74% of places | 9 KB | [docs/phonetic.md](docs/phonetic.md) |
 | `segment` | cuts 0.05% of Kannada words wrongly | `Intl.Segmenter`: 50.3% | 8 KB | [docs/segment.md](docs/segment.md) |
-| `romanize` | writes 48.9% of the words of running text as people typed them, in 11 languages; 71.3% of names (tables: 0.2–1.8 MB a language) | IndicXlit (Python, a 119 MB model): 37.9% and 49.0% | 8 KB | [docs/romanize.md](docs/romanize.md) |
-| `deromanize` | writes the right word first for 87.1% of the words of Hindi news typed in Latin, and in four for 97.4%; 69.7% of names in 19 languages (tables: 0.3–3.1 MB a language) | IndicXlit: 86.1%, 89.7% and 66.4%; behind on rare words out of context | 28 KB | [docs/deromanize.md](docs/deromanize.md) |
+| `romanize` | writes 48.9% of the words of running text as people typed them, in 11 languages; 71.3% of names; in news, 48.1% of names and 39.8% of names it was not trained on; in Hindi, the route option (names it was not trained on get the words tables; 26 KB more on a page that has them, 2.5 MB otherwise) takes news names from 55.0% to 62.2% (Hindi only: in the other languages it loses on Wikidata's names) (tables: 0.2–1.8 MB a language) | IndicXlit (Python, a 119 MB model): 37.9% and 49.0%; in news 49.6%, 58.7% and 59.1% | 9 KB | [docs/romanize.md](docs/romanize.md) |
+| `deromanize` | writes the right word first for 87.1% of the words of Hindi news typed in Latin, and in four for 97.4%; 81.1% of typed sentences in 11 languages (Dakshina, rules 2026-10-12); 69.7% of names in 19 languages; on development names 69.2% with one accepted spelling and 74.5% with every attested spelling, the key of 0.13.0 (tables: 0.3–3.1 MB a language) | IndicXlit: 86.1%, 89.7%, 77.1% and 66.4%; 66.2% and 71.4%; behind on isolated words out of context (69.4% against 72.6% on Dakshina's lexicons) | 34 KB | [docs/deromanize.md](docs/deromanize.md) |
 | `wellformed` | agrees with HarfBuzz on 100% of 135,944 held-out PDF words (9,594 of them broken) and 2,146,974 new Wikipedia words | a check that the word starts with a combining sign: finds 84% of the broken words, with 4,667 false alarms | 6 KB | [docs/wellformed.md](docs/wellformed.md) |
 
 (Browser files are gzipped.)
@@ -515,7 +517,9 @@ best of them, indic-trans (about 200 MB), gives 34.7% on running text against
 - **`romanize` is behind IndicXlit on OpenStreetMap place names and on rare
   single words**: on a held-out slice of OpenStreetMap the places mode is
   right 44.4% of the time, IndicXlit 49.1% (it leads on village names, 57.6%
-  against 52.8%; Kannada has no places tables). Santali has names only; Bodo and Dogri have
+  against 52.8%; Kannada has no places tables). On names in news that its
+  tables were not trained on, IndicXlit is ahead, 58.7% against 39.8%
+  (Hindi's route option spells them with its words tables). Santali has names only; Bodo and Dogri have
   text only. Bodo writes ख थ फ as kh th ph, as typists do, not as
   Aksharantar's k t f. Manipuri in Bengali script loses syllables in the words mode;
   its tables were trained on Meetei Mayek. See [docs/romanize.md](docs/romanize.md).
@@ -566,8 +570,8 @@ Every table and switch of a utility is in its rules file
 Go and TypeScript code is a short loop over it.
 Both are checked against a conformance file of inputs with the outputs that
 a reference implementation gave them: 345,276 inputs for `normalize`,
-418,665 for `stem`, 199,992 for `phonetic`, 858,654 for `segment`, 3,221
-for `romanize`, 2,990 for `deromanize`, 32,279 for `wellformed`, and 1,031 search rows for
+418,665 for `stem`, 199,992 for `phonetic`, 858,654 for `segment`, 3,347
+for `romanize`, 3,119 for `deromanize`, 32,279 for `wellformed`, and 1,031 search rows for
 `phonetic-search` (`*/testdata/*conformance.jsonl.gz`). A change that makes any one disagree
 on any input fails the build.