indickit 0.11.0 → 0.13.0

package/CHANGELOG.md

@@ -0,0 +1,458 @@
+# Changelog
+
+Each version is one Git tag `v<version>`, one npm version, and one Go
+module version. The **Rules** line of an entry names the rules version of
+each utility. When a utility's rules version changed, some inputs give a
+different output than before: recompute what you stored from the old
+version, and compare `RULES_VERSION` (or `SEARCH_VERSION`) to know which
+rows. A version whose rules line equals the one before it gives every
+input the same output.
+
+## 0.13.0 (2026-10-13)
+
+Rules: phonetic 2026-10-12, phonetic-search 2026-10-12, normalize 2026-10-12.1, segment 2026-10-11, stem 2026-10-12, romanize 2026-10-13, deromanize 2026-10-12.1
+
+- Changed output: phonetic 2026-10-11.8 → 2026-10-12, phonetic-search
+  follows (thresholds unchanged). Recompute stored keys. Six folds from the
+  key's misses on village names: a Latin final -on is also -o (Pimpalgaon =
+  पिंपळगाव), Bk and Kh are also budruk and khurd, a Latin rh is also r
+  (Aligarh = अलीगढ, a known miss before), Bengali-script ওয় is also ov
+  (Dewan = দেওয়ান), a Latin initial ye is also e (Yekodi = एकोडी), Tamil ா்
+  is also ர் (நாவலூா் = Navallur). The native forms of the titles Sardar,
+  Maulana, Janab and Begum are dropped like the Latin ones:
+  match("Sardar Vallabhbhai Patel", "સરદાર વલ્લભભાઈ પટેલ") is now true.
+  On OpenStreetMap places held out to this read (1,506 places), mean recall
+  rises 0.34 points (95%: +0.14 to +0.72), Marathi 2.4, Hindi 2.0, wrong
+  merges +1.2%; on the development half of LGD villages 79.4% → 83.2%.
+  Every key of 0.12.0 is kept; 1,334 of the 199,992 test words gain one.
+  rules.json gains two fields both ports read: `abbreviations` and
+  `second_readings`. The browser file grows to 9 KB gzipped.
+- New option, default output unchanged: romanize rules 2026-10-12 →
+  2026-10-13 adds a route to Hindi's names mode. Every call as before gives
+  the same output as 2026-10-12. With the route, a word in the list of Hindi
+  names the tables were trained on (7,604 words,
+  `romanize/lang/hi/hi.known.bin`, 26 KB brotli) keeps its names list, and
+  any other word gets the words mode's list. On names in news held out (PIB,
+  August 2023) Hindi's first spelling goes from 55.0% to 62.2% (+7.2
+  points, 95% interval +4.1 to +10.0; IndicXlit 59.1%). Only Hindi has a
+  route: in the other languages the words tables spell Wikidata's unseen
+  names worse.
+- API: TypeScript `load("hi", "names", { route: true })` turns the route on;
+  it also fetches the Hindi words tables and the list (26 KB more on a page
+  that has the words tables, 2.5 MB in all otherwise). `fromBytes` takes an
+  optional sixth argument, `{ known, other }` (the list's bytes and the Hindi
+  words Romanizer), and routes only when it is given. Go:
+  `romanize.WordWith` and `romanize.TextWith` take `romanize.Options{Route:
+  true}` and return an error when the list or the words tables are not
+  imported (`lang/hi` imports both). `Word`, `Text` and `Languages` are
+  unchanged. The browser file grows to 9 KB gzipped.
+- Changed output: deromanize rules 2026-10-09.2 → 2026-10-12.1. Recompute stored
+  output. Sanskrit's words model learns from all of Aksharantar's Sanskrit pairs (+2.8
+  points on development words, +1.4 on running text). A spelling a renderer would draw
+  broken (a dotted circle) or with punctuation inside (अभवत्।, ري،) goes after every
+  whole spelling, in both modes. In words mode, a spelling that writes nothing for a
+  consonant is no longer ranked first in Kashmiri, Manipuri, Sanskrit and Sindhi. The
+  word lists are keyed with normalize 2026-10-12.1 (217 keys in as bn kn ml or ta, each a
+  two-part vowel sign typed in the other order, merge into the frequent word's key). Of the
+  conformance file's 3,028 shared inputs, 297 change, 33 at the first answer (sa 17,
+  sd 5, ks 3, ur 2, as 2, bn brx ne ta 1). Held out, read once: Dakshina's typed
+  sentences (11 languages) 81.1% first against IndicXlit's 77.1%; Dakshina's test
+  lexicons 69.4% against 72.6%. The names key changed: a typed name now accepts every
+  attested spelling (development names 69.2% → 74.5%; IndicXlit 66.2% → 71.4%).
+  deromanize output is always normalized text: a spelling that normalize rewrites is
+  returned in its normalized form, and a duplicate that results is dropped (74 of 3,114
+  conformance inputs change against the unreleased 2026-10-12, 6 at the first answer, 5
+  of them in bytes only).
+  The browser file grows from 28 KB to 34 KB gzipped: it now holds `wellformed`.
+- Changed output: normalize rules 2026-10-12 → 2026-10-12.1. Recompute stored
+  text. One rule added, nothing else changes: a two-part vowel sign typed as
+  its parts in the other order (Odia କଠାେର, ା then େ, drawn as କଠୋର) becomes
+  the sign, after a consonant or a nukta, in the 16 orders of Bengali, Odia,
+  Tamil, Kannada and Malayalam that every font we hold draws the same (four
+  Kannada orders stay). On the other half of the Odia web text (IndicCorp v2,
+  read once): look-alike groups made equal 80.5% → 81.2%; of 215 groups holding
+  such a spelling, 1 → 172 joined; no word changed its look by this rule.
+  `fold` is unchanged; no search key of the conformance sample moves.
+  rules.json gains `order`, which both ports read. The browser file is 22.5 KB
+  gzipped (README budget 23 KB). deromanize's word lists are keyed by
+  normalize and are re-exported against this version.
+
+## 0.12.0 (2026-10-12)
+
+Rules: phonetic 2026-10-11.8, normalize 2026-10-12, segment 2026-10-11, stem 2026-10-12, romanize 2026-10-12, deromanize 2026-10-09.2, phonetic-search 2026-10-11.8
+
+- Faster, same output: `deromanize` 2.0x in Go and 1.35x in TypeScript (a
+  Hindi word in Go: 11.0 -> 4.4 ms); `romanize` 1.5-2.0x in Go and 1.1x in
+  TypeScript (a Hindi sentence in Go: 31 -> 15 ms). The n-gram tables are
+  sorted arrays, not one map entry per pair, and the last letter builds only
+  the spellings that can be returned, best first.
+- Faster, same output: `Search` 2.0x in Go and 2.4x in TypeScript, `NewIndex`
+  1.6x and 1.8x. A word's keys are computed once per call, and a pair of
+  words is scored once.
+- Less memory, faster loads: a loaded Hindi deromanizer keeps 18 MB of heap
+  in Go (was 35 MB) and loads in 36 ms (was 111 ms); a romanizer keeps 15 MB
+  (was 29 MB) and loads in 40 ms (was 96 ms). The Go word lists are kept as
+  bytes. (0.11.0 already made TypeScript's tables flat: a Hindi deromanizer
+  130 -> 35 MB, a romanizer 194 -> 42 MB.) A test in each port fails if a
+  loaded language grows back past its bound.
+- Changed output: phonetic 2026-10-11.7 → 2026-10-11.8, phonetic-search
+  follows. Recompute stored keys. The Perso-Arabic letter ھ (U+06BE) is read
+  as h: Sindhi writes its h with it (سنھا = Sinha, جوھي = Juhi), and the
+  reader skipped it. On Wikidata people (DEV) Sindhi rises from 78% to 83%,
+  Urdu, Kashmiri and Hindi by under a point, the wrong-match rate unchanged.
+  2,251 of the 199,992 test words (those with ھ) gain a key; every key of
+  0.11.0 is kept but for 20 of them, with ھ at the start or after a vowel,
+  where the h-less key goes (شاھ = Shah: c → ch); the one that matters is a
+  variant spelling of Chaudhry (چوھدری; the usual چودھری keeps its key).
+- Changed output: stem rules 2026-10-11 → 2026-10-12, Kannada and Marathi
+  only. Kannada gains 6 endings (ದನ್ನು ಂದು ಿವೆ ಂತೆ ಾಗ ತ್ತ) and Marathi 10
+  (postpositions written solid: ंमध्ये बद्दल मुळे ंनी ंना द्वारे कडे ंचा िक
+  ान), chosen on PIB queries inside the wrong-hit budget those languages
+  had not spent; 6,001 of the 729,361 conformance rows change (kn 3,130,
+  mr 2,871). On PIB sentences of May 2021, which no rule was chosen on and
+  which were read once, Kannada finds 0.5 points more of the right
+  sentences (95%: +0.2 to +1.0) and Marathi 0.8 (+0.5 to +1.4), at 0.4
+  and 0.2 points of precision. Recompute stored stems in kn and mr.
+- Changed output: normalize rules 2026-10-11 → 2026-10-12. Recompute stored
+  normalized text. "Looks the same" is now decided by every font we hold that
+  draws what `normalize` writes, 2 to 15 families a script compared by glyph
+  outline, not by Noto and Anek alone, and the joiner table and tree were
+  rebuilt with them. On the 12.9M words of the measurement sets, drawn in
+  those fonts, the words whose look changes fall from 54 to 17 (all malformed
+  input or a font's quirk); the held-out number (88.1%) is unchanged, and the
+  groups made equal move by under two points in Malayalam and Bengali (down)
+  and three in Perso-Arabic (up). Three faults fixed: khanda ta (ত্ + ZWJ → ৎ)
+  is not written where the ZWJ joins ত্ to a cluster with a left-side vowel
+  sign or a reph, or under a reph (উত্‍ক্ষেপন keeps its ZWJ); an old Malayalam
+  chillu before a vowel sign stays as typed; a ZWJ between a Malayalam
+  consonant and its vowel sign (ന + ZWJ + ു) is kept, which fonts with
+  traditional ligatures draw. Two more, found on web text no rule was chosen
+  on (IndicCorp v2, read once): the chillu k (ൿ) is no longer written (not in
+  Unicode's DoNotEmit list; Noto draws it apart), and no chillu is written
+  before a second virama; Malayalam words whose look changes there fall from
+  16,327 to 47 of 83M, with 91.8% of joiner groups made equal (Odia 80.5%).
+  Odia still loses a ZWNJ in some acronyms and a ZWJ after ଡ଼ + virama (310
+  of 922,000 words): docs/normalize.md. The browser file grows from 16 KB to
+  22 KB gzipped: the tree holds more contexts; `deromanize` (22 → 28 KB)
+  and `search-key` (19 → 26 KB) embed the same tree and grow with it.
+- Changed output: romanize rules 2026-10-11.2 → 2026-10-12, the names mode
+  only (words and places are unchanged). The ten best spellings by the
+  alignment score are scored again with a letter 5-gram over the training
+  English spellings, and the best four of that order are returned: the rank
+  had English-side habits wrong (the schwa, ee against i, a lost h), and the
+  accepted spelling was usually already in the list. On development names
+  that neither tool's training holds, the first spelling is right 51.7% of
+  the time, was 47.0% (IndicXlit 39.7%); any of four 72.8%, was 70.4%. On
+  names held out to the end (Wikidata people of Bangladesh, Pakistan, Nepal
+  and Sri Lanka, read once, 10,773 unseen tokens) the first spelling goes from
+  37.9% to 41.0%, no language falling. Of the 1,008 names rows of the
+  conformance file, 83 change their first spelling and 794 their order.
+  `rules.json` gains `rescore`; the model file `lang/shared/names-lm.bin`
+  (148 KB brotli) is loaded with every names page, in Go through the
+  language package. The names lookup now holds the names the shipped beam
+  spells otherwise (13,495 entries, was 16,632 at the reference's beam).
+  TypeScript `fromBytes` takes the letter model as a fifth argument for the
+  names mode.
+
+## 0.11.0 (2026-10-12)
+
+Rules: phonetic 2026-10-11.7, normalize 2026-10-11, segment 2026-10-11, stem 2026-10-11, romanize 2026-10-11.2, deromanize 2026-10-09.2, phonetic-search 2026-10-11.7
+
+- Changed output: normalize rules 2026-10-07 → 2026-10-11, Tamil only. A
+  ZWNJ or ZWJ between consonant + pulli and a consonant is deleted, unless
+  the pair is one of the three Tamil conjuncts (க்ஷ, ஶ்ர, ஸ்ர): ப் + ZWNJ + ப →
+  ப்ப. Before, a class-based rule kept most of these and deleted the ZWNJ of
+  க்ஷ in some words, a change a reader could see. On IndicCorp Tamil text,
+  look-alike groups with a joiner made equal: 89.6% → 98.8%. `fold` follows.
+  `rules.json` gains `pairs`; the ports read it.
+- Changed output: segment rules 2026-10-06 → 2026-10-11, Odia only. ଓ +
+  virama + ବ is one letter (ଓ୍ବି in ଓ୍ବିକେଟ୍ "wicket"); Odia news words split
+  wrongly: 0.13% → 0.04%. ଓ୍ୱ stays two letters (two fonts draw it apart).
+- Docs: normalize's Malayalam limits (the most possible is 71.0% in 15
+  fonts; a ZWJ before a vowel sign; Suruma lacks the atomic chillus).
+- Added: deromanize `textMixed` / Go `TextMixed`, and `isEnglish` / Go
+  `English`, for Hindi typed with English words in it (Hinglish): a word on
+  the Hindi English list (10,000 English Wikipedia words, less those a Hindi
+  typist writes often) stays as typed. TypeScript fetches the list (22 KB
+  gzipped) only with `load("hi", "words", { english: true })`. On
+  L3Cube-HingLID, read once, 87.4% of words are called right (IndicLID
+  roman: 72.1%); 2.4% of Hindi words stay in Latin. Every other output is
+  unchanged, so the deromanize rules version is unchanged.
+- Added: romanize mode `"places"` (Go `romanize.Places`), rules
+  2026-10-11 → 2026-10-11.2, for place names. Own tables for bn hi ml mr or
+  sat ta te ur, learned from Wikidata place names and, for hi mr or, village
+  names of the Local Government Directory; the words tables in the other
+  languages. Held out, read once, the training's own names left out: village
+  names 57.6% first spelling right (words mode 48.0%, IndicXlit 52.8%);
+  OpenStreetMap place names 44.4% (40.6%, IndicXlit 49.1%). The words and
+  names modes are unchanged since 2026-10-11.
+- Changed output: romanize rules 2026-10-09 → 2026-10-11, words mode.
+  Bodo writes ख थ फ as kh th ph, as Bodo speakers type them (थाखाय →
+  thakhai, was takai), not as Aksharantar's k t f. On typed Bodo sentences
+  (Bhasha-Abhijnaanam, read once) the first spelling is right 71.9% of the
+  time, was 50.3% (IndicXlit 49.3%); on Aksharantar's word list it falls
+  from 75.6% to 45.6%. The shared Brahmic table was retrained, so a few
+  lower-ranked spellings in other languages change too (61 of 1,018 words
+  conformance rows; no language's first spelling moved by more than 0.2
+  points). The names mode is unchanged.
+
+- Changed output: stem rules 2026-10-09 → 2026-10-11, Sanskrit only. 17
+  endings added (the anusvara spellings ानां ां ायां, the passive यते यन्ते,
+  the infinitive तुं, the absolutive त्वा, and others); 1,301 of the 418,730
+  conformance rows change, all Sanskrit. On lemmas of the Digital Corpus of
+  Sanskrit, read once, recall rises from 34.6% to 40.5% at 2.7% wrong hits.
+- Docs: `stem` damages some names (README Limits, `docs/stem.md#names`).
+- Changed output: phonetic 2026-10-07 → 2026-10-11.7. Recompute stored keys.
+  `Keys` gives more keys to some words; every key a word had in 0.10.0 it
+  still has (checked on every test word, but two long Urdu words whose
+  readings exceed the cap of 32 keys a word, was 16), so every pair 0.10.0
+  matched still matches. On held-out OpenStreetMap place names the key
+  finds 89% of name pairs (romanize + Soundex: 71%).
+  - Bengali-script ং and Devanagari ं also read as ŋ: সিং = Singh (Bengali
+    and Assamese), and Bodo spelling (दं = dong; Bodo running text 74% →
+    86% of words).
+  - Tamil ட்ச also reads as க்ஷ: லட்சுமி = Lakshmi. A Tamil name ending in
+    ன் or ர் is also keyed without it; a Tamil word starting with a vowel
+    also keys with h (உசைன் = Hussain). Latin zh is ழ (Kanimozhi);
+    Malayalam റ്റ is also tt and ന്റ also nt (ആന്റണി = Antony). Tamil +1.6 and
+    Malayalam +1.9 points on Wikidata (DEV); on BPCC name pairs, read once,
+    Tamil +0.9 and Malayalam +1.5.
+  - Assamese: ৱ after a vowel (not before i) is also a glide (গোৱেল =
+    Goyal), and the case endings ৰ, ৱে, লৈ, ৰপৰা are also read off.
+  - A one-letter Latin word also keys as its spoken name: M = em = एम =
+    எம். M and N, and H and S, now share a key as initials.
+  - Santali: an initial ᱚ is also a (Ol Chiki writes the Indic short a
+    with it: Arjun = ᱚᱨᱡᱩᱱ), and ᱣ ᱶ after a vowel is also a glide;
+    Santali 74.1% → 84.7% on Wikidata (DEV).
+  - Sindhi in Devanagari: the implosives ॻ ॼ ॾ ॿ are read (they had no
+    key); व्ह is also v (चव्हाण = Chavan); the long Marathi / Konkani case
+    endings (च्या ांच्या मध्ये ांना), the Konkani -ान -ीन -ाईन -ाक -ाचो -ाची
+    -ाचें, the Sanskrit -स्य and the Sindhi / Bodo -नि are also read off. On
+    BPCC name pairs, read once, +0.49 points (Bodo +3.5).
+  - Urdu, Kashmiri and Sindhi script: an initial alif or ع is also i, e or u
+    (اندر = Indra, عمران = Imran, which did not match before), and ہ ح
+    after a consonant is also silent (جہانگیانی = Jhangiani). Urdu +4.8,
+    Kashmiri +4.3 points on Wikidata (DEV); on BPCC name pairs, read once,
+    Urdu +3.2. Latin zh is ழ's l or z + h, so both Kanimozhi and Mazhar
+    (مظہر) match.
+  - Tamil's இ or உ before an initial r or l is also dropped (இராஜ்பால் =
+    Rajpal); व after a consonant and before u is also the au glide (கவுர் =
+    Kaur); ड ढ after the first letter are also r, the flap written without its nukta
+    (राबडी = Rabri). Tamil +3.8 points on Wikidata (DEV).
+- Changed output: `NameKeys`, `JoinedKeys` and `Match` (`nameKeys`,
+  `joinedKeys`, `match`) drop the leading titles (Shri, Smt., Dr, श्रीमती,
+  திரு, …; `titles` in rules.json) of each comma-separated part of a name:
+  "Bajwa, Sardar Partap Singh" reads as Bajwa Partap Singh. A name of titles
+  only keeps its last word. `Words` still splits only. A second roster as
+  written, the Rajya Sabha: the key finds 54.8% → 87.7%.
+- Changed output: `JoinedKeys` (`joinedKeys`) returns keys of 6 classes
+  or more (`JoinedMinClasses`, was 8): a name written as one word in one
+  script and as two in another meets more often (Amartya Sen =
+  अमर्त्यसेन). +1.6 points on Wikidata (DEV), +2.7 on OpenStreetMap places
+  (read once).
+- Changed output: phonetic search 2026-10-09 → 2026-10-11.7. It keys with
+  phonetic rules 2026-10-11.7, drops titles as `NameKeys` does, and its
+  thresholds were chosen again on the same data: names 74 → 76, villages
+  72 → 73, text 80 → 81 and 70 → 71 (`Thresholds` / `THRESHOLDS`). On held-out
+  OpenStreetMap places it finds 93% of places in five languages (ICU +
+  fuzzy match: 74%), at about 3 wrong places per search per 10,000 names.
+- Size: the title list, the letter names and the new rules add about
+  1.3 KB gzipped to `phonetic`: 6.9 → 8.2 KB. `phonetic-search` falls
+  17.0 → 14.5 KB: its 2^x tables are stored as second differences, and its
+  two fine rules files as patches over `rules.json` (every score unchanged).
+- Changed output: stem, in the same rules 2026-10-11. Gujarati, Marathi,
+  Maithili and Santali run their endings twice, as Kannada does; Maithili
+  and Santali gain 7 endings each. On text no rule saw, read once:
+  Gujarati +1.7 and Marathi +1.6 points of recall (PIB), Maithili +0.95
+  (BPCC), Santali +0.56 (FLORES+); precision falls by at most 1.3 points
+  against the old output. About 12,900 conformance rows change, in these
+  four languages only.
+- `stem`: Sindhi in Perso-Arabic and Manipuri in Bengali script get their
+  own endings in the same tables, chosen on FLORES+ and checked once on its
+  unseen half (Manipuri: 5.6 points more of the right sentences than exact
+  search). Sindhi in Perso-Arabic then gained more endings, checked once on
+  the unseen test part of the Sindhi UD treebank: the share of inflected
+  forms that reach their word's stem rose from 20.2% to 33.0%, level with
+  Snowball's Arabic stemmer, with fewer wrong merges. Words in those
+  scripts were returned unchanged before; no Devanagari or Meetei Mayek
+  output changes.
+- New: `indickit/search-key` (Go `searchkey.Key`), one search key per word:
+  normalize, stem, then fold, the order measured best. Fold after stem
+  finds 2.4 points more of the right passages in Hindi (MIRACL) and changes
+  nothing in Bengali or Telugu. The README's search recipe now uses it, and
+  `docs/stem.md` gives an OpenSearch/Elasticsearch setup with a trigram
+  field beside the keys.
+- New in `stem`: two options. `minStem` (Go `StemWith`, `Options.MinStem`)
+  keeps stems longer, for fields of names; 4 halves the names a search
+  loses. `indickit/stem-compounds` (Go `CompoundKey`), a separate 1.6 KB
+  file, keys a Malayalam word by its first compound member; on held-out
+  news it finds 5.5 points more than `stem` alone. The default output does
+  not change.
+- New in `stem`: Bodo, Dogri, Kashmiri, Konkani, Maithili, Manipuri
+  (Meetei Mayek), Santali and Sindhi (Devanagari), in the same rules
+  2026-10-11: `stem` now covers all 22 languages. Endings chosen on BPCC's
+  human translations; on a part no rule saw, 3.7–11.4 points more of the
+  right sentences than exact search, at most 4.4 points less precise.
+  About 254,000 conformance rows are new; none changed. The browser file
+  grows to 4 KB gzipped.
+- Language tags: `ory` (Odia) and `npi` (Nepali), as FLORES+ and NLLB
+  write them, now name the `or` and `ne` tables.
+- New in `stem`: Odia (`or`), in the same rules 2026-10-11. 11 endings
+  chosen on PIB search; on FLORES+, read once, 6.9 points more of the right
+  sentences than exact search, 3.4 points less precise. Odia's 24,578
+  conformance rows are new.
+
+## 0.10.0 (2026-10-09)
+
+Rules: phonetic 2026-10-07, normalize 2026-10-07, segment 2026-10-06, stem 2026-10-09, romanize 2026-10-09, deromanize 2026-10-09.2, phonetic-search 2026-10-09
+
+- Changed output: phonetic search 2026-10-08 → 2026-10-09, for language
+  tags only. `NewIndex` and `Score` (Go), `loadSearch` (TypeScript) took
+  a bare code (`hi`) where every other utility takes a tag: `hi-IN`,
+  `HI` and `hin` found no Latin table and scored by the fine keys alone.
+  They now read the tag as the other utilities do. The tables are
+  unchanged: every score for a bare code is the same.
+- Changed output: `kok` (the Konkani macrolanguage) names the `gom`
+  tables in every utility's tag reading. romanize and deromanize did
+  this already; stem and normalize hold no Konkani table, so nothing
+  changes there.
+- Removed exports: TypeScript `_internal` (phonetic) and `unify`
+  (romanize), which no documented use needed. The phonetic engine now
+  lives in `js/phonetic-engine.ts`, shared by `phonetic` and
+  `phonetic-search`, not an entry of the package.
+- Fixed: `phonetic.NewIndex` and `Score` (Go), `index` and `score`
+  (TypeScript) given a profile name other than `names` or `text` crashed
+  later with no cause named (Go: divide by zero in the first `Search`;
+  TypeScript: a `TypeError`). They now fail at once, naming the profile.
+- Fixed: the search index built every combination of a name's word keys
+  before cutting to 256 (a 26-word name of two-key words: 2^26 strings,
+  8 s in Go, 4.9 s in TypeScript). It now builds the first 256 only, as
+  `NameKeys` does; the keys are the same.
+- Go `stem` checks at load that `max_end + min_stem` in rules.json is
+  below its fixed walk (16): a larger value would have cut differently
+  from the reference, silently.
+
+## 0.9.0 (2026-10-09)
+
+Rules: phonetic 2026-10-07, normalize 2026-10-07, segment 2026-10-06, stem 2026-10-09, romanize 2026-10-09, deromanize 2026-10-09.2, phonetic-search 2026-10-08
+
+- New: `wellformed`, a check that a word is drawn as written
+  (`indickit/wellformed`).
+- Changed output: stem rules 2026-10-06 → 2026-10-09. Kannada applies its
+  stem twice (ಪ್ರಧಾನಮಂತ್ರಿಯವರು → ಪ್ರಧಾನಮಂತ್ರಿ): on held-out PIB text,
+  recall +0.86 points, precision −0.75. No other language changes.
+- Changed output: romanize rules 2026-10-07 → 2026-10-08. A ZWJ or ZWNJ
+  inside a word no longer cuts the word (प्राप्\u200dत gave "the"). Manipuri
+  in Bengali script is romanized from the Brahmic tables (রামেন gave "en");
+  the `meetei` table group is gone, and `words-brahmic.bin` grows from
+  2,543 to 2,679 KB.
+- Changed output: deromanize rules 2026-10-08 → 2026-10-09. Names mode no
+  longer ranks first a spelling that writes nothing for a consonant
+  (ml "murmu" gave മു, ks "rajnath" gave راج). Still open: sd "Rafi" in
+  words mode keeps a comma chunk, ranked second.
+- Changed output, Go only: NFC now equals Python and ICU on long runs of
+  combining marks (Go inserted U+034F after 30 marks). Names split on the
+  same spaces in Go and TypeScript (Go missed 24, among them the no-break
+  space).
+- Language tags: `hi-IN`, `HI` and `hin` now mean Hindi in Go and
+  TypeScript; before, they silently meant no language.
+- Fixed: TypeScript `normalize` and `fold` threw `RangeError` above about
+  125,000 code points; Go `NameKeys` took quadratic time (70.6 s → 0.09 s
+  on 1 MB of Tamil).
+- Go and TypeScript read one pinned Unicode 15.0 table for the blocks
+  indickit has rules for. Cost: about 2.9 KB gzipped in each bundle that
+  reads it; the phonetic, normalize and romanize bundles are now over their
+  README budgets.
+- Fixed: Go `phonetic.Score` and `Index.Search` panicked (integer divide
+  by zero) on a query word of about 10,000 letters; they now give the
+  reference's scores. TypeScript `score` threw `RangeError` from about
+  125,000 candidates in Node.
+- Faster on long input, same output: `wellformed` (262,144 code points:
+  2.3 s -> 0.03 s in Go, 7.2 s -> 0.04 s in TypeScript) and the `romanize`
+  and `deromanize` decoders (a 4,096-letter word ran out of 6 GB in Go). `phonetic-search`
+  still grows with (query words) x (candidate words): cap user input
+  (README, Limits).
+- Changed output, TypeScript only: `romanize` text() keeps a ZWJ or ZWNJ
+  that no run of the script goes on from (at the start, after a space,
+  after an unassigned code point), as Go always did; it deleted it.
+- Changed output: romanize rules 2026-10-08 -> 2026-10-09 and deromanize
+  2026-10-09 -> 2026-10-09.1. A word of more than 256 letters (`max_letters`)
+  gets one spelling: the first spelling of each 256-letter piece, joined.
+  Words that long are not words; the beam was quadratic on a long run of
+  characters it has no table for (32,000 random code points: 49 s).
+- Faster, same output: `deromanize` 1.5-1.6x (a Hindi word in the browser:
+  19 -> 12 ms) and `romanize` 1.2-1.4x, in Go and TypeScript. The
+  `romanize` browser file is now 9 KB.
+- Changed output: deromanize rules 2026-10-09.1 -> 2026-10-09.2. No
+  spelling is empty: "q" in Bengali gave ক, ক্ and an empty spelling; the
+  empty one is dropped, as `romanize` always did (11 of 3,033 conformance
+  rows change, each by that drop).
+- The browser files are minified: each is 0.3 to 3.5 KB smaller gzipped
+  (`phonetic` 7 KB, `normalize` 16 KB, `romanize` 7 KB, `deromanize`
+  21 KB, `phonetic-search` 17 KB). Same code, same output (`check-dist`).
+
+## 0.8.0 (2026-10-08)
+
+Rules: phonetic 2026-10-07, normalize 2026-10-07, segment 2026-10-06, stem 2026-10-06, romanize 2026-10-07, deromanize 2026-10-08, phonetic-search 2026-10-08
+
+- New: `deromanize`, Latin typing to Indian script (`indickit/deromanize`).
+- New: `phonetic-search`, a scored search over phonetic keys
+  (`indickit/phonetic-search`).
+- Changed output: phonetic rules 2026-10-06.1 → 2026-10-07, which added the
+  folds `brahmic-av-u`, `flap-r` and `latin-ngh`.
+- README: a table of the 22 languages and what each utility covers.
+- 0.7.0 was committed but never tagged or published; its work is here.
+
+## 0.6.0 (2026-10-07)
+
+Rules: phonetic 2026-10-06.1, normalize 2026-10-07, segment 2026-10-06, stem 2026-10-06, romanize 2026-10-07
+
+- New: `romanize`, Indian text in Latin letters as people spell it
+  (`indickit/romanize`). Its tables load at run time, beside the module or
+  from jsDelivr at this version's tag.
+
+## 0.5.0 (2026-10-07)
+
+Rules: phonetic 2026-10-06.1, normalize 2026-10-07, segment 2026-10-06, stem 2026-10-06
+
+- Changed output: `fold` rules 2026-10-07.
+- Changed output: phonetic 2026-10-06.1, a fix for digits.
+- Language tags on `normalize` and `fold`.
+
+## 0.4.3 (2026-10-06)
+
+Rules: phonetic 2026-10-06, normalize 2026-10-05, segment 2026-10-06, stem 2026-10-06
+
+- First version on npm. README names the floors: Go 1.22, Node 20,
+  TypeScript `moduleResolution` nodenext or bundler.
+
+## 0.4.2, 0.4.1 (2026-10-06)
+
+Rules: as 0.4.0.
+
+- Go floor 1.22. 0.4.1 failed its README test and was not published.
+
+## 0.4.0 (2026-10-06)
+
+Rules: phonetic 2026-10-06, normalize 2026-10-05, segment 2026-10-06, stem 2026-10-06
+
+- New: `stem`, one search key for the forms of a word, 13 languages.
+
+## 0.3.0 (2026-10-06)
+
+Rules: phonetic 2026-10-06, normalize 2026-10-05, segment 2026-10-06
+
+- New: `segment`, the letters a reader sees.
+
+## 0.2.0 (2026-10-06)
+
+Rules: phonetic 2026-10-06, normalize 2026-10-05
+
+- New: `normalize`, one encoding for text that looks the same.
+- Changed output: phonetic rules 2026-10-05 → 2026-10-06.
+
+## 0.1.0 (2026-10-05)
+
+Rules: phonetic 2026-10-05
+
+- New: `phonetic`, a key that matches one name across scripts.

package/README.md

@@ -99,14 +99,14 @@ TypeScript, set `moduleResolution` to `nodenext` or `bundler`; the old
 In a browser, with no build step:
 
 ```js
-import { normalize } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.11.0/dist/normalize.js";
-import { stem } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.11.0/dist/stem.js";
-import { match } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.11.0/dist/phonetic.js";
-import { segment } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.11.0/dist/segment.js";
-import { load } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.11.0/dist/romanize.js";
-import { loadSearch } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.11.0/dist/phonetic-search.js";
-import { load as loadDeromanize } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.11.0/dist/deromanize.js";
-import { isWellFormed } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.11.0/dist/wellformed.js";
+import { normalize } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.13.0/dist/normalize.js";
+import { stem } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.13.0/dist/stem.js";
+import { match } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.13.0/dist/phonetic.js";
+import { segment } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.13.0/dist/segment.js";
+import { load } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.13.0/dist/romanize.js";
+import { loadSearch } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.13.0/dist/phonetic-search.js";
+import { load as loadDeromanize } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.13.0/dist/deromanize.js";
+import { isWellFormed } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.13.0/dist/wellformed.js";
 ```
 
 Each utility is its own file, so a page loads only the rules it uses.
@@ -383,12 +383,14 @@ key("bahut") === key("bohot");     // true: both are बहुत
 key("kahaan") === key("kahan");    // false: कहाँ and कहा
 ```
 
-On words `deromanize` never trained on (Dakshina's lexicons, Bengali,
-Hindi, Marathi, Tamil, Telugu and Urdu), this key matches 30–67% of the
-pairs of spellings of one word and gives 3–20% of spellings the key of
-another word. Fuzzy matching (one edit) matches 28–65% and gives 28–51% a
-wrong match. Keys from all four spellings match 52–88%, at 16–46% wrong.
-Tamil is the weakest language. The `phonetic` key is not for this: it drops
+On words `deromanize` never trained on (Dakshina's test lexicons, read
+once, 11 languages: Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi,
+Punjabi, Sindhi, Tamil, Telugu and Urdu), this key matches 40–69% of the
+pairs of spellings of one word and gives 0–13% of spellings the key of
+another word. Fuzzy matching (one edit) matches 33–57% and gives 10–39% a
+wrong match. Keys from all four spellings match 54–84%, at 4–33% wrong.
+Malayalam is the weakest language: level with fuzzy matching, at a tenth of
+its wrong matches. The `phonetic` key is not for this: it drops
 vowels, which carry a Latin-typed word.
 
 ### The same in Go
@@ -458,12 +460,12 @@ other tools tried and each weak spot are in `docs/`.
 
 | Utility | Result | Best other tool | Browser file | Details |
 |---|---|---|---|---|
-| `normalize` | makes 88.1% of look-alike spellings equal, and changes the look of no word | Indic NLP Library: 82.4%, and changes 0.68% of words | 16 KB | [docs/normalize.md](docs/normalize.md) |
+| `normalize` | makes 88.1% of look-alike spellings equal, and changes the look of no word | Indic NLP Library: 82.4%, and changes 0.68% of words | 23 KB | [docs/normalize.md](docs/normalize.md) |
 | `stem` | finds 3.7–15 points more of the right sentences than exact search, in 22 languages | ahead in 5 languages, level in 2, a trade in 4, behind in Nepali; no other stemmer for the other 10 | 4 KB | [docs/stem.md](docs/stem.md) |
-| `phonetic` | finds 89% of place-name pairs across scripts (OpenStreetMap, held out, rules 2026-10-11.7); 84% of people on clean names in 21 languages (rules 2026-10-05, a lower bound for the rules that ship); 88% of a second roster as written (titles, initials; rules 2026-10-11); with `phonetic-search`, 93% of places (search 2026-10-11.7) and 86% of a roster as written (search 2026-10-09) | romanize + Soundex: 71% of place pairs, 23% of people in Tamil; cannot read Urdu; ICU + fuzzy match: 74% of places | 8 KB | [docs/phonetic.md](docs/phonetic.md) |
+| `phonetic` | finds 89% of place-name pairs across scripts (OpenStreetMap, held out, rules 2026-10-11.7; 91% on a second slice, rules 2026-10-11.8; rules 2026-10-12 add 0.34 points on a third); 84% of people on clean names in 21 languages (rules 2026-10-05, a lower bound for the rules that ship); 88% of a second roster as written (titles, initials; rules 2026-10-11); with `phonetic-search`, 93% of places (search 2026-10-11.7; 92% on the second slice, search 2026-10-11.8) and 86% of a roster as written (search 2026-10-09) | romanize + Soundex: 71% of place pairs, 23% of people in Tamil; cannot read Urdu; ICU + fuzzy match: 74% of places | 9 KB | [docs/phonetic.md](docs/phonetic.md) |
 | `segment` | cuts 0.05% of Kannada words wrongly | `Intl.Segmenter`: 50.3% | 8 KB | [docs/segment.md](docs/segment.md) |
-| `romanize` | writes 48.9% of the words of running text as people typed them, in 11 languages; 71.3% of names (tables: 0.2–1.8 MB a language) | IndicXlit (Python, a 119 MB model): 37.9% and 49.0% | 8 KB | [docs/romanize.md](docs/romanize.md) |
-| `deromanize` | writes the right word first for 87.1% of the words of Hindi news typed in Latin, and in four for 97.4%; 69.7% of names in 19 languages (tables: 0.3–3.1 MB a language) | IndicXlit: 86.1%, 89.7% and 66.4%; behind on rare words out of context | 22 KB | [docs/deromanize.md](docs/deromanize.md) |
+| `romanize` | writes 48.9% of the words of running text as people typed them, in 11 languages; 71.3% of names; in news, 48.1% of names and 39.8% of names it was not trained on; in Hindi, the route option (names it was not trained on get the words tables; 26 KB more on a page that has them, 2.5 MB otherwise) takes news names from 55.0% to 62.2% (Hindi only: in the other languages it loses on Wikidata's names) (tables: 0.2–1.8 MB a language) | IndicXlit (Python, a 119 MB model): 37.9% and 49.0%; in news 49.6%, 58.7% and 59.1% | 9 KB | [docs/romanize.md](docs/romanize.md) |
+| `deromanize` | writes the right word first for 87.1% of the words of Hindi news typed in Latin, and in four for 97.4%; 81.1% of typed sentences in 11 languages (Dakshina, rules 2026-10-12); 69.7% of names in 19 languages; on development names 69.2% with one accepted spelling and 74.5% with every attested spelling, the key of 0.13.0 (tables: 0.3–3.1 MB a language) | IndicXlit: 86.1%, 89.7%, 77.1% and 66.4%; 66.2% and 71.4%; behind on isolated words out of context (69.4% against 72.6% on Dakshina's lexicons) | 34 KB | [docs/deromanize.md](docs/deromanize.md) |
 | `wellformed` | agrees with HarfBuzz on 100% of 135,944 held-out PDF words (9,594 of them broken) and 2,146,974 new Wikipedia words | a check that the word starts with a combining sign: finds 84% of the broken words, with 4,667 false alarms | 6 KB | [docs/wellformed.md](docs/wellformed.md) |
 
 (Browser files are gzipped.)
@@ -515,7 +517,9 @@ best of them, indic-trans (about 200 MB), gives 34.7% on running text against
 - **`romanize` is behind IndicXlit on OpenStreetMap place names and on rare
   single words**: on a held-out slice of OpenStreetMap the places mode is
   right 44.4% of the time, IndicXlit 49.1% (it leads on village names, 57.6%
-  against 52.8%; Kannada has no places tables). Santali has names only; Bodo and Dogri have
+  against 52.8%; Kannada has no places tables). On names in news that its
+  tables were not trained on, IndicXlit is ahead, 58.7% against 39.8%
+  (Hindi's route option spells them with its words tables). Santali has names only; Bodo and Dogri have
   text only. Bodo writes ख थ फ as kh th ph, as typists do, not as
   Aksharantar's k t f. Manipuri in Bengali script loses syllables in the words mode;
   its tables were trained on Meetei Mayek. See [docs/romanize.md](docs/romanize.md).
@@ -566,8 +570,8 @@ Every table and switch of a utility is in its rules file
 Go and TypeScript code is a short loop over it.
 Both are checked against a conformance file of inputs with the outputs that
 a reference implementation gave them: 345,276 inputs for `normalize`,
-418,665 for `stem`, 199,992 for `phonetic`, 858,654 for `segment`, 2,397
-for `romanize`, 2,990 for `deromanize`, 32,279 for `wellformed`, and 1,031 search rows for
+418,665 for `stem`, 199,992 for `phonetic`, 858,654 for `segment`, 3,347
+for `romanize`, 3,119 for `deromanize`, 32,279 for `wellformed`, and 1,031 search rows for
 `phonetic-search` (`*/testdata/*conformance.jsonl.gz`). A change that makes any one disagree
 on any input fails the build.