indickit 0.5.0 → 0.6.0

package/README.md

@@ -3,12 +3,14 @@
 Small, exact utilities for text in the languages of India, in Go and
 TypeScript. They fix the places where ordinary string code goes wrong on
 Indian text: a search that misses a word because it is spelled, encoded or
-inflected another way, and a letter count that cuts a letter in two.
+inflected another way, a letter count that cuts a letter in two, and a name
+that must be written in Latin letters.
 
 ```
 stem      किताब  किताबें  किताबों              →  किताब
 phonetic  राम  ರಾಮ  രാമ  ராம  رام  Ram       →  rn
 segment   ಲಕ್ಷ್ಮಿ                              →  ಲ  ಕ್ಷ್ಮಿ
+romanize  लक्ष्मी  ലക്ഷ്മി  லக்ஷ்மி              →  lakshmi
 ```
 
 Each utility gives the same output in Go and in TypeScript, and each was
@@ -23,6 +25,7 @@ measured on data that its rules were never built on.
 | let a search for हिन्दी also find हिंदी, and keep words whole | `fold` |
 | find a person's name typed in another script | `phonetic` |
 | count letters, cut at a length, or move a cursor | `segment` |
+| write a name or a text in Latin letters (लक्ष्मी → lakshmi) | `romanize` |
 
 ## Install
 
@@ -47,13 +50,20 @@ TypeScript, set `moduleResolution` to `nodenext` or `bundler`; the old
 In a browser, with no build step:
 
 ```js
-import { normalize } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.5.0/dist/normalize.js";
-import { stem } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.5.0/dist/stem.js";
-import { match } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.5.0/dist/phonetic.js";
-import { segment } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.5.0/dist/segment.js";
+import { normalize } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.6.0/dist/normalize.js";
+import { stem } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.6.0/dist/stem.js";
+import { match } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.6.0/dist/phonetic.js";
+import { segment } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.6.0/dist/segment.js";
+import { load } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.6.0/dist/romanize.js";
 ```
 
 Each utility is its own file, so a page loads only the rules it uses.
+`romanize` also loads one language's tables when you ask for them (0.2–1.8
+MB). The npm package carries no tables: `load()` reads them beside the
+module (a jsDelivr or GitHub-tag install), else from jsDelivr at the same
+version. Offline, pass your own `fetch`. In Go, `go get` downloads the
+tables of every language once (14 MB); a program carries only the languages
+it imports.
 
 ## Recipes
 
@@ -152,6 +162,33 @@ segment("ਕ੍ਕ"); // ["ਕ੍","ਕ"]: Gurmukhi shows this virama, so these
 count("ಕನ್ನಡ"); // 3
 ```
 
+### Write it in Latin letters
+
+`romanize` gives a ranked list of spellings, most likely first. Use the
+`"words"` mode for text and the `"names"` mode for a field that holds a
+person's name:
+
+```ts
+import { load } from "indickit/romanize";
+
+const hi = await load("hi");          // fetches the Hindi tables
+hi.word("लक्ष्मी", 2);                 // ["lakshmi","laxmi"]
+hi.text("भारत के प्रधानमंत्री");          // "bharat ke pradhaanmantri"
+const ur = await load("ur", "names");
+ur.word("پرویز", 1);                   // ["parvez"]
+```
+
+In Go, import the languages you need; a program carries only their tables:
+
+```go
+import (
+	"github.com/micahchoo/indickit/romanize"
+	_ "github.com/micahchoo/indickit/romanize/lang/hi"
+)
+
+romanize.Word("लक्ष्मी", "hi", romanize.Words, 2) // [lakshmi laxmi]
+```
+
 ### The same in Go
 
 ```go
@@ -182,6 +219,9 @@ segment.Segment("ಲಕ್ಷ್ಮಿ")                       // [ಲ ಕ್ಷ
 | `phonetic.Words(name)` | `words(name)` | the words, split as the others split them |
 | `segment.Segment(s)` | `segment(text)` | the letters, in order; joined, they give the text back |
 | `segment.Count(s)` | `count(text)` | how many letters a reader sees |
+| `romanize.Word(w, lang, mode, n)` | `(await load(lang, mode)).word(w, n?)` | up to n spellings, most likely first |
+| `romanize.Text(s, lang, mode)` | `(await load(lang, mode)).text(text)` | each word of the language's script in Latin letters |
+| `romanize.Languages(mode)` | `languages(mode)` | the languages with tables (Go: those imported) |
 
 Three things hold for all of them:
 
@@ -208,6 +248,7 @@ other tools tried and each weak spot are in `docs/`.
 | `stem` | finds 6–15 points more of the right sentences than exact search, in 13 languages | ahead in 5 languages, level in 2, a trade in 4, behind in Nepali | 3 KB | [docs/stem.md](docs/stem.md) |
 | `phonetic` | finds the right person 84% of the time, in 21 languages, on clean names; 42% on a real roster as written (70% with titles and initials removed) | romanize + Soundex: 23% in Tamil, 70% in Hindi; cannot read Urdu | 4 KB | [docs/phonetic.md](docs/phonetic.md) |
 | `segment` | cuts 0.05% of Kannada words wrongly | `Intl.Segmenter`: 50.3% | 9 KB | [docs/segment.md](docs/segment.md) |
+| `romanize` | writes 48.9% of the words of running text as people typed them, in 11 languages; 71.3% of names (tables: 0.2–1.8 MB a language) | IndicXlit (Python, a 119 MB model): 37.9% and 49.0% | 5 KB | [docs/romanize.md](docs/romanize.md) |
 
 (Browser files are gzipped.)
 
@@ -231,6 +272,9 @@ other tools tried and each weak spot are in `docs/`.
 - **Some languages have no table.** `stem` covers 13 languages; Bodo,
   Dogri, Kashmiri, Konkani, Maithili, Manipuri, Odia, Santali and Sindhi
   are not among them.
+- **`romanize` is behind IndicXlit on places and on rare single words**
+  (by 7–13 and 5.5 points). Santali has names only; Bodo and Dogri have
+  text only. See [docs/romanize.md](docs/romanize.md).
 - **Go reads invalid UTF-8 as U+FFFD**, so for such input the output does
   not join back into the original bytes.
 - **"The same output" holds inside the blocks indickit reads.** Go reads
@@ -249,10 +293,12 @@ Each utility's own weak spots are at the end of its file in `docs/`.
 
 Every table and switch of a utility is in its rules file
 (`normalize/rules.json`, `stem/rules.json`, `phonetic/rules.json`,
-`segment/rules.json`); the Go and TypeScript code is a short loop over it.
+`segment/rules.json`; `romanize/rules.json` and its tables in
+`romanize/lang/`); the Go and TypeScript code is a short loop over it.
 Both are checked against a conformance file of inputs with the outputs that
 a reference implementation gave them: 345,276 inputs for `normalize`,
-418,665 for `stem`, 195,994 for `phonetic`, 858,654 for `segment`
+418,665 for `stem`, 195,994 for `phonetic`, 858,654 for `segment`, 2,397
+for `romanize`
 (`*/testdata/conformance.jsonl.gz`). A change that makes any one disagree
 on any input fails the build.
 

package/dist/romanize.js

@@ -0,0 +1,743 @@
+// romanize/rules.json
+var rules_default = {
+  beam: {
+    names: 12,
+    words: 8
+  },
+  bos: "<s>",
+  eos: "</s>",
+  families: {
+    names: {
+      as: {
+        file: "as.names.bin",
+        group: "brahmic",
+        lookup: 669,
+        pooled: "names-brahmic.bin",
+        tag: "as"
+      },
+      bn: {
+        file: "bn.names.bin",
+        group: "brahmic",
+        lookup: 1785,
+        pooled: "names-brahmic.bin",
+        tag: "bn"
+      },
+      gom: {
+        file: "gom.names.bin",
+        group: "brahmic",
+        lookup: 17,
+        pooled: "names-brahmic.bin",
+        tag: "gom"
+      },
+      gu: {
+        file: "gu.names.bin",
+        group: "brahmic",
+        lookup: 131,
+        pooled: "names-brahmic.bin",
+        tag: "gu"
+      },
+      hi: {
+        file: "hi.names.bin",
+        group: "brahmic",
+        lookup: 1152,
+        pooled: "names-brahmic.bin",
+        tag: "hi"
+      },
+      kn: {
+        file: "kn.names.bin",
+        group: "brahmic",
+        lookup: 448,
+        pooled: "names-brahmic.bin",
+        tag: "kn"
+      },
+      ks: {
+        file: "ks.names.bin",
+        group: "arabic",
+        lookup: 130,
+        pooled: "names-arabic.bin",
+        tag: "ks"
+      },
+      mai: {
+        file: "mai.names.bin",
+        group: "brahmic",
+        lookup: 150,
+        pooled: "names-brahmic.bin",
+        tag: "mai"
+      },
+      ml: {
+        file: "ml.names.bin",
+        group: "brahmic",
+        lookup: 908,
+        pooled: "names-brahmic.bin",
+        tag: "ml"
+      },
+      mni: {
+        file: "mni.names.bin",
+        group: "meetei",
+        lookup: 253,
+        pooled: "names-meetei.bin",
+        tag: "mni"
+      },
+      mr: {
+        file: "mr.names.bin",
+        group: "brahmic",
+        lookup: 716,
+        pooled: "names-brahmic.bin",
+        tag: "mr"
+      },
+      ne: {
+        file: "ne.names.bin",
+        group: "brahmic",
+        lookup: 146,
+        pooled: "names-brahmic.bin",
+        tag: "ne"
+      },
+      or: {
+        file: "or.names.bin",
+        group: "brahmic",
+        lookup: 610,
+        pooled: "names-brahmic.bin",
+        tag: "or"
+      },
+      pa: {
+        file: "pa.names.bin",
+        group: "brahmic",
+        lookup: 996,
+        pooled: "names-brahmic.bin",
+        tag: "pa"
+      },
+      sa: {
+        file: "sa.names.bin",
+        group: "brahmic",
+        lookup: 66,
+        pooled: "names-brahmic.bin",
+        tag: "sa"
+      },
+      sat: {
+        file: "sat.names.bin",
+        group: "olchiki",
+        lookup: 407,
+        pooled: "names-olchiki.bin",
+        tag: "sat"
+      },
+      sd: {
+        file: "sd.names.bin",
+        group: "arabic",
+        lookup: 72,
+        pooled: "names-arabic.bin",
+        tag: "sd"
+      },
+      ta: {
+        file: "ta.names.bin",
+        group: "brahmic",
+        lookup: 3961,
+        pooled: "names-brahmic.bin",
+        tag: "ta"
+      },
+      te: {
+        file: "te.names.bin",
+        group: "brahmic",
+        lookup: 2126,
+        pooled: "names-brahmic.bin",
+        tag: "te"
+      },
+      ur: {
+        file: "ur.names.bin",
+        group: "arabic",
+        lookup: 1961,
+        pooled: "names-arabic.bin",
+        tag: "ur"
+      }
+    },
+    words: {
+      as: {
+        file: "as.words.bin",
+        group: "brahmic",
+        lookup: 0,
+        pooled: "words-brahmic.bin",
+        tag: "as:w"
+      },
+      bn: {
+        file: "bn.words.bin",
+        group: "brahmic",
+        lookup: 0,
+        pooled: "words-brahmic.bin",
+        tag: "bn:w"
+      },
+      brx: {
+        file: "brx.words.bin",
+        group: "brahmic",
+        lookup: 0,
+        pooled: "words-brahmic.bin",
+        tag: "brx:w"
+      },
+      doi: {
+        file: "doi.words.bin",
+        group: "brahmic",
+        lookup: 0,
+        pooled: "words-brahmic.bin",
+        tag: "doi:w"
+      },
+      gom: {
+        file: "gom.words.bin",
+        group: "brahmic",
+        lookup: 0,
+        pooled: "words-brahmic.bin",
+        tag: "gom:w"
+      },
+      gu: {
+        file: "gu.words.bin",
+        group: "brahmic",
+        lookup: 0,
+        pooled: "words-brahmic.bin",
+        tag: "gu:w"
+      },
+      hi: {
+        file: "hi.words.bin",
+        group: "brahmic",
+        lookup: 0,
+        pooled: "words-brahmic.bin",
+        tag: "hi:w"
+      },
+      kn: {
+        file: "kn.words.bin",
+        group: "brahmic",
+        lookup: 0,
+        pooled: "words-brahmic.bin",
+        tag: "kn:w"
+      },
+      ks: {
+        file: "ks.words.bin",
+        group: "arabic",
+        lookup: 0,
+        pooled: "words-arabic.bin",
+        tag: "ks:w"
+      },
+      mai: {
+        file: "mai.words.bin",
+        group: "brahmic",
+        lookup: 0,
+        pooled: "words-brahmic.bin",
+        tag: "mai:w"
+      },
+      ml: {
+        file: "ml.words.bin",
+        group: "brahmic",
+        lookup: 0,
+        pooled: "words-brahmic.bin",
+        tag: "ml:w"
+      },
+      mni: {
+        file: "mni.words.bin",
+        group: "meetei",
+        lookup: 0,
+        pooled: "words-meetei.bin",
+        tag: "mni:w"
+      },
+      mr: {
+        file: "mr.words.bin",
+        group: "brahmic",
+        lookup: 0,
+        pooled: "words-brahmic.bin",
+        tag: "mr:w"
+      },
+      ne: {
+        file: "ne.words.bin",
+        group: "brahmic",
+        lookup: 0,
+        pooled: "words-brahmic.bin",
+        tag: "ne:w"
+      },
+      or: {
+        file: "or.words.bin",
+        group: "brahmic",
+        lookup: 0,
+        pooled: "words-brahmic.bin",
+        tag: "or:w"
+      },
+      pa: {
+        file: "pa.words.bin",
+        group: "brahmic",
+        lookup: 0,
+        pooled: "words-brahmic.bin",
+        tag: "pa:w"
+      },
+      sa: {
+        file: "sa.words.bin",
+        group: "brahmic",
+        lookup: 0,
+        pooled: "words-brahmic.bin",
+        tag: "sa:w"
+      },
+      sd: {
+        file: "sd.words.bin",
+        group: "arabic",
+        lookup: 0,
+        pooled: "words-arabic.bin",
+        tag: "sd:w"
+      },
+      ta: {
+        file: "ta.words.bin",
+        group: "brahmic",
+        lookup: 0,
+        pooled: "words-brahmic.bin",
+        tag: "ta:w"
+      },
+      te: {
+        file: "te.words.bin",
+        group: "brahmic",
+        lookup: 0,
+        pooled: "words-brahmic.bin",
+        tag: "te:w"
+      },
+      ur: {
+        file: "ur.words.bin",
+        group: "arabic",
+        lookup: 0,
+        pooled: "words-arabic.bin",
+        tag: "ur:w"
+      }
+    }
+  },
+  files: {
+    "as.names.bin": 89974,
+    "as.words.bin": 235757,
+    "bn.names.bin": 182011,
+    "bn.words.bin": 377491,
+    "brx.words.bin": 208654,
+    "doi.words.bin": 48681,
+    "gom.names.bin": 9361,
+    "gom.words.bin": 324293,
+    "gu.names.bin": 38634,
+    "gu.words.bin": 370117,
+    "hi.names.bin": 162471,
+    "hi.words.bin": 468425,
+    "kn.names.bin": 73095,
+    "kn.words.bin": 273776,
+    "ks.names.bin": 36223,
+    "ks.words.bin": 445883,
+    "mai.names.bin": 40421,
+    "mai.words.bin": 381362,
+    "ml.names.bin": 133130,
+    "ml.words.bin": 342656,
+    "mni.names.bin": 12382,
+    "mni.words.bin": 133350,
+    "mr.names.bin": 119082,
+    "mr.words.bin": 327753,
+    "names-arabic.bin": 188775,
+    "names-brahmic.bin": 594055,
+    "names-meetei.bin": 43805,
+    "names-olchiki.bin": 97591,
+    "ne.names.bin": 37468,
+    "ne.words.bin": 297072,
+    "or.names.bin": 83867,
+    "or.words.bin": 285972,
+    "pa.names.bin": 140045,
+    "pa.words.bin": 419426,
+    "sa.names.bin": 24600,
+    "sa.words.bin": 243993,
+    "sat.names.bin": 73397,
+    "sd.names.bin": 11896,
+    "sd.words.bin": 463475,
+    "ta.names.bin": 295030,
+    "ta.words.bin": 244532,
+    "te.names.bin": 229077,
+    "te.words.bin": 295013,
+    "ur.names.bin": 192400,
+    "ur.words.bin": 540255,
+    "words-arabic.bin": 1182486,
+    "words-brahmic.bin": 2542643,
+    "words-meetei.bin": 198229,
+    "words-olchiki.bin": 61705
+  },
+  groups: {
+    arabic: [
+      1536,
+      1791
+    ],
+    brahmic: [
+      2304,
+      3455
+    ],
+    meetei: [
+      43744,
+      44031
+    ],
+    olchiki: [
+      7248,
+      7295
+    ]
+  },
+  max_n: 3,
+  order: 3,
+  rules_version: "2026-10-07",
+  unify: {
+    from: [
+      2432,
+      3455
+    ],
+    mask: 127,
+    to_block: 2304
+  },
+  unknown_penalty: 20,
+  version: 1
+};
+// package.json
+var version = "0.6.0";
+
+// js/lang.ts
+var ISO_639_2 = "asm as ben bn guj gu hin hi kan kn kas ks mal ml mar mr nep ne ori or pan pa san sa snd sd tam ta tel te urd ur ";
+function langCode(tag) {
+  const t = (tag ?? "").toLowerCase().split(/[-_]/)[0];
+  const i = t.length === 3 ? ISO_639_2.indexOf(t + " ") : -1;
+  return i >= 0 && i % 7 === 0 ? ISO_639_2.slice(i + 4, i + 6) : t;
+}
+
+// js/romanize.ts
+var RULES = rules_default;
+var RULES_VERSION = RULES.rules_version;
+function languages(mode = "words") {
+  return Object.keys(RULES.families[mode]).sort();
+}
+var CDN = `https://cdn.jsdelivr.net/gh/micahchoo/indickit@v${version}/romanize/lang/`;
+async function fetchBytes(url) {
+  const res = await fetch(url);
+  if (!res.ok)
+    throw new Error(`romanize: ${url}: ${res.status}`);
+  return res.arrayBuffer();
+}
+function defaultFetcher(local) {
+  return async (url) => {
+    try {
+      return await fetchBytes(url);
+    } catch {
+      return fetchBytes(new URL(url.href.slice(local.href.length), CDN));
+    }
+  };
+}
+function code(lang) {
+  const c = langCode(lang);
+  return c === "kok" ? "gom" : c;
+}
+var pooledCache = new Map;
+async function load(lang, mode = "words", opts = {}) {
+  const c = code(lang);
+  const l = RULES.families[mode][c];
+  if (!l)
+    throw new Error(`romanize: no ${mode} tables for "${lang}"`);
+  const base = new URL(opts.base ?? "../romanize/lang/", import.meta.url);
+  const get = opts.fetch ?? (opts.base ? fetchBytes : defaultFetcher(base));
+  const poolKey = `${base}${l.group}/${l.pooled}`;
+  if (!pooledCache.has(poolKey))
+    pooledCache.set(poolKey, get(new URL(`${l.group}/${l.pooled}`, base)));
+  const [own, pool] = await Promise.all([get(new URL(`${c}/${l.file}`, base)), pooledCache.get(poolKey)]);
+  return fromBytes(c, mode, new Uint8Array(own), new Uint8Array(pool));
+}
+function fromBytes(lang, mode, own, pool) {
+  const l = RULES.families[mode][lang];
+  if (!l)
+    throw new Error(`romanize: no ${mode} tables for "${lang}"`);
+  const x = new Mix(own, pool, l.tag, RULES.beam[mode]);
+  const [lo, hi] = RULES.groups[l.group];
+  const inScript = (ch) => {
+    const cp = ch.codePointAt(0);
+    return cp >= lo && cp <= hi || cp === 8204 || cp === 8205;
+  };
+  const letter = /[\p{L}\p{M}‌‍]/u;
+  const word = (w, n = 4) => {
+    const s = w.normalize("NFC");
+    const known = x.lookup.get(s);
+    if (known)
+      return known.slice(0, n);
+    return x.decode(unify(s), n).filter((o) => o !== "");
+  };
+  const r = {
+    lang,
+    mode,
+    word,
+    text(t) {
+      const cs = [...t.normalize("NFC")];
+      let out = "";
+      for (let i = 0;i < cs.length; ) {
+        if (!(inScript(cs[i]) && letter.test(cs[i]))) {
+          out += cs[i++];
+          continue;
+        }
+        let j = i;
+        while (j < cs.length && inScript(cs[j]) && letter.test(cs[j]))
+          j++;
+        const run = cs.slice(i, j).join("");
+        out += word(run, 1)[0] ?? run;
+        i = j;
+      }
+      return out;
+    }
+  };
+  Object.defineProperty(r, "_mix", { value: x });
+  return r;
+}
+function unify(w) {
+  const u = RULES.unify;
+  let out = "";
+  for (const ch of w) {
+    const cp = ch.codePointAt(0);
+    out += cp >= u.from[0] && cp <= u.from[1] ? String.fromCodePoint(u.to_block + (cp & u.mask)) : ch;
+  }
+  return out;
+}
+
+class Reader {
+  b;
+  pos = 4;
+  constructor(b) {
+    this.b = b;
+    if (b[0] !== 73 || b[1] !== 75 || b[2] !== 82 || b[3] !== 49)
+      throw new Error("romanize: not an IKR1 file");
+  }
+  varint() {
+    let v = 0;
+    let shift = 1;
+    for (;; ) {
+      const byte = this.b[this.pos++];
+      v += (byte & 127) * shift;
+      if (byte < 128)
+        return v;
+      shift *= 128;
+    }
+  }
+  str() {
+    const n = this.varint();
+    const s = DECODER.decode(this.b.subarray(this.pos, this.pos + n));
+    this.pos += n;
+    return s;
+  }
+  float64() {
+    const v = new DataView(this.b.buffer, this.b.byteOffset + this.pos, 8).getFloat64(0, true);
+    this.pos += 8;
+    return v;
+  }
+}
+var DECODER = new TextDecoder;
+var ID = 2 ** 21;
+function ctxKey(ctx, from) {
+  let k = 0;
+  for (let i = from;i < ctx.length; i++)
+    k = k * ID + (ctx[i] + 1);
+  return k;
+}
+
+class Model {
+  vocab;
+  levels = [];
+  constructor(r, strs, ids) {
+    this.vocab = r.varint();
+    for (let l = r.varint();l > 0; l--) {
+      const lv = { stats: new Map, counts: new Map };
+      for (let c = r.varint();c > 0; c--) {
+        const ctx = [];
+        for (let i = r.varint();i > 0; i--)
+          ctx.push(ids.id(strs[r.varint()]));
+        const key = ctxKey(ctx, 0);
+        const counts = new Map;
+        let total = 0;
+        let prev = 0;
+        for (let t = r.varint();t > 0; t--) {
+          prev += r.varint();
+          const k = r.varint();
+          counts.set(ids.id(strs[prev]), k);
+          total += k;
+        }
+        lv.stats.set(key, [total, counts.size]);
+        lv.counts.set(key, counts);
+      }
+      this.levels.push(lv);
+    }
+  }
+  stats(ctx) {
+    const s = { counts: [], total: [], wb: [] };
+    for (let n = 0;n < this.levels.length; n++) {
+      const key = ctxKey(ctx, ctx.length - n);
+      const st = this.levels[n].stats.get(key);
+      if (st && st[1] !== 0) {
+        s.counts[n] = this.levels[n].counts.get(key);
+        s.total[n] = st[0];
+        s.wb[n] = st[0] / (st[0] + st[1]);
+      }
+    }
+    return s;
+  }
+  prob(s, tok) {
+    let p = 1 / (this.vocab + 1);
+    for (let n = 0;n < this.levels.length; n++) {
+      const c = s.counts[n];
+      if (!c)
+        continue;
+      const lam = s.wb[n];
+      p = lam * (c.get(tok) ?? 0) / s.total[n] + (1 - lam) * p;
+    }
+    return p;
+  }
+}
+
+class Interner {
+  ids = new Map;
+  strs = [];
+  id(s) {
+    let v = this.ids.get(s);
+    if (v === undefined) {
+      v = this.strs.length;
+      this.ids.set(s, v);
+      this.strs.push(s);
+    }
+    return v;
+  }
+}
+function before(a, b) {
+  if (a.score !== b.score)
+    return a.score > b.score;
+  if (a.out !== b.out)
+    return a.out < b.out;
+  return a.idx < b.idx;
+}
+function top(cs, k) {
+  const best = [];
+  for (const c of cs) {
+    if (best.length === k && !before(c, best[k - 1]))
+      continue;
+    let lo = 0;
+    let hi = best.length;
+    while (lo < hi) {
+      const mid = lo + hi >> 1;
+      if (before(c, best[mid]))
+        hi = mid;
+      else
+        lo = mid + 1;
+    }
+    best.splice(lo, 0, c);
+    if (best.length > k)
+      best.pop();
+  }
+  return best;
+}
+
+class Mix {
+  beam;
+  a;
+  b;
+  lam;
+  start = [];
+  eos;
+  options = new Map;
+  lookup = new Map;
+  constructor(own, pool, tag, beam) {
+    this.beam = beam;
+    const ids = new Interner;
+    let r = new Reader(pool);
+    let strs = readStrings(r);
+    this.b = new Model(r, strs, ids);
+    r = new Reader(own);
+    strs = readStrings(r);
+    this.lam = r.float64();
+    this.a = new Model(r, strs, ids);
+    for (let k = r.varint();k > 0; k--) {
+      const native = r.str();
+      const sp = [];
+      for (let i = r.varint();i > 0; i--)
+        sp.push(strs[r.varint()]);
+      this.lookup.set(native, sp);
+    }
+    for (let i = 0;i < RULES.order - 2; i++)
+      this.start.push(ids.id(RULES.bos));
+    this.start.push(ids.id(`<s:${tag}>`));
+    this.eos = ids.id(RULES.eos);
+    const seen = new Set;
+    for (const m of [this.a, this.b]) {
+      for (const counts of m.levels[0].counts.values()) {
+        for (const t of counts.keys()) {
+          if (t === this.eos || seen.has(t))
+            continue;
+          seen.add(t);
+          const s = ids.strs[t];
+          const i = s.indexOf("|");
+          const chunk = s.slice(0, i);
+          const list = this.options.get(chunk) ?? [];
+          list.push({ tok: t, en: s.slice(i + 1) });
+          this.options.set(chunk, list);
+        }
+      }
+    }
+    for (const list of this.options.values())
+      list.sort((p, q) => p.en < q.en ? -1 : p.en > q.en ? 1 : 0);
+    if (this.start.length > 2 || ids.strs.length >= ID)
+      throw new Error("romanize: beyond the engine's limits");
+  }
+  logp(sa, sb, tok) {
+    const pa = Math.exp(Math.log(this.a.prob(sa, tok)));
+    const pb = Math.exp(Math.log(this.b.prob(sb, tok)));
+    return Math.log(this.lam * pa + (1 - this.lam) * pb);
+  }
+  decode(word, n) {
+    const w = [...word];
+    const h = RULES.order - 1;
+    const root = { score: 0, out: "", ctx: this.start };
+    const beams = Array.from({ length: w.length + 1 }, () => []);
+    beams[0].push({ score: 0, out: "", parent: null, tok: -1, idx: 0 });
+    const context = (c) => {
+      if (c.tok < 0)
+        return c.parent.ctx;
+      const ctx = [...c.parent.ctx, c.tok];
+      return ctx.slice(ctx.length - h);
+    };
+    for (let i = 0;i < w.length; i++) {
+      if (beams[i].length === 0)
+        continue;
+      for (const c of top(beams[i], this.beam)) {
+        const hy = i === 0 ? root : { score: c.score, out: c.out, ctx: context(c) };
+        let moved = false;
+        const sa = this.a.stats(hy.ctx);
+        const sb = this.b.stats(hy.ctx);
+        for (let k = 1;k <= RULES.max_n && i + k <= w.length; k++) {
+          const opts = this.options.get(w.slice(i, i + k).join(""));
+          if (!opts)
+            continue;
+          for (const o of opts) {
+            const next = beams[i + k];
+            next.push({ score: hy.score + this.logp(sa, sb, o.tok), out: hy.out + o.en, parent: hy, tok: o.tok, idx: next.length });
+            moved = true;
+          }
+        }
+        if (!moved) {
+          const next = beams[i + 1];
+          next.push({ score: hy.score - RULES.unknown_penalty, out: hy.out, parent: hy, tok: -1, idx: next.length });
+        }
+      }
+    }
+    const final = new Map;
+    for (const c of beams[w.length]) {
+      const ctx = c.parent === null ? this.start : context(c);
+      const s = c.score + this.logp(this.a.stats(ctx), this.b.stats(ctx), this.eos);
+      const v = final.get(c.out);
+      if (v === undefined || s > v)
+        final.set(c.out, s);
+    }
+    return [...final].sort((p, q) => p[1] !== q[1] ? q[1] - p[1] : p[0] < q[0] ? -1 : p[0] > q[0] ? 1 : 0).slice(0, n).map((e) => e[0]);
+  }
+}
+function readStrings(r) {
+  const strs = [];
+  for (let i = r.varint();i > 0; i--)
+    strs.push(r.str());
+  return strs;
+}
+function _decode(r, word, n) {
+  return r._mix.decode(word, n);
+}
+export {
+  unify,
+  load,
+  languages,
+  fromBytes,
+  _decode,
+  RULES_VERSION,
+  CDN
+};

package/dist/types/js/romanize.d.ts

@@ -0,0 +1,55 @@
+/**
+ * Indian-language text in Latin letters, the way people spell it:
+ * लक्ष्मी → lakshmi, മലൈക → malaika, پرویز → parvez.
+ *
+ * A ranked list: one word has several accepted spellings (Choudhury,
+ * Chowdhury), and the first is the most likely. Two modes: "words" for
+ * running text (the default) and "names" for fields known to hold person
+ * names, where a lookup of known names comes first.
+ *
+ * The tables are learned from data and loaded per language, on demand:
+ *
+ *     const r = await load("hi");        // fetches the Hindi and Brahmic files (beside this module, else jsDelivr)
+ *     r.word("लक्ष्मी");                   // ["lakshmi", ...]
+ *
+ * The output is deterministic: the same input gives the same list in Go and
+ * in TypeScript.
+ *
+ * @module
+ */
+/** The tables: "words" for running text, "names" for person names. */
+export type Mode = "words" | "names";
+/** Changes whenever any output changes: store it beside romanized text. */
+export declare const RULES_VERSION: string;
+/** The language codes that have tables, for one mode. */
+export declare function languages(mode?: Mode): string[];
+/** Fetches one data file's bytes. The default reads it beside this module. */
+export type Fetcher = (url: URL) => Promise<ArrayBuffer>;
+/** Where the tables are when they are not beside this module: jsDelivr, at this package's version. */
+export declare const CDN: string;
+/** One language's loaded tables. */
+export interface Romanizer {
+    readonly lang: string;
+    readonly mode: Mode;
+    /** Up to n spellings of one word (default 4), most likely first. */
+    word(word: string, n?: number): string[];
+    /** Every run of the language's script romanized (its top spelling); everything else kept. */
+    text(text: string): string;
+}
+/**
+ * Loads a language's tables for one mode: its own file and its script
+ * group's pooled file (shared by the group, fetched once). It reads them
+ * beside this module, else from jsDelivr at this package's version (the npm
+ * package carries no tables); `base` or `fetch` replaces both. Rejects when
+ * the language has no tables for the mode.
+ */
+export declare function load(lang: string, mode?: Mode, opts?: {
+    fetch?: Fetcher;
+    base?: URL | string;
+}): Promise<Romanizer>;
+/** Builds a Romanizer from a language's two files (for tests and offline use). */
+export declare function fromBytes(lang: string, mode: Mode, own: Uint8Array, pool: Uint8Array): Romanizer;
+/** Moves every Brahmic code point to the Devanagari block by its offset (rules.json "unify"). */
+export declare function unify(w: string): string;
+/** @internal The model's own list for a unified word (no lookup), for the conformance test. */
+export declare function _decode(r: Romanizer, word: string, n: number): string[];

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "indickit",
-  "version": "0.5.0",
+  "version": "0.6.0",
   "description": "Small, exact utilities for text in Indian languages: a phonetic key that matches a name across 21 Indian scripts and English, a normalizer that gives text that looks the same one encoding, a segmenter for the letters a reader sees, and a stemmer for search.",
   "license": "MIT",
   "packageManager": "bun@1.3.14",
@@ -25,6 +25,10 @@
     "./stem": {
       "types": "./dist/types/js/stem.d.ts",
       "default": "./dist/stem.js"
+    },
+    "./romanize": {
+      "types": "./dist/types/js/romanize.d.ts",
+      "default": "./dist/romanize.js"
     }
   },
   "files": [
@@ -32,7 +36,7 @@
   ],
   "sideEffects": false,
   "scripts": {
-    "build": "bun build js/phonetic.ts --outfile dist/phonetic.js --target browser --format esm && bun build js/normalize.ts --outfile dist/normalize.js --target browser --format esm && bun build js/segment.ts --outfile dist/segment.js --target browser --format esm && bun build js/stem.ts --outfile dist/stem.js --target browser --format esm && tsc -p tsconfig.json",
+    "build": "bun build js/phonetic.ts --outfile dist/phonetic.js --target browser --format esm && bun build js/normalize.ts --outfile dist/normalize.js --target browser --format esm && bun build js/segment.ts --outfile dist/segment.js --target browser --format esm && bun build js/stem.ts --outfile dist/stem.js --target browser --format esm && bun build js/romanize.ts --outfile dist/romanize.js --target browser --format esm && tsc -p tsconfig.json",
     "test": "bun test",
     "check-dist": "node js/check-dist.mjs"
   },