indickit 0.9.0 → 0.11.0

package/README.md

@@ -47,29 +47,31 @@ others hold tables for some languages only.
 |---|---|---|---|---|---|---|
 | Assamese | `as` | Bengali-Assamese | ✓ | ✓ | words, names | words, names |
 | Bengali | `bn` | Bengali-Assamese | ✓ | ✓ | words, names | words, names |
-| Bodo | `brx` | Devanagari | – | ✓ | words | words |
-| Dogri | `doi` | Devanagari | – | ✓ | words | words |
+| Bodo | `brx` | Devanagari | ✓ | ✓ | words | words |
+| Dogri | `doi` | Devanagari | ✓ | ✓ | words | words |
 | Gujarati | `gu` | Gujarati | ✓ | ✓ | words, names | words, names |
 | Hindi | `hi` | Devanagari | ✓ | ✓ | words, names | words, names |
 | Kannada | `kn` | Kannada | ✓ | ✓ | words, names | words, names |
-| Kashmiri | `ks` | Perso-Arabic | – | ✓ | words, names | words, names |
-| Konkani | `gom` | Devanagari | – | ✓ | words, names | words, names |
-| Maithili | `mai` | Devanagari | – | ✓ | words, names | words, names |
+| Kashmiri | `ks` | Perso-Arabic | ✓ | ✓ | words, names | words, names |
+| Konkani | `gom` | Devanagari | ✓ | ✓ | words, names | words, names |
+| Maithili | `mai` | Devanagari | ✓ | ✓ | words, names | words, names |
 | Malayalam | `ml` | Malayalam | ✓ | ✓ | words, names | words, names |
-| Manipuri | `mni` | Meetei Mayek, Bengali-Assamese | – | ✓ | words, names | words, names |
+| Manipuri | `mni` | Meetei Mayek, Bengali-Assamese | ✓ | ✓ | words, names | words, names |
 | Marathi | `mr` | Devanagari | ✓ | ✓ | words, names | words, names |
 | Nepali | `ne` | Devanagari | ✓ | ✓ | words, names | words, names |
-| Odia | `or` | Odia | – | ✓ | words, names | words, names |
+| Odia | `or` | Odia | ✓ | ✓ | words, names | words, names |
 | Punjabi | `pa` | Gurmukhi | ✓ | ✓ | words, names | words, names |
 | Sanskrit | `sa` | Devanagari | ✓ | ✓ | words, names | words, names |
-| Santali | `sat` | Ol Chiki | – | keys only | names | names |
-| Sindhi | `sd` | Perso-Arabic | – | ✓ | words, names | words, names |
+| Santali | `sat` | Ol Chiki | ✓ | keys only | names | names |
+| Sindhi | `sd` | Perso-Arabic | ✓ | ✓ | words, names | words, names |
 | Tamil | `ta` | Tamil | ✓ | ✓ | words, names | words, names |
 | Telugu | `te` | Telugu | ✓ | ✓ | words, names | words, names |
 | Urdu | `ur` | Perso-Arabic | ✓ | ✓ | words, names | words, names |
 
-"words" and "names" are the two modes of `romanize` and `deromanize`. In
-`deromanize`, the names mode of Bodo and Dogri uses their words tables.
+"words" and "names" are the two modes of `romanize` and `deromanize`;
+`romanize` has a third, "places", in all 22 languages (its own tables in
+nine, the words tables in the others). In `deromanize`, the names mode of
+Bodo and Dogri uses their words tables.
 Santali has no words tables: no dataset of Santali words in Latin letters
 was found. `phonetic-search` in Santali ranks by the keys alone. Measured
 quality differs by language; see `docs/` for each utility.
@@ -97,14 +99,14 @@ TypeScript, set `moduleResolution` to `nodenext` or `bundler`; the old
 In a browser, with no build step:
 
 ```js
-import { normalize } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.9.0/dist/normalize.js";
-import { stem } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.9.0/dist/stem.js";
-import { match } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.9.0/dist/phonetic.js";
-import { segment } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.9.0/dist/segment.js";
-import { load } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.9.0/dist/romanize.js";
-import { loadSearch } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.9.0/dist/phonetic-search.js";
-import { load as loadDeromanize } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.9.0/dist/deromanize.js";
-import { isWellFormed } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.9.0/dist/wellformed.js";
+import { normalize } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.11.0/dist/normalize.js";
+import { stem } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.11.0/dist/stem.js";
+import { match } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.11.0/dist/phonetic.js";
+import { segment } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.11.0/dist/segment.js";
+import { load } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.11.0/dist/romanize.js";
+import { loadSearch } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.11.0/dist/phonetic-search.js";
+import { load as loadDeromanize } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.11.0/dist/deromanize.js";
+import { isWellFormed } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.11.0/dist/wellformed.js";
 ```
 
 Each utility is its own file, so a page loads only the rules it uses.
@@ -151,29 +153,40 @@ brokenAt("क्ि"); // 2: a vowel sign after a virama
 
 ### Search words in their other forms
 
-Give every word of a document a key, and give the query the same key. The
-key for a word is `stem(normalize(word, lang), lang)`:
+Give every word of a document a key, and give the query the same key.
+`searchKey` normalizes, stems, then folds accepted spellings, in the order
+that was measured best:
 
 ```ts
-import { normalize } from "indickit/normalize";
-import { stem } from "indickit/stem";
-
-stem(normalize("किताबों", "hi"), "hi"); // "किताब"
-stem(normalize("ஆண்டில்", "ta"), "ta"); // "ஆண்ட"
-stem(normalize("ஆண்டு", "ta"), "ta"); // "ஆண்ட"
-stem(normalize("ಸರ್ಕಾರಕ್ಕೆ", "kn"), "kn"); // "ಸರ್ಕಾರ"
-stem(normalize("حکومتوں", "ur"), "ur"); // "حکومت"
+import { searchKey } from "indickit/search-key";
+
+searchKey("किताबों", "hi"); // "किताब"
+searchKey("हिन्दी", "hi"); // "हिंद"
+searchKey("हिंदी", "hi"); // "हिंद"
+searchKey("ஆண்டில்", "ta"); // "ஆண்ட"
+searchKey("ஆண்டு", "ta"); // "ஆண்ட"
+searchKey("ಸರ್ಕಾರಕ್ಕೆ", "kn"); // "ಸರ್ಕಾರ"
+searchKey("حکومتوں", "ur"); // "حکومت"
 ```
 
-In 13 languages, a search with stems finds 6–15 points more of the right
-sentences than an exact search. In a search engine over Wikipedia passages,
+In a search engine, index the key beside a trigram field (each word cut
+into its three-letter pieces), and add the two scores: over Wikipedia in
+Hindi, Bengali and Telugu, the two fields found 3.0 points more of the
+right passages than trigrams alone, and ranked them better. A setup for
+OpenSearch and Elasticsearch is in [docs/stem.md](docs/stem.md#in-a-search-engine).
+In Go: `searchkey.Key(word, lang)`.
+
+In 22 languages, a search with stems finds more of the right sentences
+than an exact search: 6–15 points in the 14 measured on FLORES+, 3.7–11.4
+in the 8 measured on BPCC. In a search engine over Wikipedia passages,
 it finds 2–11.5 points more in 10 of 13 test sets, and nothing more in
 Hindi or Punjabi. It also finds some wrong ones: a stem merges forms by
 their endings, not by their meaning. Do not add fuzzy matching to stems:
 together they find less than stems alone.
 
-You need only one of `stem` and `fold`. If you use both, stem first, then
-fold: `fold` before `stem` loses 3–10 points in Assamese. `fold` alone
+If you call the steps yourself, stem first, then fold: `fold` before
+`stem` loses 3–10 points in Assamese, and fold after stem finds 2.4 points
+more in Hindi. `fold` alone
 merges accepted spellings of one word (हिन्दी and हिंदी) and keeps the word
 whole, for a search that must not merge forms:
 
@@ -206,7 +219,8 @@ match("राम", "काम"); // false: Rām is not kām
 
 To search a list of names, store every key of each name, and look up every
 key of the query. A name can have more than one key: Tamil writes both k
-and h with க, so மோகன், Mohan, has two.
+and h with க, and a Tamil name ending in ன் is also keyed without it, so
+மோகன், Mohan, has four.
 
 ```ts
 import { nameKeys, RULES_VERSION } from "indickit/phonetic";
@@ -235,16 +249,17 @@ ix.search("Narendra Modi", THRESHOLDS.names); // [{ name: 1, score: 100 }]
 ix.search("Manmohan Singh", THRESHOLDS.names); // [{ name: 2, score: 96 }]
 ```
 
-Use the threshold for your task: `THRESHOLDS.names` (74) for a list of
-people, `THRESHOLDS.villages` (72) for place names, `THRESHOLDS.text_10`
-(80) or `text_20` (70) with `s.index(words, "text")` for a name among the
+Use the threshold for your task: `THRESHOLDS.names` (76) for a list of
+people, `THRESHOLDS.villages` (73) for place names, `THRESHOLDS.text_10`
+(81) or `text_20` (71) with `s.index(words, "text")` for a name among the
 words of a text. A score is relative to the other candidates of its search.
 
 ### Count and cut letters
 
 A cursor, a backspace, a letter count and a cut at a length limit must
 treat ಕ್ಷ್ಮಿ as one letter. `Intl.Segmenter` splits it in Kannada and
-Gurmukhi; `segment` keeps it whole.
+Gurmukhi; `segment` keeps it whole. Each letter is an akshara, the
+orthographic syllable of the script.
 
 ```ts
 import { segment, count } from "indickit/segment";
@@ -258,8 +273,8 @@ count("ಕನ್ನಡ"); // 3
 ### Write it in Latin letters
 
 `romanize` gives a ranked list of spellings, most likely first. Use the
-`"words"` mode for text and the `"names"` mode for a field that holds a
-person's name:
+`"words"` mode for text, the `"names"` mode for a field that holds a
+person's name, and the `"places"` mode for a place name:
 
 ```ts
 import { load } from "indickit/romanize";
@@ -269,6 +284,8 @@ hi.word("लक्ष्मी", 2);                 // ["lakshmi","laxmi"]
 hi.text("भारत के प्रधानमंत्री");          // "bharat ke pradhaanmantri"
 const ur = await load("ur", "names");
 ur.word("پرویز", 1);                   // ["parvez"]
+const ta = await load("ta", "places");
+ta.word("காஞ்சிபுரம்", 1);              // ["kanchipuram"]
 ```
 
 In Go, import the languages you need; a program carries only their tables:
@@ -322,8 +339,9 @@ ranked list the romanized index is as good and needs no second download.
 
 `deromanize` is the reverse: it gives a ranked list of native spellings for
 a Latin-typed word, most likely first. It writes every word it is given in
-the Indian script, English words too: in mixed text, find each word's
-language first.
+the Indian script, English words too. For Hindi typed with English words in
+it (Hinglish), load the English list and use `textMixed`: a word on the list
+stays as typed.
 
 ```ts
 import { load } from "indickit/deromanize";
@@ -333,6 +351,8 @@ hi.word("namaste", 1);                // ["नमस्ते"]
 hi.text("bharat ke pradhanmantri");   // "भारत के प्रधानमंत्री"
 const ur = await load("ur", "names");
 ur.word("ahmad", 1);                  // ["احمد"]
+const mixed = await load("hi", "words", { english: true });  // also fetches the English list (22 KB)
+mixed.textMixed("mera office band hai");  // "मेरा office बंद है"
 ```
 
 In Go:
@@ -346,6 +366,31 @@ import (
 deromanize.Word("namaste", "hi", deromanize.Words, 1) // [नमस्ते]
 ```
 
+### Match two Latin spellings of one word
+
+People type one word in many ways: "zindagi" and "jindagi", "bahut" and
+"bohot". To match them, write each in the Indian script and fold it; the
+result is a key for search or for removing duplicates:
+
+```ts
+import { load } from "indickit/deromanize";
+import { fold } from "indickit/normalize";
+
+const hi = await load("hi");
+const key = (w: string) => fold(hi.word(w, 1)[0] ?? w, "hi");
+key("zindagi") === key("jindagi"); // true: ज़िंदगी and जिंदगी fold to one
+key("bahut") === key("bohot");     // true: both are बहुत
+key("kahaan") === key("kahan");    // false: कहाँ and कहा
+```
+
+On words `deromanize` never trained on (Dakshina's lexicons, Bengali,
+Hindi, Marathi, Tamil, Telugu and Urdu), this key matches 30–67% of the
+pairs of spellings of one word and gives 3–20% of spellings the key of
+another word. Fuzzy matching (one edit) matches 28–65% and gives 28–51% a
+wrong match. Keys from all four spellings match 52–88%, at 16–46% wrong.
+Tamil is the weakest language. The `phonetic` key is not for this: it drops
+vowels, which carry a Latin-typed word.
+
 ### The same in Go
 
 ```go
@@ -375,9 +420,9 @@ segment.Segment("ಲಕ್ಷ್ಮಿ")                       // [ಲ ಕ್ಷ
 | `stem.Languages()` | `LANGUAGES` | the languages with a stem table |
 | `phonetic.Keys(word)` | `keys(word)` | the keys of one word |
 | `phonetic.NameKeys(name)` | `nameKeys(name)` | the keys of a whole name; index these |
-| `phonetic.Match(a, b)` | `match(a, b)` | same word count, and each pair of words shares a key |
+| `phonetic.Match(a, b)` | `match(a, b)` | titles dropped, the same word count, and each pair of words shares a key |
 | `phonetic.Words(name)` | `words(name)` | the words, split as the others split them |
-| `phonetic.JoinedKeys(name)` | `joinedKeys(name)` | the keys of the name with its words joined (8+ classes) |
+| `phonetic.JoinedKeys(name)` | `joinedKeys(name)` | the keys of the name with its words joined (6+ classes) |
 | `phonetic.NewIndex(names, lang, profile)` | `(await loadSearch(lang)).index(names, profile?)` | names keyed once for searching |
 | `ix.Search(query, threshold)` | `ix.search(query, threshold?)` | the names scoring at least threshold, best first |
 | `phonetic.Score(query, names, lang, profile)` | `(await loadSearch(lang)).score(query, names, profile?)` | one score (0..100) per candidate |
@@ -414,11 +459,11 @@ other tools tried and each weak spot are in `docs/`.
 | Utility | Result | Best other tool | Browser file | Details |
 |---|---|---|---|---|
 | `normalize` | makes 88.1% of look-alike spellings equal, and changes the look of no word | Indic NLP Library: 82.4%, and changes 0.68% of words | 16 KB | [docs/normalize.md](docs/normalize.md) |
-| `stem` | finds 6–15 points more of the right sentences than exact search, in 13 languages | ahead in 5 languages, level in 2, a trade in 4, behind in Nepali | 3 KB | [docs/stem.md](docs/stem.md) |
-| `phonetic` | finds the right person 84% of the time, in 21 languages, on clean names; 42% on a real roster as written (70% with titles and initials removed); with `phonetic-search`, 86% of a real roster as written and 85% of villages | romanize + Soundex: 23% in Tamil, 70% in Hindi; cannot read Urdu; ICU + fuzzy match: 78% and 65% | 7 KB | [docs/phonetic.md](docs/phonetic.md) |
+| `stem` | finds 3.7–15 points more of the right sentences than exact search, in 22 languages | ahead in 5 languages, level in 2, a trade in 4, behind in Nepali; no other stemmer for the other 10 | 4 KB | [docs/stem.md](docs/stem.md) |
+| `phonetic` | finds 89% of place-name pairs across scripts (OpenStreetMap, held out, rules 2026-10-11.7); 84% of people on clean names in 21 languages (rules 2026-10-05, a lower bound for the rules that ship); 88% of a second roster as written (titles, initials; rules 2026-10-11); with `phonetic-search`, 93% of places (search 2026-10-11.7) and 86% of a roster as written (search 2026-10-09) | romanize + Soundex: 71% of place pairs, 23% of people in Tamil; cannot read Urdu; ICU + fuzzy match: 74% of places | 8 KB | [docs/phonetic.md](docs/phonetic.md) |
 | `segment` | cuts 0.05% of Kannada words wrongly | `Intl.Segmenter`: 50.3% | 8 KB | [docs/segment.md](docs/segment.md) |
-| `romanize` | writes 48.9% of the words of running text as people typed them, in 11 languages; 71.3% of names (tables: 0.2–1.8 MB a language) | IndicXlit (Python, a 119 MB model): 37.9% and 49.0% | 7 KB | [docs/romanize.md](docs/romanize.md) |
-| `deromanize` | writes the right word first for 87.1% of the words of Hindi news typed in Latin, and in four for 97.4%; 69.7% of names in 19 languages (tables: 0.3–3.1 MB a language) | IndicXlit: 86.1%, 89.7% and 66.4%; behind on rare words out of context | 21 KB | [docs/deromanize.md](docs/deromanize.md) |
+| `romanize` | writes 48.9% of the words of running text as people typed them, in 11 languages; 71.3% of names (tables: 0.2–1.8 MB a language) | IndicXlit (Python, a 119 MB model): 37.9% and 49.0% | 8 KB | [docs/romanize.md](docs/romanize.md) |
+| `deromanize` | writes the right word first for 87.1% of the words of Hindi news typed in Latin, and in four for 97.4%; 69.7% of names in 19 languages (tables: 0.3–3.1 MB a language) | IndicXlit: 86.1%, 89.7% and 66.4%; behind on rare words out of context | 22 KB | [docs/deromanize.md](docs/deromanize.md) |
 | `wellformed` | agrees with HarfBuzz on 100% of 135,944 held-out PDF words (9,594 of them broken) and 2,146,974 new Wikipedia words | a check that the word starts with a combining sign: finds 84% of the broken words, with 4,667 false alarms | 6 KB | [docs/wellformed.md](docs/wellformed.md) |
 
 (Browser files are gzipped.)
@@ -430,32 +475,49 @@ best of them, indic-trans (about 200 MB), gives 34.7% on running text against
 
 ## Limits
 
-- **`deromanize` writes English words in the Indian script too.** About 8%
-  of the words of romanized Hinglish are English; find each word's language
-  first. On rare words out of context it is 10 points behind IndicXlit. See
-  [docs/deromanize.md](docs/deromanize.md).
+- **`deromanize` writes English words in the Indian script too**, unless
+  you use `textMixed` (Hindi only). Its English list finds 62% of the English
+  words of typed Hinglish and leaves 2.4% of the Hindi words in Latin, most
+  of them chat spellings ("to" for तो, "he" for है). On rare words out of
+  context it is 10 points behind IndicXlit. See
+  [docs/deromanize.md](docs/deromanize.md#english-in-hindi-text).
 - **Only HarfBuzz was tested** for "looks the same" (`normalize`,
   `segment`). Windows and Apple draw text with their own engines.
-- **The `phonetic` key expects clean names.** Remove titles (Shri, Smt.)
-  and initials before you key a name, or use `phonetic-search`. On one-word
-  place names the key alone returns many wrong matches. See
+- **The `phonetic` key reads names, not addresses.** It drops a name's
+  leading titles (Shri, Smt., श्रीमती, திரு) and reads one-letter initials
+  (M = एम) from rules 2026-10-11; a title in the middle of a name, or any
+  other honorific, stays a word. On one-word place names the key alone
+  returns many wrong matches: use `phonetic-search`. See
   [docs/phonetic.md](docs/phonetic.md).
 - **Tamil is the weakest language** for `phonetic` (76%), and `stem` is
-  behind Snowball's recall in Tamil.
+  behind Snowball's recall in Tamil: in Python or Java, Snowball's Tamil
+  stemmer finds 2.8 points more in passage search. A table that matched
+  its recall merged short Tamil words (இடம் "place" with இடையே "between")
+  and broke the wrong-results budget.
 - **`stem` does not help passage search in Hindi or Punjabi.** Short
   question words merge with other words (ਕਿਸ "which" with ਕਿਸਾਨ "farmer"),
   and names lose their ends (सीता → सी). In Hindi, Lucene's analyzer finds
   4.8 points more, from its spelling folds. See [docs/stem.md](docs/stem.md).
+- **`stem` damages some names.** In news text, 1.3–3.0% of person-name
+  tokens are swamped: most of their stem search's hits are other words
+  (करीना → कर finds करने). For a field of names, pass `minStem: 4`: it
+  halves that damage. See [docs/stem.md](docs/stem.md#options).
 - **An n-gram field can find more than `stem`** in passage search: in
   Malayalam, Kannada, Tamil, Bengali, Marathi and Hindi, indexing each
   word's three-letter pieces found 3–10 points more, at 2–3 times the
-  index size. See [docs/stem.md](docs/stem.md#in-a-search-engine).
-- **Some languages have no table.** `stem` covers 13 languages; Bodo,
-  Dogri, Kashmiri, Konkani, Maithili, Manipuri, Odia, Santali and Sindhi
-  are not among them.
-- **`romanize` is behind IndicXlit on places and on rare single words**
-  (by 7–13 and 5.5 points). Santali has names only; Bodo and Dogri have
-  text only. Manipuri in Bengali script loses syllables in the words mode;
+  index size. Index `stem` beside the trigram field, not instead of it: in
+  Hindi, Bengali and Telugu, both fields together found 3.0 points more
+  than trigrams alone and ranked better. See
+  [docs/stem.md](docs/stem.md#in-a-search-engine).
+- **Thin evidence for the newest tables.** Eight languages rest on one
+  test source (BPCC); Manipuri in Bengali script on 1,012 FLORES+
+  sentences. Santali gains little (3.7 points).
+- **`romanize` is behind IndicXlit on OpenStreetMap place names and on rare
+  single words**: on a held-out slice of OpenStreetMap the places mode is
+  right 44.4% of the time, IndicXlit 49.1% (it leads on village names, 57.6%
+  against 52.8%; Kannada has no places tables). Santali has names only; Bodo and Dogri have
+  text only. Bodo writes ख थ फ as kh th ph, as typists do, not as
+  Aksharantar's k t f. Manipuri in Bengali script loses syllables in the words mode;
   its tables were trained on Meetei Mayek. See [docs/romanize.md](docs/romanize.md).
 - **`normalize` keeps the direction marks LRM and RLM** (U+200E, U+200F).
   Text copied from the web carries them, and they hide a match on 0.04% of
@@ -504,8 +566,8 @@ Every table and switch of a utility is in its rules file
 Go and TypeScript code is a short loop over it.
 Both are checked against a conformance file of inputs with the outputs that
 a reference implementation gave them: 345,276 inputs for `normalize`,
-418,665 for `stem`, 195,994 for `phonetic`, 858,654 for `segment`, 2,397
-for `romanize`, 2,990 for `deromanize`, 32,279 for `wellformed`, and 1,021 search rows for
+418,665 for `stem`, 199,992 for `phonetic`, 858,654 for `segment`, 2,397
+for `romanize`, 2,990 for `deromanize`, 32,279 for `wellformed`, and 1,031 search rows for
 `phonetic-search` (`*/testdata/*conformance.jsonl.gz`). A change that makes any one disagree
 on any input fails the build.