indickit 0.5.0 → 0.8.0

package/README.md

@@ -3,12 +3,16 @@
 Small, exact utilities for text in the languages of India, in Go and
 TypeScript. They fix the places where ordinary string code goes wrong on
 Indian text: a search that misses a word because it is spelled, encoded or
-inflected another way, and a letter count that cuts a letter in two.
+inflected another way, a letter count that cuts a letter in two, a name
+that must be written in Latin letters, and Latin typing that must be written
+in an Indian script.
 
 ```
 stem      किताब  किताबें  किताबों              →  किताब
 phonetic  राम  ರಾಮ  രാമ  ராம  رام  Ram       →  rn
 segment   ಲಕ್ಷ್ಮಿ                              →  ಲ  ಕ್ಷ್ಮಿ
+romanize  लक्ष्मी  ലക്ഷ്മി  லக்ஷ்மி              →  lakshmi
+deromanize  namaste                         →  नमस्ते
 ```
 
 Each utility gives the same output in Go and in TypeScript, and each was
@@ -22,7 +26,47 @@ measured on data that its rules were never built on.
 | let a search for किताब also find किताबों | `normalize`, then `stem` |
 | let a search for हिन्दी also find हिंदी, and keep words whole | `fold` |
 | find a person's name typed in another script | `phonetic` |
+| find a name in a long list, as people write it (titles, initials), or in running text | `phonetic-search` |
 | count letters, cut at a length, or move a cursor | `segment` |
+| write a name or a text in Latin letters (लक्ष्मी → lakshmi) | `romanize` |
+| write Latin typing in an Indian script (namaste → नमस्ते) | `deromanize` |
+
+## Languages
+
+The 22 languages of the Eighth Schedule. `normalize`, `fold`, `segment` and
+`phonetic` read every script below, so they serve every language; the
+others hold tables for some languages only.
+
+| Language | Code | Script | `stem` | `phonetic-search` | `romanize` | `deromanize` |
+|---|---|---|---|---|---|---|
+| Assamese | `as` | Bengali-Assamese | ✓ | ✓ | words, names | words, names |
+| Bengali | `bn` | Bengali-Assamese | ✓ | ✓ | words, names | words, names |
+| Bodo | `brx` | Devanagari | – | ✓ | words | words |
+| Dogri | `doi` | Devanagari | – | ✓ | words | words |
+| Gujarati | `gu` | Gujarati | ✓ | ✓ | words, names | words, names |
+| Hindi | `hi` | Devanagari | ✓ | ✓ | words, names | words, names |
+| Kannada | `kn` | Kannada | ✓ | ✓ | words, names | words, names |
+| Kashmiri | `ks` | Perso-Arabic | – | ✓ | words, names | words, names |
+| Konkani | `gom` | Devanagari | – | ✓ | words, names | words, names |
+| Maithili | `mai` | Devanagari | – | ✓ | words, names | words, names |
+| Malayalam | `ml` | Malayalam | ✓ | ✓ | words, names | words, names |
+| Manipuri | `mni` | Meetei Mayek | – | ✓ | words, names | words, names |
+| Marathi | `mr` | Devanagari | ✓ | ✓ | words, names | words, names |
+| Nepali | `ne` | Devanagari | ✓ | ✓ | words, names | words, names |
+| Odia | `or` | Odia | – | ✓ | words, names | words, names |
+| Punjabi | `pa` | Gurmukhi | ✓ | ✓ | words, names | words, names |
+| Sanskrit | `sa` | Devanagari | ✓ | ✓ | words, names | words, names |
+| Santali | `sat` | Ol Chiki | – | keys only | names | names |
+| Sindhi | `sd` | Perso-Arabic | – | ✓ | words, names | words, names |
+| Tamil | `ta` | Tamil | ✓ | ✓ | words, names | words, names |
+| Telugu | `te` | Telugu | ✓ | ✓ | words, names | words, names |
+| Urdu | `ur` | Perso-Arabic | ✓ | ✓ | words, names | words, names |
+
+"words" and "names" are the two modes of `romanize` and `deromanize`. In
+`deromanize`, the names mode of Bodo and Dogri uses their words tables.
+Santali has no words tables: no dataset of Santali words in Latin letters
+was found. `phonetic-search` in Santali ranks by the keys alone. Measured
+quality differs by language; see `docs/` for each utility.
 
 ## Install
 
@@ -42,18 +86,28 @@ deno add npm:indickit
 
 The package is ES modules only; `require()` works from Node 20.19. In
 TypeScript, set `moduleResolution` to `nodenext` or `bundler`; the old
-`node10` cannot find the four utilities.
+`node10` cannot find the utilities.
 
 In a browser, with no build step:
 
 ```js
-import { normalize } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.5.0/dist/normalize.js";
-import { stem } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.5.0/dist/stem.js";
-import { match } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.5.0/dist/phonetic.js";
-import { segment } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.5.0/dist/segment.js";
+import { normalize } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.8.0/dist/normalize.js";
+import { stem } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.8.0/dist/stem.js";
+import { match } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.8.0/dist/phonetic.js";
+import { segment } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.8.0/dist/segment.js";
+import { load } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.8.0/dist/romanize.js";
+import { loadSearch } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.8.0/dist/phonetic-search.js";
+import { load as loadDeromanize } from "https://cdn.jsdelivr.net/gh/micahchoo/indickit@v0.8.0/dist/deromanize.js";
 ```
 
 Each utility is its own file, so a page loads only the rules it uses.
+`romanize` also loads one language's tables when you ask for them (0.2–1.8
+MB), `deromanize` one language's tables and word list (0.3–3.1 MB), and
+`phonetic-search` one language's table (2–6 KB). The npm package carries no tables: `load()` reads them beside the
+module (a jsDelivr or GitHub-tag install), else from jsDelivr at the same
+version. Offline, pass your own `fetch`. In Go, `go get` downloads the
+tables of every language once (`romanize` 14 MB, `deromanize` 52 MB); a
+program carries only the languages it imports.
 
 ## Recipes
 
@@ -137,6 +191,26 @@ Equal keys mean "sounds alike", not "the same person". Before you merge two
 records, weigh the evidence: a shared rare name says more than a shared
 common one.
 
+### Search a list of names
+
+`phonetic-search` uses the keys to find candidates, and a scorer learned
+from data to rank them from 0 to 100. It finds names as people write them,
+with titles, initials and joined words, and names in running text.
+
+```ts
+import { loadSearch, THRESHOLDS } from "indickit/phonetic-search";
+
+const s = await loadSearch("hi"); // fetches the Hindi table (3 KB)
+const ix = s.index(["नरेश मोदी", "नरेंद्र मोदी", "मनमोहन सिंह"]);
+ix.search("Narendra Modi", THRESHOLDS.names); // [{ name: 1, score: 100 }]
+ix.search("Manmohan Singh", THRESHOLDS.names); // [{ name: 2, score: 96 }]
+```
+
+Use the threshold for your task: `THRESHOLDS.names` (74) for a list of
+people, `THRESHOLDS.villages` (72) for place names, `THRESHOLDS.text_10`
+(80) or `text_20` (70) with `s.index(words, "text")` for a name among the
+words of a text. A score is relative to the other candidates of its search.
+
 ### Count and cut letters
 
 A cursor, a backspace, a letter count and a cut at a length limit must
@@ -152,6 +226,61 @@ segment("ਕ੍ਕ"); // ["ਕ੍","ਕ"]: Gurmukhi shows this virama, so these
 count("ಕನ್ನಡ"); // 3
 ```
 
+### Write it in Latin letters
+
+`romanize` gives a ranked list of spellings, most likely first. Use the
+`"words"` mode for text and the `"names"` mode for a field that holds a
+person's name:
+
+```ts
+import { load } from "indickit/romanize";
+
+const hi = await load("hi");          // fetches the Hindi tables
+hi.word("लक्ष्मी", 2);                 // ["lakshmi","laxmi"]
+hi.text("भारत के प्रधानमंत्री");          // "bharat ke pradhaanmantri"
+const ur = await load("ur", "names");
+ur.word("پرویز", 1);                   // ["parvez"]
+```
+
+In Go, import the languages you need; a program carries only their tables:
+
+```go
+import (
+	"github.com/micahchoo/indickit/romanize"
+	_ "github.com/micahchoo/indickit/romanize/lang/hi"
+)
+
+romanize.Word("लक्ष्मी", "hi", romanize.Words, 2) // [lakshmi laxmi]
+```
+
+### Write Latin typing in an Indian script
+
+`deromanize` is the reverse: it gives a ranked list of native spellings for
+a Latin-typed word, most likely first. It writes every word it is given in
+the Indian script, English words too: in mixed text, find each word's
+language first.
+
+```ts
+import { load } from "indickit/deromanize";
+
+const hi = await load("hi");          // fetches the Hindi tables and word list
+hi.word("namaste", 1);                // ["नमस्ते"]
+hi.text("bharat ke pradhanmantri");   // "भारत के प्रधानमंत्री"
+const ur = await load("ur", "names");
+ur.word("ahmad", 1);                  // ["احمد"]
+```
+
+In Go:
+
+```go
+import (
+	"github.com/micahchoo/indickit/deromanize"
+	_ "github.com/micahchoo/indickit/deromanize/lang/hi"
+)
+
+deromanize.Word("namaste", "hi", deromanize.Words, 1) // [नमस्ते]
+```
+
 ### The same in Go
 
 ```go
@@ -165,6 +294,7 @@ import (
 normalize.Text("അവന്\u200d", "ml")               // "അവൻ"
 stem.Stem(normalize.Text("किताबों", "hi"), "hi") // किताब
 phonetic.Match("Mohanlal", "മോഹൻലാൽ")            // true
+phonetic.NewIndex(names, "hi", phonetic.ProfileNames).Search("Narendra Modi", phonetic.Thresholds["names"])
 segment.Segment("ಲಕ್ಷ್ಮಿ")                       // [ಲ ಕ್ಷ್ಮಿ]
 ```
 
@@ -180,8 +310,18 @@ segment.Segment("ಲಕ್ಷ್ಮಿ")                       // [ಲ ಕ್ಷ
 | `phonetic.NameKeys(name)` | `nameKeys(name)` | the keys of a whole name; index these |
 | `phonetic.Match(a, b)` | `match(a, b)` | same word count, and each pair of words shares a key |
 | `phonetic.Words(name)` | `words(name)` | the words, split as the others split them |
+| `phonetic.JoinedKeys(name)` | `joinedKeys(name)` | the keys of the name with its words joined (8+ classes) |
+| `phonetic.NewIndex(names, lang, profile)` | `(await loadSearch(lang)).index(names, profile?)` | names keyed once for searching |
+| `ix.Search(query, threshold)` | `ix.search(query, threshold?)` | the names scoring at least threshold, best first |
+| `phonetic.Score(query, names, lang, profile)` | `(await loadSearch(lang)).score(query, names, profile?)` | one score (0..100) per candidate |
 | `segment.Segment(s)` | `segment(text)` | the letters, in order; joined, they give the text back |
 | `segment.Count(s)` | `count(text)` | how many letters a reader sees |
+| `romanize.Word(w, lang, mode, n)` | `(await load(lang, mode)).word(w, n?)` | up to n spellings, most likely first |
+| `romanize.Text(s, lang, mode)` | `(await load(lang, mode)).text(text)` | each word of the language's script in Latin letters |
+| `romanize.Languages(mode)` | `languages(mode)` | the languages with tables (Go: those imported) |
+| `deromanize.Word(latin, lang, mode, n)` | `(await load(lang, mode)).word(latin, n?)` | up to n native spellings, most likely first |
+| `deromanize.Text(s, lang, mode)` | `(await load(lang, mode)).text(text)` | each run of Latin letters in the language's script |
+| `deromanize.Languages(mode)` | `languages(mode)` | the languages with tables (Go: those imported) |
 
 Three things hold for all of them:
 
@@ -206,18 +346,30 @@ other tools tried and each weak spot are in `docs/`.
 |---|---|---|---|---|
 | `normalize` | makes 88.1% of look-alike spellings equal, and changes the look of no word | Indic NLP Library: 82.4%, and changes 0.68% of words | 14 KB | [docs/normalize.md](docs/normalize.md) |
 | `stem` | finds 6–15 points more of the right sentences than exact search, in 13 languages | ahead in 5 languages, level in 2, a trade in 4, behind in Nepali | 3 KB | [docs/stem.md](docs/stem.md) |
-| `phonetic` | finds the right person 84% of the time, in 21 languages, on clean names; 42% on a real roster as written (70% with titles and initials removed) | romanize + Soundex: 23% in Tamil, 70% in Hindi; cannot read Urdu | 4 KB | [docs/phonetic.md](docs/phonetic.md) |
+| `phonetic` | finds the right person 84% of the time, in 21 languages, on clean names; 42% on a real roster as written (70% with titles and initials removed); with `phonetic-search`, 86% of a real roster as written and 85% of villages | romanize + Soundex: 23% in Tamil, 70% in Hindi; cannot read Urdu; ICU + fuzzy match: 78% and 65% | 5 KB | [docs/phonetic.md](docs/phonetic.md) |
 | `segment` | cuts 0.05% of Kannada words wrongly | `Intl.Segmenter`: 50.3% | 9 KB | [docs/segment.md](docs/segment.md) |
+| `romanize` | writes 48.9% of the words of running text as people typed them, in 11 languages; 71.3% of names (tables: 0.2–1.8 MB a language) | IndicXlit (Python, a 119 MB model): 37.9% and 49.0% | 5 KB | [docs/romanize.md](docs/romanize.md) |
+| `deromanize` | writes the right word first for 87.1% of the words of Hindi news typed in Latin, and in four for 97.4%; 69.7% of names in 19 languages (tables: 0.3–3.1 MB a language) | IndicXlit: 86.1%, 89.7% and 66.4%; behind on rare words out of context | 20 KB | [docs/deromanize.md](docs/deromanize.md) |
 
 (Browser files are gzipped.)
 
+`romanize` is also ahead of every tool tried that runs without a neural model. The
+best of them, indic-trans (about 200 MB), gives 34.7% on running text against
+51.2%, on the development data; it is ahead only in Urdu. See
+[docs/romanize.md](docs/romanize.md).
+
 ## Limits
 
+- **`deromanize` writes English words in the Indian script too.** About 8%
+  of the words of romanized Hinglish are English; find each word's language
+  first. On rare words out of context it is 10 points behind IndicXlit. See
+  [docs/deromanize.md](docs/deromanize.md).
 - **Only HarfBuzz was tested** for "looks the same" (`normalize`,
   `segment`). Windows and Apple draw text with their own engines.
-- **`phonetic` expects clean names.** Remove titles (Shri, Smt.) and
-  initials before you key a name. On one-word place names it returns many
-  wrong matches. See [docs/phonetic.md](docs/phonetic.md).
+- **The `phonetic` key expects clean names.** Remove titles (Shri, Smt.)
+  and initials before you key a name, or use `phonetic-search`. On one-word
+  place names the key alone returns many wrong matches. See
+  [docs/phonetic.md](docs/phonetic.md).
 - **Tamil is the weakest language** for `phonetic` (76%), and `stem` is
   behind Snowball's recall in Tamil.
 - **`stem` does not help passage search in Hindi or Punjabi.** Short
@@ -231,6 +383,9 @@ other tools tried and each weak spot are in `docs/`.
 - **Some languages have no table.** `stem` covers 13 languages; Bodo,
   Dogri, Kashmiri, Konkani, Maithili, Manipuri, Odia, Santali and Sindhi
   are not among them.
+- **`romanize` is behind IndicXlit on places and on rare single words**
+  (by 7–13 and 5.5 points). Santali has names only; Bodo and Dogri have
+  text only. See [docs/romanize.md](docs/romanize.md).
 - **Go reads invalid UTF-8 as U+FFFD**, so for such input the output does
   not join back into the original bytes.
 - **"The same output" holds inside the blocks indickit reads.** Go reads
@@ -249,20 +404,33 @@ Each utility's own weak spots are at the end of its file in `docs/`.
 
 Every table and switch of a utility is in its rules file
 (`normalize/rules.json`, `stem/rules.json`, `phonetic/rules.json`,
-`segment/rules.json`); the Go and TypeScript code is a short loop over it.
+`segment/rules.json`; `romanize/rules.json` and its tables in
+`romanize/lang/`; `deromanize/rules.json` and its tables and word lists in
+`deromanize/lang/`; `phonetic-search`'s tables in `phonetic/scorer/`); the
+Go and TypeScript code is a short loop over it.
 Both are checked against a conformance file of inputs with the outputs that
 a reference implementation gave them: 345,276 inputs for `normalize`,
-418,665 for `stem`, 195,994 for `phonetic`, 858,654 for `segment`
-(`*/testdata/conformance.jsonl.gz`). A change that makes any one disagree
+418,665 for `stem`, 195,994 for `phonetic`, 858,654 for `segment`, 2,397
+for `romanize`, 2,818 for `deromanize`, and 1,021 search rows for
+`phonetic-search` (`*/testdata/*conformance.jsonl.gz`). A change that makes any one disagree
 on any input fails the build.
 
 ## Credits and licence
 
 MIT. The test words are Wikidata labels (CC0). The rules were tuned on
 Wikidata and PIB Parallel (Press Information Bureau releases), and checked
-on AI4Bharat's Aksharantar and on Wikipedia text; none of them ships here.
+on AI4Bharat's Aksharantar and on Wikipedia text. No text of theirs ships
+here, except `deromanize`'s word counts (below).
 `normalize` and `segment` were checked in fonts by Google (Noto), Ek Type
 (Anek), SIL, SMC and others. `stem`'s endings were mined from PIB text;
 Wiktionary's inflection tables (CC BY-SA), Universal Dependencies
 treebanks and FLORES+ (CC BY-SA) were used only to measure them. It was
 compared with Snowball, Lucene and the Indic NLP Library.
+
+`romanize` and `deromanize` learned from AI4Bharat's Aksharantar (CC BY for
+its hand-collected pairs, CC0 for its mined pairs), Wikidata labels (CC0),
+and IndicXlit's romanizations (MIT) of frequent Wikipedia words.
+`deromanize`'s tables hold word lists: counts of the words of Wikipedia
+(text CC BY-SA 4.0) and, for Bodo and Dogri, of IndicCorp v2 (CC0). Both
+were measured against IndicXlit, on Google's Dakshina (CC BY-SA 4.0) and,
+for `deromanize`, on COMI-LINGUA (CC BY 4.0), used only to measure.