indickit 0.10.0 → 0.12.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +389 -0
- package/README.md +128 -66
- package/dist/deromanize.js +1 -1
- package/dist/normalize.js +1 -1
- package/dist/phonetic-search.js +1 -1
- package/dist/phonetic.js +1 -1
- package/dist/romanize.js +1 -1
- package/dist/search-key.js +1 -0
- package/dist/segment.js +1 -1
- package/dist/stem-compounds.js +1 -0
- package/dist/stem.js +1 -1
- package/dist/types/js/deromanize.d.ts +12 -0
- package/dist/types/js/phonetic-engine.d.ts +17 -1
- package/dist/types/js/phonetic-search.d.ts +13 -3
- package/dist/types/js/phonetic.d.ts +6 -4
- package/dist/types/js/romanize.d.ts +20 -12
- package/dist/types/js/search-key.d.ts +14 -0
- package/dist/types/js/stem-compounds.d.ts +16 -0
- package/dist/types/js/stem.d.ts +12 -3
- package/package.json +12 -3
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,389 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
Each version is one Git tag `v<version>`, one npm version, and one Go
|
|
4
|
+
module version. The **Rules** line of an entry names the rules version of
|
|
5
|
+
each utility. When a utility's rules version changed, some inputs give a
|
|
6
|
+
different output than before: recompute what you stored from the old
|
|
7
|
+
version, and compare `RULES_VERSION` (or `SEARCH_VERSION`) to know which
|
|
8
|
+
rows. A version whose rules line equals the one before it gives every
|
|
9
|
+
input the same output.
|
|
10
|
+
|
|
11
|
+
## 0.12.0 (2026-10-12)
|
|
12
|
+
|
|
13
|
+
Rules: phonetic 2026-10-11.8, normalize 2026-10-12, segment 2026-10-11, stem 2026-10-12, romanize 2026-10-12, deromanize 2026-10-09.2, phonetic-search 2026-10-11.8
|
|
14
|
+
|
|
15
|
+
- Faster, same output: `deromanize` 2.0x in Go and 1.35x in TypeScript (a
|
|
16
|
+
Hindi word in Go: 11.0 -> 4.4 ms); `romanize` 1.5-2.0x in Go and 1.1x in
|
|
17
|
+
TypeScript (a Hindi sentence in Go: 31 -> 15 ms). The n-gram tables are
|
|
18
|
+
sorted arrays, not one map entry per pair, and the last letter builds only
|
|
19
|
+
the spellings that can be returned, best first.
|
|
20
|
+
- Faster, same output: `Search` 2.0x in Go and 2.4x in TypeScript, `NewIndex`
|
|
21
|
+
1.6x and 1.8x. A word's keys are computed once per call, and a pair of
|
|
22
|
+
words is scored once.
|
|
23
|
+
- Less memory, faster loads: a loaded Hindi deromanizer keeps 18 MB of heap
|
|
24
|
+
in Go (was 35 MB) and loads in 36 ms (was 111 ms); a romanizer keeps 15 MB
|
|
25
|
+
(was 29 MB) and loads in 40 ms (was 96 ms). The Go word lists are kept as
|
|
26
|
+
bytes. (0.11.0 already made TypeScript's tables flat: a Hindi deromanizer
|
|
27
|
+
130 -> 35 MB, a romanizer 194 -> 42 MB.) A test in each port fails if a
|
|
28
|
+
loaded language grows back past its bound.
|
|
29
|
+
- Changed output: phonetic 2026-10-11.7 → 2026-10-11.8, phonetic-search
|
|
30
|
+
follows. Recompute stored keys. The Perso-Arabic letter ھ (U+06BE) is read
|
|
31
|
+
as h: Sindhi writes its h with it (سنھا = Sinha, جوھي = Juhi), and the
|
|
32
|
+
reader skipped it. On Wikidata people (DEV) Sindhi rises from 78% to 83%,
|
|
33
|
+
Urdu, Kashmiri and Hindi by under a point, the wrong-match rate unchanged.
|
|
34
|
+
2,251 of the 199,992 test words (those with ھ) gain a key; every key of
|
|
35
|
+
0.11.0 is kept but for 20 of them, with ھ at the start or after a vowel,
|
|
36
|
+
where the h-less key goes (شاھ = Shah: c → ch); the one that matters is a
|
|
37
|
+
variant spelling of Chaudhry (چوھدری; the usual چودھری keeps its key).
|
|
38
|
+
- Changed output: stem rules 2026-10-11 → 2026-10-12, Kannada and Marathi
|
|
39
|
+
only. Kannada gains 6 endings (ದನ್ನು ಂದು ಿವೆ ಂತೆ ಾಗ ತ್ತ) and Marathi 10
|
|
40
|
+
(postpositions written solid: ंमध्ये बद्दल मुळे ंनी ंना द्वारे कडे ंचा िक
|
|
41
|
+
ान), chosen on PIB queries inside the wrong-hit budget those languages
|
|
42
|
+
had not spent; 6,001 of the 729,361 conformance rows change (kn 3,130,
|
|
43
|
+
mr 2,871). On PIB sentences of May 2021, which no rule was chosen on and
|
|
44
|
+
which were read once, Kannada finds 0.5 points more of the right
|
|
45
|
+
sentences (95%: +0.2 to +1.0) and Marathi 0.8 (+0.5 to +1.4), at 0.4
|
|
46
|
+
and 0.2 points of precision. Recompute stored stems in kn and mr.
|
|
47
|
+
- Changed output: normalize rules 2026-10-11 → 2026-10-12. Recompute stored
|
|
48
|
+
normalized text. "Looks the same" is now decided by every font we hold that
|
|
49
|
+
draws what `normalize` writes, 2 to 15 families a script compared by glyph
|
|
50
|
+
outline, not by Noto and Anek alone, and the joiner table and tree were
|
|
51
|
+
rebuilt with them. On the 12.9M words of the measurement sets, drawn in
|
|
52
|
+
those fonts, the words whose look changes fall from 54 to 17 (all malformed
|
|
53
|
+
input or a font's quirk); the held-out number (88.1%) is unchanged, and the
|
|
54
|
+
groups made equal move by under two points in Malayalam and Bengali (down)
|
|
55
|
+
and three in Perso-Arabic (up). Three faults fixed: khanda ta (ত্ + ZWJ → ৎ)
|
|
56
|
+
is not written where the ZWJ joins ত্ to a cluster with a left-side vowel
|
|
57
|
+
sign or a reph, or under a reph (উত্ক্ষেপন keeps its ZWJ); an old Malayalam
|
|
58
|
+
chillu before a vowel sign stays as typed; a ZWJ between a Malayalam
|
|
59
|
+
consonant and its vowel sign (ന + ZWJ + ു) is kept, which fonts with
|
|
60
|
+
traditional ligatures draw. Two more, found on web text no rule was chosen
|
|
61
|
+
on (IndicCorp v2, read once): the chillu k (ൿ) is no longer written (not in
|
|
62
|
+
Unicode's DoNotEmit list; Noto draws it apart), and no chillu is written
|
|
63
|
+
before a second virama; Malayalam words whose look changes there fall from
|
|
64
|
+
16,327 to 47 of 83M, with 91.8% of joiner groups made equal (Odia 80.5%).
|
|
65
|
+
Odia still loses a ZWNJ in some acronyms and a ZWJ after ଡ଼ + virama (310
|
|
66
|
+
of 922,000 words): docs/normalize.md. The browser file grows from 16 KB to
|
|
67
|
+
22 KB gzipped: the tree holds more contexts; `deromanize` (22 → 28 KB)
|
|
68
|
+
and `search-key` (19 → 26 KB) embed the same tree and grow with it.
|
|
69
|
+
- Changed output: romanize rules 2026-10-11.2 → 2026-10-12, the names mode
|
|
70
|
+
only (words and places are unchanged). The ten best spellings by the
|
|
71
|
+
alignment score are scored again with a letter 5-gram over the training
|
|
72
|
+
English spellings, and the best four of that order are returned: the rank
|
|
73
|
+
had English-side habits wrong (the schwa, ee against i, a lost h), and the
|
|
74
|
+
accepted spelling was usually already in the list. On development names
|
|
75
|
+
that neither tool's training holds, the first spelling is right 51.7% of
|
|
76
|
+
the time, was 47.0% (IndicXlit 39.7%); any of four 72.8%, was 70.4%. On
|
|
77
|
+
names held out to the end (Wikidata people of Bangladesh, Pakistan, Nepal
|
|
78
|
+
and Sri Lanka, read once, 10,773 unseen tokens) the first spelling goes from
|
|
79
|
+
37.9% to 41.0%, no language falling. Of the 1,008 names rows of the
|
|
80
|
+
conformance file, 83 change their first spelling and 794 their order.
|
|
81
|
+
`rules.json` gains `rescore`; the model file `lang/shared/names-lm.bin`
|
|
82
|
+
(148 KB brotli) is loaded with every names page, in Go through the
|
|
83
|
+
language package. The names lookup now holds the names the shipped beam
|
|
84
|
+
spells otherwise (13,495 entries, was 16,632 at the reference's beam).
|
|
85
|
+
TypeScript `fromBytes` takes the letter model as a fifth argument for the
|
|
86
|
+
names mode.
|
|
87
|
+
|
|
88
|
+
## 0.11.0 (2026-10-12)
|
|
89
|
+
|
|
90
|
+
Rules: phonetic 2026-10-11.7, normalize 2026-10-11, segment 2026-10-11, stem 2026-10-11, romanize 2026-10-11.2, deromanize 2026-10-09.2, phonetic-search 2026-10-11.7
|
|
91
|
+
|
|
92
|
+
- Changed output: normalize rules 2026-10-07 → 2026-10-11, Tamil only. A
|
|
93
|
+
ZWNJ or ZWJ between consonant + pulli and a consonant is deleted, unless
|
|
94
|
+
the pair is one of the three Tamil conjuncts (க்ஷ, ஶ்ர, ஸ்ர): ப் + ZWNJ + ப →
|
|
95
|
+
ப்ப. Before, a class-based rule kept most of these and deleted the ZWNJ of
|
|
96
|
+
க்ஷ in some words, a change a reader could see. On IndicCorp Tamil text,
|
|
97
|
+
look-alike groups with a joiner made equal: 89.6% → 98.8%. `fold` follows.
|
|
98
|
+
`rules.json` gains `pairs`; the ports read it.
|
|
99
|
+
- Changed output: segment rules 2026-10-06 → 2026-10-11, Odia only. ଓ +
|
|
100
|
+
virama + ବ is one letter (ଓ୍ବି in ଓ୍ବିକେଟ୍ "wicket"); Odia news words split
|
|
101
|
+
wrongly: 0.13% → 0.04%. ଓ୍ୱ stays two letters (two fonts draw it apart).
|
|
102
|
+
- Docs: normalize's Malayalam limits (the most possible is 71.0% in 15
|
|
103
|
+
fonts; a ZWJ before a vowel sign; Suruma lacks the atomic chillus).
|
|
104
|
+
- Added: deromanize `textMixed` / Go `TextMixed`, and `isEnglish` / Go
|
|
105
|
+
`English`, for Hindi typed with English words in it (Hinglish): a word on
|
|
106
|
+
the Hindi English list (10,000 English Wikipedia words, less those a Hindi
|
|
107
|
+
typist writes often) stays as typed. TypeScript fetches the list (22 KB
|
|
108
|
+
gzipped) only with `load("hi", "words", { english: true })`. On
|
|
109
|
+
L3Cube-HingLID, read once, 87.4% of words are called right (IndicLID
|
|
110
|
+
roman: 72.1%); 2.4% of Hindi words stay in Latin. Every other output is
|
|
111
|
+
unchanged, so the deromanize rules version is unchanged.
|
|
112
|
+
- Added: romanize mode `"places"` (Go `romanize.Places`), rules
|
|
113
|
+
2026-10-11 → 2026-10-11.2, for place names. Own tables for bn hi ml mr or
|
|
114
|
+
sat ta te ur, learned from Wikidata place names and, for hi mr or, village
|
|
115
|
+
names of the Local Government Directory; the words tables in the other
|
|
116
|
+
languages. Held out, read once, the training's own names left out: village
|
|
117
|
+
names 57.6% first spelling right (words mode 48.0%, IndicXlit 52.8%);
|
|
118
|
+
OpenStreetMap place names 44.4% (40.6%, IndicXlit 49.1%). The words and
|
|
119
|
+
names modes are unchanged since 2026-10-11.
|
|
120
|
+
- Changed output: romanize rules 2026-10-09 → 2026-10-11, words mode.
|
|
121
|
+
Bodo writes ख थ फ as kh th ph, as Bodo speakers type them (थाखाय →
|
|
122
|
+
thakhai, was takai), not as Aksharantar's k t f. On typed Bodo sentences
|
|
123
|
+
(Bhasha-Abhijnaanam, read once) the first spelling is right 71.9% of the
|
|
124
|
+
time, was 50.3% (IndicXlit 49.3%); on Aksharantar's word list it falls
|
|
125
|
+
from 75.6% to 45.6%. The shared Brahmic table was retrained, so a few
|
|
126
|
+
lower-ranked spellings in other languages change too (61 of 1,018 words
|
|
127
|
+
conformance rows; no language's first spelling moved by more than 0.2
|
|
128
|
+
points). The names mode is unchanged.
|
|
129
|
+
|
|
130
|
+
- Changed output: stem rules 2026-10-09 → 2026-10-11, Sanskrit only. 17
|
|
131
|
+
endings added (the anusvara spellings ानां ां ायां, the passive यते यन्ते,
|
|
132
|
+
the infinitive तुं, the absolutive त्वा, and others); 1,301 of the 418,730
|
|
133
|
+
conformance rows change, all Sanskrit. On lemmas of the Digital Corpus of
|
|
134
|
+
Sanskrit, read once, recall rises from 34.6% to 40.5% at 2.7% wrong hits.
|
|
135
|
+
- Docs: `stem` damages some names (README Limits, `docs/stem.md#names`).
|
|
136
|
+
- Changed output: phonetic 2026-10-07 → 2026-10-11.7. Recompute stored keys.
|
|
137
|
+
`Keys` gives more keys to some words; every key a word had in 0.10.0 it
|
|
138
|
+
still has (checked on every test word, but two long Urdu words whose
|
|
139
|
+
readings exceed the cap of 32 keys a word, was 16), so every pair 0.10.0
|
|
140
|
+
matched still matches. On held-out OpenStreetMap place names the key
|
|
141
|
+
finds 89% of name pairs (romanize + Soundex: 71%).
|
|
142
|
+
- Bengali-script ং and Devanagari ं also read as ŋ: সিং = Singh (Bengali
|
|
143
|
+
and Assamese), and Bodo spelling (दं = dong; Bodo running text 74% →
|
|
144
|
+
86% of words).
|
|
145
|
+
- Tamil ட்ச also reads as க்ஷ: லட்சுமி = Lakshmi. A Tamil name ending in
|
|
146
|
+
ன் or ர் is also keyed without it; a Tamil word starting with a vowel
|
|
147
|
+
also keys with h (உசைன் = Hussain). Latin zh is ழ (Kanimozhi);
|
|
148
|
+
Malayalam റ്റ is also tt and ന്റ also nt (ആന്റണി = Antony). Tamil +1.6 and
|
|
149
|
+
Malayalam +1.9 points on Wikidata (DEV); on BPCC name pairs, read once,
|
|
150
|
+
Tamil +0.9 and Malayalam +1.5.
|
|
151
|
+
- Assamese: ৱ after a vowel (not before i) is also a glide (গোৱেল =
|
|
152
|
+
Goyal), and the case endings ৰ, ৱে, লৈ, ৰপৰা are also read off.
|
|
153
|
+
- A one-letter Latin word also keys as its spoken name: M = em = एम =
|
|
154
|
+
எம். M and N, and H and S, now share a key as initials.
|
|
155
|
+
- Santali: an initial ᱚ is also a (Ol Chiki writes the Indic short a
|
|
156
|
+
with it: Arjun = ᱚᱨᱡᱩᱱ), and ᱣ ᱶ after a vowel is also a glide;
|
|
157
|
+
Santali 74.1% → 84.7% on Wikidata (DEV).
|
|
158
|
+
- Sindhi in Devanagari: the implosives ॻ ॼ ॾ ॿ are read (they had no
|
|
159
|
+
key); व्ह is also v (चव्हाण = Chavan); the long Marathi / Konkani case
|
|
160
|
+
endings (च्या ांच्या मध्ये ांना), the Konkani -ान -ीन -ाईन -ाक -ाचो -ाची
|
|
161
|
+
-ाचें, the Sanskrit -स्य and the Sindhi / Bodo -नि are also read off. On
|
|
162
|
+
BPCC name pairs, read once, +0.49 points (Bodo +3.5).
|
|
163
|
+
- Urdu, Kashmiri and Sindhi script: an initial alif or ع is also i, e or u
|
|
164
|
+
(اندر = Indra, عمران = Imran, which did not match before), and ہ ح
|
|
165
|
+
after a consonant is also silent (جہانگیانی = Jhangiani). Urdu +4.8,
|
|
166
|
+
Kashmiri +4.3 points on Wikidata (DEV); on BPCC name pairs, read once,
|
|
167
|
+
Urdu +3.2. Latin zh is ழ's l or z + h, so both Kanimozhi and Mazhar
|
|
168
|
+
(مظہر) match.
|
|
169
|
+
- Tamil's இ or உ before an initial r or l is also dropped (இராஜ்பால் =
|
|
170
|
+
Rajpal); व after a consonant and before u is also the au glide (கவுர் =
|
|
171
|
+
Kaur); ड ढ after the first letter are also r, the flap written without its nukta
|
|
172
|
+
(राबडी = Rabri). Tamil +3.8 points on Wikidata (DEV).
|
|
173
|
+
- Changed output: `NameKeys`, `JoinedKeys` and `Match` (`nameKeys`,
|
|
174
|
+
`joinedKeys`, `match`) drop the leading titles (Shri, Smt., Dr, श्रीमती,
|
|
175
|
+
திரு, …; `titles` in rules.json) of each comma-separated part of a name:
|
|
176
|
+
"Bajwa, Sardar Partap Singh" reads as Bajwa Partap Singh. A name of titles
|
|
177
|
+
only keeps its last word. `Words` still splits only. A second roster as
|
|
178
|
+
written, the Rajya Sabha: the key finds 54.8% → 87.7%.
|
|
179
|
+
- Changed output: `JoinedKeys` (`joinedKeys`) returns keys of 6 classes
|
|
180
|
+
or more (`JoinedMinClasses`, was 8): a name written as one word in one
|
|
181
|
+
script and as two in another meets more often (Amartya Sen =
|
|
182
|
+
अमर्त्यसेन). +1.6 points on Wikidata (DEV), +2.7 on OpenStreetMap places
|
|
183
|
+
(read once).
|
|
184
|
+
- Changed output: phonetic search 2026-10-09 → 2026-10-11.7. It keys with
|
|
185
|
+
phonetic rules 2026-10-11.7, drops titles as `NameKeys` does, and its
|
|
186
|
+
thresholds were chosen again on the same data: names 74 → 76, villages
|
|
187
|
+
72 → 73, text 80 → 81 and 70 → 71 (`Thresholds` / `THRESHOLDS`). On held-out
|
|
188
|
+
OpenStreetMap places it finds 93% of places in five languages (ICU +
|
|
189
|
+
fuzzy match: 74%), at about 3 wrong places per search per 10,000 names.
|
|
190
|
+
- Size: the title list, the letter names and the new rules add about
|
|
191
|
+
1.3 KB gzipped to `phonetic`: 6.9 → 8.2 KB. `phonetic-search` falls
|
|
192
|
+
17.0 → 14.5 KB: its 2^x tables are stored as second differences, and its
|
|
193
|
+
two fine rules files as patches over `rules.json` (every score unchanged).
|
|
194
|
+
- Changed output: stem, in the same rules 2026-10-11. Gujarati, Marathi,
|
|
195
|
+
Maithili and Santali run their endings twice, as Kannada does; Maithili
|
|
196
|
+
and Santali gain 7 endings each. On text no rule saw, read once:
|
|
197
|
+
Gujarati +1.7 and Marathi +1.6 points of recall (PIB), Maithili +0.95
|
|
198
|
+
(BPCC), Santali +0.56 (FLORES+); precision falls by at most 1.3 points
|
|
199
|
+
against the old output. About 12,900 conformance rows change, in these
|
|
200
|
+
four languages only.
|
|
201
|
+
- `stem`: Sindhi in Perso-Arabic and Manipuri in Bengali script get their
|
|
202
|
+
own endings in the same tables, chosen on FLORES+ and checked once on its
|
|
203
|
+
unseen half (Manipuri: 5.6 points more of the right sentences than exact
|
|
204
|
+
search). Sindhi in Perso-Arabic then gained more endings, checked once on
|
|
205
|
+
the unseen test part of the Sindhi UD treebank: the share of inflected
|
|
206
|
+
forms that reach their word's stem rose from 20.2% to 33.0%, level with
|
|
207
|
+
Snowball's Arabic stemmer, with fewer wrong merges. Words in those
|
|
208
|
+
scripts were returned unchanged before; no Devanagari or Meetei Mayek
|
|
209
|
+
output changes.
|
|
210
|
+
- New: `indickit/search-key` (Go `searchkey.Key`), one search key per word:
|
|
211
|
+
normalize, stem, then fold, the order measured best. Fold after stem
|
|
212
|
+
finds 2.4 points more of the right passages in Hindi (MIRACL) and changes
|
|
213
|
+
nothing in Bengali or Telugu. The README's search recipe now uses it, and
|
|
214
|
+
`docs/stem.md` gives an OpenSearch/Elasticsearch setup with a trigram
|
|
215
|
+
field beside the keys.
|
|
216
|
+
- New in `stem`: two options. `minStem` (Go `StemWith`, `Options.MinStem`)
|
|
217
|
+
keeps stems longer, for fields of names; 4 halves the names a search
|
|
218
|
+
loses. `indickit/stem-compounds` (Go `CompoundKey`), a separate 1.6 KB
|
|
219
|
+
file, keys a Malayalam word by its first compound member; on held-out
|
|
220
|
+
news it finds 5.5 points more than `stem` alone. The default output does
|
|
221
|
+
not change.
|
|
222
|
+
- New in `stem`: Bodo, Dogri, Kashmiri, Konkani, Maithili, Manipuri
|
|
223
|
+
(Meetei Mayek), Santali and Sindhi (Devanagari), in the same rules
|
|
224
|
+
2026-10-11: `stem` now covers all 22 languages. Endings chosen on BPCC's
|
|
225
|
+
human translations; on a part no rule saw, 3.7–11.4 points more of the
|
|
226
|
+
right sentences than exact search, at most 4.4 points less precise.
|
|
227
|
+
About 254,000 conformance rows are new; none changed. The browser file
|
|
228
|
+
grows to 4 KB gzipped.
|
|
229
|
+
- Language tags: `ory` (Odia) and `npi` (Nepali), as FLORES+ and NLLB
|
|
230
|
+
write them, now name the `or` and `ne` tables.
|
|
231
|
+
- New in `stem`: Odia (`or`), in the same rules 2026-10-11. 11 endings
|
|
232
|
+
chosen on PIB search; on FLORES+, read once, 6.9 points more of the right
|
|
233
|
+
sentences than exact search, 3.4 points less precise. Odia's 24,578
|
|
234
|
+
conformance rows are new.
|
|
235
|
+
|
|
236
|
+
## 0.10.0 (2026-10-09)
|
|
237
|
+
|
|
238
|
+
Rules: phonetic 2026-10-07, normalize 2026-10-07, segment 2026-10-06, stem 2026-10-09, romanize 2026-10-09, deromanize 2026-10-09.2, phonetic-search 2026-10-09
|
|
239
|
+
|
|
240
|
+
- Changed output: phonetic search 2026-10-08 → 2026-10-09, for language
|
|
241
|
+
tags only. `NewIndex` and `Score` (Go), `loadSearch` (TypeScript) took
|
|
242
|
+
a bare code (`hi`) where every other utility takes a tag: `hi-IN`,
|
|
243
|
+
`HI` and `hin` found no Latin table and scored by the fine keys alone.
|
|
244
|
+
They now read the tag as the other utilities do. The tables are
|
|
245
|
+
unchanged: every score for a bare code is the same.
|
|
246
|
+
- Changed output: `kok` (the Konkani macrolanguage) names the `gom`
|
|
247
|
+
tables in every utility's tag reading. romanize and deromanize did
|
|
248
|
+
this already; stem and normalize hold no Konkani table, so nothing
|
|
249
|
+
changes there.
|
|
250
|
+
- Removed exports: TypeScript `_internal` (phonetic) and `unify`
|
|
251
|
+
(romanize), which no documented use needed. The phonetic engine now
|
|
252
|
+
lives in `js/phonetic-engine.ts`, shared by `phonetic` and
|
|
253
|
+
`phonetic-search`, not an entry of the package.
|
|
254
|
+
- Fixed: `phonetic.NewIndex` and `Score` (Go), `index` and `score`
|
|
255
|
+
(TypeScript) given a profile name other than `names` or `text` crashed
|
|
256
|
+
later with no cause named (Go: divide by zero in the first `Search`;
|
|
257
|
+
TypeScript: a `TypeError`). They now fail at once, naming the profile.
|
|
258
|
+
- Fixed: the search index built every combination of a name's word keys
|
|
259
|
+
before cutting to 256 (a 26-word name of two-key words: 2^26 strings,
|
|
260
|
+
8 s in Go, 4.9 s in TypeScript). It now builds the first 256 only, as
|
|
261
|
+
`NameKeys` does; the keys are the same.
|
|
262
|
+
- Go `stem` checks at load that `max_end + min_stem` in rules.json is
|
|
263
|
+
below its fixed walk (16): a larger value would have cut differently
|
|
264
|
+
from the reference, silently.
|
|
265
|
+
|
|
266
|
+
## 0.9.0 (2026-10-09)
|
|
267
|
+
|
|
268
|
+
Rules: phonetic 2026-10-07, normalize 2026-10-07, segment 2026-10-06, stem 2026-10-09, romanize 2026-10-09, deromanize 2026-10-09.2, phonetic-search 2026-10-08
|
|
269
|
+
|
|
270
|
+
- New: `wellformed`, a check that a word is drawn as written
|
|
271
|
+
(`indickit/wellformed`).
|
|
272
|
+
- Changed output: stem rules 2026-10-06 → 2026-10-09. Kannada applies its
|
|
273
|
+
stem twice (ಪ್ರಧಾನಮಂತ್ರಿಯವರು → ಪ್ರಧಾನಮಂತ್ರಿ): on held-out PIB text,
|
|
274
|
+
recall +0.86 points, precision −0.75. No other language changes.
|
|
275
|
+
- Changed output: romanize rules 2026-10-07 → 2026-10-08. A ZWJ or ZWNJ
|
|
276
|
+
inside a word no longer cuts the word (प्राप्\u200dत gave "the"). Manipuri
|
|
277
|
+
in Bengali script is romanized from the Brahmic tables (রামেন gave "en");
|
|
278
|
+
the `meetei` table group is gone, and `words-brahmic.bin` grows from
|
|
279
|
+
2,543 to 2,679 KB.
|
|
280
|
+
- Changed output: deromanize rules 2026-10-08 → 2026-10-09. Names mode no
|
|
281
|
+
longer ranks first a spelling that writes nothing for a consonant
|
|
282
|
+
(ml "murmu" gave മു, ks "rajnath" gave راج). Still open: sd "Rafi" in
|
|
283
|
+
words mode keeps a comma chunk, ranked second.
|
|
284
|
+
- Changed output, Go only: NFC now equals Python and ICU on long runs of
|
|
285
|
+
combining marks (Go inserted U+034F after 30 marks). Names split on the
|
|
286
|
+
same spaces in Go and TypeScript (Go missed 24, among them the no-break
|
|
287
|
+
space).
|
|
288
|
+
- Language tags: `hi-IN`, `HI` and `hin` now mean Hindi in Go and
|
|
289
|
+
TypeScript; before, they silently meant no language.
|
|
290
|
+
- Fixed: TypeScript `normalize` and `fold` threw `RangeError` above about
|
|
291
|
+
125,000 code points; Go `NameKeys` took quadratic time (70.6 s → 0.09 s
|
|
292
|
+
on 1 MB of Tamil).
|
|
293
|
+
- Go and TypeScript read one pinned Unicode 15.0 table for the blocks
|
|
294
|
+
indickit has rules for. Cost: about 2.9 KB gzipped in each bundle that
|
|
295
|
+
reads it; the phonetic, normalize and romanize bundles are now over their
|
|
296
|
+
README budgets.
|
|
297
|
+
- Fixed: Go `phonetic.Score` and `Index.Search` panicked (integer divide
|
|
298
|
+
by zero) on a query word of about 10,000 letters; they now give the
|
|
299
|
+
reference's scores. TypeScript `score` threw `RangeError` from about
|
|
300
|
+
125,000 candidates in Node.
|
|
301
|
+
- Faster on long input, same output: `wellformed` (262,144 code points:
|
|
302
|
+
2.3 s -> 0.03 s in Go, 7.2 s -> 0.04 s in TypeScript) and the `romanize`
|
|
303
|
+
and `deromanize` decoders (a 4,096-letter word ran out of 6 GB in Go). `phonetic-search`
|
|
304
|
+
still grows with (query words) x (candidate words): cap user input
|
|
305
|
+
(README, Limits).
|
|
306
|
+
- Changed output, TypeScript only: `romanize` text() keeps a ZWJ or ZWNJ
|
|
307
|
+
that no run of the script goes on from (at the start, after a space,
|
|
308
|
+
after an unassigned code point), as Go always did; it deleted it.
|
|
309
|
+
- Changed output: romanize rules 2026-10-08 -> 2026-10-09 and deromanize
|
|
310
|
+
2026-10-09 -> 2026-10-09.1. A word of more than 256 letters (`max_letters`)
|
|
311
|
+
gets one spelling: the first spelling of each 256-letter piece, joined.
|
|
312
|
+
Words that long are not words; the beam was quadratic on a long run of
|
|
313
|
+
characters it has no table for (32,000 random code points: 49 s).
|
|
314
|
+
- Faster, same output: `deromanize` 1.5-1.6x (a Hindi word in the browser:
|
|
315
|
+
19 -> 12 ms) and `romanize` 1.2-1.4x, in Go and TypeScript. The
|
|
316
|
+
`romanize` browser file is now 9 KB.
|
|
317
|
+
- Changed output: deromanize rules 2026-10-09.1 -> 2026-10-09.2. No
|
|
318
|
+
spelling is empty: "q" in Bengali gave ক, ক্ and an empty spelling; the
|
|
319
|
+
empty one is dropped, as `romanize` always did (11 of 3,033 conformance
|
|
320
|
+
rows change, each by that drop).
|
|
321
|
+
- The browser files are minified: each is 0.3 to 3.5 KB smaller gzipped
|
|
322
|
+
(`phonetic` 7 KB, `normalize` 16 KB, `romanize` 7 KB, `deromanize`
|
|
323
|
+
21 KB, `phonetic-search` 17 KB). Same code, same output (`check-dist`).
|
|
324
|
+
|
|
325
|
+
## 0.8.0 (2026-10-08)
|
|
326
|
+
|
|
327
|
+
Rules: phonetic 2026-10-07, normalize 2026-10-07, segment 2026-10-06, stem 2026-10-06, romanize 2026-10-07, deromanize 2026-10-08, phonetic-search 2026-10-08
|
|
328
|
+
|
|
329
|
+
- New: `deromanize`, Latin typing to Indian script (`indickit/deromanize`).
|
|
330
|
+
- New: `phonetic-search`, a scored search over phonetic keys
|
|
331
|
+
(`indickit/phonetic-search`).
|
|
332
|
+
- Changed output: phonetic rules 2026-10-06.1 → 2026-10-07, which added the
|
|
333
|
+
folds `brahmic-av-u`, `flap-r` and `latin-ngh`.
|
|
334
|
+
- README: a table of the 22 languages and what each utility covers.
|
|
335
|
+
- 0.7.0 was committed but never tagged or published; its work is here.
|
|
336
|
+
|
|
337
|
+
## 0.6.0 (2026-10-07)
|
|
338
|
+
|
|
339
|
+
Rules: phonetic 2026-10-06.1, normalize 2026-10-07, segment 2026-10-06, stem 2026-10-06, romanize 2026-10-07
|
|
340
|
+
|
|
341
|
+
- New: `romanize`, Indian text in Latin letters as people spell it
|
|
342
|
+
(`indickit/romanize`). Its tables load at run time, beside the module or
|
|
343
|
+
from jsDelivr at this version's tag.
|
|
344
|
+
|
|
345
|
+
## 0.5.0 (2026-10-07)
|
|
346
|
+
|
|
347
|
+
Rules: phonetic 2026-10-06.1, normalize 2026-10-07, segment 2026-10-06, stem 2026-10-06
|
|
348
|
+
|
|
349
|
+
- Changed output: `fold` rules 2026-10-07.
|
|
350
|
+
- Changed output: phonetic 2026-10-06.1, a fix for digits.
|
|
351
|
+
- Language tags on `normalize` and `fold`.
|
|
352
|
+
|
|
353
|
+
## 0.4.3 (2026-10-06)
|
|
354
|
+
|
|
355
|
+
Rules: phonetic 2026-10-06, normalize 2026-10-05, segment 2026-10-06, stem 2026-10-06
|
|
356
|
+
|
|
357
|
+
- First version on npm. README names the floors: Go 1.22, Node 20,
|
|
358
|
+
TypeScript `moduleResolution` nodenext or bundler.
|
|
359
|
+
|
|
360
|
+
## 0.4.2, 0.4.1 (2026-10-06)
|
|
361
|
+
|
|
362
|
+
Rules: as 0.4.0.
|
|
363
|
+
|
|
364
|
+
- Go floor 1.22. 0.4.1 failed its README test and was not published.
|
|
365
|
+
|
|
366
|
+
## 0.4.0 (2026-10-06)
|
|
367
|
+
|
|
368
|
+
Rules: phonetic 2026-10-06, normalize 2026-10-05, segment 2026-10-06, stem 2026-10-06
|
|
369
|
+
|
|
370
|
+
- New: `stem`, one search key for the forms of a word, 13 languages.
|
|
371
|
+
|
|
372
|
+
## 0.3.0 (2026-10-06)
|
|
373
|
+
|
|
374
|
+
Rules: phonetic 2026-10-06, normalize 2026-10-05, segment 2026-10-06
|
|
375
|
+
|
|
376
|
+
- New: `segment`, the letters a reader sees.
|
|
377
|
+
|
|
378
|
+
## 0.2.0 (2026-10-06)
|
|
379
|
+
|
|
380
|
+
Rules: phonetic 2026-10-06, normalize 2026-10-05
|
|
381
|
+
|
|
382
|
+
- New: `normalize`, one encoding for text that looks the same.
|
|
383
|
+
- Changed output: phonetic rules 2026-10-05 → 2026-10-06.
|
|
384
|
+
|
|
385
|
+
## 0.1.0 (2026-10-05)
|
|
386
|
+
|
|
387
|
+
Rules: phonetic 2026-10-05
|
|
388
|
+
|
|
389
|
+
- New: `phonetic`, a key that matches one name across scripts.
|