which-dialect 0.4.0 → 0.5.1

package/README.md

@@ -138,8 +138,8 @@ The result has `text` (the input in Unicode NFC form), `issues` and `parts` (the
 
 **On by default (the spellchecker):**
 
-- **`spelling`**: syllables that aren't in the language get the accented forms they could be (*khong* → *không*); rare plain-letter words next to a far more common accented form too (*toi* → *tôi*, *hom* → *hôm*), and, in a sentence typed without accents, any more common accented form (*an* → *ăn*, *di* → *đi*; not *con* → *còn*, about as common). Names (capitalized mid-sentence), the language's listed pronouns (*tui*) and plain-letter words with no accented form (*email*) are left alone. Languages written in syllables only (Vietnamese).
-- **`foreign-word`**: words in the learner's own language get the target-language word, in the region and register (*market* → *chợ*, *ice cream* → *kem*). Phrases the base dictionary knows are translated as one, falling back to single words. Words the language has no word for are suggested to be left out (`leaveOut` in its checker settings: Vietnamese has no *the*, *a*, *an*). Which language each word is in comes from the two dictionaries: an English word that's Vietnamese only with accents added (*but*: *bút*) is English; Vietnamese typed without accents (*khong*) is Vietnamese; words that could be either (*an*, *tui*) follow their neighbors, leaning to the target language.
+- **`spelling`**: a doubtful syllable is first tried with its neighbors: if fixing its accents makes a dictionary word with the syllables next to it, that word is suggested (*hom nay* → *hôm nay* "today", where *nay* alone would become *này* "this"; *an com* → *ăn cơm*; *Chung toi* → *Chúng tôi*), changing only the doubtful syllables. Less common words written without accents can change the same way when one dictionary word clearly wins (*mua sam* → *mua sắm* "shopping", *nha sach* → *nhà sách*), even common ones when the whole word has no accents (*nha bang* → *nhà băng* "bank"), while a common one left plain next to accented syllables is taken as meant (*tháng sau* stays), *cho con* never becomes *chó con* (not a headword) and runs that could be two words equally are left alone (*ban an*: *bản án* or *bàn ăn*). Otherwise, syllables that aren't in the language get the accented forms they could be (*khong* → *không*); rare plain-letter words next to a far more common accented form too (*toi* → *tôi*, *hom* → *hôm*), and, in a sentence typed without accents, any more common accented form (*an* → *ăn*, *di* → *đi*; not *con* → *còn*, about as common). Names (capitalized mid-sentence), the language's listed pronouns (*tui*) and plain-letter words with no accented form (*email*) are left alone. Languages written in syllables only (Vietnamese).
+- **`foreign-word`**: words in the learner's own language get the target-language word, in the region and register (*market* → *chợ*, *ice cream* → *kem*). Phrases the base dictionary knows are translated as one, falling back to single words. Words the language has no word for are suggested to be left out (`leaveOut` in its checker settings: Vietnamese has no *the*, *a*, *an*). Which language each word is in comes from the two dictionaries: an English word that's Vietnamese only with accents added (*but*: *bút*) is English; Vietnamese typed without accents (*khong*) is Vietnamese; words both languages have go by how common they are in each (*em*, *di*, *tui* are Vietnamese even between English words; *the* is English), and close calls (*an*, *to*) follow their neighbors, leaning to the target language.
 
 **Opt-in (`rules`):**
 
@@ -212,6 +212,8 @@ One language's dictionary. `baseUrl` defaults to `https://cdn.jsdelivr.net/npm/w
 - `dictionary.lookup(word)`: all entries for a word, with every sense's definitions, regions, labels, synonyms and variant links (`altOf`).
 - `dictionary.searchEnglish(term, { region?, pos?, exclude?, limit?, allSenses? })`: words for an English term in this language, one per word, best first (no meaning handling: use the translator for that). `allSenses: true` returns every matching sense of those words instead.
 - `dictionary.frames()`: the language's sentence frames and frame words, unfilled (`{ frames, words }`; empty without frames).
+- `dictionary.segment(text)`: the text split into the language's words, longest dictionary match first (up to the checker's `maxWordUnits`, 3 for Vietnamese): `Hôm nay tôi ăn cơm.` → *Hôm nay* · *tôi* · *ăn cơm*, each `{ text, start, end, entries }` (offsets into the NFC text; `entries` empty for words the dictionary doesn't have). Use it to treat a word of several syllables as one, e.g. to show its meaning.
+- `dictionary.variants(word, { limit? })`: other words with the same letters but other accents that the dictionary has, most common first, each `{ word, entries, frequency? }`: *muộn* → *muốn*, *mượn*, *muôn*…; for words of several syllables every syllable varies (*hom nay* → *hôm nay*). For apps that let learners pick the word they meant: a real word with the wrong tone passes the spellchecker. Languages written in syllables only (`[]` otherwise).
 - `dictionary.checker()` / `dictionary.syllables()`: the language's checker settings and syllable list (with frequencies), used by `createChecker` (`{}` for languages without them).
 - `dictionary.picks()`: the language's [hand-picked words](#hand-picked-words), `{ word, pos, gloss, picks: [{ word, tags? }], first? }` per English meaning (empty for languages without any).
 - `dictionary.meta()`: the language's name, regions, region groups, source, license, build date and counts.

package/dist/check.js

@@ -11,7 +11,7 @@
 // The checker only speaks up when it's confident and stays silent otherwise, so no issues means none of
 // the checks found anything, not that the text is correct.
 import { createDictionary, resolveRegion } from "./index.js";
-import { joined, matchCase, plain, segment, sentencesOf, units } from "./text.js";
+import { accentCombinations, joined, matchCase, plain, segment, sentencesOf, syllablesByPlain, units } from "./text.js";
 import { markLanguages } from "./detect.js";
 import { createTranslator } from "./translate.js";
 import { PRONOUN_PERSONS, } from "./types.js";
@@ -50,19 +50,30 @@ const unique = (items) => [...new Set(items)];
 //   because filled-in sentence frames add accented words), any accented form that's more common
 //   counts ("an" → "ăn"; but not "con" → "còn", about as common). Left alone: capitalized words
 //   mid-sentence (names) and the language's listed pronouns, some written without accents ("tui").
+// First, a doubtful syllable (one of the above) is tried with its neighbors: when fixing its accents makes a
+// dictionary word with them, that word is suggested for the run ("hom nay" → "hôm nay", not "hôm này").
 // Units inside a known word, or in the learner's own language, aren't checked.
 // Zipf points by which an accented form must be more common than a plain-letter word to be suggested.
 const ACCENTS_GAP = 1.5;
 const ACCENTS_GAP_UNACCENTED_TEXT = 0.2;
+// Accent forms tried per doubtful syllable when looking for a word with its neighbors, and how many
+// combinations are looked up at most.
+const COMPOUND_FORMS_PER_SYLLABLE = 5;
+const COMPOUND_MAX_COMBINATIONS = 64;
+// When none of a run's syllables is doubtful on its own (all real words written without accents, "mua sam"),
+// its accented form is suggested only if one dictionary word clearly wins: ahead of the next by this many
+// Zipf points ("ban an" could be "bản án" or "bàn ăn", equally common, so it's left alone).
+const PLAIN_RUN_MARGIN = 0.5;
+// In a run that has accents elsewhere, only syllables less common than this (Zipf) may change that way: a
+// common word left without accents next to accented ones is usually meant ("tháng sau", next month, isn't
+// "tháng sáu", June). A run with no accents at all ("nha bang") was probably typed without them, so any of
+// its syllables may change ("nhà băng", bank), still only when one word clearly wins.
+const PLAIN_RUN_MAX_ZIPF = 5;
 async function spelling(ctx) {
     if (ctx.config.units !== 'syllables' || !ctx.meta.syllables)
         return [];
     const syllables = await ctx.dict.syllables();
-    const byPlain = new Map();
-    for (const s of Object.keys(syllables))
-        byPlain.set(plain(s), [...(byPlain.get(plain(s)) ?? []), s]);
-    for (const list of byPlain.values())
-        list.sort((a, b) => syllables[b] - syllables[a]);
+    const byPlain = syllablesByPlain(syllables);
     const issues = [];
     const suggest = (u, candidates, severity, message) => issues.push({ rule: 'spelling', severity, start: u.start, end: u.end, text: u.text, message, suggestions: candidates.map((c) => matchCase(u.text, c)) });
     const options = (u, candidates) => listOf(candidates.map((c) => quote(matchCase(u.text, c))), 'or');
@@ -70,44 +81,97 @@ async function spelling(ctx) {
     for (const sentence of ctx.sentences) {
         const lone = sentence.words.filter((w) => !w.entries.length || w.units.length === 1).flatMap((w) => w.units).filter((u) => !ctx.inBase(u));
         const target = sentence.units.filter((u) => !ctx.inBase(u) && /\p{L}/u.test(u.text));
-        const mostlyPlain = target.filter((u) => u.text.toLowerCase() === plain(u.text.toLowerCase())).length >= target.length * 0.5;
-        let missingAccents = false;
-        const rare = [];
-        for (const u of lone) {
-            const lower = u.text.toLowerCase().replace(/[’']/g, '');
-            if (/\d/.test(lower))
-                continue;
-            const inWord = sentence.words.some((w) => w.entries.length && w.units.length > 1 && w.units.includes(u));
-            if (inWord)
+        const isPlain = (u) => u.text.toLowerCase() === plain(u.text.toLowerCase());
+        const mostlyPlain = target.filter(isPlain).length >= target.length * 0.5;
+        const lowerOf = (u) => u.text.toLowerCase().replace(/[’']/g, '');
+        const inWord = (u) => sentence.words.some((w) => w.entries.length && w.units.length > 1 && w.units.includes(u));
+        // What each lone unit is: not a syllable of the language (with forms it could be), or a plain-letter
+        // syllable that could be missing its accents.
+        const checked = lone.filter((u) => !/\d/.test(lowerOf(u)) && !inWord(u));
+        const unknown = new Set(checked.filter((u) => !(lowerOf(u) in syllables) && ((byPlain.get(plain(lowerOf(u))) ?? []).length || !isPlain(u))));
+        const plainWords = new Set(checked.filter((u) => {
+            const lower = lowerOf(u);
+            return lower in syllables && lower === plain(lower) && !pronouns.has(lower) && (u === sentence.units[0] || !/\p{Lu}/u.test(u.text.charAt(0)));
+        }));
+        const missingAccents = [...unknown].some(isPlain);
+        // Typed without accents: mostly plain letters, and an accent error already, or 3+ words with no accent at all.
+        const unaccented = mostlyPlain && (missingAccents || (target.length >= 3 && target.every(isPlain)));
+        const gap = unaccented ? ACCENTS_GAP_UNACCENTED_TEXT : ACCENTS_GAP;
+        const betterForms = (u) => {
+            const lower = lowerOf(u);
+            return (byPlain.get(lower) ?? []).filter((c) => c !== lower && syllables[c] >= syllables[lower] + gap - 1e-9).slice(0, 3);
+        };
+        const suspect = (u) => unknown.has(u) || (plainWords.has(u) && betterForms(u).length > 0);
+        // First, runs of syllables with a doubtful one that make a word once their accents are fixed: "hom nay"
+        // → "hôm nay", "an com" → "ăn cơm", "hôm nây" → "hôm nay". The word decides between forms a syllable
+        // alone can't ("nay" alone could be này, nảy…). Syllables written without accents may change too
+        // ("mua sam" → "mua sắm": "sam" is a word, but "mua sam" isn't, and "mua sắm" is); when none of them is
+        // doubtful on its own, only if one word clearly wins. Syllables written with accents never change
+        // unless they're doubtful.
+        const covered = new Set();
+        const loneSet = new Set(checked);
+        const maxUnits = ctx.config.maxWordUnits ?? 3;
+        for (let i = 0; i < sentence.units.length; i++) {
+            for (let k = Math.min(maxUnits, sentence.units.length - i); k >= 2; k--) {
+                const span = sentence.units.slice(i, i + k);
+                if (span.some((u, j) => !loneSet.has(u) || covered.has(u) || (j > 0 && !joined(ctx.text, span[j - 1], u))))
+                    continue;
+                const doubtful = span.some(suspect);
+                const allPlain = span.every(isPlain);
+                const rarePlain = (u) => plainWords.has(u) && (allPlain || (syllables[lowerOf(u)] ?? 0) < PLAIN_RUN_MAX_ZIPF);
+                const mayChange = (u) => suspect(u) || rarePlain(u);
+                if (!doubtful && !span.some(rarePlain))
+                    continue;
+                const written = span.map(lowerOf);
+                const combos = accentCombinations(written, byPlain, {
+                    fixed: span.map((u) => !mayChange(u)),
+                    perUnit: COMPOUND_FORMS_PER_SYLLABLE,
+                    max: COMPOUND_MAX_COMBINATIONS,
+                }).filter((c) => c !== written.join(' '));
+                const found = (await Promise.all(combos.map(async (c) => ({ word: c, entries: await ctx.dict.lookup(c) }))))
+                    .filter((f) => f.entries.length)
+                    .map((f) => ({ word: f.word, frequency: Math.max(-1, ...f.entries.map((e) => e.frequency ?? -1)) }))
+                    .sort((a, b) => b.frequency - a.frequency)
+                    .slice(0, 3);
+                if (!found.length)
+                    continue;
+                if (!doubtful && found.length > 1 && found[0].frequency - found[1].frequency < PLAIN_RUN_MARGIN)
+                    continue;
+                const at = { start: span[0].start, end: span[k - 1].end, text: ctx.text.slice(span[0].start, span[k - 1].end) };
+                const suggestions = found.map((f) => matchCase(at.text, f.word));
+                const quoted = listOf(suggestions.map(quote), 'or');
+                issues.push({
+                    rule: 'spelling', severity: 'warning', ...at, suggestions,
+                    message: span.filter(mayChange).every(isPlain)
+                        ? `${quote(at.text)} looks typed without accents. Did you mean ${quoted}?`
+                        : `${quote(at.text)} isn't spelled right. Did you mean ${quoted}?`,
+                });
+                span.forEach((u) => covered.add(u));
+                break;
+            }
+        }
+        // Then each remaining syllable on its own.
+        for (const u of checked) {
+            if (covered.has(u))
                 continue;
-            if (!(lower in syllables)) {
+            const lower = lowerOf(u);
+            if (unknown.has(u)) {
                 const candidates = (byPlain.get(plain(lower)) ?? []).slice(0, 3);
-                const accented = lower !== plain(lower);
-                if (!candidates.length && !accented)
-                    continue;
-                if (!accented)
-                    missingAccents = true;
+                const accented = !isPlain(u);
                 // Missing accents on plain letters is the common learner slip; a wrong accent is more surely wrong.
                 suggest(u, candidates, accented ? 'error' : 'warning', candidates.length
                     ? `${quote(u.text)} isn't a ${ctx.meta.name} syllable. Did you mean ${options(u, candidates)}?`
                     : `${quote(u.text)} isn't a ${ctx.meta.name} syllable. Check the spelling and accents.`);
             }
-            else if (lower === plain(lower) && !pronouns.has(lower) && (u === sentence.units[0] || !/\p{Lu}/u.test(u.text.charAt(0)))) {
-                rare.push(u);
+            else if (plainWords.has(u)) {
+                const better = betterForms(u);
+                if (!better.length)
+                    continue;
+                suggest(u, better, 'warning', unaccented
+                    ? `${quote(u.text)} looks typed without accents. Did you mean ${options(u, better)}?`
+                    : `${quote(u.text)} is a rare word; did you mean ${options(u, better)}?`);
             }
         }
-        // Typed without accents: mostly plain letters, and an accent error already, or 3+ words with no accent at all.
-        const unaccented = mostlyPlain && (missingAccents || (target.length >= 3 && target.every((u) => u.text.toLowerCase() === plain(u.text.toLowerCase()))));
-        for (const u of rare) {
-            const lower = u.text.toLowerCase();
-            const gap = unaccented ? ACCENTS_GAP_UNACCENTED_TEXT : ACCENTS_GAP;
-            const better = (byPlain.get(lower) ?? []).filter((c) => c !== lower && syllables[c] >= syllables[lower] + gap - 1e-9).slice(0, 3);
-            if (!better.length)
-                continue;
-            suggest(u, better, 'warning', unaccented
-                ? `${quote(u.text)} looks typed without accents. Did you mean ${options(u, better)}?`
-                : `${quote(u.text)} is a rare word; did you mean ${options(u, better)}?`);
-        }
     }
     return issues;
 }

package/dist/detect.js

@@ -3,15 +3,18 @@
 import { plain } from "./text.js";
 /**
  * Marks each unit (of one sentence) with `lang` or `base`. A unit the target knows and the base doesn't
- * is the target's, and the other way round; units both or neither know ("an", "to", names) take the
- * nearest decided neighbors' language when they agree. When they don't, the target's, unless the word is
- * far more common in the base (the text is mainly the target language: Vietnamese "an" stays Vietnamese
- * next to English "the", and "tui" between "but" and "không"). In languages written in syllables, a base
+ * is the target's, and the other way round. A unit both know ("em", "an", "to") goes by how common it is
+ * in each: clearly more common in the target, the target's (Vietnamese "em", "di" next to English
+ * "hello", "shopping": English has rare "em", "di"); far more common in the base, the base's ("the");
+ * otherwise, and for units neither knows (names), the nearest decided neighbors' language when they agree,
+ * else the target's (the text is mainly the target language: "an" next to English "the"). In languages written in syllables, a base
  * word that's a target syllable only once accents are added ("but": bút) is the base's; a word only the
  * target knows without its accents ("khong") is still the target's. The base is known through the translator, so
  * inflected forms count ("went", "cooked").
  */
-// Zipf points by which a word between the two languages must be more common in the base to count as base.
+// Zipf points by which a word both languages know must be more common in one to count as that language's
+// whatever its neighbors: less for the target, since the text is mainly the target language.
+const TARGET_MARGIN = 1;
 const BASE_MARGIN = 2;
 export async function markLanguages(list, o) {
     const syllables = await o.target.syllables();
@@ -32,7 +35,7 @@ export async function markLanguages(list, o) {
             return '?';
         const [t, b] = await Promise.all([targetKnows(u), baseKnows(u)]);
         if (t === 'exact')
-            return b ? '?' : o.lang;
+            return b ? 'both' : o.lang;
         if (b)
             return o.base;
         return t === 'accents' ? o.lang : '?';
@@ -45,14 +48,18 @@ export async function markLanguages(list, o) {
         return (await o.target.lookup(lower))[0]?.frequency ?? 0;
     };
     const baseFrequency = async (u) => (await o.baseDictionary.lookup(u.text.toLowerCase()))[0]?.frequency ?? 0;
-    return Promise.all(marks.map(async (m, i) => {
-        if (m !== '?')
+    // Words both languages know, decided by frequency when it's clear.
+    const decided = await Promise.all(marks.map(async (m, i) => {
+        if (m !== 'both')
             return m;
-        const left = marks.slice(0, i).reverse().find((x) => x !== '?');
-        const right = marks.slice(i + 1).find((x) => x !== '?');
-        if (left && right && left !== right) {
-            return (await baseFrequency(list[i])) - (await targetFrequency(list[i])) >= BASE_MARGIN ? o.base : o.lang;
-        }
-        return left ?? right ?? o.lang;
+        const [t, b] = await Promise.all([targetFrequency(list[i]), baseFrequency(list[i])]);
+        return t - b >= TARGET_MARGIN ? o.lang : b - t >= BASE_MARGIN ? o.base : '?';
     }));
+    return decided.map((m, i) => {
+        if (m !== '?')
+            return m;
+        const left = decided.slice(0, i).reverse().find((x) => x !== '?');
+        const right = decided.slice(i + 1).find((x) => x !== '?');
+        return left && right && left !== right ? o.lang : left ?? right ?? o.lang;
+    });
 }

package/dist/index.d.ts

@@ -49,6 +49,22 @@ export type PronounOptions = {
     /** Labels to leave out. Defaults to DEFAULT_EXCLUDED_LABELS; pass [] to include everything. */
     exclude?: string[];
 };
+/** One word of a text, as the dictionary splits it (see Dictionary.segment). */
+export type WordSegment = {
+    /** The word as written; offsets are into the text in Unicode NFC form. */
+    text: string;
+    start: number;
+    end: number;
+    /** The word's dictionary entries; empty for words the dictionary doesn't have (names, typos). */
+    entries: Entry[];
+};
+/** Another way to write a word with the same letters (see Dictionary.variants). */
+export type WordVariant = {
+    word: string;
+    entries: Entry[];
+    /** Zipf frequency (see Entry.frequency), or the syllable's for one syllable; undefined when unknown. */
+    frequency?: number;
+};
 export type Dictionary = {
     /** The language's name, regions, source, license and counts. */
     meta(): Promise<LanguageMeta>;
@@ -73,6 +89,22 @@ export type Dictionary = {
     syllables(): Promise<Record<string, number>>;
     /** The language's sentence frames and frame words, unfilled (see FramesData); empty without frames. */
     frames(): Promise<FramesData>;
+    /**
+     * The text split into the language's words: at each point, the longest run of units (syllables or
+     * words, up to the checker's `maxWordUnits`, default 3) that's a headword, so Vietnamese "hôm nay" is one
+     * word ("today"), not "hôm" + "nay". Units join only across spaces. Units the dictionary doesn't know
+     * are words of their own with no entries. Punctuation isn't returned.
+     */
+    segment(text: string): Promise<WordSegment[]>;
+    /**
+     * Other words written with the same letters but other accents, that the dictionary has, most common
+     * first: "muộn" → muốn, mượn, …; for words of several syllables, every syllable varies ("hom nay" →
+     * "hôm nay"). For languages written in syllables (with a syllable list); `[]` otherwise. The word
+     * itself isn't included.
+     */
+    variants(word: string, options?: {
+        limit?: number;
+    }): Promise<WordVariant[]>;
 };
 /**
  * The regions a region or region-group name stands for (e.g. English "North America" -> US, Canada),

package/dist/index.js

@@ -1,4 +1,5 @@
 import { PRONOUN_PERSONS, fromStored, normalizeEnglish, shardKey, } from "./types.js";
+import { accentCombinations, segment as segmentUnits, syllablesByPlain, units } from "./text.js";
 export * from "./types.js";
 export * from "./pos.js";
 /**
@@ -14,6 +15,11 @@ export const DEFAULT_EXCLUDED_LABELS = [
 // just a particular register); the second set costs more (specialized, playful or uncommon).
 const MILD_LABELS = new Set(['colloquial', 'informal', 'familiar', 'polite', 'formal', 'endearing', 'honorific']);
 const LABEL_PENALTY = (h) => h.labels.reduce((sum, l) => sum + (MILD_LABELS.has(l) ? 1 : 2), 0);
+// Accent forms tried per syllable: more for one syllable, fewer for longer words, where every
+// combination is looked up (at most MAX_VARIANT_COMBINATIONS).
+const VARIANT_FORMS_ONE_SYLLABLE = 12;
+const VARIANT_FORMS_PER_SYLLABLE = 5;
+const MAX_VARIANT_COMBINATIONS = 64;
 /**
  * The regions a region or region-group name stands for (e.g. English "North America" -> US, Canada),
  * or null when no region is given. Throws for names the language doesn't have, listing the valid ones.
@@ -63,13 +69,52 @@ export function createDictionary(options) {
         const key = shardKey(term, m.shardLength);
         return m.shards[kind].includes(key) ? loadOnce(`${kind}/${key}.json`) : null;
     }
+    async function lookup(word) {
+        const w = word.trim();
+        const [m, data] = await Promise.all([meta(), shard('words', w)]);
+        const stored = data?.[w] ?? data?.[w.toLowerCase()] ?? [];
+        return stored.map((e) => ({ ...e, senses: e.senses.map((s) => fromStored(s, m.regions)) }));
+    }
+    const checker = async () => {
+        const m = await meta();
+        return m.checker ? loadOnce('checker.json') : {};
+    };
+    const syllables = async () => {
+        const m = await meta();
+        return m.syllables ? loadOnce('syllables.json') : {};
+    };
     return {
         meta,
-        async lookup(word) {
-            const w = word.trim();
-            const [m, data] = await Promise.all([meta(), shard('words', w)]);
-            const stored = data?.[w] ?? data?.[w.toLowerCase()] ?? [];
-            return stored.map((e) => ({ ...e, senses: e.senses.map((s) => fromStored(s, m.regions)) }));
+        lookup,
+        checker,
+        syllables,
+        async segment(input) {
+            const text = input.normalize('NFC');
+            const config = await checker();
+            const words = await segmentUnits(text, units(text), config.maxWordUnits ?? 3, lookup);
+            return words.map(({ text: t, start, end, entries }) => ({ text: t, start, end, entries }));
+        },
+        async variants(word, { limit = 5 } = {}) {
+            const list = await syllables();
+            if (!Object.keys(list).length)
+                return [];
+            const written = word.trim().normalize('NFC').toLowerCase();
+            const parts = written.split(/\s+/);
+            const combos = accentCombinations(parts, syllablesByPlain(list), {
+                perUnit: parts.length === 1 ? VARIANT_FORMS_ONE_SYLLABLE : VARIANT_FORMS_PER_SYLLABLE,
+                max: MAX_VARIANT_COMBINATIONS,
+            }).filter((c) => c !== written);
+            const found = await Promise.all(combos.map(async (c) => ({ word: c, entries: await lookup(c) })));
+            const frequency = (v) => {
+                const known = v.entries.map((e) => e.frequency).filter((f) => f !== undefined);
+                return known.length ? Math.max(...known) : parts.length === 1 ? list[v.word] || undefined : undefined;
+            };
+            // Most common first; ties (and unknown frequencies) keep the order of more common syllable forms.
+            return found
+                .filter((v) => v.entries.length)
+                .map((v) => ({ ...v, frequency: frequency(v) }))
+                .sort((a, b) => (b.frequency ?? -1) - (a.frequency ?? -1))
+                .slice(0, limit);
         },
         async searchEnglish(term, { region, pos, exclude = DEFAULT_EXCLUDED_LABELS, limit = 10, allSenses = false } = {}) {
             const key = normalizeEnglish(term);
@@ -107,14 +152,6 @@ export function createDictionary(options) {
             const m = await meta();
             return m.picks ? loadOnce('picks.json') : [];
         },
-        async checker() {
-            const m = await meta();
-            return m.checker ? loadOnce('checker.json') : {};
-        },
-        async syllables() {
-            const m = await meta();
-            return m.syllables ? loadOnce('syllables.json') : {};
-        },
         async frames() {
             const m = await meta();
             if (!m.frames)

package/dist/text.d.ts

@@ -29,3 +29,16 @@ export declare function segment(text: string, list: Unit[], maxUnits: number, lo
 export declare const plain: (s: string) => string;
 /** Keeps a capital first letter when replacing a capitalized word ("Tôi" → "Con"). */
 export declare const matchCase: (original: string, replacement: string) => string;
+/** A language's syllables grouped by their form without accents, most common first. */
+export declare function syllablesByPlain(syllables: Record<string, number>): Map<string, string[]>;
+/**
+ * Ways to write a run of syllables with other accents ("hom nay" → "hôm nay", "hơm nay", …): each
+ * syllable's forms that share its letters (the most common `perUnit`, plus the syllable as written), in
+ * every combination, the ones made of more common forms first, at most `max`. Syllables that mustn't
+ * change (`fixed[i]`) keep their written form. The written run itself is included.
+ */
+export declare function accentCombinations(written: string[], byPlain: Map<string, string[]>, { fixed, perUnit, max }: {
+    fixed?: boolean[];
+    perUnit: number;
+    max: number;
+}): string[];

package/dist/text.js

@@ -55,3 +55,43 @@ export async function segment(text, list, maxUnits, lookup) {
 export const plain = (s) => s.normalize('NFD').replace(/\p{M}/gu, '').replace(/đ/g, 'd').replace(/Đ/g, 'D').normalize('NFC');
 /** Keeps a capital first letter when replacing a capitalized word ("Tôi" → "Con"). */
 export const matchCase = (original, replacement) => original.charAt(0) !== original.charAt(0).toLowerCase() ? replacement.charAt(0).toUpperCase() + replacement.slice(1) : replacement;
+// Syllables grouped by their plain form, most common first: "khong" → ["không", "khống", …]. Built once
+// per syllable list.
+const byPlainCache = new WeakMap();
+/** A language's syllables grouped by their form without accents, most common first. */
+export function syllablesByPlain(syllables) {
+    let byPlain = byPlainCache.get(syllables);
+    if (!byPlain) {
+        byPlain = new Map();
+        for (const s of Object.keys(syllables))
+            byPlain.set(plain(s), [...(byPlain.get(plain(s)) ?? []), s]);
+        for (const list of byPlain.values())
+            list.sort((a, b) => syllables[b] - syllables[a]);
+        byPlainCache.set(syllables, byPlain);
+    }
+    return byPlain;
+}
+/**
+ * Ways to write a run of syllables with other accents ("hom nay" → "hôm nay", "hơm nay", …): each
+ * syllable's forms that share its letters (the most common `perUnit`, plus the syllable as written), in
+ * every combination, the ones made of more common forms first, at most `max`. Syllables that mustn't
+ * change (`fixed[i]`) keep their written form. The written run itself is included.
+ */
+export function accentCombinations(written, byPlain, { fixed = [], perUnit, max }) {
+    const choices = written.map((w, i) => {
+        const lower = w.toLowerCase();
+        if (fixed[i])
+            return [lower];
+        const forms = (byPlain.get(plain(lower)) ?? []).slice(0, perUnit);
+        return forms.includes(lower) ? forms : [...forms, lower];
+    });
+    // Every combination with the sum of its forms' ranks: lower sums are made of more common forms.
+    let combos = [{ words: [], rank: 0 }];
+    for (const options of choices) {
+        combos = combos.flatMap((c) => options.map((o, r) => ({ words: [...c.words, o], rank: c.rank + r })));
+    }
+    return combos
+        .sort((a, b) => a.rank - b.rank)
+        .slice(0, max)
+        .map((c) => c.words.join(' '));
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "which-dialect",
-  "version": "0.4.0",
+  "version": "0.5.1",
   "description": "Translate words between languages and dialects, meaning by meaning, starting with Vietnamese (Northern, Central, Southern) and English, with Wiktionary data loaded on demand. More languages are planned.",
   "type": "module",
   "main": "./dist/index.js",