which-dialect 0.5.1 → 0.5.4

package/README.md

@@ -80,7 +80,7 @@ The meanings of a word, following forms and variant spellings: useful for lettin
 Every language's data defines its words in English, so English is the bridge between any two languages. For each **meaning** of the source word:
 
 1. **Forms and variants go to the word they belong to:** irregular forms from the data (*said* → *say*), regular English forms by rule (*walked* → *walk*, *running* → *run*), texting spellings (*dont* → *don't*), and regional variants (*hông* → *không*, keeping its Southern region).
-2. **English terms carry the meaning:** the terms in its definition ("bắp": *corn*), or for English words the word itself and its synonyms.
+2. **English terms carry the meaning:** the terms in its definition ("bắp": *corn*), or for English words the word itself and its synonyms. **Place names** carry their own name: *Japan* ↔ *Nhật Bản* (defined "Japan (a country in East Asia)"), also *Nhật*; *South Korea* ↔ *Hàn Quốc*; *Kyoto* → *Kinh Đô*. A word counts only when its definition is of the place itself, not one that mentions it (a robe "worn by the emperors of China, Japan, Korea and Vietnam" isn't Japan). Places come from Wiktionary's place categories: every place name in the target language, and in English every country plus about 4,200 common cities and regions, leaving out ones that are also ordinary words (*Reading*, *Bath*, *Well*). Given names and surnames aren't translated.
 3. **The target is searched with a compatible part of speech** (`compatiblePos`): a noun for a noun, but an English auxiliary verb may be a Vietnamese particle, and an adjective may be a verb in another language.
 4. **Wiktionary's translation tables** give the usual translation of each English meaning in each language, often with region and register tags ("I" → Vietnamese *tôi*, *tớ* (informal), *tui* (South); *cool* "mildly low temperature" → *mát*). Listed words get a strong boost, more when tagged for the target region, less when tagged only for other regions. From another language, the English meaning whose table lists the source word is the same meaning.
 5. **Within one language, synonyms are direct equivalents.** Wiktionary lists dialect words as synonyms (*ngô* → *bắp*, *lợn* → *heo*, *lift* → *elevator*), in either direction (*dạ* lists *vâng*).
@@ -139,7 +139,8 @@ The result has `text` (the input in Unicode NFC form), `issues` and `parts` (the
 **On by default (the spellchecker):**
 
 - **`spelling`**: a doubtful syllable is first tried with its neighbors: if fixing its accents makes a dictionary word with the syllables next to it, that word is suggested (*hom nay* → *hôm nay* "today", where *nay* alone would become *này* "this"; *an com* → *ăn cơm*; *Chung toi* → *Chúng tôi*), changing only the doubtful syllables. Less common words written without accents can change the same way when one dictionary word clearly wins (*mua sam* → *mua sắm* "shopping", *nha sach* → *nhà sách*), even common ones when the whole word has no accents (*nha bang* → *nhà băng* "bank"), while a common one left plain next to accented syllables is taken as meant (*tháng sau* stays), *cho con* never becomes *chó con* (not a headword) and runs that could be two words equally are left alone (*ban an*: *bản án* or *bàn ăn*). Otherwise, syllables that aren't in the language get the accented forms they could be (*khong* → *không*); rare plain-letter words next to a far more common accented form too (*toi* → *tôi*, *hom* → *hôm*), and, in a sentence typed without accents, any more common accented form (*an* → *ăn*, *di* → *đi*; not *con* → *còn*, about as common). Names (capitalized mid-sentence), the language's listed pronouns (*tui*) and plain-letter words with no accented form (*email*) are left alone. Languages written in syllables only (Vietnamese).
-- **`foreign-word`**: words in the learner's own language get the target-language word, in the region and register (*market* → *chợ*, *ice cream* → *kem*). Phrases the base dictionary knows are translated as one, falling back to single words. Words the language has no word for are suggested to be left out (`leaveOut` in its checker settings: Vietnamese has no *the*, *a*, *an*). Which language each word is in comes from the two dictionaries: an English word that's Vietnamese only with accents added (*but*: *bút*) is English; Vietnamese typed without accents (*khong*) is Vietnamese; words both languages have go by how common they are in each (*em*, *di*, *tui* are Vietnamese even between English words; *the* is English), and close calls (*an*, *to*) follow their neighbors, leaning to the target language.
+  **Letter typos:** a syllable that isn't in the language also gets one-letter fixes (a letter missing, extra, wrong or two swapped) with any accents: *khôg*, *khôngg* → *không*; *họk* → *học*; *tihch* → *thích*; *muốm* → *muốn*; with its neighbors, *Cảm ơm* → *Cảm ơn*. Candidates keeping the accent marks the writer typed come first (*ngừoi* → *người*, *ơm* → *ơn* rather than *ôm*), then accent-only fixes, then letters often confused (`similarLetters` in the checker settings; Vietnamese c/k/q, s/x, i/y, m/n count as half a change), then other letter changes, then the more common syllable. Plain-letter non-words are only treated as typos when a one-letter fix is a common syllable (else they're names or foreign words). Real syllables are never letter-changed, so a typo that makes another real word (*kông*, *mún*) isn't caught. Abbreviations with inner capitals (*GĐ*, *TKiều*) and other scripts (Chinese characters) are left alone.
+- **`foreign-word`**: words in the learner's own language get the target-language word, in the region and register (*market* → *chợ*, *ice cream* → *kem*). Phrases the base dictionary knows are translated as one, falling back to single words. Words the language has no word for are suggested to be left out (`leaveOut` in its checker settings: Vietnamese has no *the*, *a*, *an*). Which language each word is in comes from the two dictionaries: an English word that's also Vietnamese once accents are added is weighed by how common each is (*rat* is *rất*, far more common than English *rat*; *but* stays English, far more common than *bút*), and close calls (*met*: *mệt* or English *met*) are Vietnamese only among Vietnamese words; Vietnamese typed without accents (*khong*) is Vietnamese; words both languages have go by how common they are in each (*em*, *di*, *tui* are Vietnamese even between English words; *the* is English), and close calls (*an*, *to*) follow their neighbors, leaning to the target language.
 
 **Opt-in (`rules`):**
 
@@ -213,7 +214,7 @@ One language's dictionary. `baseUrl` defaults to `https://cdn.jsdelivr.net/npm/w
 - `dictionary.searchEnglish(term, { region?, pos?, exclude?, limit?, allSenses? })`: words for an English term in this language, one per word, best first (no meaning handling: use the translator for that). `allSenses: true` returns every matching sense of those words instead.
 - `dictionary.frames()`: the language's sentence frames and frame words, unfilled (`{ frames, words }`; empty without frames).
 - `dictionary.segment(text)`: the text split into the language's words, longest dictionary match first (up to the checker's `maxWordUnits`, 3 for Vietnamese): `Hôm nay tôi ăn cơm.` → *Hôm nay* · *tôi* · *ăn cơm*, each `{ text, start, end, entries }` (offsets into the NFC text; `entries` empty for words the dictionary doesn't have). Use it to treat a word of several syllables as one, e.g. to show its meaning.
-- `dictionary.variants(word, { limit? })`: other words with the same letters but other accents that the dictionary has, most common first, each `{ word, entries, frequency? }`: *muộn* → *muốn*, *mượn*, *muôn*…; for words of several syllables every syllable varies (*hom nay* → *hôm nay*). For apps that let learners pick the word they meant: a real word with the wrong tone passes the spellchecker. Languages written in syllables only (`[]` otherwise).
+- `dictionary.variants(word, { limit? })`: other words with the same letters but other accents that the dictionary has, most common first, each `{ word, entries, frequency? }`: *muộn* → *muốn*, *mượn*, *muôn*…; for words of several syllables every syllable varies (*hom nay* → *hôm nay*). Capitalized headwords count too, unless they're only a person's name: *nhat* → *nhất*, *Nhật* (Japan), *nhạt*…; *nhat ban* → *Nhật Bản*. For apps that let learners pick the word they meant: a real word with the wrong tone passes the spellchecker. Languages written in syllables only (`[]` otherwise).
 - `dictionary.checker()` / `dictionary.syllables()`: the language's checker settings and syllable list (with frequencies), used by `createChecker` (`{}` for languages without them).
 - `dictionary.picks()`: the language's [hand-picked words](#hand-picked-words), `{ word, pos, gloss, picks: [{ word, tags? }], first? }` per English meaning (empty for languages without any).
 - `dictionary.meta()`: the language's name, regions, region groups, source, license, build date and counts.
@@ -251,6 +252,7 @@ This is a suggestion tool, not a curated translation dictionary. Check results b
 - **No word frequencies.** The source doesn't say how common a word or meaning is. Without `pos`/`meaning`, the first meaning follows the dictionary's order, which isn't always the most common one.
 - **English is the bridge.** Translating between two non-English languages goes through English definitions, so nuance English doesn't mark can be lost.
 - **`meaning` matches words, not ideas.** It compares your words with the dictionary's definitions (and the top translation's), so "awesome" won't find a sense defined as "Fashionable; trendy; hip." Use the definition's own wording, or let users pick from `senses()`.
+- **Nationality phrases aren't built yet.** *Japan* → *Nhật Bản* works, but "Japanese food" → *món Nhật* (the thing, then the country's short name) doesn't; see FUTURE_IMPROVEMENTS.md.
 - **Grammar words translate poorly.** Words defined by their function ("marks the future tense" for Vietnamese *sẽ*) aren't reached from English *will*; *the*, *is* or *gonna* may give nothing useful.
 
 ## Word frequencies
@@ -307,11 +309,11 @@ Generated data isn't committed (it would bloat git history); it's built before p
 
 ### Adding a language
 
-1. Add `languages/<code>.ts`: the Kaikki language name, the language's regions (and groups, if any), and how to read regions from Wiktionary's tags. See [`languages/vi.ts`](languages/vi.ts) (dialect areas and a pronoun table) and [`languages/en.ts`](languages/en.ts) (countries with region groups, and trimming a very large language). Spanish was built this way too (countries and region groups; `languages/es.ts` in the git history). Set `skipFormOf` for heavily inflected languages; for very large ones, see the trimming options in [`scripts/language-config.ts`](scripts/language-config.ts) (`dropTechnical`, `formOfPos`, `keepFormOf`, `maxSensesPerEntry`, `shardLength`, …).
+1. Add `languages/<code>.ts`: the Kaikki language name, the language's regions (and groups, if any), and how to read regions from Wiktionary's tags. See [`languages/vi.ts`](languages/vi.ts) (dialect areas and a pronoun table) and [`languages/en.ts`](languages/en.ts) (countries with region groups, and trimming a very large language). Spanish was built this way too (countries and region groups; `languages/es.ts` in the git history). Set `placeNames: {}` so its place names translate ("Japan" ↔ its name for Japan). Set `skipFormOf` for heavily inflected languages; for very large ones, see the trimming options in [`scripts/language-config.ts`](scripts/language-config.ts) (`dropTechnical`, `formOfPos`, `keepFormOf`, `maxSensesPerEntry`, `shardLength`, …).
 2. Copy a data package (`packages/vi`) to `packages/<code>` and update its `package.json` and `README.md`.
 3. Add the language's code to `translationLangs` in `languages/en.ts` (so English keeps its translation tables for it) and rebuild English.
 4. `npm run build:data -- <code>`, check the results, add a few real-data tests and evaluation cases, and publish.
-5. Optional: [checker](#checking-text) settings (`checker` in `languages/<code>.ts`): `units: 'syllables'` for languages written in syllables (also builds the syllable list for accent suggestions), `maxWordUnits`, the reliable `pronouns` per pronoun-table column, `ambiguousPronouns`, and `politeEndings`. Without them, the language is read as space-separated words and gets the region check.
+5. Optional: [checker](#checking-text) settings (`checker` in `languages/<code>.ts`): `units: 'syllables'` for languages written in syllables (also builds the syllable list for accent suggestions), `maxWordUnits`, the reliable `pronouns` per pronoun-table column, `ambiguousPronouns`, and `politeEndings`, `similarLetters` (groups of letters often confused, for typo suggestions). Without them, the language is read as space-separated words and gets the region check.
 6. Optional: [sentence frames](#sentence-frames): `frames` (how the language says each frame in `languages/frames.ts`) and `frameWords` (the words that change by region, each from a definition or an override with a note). Frames reach other languages through the shared catalog; a new frame goes in the catalog once.
 7. Optional, later: [hand-picked words](#hand-picked-words). A speaker runs `npm run picks-sheet -- <code>`, fills in `your_pick` only where the first choice is wrong, and pastes the `--apply` output into `picks`.
 

package/dist/check.js

@@ -11,7 +11,7 @@
 // The checker only speaks up when it's confident and stays silent otherwise, so no issues means none of
 // the checks found anything, not that the text is correct.
 import { createDictionary, resolveRegion } from "./index.js";
-import { accentCombinations, joined, matchCase, plain, segment, sentencesOf, syllablesByPlain, units } from "./text.js";
+import { accentCombinations, joined, matchCase, plain, segment, sentencesOf, spellingCandidates, syllablesByPlain, units } from "./text.js";
 import { markLanguages } from "./detect.js";
 import { createTranslator } from "./translate.js";
 import { PRONOUN_PERSONS, } from "./types.js";
@@ -50,8 +50,15 @@ const unique = (items) => [...new Set(items)];
 //   because filled-in sentence frames add accented words), any accented form that's more common
 //   counts ("an" → "ăn"; but not "con" → "còn", about as common). Left alone: capitalized words
 //   mid-sentence (names) and the language's listed pronouns, some written without accents ("tui").
-// First, a doubtful syllable (one of the above) is tried with its neighbors: when fixing its accents makes a
-// dictionary word with them, that word is suggested for the run ("hom nay" → "hôm nay", not "hôm này").
+// - a syllable that isn't in the language may also be a letter typo: one letter missing, extra, wrong or two
+//   swapped ("khôg", "khôngg" → "không"; "họk" → "học"), with any accents (spellingCandidates). Plain-letter
+//   ones count only when a one-letter fix gives a common syllable ("tihch" → "thích"); real syllables are
+//   never letter-changed, so correct words stay.
+// Not checked: abbreviations and names with a capital after the first letter ("GĐ", "TKiều"), and words
+// with letters the language's syllables don't use (Chinese characters in Vietnamese text).
+// First, a doubtful syllable (one of the above) is tried with its neighbors: when fixing its accents (or a
+// letter) makes a dictionary word with them, that word is suggested for the run ("hom nay" → "hôm nay", not
+// "hôm này"; "cảm ơm" → "cảm ơn", not "cảm ôm").
 // Units inside a known word, or in the learner's own language, aren't checked.
 // Zipf points by which an accented form must be more common than a plain-letter word to be suggested.
 const ACCENTS_GAP = 1.5;
@@ -69,6 +76,11 @@ const PLAIN_RUN_MARGIN = 0.5;
 // "tháng sáu", June). A run with no accents at all ("nha bang") was probably typed without them, so any of
 // its syllables may change ("nhà băng", bank), still only when one word clearly wins.
 const PLAIN_RUN_MAX_ZIPF = 5;
+// A plain-letter non-syllable with no accented form is checked as a typo only if a one-letter fix gives a
+// syllable at least this common (Zipf); otherwise it's probably a name or a foreign word.
+const PLAIN_TYPO_MIN_ZIPF = 4;
+// Candidates kept per misspelled syllable.
+const SPELLING_CANDIDATES = 3;
 async function spelling(ctx) {
     if (ctx.config.units !== 'syllables' || !ctx.meta.syllables)
         return [];
@@ -78,6 +90,10 @@ async function spelling(ctx) {
     const suggest = (u, candidates, severity, message) => issues.push({ rule: 'spelling', severity, start: u.start, end: u.end, text: u.text, message, suggestions: candidates.map((c) => matchCase(u.text, c)) });
     const options = (u, candidates) => listOf(candidates.map((c) => quote(matchCase(u.text, c))), 'or');
     const pronouns = new Set(Object.values(ctx.config.pronouns ?? {}).flat().map((w) => w.toLowerCase()));
+    // The language's letters (without accents), from its syllables: words using others aren't its own.
+    const alphabet = new Set([...byPlain.keys()].join(''));
+    const ownLetters = (u) => [...plain(u.text.toLowerCase()).replace(/[’']/g, '')].every((c) => alphabet.has(c));
+    const abbreviation = (u) => /\p{Lu}/u.test(u.text.slice(1));
     for (const sentence of ctx.sentences) {
         const lone = sentence.words.filter((w) => !w.entries.length || w.units.length === 1).flatMap((w) => w.units).filter((u) => !ctx.inBase(u));
         const target = sentence.units.filter((u) => !ctx.inBase(u) && /\p{L}/u.test(u.text));
@@ -87,13 +103,38 @@ async function spelling(ctx) {
         const inWord = (u) => sentence.words.some((w) => w.entries.length && w.units.length > 1 && w.units.includes(u));
         // What each lone unit is: not a syllable of the language (with forms it could be), or a plain-letter
         // syllable that could be missing its accents.
-        const checked = lone.filter((u) => !/\d/.test(lowerOf(u)) && !inWord(u));
-        const unknown = new Set(checked.filter((u) => !(lowerOf(u) in syllables) && ((byPlain.get(plain(lowerOf(u))) ?? []).length || !isPlain(u))));
+        const checked = lone.filter((u) => !/\d/.test(lowerOf(u)) && !inWord(u) && !abbreviation(u) && ownLetters(u));
+        const name = (u) => u !== sentence.units[0] && /\p{Lu}/u.test(u.text.charAt(0));
+        // Each non-syllable's candidates: accents and one-letter changes, the writer's own accents first.
+        const candidatesOf = new Map();
+        const similar = ctx.config.similarLetters ?? [];
+        const candidates = (u, limit = SPELLING_CANDIDATES) => spellingCandidates(lowerOf(u), syllables, byPlain, { limit, similar }).map((c) => c.word);
+        // For words with neighbors, only the best-ranked candidates: "muốm ăn" mustn't become "muối ăn" (salt)
+        // through a worse fix of "muốm" than "muốn" just because "muối ăn" is a dictionary word.
+        const bestCandidates = (u, limit) => {
+            const all = spellingCandidates(lowerOf(u), syllables, byPlain, { limit: 50, similar });
+            return all.filter((c) => c.marks === all[0]?.marks && c.edits === all[0]?.edits).slice(0, limit).map((c) => c.word);
+        };
+        const unknown = new Set(checked.filter((u) => {
+            const lower = lowerOf(u);
+            if (lower in syllables)
+                return false;
+            if ((byPlain.get(plain(lower)) ?? []).length || !isPlain(u))
+                return true;
+            // Plain letters, no accented form: a typo only if a one-letter fix is a common syllable, and not a name.
+            if (name(u))
+                return false;
+            const best = spellingCandidates(lower, syllables, byPlain, { limit: 1, minZipf: PLAIN_TYPO_MIN_ZIPF, similar })[0];
+            return Boolean(best);
+        }));
+        for (const u of unknown)
+            candidatesOf.set(u, candidates(u));
         const plainWords = new Set(checked.filter((u) => {
             const lower = lowerOf(u);
             return lower in syllables && lower === plain(lower) && !pronouns.has(lower) && (u === sentence.units[0] || !/\p{Lu}/u.test(u.text.charAt(0)));
         }));
-        const missingAccents = [...unknown].some(isPlain);
+        // Only accent slips count here ("khong"), not letter typos ("tihch").
+        const missingAccents = [...unknown].some((u) => isPlain(u) && (byPlain.get(lowerOf(u)) ?? []).length > 0);
         // Typed without accents: mostly plain letters, and an accent error already, or 3+ words with no accent at all.
         const unaccented = mostlyPlain && (missingAccents || (target.length >= 3 && target.every(isPlain)));
         const gap = unaccented ? ACCENTS_GAP_UNACCENTED_TEXT : ACCENTS_GAP;
@@ -125,6 +166,8 @@ async function spelling(ctx) {
                 const written = span.map(lowerOf);
                 const combos = accentCombinations(written, byPlain, {
                     fixed: span.map((u) => !mayChange(u)),
+                    // Misspelled syllables may change letters too ("cảm ơm" → "cảm ơn").
+                    forms: span.map((u) => (unknown.has(u) ? bestCandidates(u, COMPOUND_FORMS_PER_SYLLABLE) : undefined)),
                     perUnit: COMPOUND_FORMS_PER_SYLLABLE,
                     max: COMPOUND_MAX_COMBINATIONS,
                 }).filter((c) => c !== written.join(' '));
@@ -156,11 +199,11 @@ async function spelling(ctx) {
                 continue;
             const lower = lowerOf(u);
             if (unknown.has(u)) {
-                const candidates = (byPlain.get(plain(lower)) ?? []).slice(0, 3);
+                const found = candidatesOf.get(u) ?? [];
                 const accented = !isPlain(u);
                 // Missing accents on plain letters is the common learner slip; a wrong accent is more surely wrong.
-                suggest(u, candidates, accented ? 'error' : 'warning', candidates.length
-                    ? `${quote(u.text)} isn't a ${ctx.meta.name} syllable. Did you mean ${options(u, candidates)}?`
+                suggest(u, found, accented ? 'error' : 'warning', found.length
+                    ? `${quote(u.text)} isn't a ${ctx.meta.name} syllable. Did you mean ${options(u, found)}?`
                     : `${quote(u.text)} isn't a ${ctx.meta.name} syllable. Check the spelling and accents.`);
             }
             else if (plainWords.has(u)) {

package/dist/detect.js

@@ -7,15 +7,21 @@ import { plain } from "./text.js";
  * in each: clearly more common in the target, the target's (Vietnamese "em", "di" next to English
  * "hello", "shopping": English has rare "em", "di"); far more common in the base, the base's ("the");
  * otherwise, and for units neither knows (names), the nearest decided neighbors' language when they agree,
- * else the target's (the text is mainly the target language: "an" next to English "the"). In languages written in syllables, a base
- * word that's a target syllable only once accents are added ("but": bút) is the base's; a word only the
- * target knows without its accents ("khong") is still the target's. The base is known through the translator, so
+ * else the target's (the text is mainly the target language: "an" next to English "the"). In languages
+ * written in syllables, a base word that's a target word only once accents are added is weighed the same
+ * way, using its accented form's frequency: "rat" is "rất" (far more common than English "rat"); close
+ * calls ("met": mệt or English met) go to the target only when their neighbors are the target's ("Tôi met
+ * quá"), so English stays English among English ("kem but it was": but, not bút). A word only the target
+ * knows without its accents ("khong") is still the target's. The base is known through the translator, so
  * inflected forms count ("went", "cooked").
  */
 // Zipf points by which a word both languages know must be more common in one to count as that language's
 // whatever its neighbors: less for the target, since the text is mainly the target language.
 const TARGET_MARGIN = 1;
 const BASE_MARGIN = 2;
+// For a base word that's a target word only with accents added, a smaller lead keeps it the base's: it was
+// typed exactly as the base word ("but" is 1.9 more common in English than "bút" in Vietnamese).
+const BASE_MARGIN_ACCENTS = 1;
 export async function markLanguages(list, o) {
     const syllables = await o.target.syllables();
     const syllableLanguage = Object.keys(syllables).length > 0;
@@ -37,9 +43,18 @@ export async function markLanguages(list, o) {
         if (t === 'exact')
             return b ? 'both' : o.lang;
         if (b)
-            return o.base;
+            return t === 'accents' ? 'accents' : o.base;
         return t === 'accents' ? o.lang : '?';
     }));
+    // The most common form a plain-letter unit takes once accents are added ("rat" → rất, 6.4).
+    const accentedFrequency = (u) => {
+        const bare = plain(u.text.toLowerCase());
+        let best = 0;
+        for (const [s, f] of Object.entries(syllables))
+            if (f > best && plain(s) === bare)
+                best = f;
+        return best;
+    };
     // How common a word is in each language (Zipf; 0 when unknown).
     const targetFrequency = async (u) => {
         const lower = u.text.toLowerCase();
@@ -48,18 +63,29 @@ export async function markLanguages(list, o) {
         return (await o.target.lookup(lower))[0]?.frequency ?? 0;
     };
     const baseFrequency = async (u) => (await o.baseDictionary.lookup(u.text.toLowerCase()))[0]?.frequency ?? 0;
-    // Words both languages know, decided by frequency when it's clear.
+    // Words both languages know, decided by frequency when it's clear. Base words that are target words with
+    // accents added use that form's frequency, and close calls are marked '?accents' (decided by neighbors,
+    // more strictly).
     const decided = await Promise.all(marks.map(async (m, i) => {
-        if (m !== 'both')
+        if (m !== 'both' && m !== 'accents')
             return m;
-        const [t, b] = await Promise.all([targetFrequency(list[i]), baseFrequency(list[i])]);
-        return t - b >= TARGET_MARGIN ? o.lang : b - t >= BASE_MARGIN ? o.base : '?';
+        const [t, b] = await Promise.all([m === 'accents' ? accentedFrequency(list[i]) : targetFrequency(list[i]), baseFrequency(list[i])]);
+        if (t - b >= TARGET_MARGIN)
+            return o.lang;
+        if (b - t >= (m === 'accents' ? BASE_MARGIN_ACCENTS : BASE_MARGIN))
+            return o.base;
+        return m === 'accents' ? '?accents' : '?';
     }));
+    const known = (x) => x !== '?' && x !== '?accents';
     return decided.map((m, i) => {
-        if (m !== '?')
+        if (known(m))
             return m;
-        const left = decided.slice(0, i).reverse().find((x) => x !== '?');
-        const right = decided.slice(i + 1).find((x) => x !== '?');
+        const left = decided.slice(0, i).reverse().find(known);
+        const right = decided.slice(i + 1).find(known);
+        // A word written as typed in the base language needs every decided neighbor to be the target's to be
+        // read as the target's without its accents ("Tôi met quá": mệt); otherwise it's the base's.
+        if (m === '?accents')
+            return (left ?? o.lang) === o.lang && (right ?? o.lang) === o.lang && (left || right) ? o.lang : o.base;
         return left && right && left !== right ? o.lang : left ?? right ?? o.lang;
     });
 }

package/dist/index.js

@@ -1,4 +1,4 @@
-import { PRONOUN_PERSONS, fromStored, normalizeEnglish, shardKey, } from "./types.js";
+import { PERSONAL_NAME, PRONOUN_PERSONS, fromStored, normalizeEnglish, shardKey, } from "./types.js";
 import { accentCombinations, segment as segmentUnits, syllablesByPlain, units } from "./text.js";
 export * from "./types.js";
 export * from "./pos.js";
@@ -104,10 +104,17 @@ export function createDictionary(options) {
                 perUnit: parts.length === 1 ? VARIANT_FORMS_ONE_SYLLABLE : VARIANT_FORMS_PER_SYLLABLE,
                 max: MAX_VARIANT_COMBINATIONS,
             }).filter((c) => c !== written);
-            const found = await Promise.all(combos.map(async (c) => ({ word: c, entries: await lookup(c) })));
+            // Each form as written in lowercase and capitalized, for names ("nhat" → "Nhật", Japan; "nhat ban" →
+            // "Nhật Bản").
+            const forms = [...new Set(combos.flatMap((c) => [c, c.replace(/(^|\s)(\S)/gu, (_, sep, ch) => sep + ch.toUpperCase())]))];
+            // A capitalized form counts when it's its own headword (lookup ignores case) and more than a person's
+            // name (a place, a people), so given names don't crowd out words.
+            const found = (await Promise.all(forms.map(async (c) => ({ word: c, entries: await lookup(c) })))).map((v) => v.word === v.word.toLowerCase()
+                ? v
+                : { ...v, entries: v.entries.filter((e) => e.word === v.word && e.senses.some((s) => !PERSONAL_NAME.test(s.glosses[0] ?? ''))) });
             const frequency = (v) => {
                 const known = v.entries.map((e) => e.frequency).filter((f) => f !== undefined);
-                return known.length ? Math.max(...known) : parts.length === 1 ? list[v.word] || undefined : undefined;
+                return known.length ? Math.max(...known) : parts.length === 1 ? list[v.word.toLowerCase()] || undefined : undefined;
             };
             // Most common first; ties (and unknown frequencies) keep the order of more common syllable forms.
             return found

package/dist/text.d.ts

@@ -37,8 +37,28 @@ export declare function syllablesByPlain(syllables: Record<string, number>): Map
  * every combination, the ones made of more common forms first, at most `max`. Syllables that mustn't
  * change (`fixed[i]`) keep their written form. The written run itself is included.
  */
-export declare function accentCombinations(written: string[], byPlain: Map<string, string[]>, { fixed, perUnit, max }: {
+export declare function accentCombinations(written: string[], byPlain: Map<string, string[]>, { fixed, forms: given, perUnit, max }: {
     fixed?: boolean[];
+    forms?: (string[] | undefined)[];
     perUnit: number;
     max: number;
 }): string[];
+/**
+ * The syllables a misspelled one could be, best first: the same letters with other accents ("khong" →
+ * "không"), and one letter changed: missing, extra, wrong or two swapped ("khôg", "khôngg" → "không";
+ * "tihch" → "thích"; "ơm" → "ơn"), each with any accents. First, same-letter fixes keeping every accent mark
+ * the writer typed ("ngừoi" → "người": the marks were on the wrong letter). Then candidates keeping more of
+ * the writer's marks ("muốm" → "muốn", not "muỗm"; "ơm" → "ơn", not "ôm"); then accent-only fixes, then changes between
+ * letters often confused (`similar`: groups like "ckq", counted as half a change: "họk" → "học", not "họ"),
+ * then other letter changes, then the more common syllable. Letter changes only give syllables at least
+ * `minZipf` common. `edits` is 0, 0.5 or 1; `marks` how well the writer's accent marks are kept.
+ */
+export declare function spellingCandidates(written: string, syllables: Record<string, number>, byPlain: Map<string, string[]>, { limit, minZipf, similar }: {
+    limit: number;
+    minZipf?: number;
+    similar?: string[];
+}): {
+    word: string;
+    edits: number;
+    marks: number;
+}[];

package/dist/text.js

@@ -77,12 +77,13 @@ export function syllablesByPlain(syllables) {
  * every combination, the ones made of more common forms first, at most `max`. Syllables that mustn't
  * change (`fixed[i]`) keep their written form. The written run itself is included.
  */
-export function accentCombinations(written, byPlain, { fixed = [], perUnit, max }) {
+export function accentCombinations(written, byPlain, { fixed = [], forms: given = [], perUnit, max }) {
     const choices = written.map((w, i) => {
         const lower = w.toLowerCase();
         if (fixed[i])
             return [lower];
-        const forms = (byPlain.get(plain(lower)) ?? []).slice(0, perUnit);
+        // `forms[i]`, when given, replaces the accent forms (e.g. spellingCandidates, with letter changes too).
+        const forms = (given[i] ?? byPlain.get(plain(lower)) ?? []).slice(0, perUnit);
         return forms.includes(lower) ? forms : [...forms, lower];
     });
     // Every combination with the sum of its forms' ranks: lower sums are made of more common forms.
@@ -95,3 +96,74 @@ export function accentCombinations(written, byPlain, { fixed = [], perUnit, max
         .slice(0, max)
         .map((c) => c.words.join(' '));
 }
+// The letters a plain-letter typo can be changed to (accents come from the syllable list afterwards).
+const LETTERS = 'abcdefghijklmnopqrstuvwxyz';
+// A syllable's accent marks, as a multiset of combining characters ("muốn": circumflex, acute).
+const marksOf = (s) => [...s.normalize('NFD')].filter((c) => /\p{M}/u.test(c));
+/**
+ * The syllables a misspelled one could be, best first: the same letters with other accents ("khong" →
+ * "không"), and one letter changed: missing, extra, wrong or two swapped ("khôg", "khôngg" → "không";
+ * "tihch" → "thích"; "ơm" → "ơn"), each with any accents. First, same-letter fixes keeping every accent mark
+ * the writer typed ("ngừoi" → "người": the marks were on the wrong letter). Then candidates keeping more of
+ * the writer's marks ("muốm" → "muốn", not "muỗm"; "ơm" → "ơn", not "ôm"); then accent-only fixes, then changes between
+ * letters often confused (`similar`: groups like "ckq", counted as half a change: "họk" → "học", not "họ"),
+ * then other letter changes, then the more common syllable. Letter changes only give syllables at least
+ * `minZipf` common. `edits` is 0, 0.5 or 1; `marks` how well the writer's accent marks are kept.
+ */
+export function spellingCandidates(written, syllables, byPlain, { limit, minZipf = 3, similar = [] }) {
+    const lower = written.toLowerCase();
+    const base = plain(lower);
+    const alike = (x, y) => similar.some((g) => g.includes(x) && g.includes(y));
+    // Each one-letter change of the plain letters, with its cost: half for letters often confused.
+    const variants = new Map();
+    const add = (v, cost) => {
+        if (v !== base && cost < (variants.get(v) ?? Infinity))
+            variants.set(v, cost);
+    };
+    for (let i = 0; i <= base.length; i++) {
+        const [a, b] = [base.slice(0, i), base.slice(i)];
+        if (b)
+            add(a + b.slice(1), 1);
+        if (b.length > 1)
+            add(a + b[1] + b[0] + b.slice(2), 1);
+        for (const c of LETTERS) {
+            if (b)
+                add(a + c + b.slice(1), alike(b[0], c) ? 0.5 : 1);
+            add(a + c + b, 1);
+        }
+    }
+    const found = new Map();
+    for (const w of byPlain.get(base) ?? [])
+        if (w !== lower)
+            found.set(w, 0);
+    for (const [v, cost] of variants) {
+        for (const w of byPlain.get(v) ?? []) {
+            if ((syllables[w] ?? 0) >= minZipf && cost < (found.get(w) ?? Infinity))
+                found.set(w, cost);
+        }
+    }
+    const mine = marksOf(lower);
+    // How a candidate keeps the writer's marks: how many it shares, and a score where missing or extra marks
+    // count down (0 when the writer typed none, so plain-letter writing isn't held against any candidate).
+    const keep = (w) => {
+        const left = marksOf(w);
+        let shared = 0;
+        for (const m of mine) {
+            const at = left.indexOf(m);
+            if (at >= 0) {
+                shared++;
+                left.splice(at, 1);
+            }
+        }
+        return { all: shared === mine.length, score: mine.length ? shared - (mine.length - shared) - left.length : 0 };
+    };
+    return [...found]
+        .map(([word, edits]) => {
+        const k = keep(word);
+        // Tier 0: same letters, every typed mark kept (accents only moved or added).
+        return { word, edits, tier: edits === 0 && k.all ? 0 : 1, marks: k.score, zipf: syllables[word] ?? 0 };
+    })
+        .sort((a, b) => a.tier - b.tier || b.marks - a.marks || a.edits - b.edits || b.zipf - a.zipf)
+        .slice(0, limit)
+        .map(({ word, edits, tier, marks }) => ({ word, edits, marks: marks - tier * 100 }));
+}

package/dist/translate.js

@@ -5,7 +5,7 @@
 // for those terms with a compatible part of speech, in the target region. Each source sense is translated
 // on its own, so a word's meanings don't get mixed up ("cool" the temperature vs. "cool" the compliment).
 import { compatiblePos } from "./pos.js";
-import { displayGloss, glossTerms, normalizeEnglish, PRONOUN_PERSONS, TRAILING_PARTICLE, } from "./types.js";
+import { displayGloss, glossTerms, normalizeEnglish, PRONOUN_PERSONS, TRAILING_PARTICLE, withoutNotes, } from "./types.js";
 import { createDictionary, DEFAULT_EXCLUDED_LABELS, resolveRegion, } from "./index.js";
 /**
  * Possible base words for a regularly inflected English word, most likely first: "walked" -> walk,
@@ -238,6 +238,13 @@ function bridgeTerms(sense, from) {
     // Main terms get the full "main meaning" bonus: the English word itself, and the first term of the
     // definition ("will": "Used to express the future tense" → "future tense", which is what "sẽ" means).
     const main = new Set();
+    // An English place name is its own bridge: "Japan" finds "Nhật Bản" (defined "Japan (a country in East
+    // Asia)"). Its definition describes it ("A country in East Asia. Capital and largest city: Tokyo.") and
+    // would find other places.
+    if (from === 'en' && sense.pos === 'name') {
+        const name = normalizeEnglish(sense.lemma);
+        return { terms: [name], synonymsFrom: 1, main: new Set([name]) };
+    }
     if (from === 'en') {
         terms.push(normalizeEnglish(sense.lemma));
         main.add(normalizeEnglish(sense.lemma));
@@ -272,6 +279,14 @@ function bridgeTerms(sense, from) {
     }
     return { terms: unique.slice(0, 8), synonymsFrom, main };
 }
+// Whether a definition is of the place `name` itself: it starts with the name, notes aside ("Japan (a
+// country in East Asia)"), or is a short list of equivalents including it ("Japano-; Nippo-; Japan").
+function namesPlace(gloss, name) {
+    const text = normalizeEnglish(withoutNotes(gloss));
+    if (text === name || new RegExp(`^${name.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')}(?=[\\s,;:.]|$)`).test(text))
+        return true;
+    return isTermList(gloss, glossTerms(gloss).length) && glossTerms(gloss).some(([t]) => t === name);
+}
 // Every comma/semicolon part of the definition became a term, and each is at most three words.
 function isTermList(gloss, termCount) {
     const parts = gloss.replace(/\.\s*$/, '').split(/[;,]/);
@@ -361,8 +376,11 @@ export function createTranslator(options = {}) {
         const hits = [];
         // Terms are lowercase, so the capitalized word too ("i" is the letter; the pronoun is "I"). When the
         // exact part of speech is there, other compatible ones are left out (the letter "i" for pronoun "I").
+        // Names of several words are capitalized throughout ("South Korea", "United Kingdom").
         const capitalized = term.charAt(0).toUpperCase() + term.slice(1);
-        let entries = [...(await en.lookup(term)), ...(capitalized !== term ? await en.lookup(capitalized) : [])];
+        const titled = term.replace(/(^|[ -])([a-z])/g, (_, sep, c) => sep + c.toUpperCase());
+        const forms = [...new Set([term, capitalized, titled])];
+        let entries = (await Promise.all(forms.map((f) => en.lookup(f)))).flat();
         const usable = (s) => !s.altOf && intersects(s.regions, wanted) && !s.labels.some((l) => exclude.includes(l));
         if (entries.some((e) => e.pos === posList[0] && e.senses.some(usable)))
             entries = entries.filter((e) => e.pos === posList[0]);
@@ -580,6 +598,7 @@ export function createTranslator(options = {}) {
                     continue;
                 }
                 const posList = compatiblePos(sense.pos);
+                const placeName = from === 'en' && sense.pos === 'name' && to !== 'en';
                 const context = contentWords(`${sense.glosses.join(' ')} ${meaning ?? ''}`);
                 const scored = new Map();
                 const keep = (hit, score, bridgeTerm) => {
@@ -616,6 +635,12 @@ export function createTranslator(options = {}) {
                         if (!wordRank.has(h.word))
                             wordRank.set(h.word, wordRank.size);
                     hits.forEach((hit) => {
+                        // An English place name matches a word that means that place: a definition starting with the
+                        // name ("Japan (a country in East Asia)"), a short list ("Nhật": "Japano-; Nippo-; Japan"), or a
+                        // spelling of such a word; not a definition that mentions it ("hoàng bào": a robe worn by the
+                        // emperors of China, Japan, Korea and Vietnam; "Vinh": the provincial capital of …, Vietnam).
+                        if (placeName && !hit.altOf && !namesPlace(hit.gloss, term))
+                            return;
                         const rank = wordRank.get(hit.word);
                         // Translating between dialects of one language: the word itself only counts if it's tagged for the target region.
                         if (to === from && hit.word === sense.lemma && toWanted && !hit.regionTagged)

package/dist/types.d.ts

@@ -200,6 +200,11 @@ export type CheckerConfig = {
      * leaving them out rather than translating them: Vietnamese has no articles ("the", "a", "an").
      */
     leaveOut?: string[];
+    /**
+     * Groups of letters often confused, by sound or by neighboring keys ("ckq": Vietnamese c, k and q sound
+     * alike). The spellchecker counts swapping within a group as half a change, so "họk" → "học" before "họ".
+     */
+    similarLetters?: string[];
 };
 /**
  * Hand-picked first choices for one English meaning, in one target language (picks.json). A speaker's
@@ -329,6 +334,13 @@ export declare function normalizeEnglish(term: string): string;
  */
 export declare function displayGloss(glosses: string[]): string;
 export declare const TRAILING_PARTICLE: RegExp;
+/** Definitions of personal names ("a male given name from Chinese", "a surname"), which aren't translated. */
+export declare const PERSONAL_NAME: RegExp;
+/**
+ * A definition without its parenthesized notes, nested ones too ("South Korea (a country …; official name:
+ * Đại Hàn Dân Quốc (“Republic of Korea”))" → "South Korea"), and a note left open by a cut-off definition.
+ */
+export declare function withoutNotes(gloss: string): string;
 /**
  * Splits an English gloss into search terms, main meaning first: "now, today, this time" ->
  * [["now", 0], ["today", 1], ["this time", 2]]. Parenthesized notes and final punctuation are dropped.

package/dist/types.js

@@ -69,6 +69,20 @@ export function displayGloss(glosses) {
 }
 // Trailing words dropped to also match the bare verb: "wait for" is found by "wait" too.
 export const TRAILING_PARTICLE = / (for|to|at|on|with|about|of|in|into|up|out|off|over)$/;
+/** Definitions of personal names ("a male given name from Chinese", "a surname"), which aren't translated. */
+export const PERSONAL_NAME = /\b(?:given name|surname|family name|patronymic|nickname)\b/i;
+/**
+ * A definition without its parenthesized notes, nested ones too ("South Korea (a country …; official name:
+ * Đại Hàn Dân Quốc (“Republic of Korea”))" → "South Korea"), and a note left open by a cut-off definition.
+ */
+export function withoutNotes(gloss) {
+    let text = gloss;
+    for (let previous = ''; previous !== text;) {
+        previous = text;
+        text = text.replace(/\([^()]*\)/g, ' ');
+    }
+    return text.replace(/\([^)]*$/, ' ');
+}
 /**
  * Splits an English gloss into search terms, main meaning first: "now, today, this time" ->
  * [["now", 0], ["today", 1], ["this time", 2]]. Parenthesized notes and final punctuation are dropped.
@@ -80,7 +94,7 @@ export function glossTerms(gloss) {
     // express the future tense" (English "will"). The phrase after marks/expresses/indicates/denotes is
     // their meaning, so it's a term too, placed after the gloss's own terms.
     const grammar = [...gloss.matchAll(GRAMMAR_PHRASE)].map((m) => normalizeEnglish(m[1])).filter((t) => t.split(' ').length <= 3);
-    let cleaned = gloss.replace(/\([^)]*\)/g, ' ').replace(/[“”"]/g, '');
+    let cleaned = withoutNotes(gloss).replace(/[“”"]/g, '');
     if (cleaned.includes(':'))
         cleaned = cleaned.slice(cleaned.lastIndexOf(':') + 1);
     const terms = [];

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "which-dialect",
-  "version": "0.5.1",
+  "version": "0.5.4",
   "description": "Translate words between languages and dialects, meaning by meaning, starting with Vietnamese (Northern, Central, Southern) and English, with Wiktionary data loaded on demand. More languages are planned.",
   "type": "module",
   "main": "./dist/index.js",