which-dialect 0.5.0 → 0.5.2

package/README.md

@@ -138,8 +138,8 @@ The result has `text` (the input in Unicode NFC form), `issues` and `parts` (the
 
 **On by default (the spellchecker):**
 
-- **`spelling`**: a doubtful syllable is first tried with its neighbors: if fixing its accents makes a dictionary word with the syllables next to it, that word is suggested (*hom nay* → *hôm nay* "today", where *nay* alone would become *này* "this"; *an com* → *ăn cơm*; *Chung toi* → *Chúng tôi*), changing only the doubtful syllables. Otherwise, syllables that aren't in the language get the accented forms they could be (*khong* → *không*); rare plain-letter words next to a far more common accented form too (*toi* → *tôi*, *hom* → *hôm*), and, in a sentence typed without accents, any more common accented form (*an* → *ăn*, *di* → *đi*; not *con* → *còn*, about as common). Names (capitalized mid-sentence), the language's listed pronouns (*tui*) and plain-letter words with no accented form (*email*) are left alone. Languages written in syllables only (Vietnamese).
-- **`foreign-word`**: words in the learner's own language get the target-language word, in the region and register (*market* → *chợ*, *ice cream* → *kem*). Phrases the base dictionary knows are translated as one, falling back to single words. Words the language has no word for are suggested to be left out (`leaveOut` in its checker settings: Vietnamese has no *the*, *a*, *an*). Which language each word is in comes from the two dictionaries: an English word that's Vietnamese only with accents added (*but*: *bút*) is English; Vietnamese typed without accents (*khong*) is Vietnamese; words both languages have go by how common they are in each (*em*, *di*, *tui* are Vietnamese even between English words; *the* is English), and close calls (*an*, *to*) follow their neighbors, leaning to the target language.
+- **`spelling`**: a doubtful syllable is first tried with its neighbors: if fixing its accents makes a dictionary word with the syllables next to it, that word is suggested (*hom nay* → *hôm nay* "today", where *nay* alone would become *này* "this"; *an com* → *ăn cơm*; *Chung toi* → *Chúng tôi*), changing only the doubtful syllables. Less common words written without accents can change the same way when one dictionary word clearly wins (*mua sam* → *mua sắm* "shopping", *nha sach* → *nhà sách*), even common ones when the whole word has no accents (*nha bang* → *nhà băng* "bank"), while a common one left plain next to accented syllables is taken as meant (*tháng sau* stays), *cho con* never becomes *chó con* (not a headword) and runs that could be two words equally are left alone (*ban an*: *bản án* or *bàn ăn*). Otherwise, syllables that aren't in the language get the accented forms they could be (*khong* → *không*); rare plain-letter words next to a far more common accented form too (*toi* → *tôi*, *hom* → *hôm*), and, in a sentence typed without accents, any more common accented form (*an* → *ăn*, *di* → *đi*; not *con* → *còn*, about as common). Names (capitalized mid-sentence), the language's listed pronouns (*tui*) and plain-letter words with no accented form (*email*) are left alone. Languages written in syllables only (Vietnamese).
+- **`foreign-word`**: words in the learner's own language get the target-language word, in the region and register (*market* → *chợ*, *ice cream* → *kem*). Phrases the base dictionary knows are translated as one, falling back to single words. Words the language has no word for are suggested to be left out (`leaveOut` in its checker settings: Vietnamese has no *the*, *a*, *an*). Which language each word is in comes from the two dictionaries: an English word that's also Vietnamese once accents are added is weighed by how common each is (*rat* is *rất*, far more common than English *rat*; *but* stays English, far more common than *bút*), and close calls (*met*: *mệt* or English *met*) are Vietnamese only among Vietnamese words; Vietnamese typed without accents (*khong*) is Vietnamese; words both languages have go by how common they are in each (*em*, *di*, *tui* are Vietnamese even between English words; *the* is English), and close calls (*an*, *to*) follow their neighbors, leaning to the target language.
 
 **Opt-in (`rules`):**
 

package/dist/check.js

@@ -60,6 +60,15 @@ const ACCENTS_GAP_UNACCENTED_TEXT = 0.2;
 // combinations are looked up at most.
 const COMPOUND_FORMS_PER_SYLLABLE = 5;
 const COMPOUND_MAX_COMBINATIONS = 64;
+// When none of a run's syllables is doubtful on its own (all real words written without accents, "mua sam"),
+// its accented form is suggested only if one dictionary word clearly wins: ahead of the next by this many
+// Zipf points ("ban an" could be "bản án" or "bàn ăn", equally common, so it's left alone).
+const PLAIN_RUN_MARGIN = 0.5;
+// In a run that has accents elsewhere, only syllables less common than this (Zipf) may change that way: a
+// common word left without accents next to accented ones is usually meant ("tháng sau", next month, isn't
+// "tháng sáu", June). A run with no accents at all ("nha bang") was probably typed without them, so any of
+// its syllables may change ("nhà băng", bank), still only when one word clearly wins.
+const PLAIN_RUN_MAX_ZIPF = 5;
 async function spelling(ctx) {
     if (ctx.config.units !== 'syllables' || !ctx.meta.syllables)
         return [];
@@ -94,8 +103,11 @@ async function spelling(ctx) {
         };
         const suspect = (u) => unknown.has(u) || (plainWords.has(u) && betterForms(u).length > 0);
         // First, runs of syllables with a doubtful one that make a word once their accents are fixed: "hom nay"
-        // → "hôm nay", "an com" → "ăn cơm", "hôm nây" → "hôm nay". Only the doubtful syllables change. The word
-        // decides between forms a syllable alone can't ("nay" alone could be này, nảy…).
+        // → "hôm nay", "an com" → "ăn cơm", "hôm nây" → "hôm nay". The word decides between forms a syllable
+        // alone can't ("nay" alone could be này, nảy…). Syllables written without accents may change too
+        // ("mua sam" → "mua sắm": "sam" is a word, but "mua sam" isn't, and "mua sắm" is); when none of them is
+        // doubtful on its own, only if one word clearly wins. Syllables written with accents never change
+        // unless they're doubtful.
         const covered = new Set();
         const loneSet = new Set(checked);
         const maxUnits = ctx.config.maxWordUnits ?? 3;
@@ -104,11 +116,15 @@ async function spelling(ctx) {
                 const span = sentence.units.slice(i, i + k);
                 if (span.some((u, j) => !loneSet.has(u) || covered.has(u) || (j > 0 && !joined(ctx.text, span[j - 1], u))))
                     continue;
-                if (!span.some(suspect))
+                const doubtful = span.some(suspect);
+                const allPlain = span.every(isPlain);
+                const rarePlain = (u) => plainWords.has(u) && (allPlain || (syllables[lowerOf(u)] ?? 0) < PLAIN_RUN_MAX_ZIPF);
+                const mayChange = (u) => suspect(u) || rarePlain(u);
+                if (!doubtful && !span.some(rarePlain))
                     continue;
                 const written = span.map(lowerOf);
                 const combos = accentCombinations(written, byPlain, {
-                    fixed: span.map((u) => !suspect(u)),
+                    fixed: span.map((u) => !mayChange(u)),
                     perUnit: COMPOUND_FORMS_PER_SYLLABLE,
                     max: COMPOUND_MAX_COMBINATIONS,
                 }).filter((c) => c !== written.join(' '));
@@ -119,12 +135,14 @@ async function spelling(ctx) {
                     .slice(0, 3);
                 if (!found.length)
                     continue;
+                if (!doubtful && found.length > 1 && found[0].frequency - found[1].frequency < PLAIN_RUN_MARGIN)
+                    continue;
                 const at = { start: span[0].start, end: span[k - 1].end, text: ctx.text.slice(span[0].start, span[k - 1].end) };
                 const suggestions = found.map((f) => matchCase(at.text, f.word));
                 const quoted = listOf(suggestions.map(quote), 'or');
                 issues.push({
                     rule: 'spelling', severity: 'warning', ...at, suggestions,
-                    message: span.filter(suspect).every(isPlain)
+                    message: span.filter(mayChange).every(isPlain)
                         ? `${quote(at.text)} looks typed without accents. Did you mean ${quoted}?`
                         : `${quote(at.text)} isn't spelled right. Did you mean ${quoted}?`,
                 });

package/dist/detect.js

@@ -7,15 +7,21 @@ import { plain } from "./text.js";
  * in each: clearly more common in the target, the target's (Vietnamese "em", "di" next to English
  * "hello", "shopping": English has rare "em", "di"); far more common in the base, the base's ("the");
  * otherwise, and for units neither knows (names), the nearest decided neighbors' language when they agree,
- * else the target's (the text is mainly the target language: "an" next to English "the"). In languages written in syllables, a base
- * word that's a target syllable only once accents are added ("but": bút) is the base's; a word only the
- * target knows without its accents ("khong") is still the target's. The base is known through the translator, so
+ * else the target's (the text is mainly the target language: "an" next to English "the"). In languages
+ * written in syllables, a base word that's a target word only once accents are added is weighed the same
+ * way, using its accented form's frequency: "rat" is "rất" (far more common than English "rat"); close
+ * calls ("met": mệt or English met) go to the target only when their neighbors are the target's ("Tôi met
+ * quá"), so English stays English among English ("kem but it was": but, not bút). A word only the target
+ * knows without its accents ("khong") is still the target's. The base is known through the translator, so
  * inflected forms count ("went", "cooked").
  */
 // Zipf points by which a word both languages know must be more common in one to count as that language's
 // whatever its neighbors: less for the target, since the text is mainly the target language.
 const TARGET_MARGIN = 1;
 const BASE_MARGIN = 2;
+// For a base word that's a target word only with accents added, a smaller lead keeps it the base's: it was
+// typed exactly as the base word ("but" is 1.9 more common in English than "bút" in Vietnamese).
+const BASE_MARGIN_ACCENTS = 1;
 export async function markLanguages(list, o) {
     const syllables = await o.target.syllables();
     const syllableLanguage = Object.keys(syllables).length > 0;
@@ -37,9 +43,18 @@ export async function markLanguages(list, o) {
         if (t === 'exact')
             return b ? 'both' : o.lang;
         if (b)
-            return o.base;
+            return t === 'accents' ? 'accents' : o.base;
         return t === 'accents' ? o.lang : '?';
     }));
+    // The most common form a plain-letter unit takes once accents are added ("rat" → rất, 6.4).
+    const accentedFrequency = (u) => {
+        const bare = plain(u.text.toLowerCase());
+        let best = 0;
+        for (const [s, f] of Object.entries(syllables))
+            if (f > best && plain(s) === bare)
+                best = f;
+        return best;
+    };
     // How common a word is in each language (Zipf; 0 when unknown).
     const targetFrequency = async (u) => {
         const lower = u.text.toLowerCase();
@@ -48,18 +63,29 @@ export async function markLanguages(list, o) {
         return (await o.target.lookup(lower))[0]?.frequency ?? 0;
     };
     const baseFrequency = async (u) => (await o.baseDictionary.lookup(u.text.toLowerCase()))[0]?.frequency ?? 0;
-    // Words both languages know, decided by frequency when it's clear.
+    // Words both languages know, decided by frequency when it's clear. Base words that are target words with
+    // accents added use that form's frequency, and close calls are marked '?accents' (decided by neighbors,
+    // more strictly).
     const decided = await Promise.all(marks.map(async (m, i) => {
-        if (m !== 'both')
+        if (m !== 'both' && m !== 'accents')
             return m;
-        const [t, b] = await Promise.all([targetFrequency(list[i]), baseFrequency(list[i])]);
-        return t - b >= TARGET_MARGIN ? o.lang : b - t >= BASE_MARGIN ? o.base : '?';
+        const [t, b] = await Promise.all([m === 'accents' ? accentedFrequency(list[i]) : targetFrequency(list[i]), baseFrequency(list[i])]);
+        if (t - b >= TARGET_MARGIN)
+            return o.lang;
+        if (b - t >= (m === 'accents' ? BASE_MARGIN_ACCENTS : BASE_MARGIN))
+            return o.base;
+        return m === 'accents' ? '?accents' : '?';
     }));
+    const known = (x) => x !== '?' && x !== '?accents';
     return decided.map((m, i) => {
-        if (m !== '?')
+        if (known(m))
             return m;
-        const left = decided.slice(0, i).reverse().find((x) => x !== '?');
-        const right = decided.slice(i + 1).find((x) => x !== '?');
+        const left = decided.slice(0, i).reverse().find(known);
+        const right = decided.slice(i + 1).find(known);
+        // A word written as typed in the base language needs every decided neighbor to be the target's to be
+        // read as the target's without its accents ("Tôi met quá": mệt); otherwise it's the base's.
+        if (m === '?accents')
+            return (left ?? o.lang) === o.lang && (right ?? o.lang) === o.lang && (left || right) ? o.lang : o.base;
         return left && right && left !== right ? o.lang : left ?? right ?? o.lang;
     });
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "which-dialect",
-  "version": "0.5.0",
+  "version": "0.5.2",
   "description": "Translate words between languages and dialects, meaning by meaning, starting with Vietnamese (Northern, Central, Southern) and English, with Wiktionary data loaded on demand. More languages are planned.",
   "type": "module",
   "main": "./dist/index.js",