which-dialect 0.5.1 → 0.5.2

package/README.md

@@ -139,7 +139,7 @@ The result has `text` (the input in Unicode NFC form), `issues` and `parts` (the
 **On by default (the spellchecker):**
 
 - **`spelling`**: a doubtful syllable is first tried with its neighbors: if fixing its accents makes a dictionary word with the syllables next to it, that word is suggested (*hom nay* → *hôm nay* "today", where *nay* alone would become *này* "this"; *an com* → *ăn cơm*; *Chung toi* → *Chúng tôi*), changing only the doubtful syllables. Less common words written without accents can change the same way when one dictionary word clearly wins (*mua sam* → *mua sắm* "shopping", *nha sach* → *nhà sách*), even common ones when the whole word has no accents (*nha bang* → *nhà băng* "bank"), while a common one left plain next to accented syllables is taken as meant (*tháng sau* stays), *cho con* never becomes *chó con* (not a headword) and runs that could be two words equally are left alone (*ban an*: *bản án* or *bàn ăn*). Otherwise, syllables that aren't in the language get the accented forms they could be (*khong* → *không*); rare plain-letter words next to a far more common accented form too (*toi* → *tôi*, *hom* → *hôm*), and, in a sentence typed without accents, any more common accented form (*an* → *ăn*, *di* → *đi*; not *con* → *còn*, about as common). Names (capitalized mid-sentence), the language's listed pronouns (*tui*) and plain-letter words with no accented form (*email*) are left alone. Languages written in syllables only (Vietnamese).
-- **`foreign-word`**: words in the learner's own language get the target-language word, in the region and register (*market* → *chợ*, *ice cream* → *kem*). Phrases the base dictionary knows are translated as one, falling back to single words. Words the language has no word for are suggested to be left out (`leaveOut` in its checker settings: Vietnamese has no *the*, *a*, *an*). Which language each word is in comes from the two dictionaries: an English word that's Vietnamese only with accents added (*but*: *bút*) is English; Vietnamese typed without accents (*khong*) is Vietnamese; words both languages have go by how common they are in each (*em*, *di*, *tui* are Vietnamese even between English words; *the* is English), and close calls (*an*, *to*) follow their neighbors, leaning to the target language.
+- **`foreign-word`**: words in the learner's own language get the target-language word, in the region and register (*market* → *chợ*, *ice cream* → *kem*). Phrases the base dictionary knows are translated as one, falling back to single words. Words the language has no word for are suggested to be left out (`leaveOut` in its checker settings: Vietnamese has no *the*, *a*, *an*). Which language each word is in comes from the two dictionaries: an English word that's also Vietnamese once accents are added is weighed by how common each is (*rat* is *rất*, far more common than English *rat*; *but* stays English, far more common than *bút*), and close calls (*met*: *mệt* or English *met*) are Vietnamese only among Vietnamese words; Vietnamese typed without accents (*khong*) is Vietnamese; words both languages have go by how common they are in each (*em*, *di*, *tui* are Vietnamese even between English words; *the* is English), and close calls (*an*, *to*) follow their neighbors, leaning to the target language.
 
 **Opt-in (`rules`):**
 

package/dist/detect.js

@@ -7,15 +7,21 @@ import { plain } from "./text.js";
  * in each: clearly more common in the target, the target's (Vietnamese "em", "di" next to English
  * "hello", "shopping": English has rare "em", "di"); far more common in the base, the base's ("the");
  * otherwise, and for units neither knows (names), the nearest decided neighbors' language when they agree,
- * else the target's (the text is mainly the target language: "an" next to English "the"). In languages written in syllables, a base
- * word that's a target syllable only once accents are added ("but": bút) is the base's; a word only the
- * target knows without its accents ("khong") is still the target's. The base is known through the translator, so
+ * else the target's (the text is mainly the target language: "an" next to English "the"). In languages
+ * written in syllables, a base word that's a target word only once accents are added is weighed the same
+ * way, using its accented form's frequency: "rat" is "rất" (far more common than English "rat"); close
+ * calls ("met": mệt or English met) go to the target only when their neighbors are the target's ("Tôi met
+ * quá"), so English stays English among English ("kem but it was": but, not bút). A word only the target
+ * knows without its accents ("khong") is still the target's. The base is known through the translator, so
  * inflected forms count ("went", "cooked").
  */
 // Zipf points by which a word both languages know must be more common in one to count as that language's
 // whatever its neighbors: less for the target, since the text is mainly the target language.
 const TARGET_MARGIN = 1;
 const BASE_MARGIN = 2;
+// For a base word that's a target word only with accents added, a smaller lead keeps it the base's: it was
+// typed exactly as the base word ("but" is 1.9 more common in English than "bút" in Vietnamese).
+const BASE_MARGIN_ACCENTS = 1;
 export async function markLanguages(list, o) {
     const syllables = await o.target.syllables();
     const syllableLanguage = Object.keys(syllables).length > 0;
@@ -37,9 +43,18 @@ export async function markLanguages(list, o) {
         if (t === 'exact')
             return b ? 'both' : o.lang;
         if (b)
-            return o.base;
+            return t === 'accents' ? 'accents' : o.base;
         return t === 'accents' ? o.lang : '?';
     }));
+    // The most common form a plain-letter unit takes once accents are added ("rat" → rất, 6.4).
+    const accentedFrequency = (u) => {
+        const bare = plain(u.text.toLowerCase());
+        let best = 0;
+        for (const [s, f] of Object.entries(syllables))
+            if (f > best && plain(s) === bare)
+                best = f;
+        return best;
+    };
     // How common a word is in each language (Zipf; 0 when unknown).
     const targetFrequency = async (u) => {
         const lower = u.text.toLowerCase();
@@ -48,18 +63,29 @@ export async function markLanguages(list, o) {
         return (await o.target.lookup(lower))[0]?.frequency ?? 0;
     };
     const baseFrequency = async (u) => (await o.baseDictionary.lookup(u.text.toLowerCase()))[0]?.frequency ?? 0;
-    // Words both languages know, decided by frequency when it's clear.
+    // Words both languages know, decided by frequency when it's clear. Base words that are target words with
+    // accents added use that form's frequency, and close calls are marked '?accents' (decided by neighbors,
+    // more strictly).
     const decided = await Promise.all(marks.map(async (m, i) => {
-        if (m !== 'both')
+        if (m !== 'both' && m !== 'accents')
             return m;
-        const [t, b] = await Promise.all([targetFrequency(list[i]), baseFrequency(list[i])]);
-        return t - b >= TARGET_MARGIN ? o.lang : b - t >= BASE_MARGIN ? o.base : '?';
+        const [t, b] = await Promise.all([m === 'accents' ? accentedFrequency(list[i]) : targetFrequency(list[i]), baseFrequency(list[i])]);
+        if (t - b >= TARGET_MARGIN)
+            return o.lang;
+        if (b - t >= (m === 'accents' ? BASE_MARGIN_ACCENTS : BASE_MARGIN))
+            return o.base;
+        return m === 'accents' ? '?accents' : '?';
     }));
+    const known = (x) => x !== '?' && x !== '?accents';
     return decided.map((m, i) => {
-        if (m !== '?')
+        if (known(m))
             return m;
-        const left = decided.slice(0, i).reverse().find((x) => x !== '?');
-        const right = decided.slice(i + 1).find((x) => x !== '?');
+        const left = decided.slice(0, i).reverse().find(known);
+        const right = decided.slice(i + 1).find(known);
+        // A word written as typed in the base language needs every decided neighbor to be the target's to be
+        // read as the target's without its accents ("Tôi met quá": mệt); otherwise it's the base's.
+        if (m === '?accents')
+            return (left ?? o.lang) === o.lang && (right ?? o.lang) === o.lang && (left || right) ? o.lang : o.base;
         return left && right && left !== right ? o.lang : left ?? right ?? o.lang;
     });
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "which-dialect",
-  "version": "0.5.1",
+  "version": "0.5.2",
   "description": "Translate words between languages and dialects, meaning by meaning, starting with Vietnamese (Northern, Central, Southern) and English, with Wiktionary data loaded on demand. More languages are planned.",
   "type": "module",
   "main": "./dist/index.js",