which-dialect 0.5.0 → 0.5.1

package/README.md

@@ -138,7 +138,7 @@ The result has `text` (the input in Unicode NFC form), `issues` and `parts` (the
 
 **On by default (the spellchecker):**
 
-- **`spelling`**: a doubtful syllable is first tried with its neighbors: if fixing its accents makes a dictionary word with the syllables next to it, that word is suggested (*hom nay* → *hôm nay* "today", where *nay* alone would become *này* "this"; *an com* → *ăn cơm*; *Chung toi* → *Chúng tôi*), changing only the doubtful syllables. Otherwise, syllables that aren't in the language get the accented forms they could be (*khong* → *không*); rare plain-letter words next to a far more common accented form too (*toi* → *tôi*, *hom* → *hôm*), and, in a sentence typed without accents, any more common accented form (*an* → *ăn*, *di* → *đi*; not *con* → *còn*, about as common). Names (capitalized mid-sentence), the language's listed pronouns (*tui*) and plain-letter words with no accented form (*email*) are left alone. Languages written in syllables only (Vietnamese).
+- **`spelling`**: a doubtful syllable is first tried with its neighbors: if fixing its accents makes a dictionary word with the syllables next to it, that word is suggested (*hom nay* → *hôm nay* "today", where *nay* alone would become *này* "this"; *an com* → *ăn cơm*; *Chung toi* → *Chúng tôi*), changing only the doubtful syllables. Less common words written without accents can change the same way when one dictionary word clearly wins (*mua sam* → *mua sắm* "shopping", *nha sach* → *nhà sách*), even common ones when the whole word has no accents (*nha bang* → *nhà băng* "bank"), while a common one left plain next to accented syllables is taken as meant (*tháng sau* stays), *cho con* never becomes *chó con* (not a headword) and runs that could be two words equally are left alone (*ban an*: *bản án* or *bàn ăn*). Otherwise, syllables that aren't in the language get the accented forms they could be (*khong* → *không*); rare plain-letter words next to a far more common accented form too (*toi* → *tôi*, *hom* → *hôm*), and, in a sentence typed without accents, any more common accented form (*an* → *ăn*, *di* → *đi*; not *con* → *còn*, about as common). Names (capitalized mid-sentence), the language's listed pronouns (*tui*) and plain-letter words with no accented form (*email*) are left alone. Languages written in syllables only (Vietnamese).
 - **`foreign-word`**: words in the learner's own language get the target-language word, in the region and register (*market* → *chợ*, *ice cream* → *kem*). Phrases the base dictionary knows are translated as one, falling back to single words. Words the language has no word for are suggested to be left out (`leaveOut` in its checker settings: Vietnamese has no *the*, *a*, *an*). Which language each word is in comes from the two dictionaries: an English word that's Vietnamese only with accents added (*but*: *bút*) is English; Vietnamese typed without accents (*khong*) is Vietnamese; words both languages have go by how common they are in each (*em*, *di*, *tui* are Vietnamese even between English words; *the* is English), and close calls (*an*, *to*) follow their neighbors, leaning to the target language.
 
 **Opt-in (`rules`):**

package/dist/check.js

@@ -60,6 +60,15 @@ const ACCENTS_GAP_UNACCENTED_TEXT = 0.2;
 // combinations are looked up at most.
 const COMPOUND_FORMS_PER_SYLLABLE = 5;
 const COMPOUND_MAX_COMBINATIONS = 64;
+// When none of a run's syllables is doubtful on its own (all real words written without accents, "mua sam"),
+// its accented form is suggested only if one dictionary word clearly wins: ahead of the next by this many
+// Zipf points ("ban an" could be "bản án" or "bàn ăn", equally common, so it's left alone).
+const PLAIN_RUN_MARGIN = 0.5;
+// In a run that has accents elsewhere, only syllables less common than this (Zipf) may change that way: a
+// common word left without accents next to accented ones is usually meant ("tháng sau", next month, isn't
+// "tháng sáu", June). A run with no accents at all ("nha bang") was probably typed without them, so any of
+// its syllables may change ("nhà băng", bank), still only when one word clearly wins.
+const PLAIN_RUN_MAX_ZIPF = 5;
 async function spelling(ctx) {
     if (ctx.config.units !== 'syllables' || !ctx.meta.syllables)
         return [];
@@ -94,8 +103,11 @@ async function spelling(ctx) {
         };
         const suspect = (u) => unknown.has(u) || (plainWords.has(u) && betterForms(u).length > 0);
         // First, runs of syllables with a doubtful one that make a word once their accents are fixed: "hom nay"
-        // → "hôm nay", "an com" → "ăn cơm", "hôm nây" → "hôm nay". Only the doubtful syllables change. The word
-        // decides between forms a syllable alone can't ("nay" alone could be này, nảy…).
+        // → "hôm nay", "an com" → "ăn cơm", "hôm nây" → "hôm nay". The word decides between forms a syllable
+        // alone can't ("nay" alone could be này, nảy…). Syllables written without accents may change too
+        // ("mua sam" → "mua sắm": "sam" is a word, but "mua sam" isn't, and "mua sắm" is); when none of them is
+        // doubtful on its own, only if one word clearly wins. Syllables written with accents never change
+        // unless they're doubtful.
         const covered = new Set();
         const loneSet = new Set(checked);
         const maxUnits = ctx.config.maxWordUnits ?? 3;
@@ -104,11 +116,15 @@ async function spelling(ctx) {
                 const span = sentence.units.slice(i, i + k);
                 if (span.some((u, j) => !loneSet.has(u) || covered.has(u) || (j > 0 && !joined(ctx.text, span[j - 1], u))))
                     continue;
-                if (!span.some(suspect))
+                const doubtful = span.some(suspect);
+                const allPlain = span.every(isPlain);
+                const rarePlain = (u) => plainWords.has(u) && (allPlain || (syllables[lowerOf(u)] ?? 0) < PLAIN_RUN_MAX_ZIPF);
+                const mayChange = (u) => suspect(u) || rarePlain(u);
+                if (!doubtful && !span.some(rarePlain))
                     continue;
                 const written = span.map(lowerOf);
                 const combos = accentCombinations(written, byPlain, {
-                    fixed: span.map((u) => !suspect(u)),
+                    fixed: span.map((u) => !mayChange(u)),
                     perUnit: COMPOUND_FORMS_PER_SYLLABLE,
                     max: COMPOUND_MAX_COMBINATIONS,
                 }).filter((c) => c !== written.join(' '));
@@ -119,12 +135,14 @@ async function spelling(ctx) {
                     .slice(0, 3);
                 if (!found.length)
                     continue;
+                if (!doubtful && found.length > 1 && found[0].frequency - found[1].frequency < PLAIN_RUN_MARGIN)
+                    continue;
                 const at = { start: span[0].start, end: span[k - 1].end, text: ctx.text.slice(span[0].start, span[k - 1].end) };
                 const suggestions = found.map((f) => matchCase(at.text, f.word));
                 const quoted = listOf(suggestions.map(quote), 'or');
                 issues.push({
                     rule: 'spelling', severity: 'warning', ...at, suggestions,
-                    message: span.filter(suspect).every(isPlain)
+                    message: span.filter(mayChange).every(isPlain)
                         ? `${quote(at.text)} looks typed without accents. Did you mean ${quoted}?`
                         : `${quote(at.text)} isn't spelled right. Did you mean ${quoted}?`,
                 });

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "which-dialect",
-  "version": "0.5.0",
+  "version": "0.5.1",
   "description": "Translate words between languages and dialects, meaning by meaning, starting with Vietnamese (Northern, Central, Southern) and English, with Wiktionary data loaded on demand. More languages are planned.",
   "type": "module",
   "main": "./dist/index.js",