which-dialect 0.2.0 → 0.4.0

package/README.md

@@ -111,43 +111,97 @@ The cases involving Spanish were removed with it. The held-out set has since bee
 
 ### `createChecker({ baseUrl?, load?, dictionary? })` → `checker.check(text, options)`
 
-Checks a learner's sentence in one language and returns issues with the exact place in the text, a short explanation and suggested fixes. It runs in the browser like the rest of the library: no API calls, no cost.
+A spellchecker for learners: it gives a correction for every word it can, with the exact place in the text and a short reason. Words written in the learner's own language (English by default) get the target-language word. Further checks (regions, pronouns, politeness) can be turned on. It runs in the browser like the rest of the library: no API calls, no cost.
 
 ```ts
 import { createChecker } from 'which-dialect'
 
 const checker = createChecker()
-const { text, issues } = await checker.check('Tôi muốn ăn thịt lợn và bắp khong?', {
-  lang: 'vi', region: 'Southern', listener: 'parent',
+const { text, issues, parts } = await checker.check('Toi muon an the ice cream but it was very expensive.', {
+  lang: 'vi', region: 'Southern',
 })
-// “Tôi”: Talking to your parents, say “con” for “I”, not “Tôi”.               → Con
-// “lợn”: “lợn” is Northern; in Southern Vietnamese, say “heo”.                → heo, cúi, ỉn
-// “khong”: “khong” isn't a Vietnamese syllable. Did you mean “không”, “khổng” or “khống”?
-// “khong”: Talking to your parents, end the sentence with “ạ” to sound polite. (suggestion)
+// Toi → Tôi · muon → muốn · an → ăn          (accents)
+// the → (left out: Vietnamese has no "the") · ice cream → kem · but → nhưng · it → nó
+// was → là · very → rất · expensive → mắc    (English words, in Southern Vietnamese)
 ```
 
 | Option | Meaning |
 |---|---|
-| `lang` | The text's language. |
-| `region` | The region the text should be in. Without it, words that don't fit the rest of the text's region are pointed out (*lợn*, Northern, next to *nha*, Southern). |
-| `listener` | Who you're talking to, as a [pronoun-table](#looking-words-up) row id (`'parent'`, `'teacher'`, `'friend'`, …): pronouns and polite endings are checked against it. |
-| `about` | Who you're talking about, for "he/she" and "they" pronouns. |
-| `speaker` | `'male'` or `'female'`, for pronouns that depend on it. |
-| `rules` | Turn checks off, e.g. `{ 'polite-ending': false }`. |
+| `lang` | The text's language (the target). |
+| `base` | The learner's own language, whose words get translated (default `'en'`). |
+| `region` | The region's words are used for translations (*expensive* → Southern *mắc*, Northern *đắt*), and the `dialect` check, when on, flags other regions' words. |
+| `register` | `'casual'` allows colloquial translations; otherwise plain ones come first. |
+| `rules` | Turn checks on or off. By default only `spelling` and `foreign-word` run; e.g. `{ dialect: true, 'pronoun-relationship': true }`. |
+| `listener`, `about`, `speaker` | For the pronoun and polite-ending checks: who you're talking to and about (pronoun-table row ids), and the speaker's gender. |
 
-Each issue has `rule`, `severity` (`'error'`: not a word of the language; `'warning'`: wrong for the chosen region or relationship; `'suggestion'`: often better), `start`/`end` (offsets into the returned `text`, the input in Unicode NFC form), `text`, `message` and `suggestions` (replacements for `text`, best first).
+The result has `text` (the input in Unicode NFC form), `issues` and `parts` (the text split by language). Each issue has `rule`, `severity` (`'error'`: not a word of the language; `'warning'`: probably wrong, or wrong for the region or relationship; `'suggestion'`: often better), `start`/`end` (offsets into `text`), `text`, `message` and `suggestions` (replacements for `text`, best first; `''` means leave it out).
 
-The checks:
+**On by default (the spellchecker):**
 
-- **`spelling`**: syllables that aren't in the language, with the accented forms they could be (*khong* → *không*). Plain-letter words with no accented form are left alone (names, foreign words). Languages written in syllables only (Vietnamese).
-- **`dialect`**: words whose main meaning is tagged for another region (*lợn* in Southern text → *heo*; *muỗng* in Northern → *thìa*; in English, US *apartment* in UK text → *flat*). Words with an untagged main meaning are never flagged (*má* is "cheek" everywhere).
+- **`spelling`**: syllables that aren't in the language get the accented forms they could be (*khong* → *không*); rare plain-letter words next to a far more common accented form too (*toi* → *tôi*, *hom* → *hôm*), and, in a sentence typed without accents, any more common accented form (*an* → *ăn*, *di* → *đi*; not *con* → *còn*, about as common). Names (capitalized mid-sentence), the language's listed pronouns (*tui*) and plain-letter words with no accented form (*email*) are left alone. Languages written in syllables only (Vietnamese).
+- **`foreign-word`**: words in the learner's own language get the target-language word, in the region and register (*market* → *chợ*, *ice cream* → *kem*). Phrases the base dictionary knows are translated as one, falling back to single words. Words the language has no word for are suggested to be left out (`leaveOut` in its checker settings: Vietnamese has no *the*, *a*, *an*). Which language each word is in comes from the two dictionaries: an English word that's Vietnamese only with accents added (*but*: *bút*) is English; Vietnamese typed without accents (*khong*) is Vietnamese; words that could be either (*an*, *tui*) follow their neighbors, leaning to the target language.
+
+**Opt-in (`rules`):**
+
+- **`dialect`**: words whose main meaning is tagged for another region (*lợn* in Southern text → *heo*; *muỗng* in Northern → *thìa*; in English, US *apartment* in UK text → *flat*). Without `region`, words that don't fit the rest of the text's region are pointed out. Words with an untagged main meaning are never flagged (*má* is "cheek" everywhere).
 - **`pronoun-relationship`**: pronouns that don't fit the listener (*tôi* to a parent → *con*; to a teacher → *em*).
 - **`pronoun-pair`**: "I" and "you" pronouns that never go together (*tao* goes with *mày*, not *bạn*), when no listener is given.
+- **`pronoun-consistency`**: switching words for the same person within one text (*tôi* in one sentence, *tui* in the next): keep the first.
 - **`polite-ending`**: a suggestion to end sentences with *ạ* when talking to parents, elders and teachers (not for exclamations).
 
-The checker only speaks up when it's confident, so **no issues means none of the checks found anything, not that the text is correct**. It doesn't check word order, classifiers or whether a sentence sounds natural yet. Only reliable pronouns are checked: kinship words like *con* (also "child", and a classifier in *con chó*) aren't, and *bạn* (also "friend") only gets suggestions.
+Corrections are word by word: they don't fix word order or grammar, so English translated word by word reads like English in Vietnamese words (*it was very expensive* → *nó là rất mắc*). Whole English sentences or clauses that follow a [sentence frame](#sentence-frames) get the frame's natural wording instead in the [journal review](#reviewing-journal-entries). The checker only speaks up when it's confident, so **no issues means none of the checks found anything, not that the text is correct**.
+
+**Every language uses the same checks.** They read only the language's data and its `checker` settings in `languages/<code>.ts` (how text splits into words, which pronouns are reliable, polite endings, words with no equivalent), so a check whose data a language doesn't have is skipped. A language without checker settings is read as space-separated words: English words in it are still translated, and the `dialect` check works.
+
+## Sentence frames
+
+Common sentences with slots, filled for a region, a listener and a register: *Where is {place}?* is *{place} ở **đâu**?*, and in Central Vietnamese *{place} ở **mô**?*; *Have you eaten yet?* to a parent in the South is *Ba ăn cơm chưa ạ?*, to a friend *Bạn ăn cơm chưa?*.
+
+```ts
+import { createPhrasebook } from 'which-dialect'
+
+const pb = createPhrasebook()
+await pb.frames({ lang: 'vi', region: 'Southern', topic: 'questions' })       // every frame, slots open
+await pb.render('ask-where', { lang: 'vi', region: 'Central', slots: { place: 'bathroom' } })
+// → { text: 'Phòng tắm ở mô?', parts: [{ text: 'Phòng tắm', kind: 'content', from: 'bathroom' }, …,
+//      { text: 'mô', kind: 'word', why: 'where (Central)' }, …], complete: true }
+await pb.match('Do you want coffee?', { lang: 'vi' })                         // → 'Bạn có muốn cà phê không?'
+await pb.matchFrames('Bạn có đi?', { lang: 'vi' })                            // → 'Bạn có đi không?' (missing "không")
+```
+
+Options: `lang`, `region`, `listener` (a pronoun-table row id; without it, the neutral default row), `speaker`, `register` (`'casual'` allows colloquial words like Southern *hông*, *tui*). Content slots take English (translated for you, in the region) or `{ text }` in the language itself. `parts` says why each word was chosen, for highlighting. `matchFrames` recognizes a frame in a sentence in the language itself (the journal review uses it to list the frames a sentence follows) and returns the changes the frame would suggest: a missing required word (*không*), a word from another region (*mô* in Southern text → *đâu*), missing accents in a frame position (*Toi* → *Tôi*), and, only when a listener is given, a pronoun that doesn't fit them.
+
+The frames themselves are written once, in English, in [`languages/frames.ts`](languages/frames.ts) (34 so far: journal sentences, feelings, questions, requests, greetings). Each language says how it expresses them (`frames` in `languages/<code>.ts`), with pronoun slots (`{I}`, `{YOU}`, `{WE_INCL}`, …, filled from the pronoun table) and **frame words** for what changes by region (`frameWords`: Vietnamese `{WHERE}` *đâu* / Central *mô*, `{HOW}` Northern *thế nào* / *sao* / Central *răng*, `{SOFT}` Northern *nhé* / Central and Southern *nha*, `{Q_END}` *không* / Southern casual *hông*, `{POLITE}` *ạ* when speaking up). Each frame word comes from a dictionary definition, keeping its region and labels, or from an override with a note, like the pronoun table. A language without frames returns none.
+
+## Reviewing journal entries
+
+`createReviewer().review(entry, { lang, base?, region?, register?, checks?, listener?, speaker? })` reviews an entry written in the target language, the learner's own language (`base`, default `'en'`), or a mix of both, and returns the corrected entry, in the target language, sentence by sentence:
+
+```ts
+import { createReviewer } from 'which-dialect'
+
+const review = await createReviewer().review(
+  'Hôm nay tôi đi market với má. Tôi muốn ăn thịt lợn but it was expensive. Ngày mai tui đi lại khong?',
+  { lang: 'vi', region: 'Southern' },
+)
+review.corrected
+// 'Hôm nay tôi đi chợ với má. Tôi muốn ăn thịt lợn nhưng mắc quá. Ngày mai tui đi lại không?'
+review.sentences[1].changes
+// [{ from: 'but it was expensive', to: 'nhưng mắc quá', kind: 'frame', why: 'Sentence frame “but it was {quality}”' }]
+
+// With the regional and consistency checks on:
+await createReviewer().review(entry, { lang: 'vi', region: 'Southern', checks: { dialect: true, 'pronoun-consistency': true } })
+// 'Hôm nay tôi đi chợ với má. Tôi muốn ăn thịt heo nhưng mắc quá. Ngày mai tôi đi lại không?'
+```
+
+For each sentence:
+
+1. **Each word is marked as the target or the base language** (`parts`), as in the checker's `foreign-word` check.
+2. **A whole base-language sentence or clause that follows a sentence frame is filled in** with the frame's wording (*but it was expensive* → *nhưng mắc quá*; *Where is the bathroom?* → *Phòng tắm ở đâu?*).
+3. **The checker runs on the whole entry**: by default the spellchecker, so every remaining English word gets its translation and every misspelled word its accents; `checks` turns on more (regions, pronouns, consistency across sentences, polite endings). Its errors and warnings are applied (`changes`); its suggestions, like ending with *ạ*, are returned as `hints`.
+4. **The frames the corrected sentence follows are listed** (`frames`), for suggesting patterns to reuse. Frames don't change the sentence.
 
-**Every language uses the same checks.** They read only the language's data and its `checker` settings in `languages/<code>.ts` (how text splits into words, which pronouns are reliable, polite endings), so a check whose data a language doesn't have is skipped. A language without checker settings still gets the `dialect` check.
+Each sentence has `original`, `parts`, `corrected`, `changes` (`{ from, to, why, kind }`; `to` is `''` when a word is left out), `hints`, `frames` (`{ id, en, text }`, the frame's English and its pattern) and `unchecked` (English words with no translation, left as written). For a journal, leave `listener` out: people written about (*má*, *thầy*) aren't treated as the listener. Corrections are word by word except for frames: word order and grammar aren't fixed (*I am very happy* → *Tôi là rất mừng*).
 
 ## Looking words up
 
@@ -157,6 +211,7 @@ One language's dictionary. `baseUrl` defaults to `https://cdn.jsdelivr.net/npm/w
 
 - `dictionary.lookup(word)`: all entries for a word, with every sense's definitions, regions, labels, synonyms and variant links (`altOf`).
 - `dictionary.searchEnglish(term, { region?, pos?, exclude?, limit?, allSenses? })`: words for an English term in this language, one per word, best first (no meaning handling: use the translator for that). `allSenses: true` returns every matching sense of those words instead.
+- `dictionary.frames()`: the language's sentence frames and frame words, unfilled (`{ frames, words }`; empty without frames).
 - `dictionary.checker()` / `dictionary.syllables()`: the language's checker settings and syllable list (with frequencies), used by `createChecker` (`{}` for languages without them).
 - `dictionary.picks()`: the language's [hand-picked words](#hand-picked-words), `{ word, pos, gloss, picks: [{ word, tags? }], first? }` per English meaning (empty for languages without any).
 - `dictionary.meta()`: the language's name, regions, region groups, source, license, build date and counts.
@@ -255,7 +310,8 @@ Generated data isn't committed (it would bloat git history); it's built before p
 3. Add the language's code to `translationLangs` in `languages/en.ts` (so English keeps its translation tables for it) and rebuild English.
 4. `npm run build:data -- <code>`, check the results, add a few real-data tests and evaluation cases, and publish.
 5. Optional: [checker](#checking-text) settings (`checker` in `languages/<code>.ts`): `units: 'syllables'` for languages written in syllables (also builds the syllable list for accent suggestions), `maxWordUnits`, the reliable `pronouns` per pronoun-table column, `ambiguousPronouns`, and `politeEndings`. Without them, the language is read as space-separated words and gets the region check.
-6. Optional, later: [hand-picked words](#hand-picked-words). A speaker runs `npm run picks-sheet -- <code>`, fills in `your_pick` only where the first choice is wrong, and pastes the `--apply` output into `picks`.
+6. Optional: [sentence frames](#sentence-frames): `frames` (how the language says each frame in `languages/frames.ts`) and `frameWords` (the words that change by region, each from a definition or an override with a note). Frames reach other languages through the shared catalog; a new frame goes in the catalog once.
+7. Optional, later: [hand-picked words](#hand-picked-words). A speaker runs `npm run picks-sheet -- <code>`, fills in `your_pick` only where the first choice is wrong, and pastes the `--apply` output into `picks`.
 
 Languages in non-Latin scripts (Chinese, Arabic, Russian, …) will need a script-aware version of `shardKey` first: today files are split by the first two Latin letters.
 

package/dist/check.d.ts

@@ -1,5 +1,10 @@
 import { type TranslatorOptions } from './translate.ts';
-export type CheckRule = 'spelling' | 'dialect' | 'pronoun-relationship' | 'pronoun-pair' | 'polite-ending';
+export type CheckRule = 'spelling' | 'foreign-word' | 'dialect' | 'pronoun-relationship' | 'pronoun-pair' | 'pronoun-consistency' | 'polite-ending';
+/**
+ * Which checks run unless `rules` says otherwise: the spellchecker (spelling, and words in the learner's
+ * own language, with their translation). The rest are opt-in, e.g. `rules: { dialect: true }`.
+ */
+export declare const DEFAULT_CHECK_RULES: Record<CheckRule, boolean>;
 export type CheckIssue = {
     rule: CheckRule;
     /**
@@ -20,15 +25,28 @@ export type CheckIssue = {
 export type CheckOptions = {
     /** Language code of the text, e.g. "vi". */
     lang: string;
-    /** Region or region group the text should be in, e.g. "Southern". Without it, mixed regions are flagged. */
+    /**
+     * The learner's own language, whose words in the text are translated (`foreign-word`). Default "en";
+     * none when it's the text's language.
+     */
+    base?: string;
+    /**
+     * Region or region group the text should be in, e.g. "Southern": translations use its words, and the
+     * `dialect` check flags other regions' words (without it, mixed regions).
+     */
     region?: string;
+    /** 'casual' allows colloquial translations (Southern "tui"); otherwise plain ones come first. */
+    register?: 'casual' | 'neutral' | 'polite';
     /** Who you're talking to, as a pronoun-table row id (see Dictionary.pronouns), e.g. "parent". */
     listener?: string;
     /** Who you're talking about, as a pronoun-table row id, for "he/she" and "they" pronouns. */
     about?: string;
     /** The speaker's gender, for pronouns that depend on it. */
     speaker?: 'male' | 'female';
-    /** Turn rules off, e.g. { 'polite-ending': false }. All are on by default. */
+    /**
+     * Turn checks on or off. By default only the spellchecker runs (DEFAULT_CHECK_RULES: `spelling` and
+     * `foreign-word`); opt in to others, e.g. { dialect: true, 'pronoun-relationship': true }.
+     */
     rules?: Partial<Record<CheckRule, boolean>>;
 };
 export type CheckResult = {
@@ -36,6 +54,13 @@ export type CheckResult = {
     text: string;
     /** Sorted by position. */
     issues: CheckIssue[];
+    /** The text split by language (the target and the base), in order; one part when no base is given. */
+    parts: {
+        text: string;
+        lang: string;
+        start: number;
+        end: number;
+    }[];
 };
 export type Checker = {
     check(text: string, options: CheckOptions): Promise<CheckResult>;

package/dist/check.js

@@ -1,16 +1,33 @@
-// Checking text in one language: spelling and accents, words from another region, and pronouns and
-// polite endings that fit who you're talking to.
+// Checking text in one language. By default it's a spellchecker that gives a correction for every word it
+// can: syllables that aren't in the language get their accented forms ("khong" → "không"), and words in the
+// learner's own language (English by default) get the target-language word ("market" → "chợ"). Further
+// checks are opt-in (`rules`): words from another region, pronouns and polite endings that fit who you're
+// talking to, and consistency within the text.
 //
 // Every rule reads only the language's data: its dictionary, pronoun table, checker.json and
 // syllables.json (see CheckerConfig). Adding a language means adding data and settings in
-// languages/<lang>.ts, not code here. A rule whose data a language doesn't have is skipped, so a language
-// without checker settings is still checked for words from another region.
+// languages/<lang>.ts, not code here. A rule whose data a language doesn't have is skipped.
 //
 // The checker only speaks up when it's confident and stays silent otherwise, so no issues means none of
 // the checks found anything, not that the text is correct.
 import { createDictionary, resolveRegion } from "./index.js";
+import { joined, matchCase, plain, segment, sentencesOf, units } from "./text.js";
+import { markLanguages } from "./detect.js";
 import { createTranslator } from "./translate.js";
 import { PRONOUN_PERSONS, } from "./types.js";
+/**
+ * Which checks run unless `rules` says otherwise: the spellchecker (spelling, and words in the learner's
+ * own language, with their translation). The rest are opt-in, e.g. `rules: { dialect: true }`.
+ */
+export const DEFAULT_CHECK_RULES = {
+    spelling: true,
+    'foreign-word': true,
+    dialect: false,
+    'pronoun-relationship': false,
+    'pronoun-pair': false,
+    'pronoun-consistency': false,
+    'polite-ending': false,
+};
 // How each pronoun-table column is named in messages.
 const PERSON_NAMES = {
     self: 'I', addressee: 'you', third: 'he/she', selfPlural: 'we', addresseePlural: 'you (plural)', thirdPlural: 'they',
@@ -19,65 +36,24 @@ const PERSON_NAMES = {
 const ABOUT_PERSONS = new Set(['third', 'thirdPlural']);
 const SELF_PERSONS = new Set(['self', 'selfPlural']);
 const ADDRESSEE_PERSONS = new Set(['addressee', 'addresseePlural']);
-// A run of letters (with their accents), digits and apostrophes: a word, or a syllable in syllable languages.
-const UNIT = /[\p{L}\p{M}\p{N}'’]+/gu;
-const SENTENCE_END = /[.!?…]+|\n+/g;
 const quote = (s) => `“${s}”`;
 const lowerFirst = (s) => s.charAt(0).toLowerCase() + s.slice(1);
 const listOf = (items, conjunction = 'and') => items.length <= 1 ? items.join('') : `${items.slice(0, -1).join(', ')} ${conjunction} ${items[items.length - 1]}`;
-// Keeps a capital first letter when replacing a capitalized word ("Tôi" → "Con").
-const matchCase = (original, replacement) => original.charAt(0) !== original.charAt(0).toLowerCase() ? replacement.charAt(0).toUpperCase() + replacement.slice(1) : replacement;
-// Without accents, for finding the accented forms of a syllable: "không" → "khong", "đi" → "di".
-const plain = (s) => s.normalize('NFD').replace(/\p{M}/gu, '').replace(/đ/g, 'd').replace(/Đ/g, 'D').normalize('NFC');
 const unique = (items) => [...new Set(items)];
-function units(text) {
-    return [...text.matchAll(UNIT)].map((m) => ({ text: m[0], start: m.index, end: m.index + m[0].length }));
-}
-function sentencesOf(text, all) {
-    const ends = [...text.matchAll(SENTENCE_END)].map((m) => m.index);
-    const out = [[]];
-    let next = 0;
-    for (const u of all) {
-        while (next < ends.length && ends[next] < u.start) {
-            if (out[out.length - 1].length)
-                out.push([]);
-            next++;
-        }
-        out[out.length - 1].push(u);
-    }
-    return out.filter((s) => s.length);
-}
-// Units join into one word only when just spaces separate them ("thịt heo", not "thịt, heo").
-const joined = (text, a, b) => /^[^\S\n]+$/.test(text.slice(a.end, b.start));
-// Longest match against the dictionary: at each position, the longest run of units that's a headword
-// ("kết quả" is one word, so its "quả" isn't checked on its own). Unknown single units become words with
-// no entries.
-async function segment(text, list, maxUnits, lookup) {
-    const words = [];
-    for (let i = 0; i < list.length;) {
-        let taken = null;
-        for (let k = Math.min(maxUnits, list.length - i); k >= 1; k--) {
-            const span = list.slice(i, i + k);
-            if (span.some((u, j) => j > 0 && !joined(text, span[j - 1], u)))
-                continue;
-            const phrase = span.map((u) => u.text).join(' ');
-            const entries = /\d/.test(phrase) ? [] : await lookup(phrase);
-            if (entries.length || k === 1) {
-                const start = span[0].start;
-                const end = span[k - 1].end;
-                taken = { text: text.slice(start, end), lower: phrase.toLowerCase(), start, end, units: span, entries };
-                break;
-            }
-        }
-        words.push(taken);
-        i += taken.units.length;
-    }
-    return words;
-}
 // --- Rules -----------------------------------------------------------------------------------------------
-// Spelling and accents, for languages written in syllables with a syllable list: a syllable that isn't in
-// the language is flagged, with the accented forms it could be ("khong" → "không"). Plain-letter
-// syllables with no accented form are left alone: usually a name or a foreign word ("email").
+// Spelling and accents, for languages written in syllables with a syllable list (with frequencies):
+// - a syllable that isn't in the language is flagged, with the accented forms it could be ("khong" →
+//   "không"); plain-letter syllables with no accented form are left alone (names, foreign words: "email");
+// - a plain-letter syllable that is a word but a rare one, next to a much more common accented form, is
+//   probably missing its accents ("toi" → "tôi", "hom" → "hôm"). In a sentence typed without accents (at
+//   least half plain letters and a missing-accents error already, or 3+ words with no accent at all; half,
+//   because filled-in sentence frames add accented words), any accented form that's more common
+//   counts ("an" → "ăn"; but not "con" → "còn", about as common). Left alone: capitalized words
+//   mid-sentence (names) and the language's listed pronouns, some written without accents ("tui").
+// Units inside a known word, or in the learner's own language, aren't checked.
+// Zipf points by which an accented form must be more common than a plain-letter word to be suggested.
+const ACCENTS_GAP = 1.5;
+const ACCENTS_GAP_UNACCENTED_TEXT = 0.2;
 async function spelling(ctx) {
     if (ctx.config.units !== 'syllables' || !ctx.meta.syllables)
         return [];
@@ -88,29 +64,111 @@ async function spelling(ctx) {
     for (const list of byPlain.values())
         list.sort((a, b) => syllables[b] - syllables[a]);
     const issues = [];
-    for (const word of ctx.sentences.flatMap((s) => s.words)) {
-        if (word.entries.length)
-            continue;
-        for (const u of word.units) {
+    const suggest = (u, candidates, severity, message) => issues.push({ rule: 'spelling', severity, start: u.start, end: u.end, text: u.text, message, suggestions: candidates.map((c) => matchCase(u.text, c)) });
+    const options = (u, candidates) => listOf(candidates.map((c) => quote(matchCase(u.text, c))), 'or');
+    const pronouns = new Set(Object.values(ctx.config.pronouns ?? {}).flat().map((w) => w.toLowerCase()));
+    for (const sentence of ctx.sentences) {
+        const lone = sentence.words.filter((w) => !w.entries.length || w.units.length === 1).flatMap((w) => w.units).filter((u) => !ctx.inBase(u));
+        const target = sentence.units.filter((u) => !ctx.inBase(u) && /\p{L}/u.test(u.text));
+        const mostlyPlain = target.filter((u) => u.text.toLowerCase() === plain(u.text.toLowerCase())).length >= target.length * 0.5;
+        let missingAccents = false;
+        const rare = [];
+        for (const u of lone) {
             const lower = u.text.toLowerCase().replace(/[’']/g, '');
-            if (/\d/.test(lower) || lower in syllables)
+            if (/\d/.test(lower))
                 continue;
-            const candidates = (byPlain.get(plain(lower)) ?? []).slice(0, 3);
-            const accented = lower !== plain(lower);
-            if (!candidates.length && !accented)
+            const inWord = sentence.words.some((w) => w.entries.length && w.units.length > 1 && w.units.includes(u));
+            if (inWord)
                 continue;
-            issues.push({
-                rule: 'spelling',
+            if (!(lower in syllables)) {
+                const candidates = (byPlain.get(plain(lower)) ?? []).slice(0, 3);
+                const accented = lower !== plain(lower);
+                if (!candidates.length && !accented)
+                    continue;
+                if (!accented)
+                    missingAccents = true;
                 // Missing accents on plain letters is the common learner slip; a wrong accent is more surely wrong.
-                severity: accented ? 'error' : 'warning',
-                start: u.start,
-                end: u.end,
-                text: u.text,
-                message: candidates.length
-                    ? `${quote(u.text)} isn't a ${ctx.meta.name} syllable. Did you mean ${listOf(candidates.map((c) => quote(matchCase(u.text, c))), 'or')}?`
-                    : `${quote(u.text)} isn't a ${ctx.meta.name} syllable. Check the spelling and accents.`,
-                suggestions: candidates.map((c) => matchCase(u.text, c)),
-            });
+                suggest(u, candidates, accented ? 'error' : 'warning', candidates.length
+                    ? `${quote(u.text)} isn't a ${ctx.meta.name} syllable. Did you mean ${options(u, candidates)}?`
+                    : `${quote(u.text)} isn't a ${ctx.meta.name} syllable. Check the spelling and accents.`);
+            }
+            else if (lower === plain(lower) && !pronouns.has(lower) && (u === sentence.units[0] || !/\p{Lu}/u.test(u.text.charAt(0)))) {
+                rare.push(u);
+            }
+        }
+        // Typed without accents: mostly plain letters, and an accent error already, or 3+ words with no accent at all.
+        const unaccented = mostlyPlain && (missingAccents || (target.length >= 3 && target.every((u) => u.text.toLowerCase() === plain(u.text.toLowerCase()))));
+        for (const u of rare) {
+            const lower = u.text.toLowerCase();
+            const gap = unaccented ? ACCENTS_GAP_UNACCENTED_TEXT : ACCENTS_GAP;
+            const better = (byPlain.get(lower) ?? []).filter((c) => c !== lower && syllables[c] >= syllables[lower] + gap - 1e-9).slice(0, 3);
+            if (!better.length)
+                continue;
+            suggest(u, better, 'warning', unaccented
+                ? `${quote(u.text)} looks typed without accents. Did you mean ${options(u, better)}?`
+                : `${quote(u.text)} is a rare word; did you mean ${options(u, better)}?`);
+        }
+    }
+    return issues;
+}
+// Words in the learner's own language (English by default), with the target-language word to use: "market"
+// → "chợ". Runs of base units are split into the longest phrases the base dictionary knows ("ice cream"),
+// each translated in the region and register. Words the language usually has no word for
+// (CheckerConfig.leaveOut: Vietnamese has no articles) are suggested to be left out.
+async function foreignWord(ctx) {
+    const base = ctx.base;
+    if (!base)
+        return [];
+    const leaveOut = new Set((ctx.config.leaveOut ?? []).map((w) => w.toLowerCase()));
+    const issues = [];
+    for (const s of ctx.sentences) {
+        const runs = [];
+        s.units.forEach((u, i) => {
+            if (!ctx.inBase(u))
+                return;
+            const last = runs[runs.length - 1];
+            if (last && last[last.length - 1] === s.units[i - 1] && joined(ctx.text, s.units[i - 1], u))
+                last.push(u);
+            else
+                runs.push([u]);
+        });
+        for (const run of runs) {
+            for (let i = 0; i < run.length;) {
+                // The longest phrase the base dictionary knows and the language has a word for; single words
+                // always give an issue, with or without a translation.
+                let found;
+                for (let k = Math.min(MAX_FOREIGN_UNITS, run.length - i); k >= 1 && !found; k--) {
+                    const phrase = ctx.text.slice(run[i].start, run[i + k - 1].end);
+                    if (k === 1 && leaveOut.has(phrase.toLowerCase())) {
+                        found = { k, words: [], leave: true };
+                        break;
+                    }
+                    if (k > 1 && !(await ctx.translator.senses(phrase, { from: base }).catch(() => [])).length)
+                        continue;
+                    const [group] = await ctx.translator.translate(phrase, {
+                        from: base, to: ctx.options.lang, toRegion: ctx.options.region, register: ctx.options.register, listener: ctx.options.listener, limit: 3,
+                    }).catch(() => []);
+                    const words = unique((group?.translations ?? []).map((t) => t.word));
+                    if (words.length || k === 1)
+                        found = { k, words, leave: false };
+                }
+                const { k, words, leave } = found;
+                const span = run.slice(i, i + k);
+                const text = ctx.text.slice(span[0].start, span[k - 1].end);
+                i += k;
+                const at = { rule: 'foreign-word', severity: 'error', start: span[0].start, end: span[k - 1].end, text };
+                if (leave) {
+                    issues.push({ ...at, message: `${quote(text)} isn't ${ctx.meta.name}; ${ctx.meta.name} usually has no word for it, so leave it out.`, suggestions: [''] });
+                    continue;
+                }
+                issues.push({
+                    ...at,
+                    message: words.length
+                        ? `${quote(text)} isn't ${ctx.meta.name}: say ${listOf(words.map(quote), 'or')}.`
+                        : `${quote(text)} isn't ${ctx.meta.name}, and no translation was found.`,
+                    suggestions: words.map((w) => (span[0].start === s.units[0].start ? matchCase(text, w) : w)),
+                });
+            }
         }
     }
     return issues;
@@ -123,6 +181,7 @@ async function dialect(ctx) {
     const all = new Set(ctx.meta.regions);
     const regional = ctx.sentences
         .flatMap((s) => s.words)
+        .filter((word) => !word.units.some((u) => ctx.inBase(u)))
         .map((word) => ({ word, sense: word.entries[0]?.senses[0] }))
         .filter((x) => Boolean(x.sense?.regionTagged && x.sense.regions.length < all.size));
     if (!regional.length)
@@ -233,6 +292,33 @@ async function pronounPair(ctx) {
     }
     return issues;
 }
+// Switching words for the same person within one text ("tôi" in one sentence, "tui" in the next): the first
+// one used is kept. Useful for journals and longer messages. Ambiguous pronouns ("bạn": also "friend") are
+// never the ones flagged as a warning.
+async function pronounConsistency(ctx) {
+    const first = new Map();
+    const issues = [];
+    for (const m of ctx.pronounMatches()) {
+        const earlier = first.get(m.person);
+        if (!earlier) {
+            first.set(m.person, m);
+            continue;
+        }
+        if (earlier.lower === m.lower)
+            continue;
+        const ambiguous = ctx.config.ambiguousPronouns?.includes(m.lower) || ctx.config.ambiguousPronouns?.includes(earlier.lower);
+        issues.push({
+            rule: 'pronoun-consistency',
+            severity: ambiguous ? 'suggestion' : 'warning',
+            start: m.start,
+            end: m.end,
+            text: m.text,
+            message: `You used ${quote(earlier.text)} for “${PERSON_NAMES[m.person]}” earlier; keep one word: ${quote(earlier.lower)}.`,
+            suggestions: [matchCase(m.text, earlier.lower)],
+        });
+    }
+    return issues;
+}
 // Polite endings when speaking up (the listener's row is marked `respect`): Vietnamese sentences to parents,
 // elders and teachers usually end in "ạ". A suggestion, not an error: it isn't always needed.
 async function politeEnding(ctx) {
@@ -262,11 +348,15 @@ async function politeEnding(ctx) {
     }
     return issues;
 }
+// The longest base-language phrase looked up as one ("ice cream", "take care of").
+const MAX_FOREIGN_UNITS = 3;
 const RULES = {
     spelling,
+    'foreign-word': foreignWord,
     dialect,
     'pronoun-relationship': pronounRelationship,
     'pronoun-pair': pronounPair,
+    'pronoun-consistency': pronounConsistency,
     'polite-ending': politeEnding,
 };
 const SEVERITY_ORDER = { error: 0, warning: 1, suggestion: 2 };
@@ -303,10 +393,13 @@ export function createChecker(options = {}) {
                 }
                 return pending;
             };
+            const base = (opts.base ?? 'en') === opts.lang ? undefined : (opts.base ?? 'en');
             const sentences = [];
             for (const list of sentencesOf(text, units(text))) {
-                sentences.push({ units: list, words: await segment(text, list, config.maxWordUnits ?? 3, lookup) });
+                const langs = base ? await markLanguages(list, { lang: opts.lang, base, target: d, baseDictionary: dict(base), translator }) : list.map(() => opts.lang);
+                sentences.push({ units: list, langs, words: await segment(text, list, config.maxWordUnits ?? 3, lookup) });
             }
+            const baseUnits = new Set(sentences.flatMap((s) => s.units.filter((_, i) => s.langs[i] !== opts.lang).map((u) => u.start)));
             let rows;
             let matches;
             const ctx = {
@@ -320,6 +413,9 @@ export function createChecker(options = {}) {
                 // The language's reliable pronouns in the text, longest first ("chúng tôi" before "tôi"), matched on
                 // units directly so pronouns the dictionary doesn't list as one word ("tụi mày") are found too.
                 pronounMatches: () => (matches ??= findPronouns(text, sentences, config)),
+                base,
+                inBase: (u) => baseUnits.has(u.start),
+                translator,
                 suggestRegional: async (word, region) => {
                     try {
                         const [group] = await translator.translate(word.lower, {
@@ -332,9 +428,22 @@ export function createChecker(options = {}) {
                     }
                 },
             };
-            const results = await Promise.all(Object.keys(RULES).filter((r) => opts.rules?.[r] !== false).map((r) => RULES[r](ctx)));
+            const results = await Promise.all(Object.keys(RULES).filter((r) => opts.rules?.[r] ?? DEFAULT_CHECK_RULES[r]).map((r) => RULES[r](ctx)));
             const issues = results.flat().sort((a, b) => a.start - b.start || SEVERITY_ORDER[a.severity] - SEVERITY_ORDER[b.severity]);
-            return { text, issues };
+            // The text split by language: consecutive units in one language, with the text between them.
+            const parts = [];
+            for (const s of sentences) {
+                s.units.forEach((u, i) => {
+                    const last = parts[parts.length - 1];
+                    if (last && last.lang === s.langs[i])
+                        last.end = u.end;
+                    else
+                        parts.push({ text: '', lang: s.langs[i], start: u.start, end: u.end });
+                });
+            }
+            for (const p of parts)
+                p.text = text.slice(p.start, p.end);
+            return { text, issues, parts };
         },
     };
 }

package/dist/detect.d.ts

@@ -0,0 +1,10 @@
+import type { Dictionary } from './index.ts';
+import { type Unit } from './text.ts';
+import type { Translator } from './translate.ts';
+export declare function markLanguages(list: Unit[], o: {
+    lang: string;
+    base: string;
+    target: Dictionary;
+    baseDictionary: Dictionary;
+    translator: Translator;
+}): Promise<string[]>;