which-dialect 0.3.0 → 0.5.0

package/README.md

@@ -111,44 +111,47 @@ The cases involving Spanish were removed with it. The held-out set has since bee
 
 ### `createChecker({ baseUrl?, load?, dictionary? })` → `checker.check(text, options)`
 
-Checks a learner's sentence in one language and returns issues with the exact place in the text, a short explanation and suggested fixes. It runs in the browser like the rest of the library: no API calls, no cost.
+A spellchecker for learners: it gives a correction for every word it can, with the exact place in the text and a short reason. Words written in the learner's own language (English by default) get the target-language word. Further checks (regions, pronouns, politeness) can be turned on. It runs in the browser like the rest of the library: no API calls, no cost.
 
 ```ts
 import { createChecker } from 'which-dialect'
 
 const checker = createChecker()
-const { text, issues } = await checker.check('Tôi muốn ăn thịt lợn và bắp khong?', {
-  lang: 'vi', region: 'Southern', listener: 'parent',
+const { text, issues, parts } = await checker.check('Toi muon an the ice cream but it was very expensive.', {
+  lang: 'vi', region: 'Southern',
 })
-// “Tôi”: Talking to your parents, say “con” for “I”, not “Tôi”.               → Con
-// “lợn”: “lợn” is Northern; in Southern Vietnamese, say “heo”.                → heo, cúi, ỉn
-// “khong”: “khong” isn't a Vietnamese syllable. Did you mean “không”, “khổng” or “khống”?
-// “khong”: Talking to your parents, end the sentence with “ạ” to sound polite. (suggestion)
+// Toi → Tôi · muon → muốn · an → ăn          (accents)
+// the → (left out: Vietnamese has no "the") · ice cream → kem · but → nhưng · it → nó
+// was → là · very → rất · expensive → mắc    (English words, in Southern Vietnamese)
 ```
 
 | Option | Meaning |
 |---|---|
-| `lang` | The text's language. |
-| `region` | The region the text should be in. Without it, words that don't fit the rest of the text's region are pointed out (*lợn*, Northern, next to *nha*, Southern). |
-| `listener` | Who you're talking to, as a [pronoun-table](#looking-words-up) row id (`'parent'`, `'teacher'`, `'friend'`, …): pronouns and polite endings are checked against it. |
-| `about` | Who you're talking about, for "he/she" and "they" pronouns. |
-| `speaker` | `'male'` or `'female'`, for pronouns that depend on it. |
-| `rules` | Turn checks off, e.g. `{ 'polite-ending': false }`. |
+| `lang` | The text's language (the target). |
+| `base` | The learner's own language, whose words get translated (default `'en'`). |
+| `region` | The region's words are used for translations (*expensive* → Southern *mắc*, Northern *đắt*), and the `dialect` check, when on, flags other regions' words. |
+| `register` | `'casual'` allows colloquial translations; otherwise plain ones come first. |
+| `rules` | Turn checks on or off. By default only `spelling` and `foreign-word` run; e.g. `{ dialect: true, 'pronoun-relationship': true }`. |
+| `listener`, `about`, `speaker` | For the pronoun and polite-ending checks: who you're talking to and about (pronoun-table row ids), and the speaker's gender. |
 
-Each issue has `rule`, `severity` (`'error'`: not a word of the language; `'warning'`: wrong for the chosen region or relationship; `'suggestion'`: often better), `start`/`end` (offsets into the returned `text`, the input in Unicode NFC form), `text`, `message` and `suggestions` (replacements for `text`, best first).
+The result has `text` (the input in Unicode NFC form), `issues` and `parts` (the text split by language). Each issue has `rule`, `severity` (`'error'`: not a word of the language; `'warning'`: probably wrong, or wrong for the region or relationship; `'suggestion'`: often better), `start`/`end` (offsets into `text`), `text`, `message` and `suggestions` (replacements for `text`, best first; `''` means leave it out).
 
-The checks:
+**On by default (the spellchecker):**
 
-- **`spelling`**: syllables that aren't in the language, with the accented forms they could be (*khong* → *không*). Plain-letter words with no accented form are left alone (names, foreign words). Languages written in syllables only (Vietnamese).
-- **`dialect`**: words whose main meaning is tagged for another region (*lợn* in Southern text → *heo*; *muỗng* in Northern → *thìa*; in English, US *apartment* in UK text → *flat*). Words with an untagged main meaning are never flagged (*má* is "cheek" everywhere).
+- **`spelling`**: a doubtful syllable is first tried with its neighbors: if fixing its accents makes a dictionary word with the syllables next to it, that word is suggested (*hom nay* → *hôm nay* "today", where *nay* alone would become *này* "this"; *an com* → *ăn cơm*; *Chung toi* → *Chúng tôi*), changing only the doubtful syllables. Otherwise, syllables that aren't in the language get the accented forms they could be (*khong* → *không*); rare plain-letter words next to a far more common accented form too (*toi* → *tôi*, *hom* → *hôm*), and, in a sentence typed without accents, any more common accented form (*an* → *ăn*, *di* → *đi*; not *con* → *còn*, about as common). Names (capitalized mid-sentence), the language's listed pronouns (*tui*) and plain-letter words with no accented form (*email*) are left alone. Languages written in syllables only (Vietnamese).
+- **`foreign-word`**: words in the learner's own language get the target-language word, in the region and register (*market* → *chợ*, *ice cream* → *kem*). Phrases the base dictionary knows are translated as one, falling back to single words. Words the language has no word for are suggested to be left out (`leaveOut` in its checker settings: Vietnamese has no *the*, *a*, *an*). Which language each word is in comes from the two dictionaries: an English word that's Vietnamese only with accents added (*but*: *bút*) is English; Vietnamese typed without accents (*khong*) is Vietnamese; words both languages have go by how common they are in each (*em*, *di*, *tui* are Vietnamese even between English words; *the* is English), and close calls (*an*, *to*) follow their neighbors, leaning to the target language.
+
+**Opt-in (`rules`):**
+
+- **`dialect`**: words whose main meaning is tagged for another region (*lợn* in Southern text → *heo*; *muỗng* in Northern → *thìa*; in English, US *apartment* in UK text → *flat*). Without `region`, words that don't fit the rest of the text's region are pointed out. Words with an untagged main meaning are never flagged (*má* is "cheek" everywhere).
 - **`pronoun-relationship`**: pronouns that don't fit the listener (*tôi* to a parent → *con*; to a teacher → *em*).
 - **`pronoun-pair`**: "I" and "you" pronouns that never go together (*tao* goes with *mày*, not *bạn*), when no listener is given.
 - **`pronoun-consistency`**: switching words for the same person within one text (*tôi* in one sentence, *tui* in the next): keep the first.
 - **`polite-ending`**: a suggestion to end sentences with *ạ* when talking to parents, elders and teachers (not for exclamations).
 
-The checker only speaks up when it's confident, so **no issues means none of the checks found anything, not that the text is correct**. It doesn't check word order, classifiers or whether a sentence sounds natural yet. Only reliable pronouns are checked: kinship words like *con* (also "child", and a classifier in *con chó*) aren't, and *bạn* (also "friend") only gets suggestions.
+Corrections are word by word: they don't fix word order or grammar, so English translated word by word reads like English in Vietnamese words (*it was very expensive* → *nó là rất mắc*). Whole English sentences or clauses that follow a [sentence frame](#sentence-frames) get the frame's natural wording instead in the [journal review](#reviewing-journal-entries). The checker only speaks up when it's confident, so **no issues means none of the checks found anything, not that the text is correct**.
 
-**Every language uses the same checks.** They read only the language's data and its `checker` settings in `languages/<code>.ts` (how text splits into words, which pronouns are reliable, polite endings), so a check whose data a language doesn't have is skipped. A language without checker settings still gets the `dialect` check.
+**Every language uses the same checks.** They read only the language's data and its `checker` settings in `languages/<code>.ts` (how text splits into words, which pronouns are reliable, polite endings, words with no equivalent), so a check whose data a language doesn't have is skipped. A language without checker settings is read as space-separated words: English words in it are still translated, and the `dialect` check works.
 
 ## Sentence frames
 
@@ -166,13 +169,13 @@ await pb.match('Do you want coffee?', { lang: 'vi' })                         //
 await pb.matchFrames('Bạn có đi?', { lang: 'vi' })                            // → 'Bạn có đi không?' (missing "không")
 ```
 
-Options: `lang`, `region`, `listener` (a pronoun-table row id; without it, the neutral default row), `speaker`, `register` (`'casual'` allows colloquial words like Southern *hông*, *tui*). Content slots take English (translated for you, in the region) or `{ text }` in the language itself. `parts` says why each word was chosen, for highlighting. `matchFrames` recognizes a frame in a sentence in the language itself and returns the changes the frame suggests: a missing required word (*không*), a word from another region (*mô* in Southern text → *đâu*), missing accents in a frame position (*Toi* → *Tôi*), and, only when a listener is given, a pronoun that doesn't fit them.
+Options: `lang`, `region`, `listener` (a pronoun-table row id; without it, the neutral default row), `speaker`, `register` (`'casual'` allows colloquial words like Southern *hông*, *tui*). Content slots take English (translated for you, in the region) or `{ text }` in the language itself. `parts` says why each word was chosen, for highlighting. `matchFrames` recognizes a frame in a sentence in the language itself (the journal review uses it to list the frames a sentence follows) and returns the changes the frame would suggest: a missing required word (*không*), a word from another region (*mô* in Southern text → *đâu*), missing accents in a frame position (*Toi* → *Tôi*), and, only when a listener is given, a pronoun that doesn't fit them.
 
 The frames themselves are written once, in English, in [`languages/frames.ts`](languages/frames.ts) (34 so far: journal sentences, feelings, questions, requests, greetings). Each language says how it expresses them (`frames` in `languages/<code>.ts`), with pronoun slots (`{I}`, `{YOU}`, `{WE_INCL}`, …, filled from the pronoun table) and **frame words** for what changes by region (`frameWords`: Vietnamese `{WHERE}` *đâu* / Central *mô*, `{HOW}` Northern *thế nào* / *sao* / Central *răng*, `{SOFT}` Northern *nhé* / Central and Southern *nha*, `{Q_END}` *không* / Southern casual *hông*, `{POLITE}` *ạ* when speaking up). Each frame word comes from a dictionary definition, keeping its region and labels, or from an override with a note, like the pronoun table. A language without frames returns none.
 
 ## Reviewing journal entries
 
-`createReviewer().review(entry, { lang, base?, region?, listener?, speaker?, register? })` reviews an entry written in the target language, the learner's own language (`base`, default `'en'`), or a mix of both, and returns the corrected entry sentence by sentence:
+`createReviewer().review(entry, { lang, base?, region?, register?, checks?, listener?, speaker? })` reviews an entry written in the target language, the learner's own language (`base`, default `'en'`), or a mix of both, and returns the corrected entry, in the target language, sentence by sentence:
 
 ```ts
 import { createReviewer } from 'which-dialect'
@@ -182,20 +185,23 @@ const review = await createReviewer().review(
   { lang: 'vi', region: 'Southern' },
 )
 review.corrected
-// 'Hôm nay tôi đi chợ với má. Tôi muốn ăn thịt heo nhưng mắc quá. Ngày mai tôi đi lại không?'
+// 'Hôm nay tôi đi chợ với má. Tôi muốn ăn thịt lợn nhưng mắc quá. Ngày mai tui đi lại không?'
 review.sentences[1].changes
-// [{ from: 'but it was expensive', to: 'nhưng mắc quá', kind: 'frame', why: 'Sentence frame “but it was {quality}”' },
-//  { from: 'lợn', to: 'heo', kind: 'dialect', why: '“lợn” is Northern; in Southern Vietnamese, say “heo”.' }]
+// [{ from: 'but it was expensive', to: 'nhưng mắc quá', kind: 'frame', why: 'Sentence frame “but it was {quality}”' }]
+
+// With the regional and consistency checks on:
+await createReviewer().review(entry, { lang: 'vi', region: 'Southern', checks: { dialect: true, 'pronoun-consistency': true } })
+// 'Hôm nay tôi đi chợ với má. Tôi muốn ăn thịt heo nhưng mắc quá. Ngày mai tôi đi lại không?'
 ```
 
 For each sentence:
 
-1. **Each word is marked as the target or the base language** (`parts`), from the two dictionaries alone. Words that could be either (*an*, *to*) take their neighbors' language; an English word that's Vietnamese only with accents added (*but*: *bút*) stays English, while Vietnamese typed without accents (*khong*) stays Vietnamese.
-2. **Base-language parts are replaced:** a whole sentence or clause that follows a sentence frame is filled in (*but it was expensive* → *nhưng mắc quá*; *Where is the bathroom?* → *Phòng tắm ở đâu?*), and words or short phrases (up to 3 words) are translated in the region (*market* → *chợ*). Anything longer is left as written and listed in `unchecked`: the review doesn't guess.
-3. **The grammar checker runs on the whole entry**, so consistency checks see every sentence (*tôi* earlier, *tui* later). Its errors and warnings are applied (`changes`); its suggestions, like ending with *ạ*, are returned as `hints`.
-4. **The corrected sentence is checked against sentence frames** (*Bạn có đi chợ?* → *Bạn có đi chợ không?*), and the frames it follows are listed (`frames`), for suggesting patterns to reuse.
+1. **Each word is marked as the target or the base language** (`parts`), as in the checker's `foreign-word` check.
+2. **A whole base-language sentence or clause that follows a sentence frame is filled in** with the frame's wording (*but it was expensive* → *nhưng mắc quá*; *Where is the bathroom?* → *Phòng tắm ở đâu?*).
+3. **The checker runs on the whole entry**: by default the spellchecker, so every remaining English word gets its translation and every misspelled word its accents; `checks` turns on more (regions, pronouns, consistency across sentences, polite endings). Its errors and warnings are applied (`changes`); its suggestions, like ending with *ạ*, are returned as `hints`.
+4. **The frames the corrected sentence follows are listed** (`frames`), for suggesting patterns to reuse. Frames don't change the sentence.
 
-Each sentence has `original`, `parts`, `corrected`, `changes` (`{ from, to, why, kind }`), `hints`, `frames` (`{ id, en, text }`, the frame's English and its pattern) and `unchecked`. For a journal, leave `listener` out: people written about (*má*, *thầy*) aren't treated as the listener. Rules can't fix everything: scrambled word order, missing verbs, or English sentences no frame covers come back as written, and plain-letter words that are also Vietnamese words (*toi*, *di*) are only fixed when a sentence frame shows what they should be.
+Each sentence has `original`, `parts`, `corrected`, `changes` (`{ from, to, why, kind }`; `to` is `''` when a word is left out), `hints`, `frames` (`{ id, en, text }`, the frame's English and its pattern) and `unchecked` (English words with no translation, left as written). For a journal, leave `listener` out: people written about (*má*, *thầy*) aren't treated as the listener. Corrections are word by word except for frames: word order and grammar aren't fixed (*I am very happy* → *Tôi là rất mừng*).
 
 ## Looking words up
 
@@ -206,6 +212,8 @@ One language's dictionary. `baseUrl` defaults to `https://cdn.jsdelivr.net/npm/w
 - `dictionary.lookup(word)`: all entries for a word, with every sense's definitions, regions, labels, synonyms and variant links (`altOf`).
 - `dictionary.searchEnglish(term, { region?, pos?, exclude?, limit?, allSenses? })`: words for an English term in this language, one per word, best first (no meaning handling: use the translator for that). `allSenses: true` returns every matching sense of those words instead.
 - `dictionary.frames()`: the language's sentence frames and frame words, unfilled (`{ frames, words }`; empty without frames).
+- `dictionary.segment(text)`: the text split into the language's words, longest dictionary match first (up to the checker's `maxWordUnits`, 3 for Vietnamese): `Hôm nay tôi ăn cơm.` → *Hôm nay* · *tôi* · *ăn cơm*, each `{ text, start, end, entries }` (offsets into the NFC text; `entries` empty for words the dictionary doesn't have). Use it to treat a word of several syllables as one, e.g. to show its meaning.
+- `dictionary.variants(word, { limit? })`: other words with the same letters but other accents that the dictionary has, most common first, each `{ word, entries, frequency? }`: *muộn* → *muốn*, *mượn*, *muôn*…; for words of several syllables every syllable varies (*hom nay* → *hôm nay*). For apps that let learners pick the word they meant: a real word with the wrong tone passes the spellchecker. Languages written in syllables only (`[]` otherwise).
 - `dictionary.checker()` / `dictionary.syllables()`: the language's checker settings and syllable list (with frequencies), used by `createChecker` (`{}` for languages without them).
 - `dictionary.picks()`: the language's [hand-picked words](#hand-picked-words), `{ word, pos, gloss, picks: [{ word, tags? }], first? }` per English meaning (empty for languages without any).
 - `dictionary.meta()`: the language's name, regions, region groups, source, license, build date and counts.

package/dist/check.d.ts

@@ -1,5 +1,10 @@
 import { type TranslatorOptions } from './translate.ts';
-export type CheckRule = 'spelling' | 'dialect' | 'pronoun-relationship' | 'pronoun-pair' | 'pronoun-consistency' | 'polite-ending';
+export type CheckRule = 'spelling' | 'foreign-word' | 'dialect' | 'pronoun-relationship' | 'pronoun-pair' | 'pronoun-consistency' | 'polite-ending';
+/**
+ * Which checks run unless `rules` says otherwise: the spellchecker (spelling, and words in the learner's
+ * own language, with their translation). The rest are opt-in, e.g. `rules: { dialect: true }`.
+ */
+export declare const DEFAULT_CHECK_RULES: Record<CheckRule, boolean>;
 export type CheckIssue = {
     rule: CheckRule;
     /**
@@ -20,15 +25,28 @@ export type CheckIssue = {
 export type CheckOptions = {
     /** Language code of the text, e.g. "vi". */
     lang: string;
-    /** Region or region group the text should be in, e.g. "Southern". Without it, mixed regions are flagged. */
+    /**
+     * The learner's own language, whose words in the text are translated (`foreign-word`). Default "en";
+     * none when it's the text's language.
+     */
+    base?: string;
+    /**
+     * Region or region group the text should be in, e.g. "Southern": translations use its words, and the
+     * `dialect` check flags other regions' words (without it, mixed regions).
+     */
     region?: string;
+    /** 'casual' allows colloquial translations (Southern "tui"); otherwise plain ones come first. */
+    register?: 'casual' | 'neutral' | 'polite';
     /** Who you're talking to, as a pronoun-table row id (see Dictionary.pronouns), e.g. "parent". */
     listener?: string;
     /** Who you're talking about, as a pronoun-table row id, for "he/she" and "they" pronouns. */
     about?: string;
     /** The speaker's gender, for pronouns that depend on it. */
     speaker?: 'male' | 'female';
-    /** Turn rules off, e.g. { 'polite-ending': false }. All are on by default. */
+    /**
+     * Turn checks on or off. By default only the spellchecker runs (DEFAULT_CHECK_RULES: `spelling` and
+     * `foreign-word`); opt in to others, e.g. { dialect: true, 'pronoun-relationship': true }.
+     */
     rules?: Partial<Record<CheckRule, boolean>>;
 };
 export type CheckResult = {
@@ -36,6 +54,13 @@ export type CheckResult = {
     text: string;
     /** Sorted by position. */
     issues: CheckIssue[];
+    /** The text split by language (the target and the base), in order; one part when no base is given. */
+    parts: {
+        text: string;
+        lang: string;
+        start: number;
+        end: number;
+    }[];
 };
 export type Checker = {
     check(text: string, options: CheckOptions): Promise<CheckResult>;

package/dist/check.js

@@ -1,17 +1,33 @@
-// Checking text in one language: spelling and accents, words from another region, and pronouns and
-// polite endings that fit who you're talking to.
+// Checking text in one language. By default it's a spellchecker that gives a correction for every word it
+// can: syllables that aren't in the language get their accented forms ("khong" → "không"), and words in the
+// learner's own language (English by default) get the target-language word ("market" → "chợ"). Further
+// checks are opt-in (`rules`): words from another region, pronouns and polite endings that fit who you're
+// talking to, and consistency within the text.
 //
 // Every rule reads only the language's data: its dictionary, pronoun table, checker.json and
 // syllables.json (see CheckerConfig). Adding a language means adding data and settings in
-// languages/<lang>.ts, not code here. A rule whose data a language doesn't have is skipped, so a language
-// without checker settings is still checked for words from another region.
+// languages/<lang>.ts, not code here. A rule whose data a language doesn't have is skipped.
 //
 // The checker only speaks up when it's confident and stays silent otherwise, so no issues means none of
 // the checks found anything, not that the text is correct.
 import { createDictionary, resolveRegion } from "./index.js";
-import { joined, matchCase, plain, segment, sentencesOf, units } from "./text.js";
+import { accentCombinations, joined, matchCase, plain, segment, sentencesOf, syllablesByPlain, units } from "./text.js";
+import { markLanguages } from "./detect.js";
 import { createTranslator } from "./translate.js";
 import { PRONOUN_PERSONS, } from "./types.js";
+/**
+ * Which checks run unless `rules` says otherwise: the spellchecker (spelling, and words in the learner's
+ * own language, with their translation). The rest are opt-in, e.g. `rules: { dialect: true }`.
+ */
+export const DEFAULT_CHECK_RULES = {
+    spelling: true,
+    'foreign-word': true,
+    dialect: false,
+    'pronoun-relationship': false,
+    'pronoun-pair': false,
+    'pronoun-consistency': false,
+    'polite-ending': false,
+};
 // How each pronoun-table column is named in messages.
 const PERSON_NAMES = {
     self: 'I', addressee: 'you', third: 'he/she', selfPlural: 'we', addresseePlural: 'you (plural)', thirdPlural: 'they',
@@ -25,42 +41,180 @@ const lowerFirst = (s) => s.charAt(0).toLowerCase() + s.slice(1);
 const listOf = (items, conjunction = 'and') => items.length <= 1 ? items.join('') : `${items.slice(0, -1).join(', ')} ${conjunction} ${items[items.length - 1]}`;
 const unique = (items) => [...new Set(items)];
 // --- Rules -----------------------------------------------------------------------------------------------
-// Spelling and accents, for languages written in syllables with a syllable list: a syllable that isn't in
-// the language is flagged, with the accented forms it could be ("khong" → "không"). Plain-letter
-// syllables with no accented form are left alone: usually a name or a foreign word ("email").
+// Spelling and accents, for languages written in syllables with a syllable list (with frequencies):
+// - a syllable that isn't in the language is flagged, with the accented forms it could be ("khong" →
+//   "không"); plain-letter syllables with no accented form are left alone (names, foreign words: "email");
+// - a plain-letter syllable that is a word but a rare one, next to a much more common accented form, is
+//   probably missing its accents ("toi" → "tôi", "hom" → "hôm"). In a sentence typed without accents (at
+//   least half plain letters and a missing-accents error already, or 3+ words with no accent at all; half,
+//   because filled-in sentence frames add accented words), any accented form that's more common
+//   counts ("an" → "ăn"; but not "con" → "còn", about as common). Left alone: capitalized words
+//   mid-sentence (names) and the language's listed pronouns, some written without accents ("tui").
+// First, a doubtful syllable (one of the above) is tried with its neighbors: when fixing its accents makes a
+// dictionary word with them, that word is suggested for the run ("hom nay" → "hôm nay", not "hôm này").
+// Units inside a known word, or in the learner's own language, aren't checked.
+// Zipf points by which an accented form must be more common than a plain-letter word to be suggested.
+const ACCENTS_GAP = 1.5;
+const ACCENTS_GAP_UNACCENTED_TEXT = 0.2;
+// Accent forms tried per doubtful syllable when looking for a word with its neighbors, and how many
+// combinations are looked up at most.
+const COMPOUND_FORMS_PER_SYLLABLE = 5;
+const COMPOUND_MAX_COMBINATIONS = 64;
 async function spelling(ctx) {
     if (ctx.config.units !== 'syllables' || !ctx.meta.syllables)
         return [];
     const syllables = await ctx.dict.syllables();
-    const byPlain = new Map();
-    for (const s of Object.keys(syllables))
-        byPlain.set(plain(s), [...(byPlain.get(plain(s)) ?? []), s]);
-    for (const list of byPlain.values())
-        list.sort((a, b) => syllables[b] - syllables[a]);
+    const byPlain = syllablesByPlain(syllables);
     const issues = [];
-    for (const word of ctx.sentences.flatMap((s) => s.words)) {
-        if (word.entries.length)
-            continue;
-        for (const u of word.units) {
-            const lower = u.text.toLowerCase().replace(/[’']/g, '');
-            if (/\d/.test(lower) || lower in syllables)
-                continue;
-            const candidates = (byPlain.get(plain(lower)) ?? []).slice(0, 3);
-            const accented = lower !== plain(lower);
-            if (!candidates.length && !accented)
+    const suggest = (u, candidates, severity, message) => issues.push({ rule: 'spelling', severity, start: u.start, end: u.end, text: u.text, message, suggestions: candidates.map((c) => matchCase(u.text, c)) });
+    const options = (u, candidates) => listOf(candidates.map((c) => quote(matchCase(u.text, c))), 'or');
+    const pronouns = new Set(Object.values(ctx.config.pronouns ?? {}).flat().map((w) => w.toLowerCase()));
+    for (const sentence of ctx.sentences) {
+        const lone = sentence.words.filter((w) => !w.entries.length || w.units.length === 1).flatMap((w) => w.units).filter((u) => !ctx.inBase(u));
+        const target = sentence.units.filter((u) => !ctx.inBase(u) && /\p{L}/u.test(u.text));
+        const isPlain = (u) => u.text.toLowerCase() === plain(u.text.toLowerCase());
+        const mostlyPlain = target.filter(isPlain).length >= target.length * 0.5;
+        const lowerOf = (u) => u.text.toLowerCase().replace(/[’']/g, '');
+        const inWord = (u) => sentence.words.some((w) => w.entries.length && w.units.length > 1 && w.units.includes(u));
+        // What each lone unit is: not a syllable of the language (with forms it could be), or a plain-letter
+        // syllable that could be missing its accents.
+        const checked = lone.filter((u) => !/\d/.test(lowerOf(u)) && !inWord(u));
+        const unknown = new Set(checked.filter((u) => !(lowerOf(u) in syllables) && ((byPlain.get(plain(lowerOf(u))) ?? []).length || !isPlain(u))));
+        const plainWords = new Set(checked.filter((u) => {
+            const lower = lowerOf(u);
+            return lower in syllables && lower === plain(lower) && !pronouns.has(lower) && (u === sentence.units[0] || !/\p{Lu}/u.test(u.text.charAt(0)));
+        }));
+        const missingAccents = [...unknown].some(isPlain);
+        // Typed without accents: mostly plain letters, and an accent error already, or 3+ words with no accent at all.
+        const unaccented = mostlyPlain && (missingAccents || (target.length >= 3 && target.every(isPlain)));
+        const gap = unaccented ? ACCENTS_GAP_UNACCENTED_TEXT : ACCENTS_GAP;
+        const betterForms = (u) => {
+            const lower = lowerOf(u);
+            return (byPlain.get(lower) ?? []).filter((c) => c !== lower && syllables[c] >= syllables[lower] + gap - 1e-9).slice(0, 3);
+        };
+        const suspect = (u) => unknown.has(u) || (plainWords.has(u) && betterForms(u).length > 0);
+        // First, runs of syllables with a doubtful one that make a word once their accents are fixed: "hom nay"
+        // → "hôm nay", "an com" → "ăn cơm", "hôm nây" → "hôm nay". Only the doubtful syllables change. The word
+        // decides between forms a syllable alone can't ("nay" alone could be này, nảy…).
+        const covered = new Set();
+        const loneSet = new Set(checked);
+        const maxUnits = ctx.config.maxWordUnits ?? 3;
+        for (let i = 0; i < sentence.units.length; i++) {
+            for (let k = Math.min(maxUnits, sentence.units.length - i); k >= 2; k--) {
+                const span = sentence.units.slice(i, i + k);
+                if (span.some((u, j) => !loneSet.has(u) || covered.has(u) || (j > 0 && !joined(ctx.text, span[j - 1], u))))
+                    continue;
+                if (!span.some(suspect))
+                    continue;
+                const written = span.map(lowerOf);
+                const combos = accentCombinations(written, byPlain, {
+                    fixed: span.map((u) => !suspect(u)),
+                    perUnit: COMPOUND_FORMS_PER_SYLLABLE,
+                    max: COMPOUND_MAX_COMBINATIONS,
+                }).filter((c) => c !== written.join(' '));
+                const found = (await Promise.all(combos.map(async (c) => ({ word: c, entries: await ctx.dict.lookup(c) }))))
+                    .filter((f) => f.entries.length)
+                    .map((f) => ({ word: f.word, frequency: Math.max(-1, ...f.entries.map((e) => e.frequency ?? -1)) }))
+                    .sort((a, b) => b.frequency - a.frequency)
+                    .slice(0, 3);
+                if (!found.length)
+                    continue;
+                const at = { start: span[0].start, end: span[k - 1].end, text: ctx.text.slice(span[0].start, span[k - 1].end) };
+                const suggestions = found.map((f) => matchCase(at.text, f.word));
+                const quoted = listOf(suggestions.map(quote), 'or');
+                issues.push({
+                    rule: 'spelling', severity: 'warning', ...at, suggestions,
+                    message: span.filter(suspect).every(isPlain)
+                        ? `${quote(at.text)} looks typed without accents. Did you mean ${quoted}?`
+                        : `${quote(at.text)} isn't spelled right. Did you mean ${quoted}?`,
+                });
+                span.forEach((u) => covered.add(u));
+                break;
+            }
+        }
+        // Then each remaining syllable on its own.
+        for (const u of checked) {
+            if (covered.has(u))
                 continue;
-            issues.push({
-                rule: 'spelling',
+            const lower = lowerOf(u);
+            if (unknown.has(u)) {
+                const candidates = (byPlain.get(plain(lower)) ?? []).slice(0, 3);
+                const accented = !isPlain(u);
                 // Missing accents on plain letters is the common learner slip; a wrong accent is more surely wrong.
-                severity: accented ? 'error' : 'warning',
-                start: u.start,
-                end: u.end,
-                text: u.text,
-                message: candidates.length
-                    ? `${quote(u.text)} isn't a ${ctx.meta.name} syllable. Did you mean ${listOf(candidates.map((c) => quote(matchCase(u.text, c))), 'or')}?`
-                    : `${quote(u.text)} isn't a ${ctx.meta.name} syllable. Check the spelling and accents.`,
-                suggestions: candidates.map((c) => matchCase(u.text, c)),
-            });
+                suggest(u, candidates, accented ? 'error' : 'warning', candidates.length
+                    ? `${quote(u.text)} isn't a ${ctx.meta.name} syllable. Did you mean ${options(u, candidates)}?`
+                    : `${quote(u.text)} isn't a ${ctx.meta.name} syllable. Check the spelling and accents.`);
+            }
+            else if (plainWords.has(u)) {
+                const better = betterForms(u);
+                if (!better.length)
+                    continue;
+                suggest(u, better, 'warning', unaccented
+                    ? `${quote(u.text)} looks typed without accents. Did you mean ${options(u, better)}?`
+                    : `${quote(u.text)} is a rare word; did you mean ${options(u, better)}?`);
+            }
+        }
+    }
+    return issues;
+}
+// Words in the learner's own language (English by default), with the target-language word to use: "market"
+// → "chợ". Runs of base units are split into the longest phrases the base dictionary knows ("ice cream"),
+// each translated in the region and register. Words the language usually has no word for
+// (CheckerConfig.leaveOut: Vietnamese has no articles) are suggested to be left out.
+async function foreignWord(ctx) {
+    const base = ctx.base;
+    if (!base)
+        return [];
+    const leaveOut = new Set((ctx.config.leaveOut ?? []).map((w) => w.toLowerCase()));
+    const issues = [];
+    for (const s of ctx.sentences) {
+        const runs = [];
+        s.units.forEach((u, i) => {
+            if (!ctx.inBase(u))
+                return;
+            const last = runs[runs.length - 1];
+            if (last && last[last.length - 1] === s.units[i - 1] && joined(ctx.text, s.units[i - 1], u))
+                last.push(u);
+            else
+                runs.push([u]);
+        });
+        for (const run of runs) {
+            for (let i = 0; i < run.length;) {
+                // The longest phrase the base dictionary knows and the language has a word for; single words
+                // always give an issue, with or without a translation.
+                let found;
+                for (let k = Math.min(MAX_FOREIGN_UNITS, run.length - i); k >= 1 && !found; k--) {
+                    const phrase = ctx.text.slice(run[i].start, run[i + k - 1].end);
+                    if (k === 1 && leaveOut.has(phrase.toLowerCase())) {
+                        found = { k, words: [], leave: true };
+                        break;
+                    }
+                    if (k > 1 && !(await ctx.translator.senses(phrase, { from: base }).catch(() => [])).length)
+                        continue;
+                    const [group] = await ctx.translator.translate(phrase, {
+                        from: base, to: ctx.options.lang, toRegion: ctx.options.region, register: ctx.options.register, listener: ctx.options.listener, limit: 3,
+                    }).catch(() => []);
+                    const words = unique((group?.translations ?? []).map((t) => t.word));
+                    if (words.length || k === 1)
+                        found = { k, words, leave: false };
+                }
+                const { k, words, leave } = found;
+                const span = run.slice(i, i + k);
+                const text = ctx.text.slice(span[0].start, span[k - 1].end);
+                i += k;
+                const at = { rule: 'foreign-word', severity: 'error', start: span[0].start, end: span[k - 1].end, text };
+                if (leave) {
+                    issues.push({ ...at, message: `${quote(text)} isn't ${ctx.meta.name}; ${ctx.meta.name} usually has no word for it, so leave it out.`, suggestions: [''] });
+                    continue;
+                }
+                issues.push({
+                    ...at,
+                    message: words.length
+                        ? `${quote(text)} isn't ${ctx.meta.name}: say ${listOf(words.map(quote), 'or')}.`
+                        : `${quote(text)} isn't ${ctx.meta.name}, and no translation was found.`,
+                    suggestions: words.map((w) => (span[0].start === s.units[0].start ? matchCase(text, w) : w)),
+                });
+            }
         }
     }
     return issues;
@@ -73,6 +227,7 @@ async function dialect(ctx) {
     const all = new Set(ctx.meta.regions);
     const regional = ctx.sentences
         .flatMap((s) => s.words)
+        .filter((word) => !word.units.some((u) => ctx.inBase(u)))
         .map((word) => ({ word, sense: word.entries[0]?.senses[0] }))
         .filter((x) => Boolean(x.sense?.regionTagged && x.sense.regions.length < all.size));
     if (!regional.length)
@@ -239,8 +394,11 @@ async function politeEnding(ctx) {
     }
     return issues;
 }
+// The longest base-language phrase looked up as one ("ice cream", "take care of").
+const MAX_FOREIGN_UNITS = 3;
 const RULES = {
     spelling,
+    'foreign-word': foreignWord,
     dialect,
     'pronoun-relationship': pronounRelationship,
     'pronoun-pair': pronounPair,
@@ -281,10 +439,13 @@ export function createChecker(options = {}) {
                 }
                 return pending;
             };
+            const base = (opts.base ?? 'en') === opts.lang ? undefined : (opts.base ?? 'en');
             const sentences = [];
             for (const list of sentencesOf(text, units(text))) {
-                sentences.push({ units: list, words: await segment(text, list, config.maxWordUnits ?? 3, lookup) });
+                const langs = base ? await markLanguages(list, { lang: opts.lang, base, target: d, baseDictionary: dict(base), translator }) : list.map(() => opts.lang);
+                sentences.push({ units: list, langs, words: await segment(text, list, config.maxWordUnits ?? 3, lookup) });
             }
+            const baseUnits = new Set(sentences.flatMap((s) => s.units.filter((_, i) => s.langs[i] !== opts.lang).map((u) => u.start)));
             let rows;
             let matches;
             const ctx = {
@@ -298,6 +459,9 @@ export function createChecker(options = {}) {
                 // The language's reliable pronouns in the text, longest first ("chúng tôi" before "tôi"), matched on
                 // units directly so pronouns the dictionary doesn't list as one word ("tụi mày") are found too.
                 pronounMatches: () => (matches ??= findPronouns(text, sentences, config)),
+                base,
+                inBase: (u) => baseUnits.has(u.start),
+                translator,
                 suggestRegional: async (word, region) => {
                     try {
                         const [group] = await translator.translate(word.lower, {
@@ -310,9 +474,22 @@ export function createChecker(options = {}) {
                     }
                 },
             };
-            const results = await Promise.all(Object.keys(RULES).filter((r) => opts.rules?.[r] !== false).map((r) => RULES[r](ctx)));
+            const results = await Promise.all(Object.keys(RULES).filter((r) => opts.rules?.[r] ?? DEFAULT_CHECK_RULES[r]).map((r) => RULES[r](ctx)));
             const issues = results.flat().sort((a, b) => a.start - b.start || SEVERITY_ORDER[a.severity] - SEVERITY_ORDER[b.severity]);
-            return { text, issues };
+            // The text split by language: consecutive units in one language, with the text between them.
+            const parts = [];
+            for (const s of sentences) {
+                s.units.forEach((u, i) => {
+                    const last = parts[parts.length - 1];
+                    if (last && last.lang === s.langs[i])
+                        last.end = u.end;
+                    else
+                        parts.push({ text: '', lang: s.langs[i], start: u.start, end: u.end });
+                });
+            }
+            for (const p of parts)
+                p.text = text.slice(p.start, p.end);
+            return { text, issues, parts };
         },
     };
 }

package/dist/detect.d.ts

@@ -0,0 +1,10 @@
+import type { Dictionary } from './index.ts';
+import { type Unit } from './text.ts';
+import type { Translator } from './translate.ts';
+export declare function markLanguages(list: Unit[], o: {
+    lang: string;
+    base: string;
+    target: Dictionary;
+    baseDictionary: Dictionary;
+    translator: Translator;
+}): Promise<string[]>;

package/dist/detect.js

@@ -0,0 +1,65 @@
+// Which language each word of a mixed text is in: the target language or the learner's own (the base,
+// English by default), from the two languages' data alone. Shared by the checker and the journal review.
+import { plain } from "./text.js";
+/**
+ * Marks each unit (of one sentence) with `lang` or `base`. A unit the target knows and the base doesn't
+ * is the target's, and the other way round. A unit both know ("em", "an", "to") goes by how common it is
+ * in each: clearly more common in the target, the target's (Vietnamese "em", "di" next to English
+ * "hello", "shopping": English has rare "em", "di"); far more common in the base, the base's ("the");
+ * otherwise, and for units neither knows (names), the nearest decided neighbors' language when they agree,
+ * else the target's (the text is mainly the target language: "an" next to English "the"). In languages written in syllables, a base
+ * word that's a target syllable only once accents are added ("but": bút) is the base's; a word only the
+ * target knows without its accents ("khong") is still the target's. The base is known through the translator, so
+ * inflected forms count ("went", "cooked").
+ */
+// Zipf points by which a word both languages know must be more common in one to count as that language's
+// whatever its neighbors: less for the target, since the text is mainly the target language.
+const TARGET_MARGIN = 1;
+const BASE_MARGIN = 2;
+export async function markLanguages(list, o) {
+    const syllables = await o.target.syllables();
+    const syllableLanguage = Object.keys(syllables).length > 0;
+    const plainSyllables = syllableLanguage ? new Set(Object.keys(syllables).map(plain)) : new Set();
+    // 'exact': a target word as written; 'accents': one only with accents added ("khong" → "không").
+    const targetKnows = async (u) => {
+        const lower = u.text.toLowerCase();
+        if (!syllableLanguage)
+            return (await o.target.lookup(lower)).length > 0 ? 'exact' : false;
+        if (lower in syllables || lower !== plain(lower))
+            return 'exact';
+        return plainSyllables.has(lower) ? 'accents' : false;
+    };
+    const baseKnows = async (u) => (await o.translator.senses(u.text, { from: o.base }).catch(() => [])).length > 0;
+    const marks = await Promise.all(list.map(async (u) => {
+        if (/\d/.test(u.text))
+            return '?';
+        const [t, b] = await Promise.all([targetKnows(u), baseKnows(u)]);
+        if (t === 'exact')
+            return b ? 'both' : o.lang;
+        if (b)
+            return o.base;
+        return t === 'accents' ? o.lang : '?';
+    }));
+    // How common a word is in each language (Zipf; 0 when unknown).
+    const targetFrequency = async (u) => {
+        const lower = u.text.toLowerCase();
+        if (syllableLanguage)
+            return syllables[lower] ?? 0;
+        return (await o.target.lookup(lower))[0]?.frequency ?? 0;
+    };
+    const baseFrequency = async (u) => (await o.baseDictionary.lookup(u.text.toLowerCase()))[0]?.frequency ?? 0;
+    // Words both languages know, decided by frequency when it's clear.
+    const decided = await Promise.all(marks.map(async (m, i) => {
+        if (m !== 'both')
+            return m;
+        const [t, b] = await Promise.all([targetFrequency(list[i]), baseFrequency(list[i])]);
+        return t - b >= TARGET_MARGIN ? o.lang : b - t >= BASE_MARGIN ? o.base : '?';
+    }));
+    return decided.map((m, i) => {
+        if (m !== '?')
+            return m;
+        const left = decided.slice(0, i).reverse().find((x) => x !== '?');
+        const right = decided.slice(i + 1).find((x) => x !== '?');
+        return left && right && left !== right ? o.lang : left ?? right ?? o.lang;
+    });
+}

package/dist/index.d.ts

@@ -49,6 +49,22 @@ export type PronounOptions = {
     /** Labels to leave out. Defaults to DEFAULT_EXCLUDED_LABELS; pass [] to include everything. */
     exclude?: string[];
 };
+/** One word of a text, as the dictionary splits it (see Dictionary.segment). */
+export type WordSegment = {
+    /** The word as written; offsets are into the text in Unicode NFC form. */
+    text: string;
+    start: number;
+    end: number;
+    /** The word's dictionary entries; empty for words the dictionary doesn't have (names, typos). */
+    entries: Entry[];
+};
+/** Another way to write a word with the same letters (see Dictionary.variants). */
+export type WordVariant = {
+    word: string;
+    entries: Entry[];
+    /** Zipf frequency (see Entry.frequency), or the syllable's for one syllable; undefined when unknown. */
+    frequency?: number;
+};
 export type Dictionary = {
     /** The language's name, regions, source, license and counts. */
     meta(): Promise<LanguageMeta>;
@@ -73,6 +89,22 @@ export type Dictionary = {
     syllables(): Promise<Record<string, number>>;
     /** The language's sentence frames and frame words, unfilled (see FramesData); empty without frames. */
     frames(): Promise<FramesData>;
+    /**
+     * The text split into the language's words: at each point, the longest run of units (syllables or
+     * words, up to the checker's `maxWordUnits`, default 3) that's a headword, so Vietnamese "hôm nay" is one
+     * word ("today"), not "hôm" + "nay". Units join only across spaces. Units the dictionary doesn't know
+     * are words of their own with no entries. Punctuation isn't returned.
+     */
+    segment(text: string): Promise<WordSegment[]>;
+    /**
+     * Other words written with the same letters but other accents, that the dictionary has, most common
+     * first: "muộn" → muốn, mượn, …; for words of several syllables, every syllable varies ("hom nay" →
+     * "hôm nay"). For languages written in syllables (with a syllable list); `[]` otherwise. The word
+     * itself isn't included.
+     */
+    variants(word: string, options?: {
+        limit?: number;
+    }): Promise<WordVariant[]>;
 };
 /**
  * The regions a region or region-group name stands for (e.g. English "North America" -> US, Canada),
@@ -86,3 +118,4 @@ export * from './check.ts';
 export * from './text.ts';
 export * from './phrasebook.ts';
 export * from './review.ts';
+export * from './detect.ts';

package/dist/index.js

@@ -1,4 +1,5 @@
 import { PRONOUN_PERSONS, fromStored, normalizeEnglish, shardKey, } from "./types.js";
+import { accentCombinations, segment as segmentUnits, syllablesByPlain, units } from "./text.js";
 export * from "./types.js";
 export * from "./pos.js";
 /**
@@ -14,6 +15,11 @@ export const DEFAULT_EXCLUDED_LABELS = [
 // just a particular register); the second set costs more (specialized, playful or uncommon).
 const MILD_LABELS = new Set(['colloquial', 'informal', 'familiar', 'polite', 'formal', 'endearing', 'honorific']);
 const LABEL_PENALTY = (h) => h.labels.reduce((sum, l) => sum + (MILD_LABELS.has(l) ? 1 : 2), 0);
+// Accent forms tried per syllable: more for one syllable, fewer for longer words, where every
+// combination is looked up (at most MAX_VARIANT_COMBINATIONS).
+const VARIANT_FORMS_ONE_SYLLABLE = 12;
+const VARIANT_FORMS_PER_SYLLABLE = 5;
+const MAX_VARIANT_COMBINATIONS = 64;
 /**
  * The regions a region or region-group name stands for (e.g. English "North America" -> US, Canada),
  * or null when no region is given. Throws for names the language doesn't have, listing the valid ones.
@@ -63,13 +69,52 @@ export function createDictionary(options) {
         const key = shardKey(term, m.shardLength);
         return m.shards[kind].includes(key) ? loadOnce(`${kind}/${key}.json`) : null;
     }
+    async function lookup(word) {
+        const w = word.trim();
+        const [m, data] = await Promise.all([meta(), shard('words', w)]);
+        const stored = data?.[w] ?? data?.[w.toLowerCase()] ?? [];
+        return stored.map((e) => ({ ...e, senses: e.senses.map((s) => fromStored(s, m.regions)) }));
+    }
+    const checker = async () => {
+        const m = await meta();
+        return m.checker ? loadOnce('checker.json') : {};
+    };
+    const syllables = async () => {
+        const m = await meta();
+        return m.syllables ? loadOnce('syllables.json') : {};
+    };
     return {
         meta,
-        async lookup(word) {
-            const w = word.trim();
-            const [m, data] = await Promise.all([meta(), shard('words', w)]);
-            const stored = data?.[w] ?? data?.[w.toLowerCase()] ?? [];
-            return stored.map((e) => ({ ...e, senses: e.senses.map((s) => fromStored(s, m.regions)) }));
+        lookup,
+        checker,
+        syllables,
+        async segment(input) {
+            const text = input.normalize('NFC');
+            const config = await checker();
+            const words = await segmentUnits(text, units(text), config.maxWordUnits ?? 3, lookup);
+            return words.map(({ text: t, start, end, entries }) => ({ text: t, start, end, entries }));
+        },
+        async variants(word, { limit = 5 } = {}) {
+            const list = await syllables();
+            if (!Object.keys(list).length)
+                return [];
+            const written = word.trim().normalize('NFC').toLowerCase();
+            const parts = written.split(/\s+/);
+            const combos = accentCombinations(parts, syllablesByPlain(list), {
+                perUnit: parts.length === 1 ? VARIANT_FORMS_ONE_SYLLABLE : VARIANT_FORMS_PER_SYLLABLE,
+                max: MAX_VARIANT_COMBINATIONS,
+            }).filter((c) => c !== written);
+            const found = await Promise.all(combos.map(async (c) => ({ word: c, entries: await lookup(c) })));
+            const frequency = (v) => {
+                const known = v.entries.map((e) => e.frequency).filter((f) => f !== undefined);
+                return known.length ? Math.max(...known) : parts.length === 1 ? list[v.word] || undefined : undefined;
+            };
+            // Most common first; ties (and unknown frequencies) keep the order of more common syllable forms.
+            return found
+                .filter((v) => v.entries.length)
+                .map((v) => ({ ...v, frequency: frequency(v) }))
+                .sort((a, b) => (b.frequency ?? -1) - (a.frequency ?? -1))
+                .slice(0, limit);
         },
         async searchEnglish(term, { region, pos, exclude = DEFAULT_EXCLUDED_LABELS, limit = 10, allSenses = false } = {}) {
             const key = normalizeEnglish(term);
@@ -107,14 +152,6 @@ export function createDictionary(options) {
             const m = await meta();
             return m.picks ? loadOnce('picks.json') : [];
         },
-        async checker() {
-            const m = await meta();
-            return m.checker ? loadOnce('checker.json') : {};
-        },
-        async syllables() {
-            const m = await meta();
-            return m.syllables ? loadOnce('syllables.json') : {};
-        },
         async frames() {
             const m = await meta();
             if (!m.frames)
@@ -154,3 +191,4 @@ export * from "./check.js";
 export * from "./text.js";
 export * from "./phrasebook.js";
 export * from "./review.js";
+export * from "./detect.js";

package/dist/phrasebook.d.ts

@@ -39,6 +39,8 @@ export type FilledFrame = {
     }[];
     /** true when every content slot is filled (and translated). */
     complete: boolean;
+    /** true for frames that can be part of a sentence ("but it was {quality}"). */
+    clause?: boolean;
 };
 /** A content slot value: English (translated for you) or `{ text }` in the target language. */
 export type SlotValue = string | {

package/dist/phrasebook.js

@@ -114,7 +114,7 @@ export function createPhrasebook(options = {}) {
         const slots = Object.entries(frame.slots).map(([name, pos]) => ({ name, pos }));
         const en = Object.keys(frame.slots).reduce((s, name) => (typeof values[name] === 'string' ? s.replace(`{${name}}`, values[name]) : s), frame.en[0]);
         return {
-            id: frame.id, topic: frame.topic, en, text,
+            id: frame.id, topic: frame.topic, en, text, ...(frame.clause ? { clause: true } : {}),
             parts: parts.filter((p) => p.text),
             slots,
             complete: slots.every(({ name }) => values[name] !== undefined) && !parts.some((p) => p.untranslated),

package/dist/review.d.ts

@@ -2,17 +2,20 @@ import { type CheckRule } from './check.ts';
 import { type FrameOptions } from './phrasebook.ts';
 import { type TranslatorOptions } from './translate.ts';
 export type ReviewOptions = FrameOptions & {
-    /** The learner's own language, for the parts written in it (default "en"). */
+    /** The learner's own language, whose words are translated into the target language (default "en"). */
     base?: string;
+    /** Checks to add to the spellchecker, e.g. { dialect: true, 'pronoun-consistency': true } (see CheckRule). */
+    checks?: Partial<Record<CheckRule, boolean>>;
 };
 export type ReviewChange = {
-    /** What was written (empty when a missing word was added). */
+    /** What was written. */
     from: string;
+    /** The correction; empty when the word is left out (Vietnamese has no "the"). */
     to: string;
     /** One sentence for the learner. */
     why: string;
-    /** 'translated' (a base-language word), 'frame' (a sentence frame), or the checker rule. */
-    kind: 'translated' | 'frame' | CheckRule;
+    /** 'frame' (a base-language sentence or clause filled into a sentence frame), or the checker rule. */
+    kind: 'frame' | CheckRule;
 };
 /** A checker suggestion that wasn't applied ("end with ạ"), for the learner to consider. */
 export type ReviewHint = {
@@ -40,7 +43,7 @@ export type ReviewSentence = {
         en: string;
         text: string;
     }[];
-    /** Parts the review couldn't handle (longer base-language text that follows no frame), left as written. */
+    /** Words the review couldn't correct (a base-language word with no translation), left as written. */
     unchecked: string[];
 };
 export type Review = {

package/dist/review.js

@@ -1,24 +1,26 @@
 // Reviewing a journal entry written in the target language, the learner's own language, or a mix of both
-// ("Hôm nay tôi đi market với má"). For each sentence:
+// ("Hôm nay tôi đi market với má"), and giving the corrected entry in the target language. For each sentence:
 //
-//   1. Each word is marked as the target or the base language, from the two dictionaries alone (accents and
-//      syllables, headwords, inflected forms); words that could be either take their neighbors' language.
-//   2. Base-language parts are replaced: a whole sentence or clause that follows a sentence frame is filled
-//      in ("but it was expensive" → "nhưng mắc quá"), short phrases are translated word by word ("market" →
-//      "chợ"), and anything longer is listed as unchecked rather than guessed.
-//   3. The grammar checker runs on the whole entry, so consistency checks see every sentence ("tôi" here,
-//      "tui" there); its errors and warnings are applied, its suggestions returned as hints.
-//   4. The corrected sentence is checked against sentence frames ("Bạn có đi?" → "Bạn có đi không?").
+//   1. Each word is marked as the target or the base language (detect.ts), from the two dictionaries alone.
+//   2. A whole base-language sentence or clause that follows a sentence frame is filled in ("but it was
+//      expensive" → "nhưng mắc quá"; "Where is the bathroom?" → "Phòng tắm ở đâu?").
+//   3. The checker runs on the whole entry. By default that's the spellchecker: every misspelled word gets
+//      its accents ("khong" → "không") and every remaining base-language word its translation ("market" →
+//      "chợ"); opt-in checks (`checks`: regions, pronouns, consistency, polite endings) join it. Errors and
+//      warnings are applied; suggestions are returned as hints.
+//   4. The sentence frames the corrected sentence follows are listed, for the learner to reuse. Frames don't
+//      change the sentence.
 //
 // Everything is rule-based and runs on the language's data: no API, and nothing language-specific here.
 import { createChecker } from "./check.js";
+import { markLanguages } from "./detect.js";
 import { createDictionary } from "./index.js";
 import { createPhrasebook } from "./phrasebook.js";
-import { plain, sentencesOf, units } from "./text.js";
+import { sentencesOf, units } from "./text.js";
 import { createTranslator } from "./translate.js";
-// The longest base-language phrase translated word by word; longer ones need a frame.
-const MAX_TRANSLATED_UNITS = 3;
 const TERMINAL = /^[.!?…]+/;
+// Spaces after a word was left out: one between words, none before punctuation.
+const tidy = (s) => s.replace(/[^\S\n]+/g, ' ').replace(/ ([.,!?…])/g, '$1').trim();
 export function createReviewer(options = {}) {
     const cache = new Map();
     const dict = (lang) => {
@@ -32,53 +34,10 @@ export function createReviewer(options = {}) {
     const translator = createTranslator({ ...options, dictionary: dict });
     const phrasebook = createPhrasebook({ ...options, dictionary: dict, translator });
     const checker = createChecker({ ...options, dictionary: dict });
-    // Which language each unit is in. A unit the target knows and the base doesn't is the target's, and the
-    // other way round; units both or neither know ("an", "to", names) take the nearest decided neighbor's.
-    // In syllable languages, a base-language word that's a target syllable only once accents are added
-    // ("but": bút) is the base language's; a word only the target knows without its accents ("khong") is
-    // still the target's.
-    async function languages(list, lang, base) {
-        const target = dict(lang);
-        const syllables = await target.syllables();
-        const syllableLanguage = Object.keys(syllables).length > 0;
-        const plainSyllables = syllableLanguage ? new Set(Object.keys(syllables).map(plain)) : new Set();
-        // 'exact': a target word as written; 'accents': one only with accents added ("khong" → "không").
-        const targetKnows = async (u) => {
-            const lower = u.text.toLowerCase();
-            if (!syllableLanguage)
-                return (await target.lookup(lower)).length > 0 ? 'exact' : false;
-            if (lower in syllables || lower !== plain(lower))
-                return 'exact';
-            return plainSyllables.has(lower) ? 'accents' : false;
-        };
-        const baseKnows = async (u) => (await translator.senses(u.text, { from: base }).catch(() => [])).length > 0;
-        const marks = await Promise.all(list.map(async (u) => {
-            if (/\d/.test(u.text))
-                return '?';
-            const [t, b] = await Promise.all([targetKnows(u), baseKnows(u)]);
-            if (t === 'exact')
-                return b ? '?' : lang;
-            if (b)
-                return base;
-            return t === 'accents' ? lang : '?';
-        }));
-        return marks.map((m, i) => {
-            if (m !== '?')
-                return m;
-            for (let d = 1; d < marks.length; d++) {
-                if (i - d >= 0 && marks[i - d] !== '?')
-                    return marks[i - d];
-                if (i + d < marks.length && marks[i + d] !== '?')
-                    return marks[i + d];
-            }
-            return lang;
-        });
-    }
     return {
         async review(entry, o) {
             const text = entry.normalize('NFC');
             const base = o.base ?? 'en';
-            const meta = await dict(o.lang).meta();
             // A frame as shown to the learner: its English and its pattern for these settings, slots open.
             const frameInfo = new Map();
             const frameOf = async (id) => {
@@ -93,7 +52,7 @@ export function createReviewer(options = {}) {
                 const start = list[0].start;
                 const coreEnd = list[list.length - 1].end;
                 const end = coreEnd + (TERMINAL.exec(text.slice(coreEnd))?.[0].length ?? 0);
-                const langs = await languages(list, o.lang, base);
+                const langs = base === o.lang ? list.map(() => o.lang) : await markLanguages(list, { lang: o.lang, base, target: dict(o.lang), baseDictionary: dict(base), translator });
                 const groups = [];
                 list.forEach((u, i) => {
                     if (groups.length && groups[groups.length - 1].lang === langs[i])
@@ -102,38 +61,45 @@ export function createReviewer(options = {}) {
                         groups.push({ lang: langs[i], units: [u] });
                 });
                 const draft = {
-                    start, coreEnd, end, replacements: [], frames: [], unchecked: [],
+                    start, coreEnd, end, replacements: [], frames: [],
                     parts: groups.map((g) => ({ text: text.slice(g.units[0].start, g.units[g.units.length - 1].end), lang: g.lang })),
                 };
                 const whole = groups.length === 1 && groups[0].lang === base;
+                const fill = async (span, frame) => {
+                    const from = text.slice(span[0].start, span[span.length - 1].end);
+                    const info = await frameOf(frame.id);
+                    let to = frame.text.replace(/[.!?…]+$/, '');
+                    if (span[0].start !== start)
+                        to = to.charAt(0).toLowerCase() + to.slice(1);
+                    draft.replacements.push({
+                        start: span[0].start, end: span[span.length - 1].end, to,
+                        change: { kind: 'frame', from, to, why: `Sentence frame “${info.en}”` },
+                    });
+                    draft.frames.push(info);
+                };
                 for (const g of groups.filter((g) => g.lang === base)) {
-                    const from = text.slice(g.units[0].start, g.units[g.units.length - 1].end);
-                    const at = { start: g.units[0].start, end: g.units[g.units.length - 1].end };
                     // A whole sentence is matched with its own punctuation ("Where is the bathroom?").
-                    const frame = await phrasebook.match(whole ? text.slice(start, end) : from, o);
-                    if (frame && frame.complete) {
-                        let to = frame.text.replace(/[.!?…]+$/, '');
-                        if (!whole && g.units[0].start !== start)
-                            to = to.charAt(0).toLowerCase() + to.slice(1);
-                        const info = await frameOf(frame.id);
-                        draft.replacements.push({ ...at, to, change: { kind: 'frame', from, to, why: `Sentence frame “${info.en}”` } });
-                        draft.frames.push(info);
+                    const whole_ = whole ? text.slice(start, end) : text.slice(g.units[0].start, g.units[g.units.length - 1].end);
+                    const frame = await phrasebook.match(whole_, o);
+                    if (frame?.complete) {
+                        await fill(g.units, frame);
                         continue;
                     }
-                    if (!whole && g.units.length <= MAX_TRANSLATED_UNITS) {
-                        const english = from.replace(/^(the|a|an)\s+/i, '');
-                        const [group] = await translator.translate(english, { from: base, to: o.lang, toRegion: o.region, register: o.register, limit: 1 }).catch(() => []);
-                        const word = group?.translations[0]?.word;
-                        if (word) {
-                            draft.replacements.push({ ...at, to: word, change: { kind: 'translated', from, to: word, why: `“${english}” in ${meta.name}` } });
+                    // Otherwise a clause frame ending the part, longest first ("the ice cream but it was expensive":
+                    // "but it was expensive"); the rest is translated word by word by the checker. Only spans ending
+                    // with the part, so a clause isn't cut short ("but it was very" with "expensive" left over).
+                    for (let len = g.units.length - 1; len >= 2; len--) {
+                        const span = g.units.slice(g.units.length - len);
+                        const m = await phrasebook.match(text.slice(span[0].start, span[len - 1].end), o);
+                        if (!m?.complete || !m.clause)
                             continue;
-                        }
+                        await fill(span, m);
+                        break;
                     }
-                    draft.unchecked.push(from);
                 }
                 drafts.push(draft);
             }
-            // The entry with base-language parts replaced, keeping where each sentence now is.
+            // The entry with those parts filled in, keeping where each sentence now is.
             let draftText = '';
             let cursor = 0;
             const placed = [];
@@ -151,41 +117,49 @@ export function createReviewer(options = {}) {
                 cursor = d.end;
             }
             draftText += text.slice(cursor);
-            // 3. The checker on the whole entry; 4. frames on each corrected sentence.
-            const { issues } = await checker.check(draftText, { lang: o.lang, region: o.region, listener: o.listener, about: undefined, speaker: o.speaker });
+            // 3. The checker on the whole entry; 4. the frames each corrected sentence follows.
+            const { issues } = await checker.check(draftText, {
+                lang: o.lang, base, region: o.region, listener: o.listener, speaker: o.speaker, register: o.register, rules: o.checks,
+            });
             const sentences = [];
             for (const [i, d] of drafts.entries()) {
                 const p = placed[i];
                 const mine = issues.filter((x) => x.start >= p.start && x.end <= p.coreEnd);
                 const changes = d.replacements.sort((a, b) => a.start - b.start).map((r) => r.change);
                 const hints = [];
+                const unchecked = [];
+                const applied = [];
                 let core = draftText.slice(p.start, p.coreEnd);
                 let lastStart = Infinity;
                 for (const x of [...mine].sort((a, b) => b.start - a.start)) {
-                    const applies = (x.severity === 'error' || x.severity === 'warning') && x.suggestions.length && x.end <= lastStart;
-                    if (!applies) {
-                        hints.push({ rule: x.rule, text: x.text, message: x.message, suggestions: x.suggestions });
+                    if (!x.suggestions.length) {
+                        if (x.rule === 'foreign-word')
+                            unchecked.unshift(x.text);
+                        else
+                            hints.unshift({ rule: x.rule, text: x.text, message: x.message, suggestions: x.suggestions });
+                        continue;
+                    }
+                    if (x.severity === 'suggestion' || x.end > lastStart) {
+                        hints.unshift({ rule: x.rule, text: x.text, message: x.message, suggestions: x.suggestions });
                         continue;
                     }
                     core = core.slice(0, x.start - p.start) + x.suggestions[0] + core.slice(x.end - p.start);
-                    changes.push({ kind: x.rule, from: x.text, to: x.suggestions[0], why: x.message });
+                    applied.unshift({ kind: x.rule, from: x.text, to: x.suggestions[0], why: x.message });
                     lastStart = x.start;
                 }
-                const punctuation = draftText.slice(p.coreEnd, p.end);
-                let corrected = core + punctuation;
+                changes.push(...applied);
+                core = tidy(core);
+                if (/\p{Lu}/u.test(draftText.charAt(p.start)))
+                    core = core.charAt(0).toUpperCase() + core.slice(1);
+                const corrected = core + draftText.slice(p.coreEnd, p.end);
                 const frames = [...d.frames];
                 for (const m of await phrasebook.matchFrames(corrected, o)) {
                     if (!frames.some((f) => f.id === m.frame.id))
                         frames.push(await frameOf(m.frame.id));
-                    if (m.changes.length && corrected === core + punctuation) {
-                        corrected = m.text;
-                        for (const c of m.changes)
-                            changes.push({ kind: 'frame', from: c.from, to: c.to, why: `${c.why} (sentence frame “${m.frame.en}”)` });
-                    }
                 }
                 sentences.push({
                     start: d.start, end: d.end, original: text.slice(d.start, d.end), parts: d.parts,
-                    corrected, changes, hints: hints.reverse(), frames, unchecked: d.unchecked,
+                    corrected, changes, hints, frames, unchecked,
                 });
             }
             let corrected = '';

package/dist/text.d.ts

@@ -29,3 +29,16 @@ export declare function segment(text: string, list: Unit[], maxUnits: number, lo
 export declare const plain: (s: string) => string;
 /** Keeps a capital first letter when replacing a capitalized word ("Tôi" → "Con"). */
 export declare const matchCase: (original: string, replacement: string) => string;
+/** A language's syllables grouped by their form without accents, most common first. */
+export declare function syllablesByPlain(syllables: Record<string, number>): Map<string, string[]>;
+/**
+ * Ways to write a run of syllables with other accents ("hom nay" → "hôm nay", "hơm nay", …): each
+ * syllable's forms that share its letters (the most common `perUnit`, plus the syllable as written), in
+ * every combination, the ones made of more common forms first, at most `max`. Syllables that mustn't
+ * change (`fixed[i]`) keep their written form. The written run itself is included.
+ */
+export declare function accentCombinations(written: string[], byPlain: Map<string, string[]>, { fixed, perUnit, max }: {
+    fixed?: boolean[];
+    perUnit: number;
+    max: number;
+}): string[];

package/dist/text.js

@@ -55,3 +55,43 @@ export async function segment(text, list, maxUnits, lookup) {
 export const plain = (s) => s.normalize('NFD').replace(/\p{M}/gu, '').replace(/đ/g, 'd').replace(/Đ/g, 'D').normalize('NFC');
 /** Keeps a capital first letter when replacing a capitalized word ("Tôi" → "Con"). */
 export const matchCase = (original, replacement) => original.charAt(0) !== original.charAt(0).toLowerCase() ? replacement.charAt(0).toUpperCase() + replacement.slice(1) : replacement;
+// Syllables grouped by their plain form, most common first: "khong" → ["không", "khống", …]. Built once
+// per syllable list.
+const byPlainCache = new WeakMap();
+/** A language's syllables grouped by their form without accents, most common first. */
+export function syllablesByPlain(syllables) {
+    let byPlain = byPlainCache.get(syllables);
+    if (!byPlain) {
+        byPlain = new Map();
+        for (const s of Object.keys(syllables))
+            byPlain.set(plain(s), [...(byPlain.get(plain(s)) ?? []), s]);
+        for (const list of byPlain.values())
+            list.sort((a, b) => syllables[b] - syllables[a]);
+        byPlainCache.set(syllables, byPlain);
+    }
+    return byPlain;
+}
+/**
+ * Ways to write a run of syllables with other accents ("hom nay" → "hôm nay", "hơm nay", …): each
+ * syllable's forms that share its letters (the most common `perUnit`, plus the syllable as written), in
+ * every combination, the ones made of more common forms first, at most `max`. Syllables that mustn't
+ * change (`fixed[i]`) keep their written form. The written run itself is included.
+ */
+export function accentCombinations(written, byPlain, { fixed = [], perUnit, max }) {
+    const choices = written.map((w, i) => {
+        const lower = w.toLowerCase();
+        if (fixed[i])
+            return [lower];
+        const forms = (byPlain.get(plain(lower)) ?? []).slice(0, perUnit);
+        return forms.includes(lower) ? forms : [...forms, lower];
+    });
+    // Every combination with the sum of its forms' ranks: lower sums are made of more common forms.
+    let combos = [{ words: [], rank: 0 }];
+    for (const options of choices) {
+        combos = combos.flatMap((c) => options.map((o, r) => ({ words: [...c.words, o], rank: c.rank + r })));
+    }
+    return combos
+        .sort((a, b) => a.rank - b.rank)
+        .slice(0, max)
+        .map((c) => c.words.join(' '));
+}

package/dist/types.d.ts

@@ -195,6 +195,11 @@ export type CheckerConfig = {
     ambiguousPronouns?: string[];
     /** Sentence endings expected when speaking up (pronoun-table rows marked `respect`), e.g. Vietnamese "ạ". */
     politeEndings?: string[];
+    /**
+     * English words (English is the bridge) this language usually has no word for, so the checker suggests
+     * leaving them out rather than translating them: Vietnamese has no articles ("the", "a", "an").
+     */
+    leaveOut?: string[];
 };
 /**
  * Hand-picked first choices for one English meaning, in one target language (picks.json). A speaker's

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "which-dialect",
-  "version": "0.3.0",
+  "version": "0.5.0",
   "description": "Translate words between languages and dialects, meaning by meaning, starting with Vietnamese (Northern, Central, Southern) and English, with Wiktionary data loaded on demand. More languages are planned.",
   "type": "module",
   "main": "./dist/index.js",