which-dialect 0.1.1 → 0.2.0

package/README.md

@@ -18,6 +18,8 @@ await tr.translate('cool', { from: 'en', to: 'vi', toRegion: 'Southern', meaning
 
 Each result is a list of **groups, one per meaning** of the source word, each with its part of speech, its definitions and its translations, so a word's meanings never get mixed up.
 
+**Data:** Wiktionary data of 2026-09-25 for both languages. See [`DATA_UPDATES.md`](DATA_UPDATES.md) for each language's dates and update history.
+
 You install one small package. Dictionary data is **not bundled**: each lookup fetches one small file (usually 5–100 kB compressed) from the language's data package on the jsDelivr CDN, and caches it.
 
 ## Languages
@@ -60,11 +62,12 @@ Loads each language's data on demand. The options are only needed to self-host t
 | `exclude` | Labels to leave out (default: vulgar, offensive, derogatory, archaic, obsolete, dated, historical, rare, abbreviation). |
 | `limit` | Translations per meaning (default 5). |
 | `allSenses` | Return every meaning, including ones with no translation, instead of only useful ones. For letting users pick a meaning. |
+| `picks` | Use the target language's [hand-picked words](#hand-picked-words) (default `true`). `false` returns the ranking alone. |
 
 It returns `TranslationGroup[]`, most relevant meaning first. Each group has:
 
 - `source`: the meaning being translated: `lemma` (the base or standard word, e.g. *say* for *said*), `pos`, `glosses` (its English definitions), `regions`, `labels`, and `via` when the word led there ("simple past of say", "Southern Vietnam form of không").
-- `translations`: target words, best first, each with its own `pos`, `gloss`, `regions`, `labels`, a `score`, and `bridge` (how it was found; `'pronouns'` for the listener's words, which also carry `relationship`, e.g. "Your parents").
+- `translations`: target words, best first, each with its own `pos`, `gloss`, `regions`, `labels`, a `score`, and `bridge` (how it was found; `'picked'` for [hand-picked words](#hand-picked-words); `'pronouns'` for the listener's words, which also carry `relationship`, e.g. "Your parents").
 - `pronounUses` (source words in the source language's pronoun table): the relationships the word is used in with this meaning, `{ id, label, person: 'self' | 'addressee', speaker?, regions, note?, warning? }`. Uses no definition covers get a group of their own, translated as "I" or "you": Vietnamese *em* → "I/me, when talking to: someone a bit older (man); …; your teacher" and "you, when talking to: someone younger; …"; *ba* → "you, when talking to: your parents". Pass `pos: 'pron'` to get only these.
 - `relationships` (singular "I"/"you" meanings into a language with a pronoun table): the word for each relationship, `{ id, label, words, warning? }`, filtered by `toRegion`, `speaker` and `exclude`, for showing "it depends on who you're talking to".
 
@@ -83,19 +86,69 @@ Every language's data defines its words in English, so English is the bridge bet
 5. **Within one language, synonyms are direct equivalents.** Wiktionary lists dialect words as synonyms (*ngô* → *bắp*, *lợn* → *heo*, *lift* → *elevator*), in either direction (*dạ* lists *vâng*).
 6. **Ranking** favors **common words** ([word frequencies](#word-frequencies): *anh* over the niche *cô nương* for "you"), target words whose definition shares the source definition's details, whose *main* meaning is the match, that are tagged for the target region, and that keep the **register**: a polite word translates to a polite word (Northern *vâng* → Southern *dạ*), slang to slang.
 
+**Hand-picked words** come before all of these for the meanings that have them (see below).
+
 Meanings are ordered by `meaning` (if given), then by being tagged for `fromRegion`, then the dictionary's own order, which lists main meanings first. That order is a guess when a word has several parts of speech: English *just* is listed as an adjective ("fair") before the adverb. Pass `pos` or `meaning` when you know which one you want.
 
+### Hand-picked words
+
+For common words, a speaker of the target language knows the natural word even where the data ranks another first: *get* "To fetch, bring, take" ranks Vietnamese *đưa* ("to hand, to bring") above *lấy*, because *đưa*'s definition matches more of the English words and no *lấy* definition says "fetch". Each language can list **hand-picked words for English meanings** (`picks` in `languages/<code>.ts`, built into `picks.json`). In a meaning that has picks, the picked words come first (`bridge: 'picked'`), then the ranked words; other meanings are unchanged. Picks can carry a region tag, like translation tables: with `toRegion`, picks tagged for it come first and picks tagged only for other regions are left to the ranking. A pick can also mark its meaning as the one to put first when no `pos` or `meaning` is given.
+
+Picks are keyed by English meaning, so each language needs one list however many languages there are: translating from any other language reaches them through the English meaning (as with translation tables). They're optional: a language without picks uses the ranking alone. `picks: false` shows the ranking alone, and `npm run evaluate` reports both. Vietnamese picks so far: *get* "fetch" → *lấy*, *mang*; *get* "obtain" → *lấy*.
+
 ### How accurate is it?
 
 `npm run evaluate` runs known-correct translations across language pairs and dialect pairs (English → Vietnamese by region, including pronouns, Northern → Southern Vietnamese, US ↔ UK, Vietnamese → English). A case passes when a correct word is in the top 3 of the first meaning:
 
 | Set | Correct word in top 3 | Correct word first |
 |---|---|---|
-| Tuning set (used while developing) | 42 / 42 (100%) | 42 / 42 (100%) |
+| Tuning set (used while developing) | 49 / 49 (100%) | 49 / 49 (100%); 48 / 49 without picks |
 | Held-out set (written afterwards, not tuned against) | 17 / 17 (100%) | 16 / 17 (94%) |
 
 The cases involving Spanish were removed with it. The held-out set has since been looked at, so new held-out cases are needed before relying on it (see [`FUTURE_IMPROVEMENTS.md`](FUTURE_IMPROVEMENTS.md)).
 
+## Checking text
+
+### `createChecker({ baseUrl?, load?, dictionary? })` → `checker.check(text, options)`
+
+Checks a learner's sentence in one language and returns issues with the exact place in the text, a short explanation and suggested fixes. It runs in the browser like the rest of the library: no API calls, no cost.
+
+```ts
+import { createChecker } from 'which-dialect'
+
+const checker = createChecker()
+const { text, issues } = await checker.check('Tôi muốn ăn thịt lợn và bắp khong?', {
+  lang: 'vi', region: 'Southern', listener: 'parent',
+})
+// “Tôi”: Talking to your parents, say “con” for “I”, not “Tôi”.               → Con
+// “lợn”: “lợn” is Northern; in Southern Vietnamese, say “heo”.                → heo, cúi, ỉn
+// “khong”: “khong” isn't a Vietnamese syllable. Did you mean “không”, “khổng” or “khống”?
+// “khong”: Talking to your parents, end the sentence with “ạ” to sound polite. (suggestion)
+```
+
+| Option | Meaning |
+|---|---|
+| `lang` | The text's language. |
+| `region` | The region the text should be in. Without it, words that don't fit the rest of the text's region are pointed out (*lợn*, Northern, next to *nha*, Southern). |
+| `listener` | Who you're talking to, as a [pronoun-table](#looking-words-up) row id (`'parent'`, `'teacher'`, `'friend'`, …): pronouns and polite endings are checked against it. |
+| `about` | Who you're talking about, for "he/she" and "they" pronouns. |
+| `speaker` | `'male'` or `'female'`, for pronouns that depend on it. |
+| `rules` | Turn checks off, e.g. `{ 'polite-ending': false }`. |
+
+Each issue has `rule`, `severity` (`'error'`: not a word of the language; `'warning'`: wrong for the chosen region or relationship; `'suggestion'`: often better), `start`/`end` (offsets into the returned `text`, the input in Unicode NFC form), `text`, `message` and `suggestions` (replacements for `text`, best first).
+
+The checks:
+
+- **`spelling`**: syllables that aren't in the language, with the accented forms they could be (*khong* → *không*). Plain-letter words with no accented form are left alone (names, foreign words). Languages written in syllables only (Vietnamese).
+- **`dialect`**: words whose main meaning is tagged for another region (*lợn* in Southern text → *heo*; *muỗng* in Northern → *thìa*; in English, US *apartment* in UK text → *flat*). Words with an untagged main meaning are never flagged (*má* is "cheek" everywhere).
+- **`pronoun-relationship`**: pronouns that don't fit the listener (*tôi* to a parent → *con*; to a teacher → *em*).
+- **`pronoun-pair`**: "I" and "you" pronouns that never go together (*tao* goes with *mày*, not *bạn*), when no listener is given.
+- **`polite-ending`**: a suggestion to end sentences with *ạ* when talking to parents, elders and teachers (not for exclamations).
+
+The checker only speaks up when it's confident, so **no issues means none of the checks found anything, not that the text is correct**. It doesn't check word order, classifiers or whether a sentence sounds natural yet. Only reliable pronouns are checked: kinship words like *con* (also "child", and a classifier in *con chó*) aren't, and *bạn* (also "friend") only gets suggestions.
+
+**Every language uses the same checks.** They read only the language's data and its `checker` settings in `languages/<code>.ts` (how text splits into words, which pronouns are reliable, polite endings), so a check whose data a language doesn't have is skipped. A language without checker settings still gets the `dialect` check.
+
 ## Looking words up
 
 ### `createDictionary({ lang, baseUrl?, load? })`
@@ -104,6 +157,8 @@ One language's dictionary. `baseUrl` defaults to `https://cdn.jsdelivr.net/npm/w
 
 - `dictionary.lookup(word)`: all entries for a word, with every sense's definitions, regions, labels, synonyms and variant links (`altOf`).
 - `dictionary.searchEnglish(term, { region?, pos?, exclude?, limit?, allSenses? })`: words for an English term in this language, one per word, best first (no meaning handling: use the translator for that). `allSenses: true` returns every matching sense of those words instead.
+- `dictionary.checker()` / `dictionary.syllables()`: the language's checker settings and syllable list (with frequencies), used by `createChecker` (`{}` for languages without them).
+- `dictionary.picks()`: the language's [hand-picked words](#hand-picked-words), `{ word, pos, gloss, picks: [{ word, tags? }], first? }` per English meaning (empty for languages without any).
 - `dictionary.meta()`: the language's name, regions, region groups, source, license, build date and counts.
 - `dictionary.pronouns({ listener?, region?, speaker?, exclude? })`: how to say "I", "you", "he/she", "we", plural "you" and "they" depending on who you're talking to (or about), one row per relationship, for languages whose pronouns depend on it (so far Vietnamese; empty for others). Each row has the columns `self`, `addressee`, `third`, `selfPlural`, `addresseePlural` and `thirdPlural`. Each word says where it comes from: a dictionary definition (`source: 'gloss'`, with the `gloss`), a grammar rule for a regular compound the dictionary doesn't list (`source: 'rule'`, e.g. *các anh*, with a `note`), or a hand-written override (`source: 'override'`, with a `note`); and, when it has one, the `gender` of the person it refers to and whether a "we" is `inclusive`.
 
@@ -172,17 +227,26 @@ If you **redistribute modified data**, it has to stay under CC BY-SA 4.0. Using
 
 Requires Node 22.6+ (TypeScript scripts run with `--experimental-strip-types`).
 
+Generated data (`packages/<code>/data`) isn't committed, so a fresh clone needs it first. Two ways:
+
+- **Quick start: `npm run fetch:data`.** Downloads the published data packages from npm (a few seconds), then rebuilds hand-picked words from the repo's configs. You get exactly the published data, so tests and `npm run evaluate` match the numbers in this README. Use this unless you're changing how data is built.
+- **Build from Wiktionary: `npm run build:data:all`.** Downloads Kaikki's files (English 3.3 GB, Vietnamese 79 MB; cached in `.cache/`) and builds every language, **English first**: English is the bridge between languages, and other languages' hand-picked words are checked against it. Kaikki always serves its latest dump, so this can be newer than the published data; the build records the dates in [`DATA_UPDATES.md`](DATA_UPDATES.md). Building one language alone (`npm run build:data -- vi`) also works once English is built.
+
 ```sh
 npm install
-npm run build:data -- vi             # download Kaikki's file for a language (cached in .cache/) and build packages/<code>/data
-npm run build:data -- vi --refresh   # re-download first
+npm run fetch:data                   # quick start: the published data (-- --force replaces data you already have)
+npm run build:data:all               # or build every language from Kaikki, English first (-- --refresh re-downloads)
+npm run build:data -- vi             # build one language (after English); -- vi --refresh re-downloads first
 npm test                             # unit tests, plus checks against whichever languages are built
-npm run evaluate                     # translation accuracy on known-correct cases (-- --verbose to see every case)
+npm run evaluate                     # translation accuracy on known-correct cases, with and without picks (-- --verbose to see every case)
+npm run build:picks -- vi            # rebuild only a language's hand-picked words (after editing `picks` in languages/vi.ts)
+npm run picks-sheet -- vi            # review sheet (.cache/picks-vi.csv) of the top English words' meanings; -- --words file.txt, -- --top 500
+npm run picks-sheet -- vi --apply sheet.csv   # config rows for the rows whose your_pick is filled in
 npm run typecheck
 npm run build                        # compile the API to packages/core/dist
 ```
 
-Generated data isn't committed (it would bloat git history); it's built before publishing. Downloads: Vietnamese 79 MB, English 3.3 GB. Builds take seconds (English: about 25 s and 1.4 GB of memory).
+Generated data isn't committed (it would bloat git history); it's built before publishing. Builds take seconds (English: about 25 s and 1.4 GB of memory).
 
 ### Adding a language
 
@@ -190,13 +254,15 @@ Generated data isn't committed (it would bloat git history); it's built before p
 2. Copy a data package (`packages/vi`) to `packages/<code>` and update its `package.json` and `README.md`.
 3. Add the language's code to `translationLangs` in `languages/en.ts` (so English keeps its translation tables for it) and rebuild English.
 4. `npm run build:data -- <code>`, check the results, add a few real-data tests and evaluation cases, and publish.
+5. Optional: [checker](#checking-text) settings (`checker` in `languages/<code>.ts`): `units: 'syllables'` for languages written in syllables (also builds the syllable list for accent suggestions), `maxWordUnits`, the reliable `pronouns` per pronoun-table column, `ambiguousPronouns`, and `politeEndings`. Without them, the language is read as space-separated words and gets the region check.
+6. Optional, later: [hand-picked words](#hand-picked-words). A speaker runs `npm run picks-sheet -- <code>`, fills in `your_pick` only where the first choice is wrong, and pastes the `--apply` output into `picks`.
 
 Languages in non-Latin scripts (Chinese, Arabic, Russian, …) will need a script-aware version of `shardKey` first: today files are split by the first two Latin letters.
 
 ### Publishing
 
 ```sh
-npm run build:data -- en --refresh && npm run build:data -- vi --refresh && npm test
+npm run build:data:all -- --refresh && npm test    # records the new dates in DATA_UPDATES.md: commit it
 npm publish -w which-dialect-en
 npm publish -w which-dialect-vi
 npm publish -w which-dialect

package/dist/check.d.ts

@@ -0,0 +1,45 @@
+import { type TranslatorOptions } from './translate.ts';
+export type CheckRule = 'spelling' | 'dialect' | 'pronoun-relationship' | 'pronoun-pair' | 'polite-ending';
+export type CheckIssue = {
+    rule: CheckRule;
+    /**
+     * 'error': almost certainly wrong (not a word of the language). 'warning': wrong for the chosen region
+     * or relationship. 'suggestion': often better; worth a look.
+     */
+    severity: 'error' | 'warning' | 'suggestion';
+    /** Character offsets of `text` in the checked text (`CheckResult.text`, the input in Unicode NFC form). */
+    start: number;
+    end: number;
+    /** The text the issue is about. */
+    text: string;
+    /** One sentence for the learner, e.g. “khong” isn't a Vietnamese syllable. Did you mean “không”? */
+    message: string;
+    /** Replacements for `text`, best first; empty when there's nothing specific to suggest. */
+    suggestions: string[];
+};
+export type CheckOptions = {
+    /** Language code of the text, e.g. "vi". */
+    lang: string;
+    /** Region or region group the text should be in, e.g. "Southern". Without it, mixed regions are flagged. */
+    region?: string;
+    /** Who you're talking to, as a pronoun-table row id (see Dictionary.pronouns), e.g. "parent". */
+    listener?: string;
+    /** Who you're talking about, as a pronoun-table row id, for "he/she" and "they" pronouns. */
+    about?: string;
+    /** The speaker's gender, for pronouns that depend on it. */
+    speaker?: 'male' | 'female';
+    /** Turn rules off, e.g. { 'polite-ending': false }. All are on by default. */
+    rules?: Partial<Record<CheckRule, boolean>>;
+};
+export type CheckResult = {
+    /** The checked text: the input in Unicode NFC form, which the issues' offsets refer to. */
+    text: string;
+    /** Sorted by position. */
+    issues: CheckIssue[];
+};
+export type Checker = {
+    check(text: string, options: CheckOptions): Promise<CheckResult>;
+};
+/** Same as the translator's: where each language's data comes from. */
+export type CheckerOptions = TranslatorOptions;
+export declare function createChecker(options?: CheckerOptions): Checker;

package/dist/check.js

@@ -0,0 +1,369 @@
+// Checking text in one language: spelling and accents, words from another region, and pronouns and
+// polite endings that fit who you're talking to.
+//
+// Every rule reads only the language's data: its dictionary, pronoun table, checker.json and
+// syllables.json (see CheckerConfig). Adding a language means adding data and settings in
+// languages/<lang>.ts, not code here. A rule whose data a language doesn't have is skipped, so a language
+// without checker settings is still checked for words from another region.
+//
+// The checker only speaks up when it's confident and stays silent otherwise, so no issues means none of
+// the checks found anything, not that the text is correct.
+import { createDictionary, resolveRegion } from "./index.js";
+import { createTranslator } from "./translate.js";
+import { PRONOUN_PERSONS, } from "./types.js";
+// How each pronoun-table column is named in messages.
+const PERSON_NAMES = {
+    self: 'I', addressee: 'you', third: 'he/she', selfPlural: 'we', addresseePlural: 'you (plural)', thirdPlural: 'they',
+};
+// Columns chosen by `about` (who you're talking about); the rest by `listener`.
+const ABOUT_PERSONS = new Set(['third', 'thirdPlural']);
+const SELF_PERSONS = new Set(['self', 'selfPlural']);
+const ADDRESSEE_PERSONS = new Set(['addressee', 'addresseePlural']);
+// A run of letters (with their accents), digits and apostrophes: a word, or a syllable in syllable languages.
+const UNIT = /[\p{L}\p{M}\p{N}'’]+/gu;
+const SENTENCE_END = /[.!?…]+|\n+/g;
+const quote = (s) => `“${s}”`;
+const lowerFirst = (s) => s.charAt(0).toLowerCase() + s.slice(1);
+const listOf = (items, conjunction = 'and') => items.length <= 1 ? items.join('') : `${items.slice(0, -1).join(', ')} ${conjunction} ${items[items.length - 1]}`;
+// Keeps a capital first letter when replacing a capitalized word ("Tôi" → "Con").
+const matchCase = (original, replacement) => original.charAt(0) !== original.charAt(0).toLowerCase() ? replacement.charAt(0).toUpperCase() + replacement.slice(1) : replacement;
+// Without accents, for finding the accented forms of a syllable: "không" → "khong", "đi" → "di".
+const plain = (s) => s.normalize('NFD').replace(/\p{M}/gu, '').replace(/đ/g, 'd').replace(/Đ/g, 'D').normalize('NFC');
+const unique = (items) => [...new Set(items)];
+function units(text) {
+    return [...text.matchAll(UNIT)].map((m) => ({ text: m[0], start: m.index, end: m.index + m[0].length }));
+}
+function sentencesOf(text, all) {
+    const ends = [...text.matchAll(SENTENCE_END)].map((m) => m.index);
+    const out = [[]];
+    let next = 0;
+    for (const u of all) {
+        while (next < ends.length && ends[next] < u.start) {
+            if (out[out.length - 1].length)
+                out.push([]);
+            next++;
+        }
+        out[out.length - 1].push(u);
+    }
+    return out.filter((s) => s.length);
+}
+// Units join into one word only when just spaces separate them ("thịt heo", not "thịt, heo").
+const joined = (text, a, b) => /^[^\S\n]+$/.test(text.slice(a.end, b.start));
+// Longest match against the dictionary: at each position, the longest run of units that's a headword
+// ("kết quả" is one word, so its "quả" isn't checked on its own). Unknown single units become words with
+// no entries.
+async function segment(text, list, maxUnits, lookup) {
+    const words = [];
+    for (let i = 0; i < list.length;) {
+        let taken = null;
+        for (let k = Math.min(maxUnits, list.length - i); k >= 1; k--) {
+            const span = list.slice(i, i + k);
+            if (span.some((u, j) => j > 0 && !joined(text, span[j - 1], u)))
+                continue;
+            const phrase = span.map((u) => u.text).join(' ');
+            const entries = /\d/.test(phrase) ? [] : await lookup(phrase);
+            if (entries.length || k === 1) {
+                const start = span[0].start;
+                const end = span[k - 1].end;
+                taken = { text: text.slice(start, end), lower: phrase.toLowerCase(), start, end, units: span, entries };
+                break;
+            }
+        }
+        words.push(taken);
+        i += taken.units.length;
+    }
+    return words;
+}
+// --- Rules -----------------------------------------------------------------------------------------------
+// Spelling and accents, for languages written in syllables with a syllable list: a syllable that isn't in
+// the language is flagged, with the accented forms it could be ("khong" → "không"). Plain-letter
+// syllables with no accented form are left alone: usually a name or a foreign word ("email").
+async function spelling(ctx) {
+    if (ctx.config.units !== 'syllables' || !ctx.meta.syllables)
+        return [];
+    const syllables = await ctx.dict.syllables();
+    const byPlain = new Map();
+    for (const s of Object.keys(syllables))
+        byPlain.set(plain(s), [...(byPlain.get(plain(s)) ?? []), s]);
+    for (const list of byPlain.values())
+        list.sort((a, b) => syllables[b] - syllables[a]);
+    const issues = [];
+    for (const word of ctx.sentences.flatMap((s) => s.words)) {
+        if (word.entries.length)
+            continue;
+        for (const u of word.units) {
+            const lower = u.text.toLowerCase().replace(/[’']/g, '');
+            if (/\d/.test(lower) || lower in syllables)
+                continue;
+            const candidates = (byPlain.get(plain(lower)) ?? []).slice(0, 3);
+            const accented = lower !== plain(lower);
+            if (!candidates.length && !accented)
+                continue;
+            issues.push({
+                rule: 'spelling',
+                // Missing accents on plain letters is the common learner slip; a wrong accent is more surely wrong.
+                severity: accented ? 'error' : 'warning',
+                start: u.start,
+                end: u.end,
+                text: u.text,
+                message: candidates.length
+                    ? `${quote(u.text)} isn't a ${ctx.meta.name} syllable. Did you mean ${listOf(candidates.map((c) => quote(matchCase(u.text, c))), 'or')}?`
+                    : `${quote(u.text)} isn't a ${ctx.meta.name} syllable. Check the spelling and accents.`,
+                suggestions: candidates.map((c) => matchCase(u.text, c)),
+            });
+        }
+    }
+    return issues;
+}
+// Words from another region: a word whose main meaning (its first sense) is tagged for regions that don't
+// include the text's region. With `region`, against that; without, against the regions most of the text's
+// regional words share (so "lợn" stands out in otherwise Southern text). Words with an untagged main
+// meaning are never flagged (Vietnamese "má" is "cheek" everywhere, "mother" only in the South).
+async function dialect(ctx) {
+    const all = new Set(ctx.meta.regions);
+    const regional = ctx.sentences
+        .flatMap((s) => s.words)
+        .map((word) => ({ word, sense: word.entries[0]?.senses[0] }))
+        .filter((x) => Boolean(x.sense?.regionTagged && x.sense.regions.length < all.size));
+    if (!regional.length)
+        return [];
+    const issue = (word, regions, message, suggestions, severity) => ({
+        rule: 'dialect',
+        severity,
+        start: word.start,
+        end: word.end,
+        text: word.text,
+        message: `${quote(word.text)} is ${listOf(regions)}${message}` +
+            // The best word only; the others (often niche: Southern "cúi", "ỉn" after "heo") are in suggestions.
+            (suggestions.length ? `, say ${quote(suggestions[0])}.` : '.'),
+        suggestions,
+    });
+    if (ctx.options.region) {
+        const target = resolveRegion(ctx.meta, ctx.options.region);
+        const where = listOf(ctx.meta.regions.filter((r) => target.has(r)));
+        const issues = [];
+        for (const { word, sense } of regional) {
+            if (sense.regions.some((r) => target.has(r)))
+                continue;
+            const suggestions = await ctx.suggestRegional(word, ctx.options.region);
+            issues.push(issue(word, sense.regions, `; in ${where} ${ctx.meta.name}`, suggestions, 'warning'));
+        }
+        return issues;
+    }
+    // No region given: the regions most regional words share are the text's region, and words that fit none
+    // of them stand out. When no region has a majority but some words can't go together ("lợn", Northern,
+    // with "nha", Central and Southern), every regional word is pointed out, without a guess.
+    const counts = new Map();
+    for (const { sense } of regional)
+        for (const r of sense.regions)
+            counts.set(r, (counts.get(r) ?? 0) + 1);
+    const most = Math.max(...counts.values());
+    const majority = new Set([...counts].filter(([, n]) => n === most).map(([r]) => r));
+    const outliers = regional.filter(({ sense }) => !sense.regions.some((r) => majority.has(r)));
+    if (outliers.length) {
+        const where = ctx.meta.regions.filter((r) => majority.has(r));
+        return Promise.all(outliers.map(async ({ word, sense }) => issue(word, sense.regions, `, but the rest of the text is ${listOf(where)}; there`, await ctx.suggestRegional(word, where[0]), 'suggestion')));
+    }
+    const clash = regional.some((a) => regional.some((b) => !a.sense.regions.some((r) => b.sense.regions.includes(r))));
+    if (!clash)
+        return [];
+    const mixed = listOf(regional.map(({ word, sense }) => `${quote(word.text)} (${listOf(sense.regions)})`));
+    return regional.map(({ word, sense }) => issue(word, sense.regions, `. This text mixes regions: ${mixed}. Pick one region`, [], 'suggestion'));
+}
+// Pronouns that don't fit who you're talking to (`listener`) or about (`about`): "tôi" to a parent, where
+// it's "con". Only the language's reliable pronouns are checked (CheckerConfig.pronouns); ones that are
+// also common nouns ("bạn": friend) only get a hedged suggestion.
+async function pronounRelationship(ctx) {
+    if (!ctx.options.listener && !ctx.options.about)
+        return [];
+    const rows = await ctx.rows();
+    const issues = [];
+    for (const m of ctx.pronounMatches()) {
+        const about = ABOUT_PERSONS.has(m.person);
+        const row = rows.find((r) => r.id === (about ? ctx.options.about : ctx.options.listener));
+        const fit = unique((row?.[m.person] ?? []).map((c) => c.word));
+        if (!row || !fit.length || fit.includes(m.lower))
+            continue;
+        const ambiguous = ctx.config.ambiguousPronouns?.includes(m.lower);
+        const who = `${about ? 'Talking about' : 'Talking to'} ${lowerFirst(row.label)}`;
+        issues.push({
+            rule: 'pronoun-relationship',
+            severity: ambiguous ? 'suggestion' : 'warning',
+            start: m.start,
+            end: m.end,
+            text: m.text,
+            message: (ambiguous ? `If ${quote(m.text)} means “${PERSON_NAMES[m.person]}” here: ${lowerFirst(who)}` : who) +
+                `, say ${listOf(fit.map(quote), 'or')} for “${PERSON_NAMES[m.person]}”, not ${quote(m.text)}.`,
+            suggestions: fit.map((w) => matchCase(m.text, w)),
+        });
+    }
+    return issues;
+}
+// "I" and "you" pronouns that never go together (Vietnamese "tao" goes with "mày", not "bạn"): no row of the
+// pronoun table has both. Only when no listener is given; with one, pronoun-relationship covers it.
+async function pronounPair(ctx) {
+    if (ctx.options.listener)
+        return [];
+    const matches = ctx.pronounMatches();
+    const selves = matches.filter((m) => SELF_PERSONS.has(m.person));
+    const yous = matches.filter((m) => ADDRESSEE_PERSONS.has(m.person));
+    if (!selves.length || !yous.length)
+        return [];
+    const rows = await ctx.rows();
+    const words = (r, persons) => [...persons].flatMap((p) => r[p].map((c) => c.word));
+    const issues = [];
+    for (const you of yous) {
+        for (const me of selves) {
+            const withMe = rows.filter((r) => words(r, SELF_PERSONS).includes(me.lower));
+            if (!withMe.length || withMe.some((r) => words(r, ADDRESSEE_PERSONS).includes(you.lower)))
+                continue;
+            const partners = unique(withMe.flatMap((r) => r[you.person].map((c) => c.word)));
+            issues.push({
+                rule: 'pronoun-pair',
+                severity: ctx.config.ambiguousPronouns?.includes(you.lower) ? 'suggestion' : 'warning',
+                start: you.start,
+                end: you.end,
+                text: you.text,
+                message: `${quote(me.text)} and ${quote(you.text)} don't go together` +
+                    (partners.length ? `: with ${quote(me.text)}, say ${listOf(partners.map(quote), 'or')}.` : '.'),
+                suggestions: partners.map((w) => matchCase(you.text, w)),
+            });
+            break;
+        }
+    }
+    return issues;
+}
+// Polite endings when speaking up (the listener's row is marked `respect`): Vietnamese sentences to parents,
+// elders and teachers usually end in "ạ". A suggestion, not an error: it isn't always needed.
+async function politeEnding(ctx) {
+    const endings = ctx.config.politeEndings ?? [];
+    if (!ctx.options.listener || !endings.length)
+        return [];
+    const row = (await ctx.rows()).find((r) => r.id === ctx.options.listener);
+    if (!row?.respect)
+        return [];
+    const issues = [];
+    for (const s of ctx.sentences) {
+        const last = s.units[s.units.length - 1];
+        if (endings.includes(last.text.toLowerCase()))
+            continue;
+        // Exclamations ("Con nhớ má quá!") are often fine without it.
+        if (/^[^\p{L}\p{N}]*!/u.test(ctx.text.slice(last.end)))
+            continue;
+        issues.push({
+            rule: 'polite-ending',
+            severity: 'suggestion',
+            start: last.start,
+            end: last.end,
+            text: last.text,
+            message: `Talking to ${lowerFirst(row.label)}, end the sentence with ${listOf(endings.map(quote), 'or')} to sound polite.`,
+            suggestions: endings.map((e) => `${last.text} ${e}`),
+        });
+    }
+    return issues;
+}
+const RULES = {
+    spelling,
+    dialect,
+    'pronoun-relationship': pronounRelationship,
+    'pronoun-pair': pronounPair,
+    'polite-ending': politeEnding,
+};
+const SEVERITY_ORDER = { error: 0, warning: 1, suggestion: 2 };
+export function createChecker(options = {}) {
+    const cache = new Map();
+    const dict = (lang) => {
+        let d = cache.get(lang);
+        if (!d) {
+            d = options.dictionary?.(lang) ?? createDictionary({ lang, baseUrl: options.baseUrl?.(lang), load: options.load?.(lang) });
+            cache.set(lang, d);
+        }
+        return d;
+    };
+    const translator = createTranslator({ ...options, dictionary: dict });
+    const lookups = new Map();
+    return {
+        async check(input, opts) {
+            const text = input.normalize('NFC');
+            const d = dict(opts.lang);
+            const [meta, config] = await Promise.all([d.meta(), d.checker()]);
+            // Unknown listener/about ids throw, listing the valid ones (as the translator does).
+            if (meta.pronouns) {
+                if (opts.listener)
+                    await d.pronouns({ listener: opts.listener });
+                if (opts.about)
+                    await d.pronouns({ listener: opts.about });
+            }
+            const lookup = (phrase) => {
+                const key = `${opts.lang}\u0000${phrase}`;
+                let pending = lookups.get(key);
+                if (!pending) {
+                    pending = d.lookup(phrase);
+                    lookups.set(key, pending);
+                }
+                return pending;
+            };
+            const sentences = [];
+            for (const list of sentencesOf(text, units(text))) {
+                sentences.push({ units: list, words: await segment(text, list, config.maxWordUnits ?? 3, lookup) });
+            }
+            let rows;
+            let matches;
+            const ctx = {
+                text,
+                options: opts,
+                meta,
+                config,
+                sentences,
+                dict: d,
+                rows: () => (rows ??= meta.pronouns ? d.pronouns({ region: opts.region, speaker: opts.speaker }) : Promise.resolve([])),
+                // The language's reliable pronouns in the text, longest first ("chúng tôi" before "tôi"), matched on
+                // units directly so pronouns the dictionary doesn't list as one word ("tụi mày") are found too.
+                pronounMatches: () => (matches ??= findPronouns(text, sentences, config)),
+                suggestRegional: async (word, region) => {
+                    try {
+                        const [group] = await translator.translate(word.lower, {
+                            from: opts.lang, to: opts.lang, toRegion: region, pos: word.entries[0]?.pos, limit: 3,
+                        });
+                        return (group?.translations ?? []).map((t) => t.word).filter((w) => w !== word.lower);
+                    }
+                    catch {
+                        return [];
+                    }
+                },
+            };
+            const results = await Promise.all(Object.keys(RULES).filter((r) => opts.rules?.[r] !== false).map((r) => RULES[r](ctx)));
+            const issues = results.flat().sort((a, b) => a.start - b.start || SEVERITY_ORDER[a.severity] - SEVERITY_ORDER[b.severity]);
+            return { text, issues };
+        },
+    };
+}
+function findPronouns(text, sentences, config) {
+    const byWord = new Map();
+    for (const person of PRONOUN_PERSONS)
+        for (const w of config.pronouns?.[person] ?? [])
+            byWord.set(w.toLowerCase(), person);
+    if (!byWord.size)
+        return [];
+    const longest = Math.max(...[...byWord.keys()].map((w) => w.split(' ').length));
+    const out = [];
+    sentences.forEach((s, sentence) => {
+        for (let i = 0; i < s.units.length;) {
+            let step = 1;
+            for (let k = Math.min(longest, s.units.length - i); k >= 1; k--) {
+                const span = s.units.slice(i, i + k);
+                if (span.some((u, j) => j > 0 && !joined(text, span[j - 1], u)))
+                    continue;
+                const lower = span.map((u) => u.text.toLowerCase()).join(' ');
+                const person = byWord.get(lower);
+                if (person) {
+                    out.push({ text: text.slice(span[0].start, span[k - 1].end), lower, start: span[0].start, end: span[k - 1].end, person, sentence });
+                    step = k;
+                    break;
+                }
+            }
+            i += step;
+        }
+    });
+    return out;
+}

package/dist/index.d.ts

@@ -1,4 +1,4 @@
-import { type PronounRow, type Entry, type Hit, type LanguageMeta, type PickRow } from './types.ts';
+import { type CheckerConfig, type PronounRow, type Entry, type Hit, type LanguageMeta, type PickRow } from './types.ts';
 export * from './types.ts';
 export * from './pos.ts';
 /**
@@ -67,6 +67,10 @@ export type Dictionary = {
     pronouns(options?: PronounOptions): Promise<PronounRow[]>;
     /** Hand-picked first choices for English meanings (see PickRow). Empty for languages without any. */
     picks(): Promise<PickRow[]>;
+    /** How the checker reads this language (see CheckerConfig); `{}` for languages without settings. */
+    checker(): Promise<CheckerConfig>;
+    /** Syllable → Zipf frequency (0 when unknown), for languages written in syllables; `{}` otherwise. */
+    syllables(): Promise<Record<string, number>>;
 };
 /**
  * The regions a region or region-group name stands for (e.g. English "North America" -> US, Canada),
@@ -76,3 +80,4 @@ export declare function resolveRegion(meta: LanguageMeta, region: string | undef
 export declare function dataUrl(lang: string): string;
 export declare function createDictionary(options: DictionaryOptions): Dictionary;
 export * from './translate.ts';
+export * from './check.ts';

package/dist/index.js

@@ -107,6 +107,14 @@ export function createDictionary(options) {
             const m = await meta();
             return m.picks ? loadOnce('picks.json') : [];
         },
+        async checker() {
+            const m = await meta();
+            return m.checker ? loadOnce('checker.json') : {};
+        },
+        async syllables() {
+            const m = await meta();
+            return m.syllables ? loadOnce('syllables.json') : {};
+        },
         async pronouns({ listener, region, speaker, exclude = DEFAULT_EXCLUDED_LABELS } = {}) {
             const m = await meta();
             if (!m.pronouns)
@@ -132,3 +140,4 @@ export function createDictionary(options) {
     };
 }
 export * from "./translate.js";
+export * from "./check.js";

package/dist/types.d.ts

@@ -110,6 +110,33 @@ export type LanguageMeta = {
     pronouns?: boolean;
     /** true when the language has hand-picked translations (picks.json, see PickRow). */
     picks?: boolean;
+    /** true when the language has checker settings (checker.json, see CheckerConfig). */
+    checker?: boolean;
+    /** true when the language has a syllable list (syllables.json: syllable → Zipf frequency, 0 when unknown). */
+    syllables?: boolean;
+};
+/**
+ * How the checker reads one language (languages/<lang>.ts `checker`, stored as data/checker.json). Every
+ * field is optional: a language without it is read as space-separated words, and gets the checks that
+ * only need its dictionary (dialect mixing).
+ */
+export type CheckerConfig = {
+    /**
+     * What spaces separate: 'words' (English), or 'syllables' that group into dictionary words (Vietnamese
+     * "thịt heo", "kết quả"). With 'syllables' the build also writes syllables.json for accent suggestions.
+     */
+    units?: 'words' | 'syllables';
+    /** The longest dictionary word to look for, in units (default 3). */
+    maxWordUnits?: number;
+    /**
+     * Pronouns that are reliably pronouns, by pronoun-table column ("tôi", "tao", "mày"). The pronoun checks
+     * look only at these: kinship words like Vietnamese "con" (also "child", a classifier) aren't listed.
+     */
+    pronouns?: Partial<Record<PronounPerson, string[]>>;
+    /** Listed pronouns that are also common nouns ("bạn": you, or friend); checked, but only as suggestions. */
+    ambiguousPronouns?: string[];
+    /** Sentence endings expected when speaking up (pronoun-table rows marked `respect`), e.g. Vietnamese "ạ". */
+    politeEndings?: string[];
 };
 /**
  * Hand-picked first choices for one English meaning, in one target language (picks.json). A speaker's
@@ -181,6 +208,8 @@ export type PronounRow = {
     thirdPlural: PronounChoice[];
     /** The row to use when the relationship isn't known (neutral words: Vietnamese "tôi", "bạn"). */
     default?: boolean;
+    /** Speaking up to this person (parents, elders, teachers): polite sentence endings are expected. */
+    respect?: boolean;
     /** A caution for the whole row, e.g. that "tao"/"mày" are rude outside close friendships. */
     warning?: string;
 };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "which-dialect",
-  "version": "0.1.1",
+  "version": "0.2.0",
   "description": "Translate words between languages and dialects, meaning by meaning, starting with Vietnamese (Northern, Central, Southern) and English, with Wiktionary data loaded on demand. More languages are planned.",
   "type": "module",
   "main": "./dist/index.js",