which-dialect 0.1.0

package/LICENSE

@@ -0,0 +1,28 @@
+MIT License
+
+Copyright (c) 2026 nguyen18
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+---
+
+This license covers the code in this repository. The dictionary data in the
+language data packages (packages/<lang>, published as
+which-dialect-<lang>) is derived from Wiktionary and licensed
+separately under CC BY-SA 4.0; see packages/<lang>/LICENSE.

package/README.md

@@ -0,0 +1,205 @@
+# which-dialect
+
+Translate words **between languages and between dialects**, meaning by meaning: Northern Vietnamese *ngô* into Southern *bắp*, English *cool* (as in "awesome") into Southern Vietnamese *ngầu*, English "I" into the right Vietnamese pronoun for who you're talking to, US *truck* into UK *lorry*. Dictionary data comes from [Wiktionary](https://en.wiktionary.org/) (via [Kaikki.org](https://kaikki.org/)) and loads only when you need it.
+
+**Status:** which-dialect is starting with **Vietnamese**, with English as the bridge language, and the current focus is cleaning up Vietnamese translations. **More languages are planned.** The translator and data format are language-general, so adding one is a config file plus a data package (see [Adding a language](#adding-a-language)). Spanish was supported early on and has been removed for now so the work can focus on Vietnamese.
+
+```ts
+import { createTranslator } from 'which-dialect'
+
+const tr = createTranslator()
+
+await tr.translate('ngô', { from: 'vi', fromRegion: 'Northern', to: 'vi', toRegion: 'Southern' }) // bắp
+await tr.translate('I', { from: 'en', to: 'vi', listener: 'parent' })                           // con
+await tr.translate('he', { from: 'en', to: 'vi', toRegion: 'Southern' })                        // ảnh, nó, …
+await tr.translate('truck', { from: 'en', fromRegion: 'US', to: 'en', toRegion: 'UK' })         // lorry
+await tr.translate('cool', { from: 'en', to: 'vi', toRegion: 'Southern', meaning: 'awesome' })  // bá cháy, chất, khét, ngầu (not mát, "cool" the temperature)
+```
+
+Each result is a list of **groups, one per meaning** of the source word, each with its part of speech, its definitions and its translations, so a word's meanings never get mixed up.
+
+You install one small package. Dictionary data is **not bundled**: each lookup fetches one small file (usually 5–100 kB compressed) from the language's data package on the jsDelivr CDN, and caches it.
+
+## Languages
+
+| Language | Code | Data package | Regions | Entries | Region-tagged senses |
+|---|---|---|---|---|---|
+| Vietnamese | `vi` | [`which-dialect-vi`](packages/vi) | Northern, Central, Southern | 35,394 | 1,330 of 42,419 (3.1%) |
+| English | `en` | [`which-dialect-en`](packages/en) | 15 countries/areas (US, UK, Australia, India, …), plus groups: North America, British Isles, Commonwealth, … | 594,544 | 37,339 of 776,085 (4.8%) |
+
+Vietnamese and English can be translated in both directions, and between any of their regions, including between two dialects of one language (Northern → Southern Vietnamese, US → UK English). English is also the bridge every language's definitions are written in.
+
+More languages are planned. Each one is a config file in [`languages/`](languages) plus a data package in `packages/<code>`; see [Adding a language](#adding-a-language).
+
+## Install
+
+```sh
+npm install which-dialect
+```
+
+Works in browsers and in Node 18+ (anywhere with `fetch`).
+
+## Translating
+
+### `createTranslator({ baseUrl?, load?, dictionary? })`
+
+Loads each language's data on demand. The options are only needed to self-host the data: `baseUrl(lang)` returns where a language's data folder is served, and `load(lang)` returns a custom loader (e.g. reading from disk in Node).
+
+### `translator.translate(word, options)`
+
+| Option | Meaning |
+|---|---|
+| `from`, `to` | Language codes. They can be the same, to translate between dialects of one language. |
+| `fromRegion`, `toRegion` | Region or region group, e.g. `'Southern'`, `'Northern'`, `'UK'`, `'North America'`. `fromRegion` keeps only the word's senses used there; `toRegion` ranks words tagged for it first and leaves out words tagged only for other regions. |
+| `pos` | Only source senses with this part of speech, e.g. `'noun'`. |
+| `meaning` | A few words describing the meaning you want, e.g. `'awesome'` for *cool*. Matching senses and words rank first. |
+| `register` | `'casual'`, `'neutral'` or `'polite'`: the register you want translations in, instead of the source's own. `'casual'` favors colloquial words (Southern *tui* for "I"). |
+| `listener` | Who you're talking to, as a row id of the target language's pronoun table (Vietnamese: `'parent'`, `'older-male'`, `'friend'`, …; see `dictionary.pronouns()`). For "I"/"me", "you", "we"/"us" and plural "you" that relationship's words come first: *I* to a parent → *con*; *you* → Southern *ba*, *má*; *we* → *chúng con*. Without it, the table's neutral default row leads (*I* → *tôi*, *you* → *bạn*). Ignored for languages without a pronoun table. |
+| `about` | Who you're talking about, as a row id like `listener`, for "he"/"him", "she"/"her" and "they"/"them": *he* about a grandparent → *ông ấy* (Southern *ổng*); *she* about a teacher → *cô*. Without it, the default row leads (*he* → *anh ấy*, Southern *ảnh*; *they* → *họ*). |
+| `speaker` | `'male'` or `'female'`, for pronouns that depend on it (*anh* vs *chị* toward someone younger). |
+| `exclude` | Labels to leave out (default: vulgar, offensive, derogatory, archaic, obsolete, dated, historical, rare, abbreviation). |
+| `limit` | Translations per meaning (default 5). |
+| `allSenses` | Return every meaning, including ones with no translation, instead of only useful ones. For letting users pick a meaning. |
+
+It returns `TranslationGroup[]`, most relevant meaning first. Each group has:
+
+- `source`: the meaning being translated: `lemma` (the base or standard word, e.g. *say* for *said*), `pos`, `glosses` (its English definitions), `regions`, `labels`, and `via` when the word led there ("simple past of say", "Southern Vietnam form of không").
+- `translations`: target words, best first, each with its own `pos`, `gloss`, `regions`, `labels`, a `score`, and `bridge` (how it was found; `'pronouns'` for the listener's words, which also carry `relationship`, e.g. "Your parents").
+- `pronounUses` (source words in the source language's pronoun table): the relationships the word is used in with this meaning, `{ id, label, person: 'self' | 'addressee', speaker?, regions, note?, warning? }`. Uses no definition covers get a group of their own, translated as "I" or "you": Vietnamese *em* → "I/me, when talking to: someone a bit older (man); …; your teacher" and "you, when talking to: someone younger; …"; *ba* → "you, when talking to: your parents". Pass `pos: 'pron'` to get only these.
+- `relationships` (singular "I"/"you" meanings into a language with a pronoun table): the word for each relationship, `{ id, label, words, warning? }`, filtered by `toRegion`, `speaker` and `exclude`, for showing "it depends on who you're talking to".
+
+### `translator.senses(word, { from, fromRegion?, pos? })`
+
+The meanings of a word, following forms and variant spellings: useful for letting a user pick which meaning they mean before translating.
+
+## How translation works
+
+Every language's data defines its words in English, so English is the bridge between any two languages. For each **meaning** of the source word:
+
+1. **Forms and variants go to the word they belong to:** irregular forms from the data (*said* → *say*), regular English forms by rule (*walked* → *walk*, *running* → *run*), texting spellings (*dont* → *don't*), and regional variants (*hông* → *không*, keeping its Southern region).
+2. **English terms carry the meaning:** the terms in its definition ("bắp": *corn*), or for English words the word itself and its synonyms.
+3. **The target is searched with a compatible part of speech** (`compatiblePos`): a noun for a noun, but an English auxiliary verb may be a Vietnamese particle, and an adjective may be a verb in another language.
+4. **Wiktionary's translation tables** give the usual translation of each English meaning in each language, often with region and register tags ("I" → Vietnamese *tôi*, *tớ* (informal), *tui* (South); *cool* "mildly low temperature" → *mát*). Listed words get a strong boost, more when tagged for the target region, less when tagged only for other regions. From another language, the English meaning whose table lists the source word is the same meaning.
+5. **Within one language, synonyms are direct equivalents.** Wiktionary lists dialect words as synonyms (*ngô* → *bắp*, *lợn* → *heo*, *lift* → *elevator*), in either direction (*dạ* lists *vâng*).
+6. **Ranking** favors **common words** ([word frequencies](#word-frequencies): *anh* over the niche *cô nương* for "you"), target words whose definition shares the source definition's details, whose *main* meaning is the match, that are tagged for the target region, and that keep the **register**: a polite word translates to a polite word (Northern *vâng* → Southern *dạ*), slang to slang.
+
+Meanings are ordered by `meaning` (if given), then by being tagged for `fromRegion`, then the dictionary's own order, which lists main meanings first. That order is a guess when a word has several parts of speech: English *just* is listed as an adjective ("fair") before the adverb. Pass `pos` or `meaning` when you know which one you want.
+
+### How accurate is it?
+
+`npm run evaluate` runs known-correct translations across language pairs and dialect pairs (English → Vietnamese by region, including pronouns, Northern → Southern Vietnamese, US ↔ UK, Vietnamese → English). A case passes when a correct word is in the top 3 of the first meaning:
+
+| Set | Correct word in top 3 | Correct word first |
+|---|---|---|
+| Tuning set (used while developing) | 42 / 42 (100%) | 42 / 42 (100%) |
+| Held-out set (written afterwards, not tuned against) | 17 / 17 (100%) | 16 / 17 (94%) |
+
+The cases involving Spanish were removed with it. The held-out set has since been looked at, so new held-out cases are needed before relying on it (see [`FUTURE_IMPROVEMENTS.md`](FUTURE_IMPROVEMENTS.md)).
+
+## Looking words up
+
+### `createDictionary({ lang, baseUrl?, load? })`
+
+One language's dictionary. `baseUrl` defaults to `https://cdn.jsdelivr.net/npm/which-dialect-<lang>@0.1/data`.
+
+- `dictionary.lookup(word)`: all entries for a word, with every sense's definitions, regions, labels, synonyms and variant links (`altOf`).
+- `dictionary.searchEnglish(term, { region?, pos?, exclude?, limit?, allSenses? })`: words for an English term in this language, one per word, best first (no meaning handling: use the translator for that). `allSenses: true` returns every matching sense of those words instead.
+- `dictionary.meta()`: the language's name, regions, region groups, source, license, build date and counts.
+- `dictionary.pronouns({ listener?, region?, speaker?, exclude? })`: how to say "I", "you", "he/she", "we", plural "you" and "they" depending on who you're talking to (or about), one row per relationship, for languages whose pronouns depend on it (so far Vietnamese; empty for others). Each row has the columns `self`, `addressee`, `third`, `selfPlural`, `addresseePlural` and `thirdPlural`. Each word says where it comes from: a dictionary definition (`source: 'gloss'`, with the `gloss`), a grammar rule for a regular compound the dictionary doesn't list (`source: 'rule'`, e.g. *các anh*, with a `note`), or a hand-written override (`source: 'override'`, with a `note`); and, when it has one, the `gender` of the person it refers to and whether a "we" is `inclusive`.
+
+```js
+const vi = createDictionary({ lang: 'vi' })
+const [parents] = await vi.pronouns({ listener: 'parent', region: 'Southern' })
+parents.self.map((c) => c.word)      // ['con']
+parents.addressee.map((c) => c.word) // ['ba', 'má', 'mẹ']
+```
+
+Vietnamese rows: `general` (the default: neutral words), `friend`, `close-friend`, `older-male`, `older-female`, `younger`, `parent`, `parents-age`, `grandparents-age`, `teacher`, `partner`, `formal`. `speaker: 'male' | 'female'` resolves choices like *anh* vs *chị* toward someone younger.
+
+## Parts of speech
+
+Entries and results use Wiktionary's part-of-speech codes (`noun`, `verb`, `adj`, `adv`, `pron`, `det`, `prep`, `conj`, `intj`, `particle`, `classifier`, `num`, `phrase`, `contraction`, …).
+
+- `POS_NAMES` / `posName(code)`: readable names and one-line explanations for learners (`posName('adj')` → "Adjective"; `particle`: "A small word that adds grammar or tone rather than meaning, e.g. Vietnamese đã (past)…").
+- `compatiblePos(pos)`: the parts of speech that can translate one from any language.
+
+## How regions work
+
+Each language defines its regions: dialect areas for Vietnamese, countries and areas for English. Wiktionary tags some senses with a region; for example Vietnamese *lợn* "pig" is tagged Northern and *heo* Central and Southern. **Senses with no region tag are treated as used in every region**, and marked `regionTagged: false`, so you can tell a confirmed regional word from an assumed one.
+
+- **Per sense, not per word:** Vietnamese *má* means "cheek" everywhere, but "mother" only in the South.
+- **Groups:** a language can define groups of regions. Wiktionary's group tags (English "British Isles", "North America") count for every region in the group, and you can search a group as a whole.
+- **Variants:** words recorded as a variant of another ("Southern Vietnam form of *không*") are found by the other word's meanings, so "not" finds *hông*, and keep their region when translated.
+
+## Limitations
+
+This is a suggestion tool, not a curated translation dictionary. Check results before teaching them.
+
+- **Few region tags.** Most senses carry no region (see the table), so they're assumed to be used everywhere, including some words that are really old-fashioned or local.
+- **No word frequencies.** The source doesn't say how common a word or meaning is. Without `pos`/`meaning`, the first meaning follows the dictionary's order, which isn't always the most common one.
+- **English is the bridge.** Translating between two non-English languages goes through English definitions, so nuance English doesn't mark can be lost.
+- **`meaning` matches words, not ideas.** It compares your words with the dictionary's definitions (and the top translation's), so "awesome" won't find a sense defined as "Fashionable; trendy; hip." Use the definition's own wording, or let users pick from `senses()`.
+- **Grammar words translate poorly.** Words defined by their function ("marks the future tense" for Vietnamese *sẽ*) aren't reached from English *will*; *the*, *is* or *gonna* may give nothing useful.
+
+## Word frequencies
+
+Translations are ranked partly by how common each word is, using **[wordfreq](https://github.com/rspeer/wordfreq)** by Robyn Speer (Robyn Speer. (2022). rspeer/wordfreq: v3.0 (v3.0.2). Zenodo. https://doi.org/10.5281/zenodo.7199437), licensed CC BY-SA 4.0. Only a single frequency score is stored for each word already in these dictionaries (`frequency` on entries and results), never wordfreq's word lists.
+
+wordfreq's data comes from, and is credited to:
+
+- Google Books Ngrams (<http://books.google.com/ngrams>) and Google Books Syntactic Ngrams.
+- The Leeds Internet Corpus, from the University of Leeds Centre for Translation Studies (<http://corpus.leeds.ac.uk/list.html>).
+- Wikipedia, the free encyclopedia (<http://www.wikipedia.org>).
+- ParaCrawl, a multilingual Web crawl (<https://paracrawl.eu>).
+- OPUS OpenSubtitles 2018 (<http://opus.nlpl.eu/OpenSubtitles.php>), whose data originates from the OpenSubtitles project (<http://www.opensubtitles.org/>).
+- SUBTLEX word lists (SUBTLEX-US, SUBTLEX-UK, SUBTLEX-CH, SUBTLEX-DE, SUBTLEX-NL) created by **Marc Brysbaert et al.**; SUBTLEX is freely available data (<http://crr.ugent.be/programs-data/subtitle-frequencies>).
+- Word statistics gathered from the Twitter streaming API (no Twitter content is included).
+
+wordfreq measures commonness **worldwide**, so a regional word can look rare even where it's the everyday word; the region bonus is kept separate from frequency for that reason.
+
+## License and attribution
+
+- **Code** (this repo, and the `which-dialect` package): [MIT](LICENSE).
+- **Data** (the `which-dialect-<code>` packages): derived from Wiktionary via Kaikki.org's [wiktextract](https://github.com/tatuylonen/wiktextract) extraction, modified (filtered and reshaped) by this project, with word frequencies from [wordfreq](#word-frequencies), and licensed under [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/).
+
+If your app **shows** this data, credit it where users can see it, for example:
+
+> Translations from [Wiktionary](https://en.wiktionary.org/), [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/), via which-dialect.
+
+If you **redistribute modified data**, it has to stay under CC BY-SA 4.0. Using the API in your app doesn't change your app's own license. This project isn't affiliated with or endorsed by Wikimedia or Kaikki.org.
+
+## Development
+
+Requires Node 22.6+ (TypeScript scripts run with `--experimental-strip-types`).
+
+```sh
+npm install
+npm run build:data -- vi             # download Kaikki's file for a language (cached in .cache/) and build packages/<code>/data
+npm run build:data -- vi --refresh   # re-download first
+npm test                             # unit tests, plus checks against whichever languages are built
+npm run evaluate                     # translation accuracy on known-correct cases (-- --verbose to see every case)
+npm run typecheck
+npm run build                        # compile the API to packages/core/dist
+```
+
+Generated data isn't committed (it would bloat git history); it's built before publishing. Downloads: Vietnamese 79 MB, English 3.3 GB. Builds take seconds (English: about 25 s and 1.4 GB of memory).
+
+### Adding a language
+
+1. Add `languages/<code>.ts`: the Kaikki language name, the language's regions (and groups, if any), and how to read regions from Wiktionary's tags. See [`languages/vi.ts`](languages/vi.ts) (dialect areas and a pronoun table) and [`languages/en.ts`](languages/en.ts) (countries with region groups, and trimming a very large language). Spanish was built this way too (countries and region groups; `languages/es.ts` in the git history). Set `skipFormOf` for heavily inflected languages; for very large ones, see the trimming options in [`scripts/language-config.ts`](scripts/language-config.ts) (`dropTechnical`, `formOfPos`, `keepFormOf`, `maxSensesPerEntry`, `shardLength`, …).
+2. Copy a data package (`packages/vi`) to `packages/<code>` and update its `package.json` and `README.md`.
+3. Add the language's code to `translationLangs` in `languages/en.ts` (so English keeps its translation tables for it) and rebuild English.
+4. `npm run build:data -- <code>`, check the results, add a few real-data tests and evaluation cases, and publish.
+
+Languages in non-Latin scripts (Chinese, Arabic, Russian, …) will need a script-aware version of `shardKey` first: today files are split by the first two Latin letters.
+
+### Publishing
+
+```sh
+npm run build:data -- en --refresh && npm run build:data -- vi --refresh && npm test
+npm publish -w which-dialect-en
+npm publish -w which-dialect-vi
+npm publish -w which-dialect
+```
+
+The API loads data versions matching `DATA_VERSION` in `packages/core/src/index.ts` (currently `0.1`). Data-only updates can publish new 0.1.x data versions without touching the API; bump `DATA_VERSION` when the data format changes.

package/dist/index.d.ts

@@ -0,0 +1,76 @@
+import { type PronounRow, type Entry, type Hit, type LanguageMeta } from './types.ts';
+export * from './types.ts';
+export * from './pos.ts';
+/**
+ * Data packages this version of the API reads by default, as a jsDelivr version range.
+ * Bump it when the data format changes.
+ */
+export declare const DATA_VERSION = "0.1";
+/** Labels left out of search results unless `exclude` is passed. */
+export declare const DEFAULT_EXCLUDED_LABELS: string[];
+/** Loads one data file by its path inside the data folder, e.g. "meta.json" or "en/no.json". */
+export type LoadJson = (path: string) => Promise<unknown>;
+export type DictionaryOptions = {
+    /** ISO 639 code of a published data package, e.g. "vi" for which-dialect-vi. */
+    lang: string;
+    /**
+     * Where the data folder is served. Defaults to the data package on jsDelivr, so nothing has to be
+     * bundled: https://cdn.jsdelivr.net/npm/which-dialect-<lang>@<DATA_VERSION>/data
+     */
+    baseUrl?: string;
+    /** Custom loader instead of fetch, e.g. reading from disk in Node. Overrides baseUrl. */
+    load?: LoadJson;
+};
+export type SearchOptions = {
+    /**
+     * Only return words used in this region, or in any region of a group (see meta().regions and
+     * meta().regionGroups, e.g. Vietnamese "Southern" or English "North America"). Untagged words count as every region.
+     */
+    region?: string;
+    /** Only return these parts of speech (Wiktionary codes, see POS_NAMES), e.g. ['verb']. */
+    pos?: string[];
+    /** Labels to leave out. Defaults to DEFAULT_EXCLUDED_LABELS; pass [] to include everything. */
+    exclude?: string[];
+    /** Maximum number of results (default 10). */
+    limit?: number;
+    /**
+     * Return every matching sense of each word instead of one result per word (its best-ranked sense), so
+     * a caller can choose between a word's senses itself. `limit` then counts words, not results.
+     */
+    allSenses?: boolean;
+};
+export type PronounOptions = {
+    /** Only this relationship's row, by id (e.g. "parent"). Unknown ids throw, listing the valid ones. */
+    listener?: string;
+    /** Only words used in this region or region group; untagged words count as every region. */
+    region?: string;
+    /** Drop words only the other gender uses ("anh" vs "chị" toward someone younger). */
+    speaker?: 'male' | 'female';
+    /** Labels to leave out. Defaults to DEFAULT_EXCLUDED_LABELS; pass [] to include everything. */
+    exclude?: string[];
+};
+export type Dictionary = {
+    /** The language's name, regions, source, license and counts. */
+    meta(): Promise<LanguageMeta>;
+    /** Entries for a word in the language, e.g. lookup("má"). */
+    lookup(word: string): Promise<Entry[]>;
+    /**
+     * Ways to say an English word or short phrase, best first, one result per word. With a region,
+     * words the source tags for that region rank ahead of untagged ones.
+     */
+    searchEnglish(term: string, options?: SearchOptions): Promise<Hit[]>;
+    /**
+     * How to say "I", "you", "he/she", "we", plural "you" and "they" depending on who you're talking to
+     * (or about), one row per relationship, for languages whose pronouns depend on it (Vietnamese). Empty
+     * for languages without a pronoun table.
+     */
+    pronouns(options?: PronounOptions): Promise<PronounRow[]>;
+};
+/**
+ * The regions a region or region-group name stands for (e.g. English "North America" -> US, Canada),
+ * or null when no region is given. Throws for names the language doesn't have, listing the valid ones.
+ */
+export declare function resolveRegion(meta: LanguageMeta, region: string | undefined): Set<string> | null;
+export declare function dataUrl(lang: string): string;
+export declare function createDictionary(options: DictionaryOptions): Dictionary;
+export * from './translate.ts';

package/dist/index.js

@@ -0,0 +1,130 @@
+import { PRONOUN_PERSONS, fromStored, normalizeEnglish, shardKey, } from "./types.js";
+export * from "./types.js";
+export * from "./pos.js";
+/**
+ * Data packages this version of the API reads by default, as a jsDelivr version range.
+ * Bump it when the data format changes.
+ */
+export const DATA_VERSION = '0.1';
+/** Labels left out of search results unless `exclude` is passed. */
+export const DEFAULT_EXCLUDED_LABELS = [
+    'vulgar', 'offensive', 'derogatory', 'archaic', 'obsolete', 'dated', 'historical', 'rare', 'abbreviation',
+];
+// Search ranks plain, everyday words first. Labels in the first set cost a little (still normal speech,
+// just a particular register); the second set costs more (specialized, playful or uncommon).
+const MILD_LABELS = new Set(['colloquial', 'informal', 'familiar', 'polite', 'formal', 'endearing', 'honorific']);
+const LABEL_PENALTY = (h) => h.labels.reduce((sum, l) => sum + (MILD_LABELS.has(l) ? 1 : 2), 0);
+/**
+ * The regions a region or region-group name stands for (e.g. English "North America" -> US, Canada),
+ * or null when no region is given. Throws for names the language doesn't have, listing the valid ones.
+ */
+export function resolveRegion(meta, region) {
+    if (!region)
+        return null;
+    const members = meta.regions.includes(region) ? [region] : meta.regionGroups?.[region];
+    if (!members) {
+        const groups = Object.keys(meta.regionGroups ?? {});
+        throw new Error(`which-dialect: "${region}" isn't a ${meta.name} region or group ` +
+            `(regions: ${meta.regions.join(', ')}${groups.length ? `; groups: ${groups.join(', ')}` : ''})`);
+    }
+    return new Set(members);
+}
+export function dataUrl(lang) {
+    return `https://cdn.jsdelivr.net/npm/which-dialect-${lang}@${DATA_VERSION}/data`;
+}
+function fetchLoader(baseUrl) {
+    const base = baseUrl.replace(/\/+$/, '');
+    return async (path) => {
+        const res = await fetch(`${base}/${path}`);
+        if (!res.ok)
+            throw new Error(`which-dialect: couldn't load ${base}/${path} (${res.status})`);
+        return res.json();
+    };
+}
+export function createDictionary(options) {
+    const load = options.load ?? fetchLoader(options.baseUrl ?? dataUrl(options.lang));
+    // Each file is loaded at most once; a failed load is forgotten so it can be retried.
+    const cache = new Map();
+    const loadOnce = (path) => {
+        let pending = cache.get(path);
+        if (!pending) {
+            pending = load(path).catch((err) => {
+                cache.delete(path);
+                throw err;
+            });
+            cache.set(path, pending);
+        }
+        return pending;
+    };
+    const meta = () => loadOnce('meta.json');
+    // Missing shards simply mean no words start with those letters, so they aren't requested.
+    async function shard(kind, term) {
+        const m = await meta();
+        const key = shardKey(term, m.shardLength);
+        return m.shards[kind].includes(key) ? loadOnce(`${kind}/${key}.json`) : null;
+    }
+    return {
+        meta,
+        async lookup(word) {
+            const w = word.trim();
+            const [m, data] = await Promise.all([meta(), shard('words', w)]);
+            const stored = data?.[w] ?? data?.[w.toLowerCase()] ?? [];
+            return stored.map((e) => ({ ...e, senses: e.senses.map((s) => fromStored(s, m.regions)) }));
+        },
+        async searchEnglish(term, { region, pos, exclude = DEFAULT_EXCLUDED_LABELS, limit = 10, allSenses = false } = {}) {
+            const key = normalizeEnglish(term);
+            if (!key)
+                return [];
+            const m = await meta();
+            const wanted = resolveRegion(m, region);
+            const inWanted = (h) => !wanted || h.regions.some((r) => wanted.has(r));
+            const hits = ((await shard('en', key))?.[key] ?? []).map((h) => fromStored(h, m.regions));
+            const excluded = new Set(exclude);
+            const regional = (h) => Boolean(wanted && h.regionTagged);
+            // Best first: the English term is the gloss's main meaning, the word is tagged for the requested
+            // region, the word is plain rather than slang or literary, the word is common (has more senses),
+            // then the word's earlier senses.
+            const ranked = hits
+                .filter((h) => !h.labels.some((l) => excluded.has(l)))
+                .filter(inWanted)
+                .filter((h) => !pos || pos.includes(h.pos))
+                .sort((a, b) => Number(b.primary) - Number(a.primary) ||
+                Number(regional(b)) - Number(regional(a)) ||
+                LABEL_PENALTY(a) - LABEL_PENALTY(b) ||
+                // Common words first: wordfreq's frequency when the data has it, else the number of senses.
+                (b.frequency ?? 0) - (a.frequency ?? 0) ||
+                b.senses - a.senses ||
+                a.senseIndex - b.senseIndex);
+            // One result per word, its best-ranked sense (or every sense of the first `limit` words).
+            const seen = new Set();
+            const words = ranked.filter((h) => !seen.has(h.word) && seen.add(h.word)).slice(0, limit);
+            if (!allSenses)
+                return words;
+            const kept = new Set(words.map((h) => h.word));
+            return ranked.filter((h) => kept.has(h.word));
+        },
+        async pronouns({ listener, region, speaker, exclude = DEFAULT_EXCLUDED_LABELS } = {}) {
+            const m = await meta();
+            if (!m.pronouns)
+                return [];
+            const rows = await loadOnce('pronouns.json');
+            if (listener && !rows.some((r) => r.id === listener)) {
+                throw new Error(`which-dialect: unknown listener "${listener}" for ${m.name}. Use one of: ${rows.map((r) => r.id).join(', ')}`);
+            }
+            const wanted = resolveRegion(m, region);
+            const excluded = new Set(exclude);
+            const keep = (c) => (!wanted || c.regions.some((r) => wanted.has(r))) &&
+                (!speaker || !c.speaker || c.speaker === speaker) &&
+                !c.labels.some((l) => excluded.has(l));
+            return rows
+                .filter((r) => !listener || r.id === listener)
+                .map((r) => {
+                const row = { ...r };
+                for (const person of PRONOUN_PERSONS)
+                    row[person] = (r[person] ?? []).map((c) => fromStored(c, m.regions)).filter(keep);
+                return row;
+            });
+        },
+    };
+}
+export * from "./translate.js";

package/dist/pos.d.ts

@@ -0,0 +1,16 @@
+export type PosInfo = {
+    name: string;
+    description: string;
+};
+export declare const POS_NAMES: Record<string, PosInfo>;
+/** Readable name for a part-of-speech code, e.g. "adj" -> "Adjective". Unknown codes are returned as-is. */
+export declare function posName(code: string): string;
+/**
+ * Parts of speech that can translate a part of speech from any language. Languages don't line up one
+ * to one: English auxiliary verbs ("will", "can") and many adverbs are particles in Vietnamese,
+ * adjectives are verbs in languages without a separate adjective class, and a Vietnamese particle can
+ * be an adverb, conjunction or interjection elsewhere. The first entry is always the same part of speech.
+ */
+export declare const COMPATIBLE_POS: Record<string, string[]>;
+/** The parts of speech to search in another language for a part of speech (from any language). */
+export declare function compatiblePos(pos: string): string[];

package/dist/pos.js

@@ -0,0 +1,70 @@
+// Parts of speech. Entries use Wiktionary's codes (via Kaikki); these give readable names and
+// short explanations for learners, and say which codes can translate each other across languages.
+export const POS_NAMES = {
+    noun: { name: 'Noun', description: 'A person, place, thing or idea: "dog", "city", "happiness".' },
+    verb: { name: 'Verb', description: 'An action or state: "run", "think", "be".' },
+    adj: { name: 'Adjective', description: 'Describes a noun: "big", "happy", "cool".' },
+    adv: { name: 'Adverb', description: 'Describes how, when, where or how much: "quickly", "now", "very".' },
+    pron: { name: 'Pronoun', description: 'Stands in for a noun: "I", "you", "she", "they".' },
+    det: { name: 'Determiner', description: 'Points to or counts a noun: "the", "a", "this", "some".' },
+    article: { name: 'Article', description: 'A determiner like "the" or "a".' },
+    prep: { name: 'Preposition', description: 'Shows position or relation: "in", "on", "with", "for".' },
+    postp: { name: 'Postposition', description: 'Like a preposition, but placed after its noun.' },
+    conj: { name: 'Conjunction', description: 'Joins words or sentences: "and", "but", "because".' },
+    intj: { name: 'Interjection', description: 'An exclamation or reaction: "oh", "wow", "yeah".' },
+    particle: {
+        name: 'Particle',
+        description: 'A small word that adds grammar or tone rather than meaning, e.g. Vietnamese "đã" (past), "sẽ" (future), "nhé" (softener).',
+    },
+    classifier: {
+        name: 'Classifier',
+        description: 'A word used when counting or pointing at a noun, e.g. Vietnamese "con" for animals, "cái" for objects.',
+    },
+    num: { name: 'Number', description: 'A number word: "one", "twenty", "first".' },
+    contraction: { name: 'Contraction', description: 'Shortened words run together: "don\'t", "I\'m", "gonna".' },
+    phrase: { name: 'Phrase', description: 'A fixed group of words with its own meaning.' },
+    prep_phrase: { name: 'Prepositional phrase', description: 'A fixed phrase starting with a preposition: "at all", "in time".' },
+    proverb: { name: 'Proverb', description: 'A traditional saying.' },
+    prefix: { name: 'Prefix', description: 'A word part added to the front of a word: "un-", "re-".' },
+    suffix: { name: 'Suffix', description: 'A word part added to the end of a word: "-ness", "-ly".' },
+    combining_form: { name: 'Combining form', description: 'A word part only used inside other words.' },
+    name: { name: 'Proper noun', description: 'The name of a specific person, place or thing.' },
+    symbol: { name: 'Symbol', description: 'A sign or abbreviation that stands for something.' },
+    character: { name: 'Character', description: 'A single written character.' },
+};
+/** Readable name for a part-of-speech code, e.g. "adj" -> "Adjective". Unknown codes are returned as-is. */
+export function posName(code) {
+    return POS_NAMES[code]?.name ?? code;
+}
+/**
+ * Parts of speech that can translate a part of speech from any language. Languages don't line up one
+ * to one: English auxiliary verbs ("will", "can") and many adverbs are particles in Vietnamese,
+ * adjectives are verbs in languages without a separate adjective class, and a Vietnamese particle can
+ * be an adverb, conjunction or interjection elsewhere. The first entry is always the same part of speech.
+ */
+export const COMPATIBLE_POS = {
+    noun: ['noun', 'name', 'phrase'],
+    name: ['name', 'noun'],
+    verb: ['verb', 'particle', 'phrase', 'adj'],
+    adj: ['adj', 'verb', 'phrase'],
+    adv: ['adv', 'particle', 'phrase', 'prep_phrase', 'adj'],
+    pron: ['pron', 'noun', 'det'],
+    det: ['det', 'article', 'num', 'pron', 'particle', 'classifier'],
+    article: ['article', 'det', 'num', 'particle', 'classifier'],
+    prep: ['prep', 'postp', 'verb', 'adv', 'particle'],
+    postp: ['postp', 'prep', 'particle'],
+    conj: ['conj', 'particle', 'adv'],
+    intj: ['intj', 'particle', 'phrase'],
+    particle: ['particle', 'adv', 'verb', 'intj', 'conj', 'det', 'prep'],
+    classifier: ['classifier', 'noun', 'det'],
+    num: ['num', 'det'],
+    contraction: ['contraction', 'verb', 'phrase', 'pron', 'adv'],
+    phrase: ['phrase', 'verb', 'adv', 'prep_phrase', 'noun', 'intj'],
+    prep_phrase: ['prep_phrase', 'phrase', 'adv'],
+    prefix: ['prefix'],
+    suffix: ['suffix'],
+};
+/** The parts of speech to search in another language for a part of speech (from any language). */
+export function compatiblePos(pos) {
+    return COMPATIBLE_POS[pos] ?? [pos];
+}