which-dialect 0.5.6 → 0.5.8

package/README.md

@@ -2,7 +2,7 @@
 
 Translate words **between languages and between dialects**, meaning by meaning: Northern Vietnamese *ngô* into Southern *bắp*, English *cool* (as in "awesome") into Southern Vietnamese *ngầu*, English "I" into the right Vietnamese pronoun for who you're talking to, US *truck* into UK *lorry*. Dictionary data comes from [Wiktionary](https://en.wiktionary.org/) (via [Kaikki.org](https://kaikki.org/)) and loads only when you need it.
 
-**Status:** which-dialect is starting with **Vietnamese**, with English as the bridge language, and the current focus is cleaning up Vietnamese translations. **More languages are planned.** The translator and data format are language-general, so adding one is a config file plus a data package (see [Adding a language](#adding-a-language)). Spanish was supported early on and has been removed for now so the work can focus on Vietnamese.
+**Status:** which-dialect is starting with **Vietnamese**, with English as the bridge language, and the current focus is cleaning up Vietnamese translations. **More languages are planned.** The translator and data format are language-general, so adding one is a config file plus a data package (see [Adding a language](#adding-a-language)). **Spanish** is supported too, by country and region group (Spain, Mexico, Latin America, Río de la Plata, …).
 
 ```ts
 import { createTranslator } from 'which-dialect'
@@ -18,7 +18,7 @@ await tr.translate('cool', { from: 'en', to: 'vi', toRegion: 'Southern', meaning
 
 Each result is a list of **groups, one per meaning** of the source word, each with its part of speech, its definitions and its translations, so a word's meanings never get mixed up.
 
-**Data:** Wiktionary data of 2026-09-25 for both languages. See [`DATA_UPDATES.md`](DATA_UPDATES.md) for each language's dates and update history.
+**Data:** Wiktionary data of 2026-09-25 (Vietnamese: 2026-09-28). See [`DATA_UPDATES.md`](DATA_UPDATES.md) for each language's dates and update history.
 
 You install one small package. Dictionary data is **not bundled**: each lookup fetches one small file (usually 5–100 kB compressed) from the language's data package on the jsDelivr CDN, and caches it.
 
@@ -27,9 +27,10 @@ You install one small package. Dictionary data is **not bundled**: each lookup f
 | Language | Code | Data package | Regions | Entries | Region-tagged senses |
 |---|---|---|---|---|---|
 | Vietnamese | `vi` | [`which-dialect-vi`](packages/vi) | Northern, Central, Southern | 35,394 | 1,330 of 42,419 (3.1%) |
+| Spanish | `es` | [`which-dialect-es`](packages/es) | 23 countries (Spain, Mexico, Argentina, Cuba, …), plus groups: Latin America, Central America, Caribbean, South America, Río de la Plata | 123,995 | 11,940 of 153,963 (7.8%) |
 | English | `en` | [`which-dialect-en`](packages/en) | 15 countries/areas (US, UK, Australia, India, …), plus groups: North America, British Isles, Commonwealth, … | 594,544 | 37,339 of 776,085 (4.8%) |
 
-Vietnamese and English can be translated in both directions, and between any of their regions, including between two dialects of one language (Northern → Southern Vietnamese, US → UK English). English is also the bridge every language's definitions are written in.
+Any of these languages can be translated into any other, in both directions and between any of their regions, including between two dialects of one language (Northern → Southern Vietnamese, US → UK English, Spain → Mexico Spanish) and dialect to dialect across languages (Mexican Spanish → Southern Vietnamese). English is also the bridge every language's definitions are written in.
 
 More languages are planned. Each one is a config file in [`languages/`](languages) plus a data package in `packages/<code>`; see [Adding a language](#adding-a-language).
 
@@ -92,20 +93,20 @@ Meanings are ordered by `meaning` (if given), then by being tagged for `fromRegi
 
 ### Hand-picked words
 
-For common words, a speaker of the target language knows the natural word even where the data ranks another first: *get* "To fetch, bring, take" ranks Vietnamese *đưa* ("to hand, to bring") above *lấy*, because *đưa*'s definition matches more of the English words and no *lấy* definition says "fetch". Each language can list **hand-picked words for English meanings** (`picks` in `languages/<code>.ts`, built into `picks.json`). In a meaning that has picks, the picked words come first (`bridge: 'picked'`), then the ranked words; other meanings are unchanged. Picks can carry a region tag, like translation tables: with `toRegion`, picks tagged for it come first and picks tagged only for other regions are left to the ranking. A pick can also mark its meaning as the one to put first when no `pos` or `meaning` is given.
+For common words, a speaker of the target language knows the natural word even where the data ranks another first: *get* "To fetch, bring, take" ranks Vietnamese *đưa* ("to hand, to bring") above *lấy*, because *đưa*'s definition matches more of the English words and no *lấy* definition says "fetch". Each language can list **hand-picked words for English meanings** (`picks` in `languages/<code>.ts`, built into `picks.json`). In a meaning that has picks, the picked words come first (`bridge: 'picked'`), then the ranked words; other meanings are unchanged. Picks can carry a region tag, like translation tables: with `toRegion`, picks tagged for it come first and picks tagged only for other regions are left to the ranking. A pick can also mark its meaning as the one to put first when no `pos` or `meaning` is given. A pick can be a phrase the dictionary doesn't list (*chưa bao giờ*, "never"; marked `phrase: true`), and a row can `exclude` words that should never be given for its meaning (*mát mặt* for *well* "in good health").
 
-Picks are keyed by English meaning, so each language needs one list however many languages there are: translating from any other language reaches them through the English meaning (as with translation tables). They're optional: a language without picks uses the ranking alone. `picks: false` shows the ranking alone, and `npm run evaluate` reports both. Vietnamese picks so far: *get* "fetch" → *lấy*, *mang*; *get* "obtain" → *lấy*.
+Picks are keyed by English meaning, so each language needs one list however many languages there are: translating from any other language reaches them through the English meaning (as with translation tables). They're optional: a language without picks uses the ranking alone. `picks: false` shows the ranking alone, and `npm run evaluate` reports both. Vietnamese picks so far: *get* "fetch" → *lấy*, *mang*; *get* "obtain" → *lấy*; a few grammar words; and 54 meanings from a native speaker's review of the 300 most common English words (*can* "know how to" → *biết*; *how* → *như thế nào*, Southern *làm sao*; *life* "a living being" → *mạng*).
 
 ### How accurate is it?
 
-`npm run evaluate` runs known-correct translations across language pairs and dialect pairs (English → Vietnamese by region, including pronouns, Northern → Southern Vietnamese, US ↔ UK, Vietnamese → English). A case passes when a correct word is in the top 3 of the first meaning:
+`npm run evaluate` runs known-correct translations across language pairs and dialect pairs (English → Vietnamese by region, including pronouns, English → Spanish by country, Spanish ↔ Vietnamese, Northern → Southern Vietnamese, Spain ↔ Mexico, US ↔ UK, Vietnamese → English). A case passes when a correct word is in the top 3 of the first meaning:
 
 | Set | Correct word in top 3 | Correct word first |
 |---|---|---|
-| Tuning set (used while developing) | 49 / 49 (100%) | 49 / 49 (100%); 48 / 49 without picks |
-| Held-out set (written afterwards, not tuned against) | 17 / 17 (100%) | 16 / 17 (94%) |
+| Tuning set (used while developing) | 88 / 91 (97%) | 82 / 91 (90%); 81 / 91 without picks |
+| Held-out set (written afterwards, not tuned against) | 29 / 30 (97%) | 28 / 30 (93%) |
 
-The cases involving Spanish were removed with it. The held-out set has since been looked at, so new held-out cases are needed before relying on it (see [`FUTURE_IMPROVEMENTS.md`](FUTURE_IMPROVEMENTS.md)).
+With Spanish back (2026-10-29), its cases are back too; the misses are Spanish (*cerdo* without a part of speech, and Mexican "popcorn" and "cat"; see [`FUTURE_IMPROVEMENTS.md`](FUTURE_IMPROVEMENTS.md)). The held-out set has since been looked at, so new held-out cases are needed before relying on it (see [`FUTURE_IMPROVEMENTS.md`](FUTURE_IMPROVEMENTS.md)).
 
 ## Checking text
 
@@ -208,7 +209,7 @@ Each sentence has `original`, `parts`, `corrected`, `changes` (`{ from, to, why,
 
 ### `createDictionary({ lang, baseUrl?, load? })`
 
-One language's dictionary. `baseUrl` defaults to `https://cdn.jsdelivr.net/npm/which-dialect-<lang>@0.1/data`.
+One language's dictionary. `baseUrl` defaults to `https://cdn.jsdelivr.net/npm/which-dialect-<lang>@0.2/data`.
 
 - `dictionary.lookup(word)`: all entries for a word, with every sense's definitions, regions, labels, synonyms and variant links (`altOf`). The word as written, else lowercase, else capitalized, so names written in lowercase are found (*nhật* → *Nhật*, Japan; *nhật bản* → *Nhật Bản*).
 - `dictionary.searchEnglish(term, { region?, pos?, exclude?, limit?, allSenses? })`: words for an English term in this language, one per word, best first (no meaning handling: use the translator for that). `allSenses: true` returns every matching sense of those words instead.
@@ -216,7 +217,7 @@ One language's dictionary. `baseUrl` defaults to `https://cdn.jsdelivr.net/npm/w
 - `dictionary.segment(text)`: the text split into the language's words, longest dictionary match first (up to the checker's `maxWordUnits`, 3 for Vietnamese): `Hôm nay tôi ăn cơm.` → *Hôm nay* · *tôi* · *ăn cơm*, each `{ text, start, end, entries }` (offsets into the NFC text; `entries` empty for words the dictionary doesn't have). Use it to treat a word of several syllables as one, e.g. to show its meaning.
 - `dictionary.variants(word, { limit? })`: other words with the same letters but other accents that the dictionary has, most common first, each `{ word, entries, frequency? }`: *muộn* → *muốn*, *mượn*, *muôn*…; for words of several syllables every syllable varies (*hom nay* → *hôm nay*). Capitalized headwords count too, unless they're only a person's name: *nhat* → *nhất*, *Nhật* (Japan), *nhạt*…; *nhat ban* → *Nhật Bản*. For apps that let learners pick the word they meant: a real word with the wrong tone passes the spellchecker. Languages written in syllables only (`[]` otherwise).
 - `dictionary.checker()` / `dictionary.syllables()`: the language's checker settings and syllable list (with frequencies), used by `createChecker` (`{}` for languages without them).
-- `dictionary.picks()`: the language's [hand-picked words](#hand-picked-words), `{ word, pos, gloss, picks: [{ word, tags? }], first? }` per English meaning (empty for languages without any).
+- `dictionary.picks()`: the language's [hand-picked words](#hand-picked-words), `{ word, pos, gloss, picks: [{ word, tags? }], exclude?, first? }` per English meaning (`gloss` is the sense's own definition: the last one, for a sense nested under a heading) (empty for languages without any).
 - `dictionary.meta()`: the language's name, regions, region groups, source, license, build date and counts.
 - `dictionary.pronouns({ listener?, region?, speaker?, exclude? })`: how to say "I", "you", "he/she", "we", plural "you" and "they" depending on who you're talking to (or about), one row per relationship, for languages whose pronouns depend on it (so far Vietnamese; empty for others). Each row has the columns `self`, `addressee`, `third`, `selfPlural`, `addresseePlural` and `thirdPlural`. Each word says where it comes from: a dictionary definition (`source: 'gloss'`, with the `gloss`), a grammar rule for a regular compound the dictionary doesn't list (`source: 'rule'`, e.g. *các anh*, with a `note`), or a hand-written override (`source: 'override'`, with a `note`); and, when it has one, the `gender` of the person it refers to and whether a "we" is `inclusive`.
 
@@ -311,7 +312,7 @@ Generated data isn't committed (it would bloat git history); it's built before p
 
 ### Adding a language
 
-1. Add `languages/<code>.ts`: the Kaikki language name, the language's regions (and groups, if any), and how to read regions from Wiktionary's tags. See [`languages/vi.ts`](languages/vi.ts) (dialect areas and a pronoun table) and [`languages/en.ts`](languages/en.ts) (countries with region groups, and trimming a very large language). Spanish was built this way too (countries and region groups; `languages/es.ts` in the git history). Set `placeNames: {}` so its place names translate ("Japan" ↔ its name for Japan). Set `skipFormOf` for heavily inflected languages; for very large ones, see the trimming options in [`scripts/language-config.ts`](scripts/language-config.ts) (`dropTechnical`, `formOfPos`, `keepFormOf`, `maxSensesPerEntry`, `shardLength`, …).
+1. Add `languages/<code>.ts`: the Kaikki language name, the language's regions (and groups, if any), and how to read regions from Wiktionary's tags. See [`languages/vi.ts`](languages/vi.ts) (dialect areas and a pronoun table) and [`languages/en.ts`](languages/en.ts) (countries with region groups, and trimming a very large language). [`languages/es.ts`](languages/es.ts) shows countries with region groups (Latin America, Río de la Plata) and `skipFormOf`. Set `placeNames: {}` so its place names translate ("Japan" ↔ its name for Japan). Set `skipFormOf` for heavily inflected languages; for very large ones, see the trimming options in [`scripts/language-config.ts`](scripts/language-config.ts) (`dropTechnical`, `formOfPos`, `keepFormOf`, `maxSensesPerEntry`, `shardLength`, …).
 2. Copy a data package (`packages/vi`) to `packages/<code>` and update its `package.json` and `README.md`.
 3. Add the language's code to `translationLangs` in `languages/en.ts` (so English keeps its translation tables for it) and rebuild English.
 4. `npm run build:data -- <code>`, check the results, add a few real-data tests and evaluation cases, and publish.
@@ -330,4 +331,4 @@ npm publish -w which-dialect-vi
 npm publish -w which-dialect
 ```
 
-The API loads data versions matching `DATA_VERSION` in `packages/core/src/index.ts` (currently `0.1`). Data-only updates can publish new 0.1.x data versions without touching the API; bump `DATA_VERSION` when the data format changes.
+The API loads data versions matching `DATA_VERSION` in `packages/core/src/index.ts` (currently `0.2`; `0.1` up to which-dialect 0.5.6). Data-only updates can publish new 0.2.x data versions without touching the API; bump `DATA_VERSION` when the data format changes.

package/dist/index.d.ts

@@ -5,7 +5,7 @@ export * from './pos.ts';
  * Data packages this version of the API reads by default, as a jsDelivr version range.
  * Bump it when the data format changes.
  */
-export declare const DATA_VERSION = "0.1";
+export declare const DATA_VERSION = "0.2";
 /** Labels left out of search results unless `exclude` is passed. */
 export declare const DEFAULT_EXCLUDED_LABELS: string[];
 /** Loads one data file by its path inside the data folder, e.g. "meta.json" or "en/no.json". */

package/dist/index.js

@@ -6,7 +6,7 @@ export * from "./pos.js";
  * Data packages this version of the API reads by default, as a jsDelivr version range.
  * Bump it when the data format changes.
  */
-export const DATA_VERSION = '0.1';
+export const DATA_VERSION = '0.2';
 /** Labels left out of search results unless `exclude` is passed. */
 export const DEFAULT_EXCLUDED_LABELS = [
     'vulgar', 'offensive', 'derogatory', 'archaic', 'obsolete', 'dated', 'historical', 'rare', 'abbreviation',

package/dist/translate.js

@@ -445,8 +445,9 @@ export function createTranslator(options = {}) {
         return hits;
     }
     async function englishMeanings(sense, from, bridge) {
+        // A meaning's gloss is its own (last) definition: a nested sense's first is the heading it shares.
         if (from === 'en')
-            return [{ word: sense.lemma, pos: sense.pos, gloss: sense.glosses[0] ?? '', translations: sense.translations }];
+            return [{ word: sense.lemma, pos: sense.pos, gloss: sense.glosses.at(-1) ?? '', translations: sense.translations }];
         const found = [];
         for (const term of bridge.slice(0, 3)) {
             for (const entry of await dict('en').lookup(term)) {
@@ -454,8 +455,9 @@ export function createTranslator(options = {}) {
                     const own = s.translations?.[from];
                     if (!own?.some((t) => t.word === sense.lemma || t.word === sense.word))
                         continue;
-                    if (!found.some((f) => f.word === entry.word && f.gloss === s.glosses[0])) {
-                        found.push({ word: entry.word, pos: entry.pos, gloss: s.glosses[0], translations: s.translations });
+                    const gloss = s.glosses.at(-1) ?? '';
+                    if (!found.some((f) => f.word === entry.word && f.gloss === gloss)) {
+                        found.push({ word: entry.word, pos: entry.pos, gloss, translations: s.translations });
                     }
                 }
             }
@@ -522,6 +524,13 @@ export function createTranslator(options = {}) {
             ...(entry.frequency !== undefined ? { frequency: entry.frequency } : {}),
         };
     }
+    // A hand-picked phrase the dictionary doesn't list: used everywhere, or where its tags say.
+    function phraseHit(word, sense, places, meta) {
+        return {
+            word, pos: sense.pos, gloss: sense.glosses.at(-1) ?? '', regions: places ? [...places] : meta.regions,
+            regionTagged: Boolean(places), labels: [], senseIndex: 0, senses: 1, primary: true, phrase: true,
+        };
+    }
     // The source word's uses in a pronoun table, with the definition each came from.
     function pronounUses(rows, word) {
         const w = word.trim();
@@ -722,7 +731,8 @@ export function createTranslator(options = {}) {
                 // Hand-picked words for this meaning go first, in the listed order: a speaker's judgment of the
                 // natural word where the ranking puts another first. With a target region, picks tagged for it
                 // lead, and picks tagged only for other regions (or words not used there) are left to the
-                // ranking. The ranked words follow. (A pronoun-table row, below, still comes before them.)
+                // ranking. The ranked words follow. (A pronoun-table row, below, still comes before them.) A pick
+                // that isn't a headword is a phrase ("bữa tối"), given as is. Words a row excludes are dropped.
                 const rows = pickRows.length
                     ? await englishMeanings(sense, from, bridge).then((ms) => pickRows.filter((r) => ms.some((m) => m.word === r.word && m.pos === r.pos && m.gloss === r.gloss)))
                     : [];
@@ -737,13 +747,17 @@ export function createTranslator(options = {}) {
                         continue;
                     if (to === from && t.word === sense.lemma && !places?.size)
                         continue;
-                    const hit = scored.get(t.word) ?? (await tableHit(t.word, to, posList, toWanted, exclude, bridge));
+                    const hit = scored.get(t.word) ??
+                        (await tableHit(t.word, to, posList, toWanted, exclude, bridge)) ??
+                        (t.word.includes(' ') ? phraseHit(t.word, sense, places, toMeta) : null);
                     if (!hit || hit.labels.some((l) => exclude.includes(l)))
                         continue;
                     if (!places && !intersects(hit.regions, toWanted))
                         continue;
                     picked.push({ hit, tagged: Boolean(toWanted && places) });
                 }
+                for (const w of rows.flatMap((r) => r.exclude ?? []))
+                    scored.delete(w);
                 if (picked.length) {
                     const ordered = [...picked.filter((p) => p.tagged), ...picked.filter((p) => !p.tagged)];
                     const top = Math.max(0, ...[...scored.values()].map((t) => t.score));

package/dist/types.d.ts

@@ -63,6 +63,8 @@ export type Hit = {
     frequency?: number;
     /** true when the English term is the first meaning listed in the gloss (e.g. "now" in "now, today"). */
     primary: boolean;
+    /** true for a hand-picked phrase that isn't a dictionary headword ("bữa tối"); its details are the pick's. */
+    phrase?: boolean;
 };
 export type LanguageMeta = {
     /** ISO 639 code, e.g. "vi". */
@@ -213,12 +215,18 @@ export type CheckerConfig = {
  * translating between two other languages reaches it through the English meaning.
  */
 export type PickRow = {
-    /** The English headword, part of speech and first definition of the meaning, as in the English data. */
+    /**
+     * The English headword, part of speech and definition of the meaning, as in the English data. For a
+     * nested sense (a heading plus its own definition: "old", "Of a living being…") it's the sense's own,
+     * last definition, so senses under one heading are told apart.
+     */
     word: string;
     pos: string;
     gloss: string;
-    /** Best first. `tags` name regions like translation-table tags ("Southern"). */
+    /** Best first. `tags` name regions like translation-table tags ("Southern"). A pick needn't be a headword ("bữa tối"). */
     picks: TableTranslation[];
+    /** Words never given for this meaning (a speaker removed them: wrong or unnatural here). */
+    exclude?: string[];
     /** Put this meaning first when the caller gives no `pos` or `meaning` ("got" → "Have/has."). */
     first?: boolean;
 };

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "which-dialect",
-  "version": "0.5.6",
-  "description": "Translate words between languages and dialects, meaning by meaning, starting with Vietnamese (Northern, Central, Southern) and English, with Wiktionary data loaded on demand. More languages are planned.",
+  "version": "0.5.8",
+  "description": "Translate words between languages and dialects, meaning by meaning, starting with Vietnamese (Northern, Central, Southern), Spanish (by country) and English, with Wiktionary data loaded on demand. More languages are planned.",
   "type": "module",
   "main": "./dist/index.js",
   "types": "./dist/index.d.ts",
@@ -26,6 +26,7 @@
     "translation",
     "language-learning",
     "wiktionary",
+    "spanish",
     "vietnamese"
   ],
   "license": "MIT",