which-dialect 0.5.4 → 0.5.6

package/README.md

@@ -88,7 +88,7 @@ Every language's data defines its words in English, so English is the bridge bet
 
 **Hand-picked words** come before all of these for the meanings that have them (see below).
 
-Meanings are ordered by `meaning` (if given), then by being tagged for `fromRegion`, then the dictionary's own order, which lists main meanings first. That order is a guess when a word has several parts of speech: English *just* is listed as an adjective ("fair") before the adverb. Pass `pos` or `meaning` when you know which one you want.
+Meanings are ordered by `meaning` (if given), then by being tagged for `fromRegion`, then the dictionary's own order, which lists main meanings first. **Specialist and sexual meanings come last:** the English data labels later meanings `technical` (physics, chemistry, geometry… from Wiktionary's topics) or `sexual` (by definition, since Wiktionary seldom labels them), so "excited" lists *enthusiastic* and "to stir the emotions" before "in a state of higher energy", "having an erection" and "sexually aroused", for every target language. A word's first meaning is never labelled, so *gold* stays the metal and *circle* the shape (`senseLabels` in `languages/en.ts`). That order is a guess when a word has several parts of speech: English *just* is listed as an adjective ("fair") before the adverb. Pass `pos` or `meaning` when you know which one you want.
 
 ### Hand-picked words
 
@@ -210,7 +210,7 @@ Each sentence has `original`, `parts`, `corrected`, `changes` (`{ from, to, why,
 
 One language's dictionary. `baseUrl` defaults to `https://cdn.jsdelivr.net/npm/which-dialect-<lang>@0.1/data`.
 
-- `dictionary.lookup(word)`: all entries for a word, with every sense's definitions, regions, labels, synonyms and variant links (`altOf`).
+- `dictionary.lookup(word)`: all entries for a word, with every sense's definitions, regions, labels, synonyms and variant links (`altOf`). The word as written, else lowercase, else capitalized, so names written in lowercase are found (*nhật* → *Nhật*, Japan; *nhật bản* → *Nhật Bản*).
 - `dictionary.searchEnglish(term, { region?, pos?, exclude?, limit?, allSenses? })`: words for an English term in this language, one per word, best first (no meaning handling: use the translator for that). `allSenses: true` returns every matching sense of those words instead.
 - `dictionary.frames()`: the language's sentence frames and frame words, unfilled (`{ frames, words }`; empty without frames).
 - `dictionary.segment(text)`: the text split into the language's words, longest dictionary match first (up to the checker's `maxWordUnits`, 3 for Vietnamese): `Hôm nay tôi ăn cơm.` → *Hôm nay* · *tôi* · *ăn cơm*, each `{ text, start, end, entries }` (offsets into the NFC text; `entries` empty for words the dictionary doesn't have). Use it to treat a word of several syllables as one, e.g. to show its meaning.
@@ -301,6 +301,8 @@ npm run evaluate                     # translation accuracy on known-correct cas
 npm run build:picks -- vi            # rebuild only a language's hand-picked words (after editing `picks` in languages/vi.ts)
 npm run picks-sheet -- vi            # review sheet (.cache/picks-vi.csv) of the top English words' meanings; -- --words file.txt, -- --top 500
 npm run picks-sheet -- vi --apply sheet.csv   # config rows for the rows whose your_pick is filled in
+npm run picks-sheet -- vi --top 300 --drafts reviews/vi-top300.json --html page.html   # the same rows as a review page
+npm run picks-sheet -- vi --apply-decisions <dir> --drafts reviews/vi-top300.json     # config rows from the page's decisions
 npm run typecheck
 npm run build                        # compile the API to packages/core/dist
 ```
@@ -315,7 +317,7 @@ Generated data isn't committed (it would bloat git history); it's built before p
 4. `npm run build:data -- <code>`, check the results, add a few real-data tests and evaluation cases, and publish.
 5. Optional: [checker](#checking-text) settings (`checker` in `languages/<code>.ts`): `units: 'syllables'` for languages written in syllables (also builds the syllable list for accent suggestions), `maxWordUnits`, the reliable `pronouns` per pronoun-table column, `ambiguousPronouns`, and `politeEndings`, `similarLetters` (groups of letters often confused, for typo suggestions). Without them, the language is read as space-separated words and gets the region check.
 6. Optional: [sentence frames](#sentence-frames): `frames` (how the language says each frame in `languages/frames.ts`) and `frameWords` (the words that change by region, each from a definition or an override with a note). Frames reach other languages through the shared catalog; a new frame goes in the catalog once.
-7. Optional, later: [hand-picked words](#hand-picked-words). A speaker runs `npm run picks-sheet -- <code>`, fills in `your_pick` only where the first choice is wrong, and pastes the `--apply` output into `picks`.
+7. Optional, later: [hand-picked words](#hand-picked-words). A speaker runs `npm run picks-sheet -- <code>`, fills in `your_pick` only where the first choice is wrong, and pastes the `--apply` output into `picks`. Or as a **review page**: draft suggestions for the meanings whose first word is wrong in `reviews/<code>-top300.json` (picks, confidence, why, an example sentence), build the page with `--html`, publish it as a claude.ai Artifact (it saves each decision in its database), and once a speaker has gone through it, export the decisions and turn them into config rows with `--apply-decisions`. In Claude Code, the project skill `/word-review <language> <N>` does all of this (and `/word-review <language> <N> apply` afterwards).
 
 Languages in non-Latin scripts (Chinese, Arabic, Russian, …) will need a script-aware version of `shardKey` first: today files are split by the first two Latin letters.
 

package/dist/check.js

@@ -172,8 +172,12 @@ async function spelling(ctx) {
                     max: COMPOUND_MAX_COMBINATIONS,
                 }).filter((c) => c !== written.join(' '));
                 const found = (await Promise.all(combos.map(async (c) => ({ word: c, entries: await ctx.dict.lookup(c) }))))
+                    // Names (found through a capitalized headword) only for a misspelled run: "nhat ban" → "Nhật Bản",
+                    // but "hoa lan" (orchid, every syllable a word) isn't "Hòa Lan" (the Netherlands).
+                    .map((f) => ({ ...f, entries: doubtful ? f.entries : f.entries.filter((e) => e.word === e.word.toLowerCase()) }))
                     .filter((f) => f.entries.length)
-                    .map((f) => ({ word: f.word, frequency: Math.max(-1, ...f.entries.map((e) => e.frequency ?? -1)) }))
+                    // Spelled as the headword: names keep their capitals ("nhat ban" → "Nhật Bản").
+                    .map((f) => ({ word: f.entries[0].word.toLowerCase() === f.word ? f.entries[0].word : f.word, frequency: Math.max(-1, ...f.entries.map((e) => e.frequency ?? -1)) }))
                     .sort((a, b) => b.frequency - a.frequency)
                     .slice(0, 3);
                 if (!found.length)

package/dist/index.js

@@ -72,7 +72,10 @@ export function createDictionary(options) {
     async function lookup(word) {
         const w = word.trim();
         const [m, data] = await Promise.all([meta(), shard('words', w)]);
-        const stored = data?.[w] ?? data?.[w.toLowerCase()] ?? [];
+        // The word as written, else lowercase ("Mẹ" → "mẹ"), else capitalized for names written in lowercase
+        // ("nhật" → "Nhật", Japan; "nhật bản" → "Nhật Bản"), which spellchecker suggestions and learners do.
+        const titled = w.toLowerCase().replace(/(^|[\s-])(\p{Ll})/gu, (_, sep, c) => sep + c.toUpperCase());
+        const stored = data?.[w] ?? data?.[w.toLowerCase()] ?? data?.[titled] ?? [];
         return stored.map((e) => ({ ...e, senses: e.senses.map((s) => fromStored(s, m.regions)) }));
     }
     const checker = async () => {
@@ -107,11 +110,15 @@ export function createDictionary(options) {
             // Each form as written in lowercase and capitalized, for names ("nhat" → "Nhật", Japan; "nhat ban" →
             // "Nhật Bản").
             const forms = [...new Set(combos.flatMap((c) => [c, c.replace(/(^|\s)(\S)/gu, (_, sep, ch) => sep + ch.toUpperCase())]))];
-            // A capitalized form counts when it's its own headword (lookup ignores case) and more than a person's
-            // name (a place, a people), so given names don't crowd out words.
-            const found = (await Promise.all(forms.map(async (c) => ({ word: c, entries: await lookup(c) })))).map((v) => v.word === v.word.toLowerCase()
-                ? v
-                : { ...v, entries: v.entries.filter((e) => e.word === v.word && e.senses.some((s) => !PERSONAL_NAME.test(s.glosses[0] ?? ''))) });
+            // Each form counts only as its own headword (lookup also finds other cases: "nhật" finds "Nhật"), and a
+            // capitalized one only when it's more than a person's name (a place, a people), so given names don't
+            // crowd out words.
+            const found = (await Promise.all(forms.map(async (c) => ({ word: c, entries: await lookup(c) })))).map((v) => ({
+                ...v,
+                entries: v.entries.filter((e) => v.word === v.word.toLowerCase()
+                    ? !(e.word !== v.word && e.word.toLowerCase() === v.word)
+                    : e.word === v.word && e.senses.some((s) => !PERSONAL_NAME.test(s.glosses[0] ?? ''))),
+            }));
             const frequency = (v) => {
                 const known = v.entries.map((e) => e.frequency).filter((f) => f !== undefined);
                 return known.length ? Math.max(...known) : parts.length === 1 ? list[v.word.toLowerCase()] || undefined : undefined;

package/dist/translate.js

@@ -130,6 +130,10 @@ const INFLECTION_VIA = /\b(?:past|participle|plural|singular|present|third-perso
 // Everyday speech, not slang: a form's own colloquial meaning ("got": "Have/has.") isn't marginal.
 const EVERYDAY_LABELS = new Set(['informal', 'colloquial']);
 const LABEL_PENALTY = new Set(['slang', 'Internet', 'humorous', 'literary', 'dialectal', 'uncommon', 'rare', 'euphemistic']);
+// Senses with any of these labels (added by the data build, LanguageConfig.senseLabels) rank after
+// everyday and marginal ones: specialist ("excited": "in a state of higher energy", physics) and sexual
+// meanings, which a learner rarely means and shouldn't see first.
+const LATER_LABELS = new Set(['technical', 'sexual']);
 // A target gloss that lists the bridge term itself as an equivalent ("muốn": "to want"; "má": "mother;
 // mom") is a plain translation, but a short gloss can't earn definition overlap, since the term's own
 // words don't count. Wordier glosses did ("mắc": "to want to (go to the bathroom, laugh, etc.); to have
@@ -859,13 +863,15 @@ export function createTranslator(options = {}) {
                 }
                 return marginal ? -1 : 0;
             };
+            const later = (g) => (g.source.labels.some((l) => LATER_LABELS.has(l)) ? 2 : 0);
             const relevance = (g) => (meaningWords
                 ? 10 * meaningOverlap(meaningWords, g.source.glosses.join(' ')) + 5 * meaningOverlap(meaningWords, g.translations[0]?.gloss ?? '')
                 : 0) -
                 (ownSenses && g.source.via && SPELLING_VIA.test(g.source.via) ? 2 : 0) +
                 (fromRegion && g.source.regionTagged ? 3 : 0) +
                 pronounRelevance(g) +
-                labelRelevance(g) +
+                labelRelevance(g) -
+                later(g) +
                 (!meaning && pickedFirst.has(g.source) ? 5 : 0);
             // A sense with no translations says nothing, so it goes after the ones that do ("got": "Expressing
             // obligation; used with have." finds nothing; "Must; have/has (to)." gives "phải").

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "which-dialect",
-  "version": "0.5.4",
+  "version": "0.5.6",
   "description": "Translate words between languages and dialects, meaning by meaning, starting with Vietnamese (Northern, Central, Southern) and English, with Wiktionary data loaded on demand. More languages are planned.",
   "type": "module",
   "main": "./dist/index.js",