locale-ai 0.6.10 → 0.6.12

package/README.md

@@ -31,11 +31,11 @@ Download 577 MB? (Y/n)
 - **Layouts**: `locales/en/*.json` (nested folders too) → `locales/fr/...`, or `messages/en.json` → `messages/fr.json`. `.json`, `.yaml`, `.yml`, `.po`/`.pot` (gettext), `.arb` (Flutter) and `.strings` (Apple) are all supported, mixed freely in the same project. Common folders (`locales`, `src/locales`, `public/locales`, `src/i18n`, `messages`, `lang`, ...) are found automatically, or pass `--input`.
 - **Placeholders are never translated or lost**: `{name}`, `{{count}}`, `%s`, `%1$s`, `%(name)s`, `%{name}`, `${name}`, `:attribute`, `$t(key)`, `@:key`, `<b>…</b>`, `<0>…</0>`, `&nbsp;`, URLs, e-mails.
 - **ICU MessageFormat**: `{count, plural, one {# file} other {# files}}`, `select`, nesting. Only the text is translated.
-- **i18next plurals**: adds the forms the target language needs (`_few`, `_many` for Arabic, Russian, Polish...), so the app doesn't fall back to the English string — each added form copies the `_other` translation verbatim rather than inflecting it, so review and adjust the wording of `_zero`/`_two`/`_few`/`_many` by hand for languages where those forms use a different noun/adjective form. The `_one` form is translated with its `{{count}}` temporarily replaced by a literal "1", since the model otherwise sometimes picks the target's plural noun form despite the English source being grammatically singular.
+- **i18next plurals**: adds the forms the target language needs (`_few`, `_many` for Arabic, Russian, Polish...), so the app doesn't fall back to the English string — each added form copies the `_other` translation verbatim rather than inflecting it, so review and adjust the wording of `_zero`/`_two`/`_few`/`_many` by hand for languages where those forms use a different noun/adjective form. The `_one` form is translated with its `{{count}}` temporarily replaced by a literal "1", since the model otherwise sometimes picks the target's plural noun form despite the English source being grammatically singular. Rails-style nested plurals (`item:` with `one:`/`other:` children instead of `item_one`/`item_other`) get the same missing-forms treatment.
 - **Incremental, and edit-aware**: only missing or changed keys are translated; untouched translations are kept, so re-runs are no-ops. A small `locale-ai.lock.json` (commit it, like a package lock) remembers a hash of each source string and of the output last written for it per target locale — so editing a source string gets it re-translated automatically (it isn't enough for the target to already hold *some* text), while editing a target string by hand is detected and reported for review instead of being silently overwritten on the next run. `--overwrite` bypasses the lock and re-translates everything.
 - **Structure**: key order, numbers, booleans, arrays, indentation and trailing newline are preserved.
 
-**Format notes**: for `.po`/`.pot`, `msgid` is the stable key (never changed) and `msgstr` is the translated value, as gettext expects — the English source's `msgstr` holds the source text when it's filled in (most generators do), and falls back to `msgid` itself when it's empty, which is normal for a `.pot` template or a project that just keeps the English text in `msgid`. That fallback only ever applies to the source file: an empty `msgstr` in an *existing target* still means "not translated yet", never "same as msgid". `msgid_plural`/`msgstr[n]` plural entries are translated too (the singular and plural source forms), then expanded to however many `msgstr[n]` the target language's plural rules need (6 for Arabic, 1 for Japanese, ...) — not full grammatical inflection for in-between categories (`few`/`many`/...), which get the plural-sourced translation verbatim, the same simplification `addPluralForms` makes for i18next. `Plural-Forms` in the header is computed for the target language, not copied from the source; an existing target file's header, and any entry's own comments (a translator's note, a hand-added `#, fuzzy`), are otherwise left exactly as last edited instead of being regenerated from the source on every rerun. `--fuzzy` flags every entry a run actually (re)translated with gettext's standard `#, fuzzy` "needs review" marker, merging onto an existing `#,` line rather than adding a second one. A `.pot` source always writes a `.po` target, never a `.pot`. For `.arb`, `@key` metadata and `@@locale` are never translated; `@@locale` is set to the target locale on output. For `.strings`, a file saved as UTF-16 (Xcode's traditional default) is read like a UTF-8 one, an unquoted key (`title = "Settings";`, without quotes around `title`) is read the same as a quoted one, and `\U00e9`-style escapes are decoded — output is always written as UTF-8 with quoted keys, regardless of how the source was saved.
+**Format notes**: for `.yaml`/`.yml`, `#` comments (a translator note, an inline `# ...` after a value) survive a translation run — the target file is edited in place rather than rebuilt from scratch, so an existing comment's own value is updated without disturbing the comment, and a brand-new key is appended with no comment of its own. A newly added plural-category key (see above) is likewise appended rather than inserted between existing ones, so its position in the file won't always match the canonical `zero`/`one`/`two`/`few`/`many`/`other` order. For `.po`/`.pot`, `msgid` is the stable key (never changed) and `msgstr` is the translated value, as gettext expects — the English source's `msgstr` holds the source text when it's filled in (most generators do), and falls back to `msgid` itself when it's empty, which is normal for a `.pot` template or a project that just keeps the English text in `msgid`. That fallback only ever applies to the source file: an empty `msgstr` in an *existing target* still means "not translated yet", never "same as msgid". `msgid_plural`/`msgstr[n]` plural entries are translated too (the singular and plural source forms), then expanded to however many `msgstr[n]` the target language's plural rules need (6 for Arabic, 1 for Japanese, ...) — not full grammatical inflection for in-between categories (`few`/`many`/...), which get the plural-sourced translation verbatim, the same simplification `addPluralForms` makes for i18next. `Plural-Forms` in the header is computed for the target language, not copied from the source; an existing target file's header, and any entry's own comments (a translator's note, a hand-added `#, fuzzy`), are otherwise left exactly as last edited instead of being regenerated from the source on every rerun. `--fuzzy` flags every entry a run actually (re)translated with gettext's standard `#, fuzzy` "needs review" marker, merging onto an existing `#,` line rather than adding a second one. A `.pot` source always writes a `.po` target, never a `.pot`. For `.arb`, `@key` metadata and `@@locale` are never translated; `@@locale` is set to the target locale on output. For `.strings`, a file saved as UTF-16 (Xcode's traditional default) is read like a UTF-8 one, an unquoted key (`title = "Settings";`, without quotes around `title`) is read the same as a quoted one, and `\U00e9`-style escapes are decoded — output is always written as UTF-8 with quoted keys, regardless of how the source was saved.
 
 ## Why locale-ai?
 
@@ -49,7 +49,7 @@ Download 577 MB? (Y/n)
 ## Commands
 
 ```bash
-locale-ai translate --to fr,pt-BR [--from en] [--input dir] [--dry-run] [--overwrite] [--fast] [--report file] [-y]
+locale-ai translate --to fr,pt-BR [--from en] [--input dir] [--dry-run] [--overwrite] [--fast] [--beams n] [--batch-size n] [--report file] [-y]
 locale-ai check [--from en] [--input dir]   # CI: fails if a translated locale has missing keys or placeholder mismatches
                                              # (also warns, without failing, about strings left identical to the source)
 locale-ai models list [--from en]     # supported language pairs
@@ -59,7 +59,7 @@ locale-ai models remove en-de
 locale-ai models convert <hf-marian-model> --out <dir>   # bring your own model (see below)
 ```
 
-`translate` uses beam search (4 beams) by default. `--fast` uses greedy decoding: 2–3x faster, about 0.5 chrF lower.
+`translate` uses beam search (4 beams) by default; `--beams n` overrides the width directly. `--fast` uses greedy decoding (equivalent to `--beams 1`): 2–3x faster, about 0.5 chrF lower. `--batch-size n` (default 32) controls how many strings go through the model per batch — lower it if you're hitting memory limits, raise it for a bit more throughput on a machine with room to spare.
 
 Models are cached once per machine in `~/.cache/locale-ai` (`LOCALE_AI_CACHE` to change). In CI, pass `--yes`.
 

package/dist/cli.js

@@ -10,7 +10,7 @@ import { translateStrings } from "./engine.js";
 import { CONFIG_FILE, glossaryLookup, loadConfig, protectRegex } from "./glossary.js";
 import { intlCode, parseLocale } from "./lang.js";
 import { editedPaths, isSourceStale, LOCK_FILENAME, loadLock, recordEntries, saveLock } from "./lock.js";
-import { addPluralForms, buildTarget, detectLayout, existingTargets, flattenStrings, keyPathProtect, pendingStrings, pluralOneStrings, readLocaleDoc, sourceFiles, targetFile, usesI18nextPlurals, } from "./locales.js";
+import { addNestedPluralForms, addPluralForms, buildTarget, detectLayout, existingTargets, flattenStrings, keyPathProtect, pendingStrings, pluralOneStrings, readLocaleDoc, sourceFiles, targetFile, usesI18nextPlurals, } from "./locales.js";
 import { cacheRoot, install, isInstalled, loadTranslator, localModelProblem, planInstall, readManifest, uninstall, } from "./models.js";
 import { hasTranslatableText, mask } from "./placeholders.js";
 /** Read from package.json so `npm version` is the single source of truth. */
@@ -37,6 +37,8 @@ Translate options:
   --overwrite           Re-translate strings that already have a translation
   --dry-run             Show what would be translated without downloading or writing
   --fast                Greedy decoding: ~2x faster, slightly lower quality
+  --beams <n>           Beam width for decoding (default: 4, or 1 with --fast)
+  --batch-size <n>      Strings translated per model batch (default: 32)
   --report <file>       Write a JSON report of strings that need review
   --fuzzy               For .po/.pot: flag every entry this run (re)translated as "#, fuzzy"
   --config <file>       Project config (default: ./${CONFIG_FILE} if present)
@@ -260,7 +262,9 @@ async function cmdTranslate(argv) {
                 }
             }
             const built = buildTarget(f.doc.data, f.existing, (s) => translations.get(s) ?? s, values.overwrite, isProtectedKey, isStale);
-            const plural = usesI18nextPlurals(f.source) ? addPluralForms(built, intlCode(job.locale)) : { doc: built, added: 0 };
+            let plural = usesI18nextPlurals(f.source) ? addPluralForms(built, intlCode(job.locale)) : { doc: built, added: 0 };
+            const nested = addNestedPluralForms(plural.doc, intlCode(job.locale));
+            plural = { doc: nested.doc, added: plural.added + nested.added };
             pluralsAdded += plural.added;
             const before = fs.existsSync(f.target) ? fs.readFileSync(f.target, "utf8") : undefined;
             const text = f.doc.serialize(plural.doc, job.locale.tag, before);

package/dist/locales.js

@@ -1,6 +1,6 @@
 import fs from "node:fs";
 import path from "node:path";
-import { parse as parseYaml, stringify as stringifyYaml } from "yaml";
+import { Document, isMap, isScalar, parseDocument } from "yaml";
 import { parseLocale } from "./lang.js";
 const EXTENSIONS = [".json", ".yaml", ".yml", ".po", ".pot", ".arb", ".strings"];
 const CANDIDATES = [
@@ -130,27 +130,78 @@ function readJsonDoc(file, text) {
     const keepNewline = finalNewline(text);
     return { data, serialize: (d) => withNewline(JSON.stringify(d, null, indent), keepNewline) };
 }
+/** Renames a single-key document's top-level key, keeping that key's own comments. */
+function rewrapKey(doc, tag) {
+    const map = doc.contents;
+    if (isMap(map) && map.items.length === 1 && isScalar(map.items[0].key)) {
+        map.items[0].key.value = tag;
+    }
+    else {
+        doc.contents = doc.createNode({ [tag]: doc.toJS() ?? {} });
+    }
+}
+/**
+ * Writes every string/number/boolean leaf of `v` into `doc` at `base` (plus the
+ * leaf's own key path). `Document#setIn` mutates an existing scalar node's value in
+ * place rather than replacing it, so any comment already attached to that node (a
+ * trailing `# ...`, or one on its own line above) survives untouched; a leaf with no
+ * existing node there yet is simply inserted, with no comment of its own.
+ */
+function setLeaves(doc, base, v) {
+    if (Array.isArray(v))
+        v.forEach((item, i) => setLeaves(doc, [...base, i], item));
+    else if (isObj(v))
+        for (const [k, val] of Object.entries(v))
+            setLeaves(doc, [...base, k], val);
+    else
+        doc.setIn(base, v);
+}
 /**
  * Rails and some other frameworks nest an entire YAML locale file under a single
  * top-level key matching the locale tag itself (`en:\n  hello: Hi`), so the file is
  * valid wherever it's loaded from. Detect that wrapper and unwrap it before handing
  * `data` to the rest of the pipeline (which otherwise treats "en" as just another
  * translatable key), then re-wrap under the target's own tag on serialize.
+ *
+ * Re-serializing edits an existing YAML document (the target file's own, when it
+ * already exists, otherwise a clone of the source's) in place via `setIn` instead of
+ * building a plain object and stringifying it from scratch — the latter would throw
+ * away every `#` comment in the file on every run.
  */
 function readYamlDoc(file, text, tag) {
-    let raw;
+    let srcDoc;
     try {
-        raw = parseYaml(text) ?? {};
+        srcDoc = parseDocument(text);
     }
     catch (e) {
         throw new Error(`Invalid YAML in ${file}: ${e.message}`);
     }
+    if (srcDoc.errors.length)
+        throw new Error(`Invalid YAML in ${file}: ${srcDoc.errors[0].message}`);
+    const raw = srcDoc.toJS() ?? {};
     const indent = Number(/\n( +)\S/.exec(text)?.[1].length) || 2;
     const keepNewline = finalNewline(text);
     const keys = isObj(raw) ? Object.keys(raw) : [];
     const wrapped = keys.length === 1 && keys[0] === tag;
     const data = wrapped ? raw[tag] : raw;
-    const serialize = (d, targetTag) => withNewline(stringifyYaml(wrapped ? { [targetTag]: d } : d, { indent }), keepNewline);
+    const serialize = (d, targetTag, existingText) => {
+        let base;
+        if (existingText !== undefined) {
+            try {
+                const parsed = parseDocument(existingText);
+                if (parsed.errors.length === 0)
+                    base = parsed;
+            }
+            catch {
+                // Fall through to the source clone below.
+            }
+        }
+        base ??= srcDoc.clone();
+        if (wrapped)
+            rewrapKey(base, targetTag);
+        setLeaves(base, wrapped ? [targetTag] : [], d);
+        return withNewline(base.toString({ indent }), keepNewline);
+    };
     return { data, serialize };
 }
 /** Apple .strings: `"key" = "value";`, optionally preceded by a `/* ... *\/` comment. */
@@ -700,3 +751,48 @@ export function addPluralForms(doc, targetLang) {
     };
     return { doc: visit(doc), added };
 }
+/**
+ * Rails and some other frameworks nest plural forms under the key itself
+ * (`item:\n  one: "%{count} item"\n  other: "%{count} items"`) instead of i18next's
+ * `item_one`/`item_other` suffix convention. A value is treated as a plural group
+ * when every one of its keys is a CLDR plural category and "other" (the mandatory
+ * fallback) is among them and there's at least one more alongside it — a single
+ * `{ other: "..." }` object is too common as an ordinary translation value (e.g. a
+ * key literally named "other") to treat as a plural group on its own.
+ */
+export function addNestedPluralForms(doc, targetLang) {
+    let added = 0;
+    const visit = (v) => {
+        if (Array.isArray(v))
+            return v.map(visit);
+        if (!isObj(v))
+            return v;
+        const keys = Object.keys(v);
+        const isGroup = keys.length >= 2 && keys.includes("other") && keys.every((k) => CATEGORIES.includes(k) && typeof v[k] === "string");
+        if (!isGroup) {
+            const out = newObj();
+            for (const k of keys)
+                out[k] = visit(v[k]);
+            return out;
+        }
+        let needed;
+        try {
+            needed = new Intl.PluralRules(targetLang).resolvedOptions().pluralCategories;
+        }
+        catch {
+            needed = keys;
+        }
+        const other = v.other;
+        const out = newObj();
+        for (const cat of CATEGORIES) {
+            if (keys.includes(cat))
+                out[cat] = v[cat];
+            else if (needed.includes(cat)) {
+                out[cat] = other;
+                added++;
+            }
+        }
+        return out;
+    };
+    return { doc: visit(doc), added };
+}

package/dist/placeholders.js

@@ -24,7 +24,7 @@ const SOURCES = [
     /\$\{[^{}]*\}/.source, // ${name}
     /%\{[^{}]+\}/.source, // %{name}  ruby / polyglot
     /%\([\w.]+\)[sdifr]/.source, // %(name)s  python
-    /%(?:\d+\$)?[-+0#]*\d*(?:\.\d+)?[sdifuxXeEgGcpo@]/.source, // %s %d %1$s %.2f %@
+    /%(?:\d+\$)?[-+0#]*\d*(?:\.\d+)?[sdifuxXeEgGcpo@](?![A-Za-z])/.source, // %s %d %1$s %.2f %@ (not "100%off")
     /@(?:\.\w+)?:\(?[\w.-]+\)?/.source, // @:key @.lower:key  vue-i18n linked messages
     /\{\s*[\w.-]+\s*(?:,[^{}]*)?\}/.source, // {name} {0} {count, number}  ICU simple args
     /<\/?[A-Za-z][\w.:-]*(?:\s[^<>]*)?\/?>/.source, // <b> </b> <a href="..."> <Trans/>
@@ -33,7 +33,7 @@ const SOURCES = [
     /https?:\/\/[^\s<>"']*[^\s<>"'.,;:!?)\]]/.source, // URLs
     /[\w.+-]+@[\w-]+(?:\.[\w-]+)+/.source, // e-mail addresses
     /(?<=^|[\s(])(?::[a-zA-Z_]\w*)/.source, // :attribute  laravel
-    /\$[A-Za-z_]\w*/.source, // $name
+    /\$(?![A-Z]{2,}(?![a-zA-Z0-9_]))[A-Za-z_]\w*/.source, // $name (not "$USD", "$CAD" — currency codes)
 ];
 const PLACEHOLDER_SRC = SOURCES.map((s) => `(?:${s})`).join("|");
 const PLACEHOLDER_RE = new RegExp(PLACEHOLDER_SRC, "gu");

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "locale-ai",
-  "version": "0.6.10",
+  "version": "0.6.12",
   "description": "Translate your i18n locale files locally on CPU. Free, offline, no API keys.",
   "author": "Taha KHABOUSS",
   "repository": {