locale-ai 0.6.9 → 0.6.11

package/README.md

@@ -31,11 +31,11 @@ Download 577 MB? (Y/n)
 - **Layouts**: `locales/en/*.json` (nested folders too) → `locales/fr/...`, or `messages/en.json` → `messages/fr.json`. `.json`, `.yaml`, `.yml`, `.po`/`.pot` (gettext), `.arb` (Flutter) and `.strings` (Apple) are all supported, mixed freely in the same project. Common folders (`locales`, `src/locales`, `public/locales`, `src/i18n`, `messages`, `lang`, ...) are found automatically, or pass `--input`.
 - **Placeholders are never translated or lost**: `{name}`, `{{count}}`, `%s`, `%1$s`, `%(name)s`, `%{name}`, `${name}`, `:attribute`, `$t(key)`, `@:key`, `<b>…</b>`, `<0>…</0>`, `&nbsp;`, URLs, e-mails.
 - **ICU MessageFormat**: `{count, plural, one {# file} other {# files}}`, `select`, nesting. Only the text is translated.
-- **i18next plurals**: adds the forms the target language needs (`_few`, `_many` for Arabic, Russian, Polish...), so the app doesn't fall back to the English string — each added form copies the `_other` translation verbatim rather than inflecting it, so review and adjust the wording of `_zero`/`_two`/`_few`/`_many` by hand for languages where those forms use a different noun/adjective form. The `_one` form is translated with its `{{count}}` temporarily replaced by a literal "1", since the model otherwise sometimes picks the target's plural noun form despite the English source being grammatically singular.
+- **i18next plurals**: adds the forms the target language needs (`_few`, `_many` for Arabic, Russian, Polish...), so the app doesn't fall back to the English string — each added form copies the `_other` translation verbatim rather than inflecting it, so review and adjust the wording of `_zero`/`_two`/`_few`/`_many` by hand for languages where those forms use a different noun/adjective form. The `_one` form is translated with its `{{count}}` temporarily replaced by a literal "1", since the model otherwise sometimes picks the target's plural noun form despite the English source being grammatically singular. Rails-style nested plurals (`item:` with `one:`/`other:` children instead of `item_one`/`item_other`) get the same missing-forms treatment.
 - **Incremental, and edit-aware**: only missing or changed keys are translated; untouched translations are kept, so re-runs are no-ops. A small `locale-ai.lock.json` (commit it, like a package lock) remembers a hash of each source string and of the output last written for it per target locale — so editing a source string gets it re-translated automatically (it isn't enough for the target to already hold *some* text), while editing a target string by hand is detected and reported for review instead of being silently overwritten on the next run. `--overwrite` bypasses the lock and re-translates everything.
 - **Structure**: key order, numbers, booleans, arrays, indentation and trailing newline are preserved.
 
-**Format notes**: for `.po`/`.pot`, `msgid` is the stable key (never changed) and `msgstr` is the translated value, as gettext expects — the English source's `msgstr` holds the source text when it's filled in (most generators do), and falls back to `msgid` itself when it's empty, which is normal for a `.pot` template or a project that just keeps the English text in `msgid`. That fallback only ever applies to the source file: an empty `msgstr` in an *existing target* still means "not translated yet", never "same as msgid". `msgid_plural`/`msgstr[n]` plural entries are translated too (the singular and plural source forms), then expanded to however many `msgstr[n]` the target language's plural rules need (6 for Arabic, 1 for Japanese, ...) — not full grammatical inflection for in-between categories (`few`/`many`/...), which get the plural-sourced translation verbatim, the same simplification `addPluralForms` makes for i18next. `Plural-Forms` in the header is computed for the target language, not copied from the source; an existing target file's header, and any entry's own comments (a translator's note, a hand-added `#, fuzzy`), are otherwise left exactly as last edited instead of being regenerated from the source on every rerun. `--fuzzy` flags every entry a run actually (re)translated with gettext's standard `#, fuzzy` "needs review" marker, merging onto an existing `#,` line rather than adding a second one. A `.pot` source always writes a `.po` target, never a `.pot`. For `.arb`, `@key` metadata and `@@locale` are never translated; `@@locale` is set to the target locale on output. For `.strings`, a file saved as UTF-16 (Xcode's traditional default) is read like a UTF-8 one, an unquoted key (`title = "Settings";`, without quotes around `title`) is read the same as a quoted one, and `\U00e9`-style escapes are decoded — output is always written as UTF-8 with quoted keys, regardless of how the source was saved.
+**Format notes**: for `.yaml`/`.yml`, `#` comments (a translator note, an inline `# ...` after a value) survive a translation run — the target file is edited in place rather than rebuilt from scratch, so an existing comment's own value is updated without disturbing the comment, and a brand-new key is appended with no comment of its own. A newly added plural-category key (see above) is likewise appended rather than inserted between existing ones, so its position in the file won't always match the canonical `zero`/`one`/`two`/`few`/`many`/`other` order. For `.po`/`.pot`, `msgid` is the stable key (never changed) and `msgstr` is the translated value, as gettext expects — the English source's `msgstr` holds the source text when it's filled in (most generators do), and falls back to `msgid` itself when it's empty, which is normal for a `.pot` template or a project that just keeps the English text in `msgid`. That fallback only ever applies to the source file: an empty `msgstr` in an *existing target* still means "not translated yet", never "same as msgid". `msgid_plural`/`msgstr[n]` plural entries are translated too (the singular and plural source forms), then expanded to however many `msgstr[n]` the target language's plural rules need (6 for Arabic, 1 for Japanese, ...) — not full grammatical inflection for in-between categories (`few`/`many`/...), which get the plural-sourced translation verbatim, the same simplification `addPluralForms` makes for i18next. `Plural-Forms` in the header is computed for the target language, not copied from the source; an existing target file's header, and any entry's own comments (a translator's note, a hand-added `#, fuzzy`), are otherwise left exactly as last edited instead of being regenerated from the source on every rerun. `--fuzzy` flags every entry a run actually (re)translated with gettext's standard `#, fuzzy` "needs review" marker, merging onto an existing `#,` line rather than adding a second one. A `.pot` source always writes a `.po` target, never a `.pot`. For `.arb`, `@key` metadata and `@@locale` are never translated; `@@locale` is set to the target locale on output. For `.strings`, a file saved as UTF-16 (Xcode's traditional default) is read like a UTF-8 one, an unquoted key (`title = "Settings";`, without quotes around `title`) is read the same as a quoted one, and `\U00e9`-style escapes are decoded — output is always written as UTF-8 with quoted keys, regardless of how the source was saved.
 
 ## Why locale-ai?
 
@@ -105,7 +105,7 @@ then declare it in `locale-ai.config.json`:
 }
 ```
 
-- `path` is a local folder (relative to the config file); `repo` is a Hugging Face repo with the converted files (uploaded output folder), downloaded and cached like built-in models.
+- `path` is a local folder (relative to the config file); `repo` is a Hugging Face repo with the converted files (uploaded output folder), downloaded and cached like built-in models — it must look like `namespace/name`, rejected otherwise (it's joined onto the cache directory with no further checks, so anything else could resolve outside it).
 - `prefix` is required for multi-target models (`convert` lists the model's `>>xx<<` tokens).
 - Project models take priority over built-ins for their language pairs and can be used for pivoting (e.g. `ja → en → tr`).
 

package/dist/catalog.js

@@ -90,10 +90,23 @@ export const CATALOG = [
     },
 ];
 const asList = (v) => (Array.isArray(v) ? v : [v]).map((x) => parseLocale(x).lang);
+/**
+ * A Hugging Face repo id is always exactly "namespace/name". `models.ts` joins it
+ * onto the cache root with no further checks, so a `repo` of "../../../etc" (or any
+ * segment that isn't a real name) would resolve outside the cache directory — and
+ * install() writes there, uninstall() recursively deletes there.
+ */
+function validRepo(repo) {
+    const parts = repo.split("/");
+    return parts.length === 2 && parts.every((p) => p !== "" && p !== "." && p !== ".." && !p.includes("\\"));
+}
 export function customEntries(models, configDir) {
     return (models ?? []).map((c, i) => {
         if (!c.repo === !c.path)
             throw new Error(`models[${i}]: set exactly one of "repo" or "path".`);
+        if (c.repo && !validRepo(c.repo)) {
+            throw new Error(`models[${i}]: "repo" must look like "namespace/name" (a Hugging Face repo id), got "${c.repo}".`);
+        }
         if (!c.source || !c.target)
             throw new Error(`models[${i}]: "source" and "target" are required.`);
         const sources = asList(c.source);

package/dist/cli.js

@@ -10,7 +10,7 @@ import { translateStrings } from "./engine.js";
 import { CONFIG_FILE, glossaryLookup, loadConfig, protectRegex } from "./glossary.js";
 import { intlCode, parseLocale } from "./lang.js";
 import { editedPaths, isSourceStale, LOCK_FILENAME, loadLock, recordEntries, saveLock } from "./lock.js";
-import { addPluralForms, buildTarget, detectLayout, existingTargets, flattenStrings, keyPathProtect, pendingStrings, pluralOneStrings, readLocaleDoc, sourceFiles, targetFile, usesI18nextPlurals, } from "./locales.js";
+import { addNestedPluralForms, addPluralForms, buildTarget, detectLayout, existingTargets, flattenStrings, keyPathProtect, pendingStrings, pluralOneStrings, readLocaleDoc, sourceFiles, targetFile, usesI18nextPlurals, } from "./locales.js";
 import { cacheRoot, install, isInstalled, loadTranslator, localModelProblem, planInstall, readManifest, uninstall, } from "./models.js";
 import { hasTranslatableText, mask } from "./placeholders.js";
 /** Read from package.json so `npm version` is the single source of truth. */
@@ -260,7 +260,9 @@ async function cmdTranslate(argv) {
                 }
             }
             const built = buildTarget(f.doc.data, f.existing, (s) => translations.get(s) ?? s, values.overwrite, isProtectedKey, isStale);
-            const plural = usesI18nextPlurals(f.source) ? addPluralForms(built, intlCode(job.locale)) : { doc: built, added: 0 };
+            let plural = usesI18nextPlurals(f.source) ? addPluralForms(built, intlCode(job.locale)) : { doc: built, added: 0 };
+            const nested = addNestedPluralForms(plural.doc, intlCode(job.locale));
+            plural = { doc: nested.doc, added: plural.added + nested.added };
             pluralsAdded += plural.added;
             const before = fs.existsSync(f.target) ? fs.readFileSync(f.target, "utf8") : undefined;
             const text = f.doc.serialize(plural.doc, job.locale.tag, before);

package/dist/locales.js

@@ -1,6 +1,6 @@
 import fs from "node:fs";
 import path from "node:path";
-import { parse as parseYaml, stringify as stringifyYaml } from "yaml";
+import { Document, isMap, isScalar, parseDocument } from "yaml";
 import { parseLocale } from "./lang.js";
 const EXTENSIONS = [".json", ".yaml", ".yml", ".po", ".pot", ".arb", ".strings"];
 const CANDIDATES = [
@@ -130,27 +130,78 @@ function readJsonDoc(file, text) {
     const keepNewline = finalNewline(text);
     return { data, serialize: (d) => withNewline(JSON.stringify(d, null, indent), keepNewline) };
 }
+/** Renames a single-key document's top-level key, keeping that key's own comments. */
+function rewrapKey(doc, tag) {
+    const map = doc.contents;
+    if (isMap(map) && map.items.length === 1 && isScalar(map.items[0].key)) {
+        map.items[0].key.value = tag;
+    }
+    else {
+        doc.contents = doc.createNode({ [tag]: doc.toJS() ?? {} });
+    }
+}
+/**
+ * Writes every string/number/boolean leaf of `v` into `doc` at `base` (plus the
+ * leaf's own key path). `Document#setIn` mutates an existing scalar node's value in
+ * place rather than replacing it, so any comment already attached to that node (a
+ * trailing `# ...`, or one on its own line above) survives untouched; a leaf with no
+ * existing node there yet is simply inserted, with no comment of its own.
+ */
+function setLeaves(doc, base, v) {
+    if (Array.isArray(v))
+        v.forEach((item, i) => setLeaves(doc, [...base, i], item));
+    else if (isObj(v))
+        for (const [k, val] of Object.entries(v))
+            setLeaves(doc, [...base, k], val);
+    else
+        doc.setIn(base, v);
+}
 /**
  * Rails and some other frameworks nest an entire YAML locale file under a single
  * top-level key matching the locale tag itself (`en:\n  hello: Hi`), so the file is
  * valid wherever it's loaded from. Detect that wrapper and unwrap it before handing
  * `data` to the rest of the pipeline (which otherwise treats "en" as just another
  * translatable key), then re-wrap under the target's own tag on serialize.
+ *
+ * Re-serializing edits an existing YAML document (the target file's own, when it
+ * already exists, otherwise a clone of the source's) in place via `setIn` instead of
+ * building a plain object and stringifying it from scratch — the latter would throw
+ * away every `#` comment in the file on every run.
  */
 function readYamlDoc(file, text, tag) {
-    let raw;
+    let srcDoc;
     try {
-        raw = parseYaml(text) ?? {};
+        srcDoc = parseDocument(text);
     }
     catch (e) {
         throw new Error(`Invalid YAML in ${file}: ${e.message}`);
     }
+    if (srcDoc.errors.length)
+        throw new Error(`Invalid YAML in ${file}: ${srcDoc.errors[0].message}`);
+    const raw = srcDoc.toJS() ?? {};
     const indent = Number(/\n( +)\S/.exec(text)?.[1].length) || 2;
     const keepNewline = finalNewline(text);
     const keys = isObj(raw) ? Object.keys(raw) : [];
     const wrapped = keys.length === 1 && keys[0] === tag;
     const data = wrapped ? raw[tag] : raw;
-    const serialize = (d, targetTag) => withNewline(stringifyYaml(wrapped ? { [targetTag]: d } : d, { indent }), keepNewline);
+    const serialize = (d, targetTag, existingText) => {
+        let base;
+        if (existingText !== undefined) {
+            try {
+                const parsed = parseDocument(existingText);
+                if (parsed.errors.length === 0)
+                    base = parsed;
+            }
+            catch {
+                // Fall through to the source clone below.
+            }
+        }
+        base ??= srcDoc.clone();
+        if (wrapped)
+            rewrapKey(base, targetTag);
+        setLeaves(base, wrapped ? [targetTag] : [], d);
+        return withNewline(base.toString({ indent }), keepNewline);
+    };
     return { data, serialize };
 }
 /** Apple .strings: `"key" = "value";`, optionally preceded by a `/* ... *\/` comment. */
@@ -700,3 +751,48 @@ export function addPluralForms(doc, targetLang) {
     };
     return { doc: visit(doc), added };
 }
+/**
+ * Rails and some other frameworks nest plural forms under the key itself
+ * (`item:\n  one: "%{count} item"\n  other: "%{count} items"`) instead of i18next's
+ * `item_one`/`item_other` suffix convention. A value is treated as a plural group
+ * when every one of its keys is a CLDR plural category and "other" (the mandatory
+ * fallback) is among them and there's at least one more alongside it — a single
+ * `{ other: "..." }` object is too common as an ordinary translation value (e.g. a
+ * key literally named "other") to treat as a plural group on its own.
+ */
+export function addNestedPluralForms(doc, targetLang) {
+    let added = 0;
+    const visit = (v) => {
+        if (Array.isArray(v))
+            return v.map(visit);
+        if (!isObj(v))
+            return v;
+        const keys = Object.keys(v);
+        const isGroup = keys.length >= 2 && keys.includes("other") && keys.every((k) => CATEGORIES.includes(k) && typeof v[k] === "string");
+        if (!isGroup) {
+            const out = newObj();
+            for (const k of keys)
+                out[k] = visit(v[k]);
+            return out;
+        }
+        let needed;
+        try {
+            needed = new Intl.PluralRules(targetLang).resolvedOptions().pluralCategories;
+        }
+        catch {
+            needed = keys;
+        }
+        const other = v.other;
+        const out = newObj();
+        for (const cat of CATEGORIES) {
+            if (keys.includes(cat))
+                out[cat] = v[cat];
+            else if (needed.includes(cat)) {
+                out[cat] = other;
+                added++;
+            }
+        }
+        return out;
+    };
+    return { doc: visit(doc), added };
+}

package/dist/models.js

@@ -25,8 +25,26 @@ export function cacheRoot() {
     return path.join(process.env.XDG_CACHE_HOME || path.join(os.homedir(), ".cache"), "locale-ai");
 }
 const modelsRoot = () => path.join(cacheRoot(), "models");
-/** Where a model's files live: its local folder, or its cache folder. */
-export const modelDir = (m) => m.path ?? path.join(modelsRoot(), ...m.repo.split("/"));
+/**
+ * Where a model's files live: its local folder, or its cache folder. `repo` is
+ * validated as a plain "namespace/name" Hugging Face id when it comes from a
+ * project config (see catalog.ts's customEntries) — this is a second,
+ * independent check that the computed cache folder actually stays inside the
+ * cache root, since install() writes there and uninstall() recursively deletes
+ * it: a `repo` that somehow smuggled a ".." segment past that first check must
+ * never be allowed to touch a path outside the cache.
+ */
+export function modelDir(m) {
+    if (m.path)
+        return m.path;
+    const root = modelsRoot();
+    const dir = path.join(root, ...m.repo.split("/"));
+    const rel = path.relative(root, dir);
+    if (rel.startsWith("..") || path.isAbsolute(rel)) {
+        throw new Error(`${m.id}: "repo" (${m.repo}) resolves outside the model cache.`);
+    }
+    return dir;
+}
 export function readManifest(m) {
     if (m.path)
         return undefined;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "locale-ai",
-  "version": "0.6.9",
+  "version": "0.6.11",
   "description": "Translate your i18n locale files locally on CPU. Free, offline, no API keys.",
   "author": "Taha KHABOUSS",
   "repository": {