locale-ai 0.6.8 → 0.6.10

package/README.md

@@ -28,14 +28,14 @@ Download 577 MB? (Y/n)
 
 ## What it handles
 
-- **Layouts**: `locales/en/*.json` (nested folders too) → `locales/fr/...`, or `messages/en.json` → `messages/fr.json`. `.json`, `.yaml`, `.yml`, `.po` (gettext), `.arb` (Flutter) and `.strings` (Apple) are all supported, mixed freely in the same project. Common folders (`locales`, `src/locales`, `public/locales`, `src/i18n`, `messages`, `lang`, ...) are found automatically, or pass `--input`.
+- **Layouts**: `locales/en/*.json` (nested folders too) → `locales/fr/...`, or `messages/en.json` → `messages/fr.json`. `.json`, `.yaml`, `.yml`, `.po`/`.pot` (gettext), `.arb` (Flutter) and `.strings` (Apple) are all supported, mixed freely in the same project. Common folders (`locales`, `src/locales`, `public/locales`, `src/i18n`, `messages`, `lang`, ...) are found automatically, or pass `--input`.
 - **Placeholders are never translated or lost**: `{name}`, `{{count}}`, `%s`, `%1$s`, `%(name)s`, `%{name}`, `${name}`, `:attribute`, `$t(key)`, `@:key`, `<b>…</b>`, `<0>…</0>`, `&nbsp;`, URLs, e-mails.
 - **ICU MessageFormat**: `{count, plural, one {# file} other {# files}}`, `select`, nesting. Only the text is translated.
 - **i18next plurals**: adds the forms the target language needs (`_few`, `_many` for Arabic, Russian, Polish...), so the app doesn't fall back to the English string — each added form copies the `_other` translation verbatim rather than inflecting it, so review and adjust the wording of `_zero`/`_two`/`_few`/`_many` by hand for languages where those forms use a different noun/adjective form. The `_one` form is translated with its `{{count}}` temporarily replaced by a literal "1", since the model otherwise sometimes picks the target's plural noun form despite the English source being grammatically singular.
 - **Incremental, and edit-aware**: only missing or changed keys are translated; untouched translations are kept, so re-runs are no-ops. A small `locale-ai.lock.json` (commit it, like a package lock) remembers a hash of each source string and of the output last written for it per target locale — so editing a source string gets it re-translated automatically (it isn't enough for the target to already hold *some* text), while editing a target string by hand is detected and reported for review instead of being silently overwritten on the next run. `--overwrite` bypasses the lock and re-translates everything.
 - **Structure**: key order, numbers, booleans, arrays, indentation and trailing newline are preserved.
 
-**Format notes**: for `.po`, `msgid` is the stable key (never changed) and `msgstr` is the translated value, as gettext expects — the English `.po` file's `msgstr` must hold the source text (most generators already fill it; an empty `msgstr` is treated as empty, not inherited from `msgid`). `msgid_plural`/`msgstr[n]` plural entries are translated too (the singular and plural source forms), then expanded to however many `msgstr[n]` the target language's plural rules need (6 for Arabic, 1 for Japanese, ...) — not full grammatical inflection for in-between categories (`few`/`many`/...), which get the plural-sourced translation verbatim, the same simplification `addPluralForms` makes for i18next. `Plural-Forms` in the header is computed for the target language, not copied from the source; an existing target file's header is otherwise left as you last edited it (a hand-fixed `Plural-Forms`, `Last-Translator`, ...) instead of being regenerated on every rerun. For `.arb`, `@key` metadata and `@@locale` are never translated; `@@locale` is set to the target locale on output. For `.strings`, a file saved as UTF-16 (Xcode's traditional default) is read like a UTF-8 one, an unquoted key (`title = "Settings";`, without quotes around `title`) is read the same as a quoted one, and `\U00e9`-style escapes are decoded — output is always written as UTF-8 with quoted keys, regardless of how the source was saved.
+**Format notes**: for `.po`/`.pot`, `msgid` is the stable key (never changed) and `msgstr` is the translated value, as gettext expects — the English source's `msgstr` holds the source text when it's filled in (most generators do), and falls back to `msgid` itself when it's empty, which is normal for a `.pot` template or a project that just keeps the English text in `msgid`. That fallback only ever applies to the source file: an empty `msgstr` in an *existing target* still means "not translated yet", never "same as msgid". `msgid_plural`/`msgstr[n]` plural entries are translated too (the singular and plural source forms), then expanded to however many `msgstr[n]` the target language's plural rules need (6 for Arabic, 1 for Japanese, ...) — not full grammatical inflection for in-between categories (`few`/`many`/...), which get the plural-sourced translation verbatim, the same simplification `addPluralForms` makes for i18next. `Plural-Forms` in the header is computed for the target language, not copied from the source; an existing target file's header, and any entry's own comments (a translator's note, a hand-added `#, fuzzy`), are otherwise left exactly as last edited instead of being regenerated from the source on every rerun. `--fuzzy` flags every entry a run actually (re)translated with gettext's standard `#, fuzzy` "needs review" marker, merging onto an existing `#,` line rather than adding a second one. A `.pot` source always writes a `.po` target, never a `.pot`. For `.arb`, `@key` metadata and `@@locale` are never translated; `@@locale` is set to the target locale on output. For `.strings`, a file saved as UTF-16 (Xcode's traditional default) is read like a UTF-8 one, an unquoted key (`title = "Settings";`, without quotes around `title`) is read the same as a quoted one, and `\U00e9`-style escapes are decoded — output is always written as UTF-8 with quoted keys, regardless of how the source was saved.
 
 ## Why locale-ai?
 
@@ -105,7 +105,7 @@ then declare it in `locale-ai.config.json`:
 }
 ```
 
-- `path` is a local folder (relative to the config file); `repo` is a Hugging Face repo with the converted files (uploaded output folder), downloaded and cached like built-in models.
+- `path` is a local folder (relative to the config file); `repo` is a Hugging Face repo with the converted files (uploaded output folder), downloaded and cached like built-in models — it must look like `namespace/name`, rejected otherwise (it's joined onto the cache directory with no further checks, so anything else could resolve outside it).
 - `prefix` is required for multi-target models (`convert` lists the model's `>>xx<<` tokens).
 - Project models take priority over built-ins for their language pairs and can be used for pivoting (e.g. `ja → en → tr`).
 

package/dist/catalog.js

@@ -90,10 +90,23 @@ export const CATALOG = [
     },
 ];
 const asList = (v) => (Array.isArray(v) ? v : [v]).map((x) => parseLocale(x).lang);
+/**
+ * A Hugging Face repo id is always exactly "namespace/name". `models.ts` joins it
+ * onto the cache root with no further checks, so a `repo` of "../../../etc" (or any
+ * segment that isn't a real name) would resolve outside the cache directory — and
+ * install() writes there, uninstall() recursively deletes there.
+ */
+function validRepo(repo) {
+    const parts = repo.split("/");
+    return parts.length === 2 && parts.every((p) => p !== "" && p !== "." && p !== ".." && !p.includes("\\"));
+}
 export function customEntries(models, configDir) {
     return (models ?? []).map((c, i) => {
         if (!c.repo === !c.path)
             throw new Error(`models[${i}]: set exactly one of "repo" or "path".`);
+        if (c.repo && !validRepo(c.repo)) {
+            throw new Error(`models[${i}]: "repo" must look like "namespace/name" (a Hugging Face repo id), got "${c.repo}".`);
+        }
         if (!c.source || !c.target)
             throw new Error(`models[${i}]: "source" and "target" are required.`);
         const sources = asList(c.source);

package/dist/cli.js

@@ -33,11 +33,12 @@ Usage:
 Translate options:
   --from <locale>       Source locale (default: en)
   --to <locales>        Target locale(s), comma separated: fr,es,pt-BR
-  --input <dir>         Folder containing <from>/ or <from>.json/.yaml/.yml/.po/.arb/.strings (auto-detected)
+  --input <dir>         Folder containing <from>/ or <from>.json/.yaml/.yml/.po/.pot/.arb/.strings (auto-detected)
   --overwrite           Re-translate strings that already have a translation
   --dry-run             Show what would be translated without downloading or writing
   --fast                Greedy decoding: ~2x faster, slightly lower quality
   --report <file>       Write a JSON report of strings that need review
+  --fuzzy               For .po/.pot: flag every entry this run (re)translated as "#, fuzzy"
   --config <file>       Project config (default: ./${CONFIG_FILE} if present)
   -y, --yes             Download required models without asking
 
@@ -149,6 +150,7 @@ async function cmdTranslate(argv) {
             beams: { type: "string" },
             report: { type: "string" },
             config: { type: "string" },
+            fuzzy: { type: "boolean", default: false },
             yes: { type: "boolean", short: "y", default: false },
         },
     });
@@ -165,7 +167,7 @@ async function cmdTranslate(argv) {
     const layout = detectLayout(process.cwd(), from.tag, values.input);
     const lockPath = path.resolve(process.cwd(), LOCK_FILENAME);
     const lock = loadLock(lockPath);
-    const sources = sourceFiles(layout, from.tag).map((f) => ({ file: f, doc: readLocaleDoc(f, from.tag) }));
+    const sources = sourceFiles(layout, from.tag).map((f) => ({ file: f, doc: readLocaleDoc(f, from.tag, true, values.fuzzy) }));
     const singularHints = new Set(sources.flatMap(({ doc }) => [...pluralOneStrings(doc.data)]));
     log(`locale-ai ${VERSION}`);
     log(`Found ${sources.length} ${from.tag} file${sources.length === 1 ? "" : "s"} in ${rel(layout.root)}/` +
@@ -417,7 +419,7 @@ async function cmdCheck(argv) {
     });
     const from = parseLocale(values.from);
     const layout = detectLayout(process.cwd(), from.tag, values.input);
-    const sources = sourceFiles(layout, from.tag).map((f) => ({ file: f, doc: readLocaleDoc(f, from.tag) }));
+    const sources = sourceFiles(layout, from.tag).map((f) => ({ file: f, doc: readLocaleDoc(f, from.tag, true) }));
     const targets = existingTargets(layout, from.tag);
     if (!targets.length) {
         log(`No translated locales found next to ${from.tag}.`);

package/dist/locales.js

@@ -2,7 +2,7 @@ import fs from "node:fs";
 import path from "node:path";
 import { parse as parseYaml, stringify as stringifyYaml } from "yaml";
 import { parseLocale } from "./lang.js";
-const EXTENSIONS = [".json", ".yaml", ".yml", ".po", ".arb", ".strings"];
+const EXTENSIONS = [".json", ".yaml", ".yml", ".po", ".pot", ".arb", ".strings"];
 const CANDIDATES = [
     "locales",
     "src/locales",
@@ -78,10 +78,14 @@ function listLocaleFiles(dir) {
 export function sourceFiles(layout, from) {
     return layout.kind === "dir" ? listLocaleFiles(path.join(layout.root, from)) : [path.join(layout.root, `${from}${layout.ext}`)];
 }
+/** A translated file is a real PO, never a template, regardless of what the source was. */
+const outputExt = (ext) => (ext === ".pot" ? ".po" : ext);
 export function targetFile(layout, from, to, src) {
     if (layout.kind === "file")
-        return path.join(layout.root, `${to}${layout.ext}`);
-    return path.join(layout.root, to, path.relative(path.join(layout.root, from), src));
+        return path.join(layout.root, `${to}${outputExt(layout.ext)}`);
+    const rel = path.relative(path.join(layout.root, from), src);
+    const ext = path.extname(rel);
+    return path.join(layout.root, to, ext === ".pot" ? `${rel.slice(0, -ext.length)}.po` : rel);
 }
 /** True when `tag` parses as a locale (e.g. "fr", "pt-BR") — rejects names like "__mocks__" or "schemas". */
 function looksLikeLocale(tag) {
@@ -364,21 +368,37 @@ function parsePo(text) {
 /** Synthetic data keys for a plural entry's two translatable forms. */
 const pluralOneKey = (e) => poKey(e) + "\u0005one";
 const pluralOtherKey = (e) => poKey(e) + "\u0005other";
-function readPoDoc(file, text) {
+/** Adds a `#, fuzzy` gettext flag to a comment block, merging onto an existing `#,` line rather than duplicating it. */
+function withFuzzyFlag(comments) {
+    const i = comments.findIndex((c) => /^#,/.test(c));
+    if (i === -1)
+        return [...comments, "#, fuzzy"];
+    if (/\bfuzzy\b/.test(comments[i]))
+        return comments;
+    return comments.map((c, j) => (j === i ? `${c}, fuzzy` : c));
+}
+function readPoDoc(file, text, isSource, fuzzy) {
     const { entries, leading } = parsePo(text);
     const data = newObj();
     for (const e of entries) {
         if (e.isHeader)
             continue;
-        if (e.msgidPlural === undefined)
-            data[poKey(e)] = e.msgstr;
+        if (e.msgidPlural === undefined) {
+            // A template (.pot) or a project that keeps the English text in msgid and
+            // leaves the source .po's own msgstr empty: fall back to msgid so the source
+            // text isn't just "", which would otherwise translate every entry to nothing
+            // with no warning. Never done for a target file, where an empty msgstr
+            // legitimately means "not translated yet".
+            data[poKey(e)] = isSource && e.msgstr === "" ? e.msgid : e.msgstr;
+        }
         else {
             // Like msgstr for a singular entry, msgstrPlural holds this FILE's own text: the
             // English source form in the source .po, the translated forms in a target .po
             // (msgid/msgid_plural are the stable keys, identical across every locale).
             const forms = e.msgstrPlural ?? [];
-            data[pluralOneKey(e)] = forms[0] ?? "";
-            data[pluralOtherKey(e)] = forms[forms.length - 1] ?? forms[0] ?? "";
+            const useMsgid = isSource && forms.every((f) => f === "");
+            data[pluralOneKey(e)] = useMsgid ? e.msgid : forms[0] ?? "";
+            data[pluralOtherKey(e)] = useMsgid ? e.msgidPlural : forms[forms.length - 1] ?? forms[0] ?? "";
         }
     }
     const keepNewline = finalNewline(text);
@@ -386,24 +406,35 @@ function readPoDoc(file, text) {
         const obj = isObj(d) ? d : newObj();
         const seen = new Set();
         const blocks = leading.length ? [leading.join("\n")] : [];
+        const existingEntries = existingText ? parsePo(existingText).entries : [];
         // A previous run may have left a header a translator then hand-fixed (a corrected
         // Plural-Forms, an added Last-Translator, ...). Keep it as-is rather than
         // regenerating from the source every time; only a brand-new target file gets a
         // freshly computed header.
-        const existingHeader = existingText && parsePo(existingText).entries.find((e) => e.isHeader);
+        const existingHeader = existingEntries.find((e) => e.isHeader);
+        // Comments (translator notes, `#, fuzzy` flags, ...) a translator may have added
+        // directly to the target file, keyed the same way entries are. Previously
+        // serialize always used the SOURCE's own comments for every entry, silently
+        // wiping anything a translator had added to the target on the very next run.
+        const existingByKey = new Map(existingEntries.filter((e) => !e.isHeader).map((e) => [poKey(e), e]));
         for (const e of entries) {
-            const lines = [...e.comments];
-            if (e.msgctxt !== undefined)
-                lines.push(`msgctxt "${escapeC(e.msgctxt)}"`);
             if (e.isHeader) {
-                lines.push(`msgid ""`);
+                const lines = [...e.comments, `msgid ""`];
                 const header = existingHeader
                     ? existingHeader.msgstr
                     : e.msgstr.replace(/^Language:.*$/m, `Language: ${targetTag}`).replace(/^Plural-Forms:.*$/m, `Plural-Forms: ${pluralFormsHeader(targetTag)}`);
                 lines.push(`msgstr "${escapeC(header)}"`);
+                blocks.push(lines.join("\n"));
+                continue;
             }
-            else if (e.msgidPlural !== undefined) {
-                lines.push(`msgid "${escapeC(e.msgid)}"`, `msgid_plural "${escapeC(e.msgidPlural)}"`);
+            const existingEntry = existingByKey.get(poKey(e));
+            let comments = existingEntry?.comments ?? e.comments;
+            const content = [];
+            if (e.msgctxt !== undefined)
+                content.push(`msgctxt "${escapeC(e.msgctxt)}"`);
+            let changed;
+            if (e.msgidPlural !== undefined) {
+                content.push(`msgid "${escapeC(e.msgid)}"`, `msgid_plural "${escapeC(e.msgidPlural)}"`);
                 const oneKey = pluralOneKey(e);
                 const otherKey = pluralOtherKey(e);
                 seen.add(oneKey);
@@ -416,15 +447,20 @@ function readPoDoc(file, text) {
                 // plural-sourced one. A language with only one category (ja, zh, ...) gets
                 // just the plural form, since it has no separate singular count.
                 for (let i = 0; i < n; i++)
-                    lines.push(`msgstr[${i}] "${escapeC(n === 1 ? other : i === 0 ? one : other)}"`);
+                    content.push(`msgstr[${i}] "${escapeC(n === 1 ? other : i === 0 ? one : other)}"`);
+                const forms = existingEntry?.msgstrPlural;
+                changed = !forms || forms[0] !== one || forms[forms.length - 1] !== other;
             }
             else {
                 const key = poKey(e);
                 seen.add(key);
                 const v = typeof obj[key] === "string" ? obj[key] : "";
-                lines.push(`msgid "${escapeC(e.msgid)}"`, `msgstr "${escapeC(v)}"`);
+                content.push(`msgid "${escapeC(e.msgid)}"`, `msgstr "${escapeC(v)}"`);
+                changed = existingEntry?.msgstr !== v;
             }
-            blocks.push(lines.join("\n"));
+            if (fuzzy && changed)
+                comments = withFuzzyFlag(comments);
+            blocks.push([...comments, ...content].join("\n"));
         }
         for (const k of Object.keys(obj)) {
             if (seen.has(k) || typeof obj[k] !== "string")
@@ -459,14 +495,21 @@ function readText(file) {
     }
     return buf.toString("utf8").replace(/^/, "");
 }
-/** `tag` is the locale this file is for (the source tag, or an existing target's own tag). */
-export function readLocaleDoc(file, tag) {
+/**
+ * `tag` is the locale this file is for (the source tag, or an existing target's own
+ * tag). `isSource` marks this as the FROM-locale file being read as the thing to
+ * translate (as opposed to an already-existing target) — only then does a `.po`
+ * entry's empty `msgstr` fall back to `msgid` as the source text (see readPoDoc).
+ * `fuzzy`, for `.po`/`.pot` only, marks every entry this run actually (re)translated
+ * with gettext's standard `#, fuzzy` "needs review" flag.
+ */
+export function readLocaleDoc(file, tag, isSource = false, fuzzy = false) {
     const text = readText(file);
     const ext = path.extname(file).toLowerCase();
     if (ext === ".yaml" || ext === ".yml")
         return readYamlDoc(file, text, tag);
-    if (ext === ".po")
-        return readPoDoc(file, text);
+    if (ext === ".po" || ext === ".pot")
+        return readPoDoc(file, text, isSource, fuzzy);
     if (ext === ".arb")
         return readArbDoc(file, text);
     if (ext === ".strings")

package/dist/models.js

@@ -25,8 +25,26 @@ export function cacheRoot() {
     return path.join(process.env.XDG_CACHE_HOME || path.join(os.homedir(), ".cache"), "locale-ai");
 }
 const modelsRoot = () => path.join(cacheRoot(), "models");
-/** Where a model's files live: its local folder, or its cache folder. */
-export const modelDir = (m) => m.path ?? path.join(modelsRoot(), ...m.repo.split("/"));
+/**
+ * Where a model's files live: its local folder, or its cache folder. `repo` is
+ * validated as a plain "namespace/name" Hugging Face id when it comes from a
+ * project config (see catalog.ts's customEntries) — this is a second,
+ * independent check that the computed cache folder actually stays inside the
+ * cache root, since install() writes there and uninstall() recursively deletes
+ * it: a `repo` that somehow smuggled a ".." segment past that first check must
+ * never be allowed to touch a path outside the cache.
+ */
+export function modelDir(m) {
+    if (m.path)
+        return m.path;
+    const root = modelsRoot();
+    const dir = path.join(root, ...m.repo.split("/"));
+    const rel = path.relative(root, dir);
+    if (rel.startsWith("..") || path.isAbsolute(rel)) {
+        throw new Error(`${m.id}: "repo" (${m.repo}) resolves outside the model cache.`);
+    }
+    return dir;
+}
 export function readManifest(m) {
     if (m.path)
         return undefined;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "locale-ai",
-  "version": "0.6.8",
+  "version": "0.6.10",
   "description": "Translate your i18n locale files locally on CPU. Free, offline, no API keys.",
   "author": "Taha KHABOUSS",
   "repository": {