locale-ai 0.6.7 → 0.6.9

package/README.md

@@ -28,14 +28,14 @@ Download 577 MB? (Y/n)
 
 ## What it handles
 
-- **Layouts**: `locales/en/*.json` (nested folders too) → `locales/fr/...`, or `messages/en.json` → `messages/fr.json`. `.json`, `.yaml`, `.yml`, `.po` (gettext), `.arb` (Flutter) and `.strings` (Apple) are all supported, mixed freely in the same project. Common folders (`locales`, `src/locales`, `public/locales`, `src/i18n`, `messages`, `lang`, ...) are found automatically, or pass `--input`.
+- **Layouts**: `locales/en/*.json` (nested folders too) → `locales/fr/...`, or `messages/en.json` → `messages/fr.json`. `.json`, `.yaml`, `.yml`, `.po`/`.pot` (gettext), `.arb` (Flutter) and `.strings` (Apple) are all supported, mixed freely in the same project. Common folders (`locales`, `src/locales`, `public/locales`, `src/i18n`, `messages`, `lang`, ...) are found automatically, or pass `--input`.
 - **Placeholders are never translated or lost**: `{name}`, `{{count}}`, `%s`, `%1$s`, `%(name)s`, `%{name}`, `${name}`, `:attribute`, `$t(key)`, `@:key`, `<b>…</b>`, `<0>…</0>`, `&nbsp;`, URLs, e-mails.
 - **ICU MessageFormat**: `{count, plural, one {# file} other {# files}}`, `select`, nesting. Only the text is translated.
 - **i18next plurals**: adds the forms the target language needs (`_few`, `_many` for Arabic, Russian, Polish...), so the app doesn't fall back to the English string — each added form copies the `_other` translation verbatim rather than inflecting it, so review and adjust the wording of `_zero`/`_two`/`_few`/`_many` by hand for languages where those forms use a different noun/adjective form. The `_one` form is translated with its `{{count}}` temporarily replaced by a literal "1", since the model otherwise sometimes picks the target's plural noun form despite the English source being grammatically singular.
 - **Incremental, and edit-aware**: only missing or changed keys are translated; untouched translations are kept, so re-runs are no-ops. A small `locale-ai.lock.json` (commit it, like a package lock) remembers a hash of each source string and of the output last written for it per target locale — so editing a source string gets it re-translated automatically (it isn't enough for the target to already hold *some* text), while editing a target string by hand is detected and reported for review instead of being silently overwritten on the next run. `--overwrite` bypasses the lock and re-translates everything.
 - **Structure**: key order, numbers, booleans, arrays, indentation and trailing newline are preserved.
 
-**Format notes**: for `.po`, `msgid` is the stable key (never changed) and `msgstr` is the translated value, as gettext expects — the English `.po` file's `msgstr` must hold the source text (most generators already fill it; an empty `msgstr` is treated as empty, not inherited from `msgid`). `msgid_plural`/`msgstr[n]` plural entries are translated too (the singular and plural source forms), then expanded to however many `msgstr[n]` the target language's plural rules need (6 for Arabic, 1 for Japanese, ...) — not full grammatical inflection for in-between categories (`few`/`many`/...), which get the plural-sourced translation verbatim, the same simplification `addPluralForms` makes for i18next. `Plural-Forms` in the header is computed for the target language, not copied from the source; an existing target file's header is otherwise left as you last edited it (a hand-fixed `Plural-Forms`, `Last-Translator`, ...) instead of being regenerated on every rerun. For `.arb`, `@key` metadata and `@@locale` are never translated; `@@locale` is set to the target locale on output.
+**Format notes**: for `.po`/`.pot`, `msgid` is the stable key (never changed) and `msgstr` is the translated value, as gettext expects — the English source's `msgstr` holds the source text when it's filled in (most generators do), and falls back to `msgid` itself when it's empty, which is normal for a `.pot` template or a project that just keeps the English text in `msgid`. That fallback only ever applies to the source file: an empty `msgstr` in an *existing target* still means "not translated yet", never "same as msgid". `msgid_plural`/`msgstr[n]` plural entries are translated too (the singular and plural source forms), then expanded to however many `msgstr[n]` the target language's plural rules need (6 for Arabic, 1 for Japanese, ...) — not full grammatical inflection for in-between categories (`few`/`many`/...), which get the plural-sourced translation verbatim, the same simplification `addPluralForms` makes for i18next. `Plural-Forms` in the header is computed for the target language, not copied from the source; an existing target file's header, and any entry's own comments (a translator's note, a hand-added `#, fuzzy`), are otherwise left exactly as last edited instead of being regenerated from the source on every rerun. `--fuzzy` flags every entry a run actually (re)translated with gettext's standard `#, fuzzy` "needs review" marker, merging onto an existing `#,` line rather than adding a second one. A `.pot` source always writes a `.po` target, never a `.pot`. For `.arb`, `@key` metadata and `@@locale` are never translated; `@@locale` is set to the target locale on output. For `.strings`, a file saved as UTF-16 (Xcode's traditional default) is read like a UTF-8 one, an unquoted key (`title = "Settings";`, without quotes around `title`) is read the same as a quoted one, and `\U00e9`-style escapes are decoded — output is always written as UTF-8 with quoted keys, regardless of how the source was saved.
 
 ## Why locale-ai?
 

package/dist/cli.js

@@ -33,11 +33,12 @@ Usage:
 Translate options:
   --from <locale>       Source locale (default: en)
   --to <locales>        Target locale(s), comma separated: fr,es,pt-BR
-  --input <dir>         Folder containing <from>/ or <from>.json/.yaml/.yml/.po/.arb/.strings (auto-detected)
+  --input <dir>         Folder containing <from>/ or <from>.json/.yaml/.yml/.po/.pot/.arb/.strings (auto-detected)
   --overwrite           Re-translate strings that already have a translation
   --dry-run             Show what would be translated without downloading or writing
   --fast                Greedy decoding: ~2x faster, slightly lower quality
   --report <file>       Write a JSON report of strings that need review
+  --fuzzy               For .po/.pot: flag every entry this run (re)translated as "#, fuzzy"
   --config <file>       Project config (default: ./${CONFIG_FILE} if present)
   -y, --yes             Download required models without asking
 
@@ -149,6 +150,7 @@ async function cmdTranslate(argv) {
             beams: { type: "string" },
             report: { type: "string" },
             config: { type: "string" },
+            fuzzy: { type: "boolean", default: false },
             yes: { type: "boolean", short: "y", default: false },
         },
     });
@@ -165,7 +167,7 @@ async function cmdTranslate(argv) {
     const layout = detectLayout(process.cwd(), from.tag, values.input);
     const lockPath = path.resolve(process.cwd(), LOCK_FILENAME);
     const lock = loadLock(lockPath);
-    const sources = sourceFiles(layout, from.tag).map((f) => ({ file: f, doc: readLocaleDoc(f, from.tag) }));
+    const sources = sourceFiles(layout, from.tag).map((f) => ({ file: f, doc: readLocaleDoc(f, from.tag, true, values.fuzzy) }));
     const singularHints = new Set(sources.flatMap(({ doc }) => [...pluralOneStrings(doc.data)]));
     log(`locale-ai ${VERSION}`);
     log(`Found ${sources.length} ${from.tag} file${sources.length === 1 ? "" : "s"} in ${rel(layout.root)}/` +
@@ -417,7 +419,7 @@ async function cmdCheck(argv) {
     });
     const from = parseLocale(values.from);
     const layout = detectLayout(process.cwd(), from.tag, values.input);
-    const sources = sourceFiles(layout, from.tag).map((f) => ({ file: f, doc: readLocaleDoc(f, from.tag) }));
+    const sources = sourceFiles(layout, from.tag).map((f) => ({ file: f, doc: readLocaleDoc(f, from.tag, true) }));
     const targets = existingTargets(layout, from.tag);
     if (!targets.length) {
         log(`No translated locales found next to ${from.tag}.`);

package/dist/locales.js

@@ -2,7 +2,7 @@ import fs from "node:fs";
 import path from "node:path";
 import { parse as parseYaml, stringify as stringifyYaml } from "yaml";
 import { parseLocale } from "./lang.js";
-const EXTENSIONS = [".json", ".yaml", ".yml", ".po", ".arb", ".strings"];
+const EXTENSIONS = [".json", ".yaml", ".yml", ".po", ".pot", ".arb", ".strings"];
 const CANDIDATES = [
     "locales",
     "src/locales",
@@ -78,10 +78,14 @@ function listLocaleFiles(dir) {
 export function sourceFiles(layout, from) {
     return layout.kind === "dir" ? listLocaleFiles(path.join(layout.root, from)) : [path.join(layout.root, `${from}${layout.ext}`)];
 }
+/** A translated file is a real PO, never a template, regardless of what the source was. */
+const outputExt = (ext) => (ext === ".pot" ? ".po" : ext);
 export function targetFile(layout, from, to, src) {
     if (layout.kind === "file")
-        return path.join(layout.root, `${to}${layout.ext}`);
-    return path.join(layout.root, to, path.relative(path.join(layout.root, from), src));
+        return path.join(layout.root, `${to}${outputExt(layout.ext)}`);
+    const rel = path.relative(path.join(layout.root, from), src);
+    const ext = path.extname(rel);
+    return path.join(layout.root, to, ext === ".pot" ? `${rel.slice(0, -ext.length)}.po` : rel);
 }
 /** True when `tag` parses as a locale (e.g. "fr", "pt-BR") — rejects names like "__mocks__" or "schemas". */
 function looksLikeLocale(tag) {
@@ -151,19 +155,28 @@ function readYamlDoc(file, text, tag) {
 }
 /** Apple .strings: `"key" = "value";`, optionally preceded by a `/* ... *\/` comment. */
 function unescapeC(s) {
-    return s.replace(/\\(.)/g, (_, c) => ({ n: "\n", t: "\t", r: "\r", '"': '"', "\\": "\\" })[c] ?? c);
+    return s.replace(/\\(?:U([0-9a-fA-F]{4})|(.))/g, (_, hex, c) => {
+        // \Uxxxx is a UTF-16 code unit; two in a row combine into one character via
+        // ordinary JS string concatenation when they're a surrogate pair.
+        if (hex !== undefined)
+            return String.fromCharCode(parseInt(hex, 16));
+        return ({ n: "\n", t: "\t", r: "\r", '"': '"', "\\": "\\" })[c] ?? c;
+    });
 }
 function escapeC(s) {
     return s.replace(/[\\"\n\t\r]/g, (c) => ({ "\\": "\\\\", '"': '\\"', "\n": "\\n", "\t": "\\t", "\r": "\\r" })[c]);
 }
 function readStringsDoc(file, text) {
     // A key's preceding comment is either a /* ... */ block, or one or more consecutive // lines.
-    const PAIR = /(?:((?:\/\*[\s\S]*?\*\/)|(?:\/\/[^\r\n]*(?:\r?\n[ \t]*\/\/[^\r\n]*)*))[ \t]*\r?\n+)?[ \t]*"((?:\\.|[^"\\])*)"\s*=\s*"((?:\\.|[^"\\])*)"\s*;/g;
+    // The old NeXT/Apple plist-string format allows a bare, unquoted key (no spaces,
+    // "=" or ";") as well as a quoted one — Xcode itself only ever writes quoted keys,
+    // but hand-edited or third-party-exported .strings files sometimes have both.
+    const PAIR = /(?:((?:\/\*[\s\S]*?\*\/)|(?:\/\/[^\r\n]*(?:\r?\n[ \t]*\/\/[^\r\n]*)*))[ \t]*\r?\n+)?[ \t]*(?:"((?:\\.|[^"\\])*)"|([^\s"=;]+))\s*=\s*"((?:\\.|[^"\\])*)"\s*;/g;
     const data = newObj();
     const entries = [];
     for (const m of text.matchAll(PAIR)) {
-        const key = unescapeC(m[2]);
-        data[key] = unescapeC(m[3]);
+        const key = m[2] !== undefined ? unescapeC(m[2]) : m[3];
+        data[key] = unescapeC(m[4]);
         entries.push({ key, comment: m[1] });
     }
     if (entries.length === 0 && text.trim() !== "") {
@@ -355,21 +368,37 @@ function parsePo(text) {
 /** Synthetic data keys for a plural entry's two translatable forms. */
 const pluralOneKey = (e) => poKey(e) + "\u0005one";
 const pluralOtherKey = (e) => poKey(e) + "\u0005other";
-function readPoDoc(file, text) {
+/** Adds a `#, fuzzy` gettext flag to a comment block, merging onto an existing `#,` line rather than duplicating it. */
+function withFuzzyFlag(comments) {
+    const i = comments.findIndex((c) => /^#,/.test(c));
+    if (i === -1)
+        return [...comments, "#, fuzzy"];
+    if (/\bfuzzy\b/.test(comments[i]))
+        return comments;
+    return comments.map((c, j) => (j === i ? `${c}, fuzzy` : c));
+}
+function readPoDoc(file, text, isSource, fuzzy) {
     const { entries, leading } = parsePo(text);
     const data = newObj();
     for (const e of entries) {
         if (e.isHeader)
             continue;
-        if (e.msgidPlural === undefined)
-            data[poKey(e)] = e.msgstr;
+        if (e.msgidPlural === undefined) {
+            // A template (.pot) or a project that keeps the English text in msgid and
+            // leaves the source .po's own msgstr empty: fall back to msgid so the source
+            // text isn't just "", which would otherwise translate every entry to nothing
+            // with no warning. Never done for a target file, where an empty msgstr
+            // legitimately means "not translated yet".
+            data[poKey(e)] = isSource && e.msgstr === "" ? e.msgid : e.msgstr;
+        }
         else {
             // Like msgstr for a singular entry, msgstrPlural holds this FILE's own text: the
             // English source form in the source .po, the translated forms in a target .po
             // (msgid/msgid_plural are the stable keys, identical across every locale).
             const forms = e.msgstrPlural ?? [];
-            data[pluralOneKey(e)] = forms[0] ?? "";
-            data[pluralOtherKey(e)] = forms[forms.length - 1] ?? forms[0] ?? "";
+            const useMsgid = isSource && forms.every((f) => f === "");
+            data[pluralOneKey(e)] = useMsgid ? e.msgid : forms[0] ?? "";
+            data[pluralOtherKey(e)] = useMsgid ? e.msgidPlural : forms[forms.length - 1] ?? forms[0] ?? "";
         }
     }
     const keepNewline = finalNewline(text);
@@ -377,24 +406,35 @@ function readPoDoc(file, text) {
         const obj = isObj(d) ? d : newObj();
         const seen = new Set();
         const blocks = leading.length ? [leading.join("\n")] : [];
+        const existingEntries = existingText ? parsePo(existingText).entries : [];
         // A previous run may have left a header a translator then hand-fixed (a corrected
         // Plural-Forms, an added Last-Translator, ...). Keep it as-is rather than
         // regenerating from the source every time; only a brand-new target file gets a
         // freshly computed header.
-        const existingHeader = existingText && parsePo(existingText).entries.find((e) => e.isHeader);
+        const existingHeader = existingEntries.find((e) => e.isHeader);
+        // Comments (translator notes, `#, fuzzy` flags, ...) a translator may have added
+        // directly to the target file, keyed the same way entries are. Previously
+        // serialize always used the SOURCE's own comments for every entry, silently
+        // wiping anything a translator had added to the target on the very next run.
+        const existingByKey = new Map(existingEntries.filter((e) => !e.isHeader).map((e) => [poKey(e), e]));
         for (const e of entries) {
-            const lines = [...e.comments];
-            if (e.msgctxt !== undefined)
-                lines.push(`msgctxt "${escapeC(e.msgctxt)}"`);
             if (e.isHeader) {
-                lines.push(`msgid ""`);
+                const lines = [...e.comments, `msgid ""`];
                 const header = existingHeader
                     ? existingHeader.msgstr
                     : e.msgstr.replace(/^Language:.*$/m, `Language: ${targetTag}`).replace(/^Plural-Forms:.*$/m, `Plural-Forms: ${pluralFormsHeader(targetTag)}`);
                 lines.push(`msgstr "${escapeC(header)}"`);
+                blocks.push(lines.join("\n"));
+                continue;
             }
-            else if (e.msgidPlural !== undefined) {
-                lines.push(`msgid "${escapeC(e.msgid)}"`, `msgid_plural "${escapeC(e.msgidPlural)}"`);
+            const existingEntry = existingByKey.get(poKey(e));
+            let comments = existingEntry?.comments ?? e.comments;
+            const content = [];
+            if (e.msgctxt !== undefined)
+                content.push(`msgctxt "${escapeC(e.msgctxt)}"`);
+            let changed;
+            if (e.msgidPlural !== undefined) {
+                content.push(`msgid "${escapeC(e.msgid)}"`, `msgid_plural "${escapeC(e.msgidPlural)}"`);
                 const oneKey = pluralOneKey(e);
                 const otherKey = pluralOtherKey(e);
                 seen.add(oneKey);
@@ -407,15 +447,20 @@ function readPoDoc(file, text) {
                 // plural-sourced one. A language with only one category (ja, zh, ...) gets
                 // just the plural form, since it has no separate singular count.
                 for (let i = 0; i < n; i++)
-                    lines.push(`msgstr[${i}] "${escapeC(n === 1 ? other : i === 0 ? one : other)}"`);
+                    content.push(`msgstr[${i}] "${escapeC(n === 1 ? other : i === 0 ? one : other)}"`);
+                const forms = existingEntry?.msgstrPlural;
+                changed = !forms || forms[0] !== one || forms[forms.length - 1] !== other;
             }
             else {
                 const key = poKey(e);
                 seen.add(key);
                 const v = typeof obj[key] === "string" ? obj[key] : "";
-                lines.push(`msgid "${escapeC(e.msgid)}"`, `msgstr "${escapeC(v)}"`);
+                content.push(`msgid "${escapeC(e.msgid)}"`, `msgstr "${escapeC(v)}"`);
+                changed = existingEntry?.msgstr !== v;
             }
-            blocks.push(lines.join("\n"));
+            if (fuzzy && changed)
+                comments = withFuzzyFlag(comments);
+            blocks.push([...comments, ...content].join("\n"));
         }
         for (const k of Object.keys(obj)) {
             if (seen.has(k) || typeof obj[k] !== "string")
@@ -431,14 +476,40 @@ function readPoDoc(file, text) {
     };
     return { data, serialize };
 }
-/** `tag` is the locale this file is for (the source tag, or an existing target's own tag). */
-export function readLocaleDoc(file, tag) {
-    const text = fs.readFileSync(file, "utf8").replace(/^/, "");
+/**
+ * Decodes a locale file's raw bytes to text, detecting a UTF-16 BOM first — Xcode's
+ * traditional default for `.strings` files, which `fs.readFileSync(file, "utf8")`
+ * would otherwise mangle into unreadable text (and then "no entries found").
+ */
+function readText(file) {
+    const buf = fs.readFileSync(file);
+    if (buf.length >= 2 && buf[0] === 0xff && buf[1] === 0xfe)
+        return buf.subarray(2).toString("utf16le");
+    if (buf.length >= 2 && buf[0] === 0xfe && buf[1] === 0xff) {
+        const swapped = Buffer.alloc(buf.length - 2);
+        for (let i = 2; i + 1 < buf.length; i += 2) {
+            swapped[i - 2] = buf[i + 1];
+            swapped[i - 1] = buf[i];
+        }
+        return swapped.toString("utf16le");
+    }
+    return buf.toString("utf8").replace(/^/, "");
+}
+/**
+ * `tag` is the locale this file is for (the source tag, or an existing target's own
+ * tag). `isSource` marks this as the FROM-locale file being read as the thing to
+ * translate (as opposed to an already-existing target) — only then does a `.po`
+ * entry's empty `msgstr` fall back to `msgid` as the source text (see readPoDoc).
+ * `fuzzy`, for `.po`/`.pot` only, marks every entry this run actually (re)translated
+ * with gettext's standard `#, fuzzy` "needs review" flag.
+ */
+export function readLocaleDoc(file, tag, isSource = false, fuzzy = false) {
+    const text = readText(file);
     const ext = path.extname(file).toLowerCase();
     if (ext === ".yaml" || ext === ".yml")
         return readYamlDoc(file, text, tag);
-    if (ext === ".po")
-        return readPoDoc(file, text);
+    if (ext === ".po" || ext === ".pot")
+        return readPoDoc(file, text, isSource, fuzzy);
     if (ext === ".arb")
         return readArbDoc(file, text);
     if (ext === ".strings")

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "locale-ai",
-  "version": "0.6.7",
+  "version": "0.6.9",
   "description": "Translate your i18n locale files locally on CPU. Free, offline, no API keys.",
   "author": "Taha KHABOUSS",
   "repository": {