locale-ai 0.4.0 → 0.6.0

package/README.md

@@ -6,7 +6,7 @@
 [![license](https://img.shields.io/npm/l/locale-ai.svg)](LICENSE)
 [![node](https://img.shields.io/node/v/locale-ai.svg)](package.json)
 
-Translate your i18n JSON/YAML locale files **locally, on CPU, for free**. No API keys, no Python, nothing leaves your machine.
+Translate your i18n locale files **locally, on CPU, for free**. No API keys, no Python, nothing leaves your machine.
 
 ```bash
 npx locale-ai translate --from en --to fr,es,de
@@ -26,13 +26,15 @@ Download 577 MB? (Y/n)
 
 ## What it handles
 
-- **Layouts**: `locales/en/*.json` (nested folders too) → `locales/fr/...`, or `messages/en.json` → `messages/fr.json`. `.json`, `.yaml` and `.yml` are all supported, mixed freely in the same project. Common folders (`locales`, `src/locales`, `public/locales`, `src/i18n`, `messages`, `lang`, ...) are found automatically, or pass `--input`.
+- **Layouts**: `locales/en/*.json` (nested folders too) → `locales/fr/...`, or `messages/en.json` → `messages/fr.json`. `.json`, `.yaml`, `.yml`, `.po` (gettext), `.arb` (Flutter) and `.strings` (Apple) are all supported, mixed freely in the same project. Common folders (`locales`, `src/locales`, `public/locales`, `src/i18n`, `messages`, `lang`, ...) are found automatically, or pass `--input`.
 - **Placeholders are never translated or lost**: `{name}`, `{{count}}`, `%s`, `%1$s`, `%(name)s`, `%{name}`, `${name}`, `:attribute`, `$t(key)`, `@:key`, `<b>…</b>`, `<0>…</0>`, `&nbsp;`, URLs, e-mails.
 - **ICU MessageFormat**: `{count, plural, one {# file} other {# files}}`, `select`, nesting. Only the text is translated.
 - **i18next plurals**: adds the forms the target language needs (`_few`, `_many` for Arabic, Russian, Polish...).
 - **Incremental**: only missing keys are translated, existing translations are kept, re-runs are no-ops. `--overwrite` re-translates everything.
 - **Structure**: key order, numbers, booleans, arrays, indentation and trailing newline are preserved.
 
+**Format notes**: for `.po`, `msgid` is the stable key (never changed) and `msgstr` is the translated value, as gettext expects — the English `.po` file's `msgstr` must hold the source text (most generators already fill it; an empty `msgstr` is treated as empty, not inherited from `msgid`), and `msgid_plural`/`msgstr[n]` plural entries pass through untranslated (translate those by hand, or use ICU plurals in `.arb`/JSON/YAML instead). For `.arb`, `@key` metadata and `@@locale` are never translated; `@@locale` is set to the target locale on output.
+
 ## Why locale-ai?
 
 - **No API keys, no cloud bill.** Google/DeepL/OpenAI-based translation tools work well but mean an account, a key, a per-character cost, and your strings leaving your machine. locale-ai runs the model on your own CPU.
@@ -111,19 +113,21 @@ then declare it in `locale-ai.config.json`:
 
 300 Tatoeba test sentences per language, int8, beam 4 unless noted; reproduce with `bench/` (see `bench/README.md`). `opus-mt-tc-big-en-ko` is broken upstream (garbage output even in PyTorch) and is not recommended.
 
+Also new: **pl** (chrF 62.6 / BLEU 40.3, `>>pol<<`), hosted at [khabouss/opus-mt-en-sla-onnx](https://huggingface.co/khabouss/opus-mt-en-sla-onnx) — no tc-big `en-pl` checkpoint exists, so this uses Helsinki-NLP's multi-target Slavic model instead.
+
 ## Languages
 
-From English: af, ar, ca, cs, da, de, es, fi, fr, gl, hi, hu, id, it, nl, oc, pt (pt-BR, pt-PT), ro, ru, sv, uk, vi, xh, zh (zh-CN, zh-TW). Other source languages are supported directly or through English (`locale-ai models list --from fr`).
+From English: af, ar, ca, cs, da, de, es, fi, fr, gl, hi, hu, id, it, nl, oc, pl, pt (pt-BR, pt-PT), ro, ru, sv, uk, vi, xh, zh (zh-CN, zh-TW). Other source languages are supported directly or through English (`locale-ai models list --from fr`).
 
 Measured on a sample app (see `test/fixtures`):
 
 | Quality (built-in models) | Languages |
 |---|---|
 | Good — light review | fr, es, de, pt-BR, nl |
-| Usable — review before shipping | it, ar, ru, zh-CN, zh-TW |
+| Usable — review before shipping | it, ar, ru, pl, zh-CN, zh-TW |
 | Untested | the other pairs in `locale-ai models list` |
 
-`en-fr` is already on the tc-big model (chrF 72.5 / BLEU 55.9) as a built-in, no config needed. With the other tc-big models above, it, ar and ru move up a tier and tr becomes available. Not available yet: ja, ko, pl and others; he works with a converted `opus-mt-en-he` (weaker). Moroccan Arabic (Darija): the tc-big en-ar model has a `>>ary<<` token, but its output is close to Modern Standard Arabic — not supported.
+`en-fr` is already on the tc-big model (chrF 72.5 / BLEU 55.9) as a built-in, no config needed. With the other tc-big models above, it, ar and ru move up a tier, and tr and pl become available. Not available yet: ja, ko and others; he works with a converted `opus-mt-en-he` (weaker). Moroccan Arabic (Darija): the tc-big en-ar model has a `>>ary<<` token, but its output is close to Modern Standard Arabic — not supported.
 
 ## How it works
 
@@ -150,5 +154,4 @@ npm run build       # → dist/
 
 ## Next steps
 
-1. **More languages**: ja, ko, pl have no usable OPUS-MT model yet; a different model family would be needed.
-2. **More formats**: `.po` (gettext), ARB (Flutter), Apple `.strings`. (JSON and YAML are supported today.)
+1. **More languages**: ja, ko have no usable OPUS-MT model yet; a different model family would be needed.

package/dist/catalog.js

@@ -47,6 +47,16 @@ export const CATALOG = [
         targets: ["ru", "uk"],
         prefix: (t) => (t.lang === "ru" ? ">>rus<<" : ">>ukr<<"),
     },
+    {
+        // New: Polish, not supported by any built-in model before this. The tc-big
+        // family has no en-pl checkpoint; this is Helsinki-NLP's multi-target Slavic
+        // model, used for its >>pol<< target only.
+        id: "en-pl",
+        repo: "khabouss/opus-mt-en-sla-onnx", // chrF 62.6, BLEU 40.3
+        sources: ["en"],
+        targets: ["pl"],
+        prefix: () => ">>pol<<",
+    },
     {
         // Note: the model card lists Moroccan Arabic ("ary") but its output stays close
         // to Modern Standard Arabic, so Darija is not treated as supported.

package/dist/cli.js

@@ -13,7 +13,7 @@ import { addPluralForms, buildTarget, detectLayout, pendingStrings, readLocaleDo
 import { cacheRoot, install, isInstalled, loadTranslator, localModelProblem, planInstall, readManifest, uninstall, } from "./models.js";
 /** Read from package.json so `npm version` is the single source of truth. */
 const VERSION = JSON.parse(fs.readFileSync(fileURLToPath(new URL("../package.json", import.meta.url)), "utf8")).version;
-const HELP = `locale-ai ${VERSION} — translate i18n JSON/YAML locale files locally, on CPU, for free.
+const HELP = `locale-ai ${VERSION} — translate i18n locale files locally, on CPU, for free.
 
 Usage:
   locale-ai translate --to <locales> [options]
@@ -27,7 +27,7 @@ Usage:
 Translate options:
   --from <locale>       Source locale (default: en)
   --to <locales>        Target locale(s), comma separated: fr,es,pt-BR
-  --input <dir>         Folder containing <from>/ or <from>.json/.yaml/.yml (auto-detected)
+  --input <dir>         Folder containing <from>/ or <from>.json/.yaml/.yml/.po/.arb/.strings (auto-detected)
   --overwrite           Re-translate strings that already have a translation
   --dry-run             Show what would be translated without downloading or writing
   --fast                Greedy decoding: ~2x faster, slightly lower quality
@@ -201,7 +201,7 @@ async function cmdTranslate(argv) {
             const built = buildTarget(f.doc.data, f.existing, (s) => translations.get(s) ?? s, values.overwrite);
             const { doc, added } = addPluralForms(built, intlCode(job.locale));
             pluralsAdded += added;
-            const text = f.doc.serialize(doc);
+            const text = f.doc.serialize(doc, job.locale.tag);
             const before = fs.existsSync(f.target) ? fs.readFileSync(f.target, "utf8") : undefined;
             if (text !== before) {
                 fs.mkdirSync(path.dirname(f.target), { recursive: true });

package/dist/locales.js

@@ -1,7 +1,7 @@
 import fs from "node:fs";
 import path from "node:path";
 import { parse as parseYaml, stringify as stringifyYaml } from "yaml";
-const EXTENSIONS = [".json", ".yaml", ".yml"];
+const EXTENSIONS = [".json", ".yaml", ".yml", ".po", ".arb", ".strings"];
 const CANDIDATES = [
     "locales",
     "src/locales",
@@ -108,9 +108,214 @@ function readYamlDoc(file, text) {
     const keepNewline = finalNewline(text);
     return { data, serialize: (d) => withNewline(stringifyYaml(d, { indent }), keepNewline) };
 }
+/** Apple .strings: `"key" = "value";`, optionally preceded by a `/* ... *\/` comment. */
+function unescapeC(s) {
+    return s.replace(/\\(.)/g, (_, c) => ({ n: "\n", t: "\t", r: "\r", '"': '"', "\\": "\\" })[c] ?? c);
+}
+function escapeC(s) {
+    return s.replace(/[\\"\n\t\r]/g, (c) => ({ "\\": "\\\\", '"': '\\"', "\n": "\\n", "\t": "\\t", "\r": "\\r" })[c]);
+}
+function readStringsDoc(file, text) {
+    const PAIR = /(?:(\/\*[\s\S]*?\*\/)[ \t]*\r?\n+)?[ \t]*"((?:\\.|[^"\\])*)"\s*=\s*"((?:\\.|[^"\\])*)"\s*;/g;
+    const data = newObj();
+    const entries = [];
+    for (const m of text.matchAll(PAIR)) {
+        const key = unescapeC(m[2]);
+        data[key] = unescapeC(m[3]);
+        entries.push({ key, comment: m[1] });
+    }
+    if (entries.length === 0 && text.trim() !== "") {
+        throw new Error(`No "key" = "value"; entries found in ${file}`);
+    }
+    const keepNewline = finalNewline(text);
+    const serialize = (d) => {
+        const obj = isObj(d) ? d : newObj();
+        const seen = new Set();
+        const blocks = [];
+        for (const e of entries) {
+            seen.add(e.key);
+            const v = typeof obj[e.key] === "string" ? obj[e.key] : "";
+            blocks.push((e.comment ? e.comment + "\n" : "") + `"${escapeC(e.key)}" = "${escapeC(v)}";`);
+        }
+        for (const k of Object.keys(obj)) {
+            if (seen.has(k) || typeof obj[k] !== "string")
+                continue;
+            blocks.push(`"${escapeC(k)}" = "${escapeC(obj[k])}";`);
+        }
+        return withNewline(blocks.join("\n\n"), keepNewline);
+    };
+    return { data, serialize };
+}
+/** ARB (Flutter): plain JSON, but "@key" / "@@locale" entries are metadata, not translatable content. */
+function readArbDoc(file, text) {
+    let raw;
+    try {
+        raw = JSON.parse(text);
+    }
+    catch (e) {
+        throw new Error(`Invalid ARB (JSON) in ${file}: ${e.message}`);
+    }
+    const data = newObj();
+    for (const k of Object.keys(raw))
+        if (!k.startsWith("@") && typeof raw[k] === "string")
+            data[k] = raw[k];
+    const indent = /^\{\r?\n([ \t]+)/.exec(text)?.[1] ?? (text.includes("\n") ? 2 : 0);
+    const keepNewline = finalNewline(text);
+    const hasLocale = typeof raw["@@locale"] === "string";
+    const serialize = (d, targetTag) => {
+        const obj = isObj(d) ? d : newObj();
+        const out = newObj();
+        if (hasLocale)
+            out["@@locale"] = targetTag;
+        for (const k of Object.keys(raw)) {
+            if (k === "@@locale" || k.startsWith("@"))
+                continue;
+            out[k] = typeof obj[k] === "string" ? obj[k] : raw[k];
+            if (`@${k}` in raw)
+                out[`@${k}`] = raw[`@${k}`];
+        }
+        for (const k of Object.keys(obj))
+            if (!(k in out) && !k.startsWith("@"))
+                out[k] = obj[k];
+        return withNewline(JSON.stringify(out, null, indent), keepNewline);
+    };
+    return { data, serialize };
+}
+const poKey = (e) => (e.msgctxt ?? "") + "\u0004" + e.msgid;
+function parsePo(text) {
+    const entries = [];
+    let leading = [];
+    let cur = { comments: [] };
+    let field = null;
+    let buf = "";
+    const flushField = () => {
+        if (field === null)
+            return;
+        const val = unescapeC(buf);
+        if (field === "msgctxt")
+            cur.msgctxt = val;
+        else if (field === "msgid")
+            cur.msgid = val;
+        else if (field === "msgid_plural")
+            cur.msgidPlural = val;
+        else if (field === "msgstr")
+            cur.msgstr = val;
+        else
+            (cur.msgstrPlural ??= [])[Number(field.slice(7, -1))] = val;
+        field = null;
+        buf = "";
+    };
+    const flushEntry = () => {
+        flushField();
+        if (cur.msgid !== undefined) {
+            entries.push({ comments: cur.comments, msgctxt: cur.msgctxt, msgid: cur.msgid, msgidPlural: cur.msgidPlural,
+                msgstrPlural: cur.msgstrPlural, msgstr: cur.msgstr ?? "", isHeader: cur.msgid === "" && cur.msgctxt === undefined });
+        }
+        else if (entries.length === 0 && cur.comments.length && leading.length === 0) {
+            leading = cur.comments;
+        }
+        cur = { comments: [] };
+    };
+    for (const line of text.split(/\r?\n/)) {
+        if (line.trim() === "") {
+            flushEntry();
+            continue;
+        }
+        let m;
+        if (/^#/.test(line)) {
+            flushField();
+            cur.comments.push(line);
+        }
+        else if ((m = /^msgctxt\s+"((?:\\.|[^"\\])*)"\s*$/.exec(line))) {
+            flushField();
+            field = "msgctxt";
+            buf = m[1];
+        }
+        else if ((m = /^msgid_plural\s+"((?:\\.|[^"\\])*)"\s*$/.exec(line))) {
+            flushField();
+            field = "msgid_plural";
+            buf = m[1];
+        }
+        else if ((m = /^msgid\s+"((?:\\.|[^"\\])*)"\s*$/.exec(line))) {
+            flushField();
+            field = "msgid";
+            buf = m[1];
+        }
+        else if ((m = /^msgstr\[(\d+)\]\s+"((?:\\.|[^"\\])*)"\s*$/.exec(line))) {
+            flushField();
+            field = `msgstr[${m[1]}]`;
+            buf = m[2];
+        }
+        else if ((m = /^msgstr\s+"((?:\\.|[^"\\])*)"\s*$/.exec(line))) {
+            flushField();
+            field = "msgstr";
+            buf = m[1];
+        }
+        else if ((m = /^"((?:\\.|[^"\\])*)"\s*$/.exec(line))) {
+            buf += m[1];
+        }
+    }
+    flushEntry();
+    return { entries, leading };
+}
+function readPoDoc(file, text) {
+    const { entries, leading } = parsePo(text);
+    const data = newObj();
+    for (const e of entries)
+        if (!e.isHeader && e.msgidPlural === undefined)
+            data[poKey(e)] = e.msgstr;
+    const keepNewline = finalNewline(text);
+    const serialize = (d, targetTag) => {
+        const obj = isObj(d) ? d : newObj();
+        const seen = new Set();
+        const blocks = leading.length ? [leading.join("\n")] : [];
+        for (const e of entries) {
+            const lines = [...e.comments];
+            if (e.msgctxt !== undefined)
+                lines.push(`msgctxt "${escapeC(e.msgctxt)}"`);
+            if (e.isHeader) {
+                lines.push(`msgid ""`);
+                const header = e.msgstr.replace(/^Language:.*$/m, `Language: ${targetTag}`);
+                lines.push(`msgstr "${escapeC(header)}"`);
+            }
+            else if (e.msgidPlural !== undefined) {
+                lines.push(`msgid "${escapeC(e.msgid)}"`, `msgid_plural "${escapeC(e.msgidPlural)}"`);
+                (e.msgstrPlural ?? []).forEach((s, i) => lines.push(`msgstr[${i}] "${escapeC(s)}"`));
+            }
+            else {
+                const key = poKey(e);
+                seen.add(key);
+                const v = typeof obj[key] === "string" ? obj[key] : "";
+                lines.push(`msgid "${escapeC(e.msgid)}"`, `msgstr "${escapeC(v)}"`);
+            }
+            blocks.push(lines.join("\n"));
+        }
+        for (const k of Object.keys(obj)) {
+            if (seen.has(k) || typeof obj[k] !== "string")
+                continue;
+            const i = k.indexOf("\u0004");
+            const ctx = i === -1 ? undefined : k.slice(0, i);
+            const id = i === -1 ? k : k.slice(i + 1);
+            const lines = ctx ? [`msgctxt "${escapeC(ctx)}"`] : [];
+            lines.push(`msgid "${escapeC(id)}"`, `msgstr "${escapeC(obj[k])}"`);
+            blocks.push(lines.join("\n"));
+        }
+        return withNewline(blocks.join("\n\n"), keepNewline);
+    };
+    return { data, serialize };
+}
 export function readLocaleDoc(file) {
     const text = fs.readFileSync(file, "utf8").replace(/^/, "");
-    return /\.ya?ml$/i.test(file) ? readYamlDoc(file, text) : readJsonDoc(file, text);
+    const ext = path.extname(file).toLowerCase();
+    if (ext === ".yaml" || ext === ".yml")
+        return readYamlDoc(file, text);
+    if (ext === ".po")
+        return readPoDoc(file, text);
+    if (ext === ".arb")
+        return readArbDoc(file, text);
+    if (ext === ".strings")
+        return readStringsDoc(file, text);
+    return readJsonDoc(file, text);
 }
 const isObj = (v) => typeof v === "object" && v !== null && !Array.isArray(v);
 /** Null-prototype object so keys such as "__proto__" are stored as plain data. */

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "locale-ai",
-  "version": "0.4.0",
+  "version": "0.6.0",
   "description": "Translate your i18n locale files locally on CPU. Free, offline, no API keys.",
   "author": "Taha KHABOUSS",
   "repository": {