locale-ai 0.6.18 → 0.6.19

package/README.md

@@ -31,13 +31,14 @@ Download 577 MB? (Y/n)
 - **Layouts**: `locales/en/*.json` (nested folders too) → `locales/fr/...`, or `messages/en.json` → `messages/fr.json`. `.json`, `.yaml`, `.yml`, `.po`/`.pot` (gettext), `.arb` (Flutter), `.strings` (Apple) and `.properties` (Java) are all supported, mixed freely in the same project. Common folders (`locales`, `src/locales`, `public/locales`, `src/i18n`, `messages`, `lang`, `config/locales` (Rails), ...) are found automatically, or pass `--input`. Framework-specific naming is also recognized: `flutter gen-l10n`'s `app_en.arb`/`app_fr.arb` filename prefix, Apple's `en.lproj`/`fr.lproj` directory-suffix convention (`--input en.lproj` or just the project root both work), and Java `ResourceBundle`'s `messages_en.properties`/`messages_fr.properties` filename prefix (region-qualified as `messages_pt_BR.properties`, with an underscore — Java's own convention, not this project's usual BCP-47 hyphen), on both the source and the generated target side.
 - **Android**: `res/values/strings.xml` → `res/values-fr/strings.xml` (or `values-pt-rBR` for a region). `<string>`, `<string-array>` and `<plurals>` are all translated; `translatable="false"` strings and `<!-- ... -->` comments are kept as-is. Resource-qualifier scripts/variants (`values-b+sr+Latn`) aren't supported — only a plain language or language-region qualifier.
 - **Apple `.xcstrings`** (Xcode 15+'s String Catalog, replacing `.strings`): every locale lives inside this one file, so point `--input` straight at it (e.g. `--input Localizable.xcstrings`; it isn't auto-detected, since a project can have several). `shouldTranslate: false` entries, and Xcode's per-device variants (`variations.iphone`/`.ipad`) or multi-variable strings (`substitutions`), are left untouched rather than mistranslated — only a single cardinal plural (`variations.plural`) is supported. A key with no explicit source-language text falls back to the key itself, matching Xcode's own convention.
+- **XLIFF 1.2** (Qt Linguist, Angular's `ng extract-i18n`, and most CAT tools): also only via `--input <file>.xlf`/`.xliff` — the source file's name carries no locale tag (XLIFF puts it in `source-language` instead), only the `<basename>.fr.xlf` convention the target file is written under. Inline markup inside a `<source>` — Angular's `<x id="INTERPOLATION" .../>`, or a generic `<g id="1">...</g>` — passes through untouched, the same as `<b>…</b>` everywhere else. `translate="no"` units, and the rarer non-nesting `<bpt>`/`<ept>` paired-tag markup, are left completely alone. A `<note>` is kept from the source.
 - **Placeholders are never translated or lost**: `{name}`, `{{count}}`, `%s`, `%1$s`, `%(name)s`, `%{name}`, `${name}`, `:attribute`, `$t(key)`, `@:key`, `<b>…</b>`, `<0>…</0>`, `&nbsp;`, URLs, e-mails.
 - **ICU MessageFormat**: `{count, plural, one {# file} other {# files}}`, `select`, nesting. Only the text is translated.
 - **i18next plurals**: adds the forms the target language needs (`_few`, `_many` for Arabic, Russian, Polish...), so the app doesn't fall back to the English string — each added form copies the `_other` translation verbatim rather than inflecting it, so review and adjust the wording of `_zero`/`_two`/`_few`/`_many` by hand for languages where those forms use a different noun/adjective form. The `_one` form is translated with its `{{count}}` temporarily replaced by a literal "1", since the model otherwise sometimes picks the target's plural noun form despite the English source being grammatically singular. Rails-style nested plurals (`item:` with `one:`/`other:` children instead of `item_one`/`item_other`) get the same missing-forms treatment — Android's `<plurals>`/`<item quantity="...">` groups and `.xcstrings`' `variations.plural` are the same shape and get it too.
 - **Incremental, and edit-aware**: only missing or changed keys are translated; untouched translations are kept, so re-runs are no-ops. A small `locale-ai.lock.json` (commit it, like a package lock) remembers a hash of each source string and of the output last written for it per target locale — so editing a source string gets it re-translated automatically (it isn't enough for the target to already hold *some* text), while editing a target string by hand is detected and reported for review instead of being silently overwritten on the next run. `--overwrite` bypasses the lock and re-translates everything.
 - **Structure**: key order, numbers, booleans, arrays, indentation and trailing newline are preserved.
 
-**Format notes**: for `.yaml`/`.yml`, `#` comments (a translator note, an inline `# ...` after a value) survive a translation run — the target file is edited in place rather than rebuilt from scratch, so an existing comment's own value is updated without disturbing the comment, and a brand-new key is appended with no comment of its own. A newly added plural-category key (see above) is likewise appended rather than inserted between existing ones, so its position in the file won't always match the canonical `zero`/`one`/`two`/`few`/`many`/`other` order. For `.po`/`.pot`, `msgid` is the stable key (never changed) and `msgstr` is the translated value, as gettext expects — the English source's `msgstr` holds the source text when it's filled in (most generators do), and falls back to `msgid` itself when it's empty, which is normal for a `.pot` template or a project that just keeps the English text in `msgid`. That fallback only ever applies to the source file: an empty `msgstr` in an *existing target* still means "not translated yet", never "same as msgid". `msgid_plural`/`msgstr[n]` plural entries are translated too (the singular and plural source forms), then expanded to however many `msgstr[n]` the target language's plural rules need (6 for Arabic, 1 for Japanese, ...) — not full grammatical inflection for in-between categories (`few`/`many`/...), which get the plural-sourced translation verbatim, the same simplification `addPluralForms` makes for i18next. `Plural-Forms` in the header is computed for the target language, not copied from the source; an existing target file's header, and any entry's own comments (a translator's note, a hand-added `#, fuzzy`), are otherwise left exactly as last edited instead of being regenerated from the source on every rerun. `--fuzzy` flags every entry a run actually (re)translated with gettext's standard `#, fuzzy` "needs review" marker, merging onto an existing `#,` line rather than adding a second one. A `.pot` source always writes a `.po` target, never a `.pot`. For `.arb`, `@key` metadata and `@@locale` are never translated; `@@locale` is set to the target locale on output. For `.strings`, a file saved as UTF-16 (Xcode's traditional default) is read like a UTF-8 one, an unquoted key (`title = "Settings";`, without quotes around `title`) is read the same as a quoted one, and `\U00e9`-style escapes are decoded — output is always written as UTF-8 with quoted keys, regardless of how the source was saved. For Android `strings.xml`, `translatable="false"` strings and each element's own `<!-- ... -->` comment are kept on first write, but — unlike YAML/`.po`/`.strings` — a rerun regenerates the file from scratch rather than editing an existing target in place, so a hand-edited comment in an already-translated file won't survive a rerun. For `.xcstrings`, every target locale is written into the same one catalog file (each locale's run re-reads it first, so several locales share it safely in one invocation); an entry's `state` (e.g. a translator's `needs_review` flag) is only touched when the value it's attached to actually changes, so hand review survives a rerun. For `.properties`, `#`/`!` comments and blank lines survive a rerun exactly like YAML's — the target file is edited in place rather than rebuilt from scratch; a `\`-continued multi-line value is read correctly but gets collapsed onto one line if it's ever actually rewritten (an untouched one keeps its original layout), and a line with no unescaped `=`/`:` at all is left completely alone rather than guessed at. `.properties` has no plural convention of its own, so ICU `{count, plural, ...}` inside a value is handled the same generic way it is everywhere else.
+**Format notes**: for `.yaml`/`.yml`, `#` comments (a translator note, an inline `# ...` after a value) survive a translation run — the target file is edited in place rather than rebuilt from scratch, so an existing comment's own value is updated without disturbing the comment, and a brand-new key is appended with no comment of its own. A newly added plural-category key (see above) is likewise appended rather than inserted between existing ones, so its position in the file won't always match the canonical `zero`/`one`/`two`/`few`/`many`/`other` order. For `.po`/`.pot`, `msgid` is the stable key (never changed) and `msgstr` is the translated value, as gettext expects — the English source's `msgstr` holds the source text when it's filled in (most generators do), and falls back to `msgid` itself when it's empty, which is normal for a `.pot` template or a project that just keeps the English text in `msgid`. That fallback only ever applies to the source file: an empty `msgstr` in an *existing target* still means "not translated yet", never "same as msgid". `msgid_plural`/`msgstr[n]` plural entries are translated too (the singular and plural source forms), then expanded to however many `msgstr[n]` the target language's plural rules need (6 for Arabic, 1 for Japanese, ...) — not full grammatical inflection for in-between categories (`few`/`many`/...), which get the plural-sourced translation verbatim, the same simplification `addPluralForms` makes for i18next. `Plural-Forms` in the header is computed for the target language, not copied from the source; an existing target file's header, and any entry's own comments (a translator's note, a hand-added `#, fuzzy`), are otherwise left exactly as last edited instead of being regenerated from the source on every rerun. `--fuzzy` flags every entry a run actually (re)translated with gettext's standard `#, fuzzy` "needs review" marker, merging onto an existing `#,` line rather than adding a second one. A `.pot` source always writes a `.po` target, never a `.pot`. For `.arb`, `@key` metadata and `@@locale` are never translated; `@@locale` is set to the target locale on output. For `.strings`, a file saved as UTF-16 (Xcode's traditional default) is read like a UTF-8 one, an unquoted key (`title = "Settings";`, without quotes around `title`) is read the same as a quoted one, and `\U00e9`-style escapes are decoded — output is always written as UTF-8 with quoted keys, regardless of how the source was saved. For Android `strings.xml`, `translatable="false"` strings and each element's own `<!-- ... -->` comment are kept on first write, but — unlike YAML/`.po`/`.strings` — a rerun regenerates the file from scratch rather than editing an existing target in place, so a hand-edited comment in an already-translated file won't survive a rerun. For `.xcstrings`, every target locale is written into the same one catalog file (each locale's run re-reads it first, so several locales share it safely in one invocation); an entry's `state` (e.g. a translator's `needs_review` flag) is only touched when the value it's attached to actually changes, so hand review survives a rerun. For `.properties`, `#`/`!` comments and blank lines survive a rerun exactly like YAML's — the target file is edited in place rather than rebuilt from scratch; a `\`-continued multi-line value is read correctly but gets collapsed onto one line if it's ever actually rewritten (an untouched one keeps its original layout), and a line with no unescaped `=`/`:` at all is left completely alone rather than guessed at. `.properties` has no plural convention of its own, so ICU `{count, plural, ...}` inside a value is handled the same generic way it is everywhere else. For XLIFF, like Android/ARB/`.xcstrings`, each `<trans-unit>` is regenerated from the *source's* own structure every run rather than edited in place — a translator's own `<target state="...">` attribute won't survive a rerun; `<file target-language="...">` is set to the target locale on output.
 
 ## Why locale-ai?
 

package/dist/cli.js

@@ -35,7 +35,8 @@ Translate options:
   --to <locales>        Target locale(s), comma separated: fr,es,pt-BR
   --input <dir>         Folder containing <from>/ or <from>.json/.yaml/.yml/.po/.pot/.arb/.strings,
                         an Android res/ folder (values/strings.xml) (auto-detected),
-                        or a path straight to a .xcstrings catalog (not auto-detected)
+                        a path straight to a .xcstrings catalog (not auto-detected),
+                        or a path straight to a .xlf/.xliff file (not auto-detected)
   --overwrite           Re-translate strings that already have a translation
   --dry-run             Show what would be translated without downloading or writing
   --fast                Greedy decoding: ~2x faster, slightly lower quality

package/dist/locales.js

@@ -124,6 +124,17 @@ export function detectLayout(cwd, from, input) {
             }
             return { kind: "xcstrings", root: path.dirname(abs), file: abs };
         }
+        if ((abs.endsWith(".xlf") || abs.endsWith(".xliff")) && isFile(abs)) {
+            const t = readText(abs);
+            const fileTag = /<file\b[^>]*>/.exec(t);
+            if (!fileTag)
+                throw new Error(`Invalid XLIFF in ${input}: no <file> element found.`);
+            const sourceLanguage = /\bsource-language="([^"]*)"/.exec(fileTag[0])?.[1];
+            if (sourceLanguage !== undefined && sourceLanguage !== from) {
+                throw new Error(`${input} has source-language "${sourceLanguage}", not "${from}" — pass --from ${sourceLanguage}.`);
+            }
+            return { kind: "xliff", root: path.dirname(abs), file: abs, ext: abs.endsWith(".xliff") ? ".xliff" : ".xlf" };
+        }
         const base = path.basename(abs);
         const directExt = EXTENSIONS.find((ext) => base === `${propertiesTag(ext, from)}${ext}`);
         if (directExt && isFile(abs))
@@ -164,7 +175,7 @@ function listLocaleFiles(dir) {
 export function sourceFiles(layout, from) {
     if (layout.kind === "android")
         return [path.join(layout.root, "values", "strings.xml")];
-    if (layout.kind === "xcstrings")
+    if (layout.kind === "xcstrings" || layout.kind === "xliff")
         return [layout.file];
     return layout.kind === "dir"
         ? listLocaleFiles(path.join(layout.root, dirNameFor(layout, from)))
@@ -177,6 +188,8 @@ export function targetFile(layout, from, to, src) {
         return path.join(layout.root, `values-${androidQualifier(to)}`, "strings.xml");
     if (layout.kind === "xcstrings")
         return layout.file;
+    if (layout.kind === "xliff")
+        return path.join(layout.root, `${path.basename(layout.file, layout.ext)}.${to}${layout.ext}`);
     if (layout.kind === "file")
         return path.join(layout.root, fileNameFor(layout, to, outputExt(layout.ext)));
     const rel = path.relative(path.join(layout.root, dirNameFor(layout, from)), src);
@@ -214,6 +227,17 @@ export function existingTargets(layout, from) {
         tags.delete(from);
         return [...tags].filter(looksLikeLocale).sort();
     }
+    if (layout.kind === "xliff") {
+        const base = path.basename(layout.file, layout.ext);
+        const prefix = `${base}.`;
+        const ext = layout.ext;
+        return fs
+            .readdirSync(layout.root)
+            .filter((f) => f.startsWith(prefix) && f.endsWith(ext) && f.length > prefix.length + ext.length)
+            .map((f) => f.slice(prefix.length, -ext.length))
+            .filter((tag) => tag !== from && looksLikeLocale(tag))
+            .sort();
+    }
     if (layout.kind === "android") {
         return fs
             .readdirSync(layout.root, { withFileTypes: true })
@@ -594,6 +618,129 @@ function readAndroidXmlDoc(file, text) {
     };
     return { data, serialize };
 }
+/** XLIFF's `&entity;` references and `<![CDATA[...]]>` — there's no backslash-escape convention to also handle, unlike Android's XML. */
+function unescapeXml(s) {
+    return s
+        .replace(/<!\[CDATA\[([\s\S]*?)\]\]>/g, (_m, inner) => inner)
+        .replace(/&lt;/g, "<")
+        .replace(/&gt;/g, ">")
+        .replace(/&quot;/g, '"')
+        .replace(/&apos;/g, "'")
+        .replace(/&amp;/g, "&");
+}
+/** Any opening/closing/self-closing tag — the same generic shape placeholders.ts already protects (`<b>`, `<x id="..."/>`, `<g id="1">...</g>`). */
+const XML_TAG_RE = /<\/?[A-Za-z][\w.:-]*(?:\s[^<>]*)?\/?>/g;
+/** Escapes `&`/`<`/`>` in plain text while leaving any inline tag (see XML_TAG_RE) completely untouched, so a translated value's own `<x .../>`/`<g>...</g>` markup — carried through verbatim by the placeholder masking — stays valid, literal XML instead of getting entity-escaped into text. */
+function escapeXmlKeepTags(s) {
+    const escapeText = (t) => t.replace(/&/g, "&amp;").replace(/</g, "&lt;").replace(/>/g, "&gt;");
+    let out = "";
+    let last = 0;
+    for (const m of s.matchAll(XML_TAG_RE)) {
+        out += escapeText(s.slice(last, m.index)) + m[0];
+        last = m.index + m[0].length;
+    }
+    return out + escapeText(s.slice(last));
+}
+/** Sets (inserting if absent) one attribute on a single opening XML tag's raw text. */
+function setXmlAttr(tag, name, value) {
+    const re = new RegExp(`\\b${name}\\s*=\\s*"[^"]*"`);
+    return re.test(tag) ? tag.replace(re, `${name}="${value}"`) : tag.replace(/\/?>$/, (end) => ` ${name}="${value}"${end}`);
+}
+/** Splits XLIFF text into literal passthrough segments and parsed `<trans-unit>` entries, ignoring the surrounding `<file>`/`<body>`/`<group>` nesting (it's never rewritten, just reproduced). */
+function parseXliffUnits(text) {
+    const unitRe = /<trans-unit\b([^>]*)>([\s\S]*?)<\/trans-unit>/g;
+    const segments = [];
+    const units = [];
+    let last = 0;
+    for (const m of text.matchAll(unitRe)) {
+        const before = text.slice(last, m.index);
+        segments.push({ text: before });
+        const [whole, attrsRaw, inner] = m;
+        const idMatch = /\bid="([^"]*)"/.exec(attrsRaw);
+        const sourceMatch = /<source\b[^>]*>([\s\S]*?)<\/source>/.exec(inner);
+        if (!idMatch || !sourceMatch) {
+            segments.push({ text: whole });
+        }
+        else {
+            const targetMatch = /<target\b[^>]*>([\s\S]*?)<\/target>/.exec(inner);
+            const noteMatch = /<note\b[^>]*>[\s\S]*?<\/note>/.exec(inner);
+            const translatable = !/\btranslate\s*=\s*"no"/.test(attrsRaw) && !/<(?:bpt|ept)\b/.test(sourceMatch[1]);
+            const lineIndent = /[ \t]*$/.exec(before)[0];
+            units.push({
+                id: unescapeXml(idMatch[1]),
+                attrsRaw,
+                translatable,
+                sourceInner: sourceMatch[1],
+                targetInner: targetMatch?.[1],
+                noteBlock: noteMatch?.[0] ?? "",
+                whole,
+                indent: before.includes("\n") ? lineIndent : "",
+            });
+            segments.push({ unit: units.length - 1 });
+        }
+        last = m.index + whole.length;
+    }
+    segments.push({ text: text.slice(last) });
+    return { segments, units };
+}
+/**
+ * XLIFF 1.2 (`<xliff version="1.2">`, the default for Qt Linguist, Angular's
+ * `ng extract-i18n`, and most CAT tools): each `<trans-unit id="...">` holds a
+ * `<source>` (always present) and a `<target>` (present once translated). A unit's
+ * inline markup — Angular's `<x id="INTERPOLATION" .../>`, or a generic `<g id="1">
+ * ... </g>` — is carried through to translation as literal text rather than stripped:
+ * it's exactly the shape `placeholders.ts` already protects for HTML-ish i18n values
+ * (`<b>`, `<Trans/>`), so it survives untouched with zero extra code. `translate="no"`
+ * units, and the rarer `<bpt>`/`<ept>` paired-tag markup (a different, non-nesting
+ * convention XLIFF also allows but modern tools rarely emit), are left completely
+ * untouched instead. `<note>` comments are kept from the source.
+ *
+ * Like Android/ARB/`.xcstrings`, this regenerates each unit from the *source's* own
+ * structure every run rather than editing an existing target in place — a translator's
+ * own `<target state="...">` attribute (if any) won't survive a rerun, the same
+ * documented tradeoff Android already makes for its own comments.
+ *
+ * Only reached via `--input <file>.xlf`/`.xliff`: the source file's name carries no
+ * locale tag at all (XLIFF puts it in `source-language` instead), so there's no
+ * filename convention to auto-detect from.
+ */
+function readXliffDoc(file, text, isSource) {
+    if (!/<file\b[^>]*>/.test(text))
+        throw new Error(`Invalid XLIFF in ${file}: no <file> element found.`);
+    const { segments, units } = parseXliffUnits(text);
+    const data = newObj();
+    for (const u of units) {
+        if (!u.translatable)
+            continue;
+        const v = isSource ? u.sourceInner : u.targetInner;
+        if (v !== undefined)
+            data[u.id] = unescapeXml(v);
+    }
+    const step = /\r?\n([ \t]+)</.exec(text)?.[1] ?? "    ";
+    const keepNewline = finalNewline(text);
+    const serialize = (d, targetTag) => {
+        const obj = isObj(d) ? d : newObj();
+        let patchedFileTag = false;
+        const parts = segments.map((seg) => {
+            if ("text" in seg) {
+                if (!patchedFileTag && /<file\b[^>]*>/.test(seg.text)) {
+                    patchedFileTag = true;
+                    return seg.text.replace(/<file\b[^>]*>/, (tag) => setXmlAttr(tag, "target-language", targetTag));
+                }
+                return seg.text;
+            }
+            const u = units[seg.unit];
+            if (!u.translatable)
+                return u.whole;
+            const v = typeof obj[u.id] === "string" ? obj[u.id] : "";
+            const inner = u.indent + step;
+            const noteLine = u.noteBlock ? `\n${inner}${u.noteBlock}` : "";
+            return `<trans-unit${u.attrsRaw}>\n${inner}<source>${u.sourceInner}</source>\n${inner}<target>${escapeXmlKeepTags(v)}</target>${noteLine}\n${u.indent}</trans-unit>`;
+        });
+        return withNewline(parts.join(""), keepNewline);
+    };
+    return { data, serialize };
+}
 /** ARB (Flutter): plain JSON, but "@key" / "@@locale" entries are metadata, not translatable content. */
 function readArbDoc(file, text) {
     let raw;
@@ -1026,6 +1173,8 @@ export function readLocaleDoc(file, tag, isSource = false, fuzzy = false) {
         return readAndroidXmlDoc(file, text);
     if (ext === ".xcstrings")
         return readXcstringsDoc(file, text, tag, isSource);
+    if (ext === ".xlf" || ext === ".xliff")
+        return readXliffDoc(file, text, isSource);
     return readJsonDoc(file, text);
 }
 const isObj = (v) => typeof v === "object" && v !== null && !Array.isArray(v);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "locale-ai",
-  "version": "0.6.18",
+  "version": "0.6.19",
   "description": "Translate your i18n locale files locally on CPU. Free, offline, no API keys.",
   "author": "Taha KHABOUSS",
   "repository": {