locale-ai 0.0.0-stage → 0.2.1

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Taha KHABOUSS
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,141 @@
-# Temporary Holding Version
+# locale-ai
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+Translate your i18n JSON locale files **locally, on CPU, for free**. No API keys, no Python, nothing leaves your machine.
+
+```bash
+npx locale-ai translate --from en --to fr,es,de
+```
+
+```
+locale-ai 0.2.0
+Found 3 en files in locales/en/
+Models to download (one time, then cached for every project):
+  en-fr         114 MB  Xenova/opus-mt-en-fr
+  ...
+Download 346 MB? (Y/n)
+✓ fr         43 strings translated, 3 files written in 2.7s, 1 plural forms added
+✓ es         43 strings translated, 3 files written in 2.5s, 1 plural forms added
+✓ de         43 strings translated, 3 files written in 2.1s
+```
+
+## What it handles
+
+- **Layouts**: `locales/en/*.json` (nested folders too) → `locales/fr/...`, or `messages/en.json` → `messages/fr.json`. Common folders (`locales`, `src/locales`, `public/locales`, `src/i18n`, `messages`, `lang`, ...) are found automatically, or pass `--input`.
+- **Placeholders are never translated or lost**: `{name}`, `{{count}}`, `%s`, `%1$s`, `%(name)s`, `%{name}`, `${name}`, `:attribute`, `$t(key)`, `@:key`, `<b>…</b>`, `<0>…</0>`, `&nbsp;`, URLs, e-mails.
+- **ICU MessageFormat**: `{count, plural, one {# file} other {# files}}`, `select`, nesting. Only the text is translated.
+- **i18next plurals**: adds the forms the target language needs (`_few`, `_many` for Arabic, Russian, Polish...).
+- **Incremental**: only missing keys are translated, existing translations are kept, re-runs are no-ops. `--overwrite` re-translates everything.
+- **Structure**: key order, numbers, booleans, arrays, indentation and trailing newline are preserved.
+
+## Commands
+
+```bash
+locale-ai translate --to fr,pt-BR [--from en] [--input dir] [--dry-run] [--overwrite] [--fast] [--report file] [-y]
+locale-ai models list [--from en]     # supported language pairs
+locale-ai models installed
+locale-ai models install en-de        # pre-download (e.g. in CI)
+locale-ai models remove en-de
+locale-ai models convert <hf-marian-model> --out <dir>   # bring your own model (see below)
+```
+
+`translate` uses beam search (4 beams) by default. `--fast` uses greedy decoding: 2–3x faster, about 0.5 chrF lower.
+
+Models are cached once per machine in `~/.cache/locale-ai` (`LOCALE_AI_CACHE` to change). In CI, pass `--yes`.
+
+## Project config (optional)
+
+`locale-ai.config.json` in the project root:
+
+```json
+{
+  "doNotTranslate": ["Acme", "Acme Pro"],
+  "glossary": {
+    "fr": { "Workspace": "Espace de travail" },
+    "de": { "Sign in": "Einloggen" }
+  }
+}
+```
+
+`glossary` entries replace whole strings. `models` adds your own translation models (next section). A built-in glossary covers ~60 standard UI labels ("Sign in", "Settings", "Log out", ...) for fr, es, de, it, pt-BR, nl and ar, because one-to-three word labels are where sentence-level MT is weakest.
+
+## Add a language, or a better model
+
+Any [MarianMT / OPUS-MT](https://huggingface.co/Helsinki-NLP) model on Hugging Face can be used. Convert it once (needs Python):
+
+```bash
+pip install "optimum-onnx[onnxruntime]" accelerate sentencepiece
+locale-ai models convert Helsinki-NLP/opus-mt-tc-big-en-tr --out models/en-tr --source en --target tr
+```
+
+then declare it in `locale-ai.config.json`:
+
+```json
+{
+  "models": [
+    { "source": "en", "target": "tr", "path": "models/en-tr" },
+    { "source": "en", "target": ["ru", "uk"], "path": "models/en-zle", "prefix": { "ru": ">>rus<<", "uk": ">>ukr<<" } },
+    { "source": "en", "target": "he", "repo": "your-org/opus-mt-en-he-onnx" }
+  ]
+}
+```
+
+- `path` is a local folder (relative to the config file); `repo` is a Hugging Face repo with the converted files (uploaded output folder), downloaded and cached like built-in models.
+- `prefix` is required for multi-target models (`convert` lists the model's `>>xx<<` tokens).
+- Project models take priority over built-ins for their language pairs and can be used for pivoting (e.g. `ja → en → tr`).
+
+**Recommended for quality**: the `opus-mt-tc-big-en-*` models (fr, es, it, pt, ar, tr, and `zle` for ru/uk). They are ~3x larger (270–370 MB int8) and ~1.5x slower, and clearly better:
+
+| Target | Built-in model, chrF / BLEU | tc-big model, chrF / BLEU |
+|---|---|---|
+| fr | 68.9 / 53.7 | **72.5 / 55.9** |
+| es | 72.9 / 56.9 | **73.6 / 57.4** |
+| it | 73.8 / 54.9 | **75.3 / 58.2** |
+| pt | 68.3 / 44.4 | **70.6 / 47.7** (`>>por<<`, or `>>pob<<` for pt-BR) |
+| ru | 65.7 / 46.0 | **68.2 / 48.1** (`>>rus<<`) |
+| ar | 47.8 / 16.6 | **49.7 / 19.9** (`>>ara<<`, with `--fast`) |
+| tr | — | **67.4 / 42.1** |
+
+300 Tatoeba test sentences per language, int8, beam 4 unless noted; reproduce with `bench/` (see `bench/README.md`). `opus-mt-tc-big-en-ko` is broken upstream (garbage output even in PyTorch) and is not recommended.
+
+## Languages
+
+From English: af, ar, ca, cs, da, de, es, fi, fr, gl, hi, hu, id, it, nl, oc, pt (pt-BR, pt-PT), ro, ru, sv, uk, vi, xh, zh (zh-CN, zh-TW). Other source languages are supported directly or through English (`locale-ai models list --from fr`).
+
+Measured on a sample app (see `test/fixtures`):
+
+| Quality (built-in models) | Languages |
+|---|---|
+| Good — light review | fr, es, de, pt-BR, nl |
+| Usable — review before shipping | it, ar, ru, zh-CN, zh-TW |
+| Untested | the other pairs in `locale-ai models list` |
+
+With the tc-big models above, it, ar and ru move up a tier and tr becomes available. Not available yet: ja, ko, pl and others; he works with a converted `opus-mt-en-he` (weaker). Moroccan Arabic (Darija): the tc-big en-ar model has a `>>ary<<` token, but its output is close to Modern Standard Arabic — not supported.
+
+## How it works
+
+- **Models**: [OPUS-MT](https://huggingface.co/Helsinki-NLP) (MarianMT) in ONNX, int8-quantized (int8 vs fp32 measured equal: en-fr chrF 68.9 vs 69.0, en-ar 47.8 vs 47.8). Downloads are pinned to a commit and verified (sha256 / git blob hash).
+- **Own runtime** on [onnxruntime](https://onnxruntime.ai) (`src/marian.ts`), no transformers.js:
+  - a SentencePiece Unigram tokenizer including the model's precompiled normalisation map, identical to Python `sentencepiece` on 3,577/3,577 test sentences across 5 models (`src/spm.ts`);
+  - real beam search with Hugging Face semantics, identical to `transformers` PyTorch beam search on 599/600 sentences; log-softmax + top-k run natively in a tiny embedded ONNX graph (`scripts/make-topk-graph.py`);
+  - KV cache with batch compaction: finished sentences leave the batch, so short and long strings share batches cheaply.
+- **Sentence splitting**: OPUS-MT drops sentences from multi-sentence inputs, so each sentence is translated separately.
+- **Placeholder masking** with three marker styles tried in order (`{0}` / `<0>…</0>`, then numbers `900`, then `_P0_`), validating every marker; numbers survive models that mangle braces (en-zh, en-ru). A last-resort fallback translates around placeholders, and anything that needed it is listed for review.
+- **Speed** (20-core CPU, ~1 GB RAM): sample app (43 strings) in ~3 s per language with beam search, ~2 s with `--fast`; full sentences at ~18/s (built-in, beam 4), ~12/s (tc-big, beam 4), ~33/s (tc-big, `--fast`).
+
+## Development
+
+```bash
+npm install
+npm test            # unit tests (no models)
+npm run test:e2e    # real models, downloads ~115 MB once
+node bench/eval.ts --dir <model dir> --lang fr --beams 4 --out out.json   # quality benchmark (see bench/README.md)
+npm run dev -- translate --to fr   # run from source (Node ≥ 22.18 strips types)
+npm run build       # → dist/
+```
+
+## Next steps
+
+1. **Host the tc-big models** (converted ONNX) on Hugging Face so they can become the built-in defaults instead of a manual conversion.
+2. **Smaller install.** `onnxruntime-node` is ~550 MB unpacked: it ships binaries for every platform plus a 260 MB CUDA provider we never use.
+3. **More languages**: ja, ko, pl have no usable OPUS-MT model yet; a different model family would be needed.
+4. **More formats**: YAML, `.po`, ARB, Apple `.strings`.

package/dist/catalog.js

@@ -0,0 +1,121 @@
+import path from "node:path";
+import { parseLocale } from "./lang.js";
+const pair = (src, tgt) => ({
+    id: `${src}-${tgt}`,
+    repo: `Xenova/opus-mt-${src}-${tgt}`,
+    sources: [src],
+    targets: [tgt],
+});
+const DIRECT = [
+    "af-en", "ar-en", "cs-en", "da-de", "da-en", "de-en", "de-es", "de-fr",
+    "en-af", "en-cs", "en-da", "en-de", "en-es", "en-fi", "en-fr", "en-hi",
+    "en-hu", "en-id", "en-it", "en-nl", "en-ro", "en-ru", "en-sv", "en-uk",
+    "en-xh", "es-de", "es-en", "es-fr", "es-it", "es-ru", "et-en", "fi-de",
+    "fi-en", "fr-de", "fr-en", "fr-es", "fr-ro", "fr-ru", "hi-en", "hu-en",
+    "id-en", "it-en", "it-es", "it-fr", "ja-en", "ko-en", "nl-en", "nl-fr",
+    "no-de", "pl-en", "ro-fr", "ru-en", "ru-es", "ru-fr", "ru-uk", "sv-en",
+    "th-en", "tr-en", "uk-en", "uk-ru", "vi-en", "xh-en", "zh-en",
+];
+const ROMANCE = ["es", "pt", "ca", "gl", "oc", "it", "ro", "fr"];
+export const CATALOG = [
+    ...DIRECT.map((p) => {
+        const [s, t] = p.split("-");
+        return pair(s, t);
+    }),
+    {
+        // Note: the model card lists Moroccan Arabic ("ary") but its vocabulary has no
+        // >>ary<< token, so Darija is not supported.
+        ...pair("en", "ar"),
+        prefix: () => ">>ara<<",
+    },
+    {
+        ...pair("en", "zh"),
+        prefix: (t) => t.script === "Hant" || ["TW", "HK", "MO"].includes(t.region ?? "") ? ">>cmn_Hant<<" : ">>cmn_Hans<<",
+    },
+    { ...pair("en", "vi"), prefix: () => ">>vie<<" },
+    {
+        id: "en-ROMANCE",
+        repo: "Xenova/opus-mt-en-ROMANCE",
+        sources: ["en"],
+        targets: ROMANCE,
+        prefix: (t) => {
+            if (t.lang === "pt" && t.region === "BR")
+                return ">>pt_BR<<";
+            if (t.lang === "pt" && t.region === "PT")
+                return ">>pt_PT<<";
+            return `>>${t.lang}<<`;
+        },
+    },
+    {
+        id: "ROMANCE-en",
+        repo: "Xenova/opus-mt-ROMANCE-en",
+        sources: ROMANCE,
+        targets: ["en"],
+    },
+];
+const asList = (v) => (Array.isArray(v) ? v : [v]).map((x) => parseLocale(x).lang);
+export function customEntries(models, configDir) {
+    return (models ?? []).map((c, i) => {
+        if (!c.repo === !c.path)
+            throw new Error(`models[${i}]: set exactly one of "repo" or "path".`);
+        if (!c.source || !c.target)
+            throw new Error(`models[${i}]: "source" and "target" are required.`);
+        const sources = asList(c.source);
+        const targets = asList(c.target);
+        const prefix = c.prefix;
+        return {
+            id: c.id ?? `custom:${sources.join("+")}-${targets.join("+")}`,
+            repo: c.repo,
+            path: c.path ? path.resolve(configDir, c.path) : undefined,
+            custom: true,
+            sources,
+            targets,
+            prefix: prefix === undefined
+                ? undefined
+                : (t) => (typeof prefix === "string" ? prefix : (prefix[t.tag] ?? prefix[t.lang])),
+        };
+    });
+}
+export function findById(id, catalog = CATALOG) {
+    const lower = id.toLowerCase();
+    return catalog.find((m) => m.id.toLowerCase() === lower || m.repo?.toLowerCase() === lower);
+}
+/** Best single model for a pair: custom models first, then dedicated pair models over multi-target ones. */
+function direct(src, tgt, catalog) {
+    const candidates = catalog.filter((m) => m.sources.includes(src) && m.targets.includes(tgt.lang));
+    const rank = (m) => (m.custom ? 0 : 1000) + m.targets.length + m.sources.length;
+    candidates.sort((a, b) => rank(a) - rank(b));
+    const model = candidates[0];
+    return model && { model, prefix: model.prefix?.(tgt) };
+}
+const EN = { tag: "en", lang: "en" };
+/**
+ * Finds the models needed to go from `source` to `target`: a direct model when one
+ * exists, otherwise a pivot through English. Returns undefined when unsupported.
+ */
+export function resolveRoute(source, target, catalog = CATALOG) {
+    if (source.lang === target.lang)
+        return undefined;
+    const d = direct(source.lang, target, catalog);
+    if (d)
+        return [d];
+    if (source.lang === "en" || target.lang === "en")
+        return undefined;
+    const a = direct(source.lang, EN, catalog);
+    const b = direct("en", target, catalog);
+    return a && b ? [a, b] : undefined;
+}
+/** Every language code reachable from `source` (direct or via English). */
+export function reachableTargets(source, catalog = CATALOG) {
+    const out = new Set();
+    for (const m of catalog)
+        if (m.sources.includes(source))
+            m.targets.forEach((t) => out.add(t));
+    if (source !== "en" && out.has("en")) {
+        for (const m of catalog)
+            if (m.sources.includes("en"))
+                m.targets.forEach((t) => out.add(t));
+    }
+    out.delete(source);
+    return [...out].sort();
+}

package/dist/cli.js

@@ -0,0 +1,370 @@
+#!/usr/bin/env node
+import fs from "node:fs";
+import path from "node:path";
+import { createInterface } from "node:readline/promises";
+import { spawnSync } from "node:child_process";
+import { fileURLToPath } from "node:url";
+import { parseArgs } from "node:util";
+import { CATALOG, customEntries, findById, reachableTargets, resolveRoute, } from "./catalog.js";
+import { translateStrings } from "./engine.js";
+import { CONFIG_FILE, glossaryLookup, loadConfig, protectRegex } from "./glossary.js";
+import { intlCode, parseLocale } from "./lang.js";
+import { addPluralForms, buildTarget, detectLayout, pendingStrings, readJson, sourceFiles, targetFile, } from "./locales.js";
+import { cacheRoot, install, isInstalled, loadTranslator, localModelProblem, planInstall, readManifest, uninstall, } from "./models.js";
+/** Read from package.json so `npm version` is the single source of truth. */
+const VERSION = JSON.parse(fs.readFileSync(fileURLToPath(new URL("../package.json", import.meta.url)), "utf8")).version;
+const HELP = `locale-ai ${VERSION} — translate i18n JSON locale files locally, on CPU, for free.
+
+Usage:
+  locale-ai translate --to <locales> [options]
+  locale-ai models list [--from <locale>]
+  locale-ai models installed
+  locale-ai models install <from-to | model-id> [--yes]
+  locale-ai models remove <model-id>
+  locale-ai models convert <hf-marian-model> --out <dir> [--source en --target ko]
+                        Convert any Hugging Face MarianMT model (needs Python, see README)
+
+Translate options:
+  --from <locale>       Source locale (default: en)
+  --to <locales>        Target locale(s), comma separated: fr,es,pt-BR
+  --input <dir>         Folder containing <from>/ or <from>.json (auto-detected)
+  --overwrite           Re-translate strings that already have a translation
+  --dry-run             Show what would be translated without downloading or writing
+  --fast                Greedy decoding: ~2x faster, slightly lower quality
+  --report <file>       Write a JSON report of strings that need review
+  --config <file>       Project config (default: ./${CONFIG_FILE} if present)
+  -y, --yes             Download required models without asking
+
+Models are cached in ${cacheRoot()} (override with LOCALE_AI_CACHE).
+`;
+const isTTY = process.stderr.isTTY;
+const log = (msg = "") => process.stderr.write(msg + "\n");
+const status = (msg) => {
+    if (isTTY)
+        process.stderr.write(`\r\x1b[2K${msg}`);
+};
+const endStatus = () => {
+    if (isTTY)
+        process.stderr.write("\r\x1b[2K");
+};
+const mb = (bytes) => `${Math.round(bytes / 1e6)} MB`;
+const rel = (p) => path.relative(process.cwd(), p) || ".";
+function bar(done, total, width = 24) {
+    const ratio = total ? Math.min(1, done / total) : 1;
+    const filled = Math.round(ratio * width);
+    return `[${"█".repeat(filled)}${"░".repeat(width - filled)}] ${Math.round(ratio * 100)}%`;
+}
+async function confirm(question, yes) {
+    if (yes)
+        return true;
+    if (!process.stdin.isTTY) {
+        throw new Error(`${question.trim()} Re-run with --yes to allow downloads in non-interactive mode.`);
+    }
+    const rl = createInterface({ input: process.stdin, output: process.stderr });
+    const answer = (await rl.question(`${question} (Y/n) `)).trim().toLowerCase();
+    rl.close();
+    return answer === "" || answer === "y" || answer === "yes";
+}
+async function ensureInstalled(models, yes) {
+    for (const m of new Set(models)) {
+        const problem = localModelProblem(m);
+        if (problem)
+            throw new Error(`Model ${m.id}: ${problem}`);
+    }
+    const missing = [...new Set(models)].filter((m) => !m.path && !isInstalled(m));
+    if (!missing.length)
+        return;
+    status("Checking model sizes...");
+    const plans = [];
+    for (const m of missing)
+        plans.push(await planInstall(m));
+    endStatus();
+    const total = plans.reduce((s, p) => s + p.bytes, 0);
+    log("Models to download (one time, then cached for every project):");
+    for (const p of plans)
+        log(`  ${p.model.id.padEnd(12)} ${mb(p.bytes).padStart(7)}  ${p.model.repo ?? p.model.path}`);
+    if (!(await confirm(`Download ${mb(total)}?`, yes)))
+        throw new Error("Download declined.");
+    for (const p of plans) {
+        await install(p, (d, t) => status(`Downloading ${p.model.id} ${bar(d, t)} ${mb(d)}/${mb(t)}`));
+        endStatus();
+        log(`✓ Installed ${p.model.id}`);
+    }
+}
+async function cmdTranslate(argv) {
+    const { values } = parseArgs({
+        args: argv,
+        options: {
+            from: { type: "string", default: "en" },
+            to: { type: "string" },
+            input: { type: "string" },
+            overwrite: { type: "boolean", default: false },
+            "dry-run": { type: "boolean", default: false },
+            "batch-size": { type: "string", default: "32" },
+            fast: { type: "boolean", default: false },
+            beams: { type: "string" },
+            report: { type: "string" },
+            config: { type: "string" },
+            out: { type: "string" },
+            source: { type: "string" },
+            target: { type: "string" },
+            python: { type: "string" },
+            yes: { type: "boolean", short: "y", default: false },
+        },
+    });
+    if (!values.to)
+        throw new Error("Missing --to. Example: locale-ai translate --from en --to fr,es");
+    const from = parseLocale(values.from);
+    const targets = values.to.split(",").map((t) => t.trim()).filter(Boolean).map(parseLocale);
+    const batchSize = Math.max(1, Number(values["batch-size"]));
+    const beams = Math.max(1, Number(values.beams ?? (values.fast ? 1 : 4)));
+    const config = loadConfig(process.cwd(), values.config);
+    const catalog = catalogFor(config);
+    const protect = protectRegex(config);
+    const layout = detectLayout(process.cwd(), from.tag, values.input);
+    const sources = sourceFiles(layout, from.tag).map((f) => ({ file: f, doc: readJson(f) }));
+    log(`locale-ai ${VERSION}`);
+    log(`Found ${sources.length} ${from.tag} file${sources.length === 1 ? "" : "s"} in ${rel(layout.root)}/` +
+        (layout.kind === "dir" ? `${from.tag}/` : ""));
+    let failures = 0;
+    const jobs = [];
+    for (const locale of targets) {
+        const route = resolveRoute(from, locale, catalog);
+        if (!route) {
+            failures++;
+            const options = reachableTargets(from.lang, catalog);
+            log(`✗ ${locale.tag}: no local model for ${from.lang} → ${locale.lang}.`);
+            log(`  Available from ${from.lang}: ${options.join(", ") || "(none)"}`);
+            continue;
+        }
+        const files = sources.map(({ file, doc }) => {
+            const target = targetFile(layout, from.tag, locale.tag, file);
+            const existing = fs.existsSync(target) ? readJson(target).data : undefined;
+            return { source: file, target, doc, existing, pending: pendingStrings(doc.data, existing, values.overwrite) };
+        });
+        const unique = new Set(files.flatMap((f) => f.pending)).size;
+        jobs.push({ locale, route, files, unique });
+    }
+    if (values["dry-run"]) {
+        for (const j of jobs) {
+            const models = j.route
+                .map((h) => `${h.model.id}${isInstalled(h.model) ? "" : " (not installed)"}`)
+                .join(" → ");
+            log(`${j.locale.tag.padEnd(8)} ${String(j.unique).padStart(5)} strings to translate   model: ${models}`);
+            for (const f of j.files)
+                if (f.pending.length)
+                    log(`         ${rel(f.target)}  +${f.pending.length}`);
+        }
+        return failures ? 1 : 0;
+    }
+    const active = jobs.filter((j) => j.unique > 0);
+    await ensureInstalled(active.flatMap((j) => j.route.map((h) => h.model)), values.yes);
+    const allIssues = [];
+    const loaded = new Map();
+    const key = (h) => `${h.model.id}|${h.prefix ?? ""}`;
+    const remainingUses = (repoKey, from) => active.slice(from).filter((j) => j.route.some((h) => key(h) === repoKey)).length;
+    for (const job of jobs) {
+        const started = Date.now();
+        let translations = new Map();
+        if (job.unique > 0) {
+            const translators = [];
+            for (const hop of job.route) {
+                if (!loaded.has(key(hop))) {
+                    status(`${job.locale.tag}: loading ${hop.model.id}...`);
+                    loaded.set(key(hop), await loadTranslator(hop, { beams }));
+                }
+                translators.push(loaded.get(key(hop)));
+            }
+            const strings = job.files.flatMap((f) => f.pending);
+            const result = await translateStrings(strings, translators, {
+                sourceLang: intlCode(from),
+                targetLang: intlCode(job.locale),
+                batchSize,
+                protect,
+                lookup: glossaryLookup(from, job.locale, config),
+                onProgress: (d, t) => status(`${job.locale.tag}: translating ${bar(d, t)} ${d}/${t}`),
+            });
+            endStatus();
+            translations = result.translations;
+            allIssues.push(...result.issues.map((i) => ({ ...i, locale: job.locale.tag })));
+            for (const hop of job.route) {
+                const k = key(hop);
+                if (remainingUses(k, active.indexOf(job) + 1) === 0 && loaded.has(k)) {
+                    await loaded.get(k).dispose();
+                    loaded.delete(k);
+                }
+            }
+        }
+        let written = 0;
+        let pluralsAdded = 0;
+        for (const f of job.files) {
+            const built = buildTarget(f.doc.data, f.existing, (s) => translations.get(s) ?? s, values.overwrite);
+            const { doc, added } = addPluralForms(built, intlCode(job.locale));
+            pluralsAdded += added;
+            const text = JSON.stringify(doc, null, f.doc.indent) + (f.doc.finalNewline ? "\n" : "");
+            const before = fs.existsSync(f.target) ? fs.readFileSync(f.target, "utf8") : undefined;
+            if (text !== before) {
+                fs.mkdirSync(path.dirname(f.target), { recursive: true });
+                fs.writeFileSync(f.target, text);
+                written++;
+            }
+        }
+        const secs = ((Date.now() - started) / 1000).toFixed(1);
+        const issues = allIssues.filter((i) => i.locale === job.locale.tag).length;
+        const via = job.route.length > 1 ? ` (via ${job.route.map((h) => h.model.id).join(" → ")})` : "";
+        log(`✓ ${job.locale.tag.padEnd(7)} ${String(job.unique).padStart(5)} strings translated, ${written} file${written === 1 ? "" : "s"} written in ${secs}s${via}` +
+            (pluralsAdded ? `, ${pluralsAdded} plural forms added` : "") +
+            (issues ? `, ${issues} to review` : ""));
+    }
+    for (const t of loaded.values())
+        await t.dispose();
+    if (allIssues.length) {
+        log();
+        log(`${allIssues.length} string${allIssues.length === 1 ? "" : "s"} worth a quick review:`);
+        for (const i of allIssues.slice(0, 10)) {
+            log(`  [${i.locale}] ${i.kind}: ${JSON.stringify(i.source)}${i.output ? ` → ${JSON.stringify(i.output)}` : ""}`);
+        }
+        if (allIssues.length > 10)
+            log(`  ... and ${allIssues.length - 10} more${values.report ? "" : " (use --report <file>)"}`);
+    }
+    if (values.report) {
+        fs.writeFileSync(values.report, JSON.stringify({ version: VERSION, from: from.tag, issues: allIssues }, null, 2) + "\n");
+        log(`Report written to ${values.report}`);
+    }
+    return failures ? 1 : 0;
+}
+/** Runs scripts/convert-marian.py and prints the config entry for the new model. */
+function convertModel(model, opts) {
+    if (!model || !opts.out)
+        throw new Error("Usage: locale-ai models convert <hf-marian-model> --out <dir>");
+    const script = fileURLToPath(new URL("../scripts/convert-marian.py", import.meta.url));
+    const python = opts.python ?? process.env.PYTHON ?? (process.platform === "win32" ? "python" : "python3");
+    const res = spawnSync(python, [script, model, "--out", opts.out], { stdio: "inherit" });
+    if (res.error || res.status !== 0) {
+        log(`\nConversion failed. It needs Python 3.10+ with: pip install "optimum-onnx[onnxruntime]" accelerate sentencepiece`);
+        log(`(use --python <path> to pick an interpreter, e.g. a virtualenv)`);
+        return 1;
+    }
+    const entry = {
+        source: opts.source ?? "en",
+        target: opts.target ?? "<target language code>",
+        path: path.relative(process.cwd(), path.resolve(opts.out)) || ".",
+    };
+    try {
+        const vocab = JSON.parse(fs.readFileSync(path.join(opts.out, "vocab.json"), "utf8"));
+        const codes = Object.keys(vocab).filter((k) => /^>>[\w-]+<<$/.test(k));
+        if (codes.length) {
+            entry.prefix = "<one of the target tokens below>";
+            log(`\nThis model is multi-target; its target tokens: ${codes.join(" ")}`);
+        }
+    }
+    catch {
+        // vocab.json missing: the script would have failed already.
+    }
+    log(`\nAdd to ${CONFIG_FILE} under "models":\n${JSON.stringify(entry, null, 2)}`);
+    return 0;
+}
+/** Built-in models plus the project's own (which take priority). */
+function catalogFor(config) {
+    return [...customEntries(config.models, config.dir ?? process.cwd()), ...CATALOG];
+}
+async function cmdModels(argv) {
+    const [sub, ...rest] = argv;
+    const { values, positionals } = parseArgs({
+        args: rest,
+        allowPositionals: true,
+        options: {
+            from: { type: "string" },
+            config: { type: "string" },
+            out: { type: "string" },
+            source: { type: "string" },
+            target: { type: "string" },
+            python: { type: "string" },
+            yes: { type: "boolean", short: "y", default: false },
+        },
+    });
+    const catalog = catalogFor(loadConfig(process.cwd(), values.config));
+    switch (sub) {
+        case "list": {
+            const filter = values.from ? parseLocale(values.from).lang : undefined;
+            for (const m of catalog) {
+                if (filter && !m.sources.includes(filter))
+                    continue;
+                const state = isInstalled(m) ? "installed" : "";
+                log(`${m.id.padEnd(12)} ${m.sources.join(",")} → ${m.targets.join(",").padEnd(22)} ${state}`);
+            }
+            if (filter)
+                log(`\nReachable from ${filter} (direct or via English): ${reachableTargets(filter, catalog).join(", ")}`);
+            return 0;
+        }
+        case "installed": {
+            const list = catalog.map((m) => ({ m, man: readManifest(m) })).filter((x) => x.man);
+            if (!list.length)
+                log("No models installed.");
+            for (const { m, man } of list)
+                log(`${m.id.padEnd(12)} ${mb(man.bytes).padStart(7)}  ${man.installedAt.slice(0, 10)}`);
+            return 0;
+        }
+        case "install": {
+            if (!positionals[0])
+                throw new Error("Usage: locale-ai models install <from-to | model-id>");
+            let models;
+            const byId = findById(positionals[0], catalog);
+            if (byId)
+                models = [byId];
+            else {
+                const [a, ...b] = positionals[0].split("-");
+                const route = resolveRoute(parseLocale(a), parseLocale(b.join("-")), catalog);
+                if (!route)
+                    throw new Error(`No model for ${positionals[0]}. See: locale-ai models list`);
+                models = route.map((h) => h.model);
+            }
+            const already = models.filter(isInstalled);
+            already.forEach((m) => log(`✓ ${m.id} already installed`));
+            await ensureInstalled(models, values.yes);
+            return 0;
+        }
+        case "remove": {
+            const m = positionals[0] && findById(positionals[0], catalog);
+            if (!m)
+                throw new Error("Usage: locale-ai models remove <model-id>  (see: locale-ai models installed)");
+            log((await uninstall(m)) ? `Removed ${m.id}` : `${m.id} is not installed`);
+            return 0;
+        }
+        case "convert":
+            return convertModel(positionals[0], values);
+        default:
+            log(HELP);
+            return sub ? 1 : 0;
+    }
+}
+async function main() {
+    const [cmd, ...rest] = process.argv.slice(2);
+    switch (cmd) {
+        case "translate":
+            return cmdTranslate(rest);
+        case "models":
+            return cmdModels(rest);
+        case "-v":
+        case "--version":
+            console.log(VERSION);
+            return 0;
+        case undefined:
+        case "-h":
+        case "--help":
+        case "help":
+            log(HELP);
+            return 0;
+        default:
+            log(`Unknown command "${cmd}".\n`);
+            log(HELP);
+            return 1;
+    }
+}
+main().then((code) => {
+    process.exitCode = code;
+}, (err) => {
+    endStatus();
+    log(`Error: ${err instanceof Error ? err.message : String(err)}`);
+    process.exitCode = 1;
+});