locale-ai 0.2.3 → 0.3.1

package/README.md

@@ -16,9 +16,9 @@ npx locale-ai translate --from en --to fr,es,de
 locale-ai 0.2.0
 Found 3 en files in locales/en/
 Models to download (one time, then cached for every project):
-  en-fr         114 MB  Xenova/opus-mt-en-fr
+  en-fr         345 MB  khabouss/opus-mt-tc-big-en-fr-onnx
   ...
-Download 346 MB? (Y/n)
+Download 577 MB? (Y/n)
 ✓ fr         43 strings translated, 3 files written in 2.7s, 1 plural forms added
 ✓ es         43 strings translated, 3 files written in 2.5s, 1 plural forms added
 ✓ de         43 strings translated, 3 files written in 2.1s
@@ -98,11 +98,10 @@ then declare it in `locale-ai.config.json`:
 - `prefix` is required for multi-target models (`convert` lists the model's `>>xx<<` tokens).
 - Project models take priority over built-ins for their language pairs and can be used for pivoting (e.g. `ja → en → tr`).
 
-**Recommended for quality**: the `opus-mt-tc-big-en-*` models (fr, es, it, pt, ar, tr, and `zle` for ru/uk). They are ~3x larger (270–370 MB int8) and ~1.5x slower, and clearly better:
+**Recommended for quality**: the `opus-mt-tc-big-en-*` models (es, it, pt, ar, tr, and `zle` for ru/uk). They are ~3x larger (270–370 MB int8) and ~1.5x slower, and clearly better. `en-fr` already uses one of these as its built-in model (see [Languages](#languages)), hosted at [khabouss/opus-mt-tc-big-en-fr-onnx](https://huggingface.co/khabouss/opus-mt-tc-big-en-fr-onnx) — no conversion needed for it.
 
 | Target | Built-in model, chrF / BLEU | tc-big model, chrF / BLEU |
 |---|---|---|
-| fr | 68.9 / 53.7 | **72.5 / 55.9** |
 | es | 72.9 / 56.9 | **73.6 / 57.4** |
 | it | 73.8 / 54.9 | **75.3 / 58.2** |
 | pt | 68.3 / 44.4 | **70.6 / 47.7** (`>>por<<`, or `>>pob<<` for pt-BR) |
@@ -124,7 +123,7 @@ Measured on a sample app (see `test/fixtures`):
 | Usable — review before shipping | it, ar, ru, zh-CN, zh-TW |
 | Untested | the other pairs in `locale-ai models list` |
 
-With the tc-big models above, it, ar and ru move up a tier and tr becomes available. Not available yet: ja, ko, pl and others; he works with a converted `opus-mt-en-he` (weaker). Moroccan Arabic (Darija): the tc-big en-ar model has a `>>ary<<` token, but its output is close to Modern Standard Arabic — not supported.
+`en-fr` is already on the tc-big model (chrF 72.5 / BLEU 55.9) as a built-in, no config needed. With the other tc-big models above, it, ar and ru move up a tier and tr becomes available. Not available yet: ja, ko, pl and others; he works with a converted `opus-mt-en-he` (weaker). Moroccan Arabic (Darija): the tc-big en-ar model has a `>>ary<<` token, but its output is close to Modern Standard Arabic — not supported.
 
 ## How it works
 
@@ -136,13 +135,14 @@ With the tc-big models above, it, ar and ru move up a tier and tr becomes availa
 - **Sentence splitting**: OPUS-MT drops sentences from multi-sentence inputs, so each sentence is translated separately.
 - **Placeholder masking** with three marker styles tried in order (`{0}` / `<0>…</0>`, then numbers `900`, then `_P0_`), validating every marker; numbers survive models that mangle braces (en-zh, en-ru). A last-resort fallback translates around placeholders, and anything that needed it is listed for review.
 - **Speed** (20-core CPU, ~1 GB RAM): sample app (43 strings) in ~3 s per language with beam search, ~2 s with `--fast`; full sentences at ~18/s (built-in, beam 4), ~12/s (tc-big, beam 4), ~33/s (tc-big, `--fast`).
+- **Install size**: `onnxruntime-node` bundles binaries for every platform, and on linux/x64 downloads a ~260 MB CUDA provider by default — locale-ai only ever runs on CPU (`executionProviders: ["cpu"]`), so a `postinstall` script ([`scripts/trim-onnxruntime.cjs`](scripts/trim-onnxruntime.cjs)) removes the unused platforms and GPU providers after install, cutting it from ~550 MB to ~45 MB on your current platform.
 
 ## Development
 
 ```bash
 npm install
 npm test            # unit tests (no models)
-npm run test:e2e    # real models, downloads ~115 MB once
+npm run test:e2e    # real models, downloads ~345 MB once (en-fr, tc-big)
 node bench/eval.ts --dir <model dir> --lang fr --beams 4 --out out.json   # quality benchmark (see bench/README.md)
 npm run dev -- translate --to fr   # run from source (Node ≥ 22.18 strips types)
 npm run build       # → dist/
@@ -150,7 +150,5 @@ npm run build       # → dist/
 
 ## Next steps
 
-1. **Host the tc-big models** (converted ONNX) on Hugging Face so they can become the built-in defaults instead of a manual conversion.
-2. **Smaller install.** `onnxruntime-node` is ~550 MB unpacked: it ships binaries for every platform plus a 260 MB CUDA provider we never use.
-3. **More languages**: ja, ko, pl have no usable OPUS-MT model yet; a different model family would be needed.
-4. **More formats**: YAML, `.po`, ARB, Apple `.strings`.
+1. **More languages**: ja, ko, pl have no usable OPUS-MT model yet; a different model family would be needed.
+2. **More formats**: YAML, `.po`, ARB, Apple `.strings`.

package/dist/catalog.js

@@ -8,8 +8,8 @@ const pair = (src, tgt) => ({
 });
 const DIRECT = [
     "af-en", "ar-en", "cs-en", "da-de", "da-en", "de-en", "de-es", "de-fr",
-    "en-af", "en-cs", "en-da", "en-de", "en-es", "en-fi", "en-fr", "en-hi",
-    "en-hu", "en-id", "en-it", "en-nl", "en-ro", "en-ru", "en-sv", "en-uk",
+    "en-af", "en-cs", "en-da", "en-de", "en-es", "en-fi", "en-hi",
+    "en-hu", "en-id", "en-it", "en-nl", "en-ro", "en-sv",
     "en-xh", "es-de", "es-en", "es-fr", "es-it", "es-ru", "et-en", "fi-de",
     "fi-en", "fr-de", "fr-en", "fr-es", "fr-ro", "fr-ru", "hi-en", "hu-en",
     "id-en", "it-en", "it-es", "it-fr", "ja-en", "ko-en", "nl-en", "nl-fr",
@@ -22,10 +22,36 @@ export const CATALOG = [
         const [s, t] = p.split("-");
         return pair(s, t);
     }),
+    // tc-big: larger OPUS-MT checkpoints, clearly better than the defaults (see
+    // bench/README.md). Converted and hosted per the README's "Add a language,
+    // or a better model" section.
+    { ...pair("en", "fr"), repo: "khabouss/opus-mt-tc-big-en-fr-onnx" }, // chrF 72.5 vs 68.9, BLEU 55.9 vs 53.7
+    { ...pair("en", "es"), repo: "khabouss/opus-mt-tc-big-en-es-onnx" }, // chrF 73.6 vs 72.9, BLEU 57.4 vs 56.9
+    { ...pair("en", "it"), repo: "khabouss/opus-mt-tc-big-en-it-onnx" }, // chrF 75.3 vs 73.8, BLEU 58.2 vs 54.9
+    // New: Turkish, not supported by any built-in model before this.
+    { ...pair("en", "tr"), repo: "khabouss/opus-mt-tc-big-en-tr-onnx" }, // chrF 67.4, BLEU 42.1
     {
-        // Note: the model card lists Moroccan Arabic ("ary") but its vocabulary has no
-        // >>ary<< token, so Darija is not supported.
+        // Multi-target: >>por<< (European) / >>pob<< (Brazilian). No individual built-in
+        // existed before; pt was only reachable via the weaker en-ROMANCE model.
+        id: "en-pt",
+        repo: "khabouss/opus-mt-tc-big-en-pt-onnx", // chrF 70.6 vs 68.3, BLEU 47.7 vs 44.4
+        sources: ["en"],
+        targets: ["pt"],
+        prefix: (t) => (t.region === "BR" ? ">>pob<<" : ">>por<<"),
+    },
+    {
+        // Multi-target East Slavic model, replaces the separate en-ru/en-uk built-ins.
+        id: "en-zle",
+        repo: "khabouss/opus-mt-tc-big-en-zle-onnx", // ru: chrF 68.2 vs 65.7, BLEU 48.1 vs 46.0
+        sources: ["en"],
+        targets: ["ru", "uk"],
+        prefix: (t) => (t.lang === "ru" ? ">>rus<<" : ">>ukr<<"),
+    },
+    {
+        // Note: the model card lists Moroccan Arabic ("ary") but its output stays close
+        // to Modern Standard Arabic, so Darija is not treated as supported.
         ...pair("en", "ar"),
+        repo: "khabouss/opus-mt-tc-big-en-ar-onnx", // chrF 49.7 vs 47.8, BLEU 19.9 vs 16.6 (greedy)
         prefix: () => ">>ara<<",
     },
     {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "locale-ai",
-  "version": "0.2.3",
+  "version": "0.3.1",
   "description": "Translate your i18n locale files locally on CPU. Free, offline, no API keys.",
   "author": "Taha KHABOUSS",
   "repository": {
@@ -17,7 +17,8 @@
   },
   "files": [
     "dist",
-    "scripts/convert-marian.py"
+    "scripts/convert-marian.py",
+    "scripts/trim-onnxruntime.cjs"
   ],
   "engines": {
     "node": ">=22.18"
@@ -28,7 +29,8 @@
     "dev": "node src/cli.ts",
     "test": "node --test test/*.test.ts",
     "test:e2e": "LOCALE_AI_E2E=1 node --test test/e2e.test.ts",
-    "prepublishOnly": "npm run build && npm test"
+    "prepublishOnly": "npm run build && npm test",
+    "postinstall": "node scripts/trim-onnxruntime.cjs"
   },
   "keywords": [
     "i18n",
@@ -48,5 +50,8 @@
   "devDependencies": {
     "@types/node": "^26.6.6",
     "typescript": "^7.0.2"
+  },
+  "allowScripts": {
+    "onnxruntime-node@1.30.0": true
   }
 }

package/scripts/trim-onnxruntime.cjs

@@ -0,0 +1,54 @@
+#!/usr/bin/env node
+'use strict';
+
+// locale-ai only ever runs onnxruntime-node with executionProviders: ["cpu"]
+// (see src/marian.ts) and only ever runs on one platform/arch per install.
+// onnxruntime-node still bundles binaries for every platform, and on
+// linux/x64 its own postinstall downloads a 260 MB CUDA provider we never
+// use (see its script/install-metadata.js). This trims all of that after
+// onnxruntime-node has finished installing, shrinking a ~550 MB unpacked
+// dependency down to the one binary locale-ai actually loads.
+//
+// Best-effort: never fails the install, even if onnxruntime-node changes
+// its layout in a future version.
+
+const fs = require('fs');
+const path = require('path');
+
+function rm(p) {
+  try {
+    fs.rmSync(p, { recursive: true, force: true });
+  } catch {
+    // ignore — best-effort cleanup only
+  }
+}
+
+try {
+  const pkgJson = require.resolve('onnxruntime-node/package.json');
+  const binRoot = path.join(path.dirname(pkgJson), 'bin', 'napi-v6');
+  if (!fs.existsSync(binRoot)) process.exit(0);
+
+  const platform = process.platform;
+  const arch = process.arch;
+
+  for (const entry of fs.readdirSync(binRoot)) {
+    const platformDir = path.join(binRoot, entry);
+    if (entry !== platform) {
+      rm(platformDir);
+      continue;
+    }
+    // Same platform: drop every other architecture.
+    for (const archEntry of fs.readdirSync(platformDir)) {
+      if (archEntry !== arch) rm(path.join(platformDir, archEntry));
+    }
+    // Drop GPU execution providers locale-ai never requests.
+    const archDir = path.join(platformDir, arch);
+    for (const name of fs.readdirSync(archDir)) {
+      if (/^libonnxruntime_providers_(cuda|tensorrt|rocm|dml)\./.test(name)) {
+        rm(path.join(archDir, name));
+      }
+    }
+  }
+} catch {
+  // onnxruntime-node not installed, or its layout changed — nothing to do.
+}