locale-ai 0.2.3 → 0.3.0

package/README.md

@@ -16,9 +16,9 @@ npx locale-ai translate --from en --to fr,es,de
 locale-ai 0.2.0
 Found 3 en files in locales/en/
 Models to download (one time, then cached for every project):
-  en-fr         114 MB  Xenova/opus-mt-en-fr
+  en-fr         345 MB  khabouss/opus-mt-tc-big-en-fr-onnx
   ...
-Download 346 MB? (Y/n)
+Download 577 MB? (Y/n)
 ✓ fr         43 strings translated, 3 files written in 2.7s, 1 plural forms added
 ✓ es         43 strings translated, 3 files written in 2.5s, 1 plural forms added
 ✓ de         43 strings translated, 3 files written in 2.1s
@@ -98,11 +98,10 @@ then declare it in `locale-ai.config.json`:
 - `prefix` is required for multi-target models (`convert` lists the model's `>>xx<<` tokens).
 - Project models take priority over built-ins for their language pairs and can be used for pivoting (e.g. `ja → en → tr`).
 
-**Recommended for quality**: the `opus-mt-tc-big-en-*` models (fr, es, it, pt, ar, tr, and `zle` for ru/uk). They are ~3x larger (270–370 MB int8) and ~1.5x slower, and clearly better:
+**Recommended for quality**: the `opus-mt-tc-big-en-*` models (es, it, pt, ar, tr, and `zle` for ru/uk). They are ~3x larger (270–370 MB int8) and ~1.5x slower, and clearly better. `en-fr` already uses one of these as its built-in model (see [Languages](#languages)), hosted at [khabouss/opus-mt-tc-big-en-fr-onnx](https://huggingface.co/khabouss/opus-mt-tc-big-en-fr-onnx) — no conversion needed for it.
 
 | Target | Built-in model, chrF / BLEU | tc-big model, chrF / BLEU |
 |---|---|---|
-| fr | 68.9 / 53.7 | **72.5 / 55.9** |
 | es | 72.9 / 56.9 | **73.6 / 57.4** |
 | it | 73.8 / 54.9 | **75.3 / 58.2** |
 | pt | 68.3 / 44.4 | **70.6 / 47.7** (`>>por<<`, or `>>pob<<` for pt-BR) |
@@ -124,7 +123,7 @@ Measured on a sample app (see `test/fixtures`):
 | Usable — review before shipping | it, ar, ru, zh-CN, zh-TW |
 | Untested | the other pairs in `locale-ai models list` |
 
-With the tc-big models above, it, ar and ru move up a tier and tr becomes available. Not available yet: ja, ko, pl and others; he works with a converted `opus-mt-en-he` (weaker). Moroccan Arabic (Darija): the tc-big en-ar model has a `>>ary<<` token, but its output is close to Modern Standard Arabic — not supported.
+`en-fr` is already on the tc-big model (chrF 72.5 / BLEU 55.9) as a built-in, no config needed. With the other tc-big models above, it, ar and ru move up a tier and tr becomes available. Not available yet: ja, ko, pl and others; he works with a converted `opus-mt-en-he` (weaker). Moroccan Arabic (Darija): the tc-big en-ar model has a `>>ary<<` token, but its output is close to Modern Standard Arabic — not supported.
 
 ## How it works
 
@@ -142,7 +141,7 @@ With the tc-big models above, it, ar and ru move up a tier and tr becomes availa
 ```bash
 npm install
 npm test            # unit tests (no models)
-npm run test:e2e    # real models, downloads ~115 MB once
+npm run test:e2e    # real models, downloads ~345 MB once (en-fr, tc-big)
 node bench/eval.ts --dir <model dir> --lang fr --beams 4 --out out.json   # quality benchmark (see bench/README.md)
 npm run dev -- translate --to fr   # run from source (Node ≥ 22.18 strips types)
 npm run build       # → dist/
@@ -150,7 +149,7 @@ npm run build       # → dist/
 
 ## Next steps
 
-1. **Host the tc-big models** (converted ONNX) on Hugging Face so they can become the built-in defaults instead of a manual conversion.
+1. **Host the tc-big models** (converted ONNX) on Hugging Face so they can become the built-in defaults instead of a manual conversion. `en-fr` is done ([khabouss/opus-mt-tc-big-en-fr-onnx](https://huggingface.co/khabouss/opus-mt-tc-big-en-fr-onnx)); es, it, pt, ar, ru/uk, tr remain.
 2. **Smaller install.** `onnxruntime-node` is ~550 MB unpacked: it ships binaries for every platform plus a 260 MB CUDA provider we never use.
 3. **More languages**: ja, ko, pl have no usable OPUS-MT model yet; a different model family would be needed.
 4. **More formats**: YAML, `.po`, ARB, Apple `.strings`.

package/dist/catalog.js

@@ -8,8 +8,8 @@ const pair = (src, tgt) => ({
 });
 const DIRECT = [
     "af-en", "ar-en", "cs-en", "da-de", "da-en", "de-en", "de-es", "de-fr",
-    "en-af", "en-cs", "en-da", "en-de", "en-es", "en-fi", "en-fr", "en-hi",
-    "en-hu", "en-id", "en-it", "en-nl", "en-ro", "en-ru", "en-sv", "en-uk",
+    "en-af", "en-cs", "en-da", "en-de", "en-es", "en-fi", "en-hi",
+    "en-hu", "en-id", "en-it", "en-nl", "en-ro", "en-sv",
     "en-xh", "es-de", "es-en", "es-fr", "es-it", "es-ru", "et-en", "fi-de",
     "fi-en", "fr-de", "fr-en", "fr-es", "fr-ro", "fr-ru", "hi-en", "hu-en",
     "id-en", "it-en", "it-es", "it-fr", "ja-en", "ko-en", "nl-en", "nl-fr",
@@ -22,10 +22,36 @@ export const CATALOG = [
         const [s, t] = p.split("-");
         return pair(s, t);
     }),
+    // tc-big: larger OPUS-MT checkpoints, clearly better than the defaults (see
+    // bench/README.md). Converted and hosted per the README's "Add a language,
+    // or a better model" section.
+    { ...pair("en", "fr"), repo: "khabouss/opus-mt-tc-big-en-fr-onnx" }, // chrF 72.5 vs 68.9, BLEU 55.9 vs 53.7
+    { ...pair("en", "es"), repo: "khabouss/opus-mt-tc-big-en-es-onnx" }, // chrF 73.6 vs 72.9, BLEU 57.4 vs 56.9
+    { ...pair("en", "it"), repo: "khabouss/opus-mt-tc-big-en-it-onnx" }, // chrF 75.3 vs 73.8, BLEU 58.2 vs 54.9
+    // New: Turkish, not supported by any built-in model before this.
+    { ...pair("en", "tr"), repo: "khabouss/opus-mt-tc-big-en-tr-onnx" }, // chrF 67.4, BLEU 42.1
     {
-        // Note: the model card lists Moroccan Arabic ("ary") but its vocabulary has no
-        // >>ary<< token, so Darija is not supported.
+        // Multi-target: >>por<< (European) / >>pob<< (Brazilian). No individual built-in
+        // existed before; pt was only reachable via the weaker en-ROMANCE model.
+        id: "en-pt",
+        repo: "khabouss/opus-mt-tc-big-en-pt-onnx", // chrF 70.6 vs 68.3, BLEU 47.7 vs 44.4
+        sources: ["en"],
+        targets: ["pt"],
+        prefix: (t) => (t.region === "BR" ? ">>pob<<" : ">>por<<"),
+    },
+    {
+        // Multi-target East Slavic model, replaces the separate en-ru/en-uk built-ins.
+        id: "en-zle",
+        repo: "khabouss/opus-mt-tc-big-en-zle-onnx", // ru: chrF 68.2 vs 65.7, BLEU 48.1 vs 46.0
+        sources: ["en"],
+        targets: ["ru", "uk"],
+        prefix: (t) => (t.lang === "ru" ? ">>rus<<" : ">>ukr<<"),
+    },
+    {
+        // Note: the model card lists Moroccan Arabic ("ary") but its output stays close
+        // to Modern Standard Arabic, so Darija is not treated as supported.
         ...pair("en", "ar"),
+        repo: "khabouss/opus-mt-tc-big-en-ar-onnx", // chrF 49.7 vs 47.8, BLEU 19.9 vs 16.6 (greedy)
         prefix: () => ">>ara<<",
     },
     {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "locale-ai",
-  "version": "0.2.3",
+  "version": "0.3.0",
   "description": "Translate your i18n locale files locally on CPU. Free, offline, no API keys.",
   "author": "Taha KHABOUSS",
   "repository": {