locale-ai 0.6.11 → 0.6.13
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +3 -3
- package/dist/cli.js +2 -0
- package/dist/placeholders.js +2 -2
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -49,7 +49,7 @@ Download 577 MB? (Y/n)
|
|
|
49
49
|
## Commands
|
|
50
50
|
|
|
51
51
|
```bash
|
|
52
|
-
locale-ai translate --to fr,pt-BR [--from en] [--input dir] [--dry-run] [--overwrite] [--fast] [--report file] [-y]
|
|
52
|
+
locale-ai translate --to fr,pt-BR [--from en] [--input dir] [--dry-run] [--overwrite] [--fast] [--beams n] [--batch-size n] [--report file] [-y]
|
|
53
53
|
locale-ai check [--from en] [--input dir] # CI: fails if a translated locale has missing keys or placeholder mismatches
|
|
54
54
|
# (also warns, without failing, about strings left identical to the source)
|
|
55
55
|
locale-ai models list [--from en] # supported language pairs
|
|
@@ -59,7 +59,7 @@ locale-ai models remove en-de
|
|
|
59
59
|
locale-ai models convert <hf-marian-model> --out <dir> # bring your own model (see below)
|
|
60
60
|
```
|
|
61
61
|
|
|
62
|
-
`translate` uses beam search (4 beams) by default. `--fast` uses greedy decoding: 2–3x faster, about 0.5 chrF lower.
|
|
62
|
+
`translate` uses beam search (4 beams) by default; `--beams n` overrides the width directly. `--fast` uses greedy decoding (equivalent to `--beams 1`): 2–3x faster, about 0.5 chrF lower. `--batch-size n` (default 32) controls how many strings go through the model per batch — lower it if you're hitting memory limits, raise it for a bit more throughput on a machine with room to spare.
|
|
63
63
|
|
|
64
64
|
Models are cached once per machine in `~/.cache/locale-ai` (`LOCALE_AI_CACHE` to change). In CI, pass `--yes`.
|
|
65
65
|
|
|
@@ -140,7 +140,7 @@ Measured on a sample app (see `test/fixtures`):
|
|
|
140
140
|
|
|
141
141
|
## How it works
|
|
142
142
|
|
|
143
|
-
- **Models**: [OPUS-MT](https://huggingface.co/Helsinki-NLP) (MarianMT) in ONNX, int8-quantized (int8 vs fp32 measured equal: en-fr chrF 68.9 vs 69.0, en-ar 47.8 vs 47.8). Downloads are pinned to a commit and verified (sha256 / git blob hash).
|
|
143
|
+
- **Models**: [OPUS-MT](https://huggingface.co/Helsinki-NLP) (MarianMT) in ONNX, int8-quantized (int8 vs fp32 measured equal: en-fr chrF 68.9 vs 69.0, en-ar 47.8 vs 47.8). Downloads are pinned to a commit and verified (sha256 / git blob hash). Every model is OPUS-MT by Helsinki-NLP (Tiedemann & Thottingal, 2020 — cite this if you use the models); each is licensed individually (most CC-BY 4.0, some Apache 2.0 — check the model's own card). The larger `tc-big` and multi-target checkpoints are re-hosted as ONNX conversions under [khabouss/*](https://huggingface.co/khabouss), each model card carrying the original's license, crediting and citing it, and listing only the ONNX conversion and quantization as locale-ai's own addition.
|
|
144
144
|
- **Own runtime** on [onnxruntime](https://onnxruntime.ai) (`src/marian.ts`), no transformers.js:
|
|
145
145
|
- a SentencePiece Unigram tokenizer including the model's precompiled normalisation map, identical to Python `sentencepiece` on 3,577/3,577 test sentences across 5 models (`src/spm.ts`);
|
|
146
146
|
- real beam search with Hugging Face semantics, identical to `transformers` PyTorch beam search on 599/600 sentences; log-softmax + top-k run natively in a tiny embedded ONNX graph (`scripts/make-topk-graph.py`);
|
package/dist/cli.js
CHANGED
|
@@ -37,6 +37,8 @@ Translate options:
|
|
|
37
37
|
--overwrite Re-translate strings that already have a translation
|
|
38
38
|
--dry-run Show what would be translated without downloading or writing
|
|
39
39
|
--fast Greedy decoding: ~2x faster, slightly lower quality
|
|
40
|
+
--beams <n> Beam width for decoding (default: 4, or 1 with --fast)
|
|
41
|
+
--batch-size <n> Strings translated per model batch (default: 32)
|
|
40
42
|
--report <file> Write a JSON report of strings that need review
|
|
41
43
|
--fuzzy For .po/.pot: flag every entry this run (re)translated as "#, fuzzy"
|
|
42
44
|
--config <file> Project config (default: ./${CONFIG_FILE} if present)
|
package/dist/placeholders.js
CHANGED
|
@@ -24,7 +24,7 @@ const SOURCES = [
|
|
|
24
24
|
/\$\{[^{}]*\}/.source, // ${name}
|
|
25
25
|
/%\{[^{}]+\}/.source, // %{name} ruby / polyglot
|
|
26
26
|
/%\([\w.]+\)[sdifr]/.source, // %(name)s python
|
|
27
|
-
/%(?:\d+\$)?[-+0#]*\d*(?:\.\d+)?[sdifuxXeEgGcpo@]/.source, // %s %d %1$s %.2f %@
|
|
27
|
+
/%(?:\d+\$)?[-+0#]*\d*(?:\.\d+)?[sdifuxXeEgGcpo@](?![A-Za-z])/.source, // %s %d %1$s %.2f %@ (not "100%off")
|
|
28
28
|
/@(?:\.\w+)?:\(?[\w.-]+\)?/.source, // @:key @.lower:key vue-i18n linked messages
|
|
29
29
|
/\{\s*[\w.-]+\s*(?:,[^{}]*)?\}/.source, // {name} {0} {count, number} ICU simple args
|
|
30
30
|
/<\/?[A-Za-z][\w.:-]*(?:\s[^<>]*)?\/?>/.source, // <b> </b> <a href="..."> <Trans/>
|
|
@@ -33,7 +33,7 @@ const SOURCES = [
|
|
|
33
33
|
/https?:\/\/[^\s<>"']*[^\s<>"'.,;:!?)\]]/.source, // URLs
|
|
34
34
|
/[\w.+-]+@[\w-]+(?:\.[\w-]+)+/.source, // e-mail addresses
|
|
35
35
|
/(?<=^|[\s(])(?::[a-zA-Z_]\w*)/.source, // :attribute laravel
|
|
36
|
-
/\$[A-Za-z_]\w*/.source, // $name
|
|
36
|
+
/\$(?![A-Z]{2,}(?![a-zA-Z0-9_]))[A-Za-z_]\w*/.source, // $name (not "$USD", "$CAD" — currency codes)
|
|
37
37
|
];
|
|
38
38
|
const PLACEHOLDER_SRC = SOURCES.map((s) => `(?:${s})`).join("|");
|
|
39
39
|
const PLACEHOLDER_RE = new RegExp(PLACEHOLDER_SRC, "gu");
|