locale-ai 0.6.1 → 0.6.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +10 -9
- package/dist/cli.js +44 -12
- package/dist/engine.js +27 -13
- package/dist/locales.js +79 -11
- package/dist/vue-plural.js +38 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -15,7 +15,7 @@ npx locale-ai translate --from en --to fr,es,de
|
|
|
15
15
|
```
|
|
16
16
|
|
|
17
17
|
```
|
|
18
|
-
locale-ai 0.
|
|
18
|
+
locale-ai 0.6.3
|
|
19
19
|
Found 3 en files in locales/en/
|
|
20
20
|
Models to download (one time, then cached for every project):
|
|
21
21
|
en-fr 345 MB khabouss/opus-mt-tc-big-en-fr-onnx
|
|
@@ -31,7 +31,7 @@ Download 577 MB? (Y/n)
|
|
|
31
31
|
- **Layouts**: `locales/en/*.json` (nested folders too) → `locales/fr/...`, or `messages/en.json` → `messages/fr.json`. `.json`, `.yaml`, `.yml`, `.po` (gettext), `.arb` (Flutter) and `.strings` (Apple) are all supported, mixed freely in the same project. Common folders (`locales`, `src/locales`, `public/locales`, `src/i18n`, `messages`, `lang`, ...) are found automatically, or pass `--input`.
|
|
32
32
|
- **Placeholders are never translated or lost**: `{name}`, `{{count}}`, `%s`, `%1$s`, `%(name)s`, `%{name}`, `${name}`, `:attribute`, `$t(key)`, `@:key`, `<b>…</b>`, `<0>…</0>`, ` `, URLs, e-mails.
|
|
33
33
|
- **ICU MessageFormat**: `{count, plural, one {# file} other {# files}}`, `select`, nesting. Only the text is translated.
|
|
34
|
-
- **i18next plurals**: adds the forms the target language needs (`_few`, `_many` for Arabic, Russian, Polish...).
|
|
34
|
+
- **i18next plurals**: adds the forms the target language needs (`_few`, `_many` for Arabic, Russian, Polish...), so the app doesn't fall back to the English string — each added form copies the `_other` translation verbatim rather than inflecting it, so review and adjust the wording of `_zero`/`_two`/`_few`/`_many` by hand for languages where those forms use a different noun/adjective form.
|
|
35
35
|
- **Incremental**: only missing keys are translated, existing translations are kept, re-runs are no-ops. `--overwrite` re-translates everything.
|
|
36
36
|
- **Structure**: key order, numbers, booleans, arrays, indentation and trailing newline are preserved.
|
|
37
37
|
|
|
@@ -102,24 +102,24 @@ then declare it in `locale-ai.config.json`:
|
|
|
102
102
|
- `prefix` is required for multi-target models (`convert` lists the model's `>>xx<<` tokens).
|
|
103
103
|
- Project models take priority over built-ins for their language pairs and can be used for pivoting (e.g. `ja → en → tr`).
|
|
104
104
|
|
|
105
|
-
**
|
|
105
|
+
**Already built-in, no conversion needed**: `fr`, `es`, `it`, `pt`, `ar` and `tr` all use the larger, clearly-better `opus-mt-tc-big-en-*` checkpoints as their built-in model (~3x larger, 270–370 MB int8, ~1.5x slower than the plain OPUS-MT default); `ru`/`uk` use the tc-big `zle` multi-target checkpoint, and `pl` uses Helsinki-NLP's multi-target Slavic model (no tc-big `en-pl` checkpoint exists). The section above is for adding a language or model that *isn't* already built in.
|
|
106
106
|
|
|
107
|
-
| Target |
|
|
107
|
+
| Target | Previous (plain OPUS-MT), chrF / BLEU | Built-in today (tc-big / better), chrF / BLEU |
|
|
108
108
|
|---|---|---|
|
|
109
|
+
| fr | 68.9 / 53.7 | **72.5 / 55.9** |
|
|
109
110
|
| es | 72.9 / 56.9 | **73.6 / 57.4** |
|
|
110
111
|
| it | 73.8 / 54.9 | **75.3 / 58.2** |
|
|
111
112
|
| pt | 68.3 / 44.4 | **70.6 / 47.7** (`>>por<<`, or `>>pob<<` for pt-BR) |
|
|
112
113
|
| ru | 65.7 / 46.0 | **68.2 / 48.1** (`>>rus<<`) |
|
|
113
114
|
| ar | 47.8 / 16.6 | **49.7 / 19.9** (`>>ara<<`, with `--fast`) |
|
|
114
|
-
| tr | — | **67.4 / 42.1** |
|
|
115
|
+
| tr | — | **67.4 / 42.1** (not supported before) |
|
|
116
|
+
| pl | — | **62.6 / 40.3** (`>>pol<<`, not supported before) |
|
|
115
117
|
|
|
116
118
|
300 Tatoeba test sentences per language, int8, beam 4 unless noted; reproduce with `bench/` (see `bench/README.md`). `opus-mt-tc-big-en-ko` is broken upstream (garbage output even in PyTorch) and is not recommended.
|
|
117
119
|
|
|
118
|
-
Also new: **pl** (chrF 62.6 / BLEU 40.3, `>>pol<<`), hosted at [khabouss/opus-mt-en-sla-onnx](https://huggingface.co/khabouss/opus-mt-en-sla-onnx) — no tc-big `en-pl` checkpoint exists, so this uses Helsinki-NLP's multi-target Slavic model instead.
|
|
119
|
-
|
|
120
120
|
## Languages
|
|
121
121
|
|
|
122
|
-
From English: af, ar, ca, cs, da, de, es, fi, fr, gl, hi, hu, id, it, nl, oc, pl, pt (pt-BR, pt-PT), ro, ru, sv, uk, vi, xh, zh (zh-CN, zh-TW). Other source languages are supported directly or through English (`locale-ai models list --from fr`).
|
|
122
|
+
From English: af, ar, ca, cs, da, de, es, fi, fr, gl, hi, hu, id, it, nl, oc, pl, pt (pt-BR, pt-PT), ro, ru, sv, tr, uk, vi, xh, zh (zh-CN, zh-TW). Other source languages are supported directly or through English (`locale-ai models list --from fr`).
|
|
123
123
|
|
|
124
124
|
Measured on a sample app (see `test/fixtures`):
|
|
125
125
|
|
|
@@ -129,7 +129,7 @@ Measured on a sample app (see `test/fixtures`):
|
|
|
129
129
|
| Usable — review before shipping | it, ar, ru, pl, zh-CN, zh-TW |
|
|
130
130
|
| Untested | the other pairs in `locale-ai models list` |
|
|
131
131
|
|
|
132
|
-
`
|
|
132
|
+
`fr`, `es`, `it`, `pt`, `ar`, `ru`/`uk`, `tr` and `pl` all ship on the tc-big (or better) models above as built-ins already — no config needed. Not available yet: ja, ko and others; he works with a converted `opus-mt-en-he` (weaker). Moroccan Arabic (Darija): the tc-big en-ar model has a `>>ary<<` token, but its output is close to Modern Standard Arabic — not supported.
|
|
133
133
|
|
|
134
134
|
## How it works
|
|
135
135
|
|
|
@@ -157,3 +157,4 @@ npm run build # → dist/
|
|
|
157
157
|
## Next steps
|
|
158
158
|
|
|
159
159
|
1. **More languages**: ja, ko have no usable OPUS-MT model yet; a different model family would be needed.
|
|
160
|
+
2. **Rare batch-dependent mistranslation**: translating a short string (e.g. "© 2026 Acme Inc. All rights reserved.") can occasionally produce an unrelated/garbled result depending on which *other* strings share its translation batch — the same string translates correctly alone or in a different batch. Root cause: the built-in models are dynamically int8-quantized per-tensor, so the quantization scale for a batched activation tensor is computed from the whole batch, not per row; most of the time this is just quantization noise, but occasionally it's enough to tip beam search into a bad output with no warning. A fix would mean re-quantizing the models with static per-channel scales (or giving up CPU-batching speed) — out of scope for a quick patch; flagging here so it's a known, understood limitation rather than a silent one.
|
package/dist/cli.js
CHANGED
|
@@ -49,16 +49,52 @@ const endStatus = () => {
|
|
|
49
49
|
};
|
|
50
50
|
const mb = (bytes) => `${Math.round(bytes / 1e6)} MB`;
|
|
51
51
|
const rel = (p) => path.relative(process.cwd(), p) || ".";
|
|
52
|
+
/**
|
|
53
|
+
* `Number("abc")` is NaN, and NaN silently poisons everything downstream (an
|
|
54
|
+
* all-garbage translation batch, or a raw ONNX crash deep in the model) instead of
|
|
55
|
+
* a clear error at the CLI boundary where the bad flag value actually came from.
|
|
56
|
+
*/
|
|
57
|
+
function positiveInt(raw, flag, fallback) {
|
|
58
|
+
if (raw === undefined) {
|
|
59
|
+
if (fallback === undefined)
|
|
60
|
+
throw new Error(`Missing --${flag}.`);
|
|
61
|
+
return fallback;
|
|
62
|
+
}
|
|
63
|
+
const n = Number(raw);
|
|
64
|
+
if (!Number.isInteger(n) || n < 1)
|
|
65
|
+
throw new Error(`Invalid --${flag} "${raw}": expected a positive integer.`);
|
|
66
|
+
return n;
|
|
67
|
+
}
|
|
52
68
|
function bar(done, total, width = 24) {
|
|
53
69
|
const ratio = total ? Math.min(1, done / total) : 1;
|
|
54
70
|
const filled = Math.round(ratio * width);
|
|
55
71
|
return `[${"█".repeat(filled)}${"░".repeat(width - filled)}] ${Math.round(ratio * 100)}%`;
|
|
56
72
|
}
|
|
73
|
+
/**
|
|
74
|
+
* A long download or translation batch can run for a minute or more. The `\r`-redrawn
|
|
75
|
+
* bar only ever reaches a real terminal, so a non-TTY run (CI logs, piped output) would
|
|
76
|
+
* otherwise print nothing at all between the start and the final result, indistinguishable
|
|
77
|
+
* from a hang. This prints one plain line per 10% there instead.
|
|
78
|
+
*/
|
|
79
|
+
function makeProgress(render) {
|
|
80
|
+
let lastPct = -1;
|
|
81
|
+
return (done, total) => {
|
|
82
|
+
if (isTTY) {
|
|
83
|
+
status(render(done, total));
|
|
84
|
+
return;
|
|
85
|
+
}
|
|
86
|
+
const pct = total ? Math.floor((done / total) * 100) : 100;
|
|
87
|
+
if (pct >= lastPct + 10 || (done >= total && lastPct < 100)) {
|
|
88
|
+
lastPct = done >= total ? 100 : pct;
|
|
89
|
+
log(render(done, total));
|
|
90
|
+
}
|
|
91
|
+
};
|
|
92
|
+
}
|
|
57
93
|
async function confirm(question, yes) {
|
|
58
94
|
if (yes)
|
|
59
95
|
return true;
|
|
60
96
|
if (!process.stdin.isTTY) {
|
|
61
|
-
throw new Error(
|
|
97
|
+
throw new Error(`Cannot ask "${question.trim()}" in a non-interactive shell. Re-run with --yes to allow downloads.`);
|
|
62
98
|
}
|
|
63
99
|
const rl = createInterface({ input: process.stdin, output: process.stderr });
|
|
64
100
|
const answer = (await rl.question(`${question} (Y/n) `)).trim().toLowerCase();
|
|
@@ -86,7 +122,7 @@ async function ensureInstalled(models, yes) {
|
|
|
86
122
|
if (!(await confirm(`Download ${mb(total)}?`, yes)))
|
|
87
123
|
throw new Error("Download declined.");
|
|
88
124
|
for (const p of plans) {
|
|
89
|
-
await install(p, (d, t) =>
|
|
125
|
+
await install(p, makeProgress((d, t) => `Downloading ${p.model.id} ${bar(d, t)} ${mb(d)}/${mb(t)}`));
|
|
90
126
|
endStatus();
|
|
91
127
|
log(`✓ Installed ${p.model.id}`);
|
|
92
128
|
}
|
|
@@ -105,10 +141,6 @@ async function cmdTranslate(argv) {
|
|
|
105
141
|
beams: { type: "string" },
|
|
106
142
|
report: { type: "string" },
|
|
107
143
|
config: { type: "string" },
|
|
108
|
-
out: { type: "string" },
|
|
109
|
-
source: { type: "string" },
|
|
110
|
-
target: { type: "string" },
|
|
111
|
-
python: { type: "string" },
|
|
112
144
|
yes: { type: "boolean", short: "y", default: false },
|
|
113
145
|
},
|
|
114
146
|
});
|
|
@@ -116,13 +148,13 @@ async function cmdTranslate(argv) {
|
|
|
116
148
|
throw new Error("Missing --to. Example: locale-ai translate --from en --to fr,es");
|
|
117
149
|
const from = parseLocale(values.from);
|
|
118
150
|
const targets = values.to.split(",").map((t) => t.trim()).filter(Boolean).map(parseLocale);
|
|
119
|
-
const batchSize =
|
|
120
|
-
const beams =
|
|
151
|
+
const batchSize = positiveInt(values["batch-size"], "batch-size");
|
|
152
|
+
const beams = positiveInt(values.beams, "beams", values.fast ? 1 : 4);
|
|
121
153
|
const config = loadConfig(process.cwd(), values.config);
|
|
122
154
|
const catalog = catalogFor(config);
|
|
123
155
|
const protect = protectRegex(config);
|
|
124
156
|
const layout = detectLayout(process.cwd(), from.tag, values.input);
|
|
125
|
-
const sources = sourceFiles(layout, from.tag).map((f) => ({ file: f, doc: readLocaleDoc(f) }));
|
|
157
|
+
const sources = sourceFiles(layout, from.tag).map((f) => ({ file: f, doc: readLocaleDoc(f, from.tag) }));
|
|
126
158
|
log(`locale-ai ${VERSION}`);
|
|
127
159
|
log(`Found ${sources.length} ${from.tag} file${sources.length === 1 ? "" : "s"} in ${rel(layout.root)}/` +
|
|
128
160
|
(layout.kind === "dir" ? `${from.tag}/` : ""));
|
|
@@ -139,7 +171,7 @@ async function cmdTranslate(argv) {
|
|
|
139
171
|
}
|
|
140
172
|
const files = sources.map(({ file, doc }) => {
|
|
141
173
|
const target = targetFile(layout, from.tag, locale.tag, file);
|
|
142
|
-
const existing = fs.existsSync(target) ? readLocaleDoc(target).data : undefined;
|
|
174
|
+
const existing = fs.existsSync(target) ? readLocaleDoc(target, locale.tag).data : undefined;
|
|
143
175
|
return { source: file, target, doc, existing, pending: pendingStrings(doc.data, existing, values.overwrite) };
|
|
144
176
|
});
|
|
145
177
|
const unique = new Set(files.flatMap((f) => f.pending)).size;
|
|
@@ -182,7 +214,7 @@ async function cmdTranslate(argv) {
|
|
|
182
214
|
batchSize,
|
|
183
215
|
protect,
|
|
184
216
|
lookup: glossaryLookup(from, job.locale, config),
|
|
185
|
-
onProgress: (d, t) =>
|
|
217
|
+
onProgress: makeProgress((d, t) => `${job.locale.tag}: translating ${bar(d, t)} ${d}/${t}`),
|
|
186
218
|
});
|
|
187
219
|
endStatus();
|
|
188
220
|
translations = result.translations;
|
|
@@ -201,8 +233,8 @@ async function cmdTranslate(argv) {
|
|
|
201
233
|
const built = buildTarget(f.doc.data, f.existing, (s) => translations.get(s) ?? s, values.overwrite);
|
|
202
234
|
const { doc, added } = addPluralForms(built, intlCode(job.locale));
|
|
203
235
|
pluralsAdded += added;
|
|
204
|
-
const text = f.doc.serialize(doc, job.locale.tag);
|
|
205
236
|
const before = fs.existsSync(f.target) ? fs.readFileSync(f.target, "utf8") : undefined;
|
|
237
|
+
const text = f.doc.serialize(doc, job.locale.tag, before);
|
|
206
238
|
if (text !== before) {
|
|
207
239
|
fs.mkdirSync(path.dirname(f.target), { recursive: true });
|
|
208
240
|
fs.writeFileSync(f.target, text);
|
package/dist/engine.js
CHANGED
|
@@ -1,4 +1,5 @@
|
|
|
1
1
|
import { extractIcu } from "./icu.js";
|
|
2
|
+
import { extractPipePlural } from "./vue-plural.js";
|
|
2
3
|
import { fixTagSpacing, hasTranslatableText, mapOutsideTokens, mask, numericStyleSafe, splitOnTokens, stripMarkers, unmask, } from "./placeholders.js";
|
|
3
4
|
import { splitSentences } from "./segment.js";
|
|
4
5
|
const letters = (s) => stripMarkers(s).replace(/[^\p{L}]/gu, "");
|
|
@@ -136,21 +137,34 @@ async function attempt(units, style, hops, opts, progress, issues) {
|
|
|
136
137
|
}
|
|
137
138
|
let ok = true;
|
|
138
139
|
const found = [];
|
|
139
|
-
const
|
|
140
|
-
.map((p) => {
|
|
140
|
+
const parts = await Promise.all(pieces.map(async (p) => {
|
|
141
141
|
// No spaces between sentences in Chinese, Japanese, Thai...
|
|
142
142
|
if (!p.translate)
|
|
143
143
|
return NO_SPACES.test(opts.targetLang) ? p.value.replace(/^[ \t]+$/, "") : p.value;
|
|
144
|
-
|
|
145
|
-
|
|
144
|
+
let out = results.get(prepare(p.value)) ?? "";
|
|
145
|
+
const suspicionKind = (o) => o.length > 3 * p.value.length + 20 ? "suspicious-length" : /(\b\p{L}+\b)(?:\W+\1\b){3,}/u.test(o) ? "repetition" : undefined;
|
|
146
|
+
let degenerate = isDegenerate(p.value, out);
|
|
147
|
+
let kind = degenerate ? undefined : suspicionKind(out);
|
|
148
|
+
// Quantizing a batched tensor computes scale factors from the whole batch, so one
|
|
149
|
+
// row's output can depend on what else shared its batch. Retrying the same input
|
|
150
|
+
// alone (batch of 1) often clears a hallucination that batch composition caused.
|
|
151
|
+
if (degenerate || kind !== undefined) {
|
|
152
|
+
const retry = await runModels(new Set([prepare(p.value)]), hops, opts, progress);
|
|
153
|
+
const retryOut = retry.get(prepare(p.value)) ?? out;
|
|
154
|
+
const retryDegenerate = isDegenerate(p.value, retryOut);
|
|
155
|
+
if (!retryDegenerate && suspicionKind(retryOut) === undefined) {
|
|
156
|
+
out = retryOut;
|
|
157
|
+
degenerate = false;
|
|
158
|
+
kind = undefined;
|
|
159
|
+
}
|
|
160
|
+
}
|
|
161
|
+
if (degenerate)
|
|
146
162
|
ok = false;
|
|
147
|
-
else if (
|
|
148
|
-
found.push({ source: unit.owner, kind
|
|
149
|
-
else if (/(\b\p{L}+\b)(?:\W+\1\b){3,}/u.test(out))
|
|
150
|
-
found.push({ source: unit.owner, kind: "repetition", output: out });
|
|
163
|
+
else if (kind)
|
|
164
|
+
found.push({ source: unit.owner, kind, output: out });
|
|
151
165
|
return postprocess(p.value, out, opts.targetLang);
|
|
152
|
-
})
|
|
153
|
-
|
|
166
|
+
}));
|
|
167
|
+
const joined = parts.join("");
|
|
154
168
|
const restored = ok ? unmask(joined, masked) : undefined;
|
|
155
169
|
if (restored === undefined)
|
|
156
170
|
failed.push(unit);
|
|
@@ -207,8 +221,8 @@ export async function translateStrings(strings, hops, opts) {
|
|
|
207
221
|
remaining.push(s);
|
|
208
222
|
}
|
|
209
223
|
const plans = remaining.map((s) => {
|
|
210
|
-
const
|
|
211
|
-
return { source: s,
|
|
224
|
+
const split = extractIcu(s) ?? extractPipePlural(s);
|
|
225
|
+
return { source: s, split, units: (split?.units ?? [s]).map((text) => ({ text, owner: s })) };
|
|
212
226
|
});
|
|
213
227
|
const units = plans.flatMap((p) => p.units);
|
|
214
228
|
let pending = units;
|
|
@@ -220,7 +234,7 @@ export async function translateStrings(strings, hops, opts) {
|
|
|
220
234
|
await splitFallback(pending, hops, opts, progress, issues);
|
|
221
235
|
for (const p of plans) {
|
|
222
236
|
const parts = p.units.map((u) => u.result ?? u.text);
|
|
223
|
-
translations.set(p.source, p.
|
|
237
|
+
translations.set(p.source, p.split ? p.split.rebuild(parts) : parts[0]);
|
|
224
238
|
}
|
|
225
239
|
return { translations, issues };
|
|
226
240
|
}
|
package/dist/locales.js
CHANGED
|
@@ -51,7 +51,7 @@ export function detectLayout(cwd, from, input) {
|
|
|
51
51
|
return { kind: "file", root: path.dirname(abs), ext: directExt };
|
|
52
52
|
const l = layoutAt(abs, from);
|
|
53
53
|
if (!l)
|
|
54
|
-
throw new Error(`No "${from}" locale found in ${input} (looked for ${from}
|
|
54
|
+
throw new Error(`No "${from}" locale found in ${input} (looked for ${from}/* and ${from}${EXTENSIONS.join("/")}).`);
|
|
55
55
|
return l;
|
|
56
56
|
}
|
|
57
57
|
for (const c of CANDIDATES) {
|
|
@@ -59,7 +59,7 @@ export function detectLayout(cwd, from, input) {
|
|
|
59
59
|
if (l)
|
|
60
60
|
return l;
|
|
61
61
|
}
|
|
62
|
-
throw new Error(`Could not find "${from}" locale files. Use --input <dir> to point at the folder containing ${from}/ or ${from}.
|
|
62
|
+
throw new Error(`Could not find "${from}" locale files. Use --input <dir> to point at the folder containing ${from}/ or ${from}${EXTENSIONS.join("/")}.`);
|
|
63
63
|
}
|
|
64
64
|
function listLocaleFiles(dir) {
|
|
65
65
|
const out = [];
|
|
@@ -96,17 +96,28 @@ function readJsonDoc(file, text) {
|
|
|
96
96
|
const keepNewline = finalNewline(text);
|
|
97
97
|
return { data, serialize: (d) => withNewline(JSON.stringify(d, null, indent), keepNewline) };
|
|
98
98
|
}
|
|
99
|
-
|
|
100
|
-
|
|
99
|
+
/**
|
|
100
|
+
* Rails and some other frameworks nest an entire YAML locale file under a single
|
|
101
|
+
* top-level key matching the locale tag itself (`en:\n hello: Hi`), so the file is
|
|
102
|
+
* valid wherever it's loaded from. Detect that wrapper and unwrap it before handing
|
|
103
|
+
* `data` to the rest of the pipeline (which otherwise treats "en" as just another
|
|
104
|
+
* translatable key), then re-wrap under the target's own tag on serialize.
|
|
105
|
+
*/
|
|
106
|
+
function readYamlDoc(file, text, tag) {
|
|
107
|
+
let raw;
|
|
101
108
|
try {
|
|
102
|
-
|
|
109
|
+
raw = parseYaml(text) ?? {};
|
|
103
110
|
}
|
|
104
111
|
catch (e) {
|
|
105
112
|
throw new Error(`Invalid YAML in ${file}: ${e.message}`);
|
|
106
113
|
}
|
|
107
114
|
const indent = Number(/\n( +)\S/.exec(text)?.[1].length) || 2;
|
|
108
115
|
const keepNewline = finalNewline(text);
|
|
109
|
-
|
|
116
|
+
const keys = isObj(raw) ? Object.keys(raw) : [];
|
|
117
|
+
const wrapped = keys.length === 1 && keys[0] === tag;
|
|
118
|
+
const data = wrapped ? raw[tag] : raw;
|
|
119
|
+
const serialize = (d, targetTag) => withNewline(stringifyYaml(wrapped ? { [targetTag]: d } : d, { indent }), keepNewline);
|
|
120
|
+
return { data, serialize };
|
|
110
121
|
}
|
|
111
122
|
/** Apple .strings: `"key" = "value";`, optionally preceded by a `/* ... *\/` comment. */
|
|
112
123
|
function unescapeC(s) {
|
|
@@ -116,7 +127,8 @@ function escapeC(s) {
|
|
|
116
127
|
return s.replace(/[\\"\n\t\r]/g, (c) => ({ "\\": "\\\\", '"': '\\"', "\n": "\\n", "\t": "\\t", "\r": "\\r" })[c]);
|
|
117
128
|
}
|
|
118
129
|
function readStringsDoc(file, text) {
|
|
119
|
-
|
|
130
|
+
// A key's preceding comment is either a /* ... */ block, or one or more consecutive // lines.
|
|
131
|
+
const PAIR = /(?:((?:\/\*[\s\S]*?\*\/)|(?:\/\/[^\r\n]*(?:\r?\n[ \t]*\/\/[^\r\n]*)*))[ \t]*\r?\n+)?[ \t]*"((?:\\.|[^"\\])*)"\s*=\s*"((?:\\.|[^"\\])*)"\s*;/g;
|
|
120
132
|
const data = newObj();
|
|
121
133
|
const entries = [];
|
|
122
134
|
for (const m of text.matchAll(PAIR)) {
|
|
@@ -182,6 +194,54 @@ function readArbDoc(file, text) {
|
|
|
182
194
|
return { data, serialize };
|
|
183
195
|
}
|
|
184
196
|
const poKey = (e) => (e.msgctxt ?? "") + "\u0004" + e.msgid;
|
|
197
|
+
/**
|
|
198
|
+
* gettext's well-established Plural-Forms formulas, keyed by language. Copying the
|
|
199
|
+
* source header's line verbatim (as a naive patch does) is wrong whenever the target
|
|
200
|
+
* needs a different count — Arabic needs 6 categories, an English source's line says
|
|
201
|
+
* 2, and anything consuming the .po indexes msgstr[] short or drops forms silently.
|
|
202
|
+
*/
|
|
203
|
+
const PLURAL_FORMS = {
|
|
204
|
+
ja: "nplurals=1; plural=0;",
|
|
205
|
+
zh: "nplurals=1; plural=0;",
|
|
206
|
+
ko: "nplurals=1; plural=0;",
|
|
207
|
+
th: "nplurals=1; plural=0;",
|
|
208
|
+
vi: "nplurals=1; plural=0;",
|
|
209
|
+
id: "nplurals=1; plural=0;",
|
|
210
|
+
ms: "nplurals=1; plural=0;",
|
|
211
|
+
fr: "nplurals=2; plural=(n > 1);",
|
|
212
|
+
pt: "nplurals=2; plural=(n != 1);",
|
|
213
|
+
cs: "nplurals=3; plural=(n==1) ? 0 : (n>=2 && n<=4) ? 1 : 2;",
|
|
214
|
+
sk: "nplurals=3; plural=(n==1) ? 0 : (n>=2 && n<=4) ? 1 : 2;",
|
|
215
|
+
pl: "nplurals=3; plural=(n==1 ? 0 : n%10>=2 && n%10<=4 && (n%100<12 || n%100>14) ? 1 : 2);",
|
|
216
|
+
ru: "nplurals=3; plural=(n%10==1 && n%100!=11 ? 0 : n%10>=2 && n%10<=4 && (n%100<12 || n%100>14) ? 1 : 2);",
|
|
217
|
+
uk: "nplurals=3; plural=(n%10==1 && n%100!=11 ? 0 : n%10>=2 && n%10<=4 && (n%100<12 || n%100>14) ? 1 : 2);",
|
|
218
|
+
ro: "nplurals=3; plural=(n==1 ? 0 : (n==0 || (n%100 > 0 && n%100 < 20)) ? 1 : 2);",
|
|
219
|
+
ar: "nplurals=6; plural=(n==0 ? 0 : n==1 ? 1 : n==2 ? 2 : n%100>=3 && n%100<=10 ? 3 : n%100>=11 && n%100<=99 ? 4 : 5);",
|
|
220
|
+
};
|
|
221
|
+
/**
|
|
222
|
+
* Falls back to CLDR's plural category count (via Intl.PluralRules) for a language
|
|
223
|
+
* not in the table above: not necessarily linguistically exact (it buckets purely by
|
|
224
|
+
* small integer value), but it keeps nplurals correct, which is the part that breaks
|
|
225
|
+
* gettext consumers when wrong.
|
|
226
|
+
*/
|
|
227
|
+
function pluralFormsHeader(lang) {
|
|
228
|
+
const known = PLURAL_FORMS[lang];
|
|
229
|
+
if (known)
|
|
230
|
+
return known;
|
|
231
|
+
let n;
|
|
232
|
+
try {
|
|
233
|
+
n = new Intl.PluralRules(lang).resolvedOptions().pluralCategories.length;
|
|
234
|
+
}
|
|
235
|
+
catch {
|
|
236
|
+
n = 2;
|
|
237
|
+
}
|
|
238
|
+
if (n <= 1)
|
|
239
|
+
return "nplurals=1; plural=0;";
|
|
240
|
+
if (n === 2)
|
|
241
|
+
return "nplurals=2; plural=(n != 1);";
|
|
242
|
+
const branches = Array.from({ length: n - 1 }, (_, i) => `n==${i + 1} ? ${i} :`).join(" ");
|
|
243
|
+
return `nplurals=${n}; plural=(${branches} ${n - 1});`;
|
|
244
|
+
}
|
|
185
245
|
function parsePo(text) {
|
|
186
246
|
const entries = [];
|
|
187
247
|
let leading = [];
|
|
@@ -265,17 +325,24 @@ function readPoDoc(file, text) {
|
|
|
265
325
|
if (!e.isHeader && e.msgidPlural === undefined)
|
|
266
326
|
data[poKey(e)] = e.msgstr;
|
|
267
327
|
const keepNewline = finalNewline(text);
|
|
268
|
-
const serialize = (d, targetTag) => {
|
|
328
|
+
const serialize = (d, targetTag, existingText) => {
|
|
269
329
|
const obj = isObj(d) ? d : newObj();
|
|
270
330
|
const seen = new Set();
|
|
271
331
|
const blocks = leading.length ? [leading.join("\n")] : [];
|
|
332
|
+
// A previous run may have left a header a translator then hand-fixed (a corrected
|
|
333
|
+
// Plural-Forms, an added Last-Translator, ...). Keep it as-is rather than
|
|
334
|
+
// regenerating from the source every time; only a brand-new target file gets a
|
|
335
|
+
// freshly computed header.
|
|
336
|
+
const existingHeader = existingText && parsePo(existingText).entries.find((e) => e.isHeader);
|
|
272
337
|
for (const e of entries) {
|
|
273
338
|
const lines = [...e.comments];
|
|
274
339
|
if (e.msgctxt !== undefined)
|
|
275
340
|
lines.push(`msgctxt "${escapeC(e.msgctxt)}"`);
|
|
276
341
|
if (e.isHeader) {
|
|
277
342
|
lines.push(`msgid ""`);
|
|
278
|
-
const header =
|
|
343
|
+
const header = existingHeader
|
|
344
|
+
? existingHeader.msgstr
|
|
345
|
+
: e.msgstr.replace(/^Language:.*$/m, `Language: ${targetTag}`).replace(/^Plural-Forms:.*$/m, `Plural-Forms: ${pluralFormsHeader(targetTag)}`);
|
|
279
346
|
lines.push(`msgstr "${escapeC(header)}"`);
|
|
280
347
|
}
|
|
281
348
|
else if (e.msgidPlural !== undefined) {
|
|
@@ -304,11 +371,12 @@ function readPoDoc(file, text) {
|
|
|
304
371
|
};
|
|
305
372
|
return { data, serialize };
|
|
306
373
|
}
|
|
307
|
-
|
|
374
|
+
/** `tag` is the locale this file is for (the source tag, or an existing target's own tag). */
|
|
375
|
+
export function readLocaleDoc(file, tag) {
|
|
308
376
|
const text = fs.readFileSync(file, "utf8").replace(/^/, "");
|
|
309
377
|
const ext = path.extname(file).toLowerCase();
|
|
310
378
|
if (ext === ".yaml" || ext === ".yml")
|
|
311
|
-
return readYamlDoc(file, text);
|
|
379
|
+
return readYamlDoc(file, text, tag);
|
|
312
380
|
if (ext === ".po")
|
|
313
381
|
return readPoDoc(file, text);
|
|
314
382
|
if (ext === ".arb")
|
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* vue-i18n's plural syntax: forms separated by a literal `|`, chosen by count at
|
|
3
|
+
* runtime (`"no items | one item | {count} items"`). The `|` carries no meaning to
|
|
4
|
+
* a translation model — it reads as stray punctuation and gets dropped or garbles
|
|
5
|
+
* the sentence around it — so each form is translated on its own and the `|`
|
|
6
|
+
* separators are kept untouched, exactly where they were.
|
|
7
|
+
*/
|
|
8
|
+
/** Returns undefined when the string has no top-level `|` (ignoring one inside `{...}`). */
|
|
9
|
+
export function extractPipePlural(s) {
|
|
10
|
+
const segments = [];
|
|
11
|
+
let depth = 0;
|
|
12
|
+
let start = 0;
|
|
13
|
+
const push = (end) => {
|
|
14
|
+
const seg = s.slice(start, end);
|
|
15
|
+
const m = /^(\s*)([\s\S]*?)(\s*)$/.exec(seg);
|
|
16
|
+
segments.push({ lead: m[1], core: m[2], trail: m[3] });
|
|
17
|
+
};
|
|
18
|
+
for (let i = 0; i < s.length; i++) {
|
|
19
|
+
const c = s[i];
|
|
20
|
+
if (c === "{")
|
|
21
|
+
depth++;
|
|
22
|
+
else if (c === "}")
|
|
23
|
+
depth = Math.max(0, depth - 1);
|
|
24
|
+
else if (c === "|" && depth === 0) {
|
|
25
|
+
push(i);
|
|
26
|
+
start = i + 1;
|
|
27
|
+
}
|
|
28
|
+
}
|
|
29
|
+
if (start === 0)
|
|
30
|
+
return undefined; // no top-level '|'
|
|
31
|
+
push(s.length);
|
|
32
|
+
if (segments.some((p) => !p.core))
|
|
33
|
+
return undefined; // "a||b" or a leading/trailing '|': not a plural list
|
|
34
|
+
return {
|
|
35
|
+
units: segments.map((p) => p.core),
|
|
36
|
+
rebuild: (translations) => segments.map((p, i) => p.lead + translations[i] + p.trail).join("|"),
|
|
37
|
+
};
|
|
38
|
+
}
|