champollion 0.3.3 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +52 -37
- package/bin/cli.js +53 -5
- package/index.js +63 -2
- package/lib/api-key.js +17 -4
- package/lib/autofix.js +83 -36
- package/lib/bridge/method_bridge.py +15 -3
- package/lib/cards/reader.js +51 -3
- package/lib/cards/remote.js +15 -0
- package/lib/cards/search-names.js +178 -0
- package/lib/command-help.js +289 -88
- package/lib/commands/audit.js +10 -3
- package/lib/commands/card.js +583 -226
- package/lib/commands/doctor.js +54 -18
- package/lib/commands/help.js +37 -32
- package/lib/commands/init.js +1689 -87
- package/lib/commands/integrity.js +127 -40
- package/lib/commands/leaderboard.js +187 -67
- package/lib/commands/models.js +9 -2
- package/lib/commands/provenance.js +7 -2
- package/lib/commands/recommend.js +43 -14
- package/lib/commands/register-corpus.js +649 -130
- package/lib/commands/seal-corpus.js +1 -1
- package/lib/commands/status.js +564 -27
- package/lib/commands/submit.js +17 -12
- package/lib/commands/sync.js +31 -7
- package/lib/commands/tm.js +16 -10
- package/lib/commands/verify.js +27 -3
- package/lib/commands/wrap.js +63 -5
- package/lib/commands/xliff.js +135 -64
- package/lib/commercial-eligibility.js +1 -1
- package/lib/config.js +196 -14
- package/lib/content-estimate.js +96 -0
- package/lib/content-refusals.js +270 -0
- package/lib/content-review.js +372 -0
- package/lib/content-sync.js +1127 -344
- package/lib/content.js +94 -7
- package/lib/corpus-registration.mjs +197 -38
- package/lib/cost-label.js +29 -0
- package/lib/cost-report.js +726 -78
- package/lib/diff.js +38 -4
- package/lib/docusaurus-sync.js +965 -253
- package/lib/edit-distance.js +31 -0
- package/lib/fallback.js +964 -0
- package/lib/file-scope.js +106 -0
- package/lib/flatten.js +80 -3
- package/lib/flutter-locales.js +124 -0
- package/lib/format.js +266 -12
- package/lib/hash.js +146 -21
- package/lib/icu-structure.js +929 -0
- package/lib/integrity.js +223 -75
- package/lib/language-pair.js +157 -0
- package/lib/lint.js +78 -16
- package/lib/local-only-marks.js +106 -0
- package/lib/locale-layout.js +1103 -0
- package/lib/locale-state.js +571 -0
- package/lib/methods/anthropic.js +5 -0
- package/lib/methods/apertium.js +6 -3
- package/lib/methods/api.js +138 -25
- package/lib/methods/base.js +17 -0
- package/lib/methods/coaching-data.js +153 -0
- package/lib/methods/content-separator.js +43 -0
- package/lib/methods/deepl.js +1 -1
- package/lib/methods/direct-llm.js +252 -103
- package/lib/methods/external.js +146 -63
- package/lib/methods/gemini.js +1 -0
- package/lib/methods/google-translate.js +1 -0
- package/lib/methods/http-utils.js +41 -0
- package/lib/methods/libretranslate.js +7 -2
- package/lib/methods/llm-coached.js +68 -128
- package/lib/methods/llm.js +80 -31
- package/lib/methods/local.js +93 -10
- package/lib/methods/microsoft-translator.js +1 -2
- package/lib/methods/openai.js +4 -2
- package/lib/methods/openrouter-client.js +20 -19
- package/lib/methods/openrouter-pricing.js +150 -13
- package/lib/methods/prompt-methods.js +20 -0
- package/lib/methods/provider-pricing.js +42 -1
- package/lib/methods/request-capture.js +104 -0
- package/lib/methods/tilde.js +1 -1
- package/lib/methods/translated.js +1 -2
- package/lib/missing-key.js +93 -0
- package/lib/models.js +11 -0
- package/lib/name-rules.js +32 -0
- package/lib/named-keys.js +172 -0
- package/lib/no-translate.js +4 -3
- package/lib/output.js +160 -19
- package/lib/pairs.js +586 -30
- package/lib/placeholders.js +394 -0
- package/lib/plugins.js +8 -0
- package/lib/plural-gap-redo.js +109 -0
- package/lib/plurals.js +323 -0
- package/lib/po.js +1187 -0
- package/lib/public-catalogue.js +74 -0
- package/lib/recommend.js +527 -32
- package/lib/redo.js +95 -0
- package/lib/refusal-category.js +44 -0
- package/lib/registers.js +255 -11
- package/lib/repair-script.js +20 -13
- package/lib/scripts.js +193 -106
- package/lib/seal.mjs +6 -5
- package/lib/sealed-qualifier.mjs +2 -2
- package/lib/segment.js +2 -1
- package/lib/seo.js +19 -9
- package/lib/serve.js +43 -6
- package/lib/shared-output-seed.js +164 -0
- package/lib/source-contexts.js +39 -0
- package/lib/submit.mjs +57 -5
- package/lib/sync.js +2923 -474
- package/lib/terminology.js +13 -4
- package/lib/tm-evict.js +179 -0
- package/lib/tm-seed.js +5 -2
- package/lib/tm.js +818 -36
- package/lib/translate-pair.js +639 -34
- package/lib/translate.js +78 -5
- package/lib/types.js +22 -3
- package/lib/validate.js +880 -17
- package/lib/verify.js +1296 -104
- package/lib/watch.js +32 -13
- package/lib/xliff.js +44 -3
- package/package.json +3 -2
- package/shared/CORPORA-CARDS.md +2 -0
- package/shared/DATA-SOVEREIGNTY.md +19 -20
- package/shared/LANGUAGE-CARD-FIELDS.md +1 -1
- package/shared/cards-fallback.json +1 -1
- package/shared/catalogue/card-config.json +1 -1
- package/shared/curated-orthography-conventions.json +26 -8
- package/shared/docent/faq.en.json +14 -16
- package/shared/docent/system-prompt.md +17 -19
- package/shared/explainers/tc-features.json +15 -15
- package/shared/gettext-plural-forms.json +45 -0
- package/shared/human-services.json +1 -1
- package/shared/method-registry.json +2 -0
- package/shared/metric-registry.json +96 -18
- package/shared/schemas/champollion-plugin.schema.json +4 -0
- package/shared/schemas/corpora-card.schema.json +20 -10
- package/shared/schemas/human-services.schema.json +2 -2
- package/shared/schemas/language-card.schema.json +1 -1
- package/shared/schemas/method-card.schema.json +1 -1
- package/shared/schemas/method-index-record.schema.json +67 -0
- package/shared/schemas/method-registry.schema.json +4 -0
- package/shared/schemas/metric-registry.schema.json +55 -1
- package/shared/docent/corpus.json +0 -11333
package/lib/commands/doctor.js
CHANGED
|
@@ -337,6 +337,58 @@ function checkFST(report, langCode) {
|
|
|
337
337
|
* @param {DiagnosticReport} report
|
|
338
338
|
* @param {string} cwd - Project root, for .env.local/.env key lookup
|
|
339
339
|
*/
|
|
340
|
+
/**
|
|
341
|
+
* OpenRouter's documented endpoint for a key's own limits and usage:
|
|
342
|
+
* authenticated, free (it translates nothing), and HTTP 401 for a key that
|
|
343
|
+
* is missing, invalid or disabled (openrouter.ai/docs/api-reference/limits).
|
|
344
|
+
*/
|
|
345
|
+
const OPENROUTER_KEY_URL = 'https://openrouter.ai/api/v1/key';
|
|
346
|
+
|
|
347
|
+
/**
|
|
348
|
+
* Ask OpenRouter whether it accepts the key — the one check here that sends
|
|
349
|
+
* it anywhere, and it costs nothing. doctor used to call the model list
|
|
350
|
+
* (/api/v1/models), which answers without a key: "Connected — N models
|
|
351
|
+
* available" for any value, a placeholder included, while claiming to
|
|
352
|
+
* validate it (Round 11). 401/403 = rejected (a failed check); any other
|
|
353
|
+
* answer, or none, = not checked (a warning). The key is never printed.
|
|
354
|
+
*
|
|
355
|
+
* @param {DiagnosticReport} report
|
|
356
|
+
* @param {string} key
|
|
357
|
+
* @param {string} source - where it was read from, for the advice
|
|
358
|
+
*/
|
|
359
|
+
async function checkOpenRouterKey(report, key, source) {
|
|
360
|
+
let resp;
|
|
361
|
+
try {
|
|
362
|
+
resp = await fetch(OPENROUTER_KEY_URL, {
|
|
363
|
+
headers: { Authorization: `Bearer ${key}` },
|
|
364
|
+
signal: AbortSignal.timeout(5000),
|
|
365
|
+
});
|
|
366
|
+
} catch (err) {
|
|
367
|
+
report.warn('OpenRouter key check', `not checked — could not reach OpenRouter (${err.name === 'TimeoutError' ? 'no answer in 5 s' : err.message})`);
|
|
368
|
+
return;
|
|
369
|
+
}
|
|
370
|
+
if (resp.status === 401 || resp.status === 403) {
|
|
371
|
+
report.fail('OpenRouter key check', `rejected by OpenRouter (HTTP ${resp.status}): the key is invalid, disabled or revoked. `
|
|
372
|
+
+ `Check the value of OPENROUTER_API_KEY (set via ${source}); the llm and llm-coached methods cannot run with it.`);
|
|
373
|
+
return;
|
|
374
|
+
}
|
|
375
|
+
if (!resp.ok) {
|
|
376
|
+
report.warn('OpenRouter key check', `not checked — OpenRouter answered HTTP ${resp.status}`);
|
|
377
|
+
return;
|
|
378
|
+
}
|
|
379
|
+
let data = {};
|
|
380
|
+
try { data = (await resp.json())?.data || {}; } catch { /* the status said accepted */ }
|
|
381
|
+
const num = (v) => (typeof v === 'number' && Number.isFinite(v) ? v : null);
|
|
382
|
+
const usage = num(data.usage);
|
|
383
|
+
const limit = num(data.limit);
|
|
384
|
+
const left = num(data.limit_remaining);
|
|
385
|
+
const credits = [
|
|
386
|
+
usage !== null ? `${usage} credits used` : null,
|
|
387
|
+
limit !== null ? `${left !== null ? `${left} of ${limit}` : `limit ${limit}`} left` : (data.limit === null ? 'no credit limit' : null),
|
|
388
|
+
].filter(Boolean).join(', ');
|
|
389
|
+
report.pass('OpenRouter key check', `accepted by OpenRouter (GET /api/v1/key — no charge)${credits ? `; ${credits}` : ''}`);
|
|
390
|
+
}
|
|
391
|
+
|
|
340
392
|
async function checkMethods(report, cwd) {
|
|
341
393
|
output.raw('');
|
|
342
394
|
output.raw(' ── Methods ──');
|
|
@@ -369,24 +421,8 @@ async function checkMethods(report, cwd) {
|
|
|
369
421
|
const openRouterKey = getEnvOrFileVar('OPENROUTER_API_KEY', cwd);
|
|
370
422
|
if (openRouterKey) {
|
|
371
423
|
const source = process.env.OPENROUTER_API_KEY ? 'environment' : '.env.local/.env';
|
|
372
|
-
report.pass('OpenRouter API key', `Set via ${source}
|
|
373
|
-
|
|
374
|
-
// Try to validate by checking the /api/v1/models endpoint
|
|
375
|
-
try {
|
|
376
|
-
const resp = await fetch('https://openrouter.ai/api/v1/models', {
|
|
377
|
-
headers: { 'Authorization': `Bearer ${openRouterKey}` },
|
|
378
|
-
signal: AbortSignal.timeout(5000),
|
|
379
|
-
});
|
|
380
|
-
if (resp.ok) {
|
|
381
|
-
const data = await resp.json();
|
|
382
|
-
const modelCount = data.data?.length || 0;
|
|
383
|
-
report.pass('OpenRouter API', `Connected — ${modelCount} models available`);
|
|
384
|
-
} else {
|
|
385
|
-
report.warn('OpenRouter API', `HTTP ${resp.status} — key may be invalid`);
|
|
386
|
-
}
|
|
387
|
-
} catch (err) {
|
|
388
|
-
report.warn('OpenRouter API', `Could not reach API: ${err.message}`);
|
|
389
|
-
}
|
|
424
|
+
report.pass('OpenRouter API key', `Set via ${source}`);
|
|
425
|
+
await checkOpenRouterKey(report, openRouterKey, source);
|
|
390
426
|
} else {
|
|
391
427
|
report.warn('OpenRouter API key', 'OPENROUTER_API_KEY not set (checked environment, .env.local, .env) — LLM methods unavailable');
|
|
392
428
|
}
|
package/lib/commands/help.js
CHANGED
|
@@ -14,75 +14,80 @@ const DEFAULT_CONFIG_FILENAME = CONFIG_FILENAMES[0];
|
|
|
14
14
|
*/
|
|
15
15
|
function run() {
|
|
16
16
|
console.log(`
|
|
17
|
-
champollion —
|
|
17
|
+
champollion — translate your project, and deploy MT you have measured
|
|
18
18
|
|
|
19
|
-
|
|
20
|
-
init
|
|
21
|
-
sync Translate
|
|
22
|
-
|
|
19
|
+
TRANSLATE YOUR PROJECT
|
|
20
|
+
init Set up: detects your framework's locale files (or --yes for defaults)
|
|
21
|
+
sync Translate what changed (--redo / --fresh to translate again)
|
|
22
|
+
verify Check translations are present and correct (CI gate)
|
|
23
|
+
status Show pair graph, methods, and config summary
|
|
23
24
|
watch Auto-sync when the source file changes
|
|
25
|
+
serve Serve this project's translation stack over HTTP (api-method contract)
|
|
24
26
|
audit List all untranslated [EN] fallback values
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
submit Propose an index entry (review-gated): print a pre-filled GitHub issue
|
|
27
|
+
integrity Audit locale files for placeholder/encoding/ICU issues
|
|
28
|
+
tm Manage Translation Memory cache (stats, clear, seed, prune)
|
|
29
|
+
xliff Export/import XLIFF 1.2 for professional review
|
|
29
30
|
lint Scan source files for hardcoded strings (pre-commit gate)
|
|
30
31
|
wrap Auto-wrap hardcoded strings in t() calls (with undo)
|
|
31
32
|
seo Generate hreflang, sitemap.xml, or JSON-LD schema
|
|
32
|
-
integrity Audit locale files for placeholder/encoding issues
|
|
33
33
|
repair-script Restore romanization where script conversion was unwanted
|
|
34
|
-
status Show pair graph, methods, and config summary
|
|
35
|
-
provenance Show licensing & resource dependencies for all pairs
|
|
36
34
|
plugin Manage method plugins (install, remove, list)
|
|
37
|
-
fonts Download web fonts for PUA script converters
|
|
38
|
-
tm Manage Translation Memory cache (stats, clear, seed, prune)
|
|
39
|
-
xliff Export/import XLIFF 1.2 for professional review
|
|
40
35
|
models List available models for a provider
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
recommend Method guidance for a pair — availability + cited evidence (--use, --json)
|
|
36
|
+
fonts Download web fonts for PUA script converters
|
|
37
|
+
provenance Show licensing & resource dependencies for all pairs
|
|
44
38
|
doctor System health check (cards, config, FSTs, methods)
|
|
45
39
|
|
|
40
|
+
THE NETWORK champollion network <command> — each also works without the prefix
|
|
41
|
+
card What the index knows about a language (card \<code\> [--json])
|
|
42
|
+
recommend Methods you can run for a pair, with the evidence for each
|
|
43
|
+
leaderboard Published results (--pair, --sort, --json)
|
|
44
|
+
register-corpus Register a test set without handing it over (local-only/private/public/sealed)
|
|
45
|
+
seal-corpus Sealed-tier crypto verbs: keygen / seal / open (organizer-node bridge)
|
|
46
|
+
submit Propose an index entry (review-gated): print a pre-filled GitHub issue
|
|
47
|
+
|
|
48
|
+
Building MT for a language end to end (measure, train, prove, deploy)?
|
|
49
|
+
https://champollion.dev/docs/build-mt-for-your-language
|
|
50
|
+
|
|
46
51
|
OPTIONS
|
|
47
52
|
--config <path> Path to config file (default: ${DEFAULT_CONFIG_FILENAME})
|
|
48
53
|
--dir <path> Override locales directory
|
|
49
|
-
--content-dir <p> Hugo content
|
|
54
|
+
--content-dir <p> Folder of Markdown/MDX to translate (Hugo content/ or any folder)
|
|
50
55
|
--source <code> Override source locale (default: en)
|
|
51
56
|
--base-url <url> Override base URL for SEO commands
|
|
52
|
-
--model <model>
|
|
53
|
-
--method <method>
|
|
54
|
-
--format <fmt> Locale file format: json, toml, yaml, or auto (default: auto)
|
|
57
|
+
--model <model> Translation model for this run only (the config's "model" is not changed)
|
|
58
|
+
--method <method> Translation method for this run only: llm, llm-coached, local (your own model), openai, anthropic, gemini, google-translate, deepl, microsoft-translator, libretranslate, api
|
|
59
|
+
--format <fmt> Locale file format: json, toml, yaml, po, arb, or auto (default: auto)
|
|
55
60
|
--dry Preview changes without writing files
|
|
56
|
-
--
|
|
61
|
+
--redo <scope> Translate again: all | keys:<k1,k2> | content | files:<glob> (cache still serves) | gaps (plural forms a model left out, asked anew)
|
|
62
|
+
--prune plural-extras Remove i18next plural keys for forms a language does not have (sync)
|
|
63
|
+
--fresh Do not use the cache for what is queued (billed again)
|
|
57
64
|
--src <path> Source directory for lint/wrap (auto-detected)
|
|
58
65
|
--min-length <n> Minimum string length to flag (default: 2)
|
|
59
66
|
--warn-only Exit 0 even with issues (lint, integrity)
|
|
60
67
|
--undo Restore files from .champollion-backup/ (wrap)
|
|
61
68
|
--out <path> Write output to file (seo sitemap, xliff export)
|
|
62
69
|
--locale <code> Target locale (xliff export, tm clear, tm seed)
|
|
63
|
-
--no-tm Skip Translation Memory cache for this sync run
|
|
64
70
|
--no-verify Skip post-sync verification pass
|
|
65
71
|
|
|
66
72
|
SUPPORTED FORMATS
|
|
67
73
|
json Standard JSON (next-intl, i18next, react-intl)
|
|
68
74
|
toml Hugo i18n TOML files (i18n/*.toml)
|
|
69
75
|
yaml Hugo i18n YAML files (i18n/*.yaml)
|
|
70
|
-
|
|
76
|
+
po gettext catalogs (Django, GNU gettext; .pot templates)
|
|
77
|
+
arb Flutter Application Resource Bundles (app_<locale>.arb)
|
|
78
|
+
auto Auto-detect from file extensions
|
|
71
79
|
|
|
72
80
|
QUICK START
|
|
73
|
-
1. Set OPENROUTER_API_KEY (or provider key
|
|
74
|
-
2.
|
|
81
|
+
1. Set OPENROUTER_API_KEY (or a provider key; or --method local for a model on this machine)
|
|
82
|
+
2. Run: champollion init # finds your locale files
|
|
75
83
|
3. Run: champollion sync
|
|
76
84
|
|
|
77
85
|
The tool will:
|
|
78
|
-
•
|
|
79
|
-
•
|
|
80
|
-
• Translate Hugo Markdown content files (if --content-dir is set)
|
|
86
|
+
• Translate only what changed; the Translation Memory never bills the same sentence twice
|
|
87
|
+
• Protect placeholders, ICU plurals and markup (a damaged translation is rejected and retried)
|
|
81
88
|
• Fail loud on any translation errors (no silent failures)
|
|
82
89
|
• Verify translations after writing (key parity, placeholders, script compliance)
|
|
83
|
-
• Batch translations to avoid token overflow
|
|
84
90
|
• Preserve your file structure and formatting
|
|
85
|
-
• Cache translations in Translation Memory to avoid redundant API calls
|
|
86
91
|
|
|
87
92
|
Run champollion <command> --help for detailed help on any command.
|
|
88
93
|
`);
|