champollion 0.3.3 → 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (142) hide show
  1. package/README.md +52 -37
  2. package/bin/cli.js +53 -5
  3. package/index.js +63 -2
  4. package/lib/api-key.js +17 -4
  5. package/lib/autofix.js +83 -36
  6. package/lib/bridge/method_bridge.py +15 -3
  7. package/lib/cards/reader.js +51 -3
  8. package/lib/cards/remote.js +15 -0
  9. package/lib/cards/search-names.js +178 -0
  10. package/lib/command-help.js +289 -88
  11. package/lib/commands/audit.js +10 -3
  12. package/lib/commands/card.js +583 -226
  13. package/lib/commands/doctor.js +54 -18
  14. package/lib/commands/help.js +37 -32
  15. package/lib/commands/init.js +1689 -87
  16. package/lib/commands/integrity.js +127 -40
  17. package/lib/commands/leaderboard.js +187 -67
  18. package/lib/commands/models.js +9 -2
  19. package/lib/commands/provenance.js +7 -2
  20. package/lib/commands/recommend.js +43 -14
  21. package/lib/commands/register-corpus.js +649 -130
  22. package/lib/commands/seal-corpus.js +1 -1
  23. package/lib/commands/status.js +564 -27
  24. package/lib/commands/submit.js +17 -12
  25. package/lib/commands/sync.js +31 -7
  26. package/lib/commands/tm.js +16 -10
  27. package/lib/commands/verify.js +27 -3
  28. package/lib/commands/wrap.js +63 -5
  29. package/lib/commands/xliff.js +135 -64
  30. package/lib/commercial-eligibility.js +1 -1
  31. package/lib/config.js +196 -14
  32. package/lib/content-estimate.js +96 -0
  33. package/lib/content-refusals.js +270 -0
  34. package/lib/content-review.js +372 -0
  35. package/lib/content-sync.js +1127 -344
  36. package/lib/content.js +94 -7
  37. package/lib/corpus-registration.mjs +197 -38
  38. package/lib/cost-label.js +29 -0
  39. package/lib/cost-report.js +726 -78
  40. package/lib/diff.js +38 -4
  41. package/lib/docusaurus-sync.js +965 -253
  42. package/lib/edit-distance.js +31 -0
  43. package/lib/fallback.js +964 -0
  44. package/lib/file-scope.js +106 -0
  45. package/lib/flatten.js +80 -3
  46. package/lib/flutter-locales.js +124 -0
  47. package/lib/format.js +266 -12
  48. package/lib/hash.js +146 -21
  49. package/lib/icu-structure.js +929 -0
  50. package/lib/integrity.js +223 -75
  51. package/lib/language-pair.js +157 -0
  52. package/lib/lint.js +78 -16
  53. package/lib/local-only-marks.js +106 -0
  54. package/lib/locale-layout.js +1103 -0
  55. package/lib/locale-state.js +571 -0
  56. package/lib/methods/anthropic.js +5 -0
  57. package/lib/methods/apertium.js +6 -3
  58. package/lib/methods/api.js +138 -25
  59. package/lib/methods/base.js +17 -0
  60. package/lib/methods/coaching-data.js +153 -0
  61. package/lib/methods/content-separator.js +43 -0
  62. package/lib/methods/deepl.js +1 -1
  63. package/lib/methods/direct-llm.js +252 -103
  64. package/lib/methods/external.js +146 -63
  65. package/lib/methods/gemini.js +1 -0
  66. package/lib/methods/google-translate.js +1 -0
  67. package/lib/methods/http-utils.js +41 -0
  68. package/lib/methods/libretranslate.js +7 -2
  69. package/lib/methods/llm-coached.js +68 -128
  70. package/lib/methods/llm.js +80 -31
  71. package/lib/methods/local.js +93 -10
  72. package/lib/methods/microsoft-translator.js +1 -2
  73. package/lib/methods/openai.js +4 -2
  74. package/lib/methods/openrouter-client.js +20 -19
  75. package/lib/methods/openrouter-pricing.js +150 -13
  76. package/lib/methods/prompt-methods.js +20 -0
  77. package/lib/methods/provider-pricing.js +42 -1
  78. package/lib/methods/request-capture.js +104 -0
  79. package/lib/methods/tilde.js +1 -1
  80. package/lib/methods/translated.js +1 -2
  81. package/lib/missing-key.js +93 -0
  82. package/lib/models.js +11 -0
  83. package/lib/name-rules.js +32 -0
  84. package/lib/named-keys.js +172 -0
  85. package/lib/no-translate.js +4 -3
  86. package/lib/output.js +160 -19
  87. package/lib/pairs.js +586 -30
  88. package/lib/placeholders.js +394 -0
  89. package/lib/plugins.js +8 -0
  90. package/lib/plural-gap-redo.js +109 -0
  91. package/lib/plurals.js +323 -0
  92. package/lib/po.js +1187 -0
  93. package/lib/public-catalogue.js +74 -0
  94. package/lib/recommend.js +527 -32
  95. package/lib/redo.js +95 -0
  96. package/lib/refusal-category.js +44 -0
  97. package/lib/registers.js +255 -11
  98. package/lib/repair-script.js +20 -13
  99. package/lib/scripts.js +193 -106
  100. package/lib/seal.mjs +6 -5
  101. package/lib/sealed-qualifier.mjs +2 -2
  102. package/lib/segment.js +2 -1
  103. package/lib/seo.js +19 -9
  104. package/lib/serve.js +43 -6
  105. package/lib/shared-output-seed.js +164 -0
  106. package/lib/source-contexts.js +39 -0
  107. package/lib/submit.mjs +57 -5
  108. package/lib/sync.js +2923 -474
  109. package/lib/terminology.js +13 -4
  110. package/lib/tm-evict.js +179 -0
  111. package/lib/tm-seed.js +5 -2
  112. package/lib/tm.js +818 -36
  113. package/lib/translate-pair.js +639 -34
  114. package/lib/translate.js +78 -5
  115. package/lib/types.js +22 -3
  116. package/lib/validate.js +880 -17
  117. package/lib/verify.js +1296 -104
  118. package/lib/watch.js +32 -13
  119. package/lib/xliff.js +44 -3
  120. package/package.json +3 -2
  121. package/shared/CORPORA-CARDS.md +2 -0
  122. package/shared/DATA-SOVEREIGNTY.md +19 -20
  123. package/shared/LANGUAGE-CARD-FIELDS.md +1 -1
  124. package/shared/cards-fallback.json +1 -1
  125. package/shared/catalogue/card-config.json +1 -1
  126. package/shared/curated-orthography-conventions.json +26 -8
  127. package/shared/docent/faq.en.json +14 -16
  128. package/shared/docent/system-prompt.md +17 -19
  129. package/shared/explainers/tc-features.json +15 -15
  130. package/shared/gettext-plural-forms.json +45 -0
  131. package/shared/human-services.json +1 -1
  132. package/shared/method-registry.json +2 -0
  133. package/shared/metric-registry.json +96 -18
  134. package/shared/schemas/champollion-plugin.schema.json +4 -0
  135. package/shared/schemas/corpora-card.schema.json +20 -10
  136. package/shared/schemas/human-services.schema.json +2 -2
  137. package/shared/schemas/language-card.schema.json +1 -1
  138. package/shared/schemas/method-card.schema.json +1 -1
  139. package/shared/schemas/method-index-record.schema.json +67 -0
  140. package/shared/schemas/method-registry.schema.json +4 -0
  141. package/shared/schemas/metric-registry.schema.json +55 -1
  142. package/shared/docent/corpus.json +0 -11333
@@ -337,6 +337,58 @@ function checkFST(report, langCode) {
337
337
  * @param {DiagnosticReport} report
338
338
  * @param {string} cwd - Project root, for .env.local/.env key lookup
339
339
  */
340
+ /**
341
+ * OpenRouter's documented endpoint for a key's own limits and usage:
342
+ * authenticated, free (it translates nothing), and HTTP 401 for a key that
343
+ * is missing, invalid or disabled (openrouter.ai/docs/api-reference/limits).
344
+ */
345
+ const OPENROUTER_KEY_URL = 'https://openrouter.ai/api/v1/key';
346
+
347
+ /**
348
+ * Ask OpenRouter whether it accepts the key — the one check here that sends
349
+ * it anywhere, and it costs nothing. doctor used to call the model list
350
+ * (/api/v1/models), which answers without a key: "Connected — N models
351
+ * available" for any value, a placeholder included, while claiming to
352
+ * validate it (Round 11). 401/403 = rejected (a failed check); any other
353
+ * answer, or none, = not checked (a warning). The key is never printed.
354
+ *
355
+ * @param {DiagnosticReport} report
356
+ * @param {string} key
357
+ * @param {string} source - where it was read from, for the advice
358
+ */
359
+ async function checkOpenRouterKey(report, key, source) {
360
+ let resp;
361
+ try {
362
+ resp = await fetch(OPENROUTER_KEY_URL, {
363
+ headers: { Authorization: `Bearer ${key}` },
364
+ signal: AbortSignal.timeout(5000),
365
+ });
366
+ } catch (err) {
367
+ report.warn('OpenRouter key check', `not checked — could not reach OpenRouter (${err.name === 'TimeoutError' ? 'no answer in 5 s' : err.message})`);
368
+ return;
369
+ }
370
+ if (resp.status === 401 || resp.status === 403) {
371
+ report.fail('OpenRouter key check', `rejected by OpenRouter (HTTP ${resp.status}): the key is invalid, disabled or revoked. `
372
+ + `Check the value of OPENROUTER_API_KEY (set via ${source}); the llm and llm-coached methods cannot run with it.`);
373
+ return;
374
+ }
375
+ if (!resp.ok) {
376
+ report.warn('OpenRouter key check', `not checked — OpenRouter answered HTTP ${resp.status}`);
377
+ return;
378
+ }
379
+ let data = {};
380
+ try { data = (await resp.json())?.data || {}; } catch { /* the status said accepted */ }
381
+ const num = (v) => (typeof v === 'number' && Number.isFinite(v) ? v : null);
382
+ const usage = num(data.usage);
383
+ const limit = num(data.limit);
384
+ const left = num(data.limit_remaining);
385
+ const credits = [
386
+ usage !== null ? `${usage} credits used` : null,
387
+ limit !== null ? `${left !== null ? `${left} of ${limit}` : `limit ${limit}`} left` : (data.limit === null ? 'no credit limit' : null),
388
+ ].filter(Boolean).join(', ');
389
+ report.pass('OpenRouter key check', `accepted by OpenRouter (GET /api/v1/key — no charge)${credits ? `; ${credits}` : ''}`);
390
+ }
391
+
340
392
  async function checkMethods(report, cwd) {
341
393
  output.raw('');
342
394
  output.raw(' ── Methods ──');
@@ -369,24 +421,8 @@ async function checkMethods(report, cwd) {
369
421
  const openRouterKey = getEnvOrFileVar('OPENROUTER_API_KEY', cwd);
370
422
  if (openRouterKey) {
371
423
  const source = process.env.OPENROUTER_API_KEY ? 'environment' : '.env.local/.env';
372
- report.pass('OpenRouter API key', `Set via ${source} (LLM methods available)`);
373
-
374
- // Try to validate by checking the /api/v1/models endpoint
375
- try {
376
- const resp = await fetch('https://openrouter.ai/api/v1/models', {
377
- headers: { 'Authorization': `Bearer ${openRouterKey}` },
378
- signal: AbortSignal.timeout(5000),
379
- });
380
- if (resp.ok) {
381
- const data = await resp.json();
382
- const modelCount = data.data?.length || 0;
383
- report.pass('OpenRouter API', `Connected — ${modelCount} models available`);
384
- } else {
385
- report.warn('OpenRouter API', `HTTP ${resp.status} — key may be invalid`);
386
- }
387
- } catch (err) {
388
- report.warn('OpenRouter API', `Could not reach API: ${err.message}`);
389
- }
424
+ report.pass('OpenRouter API key', `Set via ${source}`);
425
+ await checkOpenRouterKey(report, openRouterKey, source);
390
426
  } else {
391
427
  report.warn('OpenRouter API key', 'OPENROUTER_API_KEY not set (checked environment, .env.local, .env) — LLM methods unavailable');
392
428
  }
@@ -14,75 +14,80 @@ const DEFAULT_CONFIG_FILENAME = CONFIG_FILENAMES[0];
14
14
  */
15
15
  function run() {
16
16
  console.log(`
17
- champollion — Research-grade translation engine for i18n projects
17
+ champollion — translate your project, and deploy MT you have measured
18
18
 
19
- COMMANDS
20
- init Interactive setup wizard (or --yes for quick defaults)
21
- sync Translate & sync all locale files
22
- serve Serve this project's translation stack over HTTP (api-method contract)
19
+ TRANSLATE YOUR PROJECT
20
+ init Set up: detects your framework's locale files (or --yes for defaults)
21
+ sync Translate what changed (--redo / --fresh to translate again)
22
+ verify Check translations are present and correct (CI gate)
23
+ status Show pair graph, methods, and config summary
23
24
  watch Auto-sync when the source file changes
25
+ serve Serve this project's translation stack over HTTP (api-method contract)
24
26
  audit List all untranslated [EN] fallback values
25
- card Pretty-print a language card (card \<code\> [--json])
26
- register-corpus Register a corpus: pick a license + exposure tier (local-only/private/public/sealed)
27
- seal-corpus Sealed-tier crypto verbs: keygen / seal / open (organizer-node bridge)
28
- submit Propose an index entry (review-gated): print a pre-filled GitHub issue
27
+ integrity Audit locale files for placeholder/encoding/ICU issues
28
+ tm Manage Translation Memory cache (stats, clear, seed, prune)
29
+ xliff Export/import XLIFF 1.2 for professional review
29
30
  lint Scan source files for hardcoded strings (pre-commit gate)
30
31
  wrap Auto-wrap hardcoded strings in t() calls (with undo)
31
32
  seo Generate hreflang, sitemap.xml, or JSON-LD schema
32
- integrity Audit locale files for placeholder/encoding issues
33
33
  repair-script Restore romanization where script conversion was unwanted
34
- status Show pair graph, methods, and config summary
35
- provenance Show licensing & resource dependencies for all pairs
36
34
  plugin Manage method plugins (install, remove, list)
37
- fonts Download web fonts for PUA script converters
38
- tm Manage Translation Memory cache (stats, clear, seed, prune)
39
- xliff Export/import XLIFF 1.2 for professional review
40
35
  models List available models for a provider
41
- verify Verify translations are present and correct (CI gate)
42
- leaderboard Show MT leaderboard from Supabase (--pair, --sort, --json)
43
- recommend Method guidance for a pair — availability + cited evidence (--use, --json)
36
+ fonts Download web fonts for PUA script converters
37
+ provenance Show licensing & resource dependencies for all pairs
44
38
  doctor System health check (cards, config, FSTs, methods)
45
39
 
40
+ THE NETWORK champollion network <command> — each also works without the prefix
41
+ card What the index knows about a language (card \<code\> [--json])
42
+ recommend Methods you can run for a pair, with the evidence for each
43
+ leaderboard Published results (--pair, --sort, --json)
44
+ register-corpus Register a test set without handing it over (local-only/private/public/sealed)
45
+ seal-corpus Sealed-tier crypto verbs: keygen / seal / open (organizer-node bridge)
46
+ submit Propose an index entry (review-gated): print a pre-filled GitHub issue
47
+
48
+ Building MT for a language end to end (measure, train, prove, deploy)?
49
+ https://champollion.dev/docs/build-mt-for-your-language
50
+
46
51
  OPTIONS
47
52
  --config <path> Path to config file (default: ${DEFAULT_CONFIG_FILENAME})
48
53
  --dir <path> Override locales directory
49
- --content-dir <p> Hugo content directory for Markdown translation
54
+ --content-dir <p> Folder of Markdown/MDX to translate (Hugo content/ or any folder)
50
55
  --source <code> Override source locale (default: en)
51
56
  --base-url <url> Override base URL for SEO commands
52
- --model <model> Override translation model
53
- --method <method> Default translation method: llm, llm-coached, google-translate, api, deepl, microsoft-translator, libretranslate, openai, anthropic, gemini
54
- --format <fmt> Locale file format: json, toml, yaml, or auto (default: auto)
57
+ --model <model> Translation model for this run only (the config's "model" is not changed)
58
+ --method <method> Translation method for this run only: llm, llm-coached, local (your own model), openai, anthropic, gemini, google-translate, deepl, microsoft-translator, libretranslate, api
59
+ --format <fmt> Locale file format: json, toml, yaml, po, arb, or auto (default: auto)
55
60
  --dry Preview changes without writing files
56
- --force-keys <k> Comma-separated dot-notation keys to force re-translate
61
+ --redo <scope> Translate again: all | keys:<k1,k2> | content | files:<glob> (cache still serves) | gaps (plural forms a model left out, asked anew)
62
+ --prune plural-extras Remove i18next plural keys for forms a language does not have (sync)
63
+ --fresh Do not use the cache for what is queued (billed again)
57
64
  --src <path> Source directory for lint/wrap (auto-detected)
58
65
  --min-length <n> Minimum string length to flag (default: 2)
59
66
  --warn-only Exit 0 even with issues (lint, integrity)
60
67
  --undo Restore files from .champollion-backup/ (wrap)
61
68
  --out <path> Write output to file (seo sitemap, xliff export)
62
69
  --locale <code> Target locale (xliff export, tm clear, tm seed)
63
- --no-tm Skip Translation Memory cache for this sync run
64
70
  --no-verify Skip post-sync verification pass
65
71
 
66
72
  SUPPORTED FORMATS
67
73
  json Standard JSON (next-intl, i18next, react-intl)
68
74
  toml Hugo i18n TOML files (i18n/*.toml)
69
75
  yaml Hugo i18n YAML files (i18n/*.yaml)
70
- auto Auto-detect from file extensions in locales directory
76
+ po gettext catalogs (Django, GNU gettext; .pot templates)
77
+ arb Flutter Application Resource Bundles (app_<locale>.arb)
78
+ auto Auto-detect from file extensions
71
79
 
72
80
  QUICK START
73
- 1. Set OPENROUTER_API_KEY (or provider key like OPENAI_API_KEY, DEEPL_API_KEY, etc.) in your environment or .env.local
74
- 2. Put your source locale (en.json / en.toml / en.yaml) in a locales/ directory
81
+ 1. Set OPENROUTER_API_KEY (or a provider key; or --method local for a model on this machine)
82
+ 2. Run: champollion init # finds your locale files
75
83
  3. Run: champollion sync
76
84
 
77
85
  The tool will:
78
- • Auto-detect locale file format (JSON, TOML, or YAML)
79
- • Translate missing keys via OpenRouter (${DEFAULT_OPENROUTER_MODEL})
80
- • Translate Hugo Markdown content files (if --content-dir is set)
86
+ • Translate only what changed; the Translation Memory never bills the same sentence twice
87
+ • Protect placeholders, ICU plurals and markup (a damaged translation is rejected and retried)
81
88
  • Fail loud on any translation errors (no silent failures)
82
89
  • Verify translations after writing (key parity, placeholders, script compliance)
83
- • Batch translations to avoid token overflow
84
90
  • Preserve your file structure and formatting
85
- • Cache translations in Translation Memory to avoid redundant API calls
86
91
 
87
92
  Run champollion <command> --help for detailed help on any command.
88
93
  `);