claude-translator 1.3.0

package/references/throughput-and-cost.md

@@ -0,0 +1,124 @@
+# Throughput and cost
+
+How to budget a run, and how to make it finish in hours rather than days.
+
+---
+
+## Where the savings come from
+
+Both levers below apply whichever provider you use, and they matter more than the choice of
+provider: deduplication routinely cuts the bill by an order of magnitude, which is a bigger
+factor than the 10x between Gemini and Claude.
+
+**Deduplication is the largest lever, by far.** Header, footer, navigation and cross-link
+blocks repeat on every page — a single string can occur hundreds of times across a site.
+Translate each unique string once and reuse it everywhere.
+
+On a content site the collapse from *segment occurrences* to *unique units* is typically an
+order of magnitude. `extract.mjs` prints both numbers; the ratio between them is the factor
+by which your API bill shrinks.
+
+**Incrementality is the second.** The memory is keyed by SHA-1 of the source unit, so
+editing one page changes only the hashes it touched. A re-run after a content edit
+translates those and reuses everything else — usually cents. A full re-translation happens
+only if the memory is deleted, which is why `i18n/tm/` belongs in version control.
+
+## Budgeting
+
+Estimate from **tokens**, not pages:
+
+```
+source tokens ≈ unique-unit words × 1.3
+output tokens ≈ source tokens × script factor
+```
+
+Script factor, roughly:
+
+| Target script | Output tokens vs source |
+| --- | --- |
+| Latin (es, fr, de, pt…) | 1.2 – 1.5× |
+| Cyrillic, Greek, Arabic, Hebrew | 1.8 – 2.2× |
+| Devanagari, Thai, Bengali, Tamil | 3 – 4× |
+| CJK | 1.0 – 1.5× |
+
+Then apply your provider's per-million rate. The default is Claude, which is **not** the
+cheapest option — it is the one the project is named for, and the tradeoff is worth stating
+plainly rather than burying.
+
+Rates checked 2026-08-25, USD per million tokens:
+
+| Provider / model | In | Out | A mid-sized site, 20 locales |
+| --- | --- | --- | --- |
+| `gemini-2.5-flash-lite` | 0.10 | 0.40 | **~$2.40** |
+| `claude-haiku-4-5` *(default)* | 1 | 5 | **~$30** |
+| `claude-sonnet-5` | 3 | 15 | ~$90 |
+| `claude-opus-5` | 5 | 25 | ~$150 |
+| a local model via Ollama | — | — | **$0** |
+
+Those totals assume ~150k unique source words and Latin-script targets; scale by the table
+above for other scripts. Treat them as order-of-magnitude, not a quote.
+
+**So: money is a constraint here in a way it was not before 1.2.** If cost matters more
+than the last few percent of idiom, one line moves you:
+
+```json
+{ "provider": "gemini", "model": "gemini-2.5-flash-lite" }
+```
+
+and one line moves you to free, at the price of running the model yourself:
+
+```json
+{ "provider": "openai", "apiBaseUrl": "http://localhost:11434/v1", "model": "qwen2.5:14b" }
+```
+
+Whatever you pick, **review time is still the real constraint** — the pilot in the
+sequence below exists because layout problems and script-specific defects cost more to
+find late than any of these numbers.
+
+The Batch API halves cost on providers that offer one and runs asynchronously. Worth it
+for a full rollout; not worth the added latency for a 3-locale pilot.
+
+## Batching
+
+- Default 40 units per request.
+- Split **only** on failure — truncated JSON or a safety block. Pre-emptively shrinking
+  batches just multiplies request count for no benefit.
+- Use `--batch 6` when re-translating units that already failed once: those are the long,
+  awkward ones that hit output limits.
+- Smaller and local models want smaller batches. 40 units is tuned for hosted frontier
+  models; a 7B running on a laptop is more reliable at 8–10.
+
+## Parallelism
+
+Running one locale at a time is the slowest possible arrangement. Split the locale list
+across several concurrent streams:
+
+```bash
+node scripts/translate.mjs --lang es,it,pl,nl --concurrency 8 &
+node scripts/translate.mjs --lang fr,de,tr,sv --concurrency 8 &
+node scripts/translate.mjs --lang ja,ko,zh,th --concurrency 8 &
+```
+
+Wall-clock drops close to linearly with stream count. Around 32 concurrent requests has run
+cleanly without rate-limit errors on a paid tier; back off if you see 429s — the retry logic
+handles them, but they waste time. A local model is the exception: concurrency past what the
+GPU can hold makes it slower, not faster, so start at 2 and measure.
+
+Two rules for splitting:
+
+- **Round-robin, don't chunk.** Each stream should mix high- and low-value locales so a
+  partial run still covers what matters.
+- **Order by traffic, not alphabetically.** If the run is interrupted at 60%, you want the
+  60% that earns.
+
+## Suggested sequence
+
+1. **Pilot 3–4 locales** — one LTR, one RTL, one CJK, and one long-word language
+   (German, Finnish). This surfaces script-specific layout and heuristic problems while
+   they are still cheap to fix.
+2. Review, fix, get sign-off.
+3. **Bulk run** the remainder in parallel streams, traffic-ordered.
+4. `verify` and `audit-seo` **once** across everything at the end.
+
+The pilot is not ceremony. RTL panel clipping, CJK length-ratio false positives and
+font-fallback layout shift all appear only when those scripts are first built.

package/scripts/audit-seo.mjs

@@ -0,0 +1,261 @@
+#!/usr/bin/env node
+/**
+ * Exhaustive SEO audit of the built site — hreflang, canonicals, schema, sitemaps.
+ *
+ *   node scripts/i18n/audit-seo.mjs            # every page, every locale
+ *   node scripts/i18n/audit-seo.mjs --lang it  # one locale
+ *
+ * verify.mjs samples pages to stay fast; this reads ALL 13,328 of them, because the
+ * failure modes here are per-page (a canonical pointing at the wrong slug, one locale
+ * missing from one page's alternate set) and sampling cannot prove their absence.
+ *
+ * Checks, per page:
+ *   canonical      exactly the page's own URL, https, no trailing slash
+ *   hreflang       57 <link rel=alternate>: 55 locales + en + x-default, no duplicates,
+ *                  self-reference present and identical to the canonical, en and
+ *                  x-default both pointing at the English original, every href https
+ *   return tags    full mesh — every alternate URL must itself be a built page whose
+ *                  own set points back (guaranteed structurally, verified explicitly)
+ *   og             og:url matches canonical, og:locale matches the page's language
+ *   JSON-LD        parses; inLanguage is the page's language, never "en" on a locale
+ *                  page; WebPage @id is <page-url>#webpage and unique per page
+ *   robots         no accidental noindex on an indexable page
+ */
+
+import { readFileSync, existsSync, writeFileSync } from 'fs';
+import { join } from 'path';
+import { fileURLToPath } from 'url';
+
+import { BUILD_DIR as DIST, BASE_URL as BASE, LOCALES as LANG_ROWS, BY_PATH, RTL, getPages, I18N_DIR } from './config.mjs';
+import { creditBlock } from './credit.mjs';
+
+const ROOT = process.cwd();
+
+const args = Object.fromEntries(
+  process.argv
+    .slice(2)
+    .join(' ')
+    .split('--')
+    .filter(Boolean)
+    .map((s) => s.trim().split(/\s+/))
+    .map(([k, ...v]) => [k, v.join(' ') || true])
+);
+
+// ── Locale table ─────────────────────────────────────────────────────────────
+
+
+const EXPECTED_CODES = new Set([...LANG_ROWS.map((r) => r.hreflang), 'en', 'x-default']);
+
+const SITEMAP_SLUGS = getPages();
+
+const LOCALES = args.lang ? String(args.lang).split(',') : LANG_ROWS.map((r) => r.pathCode);
+
+// ── Helpers ──────────────────────────────────────────────────────────────────
+
+const attrs = (tag) => Object.fromEntries([...tag.matchAll(/([\w:-]+)="([^"]*)"/g)].map((m) => [m[1], m[2]]));
+
+function findTag(html, re, test) {
+  for (const m of html.matchAll(re)) {
+    const a = attrs(m[0]);
+    if (test(a)) return a;
+  }
+  return null;
+}
+
+const LINK_RE = /<link[^>]*>/g;
+const META_RE = /<meta[^>]*>/g;
+
+const pageUrl = (lang, slug) =>
+  lang ? (slug ? `${BASE}/${lang}/${slug}` : `${BASE}/${lang}`) : slug ? `${BASE}/${slug}` : `${BASE}/`;
+
+// ── Findings ─────────────────────────────────────────────────────────────────
+
+const findings = [];
+const add = (check, page, detail) => findings.push({ check, page, detail });
+const counts = {};
+const bump = (k) => (counts[k] = (counts[k] ?? 0) + 1);
+
+// ── Per-page audit ───────────────────────────────────────────────────────────
+
+function auditPage(lang, slug) {
+  const file = lang
+    ? slug
+      ? join(DIST, lang, slug, 'index.html')
+      : join(DIST, lang, 'index.html')
+    : slug
+      ? join(DIST, slug, 'index.html')
+      : join(DIST, 'index.html');
+
+  if (!existsSync(file)) {
+    add('missing-page', `${lang || 'en'}/${slug || '(home)'}`, 'file does not exist');
+    return;
+  }
+
+  const html = readFileSync(file, 'utf8');
+  const id = `${lang || 'en'}/${slug || '(home)'}`;
+  const self = pageUrl(lang, slug);
+  const enUrl = pageUrl(null, slug);
+  const row = lang ? BY_PATH[lang] : { hreflang: 'en' };
+  bump('pages');
+
+  // ── canonical ──
+  const canon = findTag(html, LINK_RE, (a) => a.rel === 'canonical');
+  if (!canon) add('canonical-missing', id, 'no rel=canonical');
+  else {
+    if (canon.href !== self) add('canonical-wrong', id, `${canon.href} ≠ ${self}`);
+    if (!canon.href?.startsWith('https://')) add('canonical-protocol', id, canon.href);
+    if (canon.href !== `${BASE}/` && canon.href?.endsWith('/')) add('canonical-trailing-slash', id, canon.href);
+  }
+
+  // ── hreflang set ──
+  const alts = [...html.matchAll(LINK_RE)].map((m) => attrs(m[0])).filter((a) => a.rel === 'alternate' && a.hreflang);
+  const codes = alts.map((a) => a.hreflang);
+  const byCode = Object.fromEntries(alts.map((a) => [a.hreflang, a.href]));
+
+  if (codes.length !== EXPECTED_CODES.size) {
+    add('hreflang-count', id, `${codes.length} tags, expected ${EXPECTED_CODES.size}`);
+  }
+  const dupes = codes.filter((c, i) => codes.indexOf(c) !== i);
+  if (dupes.length) add('hreflang-duplicate', id, [...new Set(dupes)].join(','));
+
+  for (const c of codes) if (!EXPECTED_CODES.has(c)) add('hreflang-unexpected-code', id, c);
+  for (const c of EXPECTED_CODES) if (!codes.includes(c)) add('hreflang-missing-code', id, c);
+
+  // self-reference must exist and equal the canonical
+  const selfHref = byCode[row.hreflang];
+  if (selfHref === undefined) add('hreflang-self-missing', id, row.hreflang);
+  else if (selfHref !== self) add('hreflang-self-wrong', id, `${row.hreflang}=${selfHref} ≠ ${self}`);
+  else if (canon && selfHref !== canon.href) add('hreflang-canonical-mismatch', id, `${selfHref} ≠ ${canon.href}`);
+
+  // en and x-default must point at the English original
+  if (byCode['en'] !== enUrl) add('hreflang-en-wrong', id, `${byCode['en']} ≠ ${enUrl}`);
+  if (byCode['x-default'] !== enUrl) add('hreflang-xdefault-wrong', id, `${byCode['x-default']} ≠ ${enUrl}`);
+
+  for (const [c, href] of Object.entries(byCode)) {
+    if (!href?.startsWith('https://')) add('hreflang-protocol', id, `${c}=${href}`);
+    if (href !== `${BASE}/` && href?.endsWith('/')) add('hreflang-trailing-slash', id, `${c}=${href}`);
+  }
+
+  // every alternate must be the URL this page's slug maps to in that locale
+  for (const r of LANG_ROWS) {
+    const want = pageUrl(r.pathCode, slug);
+    if (byCode[r.hreflang] && byCode[r.hreflang] !== want) {
+      add('hreflang-target-wrong', id, `${r.hreflang}=${byCode[r.hreflang]} ≠ ${want}`);
+    }
+  }
+
+  // ── og ──
+  const ogUrl = findTag(html, META_RE, (a) => a.property === 'og:url');
+  if (ogUrl && ogUrl.content !== self) add('og-url-wrong', id, `${ogUrl.content} ≠ ${self}`);
+  const ogLocale = findTag(html, META_RE, (a) => a.property === 'og:locale');
+  if (ogLocale && ogLocale.content !== row.hreflang)
+    add('og-locale-wrong', id, `${ogLocale.content} ≠ ${row.hreflang}`);
+
+  // ── robots ──
+  const robots = findTag(html, META_RE, (a) => a.name === 'robots');
+  if (robots && /noindex/i.test(robots.content ?? '')) add('noindex', id, robots.content);
+
+  // ── html lang ──
+  const htmlTag = /<html[^>]*>/.exec(html)?.[0] ?? '';
+  const htmlLang = /\slang="([^"]*)"/.exec(htmlTag)?.[1];
+  if (htmlLang !== row.hreflang) add('html-lang-wrong', id, `${htmlLang} ≠ ${row.hreflang}`);
+
+  // ── JSON-LD ──
+  let sawWebPage = false;
+  for (const m of html.matchAll(/<script type="application\/ld\+json">([\s\S]*?)<\/script>/g)) {
+    let data;
+    try {
+      data = JSON.parse(m[1]);
+    } catch {
+      add('jsonld-invalid', id, m[1].slice(0, 60));
+      continue;
+    }
+    bump('jsonld-blocks');
+    const walk = (n) => {
+      if (Array.isArray(n)) return n.forEach(walk);
+      if (!n || typeof n !== 'object') return;
+      const types = [].concat(n['@type'] ?? []);
+      if (typeof n.inLanguage === 'string' && n.inLanguage !== row.hreflang) {
+        add('jsonld-inlanguage', id, `${types.join('/')}: ${n.inLanguage} ≠ ${row.hreflang}`);
+      }
+      if (types.includes('WebPage')) {
+        sawWebPage = true;
+        if (n['@id'] !== `${self}#webpage`) add('jsonld-webpage-id', id, `${n['@id']} ≠ ${self}#webpage`);
+        if (n.url && n.url !== self) add('jsonld-webpage-url', id, `${n.url} ≠ ${self}`);
+      }
+      for (const v of Object.values(n)) if (v && typeof v === 'object') walk(v);
+    };
+    walk(data);
+  }
+  if (!sawWebPage) bump('pages-without-webpage-schema');
+}
+
+// ── Run ──────────────────────────────────────────────────────────────────────
+
+console.log(`auditing ${SITEMAP_SLUGS.length} slugs × (${LOCALES.length} locales + English)\n`);
+
+for (const slug of SITEMAP_SLUGS) auditPage(null, slug);
+for (const lang of LOCALES) for (const slug of SITEMAP_SLUGS) auditPage(lang, slug);
+
+// ── Sitemap audit ────────────────────────────────────────────────────────────
+
+console.log('── sitemaps ──');
+const smIndex = join(DIST, 'sitemap.xml');
+if (!existsSync(smIndex)) add('sitemap-index-missing', 'sitemap.xml', 'not built');
+else {
+  const idx = readFileSync(smIndex, 'utf8');
+  const children = [...idx.matchAll(/<loc>([^<]+)<\/loc>/g)].map((m) => m[1]);
+  console.log(`  index lists ${children.length} sitemaps`);
+
+  let totalUrls = 0;
+  for (const child of children) {
+    const rel = child.replace(`${BASE}/`, '');
+    const f = join(DIST, rel);
+    if (!existsSync(f)) {
+      add('sitemap-child-missing', rel, 'listed in index but not built');
+      continue;
+    }
+    const urls = [...readFileSync(f, 'utf8').matchAll(/<loc>([^<]+)<\/loc>/g)].map((m) => m[1]);
+    totalUrls += urls.length;
+    if (urls.length !== SITEMAP_SLUGS.length) {
+      add('sitemap-url-count', rel, `${urls.length} urls, expected ${SITEMAP_SLUGS.length}`);
+    }
+    // every listed URL must exist as a built page
+    for (const u of urls) {
+      const path = u.replace(BASE, '').replace(/^\//, '');
+      const file = path === '' ? join(DIST, 'index.html') : join(DIST, path, 'index.html');
+      if (!existsSync(file)) add('sitemap-url-404', rel, u);
+      if (u !== `${BASE}/` && u.endsWith('/')) add('sitemap-trailing-slash', rel, u);
+      if (!u.startsWith('https://')) add('sitemap-protocol', rel, u);
+    }
+  }
+  console.log(`  total URLs across sitemaps: ${totalUrls.toLocaleString()}`);
+  counts['sitemap-urls'] = totalUrls;
+}
+
+// ── Report ───────────────────────────────────────────────────────────────────
+
+console.log('\n── coverage ──');
+for (const [k, v] of Object.entries(counts)) console.log(`  ${k}: ${v.toLocaleString()}`);
+
+const byCheck = {};
+for (const f of findings) (byCheck[f.check] ??= []).push(f);
+
+console.log('\n── findings ──');
+if (findings.length === 0) console.log('  none — all checks passed');
+for (const [check, list] of Object.entries(byCheck).sort((a, b) => b[1].length - a[1].length)) {
+  console.log(`  ${check}: ${list.length.toLocaleString()}`);
+  for (const f of list.slice(0, 3)) console.log(`      ${f.page} — ${f.detail}`);
+  if (list.length > 3) console.log(`      … and ${(list.length - 3).toLocaleString()} more`);
+}
+
+writeFileSync(join(ROOT, 'i18n/seo-audit.json'), JSON.stringify({ counts, findings }, null, 2));
+console.log(`\nwrote i18n/seo-audit.json (${findings.length.toLocaleString()} findings)`);
+
+creditBlock(
+  findings.length === 0
+    ? ['SEO audit clean \u2014 canonicals, hreflang mesh, og, JSON-LD and sitemaps all pass']
+    : [`SEO audit finished with ${findings.length.toLocaleString()} finding(s) \u2014 see i18n/seo-audit.json`]
+);
+
+process.exit(findings.length ? 1 : 0);