@likolabs/i18nmd 0.1.1 → 0.2.0

package/CHANGELOG.md

@@ -1,5 +1,9 @@
 # Changelog
 
+## 0.2.0
+
+- **Static websites.** `extract` reads `.html` pages: it marks each sentence, the title and description, image text and form labels with a `data-i18n` attribute and leaves the English in place, and takes `/* i18n */` strings from inline scripts. `render <site> --out dist --url …` writes a copy of every page in every language (`/haw/…`) with `lang`, `hreflang` links, `og:url`, adjusted relative links, and a language switcher wherever a page has `<nav data-i18n-languages>`. No JavaScript needed.
+
 ## 0.1.1
 
 - Published to npm as `@likolabs/i18nmd` (`npm install --save-dev @likolabs/i18nmd`); the command is still `i18nmd`. npm 12 refuses GitHub and tarball URLs by default, so the 0.1.0 install instructions failed there.

package/PROMPT.md

@@ -11,6 +11,8 @@ Give these instructions to a coding agent in the application's repository. They
    npx i18nmd extract src/components --out translations/ui --in-place
    ```
 
+   For a site of plain HTML pages, extract the pages (`npx i18nmd extract site --in-place`) and build each language with `npx i18nmd render site --out dist`.
+
 3. **Review the extractor's work.** Read the diff. Then work through every diagnostic it printed:
    - *looks like a count*: make the message an ICU plural, `{count, plural, one {# item} other {# items}}`.
    - *text chosen in code*: move the wording into the message as an ICU `select` or plural, instead of passing English through a placeholder.

package/README.md

@@ -1,6 +1,6 @@
 # i18n.md
 
-Keep your interface strings in Markdown: one file per language, readable and editable by translators, reviewers and LLMs. A compiler checks every language and generates typed code your app imports.
+Keep your interface strings in Markdown, for apps and for plain HTML sites: one file per language, readable and editable by translators, reviewers and LLMs. A compiler checks every language and generates typed code your app imports.
 
 ````md
 # Français
@@ -22,6 +22,7 @@ The files are the source of truth. They diff in pull requests, and anyone can ha
 
 - [Install](#install)
 - [Quick start](#quick-start)
+- [Static websites](#static-websites)
 - [Language files](#language-files)
 - [Divisions](#divisions)
 - [Using the generated code](#using-the-generated-code)
@@ -97,6 +98,42 @@ export function LanguagePicker() {
 
 **5. Keep it current.** After editing source text, `npx i18nmd status` shows what needs translating and `npx i18nmd translate` fills it in.
 
+## Static websites
+
+A site made of HTML pages needs no JavaScript at all. i18nmd writes a copy of each page in each language:
+
+```sh
+npx i18nmd extract site --in-place                            # site/*.html → translations/i18n-en.md
+npx i18nmd --add haw                                          # or write translations/i18n-haw.md yourself
+npx i18nmd render site --out dist --url https://example.com   # dist/ and dist/haw/
+```
+
+`extract` marks each piece of text with a `data-i18n` attribute and leaves the English where it is, so the page still opens and edits as before:
+
+```html
+<h1 data-i18n="from_first_leaf_to_full_grown">From first leaf to <em class="red">full grown.</em></h1>
+```
+
+The message keeps the sentence whole, with its inline markup as tags: `From first leaf to <em>full grown.</em>`. Translators move the tag; its class and other attributes stay in the page. Extract also takes the page `<title>`, the description and link-preview `<meta>` tags, `alt`, `title`, `placeholder` and `aria-label` attributes, and strings in inline scripts marked `/* i18n */`:
+
+```js
+statusEl.textContent = /* i18n */ 'Sending…';
+```
+
+It skips addresses such as `example.com`, and anything inside an element with `translate="no"`, the standard attribute for names and code. It lists text it can't place, such as words beside a block element, and script strings that look like text people read.
+
+The English lives in your HTML. Edit a page, run `extract` again, and the source language file follows; the other languages' versions of that text become stale for `translate` to update.
+
+`render` writes the source language where the pages are and every other language in a directory named for it, `/haw/`, copying everything else in the site alongside. Each copy gets:
+
+- the translated text, with untranslated text left in the source language
+- `<html lang>`, and `dir="rtl"` for right-to-left languages
+- `<link rel="alternate" hreflang>` to every language's copy, so search engines index each one
+- with `--url`, `og:url` and canonical links pointing at that copy
+- relative links adjusted for the language directory
+
+Put `<nav data-i18n-languages></nav>` anywhere on a page, and render fills it with a link to each language, named in that language. Deploy `dist/`.
+
 ## Language files
 
 `translations/i18n-fr.md` holds French and nothing else. The filename gives the language code (`fr`, `pt-BR`, or a custom name such as `pirate`). The first heading names the language in that language, and each `##` heading is a token:
@@ -346,6 +383,7 @@ FormatJS and next-intl already use ICU, so conversion is lossless, and FormatJS
 
 | Command | |
 | --- | --- |
+| `render <site>` | `--out dist`, `--url https://example.com` |
 | `extract <src…>` | `--in-place`, `--out translations/<division>`, `--source en`, `--runtime src/i18n/i18n`, `--dest .i18n/src`, `--locale-expr locale` (calls use `i18nmd.in(locale)`) |
 | `compile` | `--out src/i18n`, `--target ts\|js\|json\|python`, `--eager`, `--skip <division,…>` |
 | `--add <language>`, `--top <n>`, `translate` | `--only fr,de`, `--dry-run`, `--provider`, `--base-url`, `--model`, `--batch 40` |
@@ -359,7 +397,7 @@ Every command takes `--dir`, and `--source` to override the source language reco
 
 ## Limits in 0.1
 
-- The extractor reads JavaScript and TypeScript. Strings in object properties (`{ label: "Save" }`) and plain `.ts` files need `/* i18n */` or the extraction prompt.
+- The extractor reads HTML, JavaScript and TypeScript. Strings in object properties (`{ label: "Save" }`) and plain `.ts` files need `/* i18n */` or the extraction prompt.
 - Switching language loads that whole language at once, not per division.
 - The Python target ignores tags and formats numbers without locale grouping; dates are passed through as given.
 

package/bin/i18nmd.mjs

@@ -6,6 +6,7 @@ import { spawn } from 'node:child_process';
 import { parseCatalog, serializeCatalog, parseLanguageFiles, languageFromFilename, renameToken, textOf, divisionOf, namespaceFor, accessorFor } from '../lib/catalog.mjs';
 import { generateModule, generateModules, divisionFile, runtimeTypes } from '../lib/compiler.mjs';
 import { extractSource, missingDivisionImports } from '../lib/extractor.mjs';
+import { extractHtml, renderHtml } from '../lib/html.mjs';
 import { lockPathFor, findLock, namedRoot, readLock, serializeLock, report, syncLock, markCurrent } from '../lib/lock.mjs';
 import { resolveLanguage, topLanguages, RANKED } from '../lib/languages.mjs';
 import { llmConfig, translateLanguage } from '../lib/llm.mjs';
@@ -32,6 +33,8 @@ Keep files in shape
                                            One Markdown file with every language, and back.
 
 Build
+  render <site> [--out dist] [--url https://example.com]
+                                           Static HTML pages, one copy per language (/haw/…).
   compile [--out src/i18n] [--target ts|js|json|python] [--skip division,...] [--eager]
   extract <src...> [--out translations/<division>] [--in-place] [--runtime src/i18n/i18n]
           [--source en] [--locale-expr locale]
@@ -60,14 +63,16 @@ function python(args) {
   });
 }
 
-async function filesAt(input) {
-  if (!(await stat(input)).isDirectory()) return /\.(?:[cm]?[jt]sx?)$/.test(input) ? [input] : [];
+// Source files under input; html: also .html pages (for extract).
+async function filesAt(input, { html = false } = {}) {
+  const wanted = html ? /\.(?:[cm]?[jt]sx?|html?)$/ : /\.(?:[cm]?[jt]sx?)$/;
+  if (!(await stat(input)).isDirectory()) return wanted.test(input) ? [input] : [];
   const files = [];
   for (const entry of (await readdir(input, { withFileTypes: true })).sort((a, b) => a.name.localeCompare(b.name))) {
     if (entry.name.startsWith('.') || ['node_modules', 'dist', 'build', 'vendor', 'coverage'].includes(entry.name)) continue;
     const file = path.join(input, entry.name);
-    if (entry.isDirectory()) files.push(...await filesAt(file));
-    else if (entry.isFile() && /\.(?:[cm]?[jt]sx?)$/.test(file) && !/\.(test|spec)\.[jt]sx?$/.test(file) && !/\.d\.[cm]?ts$/.test(file)) files.push(file);
+    if (entry.isDirectory()) files.push(...await filesAt(file, { html }));
+    else if (entry.isFile() && wanted.test(file) && !/\.(test|spec)\.[jt]sx?$/.test(file) && !/\.d\.[cm]?ts$/.test(file)) files.push(file);
   }
   return files;
 }
@@ -218,7 +223,7 @@ async function main() {
   if (!command || ['help', '--help', '-h'].includes(command)) { console.log(help); return; }
   if (['--version', '-v', 'version'].includes(command)) { console.log(JSON.parse(await readFile(new URL('../package.json', import.meta.url), 'utf8')).version); return; }
   const aliases = { locale: 'source', language: 'locale-expr' };
-  const valued = /^(skip|out|source|locale|language|locale-expr|dest|runtime|target|table|languages|syntax|in|from|to|model|base-url|provider|batch|only|dir)$/;
+  const valued = /^(url|skip|out|source|locale|language|locale-expr|dest|runtime|target|table|languages|syntax|in|from|to|model|base-url|provider|batch|only|dir)$/;
   const options = Object.create(null), positional = [];
   for (let i = 0; i < rest.length; i++) {
     const arg = rest[i];
@@ -358,6 +363,45 @@ async function main() {
     const divisions = Object.keys(files).filter(n => !/^(?:i18n|language|runtime)\.|^languages\//.test(n) && !n.endsWith('.d.mts')).length;
     console.log(`Compiled ${plural(catalog.messages.length, 'token')} into ${options.out}: ${plural(divisions, 'division module')}, ${plural(Object.keys(catalog.languages).length - 1, 'language chunk')}${options.eager ? ' (eager)' : ''}.`); return;
   }
+  if (command === 'render') {
+    // Static pages, one copy per language: the source language where the pages
+    // are, every other language under its code (/haw/), and every other file copied.
+    const site = one();
+    if (!site || !(await isDirectory(site))) throw new Error('Usage: i18nmd render <site directory> [--out dist] [--url https://example.com]');
+    const out = path.resolve(options.out || 'dist');
+    const root = path.resolve(site);
+    if (out === root) throw new Error('Render into a directory other than the site itself.');
+    const ws = await open(options.dir, options); ws.warn();
+    const { catalog } = ws;
+    const base = (options.url || '').replace(/\/+$/, '');
+    if (base && !/^https?:\/\//.test(base)) throw new Error('--url needs a full address, such as https://example.com.');
+    const codes = Object.keys(catalog.languages);
+    const tables = Object.fromEntries(codes.map(code => [code, Object.fromEntries(catalog.messages.filter(m => Object.hasOwn(m.translations, code)).map(m => [m.key, m.translations[code]]))]));
+    const files = [];
+    const walk = async dir => {
+      for (const entry of (await readdir(dir, { withFileTypes: true })).sort((a, b) => a.name.localeCompare(b.name))) {
+        const file = path.join(dir, entry.name);
+        if (entry.name.startsWith('.') || entry.name === 'node_modules' || file === out) continue;
+        if (entry.isDirectory()) await walk(file); else if (entry.isFile()) files.push(path.relative(root, file).split(path.sep).join('/'));
+      }
+    };
+    await walk(root);
+    let pages = 0;
+    for (const file of files) {
+      if (!/\.html?$/.test(file)) { await mkdir(path.dirname(path.join(out, file)), { recursive: true }); await copyFile(path.join(root, file), path.join(out, file)); continue; }
+      const html = await readFile(path.join(root, file), 'utf8');
+      const address = code => `${base}/${code === catalog.source ? '' : code + '/'}${file.replace(/(^|\/)index\.html?$/, '$1')}`;
+      const urls = Object.fromEntries(codes.map(code => [code, address(code)]));
+      for (const code of codes) {
+        const target = code === catalog.source ? file : `${code}/${file}`;
+        const relative = code === catalog.source ? '' : path.posix.relative(path.posix.dirname(target), path.posix.dirname(file)) + '/';
+        await save(path.join(out, target), renderHtml(html, { language: code, source: catalog.source, messages: tables[code], pages: urls, languages: catalog.languages, relative, absolute: !!base }));
+        pages++;
+      }
+    }
+    for (const row of ws.rows()) if (row.missing.length || row.stale.length) console.warn(`i18nmd: ${summary(row)}; untranslated text stays in ${catalog.languages[catalog.source]}.`);
+    console.log(`Rendered ${plural(pages, 'page')} in ${plural(codes.length, 'language')} into ${path.relative(process.cwd(), out) || '.'}.`); return;
+  }
   if (command === 'join') {
     if (!options.out || !/\.md$/.test(options.out)) throw new Error('Choose the joined Markdown file with --out, e.g. --out translations.md.');
     const ws = await open(one(), options);
@@ -495,8 +539,8 @@ async function main() {
     const outDir = path.dirname(sourceFile);
     const namespace = namespaceFor(await divisionPrefix(outDir, (await findLock(outDir)).file));
     if (languageFromFilename(sourceFile) !== locale) throw new Error('The output filename must match --source.');
-    const files = [...new Set((await Promise.all(positional.map(filesAt))).flat())];
-    if (!files.length) throw new Error('No JavaScript or TypeScript source files found.');
+    const files = [...new Set((await Promise.all(positional.map(input => filesAt(input, { html: true })))).flat())];
+    if (!files.length) throw new Error('No HTML, JavaScript or TypeScript files found.');
     // The deepest directory holding every input; converted copies keep their paths below it.
     const dirs = await Promise.all(positional.map(async input => (await isDirectory(input)) ? path.resolve(input) : path.dirname(path.resolve(input))));
     let root = dirs[0];
@@ -513,21 +557,26 @@ async function main() {
     // Existing tokens are always kept, so extraction can be repeated as code changes.
     let catalog = await exists(sourceFile) ? parseCatalog(await readFile(sourceFile, 'utf8'), { filename: sourceFile }) : undefined;
     const outputs = [];
-    let replacements = 0;
+    let replacements = 0, updated = 0;
     for (const file of files) {
       const relative = path.relative(root, path.resolve(file));
       const output = path.join(destination, relative);
       let importPath = path.relative(path.dirname(output), runtime).split(path.sep).join('/');
       if (!importPath.startsWith('.')) importPath = './' + importPath;
-      const result = extractSource(await readFile(file, 'utf8'), { filename: path.relative(process.cwd(), file), locale, languageExpression: options['locale-expr'], existing: catalog, importPath, namespace });
-      catalog = result.catalog; replacements += result.replacements;
+      const text = await readFile(file, 'utf8'), filename = path.relative(process.cwd(), file);
+      const result = /\.html?$/.test(file)
+        ? extractHtml(text, { filename, locale, existing: catalog, namespace })
+        : extractSource(text, { filename, locale, languageExpression: options['locale-expr'], existing: catalog, importPath, namespace });
+      catalog = result.catalog; replacements += result.replacements; updated += result.updated || 0;
       result.diagnostics.forEach(message => console.warn(message));
       if (result.replacements) outputs.push([output, result.source]);
     }
     // The lock marks the top of the translations tree, so divisions keep their names.
     const lockFile = (await findLock(outDir)).file;
     if (!(await exists(lockFile))) await save(lockFile, serializeLock({ source: locale, translations: {} }));
-    if (!replacements) { console.log(`Nothing new to extract from ${plural(files.length, 'file')}. Mark other display strings with /* i18n */.`); return; }
+    if (!replacements && !updated) { console.log(`Nothing new to extract from ${plural(files.length, 'file')}. Mark other display strings with /* i18n */.`); return; }
+    // An HTML page holds its own English: an edited sentence updates the source file.
+    if (!replacements) { await save(sourceFile, serializeCatalog(catalog, { locale })); console.log(`Updated ${plural(updated, 'string')} in ${sourceFile} from the pages' English; run i18nmd status to see what to translate.`); return; }
     const markdown = serializeCatalog(catalog, { locale });
     for (const [file, text] of outputs) await save(file, text);
     await save(sourceFile, markdown);

package/lib/extractor.mjs

@@ -7,7 +7,7 @@ const sourcePlugins = ['jsx', 'typescript'];
 
 // Readable tokens from the words of the message; a numeric suffix resolves collisions.
 // Tokens never encode the text, so editing the source message keeps the token.
-function tokenFor(text, taken) {
+export function tokenFor(text, taken) {
   const words = text.toLowerCase().replace(/\{[^}]+\}|<\/?[\w-]+>/g, ' ').normalize('NFKD').replace(/[\u0300-\u036f]/g, '').split(/[^a-z0-9]+/).filter(Boolean);
   let slug = '';
   for (const word of words) { if (slug && slug.length + word.length >= 40) break; slug += (slug ? '_' : '') + word; }

package/lib/html.mjs

@@ -0,0 +1,309 @@
+// Static HTML pages: extract marks each translatable element with
+// data-i18n="token" and leaves the English in place; render writes one page per
+// language from the Markdown files. English is authored in the HTML.
+import { posix } from 'node:path';
+import { parse } from 'parse5';
+import { parse as parseScript } from '@babel/parser';
+import { escapeMessageLiteral, parseMessage } from './messages.mjs';
+import { languageName } from './catalog.mjs';
+import { tokenFor } from './extractor.mjs';
+
+// Elements that sit inside a sentence; they become ICU tags.
+const INLINE = new Set(['a', 'abbr', 'b', 'bdi', 'bdo', 'cite', 'code', 'data', 'del', 'dfn', 'em', 'i', 'ins', 'kbd', 'mark', 'q', 's', 'samp', 'small', 'span', 'strong', 'sub', 'sup', 'time', 'u', 'var']);
+// Empty elements allowed at the start or end of a sentence, such as a logo before a name.
+const VOID = new Set(['img', 'br', 'wbr', 'input', 'svg']);
+const SKIP = new Set(['script', 'style', 'template', 'svg', 'math', 'noscript', 'pre', 'textarea', 'select', 'iframe', 'object', 'canvas']);
+const ATTRIBUTES = ['alt', 'title', 'placeholder', 'aria-label', 'aria-description', 'label'];
+const META = new Set(['description', 'og:title', 'og:description', 'og:image:alt', 'og:site_name', 'twitter:title', 'twitter:description', 'twitter:image:alt']);
+const URL_ATTRIBUTES = ['href', 'src', 'srcset', 'action', 'poster', 'formaction'];
+const WHITESPACE = /[ \t\n\r\f]+/g;
+
+const attr = (node, name) => node.attrs?.find(a => a.name === name)?.value;
+const isElement = node => !!node.tagName;
+const isText = node => node.nodeName === '#text';
+const letters = text => /\p{L}/u.test(text);
+// Addresses read the same in every language: domains, emails and URLs.
+const address = text => /^(?:[\w.+-]+@[\w-]+(?:\.[\w-]+)+|(?:https?:\/\/)?[\w-]+(?:\.[\w-]+)+(?:\/\S*)?)$/i.test(text.trim());
+const translatable = text => letters(text) && !address(text);
+const noTranslate = node => attr(node, 'translate') === 'no';
+
+function describe(node) {
+  const tag = node.tagName;
+  const kind = /^h[1-6]$/.test(tag) ? 'Heading' : { p: 'Paragraph', li: 'List item', a: 'Link', button: 'Button', label: 'Form label', title: 'Page title', option: 'Choice in a list', caption: 'Table caption', figcaption: 'Figure caption', th: 'Table heading', td: 'Table cell', legend: 'Form section title', summary: 'Disclosure summary', dt: 'Term', dd: 'Definition' }[tag] || `<${tag}>`;
+  for (let up = node.parentNode; up; up = up.parentNode) if (isElement(up) && attr(up, 'id')) return `${kind} in #${attr(up, 'id')}`;
+  return kind;
+}
+
+// A sentence: an element whose content is only text and inline elements, with
+// any empty elements at its edges.
+function sentence(node) {
+  if (!isElement(node) || SKIP.has(node.tagName) || noTranslate(node)) return null;
+  const children = node.childNodes.filter(c => c.nodeName !== '#comment');
+  let start = 0, end = children.length;
+  const blank = c => isText(c) && !c.value.trim();
+  const edge = c => blank(c) || (isElement(c) && VOID.has(c.tagName));
+  while (start < end && edge(children[start])) start++;
+  while (end > start && edge(children[end - 1])) end--;
+  const body = children.slice(start, end);
+  if (!body.length) return null;
+  const phrasing = c => isText(c) || c.nodeName === '#comment' || (isElement(c) && INLINE.has(c.tagName) && c.childNodes.every(phrasing));
+  if (!body.every(phrasing)) return null;
+  // Links side by side are separate items, not a sentence: a sentence has words of its own.
+  if (body.some(isElement) && !body.some(c => isText(c) && letters(c.value))) return null;
+  const text = body.map(function all(c) { return isText(c) ? c.value : (c.childNodes || []).map(all).join(''); }).join('');
+  return translatable(text) ? { body, lead: children.slice(0, start), trail: children.slice(end) } : null;
+}
+
+// The message for a sentence, and the element each tag name stands for. Names
+// come from the element (em, a, a_2) and are rebuilt the same way when rendering.
+function messageFor(body) {
+  const tags = new Map(), counts = new Map();
+  const walk = nodes => nodes.map(node => {
+    if (isText(node)) return escapeMessageLiteral(node.value.replace(WHITESPACE, ' '));
+    if (!isElement(node)) return '';
+    const n = (counts.get(node.tagName) || 0) + 1;
+    counts.set(node.tagName, n);
+    const name = n === 1 ? node.tagName : `${node.tagName}_${n}`;
+    tags.set(name, node);
+    return `<${name}>${walk(node.childNodes)}</${name}>`;
+  }).join('');
+  return { text: walk(body).trim(), tags };
+}
+
+/** Every translatable spot in a page: sentences, attributes and annotated script strings. */
+function units(html) {
+  const document = parse(html, { sourceCodeLocationInfo: true });
+  const found = [], diagnostics = [];
+  const attributes = node => {
+    for (const name of ATTRIBUTES) if (translatable(attr(node, name) || '')) found.push({ kind: 'attribute', node, name, text: escapeMessageLiteral(attr(node, name).replace(WHITESPACE, ' ').trim()), context: `${name} text of ${describe(node)}` });
+  };
+  const visit = node => {
+    if (!isElement(node)) { (node.childNodes || []).forEach(visit); return; }
+    if (noTranslate(node)) return;
+    const meta = node.tagName === 'meta' && (attr(node, 'name') || attr(node, 'property'));
+    if (meta && META.has(meta) && translatable(attr(node, 'content') || '')) found.push({ kind: 'attribute', node, name: 'content', text: escapeMessageLiteral(attr(node, 'content')), context: `Page ${meta.replace(/^(og|twitter):/, '')} for search and link previews` });
+    attributes(node);
+    if (node.tagName === 'input' && ['submit', 'button', 'reset'].includes(attr(node, 'type')) && letters(attr(node, 'value') || '')) found.push({ kind: 'attribute', node, name: 'value', text: escapeMessageLiteral(attr(node, 'value')), context: `Button in ${describe(node)}` });
+    if (node.tagName === 'script' && !attr(node, 'src')) { scriptStrings(node, found, diagnostics, html); return; }
+    if (SKIP.has(node.tagName)) return;
+    const unit = sentence(node);
+    if (unit) {
+      const { text, tags } = messageFor(unit.body);
+      found.push({ kind: 'sentence', node, ...unit, text, tags, context: describe(node) });
+      // Elements inside a sentence keep their own attributes, such as an icon's alt text.
+      const inside = n => { for (const c of n.childNodes || []) if (isElement(c) && !noTranslate(c)) { attributes(c); inside(c); } };
+      inside(node);
+      return;
+    }
+    for (const child of node.childNodes || []) {
+      if (isText(child) && translatable(child.value)) diagnostics.push(`line ${child.sourceCodeLocation?.startLine}: text beside other elements in <${node.tagName}> ("${child.value.trim().slice(0, 40)}"); wrap it in an element such as <span> to translate it.`);
+      else visit(child);
+    }
+  };
+  visit(document);
+  return { document, found, diagnostics };
+}
+
+// Script strings that read like words for people: a capitalised word and a space or
+// sentence punctuation, unlike ids, selectors, URLs and JSON keys.
+const shown = text => /^\s*[\p{Lu}]/u.test(text) && /[\p{L}][\s.…!?]|[.…!?]$/u.test(text.trim()) && !address(text) && !/^[\w-]+$/.test(text.trim());
+
+// Strings in an inline script marked /* i18n */ or /* i18n:token */.
+function scriptStrings(node, found, diagnostics, html) {
+  const text = node.childNodes[0];
+  if (!text?.sourceCodeLocation) return;
+  const offset = text.sourceCodeLocation.startOffset;
+  let ast;
+  try { ast = parseScript(text.value, { sourceType: 'unambiguous', errorRecovery: true }); } catch { return; }
+  const visit = n => {
+    if (!n || typeof n.type !== 'string') return;
+    const comment = n.leadingComments?.findLast(c => /^\s*i18n(?:\s*$|:)/.test(c.value));
+    if (comment && (n.type === 'StringLiteral' || (n.type === 'TemplateLiteral' && !n.expressions.length))) {
+      const value = n.type === 'StringLiteral' ? n.value : n.quasis[0].value.cooked;
+      found.push({ kind: 'script', start: offset + n.start, end: offset + n.end, commentStart: offset + comment.start, commentEnd: offset + comment.end, explicit: /i18n:\s*([\w.-]+)/.exec(comment.value)?.[1], text: escapeMessageLiteral(value), context: 'Text a script on the page shows' });
+    } else if (!comment && (n.type === 'StringLiteral' || n.type === 'TemplateLiteral') && shown(n.value ?? n.quasis.map(q => q.value.cooked).join(''))) {
+      diagnostics.push(`line ${node.sourceCodeLocation.startLine + (n.loc?.start.line || 1) - 1}: a script string looks like text people read ("${(n.value ?? n.quasis[0].value.cooked).slice(0, 40)}"); mark it /* i18n */ to translate it.`);
+    } else if (comment) diagnostics.push(`line ${node.sourceCodeLocation.startLine + (n.loc?.start.line || 1) - 1}: /* i18n */ marks a string that isn't plain; use a string without \${…}.`);
+    for (const [key, value] of Object.entries(n)) {
+      if (key === 'loc' || key.endsWith('Comments')) continue;
+      if (Array.isArray(value)) value.forEach(visit); else if (value && typeof value.type === 'string') visit(value);
+    }
+  };
+  visit(ast.program);
+}
+
+const markerOf = unit => unit.kind === 'sentence' ? 'data-i18n' : unit.kind === 'attribute' ? `data-i18n-${unit.name}` : null;
+
+/**
+ * Mark a page's translatable text with data-i18n attributes (and /* i18n:token *\/
+ * comments in scripts), adding new messages to the catalog. An element already
+ * marked keeps its token, and the page's English replaces the catalog's.
+ */
+export function extractHtml(html, { existing, locale = 'en', namespace = '', filename = 'index.html' } = {}) {
+  const catalog = existing ? structuredClone(existing) : { title: 'Site', source: locale, syntax: 'icu', languages: { [locale]: languageName(locale) }, messages: [] };
+  const byKey = new Map(catalog.messages.map(m => [m.key, m]));
+  const byText = new Map(catalog.messages.map(m => [m.translations[locale], m]));
+  const { found, diagnostics } = units(html);
+  const edits = [];
+  let marked = 0, updated = 0;
+  for (const unit of found) {
+    const marker = markerOf(unit);
+    const given = unit.kind === 'script' ? unit.explicit : attr(unit.node, marker);
+    const short = given?.startsWith(namespace) ? given.slice(namespace.length) : given;
+    let message = short ? byKey.get(short) : byText.get(unit.text);
+    if (message && short && message.translations[locale] !== unit.text) {
+      // The page is where the English is written: take its text.
+      if (byText.get(message.translations[locale]) === message) byText.delete(message.translations[locale]);
+      message.translations[locale] = unit.text; byText.set(unit.text, message); updated++;
+    }
+    if (!message) {
+      const key = short || tokenFor(unit.text, byKey);
+      message = { key, context: unit.context, optional: [], translations: { [locale]: unit.text } };
+      catalog.messages.push(message); byKey.set(key, message); byText.set(unit.text, message);
+    }
+    if (given) continue;
+    const token = namespace + message.key;
+    marked++;
+    if (unit.kind === 'script') edits.push({ start: unit.commentStart, end: unit.commentEnd, text: `/* i18n:${token} */` });
+    else {
+      const tag = unit.node.sourceCodeLocation.startTag;
+      const close = html[tag.endOffset - 2] === '/' ? tag.endOffset - 2 : tag.endOffset - 1;
+      edits.push({ start: close, end: close, text: ` ${marker}="${token}"` });
+    }
+  }
+  let source = html;
+  for (const edit of edits.sort((a, b) => b.start - a.start)) source = source.slice(0, edit.start) + edit.text + source.slice(edit.end);
+  return { source, catalog, diagnostics: diagnostics.map(d => `${filename}:${d.replace(/^line /, '')}`), replacements: marked, updated };
+}
+
+const escapeHtml = text => text.replace(/&/g, '&amp;').replace(/</g, '&lt;').replace(/>/g, '&gt;').replace(/\u00a0/g, '&nbsp;');
+const escapeAttribute = text => escapeHtml(text).replace(/"/g, '&quot;');
+// The text of a message without markup, for attributes and scripts.
+const plain = nodes => nodes.map(n => typeof n === 'string' ? n : n.type === 'tag' ? plain(n.children) : `{${n.name}}`).join('');
+
+/**
+ * One page in one language. messages maps tokens to this language's message
+ * text (missing ones keep the page's English). pages lists every language's URL
+ * for this page, for hreflang links and the language switcher. relative: the
+ * path from this copy's directory to the original's ("../" for a page moved into
+ * haw/), put before the page's relative links. absolute: pages' URLs are full
+ * URLs, so og:url and canonical links can name this copy.
+ */
+export function renderHtml(html, { language, source, messages = {}, pages = {}, languages = {}, relative = '', absolute = false } = {}) {
+  // Two passes, since a sentence's content holds tags whose attributes change too:
+  // first every tag and attribute edit, then each sentence rebuilt from those tags.
+  const first = renderAttributes(html, { language, source, messages, pages, languages, relative, absolute });
+  return renderSentences(first, { language, source, messages });
+}
+
+function translationFor(language, source, messages, token) {
+  if (language === source || !token || !Object.hasOwn(messages, token)) return null;
+  try { return parseMessage(messages[token]); } catch { return null; }
+}
+
+function applyEdits(html, edits) {
+  // From the end; insertions at one spot keep their order.
+  let out = html;
+  edits.map((edit, index) => ({ ...edit, index })).sort((a, b) => b.start - a.start || b.end - a.end || b.index - a.index)
+    .forEach(edit => { out = out.slice(0, edit.start) + edit.text + out.slice(edit.end); });
+  return out;
+}
+
+function attributeRemover(html, edits) {
+  return (node, name) => {
+    const loc = node.sourceCodeLocation.attrs?.[name];
+    if (!loc) return;
+    let start = loc.startOffset;
+    while (start > 0 && /\s/.test(html[start - 1])) start--;
+    edits.push({ start, end: loc.endOffset, text: '' });
+  };
+}
+
+function renderSentences(html, { language, source, messages }) {
+  const { found } = units(html);
+  const edits = [], removeAttribute = attributeRemover(html, edits);
+  for (const unit of found) {
+    if (unit.kind !== 'sentence') continue;
+    const token = attr(unit.node, 'data-i18n');
+    if (token === undefined) continue;
+    removeAttribute(unit.node, 'data-i18n');
+    const nodes = translationFor(language, source, messages, token);
+    if (!nodes) continue;
+    const outer = node => html.slice(node.sourceCodeLocation.startOffset, node.sourceCodeLocation.endOffset);
+    const inner = node => html.slice(node.sourceCodeLocation.startTag.endOffset, node.sourceCodeLocation.endTag?.startOffset ?? node.sourceCodeLocation.startTag.endOffset);
+    const write = list => list.map(n => {
+      if (typeof n === 'string') return escapeHtml(n);
+      if (n.type !== 'tag') return escapeHtml(`{${n.name}}`);
+      const element = unit.tags.get(n.name);
+      if (!element) return write(n.children);
+      const loc = element.sourceCodeLocation;
+      // translate="no" content stays as written.
+      return html.slice(loc.startOffset, loc.startTag.endOffset) + (noTranslate(element) ? inner(element) : write(n.children)) + (loc.endTag ? html.slice(loc.endTag.startOffset, loc.endTag.endOffset) : '');
+    }).join('');
+    // The sentence's own spacing at its edges, such as the space after a leading icon.
+    const body = unit.body.map(outer).join('');
+    const [, before, after] = /^([ \t\n\r\f]*)[\s\S]*?([ \t\n\r\f]*)$/.exec(body);
+    const loc = unit.node.sourceCodeLocation;
+    edits.push({ start: loc.startTag.endOffset, end: loc.endTag.startOffset, text: unit.lead.map(outer).join('') + before + write(nodes) + after + unit.trail.map(outer).join('') });
+  }
+  return applyEdits(html, edits);
+}
+
+function renderAttributes(html, { language, source, messages, pages, languages, relative, absolute }) {
+  const { document, found } = units(html);
+  const edits = [];
+  const replace = (start, end, text) => edits.push({ start, end, text });
+  const removeAttribute = attributeRemover(html, edits);
+  for (const unit of found) {
+    if (unit.kind === 'sentence') continue;
+    const marker = markerOf(unit);
+    const token = unit.kind === 'script' ? unit.explicit : attr(unit.node, marker);
+    const nodes = translationFor(language, source, messages, token);
+    if (unit.kind === 'script') { if (nodes) replace(unit.start, unit.end, JSON.stringify(plain(nodes))); continue; }
+    if (token) removeAttribute(unit.node, marker);
+    if (nodes) { const loc = unit.node.sourceCodeLocation.attrs[unit.name]; replace(loc.startOffset, loc.endOffset, `${unit.name}="${escapeAttribute(plain(nodes))}"`); }
+  }
+  const root = document.childNodes.find(n => n.tagName === 'html');
+  const head = root?.childNodes.find(n => n.tagName === 'head');
+  const all = function* (node) { yield node; for (const child of node.childNodes || []) yield* all(child); };
+  let direction = 'ltr';
+  try { direction = new Intl.Locale(language).getTextInfo?.().direction || new Intl.Locale(language).textInfo?.direction || 'ltr'; } catch { /* custom languages */ }
+  if (root?.sourceCodeLocation?.startTag) {
+    const tag = root.sourceCodeLocation.startTag;
+    for (const name of ['lang', 'dir']) removeAttribute(root, name);
+    replace(tag.endOffset - 1, tag.endOffset - 1, ` lang="${language}"${direction === 'rtl' ? ' dir="rtl"' : ''}`);
+  }
+  for (const node of all(document)) {
+    if (!isElement(node) || !node.sourceCodeLocation) continue;
+    // Pages in a language directory sit deeper: relative links go up to the same files.
+    if (relative) for (const name of URL_ATTRIBUTES) {
+      const value = attr(node, name);
+      if (value === undefined || node.sourceCodeLocation.attrs?.[name] === undefined) continue;
+      const fix = part => {
+        if (!part || /^(?:[a-z][a-z0-9+.-]*:|\/|#|\?)/i.test(part)) return part;
+        const [, file, rest] = /^([^?#]*)(.*)$/.exec(part);
+        return posix.normalize(relative + file) + rest;
+      };
+      const next = name === 'srcset' ? value.split(',').map(s => s.trim().replace(/^\S+/, fix)).join(', ') : fix(value);
+      if (next !== value) { const loc = node.sourceCodeLocation.attrs[name]; replace(loc.startOffset, loc.endOffset, `${name}="${escapeAttribute(next)}"`); }
+    }
+    // Links that name this page's own address point at this language's copy.
+    if (absolute && ((node.tagName === 'meta' && attr(node, 'property') === 'og:url') || (node.tagName === 'link' && attr(node, 'rel') === 'canonical'))) {
+      const name = node.tagName === 'meta' ? 'content' : 'href';
+      const loc = node.sourceCodeLocation.attrs[name];
+      if (loc) replace(loc.startOffset, loc.endOffset, `${name}="${escapeAttribute(pages[language])}"`);
+    }
+    if (attr(node, 'data-i18n-languages') !== undefined) {
+      removeAttribute(node, 'data-i18n-languages');
+      const loc = node.sourceCodeLocation;
+      const links = Object.keys(pages).map(code => `<a href="${escapeAttribute(pages[code])}" hreflang="${code}" lang="${code}"${code === language ? ' aria-current="page"' : ''}>${escapeHtml(languages[code] || code)}</a>`).join(' ');
+      if (loc.endTag) replace(loc.startTag.endOffset, loc.endTag.startOffset, links);
+    }
+  }
+  if (head?.sourceCodeLocation?.endTag && Object.keys(pages).length > 1) {
+    const alternates = Object.keys(pages).map(code => `<link rel="alternate" hreflang="${code}" href="${escapeAttribute(pages[code])}">`);
+    alternates.push(`<link rel="alternate" hreflang="x-default" href="${escapeAttribute(pages[source])}">`);
+    replace(head.sourceCodeLocation.endTag.startOffset, head.sourceCodeLocation.endTag.startOffset, alternates.join('\n') + '\n');
+  }
+  return applyEdits(html, edits);
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@likolabs/i18nmd",
-  "version": "0.1.1",
+  "version": "0.2.0",
   "description": "Interface strings in Markdown, one file per language. A compiler checks every language and generates typed code your app imports.",
   "author": "Liko Labs (https://likolabs.com)",
   "type": "module",
@@ -31,7 +31,8 @@
     "node": ">=22"
   },
   "dependencies": {
-    "@babel/parser": "7.29.0"
+    "@babel/parser": "7.29.0",
+    "parse5": "^8.0.1"
   },
   "repository": {
     "type": "git",