@trueclone/cli 0.2.0 → 0.4.0-rc.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,11 +1,11 @@
1
1
  # trueclone
2
2
 
3
- **A working twin of any website, from one URL, with scores that prove it.**
3
+ **The exact same website, from one URL, at every screen size, with the checks to prove it.**
4
4
 
5
- Give your coding agent a URL. trueclone captures the site, extracts its design system, guides the rebuild and measures the result against the original until it matches: layout, scroll states, motion curves, interactions, hover effects and computed styles, at phone, tablet and desktop widths. You get the clone as HTML and as a Next.js app, plus the design system as `DESIGN.md` and a shadcn `globals.css`.
5
+ Give your coding agent a URL. trueclone captures the site, extracts its design system, seeds the clone with the original's structure, hands the agent the original's own scripts to port every behaviour from, and measures the result against the original until it matches: layout, scroll states, canvases, motion, interactions, hovers, styles and the load sequence, at the sizes it was built from and at sizes it never saw, plus a frame-by-frame visual review. You get the clone as HTML and as a Next.js app, plus the design system as `DESIGN.md` and a shadcn `globals.css`.
6
6
 
7
7
  ```bash
8
- npx @trueclone/cli init # once per project: installs trueclone, the skill for your agent, REA as an MCP server
8
+ npx @trueclone/cli init # once per project: installs trueclone and the skill for your agent
9
9
  # then tell Claude Code, Codex or Cursor:
10
10
  # "clone https://example.com with trueclone"
11
11
  ```
@@ -23,7 +23,7 @@ npx @trueclone/cli init # once per project: installs trueclone, the s
23
23
 
24
24
  ## Proof
25
25
 
26
- Three benchmarks with very different stacks, scored by the tool itself:
26
+ Three benchmarks with very different stacks, scored by the tool itself. These numbers were measured at the three build sizes only, before trueclone 0.4 added held-out sizes and the visual review; 0.4's benchmark runs replace them.
27
27
 
28
28
  **[rea.tools](https://rea.tools/)**: static docs with a canvas game and interactive demos, exact mode (copied with the owner's permission).
29
29
 
@@ -60,33 +60,25 @@ The hand-built portfolio isn't perfect yet, and its report says so: computed sty
60
60
  ## How it works
61
61
 
62
62
  ```
63
- URL ─► capture ─► understand ─► design system ─► build ─► score ─► report
64
- 3 widths motion DESIGN.md your 7 checks REPORT.md
65
- HAR, HTML inventory globals.css agent in parallel
66
- assets + REA tokens.css ▲ │
67
- └─────┘ until every target passes
63
+ URL ─► capture ─► understand ─► design system ─► seed ─► build ─► score + review ─► report
64
+ 5 sizes the site's DESIGN.md structure your 10 checks at REPORT.md
65
+ scripts own scripts, globals.css + viewport agent build and held-out
66
+ HAR motion runtime tokens.css functions sizes, by eye too
67
+ ▲ │
68
+ └────────────┘ until every check passes
68
69
  ```
69
70
 
70
- 1. **Capture** (`trueclone capture`): full-page screenshots at 390, 768 and 1440px, sections, rendered HTML, assets, every network request.
71
- 2. **Understand** (`trueclone motion`, REA): reads the live animation runtime (GSAP timelines, ScrollTriggers, Lenis and Swiper settings, CSS animations, cursor-driven tweens) and, through REA, bundles, source maps, event listeners and API endpoints.
71
+ 1. **Capture** (`trueclone capture`): full-page screenshots at five sizes (390, 768 and 1440 wide to build from; 1280 and 1920 held out, never used to build), sections, rendered HTML, assets, every network request, every script the page ran (with what ran and where each event listener lives), and the inline values scripts set, read at extra sizes inside each of the site's breakpoint ranges.
72
+ 2. **Understand**: `reference/scripts/INDEX.md` (the site's own code, ranked by what ran), the live animation runtime (`trueclone motion`: GSAP timelines, ScrollTriggers, Lenis, Swiper, CSS animations) and the scroll states. The agent ports each behaviour's rule from the site's code instead of guessing it from effects.
72
73
  3. **Design system** (`trueclone design`): one canonical `design-tokens.json`, rendered to DESIGN.md, globals.css and the clone's tokens.
73
- 4. **Build**: `trueclone reveals` records the original's scroll reveals frame by frame and its scroll-driven class changes; `trueclone widgets` clicks every tab and menu and records each state; `trueclone seed` turns the script-free mirror into the clone (template mode: layout-measured lorem text and same-size media placeholders) and replays what was recorded. Layout is at 99%+ before anyone writes CSS; your agent writes the remaining behaviour, guided by the skill. `trueclone rules` shows the original's CSS for any element (like DevTools).
74
- 5. **Score** (`trueclone score`): layout pixel diff per section, scroll states replayed, motion curves sampled, interaction scenarios with change assertions, every hover, computed styles at matching points. Reference measurements are cached, so only the clone is re-measured.
75
- 6. **Report**: `REPORT.md` with the verdict, what to fix first, and what was not copied.
74
+ 4. **Seed** (`trueclone seed`): the script-free mirror becomes the clone (template mode: layout-measured lorem text and same-size media placeholders). A value a script computed from the screen size is carried as that function (`calc()`), never as one size's number; a value no such function explains is left to the agent and listed. Pins pin, one-shot reveals and scroll class changes replay, shadow roots render, the first paint is held until the clone's scripts have run.
75
+ 5. **Build**: the agent writes `clone/script.js`, scene by scene, from the site's own rules; `clone/patches/` for anything on top of the seed; `trueclone rules` shows the original's CSS for any element.
76
+ 6. **Score and review** (`trueclone score`, `trueclone review`): layout, scroll states, canvases, motion curves, details, hovers (outcome, not just "something changed"), interactions (same end state), load sequence, at the build sizes and the held-out sizes; then the reference and the clone side by side, frame by frame, with a verdict for each.
77
+ 7. **Report**: `REPORT.md` with every check at every size, the review, what to fix first, and what was not copied.
76
78
 
77
- ## Why not just REA?
79
+ ## REA
78
80
 
79
- trueclone **bundles** [REA](https://github.com/morluto/rea) (Reverse Engineer Anything) and uses it, but works without it too.
80
-
81
- | | REA on its own | trueclone |
82
- |---|---|---|
83
- | What it is | A microscope: typed tools that let your agent look inside apps, binaries and pages | A factory with quality control: capture, design system, build guide, scores, report |
84
- | Capture | Your Chrome over CDP, one page state | 3 widths, every scroll state, motion runtime, assets, network |
85
- | Knows when the clone is right | No: the agent judges by eye | Yes: 7 measured checks with targets |
86
- | Tells you what's wrong | No | "Footer column 2 is at x 189, the original's at 732", plus the original's CSS rule |
87
- | Outputs | Whatever the agent writes | HTML clone, Next.js app, DESIGN.md, globals.css, motion recipe, REPORT.md |
88
-
89
- REA answers *how does this work?* trueclone answers *is my copy identical, and where isn't it?* For the hard parts (a bundled physics rule, a shader, the API a component calls) the skill sends your agent to REA.
81
+ trueclone bundles [REA](https://github.com/morluto/rea) (Reverse Engineer Anything) for the rare part no script reading explains (a bundled physics rule, a shader, WASM): `npx trueclone rea --help`. It is not part of the standard flow: the site's scripts, saved and ranked by the capture, carry what the agent needs.
90
82
 
91
83
  ## Modes and guardrails
92
84
 
@@ -103,19 +95,22 @@ REA answers *how does this work?* trueclone answers *is my copy identical, and w
103
95
 
104
96
  ```bash
105
97
  npx @trueclone/cli init # set up a project for your agent (then `npx trueclone ...`)
106
- npx trueclone doctor # check Node, Chrome and REA
98
+ npx trueclone doctor # check Node and Chrome
107
99
  npx trueclone <url> [name] # capture + design-system draft
108
100
  npx trueclone capture <url> [name] [--mode exact --permission "<owner>"]
109
101
  npx trueclone motion <name> # motion inventory
110
102
  npx trueclone design <name> # DESIGN.md + globals.css
111
103
  npx trueclone reveals <name> # record scroll reveals and scroll class changes
112
104
  npx trueclone widgets <name> # record every tab and menu state
113
- npx trueclone seed <name> # clone/ from the mirror (template: placeholder text and media)
105
+ npx trueclone seed <name> # clone/ from the mirror (template: placeholder text and media); --patches re-applies clone/patches/ in a second
114
106
  npx trueclone rules <name> "<selector>" # the original's CSS rules for an element
115
107
  npx trueclone textshape <name> --write # template mode: placeholder text with the original's line breaks
116
- npx trueclone score <name> [--quick] # every check, then REPORT.md
117
- npx trueclone next <name> --verify # Next.js + Tailwind + shadcn app, scored
118
- npx trueclone rea <args> # bundled REA
108
+ npx trueclone load <name> # load sequence on both sides: settledness 0.25-4 s after navigation, cold cache, same network
109
+ npx trueclone score <name> [--quick] # every check, then REPORT.md (--only detail,load --viewport tablet after a fix; unchanged checks are kept)
110
+ npx trueclone explain <name> <y> # one place on the page: what every check saw there, plus a reference|clone|diff crop
111
+ npx trueclone review <name> --size wide # reference and clone side by side at a held-out size: load, every section, hover
112
+ npx trueclone next <name> --verify # Next.js + Tailwind + shadcn app, scored into REPORT.next.md
113
+ npx trueclone rea <args> # bundled REA, for the rare deep case
119
114
  npx trueclone help # every step
120
115
  ```
121
116
 
@@ -134,7 +129,7 @@ Node 22.19+, 24.11+ or 26+ (the same as REA), and Google Chrome for captures (it
134
129
 
135
130
  - **Whole sites:** crawl every route, shared header and footer as components.
136
131
  - **Data layer:** captured API responses turned into mock routes and fixtures, so the clone works offline.
137
- - **Benchmarks:** React with Framer Motion, a Next.js site with source maps, a data-driven app; each scored with and without REA.
132
+ - **Benchmarks by kind of site:** static, Tailwind/Next.js, Webflow, Webflow + GSAP, Framer, WordPress builders, WebGL showcases, docs, e-commerce, SPAs, scroll-jacking, embeds: each scored at build and held-out sizes, published here.
138
133
  - **A free web page:** paste a URL, get its DESIGN.md and globals.css.
139
134
 
140
135
  ## Credits
package/bin/trueclone.mjs CHANGED
@@ -19,22 +19,28 @@ const require = createRequire(import.meta.url);
19
19
  const { name, version } = JSON.parse(await fs.readFile(path.join(PKG, 'package.json'), 'utf8'));
20
20
 
21
21
  const STEPS = {
22
- capture: ['capture.mjs', 'capture a live page: screenshots at 390/768/1440, sections, tokens, rendered HTML, assets'],
23
- seed: ['seed.mjs', 'exact mode: start clone/ from the script-free mirror (markup, text, styles, media)'],
22
+ capture: ['capture.mjs', 'capture a live page at 5 sizes (3 to build from, 2 held out), its scripts, assets, rendered HTML, inline values at extra sizes'],
23
+ seed: ['seed.mjs', 'clone/ from the script-free mirror: structure, viewport functions, pins, reveals (--patches: re-apply clone/patches/ only)'],
24
24
  design: ['design.mjs', 'design-tokens.json -> DESIGN.md (linted), shadcn globals.css, clone/tokens.css'],
25
25
  motion: ['motion.mjs', 'motion inventory from the live runtime (GSAP/ScrollTrigger, Lenis, Swiper, CSS animations)'],
26
+ understand: ['understand.mjs', 'optional, rarely needed: REA code evidence -> reference/code.md (capture already saves every script, ranked, in reference/scripts/)'],
26
27
  reveals: ['reveals.mjs', 'record one-shot scroll reveals frame by frame, so seed can replay them (any library)'],
27
28
  widgets: ['widgets.mjs', 'record every tab and menu state by clicking it, so seed can replay them (template mode)'],
28
- states: ['states.mjs', 'live scroll states; --clone scores the rebuild step by step'],
29
+ states: ['states.mjs', 'live scroll states at every scored size; --clone scores the rebuild step by step (canvases too)'],
29
30
  curves: ['curves.mjs', 'motion score: sampled property curves, reference vs clone (motion-map.json)'],
30
31
  interact: ['interact.mjs', 'interaction tests: click/hover/drag/key scenarios on both sides (interactions.json)'],
31
32
  detail: ['detail.mjs', 'style diff at matching points: font, size, weight, line height, colour, background, radius'],
32
- hovers: ['hovers.mjs', 'hover every interactive element on both sides and compare what changes'],
33
- rules: ['rules.mjs', 'every CSS rule matching an element on the reference (+ ::before/::after, @media)'],
33
+ hovers: ['hovers.mjs', 'hover every control and every element with a hover listener on both sides; the outcome must match'],
34
+ load: ['load.mjs', 'load sequence: how settled each side is 0.25/0.5/1/2/4 s after navigation, cold cache, same network'],
35
+ drift: ['drift.mjs', 'has the live page changed since the capture? (page height and sections per width vs meta.json)'],
36
+ owned: ['owned.mjs', 'which sections of the clone script.js rebuilds at boot (load with scripts blocked vs running)'],
37
+ rules: ['rules.mjs', 'every CSS rule matching an element on the reference (+ ::before/::after, @media); --clone for the clone'],
38
+ review: ['review.mjs', 'reference and clone side by side at a held-out size: load, every section, hover; verdicts go in notes.md'],
39
+ explain: ['explain.mjs', 'one place on the page: what layout, states, detail and hovers saw there, plus a reference|clone|diff crop'],
34
40
  textshape: ['textshape.mjs', 'template mode: placeholder words with the reference word widths, so lines break the same'],
35
41
  diff: ['diff.mjs', 'full-page pixel diff per viewport and section (--structure masks text and media)'],
36
42
  spec: ['spec.mjs', 'measured boxes and styles of the live page (--clone for the rebuild)'],
37
- tiles: ['tiles.mjs', 'cut a tall screenshot into readable tiles'],
43
+ tiles: ['tiles.mjs', 'the full-page shots cut into one tile per screen: tiles <name> [--size wide] [--clone], or tiles <file.png>'],
38
44
  next: ['next.mjs', 'Next.js + Tailwind v4 + shadcn app from the verified clone; --verify scores it'],
39
45
  score: ['score.mjs', 'every score in parallel groups, time limit per step, writes REPORT.md'],
40
46
  report: ['report.mjs', 'REPORT.md from the latest scores: every number, every gap, what cannot be copied'],
@@ -51,14 +57,14 @@ const run = (file, args) => new Promise((resolve) => {
51
57
  });
52
58
 
53
59
  function help() {
54
- console.log(`trueclone ${version}: a working twin of a website, proven by scores
60
+ console.log(`trueclone ${version}: the exact same website, at every screen size
55
61
 
56
62
  usage
57
- npx @trueclone/cli init set up this folder for your coding agent (installs trueclone)
63
+ npx @trueclone/cli init set up this folder for your coding agent (installs trueclone; --with-rea adds REA as MCP)
58
64
  trueclone <url> [name] capture + design-system draft (template mode)
59
65
  trueclone <step> <name> [options] run one step
60
- trueclone rea <args> bundled REA (Reverse Engineer Anything)
61
- trueclone doctor check Node, browsers and REA
66
+ trueclone rea <args> bundled REA (Reverse Engineer Anything), for the rare deep case
67
+ trueclone doctor check Node and browsers
62
68
 
63
69
  steps
64
70
  ${Object.entries(STEPS).map(([k, [, d]]) => ` ${k.padEnd(10)} ${d}`).join('\n')}
@@ -80,11 +86,15 @@ async function init() {
80
86
  let agents = (await exists(agentsFile)) ? await fs.readFile(agentsFile, 'utf8') : '';
81
87
  agents = agents.includes('<!-- trueclone:start -->') ? agents.replace(/<!-- trueclone:start -->[\s\S]*?<!-- trueclone:end -->\n?/, block) : `${agents}${agents && !agents.endsWith('\n\n') ? '\n' : ''}${block}`;
82
88
  await fs.writeFile(agentsFile, agents);
83
- // 3. REA as an MCP server for agents that speak MCP (Claude Code reads .mcp.json)
84
- const mcpFile = path.join(cwd, '.mcp.json');
85
- const mcp = (await exists(mcpFile)) ? JSON.parse(await fs.readFile(mcpFile, 'utf8')) : {};
86
- mcp.mcpServers = { ...(mcp.mcpServers || {}), rea: { command: 'npx', args: ['-y', `${name}@${version}`, 'rea', 'mcp'] } };
87
- await fs.writeFile(mcpFile, `${JSON.stringify(mcp, null, 2)}\n`);
89
+ // 3. REA as an MCP server only on request (--with-rea): the standard flow reads the site's own scripts from the
90
+ // capture; REA is for the rare deep case and stays reachable as `npx trueclone rea` either way
91
+ const withRea = process.argv.includes('--with-rea');
92
+ if (withRea) {
93
+ const mcpFile = path.join(cwd, '.mcp.json');
94
+ const mcp = (await exists(mcpFile)) ? JSON.parse(await fs.readFile(mcpFile, 'utf8')) : {};
95
+ mcp.mcpServers = { ...(mcp.mcpServers || {}), rea: { command: 'npx', args: ['-y', `${name}@${version}`, 'rea', 'mcp'] } };
96
+ await fs.writeFile(mcpFile, `${JSON.stringify(mcp, null, 2)}\n`);
97
+ }
88
98
  // 4. captures, clones and browser profiles stay local
89
99
  const ignoreFile = path.join(cwd, '.gitignore');
90
100
  let ignore = (await exists(ignoreFile)) ? await fs.readFile(ignoreFile, 'utf8') : '';
@@ -103,8 +113,7 @@ async function init() {
103
113
  console.log(`trueclone set up in ${cwd}
104
114
  .claude/skills/trueclone/SKILL.md the workflow your agent follows
105
115
  AGENTS.md points Codex, Cursor and others at the skill
106
- .mcp.json REA as an MCP server
107
- .gitignore sites/ stays local
116
+ ${withRea ? ' .mcp.json REA as an MCP server\n' : ''} .gitignore sites/ stays local
108
117
  package.json ${installed}
109
118
  next: npx trueclone doctor, then ask your agent "clone <url> with trueclone"`);
110
119
  }
@@ -120,8 +129,8 @@ async function doctor() {
120
129
  try { const b = await chromium.launch(opts); await b.close(); console.log(`ok ${label}`); }
121
130
  catch { console.log(`warn ${label} not found${opts.channel ? ': install Google Chrome' : ': npx playwright install chromium'}`); }
122
131
  }
123
- try { require.resolve('rea-agents/package.json'); console.log('ok REA bundled (trueclone rea doctor checks its decompilers)'); }
124
- catch { console.log('warn REA not installed'); ok = false; }
132
+ try { require.resolve('rea-agents/package.json'); console.log('ok REA bundled, for the rare deep case (trueclone rea doctor checks its decompilers)'); }
133
+ catch { console.log('warn REA not installed (optional)'); }
125
134
  process.exitCode = ok ? 0 : 1;
126
135
  }
127
136
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@trueclone/cli",
3
- "version": "0.2.0",
3
+ "version": "0.4.0-rc.1",
4
4
  "description": "A working twin of any website from its URL: HTML clone, Next.js + Tailwind + shadcn app, DESIGN.md and globals.css, proven by layout, motion, interaction and style scores. Bundles REA.",
5
5
  "type": "module",
6
6
  "bin": {
@@ -22,6 +22,7 @@
22
22
  },
23
23
  "dependencies": {
24
24
  "@google/design.md": "^0.4.0",
25
+ "js-beautify": "2.0.3",
25
26
  "parse5": "^8.0.1",
26
27
  "pixelmatch": "^7.2.0",
27
28
  "playwright": "^1.64.0",
@@ -5,7 +5,8 @@
5
5
  import fs from 'node:fs/promises';
6
6
  import path from 'node:path';
7
7
  import crypto from 'node:crypto';
8
- import { VIEWPORTS, parseArgs, siteName, openBrowser, newPage, settle, sectionsOf, mask } from './lib.mjs';
8
+ import { SCORED, VIEWPORTS, HELD_OUT, parseArgs, siteName, openBrowser, newPage, settle, sectionsOf, mask } from './lib.mjs';
9
+ import { startScriptEvidence } from './scripts-evidence.mjs';
9
10
 
10
11
  const { pos, flags } = parseArgs(process.argv.slice(2));
11
12
  const [url, nameArg] = pos;
@@ -92,10 +93,14 @@ const browser = await openBrowser(flags.cdp);
92
93
  const manifest = [];
93
94
  const summary = [];
94
95
  const rootFontSize = {};
96
+ const inlineStyles = {};
97
+ const scriptQueries = new Set(); // media queries the page's scripts asked (matchMedia), at any size
95
98
  try {
96
- for (const vp of VIEWPORTS) {
99
+ // the seed sizes and the held-out sizes: the held-out ones are shot and measured for scoring, nothing else
100
+ for (const vp of SCORED) {
97
101
  const desktop = vp.name === 'desktop';
98
102
  const { context, page } = await newPage(browser, vp, desktop ? { recordHar: { path: path.join(ref, 'desktop.har'), content: 'omit' } } : {});
103
+ await page.addInitScript(recordMediaQueries);
99
104
  if (desktop) {
100
105
  page.on('response', async (res) => {
101
106
  const rawType = res.request().resourceType();
@@ -114,6 +119,8 @@ try {
114
119
  } catch { /* redirects, data: urls, aborted */ }
115
120
  });
116
121
  }
122
+ // desktop: the page's own scripts and what ran, saved as evidence (reference/scripts/)
123
+ const finishEvidence = desktop ? await startScriptEvidence(page, url).catch(() => null) : null;
117
124
  await page.goto(url, { waitUntil: 'load', timeout: 60000 });
118
125
  // Components that mount only while in view (a lazy wordmark, an embed) are often unmounted again by the time
119
126
  // the DOM is saved. Watch from load: an element that was empty, got content while scrolled into view and is empty
@@ -137,6 +144,16 @@ try {
137
144
  }).observe(document.body, { subtree: true, childList: true });
138
145
  });
139
146
  await settle(page, hide);
147
+ // inline styles at this size, by element path: scripts size and place things from the viewport (a grid's computed
148
+ // cells, a pin's distance), and the rendered HTML only keeps one size's values. With the extra sizes read below,
149
+ // the seed tells a function of the viewport from a constant. Read here, settled and before any screenshot, in the
150
+ // same state as the extra sizes (a full-page shot mounts content and fires reveals).
151
+ if (!vp.heldOut) inlineStyles[`${vp.width}x${vp.height}`] = await page.evaluate(readInlineStyles);
152
+ for (const q of await page.evaluate(() => window.__tcMediaQueries || []).catch(() => [])) scriptQueries.add(q);
153
+ if (finishEvidence) {
154
+ const ev = await finishEvidence(path.join(ref, 'scripts')).catch((e) => { console.warn(`WARNING scripts not saved: ${e.message}`); return null; });
155
+ if (ev) console.log(`scripts: ${ev.scripts} saved to reference/scripts/ (${ev.ran} ran during load and scroll), ${ev.listeners} event-listener handlers located (INDEX.md)`);
156
+ }
140
157
  // A full-page screenshot brings the whole page into view at once, which mounts view-mounted components and starts
141
158
  // their reveals: shoot until two shots agree, so the reference shows a settled page, not a fade caught at 0%.
142
159
  let shot = await page.screenshot({ fullPage: true });
@@ -162,6 +179,23 @@ try {
162
179
  el.setAttribute('data-tc-rich', String(i));
163
180
  restore[i++] = best.html;
164
181
  }
182
+ // open shadow roots (web components, a Webflow code island) are not part of outerHTML: each becomes a
183
+ // declarative <template shadowrootmode="open"> inside its host, which the browser turns back into a shadow
184
+ // root while parsing, no script needed. Deepest first, so a host inside a shadow root is carried too.
185
+ // Constructed stylesheets (adoptedStyleSheets) are written out as a <style> at the top of the template.
186
+ const hosts = [];
187
+ (function find(scope) { for (const el of scope.querySelectorAll('*')) if (el.shadowRoot) { find(el.shadowRoot); hosts.push(el); } })(document);
188
+ for (const host of hosts) {
189
+ const t = document.createElement('template');
190
+ t.setAttribute('shadowrootmode', 'open');
191
+ t.setAttribute('shadowrootserializable', '');
192
+ const adopted = (host.shadowRoot.adoptedStyleSheets || []).map((s) => { try { return [...s.cssRules].map((r) => r.cssText).join('\n'); } catch { return ''; } }).filter(Boolean);
193
+ if (adopted.length) { const st = document.createElement('style'); st.textContent = adopted.join('\n'); t.content.appendChild(st); }
194
+ for (const n of host.shadowRoot.childNodes) t.content.appendChild(n.cloneNode(true));
195
+ host.prepend(t);
196
+ host.setAttribute('data-tc-shadow', '');
197
+ }
198
+ if (hosts.length) console.log(`tc-shadow: serialised ${hosts.length} open shadow root(s) as declarative shadow DOM`);
165
199
  const doc = document.documentElement.cloneNode(true);
166
200
  for (const el of document.querySelectorAll('[data-tc-rich]')) el.removeAttribute('data-tc-rich');
167
201
  for (const el of doc.querySelectorAll('[data-tc-rich]')) {
@@ -180,7 +214,7 @@ try {
180
214
  }
181
215
  await mask(page);
182
216
  await page.screenshot({ path: path.join(ref, `${vp.name}.mask.png`), fullPage: true });
183
- summary.push({ viewport: vp.name, width: vp.width, pageHeight: height, sections: sections.length });
217
+ summary.push({ viewport: vp.name, width: vp.width, height: vp.height, ...(vp.heldOut ? { heldOut: true } : {}), pageHeight: height, sections: sections.length });
184
218
  await context.close(); // flushes the HAR
185
219
  }
186
220
  } finally {
@@ -188,6 +222,33 @@ try {
188
222
  }
189
223
 
190
224
  await fs.writeFile(path.join(ref, 'assets', 'manifest.json'), JSON.stringify(manifest, null, 2));
225
+
226
+ // The site's own breakpoints (its @media width queries) and extra sizes inside every range they make: inline styles
227
+ // read there let the seed fit each value a script set as a function of the viewport (a·vw + b·vh + c) per range, or
228
+ // see that it is none and leave it to the agent. Never the held-out sizes: those stay unseen by anything that builds.
229
+ const breakpoints = await siteBreakpoints(manifest, [...scriptQueries]);
230
+ const fitSizes = sizesForFit(breakpoints);
231
+ if (fitSizes.length) {
232
+ const queue = [...fitSizes];
233
+ const browser2 = await openBrowser(flags.cdp);
234
+ try {
235
+ await Promise.all(Array.from({ length: 3 }, async () => {
236
+ for (let s = queue.shift(); s; s = queue.shift()) {
237
+ const { context, page } = await newPage(browser2, { name: `fit${s.width}`, width: s.width, height: s.height });
238
+ try {
239
+ await page.goto(url, { waitUntil: 'load', timeout: 60000 });
240
+ await settle(page, hide);
241
+ inlineStyles[`${s.width}x${s.height}`] = await page.evaluate(readInlineStyles);
242
+ } catch (e) { console.warn(`WARNING inline styles at ${s.width}x${s.height} not read: ${e.message.split('\n')[0]}`); }
243
+ finally { await context.close(); }
244
+ }
245
+ }));
246
+ } finally {
247
+ if (!flags.cdp) await browser2.close();
248
+ }
249
+ }
250
+ await fs.writeFile(path.join(ref, 'inline-styles.json'), JSON.stringify({ breakpoints, scriptQueries: [...scriptQueries], sizes: Object.keys(inlineStyles), styles: inlineStyles }));
251
+ console.log(`inline styles read at ${Object.keys(inlineStyles).length} sizes (site breakpoints: ${breakpoints.join(', ') || 'none found'}; ${scriptQueries.size} width quer${scriptQueries.size === 1 ? 'y' : 'ies'} asked by its scripts)`);
191
252
  await fs.writeFile(path.join(ref, 'meta.json'), JSON.stringify({ url, name, mode, permission, hide, capturedAt: new Date().toISOString(), rootFontSize, viewports: summary }, null, 2));
192
253
  await buildMirror(manifest, url, mode === 'exact' ? `Exact mode, authorised by: ${permission}.` : "Template mode: the site's own content, kept as a reference only.");
193
254
  console.table(summary);
@@ -202,6 +263,24 @@ async function buildMirror(manifest, url, note) {
202
263
  const mirror = path.join(ref, 'mirror');
203
264
  await fs.rm(mirror, { recursive: true, force: true });
204
265
  await fs.mkdir(path.join(mirror, 'css'), { recursive: true });
266
+ // Rendered twice: the first pass shows which media the capture never saw (lazy images below the fold, hover and
267
+ // state images, sources behind a media query, fonts behind @import); fetchMissing() saves them and the second pass
268
+ // points at the local copies. A page is only as complete as its slowest-loading image.
269
+ let out = await renderMirror(manifest, url, note, mirror);
270
+ const missing = findRemoteMedia(out);
271
+ if (missing.length) {
272
+ const added = await fetchMissing(missing, manifest, url);
273
+ if (added) out = await renderMirror(manifest, url, note, mirror);
274
+ await fs.writeFile(path.join(ref, 'assets', 'manifest.json'), JSON.stringify(manifest, null, 2));
275
+ console.log(`mirror: fetched ${added}/${missing.length} media the capture had not seen (lazy, hover or conditional sources)`);
276
+ }
277
+ for (const [file, css] of out.css) await fs.writeFile(path.join(mirror, 'css', file), css);
278
+ await fs.writeFile(path.join(mirror, 'index.html'), out.html);
279
+ const left = findRemoteMedia(out);
280
+ console.log(`mirror: reference/mirror/index.html (${manifest.length} local assets${left.length ? `, ${left.length} remote media left: ${left.slice(0, 3).join(', ')}` : ''})`);
281
+ }
282
+
283
+ async function renderMirror(manifest, url, note, mirror) {
205
284
  const byUrl = new Map(manifest.map((m) => [m.url.split('#')[0], m]));
206
285
  const abs = (u, base) => { try { return new URL(u.trim().replace(/^['"]|['"]$/g, '').replaceAll('&amp;', '&'), base).href.split('#')[0]; } catch { return null; } };
207
286
  const cssName = (m) => path.basename(m.file);
@@ -218,9 +297,10 @@ async function buildMirror(manifest, url, note) {
218
297
  .replace(/@import\s+(['"])([^'"]+)\1/g, (all, q, u) => `@import "${local(abs(u, base), fromDir) || u}"`);
219
298
 
220
299
  // stylesheets: rewritten copies under mirror/css
300
+ const css = new Map();
221
301
  for (const m of manifest.filter((x) => x.type === 'stylesheet')) {
222
- const css = await fs.readFile(path.join(ref, m.file), 'utf8');
223
- await fs.writeFile(path.join(mirror, 'css', cssName(m)), rewriteCssUrls(css, m.url, path.join(mirror, 'css')));
302
+ const text = await fs.readFile(path.join(ref, m.file), 'utf8').catch(() => null);
303
+ if (text !== null) css.set(cssName(m), rewriteCssUrls(text, m.url, path.join(mirror, 'css')));
224
304
  }
225
305
 
226
306
  let html = await fs.readFile(path.join(ref, 'rendered.html'), 'utf8');
@@ -235,14 +315,164 @@ async function buildMirror(manifest, url, note) {
235
315
  const l = local(abs(u, url), mirror);
236
316
  return l ? ` ${attr}=${q}${l}${q}` : all;
237
317
  })
318
+ // a subresource-integrity hash belongs to the bytes at the original URL; after the rewrite it only makes the
319
+ // browser refuse the local copy (and the whole stylesheet with it)
320
+ .replace(/<link\b[^>]*>/gi, (tag) => tag.replace(/\s(?:integrity|crossorigin)(?:=("[^"]*"|'[^']*'|[^\s>]+))?/gi, ''))
238
321
  .replace(/\s(srcset|data-srcset)=("|')([^"']+)\2/gi, (all, attr, q, list) => ` ${attr}=${q}${list.split(',').map((part) => {
239
322
  const [u, ...rest] = part.trim().split(/\s+/);
240
323
  return [local(abs(u, url), mirror) || u, ...rest].join(' ');
241
324
  }).join(', ')}${q}`)
242
325
  .replace(/<style\b([^>]*)>([\s\S]*?)<\/style>/gi, (all, attrs, css) => `<style${attrs}>${rewriteCssUrls(css, url, mirror)}</style>`)
243
- .replace(/\sstyle=("|')([^"']*url\([^"']*)\1/gi, (all, q, css) => ` style=${q}${rewriteCssUrls(css, url, mirror)}${q}`)
326
+ // quotes inside a style attribute are HTML-escaped (url(&quot;…&quot;)): unescape before rewriting, re-escape after
327
+ .replace(/\sstyle=(?:"([^"]*)"|'([^']*)')/gi, (all, d, s) => {
328
+ const value = d ?? s;
329
+ if (!/url\(/i.test(value)) return all;
330
+ const plain = value.replaceAll('&quot;', '"').replaceAll('&#39;', "'");
331
+ return ` style="${rewriteCssUrls(plain, url, mirror).replaceAll('"', '&quot;')}"`;
332
+ })
244
333
  .replace(/<head([^>]*)>/i, `<head$1>\n<!-- trueclone local mirror of ${url}. ${note.replace(/--/g, '-')} Local reference only: do not deploy. -->\n<meta name="robots" content="noindex,nofollow">`);
245
- await fs.writeFile(path.join(mirror, 'index.html'), html);
246
- const missing = [...html.matchAll(/\s(?:src|href)=["'](https?:[^"']+)["']/gi)].map((m) => m[1]).filter((u) => !/^https?:\/\/[^/]+\/?$/.test(u));
247
- console.log(`mirror: reference/mirror/index.html (${manifest.length} local assets, ${missing.length} remote URLs left, e.g. links)`);
334
+ return { html, css };
335
+ }
336
+
337
+ // Media the page still points at on the network after the rewrite: image/video/font/stylesheet URLs in src, srcset,
338
+ // poster, data-src, <link> hrefs and CSS url(). Links to other pages (<a href>) are not media.
339
+ function findRemoteMedia({ html, css }) {
340
+ const urls = new Set();
341
+ const media = (u) => u && /^https?:\/\//.test(u) && !/^https?:\/\/[^/]+\/?$/.test(u) && !/\.(js|mjs|json|html?)(\?|$)/i.test(u);
342
+ const add = (u) => { if (media(u)) urls.add(u.replaceAll('&amp;', '&')); };
343
+ for (const m of html.matchAll(/\s(?:src|poster|data-src)=["']([^"']+)["']/gi)) add(m[1]);
344
+ for (const m of html.matchAll(/\s(?:srcset|data-srcset)=["']([^"']+)["']/gi)) for (const part of m[1].split(',')) add(part.trim().split(/\s+/)[0]);
345
+ for (const m of html.matchAll(/<link\b[^>]*>/gi)) {
346
+ const rel = /\srel=["']?([^"'\s>]+)/i.exec(m[0])?.[1] || '';
347
+ const href = /\shref=["']([^"']+)["']/i.exec(m[0])?.[1];
348
+ if (/stylesheet|icon|apple-touch-icon|preload|prefetch/i.test(rel) && !/\sas=["']?(script|document|fetch)/i.test(m[0])) add(href);
349
+ }
350
+ const cssUrls = (text) => { for (const m of text.matchAll(/url\(\s*(?:&quot;|["'])?(https?:\/\/[^)"'&\s]+)/gi)) add(m[1]); };
351
+ cssUrls(html);
352
+ for (const text of css.values()) cssUrls(text);
353
+ return [...urls].slice(0, 300);
354
+ }
355
+
356
+ async function fetchMissing(urls, manifest, pageUrl) {
357
+ const typeOf = (u, contentType = '') => {
358
+ if (/text\/css/.test(contentType) || /\.css(\?|$)/i.test(u)) return 'stylesheet';
359
+ if (/^font\/|application\/(x-)?font|application\/vnd\.ms-fontobject/.test(contentType) || /\.(woff2?|ttf|otf|eot)(\?|$)/i.test(u)) return 'font';
360
+ if (/^image\//.test(contentType) || /\.(png|jpe?g|gif|webp|avif|svg|ico|bmp)(\?|$)/i.test(u)) return 'image';
361
+ return 'media';
362
+ };
363
+ let added = 0;
364
+ const queue = [...urls];
365
+ const worker = async () => {
366
+ for (let u = queue.shift(); u; u = queue.shift()) {
367
+ try {
368
+ const res = await fetch(u, { headers: { 'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36', referer: pageUrl, accept: '*/*' }, signal: AbortSignal.timeout(20000), redirect: 'follow' });
369
+ if (!res.ok) continue;
370
+ const body = Buffer.from(await res.arrayBuffer());
371
+ if (!body.length || body.length > MAX_ASSET_BYTES) continue;
372
+ const contentType = res.headers.get('content-type') || '';
373
+ if (/text\/html/.test(contentType)) continue; // a page, not media (a soft 404)
374
+ const type = typeOf(u, contentType);
375
+ const parsed = new URL(u);
376
+ const base = (path.basename(parsed.pathname) || 'index').replace(/[^a-z0-9._-]/gi, '_').slice(-80);
377
+ const hash = crypto.createHash('sha1').update(u).digest('hex').slice(0, 8);
378
+ const file = path.join('assets', type, `${hash}-${base}`);
379
+ await fs.mkdir(path.join(ref, 'assets', type), { recursive: true });
380
+ await fs.writeFile(path.join(ref, file), body);
381
+ manifest.push({ url: u, type, file: file.replaceAll('\\', '/'), contentType, bytes: body.length, fetched: 'mirror' });
382
+ added++;
383
+ } catch { /* offline, blocked, timed out: stays remote and is listed */ }
384
+ }
385
+ };
386
+ await Promise.all(Array.from({ length: 6 }, worker));
387
+ return added;
388
+ }
389
+
390
+ // Inline styles and classes of every styled element, by element path, with the size actually laid out (clientWidth
391
+ // excludes a classic scrollbar; scripts measure that, not the window).
392
+ function readInlineStyles() {
393
+ const SKIP = new Set(['SCRIPT', 'NOSCRIPT', 'TEMPLATE', 'STYLE', 'LINK', 'META']);
394
+ const pathOf = (el) => {
395
+ const parts = [];
396
+ for (let n = el; n && n !== document.body; n = n.parentElement) {
397
+ const sibs = [...n.parentElement.children].filter((c) => !SKIP.has(c.tagName));
398
+ parts.unshift(`${n.tagName.toLowerCase()}:${sibs.indexOf(n) + 1}`);
399
+ }
400
+ return parts.join('>');
401
+ };
402
+ const styles = {};
403
+ for (const el of document.body.querySelectorAll('[style]')) {
404
+ if (SKIP.has(el.tagName) || (el.closest('svg') && el.tagName !== 'svg')) continue;
405
+ styles[pathOf(el)] = [el.getAttribute('style'), String(el.getAttribute('class') || '')];
406
+ }
407
+ return { w: document.documentElement.clientWidth, h: innerHeight, styles };
408
+ }
409
+
410
+ // Every media query the page's scripts ask (GSAP matchMedia, Webflow, a component's own breakpoint): what a script
411
+ // does can switch at a width no stylesheet mentions.
412
+ function recordMediaQueries() {
413
+ const seen = (window.__tcMediaQueries = []);
414
+ const original = window.matchMedia;
415
+ if (!original) return;
416
+ window.matchMedia = function matchMedia(q) {
417
+ if (typeof q === 'string' && /width/.test(q) && !seen.includes(q)) seen.push(q);
418
+ return original.call(this, q);
419
+ };
420
+ }
421
+
422
+ // The width breakpoints the site's stylesheets use at least twice (min-width: 992px is a boundary at 991), and every
423
+ // width its scripts ask about (always kept: a script's breakpoint is behaviour).
424
+ async function siteBreakpoints(manifest, scriptQueries = []) {
425
+ const count = new Map();
426
+ for (const q of scriptQueries) {
427
+ for (const m of q.matchAll(/(min|max)-width\s*:\s*([\d.]+)(px|em|rem)/g)) {
428
+ const px = Number(m[2]) * (m[3] === 'px' ? 1 : 16);
429
+ const edge = Math.round(m[1] === 'min' ? px - 1 : px);
430
+ if (edge >= 300 && edge <= 2600) count.set(edge, (count.get(edge) || 0) + 100);
431
+ }
432
+ }
433
+ const texts = [];
434
+ for (const m of manifest.filter((x) => x.type === 'stylesheet')) texts.push(await fs.readFile(path.join(ref, m.file), 'utf8').catch(() => ''));
435
+ const rendered = await fs.readFile(path.join(ref, 'rendered.html'), 'utf8').catch(() => '');
436
+ for (const m of rendered.matchAll(/<style\b[^>]*>([\s\S]*?)<\/style>/gi)) texts.push(m[1]);
437
+ for (const t of texts) {
438
+ for (const q of t.matchAll(/@media[^{]*/g)) {
439
+ for (const m of q[0].matchAll(/(min|max)-width\s*:\s*([\d.]+)(px|em|rem)/g)) {
440
+ const px = Number(m[2]) * (m[3] === 'px' ? 1 : 16);
441
+ const edge = Math.round(m[1] === 'min' ? px - 1 : px);
442
+ if (edge >= 300 && edge <= 2600) count.set(edge, (count.get(edge) || 0) + 1);
443
+ }
444
+ }
445
+ }
446
+ // breakpoints within 2px of each other are one (991 from max-width, 991 from min-width: 992)
447
+ const edges = [...count].filter(([, n]) => n >= 2).sort((a, b) => a[0] - b[0]);
448
+ const merged = [];
449
+ for (const [e, n] of edges) { const last = merged[merged.length - 1]; if (last && e - last.e <= 2) last.n += n; else merged.push({ e, n }); }
450
+ return merged.sort((a, b) => b.n - a.n).slice(0, 9).map((x) => x.e).sort((a, b) => a - b);
451
+ }
452
+
453
+ // Extra sizes for fitting: at least three points in every breakpoint range (counting the seed sizes in it), at
454
+ // different heights, so a value that depends on the width, the height or both can be told apart from a constant.
455
+ function sizesForFit(breakpoints) {
456
+ const edges = [320, ...breakpoints, 2400];
457
+ const taken = new Set([...VIEWPORTS, ...HELD_OUT].map((v) => `${v.width}x${v.height}`));
458
+ const out = [];
459
+ const heights = [760, 1000, 880];
460
+ for (let i = 0; i < edges.length - 1; i++) {
461
+ const lo = i === 0 ? edges[0] : edges[i] + 1, hi = edges[i + 1];
462
+ if (hi - lo < 8) { // a narrow band (760 and 767 both breakpoints) still has its own rules: sample both edges
463
+ for (const [w, h] of [[lo, heights[i % 3]], [hi, heights[(i + 1) % 3]]]) if (!taken.has(`${w}x${h}`)) { taken.add(`${w}x${h}`); out.push({ width: w, height: h }); }
464
+ continue;
465
+ }
466
+ const seeds = VIEWPORTS.filter((v) => v.width >= lo && v.width <= hi).length;
467
+ const want = Math.max(2, 3 - seeds);
468
+ for (let k = 0; k < want; k++) {
469
+ const w = Math.round(lo + (hi - lo) * [0.25, 0.75, 0.5][k]);
470
+ const h = heights[(i + k) % heights.length];
471
+ if (!taken.has(`${w}x${h}`)) { taken.add(`${w}x${h}`); out.push({ width: w, height: h }); }
472
+ }
473
+ }
474
+ // Two very wide, short screens: most sizes above are taller than 16:9, so a "cover" rule (max(100vw, 177.8vh))
475
+ // would have one point at most on its width side. Two make it checkable (see fitCover in viewport-fit.mjs).
476
+ for (const [w, h] of [[2400, 760], [2200, 900]]) if (!taken.has(`${w}x${h}`)) { taken.add(`${w}x${h}`); out.push({ width: w, height: h }); }
477
+ return out;
248
478
  }