@tractiontactics/tt-fidelity 0.2.1 → 0.2.3

package/AGENTS.md

@@ -10,7 +10,11 @@ You are measuring a prototype against a TT WP Theme draft. **You are not the gra
 4. **One WORK PLAN task**, then re-run with `--round` incremented. Do not batch speculative fixes.
 5. **Never apply a rule site-wide** until `instances_per_page` (in clusters / QUEUE) shows it is a real default. Blanket CSS from a rare seam creates hundreds of false findings.
 6. **Cache:** the tool cache-busts by default. If draft and live disagree, purge host/CDN and re-run — do not “fix” ghosts.
+6b. **Capture prep (0.2.3+):** the tool scrolls the full page, forces lazy images, and unlocks common scroll-reveal (`opacity:0`) before section shots. If prototype PNGs look blank cream with only sticky widgets, you are on an old package — upgrade. Separately, **404 images** on the prototype (or `onerror` hiding media) are real source defects, not capture bugs.
 7. **Escalate** tokens → block Style/Advanced → minimal `PUT /custom-css` → theme change → HTML template. Cite `ttRowId` / `ttBlockIds` when present.
+8. **Default `--scope content`** (page body). Use `--scope chrome` for header/footer. Use `--scope full` only when you intentionally want both (TT draft may then count header/footer rows the prototype does not).
+9. **QUEUE lifetime:** `ttRowId` values persist across *renders* of the same layout JSON, but a **republish / rebuild that regenerates row ids** invalidates the QUEUE. Re-measure after any full layout rewrite.
+10. **`--only-failing`:** optional acceleration only. Do **not** grant exit 0 after an only-failing pass — run a full measure (all sections) first.
 
 ## Loop
 
@@ -21,21 +25,26 @@ npx @tractiontactics/tt-fidelity \
   --viewports 1280,768,390 \
   --out ./fidelity \
   --mode full \
+  --scope content \
   --round N \
   --max-rounds 6
 ```
 
 - Exit **0** → page (or chrome scope) done for this URL pair.
-- Exit **1** → do TASK 1 from the plan; re-measure.
+- Exit **1** → do TASK 1 from the plan; re-measure. Section **count** mismatches and **unpaired** sections also keep exit 1.
 - Exit **2** → fix install / auth / `--roles`; do not guess missing sections.
 - Exit **3** → stop; report residuals; consider history restore.
 
+## Structural noise
+
+If a STRUCTURAL task claims the draft has “extra sections” and those are clearly header/footer chrome, re-run with `--scope content` (default since 0.2.2). Do not `PUT /pages/.../layout` to “add” chrome rows.
+
 ## Large sites
 
-Compare one prototype page to its draft at a time, or pass `--pages`. Use `--scope chrome` then `--scope content`. Prefer `--only-failing` after the first full pass.
+Compare one prototype page to its draft at a time, or pass `--pages`. Use `--scope chrome` then `--scope content`. Prefer `--only-failing` only mid-loop; finish with a full pass.
 
 ## Forbidden
 
 - Certifying “100% parity” without exit 0 from the pixel gate.
 - Substituting fetch-only CSS analysis for the main path.
-- Ignoring section count mismatches (structural first).
+- Ignoring unpaired / section count mismatches (structural first — after confirming scope).

package/README.md

@@ -12,9 +12,12 @@ npx @tractiontactics/tt-fidelity \
   --draft "https://site.example.com/?p=12&preview=true" \
   --viewports 1280,768,390 \
   --out ./fidelity \
-  --mode full
+  --mode full \
+  --scope content
 ```
 
+Default `--scope` is **`content`** (page body; excludes TT header/footer rows). Use `--scope chrome` for header/footer, `--scope full` only when you need both.
+
 Monorepo:
 
 ```bash
@@ -97,10 +100,10 @@ Override fail cut with `--threshold` (default `0.10`).
 | `--out <dir>` | `fidelity.json`, `QUEUE.md`, `shots/sections/…` |
 | `--mode full\|pixel\|styles` | See above |
 | `--viewports` | Default `1280,768,390` |
-| `--scope full\|chrome\|content` | Section / landmark scope |
+| `--scope` | Default **`content`**. `chrome` = header/footer. `full` = both (may skew TT section counts). |
 | `--cache-bust` / `--no-cache-bust` | Default **on** (`tt_nocache=`); logs cache headers |
 | `--concurrency <n>` | Parallel pages |
-| `--only-failing` | Skip sections clean in prior `fidelity.json` |
+| `--only-failing` | Skip sections clean in prior `fidelity.json` — finish with a full pass before exit 0 |
 | `--roles` | Landmark map when defaults miss |
 | `--round` / `--max-rounds` | Agent loop |
 | `--json` | Extra copy of the document |
@@ -126,12 +129,15 @@ Schema: [`fidelity.schema.json`](./fidelity.schema.json) (`schema_version: 2`).
 ## How measurement works
 
 1. Load both URLs at the same viewport (cache-busted by default).
-2. Discover **sections**: draft prefers `.tt-pb-row` / `data-tt-row`; prototype prefers `main > section`.
-3. Align by document order; count mismatch → structural task.
-4. Screenshot each pair, normalize canvas, `pixelmatch` → score + band.
-5. Optionally capture landmark computed styles (explainer).
-6. Cluster failing headings across `--pages` → `QUEUE.md` leverage list.
-7. Emit TT ids (`data-tt-row` / `data-tt-block`) when present for `PUT /pages/{id}/layout`.
+2. **Prepare for capture (0.2.3+):** full-page scroll (scroll-reveal / IntersectionObserver), eager lazy images, unlock common `opacity:0` reveal classes, hide sticky tip/chat overlays.
+3. Discover **sections**: draft prefers `.tt-pb-row` / `data-tt-row`; prototype prefers `main > section` (or `body > section`).
+4. Align by document order; count mismatch → structural task.
+5. Screenshot each pair (scroll section into view again), normalize canvas, `pixelmatch` → score + band.
+6. Optionally capture landmark computed styles (explainer).
+7. Cluster failing headings across `--pages` → `QUEUE.md` leverage list.
+8. Emit TT ids (`data-tt-row` / `data-tt-block`) when present for `PUT /pages/{id}/layout`.
+
+If prototype shots look like blank cream with only a sticky widget, you are on a pre-0.2.3 build — upgrade. Broken/404 media on the prototype itself will still appear empty.
 
 ## Agent contract
 

package/fidelity.schema.json

@@ -66,7 +66,14 @@
 									}
 								},
 								"cacheHeaders": { "type": "object" },
-								"coverage": { "type": "object" }
+								"coverage": { "type": "object" },
+								"unpaired": {
+									"type": "object",
+									"properties": {
+										"prototype": { "type": "array" },
+										"draft": { "type": "array" }
+									}
+								}
 							}
 						}
 					}

package/fixtures/self-test.mjs

@@ -8,6 +8,8 @@ import { fileURLToPath } from 'node:url';
 import { writeFileSync, mkdtempSync, rmSync } from 'node:fs';
 import { tmpdir } from 'node:os';
 import { runFidelity } from '../src/index.mjs';
+import { alignSections } from '../src/sections.mjs';
+import { PACKAGE_VERSION } from '../src/constants.mjs';
 
 const __dirname = dirname(fileURLToPath(import.meta.url));
 const FIX = __dirname;
@@ -24,6 +26,26 @@ function assert(cond, msg) {
 }
 
 async function main() {
+	assert(PACKAGE_VERSION === '0.2.3', 'package version constant is 0.2.3');
+
+	// Align: unpaired + structural when counts differ
+	{
+		const proto = [
+			{ index: 0, label: 'A', selector: '#a', ttRowId: null },
+			{ index: 1, label: 'B', selector: '#b', ttRowId: null },
+			{ index: 2, label: 'C-unpaired', selector: '#c', ttRowId: null }
+		];
+		const draft = [
+			{ index: 0, label: 'A', selector: '#a', ttRowId: 'row_a' },
+			{ index: 1, label: 'B', selector: '#b', ttRowId: 'row_b' }
+		];
+		const { pairs, structural, unpaired } = alignSections(proto, draft);
+		assert(pairs.length === 2, 'align: pairs = min length');
+		assert(unpaired.prototype.length === 1 && unpaired.prototype[0].label === 'C-unpaired', 'align: unpaired proto listed');
+		assert(unpaired.draft.length === 0, 'align: no unpaired draft');
+		assert(structural.some((s) => s.kind === 'count'), 'align: count structural finding');
+	}
+
 	// 1. Known style drift — exit 1 in styles mode
 	{
 		const r = await runFidelity({

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "@tractiontactics/tt-fidelity",
-	"version": "0.2.1",
+	"version": "0.2.3",
 	"description": "Section-first visual fidelity: hybrid SSIM+pixel gate + computed-style explainer for TT Platform prototype→site builds.",
 	"type": "module",
 	"bin": {
@@ -51,5 +51,8 @@
 	"dependencies": {
 		"pixelmatch": "^7.2.0",
 		"pngjs": "^7.0.0"
+	},
+	"devDependencies": {
+		"playwright": "^1.63.0"
 	}
 }

package/src/capture.mjs

@@ -72,9 +72,9 @@ export async function openPage(browser, {
 			}
 		} catch (e) { /* ignore */ }
 	});
-	await page.goto(target, { waitUntil: 'networkidle', timeout: 45000 });
+	await page.goto(target, { waitUntil: 'networkidle', timeout: 60000 });
 	await page.evaluate(() => document.fonts && document.fonts.ready).catch(() => {});
-	await page.waitForTimeout(250);
+	await preparePageForCapture(page);
 
 	if (menus) {
 		for (const s of ['nav li:has(ul)', '.menu-item-has-children', 'nav [aria-haspopup]']) {
@@ -92,23 +92,133 @@ export async function openPage(browser, {
 	return { page, context: ctx, cacheHeaders, finalUrl: page.url() };
 }
 
+/**
+ * Prototypes often use scroll-reveal (opacity:0 until .in) and loading=lazy images.
+ * Without a full scroll + image wait, section screenshots are blank cream and the
+ * pixel gate is meaningless.
+ */
+export async function preparePageForCapture(page) {
+	await page.evaluate(async () => {
+		const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
+
+		// Force-eager lazy images / common data-src patterns.
+		for (const img of document.querySelectorAll('img')) {
+			try {
+				img.loading = 'eager';
+				const ds = img.getAttribute('data-src') || img.getAttribute('data-lazy-src');
+				if (ds && !img.getAttribute('src')) img.setAttribute('src', ds);
+				const dss = img.getAttribute('data-srcset') || img.getAttribute('data-lazy-srcset');
+				if (dss && !img.getAttribute('srcset')) img.setAttribute('srcset', dss);
+			} catch (e) { /* ignore */ }
+		}
+
+		// Scroll the full document so IntersectionObservers / reveal classes fire.
+		const step = Math.max(240, Math.floor(window.innerHeight * 0.7));
+		const maxY = Math.max(document.body.scrollHeight, document.documentElement.scrollHeight);
+		for (let y = 0; y < maxY; y += step) {
+			window.scrollTo(0, y);
+			await sleep(80);
+		}
+		window.scrollTo(0, maxY);
+		await sleep(150);
+		window.scrollTo(0, 0);
+		await sleep(200);
+
+		// If reveal never fired (no IO), force common "in view" classes / opacity.
+		for (const el of document.querySelectorAll('section, .sec, .sec-alt, .trust, .lead, .phero, [class*=reveal i], [class*=fade i]')) {
+			try {
+				const cs = getComputedStyle(el);
+				if (parseFloat(cs.opacity) < 0.5) {
+					el.classList.add('in', 'is-inview', 'is-visible', 'visible', 'aos-animate');
+					el.style.opacity = '1';
+					el.style.transform = 'none';
+					el.style.visibility = 'visible';
+				}
+			} catch (e) { /* ignore */ }
+		}
+
+		// Hide sticky tips/chat that paint over section shots (not page content).
+		for (const el of document.querySelectorAll(
+			'[class*=did-you i], [class*=cookie i], [id*=cookie i], .bc-widget, [class*=chat-widget i], [class*=intercom i]'
+		)) {
+			try {
+				const cs = getComputedStyle(el);
+				if (cs.position === 'fixed' || cs.position === 'sticky') {
+					el.style.setProperty('display', 'none', 'important');
+				}
+			} catch (e) { /* ignore */ }
+		}
+	});
+
+	// Wait for images that already have a layout box.
+	await page.evaluate(async () => {
+		const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
+		const deadline = Date.now() + 8000;
+		while (Date.now() < deadline) {
+			const pending = [...document.images].filter((img) => {
+				if (!img.src || img.src.startsWith('data:')) return false;
+				const r = img.getBoundingClientRect();
+				if (r.width < 2 || r.height < 2) return false;
+				return !(img.complete && img.naturalWidth > 0);
+			});
+			if (!pending.length) break;
+			await sleep(150);
+		}
+		if (document.fonts?.ready) await document.fonts.ready;
+	}).catch(() => {});
+
+	await page.waitForTimeout(200);
+}
+
 /**
  * Screenshot a section by clipping from the page (element bounding box).
  */
 export async function screenshotSection(page, section) {
 	const handle = await page.$(section.selector);
-	if (!handle) {
-		// fallback clip from stored box
-		const { x, viewportTop, width, height } = section.box;
-		return page.screenshot({
-			type: 'png',
-			clip: {
-				x: Math.max(0, x),
-				y: Math.max(0, viewportTop),
-				width: Math.max(1, width),
-				height: Math.max(1, height)
-			}
-		});
+	if (handle) {
+		try {
+			await handle.evaluate((el) => {
+				el.scrollIntoView({ block: 'center', inline: 'nearest' });
+				const cs = getComputedStyle(el);
+				if (parseFloat(cs.opacity) < 0.5) {
+					el.classList.add('in', 'is-inview', 'is-visible', 'visible', 'aos-animate');
+					el.style.opacity = '1';
+					el.style.transform = 'none';
+					el.style.visibility = 'visible';
+				}
+				for (const img of el.querySelectorAll('img')) {
+					img.loading = 'eager';
+					const ds = img.getAttribute('data-src') || img.getAttribute('data-lazy-src');
+					if (ds && !img.getAttribute('src')) img.setAttribute('src', ds);
+				}
+			});
+			await page.waitForTimeout(120);
+			// Wait briefly for in-section images.
+			await handle.evaluate(async (el) => {
+				const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
+				const deadline = Date.now() + 2500;
+				while (Date.now() < deadline) {
+					const pending = [...el.querySelectorAll('img')].filter(
+						(img) => img.src && !(img.complete && img.naturalWidth > 0)
+					);
+					if (!pending.length) break;
+					await sleep(100);
+				}
+			}).catch(() => {});
+			return await handle.screenshot({ type: 'png', timeout: 20000 });
+		} catch (e) {
+			/* fall through to clip */
+		}
 	}
-	return handle.screenshot({ type: 'png' });
+	const { x, viewportTop, width, height } = section.box || {};
+	return page.screenshot({
+		type: 'png',
+		clip: {
+			x: Math.max(0, x || 0),
+			y: Math.max(0, viewportTop || 0),
+			width: Math.max(1, width || 1),
+			height: Math.max(1, Math.min(height || 1, 8000))
+		},
+		timeout: 15000
+	});
 }

package/src/cli.mjs

@@ -13,7 +13,7 @@ function parseArgs(argv) {
 		only: null,
 		menus: false,
 		quiet: false,
-		scope: 'full',
+		scope: 'content',
 		round: 1,
 		maxRounds: 6,
 		mode: 'full',
@@ -77,11 +77,12 @@ function usage() {
   --auth u:p / --draft-auth  basic auth
   --cookie "n=v"             draft cookies
   --roles roles.json         project role map
-  --scope full|chrome|content
+  --scope content|chrome|full  default content (page body; excludes TT header/footer rows)
+                               use chrome for header/footer; full only when you need both
   --round / --max-rounds     fidelity loop (exit 3 at max with diffs)
   --cache-bust / --no-cache-bust  default ON (tt_nocache=)
   --concurrency <n>          parallel pages (default 1)
-  --only-failing             skip sections clean in prior fidelity.json
+  --only-failing             skip sections clean in prior fidelity.json (do not use until a final full pass)
   --prior <path>             prior fidelity.json for --only-failing
   --json out.json            also write document
   --plan-only                suppress raw evidence tables

package/src/constants.mjs

@@ -70,4 +70,4 @@ export const BANDS = {
 };
 
 export const SCHEMA_VERSION = 2;
-export const PACKAGE_VERSION = '0.2.1';
+export const PACKAGE_VERSION = '0.2.3';

package/src/plan.mjs

@@ -121,6 +121,7 @@ export function buildPlan(d, { pixelSections = [] } = {}) {
 	}
 	for (const [role, items] of structuralByRole) {
 		const unmatched = items.some((i) => i.kind === 'unmatched' || i.kind === 'extra');
+		const hints = items.map((i) => i.hint).filter(Boolean);
 		tasks.push({
 			kind: 'structural',
 			title: unmatched
@@ -129,13 +130,34 @@ export function buildPlan(d, { pixelSections = [] } = {}) {
 			why: items.map((i) => i.text),
 			fixAt: unmatched
 				? `FIRST verify it is really absent (inspect the page). If it exists under different markup, add its selector via --roles and re-run — do not "fix" it. If genuinely absent: ${fixHintFor(null, role, 'structural')}`
-				: fixHintFor(null, role, 'structural'),
+				: (hints[0] || fixHintFor(null, role, 'structural')),
 			rows: [],
 			affects: items.length,
 			mayResolve: 0
 		});
 	}
 
+	const unpaired = d.unpaired;
+	if (unpaired && ((unpaired.prototype && unpaired.prototype.length) || (unpaired.draft && unpaired.draft.length))) {
+		const why = [];
+		for (const u of unpaired.prototype || []) {
+			why.push(`unpaired prototype [${u.index}]: "${u.label || u.selector}"${u.ttRowId ? ` (${u.ttRowId})` : ''}`);
+		}
+		for (const u of unpaired.draft || []) {
+			why.push(`unpaired draft [${u.index}]: "${u.label || u.selector}"${u.ttRowId ? ` (${u.ttRowId})` : ''}`);
+		}
+		tasks.push({
+			kind: 'structural',
+			title: 'Unpaired sections — not pixel-compared (positional truncate)',
+			why,
+			fixAt:
+				'These sections were dropped from the pixel gate because counts differ. Prefer --scope content for page body (default since 0.2.2). Fix real missing/extra content, then re-measure so every section is paired.',
+			rows: [],
+			affects: why.length,
+			mayResolve: 0
+		});
+	}
+
 	// Pixel sections — severity-ranked, before global style (rebuild first).
 	const failing = (pixelSections || [])
 		.filter((s) => s.band && s.band !== 'clean')
@@ -381,6 +403,10 @@ export function renderQueueMd(clusters, pages) {
 	lines.push('');
 	lines.push('Ranked by leverage. Do one task, re-measure, continue.');
 	lines.push('');
+	lines.push('> Default scope is **content** (page body). Use `--scope chrome` for header/footer.');
+	lines.push('> This QUEUE is valid only until the draft page layout is rewritten (row ids may change).');
+	lines.push('> Do not use `--only-failing` until a final full pass before claiming exit 0.');
+	lines.push('');
 	if (clusters?.length) {
 		lines.push('## Cross-page clusters (fix these first)');
 		lines.push('');
@@ -397,13 +423,23 @@ export function renderQueueMd(clusters, pages) {
 	for (const page of pages) {
 		let fail = 0;
 		let total = 0;
+		const unpairedNotes = [];
 		for (const vp of page.viewports) {
 			for (const s of vp.sections || []) {
 				total += 1;
 				if (s.band !== 'clean') fail += 1;
 			}
+			const u = vp.unpaired || vp.styleDiff?.unpaired;
+			if (u && (u.prototype?.length || u.draft?.length)) {
+				unpairedNotes.push(
+					`${vp.width}px: ${u.prototype?.length || 0} proto / ${u.draft?.length || 0} draft unpaired`
+				);
+			}
+		}
+		lines.push(`- \`${page.id}\`: ${fail}/${total} sections failing (compared)`);
+		for (const note of unpairedNotes) {
+			lines.push(`  - unpaired (not pixel-compared): ${note}`);
 		}
-		lines.push(`- \`${page.id}\`: ${fail}/${total} sections failing`);
 	}
 	lines.push('');
 	return lines.join('\n');

package/src/run.mjs

@@ -150,9 +150,11 @@ async function measurePagePair(browser, pageSpec, opts, roles) {
 		if (doPixel) {
 			const protoSecs = await protoOpen.page.evaluate(inPageSections, { scope: opts.scope });
 			const draftSecs = await draftOpen.page.evaluate(inPageSections, { scope: opts.scope });
-			const { pairs, structural } = alignSections(protoSecs, draftSecs);
+			const { pairs, structural, unpaired } = alignSections(protoSecs, draftSecs);
 			styleDiff.structural = [...(styleDiff.structural || []), ...structural];
-
+			if (unpaired?.prototype?.length || unpaired?.draft?.length) {
+				styleDiff.unpaired = unpaired;
+			}
 			const shotsDir = opts.outDir
 				? join(opts.outDir, 'shots', 'sections', pageSpec.id, String(width))
 				: null;
@@ -225,6 +227,7 @@ async function measurePagePair(browser, pageSpec, opts, roles) {
 			coverage,
 			styleDiff,
 			sections,
+			unpaired: styleDiff.unpaired || { prototype: [], draft: [] },
 			cacheHeaders: draftOpen.cacheHeaders,
 			plan
 		});
@@ -264,7 +267,7 @@ export async function runFidelity(opts = {}) {
 		viewports: [1280, 768, 390],
 		tolerance: 1,
 		mode: 'full',
-		scope: 'full',
+		scope: 'content',
 		round: 1,
 		maxRounds: 6,
 		cacheBust: true,

package/src/sections.mjs

@@ -95,7 +95,7 @@ export function inPageSections({ scope }) {
 /* eslint-enable no-undef */
 
 /**
- * Align sections by document order. Length mismatch → structural findings.
+ * Align sections by document order. Length mismatch → structural findings + unpaired lists.
  */
 export function alignSections(protoSections, draftSections) {
 	const n = Math.min(protoSections.length, draftSections.length);
@@ -103,17 +103,33 @@ export function alignSections(protoSections, draftSections) {
 	for (let i = 0; i < n; i += 1) {
 		pairs.push({ index: i, proto: protoSections[i], draft: draftSections[i] });
 	}
+	const unpaired = {
+		prototype: protoSections.slice(n).map(summarizeUnpaired),
+		draft: draftSections.slice(n).map(summarizeUnpaired)
+	};
 	const structural = [];
 	if (protoSections.length !== draftSections.length) {
+		const missing = protoSections.length - draftSections.length;
 		structural.push({
 			kind: 'count',
 			role: 'section',
 			text: `section: prototype has ${protoSections.length}, draft has ${draftSections.length}${
-				draftSections.length < protoSections.length
-					? ` (draft is missing ${protoSections.length - draftSections.length})`
-					: ` (draft has ${draftSections.length - protoSections.length} extra)`
-			}`
+				missing > 0
+					? ` (draft is missing ${missing})`
+					: ` (draft has ${-missing} extra)`
+			}`,
+			hint:
+				'If comparing page body only, re-run with --scope content (default since 0.2.2) so TT header/footer rows are not counted as extra sections. Chromeless count gaps may still be real layout issues.'
 		});
 	}
-	return { pairs, structural };
+	return { pairs, structural, unpaired };
+}
+
+function summarizeUnpaired(sec) {
+	return {
+		index: sec.index,
+		label: sec.label || '',
+		selector: sec.selector || '',
+		ttRowId: sec.ttRowId || null
+	};
 }