@rettangoli/vt 0.0.13 → 1.0.0-rc1

package/README.md

@@ -1,98 +1,141 @@
-
 # Rettangoli Visual Testing
 
-A visual testing framework for UI components using Playwright and screenshot comparison. Perfect for regression testing and ensuring UI consistency across changes.
+Visual regression testing for Rettangoli specs using Playwright screenshots.
 
-**In production**, this package is typically used through the `rtgl` CLI tool. For development and testing of this package itself, you should call the local CLI directly.
+## Commands
 
-## Features
+- `rtgl vt generate`
+- `rtgl vt report`
+- `rtgl vt accept`
 
-- **Screenshot Generation** - Automatically generate screenshots from HTML specifications
-- **Visual Comparison** - Compare screenshots to detect visual changes
-- **Test Reports** - Generate detailed reports with diff highlights
-- **Playwright Integration** - Uses Playwright for reliable cross-browser testing
-- **Template System** - Liquid templates for flexible HTML generation
-- **Configuration** - YAML-based configuration for easy customization
+## Public Generate Options
 
-## Development
+- `--skip-screenshots`
+- `--headed`
+- `--concurrency <number>`
+- `--timeout <ms>`
+- `--wait-event <name>`
+- `--folder <path>` (repeatable)
+- `--group <section-key>` (repeatable)
+- `--item <spec-path>` (repeatable)
 
-### Prerequisites
+## Public Report Options
 
-- Node.js 18+ or Bun
-- Playwright browsers (automatically installed)
+- `--compare-method <method>`
+- `--color-threshold <number>`
+- `--diff-threshold <number>`
+- `--folder <path>` (repeatable)
+- `--group <section-key>` (repeatable)
+- `--item <spec-path>` (repeatable)
 
-### Setup
+## Scoped Runs
 
-1. **Install dependencies**:
-```bash
-bun install
-```
+Use selectors to run only part of VT in both `generate` and `report`:
 
-2. **Install Playwright browsers** (if not already installed):
-```bash
-npx playwright install
-```
+- `folder`: matches specs by folder prefix under `vt/specs` (example: `components/forms`)
+- `group`: matches section page key from `vt.sections` (`title` for flat sections, `items[].title` for grouped sections)
+- `item`: matches a single spec path relative to `vt/specs` (with or without extension)
 
-### Project Structure
+Selector rules:
 
-```
-src/
-├── cli/
-│   ├── generate.js    # Generate screenshots from specifications
-│   ├── report.js      # Generate visual comparison reports
-│   ├── accept.js      # Accept screenshot changes as new reference
-│   ├── templates/     # HTML templates for reports
-│   └── static/        # Static assets (CSS, etc.)
-├── common.js          # Shared utilities and functions
-└── index.js           # Main export (empty - CLI focused)
-```
+- selectors are unioned (OR); any matched item is included
+- if no selector is provided, all items are included
 
-### Core Functionality
+Examples:
 
-The visual testing framework provides three main commands:
+```bash
+# Only specs under a folder
+rtgl vt generate --folder components/forms
+
+# Only one section/group key from vt.sections
+rtgl vt generate --group components_basic
 
-#### 1. Generate (`vt generate`)
-- Reads HTML specifications from `vt/specs/` directory
-- Generates screenshots using Playwright
-- Saves candidate screenshots for comparison
-- Creates a static site for viewing results
+# Only one spec item (extension optional)
+rtgl vt generate --item components/forms/login
+rtgl vt generate --item components/forms/login.html
 
-#### 2. Report (`vt report`)
-- Compares candidate screenshots with reference screenshots
-- Generates visual diff reports highlighting changes
-- Creates an HTML report with before/after comparisons
-- Uses `looks-same` library for pixel-perfect comparison
+# Combine selectors (union)
+rtgl vt generate --group components_basic --item pages/home
+
+# Same selectors for report
+rtgl vt report --folder components/forms
+rtgl vt report --group components_basic
+rtgl vt report --item components/forms/login
+```
 
-#### 3. Accept (`vt accept`)
-- Accepts candidate screenshots as new reference images
-- Updates the golden/reference screenshot directory
-- Used when visual changes are intentional
+Everything else in capture is internal and intentionally not user-configurable.
 
-### Configuration
+## Config
 
-The framework reads configuration from `rettangoli.config.yaml`:
+`rettangoli.config.yaml`:
 
 ```yaml
 vt:
+  path: ./vt
   port: 3001
-  screenshotWaitTime: 500
   skipScreenshots: false
+  concurrency: 4
+  timeout: 30000
+  waitEvent: vt:ready
+  sections:
+    - title: components_basic
+      files: components
 ```
 
-### Testing Your Changes
+Notes:
+
+- `vt.sections` is required.
+- Section page keys (`title` for flat sections and group `items[].title`) allow only letters, numbers, `-`, `_`.
+- `vt.capture` is internal and must be omitted.
+
+## Spec Frontmatter
+
+Supported frontmatter keys per spec file:
+
+- `title`
+- `description`
+- `template`
+- `url`
+- `waitEvent`
+- `waitSelector`
+- `waitStrategy` (`networkidle` | `load` | `event` | `selector`)
+- `skipScreenshot`
+- `specs`
+- `steps`
 
-**Note**: This package doesn't include example files in its directory. For testing during development, use examples from other packages (like `rettangoli-ui`) and call the CLI directly:
+Screenshot naming:
+
+- First screenshot is `-01`.
+- Then `-02`, `-03`, up to `-99`.
+
+## Development
+
+Run tests:
+
+```bash
+bun test
+```
+
+Run real-browser smoke:
 
 ```bash
-# Call the local CLI directly for development
-node ../rettangoli-cli/cli.js vt generate
-node ../rettangoli-cli/cli.js vt report
-node ../rettangoli-cli/cli.js vt accept
+VT_E2E=1 bun test spec/e2e-smoke.spec.js
 ```
 
-**Production usage** (when rtgl is installed globally):
+Run Docker E2E tests (requires Docker daemon running):
+
+```bash
+# Full pipeline: build image → verify → run all Docker E2E tests
+bun run test:docker:full
+
+# Tests only (skip image build, assumes image already exists)
+bun run test:docker
+```
+
+The Docker E2E suite builds a local `rtgl-local-test:latest` image, then runs generate/report/accept in temp directories inside containers. Tests validate WebP screenshot headers, report.json schema, metrics.json schema, HTML content, directory structure, accept file copies, multi-spec fixtures, multi-screenshot ordinals, and pixelmatch diff detection.
+
+Optional benchmark fixture:
+
 ```bash
-rtgl vt generate
-rtgl vt report
-rtgl vt accept
+bun run bench:capture
 ```

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@rettangoli/vt",
-  "version": "0.0.13",
+  "version": "1.0.0-rc1",
   "description": "Rettangoli Visual Testing",
   "type": "module",
   "main": "./src/index.js",
@@ -9,7 +9,10 @@
     "./cli": "./src/cli/index.js"
   },
   "scripts": {
-    "test": "echo \"Error: no test specified\" && exit 1"
+    "test": "vitest run --reporter=verbose",
+    "test:e2e": "node e2e/run.js",
+    "test:e2e:full": "bash scripts/docker-e2e-test.sh",
+    "bench:capture": "node ./scripts/benchmark-capture.js"
   },
   "dependencies": {
     "commander": "^13.1.0",
@@ -19,5 +22,9 @@
     "playwright": "1.57.0",
     "sharp": "^0.34.5",
     "shiki": "^3.3.0"
+  },
+  "devDependencies": {
+    "puty": "^0.1.1",
+    "vitest": "^4.0.15"
   }
 }

package/src/capture/capture-scheduler.js

@@ -0,0 +1,206 @@
+import { chromium } from "playwright";
+import { PlaywrightRunner } from "./playwright-runner.js";
+import { ResultCollector } from "./result-collector.js";
+import { resolveWorkerPlan } from "./worker-plan.js";
+
+function nowMs() {
+  return performance.now();
+}
+
+function createTaskQueues(tasks) {
+  const freshQueue = tasks
+    .map((task, sourceOrder) => ({
+      task,
+      sourceOrder,
+      attempt: 1,
+      queueType: "fresh",
+      enqueuedAtMs: nowMs(),
+    }))
+    .sort((left, right) => {
+      const leftCost = left.task.estimatedCost ?? 0;
+      const rightCost = right.task.estimatedCost ?? 0;
+      if (leftCost !== rightCost) {
+        return leftCost - rightCost;
+      }
+      return right.sourceOrder - left.sourceOrder;
+    });
+
+  const retryQueue = [];
+  let dispatchCount = 0;
+  const fairRetryInterval = 4; // 3 fresh dispatches then 1 retry when available
+
+  const getNextTask = () => {
+    const shouldDispatchRetry = retryQueue.length > 0
+      && (freshQueue.length === 0 || (dispatchCount % fairRetryInterval) === (fairRetryInterval - 1));
+
+    if (shouldDispatchRetry) {
+      dispatchCount += 1;
+      return retryQueue.shift();
+    }
+    if (freshQueue.length > 0) {
+      dispatchCount += 1;
+      return freshQueue.pop();
+    }
+    if (retryQueue.length > 0) {
+      dispatchCount += 1;
+      return retryQueue.shift();
+    }
+    return null;
+  };
+
+  const enqueueRetry = (item) => {
+    retryQueue.push({
+      task: item.task,
+      attempt: item.attempt + 1,
+      queueType: "retry",
+      enqueuedAtMs: nowMs(),
+    });
+  };
+
+  return {
+    getNextTask,
+    enqueueRetry,
+  };
+}
+
+export async function runCaptureScheduler(options) {
+  const {
+    tasks,
+    screenshotsDir,
+    workerCount: requestedWorkers,
+    isolationMode,
+    screenshotWaitTime,
+    waitEvent,
+    waitSelector,
+    waitStrategy,
+    navigationTimeout,
+    readyTimeout,
+    screenshotTimeout,
+    maxRetries,
+    recycleEvery,
+    metricsPath,
+    headless,
+  } = options;
+
+  const { workerCount, adaptivePolicy } = resolveWorkerPlan(requestedWorkers);
+  console.log(
+    `Capture scheduler: mode=${adaptivePolicy.mode}, workers=${workerCount}, isolation=${isolationMode}`,
+  );
+
+  const collector = new ResultCollector({
+    totalTasks: tasks.length,
+    metricsPath,
+    workerCount,
+    isolationMode,
+    maxRetries,
+    adaptivePolicy,
+    schedulingPolicy: {
+      type: "duration-aware-fair-retry",
+      freshBeforeRetry: 3,
+    },
+  });
+
+  if (!tasks.length) {
+    const { summary } = collector.finalize();
+    return {
+      summary,
+      failures: [],
+    };
+  }
+
+  const queue = createTaskQueues(tasks);
+  const getNextTask = queue.getNextTask;
+
+  const browser = await chromium.launch({ headless });
+  const runners = [];
+
+  try {
+    for (let index = 0; index < workerCount; index += 1) {
+      const runner = new PlaywrightRunner({
+        workerId: index + 1,
+        browser,
+        screenshotsDir,
+        isolationMode,
+        screenshotWaitTime,
+        waitEvent,
+        waitSelector,
+        waitStrategy,
+        navigationTimeout,
+        readyTimeout,
+        screenshotTimeout,
+      });
+      await runner.initialize();
+      runners.push(runner);
+    }
+
+    const workerLoops = runners.map(async (runner) => {
+      let processedSinceRecycle = 0;
+
+      while (true) {
+        const item = getNextTask();
+        if (!item) {
+          break;
+        }
+
+        const queueWaitMs = Math.max(0, nowMs() - item.enqueuedAtMs);
+        const attemptStartMs = nowMs();
+
+        try {
+          const result = await runner.runTask(item.task, item.attempt);
+          const attemptMs = nowMs() - attemptStartMs;
+          collector.recordSuccess(item.task, result, {
+            workerId: runner.workerId,
+            queueType: item.queueType,
+            queueWaitMs,
+            attemptMs,
+          });
+          processedSinceRecycle += 1;
+
+          if (
+            isolationMode === "fast"
+            && recycleEvery > 0
+            && processedSinceRecycle >= recycleEvery
+          ) {
+            await runner.recycleSharedContext();
+            collector.recordRecycle(runner.workerId, `recycleEvery=${recycleEvery}`);
+            processedSinceRecycle = 0;
+          }
+        } catch (error) {
+          const attemptMs = nowMs() - attemptStartMs;
+          if (item.attempt <= maxRetries) {
+            collector.recordRetry(item.task, item.attempt, error.message, {
+              workerId: runner.workerId,
+              queueType: item.queueType,
+              queueWaitMs,
+              attemptMs,
+            });
+            queue.enqueueRetry(item);
+            continue;
+          }
+
+          collector.recordFailure(item.task, item.attempt, error.message, {
+            workerId: runner.workerId,
+            queueType: item.queueType,
+            queueWaitMs,
+            attemptMs,
+          });
+        }
+      }
+    });
+
+    await Promise.all(workerLoops);
+  } finally {
+    await Promise.all(
+      runners.map(async (runner) => {
+        await runner.dispose();
+      }),
+    );
+    await browser.close();
+  }
+
+  const { summary, failures } = collector.finalize();
+  return {
+    summary,
+    failures,
+  };
+}