playwright-archaeologist 0.1.1

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 playwright-archaeologist contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,392 @@
+# playwright-archaeologist
+
+**Generate a complete behavioral specification of any running web app — no source code required.**
+
+[![npm version](https://img.shields.io/npm/v/playwright-archaeologist)](https://www.npmjs.com/package/playwright-archaeologist)
+[![license](https://img.shields.io/npm/l/playwright-archaeologist)](./LICENSE)
+[![node](https://img.shields.io/node/v/playwright-archaeologist)](https://nodejs.org/)
+
+Point `playwright-archaeologist` at a URL and get back a full behavioral spec: sitemap, form catalog, API map with OpenAPI 3.0 schema, screenshots, navigation flow graph, and a regression baseline you can diff later.
+
+---
+
+## Quick Start
+
+```bash
+# Install globally
+npm install -g playwright-archaeologist
+
+# Download Chromium (one-time)
+pa install
+
+# Crawl a site
+pa dig https://example.com
+
+# View the report
+open .archaeologist/report.html
+```
+
+Or use `npx` without installing:
+
+```bash
+npx playwright-archaeologist install
+npx playwright-archaeologist dig https://example.com
+```
+
+---
+
+## Features
+
+- **Zero source code access** — works on any running web app, staging or production
+- **SPA-aware crawling** — Navigation API + History API patching + MutationObserver for client-side route detection
+- **Authenticated crawling** — run auth scripts or inject cookies before crawling protected sites
+- **Screenshot atlas** — full-page and viewport screenshots with a browsable gallery
+- **API discovery** — auto-generates OpenAPI 3.0 specs from observed network traffic
+- **Form catalog** — extracts every form with field metadata, validation rules, and structure
+- **Flow graph** — Mermaid navigation diagrams showing how pages connect
+- **Regression diff** — compare two crawl snapshots, detect structural and visual changes
+- **Security-first** — SSRF protection, credential scrubbing, browser CSP hardening
+- **Resume support** — checkpoint and resume interrupted crawls
+
+---
+
+## Installation
+
+### Requirements
+
+- Node.js >= 20.0.0
+- Chromium is downloaded automatically via `pa install`
+
+### npm
+
+```bash
+npm install -g playwright-archaeologist
+pa install
+```
+
+### As a dev dependency
+
+```bash
+npm install --save-dev playwright-archaeologist
+npx pa install
+```
+
+---
+
+## Usage
+
+### Crawl a website
+
+```bash
+# Basic crawl
+pa dig https://myapp.com
+
+# Limit depth and pages
+pa dig https://myapp.com --depth 3 --max-pages 100
+
+# Custom viewport
+pa dig https://myapp.com --viewport 1440x900
+
+# Skip screenshots for a faster crawl
+pa dig https://myapp.com --no-screenshots
+
+# Enable deep click exploration for SPAs
+pa dig https://myapp.com --deep-click
+
+# Custom output directory
+pa dig https://myapp.com -o ./crawl-output
+
+# Resume an interrupted crawl
+pa dig https://myapp.com --resume
+```
+
+### Compare two snapshots
+
+```bash
+# Compare crawl bundles (exit code 0 = identical, 1 = changes)
+pa diff .archaeologist/bundle-old.zip .archaeologist/bundle-new.zip
+
+# Generate an HTML diff report
+pa diff old.zip new.zip --format-html diff-report.html
+
+# Generate a JSON diff report
+pa diff old.zip new.zip --format-json diff-report.json
+```
+
+### Authenticated crawling
+
+```bash
+# Using an auth script
+pa dig https://myapp.com --auth ./login.js
+
+# Using cookies
+pa dig https://myapp.com --cookies ./cookies.json
+```
+
+---
+
+## Configuration Reference
+
+### `pa dig` options
+
+| Option | Default | Description |
+|---|---|---|
+| `-d, --depth <n>` | `5` | Maximum crawl depth from the entry URL |
+| `--max-pages <n>` | `1000` | Maximum number of pages to visit |
+| `-c, --concurrency <n>` | `3` | Number of parallel browser contexts |
+| `--auth <script>` | — | Path to an auth script (runs before crawling) |
+| `--cookies <file>` | — | Path to a cookies JSON file |
+| `-o, --output <dir>` | `.archaeologist` | Output directory for all artifacts |
+| `--no-screenshots` | `false` | Skip screenshot capture |
+| `--viewport <WxH>` | `1280x720` | Viewport dimensions |
+| `--viewports <list>` | — | Comma-separated viewport list for multi-viewport screenshots |
+| `--deep-click` | `false` | Click interactive elements to discover SPA routes |
+| `--resume` | `false` | Resume from the last checkpoint |
+| `--include <pattern>` | — | URL patterns to include (repeatable) |
+| `--exclude <pattern>` | — | URL patterns to exclude (repeatable) |
+
+### `pa diff` options
+
+| Option | Description |
+|---|---|
+| `--format-html <path>` | Write an HTML diff report |
+| `--format-json <path>` | Write a JSON diff report |
+
+---
+
+## Output Structure
+
+After a crawl, the `.archaeologist/` directory contains:
+
+```
+.archaeologist/
+  report.html            # Browsable HTML report with all findings
+  sitemap.json           # Discovered pages with metadata
+  forms.json             # Form catalog with field details
+  api.json               # Observed API endpoints
+  openapi.yaml           # Generated OpenAPI 3.0 specification
+  flow-graph.svg         # Navigation flow diagram (Mermaid)
+  screenshots/           # Full-page and viewport screenshots
+    index.png
+    about.png
+    ...
+  bundle.zip             # Snapshot bundle for regression diffing
+  checkpoint.json        # Resume checkpoint (deleted on completion)
+```
+
+---
+
+## Programmatic API
+
+Use `playwright-archaeologist` as a library in your own tools:
+
+```typescript
+import { dig } from 'playwright-archaeologist';
+
+const result = await dig({
+  entryUrl: 'https://myapp.com',
+  depth: 3,
+  maxPages: 50,
+  concurrency: 2,
+  output: './my-output',
+  screenshots: true,
+  viewport: { width: 1280, height: 720 },
+});
+
+console.log(`Crawled ${result.pages.length} pages`);
+console.log(`Found ${result.forms.length} forms`);
+console.log(`Discovered ${result.apis.length} API endpoints`);
+```
+
+### Comparing snapshots programmatically
+
+```typescript
+import { diffBundles, generateDiffReportHtml } from 'playwright-archaeologist';
+
+const diff = await diffBundles('./old-bundle.zip', './new-bundle.zip');
+
+if (diff.hasChanges) {
+  console.log('Changes detected:');
+  console.log(`  Pages added: ${diff.pages.added.length}`);
+  console.log(`  Pages removed: ${diff.pages.removed.length}`);
+  console.log(`  APIs changed: ${diff.apis.modified.length}`);
+
+  // Generate HTML report
+  const html = generateDiffReportHtml(diff);
+  await fs.writeFile('diff-report.html', html);
+}
+```
+
+### Using individual collectors
+
+```typescript
+import { scanPage, probeForms, captureScreenshots } from 'playwright-archaeologist';
+import { chromium } from 'playwright';
+
+const browser = await chromium.launch();
+const context = await browser.newContext();
+const page = await context.newPage();
+
+await page.goto('https://myapp.com/login');
+
+// Scan page structure
+const scan = await scanPage(page);
+
+// Probe forms
+const forms = await probeForms(page);
+
+// Capture screenshots
+const screenshots = await captureScreenshots(page, {
+  viewport: { width: 1280, height: 720 },
+});
+
+await browser.close();
+```
+
+---
+
+## Auth Script Example
+
+Auth scripts run in a real browser context before crawling begins. They receive a Playwright `page` object:
+
+```javascript
+// login.js
+export default async function authenticate(page) {
+  await page.goto('https://myapp.com/login');
+  await page.fill('#email', 'test@example.com');
+  await page.fill('#password', process.env.TEST_PASSWORD);
+  await page.click('button[type="submit"]');
+  await page.waitForURL('**/dashboard');
+}
+```
+
+```bash
+TEST_PASSWORD=secret pa dig https://myapp.com --auth ./login.js
+```
+
+Auth scripts are statically analyzed before execution and require confirmation for scripts that access the filesystem, network, or run shell commands.
+
+---
+
+## Cookies File Format
+
+The cookies file follows the Playwright cookie format:
+
+```json
+[
+  {
+    "name": "session",
+    "value": "abc123",
+    "domain": "myapp.com",
+    "path": "/",
+    "httpOnly": true,
+    "secure": true
+  }
+]
+```
+
+---
+
+## Security Considerations
+
+`playwright-archaeologist` is designed to crawl potentially untrusted web applications. Several protections are built in:
+
+- **SSRF protection** — Private/internal IP ranges (10.x, 172.16-31.x, 169.254.x, 127.x, ::1) are blocked by default. Only same-origin navigation is permitted unless explicitly expanded.
+- **Credential scrubbing** — Authorization headers, cookies, and bearer tokens are redacted from all output artifacts by default.
+- **Browser hardening** — `bypassCSP: true` for instrumentation, `serviceWorkers: 'block'`, `acceptDownloads: false`, and automatic dialog dismissal to prevent crawler hangs.
+- **Auth script sandboxing** — Auth scripts undergo static analysis before execution. Scripts accessing `fs`, `child_process`, or making network requests outside the target domain trigger a confirmation prompt.
+- **Output sanitization** — All target-sourced data is entity-encoded in HTML reports. Reports include a restrictive CSP meta tag.
+
+---
+
+## CI / Regression Testing
+
+Use `playwright-archaeologist` in CI to catch behavioral regressions:
+
+```yaml
+# .github/workflows/behavioral-regression.yml
+name: Behavioral Regression
+on: [pull_request]
+
+jobs:
+  regression:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Start app
+        run: npm start &
+
+      - name: Install pa
+        run: npx playwright-archaeologist install
+
+      - name: Crawl
+        run: npx pa dig http://localhost:3000 -o ./current
+
+      - name: Download baseline
+        uses: actions/download-artifact@v4
+        with:
+          name: behavioral-baseline
+          path: ./baseline
+
+      - name: Diff
+        run: |
+          npx pa diff ./baseline/bundle.zip ./current/bundle.zip \
+            --format-html regression-report.html
+
+      - name: Upload report
+        if: failure()
+        uses: actions/upload-artifact@v4
+        with:
+          name: regression-report
+          path: regression-report.html
+```
+
+---
+
+## Contributing
+
+Contributions are welcome. Please open an issue first to discuss what you would like to change.
+
+```bash
+# Clone and install
+git clone https://github.com/AshGw/playwright-archaeologist.git
+cd playwright-archaeologist
+npm install
+
+# Build
+npm run build
+
+# Run tests
+npm test
+
+# Run tests in watch mode
+npm run test:watch
+
+# Run benchmarks
+npm run bench
+```
+
+### Project structure
+
+```
+src/
+  cli.ts              # CLI entry point (Commander.js)
+  index.ts            # Programmatic API exports
+  crawl/              # BFS crawler, frontier, context pool, checkpoints
+  collectors/         # Page scanner, form prober, network logger, screenshots
+  assembler/          # API grouper, flow graph builder
+  auth/               # Auth script handler
+  report/             # HTML report generator
+  diff/               # Snapshot diff engine and reports
+  bundle/             # ZIP bundle creator
+  security/           # SSRF guard, credential scrubber, output sanitizer
+  types/              # TypeScript interfaces, Zod schemas, error hierarchy
+  utils/              # Logger, URL utilities, progress tracker
+```
+
+---
+
+## License
+
+[MIT](./LICENSE)

package/bin/cli.js

@@ -0,0 +1,2 @@
+#!/usr/bin/env node
+import '../dist/cli.js';

package/dist/chunk-7ZQGW5OV.js

@@ -0,0 +1,255 @@
+import {
+  isSameOrigin,
+  resolveUrl
+} from "./chunk-RWPEKZOW.js";
+
+// src/collectors/page-scanner.ts
+async function scanPage(page, baseUrl, response) {
+  const url = page.url();
+  const statusCode = response?.status() ?? 200;
+  const [domData, timingData] = await Promise.all([
+    extractDomData(page),
+    extractTiming(page)
+  ]);
+  const links = processLinks(domData.links, url, baseUrl);
+  const interactiveElements = processInteractiveElements(domData.interactiveElements);
+  const hashRoutingDetected = domData.hashIndicators >= 3;
+  return {
+    url,
+    canonicalUrl: domData.canonicalHref ?? void 0,
+    statusCode,
+    title: domData.title,
+    metaTags: domData.metaTags,
+    headings: domData.headings,
+    landmarks: domData.landmarks,
+    links,
+    interactiveElements,
+    timing: timingData,
+    contentHash: domData.contentHash,
+    hashRoutingDetected
+  };
+}
+async function extractDomData(page) {
+  try {
+    return await page.evaluate(() => {
+      const canonicalEl = document.querySelector('link[rel="canonical"]');
+      const canonicalHref = canonicalEl?.href ?? null;
+      const title = document.title ?? "";
+      const metaEls = document.querySelectorAll("meta[name], meta[property], meta[content]");
+      const metaTags = [];
+      metaEls.forEach((el) => {
+        const content = el.getAttribute("content");
+        if (!content) return;
+        const entry = { content };
+        const name = el.getAttribute("name");
+        const property = el.getAttribute("property");
+        if (name) entry.name = name;
+        if (property) entry.property = property;
+        if (name || property) {
+          metaTags.push(entry);
+        }
+      });
+      const headingEls = document.querySelectorAll("h1, h2, h3, h4, h5, h6");
+      const headings = [];
+      headingEls.forEach((el) => {
+        const text = (el.textContent ?? "").trim();
+        if (!text) return;
+        const level = parseInt(el.tagName[1], 10);
+        headings.push({ level, text });
+      });
+      const landmarkSelectors = "nav, main, aside, footer, header, [role]";
+      const landmarkEls = document.querySelectorAll(landmarkSelectors);
+      const landmarks = [];
+      const seenLandmarks = /* @__PURE__ */ new Set();
+      landmarkEls.forEach((el) => {
+        const tagName = el.tagName.toLowerCase();
+        const explicitRole = el.getAttribute("role");
+        let role;
+        if (explicitRole) {
+          role = explicitRole;
+        } else {
+          const implicitRoles = {
+            nav: "navigation",
+            main: "main",
+            aside: "complementary",
+            footer: "contentinfo",
+            header: "banner"
+          };
+          role = implicitRoles[tagName] ?? tagName;
+        }
+        const label = el.getAttribute("aria-label") ?? el.getAttribute("aria-labelledby") ?? void 0;
+        const key = `${role}|${tagName}|${label ?? ""}`;
+        if (seenLandmarks.has(key)) return;
+        seenLandmarks.add(key);
+        const entry = { role, tagName };
+        if (label) entry.label = label;
+        landmarks.push(entry);
+      });
+      const linkEls = document.querySelectorAll("a[href]");
+      const links = [];
+      const seenHrefs = /* @__PURE__ */ new Set();
+      linkEls.forEach((el) => {
+        const href = el.getAttribute("href");
+        if (!href || href === "#" || href.startsWith("javascript:")) return;
+        const resolvedHref = el.href;
+        if (seenHrefs.has(resolvedHref)) return;
+        seenHrefs.add(resolvedHref);
+        const text = (el.textContent ?? "").trim();
+        const rel = el.getAttribute("rel");
+        links.push({ href: resolvedHref, text, rel });
+      });
+      const interactiveSelectors = 'button, input, select, textarea, [role="button"], [role="tab"], [role="combobox"], [role="listbox"], [role="slider"], [role="spinbutton"], [role="switch"]';
+      const interactiveEls = document.querySelectorAll(interactiveSelectors);
+      const interactiveElements = [];
+      interactiveEls.forEach((el, index) => {
+        const tagName = el.tagName.toLowerCase();
+        const type = el.getAttribute("type");
+        const role = el.getAttribute("role");
+        const ariaLabel = el.getAttribute("aria-label");
+        if (tagName === "input" && type === "hidden") return;
+        let text = "";
+        if (tagName === "input" || tagName === "textarea") {
+          text = el.placeholder ?? el.value ?? "";
+        } else {
+          text = (el.textContent ?? "").trim();
+        }
+        let selector;
+        const id = el.getAttribute("id");
+        if (id) {
+          selector = `${tagName}#${CSS.escape(id)}`;
+        } else {
+          const name = el.getAttribute("name");
+          if (name) {
+            selector = `${tagName}[name="${CSS.escape(name)}"]`;
+          } else {
+            const parent = el.parentElement;
+            if (parent) {
+              const siblings = Array.from(parent.querySelectorAll(`:scope > ${tagName}`));
+              const nth = siblings.indexOf(el) + 1;
+              const parentId = parent.getAttribute("id");
+              if (parentId) {
+                selector = `#${CSS.escape(parentId)} > ${tagName}:nth-of-type(${nth})`;
+              } else {
+                selector = `${tagName}:nth-of-type(${nth})`;
+              }
+            } else {
+              selector = `${tagName}[data-arch-index="${index}"]`;
+            }
+          }
+        }
+        interactiveElements.push({
+          tagName,
+          type,
+          text: text.slice(0, 200),
+          // cap text length
+          role,
+          ariaLabel,
+          selector
+        });
+      });
+      const bodyText = (document.body?.innerText ?? "").trim();
+      let hash = 5381;
+      for (let i = 0; i < bodyText.length; i++) {
+        hash = (hash << 5) + hash + bodyText.charCodeAt(i) | 0;
+      }
+      const contentHash = (hash >>> 0).toString(16).padStart(8, "0");
+      let hashIndicators = 0;
+      linkEls.forEach((el) => {
+        const h = el.getAttribute("href");
+        if (h && (h.startsWith("#/") || h.startsWith("#!/"))) {
+          hashIndicators++;
+        }
+      });
+      return {
+        canonicalHref,
+        title,
+        metaTags,
+        headings,
+        landmarks,
+        links,
+        interactiveElements,
+        contentHash,
+        hashIndicators
+      };
+    });
+  } catch {
+    return {
+      canonicalHref: null,
+      title: "",
+      metaTags: [],
+      headings: [],
+      landmarks: [],
+      links: [],
+      interactiveElements: [],
+      contentHash: "00000000",
+      hashIndicators: 0
+    };
+  }
+}
+async function extractTiming(page) {
+  try {
+    const raw = await page.evaluate(() => {
+      const entries = performance.getEntriesByType("navigation");
+      if (entries.length === 0) {
+        return { loadTime: 0, domContentLoaded: 0, firstContentfulPaint: null };
+      }
+      const nav = entries[0];
+      const loadTime = nav.loadEventEnd > 0 ? Math.round(nav.loadEventEnd - nav.startTime) : 0;
+      const domContentLoaded = nav.domContentLoadedEventEnd > 0 ? Math.round(nav.domContentLoadedEventEnd - nav.startTime) : 0;
+      let firstContentfulPaint = null;
+      const paintEntries = performance.getEntriesByType("paint");
+      for (const entry of paintEntries) {
+        if (entry.name === "first-contentful-paint") {
+          firstContentfulPaint = Math.round(entry.startTime);
+          break;
+        }
+      }
+      return { loadTime, domContentLoaded, firstContentfulPaint };
+    });
+    const timing = {
+      loadTime: raw.loadTime,
+      domContentLoaded: raw.domContentLoaded
+    };
+    if (raw.firstContentfulPaint != null) {
+      timing.firstContentfulPaint = raw.firstContentfulPaint;
+    }
+    return timing;
+  } catch {
+    return { loadTime: 0, domContentLoaded: 0 };
+  }
+}
+function processLinks(rawLinks, pageUrl, baseUrl) {
+  const results = [];
+  for (const raw of rawLinks) {
+    const resolved = resolveUrl(raw.href, pageUrl);
+    const isExternal = !isSameOrigin(resolved, baseUrl);
+    const link = {
+      href: resolved,
+      text: raw.text,
+      isExternal
+    };
+    if (raw.rel) {
+      link.rel = raw.rel;
+    }
+    results.push(link);
+  }
+  return results;
+}
+function processInteractiveElements(rawElements) {
+  return rawElements.map((raw) => {
+    const el = {
+      tagName: raw.tagName,
+      text: raw.text,
+      selector: raw.selector
+    };
+    if (raw.type) el.type = raw.type;
+    if (raw.role) el.role = raw.role;
+    if (raw.ariaLabel) el.ariaLabel = raw.ariaLabel;
+    return el;
+  });
+}
+
+export {
+  scanPage
+};
+//# sourceMappingURL=chunk-7ZQGW5OV.js.map