@sentinelqa/playwright-reporter 0.1.53 → 0.1.56

package/README.md

@@ -1,210 +1,192 @@
-# Playwright Reporter
+# @sentinelqa/playwright-reporter
 
-After every failed run, reporter prints a shareable debugging link:
-
-Sample run: https://app.sentinelqa.com/share/1f343d91-be17-4c14-b1b9-2d4e8ef448d2
-
-Open it to inspect failures instantly or share it in Slack, PRs, or GitHub issues.
+Deterministic Playwright failure triage: CLI diagnosis, root-cause grouping, and a shareable debugging report.
 
 [![npm](https://img.shields.io/npm/v/@sentinelqa/playwright-reporter)](https://www.npmjs.com/package/@sentinelqa/playwright-reporter)
 [![downloads](https://img.shields.io/npm/dm/@sentinelqa/playwright-reporter)](https://www.npmjs.com/package/@sentinelqa/playwright-reporter)
 [![license](https://img.shields.io/npm/l/@sentinelqa/playwright-reporter)](./LICENSE)
 
-From failed CI run → root cause in seconds.
-Get a shareable Playwright debugging link with traces, screenshots, and failure context — no setup required.
+## What You Get
 
-Works with no account or API key required.
+SAMPLE REPORT: https://sentinelqa.com/share/run/permanent-demo-playwright-report
 
-Use it to get a shareable hosted run link from CI or local development, then upgrade to Sentinel Cloud for richer history and intelligence.
+![CLI Output](./docs/cli.png)
 
-![Sentinel Report Example](./docs/screenshot2.png)
-![CLI Quick Diagnosis](./docs/CLI1.png)
+After a failed Playwright run, Sentinel prints:
 
-## Features
+- a deterministic CLI diagnosis (1–3 root causes)
+- how many tests are affected per root cause
+- what to inspect first (usually trace)
+- a report link (hosted by default, local in offline mode)
 
-- Free hosted debugging links by default, with no account or API key required
-- Public run page that opens on unified failures across the run
-- Within-run failure grouping so repeated failures collapse into one issue
-- Public failure pages with screenshots, evidence, parsed errors and light summaries
-- Copyable share actions for Slack, PRs, and debugging handoff
-- Deterministic quick diagnosis in the terminal after failed runs
-- Playwright traces, screenshots, videos and logs uploaded automatically
-- 48-hour public share links on the free hosted flow
-- Works with existing Playwright reporter setup
-- Optional live failure capture for richer Sentinel Cloud analysis
-- CI run history, retention, and deeper AI debugging in Sentinel Cloud
+## Requirements
 
-## Why this exists
+- Node.js 18+
+- `@playwright/test` 1.40+ (newer is recommended)
 
-Debugging Playwright failures usually means downloading traces, screenshots, and logs separately from CI.
+## Install
 
-Reporter uploads those artifacts into a single hosted Sentinel run page so you can open one link, inspect failures fast, and share that link with the rest of the team.
+```bash
+npm install -D @sentinelqa/playwright-reporter
+```
 
-## CLI diagnosis
+## Setup (Recommended)
 
-Sentinel is not just a report link.
+Wrap your `playwright.config.ts`:
 
-On failed runs, it prints a compact terminal diagnosis that answers:
+```ts
+import { defineConfig } from "@playwright/test";
+import { withSentinel } from "@sentinelqa/playwright-reporter";
 
-1. what broke
-2. why it broke
-3. where to look
-4. what changed
-5. what to do next
+export default withSentinel(
+  defineConfig({
+    reporter: [["line"]],
+    outputDir: "test-results",
+  }),
+  {
+    project: "my-app",
+  },
+);
+```
 
-The goal is simple:
+Run tests normally:
 
-you should not need to open logs first.
+```bash
+npx playwright test
+```
 
-Typical CLI output:
+## How It Works (High Level)
 
-```text
-Sentinel diagnosis
-━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+Sentinel is a Playwright reporter that:
 
-NEW FAILURE after 8 passing runs
+1. Ensures a Playwright JSON report exists (adds a JSON reporter if needed).
+2. Collects the artifacts Playwright already produces (`trace.zip`, screenshots, video, logs, report.json).
+3. Builds a **deterministic** failure summary:
+   - groups repeated failures into 1–3 canonical root causes
+   - extracts normalized evidence (where, blocker, target state, expected/received)
+4. Publishes a report:
+   - default: a hosted share link
+   - offline: a local HTML report folder (no upload)
 
-What broke: 10 tests failed
-Why: collapsed into 2 real issues
+This project is intentionally heuristic-driven (not “AI guesses”) for root-cause grouping.
 
-Issue 1: UI assertion mismatch (6 tests)
-  Why: getByTestId('metric-pass-rate') showed "82%" instead of "88%"
-  Where: app.spec.ts:43, app.spec.ts:69
-  Expected: 88%
-  Received: 82%
-  What changed: "update analytics cards"
-  Next: verify getByTestId('metric-pass-rate')
-  Impact: 6 tests failing with same root cause
-  Clears: fixing this likely clears 6 of 10 failures
-```
+## Modes
 
-On passed runs, Sentinel stays short and only prints a warning if there is a strong risk signal worth caring about.
+Sentinel behavior is controlled by `SENTINEL_MODE`.
 
-## Why teams use the free version
+## Pricing Tiers (How This Maps)
 
-- Drop one wrapper into `playwright.config.ts` and keep running `npx playwright test`
-- Get a hosted Sentinel debugging link automatically on failed runs
-- Share one public URL in Slack, PRs, or GitHub issues instead of passing around raw CI artifacts
-- See unified failures, grouped failure patterns, screenshots, and evidence in one place
-- Let teammates inspect the failure without needing your CI system or local machine
+- **Community (Open Source)**
+  - “Everything local”: generate the same report UI fully offline, no upload required.
+  - “Bring your own storage + DB”: artifacts and metadata stay in your VPC (S3/GCS + Postgres).
+  - “Deterministic root-cause clustering”: same grouping logic as the hosted product, no AI guessing.
+  - “No limits”: unlimited runs because you own infra.
+  - “Security posture”: secrets masking happens before anything leaves your environment (and in offline, nothing leaves).
+  - Share links are optional: run in hosted mode when you want a public URL, or offline mode for zero egress.
 
-## Requirements
+- **Personal ($0)**
+  - “Zero setup”: install + run, auto-upload on failures.
+  - “Share links for PRs”: reviewers open a read-only report without reproducing locally.
+  - “Retention”: keep enough history to see patterns (200 runs / 14 days).
+  - “Safe-by-default”: automatic secret masking before upload.
+  - “Works with shards”: parallel jobs still collapse into root-cause clusters.
 
-- Node.js 18+
-- `@playwright/test` 1.40+
+- **Pro ($50/mo)**
+  - “Stop repeating fixes”: recurring failure tracking across commits (first seen, frequency, impact).
+  - “Compare runs”: last known good vs now, isolate what changed.
+  - “Team workflow”: shared workspace, multiple projects, roles.
+  - “High volume”: 10,000 run history and longer retention.
+  - “Faster decision making”: regressions vs flakes becomes obvious (trend + history).
 
-## Quick Start
+The reporter package works in all tiers; the difference is where reports live and how much history you get.
 
-`withSentinel()` is the default setup for everyone:
+### Hosted Mode (Default)
 
-- best for free and local users
-- zero-friction setup
-- hosted Sentinel report link is generated automatically
-- no `SENTINEL_TOKEN` required
-- AI summaries use trace and reporter evidence, but are less precise than live page capture
+Do nothing. Sentinel uploads and prints a share link when failures happen.
 
-Install:
+Environment:
 
-```bash
-npm install -D @sentinelqa/playwright-reporter
-```
+- `SENTINEL_MODE` unset (or `SENTINEL_MODE=hosted`)
 
-Add Sentinel to your Playwright config:
+### Workspace Mode (Private History)
 
-```ts
-import { defineConfig } from "@playwright/test";
-import { withSentinel } from "@sentinelqa/playwright-reporter";
+If you have a workspace token:
 
-export default withSentinel(
-  defineConfig({
-    reporter: [["line"]],
-    outputDir: "test-results",
-    use: {
-      trace: "retain-on-failure",
-      screenshot: "only-on-failure",
-      video: "retain-on-failure",
-    },
-  }),
-  {
-    project: "my-app",
-  },
-);
+```bash
+SENTINEL_TOKEN=your_project_ingest_token npx playwright test
 ```
 
-## Example
+Environment:
+
+- `SENTINEL_MODE` unset (or `SENTINEL_MODE=hosted`)
+- `SENTINEL_TOKEN=...`
 
-Run your Playwright tests:
+### Offline Mode (No Uploads)
+
+To keep everything local:
 
 ```bash
-npx playwright test
+SENTINEL_MODE=offline npx playwright test
 ```
 
-If tests fail, Sentinel uploads a hosted debugging report and prints the shareable link in the terminal.
+Offline mode is strict:
 
-Open the hosted report to inspect:
+- uploads are skipped
+- a local report is generated (default `./sentinel-report/index.html`)
+- the CLI prints a local `file://` link (or a relative path)
 
-- failed tests across jobs
-- within-run grouped failures
-- screenshots and videos
-- trace links
-- parsed failure details
-- light summaries
-- shareable public debugging page
+If a hosted upload fails, Sentinel falls back to generating the local report automatically.
 
-The free hosted public flow is designed for distribution:
+### Local Workspace Uploads (When Not In CI)
 
-- one shareable debugging link per run
-- public read-only pages
-- fast enough to use in CI comments and Slack threads
-- clear upgrade path into a full Sentinel workspace when teams want history, retention, and deeper analysis
+If you set `SENTINEL_TOKEN` locally, Sentinel will _not_ upload by default (to avoid accidental data egress).
+To allow a local upload to your workspace, set:
 
-## Modes
+```bash
+SENTINEL_UPLOAD_LOCAL=1 SENTINEL_TOKEN=your_project_ingest_token npx playwright test
+```
+
+This is useful for quickly generating a private run history entry from your laptop.
+
+### Implicit Local Public Upload (No Token, Not In CI)
+
+If you are not in CI and you did not set `SENTINEL_TOKEN`, Sentinel can still upload in public mode by enabling:
 
-### Free hosted mode
+```bash
+SENTINEL_UPLOAD_LOCAL=1 npx playwright test
+```
 
-If `SENTINEL_TOKEN` is not set, the reporter uploads the run to a hosted public Sentinel report and prints the shareable URL.
+### Share Links Are Optional (Open Source / Self-Hosted)
 
-This free public flow includes:
+- Hosted mode prints a share URL on failures.
+- Offline mode skips uploads and produces only a local report.
 
-- hosted run page
-- hosted failure pages
-- grouped failures inside the run
-- light summaries
-- copy/share actions
-- 48-hour share links
+Pick the behavior explicitly with `SENTINEL_MODE` (and `SENTINEL_UPLOAD_LOCAL` for local uploads).
 
-### Workspace mode
+## Serving Local Reports
 
-If `SENTINEL_TOKEN` is set, the reporter uploads into your Sentinel workspace instead of the free hosted public flow.
+Local reports are static HTML + copied artifacts. You can open `index.html` directly, or run a tiny local server:
 
 ```bash
-SENTINEL_TOKEN=your_project_ingest_token npx playwright test
+cd sentinel-report
+npx --yes serve -p 4173 .
 ```
 
-For local runs outside CI, Sentinel will use your local git metadata automatically when available.
-
-## What `withSentinel()` does
+Then open `http://localhost:4173`.
 
-- Preserves your existing reporter configuration
-- Injects a Playwright JSON reporter if one is missing
-- Sets sensible artifact defaults:
-  - trace: `retain-on-failure`
-  - screenshot: `only-on-failure`
-  - video: `retain-on-failure`
-- Uploads the run to hosted Sentinel at the end of the test run
+## Secrets Masking
 
-## Recommended Cloud Setup
+Sentinel masks common secret patterns before data is shared. This includes:
 
-If you use Sentinel Cloud and want the best AI summaries and fix suggestions, keep `withSentinel()` in your Playwright config and add the live capture fixture.
+- environment-variable looking strings
+- common credential/token patterns
+- internal URLs/hosts (where possible)
 
-Why:
+If you find something that should be masked but isn’t, file an issue with a minimal reproducible example (redact the secret).
 
-- `withSentinel()` alone works from reporter and trace data
-- a Playwright reporter does not get the live `page` fixture
-- the live capture fixture lets Sentinel collect richer DOM and code context at the exact failure moment
-- this is required for the highest-quality DOM-aware patches
+## Optional: Richer Failure Evidence
 
-Create one shared test wrapper:
+`withSentinel()` is enough for most setups. If you want richer UI evidence for actionability/timeouts, you can attach the failure capture fixture:
 
 ```ts
 // tests/test.ts
@@ -215,22 +197,13 @@ export const test = attachSentinelFailureCapture(base);
 export { expect };
 ```
 
-Then import from that file in your specs instead of `@playwright/test`:
+Then import from that file in your specs:
 
 ```ts
 import { test, expect } from "./test";
 ```
 
-Use this cloud setup when you want:
-
-- best AI summaries
-- best fix suggestions
-- richer DOM-aware diagnosis
-- more reliable code patches grounded in real page state
-
-Free and local-only users do not need this. The standard `withSentinel()` setup remains the simplest path and will upload a hosted Sentinel report automatically.
-
-## Options
+## Configuration Options
 
 ```ts
 withSentinel(config, {
@@ -243,19 +216,23 @@ withSentinel(config, {
 });
 ```
 
-## Sentinel Cloud (optional)
+## Troubleshooting
+
+### “POST /api/runs failed …”
+
+- Check `SENTINEL_MODE` (offline skips uploads).
+- If using a workspace token, confirm `SENTINEL_TOKEN` is correct.
+- If a server schema changed, upgrade the reporter/uploader and/or run DB migrations for self-hosted deployments.
+
+### “Local report generation failed …”
+
+Make sure Playwright produced:
+
+- `playwright-report/report.json`
+- `test-results/` (or your configured `outputDir`)
 
-Sentinel Cloud adds:
+Then rerun with `SENTINEL_MODE=offline` to force local generation.
 
-- hosted debugging dashboards
-- CI run history
-- AI-generated failure summaries
-- flaky test detection
-- shareable run links
-- longer retention
-- compare against previous runs
-- recurring failure history
-- richer fix suggestions and team workflows
+## License
 
-Free for up to 100 CI runs per month.
-Create an account at [sentinelqa.com](https://sentinelqa.com).
+See [LICENSE](./LICENSE).