@reportforge/playwright-pdf 0.29.0 → 0.31.0

package/README.md

@@ -63,12 +63,12 @@ Three templates, one reporter. Every PDF is fully offline: fonts, charts, and sc
 - **Offline failure analysis**: sorts each failure into one of 7 root-cause buckets and renders a root-cause chip under each failed test in the `detailed` PDF (dominant-cause one-liner in `executive`). No network, no AI service: a small embedded classifier with automatic updates; pin it any time. Full details at [reportforge.org/docs/advanced/failure-analysis](https://reportforge.org/docs/advanced/failure-analysis).
 - **On-device training**: teach the classifier your team's failures without any data leaving your machine. Label locally-collected samples (`npx @reportforge/playwright-pdf label-feedback`), then `train-model` trains and cross-validates a personal layer on-device; it activates only when it measurably beats the base model on your own labeled set (`evaluate-model` shows the comparison any time). Commit the model file to your repo via `failureAnalysis.localModelPath` to share it with CI. Training data, the trained model, and evaluation are all local files; the analysis code cannot even name a network primitive (enforced by an automated source audit).
 - `npx @reportforge/playwright-pdf export-feedback`: merges your locally-collected feedback (tokenized + redacted) into one CSV for review or moving between machines. Local file only; there is no upload.
-- **Rich execution capture**: opt-in "Steps to Reproduce" outline (a nested, copy-pasteable Markdown list of your `test.step`s and assertions; add `apiSteps` for the full click/fill/check trail), Node console tail, and trace/video links under each failed test in the defect log. Read straight from the Playwright step tree: no fixtures, no test-code changes. Enable via `capture` (`steps` / `apiSteps` / `console` / `evidence`); off by default.
+- **Rich execution capture**: opt-in "Steps to Reproduce" outline (a nested, copy-pasteable Markdown list of your `test.step`s and assertions; add `apiSteps` for the full click/fill/check trail), Node console tail, and trace/video links under each failed test in the defect log. Read straight from the Playwright step tree: no fixtures, no test-code changes. Enable via `capture` (`steps` / `apiSteps` / `console` / `evidence`); off by default. Set `evidenceUrlTemplate` to turn those trace/video paths into clickable CI artifact links with a QR code beside each one, so even a printed report jumps straight to the trace viewer.
 - **Configurable sections**: add or remove any report section per template via `sections`.
 - **Executive report brief band**: leads with a plain-language brief (trend + root causes) and one hero pass-rate number instead of a flat KPI grid
 - **Data protection**: AES-256 password-protected PDFs (no external tools) and opt-in redaction of credentials/PII in report content and live streams
 - **Hybrid licensing**: one short key unlocks it; reports keep rendering offline via a locally cached license between refreshes
-- **Python and .NET support**: the same reports for pytest-playwright (`reportforge-playwright-pdf` on PyPI) and for NUnit/MSTest/xUnit (`ReportForge.Playwright.Pdf` on NuGet); one subscription key covers every ecosystem (live runs and step trees are Node.js-only today)
+- **Python and .NET support**: the same reports for pytest-playwright (`reportforge-playwright-pdf` on PyPI) and for NUnit/MSTest/xUnit (`ReportForge.Playwright.Pdf` on NuGet); one subscription key covers every ecosystem, and live runs stream from all three (step trees are Node.js-only today)
 - **Update notice**: when a newer reporter version is published, each run ends with one info line naming it and the update command; no extra network calls
 
 ---
@@ -180,7 +180,7 @@ template = "detailed"
 outputFile = "reports/{date}-{branch}-report.pdf"
 ```
 
-The plugin reuses the Node runtime and Chromium that Playwright for Python already ships, so there is nothing else to install. pytest-xdist and pytest-rerunfailures are supported; failure screenshots from `--screenshot only-on-failure` are embedded. Passing `--browser` more than once splits the report into one project per browser and unlocks the cross-browser comparison section. Live runs and step trees are Node.js-only today. Full guide: [reportforge.org/docs/python](https://reportforge.org/docs/python).
+The plugin reuses the Node runtime and Chromium that Playwright for Python already ships, so there is nothing else to install. pytest-xdist and pytest-rerunfailures are supported; failure screenshots from `--screenshot only-on-failure` are embedded, and trace/video artifacts from `--tracing` / `--video` feed the defect log's evidence links (pair with `evidenceUrlTemplate` for clickable CI links). Passing `--browser` more than once splits the report into one project per browser and unlocks the cross-browser comparison section. Enable `[tool.reportforge.live] enabled = true` and the run streams to a shareable watch page, same as Node.js; step trees stay Node.js-only today. Full guide: [reportforge.org/docs/python](https://reportforge.org/docs/python).
 
 ---
 
@@ -195,7 +195,7 @@ set RF_LICENSE_KEY=RFSU-XXXX-XXXX-XXXX-XXXX
 dotnet test --logger "reportforge;output=reports/run.pdf;template=detailed"
 ```
 
-Full options come from a JSON file with the same camelCase names as this reporter: `--logger "reportforge;optionsFile=rf-options.json"`. The logger reuses the Node runtime Microsoft.Playwright places in your test output and a Playwright-managed Chromium, so there is nothing else to install. Retried tests render as flaky with their attempt history; categories become tags. A `dotnet test` run drives one browser, so the cross-browser comparison section stays hidden here; live runs and step trees are Node.js-only today. Full guide: [reportforge.org/docs/dotnet](https://reportforge.org/docs/dotnet).
+Full options come from a JSON file with the same camelCase names as this reporter: `--logger "reportforge;optionsFile=rf-options.json"`. The logger reuses the Node runtime Microsoft.Playwright places in your test output and a Playwright-managed Chromium, so there is nothing else to install. Retried tests render as flaky with their attempt history; categories become tags. Trace archives and videos attached via `TestContext.AddTestAttachment` feed the defect log's evidence links (pair with `evidenceUrlTemplate` for clickable CI links). A `dotnet test` run drives one browser, so the cross-browser comparison section stays hidden here. Add `"live": { "enabled": true }` to the options file and the run streams to a shareable watch page (rows appear as tests finish); step trees stay Node.js-only today. Full guide: [reportforge.org/docs/dotnet](https://reportforge.org/docs/dotnet).
 
 ---
 
@@ -253,6 +253,8 @@ All options are the second element of the reporter tuple.
 | `showTrend` | `boolean` | `true` | Show the pass-rate sparkline and delta badge in the `detailed` template. Set to `false` to disable history tracking entirely. |
 | `remoteHistory` | `boolean` | `false` | Opt-in server-side trend store so the pass-rate trend survives ephemeral CI runners (the local history file is wiped with the workspace, leaving the chart stuck at one data point). One small authenticated request per run carrying aggregate numbers only — no test titles, no error text, and the branch name never travels in plaintext. Falls back to the local history file on any failure. See the History docs. |
 | `flakinessTopN` | `number` | `5` | Maximum flaky tests to show in the flakiness table (`detailed` template). Set to `0` to disable the table entirely. |
+| `quarantineThreshold` | `number` | `40` | Flake-rate percentage (0-100) at which a test gets a QUARANTINE chip in the flakiness table, plus a callout with the candidate count. Applies once a test has 3+ history runs with flakiness data. Set to `0` to disable flagging. |
+| `evidenceUrlTemplate` | `string` | n/a | URL template that turns local trace/video artifact paths (from `capture.evidence`) into clickable CI links with a QR code in the defect log. Must contain `{path}`, replaced with the forward-slashed artifact path, e.g. `https://ci.example.com/job/123/artifacts/{path}`. Unset = plain paths as before. |
 | `slowTestThreshold` | `number` | `10` | Minimum timed-test count before `SLOW` badges appear in the suite breakdown: below this, every test is trivially the "slowest" so badges stay hidden. On larger runs only genuine outliers (at least 2× the median test duration) are badged, so the badge stays rare. Set to `0` to drop the run-size gate (outliers still required). |
 | `requirementTagPattern` | `string` | `'^@?[A-Z][A-Z0-9]*-\\d+$'` | Regex deciding which tags count as requirement IDs in the Requirements Traceability section (`detailed` template). Matching tags (ticket shapes like `@ODP-5328`, `REQ-001`) get the traceability matrix; everything else (`@regression`, `@sanity`) collapses into a compact tag summary instead of repeating identical rows. Set to `''` to disable the split and list every tag in the matrix. |
 | `templatePath` | `string \| string[]` | n/a | Path to a custom Handlebars (`.hbs`) template file. Takes precedence over `template`. Pass an array to generate one PDF per custom template. See the Custom Templates docs. |
@@ -286,6 +288,8 @@ reporter: [
 
 Sharded runs need no extra config: every shard of one CI run converges on the same live view automatically. Set `RF_LIVE_RUN_ID` to override when your CI is not auto-detected.
 
+**Python and .NET stream too.** Enable live the same way (`[tool.reportforge.live] enabled = true` in `pyproject.toml`, or `"live": { "enabled": true }` in the .NET options file) with `RF_LICENSE_KEY` set, and the run prints the same watch link and streams per-test progress to the same page. pytest rows appear as tests start and finish; .NET's test platform reports results only on completion, so rows appear as each test finishes. Adapter runs are single-shard: parallel CI matrix jobs in one workflow should set a distinct `RF_LIVE_RUN_ID` per job so each gets its own watch page. The live step trail and console streaming remain Node.js-only.
+
 Live streaming is best-effort and entitlement-gated: if the server is unreachable the run is unaffected, and an active subscription with the `live` feature is required. If you've configured Slack/Teams/Discord notifications, the watch link is also posted to those channels when the run starts.
 
 > **Security:** the watch link exposes test titles, statuses, step titles, and failure details (error messages, code frames, stacks) to anyone holding it — no login needed, which is the point: share it with whoever should watch. Your `redact` rules apply to everything on the link, error text included. The token is scoped to that one run, and rotating your license key never breaks an already-shared link. Leaving `console: false` is recommended: enabling it streams your tests' stdout/stderr to anyone holding the link. Full details at [reportforge.org/docs/advanced/live](https://reportforge.org/docs/advanced/live).
@@ -409,7 +413,9 @@ Full details at [reportforge.org/docs/advanced/security](https://reportforge.org
 
 After each run the reporter appends a summary entry to a local JSON file and renders a pass-rate sparkline + verdict row in the `detailed` template.
 
-**Since Last Run**: every report also gets a section (`runDiff`, on by default in all three templates) comparing the current run against the previous run on the same branch: new failures, fixed tests, and tests still failing, with the same summary line carried into Slack, Teams, Discord, and email notifications. The first run after upgrading has no comparable prior run to diff against, so the section starts appearing from the second run onward. It reads only the local history file, so test names never leave your machine; see the ephemeral-CI note below for keeping that file around between runs. Like the flakiness table, it rides history tracking (`showTrend`, on by default), and the notification line follows history availability rather than the `runDiff` section toggle.
+**Flaky intelligence**: the flakiness table (`detailed` template) flags quarantine candidates — tests whose flake rate reaches `quarantineThreshold` (default 40%) across 3+ recorded runs get a QUARANTINE chip and a callout with the candidate count, and every row shows its current streak (consecutive latest runs flaky). Set `quarantineThreshold: 0` to turn flagging off.
+
+**Since Last Run**: every report also gets a section (`runDiff`, on by default in all three templates) comparing the current run against the previous run on the same branch: new failures, newly flaky tests, fixed tests, and tests still failing, with the same summary line carried into Slack, Teams, Discord, and email notifications. The first run after upgrading has no comparable prior run to diff against, so the section starts appearing from the second run onward. It reads only the local history file, so test names never leave your machine; see the ephemeral-CI note below for keeping that file around between runs. Like the flakiness table, it rides history tracking (`showTrend`, on by default), and the notification line follows history availability rather than the `runDiff` section toggle.
 
 **Local dev (zero config)**: history is written automatically to `~/.reportforge/{projectKey}/history.json`. The sparkline appears once two or more runs have been recorded.
 

package/dist/cli/index.js

@@ -293,20 +293,26 @@ function buildDemoCharts(passed, failed, skipped, flaky, total) {
       delta: 71 - 76
       // -5 vs previous run
     },
+    flakinessQuarantineTotal: 1,
+    flakinessQuarantineThreshold: 40,
     flakinessTable: [
       {
         title: "should handle session timeout gracefully",
         flakeRate: 60,
         flakeCount: 3,
         totalRuns: 5,
-        sparkline: [true, false, true, false, true]
+        sparkline: [true, false, true, false, true],
+        streak: 1,
+        quarantine: true
       },
       {
         title: "should apply valid promo code",
         flakeRate: 20,
         flakeCount: 1,
         totalRuns: 5,
-        sparkline: [false, false, false, true, false]
+        sparkline: [false, false, false, true, false],
+        streak: 0,
+        quarantine: false
       }
     ]
   };
@@ -320,6 +326,9 @@ function buildDemoRunDiff() {
     stillFailing: [
       { id: "co-3", title: "should reject expired promo code" }
     ],
+    newlyFlaky: [
+      { id: "auth-4", title: "should handle session timeout gracefully" }
+    ],
     fixed: [
       { id: "co-5", title: "should complete Stripe checkout" }
     ],
@@ -6381,6 +6390,7 @@ var init_defaults = __esm({
       historySize: 10,
       showTrend: true,
       flakinessTopN: 5,
+      quarantineThreshold: 40,
       slowTestThreshold: 10,
       // Ticket-ID shape: @ODP-5328, REQ-001, JIRA-123. Tags matching this land in
       // the Requirements Traceability matrix; the rest fall to the tag summary.
@@ -12250,6 +12260,10 @@ var init_schema = __esm({
       // local history file is wiped every run. Aggregate numbers only.
       remoteHistory: external_exports.boolean().default(false),
       flakinessTopN: external_exports.number().int().min(0, "flakinessTopN must be 0 or greater").optional().default(DEFAULT_OPTIONS.flakinessTopN),
+      quarantineThreshold: external_exports.number().int().min(0, "quarantineThreshold must be between 0 and 100").max(100, "quarantineThreshold must be between 0 and 100").optional().default(DEFAULT_OPTIONS.quarantineThreshold),
+      evidenceUrlTemplate: external_exports.string().refine((s) => s.includes("{path}"), {
+        message: "evidenceUrlTemplate must contain the {path} token"
+      }).optional(),
       slowTestThreshold: external_exports.number().int().min(0, "slowTestThreshold must be 0 or greater").optional().default(DEFAULT_OPTIONS.slowTestThreshold),
       requirementTagPattern: external_exports.string().refine((v) => {
         try {

package/dist/index.d.mts

@@ -368,6 +368,24 @@ interface ReporterOptions {
      * @default 5
      */
     flakinessTopN?: number;
+    /**
+     * Flake-rate percentage (0-100) at which a test is flagged as a quarantine
+     * candidate in the flakiness table: a QUARANTINE chip on the row plus a
+     * callout with the candidate count. A test qualifies once its flake rate
+     * over the stored history reaches the threshold across at least 3 runs
+     * that carry flakiness data. Set to 0 to disable flagging.
+     * @default 40
+     */
+    quarantineThreshold?: number;
+    /**
+     * URL template that turns local trace/video artifact paths (surfaced by
+     * `capture.evidence`) into clickable CI links with a QR code in the defect
+     * log. Must contain the `{path}` token, replaced with the forward-slashed
+     * artifact path (each segment percent-encoded), e.g.
+     * `https://ci.example.com/job/123/artifacts/{path}`.
+     * Unset = evidence renders as plain paths, exactly as before.
+     */
+    evidenceUrlTemplate?: string;
     /**
      * Minimum number of timed tests before `SLOW` badges appear in the suite
      * breakdown — below this, every test is trivially the "slowest", so badges

package/dist/index.d.ts

@@ -368,6 +368,24 @@ interface ReporterOptions {
      * @default 5
      */
     flakinessTopN?: number;
+    /**
+     * Flake-rate percentage (0-100) at which a test is flagged as a quarantine
+     * candidate in the flakiness table: a QUARANTINE chip on the row plus a
+     * callout with the candidate count. A test qualifies once its flake rate
+     * over the stored history reaches the threshold across at least 3 runs
+     * that carry flakiness data. Set to 0 to disable flagging.
+     * @default 40
+     */
+    quarantineThreshold?: number;
+    /**
+     * URL template that turns local trace/video artifact paths (surfaced by
+     * `capture.evidence`) into clickable CI links with a QR code in the defect
+     * log. Must contain the `{path}` token, replaced with the forward-slashed
+     * artifact path (each segment percent-encoded), e.g.
+     * `https://ci.example.com/job/123/artifacts/{path}`.
+     * Unset = evidence renders as plain paths, exactly as before.
+     */
+    evidenceUrlTemplate?: string;
     /**
      * Minimum number of timed tests before `SLOW` badges appear in the suite
      * breakdown — below this, every test is trivially the "slowest", so badges