@qartisan/playwright-loupe 0.1.2 → 0.1.3

package/CHANGELOG.md

@@ -4,6 +4,21 @@ All notable changes to this project are documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). Before 1.0, minor versions may contain breaking changes. The JSON report has its own version (`schemaVersion`), see [`docs/schema/v1.md`](docs/schema/v1.md#versioning-policy).
 
+## [0.1.3] - 2026-10-21
+
+### Added
+
+- **Common failure points.** A new Overview card, after the run errors and before the error categories, lists the functions of your code that appear in the stack of several failures, in different spec files and error groups: when one broken fixture or helper fails many tests with different messages, it shows the function they share. A lead, not a diagnosis. Each point gives its function, file and the lines seen, the functions folded under it ("via …"), its tests, files and error groups, the failed / flaky split and the categories involved, and opens the matching tests. Thresholds: 5 tests (a test in several projects counts once), 3 spec files, 2 error groups; 5 points at most. Spec files, anonymous functions, `node_modules`, Node and Playwright internals are left out; helpers outside the root dir are kept, with `../` paths. See [Common failure points](README.md#common-failure-points).
+- **Failure point filter** in the Tests view and the sidebar, after the error category (`point` in the URL, `#` encoded as `%23`).
+- **Error groups only seen after a test timeout are folded.** When the test timeout stops a test, the pending action fails too, as a second error of the attempt (`Request context disposed`, `Target page, context or browser has been closed`…). A group whose every error is such a secondary error (after `Test timeout of <n>ms exceeded.`, or the fixture and hook variants of the test timeout) now sits in a collapsed disclosure at the end of the Errors view's list and of each category, and group counts leave it out with a mention (`22 error groups · 18 more only seen after a test timeout`). Test counts and test categories are unchanged; `n` / `p` walk these groups last; their pages stay reachable, and the test detail labels the error "Raised after the test timeout". See [Groups only seen after a test timeout](README.md#groups-only-seen-after-a-test-timeout).
+- **Why a test was skipped.** The reason given in the test code (`test.skip(condition, 'reason')`, `test.fixme(…)`, `testInfo.skip(…)`, in a test, a `describe` or a hook) now shows as `Skipped: <reason>` or `Fixme: <reason>` in Suite health, on a muted line under the title in the Tests view (one line, the full text in the tooltip and the accessible name), and next to the outcome in the test detail header. Without a reason: `Skipped (no reason given)` / `Fixme (no reason given)`. A test Playwright skipped without any skip annotation (a failed dependency project or earlier test of a serial group, `--max-failures`, a stopped run) shows `Not run`, with a sentence listing these usual causes, never asserting one. The search (`q`) also matches the reason's text. Read from the existing annotations: no change to the JSON. See [Skip reasons](README.md#skip-reasons).
+- `npm run perf:sample` writes a third dataset, `cascade`, with realistic stacks through invented helpers (some outside the test dir) and 50 spec files, and also times the opening of the Overview.
+
+### Changed
+
+- **Error signatures of `toBeOK()` keep the response status.** For `expect(response).toBeOK()` (and `not.toBeOK()`), the status is only in Playwright's call log, which signatures cut: a 429, a 404 and a 503 shared one error group, shown under the category of its first error. The signature now keeps the final response line of the call log (`← 503 Service Unavailable`, the last one after redirects), so each status gets its own group, matching its category. The same status on different endpoints still shares one group. Other messages keep their signature; `toBeOK()` failures get a new one, so their groups do not match those of earlier reports. No change to the JSON shape (`schemaVersion` stays 1). See [Error signature](docs/schema/v1.md#error-signature).
+- Suite health's skipped tests use the new wording (`Skipped: <reason>`, `Fixme (no reason given)`, `Not run`) instead of a `fixme` badge and "No reason given"; a test with several skip annotations shows the first one with a reason.
+
 ## [0.1.2] - 2026-10-21
 
 ### Security

package/README.md

@@ -38,8 +38,8 @@ A run that broke outside its tests (a spec file that fails to import, a failing
 The report is a single self-contained HTML file. It works offline, straight from `file://`, in light and dark themes.
 
 - **Header:** the run's verdict and status counts, its start, duration and environment; the shortcuts and theme controls stay top-right at every width.
-- **Overview:** verdict with pass rate, run errors first when the run broke outside its tests, error categories, project cards, and the run details in three groups: Run (scope, CI, workers, shard), Code (commit, short with a copy button; branch; repository) and Environment (Playwright, Loupe, Node, OS). Empty values are left out.
-- **Errors:** every error group, built from normalized error signatures, with the failed / flaky split and links to the affected tests. A group page has previous / next buttons: from a category, they stay within that category.
+- **Overview:** verdict with pass rate, run errors first when the run broke outside its tests, [common failure points](#common-failure-points), error categories, project cards, and the run details in three groups: Run (scope, CI, workers, shard), Code (commit, short with a copy button; branch; repository) and Environment (Playwright, Loupe, Node, OS). Empty values are left out.
+- **Errors:** every error group, built from normalized error signatures, with the failed / flaky split and links to the affected tests. Groups [only seen after a test timeout](#groups-only-seen-after-a-test-timeout) are folded at the end. A group page has previous / next buttons: from a category, they stay within that category.
 - **Test detail:** title path, outcome, project, `file:line`, tags, annotations and dimensions, then one tab per attempt (the last failing attempt first; `?attempt=` in the URL): errors with Expected / Received and diffs highlighted and the code snippet, steps (`test.step` and failing steps, the failing path expanded; "Show all steps" for hooks, fixtures and API calls), attachments, stdout and stderr. Previous / next walk the Tests view's current list.
 - **Attachments**, per attempt, linked from `data/` next to the report (never inlined in the HTML), all working from `file://`:
   - **Screenshot comparisons**: a failed `toHaveScreenshot` / `toMatchSnapshot` gives one card per image name, before the other screenshots, with four modes: Diff, Side by side (expected | actual, at one scale), Slider (actual over expected, a divider to drag or move with the arrow keys) and Overlay (actual over expected, its opacity to set). Only the modes the images allow are offered; Diff is the default when there is a diff image. The chosen mode applies to every comparison until the report is closed. Each image's size is shown, with a warning when expected and actual differ: slider and overlay align them top-left. "Open in viewer" shows the same modes in the viewer below.
@@ -50,10 +50,40 @@ The report is a single self-contained HTML file. It works offline, straight from
 - **Copy buttons:** rerun commands (this test, the failed tests of its file, the failed tests of the current list) and an AI prompt describing the failure, in English. Both are only copied to the clipboard; nothing is sent anywhere. Projects failing on the same lines share one command (`--project=a --project=b`); otherwise each project gets its own, joined with `&&`.
   - The commands target **POSIX shells** (bash, zsh; not `cmd.exe`) and run from the Playwright root directory: paths in the report are relative to it.
   - A `file:line` argument targets the line where the test is declared and reruns every test declared on that line. A command is only valid while the file has not changed since the run.
-- **Suite health:** dimensions from tags and annotations (with hints for unknown or inconsistent values), slowest, skipped and flaky tests.
-- **Tests:** every test in one sortable list (by default in declaration order, as in the sidebar: project, file, line; the Status header sorts failed, flaky, passed, skipped), filtered by status, error category, project, tag and dimension, with the count each filter would give. Header counts, error categories and dimension values open it pre-filtered. Large lists show their first 500 rows, with "Show 500 more" and "Show all". Under each title, chips show what the test covers: its dimension values first (dimensions in config order, the value alone, `api`, with the dimension in the tooltip), then its other tags (`@smoke`), sorted. A tag a dimension takes appears only as that dimension's value. A chip adds its value to the filters, keeping the others and the sort; an active chip (with a ×) removes it, and Back undoes either. Up to four chips per row, active chips always among them; "+N" shows the rest.
-- **Sidebar:** search (`/` to focus; every word must match, ignoring case and accents, the title path, file, project, tags or dimension values: `smoke`, `api`, `layer:api`), projects, folders, files, `describe` blocks and tests, with status icons, failure counts and keyboard navigation. The folder shared by every file is not repeated.
-- **Filters in the URL:** search, filters and sort live in the link, so a filtered view can be shared or bookmarked, even from `file://`. A project without a `name` (Playwright's default project) cannot be selected in the project filter: an empty name cannot be written in the URL. Give your projects a name to filter by them. Each tag is filterable in one place only: a tag a dimension is built from (`@layer:ui` with a `layer` dimension, configured or automatic) is in that dimension's filter (Layer: `ui`), not in the tag filter, so the two can never be combined into an empty result. Unprefixed tags (`@smoke`) and prefixed tags no dimension uses stay in the tag filter.
+- **Suite health:** dimensions from tags and annotations (with hints for unknown or inconsistent values), slowest, skipped (with why, see [Skip reasons](#skip-reasons)) and flaky tests.
+- **Tests:** every test in one sortable list (by default in declaration order, as in the sidebar: project, file, line; the Status header sorts failed, flaky, passed, skipped), filtered by status, error category, failure point, project, tag and dimension, with the count each filter would give. Header counts, error categories, failure points and dimension values open it pre-filtered. Large lists show their first 500 rows, with "Show 500 more" and "Show all". Under each title, chips show what the test covers: its dimension values first (dimensions in config order, the value alone, `api`, with the dimension in the tooltip), then its other tags (`@smoke`), sorted. A tag a dimension takes appears only as that dimension's value. A chip adds its value to the filters, keeping the others and the sort; an active chip (with a ×) removes it, and Back undoes either. Up to four chips per row, active chips always among them; "+N" shows the rest. A skipped test shows why on a muted line under its title, above the chips.
+- **Sidebar:** search (`/` to focus; every word must match, ignoring case and accents, the title path, file, project, tags, dimension values or the reason a test was skipped: `smoke`, `api`, `layer:api`, `sandbox`), projects, folders, files, `describe` blocks and tests, with status icons, failure counts and keyboard navigation. The folder shared by every file is not repeated.
+- **Filters in the URL:** search, filters and sort live in the link, so a filtered view can be shared or bookmarked, even from `file://`. A project without a `name` (Playwright's default project) cannot be selected in the project filter: an empty name cannot be written in the URL. Give your projects a name to filter by them. Each tag is filterable in one place only: a tag a dimension is built from (`@layer:ui` with a `layer` dimension, configured or automatic) is in that dimension's filter (Layer: `ui`), not in the tag filter, so the two can never be combined into an empty result. Unprefixed tags (`@smoke`) and prefixed tags no dimension uses stay in the tag filter. The failure point filter (`point`, the value `<file>#<function>` encoded like any other, `#` as `%23`: `point=fixtures%2Fsession.ts%23signIn`) offers the points of the Overview card; a test matches when the function is in the stack of one of its failing attempts.
+
+### Skip reasons
+
+A skipped test says why, in Suite health, under its title in the Tests view and next to its outcome in the test detail. The reason is the one given in your test code, read from the `skip` / `fixme` annotation Playwright records for `test.skip(condition, 'reason')`, `test.fixme(condition, 'reason')`, `testInfo.skip(…)`, in a test, a `describe` or a hook:
+
+- `Skipped: <reason>` / `Fixme: <reason>`;
+- `Skipped (no reason given)` / `Fixme (no reason given)` when the call has no reason (`test.skip('title', body)`, `test.describe.fixme`…);
+- `Not run` when Playwright skipped the test without any skip annotation: usually because a dependency project or an earlier test of its serial group failed, the failure limit (`--max-failures`) was reached, or the run was stopped. The report does not say which, so Loupe does not guess.
+
+The search matches the reason's text (not the `Skipped` or `Not run` labels). Reasons go through the same secret redaction as every other string of the report.
+
+### Common failure points
+
+When one problem breaks many tests, their errors rarely share a message: the same function fails at different lines, with different errors, in different categories. The Overview card **Common failure points** lists the functions of your code that appear in the stack of several failures, in different spec files and error groups. **It is a lead, not a diagnosis:** a shared helper also shows up when unrelated failures go through it.
+
+- **What is counted:** every error of every failing attempt, flaky tests included (the card gives the failed / flaky split), read from its stack trace. A point is a function in a file, whatever the line: `fixtures/session.ts#signIn`, with the lines seen. Paths are relative to the Playwright root dir, with `../` for helpers outside it (`../helpers/api.ts`). Spec files (any file holding a test of the report), anonymous functions, `node_modules`, Node internals and Playwright's own frames are left out.
+- **Thresholds:** a point is shown when at least **5** distinct tests (a test run in several projects counts once), **3** spec files and **2** error groups go through it. At most **5** points, the most tests first. A point below a threshold is shown nowhere.
+- **Folding:** a function only reached through another one, by some of its tests, is listed under it (`via checkResponse`) rather than on its own. When their tests only partly overlap, both are shown.
+- Each point opens its tests in the Tests view (the Failure point filter). The root dir is found from the stacks (the frames of each test's own file); when no stack shows it, the card is hidden.
+
+Nothing is configured and nothing changes in your tests: the points are derived when the report opens, from the stacks Playwright records. The thresholds may change while the heuristic is tuned.
+
+### Groups only seen after a test timeout
+
+When a test times out, Playwright stops it and the action still running fails too (`Request context disposed`, `Target page, context or browser has been closed`, `locator.click: Test timeout of 30000ms exceeded`): a second error of the same attempt. Such follow-on errors can make up many error groups of a slow run.
+
+- An error is **secondary** when it is not the first error of its attempt and that first error is Playwright's test timeout: `Test timeout of <n>ms exceeded.`, also while setting up or tearing down a fixture, in a `beforeEach` / `afterEach` hook, a fixture's own timeout and a `beforeAll` / `afterAll` hook timeout.
+- A group is **folded** when every one of its errors is secondary. A group seen at least once as a first error stays visible.
+- In the Errors view, folded groups move to a collapsed "N groups only seen after a test timeout" at the end of the list and of each category. Group counts leave them out and say so (`22 error groups · 18 more only seen after a test timeout`), on the Overview too; test counts are unchanged. `n` / `p` walk the visible groups first, then the folded ones.
+- A folded group's page stays reachable, and in the test detail each secondary error keeps its place, labelled "Raised after the test timeout". The category of a test does not change: the second error is sometimes the most telling one.
 
 ### Keyboard shortcuts
 

package/dist/core/signature.d.ts

@@ -7,7 +7,8 @@ export interface NormalizeOptions {
 }
 /**
  * Normalizes an error message so the same failure produces the same text
- * across tests, retries and runs: ANSI codes and the call log are removed,
+ * across tests, retries and runs: ANSI codes and the call log are removed
+ * (except the final response status of a `toBeOK()` failure),
  * only the stable part of Playwright messages is kept, volatile values are
  * replaced with placeholders and whitespace is collapsed.
  */

package/dist/core/signature.js

@@ -7,6 +7,30 @@ export function stripAnsi(text) {
 }
 /** Playwright appends the action log after a line starting with "Call log:". */
 const CALL_LOG = /^[ \t]*Call log:/m;
+/**
+ * Playwright's own `toBeOK()` / `not.toBeOK()` matcher line: `expect(received).toBeOK()`
+ * (1.52), `expect(response).toBeOK() failed` (1.64), with `Error: ` unless a
+ * custom message comes first. A whole line, so a user message that only
+ * mentions `toBeOK()` in its text does not match.
+ */
+const TO_BE_OK = /^[ \t]*(?:Error: )?expect\((?:received|response)\)(?:\.not)?\.toBeOK\(\)(?: failed)?[ \t]*$/m;
+/**
+ * The response line of `toBeOK()`'s call log: `← 503 Service Unavailable`,
+ * one per response when there are redirects. The status is nowhere else in
+ * the message.
+ */
+const RESPONSE_LINE = /^[ \t]*← [1-5]\d\d\b.*$/gm;
+/**
+ * For a `toBeOK()` failure, the last response line of the call log (the
+ * final response, after any redirect); null otherwise. The request line,
+ * headers and body are left out: the same status on two endpoints is one
+ * failure.
+ */
+function toBeOkResponse(head, callLog) {
+    if (!TO_BE_OK.test(head))
+        return null;
+    return callLog.match(RESPONSE_LINE)?.at(-1)?.trim() ?? null;
+}
 /**
  * A strict mode violation lists the matching elements, one per line:
  * `    1) <li>Order 13</li> aka locator('li').first()`. Their positions and
@@ -89,7 +113,8 @@ const REPLACEMENTS = [
 const STATUS_OR_NUMBER = new RegExp(`(${httpStatusSource(STATUS_CODE)})|\\d+(?:\\.\\d+)?`, 'gi');
 /**
  * Normalizes an error message so the same failure produces the same text
- * across tests, retries and runs: ANSI codes and the call log are removed,
+ * across tests, retries and runs: ANSI codes and the call log are removed
+ * (except the final response status of a `toBeOK()` failure),
  * only the stable part of Playwright messages is kept, volatile values are
  * replaced with placeholders and whitespace is collapsed.
  */
@@ -98,9 +123,12 @@ export function normalizeErrorMessage(message, options = {}) {
     if (options.runError)
         text = stripMergeTagPrefix(text);
     const callLog = CALL_LOG.exec(text);
+    const response = callLog ? toBeOkResponse(text.slice(0, callLog.index), text.slice(callLog.index)) : null;
     if (callLog)
         text = text.slice(0, callLog.index);
     text = stablePart(text);
+    if (response)
+        text = `${text}\n${response}`;
     for (const [pattern, placeholder] of REPLACEMENTS) {
         text = text.replace(pattern, placeholder);
     }