@piwitests/reporter 0.20.0 → 0.22.0

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@piwitests/reporter",
-  "version": "0.20.0",
+  "version": "0.22.0",
   "description": "Playwright reporter that streams results, traces and HTML reports to a Piwi Dashboard instance",
   "url": "https://github.com/PiwiTests/platform",
   "homepage": "https://piwitests.github.io",
@@ -55,7 +55,8 @@
     "prepublishOnly": "npm run reporter:build"
   },
   "files": [
-    "dist/"
+    "dist/",
+    "templates/"
   ],
   "peerDependencies": {
     "@playwright/test": "^1.61.1"
@@ -65,6 +66,7 @@
   },
   "devDependencies": {
     "@piwitests/core": "*",
+    "@piwitests/picker-dom": "*",
     "tsup": "^8.5.0"
   }
 }

package/templates/skills/apply-locator-healing/SKILL.md

@@ -0,0 +1,32 @@
+---
+name: apply-locator-healing
+description: Replace brittle Playwright locators with the healed selector Piwi suggests after a failing run, then re-run to confirm. Use when a test fails because a selector no longer matches, when the user asks to "fix the broken locator", "apply Piwi's suggestion", or "heal the selector".
+---
+
+# Apply Piwi's locator healing
+
+When a locator stops matching, Piwi has already computed a ranked replacement from the locator snapshots it captured on passing runs. This skill applies that suggestion to the real spec file and verifies it — closing the last mile Piwi cannot do on its own (edit your code).
+
+## How you reach Piwi
+
+Prefer the **Piwi MCP server** if it is connected (`get_locator_healing`, `explain_failure`, `get_run`). Otherwise open the failing case in the dashboard — the healing panel shows the suggested locator and its call site — and work from that plus the repo. The reporter also attaches a `piwi-locator-suggestion` annotation to the failing test in the Playwright report.
+
+## Steps
+
+1. **Find the failing case.** From a run the user names, or the latest failed run (`list_recent_activity` → `get_run` with a failed filter). Identify the case whose failure is a locator that matched nothing (a timeout on `click` / `fill` / `expect(locator)`).
+
+2. **Get the healed locator.** Call `get_locator_healing` for that case. It returns the **recommended durable locator** plus ranked alternatives, each stamped with the **call site** (file and line) where the original locator was used. Read the call site — that is the exact spot to edit.
+
+3. **Confirm it's really a locator problem.** The suggestion is only right if the element still exists under a new selector. Skim the error and the ARIA snapshot (`get_test_run_case` / `explain_failure`): if the element is genuinely gone or the page errored before rendering, this is not a healing case — hand it to the `investigate-failure` skill instead.
+
+4. **Edit the spec.** At the call site, replace the brittle locator with the recommended one. Prefer role/label/test-id selectors (what Piwi ranks highest) over CSS/XPath. Keep the change minimal and in the user's existing style. If the same brittle selector appears elsewhere (a page object, a helper), update those too.
+
+5. **Verify.** Re-run just that spec: `npx playwright test <file>`. It should pass. If Piwi is set up to report, confirm the new run is green for that case; if it still fails, try the next-ranked alternative from step 2 before broadening the search.
+
+6. **Report.** Show the before/after locator, the file and line, and the verifying result. If several tests shared the selector, list every place you changed.
+
+## Guardrails
+
+- Change the **locator**, not the assertion's intent — if a test checked for "Sign in" and the button is now "Log in", that is a copy change to confirm with the user, not a locator to heal.
+- Never paste a suggested locator without reading its call site and the surrounding test; apply it where the original was used.
+- One verified green re-run per healed selector. Don't batch-replace across many files without running the affected specs.

package/templates/skills/investigate-failure/SKILL.md

@@ -0,0 +1,36 @@
+---
+name: investigate-failure
+description: Investigate a failed test run recorded in Piwi Dashboard and propose a fix grounded in its evidence — error, steps, console, network, locator suggestion, and the diff since the last green run. Use when the user asks "why did the last run fail", "what broke in CI", "diagnose this failure", or points at a Piwi run/cluster.
+---
+
+# Investigate a Piwi failure
+
+Turn a failed run in [Piwi Dashboard](https://piwitests.github.io) into a grounded diagnosis and a concrete fix. Piwi has already gathered the evidence — the error text, the steps that ran, console output, failing network calls, a suggested locator, and the source diff since the last passing run. Use that instead of guessing.
+
+## How you reach Piwi
+
+Prefer the **Piwi MCP server** if it is connected to this agent (tools named `list_recent_activity`, `get_run`, `explain_failure`, `get_cluster_context`, …). If it is not connected, tell the user they can connect it (the reporter CLI does not proxy MCP — see the dashboard's **MCP server** page) or paste the run URL / failure details, and work from those plus the repo.
+
+## Steps
+
+1. **Find the run.** If the user gave a run URL or id, use it. Otherwise call `list_recent_activity` (or `list_runs` for a specific project) and take the most recent failed run. Confirm with the user if several projects are in play.
+
+2. **Get the failures.** Call `get_run` with a failed-status filter to list the failing cases. For a fast single-call evidence bundle on one case, use `explain_failure` (error + steps + console + locator fix + diagnosis context).
+
+3. **Group by cause.** Call `get_failure_groups` (or `list_clusters` / `get_cluster`) — failures that share a root cause are clustered, so you fix one thing, not five. Work cluster by cluster.
+
+4. **Read the evidence per cluster.** Call `get_cluster_context` — the same SCM-grounded context the built-in diagnosis uses: representative errors, test steps, console logs, failing network requests, ARIA snapshots, and the **diff of files changed since the last green run**. If a diagnosis already exists, `get_cluster_diagnosis` returns its root cause and suggested fix.
+
+5. **Form the diagnosis.** Tie the failure to a cause with evidence: a selector that stopped matching, an assertion on changed copy, a slow/500 endpoint (`get_network_requests`), a race, a genuinely flaky test (check `get_test_stability_trend` — if it fails intermittently, treat it as flaky, not a regression). Point at the specific commit/file from the diff when the evidence supports it.
+
+6. **Propose the fix.** Make the smallest change that addresses the root cause, in the actual source or spec files. Prefer Piwi's own suggestions where they exist — for a broken selector, the ranked replacement from `get_locator_healing` (the `apply-locator-healing` skill does exactly this). Show the diff.
+
+7. **Verify.** Re-run the affected spec(s): `npx playwright test <file>`. Confirm they pass and the new run is green in the dashboard (`get_run_insights` compares against the last green run: regressions cleared, nothing new broken).
+
+8. **Close the loop (optional).** With packages/reporter/admin access, `set_cluster_status` marks the cluster resolved with a note so it drops off the triage queue.
+
+## Guardrails
+
+- Ground every claim in evidence you actually read — never invent a stack trace, a commit, or a line number. If the evidence is thin, say so and get the trace (`list_case_traces`) or ask.
+- Distinguish a **regression** (was passing, now failing — fix the cause) from a **flaky** test (intermittent — stabilize it) from an **environmental** failure (dashboard/CI/network). The fix differs for each.
+- Don't mark a cluster resolved until a run proves it. Report the diagnosis, the change, and the verifying run URL.

package/templates/skills/setup-piwi/SKILL.md

@@ -0,0 +1,62 @@
+---
+name: setup-piwi
+description: Wire a Playwright project up to a Piwi Dashboard — install the reporter, wrap the config, add capture fixtures, and confirm a run lands. Use when the user asks to "set up Piwi", "connect Playwright to the dashboard", "start reporting test results", or after they mention a Piwi/dashboard server URL.
+---
+
+# Set up Piwi in a Playwright project
+
+Connect a Playwright test suite to a [Piwi Dashboard](https://piwitests.github.io) so every run is uploaded, kept, and analyzed. The mechanical work is done by a deterministic command; your job is to gather the right inputs, run it, finish anything it flags, and prove a run reaches the dashboard.
+
+## Before you start
+
+Find out two things (ask the user only if you cannot infer them):
+
+- **Dashboard URL** — e.g. `http://localhost:3000` for a local server, or a deployed URL. Defaults to `http://localhost:3000`.
+- **Project name** — the label runs are grouped under. Defaults to the package or folder name.
+
+Confirm this is a Playwright project: there should be a `playwright.config.ts|js` and `@playwright/test` in `package.json`. If there is no Playwright at all, set that up first.
+
+## Do it
+
+Run the initializer from the project root. It installs `@piwitests/reporter`, wraps `defineConfig(...)` with `wrapConfig(...)`, creates `tests/fixtures.ts`, and records `PIWI_*` settings in `.env.example`:
+
+```bash
+npx @piwitests/reporter init --server-url <dashboard-url> --project <name> --json
+```
+
+- Preview first with `--dry-run` if you want to see the plan before anything is written.
+- `--json` prints a `steps[]` array. Read it. Each step has a `status`:
+  - `created` / `updated` / `already` — done, nothing more to do.
+  - `manual` — the tool would not edit that file safely. The `detail` field is the exact change to make; apply it yourself (see below).
+  - `error` — something failed (usually the install); the `detail` says what. Fix it and re-run — `init` is idempotent.
+
+### Finishing `manual` steps
+
+- **config** marked `manual`: the config is not a plain `export default defineConfig(...)`. Add `import { wrapConfig } from '@piwitests/reporter'` and wrap the exported config: `export default wrapConfig(defineConfig({ ... }), { serverUrl, projectName })`.
+- **fixtures** marked `manual`: a fixtures file already exists. Merge Piwi in: `import { piwiFixtures } from '@piwitests/reporter'` and compose them into the existing `base.extend(...)` (or use `extendPiwiFixtures(base)`).
+
+### Rewire the specs to the fixtures
+
+The capture fixtures (locator healing, slow-endpoint analysis, Web Vitals, console, failure-time ARIA) only apply to specs that import `test` from the fixtures file. In each spec that currently does:
+
+```ts
+import { test, expect } from '@playwright/test'
+```
+
+change it to import from the fixtures file instead, e.g. `import { test, expect } from './fixtures'` (adjust the relative path). A spec left on the direct import still runs and reports — it just is not captured.
+
+## Authentication
+
+If the dashboard has auth enabled, runs need an API key. Have the user create one in the dashboard (**Settings → Users → API keys**; keys start with `pd_`), then put it in `.env` as `PIWI_API_KEY=pd_...` and make sure `.env` is git-ignored. Never hardcode a key into `playwright.config.ts` or commit it. In CI, pass it as a secret (`PIWI_API_KEY`).
+
+## Verify it worked
+
+Setup is not done until a run reaches the dashboard.
+
+1. Run one spec: `npx playwright test` (or a single file to keep it quick).
+2. Confirm the run landed. Deterministically: set `PIWI_OUTPUT_FILE=piwi-run.json` when running, then read `runUrl`/`status` from that JSON file — its existence with a `runUrl` proves the upload. Otherwise look for the `View run: <url>` line the reporter prints, or open the dashboard and check the project's latest run.
+3. If nothing appears: verify the dashboard is reachable at the URL, that auth (if on) has a valid key, and that traces are enabled (`use: { trace: 'retain-on-failure' }`).
+
+## Wrap up
+
+Tell the user what changed (config, fixtures, `.env.example`), what they still need to do (API key if auth is on; rewire remaining specs), and the run URL you verified. If they use CI, offer to add the reporter's env there too.

package/templates/skills/stabilize-flaky-tests/SKILL.md

@@ -0,0 +1,36 @@
+---
+name: stabilize-flaky-tests
+description: Find the flakiest Playwright tests from Piwi's flaky analysis and fix the root cause of the intermittency, ranked by impact. Use when the user asks to "fix flaky tests", "reduce flakiness", "why is this test flaky", or wants to clean up an unreliable suite.
+---
+
+# Stabilize flaky tests with Piwi
+
+Piwi scores every test's flakiness over its whole run history and ranks it by impact, so you spend effort on the tests that actually cost the team — not whichever one failed most recently. This skill picks the worst offenders, fixes the *cause* of the intermittency, and confirms the score improves.
+
+## How you reach Piwi
+
+Prefer the **Piwi MCP server** if it is connected (`list_flaky_tests`, `get_test_stability_trend`, `get_test_case`, `get_spec_health`). Otherwise use the dashboard's **Flaky** tab for the project and work from what it shows plus the repo.
+
+## Steps
+
+1. **Rank the flaky tests.** Call `list_flaky_tests` for the project. It returns each test's flaky score, impact ranking, and a **root-cause category**. Take the top few by impact — do not try to fix the whole list at once.
+
+2. **Understand one test's pattern.** For a chosen test, call `get_test_stability_trend` (is it getting worse, or already stable again?) and `get_test_case` (recent pass/fail history). Read a representative failing execution (`get_test_run_case` / `explain_failure`) for the error, steps, console, and network at the moment it flaked.
+
+3. **Identify the cause.** Common categories and their fixes:
+   - **Timing / race** — replace fixed waits and manual sleeps with Playwright web-first assertions (`await expect(locator).toBeVisible()`), await the network/UI state the test depends on, not a timeout.
+   - **Locator instability** — a selector that resolves ambiguously or intermittently; heal it (see the `apply-locator-healing` skill) or scope it.
+   - **Test-order / shared state** — leaking storage, cookies, or a shared backend row; isolate setup, use a fresh context, or a unique fixture per test.
+   - **Slow / unreliable endpoint** — check `get_network_requests`; if a real endpoint is intermittently slow or 500s, that is a product bug, not a test bug — report it rather than papering over it with retries.
+
+4. **Fix the root cause.** Make the change in the spec (or the app, when the flake is a real defect). Do **not** "fix" flakiness by adding retries or increasing timeouts — that hides it; the goal is a test that passes deterministically.
+
+5. **Verify.** Re-run the test several times to shake out intermittency: `npx playwright test <file> --repeat-each=5` (raise the count for stubborn ones). It should pass every time. Over the next runs, confirm the flaky score falls in the dashboard (`get_test_stability_trend`).
+
+6. **Report.** For each test you touched: the root-cause category, the change, and the repeat-run result. List any remaining high-impact flaky tests so the user can decide whether to continue.
+
+## Guardrails
+
+- Fix the cause, never the symptom — no blanket `test.retry`, no bumped global timeout, no `waitForTimeout` sprinkles.
+- A test that flakes because a *real* endpoint is unreliable is a product finding; surface it instead of hiding it in the test.
+- Stabilize a few high-impact tests well rather than lightly touching many. Prove each one with repeated runs before moving on.