@andwhy/e2e-pipeline 0.13.2 → 0.14.1

package/plugin/skills/playwright-test-pipeline/SKILL.md

@@ -0,0 +1,547 @@
+---
+name: playwright-test-pipeline
+description: Guides the full workflow for creating and maintaining Playwright
+  end-to-end tests with the planner/generator/healer agents — setup check,
+  Plan → Generate → Heal, local runs, Checkly preview validation, and
+  deployment. Use when the user wants to create, extend, or fix e2e/Playwright
+  tests, asks to "test" a page/feature/flow, or asks how the test pipeline
+  works.
+metadata:
+  version: "0.14.1"
+---
+
+# Playwright Test Pipeline
+
+You are guiding a person through this project's test pipeline. They may not
+know it exists — narrate briefly at each phase, and pause at the marked
+decision points. The delegation rules, the project test rules, and the exact
+prompt templates for the agents are all in this skill; follow them, do not
+improvise your own prompts to the agents.
+
+Setup and workflow docs (for humans): https://app.notion.com/p/31ff837b6270805d8922c4a7c7beb6b0
+
+## Delegation rules (always)
+
+- When the user says "Use the playwright-test-planner/generator/healer agent",
+  or the pipeline reaches a step owned by one of them, delegate via the
+  **Agent** tool with the matching `subagent_type`. The agents ship with the
+  `e2e-pipeline` plugin and register under their namespaced names:
+  `e2e-pipeline:playwright-test-planner`, `e2e-pipeline:playwright-test-generator`,
+  `e2e-pipeline:playwright-test-healer` (the plain name also resolves when it
+  is unique in the session).
+- **Never call the `playwright-test` MCP tools directly** from the
+  orchestrating session. They belong to the agents.
+- **Do not implement Playwright specs in the orchestrating turn.** Writing or
+  editing `*.spec.js` / `*.spec.ts` test bodies yourself bypasses replay, the
+  generator's MCP tools, and the project rules. Generation goes through the
+  generator; failing tests go through the healer.
+- **Plans:** read `tests/e2e/<feature>/<name>.plan.md` yourself and pass its
+  contents in the Agent prompt. **Never ask the user to paste the plan** into
+  chat.
+
+| Trigger phrase in user prompt                  | `subagent_type`                          | Purpose                                                                                                                             |
+| ---------------------------------------------- | ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
+| "Use the playwright-test-planner agent to …"   | `e2e-pipeline:playwright-test-planner`   | Explores the live site, discovers real selectors/DOM, writes a `tests/e2e/<name>.plan.md`                                           |
+| "Use the playwright-test-generator agent to …" | `e2e-pipeline:playwright-test-generator` | Reads a plan, replays steps in a browser, writes or extends `tests/e2e/<group>.spec.js` or `.spec.ts` (**Project test rules** §7–8) |
+| "Use the playwright-test-healer agent to …"    | `e2e-pipeline:playwright-test-healer`    | Runs tests, debugs failures, fixes code, re-runs until green                                                                        |
+
+## Phase 0 — Setup (always first)
+
+Verify, in order:
+
+1. The three agents are available in this session. A repo is in one of two
+   states, and both are fine:
+   - **Plugin:** the agents are listed for the Agent tool as
+     `e2e-pipeline:playwright-test-planner`,
+     `e2e-pipeline:playwright-test-generator` and
+     `e2e-pipeline:playwright-test-healer`. They come from the `e2e-pipeline`
+     plugin, which the repo enables through `.claude/settings.json`
+     (`enabledPlugins` → `e2e-pipeline@andwhy`).
+   - **Original flow (repo scaffolded before 0.14.0):** the agents are files
+     under `.claude/agents/playwright-test-*.md` and are listed by their plain
+     names `playwright-test-planner`, `playwright-test-generator`,
+     `playwright-test-healer`. Use those names for `subagent_type` wherever
+     this skill writes the `e2e-pipeline:` form. Do not suggest migrating to
+     the plugin unless the user asks; the repo works as it is.
+2. `.mcp.json` at project root defines the `playwright-test` server, pointed
+   at `playwright.mcp.config.{ts,mjs}` (which must exist — it lets the MCP run
+   seeds that `testIgnore` hides from normal runs)
+3. A `playwright.config.{ts,mjs}` and `checkly.config.{ts,mjs}` exist
+
+### If the agents are missing but the repo is otherwise set up
+
+Check `.claude/settings.json` for the `e2e-pipeline@andwhy` entry.
+
+- **Entry present, agents absent:** the plugin has not loaded in this
+  session. On a developer machine that means the session started before the
+  settings were committed or the folder was trusted — the plugin installs at
+  the next session start, so tell the user to restart and re-ask. In a
+  claude.ai/code cloud session the repo's plugin settings are not loaded at
+  all; say so plainly and stop — do not run the scaffold again, it will not
+  help, and do not improvise (see the hard stop below).
+- **Entry absent, repo copies present** (`.claude/agents/playwright-test-*.md`
+  exist): original flow, nothing is missing — continue with the plain agent
+  names. Migrating to the plugin is the user's decision; when they ask, the
+  scaffold with `--force` as below writes the plugin entry and removes the
+  copies, which would otherwise shadow the plugin's agents.
+- **Entry absent, no repo copies:** the pipeline is not installed here. Run
+  the scaffold as below.
+
+### If anything is missing, install it — do not just bounce the user
+
+Say what is missing, then offer to run the scaffold. Do not run it without
+agreement: it installs dependencies and writes files.
+
+**Always pass `--format` explicitly.** Without it the CLI opens an interactive
+prompt, and in a non-interactive shell that prompt resolves empty and the CLI
+exits silently having done nothing — it looks like it worked. Pick the format
+from the repo: `ts` when there is a `tsconfig.json`, otherwise `mjs`.
+
+```bash
+pnpm dlx @andwhy/e2e-pipeline init --format ts    # or --format mjs
+```
+
+Add `--force` only when updating an existing install. It regenerates the
+package-owned files (`.mcp.json`, `playwright.mcp.config.*`), removes
+pre-plugin copies of the agents, rules and skill from `.claude/`, and never
+touches the configs or the workflows. Add `--update-workflows` as well when
+the repo still carries the full-length workflows from before 0.14.0: it
+replaces them with the short callers of the pipeline's reusable workflows
+and removes the prompt and script copies only those old workflows read.
+
+Then read the output back to the user: what was created, what was skipped,
+what was removed, and any warning it printed (it flags workflows from
+before 0.14.0 and names the flag that replaces them; have the user review
+that diff, since a repo customization becomes a `with:` input in the
+caller).
+Docs: https://app.notion.com/p/31ff837b6270805d8922c4a7c7beb6b0
+
+**After a mid-session install, a restart is required — this is a hard stop.**
+Claude Code installs plugins and registers subagents and `.mcp.json` servers
+at session start, so the planner/generator/healer and the `playwright-test`
+browser tools do not exist in the session that just scaffolded them. Do NOT
+work around this: no inlining an agent definition into a general-purpose
+agent, no hand-writing plans or specs from source code — without the browser
+tools that produces unverified selectors, which is exactly what this pipeline
+exists to prevent. Finish Phase 0 (configs, reachable URL), then tell the
+user to restart the session and re-ask; Phase 0 will pass and planning starts
+for real.
+
+### Then finish the configuration before planning
+
+The scaffold writes templates, not a working setup. Walk these with the user:
+
+1. `playwright.config` — `baseURL` for local runs.
+2. `checkly.config` — real `projectName` and `logicalId`, the production
+   domain(s) in `environmentVariables`, `testCommand` (see Phase 5), and the
+   check-suite `name` — that is what teammates see in the Checkly dashboard
+   and in alerts. **Ask the user what to call it** unless the repo makes it
+   obvious (a single clear brand or product name); do not silently invent
+   one. Also set
+   `installCommand: "pnpm install --ignore-scripts"` — Checkly's runner gets a
+   bundle of test files, not the repo, so a project postinstall (codegen etc.)
+   fails there on missing files. Placeholder values (`my-app`, `example.com`)
+   left in place will block Phase 4; local work is unaffected.
+3. A reachable URL for the planner: a running dev server or a deployed
+   environment. **Confirm this before Phase 1** — the planner explores a live
+   page, and pointing it at an empty or unreachable app produces a plan that
+   asserts nothing useful.
+4. **Repository secrets for the workflows**: if the scaffold just created
+   `.github/workflows/`, tell the user now — not at merge time — to add the
+   repository secrets (GitHub → Settings → Secrets and variables → Actions).
+   You cannot add them; a person must. Without them the first PR push and the
+   first merge fail CI in ways that look like pipeline bugs.
+   - `CHECKLY_API_KEY` — 1Password ("Checkly") first; else Checkly → User
+     Settings → API keys. The key is shown once, at creation — tell the user
+     to save it to 1Password immediately.
+   - `CHECKLY_ACCOUNT_ID` — Checkly → Account Settings → General; also in
+     1Password.
+   - `VERCEL_AUTOMATION_BYPASS_SECRET` — Vercel → project → Settings →
+     Deployment Protection → Protection Bypass for Automation. Per project —
+     never copied from another repo.
+   - `SLACK_WEBHOOK_URL` — the **org-shared** e2e-channel webhook: paste the
+     value from 1Password ("e2e Slack webhook"). Same value in every repo;
+     do not walk the user through creating a Slack app or webhook — routing
+     to the project channel happens via a Slack rule in Phase 5.
+   - `ANTHROPIC_API_KEY` usually needs **no action**: it is an
+     organization-level secret inherited by every repo — do not walk the
+     user through adding it up front. But it can fail to reach a repo
+     (org-secret scoping, or a stale stored value): if a heal run's agent
+     step fails to authenticate, have the user add a repository-level
+     `ANTHROPIC_API_KEY` — a repo secret with the same name overrides the
+     organization one.
+5. **Existing tests under `tests/`** (unit, a11y — the scaffold lists them as
+   "outside tests/e2e"): leave `testDir` at `./tests/e2e` and give that suite
+   its own Playwright project with its own `testDir` if it should run under
+   Playwright at all. **Never widen `testDir` to `./tests`** — that runs the
+   other suite in every browser project and mixes the suites. Checkly is
+   pinned to `tests/e2e` by `testCommand` either way.
+
+## Project test rules
+
+These rules apply to **all** agents (generator, healer, and orchestrator).
+Include them verbatim in every Agent prompt to the generator or healer.
+
+1. Every logical step MUST have a numbered comment above it
+   (e.g. `// 1. Navigate to the homepage`). Preserve existing step comments
+   when editing. Add them if missing.
+
+2. Never hardcode domains. Use `page.goto("/")` and derive the origin from the
+   live page URL.
+
+3. Every `test.describe` block MUST include a `beforeEach` that runs shared preview setup
+   from `tests/e2e/helpers/preview-consent-setup.js` or `.ts` (`prepareVercelPreviewPage`, and when tests
+   start after consent, `gotoPathAndAcceptCookies(page, baseURL, path)` with the correct
+   relative path). The scaffold ships `prepareVercelPreviewPage` — it exchanges
+   `VERCEL_AUTOMATION_BYPASS_SECRET` for Vercel's deployment-protection bypass cookie on the
+   first navigation and is a plain `goto` when the variable is unset. Extend the file with
+   project consent logic; do not reinvent it. The helper file extension must match rule 7.
+
+4. Never hardcode a single locale's URL pattern. Use a locale config map keyed
+   by bare hostname and resolve at runtime. Preview hosts (`*.vercel.app`) are
+   never in the map and must fall back to the primary production domain's
+   config (overridable via `PREVIEW_LOCALE_HOST`) — an exact-match-only lookup
+   skips the whole suite on every PR run.
+
+5. After all tests pass locally, verify with
+   `BASE_URL=<dev-url> pnpm playwright test` via the terminal tool. If failures
+   occur, fix and re-run until green.
+
+   **Local runs always pin `BASE_URL` on the command line** to the dev server
+   confirmed during setup (usually `http://localhost:3000`) — never rely on the
+   ambient environment. Repos commonly export `BASE_URL` for the app's own
+   purposes, pointed at the deployed site; an unpinned run silently targets
+   production, which does not contain the working tree's changes (the feature
+   under test, any `data-testid` additions). Deployed URLs enter the pipeline
+   only at preview validation and Checkly, always passed explicitly.
+
+6. **Test root:** Paths in this skill assume the default `tests/e2e/`, which keeps e2e specs
+   clear of unit tests in the same repo. **The repo wins over this default.** Check
+   `playwright.config.{ts,mjs}` for `testDir` and list existing `*.spec.*`; if the **e2e**
+   specs already live somewhere else (`tests/` flat, `e2e/`, `src/__tests__/`), write there
+   instead and read every `tests/e2e/…` path below as that directory. Never split a suite
+   across two roots. A non-e2e suite under `tests/` (unit, a11y) is **not** that signal:
+   leave `testDir` on the e2e root and give that suite its own Playwright project with its
+   own `testDir` — never widen `testDir` to cover both, which runs the other suite in every
+   browser project.
+
+7. **Spec language (generators only):** Repositories differ — some teams use **JavaScript**
+   for Playwright tests (`.spec.js`), others use **TypeScript** (`.spec.ts`). List
+   `tests/e2e/**/*.spec.*` and match the prevailing extension. If there are no specs yet, default
+   to `.spec.js` unless the project README or the user specifies TypeScript for tests. Helpers
+   and seeds must use the same language as specs. Do not add `.spec.ts` in a repo that already
+   uses `.spec.js` everywhere (and vice versa).
+
+8. **Feature groups (target layout):** Prefer **one spec file per feature group** (e.g.
+   `tests/e2e/map/map.spec.js`) with a single `test.describe('<Group name>', …)` and **multiple**
+   `test()` blocks — one per plan scenario — sharing one `beforeEach`. Do not default to one
+   file per scenario unless scenarios are unrelated. Prefer **lean** tests: assert user-visible
+   outcomes and regressions, not every intermediate DOM detail from the plan. Numbered
+   comments on **main** user steps only.
+
+## Phase 1 — Plan
+
+Trigger when: user wants to create tests for a new feature, page, or user flow.
+
+1. Ask, if not already clear: which page/flow, which URL, and whether the
+   suite targets multiple domains (check `checkly.config` for the list).
+2. Use the **Agent** tool with `subagent_type: "e2e-pipeline:playwright-test-planner"`,
+   passing the URL and user flow description in the `prompt` (template below).
+3. The planner saves output to `tests/e2e/<feature>/<name>.plan.md`.
+4. **Decision point:** present the saved plan scenario-by-scenario and ask
+   what to keep, drop, or adjust. Do not generate before the user responds.
+
+Prompt to forward:
+
+```
+Create a test plan for: <FEATURE NAME>
+
+Navigate to <URL> and explore the following user flow:
+<USER'S FLOW STEPS>
+
+While exploring, verify:
+- Actual selectors (see **Locator strategy** under *Robustness patterns* below)
+- Whether cookie consent or login gates appear
+- DOM structure of key interactive elements
+- If the suite targets **multiple domains**, explore **each** domain the tests will run against
+  (see *Multi-domain / multi-locale testing* — the plan must reflect every distinct host or market)
+
+Generate a seed file at tests/e2e/<feature>/seed.spec.js or seed.spec.ts (same extension as
+the rest of this repo's specs — see Project test rules §7) that handles the common setup
+(navigation, cookie/auth consent, waiting for key elements) discovered during exploration.
+
+Save the plan to tests/e2e/<feature>/<name>.plan.md using planner_save_plan.
+All URLs must use relative paths — never localhost or absolute domains.
+```
+
+## Phase 2 — Generate
+
+Trigger when: a `tests/e2e/<name>.plan.md` exists and the user wants generated test code.
+
+**You (the orchestrating agent) own this step end-to-end.** The human does **not** need to
+invoke the generator separately for each scenario or plan section — you chain **Agent** calls
+to `playwright-test-generator` until the feature group is implemented (see **Important rules**).
+
+1. Read `tests/e2e/<feature>/<name>.plan.md` yourself to get the full plan content; never ask the user to paste it.
+2. Use the **Agent** tool with `subagent_type: "e2e-pipeline:playwright-test-generator"` — **once per scenario**
+   if building incrementally, **or** pass the **full plan** and all scenarios for the group in one
+   prompt if the generator can emit the whole file (**Project test rules** §8)
+3. Pass the **plan content** in the `prompt` (the generator subagent cannot read arbitrary paths from disk)
+4. Include the **project test rules** above in the prompt
+5. If the plan contains a "Recommended data-testid additions" table, include it verbatim — the generator adds those attributes before replaying steps
+6. Target **one spec file per feature group** (e.g. `tests/e2e/map/map.spec.js`) with a shared
+   `test.describe` and **multiple** `test()` blocks — merge with `edit` between runs if needed
+7. Tell the user which spec file is being written and when each scenario lands.
+
+Prompt to forward (example — adjust per repo):
+
+```
+Implement tests from the plan for this feature group. Target layout: one file, one describe, multiple tests (see project test rules §8).
+
+- Group / describe: "<TOP-LEVEL PLAN HEADING e.g. Map>"
+- File: tests/e2e/<feature>/<group>.spec.js (or `.spec.ts` per project)
+- Seed: <SEED FILE PATH FROM PLAN>
+- Viewport: <WIDTH>x<HEIGHT>
+
+<plan>
+<PASTE full plan OR the scenario block(s) you are implementing this run>
+</plan>
+
+If recommended data-testid additions exist, add them to source components before replaying steps.
+
+After the spec is written or updated, run
+BASE_URL=<dev-url> pnpm playwright test tests/e2e/<feature>/ (pin the dev server per rule 5).
+
+## Project test rules (follow these strictly)
+
+<COPY the Project test rules section above into this prompt>
+```
+
+## Phase 3 — Run and heal
+
+1. Run `BASE_URL=<dev-url> pnpm playwright test tests/e2e/<feature>/`, pinning
+   the dev server confirmed in Phase 0. Phases 1–3 always target local: the
+   working tree's changes (the feature, any `data-testid` additions) exist
+   nowhere else, and an ambient `BASE_URL` export would silently point the run
+   at the deployed site. Deployed URLs enter only at Phase 4, always explicit.
+2. Green → Phase 4. Red → use the **Agent** tool with
+   `subagent_type: "e2e-pipeline:playwright-test-healer"`, passing the test file path and
+   the project test rules (template below). If the healer marks anything
+   `test.fixme()`, surface it explicitly — the user decides whether that is
+   acceptable.
+
+Prompt to forward (always include the rules block):
+
+```
+Run and fix the tests in tests/e2e/<name>.spec.js (or `.ts` per project)
+
+If any tests fail:
+
+- Debug each failure
+- Examine the page state, selectors, and assertions
+- Fix the test code
+- Re-run until all tests pass locally
+
+## Project test rules (follow these strictly)
+
+<COPY the Project test rules section above into this prompt>
+```
+
+## Important rules
+
+- **Always delegate via the Agent tool.** Never call `playwright-test/*` tools directly.
+- **Always plan before generating.** Use the planner to discover real selectors first.
+- **Always read the plan file** before passing to the generator. The generator runs in
+  isolation — include the plan or scenario steps in the prompt.
+- **You implement the full generate phase.** After plan approval, **you** invoke the
+  generator subagent **as many times as needed** (typically once per scenario, sequentially)
+  or supply the full multi-scenario prompt — **without** asking the user to run the generator
+  for each plan section. A single user request such as "Implement the tests from this plan" is enough.
+- **Run generator subagent calls sequentially.** Never parallel — the MCP browser server is shared.
+- **Do not interrupt.** Sending a message while a generator is running aborts it.
+  Wait for completion before proceeding.
+- **data-testid additions are owned by the generator.** When the plan includes a
+  "Recommended data-testid additions" table, the generator subagent adds those
+  attributes to source components before replaying steps. The orchestrator must
+  forward the table verbatim in the generator prompt and must NOT edit source
+  components itself.
+- **Present the plan for review** after planning. Unless the user narrows scope, **proceed
+  to generate all scenarios** for the agreed feature group (see **Project test rules** §8).
+- **Generator and healer self-verify locally** with
+  `BASE_URL=<dev-url> pnpm playwright test` (pinned per **Project test rules** §5).
+  Cloud validation via `checkly pw-test` is a separate human step after pushing.
+  Only invoke the healer if a generated test fails locally.
+- **Base URL**: local runs pin `BASE_URL=<dev-url>` on the command (**Project
+  test rules** §5); the config's localhost fallback is a safety net, not a
+  guarantee, because an exported `BASE_URL` overrides it silently.
+- **Plan files** are saved at `tests/e2e/<feature>/<name>.plan.md`.
+- **Test files** follow **Project test rules** §7–8 — typically `tests/e2e/<feature>/<group>.spec.js`
+  or `.spec.ts` with one `describe` and multiple tests.
+
+## Phase 4 — Preview validation (before merge)
+
+1. Auth check (first authenticated command in the pipeline): run
+   `pnpm checkly whoami`. If not logged in, have the user run
+   `pnpm checkly login` — it opens a browser flow, and the shared team
+   credentials are in 1Password (search "Checkly"). Local Playwright runs
+   never need this, and CI does not either — it uses the `CHECKLY_API_KEY` /
+   `CHECKLY_ACCOUNT_ID` repository secrets.
+2. **Decision point:** ask for the preview deployment URL, and settle how
+   Checkly's runners get through Vercel deployment protection. Preferred: the
+   project's automation bypass secret (Vercel project settings → Deployment
+   Protection; value in 1Password) — the scaffolded helper exchanges it for
+   Vercel's bypass cookie on first navigation. Fallback when the project has
+   no secret: a share link ("Anyone with the link") as the preview URL.
+3. Run: `pnpm checkly pw-test --env BASE_URL="<preview-url>" --env VERCEL_AUTOMATION_BYPASS_SECRET="<secret>" -- tests/e2e/<feature>/`
+   (omit the second `--env` on the share-link fallback). Never write the
+   secret into `checkly.config` — deployed monitors run against production
+   and do not need it.
+4. Failures here but not locally are usually cloud-environment issues
+   (timeouts, consent gates) — heal with that framing, then re-run.
+
+The user can skip this phase; note that cloud behavior then goes unverified
+until after merge.
+
+## Phase 5 — Merge and deployment
+
+The scaffold ships three workflows (seeded once — existing ones are never
+overwritten, except by an explicit `init --update-workflows`). Each is a
+short caller of the matching reusable workflow in the pipeline repository,
+so pipeline releases update the behavior without a change in the repo:
+
+- `.github/workflows/pr-playwright-tests.yml` — runs the suite against the
+  PR's own Vercel preview on every push (`deployment_status` trigger, so the
+  preview is already live and contains the PR's changes), and comments the
+  result on the PR.
+- `.github/workflows/checkly-deploy.yml` — when a **production deployment
+  succeeds** (`deployment_status` trigger), deploys the checks as Checkly
+  monitors. Not on merge: monitors must match what production serves, and a
+  merge that adds tests for new `data-testid`s would go red until the release
+  carrying them is live — with gated releases that gap can be days. A
+  `workflow_dispatch` fallback allows manual runs.
+- `.github/workflows/checkly-heal.yml` — a failing deployed monitor
+  dispatches here (via the webhook alert channel below). An agent triages
+  first: a production regression or anything needing a behavioral app change
+  alerts Slack immediately; a stale test gets fixed and a flaky test gets
+  hardened — including behavior-neutral test hooks (`data-testid`) in app
+  source when the robust fix needs them — and opened as a PR that a human
+  must review. Test-only fixes are verified against production and on
+  Checkly's infrastructure; app-side hooks are validated by the PR's own
+  preview run (pr-playwright-tests). Inert until the webhook exists.
+
+**Before the first merge, the repo needs secrets** (values in 1Password):
+`CHECKLY_API_KEY` and `CHECKLY_ACCOUNT_ID` for the deploy and auto-heal,
+`VERCEL_AUTOMATION_BYPASS_SECRET` for the PR run through deployment
+protection, `SLACK_WEBHOOK_URL` for auto-heal notifications
+(`ANTHROPIC_API_KEY` is inherited from the organization — nothing to add
+unless a heal run fails to authenticate; then a repo-level secret of the
+same name overrides it, see Phase 0).
+If the scaffold just created the workflows, say so explicitly and list the
+secrets — a merge without them fails CI and looks like a pipeline bug.
+
+After merge, monitors update once the production deployment goes live — the
+user verifies in the Checkly dashboard (https://app.checklyhq.com/); nothing
+to run by hand. Manual fallback if a repo deleted the deploy workflow:
+`pnpm checkly test --record`, then `pnpm checkly deploy` (requires the
+Phase 4 login).
+
+**Alert notifications are not automatic — walk the user through the
+auto-heal webhook.** A failing monitor only shows in the Checkly dashboard
+until an alert channel exists. The alert channel for these suites is a
+**webhook into the auto-heal workflow**, not a direct Slack channel: the
+team's Slack message is written by the pipeline *after* triage (regression
+vs. stale test), so it carries a diagnosis and, for stale tests, the fix PR.
+Set up once per project:
+
+1. Confirm the `SLACK_WEBHOOK_URL` repository secret from Phase 0 exists —
+   the workflow escalates nothing without it. (`ANTHROPIC_API_KEY` comes
+   from the organization automatically — but if the dry run in step 6 ends
+   agent-failed with an authentication error, override it with a repo-level
+   secret of the same name, per Phase 0.)
+2. The shared `GH_DISPATCH_TOKEN` secret exists in Checkly and covers this
+   repo. This is **one-time organization setup, not yours to perform**: the
+   token (`checkly-heal-dispatch`, a fine-grained PAT) is created once and
+   maintained by its owner, named in 1Password — see "One-time organization
+   setup" in the usage guide
+   (https://app.notion.com/p/32df837b627080748719c0f94a3aeaaf). Your only
+   per-project concern: if this repo is not yet on the token's repository
+   list, the webhook channel's test (step 3) fails with a 404 — then ask
+   the token owner to widen it (a one-minute edit on their side; the token
+   value doesn't change). Do not walk the user through creating a PAT or a
+   Checkly secret.
+3. Checkly → Alert channels → "Add more channels" → **Webhook**:
+   - Method `POST`, URL
+     `https://api.github.com/repos/<org>/<repo>/dispatches`
+   - Headers: `Authorization: Bearer {{GH_DISPATCH_TOKEN}}` and
+     `Accept: application/vnd.github+json`
+   - Body (Checkly's editor lists the available variables — confirm the
+     names there):
+
+     ```json
+     {
+       "event_type": "checkly-alert",
+       "client_payload": {
+         "check_name": "{{CHECK_NAME}}",
+         "check_id": "{{CHECK_ID}}",
+         "result_id": "{{CHECK_RESULT_ID}}",
+         "result_link": "{{RESULT_LINK}}",
+         "alert_type": "{{ALERT_TYPE}}"
+       }
+     }
+     ```
+
+   - Notify on **failure**; enable **reminders** while a check stays failing
+     — a reminder re-fires the webhook, the workflow dedupes repeats, and a
+     lost first delivery gets retried by the next one. Recovery/degraded
+     notifications are ignored by the workflow either way.
+4. Subscriptions: **only the current check**, and **uncheck** "Automatically
+   subscribe newly created checks to this alert channel".
+5. **Do not also subscribe a Slack alert channel to these suites.** The
+   pipeline is the single alert path; a parallel raw Checkly alert double-pings
+   the channel and trains people to ignore one of them.
+6. **One Slack Workflow Builder rule** routes this project's messages from
+   the shared e2e channel to the project channel. All notifications post to
+   the e2e channel tagged `[e2e:<repo-name>]`; the rule matches the **full
+   bracketed tag** — never a short name like `gbk`, which collides with
+   other text — and sends the message to the project channel. The sent
+   message's template is just the message-text variable (plus the
+   link-to-original variable when available, so threads happen on the
+   original) — no static prefix text. Have the user add a **second
+   manager** to the rule and record it in the rule registry in the e2e
+   channel's canvas.
+7. Dry run: GitHub → Actions → "Checkly auto-heal" → Run workflow, with the
+   suite's check name. Confirm the run completes and the message shows up in
+   the e2e channel and, via the rule, in the project channel (a passing
+   check should end as "not reproducible" — summary only, no message — so a
+   real dry run is easiest right after any genuine failure).
+
+What the user should expect afterwards: a failing monitor produces either a
+`checkly-heal` PR plus a Slack note (stale or flaky test — **review
+required, the PR is never merged automatically**; PRs carrying app-side test
+hooks say so and point at their preview run for validation), or a Slack
+escalation (regression, fix needs a behavioral app change, agent failed). A
+transient blip that doesn't reproduce stays silent by design.
+
+## Checkly auto-heal (CI, not this session)
+
+Failing deployed monitors are handled by `.github/workflows/checkly-heal.yml`:
+it triages first (production regression → Slack alert; stale test → fix; flaky
+test → harden), may add behavior-neutral test hooks (`data-testid`) to app
+source when the robust fix needs them, and opens a `checkly-heal` PR a human
+must review — app-side hooks are validated by the PR's own preview run. The
+repo's workflow file is a short caller; the triage logic, the agent prompt
+and the healer methodology come from the pipeline's reusable workflow and
+the published package, not from this repo or this plugin. If a user asks
+about a `checkly-heal` PR, that workflow authored it; reviewing one means
+verifying the updated test still describes correct production behavior, not
+that it merely passes.
+
+## Conventions to uphold
+
+The generator and healer enforce spec-level rules themselves. At the pipeline
+level, additionally keep to:
+
+- One spec file per feature group; one `describe`, multiple `test()` blocks
+- Commit specs, plans, seeds, **and** any `data-testid` source edits together
+- Never commit with locally failing tests — heal first or `test.fixme()` with
+  the user's sign-off

package/templates/github/prompts/checkly-heal.md

@@ -13,9 +13,16 @@ team. You never push, never open PRs, never send notifications.
 - `.checkly-heal/result.json` — the raw Checkly check result when the API call
   succeeded: error message, failing test and step, stack trace. May be absent;
   proceed without it if so.
-- `.claude/agents/playwright-test-healer.md` — debugging practice and
-  robustness rules. Follow its methodology. The overrides below take
-  precedence over anything it says.
+- `.checkly-heal/healer.md` — the `playwright-test-healer` agent definition:
+  debugging practice and robustness rules. The workflow copies it here from
+  the published pipeline package. Follow its methodology; ignore its
+  frontmatter (`tools:` names browser tools this run does not have — use
+  `pnpm playwright test` and the trace output instead). The overrides below
+  take precedence over anything it says. If the file is absent (a workflow
+  from before 0.14.0 that never wrote it), apply the same principles from
+  your own knowledge: resilient locators, root causes over timeouts, minimal
+  edits — and say in the outcome details that the methodology file was
+  missing.
 
 ## Overrides for this run
 

package/templates/github/workflows/checkly-deploy.yml

@@ -1,14 +1,16 @@
 name: Deploy Checkly checks
 
-# Deploys the e2e suite as Checkly monitors when PRODUCTION deploys — not on
-# merge. Monitors must match what production serves: a merge that adds tests
-# for new data-testids goes red until the app deploy carrying those testids is
-# live, and with gated releases that gap can be days. `deployment_status`
-# fires when Vercel finishes the production deployment, so monitors update at
-# the exact moment the code they were recorded against is reachable.
-# `checkly deploy` is idempotent, so redeploying on every production deploy
-# is harmless.
-# Needs repo secrets: CHECKLY_API_KEY, CHECKLY_ACCOUNT_ID (1Password, "Checkly").
+# Deploys the e2e suite as Checkly monitors when a PRODUCTION deployment
+# succeeds (not on merge — monitors must match what production serves).
+# The work happens in the pipeline's reusable workflow, so a pipeline
+# release updates it without a change here. This file owns the triggers.
+# Needs repo secrets CHECKLY_API_KEY, CHECKLY_ACCOUNT_ID (1Password, "Checkly").
+# Alternative trigger, if the repo prefers deploying on merge: replace
+# deployment_status with
+#   push:
+#     branches: [main]
+#     paths: ["tests/e2e/**", "playwright.config.*", "checkly.config.*"]
+# The reusable workflow deploys on any non-deployment_status event.
 on:
   deployment_status:
   workflow_dispatch:
@@ -19,46 +21,11 @@ concurrency:
 
 jobs:
   deploy:
-    if: >-
-      github.event_name == 'workflow_dispatch' ||
-      (github.event.deployment_status.state == 'success' &&
-       github.event.deployment.environment == 'Production')
-    runs-on: ubuntu-latest
-    timeout-minutes: 8
-
-    steps:
-      - name: Check out the deployed ref
-        uses: actions/checkout@v5
-        with:
-          ref: ${{ github.event.deployment.ref || github.ref }}
-
-      - name: Set up pnpm
-        uses: pnpm/action-setup@v4
-
-      - name: Set up Node
-        uses: actions/setup-node@v5
-        with:
-          node-version: 22
-          cache: pnpm
-
-      # --ignore-scripts: only the tests are bundled; project postinstall
-      # hooks are not needed and may want CMS tokens CI does not have.
-      - name: Install dependencies
-        run: pnpm install --frozen-lockfile --ignore-scripts
-
-      # --force skips the interactive confirmation prompt.
-      - name: Deploy checks to Checkly
-        env:
-          CHECKLY_API_KEY: ${{ secrets.CHECKLY_API_KEY }}
-          CHECKLY_ACCOUNT_ID: ${{ secrets.CHECKLY_ACCOUNT_ID }}
-        run: pnpm exec checkly deploy --force
-
-      - name: Write result to job summary
-        if: always()
-        run: |
-          {
-            echo "## Checkly deploy"
-            echo ""
-            echo "- Ref: \`${{ github.event.deployment.ref || github.ref }}\`"
-            echo "- Triggered by a successful production deployment"
-          } >> "$GITHUB_STEP_SUMMARY"
+    uses: why-agency/why-e2e-pipeline/.github/workflows/reusable-checkly-deploy.yml@main
+    # Per-repo settings, defaults shown. Uncomment to change.
+    # with:
+    #   node_version: "22"
+    #   pnpm_version: ""
+    #   install_command: pnpm install --frozen-lockfile --ignore-scripts
+    #   validate_before_deploy: false  # `checkly test` before deploying
+    secrets: inherit