@andwhy/e2e-pipeline 0.10.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (31) hide show
  1. package/README.md +106 -0
  2. package/dist/bin/init.d.ts +2 -0
  3. package/dist/bin/init.js +128 -0
  4. package/dist/utils/copy.d.ts +23 -0
  5. package/dist/utils/copy.js +249 -0
  6. package/dist/utils/format.d.ts +1 -0
  7. package/dist/utils/format.js +49 -0
  8. package/dist/utils/install.d.ts +2 -0
  9. package/dist/utils/install.js +54 -0
  10. package/dist/utils/prompt.d.ts +2 -0
  11. package/dist/utils/prompt.js +17 -0
  12. package/package.json +37 -0
  13. package/templates/claude/agents/playwright-test-generator.md +184 -0
  14. package/templates/claude/agents/playwright-test-healer.md +70 -0
  15. package/templates/claude/agents/playwright-test-planner.md +135 -0
  16. package/templates/claude/partials/_robustness-core.md +69 -0
  17. package/templates/claude/rules/playwright-workflow.md +203 -0
  18. package/templates/claude/skills/playwright-test-pipeline/SKILL.md +201 -0
  19. package/templates/github/scripts/playwright-pr-comment.mjs +95 -0
  20. package/templates/github/workflows/checkly-deploy.yml +64 -0
  21. package/templates/github/workflows/pr-playwright-tests.yml +154 -0
  22. package/templates/shared/checkly.config.template.mjs +58 -0
  23. package/templates/shared/checkly.config.template.ts +58 -0
  24. package/templates/shared/playwright.config.template.mjs +44 -0
  25. package/templates/shared/playwright.config.template.ts +44 -0
  26. package/templates/shared/playwright.mcp.config.template.mjs +12 -0
  27. package/templates/shared/playwright.mcp.config.template.ts +12 -0
  28. package/templates/shared/preview-consent-setup.template.js +32 -0
  29. package/templates/shared/preview-consent-setup.template.ts +33 -0
  30. package/templates/shared/seed.spec.js +5 -0
  31. package/templates/shared/seed.spec.ts +5 -0
@@ -0,0 +1,203 @@
1
+ # Playwright Agent Workflow
2
+
3
+ > Setup and day-to-day workflow docs (for humans): https://app.notion.com/p/31ff837b6270805d8922c4a7c7beb6b0
4
+
5
+ When the user says "Use the playwright-test-planner/generator/healer agent", always delegate via the Agent tool with the matching subagent_type (`playwright-test-planner`, `playwright-test-generator`, `playwright-test-healer`). Never attempt to call the MCP tools directly.
6
+
7
+ **Do not implement Playwright specs in the orchestrating turn.** Writing or editing `*.spec.js` test bodies yourself bypasses replay, generator MCP tools, and project rules. For generation, always use the **Agent** tool with `subagent_type: "playwright-test-generator"`.
8
+
9
+ **Plans:** Read `tests/e2e/<feature>/<name>.plan.md` yourself and pass its contents in the Agent prompt. **Never ask the user to paste the plan** into chat.
10
+
11
+ ## Agents
12
+
13
+ | Trigger phrase in user prompt | `subagent_type` | Purpose |
14
+ | ---------------------------------------------- | --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
15
+ | "Use the playwright-test-planner agent to …" | `playwright-test-planner` | Explores the live site, discovers real selectors/DOM, writes a `tests/e2e/<name>.plan.md` |
16
+ | "Use the playwright-test-generator agent to …" | `playwright-test-generator` | Reads a plan, replays steps in a browser, writes or extends `tests/e2e/<group>.spec.js` or `.spec.ts` (**Project test rules** §7–8) |
17
+ | "Use the playwright-test-healer agent to …" | `playwright-test-healer` | Runs tests, debugs failures, fixes code, re-runs until green |
18
+
19
+ ## Project test rules
20
+
21
+ These rules apply to **all** agents (generator, healer, and orchestrator). Include them in every Agent prompt to the generator or healer.
22
+
23
+ 1. Every logical step MUST have a numbered comment above it
24
+ (e.g. `// 1. Navigate to the homepage`). Preserve existing step comments
25
+ when editing. Add them if missing.
26
+
27
+ 2. Never hardcode domains. Use `page.goto("/")` and derive the origin from the
28
+ live page URL.
29
+
30
+ 3. Every `test.describe` block MUST include a `beforeEach` that runs shared preview setup
31
+ from `tests/e2e/helpers/preview-consent-setup.js` or `.ts` (`prepareVercelPreviewPage`, and when tests
32
+ start after consent, `gotoPathAndAcceptCookies(page, baseURL, path)` with the correct
33
+ relative path). The scaffold ships `prepareVercelPreviewPage` — it exchanges
34
+ `VERCEL_AUTOMATION_BYPASS_SECRET` for Vercel's deployment-protection bypass cookie on the
35
+ first navigation and is a plain `goto` when the variable is unset. Extend the file with
36
+ project consent logic; do not reinvent it. The helper file extension must match rule 7.
37
+
38
+ 4. Never hardcode a single locale's URL pattern. Use a locale config map keyed
39
+ by bare hostname and resolve at runtime.
40
+
41
+ 5. After all tests pass locally, verify with
42
+ `BASE_URL=<dev-url> pnpm playwright test` via the terminal tool. If failures
43
+ occur, fix and re-run until green.
44
+
45
+ **Local runs always pin `BASE_URL` on the command line** to the dev server
46
+ confirmed during setup (usually `http://localhost:3000`) — never rely on the
47
+ ambient environment. Repos commonly export `BASE_URL` for the app's own
48
+ purposes, pointed at the deployed site; an unpinned run silently targets
49
+ production, which does not contain the working tree's changes (the feature
50
+ under test, any `data-testid` additions). Deployed URLs enter the pipeline
51
+ only at preview validation and Checkly, always passed explicitly.
52
+
53
+ 6. **Test root:** Paths in this file assume the default `tests/e2e/`, which keeps e2e specs
54
+ clear of unit tests in the same repo. **The repo wins over this default.** Check
55
+ `playwright.config.{ts,mjs}` for `testDir` and list existing `*.spec.*`; if the **e2e**
56
+ specs already live somewhere else (`tests/` flat, `e2e/`, `src/__tests__/`), write there
57
+ instead and read every `tests/e2e/…` path below as that directory. Never split a suite
58
+ across two roots. A non-e2e suite under `tests/` (unit, a11y) is **not** that signal:
59
+ leave `testDir` on the e2e root and give that suite its own Playwright project with its
60
+ own `testDir` — never widen `testDir` to cover both, which runs the other suite in every
61
+ browser project.
62
+
63
+ 7. **Spec language (generators only):** Repositories differ — some teams use **JavaScript**
64
+ for Playwright tests (`.spec.js`), others use **TypeScript** (`.spec.ts`). List
65
+ `tests/e2e/**/*.spec.*` and match the prevailing extension. If there are no specs yet, default
66
+ to `.spec.js` unless the project README or the user specifies TypeScript for tests. Helpers
67
+ and seeds must use the same language as specs. Do not add `.spec.ts` in a repo that already
68
+ uses `.spec.js` everywhere (and vice versa).
69
+
70
+ 8. **Feature groups (target layout):** Prefer **one spec file per feature group** (e.g.
71
+ `tests/e2e/map/map.spec.js`) with a single `test.describe('<Group name>', …)` and **multiple**
72
+ `test()` blocks — one per plan scenario — sharing one `beforeEach`. Do not default to one
73
+ file per scenario unless scenarios are unrelated. Prefer **lean** tests: assert user-visible
74
+ outcomes and regressions, not every intermediate DOM detail from the plan. Numbered
75
+ comments on **main** user steps only.
76
+
77
+ ## Pipeline: Plan → Generate → Heal
78
+
79
+ ### Step 1 — Plan
80
+
81
+ Trigger when: user wants to create tests for a new feature, page, or user flow.
82
+
83
+ 1. Use the **Agent** tool with `subagent_type: "playwright-test-planner"`
84
+ 2. Pass the URL and user flow description in the `prompt`
85
+ 3. The planner saves output to `tests/e2e/<name>.plan.md`
86
+ 4. When the subagent returns, present the plan to the user for review before proceeding
87
+
88
+ Prompt to forward:
89
+
90
+ ```
91
+ Create a test plan for: <FEATURE NAME>
92
+
93
+ Navigate to <URL> and explore the following user flow:
94
+ <USER'S FLOW STEPS>
95
+
96
+ While exploring, verify:
97
+ - Actual selectors (see **Locator strategy** under *Robustness patterns* below)
98
+ - Whether cookie consent or login gates appear
99
+ - DOM structure of key interactive elements
100
+ - If the suite targets **multiple domains**, explore **each** domain the tests will run against
101
+ (see *Multi-domain / multi-locale testing* — the plan must reflect every distinct host or market)
102
+
103
+ Generate a seed file at tests/e2e/<feature>/seed.spec.js or seed.spec.ts (same extension as
104
+ the rest of this repo's specs — see Project test rules §7) that handles the common setup
105
+ (navigation, cookie/auth consent, waiting for key elements) discovered during exploration.
106
+
107
+ Save the plan to tests/e2e/<feature>/<name>.plan.md using planner_save_plan.
108
+ All URLs must use relative paths — never localhost or absolute domains.
109
+ ```
110
+
111
+ ### Step 2 — Generate
112
+
113
+ Trigger when: a `tests/e2e/<name>.plan.md` exists and the user wants generated test code.
114
+
115
+ **You (the orchestrating agent) own this step end-to-end.** The human does **not** need to
116
+ invoke the generator separately for each scenario or plan section — you chain **Agent** calls
117
+ to `playwright-test-generator` until the feature group is implemented (see **Important rules**).
118
+
119
+ 1. Read `tests/e2e/<name>.plan.md` to get the full plan content
120
+ 2. Use the **Agent** tool with `subagent_type: "playwright-test-generator"` — **once per scenario**
121
+ if building incrementally, **or** pass the **full plan** and all scenarios for the group in one
122
+ prompt if the generator can emit the whole file (**Project test rules** §8)
123
+ 3. Pass the **plan content** in the `prompt` (the generator subagent cannot read arbitrary paths from disk)
124
+ 4. Include the **project test rules** above in the prompt
125
+ 5. If the plan contains a "Recommended data-testid additions" table, include it — the generator adds those attributes before replaying steps
126
+ 6. Target **one spec file per feature group** (e.g. `tests/e2e/map/map.spec.js`) with a shared
127
+ `test.describe` and **multiple** `test()` blocks — merge with `edit` between runs if needed
128
+
129
+ Prompt to forward (example — adjust per repo):
130
+
131
+ ```
132
+ Implement tests from the plan for this feature group. Target layout: one file, one describe, multiple tests (see project test rules §8).
133
+
134
+ - Group / describe: "<TOP-LEVEL PLAN HEADING e.g. Map>"
135
+ - File: tests/e2e/<feature>/<group>.spec.js (or `.spec.ts` per project)
136
+ - Seed: <SEED FILE PATH FROM PLAN>
137
+ - Viewport: <WIDTH>x<HEIGHT>
138
+
139
+ <plan>
140
+ <PASTE full plan OR the scenario block(s) you are implementing this run>
141
+ </plan>
142
+
143
+ If recommended data-testid additions exist, add them to source components before replaying steps.
144
+
145
+ After the spec is written or updated, run
146
+ BASE_URL=<dev-url> pnpm playwright test tests/e2e/<feature>/ (pin the dev server per rule 5).
147
+ ```
148
+
149
+ ### Step 3 — Heal
150
+
151
+ Trigger when: tests are failing (after generation, after site changes, or on user request).
152
+
153
+ 1. Use the **Agent** tool with `subagent_type: "playwright-test-healer"`
154
+ 2. Pass the test file path in the `prompt`
155
+ 3. Include the **project test rules** above in the prompt so the healer follows them
156
+
157
+ Prompt to forward (always include the rules block):
158
+
159
+ ```
160
+ Run and fix the tests in tests/e2e/<name>.spec.js (or `.ts` per project)
161
+
162
+ If any tests fail:
163
+
164
+ - Debug each failure
165
+ - Examine the page state, selectors, and assertions
166
+ - Fix the test code
167
+ - Re-run until all tests pass locally
168
+
169
+ ## Project test rules (follow these strictly)
170
+
171
+ <COPY the Project test rules section above into this prompt>
172
+ ```
173
+
174
+ ## Important rules
175
+
176
+ - **Always delegate via the Agent tool.** Never call `playwright-test/*` tools directly.
177
+ - **Always plan before generating.** Use the planner to discover real selectors first.
178
+ - **Always read the plan file** before passing to the generator. The generator runs in
179
+ isolation — include the plan or scenario steps in the prompt.
180
+ - **You implement the full generate phase.** After plan approval, **you** invoke the
181
+ generator subagent **as many times as needed** (typically once per scenario, sequentially)
182
+ or supply the full multi-scenario prompt — **without** asking the user to run the generator
183
+ for each plan section. A single user request such as "Implement the tests from this plan" is enough.
184
+ - **Run generator subagent calls sequentially.** Never parallel — the MCP browser server is shared.
185
+ - **Do not interrupt.** Sending a message while a generator is running aborts it.
186
+ Wait for completion before proceeding.
187
+ - **data-testid additions are owned by the generator.** When the plan includes a
188
+ "Recommended data-testid additions" table, the generator subagent adds those
189
+ attributes to source components before replaying steps. The orchestrator must
190
+ forward the table verbatim in the generator prompt and must NOT edit source
191
+ components itself.
192
+ - **Present the plan for review** after planning. Unless the user narrows scope, **proceed
193
+ to generate all scenarios** for the agreed feature group (see **Project test rules** §8).
194
+ - **Generator and healer self-verify locally** with
195
+ `BASE_URL=<dev-url> pnpm playwright test` (pinned per **Project test rules** §5).
196
+ Cloud validation via `checkly pw-test` is a separate human step after pushing.
197
+ Only invoke the healer if a generated test fails locally.
198
+ - **Base URL**: local runs pin `BASE_URL=<dev-url>` on the command (**Project
199
+ test rules** §5); the config's localhost fallback is a safety net, not a
200
+ guarantee, because an exported `BASE_URL` overrides it silently.
201
+ - **Plan files** are saved at `tests/e2e/<feature>/<name>.plan.md`.
202
+ - **Test files** follow **Project test rules** §7–8 — typically `tests/e2e/<feature>/<group>.spec.js`
203
+ or `.spec.ts` with one `describe` and multiple tests.
@@ -0,0 +1,201 @@
1
+ ---
2
+ name: playwright-test-pipeline
3
+ description: Guides the full workflow for creating and maintaining Playwright
4
+ end-to-end tests with the planner/generator/healer agents — setup check,
5
+ Plan → Generate → Heal, local runs, Checkly preview validation, and
6
+ deployment. Use when the user wants to create, extend, or fix e2e/Playwright
7
+ tests, asks to "test" a page/feature/flow, or asks how the test pipeline
8
+ works.
9
+ ---
10
+
11
+ # Playwright Test Pipeline
12
+
13
+ You are guiding a person through this project's test pipeline. They may not
14
+ know it exists — narrate briefly at each phase, and pause at the marked
15
+ decision points. The delegation rules and prompt templates live in
16
+ `.claude/rules/playwright-workflow.md` (already in your context); follow them,
17
+ do not improvise your own prompts to the agents.
18
+
19
+ ## Phase 0 — Setup (always first)
20
+
21
+ Verify, in order:
22
+
23
+ 1. `.claude/agents/playwright-test-planner.md` (and generator/healer) exist
24
+ 2. `.mcp.json` at project root defines the `playwright-test` server, pointed
25
+ at `playwright.mcp.config.{ts,mjs}` (which must exist — it lets the MCP run
26
+ seeds that `testIgnore` hides from normal runs)
27
+ 3. A `playwright.config.{ts,mjs}` and `checkly.config.{ts,mjs}` exist
28
+
29
+ ### If anything is missing, install it — do not just bounce the user
30
+
31
+ Say what is missing, then offer to run the scaffold. Do not run it without
32
+ agreement: it installs dependencies and writes files.
33
+
34
+ **Always pass `--format` explicitly.** Without it the CLI opens an interactive
35
+ prompt, and in a non-interactive shell that prompt resolves empty and the CLI
36
+ exits silently having done nothing — it looks like it worked. Pick the format
37
+ from the repo: `ts` when there is a `tsconfig.json`, otherwise `mjs`.
38
+
39
+ ```bash
40
+ pnpm dlx @andwhy/e2e-pipeline init --format ts # or --format mjs
41
+ ```
42
+
43
+ Add `--force` only when updating an existing install. It regenerates
44
+ `.claude/**` and `.mcp.json` and never touches the configs.
45
+
46
+ Then read the output back to the user: what was created, what was skipped.
47
+ Docs: https://app.notion.com/p/31ff837b6270805d8922c4a7c7beb6b0
48
+
49
+ **After a mid-session install, a restart is required — this is a hard stop.**
50
+ Claude Code registers subagents and `.mcp.json` servers at session start, so
51
+ the planner/generator/healer and the `playwright-test` browser tools do not
52
+ exist in the session that just scaffolded them. Do NOT work around this:
53
+ no inlining an agent definition into a general-purpose agent, no hand-writing
54
+ plans or specs from source code — without the browser tools that produces
55
+ unverified selectors, which is exactly what this pipeline exists to prevent.
56
+ Finish Phase 0 (configs, reachable URL), then tell the user to restart the
57
+ session and re-ask; Phase 0 will pass and planning starts for real.
58
+
59
+ ### Then finish the configuration before planning
60
+
61
+ The scaffold writes templates, not a working setup. Walk these with the user:
62
+
63
+ 1. `playwright.config` — `baseURL` for local runs.
64
+ 2. `checkly.config` — real `projectName` and `logicalId`, the production
65
+ domain(s) in `environmentVariables`, `testCommand` (see Phase 5), and the
66
+ check-suite `name` — that is what teammates see in the Checkly dashboard
67
+ and in alerts. **Ask the user what to call it** unless the repo makes it
68
+ obvious (a single clear brand or product name); do not silently invent
69
+ one. Also set
70
+ `installCommand: "pnpm install --ignore-scripts"` — Checkly's runner gets a
71
+ bundle of test files, not the repo, so a project postinstall (codegen etc.)
72
+ fails there on missing files. Placeholder values (`my-app`, `example.com`)
73
+ left in place will block Phase 4; local work is unaffected.
74
+ 3. A reachable URL for the planner: a running dev server or a deployed
75
+ environment. **Confirm this before Phase 1** — the planner explores a live
76
+ page, and pointing it at an empty or unreachable app produces a plan that
77
+ asserts nothing useful.
78
+ 4. **Repository secrets for the workflows**: if the scaffold just created
79
+ `.github/workflows/`, tell the user now — not at merge time — to add three
80
+ repository secrets (GitHub → Settings → Secrets and variables → Actions).
81
+ You cannot add them; a person must. Without them the first PR push and the
82
+ first merge fail CI in ways that look like pipeline bugs.
83
+ - `CHECKLY_API_KEY` — 1Password ("Checkly") first; else Checkly → User
84
+ Settings → API keys. The key is shown once, at creation — tell the user
85
+ to save it to 1Password immediately.
86
+ - `CHECKLY_ACCOUNT_ID` — Checkly → Account Settings → General; also in
87
+ 1Password.
88
+ - `VERCEL_AUTOMATION_BYPASS_SECRET` — Vercel → project → Settings →
89
+ Deployment Protection → Protection Bypass for Automation. Per project —
90
+ never copied from another repo.
91
+ 5. **Existing tests under `tests/`** (unit, a11y — the scaffold lists them as
92
+ "outside tests/e2e"): leave `testDir` at `./tests/e2e` and give that suite
93
+ its own Playwright project with its own `testDir` if it should run under
94
+ Playwright at all. **Never widen `testDir` to `./tests`** — that runs the
95
+ other suite in every browser project and mixes the suites. Checkly is
96
+ pinned to `tests/e2e` by `testCommand` either way.
97
+
98
+ ## Phase 1 — Plan
99
+
100
+ 1. Ask, if not already clear: which page/flow, which URL, and whether the
101
+ suite targets multiple domains (check `checkly.config` for the list).
102
+ 2. Delegate to `playwright-test-planner` using the rules-file template.
103
+ 3. **Decision point:** present the saved plan (`tests/e2e/<feature>/<name>.plan.md`)
104
+ scenario-by-scenario and ask what to keep, drop, or adjust. Do not
105
+ generate before the user responds.
106
+
107
+ ## Phase 2 — Generate
108
+
109
+ 1. Read the plan file yourself; never ask the user to paste it.
110
+ 2. Delegate to `playwright-test-generator` per the rules file (sequential
111
+ calls, project test rules included, data-testid table forwarded verbatim).
112
+ 3. Tell the user which spec file is being written and when each scenario lands.
113
+
114
+ ## Phase 3 — Run and heal
115
+
116
+ 1. Run `BASE_URL=<dev-url> pnpm playwright test tests/e2e/<feature>/`, pinning
117
+ the dev server confirmed in Phase 0. Phases 1–3 always target local: the
118
+ working tree's changes (the feature, any `data-testid` additions) exist
119
+ nowhere else, and an ambient `BASE_URL` export would silently point the run
120
+ at the deployed site. Deployed URLs enter only at Phase 4, always explicit.
121
+ 2. Green → Phase 4. Red → delegate to `playwright-test-healer` per the rules
122
+ file. If the healer marks anything `test.fixme()`, surface it explicitly —
123
+ the user decides whether that is acceptable.
124
+
125
+ ## Phase 4 — Preview validation (before merge)
126
+
127
+ 1. Auth check (first authenticated command in the pipeline): run
128
+ `pnpm checkly whoami`. If not logged in, have the user run
129
+ `pnpm checkly login` — it opens a browser flow, and the shared team
130
+ credentials are in 1Password (search "Checkly"). Local Playwright runs
131
+ never need this, and CI does not either — it uses the `CHECKLY_API_KEY` /
132
+ `CHECKLY_ACCOUNT_ID` repository secrets.
133
+ 2. **Decision point:** ask for the preview deployment URL, and settle how
134
+ Checkly's runners get through Vercel deployment protection. Preferred: the
135
+ project's automation bypass secret (Vercel project settings → Deployment
136
+ Protection; value in 1Password) — the scaffolded helper exchanges it for
137
+ Vercel's bypass cookie on first navigation. Fallback when the project has
138
+ no secret: a share link ("Anyone with the link") as the preview URL.
139
+ 3. Run: `pnpm checkly pw-test --env BASE_URL="<preview-url>" --env VERCEL_AUTOMATION_BYPASS_SECRET="<secret>" -- tests/e2e/<feature>/`
140
+ (omit the second `--env` on the share-link fallback). Never write the
141
+ secret into `checkly.config` — deployed monitors run against production
142
+ and do not need it.
143
+ 4. Failures here but not locally are usually cloud-environment issues
144
+ (timeouts, consent gates) — heal with that framing, then re-run.
145
+
146
+ The user can skip this phase; note that cloud behavior then goes unverified
147
+ until after merge.
148
+
149
+ ## Phase 5 — Merge and deployment
150
+
151
+ The scaffold ships two workflows (seeded once — existing ones are never
152
+ overwritten):
153
+
154
+ - `.github/workflows/pr-playwright-tests.yml` — runs the suite against the
155
+ PR's own Vercel preview on every push (`deployment_status` trigger, so the
156
+ preview is already live and contains the PR's changes), and comments the
157
+ result on the PR.
158
+ - `.github/workflows/checkly-deploy.yml` — when a **production deployment
159
+ succeeds** (`deployment_status` trigger), deploys the checks as Checkly
160
+ monitors. Not on merge: monitors must match what production serves, and a
161
+ merge that adds tests for new `data-testid`s would go red until the release
162
+ carrying them is live — with gated releases that gap can be days. A
163
+ `workflow_dispatch` fallback allows manual runs.
164
+
165
+ **Before the first merge, the repo needs secrets** (values in 1Password):
166
+ `CHECKLY_API_KEY` and `CHECKLY_ACCOUNT_ID` for the deploy,
167
+ `VERCEL_AUTOMATION_BYPASS_SECRET` for the PR run through deployment
168
+ protection. If the scaffold just created the workflows, say so explicitly and
169
+ list the three secrets — a merge without them fails CI and looks like a
170
+ pipeline bug.
171
+
172
+ After merge, monitors update once the production deployment goes live — the
173
+ user verifies in the Checkly dashboard (https://app.checklyhq.com/); nothing
174
+ to run by hand. Manual fallback if a repo deleted the deploy workflow:
175
+ `pnpm checkly test --record`, then `pnpm checkly deploy` (requires the
176
+ Phase 4 login).
177
+
178
+ **Alert notifications are not automatic — walk the user through it.** A
179
+ failing monitor only shows in the Checkly dashboard until an alert channel
180
+ exists, so nobody gets notified. In Checkly:
181
+
182
+ 1. Alert channels → "Add more channels" → select **Slack**.
183
+ 2. Channel configuration: the project's Slack notification channel
184
+ (`#…`). A dedicated test-notification channel may exist by now — ask
185
+ which channel to use rather than assuming.
186
+ 3. Verify the subscriptions: **only the current check** subscribed, and
187
+ **uncheck** "Automatically subscribe newly created checks to this alert
188
+ channel".
189
+ 4. Test the alert channel (button) and confirm the notification arrived in
190
+ Slack.
191
+ 5. "Create alert Slack channel".
192
+
193
+ ## Conventions to uphold
194
+
195
+ The generator and healer enforce spec-level rules themselves. At the pipeline
196
+ level, additionally keep to:
197
+
198
+ - One spec file per feature group; one `describe`, multiple `test()` blocks
199
+ - Commit specs, plans, seeds, **and** any `data-testid` source edits together
200
+ - Never commit with locally failing tests — heal first or `test.fixme()` with
201
+ the user's sign-off
@@ -0,0 +1,95 @@
1
+ // Renders a Playwright JSON report as a PR comment body on stdout.
2
+ import { readFileSync } from 'node:fs'
3
+
4
+ const MARKER = '<!-- playwright-preview-report -->'
5
+ const { PREVIEW_URL = '', RUN_URL = '' } = process.env
6
+
7
+ const truncate = (text, max) =>
8
+ text.length > max ? `${text.slice(0, max)}…` : text
9
+
10
+ // Strips the ANSI colour codes Playwright writes into JSON error messages.
11
+ const clean = text =>
12
+ String(text ?? '')
13
+ .replace(/\u001b\[[0-9;]*m/g, '')
14
+ .trim()
15
+
16
+ const collect = (suite, acc = []) => {
17
+ for (const spec of suite.specs ?? []) {
18
+ for (const test of spec.tests ?? []) {
19
+ const last = test.results?.[test.results.length - 1]
20
+ if (!last || ['passed', 'skipped'].includes(last.status)) continue
21
+ acc.push({
22
+ title: spec.title,
23
+ file: spec.file ?? suite.file,
24
+ line: spec.line,
25
+ status: last.status,
26
+ error: clean(last.error?.message ?? last.errors?.[0]?.message ?? '')
27
+ })
28
+ }
29
+ }
30
+ for (const child of suite.suites ?? []) collect(child, acc)
31
+ return acc
32
+ }
33
+
34
+ const render = report => {
35
+ const failures = (report.suites ?? []).flatMap(s => collect(s))
36
+ const { expected = 0, skipped = 0, flaky = 0 } = report.stats ?? {}
37
+ // Playwright reports a global-timeout abort as a top-level error.
38
+ const truncated = (report.errors ?? []).some(e =>
39
+ /^Timed out waiting .* for the/.test(clean(e.message))
40
+ )
41
+
42
+ const lines = [MARKER, '## Playwright on the preview', '']
43
+
44
+ if (!failures.length && !truncated) {
45
+ lines.push(`All tests passed. ${expected} passed, ${skipped} skipped.`)
46
+ } else {
47
+ lines.push(
48
+ `**${failures.length} failed**, ${expected} passed, ${skipped} skipped` +
49
+ (flaky ? `, ${flaky} flaky` : '') +
50
+ '.'
51
+ )
52
+ }
53
+
54
+ if (truncated) {
55
+ lines.push(
56
+ '',
57
+ '> The run hit its global timeout, so tests after the ones below never ran.'
58
+ )
59
+ }
60
+
61
+ for (const f of failures) {
62
+ lines.push('', `### \`${f.file}:${f.line}\` — ${f.title}`)
63
+ if (f.error) {
64
+ lines.push('', '```', truncate(f.error, 1200), '```')
65
+ }
66
+ }
67
+
68
+ lines.push('', '---', '')
69
+ if (PREVIEW_URL) lines.push(`Preview: ${PREVIEW_URL} `)
70
+ if (RUN_URL) lines.push(`[Full run and HTML report](${RUN_URL}) `)
71
+ lines.push('Chrome only. Firefox and Edge run on Checkly against production.')
72
+
73
+ return lines.join('\n')
74
+ }
75
+
76
+ let report
77
+ try {
78
+ report = JSON.parse(readFileSync(process.argv[2], 'utf8'))
79
+ } catch {
80
+ // No JSON at all means Playwright died before writing one.
81
+ console.log(
82
+ [
83
+ MARKER,
84
+ '## Playwright on the preview',
85
+ '',
86
+ 'The run produced no report. It most likely crashed or was cancelled ' +
87
+ 'before any test finished.',
88
+ '',
89
+ RUN_URL ? `[Full run](${RUN_URL})` : ''
90
+ ].join('\n')
91
+ )
92
+ process.exit(0)
93
+ }
94
+
95
+ console.log(render(report))
@@ -0,0 +1,64 @@
1
+ name: Deploy Checkly checks
2
+
3
+ # Deploys the e2e suite as Checkly monitors when PRODUCTION deploys — not on
4
+ # merge. Monitors must match what production serves: a merge that adds tests
5
+ # for new data-testids goes red until the app deploy carrying those testids is
6
+ # live, and with gated releases that gap can be days. `deployment_status`
7
+ # fires when Vercel finishes the production deployment, so monitors update at
8
+ # the exact moment the code they were recorded against is reachable.
9
+ # `checkly deploy` is idempotent, so redeploying on every production deploy
10
+ # is harmless.
11
+ # Needs repo secrets: CHECKLY_API_KEY, CHECKLY_ACCOUNT_ID (1Password, "Checkly").
12
+ on:
13
+ deployment_status:
14
+ workflow_dispatch:
15
+
16
+ concurrency:
17
+ group: checkly-deploy
18
+ cancel-in-progress: false
19
+
20
+ jobs:
21
+ deploy:
22
+ if: >-
23
+ github.event_name == 'workflow_dispatch' ||
24
+ (github.event.deployment_status.state == 'success' &&
25
+ github.event.deployment.environment == 'Production')
26
+ runs-on: ubuntu-latest
27
+ timeout-minutes: 8
28
+
29
+ steps:
30
+ - name: Check out the deployed ref
31
+ uses: actions/checkout@v5
32
+ with:
33
+ ref: ${{ github.event.deployment.ref || github.ref }}
34
+
35
+ - name: Set up pnpm
36
+ uses: pnpm/action-setup@v4
37
+
38
+ - name: Set up Node
39
+ uses: actions/setup-node@v5
40
+ with:
41
+ node-version: 22
42
+ cache: pnpm
43
+
44
+ # --ignore-scripts: only the tests are bundled; project postinstall
45
+ # hooks are not needed and may want CMS tokens CI does not have.
46
+ - name: Install dependencies
47
+ run: pnpm install --frozen-lockfile --ignore-scripts
48
+
49
+ # --force skips the interactive confirmation prompt.
50
+ - name: Deploy checks to Checkly
51
+ env:
52
+ CHECKLY_API_KEY: ${{ secrets.CHECKLY_API_KEY }}
53
+ CHECKLY_ACCOUNT_ID: ${{ secrets.CHECKLY_ACCOUNT_ID }}
54
+ run: pnpm exec checkly deploy --force
55
+
56
+ - name: Write result to job summary
57
+ if: always()
58
+ run: |
59
+ {
60
+ echo "## Checkly deploy"
61
+ echo ""
62
+ echo "- Ref: \`${{ github.event.deployment.ref || github.ref }}\`"
63
+ echo "- Triggered by a successful production deployment"
64
+ } >> "$GITHUB_STEP_SUMMARY"