@andwhy/e2e-pipeline 0.13.2 → 0.14.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +40 -10
- package/dist/bin/init.js +29 -5
- package/dist/utils/copy.d.ts +15 -1
- package/dist/utils/copy.js +152 -91
- package/package.json +7 -3
- package/plugin/.claude-plugin/plugin.json +18 -0
- package/plugin/agents/playwright-test-generator.md +273 -0
- package/plugin/agents/playwright-test-healer.md +159 -0
- package/plugin/agents/playwright-test-planner.md +135 -0
- package/plugin/hooks/hooks.json +16 -0
- package/plugin/hooks/session-context.md +9 -0
- package/plugin/hooks/session-start.mjs +49 -0
- package/plugin/skills/playwright-test-pipeline/SKILL.md +547 -0
- package/templates/github/prompts/checkly-heal.md +10 -3
- package/templates/github/workflows/checkly-deploy.yml +19 -52
- package/templates/github/workflows/checkly-heal.yml +20 -419
- package/templates/github/workflows/pr-playwright-tests.yml +17 -143
- package/templates/claude/rules/playwright-workflow.md +0 -219
- package/templates/claude/skills/playwright-test-pipeline/SKILL.md +0 -286
|
@@ -1,219 +0,0 @@
|
|
|
1
|
-
# Playwright Agent Workflow
|
|
2
|
-
|
|
3
|
-
> Setup and day-to-day workflow docs (for humans): https://app.notion.com/p/31ff837b6270805d8922c4a7c7beb6b0
|
|
4
|
-
|
|
5
|
-
When the user says "Use the playwright-test-planner/generator/healer agent", always delegate via the Agent tool with the matching subagent_type (`playwright-test-planner`, `playwright-test-generator`, `playwright-test-healer`). Never attempt to call the MCP tools directly.
|
|
6
|
-
|
|
7
|
-
**Do not implement Playwright specs in the orchestrating turn.** Writing or editing `*.spec.js` test bodies yourself bypasses replay, generator MCP tools, and project rules. For generation, always use the **Agent** tool with `subagent_type: "playwright-test-generator"`.
|
|
8
|
-
|
|
9
|
-
**Plans:** Read `tests/e2e/<feature>/<name>.plan.md` yourself and pass its contents in the Agent prompt. **Never ask the user to paste the plan** into chat.
|
|
10
|
-
|
|
11
|
-
## Agents
|
|
12
|
-
|
|
13
|
-
| Trigger phrase in user prompt | `subagent_type` | Purpose |
|
|
14
|
-
| ---------------------------------------------- | --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
|
|
15
|
-
| "Use the playwright-test-planner agent to …" | `playwright-test-planner` | Explores the live site, discovers real selectors/DOM, writes a `tests/e2e/<name>.plan.md` |
|
|
16
|
-
| "Use the playwright-test-generator agent to …" | `playwright-test-generator` | Reads a plan, replays steps in a browser, writes or extends `tests/e2e/<group>.spec.js` or `.spec.ts` (**Project test rules** §7–8) |
|
|
17
|
-
| "Use the playwright-test-healer agent to …" | `playwright-test-healer` | Runs tests, debugs failures, fixes code, re-runs until green |
|
|
18
|
-
|
|
19
|
-
## Project test rules
|
|
20
|
-
|
|
21
|
-
These rules apply to **all** agents (generator, healer, and orchestrator). Include them in every Agent prompt to the generator or healer.
|
|
22
|
-
|
|
23
|
-
1. Every logical step MUST have a numbered comment above it
|
|
24
|
-
(e.g. `// 1. Navigate to the homepage`). Preserve existing step comments
|
|
25
|
-
when editing. Add them if missing.
|
|
26
|
-
|
|
27
|
-
2. Never hardcode domains. Use `page.goto("/")` and derive the origin from the
|
|
28
|
-
live page URL.
|
|
29
|
-
|
|
30
|
-
3. Every `test.describe` block MUST include a `beforeEach` that runs shared preview setup
|
|
31
|
-
from `tests/e2e/helpers/preview-consent-setup.js` or `.ts` (`prepareVercelPreviewPage`, and when tests
|
|
32
|
-
start after consent, `gotoPathAndAcceptCookies(page, baseURL, path)` with the correct
|
|
33
|
-
relative path). The scaffold ships `prepareVercelPreviewPage` — it exchanges
|
|
34
|
-
`VERCEL_AUTOMATION_BYPASS_SECRET` for Vercel's deployment-protection bypass cookie on the
|
|
35
|
-
first navigation and is a plain `goto` when the variable is unset. Extend the file with
|
|
36
|
-
project consent logic; do not reinvent it. The helper file extension must match rule 7.
|
|
37
|
-
|
|
38
|
-
4. Never hardcode a single locale's URL pattern. Use a locale config map keyed
|
|
39
|
-
by bare hostname and resolve at runtime. Preview hosts (`*.vercel.app`) are
|
|
40
|
-
never in the map and must fall back to the primary production domain's
|
|
41
|
-
config (overridable via `PREVIEW_LOCALE_HOST`) — an exact-match-only lookup
|
|
42
|
-
skips the whole suite on every PR run.
|
|
43
|
-
|
|
44
|
-
5. After all tests pass locally, verify with
|
|
45
|
-
`BASE_URL=<dev-url> pnpm playwright test` via the terminal tool. If failures
|
|
46
|
-
occur, fix and re-run until green.
|
|
47
|
-
|
|
48
|
-
**Local runs always pin `BASE_URL` on the command line** to the dev server
|
|
49
|
-
confirmed during setup (usually `http://localhost:3000`) — never rely on the
|
|
50
|
-
ambient environment. Repos commonly export `BASE_URL` for the app's own
|
|
51
|
-
purposes, pointed at the deployed site; an unpinned run silently targets
|
|
52
|
-
production, which does not contain the working tree's changes (the feature
|
|
53
|
-
under test, any `data-testid` additions). Deployed URLs enter the pipeline
|
|
54
|
-
only at preview validation and Checkly, always passed explicitly.
|
|
55
|
-
|
|
56
|
-
6. **Test root:** Paths in this file assume the default `tests/e2e/`, which keeps e2e specs
|
|
57
|
-
clear of unit tests in the same repo. **The repo wins over this default.** Check
|
|
58
|
-
`playwright.config.{ts,mjs}` for `testDir` and list existing `*.spec.*`; if the **e2e**
|
|
59
|
-
specs already live somewhere else (`tests/` flat, `e2e/`, `src/__tests__/`), write there
|
|
60
|
-
instead and read every `tests/e2e/…` path below as that directory. Never split a suite
|
|
61
|
-
across two roots. A non-e2e suite under `tests/` (unit, a11y) is **not** that signal:
|
|
62
|
-
leave `testDir` on the e2e root and give that suite its own Playwright project with its
|
|
63
|
-
own `testDir` — never widen `testDir` to cover both, which runs the other suite in every
|
|
64
|
-
browser project.
|
|
65
|
-
|
|
66
|
-
7. **Spec language (generators only):** Repositories differ — some teams use **JavaScript**
|
|
67
|
-
for Playwright tests (`.spec.js`), others use **TypeScript** (`.spec.ts`). List
|
|
68
|
-
`tests/e2e/**/*.spec.*` and match the prevailing extension. If there are no specs yet, default
|
|
69
|
-
to `.spec.js` unless the project README or the user specifies TypeScript for tests. Helpers
|
|
70
|
-
and seeds must use the same language as specs. Do not add `.spec.ts` in a repo that already
|
|
71
|
-
uses `.spec.js` everywhere (and vice versa).
|
|
72
|
-
|
|
73
|
-
8. **Feature groups (target layout):** Prefer **one spec file per feature group** (e.g.
|
|
74
|
-
`tests/e2e/map/map.spec.js`) with a single `test.describe('<Group name>', …)` and **multiple**
|
|
75
|
-
`test()` blocks — one per plan scenario — sharing one `beforeEach`. Do not default to one
|
|
76
|
-
file per scenario unless scenarios are unrelated. Prefer **lean** tests: assert user-visible
|
|
77
|
-
outcomes and regressions, not every intermediate DOM detail from the plan. Numbered
|
|
78
|
-
comments on **main** user steps only.
|
|
79
|
-
|
|
80
|
-
## Pipeline: Plan → Generate → Heal
|
|
81
|
-
|
|
82
|
-
### Step 1 — Plan
|
|
83
|
-
|
|
84
|
-
Trigger when: user wants to create tests for a new feature, page, or user flow.
|
|
85
|
-
|
|
86
|
-
1. Use the **Agent** tool with `subagent_type: "playwright-test-planner"`
|
|
87
|
-
2. Pass the URL and user flow description in the `prompt`
|
|
88
|
-
3. The planner saves output to `tests/e2e/<name>.plan.md`
|
|
89
|
-
4. When the subagent returns, present the plan to the user for review before proceeding
|
|
90
|
-
|
|
91
|
-
Prompt to forward:
|
|
92
|
-
|
|
93
|
-
```
|
|
94
|
-
Create a test plan for: <FEATURE NAME>
|
|
95
|
-
|
|
96
|
-
Navigate to <URL> and explore the following user flow:
|
|
97
|
-
<USER'S FLOW STEPS>
|
|
98
|
-
|
|
99
|
-
While exploring, verify:
|
|
100
|
-
- Actual selectors (see **Locator strategy** under *Robustness patterns* below)
|
|
101
|
-
- Whether cookie consent or login gates appear
|
|
102
|
-
- DOM structure of key interactive elements
|
|
103
|
-
- If the suite targets **multiple domains**, explore **each** domain the tests will run against
|
|
104
|
-
(see *Multi-domain / multi-locale testing* — the plan must reflect every distinct host or market)
|
|
105
|
-
|
|
106
|
-
Generate a seed file at tests/e2e/<feature>/seed.spec.js or seed.spec.ts (same extension as
|
|
107
|
-
the rest of this repo's specs — see Project test rules §7) that handles the common setup
|
|
108
|
-
(navigation, cookie/auth consent, waiting for key elements) discovered during exploration.
|
|
109
|
-
|
|
110
|
-
Save the plan to tests/e2e/<feature>/<name>.plan.md using planner_save_plan.
|
|
111
|
-
All URLs must use relative paths — never localhost or absolute domains.
|
|
112
|
-
```
|
|
113
|
-
|
|
114
|
-
### Step 2 — Generate
|
|
115
|
-
|
|
116
|
-
Trigger when: a `tests/e2e/<name>.plan.md` exists and the user wants generated test code.
|
|
117
|
-
|
|
118
|
-
**You (the orchestrating agent) own this step end-to-end.** The human does **not** need to
|
|
119
|
-
invoke the generator separately for each scenario or plan section — you chain **Agent** calls
|
|
120
|
-
to `playwright-test-generator` until the feature group is implemented (see **Important rules**).
|
|
121
|
-
|
|
122
|
-
1. Read `tests/e2e/<name>.plan.md` to get the full plan content
|
|
123
|
-
2. Use the **Agent** tool with `subagent_type: "playwright-test-generator"` — **once per scenario**
|
|
124
|
-
if building incrementally, **or** pass the **full plan** and all scenarios for the group in one
|
|
125
|
-
prompt if the generator can emit the whole file (**Project test rules** §8)
|
|
126
|
-
3. Pass the **plan content** in the `prompt` (the generator subagent cannot read arbitrary paths from disk)
|
|
127
|
-
4. Include the **project test rules** above in the prompt
|
|
128
|
-
5. If the plan contains a "Recommended data-testid additions" table, include it — the generator adds those attributes before replaying steps
|
|
129
|
-
6. Target **one spec file per feature group** (e.g. `tests/e2e/map/map.spec.js`) with a shared
|
|
130
|
-
`test.describe` and **multiple** `test()` blocks — merge with `edit` between runs if needed
|
|
131
|
-
|
|
132
|
-
Prompt to forward (example — adjust per repo):
|
|
133
|
-
|
|
134
|
-
```
|
|
135
|
-
Implement tests from the plan for this feature group. Target layout: one file, one describe, multiple tests (see project test rules §8).
|
|
136
|
-
|
|
137
|
-
- Group / describe: "<TOP-LEVEL PLAN HEADING e.g. Map>"
|
|
138
|
-
- File: tests/e2e/<feature>/<group>.spec.js (or `.spec.ts` per project)
|
|
139
|
-
- Seed: <SEED FILE PATH FROM PLAN>
|
|
140
|
-
- Viewport: <WIDTH>x<HEIGHT>
|
|
141
|
-
|
|
142
|
-
<plan>
|
|
143
|
-
<PASTE full plan OR the scenario block(s) you are implementing this run>
|
|
144
|
-
</plan>
|
|
145
|
-
|
|
146
|
-
If recommended data-testid additions exist, add them to source components before replaying steps.
|
|
147
|
-
|
|
148
|
-
After the spec is written or updated, run
|
|
149
|
-
BASE_URL=<dev-url> pnpm playwright test tests/e2e/<feature>/ (pin the dev server per rule 5).
|
|
150
|
-
```
|
|
151
|
-
|
|
152
|
-
### Step 3 — Heal
|
|
153
|
-
|
|
154
|
-
Trigger when: tests are failing (after generation, after site changes, or on user request).
|
|
155
|
-
|
|
156
|
-
1. Use the **Agent** tool with `subagent_type: "playwright-test-healer"`
|
|
157
|
-
2. Pass the test file path in the `prompt`
|
|
158
|
-
3. Include the **project test rules** above in the prompt so the healer follows them
|
|
159
|
-
|
|
160
|
-
Prompt to forward (always include the rules block):
|
|
161
|
-
|
|
162
|
-
```
|
|
163
|
-
Run and fix the tests in tests/e2e/<name>.spec.js (or `.ts` per project)
|
|
164
|
-
|
|
165
|
-
If any tests fail:
|
|
166
|
-
|
|
167
|
-
- Debug each failure
|
|
168
|
-
- Examine the page state, selectors, and assertions
|
|
169
|
-
- Fix the test code
|
|
170
|
-
- Re-run until all tests pass locally
|
|
171
|
-
|
|
172
|
-
## Project test rules (follow these strictly)
|
|
173
|
-
|
|
174
|
-
<COPY the Project test rules section above into this prompt>
|
|
175
|
-
```
|
|
176
|
-
|
|
177
|
-
## Important rules
|
|
178
|
-
|
|
179
|
-
- **Always delegate via the Agent tool.** Never call `playwright-test/*` tools directly.
|
|
180
|
-
- **Always plan before generating.** Use the planner to discover real selectors first.
|
|
181
|
-
- **Always read the plan file** before passing to the generator. The generator runs in
|
|
182
|
-
isolation — include the plan or scenario steps in the prompt.
|
|
183
|
-
- **You implement the full generate phase.** After plan approval, **you** invoke the
|
|
184
|
-
generator subagent **as many times as needed** (typically once per scenario, sequentially)
|
|
185
|
-
or supply the full multi-scenario prompt — **without** asking the user to run the generator
|
|
186
|
-
for each plan section. A single user request such as "Implement the tests from this plan" is enough.
|
|
187
|
-
- **Run generator subagent calls sequentially.** Never parallel — the MCP browser server is shared.
|
|
188
|
-
- **Do not interrupt.** Sending a message while a generator is running aborts it.
|
|
189
|
-
Wait for completion before proceeding.
|
|
190
|
-
- **data-testid additions are owned by the generator.** When the plan includes a
|
|
191
|
-
"Recommended data-testid additions" table, the generator subagent adds those
|
|
192
|
-
attributes to source components before replaying steps. The orchestrator must
|
|
193
|
-
forward the table verbatim in the generator prompt and must NOT edit source
|
|
194
|
-
components itself.
|
|
195
|
-
- **Present the plan for review** after planning. Unless the user narrows scope, **proceed
|
|
196
|
-
to generate all scenarios** for the agreed feature group (see **Project test rules** §8).
|
|
197
|
-
- **Generator and healer self-verify locally** with
|
|
198
|
-
`BASE_URL=<dev-url> pnpm playwright test` (pinned per **Project test rules** §5).
|
|
199
|
-
Cloud validation via `checkly pw-test` is a separate human step after pushing.
|
|
200
|
-
Only invoke the healer if a generated test fails locally.
|
|
201
|
-
- **Base URL**: local runs pin `BASE_URL=<dev-url>` on the command (**Project
|
|
202
|
-
test rules** §5); the config's localhost fallback is a safety net, not a
|
|
203
|
-
guarantee, because an exported `BASE_URL` overrides it silently.
|
|
204
|
-
- **Plan files** are saved at `tests/e2e/<feature>/<name>.plan.md`.
|
|
205
|
-
- **Test files** follow **Project test rules** §7–8 — typically `tests/e2e/<feature>/<group>.spec.js`
|
|
206
|
-
or `.spec.ts` with one `describe` and multiple tests.
|
|
207
|
-
|
|
208
|
-
## Checkly auto-heal (CI, not this session)
|
|
209
|
-
|
|
210
|
-
Failing deployed monitors are handled by `.github/workflows/checkly-heal.yml`:
|
|
211
|
-
it triages first (production regression → Slack alert; stale test → fix; flaky
|
|
212
|
-
test → harden), may add behavior-neutral test hooks (`data-testid`) to app
|
|
213
|
-
source when the robust fix needs them, and opens a `checkly-heal` PR a human
|
|
214
|
-
must review — app-side hooks are validated by the PR's own preview run. Its
|
|
215
|
-
agent instructions live in `.github/prompts/checkly-heal.md` (package-owned —
|
|
216
|
-
update via `init --force`, don't hand-edit). If a user asks about a
|
|
217
|
-
`checkly-heal` PR, that workflow authored it; reviewing one means verifying the
|
|
218
|
-
updated test still describes correct production behavior, not that it merely
|
|
219
|
-
passes.
|
|
@@ -1,286 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: playwright-test-pipeline
|
|
3
|
-
description: Guides the full workflow for creating and maintaining Playwright
|
|
4
|
-
end-to-end tests with the planner/generator/healer agents — setup check,
|
|
5
|
-
Plan → Generate → Heal, local runs, Checkly preview validation, and
|
|
6
|
-
deployment. Use when the user wants to create, extend, or fix e2e/Playwright
|
|
7
|
-
tests, asks to "test" a page/feature/flow, or asks how the test pipeline
|
|
8
|
-
works.
|
|
9
|
-
---
|
|
10
|
-
|
|
11
|
-
# Playwright Test Pipeline
|
|
12
|
-
|
|
13
|
-
You are guiding a person through this project's test pipeline. They may not
|
|
14
|
-
know it exists — narrate briefly at each phase, and pause at the marked
|
|
15
|
-
decision points. The delegation rules and prompt templates live in
|
|
16
|
-
`.claude/rules/playwright-workflow.md` (already in your context); follow them,
|
|
17
|
-
do not improvise your own prompts to the agents.
|
|
18
|
-
|
|
19
|
-
## Phase 0 — Setup (always first)
|
|
20
|
-
|
|
21
|
-
Verify, in order:
|
|
22
|
-
|
|
23
|
-
1. `.claude/agents/playwright-test-planner.md` (and generator/healer) exist
|
|
24
|
-
2. `.mcp.json` at project root defines the `playwright-test` server, pointed
|
|
25
|
-
at `playwright.mcp.config.{ts,mjs}` (which must exist — it lets the MCP run
|
|
26
|
-
seeds that `testIgnore` hides from normal runs)
|
|
27
|
-
3. A `playwright.config.{ts,mjs}` and `checkly.config.{ts,mjs}` exist
|
|
28
|
-
|
|
29
|
-
### If anything is missing, install it — do not just bounce the user
|
|
30
|
-
|
|
31
|
-
Say what is missing, then offer to run the scaffold. Do not run it without
|
|
32
|
-
agreement: it installs dependencies and writes files.
|
|
33
|
-
|
|
34
|
-
**Always pass `--format` explicitly.** Without it the CLI opens an interactive
|
|
35
|
-
prompt, and in a non-interactive shell that prompt resolves empty and the CLI
|
|
36
|
-
exits silently having done nothing — it looks like it worked. Pick the format
|
|
37
|
-
from the repo: `ts` when there is a `tsconfig.json`, otherwise `mjs`.
|
|
38
|
-
|
|
39
|
-
```bash
|
|
40
|
-
pnpm dlx @andwhy/e2e-pipeline init --format ts # or --format mjs
|
|
41
|
-
```
|
|
42
|
-
|
|
43
|
-
Add `--force` only when updating an existing install. It regenerates
|
|
44
|
-
`.claude/**` and `.mcp.json` and never touches the configs.
|
|
45
|
-
|
|
46
|
-
Then read the output back to the user: what was created, what was skipped.
|
|
47
|
-
Docs: https://app.notion.com/p/31ff837b6270805d8922c4a7c7beb6b0
|
|
48
|
-
|
|
49
|
-
**After a mid-session install, a restart is required — this is a hard stop.**
|
|
50
|
-
Claude Code registers subagents and `.mcp.json` servers at session start, so
|
|
51
|
-
the planner/generator/healer and the `playwright-test` browser tools do not
|
|
52
|
-
exist in the session that just scaffolded them. Do NOT work around this:
|
|
53
|
-
no inlining an agent definition into a general-purpose agent, no hand-writing
|
|
54
|
-
plans or specs from source code — without the browser tools that produces
|
|
55
|
-
unverified selectors, which is exactly what this pipeline exists to prevent.
|
|
56
|
-
Finish Phase 0 (configs, reachable URL), then tell the user to restart the
|
|
57
|
-
session and re-ask; Phase 0 will pass and planning starts for real.
|
|
58
|
-
|
|
59
|
-
### Then finish the configuration before planning
|
|
60
|
-
|
|
61
|
-
The scaffold writes templates, not a working setup. Walk these with the user:
|
|
62
|
-
|
|
63
|
-
1. `playwright.config` — `baseURL` for local runs.
|
|
64
|
-
2. `checkly.config` — real `projectName` and `logicalId`, the production
|
|
65
|
-
domain(s) in `environmentVariables`, `testCommand` (see Phase 5), and the
|
|
66
|
-
check-suite `name` — that is what teammates see in the Checkly dashboard
|
|
67
|
-
and in alerts. **Ask the user what to call it** unless the repo makes it
|
|
68
|
-
obvious (a single clear brand or product name); do not silently invent
|
|
69
|
-
one. Also set
|
|
70
|
-
`installCommand: "pnpm install --ignore-scripts"` — Checkly's runner gets a
|
|
71
|
-
bundle of test files, not the repo, so a project postinstall (codegen etc.)
|
|
72
|
-
fails there on missing files. Placeholder values (`my-app`, `example.com`)
|
|
73
|
-
left in place will block Phase 4; local work is unaffected.
|
|
74
|
-
3. A reachable URL for the planner: a running dev server or a deployed
|
|
75
|
-
environment. **Confirm this before Phase 1** — the planner explores a live
|
|
76
|
-
page, and pointing it at an empty or unreachable app produces a plan that
|
|
77
|
-
asserts nothing useful.
|
|
78
|
-
4. **Repository secrets for the workflows**: if the scaffold just created
|
|
79
|
-
`.github/workflows/`, tell the user now — not at merge time — to add the
|
|
80
|
-
repository secrets (GitHub → Settings → Secrets and variables → Actions).
|
|
81
|
-
You cannot add them; a person must. Without them the first PR push and the
|
|
82
|
-
first merge fail CI in ways that look like pipeline bugs.
|
|
83
|
-
- `CHECKLY_API_KEY` — 1Password ("Checkly") first; else Checkly → User
|
|
84
|
-
Settings → API keys. The key is shown once, at creation — tell the user
|
|
85
|
-
to save it to 1Password immediately.
|
|
86
|
-
- `CHECKLY_ACCOUNT_ID` — Checkly → Account Settings → General; also in
|
|
87
|
-
1Password.
|
|
88
|
-
- `VERCEL_AUTOMATION_BYPASS_SECRET` — Vercel → project → Settings →
|
|
89
|
-
Deployment Protection → Protection Bypass for Automation. Per project —
|
|
90
|
-
never copied from another repo.
|
|
91
|
-
- `SLACK_WEBHOOK_URL` — the **org-shared** e2e-channel webhook: paste the
|
|
92
|
-
value from 1Password ("e2e Slack webhook"). Same value in every repo;
|
|
93
|
-
do not walk the user through creating a Slack app or webhook — routing
|
|
94
|
-
to the project channel happens via a Slack rule in Phase 5.
|
|
95
|
-
- `ANTHROPIC_API_KEY` usually needs **no action**: it is an
|
|
96
|
-
organization-level secret inherited by every repo — do not walk the
|
|
97
|
-
user through adding it up front. But it can fail to reach a repo
|
|
98
|
-
(org-secret scoping, or a stale stored value): if a heal run's agent
|
|
99
|
-
step fails to authenticate, have the user add a repository-level
|
|
100
|
-
`ANTHROPIC_API_KEY` — a repo secret with the same name overrides the
|
|
101
|
-
organization one.
|
|
102
|
-
5. **Existing tests under `tests/`** (unit, a11y — the scaffold lists them as
|
|
103
|
-
"outside tests/e2e"): leave `testDir` at `./tests/e2e` and give that suite
|
|
104
|
-
its own Playwright project with its own `testDir` if it should run under
|
|
105
|
-
Playwright at all. **Never widen `testDir` to `./tests`** — that runs the
|
|
106
|
-
other suite in every browser project and mixes the suites. Checkly is
|
|
107
|
-
pinned to `tests/e2e` by `testCommand` either way.
|
|
108
|
-
|
|
109
|
-
## Phase 1 — Plan
|
|
110
|
-
|
|
111
|
-
1. Ask, if not already clear: which page/flow, which URL, and whether the
|
|
112
|
-
suite targets multiple domains (check `checkly.config` for the list).
|
|
113
|
-
2. Delegate to `playwright-test-planner` using the rules-file template.
|
|
114
|
-
3. **Decision point:** present the saved plan (`tests/e2e/<feature>/<name>.plan.md`)
|
|
115
|
-
scenario-by-scenario and ask what to keep, drop, or adjust. Do not
|
|
116
|
-
generate before the user responds.
|
|
117
|
-
|
|
118
|
-
## Phase 2 — Generate
|
|
119
|
-
|
|
120
|
-
1. Read the plan file yourself; never ask the user to paste it.
|
|
121
|
-
2. Delegate to `playwright-test-generator` per the rules file (sequential
|
|
122
|
-
calls, project test rules included, data-testid table forwarded verbatim).
|
|
123
|
-
3. Tell the user which spec file is being written and when each scenario lands.
|
|
124
|
-
|
|
125
|
-
## Phase 3 — Run and heal
|
|
126
|
-
|
|
127
|
-
1. Run `BASE_URL=<dev-url> pnpm playwright test tests/e2e/<feature>/`, pinning
|
|
128
|
-
the dev server confirmed in Phase 0. Phases 1–3 always target local: the
|
|
129
|
-
working tree's changes (the feature, any `data-testid` additions) exist
|
|
130
|
-
nowhere else, and an ambient `BASE_URL` export would silently point the run
|
|
131
|
-
at the deployed site. Deployed URLs enter only at Phase 4, always explicit.
|
|
132
|
-
2. Green → Phase 4. Red → delegate to `playwright-test-healer` per the rules
|
|
133
|
-
file. If the healer marks anything `test.fixme()`, surface it explicitly —
|
|
134
|
-
the user decides whether that is acceptable.
|
|
135
|
-
|
|
136
|
-
## Phase 4 — Preview validation (before merge)
|
|
137
|
-
|
|
138
|
-
1. Auth check (first authenticated command in the pipeline): run
|
|
139
|
-
`pnpm checkly whoami`. If not logged in, have the user run
|
|
140
|
-
`pnpm checkly login` — it opens a browser flow, and the shared team
|
|
141
|
-
credentials are in 1Password (search "Checkly"). Local Playwright runs
|
|
142
|
-
never need this, and CI does not either — it uses the `CHECKLY_API_KEY` /
|
|
143
|
-
`CHECKLY_ACCOUNT_ID` repository secrets.
|
|
144
|
-
2. **Decision point:** ask for the preview deployment URL, and settle how
|
|
145
|
-
Checkly's runners get through Vercel deployment protection. Preferred: the
|
|
146
|
-
project's automation bypass secret (Vercel project settings → Deployment
|
|
147
|
-
Protection; value in 1Password) — the scaffolded helper exchanges it for
|
|
148
|
-
Vercel's bypass cookie on first navigation. Fallback when the project has
|
|
149
|
-
no secret: a share link ("Anyone with the link") as the preview URL.
|
|
150
|
-
3. Run: `pnpm checkly pw-test --env BASE_URL="<preview-url>" --env VERCEL_AUTOMATION_BYPASS_SECRET="<secret>" -- tests/e2e/<feature>/`
|
|
151
|
-
(omit the second `--env` on the share-link fallback). Never write the
|
|
152
|
-
secret into `checkly.config` — deployed monitors run against production
|
|
153
|
-
and do not need it.
|
|
154
|
-
4. Failures here but not locally are usually cloud-environment issues
|
|
155
|
-
(timeouts, consent gates) — heal with that framing, then re-run.
|
|
156
|
-
|
|
157
|
-
The user can skip this phase; note that cloud behavior then goes unverified
|
|
158
|
-
until after merge.
|
|
159
|
-
|
|
160
|
-
## Phase 5 — Merge and deployment
|
|
161
|
-
|
|
162
|
-
The scaffold ships three workflows (seeded once — existing ones are never
|
|
163
|
-
overwritten):
|
|
164
|
-
|
|
165
|
-
- `.github/workflows/pr-playwright-tests.yml` — runs the suite against the
|
|
166
|
-
PR's own Vercel preview on every push (`deployment_status` trigger, so the
|
|
167
|
-
preview is already live and contains the PR's changes), and comments the
|
|
168
|
-
result on the PR.
|
|
169
|
-
- `.github/workflows/checkly-deploy.yml` — when a **production deployment
|
|
170
|
-
succeeds** (`deployment_status` trigger), deploys the checks as Checkly
|
|
171
|
-
monitors. Not on merge: monitors must match what production serves, and a
|
|
172
|
-
merge that adds tests for new `data-testid`s would go red until the release
|
|
173
|
-
carrying them is live — with gated releases that gap can be days. A
|
|
174
|
-
`workflow_dispatch` fallback allows manual runs.
|
|
175
|
-
- `.github/workflows/checkly-heal.yml` — a failing deployed monitor
|
|
176
|
-
dispatches here (via the webhook alert channel below). An agent triages
|
|
177
|
-
first: a production regression or anything needing a behavioral app change
|
|
178
|
-
alerts Slack immediately; a stale test gets fixed and a flaky test gets
|
|
179
|
-
hardened — including behavior-neutral test hooks (`data-testid`) in app
|
|
180
|
-
source when the robust fix needs them — and opened as a PR that a human
|
|
181
|
-
must review. Test-only fixes are verified against production and on
|
|
182
|
-
Checkly's infrastructure; app-side hooks are validated by the PR's own
|
|
183
|
-
preview run (pr-playwright-tests). Inert until the webhook exists.
|
|
184
|
-
|
|
185
|
-
**Before the first merge, the repo needs secrets** (values in 1Password):
|
|
186
|
-
`CHECKLY_API_KEY` and `CHECKLY_ACCOUNT_ID` for the deploy and auto-heal,
|
|
187
|
-
`VERCEL_AUTOMATION_BYPASS_SECRET` for the PR run through deployment
|
|
188
|
-
protection, `SLACK_WEBHOOK_URL` for auto-heal notifications
|
|
189
|
-
(`ANTHROPIC_API_KEY` is inherited from the organization — nothing to add
|
|
190
|
-
unless a heal run fails to authenticate; then a repo-level secret of the
|
|
191
|
-
same name overrides it, see Phase 0).
|
|
192
|
-
If the scaffold just created the workflows, say so explicitly and list the
|
|
193
|
-
secrets — a merge without them fails CI and looks like a pipeline bug.
|
|
194
|
-
|
|
195
|
-
After merge, monitors update once the production deployment goes live — the
|
|
196
|
-
user verifies in the Checkly dashboard (https://app.checklyhq.com/); nothing
|
|
197
|
-
to run by hand. Manual fallback if a repo deleted the deploy workflow:
|
|
198
|
-
`pnpm checkly test --record`, then `pnpm checkly deploy` (requires the
|
|
199
|
-
Phase 4 login).
|
|
200
|
-
|
|
201
|
-
**Alert notifications are not automatic — walk the user through the
|
|
202
|
-
auto-heal webhook.** A failing monitor only shows in the Checkly dashboard
|
|
203
|
-
until an alert channel exists. The alert channel for these suites is a
|
|
204
|
-
**webhook into the auto-heal workflow**, not a direct Slack channel: the
|
|
205
|
-
team's Slack message is written by the pipeline *after* triage (regression
|
|
206
|
-
vs. stale test), so it carries a diagnosis and, for stale tests, the fix PR.
|
|
207
|
-
Set up once per project:
|
|
208
|
-
|
|
209
|
-
1. Confirm the `SLACK_WEBHOOK_URL` repository secret from Phase 0 exists —
|
|
210
|
-
the workflow escalates nothing without it. (`ANTHROPIC_API_KEY` comes
|
|
211
|
-
from the organization automatically — but if the dry run in step 6 ends
|
|
212
|
-
agent-failed with an authentication error, override it with a repo-level
|
|
213
|
-
secret of the same name, per Phase 0.)
|
|
214
|
-
2. The shared `GH_DISPATCH_TOKEN` secret exists in Checkly and covers this
|
|
215
|
-
repo. This is **one-time organization setup, not yours to perform**: the
|
|
216
|
-
token (`checkly-heal-dispatch`, a fine-grained PAT) is created once and
|
|
217
|
-
maintained by its owner, named in 1Password — see "One-time organization
|
|
218
|
-
setup" in the usage guide
|
|
219
|
-
(https://app.notion.com/p/32df837b627080748719c0f94a3aeaaf). Your only
|
|
220
|
-
per-project concern: if this repo is not yet on the token's repository
|
|
221
|
-
list, the webhook channel's test (step 3) fails with a 404 — then ask
|
|
222
|
-
the token owner to widen it (a one-minute edit on their side; the token
|
|
223
|
-
value doesn't change). Do not walk the user through creating a PAT or a
|
|
224
|
-
Checkly secret.
|
|
225
|
-
3. Checkly → Alert channels → "Add more channels" → **Webhook**:
|
|
226
|
-
- Method `POST`, URL
|
|
227
|
-
`https://api.github.com/repos/<org>/<repo>/dispatches`
|
|
228
|
-
- Headers: `Authorization: Bearer {{GH_DISPATCH_TOKEN}}` and
|
|
229
|
-
`Accept: application/vnd.github+json`
|
|
230
|
-
- Body (Checkly's editor lists the available variables — confirm the
|
|
231
|
-
names there):
|
|
232
|
-
|
|
233
|
-
```json
|
|
234
|
-
{
|
|
235
|
-
"event_type": "checkly-alert",
|
|
236
|
-
"client_payload": {
|
|
237
|
-
"check_name": "{{CHECK_NAME}}",
|
|
238
|
-
"check_id": "{{CHECK_ID}}",
|
|
239
|
-
"result_id": "{{CHECK_RESULT_ID}}",
|
|
240
|
-
"result_link": "{{RESULT_LINK}}",
|
|
241
|
-
"alert_type": "{{ALERT_TYPE}}"
|
|
242
|
-
}
|
|
243
|
-
}
|
|
244
|
-
```
|
|
245
|
-
|
|
246
|
-
- Notify on **failure**; enable **reminders** while a check stays failing
|
|
247
|
-
— a reminder re-fires the webhook, the workflow dedupes repeats, and a
|
|
248
|
-
lost first delivery gets retried by the next one. Recovery/degraded
|
|
249
|
-
notifications are ignored by the workflow either way.
|
|
250
|
-
4. Subscriptions: **only the current check**, and **uncheck** "Automatically
|
|
251
|
-
subscribe newly created checks to this alert channel".
|
|
252
|
-
5. **Do not also subscribe a Slack alert channel to these suites.** The
|
|
253
|
-
pipeline is the single alert path; a parallel raw Checkly alert double-pings
|
|
254
|
-
the channel and trains people to ignore one of them.
|
|
255
|
-
6. **One Slack Workflow Builder rule** routes this project's messages from
|
|
256
|
-
the shared e2e channel to the project channel. All notifications post to
|
|
257
|
-
the e2e channel tagged `[e2e:<repo-name>]`; the rule matches the **full
|
|
258
|
-
bracketed tag** — never a short name like `gbk`, which collides with
|
|
259
|
-
other text — and sends the message to the project channel. The sent
|
|
260
|
-
message's template is just the message-text variable (plus the
|
|
261
|
-
link-to-original variable when available, so threads happen on the
|
|
262
|
-
original) — no static prefix text. Have the user add a **second
|
|
263
|
-
manager** to the rule and record it in the rule registry in the e2e
|
|
264
|
-
channel's canvas.
|
|
265
|
-
7. Dry run: GitHub → Actions → "Checkly auto-heal" → Run workflow, with the
|
|
266
|
-
suite's check name. Confirm the run completes and the message shows up in
|
|
267
|
-
the e2e channel and, via the rule, in the project channel (a passing
|
|
268
|
-
check should end as "not reproducible" — summary only, no message — so a
|
|
269
|
-
real dry run is easiest right after any genuine failure).
|
|
270
|
-
|
|
271
|
-
What the user should expect afterwards: a failing monitor produces either a
|
|
272
|
-
`checkly-heal` PR plus a Slack note (stale or flaky test — **review
|
|
273
|
-
required, the PR is never merged automatically**; PRs carrying app-side test
|
|
274
|
-
hooks say so and point at their preview run for validation), or a Slack
|
|
275
|
-
escalation (regression, fix needs a behavioral app change, agent failed). A
|
|
276
|
-
transient blip that doesn't reproduce stays silent by design.
|
|
277
|
-
|
|
278
|
-
## Conventions to uphold
|
|
279
|
-
|
|
280
|
-
The generator and healer enforce spec-level rules themselves. At the pipeline
|
|
281
|
-
level, additionally keep to:
|
|
282
|
-
|
|
283
|
-
- One spec file per feature group; one `describe`, multiple `test()` blocks
|
|
284
|
-
- Commit specs, plans, seeds, **and** any `data-testid` source edits together
|
|
285
|
-
- Never commit with locally failing tests — heal first or `test.fixme()` with
|
|
286
|
-
the user's sign-off
|