@andwhy/e2e-pipeline 0.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +106 -0
- package/dist/bin/init.d.ts +2 -0
- package/dist/bin/init.js +128 -0
- package/dist/utils/copy.d.ts +23 -0
- package/dist/utils/copy.js +249 -0
- package/dist/utils/format.d.ts +1 -0
- package/dist/utils/format.js +49 -0
- package/dist/utils/install.d.ts +2 -0
- package/dist/utils/install.js +54 -0
- package/dist/utils/prompt.d.ts +2 -0
- package/dist/utils/prompt.js +17 -0
- package/package.json +37 -0
- package/templates/claude/agents/playwright-test-generator.md +184 -0
- package/templates/claude/agents/playwright-test-healer.md +70 -0
- package/templates/claude/agents/playwright-test-planner.md +135 -0
- package/templates/claude/partials/_robustness-core.md +69 -0
- package/templates/claude/rules/playwright-workflow.md +203 -0
- package/templates/claude/skills/playwright-test-pipeline/SKILL.md +201 -0
- package/templates/github/scripts/playwright-pr-comment.mjs +95 -0
- package/templates/github/workflows/checkly-deploy.yml +64 -0
- package/templates/github/workflows/pr-playwright-tests.yml +154 -0
- package/templates/shared/checkly.config.template.mjs +58 -0
- package/templates/shared/checkly.config.template.ts +58 -0
- package/templates/shared/playwright.config.template.mjs +44 -0
- package/templates/shared/playwright.config.template.ts +44 -0
- package/templates/shared/playwright.mcp.config.template.mjs +12 -0
- package/templates/shared/playwright.mcp.config.template.ts +12 -0
- package/templates/shared/preview-consent-setup.template.js +32 -0
- package/templates/shared/preview-consent-setup.template.ts +33 -0
- package/templates/shared/seed.spec.js +5 -0
- package/templates/shared/seed.spec.ts +5 -0
|
@@ -0,0 +1,203 @@
|
|
|
1
|
+
# Playwright Agent Workflow
|
|
2
|
+
|
|
3
|
+
> Setup and day-to-day workflow docs (for humans): https://app.notion.com/p/31ff837b6270805d8922c4a7c7beb6b0
|
|
4
|
+
|
|
5
|
+
When the user says "Use the playwright-test-planner/generator/healer agent", always delegate via the Agent tool with the matching subagent_type (`playwright-test-planner`, `playwright-test-generator`, `playwright-test-healer`). Never attempt to call the MCP tools directly.
|
|
6
|
+
|
|
7
|
+
**Do not implement Playwright specs in the orchestrating turn.** Writing or editing `*.spec.js` test bodies yourself bypasses replay, generator MCP tools, and project rules. For generation, always use the **Agent** tool with `subagent_type: "playwright-test-generator"`.
|
|
8
|
+
|
|
9
|
+
**Plans:** Read `tests/e2e/<feature>/<name>.plan.md` yourself and pass its contents in the Agent prompt. **Never ask the user to paste the plan** into chat.
|
|
10
|
+
|
|
11
|
+
## Agents
|
|
12
|
+
|
|
13
|
+
| Trigger phrase in user prompt | `subagent_type` | Purpose |
|
|
14
|
+
| ---------------------------------------------- | --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
|
|
15
|
+
| "Use the playwright-test-planner agent to …" | `playwright-test-planner` | Explores the live site, discovers real selectors/DOM, writes a `tests/e2e/<name>.plan.md` |
|
|
16
|
+
| "Use the playwright-test-generator agent to …" | `playwright-test-generator` | Reads a plan, replays steps in a browser, writes or extends `tests/e2e/<group>.spec.js` or `.spec.ts` (**Project test rules** §7–8) |
|
|
17
|
+
| "Use the playwright-test-healer agent to …" | `playwright-test-healer` | Runs tests, debugs failures, fixes code, re-runs until green |
|
|
18
|
+
|
|
19
|
+
## Project test rules
|
|
20
|
+
|
|
21
|
+
These rules apply to **all** agents (generator, healer, and orchestrator). Include them in every Agent prompt to the generator or healer.
|
|
22
|
+
|
|
23
|
+
1. Every logical step MUST have a numbered comment above it
|
|
24
|
+
(e.g. `// 1. Navigate to the homepage`). Preserve existing step comments
|
|
25
|
+
when editing. Add them if missing.
|
|
26
|
+
|
|
27
|
+
2. Never hardcode domains. Use `page.goto("/")` and derive the origin from the
|
|
28
|
+
live page URL.
|
|
29
|
+
|
|
30
|
+
3. Every `test.describe` block MUST include a `beforeEach` that runs shared preview setup
|
|
31
|
+
from `tests/e2e/helpers/preview-consent-setup.js` or `.ts` (`prepareVercelPreviewPage`, and when tests
|
|
32
|
+
start after consent, `gotoPathAndAcceptCookies(page, baseURL, path)` with the correct
|
|
33
|
+
relative path). The scaffold ships `prepareVercelPreviewPage` — it exchanges
|
|
34
|
+
`VERCEL_AUTOMATION_BYPASS_SECRET` for Vercel's deployment-protection bypass cookie on the
|
|
35
|
+
first navigation and is a plain `goto` when the variable is unset. Extend the file with
|
|
36
|
+
project consent logic; do not reinvent it. The helper file extension must match rule 7.
|
|
37
|
+
|
|
38
|
+
4. Never hardcode a single locale's URL pattern. Use a locale config map keyed
|
|
39
|
+
by bare hostname and resolve at runtime.
|
|
40
|
+
|
|
41
|
+
5. After all tests pass locally, verify with
|
|
42
|
+
`BASE_URL=<dev-url> pnpm playwright test` via the terminal tool. If failures
|
|
43
|
+
occur, fix and re-run until green.
|
|
44
|
+
|
|
45
|
+
**Local runs always pin `BASE_URL` on the command line** to the dev server
|
|
46
|
+
confirmed during setup (usually `http://localhost:3000`) — never rely on the
|
|
47
|
+
ambient environment. Repos commonly export `BASE_URL` for the app's own
|
|
48
|
+
purposes, pointed at the deployed site; an unpinned run silently targets
|
|
49
|
+
production, which does not contain the working tree's changes (the feature
|
|
50
|
+
under test, any `data-testid` additions). Deployed URLs enter the pipeline
|
|
51
|
+
only at preview validation and Checkly, always passed explicitly.
|
|
52
|
+
|
|
53
|
+
6. **Test root:** Paths in this file assume the default `tests/e2e/`, which keeps e2e specs
|
|
54
|
+
clear of unit tests in the same repo. **The repo wins over this default.** Check
|
|
55
|
+
`playwright.config.{ts,mjs}` for `testDir` and list existing `*.spec.*`; if the **e2e**
|
|
56
|
+
specs already live somewhere else (`tests/` flat, `e2e/`, `src/__tests__/`), write there
|
|
57
|
+
instead and read every `tests/e2e/…` path below as that directory. Never split a suite
|
|
58
|
+
across two roots. A non-e2e suite under `tests/` (unit, a11y) is **not** that signal:
|
|
59
|
+
leave `testDir` on the e2e root and give that suite its own Playwright project with its
|
|
60
|
+
own `testDir` — never widen `testDir` to cover both, which runs the other suite in every
|
|
61
|
+
browser project.
|
|
62
|
+
|
|
63
|
+
7. **Spec language (generators only):** Repositories differ — some teams use **JavaScript**
|
|
64
|
+
for Playwright tests (`.spec.js`), others use **TypeScript** (`.spec.ts`). List
|
|
65
|
+
`tests/e2e/**/*.spec.*` and match the prevailing extension. If there are no specs yet, default
|
|
66
|
+
to `.spec.js` unless the project README or the user specifies TypeScript for tests. Helpers
|
|
67
|
+
and seeds must use the same language as specs. Do not add `.spec.ts` in a repo that already
|
|
68
|
+
uses `.spec.js` everywhere (and vice versa).
|
|
69
|
+
|
|
70
|
+
8. **Feature groups (target layout):** Prefer **one spec file per feature group** (e.g.
|
|
71
|
+
`tests/e2e/map/map.spec.js`) with a single `test.describe('<Group name>', …)` and **multiple**
|
|
72
|
+
`test()` blocks — one per plan scenario — sharing one `beforeEach`. Do not default to one
|
|
73
|
+
file per scenario unless scenarios are unrelated. Prefer **lean** tests: assert user-visible
|
|
74
|
+
outcomes and regressions, not every intermediate DOM detail from the plan. Numbered
|
|
75
|
+
comments on **main** user steps only.
|
|
76
|
+
|
|
77
|
+
## Pipeline: Plan → Generate → Heal
|
|
78
|
+
|
|
79
|
+
### Step 1 — Plan
|
|
80
|
+
|
|
81
|
+
Trigger when: user wants to create tests for a new feature, page, or user flow.
|
|
82
|
+
|
|
83
|
+
1. Use the **Agent** tool with `subagent_type: "playwright-test-planner"`
|
|
84
|
+
2. Pass the URL and user flow description in the `prompt`
|
|
85
|
+
3. The planner saves output to `tests/e2e/<name>.plan.md`
|
|
86
|
+
4. When the subagent returns, present the plan to the user for review before proceeding
|
|
87
|
+
|
|
88
|
+
Prompt to forward:
|
|
89
|
+
|
|
90
|
+
```
|
|
91
|
+
Create a test plan for: <FEATURE NAME>
|
|
92
|
+
|
|
93
|
+
Navigate to <URL> and explore the following user flow:
|
|
94
|
+
<USER'S FLOW STEPS>
|
|
95
|
+
|
|
96
|
+
While exploring, verify:
|
|
97
|
+
- Actual selectors (see **Locator strategy** under *Robustness patterns* below)
|
|
98
|
+
- Whether cookie consent or login gates appear
|
|
99
|
+
- DOM structure of key interactive elements
|
|
100
|
+
- If the suite targets **multiple domains**, explore **each** domain the tests will run against
|
|
101
|
+
(see *Multi-domain / multi-locale testing* — the plan must reflect every distinct host or market)
|
|
102
|
+
|
|
103
|
+
Generate a seed file at tests/e2e/<feature>/seed.spec.js or seed.spec.ts (same extension as
|
|
104
|
+
the rest of this repo's specs — see Project test rules §7) that handles the common setup
|
|
105
|
+
(navigation, cookie/auth consent, waiting for key elements) discovered during exploration.
|
|
106
|
+
|
|
107
|
+
Save the plan to tests/e2e/<feature>/<name>.plan.md using planner_save_plan.
|
|
108
|
+
All URLs must use relative paths — never localhost or absolute domains.
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
### Step 2 — Generate
|
|
112
|
+
|
|
113
|
+
Trigger when: a `tests/e2e/<name>.plan.md` exists and the user wants generated test code.
|
|
114
|
+
|
|
115
|
+
**You (the orchestrating agent) own this step end-to-end.** The human does **not** need to
|
|
116
|
+
invoke the generator separately for each scenario or plan section — you chain **Agent** calls
|
|
117
|
+
to `playwright-test-generator` until the feature group is implemented (see **Important rules**).
|
|
118
|
+
|
|
119
|
+
1. Read `tests/e2e/<name>.plan.md` to get the full plan content
|
|
120
|
+
2. Use the **Agent** tool with `subagent_type: "playwright-test-generator"` — **once per scenario**
|
|
121
|
+
if building incrementally, **or** pass the **full plan** and all scenarios for the group in one
|
|
122
|
+
prompt if the generator can emit the whole file (**Project test rules** §8)
|
|
123
|
+
3. Pass the **plan content** in the `prompt` (the generator subagent cannot read arbitrary paths from disk)
|
|
124
|
+
4. Include the **project test rules** above in the prompt
|
|
125
|
+
5. If the plan contains a "Recommended data-testid additions" table, include it — the generator adds those attributes before replaying steps
|
|
126
|
+
6. Target **one spec file per feature group** (e.g. `tests/e2e/map/map.spec.js`) with a shared
|
|
127
|
+
`test.describe` and **multiple** `test()` blocks — merge with `edit` between runs if needed
|
|
128
|
+
|
|
129
|
+
Prompt to forward (example — adjust per repo):
|
|
130
|
+
|
|
131
|
+
```
|
|
132
|
+
Implement tests from the plan for this feature group. Target layout: one file, one describe, multiple tests (see project test rules §8).
|
|
133
|
+
|
|
134
|
+
- Group / describe: "<TOP-LEVEL PLAN HEADING e.g. Map>"
|
|
135
|
+
- File: tests/e2e/<feature>/<group>.spec.js (or `.spec.ts` per project)
|
|
136
|
+
- Seed: <SEED FILE PATH FROM PLAN>
|
|
137
|
+
- Viewport: <WIDTH>x<HEIGHT>
|
|
138
|
+
|
|
139
|
+
<plan>
|
|
140
|
+
<PASTE full plan OR the scenario block(s) you are implementing this run>
|
|
141
|
+
</plan>
|
|
142
|
+
|
|
143
|
+
If recommended data-testid additions exist, add them to source components before replaying steps.
|
|
144
|
+
|
|
145
|
+
After the spec is written or updated, run
|
|
146
|
+
BASE_URL=<dev-url> pnpm playwright test tests/e2e/<feature>/ (pin the dev server per rule 5).
|
|
147
|
+
```
|
|
148
|
+
|
|
149
|
+
### Step 3 — Heal
|
|
150
|
+
|
|
151
|
+
Trigger when: tests are failing (after generation, after site changes, or on user request).
|
|
152
|
+
|
|
153
|
+
1. Use the **Agent** tool with `subagent_type: "playwright-test-healer"`
|
|
154
|
+
2. Pass the test file path in the `prompt`
|
|
155
|
+
3. Include the **project test rules** above in the prompt so the healer follows them
|
|
156
|
+
|
|
157
|
+
Prompt to forward (always include the rules block):
|
|
158
|
+
|
|
159
|
+
```
|
|
160
|
+
Run and fix the tests in tests/e2e/<name>.spec.js (or `.ts` per project)
|
|
161
|
+
|
|
162
|
+
If any tests fail:
|
|
163
|
+
|
|
164
|
+
- Debug each failure
|
|
165
|
+
- Examine the page state, selectors, and assertions
|
|
166
|
+
- Fix the test code
|
|
167
|
+
- Re-run until all tests pass locally
|
|
168
|
+
|
|
169
|
+
## Project test rules (follow these strictly)
|
|
170
|
+
|
|
171
|
+
<COPY the Project test rules section above into this prompt>
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
## Important rules
|
|
175
|
+
|
|
176
|
+
- **Always delegate via the Agent tool.** Never call `playwright-test/*` tools directly.
|
|
177
|
+
- **Always plan before generating.** Use the planner to discover real selectors first.
|
|
178
|
+
- **Always read the plan file** before passing to the generator. The generator runs in
|
|
179
|
+
isolation — include the plan or scenario steps in the prompt.
|
|
180
|
+
- **You implement the full generate phase.** After plan approval, **you** invoke the
|
|
181
|
+
generator subagent **as many times as needed** (typically once per scenario, sequentially)
|
|
182
|
+
or supply the full multi-scenario prompt — **without** asking the user to run the generator
|
|
183
|
+
for each plan section. A single user request such as "Implement the tests from this plan" is enough.
|
|
184
|
+
- **Run generator subagent calls sequentially.** Never parallel — the MCP browser server is shared.
|
|
185
|
+
- **Do not interrupt.** Sending a message while a generator is running aborts it.
|
|
186
|
+
Wait for completion before proceeding.
|
|
187
|
+
- **data-testid additions are owned by the generator.** When the plan includes a
|
|
188
|
+
"Recommended data-testid additions" table, the generator subagent adds those
|
|
189
|
+
attributes to source components before replaying steps. The orchestrator must
|
|
190
|
+
forward the table verbatim in the generator prompt and must NOT edit source
|
|
191
|
+
components itself.
|
|
192
|
+
- **Present the plan for review** after planning. Unless the user narrows scope, **proceed
|
|
193
|
+
to generate all scenarios** for the agreed feature group (see **Project test rules** §8).
|
|
194
|
+
- **Generator and healer self-verify locally** with
|
|
195
|
+
`BASE_URL=<dev-url> pnpm playwright test` (pinned per **Project test rules** §5).
|
|
196
|
+
Cloud validation via `checkly pw-test` is a separate human step after pushing.
|
|
197
|
+
Only invoke the healer if a generated test fails locally.
|
|
198
|
+
- **Base URL**: local runs pin `BASE_URL=<dev-url>` on the command (**Project
|
|
199
|
+
test rules** §5); the config's localhost fallback is a safety net, not a
|
|
200
|
+
guarantee, because an exported `BASE_URL` overrides it silently.
|
|
201
|
+
- **Plan files** are saved at `tests/e2e/<feature>/<name>.plan.md`.
|
|
202
|
+
- **Test files** follow **Project test rules** §7–8 — typically `tests/e2e/<feature>/<group>.spec.js`
|
|
203
|
+
or `.spec.ts` with one `describe` and multiple tests.
|
|
@@ -0,0 +1,201 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: playwright-test-pipeline
|
|
3
|
+
description: Guides the full workflow for creating and maintaining Playwright
|
|
4
|
+
end-to-end tests with the planner/generator/healer agents — setup check,
|
|
5
|
+
Plan → Generate → Heal, local runs, Checkly preview validation, and
|
|
6
|
+
deployment. Use when the user wants to create, extend, or fix e2e/Playwright
|
|
7
|
+
tests, asks to "test" a page/feature/flow, or asks how the test pipeline
|
|
8
|
+
works.
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
# Playwright Test Pipeline
|
|
12
|
+
|
|
13
|
+
You are guiding a person through this project's test pipeline. They may not
|
|
14
|
+
know it exists — narrate briefly at each phase, and pause at the marked
|
|
15
|
+
decision points. The delegation rules and prompt templates live in
|
|
16
|
+
`.claude/rules/playwright-workflow.md` (already in your context); follow them,
|
|
17
|
+
do not improvise your own prompts to the agents.
|
|
18
|
+
|
|
19
|
+
## Phase 0 — Setup (always first)
|
|
20
|
+
|
|
21
|
+
Verify, in order:
|
|
22
|
+
|
|
23
|
+
1. `.claude/agents/playwright-test-planner.md` (and generator/healer) exist
|
|
24
|
+
2. `.mcp.json` at project root defines the `playwright-test` server, pointed
|
|
25
|
+
at `playwright.mcp.config.{ts,mjs}` (which must exist — it lets the MCP run
|
|
26
|
+
seeds that `testIgnore` hides from normal runs)
|
|
27
|
+
3. A `playwright.config.{ts,mjs}` and `checkly.config.{ts,mjs}` exist
|
|
28
|
+
|
|
29
|
+
### If anything is missing, install it — do not just bounce the user
|
|
30
|
+
|
|
31
|
+
Say what is missing, then offer to run the scaffold. Do not run it without
|
|
32
|
+
agreement: it installs dependencies and writes files.
|
|
33
|
+
|
|
34
|
+
**Always pass `--format` explicitly.** Without it the CLI opens an interactive
|
|
35
|
+
prompt, and in a non-interactive shell that prompt resolves empty and the CLI
|
|
36
|
+
exits silently having done nothing — it looks like it worked. Pick the format
|
|
37
|
+
from the repo: `ts` when there is a `tsconfig.json`, otherwise `mjs`.
|
|
38
|
+
|
|
39
|
+
```bash
|
|
40
|
+
pnpm dlx @andwhy/e2e-pipeline init --format ts # or --format mjs
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
Add `--force` only when updating an existing install. It regenerates
|
|
44
|
+
`.claude/**` and `.mcp.json` and never touches the configs.
|
|
45
|
+
|
|
46
|
+
Then read the output back to the user: what was created, what was skipped.
|
|
47
|
+
Docs: https://app.notion.com/p/31ff837b6270805d8922c4a7c7beb6b0
|
|
48
|
+
|
|
49
|
+
**After a mid-session install, a restart is required — this is a hard stop.**
|
|
50
|
+
Claude Code registers subagents and `.mcp.json` servers at session start, so
|
|
51
|
+
the planner/generator/healer and the `playwright-test` browser tools do not
|
|
52
|
+
exist in the session that just scaffolded them. Do NOT work around this:
|
|
53
|
+
no inlining an agent definition into a general-purpose agent, no hand-writing
|
|
54
|
+
plans or specs from source code — without the browser tools that produces
|
|
55
|
+
unverified selectors, which is exactly what this pipeline exists to prevent.
|
|
56
|
+
Finish Phase 0 (configs, reachable URL), then tell the user to restart the
|
|
57
|
+
session and re-ask; Phase 0 will pass and planning starts for real.
|
|
58
|
+
|
|
59
|
+
### Then finish the configuration before planning
|
|
60
|
+
|
|
61
|
+
The scaffold writes templates, not a working setup. Walk these with the user:
|
|
62
|
+
|
|
63
|
+
1. `playwright.config` — `baseURL` for local runs.
|
|
64
|
+
2. `checkly.config` — real `projectName` and `logicalId`, the production
|
|
65
|
+
domain(s) in `environmentVariables`, `testCommand` (see Phase 5), and the
|
|
66
|
+
check-suite `name` — that is what teammates see in the Checkly dashboard
|
|
67
|
+
and in alerts. **Ask the user what to call it** unless the repo makes it
|
|
68
|
+
obvious (a single clear brand or product name); do not silently invent
|
|
69
|
+
one. Also set
|
|
70
|
+
`installCommand: "pnpm install --ignore-scripts"` — Checkly's runner gets a
|
|
71
|
+
bundle of test files, not the repo, so a project postinstall (codegen etc.)
|
|
72
|
+
fails there on missing files. Placeholder values (`my-app`, `example.com`)
|
|
73
|
+
left in place will block Phase 4; local work is unaffected.
|
|
74
|
+
3. A reachable URL for the planner: a running dev server or a deployed
|
|
75
|
+
environment. **Confirm this before Phase 1** — the planner explores a live
|
|
76
|
+
page, and pointing it at an empty or unreachable app produces a plan that
|
|
77
|
+
asserts nothing useful.
|
|
78
|
+
4. **Repository secrets for the workflows**: if the scaffold just created
|
|
79
|
+
`.github/workflows/`, tell the user now — not at merge time — to add three
|
|
80
|
+
repository secrets (GitHub → Settings → Secrets and variables → Actions).
|
|
81
|
+
You cannot add them; a person must. Without them the first PR push and the
|
|
82
|
+
first merge fail CI in ways that look like pipeline bugs.
|
|
83
|
+
- `CHECKLY_API_KEY` — 1Password ("Checkly") first; else Checkly → User
|
|
84
|
+
Settings → API keys. The key is shown once, at creation — tell the user
|
|
85
|
+
to save it to 1Password immediately.
|
|
86
|
+
- `CHECKLY_ACCOUNT_ID` — Checkly → Account Settings → General; also in
|
|
87
|
+
1Password.
|
|
88
|
+
- `VERCEL_AUTOMATION_BYPASS_SECRET` — Vercel → project → Settings →
|
|
89
|
+
Deployment Protection → Protection Bypass for Automation. Per project —
|
|
90
|
+
never copied from another repo.
|
|
91
|
+
5. **Existing tests under `tests/`** (unit, a11y — the scaffold lists them as
|
|
92
|
+
"outside tests/e2e"): leave `testDir` at `./tests/e2e` and give that suite
|
|
93
|
+
its own Playwright project with its own `testDir` if it should run under
|
|
94
|
+
Playwright at all. **Never widen `testDir` to `./tests`** — that runs the
|
|
95
|
+
other suite in every browser project and mixes the suites. Checkly is
|
|
96
|
+
pinned to `tests/e2e` by `testCommand` either way.
|
|
97
|
+
|
|
98
|
+
## Phase 1 — Plan
|
|
99
|
+
|
|
100
|
+
1. Ask, if not already clear: which page/flow, which URL, and whether the
|
|
101
|
+
suite targets multiple domains (check `checkly.config` for the list).
|
|
102
|
+
2. Delegate to `playwright-test-planner` using the rules-file template.
|
|
103
|
+
3. **Decision point:** present the saved plan (`tests/e2e/<feature>/<name>.plan.md`)
|
|
104
|
+
scenario-by-scenario and ask what to keep, drop, or adjust. Do not
|
|
105
|
+
generate before the user responds.
|
|
106
|
+
|
|
107
|
+
## Phase 2 — Generate
|
|
108
|
+
|
|
109
|
+
1. Read the plan file yourself; never ask the user to paste it.
|
|
110
|
+
2. Delegate to `playwright-test-generator` per the rules file (sequential
|
|
111
|
+
calls, project test rules included, data-testid table forwarded verbatim).
|
|
112
|
+
3. Tell the user which spec file is being written and when each scenario lands.
|
|
113
|
+
|
|
114
|
+
## Phase 3 — Run and heal
|
|
115
|
+
|
|
116
|
+
1. Run `BASE_URL=<dev-url> pnpm playwright test tests/e2e/<feature>/`, pinning
|
|
117
|
+
the dev server confirmed in Phase 0. Phases 1–3 always target local: the
|
|
118
|
+
working tree's changes (the feature, any `data-testid` additions) exist
|
|
119
|
+
nowhere else, and an ambient `BASE_URL` export would silently point the run
|
|
120
|
+
at the deployed site. Deployed URLs enter only at Phase 4, always explicit.
|
|
121
|
+
2. Green → Phase 4. Red → delegate to `playwright-test-healer` per the rules
|
|
122
|
+
file. If the healer marks anything `test.fixme()`, surface it explicitly —
|
|
123
|
+
the user decides whether that is acceptable.
|
|
124
|
+
|
|
125
|
+
## Phase 4 — Preview validation (before merge)
|
|
126
|
+
|
|
127
|
+
1. Auth check (first authenticated command in the pipeline): run
|
|
128
|
+
`pnpm checkly whoami`. If not logged in, have the user run
|
|
129
|
+
`pnpm checkly login` — it opens a browser flow, and the shared team
|
|
130
|
+
credentials are in 1Password (search "Checkly"). Local Playwright runs
|
|
131
|
+
never need this, and CI does not either — it uses the `CHECKLY_API_KEY` /
|
|
132
|
+
`CHECKLY_ACCOUNT_ID` repository secrets.
|
|
133
|
+
2. **Decision point:** ask for the preview deployment URL, and settle how
|
|
134
|
+
Checkly's runners get through Vercel deployment protection. Preferred: the
|
|
135
|
+
project's automation bypass secret (Vercel project settings → Deployment
|
|
136
|
+
Protection; value in 1Password) — the scaffolded helper exchanges it for
|
|
137
|
+
Vercel's bypass cookie on first navigation. Fallback when the project has
|
|
138
|
+
no secret: a share link ("Anyone with the link") as the preview URL.
|
|
139
|
+
3. Run: `pnpm checkly pw-test --env BASE_URL="<preview-url>" --env VERCEL_AUTOMATION_BYPASS_SECRET="<secret>" -- tests/e2e/<feature>/`
|
|
140
|
+
(omit the second `--env` on the share-link fallback). Never write the
|
|
141
|
+
secret into `checkly.config` — deployed monitors run against production
|
|
142
|
+
and do not need it.
|
|
143
|
+
4. Failures here but not locally are usually cloud-environment issues
|
|
144
|
+
(timeouts, consent gates) — heal with that framing, then re-run.
|
|
145
|
+
|
|
146
|
+
The user can skip this phase; note that cloud behavior then goes unverified
|
|
147
|
+
until after merge.
|
|
148
|
+
|
|
149
|
+
## Phase 5 — Merge and deployment
|
|
150
|
+
|
|
151
|
+
The scaffold ships two workflows (seeded once — existing ones are never
|
|
152
|
+
overwritten):
|
|
153
|
+
|
|
154
|
+
- `.github/workflows/pr-playwright-tests.yml` — runs the suite against the
|
|
155
|
+
PR's own Vercel preview on every push (`deployment_status` trigger, so the
|
|
156
|
+
preview is already live and contains the PR's changes), and comments the
|
|
157
|
+
result on the PR.
|
|
158
|
+
- `.github/workflows/checkly-deploy.yml` — when a **production deployment
|
|
159
|
+
succeeds** (`deployment_status` trigger), deploys the checks as Checkly
|
|
160
|
+
monitors. Not on merge: monitors must match what production serves, and a
|
|
161
|
+
merge that adds tests for new `data-testid`s would go red until the release
|
|
162
|
+
carrying them is live — with gated releases that gap can be days. A
|
|
163
|
+
`workflow_dispatch` fallback allows manual runs.
|
|
164
|
+
|
|
165
|
+
**Before the first merge, the repo needs secrets** (values in 1Password):
|
|
166
|
+
`CHECKLY_API_KEY` and `CHECKLY_ACCOUNT_ID` for the deploy,
|
|
167
|
+
`VERCEL_AUTOMATION_BYPASS_SECRET` for the PR run through deployment
|
|
168
|
+
protection. If the scaffold just created the workflows, say so explicitly and
|
|
169
|
+
list the three secrets — a merge without them fails CI and looks like a
|
|
170
|
+
pipeline bug.
|
|
171
|
+
|
|
172
|
+
After merge, monitors update once the production deployment goes live — the
|
|
173
|
+
user verifies in the Checkly dashboard (https://app.checklyhq.com/); nothing
|
|
174
|
+
to run by hand. Manual fallback if a repo deleted the deploy workflow:
|
|
175
|
+
`pnpm checkly test --record`, then `pnpm checkly deploy` (requires the
|
|
176
|
+
Phase 4 login).
|
|
177
|
+
|
|
178
|
+
**Alert notifications are not automatic — walk the user through it.** A
|
|
179
|
+
failing monitor only shows in the Checkly dashboard until an alert channel
|
|
180
|
+
exists, so nobody gets notified. In Checkly:
|
|
181
|
+
|
|
182
|
+
1. Alert channels → "Add more channels" → select **Slack**.
|
|
183
|
+
2. Channel configuration: the project's Slack notification channel
|
|
184
|
+
(`#…`). A dedicated test-notification channel may exist by now — ask
|
|
185
|
+
which channel to use rather than assuming.
|
|
186
|
+
3. Verify the subscriptions: **only the current check** subscribed, and
|
|
187
|
+
**uncheck** "Automatically subscribe newly created checks to this alert
|
|
188
|
+
channel".
|
|
189
|
+
4. Test the alert channel (button) and confirm the notification arrived in
|
|
190
|
+
Slack.
|
|
191
|
+
5. "Create alert Slack channel".
|
|
192
|
+
|
|
193
|
+
## Conventions to uphold
|
|
194
|
+
|
|
195
|
+
The generator and healer enforce spec-level rules themselves. At the pipeline
|
|
196
|
+
level, additionally keep to:
|
|
197
|
+
|
|
198
|
+
- One spec file per feature group; one `describe`, multiple `test()` blocks
|
|
199
|
+
- Commit specs, plans, seeds, **and** any `data-testid` source edits together
|
|
200
|
+
- Never commit with locally failing tests — heal first or `test.fixme()` with
|
|
201
|
+
the user's sign-off
|
|
@@ -0,0 +1,95 @@
|
|
|
1
|
+
// Renders a Playwright JSON report as a PR comment body on stdout.
|
|
2
|
+
import { readFileSync } from 'node:fs'
|
|
3
|
+
|
|
4
|
+
const MARKER = '<!-- playwright-preview-report -->'
|
|
5
|
+
const { PREVIEW_URL = '', RUN_URL = '' } = process.env
|
|
6
|
+
|
|
7
|
+
const truncate = (text, max) =>
|
|
8
|
+
text.length > max ? `${text.slice(0, max)}…` : text
|
|
9
|
+
|
|
10
|
+
// Strips the ANSI colour codes Playwright writes into JSON error messages.
|
|
11
|
+
const clean = text =>
|
|
12
|
+
String(text ?? '')
|
|
13
|
+
.replace(/\u001b\[[0-9;]*m/g, '')
|
|
14
|
+
.trim()
|
|
15
|
+
|
|
16
|
+
const collect = (suite, acc = []) => {
|
|
17
|
+
for (const spec of suite.specs ?? []) {
|
|
18
|
+
for (const test of spec.tests ?? []) {
|
|
19
|
+
const last = test.results?.[test.results.length - 1]
|
|
20
|
+
if (!last || ['passed', 'skipped'].includes(last.status)) continue
|
|
21
|
+
acc.push({
|
|
22
|
+
title: spec.title,
|
|
23
|
+
file: spec.file ?? suite.file,
|
|
24
|
+
line: spec.line,
|
|
25
|
+
status: last.status,
|
|
26
|
+
error: clean(last.error?.message ?? last.errors?.[0]?.message ?? '')
|
|
27
|
+
})
|
|
28
|
+
}
|
|
29
|
+
}
|
|
30
|
+
for (const child of suite.suites ?? []) collect(child, acc)
|
|
31
|
+
return acc
|
|
32
|
+
}
|
|
33
|
+
|
|
34
|
+
const render = report => {
|
|
35
|
+
const failures = (report.suites ?? []).flatMap(s => collect(s))
|
|
36
|
+
const { expected = 0, skipped = 0, flaky = 0 } = report.stats ?? {}
|
|
37
|
+
// Playwright reports a global-timeout abort as a top-level error.
|
|
38
|
+
const truncated = (report.errors ?? []).some(e =>
|
|
39
|
+
/^Timed out waiting .* for the/.test(clean(e.message))
|
|
40
|
+
)
|
|
41
|
+
|
|
42
|
+
const lines = [MARKER, '## Playwright on the preview', '']
|
|
43
|
+
|
|
44
|
+
if (!failures.length && !truncated) {
|
|
45
|
+
lines.push(`All tests passed. ${expected} passed, ${skipped} skipped.`)
|
|
46
|
+
} else {
|
|
47
|
+
lines.push(
|
|
48
|
+
`**${failures.length} failed**, ${expected} passed, ${skipped} skipped` +
|
|
49
|
+
(flaky ? `, ${flaky} flaky` : '') +
|
|
50
|
+
'.'
|
|
51
|
+
)
|
|
52
|
+
}
|
|
53
|
+
|
|
54
|
+
if (truncated) {
|
|
55
|
+
lines.push(
|
|
56
|
+
'',
|
|
57
|
+
'> The run hit its global timeout, so tests after the ones below never ran.'
|
|
58
|
+
)
|
|
59
|
+
}
|
|
60
|
+
|
|
61
|
+
for (const f of failures) {
|
|
62
|
+
lines.push('', `### \`${f.file}:${f.line}\` — ${f.title}`)
|
|
63
|
+
if (f.error) {
|
|
64
|
+
lines.push('', '```', truncate(f.error, 1200), '```')
|
|
65
|
+
}
|
|
66
|
+
}
|
|
67
|
+
|
|
68
|
+
lines.push('', '---', '')
|
|
69
|
+
if (PREVIEW_URL) lines.push(`Preview: ${PREVIEW_URL} `)
|
|
70
|
+
if (RUN_URL) lines.push(`[Full run and HTML report](${RUN_URL}) `)
|
|
71
|
+
lines.push('Chrome only. Firefox and Edge run on Checkly against production.')
|
|
72
|
+
|
|
73
|
+
return lines.join('\n')
|
|
74
|
+
}
|
|
75
|
+
|
|
76
|
+
let report
|
|
77
|
+
try {
|
|
78
|
+
report = JSON.parse(readFileSync(process.argv[2], 'utf8'))
|
|
79
|
+
} catch {
|
|
80
|
+
// No JSON at all means Playwright died before writing one.
|
|
81
|
+
console.log(
|
|
82
|
+
[
|
|
83
|
+
MARKER,
|
|
84
|
+
'## Playwright on the preview',
|
|
85
|
+
'',
|
|
86
|
+
'The run produced no report. It most likely crashed or was cancelled ' +
|
|
87
|
+
'before any test finished.',
|
|
88
|
+
'',
|
|
89
|
+
RUN_URL ? `[Full run](${RUN_URL})` : ''
|
|
90
|
+
].join('\n')
|
|
91
|
+
)
|
|
92
|
+
process.exit(0)
|
|
93
|
+
}
|
|
94
|
+
|
|
95
|
+
console.log(render(report))
|
|
@@ -0,0 +1,64 @@
|
|
|
1
|
+
name: Deploy Checkly checks
|
|
2
|
+
|
|
3
|
+
# Deploys the e2e suite as Checkly monitors when PRODUCTION deploys — not on
|
|
4
|
+
# merge. Monitors must match what production serves: a merge that adds tests
|
|
5
|
+
# for new data-testids goes red until the app deploy carrying those testids is
|
|
6
|
+
# live, and with gated releases that gap can be days. `deployment_status`
|
|
7
|
+
# fires when Vercel finishes the production deployment, so monitors update at
|
|
8
|
+
# the exact moment the code they were recorded against is reachable.
|
|
9
|
+
# `checkly deploy` is idempotent, so redeploying on every production deploy
|
|
10
|
+
# is harmless.
|
|
11
|
+
# Needs repo secrets: CHECKLY_API_KEY, CHECKLY_ACCOUNT_ID (1Password, "Checkly").
|
|
12
|
+
on:
|
|
13
|
+
deployment_status:
|
|
14
|
+
workflow_dispatch:
|
|
15
|
+
|
|
16
|
+
concurrency:
|
|
17
|
+
group: checkly-deploy
|
|
18
|
+
cancel-in-progress: false
|
|
19
|
+
|
|
20
|
+
jobs:
|
|
21
|
+
deploy:
|
|
22
|
+
if: >-
|
|
23
|
+
github.event_name == 'workflow_dispatch' ||
|
|
24
|
+
(github.event.deployment_status.state == 'success' &&
|
|
25
|
+
github.event.deployment.environment == 'Production')
|
|
26
|
+
runs-on: ubuntu-latest
|
|
27
|
+
timeout-minutes: 8
|
|
28
|
+
|
|
29
|
+
steps:
|
|
30
|
+
- name: Check out the deployed ref
|
|
31
|
+
uses: actions/checkout@v5
|
|
32
|
+
with:
|
|
33
|
+
ref: ${{ github.event.deployment.ref || github.ref }}
|
|
34
|
+
|
|
35
|
+
- name: Set up pnpm
|
|
36
|
+
uses: pnpm/action-setup@v4
|
|
37
|
+
|
|
38
|
+
- name: Set up Node
|
|
39
|
+
uses: actions/setup-node@v5
|
|
40
|
+
with:
|
|
41
|
+
node-version: 22
|
|
42
|
+
cache: pnpm
|
|
43
|
+
|
|
44
|
+
# --ignore-scripts: only the tests are bundled; project postinstall
|
|
45
|
+
# hooks are not needed and may want CMS tokens CI does not have.
|
|
46
|
+
- name: Install dependencies
|
|
47
|
+
run: pnpm install --frozen-lockfile --ignore-scripts
|
|
48
|
+
|
|
49
|
+
# --force skips the interactive confirmation prompt.
|
|
50
|
+
- name: Deploy checks to Checkly
|
|
51
|
+
env:
|
|
52
|
+
CHECKLY_API_KEY: ${{ secrets.CHECKLY_API_KEY }}
|
|
53
|
+
CHECKLY_ACCOUNT_ID: ${{ secrets.CHECKLY_ACCOUNT_ID }}
|
|
54
|
+
run: pnpm exec checkly deploy --force
|
|
55
|
+
|
|
56
|
+
- name: Write result to job summary
|
|
57
|
+
if: always()
|
|
58
|
+
run: |
|
|
59
|
+
{
|
|
60
|
+
echo "## Checkly deploy"
|
|
61
|
+
echo ""
|
|
62
|
+
echo "- Ref: \`${{ github.event.deployment.ref || github.ref }}\`"
|
|
63
|
+
echo "- Triggered by a successful production deployment"
|
|
64
|
+
} >> "$GITHUB_STEP_SUMMARY"
|