e2e-qa-pilot 0.0.0-stage → 0.4.5

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,48 @@
1
+ # Settings
2
+
3
+ `settings.md` holds the keys every project shares. What is specific to your repository (ticket id, branch and commit format, case tool, Figma, feature flags, other repositories) goes in `.e2e-qa-pilot/project.md`. See [Project file](project.md). A value there answers the matching key below, so the assistant does not ask for it.
4
+
5
+ The first `review_case` run reads `.e2e-qa-pilot/settings.md`. If a key it needs is empty, the assistant asks once in chat and writes your answers into that file. A case id needs `case_tool`, `fetch_case`, and `update_case`. Pasted case text does not. Both inputs need `figma` and `test_command`, and `read_figma` when `figma` is `on`. `png_dir` is required unless `## Test layout` in `project.md` names the screenshot directory. `branch_pattern` has a default.
6
+
7
+ The settings file is included in every tool result, so do not store tokens in it. Put how to fetch and update a case in the file, and keep tokens in your own MCP configuration. Do not paste a secret into the chat.
8
+
9
+ | Key | Meaning |
10
+ |-----|---------|
11
+ | `case_tool`, `fetch_case`, `update_case` | Where cases live and how to read and update one. Used only for a case id |
12
+ | `figma`, `read_figma` | `on` or `off`, and how to read a frame |
13
+ | `branch_pattern` | Branch name. `{title_slug}` is the case title in kebab-case and `{case_id}` is the case id, for a case id only. Default `e2e/{title_slug}`, or `e2e/{case_id}-{title_slug}` for a case id. A name with no placeholder is used as it is |
14
+ | `test_command`, `png_dir` | How to run the spec, and where PNGs go |
15
+ | `pr_mode` | `create_pr_for_review` (default) or `ask_before_pr`. A missing value means `create_pr_for_review` |
16
+ | `pr_label_ready` | Label when behavior matches and required screenshot evidence is adequate. Default `e2e:ready` |
17
+ | `pr_label_minor_text_diff` | Label when the only issue is a narrow wording difference. Default `e2e:minor-text-diff` |
18
+ | `pr_label_mismatch` | Label for a functional or unresolved mismatch, or inadequate required screenshot evidence. Default `e2e:mismatch` |
19
+
20
+ No Figma? Answer `off` and leave `read_figma` blank. `read_figma` is required only when `figma` is `on`. With `figma: off`, in `settings.md` or in `## Design` of `project.md`, leave it blank. The assistant does not ask for it.
21
+
22
+ ## Pull request outcomes
23
+
24
+ | Label | Color | Meaning |
25
+ |-------|-------|---------|
26
+ | `e2e:ready` | 🟢 green `#0E8A16` | Behavior matches and the required screenshots are adequate |
27
+ | `e2e:minor-text-diff` | 🟡 yellow `#FBCA04` | Behavior matches and the only issue is a narrow wording difference |
28
+ | `e2e:mismatch` | 🔴 red `#D73A4A` | A functional or unresolved mismatch, or inadequate screenshots. Draft pull request |
29
+
30
+ When a label is missing in your repository, the assistant creates it with these colors and descriptions. It does not change a label that already exists. The colors are not a setting, so a renamed label (`pr_label_*`) is created with the same color as the one it replaces.
31
+
32
+ When the assistant has the tools and permissions the commands name, the intended deliverables are a Playwright spec, a step-by-step results table, screenshot evidence, pull request comments, and a pull request for human QA review. A label is a review outcome. It does not mean the change is approved or merged.
33
+
34
+ A green test run means only that the assertions executed by that run passed. It does not prove that every expected behavior was tested or that the screenshots demonstrate every required state. A failed assertion and inadequate screenshot evidence are separate findings. Inadequate screenshot evidence does not, by itself, prove that a functional assertion failed.
35
+
36
+ - `e2e:ready` — functional assertions passed, the expected behavior matches, and all required screenshot evidence meets the acceptance criteria.
37
+ - `e2e:minor-text-diff` — functional behavior matches, and the only discrepancy is a minor wording difference that meets the documented classification rule. The difference is explicitly recorded. It is not reported as a passed exact-text assertion.
38
+ - `e2e:mismatch` — a functional mismatch or unresolved discrepancy exists, or required screenshot evidence fails to demonstrate the expected end state. The pull request is a draft. It is not a confirmed product regression.
39
+
40
+ Nothing is auto-merged. The default `pr_mode` is `create_pr_for_review`. It allows the assistant to create a pull request for review without auto-merging it. The optional `ask_before_pr` mode requires the assistant to stop and ask before creating a pull request when a functional mismatch, unresolved discrepancy, or failed final-screenshot gate remains. If the discrepancy remains after clarification, the assistant must obtain explicit authorization before creating the pull request.
41
+
42
+ ## Required inputs
43
+
44
+ The server requires exactly one of `case_id` or `case_text`. Pasted text needs no case tool. A case id needs `case_tool`, `fetch_case`, `update_case`, and an external case-tool MCP. Generating a spec needs `figma` and `test_command`, and `read_figma` when `figma` is `on`. It also needs `png_dir` unless `## Test layout` in `project.md` names the screenshot directory. `branch_pattern`, `pr_mode`, `pr_label_ready`, `pr_label_minor_text_diff`, and `pr_label_mismatch` have defaults, so a missing value is not a question. The input contract is in [mcp.md](../mcp.md).
45
+
46
+ ## Reference conventions
47
+
48
+ Reference conventions in the commands, which another project may adapt (set the locations in `## Test layout` of [`project.md`](project.md)), include `overview.md`, Gherkin under `specs/`, page contracts under `pages/`, page objects, flows, and a Playwright fixture. There is no settings key for those paths. A project does not have to copy that layout for the server to start. The [Cabana example](https://github.com/bogdanOwner/e2e-qa-pilot/blob/main/examples/boutique-cabin/README.md) uses it so the reference workflow can be read in one place. That directory is in this git repository and is not part of the published package. Figma helpers such as `attachMilestoneScreenshot` are not in the example and are not provided by this package.
package/mcp.md ADDED
@@ -0,0 +1,174 @@
1
+ # MCP
2
+
3
+ MCP is the connection between your project and the assistant you already use. `init` sets it up for Cursor and Claude Code, and both use the same four tools: `review_case`, `add_from_steps`, `add_from_design`, and `setup_project`. You ask in plain words. There is no new screen to learn.
4
+
5
+ Install steps, settings keys, and the trust statement are in [docs](docs/README.md). This page is the tool contract.
6
+
7
+ ## What the server does and does not do
8
+
9
+ The server is small on purpose: [src/mcp.mjs](src/mcp.mjs), one file, no dependencies. It returns text. It does not run the workflow, run tests, call a case tool, Figma, or GitHub, or write to your project. It makes no network requests.
10
+
11
+ Each tool returns:
12
+
13
+ - the input block: the case id, or the pasted case text, as the client sent it
14
+ - your `.e2e-qa-pilot/settings.md`, verbatim, or a notice that it is missing
15
+ - your `.e2e-qa-pilot/project.md`, verbatim, or a notice that it is missing. It is optional for a case run
16
+ - the command file and the shared rules, from the installed package
17
+
18
+ Your connected assistant then follows the instructions with whatever tools you have configured: repository access, a test runner, Playwright, your case tool when you passed a case id, Figma for design comparison, a feature-flag tool, the `gh` CLI for a read-only frontend repository, and GitHub for the pull request. Which of these a project uses is written in `project.md`. See [Project file](docs/project.md). Not every assistant or client has all of these. If one is missing, the step that needs it cannot be completed.
19
+
20
+ Because the settings file and the project file are returned in every result, keep tokens out of both. Keep them in your own MCP configuration.
21
+
22
+ ## Connect once
23
+
24
+ In your own project:
25
+
26
+ ```bash
27
+ npx e2e-qa-pilot init
28
+ ```
29
+
30
+ `init` writes the skills, an always-on instruction (a rule file for Cursor, a marked block in `CLAUDE.md` for Claude Code), and adds this entry. Other servers in the file are kept. An entry that is already listed is left as it is, and a file that is not valid JSON is left alone.
31
+
32
+ Cursor, `.cursor/mcp.json`:
33
+
34
+ ```json
35
+ {
36
+ "mcpServers": {
37
+ "e2e-qa-pilot": {
38
+ "command": "npx",
39
+ "args": ["-y", "e2e-qa-pilot", "mcp"]
40
+ }
41
+ }
42
+ }
43
+ ```
44
+
45
+ Claude Code, `.mcp.json` in the project root (same shape). To add it by hand instead, run the command below. It is not exercised by this repository's tests, so check `claude mcp add --help` for the current syntax:
46
+
47
+ ```bash
48
+ claude mcp add e2e-qa-pilot -- npx -y e2e-qa-pilot mcp
49
+ ```
50
+
51
+ Restart the editor. The server reads `.e2e-qa-pilot/settings.md` from the folder the client starts it in. That is normally your project root, but it depends on the client.
52
+
53
+ The entry is not pinned to a version, so `npx -y` uses the version npm resolves, or has cached. The command files that `init` copied into your project came from the version that ran `init`. The two can differ, as described in [Without MCP](#without-mcp). When they do, every tool result starts with a line that says to run `npx e2e-qa-pilot upgrade`. Pinning and upgrading are in [Install](docs/install.md#upgrade).
54
+
55
+ Also connect the MCP servers your project uses: a case tool when you review a case id, a design tool for design comparison, and a feature-flag tool. e2e-qa-pilot does not replace them, and pasted case text needs none. A frontend repository is read with the `gh` CLI after `gh auth login`. Which servers and repositories this project uses is written once in `.e2e-qa-pilot/project.md`.
56
+
57
+ ## Ask in plain words
58
+
59
+ Copy a phrase and change the case id, or paste the case.
60
+
61
+ | You want | Ask |
62
+ |----------|-----|
63
+ | Review a case id | Review case `<id>`. Use `.e2e-qa-pilot/settings.md`. |
64
+ | Review pasted case text | Paste the full case, including a `Title:` line, preconditions, steps, and expected results, and ask to review it. |
65
+ | Do everything for a case id | Implement case `<id>` with e2e-qa-pilot. |
66
+ | Automate a case that has no Figma link | Add the end-to-end test from the steps of case `<id>`, or from the pasted case. |
67
+ | Automate a case that has a Figma link | Case `<id>` has a Figma link. Compare the screen to the frame and stop if they differ. |
68
+
69
+ The assistant tells you what it is about to change. For a case id, it updates that same case only after you answer. It does not invent an expected result.
70
+
71
+ ## Tools
72
+
73
+ `review_case`, `add_from_steps`, and `add_from_design` take the same input object. `setup_project` takes no input and is described [below](#setup_project). For the three case tools, `required` is `[]`, because the handler enforces the rule below. Send exactly one of `case_id` or `case_text`, as a string with some text in it.
74
+
75
+ - `case_id`: the id of an existing case in the configured case tool. It needs a connected case tool and the `case_tool`, `fetch_case`, and `update_case` settings. A provider error or a missing case stops the turn. Only an id-based review updates a case.
76
+ - `case_text`: the complete pasted case, including a `Title:` line when the title is known. It needs no case tool and none of those three settings. The server does not fetch or update a case.
77
+
78
+ The server trims both values. A blank value counts as absent, so a blank `case_id` with a real `case_text` is a `case_text` call. These return a tool error (`isError: true`) and no instructions:
79
+
80
+ - both fields with text, or neither
81
+ - a value that is not a string
82
+ - arguments that are not an object
83
+ - any other key
84
+
85
+ An unknown tool name returns a JSON-RPC error instead of a tool result.
86
+
87
+ A call with pasted text:
88
+
89
+ ```json
90
+ {
91
+ "name": "review_case",
92
+ "arguments": {
93
+ "case_text": "Title: Open home\nPreconditions: a user is signed in.\n1. Open the home page. Expected: the home page is visible."
94
+ }
95
+ }
96
+ ```
97
+
98
+ A call that returns a tool error, because it sends both fields:
99
+
100
+ ```json
101
+ {
102
+ "name": "review_case",
103
+ "arguments": {
104
+ "case_id": "QA-123",
105
+ "case_text": "Title: Open home"
106
+ }
107
+ }
108
+ ```
109
+
110
+ ### Routing
111
+
112
+ `review_case` returns the instructions for the assistant to route. The server does not choose the next tool.
113
+
114
+ - A `figma.com` URL and `figma: on` in the settings go to `add_from_design`. For a case id, the URL is in the description or a custom field. For pasted text, it is anywhere in the text.
115
+ - No Figma URL, or `figma: off`, goes to `add_from_steps`.
116
+ - `add_from_design` also needs `read_figma` to work, so your assistant must be able to read the Figma frame. `read_figma` is not needed with `figma: off`. It can come from `settings.md` or from `## Design` in `project.md`.
117
+
118
+ The assistant may run the next command by calling its MCP tool or by reading the installed command file.
119
+
120
+ ### review_case
121
+
122
+ - Spec: [processes/create-e2e/commands/review-case/SKILL.md](processes/create-e2e/commands/review-case/SKILL.md)
123
+ - Result: the case is ready and routed, or one question per hole was asked and, for a case id, that same case was updated
124
+ - If the case is ready, the instructions tell the assistant to continue into the routed command in the same turn. That can end in a pull request. See [Known limitations](#known-limitations)
125
+ - The assistant still asks, and does not invent an expected result
126
+
127
+ ### add_from_steps
128
+
129
+ - Spec: [processes/create-e2e/commands/add-from-steps/SKILL.md](processes/create-e2e/commands/add-from-steps/SKILL.md)
130
+ - Result: one label, `e2e:ready`, `e2e:minor-text-diff`, or a draft `e2e:mismatch`, unless `ask_before_pr` is still waiting for an answer or for explicit authorization
131
+ - The assistant still reads the screen. A narrow minor text difference does not stop the run. `pr_mode` is read from settings by the assistant. The server does not parse it.
132
+
133
+ ### add_from_design
134
+
135
+ - Spec: [processes/create-e2e/commands/add-from-design/SKILL.md](processes/create-e2e/commands/add-from-design/SKILL.md)
136
+ - Needs a Figma URL in the case and `figma: on`
137
+ - Result: one label, `e2e:ready`, `e2e:minor-text-diff`, or a draft `e2e:mismatch`, unless `ask_before_pr` is still waiting for an answer or for explicit authorization
138
+ - The assistant still reads the screen. A narrow minor text difference does not stop the run. `ask_before_pr` stops for a functional mismatch, an unresolved difference, or a failed final-screenshot gate.
139
+
140
+ ### setup_project
141
+
142
+ - Spec: [processes/create-e2e/commands/setup-project/SKILL.md](processes/create-e2e/commands/setup-project/SKILL.md)
143
+ - Input: none. Any argument returns a tool error
144
+ - Result: a filled `.e2e-qa-pilot/project.md` that you approved, or a list of open questions
145
+ - The assistant inspects the repository and the connected tools first, asks one batch of questions for what it cannot detect, and never writes a token
146
+
147
+ ## What stays in the assistant
148
+
149
+ These are not server work, and the server cannot enforce them.
150
+
151
+ - Ask when an expected result is missing, vague, or uses two names for one control.
152
+ - Read the screenshot. One PNG must show that step's control. A repeat of an earlier screen does not count. An omitted earlier image is not, by itself, a failed assertion. The required end-state screenshot must show the expected end state. A screenshot finding does not, by itself, prove that a functional assertion failed.
153
+ - Follow `pr_mode`. `create_pr_for_review` is the default and does not auto-merge. `ask_before_pr` stops before a pull request for a functional mismatch, an unresolved difference, or a failed final-screenshot gate, and creates that pull request only after explicit authorization when the discrepancy remains. A narrow minor text difference does not trigger that stop.
154
+
155
+ The server reads `.e2e-qa-pilot/settings.md` and returns it verbatim. It does not parse `pr_mode`, `pr_label_ready`, `pr_label_minor_text_diff`, or `pr_label_mismatch`, and it does not choose a pull request label. The command files tell the assistant what those keys mean. A missing `pr_mode` means `create_pr_for_review`.
156
+
157
+ ## Without MCP
158
+
159
+ The skills work alone. `.cursor/skills/` and `.claude/skills/` each hold a stub that points at the files under `.e2e-qa-pilot/processes/create-e2e/commands/`. If you delete the MCP entry, the four skills can still be followed from those files. You can also copy those Markdown files by hand. Nothing here is hidden, so read every line before you give an assistant access to anything.
160
+
161
+ The MCP tools and the stubs do not always give the same instructions. A tool returns the files bundled with the installed package. A stub points at the copy in your project. They match only when both come from the same package version, and an edit to the project copy does not change what a tool returns.
162
+
163
+ ## What is and is not verified
164
+
165
+ The tests start the server over stdio, run `init` in temporary folders, and check that the instruction text contains the rules. They do not run Cursor or Claude Code, so compatibility with a specific client or version is not verified here, and they do not prove that an assistant follows the instructions. See [What the tests check](docs/scope-and-trust.md#what-the-tests-check).
166
+
167
+ ## Known limitations
168
+
169
+ - **Review can continue into implementation.** The `review-case` command says it does not write a spec, yet its silent-pass step says to continue into `add-from-steps` or `add-from-design` in the same turn. A ready case can therefore end in a pull request. This wording has not been changed in the documentation update, and needs a separate decision.
170
+ - **Cursor and Claude Code behavior is not verified.** The instructions are Markdown. This repository's tests check that the files and the server are consistent. They do not run an editor and do not prove that an assistant follows the instructions.
171
+
172
+ ## Not implemented
173
+
174
+ [AutoFix nightly](processes/autofix-nightly/README.md) is not implemented. It has no tools here.
package/package.json CHANGED
@@ -1,6 +1,49 @@
1
1
  {
2
2
  "name": "e2e-qa-pilot",
3
- "version": "0.0.0-stage",
4
- "stub": true,
5
- "description": "Temporary package placeholder for staged publishing"
6
- }
3
+ "version": "0.4.5",
4
+ "description": "An MCP server and set of Markdown commands that guide your AI assistant from a manual test case to a reviewed Playwright spec and a pull request, with a human deciding when the case or the screen is unclear.",
5
+ "type": "module",
6
+ "license": "MIT",
7
+ "repository": {
8
+ "type": "git",
9
+ "url": "git+https://github.com/bogdanOwner/e2e-qa-pilot.git"
10
+ },
11
+ "homepage": "https://github.com/bogdanOwner/e2e-qa-pilot#readme",
12
+ "bugs": {
13
+ "url": "https://github.com/bogdanOwner/e2e-qa-pilot/issues"
14
+ },
15
+ "bin": {
16
+ "e2e-qa-pilot": "bin/cli.mjs"
17
+ },
18
+ "files": [
19
+ "bin",
20
+ "src",
21
+ "processes",
22
+ "templates",
23
+ "docs",
24
+ "assets",
25
+ "README.md",
26
+ "mcp.md",
27
+ "CONTRIBUTING.md",
28
+ "SECURITY.md",
29
+ "CODE_OF_CONDUCT.md"
30
+ ],
31
+ "engines": {
32
+ "node": ">=22"
33
+ },
34
+ "publishConfig": {
35
+ "access": "public"
36
+ },
37
+ "keywords": [
38
+ "mcp",
39
+ "playwright",
40
+ "e2e",
41
+ "qa",
42
+ "cursor",
43
+ "claude-code",
44
+ "skills"
45
+ ],
46
+ "scripts": {
47
+ "test": "node --test test/*.test.mjs"
48
+ }
49
+ }
@@ -0,0 +1,18 @@
1
+ # AutoFix nightly
2
+
3
+ **Status: not implemented.**
4
+
5
+ This directory records an intended future process: explaining a red nightly run and proposing one fix for a human to review. There are no commands, tools, or install step for it. No delivery date is committed, and nothing here is a roadmap commitment.
6
+
7
+ Create e2e does not call it.
8
+
9
+ ## Planned flow
10
+
11
+ When this process exists, a nightly run would:
12
+
13
+ 1. Read the real frontend changes since the last run.
14
+ 2. Rerun the failed tests on Claude to check whether the failure is flaky.
15
+ 3. Report the reason for the failure.
16
+ 4. Open a pull request with one proposed fix for a person to review.
17
+
18
+ None of these steps run today. There is still no command, tool, or install step, and no delivery date.
@@ -0,0 +1,43 @@
1
+ # Create e2e
2
+
3
+ Author end-to-end coverage from one case. A case id or pasted case text, plus `.e2e-qa-pilot/settings.md` and `.e2e-qa-pilot/project.md`, are the inputs. [review-case](commands/review-case/SKILL.md) is the only router. A case tool is required only for a case id.
4
+
5
+ ```mermaid
6
+ flowchart TD
7
+ reviewNode[review-case]
8
+ implNode[implementation command]
9
+ readyNode[Behavior matches and evidence is adequate]
10
+ minorNode[Minor text difference only]
11
+ assertNode[Functional or unresolved assertion]
12
+ shotNode[Required screenshot evidence fails]
13
+ modeNode{pr_mode}
14
+ readyPr["PR labeled e2e:ready"]
15
+ minorPr["PR labeled e2e:minor-text-diff"]
16
+ draftPr["Draft PR labeled e2e:mismatch"]
17
+ askNode[Ask then continue only if authorized]
18
+ reviewNode -->|case not ready| askClarify[Ask and do not invent]
19
+ reviewNode -->|case ready same turn| implNode
20
+ implNode --> readyNode
21
+ implNode --> minorNode
22
+ implNode --> assertNode
23
+ implNode --> shotNode
24
+ readyNode --> readyPr
25
+ minorNode --> minorPr
26
+ assertNode --> modeNode
27
+ shotNode --> modeNode
28
+ modeNode -->|create_pr_for_review| draftPr
29
+ modeNode -->|ask_before_pr| askNode
30
+ ```
31
+
32
+ `figma: off` in the settings file ignores a Figma URL and sends the case to [add-from-steps](commands/add-from-steps/SKILL.md). A missing `pr_mode` means `create_pr_for_review`. The server does not parse or enforce it. Apply one label: `e2e:ready`, `e2e:minor-text-diff`, or draft `e2e:mismatch`. Nothing is auto-merged. Assertion findings and screenshot findings stay separate.
33
+
34
+ | Command | When |
35
+ |---------|------|
36
+ | [review-case](commands/review-case/SKILL.md) | First. Ready checks, one question per hole, update that same case when it came from an id, then route |
37
+ | [add-from-design](commands/add-from-design/SKILL.md) | The case has a `figma.com` URL and `figma: on` |
38
+ | [add-from-steps](commands/add-from-steps/SKILL.md) | No Figma URL, or `figma: off` |
39
+ | [setup-project](commands/setup-project/SKILL.md) | Once per project, and when its setup changes. Fills in `.e2e-qa-pilot/project.md`. Takes no case |
40
+
41
+ Shared rules: [ready-rules.md](commands/shared/ready-rules.md), [pr-evidence.md](commands/shared/pr-evidence.md), [project-config.md](commands/shared/project-config.md) (how `.e2e-qa-pilot/project.md` is read), and [workflow.md](commands/shared/workflow.md) (case title, duplicate check, branch name, git steps, where files live).
42
+
43
+ Install with `npx e2e-qa-pilot init` in your project. What to ask, the input contract, and the tool results are in `mcp.md` in the e2e-qa-pilot package and repository.
@@ -0,0 +1,132 @@
1
+ ---
2
+ name: add-from-design
3
+ description: >-
4
+ Add or extend end-to-end tests from a case that has a Figma URL. The case
5
+ is an existing id or pasted test case text. Docs first. Compare each
6
+ milestone to the case, the page contract, and the Figma frame. Follow
7
+ pr_mode for a functional mismatch, an unresolved difference, or a failed
8
+ final screenshot. A narrow minor text difference does not stop the run.
9
+ ---
10
+
11
+ # Add end-to-end coverage from Figma
12
+
13
+ Runs only after [review-case](../review-case/SKILL.md) routes here: the settings file has `figma: on`, and the resolved case has a `figma.com` URL. `case_id`: that URL is in the description or a custom field. `case_text`: that URL is anywhere in the pasted text.
14
+
15
+ Input is exactly one of `case_id` or `case_text`, plus `.e2e-qa-pilot/settings.md` and `.e2e-qa-pilot/project.md`. Read both first. [project-config.md](../shared/project-config.md) says how `project.md` is read and what to do instead of asking. The tool call names the mode. Do not decide which one a string is by reading its contents.
16
+
17
+ `case_id`: fetch the case with `fetch_case` from that file. If the case tool errors or the case does not exist, stop and report that. Do not invent the case. Do not treat this id as test case text.
18
+
19
+ `case_text`: the pasted text is the case. Do not call `fetch_case`. Do not call `update_case`. A case tool is not required.
20
+
21
+ Read `read_figma`, `png_dir`, `branch_pattern`, and `test_command` from that file. A missing or empty `branch_pattern` uses the default in [workflow.md](../shared/workflow.md#workflow-resolution).
22
+
23
+ If `figma`, `read_figma`, or `test_command` is still empty, stop and run [review-case](../review-case/SKILL.md). This command runs only with `figma: on`, so `read_figma` is needed here. `## Design` in `project.md` answers `figma` and `read_figma`. Do the same for an empty `png_dir` when `project.md` does not name the screenshot directory. Do not ask a second time. A missing `pr_mode` means `create_pr_for_review`. A missing label uses `e2e:ready`, `e2e:minor-text-diff`, or `e2e:mismatch`. Empty `case_tool`, `fetch_case`, and `update_case` are expected in `case_text` mode.
24
+
25
+ Title, duplicate check, branch name, git steps, commit subject, and file locations are in [workflow.md](../shared/workflow.md). Follow it. Stop on a duplicate before any docs or spec. Do not open a pull request until Phase D is clear.
26
+
27
+ Context repos are read-only: read them for locators and test ids, never write to one, and open the pull request only in this repository ([project-config.md](../shared/project-config.md#context-repos)). Feature flags are never changed, by you or by the spec. Keep any `Feature flag:` precondition lines from review-case and copy them into the pull request body. Use the `{ticket_id}` that review-case recorded, and ask once only when `## Ticket` requires an id and none was recorded.
28
+
29
+ **Match the repo, then code.** Resolve locations in [Where files live](../shared/workflow.md#where-files-live). When that resolution is the reference layout, do not add a spec until the feature has an overview, Gherkin, and page contracts. Otherwise do not create `overview.md`, `specs/`, or `pages/`.
30
+
31
+ Case steps define *what* to automate. Page contracts and Figma define *what the screen should look like*.
32
+
33
+ ## Phase A — Intake: step coverage map (before coding)
34
+
35
+ 1. From the resolved case, extract **steps**, **preconditions**, and the **Figma URL**. `case_id` reads that URL from the description or a custom field. `case_text` reads it from the pasted text.
36
+ 2. When the resolved layout is the reference layout, add a **step coverage table** to the feature `overview.md`. Otherwise do not create `overview.md`. Record the same coverage beside the spec, in the doc the repo already uses for that feature.
37
+
38
+ | Step | Gherkin / action | Page contract | Milestone attachment | Figma frame |
39
+ |------|------------------|---------------|----------------------|-------------|
40
+ | 1. Open Personal Space | `When the user opens Personal Space…` | the page contract for that screen | `02-personal-space` | node id from the Figma URL |
41
+
42
+ 3. For each page in the table, capture a **reference image**:
43
+ - **Preferred:** `read_figma` from the settings file, saved under `png_dir`
44
+ - **Fallback:** a capture from the running app if the Figma frame is missing or outdated
45
+ 4. When page contracts already exist, add `## Case coverage` to the affected ones. Fill `## Visible copy` and `## States` from the Figma frame: exact titles, button labels, placeholders, and one state block per distinct screen, each citing its screenshot under `png_dir`. Do not create a page-contract file for a repo that does not use them.
46
+
47
+ ## Phase B — Implementation: annotate milestones in the report
48
+
49
+ - **Always** pass `testInfo` from the spec into flows.
50
+ - Before each milestone attachment, push annotations. The example calls `attachMilestoneScreenshot`, a helper from one project. This package does not provide it. Use the helper the repo already has. A repo without one attaches the image in the same place with `await testInfo.attach(name, { body: await page.screenshot(), contentType: 'image/png' })`.
51
+
52
+ ```typescript
53
+ testInfo.annotations.push({
54
+ type: 'case-step',
55
+ description: '1. Open Personal Space from bottom navigation',
56
+ });
57
+ testInfo.annotations.push({
58
+ type: 'page-contract',
59
+ description: '<page contract path from the settings file>',
60
+ });
61
+ await attachMilestoneScreenshot(page, testInfo, MILESTONE_SCREENSHOTS.personalSpace);
62
+ ```
63
+
64
+ - Add a new constant in the milestone screenshot helper when a case step has no existing attachment name.
65
+ - When the reference layout is in use, include the case id in the feature `overview.md`, or the inline display title when there is no id. Do not create `overview.md` for that line.
66
+ - **Spec test title:** `case_id` uses the exact case title prefixed with the case id — for example `test('<case-id> As a User I can edit the personal goal.', …)`. Use the title from `fetch_case`. `case_text` uses the trimmed title, with no invented id. Do not invent a separate descriptive name.
67
+
68
+ Use the locations from [Where files live](../shared/workflow.md#where-files-live). Reuse page objects and flows only when the repo already has them. Import `test` from the fixture existing specs already use. When the repo has no fixture, import `test` the way those specs do. Do not add a fixture file. The spec goes in the resolved spec directory.
69
+
70
+ A step that types or selects data follows the Test data rules in [pr-evidence.md](../shared/pr-evidence.md#test-data): values from the case, or declared fake values, and the filled state on a screenshot before the final click.
71
+
72
+ ## Phase C — Mandatory report review
73
+
74
+ Read `test_command` from the settings file and run it for this spec. Then open the report that command produces.
75
+
76
+ **Check the screenshot at the end to be sure it makes sense and covers the case.**
77
+
78
+ Priority order:
79
+
80
+ 1. **`01-final-screen` (required)** — must show the end state that proves the case passed. If this screen is the wrong page, missing the asserted UI, or does not cover the case expected results → treat as `BLOCKER`.
81
+ 2. **Each milestone** (`02-*` … `07-*`) — same checks below.
82
+
83
+ For **each** attachment:
84
+
85
+ 1. **Case** — does the screenshot match the step action and expected result?
86
+ 2. **Page contract** — are key elements visible (headings, calls to action, form state)?
87
+ 3. **Figma** — does layout and copy roughly match the reference screenshot and the Figma frame?
88
+
89
+ Record per milestone:
90
+
91
+ - `MATCH` — behavior matches and required screenshot evidence shows the expected end state. No text difference: `pr_label_ready` (default `e2e:ready`).
92
+ - `MINOR` — copy, spacing, or icon drift. A sole difference that meets the narrow text rule in [pr-evidence.md](../shared/pr-evidence.md) is `pr_label_minor_text_diff` (default `e2e:minor-text-diff`) and does not stop the run. Any other difference is unresolved and follows `pr_mode`.
93
+ - `BLOCKER` — wrong screen, missing step, broken flow, or `01-final-screen` does not cover the case. This is a functional finding or a screenshot finding. Report them separately. A screenshot that misses the end state does not, by itself, prove that a functional assertion failed. Follow `pr_mode`.
94
+
95
+ ## Phase D — Clarify unexpected diffs
96
+
97
+ Read `pr_mode`. A missing value means `create_pr_for_review`. The pull request labels and the narrow text rule are in [pr-evidence.md](../shared/pr-evidence.md).
98
+
99
+ A sole narrow text difference does not stop the run. Document the case sentence and the screen sentence. Do not treat the exact sentence as a passed assertion. Do not rewrite the expected text. Apply `pr_label_minor_text_diff`.
100
+
101
+ `create_pr_for_review` opens the pull request for the label in that file. Never auto-merge. A functional or unresolved difference, or a final screenshot that misses the expected end state, is a draft pull request labeled `pr_label_mismatch` (default `e2e:mismatch`). It is not, by itself, a confirmed product regression.
102
+
103
+ `ask_before_pr` stops before creating a pull request when a functional mismatch, an unresolved difference, or a failed final-screenshot gate remains. Stop and ask in chat. Do not assume drift is acceptable.
104
+
105
+ For **each** such diff, present:
106
+
107
+ 1. **Milestone** — attachment name (for example `02-personal-space`)
108
+ 2. **Case step** — expected action and result from the case
109
+ 3. **Figma / page contract** — what the design and the docs say should appear
110
+ 4. **Actual screenshot** — describe the real diff (missing element, wrong copy, extra modal, wrong tab state)
111
+ 5. **Finding** — say whether this is a functional assertion finding, a screenshot evidence finding, or both. Do not treat a screenshot finding as proof that the functional assertion failed.
112
+ 6. **Question** — *How should this difference be resolved?*
113
+
114
+ After the user responds, continue from the clarified requirements. If the user changes the expected behavior, later assertions follow that clarification and the pull request quotes the change. If the discrepancy remains, create the pull request only after the user explicitly authorizes it. When the expected behavior has not changed, preserve the failing assertion. If the answer is still unclear, ask one follow-up and do not open a pull request.
115
+
116
+ When the user accepts the difference as a known visual gap, record it under `## Known visual gaps` in the feature `overview.md` and add visual parity notes in the pull request. That record does not rewrite an expected result and is not an assertion edit. If the functional assertions passed and the required end-state screenshot shows the expected end state, apply `pr_label_ready`. If a functional assertion is still failing, preserve it and apply `pr_label_mismatch` under the `pr_mode` rules above.
117
+
118
+ "Fix the test" means a locator or a wait. It does not mean rewriting an expected result.
119
+
120
+ Do not apply a label while `ask_before_pr` is still waiting for an answer or for explicit authorization. Otherwise apply exactly one of `pr_label_ready`, `pr_label_minor_text_diff`, or `pr_label_mismatch` from the settings file. The defaults are `e2e:ready`, `e2e:minor-text-diff`, and `e2e:mismatch`. Full `MATCH`, with the end state shown and no text difference, uses `pr_label_ready`. If GitHub access is missing, report that and do not claim a pull request exists.
121
+
122
+ ## Pull request body
123
+
124
+ The body layout, the steps table with its `✅` and `❌` icons, and the screenshot links are in [pr-evidence.md](../shared/pr-evidence.md). The Actual cell is `✅ As expected` when it equals the expected result.
125
+
126
+ ## Tool contract
127
+
128
+ `add_from_design`
129
+
130
+ - Inputs: exactly one of `case_id` or `case_text`, and the settings file `.e2e-qa-pilot/settings.md`
131
+ - Result: one label, `e2e:ready`, `e2e:minor-text-diff`, or a draft `e2e:mismatch`, unless `ask_before_pr` is still waiting for an answer or for explicit authorization
132
+ - The agent still reads the screen. A narrow minor text difference does not stop the run. `ask_before_pr` stops for a functional mismatch, an unresolved difference, or a failed final-screenshot gate.
@@ -0,0 +1,145 @@
1
+ ---
2
+ name: add-from-steps
3
+ description: >-
4
+ Add an end-to-end spec from one case whose steps name a screen, control, or
5
+ visible change, and whose description has no Figma URL. Branches from the
6
+ resolved starting branch and opens a pull request to the resolved base.
7
+ The fallback base is main. Assertions stay on the
8
+ expected results. The case is an existing id or pasted test case text.
9
+ Use when review-case passes with no Figma link, or the user asks to
10
+ automate a case from its steps.
11
+ ---
12
+
13
+ # Add end-to-end coverage from case steps
14
+
15
+ The steps are the contract. Input is exactly one of `case_id` or `case_text`, plus `.e2e-qa-pilot/settings.md` and `.e2e-qa-pilot/project.md`. Read both first. [project-config.md](../shared/project-config.md) says how `project.md` is read and what to do instead of asking. The tool call names the mode. Do not decide which one a string is by reading its contents.
16
+
17
+ `case_id`: fetch the case with `fetch_case` from that file. If the case tool errors or the case does not exist, stop and report that. Do not invent the case. Do not treat this id as test case text.
18
+
19
+ `case_text`: the pasted text is the case. Do not call `fetch_case`. Do not call `update_case`. A case tool is not required.
20
+
21
+ This skill does not call `update_case`. A case that is not ready to assert is [review-case](../review-case/SKILL.md).
22
+
23
+ This skill checks the expected-result text. It does not run a Figma visual review. A narrow minor text difference does not stop the run. `ask_before_pr` stops for a functional mismatch, an unresolved difference, or a failed final-screenshot gate, as [pr-evidence.md](../shared/pr-evidence.md) describes.
24
+
25
+ PR body, the word rule, and the screenshot gate: [pr-evidence.md](../shared/pr-evidence.md). Ready checks: [ready-rules.md](../shared/ready-rules.md).
26
+
27
+ If `figma` or `test_command` is still empty, or `read_figma` is empty while `figma` is `on`, stop and run [review-case](../review-case/SKILL.md). Do the same for an empty `png_dir` when `project.md` does not name the screenshot directory. Do not ask a second time. A missing `branch_pattern` uses the default in [workflow.md](../shared/workflow.md#workflow-resolution). A missing `pr_mode` means `create_pr_for_review`. A missing label uses `e2e:ready`, `e2e:minor-text-diff`, or `e2e:mismatch`. Empty `case_tool`, `fetch_case`, and `update_case` are expected in `case_text` mode.
28
+
29
+ ## Workflow and project file
30
+
31
+ Title, duplicate check, branch name, git steps, commit subject, and file locations are in [workflow.md](../shared/workflow.md). Follow it.
32
+
33
+ - Context repos are read-only. Read them to choose locators and test ids, as [project-config.md](../shared/project-config.md#context-repos) says. Never write to one, and open the pull request only in this repository. Mark what you could not read as unverified.
34
+ - Feature flags are never changed, by you or by the spec. If the case preconditions carry `Feature flag:` lines, keep them and copy them into the pull request body with the case preconditions.
35
+ - Ticket: use the `{ticket_id}` that review-case recorded. Ask once only when `## Ticket` requires an id and none was recorded.
36
+
37
+ ## Gate
38
+
39
+ Stop before any docs or spec when any of these is true:
40
+
41
+ - The case already has a spec, by the duplicate check in [workflow.md](../shared/workflow.md#duplicates).
42
+ - `case_id`: the description or a custom field contains a `figma.com` URL and the settings file has `figma: on`. `case_text`: a `figma.com` URL anywhere in the pasted text and `figma: on`. Use [add-from-design](../add-from-design/SKILL.md).
43
+ - A step has no expected result, uses `correct`, `properly`, or `etc.`, gives one control two names, or contradicts another step. Use [review-case](../review-case/SKILL.md). Do not implement in that turn.
44
+
45
+ `source-hash` in the description is not a gate. Do not compare it to another case. Do not open a hash-mismatch pull request. The contract is the current steps.
46
+
47
+ ## Docs, then code
48
+
49
+ Read `png_dir` from the settings file. Use the locations from [Where files live](../shared/workflow.md#where-files-live).
50
+
51
+ 1. When the resolved layout is the reference layout, write `overview.md`, Gherkin under `specs/`, and page contracts under `pages/`. Copy each expected result into `## Expected behavior`. Locator hints in `## Elements` may change between runs. Expected-behavior text may not. `## Reference` says "Captured from the running app" and cites `screenshots/<state>.png` under `png_dir`. Do not invent a Figma node. `## Visible copy` records the exact on-screen title, button labels, placeholders, tab names, and toast text. `## Test data` lists each value a step types or selects, and whether the case gave it or the run chose it ([Test data rules](../shared/pr-evidence.md#test-data)). `## States` has one block per distinct screen (open picker, filled chip, next tab, toast), each naming its screenshot and case step. The overview coverage table has the step, the action, the page contract, and the screenshot. No Figma column. One modal stays one page contract. A different dialog gets its own page contract. When the resolved layout is not the reference layout, do not create those files.
52
+ 2. Page objects, only when the repo already has them. Reuse those.
53
+ 3. Flows, only when the repo already has them. Reuse those. Pass `testInfo`. Keep the values a flow types in one named constant, so the spec, the page contract when one exists, and the pull request use the same ones. Before each screenshot:
54
+
55
+ ```typescript
56
+ testInfo.annotations.push({
57
+ type: 'case-step',
58
+ description: '1. <action from the case step>',
59
+ });
60
+ testInfo.annotations.push({
61
+ type: 'page-contract',
62
+ description: '<page contract path from the settings file>',
63
+ });
64
+ ```
65
+
66
+ 4. Thin spec in the resolved spec directory. `case_id` title: `<case-id> <case title>`. `case_text` title: the trimmed title, with no invented id. Import `test` from the fixture existing specs already use. When the repo has no fixture, import `test` the way those specs do. Do not add a fixture file.
67
+
68
+ Before the final click of a step that enters data, assert each entered value and each toggled control. Take the filled-state screenshot at that point.
69
+
70
+ Prefer `getByRole`, `getByLabel`, `getByPlaceholder`. Wait with `expect(...).toBeVisible()` or `toHaveURL()`.
71
+
72
+ On each run, compare the screenshot to the page contract's Elements, Visible copy, and the expected result. Do not edit the app.
73
+
74
+ After a green run, copy a PNG into `png_dir` only when it passes the Screenshot gate in [pr-evidence.md](../shared/pr-evidence.md). The page contract and the overview step-coverage table cite those files. A step with no image cites none. Do not write a second copy under a case folder. Do not write docs from the spec on every run.
75
+
76
+ ## Assertions
77
+
78
+ Follow the word rule in [pr-evidence.md](../shared/pr-evidence.md). The same rule is repeated here.
79
+
80
+ The spec asserts functional outcomes: the control is present, the modal closes, the preview updates, the chart dot appears, and the other visible behavior in the expected result. Copy that behavior from the case into the assertion, including awkward grammar.
81
+
82
+ Do not put the quoted case sentence in an assertion when, after lowercasing, stripping punctuation, and dropping the filler words `a`, `the`, `has`, and `been`, the remaining words match in the same order. Record the case sentence and the screen sentence in the steps table and in the pull request comments. The verdict for that quoted sentence is not `pass`. Example: case **"Your diary entry has successfully created"** and screen **Your Diary entry successfully created!**
83
+
84
+ That difference is not an exact-text pass. Functional checks in the same step stay `pass` when they pass. State that the original exact sentence was not a passing assertion.
85
+
86
+ When the remaining words do not match in the same order, the text difference is unresolved. Assert the case sentence and leave that assertion failing. The spec stays red. Do not treat it as a minor text difference.
87
+
88
+ Do not rewrite the expected sentence. Do not change an assertion so the exact case sentence appears to pass. `assertion_edits` stays 0. Comparing with this rule is not an assertion edit. A green run bought by editing the expected sentence is a failed skill run. Revert it and open the mismatch pull request.
89
+
90
+ When this minor difference is the only issue, and required screenshot evidence meets the acceptance criteria, the pull request label is `pr_label_minor_text_diff` from the settings file. The default is `e2e:minor-text-diff`. It is not `e2e:ready`.
91
+
92
+ A different message (another verb, a missing screen, a missing control) is a functional mismatch. Leave the functional assertion. The spec stays red. A failed functional assertion is a mismatch. It is not, by itself, a confirmed product regression.
93
+
94
+ Take the step PNG of the control that proves the step (modal, open picker, toast) while that result is still on screen. A toast that auto-dismisses is captured while it is visible.
95
+
96
+ ## Screenshot gate
97
+
98
+ Follow [pr-evidence.md](../shared/pr-evidence.md).
99
+
100
+ Look at the PNG before it is copied into `png_dir` or linked in the pull request.
101
+
102
+ - Keep it when the image shows the control or change named in that row (open modal, open picker, filled chip, new tab, toast).
103
+ - A picture that matches an earlier step does not prove this step.
104
+ - A step that enters or selects data needs the filled state on screen: the values and the checked controls, before the final click. When the step also submits, add a second image of the result. Links are `step-03a` and `step-03b`. An empty form or a result card alone does not prove the entry.
105
+ - One retry only: capture the proving control (`locator.screenshot` or a clip of that control), not the same full viewport.
106
+ - If the second image still does not show the step, omit the file. The table row stays. The Screenshot cell is `—`. No image heading for that step.
107
+ - An omitted earlier image is not, by itself, a failed assertion.
108
+ - The required end-state screenshot must show the expected end state. A narrow wording difference on that image does not, by itself, mean the screenshot missed the expected end state. If the image does not show the expected end state, do not publish it as passing evidence. That screenshot finding does not, by itself, prove that a functional assertion failed. Report the findings separately. Either finding blocks `e2e:ready`. A required end-state screenshot that misses the expected end state selects `e2e:mismatch`, unless the only issue in the run is the narrow text difference.
109
+ - Do not change the expected result to match a useless screenshot.
110
+
111
+ `assertion_edits` stays 0. Rewriting the expected sentence so the spec goes green is an assertion edit. Comparing with the word rule above is not. A green run bought by editing the expected sentence is a failed skill run. Revert it and open the mismatch pull request.
112
+
113
+ ## Run loop
114
+
115
+ Read `test_command` from the settings file. Run that command for this spec with retries off. Never install browsers. The browser is already installed.
116
+
117
+ The five-run ceiling is for the whole spec, not five per step. Each command is one run. Retries off disables the runner's automatic repeat of the same failure. Stop early when the expected result passes.
118
+
119
+ - Locator or timing: the control named by the action is not on screen yet. Change the locator or the wait. Run again.
120
+ - Narrow text difference: the on-screen sentence matches the case under the word rule in Assertions. Record the screen sentence. Do not mark the exact sentence `pass`. Continue the later sentences of that step.
121
+ - Behavior: a found control's text or state is a different result. Stop on that run. Do not spend the remaining attempts searching for a different string. Leave the assertion. Do not treat later steps as passed.
122
+
123
+ ## Pull request
124
+
125
+ One pull request to the resolved pull-request base. The fallback is `main`. Never auto-merge. Evidence rules, the body layout, and the steps table are in [pr-evidence.md](../shared/pr-evidence.md). In the steps table the Actual cell is `✅ As expected` when it equals the expected result. Only a difference carries text.
126
+
127
+ Read `pr_mode`. A missing value means `create_pr_for_review`. `ask_before_pr` stops before a pull request for a functional mismatch, an unresolved difference, or a failed final-screenshot gate. Stop and ask in chat. A narrow minor text difference does not trigger that stop. If the discrepancy remains, create the pull request only after the user explicitly authorizes it. When the expected behavior has not changed, preserve the failing assertion.
128
+
129
+ | Outcome | Label | PR |
130
+ |---------|--------|----|
131
+ | Functional assertions passed, the expected behavior matches, and required screenshot evidence meets the acceptance criteria. `assertion_edits` is 0 | `pr_label_ready` from the settings file, default `e2e:ready` | Open for review. Not a draft |
132
+ | Functional behavior matches, required screenshots meet the acceptance criteria, and the only difference is the narrow text rule | `pr_label_minor_text_diff` from the settings file, default `e2e:minor-text-diff` | Open for review. Not a draft. Not `e2e:ready` |
133
+ | A functional or unresolved mismatch, or required screenshot evidence does not show the expected end state | `pr_label_mismatch` from the settings file, default `e2e:mismatch` | Draft. A failed assertion stays red |
134
+
135
+ Screenshot links in the body are head-branch blob and raw URLs. The path is `png_dir` from the settings file. A relative path does not open the file. Link text stays `step-01`. A step whose image failed the Screenshot gate has `—` in the Screenshot cell and no image block. Assertion findings and screenshot findings are separate. A screenshot failure does not, by itself, prove that a functional assertion failed. When the only issue is the narrow text rule, the body records both sentences and states that the exact sentence was not a passing assertion. The agent does not edit the case. That difference is not an assertion edit. Steps that never ran have no PNG. Do not commit `test-results/` or the HTML report folder. Do not use `test.fixme`. A red pull-request check is the expected CI result on a mismatch. If GitHub access is missing, report that and do not claim a pull request exists.
136
+
137
+ Do not edit the nightly workflow.
138
+
139
+ ## Tool contract
140
+
141
+ `add_from_steps`
142
+
143
+ - Inputs: exactly one of `case_id` or `case_text`, and the settings file `.e2e-qa-pilot/settings.md`
144
+ - Result: one label, `e2e:ready`, `e2e:minor-text-diff`, or a draft `e2e:mismatch`, unless `ask_before_pr` is still waiting for an answer or for explicit authorization
145
+ - The agent still reads the screen. A narrow minor text difference does not stop the run.