e2e-qa-pilot 0.0.0-stage → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CODE_OF_CONDUCT.md +126 -0
- package/CONTRIBUTING.md +11 -0
- package/LICENSE +21 -0
- package/README.md +111 -2
- package/SECURITY.md +13 -0
- package/assets/logo.svg +12 -0
- package/bin/cli.mjs +32 -0
- package/docs/README.md +9 -0
- package/docs/install.md +111 -0
- package/docs/scope-and-trust.md +19 -0
- package/docs/settings.md +38 -0
- package/mcp.md +166 -0
- package/package.json +47 -4
- package/processes/autofix-nightly/README.md +18 -0
- package/processes/create-e2e/README.md +42 -0
- package/processes/create-e2e/commands/add-from-design/SKILL.md +173 -0
- package/processes/create-e2e/commands/add-from-steps/SKILL.md +178 -0
- package/processes/create-e2e/commands/review-case/SKILL.md +130 -0
- package/processes/create-e2e/commands/shared/pr-evidence.md +211 -0
- package/processes/create-e2e/commands/shared/ready-rules.md +42 -0
- package/src/init.mjs +267 -0
- package/src/kit.mjs +89 -0
- package/src/mcp.mjs +227 -0
- package/templates/settings.md +57 -0
package/mcp.md
ADDED
|
@@ -0,0 +1,166 @@
|
|
|
1
|
+
# MCP
|
|
2
|
+
|
|
3
|
+
MCP is the connection between your project and the assistant you already use. `init` sets it up for Cursor and Claude Code, and both use the same three tools. You ask in plain words. There is no new screen to learn.
|
|
4
|
+
|
|
5
|
+
Install steps, settings keys, and the trust statement are in [docs](docs/README.md). This page is the tool contract.
|
|
6
|
+
|
|
7
|
+
## What the server does and does not do
|
|
8
|
+
|
|
9
|
+
The server is small on purpose: [src/mcp.mjs](src/mcp.mjs), one file, no dependencies. It returns text. It does not run the workflow, run tests, call a case tool, Figma, or GitHub, or write to your project. It makes no network requests.
|
|
10
|
+
|
|
11
|
+
Each tool returns:
|
|
12
|
+
|
|
13
|
+
- the input block: the case id, or the pasted case text, as the client sent it
|
|
14
|
+
- your `.e2e-qa-pilot/settings.md`, verbatim, or a notice that it is missing
|
|
15
|
+
- the command file and the shared rules, from the installed package
|
|
16
|
+
|
|
17
|
+
Your connected assistant then follows the instructions with whatever tools you have configured: repository access, a test runner, Playwright, your case tool when you passed a case id, Figma for design comparison, and GitHub for the pull request. Not every assistant or client has all of these. If one is missing, the step that needs it cannot be completed.
|
|
18
|
+
|
|
19
|
+
Because the settings file is returned in every result, keep tokens out of it. Keep them in your own MCP configuration.
|
|
20
|
+
|
|
21
|
+
## Connect once
|
|
22
|
+
|
|
23
|
+
In your own project:
|
|
24
|
+
|
|
25
|
+
```bash
|
|
26
|
+
npx e2e-qa-pilot init
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
`init` writes the skills and adds this entry. Other servers in the file are kept. An entry that is already listed is left as it is, and a file that is not valid JSON is left alone.
|
|
30
|
+
|
|
31
|
+
Cursor, `.cursor/mcp.json`:
|
|
32
|
+
|
|
33
|
+
```json
|
|
34
|
+
{
|
|
35
|
+
"mcpServers": {
|
|
36
|
+
"e2e-qa-pilot": {
|
|
37
|
+
"command": "npx",
|
|
38
|
+
"args": ["-y", "e2e-qa-pilot", "mcp"]
|
|
39
|
+
}
|
|
40
|
+
}
|
|
41
|
+
}
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
Claude Code, `.mcp.json` in the project root (same shape). To add it by hand instead, run the command below. It is not exercised by this repository's tests, so check `claude mcp add --help` for the current syntax:
|
|
45
|
+
|
|
46
|
+
```bash
|
|
47
|
+
claude mcp add e2e-qa-pilot -- npx -y e2e-qa-pilot mcp
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
Restart the editor. The server reads `.e2e-qa-pilot/settings.md` from the folder the client starts it in. That is normally your project root, but it depends on the client.
|
|
51
|
+
|
|
52
|
+
The entry is not pinned to a version. `npx -y` uses the package version that npm resolves, or has cached, when the server starts. The command files that `init` copied into your project were taken from the version that ran `init`. The two can differ, as described in [Without MCP](#without-mcp).
|
|
53
|
+
|
|
54
|
+
Also connect your own case tool (for example Qase) when you review a case id. e2e-qa-pilot does not replace it, and pasted case text does not need it.
|
|
55
|
+
|
|
56
|
+
## Ask in plain words
|
|
57
|
+
|
|
58
|
+
Copy a phrase and change the case id, or paste the case.
|
|
59
|
+
|
|
60
|
+
| You want | Ask |
|
|
61
|
+
|----------|-----|
|
|
62
|
+
| Review a case id | Review case `<id>`. Use `.e2e-qa-pilot/settings.md`. |
|
|
63
|
+
| Review pasted case text | Paste the full case, including a `Title:` line, preconditions, steps, and expected results, and ask to review it. |
|
|
64
|
+
| Do everything for a case id | Implement case `<id>` with e2e-qa-pilot. |
|
|
65
|
+
| Automate a case that has no Figma link | Add the end-to-end test from the steps of case `<id>`, or from the pasted case. |
|
|
66
|
+
| Automate a case that has a Figma link | Case `<id>` has a Figma link. Compare the screen to the frame and stop if they differ. |
|
|
67
|
+
|
|
68
|
+
The assistant tells you what it is about to change. For a case id, it updates that same case only after you answer. It does not invent an expected result.
|
|
69
|
+
|
|
70
|
+
## Tools
|
|
71
|
+
|
|
72
|
+
Every tool takes the same input object. `required` is `[]`, because the handler enforces the rule below. Send exactly one of `case_id` or `case_text`, as a string with some text in it.
|
|
73
|
+
|
|
74
|
+
- `case_id`: the id of an existing case in the configured case tool. It needs a connected case tool and the `case_tool`, `fetch_case`, and `update_case` settings. A provider error or a missing case stops the turn. Only an id-based review updates a case.
|
|
75
|
+
- `case_text`: the complete pasted case, including a `Title:` line when the title is known. It needs no case tool and none of those three settings. The server does not fetch or update a case.
|
|
76
|
+
|
|
77
|
+
The server trims both values. A blank value counts as absent, so a blank `case_id` with a real `case_text` is a `case_text` call. These return a tool error (`isError: true`) and no instructions:
|
|
78
|
+
|
|
79
|
+
- both fields with text, or neither
|
|
80
|
+
- a value that is not a string
|
|
81
|
+
- arguments that are not an object
|
|
82
|
+
- any other key
|
|
83
|
+
|
|
84
|
+
An unknown tool name returns a JSON-RPC error instead of a tool result.
|
|
85
|
+
|
|
86
|
+
A call with pasted text:
|
|
87
|
+
|
|
88
|
+
```json
|
|
89
|
+
{
|
|
90
|
+
"name": "review_case",
|
|
91
|
+
"arguments": {
|
|
92
|
+
"case_text": "Title: Open home\nPreconditions: a user is signed in.\n1. Open the home page. Expected: the home page is visible."
|
|
93
|
+
}
|
|
94
|
+
}
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
A call that returns a tool error, because it sends both fields:
|
|
98
|
+
|
|
99
|
+
```json
|
|
100
|
+
{
|
|
101
|
+
"name": "review_case",
|
|
102
|
+
"arguments": {
|
|
103
|
+
"case_id": "QA-123",
|
|
104
|
+
"case_text": "Title: Open home"
|
|
105
|
+
}
|
|
106
|
+
}
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
### Routing
|
|
110
|
+
|
|
111
|
+
`review_case` returns the instructions for the assistant to route. The server does not choose the next tool.
|
|
112
|
+
|
|
113
|
+
- A `figma.com` URL and `figma: on` in the settings go to `add_from_design`. For a case id, the URL is in the description or a custom field. For pasted text, it is anywhere in the text.
|
|
114
|
+
- No Figma URL, or `figma: off`, goes to `add_from_steps`.
|
|
115
|
+
- `add_from_design` also needs `read_figma` to work, so your assistant must be able to read the Figma frame.
|
|
116
|
+
|
|
117
|
+
The assistant may run the next command by calling its MCP tool or by reading the installed command file.
|
|
118
|
+
|
|
119
|
+
### review_case
|
|
120
|
+
|
|
121
|
+
- Spec: [processes/create-e2e/commands/review-case/SKILL.md](processes/create-e2e/commands/review-case/SKILL.md)
|
|
122
|
+
- Result: the case is ready and routed, or one question per hole was asked and, for a case id, that same case was updated
|
|
123
|
+
- If the case is ready, the instructions tell the assistant to continue into the routed command in the same turn. That can end in a pull request. See [Known limitations](#known-limitations)
|
|
124
|
+
- The assistant still asks, and does not invent an expected result
|
|
125
|
+
|
|
126
|
+
### add_from_steps
|
|
127
|
+
|
|
128
|
+
- Spec: [processes/create-e2e/commands/add-from-steps/SKILL.md](processes/create-e2e/commands/add-from-steps/SKILL.md)
|
|
129
|
+
- Result: one label, `e2e:ready`, `e2e:minor-text-diff`, or a draft `e2e:mismatch`, unless `ask_before_pr` is still waiting for an answer or for explicit authorization
|
|
130
|
+
- The assistant still reads the screen. A narrow minor text difference does not stop the run. `pr_mode` is read from settings by the assistant. The server does not parse it.
|
|
131
|
+
|
|
132
|
+
### add_from_design
|
|
133
|
+
|
|
134
|
+
- Spec: [processes/create-e2e/commands/add-from-design/SKILL.md](processes/create-e2e/commands/add-from-design/SKILL.md)
|
|
135
|
+
- Needs a Figma URL in the case and `figma: on`
|
|
136
|
+
- Result: one label, `e2e:ready`, `e2e:minor-text-diff`, or a draft `e2e:mismatch`, unless `ask_before_pr` is still waiting for an answer or for explicit authorization
|
|
137
|
+
- The assistant still reads the screen. A narrow minor text difference does not stop the run. `ask_before_pr` stops for a functional mismatch, an unresolved difference, or a failed final-screenshot gate.
|
|
138
|
+
|
|
139
|
+
## What stays in the assistant
|
|
140
|
+
|
|
141
|
+
These are not server work, and the server cannot enforce them.
|
|
142
|
+
|
|
143
|
+
- Ask when an expected result is missing, vague, or uses two names for one control.
|
|
144
|
+
- Read the screenshot. One PNG must show that step's control. A repeat of an earlier screen does not count. An omitted earlier image is not, by itself, a failed assertion. The required end-state screenshot must show the expected end state. A screenshot finding does not, by itself, prove that a functional assertion failed.
|
|
145
|
+
- Follow `pr_mode`. `create_pr_for_review` is the default and does not auto-merge. `ask_before_pr` stops before a pull request for a functional mismatch, an unresolved difference, or a failed final-screenshot gate, and creates that pull request only after explicit authorization when the discrepancy remains. A narrow minor text difference does not trigger that stop.
|
|
146
|
+
|
|
147
|
+
The server reads `.e2e-qa-pilot/settings.md` and returns it verbatim. It does not parse `pr_mode`, `pr_label_ready`, `pr_label_minor_text_diff`, or `pr_label_mismatch`, and it does not choose a pull request label. The command files tell the assistant what those keys mean. A missing `pr_mode` means `create_pr_for_review`.
|
|
148
|
+
|
|
149
|
+
## Without MCP
|
|
150
|
+
|
|
151
|
+
The skills work alone. `.cursor/skills/` and `.claude/skills/` each hold a stub that points at the files under `.e2e-qa-pilot/processes/create-e2e/commands/`. If you delete the MCP entry, the three skills can still be followed from those files. You can also copy those Markdown files by hand. Nothing here is hidden, so read every line before you give an assistant access to anything.
|
|
152
|
+
|
|
153
|
+
The MCP tools and the stubs do not always give the same instructions. A tool returns the files bundled with the installed package. A stub points at the copy in your project. They match only when both come from the same package version, and an edit to the project copy does not change what a tool returns.
|
|
154
|
+
|
|
155
|
+
## What is and is not verified
|
|
156
|
+
|
|
157
|
+
The tests in `test/cli.test.mjs` start the server over stdio and send JSON-RPC messages, and they run `init` in temporary folders. They check that the instruction text contains the rules. They do not run Cursor or Claude Code, so compatibility with a specific client or version is not verified here, and they do not prove that an assistant follows the instructions. `test/docs.test.mjs` checks these documents against the code. It is a documentation check, not a behavioral or integration test. A longer account of scope and trust is in [scope and trust](docs/scope-and-trust.md).
|
|
158
|
+
|
|
159
|
+
## Known limitations
|
|
160
|
+
|
|
161
|
+
- **Review can continue into implementation.** The `review-case` command says it does not write a spec, yet its silent-pass step says to continue into `add-from-steps` or `add-from-design` in the same turn. A ready case can therefore end in a pull request. This wording has not been changed in the documentation update, and needs a separate decision.
|
|
162
|
+
- **`read_figma` with `figma: off`.** The command files list `read_figma` among the generation keys that must have a value, and tell the assistant to stop and return to `review-case` when one is empty. The settings template leaves `read_figma` blank for a Figma-off setup. The assistant may therefore ask for a value that this setup does not use. This is a possible inconsistency in the workflow, and fixing it would change behavior.
|
|
163
|
+
|
|
164
|
+
## Not implemented
|
|
165
|
+
|
|
166
|
+
[AutoFix nightly](processes/autofix-nightly/README.md) is not implemented. It has no tools here.
|
package/package.json
CHANGED
|
@@ -1,6 +1,49 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "e2e-qa-pilot",
|
|
3
|
-
"version": "0.
|
|
4
|
-
"
|
|
5
|
-
"
|
|
6
|
-
|
|
3
|
+
"version": "0.4.0",
|
|
4
|
+
"description": "An MCP server and set of Markdown commands that guide your AI assistant from a manual test case to a reviewed Playwright spec and a pull request, with a human deciding when the case or the screen is unclear.",
|
|
5
|
+
"type": "module",
|
|
6
|
+
"license": "MIT",
|
|
7
|
+
"repository": {
|
|
8
|
+
"type": "git",
|
|
9
|
+
"url": "git+https://github.com/bogdanOwner/e2e-qa-pilot.git"
|
|
10
|
+
},
|
|
11
|
+
"homepage": "https://github.com/bogdanOwner/e2e-qa-pilot#readme",
|
|
12
|
+
"bugs": {
|
|
13
|
+
"url": "https://github.com/bogdanOwner/e2e-qa-pilot/issues"
|
|
14
|
+
},
|
|
15
|
+
"bin": {
|
|
16
|
+
"e2e-qa-pilot": "bin/cli.mjs"
|
|
17
|
+
},
|
|
18
|
+
"files": [
|
|
19
|
+
"bin",
|
|
20
|
+
"src",
|
|
21
|
+
"processes",
|
|
22
|
+
"templates",
|
|
23
|
+
"docs",
|
|
24
|
+
"assets",
|
|
25
|
+
"README.md",
|
|
26
|
+
"mcp.md",
|
|
27
|
+
"CONTRIBUTING.md",
|
|
28
|
+
"SECURITY.md",
|
|
29
|
+
"CODE_OF_CONDUCT.md"
|
|
30
|
+
],
|
|
31
|
+
"engines": {
|
|
32
|
+
"node": ">=22"
|
|
33
|
+
},
|
|
34
|
+
"publishConfig": {
|
|
35
|
+
"access": "public"
|
|
36
|
+
},
|
|
37
|
+
"keywords": [
|
|
38
|
+
"mcp",
|
|
39
|
+
"playwright",
|
|
40
|
+
"e2e",
|
|
41
|
+
"qa",
|
|
42
|
+
"cursor",
|
|
43
|
+
"claude-code",
|
|
44
|
+
"skills"
|
|
45
|
+
],
|
|
46
|
+
"scripts": {
|
|
47
|
+
"test": "node --test test/*.test.mjs"
|
|
48
|
+
}
|
|
49
|
+
}
|
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
# AutoFix nightly
|
|
2
|
+
|
|
3
|
+
**Status: not implemented.**
|
|
4
|
+
|
|
5
|
+
This directory records an intended future process: explaining a red nightly run and proposing one fix for a human to review. There are no commands, tools, or install step for it. No delivery date is committed, and nothing here is a roadmap commitment.
|
|
6
|
+
|
|
7
|
+
Create e2e does not call it.
|
|
8
|
+
|
|
9
|
+
## Planned flow
|
|
10
|
+
|
|
11
|
+
When this process exists, a nightly run would:
|
|
12
|
+
|
|
13
|
+
1. Read the real frontend changes since the last run.
|
|
14
|
+
2. Rerun the failed tests on Claude to check whether the failure is flaky.
|
|
15
|
+
3. Report the reason for the failure.
|
|
16
|
+
4. Open a pull request with one proposed fix for a person to review.
|
|
17
|
+
|
|
18
|
+
None of these steps run today. There is still no command, tool, or install step, and no delivery date.
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
# Create e2e
|
|
2
|
+
|
|
3
|
+
Author end-to-end coverage from one case. A case id or pasted case text, plus `.e2e-qa-pilot/settings.md`, are the inputs. [review-case](commands/review-case/SKILL.md) is the only router. A case tool is required only for a case id.
|
|
4
|
+
|
|
5
|
+
```mermaid
|
|
6
|
+
flowchart TD
|
|
7
|
+
reviewNode[review-case]
|
|
8
|
+
implNode[implementation command]
|
|
9
|
+
readyNode[Behavior matches and evidence is adequate]
|
|
10
|
+
minorNode[Minor text difference only]
|
|
11
|
+
assertNode[Functional or unresolved assertion]
|
|
12
|
+
shotNode[Required screenshot evidence fails]
|
|
13
|
+
modeNode{pr_mode}
|
|
14
|
+
readyPr["PR labeled e2e:ready"]
|
|
15
|
+
minorPr["PR labeled e2e:minor-text-diff"]
|
|
16
|
+
draftPr["Draft PR labeled e2e:mismatch"]
|
|
17
|
+
askNode[Ask then continue only if authorized]
|
|
18
|
+
reviewNode -->|case not ready| askClarify[Ask and do not invent]
|
|
19
|
+
reviewNode -->|case ready same turn| implNode
|
|
20
|
+
implNode --> readyNode
|
|
21
|
+
implNode --> minorNode
|
|
22
|
+
implNode --> assertNode
|
|
23
|
+
implNode --> shotNode
|
|
24
|
+
readyNode --> readyPr
|
|
25
|
+
minorNode --> minorPr
|
|
26
|
+
assertNode --> modeNode
|
|
27
|
+
shotNode --> modeNode
|
|
28
|
+
modeNode -->|create_pr_for_review| draftPr
|
|
29
|
+
modeNode -->|ask_before_pr| askNode
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
`figma: off` in the settings file ignores a Figma URL and sends the case to [add-from-steps](commands/add-from-steps/SKILL.md). A missing `pr_mode` means `create_pr_for_review`. The server does not parse or enforce it. Apply one label: `e2e:ready`, `e2e:minor-text-diff`, or draft `e2e:mismatch`. Nothing is auto-merged. Assertion findings and screenshot findings stay separate.
|
|
33
|
+
|
|
34
|
+
| Command | When |
|
|
35
|
+
|---------|------|
|
|
36
|
+
| [review-case](commands/review-case/SKILL.md) | First. Ready checks, one question per hole, update that same case when it came from an id, then route |
|
|
37
|
+
| [add-from-design](commands/add-from-design/SKILL.md) | The case has a `figma.com` URL and `figma: on` |
|
|
38
|
+
| [add-from-steps](commands/add-from-steps/SKILL.md) | No Figma URL, or `figma: off` |
|
|
39
|
+
|
|
40
|
+
Shared rules: [ready-rules.md](commands/shared/ready-rules.md) and [pr-evidence.md](commands/shared/pr-evidence.md).
|
|
41
|
+
|
|
42
|
+
Install with `npx e2e-qa-pilot init` in your project, once the package is published to npm. What to ask, the input contract, and the three tool results are in `mcp.md` in the e2e-qa-pilot package and repository.
|
|
@@ -0,0 +1,173 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: add-from-design
|
|
3
|
+
description: >-
|
|
4
|
+
Add or extend end-to-end tests from a case that has a Figma URL. The case
|
|
5
|
+
is an existing id or pasted test case text. Docs first. Compare each
|
|
6
|
+
milestone to the case, the page contract, and the Figma frame. Follow
|
|
7
|
+
pr_mode for a functional mismatch, an unresolved difference, or a failed
|
|
8
|
+
final screenshot. A narrow minor text difference does not stop the run.
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
# Add end-to-end coverage from Figma
|
|
12
|
+
|
|
13
|
+
Runs only after [review-case](../review-case/SKILL.md) routes here: the settings file has `figma: on`, and the resolved case has a `figma.com` URL. `case_id`: that URL is in the description or a custom field. `case_text`: that URL is anywhere in the pasted text.
|
|
14
|
+
|
|
15
|
+
Input is exactly one of `case_id` or `case_text`, plus `.e2e-qa-pilot/settings.md`. The tool call names the mode. Do not decide which one a string is by reading its contents.
|
|
16
|
+
|
|
17
|
+
`case_id`: fetch the case with `fetch_case` from that file. If the case tool errors or the case does not exist, stop and report that. Do not invent the case. Do not treat this id as test case text.
|
|
18
|
+
|
|
19
|
+
`case_text`: the pasted text is the case. Do not call `fetch_case`. Do not call `update_case`. A case tool is not required.
|
|
20
|
+
|
|
21
|
+
Read `read_figma`, `png_dir`, `branch_pattern`, and `test_command` from that file. A missing or empty `branch_pattern` uses the default in Branch below.
|
|
22
|
+
|
|
23
|
+
If `figma`, `read_figma`, `test_command`, or `png_dir` is still empty, stop and run [review-case](../review-case/SKILL.md). Do not ask a second time. A missing `branch_pattern` uses the default in Branch below. A missing `pr_mode` means `create_pr_for_review`. A missing label uses `e2e:ready`, `e2e:minor-text-diff`, or `e2e:mismatch`. Empty `case_tool`, `fetch_case`, and `update_case` are expected in `case_text` mode.
|
|
24
|
+
|
|
25
|
+
**Docs first, code second.** Do not add a spec until the feature has an overview, Gherkin, and page contracts.
|
|
26
|
+
|
|
27
|
+
Case steps define *what* to automate. Page contracts and Figma define *what the screen should look like*.
|
|
28
|
+
|
|
29
|
+
## Title
|
|
30
|
+
|
|
31
|
+
`case_id` uses the title from `fetch_case`. The spec and pull request title stay `<case-id> <case title>`.
|
|
32
|
+
|
|
33
|
+
`case_text` chooses a title before a spec, a filename, or a pull request is named:
|
|
34
|
+
|
|
35
|
+
- Prefer a `Title:` line.
|
|
36
|
+
- If that line is absent, use the first Markdown H1 (`# `, not `##`).
|
|
37
|
+
- If both exist, compare the trimmed texts. A difference in case or surrounding whitespace still uses the `Title:` line. Any other difference asks one question. Do not pick silently.
|
|
38
|
+
- If neither exists, ask for a short title before generating the spec.
|
|
39
|
+
- Do not take a precondition or a step as the title.
|
|
40
|
+
- Do not invent an external id or a case URL.
|
|
41
|
+
|
|
42
|
+
Display text preserves the original casing and internal wording, with surrounding whitespace removed. Filenames, the pull request title, the spec title, and the duplicate check use that trimmed title. Do not leave leading or trailing whitespace in those names.
|
|
43
|
+
|
|
44
|
+
## Duplicates
|
|
45
|
+
|
|
46
|
+
`case_id`: a spec whose title starts with `<case-id>` already exists. Do not add a second spec.
|
|
47
|
+
|
|
48
|
+
`case_text`: search existing spec titles in the spec location from settings. Normalize by trimming, lowercasing, collapsing whitespace, and stripping punctuation. An equal normalized title is the existing spec. Do not add another. A normalized title that contains the other, in either direction, is only a likely match. Read that spec. Ask one question only when the read leaves it ambiguous whether it is the same test. Do not block a genuinely different test because its title contains similar words. If the spec location cannot be listed or read, say the duplicate check could not be completed. Do not claim that no duplicate exists.
|
|
49
|
+
|
|
50
|
+
## Branch
|
|
51
|
+
|
|
52
|
+
Read `branch_pattern` from the settings file. It names the branch for this case. Placeholders are filled in:
|
|
53
|
+
|
|
54
|
+
- `{title_slug}`: the trimmed case title in kebab-case. Lowercase, every run of characters other than letters and digits becomes one `-`, no leading or trailing `-`, at most 50 characters, cut at a word boundary. "Book Chalet 4 at Cabana" becomes `book-chalet-4-at-cabana`.
|
|
55
|
+
- `{case_id}`: the case id in lowercase kebab-case. It is filled in only in `case_id` mode. In `case_text` mode drop this placeholder and the separator next to it. Never invent an id.
|
|
56
|
+
|
|
57
|
+
A missing or empty `branch_pattern` uses `e2e/{title_slug}` in `case_text` mode and `e2e/{case_id}-{title_slug}` in `case_id` mode, for example `e2e/book-chalet-4-at-cabana`. Do not ask.
|
|
58
|
+
|
|
59
|
+
A value with no placeholder is a literal branch name and is used as it is. It puts every case on that one branch, so the template recommends a placeholder.
|
|
60
|
+
|
|
61
|
+
Run `git check-ref-format --branch` on the final name. If Git accepts it, use it. If Git rejects it, ask once for another name.
|
|
62
|
+
|
|
63
|
+
From fresh `main`:
|
|
64
|
+
|
|
65
|
+
```bash
|
|
66
|
+
git fetch origin
|
|
67
|
+
git checkout main
|
|
68
|
+
git pull origin main
|
|
69
|
+
git checkout -b <branch from branch_pattern>
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
If that branch already exists on the remote, check it out and pull. Do not create a second branch. The pull request targets `main`. Never auto-merge. Do not open a pull request until Phase D is clear.
|
|
73
|
+
|
|
74
|
+
## Phase A — Intake: step coverage map (before coding)
|
|
75
|
+
|
|
76
|
+
1. From the resolved case, extract **steps**, **preconditions**, and the **Figma URL**. `case_id` reads that URL from the description or a custom field. `case_text` reads it from the pasted text.
|
|
77
|
+
2. Add a **step coverage table** to the feature `overview.md`:
|
|
78
|
+
|
|
79
|
+
| Step | Gherkin / action | Page contract | Milestone attachment | Figma frame |
|
|
80
|
+
|------|------------------|---------------|----------------------|-------------|
|
|
81
|
+
| 1. Open Personal Space | `When the user opens Personal Space…` | the page contract for that screen | `02-personal-space` | node id from the Figma URL |
|
|
82
|
+
|
|
83
|
+
3. For each page in the table, capture a **reference image**:
|
|
84
|
+
- **Preferred:** `read_figma` from the settings file, saved under `png_dir`
|
|
85
|
+
- **Fallback:** a capture from the running app if the Figma frame is missing or outdated
|
|
86
|
+
4. Add `## Case coverage` to affected page contracts. Fill `## Visible copy` and `## States` from the Figma frame: exact titles, button labels, placeholders, and one state block per distinct screen, each citing its screenshot under `png_dir`.
|
|
87
|
+
|
|
88
|
+
## Phase B — Implementation: annotate milestones in the report
|
|
89
|
+
|
|
90
|
+
- **Always** pass `testInfo` from the spec into flows.
|
|
91
|
+
- Before each `attachMilestoneScreenshot`, push annotations:
|
|
92
|
+
|
|
93
|
+
```typescript
|
|
94
|
+
testInfo.annotations.push({
|
|
95
|
+
type: 'case-step',
|
|
96
|
+
description: '1. Open Personal Space from bottom navigation',
|
|
97
|
+
});
|
|
98
|
+
testInfo.annotations.push({
|
|
99
|
+
type: 'page-contract',
|
|
100
|
+
description: '<page contract path from the settings file>',
|
|
101
|
+
});
|
|
102
|
+
await attachMilestoneScreenshot(page, testInfo, MILESTONE_SCREENSHOTS.personalSpace);
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
- Add a new constant in the milestone screenshot helper when a case step has no existing attachment name.
|
|
106
|
+
- Include the case id in the feature `overview.md`, or the inline display title when there is no id.
|
|
107
|
+
- **Spec test title:** `case_id` uses the exact case title prefixed with the case id — for example `test('<case-id> As a User I can edit the personal goal.', …)`. Use the title from `fetch_case`. `case_text` uses the trimmed title, with no invented id. Do not invent a separate descriptive name.
|
|
108
|
+
|
|
109
|
+
Read the settings file for where page objects, flows, and the spec live. Reuse existing page objects and flows.
|
|
110
|
+
|
|
111
|
+
A step that types or selects data follows the Test data rules in [pr-evidence.md](../shared/pr-evidence.md#test-data): values from the case, or declared fake values, and the filled state on a screenshot before the final click.
|
|
112
|
+
|
|
113
|
+
## Phase C — Mandatory report review
|
|
114
|
+
|
|
115
|
+
Read `test_command` from the settings file and run it for this spec. Then open the report that command produces.
|
|
116
|
+
|
|
117
|
+
**Check the screenshot at the end to be sure it makes sense and covers the case.**
|
|
118
|
+
|
|
119
|
+
Priority order:
|
|
120
|
+
|
|
121
|
+
1. **`01-final-screen` (required)** — must show the end state that proves the case passed. If this screen is the wrong page, missing the asserted UI, or does not cover the case expected results → treat as `BLOCKER`.
|
|
122
|
+
2. **Each milestone** (`02-*` … `07-*`) — same checks below.
|
|
123
|
+
|
|
124
|
+
For **each** attachment:
|
|
125
|
+
|
|
126
|
+
1. **Case** — does the screenshot match the step action and expected result?
|
|
127
|
+
2. **Page contract** — are key elements visible (headings, calls to action, form state)?
|
|
128
|
+
3. **Figma** — does layout and copy roughly match the reference screenshot and the Figma frame?
|
|
129
|
+
|
|
130
|
+
Record per milestone:
|
|
131
|
+
|
|
132
|
+
- `MATCH` — behavior matches and required screenshot evidence shows the expected end state. No text difference: `pr_label_ready` (default `e2e:ready`).
|
|
133
|
+
- `MINOR` — copy, spacing, or icon drift. A sole difference that meets the narrow text rule in [pr-evidence.md](../shared/pr-evidence.md) is `pr_label_minor_text_diff` (default `e2e:minor-text-diff`) and does not stop the run. Any other difference is unresolved and follows `pr_mode`.
|
|
134
|
+
- `BLOCKER` — wrong screen, missing step, broken flow, or `01-final-screen` does not cover the case. This is a functional finding or a screenshot finding. Report them separately. A screenshot that misses the end state does not, by itself, prove that a functional assertion failed. Follow `pr_mode`.
|
|
135
|
+
|
|
136
|
+
## Phase D — Clarify unexpected diffs
|
|
137
|
+
|
|
138
|
+
Read `pr_mode`. A missing value means `create_pr_for_review`. The pull request labels and the narrow text rule are in [pr-evidence.md](../shared/pr-evidence.md).
|
|
139
|
+
|
|
140
|
+
A sole narrow text difference does not stop the run. Document the case sentence and the screen sentence. Do not treat the exact sentence as a passed assertion. Do not rewrite the expected text. Apply `pr_label_minor_text_diff`.
|
|
141
|
+
|
|
142
|
+
`create_pr_for_review` opens the pull request for the label in that file. Never auto-merge. A functional or unresolved difference, or a final screenshot that misses the expected end state, is a draft pull request labeled `pr_label_mismatch` (default `e2e:mismatch`). It is not, by itself, a confirmed product regression.
|
|
143
|
+
|
|
144
|
+
`ask_before_pr` stops before creating a pull request when a functional mismatch, an unresolved difference, or a failed final-screenshot gate remains. Stop and ask in chat. Do not assume drift is acceptable.
|
|
145
|
+
|
|
146
|
+
For **each** such diff, present:
|
|
147
|
+
|
|
148
|
+
1. **Milestone** — attachment name (for example `02-personal-space`)
|
|
149
|
+
2. **Case step** — expected action and result from the case
|
|
150
|
+
3. **Figma / page contract** — what the design and the docs say should appear
|
|
151
|
+
4. **Actual screenshot** — describe the real diff (missing element, wrong copy, extra modal, wrong tab state)
|
|
152
|
+
5. **Finding** — say whether this is a functional assertion finding, a screenshot evidence finding, or both. Do not treat a screenshot finding as proof that the functional assertion failed.
|
|
153
|
+
6. **Question** — *How should this difference be resolved?*
|
|
154
|
+
|
|
155
|
+
After the user responds, continue from the clarified requirements. If the user changes the expected behavior, later assertions follow that clarification and the pull request quotes the change. If the discrepancy remains, create the pull request only after the user explicitly authorizes it. When the expected behavior has not changed, preserve the failing assertion. If the answer is still unclear, ask one follow-up and do not open a pull request.
|
|
156
|
+
|
|
157
|
+
When the user accepts the difference as a known visual gap, record it under `## Known visual gaps` in the feature `overview.md` and add visual parity notes in the pull request. That record does not rewrite an expected result and is not an assertion edit. If the functional assertions passed and the required end-state screenshot shows the expected end state, apply `pr_label_ready`. If a functional assertion is still failing, preserve it and apply `pr_label_mismatch` under the `pr_mode` rules above.
|
|
158
|
+
|
|
159
|
+
"Fix the test" means a locator or a wait. It does not mean rewriting an expected result.
|
|
160
|
+
|
|
161
|
+
Do not apply a label while `ask_before_pr` is still waiting for an answer or for explicit authorization. Otherwise apply exactly one of `pr_label_ready`, `pr_label_minor_text_diff`, or `pr_label_mismatch` from the settings file. The defaults are `e2e:ready`, `e2e:minor-text-diff`, and `e2e:mismatch`. Full `MATCH`, with the end state shown and no text difference, uses `pr_label_ready`. If GitHub access is missing, report that and do not claim a pull request exists.
|
|
162
|
+
|
|
163
|
+
## Pull request body
|
|
164
|
+
|
|
165
|
+
The body layout, the steps table with its `✅` and `❌` icons, and the screenshot links are in [pr-evidence.md](../shared/pr-evidence.md). The Actual cell is `✅ As expected` when it equals the expected result.
|
|
166
|
+
|
|
167
|
+
## Tool contract
|
|
168
|
+
|
|
169
|
+
`add_from_design`
|
|
170
|
+
|
|
171
|
+
- Inputs: exactly one of `case_id` or `case_text`, and the settings file `.e2e-qa-pilot/settings.md`
|
|
172
|
+
- Result: one label, `e2e:ready`, `e2e:minor-text-diff`, or a draft `e2e:mismatch`, unless `ask_before_pr` is still waiting for an answer or for explicit authorization
|
|
173
|
+
- The agent still reads the screen. A narrow minor text difference does not stop the run. `ask_before_pr` stops for a functional mismatch, an unresolved difference, or a failed final-screenshot gate.
|
|
@@ -0,0 +1,178 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: add-from-steps
|
|
3
|
+
description: >-
|
|
4
|
+
Add an end-to-end spec from one case whose steps name a screen, control, or
|
|
5
|
+
visible change, and whose description has no Figma URL. Branches from the
|
|
6
|
+
settings file and opens a pull request to main. Assertions stay on the
|
|
7
|
+
expected results. The case is an existing id or pasted test case text.
|
|
8
|
+
Use when review-case passes with no Figma link, or the user asks to
|
|
9
|
+
automate a case from its steps.
|
|
10
|
+
---
|
|
11
|
+
|
|
12
|
+
# Add end-to-end coverage from case steps
|
|
13
|
+
|
|
14
|
+
The steps are the contract. Input is exactly one of `case_id` or `case_text`, plus `.e2e-qa-pilot/settings.md`. The tool call names the mode. Do not decide which one a string is by reading its contents.
|
|
15
|
+
|
|
16
|
+
`case_id`: fetch the case with `fetch_case` from that file. If the case tool errors or the case does not exist, stop and report that. Do not invent the case. Do not treat this id as test case text.
|
|
17
|
+
|
|
18
|
+
`case_text`: the pasted text is the case. Do not call `fetch_case`. Do not call `update_case`. A case tool is not required.
|
|
19
|
+
|
|
20
|
+
This skill does not call `update_case`. A case that is not ready to assert is [review-case](../review-case/SKILL.md).
|
|
21
|
+
|
|
22
|
+
This skill checks the expected-result text. It does not run a Figma visual review. A narrow minor text difference does not stop the run. `ask_before_pr` stops for a functional mismatch, an unresolved difference, or a failed final-screenshot gate, as [pr-evidence.md](../shared/pr-evidence.md) describes.
|
|
23
|
+
|
|
24
|
+
PR body, the word rule, and the screenshot gate: [pr-evidence.md](../shared/pr-evidence.md). Ready checks: [ready-rules.md](../shared/ready-rules.md).
|
|
25
|
+
|
|
26
|
+
If `figma`, `read_figma`, `test_command`, or `png_dir` is still empty, stop and run [review-case](../review-case/SKILL.md). Do not ask a second time. A missing `branch_pattern` uses the default in Branch below. A missing `pr_mode` means `create_pr_for_review`. A missing label uses `e2e:ready`, `e2e:minor-text-diff`, or `e2e:mismatch`. Empty `case_tool`, `fetch_case`, and `update_case` are expected in `case_text` mode.
|
|
27
|
+
|
|
28
|
+
## Title
|
|
29
|
+
|
|
30
|
+
`case_id` uses the title from `fetch_case`. The spec and pull request title stay `<case-id> <case title>`.
|
|
31
|
+
|
|
32
|
+
`case_text` chooses a title before a spec, a filename, or a pull request is named:
|
|
33
|
+
|
|
34
|
+
- Prefer a `Title:` line.
|
|
35
|
+
- If that line is absent, use the first Markdown H1 (`# `, not `##`).
|
|
36
|
+
- If both exist, compare the trimmed texts. A difference in case or surrounding whitespace still uses the `Title:` line. Any other difference asks one question. Do not pick silently.
|
|
37
|
+
- If neither exists, ask for a short title before generating the spec.
|
|
38
|
+
- Do not take a precondition or a step as the title.
|
|
39
|
+
- Do not invent an external id or a case URL.
|
|
40
|
+
|
|
41
|
+
Display text preserves the original casing and internal wording, with surrounding whitespace removed. Filenames, the pull request title, the spec title, and the duplicate check use that trimmed title. Do not leave leading or trailing whitespace in those names.
|
|
42
|
+
|
|
43
|
+
## Gate
|
|
44
|
+
|
|
45
|
+
Stop before any docs or spec when any of these is true:
|
|
46
|
+
|
|
47
|
+
- `case_id`: a spec whose title starts with `<case-id>` already exists. Do not add a second spec.
|
|
48
|
+
- `case_text`: search existing spec titles in the spec location from settings. Normalize by trimming, lowercasing, collapsing whitespace, and stripping punctuation. An equal normalized title is the existing spec. Do not add another. A normalized title that contains the other, in either direction, is only a likely match. Read that spec. Ask one question only when the read leaves it ambiguous whether it is the same test. Do not block a genuinely different test because its title contains similar words. If the spec location cannot be listed or read, say the duplicate check could not be completed. Do not claim that no duplicate exists.
|
|
49
|
+
- `case_id`: the description or a custom field contains a `figma.com` URL and the settings file has `figma: on`. `case_text`: a `figma.com` URL anywhere in the pasted text and `figma: on`. Use [add-from-design](../add-from-design/SKILL.md).
|
|
50
|
+
- A step has no expected result, uses `correct`, `properly`, or `etc.`, gives one control two names, or contradicts another step. Use [review-case](../review-case/SKILL.md). Do not implement in that turn.
|
|
51
|
+
|
|
52
|
+
`source-hash` in the description is not a gate. Do not compare it to another case. Do not open a hash-mismatch pull request. The contract is the current steps.
|
|
53
|
+
|
|
54
|
+
## Branch
|
|
55
|
+
|
|
56
|
+
Read `branch_pattern` from the settings file. It names the branch for this case. Placeholders are filled in:
|
|
57
|
+
|
|
58
|
+
- `{title_slug}`: the trimmed case title in kebab-case. Lowercase, every run of characters other than letters and digits becomes one `-`, no leading or trailing `-`, at most 50 characters, cut at a word boundary. "Book Chalet 4 at Cabana" becomes `book-chalet-4-at-cabana`.
|
|
59
|
+
- `{case_id}`: the case id in lowercase kebab-case. It is filled in only in `case_id` mode. In `case_text` mode drop this placeholder and the separator next to it. Never invent an id.
|
|
60
|
+
|
|
61
|
+
A missing or empty `branch_pattern` uses `e2e/{title_slug}` in `case_text` mode and `e2e/{case_id}-{title_slug}` in `case_id` mode, for example `e2e/book-chalet-4-at-cabana`. Do not ask.
|
|
62
|
+
|
|
63
|
+
A value with no placeholder is a literal branch name and is used as it is. It puts every case on that one branch, so the template recommends a placeholder.
|
|
64
|
+
|
|
65
|
+
Run `git check-ref-format --branch` on the final name. If Git accepts it, use it. If Git rejects it, ask once for another name.
|
|
66
|
+
|
|
67
|
+
From fresh `main`:
|
|
68
|
+
|
|
69
|
+
```bash
|
|
70
|
+
git fetch origin
|
|
71
|
+
git checkout main
|
|
72
|
+
git pull origin main
|
|
73
|
+
git checkout -b <branch from branch_pattern>
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
If that branch already exists on the remote, check it out and pull. Do not create a second branch. The pull request targets `main`. Never auto-merge.
|
|
77
|
+
|
|
78
|
+
Commit messages: `fix: <short explanation>`.
|
|
79
|
+
|
|
80
|
+
## Docs, then code
|
|
81
|
+
|
|
82
|
+
Read `png_dir` from the settings file. Read the settings file for where page objects, flows, and the spec live.
|
|
83
|
+
|
|
84
|
+
1. `overview.md`, Gherkin under `specs/`, page contracts under `pages/`. Copy each expected result into `## Expected behavior`. Locator hints in `## Elements` may change between runs. Expected-behavior text may not. `## Reference` says "Captured from the running app" and cites `screenshots/<state>.png` under `png_dir`. Do not invent a Figma node. `## Visible copy` records the exact on-screen title, button labels, placeholders, tab names, and toast text. `## Test data` lists each value a step types or selects, and whether the case gave it or the run chose it ([Test data rules](../shared/pr-evidence.md#test-data)). `## States` has one block per distinct screen (open picker, filled chip, next tab, toast), each naming its screenshot and case step. The overview coverage table has the step, the action, the page contract, and the screenshot. No Figma column. One modal stays one page contract. A different dialog gets its own page contract.
|
|
85
|
+
2. Page objects. Reuse existing ones. Read the settings file for where they live.
|
|
86
|
+
3. Flows. Reuse existing ones. Pass `testInfo`. Keep the values a flow types in one named constant, so the spec, the page contract, and the pull request use the same ones. Before each screenshot:
|
|
87
|
+
|
|
88
|
+
```typescript
|
|
89
|
+
testInfo.annotations.push({
|
|
90
|
+
type: 'case-step',
|
|
91
|
+
description: '1. <action from the case step>',
|
|
92
|
+
});
|
|
93
|
+
testInfo.annotations.push({
|
|
94
|
+
type: 'page-contract',
|
|
95
|
+
description: '<page contract path from the settings file>',
|
|
96
|
+
});
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
4. Thin spec. `case_id` title: `<case-id> <case title>`. `case_text` title: the trimmed title, with no invented id. Import `test` from the fixture named in the settings file.
|
|
100
|
+
|
|
101
|
+
Before the final click of a step that enters data, assert each entered value and each toggled control. Take the filled-state screenshot at that point.
|
|
102
|
+
|
|
103
|
+
Prefer `getByRole`, `getByLabel`, `getByPlaceholder`. Wait with `expect(...).toBeVisible()` or `toHaveURL()`.
|
|
104
|
+
|
|
105
|
+
On each run, compare the screenshot to the page contract's Elements, Visible copy, and the expected result. Do not edit the app.
|
|
106
|
+
|
|
107
|
+
After a green run, copy a PNG into `png_dir` only when it passes the Screenshot gate in [pr-evidence.md](../shared/pr-evidence.md). The page contract and the overview step-coverage table cite those files. A step with no image cites none. Do not write a second copy under a case folder. Do not write docs from the spec on every run.
|
|
108
|
+
|
|
109
|
+
## Assertions
|
|
110
|
+
|
|
111
|
+
Follow the word rule in [pr-evidence.md](../shared/pr-evidence.md). The same rule is repeated here.
|
|
112
|
+
|
|
113
|
+
The spec asserts functional outcomes: the control is present, the modal closes, the preview updates, the chart dot appears, and the other visible behavior in the expected result. Copy that behavior from the case into the assertion, including awkward grammar.
|
|
114
|
+
|
|
115
|
+
Do not put the quoted case sentence in an assertion when, after lowercasing, stripping punctuation, and dropping the filler words `a`, `the`, `has`, and `been`, the remaining words match in the same order. Record the case sentence and the screen sentence in the steps table and in the pull request comments. The verdict for that quoted sentence is not `pass`. Example: case **"Your diary entry has successfully created"** and screen **Your Diary entry successfully created!**
|
|
116
|
+
|
|
117
|
+
That difference is not an exact-text pass. Functional checks in the same step stay `pass` when they pass. State that the original exact sentence was not a passing assertion.
|
|
118
|
+
|
|
119
|
+
When the remaining words do not match in the same order, the text difference is unresolved. Assert the case sentence and leave that assertion failing. The spec stays red. Do not treat it as a minor text difference.
|
|
120
|
+
|
|
121
|
+
Do not rewrite the expected sentence. Do not change an assertion so the exact case sentence appears to pass. `assertion_edits` stays 0. Comparing with this rule is not an assertion edit. A green run bought by editing the expected sentence is a failed skill run. Revert it and open the mismatch pull request.
|
|
122
|
+
|
|
123
|
+
When this minor difference is the only issue, and required screenshot evidence meets the acceptance criteria, the pull request label is `pr_label_minor_text_diff` from the settings file. The default is `e2e:minor-text-diff`. It is not `e2e:ready`.
|
|
124
|
+
|
|
125
|
+
A different message (another verb, a missing screen, a missing control) is a functional mismatch. Leave the functional assertion. The spec stays red. A failed functional assertion is a mismatch. It is not, by itself, a confirmed product regression.
|
|
126
|
+
|
|
127
|
+
Take the step PNG of the control that proves the step (modal, open picker, toast) while that result is still on screen. A toast that auto-dismisses is captured while it is visible.
|
|
128
|
+
|
|
129
|
+
## Screenshot gate
|
|
130
|
+
|
|
131
|
+
Follow [pr-evidence.md](../shared/pr-evidence.md).
|
|
132
|
+
|
|
133
|
+
Look at the PNG before it is copied into `png_dir` or linked in the pull request.
|
|
134
|
+
|
|
135
|
+
- Keep it when the image shows the control or change named in that row (open modal, open picker, filled chip, new tab, toast).
|
|
136
|
+
- A picture that matches an earlier step does not prove this step.
|
|
137
|
+
- A step that enters or selects data needs the filled state on screen: the values and the checked controls, before the final click. When the step also submits, add a second image of the result. Links are `step-03a` and `step-03b`. An empty form or a result card alone does not prove the entry.
|
|
138
|
+
- One retry only: capture the proving control (`locator.screenshot` or a clip of that control), not the same full viewport.
|
|
139
|
+
- If the second image still does not show the step, omit the file. The table row stays. The Screenshot cell is `—`. No image heading for that step.
|
|
140
|
+
- An omitted earlier image is not, by itself, a failed assertion.
|
|
141
|
+
- The required end-state screenshot must show the expected end state. A narrow wording difference on that image does not, by itself, mean the screenshot missed the expected end state. If the image does not show the expected end state, do not publish it as passing evidence. That screenshot finding does not, by itself, prove that a functional assertion failed. Report the findings separately. Either finding blocks `e2e:ready`. A required end-state screenshot that misses the expected end state selects `e2e:mismatch`, unless the only issue in the run is the narrow text difference.
|
|
142
|
+
- Do not change the expected result to match a useless screenshot.
|
|
143
|
+
|
|
144
|
+
`assertion_edits` stays 0. Rewriting the expected sentence so the spec goes green is an assertion edit. Comparing with the word rule above is not. A green run bought by editing the expected sentence is a failed skill run. Revert it and open the mismatch pull request.
|
|
145
|
+
|
|
146
|
+
## Run loop
|
|
147
|
+
|
|
148
|
+
Read `test_command` from the settings file. Run that command for this spec with retries off. Never install browsers. The browser is already installed.
|
|
149
|
+
|
|
150
|
+
The five-run ceiling is for the whole spec, not five per step. Each command is one run. Retries off disables the runner's automatic repeat of the same failure. Stop early when the expected result passes.
|
|
151
|
+
|
|
152
|
+
- Locator or timing: the control named by the action is not on screen yet. Change the locator or the wait. Run again.
|
|
153
|
+
- Narrow text difference: the on-screen sentence matches the case under the word rule in Assertions. Record the screen sentence. Do not mark the exact sentence `pass`. Continue the later sentences of that step.
|
|
154
|
+
- Behavior: a found control's text or state is a different result. Stop on that run. Do not spend the remaining attempts searching for a different string. Leave the assertion. Do not treat later steps as passed.
|
|
155
|
+
|
|
156
|
+
## Pull request
|
|
157
|
+
|
|
158
|
+
One PR to `main`. Never auto-merge. Evidence rules, the body layout, and the steps table are in [pr-evidence.md](../shared/pr-evidence.md). In the steps table the Actual cell is `✅ As expected` when it equals the expected result. Only a difference carries text.
|
|
159
|
+
|
|
160
|
+
Read `pr_mode`. A missing value means `create_pr_for_review`. `ask_before_pr` stops before a pull request for a functional mismatch, an unresolved difference, or a failed final-screenshot gate. Stop and ask in chat. A narrow minor text difference does not trigger that stop. If the discrepancy remains, create the pull request only after the user explicitly authorizes it. When the expected behavior has not changed, preserve the failing assertion.
|
|
161
|
+
|
|
162
|
+
| Outcome | Label | PR |
|
|
163
|
+
|---------|--------|----|
|
|
164
|
+
| Functional assertions passed, the expected behavior matches, and required screenshot evidence meets the acceptance criteria. `assertion_edits` is 0 | `pr_label_ready` from the settings file, default `e2e:ready` | Open for review. Not a draft |
|
|
165
|
+
| Functional behavior matches, required screenshots meet the acceptance criteria, and the only difference is the narrow text rule | `pr_label_minor_text_diff` from the settings file, default `e2e:minor-text-diff` | Open for review. Not a draft. Not `e2e:ready` |
|
|
166
|
+
| A functional or unresolved mismatch, or required screenshot evidence does not show the expected end state | `pr_label_mismatch` from the settings file, default `e2e:mismatch` | Draft. A failed assertion stays red |
|
|
167
|
+
|
|
168
|
+
Screenshot links in the body are head-branch blob and raw URLs. The path is `png_dir` from the settings file. A relative path does not open the file. Link text stays `step-01`. A step whose image failed the Screenshot gate has `—` in the Screenshot cell and no image block. Assertion findings and screenshot findings are separate. A screenshot failure does not, by itself, prove that a functional assertion failed. When the only issue is the narrow text rule, the body records both sentences and states that the exact sentence was not a passing assertion. The agent does not edit the case. That difference is not an assertion edit. Steps that never ran have no PNG. Do not commit `test-results/` or the HTML report folder. Do not use `test.fixme`. A red pull-request check is the expected CI result on a mismatch. If GitHub access is missing, report that and do not claim a pull request exists.
|
|
169
|
+
|
|
170
|
+
Do not edit the nightly workflow.
|
|
171
|
+
|
|
172
|
+
## Tool contract
|
|
173
|
+
|
|
174
|
+
`add_from_steps`
|
|
175
|
+
|
|
176
|
+
- Inputs: exactly one of `case_id` or `case_text`, and the settings file `.e2e-qa-pilot/settings.md`
|
|
177
|
+
- Result: one label, `e2e:ready`, `e2e:minor-text-diff`, or a draft `e2e:mismatch`, unless `ask_before_pr` is still waiting for an answer or for explicit authorization
|
|
178
|
+
- The agent still reads the screen. A narrow minor text difference does not stop the run.
|