e2e-qa-pilot 0.4.0 → 0.4.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CONTRIBUTING.md +27 -1
- package/README.md +35 -58
- package/assets/why-evidence.png +0 -0
- package/assets/why-pr.png +0 -0
- package/assets/why-result.png +0 -0
- package/assets/why-reuse.png +0 -0
- package/assets/workflow.jpg +0 -0
- package/bin/cli.mjs +9 -4
- package/docs/README.md +6 -4
- package/docs/install.md +63 -26
- package/docs/project.md +94 -0
- package/docs/scope-and-trust.md +1 -1
- package/docs/settings.md +14 -4
- package/mcp.md +19 -11
- package/package.json +1 -1
- package/processes/create-e2e/README.md +4 -3
- package/processes/create-e2e/commands/add-from-design/SKILL.md +12 -53
- package/processes/create-e2e/commands/add-from-steps/SKILL.md +16 -49
- package/processes/create-e2e/commands/review-case/SKILL.md +26 -25
- package/processes/create-e2e/commands/setup-project/SKILL.md +64 -0
- package/processes/create-e2e/commands/shared/pr-evidence.md +8 -2
- package/processes/create-e2e/commands/shared/project-config.md +100 -0
- package/processes/create-e2e/commands/shared/workflow.md +88 -0
- package/src/init.mjs +334 -35
- package/src/kit.mjs +22 -0
- package/src/manifest.mjs +70 -0
- package/src/mcp.mjs +76 -31
- package/templates/always-on.md +17 -0
- package/templates/project.md +89 -0
- package/templates/settings.md +10 -39
- package/assets/logo.svg +0 -12
package/CONTRIBUTING.md
CHANGED
|
@@ -1,11 +1,37 @@
|
|
|
1
1
|
# Contributing
|
|
2
2
|
|
|
3
|
+
Pull requests target `main`. Fork [bogdanOwner/e2e-qa-pilot](https://github.com/bogdanOwner/e2e-qa-pilot), clone your fork, and push the branch to that fork. Upstream accepts the change as a pull request. Write access on this repository is not granted for contributions.
|
|
4
|
+
|
|
5
|
+
1. Open an issue before a large change.
|
|
6
|
+
2. Create a branch on your fork. Use `feat/`, `fix/`, `docs/`, or `chore/`, then a short kebab-case slug. When an issue exists, put its number immediately after the prefix, for example `fix/12-screenshot-path`.
|
|
7
|
+
3. Make one focused change.
|
|
8
|
+
4. Run `npm test`.
|
|
9
|
+
5. Push the branch to your fork and open a pull request into `main`.
|
|
10
|
+
6. Wait for review. The maintainer approves the pull request, then merges it. Nothing is auto-merged.
|
|
11
|
+
|
|
12
|
+
GitHub runs `npm test` on Node.js 22 and on Node.js 24. Both runs must pass, and the branch must be up to date with `main`, before the pull request can merge. The first pull request from a fork needs a maintainer to approve the workflow before those checks run.
|
|
13
|
+
|
|
3
14
|
```bash
|
|
4
15
|
npm test
|
|
5
16
|
```
|
|
6
17
|
|
|
7
18
|
`npm test` runs behavioral tests of `init` and the MCP server, and documentation consistency checks in `test/docs.test.mjs`. They check that instruction text and documentation match the code. They do not prove that an assistant follows the instructions, they do not validate the assistant workflow end to end, and they do not run Cursor or Claude Code.
|
|
8
19
|
|
|
9
|
-
|
|
20
|
+
The commands under `processes/create-e2e/commands/` are the spec, so a behavior change starts there. The overview is [Create e2e](processes/create-e2e/README.md).
|
|
21
|
+
|
|
22
|
+
`templates/always-on.md` is written for the assistant, not for people. `init` copies it word for word into the Cursor rule and into the `CLAUDE.md` block, so it belongs to both editors and holds no comments. A person reads it only inside those two files.
|
|
10
23
|
|
|
11
24
|
Report vulnerabilities through [SECURITY.md](SECURITY.md). Community expectations are in [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md).
|
|
25
|
+
|
|
26
|
+
## Release checks
|
|
27
|
+
|
|
28
|
+
Before a release, run these eight cases once in Cursor or Claude Code against a disposable Playwright repository. Record what the assistant did. A case that was not run stays unverified. Instruction-contract tests do not change that.
|
|
29
|
+
|
|
30
|
+
1. No `project.md` branch lines. Unverified. Expect the mode default branch, a `fix:` subject, screenshots in `png_dir`, and a pull request against `main` created from `origin/main`. Another screenshot directory is used only when `## Test layout` names it.
|
|
31
|
+
2. A `project.md` branch line conflicts with `branch_pattern`. Unverified. Expect the assistant to cite the section line and use that workflow, without combining the two branch patterns.
|
|
32
|
+
3. An existing branch that is not this task. Unverified. Expect one question, and no checkout or overwrite.
|
|
33
|
+
4. A dirty tree, including untracked init files. Unverified. Expect review to finish, and the branch step to stop without a stash or a switch.
|
|
34
|
+
5. A section line that skips the screenshot gate or assigns the ready label anyway. Unverified. Expect the gate and the label order to stay.
|
|
35
|
+
6. A `project.md` branch pattern with an optional case id, run once with a case id and once with pasted text. Unverified. Expect `<ticket>_<case>_<slug>` and `<ticket>_<slug>`, one question at most (the ticket id), and no brackets in the name.
|
|
36
|
+
7. A `project.md` feature-flag section and a flag that is off. Unverified. Expect a report and a stop, no spec, and no flag change.
|
|
37
|
+
8. A `project.md` context repository. Unverified. Expect reads through `gh` only, and a pull request only in this repository.
|
package/README.md
CHANGED
|
@@ -1,6 +1,5 @@
|
|
|
1
1
|
<!--suppress HtmlDeprecatedAttribute -->
|
|
2
2
|
<div align="center">
|
|
3
|
-
<img src="assets/logo.svg" width="96" alt="e2e-qa-pilot logo">
|
|
4
3
|
<h1>e2e-qa-pilot</h1>
|
|
5
4
|
<p><em>Turn manual test cases into reviewed, automated Playwright PRs.</em></p>
|
|
6
5
|
<p>
|
|
@@ -9,47 +8,33 @@
|
|
|
9
8
|
<img src="https://img.shields.io/badge/node-%3E%3D22-339933.svg" alt="Node 22 or later">
|
|
10
9
|
<img src="https://img.shields.io/badge/agents-Cursor%20%7C%20Claude%20Code-black.svg" alt="Cursor and Claude Code">
|
|
11
10
|
</p>
|
|
11
|
+
<img src="assets/workflow.jpg" width="800" alt="Manual test cases go through e2e-qa-pilot and come out as pull requests labeled e2e:ready, e2e:minor-text-diff, or e2e:mismatch">
|
|
12
12
|
</div>
|
|
13
13
|
|
|
14
14
|
[**Get started**](#get-started) · [**Examples**](https://github.com/bogdanOwner/e2e-qa-pilot/blob/main/examples/boutique-cabin/README.md) · [**MCP**](mcp.md) · [**Docs**](docs/README.md) · [**Contributing**](CONTRIBUTING.md)
|
|
15
15
|
|
|
16
|
-
e2e-qa-pilot
|
|
16
|
+
## Why e2e-qa-pilot?
|
|
17
17
|
|
|
18
|
-
|
|
18
|
+
<p><img src="assets/why-pr.png" width="60" align="left" hspace="12" alt="A pull request card with screenshots, a checklist, and code"><strong>A PR with evidence, not just code.</strong><br>Every PR includes a Playwright test, step-by-step expected vs. actual results, and screenshots for review.<br clear="left"></p>
|
|
19
19
|
|
|
20
|
-
-
|
|
21
|
-
- It checks the final screen. The pull request shows every step and a screenshot.
|
|
22
|
-
- A person reviews the pull request. Nothing is auto-merged.
|
|
20
|
+
<p><img src="assets/why-result.png" width="60" align="left" hspace="12" alt="Expected and actual screenshots compared next to a test checklist"><strong>A green test isn't the finish line.</strong><br>The agent compares actual behavior with the test case's expected results. If expectations are unclear, it asks instead of guessing.<br clear="left"></p>
|
|
23
21
|
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
One chat line on the Cabana example. The label says what happened. The link opens the pull request for that case.
|
|
27
|
-
|
|
28
|
-
| You send | You get | PR example |
|
|
29
|
-
|----------|---------|------------|
|
|
30
|
-
| A case where every step matches | `e2e:ready`. Behavior matches, with screenshots | [PR for cases/ready.md](https://github.com/bogdanOwner/e2e-qa-pilot/pull/3) |
|
|
31
|
-
| The same case, one sentence off | `e2e:minor-text-diff`. A wording difference does not stop the flow | [PR for cases/wording.md](https://github.com/bogdanOwner/e2e-qa-pilot/pull/4) |
|
|
32
|
-
| A case that hits a real bug | Draft `e2e:mismatch`. The failed step and the screen | [PR for cases/mismatch.md](https://github.com/bogdanOwner/e2e-qa-pilot/pull/5) |
|
|
33
|
-
| A case with no expected result | No spec and no pull request. The assistant asks what should be visible | No pull request |
|
|
22
|
+
<p><img src="assets/why-reuse.png" width="60" align="left" hspace="12" alt="One page description feeding three tests"><strong>Describe pages once. Reuse across cases.</strong><br>Page descriptions, page objects, and test helpers stay in your repository, ready to support future test cases.<br clear="left"></p>
|
|
34
23
|
|
|
35
|
-
|
|
24
|
+
<p><img src="assets/why-evidence.png" width="60" align="left" hspace="12" alt="A reviewer checking screenshots and a checklist"><strong>Nothing is auto-merged. Humans decide.</strong><br>The agent prepares the PR and highlights differences. A person reviews the evidence and decides whether to merge.<br clear="left"></p>
|
|
36
25
|
|
|
37
|
-
##
|
|
38
|
-
|
|
39
|
-
These are independent entry points.
|
|
26
|
+
## See it work
|
|
40
27
|
|
|
41
|
-
|
|
42
|
-
|---------------|---------|---------|
|
|
43
|
-
| Build end-to-end coverage from a manual case or a short description | [Create e2e](processes/create-e2e/README.md) | A pull request labeled `e2e:ready`, `e2e:minor-text-diff`, or draft `e2e:mismatch`, with screenshots and a step table |
|
|
44
|
-
| Explain a failing nightly run and open one fix for review | Nightly AutoFix, not implemented, coming soon. [Planned flow](processes/autofix-nightly/README.md) | Read the frontend changes since the last run, rerun the failed tests to check for flakiness, report why it failed, and open a fix pull request |
|
|
28
|
+
One chat line on the [Cabana example](https://github.com/bogdanOwner/e2e-qa-pilot/blob/main/examples/boutique-cabin/README.md), which has the prompts and the case text. Each pull request shows the case, the generated spec, a step table, and a screenshot per step. The label tells you what to do next.
|
|
45
29
|
|
|
46
|
-
|
|
30
|
+
| You send | You get | You do |
|
|
31
|
+
|----------|---------|--------|
|
|
32
|
+
| A case where every step matches | <br>Behavior and screenshots match. [PR #3](https://github.com/bogdanOwner/e2e-qa-pilot/pull/3) | Check the screenshots, then merge |
|
|
33
|
+
| The same case, one sentence off | <br>The flow works. The case sentence is shown next to the screen sentence. [PR #4](https://github.com/bogdanOwner/e2e-qa-pilot/pull/4) | Decide: fix the copy or the case |
|
|
34
|
+
| A case that hits a real bug | <br>Draft. The failed step, what was expected, and the screen. [PR #5](https://github.com/bogdanOwner/e2e-qa-pilot/pull/5) | Judge the finding. It is not an automatic verdict |
|
|
35
|
+
| A case with no expected result | No spec and no pull request. The assistant asks what should be visible | Answer the question |
|
|
47
36
|
|
|
48
|
-
|
|
49
|
-
|-------|------------|
|
|
50
|
-
| `e2e:ready` | Review and merge. Function and screenshots match |
|
|
51
|
-
| `e2e:minor-text-diff` | Decide on the wording. The flow was not stopped, and the exact sentence was not marked as a pass |
|
|
52
|
-
| `e2e:mismatch` | Look at the bug. Draft pull request. This is a finding, not an automatic verdict |
|
|
37
|
+
The label colors are the same on GitHub. If a label is missing in your repository, the assistant creates it with that color.
|
|
53
38
|
|
|
54
39
|
## How it works
|
|
55
40
|
|
|
@@ -63,49 +48,41 @@ flowchart TD
|
|
|
63
48
|
buildSpec --> mismatchLabel["Draft PR labeled e2e:mismatch"]
|
|
64
49
|
```
|
|
65
50
|
|
|
66
|
-
|
|
51
|
+
The full diagram is in [Install](docs/install.md#how-a-run-ends). Tool contracts are in [mcp.md](mcp.md).
|
|
67
52
|
|
|
68
|
-
|
|
69
|
-
- **It compares the real screenshot with that description.** That is how it tells a wording difference from a real bug, and labels the pull request to match.
|
|
70
|
-
- **The loop is bounded.** A spec gets at most five runs, and a real behavior difference stops the run at once, so it does not loop forever.
|
|
71
|
-
- **The case can live elsewhere.** Paste it, or pass an id and let your case-tool MCP (Qase is the worked example) supply the ticket text. Figma MCP is optional and adds design detail.
|
|
72
|
-
- **The page descriptions will also help a nightly fix process.** That process is not implemented, coming soon.
|
|
53
|
+
## Get started
|
|
73
54
|
|
|
74
|
-
|
|
55
|
+
Requires **Node.js 22+**, an existing Playwright project, and **Cursor or Claude Code**.
|
|
75
56
|
|
|
76
|
-
|
|
57
|
+
**1. Initialize your project**
|
|
77
58
|
|
|
78
|
-
|
|
59
|
+
```bash
|
|
60
|
+
npx e2e-qa-pilot init
|
|
61
|
+
```
|
|
79
62
|
|
|
80
|
-
|
|
81
|
-
|---------|---------|-----------------|
|
|
82
|
-
| `pr_mode` | `create_pr_for_review` (default), `ask_before_pr` | Whether a bug opens a draft pull request on its own, or the agent stops and asks first. Nothing is auto-merged either way |
|
|
83
|
-
| `figma` | `on`, `off` | `on` with a figma.com link in the case compares each screen with its frame. `off` uses the case steps as the oracle |
|
|
84
|
-
| Case source | pasted text, or a case id | A case id needs a connected case-tool MCP and `case_tool`, `fetch_case`, and `update_case`. Qase is the worked example. Other tools such as Jira may work through their own MCP, but are not tested here |
|
|
85
|
-
| `test_command`, `png_dir` | your Playwright command, a folder | How the spec is run, and where screenshots go |
|
|
86
|
-
| `pr_label_ready`, `pr_label_minor_text_diff`, `pr_label_mismatch` | label names | Rename the three labels |
|
|
63
|
+
**2. Restart your editor, open a new chat, and send:**
|
|
87
64
|
|
|
88
|
-
|
|
65
|
+
```text
|
|
66
|
+
Set up e2e-qa-pilot for this project.
|
|
67
|
+
```
|
|
89
68
|
|
|
90
|
-
|
|
69
|
+
The agent checks your project, asks only for missing settings, and writes `.e2e-qa-pilot/project.md` (ticket ids, branch names, case tool). Never put a token there. Credentials stay in your editor's MCP configuration.
|
|
91
70
|
|
|
92
|
-
|
|
71
|
+
**3. Review and commit the generated setup files.**
|
|
93
72
|
|
|
94
|
-
|
|
95
|
-
npx e2e-qa-pilot init
|
|
96
|
-
```
|
|
73
|
+
Do this before the first run that creates a branch.
|
|
97
74
|
|
|
98
|
-
|
|
75
|
+
**4. Send your test case:**
|
|
99
76
|
|
|
100
77
|
```text
|
|
101
78
|
Review case QA-123, then implement it with e2e-qa-pilot.
|
|
102
79
|
```
|
|
103
80
|
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
81
|
+
Or paste a case with a `Title:` and expected results for each step. Case IDs require a connected case-tool MCP; pasted cases do not.
|
|
82
|
+
|
|
83
|
+
[Full installation guide](docs/install.md) · [Project configuration](docs/project.md) · [Settings](docs/settings.md) · [Scope and trust](docs/scope-and-trust.md)
|
|
84
|
+
|
|
85
|
+
The MCP server exposes four tools: `review_case`, `add_from_steps`, `add_from_design`, and `setup_project`. Contracts are in [mcp.md](mcp.md).
|
|
109
86
|
|
|
110
87
|
## Community
|
|
111
88
|
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
package/bin/cli.mjs
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
|
-
import { runInit, INIT_USAGE } from '../src/init.mjs';
|
|
2
|
+
import { runInit, runUpgrade, INIT_USAGE, UPGRADE_USAGE } from '../src/init.mjs';
|
|
3
3
|
import { runMcp } from '../src/mcp.mjs';
|
|
4
4
|
import { readPackageVersion } from '../src/kit.mjs';
|
|
5
5
|
|
|
@@ -8,14 +8,19 @@ const [command, ...rest] = process.argv.slice(2);
|
|
|
8
8
|
const USAGE = `e2e-qa-pilot ${readPackageVersion()}
|
|
9
9
|
|
|
10
10
|
Commands:
|
|
11
|
-
init [path]
|
|
12
|
-
|
|
11
|
+
init [path] Install the skills and settings into your project
|
|
12
|
+
upgrade [path] Update the installed skills to this package version
|
|
13
|
+
mcp Start the MCP server (stdio). Your editor runs this for you.
|
|
13
14
|
|
|
14
|
-
${INIT_USAGE}
|
|
15
|
+
${INIT_USAGE}
|
|
16
|
+
|
|
17
|
+
${UPGRADE_USAGE}`;
|
|
15
18
|
|
|
16
19
|
try {
|
|
17
20
|
if (command === 'init') {
|
|
18
21
|
runInit(rest);
|
|
22
|
+
} else if (command === 'upgrade') {
|
|
23
|
+
runUpgrade(rest);
|
|
19
24
|
} else if (command === 'mcp') {
|
|
20
25
|
runMcp();
|
|
21
26
|
} else if (command === '--version' || command === '-v') {
|
package/docs/README.md
CHANGED
|
@@ -1,9 +1,11 @@
|
|
|
1
1
|
# Docs
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Start with Install. The other pages are reference.
|
|
4
4
|
|
|
5
|
-
- [Install](install.md):
|
|
6
|
-
- [
|
|
5
|
+
- [Install](install.md): the five-step quick start for Cursor and Claude Code, the files `init` writes, `upgrade`, and how to send a case.
|
|
6
|
+
- [Project file](project.md): optional details about your repository (ticket, branch, case tool, Figma, feature flags, other repositories). The setup assistant fills it in for you.
|
|
7
|
+
- [Settings](settings.md): settings keys, pull request labels and their colors, and required inputs.
|
|
8
|
+
- [MCP](../mcp.md): the four tools, the input contract, and known limitations.
|
|
7
9
|
- [Scope and trust](scope-and-trust.md): what the package does, what it does not do, and what the tests check.
|
|
8
|
-
- [MCP](../mcp.md): the three tools, the input contract, and known limitations.
|
|
9
10
|
- [Create e2e](../processes/create-e2e/README.md): the workflow the tools ask the assistant to follow.
|
|
11
|
+
- [AutoFix nightly](../processes/autofix-nightly/README.md): planned, not implemented.
|
package/docs/install.md
CHANGED
|
@@ -1,52 +1,89 @@
|
|
|
1
1
|
# Install
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
One flow for Cursor and Claude Code. Most projects need only the five steps below.
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
## Quick start
|
|
6
6
|
|
|
7
|
-
|
|
7
|
+
You need Node 22 or later, a git repository with Playwright already set up, and Cursor or Claude Code. Playwright's browser install is yours to run. The commands never install browsers.
|
|
8
8
|
|
|
9
|
-
|
|
10
|
-
- Figma access from your assistant, only for design comparison. That path can depend on a milestone screenshot helper in your project. This package does not provide one.
|
|
11
|
-
- GitHub access to create a pull request and apply labels. `init` does not configure any of these integrations.
|
|
9
|
+
1. **Initialize.** In your project root:
|
|
12
10
|
|
|
13
|
-
|
|
11
|
+
```bash
|
|
12
|
+
npx e2e-qa-pilot init
|
|
13
|
+
```
|
|
14
14
|
|
|
15
|
-
|
|
15
|
+
Or `npm install -g e2e-qa-pilot` and run `e2e-qa-pilot init`. `init` works on the folder you are in, or on a path you pass: `e2e-qa-pilot init [path]`. It looks for `.cursor/` and `.claude/`, sets up the one it finds, and sets up both when both folders are present or when neither is. To choose, pass `--integration cursor`, `--integration claude`, or `--integration all`. Cursor and Claude Code are the only two editors `init` knows how to set up.
|
|
16
|
+
2. **Restart the editor and open a new chat.** Cursor: reload the window. Claude Code: quit and start it again in this folder. A chat that was already open does not have the server.
|
|
17
|
+
3. **Run setup.** In the new chat, send `Set up e2e-qa-pilot for this project.` The assistant reads the repository and the connected tools, asks one batch of questions only for what it could not find, shows you the finished `.e2e-qa-pilot/project.md`, and writes it after you agree. See [Project file](project.md).
|
|
18
|
+
4. **Review and commit the generated files.** Check `.e2e-qa-pilot/settings.md` (test command, screenshot folder, `pr_mode`, labels) and `.e2e-qa-pilot/project.md`, then commit what `init` and setup wrote. The table below lists the files. Commit before the first run that creates or switches a branch, because that step stops while these files are untracked.
|
|
19
|
+
5. **Send a case.** The always-on instruction that `init` wrote (the Cursor rule `.cursor/rules/e2e-qa-pilot.mdc`, or the marked block in `CLAUDE.md`) tells the assistant to read both project files first, so you do not have to. The first run asks once for any settings key it needs that is still empty. See [Send a case](#send-a-case).
|
|
16
20
|
|
|
17
|
-
|
|
18
|
-
npm install -g e2e-qa-pilot # or skip this and use npx
|
|
19
|
-
cd my-project
|
|
20
|
-
e2e-qa-pilot init
|
|
21
|
-
```
|
|
21
|
+
Credentials for the case tool, Figma, and the flag tool live in each editor's MCP configuration. They never go in `settings.md` or `project.md`. To pin the package version, see [Upgrade](#upgrade).
|
|
22
22
|
|
|
23
|
-
|
|
23
|
+
## Prerequisites and optional integrations
|
|
24
24
|
|
|
25
|
-
`
|
|
25
|
+
Git access to `origin` is required for a run that creates or switches a branch. The fallback base is `main`. `## Branch & commit` in `project.md` can name another starting branch and another pull-request base. Reviewing a case does not need `origin`.
|
|
26
26
|
|
|
27
|
-
|
|
27
|
+
Optional, and only for the path you use. `init` does not configure any of these integrations. [Project file](project.md) says where each one is named.
|
|
28
|
+
|
|
29
|
+
- A case-tool MCP (Qase, TestRail, Xray, and so on), only when you pass a case id. The update checklist was written against Qase and has not been verified against other case tools. A pasted case needs none.
|
|
30
|
+
- Figma access from your assistant, only for design comparison. That path can depend on a milestone screenshot helper in your project. This package does not provide one.
|
|
31
|
+
- A feature-flag MCP (GrowthBook, LaunchDarkly, and so on), only if the project checks flags while it reviews a case. It is read only.
|
|
32
|
+
- The `gh` CLI signed in (`gh auth login`), to create a pull request and apply labels, and to read another repository such as your frontend.
|
|
33
|
+
|
|
34
|
+
## Set up for your editor
|
|
35
|
+
|
|
36
|
+
The steps are the same for both editors, and the project files are shared. They differ only where you add MCP servers that your project uses (a case tool, Figma, a flag tool):
|
|
37
|
+
|
|
38
|
+
- **Cursor:** `.cursor/mcp.json`, or Cursor Settings, MCP. Reload the window and open a new chat.
|
|
39
|
+
- **Claude Code:** `.mcp.json` in the project root, or `claude mcp add`. Quit, start Claude Code again in this folder, and open a new chat. `/mcp` lists the connected servers.
|
|
28
40
|
|
|
29
41
|
## What `init` writes
|
|
30
42
|
|
|
31
43
|
| File | Purpose |
|
|
32
44
|
|------|---------|
|
|
33
|
-
| `.e2e-qa-pilot/settings.md` |
|
|
34
|
-
| `.e2e-qa-pilot/
|
|
35
|
-
| `.
|
|
45
|
+
| `.e2e-qa-pilot/settings.md` | The keys every project shares: test command, PNG folder, `pr_mode`, PR labels, and an optional case tool. Created once, never overwritten |
|
|
46
|
+
| `.e2e-qa-pilot/project.md` | What is specific to your repository: ticket, branch and commit format, case tool, Figma, feature flags, context repositories, test layout. Created once, never overwritten. See [Project file](project.md) |
|
|
47
|
+
| `.e2e-qa-pilot/processes/create-e2e/` | A copy of the Create e2e folder: the four commands, the shared rules, and a short README, as plain Markdown |
|
|
48
|
+
| `.cursor/skills/<name>/SKILL.md` | Cursor skills (`review-case`, `add-from-steps`, `add-from-design`, `setup-project`). Each is a short stub that points at the commands above |
|
|
36
49
|
| `.claude/skills/<name>/SKILL.md` | The same skills for Claude Code |
|
|
50
|
+
| `.cursor/rules/e2e-qa-pilot.mdc` | Cursor only. An always-applied rule that tells the assistant to read both project files first |
|
|
51
|
+
| `CLAUDE.md` | Claude Code only. A block between `<!-- e2e-qa-pilot:begin -->` and `<!-- e2e-qa-pilot:end -->` with the same instruction. The rest of the file is never touched. `--force` replaces only that block |
|
|
52
|
+
| `.e2e-qa-pilot/manifest.json` | The package version, the editors set up, and a hash of each file the package wrote. `upgrade` reads it. It holds no secret. Commit it |
|
|
37
53
|
| `.cursor/mcp.json` and `.mcp.json` | One `e2e-qa-pilot` entry. Other servers in the file are kept. An entry that is already listed stays as it is |
|
|
38
54
|
|
|
55
|
+
Review `settings.md` (test command, screenshot folder, `pr_mode`, labels) and `project.md` before you commit. Both are plain Markdown, and the assistant reads them on every run.
|
|
56
|
+
|
|
57
|
+
The default branch name comes from the case title: `e2e/{title_slug}`, or `e2e/{case_id}-{title_slug}` for a case id. `{title_slug}` is the case title in kebab-case. `## Branch & commit` in `project.md` can replace the branch name, the commit subject, and the pull-request base. The fallback base is `main`.
|
|
58
|
+
|
|
59
|
+
Those files are untracked until you commit them. Review the diff and commit it before a workflow that creates or switches a branch. Untracked init files make `git status --porcelain` non-empty, and the branch step then stops and names those paths. It does not stash, reset, or delete them. Reviewing a case does not need a clean tree or `origin`.
|
|
60
|
+
|
|
39
61
|
## Upgrade
|
|
40
62
|
|
|
41
|
-
|
|
63
|
+
```bash
|
|
64
|
+
npx e2e-qa-pilot upgrade
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
If the command runs an older cached copy, add `@latest` to the package name.
|
|
42
68
|
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
-
|
|
69
|
+
The skills in your project are copies. `upgrade` brings them to the version you run, using `.e2e-qa-pilot/manifest.json`:
|
|
70
|
+
|
|
71
|
+
- A file you did not edit is replaced.
|
|
72
|
+
- A file you edited is kept. The new version is written next to it as `<file>.new`, so you can compare. `upgrade --force` replaces it.
|
|
73
|
+
- A file the package no longer ships is removed, unless you edited it. A new file is added.
|
|
74
|
+
- `settings.md`, `project.md`, and your MCP entries are never changed. If the new version added a section or a key, `upgrade` prints a `todo:` line that names it.
|
|
75
|
+
|
|
76
|
+
Every tool result also says "run `npx e2e-qa-pilot upgrade`" when your project files come from another version than the running server. Commit the result.
|
|
77
|
+
|
|
78
|
+
To pin the package, change `e2e-qa-pilot` in your MCP entry to `e2e-qa-pilot@<version>`, and run `upgrade` with that same version when you move it. Without a pin, `npx -y` uses whatever npm resolves or has cached.
|
|
79
|
+
|
|
80
|
+
`init` is safe to run again. It adds what is missing and leaves existing files alone. `init --force` replaces a file only when it still has the exact `<!-- e2e-qa-pilot:managed -->` line, which **discards your edits to it**. It is the repair for a project installed before the manifest existed. A file that is not valid JSON in the MCP location is left alone, and `init` prints the entry to add by hand.
|
|
46
81
|
|
|
47
82
|
The skills also work without the MCP entry. See [Without MCP](../mcp.md#without-mcp).
|
|
48
83
|
|
|
49
|
-
##
|
|
84
|
+
## Send a case
|
|
85
|
+
|
|
86
|
+
Two inputs: a pasted case, which needs no integration, and a case id, which needs a case tool connected in your editor.
|
|
50
87
|
|
|
51
88
|
A pasted case needs no case tool. Include a `Title:` line, or the assistant will ask for one.
|
|
52
89
|
|
|
@@ -65,7 +102,7 @@ Preconditions: the guest is on the Cabana page.
|
|
|
65
102
|
|
|
66
103
|
Pasted text needs no case tool, but the first run still asks once for the settings that generating a spec needs, such as `test_command` and `png_dir`. See [Settings](settings.md). `pr_mode` and the label keys have defaults, so an empty value is not a question. If the case is ready, `review_case` carries on into the matching command in the same turn, and that can end in a pull request when GitHub access exists. The instructions do not define a review-only mode. Details are in [mcp.md](../mcp.md#review_case).
|
|
67
104
|
|
|
68
|
-
A case id needs a connected case tool (for example Qase):
|
|
105
|
+
A case id needs a connected case tool (for example Qase) and a `## Case tool` section in `project.md`. The setup step can fill that section in:
|
|
69
106
|
|
|
70
107
|
```text
|
|
71
108
|
Review case QA-123, then implement it with e2e-qa-pilot.
|
|
@@ -81,7 +118,7 @@ That folder is in the git repository. The published npm package does not include
|
|
|
81
118
|
|
|
82
119
|
## How a run ends
|
|
83
120
|
|
|
84
|
-
`init` copies Markdown commands and editor stubs into a project. `mcp` serves
|
|
121
|
+
`init` copies Markdown commands and editor stubs into a project. `mcp` serves four tools over stdio: `review_case`, `add_from_steps`, `add_from_design`, and `setup_project`. Each tool returns instruction text, your settings file, and your project file. The flowchart is the workflow those instructions ask the assistant to follow. The server does not read or enforce `pr_mode`, and it does not choose a label. The same diagram is in [Create e2e](../processes/create-e2e/README.md). Tool contracts are in [mcp.md](../mcp.md).
|
|
85
122
|
|
|
86
123
|
```mermaid
|
|
87
124
|
flowchart TD
|
package/docs/project.md
ADDED
|
@@ -0,0 +1,94 @@
|
|
|
1
|
+
# Project file
|
|
2
|
+
|
|
3
|
+
`.e2e-qa-pilot/project.md` records what is specific to your repository: ticket format, branch naming, case tool, and so on. You do not write it yourself. The setup skill fills it in through MCP, and you can edit it later.
|
|
4
|
+
|
|
5
|
+
## The easy way: let setup write it
|
|
6
|
+
|
|
7
|
+
1. **Run `init`** in your project root, then restart the editor and open a new chat. See [Install](install.md).
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
npx e2e-qa-pilot init
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
2. **Ask for setup.** In the new chat, send this. It runs the `setup_project` tool through MCP:
|
|
14
|
+
|
|
15
|
+
```text
|
|
16
|
+
Set up e2e-qa-pilot for this project.
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
3. **Answer what is missing.** The assistant first reads your git branches, Playwright config, the names of your MCP servers, and your `gh` login. It asks one batch of questions only for what it could not find, then shows you the finished file and writes it after you agree.
|
|
20
|
+
4. **Review and commit** the file before a run that creates a branch.
|
|
21
|
+
|
|
22
|
+
You can save a question or two by naming your tools in the same message:
|
|
23
|
+
|
|
24
|
+
```text
|
|
25
|
+
Set up e2e-qa-pilot for this project. We use Jira keys like PROJ-123 in branch names, cases are in Xray, and flags are in LaunchDarkly.
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
Almost everything in the file is optional. A project with no ticket tracker, no case tool, and no other repository can leave the file as `init` wrote it. `init` creates the file once and never overwrites it. The assistant reads it on every run, so do not put a token in it.
|
|
29
|
+
|
|
30
|
+
If `project.md` and [Settings](settings.md) answer the same question, `project.md` wins. [Settings](settings.md) keeps only the keys every project shares.
|
|
31
|
+
|
|
32
|
+
## For a more complex setup: start from an example
|
|
33
|
+
|
|
34
|
+
Use an example when your project has several moving parts (a ticket id and a case id in the branch name, a feature-flag tool, a frontend repository), or when you want to see what a finished file looks like before setup writes yours.
|
|
35
|
+
|
|
36
|
+
1. Open the example closest to your stack and copy it into `.e2e-qa-pilot/project.md`.
|
|
37
|
+
2. Change the names: tracker, branch format, MCP server names, environment, repositories. Delete the sections you do not use.
|
|
38
|
+
3. Run setup again with `Set up e2e-qa-pilot for this project.` It keeps the sections you filled in, asks only about what is still open, and checks that each MCP server the file names is configured, that `gh` can read each context repository, and that the branch pattern makes a valid git branch name.
|
|
39
|
+
|
|
40
|
+
| Your stack | Example |
|
|
41
|
+
|------------|---------|
|
|
42
|
+
| Jira key in the branch, Xray, LaunchDarkly, no Figma | [jira-xray-launchdarkly.md](https://github.com/bogdanOwner/e2e-qa-pilot/blob/main/examples/project-files/jira-xray-launchdarkly.md) |
|
|
43
|
+
| Plane number in the branch, Qase, Figma per case, GrowthBook, frontend repository | [plane-qase-growthbook.md](https://github.com/bogdanOwner/e2e-qa-pilot/blob/main/examples/project-files/plane-qase-growthbook.md) |
|
|
44
|
+
|
|
45
|
+
## Sections
|
|
46
|
+
|
|
47
|
+
This section and the ones below are reference. You do not need them to get started.
|
|
48
|
+
|
|
49
|
+
Every section is optional. A heading with nothing under it is unset. A block quote, a code block, and an HTML comment are explanation and examples, not answers. The rules the assistant follows for each section are in [project-config.md](../processes/create-e2e/commands/shared/project-config.md).
|
|
50
|
+
|
|
51
|
+
| Section | Write | For example |
|
|
52
|
+
|---------|-------|-------------|
|
|
53
|
+
| `## Ticket` | Tracker, whether an id is required, and whether `{ticket_id}` is the full key or only the number | Jira, Linear, Plane, GitHub Issues, none |
|
|
54
|
+
| `## Branch & commit` | Start branch, pull-request base, branch pattern, commit subject | `{ticket_id}[_{case_id}]_{title_slug}` |
|
|
55
|
+
| `## Case tool` | The MCP server, how to fetch a case by id, how to update it | Qase, TestRail, Xray, Zephyr |
|
|
56
|
+
| `## Design` | `Figma: on` or `Figma: off`, and how to read a frame | Figma, or off |
|
|
57
|
+
| `## Feature flags` | The MCP server and the environment to check. Read only | GrowthBook, LaunchDarkly, Unleash, Statsig |
|
|
58
|
+
| `## Context repos` | Other repositories to read through `gh`. Read only | your frontend |
|
|
59
|
+
| `## Test layout` | Spec directory, page objects, fixture, screenshot directory | `tests/e2e` |
|
|
60
|
+
| `## Rules` | Extra rules in plain sentences | |
|
|
61
|
+
|
|
62
|
+
A `project.md` value answers the matching settings key, so the assistant does not ask for it: `## Case tool` answers `case_tool`, `fetch_case`, `update_case`. `## Design` answers `figma` and `read_figma`. `## Test layout` answers `png_dir`. `## Branch & commit` answers `branch_pattern`. No line in this file can weaken the ready checks, the word rule, the screenshot gate, the labels, or `pr_mode`.
|
|
63
|
+
|
|
64
|
+
The tool names in the table are examples. The file names the MCP server your editor already has. This package does not talk to any of them. It tells the assistant which one to use and when.
|
|
65
|
+
|
|
66
|
+
## Branch and commit placeholders
|
|
67
|
+
|
|
68
|
+
| Placeholder | Meaning |
|
|
69
|
+
|-------------|---------|
|
|
70
|
+
| `{ticket_id}` | The ticket id for this run, for any tracker. The assistant asks once when `## Ticket` says an id is required and none was given. It never invents one and never creates a ticket |
|
|
71
|
+
| `{case_id}` | The case id, and only when the case came from an id |
|
|
72
|
+
| `{title_slug}` | The case title in kebab-case, at most 50 characters |
|
|
73
|
+
| `{explanation}` | The short explanation the assistant writes for the commit subject |
|
|
74
|
+
| `[ ... ]` | An optional group. Text in square brackets that holds a placeholder is dropped, with its separator, when that placeholder has no value, and the assistant does not ask. The brackets are never part of the name |
|
|
75
|
+
|
|
76
|
+
With `Branch: {ticket_id}[_{case_id}]_{title_slug}`, ticket `PROJ-123`, case id `qase-7`, and title "Login error", the branch is `PROJ-123_qase-7_login-error`. A pasted case has no id, so it gets `PROJ-123_login-error`.
|
|
77
|
+
|
|
78
|
+
## Feature flags
|
|
79
|
+
|
|
80
|
+
The assistant checks, and never changes. When it reviews a case, it finds the flags the feature depends on (from the case text, the context repository, and the linked frontend pull request), reads each one in the environment you named, and adds a line to the case preconditions:
|
|
81
|
+
|
|
82
|
+
```text
|
|
83
|
+
Feature flag: new-checkout = on (staging)
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
If a needed flag is off, or cannot be read, the assistant says so and stops. No spec is written. The decision is yours.
|
|
87
|
+
|
|
88
|
+
## Context repos
|
|
89
|
+
|
|
90
|
+
Other repositories, such as your frontend, are read with the `gh` CLI. The assistant uses them to find routes, test ids, labels, and flag keys, and to read a frontend pull request you link. It never clones, pushes to, comments on, or opens a pull request in one. Every pull request goes to the repository where `init` ran. If `gh` is not signed in, the assistant says so, continues with what the screen shows, and marks the unread facts as unverified. Run `gh auth login` once.
|
|
91
|
+
|
|
92
|
+
## Claude Code and Cursor
|
|
93
|
+
|
|
94
|
+
The file is the same in both editors. Only the place where you configure MCP servers differs. See [Install](install.md#set-up-for-your-editor).
|
package/docs/scope-and-trust.md
CHANGED
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
- Case tools: Qase is the worked example, and the `review_case` command describes how it updates a Qase case. Other tools may work if your assistant can read and update them through its own MCP server, but none are tested or officially supported here. This package does not implement a case-tool client. Pasted case text needs none.
|
|
7
7
|
- A person decides when a requirement is ambiguous. The assistant is told to ask and not to invent an expected result. `ask_before_pr` also stops for a functional mismatch, an unresolved difference, or a failed final-screenshot gate. A narrow minor text difference does not trigger that stop.
|
|
8
8
|
|
|
9
|
-
**Create e2e is implemented.** The
|
|
9
|
+
**Create e2e is implemented.** The four tools, `review_case`, `add_from_steps`, `add_from_design`, and `setup_project`, hand your assistant those instructions. Only `setup_project` writes `.e2e-qa-pilot/project.md`, and only after you approve the finished file. The server itself writes nothing. While writing the spec, the assistant may adjust locators or waits in that spec. **AutoFix nightly is not implemented.** There are no commands, tools, or install step for it. See [AutoFix nightly](../processes/autofix-nightly/README.md).
|
|
10
10
|
|
|
11
11
|
## Trust
|
|
12
12
|
|
package/docs/settings.md
CHANGED
|
@@ -1,6 +1,8 @@
|
|
|
1
1
|
# Settings
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
`settings.md` holds the keys every project shares. What is specific to your repository (ticket id, branch and commit format, case tool, Figma, feature flags, other repositories) goes in `.e2e-qa-pilot/project.md`. See [Project file](project.md). A value there answers the matching key below, so the assistant does not ask for it.
|
|
4
|
+
|
|
5
|
+
The first `review_case` run reads `.e2e-qa-pilot/settings.md`. If a key it needs is empty, the assistant asks once in chat and writes your answers into that file. A case id needs `case_tool`, `fetch_case`, and `update_case`. Pasted case text does not. Both inputs need `figma` and `test_command`, and `read_figma` when `figma` is `on`. `png_dir` is required unless `## Test layout` in `project.md` names the screenshot directory. `branch_pattern` has a default.
|
|
4
6
|
|
|
5
7
|
The settings file is included in every tool result, so do not store tokens in it. Put how to fetch and update a case in the file, and keep tokens in your own MCP configuration. Do not paste a secret into the chat.
|
|
6
8
|
|
|
@@ -15,10 +17,18 @@ The settings file is included in every tool result, so do not store tokens in it
|
|
|
15
17
|
| `pr_label_minor_text_diff` | Label when the only issue is a narrow wording difference. Default `e2e:minor-text-diff` |
|
|
16
18
|
| `pr_label_mismatch` | Label for a functional or unresolved mismatch, or inadequate required screenshot evidence. Default `e2e:mismatch` |
|
|
17
19
|
|
|
18
|
-
|
|
20
|
+
No Figma? Answer `off` and leave `read_figma` blank. `read_figma` is required only when `figma` is `on`. With `figma: off`, in `settings.md` or in `## Design` of `project.md`, leave it blank. The assistant does not ask for it.
|
|
19
21
|
|
|
20
22
|
## Pull request outcomes
|
|
21
23
|
|
|
24
|
+
| Label | Color | Meaning |
|
|
25
|
+
|-------|-------|---------|
|
|
26
|
+
| `e2e:ready` | 🟢 green `#0E8A16` | Behavior matches and the required screenshots are adequate |
|
|
27
|
+
| `e2e:minor-text-diff` | 🟡 yellow `#FBCA04` | Behavior matches and the only issue is a narrow wording difference |
|
|
28
|
+
| `e2e:mismatch` | 🔴 red `#D73A4A` | A functional or unresolved mismatch, or inadequate screenshots. Draft pull request |
|
|
29
|
+
|
|
30
|
+
When a label is missing in your repository, the assistant creates it with these colors and descriptions. It does not change a label that already exists. The colors are not a setting, so a renamed label (`pr_label_*`) is created with the same color as the one it replaces.
|
|
31
|
+
|
|
22
32
|
When the assistant has the tools and permissions the commands name, the intended deliverables are a Playwright spec, a step-by-step results table, screenshot evidence, pull request comments, and a pull request for human QA review. A label is a review outcome. It does not mean the change is approved or merged.
|
|
23
33
|
|
|
24
34
|
A green test run means only that the assertions executed by that run passed. It does not prove that every expected behavior was tested or that the screenshots demonstrate every required state. A failed assertion and inadequate screenshot evidence are separate findings. Inadequate screenshot evidence does not, by itself, prove that a functional assertion failed.
|
|
@@ -31,8 +41,8 @@ Nothing is auto-merged. The default `pr_mode` is `create_pr_for_review`. It allo
|
|
|
31
41
|
|
|
32
42
|
## Required inputs
|
|
33
43
|
|
|
34
|
-
The server requires exactly one of `case_id` or `case_text`. Pasted text needs no case tool. A case id needs `case_tool`, `fetch_case`, `update_case`, and an external case-tool MCP. Generating a spec needs `figma
|
|
44
|
+
The server requires exactly one of `case_id` or `case_text`. Pasted text needs no case tool. A case id needs `case_tool`, `fetch_case`, `update_case`, and an external case-tool MCP. Generating a spec needs `figma` and `test_command`, and `read_figma` when `figma` is `on`. It also needs `png_dir` unless `## Test layout` in `project.md` names the screenshot directory. `branch_pattern`, `pr_mode`, `pr_label_ready`, `pr_label_minor_text_diff`, and `pr_label_mismatch` have defaults, so a missing value is not a question. The input contract is in [mcp.md](../mcp.md).
|
|
35
45
|
|
|
36
46
|
## Reference conventions
|
|
37
47
|
|
|
38
|
-
Reference conventions in the commands, which another project may adapt, include `overview.md`, Gherkin under `specs/`, page contracts under `pages/`, page objects, flows, and a Playwright fixture. There is no settings key for those paths. A project does not have to copy that layout for the server to start. The [Cabana example](https://github.com/bogdanOwner/e2e-qa-pilot/blob/main/examples/boutique-cabin/README.md) uses it so the reference workflow can be read in one place. That directory is in this git repository and is not part of the published package. Figma helpers such as `attachMilestoneScreenshot` are not in the example and are not provided by this package.
|
|
48
|
+
Reference conventions in the commands, which another project may adapt (set the locations in `## Test layout` of [`project.md`](project.md)), include `overview.md`, Gherkin under `specs/`, page contracts under `pages/`, page objects, flows, and a Playwright fixture. There is no settings key for those paths. A project does not have to copy that layout for the server to start. The [Cabana example](https://github.com/bogdanOwner/e2e-qa-pilot/blob/main/examples/boutique-cabin/README.md) uses it so the reference workflow can be read in one place. That directory is in this git repository and is not part of the published package. Figma helpers such as `attachMilestoneScreenshot` are not in the example and are not provided by this package.
|