e2e-qa-pilot 0.0.0-stage → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CODE_OF_CONDUCT.md +126 -0
- package/CONTRIBUTING.md +11 -0
- package/LICENSE +21 -0
- package/README.md +111 -2
- package/SECURITY.md +13 -0
- package/assets/logo.svg +12 -0
- package/bin/cli.mjs +32 -0
- package/docs/README.md +9 -0
- package/docs/install.md +111 -0
- package/docs/scope-and-trust.md +19 -0
- package/docs/settings.md +38 -0
- package/mcp.md +166 -0
- package/package.json +47 -4
- package/processes/autofix-nightly/README.md +18 -0
- package/processes/create-e2e/README.md +42 -0
- package/processes/create-e2e/commands/add-from-design/SKILL.md +173 -0
- package/processes/create-e2e/commands/add-from-steps/SKILL.md +178 -0
- package/processes/create-e2e/commands/review-case/SKILL.md +130 -0
- package/processes/create-e2e/commands/shared/pr-evidence.md +211 -0
- package/processes/create-e2e/commands/shared/ready-rules.md +42 -0
- package/src/init.mjs +267 -0
- package/src/kit.mjs +89 -0
- package/src/mcp.mjs +227 -0
- package/templates/settings.md +57 -0
|
@@ -0,0 +1,130 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: review-case
|
|
3
|
+
description: >-
|
|
4
|
+
Check whether a case has steps specific enough to assert. The case is an
|
|
5
|
+
existing id or pasted test case text. Ask when a step cannot be checked,
|
|
6
|
+
then update that same case when it came from an id. Use before automating
|
|
7
|
+
a case. Does not write a spec.
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
# Review case steps
|
|
11
|
+
|
|
12
|
+
Input is exactly one of `case_id` or `case_text`, plus `.e2e-qa-pilot/settings.md` in this project.
|
|
13
|
+
|
|
14
|
+
- `case_id`: the id of an existing case in the configured case tool.
|
|
15
|
+
- `case_text`: the complete test case, including preconditions, steps, and expected results.
|
|
16
|
+
|
|
17
|
+
The tool call names the mode. Do not decide which one a string is by reading its contents. Both values are ambiguous: ask the human which one to use and stop. Neither value: ask for one of them and stop.
|
|
18
|
+
|
|
19
|
+
This skill does not write a spec and does not create a second case. Implementation is a later skill, and only after a silent pass.
|
|
20
|
+
|
|
21
|
+
Ready checks, the question shape, and the mood-diary example: [ready-rules.md](../shared/ready-rules.md).
|
|
22
|
+
|
|
23
|
+
## Settings
|
|
24
|
+
|
|
25
|
+
Read `.e2e-qa-pilot/settings.md`. The keys live in the template comment.
|
|
26
|
+
|
|
27
|
+
Generation keys, both modes: `figma`, `read_figma`, `branch_pattern`, `test_command`, `png_dir`, `pr_mode`, `pr_label_ready`, `pr_label_minor_text_diff`, `pr_label_mismatch`.
|
|
28
|
+
|
|
29
|
+
Provider keys, `case_id` mode only: `case_tool`, `fetch_case`, `update_case`.
|
|
30
|
+
|
|
31
|
+
`read_figma` is required even when `figma` is `off`.
|
|
32
|
+
|
|
33
|
+
If a required key for this mode has no value, ask once in chat for every empty key in that set, except `branch_pattern`, `pr_mode`, and the label keys. Those have defaults below. Write the answers into the file, then continue:
|
|
34
|
+
|
|
35
|
+
- Context: `figma`, `read_figma`, and in `case_id` mode also `case_tool`, `fetch_case`, `update_case`
|
|
36
|
+
- Requirements: `test_command`, `png_dir`
|
|
37
|
+
- Goal: `branch_pattern`, `pr_mode`, `pr_label_ready`, `pr_label_minor_text_diff`, `pr_label_mismatch`
|
|
38
|
+
|
|
39
|
+
`branch_pattern`: a missing or empty value is `e2e/{title_slug}` in `case_text` mode and `e2e/{case_id}-{title_slug}` in `case_id` mode. Write that default. Do not ask. The placeholders are filled in by the implementation skill. A value that is already set stays as it is.
|
|
40
|
+
|
|
41
|
+
`pr_mode`: a missing or empty value means `create_pr_for_review`. Do not ask. The other legal value is `ask_before_pr`. Any other value: ask once, then write the answer.
|
|
42
|
+
|
|
43
|
+
Label keys: a missing or empty `pr_label_ready` is `e2e:ready`. A missing or empty `pr_label_minor_text_diff` is `e2e:minor-text-diff`. A missing or empty `pr_label_mismatch` is `e2e:mismatch`. Write those defaults. Do not ask. A value that is already set stays as it is.
|
|
44
|
+
|
|
45
|
+
`case_text` mode does not ask for `case_tool`, `fetch_case`, or `update_case`, and does not write placeholder values for them.
|
|
46
|
+
|
|
47
|
+
Do not ask again on a later run when the keys already have values.
|
|
48
|
+
|
|
49
|
+
## Resolve the case
|
|
50
|
+
|
|
51
|
+
`case_id` mode: fetch the case with `fetch_case` from the settings file. If the case tool errors or the case does not exist, stop and report that. Do not invent the case. Do not treat this id as test case text. Do not switch this request to `case_text`.
|
|
52
|
+
|
|
53
|
+
`case_text` mode: the pasted text is the case. Do not call `fetch_case`. Do not call `update_case`. A case tool is not required. A configured provider is ignored for this call.
|
|
54
|
+
|
|
55
|
+
The same ready checks apply to the resolved case from either source.
|
|
56
|
+
|
|
57
|
+
## Title
|
|
58
|
+
|
|
59
|
+
`case_id` mode uses the title returned by `fetch_case`.
|
|
60
|
+
|
|
61
|
+
`case_text` mode chooses a title before a spec, a filename, or a pull request is named:
|
|
62
|
+
|
|
63
|
+
- Prefer a `Title:` line.
|
|
64
|
+
- If that line is absent, use the first Markdown H1 (`# `, not `##`).
|
|
65
|
+
- If both exist, compare the trimmed texts. A difference in case or surrounding whitespace still uses the `Title:` line. Any other difference asks one question. Do not pick silently.
|
|
66
|
+
- If neither exists, ask for a short title before generating the spec.
|
|
67
|
+
- Do not take a precondition or a step as the title.
|
|
68
|
+
- Do not invent an external id or a case URL.
|
|
69
|
+
|
|
70
|
+
Display text preserves the original casing and internal wording, with surrounding whitespace removed. Filenames, the pull request title, the spec title, and the duplicate check use that trimmed title. Do not leave leading or trailing whitespace in those names.
|
|
71
|
+
|
|
72
|
+
## Ready
|
|
73
|
+
|
|
74
|
+
Follow [ready-rules.md](../shared/ready-rules.md).
|
|
75
|
+
|
|
76
|
+
A missing Figma URL is recorded as "steps are the oracle". It does not fail the case.
|
|
77
|
+
|
|
78
|
+
## Silent pass
|
|
79
|
+
|
|
80
|
+
Say the case is ready in one or two sentences, then continue in the same turn:
|
|
81
|
+
|
|
82
|
+
- `case_id` mode: a `figma.com` URL in the description or a custom field, and `figma: on` in the settings file → [add-from-design](../add-from-design/SKILL.md).
|
|
83
|
+
- `case_text` mode: a `figma.com` URL anywhere in the pasted text, and `figma: on` → [add-from-design](../add-from-design/SKILL.md).
|
|
84
|
+
- No such URL, or `figma: off` → [add-from-steps](../add-from-steps/SKILL.md).
|
|
85
|
+
|
|
86
|
+
Pass the same input mode to that skill. `case_text` mode passes the pasted text, including any clarified steps and the chosen title. That skill applies the duplicate search and the `branch_pattern` rule.
|
|
87
|
+
|
|
88
|
+
## Not ready
|
|
89
|
+
|
|
90
|
+
Stop. Ask in chat. One question per hole. Quote the step. Do not invent the expected result. Do not implement in that turn.
|
|
91
|
+
|
|
92
|
+
Holes: no expected result, a vague word, two names for one control, contradictory steps, or preconditions that do not say who is signed in and where they start.
|
|
93
|
+
|
|
94
|
+
After the user answers, show the step diff.
|
|
95
|
+
|
|
96
|
+
`case_id` mode: update that same case with `update_case` from the settings file. Re-read the case with `fetch_case`. Confirm the written preconditions and steps match the diff. If an answer is still missing, do not write.
|
|
97
|
+
|
|
98
|
+
`case_text` mode: do not call `update_case` and do not call `fetch_case`. Do not create a case. The clarified text is the case for the next turn.
|
|
99
|
+
|
|
100
|
+
The next turn, after the re-read matches or the clarified text is the case, starts the implementation skill from the silent-pass routes above.
|
|
101
|
+
|
|
102
|
+
## Update
|
|
103
|
+
|
|
104
|
+
This section applies only in `case_id` mode.
|
|
105
|
+
|
|
106
|
+
`update_case` updates when the id is set. Omitting the id creates a new case. Always pass the id.
|
|
107
|
+
|
|
108
|
+
- id: the existing case id
|
|
109
|
+
- title: the current title, unless the user's answer changes it
|
|
110
|
+
- automation, status, suite id, tags: the values just read. Leave automation status alone
|
|
111
|
+
- preconditions: the clarified text
|
|
112
|
+
- steps type: `classic`
|
|
113
|
+
- steps: action, expected result, and data when the step has input data
|
|
114
|
+
- description: keep the existing text. Append:
|
|
115
|
+
|
|
116
|
+
```text
|
|
117
|
+
Steps clarified YYYY-MM-DD. Assertions follow this case.
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
Do not delete the case.
|
|
121
|
+
|
|
122
|
+
Then `fetch_case` the same id. The stored preconditions and steps match the diff you showed. A mismatch stops. Do not write again with a guessed fix.
|
|
123
|
+
|
|
124
|
+
## Tool contract
|
|
125
|
+
|
|
126
|
+
`review_case`
|
|
127
|
+
|
|
128
|
+
- Inputs: exactly one of `case_id` or `case_text`, and the settings file `.e2e-qa-pilot/settings.md`
|
|
129
|
+
- Result: the case is ready and routed, or one question per hole was asked and that same case was updated when it came from an id
|
|
130
|
+
- The agent still asks, and does not invent an expected result
|
|
@@ -0,0 +1,211 @@
|
|
|
1
|
+
# Pull request evidence
|
|
2
|
+
|
|
3
|
+
PR template and evidence names for [add-from-steps](../add-from-steps/SKILL.md).
|
|
4
|
+
|
|
5
|
+
This command checks the expected-result text. It does not use the Figma visual review.
|
|
6
|
+
|
|
7
|
+
A failed functional assertion is a mismatch. It is not, by itself, a confirmed product regression. A narrow text difference is not a failed assertion.
|
|
8
|
+
|
|
9
|
+
## Word rule
|
|
10
|
+
|
|
11
|
+
The spec asserts functional outcomes: the control is present, the modal closes, the preview updates, the chart dot appears, and the other visible behavior in the expected result. Copy that behavior from the case into the assertion, including awkward grammar.
|
|
12
|
+
|
|
13
|
+
Do not put the quoted case sentence in an assertion when, after lowercasing, stripping punctuation, and dropping the filler words `a`, `the`, `has`, and `been`, the remaining words match in the same order. Record the case sentence and the screen sentence in the steps table and in the pull request comments. The verdict for that quoted sentence is not `pass`. Example: case **"Your diary entry has successfully created"** and screen **Your Diary entry successfully created!**
|
|
14
|
+
|
|
15
|
+
That difference is not an exact-text pass. Functional checks in the same step stay `pass` when they pass. State that the original exact sentence was not a passing assertion.
|
|
16
|
+
|
|
17
|
+
When the remaining words do not match in the same order, the text difference is unresolved. Assert the case sentence and leave that assertion failing. The spec stays red. Do not treat it as a minor text difference.
|
|
18
|
+
|
|
19
|
+
Do not rewrite the expected sentence. Do not change an assertion so the exact case sentence appears to pass. `assertion_edits` stays 0. Comparing with this rule is not an assertion edit. A green run bought by editing the expected sentence is a failed skill run. Revert it and open the mismatch pull request.
|
|
20
|
+
|
|
21
|
+
When this minor difference is the only issue, and required screenshot evidence meets the acceptance criteria, the pull request label is `pr_label_minor_text_diff` from the settings file. The default is `e2e:minor-text-diff`. It is not `e2e:ready`.
|
|
22
|
+
|
|
23
|
+
A different message (another verb, a missing screen, a missing control) is a functional mismatch. Leave the functional assertion. The spec stays red. A failed functional assertion is a mismatch. It is not, by itself, a confirmed product regression.
|
|
24
|
+
|
|
25
|
+
## Test data
|
|
26
|
+
|
|
27
|
+
A step that types, selects, or toggles uses the values written in the case. When the case gives no values, that is not a hole and the run does not ask. Choose obviously fake values (for example a made-up name and an address on `example.com`). Never use real personal data. Take the values from one named constant in the flow, so the spec, the screenshots, and the pull request use the same ones.
|
|
28
|
+
|
|
29
|
+
Declare the values wherever the step is reported: the `## Test data` table in the pull request, the `## Test data` block in the page contract, and the Actual cell of that step. A reader must be able to tell what was typed without opening the code.
|
|
30
|
+
|
|
31
|
+
Before the final click of such a step, the spec asserts each entered value and each toggled control (`toHaveValue`, `toBeChecked`). This checks that the action happened. It is not an expected result, and it is not an assertion edit.
|
|
32
|
+
|
|
33
|
+
## Screenshot gate
|
|
34
|
+
|
|
35
|
+
Look at the PNG before it is copied into `png_dir` from the settings file or linked in the pull request.
|
|
36
|
+
|
|
37
|
+
- Keep it when the image shows the control or change named in that row (open modal, open picker, filled chip, new tab, toast).
|
|
38
|
+
- A picture that matches an earlier step does not prove this step.
|
|
39
|
+
- One retry only: capture the proving control (`locator.screenshot` or a clip of that control), not the same full viewport.
|
|
40
|
+
- A step that enters or selects data proves itself with the data on screen. Take the filled state before the final click: the entered values and the checked controls (for example a terms checkbox) are visible. Clip the form or the section that holds them, not only the result. When the same step also submits, take a second image of the result while the proof is still on screen. The links are `step-03a` (filled) and `step-03b` (result). An empty form does not prove the entry. A result card alone does not prove it either.
|
|
41
|
+
- If the second image still does not show the step, omit the file. The table row stays. The Screenshot cell is `—`. No image heading for that step.
|
|
42
|
+
- An omitted earlier image is not, by itself, a failed assertion.
|
|
43
|
+
- The required end-state screenshot must show the expected end state. A narrow wording difference on that image does not, by itself, mean the screenshot missed the expected end state. If the image does not show the expected end state, do not publish it as passing evidence. That is a screenshot finding. It does not, by itself, prove that a functional assertion failed. A functional assertion failure does not, by itself, mark the screenshot inadequate. Report the two findings separately in the steps table and in the pull request comments.
|
|
44
|
+
- Either finding blocks `e2e:ready`. A required end-state screenshot that misses the expected end state selects `e2e:mismatch`, unless the only issue in the run is the narrow text difference above.
|
|
45
|
+
- Do not change the expected result to match a useless screenshot.
|
|
46
|
+
|
|
47
|
+
`assertion_edits` stays 0. Rewriting the expected sentence so the spec goes green is an assertion edit. Comparing with the word rule above is not. A green run bought by editing the expected sentence is a failed skill run. Revert it and open the mismatch pull request.
|
|
48
|
+
|
|
49
|
+
Name the file for the screen state (`diary-date-picker.png`), not for the case id. Kebab rules: lowercase, spaces to `-`, drop quotes and trailing periods. One file per distinct screen that shows the step (open modal, open picker, filled chip, filled form, new tab, toast). A step may have more than one file when it has more than one distinct screen. The page contract `## States` block and the overview coverage table cite the same path. A step whose capture repeated an earlier screen has no file. The end-state file is kept only when it shows the expected end state.
|
|
50
|
+
|
|
51
|
+
Mismatch pull request: one file, the failed step, named for that state, when that image shows the mismatch. Steps that never ran have no file.
|
|
52
|
+
|
|
53
|
+
Do not commit `test-results/` or the HTML report folder.
|
|
54
|
+
|
|
55
|
+
In the pull request body, do not use a relative path. The host turns that into a compare URL and the image does not render. Use the head branch. Link text stays `step-01`. `<blob-url>` and `<raw-url>` are the head-branch blob and raw URLs for `png_dir` from the settings file.
|
|
56
|
+
|
|
57
|
+
## Outcome
|
|
58
|
+
|
|
59
|
+
One pull request to `main`. Never auto-merge. Read `pr_mode`, `pr_label_ready`, `pr_label_minor_text_diff`, and `pr_label_mismatch` from the settings file. A missing or empty `pr_mode` means `create_pr_for_review`. Do not ask. The other legal value is `ask_before_pr`. Any other value: ask once. A missing or empty label uses its default: `e2e:ready`, `e2e:minor-text-diff`, or `e2e:mismatch`. Create each label you apply if it is missing, then apply exactly one.
|
|
60
|
+
|
|
61
|
+
Precedence:
|
|
62
|
+
|
|
63
|
+
1. A functional assertion failure, an unresolved difference, or required screenshot evidence that does not show the expected end state: `pr_label_mismatch`. Draft. If a functional assertion failed, the spec stays red. A screenshot finding does not, by itself, fail the functional assertion. A narrow wording difference is not, by itself, a screenshot miss. If a narrow minor text difference is also present, document it and still use this label.
|
|
64
|
+
2. Functional behavior matches, required screenshot evidence meets the acceptance criteria, and the only difference is the narrow text rule: `pr_label_minor_text_diff`. Open for review. Not a draft. Not `e2e:ready`. The exact sentence is not a passed assertion.
|
|
65
|
+
3. Functional assertions passed, the expected behavior matches, and required screenshot evidence meets the acceptance criteria: `pr_label_ready`. Open for review. Not a draft. The label does not mean approved or merged.
|
|
66
|
+
|
|
67
|
+
`create_pr_for_review` creates that pull request for human QA review. It does not stop first for a narrow minor text difference.
|
|
68
|
+
|
|
69
|
+
`ask_before_pr` stops before creating a pull request when a functional mismatch, an unresolved difference, or a failed final-screenshot gate remains. Stop and ask in chat. A narrow minor text difference does not trigger this stop, so the `e2e:minor-text-diff` pull request is still created. After the user responds, continue from the clarified requirements. If the user changes the expected behavior, later assertions follow that clarification and the pull request quotes the change. If the discrepancy remains, create the pull request only after the user explicitly authorizes it. When the expected behavior has not changed, preserve the failing assertion. If the answer is still unclear, ask once more and do not open a pull request.
|
|
70
|
+
|
|
71
|
+
If GitHub access is missing, report that the pull request step cannot be completed. Leave the local spec, table, and screenshots. Do not claim a pull request exists.
|
|
72
|
+
|
|
73
|
+
| Outcome | Label | PR |
|
|
74
|
+
|---------|--------|----|
|
|
75
|
+
| Functional assertions passed, the expected behavior matches, and required screenshot evidence meets the acceptance criteria. `assertion_edits` is 0 | `pr_label_ready` (default `e2e:ready`) | Open for review. Not a draft |
|
|
76
|
+
| Functional behavior matches, required screenshots meet the acceptance criteria, and the only difference is the narrow text rule | `pr_label_minor_text_diff` (default `e2e:minor-text-diff`) | Open for review. Not a draft. Not `e2e:ready` |
|
|
77
|
+
| A functional or unresolved mismatch, or required screenshot evidence does not show the expected end state | `pr_label_mismatch` (default `e2e:mismatch`) | Draft. A failed assertion stays red |
|
|
78
|
+
|
|
79
|
+
Do not apply `pr_label_ready` when `assertion_edits` is anything but 0. Do not apply `pr_label_ready` when the only difference is a minor text difference.
|
|
80
|
+
|
|
81
|
+
Screenshot links in the body are head-branch blob and raw URLs. A relative path does not open the file. Link text stays `step-01`. A step with two images uses `step-03a` and `step-03b`. A step whose image failed the Screenshot gate has `—` in the Screenshot cell and no image block. Assertion findings and screenshot findings are separate comments. When the only issue is the narrow text rule, the body records the case sentence and the screen sentence and states that the exact sentence was not a passing assertion. The agent does not edit the case. That difference is not an assertion edit. Steps that never ran have no PNG. Do not use `test.fixme`. A red pull-request check is the expected CI result on a mismatch.
|
|
82
|
+
|
|
83
|
+
Do not edit the nightly workflow.
|
|
84
|
+
|
|
85
|
+
Title:
|
|
86
|
+
|
|
87
|
+
- `case_id`: `<case-id> <case title>`
|
|
88
|
+
- `case_text`: the trimmed title. Trim surrounding whitespace. Keep the original casing and internal wording. Do not invent an id. The spec `test()` title and filenames use that trimmed string. Do not leave leading or trailing whitespace in those names. The duplicate check starts from that trimmed title. Display text in the case heading preserves the original casing and internal wording.
|
|
89
|
+
|
|
90
|
+
Base: `main`.
|
|
91
|
+
|
|
92
|
+
## Body layout
|
|
93
|
+
|
|
94
|
+
Sections, in this order. Do not add a section that repeats another one.
|
|
95
|
+
|
|
96
|
+
1. `## Summary`: one result line, then at most three bullets naming what was added (the spec, the new page objects and flows, the docs). The result line has the icon, the step count, the run count, and `assertion_edits`. Example: `✅ ready · 3 of 3 steps passed · 1 run (retries off) · assertion_edits 0`. Use `⚠️ minor text difference` or `❌ mismatch` in place of `✅ ready`. The label is named in this line and is not repeated elsewhere.
|
|
97
|
+
2. `## Case`: the case heading below, then the preconditions.
|
|
98
|
+
3. `## Test data`: a table of the values typed or selected (field, value, source). The source is `case` when the case gave the value and `chosen by the run` when it did not. Omit the section when no step enters data.
|
|
99
|
+
4. `## Steps`: the table.
|
|
100
|
+
5. `## Screenshots`: one heading per step that has a PNG. Under each image, one caption line that says what the reviewer should see (for example `Filled form: names, email, and the checked terms box are visible.`).
|
|
101
|
+
6. Findings, only when something is not `pass`: `## Text difference`, `## Assertion findings`, `## Screenshot findings`. Each is its own section.
|
|
102
|
+
7. `## How to run`: the `test_command` and the folder to run it from, with the spec path.
|
|
103
|
+
8. `## Review checklist`: only what a person must check and the run cannot (the screenshots show what the table says, the test data is acceptable, a finding is understood). Never copy an expected result into the checklist. Omit Attempts and Not verified.
|
|
104
|
+
|
|
105
|
+
## Steps table
|
|
106
|
+
|
|
107
|
+
Columns: Step, Expected result, Actual result, Screenshot, Verdict.
|
|
108
|
+
|
|
109
|
+
- The Expected cell is the case sentence. It is the only place the full sentence appears.
|
|
110
|
+
- When the actual outcome equals the expected result, the Actual cell is `✅ As expected`. Do not repeat the expected sentence. When the step entered data, add it: `✅ As expected. Entered: First Name Ana, Last Name Pop.`
|
|
111
|
+
- When the actual outcome differs, the Actual cell starts with an icon and holds only what the screen showed: `⚠️` for the narrow text rule, `❌` for a mismatch. When they match only under the narrow text rule, write the on-screen sentence and do not mark that quoted sentence `pass`.
|
|
112
|
+
- The Verdict cell carries an icon: `✅ pass`, `⚠️ minor-text-diff`, `❌ mismatch`.
|
|
113
|
+
- A step with two images lists both: `[step-03a](<blob-url>/x.png) [step-03b](<blob-url>/y.png)`.
|
|
114
|
+
|
|
115
|
+
Under the table, one heading per step that has a PNG, with one raw image for each of its files. A step whose Screenshot cell is `—` has no heading. The end-state image is included only when it shows the expected end state.
|
|
116
|
+
|
|
117
|
+
When the only issue is the narrow text rule, add the text difference block. Omit it when every quoted string already matches the screen. The agent does not edit the case. In `case_text` mode, omit the case link. Record the case sentence and the screen sentence. Do not create a case.
|
|
118
|
+
|
|
119
|
+
## Case heading
|
|
120
|
+
|
|
121
|
+
- `case_id`: `- Automation case: [<case-id>](<case-url>)`
|
|
122
|
+
- `case_text`: `- Case: <display title>`. No URL. Do not invent an id. The heading shows the original casing and internal wording, with surrounding whitespace removed.
|
|
123
|
+
|
|
124
|
+
The worked example below is `case_id` mode.
|
|
125
|
+
|
|
126
|
+
On a mismatch, the steps table has rows only through the failed step. Verdict for that row is `❌ mismatch`. Actual result is `❌` and what the screen showed. The image uses the same blob and raw URLs. The spec stays red. The pull request is a draft.
|
|
127
|
+
|
|
128
|
+
## Worked example
|
|
129
|
+
|
|
130
|
+
```markdown
|
|
131
|
+
## Summary
|
|
132
|
+
|
|
133
|
+
⚠️ minor text difference · 7 of 8 steps passed, 1 minor-text-diff · 1 run (retries off) · assertion_edits 0
|
|
134
|
+
|
|
135
|
+
- Add a live spec for the Mood Diary entry case.
|
|
136
|
+
- Add the Create Diary Entry page object and flow.
|
|
137
|
+
|
|
138
|
+
## Case
|
|
139
|
+
|
|
140
|
+
- Automation case: [<case-id>](<case-url>)
|
|
141
|
+
- Preconditions: the user is signed in and on the Personal Space page.
|
|
142
|
+
|
|
143
|
+
## Test data
|
|
144
|
+
|
|
145
|
+
| Field | Value | Source |
|
|
146
|
+
|-------|-------|--------|
|
|
147
|
+
| Time | 14:30 | chosen by the run |
|
|
148
|
+
| Thoughts text | Test thought | chosen by the run |
|
|
149
|
+
|
|
150
|
+
## Steps
|
|
151
|
+
|
|
152
|
+
| Step | Expected result | Actual result | Screenshot | Verdict |
|
|
153
|
+
|------|-----------------|---------------|------------|---------|
|
|
154
|
+
| 1. Open the Personal Space page | The Mood Diary component is displayed. | ✅ As expected | [step-01](<blob-url>/mood-diary.png) | ✅ pass |
|
|
155
|
+
| 2. Click the Awesome mood emoji | The Create Diary Entry modal opens with Awesome preselected. | ✅ As expected | — | ✅ pass |
|
|
156
|
+
| 3. Click the date picker | The date picker opens. | ✅ As expected | [step-03](<blob-url>/diary-date-picker.png) | ✅ pass |
|
|
157
|
+
| 4. Select any date in the past and click Next | The date picker closes, and the selected date is applied. | ✅ As expected | — | ✅ pass |
|
|
158
|
+
| 5. Open the time picker, select a time, and click Next | The time picker closes, and the selected time is applied. | ✅ As expected. Entered: time 14:30. | [step-05](<blob-url>/diary-time-picker.png) | ✅ pass |
|
|
159
|
+
| 6. Select several emotions and click Next | The Reasons tab opens. | ✅ As expected | — | ✅ pass |
|
|
160
|
+
| 7. Select one option in each category and click Next | The Thoughts tab opens. | ✅ As expected | — | ✅ pass |
|
|
161
|
+
| 8. Enter the thoughts text and click Save | The success toast message appears: "Your diary entry has successfully created". The Create Diary Entry modal closes. The newly created diary entry appears in the Mood Diary component preview. A dot representing the new entry appears on the Mood Diary chart. | ⚠️ Functional: the modal closed, the preview showed Awesome, and a chart dot was visible. Entered: thoughts "Test thought". Quoted toast on screen: Your Diary entry successfully created! | [step-08a](<blob-url>/diary-thoughts-filled.png) [step-08b](<blob-url>/diary-entry-saved.png) | ⚠️ minor-text-diff |
|
|
162
|
+
|
|
163
|
+
## Screenshots
|
|
164
|
+
|
|
165
|
+
### 1. Open the Personal Space page
|
|
166
|
+
|
|
167
|
+

|
|
168
|
+
|
|
169
|
+
Mood Diary component on the Personal Space page.
|
|
170
|
+
|
|
171
|
+
### 3. Click the date picker
|
|
172
|
+
|
|
173
|
+

|
|
174
|
+
|
|
175
|
+
Open date picker.
|
|
176
|
+
|
|
177
|
+
### 5. Open the time picker, select a time, and click Next
|
|
178
|
+
|
|
179
|
+

|
|
180
|
+
|
|
181
|
+
Time picker with 14:30 selected, before Next.
|
|
182
|
+
|
|
183
|
+
### 8. Enter the thoughts text and click Save
|
|
184
|
+
|
|
185
|
+

|
|
186
|
+
|
|
187
|
+
Filled Thoughts tab: the text "Test thought" is in the field, before Save.
|
|
188
|
+
|
|
189
|
+

|
|
190
|
+
|
|
191
|
+
Result: the toast, the new entry in the preview, and the chart dot.
|
|
192
|
+
|
|
193
|
+
## Text difference
|
|
194
|
+
|
|
195
|
+
Step 8 quoted toast. The functional checks passed. The exact sentence did not.
|
|
196
|
+
|
|
197
|
+
- Case: "Your diary entry has successfully created"
|
|
198
|
+
- Screen: Your Diary entry successfully created!
|
|
199
|
+
|
|
200
|
+
This is not a passed exact-text assertion. The expected sentence was not rewritten. Label: `e2e:minor-text-diff`.
|
|
201
|
+
|
|
202
|
+
## How to run
|
|
203
|
+
|
|
204
|
+
From `<project folder>`: `<test_command> -- tests/<spec-file> --retries=0`
|
|
205
|
+
|
|
206
|
+
## Review checklist
|
|
207
|
+
|
|
208
|
+
- [ ] The screenshots show what the Steps table says, including the filled Thoughts tab.
|
|
209
|
+
- [ ] The test data above is acceptable.
|
|
210
|
+
- [ ] The text difference in step 8 is understood.
|
|
211
|
+
```
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
# Ready rules
|
|
2
|
+
|
|
3
|
+
Templates for [review-case](../review-case/SKILL.md).
|
|
4
|
+
|
|
5
|
+
These checks apply to the resolved case, whether it was fetched by id or supplied as text.
|
|
6
|
+
|
|
7
|
+
All of these hold:
|
|
8
|
+
|
|
9
|
+
- Preconditions say who is signed in and where they start.
|
|
10
|
+
- Every step has an action and an expected result.
|
|
11
|
+
- The expected result names a screen, quotes a control, or states a visible change.
|
|
12
|
+
- No `correct`, `properly`, `etc.`, and no second name for the same control.
|
|
13
|
+
- A step that enters data may leave the values out. That is not a hole. The run chooses fake values and declares them ([test data](pr-evidence.md#test-data)). A case that names its values is preferred.
|
|
14
|
+
- Steps do not contradict each other.
|
|
15
|
+
|
|
16
|
+
A missing Figma URL is recorded as "steps are the oracle". It does not fail the case. A case written like the mood diary list below passes with no question.
|
|
17
|
+
|
|
18
|
+
## Ready example
|
|
19
|
+
|
|
20
|
+
A mood-diary list passes with no question. Each expected result names something visible:
|
|
21
|
+
|
|
22
|
+
1. Action: Open the Personal Space page. Expected: The Mood Diary component is displayed.
|
|
23
|
+
2. Action: Click the Awesome mood emoji. Expected: The Create Diary Entry modal opens with Awesome preselected.
|
|
24
|
+
3. Action: Click Save, after the earlier fields. Expected: toast "Your diary entry has successfully created", the modal closes, the entry appears in the preview, a dot appears on the chart.
|
|
25
|
+
|
|
26
|
+
Awkward product copy stays quoted. Do not correct it.
|
|
27
|
+
|
|
28
|
+
## Not ready example
|
|
29
|
+
|
|
30
|
+
Steps 3–6 say "Save button". Step 7 says "Start achieving" for the same submit control. That is two names. Ask which control the later step means. Do not pick one.
|
|
31
|
+
|
|
32
|
+
## Question
|
|
33
|
+
|
|
34
|
+
One question per hole. Quote the step. Do not invent the expected result.
|
|
35
|
+
|
|
36
|
+
```text
|
|
37
|
+
Step 7 expected result: "Start achieving button remains enabled."
|
|
38
|
+
Steps 3–6 call the same control "Save button".
|
|
39
|
+
Which label is on the control after the time is selected?
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
Holes: no expected result, a vague word, two names for one control, contradictory steps, or preconditions that do not say who is signed in and where they start.
|
package/src/init.mjs
ADDED
|
@@ -0,0 +1,267 @@
|
|
|
1
|
+
import path from 'node:path';
|
|
2
|
+
import fs from 'node:fs';
|
|
3
|
+
import {
|
|
4
|
+
kitRoot,
|
|
5
|
+
MARKER,
|
|
6
|
+
MARKER_LINE,
|
|
7
|
+
CATALOG,
|
|
8
|
+
EDITORS,
|
|
9
|
+
SERVER_NAME,
|
|
10
|
+
listFiles,
|
|
11
|
+
} from './kit.mjs';
|
|
12
|
+
|
|
13
|
+
// Installs the Create e2e commands into a project.
|
|
14
|
+
//
|
|
15
|
+
// Overwrite rules:
|
|
16
|
+
// - missing file: write
|
|
17
|
+
// - present, without --force: leave
|
|
18
|
+
// - --force and an exact marker line: refresh
|
|
19
|
+
// - --force without that line: leave (the file belongs to the user)
|
|
20
|
+
// - .e2e-qa-pilot/settings.md: always leave once it exists
|
|
21
|
+
// - an MCP entry that is already listed: always leave
|
|
22
|
+
|
|
23
|
+
const INTEGRATIONS = [...Object.keys(EDITORS), 'all'];
|
|
24
|
+
|
|
25
|
+
const FLAGS = {
|
|
26
|
+
'--force': {
|
|
27
|
+
value: false,
|
|
28
|
+
apply(parsed) {
|
|
29
|
+
parsed.force = true;
|
|
30
|
+
},
|
|
31
|
+
},
|
|
32
|
+
'--integration': {
|
|
33
|
+
value: true,
|
|
34
|
+
apply(parsed, value) {
|
|
35
|
+
parsed.integration = value ?? '';
|
|
36
|
+
},
|
|
37
|
+
},
|
|
38
|
+
};
|
|
39
|
+
|
|
40
|
+
export const INIT_USAGE = `Usage: e2e-qa-pilot init [path] [--integration cursor|claude|all] [--force]
|
|
41
|
+
|
|
42
|
+
path Project folder. Default: the current folder.
|
|
43
|
+
--integration Which editor to set up. Default: the editor whose folder
|
|
44
|
+
already exists (.cursor/ or .claude/). Both, when both
|
|
45
|
+
folders exist or neither does.
|
|
46
|
+
--force Refresh files that still have the exact ${MARKER_LINE} line.
|
|
47
|
+
Settings and an existing MCP entry stay as they are.`;
|
|
48
|
+
|
|
49
|
+
function parseArgs(argv) {
|
|
50
|
+
const parsed = { projectArg: '', force: false };
|
|
51
|
+
for (let i = 0; i < argv.length; i += 1) {
|
|
52
|
+
const arg = argv[i];
|
|
53
|
+
const equalsAt = arg.indexOf('=');
|
|
54
|
+
const name = equalsAt === -1 ? arg : arg.slice(0, equalsAt);
|
|
55
|
+
const flag = FLAGS[name];
|
|
56
|
+
if (flag) {
|
|
57
|
+
if (!flag.value) {
|
|
58
|
+
if (equalsAt !== -1) throw new Error(`Unknown argument: ${arg}`);
|
|
59
|
+
flag.apply(parsed);
|
|
60
|
+
continue;
|
|
61
|
+
}
|
|
62
|
+
const value = equalsAt === -1 ? argv[i + 1] : arg.slice(equalsAt + 1);
|
|
63
|
+
if (equalsAt === -1) i += 1;
|
|
64
|
+
flag.apply(parsed, value);
|
|
65
|
+
continue;
|
|
66
|
+
}
|
|
67
|
+
if (arg.startsWith('-')) throw new Error(`Unknown argument: ${arg}`);
|
|
68
|
+
if (parsed.projectArg) throw new Error(`Unexpected argument: ${arg}`);
|
|
69
|
+
parsed.projectArg = arg;
|
|
70
|
+
}
|
|
71
|
+
if (parsed.integration !== undefined && !INTEGRATIONS.includes(parsed.integration)) {
|
|
72
|
+
throw new Error(`--integration must be one of: ${INTEGRATIONS.join(', ')}`);
|
|
73
|
+
}
|
|
74
|
+
return {
|
|
75
|
+
projectArg: parsed.projectArg,
|
|
76
|
+
integration: parsed.integration ?? '',
|
|
77
|
+
force: parsed.force,
|
|
78
|
+
};
|
|
79
|
+
}
|
|
80
|
+
|
|
81
|
+
function hasMarker(text) {
|
|
82
|
+
return text.split(/\r?\n/).some((line) => line.trim() === MARKER_LINE);
|
|
83
|
+
}
|
|
84
|
+
|
|
85
|
+
function withMarker(text) {
|
|
86
|
+
const body = text.endsWith('\n') ? text : `${text}\n`;
|
|
87
|
+
if (hasMarker(body)) return body;
|
|
88
|
+
return `${body}${MARKER_LINE}\n`;
|
|
89
|
+
}
|
|
90
|
+
|
|
91
|
+
function stubText(command, tool) {
|
|
92
|
+
return `---
|
|
93
|
+
name: ${command}
|
|
94
|
+
description: >-
|
|
95
|
+
Follow the installed ${command} command for e2e-qa-pilot Create e2e.
|
|
96
|
+
---
|
|
97
|
+
|
|
98
|
+
# ${command}
|
|
99
|
+
|
|
100
|
+
Read and follow \`.e2e-qa-pilot/processes/create-e2e/commands/${command}/SKILL.md\` in this project.
|
|
101
|
+
|
|
102
|
+
If the \`${SERVER_NAME}\` MCP server is connected, its \`${tool}\` tool returns the instructions bundled with the installed package. They match this project's copy only when both come from the same package version.
|
|
103
|
+
|
|
104
|
+
${MARKER_LINE}
|
|
105
|
+
`;
|
|
106
|
+
}
|
|
107
|
+
|
|
108
|
+
function writeFile(dest, text) {
|
|
109
|
+
fs.mkdirSync(path.dirname(dest), { recursive: true });
|
|
110
|
+
fs.writeFileSync(dest, text);
|
|
111
|
+
}
|
|
112
|
+
|
|
113
|
+
function managedAction(exists, marked, force) {
|
|
114
|
+
if (!exists) return 'write';
|
|
115
|
+
if (force && marked) return 'refresh';
|
|
116
|
+
if (force) return 'skip-unmarked';
|
|
117
|
+
return 'skip-present';
|
|
118
|
+
}
|
|
119
|
+
|
|
120
|
+
const MANAGED_LOG = {
|
|
121
|
+
write: (shown) => `wrote ${shown}`,
|
|
122
|
+
refresh: (shown) => `refreshed ${shown}`,
|
|
123
|
+
'skip-unmarked': (shown) => `left ${shown} (no ${MARKER} marker)`,
|
|
124
|
+
'skip-present': (shown) => `left ${shown} (already present)`,
|
|
125
|
+
};
|
|
126
|
+
|
|
127
|
+
function installManaged(dest, text, force, log, root) {
|
|
128
|
+
const shown = path.relative(root, dest);
|
|
129
|
+
const exists = fs.existsSync(dest);
|
|
130
|
+
const marked = exists && hasMarker(fs.readFileSync(dest, 'utf8'));
|
|
131
|
+
const action = managedAction(exists, marked, force);
|
|
132
|
+
if (action === 'write' || action === 'refresh') writeFile(dest, text);
|
|
133
|
+
log(MANAGED_LOG[action](shown));
|
|
134
|
+
}
|
|
135
|
+
|
|
136
|
+
function detectIntegration(project) {
|
|
137
|
+
const present = Object.keys(EDITORS).filter((name) =>
|
|
138
|
+
fs.existsSync(path.join(project, EDITORS[name].skillsDir)),
|
|
139
|
+
);
|
|
140
|
+
if (present.length === 1) return present[0];
|
|
141
|
+
return 'all';
|
|
142
|
+
}
|
|
143
|
+
|
|
144
|
+
export const serverEntry = { command: 'npx', args: ['-y', 'e2e-qa-pilot', 'mcp'] };
|
|
145
|
+
|
|
146
|
+
function mcpSnippet() {
|
|
147
|
+
return JSON.stringify({ mcpServers: { [SERVER_NAME]: serverEntry } }, null, 2);
|
|
148
|
+
}
|
|
149
|
+
|
|
150
|
+
function isRecord(value) {
|
|
151
|
+
return value !== null && typeof value === 'object' && !Array.isArray(value);
|
|
152
|
+
}
|
|
153
|
+
|
|
154
|
+
function mcpAction(exists, config, parseError) {
|
|
155
|
+
if (!exists) return { action: 'write' };
|
|
156
|
+
if (parseError) return { action: 'skip', reason: 'not valid JSON' };
|
|
157
|
+
if (!isRecord(config)) return { action: 'skip', reason: 'unexpected shape' };
|
|
158
|
+
if (config.mcpServers !== undefined && !isRecord(config.mcpServers)) {
|
|
159
|
+
return { action: 'skip', reason: 'unexpected shape' };
|
|
160
|
+
}
|
|
161
|
+
if (isRecord(config.mcpServers) && config.mcpServers[SERVER_NAME]) {
|
|
162
|
+
return { action: 'skip', reason: `${SERVER_NAME} already listed` };
|
|
163
|
+
}
|
|
164
|
+
return { action: 'add' };
|
|
165
|
+
}
|
|
166
|
+
|
|
167
|
+
function installMcpEntry(dest, log, root) {
|
|
168
|
+
const shown = path.relative(root, dest);
|
|
169
|
+
const exists = fs.existsSync(dest);
|
|
170
|
+
let config;
|
|
171
|
+
let parseError = false;
|
|
172
|
+
if (exists) {
|
|
173
|
+
try {
|
|
174
|
+
config = JSON.parse(fs.readFileSync(dest, 'utf8'));
|
|
175
|
+
} catch (error) {
|
|
176
|
+
if (error instanceof SyntaxError) parseError = true;
|
|
177
|
+
else throw error;
|
|
178
|
+
}
|
|
179
|
+
}
|
|
180
|
+
const outcome = mcpAction(exists, config, parseError);
|
|
181
|
+
if (outcome.action === 'write') {
|
|
182
|
+
writeFile(dest, `${mcpSnippet()}\n`);
|
|
183
|
+
log(`wrote ${shown}`);
|
|
184
|
+
return;
|
|
185
|
+
}
|
|
186
|
+
if (outcome.action === 'add') {
|
|
187
|
+
config.mcpServers = { ...(config.mcpServers ?? {}), [SERVER_NAME]: serverEntry };
|
|
188
|
+
writeFile(dest, `${JSON.stringify(config, null, 2)}\n`);
|
|
189
|
+
log(`added ${SERVER_NAME} to ${shown}`);
|
|
190
|
+
return;
|
|
191
|
+
}
|
|
192
|
+
const hand =
|
|
193
|
+
outcome.reason === 'not valid JSON' || outcome.reason === 'unexpected shape'
|
|
194
|
+
? `. Add this by hand:\n${mcpSnippet()}`
|
|
195
|
+
: '';
|
|
196
|
+
log(`left ${shown} (${outcome.reason})${hand}`);
|
|
197
|
+
}
|
|
198
|
+
|
|
199
|
+
function editorNames(integration) {
|
|
200
|
+
return integration === 'all' ? Object.keys(EDITORS) : [integration];
|
|
201
|
+
}
|
|
202
|
+
|
|
203
|
+
export function runInit(argv, { cwd = process.cwd(), log = console.log } = {}) {
|
|
204
|
+
const { projectArg, integration: requested, force } = parseArgs(argv);
|
|
205
|
+
const project = path.resolve(cwd, projectArg || '.');
|
|
206
|
+
|
|
207
|
+
if (!fs.existsSync(project) || !fs.statSync(project).isDirectory()) {
|
|
208
|
+
throw new Error(`Not a directory: ${project}`);
|
|
209
|
+
}
|
|
210
|
+
|
|
211
|
+
const editors = editorNames(requested || detectIntegration(project));
|
|
212
|
+
|
|
213
|
+
const sourceTree = path.join(kitRoot, 'processes', 'create-e2e');
|
|
214
|
+
const destTree = path.join(project, '.e2e-qa-pilot', 'processes', 'create-e2e');
|
|
215
|
+
if (!fs.existsSync(sourceTree)) {
|
|
216
|
+
throw new Error(`Missing Create e2e commands: ${sourceTree}`);
|
|
217
|
+
}
|
|
218
|
+
|
|
219
|
+
log(`e2e-qa-pilot init in ${project} (${editors.join(' + ')})`);
|
|
220
|
+
|
|
221
|
+
for (const source of listFiles(sourceTree)) {
|
|
222
|
+
const rel = path.relative(sourceTree, source);
|
|
223
|
+
installManaged(
|
|
224
|
+
path.join(destTree, rel),
|
|
225
|
+
withMarker(fs.readFileSync(source, 'utf8')),
|
|
226
|
+
force,
|
|
227
|
+
log,
|
|
228
|
+
project,
|
|
229
|
+
);
|
|
230
|
+
}
|
|
231
|
+
|
|
232
|
+
const settingsDest = path.join(project, '.e2e-qa-pilot', 'settings.md');
|
|
233
|
+
const settingsShown = path.relative(project, settingsDest);
|
|
234
|
+
if (fs.existsSync(settingsDest)) {
|
|
235
|
+
log(`left ${settingsShown} (your settings are never overwritten)`);
|
|
236
|
+
} else {
|
|
237
|
+
writeFile(settingsDest, fs.readFileSync(path.join(kitRoot, 'templates', 'settings.md'), 'utf8'));
|
|
238
|
+
log(`wrote ${settingsShown}`);
|
|
239
|
+
}
|
|
240
|
+
|
|
241
|
+
for (const editor of editors) {
|
|
242
|
+
const { skillsDir, mcpFile } = EDITORS[editor];
|
|
243
|
+
for (const { command, tool } of CATALOG) {
|
|
244
|
+
installManaged(
|
|
245
|
+
path.join(project, skillsDir, 'skills', command, 'SKILL.md'),
|
|
246
|
+
stubText(command, tool),
|
|
247
|
+
force,
|
|
248
|
+
log,
|
|
249
|
+
project,
|
|
250
|
+
);
|
|
251
|
+
}
|
|
252
|
+
installMcpEntry(path.join(project, mcpFile), log, project);
|
|
253
|
+
}
|
|
254
|
+
|
|
255
|
+
const restart = {
|
|
256
|
+
cursor:
|
|
257
|
+
'Reload the Cursor window, then open a new chat. The chat already open will not see the new MCP server.',
|
|
258
|
+
claude:
|
|
259
|
+
'Quit Claude Code and start it again in this folder, then open a new chat. The session already open will not see the new MCP server.',
|
|
260
|
+
};
|
|
261
|
+
log(`
|
|
262
|
+
Next:
|
|
263
|
+
1. ${editors.map((name) => restart[name]).join('\n ')}
|
|
264
|
+
2. To review an existing case, connect your case tool (for example Qase) in your editor's MCP settings. To review pasted case text, no case tool is required.
|
|
265
|
+
3. Ask in chat: "Review case <id>." or paste the full case and ask to review it.
|
|
266
|
+
The first run asks once for the keys in .e2e-qa-pilot/settings.md that this input needs.`);
|
|
267
|
+
}
|