@muggleai/works 4.4.0 → 4.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +31 -13
- package/dist/{chunk-PMI2DI3V.js → chunk-MNCBJEPQ.js} +311 -35
- package/dist/cli.js +1 -1
- package/dist/index.js +1 -1
- package/dist/plugin/.claude-plugin/plugin.json +1 -1
- package/dist/plugin/.cursor-plugin/plugin.json +1 -1
- package/dist/plugin/README.md +1 -0
- package/dist/plugin/skills/do/open-prs.md +35 -74
- package/dist/plugin/skills/muggle-pr-visual-walkthrough/SKILL.md +181 -0
- package/dist/plugin/skills/muggle-test/SKILL.md +44 -48
- package/dist/plugin/skills/muggle-test-feature-local/SKILL.md +51 -1
- package/dist/plugin/skills/muggle-test-import/SKILL.md +122 -23
- package/dist/plugin/skills/muggle-test-regenerate-missing/SKILL.md +196 -0
- package/dist/plugin/skills/muggle-test-regenerate-missing/evals/evals.json +58 -0
- package/dist/plugin/skills/muggle-test-regenerate-missing/evals/trigger-eval.json +22 -0
- package/dist/release-manifest.json +7 -0
- package/package.json +7 -7
- package/plugin/.claude-plugin/plugin.json +1 -1
- package/plugin/.cursor-plugin/plugin.json +1 -1
- package/plugin/README.md +1 -0
- package/plugin/skills/do/open-prs.md +35 -74
- package/plugin/skills/muggle-pr-visual-walkthrough/SKILL.md +181 -0
- package/plugin/skills/muggle-test/SKILL.md +44 -48
- package/plugin/skills/muggle-test-feature-local/SKILL.md +51 -1
- package/plugin/skills/muggle-test-import/SKILL.md +122 -23
- package/plugin/skills/muggle-test-regenerate-missing/SKILL.md +196 -0
- package/plugin/skills/muggle-test-regenerate-missing/evals/evals.json +58 -0
- package/plugin/skills/muggle-test-regenerate-missing/evals/trigger-eval.json +22 -0
|
@@ -23,80 +23,15 @@ For each repo with changes:
|
|
|
23
23
|
- If E2E acceptance tests have failures: `[E2E FAILING] <goal>`
|
|
24
24
|
- Otherwise: `<goal>`
|
|
25
25
|
- Keep under 70 characters
|
|
26
|
-
3. **
|
|
26
|
+
3. **Render the E2E acceptance block** by invoking the shared `muggle-pr-visual-walkthrough` skill in **Mode B** (render-only for embedding). See "Rendering the E2E acceptance block via the shared skill" below. You receive `{body, comment}` where `body` is the E2E markdown block and `comment` is a non-null overflow comment only when the content exceeds the CLI's byte budget.
|
|
27
|
+
4. **Build the PR body** by concatenating, in order:
|
|
27
28
|
- `## Goal` — the requirements goal
|
|
28
29
|
- `## Acceptance Criteria` — bulleted list (omit section if empty)
|
|
29
30
|
- `## Changes` — summary of what changed in this repo
|
|
30
|
-
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
## Rendering the E2E acceptance results block
|
|
36
|
-
|
|
37
|
-
Do **not** hand-write the `## E2E Acceptance Results` markdown. Use the `muggle build-pr-section` CLI, which renders a deterministic block and decides whether the evidence fits in the PR description or needs to spill into an overflow comment.
|
|
38
|
-
|
|
39
|
-
### Step A: Build the report JSON
|
|
40
|
-
|
|
41
|
-
Assemble the e2e-acceptance report you collected in `e2e-acceptance.md` into a JSON object with this shape:
|
|
42
|
-
|
|
43
|
-
```json
|
|
44
|
-
{
|
|
45
|
-
"projectId": "<project UUID>",
|
|
46
|
-
"tests": [
|
|
47
|
-
{
|
|
48
|
-
"name": "<test case name>",
|
|
49
|
-
"testCaseId": "<UUID>",
|
|
50
|
-
"testScriptId": "<UUID or omitted>",
|
|
51
|
-
"runId": "<UUID>",
|
|
52
|
-
"viewUrl": "<muggle-ai.com run URL>",
|
|
53
|
-
"status": "passed",
|
|
54
|
-
"steps": [
|
|
55
|
-
{ "stepIndex": 0, "action": "<action>", "screenshotUrl": "<URL>" }
|
|
56
|
-
]
|
|
57
|
-
},
|
|
58
|
-
{
|
|
59
|
-
"name": "<test case name>",
|
|
60
|
-
"testCaseId": "<UUID>",
|
|
61
|
-
"runId": "<UUID>",
|
|
62
|
-
"viewUrl": "<muggle-ai.com run URL>",
|
|
63
|
-
"status": "failed",
|
|
64
|
-
"failureStepIndex": 2,
|
|
65
|
-
"error": "<error message>",
|
|
66
|
-
"artifactsDir": "<path, optional>",
|
|
67
|
-
"steps": [
|
|
68
|
-
{ "stepIndex": 0, "action": "<action>", "screenshotUrl": "<URL>" }
|
|
69
|
-
]
|
|
70
|
-
}
|
|
71
|
-
]
|
|
72
|
-
}
|
|
73
|
-
```
|
|
74
|
-
|
|
75
|
-
### Step B: Render the evidence block
|
|
76
|
-
|
|
77
|
-
Pipe the JSON into `muggle build-pr-section`. It writes `{ "body": "...", "comment": "..." | null }` to stdout:
|
|
78
|
-
|
|
79
|
-
```bash
|
|
80
|
-
echo "$REPORT_JSON" | muggle build-pr-section > /tmp/muggle-pr-section.json
|
|
81
|
-
```
|
|
82
|
-
|
|
83
|
-
The command exits nonzero on malformed input and writes a descriptive error to stderr — do not swallow that error, surface it to the user.
|
|
84
|
-
|
|
85
|
-
### Step C: Build the PR body
|
|
86
|
-
|
|
87
|
-
Build the PR body by concatenating, in order:
|
|
88
|
-
|
|
89
|
-
- `## Goal` — the requirements goal
|
|
90
|
-
- `## Acceptance Criteria` — bulleted list (omit section if empty)
|
|
91
|
-
- `## Changes` — summary of what changed in this repo
|
|
92
|
-
- The `body` field from the CLI output (already contains its own `## E2E Acceptance Results` header)
|
|
93
|
-
|
|
94
|
-
### Step D: Create the PR, then post the overflow comment only if present
|
|
95
|
-
|
|
96
|
-
1. Create the PR with `gh pr create --title "..." --body "..." --head <branch>`.
|
|
97
|
-
2. Capture the PR URL and extract the PR number.
|
|
98
|
-
3. If the CLI output's `comment` field is `null`, **do not post a comment** — everything is already in the PR description.
|
|
99
|
-
4. If the CLI output's `comment` field is a non-null string, post it as a follow-up comment:
|
|
31
|
+
- The `body` field from the skill output (already contains its own `## E2E Acceptance Results` header — do not add another)
|
|
32
|
+
5. **Create the PR** using `gh pr create --title "..." --body "..." --head <branch>` in the repo directory.
|
|
33
|
+
6. **Capture the PR URL** and extract the PR number.
|
|
34
|
+
7. **Post the overflow `comment` only if it is non-null.** In the common case, `comment` is `null` and nothing is posted. Never post speculatively.
|
|
100
35
|
|
|
101
36
|
```bash
|
|
102
37
|
gh pr comment <PR#> --body "$(cat <<'EOF'
|
|
@@ -105,11 +40,37 @@ Build the PR body by concatenating, in order:
|
|
|
105
40
|
)"
|
|
106
41
|
```
|
|
107
42
|
|
|
43
|
+
## Rendering the E2E acceptance block via the shared skill
|
|
44
|
+
|
|
45
|
+
**Do not hand-write the `## E2E Acceptance Results` markdown, and do not call `muggle build-pr-section` directly from this stage.** The rendering workflow is owned by the shared **`muggle-pr-visual-walkthrough`** skill (see `plugin/skills/muggle-pr-visual-walkthrough/SKILL.md`), which wraps the CLI and enforces the `E2eReport` input contract with a Zod schema.
|
|
46
|
+
|
|
47
|
+
### Input — the `E2eReport` JSON
|
|
48
|
+
|
|
49
|
+
The `e2e-acceptance.md` stage already produces an `E2eReport` with the exact shape the skill expects (`projectId` + `tests[]` with per-test `name`, `testCaseId`, `testScriptId`, `runId`, `viewUrl`, `status`, and `steps[]` of `{stepIndex, action, screenshotUrl}`; failed tests additionally have `failureStepIndex`, `error`, and optionally `artifactsDir`). Pass it through unchanged — do not reshape it. The full schema is documented in the shared skill.
|
|
50
|
+
|
|
51
|
+
### Invocation — Mode B (render-only)
|
|
52
|
+
|
|
53
|
+
Invoke `muggle-pr-visual-walkthrough` via the `Skill` tool with the `E2eReport` already in context. The skill will:
|
|
54
|
+
|
|
55
|
+
1. Validate the `E2eReport` and call `muggle build-pr-section` (piping the JSON to stdin).
|
|
56
|
+
2. Parse the CLI's `{body, comment}` stdout.
|
|
57
|
+
3. **Return `{body, comment}` to this stage's conversation** without posting anything — because Mode B is render-only. `body` is the E2E markdown block; `comment` is a non-null overflow follow-up comment only when content exceeds the byte budget, otherwise `null`.
|
|
58
|
+
|
|
59
|
+
Mode A (where the skill itself finds an existing PR and posts a `gh pr comment`) is **not used by `muggle-do`** — it's for interactive callers like `muggle-test` that are mid-development with a PR already open. `muggle-do` always creates new PRs, so it always uses Mode B.
|
|
60
|
+
|
|
61
|
+
### After rendering
|
|
62
|
+
|
|
63
|
+
Back in this stage:
|
|
64
|
+
|
|
65
|
+
- Embed `body` in the `gh pr create --body` body (see step 4 above).
|
|
66
|
+
- Post the overflow `comment` as a follow-up **only when it is non-null** (see step 7 above).
|
|
67
|
+
- If the CLI exited non-zero, the skill surfaces the stderr error — do not swallow it, surface it to the user.
|
|
68
|
+
|
|
108
69
|
### Notes on fit vs. overflow
|
|
109
70
|
|
|
110
|
-
- **
|
|
111
|
-
- **
|
|
112
|
-
- You do not make the fit-vs-overflow decision — the CLI does. Never post the comment
|
|
71
|
+
- **Common case (fit):** the full evidence (summary, per-test rows, collapsible failure details) lives in the PR description, `comment` is `null`, no follow-up comment is posted.
|
|
72
|
+
- **Overflow case:** the CLI detects the full body would exceed its byte budget; `body` contains the summary, per-test rows, and a pointer line; `comment` contains the overflow details. Post both.
|
|
73
|
+
- You do not make the fit-vs-overflow decision — the CLI does. Never post the comment when it is `null`.
|
|
113
74
|
|
|
114
75
|
## Output
|
|
115
76
|
|
|
@@ -0,0 +1,181 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: muggle-pr-visual-walkthrough
|
|
3
|
+
description: Renders and posts a visual walkthrough of Muggle AI E2E acceptance test results to a PR — per-test-case dashboard links, step-by-step screenshots, and pass/fail summary — using the `muggle build-pr-section` CLI for deterministic formatting with automatic fit-vs-overflow. Use at the end of any Muggle test run (local or remote) to give PR reviewers clickable visual evidence that user flows work. Triggers on 'post results to PR', 'attach walkthrough to PR', 'share E2E screenshots on the PR', 'add visual walkthrough to PR'.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Muggle PR Visual Walkthrough
|
|
7
|
+
|
|
8
|
+
Renders a visual walkthrough of Muggle AI E2E acceptance test results and posts it to a PR. Each test case is linked to its detail page on the Muggle AI dashboard, so PR reviewers can click through to see step-by-step screenshots and action scripts — not just a pass/fail flag.
|
|
9
|
+
|
|
10
|
+
This is the **canonical PR-walkthrough workflow** shared across every Muggle entry point:
|
|
11
|
+
|
|
12
|
+
| Caller | Mode | When to invoke |
|
|
13
|
+
| :--- | :--- | :--- |
|
|
14
|
+
| `muggle-test` | **Mode A** (post to existing PR) | After publishing results, user opts in via `AskQuestion` |
|
|
15
|
+
| `muggle-test-feature-local` | **Mode A** (post to existing PR) | After publishing the run, user opts in via `AskQuestion` |
|
|
16
|
+
| `muggle-do` / `open-prs.md` | **Mode B** (render-only for embedding) | During PR creation — caller embeds `body` in `gh pr create` and posts `comment` as follow-up |
|
|
17
|
+
|
|
18
|
+
Rendering is always done by `muggle build-pr-section`, a battle-tested CLI that handles deterministic markdown layout, per-step screenshots, and automatic fit-vs-overflow (oversized content spills into a follow-up comment). Never hand-write the walkthrough markdown.
|
|
19
|
+
|
|
20
|
+
## Input contract: the `E2eReport` JSON
|
|
21
|
+
|
|
22
|
+
Every caller must build an `E2eReport` JSON object and have it in conversation context before invoking this skill. The schema is defined in `src/cli/pr-section/types.ts` (`E2eReportSchema`) and enforced by the CLI with Zod — malformed input exits non-zero with a descriptive stderr message.
|
|
23
|
+
|
|
24
|
+
```json
|
|
25
|
+
{
|
|
26
|
+
"projectId": "<UUID>",
|
|
27
|
+
"tests": [
|
|
28
|
+
{
|
|
29
|
+
"name": "<test case name>",
|
|
30
|
+
"testCaseId": "<UUID>",
|
|
31
|
+
"testScriptId": "<UUID>",
|
|
32
|
+
"runId": "<UUID>",
|
|
33
|
+
"viewUrl": "https://www.muggle-ai.com/...",
|
|
34
|
+
"status": "passed",
|
|
35
|
+
"steps": [
|
|
36
|
+
{ "stepIndex": 0, "action": "Click login button", "screenshotUrl": "https://..." },
|
|
37
|
+
{ "stepIndex": 1, "action": "Type email", "screenshotUrl": "https://..." }
|
|
38
|
+
]
|
|
39
|
+
},
|
|
40
|
+
{
|
|
41
|
+
"name": "Checkout flow",
|
|
42
|
+
"testCaseId": "<UUID>",
|
|
43
|
+
"testScriptId": "<UUID>",
|
|
44
|
+
"runId": "<UUID>",
|
|
45
|
+
"viewUrl": "https://www.muggle-ai.com/...",
|
|
46
|
+
"status": "failed",
|
|
47
|
+
"steps": [
|
|
48
|
+
{ "stepIndex": 0, "action": "Open cart", "screenshotUrl": "https://..." }
|
|
49
|
+
],
|
|
50
|
+
"failureStepIndex": 2,
|
|
51
|
+
"error": "Element not found: Click checkout button",
|
|
52
|
+
"artifactsDir": "/Users/.../~/.muggle-ai/sessions/<runId>"
|
|
53
|
+
}
|
|
54
|
+
]
|
|
55
|
+
}
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
Required fields per test: `name`, `testCaseId`, `runId`, `viewUrl`, `status`, `steps[]` with `{stepIndex, action, screenshotUrl}`. Failed tests additionally require `failureStepIndex` and `error`. `testScriptId` and `artifactsDir` are optional.
|
|
59
|
+
|
|
60
|
+
If any required field is missing, stop and tell the caller exactly what's missing. Never fabricate data.
|
|
61
|
+
|
|
62
|
+
## Step 1: Gather the `E2eReport`
|
|
63
|
+
|
|
64
|
+
How to assemble the JSON depends on which caller you are:
|
|
65
|
+
|
|
66
|
+
### From `muggle-test` / `muggle-test-feature-local` (local mode)
|
|
67
|
+
|
|
68
|
+
After `muggle-local-publish-test-script` returns `{testScriptId, viewUrl, ...}` for each run:
|
|
69
|
+
|
|
70
|
+
1. Call `muggle-remote-test-script-get` with `testScriptId` to fetch the published script.
|
|
71
|
+
2. Extract `steps[]` — for each step, build `{stepIndex: <index>, action: operation.action, screenshotUrl: operation.screenshotUrl}`.
|
|
72
|
+
3. Determine `status` from the local run result (`muggle-local-run-result-get`).
|
|
73
|
+
4. For failures, read `failureStepIndex`, `error`, and `artifactsDir` from the run result.
|
|
74
|
+
5. Assemble the `E2eReport` with `projectId` from the test run.
|
|
75
|
+
|
|
76
|
+
### From `muggle-do` (`open-prs.md`)
|
|
77
|
+
|
|
78
|
+
The `e2e-acceptance.md` stage already produces an `E2eReport` with the exact shape above — that is the report's native output format. Pass it through unchanged.
|
|
79
|
+
|
|
80
|
+
### Direct invocation (user asked to post existing results)
|
|
81
|
+
|
|
82
|
+
The caller must have already executed tests and published them. If the `E2eReport` is not in context, stop and tell the user to run `muggle-test` or `muggle-test-feature-local` first.
|
|
83
|
+
|
|
84
|
+
## Step 2: Render via `muggle build-pr-section`
|
|
85
|
+
|
|
86
|
+
Pipe the `E2eReport` JSON to the CLI. It writes `{"body": "...", "comment": "..." | null}` to stdout — the `body` is the E2E markdown block, and `comment` is a non-null overflow comment only when the full body exceeds the byte budget (default 60 KB).
|
|
87
|
+
|
|
88
|
+
```bash
|
|
89
|
+
echo "$REPORT_JSON" | muggle build-pr-section > /tmp/muggle-pr-section.json
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
- Exit **non-zero** → the CLI wrote a descriptive error to stderr. Surface it to the user; do not swallow it.
|
|
93
|
+
- `comment` is **`null`** (fit case) → everything is inline in `body`. Post `body` only.
|
|
94
|
+
- `comment` is a **non-null string** (overflow case) → `body` contains the summary + a pointer; `comment` contains the full per-step details. Post both, in order.
|
|
95
|
+
|
|
96
|
+
Never hand-write the walkthrough markdown. Never modify the CLI's output before posting. The CLI owns the format.
|
|
97
|
+
|
|
98
|
+
## Step 3 — Mode A: Post to an existing PR
|
|
99
|
+
|
|
100
|
+
Used by `muggle-test` and `muggle-test-feature-local`, where the user is mid-development and a PR already exists on the current branch.
|
|
101
|
+
|
|
102
|
+
### 3A.1: Find the PR
|
|
103
|
+
|
|
104
|
+
```bash
|
|
105
|
+
gh pr view --json number,url,title 2>/dev/null
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
- **PR exists** → continue to 3A.2
|
|
109
|
+
- **No PR exists** → use `AskQuestion`:
|
|
110
|
+
- "Create a new PR with the visual walkthrough in the body"
|
|
111
|
+
- "Skip posting"
|
|
112
|
+
- If the user chooses to create a new PR, switch to Mode B and return the rendered `body`/`comment` to the caller for embedding in `gh pr create`. Do not create the PR directly from this skill unless the caller has no better way to do it.
|
|
113
|
+
- **`gh` not installed or not authenticated** → tell the user, suggest `gh auth login`, stop.
|
|
114
|
+
|
|
115
|
+
### 3A.2: Post the body as a PR comment
|
|
116
|
+
|
|
117
|
+
```bash
|
|
118
|
+
gh pr comment <pr-number> --body "$(cat <<'EOF'
|
|
119
|
+
<contents of body field from CLI output>
|
|
120
|
+
EOF
|
|
121
|
+
)"
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
### 3A.3: Post the overflow comment only if the CLI emitted one
|
|
125
|
+
|
|
126
|
+
```bash
|
|
127
|
+
gh pr comment <pr-number> --body "$(cat <<'EOF'
|
|
128
|
+
<contents of comment field from CLI output>
|
|
129
|
+
EOF
|
|
130
|
+
)"
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
**Skip this step entirely if `comment` is `null`** — do not post a placeholder. The CLI decides fit-vs-overflow; never post the overflow comment speculatively.
|
|
134
|
+
|
|
135
|
+
### 3A.4: Confirm to the user
|
|
136
|
+
|
|
137
|
+
> "Visual walkthrough posted to PR #<number>. Reviewers can click any test case link to see the step-by-step screenshots on the Muggle AI dashboard."
|
|
138
|
+
|
|
139
|
+
Include the PR URL in the confirmation.
|
|
140
|
+
|
|
141
|
+
## Step 3 — Mode B: Return rendered block for embedding in a new PR
|
|
142
|
+
|
|
143
|
+
Used by `muggle-do`'s `open-prs.md`, where the PR does not exist yet and the caller is assembling the PR body from multiple sections (`## Goal`, `## Acceptance Criteria`, `## Changes`, plus this walkthrough).
|
|
144
|
+
|
|
145
|
+
Instead of posting, **return** the CLI output to the caller's context so they can:
|
|
146
|
+
|
|
147
|
+
1. **Embed `body`** in their PR body, concatenated after `## Changes`. `body` already includes its own `## E2E Acceptance Results` header — do not add another.
|
|
148
|
+
2. **Create the PR** with `gh pr create --title "..." --body "..."` using the concatenated body.
|
|
149
|
+
3. **Post `comment` as a follow-up only if the CLI emitted one:**
|
|
150
|
+
|
|
151
|
+
```bash
|
|
152
|
+
gh pr comment <new-pr-number> --body "$(cat <<'EOF'
|
|
153
|
+
<contents of comment field>
|
|
154
|
+
EOF
|
|
155
|
+
)"
|
|
156
|
+
```
|
|
157
|
+
|
|
158
|
+
Skip if `comment` is `null`.
|
|
159
|
+
|
|
160
|
+
In Mode B, this skill does not call `gh pr comment` or `gh pr create` itself — the caller owns PR creation because it also owns branch pushing, title building (including `[E2E FAILING]` prefix on failures), and multi-repo orchestration.
|
|
161
|
+
|
|
162
|
+
## Tool Reference
|
|
163
|
+
|
|
164
|
+
| Phase | Tool |
|
|
165
|
+
|:------|:-----|
|
|
166
|
+
| Gather per-step data (muggle-test, muggle-test-feature-local) | `muggle-remote-test-script-get` |
|
|
167
|
+
| Render the walkthrough markdown | `muggle build-pr-section` (shell) |
|
|
168
|
+
| Find existing PR (Mode A) | `gh pr view` |
|
|
169
|
+
| Post comment(s) (Mode A) | `gh pr comment` |
|
|
170
|
+
| Create new PR (Mode B, caller handles) | `gh pr create` |
|
|
171
|
+
| User confirmation (Mode A no-PR branch) | `AskQuestion` |
|
|
172
|
+
|
|
173
|
+
## Guardrails
|
|
174
|
+
|
|
175
|
+
- **Never hand-write the walkthrough markdown** — always call `muggle build-pr-section`. The CLI is the single source of truth for formatting.
|
|
176
|
+
- **Never modify the CLI's output** — post `body` and (if present) `comment` verbatim. Any reformatting defeats the fit-vs-overflow budget math.
|
|
177
|
+
- **Never invent report fields** — if `projectId`, a per-test `viewUrl`, or per-step `screenshotUrl` is missing, stop and report what's missing. Do not fabricate URLs or fill in placeholders.
|
|
178
|
+
- **Never post the overflow comment when `comment` is `null`** — the CLI decides fit-vs-overflow.
|
|
179
|
+
- **Never create a PR without confirmation in Mode A** — if no PR exists, ask the user or switch to Mode B and hand back to the caller.
|
|
180
|
+
- **Don't run tests** — this skill only renders and posts existing results. If the `E2eReport` is not in context, redirect the caller to `muggle-test`, `muggle-test-feature-local`, or `muggle-do`.
|
|
181
|
+
- **Mode A vs Mode B is chosen by the caller, not the user** — `muggle-test` always uses Mode A; `muggle-do` always uses Mode B. Don't ask the user which mode to use.
|
|
@@ -117,7 +117,7 @@ The user MUST explicitly tell you which use case(s) to use.
|
|
|
117
117
|
1. Ask the user to describe the use case in plain English
|
|
118
118
|
2. Call `muggle-remote-use-case-create-from-prompts`:
|
|
119
119
|
- `projectId`: The project ID
|
|
120
|
-
- `
|
|
120
|
+
- `instructions`: A plain array of strings, one per use case — e.g. `["<user's description>"]`
|
|
121
121
|
3. Present the created use case and confirm it's correct
|
|
122
122
|
|
|
123
123
|
## Step 6: Select Test Case (User Must Choose)
|
|
@@ -279,66 +279,62 @@ open "https://www.muggle-ai.com/muggleTestV0/dashboard/projects/{projectId}/runs
|
|
|
279
279
|
Tell the user:
|
|
280
280
|
> "I've opened the Muggle AI dashboard in your browser — you can see the test results, step-by-step screenshots, and action scripts there."
|
|
281
281
|
|
|
282
|
-
## Step 9: Post
|
|
282
|
+
## Step 9: Offer to Post Visual Walkthrough to PR
|
|
283
283
|
|
|
284
|
-
After reporting results,
|
|
284
|
+
After reporting results, ask the user if they want to attach a **visual walkthrough** — a markdown block with per-test-case dashboard links and step-by-step screenshots — to the current branch's open PR. The rendering and posting workflow lives in the shared `muggle-pr-visual-walkthrough` skill; this step gathers the required input and hands off.
|
|
285
285
|
|
|
286
|
-
### 9a:
|
|
286
|
+
### 9a: Gather per-step screenshots (required input for the shared skill)
|
|
287
287
|
|
|
288
|
-
|
|
289
|
-
gh pr view --json number,url,title 2>/dev/null
|
|
290
|
-
```
|
|
288
|
+
The shared skill takes an **`E2eReport` JSON** that includes per-step screenshot URLs. You already have `projectId`, `testCaseId`, `runId`, `viewUrl`, and `status` from earlier steps — you still need the step-level data.
|
|
291
289
|
|
|
292
|
-
|
|
293
|
-
- If no PR exists → use `AskQuestion`:
|
|
294
|
-
- "Create PR with E2E acceptance results"
|
|
295
|
-
- "Skip posting to PR"
|
|
290
|
+
For each published run from Step 7A:
|
|
296
291
|
|
|
297
|
-
|
|
292
|
+
1. Call `muggle-remote-test-script-get` with the `testScriptId` returned by `muggle-local-publish-test-script`.
|
|
293
|
+
2. Extract from the response: `steps[].operation.action` (description) and `steps[].operation.screenshotUrl` (cloud URL).
|
|
294
|
+
3. Build a `steps` array: `[{ stepIndex: 0, action: "...", screenshotUrl: "..." }, ...]`.
|
|
295
|
+
4. If the run failed, also capture `failureStepIndex`, `error`, and the local `artifactsDir` from `muggle-local-run-result-get`.
|
|
298
296
|
|
|
299
|
-
|
|
297
|
+
Assemble the report:
|
|
300
298
|
|
|
301
|
-
```
|
|
302
|
-
|
|
299
|
+
```json
|
|
300
|
+
{
|
|
301
|
+
"projectId": "<projectId>",
|
|
302
|
+
"tests": [
|
|
303
|
+
{
|
|
304
|
+
"name": "<test case title>",
|
|
305
|
+
"testCaseId": "<id>",
|
|
306
|
+
"testScriptId": "<id>",
|
|
307
|
+
"runId": "<id>",
|
|
308
|
+
"viewUrl": "<publish response viewUrl>",
|
|
309
|
+
"status": "passed",
|
|
310
|
+
"steps": [{ "stepIndex": 0, "action": "...", "screenshotUrl": "..." }]
|
|
311
|
+
}
|
|
312
|
+
]
|
|
313
|
+
}
|
|
314
|
+
```
|
|
303
315
|
|
|
304
|
-
|
|
316
|
+
See the shared skill for the full schema (including the failed-test shape with `failureStepIndex` and `error`).
|
|
305
317
|
|
|
306
|
-
|
|
307
|
-
|-----------|--------|---------|
|
|
308
|
-
| [Login with valid creds](https://www.muggle-ai.com/muggleTestV0/dashboard/projects/{projectId}/scripts?modal=script-details&testCaseId={testCaseId}) | ✅ PASSED | 8 steps, 12.3s |
|
|
309
|
-
| [Login with invalid creds](https://www.muggle-ai.com/muggleTestV0/dashboard/projects/{projectId}/scripts?modal=script-details&testCaseId={testCaseId}) | ✅ PASSED | 6 steps, 9.1s |
|
|
310
|
-
| [Checkout flow](https://www.muggle-ai.com/muggleTestV0/dashboard/projects/{projectId}/scripts?modal=script-details&testCaseId={testCaseId}) | ❌ FAILED | Step 7: "Click checkout button" — element not found |
|
|
318
|
+
### 9b: Ask the user
|
|
311
319
|
|
|
312
|
-
|
|
313
|
-
<summary>Failed test details</summary>
|
|
320
|
+
Use `AskQuestion`:
|
|
314
321
|
|
|
315
|
-
|
|
316
|
-
- **Failed at**: Step 7 — "Click checkout button"
|
|
317
|
-
- **Error**: Element not found
|
|
318
|
-
- **Local artifacts**: `~/.muggle-ai/sessions/{runId}/`
|
|
319
|
-
- **Screenshots**: `~/.muggle-ai/sessions/{runId}/screenshots/`
|
|
322
|
+
> "Post a visual walkthrough of these results to the PR? Reviewers can click each test case to see step-by-step screenshots on the Muggle AI dashboard."
|
|
320
323
|
|
|
321
|
-
|
|
324
|
+
- Option 1: "Yes, post to PR"
|
|
325
|
+
- Option 2: "Skip"
|
|
322
326
|
|
|
323
|
-
|
|
324
|
-
*Generated by [Muggle AI](https://www.muggle-ai.com) — change-driven E2E acceptance testing*
|
|
325
|
-
```
|
|
326
|
-
|
|
327
|
-
### 9c: Post to the PR
|
|
328
|
-
|
|
329
|
-
If PR already exists — add as a comment:
|
|
330
|
-
```bash
|
|
331
|
-
gh pr comment {pr-number} --body "$(cat <<'EOF'
|
|
332
|
-
{the E2E acceptance comment body from 9b}
|
|
333
|
-
EOF
|
|
334
|
-
)"
|
|
335
|
-
```
|
|
327
|
+
### 9c: Invoke the shared skill in Mode A
|
|
336
328
|
|
|
337
|
-
If
|
|
329
|
+
If the user chooses "Yes, post to PR", invoke the `muggle-pr-visual-walkthrough` skill via the `Skill` tool. With the `E2eReport` already in context, the skill will:
|
|
338
330
|
|
|
339
|
-
|
|
331
|
+
1. Call `muggle build-pr-section` to render the markdown block (fit-vs-overflow automatic)
|
|
332
|
+
2. Find the PR via `gh pr view`
|
|
333
|
+
3. Post `body` as a `gh pr comment`
|
|
334
|
+
4. Post the overflow `comment` as a second comment (only if the CLI emitted one)
|
|
335
|
+
5. Confirm the PR URL to the user
|
|
340
336
|
|
|
341
|
-
|
|
337
|
+
This skill always uses **Mode A** (post to an existing PR); `muggle-do` is the only caller that uses Mode B. Do not attempt to render the walkthrough markdown yourself — delegate to the shared skill.
|
|
342
338
|
|
|
343
339
|
## Tool Reference
|
|
344
340
|
|
|
@@ -360,8 +356,9 @@ If creating a new PR — include the E2E acceptance section in the PR body along
|
|
|
360
356
|
| Results | `muggle-local-run-result-get` | Local |
|
|
361
357
|
| Results | `muggle-remote-wf-get-ts-gen-latest-run` | Remote |
|
|
362
358
|
| Publish | `muggle-local-publish-test-script` | Local |
|
|
359
|
+
| Per-step screenshots (for walkthrough) | `muggle-remote-test-script-get` | Both |
|
|
363
360
|
| Browser | `open` (shell command) | Both |
|
|
364
|
-
| PR | `
|
|
361
|
+
| PR walkthrough | `muggle-pr-visual-walkthrough` (shared skill) | Both |
|
|
365
362
|
|
|
366
363
|
## Guardrails
|
|
367
364
|
|
|
@@ -375,6 +372,5 @@ If creating a new PR — include the E2E acceptance section in the PR body along
|
|
|
375
372
|
- **Never silently drop test cases** — log failures and continue, then report them
|
|
376
373
|
- **Never guess the URL** — always ask the user for localhost or preview URL
|
|
377
374
|
- **Always publish before opening browser** — the dashboard needs the published data to show results
|
|
378
|
-
- **
|
|
379
|
-
- **Always check for PR before posting** — don't create a PR comment if there's no PR (ask user first)
|
|
375
|
+
- **Delegate PR posting to `muggle-pr-visual-walkthrough`** — never inline the walkthrough markdown or call `gh pr comment` directly from this skill; ask the user and hand off
|
|
380
376
|
- **Can be invoked at any state** — if the user already has a project or use cases set up, skip to the relevant step rather than re-doing everything
|
|
@@ -57,7 +57,7 @@ Prompt for projects: "Pick the project to group this test into:"
|
|
|
57
57
|
- **Project — Create new project:** Collect `projectName`, `description`, and `url` (may be the local app URL, e.g. `http://localhost:3999`). Call `muggle-remote-project-create`. Use the returned `projectId` and continue.
|
|
58
58
|
- **Use case — Create new use case:** User provides a natural-language instruction (or you reuse their testing goal).
|
|
59
59
|
1. `muggle-remote-use-case-prompt-preview` with `projectId`, `instruction` — show preview; get confirmation via `AskQuestion`.
|
|
60
|
-
2. `muggle-remote-use-case-create-from-prompts` with `projectId
|
|
60
|
+
2. `muggle-remote-use-case-create-from-prompts` with `projectId` and `instructions: ["<the user's natural-language instruction>"]` — persist. Use the created use case id and continue to test-case selection.
|
|
61
61
|
- **Test case — Create new test case** (requires a chosen `useCaseId`): User provides an instruction describing what to test.
|
|
62
62
|
1. `muggle-remote-test-case-generate-from-prompt` with `projectId`, `useCaseId`, `instruction` — **preview only** (server test-case prompt preview); show the returned draft(s); get confirmation via `AskQuestion`.
|
|
63
63
|
2. Persist the accepted draft with `muggle-remote-test-case-create`, mapping preview fields into the required properties (`title`, `description`, `goal`, `expectedResult`, `url`, etc.). Then continue from **section 4** with that `testCaseId`.
|
|
@@ -126,6 +126,55 @@ Only call local execute tools with `approveElectronAppLaunch: true` after the us
|
|
|
126
126
|
- `muggle-local-run-result-get` with the run id from execute.
|
|
127
127
|
- Include: status, duration, pass/fail summary, per-step summary, artifact/screenshot paths, errors if failed, and script view URL when publishing ran.
|
|
128
128
|
|
|
129
|
+
### 9. Offer to post a visual walkthrough to the PR
|
|
130
|
+
|
|
131
|
+
After reporting results, gather the required input and hand off to the shared **`muggle-pr-visual-walkthrough`** skill, which renders the walkthrough via `muggle build-pr-section` and posts it to the current branch's open PR.
|
|
132
|
+
|
|
133
|
+
#### 9a: Gather per-step screenshots
|
|
134
|
+
|
|
135
|
+
The shared skill takes an **`E2eReport` JSON** that includes per-step screenshot URLs. After step 7 has called `muggle-local-publish-test-script` and you have the `testScriptId`:
|
|
136
|
+
|
|
137
|
+
1. Call `muggle-remote-test-script-get` with the `testScriptId`.
|
|
138
|
+
2. Extract per step: `steps[].operation.action` and `steps[].operation.screenshotUrl`.
|
|
139
|
+
3. Build the `steps` array: `[{ stepIndex: 0, action: "...", screenshotUrl: "..." }, ...]`.
|
|
140
|
+
4. If the run failed, capture `failureStepIndex`, `error`, and the local `artifactsDir` from the run result in step 8.
|
|
141
|
+
|
|
142
|
+
Assemble the `E2eReport`:
|
|
143
|
+
|
|
144
|
+
```json
|
|
145
|
+
{
|
|
146
|
+
"projectId": "<projectId from step 2>",
|
|
147
|
+
"tests": [
|
|
148
|
+
{
|
|
149
|
+
"name": "<test case title>",
|
|
150
|
+
"testCaseId": "<id>",
|
|
151
|
+
"testScriptId": "<id from publish>",
|
|
152
|
+
"runId": "<runId from execute>",
|
|
153
|
+
"viewUrl": "<viewUrl from publish>",
|
|
154
|
+
"status": "passed",
|
|
155
|
+
"steps": [{ "stepIndex": 0, "action": "...", "screenshotUrl": "..." }]
|
|
156
|
+
}
|
|
157
|
+
]
|
|
158
|
+
}
|
|
159
|
+
```
|
|
160
|
+
|
|
161
|
+
See the `muggle-pr-visual-walkthrough` skill for the full schema including the failed-test shape.
|
|
162
|
+
|
|
163
|
+
#### 9b: Ask the user
|
|
164
|
+
|
|
165
|
+
Use `AskQuestion`:
|
|
166
|
+
|
|
167
|
+
> "Post a visual walkthrough of this run to the PR? Reviewers can click the test case to see step-by-step screenshots on the Muggle AI dashboard."
|
|
168
|
+
|
|
169
|
+
- Option 1: "Yes, post to PR"
|
|
170
|
+
- Option 2: "Skip"
|
|
171
|
+
|
|
172
|
+
#### 9c: Invoke the shared skill in Mode A
|
|
173
|
+
|
|
174
|
+
If the user chooses "Yes, post to PR", invoke the `muggle-pr-visual-walkthrough` skill via the `Skill` tool. With the `E2eReport` in context, the skill renders the markdown block via the CLI, finds the PR via `gh pr view`, posts `body` as a comment, posts the overflow `comment` only if the CLI emitted one, and confirms the PR URL to the user.
|
|
175
|
+
|
|
176
|
+
Always use **Mode A** (post to existing PR) from this skill. Never hand-write the walkthrough markdown or call `gh pr comment` directly — delegate to `muggle-pr-visual-walkthrough`.
|
|
177
|
+
|
|
129
178
|
## Non-negotiables
|
|
130
179
|
|
|
131
180
|
- No silent auth skip; no launching Electron without approval via `AskQuestion`.
|
|
@@ -134,3 +183,4 @@ Only call local execute tools with `approveElectronAppLaunch: true` after the us
|
|
|
134
183
|
- Replay: never hand-built or simplified `actionScript` — only from `muggle-remote-action-script-get`.
|
|
135
184
|
- Use `AskQuestion` for every selection — project, use case, test case, script, and approval. Never ask the user to type a number.
|
|
136
185
|
- Project, use case, and test case selection lists must always include "Create new ...". Include "Show full list" whenever the API returned at least one row for that step; omit "Show full list" when the list is empty (offer "Create new ..." only). For creates, use preview tools (`muggle-remote-use-case-prompt-preview`, `muggle-remote-test-case-generate-from-prompt`) before persisting.
|
|
186
|
+
- PR posting is always optional and always delegated to the `muggle-pr-visual-walkthrough` skill — never inline the walkthrough markdown or call `gh pr comment` directly from this skill.
|