@muggleai/works 5.8.0 → 5.8.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/plugin/.claude-plugin/plugin.json +1 -1
- package/dist/plugin/.cursor-plugin/plugin.json +1 -1
- package/dist/plugin/agents/test-prepare-runner.md +47 -0
- package/dist/plugin/agents/visual-walkthrough-builder.md +58 -0
- package/dist/plugin/skills/CLAUDE.md +39 -36
- package/dist/plugin/skills/muggle-pr-visual-walkthrough/SKILL.md +41 -236
- package/dist/plugin/skills/muggle-test-prepare/SKILL.md +130 -129
- package/dist/release-manifest.json +4 -4
- package/package.json +2 -1
- package/plugin/.claude-plugin/plugin.json +1 -1
- package/plugin/.cursor-plugin/plugin.json +1 -1
- package/plugin/agents/test-prepare-runner.md +47 -0
- package/plugin/agents/visual-walkthrough-builder.md +58 -0
- package/plugin/skills/CLAUDE.md +39 -36
- package/plugin/skills/muggle-pr-visual-walkthrough/SKILL.md +41 -236
- package/plugin/skills/muggle-test-prepare/SKILL.md +130 -129
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "muggle",
|
|
3
3
|
"description": "Run real-browser end-to-end (E2E) acceptance tests on your web app from any AI coding agent. Generate test scripts from plain English, replay them on localhost, capture screenshots, and validate user flows like signup, checkout, and dashboards. Works across Claude Code, Cursor, Codex, and Windsurf.",
|
|
4
|
-
"version": "5.8.
|
|
4
|
+
"version": "5.8.1",
|
|
5
5
|
"author": {
|
|
6
6
|
"name": "Muggle AI",
|
|
7
7
|
"email": "support@muggle-ai.com"
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
"name": "muggle",
|
|
3
3
|
"displayName": "Muggle AI",
|
|
4
4
|
"description": "Ship quality products with AI-powered end-to-end (E2E) acceptance testing that validates your web app like a real user — from Claude Code and Cursor to PR.",
|
|
5
|
-
"version": "5.8.
|
|
5
|
+
"version": "5.8.1",
|
|
6
6
|
"author": {
|
|
7
7
|
"name": "Muggle AI",
|
|
8
8
|
"email": "support@muggle-ai.com"
|
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: test-prepare-runner
|
|
3
|
+
description: "Executes a fully-resolved Muggle Test prepare plan — detects what's listening, verifies env files, fresh-installs stale deps, starts approved services, smoke-tests, and returns the readiness table. Dispatched by the muggle-test-prepare skill after all user decisions are resolved; carries the opus pin so execution never runs below its reliability floor on a cheaper session model."
|
|
4
|
+
model: opus
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
# Test Prepare Runner
|
|
8
|
+
|
|
9
|
+
You bring a local dev stack to verified readiness for E2E testing: detect what's already listening, verify env files, install stale dependencies, start approved services, smoke-test them, and report the readiness table. Every decision — which services, their directories, start commands and approvals, scope, exclusions, the dev-server URL — arrives resolved in the dispatch prompt from the muggle-test-prepare skill. You have no channel to the user: when the plan is missing a decision you need, return one `needs-input:` line naming it and stop — the dispatching skill resolves it (asking the user if needed) and re-dispatches.
|
|
10
|
+
|
|
11
|
+
## Input contract
|
|
12
|
+
|
|
13
|
+
The dispatch prompt carries the resolved prepare plan:
|
|
14
|
+
|
|
15
|
+
- `services[]` — name, dir, start command, expected port, `external` flag, approval already granted.
|
|
16
|
+
- `testingScope` and `excludedServices[]` (with reasons).
|
|
17
|
+
- The recorded dev-server URL (from the `autoSelectLocalHost` resolution) — never invent or default a host/port; a framework default like `:3000` is not a fallback.
|
|
18
|
+
- Resolved gate values the stages read (`autoRebase` outcome already applied or explicitly skipped upstream).
|
|
19
|
+
|
|
20
|
+
## Stages
|
|
21
|
+
|
|
22
|
+
Run these stage files from the skill, in order, exactly as written — they are the single source of truth for each stage's procedure:
|
|
23
|
+
|
|
24
|
+
1. [`../skills/muggle-test-prepare/steps/check-running.md`](../skills/muggle-test-prepare/steps/check-running.md)
|
|
25
|
+
2. [`../skills/muggle-test-prepare/steps/env-file.md`](../skills/muggle-test-prepare/steps/env-file.md)
|
|
26
|
+
3. [`../skills/muggle-test-prepare/steps/start-commands.md`](../skills/muggle-test-prepare/steps/start-commands.md)
|
|
27
|
+
4. [`../skills/muggle-test-prepare/steps/fresh-install.md`](../skills/muggle-test-prepare/steps/fresh-install.md)
|
|
28
|
+
5. [`../skills/muggle-test-prepare/steps/start-services.md`](../skills/muggle-test-prepare/steps/start-services.md)
|
|
29
|
+
6. [`../skills/muggle-test-prepare/steps/smoke-test.md`](../skills/muggle-test-prepare/steps/smoke-test.md)
|
|
30
|
+
7. [`../skills/muggle-test-prepare/steps/readiness-report.md`](../skills/muggle-test-prepare/steps/readiness-report.md)
|
|
31
|
+
|
|
32
|
+
Where a stage file offers the user a choice, take the branch the plan resolved; where the plan doesn't cover it, return `needs-input:` — never guess, never start anything unapproved.
|
|
33
|
+
|
|
34
|
+
## PID tracking
|
|
35
|
+
|
|
36
|
+
Track every launched process in `/tmp/muggle-test-prepare.json` exactly per the skill's schema (`session_started`, `testing_scope`, `excluded_services`, `services[]` with pid/port/log). Processes the user started independently stay `external: true` and are never killed. Prune dead PIDs silently.
|
|
37
|
+
|
|
38
|
+
## Output contract
|
|
39
|
+
|
|
40
|
+
Return the readiness-report table verbatim as your report, prefixed by one line: `READY` (all services green), `DEGRADED: <which service, why>` (something is up but failed its smoke test after the clean-restart loop), or `needs-input: <decision>`. The dispatcher relays this to its caller — other skills gate on it, so a wrong `READY` is expensive; when in doubt between READY and DEGRADED, pick DEGRADED and say why. `needs-input:` is only for a decision the plan failed to resolve (a missing URL, an unapproved start, an unknown directory) — a service that stays broken after the loop is `DEGRADED` with the diagnosis, never `needs-input:`, even when no further automated fix exists. Repairing the app's own source code is out of scope entirely: a source-level bug surfaced by the smoke test is a `DEGRADED` diagnosis to report, not a decision to escalate.
|
|
41
|
+
|
|
42
|
+
## Guardrails
|
|
43
|
+
|
|
44
|
+
- Privacy boundary as the skill defines it: file contents only inside directories the plan names; never traverse upward past one level.
|
|
45
|
+
- Port-listening is never enough — smoke-test (HTTP + body sniff + log tail) is mandatory before the report.
|
|
46
|
+
- Clean Restart is the first fix in the smoke-test loop; fresh-install notifies, doesn't ask.
|
|
47
|
+
- Never leave an orphan process untracked; never kill an `external` one.
|
|
@@ -0,0 +1,58 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: visual-walkthrough-builder
|
|
3
|
+
description: "Renders the Muggle Test E2E visual walkthrough for a PR — assembles the E2eReport, runs `muggle build-pr-section`, and either posts to the PR (Mode A) or returns the rendered block to the dispatcher (Modes B/C). Dispatched by the muggle-pr-visual-walkthrough skill; carries its sonnet pin so the render runs on sonnet regardless of the session model."
|
|
4
|
+
model: sonnet
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
# Visual Walkthrough Builder
|
|
8
|
+
|
|
9
|
+
You render and (in Mode A) post the Muggle Test E2E visual walkthrough. The dispatching skill has already resolved the mode, the PR, and user consent. You have no channel to the user: if an input you need is missing, return a single `needs-input:` line naming it and stop — the dispatching skill resolves it and re-dispatches.
|
|
10
|
+
|
|
11
|
+
## Input contract
|
|
12
|
+
|
|
13
|
+
The dispatch prompt carries:
|
|
14
|
+
|
|
15
|
+
- `mode` — `post` (Mode A), `render-for-new-pr` (Mode B), or `embed` (Mode C).
|
|
16
|
+
- `prNumber` + repo — Mode A only, already verified to exist.
|
|
17
|
+
- The `E2eReport` JSON inline, **or** the run identifiers (`projectId`, per-test `runId`/`testCaseId` list) to assemble it from.
|
|
18
|
+
|
|
19
|
+
When assembling from identifiers, follow [`../skills/muggle-pr-visual-walkthrough/e2e-report-assembly.md`](../skills/muggle-pr-visual-walkthrough/e2e-report-assembly.md). The `E2eReport` schema, required fields, and the inconclusive rule live there and in the CLI's Zod schema (`src/cli/pr-section/types.ts`) — a run that couldn't produce pass/fail is `inconclusive` with a `reason`, never dropped.
|
|
20
|
+
|
|
21
|
+
## Render
|
|
22
|
+
|
|
23
|
+
Pipe the report to the CLI; it writes `{"body": "...", "comment": "..." | null}`:
|
|
24
|
+
|
|
25
|
+
```bash
|
|
26
|
+
echo "$REPORT_JSON" | muggle build-pr-section > /tmp/muggle-pr-section.json
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
- Non-zero exit → surface the CLI's stderr verbatim; do not swallow or retry blindly.
|
|
30
|
+
- `comment` is non-null only in the overflow case; the CLI owns fit-vs-overflow.
|
|
31
|
+
|
|
32
|
+
## Deliver
|
|
33
|
+
|
|
34
|
+
**Mode A (`post`)** — post `body` as a PR comment, then `comment` only if non-null. Append the Muggle Works signature to each posted body per [`../skills/_shared/vcs/post-signature.md`](../skills/_shared/vcs/post-signature.md) — this post is the walkthrough's own, so the command it names is `/muggle-pr-visual-walkthrough`:
|
|
35
|
+
|
|
36
|
+
```bash
|
|
37
|
+
sig='🤖 _Posted by `/muggle-pr-visual-walkthrough` · [Muggle Works](https://github.com/multiplex-ai/muggle-ai-works)_'
|
|
38
|
+
{ jq -r '.body' /tmp/muggle-pr-section.json; printf '\n\n%s\n' "$sig"; } | gh pr comment <prNumber> --body-file -
|
|
39
|
+
{ jq -r '.comment' /tmp/muggle-pr-section.json; printf '\n\n%s\n' "$sig"; } | gh pr comment <prNumber> --body-file - # skip when null
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
Report back: PR URL + whether an overflow comment was posted.
|
|
43
|
+
|
|
44
|
+
**Modes B/C (`render-for-new-pr` / `embed`)** — do not post, do not touch `gh`. Return the CLI output verbatim as your report:
|
|
45
|
+
|
|
46
|
+
```
|
|
47
|
+
body:
|
|
48
|
+
<body>
|
|
49
|
+
comment:
|
|
50
|
+
<comment or null>
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
## Guardrails
|
|
54
|
+
|
|
55
|
+
- Never hand-write or modify the walkthrough markdown — the CLI is the single source of truth. No custom tables, no added "Verdict" lines, no `Tested on:`/`Project:` footers; the CLI computes the verdict and emits per-test dashboard links.
|
|
56
|
+
- Never invent report fields — missing `projectId`, `viewUrl`, or `screenshotUrl` → `needs-input:`, never a placeholder.
|
|
57
|
+
- Never post the overflow comment when `comment` is null.
|
|
58
|
+
- Never create a PR, never choose a mode — both belong to the dispatcher.
|
|
@@ -1,36 +1,39 @@
|
|
|
1
|
-
# Skill authoring conventions
|
|
2
|
-
|
|
3
|
-
Rules for every skill under `plugin/skills/`. Read before adding or editing one.
|
|
4
|
-
|
|
5
|
-
## One-way dependencies — no reverse references
|
|
6
|
-
|
|
7
|
-
Skill cross-references form a one-way graph. If any file in skill **A** references skill **B** — a markdown link to B's files, or a documented dependency on B's internals — then **no file in B may reference A back**. Reference *downward*, toward the more general / lower-level skill you depend on; pass anything the other direction needs as input, not as a link.
|
|
8
|
-
|
|
9
|
-
A reverse reference (A → B and B → A) couples the depended-on skill to its caller, creates a cycle no one can reason about in isolation, and makes every edit ripple both ways. The lower-level skill must stay reusable by callers it has never heard of.
|
|
10
|
-
|
|
11
|
-
**Runtime dispatch is not a doc reference.** A dumb-pipe skill may *fire* another skill's slash command at runtime (hand off and forget) — that is an action, not a dependency. What the rule forbids is a procedure file **linking to** or **encoding the internals of** the skill it hands off to.
|
|
12
|
-
|
|
13
|
-
**Worked example.** `muggle-pr-followup` (the dumb-pipe watcher) is lower-level than `muggle-do` (the executor that orchestrates it). `muggle-do` references `muggle-pr-followup`; `muggle-pr-followup`'s files must not link back to `muggle-do`. A watcher tick still dispatches `/muggle-do …` at runtime — allowed — but no watcher file links a `do/` file or restates its steps, and shared primitives like `muggle-pr-followup/finalize.md` stay dispatch-free so any caller can reuse them.
|
|
14
|
-
|
|
15
|
-
When you feel the urge to link "up" to a caller, that is the smell — restructure so the caller passes what is needed in.
|
|
16
|
-
|
|
17
|
-
### Enforcement
|
|
18
|
-
|
|
19
|
-
`scripts/check-skill-deps.mjs` derives the cross-skill link graph and fails on any cycle. A "reference" is a markdown file-link into another skill's directory — runtime slash-command dispatch is not a link and is not counted. It runs three ways: the `skill-deps` CI job on every PR, a `PreToolUse` hook (`.claude/settings.json`) that blocks the write mid-session with the offending link named, and `pnpm run verify:skill-deps` locally.
|
|
20
|
-
|
|
21
|
-
`plugin/skills/skill-deps.config.json` declares support dirs grouped into their owning skill (`do/` → `muggle-do`), shared namespaces exploded to per-file nodes (`_shared`), and `knownReverseDeps` — pre-existing violations grandfathered so CI stays green. That list is debt: fix each link and delete its entry. A new reverse dependency is blocked whether or not it is on the list.
|
|
22
|
-
|
|
23
|
-
## Model tiers
|
|
24
|
-
|
|
25
|
-
Each skill sets a `model:` in its `SKILL.md` frontmatter sized to its cognitive load. `model:` is a native Claude Code field — the override applies while the skill is active and reverts to the session model when it exits. Cheaper, faster models run the mechanical skills; the default (Opus) is reserved for the ones that actually reason. Cost and latency scale with the model, and these skills run often (the watcher fires every minute), so the tier is a real lever, not cosmetics.
|
|
26
|
-
|
|
27
|
-
| Model | Skills | Why this tier |
|
|
28
|
-
|-------|--------|---------------|
|
|
29
|
-
| `haiku` | `muggle`, `muggle-status`, `muggle-repair`, `muggle-upgrade`, `muggle-preferences`, `muggle-feedback`, `muggle-pr-followup` | Routers and dumb pipes. They follow an explicit procedure with no open-ended reasoning: route intent to a downstream skill, run a fixed CLI sequence, CRUD a config file, format a status report, or poll provider state and branch on conditions. `muggle-pr-followup` is the canonical case — a watcher that reads GitHub state and dispatches; all judgment lives in the `muggle-do` it hands off to. |
|
|
30
|
-
| `sonnet` | `muggle-pr-visual-walkthrough
|
|
31
|
-
| `opus` (explicit pin) | `muggle-test-prepare` | Pinned for **reliability**, not raw reasoning load: it's flaky on smaller models, and since other skills gate on the environment it readies, a wrong call is expensive. Pin explicitly rather than leaving `model:` unset so it stays on Opus even when the user's session runs a cheaper model. |
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
1
|
+
# Skill authoring conventions
|
|
2
|
+
|
|
3
|
+
Rules for every skill under `plugin/skills/`. Read before adding or editing one.
|
|
4
|
+
|
|
5
|
+
## One-way dependencies — no reverse references
|
|
6
|
+
|
|
7
|
+
Skill cross-references form a one-way graph. If any file in skill **A** references skill **B** — a markdown link to B's files, or a documented dependency on B's internals — then **no file in B may reference A back**. Reference *downward*, toward the more general / lower-level skill you depend on; pass anything the other direction needs as input, not as a link.
|
|
8
|
+
|
|
9
|
+
A reverse reference (A → B and B → A) couples the depended-on skill to its caller, creates a cycle no one can reason about in isolation, and makes every edit ripple both ways. The lower-level skill must stay reusable by callers it has never heard of.
|
|
10
|
+
|
|
11
|
+
**Runtime dispatch is not a doc reference.** A dumb-pipe skill may *fire* another skill's slash command at runtime (hand off and forget) — that is an action, not a dependency. What the rule forbids is a procedure file **linking to** or **encoding the internals of** the skill it hands off to.
|
|
12
|
+
|
|
13
|
+
**Worked example.** `muggle-pr-followup` (the dumb-pipe watcher) is lower-level than `muggle-do` (the executor that orchestrates it). `muggle-do` references `muggle-pr-followup`; `muggle-pr-followup`'s files must not link back to `muggle-do`. A watcher tick still dispatches `/muggle-do …` at runtime — allowed — but no watcher file links a `do/` file or restates its steps, and shared primitives like `muggle-pr-followup/finalize.md` stay dispatch-free so any caller can reuse them.
|
|
14
|
+
|
|
15
|
+
When you feel the urge to link "up" to a caller, that is the smell — restructure so the caller passes what is needed in.
|
|
16
|
+
|
|
17
|
+
### Enforcement
|
|
18
|
+
|
|
19
|
+
`scripts/check-skill-deps.mjs` derives the cross-skill link graph and fails on any cycle. A "reference" is a markdown file-link into another skill's directory — runtime slash-command dispatch is not a link and is not counted. It runs three ways: the `skill-deps` CI job on every PR, a `PreToolUse` hook (`.claude/settings.json`) that blocks the write mid-session with the offending link named, and `pnpm run verify:skill-deps` locally.
|
|
20
|
+
|
|
21
|
+
`plugin/skills/skill-deps.config.json` declares support dirs grouped into their owning skill (`do/` → `muggle-do`), shared namespaces exploded to per-file nodes (`_shared`), and `knownReverseDeps` — pre-existing violations grandfathered so CI stays green. That list is debt: fix each link and delete its entry. A new reverse dependency is blocked whether or not it is on the list.
|
|
22
|
+
|
|
23
|
+
## Model tiers
|
|
24
|
+
|
|
25
|
+
Each skill sets a `model:` in its `SKILL.md` frontmatter sized to its cognitive load. `model:` is a native Claude Code field — the override applies while the skill is active and reverts to the session model when it exits. Cheaper, faster models run the mechanical skills; the default (Opus) is reserved for the ones that actually reason. Cost and latency scale with the model, and these skills run often (the watcher fires every minute), so the tier is a real lever, not cosmetics.
|
|
26
|
+
|
|
27
|
+
| Model | Skills | Why this tier |
|
|
28
|
+
|-------|--------|---------------|
|
|
29
|
+
| `haiku` | `muggle`, `muggle-status`, `muggle-repair`, `muggle-upgrade`, `muggle-preferences`, `muggle-feedback`, `muggle-pr-followup` | Routers and dumb pipes. They follow an explicit procedure with no open-ended reasoning: route intent to a downstream skill, run a fixed CLI sequence, CRUD a config file, format a status report, or poll provider state and branch on conditions. `muggle-pr-followup` is the canonical case — a watcher that reads GitHub state and dispatches; all judgment lives in the `muggle-do` it hands off to. |
|
|
30
|
+
| `sonnet` | `muggle-pr-visual-walkthrough` (executes via the `visual-walkthrough-builder` agent), `muggle-test-regenerate-missing` | Multi-step orchestration with light judgment, short of deep reasoning: assemble run data and build a PR section with fit-vs-overflow handling; scan, filter, bulk-dispatch, and classify per-item failures into buckets. More moving parts than a router, but each step is well-defined. |
|
|
31
|
+
| `opus` (explicit pin) | `muggle-test-prepare` (executes via the `test-prepare-runner` agent) | Pinned for **reliability**, not raw reasoning load: it's flaky on smaller models, and since other skills gate on the environment it readies, a wrong call is expensive. Pin explicitly rather than leaving `model:` unset so it stays on Opus even when the user's session runs a cheaper model. |
|
|
32
|
+
|
|
33
|
+
| default (Opus) — no `model:` set | `muggle-do`, `muggle-test`, `muggle-test-feature-local`, `muggle-browser-task`, `muggle-test-import` | Reasoning-heavy. Authoring code to a PR, mapping a code diff to affected user flows and interpreting E2E results, reasoning about an arbitrary website's flow to drive a browser, translating Playwright/Cypress/PRD artifacts into Muggle test cases. Leave `model:` unset so the skill inherits the session model. |
|
|
34
|
+
|
|
35
|
+
**A `SKILL.md` `model:` only bites when the skill starts a fresh session; invoked mid-session the session model keeps running.** A skill whose pin must hold regardless resolves its user interaction in the `SKILL.md`, then dispatches a `plugin/agents/*` agent that carries the pin — the harness applies an agent's `model:` on every dispatch. That is why `muggle-pr-visual-walkthrough`, `muggle-test-prepare`, and `muggle-pr-followup` execute through agents. Agents have no user channel; an unresolved decision comes back as `needs-input:` for the skill to resolve.
|
|
36
|
+
|
|
37
|
+
**Choosing a tier for a new skill.** Ask what the skill actually does. Pure routing / fixed procedure / CRUD / reporting → `haiku`. Several well-defined steps with some judgment or classification → `sonnet`. Open-ended reasoning, code authoring, or interpreting ambiguous real-world state → leave `model:` unset (Opus). When unsure between two tiers, pick the cheaper one and watch for misbehavior — the likeliest to need a bump is anything doing AI-based classification. If a skill proves flaky on its tier and reliability matters more than cost (other skills depend on it, or a wrong call is expensive), pin it up explicitly — `model: opus` — rather than leaving it unset, so the floor holds regardless of the user's session model.
|
|
38
|
+
|
|
39
|
+
**Never set `model:` on aliases or commands.** The alias skills (`m`, `mstatus`, …) and `plugin/commands/*.md` are thin routers that re-invoke the canonical skill via the `Skill` tool. The canonical `SKILL.md`'s `model:` takes effect once it loads, so a model on the alias would only apply to the negligible one-line hand-off — and risks drifting from the canonical value.
|
|
@@ -1,236 +1,41 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: muggle-pr-visual-walkthrough
|
|
3
|
-
model: sonnet
|
|
4
|
-
description: Renders and posts a visual walkthrough of Muggle AI E2E acceptance test results to a PR — per-test-case dashboard links, step-by-step screenshots, and pass/fail summary — using the `muggle build-pr-section` CLI for deterministic formatting with automatic fit-vs-overflow. Use at the end of any Muggle Test test run (local or remote) to give PR reviewers clickable visual evidence that user flows work. Triggers on 'post results to PR', 'attach walkthrough to PR', 'share E2E screenshots on the PR', 'add visual walkthrough to PR'.
|
|
5
|
-
---
|
|
6
|
-
|
|
7
|
-
# Muggle Test PR Visual Walkthrough
|
|
8
|
-
|
|
9
|
-
> Telemetry first step: see [`_shared/telemetry-emit.md`](../_shared/telemetry-emit.md). Use `skillName: "muggle-pr-visual-walkthrough"`.
|
|
10
|
-
|
|
11
|
-
Renders a visual walkthrough of Muggle AI E2E acceptance test results and posts it to a PR. Each test case
|
|
12
|
-
|
|
13
|
-
This is
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
|
18
|
-
|
|
|
19
|
-
| `muggle-
|
|
20
|
-
| `muggle-test
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
## Preferences
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
{
|
|
43
|
-
"name": "<test case title>",
|
|
44
|
-
"description": "<one-line description of what this test verifies>",
|
|
45
|
-
"useCaseName": "<parent use case title>",
|
|
46
|
-
"testCaseId": "<UUID>",
|
|
47
|
-
"testScriptId": "<UUID>",
|
|
48
|
-
"runId": "<UUID>",
|
|
49
|
-
"viewUrl": "https://www.muggle-ai.com/...",
|
|
50
|
-
"status": "passed",
|
|
51
|
-
"steps": [
|
|
52
|
-
{ "stepIndex": 0, "action": "Click login button", "screenshotUrl": "https://..." },
|
|
53
|
-
{ "stepIndex": 1, "action": "Type email", "screenshotUrl": "https://..." }
|
|
54
|
-
]
|
|
55
|
-
},
|
|
56
|
-
{
|
|
57
|
-
"name": "Checkout flow",
|
|
58
|
-
"description": "Verify a shopper can complete checkout with a saved card.",
|
|
59
|
-
"useCaseName": "Purchase",
|
|
60
|
-
"testCaseId": "<UUID>",
|
|
61
|
-
"testScriptId": "<UUID>",
|
|
62
|
-
"runId": "<UUID>",
|
|
63
|
-
"viewUrl": "https://www.muggle-ai.com/...",
|
|
64
|
-
"status": "failed",
|
|
65
|
-
"steps": [
|
|
66
|
-
{ "stepIndex": 0, "action": "Open cart", "screenshotUrl": "https://..." }
|
|
67
|
-
],
|
|
68
|
-
"failureStepIndex": 2,
|
|
69
|
-
"error": "Element not found: Click checkout button",
|
|
70
|
-
"artifactsDir": "/Users/.../~/.muggle-ai/sessions/<runId>"
|
|
71
|
-
},
|
|
72
|
-
{
|
|
73
|
-
"name": "Clear search input restores full list",
|
|
74
|
-
"description": "Verify clearing the search input restores all options.",
|
|
75
|
-
"useCaseName": "Filter Dropdowns",
|
|
76
|
-
"testCaseId": "<UUID>",
|
|
77
|
-
"runId": "<UUID>",
|
|
78
|
-
"viewUrl": "https://www.muggle-ai.com/...",
|
|
79
|
-
"status": "inconclusive",
|
|
80
|
-
"steps": [],
|
|
81
|
-
"reason": "No replayable script exists yet — needs first generation run."
|
|
82
|
-
}
|
|
83
|
-
]
|
|
84
|
-
}
|
|
85
|
-
```
|
|
86
|
-
|
|
87
|
-
Required fields per test: `name`, `testCaseId`, `runId`, `viewUrl`, `status`, `steps[]` with `{stepIndex, action, screenshotUrl}`.
|
|
88
|
-
|
|
89
|
-
- **Failed** tests additionally require `failureStepIndex` and `error`.
|
|
90
|
-
- **Inconclusive** tests additionally require `reason` (one short sentence on why the result is neither pass nor fail). `steps[]` may be empty — that's fine. Inconclusive is for runs that couldn't be classified pass/fail (no replayable script, environment precondition unmet, infra error blocked execution, agent stalled before reaching the assertion). **Never silently drop these — always emit them as `inconclusive`.** The CLI counts them in the overview and renders an `⚠️` row with the dashboard link so reviewers can click through. If you find yourself wanting to skip a test or hand-write a comment because the schema "doesn't fit," that is the schema fitting — use `inconclusive`.
|
|
91
|
-
|
|
92
|
-
### Verdict
|
|
93
|
-
|
|
94
|
-
The renderer computes a verdict from the tests and emits a `**Verdict:** ✅ PASS | ❌ FAIL | ⚠️ INCONCLUSIVE` line at the top of the overview. The policy is strict:
|
|
95
|
-
|
|
96
|
-
- Any failed test → **FAIL** (regardless of how many passed or are inconclusive).
|
|
97
|
-
- No failures but any inconclusive → **INCONCLUSIVE**.
|
|
98
|
-
- All passed → **PASS**.
|
|
99
|
-
- Empty report → no verdict line.
|
|
100
|
-
|
|
101
|
-
You do not compute or render the verdict yourself — the CLI does. Never write a "Verdict: PASS" line into a hand-edited summary; it will conflict with the CLI's computed verdict.
|
|
102
|
-
|
|
103
|
-
**Optional but recommended** per test:
|
|
104
|
-
- `description` — a one-line summary of what the test case verifies. Shown in the collapsible header for each test and helps reviewers understand the test without expanding it. Pull from the test case's `title`/`description` via `muggle-remote-test-case-get`.
|
|
105
|
-
- `useCaseName` — the parent use case title. When present on any test, the overview list is grouped by use case; otherwise it is rendered as a flat list. Pull from `muggle-remote-use-case-get` using the test case's parent use-case id.
|
|
106
|
-
- `testScriptId` and `artifactsDir` are also optional.
|
|
107
|
-
|
|
108
|
-
If any required field is missing, stop and tell the caller exactly what's missing. Never fabricate data.
|
|
109
|
-
|
|
110
|
-
## Step 1: Assemble the `E2eReport`
|
|
111
|
-
|
|
112
|
-
Read `plugin/skills/muggle-pr-visual-walkthrough/e2e-report-assembly.md` for the full assembly guide.
|
|
113
|
-
|
|
114
|
-
## Step 2: Render via `muggle build-pr-section`
|
|
115
|
-
|
|
116
|
-
Pipe the `E2eReport` JSON to the CLI. It writes `{"body": "...", "comment": "..." | null}` to stdout — the `body` is the E2E markdown block, and `comment` is a non-null overflow comment only when the full body exceeds the byte budget (default 60 KB).
|
|
117
|
-
|
|
118
|
-
```bash
|
|
119
|
-
echo "$REPORT_JSON" | muggle build-pr-section > /tmp/muggle-pr-section.json
|
|
120
|
-
```
|
|
121
|
-
|
|
122
|
-
- Exit **non-zero** → the CLI wrote a descriptive error to stderr. Surface it to the user; do not swallow it.
|
|
123
|
-
- `comment` is **`null`** (fit case) → everything is inline in `body`. Post `body` only.
|
|
124
|
-
- `comment` is a **non-null string** (overflow case) → `body` contains the summary + a pointer; `comment` contains the full per-step details. Post both, in order.
|
|
125
|
-
|
|
126
|
-
Never hand-write the walkthrough markdown. Never modify the CLI's output before posting. The CLI owns the format.
|
|
127
|
-
|
|
128
|
-
## Step 3 — Mode A: Post to an existing PR
|
|
129
|
-
|
|
130
|
-
Used by `muggle-test` and `muggle-test-feature-local`, where the user is mid-development and a PR already exists on the current branch.
|
|
131
|
-
|
|
132
|
-
### 3A.1: Find the PR
|
|
133
|
-
|
|
134
|
-
```bash
|
|
135
|
-
gh pr view --json number,url,title 2>/dev/null
|
|
136
|
-
```
|
|
137
|
-
|
|
138
|
-
- **PR exists** → continue to 3A.2
|
|
139
|
-
- **No PR exists** → use `AskUserQuestion`:
|
|
140
|
-
- "Create a new PR with the visual walkthrough in the body"
|
|
141
|
-
- "Skip posting"
|
|
142
|
-
- If the user chooses to create a new PR, switch to Mode B and return the rendered `body`/`comment` to the caller for embedding in `gh pr create`. Do not create the PR directly from this skill unless the caller has no better way to do it.
|
|
143
|
-
- **`gh` not installed or not authenticated** → tell the user, suggest `gh auth login`, stop.
|
|
144
|
-
|
|
145
|
-
### 3A.2: Post the body as a PR comment
|
|
146
|
-
|
|
147
|
-
Extract the `body` field with `jq -r` (not `sed`) so JSON escape sequences are properly decoded, append the Muggle Works signature (command `/muggle-pr-visual-walkthrough`) per [`../_shared/vcs/post-signature.md`](../_shared/vcs/post-signature.md), then pipe to `--body-file -`. The renderer's sentinel stays at the top of `body`, so the report-format guardrail still recognises the post:
|
|
148
|
-
|
|
149
|
-
```bash
|
|
150
|
-
{
|
|
151
|
-
jq -r '.body' /tmp/muggle-pr-section.json
|
|
152
|
-
printf '\n\n%s\n' '🤖 _Posted by `/muggle-pr-visual-walkthrough` · [Muggle Works](https://github.com/multiplex-ai/muggle-ai-works)_'
|
|
153
|
-
} | gh pr comment <pr-number> --body-file -
|
|
154
|
-
```
|
|
155
|
-
|
|
156
|
-
### 3A.3: Post the overflow comment only if the CLI emitted one
|
|
157
|
-
|
|
158
|
-
```bash
|
|
159
|
-
{
|
|
160
|
-
jq -r '.comment' /tmp/muggle-pr-section.json
|
|
161
|
-
printf '\n\n%s\n' '🤖 _Posted by `/muggle-pr-visual-walkthrough` · [Muggle Works](https://github.com/multiplex-ai/muggle-ai-works)_'
|
|
162
|
-
} | gh pr comment <pr-number> --body-file -
|
|
163
|
-
```
|
|
164
|
-
|
|
165
|
-
**Skip this step entirely if `comment` is `null`** — do not post a placeholder. The CLI decides fit-vs-overflow; never post the overflow comment speculatively.
|
|
166
|
-
|
|
167
|
-
### 3A.4: Confirm to the user
|
|
168
|
-
|
|
169
|
-
> "Visual walkthrough posted to PR #<number>. Reviewers can click any test case link to see the step-by-step screenshots on the Muggle AI dashboard."
|
|
170
|
-
|
|
171
|
-
Include the PR URL in the confirmation.
|
|
172
|
-
|
|
173
|
-
## Step 3 — Mode B: Return rendered block for embedding in a new PR
|
|
174
|
-
|
|
175
|
-
Used by `muggle-do`'s `open-prs.md`, where the PR does not exist yet and the caller is assembling the PR body from multiple sections (`## Goal`, `## Acceptance Criteria`, `## Changes`, plus this walkthrough).
|
|
176
|
-
|
|
177
|
-
Instead of posting, **return** the CLI output to the caller's context so they can:
|
|
178
|
-
|
|
179
|
-
1. **Embed `body`** in their PR body, concatenated after `## Changes`. `body` already includes its own `## E2E Acceptance Results` header — do not add another.
|
|
180
|
-
2. **Create the PR** with `gh pr create --title "..." --body "..."` using the concatenated body.
|
|
181
|
-
3. **Post `comment` as a follow-up only if the CLI emitted one**, ending the posted body with the signature (the caller owns this post, so command `/muggle-do`) per [`../_shared/vcs/post-signature.md`](../_shared/vcs/post-signature.md):
|
|
182
|
-
|
|
183
|
-
```bash
|
|
184
|
-
{
|
|
185
|
-
jq -r '.comment' /tmp/muggle-pr-section.json
|
|
186
|
-
printf '\n\n%s\n' '🤖 _Posted by `/muggle-do` · [Muggle Works](https://github.com/multiplex-ai/muggle-ai-works)_'
|
|
187
|
-
} | gh pr comment <new-pr-number> --body-file -
|
|
188
|
-
```
|
|
189
|
-
|
|
190
|
-
Skip if `comment` is `null`.
|
|
191
|
-
|
|
192
|
-
In Mode B, this skill does not call `gh pr comment` or `gh pr create` itself — the caller owns PR creation because it also owns branch pushing, title building (including `[E2E FAILING]` prefix on failures), and multi-repo orchestration.
|
|
193
|
-
|
|
194
|
-
## Step 3 — Mode C: Embed mode for open-PR sweep orchestrators
|
|
195
|
-
|
|
196
|
-
Used when invoked as a sub-step of a open-PR sweep orchestrator (see `plugin/skills/muggle-test/SKILL.md` Mode C) rather than as a top-level user invocation. The orchestrator's `acceptance-tester` subagent (see `plugin/agents/acceptance-tester.md`) composes a **single per-PR verdict comment** and folds the walkthrough into it — posting separately would create 2–3 disparate comments per test cycle and clutter the PR.
|
|
197
|
-
|
|
198
|
-
### 3C.1: Detect embed mode
|
|
199
|
-
|
|
200
|
-
The caller passes `mode: "embed"` as a skill argument. Default behavior (no `mode` passed, or `mode: "post"`) is Mode A — unchanged.
|
|
201
|
-
|
|
202
|
-
### 3C.2: Render but do not post
|
|
203
|
-
|
|
204
|
-
Run `muggle build-pr-section` exactly as in Step 2. Then, **instead of calling `gh pr comment`**:
|
|
205
|
-
|
|
206
|
-
1. **Return** the CLI output (`{ body, comment }`) to the caller as this skill's result.
|
|
207
|
-
2. Do **not** find a PR, do **not** post a standalone comment, do **not** prompt the user.
|
|
208
|
-
3. If `comment` is non-null (overflow case), return it alongside `body` — the orchestrator decides how to handle overflow (typically: inline `body` in the verdict comment, post `comment` as a follow-up).
|
|
209
|
-
|
|
210
|
-
### 3C.3: Hand off
|
|
211
|
-
|
|
212
|
-
The orchestrator (`acceptance-tester`) concatenates the returned `body` into its verdict comment template alongside the verdict summary, change-list, and any other sections it owns, then posts a single `gh pr comment` itself. This skill's job ends at returning the rendered markdown.
|
|
213
|
-
|
|
214
|
-
In Mode C, the same fit-vs-overflow contract from Step 2 applies — never modify the CLI's output, never fabricate fields, never post anything. The caller owns posting.
|
|
215
|
-
|
|
216
|
-
## Tool Reference
|
|
217
|
-
|
|
218
|
-
| Phase | Tool |
|
|
219
|
-
|:------|:-----|
|
|
220
|
-
| Gather per-step data (muggle-test, muggle-test-feature-local) | `muggle-remote-test-script-get` |
|
|
221
|
-
| Render the walkthrough markdown | `muggle build-pr-section` (shell) |
|
|
222
|
-
| Find existing PR (Mode A) | `gh pr view` |
|
|
223
|
-
| Post comment(s) (Mode A) | `gh pr comment` |
|
|
224
|
-
| Create new PR (Mode B, caller handles) | `gh pr create` |
|
|
225
|
-
| User confirmation (Mode A no-PR branch) | `AskUserQuestion` |
|
|
226
|
-
|
|
227
|
-
## Guardrails
|
|
228
|
-
|
|
229
|
-
- **Never hand-write the walkthrough markdown** — always call `muggle build-pr-section`. The CLI is the single source of truth for formatting. If a test outcome doesn't fit `passed | failed`, that's what `inconclusive` is for — never fall back to a hand-written summary, a custom table, a "Verdict: PASS" header, a `Tested on:`/`Project:` footer, or any other freeform text. The CLI already emits per-test-case dashboard links (no project-level link), uses `https://www.muggle-ai.com/...` URLs (never the test target's `localhost` URL), and computes the verdict — anything you would manually add is wrong by construction.
|
|
230
|
-
- **Never modify the CLI's output** — post `body` and (if present) `comment` verbatim. Any reformatting defeats the fit-vs-overflow budget math.
|
|
231
|
-
- **Never invent report fields** — if `projectId`, a per-test `viewUrl`, or per-step `screenshotUrl` is missing, stop and report what's missing. Do not fabricate URLs or fill in placeholders.
|
|
232
|
-
- **Never post the overflow comment when `comment` is `null`** — the CLI decides fit-vs-overflow.
|
|
233
|
-
- **Never create a PR without confirmation in Mode A** — if no PR exists, ask the user or switch to Mode B and hand back to the caller.
|
|
234
|
-
- **Don't run tests** — this skill only renders and posts existing results. If the `E2eReport` is not in context, redirect the caller to `muggle-test`, `muggle-test-feature-local`, or `muggle-do`.
|
|
235
|
-
- **Mode is chosen by the caller, not the user** — `muggle-test` top-level uses Mode A; `muggle-do` uses Mode B; open-PR sweep orchestrators (`muggle-test` Mode C / `acceptance-tester`) pass `mode: "embed"` for Mode C. Don't ask the user which mode to use.
|
|
236
|
-
- **Never post in Mode C** — when `mode: "embed"` is passed, return the rendered body to the caller and stop. The orchestrator owns the single verdict comment.
|
|
1
|
+
---
|
|
2
|
+
name: muggle-pr-visual-walkthrough
|
|
3
|
+
model: sonnet
|
|
4
|
+
description: Renders and posts a visual walkthrough of Muggle AI E2E acceptance test results to a PR — per-test-case dashboard links, step-by-step screenshots, and pass/fail summary — using the `muggle build-pr-section` CLI for deterministic formatting with automatic fit-vs-overflow. Use at the end of any Muggle Test test run (local or remote) to give PR reviewers clickable visual evidence that user flows work. Triggers on 'post results to PR', 'attach walkthrough to PR', 'share E2E screenshots on the PR', 'add visual walkthrough to PR'.
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
# Muggle Test PR Visual Walkthrough
|
|
8
|
+
|
|
9
|
+
> Telemetry first step: see [`_shared/telemetry-emit.md`](../_shared/telemetry-emit.md). Use `skillName: "muggle-pr-visual-walkthrough"`.
|
|
10
|
+
|
|
11
|
+
Renders a visual walkthrough of Muggle AI E2E acceptance test results and posts it to a PR. Each test case links to its detail page on the Muggle AI dashboard, so reviewers can click through to step-by-step screenshots — not just a pass/fail flag.
|
|
12
|
+
|
|
13
|
+
This skill is a **dispatcher**: it resolves the mode and any user interaction, then hands execution to the `visual-walkthrough-builder` agent (`plugin/agents/visual-walkthrough-builder.md`), which renders and delivers per its own contract.
|
|
14
|
+
|
|
15
|
+
This is the **canonical PR-walkthrough workflow** shared across every Muggle Test entry point:
|
|
16
|
+
|
|
17
|
+
| Caller | Mode | When to invoke |
|
|
18
|
+
| :--- | :--- | :--- |
|
|
19
|
+
| `muggle-test` | **Mode A** (post to existing PR) | After publishing results, user opts in via `AskUserQuestion` |
|
|
20
|
+
| `muggle-test-feature-local` | **Mode A** (post to existing PR) | After publishing the run, user opts in via `AskUserQuestion` |
|
|
21
|
+
| `muggle-do` / `open-prs.md` | **Mode B** (render-only for embedding) | During PR creation — caller embeds `body` in the PR create call and posts `comment` as follow-up |
|
|
22
|
+
| `muggle-test` Mode C / `acceptance-tester` agent | **Mode C** (embed in verdict comment) | Inside an open-PR sweep orchestrator — caller folds the rendered body into a single per-PR verdict comment |
|
|
23
|
+
|
|
24
|
+
## Preferences
|
|
25
|
+
|
|
26
|
+
Callers consult the `postPRVisualWalkthrough` gate **before** invoking this skill — by the time it runs, posting is already approved. Per-key gate definitions live in `plugin/skills/muggle-preferences/preference-gates/`.
|
|
27
|
+
|
|
28
|
+
## Procedure
|
|
29
|
+
|
|
30
|
+
1. **Resolve the mode.** Chosen by the caller, never the user: top-level `muggle-test`/`muggle-test-feature-local` → `post` (Mode A); `muggle-do` PR creation → `render-for-new-pr` (Mode B); an orchestrator passing `mode: "embed"` → Mode C.
|
|
31
|
+
2. **Mode A only — find the PR** with `gh pr view --json number,url,title`. No PR on the branch → `AskUserQuestion`: create a new PR with the walkthrough in the body (switch to Mode B and hand the rendered block back to the caller), or skip posting. `gh` missing/unauthenticated → tell the user, suggest `gh auth login`, stop. This is the skill's only interactive branch — resolve it **before** dispatching.
|
|
32
|
+
3. **Gather the inputs.** The `E2eReport` JSON if the caller already assembled it (see [`e2e-report-assembly.md`](e2e-report-assembly.md)), else the run identifiers (`projectId`, per-test `runId`/`testCaseId`) the agent needs to assemble it.
|
|
33
|
+
4. **Dispatch** the `visual-walkthrough-builder` agent (subagent type `muggle:visual-walkthrough-builder`; bare `visual-walkthrough-builder` where the plugin namespace is absent), synchronously, passing: mode, PR number + repo (Mode A), and the report JSON or identifiers. In a harness with no agent/subagent facility, execute `plugin/agents/visual-walkthrough-builder.md` inline instead.
|
|
34
|
+
5. **Relay the result.** Mode A → confirm to the user with the PR URL. Modes B/C → return the agent's `{body, comment}` to the caller verbatim; the caller owns PR creation (Mode B) or the single verdict comment (Mode C). A `needs-input:` line from the agent names a missing report field — surface it to the caller; never fabricate the field.
|
|
35
|
+
|
|
36
|
+
## Guardrails
|
|
37
|
+
|
|
38
|
+
- The rendered markdown is CLI-owned end to end — this skill never writes, edits, or post-processes it; those rules ride with the agent.
|
|
39
|
+
- Mode is chosen by the caller, not the user.
|
|
40
|
+
- Never create a PR without confirmation in Mode A.
|
|
41
|
+
- Don't run tests — this skill only dispatches rendering/posting of existing results. No report and no run identifiers in context → redirect the caller to `muggle-test`, `muggle-test-feature-local`, or `muggle-do`.
|