@muggleai/works 5.8.0 → 5.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/plugin/.claude-plugin/plugin.json +1 -1
- package/dist/plugin/.cursor-plugin/plugin.json +1 -1
- package/dist/plugin/agents/test-prepare-runner.md +47 -0
- package/dist/plugin/agents/visual-walkthrough-builder.md +58 -0
- package/dist/plugin/hooks/hooks.json +6 -0
- package/dist/plugin/scripts/guardrail-watch-gate.sh +40 -0
- package/dist/plugin/scripts/guardrails.mjs +84 -2
- package/dist/plugin/scripts/pr-watch-guards.sh +17 -0
- package/dist/plugin/skills/CLAUDE.md +39 -36
- package/dist/plugin/skills/muggle-pr-followup/arm-watcher.md +5 -3
- package/dist/plugin/skills/muggle-pr-followup/blocked-tick.md +2 -2
- package/dist/plugin/skills/muggle-pr-followup/cancel-cron.md +5 -3
- package/dist/plugin/skills/muggle-pr-followup/contract.md +8 -2
- package/dist/plugin/skills/muggle-pr-followup/evals/evals.json +30 -3
- package/dist/plugin/skills/muggle-pr-followup/finalize.md +1 -1
- package/dist/plugin/skills/muggle-pr-followup/state-schemas.md +3 -1
- package/dist/plugin/skills/muggle-pr-visual-walkthrough/SKILL.md +41 -236
- package/dist/plugin/skills/muggle-test-prepare/SKILL.md +130 -129
- package/dist/release-manifest.json +4 -4
- package/package.json +7 -6
- package/plugin/.claude-plugin/plugin.json +1 -1
- package/plugin/.cursor-plugin/plugin.json +1 -1
- package/plugin/agents/test-prepare-runner.md +47 -0
- package/plugin/agents/visual-walkthrough-builder.md +58 -0
- package/plugin/hooks/hooks.json +6 -0
- package/plugin/scripts/guardrail-watch-gate.sh +40 -0
- package/plugin/scripts/guardrails.mjs +84 -2
- package/plugin/scripts/pr-watch-guards.sh +17 -0
- package/plugin/skills/CLAUDE.md +39 -36
- package/plugin/skills/muggle-pr-followup/arm-watcher.md +5 -3
- package/plugin/skills/muggle-pr-followup/blocked-tick.md +2 -2
- package/plugin/skills/muggle-pr-followup/cancel-cron.md +5 -3
- package/plugin/skills/muggle-pr-followup/contract.md +8 -2
- package/plugin/skills/muggle-pr-followup/evals/evals.json +30 -3
- package/plugin/skills/muggle-pr-followup/finalize.md +1 -1
- package/plugin/skills/muggle-pr-followup/state-schemas.md +3 -1
- package/plugin/skills/muggle-pr-visual-walkthrough/SKILL.md +41 -236
- package/plugin/skills/muggle-test-prepare/SKILL.md +130 -129
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"skill_name": "muggle-pr-followup",
|
|
3
|
-
"notes": "These evals test the TICK DECISION behavior plus the recovery/stop substrate. A real tick needs live GitHub state and the Cron tools, so each prompt hands the model a concrete slot + provider state and asks for the step-by-step plan it would execute; assertions check the plan takes the branch the contract mandates. Cases 0–10 cover per-tick dispatch/idle decisions; cases 11–17 cover cron-delivered ticks handing back to the token-free monitor, the owner stop switch, and the stopped-slot / kill-file absorb that makes an unreachable orphaned cron inert. Run via skill-creator's eval runner — CI does not execute these (ci.yml gates on vitest/check-skill-deps; skill-eval.yml runs gate scenarios sourced from muggle-ai-brain and the routing eval).",
|
|
3
|
+
"notes": "These evals test the TICK DECISION behavior plus the recovery/stop substrate. A real tick needs live GitHub state and the Cron tools, so each prompt hands the model a concrete slot + provider state and asks for the step-by-step plan it would execute; assertions check the plan takes the branch the contract mandates. Cases 0–10 cover per-tick dispatch/idle decisions; cases 11–17 cover cron-delivered ticks handing back to the token-free monitor, the owner stop switch, and the stopped-slot / kill-file absorb that makes an unreachable orphaned cron inert; cases 8 and 18–19 cover the monitor's own wake signals (settled-red CI, branch-behind/conflicting, and blocked-resume on any CI-digest move). Run via skill-creator's eval runner — CI does not execute these (ci.yml gates on vitest/check-skill-deps; skill-eval.yml runs gate scenarios sourced from muggle-ai-brain and the routing eval).",
|
|
4
4
|
"evals": [
|
|
5
5
|
{
|
|
6
6
|
"id": 0,
|
|
@@ -88,10 +88,12 @@
|
|
|
88
88
|
{
|
|
89
89
|
"id": 7,
|
|
90
90
|
"eval_name": "repeated-stale-fires-escalate-orphan-once",
|
|
91
|
-
"prompt": "Tick muggle-ai-works-pr507 507. prs.json records state merged and result.md exists. followup.log
|
|
91
|
+
"prompt": "Tick muggle-ai-works-pr507 507. prs.json records state merged and result.md exists. The watcher's session was continued, so neither the recorded cron id nor CronList can reach this slug's cron. followup.log shows finalize recorded cancel=not-found and one earlier stale-tick line also recorded cancel=not-found — there is no cancel=found line anywhere for this slot — and no stale-orphan-escalated line. This fire's defensive cancel per cancel-cron.md again reports not-found. Walk me through the plan for this tick.",
|
|
92
92
|
"files": [],
|
|
93
93
|
"assertions": [
|
|
94
|
-
{ "name": "detects_orphaned_cron", "text": "Plan concludes the cron is orphaned —
|
|
94
|
+
{ "name": "detects_orphaned_cron", "text": "Plan concludes the cron is orphaned because no cancel has ever reached it — finalize's teardown cancel and every stale fire's defensive cancel all report not-found — and the stale fires persist (a prior stale-tick line already exists). It reaches this from the reachability signal, not from counting stale fires to a threshold." },
|
|
95
|
+
{ "name": "requires_persistence_not_a_lone_fire", "text": "Plan escalates because this is not the first stale fire (a prior stale-tick line exists); it would treat a single not-found fire as a possible manually-run tick or just-expired cron, not an orphan." },
|
|
96
|
+
{ "name": "would_not_escalate_a_drained_backlog", "text": "Plan would NOT escalate had any cancel reported found — finalize's teardown cancel or an earlier stale fire's defensive cancel — because a reachable cron's later fires are a finite queued-drain backlog, not an orphan." },
|
|
95
97
|
{ "name": "escalates_to_owner_once", "text": "Plan surfaces a one-line owner notice that only a session restart clears the orphan, and appends a stale-orphan-escalated marker line so the notice never repeats." },
|
|
96
98
|
{ "name": "later_fires_absorb_silently", "text": "Plan treats any stale fire after the marker exists as a silent absorb — log stale-tick and exit, no repeated notice." },
|
|
97
99
|
{ "name": "never_guess_deletes_other_ids", "text": "Plan does NOT delete cron ids that CronList surfaces for other slugs — those belong to other live watchers." }
|
|
@@ -216,6 +218,31 @@
|
|
|
216
218
|
{ "name": "then_seeds_and_arms", "text": "Plan then seeds the slot and arms the watch normally — leaving the kill file in place would make every tick of the new watch absorb itself stillborn." },
|
|
217
219
|
{ "name": "only_bootstrap_clears_it", "text": "Plan notes only bootstrap (an explicit URL) clears the kill file; auto-track and reconcile never do." }
|
|
218
220
|
]
|
|
221
|
+
},
|
|
222
|
+
{
|
|
223
|
+
"id": 18,
|
|
224
|
+
"eval_name": "arm-monitor-wakes-on-branch-behind",
|
|
225
|
+
"prompt": "Arm the watch for muggle-ai-works-pr525 525 after a clean drain (no reviews, no unresolved threads, CI green, branch level with master at seed time). Describe the watch.sh monitor loop you would write and specifically what makes it wake the session when the base branch later advances and the branch falls behind.",
|
|
226
|
+
"files": [],
|
|
227
|
+
"assertions": [
|
|
228
|
+
{ "name": "polls_branch_standing", "text": "The loop polls the branch's standing against its base each iteration (behind_by via the compare call and the conflict signal per branch-standing.md), not just reviews, threads, and CI." },
|
|
229
|
+
{ "name": "wakes_on_behind_or_conflict", "text": "The loop wakes the session when behind_by > 0 or the branch conflicts with its base, so a silently-stale branch triggers a tick without waiting for an unrelated wake." },
|
|
230
|
+
{ "name": "dedups_on_rebase_key_pair", "text": "The loop dedups the rebase wake on the rebase_key pair (head_sha..base_tip_sha), firing once per newly-due pair and re-arming when the head or base moves — not on a head-only key, which would wedge permanently once the base advances." },
|
|
231
|
+
{ "name": "not_a_wake_when_unknown_and_level", "text": "The loop does NOT wake when behind_by == 0 and mergeable is UNKNOWN (conflict state still computing) — it waits, matching the tick's Step 5." },
|
|
232
|
+
{ "name": "stays_a_dumb_pipe", "text": "The loop only signals a rebase is due and hands off to a tick; it does NOT apply the conflict_resolve_attempts budget, resolve conflicts, or dispatch the rebase itself — those belong to the tick and /muggle-do." }
|
|
233
|
+
]
|
|
234
|
+
},
|
|
235
|
+
{
|
|
236
|
+
"id": 19,
|
|
237
|
+
"eval_name": "arm-monitor-wakes-on-cidigest-change-while-blocked",
|
|
238
|
+
"prompt": "Arm the watch for muggle-ai-works-pr526 526 while it is blocked pending a human (last_seen.blocked present, reason ci_escalated). Describe the watch.sh monitor loop and specifically what makes it resume the block when the head's CI later moves to green, or a rerun/external deploy check posts — i.e. a non-red CI change.",
|
|
239
|
+
"files": [],
|
|
240
|
+
"assertions": [
|
|
241
|
+
{ "name": "blocked_probe_is_conditional", "text": "The loop runs the blocked-resume CI probe only while the watch is blocked — it keys off BLOCKED_CIDIGEST being non-empty, and the probe stays dormant when the watch is not blocked." },
|
|
242
|
+
{ "name": "wakes_on_any_digest_move_not_just_red", "text": "While blocked, the loop wakes on ANY change to the head's CI digest away from BLOCKED_CIDIGEST — a green pass, a rerun, or an external deploy check — not only on a settled-red transition." },
|
|
243
|
+
{ "name": "digest_matches_fingerprint_definition", "text": "The digest the loop computes is the same bucket-plus-sorted-name/conclusion signature the blocked fingerprint's ci_digest uses, so the loop and the tick's Step 2.5 resume gate agree by construction." },
|
|
244
|
+
{ "name": "resume_is_the_tick_job", "text": "The loop only wakes on the digest move; clearing last_seen.blocked and re-evaluating belongs to the tick's Step 2.5 fingerprint gate, and the blocked-tick flag/resume path is what maintains BLOCKED_CIDIGEST — the loop never writes it." }
|
|
245
|
+
]
|
|
219
246
|
}
|
|
220
247
|
]
|
|
221
248
|
}
|
|
@@ -26,4 +26,4 @@ Append the terminal line per [`output-templates/watcher-log.md`](output-template
|
|
|
26
26
|
|
|
27
27
|
### Step 4 — Unschedule the cron
|
|
28
28
|
|
|
29
|
-
Cancel this slot's cron per [`cancel-cron.md`](cancel-cron.md).
|
|
29
|
+
Cancel this slot's cron per [`cancel-cron.md`](cancel-cron.md), which reports **found** or **not-found**. Record the result in `followup.log` — `finalize cancel=<found|not-found>` — so a later stale fire's orphan gate ([`contract.md`](contract.md) Step 0) can tell a reachable cron whose queued fires merely drain (**found** here) from one it could never reach (**not-found** here, the orphan case). A **not-found** is otherwise benign — a manually-run tick, or a cron that already expired. Recurring `/loop` crons auto-expire after 7 days; that lapse is the gap [`reconcile.md`](reconcile.md) exists to catch.
|
|
@@ -176,8 +176,10 @@ The watch loop's comparison floor — plain `KEY=VALUE` lines, one file per slot
|
|
|
176
176
|
- `COM` — highest thread-comment id already handled
|
|
177
177
|
- `THREADS` — semicolon-joined ids of threads already known unresolved
|
|
178
178
|
- `CIRED` — head SHA whose settled-red CI the drain already handled; empty when the checks are green, still pending, or unseen. The CI floor is a SHA rather than a monotonic id because the check rollup flips green↔red and resets on each push — keying on the head SHA fires the loop once per red head and re-arms on the next push ([`arm-watcher.md`](arm-watcher.md)).
|
|
179
|
+
- `REBASED` — the `rebase_key` (`<head_sha>..<base_tip_sha>`) whose behind/conflicting state the drain already handled; empty when the branch is current with its base or unseen. Keyed on the pair, not a monotonic id, because staleness depends on both sides — the loop fires once per newly-due pair and re-arms when the head or the base moves. Same key [`contract.md`](contract.md) Step 5 dedups rebase dispatch on; the monitor only signals a due rebase, the tick applies the `conflict_resolve_attempts` budget.
|
|
180
|
+
- `BLOCKED_CIDIGEST` — the head's `ci_digest` at the moment a block was flagged (the same signature `blocked.fingerprint.ci_digest` carries); empty whenever the watch is not blocked. Non-empty arms the loop's blocked-resume probe: while set, the loop wakes on any move of the live digest away from this value, so a block resumes on a green pass, a rerun, or an external deploy check — not only on red. Written by [`blocked-tick.md`](blocked-tick.md) when a block is flagged and cleared to empty when it resumes.
|
|
179
181
|
|
|
180
|
-
Written whole-file by the orchestrating session — seeded at arm time to the **drain's own observed max
|
|
182
|
+
Written whole-file by the orchestrating session — the review/comment/CI/rebase floors seeded at arm time (`REV`/`COM` to the **drain's own observed max**, `CIRED`/`REBASED` to what the drain handled) and advanced after every cycle to the **handled wave's snapshot** (the max review-id and comment-id the cycle read at the start of handling, captured before replies were posted); `BLOCKED_CIDIGEST` is maintained by the blocked-tick flag/resume path rather than the arm/cycle path. **Never** derived from a fresh fetch taken after the drain or after replies land: a later fetch includes ids that arrived in the handling window and were never read, and marking them seen swallows them ([`arm-watcher.md`](arm-watcher.md)). The advance also never moves any floor past an unresolved thread the cycle left unaddressed. Read by the watch loop each iteration; the loop never writes it. A stale watermark makes the next reported event the loop's own reply.
|
|
181
183
|
|
|
182
184
|
**Never `source` this file, and quote or extract values.** `THREADS` holds bare semicolons: sourced unquoted, the shell splits the line at the first `;` and silently drops every id after it — the loop then re-reports known threads as new. The watch loop must extract values verbatim (e.g. `grep '^THREADS=' | cut -d= -f2- | tr -d '"\r'`), tolerating quotes and CRLF; writers should quote the value anyway.
|
|
183
185
|
|
|
@@ -1,236 +1,41 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: muggle-pr-visual-walkthrough
|
|
3
|
-
model: sonnet
|
|
4
|
-
description: Renders and posts a visual walkthrough of Muggle AI E2E acceptance test results to a PR — per-test-case dashboard links, step-by-step screenshots, and pass/fail summary — using the `muggle build-pr-section` CLI for deterministic formatting with automatic fit-vs-overflow. Use at the end of any Muggle Test test run (local or remote) to give PR reviewers clickable visual evidence that user flows work. Triggers on 'post results to PR', 'attach walkthrough to PR', 'share E2E screenshots on the PR', 'add visual walkthrough to PR'.
|
|
5
|
-
---
|
|
6
|
-
|
|
7
|
-
# Muggle Test PR Visual Walkthrough
|
|
8
|
-
|
|
9
|
-
> Telemetry first step: see [`_shared/telemetry-emit.md`](../_shared/telemetry-emit.md). Use `skillName: "muggle-pr-visual-walkthrough"`.
|
|
10
|
-
|
|
11
|
-
Renders a visual walkthrough of Muggle AI E2E acceptance test results and posts it to a PR. Each test case
|
|
12
|
-
|
|
13
|
-
This is
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
|
18
|
-
|
|
|
19
|
-
| `muggle-
|
|
20
|
-
| `muggle-test
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
## Preferences
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
{
|
|
43
|
-
"name": "<test case title>",
|
|
44
|
-
"description": "<one-line description of what this test verifies>",
|
|
45
|
-
"useCaseName": "<parent use case title>",
|
|
46
|
-
"testCaseId": "<UUID>",
|
|
47
|
-
"testScriptId": "<UUID>",
|
|
48
|
-
"runId": "<UUID>",
|
|
49
|
-
"viewUrl": "https://www.muggle-ai.com/...",
|
|
50
|
-
"status": "passed",
|
|
51
|
-
"steps": [
|
|
52
|
-
{ "stepIndex": 0, "action": "Click login button", "screenshotUrl": "https://..." },
|
|
53
|
-
{ "stepIndex": 1, "action": "Type email", "screenshotUrl": "https://..." }
|
|
54
|
-
]
|
|
55
|
-
},
|
|
56
|
-
{
|
|
57
|
-
"name": "Checkout flow",
|
|
58
|
-
"description": "Verify a shopper can complete checkout with a saved card.",
|
|
59
|
-
"useCaseName": "Purchase",
|
|
60
|
-
"testCaseId": "<UUID>",
|
|
61
|
-
"testScriptId": "<UUID>",
|
|
62
|
-
"runId": "<UUID>",
|
|
63
|
-
"viewUrl": "https://www.muggle-ai.com/...",
|
|
64
|
-
"status": "failed",
|
|
65
|
-
"steps": [
|
|
66
|
-
{ "stepIndex": 0, "action": "Open cart", "screenshotUrl": "https://..." }
|
|
67
|
-
],
|
|
68
|
-
"failureStepIndex": 2,
|
|
69
|
-
"error": "Element not found: Click checkout button",
|
|
70
|
-
"artifactsDir": "/Users/.../~/.muggle-ai/sessions/<runId>"
|
|
71
|
-
},
|
|
72
|
-
{
|
|
73
|
-
"name": "Clear search input restores full list",
|
|
74
|
-
"description": "Verify clearing the search input restores all options.",
|
|
75
|
-
"useCaseName": "Filter Dropdowns",
|
|
76
|
-
"testCaseId": "<UUID>",
|
|
77
|
-
"runId": "<UUID>",
|
|
78
|
-
"viewUrl": "https://www.muggle-ai.com/...",
|
|
79
|
-
"status": "inconclusive",
|
|
80
|
-
"steps": [],
|
|
81
|
-
"reason": "No replayable script exists yet — needs first generation run."
|
|
82
|
-
}
|
|
83
|
-
]
|
|
84
|
-
}
|
|
85
|
-
```
|
|
86
|
-
|
|
87
|
-
Required fields per test: `name`, `testCaseId`, `runId`, `viewUrl`, `status`, `steps[]` with `{stepIndex, action, screenshotUrl}`.
|
|
88
|
-
|
|
89
|
-
- **Failed** tests additionally require `failureStepIndex` and `error`.
|
|
90
|
-
- **Inconclusive** tests additionally require `reason` (one short sentence on why the result is neither pass nor fail). `steps[]` may be empty — that's fine. Inconclusive is for runs that couldn't be classified pass/fail (no replayable script, environment precondition unmet, infra error blocked execution, agent stalled before reaching the assertion). **Never silently drop these — always emit them as `inconclusive`.** The CLI counts them in the overview and renders an `⚠️` row with the dashboard link so reviewers can click through. If you find yourself wanting to skip a test or hand-write a comment because the schema "doesn't fit," that is the schema fitting — use `inconclusive`.
|
|
91
|
-
|
|
92
|
-
### Verdict
|
|
93
|
-
|
|
94
|
-
The renderer computes a verdict from the tests and emits a `**Verdict:** ✅ PASS | ❌ FAIL | ⚠️ INCONCLUSIVE` line at the top of the overview. The policy is strict:
|
|
95
|
-
|
|
96
|
-
- Any failed test → **FAIL** (regardless of how many passed or are inconclusive).
|
|
97
|
-
- No failures but any inconclusive → **INCONCLUSIVE**.
|
|
98
|
-
- All passed → **PASS**.
|
|
99
|
-
- Empty report → no verdict line.
|
|
100
|
-
|
|
101
|
-
You do not compute or render the verdict yourself — the CLI does. Never write a "Verdict: PASS" line into a hand-edited summary; it will conflict with the CLI's computed verdict.
|
|
102
|
-
|
|
103
|
-
**Optional but recommended** per test:
|
|
104
|
-
- `description` — a one-line summary of what the test case verifies. Shown in the collapsible header for each test and helps reviewers understand the test without expanding it. Pull from the test case's `title`/`description` via `muggle-remote-test-case-get`.
|
|
105
|
-
- `useCaseName` — the parent use case title. When present on any test, the overview list is grouped by use case; otherwise it is rendered as a flat list. Pull from `muggle-remote-use-case-get` using the test case's parent use-case id.
|
|
106
|
-
- `testScriptId` and `artifactsDir` are also optional.
|
|
107
|
-
|
|
108
|
-
If any required field is missing, stop and tell the caller exactly what's missing. Never fabricate data.
|
|
109
|
-
|
|
110
|
-
## Step 1: Assemble the `E2eReport`
|
|
111
|
-
|
|
112
|
-
Read `plugin/skills/muggle-pr-visual-walkthrough/e2e-report-assembly.md` for the full assembly guide.
|
|
113
|
-
|
|
114
|
-
## Step 2: Render via `muggle build-pr-section`
|
|
115
|
-
|
|
116
|
-
Pipe the `E2eReport` JSON to the CLI. It writes `{"body": "...", "comment": "..." | null}` to stdout — the `body` is the E2E markdown block, and `comment` is a non-null overflow comment only when the full body exceeds the byte budget (default 60 KB).
|
|
117
|
-
|
|
118
|
-
```bash
|
|
119
|
-
echo "$REPORT_JSON" | muggle build-pr-section > /tmp/muggle-pr-section.json
|
|
120
|
-
```
|
|
121
|
-
|
|
122
|
-
- Exit **non-zero** → the CLI wrote a descriptive error to stderr. Surface it to the user; do not swallow it.
|
|
123
|
-
- `comment` is **`null`** (fit case) → everything is inline in `body`. Post `body` only.
|
|
124
|
-
- `comment` is a **non-null string** (overflow case) → `body` contains the summary + a pointer; `comment` contains the full per-step details. Post both, in order.
|
|
125
|
-
|
|
126
|
-
Never hand-write the walkthrough markdown. Never modify the CLI's output before posting. The CLI owns the format.
|
|
127
|
-
|
|
128
|
-
## Step 3 — Mode A: Post to an existing PR
|
|
129
|
-
|
|
130
|
-
Used by `muggle-test` and `muggle-test-feature-local`, where the user is mid-development and a PR already exists on the current branch.
|
|
131
|
-
|
|
132
|
-
### 3A.1: Find the PR
|
|
133
|
-
|
|
134
|
-
```bash
|
|
135
|
-
gh pr view --json number,url,title 2>/dev/null
|
|
136
|
-
```
|
|
137
|
-
|
|
138
|
-
- **PR exists** → continue to 3A.2
|
|
139
|
-
- **No PR exists** → use `AskUserQuestion`:
|
|
140
|
-
- "Create a new PR with the visual walkthrough in the body"
|
|
141
|
-
- "Skip posting"
|
|
142
|
-
- If the user chooses to create a new PR, switch to Mode B and return the rendered `body`/`comment` to the caller for embedding in `gh pr create`. Do not create the PR directly from this skill unless the caller has no better way to do it.
|
|
143
|
-
- **`gh` not installed or not authenticated** → tell the user, suggest `gh auth login`, stop.
|
|
144
|
-
|
|
145
|
-
### 3A.2: Post the body as a PR comment
|
|
146
|
-
|
|
147
|
-
Extract the `body` field with `jq -r` (not `sed`) so JSON escape sequences are properly decoded, append the Muggle Works signature (command `/muggle-pr-visual-walkthrough`) per [`../_shared/vcs/post-signature.md`](../_shared/vcs/post-signature.md), then pipe to `--body-file -`. The renderer's sentinel stays at the top of `body`, so the report-format guardrail still recognises the post:
|
|
148
|
-
|
|
149
|
-
```bash
|
|
150
|
-
{
|
|
151
|
-
jq -r '.body' /tmp/muggle-pr-section.json
|
|
152
|
-
printf '\n\n%s\n' '🤖 _Posted by `/muggle-pr-visual-walkthrough` · [Muggle Works](https://github.com/multiplex-ai/muggle-ai-works)_'
|
|
153
|
-
} | gh pr comment <pr-number> --body-file -
|
|
154
|
-
```
|
|
155
|
-
|
|
156
|
-
### 3A.3: Post the overflow comment only if the CLI emitted one
|
|
157
|
-
|
|
158
|
-
```bash
|
|
159
|
-
{
|
|
160
|
-
jq -r '.comment' /tmp/muggle-pr-section.json
|
|
161
|
-
printf '\n\n%s\n' '🤖 _Posted by `/muggle-pr-visual-walkthrough` · [Muggle Works](https://github.com/multiplex-ai/muggle-ai-works)_'
|
|
162
|
-
} | gh pr comment <pr-number> --body-file -
|
|
163
|
-
```
|
|
164
|
-
|
|
165
|
-
**Skip this step entirely if `comment` is `null`** — do not post a placeholder. The CLI decides fit-vs-overflow; never post the overflow comment speculatively.
|
|
166
|
-
|
|
167
|
-
### 3A.4: Confirm to the user
|
|
168
|
-
|
|
169
|
-
> "Visual walkthrough posted to PR #<number>. Reviewers can click any test case link to see the step-by-step screenshots on the Muggle AI dashboard."
|
|
170
|
-
|
|
171
|
-
Include the PR URL in the confirmation.
|
|
172
|
-
|
|
173
|
-
## Step 3 — Mode B: Return rendered block for embedding in a new PR
|
|
174
|
-
|
|
175
|
-
Used by `muggle-do`'s `open-prs.md`, where the PR does not exist yet and the caller is assembling the PR body from multiple sections (`## Goal`, `## Acceptance Criteria`, `## Changes`, plus this walkthrough).
|
|
176
|
-
|
|
177
|
-
Instead of posting, **return** the CLI output to the caller's context so they can:
|
|
178
|
-
|
|
179
|
-
1. **Embed `body`** in their PR body, concatenated after `## Changes`. `body` already includes its own `## E2E Acceptance Results` header — do not add another.
|
|
180
|
-
2. **Create the PR** with `gh pr create --title "..." --body "..."` using the concatenated body.
|
|
181
|
-
3. **Post `comment` as a follow-up only if the CLI emitted one**, ending the posted body with the signature (the caller owns this post, so command `/muggle-do`) per [`../_shared/vcs/post-signature.md`](../_shared/vcs/post-signature.md):
|
|
182
|
-
|
|
183
|
-
```bash
|
|
184
|
-
{
|
|
185
|
-
jq -r '.comment' /tmp/muggle-pr-section.json
|
|
186
|
-
printf '\n\n%s\n' '🤖 _Posted by `/muggle-do` · [Muggle Works](https://github.com/multiplex-ai/muggle-ai-works)_'
|
|
187
|
-
} | gh pr comment <new-pr-number> --body-file -
|
|
188
|
-
```
|
|
189
|
-
|
|
190
|
-
Skip if `comment` is `null`.
|
|
191
|
-
|
|
192
|
-
In Mode B, this skill does not call `gh pr comment` or `gh pr create` itself — the caller owns PR creation because it also owns branch pushing, title building (including `[E2E FAILING]` prefix on failures), and multi-repo orchestration.
|
|
193
|
-
|
|
194
|
-
## Step 3 — Mode C: Embed mode for open-PR sweep orchestrators
|
|
195
|
-
|
|
196
|
-
Used when invoked as a sub-step of a open-PR sweep orchestrator (see `plugin/skills/muggle-test/SKILL.md` Mode C) rather than as a top-level user invocation. The orchestrator's `acceptance-tester` subagent (see `plugin/agents/acceptance-tester.md`) composes a **single per-PR verdict comment** and folds the walkthrough into it — posting separately would create 2–3 disparate comments per test cycle and clutter the PR.
|
|
197
|
-
|
|
198
|
-
### 3C.1: Detect embed mode
|
|
199
|
-
|
|
200
|
-
The caller passes `mode: "embed"` as a skill argument. Default behavior (no `mode` passed, or `mode: "post"`) is Mode A — unchanged.
|
|
201
|
-
|
|
202
|
-
### 3C.2: Render but do not post
|
|
203
|
-
|
|
204
|
-
Run `muggle build-pr-section` exactly as in Step 2. Then, **instead of calling `gh pr comment`**:
|
|
205
|
-
|
|
206
|
-
1. **Return** the CLI output (`{ body, comment }`) to the caller as this skill's result.
|
|
207
|
-
2. Do **not** find a PR, do **not** post a standalone comment, do **not** prompt the user.
|
|
208
|
-
3. If `comment` is non-null (overflow case), return it alongside `body` — the orchestrator decides how to handle overflow (typically: inline `body` in the verdict comment, post `comment` as a follow-up).
|
|
209
|
-
|
|
210
|
-
### 3C.3: Hand off
|
|
211
|
-
|
|
212
|
-
The orchestrator (`acceptance-tester`) concatenates the returned `body` into its verdict comment template alongside the verdict summary, change-list, and any other sections it owns, then posts a single `gh pr comment` itself. This skill's job ends at returning the rendered markdown.
|
|
213
|
-
|
|
214
|
-
In Mode C, the same fit-vs-overflow contract from Step 2 applies — never modify the CLI's output, never fabricate fields, never post anything. The caller owns posting.
|
|
215
|
-
|
|
216
|
-
## Tool Reference
|
|
217
|
-
|
|
218
|
-
| Phase | Tool |
|
|
219
|
-
|:------|:-----|
|
|
220
|
-
| Gather per-step data (muggle-test, muggle-test-feature-local) | `muggle-remote-test-script-get` |
|
|
221
|
-
| Render the walkthrough markdown | `muggle build-pr-section` (shell) |
|
|
222
|
-
| Find existing PR (Mode A) | `gh pr view` |
|
|
223
|
-
| Post comment(s) (Mode A) | `gh pr comment` |
|
|
224
|
-
| Create new PR (Mode B, caller handles) | `gh pr create` |
|
|
225
|
-
| User confirmation (Mode A no-PR branch) | `AskUserQuestion` |
|
|
226
|
-
|
|
227
|
-
## Guardrails
|
|
228
|
-
|
|
229
|
-
- **Never hand-write the walkthrough markdown** — always call `muggle build-pr-section`. The CLI is the single source of truth for formatting. If a test outcome doesn't fit `passed | failed`, that's what `inconclusive` is for — never fall back to a hand-written summary, a custom table, a "Verdict: PASS" header, a `Tested on:`/`Project:` footer, or any other freeform text. The CLI already emits per-test-case dashboard links (no project-level link), uses `https://www.muggle-ai.com/...` URLs (never the test target's `localhost` URL), and computes the verdict — anything you would manually add is wrong by construction.
|
|
230
|
-
- **Never modify the CLI's output** — post `body` and (if present) `comment` verbatim. Any reformatting defeats the fit-vs-overflow budget math.
|
|
231
|
-
- **Never invent report fields** — if `projectId`, a per-test `viewUrl`, or per-step `screenshotUrl` is missing, stop and report what's missing. Do not fabricate URLs or fill in placeholders.
|
|
232
|
-
- **Never post the overflow comment when `comment` is `null`** — the CLI decides fit-vs-overflow.
|
|
233
|
-
- **Never create a PR without confirmation in Mode A** — if no PR exists, ask the user or switch to Mode B and hand back to the caller.
|
|
234
|
-
- **Don't run tests** — this skill only renders and posts existing results. If the `E2eReport` is not in context, redirect the caller to `muggle-test`, `muggle-test-feature-local`, or `muggle-do`.
|
|
235
|
-
- **Mode is chosen by the caller, not the user** — `muggle-test` top-level uses Mode A; `muggle-do` uses Mode B; open-PR sweep orchestrators (`muggle-test` Mode C / `acceptance-tester`) pass `mode: "embed"` for Mode C. Don't ask the user which mode to use.
|
|
236
|
-
- **Never post in Mode C** — when `mode: "embed"` is passed, return the rendered body to the caller and stop. The orchestrator owns the single verdict comment.
|
|
1
|
+
---
|
|
2
|
+
name: muggle-pr-visual-walkthrough
|
|
3
|
+
model: sonnet
|
|
4
|
+
description: Renders and posts a visual walkthrough of Muggle AI E2E acceptance test results to a PR — per-test-case dashboard links, step-by-step screenshots, and pass/fail summary — using the `muggle build-pr-section` CLI for deterministic formatting with automatic fit-vs-overflow. Use at the end of any Muggle Test test run (local or remote) to give PR reviewers clickable visual evidence that user flows work. Triggers on 'post results to PR', 'attach walkthrough to PR', 'share E2E screenshots on the PR', 'add visual walkthrough to PR'.
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
# Muggle Test PR Visual Walkthrough
|
|
8
|
+
|
|
9
|
+
> Telemetry first step: see [`_shared/telemetry-emit.md`](../_shared/telemetry-emit.md). Use `skillName: "muggle-pr-visual-walkthrough"`.
|
|
10
|
+
|
|
11
|
+
Renders a visual walkthrough of Muggle AI E2E acceptance test results and posts it to a PR. Each test case links to its detail page on the Muggle AI dashboard, so reviewers can click through to step-by-step screenshots — not just a pass/fail flag.
|
|
12
|
+
|
|
13
|
+
This skill is a **dispatcher**: it resolves the mode and any user interaction, then hands execution to the `visual-walkthrough-builder` agent (`plugin/agents/visual-walkthrough-builder.md`), which renders and delivers per its own contract.
|
|
14
|
+
|
|
15
|
+
This is the **canonical PR-walkthrough workflow** shared across every Muggle Test entry point:
|
|
16
|
+
|
|
17
|
+
| Caller | Mode | When to invoke |
|
|
18
|
+
| :--- | :--- | :--- |
|
|
19
|
+
| `muggle-test` | **Mode A** (post to existing PR) | After publishing results, user opts in via `AskUserQuestion` |
|
|
20
|
+
| `muggle-test-feature-local` | **Mode A** (post to existing PR) | After publishing the run, user opts in via `AskUserQuestion` |
|
|
21
|
+
| `muggle-do` / `open-prs.md` | **Mode B** (render-only for embedding) | During PR creation — caller embeds `body` in the PR create call and posts `comment` as follow-up |
|
|
22
|
+
| `muggle-test` Mode C / `acceptance-tester` agent | **Mode C** (embed in verdict comment) | Inside an open-PR sweep orchestrator — caller folds the rendered body into a single per-PR verdict comment |
|
|
23
|
+
|
|
24
|
+
## Preferences
|
|
25
|
+
|
|
26
|
+
Callers consult the `postPRVisualWalkthrough` gate **before** invoking this skill — by the time it runs, posting is already approved. Per-key gate definitions live in `plugin/skills/muggle-preferences/preference-gates/`.
|
|
27
|
+
|
|
28
|
+
## Procedure
|
|
29
|
+
|
|
30
|
+
1. **Resolve the mode.** Chosen by the caller, never the user: top-level `muggle-test`/`muggle-test-feature-local` → `post` (Mode A); `muggle-do` PR creation → `render-for-new-pr` (Mode B); an orchestrator passing `mode: "embed"` → Mode C.
|
|
31
|
+
2. **Mode A only — find the PR** with `gh pr view --json number,url,title`. No PR on the branch → `AskUserQuestion`: create a new PR with the walkthrough in the body (switch to Mode B and hand the rendered block back to the caller), or skip posting. `gh` missing/unauthenticated → tell the user, suggest `gh auth login`, stop. This is the skill's only interactive branch — resolve it **before** dispatching.
|
|
32
|
+
3. **Gather the inputs.** The `E2eReport` JSON if the caller already assembled it (see [`e2e-report-assembly.md`](e2e-report-assembly.md)), else the run identifiers (`projectId`, per-test `runId`/`testCaseId`) the agent needs to assemble it.
|
|
33
|
+
4. **Dispatch** the `visual-walkthrough-builder` agent (subagent type `muggle:visual-walkthrough-builder`; bare `visual-walkthrough-builder` where the plugin namespace is absent), synchronously, passing: mode, PR number + repo (Mode A), and the report JSON or identifiers. In a harness with no agent/subagent facility, execute `plugin/agents/visual-walkthrough-builder.md` inline instead.
|
|
34
|
+
5. **Relay the result.** Mode A → confirm to the user with the PR URL. Modes B/C → return the agent's `{body, comment}` to the caller verbatim; the caller owns PR creation (Mode B) or the single verdict comment (Mode C). A `needs-input:` line from the agent names a missing report field — surface it to the caller; never fabricate the field.
|
|
35
|
+
|
|
36
|
+
## Guardrails
|
|
37
|
+
|
|
38
|
+
- The rendered markdown is CLI-owned end to end — this skill never writes, edits, or post-processes it; those rules ride with the agent.
|
|
39
|
+
- Mode is chosen by the caller, not the user.
|
|
40
|
+
- Never create a PR without confirmation in Mode A.
|
|
41
|
+
- Don't run tests — this skill only dispatches rendering/posting of existing results. No report and no run identifiers in context → redirect the caller to `muggle-test`, `muggle-test-feature-local`, or `muggle-do`.
|