@muggleai/works 4.10.0 → 4.10.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +3 -3
- package/dist/{chunk-RK6XKSZK.js → chunk-Q7WNEMN4.js} +109 -43
- package/dist/{chunk-UVXOHXY2.js → chunk-ZKRHFTIC.js} +832 -89
- package/dist/cli.js +2 -2
- package/dist/index.js +2 -2
- package/dist/plugin/.claude-plugin/plugin.json +1 -1
- package/dist/plugin/.cursor-plugin/plugin.json +1 -1
- package/dist/plugin/README.md +4 -2
- package/dist/plugin/agents/acceptance-tester.md +24 -10
- package/dist/plugin/commands/m.md +7 -0
- package/dist/plugin/commands/mdo.md +7 -0
- package/dist/plugin/commands/mfeedback.md +7 -0
- package/dist/plugin/commands/mimport.md +7 -0
- package/dist/plugin/commands/mpr.md +7 -0
- package/dist/plugin/commands/mprefs.md +7 -0
- package/dist/plugin/commands/mregen.md +7 -0
- package/dist/plugin/commands/mrelease.md +7 -0
- package/dist/plugin/commands/mrepair.md +7 -0
- package/dist/plugin/commands/mstatus.md +7 -0
- package/dist/plugin/commands/mtest.md +7 -0
- package/dist/plugin/commands/mtestlocal.md +7 -0
- package/dist/plugin/commands/mtestprep.md +7 -0
- package/dist/plugin/commands/mupgrade.md +7 -0
- package/dist/plugin/scripts/ensure-electron-app.sh +35 -8
- package/dist/plugin/skills/_aliases.json +18 -0
- package/dist/plugin/skills/_shared/dev-server-readiness.md +231 -0
- package/dist/plugin/skills/_shared/failure-mode-handling.md +244 -0
- package/dist/plugin/skills/_shared/post-merge-cleanup.md +10 -0
- package/dist/plugin/skills/_shared/rebase-before-e2e.md +15 -0
- package/dist/plugin/skills/_shared/telemetry-emit.md +15 -0
- package/dist/plugin/skills/_shared/use-worktrees.md +53 -0
- package/dist/plugin/skills/do/e2e-acceptance.md +63 -18
- package/dist/plugin/skills/do/impact-analysis.md +45 -5
- package/dist/plugin/skills/do/open-prs.md +13 -4
- package/dist/plugin/skills/do/pr-followup.md +225 -0
- package/dist/plugin/skills/do/pre-flight.md +18 -11
- package/dist/plugin/skills/m/SKILL.md +8 -0
- package/dist/plugin/skills/mdo/SKILL.md +8 -0
- package/dist/plugin/skills/mfeedback/SKILL.md +8 -0
- package/dist/plugin/skills/mimport/SKILL.md +8 -0
- package/dist/plugin/skills/mpr/SKILL.md +8 -0
- package/dist/plugin/skills/mprefs/SKILL.md +8 -0
- package/dist/plugin/skills/mregen/SKILL.md +8 -0
- package/dist/plugin/skills/mrelease/SKILL.md +8 -0
- package/dist/plugin/skills/mrepair/SKILL.md +8 -0
- package/dist/plugin/skills/mstatus/SKILL.md +8 -0
- package/dist/plugin/skills/mtest/SKILL.md +8 -0
- package/dist/plugin/skills/mtestlocal/SKILL.md +8 -0
- package/dist/plugin/skills/mtestprep/SKILL.md +8 -0
- package/dist/plugin/skills/muggle/SKILL.md +9 -6
- package/dist/plugin/skills/muggle-do/SKILL.md +22 -4
- package/dist/plugin/skills/muggle-do-pr-followup/SKILL.md +37 -0
- package/dist/plugin/skills/muggle-do-task/SKILL.md +134 -0
- package/dist/plugin/skills/muggle-feedback/SKILL.md +38 -0
- package/dist/plugin/skills/muggle-feedback/ops/delete.md +49 -0
- package/dist/plugin/skills/muggle-feedback/ops/list.md +65 -0
- package/dist/plugin/skills/muggle-feedback/ops/submit.md +117 -0
- package/dist/plugin/skills/muggle-pr-visual-walkthrough/SKILL.md +68 -43
- package/dist/plugin/skills/muggle-pr-visual-walkthrough/e2e-report-assembly.md +86 -0
- package/dist/plugin/skills/muggle-preferences/SKILL.md +16 -108
- package/dist/plugin/skills/muggle-preferences/ops/change-one.md +10 -0
- package/dist/plugin/skills/muggle-preferences/ops/configure.md +49 -0
- package/dist/plugin/skills/muggle-preferences/ops/list.md +9 -0
- package/dist/plugin/skills/muggle-preferences/ops/reset.md +6 -0
- package/dist/plugin/skills/muggle-preferences/ops/set.md +10 -0
- package/dist/plugin/skills/muggle-preferences/preference-gates/README.md +1 -1
- package/dist/plugin/skills/muggle-preferences/preference-gates/autoCleanup.md +11 -0
- package/dist/plugin/skills/muggle-preferences/preference-gates/autoCreatePR.md +13 -0
- package/dist/plugin/skills/muggle-preferences/preference-gates/autoDetectChanges.md +3 -3
- package/dist/plugin/skills/muggle-preferences/preference-gates/autoE2ETest.md +11 -0
- package/dist/plugin/skills/muggle-preferences/preference-gates/autoLogin.md +1 -1
- package/dist/plugin/skills/muggle-preferences/preference-gates/autoPublishLocalResults.md +2 -2
- package/dist/plugin/skills/muggle-preferences/preference-gates/autoRebase.md +11 -0
- package/dist/plugin/skills/muggle-preferences/preference-gates/autoSelectLocalHost.md +19 -0
- package/dist/plugin/skills/muggle-preferences/preference-gates/autoSelectProject.md +1 -2
- package/dist/plugin/skills/muggle-preferences/preference-gates/autoUseWorktree.md +11 -0
- package/dist/plugin/skills/muggle-preferences/preference-gates/checkForUpdates.md +4 -4
- package/dist/plugin/skills/muggle-preferences/preference-gates/defaultExecutionMode.md +1 -1
- package/dist/plugin/skills/muggle-preferences/preference-gates/openTestResultsAfterRun.md +1 -1
- package/dist/plugin/skills/muggle-preferences/preference-gates/postPRVisualWalkthrough.md +8 -25
- package/dist/plugin/skills/muggle-preferences/preference-gates/showElectronBrowser.md +1 -1
- package/dist/plugin/skills/muggle-preferences/preference-gates/suggestRelatedTestCases.md +4 -4
- package/dist/plugin/skills/muggle-preferences/preference-gates/suggestRelatedUseCases.md +4 -4
- package/dist/plugin/skills/muggle-repair/SKILL.md +4 -2
- package/dist/plugin/skills/muggle-status/SKILL.md +8 -5
- package/dist/plugin/skills/muggle-test/SKILL.md +110 -86
- package/dist/plugin/skills/muggle-test-feature-local/SKILL.md +73 -66
- package/dist/plugin/skills/muggle-test-import/SKILL.md +28 -23
- package/dist/plugin/skills/muggle-test-prepare/SKILL.md +79 -18
- package/dist/plugin/skills/muggle-test-regenerate-missing/SKILL.md +34 -18
- package/dist/plugin/skills/muggle-test-regenerate-missing/evals/evals.json +9 -9
- package/dist/plugin/skills/muggle-upgrade/SKILL.md +4 -2
- package/dist/plugin/skills/muggle-works-npm-release/SKILL.md +6 -4
- package/dist/plugin/skills/mupgrade/SKILL.md +8 -0
- package/dist/release-manifest.json +4 -4
- package/dist/{src-GD3YFCCW.js → src-2G7ENRGI.js} +1 -1
- package/package.json +9 -7
- package/plugin/.claude-plugin/plugin.json +1 -1
- package/plugin/.cursor-plugin/plugin.json +1 -1
- package/plugin/README.md +4 -2
- package/plugin/agents/acceptance-tester.md +24 -10
- package/plugin/commands/m.md +7 -0
- package/plugin/commands/mdo.md +7 -0
- package/plugin/commands/mfeedback.md +7 -0
- package/plugin/commands/mimport.md +7 -0
- package/plugin/commands/mpr.md +7 -0
- package/plugin/commands/mprefs.md +7 -0
- package/plugin/commands/mregen.md +7 -0
- package/plugin/commands/mrelease.md +7 -0
- package/plugin/commands/mrepair.md +7 -0
- package/plugin/commands/mstatus.md +7 -0
- package/plugin/commands/mtest.md +7 -0
- package/plugin/commands/mtestlocal.md +7 -0
- package/plugin/commands/mtestprep.md +7 -0
- package/plugin/commands/mupgrade.md +7 -0
- package/plugin/scripts/ensure-electron-app.sh +35 -8
- package/plugin/skills/_aliases.json +18 -0
- package/plugin/skills/_shared/dev-server-readiness.md +231 -0
- package/plugin/skills/_shared/failure-mode-handling.md +244 -0
- package/plugin/skills/_shared/post-merge-cleanup.md +10 -0
- package/plugin/skills/_shared/rebase-before-e2e.md +15 -0
- package/plugin/skills/_shared/telemetry-emit.md +15 -0
- package/plugin/skills/_shared/use-worktrees.md +53 -0
- package/plugin/skills/do/e2e-acceptance.md +63 -18
- package/plugin/skills/do/impact-analysis.md +45 -5
- package/plugin/skills/do/open-prs.md +13 -4
- package/plugin/skills/do/pr-followup.md +225 -0
- package/plugin/skills/do/pre-flight.md +18 -11
- package/plugin/skills/m/SKILL.md +8 -0
- package/plugin/skills/mdo/SKILL.md +8 -0
- package/plugin/skills/mfeedback/SKILL.md +8 -0
- package/plugin/skills/mimport/SKILL.md +8 -0
- package/plugin/skills/mpr/SKILL.md +8 -0
- package/plugin/skills/mprefs/SKILL.md +8 -0
- package/plugin/skills/mregen/SKILL.md +8 -0
- package/plugin/skills/mrelease/SKILL.md +8 -0
- package/plugin/skills/mrepair/SKILL.md +8 -0
- package/plugin/skills/mstatus/SKILL.md +8 -0
- package/plugin/skills/mtest/SKILL.md +8 -0
- package/plugin/skills/mtestlocal/SKILL.md +8 -0
- package/plugin/skills/mtestprep/SKILL.md +8 -0
- package/plugin/skills/muggle/SKILL.md +9 -6
- package/plugin/skills/muggle-do/SKILL.md +22 -4
- package/plugin/skills/muggle-do-pr-followup/SKILL.md +37 -0
- package/plugin/skills/muggle-do-task/SKILL.md +134 -0
- package/plugin/skills/muggle-feedback/SKILL.md +38 -0
- package/plugin/skills/muggle-feedback/ops/delete.md +49 -0
- package/plugin/skills/muggle-feedback/ops/list.md +65 -0
- package/plugin/skills/muggle-feedback/ops/submit.md +117 -0
- package/plugin/skills/muggle-pr-visual-walkthrough/SKILL.md +68 -43
- package/plugin/skills/muggle-pr-visual-walkthrough/e2e-report-assembly.md +86 -0
- package/plugin/skills/muggle-preferences/SKILL.md +16 -108
- package/plugin/skills/muggle-preferences/ops/change-one.md +10 -0
- package/plugin/skills/muggle-preferences/ops/configure.md +49 -0
- package/plugin/skills/muggle-preferences/ops/list.md +9 -0
- package/plugin/skills/muggle-preferences/ops/reset.md +6 -0
- package/plugin/skills/muggle-preferences/ops/set.md +10 -0
- package/plugin/skills/muggle-preferences/preference-gates/README.md +1 -1
- package/plugin/skills/muggle-preferences/preference-gates/autoCleanup.md +11 -0
- package/plugin/skills/muggle-preferences/preference-gates/autoCreatePR.md +13 -0
- package/plugin/skills/muggle-preferences/preference-gates/autoDetectChanges.md +3 -3
- package/plugin/skills/muggle-preferences/preference-gates/autoE2ETest.md +11 -0
- package/plugin/skills/muggle-preferences/preference-gates/autoLogin.md +1 -1
- package/plugin/skills/muggle-preferences/preference-gates/autoPublishLocalResults.md +2 -2
- package/plugin/skills/muggle-preferences/preference-gates/autoRebase.md +11 -0
- package/plugin/skills/muggle-preferences/preference-gates/autoSelectLocalHost.md +19 -0
- package/plugin/skills/muggle-preferences/preference-gates/autoSelectProject.md +1 -2
- package/plugin/skills/muggle-preferences/preference-gates/autoUseWorktree.md +11 -0
- package/plugin/skills/muggle-preferences/preference-gates/checkForUpdates.md +4 -4
- package/plugin/skills/muggle-preferences/preference-gates/defaultExecutionMode.md +1 -1
- package/plugin/skills/muggle-preferences/preference-gates/openTestResultsAfterRun.md +1 -1
- package/plugin/skills/muggle-preferences/preference-gates/postPRVisualWalkthrough.md +8 -25
- package/plugin/skills/muggle-preferences/preference-gates/showElectronBrowser.md +1 -1
- package/plugin/skills/muggle-preferences/preference-gates/suggestRelatedTestCases.md +4 -4
- package/plugin/skills/muggle-preferences/preference-gates/suggestRelatedUseCases.md +4 -4
- package/plugin/skills/muggle-repair/SKILL.md +4 -2
- package/plugin/skills/muggle-status/SKILL.md +8 -5
- package/plugin/skills/muggle-test/SKILL.md +110 -86
- package/plugin/skills/muggle-test-feature-local/SKILL.md +73 -66
- package/plugin/skills/muggle-test-import/SKILL.md +28 -23
- package/plugin/skills/muggle-test-prepare/SKILL.md +79 -18
- package/plugin/skills/muggle-test-regenerate-missing/SKILL.md +34 -18
- package/plugin/skills/muggle-test-regenerate-missing/evals/evals.json +9 -9
- package/plugin/skills/muggle-upgrade/SKILL.md +4 -2
- package/plugin/skills/muggle-works-npm-release/SKILL.md +6 -4
- package/plugin/skills/mupgrade/SKILL.md +8 -0
- package/scripts/postinstall.mjs +11 -1
|
@@ -0,0 +1,244 @@
|
|
|
1
|
+
# Failure-Mode Handling — Shared Reference
|
|
2
|
+
|
|
3
|
+
> Source of truth for **(a)** the pre-execution replay-vs-regen choice and **(b)** the post-execution failure router used by `muggle-test`, `muggle-test-feature-local`, `muggle-do-task`, and `muggle-test-regenerate-missing`. Skills MUST link here rather than restate the rules.
|
|
4
|
+
|
|
5
|
+
## The contract
|
|
6
|
+
|
|
7
|
+
Every decision in this doc follows the same shape:
|
|
8
|
+
|
|
9
|
+
1. **Classify** — pick a bucket from a fixed taxonomy based on the available signals.
|
|
10
|
+
2. **Suggest** — present the bucket-specific suggestion to the user via `AskUserQuestion`.
|
|
11
|
+
3. **User decides** — the user always makes the final call. Never auto-act.
|
|
12
|
+
4. **Emit telemetry** — record `(aiClassification, aiSuggestion, userAction, signals)` via `muggle-local-telemetry-event-emit` so the rules can be refined later from real data.
|
|
13
|
+
|
|
14
|
+
The classification rules below are **starting heuristics**. Trust the AI's bucket only enough to phrase a default suggestion — let the user override freely.
|
|
15
|
+
|
|
16
|
+
---
|
|
17
|
+
|
|
18
|
+
## F. Run-level verdict taxonomy
|
|
19
|
+
|
|
20
|
+
The buckets in sections A–C operate **per script** (one classification per test-case execution). This section defines the **single verdict** an acceptance-tester subagent returns to its orchestrator — one verdict per PR / per dispatched run, summarizing every test case it touched.
|
|
21
|
+
|
|
22
|
+
### Verdicts
|
|
23
|
+
|
|
24
|
+
| Verdict | Meaning |
|
|
25
|
+
|---|---|
|
|
26
|
+
| **PASS** | All relevant test cases ran and passed. The PR's user-visible change is exercised and works. |
|
|
27
|
+
| **FAIL** | At least one test case demonstrated wrong behavior caused by the PR code. The PR is regressive. |
|
|
28
|
+
| **PARTIAL** | Some relevant tests passed; some were INCONCLUSIVE (not failed). Useful coverage, with gaps. |
|
|
29
|
+
| **INCONCLUSIVE** | All relevant tests were blocked by environment factors — script staleness, empty local DB, missing fixtures, tests out of date relative to current master UI. **Not a regression signal.** |
|
|
30
|
+
| **BLOCKED** | An infra blocker prevented any meaningful execution: Auth0 tenant rejecting test emails, dev server unreachable, MCP service down, port wouldn't bind. **Not a regression signal.** |
|
|
31
|
+
| **SKIPPED** | No code under test — placeholder branch, empty diff vs. master, deliberate no-op change. |
|
|
32
|
+
|
|
33
|
+
PASS and FAIL are the only verdicts that say something about the PR itself. PARTIAL means "some coverage, some gaps." INCONCLUSIVE, BLOCKED, and SKIPPED all mean **don't draw a regression conclusion from this run** — they signal an environment, script, or input problem, not a product defect.
|
|
34
|
+
|
|
35
|
+
### Mapping per-script buckets to run verdicts
|
|
36
|
+
|
|
37
|
+
When the subagent aggregates per-script results from sections B and C into one verdict:
|
|
38
|
+
|
|
39
|
+
| Per-script bucket | Section | Typical run verdict |
|
|
40
|
+
|---|---|---|
|
|
41
|
+
| `infra` (replay or regen) | B / C | **BLOCKED** |
|
|
42
|
+
| `stale-script` (replay) | B | **INCONCLUSIVE** |
|
|
43
|
+
| `product-defect` (replay) | B | **FAIL** |
|
|
44
|
+
| `product-uxux` (regen) | C | **FAIL** |
|
|
45
|
+
| `agent-course` (regen) | C | **INCONCLUSIVE** (agent couldn't complete, not a PR defect) |
|
|
46
|
+
| `transient` (regen) | C | retry once; if still transient → **INCONCLUSIVE** |
|
|
47
|
+
|
|
48
|
+
When a run mixes buckets across multiple test cases, take the most-significant verdict in this priority order: **FAIL > PARTIAL > BLOCKED > INCONCLUSIVE > SKIPPED > PASS**. If any one test case maps to FAIL, the run is FAIL. If some passed and some were INCONCLUSIVE, the run is PARTIAL.
|
|
49
|
+
|
|
50
|
+
### How orchestrators consume the verdict
|
|
51
|
+
|
|
52
|
+
Subagents return the verdict as part of a structured block, not free-form prose. The orchestrator parses this block to decide whether to post a "regression" PR comment, a "needs investigation" comment, or skip the PR. See `agents/acceptance-tester.md` for the exact block format the subagent must emit.
|
|
53
|
+
|
|
54
|
+
---
|
|
55
|
+
|
|
56
|
+
## A. Pre-execution: replay vs regen (used by `muggle-test`)
|
|
57
|
+
|
|
58
|
+
Run during change analysis, **per impacted test case**. Picks the initial execution mode before Step 7. Other skills with a single user-picked target (`muggle-test-feature-local`, `muggle-do-task`) skip this section — the user already chose.
|
|
59
|
+
|
|
60
|
+
### Inputs
|
|
61
|
+
|
|
62
|
+
- The change summary from `git diff` (file paths + diff content).
|
|
63
|
+
- The test case (title, description, instructions, last passing run timestamp).
|
|
64
|
+
- Existing test scripts for that test case from `muggle-remote-test-script-list`.
|
|
65
|
+
|
|
66
|
+
### Rules (fire in order; first match wins)
|
|
67
|
+
|
|
68
|
+
| # | Condition | Mode | Reason |
|
|
69
|
+
|---|---|---|---|
|
|
70
|
+
| 1 | No replayable/succeeded script exists for the test case | `regen` | Nothing to replay. |
|
|
71
|
+
| 2 | A changed file looks like UI/markup mapped to this test case (component, page, route, template, JSX/TSX/Vue/Svelte/HTML, CSS that changes layout or selectors) | `regen` | Selectors likely broken — replay would fail on staleness. |
|
|
72
|
+
| 3 | Last successful run was > **30 days** ago | `regen` | Drift accumulates; the saved script is stale even without a flagged change. |
|
|
73
|
+
| 4 | Otherwise — changes are logic-only / backend / styling-without-DOM-impact | `replay` | Selectors should still work; replay catches real regressions. |
|
|
74
|
+
|
|
75
|
+
### Mapping "changed file → test case"
|
|
76
|
+
|
|
77
|
+
Use the test case's `instructions` and `goal` text plus filenames in the diff. Match heuristically — a test case titled "Submit signup form" plus a diff in `app/auth/signup/page.tsx` is a clear match. When in doubt, prefer `regen` (rule 2 wins) — a needless regen wastes budget; a stale replay fails the test for the wrong reason.
|
|
78
|
+
|
|
79
|
+
### Telemetry
|
|
80
|
+
|
|
81
|
+
Emit **`pre-execution-classification`** for every test case in the batch, before execution starts:
|
|
82
|
+
|
|
83
|
+
```json
|
|
84
|
+
{
|
|
85
|
+
"eventType": "pre-execution-classification",
|
|
86
|
+
"skillName": "muggle-test",
|
|
87
|
+
"aiClassification": "replay" | "regen",
|
|
88
|
+
"aiSuggestion": "<same as classification>",
|
|
89
|
+
"testCaseId": "<id>",
|
|
90
|
+
"projectId": "<id>",
|
|
91
|
+
"signals": ["rule-1-no-script" | "rule-2-ui-changed" | "rule-3-stale-30d" | "rule-4-default-replay"],
|
|
92
|
+
"metadata": { "lastPassedAgeDays": <n>, "changedFilesMatched": ["..."] }
|
|
93
|
+
}
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
Skill MAY surface the per-test-case decision in the report (e.g., "regenerating 3, replaying 5") so the user can override before execution. If the user overrides, emit a follow-up event with `userAction` set to the overriding mode.
|
|
97
|
+
|
|
98
|
+
---
|
|
99
|
+
|
|
100
|
+
## B. Post-replay failure (used by all four skills with replays)
|
|
101
|
+
|
|
102
|
+
Triggered when `muggle-local-execute-replay` returns `status: "failed"` (or non-zero Electron exit) **and** the run is not an orchestration timeout already covered in `muggle-test-feature-local/SKILL.md` Step 6.
|
|
103
|
+
|
|
104
|
+
### Buckets
|
|
105
|
+
|
|
106
|
+
| Bucket | Meaning |
|
|
107
|
+
|---|---|
|
|
108
|
+
| **infra** | Something is wrong inside Muggle Test itself (e.g., click didn't register on a clearly-clickable element, Electron crash, browser engine quirk). Not the user's fault and not a stale script. |
|
|
109
|
+
| **stale-script** | The test script no longer matches the live UI (selectors moved, label paths changed, page renamed). The product still works; the script is out of date. |
|
|
110
|
+
| **product-defect** | The script and infra are fine; the user's app actually misbehaved (assertion failure on previously-passing step, unexpected error, wrong page after action). This is the failure mode acceptance testing exists to catch. |
|
|
111
|
+
|
|
112
|
+
### Initial signal heuristics
|
|
113
|
+
|
|
114
|
+
Derive signals from the run's per-step results, error messages, and step screenshots (`muggle-local-run-result-get`):
|
|
115
|
+
|
|
116
|
+
- **infra** signals: `electron-crash`, `chromium-error`, `click-no-effect-on-clickable-element`, `timeout-on-trivial-wait`, `internal-error-in-mcp-output`.
|
|
117
|
+
- **stale-script** signals: `element-not-found`, `selector-timeout`, `label-path-mismatch`, `nav-target-404`, `aria-label-changed`.
|
|
118
|
+
- **product-defect** signals: `assertion-failed-on-passing-step`, `unexpected-error-toast`, `wrong-page-after-action`, `network-500-from-app`, `form-validation-rejected-valid-input`.
|
|
119
|
+
|
|
120
|
+
If signals span multiple buckets, pick the most specific one and list all signals in telemetry — the user can override.
|
|
121
|
+
|
|
122
|
+
### Suggestions per bucket
|
|
123
|
+
|
|
124
|
+
Present via `AskUserQuestion`. The first option is the AI's recommendation (label it `(Recommended)`); always include the others so the user can redirect. **Always include "Skip — just report" so the user can opt out without committing to anything.**
|
|
125
|
+
|
|
126
|
+
| Bucket | Recommended suggestion | Other options |
|
|
127
|
+
|---|---|---|
|
|
128
|
+
| **infra** | Report bug to Muggle AI → invoke `muggle-feedback` skill with `category: "muggle-infra"` (and run id / signals). | Retry; muggle-feedback (different category); Skip. |
|
|
129
|
+
| **stale-script** | Regenerate the script → call `muggle-local-execute-test-generation` (or remote equivalent) for this test case, then re-replay. | Retry as-is; muggle-feedback; Skip. |
|
|
130
|
+
| **product-defect** | Surface as a real defect — show the failing step + screenshot + one-line summary; offer to share via email/SMS/file ticket (see "Sharing real defects" below). | muggle-feedback (if the script's *expectation* was wrong); Retry; Skip. |
|
|
131
|
+
|
|
132
|
+
### Sharing real defects
|
|
133
|
+
|
|
134
|
+
When the user picks "share / file ticket" on a `product-defect`:
|
|
135
|
+
|
|
136
|
+
1. Build a one-paragraph summary: test case title, failing step, expected-vs-actual, screenshot path, run id.
|
|
137
|
+
2. Ask `AskUserQuestion`: where to share?
|
|
138
|
+
- Email — open `mailto:` with the summary pre-filled.
|
|
139
|
+
- SMS — copy the summary to clipboard, instruct user to paste.
|
|
140
|
+
- File ticket — if `gh` is available and there's a repo, offer `gh issue create` with the summary as the body; otherwise copy summary and instruct user.
|
|
141
|
+
- Skip — just keep the report.
|
|
142
|
+
3. Record the choice in telemetry (`userAction: "share-email" | "share-sms" | "share-ticket" | "skip"`).
|
|
143
|
+
|
|
144
|
+
### Telemetry
|
|
145
|
+
|
|
146
|
+
Emit **two events per failure**: one when the AI classifies (before asking the user) and one when the user picks an action.
|
|
147
|
+
|
|
148
|
+
```json
|
|
149
|
+
{
|
|
150
|
+
"eventType": "replay-failure-classified",
|
|
151
|
+
"skillName": "<this skill>",
|
|
152
|
+
"aiClassification": "infra" | "stale-script" | "product-defect",
|
|
153
|
+
"aiSuggestion": "report-bug" | "regenerate" | "share-defect",
|
|
154
|
+
"runId": "<local run id>",
|
|
155
|
+
"testCaseId": "<id>",
|
|
156
|
+
"projectId": "<id>",
|
|
157
|
+
"signals": ["element-not-found", "selector-timeout"],
|
|
158
|
+
"metadata": { "failingStep": "<step name>", "errorExcerpt": "<first 200 chars>" }
|
|
159
|
+
}
|
|
160
|
+
```
|
|
161
|
+
|
|
162
|
+
```json
|
|
163
|
+
{
|
|
164
|
+
"eventType": "replay-failure-resolved",
|
|
165
|
+
"skillName": "<this skill>",
|
|
166
|
+
"aiClassification": "<same as above>",
|
|
167
|
+
"aiSuggestion": "<same as above>",
|
|
168
|
+
"userAction": "regenerate" | "report-bug" | "share-email" | "retry" | "muggle-feedback" | "skip",
|
|
169
|
+
"runId": "<local run id>",
|
|
170
|
+
"testCaseId": "<id>",
|
|
171
|
+
"projectId": "<id>"
|
|
172
|
+
}
|
|
173
|
+
```
|
|
174
|
+
|
|
175
|
+
The `(aiSuggestion, userAction)` pair is the metric we tune from — when they diverge, the classifier needs work.
|
|
176
|
+
|
|
177
|
+
---
|
|
178
|
+
|
|
179
|
+
## C. Post-regen failure (used by all four skills with generation)
|
|
180
|
+
|
|
181
|
+
Triggered when `muggle-local-execute-test-generation` (or the remote equivalent) returns `failed`, exit 26, `goal_not_achievable`, or any other non-passing terminal state.
|
|
182
|
+
|
|
183
|
+
### Buckets
|
|
184
|
+
|
|
185
|
+
| Bucket | Meaning |
|
|
186
|
+
|---|---|
|
|
187
|
+
| **transient** | Network blip, single LLM call failed, intermittent flake. Likely succeeds on retry without changing anything. |
|
|
188
|
+
| **infra** | A Muggle Test bug stopped generation from progressing (handler crash, schema validation in our code, deterministic LLM-pipeline failure). |
|
|
189
|
+
| **agent-course** | The generation agent went down a wrong path (chose the wrong button, misread the goal, looped on a blocking modal). The product is fine and the test case is fine — the agent's *course* needs steering. |
|
|
190
|
+
| **product-uxux** | The product itself blocks the test (broken page, missing element, server error). Agent can't proceed because the feature doesn't actually work. |
|
|
191
|
+
|
|
192
|
+
### Initial signal heuristics
|
|
193
|
+
|
|
194
|
+
From `muggle-local-run-result-get` (summary, structured summary, last steps, error):
|
|
195
|
+
|
|
196
|
+
- **transient**: `network-error`, `llm-rate-limit`, `single-tool-call-error`, run had partial progress then died.
|
|
197
|
+
- **infra**: `electron-mcp-handler-crash`, `internal-validation-error`, `pipeline-stuck`, identical failure repeated more than twice.
|
|
198
|
+
- **agent-course**: `goal_not_achievable` with summary mentioning the agent picked a different element, looped on a modal, kept trying the same wrong action; many steps but no real progress.
|
|
199
|
+
- **product-uxux**: server 5xx in step screenshots, "page not found" reached repeatedly, expected element provably absent (visible in screenshot), product clearly broken in the artifact log.
|
|
200
|
+
|
|
201
|
+
### Suggestions per bucket
|
|
202
|
+
|
|
203
|
+
Present via `AskUserQuestion`. **Always show the run summary first** so the user has context, then offer the bucket's recommended action plus alternatives.
|
|
204
|
+
|
|
205
|
+
| Bucket | Recommended suggestion | Other options |
|
|
206
|
+
|---|---|---|
|
|
207
|
+
| **transient** | Retry as-is. | muggle-feedback; Edit test case; Skip. |
|
|
208
|
+
| **infra** | Report bug to Muggle AI → `muggle-feedback` with `category: "muggle-infra"` (run id, signals, summary excerpt). | Retry; Skip. |
|
|
209
|
+
| **agent-course** | Steer the agent → invoke `muggle-feedback` skill so the user describes what should have happened; the workflow re-runs with the corrected course. | Retry; Edit test case; Skip. |
|
|
210
|
+
| **product-uxux** | Wait for fix — share the run summary + screenshot to the dev. Offer email / SMS / file ticket (see "Sharing real defects" in section B). | Retry once the fix lands; muggle-feedback; Skip. |
|
|
211
|
+
|
|
212
|
+
### Telemetry
|
|
213
|
+
|
|
214
|
+
Same two-event pattern as section B, with `eventType` values `regen-failure-classified` and `regen-failure-resolved`. `aiClassification` is one of the four bucket strings; `userAction` is the user's pick.
|
|
215
|
+
|
|
216
|
+
```json
|
|
217
|
+
{
|
|
218
|
+
"eventType": "regen-failure-classified",
|
|
219
|
+
"skillName": "<this skill>",
|
|
220
|
+
"aiClassification": "transient" | "infra" | "agent-course" | "product-uxux",
|
|
221
|
+
"aiSuggestion": "retry" | "report-bug" | "muggle-feedback" | "wait-and-share",
|
|
222
|
+
"runId": "<local run id>",
|
|
223
|
+
"testCaseId": "<id>",
|
|
224
|
+
"projectId": "<id>",
|
|
225
|
+
"signals": ["goal_not_achievable", "loop-on-modal"],
|
|
226
|
+
"metadata": { "summary": "<excerpt>", "stepsCompleted": <n> }
|
|
227
|
+
}
|
|
228
|
+
```
|
|
229
|
+
|
|
230
|
+
---
|
|
231
|
+
|
|
232
|
+
## D. Emitting telemetry — implementation notes
|
|
233
|
+
|
|
234
|
+
- Tool: `muggle-local-telemetry-event-emit` (local-only, fire-and-forget).
|
|
235
|
+
- Sink: `~/.muggle-ai/telemetry/failure-events.jsonl` (one JSON record per line, append-only).
|
|
236
|
+
- Never block the skill on a telemetry call. If the tool errors, log and continue.
|
|
237
|
+
- Always emit the **classified** event before asking the user, and the **resolved** event immediately after the user picks. Don't merge them into one event after the fact — the *latency* between AI suggestion and user choice is also data.
|
|
238
|
+
- For the pre-execution classifier (section A), the event has no separate "resolved" pair unless the user actively overrides; if they do, emit a single follow-up `pre-execution-classification` event with `userAction` set.
|
|
239
|
+
|
|
240
|
+
## E. What this doc deliberately does not bake in
|
|
241
|
+
|
|
242
|
+
- **Quantitative thresholds** beyond rule A.3 (30-day drift). Bucket signal lists above are an initial best guess; once telemetry has data, the user will refine which signals reliably indicate which bucket.
|
|
243
|
+
- **Auto-retry**. There is no "if X then automatically rerun" path anywhere. The user decides every time — that's the whole point of the contract.
|
|
244
|
+
- **Cross-bucket fallbacks**. If the AI mis-classifies, the user picks a different option from the AskUserQuestion list; the AI does not retry classification.
|
|
@@ -0,0 +1,10 @@
|
|
|
1
|
+
# Cleanup After the Change Is Merged
|
|
2
|
+
|
|
3
|
+
Gated by [`autoCleanup`](../muggle-preferences/preference-gates/autoCleanup.md). Follow the standard procedure in [`preference-gates/README.md`](../muggle-preferences/preference-gates/README.md). Fire only after the PR is **merged** — never while it's still open.
|
|
4
|
+
|
|
5
|
+
On `always`, the four steps below run as one pre-authorized sequence (no per-step prompts). Stop on the first failure; do not force.
|
|
6
|
+
|
|
7
|
+
1. `git worktree remove {worktreePath}` — only if a worktree was used.
|
|
8
|
+
2. `git branch -d {branch}` then `git push origin --delete {branch}`.
|
|
9
|
+
3. Clear `.muggle-ai/` session folders for this branch's runs and stale `/tmp/muggle-prepare-*.log` files. Cloud results stay.
|
|
10
|
+
4. Invoke `commit-commands:clean_gone` via the `Skill` tool.
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
# Rebase Onto the Default Branch Before Dev Server / E2E
|
|
2
|
+
|
|
3
|
+
Gated by [`autoRebase`](../muggle-preferences/preference-gates/autoRebase.md). Follow the standard procedure in [`preference-gates/README.md`](../muggle-preferences/preference-gates/README.md).
|
|
4
|
+
|
|
5
|
+
**Fire only when `behind > 0`:**
|
|
6
|
+
|
|
7
|
+
```bash
|
|
8
|
+
git fetch origin
|
|
9
|
+
default=$(git symbolic-ref refs/remotes/origin/HEAD --short | sed 's|origin/||')
|
|
10
|
+
behind=$(git rev-list --count "HEAD..origin/${default}")
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
Pass `{behind}` and `{default}` to the picker prompts. On `always`, run `git rebase origin/${default}`; stop and report on conflicts — never auto-resolve.
|
|
14
|
+
|
|
15
|
+
Stale branches produce false failures and false greens — that's why this gate exists.
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
# Skill Telemetry — Shared Convention
|
|
2
|
+
|
|
3
|
+
At the start of every `muggle-*` skill, call `muggle-local-telemetry-skill-emit` once. Fire-and-forget — never await, never let a telemetry failure block the skill.
|
|
4
|
+
|
|
5
|
+
**Args:**
|
|
6
|
+
|
|
7
|
+
- `skillName`: this skill's name (e.g. `"muggle-test"`)
|
|
8
|
+
- `trigger`: one of
|
|
9
|
+
- `"user-slash"` — user typed `/<skill-name>` (the common case; default this)
|
|
10
|
+
- `"claude-proactive"` — Claude matched on the skill's description, not a slash command
|
|
11
|
+
- `"nested-skill"` — invoked from another skill via the `Skill` tool
|
|
12
|
+
|
|
13
|
+
## Failure-mode events
|
|
14
|
+
|
|
15
|
+
Skills that run replays or generations also emit structured failure-mode events through a separate tool — `muggle-local-telemetry-event-emit`. Don't reinvent the schema or sink in the skill; follow [`failure-mode-handling.md`](./failure-mode-handling.md), which defines the bucket taxonomies, the AI-classify → user-pick contract, and the exact event shape.
|
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
# Worktrees for Dev Work
|
|
2
|
+
|
|
3
|
+
Use this for feature development, local validation, and PR iteration.
|
|
4
|
+
|
|
5
|
+
## Core guidance
|
|
6
|
+
|
|
7
|
+
- **One worktree per branch.** Never switch branches inside a long-lived checkout.
|
|
8
|
+
- **Isolate runtime resources.** Parallel worktrees need unique ports and isolated mutable test state.
|
|
9
|
+
- **Keep worktrees disposable.** Create for focused work, remove after merge.
|
|
10
|
+
|
|
11
|
+
## Start new change work
|
|
12
|
+
|
|
13
|
+
1. Update the base branch (usually `main`).
|
|
14
|
+
2. Create a new worktree and branch:
|
|
15
|
+
```bash
|
|
16
|
+
git worktree add <repo>-worktrees/<slug> -b <branch>
|
|
17
|
+
```
|
|
18
|
+
3. Install dependencies and run setup in that worktree.
|
|
19
|
+
4. Keep all edits and commits for the change in that same worktree.
|
|
20
|
+
|
|
21
|
+
## Test in the worktree
|
|
22
|
+
|
|
23
|
+
- Run lint, typecheck, and unit/integration tests from the worktree.
|
|
24
|
+
- Run your local E2E acceptance flow from that same worktree for user-facing validation.
|
|
25
|
+
- Run the dev server from the same worktree so code, env, and cache all match.
|
|
26
|
+
- In parallel runs, ensure each worktree has a unique port and isolated test data.
|
|
27
|
+
|
|
28
|
+
## E2E + worktree checklist
|
|
29
|
+
|
|
30
|
+
1. Confirm command cwd, checked-out branch, and server source are the same worktree.
|
|
31
|
+
2. Start the app from that worktree and verify readiness first (see [`dev-server-readiness.md`](dev-server-readiness.md)).
|
|
32
|
+
3. Run local E2E acceptance checks in that same session.
|
|
33
|
+
4. Isolate test data across parallel branches (separate users/accounts/orgs or resettable fixtures).
|
|
34
|
+
5. Re-run E2E acceptance checks after meaningful code changes during PR review.
|
|
35
|
+
|
|
36
|
+
## Common failure modes to avoid
|
|
37
|
+
|
|
38
|
+
- **Mixed checkouts:** Test command and server run from different directories.
|
|
39
|
+
- **Port collision:** Multiple branches share a dev-server port.
|
|
40
|
+
- **Shared mutable state:** Parallel runs mutate the same account or fixtures.
|
|
41
|
+
- **Stale server:** Old process still serves previous branch code.
|
|
42
|
+
|
|
43
|
+
## Pull request workflow
|
|
44
|
+
|
|
45
|
+
1. Commit on the branch tied to the worktree.
|
|
46
|
+
2. Push the branch and open the PR.
|
|
47
|
+
3. Keep review fixes in that same worktree.
|
|
48
|
+
4. Re-run relevant checks before each push, including E2E acceptance checks for user-facing changes.
|
|
49
|
+
|
|
50
|
+
## Tear down
|
|
51
|
+
|
|
52
|
+
After merge, remove the worktree.
|
|
53
|
+
See [`post-merge-cleanup.md`](post-merge-cleanup.md).
|
|
@@ -1,6 +1,10 @@
|
|
|
1
|
-
# E2E
|
|
1
|
+
# E2E Acceptance Agent (Stage 6/7)
|
|
2
2
|
|
|
3
|
-
You are
|
|
3
|
+
You are executing E2E acceptance validation for the muggle-do cycle.
|
|
4
|
+
|
|
5
|
+
Stage 6 of `muggle-do`. Runs browser acceptance tests against code changes and records evidence for downstream PR rendering.
|
|
6
|
+
|
|
7
|
+
Standalone subagent (different invocation path, used by `muggle-test` Mode C): [`../../agents/acceptance-tester.md`](../../agents/acceptance-tester.md).
|
|
4
8
|
|
|
5
9
|
## Turn preamble
|
|
6
10
|
|
|
@@ -12,23 +16,29 @@ Start the turn with:
|
|
|
12
16
|
|
|
13
17
|
## Design
|
|
14
18
|
|
|
15
|
-
|
|
19
|
+
This stage is **mode-driven by pre-flight**:
|
|
20
|
+
|
|
21
|
+
- `local-e2e` runs the local browser flow (`test-feature-local` approach).
|
|
22
|
+
- `unit-only` or `skip` does not execute browser runs and must emit an explicit non-pass verdict (`SKIPPED` / `UNIT-ONLY` equivalent in downstream reporting).
|
|
23
|
+
- `staging-replay` is not executed in this stage path and should be surfaced as `INCONCLUSIVE` unless the caller has already routed to a dedicated staging runner.
|
|
24
|
+
|
|
25
|
+
For local runs, the tool boundaries are:
|
|
16
26
|
|
|
17
27
|
| Scope | MCP tools |
|
|
18
28
|
| :---- | :-------- |
|
|
19
29
|
| Cloud (projects, cases, scripts, auth) | `muggle-remote-*` |
|
|
20
30
|
| Local (Electron run, publish, results) | `muggle-local-*` |
|
|
21
31
|
|
|
22
|
-
This
|
|
32
|
+
This keeps execution deterministic: local runs do not depend on cloud replay execution availability.
|
|
23
33
|
|
|
24
34
|
## Input
|
|
25
35
|
|
|
26
36
|
You receive everything from `state.md` already — pre-flight resolved it:
|
|
27
37
|
|
|
28
|
-
- `localUrl` — the
|
|
29
|
-
- `projectId` — the chosen Muggle project
|
|
38
|
+
- `localUrl` — the running validation target URL (typically localhost in local mode)
|
|
39
|
+
- `projectId` — the chosen Muggle Test project
|
|
30
40
|
- The validation strategy (`local-e2e`, `staging-replay`, `unit-only`, `skip`)
|
|
31
|
-
- Test-user credential status (existing / new / skip)
|
|
41
|
+
- Test-user credential status (existing / new / skip), when credentials are needed
|
|
32
42
|
- The list of changed repos, files, and a summary of changes
|
|
33
43
|
- The requirements goal
|
|
34
44
|
|
|
@@ -36,23 +46,41 @@ You receive everything from `state.md` already — pre-flight resolved it:
|
|
|
36
46
|
|
|
37
47
|
### Step 0: Consume pre-flight (no user questions)
|
|
38
48
|
|
|
39
|
-
Read `state.md`.
|
|
49
|
+
Read `state.md`. Resolve [`autoE2ETest`](../muggle-preferences/preference-gates/autoE2ETest.md) first — `always` (default, including when unset) runs this stage. `ask` should already have been resolved by pre-flight Q13. Use the resolved validation mode (`local-e2e`, `staging-replay`, `unit-only`, `skip`) to pick execution vs early-exit behavior.
|
|
40
50
|
|
|
41
|
-
|
|
51
|
+
Use `localUrl`, `projectId`, and `worktreePath` from `state.md`. Missing any → pre-flight bug; escalate with the session path and halt; do not ask the user.
|
|
42
52
|
|
|
43
53
|
### Step 0.5: Pre-flight verification probes
|
|
44
54
|
|
|
45
|
-
Before launching
|
|
55
|
+
Before launching the local runner:
|
|
46
56
|
|
|
47
|
-
1.
|
|
48
|
-
2.
|
|
49
|
-
3.
|
|
50
|
-
4. If test credentials were marked `existing`, confirm the Auth0 tenant in the repo's env matches the tenant the secrets were created under (recorded in `state.md`). Tenant mismatch → halt with "existing secrets target tenant X, local dev targets tenant Y — update pre-flight to collect new credentials."
|
|
57
|
+
1. **Dev-server + backend readiness** — per [`../_shared/dev-server-readiness.md`](../_shared/dev-server-readiness.md) (port + compile log + backend health). Halt on any failure.
|
|
58
|
+
2. **Auth** — `muggle-remote-auth-status` must be `authenticated`; else escalate.
|
|
59
|
+
3. **Identity tenant/domain match** — if test credentials were marked `existing`, confirm the repo's configured identity tenant/domain matches the recorded tenant/domain. Mismatch → halt.
|
|
51
60
|
|
|
52
61
|
### Step 1: Authentication already verified
|
|
53
62
|
|
|
54
63
|
Pre-flight handled auth. If `muggle-remote-auth-status` somehow shows expired here (session clock skew, etc.), re-auth silently via `muggle-remote-auth-login` + `muggle-remote-auth-poll` — but do not ask the user "continue with this account?" again.
|
|
55
64
|
|
|
65
|
+
If validation is `unit-only` or `skip`, emit a `SKIPPED` report with a one-line reason and exit cleanly.
|
|
66
|
+
|
|
67
|
+
If validation is `staging-replay`, emit `INCONCLUSIVE` with reason `staging replay not handled in Stage 6 local runner path` and exit cleanly.
|
|
68
|
+
|
|
69
|
+
### Step 1.5: Placeholder branch detection
|
|
70
|
+
|
|
71
|
+
Read `pathClassification` from the impact-analysis output (emitted by `do/impact-analysis.md`). If it is `none` — i.e. `git diff <default-branch>...HEAD --stat` was empty after rebase — there is no code under test and running test cases would only re-test master. Write a one-paragraph SKIPPED result to the E2E report (or return a SKIPPED verdict to the caller) and exit the stage cleanly. **Do not** synthesize test cases or run anything.
|
|
72
|
+
|
|
73
|
+
### Step 1.7: Route + project classification
|
|
74
|
+
|
|
75
|
+
Consume `pathClassification` from impact-analysis and resolve the dispatch target:
|
|
76
|
+
|
|
77
|
+
- `surface-a` → use the classification-specific route + project mapping defined by impact-analysis output
|
|
78
|
+
- `surface-b` → use the classification-specific route + project mapping defined by impact-analysis output
|
|
79
|
+
- `mixed` → run once per classification mapping (route + project), or surface as INCONCLUSIVE if running both is over the wall-time budget
|
|
80
|
+
- `none` → already handled in Step 1.5
|
|
81
|
+
|
|
82
|
+
The `devServerUrl` and project resolved here override any defaults in `state.md` for the remainder of this stage. Treat classification labels as routing hints provided by impact-analysis; do not hardcode product-specific paths in this stage.
|
|
83
|
+
|
|
56
84
|
### Step 2: Get Test Cases
|
|
57
85
|
|
|
58
86
|
Use `muggle-remote-test-case-list` with the project ID to fetch all test cases.
|
|
@@ -93,7 +121,7 @@ For each relevant test case:
|
|
|
93
121
|
|
|
94
122
|
### Local Execution Timeout (`timeoutMs`)
|
|
95
123
|
|
|
96
|
-
The MCP client often uses a **default wait of 300000 ms (5 minutes)**. **Exploratory script generation** (
|
|
124
|
+
The MCP client often uses a **default wait of 300000 ms (5 minutes)**. **Exploratory script generation** (identity login, multi-step app flows, many LLM iterations) routinely **runs longer than 5 minutes** while Electron is still healthy.
|
|
97
125
|
|
|
98
126
|
- **Always pass `timeoutMs`** — `600000` (10 min) or `900000` (15 min) — unless the test case is known to be simple.
|
|
99
127
|
- If the tool reports **`Electron execution timed out after 300000ms`** but Electron logs show the run still progressing (steps, screenshots, LLM calls), treat it as **orchestration timeout**, not an Electron app defect: **increase `timeoutMs` and retry**.
|
|
@@ -135,7 +163,7 @@ For each published test script:
|
|
|
135
163
|
For each test case:
|
|
136
164
|
- Record pass or fail from the run result
|
|
137
165
|
- If failed, capture the error message, failure step index, and `artifactsDir` for local debugging
|
|
138
|
-
-
|
|
166
|
+
- In `local-e2e` mode, every relevant test case must be executed — generate a new script if none exists (no skips)
|
|
139
167
|
|
|
140
168
|
## Output
|
|
141
169
|
|
|
@@ -160,10 +188,26 @@ For each test case:
|
|
|
160
188
|
- steps: `[{ stepIndex, action, screenshotUrl }, ...]`
|
|
161
189
|
- artifactsDir: `<path>` (for local debugging)
|
|
162
190
|
|
|
191
|
+
**Inconclusive:** (count) — use for runs that couldn't yield a pass/fail signal: no replayable script, environment precondition unmet, infra error, agent stalled on auth/cookie banner before reaching the assertion, missing secrets. The product is **not** implicated — that's `failed`, not `inconclusive`.
|
|
192
|
+
- (test case name):
|
|
193
|
+
- testCaseId: `<id>`
|
|
194
|
+
- runId: `<id>` (synthesize a UUID if no run started)
|
|
195
|
+
- viewUrl: `<url>` (project-level run-results fallback when no specific run URL exists)
|
|
196
|
+
- reason: `<one short sentence>`
|
|
197
|
+
- steps: `[{ stepIndex, action, screenshotUrl }, ...]` (may be empty)
|
|
198
|
+
|
|
163
199
|
**Metadata:**
|
|
164
200
|
- projectId: `<projectId>`
|
|
165
201
|
|
|
166
|
-
**Overall:**
|
|
202
|
+
**Overall:** PASS | FAIL | PARTIAL | INCONCLUSIVE | BLOCKED | SKIPPED — see [`../_shared/failure-mode-handling.md`](../_shared/failure-mode-handling.md) section F for the canonical taxonomy.
|
|
203
|
+
|
|
204
|
+
## Hard constraints
|
|
205
|
+
|
|
206
|
+
- **Do NOT shut down the dev server.** The caller manages dev-server lifecycle.
|
|
207
|
+
- **Do NOT delete or move workspace config/state files** (for example `.muggle-ai/`, `.env.local`, or equivalent runtime config artifacts) in the worktree.
|
|
208
|
+
- **Do NOT call destructive remote MCP tools** — no `*-delete`, `*-revoke`, `*-cancel`, or `*-update` against remote-owned definitions.
|
|
209
|
+
- **One replacement script generation max per stage cycle.**
|
|
210
|
+
- **Honor `wallTimeBudgetSec` from the caller** — on approach, write a PARTIAL report; never silently exceed.
|
|
167
211
|
|
|
168
212
|
## Non-negotiables
|
|
169
213
|
|
|
@@ -171,5 +215,6 @@ For each test case:
|
|
|
171
215
|
- Replay: never hand-build or simplify `actionScript` — only use full response from `muggle-remote-action-script-get`.
|
|
172
216
|
- Always pass `timeoutMs` for execution calls; do not rely on default 5-minute timeout.
|
|
173
217
|
- No hiding failures: surface errors, exit codes, and artifact paths.
|
|
174
|
-
-
|
|
218
|
+
- In `local-e2e` mode, every relevant test case must be executed — generate a new script if none exists (no skips).
|
|
175
219
|
- Always publish after execution to ensure screenshots are cloud-accessible for PR comments.
|
|
220
|
+
- **Never drop a test case from the report because it "couldn't run cleanly."** A test that didn't reach its assertion is `inconclusive`, not absent. Dropping it produces misleading verdicts and pushes downstream PR-comment renderers to hand-write the comment — which is the failure mode this stage exists to prevent.
|
|
@@ -15,22 +15,34 @@ Start the turn with:
|
|
|
15
15
|
You receive:
|
|
16
16
|
- A list of repos with their local filesystem paths
|
|
17
17
|
- The requirements goal and affected repos from the requirements stage
|
|
18
|
+
- Pre-flight state, including `worktreePath` when a worktree was selected (see [`../_shared/use-worktrees.md`](../_shared/use-worktrees.md))
|
|
18
19
|
|
|
19
20
|
## Your Job
|
|
20
21
|
|
|
21
22
|
For each repo path provided:
|
|
22
23
|
|
|
23
|
-
1. **
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
24
|
+
1. **Resolve analysis path with worktree awareness:**
|
|
25
|
+
- If `worktreePath` exists for this repo/session, run all git commands from `worktreePath`.
|
|
26
|
+
- Otherwise run from the repo path.
|
|
27
|
+
- Never mix command cwd across base repo + worktree in one analysis pass.
|
|
28
|
+
2. **Detect worktree context:** Run `git worktree list --porcelain` and determine whether the analysis path is:
|
|
29
|
+
- a linked worktree checkout, or
|
|
30
|
+
- the primary checkout.
|
|
31
|
+
Record this in output as `checkoutType`.
|
|
32
|
+
3. **Check the current branch:** Run `git branch --show-current` in the resolved analysis path. If it returns empty (detached HEAD), report an error for that repo.
|
|
33
|
+
4. **Detect the default branch:** Run `git symbolic-ref refs/remotes/origin/HEAD --short` to find the default branch (e.g., `origin/main`). Strip the `origin/` prefix. If this fails, check if `main` or `master` exist locally via `git rev-parse --verify`.
|
|
34
|
+
5. **Verify it's a feature branch:** The current branch must NOT be the default branch. If it is, report an error.
|
|
35
|
+
6. **List changed files:** Run `git diff --name-only <default-branch>...HEAD` to find files changed on this branch relative to the default branch. If no merge base exists, fall back to `git diff --name-only HEAD`.
|
|
36
|
+
7. **Get the diff:** Run `git diff <default-branch>...HEAD` for the full diff.
|
|
37
|
+
8. **Classify changed paths** to drive downstream test routing — emit `pathClassification` (see "Output state — pathClassification" below).
|
|
28
38
|
|
|
29
39
|
## Output
|
|
30
40
|
|
|
31
41
|
Report per repo:
|
|
32
42
|
|
|
33
43
|
**Repo: (name)**
|
|
44
|
+
- Analysis path: (resolved path used for git commands)
|
|
45
|
+
- Checkout type: `worktree` | `primary`
|
|
34
46
|
- Branch: (current branch name)
|
|
35
47
|
- Default branch: (detected default branch)
|
|
36
48
|
- Changed files: (list)
|
|
@@ -40,3 +52,31 @@ Report per repo:
|
|
|
40
52
|
**Summary:** (which repos have changes, which don't, any errors)
|
|
41
53
|
|
|
42
54
|
If NO repos have any changes, clearly state: "No changes detected in any repo."
|
|
55
|
+
|
|
56
|
+
## Output state — pathClassification
|
|
57
|
+
|
|
58
|
+
Per repo, emit one additional field that the E2E acceptance stage (`do/e2e-acceptance.md` Steps 1.5 and 1.7) consumes to pick routing + project mapping without hardcoding product paths:
|
|
59
|
+
|
|
60
|
+
```
|
|
61
|
+
pathClassification: <surface-a | surface-b | mixed | none>
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
Definitions:
|
|
65
|
+
- **`surface-a`** — all changed files match the repo's configured pattern set A.
|
|
66
|
+
- **`surface-b`** — all changed files match the repo's configured pattern set B.
|
|
67
|
+
- **`mixed`** — at least one file matches pattern set A and at least one file matches pattern set B.
|
|
68
|
+
- **`none`** — empty diff (placeholder branch — see standardized output below).
|
|
69
|
+
|
|
70
|
+
Pattern sets must be treated as repo-configured routing buckets (for example, marketing-vs-app areas), not global hardcoded paths in this stage.
|
|
71
|
+
|
|
72
|
+
### Standardized "no changes" output (placeholder branch)
|
|
73
|
+
|
|
74
|
+
If `git diff <default>...HEAD --stat` is empty for a repo, emit:
|
|
75
|
+
|
|
76
|
+
```
|
|
77
|
+
pathClassification: none
|
|
78
|
+
filesChanged: []
|
|
79
|
+
recommendation: SKIPPED — placeholder branch, no code under test
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
`do/e2e-acceptance.md` Step 1.5 reads this and exits the stage cleanly with a SKIPPED verdict — do not silently run tests against master.
|
|
@@ -41,6 +41,8 @@ You receive:
|
|
|
41
41
|
|
|
42
42
|
For each repo with changes:
|
|
43
43
|
|
|
44
|
+
0. **Apply the `autoCreatePR` gate** per [`../muggle-preferences/preference-gates/README.md`](../muggle-preferences/preference-gates/README.md) + [`autoCreatePR.md`](../muggle-preferences/preference-gates/autoCreatePR.md). On the skip path, record the reason in `result.md` and move to the next repo.
|
|
45
|
+
|
|
44
46
|
1. **Push the branch** to origin: `git push -u origin <branch-name>` in the repo directory.
|
|
45
47
|
2. **Build the PR title:**
|
|
46
48
|
- If E2E acceptance tests have failures: `[E2E FAILING] <goal>`
|
|
@@ -57,10 +59,7 @@ For each repo with changes:
|
|
|
57
59
|
7. **Post the overflow `comment` only if it is non-null.** In the common case, `comment` is `null` and nothing is posted. Never post speculatively.
|
|
58
60
|
|
|
59
61
|
```bash
|
|
60
|
-
gh pr comment <PR#> --body
|
|
61
|
-
<comment field contents>
|
|
62
|
-
EOF
|
|
63
|
-
)"
|
|
62
|
+
jq -r '.comment' /tmp/muggle-pr-section.json | gh pr comment <PR#> --body-file -
|
|
64
63
|
```
|
|
65
64
|
|
|
66
65
|
## Rendering the E2E acceptance block via the shared skill
|
|
@@ -104,3 +103,13 @@ Back in this stage:
|
|
|
104
103
|
- (repo name): comment posted to PR #(number)
|
|
105
104
|
|
|
106
105
|
**Errors:** (any repos where PR creation or comment posting failed, with the error message)
|
|
106
|
+
|
|
107
|
+
## Post-merge cleanup (gated by `autoCleanup`)
|
|
108
|
+
|
|
109
|
+
Cleanup fires in a follow-up turn after merge — never from this stage. See [`../_shared/post-merge-cleanup.md`](../_shared/post-merge-cleanup.md).
|
|
110
|
+
|
|
111
|
+
After printing PR URLs, append one short reminder tied to the gate's current value:
|
|
112
|
+
|
|
113
|
+
- `always` → `Once merged, I'll run the cleanup sequence automatically.`
|
|
114
|
+
- `never` → omit.
|
|
115
|
+
- `ask` / absent → `Once merged, I'll check with you about cleanup.`
|