@muggleai/works 4.2.2 → 4.3.0

package/plugin/skills/do/e2e-acceptance.md

@@ -0,0 +1,161 @@
+# E2E / acceptance agent
+
+You are running **end-to-end (E2E) acceptance** test cases against code changes using Muggle AI's local testing infrastructure. These tests simulate real users in a browser — they are not unit tests.
+
+## Design
+
+E2E acceptance testing runs **locally** using the `test-feature-local` approach:
+
+| Scope | MCP tools |
+| :---- | :-------- |
+| Cloud (projects, cases, scripts, auth) | `muggle-remote-*` |
+| Local (Electron run, publish, results) | `muggle-local-*` |
+
+This guarantees E2E acceptance tests always run — no dependency on cloud replay service availability.
+
+## Input
+
+You receive:
+- The Muggle project ID
+- The list of changed repos, files, and a summary of changes
+- The requirements goal
+- `localUrl` per repo (from `muggle-repos.json`) — the locally running dev server URL
+
+## Your Job
+
+### Step 0: Resolve Local URL
+
+Read `localUrl` for each repo from the context. If it is not provided, ask the user:
+> "E2E acceptance testing requires a running local server. What URL is the `<repo>` app running on? (e.g. `http://localhost:3000`)"
+
+**Do not skip E2E acceptance tests.** Wait for the user to provide the URL before proceeding.
+
+### Step 1: Check Authentication
+
+- `muggle-remote-auth-status`
+- If not signed in: `muggle-remote-auth-login` then `muggle-remote-auth-poll`
+
+Do not skip or assume auth.
+
+### Step 2: Get Test Cases
+
+Use `muggle-remote-test-case-list` with the project ID to fetch all test cases.
+
+### Step 3: Filter Relevant Test Cases
+
+Based on the changed files and the requirements goal, determine which test cases are relevant:
+- Test cases whose use cases directly relate to the changed functionality
+- Test cases that cover areas potentially affected by the changes
+- When in doubt, include the test case (better to over-test than miss a regression)
+
+### Step 4: Execute Tests Locally
+
+For each relevant test case:
+
+1. Call `muggle-remote-test-script-list` filtered by `testCaseId` to check for an existing script.
+
+2. **If a script exists** (replay path):
+   - `muggle-remote-test-script-get` with `testScriptId` → note `actionScriptId`
+   - `muggle-remote-action-script-get` with that id → full `actionScript`
+   - **Use the API response as-is.** Do not edit, shorten, or rebuild `actionScript`; replay needs full `label` paths for element lookup.
+   - `muggle-local-execute-replay` with:
+     - `testScript`: the full script object
+     - `actionScript`: the full action script object (from `muggle-remote-action-script-get`)
+     - `localUrl`: the resolved local URL
+     - `approveElectronAppLaunch`: `true` *(pipeline context — user starting `muggle-do` is implicit approval)*
+     - `timeoutMs`: `600000` (10 min) or `900000` (15 min) for complex flows
+
+3. **If no script exists** (generation path):
+   - `muggle-remote-test-case-get` with `testCaseId` to fetch the full test case object.
+   - `muggle-local-execute-test-generation` with:
+     - `testCase`: the full test case object
+     - `localUrl`: the resolved local URL
+     - `approveElectronAppLaunch`: `true`
+     - `timeoutMs`: `600000` (10 min) or `900000` (15 min) for complex flows
+
+4. When execution completes, call `muggle-local-run-result-get` with the `runId` returned by the execute call.
+
+5. **Retain per test case:** `testCaseId`, `testScriptId` (if present), `runId`, `status` (passed/failed), `artifactsDir`.
+
+### Local Execution Timeout (`timeoutMs`)
+
+The MCP client often uses a **default wait of 300000 ms (5 minutes)**. **Exploratory script generation** (Auth0 login, dashboards, multi-step wizards, many LLM iterations) routinely **runs longer than 5 minutes** while Electron is still healthy.
+
+- **Always pass `timeoutMs`** — `600000` (10 min) or `900000` (15 min) — unless the test case is known to be simple.
+- If the tool reports **`Electron execution timed out after 300000ms`** but Electron logs show the run still progressing (steps, screenshots, LLM calls), treat it as **orchestration timeout**, not an Electron app defect: **increase `timeoutMs` and retry**.
+
+### Interpreting Failures
+
+- **`Electron execution timed out after 300000ms`:** Orchestration wait too short — see `timeoutMs` above.
+- **Exit code 26** (and messages like **LLM failed to generate / replay action script**): Often corresponds to a completed exploration whose **outcome was goal not achievable** (`goal_not_achievable`, summary with `halt`). Use `muggle-local-run-result-get` and read the **summary / structured summary**; do not assume an Electron crash.
+- **Fix for precondition failures:** Choose a project/account that already has the needed state, or narrow the test goal so generation does not try to create resources from scratch unless intentional.
+
+### Step 5: Publish Test Scripts
+
+After each test execution completes (whether pass or fail):
+
+1. Call `muggle-local-publish-test-script` with:
+   - `runId`: the run ID from execution
+   - `cloudTestCaseId`: the test case ID
+
+2. **Retain from publish response:**
+   - `testScriptId`: the cloud test script ID
+   - `viewUrl`: the URL to view the run on muggle-ai.com
+
+This ensures all screenshots are uploaded to the cloud and accessible via URLs for PR comments.
+
+### Step 6: Fetch Screenshot URLs
+
+For each published test script:
+
+1. Call `muggle-remote-test-script-get` with the `testScriptId` from publish.
+
+2. Extract from the response:
+   - `steps[].operation.screenshotUrl`: cloud URL for each step's screenshot
+   - `steps[].operation.action`: the action description for each step
+
+3. **Retain per test case:** array of `{ stepIndex, action, screenshotUrl }`.
+
+### Step 7: Collect Results
+
+For each test case:
+- Record pass or fail from the run result
+- If failed, capture the error message, failure step index, and `artifactsDir` for local debugging
+- Every test case must be executed — generate a new script if none exists (no skips)
+
+## Output
+
+**E2E acceptance report:**
+
+**Passed:** (count)
+- (test case name):
+  - testCaseId: `<id>`
+  - testScriptId: `<id>`
+  - runId: `<id>`
+  - viewUrl: `<url>`
+  - steps: `[{ stepIndex, action, screenshotUrl }, ...]`
+
+**Failed:** (count)
+- (test case name):
+  - testCaseId: `<id>`
+  - testScriptId: `<id>`
+  - runId: `<id>`
+  - viewUrl: `<url>`
+  - failureStepIndex: `<index>`
+  - error: `<message>`
+  - steps: `[{ stepIndex, action, screenshotUrl }, ...]`
+  - artifactsDir: `<path>` (for local debugging)
+
+**Metadata:**
+- projectId: `<projectId>`
+
+**Overall:** ALL PASSED | FAILURES DETECTED
+
+## Non-negotiables
+
+- No silent auth skip; always verify with `muggle-remote-auth-status` first.
+- Replay: never hand-build or simplify `actionScript` — only use full response from `muggle-remote-action-script-get`.
+- Always pass `timeoutMs` for execution calls; do not rely on default 5-minute timeout.
+- No hiding failures: surface errors, exit codes, and artifact paths.
+- Every test case must be executed — generate a new script if none exists (no skips).
+- Always publish after execution to ensure screenshots are cloud-accessible for PR comments.

package/plugin/skills/do/open-prs.md

@@ -7,7 +7,12 @@ You are creating pull requests for each repository that has changes after a succ
 You receive:
 - Per-repo: repo name, path, branch name
 - Requirements: goal, acceptance criteria
-- QA report: passed/failed test cases, each with testCaseId, testScriptId, runId, artifactsDir, and projectId
+- E2E acceptance report: passed/failed test cases, each with:
+  - `testCaseId`, `testScriptId`, `runId`, `projectId`
+  - `viewUrl`: link to view run on muggle-ai.com
+  - `steps`: array of `{ stepIndex, action, screenshotUrl }`
+  - `failureStepIndex` and `error` (if failed)
+  - `artifactsDir` (for local debugging)
 
 ## Your Job
 
@@ -15,38 +20,97 @@ For each repo with changes:
 
 1. **Push the branch** to origin: `git push -u origin <branch-name>` in the repo directory.
 2. **Build the PR title:**
-   - If QA has failures: `[QA FAILING] <goal>`
+   - If E2E acceptance tests have failures: `[E2E FAILING] <goal>`
    - Otherwise: `<goal>`
    - Keep under 70 characters
 3. **Build the PR body** with these sections:
    - `## Goal` — the requirements goal
    - `## Acceptance Criteria` — bulleted list (omit section if empty)
    - `## Changes` — summary of what changed in this repo
-   - `## QA Results` — full test case breakdown (see format below)
+   - `## E2E Acceptance Results` — summary table (see format below)
 4. **Create the PR** using `gh pr create --title "..." --body "..." --head <branch>` in the repo directory.
-5. **Capture the PR URL** from the output.
+5. **Capture the PR URL** and extract the PR number.
+6. **Post E2E acceptance evidence comment** with screenshots (see format below).
 
-## QA Results Section Format
+## E2E acceptance results section format (PR body)
 
-```
-## QA Results
+```markdown
+## E2E Acceptance Results
 
 **X passed / Y failed**
 
 | Test Case | Status | Details |
 |-----------|--------|---------|
-| [Name](https://www.muggle-ai.com/muggleTestV0/dashboard/projects/{projectId}/scripts?modal=details&testCaseId={testCaseId}) | ✅ PASSED | — |
-| [Name](https://www.muggle-ai.com/muggleTestV0/dashboard/projects/{projectId}/scripts?modal=details&testCaseId={testCaseId}) | ❌ FAILED | {error} — artifacts: `{artifactsDir}` |
+| [Name]({viewUrl}) | ✅ PASSED | — |
+| [Name]({viewUrl}) | ❌ FAILED | {error} |
+```
+
+## E2E acceptance evidence comment format
+
+After creating the PR, post a comment with embedded screenshots:
+
+```bash
+gh pr comment <PR#> --body "$(cat <<'EOF'
+## 🧪 E2E acceptance evidence
+
+**X passed / Y failed**
+
+| Test Case | Status | Summary |
+|-----------|--------|---------|
+| [Login Flow]({viewUrl}) | ✅ PASSED | <a href="{lastStepScreenshotUrl}"><img src="{lastStepScreenshotUrl}" width="120"></a> |
+| [Checkout]({viewUrl}) | ❌ FAILED | <a href="{failureStepScreenshotUrl}"><img src="{failureStepScreenshotUrl}" width="120"></a> |
+
+<details>
+<summary>📸 <strong>Login Flow</strong> — 5 steps</summary>
+
+| # | Action | Screenshot |
+|---|--------|------------|
+| 1 | Navigate to `/login` | <a href="{screenshotUrl}"><img src="{screenshotUrl}" width="200"></a> |
+| 2 | Enter username | <a href="{screenshotUrl}"><img src="{screenshotUrl}" width="200"></a> |
+| 3 | Click "Sign In" | <a href="{screenshotUrl}"><img src="{screenshotUrl}" width="200"></a> |
+
+</details>
+
+<details>
+<summary>📸 <strong>Checkout</strong> — 4 steps (failed at step 3)</summary>
+
+| # | Action | Screenshot |
+|---|--------|------------|
+| 1 | Add item to cart | <a href="{screenshotUrl}"><img src="{screenshotUrl}" width="200"></a> |
+| 2 | View cart | <a href="{screenshotUrl}"><img src="{screenshotUrl}" width="200"></a> |
+| 3 ⚠️ | Click confirm — **Element not found** | <a href="{screenshotUrl}"><img src="{screenshotUrl}" width="200"></a> |
+
+</details>
+EOF
+)"
 ```
 
-Rules:
-- Link each test case name to its details page on www.muggle-ai.com using the URL pattern above (requires `testCaseId` and `projectId` from the QA report).
-- For failed tests, include the error message and the local `artifactsDir` path so the developer can inspect screenshots.
-- Screenshots are in `{artifactsDir}/screenshots/` and viewable locally.
+### Comment Building Rules
+
+1. **Summary table:**
+   - Show thumbnail (120px) of **last step** for passed tests
+   - Show thumbnail of **failure step** for failed tests
+   - Thumbnail links to full-size image
+
+2. **Collapsible details per test case:**
+   - Show all steps with 200px thumbnails
+   - Mark failure step with ⚠️ and inline error message
+   - Include step count in summary line
+
+3. **HTML for thumbnails:**
+   - Use `<a href="{url}"><img src="{url}" width="N"></a>` for clickable thumbnails
+   - 120px width in summary table, 200px in details
+
+4. **All tests get screenshots:**
+   - Passing tests show proof of success
+   - Failing tests highlight the failure point
 
 ## Output
 
 **PRs Created:**
 - (repo name): (PR URL)
 
-**Errors:** (any repos where PR creation failed, with the error message)
+**E2E acceptance evidence comments posted:**
+- (repo name): comment posted to PR #(number)
+
+**Errors:** (any repos where PR creation or comment posting failed, with the error message)

package/plugin/skills/muggle/SKILL.md

@@ -12,7 +12,8 @@ Use this as the top-level Muggle command router.
 When user asks for "muggle" with no specific subcommand, show this command set:
 
 - `/muggle:muggle-do` — autonomous dev pipeline
-- `/muggle:muggle-test-feature-local` — local feature QA
+- `/muggle:muggle-test` — change-driven E2E acceptance testing (local or remote, with PR posting)
+- `/muggle:muggle-test-feature-local` — local feature E2E acceptance testing
 - `/muggle:muggle-status` — health check
 - `/muggle:muggle-repair` — repair broken installation
 - `/muggle:muggle-upgrade` — upgrade local installation
@@ -24,7 +25,8 @@ If the user intent clearly matches one command, route to that command behavior:
 - status/health/check -> `muggle-status`
 - repair/fix/install broken -> `muggle-repair`
 - upgrade/update latest -> `muggle-upgrade`
-- test localhost/validate feature -> `muggle-test-feature-local`
+- test my changes/acceptance test my work/test before push/post E2E acceptance results to PR/test on staging/test on preview -> `muggle-test`
+- test localhost/validate single feature -> `muggle-test-feature-local`
 - build/implement from request -> `muggle-do`
 
 If intent is ambiguous, ask one concise clarification question.

package/plugin/skills/muggle-do/SKILL.md

@@ -6,9 +6,9 @@ disable-model-invocation: true
 
 # Muggle Do
 
-Muggle Do is the top-level command for the Muggle AI development workflow.
+Muggle Do is the command for the Muggle AI development workflow.
 
-It runs the autonomous dev cycle: requirements -> impact analysis -> validate code -> coding -> unit tests -> QA -> open PRs.
+It runs a battle-tested autonomous dev cycle: requirements -> impact analysis -> validate code -> coding -> unit tests -> E2E acceptance tests -> open PRs.
 
 For maintenance tasks, use the dedicated skills:
 
@@ -42,12 +42,12 @@ Use the supporting files in the `../do/` directory as stage-specific instruction
 - [impact-analysis.md](../do/impact-analysis.md)
 - [validate-code.md](../do/validate-code.md)
 - [unit-tests.md](../do/unit-tests.md)
-- [qa.md](../do/qa.md)
+- [e2e-acceptance.md](../do/e2e-acceptance.md)
 - [open-prs.md](../do/open-prs.md)
 
 ## Guardrails
 
-- Do not skip unit tests before QA.
-- Do not skip QA due to missing scripts; generate when needed.
+- Do not skip unit tests before E2E acceptance tests.
+- Do not skip E2E acceptance tests due to missing scripts; generate when needed.
 - If the same stage fails 3 times in a row, escalate with details.
-- If total iterations reach 3 and QA still fails, continue to PR creation with `[QA FAILING]`.
+- If total iterations reach 3 and E2E acceptance tests still fail, continue to PR creation with `[E2E FAILING]`.