@muggleai/works 4.2.2 → 4.4.0

package/plugin/skills/do/e2e-acceptance.md

@@ -0,0 +1,161 @@
+# E2E / acceptance agent
+
+You are running **end-to-end (E2E) acceptance** test cases against code changes using Muggle AI's local testing infrastructure. These tests simulate real users in a browser — they are not unit tests.
+
+## Design
+
+E2E acceptance testing runs **locally** using the `test-feature-local` approach:
+
+| Scope | MCP tools |
+| :---- | :-------- |
+| Cloud (projects, cases, scripts, auth) | `muggle-remote-*` |
+| Local (Electron run, publish, results) | `muggle-local-*` |
+
+This guarantees E2E acceptance tests always run — no dependency on cloud replay service availability.
+
+## Input
+
+You receive:
+- The Muggle project ID
+- The list of changed repos, files, and a summary of changes
+- The requirements goal
+- `localUrl` per repo (from `muggle-repos.json`) — the locally running dev server URL
+
+## Your Job
+
+### Step 0: Resolve Local URL
+
+Read `localUrl` for each repo from the context. If it is not provided, ask the user:
+> "E2E acceptance testing requires a running local server. What URL is the `<repo>` app running on? (e.g. `http://localhost:3000`)"
+
+**Do not skip E2E acceptance tests.** Wait for the user to provide the URL before proceeding.
+
+### Step 1: Check Authentication
+
+- `muggle-remote-auth-status`
+- If not signed in: `muggle-remote-auth-login` then `muggle-remote-auth-poll`
+
+Do not skip or assume auth.
+
+### Step 2: Get Test Cases
+
+Use `muggle-remote-test-case-list` with the project ID to fetch all test cases.
+
+### Step 3: Filter Relevant Test Cases
+
+Based on the changed files and the requirements goal, determine which test cases are relevant:
+- Test cases whose use cases directly relate to the changed functionality
+- Test cases that cover areas potentially affected by the changes
+- When in doubt, include the test case (better to over-test than miss a regression)
+
+### Step 4: Execute Tests Locally
+
+For each relevant test case:
+
+1. Call `muggle-remote-test-script-list` filtered by `testCaseId` to check for an existing script.
+
+2. **If a script exists** (replay path):
+   - `muggle-remote-test-script-get` with `testScriptId` → note `actionScriptId`
+   - `muggle-remote-action-script-get` with that id → full `actionScript`
+   - **Use the API response as-is.** Do not edit, shorten, or rebuild `actionScript`; replay needs full `label` paths for element lookup.
+   - `muggle-local-execute-replay` with:
+     - `testScript`: the full script object
+     - `actionScript`: the full action script object (from `muggle-remote-action-script-get`)
+     - `localUrl`: the resolved local URL
+     - `approveElectronAppLaunch`: `true` *(pipeline context — user starting `muggle-do` is implicit approval)*
+     - `timeoutMs`: `600000` (10 min) or `900000` (15 min) for complex flows
+
+3. **If no script exists** (generation path):
+   - `muggle-remote-test-case-get` with `testCaseId` to fetch the full test case object.
+   - `muggle-local-execute-test-generation` with:
+     - `testCase`: the full test case object
+     - `localUrl`: the resolved local URL
+     - `approveElectronAppLaunch`: `true`
+     - `timeoutMs`: `600000` (10 min) or `900000` (15 min) for complex flows
+
+4. When execution completes, call `muggle-local-run-result-get` with the `runId` returned by the execute call.
+
+5. **Retain per test case:** `testCaseId`, `testScriptId` (if present), `runId`, `status` (passed/failed), `artifactsDir`.
+
+### Local Execution Timeout (`timeoutMs`)
+
+The MCP client often uses a **default wait of 300000 ms (5 minutes)**. **Exploratory script generation** (Auth0 login, dashboards, multi-step wizards, many LLM iterations) routinely **runs longer than 5 minutes** while Electron is still healthy.
+
+- **Always pass `timeoutMs`** — `600000` (10 min) or `900000` (15 min) — unless the test case is known to be simple.
+- If the tool reports **`Electron execution timed out after 300000ms`** but Electron logs show the run still progressing (steps, screenshots, LLM calls), treat it as **orchestration timeout**, not an Electron app defect: **increase `timeoutMs` and retry**.
+
+### Interpreting Failures
+
+- **`Electron execution timed out after 300000ms`:** Orchestration wait too short — see `timeoutMs` above.
+- **Exit code 26** (and messages like **LLM failed to generate / replay action script**): Often corresponds to a completed exploration whose **outcome was goal not achievable** (`goal_not_achievable`, summary with `halt`). Use `muggle-local-run-result-get` and read the **summary / structured summary**; do not assume an Electron crash.
+- **Fix for precondition failures:** Choose a project/account that already has the needed state, or narrow the test goal so generation does not try to create resources from scratch unless intentional.
+
+### Step 5: Publish Test Scripts
+
+After each test execution completes (whether pass or fail):
+
+1. Call `muggle-local-publish-test-script` with:
+   - `runId`: the run ID from execution
+   - `cloudTestCaseId`: the test case ID
+
+2. **Retain from publish response:**
+   - `testScriptId`: the cloud test script ID
+   - `viewUrl`: the URL to view the run on muggle-ai.com
+
+This ensures all screenshots are uploaded to the cloud and accessible via URLs for PR comments.
+
+### Step 6: Fetch Screenshot URLs
+
+For each published test script:
+
+1. Call `muggle-remote-test-script-get` with the `testScriptId` from publish.
+
+2. Extract from the response:
+   - `steps[].operation.screenshotUrl`: cloud URL for each step's screenshot
+   - `steps[].operation.action`: the action description for each step
+
+3. **Retain per test case:** array of `{ stepIndex, action, screenshotUrl }`.
+
+### Step 7: Collect Results
+
+For each test case:
+- Record pass or fail from the run result
+- If failed, capture the error message, failure step index, and `artifactsDir` for local debugging
+- Every test case must be executed — generate a new script if none exists (no skips)
+
+## Output
+
+**E2E acceptance report:**
+
+**Passed:** (count)
+- (test case name):
+  - testCaseId: `<id>`
+  - testScriptId: `<id>`
+  - runId: `<id>`
+  - viewUrl: `<url>`
+  - steps: `[{ stepIndex, action, screenshotUrl }, ...]`
+
+**Failed:** (count)
+- (test case name):
+  - testCaseId: `<id>`
+  - testScriptId: `<id>`
+  - runId: `<id>`
+  - viewUrl: `<url>`
+  - failureStepIndex: `<index>`
+  - error: `<message>`
+  - steps: `[{ stepIndex, action, screenshotUrl }, ...]`
+  - artifactsDir: `<path>` (for local debugging)
+
+**Metadata:**
+- projectId: `<projectId>`
+
+**Overall:** ALL PASSED | FAILURES DETECTED
+
+## Non-negotiables
+
+- No silent auth skip; always verify with `muggle-remote-auth-status` first.
+- Replay: never hand-build or simplify `actionScript` — only use full response from `muggle-remote-action-script-get`.
+- Always pass `timeoutMs` for execution calls; do not rely on default 5-minute timeout.
+- No hiding failures: surface errors, exit codes, and artifact paths.
+- Every test case must be executed — generate a new script if none exists (no skips).
+- Always publish after execution to ensure screenshots are cloud-accessible for PR comments.

package/plugin/skills/do/open-prs.md

@@ -7,7 +7,12 @@ You are creating pull requests for each repository that has changes after a succ
 You receive:
 - Per-repo: repo name, path, branch name
 - Requirements: goal, acceptance criteria
-- QA report: passed/failed test cases, each with testCaseId, testScriptId, runId, artifactsDir, and projectId
+- E2E acceptance report: passed/failed test cases, each with:
+  - `testCaseId`, `testScriptId`, `runId`, `projectId`
+  - `viewUrl`: link to view run on muggle-ai.com
+  - `steps`: array of `{ stepIndex, action, screenshotUrl }`
+  - `failureStepIndex` and `error` (if failed)
+  - `artifactsDir` (for local debugging)
 
 ## Your Job
 
@@ -15,38 +20,103 @@ For each repo with changes:
 
 1. **Push the branch** to origin: `git push -u origin <branch-name>` in the repo directory.
 2. **Build the PR title:**
-   - If QA has failures: `[QA FAILING] <goal>`
+   - If E2E acceptance tests have failures: `[E2E FAILING] <goal>`
    - Otherwise: `<goal>`
    - Keep under 70 characters
 3. **Build the PR body** with these sections:
    - `## Goal` — the requirements goal
    - `## Acceptance Criteria` — bulleted list (omit section if empty)
    - `## Changes` — summary of what changed in this repo
-   - `## QA Results` — full test case breakdown (see format below)
+   - E2E acceptance evidence block from `muggle build-pr-section` (see "Rendering the E2E acceptance results block" below)
 4. **Create the PR** using `gh pr create --title "..." --body "..." --head <branch>` in the repo directory.
-5. **Capture the PR URL** from the output.
+5. **Capture the PR URL** and extract the PR number.
+6. **Post the overflow comment only if `muggle build-pr-section` emitted one** (see "Rendering the E2E acceptance results block" below). In the common case, no comment is posted.
 
-## QA Results Section Format
+## Rendering the E2E acceptance results block
 
+Do **not** hand-write the `## E2E Acceptance Results` markdown. Use the `muggle build-pr-section` CLI, which renders a deterministic block and decides whether the evidence fits in the PR description or needs to spill into an overflow comment.
+
+### Step A: Build the report JSON
+
+Assemble the e2e-acceptance report you collected in `e2e-acceptance.md` into a JSON object with this shape:
+
+```json
+{
+  "projectId": "<project UUID>",
+  "tests": [
+    {
+      "name": "<test case name>",
+      "testCaseId": "<UUID>",
+      "testScriptId": "<UUID or omitted>",
+      "runId": "<UUID>",
+      "viewUrl": "<muggle-ai.com run URL>",
+      "status": "passed",
+      "steps": [
+        { "stepIndex": 0, "action": "<action>", "screenshotUrl": "<URL>" }
+      ]
+    },
+    {
+      "name": "<test case name>",
+      "testCaseId": "<UUID>",
+      "runId": "<UUID>",
+      "viewUrl": "<muggle-ai.com run URL>",
+      "status": "failed",
+      "failureStepIndex": 2,
+      "error": "<error message>",
+      "artifactsDir": "<path, optional>",
+      "steps": [
+        { "stepIndex": 0, "action": "<action>", "screenshotUrl": "<URL>" }
+      ]
+    }
+  ]
+}
 ```
-## QA Results
 
-**X passed / Y failed**
+### Step B: Render the evidence block
+
+Pipe the JSON into `muggle build-pr-section`. It writes `{ "body": "...", "comment": "..." | null }` to stdout:
 
-| Test Case | Status | Details |
-|-----------|--------|---------|
-| [Name](https://www.muggle-ai.com/muggleTestV0/dashboard/projects/{projectId}/scripts?modal=details&testCaseId={testCaseId}) | ✅ PASSED | — |
-| [Name](https://www.muggle-ai.com/muggleTestV0/dashboard/projects/{projectId}/scripts?modal=details&testCaseId={testCaseId}) | ❌ FAILED | {error} — artifacts: `{artifactsDir}` |
+```bash
+echo "$REPORT_JSON" | muggle build-pr-section > /tmp/muggle-pr-section.json
 ```
 
-Rules:
-- Link each test case name to its details page on www.muggle-ai.com using the URL pattern above (requires `testCaseId` and `projectId` from the QA report).
-- For failed tests, include the error message and the local `artifactsDir` path so the developer can inspect screenshots.
-- Screenshots are in `{artifactsDir}/screenshots/` and viewable locally.
+The command exits nonzero on malformed input and writes a descriptive error to stderr — do not swallow that error, surface it to the user.
+
+### Step C: Build the PR body
+
+Build the PR body by concatenating, in order:
+
+- `## Goal` — the requirements goal
+- `## Acceptance Criteria` — bulleted list (omit section if empty)
+- `## Changes` — summary of what changed in this repo
+- The `body` field from the CLI output (already contains its own `## E2E Acceptance Results` header)
+
+### Step D: Create the PR, then post the overflow comment only if present
+
+1. Create the PR with `gh pr create --title "..." --body "..." --head <branch>`.
+2. Capture the PR URL and extract the PR number.
+3. If the CLI output's `comment` field is `null`, **do not post a comment** — everything is already in the PR description.
+4. If the CLI output's `comment` field is a non-null string, post it as a follow-up comment:
+
+   ```bash
+   gh pr comment <PR#> --body "$(cat <<'EOF'
+   <comment field contents>
+   EOF
+   )"
+   ```
+
+### Notes on fit vs. overflow
+
+- **The common case is fit**: the full evidence (summary, per-test rows, collapsible failure details) lives in the PR description, no comment is posted.
+- **The overflow case** is triggered automatically when the full inline body would exceed the CLI's budget. In that case the PR description contains the summary, the per-test rows, and a pointer line; the full step-by-step failure details live in the follow-up comment.
+- You do not make the fit-vs-overflow decision — the CLI does. Never post the comment speculatively.
 
 ## Output
 
 **PRs Created:**
 - (repo name): (PR URL)
 
-**Errors:** (any repos where PR creation failed, with the error message)
+**E2E acceptance overflow comments posted:** (only include repos where an overflow comment was actually posted)
+- (repo name): comment posted to PR #(number)
+
+**Errors:** (any repos where PR creation or comment posting failed, with the error message)

package/plugin/skills/muggle/SKILL.md

@@ -9,22 +9,24 @@ Use this as the top-level Muggle command router.
 
 ## Menu
 
-When user asks for "muggle" with no specific subcommand, show this command set:
+When user asks for "muggle" with no specific subcommand, use `AskQuestion` to present the command set as clickable options:
 
-- `/muggle:muggle-do` — autonomous dev pipeline
-- `/muggle:muggle-test-feature-local` — local feature QA
-- `/muggle:muggle-status` — health check
-- `/muggle:muggle-repair` — repair broken installation
-- `/muggle:muggle-upgrade` — upgrade local installation
+- "Test my changes — change-driven E2E acceptance testing (local or remote)" → `muggle-test`
+- "Test a feature on localhost — run a single E2E test locally" → `muggle-test-feature-local`
+- "Autonomous dev pipeline — requirements to PR" → `muggle-do`
+- "Health check — verify installation status" → `muggle-status`
+- "Repair — fix broken installation" → `muggle-repair`
+- "Upgrade — update to latest version" → `muggle-upgrade`
 
 ## Routing
 
-If the user intent clearly matches one command, route to that command behavior:
+If the user intent clearly matches one command, route directly — no menu needed:
 
-- status/health/check -> `muggle-status`
-- repair/fix/install broken -> `muggle-repair`
-- upgrade/update latest -> `muggle-upgrade`
-- test localhost/validate feature -> `muggle-test-feature-local`
-- build/implement from request -> `muggle-do`
+- status/health/check → `muggle-status`
+- repair/fix/install broken → `muggle-repair`
+- upgrade/update latest → `muggle-upgrade`
+- test my changes/acceptance test my work/test before push/post E2E acceptance results to PR/test on staging/test on preview → `muggle-test`
+- test localhost/validate single feature → `muggle-test-feature-local`
+- build/implement from request → `muggle-do`
 
-If intent is ambiguous, ask one concise clarification question.
+If intent is ambiguous, use `AskQuestion` with the most likely options rather than asking the user to type a clarification.

package/plugin/skills/muggle-do/SKILL.md

@@ -6,9 +6,9 @@ disable-model-invocation: true
 
 # Muggle Do
 
-Muggle Do is the top-level command for the Muggle AI development workflow.
+Muggle Do is the command for the Muggle AI development workflow.
 
-It runs the autonomous dev cycle: requirements -> impact analysis -> validate code -> coding -> unit tests -> QA -> open PRs.
+It runs a battle-tested autonomous dev cycle: requirements -> impact analysis -> validate code -> coding -> unit tests -> E2E acceptance tests -> open PRs.
 
 For maintenance tasks, use the dedicated skills:
 
@@ -42,12 +42,12 @@ Use the supporting files in the `../do/` directory as stage-specific instruction
 - [impact-analysis.md](../do/impact-analysis.md)
 - [validate-code.md](../do/validate-code.md)
 - [unit-tests.md](../do/unit-tests.md)
-- [qa.md](../do/qa.md)
+- [e2e-acceptance.md](../do/e2e-acceptance.md)
 - [open-prs.md](../do/open-prs.md)
 
 ## Guardrails
 
-- Do not skip unit tests before QA.
-- Do not skip QA due to missing scripts; generate when needed.
+- Do not skip unit tests before E2E acceptance tests.
+- Do not skip E2E acceptance tests due to missing scripts; generate when needed.
 - If the same stage fails 3 times in a row, escalate with details.
-- If total iterations reach 3 and QA still fails, continue to PR creation with `[QA FAILING]`.
+- If total iterations reach 3 and E2E acceptance tests still fail, continue to PR creation with `[E2E FAILING]`.