@muggleai/works 5.28.0-staging.249 → 5.28.0-staging.251
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/dist/{chunk-HBCASFGN.js → chunk-JRYEMMKU.js} +39 -0
- package/dist/{chunk-USALT4LJ.js → chunk-XEFALLR5.js} +27 -48
- package/dist/cli.js +2 -2
- package/dist/index.js +2 -2
- package/dist/plugin/.claude-plugin/plugin.json +1 -1
- package/dist/plugin/.codex-mcp.json +1 -1
- package/dist/plugin/.codex-plugin/plugin.json +1 -1
- package/dist/plugin/.cursor-plugin/plugin.json +1 -1
- package/dist/plugin/.mcp.json +1 -1
- package/dist/plugin/agents/acceptance-tester.md +4 -0
- package/dist/plugin/skills/_shared/deliverable-categories/audio.md +29 -0
- package/dist/plugin/skills/_shared/deliverable-categories/data.md +31 -0
- package/dist/plugin/skills/_shared/deliverable-categories/image.md +34 -0
- package/dist/plugin/skills/_shared/deliverable-categories/research.md +28 -0
- package/dist/plugin/skills/_shared/deliverable-categories/video.md +32 -0
- package/dist/plugin/skills/_shared/deliverable-categories/web.md +25 -0
- package/dist/plugin/skills/_shared/deliverable-categories.md +69 -0
- package/dist/plugin/skills/_shared/dev-loop/failures.md +2 -1
- package/dist/plugin/skills/_shared/failure-mode-handling.md +5 -0
- package/dist/plugin/skills/do/e2e-acceptance.md +14 -3
- package/dist/plugin/skills/do/e2e-repair.md +3 -1
- package/dist/plugin/skills/do/impact-analysis.md +17 -3
- package/dist/plugin/skills/do/open-prs/forward.md +2 -2
- package/dist/plugin/skills/do/pre-flight.md +1 -1
- package/dist/plugin/skills/do/requirements.md +3 -3
- package/dist/plugin/skills/muggle-do/SKILL.md +2 -1
- package/dist/plugin/skills/muggle-pr-visual-walkthrough/e2e-report-assembly.md +2 -0
- package/dist/release-manifest.json +3 -3
- package/dist/{src-ZOCXV6PT.js → src-7OY7RXTE.js} +1 -1
- package/package.json +1 -1
- package/plugin/.claude-plugin/plugin.json +1 -1
- package/plugin/.codex-mcp.json +1 -1
- package/plugin/.codex-plugin/plugin.json +1 -1
- package/plugin/.cursor-plugin/plugin.json +1 -1
- package/plugin/.mcp.json +1 -1
- package/plugin/agents/acceptance-tester.md +4 -0
- package/plugin/skills/_shared/deliverable-categories/audio.md +29 -0
- package/plugin/skills/_shared/deliverable-categories/data.md +31 -0
- package/plugin/skills/_shared/deliverable-categories/image.md +34 -0
- package/plugin/skills/_shared/deliverable-categories/research.md +28 -0
- package/plugin/skills/_shared/deliverable-categories/video.md +32 -0
- package/plugin/skills/_shared/deliverable-categories/web.md +25 -0
- package/plugin/skills/_shared/deliverable-categories.md +69 -0
- package/plugin/skills/_shared/dev-loop/failures.md +2 -1
- package/plugin/skills/_shared/failure-mode-handling.md +5 -0
- package/plugin/skills/do/e2e-acceptance.md +14 -3
- package/plugin/skills/do/e2e-repair.md +3 -1
- package/plugin/skills/do/impact-analysis.md +17 -3
- package/plugin/skills/do/open-prs/forward.md +2 -2
- package/plugin/skills/do/pre-flight.md +1 -1
- package/plugin/skills/do/requirements.md +3 -3
- package/plugin/skills/muggle-do/SKILL.md +2 -1
- package/plugin/skills/muggle-pr-visual-walkthrough/e2e-report-assembly.md +2 -0
|
@@ -73,7 +73,7 @@ When other work is in front of the session as this change reaches Stage 6, the m
|
|
|
73
73
|
|
|
74
74
|
### Step 0.5: Readiness check
|
|
75
75
|
|
|
76
|
-
A `local-e2e` cycle runs these before any `muggle-local-execute-*` call, every cycle — even when pre-flight already reported the app ready, since a server can die while stages 2–5 run and a healthy one fast-exits:
|
|
76
|
+
A scope with deliverable checks and no browser case runs none of these and goes to Step 5. Otherwise a `local-e2e` cycle runs these before any `muggle-local-execute-*` call, every cycle — even when pre-flight already reported the app ready, since a server can die while stages 2–5 run and a healthy one fast-exits:
|
|
77
77
|
|
|
78
78
|
1. **Dev-server + backend readiness** — invoke [`muggle-test-prepare`](../muggle-test-prepare/SKILL.md) from the cycle's worktree, passing the PR target as `base` — the readiness/service-start owner (idempotent fast-exit when healthy; probes via `dev-server-readiness.md` and starts what is down). It syncs the worktree onto the latest base before starting anything, and serves every other repo from its latest default branch. Its readiness table must show the cycle's worktree as the `Source` of the changed repo's services, at the worktree's current HEAD; any other source is not ready. Still not ready after its start attempts → the readiness halt below.
|
|
79
79
|
2. **Auth** — `muggle-remote-auth-status` must be `authenticated`; else escalate.
|
|
@@ -87,7 +87,7 @@ Only the app not answering is this halt. A run the runner itself lost while read
|
|
|
87
87
|
|
|
88
88
|
Pre-flight handled auth. If `muggle-remote-auth-status` somehow shows expired here (session clock skew, etc.), re-auth silently via `muggle-remote-auth-login` + `muggle-remote-auth-poll` — but do not ask the user "continue with this account?" again.
|
|
89
89
|
|
|
90
|
-
If validation is `unit-only` or `skip`, emit a `SKIPPED` report citing the code that holds — `NO_WEB_SURFACE` when the repo has nothing to drive, `DEV_SERVER_UNREACHABLE` when it does but nothing answers — and exit cleanly. This is the only exit `DEV_SERVER_UNREACHABLE` belongs to, because pre-flight never planned a browser run; on a `local-e2e` cycle an unreachable server is the Step 0.5 readiness halt.
|
|
90
|
+
If validation is `unit-only` or `skip`, the scope's test cases — the criteria whose category flow is **Verified by** test cases ([`../_shared/deliverable-categories.md`](../_shared/deliverable-categories.md#category-flows)) — get no run: emit a `SKIPPED` report citing the code that holds — `NO_WEB_SURFACE` when the repo has nothing to drive, `DEV_SERVER_UNREACHABLE` when it does but nothing answers — and exit cleanly. Every other criterion follows its own flow: under `unit-only`, a scope that also holds deliverable checks records that code as the report's `**Browser skip:**` line instead and goes straight to Step 5 rather than exiting. This is the only exit `DEV_SERVER_UNREACHABLE` belongs to, because pre-flight never planned a browser run; on a `local-e2e` cycle an unreachable server is the Step 0.5 readiness halt.
|
|
91
91
|
|
|
92
92
|
If validation is `staging-replay`, emit `INCONCLUSIVE` with reason `staging replay not handled in Stage 6 local runner path` and exit cleanly.
|
|
93
93
|
|
|
@@ -141,11 +141,18 @@ Inputs to the loop: `mode` from the script-exists check, `localUrl`/project from
|
|
|
141
141
|
|
|
142
142
|
Each record's run is its own report entry carrying its `runSettingsId` and `resolutionLabel`. A record skipped because its primary failed is an Inconclusive entry with reason `primary failed`, not a skipped test case.
|
|
143
143
|
|
|
144
|
+
### Step 5: Run the deliverable checks
|
|
145
|
+
|
|
146
|
+
Run every `deliverableChecks` entry of the scope per [`../_shared/deliverable-categories.md`](../_shared/deliverable-categories.md): obtain the artifact from the working tree recorded in `state.md`, run each planned probe plus the category's review, and record one report entry per check in the **Deliverables** block. Probe output, extracted frames, and transcripts go in a scratch directory outside the worktree, recorded as the entry's `evidenceDir`. A check that could not run is an `INCONCLUSIVE` entry with its reason, never absent.
|
|
147
|
+
|
|
148
|
+
Deliverable checks do not satisfy the session's browser E2E gate. A cycle that ran no browser case still declares its browser skip with `echo "MUGGLE_E2E_SKIP: <CODE>: <detail>"`: `NO_WEB_SURFACE` when that holds, else `NOT_APPLICABLE` with a detail naming the categories checked instead, e.g. `NOT_APPLICABLE: change ships only video and image deliverables, verified by deliverable checks DC1-DC2`.
|
|
149
|
+
|
|
144
150
|
### Step 7: Collect Results
|
|
145
151
|
|
|
146
152
|
For each test case:
|
|
147
153
|
- Record pass or fail from the run result
|
|
148
154
|
- If failed, capture the error message, failure step index, and `artifactsDir` for local debugging
|
|
155
|
+
- A studio verdict of `bug_detected` (`**Studio Verdict:** Bug Detected (bug_detected)` on `muggle-local-run-result-get`) is failed: it is a product bug the run proved. Surface it to the user as a bug in their app; do not retry, regenerate, or debug the test
|
|
149
156
|
- In `local-e2e` mode, every scoped test case must be executed — generate a new script if none exists (no skips)
|
|
150
157
|
|
|
151
158
|
## Output
|
|
@@ -185,6 +192,10 @@ For each test case:
|
|
|
185
192
|
- reason: `<one short sentence>`
|
|
186
193
|
- steps: `[{ stepIndex, action, description, screenshotUrl }, ...]` (empty only when the run produced no steps)
|
|
187
194
|
|
|
195
|
+
**Deliverables:** (count) — one entry per deliverable check from Step 5, in the report-entry shape of [`../_shared/deliverable-categories.md`](../_shared/deliverable-categories.md#report-entry). Omitted when the scope planned none.
|
|
196
|
+
|
|
197
|
+
**Browser skip:** `<CODE>: <detail>` — only when Step 1 skipped the browser part of a cycle that still ran deliverable checks.
|
|
198
|
+
|
|
188
199
|
**Scope:** (from `e2eScope`, after Step 3)
|
|
189
200
|
- impacted: `<testCaseId>`, ...
|
|
190
201
|
- new: `<key>` → `<testCaseId>`, ...
|
|
@@ -194,7 +205,7 @@ For each test case:
|
|
|
194
205
|
**Metadata:**
|
|
195
206
|
- projectId: `<projectId>`
|
|
196
207
|
|
|
197
|
-
**Overall:** PASS | FAIL | PARTIAL | INCONCLUSIVE | BLOCKED | SKIPPED — see [`../_shared/failure-mode-handling.md`](../_shared/failure-mode-handling.md) section F for the canonical taxonomy.
|
|
208
|
+
**Overall:** PASS | FAIL | PARTIAL | INCONCLUSIVE | BLOCKED | SKIPPED — see [`../_shared/failure-mode-handling.md`](../_shared/failure-mode-handling.md) section F for the canonical taxonomy. Deliverable entries count toward it like test cases: a `FAIL` entry fails the run, an `INCONCLUSIVE` one is a gap. A browser skip beside deliverable entries is not `SKIPPED` — the run's verdict comes from the entries that ran.
|
|
198
209
|
|
|
199
210
|
A non-passing verdict does not end the cycle. [`e2e-repair.md`](e2e-repair.md) consumes this report at Stage 7.5 and root-causes every Failed and Inconclusive entry before a watcher is armed, so write those blocks in full — the bucket, failing step, and `artifactsDir` recorded here are that stage's entire input.
|
|
200
211
|
|
|
@@ -26,7 +26,7 @@ Read the Stage 6 run-level verdict, defined in [`../_shared/failure-mode-handlin
|
|
|
26
26
|
- `PASS` or `SKIPPED` → nothing to repair. Append `e2e-repair: clear (<verdict>)` to the iteration log and hand to Stage 8.
|
|
27
27
|
- A verdict the user waived earlier in this session → clear on the recorded waiver; never re-ask.
|
|
28
28
|
- An entry whose run failed because the app did not answer — connection refused, page never loaded — is the readiness halt of [`e2e-acceptance.md`](e2e-acceptance.md) Step 0.5 arriving late. Nothing ran, so there is nothing to repair and nothing to waive, attended or not: re-check readiness there, and if the app is still down, stop as Step 4's **Stop here** and report the halt. A run the runner lost while the app answers — an Electron, MCP or Studio crash, even before the first step — is not this: it is an `infra` failure and takes the loop below.
|
|
29
|
-
- `FAIL`, `PARTIAL`, `INCONCLUSIVE`, or `BLOCKED` → the loop runs over every entry in the Failed and Inconclusive blocks.
|
|
29
|
+
- `FAIL`, `PARTIAL`, `INCONCLUSIVE`, or `BLOCKED` → the loop runs over every entry in the Failed and Inconclusive blocks, and every non-`PASS` entry in the Deliverables block.
|
|
30
30
|
|
|
31
31
|
## Step 1: Root-cause each failure
|
|
32
32
|
|
|
@@ -56,6 +56,8 @@ A `repair` disposition requires a named defect: the failing step, expected-versu
|
|
|
56
56
|
|
|
57
57
|
**A failure the diff cannot explain may predate it.** When a `product-defect` or `product-uxux` failure names no code path the diff touched, replay the same test case against a checkout of the base before repairing anything: a worktree at the base tip's full SHA, with its own stack claim and services, under the same runSettings record. If that run fails the same way, its disposition is `baseline` — the defect is on the base, not this change's to fix — and the entry carries that run's id as `baselineRunId` into the walkthrough report ([`../muggle-pr-visual-walkthrough/e2e-report-assembly.md`](../muggle-pr-visual-walkthrough/e2e-report-assembly.md)). If it passes, the change caused the failure and the original disposition stands. A `baseline` entry is never re-run or waived; when every remaining failure is `baseline`, the loop clears.
|
|
58
58
|
|
|
59
|
+
A Deliverables entry carries no bucket. A `FAIL` entry is already a named defect — the failed probe or review with its expected and observed values — so its disposition is `repair`. An `INCONCLUSIVE` entry is `ask`, naming its reason: a missing tool or unreachable source is the machine's to fix, not the diff's. A Step 5 re-run re-checks only the deliverable entries the repair touched.
|
|
60
|
+
|
|
59
61
|
An Inconclusive entry Stage 6 recorded as `prerequisites unmet` never executed. It carries no bucket and none may be derived from it — there is no run to read. Its disposition is `ask`, and the question names the missing prerequisite: what failed is the project's test-plan chain, not the diff.
|
|
60
62
|
|
|
61
63
|
An Inconclusive entry Stage 6 recorded as `primary failed` never executed either: it is a record held back by its test case's failed primary. It takes no disposition of its own; the primary's entry is the one repaired, and re-running the test case runs it again.
|
|
@@ -101,6 +101,8 @@ The case selection [`e2e-acceptance.md`](e2e-acceptance.md) runs. Stage 6 execut
|
|
|
101
101
|
|
|
102
102
|
Scope only when this cycle runs E2E: `Validation` is `local-e2e`, `pathClassification` is not `none`, and in a watcher cycle `cycleSize` is `full`. Otherwise emit `e2eScope: not-scoped — <the condition that failed>`.
|
|
103
103
|
|
|
104
|
+
Deliverable checks ([`../_shared/deliverable-categories.md`](../_shared/deliverable-categories.md)) need no browser, dev server, or Muggle session, so they are scoped on their own rule: whenever `Validation` is not `skip`, `pathClassification` is not `none`, and in a watcher cycle `cycleSize` is `full`. A cycle that scopes deliverable checks but no browser cases emits only `projectId: none`, `deliverableChecks`, and `criteria`; the browser lists are written empty.
|
|
105
|
+
|
|
104
106
|
The criteria are the acceptance criteria in the session's `requirements.md`, plus each review-derived change item that alters behavior in an address-reviews cycle. A session with no `requirements.md` (an adopted PR) scopes from the diff alone and writes `criteria: []`.
|
|
105
107
|
|
|
106
108
|
### Procedure
|
|
@@ -113,9 +115,11 @@ The criteria are the acceptance criteria in the session's `requirements.md`, plu
|
|
|
113
115
|
- **Removed** — the change removes the feature it verifies. `replacedBy: none`.
|
|
114
116
|
|
|
115
117
|
A use case is retired only when the change removes its whole flow **and** every test case under it is retired: deleting a use case cascades to all its test cases and scripts, so one surviving test case keeps the use case. Retired cases are deleted after the PR merges ([`cleanup.md`](cleanup.md)), never during the cycle — an unmerged change must leave the project's suite intact for the default branch.
|
|
116
|
-
5. **Map every criterion.**
|
|
118
|
+
5. **Map every criterion.** A criterion's mapping follows its category's flow file ([`../_shared/deliverable-categories.md`](../_shared/deliverable-categories.md#category-flows)), never the category's name. A flow **Verified by** test cases gives the criterion `coveredBy` (impacted or new test cases that verify it); a flow **Verified by** a deliverable check gives it `verifiedBy` (the deliverable checks that verify it). Either may instead carry `unobservable` (why no check can observe it: a log line, a unit-test requirement, a config value). A criterion with none of the three is a scoping failure.
|
|
117
119
|
6. **Plan the new cases.** Each uncovered criterion gets a new test case under the impacted use case its behavior belongs to, or under a new use case when none fits. A test case instruction names one atomic behavior to verify, with no signup, login, navigation, or seed-data steps, per [`../_shared/test-case-design.md`](../_shared/test-case-design.md); prerequisites come from the test-plan graph at run time ([`../_shared/test-case-chain-readiness.md`](../_shared/test-case-chain-readiness.md)).
|
|
118
120
|
|
|
121
|
+
7. **Plan the deliverable checks.** One check per artifact that the criteria mapped to `verifiedBy` are about, keyed `DC1`, `DC2`, …: its category, the artifact and how it is obtained (committed path, or the build command), the probes with their expected values, and the criteria it verifies. A changed artifact that no criterion names is still checked when the diff touches it — the probes alone, with expected values from its base-branch version.
|
|
122
|
+
|
|
119
123
|
This stage plans new cases and creates nothing. Stage 6 creates them, so a cycle that never reaches E2E leaves no cases behind in the project.
|
|
120
124
|
|
|
121
125
|
### Shape
|
|
@@ -148,11 +152,21 @@ e2eScope:
|
|
|
148
152
|
- key: NTC1
|
|
149
153
|
useCase: <useCaseId | new use case key>
|
|
150
154
|
instruction: <one atomic behavior to verify>
|
|
155
|
+
deliverableChecks:
|
|
156
|
+
- key: DC1
|
|
157
|
+
category: <video | image | audio | data | research>
|
|
158
|
+
artifact: <repo-relative path | build: <command> → <output>>
|
|
159
|
+
probes:
|
|
160
|
+
- probe: <property>
|
|
161
|
+
expected: <value or range>
|
|
162
|
+
criteria: [<criterion>, ...]
|
|
151
163
|
criteria:
|
|
152
|
-
- criterion: <acceptance criterion>
|
|
164
|
+
- criterion: <[<category verified by test cases>] acceptance criterion>
|
|
153
165
|
coveredBy: [<testCaseId | new test case key>, ...]
|
|
166
|
+
- criterion: <[<category verified by a deliverable check>] acceptance criterion>
|
|
167
|
+
verifiedBy: [<DC key>, ...]
|
|
154
168
|
- criterion: <acceptance criterion>
|
|
155
|
-
unobservable: <why no
|
|
169
|
+
unobservable: <why no check can observe it>
|
|
156
170
|
```
|
|
157
171
|
|
|
158
172
|
An empty list is written empty, never omitted; an absent list reads as "not scoped". Record the scope in this stage's marker in `iterations/<NNN>.md`, so Stage 6, a Stage 7.5 re-run, and a resumed session read the same one.
|
|
@@ -39,7 +39,7 @@ Forward pipeline's Stage 7. Invoked by `/muggle-do` after stages 1–6 of a fres
|
|
|
39
39
|
- `## Goal` — from requirements.
|
|
40
40
|
- `## Acceptance Criteria` — bulleted; omit if empty.
|
|
41
41
|
- `## Changes` — summary of what changed in this repo.
|
|
42
|
-
- `## Validation` — one line: link to E2E report, `unit-only`, or `skip — <reason>`.
|
|
42
|
+
- `## Validation` — one line: link to E2E report, `unit-only`, or `skip — <reason>`. A report with a **Deliverables** block adds its tables below that line, per [`../../_shared/deliverable-categories.md`](../../_shared/deliverable-categories.md#evidence-on-the-pr).
|
|
43
43
|
- The walkthrough does **not** go in the body. It settles the PR's designated walkthrough comment in Step 6, so a rerun updates one comment instead of rewriting the description.
|
|
44
44
|
- **Signature** — write the assembled body to a file and sign it with `--command /muggle-do --mode editable` per [`../../_shared/vcs/post-signature.md`](../../_shared/vcs/post-signature.md). The signature lands last, after the walkthrough block; `editable` is the mode a description carries so later refreshes replace it instead of stacking.
|
|
45
45
|
|
|
@@ -49,7 +49,7 @@ Forward pipeline's Stage 7. Invoked by `/muggle-do` after stages 1–6 of a fres
|
|
|
49
49
|
|
|
50
50
|
6. **Settle the designated walkthrough comment.** Creating the PR reserves one comment for the walkthrough, marked `muggle-pr-walkthrough` and empty until settled — the PR's walkthrough check fails while it stays that way.
|
|
51
51
|
- E2E report exists → fire [`postPRVisualWalkthrough`](../../muggle-preferences/preference-gates/postPRVisualWalkthrough.md); on skip, leave the comment to the gate's own skip record. Otherwise invoke [`../../muggle-pr-visual-walkthrough/SKILL.md`](../../muggle-pr-visual-walkthrough/SKILL.md) Mode A with the PR number and repo passed explicitly — it fills the reserved comment in place. Explicit because a session carrying several changes may sit in another change's worktree, where the current branch's PR is the wrong one.
|
|
52
|
-
- No E2E report (validation was `unit-only` or `skip`),
|
|
52
|
+
- No E2E report (validation was `unit-only` or `skip`), a `SKIPPED` report, or a report whose browser part was skipped (a `**Browser skip:**` line, or deliverable entries and no test cases) → now that the PR exists, re-declare the skip Stage 6 recorded with the same code, from this PR's worktree: `echo "MUGGLE_E2E_SKIP: <CODE>: <detail>"`, codes per [`../e2e-acceptance.md`](../e2e-acceptance.md). An accepted code settles only the PRs of that worktree's change — the PRs opened from it and its checked-out branch's PR; a free-text reason is rejected and leaves it unsettled. A PR whose title says `[UNIT-ONLY]` still owes reviewers the why. The write runs detached, so confirm it landed: the walkthrough Stop gate holds the turn while the PR still shows the placeholder, and its block names a foreground command that prints `{"settled":true}` once the PR reads back the skip — run it, never a hand-written PATCH.
|
|
53
53
|
|
|
54
54
|
7. **Overflow comment:** if the walkthrough skill returned a non-null `comment`, post it once using the provider resolved in Step 5 — `github` per [`../../_shared/vcs/github/top-level-comment.md`](../../_shared/vcs/github/top-level-comment.md), `gitlab` per [`../../_shared/vcs/gitlab/mr-note.md`](../../_shared/vcs/gitlab/mr-note.md). End the posted body with the signature line (command `/muggle-do`) per [`../../_shared/vcs/post-signature.md`](../../_shared/vcs/post-signature.md). Never post when `comment` is `null`.
|
|
55
55
|
|
|
@@ -60,7 +60,7 @@ Present **one `AskUserQuestion`** (or the platform's structured-selection equiva
|
|
|
60
60
|
4. **Validation strategy** — the single most important question. Options:
|
|
61
61
|
- **Local E2E** (Muggle Test Electron against a running localhost) — default if a dev server was detected.
|
|
62
62
|
- **Staging replay** — for changes already deployed to a preview URL.
|
|
63
|
-
- **Unit tests only** — skip E2E, acceptable for pure refactors or backend-only changes.
|
|
63
|
+
- **Unit tests only** — skip browser E2E, acceptable for pure refactors or backend-only changes. Deliverable checks on video, image, audio, data, or research artifacts still run.
|
|
64
64
|
- **Skip validation** — explicit opt-out; the PR title gets `[UNVERIFIED]`.
|
|
65
65
|
5. **Local URL** — only if validation is Local E2E. Default: the detected port. "Confirm `<detected>`" / "Type a different URL".
|
|
66
66
|
6. **Backend reachable?** — only if validation is Local E2E and a backend URL is declared. If the health probe failed, ask "Start the backend now and I'll re-probe" / "Proceed anyway" / "Skip to unit tests only".
|
|
@@ -22,7 +22,7 @@ You receive:
|
|
|
22
22
|
|
|
23
23
|
1. **Read the task description carefully.** Understand what the user wants to build, fix, or change.
|
|
24
24
|
2. **Extract the goal** — one clear sentence describing the outcome.
|
|
25
|
-
3. **Extract acceptance criteria** — specific, verifiable conditions that must be true when the task is done. Each criterion should be independently testable. If the user's description is vague, infer reasonable criteria but flag them as inferred.
|
|
25
|
+
3. **Extract acceptance criteria** — specific, verifiable conditions that must be true when the task is done. Each criterion should be independently testable. If the user's description is vague, infer reasonable criteria but flag them as inferred. Tag each criterion with its deliverable category per [`../_shared/deliverable-categories.md`](../_shared/deliverable-categories.md#tagging-a-criterion), splitting any criterion that spans two.
|
|
26
26
|
4. **Identify which repos are likely affected** — based on the task description and the repo names provided.
|
|
27
27
|
|
|
28
28
|
## Output
|
|
@@ -33,8 +33,8 @@ Write the findings to `~/.muggle-ai/muggle-do/sessions/<slug>/requirements.md`.
|
|
|
33
33
|
**Goal:** (one sentence)
|
|
34
34
|
|
|
35
35
|
**Acceptance Criteria:**
|
|
36
|
-
- (criterion 1)
|
|
37
|
-
- (criterion 2)
|
|
36
|
+
- [<category>] (criterion 1)
|
|
37
|
+
- [<category>] (criterion 2)
|
|
38
38
|
|
|
39
39
|
**Affected Repos:** (comma-separated list)
|
|
40
40
|
|
|
@@ -55,7 +55,8 @@ Do not create or update a PR until each line holds, or is waived by a one-line r
|
|
|
55
55
|
- Build clean — typecheck + lint on changed files
|
|
56
56
|
- New/changed logic carries unit tests (authored in Stage 3; Stage 5 only runs the suite)
|
|
57
57
|
- Unit suite run, PASS recorded
|
|
58
|
-
- Stage 4's `e2eScope` recorded, mapping every acceptance criterion to a test case or
|
|
58
|
+
- Stage 4's `e2eScope` recorded, mapping every acceptance criterion to a test case, a deliverable check, or `unobservable` — or `not-scoped` with the condition that failed
|
|
59
|
+
- A verdict recorded for every deliverable check in that scope
|
|
59
60
|
- E2E verdict recorded with `runId` for every case in that scope, per `autoE2ETest` — or `[E2E FAILING]`, or `SKIPPED` citing a verified skip code from [`../do/e2e-acceptance.md`](../do/e2e-acceptance.md), where each code's verification rule lives. Free-text reasons are rejected by the gate
|
|
60
61
|
- UI changes verified in a real browser with evidence (screenshot path or muggle `runId`); `curl` + `grep` is not verification
|
|
61
62
|
|
|
@@ -10,6 +10,8 @@ Include **all** runs — passed, failed, **and inconclusive**. Never drop a run.
|
|
|
10
10
|
| `failed` | Run completed and the assertion failed, or the product itself broke before the assertion could be made (server error, 500, broken page). | `failureStepIndex`, `error` |
|
|
11
11
|
| `inconclusive` | Run could not yield a pass/fail signal for reasons **outside the product**: no replayable script existed, environment precondition unmet, infra/Electron error, agent stalled on a cookie banner or login wall, missing secrets, agent went off-course. | `reason` (one short sentence). Include every step the run produced; `steps[]` is empty only when it produced none. |
|
|
12
12
|
|
|
13
|
+
A run whose studio verdict is `bug_detected` (`**Studio Verdict:** Bug Detected (bug_detected)` on `muggle-local-run-result-get`) is `failed`, never `inconclusive`: it is a product bug the run proved. Put what the app did against what was expected in `error`; do not retry, regenerate, or debug the test.
|
|
14
|
+
|
|
13
15
|
If a test should be inconclusive but you don't have a real `runId` / `viewUrl` (e.g., generation never ran), use the same project-level dashboard fallback as the "last-resort" failed branch below: synthesize a UUID-shaped runId and use the project runs page as `viewUrl`. The schema requires both fields — but a working dashboard link is better than nothing.
|
|
14
16
|
|
|
15
17
|
A record skipped because its test case's primary failed never ran either: an `inconclusive` entry with `reason: "primary failed"`, the `runSettingsId` and `resolutionLabel` the caller carried for it, `steps: []`, and that same synthesized `runId` and project-level `viewUrl`.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"release": "5.26.0",
|
|
3
|
-
"buildId": "run-
|
|
4
|
-
"commitSha": "
|
|
5
|
-
"buildTime": "2026-10-
|
|
3
|
+
"buildId": "run-251-1",
|
|
4
|
+
"commitSha": "1e6cfe30749385ff8e553c34e1f43494668d3cb2",
|
|
5
|
+
"buildTime": "2026-10-30T04:38:24Z",
|
|
6
6
|
"serviceName": "muggle-ai-works-mcp"
|
|
7
7
|
}
|
|
@@ -1 +1 @@
|
|
|
1
|
-
export { ACTION_SCRIPT_FILE_NAME, BASE_REMOTE_NAME, BLOCKING_FRESHNESS_VERDICTS, BRANCH_CREATION_REFLOG_PREFIX, BUSY_EXIT_CODE, CLAUDE_CODE_AUTH_STATUS_ARGS, CLAUDE_CODE_CHECK_TIMEOUT_MS, CLAUDE_CODE_EXECUTABLE, CLAUDE_CODE_EXECUTABLE_NOT_FOUND_CODE, CLAUDE_CODE_SESSION_LINK_ENV_VARS, CLAUDE_CODE_VERSION_ARGS, CheckoutFreshnessVerdict, ClaudeCodeReadiness, DEFAULT_LOCAL_LLM_BASE_URL, DEFAULT_PREFERENCES, ELECTRON_APP_EXECUTABLE_PATH_SEGMENTS, ElectronAppPlatform, ElectronAppReleaseStream, HTML_VIEW_OFFER_LINE, LLM_CREDENTIALS_FILE_MODE, LLM_CREDENTIALS_FILE_NAME, LLM_SOURCE_HEADER, LLM_SOURCE_OPTIONS, LLM_SOURCE_OWNED_ENV_NAMES, LLM_SOURCE_PROMPT, LLM_SOURCE_SCOPE_NOTE, LlmSource, MASKED_API_KEY_PREFIX, MASKED_API_KEY_VISIBLE_CHARS, MAX_LISTED_INHERITED_COMMITS, ONBOARDING_BLANKET_OPTIONS, ONBOARDING_CHOICE_GROUPS, ONBOARDING_MAX_OFFERS, ONBOARDING_PRIMER_BULLETS, ONBOARDING_PRIMER_HEADLINE, ONBOARDING_TOGGLE_GROUPS, ONBOARDING_TOGGLE_PROMPT, ONBOARDING_UNSELECTED_TOGGLE_VALUE, OnboardingBlanketChoice, OnboardingGroupKind, PREFERENCES_FILE_NAME, PREFERENCES_PROJECT_DIR_NAME, PREFERENCES_SCHEMA, PREFERENCES_VERSION, PREFERENCE_ALLOWED_VALUES, PreferenceKey, PreferenceValue, ProjectPreferencesReconcileOutcome, RUN_SESSIONS_DIR_NAME, RUN_VIEW_DASHBOARD_LINK_LABEL, RunViewError, RuntimeTarget, SCREENSHOT_DIR_RELATIVE_PATH, SCREENSHOT_FILE_EXTENSION, StackCheckVerdict, StackClaimVerdict, StackLeaseEndReason, StackMismatchKind, StackPortAction, WATCHER_LIFETIME_SECONDS, WATCHER_LIFETIME_UNBOUNDED_SECONDS, applyLlmSourceSelection, applyOnboardingAnswers, assertDeviceCodeClientProvisioned, buildElectronAppChecksumsUrl, buildElectronAppDirName, buildElectronAppReleaseAssetUrl, buildElectronAppReleaseTag, buildLlmSourcePlan, buildOnboardingPlan, buildToolInputJsonSchema, calculateFileChecksum, checkServiceStack, claimServiceStack, compareQueuePlaces, completeOnboarding, createApiKeyWithToken, createChildLogger, decideStackClaim, decideStackReap, decideStackRelease, decideStackRunEnd, decideStackRunStart, deleteApiKeyData, deleteCredentials, describeLlmSourceStatus, describeStackMismatch, e2e_exports as e2e, evaluateCheckoutFreshness, findVacatedNeeds, finishStackRun, formatPreferencesOneLiner, getActiveElectronAppReleaseStream, getActiveRuntimeTarget, getApiKey, getApiKeyFilePath, getAuthService, getBundledElectronAppVersion, getCallerCredentials, getCallerCredentialsAsync, getChecksumForPlatform, getConfig, getCredentialsFilePath, getDashboardBaseUrl, getDataDir2 as getDataDir, getDownloadBaseUrl, getElectronAppChecksums, getElectronAppDir, getElectronAppReleaseTagPrefix, getElectronAppSignedFromVersion, getElectronAppVersion, getElectronAppVersionSource, getLlmSourceOption, getLocalQaTools, getLogger, getOnboardingToggleKeys, getPlatformKey, getQaTools, getReleaseSignerIdentityUri, getRunSessionsDir, getValidApiKeyData, getValidCredentials, hasApiKey, isElectronAppInstalled, isElectronAppPlatform, isFirstRun, isValidLocalBaseUrl, loadApiKeyData, loadCredentials, local_exports as localQa, lookUpCheckoutPullRequest, markOnboardingCompleted, maskLlmApiKey, mcp_exports as mcp, needsConflict, needsOnboarding, openBrowserUrl, parseElectronAppDirName, performLogin, performLogout, pollDeviceCode, portOfUrl, e2e_exports as qa, readPreferencesMetadata, readRunView, readServiceStackStatus, reapServiceStack, reconcileProjectPreferences, recordOnboardingSkip, releaseServiceStack, renderRunViewHtml, renderRunViewText, resetConfig, resetLogger, resetPreference, resolveActiveProfile, resolveActiveReleaseStream, resolveActiveReleaseTagPrefix, resolveBlanketPreferences, resolveElectronAppExecutablePath, resolveElectronAppPathOrNull, resolveLlmSourceStatus, resolveOnboardingAnswers, resolvePreferences, resolveRunSessionPath, resolveRuntimeTarget, resolveTogglePreferences, runGitCommand, saveApiKey, saveApiKeyData, saveCredentials, startDeviceCodeFlow, startStackRun, stopVacatedServices, toDisplayRef, toolRequiresAuth, validatePreference, verifyFileChecksum, writeLlmEnvBlock, writePreferences, writePreferencesWithMetadata, writeRunViewHtml } from './chunk-
|
|
1
|
+
export { ACTION_SCRIPT_FILE_NAME, BASE_REMOTE_NAME, BLOCKING_FRESHNESS_VERDICTS, BRANCH_CREATION_REFLOG_PREFIX, BUSY_EXIT_CODE, CLAUDE_CODE_AUTH_STATUS_ARGS, CLAUDE_CODE_CHECK_TIMEOUT_MS, CLAUDE_CODE_EXECUTABLE, CLAUDE_CODE_EXECUTABLE_NOT_FOUND_CODE, CLAUDE_CODE_SESSION_LINK_ENV_VARS, CLAUDE_CODE_VERSION_ARGS, CheckoutFreshnessVerdict, ClaudeCodeReadiness, DEFAULT_LOCAL_LLM_BASE_URL, DEFAULT_PREFERENCES, ELECTRON_APP_EXECUTABLE_PATH_SEGMENTS, ElectronAppPlatform, ElectronAppReleaseStream, HTML_VIEW_OFFER_LINE, LLM_CREDENTIALS_FILE_MODE, LLM_CREDENTIALS_FILE_NAME, LLM_SOURCE_HEADER, LLM_SOURCE_OPTIONS, LLM_SOURCE_OWNED_ENV_NAMES, LLM_SOURCE_PROMPT, LLM_SOURCE_SCOPE_NOTE, LlmSource, MASKED_API_KEY_PREFIX, MASKED_API_KEY_VISIBLE_CHARS, MAX_LISTED_INHERITED_COMMITS, ONBOARDING_BLANKET_OPTIONS, ONBOARDING_CHOICE_GROUPS, ONBOARDING_MAX_OFFERS, ONBOARDING_PRIMER_BULLETS, ONBOARDING_PRIMER_HEADLINE, ONBOARDING_TOGGLE_GROUPS, ONBOARDING_TOGGLE_PROMPT, ONBOARDING_UNSELECTED_TOGGLE_VALUE, OnboardingBlanketChoice, OnboardingGroupKind, PREFERENCES_FILE_NAME, PREFERENCES_PROJECT_DIR_NAME, PREFERENCES_SCHEMA, PREFERENCES_VERSION, PREFERENCE_ALLOWED_VALUES, PreferenceKey, PreferenceValue, ProjectPreferencesReconcileOutcome, RUN_SESSIONS_DIR_NAME, RUN_VIEW_DASHBOARD_LINK_LABEL, RunViewError, RuntimeTarget, SCREENSHOT_DIR_RELATIVE_PATH, SCREENSHOT_FILE_EXTENSION, StackCheckVerdict, StackClaimVerdict, StackLeaseEndReason, StackMismatchKind, StackPortAction, WATCHER_LIFETIME_SECONDS, WATCHER_LIFETIME_UNBOUNDED_SECONDS, applyLlmSourceSelection, applyOnboardingAnswers, assertDeviceCodeClientProvisioned, buildElectronAppChecksumsUrl, buildElectronAppDirName, buildElectronAppReleaseAssetUrl, buildElectronAppReleaseTag, buildLlmSourcePlan, buildOnboardingPlan, buildToolInputJsonSchema, calculateFileChecksum, checkServiceStack, claimServiceStack, compareQueuePlaces, completeOnboarding, createApiKeyWithToken, createChildLogger, decideStackClaim, decideStackReap, decideStackRelease, decideStackRunEnd, decideStackRunStart, deleteApiKeyData, deleteCredentials, describeLlmSourceStatus, describeStackMismatch, e2e_exports as e2e, evaluateCheckoutFreshness, findVacatedNeeds, finishStackRun, formatPreferencesOneLiner, getActiveElectronAppReleaseStream, getActiveRuntimeTarget, getApiKey, getApiKeyFilePath, getAuthService, getBundledElectronAppVersion, getCallerCredentials, getCallerCredentialsAsync, getChecksumForPlatform, getConfig, getCredentialsFilePath, getDashboardBaseUrl, getDataDir2 as getDataDir, getDownloadBaseUrl, getElectronAppChecksums, getElectronAppDir, getElectronAppReleaseTagPrefix, getElectronAppSignedFromVersion, getElectronAppVersion, getElectronAppVersionSource, getLlmSourceOption, getLocalQaTools, getLogger, getOnboardingToggleKeys, getPlatformKey, getQaTools, getReleaseSignerIdentityUri, getRunSessionsDir, getValidApiKeyData, getValidCredentials, hasApiKey, isElectronAppInstalled, isElectronAppPlatform, isFirstRun, isValidLocalBaseUrl, loadApiKeyData, loadCredentials, local_exports as localQa, lookUpCheckoutPullRequest, markOnboardingCompleted, maskLlmApiKey, mcp_exports as mcp, needsConflict, needsOnboarding, openBrowserUrl, parseElectronAppDirName, performLogin, performLogout, pollDeviceCode, portOfUrl, e2e_exports as qa, readPreferencesMetadata, readRunView, readServiceStackStatus, reapServiceStack, reconcileProjectPreferences, recordOnboardingSkip, releaseServiceStack, renderRunViewHtml, renderRunViewText, resetConfig, resetLogger, resetPreference, resolveActiveProfile, resolveActiveReleaseStream, resolveActiveReleaseTagPrefix, resolveBlanketPreferences, resolveElectronAppExecutablePath, resolveElectronAppPathOrNull, resolveLlmSourceStatus, resolveOnboardingAnswers, resolvePreferences, resolveRunSessionPath, resolveRuntimeTarget, resolveTogglePreferences, runGitCommand, saveApiKey, saveApiKeyData, saveCredentials, startDeviceCodeFlow, startStackRun, stopVacatedServices, toDisplayRef, toolRequiresAuth, validatePreference, verifyFileChecksum, writeLlmEnvBlock, writePreferences, writePreferencesWithMetadata, writeRunViewHtml } from './chunk-JRYEMMKU.js';
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@muggleai/works",
|
|
3
3
|
"mcpName": "io.github.multiplex-ai/muggle",
|
|
4
|
-
"version": "5.28.0-staging.
|
|
4
|
+
"version": "5.28.0-staging.251",
|
|
5
5
|
"description": "13 years of Microsoft hyperscale engineering, packaged as a harness for your AI coding agent.",
|
|
6
6
|
"type": "module",
|
|
7
7
|
"main": "dist/index.js",
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "muggle",
|
|
3
3
|
"description": "Real-browser E2E acceptance for your coding agent: test the flows you changed and open the PR with screenshots. Built for flows other tools can't reach: managed test identities with live inboxes (magic links, email OTP and 2FA, verification mail, password resets), an encrypted secret store for login and test credentials, and a test graph that runs prerequisite flows first. Plain-English tests become replayable scripts on localhost or preview URLs.",
|
|
4
|
-
"version": "5.28.0-staging.
|
|
4
|
+
"version": "5.28.0-staging.251",
|
|
5
5
|
"author": {
|
|
6
6
|
"name": "Muggle AI",
|
|
7
7
|
"email": "support@muggle-ai.com"
|
package/plugin/.codex-mcp.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "muggle",
|
|
3
|
-
"version": "5.28.0-staging.
|
|
3
|
+
"version": "5.28.0-staging.251",
|
|
4
4
|
"description": "Real-browser E2E acceptance for your coding agent: test the flows you changed and open the PR with screenshots. Built for flows other tools can't reach: managed test identities with live inboxes (magic links, email OTP and 2FA, verification mail, password resets), an encrypted secret store for login and test credentials, and a test graph that runs prerequisite flows first. Plain-English tests become replayable scripts on localhost or preview URLs.",
|
|
5
5
|
"author": {
|
|
6
6
|
"name": "Muggle AI",
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
"name": "muggle",
|
|
3
3
|
"displayName": "Muggle AI",
|
|
4
4
|
"description": "Real-browser E2E acceptance for your coding agent: test the flows you changed and open the PR with screenshots. Built for flows other tools can't reach: managed test identities with live inboxes (magic links, email OTP and 2FA, verification mail, password resets), an encrypted secret store for login and test credentials, and a test graph that runs prerequisite flows first. Plain-English tests become replayable scripts on localhost or preview URLs.",
|
|
5
|
-
"version": "5.28.0-staging.
|
|
5
|
+
"version": "5.28.0-staging.251",
|
|
6
6
|
"author": {
|
|
7
7
|
"name": "Muggle AI",
|
|
8
8
|
"email": "support@muggle-ai.com"
|
package/plugin/.mcp.json
CHANGED
|
@@ -22,6 +22,8 @@ You operate through skills — never call raw MCP tools directly.
|
|
|
22
22
|
|
|
23
23
|
Select the skill based on what the orchestrator asks you to do. If the task doesn't clearly map to one skill, ask for clarification.
|
|
24
24
|
|
|
25
|
+
A criterion about a video, image, audio, data, or research artifact is verified by a deliverable check rather than a browser run, per [`../skills/_shared/deliverable-categories.md`](../skills/_shared/deliverable-categories.md). Its results are reported in the **Deliverables** block and never as a Muggle run.
|
|
26
|
+
|
|
25
27
|
The run loop these skills execute (replay/regen, timeouts, publish, screenshots) lives in [`../skills/_shared/dev-loop/run.md`](../skills/_shared/dev-loop/run.md) — you reach it through `muggle-test`, never by calling MCP tools yourself.
|
|
26
28
|
|
|
27
29
|
## Input Contract
|
|
@@ -58,6 +60,8 @@ Verdict policy (matches the CLI's `computeVerdict`):
|
|
|
58
60
|
|
|
59
61
|
A test is **inconclusive** when the run could not yield a pass/fail signal for reasons outside the product itself: no replayable script, environment precondition unmet, infra error blocked execution, agent stalled before reaching the assertion (auth/cookie banner, missing secret). The product is not implicated. If you are tempted to call a run "passed except for one that didn't really run," that one is inconclusive.
|
|
60
62
|
|
|
63
|
+
A run whose studio verdict is `bug_detected` (`**Studio Verdict:** Bug Detected (bug_detected)` on `muggle-local-run-result-get`) is **failed**: it is a product bug the run proved. Report it as a FAIL whose blocking issue is that bug in the user's app; do not retry, regenerate, or debug the test.
|
|
64
|
+
|
|
61
65
|
**Section 2 — Per-Test Highlights**
|
|
62
66
|
|
|
63
67
|
Order failures first, then inconclusives, then passes.
|
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
# Audio deliverable check
|
|
2
|
+
|
|
3
|
+
The flow for an `[audio]` criterion. Shared rules — obtaining the artifact, verdicts, the report entry — live in the `deliverable-categories.md` index beside this folder, which opens this file.
|
|
4
|
+
|
|
5
|
+
**Verified by:** deliverable check.
|
|
6
|
+
|
|
7
|
+
## Artifact
|
|
8
|
+
|
|
9
|
+
An audio file: committed at head, or produced by the repo's render or text-to-speech command.
|
|
10
|
+
|
|
11
|
+
## Probes
|
|
12
|
+
|
|
13
|
+
Read with `ffprobe` unless a step names another tool.
|
|
14
|
+
|
|
15
|
+
- Codec.
|
|
16
|
+
- Duration, against the stated length or range.
|
|
17
|
+
- Sample rate and channel count.
|
|
18
|
+
- Integrated loudness and true peak (`ffmpeg` `ebur128`), against any stated target.
|
|
19
|
+
- Silence stretches (`silencedetect`) longer than two seconds, unless the criterion calls for them.
|
|
20
|
+
|
|
21
|
+
## Review
|
|
22
|
+
|
|
23
|
+
Transcribe spoken content with a speech-to-text tool available on the machine, and compare the transcript with the script or the criterion.
|
|
24
|
+
|
|
25
|
+
## Inconclusive when
|
|
26
|
+
|
|
27
|
+
- `ffprobe` or `ffmpeg` is missing.
|
|
28
|
+
- No transcription is available. Only the spoken-content review is `INCONCLUSIVE`, with reason `no transcription available`; the probes still run and report.
|
|
29
|
+
- The audio cannot be produced for an environment reason.
|
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
# Data deliverable check
|
|
2
|
+
|
|
3
|
+
The flow for a `[data]` criterion. Shared rules — obtaining the artifact, verdicts, the report entry — live in the `deliverable-categories.md` index beside this folder, which opens this file.
|
|
4
|
+
|
|
5
|
+
**Verified by:** deliverable check.
|
|
6
|
+
|
|
7
|
+
## Artifact
|
|
8
|
+
|
|
9
|
+
A dataset or export — CSV, JSON, JSONL, Parquet, seed data: committed at head, or produced by the repo's export command.
|
|
10
|
+
|
|
11
|
+
## Probes
|
|
12
|
+
|
|
13
|
+
Run as a script written outside the worktree, so the check leaves the change untouched.
|
|
14
|
+
|
|
15
|
+
- The file parses in its format.
|
|
16
|
+
- Schema fields and types.
|
|
17
|
+
- Row count, against the expected count or range.
|
|
18
|
+
- Required fields filled.
|
|
19
|
+
- Key uniqueness.
|
|
20
|
+
- Values inside stated ranges or enumerations.
|
|
21
|
+
- References to other files or tables resolve.
|
|
22
|
+
|
|
23
|
+
A changed dataset is also diffed against its base-branch version: fields added or removed, type changes, row-count delta.
|
|
24
|
+
|
|
25
|
+
## Review
|
|
26
|
+
|
|
27
|
+
Read the first rows and a random sample of rows against the criterion.
|
|
28
|
+
|
|
29
|
+
## Inconclusive when
|
|
30
|
+
|
|
31
|
+
- The dataset cannot be produced for an environment reason.
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
# Image deliverable check
|
|
2
|
+
|
|
3
|
+
The flow for an `[image]` criterion. Shared rules — obtaining the artifact, verdicts, the report entry — live in the `deliverable-categories.md` index beside this folder, which opens this file.
|
|
4
|
+
|
|
5
|
+
**Verified by:** deliverable check.
|
|
6
|
+
|
|
7
|
+
## Artifact
|
|
8
|
+
|
|
9
|
+
A raster or vector image: committed at head, or produced by the repo's build or export command.
|
|
10
|
+
|
|
11
|
+
## Probes
|
|
12
|
+
|
|
13
|
+
Read with `ffprobe` for raster images; read an SVG's markup directly.
|
|
14
|
+
|
|
15
|
+
- Format.
|
|
16
|
+
- Pixel dimensions, or the `viewBox` for SVG.
|
|
17
|
+
- Aspect ratio.
|
|
18
|
+
- File size, against any stated budget.
|
|
19
|
+
- Alpha channel, when the criterion requires transparency.
|
|
20
|
+
|
|
21
|
+
## Review
|
|
22
|
+
|
|
23
|
+
View the image against the criterion:
|
|
24
|
+
|
|
25
|
+
- Text legible and spelled as specified.
|
|
26
|
+
- Required elements present.
|
|
27
|
+
- Nothing cropped or clipped.
|
|
28
|
+
|
|
29
|
+
A changed image is also viewed beside its base-branch version; every difference the criterion does not ask for is a finding.
|
|
30
|
+
|
|
31
|
+
## Inconclusive when
|
|
32
|
+
|
|
33
|
+
- `ffprobe` is missing for a raster image.
|
|
34
|
+
- The image cannot be produced for an environment reason.
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
# Research deliverable check
|
|
2
|
+
|
|
3
|
+
The flow for a `[research]` criterion. Shared rules — obtaining the artifact, verdicts, the report entry — live in the `deliverable-categories.md` index beside this folder, which opens this file.
|
|
4
|
+
|
|
5
|
+
**Verified by:** deliverable check.
|
|
6
|
+
|
|
7
|
+
## Artifact
|
|
8
|
+
|
|
9
|
+
A report or brief: committed at head, or produced by the repo's generation command.
|
|
10
|
+
|
|
11
|
+
## Probes
|
|
12
|
+
|
|
13
|
+
- Every question the criterion poses has an answer, cited by section.
|
|
14
|
+
- Every claim a criterion depends on cites a source.
|
|
15
|
+
- Every cited source resolves.
|
|
16
|
+
- No placeholders remain: `TODO`, `TBD`, `[citation needed]`, an empty section.
|
|
17
|
+
- A figure quoted in two places agrees.
|
|
18
|
+
- Every source falls inside any stated recency bound.
|
|
19
|
+
|
|
20
|
+
## Review
|
|
21
|
+
|
|
22
|
+
Read each cited source and confirm it says what the claim says.
|
|
23
|
+
|
|
24
|
+
A source that answers gone or not found is a `FAIL`: a dead citation is a defect in the deliverable.
|
|
25
|
+
|
|
26
|
+
## Inconclusive when
|
|
27
|
+
|
|
28
|
+
- A source is unreachable for a network reason.
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
# Video deliverable check
|
|
2
|
+
|
|
3
|
+
The flow for a `[video]` criterion. Shared rules — obtaining the artifact, verdicts, the report entry — live in the `deliverable-categories.md` index beside this folder, which opens this file.
|
|
4
|
+
|
|
5
|
+
**Verified by:** deliverable check.
|
|
6
|
+
|
|
7
|
+
## Artifact
|
|
8
|
+
|
|
9
|
+
A rendered video file: committed at head, or produced by the repo's render command.
|
|
10
|
+
|
|
11
|
+
## Probes
|
|
12
|
+
|
|
13
|
+
Read with `ffprobe` unless a step names another tool.
|
|
14
|
+
|
|
15
|
+
- Container and codec.
|
|
16
|
+
- Duration, against the stated length or range.
|
|
17
|
+
- Resolution and frame rate.
|
|
18
|
+
- Audio-track presence, when the criterion implies sound.
|
|
19
|
+
- Black stretches (`ffmpeg` `blackdetect`) and frozen stretches (`freezedetect`) longer than one second, unless the criterion calls for them.
|
|
20
|
+
|
|
21
|
+
## Review
|
|
22
|
+
|
|
23
|
+
Extract a frame with `ffmpeg` at every timestamp a criterion names, plus the first frame, the last, and one every 5 seconds — at most 24 in all. View each one against the criterion:
|
|
24
|
+
|
|
25
|
+
- On-screen text present and spelled as specified.
|
|
26
|
+
- Scenes in the stated order.
|
|
27
|
+
- Captions on screen when they should be.
|
|
28
|
+
|
|
29
|
+
## Inconclusive when
|
|
30
|
+
|
|
31
|
+
- `ffprobe` or `ffmpeg` is missing.
|
|
32
|
+
- The render fails for an environment reason.
|
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
# Web deliverable check
|
|
2
|
+
|
|
3
|
+
The flow for a `[web]` criterion. Shared rules — tagging, verdicts, the report entry — live in the `deliverable-categories.md` index beside this folder, which opens this file.
|
|
4
|
+
|
|
5
|
+
**Verified by:** test cases.
|
|
6
|
+
|
|
7
|
+
## Artifact
|
|
8
|
+
|
|
9
|
+
The running app at the cycle's validation target: the dev server the readiness check brings up from the PR's worktree, or the preview URL a staging replay names.
|
|
10
|
+
|
|
11
|
+
## Test cases
|
|
12
|
+
|
|
13
|
+
Each criterion maps to the impacted or new Muggle test cases that exercise it, recorded as `coveredBy` in the scope. A new test case names one atomic behavior to verify, with prerequisites resolved from the project's test-plan graph rather than written into the case.
|
|
14
|
+
|
|
15
|
+
## Review
|
|
16
|
+
|
|
17
|
+
The run's step screenshots, read against the criterion: the element the criterion names is on screen, in the stated state, at the step that asserts it.
|
|
18
|
+
|
|
19
|
+
## Inconclusive when
|
|
20
|
+
|
|
21
|
+
- No replayable script exists and generation fails.
|
|
22
|
+
- The test case's prerequisites cannot be satisfied.
|
|
23
|
+
- The runner fails before reaching the assertion for a reason outside the product: an infrastructure error, a missing secret, an auth or cookie banner it could not clear.
|
|
24
|
+
|
|
25
|
+
An app that never answers is not inconclusive: the cycle halts for readiness and owes the run.
|
|
@@ -0,0 +1,69 @@
|
|
|
1
|
+
# Deliverable categories
|
|
2
|
+
|
|
3
|
+
A **deliverable** is the artifact an acceptance criterion is about. Its **deliverable category** is one of `web`, `video`, `image`, `audio`, `data`, `research` — the one list of categories, which other files reference rather than restate. No category is special-cased: each criterion is verified the way its category's flow file declares under **Verified by** — **test cases** (Muggle browser test cases, run against the app) or a **deliverable check** (obtain the artifact, probe it, review it, return a verdict). Scoping and the acceptance stage read that declaration from the criterion's category and never branch on a category's name.
|
|
4
|
+
|
|
5
|
+
## Tagging a criterion
|
|
6
|
+
|
|
7
|
+
Every criterion carries exactly one category, written as a leading tag: `- [video] The promo renders at 1080×1920 and runs 28–32 s`. A criterion spanning two categories (a page that embeds a new video) is split into one criterion per category. A criterion about something a user does in a browser is `[web]`. A criterion no category can observe — a log line, a config value, a unit-test requirement — is still tagged with the category of the artifact it concerns, and impact analysis scopes it `unobservable`.
|
|
8
|
+
|
|
9
|
+
## Obtaining the artifact
|
|
10
|
+
|
|
11
|
+
- **Committed** — a file at the branch head is checked as committed, read from the PR's worktree.
|
|
12
|
+
- **Built** — an artifact the change produces (a render, an export, a generated report) is built with the repo's own command, run from the PR's worktree at head, writing into a scratch directory outside the worktree.
|
|
13
|
+
|
|
14
|
+
A check never commits, moves, or edits what it obtains. A build that fails because the change is broken is a `FAIL`; a build that fails because the environment lacks something (a missing binary, no network) is `INCONCLUSIVE`, naming it.
|
|
15
|
+
|
|
16
|
+
## Probes and review
|
|
17
|
+
|
|
18
|
+
A **probe** measures one property and compares it with the expected value the criterion states, or the value the scope planned when the criterion implies one. A rerun on the same artifact measures the same value. A **review** is the agent's inspection of the content against the criterion; the report labels every review finding as agent judgment.
|
|
19
|
+
|
|
20
|
+
Media probes use `ffprobe`, and frame extraction and media filters use `ffmpeg`. A probe whose tool is missing makes the check `INCONCLUSIVE` with reason `probe tool missing: <tool>` — never a silent pass, never a skipped probe.
|
|
21
|
+
|
|
22
|
+
## Category flows
|
|
23
|
+
|
|
24
|
+
Every category's flow lives in its own file, so it can be customized without touching the others. Each file declares **Verified by**, then names the artifact, how it is verified, and when the verification is `INCONCLUSIVE`; the rules on this page apply to all of them. Changing a category's **Verified by** changes how its criteria are scoped and run, with no edit anywhere else.
|
|
25
|
+
|
|
26
|
+
| Category | Verified by | Flow |
|
|
27
|
+
| :-- | :-- | :-- |
|
|
28
|
+
| web | test cases | [`deliverable-categories/web.md`](deliverable-categories/web.md) |
|
|
29
|
+
| video | deliverable check | [`deliverable-categories/video.md`](deliverable-categories/video.md) |
|
|
30
|
+
| image | deliverable check | [`deliverable-categories/image.md`](deliverable-categories/image.md) |
|
|
31
|
+
| audio | deliverable check | [`deliverable-categories/audio.md`](deliverable-categories/audio.md) |
|
|
32
|
+
| data | deliverable check | [`deliverable-categories/data.md`](deliverable-categories/data.md) |
|
|
33
|
+
| research | deliverable check | [`deliverable-categories/research.md`](deliverable-categories/research.md) |
|
|
34
|
+
|
|
35
|
+
Scoping and the acceptance stage open the flow file of every category the cycle's criteria carry before planning or running anything.
|
|
36
|
+
|
|
37
|
+
## Verdict
|
|
38
|
+
|
|
39
|
+
- **PASS** — every probe and review passed.
|
|
40
|
+
- **FAIL** — a probe or review failed on an artifact that was obtained. The entry names the probe or review, the expected value, and the observed one.
|
|
41
|
+
- **INCONCLUSIVE** — the artifact could not be obtained for an environment reason, a required tool is missing, or a source could not be reached. The product is not implicated.
|
|
42
|
+
|
|
43
|
+
## Report entry
|
|
44
|
+
|
|
45
|
+
```
|
|
46
|
+
- (deliverable name):
|
|
47
|
+
- key: <DC key from the scope>
|
|
48
|
+
- category: <video | image | audio | data | research>
|
|
49
|
+
- artifact: <repo-relative path, or the build command and its scratch output>
|
|
50
|
+
- criteria: <the criteria it verifies>
|
|
51
|
+
- checks: [{ check, kind: probe | review, expected, observed, passed }, ...]
|
|
52
|
+
- verdict: PASS | FAIL | INCONCLUSIVE
|
|
53
|
+
- reason: <one short sentence; FAIL and INCONCLUSIVE only>
|
|
54
|
+
- evidenceDir: <scratch directory holding extracted frames, transcripts, probe output>
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
## Evidence on the PR
|
|
58
|
+
|
|
59
|
+
Deliverable checks are run by the agent and are not recorded on the Muggle dashboard, so they never go under the walkthrough heading or its slot marker — that heading is Muggle vouching for a recorded run. They go in the PR body's `## Validation` section as one table per check:
|
|
60
|
+
|
|
61
|
+
```
|
|
62
|
+
**[video] Promo render** — PASS
|
|
63
|
+
| Check | Kind | Expected | Observed |
|
|
64
|
+
| :-- | :-- | :-- | :-- |
|
|
65
|
+
| duration | probe | 28–32 s | 30.04 s |
|
|
66
|
+
| end-card text | review (agent judgment) | "Try Muggle" | "Try Muggle" |
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
The table is followed by one line: `Deliverable checks ran in this PR's worktree at <short sha>; they are not recorded on the Muggle dashboard.` An image committed at head is embedded under its table by its blob URL at that commit, with `?raw=true`.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Dev Loop — Reading the Result
|
|
2
2
|
|
|
3
|
-
For each `runId`, call `muggle-local-run-result-get` and read **structured fields**, never `execute`'s stdout tail (a truncated display excerpt). Order: `Status` → `Error` → `Artifacts`. Retain per run: `testCaseId`, `testScriptId` (if any), `runId`, `status`, `artifactsDir`, plus the record it ran: `runSettingsId` from the `cloudRunSettingsId` the result prints as `**Cloud Run Settings ID:**` (absent on a run filed under no record), and the `resolutionLabel` of the plan record it ran.
|
|
3
|
+
For each `runId`, call `muggle-local-run-result-get` and read **structured fields**, never `execute`'s stdout tail (a truncated display excerpt). Order: `Status` → `Studio Verdict` → `Error` → `Artifacts`. Retain per run: `testCaseId`, `testScriptId` (if any), `runId`, `status`, `artifactsDir`, plus the record it ran: `runSettingsId` from the `cloudRunSettingsId` the result prints as `**Cloud Run Settings ID:**` (absent on a run filed under no record), and the `resolutionLabel` of the plan record it ran.
|
|
4
4
|
|
|
5
5
|
## Artifacts
|
|
6
6
|
|
|
@@ -11,6 +11,7 @@ Per-step frames are **not** absent on the failure path — the `Artifacts` secti
|
|
|
11
11
|
## Common failures
|
|
12
12
|
|
|
13
13
|
- `Electron execution timed out after 300000ms` → orchestration wait too short; see [`timeouts.md`](timeouts.md).
|
|
14
|
+
- **`Studio Verdict: Bug Detected (bug_detected)`** → status `failed`: it is a product bug the run proved. Surface it to the user as a bug in their app; do not retry, regenerate, or debug the test.
|
|
14
15
|
- **Exit code 26** (with "LLM failed to generate / replay action script") is usually a completed exploration whose outcome was **goal not achievable** (`goal_not_achievable`, summary with `halt`) — e.g. asserting "view a completed run" on an account that has none. Read the summary; don't assume a crash. Fix the precondition: pick a project/account that already has the needed state, or narrow the goal so generation isn't forced to create resources from scratch.
|
|
15
16
|
|
|
16
17
|
Then route via [`../failure-mode-handling.md`](../failure-mode-handling.md).
|