greenrun-cli 0.2.11 → 0.2.13

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "greenrun-cli",
-  "version": "0.2.11",
+  "version": "0.2.13",
   "description": "CLI and MCP server for Greenrun - browser test management for Claude Code",
   "type": "module",
   "main": "dist/server.js",

package/templates/commands/procedures.md

@@ -44,66 +44,61 @@ If all tests are scripted, skip to Step 4.
 
 ### Step 3: Generate scripts for unscripted tests
 
-For each **unscripted** test, one at a time:
+For each **unscripted** test, launch a Task agent sequentially (one at a time, wait for each to complete before starting the next). This keeps browser snapshot data out of the parent context.
 
-1. Call `get_test(test_id)` to fetch the full instructions
-2. Do a **scouting pass** — follow the test instructions in the browser to observe all UI states:
-   - Navigate to the test's starting page via `browser_navigate`
-   - Take a `browser_snapshot` to see initial elements
-   - Follow the test instructions step by step using Playwright MCP tools (`browser_click`, `browser_type`, `browser_snapshot` after each action)
-   - Snapshot after each state change to capture: validation errors, success banners, modal dialogs, redirected pages, dynamically loaded content
-   - Collect all observed elements and selectors as context
-
-#### Handling failures during scouting
-
-If a step doesn't work as expected during the scouting pass, investigate before moving on:
-
-1. **Determine the cause**: Is it a test problem (wrong instructions, bad selectors, missing prerequisite) or an application bug (form won't submit, unexpected error, broken functionality)?
+```
+Task tool with:
+- subagent_type: "general-purpose"
+- max_turns: 30
+- model: "sonnet"
+- prompt: (see agent prompt below)
+```
 
-2. **If the test is wrong** — fix and retry:
-   - Adjust the instructions to match what the UI actually requires (e.g. a required field the instructions missed, a different button label, an extra confirmation step)
-   - Update the test via `update_test` with corrected instructions
-   - Retry the failing step
+#### Script generation agent prompt
 
-3. **If it's an application bug** — work around it and record the bug:
-   - Find a way to make the original test pass by avoiding the broken path (e.g. if a discount code field breaks form submission, leave it blank)
-   - Update the original test instructions if needed to use the working path
-   - Create a **new bug test** that reproduces the specific failure:
-     ```
-     create_test(project_id, {
-       name: "BUG: [description of the failure]",
-       instructions: "[steps that reproduce the bug, ending with the expected vs actual behaviour]",
-       tags: ["bug"],
-       page_ids: [relevant page IDs],
-       credential_name: same as original test
-     })
-     ```
-   - Start a run for the bug test and immediately complete it as failed:
-     ```
-     start_run(bug_test_id) → complete_run(run_id, "failed", "description of what went wrong")
-     ```
-   - Continue scouting the original test with the workaround
+Include the following in the prompt, substituting the actual values:
 
-This ensures the original test captures the happy path while bugs are tracked as separate failing tests that will show up in future runs.
+```
+Greenrun script generation for test: {test_name}
+Test ID: {test_id}
+Project ID: {project_id}
 
-After each test's scouting pass, close the browser so the next test starts with a clean context (no leftover cookies, storage, or page state):
+Project auth: {auth_mode}, login_url: {login_url}
+Credentials: {credential_name} — email: {email}, password: {password}
 
-Call `browser_close` after collecting all observations. The next `browser_navigate` call will automatically open a fresh browser context.
+## Task
 
-Then generate a `.spec.ts` script using the observed elements:
+1. Call `get_test("{test_id}")` to fetch the full test instructions
+2. Authenticate: navigate to {login_url} and log in with the credential above using `browser_navigate`, `browser_snapshot`, `browser_click`, `browser_type`
+3. Do a scouting pass — follow the test instructions step by step in the browser:
+   - Navigate to the test's starting page via `browser_navigate`
+   - Take a `browser_snapshot` to see initial elements
+   - Follow each instruction using Playwright MCP tools (`browser_click`, `browser_type`, `browser_snapshot` after each action)
+   - Snapshot after each state change to capture selectors, validation errors, success banners, modal dialogs, redirected pages
+4. Handle failures:
+   - If a step fails because the test instructions are wrong (wrong field name, missing step, bad selector), fix the instructions and retry. Update the test via `update_test` with corrected instructions.
+   - If a step fails because of an application bug, work around it for the main test and create a new bug test:
+     `create_test({project_id}, { name: "BUG: [description]", instructions: "[repro steps]", tags: ["bug"], page_ids: [...], credential_name: "{credential_name}" })`
+     Then: `start_run(bug_test_id)` → `complete_run(run_id, "failed", "description")`
+5. After scouting, generate a Playwright `.spec.ts` script:
 
-```ts
 import { test, expect } from '@playwright/test';
 test('{test_name}', async ({ page }) => {
-  // If the test has a credential_name, include login steps using the matching
-  // credential from project.credentials (email + password) at the login_url
+  // Include login steps using the credential email + password at login_url
   await page.goto('{start_url}');
-  // Steps generated from scouting pass observations
+  // Steps from scouting observations
   // Use getByRole, getByText, getByLabel, getByPlaceholder for selectors
 });
+
+6. Save: `update_test("{test_id}", { script: <generated_script>, script_generated_at: "<ISO_now>" })`
+7. Close browser: `browser_close`
+
+## Return
+
+Return a one-line summary: {test_name} | script generated | or | {test_name} | failed | {reason}
 ```
 
-Save via `update_test(test_id, { script: <generated_script>, script_generated_at: <ISO_now> })`.
+After each agent completes, note the result and proceed to the next unscripted test.
 
 ### Step 4: Export auth state
 
@@ -123,9 +118,22 @@ Call this via `browser_run_code`. If `auth_mode` is `none`, skip this step.
 
 Gather all tests that have scripts (previously scripted + newly generated from Step 3).
 
-1. **Fetch and write test files**: For each scripted test, call `get_test(test_id)` to retrieve the full script content, then write it to `/tmp/greenrun-tests/{test_id}.spec.ts`. Fetch scripts in parallel to minimize latency.
+**1. Fetch scripts and write test files** — launch one Task agent per scripted test, all in parallel. These are just API calls + file writes so they don't conflict. Also write the Playwright config directly (it's small).
+
+For each scripted test, launch in parallel:
 
-2. **Write config**: Write `/tmp/greenrun-tests/playwright.config.ts`:
+```
+Task tool with:
+- subagent_type: "general-purpose"
+- max_turns: 5
+- model: "haiku"
+- prompt: "Fetch the Playwright script for test {test_id} and write it to a file.
+  1. Call `get_test(\"{test_id}\")` to fetch the test
+  2. Write the `script` field to `/tmp/greenrun-tests/{test_id}.spec.ts` using the Write tool
+  3. Return: \"{test_name} | written\""
+```
+
+While the agents run, write `/tmp/greenrun-tests/playwright.config.ts` directly:
 
 ```ts
 import { defineConfig } from '@playwright/test';
@@ -136,23 +144,24 @@ export default defineConfig({
   reporter: [['json', { outputFile: 'results.json' }]],
   use: {
     baseURL: '{base_url}',
-    storageState: '/tmp/greenrun-auth-state.json',  // omit 'use.storageState' entirely if auth_mode is 'none'
+    // include storageState ONLY if auth_mode is not 'none':
+    storageState: '/tmp/greenrun-auth-state.json',
   },
 });
 ```
 
-Replace `{base_url}` with the project's base_url.
+Wait for all agents to complete before executing.
 
-3. **Execute**: Run via Bash:
+**2. Execute** — run via Bash:
 ```
 npx playwright test --config /tmp/greenrun-tests/playwright.config.ts
 ```
 
-4. **Parse results**: Read `/tmp/greenrun-tests/results.json`. Map each result back to a run ID via the filename: `{test_id}.spec.ts` → test_id → find the matching run_id from the batch.
+**3. Parse results**: Read `/tmp/greenrun-tests/results.json`. Map each result back to a run ID via the filename: `{test_id}.spec.ts` → test_id → find the matching run_id from the batch.
 
-5. **Report results**: Call `complete_run(run_id, status, result_summary)` for each test. Map Playwright statuses: `passed` → `passed`, `failed`/`timedOut` → `failed`, other → `error`.
+**4. Report results**: Call `complete_run(run_id, status, result_summary)` for each test. Map Playwright statuses: `passed` → `passed`, `failed`/`timedOut` → `failed`, other → `error`.
 
-6. **Clean up**: Call `browser_close` to reset the MCP browser context.
+**5. Clean up**: Call `browser_close` to reset the MCP browser context.
 
 ### Step 6: Circuit breaker
 
@@ -165,23 +174,50 @@ After parsing all native results, walk through them in completion order. Track c
 
 ### Step 7: AI fallback for native failures
 
-For tests that **failed** in native execution (and circuit breaker has not tripped), execute them one at a time using the AI agent approach:
+For tests that **failed** in native execution (and circuit breaker has not tripped), execute them one at a time via Task agents. This keeps snapshot data out of the parent context.
 
-1. Close the current browser context with `browser_close` so the fallback starts fresh
-2. Re-authenticate by navigating to the login page and following the Authenticate procedure
-3. For each failed test:
-   - Call `get_test(test_id)` to fetch the full instructions
-   - Start a new run via `start_run(test_id)` (the original run was already completed in Step 5)
-   - Navigate to the test's starting page via `browser_navigate`
-   - Follow the test instructions step by step using Playwright MCP tools
-   - Determine if this is a stale script (UI changed) or an actual bug
-   - If the test passes manually, invalidate the cached script: `update_test(test_id, { script: null, script_generated_at: null })`
-   - Call `complete_run(run_id, status, brief_summary)`
-   - Call `browser_close` before the next test to reset state
+For each failed test, launch a Task agent sequentially (wait for each to complete before the next):
+
+```
+Task tool with:
+- subagent_type: "general-purpose"
+- max_turns: 25
+- model: "sonnet"
+- prompt: (see agent prompt below)
+```
+
+#### AI fallback agent prompt
+
+```
+Greenrun AI fallback test. Test: {test_name}
+Test ID: {test_id}
+
+Project auth: {auth_mode}, login_url: {login_url}
+Credentials: {credential_name} — email: {email}, password: {password}
+
+Native execution failed with: {failure_message}
+
+## Task
+
+1. Call `get_test("{test_id}")` to fetch the full test instructions
+2. Start a new run: `start_run("{test_id}")` — note the run_id
+3. Authenticate: navigate to {login_url} and log in with the credential above
+4. Follow the test instructions step by step using Playwright MCP tools (`browser_navigate`, `browser_snapshot`, `browser_click`, `browser_type`)
+5. Determine if the native failure was a stale script (UI changed) or an actual application bug
+6. If the test passes manually, invalidate the stale cached script: `update_test("{test_id}", { script: null, script_generated_at: null })`
+7. Call `complete_run(run_id, status, brief_summary)` — ALWAYS call this, even on error
+8. Call `browser_close`
+
+## Return
+
+Return: {test_name} | {status} | {summary}
+```
+
+After each agent completes, note the result. If the agent fails to call `complete_run`, call it yourself with status "error".
 
 ### Step 8: Handle unscripted tests without scripts
 
-Any tests that didn't get scripts generated in Step 3 (e.g. if script generation failed) need to be executed the same way as Step 7 — one at a time using the AI agent approach. Follow the same pattern: get instructions, start run, execute in browser, complete run, close browser.
+Any tests that didn't get scripts generated in Step 3 (e.g. if script generation failed) need to be executed the same way as Step 7 — launch a Task agent for each one sequentially using the AI fallback agent prompt above (omit the "Native execution failed with" line).
 
 ## Summarize