playwright-test-agent 0.1.9 → 0.2.2

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "playwright-test-agent",
-  "version": "0.1.9",
+  "version": "0.2.2",
   "description": "Initialize Playwright Test agents with a Playwright CLI-first browser workflow.",
   "type": "module",
   "bin": {

package/playwright-test-agent/SKILL.md

@@ -5,39 +5,45 @@ description: Use first when a user asks to investigate or test a website, browse
 
 # Playwright Test Agent
 
-Use Planner -> Generator -> Healer for durable tests. `AGENTS.md` or `CLAUDE.md` defines the platform-specific role routing. Generated role definitions use `playwright-cli` as their primary browser interface, so MCP tool injection is not required.
+Use Planner -> Generator -> Healer for durable tests. `AGENTS.md` or `CLAUDE.md` defines the platform-specific role routing. The role that owns the current phase uses `playwright-cli`; the parent does not perform another role's browser work.
 
-## Fast start
+## Required phase handoff
 
-When the request contains a test objective and deployed URL, enter Planner immediately and run:
+1. Planner investigates the application and saves a Markdown plan under `specs/`.
+2. The parent presents that plan and waits for user confirmation.
+3. Generator validates the confirmed plan and writes executable tests.
+4. The parent executes the generated tests with `npx playwright test`.
+5. If all generated tests pass, report the result. If any fail, Healer diagnoses, fixes justified test defects, and reruns the affected tests until they pass or a genuine blocker is identified.
 
-```bash
-playwright-cli open <deployed-url>
-```
+Every phase is required in that order. Planner exploration without a saved plan is incomplete. Generator output without test files is incomplete. Generated test files that have not been executed are not a successful test result. Healer must never start before a real generated-test failure exists.
 
-Before opening the page, do not inspect the local project and do not locate or read a seed file, Playwright config, test directory, agent definition, fixtures, source tree, or existing tests. Do not ask optional questions about roles, browsers, edge cases, expected results, or test data when Planner can discover or defer them.
+On Claude Code, the parent must start `playwright-test-planner`; only that Planner may open the website. The parent must not call `playwright-cli`, Chrome DevTools, browser MCP tools, or another browser integration. On Codex, the main agent does not spawn a Playwright subagent; it explicitly assumes one phase at a time, and may call CLI only while acting in that phase. Preserve the official Playwright Test MCP configuration and role tools; CLI is the preferred live-browser path and prevents missing MCP tools from blocking the workflow.
 
-Ask before Planner only when the objective or deployed URL itself is missing. During exploration, ask only when a real blocker is reached: required credentials, an invalid URL, ambiguous expected behavior that changes assertions, or authorization for a consequential action. Resume Planner after the answer. Do not guess credentials, probe nearby environments, bypass access controls, or wander through unrelated pages.
+Before Planner, the parent may read relevant local project information to understand routes, requirements, fixtures, and constraints. It may ask for information that is necessary or could materially change the test purpose, scope, assertions, access, or safety, such as the deployed URL, account or role, expected behavior, target environment, allowed data mutations, and authorization boundary. Keep this preparation focused: do not locate a seed file or turn config, test-directory, agent-definition, fixture, or generated-file discovery into a prerequisite.
+
+If the available context is sufficient, start/enter Planner without further confirmation. During exploration, pause when required information is missing, incorrect, or contradictory, including credentials, URL, role, prerequisites, expected behavior that changes assertions, or authorization for a consequential action. Ask the user for the specific correction, then resume or restart Planner with the answer and blocking observation. Do not guess credentials, probe nearby environments, bypass access controls, or wander through unrelated pages.
 
 Keep credentials and tokens in environment variables or ignored secret files. Never copy them into plans, tests, screenshots, evidence, or chat.
 
 ## Planner
 
-Planner investigates the live application with `playwright-cli` and saves a human-readable Markdown plan under `specs/`. Start with the supplied URL, then use compact `snapshot` or `find` output and refs for interaction. A seed is not a Planner prerequisite and must not be located before exploration.
+Planner—not the parent—opens the supplied URL with `playwright-cli open <deployed-url>`, investigates the application, converts its findings into a human-readable Markdown test plan, and saves it under `specs/`. Once the parent has supplied the objective, URL, access information, relevant project context, and constraints, Planner's first browser action must be that CLI command. It then uses compact `snapshot` or `find` output and refs for interaction. A seed is not a prerequisite and must not be located before exploration.
 
 The plan contains prerequisites, test data, independent scenarios, steps, observable expected results, exclusions, and intended output files. Reconnaissance must not mutate durable/shared data or perform consequential actions unless authorized.
 
-Show the scenario list and exclusions to the user. Generator starts only after confirmation.
+Planner returns the saved plan path and a scenario summary. The parent shows the plan and exclusions to the user. Generator starts only after the user explicitly confirms that plan.
 
 ## Generator
 
-Generator converts the confirmed plan into executable tests. Use `playwright-cli` to validate flows, locators, and assertions against the live application, then write the tests directly. Generate only confirmed scenarios; prefer independent tests, semantic locators, explicit assertions, and environment-based secrets.
+Generator starts only after confirmation and converts the confirmed plan into executable Playwright test code. Generator—not the parent—uses `playwright-cli` to validate flows, locators, and assertions, writes the test files, and returns their paths. Generate only confirmed scenarios; prefer independent tests, semantic locators, explicit assertions, and environment-based secrets.
+
+After Generator returns, the parent executes the generated test files with `npx playwright test`. This execution step is mandatory. If all tests pass, report the executed result. If any fail, pass the failed test names, failure output, confirmed plan, and generated file paths to Healer.
 
 For API-only scenarios, use Playwright `APIRequestContext` directly and assert status, headers, schema, stable business invariants, and safe mutation cleanup.
 
 ## Healer
 
-Use Healer only for failures. Run the failing test with `npx playwright test`, reproduce and inspect the current UI with `playwright-cli`, apply justified test-side changes, and rerun within guardrails.
+Use Healer only for real failures from executing generated tests. Healer—not the parent—reproduces the failing tests with `npx playwright test`, inspects the UI with `playwright-cli`, applies justified test-side changes, and reruns the affected tests within guardrails until they pass or a genuine blocker is identified.
 
 Do not weaken assertions, add arbitrary sleeps, retry blindly, or skip merely to pass. If behavior conflicts with the confirmed plan, preserve evidence and ask whether it is a regression or intended change. Classify unresolved failures as application defect, test defect, environment/data problem, or product decision.
 

package/playwright-test-agent/scripts/init-playwright.mjs

@@ -13,24 +13,44 @@ const cliFirstInstructions = (role) => `${ROLE_BLOCK_START}
 
 ## Browser tool priority
 
-Use the installed \`playwright-cli\` command as the primary browser interface. Read the globally installed \`playwright-cli\` skill when available. Start or attach a session with \`playwright-cli open\`/\`attach\`, navigate with \`goto\`, inspect compact state with \`snapshot\`, and interact through refs from the latest snapshot. Prefer \`snapshot\` and \`find\` over screenshots. Keep credentials in environment variables or ignored files and never print them.
+Use the installed \`playwright-cli\` command as the primary browser interface for this role. Read the globally installed \`playwright-cli\` skill when available. Start or attach a session with \`playwright-cli open\`/\`attach\`, navigate with \`goto\`, inspect compact state with \`snapshot\`, and interact through refs from the latest snapshot. Prefer \`snapshot\` and \`find\` over screenshots. Keep credentials in environment variables or ignored files and never print them.
 
-CLI replaces any conflicting requirement above to call Playwright Test MCP setup, browser, save, generation, or debug tools. Do not stop merely because \`planner_*\`, \`generator_*\`, \`test_*\`, or \`browser_*\` MCP tools are absent, and do not spawn a nested or same-role agent.
+Keep the official Playwright Test MCP configuration and tools generated for this role; do not remove or disable them. Prefer CLI for live page interaction and use it when those MCP tools are unavailable, so missing \`planner_*\`, \`generator_*\`, \`test_*\`, or \`browser_*\` tools must not block the phase. Do not call Chrome DevTools or an unrelated browser integration, and do not spawn a nested or same-role agent.
 
-${role === 'planner' ? 'FAST START: when the request contains a test objective and deployed URL, immediately run `playwright-cli open <url>` and investigate. Do not first locate or read a seed file, Playwright config, test directory, agent definition, source tree, fixtures, or existing tests. Do not ask optional planning questions before opening the page. Ask only when exploration reaches a real blocker such as required credentials, missing/incorrect URL, ambiguous expected behavior that changes assertions, or authorization for a consequential action. Save the completed Markdown test plan directly under `specs/` using filesystem tools.' : ''}${role === 'generator' ? 'Use CLI to validate the confirmed plan against the live UI, then write executable Playwright test files directly with filesystem tools.' : ''}${role === 'healer' ? 'Run failures with `npx playwright test`, use CLI to inspect and reproduce the current UI, patch the tests directly, and rerun within the healer guardrails.' : ''}
+${role === 'planner' ? 'FAST START: you are the Planner. The parent may pass relevant local-project context and answers gathered before this phase. When the request contains the test objective, deployed URL, and any information required to access the target, your first browser action must be `playwright-cli open <url>`; do not call `planner_setup_page` or any other browser/MCP action before this CLI open. After the session is open, use compact CLI `snapshot`/`find` output for exploration; MCP tools remain available as an optional supplement. Do not wait for the parent agent to open a browser, and do not first locate or read a seed file, Playwright config, test directory, agent definition, source tree, fixtures, or existing tests. Ask only when the supplied information is still missing, incorrect, contradictory, or exploration reaches a real blocker such as required credentials, ambiguous expected behavior that changes assertions, or authorization for a consequential action. Investigation is not the final output: you must turn the findings into a complete Markdown test plan and save it under `specs/` using filesystem tools or the available planner save tool. Return the saved plan path and scenario summary to the parent.' : ''}${role === 'generator' ? 'You are the Generator. Start only after the user confirms the saved test plan. Begin live validation with `playwright-cli open`/`attach` and use CLI snapshots/find to verify the confirmed plan; MCP setup and browser tools remain optional. Generate executable Playwright test code for the confirmed scenarios and write the test files using filesystem tools or the available generator write tool. Return the generated test file paths to the parent; generating code does not complete the workflow because the parent must execute the generated tests next.' : ''}${role === 'healer' ? 'You are the Healer. Start only after execution of the generated tests reports failures. Receive the failing test names and failure output, reproduce them with `npx playwright test`, begin UI diagnosis with `playwright-cli open`/`attach`, and use CLI snapshots/find to inspect the current UI; MCP tools remain optional. Diagnose and patch justified test defects, rerun the affected tests, and continue within the healer guardrails until they pass or a genuine application/environment/product blocker is identified. Return the final run result and classification to the parent.' : ''}
 
 ${ROLE_BLOCK_END}`;
 const CODEX_INSTRUCTIONS = `${BLOCK_START}
 ## Playwright Test Agent
 
-For website, browser workflow, HTTP API, or application-feature testing, load \`.agents/skills/playwright-test-agent/SKILL.md\` first. Codex must not start Playwright subagents; the main agent performs Planner -> Generator -> Healer in this session. Use \`playwright-cli\` as the primary browser interface. When the request contains a test objective and deployed URL, begin Planner immediately; do not first inspect the project or locate a seed, config, test directory, agent definition, fixtures, or existing tests. Ask only at a real access, expected-behavior, or safety blocker. Do not guess or create nested agents.
+Codex routing is mandatory for website, browser workflow, HTTP API, or application-feature testing:
+
+1. Load \`.agents/skills/playwright-test-agent/SKILL.md\`.
+2. Before Planner, the main agent may read relevant local project information and ask for information that materially affects the test, such as the deployed URL, account or role, expected behavior, environment, and authorization boundary. Do not perform a seed/config/generated-file audit or open the website during preparation.
+3. Do not start Playwright subagents. Once the required context is available, the main agent explicitly assumes the Planner role in this session.
+4. Planner's first live browser action is \`playwright-cli open <url>\`. Prefer CLI while preserving the generated Playwright Test MCP configuration and tools. Do not call Chrome DevTools or an unrelated browser integration.
+5. Planner must convert its investigation into a complete Markdown test plan under \`specs/\`. Show that saved plan to the user and wait for explicit confirmation; investigation alone is not completion.
+6. After confirmation, assume Generator and generate executable Playwright test code for the confirmed plan. When generation finishes, execute the generated tests with \`npx playwright test\`; generating files alone is not completion.
+7. If every generated test passes, report the result. If any generated test fails, assume Healer with the failed test names and output; diagnose, patch, and rerun until the tests pass or a genuine application/environment/product blocker is identified. Never skip execution and never report generated-but-unexecuted tests as passing.
+
+Ask only for information that is necessary or could materially change scope, assertions, access, or safety. Do not guess or create nested agents.
 
 ${BLOCK_END}`;
 
 const CLAUDE_INSTRUCTIONS = `${BLOCK_START}
 ## Playwright Test Agent
 
-For website, browser workflow, HTTP API, or application-feature testing, load \`.claude/skills/playwright-test-agent/SKILL.md\` first. Claude Code starts the official \`playwright-test-planner\`, \`playwright-test-generator\`, and \`playwright-test-healer\` directly. When the request contains a test objective and deployed URL, start Planner immediately; do not first inspect the project or locate a seed, config, test directory, agent definition, fixtures, or existing tests. These roles use \`playwright-cli\` as their primary browser interface and must not create nested agents. Ask only at a real access, expected-behavior, or safety blocker; do not guess.
+Claude Code routing is mandatory for website, browser workflow, HTTP API, or application-feature testing:
+
+1. Load \`.claude/skills/playwright-test-agent/SKILL.md\`.
+2. Before starting Planner, the main agent may read relevant local project information and ask for information that materially affects the test, such as the deployed URL, account or role, expected behavior, environment, and authorization boundary. Do not perform a seed/config/generated-file audit.
+3. Once the required context is available, directly start \`playwright-test-planner\` with the objective, deployed URL, supplied access information, relevant project context, and constraints. The main agent must not open or inspect the website itself and must not call \`playwright-cli\`, Chrome DevTools, browser MCP tools, or another browser integration.
+4. Planner opens and investigates the website with \`playwright-cli\`, converts its findings into a complete Markdown test plan under \`specs/\`, and returns the saved plan path. After Planner returns, show that plan to the user and wait for explicit confirmation; investigation alone is not completion.
+5. Only after confirmation, directly start \`playwright-test-generator\` with the confirmed plan. Generator must validate the confirmed scenarios and write executable Playwright test files, then return their paths.
+6. After Generator returns, the main agent must execute the generated tests with \`npx playwright test\`. Generating test files alone never completes the workflow.
+7. If every generated test passes, report the result. If any generated test fails, directly start \`playwright-test-healer\` with the failed test names, failure output, confirmed plan, and generated file paths. Healer diagnoses and patches justified test defects and reruns the affected tests until they pass or it identifies a genuine application/environment/product blocker. Never report generated-but-unexecuted tests as passing.
+
+Start each role directly and never create a nested or same-role intermediary. Do not locate a seed file or make setup-file discovery a prerequisite to Planner. If exploration finds missing, incorrect, or contradictory required information, let Planner pause and return the precise question; after the user answers, start or resume Planner with that answer and the blocking observation. Do not guess.
 
 ${BLOCK_END}`;
 
@@ -141,11 +161,11 @@ function patchClaudeRole(source, role) {
     const remainder = header.slice(blockStart);
     const nextKey = remainder.search(/\r?\n(?=[A-Za-z0-9_-]+:\s*)/);
     const blockEnd = nextKey === -1 ? header.length : blockStart + nextKey;
-    const toolsBlock = header.slice(blockStart, blockEnd);
+    let toolsBlock = header.slice(blockStart, blockEnd);
     if (!/^\s*-\s+Bash\s*$/m.test(toolsBlock)) {
-      header = header.slice(0, blockStart) + '\n  - Bash' +
-        header.slice(blockStart);
+      toolsBlock = `\n  - Bash${toolsBlock}`;
     }
+    header = header.slice(0, blockStart) + toolsBlock + header.slice(blockEnd);
   } else {
     header = `${header.trimEnd()}\ntools: Bash`;
   }
@@ -171,6 +191,61 @@ async function configureCliFirstRoles(projectDir) {
   }
 }
 
+function escapeRegExp(value) {
+  return value.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
+}
+
+function upsertTomlSection(source, section, settings) {
+  const headerPattern = new RegExp(`^\\[${escapeRegExp(section)}\\][ \\t]*$`, 'm');
+  const header = source.match(headerPattern);
+
+  if (!header) {
+    const body = Object.entries(settings).map(([key, value]) => `${key} = ${value}`).join('\n');
+    const separator = source.length === 0 ? '' : source.endsWith('\n') ? '\n' : '\n\n';
+    return `${source}${separator}[${section}]\n${body}\n`;
+  }
+
+  const bodyStart = header.index + header[0].length;
+  const remainder = source.slice(bodyStart);
+  const nextHeader = remainder.search(/\r?\n(?=\[[^\]]+\][ \\t]*(?:\r?\n|$))/);
+  const bodyEnd = nextHeader === -1 ? source.length : bodyStart + nextHeader;
+  let body = source.slice(bodyStart, bodyEnd);
+
+  for (const [key, value] of Object.entries(settings)) {
+    const settingPattern = new RegExp(`(^|\\n)${escapeRegExp(key)}[ \\t]*=[^\\r\\n]*`);
+    if (settingPattern.test(body)) {
+      body = body.replace(settingPattern, `$1${key} = ${value}`);
+    } else {
+      body = `${body.trimEnd()}\n${key} = ${value}\n`;
+    }
+  }
+
+  return source.slice(0, bodyStart) + body + source.slice(bodyEnd);
+}
+
+async function configureCodexMcp(projectDir) {
+  const codexDir = path.join(projectDir, '.codex');
+  const file = path.join(codexDir, 'config.toml');
+  await mkdir(codexDir, { recursive: true });
+  let source = await exists(file) ? await readFile(file, 'utf8') : '';
+  const command = process.platform === 'win32' ? '"cmd"' : '"npx"';
+  const args = process.platform === 'win32'
+    ? '["/d", "/s", "/c", "npx", "--no-install", "playwright", "run-test-mcp-server"]'
+    : '["--no-install", "playwright", "run-test-mcp-server"]';
+
+  source = upsertTomlSection(source, 'mcp_servers.playwright-test', {
+    command,
+    args,
+    cwd: '"."',
+    enabled: 'true',
+    default_tools_approval_mode: '"approve"',
+  });
+  source = upsertTomlSection(source, 'mcp_servers.playwright-test.env', {
+    PLAYWRIGHT_MCP_OUTPUT_DIR: '".playwright-evidence/mcp"',
+  });
+  await writeFile(file, source, 'utf8');
+}
+
 async function installPlaywrightCli(projectDir, run) {
   await run(projectDir, 'npm', ['install', '-g', '@playwright/cli@latest']);
   await run(projectDir, 'playwright-cli', ['install', '--skills=agents', '--global']);
@@ -222,33 +297,7 @@ async function disableTraceScreenshots(projectDir) {
   }
 }
 
-async function configureClaudeMcp(projectDir) {
-  const file = path.join(projectDir, '.mcp.json');
-  if (!await exists(file)) return;
-  const config = JSON.parse(await readFile(file, 'utf8'));
-  const server = config.mcpServers?.['playwright-test'];
-  if (!server) return;
-  server.env = {
-    ...(server.env ?? {}),
-    PLAYWRIGHT_MCP_OUTPUT_DIR: '.playwright-evidence/mcp',
-  };
-  await writeFile(file, `${JSON.stringify(config, null, 2)}\n`, 'utf8');
-}
-
-async function configureCodexAgents(projectDir) {
-  for (const name of ['planner', 'generator', 'healer']) {
-    const file = path.join(projectDir, '.codex', 'agents', `playwright_test_${name}.toml`);
-    if (!await exists(file)) continue;
-    let source = await readFile(file, 'utf8');
-    if (!source.includes('[mcp_servers.playwright-test.env]')) {
-      source = `${source.trimEnd()}\n\n[mcp_servers.playwright-test.env]\n` +
-        'PLAYWRIGHT_MCP_OUTPUT_DIR = ".playwright-evidence/mcp"\n';
-      await writeFile(file, source, 'utf8');
-    }
-  }
-}
-
-async function initializePlaywright(projectDir, run) {
+async function initializePlaywright(projectDir, run) {
   const configPath = path.join(projectDir, 'playwright.config.ts');
   if (!await exists(configPath)) {
     await run(projectDir, 'npm', [
@@ -272,12 +321,11 @@ async function initializePlaywright(projectDir, run) {
   await disableTraceScreenshots(projectDir);
 
   await run(projectDir, 'npx', ['--no-install', 'playwright', 'init-agents', '--loop=codex']);
-  await run(projectDir, 'npx', ['--no-install', 'playwright', 'init-agents', '--loop=claude']);
-  await mkdir(path.join(projectDir, '.playwright-evidence', 'mcp'), { recursive: true });
-  await configureClaudeMcp(projectDir);
-  await configureCodexAgents(projectDir);
+  await run(projectDir, 'npx', ['--no-install', 'playwright', 'init-agents', '--loop=claude']);
+  await mkdir(path.join(projectDir, '.playwright-evidence', 'mcp'), { recursive: true });
+  await configureCodexMcp(projectDir);
   await configureCliFirstRoles(projectDir);
-}
+}
 
 async function runStage(name, action) {
   process.stdout.write(`[playwright-test-agent] ${name}...\n`);