playwright-test-agent 0.1.9 → 0.2.1

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "playwright-test-agent",
-  "version": "0.1.9",
+  "version": "0.2.1",
   "description": "Initialize Playwright Test agents with a Playwright CLI-first browser workflow.",
   "type": "module",
   "bin": {

package/playwright-test-agent/SKILL.md

@@ -5,25 +5,26 @@ description: Use first when a user asks to investigate or test a website, browse
 
 # Playwright Test Agent
 
-Use Planner -> Generator -> Healer for durable tests. `AGENTS.md` or `CLAUDE.md` defines the platform-specific role routing. Generated role definitions use `playwright-cli` as their primary browser interface, so MCP tool injection is not required.
+Use Planner -> Generator -> Healer for durable tests. `AGENTS.md` or `CLAUDE.md` defines the platform-specific role routing. The role that owns the current phase uses `playwright-cli`; the parent does not perform another role's browser work.
 
-## Fast start
+## Required phase handoff
 
-When the request contains a test objective and deployed URL, enter Planner immediately and run:
+1. Planner investigates the application and saves a Markdown plan under `specs/`.
+2. The parent presents that plan and waits for user confirmation.
+3. Generator validates the confirmed plan and writes executable tests.
+4. Healer runs only for failing generated tests.
 
-```bash
-playwright-cli open <deployed-url>
-```
+On Claude Code, the parent must start `playwright-test-planner`; only that Planner may open the website. The parent must not call `playwright-cli`, Chrome DevTools, browser MCP tools, or another browser integration. On Codex, the main agent does not spawn a Playwright subagent; it explicitly assumes one phase at a time, and may call CLI only while acting in that phase. Preserve the official Playwright Test MCP configuration and role tools; CLI is the preferred live-browser path and prevents missing MCP tools from blocking the workflow.
 
-Before opening the page, do not inspect the local project and do not locate or read a seed file, Playwright config, test directory, agent definition, fixtures, source tree, or existing tests. Do not ask optional questions about roles, browsers, edge cases, expected results, or test data when Planner can discover or defer them.
+Before Planner, the parent may read relevant local project information to understand routes, requirements, fixtures, and constraints. It may ask for information that is necessary or could materially change the test purpose, scope, assertions, access, or safety, such as the deployed URL, account or role, expected behavior, target environment, allowed data mutations, and authorization boundary. Keep this preparation focused: do not locate a seed file or turn config, test-directory, agent-definition, fixture, or generated-file discovery into a prerequisite.
 
-Ask before Planner only when the objective or deployed URL itself is missing. During exploration, ask only when a real blocker is reached: required credentials, an invalid URL, ambiguous expected behavior that changes assertions, or authorization for a consequential action. Resume Planner after the answer. Do not guess credentials, probe nearby environments, bypass access controls, or wander through unrelated pages.
+If the available context is sufficient, start/enter Planner without further confirmation. During exploration, pause when required information is missing, incorrect, or contradictory, including credentials, URL, role, prerequisites, expected behavior that changes assertions, or authorization for a consequential action. Ask the user for the specific correction, then resume or restart Planner with the answer and blocking observation. Do not guess credentials, probe nearby environments, bypass access controls, or wander through unrelated pages.
 
 Keep credentials and tokens in environment variables or ignored secret files. Never copy them into plans, tests, screenshots, evidence, or chat.
 
 ## Planner
 
-Planner investigates the live application with `playwright-cli` and saves a human-readable Markdown plan under `specs/`. Start with the supplied URL, then use compact `snapshot` or `find` output and refs for interaction. A seed is not a Planner prerequisite and must not be located before exploration.
+Planner—not the parent—opens the supplied URL with `playwright-cli open <deployed-url>`, investigates the application, and saves a human-readable Markdown plan under `specs/`. Once the parent has supplied the objective, URL, access information, relevant project context, and constraints, Planner's first browser action must be that CLI command. It then uses compact `snapshot` or `find` output and refs for interaction. A seed is not a prerequisite and must not be located before exploration.
 
 The plan contains prerequisites, test data, independent scenarios, steps, observable expected results, exclusions, and intended output files. Reconnaissance must not mutate durable/shared data or perform consequential actions unless authorized.
 
@@ -31,13 +32,13 @@ Show the scenario list and exclusions to the user. Generator starts only after c
 
 ## Generator
 
-Generator converts the confirmed plan into executable tests. Use `playwright-cli` to validate flows, locators, and assertions against the live application, then write the tests directly. Generate only confirmed scenarios; prefer independent tests, semantic locators, explicit assertions, and environment-based secrets.
+Generator starts only after confirmation and converts the confirmed plan into executable tests. Generator—not the parent—uses `playwright-cli` to validate flows, locators, and assertions, then writes the tests directly. Generate only confirmed scenarios; prefer independent tests, semantic locators, explicit assertions, and environment-based secrets.
 
 For API-only scenarios, use Playwright `APIRequestContext` directly and assert status, headers, schema, stable business invariants, and safe mutation cleanup.
 
 ## Healer
 
-Use Healer only for failures. Run the failing test with `npx playwright test`, reproduce and inspect the current UI with `playwright-cli`, apply justified test-side changes, and rerun within guardrails.
+Use Healer only for failures. Healer—not the parent—runs the failing test with `npx playwright test`, reproduces and inspects the UI with `playwright-cli`, applies justified test-side changes, and reruns within guardrails.
 
 Do not weaken assertions, add arbitrary sleeps, retry blindly, or skip merely to pass. If behavior conflicts with the confirmed plan, preserve evidence and ask whether it is a regression or intended change. Classify unresolved failures as application defect, test defect, environment/data problem, or product decision.
 

package/playwright-test-agent/scripts/init-playwright.mjs

@@ -13,24 +13,40 @@ const cliFirstInstructions = (role) => `${ROLE_BLOCK_START}
 
 ## Browser tool priority
 
-Use the installed \`playwright-cli\` command as the primary browser interface. Read the globally installed \`playwright-cli\` skill when available. Start or attach a session with \`playwright-cli open\`/\`attach\`, navigate with \`goto\`, inspect compact state with \`snapshot\`, and interact through refs from the latest snapshot. Prefer \`snapshot\` and \`find\` over screenshots. Keep credentials in environment variables or ignored files and never print them.
+Use the installed \`playwright-cli\` command as the primary browser interface for this role. Read the globally installed \`playwright-cli\` skill when available. Start or attach a session with \`playwright-cli open\`/\`attach\`, navigate with \`goto\`, inspect compact state with \`snapshot\`, and interact through refs from the latest snapshot. Prefer \`snapshot\` and \`find\` over screenshots. Keep credentials in environment variables or ignored files and never print them.
 
-CLI replaces any conflicting requirement above to call Playwright Test MCP setup, browser, save, generation, or debug tools. Do not stop merely because \`planner_*\`, \`generator_*\`, \`test_*\`, or \`browser_*\` MCP tools are absent, and do not spawn a nested or same-role agent.
+Keep the official Playwright Test MCP configuration and tools generated for this role; do not remove or disable them. Prefer CLI for live page interaction and use it when those MCP tools are unavailable, so missing \`planner_*\`, \`generator_*\`, \`test_*\`, or \`browser_*\` tools must not block the phase. Do not call Chrome DevTools or an unrelated browser integration, and do not spawn a nested or same-role agent.
 
-${role === 'planner' ? 'FAST START: when the request contains a test objective and deployed URL, immediately run `playwright-cli open <url>` and investigate. Do not first locate or read a seed file, Playwright config, test directory, agent definition, source tree, fixtures, or existing tests. Do not ask optional planning questions before opening the page. Ask only when exploration reaches a real blocker such as required credentials, missing/incorrect URL, ambiguous expected behavior that changes assertions, or authorization for a consequential action. Save the completed Markdown test plan directly under `specs/` using filesystem tools.' : ''}${role === 'generator' ? 'Use CLI to validate the confirmed plan against the live UI, then write executable Playwright test files directly with filesystem tools.' : ''}${role === 'healer' ? 'Run failures with `npx playwright test`, use CLI to inspect and reproduce the current UI, patch the tests directly, and rerun within the healer guardrails.' : ''}
+${role === 'planner' ? 'FAST START: you are the Planner. The parent may pass relevant local-project context and answers gathered before this phase. When the request contains the test objective, deployed URL, and any information required to access the target, your first browser action must be `playwright-cli open <url>`; do not call `planner_setup_page` or any other browser/MCP action before this CLI open. After the session is open, use compact CLI `snapshot`/`find` output for exploration; MCP tools remain available as an optional supplement. Do not wait for the parent agent to open a browser, and do not first locate or read a seed file, Playwright config, test directory, agent definition, source tree, fixtures, or existing tests. Ask only when the supplied information is still missing, incorrect, contradictory, or exploration reaches a real blocker such as required credentials, ambiguous expected behavior that changes assertions, or authorization for a consequential action. Save the completed Markdown test plan directly under `specs/` using filesystem tools or the available planner save tool.' : ''}${role === 'generator' ? 'You are the Generator. Start only after the parent agent confirms the plan. Begin live validation with `playwright-cli open`/`attach` and use CLI snapshots/find to verify the confirmed plan; MCP setup and browser tools remain optional. Then write executable Playwright test files directly with filesystem tools or the available generator write tool.' : ''}${role === 'healer' ? 'You are the Healer. Run failures with `npx playwright test`, begin UI diagnosis with `playwright-cli open`/`attach`, and use CLI snapshots/find to inspect and reproduce the current UI; MCP tools remain optional. Patch the tests directly and rerun within the healer guardrails.' : ''}
 
 ${ROLE_BLOCK_END}`;
 const CODEX_INSTRUCTIONS = `${BLOCK_START}
 ## Playwright Test Agent
 
-For website, browser workflow, HTTP API, or application-feature testing, load \`.agents/skills/playwright-test-agent/SKILL.md\` first. Codex must not start Playwright subagents; the main agent performs Planner -> Generator -> Healer in this session. Use \`playwright-cli\` as the primary browser interface. When the request contains a test objective and deployed URL, begin Planner immediately; do not first inspect the project or locate a seed, config, test directory, agent definition, fixtures, or existing tests. Ask only at a real access, expected-behavior, or safety blocker. Do not guess or create nested agents.
+Codex routing is mandatory for website, browser workflow, HTTP API, or application-feature testing:
+
+1. Load \`.agents/skills/playwright-test-agent/SKILL.md\`.
+2. Before Planner, the main agent may read relevant local project information and ask for information that materially affects the test, such as the deployed URL, account or role, expected behavior, environment, and authorization boundary. Do not perform a seed/config/generated-file audit or open the website during preparation.
+3. Do not start Playwright subagents. Once the required context is available, the main agent explicitly assumes the Planner role in this session.
+4. Planner's first live browser action is \`playwright-cli open <url>\`. Prefer CLI while preserving the generated Playwright Test MCP configuration and tools. Do not call Chrome DevTools or an unrelated browser integration.
+5. After Planner saves the plan, show it to the user and wait for confirmation. Then assume Generator; use Healer only for failing generated tests.
+
+Ask only for information that is necessary or could materially change scope, assertions, access, or safety. Do not guess or create nested agents.
 
 ${BLOCK_END}`;
 
 const CLAUDE_INSTRUCTIONS = `${BLOCK_START}
 ## Playwright Test Agent
 
-For website, browser workflow, HTTP API, or application-feature testing, load \`.claude/skills/playwright-test-agent/SKILL.md\` first. Claude Code starts the official \`playwright-test-planner\`, \`playwright-test-generator\`, and \`playwright-test-healer\` directly. When the request contains a test objective and deployed URL, start Planner immediately; do not first inspect the project or locate a seed, config, test directory, agent definition, fixtures, or existing tests. These roles use \`playwright-cli\` as their primary browser interface and must not create nested agents. Ask only at a real access, expected-behavior, or safety blocker; do not guess.
+Claude Code routing is mandatory for website, browser workflow, HTTP API, or application-feature testing:
+
+1. Load \`.claude/skills/playwright-test-agent/SKILL.md\`.
+2. Before starting Planner, the main agent may read relevant local project information and ask for information that materially affects the test, such as the deployed URL, account or role, expected behavior, environment, and authorization boundary. Do not perform a seed/config/generated-file audit.
+3. Once the required context is available, directly start \`playwright-test-planner\` with the objective, deployed URL, supplied access information, relevant project context, and constraints. The main agent must not open or inspect the website itself and must not call \`playwright-cli\`, Chrome DevTools, browser MCP tools, or another browser integration.
+4. Planner opens and investigates the website with \`playwright-cli\` and saves the Markdown plan. After Planner returns, show the plan to the user and wait for confirmation.
+5. Only after confirmation start \`playwright-test-generator\`. Start \`playwright-test-healer\` only for failing generated tests.
+
+Start each role directly and never create a nested or same-role intermediary. Do not locate a seed file or make setup-file discovery a prerequisite to Planner. If exploration finds missing, incorrect, or contradictory required information, let Planner pause and return the precise question; after the user answers, start or resume Planner with that answer and the blocking observation. Do not guess.
 
 ${BLOCK_END}`;
 
@@ -141,11 +157,11 @@ function patchClaudeRole(source, role) {
     const remainder = header.slice(blockStart);
     const nextKey = remainder.search(/\r?\n(?=[A-Za-z0-9_-]+:\s*)/);
     const blockEnd = nextKey === -1 ? header.length : blockStart + nextKey;
-    const toolsBlock = header.slice(blockStart, blockEnd);
+    let toolsBlock = header.slice(blockStart, blockEnd);
     if (!/^\s*-\s+Bash\s*$/m.test(toolsBlock)) {
-      header = header.slice(0, blockStart) + '\n  - Bash' +
-        header.slice(blockStart);
+      toolsBlock = `\n  - Bash${toolsBlock}`;
     }
+    header = header.slice(0, blockStart) + toolsBlock + header.slice(blockEnd);
   } else {
     header = `${header.trimEnd()}\ntools: Bash`;
   }
@@ -171,6 +187,61 @@ async function configureCliFirstRoles(projectDir) {
   }
 }
 
+function escapeRegExp(value) {
+  return value.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
+}
+
+function upsertTomlSection(source, section, settings) {
+  const headerPattern = new RegExp(`^\\[${escapeRegExp(section)}\\][ \\t]*$`, 'm');
+  const header = source.match(headerPattern);
+
+  if (!header) {
+    const body = Object.entries(settings).map(([key, value]) => `${key} = ${value}`).join('\n');
+    const separator = source.length === 0 ? '' : source.endsWith('\n') ? '\n' : '\n\n';
+    return `${source}${separator}[${section}]\n${body}\n`;
+  }
+
+  const bodyStart = header.index + header[0].length;
+  const remainder = source.slice(bodyStart);
+  const nextHeader = remainder.search(/\r?\n(?=\[[^\]]+\][ \\t]*(?:\r?\n|$))/);
+  const bodyEnd = nextHeader === -1 ? source.length : bodyStart + nextHeader;
+  let body = source.slice(bodyStart, bodyEnd);
+
+  for (const [key, value] of Object.entries(settings)) {
+    const settingPattern = new RegExp(`(^|\\n)${escapeRegExp(key)}[ \\t]*=[^\\r\\n]*`);
+    if (settingPattern.test(body)) {
+      body = body.replace(settingPattern, `$1${key} = ${value}`);
+    } else {
+      body = `${body.trimEnd()}\n${key} = ${value}\n`;
+    }
+  }
+
+  return source.slice(0, bodyStart) + body + source.slice(bodyEnd);
+}
+
+async function configureCodexMcp(projectDir) {
+  const codexDir = path.join(projectDir, '.codex');
+  const file = path.join(codexDir, 'config.toml');
+  await mkdir(codexDir, { recursive: true });
+  let source = await exists(file) ? await readFile(file, 'utf8') : '';
+  const command = process.platform === 'win32' ? '"cmd"' : '"npx"';
+  const args = process.platform === 'win32'
+    ? '["/d", "/s", "/c", "npx", "--no-install", "playwright", "run-test-mcp-server"]'
+    : '["--no-install", "playwright", "run-test-mcp-server"]';
+
+  source = upsertTomlSection(source, 'mcp_servers.playwright-test', {
+    command,
+    args,
+    cwd: '"."',
+    enabled: 'true',
+    default_tools_approval_mode: '"approve"',
+  });
+  source = upsertTomlSection(source, 'mcp_servers.playwright-test.env', {
+    PLAYWRIGHT_MCP_OUTPUT_DIR: '".playwright-evidence/mcp"',
+  });
+  await writeFile(file, source, 'utf8');
+}
+
 async function installPlaywrightCli(projectDir, run) {
   await run(projectDir, 'npm', ['install', '-g', '@playwright/cli@latest']);
   await run(projectDir, 'playwright-cli', ['install', '--skills=agents', '--global']);
@@ -222,33 +293,7 @@ async function disableTraceScreenshots(projectDir) {
   }
 }
 
-async function configureClaudeMcp(projectDir) {
-  const file = path.join(projectDir, '.mcp.json');
-  if (!await exists(file)) return;
-  const config = JSON.parse(await readFile(file, 'utf8'));
-  const server = config.mcpServers?.['playwright-test'];
-  if (!server) return;
-  server.env = {
-    ...(server.env ?? {}),
-    PLAYWRIGHT_MCP_OUTPUT_DIR: '.playwright-evidence/mcp',
-  };
-  await writeFile(file, `${JSON.stringify(config, null, 2)}\n`, 'utf8');
-}
-
-async function configureCodexAgents(projectDir) {
-  for (const name of ['planner', 'generator', 'healer']) {
-    const file = path.join(projectDir, '.codex', 'agents', `playwright_test_${name}.toml`);
-    if (!await exists(file)) continue;
-    let source = await readFile(file, 'utf8');
-    if (!source.includes('[mcp_servers.playwright-test.env]')) {
-      source = `${source.trimEnd()}\n\n[mcp_servers.playwright-test.env]\n` +
-        'PLAYWRIGHT_MCP_OUTPUT_DIR = ".playwright-evidence/mcp"\n';
-      await writeFile(file, source, 'utf8');
-    }
-  }
-}
-
-async function initializePlaywright(projectDir, run) {
+async function initializePlaywright(projectDir, run) {
   const configPath = path.join(projectDir, 'playwright.config.ts');
   if (!await exists(configPath)) {
     await run(projectDir, 'npm', [
@@ -272,12 +317,11 @@ async function initializePlaywright(projectDir, run) {
   await disableTraceScreenshots(projectDir);
 
   await run(projectDir, 'npx', ['--no-install', 'playwright', 'init-agents', '--loop=codex']);
-  await run(projectDir, 'npx', ['--no-install', 'playwright', 'init-agents', '--loop=claude']);
-  await mkdir(path.join(projectDir, '.playwright-evidence', 'mcp'), { recursive: true });
-  await configureClaudeMcp(projectDir);
-  await configureCodexAgents(projectDir);
+  await run(projectDir, 'npx', ['--no-install', 'playwright', 'init-agents', '--loop=claude']);
+  await mkdir(path.join(projectDir, '.playwright-evidence', 'mcp'), { recursive: true });
+  await configureCodexMcp(projectDir);
   await configureCliFirstRoles(projectDir);
-}
+}
 
 async function runStage(name, action) {
   process.stdout.write(`[playwright-test-agent] ${name}...\n`);