playwright-test-agent 0.1.7 → 0.2.1

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "playwright-test-agent",
-  "version": "0.1.7",
+  "version": "0.2.1",
   "description": "Initialize Playwright Test agents with a Playwright CLI-first browser workflow.",
   "type": "module",
   "bin": {

package/playwright-test-agent/SKILL.md

@@ -5,21 +5,26 @@ description: Use first when a user asks to investigate or test a website, browse
 
 # Playwright Test Agent
 
-Use Planner -> Generator -> Healer for durable tests. `AGENTS.md` or `CLAUDE.md` defines the platform-specific role routing. Generated role definitions use `playwright-cli` as their primary browser interface, so MCP tool injection is not required.
+Use Planner -> Generator -> Healer for durable tests. `AGENTS.md` or `CLAUDE.md` defines the platform-specific role routing. The role that owns the current phase uses `playwright-cli`; the parent does not perform another role's browser work.
 
-## Prepare the objective
+## Required phase handoff
 
-The main agent may inspect the local project for relevant requirements, routes, fixtures, and constraints. Do not require a seed/config audit before planning.
+1. Planner investigates the application and saves a Markdown plan under `specs/`.
+2. The parent presents that plan and waits for user confirmation.
+3. Generator validates the confirmed plan and writes executable tests.
+4. Healer runs only for failing generated tests.
 
-Before Planner, ask for information that could materially change purpose, scope, safety, or expected behavior: deployed URL, intended role, success criteria, allowed data mutations, and environment constraints. Do not ask for UI details Planner can safely discover.
+On Claude Code, the parent must start `playwright-test-planner`; only that Planner may open the website. The parent must not call `playwright-cli`, Chrome DevTools, browser MCP tools, or another browser integration. On Codex, the main agent does not spawn a Playwright subagent; it explicitly assumes one phase at a time, and may call CLI only while acting in that phase. Preserve the official Playwright Test MCP configuration and role tools; CLI is the preferred live-browser path and prevents missing MCP tools from blocking the workflow.
 
-If the request is sufficient, begin Planner immediately. If investigation finds a missing, incorrect, or contradictory URL, credential, role, prerequisite, expected result, or authorization boundary, pause and ask for the specific correction. Resume Planner with the correction and blocking observation. Do not guess credentials, probe nearby environments, bypass access controls, or wander through unrelated pages.
+Before Planner, the parent may read relevant local project information to understand routes, requirements, fixtures, and constraints. It may ask for information that is necessary or could materially change the test purpose, scope, assertions, access, or safety, such as the deployed URL, account or role, expected behavior, target environment, allowed data mutations, and authorization boundary. Keep this preparation focused: do not locate a seed file or turn config, test-directory, agent-definition, fixture, or generated-file discovery into a prerequisite.
+
+If the available context is sufficient, start/enter Planner without further confirmation. During exploration, pause when required information is missing, incorrect, or contradictory, including credentials, URL, role, prerequisites, expected behavior that changes assertions, or authorization for a consequential action. Ask the user for the specific correction, then resume or restart Planner with the answer and blocking observation. Do not guess credentials, probe nearby environments, bypass access controls, or wander through unrelated pages.
 
 Keep credentials and tokens in environment variables or ignored secret files. Never copy them into plans, tests, screenshots, evidence, or chat.
 
 ## Planner
 
-Planner investigates the live application with `playwright-cli` and saves a human-readable Markdown plan under `specs/`. Use `open`/`goto`, compact `snapshot` or `find`, and refs for interaction. Supply a seed only when explicitly identified; do not perform a separate seed discovery pass.
+Planner—not the parent—opens the supplied URL with `playwright-cli open <deployed-url>`, investigates the application, and saves a human-readable Markdown plan under `specs/`. Once the parent has supplied the objective, URL, access information, relevant project context, and constraints, Planner's first browser action must be that CLI command. It then uses compact `snapshot` or `find` output and refs for interaction. A seed is not a prerequisite and must not be located before exploration.
 
 The plan contains prerequisites, test data, independent scenarios, steps, observable expected results, exclusions, and intended output files. Reconnaissance must not mutate durable/shared data or perform consequential actions unless authorized.
 
@@ -27,13 +32,13 @@ Show the scenario list and exclusions to the user. Generator starts only after c
 
 ## Generator
 
-Generator converts the confirmed plan into executable tests. Use `playwright-cli` to validate flows, locators, and assertions against the live application, then write the tests directly. Generate only confirmed scenarios; prefer independent tests, semantic locators, explicit assertions, and environment-based secrets.
+Generator starts only after confirmation and converts the confirmed plan into executable tests. Generator—not the parent—uses `playwright-cli` to validate flows, locators, and assertions, then writes the tests directly. Generate only confirmed scenarios; prefer independent tests, semantic locators, explicit assertions, and environment-based secrets.
 
 For API-only scenarios, use Playwright `APIRequestContext` directly and assert status, headers, schema, stable business invariants, and safe mutation cleanup.
 
 ## Healer
 
-Use Healer only for failures. Run the failing test with `npx playwright test`, reproduce and inspect the current UI with `playwright-cli`, apply justified test-side changes, and rerun within guardrails.
+Use Healer only for failures. Healer—not the parent—runs the failing test with `npx playwright test`, reproduces and inspects the UI with `playwright-cli`, applies justified test-side changes, and reruns within guardrails.
 
 Do not weaken assertions, add arbitrary sleeps, retry blindly, or skip merely to pass. If behavior conflicts with the confirmed plan, preserve evidence and ask whether it is a regression or intended change. Classify unresolved failures as application defect, test defect, environment/data problem, or product decision.
 

package/playwright-test-agent/scripts/init-playwright.mjs

@@ -13,24 +13,40 @@ const cliFirstInstructions = (role) => `${ROLE_BLOCK_START}
 
 ## Browser tool priority
 
-Use the installed \`playwright-cli\` command as the primary browser interface. Read the globally installed \`playwright-cli\` skill when available. Start or attach a session with \`playwright-cli open\`/\`attach\`, navigate with \`goto\`, inspect compact state with \`snapshot\`, and interact through refs from the latest snapshot. Prefer \`snapshot\` and \`find\` over screenshots. Keep credentials in environment variables or ignored files and never print them.
+Use the installed \`playwright-cli\` command as the primary browser interface for this role. Read the globally installed \`playwright-cli\` skill when available. Start or attach a session with \`playwright-cli open\`/\`attach\`, navigate with \`goto\`, inspect compact state with \`snapshot\`, and interact through refs from the latest snapshot. Prefer \`snapshot\` and \`find\` over screenshots. Keep credentials in environment variables or ignored files and never print them.
 
-CLI replaces any conflicting requirement above to call Playwright Test MCP setup, browser, save, generation, or debug tools. Do not stop merely because \`planner_*\`, \`generator_*\`, \`test_*\`, or \`browser_*\` MCP tools are absent, and do not spawn a nested or same-role agent.
+Keep the official Playwright Test MCP configuration and tools generated for this role; do not remove or disable them. Prefer CLI for live page interaction and use it when those MCP tools are unavailable, so missing \`planner_*\`, \`generator_*\`, \`test_*\`, or \`browser_*\` tools must not block the phase. Do not call Chrome DevTools or an unrelated browser integration, and do not spawn a nested or same-role agent.
 
-${role === 'planner' ? 'Explore with CLI and save the completed Markdown test plan directly under `specs/` using filesystem tools.' : ''}${role === 'generator' ? 'Use CLI to validate the confirmed plan against the live UI, then write executable Playwright test files directly with filesystem tools.' : ''}${role === 'healer' ? 'Run failures with `npx playwright test`, use CLI to inspect and reproduce the current UI, patch the tests directly, and rerun within the healer guardrails.' : ''}
+${role === 'planner' ? 'FAST START: you are the Planner. The parent may pass relevant local-project context and answers gathered before this phase. When the request contains the test objective, deployed URL, and any information required to access the target, your first browser action must be `playwright-cli open <url>`; do not call `planner_setup_page` or any other browser/MCP action before this CLI open. After the session is open, use compact CLI `snapshot`/`find` output for exploration; MCP tools remain available as an optional supplement. Do not wait for the parent agent to open a browser, and do not first locate or read a seed file, Playwright config, test directory, agent definition, source tree, fixtures, or existing tests. Ask only when the supplied information is still missing, incorrect, contradictory, or exploration reaches a real blocker such as required credentials, ambiguous expected behavior that changes assertions, or authorization for a consequential action. Save the completed Markdown test plan directly under `specs/` using filesystem tools or the available planner save tool.' : ''}${role === 'generator' ? 'You are the Generator. Start only after the parent agent confirms the plan. Begin live validation with `playwright-cli open`/`attach` and use CLI snapshots/find to verify the confirmed plan; MCP setup and browser tools remain optional. Then write executable Playwright test files directly with filesystem tools or the available generator write tool.' : ''}${role === 'healer' ? 'You are the Healer. Run failures with `npx playwright test`, begin UI diagnosis with `playwright-cli open`/`attach`, and use CLI snapshots/find to inspect and reproduce the current UI; MCP tools remain optional. Patch the tests directly and rerun within the healer guardrails.' : ''}
 
 ${ROLE_BLOCK_END}`;
 const CODEX_INSTRUCTIONS = `${BLOCK_START}
 ## Playwright Test Agent
 
-For website, browser workflow, HTTP API, or application-feature testing, load \`.agents/skills/playwright-test-agent/SKILL.md\` first. Codex must not start Playwright subagents; the main agent performs Planner -> Generator -> Healer in this session. Use \`playwright-cli\` as the primary browser interface and read generated TOML role definitions only for responsibilities. Ask for missing or contradictory purpose, access, role, data, or safety information before or during planning; do not guess or create nested agents.
+Codex routing is mandatory for website, browser workflow, HTTP API, or application-feature testing:
+
+1. Load \`.agents/skills/playwright-test-agent/SKILL.md\`.
+2. Before Planner, the main agent may read relevant local project information and ask for information that materially affects the test, such as the deployed URL, account or role, expected behavior, environment, and authorization boundary. Do not perform a seed/config/generated-file audit or open the website during preparation.
+3. Do not start Playwright subagents. Once the required context is available, the main agent explicitly assumes the Planner role in this session.
+4. Planner's first live browser action is \`playwright-cli open <url>\`. Prefer CLI while preserving the generated Playwright Test MCP configuration and tools. Do not call Chrome DevTools or an unrelated browser integration.
+5. After Planner saves the plan, show it to the user and wait for confirmation. Then assume Generator; use Healer only for failing generated tests.
+
+Ask only for information that is necessary or could materially change scope, assertions, access, or safety. Do not guess or create nested agents.
 
 ${BLOCK_END}`;
 
 const CLAUDE_INSTRUCTIONS = `${BLOCK_START}
 ## Playwright Test Agent
 
-For website, browser workflow, HTTP API, or application-feature testing, load \`.claude/skills/playwright-test-agent/SKILL.md\` first. Claude Code starts the official \`playwright-test-planner\`, \`playwright-test-generator\`, and \`playwright-test-healer\` directly. These roles use \`playwright-cli\` as their primary browser interface and must not create nested agents. Ask for missing or contradictory purpose, access, role, data, or safety information before or during planning; do not guess.
+Claude Code routing is mandatory for website, browser workflow, HTTP API, or application-feature testing:
+
+1. Load \`.claude/skills/playwright-test-agent/SKILL.md\`.
+2. Before starting Planner, the main agent may read relevant local project information and ask for information that materially affects the test, such as the deployed URL, account or role, expected behavior, environment, and authorization boundary. Do not perform a seed/config/generated-file audit.
+3. Once the required context is available, directly start \`playwright-test-planner\` with the objective, deployed URL, supplied access information, relevant project context, and constraints. The main agent must not open or inspect the website itself and must not call \`playwright-cli\`, Chrome DevTools, browser MCP tools, or another browser integration.
+4. Planner opens and investigates the website with \`playwright-cli\` and saves the Markdown plan. After Planner returns, show the plan to the user and wait for confirmation.
+5. Only after confirmation start \`playwright-test-generator\`. Start \`playwright-test-healer\` only for failing generated tests.
+
+Start each role directly and never create a nested or same-role intermediary. Do not locate a seed file or make setup-file discovery a prerequisite to Planner. If exploration finds missing, incorrect, or contradictory required information, let Planner pause and return the precise question; after the user answers, start or resume Planner with that answer and the blocking observation. Do not guess.
 
 ${BLOCK_END}`;
 
@@ -126,13 +142,31 @@ function patchCodexRole(source, role) {
 
 function patchClaudeRole(source, role) {
   let updated = source;
-  if (!/^\s*-\s+Bash\(playwright-cli:\*\)\s*$/m.test(updated)) {
-    const tools = updated.match(/^tools:\s*\r?\n/m);
-    if (!tools) throw new Error('tools list was not found in a Claude role definition');
-    const insertion = '  - Bash(playwright-cli:*)\n  - Bash(npx:*)\n';
-    updated = updated.slice(0, tools.index + tools[0].length) + insertion +
-      updated.slice(tools.index + tools[0].length);
+  const frontmatter = updated.match(/^---\s*\r?\n([\s\S]*?)\r?\n---/);
+  if (!frontmatter) throw new Error('YAML frontmatter was not found in a Claude role definition');
+
+  let header = frontmatter[1];
+  const inlineTools = header.match(/^tools:[ \t]*(\S.*?)[ \t]*$/m);
+  const blockTools = header.match(/^tools:[ \t]*$/m);
+  if (inlineTools) {
+    const tools = inlineTools[1].split(',').map((tool) => tool.trim());
+    if (!tools.includes('Bash')) tools.push('Bash');
+    header = header.replace(inlineTools[0], `tools: ${tools.join(', ')}`);
+  } else if (blockTools) {
+    const blockStart = blockTools.index + blockTools[0].length;
+    const remainder = header.slice(blockStart);
+    const nextKey = remainder.search(/\r?\n(?=[A-Za-z0-9_-]+:\s*)/);
+    const blockEnd = nextKey === -1 ? header.length : blockStart + nextKey;
+    let toolsBlock = header.slice(blockStart, blockEnd);
+    if (!/^\s*-\s+Bash\s*$/m.test(toolsBlock)) {
+      toolsBlock = `\n  - Bash${toolsBlock}`;
+    }
+    header = header.slice(0, blockStart) + toolsBlock + header.slice(blockEnd);
+  } else {
+    header = `${header.trimEnd()}\ntools: Bash`;
   }
+  updated = updated.slice(0, frontmatter.index) + `---\n${header}\n---` +
+    updated.slice(frontmatter.index + frontmatter[0].length);
   return withRoleBlock(updated, role);
 }
 
@@ -153,6 +187,61 @@ async function configureCliFirstRoles(projectDir) {
   }
 }
 
+function escapeRegExp(value) {
+  return value.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
+}
+
+function upsertTomlSection(source, section, settings) {
+  const headerPattern = new RegExp(`^\\[${escapeRegExp(section)}\\][ \\t]*$`, 'm');
+  const header = source.match(headerPattern);
+
+  if (!header) {
+    const body = Object.entries(settings).map(([key, value]) => `${key} = ${value}`).join('\n');
+    const separator = source.length === 0 ? '' : source.endsWith('\n') ? '\n' : '\n\n';
+    return `${source}${separator}[${section}]\n${body}\n`;
+  }
+
+  const bodyStart = header.index + header[0].length;
+  const remainder = source.slice(bodyStart);
+  const nextHeader = remainder.search(/\r?\n(?=\[[^\]]+\][ \\t]*(?:\r?\n|$))/);
+  const bodyEnd = nextHeader === -1 ? source.length : bodyStart + nextHeader;
+  let body = source.slice(bodyStart, bodyEnd);
+
+  for (const [key, value] of Object.entries(settings)) {
+    const settingPattern = new RegExp(`(^|\\n)${escapeRegExp(key)}[ \\t]*=[^\\r\\n]*`);
+    if (settingPattern.test(body)) {
+      body = body.replace(settingPattern, `$1${key} = ${value}`);
+    } else {
+      body = `${body.trimEnd()}\n${key} = ${value}\n`;
+    }
+  }
+
+  return source.slice(0, bodyStart) + body + source.slice(bodyEnd);
+}
+
+async function configureCodexMcp(projectDir) {
+  const codexDir = path.join(projectDir, '.codex');
+  const file = path.join(codexDir, 'config.toml');
+  await mkdir(codexDir, { recursive: true });
+  let source = await exists(file) ? await readFile(file, 'utf8') : '';
+  const command = process.platform === 'win32' ? '"cmd"' : '"npx"';
+  const args = process.platform === 'win32'
+    ? '["/d", "/s", "/c", "npx", "--no-install", "playwright", "run-test-mcp-server"]'
+    : '["--no-install", "playwright", "run-test-mcp-server"]';
+
+  source = upsertTomlSection(source, 'mcp_servers.playwright-test', {
+    command,
+    args,
+    cwd: '"."',
+    enabled: 'true',
+    default_tools_approval_mode: '"approve"',
+  });
+  source = upsertTomlSection(source, 'mcp_servers.playwright-test.env', {
+    PLAYWRIGHT_MCP_OUTPUT_DIR: '".playwright-evidence/mcp"',
+  });
+  await writeFile(file, source, 'utf8');
+}
+
 async function installPlaywrightCli(projectDir, run) {
   await run(projectDir, 'npm', ['install', '-g', '@playwright/cli@latest']);
   await run(projectDir, 'playwright-cli', ['install', '--skills=agents', '--global']);
@@ -204,33 +293,7 @@ async function disableTraceScreenshots(projectDir) {
   }
 }
 
-async function configureClaudeMcp(projectDir) {
-  const file = path.join(projectDir, '.mcp.json');
-  if (!await exists(file)) return;
-  const config = JSON.parse(await readFile(file, 'utf8'));
-  const server = config.mcpServers?.['playwright-test'];
-  if (!server) return;
-  server.env = {
-    ...(server.env ?? {}),
-    PLAYWRIGHT_MCP_OUTPUT_DIR: '.playwright-evidence/mcp',
-  };
-  await writeFile(file, `${JSON.stringify(config, null, 2)}\n`, 'utf8');
-}
-
-async function configureCodexAgents(projectDir) {
-  for (const name of ['planner', 'generator', 'healer']) {
-    const file = path.join(projectDir, '.codex', 'agents', `playwright_test_${name}.toml`);
-    if (!await exists(file)) continue;
-    let source = await readFile(file, 'utf8');
-    if (!source.includes('[mcp_servers.playwright-test.env]')) {
-      source = `${source.trimEnd()}\n\n[mcp_servers.playwright-test.env]\n` +
-        'PLAYWRIGHT_MCP_OUTPUT_DIR = ".playwright-evidence/mcp"\n';
-      await writeFile(file, source, 'utf8');
-    }
-  }
-}
-
-async function initializePlaywright(projectDir, run) {
+async function initializePlaywright(projectDir, run) {
   const configPath = path.join(projectDir, 'playwright.config.ts');
   if (!await exists(configPath)) {
     await run(projectDir, 'npm', [
@@ -254,12 +317,11 @@ async function initializePlaywright(projectDir, run) {
   await disableTraceScreenshots(projectDir);
 
   await run(projectDir, 'npx', ['--no-install', 'playwright', 'init-agents', '--loop=codex']);
-  await run(projectDir, 'npx', ['--no-install', 'playwright', 'init-agents', '--loop=claude']);
-  await mkdir(path.join(projectDir, '.playwright-evidence', 'mcp'), { recursive: true });
-  await configureClaudeMcp(projectDir);
-  await configureCodexAgents(projectDir);
+  await run(projectDir, 'npx', ['--no-install', 'playwright', 'init-agents', '--loop=claude']);
+  await mkdir(path.join(projectDir, '.playwright-evidence', 'mcp'), { recursive: true });
+  await configureCodexMcp(projectDir);
   await configureCliFirstRoles(projectDir);
-}
+}
 
 async function runStage(name, action) {
   process.stdout.write(`[playwright-test-agent] ${name}...\n`);