playwright-test-agent 1.0.2 → 1.0.4

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "playwright-test-agent",
-  "version": "1.0.2",
+  "version": "1.0.4",
   "description": "Initialize Playwright Test agents with a Playwright CLI-first browser workflow.",
   "type": "module",
   "bin": {

package/playwright-test-agent/SKILL.md

@@ -23,7 +23,7 @@ Before Planner, the parent may read relevant local project information to unders
 
 If the available context is sufficient, start/enter Planner without further confirmation. During exploration, pause when required information is missing, incorrect, or contradictory, including credentials, URL, role, prerequisites, expected behavior that changes assertions, or authorization for a consequential action. Ask the user for the specific correction, then resume or restart Planner with the answer and blocking observation. Do not guess credentials, probe nearby environments, bypass access controls, or wander through unrelated pages.
 
-Keep credentials and tokens in environment variables or ignored secret files. Never copy them into plans, tests, screenshots, evidence, or chat. Browser snapshots and session artifacts produced by `playwright-cli` under `.playwright-cli` are reusable workflow state: preserve them across Planner, Generator, and Healer phases by default. Do not clean up snapshots, sessions, or other transient browser artifacts merely because they were created during validation. Remove or redact only a file that is confirmed to contain an exposed credential/token, or when the user explicitly requests cleanup.
+Keep all runtime configuration—including URLs, accounts, passwords, tokens, and feature flags—in the project-root `.env` file. Commit only `.env.example` with variable names and safe placeholders; `.env` must be ignored by Git. Load it through the project's Playwright/Node configuration (for example, `dotenv/config`) so settings persist for the project without relying on global or one-shot shell environment variables. Never copy secrets into plans, tests, screenshots, evidence, or chat. Browser snapshots and session artifacts produced by `playwright-cli` under `.playwright-cli` are reusable workflow state: preserve them across Planner, Generator, and Healer phases by default. Do not clean up snapshots, sessions, or other transient browser artifacts merely because they were created during validation. Remove or redact only a file that is confirmed to contain an exposed credential/token, or when the user explicitly requests cleanup.
 
 ## Planner
 
@@ -37,7 +37,7 @@ Planner, Generator, and Healer may reuse the existing `.playwright-cli` session
 
 ## Generator
 
-Generator starts only after confirmation and converts the confirmed plan into executable Playwright test code. Pass the resolved Playwright `testDir` to Generator. Generator—not the parent—uses `playwright-cli` to validate flows, locators, and assertions, writes every new test inside that configured `testDir`, and returns their paths. Generic `tests/` paths in generated role examples, plans, or seed references do not override the configured directory. Generate only confirmed scenarios; prefer independent tests, semantic locators, explicit assertions, and environment-based secrets.
+Generator starts only after confirmation and converts the confirmed plan into executable Playwright test code. Pass the resolved Playwright `testDir` to Generator. Generator—not the parent—uses `playwright-cli` to validate flows, locators, and assertions, writes every new test inside that configured `testDir`, and returns their paths. Generic `tests/` paths in generated role examples, plans, or seed references do not override the configured directory. Generate only confirmed scenarios; prefer independent tests, semantic locators, explicit assertions, and values loaded from the project's `.env` file.
 
 After Generator returns, the parent executes the generated test files with `npx playwright test`. This execution step is mandatory. If all tests pass, report the executed result. If any fail, pass the failed test names, failure output, confirmed plan, and generated file paths to Healer.
 
@@ -49,11 +49,21 @@ Use Healer only for real failures from executing generated tests. Healer—not t
 
 Do not weaken assertions, add arbitrary sleeps, retry blindly, or skip merely to pass. If behavior conflicts with the confirmed plan, preserve evidence and ask whether it is a regression or intended change. Classify unresolved failures as application defect, test defect, environment/data problem, or product decision.
 
+For every failed test, provide a concise failure summary before the classification. Extract the first actionable error (and its relevant locator, URL, or assertion when present) and map it to one primary reason:
+
+- **Element location failure** — locator found no element, timed out waiting for a locator, or strict mode matched multiple elements.
+- **Timeout/navigation or network failure** — page/action/API timed out, navigation failed, or the target was unreachable.
+- **Assertion failure** — the expected value, text, state, count, or response did not match the observed result.
+- **Test data/environment failure** — missing credentials, fixture/seed data, configuration, dependency, or incompatible runtime.
+- **Other** — no rule matches; retain the original error's first meaningful line.
+
+Use the format `Reason: <category> — <one-sentence cause>`. Do not label a failure only as “test failed” and do not hide the original error when the category is uncertain.
+
 ## Evidence
 
 Keep evidence under `.playwright-evidence/`. Prefer DOM snapshots and targeted `find` output; take screenshots only when visual evidence is necessary. Keep trace DOM snapshots, sources, network data, and attachments, but set trace `screenshots: false` to avoid hundreds of screencast JPEGs.
 
-Report the plan path, generated tests, counts when tests ran, evidence paths, and failure classification. Never report skipped or unresolved tests as success.
+Report the plan path, generated tests, counts when tests ran, evidence paths, and, for each failure, the reason summary plus failure classification. Never report skipped or unresolved tests as success.
 
 ## Initialization
 

package/playwright-test-agent/scripts/init-playwright.mjs

@@ -13,11 +13,11 @@ const cliFirstInstructions = (role) => `${ROLE_BLOCK_START}
 
 ## Browser tool priority
 
-Use the installed \`playwright-cli\` command as the primary browser interface for this role. Read the globally installed \`playwright-cli\` skill when available. Start or attach a session with \`playwright-cli open\`/\`attach\`, navigate with \`goto\`, inspect compact state with \`snapshot\`, and interact through refs from the latest snapshot. Prefer \`snapshot\` and \`find\` over screenshots. Keep credentials in environment variables or ignored files and never print them. Preserve \`.playwright-cli\` snapshots, sessions, and other browser artifacts for reuse by later phases; do not delete validation artifacts by default. Only remove or redact a confirmed credential/token leak, or act on an explicit cleanup request.
+Use the installed \`playwright-cli\` command as the primary browser interface for this role. Read the globally installed \`playwright-cli\` skill when available. Start or attach a session with \`playwright-cli open\`/\`attach\`, navigate with \`goto\`, inspect compact state with \`snapshot\`, and interact through refs from the latest snapshot. Prefer \`snapshot\` and \`find\` over screenshots. Keep all runtime configuration in the project-root \`.env\` file (commit only \`.env.example\`); load it through Playwright/Node configuration and never rely on global or one-shot shell variables. Never print secrets. Preserve \`.playwright-cli\` snapshots, sessions, and other browser artifacts for reuse by later phases; do not delete validation artifacts by default. Only remove or redact a confirmed credential/token leak, or act on an explicit cleanup request.
 
 Keep the official Playwright Test MCP configuration and tools generated for this role; do not remove or disable them. Prefer CLI for live page interaction and use it when those MCP tools are unavailable, so missing \`planner_*\`, \`generator_*\`, \`test_*\`, or \`browser_*\` tools must not block the phase. Do not call Chrome DevTools or an unrelated browser integration, and do not spawn a nested or same-role agent.
 
-${role === 'planner' ? 'FAST START: you are the Planner. The parent must first perform a focused preflight of the project and tests under Playwright\'s configured `testDir`, then pass the findings and any user-provided answers to you. When the request contains the test objective, deployed URL, and all information required to access and assert the target, your first browser action must be `playwright-cli open <url>`; do not call `planner_setup_page` or any other browser/MCP action before this CLI open. After the session is open, use compact CLI `snapshot`/`find` output for exploration; MCP tools remain available as an optional supplement. Do not wait for the parent agent to open a browser. If the preflight context is insufficient, missing, or contradictory, stop and return the precise question for the user instead of guessing. Investigation is not the final output: you must turn the findings into a complete Markdown test plan and save it under `specs/` using filesystem tools or the available planner save tool. Return the saved plan path and scenario summary to the parent.' : ''}${role === 'generator' ? 'You are the Generator. Start only after the user confirms the saved test plan. The parent must pass Playwright\'s configured `testDir`; resolve it from `playwright.config.*` yourself if it was omitted. Write every new test file inside that resolved directory. Ignore generic `tests/` paths in upstream role descriptions, examples, plans, or seed references when they conflict with the configured `testDir`. Begin live validation with `playwright-cli open`/`attach` and use CLI snapshots/find to verify the confirmed plan; MCP setup and browser tools remain optional. Generate executable Playwright test code for the confirmed scenarios and write the test files using filesystem tools or the available generator write tool. Return the generated test file paths to the parent; generating code does not complete the workflow because the parent must execute the generated tests next.' : ''}${role === 'healer' ? 'You are the Healer. Start only after execution of the generated tests reports failures. Receive the failing test names and failure output, reproduce them with `npx playwright test`, begin UI diagnosis with `playwright-cli open`/`attach`, and use CLI snapshots/find to inspect the current UI; MCP tools remain optional. Diagnose and patch justified test defects, rerun the affected tests, and continue within the healer guardrails until they pass or a genuine application/environment/product blocker is identified. Return the final run result and classification to the parent.' : ''}
+${role === 'planner' ? 'FAST START: you are the Planner. The parent must first perform a focused preflight of the project and tests under Playwright\'s configured `testDir`, then pass the findings and any user-provided answers to you. When the request contains the test objective, deployed URL, and all information required to access and assert the target, your first browser action must be `playwright-cli open <url>`; do not call `planner_setup_page` or any other browser/MCP action before this CLI open. After the session is open, use compact CLI `snapshot`/`find` output for exploration; MCP tools remain available as an optional supplement. Do not wait for the parent agent to open a browser. If the preflight context is insufficient, missing, or contradictory, stop and return the precise question for the user instead of guessing. Investigation is not the final output: you must turn the findings into a complete Markdown test plan and save it under `specs/` using filesystem tools or the available planner save tool. Return the saved plan path and scenario summary to the parent.' : ''}${role === 'generator' ? 'You are the Generator. Start only after the user confirms the saved test plan. The parent must pass Playwright\'s configured `testDir`; resolve it from `playwright.config.*` yourself if it was omitted. Write every new test file inside that resolved directory. Ignore generic `tests/` paths in upstream role descriptions, examples, plans, or seed references when they conflict with the configured `testDir`. Begin live validation with `playwright-cli open`/`attach` and use CLI snapshots/find to verify the confirmed plan; MCP setup and browser tools remain optional. Generate executable Playwright test code for the confirmed scenarios and write the test files using filesystem tools or the available generator write tool. Return the generated test file paths to the parent; generating code does not complete the workflow because the parent must execute the generated tests next.' : ''}${role === 'healer' ? 'You are the Healer. Start only after execution of the generated tests reports failures. Receive the failing test names and failure output, reproduce them with `npx playwright test`, begin UI diagnosis with `playwright-cli open`/`attach`, and use CLI snapshots/find to inspect the current UI; MCP tools remain optional. Diagnose and patch justified test defects, rerun the affected tests, and continue within the healer guardrails until they pass or a genuine application/environment/product blocker is identified. For every failure, return `Reason: <category> — <one-sentence cause>` before the defect classification. Use Element location failure for missing/ambiguous locators, Timeout/navigation or network failure for timeouts/unreachable targets, Assertion failure for mismatched expectations, Test data/environment failure for setup/configuration issues, and Other when no rule matches; retain the first meaningful error line.' : ''}
 
 ${ROLE_BLOCK_END}`;
 const CODEX_INSTRUCTIONS = `${BLOCK_START}
@@ -31,7 +31,7 @@ Codex routing is mandatory for website, browser workflow, HTTP API, or applicati
 4. Planner opens and investigates the website with \`playwright-cli\`, converts its findings into a complete Markdown test plan under \`specs/\`, and returns the saved plan path. After Planner returns, show that plan to the user and wait for explicit confirmation; investigation alone is not completion.
 5. Only after confirmation, directly start a subagent with \`agent_type: "playwright_test_generator"\`, passing the confirmed plan and resolved \`testDir\`. Generator validates the confirmed scenarios with \`playwright-cli\`, writes every executable test inside that \`testDir\`, and returns their paths.
 6. After Generator returns, the main agent must execute the generated tests with \`npx playwright test\`. Generating test files alone never completes the workflow.
-7. If every generated test passes, report the result. If any generated test fails, directly start a subagent with \`agent_type: "playwright_test_healer"\`, passing the failed test names, failure output, confirmed plan, and generated file paths. Healer diagnoses and patches justified test defects and reruns the affected tests until they pass or it identifies a genuine application/environment/product blocker. Never report generated-but-unexecuted tests as passing.
+7. If every generated test passes, report the result. If any generated test fails, directly start a subagent with \`agent_type: "playwright_test_healer"\`, passing the failed test names, failure output, confirmed plan, and generated file paths. Healer diagnoses and patches justified test defects and reruns the affected tests until they pass or it identifies a genuine application/environment/product blocker. The final report must summarize each failure as \`Reason: <category> — <one-sentence cause>\` (including whether element location failed) before the defect classification. Never report generated-but-unexecuted tests as passing.
 
 Start each role as its configured subagent and never make the main agent perform that role. Do not create a nested or same-role intermediary. Do not locate a seed file or make setup-file discovery a prerequisite to Planner. If exploration finds missing, incorrect, or contradictory required information, let Planner pause and return the precise question; after the user answers, resume the same Planner subagent when possible, or restart it with that answer and the blocking observation. Ask only for information that is necessary or could materially change scope, assertions, access, or safety. Do not guess.
 
@@ -48,7 +48,7 @@ Claude Code routing is mandatory for website, browser workflow, HTTP API, or app
 4. Planner opens and investigates the website with \`playwright-cli\`, converts its findings into a complete Markdown test plan under \`specs/\`, and returns the saved plan path. After Planner returns, show that plan to the user and wait for explicit confirmation; investigation alone is not completion.
 5. Only after confirmation, directly start \`playwright-test-generator\` with the confirmed plan and resolved \`testDir\`. Generator must validate the confirmed scenarios, write every executable test inside that \`testDir\`, and return their paths.
 6. After Generator returns, the main agent must execute the generated tests with \`npx playwright test\`. Generating test files alone never completes the workflow.
-7. If every generated test passes, report the result. If any generated test fails, directly start \`playwright-test-healer\` with the failed test names, failure output, confirmed plan, and generated file paths. Healer diagnoses and patches justified test defects and reruns the affected tests until they pass or it identifies a genuine application/environment/product blocker. Never report generated-but-unexecuted tests as passing.
+7. If every generated test passes, report the result. If any generated test fails, directly start \`playwright-test-healer\` with the failed test names, failure output, confirmed plan, and generated file paths. Healer diagnoses and patches justified test defects, reruns the affected tests, and includes \`Reason: <category> — <one-sentence cause>\` (including element location failures) before the defect classification. Never report generated-but-unexecuted tests as passing.
 
 Start each role directly and never create a nested or same-role intermediary. Do not locate a seed file or make setup-file discovery a prerequisite to Planner. If exploration finds missing, incorrect, or contradictory required information, let Planner pause and return the precise question; after the user answers, start or resume Planner with that answer and the blocking observation. Do not guess.
 
@@ -113,6 +113,27 @@ async function updateInstructionFiles(projectDir) {
   }
 }
 
+async function ensureEnvFiles(projectDir) {
+  const examplePath = path.join(projectDir, '.env.example');
+  if (!await exists(examplePath)) {
+    await writeFile(examplePath,
+      '# Copy this file to .env and fill in project-specific runtime settings.\n' +
+      '# Keep .env local; never commit secrets.\n' +
+      '# BASE_URL=https://example.test\n' +
+      '# TEST_USERNAME=\n' +
+      '# TEST_PASSWORD=\n', 'utf8');
+  }
+
+  const gitignorePath = path.join(projectDir, '.gitignore');
+  const source = await exists(gitignorePath) ? await readFile(gitignorePath, 'utf8') : '';
+  const entries = ['.env', '.env.*', '!.env.example', '.playwright-evidence/', '.playwright-cli/'];
+  const missing = entries.filter((entry) => !source.split(/\r?\n/).includes(entry));
+  if (missing.length > 0) {
+    const prefix = source.length === 0 || source.endsWith('\n') ? source : `${source}\n`;
+    await writeFile(gitignorePath, `${prefix}\n# Playwright Test Agent local configuration\n${missing.join('\n')}\n`, 'utf8');
+  }
+}
+
 function withRoleBlock(source, role) {
   const start = source.indexOf(ROLE_BLOCK_START);
   const end = source.indexOf(ROLE_BLOCK_END);
@@ -253,9 +274,12 @@ async function installPlaywrightCli(projectDir, run) {
   await run(projectDir, 'playwright-cli', ['install-browser']);
 }
 
-async function patchPlaywrightConfig(projectDir) {
-  const configPath = path.join(projectDir, 'playwright.config.ts');
-  let source = await readFile(configPath, 'utf8');
+async function patchPlaywrightConfig(projectDir) {
+  const configPath = path.join(projectDir, 'playwright.config.ts');
+  let source = await readFile(configPath, 'utf8');
+  if (!/^import\s+['"]dotenv\/config['"];?/m.test(source)) {
+    source = `import 'dotenv/config';\n${source}`;
+  }
   source = source.replace(/testDir:\s*['"]\.\/tests['"]/, "testDir: './playwright-tests'");
   source = source.replace(
     /reporter:\s*['"]html['"],?/,
@@ -315,10 +339,15 @@ async function initializePlaywright(projectDir, run) {
       await mkdir(targetDir, { recursive: true });
       await rename(generatedExample, path.join(targetDir, 'example.spec.ts'));
     }
-    await patchPlaywrightConfig(projectDir);
-  }
+  }
+
+  await patchPlaywrightConfig(projectDir);
 
-  await disableTraceScreenshots(projectDir);
+  await disableTraceScreenshots(projectDir);
+
+  if (await exists(path.join(projectDir, 'package.json'))) {
+    await run(projectDir, 'npm', ['install', '--save-dev', 'dotenv']);
+  }
 
   await run(projectDir, 'npx', ['--no-install', 'playwright', 'init-agents', '--loop=codex']);
   await run(projectDir, 'npx', ['--no-install', 'playwright', 'init-agents', '--loop=claude']);
@@ -347,7 +376,8 @@ export async function initializeProject({
   await runStage('installing skill', () => installSkill(target, skillSourceDir));
   await runStage('installing Playwright CLI and browser', () => installPlaywrightCli(target, run));
   await runStage('initializing Playwright Test agents', () => initializePlaywright(target, run));
-  await runStage('updating project instructions', () => updateInstructionFiles(target));
+  await runStage('configuring project .env files', () => ensureEnvFiles(target));
+  await runStage('updating project instructions', () => updateInstructionFiles(target));
 
   process.stdout.write(
     'Playwright Test Agent ready: Playwright CLI and skills installed globally, ' +