playwright-e2e-mcp 0.2.1 → 0.3.0

package/README.md

@@ -1,6 +1,6 @@
 # playwright-e2e-mcp
 
-[![CI](https://github.com/trajectiq-ai/E2E/actions/workflows/ci.yml/badge.svg)](https://github.com/trajectiq-ai/E2E/actions/workflows/ci.yml) [![release](https://img.shields.io/github/v/release/trajectiq-ai/E2E)](https://github.com/trajectiq-ai/E2E/releases) [![MCP Registry](https://img.shields.io/badge/MCP%20Registry-io.github.trajectiq--ai%2FE2E-2563eb)](https://registry.modelcontextprotocol.io/)
+[![CI](https://github.com/xsignon-secure/E2E/actions/workflows/ci.yml/badge.svg)](https://github.com/xsignon-secure/E2E/actions/workflows/ci.yml) [![release](https://img.shields.io/github/v/release/xsignon-secure/E2E)](https://github.com/xsignon-secure/E2E/releases) [![MCP Registry](https://img.shields.io/badge/MCP%20Registry-io.github.xsignon--secure%2FE2E-2563eb)](https://registry.modelcontextprotocol.io/)
 [![M8ven Score](https://m8ven.ai/badge/mcp/trajectiq-ai/e2e)](https://m8ven.ai/mcp/trajectiq-ai/e2e?s=readme)
 
 An [MCP](https://modelcontextprotocol.io) server that **diagnoses and fixes failing Playwright
@@ -26,9 +26,13 @@ What it does that a browser-driving MCP or a raw `npx playwright test` does not:
   (`getByRole('button', { name: 'Proceed to checkout' })`), re-runs the test and reverts the
   edit if it is still red. When nothing on the page plausibly matches, it says the element is
   missing and does not patch the test to hide a real bug.
-- **Flakiness across runs.** Every run is recorded locally; `analyze-history` separates
-  tests that flip (flaky) from ones that started failing (regressed) or always fail (broken),
-  and groups failures that share one cause.
+- **Flakiness with evidence.** `diagnose-flaky` repeats failing tests in one process and
+  gives a verdict per test; with `isolate: true` it re-runs them alone to tell a shared-state
+  bug that only bites in parallel from a race inside the test. Every run is recorded locally;
+  `analyze-history` separates tests that flip (flaky) from ones that started failing
+  (regressed) or always fail (broken), and groups failures that share one cause.
+- **In CI too.** A [GitHub Action and `ci` command](#ci-github-action-and-ci-command) post the
+  same diagnosis, flaky verdict and fix diffs to the job summary and the PR.
 - **Pages behind a login.** Every page tool accepts a Playwright `storageState`, extra
   headers, and click/fill steps to reach a modal or a later step.
 - **Small on the wire.** In the [benchmark](bench/RESULTS.md), diagnosing and fixing one
@@ -48,12 +52,12 @@ work side by side in the same client.
 `https://playwright-e2e-mcp.vercel.app/api/mcp` returns for a client that sends the
 bearer token (without one, only `list-tests` and `get-failure` are listed):
 
-![Token-protected endpoint: initialize handshake and the tool list](https://raw.githubusercontent.com/trajectiq-ai/E2E/main/docs/demo-endpoint.png)
+![Token-protected endpoint: initialize handshake and the tool list](https://raw.githubusercontent.com/xsignon-secure/E2E/main/docs/demo-endpoint.png)
 
 **A real test run** — `run-test` served over stdio by `npx -y playwright-e2e-mcp`
 against the bundled `examples/sample-test.spec.ts` (actual output, unedited):
 
-![run-test result: 4 passed, 0 failed, 5.1s](https://raw.githubusercontent.com/trajectiq-ai/E2E/main/docs/demo-run-test.png)
+![run-test result: 4 passed, 0 failed, 5.1s](https://raw.githubusercontent.com/xsignon-secure/E2E/main/docs/demo-run-test.png)
 
 Images are rendered with `node scripts/gen-demo-images.mjs`: the run-test card is real
 captured output; the endpoint card is an illustration of the authenticated listing.
@@ -66,9 +70,9 @@ and every other MCP client — pick whichever route fits:
 | Route | How |
 | --- | --- |
 | **npm (canonical, fastest)** | `npx -y playwright-e2e-mcp` |
-| **MCP Registry** (registry-aware clients discover it automatically) | `io.github.trajectiq-ai/E2E` — [listing](https://registry.modelcontextprotocol.io/) |
-| **Any client, no npm account needed** | `npx -y github:trajectiq-ai/E2E#v0.2.1` (pin a release tag) |
-| **Claude Desktop, zero Node setup** | double-click the [`.mcpb` extension](https://github.com/trajectiq-ai/E2E/releases) |
+| **MCP Registry** (registry-aware clients discover it automatically) | `io.github.xsignon-secure/E2E` — [listing](https://registry.modelcontextprotocol.io/) |
+| **Any client, no npm account needed** | `npx -y github:xsignon-secure/E2E#v0.3.0` (pin a release tag) |
+| **Claude Desktop, zero Node setup** | double-click the [`.mcpb` extension](https://github.com/xsignon-secure/E2E/releases) |
 | **Remote-only clients (ChatGPT connectors)** | `https://playwright-e2e-mcp.vercel.app/api/mcp` |
 
 One line for the common clients:
@@ -78,6 +82,9 @@ claude mcp add playwright-e2e -- npx -y playwright-e2e-mcp
 code --add-mcp '{"name":"playwright-e2e","command":"npx","args":["-y","playwright-e2e-mcp"]}'
 ```
 
+Only need the fix loop? `--tools=core` (or `PW_MCP_TOOLS=core`) exposes six tools and
+halves the tool-definition context; see [Configuration](#configuration).
+
 Details and per-client config: [Installation](#installation) ·
 [MCP client configuration](#mcp-client-configuration).
 
@@ -283,14 +290,21 @@ and write a red-highlighted diff image for review.
 | Argument | Type | Description |
 | --- | --- | --- |
 | `testFiles` | string[] | Tests to diagnose (`file:line` supported). Defaults to the tests that failed in the most recent run |
-| `runs` | number | Times to run them, `2–10` (default `3`) |
+| `runs` | number | Repetitions, `2–10` (default `3`) |
+| `isolate` | boolean | Also tell parallelism bugs from flakiness: repeat fully parallel (`workers` default `4`), then repeat the failing tests with one worker |
+| `separateProcesses` | boolean | Start a fresh Playwright process per repetition instead of one `--repeat-each` process (slower; for flakes that depend on global setup or a cold start) |
 | `browser` / `headed` / `workers` / `config` | — | As with `run-test` |
-| `timeoutMs` | number | Hard wall-clock limit **per run** (default `120000`) |
+| `timeoutMs` | number | Hard wall-clock limit **per repetition** (default `120000`); one `--repeat-each` process gets this × `runs` |
 | `projectRoot` | string | Project directory |
 
-Every run executes with `--retries=0` and auto-retry disabled, so each result is
-honest evidence. The response contains a per-run table (status, duration, first
-failure), the count of **distinct normalized error signatures**, and one of:
+All repetitions run in **one Playwright process** with `--repeat-each` and
+`--retries=0`, so each attempt is honest evidence and the browser and workers
+start once instead of once per run (about 25% faster on the integration fixture,
+more for suites with expensive setup). A small extra reporter records every
+attempt, because Playwright's JSON report keeps only one result per test.
+
+The verdict is **per test**: a table of passes, failures, duration spread and
+first error for each test, plus the verdict block for the worst one:
 
 - **❌ CONSISTENTLY FAILING** — failed every run (same error → reproducible bug,
   different errors → still broken, just noisy). Fix it; it is not flaky.
@@ -299,7 +313,17 @@ failure), the count of **distinct normalized error signatures**, and one of:
   instability).
 - **✅ NOT REPRODUCING** — passed every re-run; the original failure was one-off.
 
-The last run is stored, so `get-failure` can analyze it immediately afterwards.
+With `isolate: true` a second table compares each failing test **in parallel**
+and **alone**:
+
+| Reading | Meaning |
+| --- | --- |
+| 🔀 fails only in parallel | Passes every run alone: shared state between tests (same account, DB rows, files, ports). Give each worker its own data or run the file serially |
+| ⏱️ flaky even alone | The race is inside the test or the app: missing `await`, asserting before the UI settles |
+| ❌ fails alone too | Not a parallelism problem; a real bug |
+
+Each repetition is recorded in the run history, and the last failing run is
+stored, so `analyze-history` and `get-failure` work right afterwards.
 
 ### `suggest-fix`
 
@@ -367,11 +391,11 @@ Requirements:
 
 **No npm account needed** — install straight from GitHub (the `prepare` script
 builds `dist/` automatically on install). Pin a release tag: an unpinned
-`github:trajectiq-ai/E2E` runs whatever is on the default branch at that moment.
+`github:xsignon-secure/E2E` runs whatever is on the default branch at that moment.
 
 ```bash
-npx -y github:trajectiq-ai/E2E#v0.2.1
-npm install -D github:trajectiq-ai/E2E#v0.2.1 @playwright/test   # or as a project dependency
+npx -y github:xsignon-secure/E2E#v0.3.0
+npm install -D github:xsignon-secure/E2E#v0.3.0 @playwright/test   # or as a project dependency
 npx playwright install chromium
 ```
 
@@ -379,11 +403,11 @@ Or grab the packaged tarball from the repo's **GitHub Releases** page and instal
 it locally:
 
 ```bash
-npm install -D https://github.com/trajectiq-ai/E2E/releases/download/v0.2.1/playwright-e2e-mcp-0.2.1.tgz
+npm install -D https://github.com/xsignon-secure/E2E/releases/download/v0.3.0/playwright-e2e-mcp-0.3.0.tgz
 ```
 
 Listed in the **official [MCP Registry](https://registry.modelcontextprotocol.io/)** as
-`io.github.trajectiq-ai/E2E` — registry-aware clients discover it there, and every
+`io.github.xsignon-secure/E2E` — registry-aware clients discover it there, and every
 `v*` release tag republishes the entry from CI via [`server.json`](server.json).
 
 ### MCP client configuration
@@ -395,7 +419,7 @@ Listed in the **official [MCP Registry](https://registry.modelcontextprotocol.io
   "mcpServers": {
     "playwright-e2e": {
       "command": "npx",
-      "args": ["-y", "github:trajectiq-ai/E2E#v0.2.1"],
+      "args": ["-y", "github:xsignon-secure/E2E#v0.3.0"],
       "env": { "PW_MCP_PROJECT_ROOT": "/absolute/path/to/your/project" }
     }
   }
@@ -409,8 +433,8 @@ when the client launches it somewhere else (e.g. your home directory).
 **Codex / VS Code / Copilot CLIs:**
 
 ```bash
-codex mcp add playwright-e2e -- npx -y github:trajectiq-ai/E2E#v0.2.1
-code --add-mcp '{"name":"playwright-e2e","command":"npx","args":["-y","github:trajectiq-ai/E2E#v0.2.1"]}'
+codex mcp add playwright-e2e -- npx -y github:xsignon-secure/E2E#v0.3.0
+code --add-mcp '{"name":"playwright-e2e","command":"npx","args":["-y","github:xsignon-secure/E2E#v0.3.0"]}'
 ```
 
 Codex's defaults fight this server: the first launch clones the repo and runs
@@ -421,13 +445,13 @@ Codex's defaults fight this server: the first launch clones the repo and runs
 ```toml
 [mcp_servers.playwright-e2e]
 command = "npx"
-args = ["-y", "github:trajectiq-ai/E2E#v0.2.1"]
+args = ["-y", "github:xsignon-secure/E2E#v0.3.0"]
 startup_timeout_sec = 60
 tool_timeout_sec = 600
 ```
 
 **Claude Desktop (one-click):** download and double-click the `.mcpb` Desktop
-Extension attached to the [latest release](https://github.com/trajectiq-ai/E2E/releases) —
+Extension attached to the [latest release](https://github.com/xsignon-secure/E2E/releases) —
 the bundle ships its own dependencies, so no Node setup is required. On install it
 asks you to pick your **project root** (required, no default: choose the project
 folder, not your home directory) and wires it
@@ -436,7 +460,7 @@ into `PW_MCP_PROJECT_ROOT`, so the tools point at a real project from the first
 **Claude Code:**
 
 ```bash
-claude mcp add playwright-e2e -- npx -y github:trajectiq-ai/E2E#v0.2.1
+claude mcp add playwright-e2e -- npx -y github:xsignon-secure/E2E#v0.3.0
 ```
 
 **Gemini CLI / Qwen Code:** paste the `mcpServers` block above into
@@ -545,11 +569,60 @@ Redeploy after a change: merge to `main`. Vercel's Git integration deploys
 every push, so there is no token to manage and no CLI step — watch the
 `Vercel` commit status for the deployment result.
 
+## CI: GitHub Action and `ci` command
+
+The same diagnosis runs without an agent. `playwright-e2e-mcp ci` runs the
+suite (auto-retry once, so flaky tests are reported as flaky, not failed),
+optionally drafts `suggest-fix` diffs and a flaky verdict for the failures, and
+writes one Markdown report to stdout, to `--output`, and to the GitHub job
+summary when `GITHUB_STEP_SUMMARY` is set. Exit code `0` passed, `1` tests
+failed, `2` the run itself failed.
+
+```bash
+npx -y playwright-e2e-mcp ci --flaky-runs 3 --suggest 2 --output report.md
+npx -y playwright-e2e-mcp ci --help
+```
+
+As a GitHub Action, it adds the report to the job summary and keeps one PR
+comment up to date when the run fails:
+
+```yaml
+permissions:
+  contents: read
+  pull-requests: write   # only for the PR comment
+steps:
+  - uses: actions/checkout@v6
+  - uses: actions/setup-node@v6
+    with: { node-version: 24 }
+  - run: npm ci && npx playwright install --with-deps chromium
+  - uses: xsignon-secure/E2E@v0.3.0
+    with:
+      flaky-runs: 3      # 0 skips the flaky verdict
+      suggest: 2         # suggest-fix diffs for the first 2 failures
+      # args: --browser chromium tests/checkout
+```
+
+Inputs: `project-root`, `args`, `flaky-runs`, `suggest`, `comment`,
+`fail-on-failure`, `version`, `github-token`. Outputs: `exit-code`, `report`.
+
+## Docker
+
+```bash
+docker build -t playwright-e2e-mcp .
+docker run -i --rm -v "$PWD:/work" playwright-e2e-mcp            # stdio MCP server
+docker run --rm -v "$PWD:/work" playwright-e2e-mcp ci            # CI report
+```
+
+The image is based on `mcr.microsoft.com/playwright:v1.63.0-noble`, so its
+browsers match Playwright 1.63. Mount your project at `/work`; its own
+`node_modules` provide `@playwright/test`.
+
 ## Configuration
 
 | Environment variable | Default | Purpose |
 | --- | --- | --- |
 | `PW_MCP_PROJECT_ROOT` | server cwd | Default project root for every tool |
+| `PW_MCP_TOOLS` | all | Which tools to expose, as tool names and/or profiles, comma separated (same as the `--tools=` flag). Profiles: `core` (run, status, failure, fix, flaky, list), `triage` (run, status, failure, flaky, history, list), `inspect` (page, selector, visual), `author` (list, page, selector, generate, run). All eleven tool definitions cost about 5.2k tokens of context per request; `core` costs about 2.4k |
 | `PW_MCP_STORAGE_STATE` | — | Default `storageState` file (inside the project) for the page tools |
 | `PW_MCP_HISTORY` | on | `0` stops recording runs to `.playwright-e2e-mcp/history.jsonl` |
 | `PW_MCP_ALLOWED_ROOTS` | — | Extra directories a caller may pass as `projectRoot` (`:`-separated, `;` on Windows). Anything outside these and the default root is rejected |
@@ -685,6 +758,10 @@ npm test        # build + test/run-tests.mjs (unit tests, any Node ≥20)
 npm run e2e     # build + e2e/run.mjs: live MCP ↔ Playwright integration suite
 npm run mcpb    # build + bundle dist/ into playwright-e2e-mcp-<version>.mcpb (+ .sha256)
 npm run mcpb:smoke   # ...then install that bundle in a temp dir, launch it and assert the handshake
+npm run mcpb:sign    # ...then sign it with MCPB_CERT/MCPB_KEY → <version>.signed.mcpb
+npm run skill:zip    # just the archive: dist-skill/playwright-e2e-mcp.zip
+npm run skill:package # the uploadable folder: ../agensi-skill-<version>/ (skill copy, README, zip)
+npm run skill:check  # build, then fail if a packaged copy has drifted from skill/
 ```
 
 Tests cover the report parser (sample Playwright JSON, trace attachments), path utils
@@ -694,11 +771,15 @@ helpers, the **trace reader** (synthetic trace.zip: error, failed action, DOM sn
 `*.network` failed-request parsing, console error/warning events), the **image diff**
 (PNG round-trip, regions, color shift, dimension changes), the **change analyzer**
 (selector extraction, git + mtime paths), the **flaky verdict logic**
-(failure signatures, CONSISTENTLY FAILING / FLAKY / NOT REPRODUCING / NO TESTS RAN),
+(failure signatures, CONSISTENTLY FAILING / FLAKY / NOT REPRODUCING / NO TESTS RAN,
+per-test grouping of repeat attempts and the parallel-vs-alone reading), the `ci`
+argument parser and tool profiles,
 the **HTTP bridge** (token, restricted mode, Host allowlist), the **security
 regressions** (sandbox escapes, argument smuggling, symlink writes, code injection,
 SSRF ranges, env scrubbing, decode limits), the `.mcpb` manifest, the `.mcpb` zip
-extractor, and the MCP config files.
+extractor, the **skill package** (byte-identical rebuilds, a CRLF checkout hashing the
+same, refusal of junk/binary/symlinks, and drift detection between `skill/` and a
+packaged copy), and the MCP config files.
 
 ### Integration suite (`npm run e2e`)
 
@@ -718,13 +799,174 @@ that only appear end-to-end:
 - `get-failure` trace diagnostics: DOM at failure, the 404 network request,
   the `console.error` message, diagnosis and next steps,
 - auto-retry turning a first-run failure into `PASSED (1 flaky)`,
-- `diagnose-flaky` verdict **FLAKY** (2 of 3 runs) with retries disabled,
+- `diagnose-flaky` verdict **FLAKY** (2 of 3 runs) with retries disabled, in one
+  `--repeat-each` process, and `isolate` reading a shared-slot test as "fails only in parallel",
+- `--tools=` profiles trimming `tools/list`, and the `ci` command's exit code, report,
+  `--output` file and GitHub job summary,
 - `generate-e2e-test` writing its scaffold, plus error paths
   (missing test path, unknown tool, unreachable server).
 
 First run needs the browser once: `npx playwright install chromium`.
 CI runs the suite on Ubuntu and Windows (see `.github/workflows/ci.yml`).
 
+### Signing the bundle
+
+```bash
+MCPB_CERT=~/signing/cert.pem MCPB_KEY=~/signing/key.pem npm run mcpb:sign
+```
+
+Signing appends a PKCS#7 signature block to the bundle, so the signed file is
+*the reproducible bundle plus that block*. That ordering is the point: the bytes
+CI proved identical across platforms stay inside the signed file, and
+`mcpb unsign` returns them exactly. `scripts/sign-mcpb.mjs` asserts that round
+trip on every run and deletes its output if it fails, which is what keeps
+`<version>.signed.mcpb` tied to `<version>.mcpb`.
+
+The private key is read from a path you pass in (`--cert`/`--key` or
+`MCPB_CERT`/`MCPB_KEY`) and never from the repository — keep it outside version
+control, and back it up: the same key across releases is what makes the signer
+recognisable.
+
+What signing does and does not buy you is decided by the certificate, not by
+this script. `mcpb verify` and `mcpb info` accept a signature only when the
+certificate chains to something in the OS trust store, so with a self-signed
+certificate a host still reports **Not signed** — the signature is real
+integrity and key continuity, verifiable by the fingerprint the script prints,
+but not third-party identity. A certificate issued by a public CA, or an
+internal one installed on your machines, is what makes hosts report it as
+signed.
+
+### Assembling a release
+
+```bash
+npm run mcpb && npm pack                  # the bundle, then the npm tarball
+npm run release:assemble -- --tag v0.2.3  # sign, checksum and write the notes
+npm run release:publish -- --tag v0.2.3 --assets dist-release/assets \
+  --title-file dist-release/RELEASE_TITLE.txt \
+  --body-file dist-release/RELEASE_BODY.md --download dist-release/downloaded
+npm run release:verify -- v0.2.3 --dir dist-release/downloaded
+npm run release:publish -- --tag v0.2.3 --publish
+```
+
+Pushing a `v*` tag runs exactly that sequence in
+`.github/workflows/release.yml`. The release used to be assembled by hand, and
+the notes were written from what those commands printed: every hash in them was
+a copy, and nothing compared the bundle hash printed there with the bundle that
+shipped.
+
+`scripts/assemble-release.mjs` stages the whole release into
+`dist-release/assets/`: the bundle and its sidecar, the **signed** bundle and the
+signing certificate when a key is configured, the npm tarball, the skill package
+built by the same writer as `npm run skill:zip`, and `SHA256SUMS.txt` covering
+every file it staged. It empties that directory first, so a file left over from
+an earlier run cannot be hashed and published as if this run had produced it.
+
+The notes are half written, half generated. `release-notes/<tag>.md` holds the
+prose — its H1 becomes the release title and has to name the tag — and the script
+appends the asset table, the reproduction recipe carrying the real bundle hash,
+the signer and fingerprint, and the commit list. The prose is held to the same
+rule: a 64-character hex string in it has to be a hash this release publishes,
+and writing an `### Assets` section of your own fails the assembly rather than
+shipping a second, staler table. See [release-notes/README.md](release-notes/README.md).
+
+`scripts/publish-release.mjs` fills the release **as a draft**, uploads the
+assets, downloads every one of them back and compares it byte for byte, and only
+`--publish` makes it public — after `release:verify` has re-derived the release
+from the tag. Re-running is safe: a draft this workflow assembled is refreshed in
+place, while a release it did not assemble, or one already published, is refused
+by name instead of overwritten.
+
+Signing is optional and its absence is stated rather than hidden: without the
+`MCPB_KEY` and `MCPB_CERT` repository secrets the release is assembled and
+published **unsigned**, with a notice in the run and a line saying so in the
+notes. Add both — the private key stays out of the repository and out of the logs:
+
+```bash
+gh secret set MCPB_KEY  --repo xsignon-secure/E2E < ~/signing/key.pem
+gh secret set MCPB_CERT --repo xsignon-secure/E2E < ~/signing/cert.pem
+```
+
+Creating the release and uploading its assets needs `contents: write`, which the
+job requests for itself; if it is refused, the run names the setting to change.
+
+### Verifying a published release
+
+```bash
+git checkout v0.3.0 && npm ci && npm run mcpb
+npm run release:verify -- v0.3.0
+```
+
+The release notes promise that the tag rebuilds the published bundle byte for
+byte. `scripts/verify-release.mjs` checks that promise from a checkout of the
+tag: the published bundle against your fresh build, every checksum the release
+publishes (and that nothing shippable is missing from `SHA256SUMS.txt`), a
+**signed** bundle against the bytes its signature wraps, the **skill package**
+against a rebuild from the same tag, and that the published archive still
+extracts with that version's manifest. The skill package is required of a tag
+whose tree carries the publishing workflow; older tags predate it, so there it
+is checked when present and skipped when not, like the signing certificate.
+`.github/workflows/verify-release.yml` runs it on demand for any past tag, and by
+itself for a release published by hand. A release assembled by
+`.github/workflows/release.yml` runs the same check on its own assets *before*
+it is published — GitHub does not raise `release: published` for a release created
+with the repository's own token, so the check is a step there rather than a
+trigger. The `.tgz` is checked for download integrity only: npm
+records the packer's line endings, so a tarball packed locally on another OS is
+legitimately different.
+
+### Packaging the agent skill
+
+```bash
+npm run skill:package                    # ../agensi-skill-<version>/, ready to upload
+npm run skill:zip                        # only the archive: dist-skill/playwright-e2e-mcp.zip
+npm run skill:check                      # build, then compare every copy
+npm run skill:check -- ../agensi-skill-0.3.0
+```
+
+`skill/` is an [Agent Skill](https://agentskills.io/specification) — the
+instructions an agent follows to drive this server — and the marketplace gets
+the same Markdown as a zip with `playwright-e2e-mcp/` at its root.
+`scripts/package-skill.mjs` produces that zip, so the uploaded copy is built
+rather than assembled by hand, and it refuses what the marketplace rejects: a
+missing `SKILL.md`, a non-text file, a symlink, `Thumbs.db`.
+
+`skill:package` writes the whole folder a release would otherwise assemble by
+hand — the skill copy, `skill/README.md` and the archive — to
+`../agensi-skill-<version>` (or `--out <dir>`), then verifies it with the same
+comparison `check` uses, so a successful exit means the folder really is the
+skill tree. It is idempotent: a folder that already matches is left untouched.
+A folder that has drifted is refused instead of overwritten, because it may
+already be published — bump `version` for a new package, or pass `--force` to
+replace that one.
+
+The archive comes from the same deterministic writer as the `.mcpb`, so the
+same tree always hashes the same, and text payloads are LF-normalised first:
+a CRLF checkout builds the bytes an LF one does. CI runs `skill:check`, generates
+a package folder, and uploads the built zip as an artifact
+(`.github/workflows/ci.yml`).
+
+**Every release publishes the package too.** Assembling a release builds it from
+the release tag and attaches it as `<name>-<version>-skill.zip`, and the
+`.github/workflows/release.yml` run downloads that asset back off the draft and
+byte-compares it with the build before anything is published. The marketplace
+upload is that asset, not a folder someone assembled — a deterministic archive
+means "the same build as the tagged release" is an exact statement, and
+`release:verify` checks it again against a rebuild from the tag.
+`.github/workflows/publish-skill-package.yml` stays for the repair path: run it
+by hand to attach or rebuild the package for a release that is missing it.
+
+`skill:check` is the other half of that, and the part that matters when a
+package has already been uploaded. It builds twice (the bytes must agree),
+reads the archive back and compares its contents with `skill/` file by file,
+then compares a packaged copy — `../agensi-skill-<version>` by default, or a
+path you pass: the package folder, the bare `playwright-e2e-mcp/` folder, or
+the zip on its own. Comparisons are by content, never by archive bytes, because
+a zip written by a platform `zip` tool carries real timestamps that say nothing
+about whether the Markdown inside it is current. Any file that differs, is
+missing or is extra is named on stderr and the command exits 1 — the repository
+copy and the uploaded copy are the same document, so an edit to either one
+should not be invisible.
+
 ### Try the example
 
 With Playwright installed in your project:

package/dist/cli/ci.d.ts

@@ -0,0 +1,29 @@
+/**
+ * `playwright-e2e-mcp ci`: the same diagnosis the MCP tools give an agent,
+ * as a plain command for CI. It runs the suite (auto-retry once, so flaky
+ * tests are reported as flaky rather than failed), optionally repeats the
+ * failures through diagnose-flaky and drafts suggest-fix diffs, and writes
+ * one Markdown report: to stdout, to --output, and to the GitHub job
+ * summary when GITHUB_STEP_SUMMARY is set.
+ *
+ * Exit codes: 0 all passed (flaky allowed), 1 tests failed, 2 the run
+ * itself failed (no Playwright, config error, timeout) or bad arguments.
+ */
+export declare const CI_USAGE: string;
+export interface CiOptions {
+    projectRoot?: string;
+    config?: string;
+    browser?: 'chromium' | 'firefox' | 'webkit';
+    grep?: string;
+    workers?: number;
+    timeoutMs: number;
+    retry: boolean;
+    flakyRuns?: number;
+    suggest: number;
+    output?: string;
+    testFiles: string[];
+}
+/** Parse `ci` arguments (everything after the subcommand). Throws on bad input. */
+export declare function parseCiArgs(argv: string[]): CiOptions;
+/** Run the CI flow; returns the exit code. */
+export declare function runCi(argv: string[]): Promise<number>;

package/dist/cli/ci.js

@@ -0,0 +1,145 @@
+/**
+ * `playwright-e2e-mcp ci`: the same diagnosis the MCP tools give an agent,
+ * as a plain command for CI. It runs the suite (auto-retry once, so flaky
+ * tests are reported as flaky rather than failed), optionally repeats the
+ * failures through diagnose-flaky and drafts suggest-fix diffs, and writes
+ * one Markdown report: to stdout, to --output, and to the GitHub job
+ * summary when GITHUB_STEP_SUMMARY is set.
+ *
+ * Exit codes: 0 all passed (flaky allowed), 1 tests failed, 2 the run
+ * itself failed (no Playwright, config error, timeout) or bad arguments.
+ */
+import { appendFile, writeFile } from 'node:fs/promises';
+import { parseArgs } from 'node:util';
+import { createLogger } from '../utils/logger.js';
+import { createToolStore } from '../tools/shared.js';
+import { runTestTool } from '../tools/run-test.js';
+import { diagnoseFlakyTool } from '../tools/diagnose-flaky.js';
+import { suggestFixTool } from '../tools/suggest-fix.js';
+export const CI_USAGE = [
+    'Usage: playwright-e2e-mcp ci [options] [test files…]',
+    '',
+    'Runs the Playwright suite and writes a Markdown diagnosis (stdout, --output,',
+    'and $GITHUB_STEP_SUMMARY when set). Exit 0 = passed, 1 = tests failed, 2 = run error.',
+    '',
+    '  --project-root <dir>   project directory (default: PW_MCP_PROJECT_ROOT or cwd)',
+    '  --config <path|n>      playwright.config path or 1-based index',
+    '  --browser <name>       chromium | firefox | webkit',
+    '  --grep <regex>         only tests whose title matches',
+    '  --workers <n>          Playwright workers',
+    '  --timeout <ms>         wall-clock limit for the run (default 600000)',
+    '  --no-retry             do not auto-retry failures once',
+    '  --flaky-runs <n>       repeat failing tests n times (2-10) for a flaky verdict',
+    '  --suggest <n>          draft suggest-fix diffs for the first n failures (max 5)',
+    '  --output <file>        also write the report to this file (for a PR comment)',
+    '',
+].join('\n');
+function intOption(value, name, min, max) {
+    if (value === undefined)
+        return undefined;
+    const n = Number(value);
+    if (!Number.isInteger(n) || n < min || n > max)
+        throw new Error(`--${name} must be an integer from ${min} to ${max}`);
+    return n;
+}
+/** Parse `ci` arguments (everything after the subcommand). Throws on bad input. */
+export function parseCiArgs(argv) {
+    const { values, positionals } = parseArgs({
+        args: argv,
+        allowPositionals: true,
+        strict: true,
+        options: {
+            'project-root': { type: 'string' },
+            config: { type: 'string' },
+            browser: { type: 'string' },
+            grep: { type: 'string' },
+            workers: { type: 'string' },
+            timeout: { type: 'string' },
+            'no-retry': { type: 'boolean' },
+            'flaky-runs': { type: 'string' },
+            suggest: { type: 'string' },
+            output: { type: 'string' },
+        },
+    });
+    const browser = values.browser;
+    if (browser !== undefined && browser !== 'chromium' && browser !== 'firefox' && browser !== 'webkit') {
+        throw new Error('--browser must be chromium, firefox or webkit');
+    }
+    return {
+        projectRoot: values['project-root'],
+        config: values.config,
+        browser,
+        grep: values.grep,
+        workers: intOption(values.workers, 'workers', 1, 64),
+        timeoutMs: intOption(values.timeout, 'timeout', 1_000, 3_600_000) ?? 600_000,
+        retry: values['no-retry'] !== true,
+        flakyRuns: intOption(values['flaky-runs'], 'flaky-runs', 2, 10),
+        suggest: intOption(values.suggest, 'suggest', 0, 5) ?? 0,
+        output: values.output,
+        testFiles: positionals,
+    };
+}
+function text(response) {
+    return response.content.map((block) => block.text).join('\n');
+}
+/** Run the CI flow; returns the exit code. */
+export async function runCi(argv) {
+    let options;
+    try {
+        options = parseCiArgs(argv);
+    }
+    catch (err) {
+        process.stderr.write(`${err.message}\n\n${CI_USAGE}`);
+        return 2;
+    }
+    const projectRoot = options.projectRoot ?? process.env.PW_MCP_PROJECT_ROOT ?? process.cwd();
+    const ctx = {
+        // Info logs would bury the report in CI output; LOG_LEVEL still wins.
+        logger: createLogger({
+            level: process.env.LOG_LEVEL || process.env.MCP_LOG_LEVEL ? undefined : 'warn',
+            bindings: { name: 'playwright-e2e-mcp', mode: 'ci' },
+        }),
+        store: createToolStore(),
+        projectRoot,
+    };
+    const sections = [];
+    const run = await runTestTool.handler({
+        testFiles: options.testFiles.length > 0 ? options.testFiles : undefined,
+        grep: options.grep,
+        browser: options.browser,
+        workers: options.workers,
+        timeoutMs: options.timeoutMs,
+        retryOnFailure: options.retry,
+        config: options.config,
+    }, ctx);
+    sections.push(text(run));
+    const result = ctx.store.lastRun?.result;
+    const failures = result?.failures ?? [];
+    let code = result === undefined || run.isError ? 2 : result.ok ? 0 : failures.length > 0 ? 1 : 2;
+    if (code === 1 && options.suggest > 0) {
+        // Before diagnose-flaky, which replaces the last run suggest-fix indexes into.
+        const count = Math.min(options.suggest, ctx.store.lastRun?.result.failures.length ?? 0);
+        for (let index = 1; index <= count; index += 1) {
+            const fix = await suggestFixTool.handler({ index, config: options.config }, ctx);
+            sections.push(text(fix).replace(/^## /m, '### '));
+        }
+    }
+    if (code === 1 && options.flakyRuns) {
+        const flaky = await diagnoseFlakyTool.handler({ runs: options.flakyRuns, config: options.config, browser: options.browser }, ctx);
+        sections.push(text(flaky).replace(/^## /m, '### '));
+    }
+    const report = `${sections.join('\n\n---\n\n')}\n\n<sub>Generated by [playwright-e2e-mcp](https://github.com/xsignon-secure/E2E) \`ci\`.</sub>\n`;
+    process.stdout.write(report);
+    try {
+        if (options.output)
+            await writeFile(options.output, report, 'utf8');
+        if (process.env.GITHUB_STEP_SUMMARY)
+            await appendFile(process.env.GITHUB_STEP_SUMMARY, report, 'utf8');
+    }
+    catch (err) {
+        process.stderr.write(`could not write the report: ${err.message}\n`);
+        code = Math.max(code, 2);
+    }
+    return code;
+}
+//# sourceMappingURL=ci.js.map

package/dist/cli/ci.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"ci.js","sourceRoot":"","sources":["../../src/cli/ci.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;GAUG;AAEH,OAAO,EAAE,UAAU,EAAE,SAAS,EAAE,MAAM,kBAAkB,CAAC;AACzD,OAAO,EAAE,SAAS,EAAE,MAAM,WAAW,CAAC;AAEtC,OAAO,EAAE,YAAY,EAAE,MAAM,oBAAoB,CAAC;AAClD,OAAO,EAAE,eAAe,EAAE,MAAM,oBAAoB,CAAC;AACrD,OAAO,EAAE,WAAW,EAAE,MAAM,sBAAsB,CAAC;AACnD,OAAO,EAAE,iBAAiB,EAAE,MAAM,4BAA4B,CAAC;AAC/D,OAAO,EAAE,cAAc,EAAE,MAAM,yBAAyB,CAAC;AAEzD,MAAM,CAAC,MAAM,QAAQ,GAAG;IACtB,sDAAsD;IACtD,EAAE;IACF,8EAA8E;IAC9E,uFAAuF;IACvF,EAAE;IACF,kFAAkF;IAClF,kEAAkE;IAClE,sDAAsD;IACtD,yDAAyD;IACzD,6CAA6C;IAC7C,wEAAwE;IACxE,0DAA0D;IAC1D,kFAAkF;IAClF,mFAAmF;IACnF,gFAAgF;IAChF,EAAE;CACH,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AAgBb,SAAS,SAAS,CAAC,KAAyB,EAAE,IAAY,EAAE,GAAW,EAAE,GAAW;IAClF,IAAI,KAAK,KAAK,SAAS;QAAE,OAAO,SAAS,CAAC;IAC1C,MAAM,CAAC,GAAG,MAAM,CAAC,KAAK,CAAC,CAAC;IACxB,IAAI,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,GAAG,IAAI,CAAC,GAAG,GAAG;QAAE,MAAM,IAAI,KAAK,CAAC,KAAK,IAAI,4BAA4B,GAAG,OAAO,GAAG,EAAE,CAAC,CAAC;IACtH,OAAO,CAAC,CAAC;AACX,CAAC;AAED,mFAAmF;AACnF,MAAM,UAAU,WAAW,CAAC,IAAc;IACxC,MAAM,EAAE,MAAM,EAAE,WAAW,EAAE,GAAG,SAAS,CAAC;QACxC,IAAI,EAAE,IAAI;QACV,gBAAgB,EAAE,IAAI;QACtB,MAAM,EAAE,IAAI;QACZ,OAAO,EAAE;YACP,cAAc,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;YAClC,MAAM,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;YAC1B,OAAO,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;YAC3B,IAAI,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;YACxB,OAAO,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;YAC3B,OAAO,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;YAC3B,UAAU,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE;YAC/B,YAAY,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;YAChC,OAAO,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;YAC3B,MAAM,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;SAC3B;KACF,CAAC,CAAC;IACH,MAAM,OAAO,GAAG,MAAM,CAAC,OAAO,CAAC;IAC/B,IAAI,OAAO,KAAK,SAAS,IAAI,OAAO,KAAK,UAAU,IAAI,OAAO,KAAK,SAAS,IAAI,OAAO,KAAK,QAAQ,EAAE,CAAC;QACrG,MAAM,IAAI,KAAK,CAAC,+CAA+C,CAAC,CAAC;IACnE,CAAC;IACD,OAAO;QACL,WAAW,EAAE,MAAM,CAAC,cAAc,CAAC;QACnC,MAAM,EAAE,MAAM,CAAC,MAAM;QACrB,OAAO;QACP,IAAI,EAAE,MAAM,CAAC,IAAI;QACjB,OAAO,EAAE,SAAS,CAAC,MAAM,CAAC,OAAO,EAAE,SAAS,EAAE,CAAC,EAAE,EAAE,CAAC;QACpD,SAAS,EAAE,SAAS,CAAC,MAAM,CAAC,OAAO,EAAE,SAAS,EAAE,KAAK,EAAE,SAAS,CAAC,IAAI,OAAO;QAC5E,KAAK,EAAE,MAAM,CAAC,UAAU,CAAC,KAAK,IAAI;QAClC,SAAS,EAAE,SAAS,CAAC,MAAM,CAAC,YAAY,CAAC,EAAE,YAAY,EAAE,CAAC,EAAE,EAAE,CAAC;QAC/D,OAAO,EAAE,SAAS,CAAC,MAAM,CAAC,OAAO,EAAE,SAAS,EAAE,CAAC,EAAE,CAAC,CAAC,IAAI,CAAC;QACxD,MAAM,EAAE,MAAM,CAAC,MAAM;QACrB,SAAS,EAAE,WAAW;KACvB,CAAC;AACJ,CAAC;AAED,SAAS,IAAI,CAAC,QAAsB;IAClC,OAAO,QAAQ,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AAChE,CAAC;AAED,8CAA8C;AAC9C,MAAM,CAAC,KAAK,UAAU,KAAK,CAAC,IAAc;IACxC,IAAI,OAAkB,CAAC;IACvB,IAAI,CAAC;QACH,OAAO,GAAG,WAAW,CAAC,IAAI,CAAC,CAAC;IAC9B,CAAC;IAAC,OAAO,GAAG,EAAE,CAAC;QACb,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAI,GAAa,CAAC,OAAO,OAAO,QAAQ,EAAE,CAAC,CAAC;QACjE,OAAO,CAAC,CAAC;IACX,CAAC;IAED,MAAM,WAAW,GAAG,OAAO,CAAC,WAAW,IAAI,OAAO,CAAC,GAAG,CAAC,mBAAmB,IAAI,OAAO,CAAC,GAAG,EAAE,CAAC;IAC5F,MAAM,GAAG,GAAgB;QACvB,sEAAsE;QACtE,MAAM,EAAE,YAAY,CAAC;YACnB,KAAK,EAAE,OAAO,CAAC,GAAG,CAAC,SAAS,IAAI,OAAO,CAAC,GAAG,CAAC,aAAa,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,MAAM;YAC9E,QAAQ,EAAE,EAAE,IAAI,EAAE,oBAAoB,EAAE,IAAI,EAAE,IAAI,EAAE;SACrD,CAAC;QACF,KAAK,EAAE,eAAe,EAAE;QACxB,WAAW;KACZ,CAAC;IAEF,MAAM,QAAQ,GAAa,EAAE,CAAC;IAC9B,MAAM,GAAG,GAAG,MAAM,WAAW,CAAC,OAAO,CACnC;QACE,SAAS,EAAE,OAAO,CAAC,SAAS,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC,CAAC,SAAS;QACvE,IAAI,EAAE,OAAO,CAAC,IAAI;QAClB,OAAO,EAAE,OAAO,CAAC,OAAO;QACxB,OAAO,EAAE,OAAO,CAAC,OAAO;QACxB,SAAS,EAAE,OAAO,CAAC,SAAS;QAC5B,cAAc,EAAE,OAAO,CAAC,KAAK;QAC7B,MAAM,EAAE,OAAO,CAAC,MAAM;KACvB,EACD,GAAG,CACJ,CAAC;IACF,QAAQ,CAAC,IAAI,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC;IAEzB,MAAM,MAAM,GAAG,GAAG,CAAC,KAAK,CAAC,OAAO,EAAE,MAAM,CAAC;IACzC,MAAM,QAAQ,GAAG,MAAM,EAAE,QAAQ,IAAI,EAAE,CAAC;IACxC,IAAI,IAAI,GAAG,MAAM,KAAK,SAAS,IAAI,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,QAAQ,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IAEjG,IAAI,IAAI,KAAK,CAAC,IAAI,OAAO,CAAC,OAAO,GAAG,CAAC,EAAE,CAAC;QACtC,+EAA+E;QAC/E,MAAM,KAAK,GAAG,IAAI,CAAC,GAAG,CAAC,OAAO,CAAC,OAAO,EAAE,GAAG,CAAC,KAAK,CAAC,OAAO,EAAE,MAAM,CAAC,QAAQ,CAAC,MAAM,IAAI,CAAC,CAAC,CAAC;QACxF,KAAK,IAAI,KAAK,GAAG,CAAC,EAAE,KAAK,IAAI,KAAK,EAAE,KAAK,IAAI,CAAC,EAAE,CAAC;YAC/C,MAAM,GAAG,GAAG,MAAM,cAAc,CAAC,OAAO,CAAC,EAAE,KAAK,EAAE,MAAM,EAAE,OAAO,CAAC,MAAM,EAAE,EAAE,GAAG,CAAC,CAAC;YACjF,QAAQ,CAAC,IAAI,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,OAAO,CAAC,OAAO,EAAE,MAAM,CAAC,CAAC,CAAC;QACpD,CAAC;IACH,CAAC;IAED,IAAI,IAAI,KAAK,CAAC,IAAI,OAAO,CAAC,SAAS,EAAE,CAAC;QACpC,MAAM,KAAK,GAAG,MAAM,iBAAiB,CAAC,OAAO,CAAC,EAAE,IAAI,EAAE,OAAO,CAAC,SAAS,EAAE,MAAM,EAAE,OAAO,CAAC,MAAM,EAAE,OAAO,EAAE,OAAO,CAAC,OAAO,EAAE,EAAE,GAAG,CAAC,CAAC;QAClI,QAAQ,CAAC,IAAI,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,OAAO,CAAC,OAAO,EAAE,MAAM,CAAC,CAAC,CAAC;IACtD,CAAC;IAED,MAAM,MAAM,GAAG,GAAG,QAAQ,CAAC,IAAI,CAAC,aAAa,CAAC,mGAAmG,CAAC;IAClJ,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC;IAC7B,IAAI,CAAC;QACH,IAAI,OAAO,CAAC,MAAM;YAAE,MAAM,SAAS,CAAC,OAAO,CAAC,MAAM,EAAE,MAAM,EAAE,MAAM,CAAC,CAAC;QACpE,IAAI,OAAO,CAAC,GAAG,CAAC,mBAAmB;YAAE,MAAM,UAAU,CAAC,OAAO,CAAC,GAAG,CAAC,mBAAmB,EAAE,MAAM,EAAE,MAAM,CAAC,CAAC;IACzG,CAAC;IAAC,OAAO,GAAG,EAAE,CAAC;QACb,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,+BAAgC,GAAa,CAAC,OAAO,IAAI,CAAC,CAAC;QAChF,IAAI,GAAG,IAAI,CAAC,GAAG,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC;IAC3B,CAAC;IACD,OAAO,IAAI,CAAC;AACd,CAAC"}

package/dist/index.d.ts

@@ -7,6 +7,6 @@
  * tests and tooling can construct a server without side effects.
  */
 import { logger } from './utils/logger.js';
-export { createServer, startServer, SERVER_NAME, SERVER_VERSION } from './server.js';
+export { createServer, startServer, resolveToolSelection, TOOL_PROFILES, SERVER_NAME, SERVER_VERSION } from './server.js';
 export { createMcpHttpHandler, handleNodeRequest, MAX_BODY_BYTES } from './http.js';
 export { logger };