testeiya 0.2.1 → 0.3.10

package/skills/codeceptjs/codeceptjs-run-analysis/SKILL.md

@@ -1,90 +1,83 @@
 ---
 name: codeceptjs-run-analysis
-description: Use after running CodeceptJS tests with the `aiTrace` plugin enabled — analyse the trace artifacts (trace.md, per-step HTML/ARIA/screenshots, console logs) via bash tools. Toolkit, not a workflow — use cases include verifying a fix, clustering errors across a CI fail-storm, diagnosing flakiness across reruns, or investigating a single failure. Other skills (`writing-codeceptjs-tests`, `debugging-codeceptjs-tests`, `refactoring-codeceptjs-tests`) invoke this whenever a run has happened and needs to be reviewed. Trigger on phrases like "what failed", "analyse the run", "cluster these errors", "is it flaky", "did the fix hold".
+description: >
+  Use after running CodeceptJS tests with the `aiTrace` plugin enabled and the
+  results need review — verify a fix held, investigate a single failure, cluster
+  errors across a CI fail-storm, diagnose flakiness across reruns. Invoked by
+  other CodeceptJS skills whenever a run has happened. Trigger on "what failed",
+  "analyse the run", "cluster these errors", "is it flaky", "did the fix hold".
 ---
 
 # CodeceptJS Run Analysis
 
-After `npx codeceptjs run`, the trace artifacts land in `output/`. This skill is the playbook for reading them efficiently — pulling out the right step, the right file, and the right slice of a giant HTML snapshot using bash tools rather than re-running the test through MCP.
+After `npx codeceptjs run`, trace artifacts land in `output/`. This skill reads them efficiently — right step, right file, right slice of a giant HTML snapshot — via bash tools rather than re-running through MCP.
 
-There's no single end goal — pick the use case that matches the situation. The foundations (where artifacts live, how to read them, what tools to reach for) apply to all of them.
+No single end goal: pick the use case matching the situation. The foundations apply to all of them.
 
 ## Foundations
 
-### Ensure aiTrace is on
-The whole skill leans on `output/trace_<TestName>_<hash>/trace.md` and its sibling artifacts. Confirm `plugins: { aiTrace: { enabled: true } }` in the active config (run `codeceptjs-fundamentals` if you don't already know). Without aiTrace there are screenshots and `pageInfo` dumps at most — useful but partial; suggest enabling and re-running before deep analysis.
+- **aiTrace must be on.** Everything leans on `output/trace_<TestName>_<hash>/trace.md`. Confirm in the active config (run `codeceptjs-fundamentals` if unknown). Without it there are only screenshots and `pageInfo` dumps — suggest enabling and re-running before deep analysis.
+- `run_step_by_step` is interactive only; `aiTrace` is the sole source of per-step files. Ad-hoc `run_code` / `snapshot` still produce single-shot bundles under `output/trace_run_code_*` / `output/snapshot_*`.
+- **Locate traces**: reruns create new dirs — when unclear, most recent wins (`ls -dt output/trace_*`).
+- **Read trace.md first** — it's the index linking each step to its artifacts.
+  - Focus on the failed step; none marked → last step in the trace.
+  - Multiple failures marked → the **first** is usually the cause; the rest cascade.
 
-The MCP `run_step_by_step` tool is now interactive (pauses after every step; agent advances via `continue`) and no longer auto-writes a per-run artifact bundle. `aiTrace` is the only source of per-step trace files now — for ad-hoc `run_code` and `snapshot` calls you still get a single-shot artifact set under `output/trace_run_code_*` / `output/snapshot_*`.
-
-### Locate traces
-Trace directories are `output/trace_<TestName>_<short-hash>/`; reruns produce a new dir per run. **When the right trace isn't obvious, sort by modification time and take the most recent** (`ls -dt`) — almost always the run you just kicked off.
-
-### Read trace.md first
-trace.md is the index — each step block links to its screenshot/HTML/ARIA/console. **Focus on the step marked failed; if no failure is explicitly marked, jump to the last step in the trace** — execution usually stops where the failure happened. When more than one step shows failed, the **first** is usually the cause; the rest are cascading side effects.
-
-### Open artifacts in the right order
-For the focus step `NNNN_<step>`:
+### Artifact order for the focus step (`NNNN_<step>`)
 
 | Artifact | When |
 |---|---|
-| `NNNN_*_aria.txt` | First read — leaner than HTML, structured, easy to scan for duplicates |
+| `NNNN_*_aria.txt` | First read — lean, structured, easy to scan for duplicates |
 | `NNNN_*_screenshot.png` | Visual confirmation — layout, animation, "rendered but wrong" |
-| `NNNN_*_page.html` | Only when ARIA is missing context. **Use `grep`, not `cat`** |
-| `NNNN_*_console.json` | JS errors, 4xx/5xx, deprecation warnings explaining "vanished" elements |
-| `NNNN_*_storage.json` | Cookies + localStorage at this step. First place to look when auth is the suspected culprit. |
+| `NNNN_*_console.json` | JS errors, 4xx/5xx, deprecation warnings explaining vanished elements |
+| `NNNN_*_storage.json` | Cookies + localStorage at this step — first stop when auth suspected |
+| `NNNN_*_page.html` | Last resort, and only via `grep` |
 
 ### Never read big files whole
-HTML snapshots can be megabytes; `console.json` arrays can be long.
 
-- Use `grep` to search HTML for expected elements / attributes / locator variants (text, class, aria-label, `data-*`). Line numbers and context flags make the match readable.
-- Use `jq` to filter `console.json` for errors or specific event types instead of reading every entry.
-- When scanning many files, `grep -l` returns filenames-only.
+- HTML snapshots: search with `grep` (text, class, aria-label, `data-*`) — line numbers + context flags keep matches readable.
+- `console.json`: filter with `jq`, don't read every entry.
+- Scanning many files: `grep -l` for filenames only.
 
 ## Use cases
 
 ### Verify a fix held
-After editing a test (e.g. via MCP) and re-running it via CLI: locate the latest trace, read trace.md, confirm no FAILED markers. Glance at `console.json` for warnings worth fixing while you're there.
+Locate latest trace → read trace.md → confirm no FAILED markers. Glance at `console.json` for warnings worth fixing while you're there.
 
 ### Cluster errors across a CI batch
-When many tests failed in one run: extract the failing-step lines from every `trace.md` in the batch, group by signature (`grep` + `sort` + `uniq -c`), rank by frequency.
-
-- **Same error in many tests = systemic.** Env var missing, auth broken, base URL wrong, deploy regression. Fix the root cause once and rerun the batch.
-- **Different errors per test = local issues.** Triage one at a time.
-
-Pick the most-frequent root cause first; rerun to see how many tests came back along with it.
-
-### Diagnose flakiness across reruns
-Run the same test 5–10 times. Extract which step failed in each trace and look at how the answer varies. If the failing step is the *same* but the surrounding state differs, `diff` the ARIA snapshots at that step between two runs to spot what changed.
-
-- **Different step each run** → timing, environment, external service.
-- **Same step, different reason** → likely a missing or wrong wait (see Waiting in writing/debug skills).
-- **`console.json` differs between runs** → transient backend / network errors.
-- **Bounding box differs** → layout reflow or late-loading content shifting things.
+Extract failing-step lines from every `trace.md`, group by signature (`grep` + `sort` + `uniq -c`), rank by frequency.
+- Same error in many tests = **systemic** (env var, auth, base URL, deploy regression) — fix root cause once, rerun the batch.
+- Different errors per test = local — triage one at a time.
+Start with the most frequent root cause; rerun to see how many tests came back with it.
+
+### Diagnose flakiness
+Rerun the same test 5–10 times; compare which step failed in each trace.
+- Different step each run → timing, environment, external service
+- Same step, different state → missing/wrong wait — `diff` the ARIA snapshots of that step between runs
+- `console.json` differs between runs → transient backend/network errors
+- Bounding box differs → layout reflow or late-loading content
 
 ### Investigate a single failure
-Locate the trace, jump to the failed-or-last step, open ARIA + screenshot first, console.json second, HTML last (and only via `grep`). Form a hypothesis. If the trace alone isn't enough — the failure resists static analysis or the page state needs live poking — hand off to **debugging-codeceptjs-tests** for the MCP-driven loop.
+Failed-or-last step → ARIA + screenshot first, console second, HTML last (grep only). Form a hypothesis. Trace not enough / page needs live poking → hand off to `debugging-codeceptjs-tests`.
 
-## After the analysis
+## After analysis
 
-- **Systemic cause across many tests** — fix the root once (env, auth, deploy, base URL). Not per test.
-- **Locator drift** — invoke **codeceptjs-exploration** to pick a new stable locator.
-- **Timing / wait issue** — apply the Waiting guidance from the writing/debugging skills, replacing `I.wait(N)` with a specific `waitFor*`.
-- **Failure resists static analysis** — invoke **debugging-codeceptjs-tests** for the live MCP loop.
-- **Test passed cleanly** — done. Still glance at `console.json` for warnings worth fixing now.
+- Systemic cause → fix root once (env, auth, deploy), not per test
+- Locator drift → `codeceptjs-exploration`
+- Timing/wait issue → Waiting guidance from fundamentals/writing skills; replace `I.wait(N)` with specific `waitFor*`
+- Resists static analysis → `debugging-codeceptjs-tests` (live MCP loop)
+- Clean pass → done, but glance at `console.json` anyway
 
 ## Things to avoid
 
-- Reading large HTML or `console.json` files whole — search them with `grep` / `jq` instead.
-- When more than one step is marked failed, stopping at the last one — the **first** failure is usually the cause and the rest are cascading.
-- Triaging individual failures before clustering — fixing 12 symptoms of one bug is wasted effort.
-- Drawing flakiness conclusions from a single run — needs 5+ reruns.
-- Deleting `output/` mid-investigation; the artifacts are the only record of the run.
+- Reading large HTML or `console.json` whole — `grep` / `jq`.
+- Stopping at the last marked failure instead of the first.
+- Triaging individual failures before clustering.
+- Flakiness conclusions from a single run — needs 5+ reruns.
+- Deleting `output/` mid-investigation.
 
-## Pointers
+## Related skills
 
-- `node_modules/codeceptjs/docs/aitrace.md` — trace format and config knobs
-- `node_modules/codeceptjs/docs/debugging.md` — verbose flags, `pause` plugin's `on=` modes
-- `node_modules/codeceptjs/docs/reports.md` — alternative reporters
-- `codeceptjs-fundamentals` — what's actually configured
-- `codeceptjs-exploration` — when locator drift is the cause
-- `debugging-codeceptjs-tests` — when post-mortem isn't enough
+- `codeceptjs-fundamentals` — what's configured, aiTrace `-p` overrides
+- `codeceptjs-exploration` — locator drift fixes
+- `debugging-codeceptjs-tests` — live loop + offline locator resolution via `codeceptq`

package/skills/codeceptjs/debugging-codeceptjs-tests/SKILL.md

@@ -1,234 +1,126 @@
 ---
 name: debugging-codeceptjs-tests
-description: "Use when a CodeceptJS 4 test is failing, flaky, or behaving unexpectedly — stack traces from `npx codeceptjs run`, intermittent failures, locator drift, timing issues, \"works locally fails in CI\", \"step through this test\", \"pause at step N\", \"set a breakpoint\". For AI agents the primary path is **MCP with pause** — drop a `pause()` in the test (or pass `pauseAt: N` to `run_test` for a no-edit breakpoint), inspect via `run_code` / `snapshot` against the live browser, release with `continue`. Step indices for `pauseAt` come from `npx codeceptjs dry-run --debug --grep <test> --numbers --no-ansi`. CLI debugging (`npx codeceptjs run --debug`, `DEBUG=\"codeceptjs:*\"`) is the fallback for humans, CI repros, and framework-internal issues (recorder hangs, leaks, plugin races). Don't fix from the error message alone; capture page state and read it. Trigger on broken or flaky tests, run errors, \"why does this fail\", trace/screenshot/console mentions, breakpoint/pause/step-through requests."
+description: >
+  Use when a CodeceptJS 4 test fails, flakes, or behaves unexpectedly. Trigger
+  on run errors and stack traces from `npx codeceptjs run`, intermittent
+  failures, locator drift, timing issues, "works locally fails in CI", "why
+  does this fail", trace/screenshot/console mentions, and breakpoint /
+  step-through / "pause at step N" requests.
 ---
 
 # Debugging CodeceptJS 4 Tests
 
-Failures lie. The error usually points at a step that's a side effect of something earlier — auth expired, a frame switch missed, a network call still pending. Reproduce, capture state, and read it before fixing.
+Failures lie — the error usually points at a step that's a side effect of something earlier (auth expired, frame switch missed, network call pending). Reproduce, capture state, read it, then fix.
 
-Two paths, picked by who's driving:
+## Paths
 
-- **MCP-first (for AI agents)** — drive the test through the MCP server. In-test `pause()` and the `pauseAt: N` option on `run_test` both yield control back to the agent in-process — same `I` / browser the test is using. Inspect via `run_code` / `snapshot`, advance one step at a time via `run_step_by_step` + `continue`, release a pause via `continue`. `aiTrace` artifacts cover the prior steps.
-- **CLI fallback (for humans / CI / framework internals)** — `npx codeceptjs run --debug` for verbose framework output. Escalate to `DEBUG="codeceptjs:*"` when the *framework itself* looks at fault: recorder hangs, plugin races, event leaks, "step never ran". Use this path for CI repros, headless servers, and framework-internal bugs.
+- **MCP-first (AI agents)**: `run_test` / in-test `pause()` yield control to the agent on the same `I` / browser the test uses. Inspect via `run_code` / `snapshot`; step through via `run_step_by_step` + `continue`.
+- **CLI fallback (humans / CI / framework internals)**: `--debug` → `--verbose` → `DEBUG="codeceptjs:*"`. The DEBUG escape hatch is only for framework-internal suspicion: recorder hangs, plugin races, event leaks, "step never ran". Namespaces: `codeceptjs:recorder`, `codeceptjs:pause`, `codeceptjs:ai`, `codeceptjs:plugin:<name>`.
 
-In-test `pause()` adapts to who's driving: at a TTY it opens the readline REPL; under MCP it yields control to the agent (same in-process `I` / browser); in a non-TTY non-MCP subprocess it prints a notice and resolves immediately so leftover `pause()` calls don't deadlock CI. **Adding `pause()` is now the primary MCP breakpoint** — drop it where you want to look, run via `run_test`, drive the live page through `run_code`, release with `continue`.
+`pause()` adapts to who's driving: TTY → readline REPL; MCP → yields control to the agent; non-TTY non-MCP subprocess → prints a notice and resolves (no CI deadlock).
 
 ## Workflow
 
-### 1. Read the project (fundamentals)
-Run the **codeceptjs-fundamentals** skill. You need: helper, plugins on (especially `aiTrace`, `screenshot`, `pageInfo`, `retryFailedStep`, `pause`, `auth`), env vars, whether MCP is wired up. If `aiTrace` is **not** declared, add it **once** with `plugins: { aiTrace: { enabled: true } }` — most of this skill leans on its output.
-
-**Declare `aiTrace` once; never edit config to change its trigger.** The capture mode is controlled per-run from the CLI, exactly like the `pause` plugin:
-
-```bash
-npx codeceptjs run -p aiTrace:on=step   # persist every step (default)
-npx codeceptjs run -p aiTrace:on=fail   # persist only the failed step
-npx codeceptjs run -p aiTrace:on=test   # persist only the last step of each test
-npx codeceptjs run -p aiTrace:on=file:path=tests/login_test.js;line=43
-npx codeceptjs run -p aiTrace:on=url:pattern=/checkout/*
-```
-
-`-p aiTrace:on=...` overrides the config-declared mode for that run only. Reach for `on=fail` to keep a CI repro lean, `on=step` while actively diagnosing. Don't flip `on:` in `codecept.conf.js` between runs — it churns the config and the change leaks into other runs.
-
-### 2. Reproduce minimally
-Run only the failing test, with steps printed:
-```bash
-npx codeceptjs run --grep '<scenario>' --steps
-```
-Add `--config codecept.ci.conf.js` if the failure is CI-specific. Confirm reproduction before instrumenting further.
-
-### 3. Pick a path
-
-**MCP (primary for AI agents):**
-- `run_test <test>` — runs a specific test in-process; shares `I` / browser with `run_code` and `snapshot`. Returns the JSON reporter result on completion, **or** `{ status: 'paused', pausedAfter, page, suggestions }` if the test calls `pause()` or hits the optional `pauseAt: N` breakpoint. From a paused state, drive the live page via `run_code` / `snapshot` and release with `continue`.
-- `run_step_by_step <test>` — interactive: pauses after every step. After each `continue`, the test advances one step and re-pauses (or completes). Use when you want to watch the whole flow tick by; use `run_test` with `pauseAt: N` instead for a single targeted breakpoint.
-- `continue` — releases a paused test. After `pause()` or `pauseAt`: runs to completion (or to the next `pause()`). After `run_step_by_step`: advances one step.
-- `run_code <CodeceptJS lines>` — runs arbitrary CodeceptJS code in the live session (works fresh **and** while a test is paused — same container). Returns the **value the code produced**, captures `console.log` / `info` / `warn` / `error` / `debug` output, and saves a final-state snapshot (URL, ARIA, HTML, screenshot, storage). Use to test a locator hypothesis or grab a value at the failure point.
-- `snapshot` — captures current browser state without performing any action (URL, cookies, localStorage, HTML, ARIA, screenshot, console). Use between actions when you want to reason about what to do next without re-running anything.
-- `list_actions` — sanity-check that an `I.*` method exists on the active helper.
-
-**CLI (fallback):**
-- `npx codeceptjs run --grep '<scenario>' --debug` — first move when MCP isn't available. Steps + helper internals + URLs + plugin events.
-- `npx codeceptjs run ... --verbose` — adds promise-queue / retry / timeout logs on top of `--debug`.
-- `DEBUG="codeceptjs:*" npx codeceptjs run ...` — turns on CodeceptJS's internal debug streams. Reach for this when `--debug` doesn't explain the failure: orphaned timers, event leaks, recorder hangs, plugin races, double-emitted events, "step disappeared from the queue". Narrow with namespaces: `codeceptjs:recorder` (promise queue), `codeceptjs:pause`, `codeceptjs:ai`, `codeceptjs:plugin:<name>`. Most user-level test failures don't need this — it's the framework-internal escape hatch.
-
-### 4. Set a breakpoint with `pause()` or `pauseAt`
-
-The MCP server installs an in-process pause handler at startup. Whenever a test running through `run_test` hits `pause()` (or completes the `pauseAt: N` step), control yields back to the agent on the same `I` / browser. There's no subprocess, no IPC, and `run_code` / `snapshot` work against the live page — exactly what a paused REPL would give you.
-
-Two ways to land at a breakpoint:
-
-- **In-test `pause()`** — drop `pause()` directly in the test where you want to look. Best when you're already editing the file or want to break inside a `within` / loop / hook.
-- **`pauseAt: N` on `run_test`** — programmatic, no test edit required. Pauses after the Nth leaf step completes.
-
-To pick `N`, list the steps with their indices:
-
-```bash
-npx codeceptjs dry-run --debug --grep '<scenario>' --numbers --no-ansi
-```
-
-Output is one numbered line per leaf step (1-based, per-test). The number on the line you want to stop *after* is the value to pass as `pauseAt`. `--no-ansi` strips colors so the output is clean for parsing.
-
-Once paused (`{ status: 'paused', pausedAfter, page, suggestions }`):
-
-1. **Inspect with `run_code`** — `await I.grabCurrentUrl()`, `await I.grabWebElement(...)`, `await I.seeElement({ role: 'dialog' })`. Each call returns URL + ARIA + console + storage from the live page.
-2. **Capture clean state with `snapshot`** between hypotheses — no action, just the artifact bundle.
-3. **Walk earlier steps via `aiTrace`** — `output/trace_<TestName>_<hash>/trace.md` has the per-step state for everything that ran before the breakpoint.
-4. **Release with `continue`** — runs to completion (or to the next `pause()`). For a step-by-step walk, use `run_step_by_step` instead of `run_test`; each `continue` then advances one step.
-
-### 5. Read the trace
-Hand off to **codeceptjs-run-analysis** to walk `output/trace_<TestName>_<hash>/trace.md` and the per-step artifacts. Focus on the **first** failed step — late failures are usually side effects of an earlier silent miss. The run-analysis skill also covers grepping into large HTML, clustering errors across many traces, and comparing reruns when flakiness is in play.
-
-For locator-level questions on a saved snapshot ("would `.btn-primary` have matched here?", "is `Username` a field at step 7?") use **`codeceptq`** against the per-step `<NNNN>_<step>_page.html` — see the "Query trace HTML with `codeceptq`" section below. Faster feedback loop than `run_code` when you're iterating selector candidates.
-
-### 6. Form a hypothesis
-
-| Symptom | Likely cause |
-|---|---|
-| Element missing, page is `/login` | Auth: stale `check`, expired session, missing env var |
-| Element in HTML but `display: none` | `waitForVisible`, not `waitForElement` |
-| Locator matches 2+ (strict mode) | Disambiguate: ARIA role, `step.opts({ elementIndex })`, `within` |
-| Element in screenshot N+1, missing in N | Animation / lazy load — `waitForVisible(loc, t)` |
-| 401/403 in console.json | API token expired or env var missing |
-| Steps pass, next `I.see` fails | Frame switch missed — wrap in `within({ frame })` |
-| Different result CI vs local | `setHeadlessWhen(CI)`, viewport, timing, env var |
-| Recorder hangs, step never fires | `DEBUG="codeceptjs:recorder"` to inspect the queue |
-| Plugin misbehaves | `DEBUG="codeceptjs:plugin:<name>"` |
-
-### 7. Verify the fix on the live page
-For agents driving MCP, use `run_code` to try the candidate fix in the live session **before editing the file**. If it works there, it'll work in the test. While paused (in-test `pause()` or `pauseAt`), `run_code` operates on the same `I` / browser the test is using, so a candidate replacement step can be tried in place. Humans running with `--debug` at a TTY can use in-test `pause()` for the same purpose at a readline REPL.
-
-### 8. Apply and re-run
-Edit the test, then `npx codeceptjs run --grep '<scenario>' --steps`. Use **codeceptjs-run-analysis** to verify the trace looks right after the fix — and to confirm the failure didn't shift to another step. If the fix introduces a `waitFor*` or `step.opts`, leave a one-line `Why:` comment — those are the comments worth keeping.
-
-## When to reach for which plugin / mode
-
-| You want to … | Use |
-|---|---|
-| Per-step artifacts after a run | `aiTrace` plugin (`output/trace_*/`), declared once in config |
-| Capture every step's state | `npx codeceptjs run -p aiTrace:on=step` |
-| Capture only the failed step (lean CI repro) | `npx codeceptjs run -p aiTrace:on=fail` |
-| Capture last step per test | `npx codeceptjs run -p aiTrace:on=test` |
-| Capture steps from a file/line or URL | `-p aiTrace:on=file:path=<file>;line=<N>` / `-p aiTrace:on=url:pattern=<glob>` |
-| REPL on first failure | `npx codeceptjs run -p pause` (default `on=fail`) |
-| Single-step interactively | `npx codeceptjs run -p pause:on=step` |
-| Break on a file or URL | `pause:on=file:path=<file>;line=<N>` / `pause:on=url:pattern=<glob>` |
-| Programmatic breakpoint at step N (no test edit) | MCP `run_test` with `pauseAt: N` (discover N via `dry-run --numbers`) |
-| In-test breakpoint at a specific line | drop `pause()` in the test, then MCP `run_test` |
-| Step-by-step REPL from an AI agent | MCP `run_step_by_step`, then `continue` between steps |
-| Release a paused test | MCP `continue` |
-| Test a hypothesis on the live page (agent) | MCP `run_code` (works fresh **and** while paused) |
-| Capture state without acting (agent) | MCP `snapshot` |
-| Test a hypothesis on the live page (human, TTY) | in-test `pause()` + `npx codeceptjs run --debug` |
-| List steps with their indices (for `pauseAt`) | `npx codeceptjs dry-run --debug --grep '<test>' --numbers --no-ansi` |
-| Visual replay slideshow | `screenshot:slides=true` → `output/records.html` |
-| Auto-suggest fixes for broken locators | `heal` plugin + `--ai` (disabled in `--debug`) |
-| Diagnose framework-internal behaviour | `DEBUG="codeceptjs:*"` (or a specific namespace) |
-| Inspect specific elements — state, markup, position, children | `I.grabWebElement` / `I.grabWebElements` (cross-helper WebElement API) |
-| Drop to native helper APIs when nothing else works | `I.usePlaywrightTo` / `I.usePuppeteerTo` / `I.useWebDriverTo` |
-| Verify a locator against a saved trace snapshot (offline) | `codeceptq <locator> --file output/trace_*/<NNNN>_<step>_page.html` |
-
-## Waiting (a common cause of flakes)
-
-Most "intermittent" failures are missed waits. Use the trace HTML / ARIA to find the *actual* gating element rather than adding a generic delay:
-- a **loader / spinner / skeleton** still on the page → `I.waitForInvisible('.spinner')` / `I.waitForDetached('.skeleton')`
-- a **modal / drawer / panel** that hasn't appeared yet → `I.waitForVisible('.modal')` / `I.waitForElement({ role: 'dialog' })`
-- async data — list rows, cards, charts, async-rendered text → `I.waitForElement('.user-row', 10)` / `I.waitForText('Loaded', 10, '.status')`
-
-`I.wait(N)` (raw seconds) is fine **during debugging** to confirm a timing hypothesis — if a 5-second sleep makes the test pass, you've found the cause. **Replace it with the specific `I.waitFor*` before committing.** Raw sleeps are slow on fast machines, flaky on slow ones, and hide the real sync point so the next person to touch the test inherits the same problem.
-
-## Inspect the page when the trace isn't enough
-
-When the trace tells you *what* failed but you need more page-state detail to diagnose — "is this button actually disabled?", "are there really two Save buttons?", "what's the rendered markup of this row?" — hand off to the **codeceptjs-exploration** skill. It covers the WebElement API (`I.grabWebElement` / `I.grabWebElements`, state checks, `toSimplifiedHTML`, `toAbsoluteXPath`, iframe walking) and the broad-XPath candidate-discovery technique.
-
-Debug-specific reaches into that toolkit:
-
-- **Button rendered but the click had no effect** — `grabWebElement('Submit')`, then `isEnabled()` + `getBoundingBox()`. Disabled? offscreen? zero-sized?
-- **Strict-mode "matched 2 elements"** — exploration's broad-XPath + iterate-and-disambiguate pattern is the canonical fix.
-- **Iframe content** — exploration's `inIframe` pattern; the failing step likely needs to be wrapped in `within({ frame })`.
-
-Prefer this over `usePlaywrightTo` / `useWebDriverTo` for inspection: same code across helpers, less boilerplate.
+1. **Fundamentals** — run `codeceptjs-fundamentals`: helper, plugins (`aiTrace`, `screenshot`, `pageInfo`, `retryFailedStep`, `pause`, `auth`), env vars, MCP availability. If `aiTrace` isn't declared, add it once: `plugins: { aiTrace: { enabled: true } }` — most of this skill leans on its output.
+   - **Declare once; never edit config to change its trigger.** Override per-run with `-p aiTrace:on=step|fail|test|file|url` (fundamentals § Plugins from CLI). `on=fail` for lean CI repros, `on=step` while diagnosing. Flipping `on:` in config churns the repo and leaks into other runs.
+2. **Reproduce minimally**: `npx codeceptjs run --grep '<scenario>' --steps`. Add `-c codecept.ci.conf.js` if CI-specific. Confirm reproduction before instrumenting.
+3. **Pick tools**:
+   - MCP `run_test <test>` — runs in-process; returns reporter result or `{ status: 'paused', pausedAfter, page, suggestions }`
+   - MCP `run_step_by_step` + `continue` — pause after every step; for watching the whole flow
+   - MCP `run_code` — arbitrary code in the live session; works fresh *and* paused; returns produced values + console output + final-state snapshot
+   - MCP `snapshot` — current browser state without acting (URL, cookies, storage, HTML, ARIA, screenshot, console)
+   - MCP `list_actions` — sanity-check an `I.*` method exists
+   - CLI `npx codeceptjs run --grep '<scenario>' --debug` — first move without MCP; `--verbose` adds promise-queue/retry/timeout logs
+4. **Breakpoint**:
+   - In-test `pause()` — best when already editing or breaking inside `within`/loop/hook
+   - `pauseAt: N` on `run_test` — no test edit; pauses after the Nth leaf step
+   - Find N: `npx codeceptjs dry-run --debug --grep '<scenario>' --numbers --no-ansi` (1-based, per-test; number of the line to stop *after*)
+   - While paused: inspect with `run_code`, capture clean state with `snapshot`, walk prior steps in `output/trace_<TestName>_<hash>/trace.md`, release with `continue`
+5. **Read the trace** — hand off to `codeceptjs-run-analysis`; focus on the **first** failed step — late failures are usually side effects of an earlier silent miss. It also covers grepping large HTML, clustering errors across traces, comparing reruns.
+6. **Form a hypothesis**:
+
+   | Symptom | Likely cause |
+   |---|---|
+   | Element missing, page is `/login` | Auth: stale session cache, missing env var |
+   | Element in HTML but `display: none` | `waitForVisible`, not `waitForElement` |
+   | Locator matches 2+ (strict mode) | Scope it: context arg, then ARIA role, then `step.opts({ elementIndex })` |
+   | Element present in screenshot N+1, missing in N | Animation / lazy load → `waitForVisible(loc, t)` |
+   | 401/403 in console.json | API token expired or env var missing |
+   | Steps pass, next `I.see` fails | Frame switch missed → `within({ frame })` |
+   | Different result CI vs local | `setHeadlessWhen(CI)`, viewport, timing, env vars |
+   | Recorder hangs, step never fires | `DEBUG="codeceptjs:recorder"` |
+   | Plugin misbehaves | `DEBUG="codeceptjs:plugin:<name>"` |
+
+7. **Verify the fix on the live page** — try the candidate replacement step via `run_code` (works while paused, same `I`) **before editing the file**. Humans at a TTY get the same via in-test `pause()`.
+8. **Apply and re-run**: edit, then `npx codeceptjs run --grep '<scenario>' --steps`; confirm via `codeceptjs-run-analysis` that the failure didn't shift steps. Leave a one-line `Why:` comment when the fix introduces `waitFor*` or `step.opts` — those comments are worth keeping.
 
 ## Query trace HTML with `codeceptq`
 
-`aiTrace` writes a per-step `<NNNN>_<step>_page.html` snapshot of the live DOM for every step (formatted so each element sits on its own line — line numbers map 1:1 to elements). To answer "would my locator have matched at step N?", use `codeceptq` — a CLI that resolves any CodeceptJS locator (CSS / XPath / fuzzy / semantic) against a saved HTML snapshot and prints the matched elements with their source lines.
+`aiTrace` writes per-step `<NNNN>_<step>_page.html` snapshots (one element per line — line numbers map 1:1 to elements). `codeceptq` resolves any CodeceptJS locator against a saved snapshot.
 
-**Never load the page HTML into your context to inspect it manually.** Real-world `*_page.html` files are thousands of lines and burn context for nothing — `codeceptq` does the locator resolution and returns only the relevant elements. Reach for it instead of `Read`-ing the snapshot.
+- **Never load page HTML into context manually** — snapshots are thousands of lines; `codeceptq` returns only matched elements with their line numbers.
+- Test candidate locators offline before applying via `run_code` — a hit is a green light to try live, not a guarantee (visibility, re-renders).
+- Multiple matches → don't write a brittler XPath; disambiguate with `step.opts({ elementIndex })` following the order `codeceptq` prints.
 
 ```bash
-# does this CSS resolve?
-npx codeceptq '#submit-btn' --file output/trace_<TestName>_<hash>/0007_I_click_Submit_page.html
-
-# semantic field lookup against a saved snapshot
-npx codeceptq 'Email' --field --file output/trace_*/0003_*_page.html
-
-# semantic clickable, scoped to a context
-npx codeceptq 'Save' '.modal' --click --file output/trace_*/0005_*_page.html
-
-# pipe directly from stdin
-cat output/trace_*/0001_*_page.html | npx codeceptq './/form//input[@required]'
-
-# machine-readable for chained tooling
-npx codeceptq 'Username' --field --json --file output/trace_*/0002_*_page.html
+npx codeceptq '#submit-btn' --file output/trace_*/0007_*_page.html     # CSS
+npx codeceptq 'Email' --field --file output/trace_*/0003_*_page.html   # semantic field
+npx codeceptq 'Save' '.modal' --click --file output/trace_*/0005_*_page.html  # scoped clickable
+npx codeceptq 'Username' --field --json --file ...                     # machine-readable
 ```
 
-What you get back: a count, the resolved XPath, and one entry per match with the **line number** in the snapshot file plus the element's outerHTML.
+Key flags: `--field/--click/--checkable/--select` force semantic strategies; `--xpath`/`--css` override auto-detection (a bare tag name like `select.foo` is treated as fuzzy text); exit codes `0` match / `1` none / `2` invalid input.
 
-Flags worth knowing:
-- `--field` / `--click` / `--checkable` / `--select` — force a CodeceptJS semantic strategy (label, button text, checkbox, option). Without a flag, the locator type is auto-detected (CSS if it starts with `#`/`.`/`[`; XPath if it starts with `//` or `./`; fuzzy text otherwise).
-- `--xpath` / `--css` — force interpretation when auto-detection wouldn't pick the right one (e.g., a bare tag name like `select.foo` without `--css` would be treated as fuzzy text).
-- `[context]` — second positional arg restricts matches to descendants of the context locator (e.g., `'Save' '.modal' --click`).
-- `--limit N` (default 20), `--snippet N` (default 500), `--full`, `--json`.
-- Exit codes: `0` matches, `1` no match, `2` invalid input/XPath — useful for scripted "did this locator break?" checks.
+> ⚠ **The `[context]` second positional does not scope** (CodeceptJS 4.1.0). It prints `N matches within '<ctx>'` but returns page-wide results — `lib/command/query.js` evaluates an absolute XPath (`//…`) against the context node, and `//foo` re-roots at the document. It will report a match inside a container that does not hold the element. To check a scoped locator offline, pass one composed selector (`codeceptq '.modal button[aria-label="Save"]'`) and compare its count against the unscoped form; a context-dependent locator is only truly verified by running the step.
 
-Use `codeceptq` to test locators against the snapshot **before** applying them via `run_code`. If a candidate matches, you've validated the locator string against the DOM as captured. It can still fail live — element not visible, removed by a re-render, or the snapshot is from a different step — so treat a hit as a green light to try, not a guarantee.
+## Inspect deeper
 
-When `codeceptq` returns multiple matches, **don't write a brittler XPath** — disambiguate with `step.opts({ elementIndex })`. Indexing is 1-based and follows the order `codeceptq` prints; supports `'first'`, `'last'`, and negatives.
+Hand off to `codeceptjs-exploration` when the trace says *what* failed but you need more page-state detail ("is this button actually disabled?", "are there really two Save buttons?"). Debug-specific reaches:
 
-```js
-I.click('Edit', step.opts({ elementIndex: 2 }))
-I.fillField('input', 'value', step.opts({ elementIndex: 'last' }))
-```
+- Button rendered but click had no effect → `grabWebElement('Submit')` + `isEnabled()` + `getBoundingBox()` (disabled? offscreen?)
+- Strict-mode multi-match → exploration's broad-XPath iterate-and-disambiguate pattern
+- Iframe content → wrap failing steps in `within({ frame })`
 
-## Native helper API escape hatch
+Prefer this over `usePlaywrightTo`/`useWebDriverTo` for inspection — same code across helpers, less boilerplate.
 
-When `I.grabWebElement` and the rest of the `I.*` surface still don't cover it — listening to network requests, manipulating storage, calling a Playwright-only API, raw browser context work — drop down to the underlying helper:
+## Native helper escape hatch
 
-- **Playwright** — `I.usePlaywrightTo('description', async ({ browser, browserContext, page }) => { ... })`
-- **Puppeteer** — `I.usePuppeteerTo('description', async ({ page }) => { ... })`
-- **WebDriver** — `I.useWebDriverTo('description', async ({ browser }) => { ... })`
+Only when the `I.*` surface truly doesn't cover it (network interception, storage manipulation, helper-only APIs):
 
-The first arg is a label that shows up in step output and traces. The callback receives the helper's native objects. Use these to inspect or manipulate state CodeceptJS doesn't expose — `page.evaluate(() => performance.timing)`, `page.context().cookies()`, `browserContext.on('request', …)`, raw `executeScript` chains. They work inside MCP `run_code` too, so you can poke at internals during a live debug session.
+- Playwright: `I.usePlaywrightTo('label', async ({ browser, browserContext, page }) => { ... })`
+- Puppeteer: `I.usePuppeteerTo('label', async ({ page }) => { ... })`
+- WebDriver: `I.useWebDriverTo('label', async ({ browser }) => { ... })`
 
-Try the regular `I.*` API first — these escape hatches couple the test to a specific helper. Reach for them only when nothing else works.
+The label shows up in step output and traces. Works inside MCP `run_code` too. These couple tests to a specific helper — last resort.
 
-## Helper-specific gotchas
+## Helper gotchas
 
-- **Playwright** — `strict: true` throws on multi-match. `trace: 'on'` produces `output/trace.zip` (open with `npx playwright show-trace`). Prefer `'load'` / `'domcontentloaded'` over `'networkidle'`.
-- **Puppeteer** — `'networkidle0'` can hang on long-polling pages; try `'networkidle2'` or `'domcontentloaded'`.
-- **WebDriver** — `smartWait` applies to actions only, not assertions. `executeScript` args must be JSON-serializable.
+- Playwright: `strict: true` throws on multi-match; prefer `'load'`/`'domcontentloaded'` over `'networkidle'`; `trace: 'on'` → `output/trace.zip` (`npx playwright show-trace`)
+- Puppeteer: `'networkidle0'` hangs on long-polling pages — use `'networkidle2'` or `'domcontentloaded'`
+- WebDriver: `smartWait` covers actions only, not assertions; `executeScript` args must be JSON-serializable
 
 ## Auth-related failures
 
-If the trace shows a redirect to `/login` mid-test, or 401/403 in console, fix **auth**, not the failing step. Check the `auth` plugin's `check`, that credential env vars are exported, and that the cached session under `output/<role>_session.json` isn't stale (delete it to force re-login). The **codeceptjs-auth** skill has the full pattern.
+Redirect to `/login` mid-test or 401/403 in console → fix **auth**, not the failing step: check the plugin's `check`, credential env vars, and stale cached session under `output/<role>_session.json` (delete to force re-login). Full pattern in `codeceptjs-auth`.
+
+## Flakiness and waits
+
+Most "intermittent" failures are missed waits. Use trace HTML/ARIA to find the actual gating element instead of adding generic delay (fundamentals § Waiting has the mapping). `I.wait(N)` confirms a timing hypothesis while debugging — replace with the specific `waitFor*` before committing.
 
 ## Things to avoid
 
 - Fixing from the error message without reading the trace.
-- Editing the test before verifying the fix in `run_code` — you'll iterate without ground truth.
-- Committing `pause()` calls. They're a debugging tool — remove (or replace with `pauseAt`) before merging. A `pause()` left in a test that runs in a non-TTY non-MCP CI subprocess will print a notice and skip, but it's still noise on every run.
-- Adding `waitFor*` blindly instead of identifying the real gating element from HTML/ARIA.
-- Leaving `I.wait(N)` (raw seconds) in committed tests — keep them only while debugging, then replace with the specific `waitFor*`.
-- Editing `aiTrace`'s `on:` in `codecept.conf.js` to switch capture modes — declare it once and override per-run with `-p aiTrace:on=...`. Repeated config edits churn the repo and leak the mode into unrelated runs.
-- Skipping the config check — `setHeadlessWhen(CI)` or env-driven URLs explain many "works locally fails in CI" reports.
-- Hiding the failure with `retries` instead of fixing the cause.
-
-## Pointers
-
-- `node_modules/codeceptjs/docs/mcp.md` — MCP tool list and client config
-- `node_modules/codeceptjs/docs/aitrace.md` — plugin config, trace.md format
-- `node_modules/codeceptjs/docs/debugging.md` — in-test `pause()`, the `pause` plugin's `on=` modes, IDE setup, DEBUG namespaces
-- `node_modules/codeceptjs/docs/heal.md` — self-healing recipes
-- `node_modules/codeceptjs/docs/retry.md` — retry semantics across step / scenario / hook
-- `node_modules/codeceptjs/lib/plugin/aiTrace.js`, `lib/plugin/pause.js`, `lib/plugin/screenshot.js`, `lib/plugin/browser.js`, `bin/mcp-server.js` — source if docs and code disagree
+- Editing the test before verifying the fix via `run_code`.
+- Committing `pause()` calls — debugging tool only.
+- Blind `waitFor*` instead of identifying the real gating element.
+- Leaving `I.wait(N)` in committed tests.
+- Editing `aiTrace`'s `on:` in config between runs — declare once, override per-run.
+- Skipping the config check — `setHeadlessWhen(CI)` / env-driven URLs explain many "works locally fails in CI".
+- Hiding failures with `retries` instead of fixing the cause.
+
+## Related skills
+
+- `codeceptjs-fundamentals` — effects, plugins-from-CLI, waiting rules
+- `codeceptjs-exploration` — WebElement inspection, broad-XPath disambiguation
+- `codeceptjs-run-analysis` — trace.md walking, error clustering, rerun comparison
+- `codeceptjs-auth` — auth failure patterns

package/skills/codeceptjs/migrate-codeceptjs-4/SKILL.md

@@ -1,9 +1,8 @@
 ---
 name: migrate-codeceptjs-4
-description: "Migrate a CodeceptJS '3.x' project to '4.x'. Trigger when 'package.json' pins 'codeceptjs' at '3.x' or is missing '\"type\": \"module\"', when test files still use CommonJS ('require()' / 'module.exports') against CodeceptJS APIs, when config references removed helpers ('Nightmare', 'Protractor', 'TestCafe', 'AI', 'SoftExpectHelper', 'Mochawesome') or removed plugins ('autoLogin', 'tryTo', 'retryTo', 'eachElement', 'commentStep', 'fakerTransform', 'enhancedRetryFailedStep', 'allure', 'htmlReporter', 'wdio', 'selenoid', 'screenshotOnFail', 'pauseOnFail', 'stepByStepReport'), or when '3.x' APIs are in use ('ai.request' function, Joi schemas in 'seeResponseMatchesJsonSchema', 'restart: 'browser'', 'I.retry()', 'I.limitTime()', Playwright 'customLocators'). Walks the project through Node + package upgrade, ESM conversion, helper/plugin replacements, AI/Zod/effects API changes, 'noGlobals: true' adoption, dependency bumps, and the post-upgrade verify pass."
+description: "Migrate a CodeceptJS 3.x project to 4.x. Trigger when `package.json` pins `codeceptjs` at 3.x or is missing `\"type\": \"module\"`, when test files still use CommonJS (`require()` / `module.exports`) against CodeceptJS APIs, when config references removed helpers (`Nightmare`, `Protractor`, `TestCafe`, `AI`, `SoftExpectHelper`, `Mochawesome`) or removed plugins (`autoLogin`, `tryTo`, `retryTo`, `eachElement`, `commentStep`, `fakerTransform`, `enhancedRetryFailedStep`, `allure`, `htmlReporter`, `wdio`, `selenoid`, `screenshotOnFail`, `pauseOnFail`, `stepByStepReport`), or when 3.x APIs are in use (`ai.request` function, Joi schemas in `seeResponseMatchesJsonSchema`, `restart: 'browser'`, `I.retry()`, `I.limitTime()`, Playwright `customLocators`)."
 ---
 
-
 # Migrate CodeceptJS 3.x → 4.x
 
 CodeceptJS 4 is **ESM-only and TypeScript-first**. There is no compatibility shim for CommonJS — every helper, page object, custom step, and config file must be ESM. This skill drives a project through the upgrade end-to-end.
@@ -103,8 +102,8 @@ Run, in order:
 5. Grep the repo for `tryTo(`, `retryTo(`, `eachElement(`, `commentStep(`, `softExpect`, `I.softExpect`, `Joi.`, `restart: 'browser'`, `I.retry(`, `I.limitTime(`, and (unless the user chose to keep it) `Mochawesome` / `--reporter mochawesome` — none should remain. `step.retry(` / `step.timeout(` passed as a step argument is the expected replacement, not a leftover.
 6. If the project used `autoLogin`: confirm the `auth` plugin restores sessions and roles.
 
-## Pointers
+## Related skills
 
-- `node_modules/codeceptjs/docs/migration-4.md` — full reference (this skill is a workflow over it)
 - `codeceptjs-auth` — replacement for the removed `autoLogin` plugin
-- `codeceptjs-fundamentals` — run **after** migration to confirm the new setup is wired correctly
+- `codeceptjs-fundamentals` — run **after** migration to confirm wiring
+- Full reference: `node_modules/codeceptjs/docs/migration-4.md`