codeceptjs 4.0.0-rc.17 → 4.0.0-rc.19

package/docs/advanced.md

@@ -0,0 +1,201 @@
+---
+permalink: /advanced
+title: Advanced Usage
+---
+
+# Advanced Usage
+
+## Data Driven Tests
+
+Execute the same scenario on a different data set.
+
+Let's say you want to test login for different user accounts.
+In this case, you need to create a datatable and fill it in with credentials.
+Then use `Data().Scenario` to include this data and generate multiple scenarios:
+
+```js
+// Define data table inside a test or load from another module
+let accounts = new DataTable(['login', 'password']); //
+accounts.add(['davert', '123456']); // adding records to a table
+accounts.add(['admin', '123456']);
+
+// You can skip some data. But add them to report as skipped (just like with usual scenarios):
+accounts.xadd(['admin', '23456'])
+
+// Pass dataTable to Data()
+// Use special param `current` to get current data set
+Data(accounts).Scenario('Test Login', ({ I, current }) => {
+  I.fillField('Username', current.login); // current is reserved!
+  I.fillField('Password', current.password);
+  I.click('Sign In');
+  I.see('Welcome '+ current.login);
+});
+
+
+// Also you can set only for Data tests. It will launch executes only the current test but with all data options
+Data(accounts).only.Scenario('Test Login', ({ I, current }) => {
+  I.fillField('Username', current.login); // current is reserved!
+  I.fillField('Password', current.password);
+  I.click('Sign In');
+  I.see('Welcome '+ current.login);
+});
+```
+
+*Important: you can't use name `current` for pageObjects or helpers in data scenarios*
+
+This will produce 2 tests with different data sets.
+Current data set is appended to a test name in output:
+
+```sh
+✓ Test Login | {"login":"davert","password":"123456"}
+✓ Test Login | {"login":"admin","password":"123456"}
+S Test Login | {"login":"admin","password":"23456"}
+```
+
+```js
+// You can filter your data table
+Data(accounts.filter(account => account.login == 'admin')
+.Scenario('Test Login', ({ I, current }) => {
+  I.fillField('Username', current.login);
+  I.fillField('Password', current.password);
+  I.click('Sign In');
+  I.see('Welcome '+ current.login);
+});
+```
+
+This will limit data sets accoring passed function:
+
+```sh
+✓ Test Login | {"login":"admin","password":"123456"}
+S Test Login | {"login":"admin","password":"23456"}
+```
+
+Data sets can also be defined with array, generator, or a function.
+
+```js
+Data(function*() {
+  yield { user: 'davert'};
+  yield { user: 'andrey'};
+}).Scenario() // ...
+```
+
+*HINT: If you don't use DataTable. add `toString()` method to each object added to data set, so the data could be pretty printed in a test name*
+
+
+## Debug
+
+CodeceptJS provides a debug mode in which additional information is printed.
+It can be turned on with `--debug` flag.
+
+```sh
+npx codeceptjs run --debug
+```
+
+to receive even more information turn on `--verbose` flag:
+
+```sh
+npx codeceptjs run --verbose
+```
+
+> You can pause execution and enter **interactive console** mode by calling `pause()` inside your test.
+
+To see a complete internal debug of CodeceptJS use `DEBUG` env variable:
+
+```sh
+DEBUG=codeceptjs:* npx codeceptjs run
+```
+
+For an interactive debugging use NodeJS debugger. In **WebStorm**:
+
+```sh
+node $NODE_DEBUG_OPTION ./node_modules/.bin/codeceptjs run
+```
+
+For **Visual Studio Code**, add the following configuration in launch.json:
+
+```json
+{
+  "type": "node",
+  "request": "launch",
+  "name": "codeceptjs",
+  "args": ["run", "--grep", "@your_test_tag"],
+  "program": "${workspaceFolder}/node_modules/codeceptjs/bin/codecept.js"
+}
+```
+
+
+## Test Options
+
+Features and Scenarios have their options that can be set by passing a hash after their names:
+
+```js
+Feature('My feature', {key: val});
+
+Scenario('My scenario', {key: val},({ I }) => {});
+```
+
+You can use this options for build your own [plugins](https://codecept.io/hooks/#plugins) with [event listners](https://codecept.io/hooks/#api). Example: 
+
+```js
+  // for test
+  event.dispatcher.on(event.test.before, (test) => {
+    ...
+    if (test.opts.key) {
+      ...
+    }
+    ...
+  });
+  // or for suite
+  event.dispatcher.on(event.suite.before, (suite) => {
+    ...
+    if (suite.opts.key) {
+      ...
+    }
+    ...
+  });
+```
+
+## Direct Helper Access
+
+Some scenarios need the underlying SDK directly — a raw `page.evaluate`, a `page.on('request')` listener, an experimental Playwright API, or a wdio command the `WebDriver` helper doesn't expose. The `expose` plugin injects helper internals as scenario arguments so you can call them inline.
+
+```js
+Scenario('intercept network', async ({ I, page }) => {
+  page.on('request', req => console.log(req.method(), req.url()))
+  I.amOnPage('/')
+  const title = await page.evaluate(() => document.title)
+  I.see(title)
+})
+```
+Enable `expose` plugin in config and use public properties from a corresponding helper.
+Map each injection name to `HelperName.propertyName`:
+
+```js
+plugins: {
+  expose: {
+    enabled: true,
+    inject: {
+      page: 'Playwright.page',
+      browser: 'Playwright.browser',
+      browserContext: 'Playwright.browserContext',
+      wdio: 'WebDriver.browser',
+    }
+  }
+}
+```
+
+There is a shorthand mode:
+
+```js
+plugins: {
+  expose: {
+    enabled: true,
+    inject: { page: 'page' }   // resolves Playwright.page or Puppeteer.page
+  }
+}
+```
+A value with no dot is shorthand for "the first configured browser helper that exposes this property". Allowed properties: `page`, `browser`, `browserContext`, `context`.
+
+The injected value is a live proxy. Every property access reads the current helper property at that moment, so tab switches (`I.openNewTab`, `I.switchToNextTab`) propagate automatically — the next call through `page` targets the new tab.
+
+Calls pass straight to the underlying SDK. They aren't wrapped as CodeceptJS steps and don't appear in step output, so `await page.evaluate(...)` behaves as native Playwright.

package/docs/agents.md

@@ -0,0 +1,159 @@
+---
+permalink: /agents
+title: Agentic Testing
+---
+
+# Agentic Testing
+
+CodeceptJS ships an **MCP server and a skillset** that lets an AI agent (Claude Code, Cursor, Codex, others) write and fix tests by driving the real browser. The agent runs the same `I.*` commands the test does, reads how the page responds, and only commits the lines that succeeded.
+
+## Why MCP
+
+The traditional agent testing loop is test/fix/retry, where the agent executes a test, watches it fail, reads artifacts, performs code fixes, and reruns the test. The agent applies fixes by intelligent guess — looking at the ARIA tree, HTML, and screenshot — then assumes the fix is enough and reruns the test hoping it will pass. If the guess is wrong and the test runs for over a minute, it may take dozens of minutes of iteration and a lot of wasted tokens.
+
+To improve that flow, the agent can spawn a browser and open the page the way the test does. This lets it interact with the page more freely and perform multi-step actions. But putting that experience back into test code is not efficient either: actions executed in the browser may not be relevant in test context, so the agent ends up in another guess-and-try loop.
+
+The problem is that **the test runs in a different context than the agent**.
+
+The agent can launch a test but can't control it while it's running. It can't access the browser. It can't set a breakpoint.
+
+This is where CodeceptJS MCP steps in. Connected to the agent, it can:
+
+- run a test and pause it on failure
+- interact with the browser in a test context
+- test locators and perform actions live while the test is running
+- write successful actions to the test file
+
+This lets the agent get a test working in one iteration. The agent can live-write the test before your eyes by exploring the page and performing actions that eventually land in the CodeceptJS test file.
+
+**Live debugging of tests** is what CodeceptJS MCP provides. The agent receives feedback faster — not from a whole test execution but from specific actions on a specific page — so it can adjust and react faster, trying different approaches.
+
+The MCP server is the agent-facing equivalent of the `pause()` REPL — same access, driven by tool calls instead of keystrokes. Full tool reference at [/mcp](/mcp).
+
+## The loop
+
+Whether the agent is writing a new test or fixing an old one, it follows the same cycle.
+
+1. **Open the page.** Run a stub test (new work) or set a breakpoint at the failing step (fix). The browser lands at the right starting point and yields control to the agent.
+2. **Read the page.** MCP saves HTML, ARIA, and screenshot of the page to files (and the agent can call the `snapshot` tool to refresh them). The agent reads those files before deciding what to try next, controlling its token usage.
+3. **Run a CodeceptJS command.** The agent tries `I.*` commands like `I.click('Add to cart')`, `I.fillField('Email', secret(process.env.EMAIL))`, `I.see('Confirmed')`. On success, that line goes into the test — same syntax.
+4. **Check the result.** The response after each command shows the new page state. If the URL changed and the modal opened, the line goes into the verified sequence. If not, the agent reads the page again and tries a different locator or a wait.
+5. **Move forward.** The agent looks at the new state and chooses the next command. Steps 2–4 repeat until the scenario is whole.
+6. **Commit to the file.** The agent edits the test — replaces `pause()` (new tests) or the broken line (fixes) with the verified sequence — then reruns end-to-end and reads the trace to confirm.
+
+## How the agent reads the page
+
+MCP commands are token efficient — they don't stream large HTML pages back to the model. MCP writes artifacts to disk under `output/trace_*/` and returns file paths. The agent reads each artifact with its own bash tools — `cat`, `grep`, `jq`.
+
+A `run_code` response, for example, looks like this:
+
+```json
+{
+  "status": "success",
+  "artifacts": {
+    "url": "http://localhost:8000/",
+    "html": "file:///output/trace_run_code_.../mcp_page.html",
+    "aria": "file:///output/trace_run_code_.../mcp_aria.txt",
+    "screenshot": "file:///output/trace_run_code_.../mcp_screenshot.png",
+    "console": "file:///output/trace_run_code_.../mcp_console.json",
+    "storage": "file:///output/trace_run_code_.../mcp_storage.json"
+  }
+}
+```
+
+Only `url` is inline. The rest are paths the agent opens with the right tool:
+
+| Artifact | How the agent reads it |
+|----------|------------------------|
+| `*_screenshot.png` | As an image — most agents are multimodal |
+| `*_aria.txt` | Whole — small and structured |
+| `*_page.html` | With `grep` — too large for context, searchable for specific elements/attributes |
+| `*_console.json` | With `jq` — filter for errors, 4xx/5xx, deprecation warnings |
+| `*_storage.json` | Whole — cookies and `localStorage` snapshot |
+| `trace.md` | Whole — markdown index linking every step to its artifacts |
+
+Saved HTML is formatted, with non-semantic elements stripped out: `<style>`, `<script>`, Tailwind-style trash classes, and inline `style=""` attributes. `grep` can then effectively find the correct tree branch in raw page source. ARIA snapshots are smaller and more structured than HTML, which is why the agent prefers them when picking locators.
+
+## Setup
+
+When CodeceptJS is installed, the MCP server can be launched with this command:
+
+```bash
+npx codeceptjs-mcp
+```
+
+> See [/mcp](/mcp) for detailed client setup.
+
+We recommend pairing CodeceptJS MCP with the skills bundle.
+
+Install for any agent:
+
+```bash
+npx skills add codeceptjs/skills
+```
+
+Or, in Claude Code:
+
+```text
+/plugin marketplace add codeceptjs/skills
+/plugin install codeceptjs@codeceptjs-skills
+```
+
+## Usage Examples
+
+When MCP and skills are connected, the agent receives predefined workflows and can act effectively for testing purposes. Common scenarios it handles:
+
+### Writing a new test
+
+You ask: "Add a test for the checkout flow."
+
+The agent writes a stub:
+
+```javascript
+Scenario('checkout', ({ I }) => {
+  I.amOnPage('/cart')
+  pause()
+})
+```
+
+It runs the stub. The browser opens at `/cart` and yields control at `pause()`. The agent reads the ARIA tree, runs `I.click('Add to cart')`, sees the cart total update — that line goes into the verified sequence. It runs `I.fillField('Email', '...')`, sees the field accept the value, records it. Through `I.click('Continue to payment')`, `I.see('Payment')`, `I.fillField('Card', secret(process.env.TEST_CARD))`, `I.click('Pay')`, `I.see('Order confirmed')` — each command commits only after the response confirms it worked.
+
+When the scenario is whole, the agent edits the test file: replaces `pause()` with the verified sequence, renames the scenario, wraps credentials with `secret()`. It reruns the file end-to-end with `aiTrace` on and hands you the diff.
+
+### Fixing a failing test
+
+A test fails. You point the agent at the scenario.
+
+It opens `output/trace_<TestName>_*/trace.md` from the last run, reads the steps, and finds the one marked failed. Most of the time the screenshot and ARIA from that step explain the cause — "Save" is now "Save changes," or a spinner is gating the next action. The agent patches the line and reruns.
+
+When the trace doesn't say enough, the agent passes a step number to `run_test` so the test pauses right before the failing step. From the live page, it tries `I.click({ role: 'button', name: 'Save changes' })`, sees the modal close. Or `I.waitForInvisible('.spinner', 10)` followed by the original click — watches it pass. Whatever holds goes into the test.
+
+The fix lands with a one-line note explaining what changed.
+
+### Auto-fixing on CI
+
+After a failed run, the agent reads every trace under `output/`, clusters failures by signature, and patches what fits a small set of safe fixes (locator drift, missing waits, raw `I.wait(N)` replacement). It reruns only the failing scenarios, compares against the baseline, and writes a markdown report at `output/ci-fix.md`.
+
+If the fix held, the PR goes green. If it didn't, every edit is rolled back with `git checkout` and the report says which patterns the agent couldn't safely handle. No half-applied fixes left behind, no `retries: 3` masking the problem.
+
+## Skills bundle
+
+Skills teach the agent best practices for using CodeceptJS. Plug them in when you develop tests with agents, and update them regularly to ensure you use CodeceptJS in the most effective way.
+
+| Skill | Use case |
+|-------|----------|
+| `writing-codeceptjs-tests` | Author or extend a scenario. Runs the loop above with a stub-and-pause flow for greenfield work, incremental `run_code` for known flows. |
+| `debugging-codeceptjs-tests` | A test is failing or flaky. Reads the trace, decides whether to patch from the trace alone or set a breakpoint on the live page. |
+| `ci-fix-tests` | Conservative auto-repair on CI |
+| `refactoring-codeceptjs-tests` | Extract page objects, tame long locators, move raw browser code into helpers. Proposes in batches. |
+| `codeceptjs-fundamentals` | Obtain actual CodeceptJS knowledge. |
+| `codeceptjs-exploration` | Pick a stable locator from messy markup. |
+| `codeceptjs-run-analysis` | Read trace artifacts, cluster CI failures into root causes, verify a fix held across many traces. |
+| `codeceptjs-auth` | Authorize efficiently with `auth` plugin. |
+
+## Pointers
+
+- [/mcp](/mcp) — full MCP tool reference, client setup
+- [/aitrace](/aitrace) — trace plugin configuration and capture options
+- [/debugging](/debugging) — pause modes, IDE setup, the `pause` plugin
+- [skills repo](https://github.com/codeceptjs/skills) — source and install for non-Claude clients