codeceptjs 4.0.0-rc.18 → 4.0.0-rc.19

package/docs/aitrace.md

@@ -0,0 +1,266 @@
+---
+permalink: /aitrace
+title: AI Trace Plugin
+---
+
+# AI Trace Plugin
+
+AI Trace Plugin generates AI-friendly trace files for debugging with AI agents like Claude Code.
+
+When a test fails, you need to understand what went wrong: what the page looked like, what elements were present, what errors occurred, and what led to the failure. This plugin automatically captures all that information and organizes it in a format optimized for AI analysis.
+
+## Quick Start
+
+Enable the plugin in your `codecept.conf.js`:
+
+```javascript
+export const config = {
+  tests: './*_test.js',
+  output: './output',
+  helpers: {
+    Playwright: {
+      url: 'https://example.com',
+      // Optional: Enable HAR/trace for HTTP capture
+      recordHar: {
+        mode: 'minimal',
+        content: 'embed',
+      },
+      trace: 'on',
+      keepTraceForPassedTests: true,
+    },
+  },
+  plugins: {
+    aiTrace: {
+      enabled: true,
+    },
+  },
+}
+```
+
+Run tests:
+
+```bash
+npx codeceptjs run
+```
+
+After test execution, trace files are created in `output/trace_*/trace.md`.
+
+## Artifacts Created
+
+For each test, a `trace_<sha256>` directory is created with the following files:
+
+**trace.md** - AI-friendly markdown file with test execution history
+
+**0000_step_name_screenshot.png** - Screenshot for each step (file names include step names)
+
+**0000_step_name_page.html** - Full HTML of the page at each step. Processed through a `minify -> clean -> beautify` pipeline so the file is multi-line indented, free of `<script>` / `<style>` / `<noscript>` content, free of inline `style=""` attributes, and free of trash class names (Tailwind utilities, framework-generated `v-*` / `ember-*`, hashed classes, scoped `xl:hidden`-style classes). Semantic attributes (`id`, `aria-*`, `data-*`, `role`, etc.) are preserved.
+
+**0000_step_name_aria.txt** - ARIA accessibility snapshot (AI-readable structure without HTML noise; Playwright only)
+
+**0000_step_name_console.json** - Browser console logs, normalized to plain `{ type, text }` objects (Playwright `ConsoleMessage` instances are coerced via their `.type()` / `.text()` methods so the JSON file is genuinely useful — not full of empty objects).
+
+When HAR or trace recording is enabled in your helper config, links to those files are also included.
+
+**Note:** Artifact files are named using step names for easier identification (e.g., `0000_I_see_Product_screenshot.png` instead of just `0000_screenshot.png`).
+
+**Storage state:** Cookies and `localStorage` are intentionally **not** captured per-step by `aiTrace` (they rarely change between actions, so per-step `_storage.json` files would be noise). Use the `pageInfo` plugin or the MCP `snapshot()` tool when you need a storage snapshot.
+
+## Trace File Format
+
+The `trace.md` file contains a structured execution log with links to all artifacts:
+
+```markdown
+file: /path/to/test.js
+name: My test scenario
+time: 3.45s
+---
+
+I am on page "https://example.com"
+  > navigated to https://example.com/
+  > [HTML](./0000_I_am_on_page_https_example.com_page.html)
+  > [ARIA Snapshot](./0000_I_am_on_page_https_example.com_aria.txt)
+  > [Screenshot](./0000_I_am_on_page_https_example.com_screenshot.png)
+  > [Browser Logs](./0000_I_am_on_page_https_example.com_console.json) (7 entries)
+  > HTTP: see [HAR file](../har/...) for network requests
+
+I see "Welcome"
+  > navigated to https://example.com/
+  > [HTML](./0001_I_see_Welcome_page.html)
+  > [ARIA Snapshot](./0001_I_see_Welcome_aria.txt)
+  > [Screenshot](./0001_I_see_Welcome_screenshot.png)
+  > [Browser Logs](./0001_I_see_Welcome_console.json) (0 entries)
+```
+
+Files are named with step descriptions for easier navigation and debugging.
+
+## Configuration
+
+### Basic Configuration
+
+```javascript
+plugins: {
+  aiTrace: {
+    enabled: true,
+  }
+}
+```
+
+### Advanced Configuration
+
+```javascript
+plugins: {
+  aiTrace: {
+    enabled: true,
+
+    // Artifact capture options
+    captureHTML: true,          // Save HTML for each step
+    captureARIA: true,          // Save ARIA snapshots
+    captureBrowserLogs: true,   // Save console logs
+    captureHTTP: true,          // Links to HAR/trace files
+    captureDebugOutput: true,   // CodeceptJS debug messages
+
+    // Screenshot options
+    fullPageScreenshots: false, // Full page or viewport only
+
+    // Output options
+    output: './output',         // Where to save traces
+    deleteSuccessful: false,    // Delete traces for passed tests
+
+    // Step filtering
+    ignoreSteps: [
+      /^grab/,                  // Ignore all grab* steps
+      /^wait/,                  // Ignore all wait* steps
+    ],
+  }
+}
+```
+
+### Configuration Options
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `enabled` | boolean | `false` | Enable/disable the plugin |
+| `captureHTML` | boolean | `true` | Capture HTML for each step |
+| `captureARIA` | boolean | `true` | Capture ARIA snapshots |
+| `captureBrowserLogs` | boolean | `true` | Capture browser console logs |
+| `captureHTTP` | boolean | `true` | Capture HTTP requests (requires HAR/trace) |
+| `captureDebugOutput` | boolean | `true` | Capture CodeceptJS debug output |
+| `fullPageScreenshots` | boolean | `false` | Use full page screenshots |
+| `output` | string | `'./output'` | Directory for trace files |
+| `deleteSuccessful` | boolean | `false` | Delete traces for passed tests |
+| `ignoreSteps` | array | `[]` | Steps to ignore (regex patterns) |
+
+## Best Practices
+
+### Optimize for Failing Tests
+
+Save disk space by only keeping traces for failed tests:
+
+```javascript
+plugins: {
+  aiTrace: {
+    enabled: true,
+    deleteSuccessful: true,  // Only keep failing tests
+  }
+}
+```
+
+### Ignore Noise
+
+Don't capture logs for `grab` and `wait` steps:
+
+```javascript
+plugins: {
+  aiTrace: {
+    enabled: true,
+    ignoreSteps: [/^grab/, /^wait/],
+  }
+}
+```
+
+### Selective Artifact Capture
+
+Capture only what you need to reduce file sizes:
+
+```javascript
+plugins: {
+  aiTrace: {
+    enabled: true,
+    captureHTML: false,        // Skip HTML (saves ~500KB per step)
+    captureARIA: true,         // Keep ARIA (only ~16KB)
+    captureBrowserLogs: false, // Skip console logs
+  }
+}
+```
+
+### Enable HTTP Capture
+
+For network debugging, enable HAR/trace in your helper:
+
+```javascript
+helpers: {
+  Playwright: {
+    recordHar: {
+      mode: 'minimal',
+      content: 'embed',
+    },
+    trace: 'on',
+    keepTraceForPassedTests: true,
+  },
+  plugins: {
+    aiTrace: {
+      enabled: true,
+      captureHTTP: true,  // Links to HAR/trace files
+    },
+  },
+}
+```
+
+## Using with AI Agents
+
+The trace format is optimized for AI agents like Claude Code. When debugging a failing test, just point the AI agent to the `trace.md` file - it will read the file and all linked artifacts automatically to analyze the failure.
+
+## Troubleshooting
+
+### No trace files created
+
+**Possible causes:**
+1. Plugin not enabled
+2. No steps executed
+3. Tests skipped
+
+**Solution:**
+```bash
+# Check if plugin is enabled
+grep -A 5 "aiTrace" codecept.conf.js
+
+# Run with verbose output
+npx codeceptjs run --verbose
+```
+
+### ARIA snapshots missing
+
+**Possible cause:** Helper doesn't support `grabAriaSnapshot`
+
+**Solution:** Use Playwright or update to latest CodeceptJS
+
+### HAR files missing
+
+**Possible cause:** HAR/trace not enabled in helper config
+
+**Solution:**
+```javascript
+helpers: {
+  Playwright: {
+    recordHar: { mode: 'minimal' },
+    trace: 'on',
+  },
+}
+```
+
+## Related
+
+- [AI Features](/ai) - AI-powered testing features
+- [Plugins](/plugins) - All available plugins
+- [Configuration](/configuration) - General configuration
+- [Playwright Helper](/playwright) - Playwright-specific configuration

package/docs/api.md

@@ -0,0 +1,332 @@
+---
+permalink: /api
+title: API Testing
+---
+
+## API Testing
+
+CodeceptJS provides a way to write tests in declarative manner for REST and GraphQL APIs. 
+
+Take a look:
+
+```js
+I.sendGetRequest('/users/1');
+// returns { "user": { "name": "jon" }, "projects": [] }
+I.seeResponseCodeIsSuccessful();
+I.seeResponseContainsKeys(['user', 'projects']);
+I.seeResponseContainsJson({ user: { name: 'jon' } });
+I.seeResponseMatchesJsonSchema($ => {
+  return $.object(
+    user: $.object({
+      name: $.string(),
+    }),
+    projects: $.array()
+  )
+});
+```
+In this code we checked API request for:
+
+* status code
+* data inclusion
+* data structure
+
+These are the things you should generally test your APIs for.
+
+> 🤓 It is recommended to check only invariable parts of responses. Check for required fields and only values you control. For instance, it is not recommended to check id fields, date fields, as they can be frequently changed.
+
+## Installation
+
+Install CodeceptJS if it is not installed yet.
+
+```
+npm i codeceptjs --save-dev
+```
+
+Initialize CodeceptJS and select REST or GraphQL helper when asked for a helper:
+
+```
+npx codeceptjs init
+```
+
+## Configuration
+
+Ensure that inside `codecept.conf.js` in helpers section `REST` or `GraphQL` helpers are enabled.
+
+* If you use `REST` helper add `JSONResponse` helper below with no extra config:
+
+```js
+// inside codecept.conf.js
+// ...
+  helpers: {
+    REST: {
+      endpoint: 'http://localhost:3000/api'
+    },
+    // .. add JSONResponse helper here
+    JSONResponse: {}
+  }
+```
+* If you use `GraphQL` helper add `JSONResponse` helper, configuring it to use GraphQL for requests:
+
+```js
+  helpers: {
+    GraphQL: {
+      endpoint: 'http://localhost:3000/graphql'
+    },
+    // .. add JSONResponse helper here
+    JSONResponse: {
+      requestHelper: 'GraphQL',
+    }
+  }
+```
+
+Originally, REST and GraphQL helpers were not designed for API testing. 
+They were used to perform API requests for browser tests. As so, they lack assertion methods to API responses.
+
+[`JSONResponse`](/helpers/JSONResponse/) helper adds response assertions. 
+
+> 💡 In CodeceptJS assertions start with `see` prefix. Learn more about assertions by [opening reference for JSONResponse](/helpers/JSONResponse/) helper.
+
+Generate TypeScript definitions to get auto-completions for JSONResponse:
+
+```
+npx codeceptjs def
+```
+
+After helpers were configured and typings were generated, you can start writing first API test. By default, CodeceptJS saves tests in `tests` directory and uses `*_test.js` suffix. The `init` command created the first test for you to start.
+
+> Check [API Examples](https://github.com/codeceptjs/api-examples) to see tests implementations.
+
+## Requests
+
+[REST](/helpers/REST/) or [GraphQL](/helpers/GraphQL/) helpers implement methods for making API requests.
+Both helpers send requests via HTTP protocol from CodeceptJS process. 
+For most cases, you will need to have authentication. It can be passed via headers, which can be added to helper's configuration in `codecept.conf.js`. 
+
+```js
+helpers: {
+  REST: {
+    defaultHeaders: {
+      // use Bearer Authorization
+      'Authorization': 'Bearer 11111',
+      'Content-Type': 'application/json',
+      'Accept': 'application/json',
+    },
+  }
+}
+```
+
+Or you can use the browser cookies if you are running browser session.
+In this case use `setSharedCookies()` from `@codeceptjs/configure` package:
+
+```js
+import { setSharedCookies } from '@codeceptjs/configure'
+
+// call before exporting config
+setSharedCookies()
+
+export const config = {
+  // ...
+  helpers: {  
+    // also works with Playwright or Puppeteer
+    WebDriver: {
+      //... 
+    },
+
+    REST: { 
+      // ...
+    }
+  }
+}
+```
+
+### REST
+
+REST helper can send GET/POST/PATCH/etc requests to REST API endpoint:
+
+* [`I.sendGetRequest()`](/helpers/REST#sendGetRequest)
+* [`I.sendPostRequest()`](/helpers/REST#sendPostRequest)
+* [`I.sendPutRequest()`](/helpers/REST#sendPutRequest)
+* [`I.sendPatchRequest()`](/helpers/REST#sendPatchRequest)
+* [`I.sendDeleteRequest()`](/helpers/REST#sendDeleteRequest)
+* [`I.sendDeleteRequestWithPayload()`](/helpers/REST#sendDeleteRequestWithPayload)
+* ...
+
+Authentication headers can be set in [helper's config](https://codecept.io/helpers/REST/#configuration) or per test with headers or special methods like `I.amBearerAuthenticated`.
+
+Example:
+
+```js
+Feature('Users endpoint')
+
+Scenario('create user', ({ I }) => {
+    // this way we pass Bearer token
+  I.amBearerAuthenticated(secret('token-is-here'));
+  // for custom authorization with headers use
+  // I.haveRequestHeaders method
+
+  // here we send a POST request
+  const response = await I.sendPostRequest('/users', {
+    name: 'joe',
+    email: 'joe@mail.com'
+  });
+  // usually we won't need direct access to response object for API testing 
+  // but you can obtain it from request
+
+  // check the last request was successful
+  // this method introduced by JSONResponse helper
+  I.seeResponseCodeIsSuccessful();
+})
+```
+
+### GraphQL
+
+GraphQL have request format different then in REST API, but the response format is the same.
+It's plain old JSON. This why `JSONResponse` helper works for both API types.
+Configure authorization headers in `codecept.conf.js` and make your first query:  
+
+```js
+Feature('Users endpoint')
+
+Scenario('get user by query', ({ I }) => {
+  // make GraphQL query or mutation
+  const resp = await I.sendQuery('{ user(id: 0) { id name email }}');
+  I.seeResponseCodeIsSuccessful();
+
+  // GraphQL always returns key data as part of response
+  I.seeResponseContainsKeys(['data']);
+
+  // check data for partial inclusion
+  I.seeResponseContainsJson({
+    data: {
+      user: {
+        name: 'john doe',
+        email: 'johnd@mutex.com',
+      },
+    },
+  });
+});
+```
+
+GraphQL helper has two methods available:
+
+* [`I.sendQuery()`](/helpers/GraphQL#sendQuery)
+* [`I.sendMutation()`](/helpers/GraphQL#sendMutation)
+
+## Assertions
+
+`JSONResponse` provides set of assertions for responses in JSON format. These assertions were designed to check only invariable parts of responses. So instead of checking that response equals to the one provided, we will check for data inclusion and structure matching.
+
+For most of cases, you won't need to perform assertions by accessing `response` object directly. All assretions are performed under hood inside `JSONResponse` module. It is recommended to keep it that way, to keep tests readable and make test log to contain all assertions.
+
+```js
+Scenario('I make API call', ({ I }) => {
+  // request was made by REST 
+  // or by GraphQL helper
+
+  // check that response code is 2xx
+  I.seeResponseCodeIsSuccessful();
+
+  // check that response contains keys
+  I.seeResponseContainsKeys(['data', 'pages', 'meta']);
+});
+```
+
+### Response Status Codes
+
+[Response status codes](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status) can be checked to be equal to some value or to be in a specific range. 
+To check that response code is `200` call `I.seeResponseCodeIs`:
+
+```js
+I.seeResponseCodeIs(200);
+```
+But because other response codes in 2xx range are also valid responses, you can use `seeResponseCodeIsSuccessful()` which will match 200 (OK), 201 (Created), 206 (Partial Content) and others. Methods to check 3xx, 4xx, 5xx response statuses also available.
+
+```js
+// matches 200, 201, 202, ... 206
+I.seeResponseCodeIsSuccessful();
+
+// matches 300...308
+I.seeResponseCodeIsRedirection();
+
+// matches 400..451
+I.seeResponseCodeIsClientError();
+
+// matches 500-511
+I.seeResponseCodeIsServerError();
+```
+
+### Structure
+
+The most basic thing to check in response is existence of keys in JSON object. Use [`I.seeResponseContainsKeys()`](/helpers/JSONResponse#seeResponseContainsKeys) method for it:
+
+```js
+// response is { "name": "joe", "email": "joe@joe.com" }
+I.seeResponseContainsKeys(['name', 'email']);
+```
+
+> ℹ️ If response is an array, it will check that every element in array have provided keys
+
+However, this is a very naive approach. It won't work for arrays or nested objects.
+To check complex JSON structures `JSONResponse` helper uses [`Zod`](https://zod.dev) library.
+It has rich API to validate JSON by the schema defined using JavaScript.
+
+```js
+// import zod library,
+// it is installed with CodeceptJS
+import { z } from 'zod';
+
+// create schema definition using Zod API
+const schema = z.object({
+  email: z.string().email(),
+  phone: z.string().regex(/^\d{3}-\d{3}-\d{4}$/),
+  birthday: z.string().datetime().max(new Date('2004-01-01'))
+});
+
+// check that response matches that schema
+I.seeResponseMatchesJsonSchema(schema);
+```
+
+> 📋 **Migration Note**: CodeceptJS has migrated from Joi to Zod v4 for JSON schema validation.
+> If you have existing tests using Joi, please update them:
+> * Replace `const Joi = require('joi')` with `import { z } from 'zod'`
+> * Replace `Joi.object().keys({...})` with `z.object({...})`
+> * Replace `Joi.string().email()` with `z.string().email()`
+> * Replace `Joi.date()` with appropriate `z.string()` or `z.date()` types
+> * See [Zod documentation](https://zod.dev) for complete API reference
+
+### Data Inclusion
+
+To check that response contains expected data use `I.seeResponseContainsJson` method. 
+It will check the response data for partial match. 
+
+```js
+I.seeResponseContainsJson({
+  user: {
+    email: 'user@user.com'
+  }
+})
+```
+
+> ℹ️ If response is an array, it will check that at least one element in array matches JSON
+
+To perform arbitrary assertions on a response object use `seeResponseValidByCallback`. 
+It allows you to do any kind of assertions by using `expect` from [`chai`](https://www.chaijs.com) library.
+
+```js
+I.seeResponseValidByCallback(({ data, status, expect }) => {
+  // we receive data and expect to combine them for good assertion
+  expect(data.users.length).to.be.gte(10);
+})
+```
+
+## Extending JSONResponse
+
+To add more assertions it is recommended to create a custom helper.
+Inside it you can get access to latest JSON response:
+
+```js
+// inside a custom helper
+makeSomeCustomAssertion() {
+  const response = this.helpers.JSONResponse.response;
+}
+```