@testdriverai/agent 7.11.229-canary → 7.11.231-test

package/docs/events.mdx

@@ -1,358 +1,201 @@
 ---
 title: "Events"
 sidebarTitle: "Events"
-description: "Listen to SDK lifecycle events with wildcard support"
+description: "Listen to what TestDriver does during a test through its event emitter."
 icon: "bolt"
 mode: "wide"
 ---
 
-## Overview
-
-TestDriver uses [EventEmitter2](https://github.com/EventEmitter2/EventEmitter2) for its event system. Events use a namespace pattern with colons. They permit wildcard listeners.
-
-Use the emitter through `testdriver.emitter`:
-
-```javascript
-testdriver.emitter.on('command:start', (data) => {
-  console.log(`Running: ${data.command}`);
-});
-```
-
-### Configuration
-
-The internal emitter is created with:
-
-```javascript
-new EventEmitter2({
-  wildcard: true,
-  delimiter: ':',
-  maxListeners: 20,
-  verboseMemoryLeak: false,
-  ignoreErrors: false,
-});
-```
-
-### Wildcard Listeners
-
-Use `*` to match one level. Use `**` to match many levels:
-
-```javascript
-// Match all log events
-testdriver.emitter.on('log:*', (message) => {
-  console.log(message);
-});
-
-// Match all events in any namespace
-testdriver.emitter.on('**', (...args) => {
-  console.log('Event:', this.event, args);
-});
-```
-
-## Event Reference
-
-### Command Events
-
-Emitted during the execution of SDK commands (`click`, `type`, `find`, etc.).
-
-| Event | Payload |
-|---|---|
-| `command:start` | `{ command, depth, data, timestamp, sourcePosition }` |
-| `command:success` | `{ command, depth, data, duration, response, timestamp, sourcePosition }` |
-| `command:error` | `{ command, depth, data, error, duration, timestamp, sourcePosition }` |
-| `command:status` | `{ command, status: "executing", data, depth, timestamp }` |
-| `command:progress` | `{ command, status: "completed", timing, data, depth, timestamp }` |
-
-```javascript
-testdriver.emitter.on('command:start', ({ command, data }) => {
-  console.log(`Starting ${command}`, data);
-});
-
-testdriver.emitter.on('command:error', ({ command, error, duration }) => {
-  console.error(`${command} failed after ${duration}ms: ${error}`);
-});
-```
-
-### Step Events
-
-Emitted for each AI reasoning step within a command.
-
-| Event | Payload |
-|---|---|
-| `step:start` | `{ stepIndex, prompt, commandCount, timestamp, sourcePosition }` |
-| `step:success` | `{ stepIndex, prompt, commandCount, duration, timestamp, sourcePosition }` |
-| `step:error` | `{ stepIndex, prompt, error, duration?, timestamp, sourcePosition? }` |
-
-```javascript
-testdriver.emitter.on('step:start', ({ stepIndex, prompt }) => {
-  console.log(`Step ${stepIndex}: ${prompt}`);
-});
-```
-
-### Test Events
-
-Emitted when a test file execution starts.
-
-| Event | Payload |
-|---|---|
-| `test:start` | `{ filePath, timestamp }` |
-| `test:success` | *Emitted on test completion* |
-| `test:error` | *Emitted on test failure* |
-
-### Log Events
-
-Emitted for all log output from the SDK.
-
-| Event | Payload |
-|---|---|
-| `log:log` | `(message: string)` — general log message |
-| `log:warn` | `(message: string)` — warning message |
-| `log:debug` | `(message: string)` — debug output (only when `VERBOSE`/`DEBUG`/`TD_DEBUG` env set) |
-| `log:info` | `(message: string)` — informational message |
-| `log:error` | `(message: string)` — error message |
-| `log:narration` | `(message: string, overwrite?: boolean)` — in-place status line |
-| `log:markdown` | `(markdown: string)` — full static markdown content |
-| `log:markdown:start` | `(streamId: string)` — begin streaming markdown |
-| `log:markdown:chunk` | `(streamId: string, chunk: string)` — incremental chunk |
-| `log:markdown:end` | `(streamId: string)` — end streaming markdown |
+TestDriver reports what it's doing through an event emitter. Get it with [`getEmitter()`](/client#getemitter) and listen with `on()`:
 
 ```javascript
-// Capture all logs
-testdriver.emitter.on('log:*', function (message) {
-  console.log(`[${this.event}]`, message);
-});
-```
-
-### Screen Capture Events
-
-Emitted during screenshot capture.
+const emitter = testdriver.getEmitter();
 
-| Event | Payload |
-|---|---|
-| `screen-capture:start` | `{ scale, silent, display }` |
-| `screen-capture:end` | `{ scale, silent, display }` |
-| `screen-capture:error` | `{ error, scale, silent, display }` |
-
-### Sandbox Events
-
-Emitted for sandbox WebSocket lifecycle and communication.
-
-| Event | Payload |
-|---|---|
-| `sandbox:connected` | *No payload* — WebSocket connection established |
-| `sandbox:authenticated` | `{ traceId }` — authentication successful |
-| `sandbox:error` | `(err: Error \| string)` — connection or sandbox error |
-| `sandbox:sent` | `(message: object)` — WebSocket message sent |
-| `sandbox:received` | *No payload* — successful message reply received |
-| `sandbox:progress` | `{ step, message }` — sandbox setup progress |
-
-```javascript
-testdriver.emitter.on('sandbox:connected', () => {
-  console.log('Connected to sandbox');
-});
-
-testdriver.emitter.on('sandbox:progress', ({ step, message }) => {
-  console.log(`Sandbox: [${step}] ${message}`);
+emitter.on('sandbox:progress', ({ step, message }) => {
+  console.log(`[${step}] ${message}`);
 });
 ```
 
-### Redraw Events
-
-Emitted during screen stability detection. See [Redraw](/redraw) for more details.
-
-| Event | Payload |
-|---|---|
-| `redraw:status` | `{ redraw: { enabled, settled, hasChangedFromInitial, consecutiveFramesStable, diffFromInitial, diffFromLast, text }, network: { enabled, settled, rxBytes, txBytes, text }, timeout: { isTimeout, elapsed, max, text } }` |
-| `redraw:complete` | `{ screenSettled, hasChangedFromInitial, consecutiveFramesStable, networkSettled, isTimeout, timeElapsed }` |
-
-```javascript
-testdriver.emitter.on('redraw:complete', (result) => {
-  if (result.isTimeout) {
-    console.warn('Redraw timed out after', result.timeElapsed, 'ms');
-  }
-});
-```
+The emitter is an [EventEmitter2](https://github.com/EventEmitter2/EventEmitter2) instance. Event names are segments separated by colons, such as `redraw:complete`.
 
-### File Events
+## Listen in a test
 
-Emitted during file load/save operations in the agent.
+The emitter exists as soon as you create the client, so add listeners before the first command:
 
-| Event | Payload |
-|---|---|
-| `file:start` | `{ operation: "load" \| "save" \| "run", filePath, timestamp }` |
-| `file:stop` | `{ operation, filePath, duration, success, sourceMap?, reason?, timestamp }` |
-| `file:load` | `{ filePath, size, timestamp }` |
-| `file:save` | `{ filePath, size, timestamp }` |
-| `file:diff` | `{ filePath, diff: { patches, sourceMaps, summary: { additions, deletions, modifications } }, timestamp }` |
-| `file:error` | `{ operation, filePath, error, duration?, timestamp }` |
+```javascript tests/login.test.mjs
+import { describe, expect, it } from 'vitest';
+import { TestDriver } from 'testdriverai/vitest/hooks';
 
-### Error Events
+describe('Login', () => {
+  it('shows an error for an empty form', async (context) => {
+    const testdriver = TestDriver(context);
+    const emitter = testdriver.getEmitter();
 
-Emitted for errors at various severity levels.
+    const clicks = [];
+    emitter.on('mouse-click', ({ x, y, button }) => {
+      clicks.push({ x, y, button });
+    });
 
-| Event | Payload |
-|---|---|
-| `error:fatal` | `(error: string \| Error)` — terminates the process |
-| `error:general` | `(message: string)` — non-fatal error |
-| `error:sandbox` | `(err: Error \| string)` — sandbox/WebSocket error |
+    await testdriver.provision.chrome({ url: 'https://testdriver-sandbox.vercel.app/login' });
+    await testdriver.find('Sign in button below the password field').click();
 
-```javascript
-testdriver.emitter.on('error:*', function (err) {
-  console.error(`[${this.event}]`, err);
+    console.log(clicks); // e.g. [{ x: 683, y: 412, button: 'left' }]
+    expect(await testdriver.assert('an error says the fields are required')).toBe(true);
+  });
 });
 ```
 
-### SDK Events
-
-Emitted for API request lifecycle.
+## Use wildcards
 
-| Event | Payload |
-|---|---|
-| `sdk:request` | `{ path }` — outgoing API request |
-| `sdk:response` | `{ path }` — API response received |
-| `sdk:retry` | `{ path, attempt, error, delayMs }` — request retry |
-
-### Other Events
-
-| Event | Payload |
-|---|---|
-| `exit` | `(exitCode: number)` — `0` for success, `1` for failure |
-| `status` | `(message: string)` — general status updates |
-| `mouse-click` | `{ x, y, button, click, double }` — mouse click performed |
-| `terminal:stdout` | Terminal stdout output |
-| `terminal:stderr` | Terminal stderr output |
-
-## Practical Examples
-
-### Custom Test Reporter
-
-```javascript
-const results = [];
-
-testdriver.emitter.on('command:success', ({ command, duration }) => {
-  results.push({ command, duration, status: 'pass' });
-});
-
-testdriver.emitter.on('command:error', ({ command, duration, error }) => {
-  results.push({ command, duration, status: 'fail', error });
-});
-
-// After test completes
-afterAll(() => {
-  console.table(results);
-});
-```
-
-### Progress Monitoring
+`*` matches one segment. `**` matches any number of segments. Use a `function` listener, not an arrow function, to read the event's name from `this.event`:
 
 ```javascript
-testdriver.emitter.on('step:start', ({ stepIndex, prompt }) => {
-  process.stdout.write(`\r  Step ${stepIndex}: ${prompt}`);
-});
-
-testdriver.emitter.on('command:progress', ({ command, timing }) => {
-  process.stdout.write(`\r  ${command} completed in ${timing}ms`);
+// Every error event: error:fatal, error:sdk, and so on
+emitter.on('error:*', function (error) {
+  console.error(`[${this.event}]`, error);
 });
-```
-
-### Debug Logging
 
-```javascript
-// Log every event (verbose)
-testdriver.emitter.on('**', function (...args) {
-  console.debug(`[EVENT] ${this.event}`, ...args);
+// Every event, with its arguments
+emitter.on('**', function (...args) {
+  console.debug(this.event, ...args);
 });
 ```
 
-## Types
-
-```typescript
-interface CommandStartEvent {
-  command: string;
-  depth: number;
-  data: Record<string, any>;
-  timestamp: number;
-  sourcePosition: SourcePosition;
-}
-
-interface CommandSuccessEvent {
-  command: string;
-  depth: number;
-  data: Record<string, any>;
-  duration: number;
-  response: any;
-  timestamp: number;
-  sourcePosition: SourcePosition;
-}
-
-interface CommandErrorEvent {
-  command: string;
-  depth: number;
-  data: Record<string, any>;
-  error: string;
-  duration: number;
-  timestamp: number;
-  sourcePosition: SourcePosition;
-}
-
-interface StepStartEvent {
-  stepIndex: number;
-  prompt: string;
-  commandCount: number;
-  timestamp: number;
-  sourcePosition: SourcePosition;
-}
-
-interface StepSuccessEvent {
-  stepIndex: number;
-  prompt: string;
-  commandCount: number;
-  duration: number;
-  timestamp: number;
-  sourcePosition: SourcePosition;
-}
-
-interface RedrawStatusEvent {
-  redraw: {
-    enabled: boolean;
-    settled: boolean;
-    hasChangedFromInitial: boolean;
-    consecutiveFramesStable: number;
-    diffFromInitial: number;
-    diffFromLast: number;
-    text: string;
-  };
-  network: {
-    enabled: boolean;
-    settled: boolean;
-    rxBytes: number;
-    txBytes: number;
-    text: string;
-  };
-  timeout: {
-    isTimeout: boolean;
-    elapsed: number;
-    max: number;
-    text: string;
-  };
-}
-
-interface RedrawCompleteEvent {
-  screenSettled: boolean;
-  hasChangedFromInitial: boolean;
-  consecutiveFramesStable: number;
-  networkSettled: boolean;
-  isTimeout: boolean;
-  timeElapsed: number;
-}
-
-interface SandboxProgressEvent {
-  step: string;
-  message: string;
-}
-
-interface SourcePosition {
-  file: string;
-  line: number;
-  column: number;
-}
-```
+`log:*` matches `log:log` and `log:warn` but not `log:markdown:start`, which has three segments. Use `log:**` to match all of them.
+
+## Log events
+
+The SDK prints these to the console unless you turn logging off with the `logging` option.
+
+| Event | Arguments | When |
+| --- | --- | --- |
+| `log:log` | `(message: string)` | General output, such as each action and `find()` |
+| `log:warn` | `(message: string)` | A warning |
+| `log:debug` | `(message: string)` | Debug detail. Always emitted, but printed only when `VERBOSE` or `TD_DEBUG` is set |
+| `log:narration` | `(message: string, overwrite?: boolean)` | Progress detail for an action, such as how long a redraw wait took |
+| `log:markdown` | `(markdown: string)` | A complete Markdown message from `ai()` |
+| `log:markdown:start` | `(streamId: string)` | `ai()` starts streaming a Markdown message |
+| `log:markdown:chunk` | `(streamId: string, chunk: string)` | The next piece of that message |
+| `log:markdown:end` | `(streamId: string)` | The message is complete |
+
+There's no `log:info` or `log:error` in the event list. Listen to `log:log` and `error:*` instead.
+
+## Error events
+
+| Event | Arguments | When |
+| --- | --- | --- |
+| `error:fatal` | `(error: string \| Error)` | TestDriver hit an error it can't recover from |
+| `error:general` | `(message: string)` | A non-fatal error |
+| `error:sdk` | `({ message, code, ... })` | A request to the TestDriver API failed |
+| `error:sandbox` | `(message: string)` | The sandbox reported an error, or the realtime connection to it failed |
+
+## Sandbox events
+
+| Event | Arguments | When |
+| --- | --- | --- |
+| `sandbox:authenticated` | `({ traceId })` | The connection to the sandbox is authenticated |
+| `sandbox:connected` | none | The client is connected to a sandbox |
+| `sandbox:progress` | `({ step, message })` | A setup step while the sandbox starts, or recovery after a dropped connection |
+| `sandbox:sent` | `(message: object)` | A message was sent to the sandbox |
+| `sandbox:received` | none | The sandbox replied to a message |
+| `sandbox:file` | `(message: object)` | The sandbox sent a file message |
+| `show-window` | `(url: string)` | The live preview URL is ready to open |
+
+## Action events
+
+| Event | Arguments | When |
+| --- | --- | --- |
+| `mouse-click` | `({ x, y, button, double })` | A click, double-click, right-click, `mouseDown()`, or `mouseUp()` was sent. `button` is `'left'` or `'right'`, and `double` is `true` for a double-click |
+| `exec:output` | `({ chunk, requestId })` | A piece of output from a running [`exec()`](/exec) command |
+| `screen-capture:start` | `({ scale, silent, display })` | TestDriver starts capturing the screen |
+| `screen-capture:end` | `({ scale, silent, display })` | The capture finished |
+| `screen-capture:error` | `({ error, scale, silent, display })` | The capture failed |
+
+## Redraw events
+
+Emitted only when [redraw detection](/redraw) is on.
+
+| Event | Arguments | When |
+| --- | --- | --- |
+| `redraw:status` | `(status: object)` | Every 500 ms while TestDriver waits for the screen to settle |
+| `redraw:complete` | `(result: object)` | The wait ended, either settled or timed out |
+
+<ResponseField name="redraw:status" type="object">
+  <Expandable title="properties">
+    <ResponseField name="redraw" type="object">
+      Screen state: `enabled`, `settled`, `hasChangedFromInitial`, `consecutiveFramesStable` (booleans), `diffFromInitial` and `diffFromLast` (percent of pixels that differ), and `text` (a short summary for the console).
+    </ResponseField>
+    <ResponseField name="network" type="object">
+      Network state: `enabled`, `settled` (booleans), `rxBytes` and `txBytes` (bytes since the last check), and `text`.
+    </ResponseField>
+    <ResponseField name="timeout" type="object">
+      Timing: `isTimeout` (boolean), `elapsed` and `max` (milliseconds), and `text`.
+    </ResponseField>
+  </Expandable>
+</ResponseField>
+
+<ResponseField name="redraw:complete" type="object">
+  <Expandable title="properties">
+    <ResponseField name="screenSettled" type="boolean">
+      Whether the screen settled. `true` when the screen check is off.
+    </ResponseField>
+    <ResponseField name="networkSettled" type="boolean">
+      Whether network traffic settled. `true` when the network check is off.
+    </ResponseField>
+    <ResponseField name="hasChangedFromInitial" type="boolean">
+      Whether the screen changed after the action.
+    </ResponseField>
+    <ResponseField name="consecutiveFramesStable" type="boolean">
+      Whether the last frames stopped changing.
+    </ResponseField>
+    <ResponseField name="isTimeout" type="boolean">
+      Whether the wait ended because it ran out of time.
+    </ResponseField>
+    <ResponseField name="timeElapsed" type="number">
+      How long the wait took, in milliseconds.
+    </ResponseField>
+  </Expandable>
+</ResponseField>
+
+## API request events
+
+| Event | Arguments | When |
+| --- | --- | --- |
+| `sdk:request` | `({ path })` | A request to the TestDriver API starts |
+| `sdk:response` | `({ path })` | The API responded |
+| `sdk:retry` | `({ path, attempt, error, delayMs })` | A failed request will be retried after `delayMs` milliseconds |
+
+## ai() command events
+
+These fire for the commands the AI runs inside [`ai()`](/errors#aierror). Commands you call yourself, such as `testdriver.click()`, don't emit them.
+
+| Event | Arguments |
+| --- | --- |
+| `command:start` | `({ command, depth, data, timestamp, sourcePosition })` |
+| `command:status` | `({ command, status: 'executing', data, depth, timestamp })` |
+| `command:progress` | `({ command, status: 'completed', timing, data, depth, timestamp })` |
+| `command:success` | `({ command, depth, data, duration, response, timestamp, sourcePosition })` |
+| `command:error` | `({ command, depth, data, error, duration, timestamp, sourcePosition })` |
+
+- `command` is the command name, and `data` is the full command the AI generated.
+- `error` is the error message as a string.
+- `timing` and `duration` are in milliseconds.
+
+## Other events
+
+| Event | Arguments | When |
+| --- | --- | --- |
+| `runner:log` | `({ level, message, timestamp })` | A log line from the sandbox's runner. Sent only when debug logging is on |
+| `exit` | `(exitCode: number)` | The agent stopped. `0` means success, `1` means failure |
+
+The event list also contains names that Vitest tests don't emit:
+
+- `step:*`, `test:*`, and `file:*` come from running YAML test files.
+- `command:stop`, `command:location`, `sandbox:error`, `sandbox:disconnected`, `sdk:error`, `sdk:progress`, `status`, `mouse-move`, `terminal:stdout`, `terminal:stderr`, `matches:show`, and `vm:show` aren't currently emitted.
+
+## Related
+
+- [getEmitter()](/client#getemitter): get the client's emitter
+- [Redraw](/redraw): what the redraw events measure
+- [Errors](/errors): the errors TestDriver throws
+- [Debug a failing test](/debugging-with-screenshots): find out why a test failed

package/docs/examples/{hover-text-with-description.mdx → add-to-cart.mdx}

@@ -1,22 +1,16 @@
 ---
-title: "Hover Text With Description Test Example"
-sidebarTitle: "Hover Text With Description"
-description: "TestDriver SDK - Hover Text With Description Test (Vitest) Converted from: testdriver/acceptance/hover-text-with-description.yaml."
+title: "Add an item to a cart"
+sidebarTitle: "Add to cart"
+description: "Log in to a demo store, add a product to the cart, and assert that it's there."
 icon: "hand-pointer"
 mode: "wide"
 ---
 
-## Overview
+This test logs in to a demo store, adds a product to the cart, and asserts that the cart has an item.
 
-TestDriver SDK - Hover Text With Description Test (Vitest) Converted from: testdriver/acceptance/hover-text-with-description.yaml
+## Watch it run
 
-Review the source code below to understand the implementation details and patterns used.
-
-## Live Test Run
-
-Watch this test execute in a real sandbox environment:
-
-{/* hover-text-with-description.test.mjs output */}
+{/* add-to-cart.test.mjs output */}
 <iframe 
   src="https://api-test.testdriver.ai/api/v1/testdriver/testcase/6ad6b5f0022b6043aeec2beb/replay" 
   width="100%" 
@@ -25,20 +19,15 @@ Watch this test execute in a real sandbox environment:
   allow="fullscreen"
 />
 
-## Source Code
-
-```javascript title="hover-text-with-description.test.mjs"
-/**
- * TestDriver SDK - Hover Text With Description Test (Vitest)
- * Converted from: testdriver/acceptance/hover-text-with-description.yaml
- */
+## The test
 
+```javascript title="add-to-cart.test.mjs"
 import { describe, expect, it } from "vitest";
 import { TestDriver } from "testdriverai/vitest/hooks";
 
 /**
- * Perform login flow for SauceLabs demo app
- * @param {TestDriver} client - TestDriver client
+ * Log in to the TestDriver sandbox demo store.
+ * @param {object} client - TestDriver instance
  * @param {string} username - Username (default: 'standard_user')
  */
 async function performLogin(client, username = "standard_user") {
@@ -55,14 +44,11 @@ async function performLogin(client, username = "standard_user") {
 
 describe("Hover Text With Description Test", () => {
   it("should add TestDriver Hat to cart and verify", async (context) => {
-    const testdriver = TestDriver(context, { ip: context.ip || process.env.TD_IP,
-      headless: true,
-    });
+    const testdriver = TestDriver(context);
     await testdriver.provision.chrome({
       url: "http://testdriver-sandbox.vercel.app/login",
     });
 
-    //
     // Perform login first
     await performLogin(testdriver);
 
@@ -85,20 +71,29 @@ describe("Hover Text With Description Test", () => {
 });
 ```
 
-## Running This Example
+## How it works
+
+- Each `find()` description starts with the visible label, then adds context ("Add to Cart, add to cart button under TestDriver Hat").
+- The context tells the AI which of several identical buttons to pick.
+- `performLogin()` is the reusable helper from [Click an icon](/examples/click-icon). It reads the on-screen password with `extract()` and types it with `secret: true`.
+- `assert()` checks the result in plain language, not by inspecting markup.
+
+## Run it
+
+The test is in the [TestDriver SDK repository](https://github.com/testdriverai/testdriverai/tree/main/examples). Clone it, install dependencies, and run the file:
 
 ```bash
-# Clone the TestDriver repository
 git clone https://github.com/testdriverai/testdriverai
-
-# Install dependencies
 cd testdriverai
 npm install
-
-# Run this specific example
-vitest run examples/hover-text-with-description.test.mjs
+npx vitest run examples/add-to-cart.test.mjs
 ```
 
-<Note>
-  Make sure you have `TD_API_KEY` set in your environment. Get one at [testdriver.ai](https://testdriver.ai).
-</Note>
+The test reads your API key from `TD_API_KEY`. Create a key at [console.testdriver.ai/team/api-keys](https://console.testdriver.ai/team/api-keys). Then set it in your environment or a `.env` file.
+
+## Related
+
+- [find()](/find)
+- [click()](/click)
+- [Locate elements](/locating-elements)
+- [Reuse code across tests](/reusable-code)