@testdriverai/agent 7.11.229-test → 7.11.232-test

package/docs/test-results-json.mdx

@@ -1,258 +1,221 @@
 ---
-title: "Test Result JSON"
-sidebarTitle: "Test Result JSON"
-description: "Per-test JSON result files with metadata, versions, and infrastructure details"
+title: "Test result JSON"
+sidebarTitle: "Test result JSON"
+description: "Read each test's result, replay link, and run details from a JSON file."
 icon: "file-code"
 ---
 
-## Overview
+TestDriver writes a JSON file for each test after it finishes.
 
-TestDriver writes a JSON result file for each test case after it finishes. These files have full metadata about the test run. This includes the SDK version, the runner version, infrastructure details, interaction statistics, and links to recordings.
+- It has the test's result, error, and links to its report and replay.
+- It also records versions, sandbox details, and how many `find()` calls hit the cache.
+- Use it in CI scripts instead of parsing console output.
 
-TestDriver writes the result files to:
+## Turn it on
 
-```
-.testdriver/results/<testFile>/<testName>.json
-```
+The TestDriver Vitest reporter writes the files. `npx testdriverai init` adds it to `vitest.config.js` for you:
 
-For example, a test file `tests/login.test.mjs` with a test named `"should log in"` produces:
+```javascript vitest.config.js
+import { defineConfig } from 'vitest/config';
+import TestDriver from 'testdriverai/vitest';
 
+export default defineConfig({
+  test: {
+    testTimeout: 300000,
+    hookTimeout: 300000,
+    reporters: ['default', TestDriver()],
+    setupFiles: ['testdriverai/vitest/setup'],
+  },
+});
 ```
-.testdriver/results/tests/login.test.mjs/should_log_in.json
-```
-
-<Note>
-  TestDriver makes the test names safe for the file system. It changes special characters to underscores. It cuts the names to 200 characters.
-</Note>
-
-## Enabling
-
-You do not need configuration. The TestDriver Vitest reporter plugin writes the JSON files automatically each time the tests run.
-
-## JSON Schema
-
-Each result file is organized into logical groups:
-
-### `versions`
-
-| Field | Type | Description |
-|---|---|---|
-| `versions.sdk` | `string \| null` | TestDriver SDK version (e.g. `"7.8.0"`) |
-| `versions.vitest` | `string \| null` | Vitest version used to run the test |
-| `versions.api` | `string \| null` | TestDriver API server version |
-| `versions.runnerBefore` | `string \| null` | Runner version at sandbox start |
-| `versions.runnerAfter` | `string \| null` | Runner version after auto-update |
-| `versions.runnerWasUpdated` | `boolean` | Whether the runner was auto-updated during provisioning |
-
-### `test`
-
-| Field | Type | Description |
-|---|---|---|
-| `test.file` | `string \| null` | Relative path to the test file |
-| `test.name` | `string \| null` | Name of the test case |
-| `test.suite` | `string \| null` | Name of the parent `describe` block |
-| `test.passed` | `boolean` | Whether the test passed |
-| `test.caseId` | `string \| null` | Database ID for this test case |
-| `test.runId` | `string \| null` | Database ID for the overall test run |
-| `test.error` | `string \| null` | Error message if the test failed |
-| `test.errorStack` | `string \| null` | Error stack trace if the test failed |
-
-### `urls`
 
-| Field | Type | Description |
-|---|---|---|
-| `urls.api` | `string \| null` | API root URL used for this test |
-| `urls.console` | `string \| null` | TestDriver console base URL |
-| `urls.vnc` | `string \| null` | VNC URL for the sandbox |
-| `urls.testRun` | `string \| null` | Direct link to this test case in the console |
-
-### `replay`
-
-The `replay` object contains the recording replay URL and derived embed links. The `gifUrl` and `embedUrl` are generated automatically from the replay URL.
-
-| Field | Type | Description |
-|---|---|---|
-| `replay.url` | `string \| null` | Recording replay URL |
-| `replay.gifUrl` | `string \| null` | Animated GIF thumbnail of the recording |
-| `replay.embedUrl` | `string \| null` | Embeddable replay URL (appends `&embed=true`) |
-| `replay.markdown` | `string \| null` | Ready-to-use Markdown embed with GIF linking to the replay |
+The reporter records each run in TestDriver, so it needs your API key in `TD_API_KEY`. If it can't record the run, it doesn't write the files.
 
-The `replay.markdown` field produces a clickable GIF badge you can paste directly into PR comments, README files, or issue descriptions:
+## Where the files go
 
-```markdown
-[![Test Recording](https://api.testdriver.ai/replay/abc123/gif?shareKey=xyz)](https://console.testdriver.ai/replay/abc123?share=xyz)
+```text
+.testdriver/results/<test file>/<test id>.json
 ```
 
-### `date`
+- `<test file>` is the test file's path relative to the project root, such as `tests/login.test.mjs`.
+- `<test id>` is Vitest's ID for the test, with characters other than letters, digits, `_`, `.`, and `-` replaced by `_`. Read the test's name from `test.name` inside the file.
 
-| Field | Type | Description |
-|---|---|---|
-| `date` | `string` | ISO 8601 timestamp when the test finished |
-
-### `team`
-
-| Field | Type | Description |
-|---|---|---|
-| `team.id` | `string \| null` | Team ID from the sandbox |
-| `team.sessionId` | `string \| null` | SDK session ID |
+The folder isn't cleared between runs. Delete `.testdriver/results` before a run if you only want the latest results.
 
-### `infrastructure`
-
-| Field | Type | Description |
-|---|---|---|
-| `infrastructure.sandboxId` | `string \| null` | Sandbox instance ID |
-| `infrastructure.instanceId` | `string \| null` | Instance ID |
-| `infrastructure.os` | `string \| null` | Operating system of the sandbox (`"linux"` or `"windows"`) |
-| `infrastructure.amiId` | `string \| null` | AWS AMI ID used for provisioning |
-| `infrastructure.e2bTemplateId` | `string \| null` | E2B template ID used for provisioning |
-| `infrastructure.imageVersion` | `string \| null` | Sandbox image version |
-
-### `realtime`
-
-| Field | Type | Description |
-|---|---|---|
-| `realtime.channel` | `string \| null` | Ably channel name used for communication |
-| `realtime.messageCount` | `number` | Number of messages published to the realtime channel |
-
-### `interactions`
-
-| Field | Type | Description |
-|---|---|---|
-| `interactions.total` | `number` | Total number of interactions recorded |
-| `interactions.cached` | `number` | Number of interactions served from cache |
-| `interactions.byType` | `object` | Breakdown of interactions by type (e.g. `find`, `click`, `assert`) |
+## Example
 
-## Example Output
-
-```json
+```json .testdriver/results/tests/login.test.mjs/1223128da3_0_0.json
 {
-  "sdkVersion": "7.8.0",
-  "vitestVersion": "4.0.0",
-  "apiVersion": "1.45.0",
-  "runnerVersionBefore": "2.1.0",
-  "runnerVersionAfter": "2.1.1",
-  "wasUpdated": true,
-  "apiUrl": "https://api.testdriver.ai",
-  "consoleUrl": "https://console.testdriver.ai",
-  "testRunLink": "https://console.testdriver.ai/runs/abc123/def456",
-  "dashcamUrl": "https://app.dashcam.io/replay/abc123",
-  "vncUrl": "wss://sandbox-123.testdriver.ai/vnc",
-  "date": "2025-01-15T14:30:00.000Z",
+  "versions": {
+    "sdk": "7.11.228",
+    "vitest": "4.0.8",
+    "api": "7.11.228",
+    "runnerBefore": "2.4.1",
+    "runnerAfter": "2.4.1",
+    "runnerWasUpdated": false
+  },
+  "test": {
+    "file": "tests/login.test.mjs",
+    "name": "shows an error for an empty form",
+    "suite": "Login",
+    "passed": false,
+    "caseId": "6712f0c4e1a2b3c4d5e6f701",
+    "runId": "6712f0b9e1a2b3c4d5e6f700",
+    "error": "AI Assertion failed: an error says the fields are required",
+    "errorStack": "Error: AI Assertion failed: ..."
+  },
+  "urls": {
+    "api": "https://api.testdriver.ai",
+    "console": "https://console.testdriver.ai",
+    "vnc": null,
+    "testRun": "https://console.testdriver.ai/runs/6712f0b9e1a2b3c4d5e6f700/6712f0c4e1a2b3c4d5e6f701"
+  },
+  "replay": {
+    "url": "https://console.testdriver.ai/replay/6789abcdef0123456789abcd?share=k3y",
+    "gifUrl": "https://api.testdriver.ai/replay/6789abcdef0123456789abcd/gif?shareKey=k3y",
+    "embedUrl": "https://console.testdriver.ai/replay/6789abcdef0123456789abcd?share=k3y&embed=true",
+    "markdown": "[![Test Recording](https://api.testdriver.ai/replay/6789abcdef0123456789abcd/gif?shareKey=k3y)](https://console.testdriver.ai/replay/6789abcdef0123456789abcd?share=k3y)"
+  },
+  "date": "2026-10-21T14:30:00.000Z",
   "team": {
     "id": "team_abc123",
     "sessionId": "sess_xyz789"
   },
   "infrastructure": {
-    "sandboxId": "sandbox-123",
-    "instanceId": "i-abc123",
+    "sandboxId": "sbx-123",
+    "instanceId": null,
     "os": "linux",
-    "amiId": "ami-0abc123",
+    "amiId": null,
     "e2bTemplateId": null,
-    "imageVersion": "v2.1.0"
+    "imageVersion": null
   },
   "realtime": {
-    "channel": "sandbox:sandbox-123",
+    "channel": "sandbox:sbx-123",
     "messageCount": 42
   },
   "interactions": {
-    "total": 15,
-    "cached": 3,
-    "byType": {
-      "find": 8,
-      "click": 5,
-      "assert": 2
-    }
+    "total": 3,
+    "cached": 1,
+    "byType": { "find": 3 }
   }
 }
 ```
 
-## Using Result Files in CI
+Values that TestDriver doesn't know are `null`.
 
-Result files are useful for extracting test metadata in CI pipelines without parsing log output.
+## Fields
 
-### GitHub Actions Example
+### versions
 
-Use `fromJSON` to parse a result file into a GitHub Actions expression you can reference in subsequent steps:
+| Field | Type | Description |
+| --- | --- | --- |
+| `versions.sdk` | `string \| null` | TestDriver SDK version |
+| `versions.vitest` | `string \| null` | Vitest version |
+| `versions.api` | `string \| null` | TestDriver API version |
+| `versions.runnerBefore` | `string \| null` | Sandbox runner version when the sandbox started |
+| `versions.runnerAfter` | `string \| null` | Runner version after any update during setup |
+| `versions.runnerWasUpdated` | `boolean` | Whether the runner was updated during setup |
 
-```yaml
-- name: Run tests
-  run: vitest run tests/login.test.mjs
+### test
 
-- name: Parse result
-  id: result
-  run: |
-    # Read the first JSON result file
-    FILE=$(find .testdriver/results -name '*.json' | head -n 1)
-    echo "json=$(cat "$FILE")" >> "$GITHUB_OUTPUT"
-
-- name: Comment on PR
-  if: fromJSON(steps.result.outputs.json).test.passed == false
-  uses: actions/github-script@v7
-  with:
-    script: |
-      const result = ${{ steps.result.outputs.json }};
-      await github.rest.issues.createComment({
-        owner: context.repo.owner,
-        repo: context.repo.repo,
-        issue_number: context.issue.number,
-        body: [
-          `❌ **${result.test.name}** failed`,
-          ``,
-          `Error: ${result.test.error}`,
-          ``,
-          result.replay.markdown,
-          ``,
-          `[View full recording](${result.urls.testRun})`
-        ].join('\n')
-      });
-```
+| Field | Type | Description |
+| --- | --- | --- |
+| `test.file` | `string \| null` | Test file path, relative to the directory you ran the tests from |
+| `test.name` | `string \| null` | The test's name |
+| `test.suite` | `string \| null` | The enclosing `describe` names, outermost first, joined with ` > ` |
+| `test.passed` | `boolean` | Whether the test passed |
+| `test.caseId` | `string \| null` | ID of this test's result in TestDriver |
+| `test.runId` | `string \| null` | ID of the whole run in TestDriver |
+| `test.error` | `string \| null` | The error message, if the test failed |
+| `test.errorStack` | `string \| null` | The error's stack trace, if the test failed |
+
+### urls
 
-You can also load all results into a matrix or iterate over them:
+| Field | Type | Description |
+| --- | --- | --- |
+| `urls.api` | `string \| null` | The TestDriver API the test used |
+| `urls.console` | `string \| null` | The TestDriver console |
+| `urls.vnc` | `string \| null` | The sandbox's VNC address |
+| `urls.testRun` | `string \| null` | This test's report in the console |
+
+### replay
+
+The test's [Dashcam](/dashcam) recording.
+
+| Field | Type | Description |
+| --- | --- | --- |
+| `replay.url` | `string \| null` | The replay link |
+| `replay.gifUrl` | `string \| null` | An animated GIF of the recording |
+| `replay.embedUrl` | `string \| null` | The replay link with `&embed=true`, for embedding |
+| `replay.markdown` | `string \| null` | Markdown for the GIF, linked to the replay, ready to paste into a pull request comment or README |
+
+`gifUrl`, `embedUrl`, and `markdown` are filled in only when the replay link includes a `?share=` key.
+
+### date, team, and infrastructure
+
+| Field | Type | Description |
+| --- | --- | --- |
+| `date` | `string` | When the test finished, as an ISO 8601 string |
+| `team.id` | `string \| null` | Your team's ID |
+| `team.sessionId` | `string \| null` | The SDK session ID |
+| `infrastructure.sandboxId` | `string \| null` | The sandbox's ID |
+| `infrastructure.instanceId` | `string \| null` | The sandbox's instance ID |
+| `infrastructure.os` | `string \| null` | `"linux"` or `"windows"` |
+| `infrastructure.amiId` | `string \| null` | The machine image the sandbox started from |
+| `infrastructure.e2bTemplateId` | `string \| null` | The sandbox template the sandbox started from |
+| `infrastructure.imageVersion` | `string \| null` | The sandbox image version |
+
+### realtime and interactions
+
+| Field | Type | Description |
+| --- | --- | --- |
+| `realtime.channel` | `string \| null` | The realtime channel used to talk to the sandbox |
+| `realtime.messageCount` | `number` | Messages sent on that channel |
+| `interactions.total` | `number` | Number of `find()` and `findAll()` calls |
+| `interactions.cached` | `number` | How many of those were answered from the [cache](/caching) |
+| `interactions.byType` | `object` | Counts per method, such as `{ "find": 3, "findAll": 1 }` |
+
+## Use the results in CI
+
+This GitHub Actions step adds a table of results to the job summary:
 
 ```yaml
 - name: Run tests
-  run: vitest run tests/*.test.mjs
-
-- name: Collect results
-  id: results
-  run: |
-    # Merge all result files into a JSON array
-    echo "json=$(find .testdriver/results -name '*.json' -exec cat {} + | jq -s '.')" >> "$GITHUB_OUTPUT"
+  run: npx vitest run
 
-- name: Summary
+- name: Summarize results
+  if: always()
   run: |
-    echo '## Test Results' >> $GITHUB_STEP_SUMMARY
-    RESULTS='${{ steps.results.outputs.json }}'
-    echo "$RESULTS" | jq -r '.[] | "| \(.test.name) | \(if .test.passed then "✅" else "❌" end) | \(.urls.testRun) |"' >> $GITHUB_STEP_SUMMARY
+    echo '| Test | Result | Report |' >> "$GITHUB_STEP_SUMMARY"
+    echo '| --- | --- | --- |' >> "$GITHUB_STEP_SUMMARY"
+    find .testdriver/results -name '*.json' -exec cat {} + \
+      | jq -rs '.[] | "| \(.test.name) | \(if .test.passed then "Passed" else "Failed" end) | \(.urls.testRun) |"' \
+      >> "$GITHUB_STEP_SUMMARY"
 ```
 
-### Reading Results Programmatically
-
-```javascript
-import fs from "fs";
-import path from "path";
+To read the results in Node.js:
 
-const resultsDir = ".testdriver/results";
+```javascript scripts/report.mjs
+import fs from 'node:fs';
+import path from 'node:path';
 
-function readResults(dir) {
-  const results = [];
-  for (const testDir of fs.readdirSync(dir, { recursive: true })) {
-    const fullPath = path.join(dir, testDir);
-    if (fullPath.endsWith(".json") && fs.statSync(fullPath).isFile()) {
-      results.push(JSON.parse(fs.readFileSync(fullPath, "utf-8")));
-    }
-  }
-  return results;
-}
+const dir = '.testdriver/results';
+const results = fs
+  .readdirSync(dir, { recursive: true })
+  .filter((file) => file.endsWith('.json'))
+  .map((file) => JSON.parse(fs.readFileSync(path.join(dir, file), 'utf8')));
 
-const results = readResults(resultsDir);
-const passed = results.filter(r => r.test.passed);
-const failed = results.filter(r => !r.test.passed);
+const failed = results.filter((result) => !result.test.passed);
+console.log(`${results.length - failed.length} passed, ${failed.length} failed`);
 
-console.log(`${passed.length} passed, ${failed.length} failed`);
-for (const r of failed) {
-  console.log(`  FAIL: ${r.test.name} — ${r.test.error}`);
-  console.log(`  Recording: ${r.urls.testRun}`);
-  console.log(`  Embed: ${r.replay.markdown}`);
+for (const result of failed) {
+  console.log(`FAIL ${result.test.name}: ${result.test.error}`);
+  console.log(`  Report: ${result.urls.testRun}`);
 }
 ```
+
+## Related
+
+- [Dashcam](/dashcam): how the replay is recorded
+- [Run in CI/CD](/ci-cd): run your tests on every pull request
+- [Run tests](/copilot/running-tests): run tests locally and in parallel