testdriverai 7.9.0 → 7.9.1-test

package/ai/skills/testdriver-self-hosted/SKILL.md

@@ -1,23 +1,48 @@
 ---
 name: testdriver:self-hosted
-description: Unlimited test execution, complete privacy, and the ability to customize everything — all for a predictable flat license fee.
+description: Our enterprise solution with unlimited test execution, assisted setup, and dedicated support.
 ---
 <!-- Generated from self-hosted.mdx. DO NOT EDIT. -->
 
-Self-hosted pricing is based on **parallel test capacity**: the number of tests you can run simultaneously on **your infrastructure**. 
-
-With self-hosting, you get:.
-
-- **Flat license fee** per parallel test slot
-- **Unlimited test execution** — run as many tests as you want
-- **No device-second metering** — predictable monthly costs
-- **Use your own AI keys** — control data usage with your own OpenAI, Anthropic, or other AI provider keys
-- **Custom hardware & software** — choose instance types, resolution, install specific software, and configure networking as needed
-- **Debug & Customize** — RDP into test machines, install custom software, modify the AMI, and debug issues directly. No black boxes.
-
-## Get Started
-
-Ready to self-host? Follow our comprehensive AWS setup guide:
+Self-hosted is our enterprise solution for teams that need unlimited test execution, infrastructure control, and dedicated support. Pricing is based on **parallel test capacity** with a flat license fee — no per-second billing.
+
+<CardGroup cols={2}>
+  <Card title="Unlimited Execution" icon="infinity">
+    Run as many tests as you want with no device-second metering. Predictable monthly costs.
+  </Card>
+  <Card title="Assisted Setup & Support" icon="headset">
+    Our team helps you deploy, configure, and optimize your infrastructure. Dedicated engineering support included.
+  </Card>
+  <Card title="Full Control" icon="gear">
+    Use your own AI keys, custom hardware, specific software, and network configurations. RDP into test machines for debugging.
+  </Card>
+  <Card title="Security & Compliance" icon="shield-check">
+    Keep data in your environment. Air-gapped deployment available for regulated industries.
+  </Card>
+</CardGroup>
+
+## Deployment Options
+
+Choose the level of control you need:
+
+| Component | Standard | Air-Gapped |
+|-----------|----------|------------|
+| **Test Sandboxes** | Your AWS | Your infrastructure (any cloud or on-prem) |
+| **Dashboard** | TestDriver hosted | Your infrastructure |
+| **API** | TestDriver hosted | Your infrastructure |
+| **AI Processing** | Your API keys | Your infrastructure |
+| **Data Storage** | Your AWS account | 100% your infrastructure |
+| **Network** | Internet access required | Fully air-gapped |
+| **Cloud Providers** | AWS | AWS, Azure, GCP, on-prem |
+
+### Standard Deployment
+
+Run test sandboxes on your AWS infrastructure while using TestDriver's hosted dashboard and API:
+
+- **Quick setup** via CloudFormation — deploy in hours
+- **Dashboard access** at [console.testdriver.ai](https://console.testdriver.ai)
+- **Your AI keys** — control costs with your own OpenAI, Anthropic, or other provider
+- **Custom AMIs** — install specific software, configure networking
 
 <Card
   title="AWS Setup Guide"
@@ -27,39 +52,96 @@ Ready to self-host? Follow our comprehensive AWS setup guide:
   Step-by-step instructions for deploying TestDriver on your AWS infrastructure using CloudFormation.
 </Card>
 
+### Air-Gapped Deployment
+
+Deploy the entire TestDriver stack in your environment for complete isolation:
+
+- **Full stack** — dashboard, API, and test infrastructure all in your environment
+- **No external dependencies** — data never leaves your network perimeter
+- **Any infrastructure** — AWS, Azure, GCP, or on-premises
+- **Regulated industries** — government, defense, healthcare, finance
+
+## Custom VM Images
+
+Build test environments with your applications, dependencies, and user data pre-installed. You get full access to:
+
+- **Golden VM** — our pre-configured base image with TestDriver agent, drivers, and optimizations
+- **Packer scripts** — build custom AMIs with your applications, user data, and configurations
+- **Faster test startup** — skip installation steps by baking dependencies into your image
+- **Consistent environments** — every test runs on an identical, reproducible machine
+
+<AccordionGroup>
+  <Accordion title="What can you customize?">
+    - Install applications (browsers, desktop apps, dev tools)
+    - Configure user accounts and credentials
+    - Set up network proxies and certificates
+    - Install fonts, language packs, and locales
+    - Pre-seed databases or test fixtures
+    - Configure Windows/Linux settings
+  </Accordion>
+  
+  <Accordion title="How it works">
+    1. We provide our golden VM base image and Packer scripts
+    2. You customize the scripts to install your software and configuration
+    3. Run Packer to build your custom AMI
+    4. Configure TestDriver to use your custom AMI for test sandboxes
+    5. Tests spin up with everything pre-installed — no setup time wasted
+  </Accordion>
+</AccordionGroup>
+
+## Implementation Process
+
+<Steps>
+  <Step title="Discovery Call">
+    Discuss your requirements, security constraints, and integration needs with our team.
+  </Step>
+  
+  <Step title="Architecture Review">
+    Our engineers design a deployment architecture that meets your security and compliance requirements.
+  </Step>
+  
+  <Step title="Deployment">
+    We work with your team to deploy TestDriver, including assisted setup and configuration.
+  </Step>
+  
+  <Step title="Integration">
+    Connect TestDriver to your CI/CD pipelines, internal tools, and workflows.
+  </Step>
+  
+  <Step title="Training & Handoff">
+    Comprehensive training for your team on operating and maintaining the deployment.
+  </Step>
+</Steps>
+
+## What's Included
 
-## Who Should Self-Host?
-
-Self-hosting is ideal for teams that:
-
-- **Run high test volumes** — Flat pricing becomes more economical at scale
-- **Want infrastructure control** — Custom hardware, specific software dependencies, or network configurations
-- **Prefer predictable costs** — Budget with confidence using flat monthly fees
-
-
-## How It Works
-
-With self-hosting, you run test sandboxes on your own AWS infrastructure. TestDriver still provides:
-
-- **Dashboard** — View test results, analytics, and reports at [console.testdriver.ai](https://console.testdriver.ai)
-- **API** — Orchestration and AI-powered test execution
-- **License Management** — Your parallel test capacity
-
-You provide:
-
-- **AWS Infrastructure** — EC2 instances running in your account
-- **AI API Keys** — Use your own OpenAI, Anthropic, or other AI provider keys
-- **Custom Configuration** — Hardware specs, networking, installed software
+- **Flat license fee** per parallel test slot
+- **Unlimited test execution** — no device-second charges
+- **Assisted setup** — our team helps you deploy and configure
+- **Dedicated support** — direct access to our engineering team
+- **Custom contract terms** — volume-based pricing, custom SLAs
+- **Professional services** — implementation assistance and training
 
-## Comparison vs Cloud
+## Comparison: Hosted vs Self-Hosted
 
-| Feature | Cloud | Self-Hosted |
-|---------|-------|-------------|
-| **Setup Time** | Minutes | Hours |
+| Feature | Hosted | Self-Hosted |
+|---------|--------|-------------|
+| **Setup Time** | Minutes | Hours (assisted) |
 | **Pricing Model** | Device-seconds | Flat license fee |
-| **Infrastructure Management** | TestDriver | You |
-| **Device Location** | TestDriver cloud | Your AWS account |
+| **Infrastructure** | TestDriver | Your AWS or any cloud |
 | **AI API Keys** | TestDriver's | Your own |
 | **Custom Software** | Limited | Full control |
 | **Hardware Selection** | Standard | Your choice |
 | **Debugging Access** | Replays only | Full RDP access |
+| **Support** | Community/Standard | Dedicated engineering |
+| **Air-Gapped Option** | No | Yes |
+
+## Get Started
+
+<Card
+  title="Schedule a Consultation"
+  icon="calendar"
+  href="https://testdriver.ai/demo"
+>
+  Discuss your requirements with our team and get a custom proposal for your self-hosted deployment.
+</Card>

package/ai/skills/testdriver-test-results-json/SKILL.md

@@ -0,0 +1,257 @@
+---
+name: testdriver:test-results-json
+description: Per-test JSON result files with metadata, versions, and infrastructure details
+---
+<!-- Generated from test-results-json.mdx. DO NOT EDIT. -->
+
+## Overview
+
+TestDriver automatically writes a JSON result file for each test case after it finishes. These files contain comprehensive metadata about the test run, including SDK and runner versions, infrastructure details, interaction statistics, and links to recordings.
+
+Result files are written to:
+
+```
+.testdriver/results/<testFile>/<testName>.json
+```
+
+For example, a test file `tests/login.test.mjs` with a test named `"should log in"` produces:
+
+```
+.testdriver/results/tests/login.test.mjs/should_log_in.json
+```
+
+<Note>
+  Test names are sanitized for filesystem use — special characters are replaced with underscores and names are truncated to 200 characters.
+</Note>
+
+## Enabling
+
+No configuration is required. The JSON files are written automatically by the TestDriver Vitest reporter plugin whenever tests run.
+
+## JSON Schema
+
+Each result file is organized into logical groups:
+
+### `versions`
+
+| Field | Type | Description |
+|---|---|---|
+| `versions.sdk` | `string \| null` | TestDriver SDK version (e.g. `"7.8.0"`) |
+| `versions.vitest` | `string \| null` | Vitest version used to run the test |
+| `versions.api` | `string \| null` | TestDriver API server version |
+| `versions.runnerBefore` | `string \| null` | Runner version at sandbox start |
+| `versions.runnerAfter` | `string \| null` | Runner version after auto-update |
+| `versions.runnerWasUpdated` | `boolean` | Whether the runner was auto-updated during provisioning |
+
+### `test`
+
+| Field | Type | Description |
+|---|---|---|
+| `test.file` | `string \| null` | Relative path to the test file |
+| `test.name` | `string \| null` | Name of the test case |
+| `test.suite` | `string \| null` | Name of the parent `describe` block |
+| `test.passed` | `boolean` | Whether the test passed |
+| `test.caseId` | `string \| null` | Database ID for this test case |
+| `test.runId` | `string \| null` | Database ID for the overall test run |
+| `test.error` | `string \| null` | Error message if the test failed |
+| `test.errorStack` | `string \| null` | Error stack trace if the test failed |
+
+### `urls`
+
+| Field | Type | Description |
+|---|---|---|
+| `urls.api` | `string \| null` | API root URL used for this test |
+| `urls.console` | `string \| null` | TestDriver console base URL |
+| `urls.vnc` | `string \| null` | VNC URL for the sandbox |
+| `urls.testRun` | `string \| null` | Direct link to this test case in the console |
+
+### `replay`
+
+The `replay` object contains the recording replay URL and derived embed links. The `gifUrl` and `embedUrl` are generated automatically from the replay URL.
+
+| Field | Type | Description |
+|---|---|---|
+| `replay.url` | `string \| null` | Recording replay URL |
+| `replay.gifUrl` | `string \| null` | Animated GIF thumbnail of the recording |
+| `replay.embedUrl` | `string \| null` | Embeddable replay URL (appends `&embed=true`) |
+| `replay.markdown` | `string \| null` | Ready-to-use Markdown embed with GIF linking to the replay |
+
+The `replay.markdown` field produces a clickable GIF badge you can paste directly into PR comments, README files, or issue descriptions:
+
+```markdown
+[![Test Recording](https://api.testdriver.ai/replay/abc123/gif?shareKey=xyz)](https://console.testdriver.ai/replay/abc123?share=xyz)
+```
+
+### `date`
+
+| Field | Type | Description |
+|---|---|---|
+| `date` | `string` | ISO 8601 timestamp when the test finished |
+
+### `team`
+
+| Field | Type | Description |
+|---|---|---|
+| `team.id` | `string \| null` | Team ID from the sandbox |
+| `team.sessionId` | `string \| null` | SDK session ID |
+
+### `infrastructure`
+
+| Field | Type | Description |
+|---|---|---|
+| `infrastructure.sandboxId` | `string \| null` | Sandbox instance ID |
+| `infrastructure.instanceId` | `string \| null` | Instance ID |
+| `infrastructure.os` | `string \| null` | Operating system of the sandbox (`"linux"` or `"windows"`) |
+| `infrastructure.amiId` | `string \| null` | AWS AMI ID used for provisioning |
+| `infrastructure.e2bTemplateId` | `string \| null` | E2B template ID used for provisioning |
+| `infrastructure.imageVersion` | `string \| null` | Sandbox image version |
+
+### `realtime`
+
+| Field | Type | Description |
+|---|---|---|
+| `realtime.channel` | `string \| null` | Ably channel name used for communication |
+| `realtime.messageCount` | `number` | Number of messages published to the realtime channel |
+
+### `interactions`
+
+| Field | Type | Description |
+|---|---|---|
+| `interactions.total` | `number` | Total number of interactions recorded |
+| `interactions.cached` | `number` | Number of interactions served from cache |
+| `interactions.byType` | `object` | Breakdown of interactions by type (e.g. `find`, `click`, `assert`) |
+
+## Example Output
+
+```json
+{
+  "sdkVersion": "7.8.0",
+  "vitestVersion": "4.0.0",
+  "apiVersion": "1.45.0",
+  "runnerVersionBefore": "2.1.0",
+  "runnerVersionAfter": "2.1.1",
+  "wasUpdated": true,
+  "apiUrl": "https://api.testdriver.ai",
+  "consoleUrl": "https://console.testdriver.ai",
+  "testRunLink": "https://console.testdriver.ai/runs/abc123/def456",
+  "dashcamUrl": "https://app.dashcam.io/replay/abc123",
+  "vncUrl": "wss://sandbox-123.testdriver.ai/vnc",
+  "date": "2025-01-15T14:30:00.000Z",
+  "team": {
+    "id": "team_abc123",
+    "sessionId": "sess_xyz789"
+  },
+  "infrastructure": {
+    "sandboxId": "sandbox-123",
+    "instanceId": "i-abc123",
+    "os": "linux",
+    "amiId": "ami-0abc123",
+    "e2bTemplateId": null,
+    "imageVersion": "v2.1.0"
+  },
+  "realtime": {
+    "channel": "sandbox:sandbox-123",
+    "messageCount": 42
+  },
+  "interactions": {
+    "total": 15,
+    "cached": 3,
+    "byType": {
+      "find": 8,
+      "click": 5,
+      "assert": 2
+    }
+  }
+}
+```
+
+## Using Result Files in CI
+
+Result files are useful for extracting test metadata in CI pipelines without parsing log output.
+
+### GitHub Actions Example
+
+Use `fromJSON` to parse a result file into a GitHub Actions expression you can reference in subsequent steps:
+
+```yaml
+- name: Run tests
+  run: npx vitest run tests/login.test.mjs
+
+- name: Parse result
+  id: result
+  run: |
+    # Read the first JSON result file
+    FILE=$(find .testdriver/results -name '*.json' | head -n 1)
+    echo "json=$(cat "$FILE")" >> "$GITHUB_OUTPUT"
+
+- name: Comment on PR
+  if: fromJSON(steps.result.outputs.json).test.passed == false
+  uses: actions/github-script@v7
+  with:
+    script: |
+      const result = ${{ steps.result.outputs.json }};
+      await github.rest.issues.createComment({
+        owner: context.repo.owner,
+        repo: context.repo.repo,
+        issue_number: context.issue.number,
+        body: [
+          `❌ **${result.test.name}** failed`,
+          ``,
+          `Error: ${result.test.error}`,
+          ``,
+          result.replay.markdown,
+          ``,
+          `[View full recording](${result.urls.testRun})`
+        ].join('\n')
+      });
+```
+
+You can also load all results into a matrix or iterate over them:
+
+```yaml
+- name: Run tests
+  run: npx vitest run tests/*.test.mjs
+
+- name: Collect results
+  id: results
+  run: |
+    # Merge all result files into a JSON array
+    echo "json=$(find .testdriver/results -name '*.json' -exec cat {} + | jq -s '.')" >> "$GITHUB_OUTPUT"
+
+- name: Summary
+  run: |
+    echo '## Test Results' >> $GITHUB_STEP_SUMMARY
+    RESULTS='${{ steps.results.outputs.json }}'
+    echo "$RESULTS" | jq -r '.[] | "| \(.test.name) | \(if .test.passed then "✅" else "❌" end) | \(.urls.testRun) |"' >> $GITHUB_STEP_SUMMARY
+```
+
+### Reading Results Programmatically
+
+```javascript
+import fs from "fs";
+import path from "path";
+
+const resultsDir = ".testdriver/results";
+
+function readResults(dir) {
+  const results = [];
+  for (const testDir of fs.readdirSync(dir, { recursive: true })) {
+    const fullPath = path.join(dir, testDir);
+    if (fullPath.endsWith(".json") && fs.statSync(fullPath).isFile()) {
+      results.push(JSON.parse(fs.readFileSync(fullPath, "utf-8")));
+    }
+  }
+  return results;
+}
+
+const results = readResults(resultsDir);
+const passed = results.filter(r => r.test.passed);
+const failed = results.filter(r => !r.test.passed);
+
+console.log(`${passed.length} passed, ${failed.length} failed`);
+for (const r of failed) {
+  console.log(`  FAIL: ${r.test.name} — ${r.test.error}`);
+  console.log(`  Recording: ${r.urls.testRun}`);
+  console.log(`  Embed: ${r.replay.markdown}`);
+}
+```