testdriverai 7.0.0 → 7.1.1

package/docs/v7/_drafts/performance.mdx

@@ -0,0 +1,517 @@
+---
+title: "Performance Optimization"
+description: "Speed up your TestDriver tests"
+icon: "gauge"
+---
+
+## Overview
+
+TestDriver tests run in cloud sandboxes, which adds latency. This guide shows you how to minimize overhead and maximize test speed.
+
+## Baseline Performance
+
+Typical test execution:
+
+| Phase | Time | What's Happening |
+|-------|------|------------------|
+| Sandbox startup | 20-60s | First test creates VM |
+| Sandbox reuse | 0s | Subsequent tests reuse VM |
+| Page load | 1-5s | Browser navigates to URL |
+| Element finding | 0.5-3s | AI locates element |
+| Element finding (cached) | 0.1-0.5s | Cache hit |
+| Command execution | 0.1-1s | Click, type, etc. |
+| AI command | 2-5s | Natural language parsing |
+| AI command (cached) | 0.1-0.5s | Cache hit |
+
+**First test**: 60-90s  
+**Subsequent tests**: 5-30s  
+**Optimized tests**: 2-10s
+
+## Optimization Strategies
+
+### 1. Reuse Sandboxes
+
+The biggest performance win is reusing sandboxes across tests.
+
+#### ❌ Slow - Create new sandbox per test
+
+```javascript
+test('test 1', async () => {
+  const testdriver = await TestDriver.create({
+    apiKey: process.env.TD_API_KEY
+  });
+  // Test logic
+  await testdriver.cleanup();
+});
+
+test('test 2', async () => {
+  const testdriver = await TestDriver.create({
+    apiKey: process.env.TD_API_KEY
+  });
+  // Test logic
+  await testdriver.cleanup();
+});
+```
+
+**Time**: 60s + 60s = 120s
+
+#### ✅ Fast - Reuse sandbox with context
+
+```javascript
+import { chrome } from '../setup/lifecycleHelpers.mjs';
+
+test('test 1', async (context) => {
+  const { testdriver } = await chrome(context, { url: 'https://app.com' });
+  // Test logic
+});
+
+test('test 2', async (context) => {
+  const { testdriver } = await chrome(context, { url: 'https://app.com' });
+  // Test logic
+});
+```
+
+**Time**: 60s + 5s = 65s (54% faster)
+
+### 2. Enable Caching
+
+TestDriver has two cache layers:
+
+#### AI Prompt Cache
+
+Caches AI-generated commands locally:
+
+```yaml
+# testdriver.yaml
+cache:
+  ai: true  # Enable prompt caching
+```
+
+**Impact**: 2-5s → 0.1-0.5s per AI command
+
+#### Selector Cache
+
+Caches element locations on server:
+
+```yaml
+# testdriver.yaml
+cache:
+  selectors: true
+  selectorThreshold: 0.95  # 95% similarity required
+```
+
+**Impact**: 0.5-3s → 0.1-0.5s per element find
+
+**Combined**: Tests run 3-5x faster on subsequent runs.
+
+### 3. Parallel Test Execution
+
+Run multiple tests simultaneously:
+
+```javascript
+// vitest.config.mjs
+export default defineConfig({
+  test: {
+    maxConcurrency: 5,  // Run 5 tests at once
+    fileParallelism: true
+  }
+});
+```
+
+**Impact**: 5 tests in 300s → 5 tests in 90s (70% faster)
+
+**Tradeoffs**:
+- Uses more test minutes
+- May hit API rate limits
+- Requires adequate plan limits
+
+### 4. Reduce AI Commands
+
+AI commands are slower than direct commands.
+
+#### ❌ Slow - AI for everything
+
+```javascript
+await testdriver.ai('click the login button');
+await testdriver.ai('type email@example.com');
+await testdriver.ai('type mypassword');
+await testdriver.ai('press enter');
+```
+
+**Time**: ~15s
+
+#### ✅ Fast - AI only for finding
+
+```javascript
+await testdriver.find('email field').then(el => el.click());
+await testdriver.type('email@example.com');
+
+await testdriver.find('password field').then(el => el.click());
+await testdriver.type('mypassword');
+
+await testdriver.find('login button').then(el => el.click());
+```
+
+**Time**: ~5s (66% faster)
+
+### 5. Batch Operations
+
+Minimize round trips to sandbox:
+
+#### ❌ Slow - Sequential operations
+
+```javascript
+await testdriver.find('first name').then(el => el.click());
+await testdriver.type('John');
+
+await testdriver.find('last name').then(el => el.click());
+await testdriver.type('Doe');
+
+await testdriver.find('email').then(el => el.click());
+await testdriver.type('john@example.com');
+```
+
+**Time**: 6 round trips
+
+#### ✅ Fast - Batch with keyboard navigation
+
+```javascript
+await testdriver.find('first name').then(el => el.click());
+await testdriver.type('John');
+
+await testdriver.pressKeys(['tab']);
+await testdriver.type('Doe');
+
+await testdriver.pressKeys(['tab']);
+await testdriver.type('john@example.com');
+```
+
+**Time**: 4 round trips (33% faster)
+
+### 6. Smart Waiting
+
+Avoid unnecessary delays:
+
+#### ❌ Slow - Fixed delays
+
+```javascript
+await testdriver.find('button').then(el => el.click());
+await new Promise(r => setTimeout(r, 5000));  // Always wait 5s
+await testdriver.find('success message');
+```
+
+#### ✅ Fast - Poll until found
+
+```javascript
+await testdriver.find('button').then(el => el.click());
+
+// Poll for success message
+let element;
+for (let i = 0; i < 30; i++) {
+  element = await testdriver.find('success message');
+  if (element.found()) break;
+  await new Promise(r => setTimeout(r, 500));
+}
+```
+
+#### ✅ Better - Use assertions
+
+```javascript
+await testdriver.find('button').then(el => el.click());
+await testdriver.assert('success message appeared');  // Waits intelligently
+```
+
+### 7. Optimize Element Descriptions
+
+More specific = faster finding:
+
+#### ❌ Slow - Vague description
+
+```javascript
+await testdriver.find('button');  // Many buttons to check
+```
+
+**Time**: 2-3s (checks all buttons)
+
+#### ✅ Fast - Specific description
+
+```javascript
+await testdriver.find('blue submit button at bottom right');
+```
+
+**Time**: 0.5-1s (narrows search area)
+
+### 8. Preload Resources
+
+Speed up page load:
+
+#### ❌ Slow - Load on demand
+
+```javascript
+test('test 1', async (context) => {
+  const { testdriver } = await chrome(context, { url: 'https://app.com' });
+  // First test loads everything
+});
+```
+
+#### ✅ Fast - Preload in beforeAll
+
+```javascript
+import { beforeAll, test } from 'vitest';
+
+beforeAll(async (context) => {
+  const { testdriver } = await chrome(context, { url: 'https://app.com' });
+  // Preload app, login, navigate to test area
+  await testdriver.find('email').then(el => el.click());
+  await testdriver.type(process.env.TEST_EMAIL);
+  await testdriver.find('password').then(el => el.click());
+  await testdriver.type(process.env.TEST_PASSWORD);
+  await testdriver.find('login').then(el => el.click());
+}, 120000);
+```
+
+### 9. Clean Cache Strategically
+
+Don't clear cache unnecessarily:
+
+#### ❌ Slow - Clear cache every run
+
+```bash
+# package.json
+{
+  "scripts": {
+    "test": "rm -rf .testdriver/.cache && vitest"
+  }
+}
+```
+
+#### ✅ Fast - Clear cache when needed
+
+```bash
+# Only clear when UI changes
+npm run test:clean  # Separate script
+
+# Normal runs use cache
+npm test
+```
+
+### 10. Monitor Performance
+
+Track test execution time:
+
+```javascript
+import { test } from 'vitest';
+
+test('slow test detector', async (context) => {
+  const start = Date.now();
+  
+  // Test logic
+  const { testdriver } = await chrome(context, { url });
+  await testdriver.find('button').then(el => el.click());
+  
+  const duration = Date.now() - start;
+  console.log(`Test took ${duration}ms`);
+  
+  if (duration > 30000) {
+    console.warn('⚠️ Test exceeded 30s threshold');
+  }
+});
+```
+
+## Configuration Tuning
+
+### Timeout Settings
+
+Balance reliability vs speed:
+
+```javascript
+// vitest.config.mjs
+export default defineConfig({
+  test: {
+    testTimeout: 60000,  // Default: 60s
+    hookTimeout: 30000   // Setup/teardown: 30s
+  }
+});
+```
+
+For fast, stable tests:
+```javascript
+testTimeout: 30000  // 30s
+```
+
+For slower, complex tests:
+```javascript
+testTimeout: 120000  // 2 minutes
+```
+
+### Verbosity Level
+
+Lower verbosity = less overhead:
+
+```javascript
+const testdriver = await TestDriver.create({
+  apiKey: process.env.TD_API_KEY,
+  verbosity: 0  // 0=silent, 1=normal, 2=debug
+});
+```
+
+**Impact**: Small (~100ms per test), but adds up.
+
+### Resolution
+
+Lower resolution = faster screenshots:
+
+```javascript
+const testdriver = await TestDriver.create({
+  apiKey: process.env.TD_API_KEY,
+  resolution: '1280x720'  // Default: 1920x1080
+});
+```
+
+**Impact**: 10-20% faster element finding.
+
+## Advanced Techniques
+
+### Connection Pooling
+
+Reuse connections across test suites:
+
+```javascript
+// setup/pool.mjs
+const clients = new Map();
+
+export async function getClient(context) {
+  const key = `${context.task.file.name}`;
+  
+  if (!clients.has(key)) {
+    clients.set(key, await TestDriver.create({
+      apiKey: process.env.TD_API_KEY
+    }));
+  }
+  
+  return clients.get(key);
+}
+
+export async function cleanup() {
+  for (const client of clients.values()) {
+    await client.cleanup();
+  }
+  clients.clear();
+}
+```
+
+### Lazy Loading
+
+Only load what you need:
+
+```javascript
+// ❌ Load everything upfront
+import { chrome, firefox, electron } from './setup/lifecycleHelpers.mjs';
+
+// ✅ Import only what you use
+import { chrome } from './setup/lifecycleHelpers.mjs';
+```
+
+### Snapshot Testing
+
+Compare screenshots instead of re-running:
+
+```javascript
+test('visual regression', async (context) => {
+  const { testdriver } = await chrome(context, { url });
+  
+  const element = await testdriver.find('hero section');
+  
+  // First run: saves screenshot
+  // Subsequent runs: compares screenshot
+  expect(element.screenshot).toMatchImageSnapshot();
+});
+```
+
+## Profiling
+
+### Find Slow Tests
+
+```bash
+# Run with timing report
+npm test -- --reporter=verbose
+
+# Output shows duration per test
+✓ fast test (1.2s)
+✓ medium test (5.4s)
+✗ slow test (45.2s)
+```
+
+### Measure Operations
+
+```javascript
+async function timedOperation(name, fn) {
+  const start = Date.now();
+  const result = await fn();
+  console.log(`${name}: ${Date.now() - start}ms`);
+  return result;
+}
+
+test('profiled test', async (context) => {
+  const { testdriver } = await timedOperation('Setup', async () => {
+    return await chrome(context, { url });
+  });
+  
+  await timedOperation('Find button', async () => {
+    return await testdriver.find('button');
+  });
+  
+  await timedOperation('Click button', async () => {
+    const el = await testdriver.find('button');
+    return await el.click();
+  });
+});
+```
+
+### Identify Bottlenecks
+
+Common slow operations:
+
+1. **Sandbox creation** (20-60s) - Reuse sandboxes
+2. **Page navigation** (1-5s) - Minimize navigation
+3. **AI commands** (2-5s) - Use direct commands
+4. **Element finding** (0.5-3s) - Enable caching
+5. **Fixed delays** (varies) - Replace with smart waiting
+
+## Production Checklist
+
+- [ ] Reuse sandboxes via context
+- [ ] Enable AI and selector caching
+- [ ] Use parallel execution (maxConcurrency: 5)
+- [ ] Minimize AI commands
+- [ ] Batch operations
+- [ ] Use smart waiting (assertions)
+- [ ] Specific element descriptions
+- [ ] Appropriate timeout settings
+- [ ] Monitor test duration
+- [ ] Profile slow tests
+
+Expected performance:
+- **First run**: 60-90s for suite
+- **Cached runs**: 10-30s for suite
+- **Per test**: 2-10s (cached)
+
+## See Also
+
+<CardGroup cols={2}>
+  <Card title="Caching (AI)" icon="brain" href="/v7/guides/caching-ai">
+    AI prompt caching
+  </Card>
+  
+  <Card title="Caching (Selectors)" icon="bullseye" href="/v7/guides/caching-selectors">
+    Selector caching
+  </Card>
+  
+  <Card title="Best Practices" icon="star" href="/v7/guides/best-practices">
+    Testing patterns
+  </Card>
+  
+  <Card title="CI/CD Integration" icon="arrows-spin" href="/v7/guides/ci-cd">
+    Optimize CI pipelines
+  </Card>
+</CardGroup>

package/docs/v7/_drafts/presets.mdx

@@ -0,0 +1,210 @@
+# TestDriver Presets
+
+Presets provide pre-configured setups for common applications, reducing boilerplate and making your tests easier to write.
+
+## Available Presets
+
+### chromePreset
+
+Automatically sets up Chrome browser with TestDriver and Dashcam.
+
+```javascript
+import { test } from 'vitest';
+import { chromePreset } from 'testdriverai/presets';
+
+test('my test', async (context) => {
+  const { client } = await chromePreset(context, {
+    url: 'https://myapp.com/login'
+  });
+  
+  await client.find('email input').type('user@example.com');
+  await client.find('Login button').click();
+});
+```
+
+**Options:**
+- `url` - URL to navigate to (default: 'http://testdriver-sandbox.vercel.app/')
+- `os` - Target OS: 'linux', 'mac', 'windows' (default: 'linux')
+- `dashcam` - Enable Dashcam recording (default: true)
+- `maximized` - Start maximized (default: true)
+- `guest` - Use guest mode (default: true)
+
+**Returns:**
+- `client` - TestDriver instance
+- `dashcam` - Dashcam instance (if enabled)
+
+### vscodePreset
+
+Automatically sets up VS Code with TestDriver and Dashcam.
+
+```javascript
+import { vscodePreset } from 'testdriverai/presets';
+
+test('extension test', async (context) => {
+  const { vscode } = await vscodePreset(context, {
+    workspace: '/tmp/test-project',
+    extensions: ['ms-python.python']
+  });
+  
+  await vscode.find('File menu').click();
+  await vscode.find('New File').click();
+});
+```
+
+**Options:**
+- `workspace` - Workspace/folder to open
+- `os` - Target OS (default: 'linux')
+- `dashcam` - Enable Dashcam recording (default: true)
+- `extensions` - Array of extension IDs to install
+
+**Returns:**
+- `client` - TestDriver instance
+- `vscode` - Alias for client
+- `dashcam` - Dashcam instance (if enabled)
+
+### electronPreset
+
+Automatically sets up an Electron application.
+
+```javascript
+import { electronPreset } from 'testdriverai/presets';
+
+test('electron app test', async (context) => {
+  const { app } = await electronPreset(context, {
+    appPath: '/path/to/electron/app',
+    args: ['--enable-logging']
+  });
+  
+  await app.find('main window').click();
+});
+```
+
+**Options:**
+- `appPath` - Path to Electron app (required)
+- `os` - Target OS (default: 'linux')
+- `dashcam` - Enable Dashcam recording (default: true)
+- `args` - Additional electron arguments
+
+**Returns:**
+- `client` - TestDriver instance
+- `app` - Alias for client
+- `dashcam` - Dashcam instance (if enabled)
+
+### webAppPreset
+
+Generic web application preset (currently uses Chrome).
+
+```javascript
+import { webAppPreset } from 'testdriverai/presets';
+
+test('web app test', async (context) => {
+  const { client } = await webAppPreset(context, {
+    url: 'https://example.com',
+    browser: 'chrome' // Only Chrome supported currently
+  });
+  
+  await client.find('login form').click();
+});
+```
+
+## Creating Custom Presets
+
+Use `createPreset` to build your own presets:
+
+```javascript
+import { createPreset } from 'testdriverai/presets';
+
+const firefoxPreset = createPreset({
+  name: 'Firefox Browser',
+  defaults: { os: 'linux', dashcam: true },
+  async setup(context, client, dashcam, options) {
+    const { url } = options;
+    
+    // Launch Firefox
+    await client.exec('sh', `firefox "${url}" >/dev/null 2>&1 &`, 30000);
+    await client.focusApplication('Firefox');
+    
+    return {
+      // client is already included automatically
+    };
+  },
+});
+
+// Use your custom preset
+test('my test', async (context) => {
+  const { client } = await firefoxPreset(context, {
+    url: 'https://example.com',
+  });
+  
+  await client.find('page content').click();
+});
+```
+
+### createPreset API
+
+```javascript
+createPreset({
+  name: string,           // Preset name (for errors)
+  defaults: object,       // Default options
+  setup: async function   // Setup function
+})
+```
+
+The `setup` function receives:
+- `context` - Vitest test context
+- `client` - TestDriver instance (already connected)
+- `dashcam` - Dashcam instance (if enabled)
+- `options` - Merged defaults + user options
+
+The `setup` function should return an object with any custom properties. The returned object will automatically include `client` and `dashcam`.
+
+## How Presets Work
+
+Presets automatically:
+
+1. **Create TestDriver client** - Uses `useTestDriver` hook
+2. **Connect to sandbox** - Authenticates and connects
+3. **Set up Dashcam** - If enabled (default: true)
+4. **Configure application** - Launch and focus the app
+5. **Handle cleanup** - Automatic disconnect and dashcam stop
+
+All lifecycle is managed automatically via Vitest hooks.
+
+## Progressive Disclosure
+
+Presets fit into the progressive disclosure pattern:
+
+### Beginner (Presets)
+```javascript
+const { client } = await chromePreset(context, { url: 'https://example.com' });
+```
+Everything automatic - just pass URL and start testing.
+
+### Intermediate (Hooks)
+```javascript
+const client = useTestDriver(context, { os: 'linux' });
+const dashcam = useDashcam(context, client, { autoStart: true });
+// Custom setup code
+```
+More control over lifecycle, still automatic cleanup.
+
+### Advanced (Direct)
+```javascript
+const client = new TestDriver(apiKey, { os: 'linux' });
+await client.auth();
+await client.connect();
+// Full manual control
+```
+Complete control, manual everything.
+
+## Best Practices
+
+1. **Use presets for common scenarios** - Chrome, VS Code, Electron
+2. **Create custom presets for your apps** - Encapsulate setup logic
+3. **Enable dashcam by default** - Great for debugging failures
+4. **Keep presets focused** - One app/scenario per preset
+5. **Use descriptive variable names** - `client`, `vscode`, `app` based on what you're testing
+
+## Examples
+
+See `testdriver/acceptance-sdk/presets-example.test.mjs` for working examples.