testdriverai 7.2.21 → 7.2.23

package/docs/v7/_drafts/performance.mdx

@@ -1,517 +0,0 @@
----
-title: "Performance Optimization"
-description: "Speed up your TestDriver tests"
-icon: "gauge"
----
-
-## Overview
-
-TestDriver tests run in cloud sandboxes, which adds latency. This guide shows you how to minimize overhead and maximize test speed.
-
-## Baseline Performance
-
-Typical test execution:
-
-| Phase | Time | What's Happening |
-|-------|------|------------------|
-| Sandbox startup | 20-60s | First test creates VM |
-| Sandbox reuse | 0s | Subsequent tests reuse VM |
-| Page load | 1-5s | Browser navigates to URL |
-| Element finding | 0.5-3s | AI locates element |
-| Element finding (cached) | 0.1-0.5s | Cache hit |
-| Command execution | 0.1-1s | Click, type, etc. |
-| AI command | 2-5s | Natural language parsing |
-| AI command (cached) | 0.1-0.5s | Cache hit |
-
-**First test**: 60-90s  
-**Subsequent tests**: 5-30s  
-**Optimized tests**: 2-10s
-
-## Optimization Strategies
-
-### 1. Reuse Sandboxes
-
-The biggest performance win is reusing sandboxes across tests.
-
-#### ❌ Slow - Create new sandbox per test
-
-```javascript
-test('test 1', async () => {
-  const testdriver = await TestDriver.create({
-    apiKey: process.env.TD_API_KEY
-  });
-  // Test logic
-  await testdriver.cleanup();
-});
-
-test('test 2', async () => {
-  const testdriver = await TestDriver.create({
-    apiKey: process.env.TD_API_KEY
-  });
-  // Test logic
-  await testdriver.cleanup();
-});
-```
-
-**Time**: 60s + 60s = 120s
-
-#### ✅ Fast - Reuse sandbox with context
-
-```javascript
-import { chrome } from '../setup/lifecycleHelpers.mjs';
-
-test('test 1', async (context) => {
-  const { testdriver } = await chrome(context, { url: 'https://app.com' });
-  // Test logic
-});
-
-test('test 2', async (context) => {
-  const { testdriver } = await chrome(context, { url: 'https://app.com' });
-  // Test logic
-});
-```
-
-**Time**: 60s + 5s = 65s (54% faster)
-
-### 2. Enable Caching
-
-TestDriver has two cache layers:
-
-#### AI Prompt Cache
-
-Caches AI-generated commands locally:
-
-```yaml
-# testdriver.yaml
-cache:
-  ai: true  # Enable prompt caching
-```
-
-**Impact**: 2-5s → 0.1-0.5s per AI command
-
-#### Selector Cache
-
-Caches element locations on server:
-
-```yaml
-# testdriver.yaml
-cache:
-  selectors: true
-  selectorThreshold: 0.95  # 95% similarity required
-```
-
-**Impact**: 0.5-3s → 0.1-0.5s per element find
-
-**Combined**: Tests run 3-5x faster on subsequent runs.
-
-### 3. Parallel Test Execution
-
-Run multiple tests simultaneously:
-
-```javascript
-// vitest.config.mjs
-export default defineConfig({
-  test: {
-    maxConcurrency: 5,  // Run 5 tests at once
-    fileParallelism: true
-  }
-});
-```
-
-**Impact**: 5 tests in 300s → 5 tests in 90s (70% faster)
-
-**Tradeoffs**:
-- Uses more test minutes
-- May hit API rate limits
-- Requires adequate plan limits
-
-### 4. Reduce AI Commands
-
-AI commands are slower than direct commands.
-
-#### ❌ Slow - AI for everything
-
-```javascript
-await testdriver.ai('click the login button');
-await testdriver.ai('type email@example.com');
-await testdriver.ai('type mypassword');
-await testdriver.ai('press enter');
-```
-
-**Time**: ~15s
-
-#### ✅ Fast - AI only for finding
-
-```javascript
-await testdriver.find('email field').then(el => el.click());
-await testdriver.type('email@example.com');
-
-await testdriver.find('password field').then(el => el.click());
-await testdriver.type('mypassword');
-
-await testdriver.find('login button').then(el => el.click());
-```
-
-**Time**: ~5s (66% faster)
-
-### 5. Batch Operations
-
-Minimize round trips to sandbox:
-
-#### ❌ Slow - Sequential operations
-
-```javascript
-await testdriver.find('first name').then(el => el.click());
-await testdriver.type('John');
-
-await testdriver.find('last name').then(el => el.click());
-await testdriver.type('Doe');
-
-await testdriver.find('email').then(el => el.click());
-await testdriver.type('john@example.com');
-```
-
-**Time**: 6 round trips
-
-#### ✅ Fast - Batch with keyboard navigation
-
-```javascript
-await testdriver.find('first name').then(el => el.click());
-await testdriver.type('John');
-
-await testdriver.pressKeys(['tab']);
-await testdriver.type('Doe');
-
-await testdriver.pressKeys(['tab']);
-await testdriver.type('john@example.com');
-```
-
-**Time**: 4 round trips (33% faster)
-
-### 6. Smart Waiting
-
-Avoid unnecessary delays:
-
-#### ❌ Slow - Fixed delays
-
-```javascript
-await testdriver.find('button').then(el => el.click());
-await new Promise(r => setTimeout(r, 5000));  // Always wait 5s
-await testdriver.find('success message');
-```
-
-#### ✅ Fast - Poll until found
-
-```javascript
-await testdriver.find('button').then(el => el.click());
-
-// Poll for success message
-let element;
-for (let i = 0; i < 30; i++) {
-  element = await testdriver.find('success message');
-  if (element.found()) break;
-  await new Promise(r => setTimeout(r, 500));
-}
-```
-
-#### ✅ Better - Use assertions
-
-```javascript
-await testdriver.find('button').then(el => el.click());
-await testdriver.assert('success message appeared');  // Waits intelligently
-```
-
-### 7. Optimize Element Descriptions
-
-More specific = faster finding:
-
-#### ❌ Slow - Vague description
-
-```javascript
-await testdriver.find('button');  // Many buttons to check
-```
-
-**Time**: 2-3s (checks all buttons)
-
-#### ✅ Fast - Specific description
-
-```javascript
-await testdriver.find('blue submit button at bottom right');
-```
-
-**Time**: 0.5-1s (narrows search area)
-
-### 8. Preload Resources
-
-Speed up page load:
-
-#### ❌ Slow - Load on demand
-
-```javascript
-test('test 1', async (context) => {
-  const { testdriver } = await chrome(context, { url: 'https://app.com' });
-  // First test loads everything
-});
-```
-
-#### ✅ Fast - Preload in beforeAll
-
-```javascript
-import { beforeAll, test } from 'vitest';
-
-beforeAll(async (context) => {
-  const { testdriver } = await chrome(context, { url: 'https://app.com' });
-  // Preload app, login, navigate to test area
-  await testdriver.find('email').then(el => el.click());
-  await testdriver.type(process.env.TEST_EMAIL);
-  await testdriver.find('password').then(el => el.click());
-  await testdriver.type(process.env.TEST_PASSWORD);
-  await testdriver.find('login').then(el => el.click());
-}, 120000);
-```
-
-### 9. Clean Cache Strategically
-
-Don't clear cache unnecessarily:
-
-#### ❌ Slow - Clear cache every run
-
-```bash
-# package.json
-{
-  "scripts": {
-    "test": "rm -rf .testdriver/.cache && vitest"
-  }
-}
-```
-
-#### ✅ Fast - Clear cache when needed
-
-```bash
-# Only clear when UI changes
-npm run test:clean  # Separate script
-
-# Normal runs use cache
-npm test
-```
-
-### 10. Monitor Performance
-
-Track test execution time:
-
-```javascript
-import { test } from 'vitest';
-
-test('slow test detector', async (context) => {
-  const start = Date.now();
-  
-  // Test logic
-  const { testdriver } = await chrome(context, { url });
-  await testdriver.find('button').then(el => el.click());
-  
-  const duration = Date.now() - start;
-  console.log(`Test took ${duration}ms`);
-  
-  if (duration > 30000) {
-    console.warn('⚠️ Test exceeded 30s threshold');
-  }
-});
-```
-
-## Configuration Tuning
-
-### Timeout Settings
-
-Balance reliability vs speed:
-
-```javascript
-// vitest.config.mjs
-export default defineConfig({
-  test: {
-    testTimeout: 60000,  // Default: 60s
-    hookTimeout: 30000   // Setup/teardown: 30s
-  }
-});
-```
-
-For fast, stable tests:
-```javascript
-testTimeout: 30000  // 30s
-```
-
-For slower, complex tests:
-```javascript
-testTimeout: 120000  // 2 minutes
-```
-
-### Verbosity Level
-
-Lower verbosity = less overhead:
-
-```javascript
-const testdriver = await TestDriver.create({
-  apiKey: process.env.TD_API_KEY,
-  verbosity: 0  // 0=silent, 1=normal, 2=debug
-});
-```
-
-**Impact**: Small (~100ms per test), but adds up.
-
-### Resolution
-
-Lower resolution = faster screenshots:
-
-```javascript
-const testdriver = await TestDriver.create({
-  apiKey: process.env.TD_API_KEY,
-  resolution: '1280x720'  // Default: 1920x1080
-});
-```
-
-**Impact**: 10-20% faster element finding.
-
-## Advanced Techniques
-
-### Connection Pooling
-
-Reuse connections across test suites:
-
-```javascript
-// setup/pool.mjs
-const clients = new Map();
-
-export async function getClient(context) {
-  const key = `${context.task.file.name}`;
-  
-  if (!clients.has(key)) {
-    clients.set(key, await TestDriver.create({
-      apiKey: process.env.TD_API_KEY
-    }));
-  }
-  
-  return clients.get(key);
-}
-
-export async function cleanup() {
-  for (const client of clients.values()) {
-    await client.cleanup();
-  }
-  clients.clear();
-}
-```
-
-### Lazy Loading
-
-Only load what you need:
-
-```javascript
-// ❌ Load everything upfront
-import { chrome, firefox, electron } from './setup/lifecycleHelpers.mjs';
-
-// ✅ Import only what you use
-import { chrome } from './setup/lifecycleHelpers.mjs';
-```
-
-### Snapshot Testing
-
-Compare screenshots instead of re-running:
-
-```javascript
-test('visual regression', async (context) => {
-  const { testdriver } = await chrome(context, { url });
-  
-  const element = await testdriver.find('hero section');
-  
-  // First run: saves screenshot
-  // Subsequent runs: compares screenshot
-  expect(element.screenshot).toMatchImageSnapshot();
-});
-```
-
-## Profiling
-
-### Find Slow Tests
-
-```bash
-# Run with timing report
-npm test -- --reporter=verbose
-
-# Output shows duration per test
-✓ fast test (1.2s)
-✓ medium test (5.4s)
-✗ slow test (45.2s)
-```
-
-### Measure Operations
-
-```javascript
-async function timedOperation(name, fn) {
-  const start = Date.now();
-  const result = await fn();
-  console.log(`${name}: ${Date.now() - start}ms`);
-  return result;
-}
-
-test('profiled test', async (context) => {
-  const { testdriver } = await timedOperation('Setup', async () => {
-    return await chrome(context, { url });
-  });
-  
-  await timedOperation('Find button', async () => {
-    return await testdriver.find('button');
-  });
-  
-  await timedOperation('Click button', async () => {
-    const el = await testdriver.find('button');
-    return await el.click();
-  });
-});
-```
-
-### Identify Bottlenecks
-
-Common slow operations:
-
-1. **Sandbox creation** (20-60s) - Reuse sandboxes
-2. **Page navigation** (1-5s) - Minimize navigation
-3. **AI commands** (2-5s) - Use direct commands
-4. **Element finding** (0.5-3s) - Enable caching
-5. **Fixed delays** (varies) - Replace with smart waiting
-
-## Production Checklist
-
-- [ ] Reuse sandboxes via context
-- [ ] Enable AI and selector caching
-- [ ] Use parallel execution (maxConcurrency: 5)
-- [ ] Minimize AI commands
-- [ ] Batch operations
-- [ ] Use smart waiting (assertions)
-- [ ] Specific element descriptions
-- [ ] Appropriate timeout settings
-- [ ] Monitor test duration
-- [ ] Profile slow tests
-
-Expected performance:
-- **First run**: 60-90s for suite
-- **Cached runs**: 10-30s for suite
-- **Per test**: 2-10s (cached)
-
-## See Also
-
-<CardGroup cols={2}>
-  <Card title="Caching (AI)" icon="brain" href="/v7/guides/caching-ai">
-    AI prompt caching
-  </Card>
-  
-  <Card title="Caching (Selectors)" icon="bullseye" href="/v7/guides/caching-selectors">
-    Selector caching
-  </Card>
-  
-  <Card title="Best Practices" icon="star" href="/v7/guides/best-practices">
-    Testing patterns
-  </Card>
-  
-  <Card title="CI/CD Integration" icon="arrows-spin" href="/v7/guides/ci-cd">
-    Optimize CI pipelines
-  </Card>
-</CardGroup>