testdriverai 6.2.2 → 7.0.0

package/PLUGIN_MIGRATION.md

@@ -0,0 +1,222 @@
+# Vitest Reporter → Plugin Migration
+
+## Summary
+
+Converted the Vitest reporter to a plugin architecture with much simpler dashcam URL tracking.
+
+## Key Changes
+
+### 1. Plugin Architecture ✅
+
+**Before (Reporter):**
+
+```javascript
+class TestDriverReporter {
+  constructor(options = {}) {
+    this.options = options;
+    this.testRun = null;
+    // ... many instance properties
+  }
+}
+```
+
+**After (Plugin):**
+
+```javascript
+const pluginState = {
+  testRun: null,
+  testRunId: null,
+  dashcamUrls: new Map(), // In-memory tracking!
+  // ... all state in one place
+};
+
+export default function testDriverPlugin(options = {}) {
+  return {
+    name: "testdriver-plugin",
+    // ... hooks
+  };
+}
+```
+
+### 2. Dashcam URL Tracking (MUCH SIMPLER) ✅
+
+**Before:**
+
+- ❌ Write JSON to temp files in `/tmp/`
+- ❌ Read temp files on test completion
+- ❌ Complex matching logic (sessionId, taskId, filename patterns)
+- ❌ Manual cleanup of temp files
+- ❌ Race conditions with parallel tests
+- ❌ Multiple fallback mechanisms (globalThis, registry, meta)
+
+**After:**
+
+- ✅ Simple in-memory Map (testId → dashcamUrl)
+- ✅ Direct API: `registerDashcamUrl(testId, url, platform)`
+- ✅ No file system operations
+- ✅ No complex matching
+- ✅ No cleanup needed
+- ✅ Thread-safe for parallel tests
+
+### 3. Usage in Tests
+
+**Before:**
+
+```javascript
+// Tests had to write to temp files
+const tempFile = path.join(
+  os.tmpdir(),
+  `testdriver-dashcam-${sessionId}-${taskId}-${Date.now()}.json`,
+);
+fs.writeFileSync(
+  tempFile,
+  JSON.stringify({
+    dashcamUrl,
+    replayObjectId,
+    platform: client.os,
+    timestamp: Date.now(),
+    pid: process.pid,
+    sessionId,
+  }),
+);
+
+// AND set global variables
+globalThis.__testdriverMeta.__lastDashcamUrl__ = dashcamUrl;
+
+// AND register in registry
+globalThis.__testdriverRegistry.setClient(taskId, client);
+
+// AND store in task.meta
+task.meta.testdriverDashcamUrl = dashcamUrl;
+```
+
+**After:**
+
+```javascript
+// Just one simple call
+globalThis.__testdriverPlugin.registerDashcamUrl(taskId, dashcamUrl, client.os);
+```
+
+### 4. Configuration
+
+**Before:**
+
+```javascript
+export default defineConfig({
+  test: {
+    reporters: [
+      "default",
+      [
+        "./interfaces/vitest-reporter.js",
+        {
+          apiKey: process.env.TD_API_KEY,
+          apiRoot: process.env.TD_API_ROOT,
+        },
+      ],
+    ],
+  },
+});
+```
+
+**After:**
+
+```javascript
+export default defineConfig({
+  plugins: [
+    testDriverPlugin({
+      apiKey: process.env.TD_API_KEY,
+      apiRoot: process.env.TD_API_ROOT,
+    }),
+  ],
+});
+```
+
+## Files Modified
+
+### Plugin
+
+- ✅ **Created**: `interfaces/vitest-plugin.mjs` (new plugin)
+- ✅ **Deleted**: `interfaces/vitest-reporter.js` (old reporter)
+- ✅ **Updated**: `vitest.config.mjs` (uses plugin)
+- ✅ **Updated**: `vitest.config.example.js` (uses plugin)
+
+### Test Helpers
+
+- ✅ **Updated**: `testdriver/acceptance-sdk/setup/testHelpers.mjs`
+  - Removed temp file writing
+  - Removed `__testdriverRegistry` initialization
+  - Removed `__testdriverMeta` initialization
+  - Uses `globalThis.__testdriverPlugin.registerDashcamUrl()` instead
+
+### Documentation
+
+- ✅ **Updated**: `docs/QUICK_START_TEST_RECORDING.md`
+- ✅ **Updated**: `docs/TEST_RECORDING.md`
+- ✅ **Updated**: `docs/ARCHITECTURE.md`
+
+## Code Removed
+
+### From Plugin (vs old reporter)
+
+- ❌ ~200 lines of file system operations
+- ❌ `indexDashcamFiles()` - complex temp file reading
+- ❌ `findReplayForTest()` - complex sessionId matching
+- ❌ `findReplayForTestInline()` - complex fallback logic
+- ❌ Imports: `fs`, `os` (no longer needed!)
+
+### From Test Helpers
+
+- ❌ ~60 lines of temp file writing
+- ❌ Global registry setup
+- ❌ Global meta initialization
+- ❌ Multiple storage mechanisms
+
+## Benefits
+
+1. **Simpler**: ~300 fewer lines of complex code
+2. **Faster**: No file I/O overhead
+3. **More Reliable**: No file system race conditions
+4. **Better DX**: Single API call to register dashcam URLs
+5. **Easier to Debug**: All state in one place
+6. **Thread-Safe**: Map-based storage handles parallel tests
+7. **Cleaner**: No temp file cleanup needed
+
+## Testing
+
+```bash
+# Verify plugin loads
+node -e "import('./interfaces/vitest-plugin.mjs').then(m => console.log('Exports:', Object.keys(m)))"
+
+# Run tests
+TD_API_KEY=xxx npx vitest run
+```
+
+## Migration for Users
+
+If you're using the TestDriver SDK in your tests:
+
+**Old way (still works but deprecated):**
+
+```javascript
+// Multiple places to set dashcam URL
+globalThis.__testdriverMeta.__lastDashcamUrl__ = dashcamUrl;
+```
+
+**New way:**
+
+```javascript
+// Single plugin API
+if (globalThis.__testdriverPlugin) {
+  globalThis.__testdriverPlugin.registerDashcamUrl(
+    testId,
+    dashcamUrl,
+    platform,
+  );
+}
+```
+
+## Notes
+
+- The plugin automatically makes itself available globally as `globalThis.__testdriverPlugin`
+- Test helpers now use this API automatically
+- No breaking changes for end users - just better internals!

package/PROMPT_CACHE.md

@@ -0,0 +1,200 @@
+# Prompt Cache System
+
+## Overview
+
+The TestDriver SDK includes automatic caching of `.ai()` responses. When you use the SDK to execute natural language prompts, the YAML commands generated by the AI are cached locally to improve performance and reduce API calls.
+
+**Caching is enabled by default** and can be controlled per-call using the `cache` parameter.
+
+## How It Works
+
+1. **First Call**: When you call `.ai('click the submit button')` for the first time:
+
+   - The SDK sends the prompt and screenshot to the API
+   - The API returns YAML commands in markdown format
+   - The response is cached to `.testdriver/.cache/{prompt-hash}.yaml`
+
+2. **Subsequent Calls**: When you call `.ai('click the submit button')` again:
+
+   - The SDK checks the cache first
+   - If found, it uses the cached YAML instead of calling the API
+   - You'll see `(using cached response)` in the output
+
+3. **Disable Caching**: Pass `false` as the second parameter:
+   - `.ai('click the submit button', false)` bypasses the cache
+   - Forces a fresh API call and updates the cache
+
+## Cache Location
+
+Cached responses are stored in:
+
+```
+.testdriver/.cache/
+```
+
+Files are named using a combination of:
+
+- Sanitized prompt (first 50 chars, alphanumeric only)
+- MD5 hash of the full prompt for uniqueness
+
+Example:
+
+```
+.testdriver/.cache/click-the-submit-button-a1b2c3d4e5f6.yaml
+```
+
+## Benefits
+
+- **Faster execution**: No API call needed for repeated prompts
+- **Cost savings**: Reduces API usage for repeated operations
+- **Offline testing**: Can rerun tests without network access
+- **Deterministic tests**: Same prompt always produces same commands
+
+## Disabling Cache
+
+To disable caching for a specific `.ai()` call, pass `false` as the second parameter:
+
+```javascript
+// Force fresh API call, bypass cache
+await client.ai("click the submit button", false);
+```
+
+Caching is **enabled by default** (equivalent to passing `true`):
+
+```javascript
+// These are equivalent
+await client.ai("click the submit button");
+await client.ai("click the submit button", true);
+```
+
+## Clearing Cache
+
+To clear all cached prompts:
+
+```bash
+rm -rf .testdriver/.cache/*.yaml
+```
+
+Or programmatically:
+
+```javascript
+const promptCache = require("testdriverai/agent/lib/cache.js");
+promptCache.clearCache();
+```
+
+## Usage Examples
+
+### Basic Usage (Automatic Caching)
+
+```javascript
+const TestDriver = require("testdriverai");
+
+const client = new TestDriver(process.env.TD_API_KEY);
+await client.connect();
+
+// First call - makes API request and caches
+await client.ai("click the login button");
+
+// Second call - uses cache (instant)
+await client.ai("click the login button");
+
+// Force fresh API call, bypass cache
+await client.ai("click the login button", false);
+```
+
+### Check Cache Status
+
+```javascript
+const promptCache = require("testdriverai/agent/lib/cache.js");
+
+// Check if a prompt is cached
+if (promptCache.hasCache("click the login button")) {
+  console.log("This prompt is cached");
+}
+
+// Get cache statistics
+const stats = promptCache.getCacheStats();
+console.log(`Cached prompts: ${stats.count}`);
+console.log(`Files: ${stats.files}`);
+```
+
+### Read Cached YAML
+
+```javascript
+const promptCache = require("testdriverai/agent/lib/cache.js");
+
+const yaml = promptCache.readCache("click the login button");
+if (yaml) {
+  console.log("Cached YAML:", yaml);
+}
+```
+
+### Manual Cache Control
+
+```javascript
+const promptCache = require("testdriverai/agent/lib/cache.js");
+
+// Write to cache manually
+promptCache.writeCache("custom prompt", yamlContent);
+
+// Clear entire cache
+promptCache.clearCache();
+
+// Get cache path for a prompt
+const path = promptCache.getCachePath("my prompt");
+console.log(`Cache file: ${path}`);
+```
+
+## Important Notes
+
+1. **Prompt Matching**: Cache keys are based on the exact prompt text (case-insensitive, trimmed)
+
+   - `'Click the button'` and `'click the button'` will match
+   - `'Click the button'` and `'Click the button '` will match (trailing space trimmed)
+   - `'Click button'` and `'Click the button'` will NOT match
+
+2. **No Screenshot Comparison**: The cache does NOT compare screenshots
+
+   - Cached responses assume the UI state is similar
+   - Use different prompts if the UI state has changed significantly
+
+3. **Manual Cache Management**: The cache never expires automatically
+
+   - Clear it periodically if needed
+   - Delete specific files if a prompt's behavior should change
+
+4. **Version Compatibility**: Cache files are plain YAML
+   - Compatible across TestDriver versions
+   - Safe to commit to version control (though not recommended)
+
+## Testing
+
+Run the test suite to verify caching:
+
+```bash
+node test-prompt-cache.js
+```
+
+This will:
+
+- Make an initial `.ai()` call that caches the response
+- Make a second `.ai()` call that uses the cache
+- Show cache statistics and file locations
+
+## Troubleshooting
+
+### Cache not being used
+
+- Check that prompts match exactly (case-insensitive)
+- Verify `TD_NO_PROMPT_CACHE` is not set
+- Check `.testdriver/.cache/` directory exists and is writable
+
+### Stale cache data
+
+- Clear the cache directory
+- Or delete specific cached prompt files
+
+### Permission errors
+
+- Ensure `.testdriver/.cache/` is writable
+- Check file system permissions

package/SDK_LOGGING.md

@@ -0,0 +1,222 @@
+# TestDriver SDK - Formatted Logging for Dashcam
+
+The TestDriver SDK now includes clean, structured logging that makes logs easy to read when replayed in Dashcam.
+
+## Features
+
+✨ **Clean, structured formatting** with clear labels and timestamps  
+⏱️ **Timestamp tracking** showing elapsed time from test start  
+🎯 **Action-specific formatting** for find, click, type, hover, scroll, assert  
+⚡ **Cache indicators** showing when elements are found from cache  
+📊 **Test context integration** with Vitest test information  
+🎥 **Dashcam-optimized** - no emojis or ANSI codes, pure text format
+
+## How It Works
+
+All logs sent through the `log:log` event are automatically formatted before being sent to Dashcam. This means when you replay your test in Dashcam, you'll see beautiful, easy-to-read logs with:
+
+- **Clear prefixes** like `[FIND]`, `[CLICK]`, `[ASSERT]` for different action types
+- **Highlighted information** for element descriptions and coordinates
+- **Elapsed timestamps** from test start like `[30.59s]`
+- **Cache hit indicators** showing performance optimizations with `(cached)`
+- **Duration information** for operations
+
+## Usage
+
+### Basic Setup
+
+The formatter is automatically integrated into the SDK. Just use the SDK normally:
+
+```javascript
+import { afterAll, beforeAll, describe, it } from "vitest";
+import {
+  createTestClient,
+  setupTest,
+  teardownTest,
+} from "./setup/testHelpers.mjs";
+
+describe("My Test", () => {
+  let testdriver;
+
+  beforeAll(async () => {
+    testdriver = createTestClient();
+    await setupTest(testdriver);
+  });
+
+  afterAll(async () => {
+    await teardownTest(testdriver);
+  });
+
+  it("should have nice logs", async () => {
+    // Logs are automatically formatted!
+    const button = await testdriver.find("Submit button");
+    await button.click();
+  });
+});
+```
+
+### Enhanced Logging with Test Context
+
+For even better logging with timestamps, set the test context:
+
+```javascript
+beforeAll(async () => {
+  testdriver = createTestClient();
+
+  // Set test context for enhanced logging
+  testdriver.setTestContext({
+    file: "my-test.spec.mjs",
+    test: "My Test Suite",
+    startTime: Date.now(),
+  });
+
+  await setupTest(testdriver);
+});
+```
+
+This enables elapsed time display like `[30.59s]` in your logs.
+
+## Log Format Examples
+
+### Element Found
+
+```
+[30.59s] [FIND] Found "Submit button" at (682, 478) 1597ms (cached)
+```
+
+### Click Action
+
+```
+[35.43s] [CLICK] Click "Submit button"
+```
+
+### Hover Action
+
+```
+[12.15s] [HOVER] Hover "Menu item"
+```
+
+### Type Action
+
+```
+[8.32s] [TYPE] Type "user@example.com"
+```
+
+### Assertion
+
+```
+[42.10s] [ASSERT] "form submission successful" PASSED
+```
+
+### Error
+
+```
+[15.23s] [FAIL] Failed to save debug image - Error: ENOENT: no such file or directory
+```
+
+## Formatter API
+
+The formatter is available through the `sdk-log-formatter.js` module:
+
+```javascript
+const { formatter } = require("./sdk-log-formatter");
+
+// Format different types of messages
+formatter.formatElementFound("Button", {
+  x: 100,
+  y: 200,
+  duration: "1500ms",
+  cacheHit: true,
+});
+formatter.formatAction("click", "Submit button");
+formatter.formatAssertion("form is visible", true);
+formatter.formatError("Connection failed", error);
+```
+
+### Available Methods
+
+- `formatElementFound(description, meta)` - Format element discovery
+- `formatAction(action, description, meta)` - Format user actions
+- `formatAssertion(assertion, passed, meta)` - Format test assertions
+- `formatError(message, error)` - Format error messages
+- `formatHeader(title)` - Create section headers
+- `formatSummary(stats)` - Format test summaries
+- `setTestContext(context)` - Update test context for timing
+
+## Dashcam Compatibility
+
+The formatter is designed specifically for Dashcam replay compatibility:
+
+- **No emojis** - Uses text labels like `[FIND]`, `[CLICK]` instead of icons
+- **No ANSI colors** - Plain text formatting that displays correctly in all environments
+- **No special characters** - Simple ASCII characters only
+- **Clean structure** - Easy to read in logs without terminal formatting
+
+## Integration with Dashcam
+
+When you run tests with Dashcam recording:
+
+1. The SDK sends formatted logs to the `log:log` event
+2. These logs are forwarded to the sandbox via `_forwardLogToSandbox()`
+3. Dashcam captures and stores these logs with precise timestamps
+4. When you replay in Dashcam, you see beautifully formatted logs synchronized with the video
+
+### Example Workflow
+
+```bash
+# Run tests with Dashcam
+npx vitest run testdriver/acceptance-sdk/formatted-logging.test.mjs
+
+# View the replay with formatted logs
+# Open the Dashcam URL from the test output
+```
+
+## Customization
+
+### Custom Action Types
+
+Add custom action types to the formatter:
+
+```javascript
+// In sdk-log-formatter.js, add to getPrefix():
+const prefixes = {
+  // ...existing prefixes
+  myAction: "[CUSTOM]",
+};
+```
+
+### Custom Message Formatting
+
+Override the `formatMessage` method for custom text highlighting:
+
+```javascript
+formatMessage(type, message) {
+  // Add custom highlighting
+  message = message.replace(/\[custom\]/g, chalk.green('[custom]'));
+  return super.formatMessage(type, message);
+}
+```
+
+## Examples
+
+See `testdriver/acceptance-sdk/formatted-logging.test.mjs` for a complete example.
+
+## Benefits for Dashcam Replay
+
+- **Better debugging**: Quickly identify what happened at each step
+- **Professional appearance**: Share polished test recordings with stakeholders
+- **Faster analysis**: Labeled actions make it easy to scan logs
+- **Context awareness**: Timestamps help correlate logs with video timeline
+- **Performance insights**: Cache indicators show optimization opportunities
+- **Universal compatibility**: Works in any environment without terminal support
+
+## Technical Details
+
+The formatter uses:
+
+- **Plain text formatting** for universal compatibility
+- **Event emitters** to intercept log events
+- **Base64 encoding** for safe transmission to sandbox
+- **Test context** from Vitest for timing information
+
+Logs are sent through the existing `log:log` event system, ensuring compatibility with all existing TestDriver infrastructure.