testdriverai 7.8.0-test.7 → 7.8.0-test.8

package/ai/skills/testdriver-cache/SKILL.md

@@ -0,0 +1,221 @@
+---
+name: testdriver:cache
+description: Speed up tests with screenshot-based caching
+---
+<!-- Generated from cache.mdx. DO NOT EDIT. -->
+
+## Overview
+
+The cache system speeds up repeated test runs by comparing screenshots to cached results. When the screen hasn't changed significantly, cached element positions are reused instead of making an AI call.
+
+Cache works at two levels:
+- **Screen cache** — pixel diff comparison between the current screenshot and the cached screenshot
+- **Element cache** — OpenCV template matching to verify the cached element position is still correct
+
+## How It Works
+
+1. On `find()`, the SDK sends the current screenshot and cache metadata to the API
+2. The API compares the screenshot against previously cached results for the same `cacheKey`
+3. If the screen pixel diff is within the `screen` threshold AND the element template match exceeds the `element` threshold, the cached position is returned
+4. Otherwise, a new AI call is made and the result is cached
+
+```mermaid
+flowchart LR
+    A[Screenshot + cacheKey] --> B{Screen Diff\n< threshold?}
+    B -- Yes --> C{Template\nMatch OK?}
+    C -- Yes --> D[Return cached position]
+    B -- No --> E[AI Call - fresh]
+    C -- No --> F[AI Call - fresh]
+```
+
+## Configuration
+
+### Constructor Options
+
+```javascript
+const testdriver = new TestDriver({
+  cache: {
+    enabled: true,
+    thresholds: {
+      find: {
+        screen: 0.05,     // 5% pixel diff allowed (default)
+        element: 0.8,     // 80% OpenCV correlation required (default)
+      },
+      assert: 0.05,       // 5% pixel diff for assertions (default)
+    },
+  },
+  cacheKey: 'my-custom-key',   // overrides auto-generated key
+});
+```
+
+<ParamField path="cache" type="CacheConfig | false">
+  Cache configuration object, or `false` to disable entirely.
+
+  <Expandable title="properties">
+    <ParamField path="enabled" type="boolean" default={true}>
+      Enable or disable the cache system. Requires a valid `cacheKey` to actually activate.
+    </ParamField>
+    
+    <ParamField path="thresholds" type="CacheThresholds">
+      Threshold configuration for different command types.
+
+      <Expandable title="properties">
+        <ParamField path="find" type="FindCacheThresholds">
+          Thresholds for `find()` and `findAll()`.
+
+          <Expandable title="properties">
+            <ParamField path="screen" type="number" default={0.05}>
+              Maximum pixel diff percentage allowed between the current screenshot and the cached screenshot. Lower values require a closer match. Range: `0` to `1`.
+            </ParamField>
+            
+            <ParamField path="element" type="number" default={0.8}>
+              Minimum OpenCV template matching correlation required for the cached element crop. Higher values require a closer match. Range: `0` to `1`. Only used for `find()`, not `findAll()`.
+            </ParamField>
+          </Expandable>
+        </ParamField>
+        
+        <ParamField path="assert" type="number" default={0.05}>
+          Maximum pixel diff allowed for assertion cache hits.
+        </ParamField>
+      </Expandable>
+    </ParamField>
+  </Expandable>
+</ParamField>
+
+<ParamField path="cacheKey" type="string">
+  Unique key for cache lookups. If not provided, an auto-generated key is created from a SHA-256 hash of the calling test file (first 16 hex characters). The cache key changes automatically when your test file changes, providing automatic cache invalidation.
+</ParamField>
+
+### Disabling Cache
+
+```javascript
+// Via constructor
+const testdriver = new TestDriver({ cache: false });
+
+// Via environment variable
+// TD_NO_CACHE=true npx vitest run
+```
+
+When cache is disabled, all thresholds are set to `-1` internally, causing the API to skip cache lookups.
+
+### Per-Command Overrides
+
+Override cache thresholds for individual commands:
+
+```javascript
+// Stricter screen matching for this specific find
+const el = await testdriver.find('submit button', {
+  cache: {
+    thresholds: { screen: 0.01, element: 0.95 },
+  },
+});
+
+// Custom cache key for a specific assertion
+await testdriver.assert('dashboard loaded', {
+  cache: { threshold: 0.01 },
+  cacheKey: 'dashboard-check',
+});
+```
+
+## Threshold Priority
+
+Thresholds are resolved in priority order (highest wins):
+
+| Priority | Source | Example |
+|----------|--------|---------|
+| 1 (highest) | Per-command option | `find(desc, { cache: { thresholds: { screen: 0.1 } } })` |
+| 2 | Legacy number argument | `find(desc, 0.1)` |
+| 3 | Global constructor config | `new TestDriver({ cache: { thresholds: { find: { screen: 0.1 } } } })` |
+| 4 (lowest) | Hard-coded defaults | `screen: 0.05`, `element: 0.8`, `assert: 0.05` |
+
+## Auto-Generated Cache Key
+
+When you don't specify a `cacheKey`, the SDK automatically generates one:
+
+1. Walks the call stack to find your test file
+2. Reads the file content
+3. Computes a **SHA-256 hash** of the content
+4. Uses the **first 16 hex characters** as the cache key
+
+This means:
+- **Same test file** → same cache key → cache hits
+- **Modified test file** → different hash → automatic cache invalidation
+- **Different test files** → different keys → isolated caches
+
+```javascript
+// Auto-generated cache key from test file hash
+const testdriver = new TestDriver();
+// cacheKey = "a3f2b1c4d5e6f7a8" (auto)
+
+// Manual override
+const testdriver = new TestDriver({ cacheKey: 'login-test-v2' });
+```
+
+## Template Matching (OpenCV)
+
+Element cache validation uses OpenCV's normalized cross-correlation coefficient (`TM_CCOEFF_NORMED`) to verify that the cached element is still visible at the expected position.
+
+**Algorithm:**
+1. Load the cached element crop (needle) and current screenshot (haystack)
+2. Run `cv.matchTemplate()` with `TM_CCOEFF_NORMED`
+3. Binary threshold at the configured element threshold
+4. Find contours to extract match positions
+5. Return matches with `{ x, y, width, height, centerX, centerY }`
+
+**Scale factors tried:** `[1, 0.5, 2, 0.75, 1.25, 1.5]`
+**Thresholds tried:** `[0.9, 0.8, 0.7]` (picks highest matching threshold)
+
+This accounts for minor scaling differences between screenshots taken at different times or resolutions.
+
+## Cache Partitioning
+
+Cache entries are partitioned by:
+- **`cacheKey`** — identifies the test file
+- **`os`** — operating system (linux, windows, darwin)
+- **`resolution`** — screen resolution
+
+This means cache from a Linux run won't be used for a Windows run, even with the same cache key.
+
+## Debugging Cache
+
+API responses include cache metadata:
+
+| Field | Description |
+|---|---|
+| `cacheHit` | `true` if cache was used |
+| `similarity` | Pixel diff percentage between screenshots |
+| `cacheSimilarity` | OpenCV template match score |
+
+Use `getDebugInfo()` on an element to inspect cache results:
+
+```javascript
+const el = await testdriver.find('submit button');
+const debug = el.getDebugInfo();
+console.log(debug);
+// { cacheHit: true, similarity: 0.02, cacheSimilarity: 0.92, ... }
+```
+
+## Types
+
+```typescript
+interface CacheConfig {
+  enabled?: boolean;            // Default: true
+  thresholds?: CacheThresholds;
+}
+
+interface CacheThresholds {
+  find?: FindCacheThresholds;
+  assert?: number;              // Default: 0.05
+}
+
+interface FindCacheThresholds {
+  screen?: number;              // Default: 0.05
+  element?: number;             // Default: 0.8
+}
+
+interface CacheDebugInfo {
+  cacheHit: boolean;
+  similarity: number;
+  cacheSimilarity: number;
+}
+```

package/ai/skills/testdriver-errors/SKILL.md

@@ -0,0 +1,246 @@
+---
+name: testdriver:errors
+description: Custom error classes and error handling
+---
+<!-- Generated from errors.mdx. DO NOT EDIT. -->
+
+## Overview
+
+TestDriver provides custom error classes with rich debugging information. These are exported from the SDK and can be used for `instanceof` checks in your tests.
+
+```javascript
+import TestDriver, { ElementNotFoundError, AIError } from 'testdriverai';
+```
+
+## ElementNotFoundError
+
+Thrown when `find()` cannot locate an element on screen, or when calling `click()`/`hover()` on an unfound element.
+
+```javascript
+try {
+  await testdriver.find('nonexistent button').click();
+} catch (error) {
+  if (error instanceof ElementNotFoundError) {
+    console.log(error.description);       // "nonexistent button"
+    console.log(error.screenshotPath);    // path to debug screenshot
+    console.log(error.pixelDiffPath);     // path to pixel diff image
+  }
+}
+```
+
+### Properties
+
+<ParamField path="name" type="string">
+  Always `"ElementNotFoundError"`.
+</ParamField>
+
+<ParamField path="message" type="string">
+  Enhanced message with a debug block containing element description, cache status, similarity scores, and AI response details.
+</ParamField>
+
+<ParamField path="description" type="string">
+  The original element description passed to `find()`.
+</ParamField>
+
+<ParamField path="screenshotPath" type="string | null">
+  Absolute path to a debug screenshot saved at the time of failure. Written to `<os.tmpdir>/testdriver-debug/screenshot-<timestamp>.png`.
+</ParamField>
+
+<ParamField path="pixelDiffPath" type="string | null">
+  Absolute path to a pixel diff image showing the comparison between the cached and current screenshots. Written to `<os.tmpdir>/testdriver-debug/pixel-diff-error-<timestamp>.png`. Only present when cache was involved.
+</ParamField>
+
+<ParamField path="cachedImagePath" type="string | null">
+  URL to the cached image that was compared against.
+</ParamField>
+
+<ParamField path="aiResponse" type="object | null">
+  Sanitized AI response object. Large binary fields (`croppedImage`, `screenshot`, `pixelDiffImage`) are removed. Contains cache metadata like `similarity`, `cacheHit`, `cacheStrategy`, `cacheDiffPercent`, and `threshold`.
+</ParamField>
+
+<ParamField path="timestamp" type="string">
+  ISO 8601 timestamp of when the error was created.
+</ParamField>
+
+### Enhanced Message
+
+The error message is automatically enhanced with debugging information:
+
+```
+Element not found: "submit button"
+
+=== Element Debug Info ===
+Element: submit button
+Cache Hit: false
+Similarity: 0.23
+Cache Strategy: pixel-diff
+Threshold: 0.05
+AI Response Element: null
+```
+
+### Stack Trace Cleaning
+
+Stack traces are automatically cleaned to remove internal SDK frames (`Element.*`, `sdk.js` internals), showing only your test code for easier debugging.
+
+## AIError
+
+Thrown when `act()` exhausts all retry attempts.
+
+```javascript
+try {
+  await testdriver.act('perform complex workflow', { tries: 3 });
+} catch (error) {
+  if (error instanceof AIError) {
+    console.log(error.task);       // "perform complex workflow"
+    console.log(error.tries);     // 3
+    console.log(error.duration);  // 15234 (ms)
+    console.log(error.cause);     // underlying Error
+  }
+}
+```
+
+### Properties
+
+<ParamField path="name" type="string">
+  Always `"AIError"`.
+</ParamField>
+
+<ParamField path="message" type="string">
+  Enhanced message with execution details block.
+</ParamField>
+
+<ParamField path="task" type="string">
+  The task description passed to `act()`.
+</ParamField>
+
+<ParamField path="tries" type="number">
+  Number of attempts that were made.
+</ParamField>
+
+<ParamField path="maxTries" type="number">
+  Maximum number of attempts configured.
+</ParamField>
+
+<ParamField path="duration" type="number">
+  Total execution time in milliseconds.
+</ParamField>
+
+<ParamField path="cause" type="Error | undefined">
+  The underlying error that caused the final failure.
+</ParamField>
+
+<ParamField path="timestamp" type="string">
+  ISO 8601 timestamp of when the error was created.
+</ParamField>
+
+### Enhanced Message
+
+```
+AI failed: Element not found after 3 attempts
+
+=== AI Execution Details ===
+Task: perform complex workflow
+Tries: 3 / 3
+Duration: 15234ms
+Cause: ElementNotFoundError: Element not found: "submit button"
+```
+
+## Internal Error Classes
+
+These errors are used internally by the agent and are not exported, but may appear as the `cause` of an `AIError`:
+
+### MatchError
+
+Thrown when element matching fails (text, image, or assertion).
+
+| Property | Type | Description |
+|---|---|---|
+| `fatal` | `boolean` | If `true`, cannot be healed. Default: `false` |
+| `attachScreenshot` | `boolean` | Always `true` — a screenshot is attached to the error |
+
+### CommandError
+
+Thrown for invalid arguments or unsupported operations.
+
+| Property | Type | Description |
+|---|---|---|
+| `fatal` | `boolean` | Always `true` |
+| `attachScreenshot` | `boolean` | Always `false` |
+
+## Soft Assert Mode
+
+Inside `act()`, assertions run in **soft assert mode**. When an assertion fails, it returns the failure result instead of throwing, allowing the AI to process the failure and adjust its approach.
+
+```javascript
+// Inside act(), assertion failures don't throw
+await testdriver.act('verify the dashboard shows correct data', {
+  tries: 3,
+});
+// The AI can see assertion results and self-correct
+```
+
+This is automatic — you don't need to configure it. Regular `assert()` calls outside of `act()` will throw normally on failure.
+
+## Error Handling Patterns
+
+### Catching Specific Errors
+
+```javascript
+import TestDriver, { ElementNotFoundError, AIError } from 'testdriverai';
+
+try {
+  await testdriver.find('submit button').click();
+} catch (error) {
+  if (error instanceof ElementNotFoundError) {
+    // Element wasn't found — check screenshot for debugging
+    console.log('Debug screenshot:', error.screenshotPath);
+  } else if (error instanceof AIError) {
+    // AI exhausted retries
+    console.log(`Failed after ${error.tries} tries in ${error.duration}ms`);
+  } else {
+    throw error; // Unexpected error
+  }
+}
+```
+
+### Using Debug Screenshots
+
+```javascript
+try {
+  const el = await testdriver.find('checkout button');
+  await el.click();
+} catch (error) {
+  if (error instanceof ElementNotFoundError) {
+    // Screenshot of what the screen looked like
+    console.log('Screen:', error.screenshotPath);
+    // Pixel diff showing cache comparison
+    console.log('Diff:', error.pixelDiffPath);
+    // Full AI response metadata
+    console.log('AI:', JSON.stringify(error.aiResponse, null, 2));
+  }
+}
+```
+
+## Types
+
+```typescript
+class ElementNotFoundError extends Error {
+  name: 'ElementNotFoundError';
+  description: string;
+  screenshotPath: string | null;
+  pixelDiffPath: string | null;
+  cachedImagePath: string | null;
+  aiResponse: Record<string, any> | null;
+  timestamp: string;
+}
+
+class AIError extends Error {
+  name: 'AIError';
+  task: string;
+  tries: number;
+  maxTries: number;
+  duration: number;
+  cause?: Error;
+  timestamp: string;
+}
+```