ttp-agent-sdk 2.30.0 → 2.32.0

package/README.md

@@ -286,6 +286,130 @@ const voiceSDK = new VoiceSDK({
 }
 ```
 
+## Capture Screenshot Tool (`capture_screen`)
+
+The `capture_screen` tool allows AI agents to capture screenshots of the browser page during conversations. It uses `html2canvas` to render the DOM as an image.
+
+### Capturing the Entire Screen
+
+When capturing the entire screen, the tool has two modes:
+
+1. **Viewport only** (default) - Captures only the visible portion of the page (`document.body`)
+2. **Full scrollable page** - When `fullPage: true`, captures the entire page including all content below the fold
+
+**Example: Capture entire visible viewport**
+```javascript
+// The agent can call this tool with:
+{
+  tool: "capture_screen",
+  params: {
+    format: "jpeg",        // 'png' or 'jpeg' (jpeg is smaller)
+    quality: 0.85,         // JPEG quality 0-1 (only for jpeg)
+    scale: 1,              // Resolution scale (1 = normal, 2 = retina)
+    maxWidth: 1280,        // Max width in pixels (auto-resizes if larger)
+    maxHeight: 1280        // Max height in pixels (auto-resizes if larger)
+  }
+}
+```
+
+**Example: Capture full scrollable page**
+```javascript
+// To capture the entire page including content below the fold:
+{
+  tool: "capture_screen",
+  params: {
+    fullPage: true,        // Capture entire scrollable page
+    format: "jpeg",
+    quality: 0.85,
+    maxWidth: 1920,        // Higher limits for full page
+    maxHeight: 5000        // Accommodate long pages
+  }
+}
+```
+
+**How Full Page Capture Works:**
+
+When `fullPage: true` is set (and no `selector` is provided), the tool:
+- Sets `windowHeight` and `height` to `document.documentElement.scrollHeight` (total page height)
+- Sets `y: 0` to start from the top
+- Captures the entire scrollable content, not just the visible viewport
+- The resulting image height equals the full scrollable height of the page
+
+**Example: Capture specific element**
+```javascript
+// Capture a specific element using CSS selector:
+{
+  tool: "capture_screen",
+  params: {
+    selector: "#my-element",  // CSS selector (e.g., "#header", ".content", "main")
+    format: "png",            // PNG preserves transparency
+    scale: 2                  // 2x resolution for crisp screenshots
+  }
+}
+```
+
+#### Tool Parameters
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `selector` | string | `null` | CSS selector for specific element. If not provided, captures entire page. |
+| `format` | string | `'jpeg'` | Output format: `'png'` or `'jpeg'`. JPEG is smaller, PNG supports transparency. |
+| `quality` | number | `0.85` | JPEG quality (0-1). Only used when `format` is `'jpeg'`. |
+| `scale` | number | `1` | Resolution scale factor. `1` = normal, `2` = retina/2x resolution. |
+| `maxWidth` | number | `1280` | Maximum width in pixels. Image will be resized if larger (maintains aspect ratio). |
+| `maxHeight` | number | `1280` | Maximum height in pixels. Image will be resized if larger (maintains aspect ratio). |
+| `fullPage` | boolean | `false` | When `true` and no `selector` provided, captures entire scrollable page (not just viewport). |
+
+#### Return Value
+
+The tool returns a result object with:
+
+```javascript
+{
+  image: "base64_encoded_image_data",  // Base64 string (without data URI prefix)
+  mimeType: "image/jpeg",              // MIME type: "image/jpeg" or "image/png"
+  width: 1280,                          // Final image width in pixels
+  height: 720,                          // Final image height in pixels
+  captureTimeMs: 234,                   // Time taken to capture in milliseconds
+  selector: "body"                      // Element that was captured (or selector used)
+}
+```
+
+#### Events
+
+The SDK emits events when screenshots are captured:
+
+```javascript
+voiceSDK.on('screenshotCaptured', (data) => {
+  console.log(`Screenshot captured: ${data.width}x${data.height}, ${data.sizeKB}KB`);
+  console.log(`Element: ${data.selector}`);
+});
+
+voiceSDK.on('screenshotError', (error) => {
+  console.error('Screenshot failed:', error.error);
+});
+```
+
+#### How It Works Internally
+
+The tool uses `html2canvas` to render the DOM:
+
+1. **Target Selection**: If `selector` is provided, captures that element. Otherwise captures `document.body`.
+2. **Full Page Mode**: When `fullPage: true` and no selector:
+   - Sets canvas height to `document.documentElement.scrollHeight`
+   - Captures from `y: 0` to capture entire scrollable area
+3. **Rendering**: `html2canvas` renders the DOM to a canvas element
+4. **Resizing**: If dimensions exceed `maxWidth`/`maxHeight`, image is resized maintaining aspect ratio
+5. **Encoding**: Canvas is converted to base64 (JPEG or PNG format)
+
+#### Important Notes
+
+- **Full Page Capture**: When `fullPage: true` is set, the tool captures the entire scrollable height (`document.documentElement.scrollHeight`), not just the visible viewport. This includes all content below the fold.
+- **Performance**: Full page captures may take longer, especially for very long pages. Consider using `maxHeight` to limit the capture size.
+- **Image Size**: Screenshots are automatically resized if they exceed `maxWidth` or `maxHeight` while maintaining aspect ratio.
+- **Cross-Origin**: The tool handles cross-origin images when possible (`useCORS: true`).
+- **Browser Compatibility**: Requires modern browsers with Canvas API support (Chrome 66+, Firefox 60+, Safari 11.1+, Edge 79+).
+
 ## Examples
 
 See the `examples/` directory for complete usage examples: