chrometools-mcp 1.0.0 → 1.1.0

package/README.md

@@ -6,11 +6,15 @@ MCP server for Chrome automation using Puppeteer with persistent browser session
 
 - [Installation](#installation)
 - [Usage](#usage)
-- [Available Tools](#available-tools) - **16 Tools Total**
+- [AI Optimization Features](#ai-optimization-features) ⭐ **NEW**
+- [Scenario Recorder](#scenario-recorder) ⭐ **NEW** - Visual UI-based recording with smart optimization
+- [Available Tools](#available-tools) - **26+ Tools Total**
+  - [AI-Powered Tools](#ai-powered-tools) ⭐ **NEW** - smartFindElement, analyzePage, getAllInteractiveElements, findElementsByText
   - [Core Tools](#1-core-tools) - ping, openBrowser
   - [Interaction Tools](#2-interaction-tools) - click, type, scrollTo
   - [Inspection Tools](#3-inspection-tools) - getElement, getComputedCss, getBoxModel, screenshot
   - [Advanced Tools](#4-advanced-tools) - executeScript, getConsoleLogs, hover, setStyles, setViewport, getViewport, navigateTo
+  - [Recorder Tools](#5-recorder-tools) ⭐ **NEW** - enableRecorder, executeScenario, listScenarios, searchScenarios, getScenarioInfo, deleteScenario
 - [Typical Workflow Example](#typical-workflow-example)
 - [Tool Usage Tips](#tool-usage-tips)
 - [Configuration](#configuration)
@@ -40,8 +44,114 @@ Add to your MCP client configuration (e.g., Claude Desktop):
 }
 ```
 
+## AI Optimization Features
+
+⭐ **NEW**: Dramatically reduce AI agent request cycles with intelligent element finding and page analysis.
+
+### Why This Matters
+
+Traditional browser automation with AI requires many trial-and-error cycles:
+```
+AI: "Find login button"
+→ Try selector #1: Not found
+→ Try selector #2: Not found
+→ Try selector #3: Found! (3 requests, 15-30 seconds)
+```
+
+**With AI optimization:**
+```
+AI: smartFindElement("login button")
+→ Returns ranked candidates with confidence scores (1 request, 2 seconds)
+```
+
+### Key Features
+
+1. **`smartFindElement`** - Natural language element search with multilingual support
+2. **`analyzePage`** - Complete page structure in one request (cached)
+3. **AI Hints** - Automatic context in all tools (page type, available actions, suggestions)
+4. **Batch helpers** - `getAllInteractiveElements`, `findElementsByText`
+
+**Performance:** 3-5x faster, 5-10x fewer requests
+
+📚 [Full AI Optimization Guide](AI_OPTIMIZATION.md)
+
+## Scenario Recorder
+
+⭐ **NEW**: Visual UI-based recorder for creating reusable test scenarios with automatic secret detection.
+
+### Features
+
+- **Visual Widget** - Floating recorder UI with compact mode (50x50px minimize button)
+- **Smart Recording** - Captures clicks, typing, navigation with intelligent optimization
+- **Secret Detection** - Auto-detects passwords/emails and stores them securely
+- **Action Optimization** - Combines sequential actions, removes duplicates
+- **Scenario Management** - Save, load, execute, search, and delete scenarios
+- **Dependencies** - Chain scenarios together with dependency resolution
+
+### Quick Start
+
+```javascript
+// 1. Enable recorder UI
+enableRecorder()
+
+// 2. Click "Start" in widget, perform actions, click "Stop & Save"
+// 3. Execute saved scenario
+executeScenario({ name: "login_flow", parameters: { email: "user@test.com" } })
+```
+
+📚 [Full Recorder Guide](RECORDER_QUICKSTART.md) | [Recorder Spec](RECORDER_SPEC.md)
+
 ## Available Tools
 
+### AI-Powered Tools
+
+#### smartFindElement ⭐
+Find elements using natural language descriptions instead of CSS selectors.
+- **Parameters**:
+  - `description` (required): Natural language (e.g., "login button", "email field")
+  - `maxResults` (optional): Max candidates to return (default: 5)
+- **Use case**: When you don't know the exact selector
+- **Returns**: Ranked candidates with confidence scores, selectors, and reasoning
+- **Example**:
+  ```json
+  {
+    "description": "submit button",
+    "maxResults": 3
+  }
+  ```
+  Returns:
+  ```json
+  {
+    "candidates": [
+      { "selector": "button.login-btn", "confidence": 0.95, "text": "Login", "reason": "type=submit, in form, matching keyword" },
+      { "selector": "#submit", "confidence": 0.7, "text": "Send", "reason": "submit class" }
+    ],
+    "hints": { "suggestion": "Use selector: button.login-btn" }
+  }
+  ```
+
+#### analyzePage ⭐
+Get complete page structure in one request. Results are cached per URL.
+- **Parameters**:
+  - `refresh` (optional): Force refresh cache (default: false)
+- **Use case**: Understanding page structure before planning actions
+- **Returns**: Complete map of forms, inputs, buttons, links, navigation with selectors
+- **Example**: Returns structured data for all interactive elements on the page
+
+#### getAllInteractiveElements
+Get all clickable/fillable elements with their selectors.
+- **Parameters**:
+  - `includeHidden` (optional): Include hidden elements (default: false)
+- **Returns**: Array of all interactive elements with selectors and metadata
+
+#### findElementsByText
+Find elements by their visible text content.
+- **Parameters**:
+  - `text` (required): Text to search for
+  - `exact` (optional): Exact match only (default: false)
+  - `caseSensitive` (optional): Case sensitive search (default: false)
+- **Returns**: Elements containing the text with their selectors
+
 ### 1. Core Tools
 
 #### ping
@@ -167,6 +277,60 @@ Navigate to different URL while keeping browser instance.
 - **Use case**: Moving between pages in workflow
 - **Returns**: New page title
 
+### 5. Recorder Tools ⭐ NEW
+
+#### enableRecorder
+Inject visual recorder UI widget into the current page.
+- **Parameters**: None
+- **Use case**: Start recording user interactions visually
+- **Returns**: Success status
+- **Features**:
+  - Floating widget with compact mode (minimize to 50x50px)
+  - Visual recording indicator (red pulsing border)
+  - Start/Pause/Stop/Stop & Save/Clear controls
+  - Real-time action list display
+  - Metadata fields (name, description, tags)
+
+#### executeScenario
+Execute a previously recorded scenario by name.
+- **Parameters**:
+  - `name` (required): Scenario name
+  - `parameters` (optional): Runtime parameters (e.g., { email: "user@test.com" })
+- **Use case**: Run automated test scenarios
+- **Returns**: Execution result with success/failure status
+- **Features**:
+  - Automatic dependency resolution
+  - Secret parameter injection
+  - Fallback selector retry logic
+
+#### listScenarios
+Get all available scenarios with metadata.
+- **Parameters**: None
+- **Use case**: Browse recorded scenarios
+- **Returns**: Array of scenarios with names, descriptions, tags, timestamps
+
+#### searchScenarios
+Search scenarios by text or tags.
+- **Parameters**:
+  - `text` (optional): Search in name/description
+  - `tags` (optional): Array of tags to filter
+- **Use case**: Find specific scenarios
+- **Returns**: Matching scenarios
+
+#### getScenarioInfo
+Get detailed information about a scenario.
+- **Parameters**:
+  - `name` (required): Scenario name
+  - `includeSecrets` (optional): Include secret values (default: false)
+- **Use case**: Inspect scenario actions and dependencies
+- **Returns**: Full scenario details (actions, metadata, dependencies)
+
+#### deleteScenario
+Delete a scenario and its associated secrets.
+- **Parameters**: `name` (required)
+- **Use case**: Clean up unused scenarios
+- **Returns**: Success confirmation
+
 ---
 
 ## Typical Workflow Example

package/RECORDER_QUICKSTART.md

@@ -0,0 +1,408 @@
+# Scenario Recorder - Quick Start Guide
+
+## What is the Scenario Recorder?
+
+The Scenario Recorder allows you to **visually record browser interactions** and save them as reusable scenarios. Instead of writing selectors manually, you interact with the page normally while the recorder captures all your actions.
+
+## Key Features
+
+✅ **Visual Recording** - Browser UI widget for start/stop/save
+✅ **9 Action Types** - Click, type, select, scroll, hover, keypress, wait, upload, drag
+✅ **Smart Secrets** - Auto-detects passwords/emails in auth forms only
+✅ **Action Optimization** - Combines sequential actions automatically
+✅ **Dependency Chaining** - Link scenarios together
+✅ **Retry Logic** - Auto-recovery if selectors fail
+
+## Basic Workflow
+
+### 1. Enable Recorder in Browser
+
+```javascript
+// MCP Tool: enableRecorder
+await enableRecorder()
+```
+
+This injects a floating widget into the page.
+
+### 2. Start Recording
+
+Click **"Start"** button in the widget. The widget shows:
+- 🟢 Green dot when recording
+- Action count in real-time
+- Visual highlighting of interactions
+
+### 3. Perform Actions
+
+Interact with the page normally:
+- Click buttons/links
+- Fill form fields
+- Select dropdowns
+- Scroll to elements
+- Upload files
+- Press keyboard shortcuts
+
+The recorder captures everything!
+
+### 4. Fill Metadata (Optional)
+
+Click ⚙️ to expand metadata:
+- **Scenario Name** (required)
+- Description
+- Tags (comma-separated)
+
+### 5. Stop & Save
+
+Click **"Stop & Save"** button.
+
+The scenario is saved to:
+- `scenarios/<name>.json` - Shareable scenario
+- `secrets/<name>.json` - Private credentials (auto .gitignore)
+
+## Using Recorded Scenarios
+
+### Execute a Scenario
+
+```javascript
+// MCP Tool: executeScenario
+await executeScenario({
+  name: "login_flow",
+  parameters: {
+    email: "user@example.com",
+    password: "secret123"
+  }
+})
+```
+
+### List Available Scenarios
+
+```javascript
+// MCP Tool: listScenarios
+await listScenarios()
+// Returns: [{ name, description, tags, dependencies, createdAt, updatedAt }, ...]
+```
+
+### Search Scenarios
+
+```javascript
+// MCP Tool: searchScenarios
+await searchScenarios({
+  text: "checkout",
+  tags: ["ecommerce"]
+})
+```
+
+## Advanced Features
+
+### 1. Dependency Chaining
+
+Scenarios can depend on other scenarios:
+
+```json
+{
+  "name": "checkout_flow",
+  "metadata": {
+    "dependencies": [
+      {
+        "scenario": "login_flow",
+        "optional": false,
+        "condition": {
+          "check": "isAuthenticated",
+          "skipIf": true
+        }
+      }
+    ]
+  }
+}
+```
+
+When you execute `checkout_flow`, it automatically runs `login_flow` first (if not already authenticated).
+
+### 2. Parameter Substitution
+
+Use `{{paramName}}` in recorded values:
+
+```json
+{
+  "type": "type",
+  "selector": { "value": "input[name='search']" },
+  "data": {
+    "text": "{{searchQuery}}"
+  }
+}
+```
+
+At execution time, provide the parameter:
+
+```javascript
+executeScenario({
+  name: "search_products",
+  parameters: {
+    searchQuery: "laptop"
+  }
+})
+```
+
+### 3. Secret Detection
+
+Secrets are **automatically detected** in authentication forms:
+
+**✅ Detected as secrets:**
+- Password fields in login/register forms
+- Email fields in login/register forms
+- Phone fields in login/register forms
+- OTP/verification codes
+
+**❌ NOT detected as secrets:**
+- Search boxes
+- Comment fields
+- Profile update forms (non-auth)
+- Phone fields in checkout forms
+
+Secrets are:
+- Stored separately in `secrets/<name>.json`
+- Automatically .gitignored
+- Masked in UI (shown as ***)
+- Substituted with {{parameter}} in scenarios
+
+### 4. Custom Selects
+
+The recorder detects custom select components (React Select, Material UI, etc.) and records them as multi-step actions:
+
+```json
+{
+  "type": "select",
+  "data": {
+    "selectType": "custom",
+    "steps": [
+      { "action": "click", "selector": ".select-container" },
+      { "action": "wait", "duration": 300 },
+      { "action": "click", "selector": ".option[data-value='US']" }
+    ]
+  }
+}
+```
+
+### 5. Action Optimization
+
+After recording, actions are automatically optimized:
+
+**Before optimization:**
+```json
+[
+  { "type": "type", "data": { "text": "H" } },
+  { "type": "type", "data": { "text": "e" } },
+  { "type": "type", "data": { "text": "l" } },
+  { "type": "type", "data": { "text": "l" } },
+  { "type": "type", "data": { "text": "o" } }
+]
+```
+
+**After optimization:**
+```json
+[
+  { "type": "type", "data": { "text": "Hello" } }
+]
+```
+
+Other optimizations:
+- Removes duplicate clicks
+- Merges sequential waits
+- Detects custom select patterns
+- Removes unnecessary scrolls/hovers
+
+## File Structure
+
+```
+your-project/
+├── scenarios/           # Shareable scenarios (commit to git)
+│   ├── index.json      # Scenario metadata index
+│   ├── login_flow.json
+│   └── checkout_flow.json
+│
+└── secrets/            # Private credentials (auto .gitignore)
+    ├── .gitignore      # Auto-created
+    ├── login_flow.json
+    └── checkout_flow.json
+```
+
+## Example Scenario
+
+Here's what a recorded login scenario looks like:
+
+```json
+{
+  "name": "login_flow",
+  "version": "1.0",
+  "createdAt": "2025-01-15T10:30:00.000Z",
+  "metadata": {
+    "description": "Standard login flow",
+    "tags": ["auth", "login"],
+    "dependencies": [],
+    "parameters": {
+      "email": { "type": "string", "required": true },
+      "password": { "type": "string", "required": true }
+    }
+  },
+  "chain": [
+    {
+      "type": "click",
+      "selector": { "primary": "a.login-link" },
+      "timestamp": 1705320000000,
+      "data": { "text": "Login" }
+    },
+    {
+      "type": "type",
+      "selector": { "primary": "input[name='email']" },
+      "timestamp": 1705320001000,
+      "data": { "text": "{{email}}", "isSecret": true }
+    },
+    {
+      "type": "type",
+      "selector": { "primary": "input[type='password']" },
+      "timestamp": 1705320002000,
+      "data": { "text": "{{password}}", "isSecret": true }
+    },
+    {
+      "type": "click",
+      "selector": { "primary": "button[type='submit']" },
+      "timestamp": 1705320003000,
+      "data": { "text": "Sign In" }
+    }
+  ]
+}
+```
+
+## Troubleshooting
+
+### Recorder UI Not Visible
+
+**Problem:** Widget doesn't appear after calling `enableRecorder()`
+
+**Solutions:**
+1. Check browser console for errors
+2. Ensure page has finished loading
+3. Try refreshing page and enabling again
+
+### Selector Fails During Playback
+
+**Problem:** "Element not found" error during execution
+
+**Solutions:**
+1. **Automatic:** Executor retries with fallback selectors
+2. **Automatic:** Executor uses smartFindElement with element text
+3. **Manual:** Edit scenario JSON and update selector
+
+### Secret Not Detected
+
+**Problem:** Password field not marked as secret
+
+**Check:**
+1. Is the field in a `<form>` element?
+2. Does the form have auth keywords (login/register/reset)?
+3. Is the field type="password" or has password-related name/id?
+
+**Note:** Phone/email in non-auth forms are NOT secrets by design!
+
+### Actions Too Granular
+
+**Problem:** Too many small actions recorded
+
+**Solution:** Action optimizer runs automatically, but you can also:
+1. Increase debounce time in recorder settings
+2. Manually edit scenario JSON after recording
+
+## Best Practices
+
+### 1. Naming Scenarios
+
+✅ Good:
+- `login_flow`
+- `search_products`
+- `checkout_guest`
+
+❌ Bad:
+- `test1`
+- `scenario_2025_01_15`
+- `my_recording`
+
+### 2. Using Tags
+
+Group related scenarios:
+
+```javascript
+tags: ["auth", "login"]           // Login scenario
+tags: ["auth", "register"]        // Register scenario
+tags: ["ecommerce", "checkout"]   // Checkout scenario
+tags: ["admin", "users"]          // Admin user management
+```
+
+### 3. Dependency Design
+
+**Keep dependencies simple:**
+
+✅ Good:
+```
+checkout_flow → login_flow
+```
+
+❌ Bad (deep nesting):
+```
+scenario_5 → scenario_4 → scenario_3 → scenario_2 → scenario_1
+```
+
+### 4. Parameter Naming
+
+Use clear, descriptive parameter names:
+
+✅ Good:
+- `email`
+- `password`
+- `searchQuery`
+- `productId`
+
+❌ Bad:
+- `param1`
+- `val`
+- `x`
+
+## Performance Tips
+
+### Execution Speed
+
+Typical execution times:
+- Simple form fill: **2-5 seconds**
+- Multi-page flow: **10-15 seconds**
+- Complex workflow: **30-60 seconds**
+
+### Optimization
+
+1. **Use dependencies** instead of duplicating actions
+2. **Combine scenarios** when they're always used together
+3. **Remove unnecessary waits** from recorded scenarios
+4. **Use specific selectors** (ID, data-testid) when possible
+
+## API Reference
+
+### MCP Tools for Recorder
+
+| Tool | Purpose |
+|------|---------|
+| `enableRecorder()` | Inject recorder widget into page |
+| `executeScenario(name, params)` | Run a scenario with parameters |
+| `listScenarios()` | Get all scenarios |
+| `searchScenarios(query)` | Search by text/tags |
+| `getScenarioInfo(name)` | Get scenario details |
+| `deleteScenario(name)` | Delete a scenario |
+| `importScenario(json)` | Import from JSON |
+| `exportScenario(name)` | Export to JSON |
+| `getStorageStats()` | Get statistics |
+| `validateStorage()` | Check integrity |
+
+## Next Steps
+
+1. **Try recording** a simple login flow
+2. **Execute the scenario** with different credentials
+3. **Create dependencies** between scenarios
+4. **Share scenarios** with your team (commit `scenarios/` directory)
+
+For detailed technical information, see [RECORDER_SPEC.md](RECORDER_SPEC.md).