misoai-web 1.0.4 → 1.0.5

package/README.md

@@ -1,9 +1,353 @@
-## Documentation
+# @midscene/web - AI-Powered Web Automation
 
-Automate UI actions, extract data, and perform assertions using AI. It offers JavaScript SDK, Chrome extension, and support for scripting in YAML.
+[![npm version](https://badge.fury.io/js/@midscene%2Fweb.svg)](https://badge.fury.io/js/@midscene%2Fweb)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
-See https://midscenejs.com/ for details.
+AI-powered web automation library with advanced cumulative context system for intelligent, multi-step browser interactions.
 
-## License
+## 🚀 Features
 
-Midscene is MIT licensed.
+- **🧠 Cumulative Context System**: AI remembers previous actions and data across steps
+- **🗣️ Natural Language Data References**: No more `{stored.key}` syntax needed
+- **🎯 Smart Context-Aware Assertions**: Intelligent data replacement in assertions
+- **🔄 Multi-Step Workflows**: Seamless information flow between actions
+- **🛡️ Built-in CAPTCHA Solving**: Advanced AI-powered CAPTCHA detection and solving
+- **📊 Comprehensive Logging**: Full visibility into context operations and AI calls
+- **🎭 Framework Support**: Works with Puppeteer, Playwright, and more
+
+## 📦 Installation
+
+```bash
+npm install @midscene/web puppeteer
+# or
+yarn add @midscene/web puppeteer
+```
+
+## 🎯 Quick Start
+
+### Basic Usage with Cumulative Context
+
+```javascript
+import { PuppeteerAgent } from '@midscene/web';
+import puppeteer from 'puppeteer';
+
+const browser = await puppeteer.launch({ headless: false });
+const page = await browser.newPage();
+
+// Create agent with cumulative context enabled
+const agent = new PuppeteerAgent(page, {
+  enableCumulativeContext: true,  // 🔥 Enable context system
+  autoClearContext: true,         // 🧹 Start with clean context
+  testId: 'my-automation'
+});
+
+await page.goto('https://example.com');
+
+// Step 1: Store data with natural language
+await agent.aiQuery('kullanıcı adını "john_doe" olarak username şeklinde kaydet');
+
+// Step 2: Use stored data without {stored.key} syntax!
+await agent.aiAction('arama kutusuna username verisini yaz');
+
+// Step 3: Context-aware assertion
+await agent.aiAssert('arama kutusunda username değeri görünüyor');
+
+await browser.close();
+```
+
+## 🧠 Cumulative Context System
+
+### Natural Language Data Storage
+
+```javascript
+// Turkish natural language storage
+await agent.aiQuery('ürün adını "iPhone 15" olarak urun_adi şeklinde kaydet');
+await agent.aiQuery('fiyatı "999$" olarak fiyat şeklinde kaydet');
+
+// English natural language storage
+await agent.aiQuery('extract username and store as user_name');
+await agent.aiQuery('get email address, store as email');
+```
+
+### Automatic Data Usage
+
+```javascript
+// No need for {stored.key} syntax!
+await agent.aiAction('formu urun_adi ve fiyat ile doldur');
+await agent.aiAction('use user_name and email to fill the form');
+
+// Multiple data references in one action
+await agent.aiAction('sepete urun_adi ürününü fiyat fiyatıyla ekle');
+```
+
+### Context-Aware Assertions
+
+```javascript
+// Smart replacement - AI decides when to replace
+await agent.aiAssert('sepette urun_adi ürünü görünüyor'); // ✅ Will replace
+await agent.aiAssert('sayfada iPhone yazısı var');        // ❓ Context-dependent
+```
+
+## 🛡️ AI CAPTCHA Solving
+
+The `aiCaptcha` method provides intelligent CAPTCHA detection and solving capabilities with automatic execution:
+
+```javascript
+// Basic CAPTCHA solving (auto-detects complexity)
+await agent.aiCaptcha();
+
+// Advanced CAPTCHA solving with options
+await agent.aiCaptcha({
+  deepThink: true,              // Force deep analysis
+  autoDetectComplexity: true    // Auto-detect if deep thinking needed (default: true)
+});
+```
+
+### CAPTCHA Features
+
+- **🔍 Automatic Detection**: Identifies CAPTCHA type (text, image, unknown)
+- **🧠 Complexity Analysis**: Auto-detects if deep thinking is needed
+- **📝 Text CAPTCHAs**: Solves distorted text and automatically inputs solution
+- **🖼️ Image CAPTCHAs**: Handles "select all images with..." challenges with coordinate clicking
+- **🎯 Automatic Execution**: Performs all required actions (click, input, verify) automatically
+- **⚡ Smart Processing**: Uses appropriate AI model based on complexity
+- **🔄 Action Sequence**: Executes complete CAPTCHA solving workflow
+
+### CAPTCHA Response Format
+
+The AI returns a structured response with:
+
+```javascript
+{
+  captchaType: "text" | "image" | "unknown",
+  solution: "The solution text or description",
+  thought: "AI reasoning process",
+  actions: [
+    {
+      type: "click" | "input" | "verify",
+      target: "Description of target element",
+      value: "Text to input (for input actions)",
+      coordinates: [x, y] // For precise clicking
+    }
+  ]
+}
+```
+
+### CAPTCHA Example
+
+```javascript
+// Navigate to page with CAPTCHA
+await page.goto('https://example.com/login');
+
+// Fill login form
+await agent.aiInput('john@example.com', 'email field');
+await agent.aiInput('password123', 'password field');
+
+// Solve CAPTCHA automatically - AI will:
+// 1. Analyze the CAPTCHA type and complexity
+// 2. Generate solution
+// 3. Execute all required actions automatically
+const captchaResult = await agent.aiCaptcha({
+  autoDetectComplexity: true
+});
+
+console.log('CAPTCHA solved:', captchaResult.result.captchaType);
+console.log('Solution:', captchaResult.result.solution);
+console.log('Actions performed:', captchaResult.result.actions.length);
+
+// Form is now ready to submit
+await agent.aiTap('login button');
+```
+
+### Advanced CAPTCHA Scenarios
+
+```javascript
+// Complex image CAPTCHA with deep thinking
+await agent.aiCaptcha({
+  deepThink: true  // Forces detailed analysis for complex puzzles
+});
+
+// Text CAPTCHA with automatic input
+// AI will automatically:
+// - Click on input field
+// - Type the solution
+// - Click verify button
+await agent.aiCaptcha();
+
+// Using with cumulative context
+await agent.aiQuery('CAPTCHA çözümünü captcha_cozum olarak kaydet');
+await agent.aiCaptcha();
+await agent.aiAssert('CAPTCHA başarıyla çözüldü');
+```
+
+## ⚙️ Configuration Options
+
+```javascript
+const agent = new PuppeteerAgent(page, {
+  // Context System
+  enableCumulativeContext: true,    // Enable/disable context system
+  autoClearContext: false,          // Auto-clear on agent creation
+
+  // Basic Options
+  testId: 'my-test',               // Test identifier
+  cacheId: 'my-cache',             // Cache identifier
+  aiActionContext: 'context',      // AI behavior context
+
+  // Reporting
+  generateReport: true,            // Generate HTML reports
+  autoPrintReportMsg: true,        // Print report messages
+
+  // Navigation
+  forceSameTabNavigation: true,    // Force same-tab navigation
+  waitForNavigationTimeout: 30000, // Navigation timeout
+  waitForNetworkIdleTimeout: 5000  // Network idle timeout
+});
+```
+
+## 📊 Context Management
+
+### Direct Context Access
+
+```javascript
+// Get stored data
+const storedData = agent.getStoredData();
+console.log('Stored:', storedData);
+
+// Get step summary
+const summary = agent.getStepSummary();
+console.log('Steps:', summary);
+
+// Clear context manually
+agent.clearContext();
+
+// Get context store instance
+const contextStore = agent.getContextStore();
+```
+
+### Manual Data Storage
+
+```javascript
+const contextStore = agent.getContextStore();
+
+// Store data manually
+contextStore.storeData('customKey', 'customValue');
+
+// Store with aliases for natural language
+contextStore.storeDataWithAliases('productName', 'iPhone 15', ['urun', 'product']);
+
+// Get recent steps
+const recentSteps = contextStore.getRecentSteps(5);
+```
+
+## 🔍 Debug Logging
+
+Enable comprehensive logging to see context operations:
+
+```bash
+# All debug logs
+DEBUG=midscene:* node your-script.js
+
+# Context-specific logs
+DEBUG=midscene:agent node your-script.js
+
+# AI call logs
+DEBUG=midscene:ai:*,midscene:agent node your-script.js
+```
+
+### Log Examples
+
+```
+DEBUG midscene:agent Context replacement in aiAction: {
+  original: "type username in search box",
+  processed: "type john_doe in search box",
+  storedData: { "username": "john_doe" }
+}
+
+DEBUG midscene:agent Stored query result with aliases: {
+  key: "username",
+  value: "john_doe",
+  aliases: ["username", "user", "kullanici"]
+}
+```
+
+## 🎭 Multi-Step Workflow Example
+
+```javascript
+// E-commerce automation with context
+const agent = new PuppeteerAgent(page, {
+  enableCumulativeContext: true,
+  autoClearContext: true
+});
+
+await page.goto('https://shop.example.com');
+
+// Step 1: Search and store product info
+await agent.aiAction('arama kutusuna "laptop" yaz ve ara');
+await agent.aiQuery('ilk ürünün adını urun_adi olarak kaydet');
+await agent.aiQuery('ilk ürünün fiyatını urun_fiyati olarak kaydet');
+
+// Step 2: Add to cart using stored data
+await agent.aiAction('urun_adi ürününü sepete ekle');
+
+// Step 3: Verify cart contents
+await agent.aiAction('sepete git');
+await agent.aiAssert('sepette urun_adi ürünü urun_fiyati fiyatıyla görünüyor');
+
+// Step 4: Proceed to checkout
+await agent.aiAction('ödeme sayfasına git');
+await agent.aiAssert('ödeme sayfasında urun_adi ürünü listeleniyor');
+```
+
+## 🔄 Backward Compatibility
+
+The new context system is fully backward compatible:
+
+```javascript
+// Old syntax still works
+await agent.aiQuery('extract username, store as user');
+await agent.aiAction('type {stored.user} in field');
+
+// New syntax (recommended)
+await agent.aiQuery('kullanıcı adını user olarak kaydet');
+await agent.aiAction('alana user verisini yaz');
+
+// Mixed usage
+await agent.aiAction('use {stored.email} and user verisi together');
+```
+
+## 📚 API Reference
+
+### Core Methods
+
+- `aiAction(prompt)` - Perform actions with context awareness
+- `aiQuery(prompt)` - Extract data with automatic storage
+- `aiAssert(assertion)` - Context-aware assertions
+- `aiCaptcha(options)` - Intelligent CAPTCHA solving
+- `aiTap(target, options)` - Click elements
+- `aiInput(text, target, options)` - Input text
+
+### Context Methods
+
+- `getStoredData()` - Get all stored data
+- `getStepSummary()` - Get step summary
+- `clearContext()` - Clear context store
+- `getContextStore()` - Get context store instance
+
+## 🤝 Contributing
+
+We welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) for details.
+
+## 📄 License
+
+MIT License - see [LICENSE](LICENSE) file for details.
+
+## 🔗 Links
+
+- [Documentation](https://midscenejs.com)
+- [GitHub Repository](https://github.com/web-infra-dev/midscene)
+- [NPM Package](https://www.npmjs.com/package/@midscene/web)
+- [Examples](https://github.com/web-infra-dev/midscene/tree/main/packages/web-integration/examples)
+
+---
+
+**Made with ❤️ by the Midscene Team**

package/dist/es/agent.js

@@ -1660,7 +1660,7 @@ import yaml3 from "js-yaml";
 import semver from "semver";
 
 // package.json
-var version = "1.0.4";
+var version = "1.0.5";
 
 // src/common/task-cache.ts
 var debug3 = getDebug3("cache");
@@ -1842,10 +1842,13 @@ var PageAgent = class {
         generateReport: true,
         autoPrintReportMsg: true,
         groupName: "Midscene Report",
-        groupDescription: ""
+        groupDescription: "",
+        enableCumulativeContext: true,
+        autoClearContext: false
       },
       opts || {}
     );
+    this.initializeContextStore();
     if (this.page.pageType === "puppeteer" || this.page.pageType === "playwright") {
       this.page.waitForNavigationTimeout = this.opts.waitForNavigationTimeout || DEFAULT_WAIT_FOR_NAVIGATION_TIMEOUT;
       this.page.waitForNetworkIdleTimeout = this.opts.waitForNetworkIdleTimeout || DEFAULT_WAIT_FOR_NETWORK_IDLE_TIMEOUT;
@@ -1872,6 +1875,69 @@ var PageAgent = class {
       opts?.testId || this.page.pageType || "web"
     );
   }
+  /**
+   * Initialize context store for cumulative context functionality
+   */
+  async initializeContextStore() {
+    if (!this.opts.enableCumulativeContext) {
+      debug4("Cumulative context disabled via options");
+      return;
+    }
+    try {
+      const aiModel = await import("misoai-core/ai-model");
+      this.contextStore = aiModel.getContextStore();
+      debug4("Context store initialized successfully", {
+        autoClearContext: this.opts.autoClearContext,
+        testId: this.opts.testId
+      });
+      if (this.opts.autoClearContext) {
+        this.contextStore.clear();
+        debug4("Context store cleared due to autoClearContext option");
+      } else {
+        const existingData = this.contextStore.getAllData();
+        const existingSteps = this.contextStore.getRecentSteps(100).length;
+        debug4("Context store preserving existing data", {
+          existingDataKeys: Object.keys(existingData),
+          existingStepsCount: existingSteps
+        });
+      }
+    } catch (error) {
+      debug4("Failed to initialize context store:", error);
+      console.warn("⚠️ Could not initialize context store:", error);
+    }
+  }
+  /**
+   * Get the context store instance
+   */
+  getContextStore() {
+    return this.contextStore;
+  }
+  /**
+   * Clear the context store
+   */
+  clearContext() {
+    if (this.contextStore) {
+      this.contextStore.clear();
+    }
+  }
+  /**
+   * Get all stored data from context store
+   */
+  getStoredData() {
+    if (this.contextStore) {
+      return this.contextStore.getAllData();
+    }
+    return {};
+  }
+  /**
+   * Get step summary from context store
+   */
+  getStepSummary() {
+    if (this.contextStore) {
+      return this.contextStore.getStepSummary();
+    }
+    return "";
+  }
   async getUIContext(action) {
     if (action && (action === "extract" || action === "assert" || action === "captcha")) {
       return await parseContextFromWebPage(this.page, {
@@ -2107,18 +2173,30 @@ var PageAgent = class {
     };
   }
   async aiAction(taskPrompt, opt) {
-    try {
-      const aiModel = await import("misoai-core/ai-model");
-      const contextStore = aiModel.getContextStore();
-      const processedPrompt = contextStore.replaceAllReferences(taskPrompt, "action");
-      contextStore.addStep({
-        type: "action",
-        summary: `Action: ${processedPrompt}`,
-        prompt: processedPrompt
-      });
-      taskPrompt = processedPrompt;
-    } catch (error) {
-      debug4("Context store not available:", error);
+    if (this.opts.enableCumulativeContext && this.contextStore) {
+      try {
+        const originalPrompt = taskPrompt;
+        const processedPrompt = this.contextStore.replaceAllReferences(taskPrompt, "action");
+        if (originalPrompt !== processedPrompt) {
+          debug4("Context replacement in aiAction:", {
+            original: originalPrompt,
+            processed: processedPrompt,
+            storedData: this.contextStore.getAllData()
+          });
+        }
+        this.contextStore.addStep({
+          type: "action",
+          summary: `Action: ${processedPrompt}`,
+          prompt: processedPrompt
+        });
+        debug4("Added action step to context store:", {
+          stepNumber: this.contextStore.getRecentSteps(1)[0]?.stepNumber,
+          totalSteps: this.contextStore.getRecentSteps(100).length
+        });
+        taskPrompt = processedPrompt;
+      } catch (error) {
+        debug4("Context store operation failed:", error);
+      }
     }
     const cacheable = opt?.cacheable;
     const isVlmUiTars = vlLocateMode() === "vlm-ui-tars";
@@ -2189,38 +2267,50 @@ var PageAgent = class {
       debug4("Context store not available:", error);
     }
     const { output, executor } = await this.taskExecutor.query(processedDemand);
-    if (storageKey && output) {
-      try {
-        const aiModel = await import("misoai-core/ai-model");
-        const contextStore = aiModel.getContextStore();
-        const pendingAliases = contextStore._pendingAliases;
-        if (pendingAliases) {
-          contextStore.storeDataWithAliases(storageKey, output, pendingAliases, typeof processedDemand === "string" ? processedDemand : JSON.stringify(processedDemand));
-          delete contextStore._pendingAliases;
-        } else {
-          contextStore.storeData(storageKey, output);
+    if (this.opts.enableCumulativeContext && this.contextStore) {
+      if (storageKey && output) {
+        try {
+          const pendingAliases = this.contextStore._pendingAliases;
+          if (pendingAliases) {
+            this.contextStore.storeDataWithAliases(storageKey, output, pendingAliases, typeof processedDemand === "string" ? processedDemand : JSON.stringify(processedDemand));
+            delete this.contextStore._pendingAliases;
+            debug4("Stored query result with aliases:", {
+              key: storageKey,
+              value: output,
+              aliases: pendingAliases
+            });
+          } else {
+            this.contextStore.storeData(storageKey, output);
+            debug4("Stored query result:", {
+              key: storageKey,
+              value: output
+            });
+          }
+          this.contextStore.addStep({
+            type: "query",
+            summary: `Query: ${typeof processedDemand === "string" ? processedDemand : JSON.stringify(processedDemand)} (stored as ${storageKey})`,
+            data: output,
+            prompt: typeof processedDemand === "string" ? processedDemand : JSON.stringify(processedDemand)
+          });
+          debug4("Added query step to context store:", {
+            storageKey,
+            totalStoredItems: Object.keys(this.contextStore.getAllData()).length,
+            totalSteps: this.contextStore.getRecentSteps(100).length
+          });
+        } catch (error) {
+          debug4("Failed to store query result:", error);
+        }
+      } else {
+        try {
+          this.contextStore.addStep({
+            type: "query",
+            summary: `Query: ${typeof processedDemand === "string" ? processedDemand : JSON.stringify(processedDemand)}`,
+            data: output,
+            prompt: typeof processedDemand === "string" ? processedDemand : JSON.stringify(processedDemand)
+          });
+        } catch (error) {
+          debug4("Failed to add query step:", error);
         }
-        contextStore.addStep({
-          type: "query",
-          summary: `Query: ${typeof processedDemand === "string" ? processedDemand : JSON.stringify(processedDemand)} (stored as ${storageKey})`,
-          data: output,
-          prompt: typeof processedDemand === "string" ? processedDemand : JSON.stringify(processedDemand)
-        });
-      } catch (error) {
-        debug4("Failed to store query result:", error);
-      }
-    } else {
-      try {
-        const aiModel = await import("misoai-core/ai-model");
-        const contextStore = aiModel.getContextStore();
-        contextStore.addStep({
-          type: "query",
-          summary: `Query: ${typeof processedDemand === "string" ? processedDemand : JSON.stringify(processedDemand)}`,
-          data: output,
-          prompt: typeof processedDemand === "string" ? processedDemand : JSON.stringify(processedDemand)
-        });
-      } catch (error) {
-        debug4("Failed to add query step:", error);
       }
     }
     const metadata = this.afterTaskRunning(executor);
@@ -2333,17 +2423,29 @@ var PageAgent = class {
   }
   async aiAssert(assertion, msg, opt) {
     let processedAssertion = assertion;
-    try {
-      const aiModel = await import("misoai-core/ai-model");
-      const contextStore = aiModel.getContextStore();
-      processedAssertion = contextStore.replaceAllReferences(assertion, "assertion");
-      contextStore.addStep({
-        type: "assertion",
-        summary: `Assertion: ${processedAssertion}`,
-        prompt: processedAssertion
-      });
-    } catch (error) {
-      debug4("Context store not available:", error);
+    if (this.opts.enableCumulativeContext && this.contextStore) {
+      try {
+        const originalAssertion = assertion;
+        processedAssertion = this.contextStore.replaceAllReferences(assertion, "assertion");
+        if (originalAssertion !== processedAssertion) {
+          debug4("Context replacement in aiAssert:", {
+            original: originalAssertion,
+            processed: processedAssertion,
+            context: "assertion",
+            storedData: this.contextStore.getAllData()
+          });
+        }
+        this.contextStore.addStep({
+          type: "assertion",
+          summary: `Assertion: ${processedAssertion}`,
+          prompt: processedAssertion
+        });
+        debug4("Added assertion step to context store:", {
+          totalSteps: this.contextStore.getRecentSteps(100).length
+        });
+      } catch (error) {
+        debug4("Context store operation failed:", error);
+      }
     }
     let currentUrl = "";
     if (this.page.url) {