misoai-web 1.0.5 → 1.5.6

package/LICENSE

@@ -1,21 +1,21 @@
-MIT License
-
-Copyright (c) 2024-present Bytedance, Inc. and its affiliates.
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+MIT License
+
+Copyright (c) 2024-present Bytedance, Inc. and its affiliates.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,353 +1,9 @@
-# @midscene/web - AI-Powered Web Automation
-
-[![npm version](https://badge.fury.io/js/@midscene%2Fweb.svg)](https://badge.fury.io/js/@midscene%2Fweb)
-[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
-
-AI-powered web automation library with advanced cumulative context system for intelligent, multi-step browser interactions.
-
-## 🚀 Features
-
-- **🧠 Cumulative Context System**: AI remembers previous actions and data across steps
-- **🗣️ Natural Language Data References**: No more `{stored.key}` syntax needed
-- **🎯 Smart Context-Aware Assertions**: Intelligent data replacement in assertions
-- **🔄 Multi-Step Workflows**: Seamless information flow between actions
-- **🛡️ Built-in CAPTCHA Solving**: Advanced AI-powered CAPTCHA detection and solving
-- **📊 Comprehensive Logging**: Full visibility into context operations and AI calls
-- **🎭 Framework Support**: Works with Puppeteer, Playwright, and more
-
-## 📦 Installation
-
-```bash
-npm install @midscene/web puppeteer
-# or
-yarn add @midscene/web puppeteer
-```
-
-## 🎯 Quick Start
-
-### Basic Usage with Cumulative Context
-
-```javascript
-import { PuppeteerAgent } from '@midscene/web';
-import puppeteer from 'puppeteer';
-
-const browser = await puppeteer.launch({ headless: false });
-const page = await browser.newPage();
-
-// Create agent with cumulative context enabled
-const agent = new PuppeteerAgent(page, {
-  enableCumulativeContext: true,  // 🔥 Enable context system
-  autoClearContext: true,         // 🧹 Start with clean context
-  testId: 'my-automation'
-});
-
-await page.goto('https://example.com');
-
-// Step 1: Store data with natural language
-await agent.aiQuery('kullanıcı adını "john_doe" olarak username şeklinde kaydet');
-
-// Step 2: Use stored data without {stored.key} syntax!
-await agent.aiAction('arama kutusuna username verisini yaz');
-
-// Step 3: Context-aware assertion
-await agent.aiAssert('arama kutusunda username değeri görünüyor');
-
-await browser.close();
-```
-
-## 🧠 Cumulative Context System
-
-### Natural Language Data Storage
-
-```javascript
-// Turkish natural language storage
-await agent.aiQuery('ürün adını "iPhone 15" olarak urun_adi şeklinde kaydet');
-await agent.aiQuery('fiyatı "999$" olarak fiyat şeklinde kaydet');
-
-// English natural language storage
-await agent.aiQuery('extract username and store as user_name');
-await agent.aiQuery('get email address, store as email');
-```
-
-### Automatic Data Usage
-
-```javascript
-// No need for {stored.key} syntax!
-await agent.aiAction('formu urun_adi ve fiyat ile doldur');
-await agent.aiAction('use user_name and email to fill the form');
-
-// Multiple data references in one action
-await agent.aiAction('sepete urun_adi ürününü fiyat fiyatıyla ekle');
-```
-
-### Context-Aware Assertions
-
-```javascript
-// Smart replacement - AI decides when to replace
-await agent.aiAssert('sepette urun_adi ürünü görünüyor'); // ✅ Will replace
-await agent.aiAssert('sayfada iPhone yazısı var');        // ❓ Context-dependent
-```
-
-## 🛡️ AI CAPTCHA Solving
-
-The `aiCaptcha` method provides intelligent CAPTCHA detection and solving capabilities with automatic execution:
-
-```javascript
-// Basic CAPTCHA solving (auto-detects complexity)
-await agent.aiCaptcha();
-
-// Advanced CAPTCHA solving with options
-await agent.aiCaptcha({
-  deepThink: true,              // Force deep analysis
-  autoDetectComplexity: true    // Auto-detect if deep thinking needed (default: true)
-});
-```
-
-### CAPTCHA Features
-
-- **🔍 Automatic Detection**: Identifies CAPTCHA type (text, image, unknown)
-- **🧠 Complexity Analysis**: Auto-detects if deep thinking is needed
-- **📝 Text CAPTCHAs**: Solves distorted text and automatically inputs solution
-- **🖼️ Image CAPTCHAs**: Handles "select all images with..." challenges with coordinate clicking
-- **🎯 Automatic Execution**: Performs all required actions (click, input, verify) automatically
-- **⚡ Smart Processing**: Uses appropriate AI model based on complexity
-- **🔄 Action Sequence**: Executes complete CAPTCHA solving workflow
-
-### CAPTCHA Response Format
-
-The AI returns a structured response with:
-
-```javascript
-{
-  captchaType: "text" | "image" | "unknown",
-  solution: "The solution text or description",
-  thought: "AI reasoning process",
-  actions: [
-    {
-      type: "click" | "input" | "verify",
-      target: "Description of target element",
-      value: "Text to input (for input actions)",
-      coordinates: [x, y] // For precise clicking
-    }
-  ]
-}
-```
-
-### CAPTCHA Example
-
-```javascript
-// Navigate to page with CAPTCHA
-await page.goto('https://example.com/login');
-
-// Fill login form
-await agent.aiInput('john@example.com', 'email field');
-await agent.aiInput('password123', 'password field');
-
-// Solve CAPTCHA automatically - AI will:
-// 1. Analyze the CAPTCHA type and complexity
-// 2. Generate solution
-// 3. Execute all required actions automatically
-const captchaResult = await agent.aiCaptcha({
-  autoDetectComplexity: true
-});
-
-console.log('CAPTCHA solved:', captchaResult.result.captchaType);
-console.log('Solution:', captchaResult.result.solution);
-console.log('Actions performed:', captchaResult.result.actions.length);
-
-// Form is now ready to submit
-await agent.aiTap('login button');
-```
-
-### Advanced CAPTCHA Scenarios
-
-```javascript
-// Complex image CAPTCHA with deep thinking
-await agent.aiCaptcha({
-  deepThink: true  // Forces detailed analysis for complex puzzles
-});
-
-// Text CAPTCHA with automatic input
-// AI will automatically:
-// - Click on input field
-// - Type the solution
-// - Click verify button
-await agent.aiCaptcha();
-
-// Using with cumulative context
-await agent.aiQuery('CAPTCHA çözümünü captcha_cozum olarak kaydet');
-await agent.aiCaptcha();
-await agent.aiAssert('CAPTCHA başarıyla çözüldü');
-```
-
-## ⚙️ Configuration Options
-
-```javascript
-const agent = new PuppeteerAgent(page, {
-  // Context System
-  enableCumulativeContext: true,    // Enable/disable context system
-  autoClearContext: false,          // Auto-clear on agent creation
-
-  // Basic Options
-  testId: 'my-test',               // Test identifier
-  cacheId: 'my-cache',             // Cache identifier
-  aiActionContext: 'context',      // AI behavior context
-
-  // Reporting
-  generateReport: true,            // Generate HTML reports
-  autoPrintReportMsg: true,        // Print report messages
-
-  // Navigation
-  forceSameTabNavigation: true,    // Force same-tab navigation
-  waitForNavigationTimeout: 30000, // Navigation timeout
-  waitForNetworkIdleTimeout: 5000  // Network idle timeout
-});
-```
-
-## 📊 Context Management
-
-### Direct Context Access
-
-```javascript
-// Get stored data
-const storedData = agent.getStoredData();
-console.log('Stored:', storedData);
-
-// Get step summary
-const summary = agent.getStepSummary();
-console.log('Steps:', summary);
-
-// Clear context manually
-agent.clearContext();
-
-// Get context store instance
-const contextStore = agent.getContextStore();
-```
-
-### Manual Data Storage
-
-```javascript
-const contextStore = agent.getContextStore();
-
-// Store data manually
-contextStore.storeData('customKey', 'customValue');
-
-// Store with aliases for natural language
-contextStore.storeDataWithAliases('productName', 'iPhone 15', ['urun', 'product']);
-
-// Get recent steps
-const recentSteps = contextStore.getRecentSteps(5);
-```
-
-## 🔍 Debug Logging
-
-Enable comprehensive logging to see context operations:
-
-```bash
-# All debug logs
-DEBUG=midscene:* node your-script.js
-
-# Context-specific logs
-DEBUG=midscene:agent node your-script.js
-
-# AI call logs
-DEBUG=midscene:ai:*,midscene:agent node your-script.js
-```
-
-### Log Examples
-
-```
-DEBUG midscene:agent Context replacement in aiAction: {
-  original: "type username in search box",
-  processed: "type john_doe in search box",
-  storedData: { "username": "john_doe" }
-}
-
-DEBUG midscene:agent Stored query result with aliases: {
-  key: "username",
-  value: "john_doe",
-  aliases: ["username", "user", "kullanici"]
-}
-```
-
-## 🎭 Multi-Step Workflow Example
-
-```javascript
-// E-commerce automation with context
-const agent = new PuppeteerAgent(page, {
-  enableCumulativeContext: true,
-  autoClearContext: true
-});
-
-await page.goto('https://shop.example.com');
-
-// Step 1: Search and store product info
-await agent.aiAction('arama kutusuna "laptop" yaz ve ara');
-await agent.aiQuery('ilk ürünün adını urun_adi olarak kaydet');
-await agent.aiQuery('ilk ürünün fiyatını urun_fiyati olarak kaydet');
-
-// Step 2: Add to cart using stored data
-await agent.aiAction('urun_adi ürününü sepete ekle');
-
-// Step 3: Verify cart contents
-await agent.aiAction('sepete git');
-await agent.aiAssert('sepette urun_adi ürünü urun_fiyati fiyatıyla görünüyor');
-
-// Step 4: Proceed to checkout
-await agent.aiAction('ödeme sayfasına git');
-await agent.aiAssert('ödeme sayfasında urun_adi ürünü listeleniyor');
-```
-
-## 🔄 Backward Compatibility
-
-The new context system is fully backward compatible:
-
-```javascript
-// Old syntax still works
-await agent.aiQuery('extract username, store as user');
-await agent.aiAction('type {stored.user} in field');
-
-// New syntax (recommended)
-await agent.aiQuery('kullanıcı adını user olarak kaydet');
-await agent.aiAction('alana user verisini yaz');
-
-// Mixed usage
-await agent.aiAction('use {stored.email} and user verisi together');
-```
-
-## 📚 API Reference
-
-### Core Methods
-
-- `aiAction(prompt)` - Perform actions with context awareness
-- `aiQuery(prompt)` - Extract data with automatic storage
-- `aiAssert(assertion)` - Context-aware assertions
-- `aiCaptcha(options)` - Intelligent CAPTCHA solving
-- `aiTap(target, options)` - Click elements
-- `aiInput(text, target, options)` - Input text
-
-### Context Methods
-
-- `getStoredData()` - Get all stored data
-- `getStepSummary()` - Get step summary
-- `clearContext()` - Clear context store
-- `getContextStore()` - Get context store instance
-
-## 🤝 Contributing
-
-We welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) for details.
-
-## 📄 License
-
-MIT License - see [LICENSE](LICENSE) file for details.
-
-## 🔗 Links
-
-- [Documentation](https://midscenejs.com)
-- [GitHub Repository](https://github.com/web-infra-dev/midscene)
-- [NPM Package](https://www.npmjs.com/package/@midscene/web)
-- [Examples](https://github.com/web-infra-dev/midscene/tree/main/packages/web-integration/examples)
-
----
-
-**Made with ❤️ by the Midscene Team**
+## Documentation
+
+Automate UI actions, extract data, and perform assertions using AI. It offers JavaScript SDK, Chrome extension, and support for scripting in YAML.
+
+See https://midscenejs.com/ for details.
+
+## License
+
+Midscene is MIT licensed.

package/bin/midscene-playground

@@ -1,3 +1,3 @@
-#!/usr/bin/env node
-
+#!/usr/bin/env node
+
 require('../dist/lib/midscene-playground.js');

package/dist/es/agent.js

@@ -17,10 +17,11 @@ var ScriptPlayer = class {
     this.unnamedResultIndex = 0;
     this.pageAgent = null;
     this.result = {};
+    const target = script.target || script.web || script.android;
     if (ifInBrowser) {
       this.output = void 0;
-    } else if (script.target?.output) {
-      this.output = resolve(process.cwd(), script.target.output);
+    } else if (target?.output) {
+      this.output = resolve(process.cwd(), target.output);
     } else {
       this.output = join(getMidsceneRunSubDir("output"), `${process.pid}.json`);
     }
@@ -94,12 +95,13 @@ var ScriptPlayer = class {
       } else if ("aiAssert" in flowItem) {
         const assertTask = flowItem;
         const prompt = assertTask.aiAssert;
+        const msg = assertTask.errorMessage;
         assert(prompt, "missing prompt for aiAssert");
         assert(
           typeof prompt === "string",
           "prompt for aiAssert must be a string"
         );
-        await agent.aiAssert(prompt);
+        await agent.aiAssert(prompt, msg);
       } else if ("aiQuery" in flowItem) {
         const queryTask = flowItem;
         const prompt = queryTask.aiQuery;
@@ -1660,7 +1662,7 @@ import yaml3 from "js-yaml";
 import semver from "semver";
 
 // package.json
-var version = "1.0.5";
+var version = "1.5.6";
 
 // src/common/task-cache.ts
 var debug3 = getDebug3("cache");
@@ -1842,13 +1844,10 @@ var PageAgent = class {
         generateReport: true,
         autoPrintReportMsg: true,
         groupName: "Midscene Report",
-        groupDescription: "",
-        enableCumulativeContext: true,
-        autoClearContext: false
+        groupDescription: ""
       },
       opts || {}
     );
-    this.initializeContextStore();
     if (this.page.pageType === "puppeteer" || this.page.pageType === "playwright") {
       this.page.waitForNavigationTimeout = this.opts.waitForNavigationTimeout || DEFAULT_WAIT_FOR_NAVIGATION_TIMEOUT;
       this.page.waitForNetworkIdleTimeout = this.opts.waitForNetworkIdleTimeout || DEFAULT_WAIT_FOR_NETWORK_IDLE_TIMEOUT;
@@ -1875,69 +1874,6 @@ var PageAgent = class {
       opts?.testId || this.page.pageType || "web"
     );
   }
-  /**
-   * Initialize context store for cumulative context functionality
-   */
-  async initializeContextStore() {
-    if (!this.opts.enableCumulativeContext) {
-      debug4("Cumulative context disabled via options");
-      return;
-    }
-    try {
-      const aiModel = await import("misoai-core/ai-model");
-      this.contextStore = aiModel.getContextStore();
-      debug4("Context store initialized successfully", {
-        autoClearContext: this.opts.autoClearContext,
-        testId: this.opts.testId
-      });
-      if (this.opts.autoClearContext) {
-        this.contextStore.clear();
-        debug4("Context store cleared due to autoClearContext option");
-      } else {
-        const existingData = this.contextStore.getAllData();
-        const existingSteps = this.contextStore.getRecentSteps(100).length;
-        debug4("Context store preserving existing data", {
-          existingDataKeys: Object.keys(existingData),
-          existingStepsCount: existingSteps
-        });
-      }
-    } catch (error) {
-      debug4("Failed to initialize context store:", error);
-      console.warn("⚠️ Could not initialize context store:", error);
-    }
-  }
-  /**
-   * Get the context store instance
-   */
-  getContextStore() {
-    return this.contextStore;
-  }
-  /**
-   * Clear the context store
-   */
-  clearContext() {
-    if (this.contextStore) {
-      this.contextStore.clear();
-    }
-  }
-  /**
-   * Get all stored data from context store
-   */
-  getStoredData() {
-    if (this.contextStore) {
-      return this.contextStore.getAllData();
-    }
-    return {};
-  }
-  /**
-   * Get step summary from context store
-   */
-  getStepSummary() {
-    if (this.contextStore) {
-      return this.contextStore.getStepSummary();
-    }
-    return "";
-  }
   async getUIContext(action) {
     if (action && (action === "extract" || action === "assert" || action === "captcha")) {
       return await parseContextFromWebPage(this.page, {
@@ -2173,31 +2109,6 @@ var PageAgent = class {
     };
   }
   async aiAction(taskPrompt, opt) {
-    if (this.opts.enableCumulativeContext && this.contextStore) {
-      try {
-        const originalPrompt = taskPrompt;
-        const processedPrompt = this.contextStore.replaceAllReferences(taskPrompt, "action");
-        if (originalPrompt !== processedPrompt) {
-          debug4("Context replacement in aiAction:", {
-            original: originalPrompt,
-            processed: processedPrompt,
-            storedData: this.contextStore.getAllData()
-          });
-        }
-        this.contextStore.addStep({
-          type: "action",
-          summary: `Action: ${processedPrompt}`,
-          prompt: processedPrompt
-        });
-        debug4("Added action step to context store:", {
-          stepNumber: this.contextStore.getRecentSteps(1)[0]?.stepNumber,
-          totalSteps: this.contextStore.getRecentSteps(100).length
-        });
-        taskPrompt = processedPrompt;
-      } catch (error) {
-        debug4("Context store operation failed:", error);
-      }
-    }
     const cacheable = opt?.cacheable;
     const isVlmUiTars = vlLocateMode() === "vlm-ui-tars";
     const matchedCache = isVlmUiTars || cacheable === false ? void 0 : this.taskCache?.matchPlanCache(taskPrompt);
@@ -2244,75 +2155,7 @@ var PageAgent = class {
     };
   }
   async aiQuery(demand) {
-    let processedDemand = demand;
-    let storageKey;
-    try {
-      const aiModel = await import("misoai-core/ai-model");
-      const contextStore = aiModel.getContextStore();
-      if (typeof demand === "string") {
-        const storageInstruction = contextStore.parseStorageInstruction(demand);
-        if (storageInstruction) {
-          storageKey = storageInstruction.key;
-          processedDemand = storageInstruction.cleanText;
-          contextStore._pendingAliases = storageInstruction.aliases;
-        } else {
-          const storageMatch = demand.match(/store\s+(?:as\s+)?(\w+)/i);
-          if (storageMatch) {
-            storageKey = storageMatch[1];
-            processedDemand = demand.replace(/,?\s*store\s+(?:as\s+)?\w+/i, "").trim();
-          }
-        }
-      }
-    } catch (error) {
-      debug4("Context store not available:", error);
-    }
-    const { output, executor } = await this.taskExecutor.query(processedDemand);
-    if (this.opts.enableCumulativeContext && this.contextStore) {
-      if (storageKey && output) {
-        try {
-          const pendingAliases = this.contextStore._pendingAliases;
-          if (pendingAliases) {
-            this.contextStore.storeDataWithAliases(storageKey, output, pendingAliases, typeof processedDemand === "string" ? processedDemand : JSON.stringify(processedDemand));
-            delete this.contextStore._pendingAliases;
-            debug4("Stored query result with aliases:", {
-              key: storageKey,
-              value: output,
-              aliases: pendingAliases
-            });
-          } else {
-            this.contextStore.storeData(storageKey, output);
-            debug4("Stored query result:", {
-              key: storageKey,
-              value: output
-            });
-          }
-          this.contextStore.addStep({
-            type: "query",
-            summary: `Query: ${typeof processedDemand === "string" ? processedDemand : JSON.stringify(processedDemand)} (stored as ${storageKey})`,
-            data: output,
-            prompt: typeof processedDemand === "string" ? processedDemand : JSON.stringify(processedDemand)
-          });
-          debug4("Added query step to context store:", {
-            storageKey,
-            totalStoredItems: Object.keys(this.contextStore.getAllData()).length,
-            totalSteps: this.contextStore.getRecentSteps(100).length
-          });
-        } catch (error) {
-          debug4("Failed to store query result:", error);
-        }
-      } else {
-        try {
-          this.contextStore.addStep({
-            type: "query",
-            summary: `Query: ${typeof processedDemand === "string" ? processedDemand : JSON.stringify(processedDemand)}`,
-            data: output,
-            prompt: typeof processedDemand === "string" ? processedDemand : JSON.stringify(processedDemand)
-          });
-        } catch (error) {
-          debug4("Failed to add query step:", error);
-        }
-      }
-    }
+    const { output, executor } = await this.taskExecutor.query(demand);
     const metadata = this.afterTaskRunning(executor);
     return {
       result: output,
@@ -2422,31 +2265,6 @@ var PageAgent = class {
     };
   }
   async aiAssert(assertion, msg, opt) {
-    let processedAssertion = assertion;
-    if (this.opts.enableCumulativeContext && this.contextStore) {
-      try {
-        const originalAssertion = assertion;
-        processedAssertion = this.contextStore.replaceAllReferences(assertion, "assertion");
-        if (originalAssertion !== processedAssertion) {
-          debug4("Context replacement in aiAssert:", {
-            original: originalAssertion,
-            processed: processedAssertion,
-            context: "assertion",
-            storedData: this.contextStore.getAllData()
-          });
-        }
-        this.contextStore.addStep({
-          type: "assertion",
-          summary: `Assertion: ${processedAssertion}`,
-          prompt: processedAssertion
-        });
-        debug4("Added assertion step to context store:", {
-          totalSteps: this.contextStore.getRecentSteps(100).length
-        });
-      } catch (error) {
-        debug4("Context store operation failed:", error);
-      }
-    }
     let currentUrl = "";
     if (this.page.url) {
       try {
@@ -2454,7 +2272,7 @@ var PageAgent = class {
       } catch (e) {
       }
     }
-    const assertionWithContext = currentUrl ? `For the page at URL "${currentUrl}", ${processedAssertion}` : processedAssertion;
+    const assertionWithContext = currentUrl ? `For the page at URL "${currentUrl}", ${assertion}` : assertion;
     const { output, executor } = await this.taskExecutor.assert(assertionWithContext);
     const metadata = this.afterTaskRunning(executor, true);
     if (output && opt?.keepRawResponse) {