@isdk/web-fetcher 0.2.12 → 0.3.1

package/README.action.md

@@ -90,6 +90,8 @@ Navigates the browser to a new URL.
   * ...other navigation options like `waitUntil`, `timeout` which are passed to the engine.
 * **`returns`**: `response`
 
+> **Note**: This action can be called multiple times within a single session script. The engine ensures that each navigation is completed and its corresponding Action Loop is settled before processing the next action in the sequence.
+
 #### `click`
 
 Clicks on an element specified by a selector.
@@ -123,6 +125,41 @@ Submits a form.
   * `selector` (string, optional): A selector for the form element.
 * **`returns`**: `none`
 
+#### `trim`
+
+Removes specific elements from the DOM to clean up the page before extraction. This is a persistent modification to the current session's page state.
+
+* **`id`**: `trim`
+* **`params`**:
+  * **`selectors`** (string | string[], optional): One or more CSS selectors of elements to remove.
+  * **`presets`** (string | string[], optional): Predefined groups of elements to remove. Supported presets:
+    * `scripts`: Removes all `<script>` tags.
+    * `styles`: Removes all `<style>` and `<link rel="stylesheet">` tags.
+    * `svgs`: Removes all `<svg>` elements.
+    * `images`: Removes `<img>`, `<picture>`, and `<canvas>` elements.
+    * `comments`: Removes HTML comments.
+    * `hidden`: Removes elements with `hidden` attribute or inline `display:none`. In **browser** mode, it also detects and removes elements hidden via external CSS (e.g., `display: none` or `visibility: hidden` in stylesheets).
+    * `all`: Includes all of the above.
+* **`returns`**: `none`
+
+**Example: Cleaning up a page before extraction**
+
+```json
+{
+  "actions": [
+    { "action": "goto", "params": { "url": "https://example.com" } },
+    {
+      "action": "trim",
+      "params": {
+        "selectors": ["#ad-banner", ".popup"],
+        "presets": ["scripts", "styles", "comments"]
+      }
+    },
+    { "action": "extract", "params": { "schema": { "content": "#main-content" } } }
+  ]
+}
+```
+
 #### `waitFor`
 
 Pauses execution to wait for one or more conditions to be met.
@@ -210,198 +247,122 @@ Retrieves the full content of the current page state.
 * **`params`**: (none)
 * **`returns`**: `response`
 
-#### `extract`
-
-Extracts structured data from the page using a powerful and declarative Schema. This is the core Action for data collection.
-
-* **`id`**: `extract`
-* **`params`**: An `ExtractSchema` object that defines the extraction rules.
-* **`returns`**: `any` (the extracted data)
-
-##### Detailed Explanation of Extraction Schema
+#### `mouseMove`
 
-The `params` object itself is a Schema that describes the data structure you want to extract.
-
-###### 1. Extracting a Single Value
-
-The most basic extraction. You can specify a `selector` (CSS selector), an `attribute` (the name of the attribute to extract), a `type` (string, number, boolean, html), and a `mode` (text, innerText).
-
-```json
-{
-  "id": "extract",
-  "params": {
-    "selector": "h1.main-title",
-    "type": "string",
-    "mode": "innerText"
-  }
-}
-```
+Moves the mouse cursor to a specific coordinate or element. In `browser` mode, it uses a **Bézier curve** to simulate a human-like non-linear trajectory with slight jitter for realism.
 
-> **Extraction Modes:**
->
-> * **`text`** (default): Extracts the `textContent` of the element.
-> * **`innerText`**: Extracts the rendered text, respecting CSS styling and line breaks.
-> * **`html`**: Returns the `innerHTML` of the element.
-> * **`outerHTML`**: Returns the HTML including the tag itself. Useful for preserving the full element structure.
-> The example above will extract the text content of the `<h1>` tag with the class `main-title` using the `innerText` mode.
+* **`id`**: `mouseMove`
+* **`params`**:
+  * `x` (number, optional): The absolute X coordinate.
+  * `y` (number, optional): The absolute Y coordinate.
+  * `selector` (string, optional): A CSS selector. If provided, the mouse moves to the center of the element.
+  * `steps` (number, optional): The number of intermediate steps for the trajectory (default: `-1`). Set to `-1` to calculate steps automatically based on distance (simulating natural speed).
+* **`returns`**: `none`
 
-###### 2. Extracting an Object
+#### `mouseClick`
 
-Define a structured object using `type: 'object'` and the `properties` field.
+Triggers a mouse click at the current position or specified coordinates. If a `selector` is provided, the cursor will first move smoothly to the target element (using dynamic steps) before clicking.
 
-```json
-{
-  "id": "extract",
-  "params": {
-    "type": "object",
-    "selector": ".author-bio",
-    "properties": {
-      "name": { "selector": ".author-name" },
-      "email": { "selector": "a.email", "attribute": "href" }
-    }
-  }
-}
-```
+* **`id`**: `mouseClick`
+* **`params`**:
+  * `x` (number, optional): The absolute X coordinate to click.
+  * `y` (number, optional): The absolute Y coordinate to click.
+  * `selector` (string, optional): A CSS selector. If provided, moves the mouse to the element first.
+  * `button` (string, optional): The mouse button to use (`left`, `right`, or `middle`). Default is `left`.
+  * `clickCount` (number, optional): The number of clicks (e.g., 2 for double-click). Default is 1.
+  * `delay` (number, optional): Delay between mousedown and mouseup in milliseconds.
+* **`returns`**: `none`
 
-###### 3. Extracting an Array (Convenient Usage)
+#### `keyboardType`
 
-Extract a list using `type: 'array'`. To make the most common operations simpler, we provide some convenient usages.
+Simulates a person typing text into the currently focused element.
 
-* **Extracting an Array of Texts (Default Behavior)**: When you want to extract a list of text, just provide the selector and omit `items`. This is the most common usage.
+* **`id`**: `keyboardType`
+* **`params`**:
+  * `text` (string): The text to type.
+  * `delay` (number, optional): The delay between key presses in milliseconds (default: 100).
+* **`returns`**: `none`
 
-    ```json
-    {
-      "id": "extract",
-      "params": {
-        "type": "array",
-        "selector": ".tags li"
-      }
-    }
-    ```
+#### `keyboardPress`
 
-    > The example above will return an array of the text from all `<li>` tags, e.g., `["tech", "news"]`.
+Simulates pressing a single key or a key combination (e.g., `Enter`, `Control+A`).
 
-* **Extracting an Array of Attributes (Shortcut)**: When you only want to extract a list of attributes (e.g., all `href`s from links), there's no need to nest `items` either. Just declare `attribute` directly in the `array` definition.
+* **`id`**: `keyboardPress`
+* **`params`**:
+  * `key` (string): The name of the key to press (e.g., `Enter`, `Tab`, `Backspace`, `ArrowUp`).
+  * `delay` (number, optional): The delay after the key press in milliseconds.
+* **`returns`**: `none`
 
-    ```json
-    {
-      "id": "extract",
-      "params": {
-        "type": "array",
-        "selector": ".gallery img",
-        "attribute": "src"
-      }
-    }
-    ```
+#### `evaluate`
 
-    > The example above will return an array of the `src` attributes from all `<img>` tags.
+Executes custom JavaScript code or an expression within the page context.
 
-* **Array Extraction Modes**: When extracting an array, the engine supports different modes to handle various DOM structures.
+* **`id`**: `evaluate`
+* **`params`**:
+  * **`fn`**: `string | Function` - The JavaScript function or expression to execute.
+    * In `browser` mode, this runs in the real browser.
+    * In `http` mode, this runs in a safe Node.js sandbox with a mocked browser environment (`window`, `document`, `console`, `).
+  * **`args`**: `any` - A single argument to pass to the function. Use an array or object to pass multiple values.
 
-  * **`nested`** (Default): The `selector` matches individual item wrappers.
-  * **`columnar`** (formerly Zip): The `selector` matches a **container**, and fields in `items` are parallel columns stitched together by index.
-  * **`segmented`**: The `selector` matches a **container**, and items are segmented by an "anchor" field.
+**Key Features:**
 
-###### 4. Columnar Mode (formerly Zip Strategy)
+- **Automatic Navigation**: If the code modifies `window.location.href` or calls `assign()`/`replace()`, the engine automatically triggers and waits for the navigation to complete.
+- **Enhanced Mock DOM (HTTP Mode)**: Supports common DOM methods like `querySelector`, `querySelectorAll`, `getElementById`, `getElementsByClassName`, and properties like `document.body` and `document.title`.
+- **Sandbox Security**: Uses `util-ex`'s `newFunction` in HTTP mode to prevent global state pollution.
 
-This mode is used when the `selector` points to a **container** (like a results list) and item data is scattered as separate columns.
+**Example (Array Args):**
 
 ```json
 {
-  "id": "extract",
+  "action": "evaluate",
   "params": {
-    "type": "array",
-    "selector": "#search-results",
-    "mode": "columnar",
-    "items": {
-      "title": { "selector": ".item-title" },
-      "link": { "selector": "a.item-link", "attribute": "href" }
-    }
+    "fn": "([a, b]) => a + b",
+    "args": [1, 2]
   }
 }
 ```
 
-> **Heuristic Detection:** If `mode` is omitted and the `selector` matches exactly one element while `items` contains nested selectors, the engine automatically uses **columnar** mode.
-
-**Columnar Configuration:**
-
-* **`strict`** (boolean, default: `true`): If `true`, throws an error if fields have different match counts.
-* **`inference`** (boolean, default: `false`): If `true`, tries to automatically find the "item wrapper" elements to fix misaligned lists.
-
-###### 5. Segmented Mode (Anchor-based Scanning)
-
-This mode is ideal for "flat" structures where there are no item wrappers. It uses the first field (or a specified `anchor`) to segment the container's content.
+**Example (Navigation):**
 
 ```json
 {
-  "id": "extract",
+  "action": "evaluate",
   "params": {
-    "type": "array",
-    "selector": "#flat-container",
-    "mode": { "type": "segmented", "anchor": "title" },
-    "items": {
-      "title": { "selector": "h3" },
-      "desc": { "selector": "p" }
-    }
+    "fn": "() => { window.location.href = '/new-page'; }"
   }
 }
 ```
 
-> In this mode, every time the engine finds an `h3`, it starts a new item. Any `<p>` found after that `h3` (and before the next one) is assigned to that item.
-
-###### 6. Implicit Object Extraction (Simplest Syntax)
+$`).
+>
+> * **Navigation**: If the code modifies `window.location.href`, the engine will automatically trigger a `goto` to the new URL.
 
-For simpler object extraction, you can omit `type: 'object'` and `properties`. If the schema object contains keys that are not reserved keywords (like `selector`, `attribute`, `type`, etc.), it is treated as an object schema where keys are property names.
+**Example: Extracting data via custom script**
 
 ```json
 {
-  "id": "extract",
+  "action": "evaluate",
   "params": {
-    "selector": ".author-bio",
-    "name": { "selector": ".author-name" },
-    "email": { "selector": "a.email", "attribute": "href" }
-  }
+    "fn": "([selector]) => document.querySelector(selector).innerText",
+    "args": [".main-title"]
+  },
+  "storeAs": "title"
 }
 ```
 
-> This is equivalent to Example 2 but more concise.
-
-###### 6. Precise Filtering: `has` and `exclude`
-
-You can use the `has` and `exclude` fields in any schema that includes a `selector` to precisely control element selection.
-
-* `has`: A CSS selector to ensure the selected element **must contain** a descendant matching this selector.
-* `exclude`: A CSS selector to **exclude** elements matching this selector from the results.
+#### `extract`
 
-**Complete Example: Extracting links of articles that have an image and are not marked as "draft"**
+Extracts structured data from the page using a powerful and declarative Schema.
 
-```json
-{
-  "actions": [
-    { "id": "goto", "params": { "url": "https://example.com/articles" } },
-    {
-      "id": "extract",
-      "params": {
-        "type": "array",
-        "selector": "div.article-card",
-        "has": "img.cover-image",
-        "exclude": ".draft",
-        "items": {
-          "selector": "a.title-link",
-          "attribute": "href"
-        }
-      }
-    }
-  ]
-}
-```
+* **`id`**: `extract`
+* **`params`**: An `ExtractSchema` object.
+* **`returns`**: The extracted structured data.
 
-> The `extract` action above will:
+> **📚 Detailed Manual**: Because `extract` is very rich in features (including array modes, scope control, anchor jumping, etc.), we have prepared a dedicated detailed manual:
 >
-> 1. Find all `div.article-card` elements.
-> 2. Filter them to only include those that contain an `<img class="cover-image">`.
-> 3. Further filter the results to exclude any that also have the `.draft` class.
-> 4. For each of the remaining `div.article-card` elements, find its descendant `a.title-link` and extract the `href` attribute.
+> 👉 **[Click to View Extract Action Deep Dive](./README.action.extract.md)**
+
+---
 
 ### Building High-Level Semantic Actions via "Composition"
 
@@ -546,3 +507,97 @@ The `FetchAction` base class provides lifecycle hooks that allow injecting custo
 * `protected onAfterExec?()`: Called after `onExecute`.
 
 For Actions that need to manage complex state or resources, you can implement these hooks. Generally, for composite actions, writing the logic directly in `onExecute` is sufficient.
+
+---
+
+## 💎 7. Action Return Types & State Management (Advanced)
+
+In `@isdk/web-fetcher`, an Action's `static returnType` is more than a type hint. It defines how the framework manages the **session state** and automates data synchronization after execution.
+
+### 7.1 Detailed Type Breakdown
+
+#### 🟢 `response` (Page Response)
+
+* **Definition**: A `FetchResponse` object containing HTTP status, headers, body, and cookies.
+* **Purpose**: To synchronize the latest page content and state with the session.
+* **Usage**: Used by actions that navigate, refresh, or capture the current page state.
+* **System Behavior**: The framework automatically updates `context.lastResponse` with this result in the `afterExec` phase. Subsequent actions can access this via the context.
+* **Typical Actions**: `goto`, `getContent`, `fill` (in some engines).
+* **Example**:
+
+  ```typescript
+  export class MyNavigateAction extends FetchAction {
+    static override id = 'myGoto';
+    static override returnType = 'response' as const;
+
+    async onExecute(context, options) {
+      // Logic that returns a FetchResponse
+      return await this.delegateToEngine(context, 'goto', options.params.url);
+    }
+  }
+  ```
+
+#### 🟡 `any` (Generic Data - Default)
+
+* **Definition**: Any serializable data structure (Object, Array, string, etc.).
+* **Purpose**: Primary mechanism for business data extraction.
+* **Usage**: Use this when your action produces processed data that doesn't represent the whole page or system state.
+* **System Behavior**: If the action configuration includes `storeAs: "key"`, the framework automatically saves the `result` into `context.outputs["key"]`. If the target key already contains an object and the new result is also an object, they will be merged (shallow merge) instead of overwritten. This allows multiple `extract` actions to accumulate data into the same output key.
+* **Typical Actions**: `extract`.
+* **Example**:
+
+  ```typescript
+  static override returnType = 'any' as const;
+  async onExecute(context, options) {
+    return { title: 'Hello', price: 99 }; // Saved to outputs if storeAs is set
+  }
+  ```
+
+#### ⚪ `none` (No Return)
+
+* **Definition**: `void`.
+* **Purpose**: Pure interaction/side-effects without data output.
+* **Usage**: Actions that perform UI interactions or timing controls.
+* **Typical Actions**: `click`, `submit`, `pause`, `trim`, `waitFor`.
+* **Example**:
+
+  ```typescript
+  static override returnType = 'none' as const;
+  async onExecute(context, options) {
+    await this.delegateToEngine(context, 'click', options.params.selector);
+    // No return value needed
+  }
+  ```
+
+#### 🔵 `outputs` (Accumulated Results)
+
+* **Definition**: The entire `context.outputs` record (`Record<string, any>`).
+* **Purpose**: To retrieve all data extracted and stored during the current session.
+* **Usage**: Typically used as a "summary" action at the end of a chain or for debugging.
+* **Typical Actions**: Custom data summary actions.
+
+#### 🟣 `context` (Session Snapshot)
+
+* **Definition**: The full `FetchContext` object.
+* **Purpose**: Meta-programming and deep debugging.
+* **Usage**: Allows the caller to inspect current session configurations (timeouts, proxies, headers) and internal engine metadata.
+
+---
+
+### 7.2 Result Wrapping (`FetchActionResult`)
+
+Every value returned by `onExecute` is automatically wrapped by the `FetchAction.execute` method into a `FetchActionResult` object. This ensures consistent error handling and metadata tracking across all actions.
+
+**Structure of `FetchActionResult`:**
+
+* `status`: `Success`, `Failed`, or `Skipped`.
+* `returnType`: Matches the action's `static returnType`.
+* `result`: The raw data returned by `onExecute`.
+* `error`: Captured error if the action failed.
+* `meta`: Diagnostic information including execution time, engine type, and retry counts.
+
+### 7.3 Developer Best Practices
+
+1. **Choose `response` for Navigation**: Always use `response` for actions that land on a new URL to ensure the session's "current page" is kept in sync.
+2. **Leverage `any` + `storeAs`**: For data extraction, return the data as `any` and let the user decide the storage key via `storeAs` in the JSON script.
+3. **Be Explicit with `none`**: Using `none` clearly signals that the action is used for its side effects (like clicking or waiting), making the workflow easier to understand.

package/README.cn.md

@@ -20,9 +20,10 @@
 * **📜 声明式动作脚本**: 以简单、可读的 JSON 格式定义多步骤工作流（如登录、填写表单、点击按钮等）。
 * **📊 强大而灵活的数据提取**: 通过直观、强大的声明式 Schema,轻松提取从简单文本到复杂嵌套的各类结构化数据。
 * **🧠 智能引擎选择**: 可自动检测动态站点，并在需要时将引擎从 `http` 动态升级到 `browser`。
+* **🛡️ 反爬虫/反屏蔽**: 在 `browser` 模式下，一个可选的 `antibot` 标志有助于绕过常见的反机器人措施，如 Cloudflare 挑战。
+* **🕹️ 高仿真交互模拟**: 支持基于 **贝塞尔曲线** 的鼠标轨迹移动、真实的打字延迟模拟，以及复杂的键盘交互，大幅提升反爬避障能力。
 * **🧩 可扩展性**: 轻松创建自定义的、高级别的“组合动作”，以封装可复用的业务逻辑（例如，一个 `login` 动作）。
 * **🧲 高级收集器 (Collectors)**: 在主动作执行期间，由事件触发，在后台异步收集数据。
-* **🛡️ 反爬虫/反屏蔽**: 在 `browser` 模式下，一个可选的 `antibot` 标志有助于绕过常见的反机器人措施，如 Cloudflare 挑战。
 
 ---
 
@@ -132,7 +133,7 @@ searchGoogle('gemini');
 * `url` (string): 要导航的初始 URL。
 * `engine` ('http' | 'browser' | 'auto'): 要使用的引擎。默认为 `auto`。
 * `proxy` (string | string[]): 用于请求的代理 URL。
-* `debug` (boolean): 在响应中启用详细的执行元数据（耗时、使用的引擎等）。
+* `debug` (boolean | string | string[]): 在响应中启用详细的执行元数据（耗时、使用的引擎等），或启用特定类别（如 'extract', 'submit', 'request'）的调试日志。
 * `actions` (FetchActionOptions[]): 要执行的动作对象数组。（支持 `action`/`name` 作为 `id` 的别名，`args` 作为 `params` 的别名）
 * `headers` (Record<string, string>): 用于所有请求的头信息。
 * `cookies` (Cookie[]): 要使用的 Cookie 数组。
@@ -145,21 +146,30 @@ searchGoogle('gemini');
 * `output` (object): 控制 `FetchResponse` 中的输出字段。
   * `cookies` (boolean): 是否在响应中包含 Cookie（默认：`true`）。
   * `sessionState` (boolean): 是否在响应中包含会话状态（默认：`true`）。
+* `browser` (object): 浏览器引擎配置。
+  * `headless` (boolean): 是否以无头模式运行（默认：`true`）。
+  * `launchOptions` (object): Playwright 启动选项（例如 `{ slowMo: 50, args: [...] }`）。
 * `sessionPoolOptions` (SessionPoolOptions): 底层 Crawlee SessionPool 的高级配置。
 * ...以及许多其他用于代理、重试等的选项。
 
-### 内置动作
-
-以下是核心的内置动作：
-
-* `goto`: 导航到一个新的 URL。
-* `click`: 点击一个由选择器指定的元素。
-* `fill`: 用指定的值填充一个输入字段。
-* `submit`: 提交一个表单。
-* `waitFor`: 暂停执行以等待特定条件（例如，超时、选择器出现或网络空闲）。
-* `pause`: 暂停执行以进行手动干预（例如，解决验证码）。
-* `getContent`: 获取当前页面状态的完整内容（HTML、文本等）。
-* `extract`: 使用富有表现力的声明式 Schema,可轻松提取页面中的任意结构化数据。
+### 内置动作 (Built-in Actions)
+
+该库提供了一系列强大的内置动作，其中许多动作是跨引擎通用的，并由核心层统一处理以确保一致性：
+
+* `goto`: 导航到新 URL。
+* `click`: 点击选择器指定的元素（引擎相关）。
+* `fill`: 用指定值填充输入框（引擎相关）。
+* `submit`: 提交表单（引擎相关）。
+* `mouseMove`: 将鼠标指针移动到指定的坐标或元素（支持贝塞尔曲线）。
+* `mouseClick`: 在当前位置或指定坐标触发鼠标点击。
+* `keyboardType`: 模拟真人在当前获得焦点的元素中输入文本。
+* `keyboardPress`: 模拟按下单个按键或组合键。
+* `trim`: 从 DOM 中移除元素以清理页面（如脚本、广告、隐藏内容）。
+* `waitFor`: 暂停执行以等待特定条件（支持统一处理的固定超时）。
+* `pause`: 暂停执行以进行人工干预（如处理验证码，由核心层统一处理）。
+* `getContent`: 获取当前页面状态的完整内容（由核心层统一处理）。
+* `evaluate`: 在页面上下文中执行自定义 JavaScript。
+* `extract`: 使用引擎无关的核心逻辑和引擎相关的 DOM 原语提取结构化数据。支持 `required` 字段和 `strict` 验证。
 
 ### 响应结构
 
@@ -172,7 +182,7 @@ searchGoogle('gemini');
   * `cookies`: Cookie 数组。
   * `sessionState`: Crawlee 会话状态。
   * `text`, `html`: 页面内容。
-* `outputs` (Record<string, any>): 通过 `storeAs` 提取并存储的数据。
+* `outputs` (Record<string, any>): 通过 `storeAs` 提取并存储的数据。注意：当多个动作将对象存储到同一个键时，它们将被合并而不再是覆盖。
 
 ---