@isdk/web-fetcher 0.3.1 → 0.3.3

package/README.action.cn.md

@@ -251,8 +251,8 @@ await fetchWeb({
 
 * **`id`**: `mouseMove`
 * **`params`**:
-  * `x` (number, 可选): 绝对 X 坐标。
-  * `y` (number, 可选): 绝对 Y 坐标。
+  * `x` (number, 可选): 绝对 X 坐标。如果为负数，则被视为相对于当前位置的随机偏移。
+  * `y` (number, 可选): 绝对 Y 坐标。如果为负数，则被视为相对于当前位置的随机偏移。
   * `selector` (string, 可选): CSS 选择器。如果提供，鼠标将移动到该元素的中心。
   * `steps` (number, 可选): 轨迹的中间步数（默认：`-1`）。设置为 `-1` 可根据距离动态计算步数（模拟自然移动速度）。
 * **`returns`**: `none`
@@ -263,14 +263,38 @@ await fetchWeb({
 
 * **`id`**: `mouseClick`
 * **`params`**:
-  * `x` (number, 可选): 点击的绝对 X 坐标。
-  * `y` (number, 可选): 点击的绝对 Y 坐标。
+  * `x` (number, 可选): 要点击的绝对 X 坐标。如果为负数，则被视为相对于当前位置的随机偏移。
+  * `y` (number, 可选): 要点击的绝对 Y 坐标。如果为负数，则被视为相对于当前位置的随机偏移。
+
   * `selector` (string, 可选): CSS 选择器。如果提供，鼠标会先移动到该元素。
   * `button` (string, 可选): 使用的鼠标按键 (`left`, `right`, 或 `middle`)。默认为 `left`。
   * `clickCount` (number, 可选): 点击次数（例如：2 表示双击）。默认为 1。
   * `delay` (number, 可选): mousedown 和 mouseup 之间的延迟（毫秒）。
 * **`returns`**: `none`
 
+#### `mouseWheel`
+
+模拟鼠标滚轮滚动事件。如果提供了 `selector`，元素会自动滚动到视口中，并且在滚动前将鼠标指针移动到其中心。如果提供了 `steps`，滚动增量将被拆分为多个步骤，以模拟真实的滚动行为。
+
+* **`id`**: `mouseWheel`
+* **`params`**:
+  * `x` (number, 可选): 滚动的绝对 X 坐标。如果为负数，则被视为相对于当前位置的随机偏移。
+  * `y` (number, 可选): 滚动的绝对 Y 坐标。如果为负数，则被视为相对于当前位置的随机偏移。
+  * `selector` (string, 可选): CSS 选择器。如果提供，则确保元素可见并先将鼠标移动到其中心。
+  * `deltaX` (number, 可选): 水平滚动量。默认值为 0。
+  * `deltaY` (number, 可选): 垂直滚动量。默认值为 0。
+  * `steps` (number, 可选): 将滚动拆分为多少步执行（默认值：`1`）。
+* **`returns`**: `none`
+
+#### `scrollIntoView`
+
+滚动页面或可滚动容器，使特定元素在视口中可见。
+
+* **`id`**: `scrollIntoView`
+* **`params`**:
+  * `selector` (string): 要滚动到视口可见的元素的 CSS 选择器。
+* **`returns`**: `none`
+
 #### `keyboardType`
 
 模拟真人在当前获得焦点的元素中输入文本。

package/README.action.md

@@ -253,8 +253,8 @@ Moves the mouse cursor to a specific coordinate or element. In `browser` mode, i
 
 * **`id`**: `mouseMove`
 * **`params`**:
-  * `x` (number, optional): The absolute X coordinate.
-  * `y` (number, optional): The absolute Y coordinate.
+  * `x` (number, optional): The absolute X coordinate. If negative, it's treated as a relative random offset from current position.
+  * `y` (number, optional): The absolute Y coordinate. If negative, it's treated as a relative random offset from current position.
   * `selector` (string, optional): A CSS selector. If provided, the mouse moves to the center of the element.
   * `steps` (number, optional): The number of intermediate steps for the trajectory (default: `-1`). Set to `-1` to calculate steps automatically based on distance (simulating natural speed).
 * **`returns`**: `none`
@@ -265,14 +265,37 @@ Triggers a mouse click at the current position or specified coordinates. If a `s
 
 * **`id`**: `mouseClick`
 * **`params`**:
-  * `x` (number, optional): The absolute X coordinate to click.
-  * `y` (number, optional): The absolute Y coordinate to click.
+  * `x` (number, optional): The absolute X coordinate to click. If negative, it's treated as a relative random offset from current position.
+  * `y` (number, optional): The absolute Y coordinate to click. If negative, it's treated as a relative random offset from current position.
   * `selector` (string, optional): A CSS selector. If provided, moves the mouse to the element first.
   * `button` (string, optional): The mouse button to use (`left`, `right`, or `middle`). Default is `left`.
   * `clickCount` (number, optional): The number of clicks (e.g., 2 for double-click). Default is 1.
   * `delay` (number, optional): Delay between mousedown and mouseup in milliseconds.
 * **`returns`**: `none`
 
+#### `mouseWheel`
+
+Simulates a mouse wheel scroll event. If a `selector` is provided, the element is automatically scrolled into view, and the cursor is moved to its center before scrolling. If `steps` is provided, the scroll delta is split into multiple steps for realistic simulation.
+
+* **`id`**: `mouseWheel`
+* **`params`**:
+  * `x` (number, optional): The absolute X coordinate to scroll at. If negative, it's treated as a relative random offset from current position.
+  * `y` (number, optional): The absolute Y coordinate to scroll at. If negative, it's treated as a relative random offset from current position.
+  * `selector` (string, optional): A CSS selector. If provided, ensures the element is visible and moves the mouse to its center first.
+  * `deltaX` (number, optional): The horizontal scroll amount. Default is 0.
+  * `deltaY` (number, optional): The vertical scroll amount. Default is 0.
+  * `steps` (number, optional): The number of steps to split the scroll into (default: `1`).
+* **`returns`**: `none`
+
+#### `scrollIntoView`
+
+Scrolls the page or a scrollable container to make a specific element visible in the viewport.
+
+* **`id`**: `scrollIntoView`
+* **`params`**:
+  * `selector` (string): The CSS selector of the element to scroll into view.
+* **`returns`**: `none`
+
 #### `keyboardType`
 
 Simulates a person typing text into the currently focused element.

package/README.cn.md

@@ -27,6 +27,19 @@
 
 ---
 
+### 智能升级与重试策略
+
+当 `enableSmart` 开启时，系统会根据响应特征自动判断是否需要升级引擎：
+
+- 触发升级的条件包括：
+  - HTTP 状态码：`401 / 403 / 500 / 429`
+  - 页面疑似动态渲染（HTML 中检测到典型 JS 框架特征）
+  - `Retry-After` 超过 `upgradeThresholdMs`
+- 升级过程中可选择是否同步 Cookies / Session 状态（`syncStateOnUpgrade`）
+- 对于 `429` 响应，若 `Retry-After` 小于`upgradeThresholdMs`阈值，系统会优先重试而非升级
+
+---
+
 ## 📦 安装
 
 1. **安装依赖包:**
@@ -150,6 +163,12 @@ searchGoogle('gemini');
   * `headless` (boolean): 是否以无头模式运行（默认：`true`）。
   * `launchOptions` (object): Playwright 启动选项（例如 `{ slowMo: 50, args: [...] }`）。
 * `sessionPoolOptions` (SessionPoolOptions): 底层 Crawlee SessionPool 的高级配置。
+* `enableSmart` (boolean): 是否启用智能探测与自动引擎升级（默认：`true`）。
+* `syncStateOnUpgrade` (boolean): 当从 http 升级到 browser 引擎时，是否同步 Cookies / Session 状态（默认：`false`）。
+* `upgradeThresholdMs` (number): 触发引擎升级的等待时间阈值（毫秒），超过该时间或无明确重试信息则升级（默认：`5000`）。
+* `maxRetries` (number): 单个 Action 的最大重试次数（默认：`0`）。
+* `failOnError` (boolean): Action 失败时是否抛出异常（默认：主流程 `true`，collector `false`）。
+* `failOnTimeout` (boolean): 超时是否视为失败（默认：`false`）。
 * ...以及许多其他用于代理、重试等的选项。
 
 ### 内置动作 (Built-in Actions)
@@ -162,6 +181,8 @@ searchGoogle('gemini');
 * `submit`: 提交表单（引擎相关）。
 * `mouseMove`: 将鼠标指针移动到指定的坐标或元素（支持贝塞尔曲线）。
 * `mouseClick`: 在当前位置或指定坐标触发鼠标点击。
+* `mouseWheel`: 在目标位置模拟鼠标滚轮滚动，支持水平和垂直偏移、分步模拟以及自动将目标元素滚动到视口。
+* `scrollIntoView`: 滚动页面或容器，使特定元素在视口中可见。
 * `keyboardType`: 模拟真人在当前获得焦点的元素中输入文本。
 * `keyboardPress`: 模拟按下单个按键或组合键。
 * `trim`: 从 DOM 中移除元素以清理页面（如脚本、广告、隐藏内容）。

package/README.engine.cn.md

@@ -64,7 +64,13 @@ const session = new FetchSession({ engine: 'browser' });
 1. **显式选项**: 如果在 `FetchSession` 的 `options.engine`（或 `executeAll` 的临时上下文覆盖）中提供了引擎且不为 `'auto'`。
     * ⚠️ **快速失败 (Fail-Fast)**: 如果请求的引擎不可用（例如缺少依赖），将立即抛出错误。
 2. **站点注册表 (Site Registry)**: 如果设置为 `'auto'`（默认），系统会尝试根据目标 URL 匹配 `sites` 注册表。
-3. **智能升级 (Smart Upgrade)**: 如果启用，系统可能会根据响应特征（如机器人检测或大量 JS）动态从 `http` 升级到 `browser`。
+3. **智能升级 (Smart Upgrade)**:
+   - 当 `enableSmart: true` 时，系统会在以下情况自动从 `http` 升级到 `browser`：
+     - 返回 `401 / 403 / 500 / 429`
+     - HTML 内容被识别为“高度动态”（大量 JS）
+     - `Retry-After` 超过 `upgradeThresholdMs`
+   - 升级时可选择是否同步 Cookies / Session（`syncStateOnUpgrade`）
+   - 升级失败或仍不满足需求时，可继续抛出原始错误
 4. **默认**: 回退到 `'http'` (Cheerio)。
 
 #### 带有覆盖的批量执行 (Batch Execution with Overrides)
@@ -82,6 +88,20 @@ await session.executeAll([
 });
 ```
 
+#### 智能升级与重试
+
+在 `executeAll` 执行过程中，如果某个 Action 抛出 `ENGINE_UPGRADE_REQUIRED`：
+
+- 系统会尝试释放当前引擎
+- 创建新的 browser 引擎
+- **自动从头重新执行动作列表**
+- 已成功执行的副作用（如 Cookie 写入）可根据配置保留
+
+此外，对于 `429` 响应：
+
+- 若 `Retry-After` 存在且小于 `upgradeThresholdMs`
+- 系统会在同一引擎内自动重试，而不会触发升级
+
 ### 会话管理与状态持久化
 
 引擎支持在多次执行之间持久化和恢复会话状态（主要是 Cookie）。
@@ -112,6 +132,16 @@ await session.executeAll([
 
 ---
 
+### 错误增强与 Retry-After 支持
+
+- 所有 HTTP 错误现在都会将原始 `FetchResponse` 附加到错误对象（`error.response`）
+- 支持解析 HTTP `Retry-After` 头：
+  - 支持整数（秒）
+  - 支持 HTTP 日期格式
+- 错误信息中会包含重试等待时间提示
+
+---
+
 ## 🏗️ 3. 架构和工作流程
 
 引擎架构旨在解决一个关键挑战：在 Crawlee 根本上**无状态、异步**的请求处理之上，提供一个简单的、**类似有状态的 API**（先 `goto()`，再 `click()`，再 `fill()`）。
@@ -158,7 +188,7 @@ await session.executeAll([
   * ✅ **快速轻量**：非常适合追求速度和低资源消耗的场景。
   * ✅ **符合 HTTP 标准的重定向**：正确处理 301-303 和 307/308 重定向，按照 HTTP 规范保留方法/正文或转换为 GET。
   * ❌ **无 JavaScript 执行**：无法与客户端渲染的内容交互。
-  * ⚙️ **模拟交互**：像 `click` 和 `submit` 这样的动作是通过发起新的 HTTP 请求来模拟的。**仅浏览器支持的动作**（如 `mouseMove`, `keyboardType`）将抛出 `not_supported` 错误。
+  * ⚙️ **模拟交互**：像 `click` 和 `submit` 这样的动作是通过发起新的 HTTP 请求来模拟的。**仅浏览器支持的动作**（如 `mouseMove`, `mouseWheel`, `keyboardType`）将抛出 `not_supported` 错误。
 * **用例**: 抓取静态网站、服务器渲染页面或 API。
 
 ### `PlaywrightFetchEngine` (browser 模式)
@@ -187,8 +217,8 @@ await session.executeAll([
 
 您可以通过选项中的 `browser` 属性来配置浏览器引擎：
 
-*   `headless` (boolean): 是否以无头模式运行浏览器（默认：`true`）。
-*   `launchOptions` (object): 原生 Playwright [LaunchOptions](https://playwright.dev/docs/api/class-browsertype#browser-type-launch)，直接传递给浏览器启动器（例如 `slowMo`, `args`, `devtools`）。
+* `headless` (boolean): 是否以无头模式运行浏览器（默认：`true`）。
+* `launchOptions` (object): 原生 Playwright [LaunchOptions](https://playwright.dev/docs/api/class-browsertype#browser-type-launch)，直接传递给浏览器启动器（例如 `slowMo`, `args`, `devtools`）。
 
     ```typescript
     const result = await fetchWeb({
@@ -246,9 +276,7 @@ await session.executeAll([
   - 支持可选的 `depth` 参数来限制向上遍历的层级。
   - 必须包含最大深度限制 (默认 1000) 以防止无限循环。
 
-这种架构确保了诸如 **列对齐 (Columnar Alignment)**、**分段扫描 (Segmented Scanning)** 以及 **属性锚点跳转 (Anchor Jumping)** 等复杂功能在不同引擎下表现高度一致。
-
-这种解耦确保了诸如 **列对齐 (Columnar Alignment)**、**分段扫描 (Segmented Scanning)** 以及 **属性锚点跳转 (Anchor Jumping)** 等复杂功能,无论是在快速的 Cheerio 引擎还是完整的 Playwright 浏览器中,其行为都完全一致。
+这种架构确保了诸如 **列对齐 (Columnar Alignment)**、**分段扫描 (Segmented Scanning)** 以及 **属性锚点跳转 (Anchor Jumping)** 等复杂功能，无论是在快速的 Cheerio 引擎还是完整的 Playwright 浏览器中，其行为都完全一致。
 
 ### Schema 规范化
 

package/README.engine.md

@@ -64,9 +64,27 @@ The engine is initialized lazily upon the first action execution and remains fix
 1. **Explicit Option**: If `options.engine` (or temporary context override in `executeAll`) is provided and NOT set to `'auto'`.
     * ⚠️ **Fail-Fast**: If the requested engine is unavailable (e.g., missing dependencies), an error is thrown immediately.
 2. **Site Registry**: If set to `'auto'` (default), the system attempts to match the target URL against the `sites` registry.
-3. **Smart Upgrade**: If enabled, the engine may be dynamically upgraded from `http` to `browser` based on response characteristics (e.g., bot detection or heavy JS).
+3. **Smart Upgrade**: If `enableSmart: true`, the system will automatically upgrade from `http` to `browser` under the following conditions:
+   - Returns `401 / 403 / 500 / 429`
+   - HTML content is identified as "highly dynamic" (heavy JS)
+   - `Retry-After` exceeds `upgradeThresholdMs`
+   - You can optionally sync Cookies/Session during upgrade (`syncStateOnUpgrade`)
+   - If upgrade fails or still doesn't meet requirements, the original error is thrown
 4. **Default**: Falls back to `'http'` (Cheerio).
 
+#### Smart Upgrade & Retry
+
+During `executeAll`, if an Action throws `ENGINE_UPGRADE_REQUIRED`:
+
+- The system attempts to release the current engine
+- Creates a new browser engine
+- **Automatically re-executes the action list from the beginning**
+- Side effects of successfully executed actions (e.g., cookie writes) can be preserved based on configuration
+
+For `429` responses:
+- If `Retry-After` exists and is less than `upgradeThresholdMs`
+- The system will automatically retry within the same engine without triggering an upgrade
+
 #### Batch Execution with Overrides
 
 You can execute a sequence of actions with temporary configuration overrides (e.g., headers, timeout) that apply only to that specific batch, without modifying the session's global state.
@@ -112,6 +130,16 @@ If both `sessionState` and `cookies` are provided, the engine adopts a **"Merge
 
 ---
 
+### Error Enhancement & Retry-After Support
+
+- All HTTP errors now attach the original `FetchResponse` to the error object (`error.response`)
+- Support for parsing HTTP `Retry-After` header:
+  - Integer (seconds) format
+  - HTTP date format
+- Retry wait time hints are included in error messages
+
+---
+
 ## 🏗️ 3. Architecture and Workflow
 
 The engine's architecture is designed to solve a key challenge: providing a simple, **stateful-like API** (`goto()`, then `click()`, then `fill()`) on top of Crawlee's fundamentally **stateless, asynchronous** request handling.
@@ -158,7 +186,7 @@ There are two primary engine implementations:
   * ✅ **Fast and Lightweight**: Ideal for speed and low resource consumption.
   * ✅ **HTTP-Compliant Redirects**: Correctly handles 301-303 and 307/308 redirects, preserving methods/bodies or converting to GET as per HTTP specifications.
   * ❌ **No JavaScript Execution**: Cannot interact with client-side rendered content.
-  * ⚙️ **Simulated Interaction**: Actions like `click` and `submit` are simulated by making new HTTP requests. **Browser-only actions** (e.g., `mouseMove`, `keyboardType`) will throw a `not_supported` error.
+  * ⚙️ **Simulated Interaction**: Actions like `click` and `submit` are simulated by making new HTTP requests. **Browser-only actions** (e.g., `mouseMove`, `mouseWheel`, `keyboardType`) will throw a `not_supported` error.
 * **Use Case**: Scraping static websites, server-rendered pages, or APIs.
 
 ### `PlaywrightFetchEngine` (browser mode)

package/README.md

@@ -20,9 +20,23 @@ English | [简体中文](./README.cn.md)
 * **📜 Declarative Action Scripts**: Define multi-step workflows (like logging in, filling forms, and clicking buttons) in a simple, readable JSON format.
 * **📊 Powerful and Flexible Data Extraction**: Easily extract all kinds of structured data, from simple text to complex nested objects, through an intuitive and powerful declarative Schema.
 * **🧠 Smart Engine Selection**: Automatically detects dynamic sites and can upgrade the engine from `http` to `browser` on the fly.
+* **🛡️ Anti-Bot Evasion**: In `browser` mode, an optional `antibot` flag helps to bypass common anti-bot measures like Cloudflare challenges.
+* **🕹️ High-Fidelity Interaction Simulation**: Supports Bézier curve-based mouse trajectory movement, realistic typing delay simulation, and complex keyboard interactions to significantly improve anti-bot evasion.
 * **🧩 Extensible**: Easily create custom, high-level "composite" actions to encapsulate reusable business logic (e.g., a `login` action).
 * **🧲 Advanced Collectors**: Asynchronously collect data in the background, triggered by events during the execution of a main action.
-* **🛡️ Anti-Bot Evasion**: In `browser` mode, an optional `antibot` flag helps to bypass common anti-bot measures like Cloudflare challenges.
+
+---
+
+### Smart Upgrade and Retry Strategy
+
+When `enableSmart` is enabled, the system automatically determines whether an engine upgrade is needed based on response characteristics:
+
+- Triggers for upgrade include:
+  - HTTP status codes: `401 / 403 / 500 / 429`
+  - Page appears to be dynamically rendered (detected typical JS framework signatures in HTML)
+  - `Retry-After` exceeds `upgradeThresholdMs`
+- During upgrade, you can choose whether to sync Cookies / Session state (`syncStateOnUpgrade`)
+- For `429` responses, if `Retry-After` is less than the `upgradeThresholdMs` threshold, the system will prioritize retry over upgrade
 
 ---
 
@@ -149,6 +163,12 @@ This is the main entry point for the library.
   * `headless` (boolean): Run in headless mode (default: `true`).
   * `launchOptions` (object): Playwright launch options (e.g., `{ slowMo: 50, args: [...] }`).
 * `sessionPoolOptions` (SessionPoolOptions): Advanced configuration for the underlying Crawlee SessionPool.
+* `enableSmart` (boolean): Enable smart detection and automatic engine upgrade (default: `true`).
+* `syncStateOnUpgrade` (boolean): Whether to sync Cookies / Session state when upgrading from http to browser engine (default: `false`).
+* `upgradeThresholdMs` (number): Wait time threshold in milliseconds to trigger engine upgrade; upgrades if exceeded or no explicit retry info (default: `5000`).
+* `maxRetries` (number): Maximum retry attempts for a single Action (default: `0`).
+* `failOnError` (boolean): Whether to throw an exception when an Action fails (default: `true` for main flow, `false` for collector).
+* `failOnTimeout` (boolean): Whether to treat timeout as failure (default: `false`).
 * ...and many other options for proxy, retries, etc.
 
 ### Built-in Actions
@@ -161,6 +181,8 @@ The library provides a set of powerful built-in actions, many of which are engin
 * `submit`: Submits a form (Engine-specific).
 * `mouseMove`: Moves the mouse cursor to a specific coordinate or element (Bézier curve supported).
 * `mouseClick`: Triggers a mouse click at the current position or specified coordinates.
+* `mouseWheel`: Simulates a mouse wheel scroll event with horizontal and vertical deltas. Supports splitting into multiple steps and automatic scrolling to make the target element visible.
+* `scrollIntoView`: Scrolls the page or a container to make a specific element visible in the viewport.
 * `keyboardType`: Simulates human-like typing into the currently focused element.
 * `keyboardPress`: Simulates pressing a single key or a key combination.
 * `trim`: Removes elements from the DOM to clean up the page.