@isdk/web-fetcher 0.3.2 → 0.3.3

package/README.cn.md

@@ -27,6 +27,19 @@
 
 ---
 
+### 智能升级与重试策略
+
+当 `enableSmart` 开启时，系统会根据响应特征自动判断是否需要升级引擎：
+
+- 触发升级的条件包括：
+  - HTTP 状态码：`401 / 403 / 500 / 429`
+  - 页面疑似动态渲染（HTML 中检测到典型 JS 框架特征）
+  - `Retry-After` 超过 `upgradeThresholdMs`
+- 升级过程中可选择是否同步 Cookies / Session 状态（`syncStateOnUpgrade`）
+- 对于 `429` 响应，若 `Retry-After` 小于`upgradeThresholdMs`阈值，系统会优先重试而非升级
+
+---
+
 ## 📦 安装
 
 1. **安装依赖包:**
@@ -150,6 +163,12 @@ searchGoogle('gemini');
   * `headless` (boolean): 是否以无头模式运行（默认：`true`）。
   * `launchOptions` (object): Playwright 启动选项（例如 `{ slowMo: 50, args: [...] }`）。
 * `sessionPoolOptions` (SessionPoolOptions): 底层 Crawlee SessionPool 的高级配置。
+* `enableSmart` (boolean): 是否启用智能探测与自动引擎升级（默认：`true`）。
+* `syncStateOnUpgrade` (boolean): 当从 http 升级到 browser 引擎时，是否同步 Cookies / Session 状态（默认：`false`）。
+* `upgradeThresholdMs` (number): 触发引擎升级的等待时间阈值（毫秒），超过该时间或无明确重试信息则升级（默认：`5000`）。
+* `maxRetries` (number): 单个 Action 的最大重试次数（默认：`0`）。
+* `failOnError` (boolean): Action 失败时是否抛出异常（默认：主流程 `true`，collector `false`）。
+* `failOnTimeout` (boolean): 超时是否视为失败（默认：`false`）。
 * ...以及许多其他用于代理、重试等的选项。
 
 ### 内置动作 (Built-in Actions)

package/README.engine.cn.md

@@ -64,7 +64,13 @@ const session = new FetchSession({ engine: 'browser' });
 1. **显式选项**: 如果在 `FetchSession` 的 `options.engine`（或 `executeAll` 的临时上下文覆盖）中提供了引擎且不为 `'auto'`。
     * ⚠️ **快速失败 (Fail-Fast)**: 如果请求的引擎不可用（例如缺少依赖），将立即抛出错误。
 2. **站点注册表 (Site Registry)**: 如果设置为 `'auto'`（默认），系统会尝试根据目标 URL 匹配 `sites` 注册表。
-3. **智能升级 (Smart Upgrade)**: 如果启用，系统可能会根据响应特征（如机器人检测或大量 JS）动态从 `http` 升级到 `browser`。
+3. **智能升级 (Smart Upgrade)**:
+   - 当 `enableSmart: true` 时，系统会在以下情况自动从 `http` 升级到 `browser`：
+     - 返回 `401 / 403 / 500 / 429`
+     - HTML 内容被识别为“高度动态”（大量 JS）
+     - `Retry-After` 超过 `upgradeThresholdMs`
+   - 升级时可选择是否同步 Cookies / Session（`syncStateOnUpgrade`）
+   - 升级失败或仍不满足需求时，可继续抛出原始错误
 4. **默认**: 回退到 `'http'` (Cheerio)。
 
 #### 带有覆盖的批量执行 (Batch Execution with Overrides)
@@ -82,6 +88,20 @@ await session.executeAll([
 });
 ```
 
+#### 智能升级与重试
+
+在 `executeAll` 执行过程中，如果某个 Action 抛出 `ENGINE_UPGRADE_REQUIRED`：
+
+- 系统会尝试释放当前引擎
+- 创建新的 browser 引擎
+- **自动从头重新执行动作列表**
+- 已成功执行的副作用（如 Cookie 写入）可根据配置保留
+
+此外，对于 `429` 响应：
+
+- 若 `Retry-After` 存在且小于 `upgradeThresholdMs`
+- 系统会在同一引擎内自动重试，而不会触发升级
+
 ### 会话管理与状态持久化
 
 引擎支持在多次执行之间持久化和恢复会话状态（主要是 Cookie）。
@@ -112,6 +132,16 @@ await session.executeAll([
 
 ---
 
+### 错误增强与 Retry-After 支持
+
+- 所有 HTTP 错误现在都会将原始 `FetchResponse` 附加到错误对象（`error.response`）
+- 支持解析 HTTP `Retry-After` 头：
+  - 支持整数（秒）
+  - 支持 HTTP 日期格式
+- 错误信息中会包含重试等待时间提示
+
+---
+
 ## 🏗️ 3. 架构和工作流程
 
 引擎架构旨在解决一个关键挑战：在 Crawlee 根本上**无状态、异步**的请求处理之上，提供一个简单的、**类似有状态的 API**（先 `goto()`，再 `click()`，再 `fill()`）。
@@ -187,8 +217,8 @@ await session.executeAll([
 
 您可以通过选项中的 `browser` 属性来配置浏览器引擎：
 
-*   `headless` (boolean): 是否以无头模式运行浏览器（默认：`true`）。
-*   `launchOptions` (object): 原生 Playwright [LaunchOptions](https://playwright.dev/docs/api/class-browsertype#browser-type-launch)，直接传递给浏览器启动器（例如 `slowMo`, `args`, `devtools`）。
+* `headless` (boolean): 是否以无头模式运行浏览器（默认：`true`）。
+* `launchOptions` (object): 原生 Playwright [LaunchOptions](https://playwright.dev/docs/api/class-browsertype#browser-type-launch)，直接传递给浏览器启动器（例如 `slowMo`, `args`, `devtools`）。
 
     ```typescript
     const result = await fetchWeb({
@@ -246,9 +276,7 @@ await session.executeAll([
   - 支持可选的 `depth` 参数来限制向上遍历的层级。
   - 必须包含最大深度限制 (默认 1000) 以防止无限循环。
 
-这种架构确保了诸如 **列对齐 (Columnar Alignment)**、**分段扫描 (Segmented Scanning)** 以及 **属性锚点跳转 (Anchor Jumping)** 等复杂功能在不同引擎下表现高度一致。
-
-这种解耦确保了诸如 **列对齐 (Columnar Alignment)**、**分段扫描 (Segmented Scanning)** 以及 **属性锚点跳转 (Anchor Jumping)** 等复杂功能,无论是在快速的 Cheerio 引擎还是完整的 Playwright 浏览器中,其行为都完全一致。
+这种架构确保了诸如 **列对齐 (Columnar Alignment)**、**分段扫描 (Segmented Scanning)** 以及 **属性锚点跳转 (Anchor Jumping)** 等复杂功能，无论是在快速的 Cheerio 引擎还是完整的 Playwright 浏览器中，其行为都完全一致。
 
 ### Schema 规范化
 

package/README.engine.md

@@ -64,9 +64,27 @@ The engine is initialized lazily upon the first action execution and remains fix
 1. **Explicit Option**: If `options.engine` (or temporary context override in `executeAll`) is provided and NOT set to `'auto'`.
     * ⚠️ **Fail-Fast**: If the requested engine is unavailable (e.g., missing dependencies), an error is thrown immediately.
 2. **Site Registry**: If set to `'auto'` (default), the system attempts to match the target URL against the `sites` registry.
-3. **Smart Upgrade**: If enabled, the engine may be dynamically upgraded from `http` to `browser` based on response characteristics (e.g., bot detection or heavy JS).
+3. **Smart Upgrade**: If `enableSmart: true`, the system will automatically upgrade from `http` to `browser` under the following conditions:
+   - Returns `401 / 403 / 500 / 429`
+   - HTML content is identified as "highly dynamic" (heavy JS)
+   - `Retry-After` exceeds `upgradeThresholdMs`
+   - You can optionally sync Cookies/Session during upgrade (`syncStateOnUpgrade`)
+   - If upgrade fails or still doesn't meet requirements, the original error is thrown
 4. **Default**: Falls back to `'http'` (Cheerio).
 
+#### Smart Upgrade & Retry
+
+During `executeAll`, if an Action throws `ENGINE_UPGRADE_REQUIRED`:
+
+- The system attempts to release the current engine
+- Creates a new browser engine
+- **Automatically re-executes the action list from the beginning**
+- Side effects of successfully executed actions (e.g., cookie writes) can be preserved based on configuration
+
+For `429` responses:
+- If `Retry-After` exists and is less than `upgradeThresholdMs`
+- The system will automatically retry within the same engine without triggering an upgrade
+
 #### Batch Execution with Overrides
 
 You can execute a sequence of actions with temporary configuration overrides (e.g., headers, timeout) that apply only to that specific batch, without modifying the session's global state.
@@ -112,6 +130,16 @@ If both `sessionState` and `cookies` are provided, the engine adopts a **"Merge
 
 ---
 
+### Error Enhancement & Retry-After Support
+
+- All HTTP errors now attach the original `FetchResponse` to the error object (`error.response`)
+- Support for parsing HTTP `Retry-After` header:
+  - Integer (seconds) format
+  - HTTP date format
+- Retry wait time hints are included in error messages
+
+---
+
 ## 🏗️ 3. Architecture and Workflow
 
 The engine's architecture is designed to solve a key challenge: providing a simple, **stateful-like API** (`goto()`, then `click()`, then `fill()`) on top of Crawlee's fundamentally **stateless, asynchronous** request handling.

package/README.md

@@ -20,9 +20,23 @@ English | [简体中文](./README.cn.md)
 * **📜 Declarative Action Scripts**: Define multi-step workflows (like logging in, filling forms, and clicking buttons) in a simple, readable JSON format.
 * **📊 Powerful and Flexible Data Extraction**: Easily extract all kinds of structured data, from simple text to complex nested objects, through an intuitive and powerful declarative Schema.
 * **🧠 Smart Engine Selection**: Automatically detects dynamic sites and can upgrade the engine from `http` to `browser` on the fly.
+* **🛡️ Anti-Bot Evasion**: In `browser` mode, an optional `antibot` flag helps to bypass common anti-bot measures like Cloudflare challenges.
+* **🕹️ High-Fidelity Interaction Simulation**: Supports Bézier curve-based mouse trajectory movement, realistic typing delay simulation, and complex keyboard interactions to significantly improve anti-bot evasion.
 * **🧩 Extensible**: Easily create custom, high-level "composite" actions to encapsulate reusable business logic (e.g., a `login` action).
 * **🧲 Advanced Collectors**: Asynchronously collect data in the background, triggered by events during the execution of a main action.
-* **🛡️ Anti-Bot Evasion**: In `browser` mode, an optional `antibot` flag helps to bypass common anti-bot measures like Cloudflare challenges.
+
+---
+
+### Smart Upgrade and Retry Strategy
+
+When `enableSmart` is enabled, the system automatically determines whether an engine upgrade is needed based on response characteristics:
+
+- Triggers for upgrade include:
+  - HTTP status codes: `401 / 403 / 500 / 429`
+  - Page appears to be dynamically rendered (detected typical JS framework signatures in HTML)
+  - `Retry-After` exceeds `upgradeThresholdMs`
+- During upgrade, you can choose whether to sync Cookies / Session state (`syncStateOnUpgrade`)
+- For `429` responses, if `Retry-After` is less than the `upgradeThresholdMs` threshold, the system will prioritize retry over upgrade
 
 ---
 
@@ -149,6 +163,12 @@ This is the main entry point for the library.
   * `headless` (boolean): Run in headless mode (default: `true`).
   * `launchOptions` (object): Playwright launch options (e.g., `{ slowMo: 50, args: [...] }`).
 * `sessionPoolOptions` (SessionPoolOptions): Advanced configuration for the underlying Crawlee SessionPool.
+* `enableSmart` (boolean): Enable smart detection and automatic engine upgrade (default: `true`).
+* `syncStateOnUpgrade` (boolean): Whether to sync Cookies / Session state when upgrading from http to browser engine (default: `false`).
+* `upgradeThresholdMs` (number): Wait time threshold in milliseconds to trigger engine upgrade; upgrades if exceeded or no explicit retry info (default: `5000`).
+* `maxRetries` (number): Maximum retry attempts for a single Action (default: `0`).
+* `failOnError` (boolean): Whether to throw an exception when an Action fails (default: `true` for main flow, `false` for collector).
+* `failOnTimeout` (boolean): Whether to treat timeout as failure (default: `false`).
 * ...and many other options for proxy, retries, etc.
 
 ### Built-in Actions