@isdk/web-fetcher 0.2.0

package/README.engine.cn.md

@@ -0,0 +1,262 @@
+# ⚙️ Fetch Engine 架构
+
+[English](./README.engine.md) | 简体中文
+
+> 本文档全面概述了 Fetch Engine 的架构，旨在抽象化网页内容抓取和交互。它适用于需要理解、维护或扩展抓取引擎功能的开发人员。
+
+---
+
+## 🎯 1. 概述
+
+`engine` 目录包含 Web Fetcher 的核心逻辑。其主要职责是为与网页交互提供统一的接口，无论底层技术如何（例如，简单的 HTTP 请求或功能齐全的浏览器）。这是通过抽象的 `FetchEngine` 类和利用不同抓取技术的具体实现来实现的。
+
+> ℹ️ 该系统建立在 [Crawlee](https://crawlee.dev/) 库之上，并使用其强大的爬虫抽象。
+
+---
+
+## 🧩 2. 核心概念
+
+### `FetchEngine` (base.ts)
+
+这是定义所有抓取引擎契约的抽象基类。
+
+* **职责**：为导航、内容检索和用户交互等操作提供一致的高级 API。
+* **关键抽象**：
+  * **生命周期**：`initialize()` 和 `cleanup()` 方法。
+  * **核心操作**：`goto()`、`getContent()`、`click()`、`fill()`、`submit()`、`waitFor()`、`extract()`。
+  * **配置**：`headers()`、`cookies()`、`blockResources()`。
+* **静态注册表**：它维护所有可用引擎实现的静态注册表（`FetchEngine.register`），允许通过 `id` 或 `mode` 动态选择引擎。
+
+### `FetchEngine.create(context, options)`
+
+此静态工厂方法是创建引擎实例的指定入口点。它会自动选择并初始化合适的引擎。
+
+---
+
+## 🏗️ 3. 架构和工作流程
+
+引擎架构旨在解决一个关键挑战：在 Crawlee 根本上**无状态、异步**的请求处理之上，提供一个简单的、**类似有状态的 API**（先 `goto()`，再 `click()`，再 `fill()`）。
+
+### 核心问题：短暂的页面上下文
+
+在 Crawlee 中，页面上下文（Playwright 中的 `page` 对象或 Cheerio 中的 `$` 函数）**仅在 `requestHandler` 函数的同步作用域内可用**。一旦处理程序完成或 `await`，上下文就会丢失。这使得无法直接实现 `await engine.goto(); await engine.click();` 这样的序列。
+
+### 解决方案：使用动作循环桥接作用域
+
+我们的引擎通过在外部 API 调用和内部 `requestHandler` 作用域之间创建桥梁来解决此问题。这是设计中最关键的部分。
+
+**工作流程如下：**
+
+1. **初始化**：消费者调用 `FetchEngine.create()`，初始化一个在后台运行的 Crawlee 爬虫（例如 `PlaywrightCrawler`）。
+2. **导航 (`goto`)**：消费者调用 `await engine.goto(url)`。此方法将 URL 添加到 Crawlee 的 `RequestQueue`，并返回一个 `Promise`，该 `Promise` 将在页面加载后被解析。
+3. **Crawlee 处理**：后台爬虫接收请求并调用引擎的 `requestHandler`，向其传递关键的页面上下文。
+4. **页面激活和动作循环**：在 `requestHandler` 内部：
+    * 页面上下文用于解析 `goto()` 调用返回的 `Promise`。
+    * 页面被标记为“活动”状态 (`isPageActive = true`)。
+    * 至关重要的是，在 `requestHandler` 返回之前，它会启动一个**动作循环** (`_executePendingActions`)。此循环通过监听 `EventEmitter` 上的事件，有效地**暂停 `requestHandler`**，从而保持页面上下文的存活。
+5. **交互式动作 (`click`, `fill` 等)**：消费者现在可以调用 `await engine.click(...)`。此方法将一个动作分派到 `EventEmitter` 并返回一个新的 `Promise`。
+6. **动作执行**：仍在原始 `requestHandler` 作用域内运行的动作循环，会监听到该事件。因为它能访问页面上下文，所以可以执行*实际的*交互（例如 `page.click(...)`）。
+7. **清理**：循环一直持续到分派 `dispose` 动作（例如，由新的 `goto()` 调用触发），该动作会终止循环，并允许 `requestHandler` 最终完成。
+
+---
+
+## 🛠️ 4. 实现
+
+主要有两种引擎实现：
+
+### `CheerioFetchEngine` (http 模式)
+
+* **ID**: `cheerio`
+* **机制**: 使用 `CheerioCrawler` 通过原始 HTTP 请求抓取页面并解析静态 HTML。
+* **行为**:
+  * ✅ **快速轻量**：非常适合追求速度和低资源消耗的场景。
+  * ❌ **无 JavaScript 执行**：无法与客户端渲染的内容交互。
+  * ⚙️ **模拟交互**：像 `click` 和 `submit` 这样的动作是通过发起新的 HTTP 请求来模拟的。
+* **用例**: 抓取静态网站、服务器渲染页面或 API。
+
+### `PlaywrightFetchEngine` (browser 模式)
+
+* **ID**: `playwright`
+* **机制**: 使用 `PlaywrightCrawler` 控制一个真实的无头浏览器。
+* **行为**:
+  * ✅ **完整的浏览器环境**：执行 JavaScript，原生处理 Cookie、会话和复杂的用户交互。
+  * ✅ **强大的交互**：动作能准确模拟真实用户行为。
+  * ⚠️ **资源密集型**：比 Cheerio 引擎慢，需要更多的内存/CPU。
+* **用例**: 与现代动态 Web 应用程序（SPA）或任何严重依赖 JavaScript 的网站进行交互。
+
+#### 反爬虫/反屏蔽 (`antibot` 选项)
+
+为了对抗复杂的反机器人措施，`PlaywrightFetchEngine` 提供了一个 `antibot` 模式。启用后，它会集成 [Camoufox](https://github.com/prescience-data/camoufox) 来增强其规避能力。
+
+* **机制**:
+  * 通过 `camoufox-js` 使用一个经过加固的 Firefox 浏览器。
+  * 禁用默认的指纹伪装，由 Camoufox 管理浏览器指纹。
+  * 自动尝试解决在导航过程中遇到的 Cloudflare 挑战。
+* **如何启用**: 在创建 fetcher 属性时设置 `antibot: true` 选项。
+* **用例**: 抓取受 Cloudflare 或其他高级机器人检测系统保护的网站。
+* **注意**: 此功能需要额外的依赖项（`camoufox-js`, `firefox`），并可能带来性能开销。
+
+---
+
+## 📊 5. 使用 `extract()` 进行数据提取
+
+`extract()` 方法提供了一种强大的声明式方式来从网页中提取结构化数据。它通过一个 **Schema (模式)** 来定义您期望的 JSON 输出结构,引擎会自动处理 DOM 遍历和数据提取。
+
+### 核心设计: Schema 规范化
+
+为了提升易用性和灵活性,`extract` 方法在内部实现了一个**“规范化 (Normalization)”**层。这意味着您可以提供语义清晰的简写形式,在执行前,引擎会自动将其转换为标准的、更详细的内部格式。这使得编写复杂的提取规则变得简单直观。
+
+### Schema 结构
+
+一个 Schema 可以是以下三种类型之一:
+
+* **值提取 (`ExtractValueSchema`)**: 提取单个值。
+  * `selector`: (可选) 用于定位元素的 CSS 选择器。
+  * `type`: (可选) 提取值的类型,如 `'string'` (默认), `'number'`, `'boolean'`, 或 `'html'` (提取内部 HTML)。
+  * `attribute`: (可选) 如果提供,则提取元素的指定属性值(例如 `href`),而不是其文本内容。
+* **对象提取 (`ExtractObjectSchema`)**: 提取一个 JSON 对象。
+  * `selector`: (可选) 对象数据的根元素。
+  * `properties`: 定义对象每个字段的子提取规则。
+* **数组提取 (`ExtractArraySchema`)**: 提取一个数组。
+  * `selector`: 用于匹配列表中每个项目元素的 CSS 选择器。
+  * `items`: (可选) 应用于每个项目元素的提取规则。**如果省略,默认为提取元素的文本内容**。
+
+### 高级功能 (通过规范化实现)
+
+#### 1. 简写语法
+
+为了简化常见场景,我们支持以下简写:
+
+* **提取属性数组**: 您可以在 `array` 类型的 Schema 上直接使用 `attribute` 属性,作为 `items: { attribute: '...' }` 的简写。
+
+    **简写:**
+
+    ```json
+    {
+      "type": "array",
+      "selector": ".post a",
+      "attribute": "href"
+    }
+    ```
+
+    **等效于:**
+
+    ```json
+    {
+      "type": "array",
+      "selector": ".post a",
+      "items": { "attribute": "href" }
+    }
+    ```
+
+* **提取文本数组**: 只需提供选择器,省略 `items` 即可。
+
+    **简写:**
+
+    ```json
+    {
+      "type": "array",
+      "selector": ".tags li"
+    }
+    ```
+
+    **等效于:**
+
+    ```json
+    {
+      "type": "array",
+      "selector": ".tags li",
+      "items": { "type": "string" }
+    }
+    ```
+
+#### 2. 精确筛选: `has` 和 `exclude`
+
+您可以在任何包含 `selector` 的 Schema 中使用 `has` 和 `exclude` 字段来精确控制元素的选择。
+
+* `has`: 一个 CSS 选择器,用于确保所选元素**必须包含**匹配此选择器的后代元素 (类似 `:has()`)。
+* `exclude`: 一个 CSS 选择器,用于从结果中**排除**匹配此选择器的元素 (类似 `:not()`)。
+
+### 完整示例
+
+假设我们有以下 HTML,我们想提取所有"重要"文章的标题和链接,同时排除掉"存档"文章。
+
+```html
+<div id="articles">
+  <div class="post important">
+    <a href="/post/1"><h3>Post 1</h3></a>
+  </div>
+  <div class="post">
+    <!-- This one is NOT important, lacks h3 -->
+    <a href="/post/2">Post 2</a>
+  </div>
+  <div class="post important archived">
+    <!-- This one is important but archived -->
+    <a href="/post/3"><h3>Archived Post 3</h3></a>
+  </div>
+</div>
+```
+
+我们可以使用以下 Schema:
+
+```typescript
+const schema = {
+  type: 'array',
+  selector: '.post',      // 1. 选择所有 post
+  has: 'h3',              // 2. 只保留包含 <h3> 的 post (重要文章)
+  exclude: '.archived',   // 3. 排除包含 .archived 类名的 post
+  items: {
+    type: 'object',
+    properties: {
+      title: { selector: 'h3' },
+      link: { selector: 'a', attribute: 'href' },
+    }
+  }
+};
+
+const data = await engine.extract(schema);
+
+/*
+ 预计输出:
+ [
+   {
+     "title": "Post 1",
+     "link": "/post/1"
+   }
+ ]
+*/
+```
+
+---
+
+## 🧑‍💻 6. 如何扩展引擎
+
+添加新的抓取引擎非常简单：
+
+1. **创建类**：定义一个扩展泛型 `FetchEngine` 的新类，并提供具体的 `Context`、`Crawler` 和 `Options` 类型。
+
+    ```typescript
+    import { PlaywrightCrawler, PlaywrightCrawlerOptions, PlaywrightCrawlingContext } from 'crawlee';
+
+    class MyPlaywrightEngine extends FetchEngine<
+      PlaywrightCrawlingContext,
+      PlaywrightCrawler,
+      PlaywrightCrawlerOptions
+    > {
+      // ...
+    }
+    ```
+
+2. **定义静态属性**：设置唯一的 `id` 和 `mode`。
+3. **实现抽象方法**：为基类的抽象方法提供具体实现：
+    * `_getSpecificCrawlerOptions()`: 返回一个包含爬虫特定选项的对象（例如，`headless` 模式、`preNavigationHooks`）。
+    * `_createCrawler()`: 返回一个新的爬虫实例（例如，`new PlaywrightCrawler(options)`）。
+    * `buildResponse()`: 将爬取上下文转换为标准的 `FetchResponse`。
+    * `executeAction()`: 处理特定引擎的动作实现，如 `click`、`fill` 等。
+4. **注册引擎**：调用 `FetchEngine.register(MyNewEngine)`。
+
+---
+
+## ✅ 7. 测试
+
+`engine.spec.ts` 文件包含一个全面的测试套件。`engineTestSuite` 函数定义了一组标准测试，该套件会针对 `CheerioFetchEngine` 和 `PlaywrightFetchEngine` 运行，以确保它们在功能上等效并符合 `FetchEngine` API 契约。

package/README.engine.md

@@ -0,0 +1,262 @@
+# ⚙️ Fetch Engine Architecture
+
+English | [简体中文](./README.engine.cn.md)
+
+> This document provides a comprehensive overview of the Fetch Engine architecture, designed to abstract web content fetching and interaction. It's intended for developers who need to understand, maintain, or extend the fetching capabilities.
+
+---
+
+## 🎯 1. Overview
+
+The `engine` directory contains the core logic for the web fetcher. Its primary responsibility is to provide a unified interface for interacting with web pages, regardless of the underlying technology (e.g., simple HTTP requests or a full-fledged browser). This is achieved through an abstract `FetchEngine` class and concrete implementations that leverage different crawling technologies.
+
+> ℹ️ The system is built on top of the [Crawlee](https://crawlee.dev/) library, using its powerful crawler abstractions.
+
+---
+
+## 🧩 2. Core Concepts
+
+### `FetchEngine` (base.ts)
+
+This is the abstract base class that defines the contract for all fetch engines.
+
+* **Role**: To provide a consistent, high-level API for actions like navigation, content retrieval, and user interaction.
+* **Key Abstractions**:
+  * **Lifecycle**: `initialize()` and `cleanup()` methods.
+  * **Core Actions**: `goto()`, `getContent()`, `click()`, `fill()`, `submit()`, `waitFor()`, `extract()`.
+  * **Configuration**: `headers()`, `cookies()`, `blockResources()`.
+* **Static Registry**: It maintains a static registry of all available engine implementations (`FetchEngine.register`), allowing for dynamic selection by `id` or `mode`.
+
+### `FetchEngine.create(context, options)`
+
+This static factory method is the designated entry point for creating an engine instance. It automatically selects and initializes the appropriate engine.
+
+---
+
+## 🏗️ 3. Architecture and Workflow
+
+The engine's architecture is designed to solve a key challenge: providing a simple, **stateful-like API** (`goto()`, then `click()`, then `fill()`) on top of Crawlee's fundamentally **stateless, asynchronous** request handling.
+
+### The Core Problem: Ephemeral Page Context
+
+In Crawlee, the page context (the `page` object in Playwright or the `$` function in Cheerio) is **only available within the synchronous scope of the `requestHandler` function**. Once the handler finishes or `await`s, the context is lost. This makes it impossible to directly implement a sequence like `await engine.goto(); await engine.click();`.
+
+### The Solution: Bridging Scopes with an Action Loop
+
+Our engine solves this by creating a bridge between the external API calls and the internal `requestHandler` scope. This is the most critical part of the design.
+
+**The workflow is as follows:**
+
+1. **Initialization**: A consumer calls `FetchEngine.create()`, which initializes a Crawlee crawler (e.g., `PlaywrightCrawler`) that runs in the background.
+2. **Navigation (`goto`)**: The consumer calls `await engine.goto(url)`. This adds the URL to Crawlee's `RequestQueue` and returns a `Promise` that will resolve when the page is loaded.
+3. **Crawlee Processing**: The background crawler picks up the request and invokes the engine's `requestHandler`, passing it the crucial page context.
+4. **Page Activation & Action Loop**: Inside the `requestHandler`:
+    * The page context is used to resolve the `Promise` from the `goto()` call.
+    * The page is marked as "active" (`isPageActive = true`).
+    * Crucially, before the `requestHandler` returns, it starts an **action loop** (`_executePendingActions`). This loop effectively **pauses the `requestHandler`** by listening for events on an `EventEmitter`, keeping the page context alive.
+5. **Interactive Actions (`click`, `fill`, etc.)**: The consumer can now call `await engine.click(...)`. This dispatches an action to the `EventEmitter` and returns a new `Promise`.
+6. **Action Execution**: The action loop, still running within the original `requestHandler`'s scope, hears the event. Because it has access to the page context, it can perform the *actual* interaction (e.g., `page.click(...)`).
+7. **Cleanup**: The loop continues until a `dispose` action is dispatched (e.g., by a new `goto()` call), which terminates the loop and allows the `requestHandler` to finally complete.
+
+---
+
+## 🛠️ 4. Implementations
+
+There are two primary engine implementations:
+
+### `CheerioFetchEngine` (http mode)
+
+* **ID**: `cheerio`
+* **Mechanism**: Uses `CheerioCrawler` to fetch pages via raw HTTP and parse static HTML.
+* **Behavior**:
+  * ✅ **Fast and Lightweight**: Ideal for speed and low resource consumption.
+  * ❌ **No JavaScript Execution**: Cannot interact with client-side rendered content.
+  * ⚙️ **Simulated Interaction**: Actions like `click` and `submit` are simulated by making new HTTP requests.
+* **Use Case**: Scraping static websites, server-rendered pages, or APIs.
+
+### `PlaywrightFetchEngine` (browser mode)
+
+* **ID**: `playwright`
+* **Mechanism**: Uses `PlaywrightCrawler` to control a real headless browser.
+* **Behavior**:
+  * ✅ **Full Browser Environment**: Executes JavaScript, handles cookies, sessions, and complex user interactions natively.
+  * ✅ **Robust Interaction**: Actions accurately mimic real user behavior.
+  * ⚠️ **Resource Intensive**: Slower and requires more memory/CPU.
+* **Use Case**: Interacting with modern, dynamic web applications (SPAs) or any site that relies heavily on JavaScript.
+
+#### Anti-Bot Evasion (`antibot` option)
+
+To combat sophisticated anti-bot measures, the `PlaywrightFetchEngine` offers an `antibot` mode. When enabled, it integrates [Camoufox](https://github.com/prescience-data/camoufox) to enhance its stealth capabilities.
+
+* **Mechanism**:
+  * Uses a hardened Firefox browser via `camoufox-js`.
+  * Disables default fingerprint spoofing to let Camoufox manage the browser's fingerprint.
+  * Automatically attempts to solve Cloudflare challenges encountered during navigation.
+* **How to enable**: Set the `antibot: true` option when creating the fetcher properties.
+* **Use Case**: Scraping websites protected by services like Cloudflare or other advanced bot-detection systems.
+* **Note**: This feature requires additional dependencies (`camoufox-js`, `firefox`) and may have a performance overhead.
+
+---
+
+## 📊 5. Data Extraction with `extract()`
+
+The `extract()` method provides a powerful, declarative way to pull structured data from a web page. It uses a **Schema** to define the structure of your desired JSON output, and the engine automatically handles DOM traversal and data extraction.
+
+### Core Design: Schema Normalization
+
+To enhance usability and flexibility, the `extract` method internally implements a **"Normalization"** layer. This means you can provide semantically clear shorthands, and the engine will automatically convert them into a standard, more verbose internal format before execution. This makes writing complex extraction rules simple and intuitive.
+
+### Schema Structure
+
+A schema can be one of three types:
+
+* **Value Extraction (`ExtractValueSchema`)**: Extracts a single value.
+  * `selector`: (Optional) A CSS selector to locate the element.
+  * `type`: (Optional) The type of the extracted value, such as `'string'` (default), `'number'`, `'boolean'`, or `'html'` (extracts inner HTML).
+  * `attribute`: (Optional) If provided, extracts the value of the specified attribute (e.g., `href`) instead of its text content.
+* **Object Extraction (`ExtractObjectSchema`)**: Extracts a JSON object.
+  * `selector`: (Optional) The root element for the object's data.
+  * `properties`: Defines sub-extraction rules for each field of the object.
+* **Array Extraction (`ExtractArraySchema`)**: Extracts an array.
+  * `selector`: A CSS selector to match each item element in the list.
+  * `items`: (Optional) The extraction rule to apply to each item element. **If omitted, it defaults to extracting the element's text content**.
+
+### Advanced Features (via Normalization)
+
+#### 1. Shorthand Syntax
+
+To simplify common scenarios, the following shorthands are supported:
+
+* **Extracting an Array of Attributes**: You can use the `attribute` property directly on an `array` type schema as a shorthand for `items: { attribute: '...' }`.
+
+    **Shorthand:**
+
+    ```json
+    {
+      "type": "array",
+      "selector": ".post a",
+      "attribute": "href"
+    }
+    ```
+
+    **Equivalent to:**
+
+    ```json
+    {
+      "type": "array",
+      "selector": ".post a",
+      "items": { "attribute": "href" }
+    }
+    ```
+
+* **Extracting an Array of Texts**: Simply provide the selector and omit `items`.
+
+    **Shorthand:**
+
+    ```json
+    {
+      "type": "array",
+      "selector": ".tags li"
+    }
+    ```
+
+    **Equivalent to:**
+
+    ```json
+    {
+      "type": "array",
+      "selector": ".tags li",
+      "items": { "type": "string" }
+    }
+    ```
+
+#### 2. Precise Filtering: `has` and `exclude`
+
+You can use the `has` and `exclude` fields in any schema that includes a `selector` to precisely control element selection.
+
+* `has`: A CSS selector to ensure the selected element **must contain** a descendant matching this selector (similar to `:has()`).
+* `exclude`: A CSS selector to **exclude** elements matching this selector from the results (similar to `:not()`).
+
+### Complete Example
+
+Suppose we have the following HTML and we want to extract the titles and links of all "important" articles, while excluding "archived" ones.
+
+```html
+<div id="articles">
+  <div class="post important">
+    <a href="/post/1"><h3>Post 1</h3></a>
+  </div>
+  <div class="post">
+    <!-- This one is NOT important, lacks h3 -->
+    <a href="/post/2">Post 2</a>
+  </div>
+  <div class="post important archived">
+    <!-- This one is important but archived -->
+    <a href="/post/3"><h3>Archived Post 3</h3></a>
+  </div>
+</div>
+```
+
+We can use the following schema:
+
+```typescript
+const schema = {
+  type: 'array',
+  selector: '.post',      // 1. Select all posts
+  has: 'h3',              // 2. Only keep posts that contain an <h3> (important)
+  exclude: '.archived',   // 3. Exclude posts with the .archived class
+  items: {
+    type: 'object',
+    properties: {
+      title: { selector: 'h3' },
+      link: { selector: 'a', attribute: 'href' },
+    }
+  }
+};
+
+const data = await engine.extract(schema);
+
+/*
+ Expected output:
+ [
+   {
+     "title": "Post 1",
+     "link": "/post/1"
+   }
+ ]
+*/
+```
+
+---
+
+## 🧑‍💻 6. How to Extend the Engine
+
+Adding a new fetch engine is straightforward:
+
+1. **Create the Class**: Define a new class that extends the generic `FetchEngine`, providing the specific `Context`, `Crawler`, and `Options` types from Crawlee.
+
+    ```typescript
+    import { PlaywrightCrawler, PlaywrightCrawlerOptions, PlaywrightCrawlingContext } from 'crawlee';
+
+    class MyPlaywrightEngine extends FetchEngine<
+      PlaywrightCrawlingContext,
+      PlaywrightCrawler,
+      PlaywrightCrawlerOptions
+    > {
+      // ...
+    }
+    ```
+
+2. **Define Static Properties**: Set the unique `id` and `mode`.
+3. **Implement Abstract Methods**: Provide concrete implementations for the abstract methods from the base class:
+    * `_getSpecificCrawlerOptions()`: Return an object with crawler-specific options (e.g., `headless` mode, `preNavigationHooks`).
+    * `_createCrawler()`: Return a new instance of your crawler (e.g., `new PlaywrightCrawler(options)`).
+    * `buildResponse()`: Convert the crawling context to a standard `FetchResponse`.
+    * `executeAction()`: Handle engine-specific implementations for actions like `click`, `fill`, etc.
+4. **Register the Engine**: Call `FetchEngine.register(MyNewEngine)`.
+
+---
+
+## ✅ 7. Testing
+
+The file `engine.spec.ts` contains a comprehensive test suite. The `engineTestSuite` function defines a standard set of tests that is run against both `CheerioFetchEngine` and `PlaywrightFetchEngine` to ensure they are functionally equivalent and conform to the `FetchEngine` API contract.

package/README.md

@@ -0,0 +1,147 @@
+# 🕸️ @isdk/web-fetcher
+
+English | [简体中文](./README.cn.md)
+
+> A powerful and flexible web fetching and browser automation library.
+> It features a dual-engine architecture (HTTP and Browser) and a declarative action system, making it perfect for AI agents and complex data scraping tasks.
+
+---
+
+## ✨ Core Features
+
+* **⚙️ Dual-Engine Architecture**: Choose between **`http`** mode (powered by Cheerio) for speed on static sites, or **`browser`** mode (powered by Playwright) for full JavaScript execution on dynamic sites.
+* **📜 Declarative Action Scripts**: Define multi-step workflows (like logging in, filling forms, and clicking buttons) in a simple, readable JSON format.
+* **📊 Powerful and Flexible Data Extraction**: Easily extract all kinds of structured data, from simple text to complex nested objects, through an intuitive and powerful declarative Schema.
+* **🧠 Smart Engine Selection**: Automatically detects dynamic sites and can upgrade the engine from `http` to `browser` on the fly.
+* **🧩 Extensible**: Easily create custom, high-level "composite" actions to encapsulate reusable business logic (e.g., a `login` action).
+* **🧲 Advanced Collectors**: Asynchronously collect data in the background, triggered by events during the execution of a main action.
+* **🛡️ Anti-Bot Evasion**: In `browser` mode, an optional `antibot` flag helps to bypass common anti-bot measures like Cloudflare challenges.
+
+---
+
+## 📦 Installation
+
+1. **Install the Package:**
+
+    ```bash
+    npm install @isdk/web-fetcher
+    ```
+
+2. **Install Browsers (For `browser` mode):**
+
+    The `browser` engine is powered by Playwright, which requires separate browser binaries to be downloaded. If you plan to use the `browser` engine for interacting with dynamic websites, run the following command:
+
+    ```bash
+    npx playwright install
+    ```
+
+    > ℹ️ **Note:** This step is only required for `browser` mode. The lightweight `http` mode works out of the box without this installation.
+
+---
+
+## 🚀 Quick Start
+
+The following example fetches a web page and extracts its title.
+
+```typescript
+import { fetchWeb } from '@isdk/web-fetcher';
+
+async function getTitle(url: string) {
+  const { outputs } = await fetchWeb({
+    url,
+    actions: [
+      {
+        id: 'extract',
+        params: {
+          // Extracts the text content of the <title> tag
+          selector: 'title',
+        },
+        // Stores the result in the `outputs` object under the key 'pageTitle'
+        storeAs: 'pageTitle',
+      },
+    ],
+  });
+
+  console.log('Page Title:', outputs.pageTitle);
+}
+
+getTitle('https://www.google.com');
+```
+
+---
+
+## 🤖 Advanced Usage: Multi-Step Form Submission
+
+This example demonstrates how to use the `browser` engine to perform a search on Google.
+
+```typescript
+import { fetchWeb } from '@isdk/web-fetcher';
+
+async function searchGoogle(query: string) {
+  // Search for the query on Google
+  const { result, outputs } = await fetchWeb({
+    url: 'https://www.google.com',
+    engine: 'browser', // Use the full browser engine for interaction
+    actions: [
+      // The initial navigation to google.com is handled by the `url` option
+      { id: 'fill', params: { selector: 'textarea[name=q]', value: query } },
+      { id: 'submit', params: { selector: 'form' } },
+      { id: 'waitFor', params: { selector: '#search' } }, // Wait for the search results container to appear
+      { id: 'getContent', storeAs: 'searchResultsPage' },
+    ]
+  });
+
+  console.log('Search Results URL:', result?.finalUrl);
+  console.log('Outputs contains the full page content:', outputs.searchResultsPage.html.substring(0, 100));
+}
+
+searchGoogle('gemini');
+```
+
+---
+
+## 🏗️ Architecture
+
+This library is built on two core concepts: **Engines** and **Actions**.
+
+* ### Engine Architecture
+
+    The library's core is its dual-engine design. It abstracts away the complexities of web interaction behind a unified API. For detailed information on the `http` (Cheerio) and `browser` (Playwright) engines, how they manage state, and how to extend them, please see the [**Fetch Engine Architecture**](./README.engine.md) document.
+
+* ### Action Architecture
+
+    All workflows are defined as a series of "Actions". The library provides a set of built-in atomic actions and a powerful composition model for creating your own semantic actions. For a deep dive into creating and using actions, see the [**Action Script Architecture**](./README.action.md) document.
+
+---
+
+## 📚 API Reference
+
+### `fetchWeb(options)` or `fetchWeb(url, options)`
+
+This is the main entry point for the library.
+
+**Key `FetcherOptions`**:
+
+* `url` (string): The initial URL to navigate to.
+* `engine` ('http' | 'browser' | 'auto'): The engine to use. Defaults to `auto`.
+* `actions` (FetchActionOptions[]): An array of action objects to execute.
+* `headers` (Record<string, string>): Headers to use for all requests.
+* ...and many other options for proxy, cookies, retries, etc.
+
+### Built-in Actions
+
+Here are the essential built-in actions:
+
+* `goto`: Navigates to a new URL.
+* `click`: Clicks on an element specified by a selector.
+* `fill`: Fills an input field with a specified value.
+* `submit`: Submits a form.
+* `waitFor`: Pauses execution to wait for a specific condition (e.g., a timeout, a selector to appear, or network to be idle).
+* `getContent`: Retrieves the full content (HTML, text, etc.) of the current page state.
+* `extract`: Extracts any structured data from the page with ease using an expressive, declarative schema.
+
+---
+
+## 📜 License
+
+[MIT](./LICENSE-MIT)