@isdk/web-fetcher 0.2.12 → 0.3.1

package/docs/_media/README.engine.md

@@ -12,6 +12,14 @@ The `engine` directory contains the core logic for the web fetcher. Its primary
 
 > ℹ️ The system is built on top of the [Crawlee](https://crawlee.dev/) library, using its powerful crawler abstractions.
 
+### Debug Mode & Tracing
+
+When the `debug: true` option is enabled, the engine provides detailed tracing of its internal operations:
+
+1. **Detailed Tracing**: Every major step (request processing, element selection, data extraction) is logged to the console with a `[FetchEngine:id]` prefix.
+2. **Extraction Insights**: During `extract()`, the engine logs how many elements were matched for each selector and the specific values being extracted, making it easier to debug complex or misaligned schemas.
+3. **Metadata**: The `FetchResponse` will include an enriched `metadata` object containing engine details, timing metrics (where available), and proxy information.
+
 ---
 
 ## 🧩 2. Core Concepts
@@ -20,13 +28,23 @@ The `engine` directory contains the core logic for the web fetcher. Its primary
 
 This is the abstract base class that defines the contract for all fetch engines.
 
-* **Role**: To provide a consistent, high-level API for actions like navigation, content retrieval, and user interaction.
+* **Role**: To provide a consistent, high-level API for actions like navigation, content retrieval, and user interaction. It acts as the central **Action Dispatcher**, handling engine-agnostic logic (like `extract`, `pause`, `getContent`) and delegating others.
 * **Key Abstractions**:
   * **Lifecycle**: `initialize()` and `cleanup()` methods.
   * **Core Actions**: `goto()`, `getContent()`, `click()`, `fill()`, `submit()`, `waitFor()`, `extract()`.
+  * **DOM Primitives**: `_querySelectorAll()`, `_extractValue()`, `_parentElement()`, `_nextSiblingsUntil()`.
   * **Configuration & State**: `headers()`, `cookies()`, `blockResources()`, `getState()`, `sessionPoolOptions`.
 * **Static Registry**: It maintains a static registry of all available engine implementations (`FetchEngine.register`), allowing for dynamic selection by `id` or `mode`.
 
+### `FetchElementScope`
+
+The **`FetchElementScope`** is the engine-specific "handle" or "context" for a DOM element.
+
+- In **`CheerioFetchEngine`**, it is a combination of the Cheerio API (`$`) and the current selection (`el`).
+- In **`PlaywrightFetchEngine`**, it is a `Locator`.
+
+All extraction and interaction primitives operate on this scope, ensuring a unified way to reference elements across different underlying technologies.
+
 ### `FetchEngine.create(context, options)`
 
 This static factory method is the designated entry point for creating an engine instance. It automatically selects and initializes the appropriate engine.
@@ -69,14 +87,14 @@ await session.executeAll([
 The engine supports persisting and restoring session state (primarily cookies) between executions.
 
 * **Flexible Session Isolation & Storage**: The library provides fine-grained control over how session data is stored and isolated via the `storage` configuration:
-    * **`id`**: A custom string to identify the storage.
-        * **Isolation (Default)**: If omitted, each session gets a unique ID, ensuring complete isolation of `RequestQueue`, `KeyValueStore`, and `SessionPool`.
-        * **Sharing**: Providing the same `id` across sessions allows them to share the same underlying storage, useful for persistent login sessions.
-    * **`persist`**: (boolean) Whether to enable disk persistence (Crawlee's `persistStorage`). Defaults to `false` (in-memory).
-    * **`purge`**: (boolean) Whether to delete the storage (drop `RequestQueue` and `KeyValueStore`) when the session is closed. Defaults to `true`.
-        * Set `purge: false` and provide a fixed `id` to create a truly persistent session that survives across application restarts.
-    * **`config`**: Allows passing raw configuration to the underlying Crawlee instance.
-        * **Note**: When `persist` is true, use `localDataDirectory` in the config to specify the storage path (e.g., `storage: { persist: true, config: { localDataDirectory: './my-data' } }`).
+  * **`id`**: A custom string to identify the storage.
+    * **Isolation (Default)**: If omitted, each session gets a unique ID, ensuring complete isolation of `RequestQueue`, `KeyValueStore`, and `SessionPool`.
+    * **Sharing**: Providing the same `id` across sessions allows them to share the same underlying storage, useful for persistent login sessions.
+  * **`persist`**: (boolean) Whether to enable disk persistence (Crawlee's `persistStorage`). Defaults to `false` (in-memory).
+  * **`purge`**: (boolean) Whether to delete the storage (drop `RequestQueue` and `KeyValueStore`) when the session is closed. Defaults to `true`.
+    * Set `purge: false` and provide a fixed `id` to create a truly persistent session that survives across application restarts.
+  * **`config`**: Allows passing raw configuration to the underlying Crawlee instance.
+    * **Note**: When `persist` is true, use `localDataDirectory` in the config to specify the storage path (e.g., `storage: { persist: true, config: { localDataDirectory: './my-data' } }`).
 * **`sessionState`**: A comprehensive state object (derived from Crawlee's SessionPool) that can be used to fully restore a previous session. This state is **automatically included in every `FetchResponse`**, making it easy to persist and later provide back to the engine during initialization.
 * **`sessionPoolOptions`**: Allows advanced configuration of the underlying Crawlee `SessionPool` (e.g., `maxUsageCount`, `maxPoolSize`).
 * **`overrideSessionState`**: If set to `true`, it forces the engine to overwrite any existing persistent state in the storage with the provided `sessionState`. This is useful when you want to ensure the session starts with the exact state provided, ignoring any stale data in the persistence layer.
@@ -86,6 +104,7 @@ The engine supports persisting and restoring session state (primarily cookies) b
 
 **Precedence Rule:**
 If both `sessionState` and `cookies` are provided, the engine adopts a **"Merge and Override"** strategy:
+
 1. The session is first restored from the `sessionState`.
 2. The explicit `cookies` are then applied on top.
    * **Result:** Any conflicting cookies in `sessionState` will be **overwritten** by the explicit `cookies`.
@@ -114,8 +133,11 @@ Our engine solves this by creating a bridge between the external API calls and t
     * The page context is used to resolve the `Promise` from the `goto()` call.
     * The page is marked as "active" (`isPageActive = true`).
     * Crucially, before the `requestHandler` returns, it starts an **action loop** (`_executePendingActions`). This loop effectively **pauses the `requestHandler`** by listening for events on an `EventEmitter`, keeping the page context alive.
+    * **Strict Sequential Execution & Re-entrancy**: The loop uses an internal queue to ensure all actions are executed in the exact order they were dispatched. It also includes re-entrancy protection to allow composite actions to call atomic actions without deadlocking.
 5. **Interactive Actions (`click`, `fill`, etc.)**: The consumer can now call `await engine.click(...)`. This dispatches an action to the `EventEmitter` and returns a new `Promise`.
-6. **Action Execution**: The action loop, still running within the original `requestHandler`'s scope, hears the event. Because it has access to the page context, it can perform the *actual* interaction (e.g., `page.click(...)`).
+6. **Action Execution**: The action loop, still running within the original `requestHandler`'s scope, hears the event.
+    * **Centralized Actions**: Actions like `extract`, `pause`, and `getContent` are processed immediately by the `FetchEngine` base class using the unified logic.
+    * **Delegated Actions**: Engine-specific interactions (e.g., `click`, `fill`) are delegated to the subclass's `executeAction` implementation.
 7. **Robust Cleanup**: When `dispose()` or `cleanup()` is called:
     * An `isEngineDisposed` flag is set to prevent new actions.
     * A `dispose` signal is emitted to wake up and terminate the action loop.
@@ -134,8 +156,9 @@ There are two primary engine implementations:
 * **Mechanism**: Uses `CheerioCrawler` to fetch pages via raw HTTP and parse static HTML.
 * **Behavior**:
   * ✅ **Fast and Lightweight**: Ideal for speed and low resource consumption.
+  * ✅ **HTTP-Compliant Redirects**: Correctly handles 301-303 and 307/308 redirects, preserving methods/bodies or converting to GET as per HTTP specifications.
   * ❌ **No JavaScript Execution**: Cannot interact with client-side rendered content.
-  * ⚙️ **Simulated Interaction**: Actions like `click` and `submit` are simulated by making new HTTP requests.
+  * ⚙️ **Simulated Interaction**: Actions like `click` and `submit` are simulated by making new HTTP requests. **Browser-only actions** (e.g., `mouseMove`, `keyboardType`) will throw a `not_supported` error.
 * **Use Case**: Scraping static websites, server-rendered pages, or APIs.
 
 ### `PlaywrightFetchEngine` (browser mode)
@@ -160,17 +183,95 @@ To combat sophisticated anti-bot measures, the `PlaywrightFetchEngine` offers an
 * **Use Case**: Scraping websites protected by services like Cloudflare or other advanced bot-detection systems.
 * **Note**: This feature requires additional dependencies (`camoufox-js`, `firefox`) and may have a performance overhead.
 
+#### Configuration
+
+You can configure the browser engine via the `browser` property in options:
+
+*   `headless` (boolean): Whether to run browser in headless mode (default: `true`).
+*   `launchOptions` (object): Native Playwright [LaunchOptions](https://playwright.dev/docs/api/class-browsertype#browser-type-launch) passed directly to the browser launcher (e.g., `slowMo`, `args`, `devtools`).
+
+    ```typescript
+    const result = await fetchWeb({
+      url: 'https://example.com',
+      engine: 'browser',
+      browser: {
+        headless: false,
+        launchOptions: {
+          slowMo: 100, // Slow down operations by 100ms
+          args: ['--start-maximized'] // Pass custom arguments
+        }
+      }
+    });
+    ```
+
 ---
 
 ## 📊 5. Data Extraction with `extract()`
 
 The `extract()` method provides a powerful, declarative way to pull structured data from a web page. It uses a **Schema** to define the structure of your desired JSON output, and the engine automatically handles DOM traversal and data extraction.
 
-### Core Design: Schema Normalization
+### Core Design: The Three-Layer Architecture
+
+To ensure consistency across engines and maintain high quality, the extraction system is divided into three layers:
+
+1. **Normalization Layer (`src/core/normalize-extract-schema.ts`)**: Pre-processes user-provided schemas into a canonical internal format, handling CSS filter merging and implicit object detection.
+2. **Core Extraction Logic (`src/core/extract.ts`)**: An engine-agnostic layer responsible for the extraction workflow. It dispatches tasks to `_extractObject`, `_extractArray`, or `_extractValue` via a **Dispatcher (`_extract`)**. It manages recursion, strict mode, required field validation, anchor resolution, performance-optimized tree operations (LCA, bubbling), and sequential consumption cursors.
+3. **Engine Interface (`IExtractEngine`)**: Defined at the core layer and implemented by engines to provide low-level DOM primitives.
+
+#### Implementation Rules for `IExtractEngine`
+
+To maintain cross-engine consistency, all implementations MUST follow these behavior contracts:
+
+- **`_querySelectorAll`**:
+  - MUST return matching elements in **document order**.
+  - MUST check if the scope element(s) **themselves** match the selector and search their **descendants**.
+- **`_nextSiblingsUntil`**:
+  - MUST return a flat list of siblings starting *after* the anchor and stopping *before* the first element matching the `untilSelector`.
+- **`_isSameElement`**:
+  - MUST compare elements based on **identity**, not content.
+- **`_findClosestAncestor`**:
+  - MUST efficiently find the closest ancestor of an element that exists in a given set of candidates.
+  - MUST be optimized to avoid multiple IPC calls in browser-based engines.
+- **`_contains`**:
+  - MUST implement standard DOM `Node.contains()` behavior.
+  - MUST be optimized for high-frequency boundary checks.
+- **`_findCommonAncestor`**:
+  - MUST find the Lowest Common Ancestor (LCA) of two elements.
+  - **Performance Critical**: In browser engines, this MUST be executed within a single `evaluate` call to minimize IPC (Inter-Process Communication) overhead.
+- **`_findContainerChild`**:
+  - MUST find the direct child of a container that contains a specific descendant.
+  - **Performance Critical**: This replaces manual "bubble-up" loops in the Node.js context, significantly reducing overhead for deep DOM trees.
+- **`_bubbleUpToScope` (Internal Helper)**:
+  - Implements the logic to bubble up from a deep element to its direct ancestor in the current scope.
+  - Supports an optional `depth` parameter to limit how many parent levels to traverse.
+  - MUST include a maximum depth limit (default 1000) to prevent infinite loops.
+
+This architecture ensures that complex features like **Columnar Alignment**, **Segmented Scanning**, and **Anchor Jumping** behave identically across the fast Cheerio engine and the full Playwright browser.
+
+### Schema Normalization
+
+To enhance usability and flexibility, the `extract` method internally implements a **"Normalization"** layer. This allows you to provide semantically clear shorthands, which are automatically converted into a standardized internal format.
+
+#### 1. Shorthand Rules
+
+- **String Shorthand**: A simple string like `'h1'` is automatically expanded to `{ selector: 'h1', type: 'string', mode: 'text' }`.
+- **Implicit Object Shorthand**: If you provide an object without an explicit `type: 'object'`, it is automatically treated as an `object` schema where the keys are the property names.
+  - *Example*: `{ "title": "h1" }` becomes `{ "type": "object", "properties": { "title": { "selector": "h1" } } }`.
+- **Filter Shorthand**: If you provide `has` or `exclude` alongside a `selector`, they are automatically merged into the CSS selector using `:has()` and `:not()` pseudo-classes.
+- **Array Shorthand**: Providing `attribute` directly on an `array` schema acts as a shorthand for its `items`.
+  - *Example*: `{ "type": "array", "attribute": "href" }` becomes `{ "type": "array", "items": { "type": "string", "attribute": "href" } }`.
+
+#### 2. Context vs. Data (Keyword Separation)
+
+In **Implicit Objects**, the engine must distinguish between *configuration* (where to look) and *data* (what to extract).
+
+- **Context Keys**: The keys `selector`, `has`, `exclude`, `required`, `strict`, and `depth` are reserved for defining the extraction context and validation. They stay at the root of the schema.
+- **Data Keys**: All other keys (including `items`, `attribute`, `mode`, or even a field named `type`) are moved into the `properties` object as data fields to be extracted.
+- **Collision Handling**: You can safely extract a field named `type` as long as its value is not one of the reserved schema type keywords (`string`, `number`, `boolean`, `html`, `object`, `array`).
 
-To enhance usability and flexibility, the `extract` method internally implements a **"Normalization"** layer. This means you can provide semantically clear shorthands, and the engine will automatically convert them into a standard, more verbose internal format before execution. This makes writing complex extraction rules simple and intuitive.
+#### 3. Cross-Engine Consistency
 
-### Schema Structure
+This normalization layer ensures that regardless of whether you are using the `cheerio` (http) or `playwright` (browser) engine, the complex shorthand logic behaves identically, providing a consistent "AI-friendly" interface.
 
 A schema can be one of three types: