jumpy-lion 0.1.6-beta.30 → 0.1.6-beta.31

package/README.md

@@ -1,795 +1 @@
-# Crawler Documentation
-
-## Table of Contents
-
-- [Overview](#overview)
-- [NPM Package](#npm-package)
-- [Usage](#usage)
-  - [Example Project](#example-project)
-  - [Internal Guide](#internal-guide)
-  - [Examples and Configuration](#examples-and-configuration)
-  - [Advanced Fingerprints Usage](#advanced-fingerprints-usage)
-  - [Syncing BrowserPool and launchOptions fingerprints](#syncing-browserpool-and-launchoptions-fingerprints)
-  - [Stealth Consistency and Network Policies](#stealth-consistency-and-network-policies)
-- [Configurable Fingerprint Options](#configurable-fingerprint-options)
-  - [Usage](#usage-1)
-  - [Available Options](#available-options)
-    - [Core Stealth Options](#core-stealth-options)
-    - [Fingerprint Spoofing](#fingerprint-spoofing)
-    - [Platform Configuration](#platform-configuration)
-    - [Additional Features](#additional-features)
-  - [Default Behavior](#default-behavior)
-  - [Best Practices](#best-practices)
-  - [Performance Considerations](#performance-considerations)
-- [Launch Options for Network and Persistence](#launch-options-for-network-and-persistence)
-- [Session Bundle (save & restore browser state)](#session-bundle-save--restore-browser-state)
-  - [When to use it](#when-to-use-it)
-  - [Producer: capturing a bundle](#producer-capturing-a-bundle)
-  - [Consumer: rehydrating a bundle](#consumer-rehydrating-a-bundle)
-  - [What's inside a bundle](#whats-inside-a-bundle)
-  - [`saveSession()` options](#savesession-options)
-  - [`restoreSession()`](#restoresession)
-  - [Caveats](#caveats)
-- [Crawler Class Documentation](#crawler-class-documentation)
-  - [Constructor](#constructor)
-- [CdpPage Class Documentation](#cdppage-class-documentation)
-  - [Constructor](#constructor-1)
-  - [Static Methods](#static-methods)
-  - [Public Methods](#public-methods)
-- [Utility Functions](#utility-functions)
-  - [createCDPRouter](#createcdprouter)
-  - [saveSession](#savesession)
-  - [restoreSession](#restoresession-1)
-
-## Overview
-
-The `Crawler` class is a custom implementation of the `BrowserCrawler` from Crawlee, designed to utilize the Chrome DevTools Protocol (CDP) for advanced antiblocking capabilities.
-
-## NPM Package
-
-The `jumpy-lion` is official cdp crawler package. See it [here](https://www.npmjs.com/package/jumpy-lion).
-
-### Installation
-
-```bash
-npm install jumpy-lion
-```
-
-A postinstall hook pulls the matching anti-detect Chromium build from a public
-Apify key-value store — no GitHub token, no Apify token. Works the same on
-your workstation, CI, and the Apify platform. If the download fails for any
-reason (offline, proxy, etc.) the install still succeeds; the crawler falls
-back to the system Chrome on `PATH`. See `browser/BUILD.md` for env-var
-overrides and the maintainer release flow.
-
----
----
-
-## Usage
-
-### Example Project
-
-Refer to this [GitHub repository](https://github.com/apify-projects/cdp-crawler-example) for a complete example of using the `Crawler` class.
-
-### Internal Guide
-
-Check out the [CDP Crawler internal guide](https://www.notion.so/apify/CDP-Crawler-internal-guide-183f39950a2280be81d7c86dc048a47a?pvs=4) for tutorial.
-
-### Examples and Configuration
-
-For detailed examples and configuration patterns, see the [Examples README](./examples/README.md). The examples include:
-
-- **Basic Configuration**: Simple fingerprint setup for common use cases
-- **Comprehensive Configuration**: Full feature setup with all spoofing options
-- **Platform-Specific Configurations**: macOS, Windows, and Linux targeting
-- **Performance-Focused Configuration**: Optimized settings for speed
-- **Minimal Configuration**: Using intelligent defaults
-
-The examples demonstrate real-world usage patterns and best practices for different scenarios.
-
-### Advanced Fingerprints usage
-
-To use advanced fingerprints, you need to set the `useExperimentalFingerprints` option to `true` in the `launchContext.launchOptions` of the `Crawler` constructor.
-
-```typescript
-const crawler = new Crawler({
-    launchContext: {
-      launchOptions: {
-        useExperimentalFingerprints: true,
-      }
-    },
-});
-```
-
----
-
-### Syncing BrowserPool and launchOptions fingerprints
-
-**Always keep the operating system in sync between BrowserPool fingerprints and `launchOptions.fingerprintOptions`.** A mismatch can lead to inconsistent signals (for example `navigator.platform`, User-Agent, WebGL, fonts) and reduce antibot effectiveness.
-
-- **launchOptions side**: Set `launchContext.launchOptions.fingerprintOptions.platform` to the desired platform string.
-- **BrowserPool side**: When `browserPoolOptions.useFingerprints` is `true`, set `browserPoolOptions.fingerprintOptions.fingerprintGeneratorOptions.operatingSystems` to the corresponding OS.
-
-Mapping guidance:
-- `platform: 'Win32'` ↔ `operatingSystems: ['windows']`
-- `platform: 'MacIntel'` ↔ `operatingSystems: ['macos']`
-- `platform: 'Linux x86_64'` ↔ `operatingSystems: ['linux']`
-
-Example:
-
-```typescript
-const crawler = new Crawler({
-  launchContext: {
-    launchOptions: {
-      useExperimentalFingerprints: true,
-      fingerprintOptions: {
-        platform: 'Win32', // Keep this in sync with BrowserPool OS
-      },
-    },
-  },
-  browserPoolOptions: {
-    useFingerprints: true,
-    fingerprintOptions: {
-      fingerprintGeneratorOptions: {
-        browsers: ['chrome'],
-        operatingSystems: ['windows'], // Matches platform: 'Win32'
-        devices: ['desktop'],
-      },
-    },
-  },
-});
-```
-
-Note: This configuration surface will be unified later. We are currently testing our custom fingerprint injector so it works even with the BrowserPool built‑in fingerprints turned off. If you prefer, you can rely solely on the custom injector by setting `browserPoolOptions.useFingerprints: false` and keeping `launchOptions.useExperimentalFingerprints: true`.
-
----
-
-### Stealth Consistency and Network Policies
-
-Recent stealth hardening adds explicit consistency and policy controls:
-
-- **UA/Binary version alignment**: the injector aligns advertised `Chrome/x.y.z.w` with the actual running Chrome binary version to reduce fingerprint drift.
-- **WebRTC policy control**: set `fingerprintOptions.webRtcPolicy` to:
-  - `'spoof'` (default): redacts/normalizes WebRTC leak surfaces.
-  - `'disable'`: removes WebRTC APIs from page context.
-- **DNS hardening controls**: configure DoH and secure DNS through launch options (`dnsOverHttpsServer`, `secureDnsMode`).
-- **WebRTC transport policy flag**: configure `webrtcIpHandlingPolicy` at browser launch level.
-- **Persistent profile mode**: set `userDataDir` (+ optional `keepUserDataDir`) to reuse browser state across runs.
-
-## Configurable Fingerprint Options
-
-The CDP crawler supports configurable fingerprint options that can be passed through the crawler options. This allows you to customize the fingerprint spoofing behavior for different use cases.
-
-### Usage
-
-You can configure fingerprint options by adding them to the `launchContext.launchOptions.fingerprintOptions` in your crawler configuration:
-
-```typescript
-import { Crawler } from 'cdp-crawler';
-
-const crawler = new Crawler({
-    launchContext: {
-        launchOptions: {
-            fingerprintOptions: {
-                // Enable advanced stealth features
-                enableAdvancedStealth: true,
-                
-                // Bypass Runtime.enable detection
-                bypassRuntimeEnable: true,
-                
-                // Humanize mouse interactions
-                humanizeInteractions: true,
-                
-                // Spoof WebGL fingerprinting
-                spoofWebGL: true,
-                
-                // Spoof audio context fingerprinting
-                spoofAudioContext: true,
-                
-                // Add variations to client rect measurements
-                spoofClientRects: true,
-                
-                // Mask automation flags
-                maskAutomationFlags: true,
-                
-                // Use fingerprint-generator defaults when available
-                useFingerprintDefaults: true,
-                
-                // Platform to spoof (defaults to Win32 for better evasion)
-                platform: 'Win32', // 'Win32' | 'MacIntel' | 'Linux x86_64'
-                
-                // Spoof font measurements
-                spoofFonts: true,
-                
-                // Spoof performance timing
-                spoofPerformance: true,
-                
-                // Spoof locale settings
-                spoofLocale: true,
-                
-                // Detect timezone from proxy (useful with residential proxies)
-                detectTimezone: true,
-
-                // WebRTC policy: 'spoof' (default) or 'disable'
-                webRtcPolicy: 'spoof',
-            }
-        }
-    },
-    // ... other crawler options
-});
-```
-
-### Available Options
-
-#### Core Stealth Options
-
-- **`enableAdvancedStealth`** (boolean): Enables advanced stealth features including WebGPU spoofing and platform consistency
-- **`bypassRuntimeEnable`** (boolean): Prevents CDP detection through Runtime.enable bypass techniques
-- **`humanizeInteractions`** (boolean): Generates human-like mouse movements using bezier curves
-
-#### Fingerprint Spoofing
-
-- **`spoofWebGL`** (boolean): Spoofs WebGL fingerprinting by modifying GPU adapter information
-- **`spoofAudioContext`** (boolean): Adds noise to audio processing to prevent audio fingerprinting
-- **`spoofClientRects`** (boolean): Adds small variations to getBoundingClientRect results
-- **`spoofFonts`** (boolean): Hides platform-specific fonts and adds font measurement variations
-- **`spoofPerformance`** (boolean): Modifies timing characteristics to match the target platform
-- **`spoofLocale`** (boolean): Ensures consistent locale formatting across all browser properties
-
-#### Platform Configuration
-
-- **`platform`** (string): Target platform to spoof. Options: `'Win32'`, `'MacIntel'`, `'Linux x86_64'`
-- **`useFingerprintDefaults`** (boolean): Use hardcoded defaults instead of fingerprint-generator values. When `false`, uses generated fingerprint values; when `true` (default), uses hardcoded defaults
-
-#### Additional Features
-
-- **`maskAutomationFlags`** (boolean): Masks automation-related flags in the browser
-- **`detectTimezone`** (boolean): Automatically detect timezone from proxy IP (useful with residential proxies)
-- **`webRtcPolicy`** (`'spoof' | 'disable'`): Controls whether WebRTC is spoofed or fully removed from page APIs
-
-### Default Behavior
-
-When no fingerprint options are provided, the crawler uses intelligent defaults:
-
-- **On Apify**: Uses Apify-recommended settings optimized for the Apify environment
-- **On other platforms**: Uses a comprehensive set of stealth features with Windows platform spoofing
-- **Humanization defaults**: mouse, keyboard, and scroll humanization are enabled with safe defaults
-- **UA consistency**: claimed UA Chrome version is automatically aligned to the running Chrome binary
-
-### Best Practices
-
-1. **Use `platform: 'Win32'`** for better evasion on Linux servers (like Apify)
-2. **Enable `detectTimezone: true`** when using residential proxies
-3. **Use `useFingerprintDefaults: false`** to leverage fingerprint-generator's realistic values
-4. **Enable `bypassRuntimeEnable: true`** for sites that detect automation
-5. **Use `enableAdvancedStealth: true`** for maximum protection against fingerprinting
-6. **Keep OS settings in sync** between `launchOptions.fingerprintOptions.platform` and `browserPoolOptions.fingerprintOptions.fingerprintGeneratorOptions.operatingSystems`
-7. **Use `webRtcPolicy: 'disable'`** for strictest leak prevention, or `'spoof'` for compatibility-sensitive targets
-
-### Performance Considerations
-
-- More fingerprint options enabled = slightly higher CPU usage
-- WebGPU spoofing may add a small delay to page loads
-- Humanized interactions add realistic delays to mouse movements
-
-The fingerprint options are designed to provide maximum protection while maintaining good performance for web scraping tasks.
-
-For more configuration examples and patterns, see the [Examples README](./examples/README.md).
-
----
-
-## Launch Options for Network and Persistence
-
-The following options are configured in `launchContext.launchOptions`:
-
-- **`dnsOverHttpsServer`** (string): DoH endpoint template, for example `https://cloudflare-dns.com/dns-query`
-- **`secureDnsMode`** (`'off' | 'automatic' | 'secure'`): Chromium secure DNS mode
-- **`webrtcIpHandlingPolicy`** (`'default' | 'default_public_interface_only' | 'default_public_and_private_interfaces' | 'disable_non_proxied_udp'`): Browser-level WebRTC IP handling policy
-- **`userDataDir`** (string): Reuse a specific Chrome profile directory across runs
-- **`keepUserDataDir`** (boolean): Keep/cleanup profile directory on close (defaults to keep custom dir, cleanup temp dir)
-
-Example:
-
-```typescript
-const crawler = new Crawler({
-  launchContext: {
-    launchOptions: {
-      dnsOverHttpsServer: 'https://cloudflare-dns.com/dns-query',
-      secureDnsMode: 'secure',
-      webrtcIpHandlingPolicy: 'disable_non_proxied_udp',
-      userDataDir: './state/chrome-profile',
-      keepUserDataDir: true,
-      fingerprintOptions: {
-        webRtcPolicy: 'disable',
-      },
-    },
-  },
-});
-```
-
----
-
-## Session Bundle (save & restore browser state)
-
-The session bundle lets one crawler **capture** the full live browser state — cookies, per-origin localStorage and sessionStorage, the Chrome user-data-dir (which transitively carries IndexedDB, Service Workers, and Cache Storage), the C++ `fingerprintConfig`, the captured `fingerprint`, and the resolved locale triple — into a single JSON-serializable blob. A second crawler can then **rehydrate** from that blob and come up byte-for-byte identical, so a session that was authenticated upstream stays authenticated downstream (matching what `naver-session-test` does, generalized into the framework).
-
-This is mechanism, **not** persistence: nothing is written to disk for you. You stash the bundle wherever you like (Apify KV store, S3, a local file) and pass it back when you launch the next crawler.
-
-### When to use it
-
-- An Apify actor logs into a site, then hands the session to a second actor that does the actual scraping.
-- A pool of long-running actors needs to checkpoint browser state between restarts.
-- You need two crawls to look like the *exact same browser* to a bot detector (Cloudflare, DataDome, Naver) — same UA, same UA-CH, same WebGL renderer, same screen, same canvas/audio noise seed, same cookies, same localStorage, same Service Worker state.
-
-### Producer: capturing a bundle
-
-```typescript
-import CDPCrawler, { saveSession, type SessionBundle } from 'cdp-crawler';
-import { Actor } from 'apify';
-
-let bundle: SessionBundle | undefined;
-
-const producer = new CDPCrawler({
-    launchContext: {
-        launchOptions: {
-            useNativeStealth: true,
-            fingerprintOptions: { platform: 'MacIntel' },
-        },
-    },
-    requestHandler: async ({ page }) => {
-        await page.goto('https://target.example.com/login');
-        // … perform login, solve captcha, etc. …
-
-        bundle = await saveSession(page);
-    },
-});
-
-await producer.run(['https://target.example.com/login']);
-await Actor.setValue('session', bundle); // ship to KV store for the next actor
-```
-
-### Consumer: rehydrating a bundle
-
-Pass the bundle as `launchOptions.sessionBundle` on the new crawler. The plugin extracts the user-data-dir into a temp directory, feeds the captured `fingerprintConfig` straight to the C++ patches (regeneration is skipped), pins `useNonApifyFingerprints: false`, and replays cookies + per-origin storage on every new page **before** any navigation runs.
-
-```typescript
-import CDPCrawler, { type SessionBundle } from 'cdp-crawler';
-import { Actor } from 'apify';
-
-const bundle = await Actor.getValue<SessionBundle>('session');
-if (!bundle) throw new Error('No session bundle available');
-
-const consumer = new CDPCrawler({
-    launchContext: {
-        launchOptions: {
-            useNativeStealth: true,
-            sessionBundle: bundle, // ← the only new option
-        },
-    },
-    requestHandler: async ({ page }) => {
-        // The first page already has the producer's cookies, localStorage,
-        // sessionStorage, IndexedDB, Service Worker registrations, etc.
-        await page.goto('https://target.example.com/account'); // already logged in
-    },
-});
-
-await consumer.run(['https://target.example.com/account']);
-```
-
-### What's inside a bundle
-
-```typescript
-interface SessionBundle {
-    schemaVersion: 1;
-    createdAt: string;
-    createdBy?: { package: 'cdp-crawler'; version: string };
-
-    cookies: SerializedCookie[];                     // Network.Cookie-shaped
-    localStorage: Record<origin, Record<key, value>>;
-    sessionStorage: Record<origin, Record<key, value>>;
-
-    userDataDir: {
-        encoding: 'base64+gzip+tar';
-        bytes: string;                               // the whole Chrome profile
-        sizeBytes: number;
-        capturedFiles: number;
-    } | null;
-
-    fingerprintConfig: FingerprintConfigJson;        // C++ overrides JSON, byte-for-byte
-    fingerprint: BrowserFingerprintWithHeaders;      // Crawlee-shaped fp object
-    fingerprintInput: {                              // The fingerprintOptions inputs
-        locale?: string; languages?: string; timezone?: string;
-        platform?: string; seedKey?: string;
-        useNonApifyFingerprints: false;              // pinned on restore
-        [key: string]: unknown;
-    };
-    resolvedLocale: { locale: string; languages: string; timezone: string };
-
-    browserProfile: {                                // diagnostic snapshot
-        userAgent: string; platform: string; language: string;
-        screenWidth: number; screenHeight: number; devicePixelRatio: number;
-        webglRenderer: string; webglVendor: string;
-    };
-
-    proxyMeta?: { proxyUrl?: string; sessionId?: string; countryCode?: string };
-}
-```
-
-Bundles are versioned via `schemaVersion`. Loading a bundle with an unrecognized version throws an explicit error rather than misbehaving silently. A helper `assertValidBundle(value)` is exported for callers that want to validate before passing the bundle around.
-
-### `saveSession()` options
-
-```typescript
-await saveSession(page, {
-    includeUserDataDir: true,    // default true; set false for a JSON-only bundle (~50 KB)
-    flushCookies: true,          // default true → calls Storage.flushBrowserCookies first
-    userDataDirPath: undefined,  // override; defaults to the path used at launch
-    cookieUrls: undefined,       // forwarded to Network.getCookies as a fallback
-    proxyMeta: {                 // optional; stamped for inspection, NOT replayed
-        proxyUrl, sessionId, countryCode,
-    },
-});
-```
-
-The proxy is intentionally not rebuilt on restore — pass your own `proxyUrl` to the consumer crawler so cookies stay tied to the same exit IP.
-
-### `restoreSession()`
-
-`restoreSession(page, bundle)` is the manual escape hatch for advanced users who open additional pages (or targets) inside a single crawl and want them to share the bundle's cookies + storage. Most users do not need it — the `launchOptions.sessionBundle` option is the canonical path.
-
-```typescript
-import { restoreSession } from 'cdp-crawler';
-
-await restoreSession(page, bundle); // sets cookies + registers per-origin storage replay
-```
-
-Note: `restoreSession` cannot swap the user-data-dir or fingerprintConfig on a running browser — those are launch-time inputs and must travel through `launchOptions.sessionBundle`.
-
-### Caveats
-
-- **User-data-dir size**: a real Chrome profile can be several MB. Inlined as base64-gzipped-tar inside the JSON, this can push the bundle past the Apify KV-store 9 MB record limit. Use `includeUserDataDir: false` if you only need cookies + storage and can live without IndexedDB / Service Workers / Cache Storage.
-- **Locks**: Singleton* sentinels, GPU/Shader/Code caches, and Crashpad metrics are deliberately stripped from the tar — they are process-bound or freely regenerable and would otherwise break restore on a new machine.
-- **Capture timing**: the data-dir tar is taken while Chrome is still running. To avoid half-written LevelDB files, prefer to call `saveSession` after the page has gone idle (`waitForLoadState`-style settling, or right before `crawler.teardown()`). `saveSession` automatically calls `Storage.flushBrowserCookies` first.
-- **Cross-version replay**: bundles are tagged with `createdBy.version`; loading a bundle produced by a meaningfully different `cdp-crawler` version may warn or fail depending on what changed in `fingerprintConfig`'s shape. The plan is to migrate forward, not to support arbitrarily old bundles.
-- **Multi-origin localStorage**: `saveSession` captures the current page's origin only. If you need storage from multiple origins, navigate to each one before calling `saveSession`, or call `saveSession` per page and merge the bundles client-side.
-
----
-
-## `Crawler` Class Documentation
-
-### Constructor
-
-#### `constructor(options: BrowserCrawlerOptions = {}, override readonly config = Configuration.getGlobalConfig())`
-
-Initializes the `Crawler` instance with default and provided options.
-
-- **Parameters**:
-
-  - `options` (BrowserCrawlerOptions): Configuration options for the crawler.
-    - `launchContext`: Specifies browser launch parameters.
-      - Default: `{}`
-    - `headless`: Runs the browser in headless mode.
-      - Default: `false`
-    - `browserPoolOptions`: Configuration for managing browser instances.
-  - `config` (Configuration): Global Crawlee configuration.
-    - Default: `Configuration.getGlobalConfig()`
-
-- **Default Behavior**:
-  - Throws an error if `launchContext.proxyUrl` is provided. Use `proxyConfiguration` instead.
-  - Throws an error if `browserPoolOptions.browserPlugins` is set. Use `launchContext.launcher` instead.
-
----
-
-## `CdpPage` Class Documentation
-
-### Constructor
-
-#### `constructor(client: CDP.Client)`
-
-Initializes the `CdpPage` instance with a CDP client.
-
-- **Parameters**:
-
-  - `client` (CDP.Client): The Chrome DevTools Protocol client.
-
-- **Emitted Events**:
-  - `PAGE_CREATED`: Triggered upon the creation of the page.
-
-### Static Methods
-
-#### `static async create(client: CDP.Client): Promise<CdpPage>`
-
-Creates and initializes a new `CdpPage` instance.
-
-- **Parameters**:
-
-  - `client` (CDP.Client): The CDP client.
-
-- **Returns**:
-  - `Promise<CdpPage>`: A promise resolving to the new `CdpPage` instance.
-
----
-
-### Public Methods
-
-#### `async url(): Promise<string>`
-Gets the current URL of the page.
-
-- **Returns**:
-  - `Promise<string>`: The current URL.
-
-#### `async goto(url: string, options?: GotoOptions): Promise<void>`
-Navigates to a specified URL.
-
-- **Parameters**:
-  - `url` (string): The URL to navigate to.
-  - `options` (GotoOptions): Navigation options, including:
-    - `waitUntil`: When to consider navigation finished (`domcontentloaded` or `load`).
-    - `timeout`: Maximum time to wait for navigation in milliseconds.
-
-#### `async click(selector: string): Promise<void>`
-Simulates a click on an element identified by the selector.
-
-- **Parameters**:
-  - `selector` (string): CSS selector of the element.
-
-#### `async type(selector: string, text: string, options?: { delay?: number }): Promise<void>`
-Types text into an input field.
-
-- **Parameters**:
-  - `selector` (string): CSS selector of the element.
-  - `text` (string): Text to type.
-  - `options` (object): Options for typing:
-    - `delay`: Time in milliseconds between key presses.
-
-#### `async screenshot(options?: { path?: string; fullPage?: boolean; format?: 'png' | 'jpeg' }): Promise<Buffer>`
-Takes a screenshot of the page, with support for PNG and JPEG formats.
-
-- **Parameters**:
-  - `options` (object): Screenshot options:
-    - `path`: File path to save the screenshot.
-    - `fullPage`: Capture the entire page.
-    - `format`: Image format, either `'png'` (default) or `'jpeg'`.
-
-- **Returns**:
-  - `Promise<Buffer>`: The screenshot as a buffer.
-
-#### `async content(): Promise<string>`
-Gets the HTML content of the page.
-
-- **Returns**:
-  - `Promise<string>`: The page's HTML.
-
-#### `async toCheerio(): Promise<cheerio.CheerioAPI>`
-Converts the current page content to a Cheerio instance for DOM manipulation.
-
-- **Returns**:
-  - `Promise<cheerio.CheerioAPI>`: A Cheerio API instance.
-
-#### `async setViewport(viewport: Viewport): Promise<void>`
-Sets the page's viewport dimensions.
-
-- **Parameters**:
-  - `viewport` (Viewport): Object with `width` and `height` properties.
-
-#### `async setUserAgent(userAgent: string): Promise<void>`
-Overrides the user-agent string.
-
-- **Parameters**:
-  - `userAgent` (string): The new user-agent string.
-
-#### `async setExtraHTTPHeaders(headers: Record<string, string>): Promise<void>`
-Sets additional HTTP headers for requests.
-
-- **Parameters**:
-  - `headers` (Record<string, string>): Key-value pairs of headers.
-
-#### `async waitForResponse(urlPart: string, statusCode?: number, timeout?: number): Promise<any>`
-Waits for a specific network response.
-
-- **Parameters**:
-  - `urlPart` (string): Part of the URL to match.
-  - `statusCode` (number): Expected HTTP status code.
-  - `timeout` (number): Maximum wait time in milliseconds.
-
-- **Returns**:
-  - `Promise<any>`: The response.
-
-#### `async setCookies(cookies: Cookie[]): Promise<void>`
-Sets cookies for the page.
-
-- **Parameters**:
-  - `cookies` (Cookie[]): Array of cookies to set.
-
-#### `async getCookies(urls?: string[]): Promise<Cookie[]>`
-Retrieves cookies for the given URLs or all cookies if no URLs are specified.
-
-- **Parameters**:
-  - `urls` (string[]): Optional array of URLs.
-
-- **Returns**:
-  - `Promise<Cookie[]>`: Array of cookies.
-
-#### `async waitForSelector(selector: string, options?: { timeout?: number }): Promise<void>`
-Waits for an element matching the selector to appear.
-
-- **Parameters**:
-  - `selector` (string): CSS selector of the element.
-  - `options` (object): Options for waiting:
-    - `timeout`: Maximum wait time in milliseconds.
-
-#### `async elementExists(selector: string): Promise<boolean>`
-Checks if an element exists.
-
-- **Parameters**:
-  - `selector` (string): CSS selector of the element.
-
-- **Returns**:
-  - `Promise<boolean>`: `true` if the element exists, `false` otherwise.
-
-#### `async getTextContent(selector: string): Promise<string>`
-Gets the text content of an element.
-
-- **Parameters**:
-  - `selector` (string): CSS selector of the element.
-
-- **Returns**:
-  - `Promise<string>`: The element's text content.
-
-#### `async getHref(selector: string): Promise<string>`
-Gets the `href` attribute of an anchor element.
-
-- **Parameters**:
-  - `selector` (string): CSS selector of the anchor element.
-
-- **Returns**:
-  - `Promise<string>`: The `href` value.
-
-#### `async reload(options?: GotoOptions): Promise<void>`
-Reloads the current page.
-
-- **Parameters**:
-  - `options` (GotoOptions): Navigation options, including:
-    - `waitUntil`: When to consider reload finished (`domcontentloaded` or `load`).
-    - `timeout`: Maximum time to wait for reload in milliseconds.
-
-#### `async deleteInput(selector: string): Promise<void>`
-Clears the value of an input field specified by the selector.
-
-- **Parameters**:
-  - `selector` (string): CSS selector of the input element.
-
-#### `async isVisible(selector: string): Promise<boolean>`
-Checks if the element specified by selector is visible (not `display: none` and not `visibility: hidden`).
-The selector should be the root item which can be hidden, otherwise this function could return a false positive.
-
-- **Parameters**:
-  - `selector` (string): CSS selector of the element.
-- **Returns**:
-  - `Promise<boolean>`: `true` if the element is visible, `false` otherwise.
-
-#### `async selectOption(dropdownSelector: string, optionSelector: string | string[], options?: SelectOptionOptions): Promise<void>`
-Selects one or more options from a select element or dropdown with intelligent automatic handling.
-
-**Key Features**:
-- **Automatic Detection**: Distinguishes between HTML `<select>` elements and custom dropdowns
-- **Smart Trigger Discovery**: For custom dropdowns, automatically finds and clicks triggers using multiple strategies
-- **Virtualized List Support**: Handles large dropdown lists with intelligent scrolling
-- **No Manual Configuration**: No need to specify separate trigger and container selectors
-
-- **Parameters**:
-  - `dropdownSelector` (string): CSS selector for the select element or dropdown container.
-  - `optionSelector` (string | string[]): CSS selector(s) for the option(s) to select. Can be a single selector or array of selectors.
-  - `options` (SelectOptionOptions): Optional configuration object with the following properties:
-    - `timeout` (number): Maximum wait time in milliseconds. Default: 30000.
-    - `force` (boolean): Bypass visibility and disabled checks. Default: false.
-    - `waitForOptions` (boolean): Wait for dropdown options to load. Default: true.
-    - `maxScrollAttempts` (number): Maximum scroll attempts for virtualized dropdowns. Default: 10.
-
-#### `async waitForElementPositionToStabilize(selector: string, timeout?: number, checkInterval?: number, stabilityThreshold?: number, tolerance?: number): Promise<void>`
-Waits for an element's position to stabilize by polling its bounding box. Useful before interactions after scrolling/animations.
-
-- **Parameters**:
-  - `selector` (string): Target element selector
-  - `timeout` (number): Max time to wait. Default: 2000
-  - `checkInterval` (number): Polling interval. Default: 100
-  - `stabilityThreshold` (number): Consecutive stable checks required. Default: 3
-  - `tolerance` (number): Max pixel delta to consider stable. Default: 1
-
-- **Usage Examples**:
-  ```typescript
-  // Regular HTML select element - works directly
-  await page.selectOption('select#country', 'option[value="us"]');
-  
-  // Multiple selection in HTML select
-  await page.selectOption('select#languages', ['option[value="en"]', 'option[value="es"]']);
-  
-  // Custom dropdown - automatically finds and clicks trigger
-  await page.selectOption('#dropdown-menu', '[data-value="premium"]');
-  
-  // Virtualized dropdown - automatically scrolls to find option
-  await page.selectOption('#large-dropdown', '[data-item="item-500"]');
-  
-  // With custom configuration
-  await page.selectOption(
-    '#complex-dropdown',
-    '.option[data-category="business"]',
-    {
-      timeout: 10000,
-      maxScrollAttempts: 15
-    }
-  );
-  
-  // Bootstrap/Material-UI dropdowns work automatically
-  await page.selectOption('.MuiSelect-menu', '[data-value="option1"]');
-  await page.selectOption('.dropdown-menu', '.dropdown-item[data-value="choice2"]');
-  ```
-
-- **How Trigger Detection Works**:
-  The method automatically detects dropdown triggers using multiple strategies:
-  1. **Accessibility patterns**: `[aria-haspopup]`, `[role="button"]`
-  2. **Common class names**: `.dropdown-trigger`, `.select-trigger`
-  3. **Sibling elements**: Previous sibling of the dropdown container
-  4. **ID pattern matching**: `#menu-id` → `#trigger-id`, `#dropdown-menu` → `#dropdown-trigger`
-
-- **Migration from Previous API**:
-  ```typescript
-  // OLD - Complex API with manual configuration
-  await page.selectOption('#trigger', '[data-value="item"]', {
-    dropdownSelector: '#menu',
-    optionSelector: '.dropdown-item'
-  });
-  
-  // NEW - Simplified API with automatic detection
-  await page.selectOption('#menu', '[data-value="item"]');
-  ```
-
----
-
-## Utility Functions
-
-### `createCDPRouter`
-
-#### `export function createCDPRouter<Context extends CDPCrawlingContext = CDPCrawlingContext, UserData extends Dictionary = GetUserDataFromRequest<Context['request']>>(routes?: RouterRoutes<Context, UserData>): Router<Context>`
-
-Creates a custom router for handling crawling routes using CDP.
-
-- **Parameters**:
-  - `routes` (RouterRoutes<Context, UserData>): Optional routes for defining crawl logic.
-
-- **Returns**:
-  - `Router<Context>`: A configured router instance.
-
----
-
-### `saveSession`
-
-#### `export async function saveSession(page: CdpPage, options?: SaveSessionOptions): Promise<SessionBundle>`
-
-Captures the full browser state of a running page — cookies, per-origin web storage, fingerprintConfig, fingerprint object, resolved locale, and the Chrome user-data-dir (inlined as base64-gzipped-tar) — into a single JSON-serializable bundle.
-
-- **Parameters**:
-  - `page` (CdpPage): a page produced by a `CDPCrawler` instance.
-  - `options` (SaveSessionOptions): optional knobs:
-    - `includeUserDataDir` (boolean, default `true`): pack and inline the Chrome user-data-dir.
-    - `flushCookies` (boolean, default `true`): call `Storage.flushBrowserCookies` before snapshotting cookies.
-    - `userDataDirPath` (string): override the user-data-dir path; defaults to the one used at launch.
-    - `cookieUrls` (string[]): forwarded to `Network.getCookies` when `Storage.getCookies` is unavailable.
-    - `proxyMeta` ({ proxyUrl?, sessionId?, countryCode? }): stamped on the bundle for inspection only.
-
-- **Returns**:
-  - `Promise<SessionBundle>`: a fully populated, JSON-serializable bundle.
-
-See [Session Bundle](#session-bundle-save--restore-browser-state) for usage patterns.
-
-### `restoreSession`
-
-#### `export async function restoreSession(page: CdpPage, bundle: SessionBundle): Promise<void>`
-
-Manually applies a bundle's cookies and per-origin web storage to an arbitrary page. Use this only for advanced flows (e.g. opening extra targets mid-crawl). The canonical path is `launchOptions.sessionBundle`, which also restores the user-data-dir and fingerprintConfig — `restoreSession` cannot swap those on a running browser.
-
-- **Parameters**:
-  - `page` (CdpPage): the target page.
-  - `bundle` (SessionBundle): a bundle produced by `saveSession`.
-
----
+# jumpy-lion