jumpy-lion 0.1.6-beta.3 → 0.1.6-beta.31

package/README.md

@@ -1,607 +1 @@
-# Crawler Documentation
-
-## Table of Contents
-
-- [Overview](#overview)
-- [NPM Package](#npm-package)
-- [Usage](#usage)
-  - [Example Project](#example-project)
-  - [Internal Guide](#internal-guide)
-  - [Examples and Configuration](#examples-and-configuration)
-  - [Advanced Fingerprints Usage](#advanced-fingerprints-usage)
-  - [Syncing BrowserPool and launchOptions fingerprints](#syncing-browserpool-and-launchoptions-fingerprints)
-  - [Stealth Consistency and Network Policies](#stealth-consistency-and-network-policies)
-- [Configurable Fingerprint Options](#configurable-fingerprint-options)
-  - [Usage](#usage-1)
-  - [Available Options](#available-options)
-    - [Core Stealth Options](#core-stealth-options)
-    - [Fingerprint Spoofing](#fingerprint-spoofing)
-    - [Platform Configuration](#platform-configuration)
-    - [Additional Features](#additional-features)
-  - [Default Behavior](#default-behavior)
-  - [Best Practices](#best-practices)
-  - [Performance Considerations](#performance-considerations)
-- [Launch Options for Network and Persistence](#launch-options-for-network-and-persistence)
-- [Crawler Class Documentation](#crawler-class-documentation)
-  - [Constructor](#constructor)
-- [CdpPage Class Documentation](#cdppage-class-documentation)
-  - [Constructor](#constructor-1)
-  - [Static Methods](#static-methods)
-  - [Public Methods](#public-methods)
-- [Utility Functions](#utility-functions)
-  - [createCDPRouter](#createcdprouter)
-
-## Overview
-
-The `Crawler` class is a custom implementation of the `BrowserCrawler` from Crawlee, designed to utilize the Chrome DevTools Protocol (CDP) for advanced antiblocking capabilities.
-
-## NPM Package
-
-The `jumpy-lion` is official cdp crawler package. See it [here](https://www.npmjs.com/package/jumpy-lion).
-
-### Installation
-
-```bash
-npm install jumpy-lion
-```
-
-A postinstall hook pulls the matching anti-detect Chromium build from a public
-Apify key-value store — no GitHub token, no Apify token. Works the same on
-your workstation, CI, and the Apify platform. If the download fails for any
-reason (offline, proxy, etc.) the install still succeeds; the crawler falls
-back to the system Chrome on `PATH`. See `browser/BUILD.md` for env-var
-overrides and the maintainer release flow.
-
----
----
-
-## Usage
-
-### Example Project
-
-Refer to this [GitHub repository](https://github.com/apify-projects/cdp-crawler-example) for a complete example of using the `Crawler` class.
-
-### Internal Guide
-
-Check out the [CDP Crawler internal guide](https://www.notion.so/apify/CDP-Crawler-internal-guide-183f39950a2280be81d7c86dc048a47a?pvs=4) for tutorial.
-
-### Examples and Configuration
-
-For detailed examples and configuration patterns, see the [Examples README](./examples/README.md). The examples include:
-
-- **Basic Configuration**: Simple fingerprint setup for common use cases
-- **Comprehensive Configuration**: Full feature setup with all spoofing options
-- **Platform-Specific Configurations**: macOS, Windows, and Linux targeting
-- **Performance-Focused Configuration**: Optimized settings for speed
-- **Minimal Configuration**: Using intelligent defaults
-
-The examples demonstrate real-world usage patterns and best practices for different scenarios.
-
-### Advanced Fingerprints usage
-
-To use advanced fingerprints, you need to set the `useExperimentalFingerprints` option to `true` in the `launchContext.launchOptions` of the `Crawler` constructor.
-
-```typescript
-const crawler = new Crawler({
-    launchContext: {
-      launchOptions: {
-        useExperimentalFingerprints: true,
-      }
-    },
-});
-```
-
----
-
-### Syncing BrowserPool and launchOptions fingerprints
-
-**Always keep the operating system in sync between BrowserPool fingerprints and `launchOptions.fingerprintOptions`.** A mismatch can lead to inconsistent signals (for example `navigator.platform`, User-Agent, WebGL, fonts) and reduce antibot effectiveness.
-
-- **launchOptions side**: Set `launchContext.launchOptions.fingerprintOptions.platform` to the desired platform string.
-- **BrowserPool side**: When `browserPoolOptions.useFingerprints` is `true`, set `browserPoolOptions.fingerprintOptions.fingerprintGeneratorOptions.operatingSystems` to the corresponding OS.
-
-Mapping guidance:
-- `platform: 'Win32'` ↔ `operatingSystems: ['windows']`
-- `platform: 'MacIntel'` ↔ `operatingSystems: ['macos']`
-- `platform: 'Linux x86_64'` ↔ `operatingSystems: ['linux']`
-
-Example:
-
-```typescript
-const crawler = new Crawler({
-  launchContext: {
-    launchOptions: {
-      useExperimentalFingerprints: true,
-      fingerprintOptions: {
-        platform: 'Win32', // Keep this in sync with BrowserPool OS
-      },
-    },
-  },
-  browserPoolOptions: {
-    useFingerprints: true,
-    fingerprintOptions: {
-      fingerprintGeneratorOptions: {
-        browsers: ['chrome'],
-        operatingSystems: ['windows'], // Matches platform: 'Win32'
-        devices: ['desktop'],
-      },
-    },
-  },
-});
-```
-
-Note: This configuration surface will be unified later. We are currently testing our custom fingerprint injector so it works even with the BrowserPool built‑in fingerprints turned off. If you prefer, you can rely solely on the custom injector by setting `browserPoolOptions.useFingerprints: false` and keeping `launchOptions.useExperimentalFingerprints: true`.
-
----
-
-### Stealth Consistency and Network Policies
-
-Recent stealth hardening adds explicit consistency and policy controls:
-
-- **UA/Binary version alignment**: the injector aligns advertised `Chrome/x.y.z.w` with the actual running Chrome binary version to reduce fingerprint drift.
-- **WebRTC policy control**: set `fingerprintOptions.webRtcPolicy` to:
-  - `'spoof'` (default): redacts/normalizes WebRTC leak surfaces.
-  - `'disable'`: removes WebRTC APIs from page context.
-- **DNS hardening controls**: configure DoH and secure DNS through launch options (`dnsOverHttpsServer`, `secureDnsMode`).
-- **WebRTC transport policy flag**: configure `webrtcIpHandlingPolicy` at browser launch level.
-- **Persistent profile mode**: set `userDataDir` (+ optional `keepUserDataDir`) to reuse browser state across runs.
-
-## Configurable Fingerprint Options
-
-The CDP crawler supports configurable fingerprint options that can be passed through the crawler options. This allows you to customize the fingerprint spoofing behavior for different use cases.
-
-### Usage
-
-You can configure fingerprint options by adding them to the `launchContext.launchOptions.fingerprintOptions` in your crawler configuration:
-
-```typescript
-import { Crawler } from 'cdp-crawler';
-
-const crawler = new Crawler({
-    launchContext: {
-        launchOptions: {
-            fingerprintOptions: {
-                // Enable advanced stealth features
-                enableAdvancedStealth: true,
-                
-                // Bypass Runtime.enable detection
-                bypassRuntimeEnable: true,
-                
-                // Humanize mouse interactions
-                humanizeInteractions: true,
-                
-                // Spoof WebGL fingerprinting
-                spoofWebGL: true,
-                
-                // Spoof audio context fingerprinting
-                spoofAudioContext: true,
-                
-                // Add variations to client rect measurements
-                spoofClientRects: true,
-                
-                // Mask automation flags
-                maskAutomationFlags: true,
-                
-                // Use fingerprint-generator defaults when available
-                useFingerprintDefaults: true,
-                
-                // Platform to spoof (defaults to Win32 for better evasion)
-                platform: 'Win32', // 'Win32' | 'MacIntel' | 'Linux x86_64'
-                
-                // Spoof font measurements
-                spoofFonts: true,
-                
-                // Spoof performance timing
-                spoofPerformance: true,
-                
-                // Spoof locale settings
-                spoofLocale: true,
-                
-                // Detect timezone from proxy (useful with residential proxies)
-                detectTimezone: true,
-
-                // WebRTC policy: 'spoof' (default) or 'disable'
-                webRtcPolicy: 'spoof',
-            }
-        }
-    },
-    // ... other crawler options
-});
-```
-
-### Available Options
-
-#### Core Stealth Options
-
-- **`enableAdvancedStealth`** (boolean): Enables advanced stealth features including WebGPU spoofing and platform consistency
-- **`bypassRuntimeEnable`** (boolean): Prevents CDP detection through Runtime.enable bypass techniques
-- **`humanizeInteractions`** (boolean): Generates human-like mouse movements using bezier curves
-
-#### Fingerprint Spoofing
-
-- **`spoofWebGL`** (boolean): Spoofs WebGL fingerprinting by modifying GPU adapter information
-- **`spoofAudioContext`** (boolean): Adds noise to audio processing to prevent audio fingerprinting
-- **`spoofClientRects`** (boolean): Adds small variations to getBoundingClientRect results
-- **`spoofFonts`** (boolean): Hides platform-specific fonts and adds font measurement variations
-- **`spoofPerformance`** (boolean): Modifies timing characteristics to match the target platform
-- **`spoofLocale`** (boolean): Ensures consistent locale formatting across all browser properties
-
-#### Platform Configuration
-
-- **`platform`** (string): Target platform to spoof. Options: `'Win32'`, `'MacIntel'`, `'Linux x86_64'`
-- **`useFingerprintDefaults`** (boolean): Use hardcoded defaults instead of fingerprint-generator values. When `false`, uses generated fingerprint values; when `true` (default), uses hardcoded defaults
-
-#### Additional Features
-
-- **`maskAutomationFlags`** (boolean): Masks automation-related flags in the browser
-- **`detectTimezone`** (boolean): Automatically detect timezone from proxy IP (useful with residential proxies)
-- **`webRtcPolicy`** (`'spoof' | 'disable'`): Controls whether WebRTC is spoofed or fully removed from page APIs
-
-### Default Behavior
-
-When no fingerprint options are provided, the crawler uses intelligent defaults:
-
-- **On Apify**: Uses Apify-recommended settings optimized for the Apify environment
-- **On other platforms**: Uses a comprehensive set of stealth features with Windows platform spoofing
-- **Humanization defaults**: mouse, keyboard, and scroll humanization are enabled with safe defaults
-- **UA consistency**: claimed UA Chrome version is automatically aligned to the running Chrome binary
-
-### Best Practices
-
-1. **Use `platform: 'Win32'`** for better evasion on Linux servers (like Apify)
-2. **Enable `detectTimezone: true`** when using residential proxies
-3. **Use `useFingerprintDefaults: false`** to leverage fingerprint-generator's realistic values
-4. **Enable `bypassRuntimeEnable: true`** for sites that detect automation
-5. **Use `enableAdvancedStealth: true`** for maximum protection against fingerprinting
-6. **Keep OS settings in sync** between `launchOptions.fingerprintOptions.platform` and `browserPoolOptions.fingerprintOptions.fingerprintGeneratorOptions.operatingSystems`
-7. **Use `webRtcPolicy: 'disable'`** for strictest leak prevention, or `'spoof'` for compatibility-sensitive targets
-
-### Performance Considerations
-
-- More fingerprint options enabled = slightly higher CPU usage
-- WebGPU spoofing may add a small delay to page loads
-- Humanized interactions add realistic delays to mouse movements
-
-The fingerprint options are designed to provide maximum protection while maintaining good performance for web scraping tasks.
-
-For more configuration examples and patterns, see the [Examples README](./examples/README.md).
-
----
-
-## Launch Options for Network and Persistence
-
-The following options are configured in `launchContext.launchOptions`:
-
-- **`dnsOverHttpsServer`** (string): DoH endpoint template, for example `https://cloudflare-dns.com/dns-query`
-- **`secureDnsMode`** (`'off' | 'automatic' | 'secure'`): Chromium secure DNS mode
-- **`webrtcIpHandlingPolicy`** (`'default' | 'default_public_interface_only' | 'default_public_and_private_interfaces' | 'disable_non_proxied_udp'`): Browser-level WebRTC IP handling policy
-- **`userDataDir`** (string): Reuse a specific Chrome profile directory across runs
-- **`keepUserDataDir`** (boolean): Keep/cleanup profile directory on close (defaults to keep custom dir, cleanup temp dir)
-
-Example:
-
-```typescript
-const crawler = new Crawler({
-  launchContext: {
-    launchOptions: {
-      dnsOverHttpsServer: 'https://cloudflare-dns.com/dns-query',
-      secureDnsMode: 'secure',
-      webrtcIpHandlingPolicy: 'disable_non_proxied_udp',
-      userDataDir: './state/chrome-profile',
-      keepUserDataDir: true,
-      fingerprintOptions: {
-        webRtcPolicy: 'disable',
-      },
-    },
-  },
-});
-```
-
----
-
-## `Crawler` Class Documentation
-
-### Constructor
-
-#### `constructor(options: BrowserCrawlerOptions = {}, override readonly config = Configuration.getGlobalConfig())`
-
-Initializes the `Crawler` instance with default and provided options.
-
-- **Parameters**:
-
-  - `options` (BrowserCrawlerOptions): Configuration options for the crawler.
-    - `launchContext`: Specifies browser launch parameters.
-      - Default: `{}`
-    - `headless`: Runs the browser in headless mode.
-      - Default: `false`
-    - `browserPoolOptions`: Configuration for managing browser instances.
-  - `config` (Configuration): Global Crawlee configuration.
-    - Default: `Configuration.getGlobalConfig()`
-
-- **Default Behavior**:
-  - Throws an error if `launchContext.proxyUrl` is provided. Use `proxyConfiguration` instead.
-  - Throws an error if `browserPoolOptions.browserPlugins` is set. Use `launchContext.launcher` instead.
-
----
-
-## `CdpPage` Class Documentation
-
-### Constructor
-
-#### `constructor(client: CDP.Client)`
-
-Initializes the `CdpPage` instance with a CDP client.
-
-- **Parameters**:
-
-  - `client` (CDP.Client): The Chrome DevTools Protocol client.
-
-- **Emitted Events**:
-  - `PAGE_CREATED`: Triggered upon the creation of the page.
-
-### Static Methods
-
-#### `static async create(client: CDP.Client): Promise<CdpPage>`
-
-Creates and initializes a new `CdpPage` instance.
-
-- **Parameters**:
-
-  - `client` (CDP.Client): The CDP client.
-
-- **Returns**:
-  - `Promise<CdpPage>`: A promise resolving to the new `CdpPage` instance.
-
----
-
-### Public Methods
-
-#### `async url(): Promise<string>`
-Gets the current URL of the page.
-
-- **Returns**:
-  - `Promise<string>`: The current URL.
-
-#### `async goto(url: string, options?: GotoOptions): Promise<void>`
-Navigates to a specified URL.
-
-- **Parameters**:
-  - `url` (string): The URL to navigate to.
-  - `options` (GotoOptions): Navigation options, including:
-    - `waitUntil`: When to consider navigation finished (`domcontentloaded` or `load`).
-    - `timeout`: Maximum time to wait for navigation in milliseconds.
-
-#### `async click(selector: string): Promise<void>`
-Simulates a click on an element identified by the selector.
-
-- **Parameters**:
-  - `selector` (string): CSS selector of the element.
-
-#### `async type(selector: string, text: string, options?: { delay?: number }): Promise<void>`
-Types text into an input field.
-
-- **Parameters**:
-  - `selector` (string): CSS selector of the element.
-  - `text` (string): Text to type.
-  - `options` (object): Options for typing:
-    - `delay`: Time in milliseconds between key presses.
-
-#### `async screenshot(options?: { path?: string; fullPage?: boolean; format?: 'png' | 'jpeg' }): Promise<Buffer>`
-Takes a screenshot of the page, with support for PNG and JPEG formats.
-
-- **Parameters**:
-  - `options` (object): Screenshot options:
-    - `path`: File path to save the screenshot.
-    - `fullPage`: Capture the entire page.
-    - `format`: Image format, either `'png'` (default) or `'jpeg'`.
-
-- **Returns**:
-  - `Promise<Buffer>`: The screenshot as a buffer.
-
-#### `async content(): Promise<string>`
-Gets the HTML content of the page.
-
-- **Returns**:
-  - `Promise<string>`: The page's HTML.
-
-#### `async toCheerio(): Promise<cheerio.CheerioAPI>`
-Converts the current page content to a Cheerio instance for DOM manipulation.
-
-- **Returns**:
-  - `Promise<cheerio.CheerioAPI>`: A Cheerio API instance.
-
-#### `async setViewport(viewport: Viewport): Promise<void>`
-Sets the page's viewport dimensions.
-
-- **Parameters**:
-  - `viewport` (Viewport): Object with `width` and `height` properties.
-
-#### `async setUserAgent(userAgent: string): Promise<void>`
-Overrides the user-agent string.
-
-- **Parameters**:
-  - `userAgent` (string): The new user-agent string.
-
-#### `async setExtraHTTPHeaders(headers: Record<string, string>): Promise<void>`
-Sets additional HTTP headers for requests.
-
-- **Parameters**:
-  - `headers` (Record<string, string>): Key-value pairs of headers.
-
-#### `async waitForResponse(urlPart: string, statusCode?: number, timeout?: number): Promise<any>`
-Waits for a specific network response.
-
-- **Parameters**:
-  - `urlPart` (string): Part of the URL to match.
-  - `statusCode` (number): Expected HTTP status code.
-  - `timeout` (number): Maximum wait time in milliseconds.
-
-- **Returns**:
-  - `Promise<any>`: The response.
-
-#### `async setCookies(cookies: Cookie[]): Promise<void>`
-Sets cookies for the page.
-
-- **Parameters**:
-  - `cookies` (Cookie[]): Array of cookies to set.
-
-#### `async getCookies(urls?: string[]): Promise<Cookie[]>`
-Retrieves cookies for the given URLs or all cookies if no URLs are specified.
-
-- **Parameters**:
-  - `urls` (string[]): Optional array of URLs.
-
-- **Returns**:
-  - `Promise<Cookie[]>`: Array of cookies.
-
-#### `async waitForSelector(selector: string, options?: { timeout?: number }): Promise<void>`
-Waits for an element matching the selector to appear.
-
-- **Parameters**:
-  - `selector` (string): CSS selector of the element.
-  - `options` (object): Options for waiting:
-    - `timeout`: Maximum wait time in milliseconds.
-
-#### `async elementExists(selector: string): Promise<boolean>`
-Checks if an element exists.
-
-- **Parameters**:
-  - `selector` (string): CSS selector of the element.
-
-- **Returns**:
-  - `Promise<boolean>`: `true` if the element exists, `false` otherwise.
-
-#### `async getTextContent(selector: string): Promise<string>`
-Gets the text content of an element.
-
-- **Parameters**:
-  - `selector` (string): CSS selector of the element.
-
-- **Returns**:
-  - `Promise<string>`: The element's text content.
-
-#### `async getHref(selector: string): Promise<string>`
-Gets the `href` attribute of an anchor element.
-
-- **Parameters**:
-  - `selector` (string): CSS selector of the anchor element.
-
-- **Returns**:
-  - `Promise<string>`: The `href` value.
-
-#### `async reload(options?: GotoOptions): Promise<void>`
-Reloads the current page.
-
-- **Parameters**:
-  - `options` (GotoOptions): Navigation options, including:
-    - `waitUntil`: When to consider reload finished (`domcontentloaded` or `load`).
-    - `timeout`: Maximum time to wait for reload in milliseconds.
-
-#### `async deleteInput(selector: string): Promise<void>`
-Clears the value of an input field specified by the selector.
-
-- **Parameters**:
-  - `selector` (string): CSS selector of the input element.
-
-#### `async isVisible(selector: string): Promise<boolean>`
-Checks if the element specified by selector is visible (not `display: none` and not `visibility: hidden`).
-The selector should be the root item which can be hidden, otherwise this function could return a false positive.
-
-- **Parameters**:
-  - `selector` (string): CSS selector of the element.
-- **Returns**:
-  - `Promise<boolean>`: `true` if the element is visible, `false` otherwise.
-
-#### `async selectOption(dropdownSelector: string, optionSelector: string | string[], options?: SelectOptionOptions): Promise<void>`
-Selects one or more options from a select element or dropdown with intelligent automatic handling.
-
-**Key Features**:
-- **Automatic Detection**: Distinguishes between HTML `<select>` elements and custom dropdowns
-- **Smart Trigger Discovery**: For custom dropdowns, automatically finds and clicks triggers using multiple strategies
-- **Virtualized List Support**: Handles large dropdown lists with intelligent scrolling
-- **No Manual Configuration**: No need to specify separate trigger and container selectors
-
-- **Parameters**:
-  - `dropdownSelector` (string): CSS selector for the select element or dropdown container.
-  - `optionSelector` (string | string[]): CSS selector(s) for the option(s) to select. Can be a single selector or array of selectors.
-  - `options` (SelectOptionOptions): Optional configuration object with the following properties:
-    - `timeout` (number): Maximum wait time in milliseconds. Default: 30000.
-    - `force` (boolean): Bypass visibility and disabled checks. Default: false.
-    - `waitForOptions` (boolean): Wait for dropdown options to load. Default: true.
-    - `maxScrollAttempts` (number): Maximum scroll attempts for virtualized dropdowns. Default: 10.
-
-#### `async waitForElementPositionToStabilize(selector: string, timeout?: number, checkInterval?: number, stabilityThreshold?: number, tolerance?: number): Promise<void>`
-Waits for an element's position to stabilize by polling its bounding box. Useful before interactions after scrolling/animations.
-
-- **Parameters**:
-  - `selector` (string): Target element selector
-  - `timeout` (number): Max time to wait. Default: 2000
-  - `checkInterval` (number): Polling interval. Default: 100
-  - `stabilityThreshold` (number): Consecutive stable checks required. Default: 3
-  - `tolerance` (number): Max pixel delta to consider stable. Default: 1
-
-- **Usage Examples**:
-  ```typescript
-  // Regular HTML select element - works directly
-  await page.selectOption('select#country', 'option[value="us"]');
-  
-  // Multiple selection in HTML select
-  await page.selectOption('select#languages', ['option[value="en"]', 'option[value="es"]']);
-  
-  // Custom dropdown - automatically finds and clicks trigger
-  await page.selectOption('#dropdown-menu', '[data-value="premium"]');
-  
-  // Virtualized dropdown - automatically scrolls to find option
-  await page.selectOption('#large-dropdown', '[data-item="item-500"]');
-  
-  // With custom configuration
-  await page.selectOption(
-    '#complex-dropdown',
-    '.option[data-category="business"]',
-    {
-      timeout: 10000,
-      maxScrollAttempts: 15
-    }
-  );
-  
-  // Bootstrap/Material-UI dropdowns work automatically
-  await page.selectOption('.MuiSelect-menu', '[data-value="option1"]');
-  await page.selectOption('.dropdown-menu', '.dropdown-item[data-value="choice2"]');
-  ```
-
-- **How Trigger Detection Works**:
-  The method automatically detects dropdown triggers using multiple strategies:
-  1. **Accessibility patterns**: `[aria-haspopup]`, `[role="button"]`
-  2. **Common class names**: `.dropdown-trigger`, `.select-trigger`
-  3. **Sibling elements**: Previous sibling of the dropdown container
-  4. **ID pattern matching**: `#menu-id` → `#trigger-id`, `#dropdown-menu` → `#dropdown-trigger`
-
-- **Migration from Previous API**:
-  ```typescript
-  // OLD - Complex API with manual configuration
-  await page.selectOption('#trigger', '[data-value="item"]', {
-    dropdownSelector: '#menu',
-    optionSelector: '.dropdown-item'
-  });
-  
-  // NEW - Simplified API with automatic detection
-  await page.selectOption('#menu', '[data-value="item"]');
-  ```
-
----
-
-## Utility Functions
-
-### `createCDPRouter`
-
-#### `export function createCDPRouter<Context extends CDPCrawlingContext = CDPCrawlingContext, UserData extends Dictionary = GetUserDataFromRequest<Context['request']>>(routes?: RouterRoutes<Context, UserData>): Router<Context>`
-
-Creates a custom router for handling crawling routes using CDP.
-
-- **Parameters**:
-  - `routes` (RouterRoutes<Context, UserData>): Optional routes for defining crawl logic.
-
-- **Returns**:
-  - `Router<Context>`: A configured router instance.
-
----
+# jumpy-lion