jumpy-lion 0.0.34 → 0.0.36

package/README.md

@@ -0,0 +1,510 @@
+# Crawler Documentation
+
+## Table of Contents
+
+- [Overview](#overview)
+- [NPM Package](#npm-package)
+- [Usage](#usage)
+  - [Example Project](#example-project)
+  - [Internal Guide](#internal-guide)
+  - [Examples and Configuration](#examples-and-configuration)
+  - [Advanced Fingerprints Usage](#advanced-fingerprints-usage)
+  - [Syncing BrowserPool and launchOptions fingerprints](#syncing-browserpool-and-launchoptions-fingerprints)
+- [Configurable Fingerprint Options](#configurable-fingerprint-options)
+  - [Usage](#usage-1)
+  - [Available Options](#available-options)
+    - [Core Stealth Options](#core-stealth-options)
+    - [Fingerprint Spoofing](#fingerprint-spoofing)
+    - [Platform Configuration](#platform-configuration)
+    - [Additional Features](#additional-features)
+  - [Default Behavior](#default-behavior)
+  - [Best Practices](#best-practices)
+  - [Performance Considerations](#performance-considerations)
+- [Crawler Class Documentation](#crawler-class-documentation)
+  - [Constructor](#constructor)
+- [CdpPage Class Documentation](#cdppage-class-documentation)
+  - [Constructor](#constructor-1)
+  - [Static Methods](#static-methods)
+  - [Public Methods](#public-methods)
+- [Utility Functions](#utility-functions)
+  - [createCDPRouter](#createcdprouter)
+
+## Overview
+
+The `Crawler` class is a custom implementation of the `BrowserCrawler` from Crawlee, designed to utilize the Chrome DevTools Protocol (CDP) for advanced antiblocking capabilities.
+
+## NPM Package
+
+The `jumpy-lion` is official cdp crawler package. See it [here](https://www.npmjs.com/package/jumpy-lion).
+
+---
+---
+
+## Usage
+
+### Example Project
+
+Refer to this [GitHub repository](https://github.com/apify-projects/cdp-crawler-example) for a complete example of using the `Crawler` class.
+
+### Internal Guide
+
+Check out the [CDP Crawler internal guide](https://www.notion.so/apify/CDP-Crawler-internal-guide-183f39950a2280be81d7c86dc048a47a?pvs=4) for tutorial.
+
+### Examples and Configuration
+
+For detailed examples and configuration patterns, see the [Examples README](./examples/README.md). The examples include:
+
+- **Basic Configuration**: Simple fingerprint setup for common use cases
+- **Comprehensive Configuration**: Full feature setup with all spoofing options
+- **Platform-Specific Configurations**: macOS, Windows, and Linux targeting
+- **Performance-Focused Configuration**: Optimized settings for speed
+- **Minimal Configuration**: Using intelligent defaults
+
+The examples demonstrate real-world usage patterns and best practices for different scenarios.
+
+### Advanced Fingerprints usage
+
+To use advanced fingerprints, you need to set the `useExperimentalFingerprints` option to `true` in the `launchContext.launchOptions` of the `Crawler` constructor.
+
+```typescript
+const crawler = new Crawler({
+    launchContext: {
+      launchOptions: {
+        useExperimentalFingerprints: true,
+      }
+    },
+});
+```
+
+---
+
+### Syncing BrowserPool and launchOptions fingerprints
+
+**Always keep the operating system in sync between BrowserPool fingerprints and `launchOptions.fingerprintOptions`.** A mismatch can lead to inconsistent signals (for example `navigator.platform`, User-Agent, WebGL, fonts) and reduce antibot effectiveness.
+
+- **launchOptions side**: Set `launchContext.launchOptions.fingerprintOptions.platform` to the desired platform string.
+- **BrowserPool side**: When `browserPoolOptions.useFingerprints` is `true`, set `browserPoolOptions.fingerprintOptions.fingerprintGeneratorOptions.operatingSystems` to the corresponding OS.
+
+Mapping guidance:
+- `platform: 'Win32'` ↔ `operatingSystems: ['windows']`
+- `platform: 'MacIntel'` ↔ `operatingSystems: ['macos']`
+- `platform: 'Linux x86_64'` ↔ `operatingSystems: ['linux']`
+
+Example:
+
+```typescript
+const crawler = new Crawler({
+  launchContext: {
+    launchOptions: {
+      useExperimentalFingerprints: true,
+      fingerprintOptions: {
+        platform: 'Win32', // Keep this in sync with BrowserPool OS
+      },
+    },
+  },
+  browserPoolOptions: {
+    useFingerprints: true,
+    fingerprintOptions: {
+      fingerprintGeneratorOptions: {
+        browsers: ['chrome'],
+        operatingSystems: ['windows'], // Matches platform: 'Win32'
+        devices: ['desktop'],
+      },
+    },
+  },
+});
+```
+
+Note: This configuration surface will be unified later. We are currently testing our custom fingerprint injector so it works even with the BrowserPool built‑in fingerprints turned off. If you prefer, you can rely solely on the custom injector by setting `browserPoolOptions.useFingerprints: false` and keeping `launchOptions.useExperimentalFingerprints: true`.
+
+---
+
+## Configurable Fingerprint Options
+
+The CDP crawler supports configurable fingerprint options that can be passed through the crawler options. This allows you to customize the fingerprint spoofing behavior for different use cases.
+
+### Usage
+
+You can configure fingerprint options by adding them to the `launchContext.launchOptions.fingerprintOptions` in your crawler configuration:
+
+```typescript
+import { Crawler } from 'cdp-crawler';
+
+const crawler = new Crawler({
+    launchContext: {
+        launchOptions: {
+            fingerprintOptions: {
+                // Enable advanced stealth features
+                enableAdvancedStealth: true,
+                
+                // Bypass Runtime.enable detection
+                bypassRuntimeEnable: true,
+                
+                // Humanize mouse interactions
+                humanizeInteractions: true,
+                
+                // Spoof WebGL fingerprinting
+                spoofWebGL: true,
+                
+                // Spoof audio context fingerprinting
+                spoofAudioContext: true,
+                
+                // Add variations to client rect measurements
+                spoofClientRects: true,
+                
+                // Mask automation flags
+                maskAutomationFlags: true,
+                
+                // Use fingerprint-generator defaults when available
+                useFingerprintDefaults: true,
+                
+                // Platform to spoof (defaults to Win32 for better evasion)
+                platform: 'Win32', // 'Win32' | 'MacIntel' | 'Linux x86_64'
+                
+                // Spoof font measurements
+                spoofFonts: true,
+                
+                // Spoof performance timing
+                spoofPerformance: true,
+                
+                // Spoof locale settings
+                spoofLocale: true,
+                
+                // Detect timezone from proxy (useful with residential proxies)
+                detectTimezone: true,
+            }
+        }
+    },
+    // ... other crawler options
+});
+```
+
+### Available Options
+
+#### Core Stealth Options
+
+- **`enableAdvancedStealth`** (boolean): Enables advanced stealth features including WebGPU spoofing and platform consistency
+- **`bypassRuntimeEnable`** (boolean): Prevents CDP detection through Runtime.enable bypass techniques
+- **`humanizeInteractions`** (boolean): Generates human-like mouse movements using bezier curves
+
+#### Fingerprint Spoofing
+
+- **`spoofWebGL`** (boolean): Spoofs WebGL fingerprinting by modifying GPU adapter information
+- **`spoofAudioContext`** (boolean): Adds noise to audio processing to prevent audio fingerprinting
+- **`spoofClientRects`** (boolean): Adds small variations to getBoundingClientRect results
+- **`spoofFonts`** (boolean): Hides platform-specific fonts and adds font measurement variations
+- **`spoofPerformance`** (boolean): Modifies timing characteristics to match the target platform
+- **`spoofLocale`** (boolean): Ensures consistent locale formatting across all browser properties
+
+#### Platform Configuration
+
+- **`platform`** (string): Target platform to spoof. Options: `'Win32'`, `'MacIntel'`, `'Linux x86_64'`
+- **`useFingerprintDefaults`** (boolean): Use hardcoded defaults instead of fingerprint-generator values. When `false`, uses generated fingerprint values; when `true` (default), uses hardcoded defaults
+
+#### Additional Features
+
+- **`maskAutomationFlags`** (boolean): Masks automation-related flags in the browser
+- **`detectTimezone`** (boolean): Automatically detect timezone from proxy IP (useful with residential proxies)
+
+### Default Behavior
+
+When no fingerprint options are provided, the crawler uses intelligent defaults:
+
+- **On Apify**: Uses Apify-recommended settings optimized for the Apify environment
+- **On other platforms**: Uses a comprehensive set of stealth features with Windows platform spoofing
+
+### Best Practices
+
+1. **Use `platform: 'Win32'`** for better evasion on Linux servers (like Apify)
+2. **Enable `detectTimezone: true`** when using residential proxies
+3. **Use `useFingerprintDefaults: false`** to leverage fingerprint-generator's realistic values
+4. **Enable `bypassRuntimeEnable: true`** for sites that detect automation
+5. **Use `enableAdvancedStealth: true`** for maximum protection against fingerprinting
+6. **Keep OS settings in sync** between `launchOptions.fingerprintOptions.platform` and `browserPoolOptions.fingerprintOptions.fingerprintGeneratorOptions.operatingSystems`
+
+### Performance Considerations
+
+- More fingerprint options enabled = slightly higher CPU usage
+- WebGPU spoofing may add a small delay to page loads
+- Humanized interactions add realistic delays to mouse movements
+
+The fingerprint options are designed to provide maximum protection while maintaining good performance for web scraping tasks.
+
+For more configuration examples and patterns, see the [Examples README](./examples/README.md).
+
+---
+
+## `Crawler` Class Documentation
+
+### Constructor
+
+#### `constructor(options: BrowserCrawlerOptions = {}, override readonly config = Configuration.getGlobalConfig())`
+
+Initializes the `Crawler` instance with default and provided options.
+
+- **Parameters**:
+
+  - `options` (BrowserCrawlerOptions): Configuration options for the crawler.
+    - `launchContext`: Specifies browser launch parameters.
+      - Default: `{}`
+    - `headless`: Runs the browser in headless mode.
+      - Default: `false`
+    - `browserPoolOptions`: Configuration for managing browser instances.
+  - `config` (Configuration): Global Crawlee configuration.
+    - Default: `Configuration.getGlobalConfig()`
+
+- **Default Behavior**:
+  - Throws an error if `launchContext.proxyUrl` is provided. Use `proxyConfiguration` instead.
+  - Throws an error if `browserPoolOptions.browserPlugins` is set. Use `launchContext.launcher` instead.
+
+---
+
+## `CdpPage` Class Documentation
+
+### Constructor
+
+#### `constructor(client: CDP.Client)`
+
+Initializes the `CdpPage` instance with a CDP client.
+
+- **Parameters**:
+
+  - `client` (CDP.Client): The Chrome DevTools Protocol client.
+
+- **Emitted Events**:
+  - `PAGE_CREATED`: Triggered upon the creation of the page.
+
+### Static Methods
+
+#### `static async create(client: CDP.Client): Promise<CdpPage>`
+
+Creates and initializes a new `CdpPage` instance.
+
+- **Parameters**:
+
+  - `client` (CDP.Client): The CDP client.
+
+- **Returns**:
+  - `Promise<CdpPage>`: A promise resolving to the new `CdpPage` instance.
+
+---
+
+### Public Methods
+
+#### `async url(): Promise<string>`
+Gets the current URL of the page.
+
+- **Returns**:
+  - `Promise<string>`: The current URL.
+
+#### `async goto(url: string, options?: GotoOptions): Promise<void>`
+Navigates to a specified URL.
+
+- **Parameters**:
+  - `url` (string): The URL to navigate to.
+  - `options` (GotoOptions): Navigation options, including:
+    - `waitUntil`: When to consider navigation finished (`domcontentloaded` or `load`).
+    - `timeout`: Maximum time to wait for navigation in milliseconds.
+
+#### `async click(selector: string): Promise<void>`
+Simulates a click on an element identified by the selector.
+
+- **Parameters**:
+  - `selector` (string): CSS selector of the element.
+
+#### `async type(selector: string, text: string, options?: { delay?: number }): Promise<void>`
+Types text into an input field.
+
+- **Parameters**:
+  - `selector` (string): CSS selector of the element.
+  - `text` (string): Text to type.
+  - `options` (object): Options for typing:
+    - `delay`: Time in milliseconds between key presses.
+
+#### `async screenshot(options?: { path?: string; fullPage?: boolean; format?: 'png' | 'jpeg' }): Promise<Buffer>`
+Takes a screenshot of the page, with support for PNG and JPEG formats.
+
+- **Parameters**:
+  - `options` (object): Screenshot options:
+    - `path`: File path to save the screenshot.
+    - `fullPage`: Capture the entire page.
+    - `format`: Image format, either `'png'` (default) or `'jpeg'`.
+
+- **Returns**:
+  - `Promise<Buffer>`: The screenshot as a buffer.
+
+#### `async content(): Promise<string>`
+Gets the HTML content of the page.
+
+- **Returns**:
+  - `Promise<string>`: The page's HTML.
+
+#### `async toCheerio(): Promise<cheerio.CheerioAPI>`
+Converts the current page content to a Cheerio instance for DOM manipulation.
+
+- **Returns**:
+  - `Promise<cheerio.CheerioAPI>`: A Cheerio API instance.
+
+#### `async setViewport(viewport: Viewport): Promise<void>`
+Sets the page's viewport dimensions.
+
+- **Parameters**:
+  - `viewport` (Viewport): Object with `width` and `height` properties.
+
+#### `async setUserAgent(userAgent: string): Promise<void>`
+Overrides the user-agent string.
+
+- **Parameters**:
+  - `userAgent` (string): The new user-agent string.
+
+#### `async setExtraHTTPHeaders(headers: Record<string, string>): Promise<void>`
+Sets additional HTTP headers for requests.
+
+- **Parameters**:
+  - `headers` (Record<string, string>): Key-value pairs of headers.
+
+#### `async waitForResponse(urlPart: string, statusCode?: number, timeout?: number): Promise<any>`
+Waits for a specific network response.
+
+- **Parameters**:
+  - `urlPart` (string): Part of the URL to match.
+  - `statusCode` (number): Expected HTTP status code.
+  - `timeout` (number): Maximum wait time in milliseconds.
+
+- **Returns**:
+  - `Promise<any>`: The response.
+
+#### `async setCookies(cookies: Cookie[]): Promise<void>`
+Sets cookies for the page.
+
+- **Parameters**:
+  - `cookies` (Cookie[]): Array of cookies to set.
+
+#### `async getCookies(urls?: string[]): Promise<Cookie[]>`
+Retrieves cookies for the given URLs or all cookies if no URLs are specified.
+
+- **Parameters**:
+  - `urls` (string[]): Optional array of URLs.
+
+- **Returns**:
+  - `Promise<Cookie[]>`: Array of cookies.
+
+#### `async waitForSelector(selector: string, options?: { timeout?: number }): Promise<void>`
+Waits for an element matching the selector to appear.
+
+- **Parameters**:
+  - `selector` (string): CSS selector of the element.
+  - `options` (object): Options for waiting:
+    - `timeout`: Maximum wait time in milliseconds.
+
+#### `async elementExists(selector: string): Promise<boolean>`
+Checks if an element exists.
+
+- **Parameters**:
+  - `selector` (string): CSS selector of the element.
+
+- **Returns**:
+  - `Promise<boolean>`: `true` if the element exists, `false` otherwise.
+
+#### `async getTextContent(selector: string): Promise<string>`
+Gets the text content of an element.
+
+- **Parameters**:
+  - `selector` (string): CSS selector of the element.
+
+- **Returns**:
+  - `Promise<string>`: The element's text content.
+
+#### `async getHref(selector: string): Promise<string>`
+Gets the `href` attribute of an anchor element.
+
+- **Parameters**:
+  - `selector` (string): CSS selector of the anchor element.
+
+- **Returns**:
+  - `Promise<string>`: The `href` value.
+
+#### `async reload(options?: GotoOptions): Promise<void>`
+Reloads the current page.
+
+- **Parameters**:
+  - `options` (GotoOptions): Navigation options, including:
+    - `waitUntil`: When to consider reload finished (`domcontentloaded` or `load`).
+    - `timeout`: Maximum time to wait for reload in milliseconds.
+
+#### `async deleteInput(selector: string): Promise<void>`
+Clears the value of an input field specified by the selector.
+
+- **Parameters**:
+  - `selector` (string): CSS selector of the input element.
+
+#### `async isVisible(selector: string): Promise<boolean>`
+Checks if the element specified by selector is visible (not `display: none` and not `visibility: hidden`).
+The selector should be the root item which can be hidden, otherwise this function could return a false positive.
+
+- **Parameters**:
+  - `selector` (string): CSS selector of the element.
+- **Returns**:
+  - `Promise<boolean>`: `true` if the element is visible, `false` otherwise.
+
+#### `async selectOption(selector: string, targetSelector: string | string[], options?: SelectOptionOptions): Promise<void>`
+Selects one or more options from a select element or dropdown. Supports both regular HTML select elements and custom virtualized dropdowns with hidden options.
+
+- **Parameters**:
+  - `selector` (string): CSS selector for the select element or dropdown trigger.
+  - `targetSelector` (string | string[]): CSS selector(s) for the option(s) to select. Can be a single selector or array of selectors.
+  - `options` (SelectOptionOptions): Optional configuration object with the following properties:
+    - `timeout` (number): Maximum wait time in milliseconds. Default: 30000.
+    - `force` (boolean): Bypass visibility and disabled checks. Default: false.
+    - `waitForOptions` (boolean): Wait for dropdown options to load. Default: true.
+    - `maxScrollAttempts` (number): Maximum scroll attempts for virtualized dropdowns. Default: 10.
+    - `optionSelector` (string): CSS selector for option elements in custom dropdowns (required for non-select elements).
+    - `dropdownSelector` (string): CSS selector for dropdown container in custom dropdowns (required for non-select elements).
+
+#### `async waitForElementPositionToStabilize(selector: string, timeout?: number, checkInterval?: number, stabilityThreshold?: number, tolerance?: number): Promise<void>`
+Waits for an element's position to stabilize by polling its bounding box. Useful before interactions after scrolling/animations.
+
+- **Parameters**:
+  - `selector` (string): Target element selector
+  - `timeout` (number): Max time to wait. Default: 2000
+  - `checkInterval` (number): Polling interval. Default: 100
+  - `stabilityThreshold` (number): Consecutive stable checks required. Default: 3
+  - `tolerance` (number): Max pixel delta to consider stable. Default: 1
+
+- **Usage Examples**:
+  ```typescript
+  // Regular HTML select element
+  await page.selectOption('select#country', 'option[value="us"]');
+  
+  // Multiple selection
+  await page.selectOption('select#languages', ['option[value="en"]', 'option[value="es"]']);
+  
+  // Custom dropdown with specific selectors
+  await page.selectOption(
+    '.dropdown-trigger',
+    '.option[data-value="premium"]',
+    {
+      dropdownSelector: '.dropdown-menu',
+      optionSelector: '.option',
+      timeout: 5000
+    }
+  );
+  ```
+
+---
+
+## Utility Functions
+
+### `createCDPRouter`
+
+#### `export function createCDPRouter<Context extends CDPCrawlingContext = CDPCrawlingContext, UserData extends Dictionary = GetUserDataFromRequest<Context['request']>>(routes?: RouterRoutes<Context, UserData>): Router<Context>`
+
+Creates a custom router for handling crawling routes using CDP.
+
+- **Parameters**:
+  - `routes` (RouterRoutes<Context, UserData>): Optional routes for defining crawl logic.
+
+- **Returns**:
+  - `Router<Context>`: A configured router instance.
+
+---

package/dist/fingerprinting/anti-webgpu/background.js

@@ -1,6 +1,4 @@
-importScripts("lib/config.js");
-importScripts("lib/chrome.js");
-importScripts("lib/runtime.js");
-importScripts("lib/common.js");
-export {};
-//# sourceMappingURL=background.js.map
+importScripts("lib/config.js");
+importScripts("lib/chrome.js");
+importScripts("lib/runtime.js");
+importScripts("lib/common.js");

package/dist/fingerprinting/anti-webgpu/data/content_script/inject.js

@@ -1,50 +1,50 @@
 var background = (function () {
-    let tmp = {};
-    /*  */
-    chrome.runtime.onMessage.addListener(function (request) {
-        for (let id in tmp) {
-            if (tmp[id] && (typeof tmp[id] === "function")) {
-                if (request.path === "background-to-page") {
-                    if (request.method === id) {
-                        tmp[id](request.data);
-                    }
-                }
-            }
+  let tmp = {};
+  /*  */
+  chrome.runtime.onMessage.addListener(function (request) {
+    for (let id in tmp) {
+      if (tmp[id] && (typeof tmp[id] === "function")) {
+        if (request.path === "background-to-page") {
+          if (request.method === id) {
+            tmp[id](request.data);
+          }
         }
-    });
-    /*  */
-    return {
-        "receive": function (id, callback) {
-            tmp[id] = callback;
-        },
-        "send": function (id, data) {
-            chrome.runtime.sendMessage({
-                "method": id,
-                "data": data,
-                "path": "page-to-background"
-            }, function () {
-                return chrome.runtime.lastError;
-            });
-        }
-    };
+      }
+    }
+  });
+  /*  */
+  return {
+    "receive": function (id, callback) {
+      tmp[id] = callback;
+    },
+    "send": function (id, data) {
+      chrome.runtime.sendMessage({
+        "method": id, 
+        "data": data,
+        "path": "page-to-background"
+      }, function () {
+        return chrome.runtime.lastError;
+      });
+    }
+  }
 })();
+
 const ikey = "webgpu-defender-sandboxed-frame";
+
 if (document.documentElement.getAttribute(ikey) === null) {
-    parent.postMessage(ikey, '*');
-    window.top.postMessage(ikey, '*');
-}
-else {
-    document.documentElement.removeAttribute(ikey);
+  parent.postMessage(ikey, '*');
+  window.top.postMessage(ikey, '*');
+} else {
+  document.documentElement.removeAttribute(ikey);
 }
+
 window.addEventListener("message", function (e) {
-    if (e.data && e.data === "webgpu-defender-alert") {
-        e.preventDefault();
-        e.stopPropagation();
-        /*  */
-        background.send("fingerprint", {
-            "host": document.location.host
-        });
-    }
+  if (e.data && e.data === "webgpu-defender-alert") {
+    e.preventDefault();
+    e.stopPropagation();
+    /*  */
+    background.send("fingerprint", {
+      "host": document.location.host
+    });
+  }
 }, false);
-export {};
-//# sourceMappingURL=inject.js.map