jumpy-lion 0.1.6-beta.3 → 0.1.6-beta.30

package/README.md

@@ -22,6 +22,14 @@
   - [Best Practices](#best-practices)
   - [Performance Considerations](#performance-considerations)
 - [Launch Options for Network and Persistence](#launch-options-for-network-and-persistence)
+- [Session Bundle (save & restore browser state)](#session-bundle-save--restore-browser-state)
+  - [When to use it](#when-to-use-it)
+  - [Producer: capturing a bundle](#producer-capturing-a-bundle)
+  - [Consumer: rehydrating a bundle](#consumer-rehydrating-a-bundle)
+  - [What's inside a bundle](#whats-inside-a-bundle)
+  - [`saveSession()` options](#savesession-options)
+  - [`restoreSession()`](#restoresession)
+  - [Caveats](#caveats)
 - [Crawler Class Documentation](#crawler-class-documentation)
   - [Constructor](#constructor)
 - [CdpPage Class Documentation](#cdppage-class-documentation)
@@ -30,6 +38,8 @@
   - [Public Methods](#public-methods)
 - [Utility Functions](#utility-functions)
   - [createCDPRouter](#createcdprouter)
+  - [saveSession](#savesession)
+  - [restoreSession](#restoresession-1)
 
 ## Overview
 
@@ -299,6 +309,152 @@ const crawler = new Crawler({
 
 ---
 
+## Session Bundle (save & restore browser state)
+
+The session bundle lets one crawler **capture** the full live browser state — cookies, per-origin localStorage and sessionStorage, the Chrome user-data-dir (which transitively carries IndexedDB, Service Workers, and Cache Storage), the C++ `fingerprintConfig`, the captured `fingerprint`, and the resolved locale triple — into a single JSON-serializable blob. A second crawler can then **rehydrate** from that blob and come up byte-for-byte identical, so a session that was authenticated upstream stays authenticated downstream (matching what `naver-session-test` does, generalized into the framework).
+
+This is mechanism, **not** persistence: nothing is written to disk for you. You stash the bundle wherever you like (Apify KV store, S3, a local file) and pass it back when you launch the next crawler.
+
+### When to use it
+
+- An Apify actor logs into a site, then hands the session to a second actor that does the actual scraping.
+- A pool of long-running actors needs to checkpoint browser state between restarts.
+- You need two crawls to look like the *exact same browser* to a bot detector (Cloudflare, DataDome, Naver) — same UA, same UA-CH, same WebGL renderer, same screen, same canvas/audio noise seed, same cookies, same localStorage, same Service Worker state.
+
+### Producer: capturing a bundle
+
+```typescript
+import CDPCrawler, { saveSession, type SessionBundle } from 'cdp-crawler';
+import { Actor } from 'apify';
+
+let bundle: SessionBundle | undefined;
+
+const producer = new CDPCrawler({
+    launchContext: {
+        launchOptions: {
+            useNativeStealth: true,
+            fingerprintOptions: { platform: 'MacIntel' },
+        },
+    },
+    requestHandler: async ({ page }) => {
+        await page.goto('https://target.example.com/login');
+        // … perform login, solve captcha, etc. …
+
+        bundle = await saveSession(page);
+    },
+});
+
+await producer.run(['https://target.example.com/login']);
+await Actor.setValue('session', bundle); // ship to KV store for the next actor
+```
+
+### Consumer: rehydrating a bundle
+
+Pass the bundle as `launchOptions.sessionBundle` on the new crawler. The plugin extracts the user-data-dir into a temp directory, feeds the captured `fingerprintConfig` straight to the C++ patches (regeneration is skipped), pins `useNonApifyFingerprints: false`, and replays cookies + per-origin storage on every new page **before** any navigation runs.
+
+```typescript
+import CDPCrawler, { type SessionBundle } from 'cdp-crawler';
+import { Actor } from 'apify';
+
+const bundle = await Actor.getValue<SessionBundle>('session');
+if (!bundle) throw new Error('No session bundle available');
+
+const consumer = new CDPCrawler({
+    launchContext: {
+        launchOptions: {
+            useNativeStealth: true,
+            sessionBundle: bundle, // ← the only new option
+        },
+    },
+    requestHandler: async ({ page }) => {
+        // The first page already has the producer's cookies, localStorage,
+        // sessionStorage, IndexedDB, Service Worker registrations, etc.
+        await page.goto('https://target.example.com/account'); // already logged in
+    },
+});
+
+await consumer.run(['https://target.example.com/account']);
+```
+
+### What's inside a bundle
+
+```typescript
+interface SessionBundle {
+    schemaVersion: 1;
+    createdAt: string;
+    createdBy?: { package: 'cdp-crawler'; version: string };
+
+    cookies: SerializedCookie[];                     // Network.Cookie-shaped
+    localStorage: Record<origin, Record<key, value>>;
+    sessionStorage: Record<origin, Record<key, value>>;
+
+    userDataDir: {
+        encoding: 'base64+gzip+tar';
+        bytes: string;                               // the whole Chrome profile
+        sizeBytes: number;
+        capturedFiles: number;
+    } | null;
+
+    fingerprintConfig: FingerprintConfigJson;        // C++ overrides JSON, byte-for-byte
+    fingerprint: BrowserFingerprintWithHeaders;      // Crawlee-shaped fp object
+    fingerprintInput: {                              // The fingerprintOptions inputs
+        locale?: string; languages?: string; timezone?: string;
+        platform?: string; seedKey?: string;
+        useNonApifyFingerprints: false;              // pinned on restore
+        [key: string]: unknown;
+    };
+    resolvedLocale: { locale: string; languages: string; timezone: string };
+
+    browserProfile: {                                // diagnostic snapshot
+        userAgent: string; platform: string; language: string;
+        screenWidth: number; screenHeight: number; devicePixelRatio: number;
+        webglRenderer: string; webglVendor: string;
+    };
+
+    proxyMeta?: { proxyUrl?: string; sessionId?: string; countryCode?: string };
+}
+```
+
+Bundles are versioned via `schemaVersion`. Loading a bundle with an unrecognized version throws an explicit error rather than misbehaving silently. A helper `assertValidBundle(value)` is exported for callers that want to validate before passing the bundle around.
+
+### `saveSession()` options
+
+```typescript
+await saveSession(page, {
+    includeUserDataDir: true,    // default true; set false for a JSON-only bundle (~50 KB)
+    flushCookies: true,          // default true → calls Storage.flushBrowserCookies first
+    userDataDirPath: undefined,  // override; defaults to the path used at launch
+    cookieUrls: undefined,       // forwarded to Network.getCookies as a fallback
+    proxyMeta: {                 // optional; stamped for inspection, NOT replayed
+        proxyUrl, sessionId, countryCode,
+    },
+});
+```
+
+The proxy is intentionally not rebuilt on restore — pass your own `proxyUrl` to the consumer crawler so cookies stay tied to the same exit IP.
+
+### `restoreSession()`
+
+`restoreSession(page, bundle)` is the manual escape hatch for advanced users who open additional pages (or targets) inside a single crawl and want them to share the bundle's cookies + storage. Most users do not need it — the `launchOptions.sessionBundle` option is the canonical path.
+
+```typescript
+import { restoreSession } from 'cdp-crawler';
+
+await restoreSession(page, bundle); // sets cookies + registers per-origin storage replay
+```
+
+Note: `restoreSession` cannot swap the user-data-dir or fingerprintConfig on a running browser — those are launch-time inputs and must travel through `launchOptions.sessionBundle`.
+
+### Caveats
+
+- **User-data-dir size**: a real Chrome profile can be several MB. Inlined as base64-gzipped-tar inside the JSON, this can push the bundle past the Apify KV-store 9 MB record limit. Use `includeUserDataDir: false` if you only need cookies + storage and can live without IndexedDB / Service Workers / Cache Storage.
+- **Locks**: Singleton* sentinels, GPU/Shader/Code caches, and Crashpad metrics are deliberately stripped from the tar — they are process-bound or freely regenerable and would otherwise break restore on a new machine.
+- **Capture timing**: the data-dir tar is taken while Chrome is still running. To avoid half-written LevelDB files, prefer to call `saveSession` after the page has gone idle (`waitForLoadState`-style settling, or right before `crawler.teardown()`). `saveSession` automatically calls `Storage.flushBrowserCookies` first.
+- **Cross-version replay**: bundles are tagged with `createdBy.version`; loading a bundle produced by a meaningfully different `cdp-crawler` version may warn or fail depending on what changed in `fingerprintConfig`'s shape. The plan is to migrate forward, not to support arbitrarily old bundles.
+- **Multi-origin localStorage**: `saveSession` captures the current page's origin only. If you need storage from multiple origins, navigate to each one before calling `saveSession`, or call `saveSession` per page and merge the bundles client-side.
+
+---
+
 ## `Crawler` Class Documentation
 
 ### Constructor
@@ -605,3 +761,35 @@ Creates a custom router for handling crawling routes using CDP.
   - `Router<Context>`: A configured router instance.
 
 ---
+
+### `saveSession`
+
+#### `export async function saveSession(page: CdpPage, options?: SaveSessionOptions): Promise<SessionBundle>`
+
+Captures the full browser state of a running page — cookies, per-origin web storage, fingerprintConfig, fingerprint object, resolved locale, and the Chrome user-data-dir (inlined as base64-gzipped-tar) — into a single JSON-serializable bundle.
+
+- **Parameters**:
+  - `page` (CdpPage): a page produced by a `CDPCrawler` instance.
+  - `options` (SaveSessionOptions): optional knobs:
+    - `includeUserDataDir` (boolean, default `true`): pack and inline the Chrome user-data-dir.
+    - `flushCookies` (boolean, default `true`): call `Storage.flushBrowserCookies` before snapshotting cookies.
+    - `userDataDirPath` (string): override the user-data-dir path; defaults to the one used at launch.
+    - `cookieUrls` (string[]): forwarded to `Network.getCookies` when `Storage.getCookies` is unavailable.
+    - `proxyMeta` ({ proxyUrl?, sessionId?, countryCode? }): stamped on the bundle for inspection only.
+
+- **Returns**:
+  - `Promise<SessionBundle>`: a fully populated, JSON-serializable bundle.
+
+See [Session Bundle](#session-bundle-save--restore-browser-state) for usage patterns.
+
+### `restoreSession`
+
+#### `export async function restoreSession(page: CdpPage, bundle: SessionBundle): Promise<void>`
+
+Manually applies a bundle's cookies and per-origin web storage to an arbitrary page. Use this only for advanced flows (e.g. opening extra targets mid-crawl). The canonical path is `launchOptions.sessionBundle`, which also restores the user-data-dir and fingerprintConfig — `restoreSession` cannot swap those on a running browser.
+
+- **Parameters**:
+  - `page` (CdpPage): the target page.
+  - `bundle` (SessionBundle): a bundle produced by `saveSession`.
+
+---