ghost-driver 0.1.0__tar.gz

ghost_driver-0.1.0/.github/workflows/publish.yml

@@ -0,0 +1,48 @@
+name: Publish to PyPI
+
+on:
+  push:
+    tags:
+      - "v*"
+
+jobs:
+  build:
+    name: Build distribution
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.x"
+      - name: Install build
+        run: python -m pip install --upgrade build
+      - name: Build sdist and wheel
+        run: python -m build
+      - name: Store distributions
+        uses: actions/upload-artifact@v4
+        with:
+          name: python-package-distributions
+          path: dist/
+
+  publish:
+    name: Publish to PyPI
+    needs: build
+    runs-on: ubuntu-latest
+    # must match the "Environment name" set on the trusted publisher at
+    # https://pypi.org/manage/account/publishing/ -- omit this whole
+    # `environment:` block (and the environment field on PyPI's side) if
+    # you'd rather not bother creating a GitHub Environment for it.
+    environment:
+      name: pypi
+      url: https://pypi.org/project/ghost-driver/
+    permissions:
+      id-token: write  # required for PyPI trusted publishing (OIDC) -- no API token/secret needed at all
+    steps:
+      - name: Download distributions
+        uses: actions/download-artifact@v4
+        with:
+          name: python-package-distributions
+          path: dist/
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1

ghost_driver-0.1.0/.gitignore

@@ -0,0 +1,7 @@
+.venv
+.idea
+__pycache__/
+*.pyc
+*.egg-info/
+build/
+dist/

ghost_driver-0.1.0/PKG-INFO

@@ -0,0 +1,135 @@
+Metadata-Version: 2.5
+Name: ghost-driver
+Version: 0.1.0
+Summary: A nodriver wrapper for automation that needs to survive behavioral antibot checks, not just fingerprint checks: real X11 input instead of CDP-synthesized events, human trajectory replay, and pluggable captcha solvers.
+License: MIT
+Requires-Python: >=3.10
+Requires-Dist: nodriver==0.50.3
+Requires-Dist: requests>=2.31.0
+Provides-Extra: cv
+Requires-Dist: numpy<2,>=1.24; extra == 'cv'
+Requires-Dist: opencv-python-headless<5,>=4.8; extra == 'cv'
+Description-Content-Type: text/markdown
+
+# ghost-driver
+
+A [nodriver](https://github.com/ultrafunkamsterdam/nodriver) wrapper for automation that needs to survive **behavioral** antibot checks, not just fingerprint checks.
+
+## Why
+
+CDP's `Input.dispatchMouseEvent`, however carefully its parameters are set — coordinates, pressure, the `buttons` bitmask — does not trigger the same internal Blink engine state a genuine X11 input event does. This was confirmed with a rotate-captcha (drag a slider to un-rotate a photo) that a real human solved manually at ~90-100%, while every synthetic-input variant we tried — careful CDP dispatch, algorithmic "natural" mouse easing, even over real X11 — stayed stuck at 20-36%.
+
+The tell: the slider button's CSS `:active` pseudo-class engages correctly on a CDP `mousePressed`, then flips `false` on the very first subsequent `mousemove` and never recovers — 98% `false` across CDP-driven drags vs 96% `true` for real manual ones. Real X11/XTest events (the same event class VNC's own mouse injection uses, via `xdotool`) don't have this problem.
+
+Driving the **entire drag** through real X11 input, replaying an **actual recorded human trajectory** (not an algorithmic approximation), closed the gap: 15/15 (100%) on a live confirming run.
+
+ghost-driver packages that finding — plus everything that had to be fixed alongside it (WebGL being entirely absent under Xvfb, `document.hasFocus()` never being true, fractional pixel coordinates no real mouse can produce) — into a reusable library on top of `nodriver`.
+
+## Install
+
+```bash
+pip install ghost-driver
+# for the free CV-based captcha solver (needs opencv/numpy):
+pip install ghost-driver[cv]
+```
+
+nodriver itself needs two small source patches applied after install (see [`ghost_driver/patches.py`](ghost_driver/patches.py) for exactly what and why — both live deep inside methods too large to cleanly monkeypatch, so this edits the installed package on disk, idempotently, with an assertion that fails loudly if nodriver's source has changed underneath it):
+
+```bash
+ghost-driver-patch-nodriver
+```
+
+Run it once after every `pip install`/`pip install --upgrade` of `nodriver` — e.g. right after `pip install` in a Dockerfile.
+
+### System requirements
+
+- `xdotool` on `PATH` — this is the actual mechanism, not optional tooling
+- A real X11 `DISPLAY` with a window manager in front of it. **Xvfb is fine** (virtual, no GPU needed) — bare Xvfb has no window manager though, so also run something minimal like `fluxbox` alongside it
+- Google Chrome (not Debian's `chromium` package — this was developed and tested against `google-chrome-stable`)
+
+Real X11 input has no mapped window to target at all under `--headless=new` — this is an architectural limit, not a bug, and `ghost_driver.input.x11` raises `NoX11DisplayError` rather than silently degrading. See [Headless vs headful](#headless-vs-headful) below.
+
+## Quickstart
+
+```python
+import asyncio
+
+import ghost_driver
+from ghost_driver.input import trajectories, x11
+from ghost_driver.solvers.capguru import CapGuruSolver
+from ghost_driver.solvers import correction_to_percent
+from ghost_driver import page_state
+
+
+async def main():
+    browser = await ghost_driver.launch(headless=False)  # real X11 input needs headful (Xvfb is fine)
+    page = await browser.create_context(url="https://example.com/captcha-page")
+
+    state, _data = await page_state.wait_for_widget_or_content(
+        page, timeout_s=30,
+        has_widget_js_expr="!!document.querySelector('.your-widget-class')",
+    )
+    if state != "widget":
+        return
+
+    await x11.ensure_real_focus(page)  # a real xdotool click -- CDP's page.activate() alone doesn't flip document.hasFocus()
+
+    solver = CapGuruSolver(api_key="your-cap-guru-key")  # or set CAP_GURU_KEY in the environment
+    image_bytes = ...  # capture your widget's already-rendered image (see note below)
+    result = await solver.solve(image_bytes)
+    target_percent = correction_to_percent(result.correction_deg)
+
+    # geometry is yours to read off your own widget's DOM
+    start_x, start_y, max_left = ...
+    needed_delta_x = max_left * (target_percent / 100.0)
+
+    pool = trajectories.load_trajectories()  # defaults to the bundled e-disclosure.ru example recordings
+    events = trajectories.pick_for_target(pool, needed_delta_x)
+    await x11.drag_replay(page, events, start_x, start_y, start_x + needed_delta_x)
+
+
+asyncio.run(main())
+```
+
+Capture the captcha image by reading an already-rendered `<img>` through a `<canvas>` (`drawImage` + `toDataURL`), not a raw `fetch()` of the image URL — an endpoint that serves a fresh random image per request will hand a raw fetch a *different* image than the one actually shown.
+
+`trajectories.load_trajectories()` uses 60 real recorded human trajectories bundled with the package (see [`ghost_driver/examples/e_disclosure_rotate_captcha/`](ghost_driver/examples/e_disclosure_rotate_captcha/)) by default, so drag replay works out of the box without recording anything yourself first. These are calibrated to one specific site's slider though — pass your own `directory` (e.g. wherever `ghost_driver.recorder.record_round` saves to) once you're targeting a different widget.
+
+## Modules
+
+| Module | What it does |
+|---|---|
+| `ghost_driver.browser` | `launch()` — thin wrapper around `nodriver.start()` with WebGL (`swiftshader`) and a launch-retry loop already applied |
+| `ghost_driver.patches` | Idempotent, assertion-guarded source patches for nodriver's CDP connection-wait retry count and proxy-forwarder timeout |
+| `ghost_driver.input.x11` | The core mechanism: real X11 input via `xdotool` — `ensure_real_focus`, `window_geometry`, `drag_replay` |
+| `ghost_driver.input.cdp_replay` | CDP-based drag replay — a documented, **confirmed weaker** fallback for when there's no real X11 display at all |
+| `ghost_driver.input.drag` | `drag_replay()` — picks X11 when available, falls back to CDP automatically |
+| `ghost_driver.input.trajectories` | Loading, filtering, and picking recorded human drag trajectories from JSON recordings |
+| `ghost_driver.recorder` | `record_round()` — captures real human pointer/mouse events during a manual (e.g. VNC) solving session, with correct handling of a challenge that bounces through multiple navigations in one round |
+| `ghost_driver.solvers` | `RotationSolver` protocol + `RotationResult` — pluggable rotate-captcha solvers |
+| `ghost_driver.solvers.capguru` | `CapGuruSolver` — paid, full 0-360° answer in one shot |
+| `ghost_driver.solvers.cv_angle` | `CVAngleSolver` — free, local, purely geometric (needs the `cv` extra); only tells axis-alignment (mod 90°), see `RotationResult.family_period_deg` |
+| `ghost_driver.page_state` | `wait_for_widget_or_content` / `wait_for_stable_content` — reliable "did the challenge appear / did it actually pass" polling with a debounce against mid-navigation false reads |
+| `ghost_driver.fingerprint` | `webrtc` (IP-leak fix), `misc` (window dims, pointer capability, device memory, color depth, languages), `native` (`Function.toString` spoofing helper) |
+| `ghost_driver.cookies` | `to_cookie_param()` — converting a captured `Cookie` into a `CookieParam` for reuse in a fresh context |
+| `ghost_driver.diagnostics` | `dump()` / `save_html()` / `save_json()` — save what actually happened on an attempt, so a confusing result has something to inspect afterward |
+
+## Headless vs headful
+
+`ghost_driver.browser.launch(headless=...)` supports both — it's just `nodriver.start()`'s own parameter. But the reason this library exists, `ghost_driver.input.x11`, **only works headful** (Xvfb counts as headful — it's a real, if virtual, X11 display and window). Under true `--headless=new` there is no mapped window at all for `xdotool` to target; calling any `ghost_driver.input.x11` function with `headless=True` raises `NoX11DisplayError` immediately rather than silently grabbing an unrelated X11 surface (confirmed live: a blind `xdotool getactivewindow` under real headless doesn't fail cleanly, it happily returns geometry for *some* unrelated surface, and a drag built on it "succeeds" with no exception while touching nothing).
+
+`ghost_driver.input.drag_replay()` handles this automatically: it tries X11 first (unless you pass `headless=True`) and falls back to `cdp_replay` on `NoX11DisplayError`. The CDP path is real and documented, but it's the ~20-36%-pass-rate path this whole library exists to get away from — treat it as a fallback, not a primary option.
+
+## Known limitations
+
+- **Cookie reuse needs a static IP.** Reusing a verified session's cookies in a fresh context (`ghost_driver.cookies`) only works if both contexts share the same exit IP — a verification cookie can be bound server-side to the IP it was issued to, not just its own value. A rotating proxy pool between the solve and reuse phases breaks this even with the cookie set correctly.
+- **The free CV solver can't tell orientation.** `CVAngleSolver` only detects axis-alignment (mod 90°) via background texture direction — it can't tell which of the 4 rotationally-symmetric orientations is actually upright. Use `ghost_driver.solvers.candidate_corrections()` and try multiple candidates across retry rounds. `CapGuruSolver` doesn't have this problem (it looks at the actual photographed object) but costs money per solve.
+- **Don't add `Emulation.setHardwareConcurrencyOverride`.** Confirmed via an isolated A/B (10/10 passed with it off, 0/8 with it on) that this actively breaks solving — it's presumed to change the real worker-thread pool Chrome allows, not just the reported value, corrupting whatever computation signs a site's behavioral-telemetry payload. Documented in `ghost_driver.fingerprint.misc`'s module docstring; deliberately not shipped as a working helper.
+
+## Credentials
+
+`CapGuruSolver` takes `api_key` as a constructor argument, or reads the `CAP_GURU_KEY` environment variable if not passed explicitly. There is no built-in fallback key — never hardcode a real API key as a source-level default in code that might be published or shared.
+
+## License
+
+MIT

ghost_driver-0.1.0/README.md

@@ -0,0 +1,122 @@
+# ghost-driver
+
+A [nodriver](https://github.com/ultrafunkamsterdam/nodriver) wrapper for automation that needs to survive **behavioral** antibot checks, not just fingerprint checks.
+
+## Why
+
+CDP's `Input.dispatchMouseEvent`, however carefully its parameters are set — coordinates, pressure, the `buttons` bitmask — does not trigger the same internal Blink engine state a genuine X11 input event does. This was confirmed with a rotate-captcha (drag a slider to un-rotate a photo) that a real human solved manually at ~90-100%, while every synthetic-input variant we tried — careful CDP dispatch, algorithmic "natural" mouse easing, even over real X11 — stayed stuck at 20-36%.
+
+The tell: the slider button's CSS `:active` pseudo-class engages correctly on a CDP `mousePressed`, then flips `false` on the very first subsequent `mousemove` and never recovers — 98% `false` across CDP-driven drags vs 96% `true` for real manual ones. Real X11/XTest events (the same event class VNC's own mouse injection uses, via `xdotool`) don't have this problem.
+
+Driving the **entire drag** through real X11 input, replaying an **actual recorded human trajectory** (not an algorithmic approximation), closed the gap: 15/15 (100%) on a live confirming run.
+
+ghost-driver packages that finding — plus everything that had to be fixed alongside it (WebGL being entirely absent under Xvfb, `document.hasFocus()` never being true, fractional pixel coordinates no real mouse can produce) — into a reusable library on top of `nodriver`.
+
+## Install
+
+```bash
+pip install ghost-driver
+# for the free CV-based captcha solver (needs opencv/numpy):
+pip install ghost-driver[cv]
+```
+
+nodriver itself needs two small source patches applied after install (see [`ghost_driver/patches.py`](ghost_driver/patches.py) for exactly what and why — both live deep inside methods too large to cleanly monkeypatch, so this edits the installed package on disk, idempotently, with an assertion that fails loudly if nodriver's source has changed underneath it):
+
+```bash
+ghost-driver-patch-nodriver
+```
+
+Run it once after every `pip install`/`pip install --upgrade` of `nodriver` — e.g. right after `pip install` in a Dockerfile.
+
+### System requirements
+
+- `xdotool` on `PATH` — this is the actual mechanism, not optional tooling
+- A real X11 `DISPLAY` with a window manager in front of it. **Xvfb is fine** (virtual, no GPU needed) — bare Xvfb has no window manager though, so also run something minimal like `fluxbox` alongside it
+- Google Chrome (not Debian's `chromium` package — this was developed and tested against `google-chrome-stable`)
+
+Real X11 input has no mapped window to target at all under `--headless=new` — this is an architectural limit, not a bug, and `ghost_driver.input.x11` raises `NoX11DisplayError` rather than silently degrading. See [Headless vs headful](#headless-vs-headful) below.
+
+## Quickstart
+
+```python
+import asyncio
+
+import ghost_driver
+from ghost_driver.input import trajectories, x11
+from ghost_driver.solvers.capguru import CapGuruSolver
+from ghost_driver.solvers import correction_to_percent
+from ghost_driver import page_state
+
+
+async def main():
+    browser = await ghost_driver.launch(headless=False)  # real X11 input needs headful (Xvfb is fine)
+    page = await browser.create_context(url="https://example.com/captcha-page")
+
+    state, _data = await page_state.wait_for_widget_or_content(
+        page, timeout_s=30,
+        has_widget_js_expr="!!document.querySelector('.your-widget-class')",
+    )
+    if state != "widget":
+        return
+
+    await x11.ensure_real_focus(page)  # a real xdotool click -- CDP's page.activate() alone doesn't flip document.hasFocus()
+
+    solver = CapGuruSolver(api_key="your-cap-guru-key")  # or set CAP_GURU_KEY in the environment
+    image_bytes = ...  # capture your widget's already-rendered image (see note below)
+    result = await solver.solve(image_bytes)
+    target_percent = correction_to_percent(result.correction_deg)
+
+    # geometry is yours to read off your own widget's DOM
+    start_x, start_y, max_left = ...
+    needed_delta_x = max_left * (target_percent / 100.0)
+
+    pool = trajectories.load_trajectories()  # defaults to the bundled e-disclosure.ru example recordings
+    events = trajectories.pick_for_target(pool, needed_delta_x)
+    await x11.drag_replay(page, events, start_x, start_y, start_x + needed_delta_x)
+
+
+asyncio.run(main())
+```
+
+Capture the captcha image by reading an already-rendered `<img>` through a `<canvas>` (`drawImage` + `toDataURL`), not a raw `fetch()` of the image URL — an endpoint that serves a fresh random image per request will hand a raw fetch a *different* image than the one actually shown.
+
+`trajectories.load_trajectories()` uses 60 real recorded human trajectories bundled with the package (see [`ghost_driver/examples/e_disclosure_rotate_captcha/`](ghost_driver/examples/e_disclosure_rotate_captcha/)) by default, so drag replay works out of the box without recording anything yourself first. These are calibrated to one specific site's slider though — pass your own `directory` (e.g. wherever `ghost_driver.recorder.record_round` saves to) once you're targeting a different widget.
+
+## Modules
+
+| Module | What it does |
+|---|---|
+| `ghost_driver.browser` | `launch()` — thin wrapper around `nodriver.start()` with WebGL (`swiftshader`) and a launch-retry loop already applied |
+| `ghost_driver.patches` | Idempotent, assertion-guarded source patches for nodriver's CDP connection-wait retry count and proxy-forwarder timeout |
+| `ghost_driver.input.x11` | The core mechanism: real X11 input via `xdotool` — `ensure_real_focus`, `window_geometry`, `drag_replay` |
+| `ghost_driver.input.cdp_replay` | CDP-based drag replay — a documented, **confirmed weaker** fallback for when there's no real X11 display at all |
+| `ghost_driver.input.drag` | `drag_replay()` — picks X11 when available, falls back to CDP automatically |
+| `ghost_driver.input.trajectories` | Loading, filtering, and picking recorded human drag trajectories from JSON recordings |
+| `ghost_driver.recorder` | `record_round()` — captures real human pointer/mouse events during a manual (e.g. VNC) solving session, with correct handling of a challenge that bounces through multiple navigations in one round |
+| `ghost_driver.solvers` | `RotationSolver` protocol + `RotationResult` — pluggable rotate-captcha solvers |
+| `ghost_driver.solvers.capguru` | `CapGuruSolver` — paid, full 0-360° answer in one shot |
+| `ghost_driver.solvers.cv_angle` | `CVAngleSolver` — free, local, purely geometric (needs the `cv` extra); only tells axis-alignment (mod 90°), see `RotationResult.family_period_deg` |
+| `ghost_driver.page_state` | `wait_for_widget_or_content` / `wait_for_stable_content` — reliable "did the challenge appear / did it actually pass" polling with a debounce against mid-navigation false reads |
+| `ghost_driver.fingerprint` | `webrtc` (IP-leak fix), `misc` (window dims, pointer capability, device memory, color depth, languages), `native` (`Function.toString` spoofing helper) |
+| `ghost_driver.cookies` | `to_cookie_param()` — converting a captured `Cookie` into a `CookieParam` for reuse in a fresh context |
+| `ghost_driver.diagnostics` | `dump()` / `save_html()` / `save_json()` — save what actually happened on an attempt, so a confusing result has something to inspect afterward |
+
+## Headless vs headful
+
+`ghost_driver.browser.launch(headless=...)` supports both — it's just `nodriver.start()`'s own parameter. But the reason this library exists, `ghost_driver.input.x11`, **only works headful** (Xvfb counts as headful — it's a real, if virtual, X11 display and window). Under true `--headless=new` there is no mapped window at all for `xdotool` to target; calling any `ghost_driver.input.x11` function with `headless=True` raises `NoX11DisplayError` immediately rather than silently grabbing an unrelated X11 surface (confirmed live: a blind `xdotool getactivewindow` under real headless doesn't fail cleanly, it happily returns geometry for *some* unrelated surface, and a drag built on it "succeeds" with no exception while touching nothing).
+
+`ghost_driver.input.drag_replay()` handles this automatically: it tries X11 first (unless you pass `headless=True`) and falls back to `cdp_replay` on `NoX11DisplayError`. The CDP path is real and documented, but it's the ~20-36%-pass-rate path this whole library exists to get away from — treat it as a fallback, not a primary option.
+
+## Known limitations
+
+- **Cookie reuse needs a static IP.** Reusing a verified session's cookies in a fresh context (`ghost_driver.cookies`) only works if both contexts share the same exit IP — a verification cookie can be bound server-side to the IP it was issued to, not just its own value. A rotating proxy pool between the solve and reuse phases breaks this even with the cookie set correctly.
+- **The free CV solver can't tell orientation.** `CVAngleSolver` only detects axis-alignment (mod 90°) via background texture direction — it can't tell which of the 4 rotationally-symmetric orientations is actually upright. Use `ghost_driver.solvers.candidate_corrections()` and try multiple candidates across retry rounds. `CapGuruSolver` doesn't have this problem (it looks at the actual photographed object) but costs money per solve.
+- **Don't add `Emulation.setHardwareConcurrencyOverride`.** Confirmed via an isolated A/B (10/10 passed with it off, 0/8 with it on) that this actively breaks solving — it's presumed to change the real worker-thread pool Chrome allows, not just the reported value, corrupting whatever computation signs a site's behavioral-telemetry payload. Documented in `ghost_driver.fingerprint.misc`'s module docstring; deliberately not shipped as a working helper.
+
+## Credentials
+
+`CapGuruSolver` takes `api_key` as a constructor argument, or reads the `CAP_GURU_KEY` environment variable if not passed explicitly. There is no built-in fallback key — never hardcode a real API key as a source-level default in code that might be published or shared.
+
+## License
+
+MIT

ghost_driver-0.1.0/ghost_driver/__init__.py

@@ -0,0 +1,5 @@
+from .browser import launch
+from .logging_utils import enable_logging
+
+__all__ = ["launch", "enable_logging"]
+__version__ = "0.1.0"

ghost_driver-0.1.0/ghost_driver/browser.py

@@ -0,0 +1,49 @@
+import asyncio
+from typing import Sequence
+
+import nodriver as uc
+
+DEFAULT_BROWSER_ARGS = [
+    "--use-gl=swiftshader",
+    "--use-angle=swiftshader",
+]
+
+
+async def launch(
+    *,
+    headless: bool = False,
+    sandbox: bool = False,
+    browser_args: Sequence[str] | None = None,
+    max_attempts: int = 5,
+    retry_delay_s: float = 2.0,
+    **kwargs,
+) -> uc.Browser:
+    """Thin wrapper around nodriver.start() with this library's defaults
+    applied, plus a retry loop around the browser PROCESS launch itself
+    (Chrome under load -- containers, cold cache -- can fail to start
+    transiently; this is separate from nodriver's own CDP-handshake
+    connection-wait retry, which ghost_driver.patches extends).
+
+    --use-gl=swiftshader --use-angle=swiftshader default ON: under Xvfb,
+    WebGL was found completely ABSENT without these (getContext("webgl")
+    returned null outright -- not just a suspicious vendor/renderer
+    string, something no real consumer browser ever does).
+
+    Defaults to headless=False: ghost_driver.input.x11's real-X11-input
+    drag mechanism (this library's whole reason to exist) requires a real
+    mapped window, which --headless=new never creates. Pass a real
+    (virtual is fine, e.g. Xvfb) DISPLAY and a window manager alongside
+    it if you're running this outside an existing desktop session.
+    """
+    args = list(DEFAULT_BROWSER_ARGS)
+    if browser_args:
+        args.extend(browser_args)
+    last_error: Exception | None = None
+    for attempt in range(max_attempts):
+        try:
+            return await uc.start(headless=headless, sandbox=sandbox, browser_args=args, **kwargs)
+        except Exception as e:
+            last_error = e
+            if attempt + 1 < max_attempts:
+                await asyncio.sleep(retry_delay_s)
+    raise RuntimeError(f"could not launch browser after {max_attempts} attempts") from last_error

ghost_driver-0.1.0/ghost_driver/cookies.py

@@ -0,0 +1,40 @@
+import time
+from datetime import datetime, timezone
+
+from nodriver import cdp
+
+
+def describe(cookies: list) -> list[str]:
+    """One human-readable line per cookie -- name, a truncated value,
+    domain, and remaining lifetime -- for a quick print() after a solve.
+    Takes the same list `Network.get_cookies()` (or NavigateResult.cookies)
+    returns."""
+    now = time.time()
+    lines = []
+    for c in cookies:
+        if c.session or c.expires is None or c.expires < 0:
+            lifetime = "SESSION (dies when browser closes)"
+        else:
+            remaining_h = (c.expires - now) / 3600.0
+            expires_iso = datetime.fromtimestamp(c.expires, tz=timezone.utc).isoformat()
+            lifetime = f"expires {expires_iso} (~{remaining_h:.1f}h from now)"
+        lines.append(f"{c.name}={c.value[:24]}... domain={c.domain} -- {lifetime}")
+    return lines
+
+
+def to_cookie_param(cookie: "cdp.network.Cookie") -> "cdp.network.CookieParam":
+    """Converts a cdp.network.Cookie (from Network.get_cookies()) into a
+    cdp.network.CookieParam (for Network.set_cookies()) -- these are
+    DIFFERENT types despite the overlapping fields. In particular
+    Cookie.expires is a plain float; CookieParam.expires needs it wrapped
+    in TimeSinceEpoch, which has the to_json() the CDP serializer calls --
+    a bare float doesn't, and fails with an unhelpful AttributeError deep
+    in serialization if passed straight through.
+    """
+    expires = cdp.network.TimeSinceEpoch(cookie.expires) if cookie.expires is not None else None
+    return cdp.network.CookieParam(
+        name=cookie.name, value=cookie.value, domain=cookie.domain, path=cookie.path,
+        secure=cookie.secure, http_only=cookie.http_only, same_site=cookie.same_site,
+        expires=expires, priority=cookie.priority, source_scheme=cookie.source_scheme,
+        source_port=cookie.source_port,
+    )

ghost_driver-0.1.0/ghost_driver/diagnostics.py

@@ -0,0 +1,50 @@
+import json
+from datetime import datetime, timezone
+from pathlib import Path
+from urllib.parse import parse_qs, urlparse
+
+
+def save_html(findings_dir: Path | str, label: str, html: str | None) -> str:
+    findings_dir = Path(findings_dir)
+    findings_dir.mkdir(parents=True, exist_ok=True)
+    ts = datetime.now(timezone.utc).strftime("%Y-%m-%d_%H%M%S")
+    path = findings_dir / f"{ts}_{label}.html"
+    path.write_text(html or "", encoding="utf-8")
+    return str(path)
+
+
+def save_json(findings_dir: Path | str, label: str, payload: dict) -> str:
+    findings_dir = Path(findings_dir)
+    findings_dir.mkdir(parents=True, exist_ok=True)
+    ts = datetime.now(timezone.utc).strftime("%Y-%m-%d_%H%M%S")
+    path = findings_dir / f"{ts}_{label}.json"
+    path.write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8")
+    return str(path)
+
+
+def short_url(url: str) -> str:
+    """Path + query KEYS only, values dropped -- some sites' flows carry
+    large opaque behavioral-telemetry blobs in query values (thousands of
+    chars each); printing full URLs to a live console makes every run
+    unreadable. Keep the full URL in saved JSON diagnostics (save_json)
+    for later analysis; use this only for human-readable log lines."""
+    parsed = urlparse(url)
+    keys = ",".join(parse_qs(parsed.query).keys())
+    return f"{parsed.scheme}://{parsed.netloc}{parsed.path}?{keys}" if keys else url
+
+
+async def dump(page, findings_dir: Path | str, label: str, extra: dict | None = None) -> str:
+    """Saves the live page's HTML plus whatever extra diagnostic payload
+    you pass, both timestamped consistently under one label so they're
+    easy to correlate on disk. Call this at every meaningful exit point of
+    an attempt, not just failures -- a "passed" that later turns out wrong
+    has nothing to inspect after the fact otherwise."""
+    try:
+        html = await page.evaluate("document.documentElement.outerHTML")
+    except Exception:
+        html = None
+    saved_html = save_html(findings_dir, f"{label}_html", html)
+    payload = {"label": label, "saved_html": saved_html}
+    if extra:
+        payload.update(extra)
+    return save_json(findings_dir, label, payload)