webskrap 3.2.0__tar.gz → 3.2.2__tar.gz

{webskrap-3.2.0 → webskrap-3.2.2}/.gitignore

@@ -15,3 +15,4 @@ test.py
 AGENTS.md
 CLAUDE.md
 .cache
+webskrap-output/

webskrap-3.2.2/.package-dist.dZyl3m/.gitignore

@@ -0,0 +1 @@
+*

{webskrap-3.2.0 → webskrap-3.2.2}/CHANGELOG.md

@@ -9,6 +9,38 @@ history, so they summarize each release rather than list every change.
 
 ## [Unreleased]
 
+## [3.2.2] - 2026-10-30
+
+### Fixed
+
+- DataDome detection now follows the device check when it navigates its
+  frame. It reads the live challenge frame, and it reports a manual
+  challenge apart from a hard block. The user-agent probe now uses the
+  session proxy and network flags, so it cannot connect directly.
+- Full desktop profiles now set `navigator.platform` to match the user
+  agent. They also fit the viewport inside the real browser window before
+  the metrics override.
+- Cookie decline now finds the reject button of Commanders Act
+  TrustCommander notices. Their markup has no "cookie" or "consent" name, so
+  the text search did not look inside them.
+- Full profiles no longer put q-values in `navigator.languages`, and no longer
+  send a doubled q-value in `Accept-Language` (`fr;q=0.9;q=0.9`). Chromium
+  weights the header itself, so the override now gets a plain list.
+- With `profile_mode="full"` (or `"locale"` off Linux),
+  `navigator.serviceWorker.register()` no longer hangs. The page and the
+  browser both held the worker paused, and each waited for the other. Now
+  only the browser-level setup prepares service and shared workers; the page
+  only resumes them.
+- Mouse presses now report `PointerEvent.pressure` 0.5 while the button is
+  down, as a real mouse does. CDP's default of 0 was visible to pages.
+
+## [3.2.1] - 2026-10-26
+
+### Removed
+
+- `scripts/driver_parity.py` no longer offers `--click`. The script only
+  measures whether a target challenges each driver.
+
 ## [3.2.0] - 2026-10-26
 
 ### Added
@@ -929,7 +961,9 @@ entries below.
 
 Earlier releases (0.1.0 - 0.4.9) are recorded in the git history and tags only.
 
-[Unreleased]: https://github.com/kacigaya/webskrap/compare/v3.2.0...HEAD
+[Unreleased]: https://github.com/kacigaya/webskrap/compare/v3.2.2...HEAD
+[3.2.2]: https://github.com/kacigaya/webskrap/compare/v3.2.1...v3.2.2
+[3.2.1]: https://github.com/kacigaya/webskrap/compare/v3.2.0...v3.2.1
 [3.2.0]: https://github.com/kacigaya/webskrap/compare/v3.1.0...v3.2.0
 [3.1.0]: https://github.com/kacigaya/webskrap/compare/v3.0.1...v3.1.0
 [3.0.1]: https://github.com/kacigaya/webskrap/compare/v3.0.0...v3.0.1

{webskrap-3.2.0 → webskrap-3.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: webskrap
-Version: 3.2.0
+Version: 3.2.2
 Summary: Async Python scraping on Chromium over its own CDP client, with coherent browser profiles and session controls.
 Project-URL: Homepage, https://github.com/kacigaya/webskrap
 Project-URL: Repository, https://github.com/kacigaya/webskrap
@@ -82,8 +82,21 @@ asyncio.run(main())
 For deferred content, pass `ready_selector="#content"` to a Python fetch or
 `--ready-selector '#content'` to CLI `fetch`. Both MCP fetch tools accept
 `ready_selector` too. The selector must become visible after consent dismissal.
-Known challenge pages (DataDome, Cloudflare, Akamai, AWS WAF) and provider block pages (Cloudflare, Akamai) raise `blocked`;
-WebSkrap does not retry or solve them. Ordinary HTTP errors still return `ok=False`.
+Known challenge pages (DataDome, Cloudflare, Akamai, AWS WAF) and provider block
+pages (DataDome, Cloudflare, Akamai) raise `blocked`. WebSkrap waits for device
+checks that reload the page. It does not retry refusals or solve CAPTCHAs.
+Ordinary HTTP errors still return `ok=False`.
+
+For manual CAPTCHA completion, open a persistent browser with
+`webskrap browser open --headed --session manual --proxy http://localhost:8888 URL`.
+Complete the check in that browser. Keep the same session, profile, and proxy
+for later commands. On a remote host, use a desktop connection to its display.
+`virtual_display=True` alone does not provide a remote desktop.
+
+Keep `profile_mode="native"` for the browser's own identity. On Linux with Xvfb,
+`virtual_display=True` runs a headed browser without a visible desktop window.
+Full profile emulation is useful for device tests, but cannot replace every
+worker or iframe value. See the [identity limits](src/webskrap/guide.md).
 
 ## Search
 

{webskrap-3.2.0 → webskrap-3.2.2}/README.md

@@ -48,8 +48,21 @@ asyncio.run(main())
 For deferred content, pass `ready_selector="#content"` to a Python fetch or
 `--ready-selector '#content'` to CLI `fetch`. Both MCP fetch tools accept
 `ready_selector` too. The selector must become visible after consent dismissal.
-Known challenge pages (DataDome, Cloudflare, Akamai, AWS WAF) and provider block pages (Cloudflare, Akamai) raise `blocked`;
-WebSkrap does not retry or solve them. Ordinary HTTP errors still return `ok=False`.
+Known challenge pages (DataDome, Cloudflare, Akamai, AWS WAF) and provider block
+pages (DataDome, Cloudflare, Akamai) raise `blocked`. WebSkrap waits for device
+checks that reload the page. It does not retry refusals or solve CAPTCHAs.
+Ordinary HTTP errors still return `ok=False`.
+
+For manual CAPTCHA completion, open a persistent browser with
+`webskrap browser open --headed --session manual --proxy http://localhost:8888 URL`.
+Complete the check in that browser. Keep the same session, profile, and proxy
+for later commands. On a remote host, use a desktop connection to its display.
+`virtual_display=True` alone does not provide a remote desktop.
+
+Keep `profile_mode="native"` for the browser's own identity. On Linux with Xvfb,
+`virtual_display=True` runs a headed browser without a visible desktop window.
+Full profile emulation is useful for device tests, but cannot replace every
+worker or iframe value. See the [identity limits](src/webskrap/guide.md).
 
 ## Search
 

webskrap-3.2.2/docs/diagnostics/datadome.md

@@ -0,0 +1,212 @@
+# DataDome website navigation diagnosis
+
+Checked on 2026-10-29 against a DataDome-protected website.
+
+## Root cause
+
+The request reaches the website through the existing Gluetun VPN. DataDome then
+chooses whether to show a device check, a CAPTCHA, or a hard block.
+The headless browser received a hard-block document. That document has no
+CAPTCHA widget. A headed browser on the same VPN exit received a working
+manual slider. This is an external verification requirement.
+
+WebSkrap also had three integration defects:
+
+1. Detection recognized `ct.captcha-delivery.com/c.js`, but missed the
+   `i.js` device-check script before its iframe appeared.
+2. It treated the device check as a manual challenge and did not wait for
+   its own script to reload the page.
+3. It read the parent document's iframe `src`. A cross-origin iframe can
+   navigate itself while that attribute stays unchanged. A hard block then
+   received the same hint as a solvable CAPTCHA.
+
+The optional headless user-agent probe also started a separate browser
+without the session's proxy or network flags.
+
+## Evidence
+
+- Gluetun reported a healthy tunnel and a French exit. A browser request to
+  the IP check service returned the same exit as a request from the container.
+- The proxy required authentication. Browser credentials stayed in memory.
+- The container resolved the website and the CAPTCHA script, frame, and asset hosts.
+  Its BusyBox `nslookup` printed a parse error for one CNAME response. The
+  proxy still resolved and loaded that host successfully.
+- The website returned HTTP 403 with `x-datadome: protected` and `x-dd-b: 3`.
+- The device-check script and iframe returned HTTP 200. In the headed run,
+  CAPTCHA JavaScript, CSS, fonts, and logo also returned HTTP 200.
+- Browser logs showed GPU and canvas warnings, but no failed CAPTCHA resource
+  request in the captured headed flow.
+- The headless provider frame contained
+  `data-dd-response-page="hard-block"`. Its parent had no visible body text.
+- The headed provider frame showed the manual verification slider.
+- Cookies included `datadome`, `__cf_bm`, and `__cflb`. Cookie values and
+  challenge URLs are excluded from this report.
+
+## Changes
+
+Detection now reads the active provider frame through the existing CDP frame
+code. It uses provider origin and markup, not page titles or translated text.
+Device checks use the existing bounded interstitial wait. Frame inspection
+uses the remaining navigation budget. Manual challenges stay open in persistent
+sessions. Hard blocks receive the block hint.
+
+The user-agent probe now keeps the session's launch flags, including its
+proxy and WebRTC policy. No CAPTCHA solver, token injection, provider script
+patch, new dependency, or site-specific branch was added.
+
+## Manual completion
+
+1. Start a headed persistent browser on a display you can access.
+2. Configure the VPN proxy before navigation. Keep all page resources enabled.
+3. Complete the CAPTCHA yourself. Use its retry control if it has expired.
+4. Continue with the same browser, profile, and proxy. Do not copy challenge
+   tokens into another browser.
+
+A temporary headed browser session and loopback VNC listener on port 15900 were used
+to prepare manual completion. Both were closed when manual completion was
+declined. Xvfb alone does not give remote access.
+
+## Verification limits
+
+The final Linux ARM64 run passed 858 tests. It excluded 30 opt-in live tests
+for unrelated public sites. Coverage was 88.85%, above the 85% project gate.
+The focused suite passed 123 tests before the combined run.
+
+Ruff lint and format checks, Pyright for Linux, Windows, and macOS, Bandit,
+package builds, and Twine metadata checks passed. OSV found no vulnerabilities
+in the 51 public packages from the lockfile. Pip-audit also found none.
+Windows and macOS runtime tests require those operating systems and were not
+run on this host.
+
+Local regression fixtures cover device-check completion, a transition to a
+manual challenge, a transition to a hard block, cross-origin frame navigation,
+proxy authentication, image loading, and continuation with the same cookies.
+The fixture's manual button is a local test control. It does not contact or
+complete real CAPTCHA verification.
+
+Live tests reached the rendered CAPTCHA. Navigation beyond verification
+cannot be confirmed until a person completes it. The provider can still
+refuse the session after manual verification. No forms were submitted.
+
+## VPN exit retest
+
+Eight distinct French NordVPN exits were tested, including the initial
+baseline. Each headed run returned HTTP 403 and a DataDome manual challenge.
+Fresh browser profiles were used for the replacement exits. No CAPTCHA was
+completed or bypassed. This sample did not identify a CAPTCHA-free exit.
+It does not establish that IP reputation is the only input to the decision.
+
+A control run placed the browser directly inside Gluetun's network namespace.
+It used the same VPN exit without the HTTP proxy and still received a challenge.
+The proxy is therefore not the sole cause.
+
+Rotation also exposed an operational defect: Gluetun's NordVPN server catalog
+was dated March 2024. Some listed endpoints timed out or rejected authentication.
+The catalog was refreshed through Gluetun's existing updater. The original
+catalog was backed up, and the VPN was restored to a healthy state.
+
+The final verified exit was `187.13.198.190` in Marseille. It still required
+verification. The refreshed catalog is in the container's writable layer;
+container recreation can replace it because there is no catalog volume mount.
+
+## CDP identity investigation
+
+An ordinary headed Chromium process was launched on the same VPN exit with
+no debugging port, no debugging pipe, and no WebSkrap page setup. Its native
+network log recorded an HTTP 403, a device check, and then a CAPTCHA.
+This establishes that CDP is not required to trigger the current challenge.
+It does not reveal DataDome's private scoring rule.
+
+A separate public fingerprint probe reported `isBot: false` for default
+headed WebSkrap. It reported no CDP, webdriver, worker-value, or client-hint
+inconsistency. The headed Mesa/locale configuration also passed that probe.
+Both configurations still received a CAPTCHA. The headless run was
+flagged for its bot user agent and weak signals, with no CDP leak reported.
+These are observations from one public probe, not universal detection claims.
+
+Local page-owned probes compared request headers and JavaScript values in
+documents, cross-origin iframes, dedicated workers, and shared workers.
+Two full-profile setup defects were reproduced and fixed:
+
+1. A Windows or macOS user agent changed client hints but left the document's
+   `navigator.platform` at its native Linux value. The CDP override now sets
+   an unambiguous legacy platform as well. Unknown CPU values are not guessed.
+2. A desktop viewport could be wider than its real browser window. The new
+   regression reproduced `innerWidth=1365` with `outerWidth=780`. Setup now
+   resizes the real window before applying device metrics. A missing command
+   on an older browser keeps the previous behavior; other errors propagate.
+
+The changes use Chromium's own
+[user-agent override](https://chromedevtools.github.io/devtools-protocol/tot/Emulation/#method-setUserAgentOverride)
+and [content-size command](https://chromedevtools.github.io/devtools-protocol/tot/Browser/#method-setContentsSize).
+No JavaScript getter replacements were added.
+
+The probe also confirmed existing limits. UA masking removes high-entropy
+client hints. Cross-process frames keep their real screen and pixel ratio.
+Workers keep their native legacy platform, and shared workers keep their
+native user agent. Full cross-platform emulation therefore cannot promise a
+coherent identity across all contexts. Native mode leaves those values intact.
+Neither the fixes nor changing the GPU and locale removed the website's verification
+requirement in the measured runs.
+
+The final follow-up suite ran inside Gluetun's network namespace. It passed
+872 tests with 88.89% coverage and excluded 30 opt-in live tests. The focused
+suite passed 125 tests. Lint, format, Linux/Windows/macOS type checks, Bandit,
+OSV, pip-audit, package builds, and Twine checks passed. Windows and macOS
+runtime checks were not available on this Linux host.
+
+## Protected form retest
+
+A later retest compared ordinary pages with the website's quote form path.
+Each run used a fresh browser profile and a VPN exit that the run had not
+used before.
+
+The browser passed DataDome on ordinary pages. On some exits the first page
+returned HTTP 403 with a device check (`x-dd-b: 3`). The device check then
+reloaded into the requested page with HTTP 200. On other exits the first page
+returned HTTP 200 at once. DataDome's page tag ran on those pages.
+
+The quote form path returned HTTP 403 with a CAPTCHA on every exit. This
+occurred with a direct load and with a click-through from the landing page.
+These changes did not remove the CAPTCHA:
+
+1. Google Chrome stable for Linux ARM64 instead of Chrome for Testing. This
+   adds the `Google Chrome` brand.
+2. A locale and a timezone that match the exit country.
+3. Mesa WebGL.
+4. `x86` client hints instead of the host's `arm` value.
+5. A 25-second pause with mouse movement and wheel scrolling on each page.
+6. A four-page session with about 75 seconds of activity before the form.
+7. Exits in eleven countries on eight hosting ASNs.
+
+A desktop-like host environment did not change the result. It added Noto
+and Ubuntu fonts and PulseAudio audio devices. On a fresh exit, a direct load
+of the form received a device check (`x-dd-b: 3`). The check reloaded into a
+CAPTCHA. The same check reloaded into HTTP 200 on ordinary pages.
+
+SwiftShader WebGL failed the device check and received a hard block.
+The VPN provider's dedicated-IP servers refused this setup's credentials,
+so a dedicated exit was not tested.
+
+A Windows user agent made the result worse: the first page returned a hard
+block. Repeated fresh profiles on one exit also made the result worse. A later
+visit received the CAPTCHA, or a hard block, without a device check.
+
+The VPN provider's catalog contains only hosting ASNs. The measured limit is
+therefore the exit IP class for this form path, not a browser defect that
+WebSkrap can fix. A residential or mobile IP was not tested.
+
+The retest found two WebSkrap defects:
+
+1. Cookie decline did not search Commanders Act TrustCommander notices.
+2. Full profiles put q-values in `navigator.languages` and doubled them in
+   `Accept-Language`.
+
+A later local fingerprint check found two more:
+
+1. With a full profile, `navigator.serviceWorker.register()` never
+   finished. The page and the browser both held the worker paused.
+2. Mouse presses reported `PointerEvent.pressure` 0. A real mouse reports
+   0.5 while a button is down.
+
+All four are fixed and have regression tests. No CAPTCHA was solved.

{webskrap-3.2.0 → webskrap-3.2.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "webskrap"
-version = "3.2.0"
+version = "3.2.2"
 description = "Async Python scraping on Chromium over its own CDP client, with coherent browser profiles and session controls."
 readme = "README.md"
 requires-python = ">=3.11"

{webskrap-3.2.0 → webskrap-3.2.2}/scripts/driver_parity.py

@@ -31,7 +31,6 @@ from collections.abc import AsyncIterator, Awaitable, Callable
 from contextlib import AbstractAsyncContextManager, asynccontextmanager
 from dataclasses import dataclass
 from pathlib import Path
-from random import SystemRandom
 from typing import Any
 from uuid import uuid4
 
@@ -43,8 +42,6 @@ from webskrap.display import VirtualDisplay
 from webskrap.profiles import get_profile
 
 REPORT = Path(".webskrap/reports/driver-parity.json")
-# Cursor jitter only; system entropy keeps the scanner quiet, as in human.py.
-_rng = SystemRandom()
 
 # A page passes when no known challenge is on it and its title is not a
 # challenge or block title.
@@ -120,8 +117,6 @@ TARGETS: dict[str, tuple[str, str]] = {
 Evaluate = Callable[[str], Awaitable[Any]]
 #: Writes a PNG of the page's viewport to the given path.
 Screenshot = Callable[[Path], Awaitable[None]]
-#: Sends one mouse event: ("move" | "down" | "up", x, y) in viewport CSS pixels.
-Mouse = Callable[[str, float, float], Awaitable[None]]
 
 
 @dataclass
@@ -130,51 +125,11 @@ class Page:
 
     evaluate: Evaluate
     screenshot: Screenshot
-    mouse: Mouse
     #: Returns DataDome's response page kind from its iframe, or None when the
     #: path cannot read that iframe.
     datadome_kind: Callable[[], Awaitable[str | None]] | None = None
 
 
-# Finds the Turnstile widget: the parent of its response input, or else the
-# first widget-high box. The host can span the content width while the 300 px
-# iframe sits at its left edge, so only the height is checked. Returns the
-# host's viewport rectangle.
-_TURNSTILE_BOX = """(() => {
-  const fits = (r) => r.width >= 250 && r.height >= 55 && r.height <= 80;
-  const input = document.querySelector('input[name="cf-turnstile-response"]');
-  const candidates = [input && input.parentElement, ...document.querySelectorAll('div')];
-  for (const element of candidates) {
-    if (!element) continue;
-    const r = element.getBoundingClientRect();
-    if (fits(r)) return { x: r.left, y: r.top, w: r.width, h: r.height };
-  }
-  return null;
-})()"""
-
-
-async def human_click(mouse: Mouse, x: float, y: float) -> None:
-    """Move along an eased, jittered curve to (x, y), then press and release.
-
-    Both paths get the same cursor path through CDP input events, so the
-    comparison measures the browser setup, not the cursor model.
-    """
-    start_x, start_y = x + _rng.uniform(250, 450), y + _rng.uniform(150, 300)
-    control_x, control_y = (start_x + x) / 2 + _rng.uniform(-80, 80), start_y - 60
-    steps = _rng.randint(22, 32)
-    for step in range(1, steps + 1):
-        t = step / steps
-        t = t * t * (3 - 2 * t)  # ease in and out
-        px = (1 - t) ** 2 * start_x + 2 * (1 - t) * t * control_x + t * t * x
-        py = (1 - t) ** 2 * start_y + 2 * (1 - t) * t * control_y + t * t * y
-        await mouse("move", px + _rng.uniform(-0.8, 0.8), py + _rng.uniform(-0.8, 0.8))
-        await asyncio.sleep(_rng.uniform(0.008, 0.022))
-    await asyncio.sleep(_rng.uniform(0.12, 0.3))
-    await mouse("down", x, y)
-    await asyncio.sleep(_rng.uniform(0.06, 0.12))
-    await mouse("up", x, y)
-
-
 def _config(mode: str) -> SessionConfig:
     return SessionConfig(
         channel="chromium",
@@ -189,22 +144,6 @@ def _page(page: CDPPage) -> Page:
         shot = await page.send("Page.captureScreenshot", {"format": "png"})
         path.write_bytes(base64.b64decode(shot["data"]))
 
-    async def mouse(kind: str, x: float, y: float) -> None:
-        # A held left button reports buttons=1 until it is released.
-        event = {"move": "mouseMoved", "down": "mousePressed", "up": "mouseReleased"}[kind]
-        pressed = kind == "down"
-        await page.send(
-            "Input.dispatchMouseEvent",
-            {
-                "type": event,
-                "x": x,
-                "y": y,
-                "button": "none" if kind == "move" else "left",
-                "buttons": 1 if pressed else 0,
-                "clickCount": 0 if kind == "move" else 1,
-            },
-        )
-
     # Isolated worlds: the page cannot see the checks, and they need no globals.
     async def datadome_kind() -> str | None:
         for frame in await cdp_dom.frames(page):
@@ -212,7 +151,7 @@ def _page(page: CDPPage) -> Page:
                 return await frame.page.evaluate(_DATADOME_KIND)
         return None
 
-    return Page(page.evaluate, screenshot, mouse, datadome_kind)
+    return Page(page.evaluate, screenshot, datadome_kind)
 
 
 @asynccontextmanager
@@ -257,7 +196,7 @@ def _via_session() -> Callable[[str, str], AbstractAsyncContextManager[Page]]:
     """Run a target through a persistent session, as `webskrap browser` does.
 
     Every call attaches anew, like each CLI command or MCP tool call. Sessions
-    run headless or headed; a virtual display and clicks are not offered.
+    run headless or headed; a virtual display is not offered.
     """
 
     @asynccontextmanager
@@ -281,10 +220,7 @@ def _via_session() -> Callable[[str, str], AbstractAsyncContextManager[Page]]:
             async def screenshot(path: Path) -> None:
                 await act(lambda page: browser_session.screenshot(page, path, full_page=False))
 
-            async def mouse(_kind: str, _x: float, _y: float) -> None:
-                raise NotImplementedError("--click is not offered with --via session")
-
-            yield Page(evaluate, screenshot, mouse)
+            yield Page(evaluate, screenshot)
         finally:
             browser_session.close_session(name, delete_data=True)
 
@@ -293,29 +229,6 @@ def _via_session() -> Callable[[str, str], AbstractAsyncContextManager[Page]]:
 
 SESSION_DRIVERS = {"session": _via_session()}
 
-# The widget box renders before the checkbox inside its iframe does.
-WIDGET_SETTLE_S = 3.0
-
-
-async def _click_turnstile(page: Page, shot: Path | None) -> bool:
-    """Click the Turnstile checkbox once it shows; False while it is absent."""
-    box = None
-    with contextlib.suppress(Exception):
-        box = await page.evaluate(_TURNSTILE_BOX)
-    if not box:
-        await asyncio.sleep(1)
-        return False
-    await asyncio.sleep(WIDGET_SETTLE_S)
-    if shot is not None:
-        with contextlib.suppress(Exception):
-            shot.parent.mkdir(parents=True, exist_ok=True)
-            await page.screenshot(shot.with_name(f"{shot.stem}-before-click.png"))
-    # The checkbox sits near the widget's left edge, centered vertically.
-    x = box["x"] + 22 + _rng.uniform(-3, 3)
-    y = box["y"] + box["h"] / 2 + _rng.uniform(-3, 3)
-    await human_click(page.mouse, x, y)
-    return True
-
 
 async def run_once(
     driver: str,
@@ -323,7 +236,6 @@ async def run_once(
     target: str,
     wait_s: float,
     shot: Path | None = None,
-    click: bool = False,
     via: str = "launch",
 ) -> dict[str, Any]:
     url, check = TARGETS[target]
@@ -332,7 +244,6 @@ async def run_once(
     # A pass after a challenge means the challenge cleared; a pass without one
     # means the target never challenged this run.
     challenged = False
-    clicked = False
     try:
         drivers = SESSION_DRIVERS if via == "session" else DRIVERS
         async with drivers[driver](url, mode) as page:
@@ -348,9 +259,6 @@ async def run_once(
                 challenged = challenged or state in ("challenge", "datadome", "captcha")
                 if state == "passed":
                     break
-                if click and not clicked and state == "challenge":
-                    clicked = await _click_turnstile(page, shot)
-                    continue
                 if state in _SETTLING and time.monotonic() - started >= SETTLE_S:
                     break
                 await asyncio.sleep(1)
@@ -368,7 +276,6 @@ async def run_once(
         "target": target,
         "seconds": round(time.monotonic() - started, 1),
         "challenged": challenged,
-        "clicked": clicked,
         **verdict,
     }
 
@@ -390,11 +297,6 @@ async def main() -> None:
         type=Path,
         help="Write a PNG of each run's last state to this directory.",
     )
-    parser.add_argument(
-        "--click",
-        action="store_true",
-        help="Click a Cloudflare Turnstile checkbox when a challenge shows it.",
-    )
     parser.add_argument(
         "--via",
         choices=("launch", "session"),
@@ -402,8 +304,8 @@ async def main() -> None:
         help="launch a browser per run, or go through a persistent session.",
     )
     args = parser.parse_args()
-    if args.via == "session" and (args.click or "virtual-display" in (args.mode or [])):
-        parser.error("--via session runs headless or headed, without --click")
+    if args.via == "session" and "virtual-display" in (args.mode or []):
+        parser.error("--via session runs headless or headed, not on a virtual display")
     modes = args.mode or (
         ["headless"] if args.via == "session" else ["headless", "virtual-display"]
     )
@@ -429,16 +331,13 @@ async def main() -> None:
                         if args.screenshots
                         else None
                     )
-                    result = await run_once(
-                        driver, mode, target, args.wait, shot, click=args.click, via=args.via
-                    )
+                    result = await run_once(driver, mode, target, args.wait, shot, via=args.via)
                     results.append(result)
                     cleared = result["challenged"] and result["state"] == "passed"
                     print(
                         f"{target:13} {mode:15} {driver:10} {result['state']:9} "
                         f"{result['seconds']:5}s {result['detail']}"
-                        f"{' (after challenge)' if cleared else ''}"
-                        f"{' [clicked]' if result['clicked'] else ''}",
+                        f"{' (after challenge)' if cleared else ''}",
                         flush=True,
                     )
 

{webskrap-3.2.0 → webskrap-3.2.2}/src/webskrap/cdp_dom.py

@@ -38,6 +38,10 @@ from webskrap.human import MAX_WHEEL_BATCHES, VIEW_MARGIN, bezier_path, click_po
 #: then ``e`` and Chromium's backend node id.
 REF_PATTERN = re.compile(r"^(?:f(\d+))?e(\d+)$")
 _POLL_S = 0.1
+#: ``force`` for a held mouse button. CDP's default of 0 reaches pages as
+#: ``PointerEvent.pressure`` 0, but a real mouse with a button down reports
+#: 0.5, as the Pointer Events spec asks of devices without pressure sensing.
+MOUSE_PRESSED_FORCE = 0.5
 
 # Builds the snapshot text with "[ref=#N]" placeholders and returns it with
 # the listed elements, so the caller can swap each placeholder for the
@@ -917,7 +921,10 @@ async def _human_pointer(element: _Element, *, click_count: int = 0) -> None:
             msg = f"another element covers the click point of: {element.description}"
             raise WebSkrapError(msg, code=ErrorCode.USAGE)
         press = {"x": x, "y": y, "button": "left", "clickCount": count}
-        await page.send("Input.dispatchMouseEvent", {"type": "mousePressed", "buttons": 1, **press})
+        await page.send(
+            "Input.dispatchMouseEvent",
+            {"type": "mousePressed", "buttons": 1, "force": MOUSE_PRESSED_FORCE, **press},
+        )
         await asyncio.sleep(uniform(60, 140) / 1000)
         await page.send("Input.dispatchMouseEvent", {"type": "mouseReleased", **press})
 
@@ -1091,7 +1098,8 @@ async def instant_click(page: CDPPage, target: str, *, click_count: int = 1) ->
         for count in range(1, click_count + 1):
             press = {"x": x, "y": y, "button": "left", "clickCount": count}
             await element.input_page.send(
-                "Input.dispatchMouseEvent", {"type": "mousePressed", "buttons": 1, **press}
+                "Input.dispatchMouseEvent",
+                {"type": "mousePressed", "buttons": 1, "force": MOUSE_PRESSED_FORCE, **press},
             )
             await element.input_page.send(
                 "Input.dispatchMouseEvent", {"type": "mouseReleased", **press}

{webskrap-3.2.0 → webskrap-3.2.2}/src/webskrap/cdp_session.py

@@ -407,7 +407,9 @@ async def goto(
     finally:
         documents.stop()
     status = challenges.response_status(response)
-    found = await challenges.detect(page, challenges.response_headers(response), status)
+    found = await challenges.detect(
+        page, challenges.response_headers(response), status, deadline=deadline
+    )
     result: dict[str, Any] = {"status": status}
     if decline_cookies:
         result["cookie_notice_declined"] = (
@@ -750,6 +752,8 @@ _BUTTON_BITS = {"left": 1, "right": 2, "middle": 4}
 async def _mouse(
     page: CDPPage, kind: str, x: float, y: float, *, button: str = "none", **extra: Any
 ) -> None:
+    if extra.get("buttons"):
+        extra.setdefault("force", cdp_dom.MOUSE_PRESSED_FORCE)
     await page.send(
         "Input.dispatchMouseEvent",
         {"type": kind, "x": x, "y": y, "button": button, **extra},