webskrap 3.2.1__tar.gz → 3.2.2__tar.gz

webskrap-3.2.2/.package-dist.dZyl3m/.gitignore

@@ -0,0 +1 @@
+*

{webskrap-3.2.1 → webskrap-3.2.2}/CHANGELOG.md

@@ -9,6 +9,31 @@ history, so they summarize each release rather than list every change.
 
 ## [Unreleased]
 
+## [3.2.2] - 2026-10-30
+
+### Fixed
+
+- DataDome detection now follows the device check when it navigates its
+  frame. It reads the live challenge frame, and it reports a manual
+  challenge apart from a hard block. The user-agent probe now uses the
+  session proxy and network flags, so it cannot connect directly.
+- Full desktop profiles now set `navigator.platform` to match the user
+  agent. They also fit the viewport inside the real browser window before
+  the metrics override.
+- Cookie decline now finds the reject button of Commanders Act
+  TrustCommander notices. Their markup has no "cookie" or "consent" name, so
+  the text search did not look inside them.
+- Full profiles no longer put q-values in `navigator.languages`, and no longer
+  send a doubled q-value in `Accept-Language` (`fr;q=0.9;q=0.9`). Chromium
+  weights the header itself, so the override now gets a plain list.
+- With `profile_mode="full"` (or `"locale"` off Linux),
+  `navigator.serviceWorker.register()` no longer hangs. The page and the
+  browser both held the worker paused, and each waited for the other. Now
+  only the browser-level setup prepares service and shared workers; the page
+  only resumes them.
+- Mouse presses now report `PointerEvent.pressure` 0.5 while the button is
+  down, as a real mouse does. CDP's default of 0 was visible to pages.
+
 ## [3.2.1] - 2026-10-26
 
 ### Removed
@@ -936,7 +961,8 @@ entries below.
 
 Earlier releases (0.1.0 - 0.4.9) are recorded in the git history and tags only.
 
-[Unreleased]: https://github.com/kacigaya/webskrap/compare/v3.2.1...HEAD
+[Unreleased]: https://github.com/kacigaya/webskrap/compare/v3.2.2...HEAD
+[3.2.2]: https://github.com/kacigaya/webskrap/compare/v3.2.1...v3.2.2
 [3.2.1]: https://github.com/kacigaya/webskrap/compare/v3.2.0...v3.2.1
 [3.2.0]: https://github.com/kacigaya/webskrap/compare/v3.1.0...v3.2.0
 [3.1.0]: https://github.com/kacigaya/webskrap/compare/v3.0.1...v3.1.0

{webskrap-3.2.1 → webskrap-3.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: webskrap
-Version: 3.2.1
+Version: 3.2.2
 Summary: Async Python scraping on Chromium over its own CDP client, with coherent browser profiles and session controls.
 Project-URL: Homepage, https://github.com/kacigaya/webskrap
 Project-URL: Repository, https://github.com/kacigaya/webskrap
@@ -82,8 +82,21 @@ asyncio.run(main())
 For deferred content, pass `ready_selector="#content"` to a Python fetch or
 `--ready-selector '#content'` to CLI `fetch`. Both MCP fetch tools accept
 `ready_selector` too. The selector must become visible after consent dismissal.
-Known challenge pages (DataDome, Cloudflare, Akamai, AWS WAF) and provider block pages (Cloudflare, Akamai) raise `blocked`;
-WebSkrap does not retry or solve them. Ordinary HTTP errors still return `ok=False`.
+Known challenge pages (DataDome, Cloudflare, Akamai, AWS WAF) and provider block
+pages (DataDome, Cloudflare, Akamai) raise `blocked`. WebSkrap waits for device
+checks that reload the page. It does not retry refusals or solve CAPTCHAs.
+Ordinary HTTP errors still return `ok=False`.
+
+For manual CAPTCHA completion, open a persistent browser with
+`webskrap browser open --headed --session manual --proxy http://localhost:8888 URL`.
+Complete the check in that browser. Keep the same session, profile, and proxy
+for later commands. On a remote host, use a desktop connection to its display.
+`virtual_display=True` alone does not provide a remote desktop.
+
+Keep `profile_mode="native"` for the browser's own identity. On Linux with Xvfb,
+`virtual_display=True` runs a headed browser without a visible desktop window.
+Full profile emulation is useful for device tests, but cannot replace every
+worker or iframe value. See the [identity limits](src/webskrap/guide.md).
 
 ## Search
 

{webskrap-3.2.1 → webskrap-3.2.2}/README.md

@@ -48,8 +48,21 @@ asyncio.run(main())
 For deferred content, pass `ready_selector="#content"` to a Python fetch or
 `--ready-selector '#content'` to CLI `fetch`. Both MCP fetch tools accept
 `ready_selector` too. The selector must become visible after consent dismissal.
-Known challenge pages (DataDome, Cloudflare, Akamai, AWS WAF) and provider block pages (Cloudflare, Akamai) raise `blocked`;
-WebSkrap does not retry or solve them. Ordinary HTTP errors still return `ok=False`.
+Known challenge pages (DataDome, Cloudflare, Akamai, AWS WAF) and provider block
+pages (DataDome, Cloudflare, Akamai) raise `blocked`. WebSkrap waits for device
+checks that reload the page. It does not retry refusals or solve CAPTCHAs.
+Ordinary HTTP errors still return `ok=False`.
+
+For manual CAPTCHA completion, open a persistent browser with
+`webskrap browser open --headed --session manual --proxy http://localhost:8888 URL`.
+Complete the check in that browser. Keep the same session, profile, and proxy
+for later commands. On a remote host, use a desktop connection to its display.
+`virtual_display=True` alone does not provide a remote desktop.
+
+Keep `profile_mode="native"` for the browser's own identity. On Linux with Xvfb,
+`virtual_display=True` runs a headed browser without a visible desktop window.
+Full profile emulation is useful for device tests, but cannot replace every
+worker or iframe value. See the [identity limits](src/webskrap/guide.md).
 
 ## Search
 

webskrap-3.2.2/docs/diagnostics/datadome.md

@@ -0,0 +1,212 @@
+# DataDome website navigation diagnosis
+
+Checked on 2026-10-29 against a DataDome-protected website.
+
+## Root cause
+
+The request reaches the website through the existing Gluetun VPN. DataDome then
+chooses whether to show a device check, a CAPTCHA, or a hard block.
+The headless browser received a hard-block document. That document has no
+CAPTCHA widget. A headed browser on the same VPN exit received a working
+manual slider. This is an external verification requirement.
+
+WebSkrap also had three integration defects:
+
+1. Detection recognized `ct.captcha-delivery.com/c.js`, but missed the
+   `i.js` device-check script before its iframe appeared.
+2. It treated the device check as a manual challenge and did not wait for
+   its own script to reload the page.
+3. It read the parent document's iframe `src`. A cross-origin iframe can
+   navigate itself while that attribute stays unchanged. A hard block then
+   received the same hint as a solvable CAPTCHA.
+
+The optional headless user-agent probe also started a separate browser
+without the session's proxy or network flags.
+
+## Evidence
+
+- Gluetun reported a healthy tunnel and a French exit. A browser request to
+  the IP check service returned the same exit as a request from the container.
+- The proxy required authentication. Browser credentials stayed in memory.
+- The container resolved the website and the CAPTCHA script, frame, and asset hosts.
+  Its BusyBox `nslookup` printed a parse error for one CNAME response. The
+  proxy still resolved and loaded that host successfully.
+- The website returned HTTP 403 with `x-datadome: protected` and `x-dd-b: 3`.
+- The device-check script and iframe returned HTTP 200. In the headed run,
+  CAPTCHA JavaScript, CSS, fonts, and logo also returned HTTP 200.
+- Browser logs showed GPU and canvas warnings, but no failed CAPTCHA resource
+  request in the captured headed flow.
+- The headless provider frame contained
+  `data-dd-response-page="hard-block"`. Its parent had no visible body text.
+- The headed provider frame showed the manual verification slider.
+- Cookies included `datadome`, `__cf_bm`, and `__cflb`. Cookie values and
+  challenge URLs are excluded from this report.
+
+## Changes
+
+Detection now reads the active provider frame through the existing CDP frame
+code. It uses provider origin and markup, not page titles or translated text.
+Device checks use the existing bounded interstitial wait. Frame inspection
+uses the remaining navigation budget. Manual challenges stay open in persistent
+sessions. Hard blocks receive the block hint.
+
+The user-agent probe now keeps the session's launch flags, including its
+proxy and WebRTC policy. No CAPTCHA solver, token injection, provider script
+patch, new dependency, or site-specific branch was added.
+
+## Manual completion
+
+1. Start a headed persistent browser on a display you can access.
+2. Configure the VPN proxy before navigation. Keep all page resources enabled.
+3. Complete the CAPTCHA yourself. Use its retry control if it has expired.
+4. Continue with the same browser, profile, and proxy. Do not copy challenge
+   tokens into another browser.
+
+A temporary headed browser session and loopback VNC listener on port 15900 were used
+to prepare manual completion. Both were closed when manual completion was
+declined. Xvfb alone does not give remote access.
+
+## Verification limits
+
+The final Linux ARM64 run passed 858 tests. It excluded 30 opt-in live tests
+for unrelated public sites. Coverage was 88.85%, above the 85% project gate.
+The focused suite passed 123 tests before the combined run.
+
+Ruff lint and format checks, Pyright for Linux, Windows, and macOS, Bandit,
+package builds, and Twine metadata checks passed. OSV found no vulnerabilities
+in the 51 public packages from the lockfile. Pip-audit also found none.
+Windows and macOS runtime tests require those operating systems and were not
+run on this host.
+
+Local regression fixtures cover device-check completion, a transition to a
+manual challenge, a transition to a hard block, cross-origin frame navigation,
+proxy authentication, image loading, and continuation with the same cookies.
+The fixture's manual button is a local test control. It does not contact or
+complete real CAPTCHA verification.
+
+Live tests reached the rendered CAPTCHA. Navigation beyond verification
+cannot be confirmed until a person completes it. The provider can still
+refuse the session after manual verification. No forms were submitted.
+
+## VPN exit retest
+
+Eight distinct French NordVPN exits were tested, including the initial
+baseline. Each headed run returned HTTP 403 and a DataDome manual challenge.
+Fresh browser profiles were used for the replacement exits. No CAPTCHA was
+completed or bypassed. This sample did not identify a CAPTCHA-free exit.
+It does not establish that IP reputation is the only input to the decision.
+
+A control run placed the browser directly inside Gluetun's network namespace.
+It used the same VPN exit without the HTTP proxy and still received a challenge.
+The proxy is therefore not the sole cause.
+
+Rotation also exposed an operational defect: Gluetun's NordVPN server catalog
+was dated March 2024. Some listed endpoints timed out or rejected authentication.
+The catalog was refreshed through Gluetun's existing updater. The original
+catalog was backed up, and the VPN was restored to a healthy state.
+
+The final verified exit was `187.13.198.190` in Marseille. It still required
+verification. The refreshed catalog is in the container's writable layer;
+container recreation can replace it because there is no catalog volume mount.
+
+## CDP identity investigation
+
+An ordinary headed Chromium process was launched on the same VPN exit with
+no debugging port, no debugging pipe, and no WebSkrap page setup. Its native
+network log recorded an HTTP 403, a device check, and then a CAPTCHA.
+This establishes that CDP is not required to trigger the current challenge.
+It does not reveal DataDome's private scoring rule.
+
+A separate public fingerprint probe reported `isBot: false` for default
+headed WebSkrap. It reported no CDP, webdriver, worker-value, or client-hint
+inconsistency. The headed Mesa/locale configuration also passed that probe.
+Both configurations still received a CAPTCHA. The headless run was
+flagged for its bot user agent and weak signals, with no CDP leak reported.
+These are observations from one public probe, not universal detection claims.
+
+Local page-owned probes compared request headers and JavaScript values in
+documents, cross-origin iframes, dedicated workers, and shared workers.
+Two full-profile setup defects were reproduced and fixed:
+
+1. A Windows or macOS user agent changed client hints but left the document's
+   `navigator.platform` at its native Linux value. The CDP override now sets
+   an unambiguous legacy platform as well. Unknown CPU values are not guessed.
+2. A desktop viewport could be wider than its real browser window. The new
+   regression reproduced `innerWidth=1365` with `outerWidth=780`. Setup now
+   resizes the real window before applying device metrics. A missing command
+   on an older browser keeps the previous behavior; other errors propagate.
+
+The changes use Chromium's own
+[user-agent override](https://chromedevtools.github.io/devtools-protocol/tot/Emulation/#method-setUserAgentOverride)
+and [content-size command](https://chromedevtools.github.io/devtools-protocol/tot/Browser/#method-setContentsSize).
+No JavaScript getter replacements were added.
+
+The probe also confirmed existing limits. UA masking removes high-entropy
+client hints. Cross-process frames keep their real screen and pixel ratio.
+Workers keep their native legacy platform, and shared workers keep their
+native user agent. Full cross-platform emulation therefore cannot promise a
+coherent identity across all contexts. Native mode leaves those values intact.
+Neither the fixes nor changing the GPU and locale removed the website's verification
+requirement in the measured runs.
+
+The final follow-up suite ran inside Gluetun's network namespace. It passed
+872 tests with 88.89% coverage and excluded 30 opt-in live tests. The focused
+suite passed 125 tests. Lint, format, Linux/Windows/macOS type checks, Bandit,
+OSV, pip-audit, package builds, and Twine checks passed. Windows and macOS
+runtime checks were not available on this Linux host.
+
+## Protected form retest
+
+A later retest compared ordinary pages with the website's quote form path.
+Each run used a fresh browser profile and a VPN exit that the run had not
+used before.
+
+The browser passed DataDome on ordinary pages. On some exits the first page
+returned HTTP 403 with a device check (`x-dd-b: 3`). The device check then
+reloaded into the requested page with HTTP 200. On other exits the first page
+returned HTTP 200 at once. DataDome's page tag ran on those pages.
+
+The quote form path returned HTTP 403 with a CAPTCHA on every exit. This
+occurred with a direct load and with a click-through from the landing page.
+These changes did not remove the CAPTCHA:
+
+1. Google Chrome stable for Linux ARM64 instead of Chrome for Testing. This
+   adds the `Google Chrome` brand.
+2. A locale and a timezone that match the exit country.
+3. Mesa WebGL.
+4. `x86` client hints instead of the host's `arm` value.
+5. A 25-second pause with mouse movement and wheel scrolling on each page.
+6. A four-page session with about 75 seconds of activity before the form.
+7. Exits in eleven countries on eight hosting ASNs.
+
+A desktop-like host environment did not change the result. It added Noto
+and Ubuntu fonts and PulseAudio audio devices. On a fresh exit, a direct load
+of the form received a device check (`x-dd-b: 3`). The check reloaded into a
+CAPTCHA. The same check reloaded into HTTP 200 on ordinary pages.
+
+SwiftShader WebGL failed the device check and received a hard block.
+The VPN provider's dedicated-IP servers refused this setup's credentials,
+so a dedicated exit was not tested.
+
+A Windows user agent made the result worse: the first page returned a hard
+block. Repeated fresh profiles on one exit also made the result worse. A later
+visit received the CAPTCHA, or a hard block, without a device check.
+
+The VPN provider's catalog contains only hosting ASNs. The measured limit is
+therefore the exit IP class for this form path, not a browser defect that
+WebSkrap can fix. A residential or mobile IP was not tested.
+
+The retest found two WebSkrap defects:
+
+1. Cookie decline did not search Commanders Act TrustCommander notices.
+2. Full profiles put q-values in `navigator.languages` and doubled them in
+   `Accept-Language`.
+
+A later local fingerprint check found two more:
+
+1. With a full profile, `navigator.serviceWorker.register()` never
+   finished. The page and the browser both held the worker paused.
+2. Mouse presses reported `PointerEvent.pressure` 0. A real mouse reports
+   0.5 while a button is down.
+
+All four are fixed and have regression tests. No CAPTCHA was solved.

{webskrap-3.2.1 → webskrap-3.2.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "webskrap"
-version = "3.2.1"
+version = "3.2.2"
 description = "Async Python scraping on Chromium over its own CDP client, with coherent browser profiles and session controls."
 readme = "README.md"
 requires-python = ">=3.11"

{webskrap-3.2.1 → webskrap-3.2.2}/src/webskrap/cdp_dom.py

@@ -38,6 +38,10 @@ from webskrap.human import MAX_WHEEL_BATCHES, VIEW_MARGIN, bezier_path, click_po
 #: then ``e`` and Chromium's backend node id.
 REF_PATTERN = re.compile(r"^(?:f(\d+))?e(\d+)$")
 _POLL_S = 0.1
+#: ``force`` for a held mouse button. CDP's default of 0 reaches pages as
+#: ``PointerEvent.pressure`` 0, but a real mouse with a button down reports
+#: 0.5, as the Pointer Events spec asks of devices without pressure sensing.
+MOUSE_PRESSED_FORCE = 0.5
 
 # Builds the snapshot text with "[ref=#N]" placeholders and returns it with
 # the listed elements, so the caller can swap each placeholder for the
@@ -917,7 +921,10 @@ async def _human_pointer(element: _Element, *, click_count: int = 0) -> None:
             msg = f"another element covers the click point of: {element.description}"
             raise WebSkrapError(msg, code=ErrorCode.USAGE)
         press = {"x": x, "y": y, "button": "left", "clickCount": count}
-        await page.send("Input.dispatchMouseEvent", {"type": "mousePressed", "buttons": 1, **press})
+        await page.send(
+            "Input.dispatchMouseEvent",
+            {"type": "mousePressed", "buttons": 1, "force": MOUSE_PRESSED_FORCE, **press},
+        )
         await asyncio.sleep(uniform(60, 140) / 1000)
         await page.send("Input.dispatchMouseEvent", {"type": "mouseReleased", **press})
 
@@ -1091,7 +1098,8 @@ async def instant_click(page: CDPPage, target: str, *, click_count: int = 1) ->
         for count in range(1, click_count + 1):
             press = {"x": x, "y": y, "button": "left", "clickCount": count}
             await element.input_page.send(
-                "Input.dispatchMouseEvent", {"type": "mousePressed", "buttons": 1, **press}
+                "Input.dispatchMouseEvent",
+                {"type": "mousePressed", "buttons": 1, "force": MOUSE_PRESSED_FORCE, **press},
             )
             await element.input_page.send(
                 "Input.dispatchMouseEvent", {"type": "mouseReleased", **press}

{webskrap-3.2.1 → webskrap-3.2.2}/src/webskrap/cdp_session.py

@@ -407,7 +407,9 @@ async def goto(
     finally:
         documents.stop()
     status = challenges.response_status(response)
-    found = await challenges.detect(page, challenges.response_headers(response), status)
+    found = await challenges.detect(
+        page, challenges.response_headers(response), status, deadline=deadline
+    )
     result: dict[str, Any] = {"status": status}
     if decline_cookies:
         result["cookie_notice_declined"] = (
@@ -750,6 +752,8 @@ _BUTTON_BITS = {"left": 1, "right": 2, "middle": 4}
 async def _mouse(
     page: CDPPage, kind: str, x: float, y: float, *, button: str = "none", **extra: Any
 ) -> None:
+    if extra.get("buttons"):
+        extra.setdefault("force", cdp_dom.MOUSE_PRESSED_FORCE)
     await page.send(
         "Input.dispatchMouseEvent",
         {"type": kind, "x": x, "y": y, "button": button, **extra},

{webskrap-3.2.1 → webskrap-3.2.2}/src/webskrap/cdp_setup.py

@@ -18,7 +18,8 @@ Chromium refuses the device-metrics override below top-level pages, so an
 iframe still reports the real screen size and pixel ratio. A shared worker
 keeps the browser's own ``navigator.userAgent``; it is read before the
 override arrives. ``mask_headless_user_agent`` sets the user agent for the
-whole browser.
+whole browser. Workers also keep their native ``navigator.platform``.
+Desktop full profiles resize the browser window to fit the requested viewport.
 
 Proxy credentials are answered only to challenges whose source is the proxy.
 A website asking for HTTP authentication gets the browser's default answer,
@@ -37,7 +38,7 @@ from typing import Any
 from urllib.parse import urlsplit
 
 from webskrap import cdp_dom
-from webskrap.cdp import CDPBrowser, CDPConnection, CDPPage
+from webskrap.cdp import CDPBrowser, CDPConnection, CDPError, CDPPage
 from webskrap.errors import ErrorCode, WebSkrapError
 from webskrap.models import BrowserProfile, ResourcePolicy, SessionConfig, user_agent_metadata
 
@@ -60,7 +61,8 @@ _NOT_ON_WORKERS = _TOP_LEVEL_ONLY | {
     "Emulation.setEmulatedMedia",
     "Emulation.setLocaleOverride",
 }
-_WORKER_TYPES = frozenset({"worker", "shared_worker", "service_worker"})
+_BROWSER_WORKER_TYPES = frozenset({"shared_worker", "service_worker"})
+_WORKER_TYPES = _BROWSER_WORKER_TYPES | {"worker"}
 
 # A blank document served for an origin while its localStorage is restored.
 _BLANK_PAGE = base64.b64encode(b"<!doctype html><title></title>").decode()
@@ -70,6 +72,8 @@ async def prepare_page(page: CDPPage, config: SessionConfig, profile: BrowserPro
     """Apply ``config`` and ``profile`` to a fresh page before it navigates."""
     await page.enable_events()
     page.default_timeout_s = config.default_timeout_ms / 1000
+    if config.profile_mode == "full" and not profile.is_mobile:
+        await _fit_window(page, profile)
     commands = await _emulation(page.connection, config, profile)
     for method, params in commands:
         await page.send(method, params)
@@ -81,6 +85,24 @@ async def prepare_page(page: CDPPage, config: SessionConfig, profile: BrowserPro
         )
 
 
+async def _fit_window(page: CDPPage, profile: BrowserProfile) -> None:
+    """Fit a desktop viewport inside a real browser window before emulation."""
+    window = await page.connection.send("Browser.getWindowForTarget", {"targetId": page.target_id})
+    try:
+        await page.connection.send(
+            "Browser.setContentsSize",
+            {
+                "windowId": window["windowId"],
+                "width": profile.viewport.width,
+                "height": profile.viewport.height,
+            },
+        )
+    except CDPError as exc:
+        # Older Chromium versions lack this experimental browser command.
+        if not str(exc).startswith("CDP error -32601:"):
+            raise
+
+
 #: The brand each channel's browser reports next to ``Chromium``.
 _CHANNEL_BRANDS: dict[str | None, str] = {"chrome": "Google Chrome", "msedge": "Microsoft Edge"}
 
@@ -110,15 +132,15 @@ async def prepare_browser(
 ) -> None:
     """Give shared and service workers the session's identity and routing.
 
-    They are browser targets, so a page's auto-attach does not report them.
-    Each one waits for the debugger until it is set up.
+    They are browser targets: only this browser-level auto-attach sets them
+    up. Each one waits for the debugger until it is set up.
     """
     connection = browser.connection
     commands = await _emulation(connection, config, profile)
     if not commands and not _routes(config):
         return
     timeout_s = config.default_timeout_ms / 1000
-    handler = _on_attached(connection, timeout_s, config, commands)
+    handler = _on_attached(connection, timeout_s, config, commands, browser_level=True)
     connection.on("Target.attachedToTarget", handler, session_id=None)
     await connection.send(
         "Target.setAutoAttach",
@@ -136,18 +158,28 @@ def _on_attached(
     timeout_s: float,
     config: SessionConfig,
     commands: list[tuple[str, dict[str, Any]]],
+    *,
+    browser_level: bool = False,
 ) -> Callable[[dict[str, Any]], None]:
     """Return a ``Target.attachedToTarget`` handler that sets up and resumes targets.
 
     ``commands`` are the page's emulation commands, from :func:`_emulation`.
+    A page's auto-attach also reports its service workers, which the
+    browser-level handler holds paused too. Held by two sessions, a worker
+    answers no setup command until both resume it, so two setups wait for
+    each other forever: page-level handlers only resume these workers.
     """
     tasks: set[asyncio.Task[None]] = set()
 
     def attached(params: dict[str, Any]) -> None:
         info = params.get("targetInfo", {})
+        kind = info.get("type")
         child = CDPPage(connection, params["sessionId"], info.get("targetId", ""))
         child.default_timeout_s = timeout_s
-        task = asyncio.create_task(_prepare_target(child, info.get("type"), config, commands))
+        if kind in _BROWSER_WORKER_TYPES and not browser_level:
+            task = asyncio.create_task(_resume(child))
+        else:
+            task = asyncio.create_task(_prepare_target(child, kind, config, commands))
         tasks.add(task)
         task.add_done_callback(tasks.discard)
 
@@ -173,7 +205,7 @@ async def _prepare_target(
                 await _route(child, config)
                 handler = _on_attached(child.connection, child.default_timeout_s, config, commands)
                 await cdp_dom.start_auto_attach(child, handler)
-            elif kind in ("shared_worker", "service_worker"):
+            elif kind in _BROWSER_WORKER_TYPES:
                 await _route(child, config)
         with suppress(WebSkrapError):
             skipped = _NOT_ON_WORKERS if kind in _WORKER_TYPES else _TOP_LEVEL_ONLY
@@ -181,8 +213,12 @@ async def _prepare_target(
                 if method not in skipped:
                     await child.send(method, params)
     finally:
-        with suppress(WebSkrapError):
-            await child.send("Runtime.runIfWaitingForDebugger")
+        await _resume(child)
+
+
+async def _resume(child: CDPPage) -> None:
+    with suppress(WebSkrapError):
+        await child.send("Runtime.runIfWaitingForDebugger")
 
 
 def _document_switches(config: SessionConfig) -> bool:

{webskrap-3.2.1 → webskrap-3.2.2}/src/webskrap/challenges.py

@@ -28,6 +28,7 @@ from collections.abc import Mapping
 from contextlib import suppress
 from dataclasses import dataclass
 from typing import Any, Literal, Protocol
+from urllib.parse import urlsplit
 
 from webskrap.cdp import CDPError, CDPPage, EventRecorder
 from webskrap.errors import ErrorCode, WebSkrapError
@@ -128,19 +129,32 @@ def blocked_hint(kind: BlockKind) -> str:
 # 1012 (access denied), 1015 (rate limited) and 1020 (firewall rule).
 _DETECT_SCRIPT = r"""({status, ray, cloudflare, akamai}) => {
   const found = (provider, kind, reloads = false) => ({provider, kind, reloads});
+  const here = new URL(location.href);
+  if (here.hostname === 'geo.captcha-delivery.com'
+      && /^\/(captcha|interstitial)\//.test(here.pathname)) {
+    if (here.searchParams.get('t') === 'b'
+        || document.querySelector('[data-dd-response-page="hard-block"]'))
+      return found('DataDome', 'block');
+    return found('DataDome', 'challenge', here.pathname.startsWith('/interstitial/'));
+  }
+  let datadome = null;
   for (const element of document.querySelectorAll('script[src], iframe[src]')) {
     let url;
     try { url = new URL(element.getAttribute('src'), document.baseURI); }
     catch { continue; }
     if (element.tagName === 'SCRIPT' && url.hostname === 'ct.captcha-delivery.com'
-        && url.pathname === '/c.js') return found('DataDome', 'challenge');
+        && /^\/[ci]\.js$/.test(url.pathname))
+      datadome = found('DataDome', 'challenge', url.pathname === '/i.js');
     if (element.tagName === 'IFRAME' && url.hostname === 'geo.captcha-delivery.com'
-        && /^\/(captcha|interstitial)\//.test(url.pathname)) return found('DataDome', 'challenge');
+        && /^\/(captcha|interstitial)\//.test(url.pathname))
+      return found('DataDome', url.searchParams.get('t') === 'b' ? 'block' : 'challenge',
+        url.pathname.startsWith('/interstitial/'));
     if (element.tagName === 'SCRIPT'
         && url.pathname.startsWith('/cdn-cgi/challenge-platform/')
         && /^Just a moment[.!\s]*$/i.test(document.title.trim()))
       return found('Cloudflare', 'challenge');
   }
+  if (datadome) return datadome;
   if (document.getElementById('sec-if-cpt-container')) return found('Akamai', 'challenge', true);
   const text = document.body ? document.body.textContent : '';
   if (cloudflare && (status === 403 || status === 429)) {
@@ -171,7 +185,11 @@ def _cloudflare_ray(headers: Mapping[str, str]) -> str | None:
 
 
 async def detect(
-    page: ScriptRunner, headers: Mapping[str, str], status: int | None = None
+    page: ScriptRunner,
+    headers: Mapping[str, str],
+    status: int | None = None,
+    *,
+    deadline: float | None = None,
 ) -> Detection | None:
     """Return the challenge or block page ``page`` shows, or None.
 
@@ -199,11 +217,42 @@ async def detect(
     provider, kind = found.get("provider"), found.get("kind")
     if provider not in ("DataDome", "Cloudflare", "Akamai") or kind not in ("challenge", "block"):
         return None
-    return Detection(provider, kind, reloads=bool(found.get("reloads")))
+    detection = Detection(provider, kind, reloads=bool(found.get("reloads")))
+    if provider == "DataDome" and isinstance(page, CDPPage):
+        budget = page.default_timeout_s
+        if deadline is not None:
+            budget = max(0.0, min(budget, deadline - time.monotonic()))
+        with suppress(TimeoutError):
+            return await asyncio.wait_for(_datadome_frame(page, detection), budget)
+    return detection
+
+
+async def _datadome_frame(page: CDPPage, fallback: Detection) -> Detection:
+    """Read the active provider frame, whose URL can differ from its element's src."""
+    # Reuse the browser's frame handling, including out-of-process iframes.
+    from webskrap import cdp_dom
+
+    with suppress(WebSkrapError):
+        for frame in await cdp_dom.frames(page):
+            url = urlsplit(frame.url)
+            if url.hostname != "geo.captcha-delivery.com" or not url.path.startswith(
+                ("/captcha/", "/interstitial/")
+            ):
+                continue
+            found = await frame.page.evaluate(_DETECT_SCRIPT, {})
+            if isinstance(found, dict) and found.get("provider") == "DataDome":
+                kind = found.get("kind")
+                if kind in ("block", "challenge"):
+                    return Detection("DataDome", kind, reloads=bool(found.get("reloads")))
+    return fallback
 
 
 async def raise_for_challenge(
-    page: ScriptRunner, headers: Mapping[str, str], status: int | None = None
+    page: ScriptRunner,
+    headers: Mapping[str, str],
+    status: int | None = None,
+    *,
+    deadline: float | None = None,
 ) -> None:
     """Raise ``blocked`` for a recognized challenge or block page.
 
@@ -211,7 +260,7 @@ async def raise_for_challenge(
     The exception contains no page-controlled text, IPs, Ray IDs, reference
     IDs or session tokens; its hint depends on the kind.
     """
-    found = await detect(page, headers, status)
+    found = await detect(page, headers, status, deadline=deadline)
     if found is not None:
         raise found.error()
 
@@ -262,9 +311,9 @@ async def settle(
 ) -> dict[str, Any]:
     """Let a self-clearing interstitial reload into the requested page.
 
-    Akamai's sec-cpt page and AWS WAF's challenge action run their own script
-    and reload the URL, as they do in any browser; nothing here interacts with
-    them. Waits at most :data:`INTERSTITIAL_TIMEOUT_S` and never past
+    DataDome's device check, Akamai's sec-cpt page and AWS WAF's challenge run
+    their own script and reload the URL; nothing here interacts with them.
+    Waits at most :data:`INTERSTITIAL_TIMEOUT_S` and never past
     ``deadline``. ``events`` comes from :func:`record_documents`.
 
     Returns:
@@ -278,7 +327,9 @@ async def settle(
     current = document if document is not None else response
     while True:
         try:
-            found = await detect(page, response_headers(current), response_status(current))
+            found = await detect(
+                page, response_headers(current), response_status(current), deadline=limit
+            )
         except CDPError:
             found = Detection("", "challenge", reloads=True)  # the document is being replaced
         if found is None or not found.reloads or time.monotonic() >= limit: