webskrap 3.2.4__tar.gz → 3.3.0__tar.gz

{webskrap-3.2.4 → webskrap-3.3.0}/CHANGELOG.md

@@ -9,6 +9,41 @@ history, so they summarize each release rather than list every change.
 
 ## [Unreleased]
 
+## [3.3.0] - 2026-11-14
+
+### Added
+
+- MCP `browser_find` locates accessible elements and visible text with minimal
+  ancestor context. `browser_snapshot` accepts subtree targets and optional
+  bounding boxes. Existing interaction refs and response fields stay compatible.
+- Named MCP fetch sessions reuse a browser across repeated requests. Close them
+  with `fetch_session_close`.
+
+### Changed
+
+- Page extraction uses one browser evaluation.
+- Fetch defaults use `domcontentloaded` and automatic resource policies:
+  lightweight resources for text, all resources for HTML. Use explicit
+  `networkidle` or `resource_policy="all"` when a task needs them.
+- Refresh the Python dependency lock and update setup-uv to 10.3.3.
+
+### Fixed
+
+- Preserve empty HTTP error responses instead of reporting a navigation failure.
+- Keep CLI schema output compatible with Typer 0.29 type descriptors and older
+  Typer parameter types.
+- Accessibility searches retain inline paragraph, status and form-value text.
+  Snapshots follow the focused dialog when native modal dialogs are stacked.
+- Wait for the local cross-site fixture's image request before checking it in
+  browser tests.
+
+### Security
+
+- Omit password input values from accessibility snapshots. Escape URL metadata
+  and normalize line separators so page attributes cannot inject fake snapshot
+  lines. Detect detached refs before interaction.
+- Update locked OpenTelemetry API to 1.45.2 for GHSA-mf5v-w3w3-f6cg.
+
 ## [3.2.4] - 2026-11-01
 
 ### Added
@@ -1001,7 +1036,8 @@ entries below.
 
 Earlier releases (0.1.0 - 0.4.9) are recorded in the git history and tags only.
 
-[Unreleased]: https://github.com/kacigaya/webskrap/compare/v3.2.4...HEAD
+[Unreleased]: https://github.com/kacigaya/webskrap/compare/v3.3.0...HEAD
+[3.3.0]: https://github.com/kacigaya/webskrap/compare/v3.2.4...v3.3.0
 [3.2.4]: https://github.com/kacigaya/webskrap/compare/v3.2.3...v3.2.4
 [3.2.3]: https://github.com/kacigaya/webskrap/compare/v3.2.2...v3.2.3
 [3.2.2]: https://github.com/kacigaya/webskrap/compare/v3.2.1...v3.2.2

{webskrap-3.2.4 → webskrap-3.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: webskrap
-Version: 3.2.4
+Version: 3.3.0
 Summary: Async Python scraping on Chromium over its own CDP client, with coherent browser profiles and session controls.
 Project-URL: Homepage, https://github.com/kacigaya/webskrap
 Project-URL: Repository, https://github.com/kacigaya/webskrap
@@ -87,6 +87,9 @@ Known challenge pages (DataDome, Cloudflare, Akamai, AWS WAF) and provider block
 pages (DataDome, Cloudflare, Akamai) raise `blocked`. WebSkrap waits for device
 checks that reload the page. It does not retry refusals or solve CAPTCHAs.
 Ordinary HTTP errors still return `ok=False`.
+For an empty HTTP error body, Chromium can show its own error page. The returned
+text or HTML is that rendered page. The status, headers, and `final_url` still
+come from the HTTP response.
 
 For manual CAPTCHA completion, open a persistent browser with
 `webskrap browser open --headed --session manual --proxy http://localhost:8888 URL`.
@@ -99,6 +102,30 @@ Keep `profile_mode="native"` for the browser's own identity. On Linux with Xvfb,
 Full profile emulation is useful for device tests, but cannot replace every
 worker or iframe value. See the [identity limits](src/webskrap/guide.md).
 
+## Python batches
+
+Use `client.fetch_many(urls)` to fetch a finite batch in order with one browser:
+
+```python
+async with WebSkrapClient() as client:
+    results = await client.fetch_many(
+        ["https://example.com", "https://example.org"], text_only=True
+    )
+```
+
+Each batch shares cookies and storage, then closes its browser. Separate batches
+and `client.fetch()` calls use fresh sessions. An explicit `user_data_dir` can
+still share state on disk. Empty batches do not launch a browser. The first page
+error stops the batch and raises; no partial list is returned. Ordinary HTTP
+errors remain results with `ok=False`. `screenshot=True` uses a separate generated
+PNG filename for each page.
+
+For large inputs, per-page options, or state across batches, create a session
+with `session = await client.session("crawl")`. Call `await session.fetch(url)`
+for each URL. Close it with `await client.close_session("crawl")`, or let the
+client context close it. Session names belong to one client. Use separate
+sessions for unrelated callers.
+
 ## Search
 
 A search loads the engine's results page in the same stealth browser and
@@ -144,6 +171,19 @@ Coordinates are CSS pixels from the original image's top-left. Take a fresh
 view after scrolling or page changes. Headless Chromium needs no Xvfb for
 this; the agent's client and model must support image input.
 
+## Load only what the task needs
+The default resource policy is `auto`. Text-only fetches skip images, fonts
+and media. HTML fetches, screenshots and interactive pages load all resources.
+Set `resource_policy="all"`, `"lite"`, or `"documents"` to override this choice.
+Use `documents` only when removing CSS preserves the required result.
+Fetches use `domcontentloaded`; set `ready_selector` for deferred content.
+Use `browser_find(text="Save")` to locate a known control with ancestor context.
+Use `browser_snapshot(target="e15", depth=2)` for one subtree; add `boxes=true`
+only when geometry matters. Reuse returned refs with `browser_interact`.
+Use `networkidle`, screenshots and full snapshots only when the task needs them.
+CLI and MCP reject cookies by default. Disable this only for known pages where
+it is unnecessary. In Python, enable rejection with `SessionConfig(decline_cookies=True)`.
+
 ## Documentation
 
 - [Agent skill](.agents/skills/webskrap/SKILL.md)

{webskrap-3.2.4 → webskrap-3.3.0}/README.md

@@ -52,6 +52,9 @@ Known challenge pages (DataDome, Cloudflare, Akamai, AWS WAF) and provider block
 pages (DataDome, Cloudflare, Akamai) raise `blocked`. WebSkrap waits for device
 checks that reload the page. It does not retry refusals or solve CAPTCHAs.
 Ordinary HTTP errors still return `ok=False`.
+For an empty HTTP error body, Chromium can show its own error page. The returned
+text or HTML is that rendered page. The status, headers, and `final_url` still
+come from the HTTP response.
 
 For manual CAPTCHA completion, open a persistent browser with
 `webskrap browser open --headed --session manual --proxy http://localhost:8888 URL`.
@@ -64,6 +67,30 @@ Keep `profile_mode="native"` for the browser's own identity. On Linux with Xvfb,
 Full profile emulation is useful for device tests, but cannot replace every
 worker or iframe value. See the [identity limits](src/webskrap/guide.md).
 
+## Python batches
+
+Use `client.fetch_many(urls)` to fetch a finite batch in order with one browser:
+
+```python
+async with WebSkrapClient() as client:
+    results = await client.fetch_many(
+        ["https://example.com", "https://example.org"], text_only=True
+    )
+```
+
+Each batch shares cookies and storage, then closes its browser. Separate batches
+and `client.fetch()` calls use fresh sessions. An explicit `user_data_dir` can
+still share state on disk. Empty batches do not launch a browser. The first page
+error stops the batch and raises; no partial list is returned. Ordinary HTTP
+errors remain results with `ok=False`. `screenshot=True` uses a separate generated
+PNG filename for each page.
+
+For large inputs, per-page options, or state across batches, create a session
+with `session = await client.session("crawl")`. Call `await session.fetch(url)`
+for each URL. Close it with `await client.close_session("crawl")`, or let the
+client context close it. Session names belong to one client. Use separate
+sessions for unrelated callers.
+
 ## Search
 
 A search loads the engine's results page in the same stealth browser and
@@ -109,6 +136,19 @@ Coordinates are CSS pixels from the original image's top-left. Take a fresh
 view after scrolling or page changes. Headless Chromium needs no Xvfb for
 this; the agent's client and model must support image input.
 
+## Load only what the task needs
+The default resource policy is `auto`. Text-only fetches skip images, fonts
+and media. HTML fetches, screenshots and interactive pages load all resources.
+Set `resource_policy="all"`, `"lite"`, or `"documents"` to override this choice.
+Use `documents` only when removing CSS preserves the required result.
+Fetches use `domcontentloaded`; set `ready_selector` for deferred content.
+Use `browser_find(text="Save")` to locate a known control with ancestor context.
+Use `browser_snapshot(target="e15", depth=2)` for one subtree; add `boxes=true`
+only when geometry matters. Reuse returned refs with `browser_interact`.
+Use `networkidle`, screenshots and full snapshots only when the task needs them.
+CLI and MCP reject cookies by default. Disable this only for known pages where
+it is unnecessary. In Python, enable rejection with `SessionConfig(decline_cookies=True)`.
+
 ## Documentation
 
 - [Agent skill](.agents/skills/webskrap/SKILL.md)

webskrap-3.3.0/docs/benchmarks/mcp-snapshots-playwright.json

@@ -0,0 +1,83 @@
+{
+  "package": "@playwright/mcp@0.0.85",
+  "runs": [
+    {
+      "mode": "full",
+      "repeat": 0,
+      "success": true,
+      "tool_calls": 6,
+      "request_tokens": 91,
+      "response_tokens": 479,
+      "response_bytes": 1508,
+      "snapshot_ms_median": 74.09905100939795,
+      "task_ms": 2093.8407619833015,
+      "cpu_seconds": 0.6999999999999993,
+      "rss_mib_after": 1396.22265625
+    },
+    {
+      "mode": "full",
+      "repeat": 1,
+      "success": true,
+      "tool_calls": 6,
+      "request_tokens": 91,
+      "response_tokens": 479,
+      "response_bytes": 1508,
+      "snapshot_ms_median": 109.93575002066791,
+      "task_ms": 2686.555687978398,
+      "cpu_seconds": 0.7799999999999994,
+      "rss_mib_after": 1335.69921875
+    },
+    {
+      "mode": "full",
+      "repeat": 2,
+      "success": true,
+      "tool_calls": 6,
+      "request_tokens": 91,
+      "response_tokens": 479,
+      "response_bytes": 1508,
+      "snapshot_ms_median": 90.022451011464,
+      "task_ms": 2248.4715349855833,
+      "cpu_seconds": 0.7300000000000004,
+      "rss_mib_after": 1335.19140625
+    },
+    {
+      "mode": "none",
+      "repeat": 0,
+      "success": true,
+      "tool_calls": 6,
+      "request_tokens": 91,
+      "response_tokens": 351,
+      "response_bytes": 1162,
+      "snapshot_ms_median": 130.02865097951144,
+      "task_ms": 2340.031432977412,
+      "cpu_seconds": 0.7799999999999998,
+      "rss_mib_after": 1314.16796875
+    },
+    {
+      "mode": "none",
+      "repeat": 1,
+      "success": true,
+      "tool_calls": 6,
+      "request_tokens": 91,
+      "response_tokens": 351,
+      "response_bytes": 1162,
+      "snapshot_ms_median": 77.6378289447166,
+      "task_ms": 1987.0279059978202,
+      "cpu_seconds": 0.7399999999999998,
+      "rss_mib_after": 1312.30859375
+    },
+    {
+      "mode": "none",
+      "repeat": 2,
+      "success": true,
+      "tool_calls": 6,
+      "request_tokens": 91,
+      "response_tokens": 351,
+      "response_bytes": 1162,
+      "snapshot_ms_median": 73.93148995470256,
+      "task_ms": 2026.7483039642684,
+      "cpu_seconds": 0.75,
+      "rss_mib_after": 1314.65625
+    }
+  ]
+}

webskrap-3.3.0/docs/benchmarks/mcp-snapshots-webskrap.json

@@ -0,0 +1,140 @@
+{
+  "tokenizer": "tiktoken 0.14.0 o200k_base",
+  "runs": [
+    {
+      "mode": "baseline",
+      "repeat": 0,
+      "success": true,
+      "tool_calls": 15,
+      "request_tokens": 245,
+      "response_tokens": 32636,
+      "response_bytes": 101972,
+      "snapshot_ms_median": 276.9699060008861,
+      "task_ms": 15578.733453992754,
+      "cpu_seconds": 4.199999999999999,
+      "rss_mib_after": 1158.53515625,
+      "describe_calls": 8688,
+      "stealth_sensitive_calls": 0
+    },
+    {
+      "mode": "snapshot",
+      "repeat": 0,
+      "success": true,
+      "tool_calls": 9,
+      "request_tokens": 161,
+      "response_tokens": 11054,
+      "response_bytes": 34568,
+      "snapshot_ms_median": 132.4927629902959,
+      "task_ms": 13070.251779980026,
+      "cpu_seconds": 2.4899999999999998,
+      "rss_mib_after": 1248.1875,
+      "describe_calls": 2894,
+      "stealth_sensitive_calls": 0
+    },
+    {
+      "mode": "find",
+      "repeat": 0,
+      "success": true,
+      "tool_calls": 6,
+      "request_tokens": 123,
+      "response_tokens": 302,
+      "response_bytes": 974,
+      "snapshot_ms_median": 10.353491990827024,
+      "task_ms": 12541.241928993259,
+      "cpu_seconds": 2.02,
+      "rss_mib_after": 1241.4296875,
+      "describe_calls": 7,
+      "stealth_sensitive_calls": 0
+    },
+    {
+      "mode": "baseline",
+      "repeat": 1,
+      "success": true,
+      "tool_calls": 15,
+      "request_tokens": 245,
+      "response_tokens": 32636,
+      "response_bytes": 101972,
+      "snapshot_ms_median": 189.5280875323806,
+      "task_ms": 14903.852140996605,
+      "cpu_seconds": 4.0600000000000005,
+      "rss_mib_after": 1160.078125,
+      "describe_calls": 8688,
+      "stealth_sensitive_calls": 0
+    },
+    {
+      "mode": "snapshot",
+      "repeat": 1,
+      "success": true,
+      "tool_calls": 9,
+      "request_tokens": 161,
+      "response_tokens": 11054,
+      "response_bytes": 34568,
+      "snapshot_ms_median": 135.92701949528418,
+      "task_ms": 13055.62697298592,
+      "cpu_seconds": 2.8599999999999994,
+      "rss_mib_after": 1247.55078125,
+      "describe_calls": 2894,
+      "stealth_sensitive_calls": 0
+    },
+    {
+      "mode": "find",
+      "repeat": 1,
+      "success": true,
+      "tool_calls": 6,
+      "request_tokens": 123,
+      "response_tokens": 302,
+      "response_bytes": 974,
+      "snapshot_ms_median": 15.616477990988642,
+      "task_ms": 12361.839258985128,
+      "cpu_seconds": 1.96,
+      "rss_mib_after": 1244.92578125,
+      "describe_calls": 7,
+      "stealth_sensitive_calls": 0
+    },
+    {
+      "mode": "baseline",
+      "repeat": 2,
+      "success": true,
+      "tool_calls": 15,
+      "request_tokens": 245,
+      "response_tokens": 32636,
+      "response_bytes": 101972,
+      "snapshot_ms_median": 151.91562002291903,
+      "task_ms": 13915.481942996848,
+      "cpu_seconds": 4.12,
+      "rss_mib_after": 1257.5,
+      "describe_calls": 8688,
+      "stealth_sensitive_calls": 0
+    },
+    {
+      "mode": "snapshot",
+      "repeat": 2,
+      "success": true,
+      "tool_calls": 9,
+      "request_tokens": 161,
+      "response_tokens": 11054,
+      "response_bytes": 34568,
+      "snapshot_ms_median": 131.0603755118791,
+      "task_ms": 12102.267072012182,
+      "cpu_seconds": 2.6100000000000003,
+      "rss_mib_after": 1245.76171875,
+      "describe_calls": 2894,
+      "stealth_sensitive_calls": 0
+    },
+    {
+      "mode": "find",
+      "repeat": 2,
+      "success": true,
+      "tool_calls": 6,
+      "request_tokens": 123,
+      "response_tokens": 302,
+      "response_bytes": 974,
+      "snapshot_ms_median": 11.524417961481959,
+      "task_ms": 11286.688952008262,
+      "cpu_seconds": 1.9299999999999997,
+      "rss_mib_after": 1244.19140625,
+      "describe_calls": 7,
+      "stealth_sensitive_calls": 0
+    }
+  ]
+}

webskrap-3.3.0/docs/benchmarks/mcp-snapshots.md

@@ -0,0 +1,145 @@
+# MCP snapshot workflow measurements
+
+Measured on 2026-11-14 on Linux ARM64, Python 3.14.7, uv 0.13.5 and
+Chrome for Testing 156.0.8078.47, headless with sandbox explicitly disabled.
+Baseline revision: `914e24576ebe92e08d46c2867d065343b1b71fdc`.
+The measurements precede the review fixes for inline-text search, URL escaping
+and stacked dialogs. Those fixes have separate regression coverage; the
+benchmark has not been rerun for them.
+No production dependency was added.
+
+## Method
+
+The task page has 180 records, headings, descriptions and edit buttons. The
+workflow locates record 179, opens its native modal editor, locates and fills
+the Title field, locates Save changes, clicks it and checks the saved value.
+This measures a three-interaction workflow, including DOM changes and modal
+state, rather than an isolated snapshot. It is one synthetic task, not a
+representative sample of the web or an actual model inference run.
+
+Each WebSkrap mode runs three times in alternating order with a fresh browser.
+Baseline loads the original CDP module from the revision above. Snapshot mode
+uses the new code with default depth and character limits. Both page through
+8,000-character responses until the named control is present. Find mode uses
+literal name search. All modes use the same native WebSkrap interactions.
+
+The Playwright comparison uses `@playwright/mcp@0.0.85` in npm's external cache,
+the same Chromium executable and HTML, and the same names and actions. It
+runs three times each with default automatic snapshots and with
+`--snapshot-mode=none`. Both use `browser_find`. Playwright can save large snapshots to local files;
+these counts cover returned text, not unrequested file contents. The task needs
+only find results, so it does not read those files. Playwright loads a temporary
+file; WebSkrap injects the document into about:blank. Loading and browser
+startup are excluded. Verification reads are excluded from tool/token counts.
+
+Token counts use tiktoken 0.14.0, `o200k_base`. Request counts encode JSON tool
+names and arguments. Response counts encode each WebSkrap JSON payload once,
+or Playwright's returned text content once. These are measured payload tokens,
+not billed model input/output usage. They exclude JSON-RPC envelopes, duplicate
+MCP text/structured serialization, tool schemas, model reasoning and replayed
+conversation context. A client that sends both structured and text payloads to
+a model can consume more tokens. No model was invoked to select actions.
+
+Snapshot latency covers each read plus shaping; the reported number is the
+median of each run's median. WebSkrap calls Python handlers directly;
+Playwright uses MCP stdio, so cross-driver latency includes different transport
+costs. Task time includes trusted interactions and verification. WebSkrap's
+human cursor, scroll and typing timing differs from Playwright's interaction
+implementation. Compare read latency separately from total task time.
+
+CPU is the `/proc` CPU-time delta for Python plus its current descendants.
+Memory is their summed RSS after the task, not peak memory or unique physical
+memory; shared pages can be counted more than once. `/proc` sampling and
+regex-worker lifecycle are approximate. Runs shared the host with tests.
+Timing and memory are observations, not isolated performance guarantees.
+
+## Results
+
+All rows are medians over three successful runs. Each driver completed 3/3
+workflows per mode with the expected saved value.
+
+| Mode | Request tokens | Response tokens | Response bytes | Tool calls | Read ms | Task ms | CPU s | Summed RSS MiB |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| WebSkrap baseline, paged snapshot | 245 | 32,636 | 101,972 | 15 | 189.53 | 14,903.85 | 4.12 | 1,160.08 |
+| WebSkrap new, paged snapshot | 161 | 11,054 | 34,568 | 9 | 132.49 | 13,055.63 | 2.61 | 1,247.55 |
+| WebSkrap new, targeted find | 123 | 302 | 974 | 6 | 11.52 | 12,361.84 | 1.96 | 1,244.19 |
+| Playwright MCP, find + default snapshots | 91 | 479 | 1,508 | 6 | 90.02 | 2,248.47 | 0.73 | 1,335.70 |
+| Playwright MCP, find + no automatic snapshots | 91 | 351 | 1,162 | 6 | 77.64 | 2,026.75 | 0.75 | 1,314.17 |
+
+The new snapshot path saves tokens in this task mainly because it omits
+background content behind the modal. It preserves the existing format for
+ordinary nonmodal pages. Targeted find reduces response tokens by 99.1% against
+the baseline workflow and uses six calls instead of fifteen. This reduction is
+specific to a known target near the end of a long page. Small pages or tasks
+that need broad context will yield different results.
+
+Targeted find makes seven `DOM.describeNode` calls per task, compared with
+8,688 in the baseline and 2,894 for the new paged snapshots. All WebSkrap modes
+made zero `Runtime.enable`, `Console.enable`, `Accessibility.enable` or
+`Accessibility.getFullAXTree` calls during the measured task. This is a CDP
+regression indicator, not evidence that a protected website accepts the browser.
+No new public anti-bot acceptance test was run.
+
+Playwright completes trusted interactions faster in this task. WebSkrap's
+read-only find is faster in these samples, but its total task is slower.
+Summed RSS is higher than the old baseline in these samples; this method does
+not establish a memory improvement. Literal find avoids regex subprocesses;
+caller-supplied regex still pays worker startup and timeout protection costs.
+
+The server instruction text is 656 tokens versus 658 before this change.
+The new find tool definition is 425 tokens when encoded as its full MCP tool
+record. A client that includes all tool schemas in each model prompt pays
+that extra context cost, which is excluded from the task table.
+
+## Reproduce
+
+1. Run `git show 914e24576ebe92e08d46c2867d065343b1b71fdc:src/webskrap/cdp_dom.py > /tmp/webskrap-snapshot-baseline.py`.
+2. Run `uv run --with tiktoken==0.14.0 python scripts/benchmark_snapshots.py --baseline /tmp/webskrap-snapshot-baseline.py`.
+3. For the optional external comparison, run `uv run --with tiktoken==0.14.0 python scripts/benchmark_playwright_snapshots.py`.
+
+The first script accepts `--repeats`. The second requires npx and installs only
+into its external cache. Neither adds a WebSkrap runtime dependency. Both
+explicitly disable Chromium's sandbox for this host. Use a sandbox on hosts
+that support it. Load only a trusted baseline source file.
+
+Raw measurements: [WebSkrap](mcp-snapshots-webskrap.json) and
+[Playwright MCP](mcp-snapshots-playwright.json). Implementation decisions and
+semantic limits are in [the design note](../design/mcp-snapshots.md).
+
+## Validation and review
+
+The complete local non-live suite passed: 966 tests, 30 live tests deselected,
+88.20% coverage. Review then found that the existing frame-navigation error
+handler could swallow an iframe regex-worker timeout. The handler now propagates
+search usage/internal errors. The final focused run passed 47 tests with appended
+coverage of 88.34%, including an independent regression test for that error path.
+
+Final checks passed: Ruff lint and format, Pyright 1.1.411 for Linux, Windows
+and Darwin, Bandit, source/wheel build and Twine metadata validation. The review
+was performed locally. Public live tests and Windows/macOS runtime tests were
+not run on this Linux host. Cross-platform typing does not replace those tests.
+
+The lockfile pins `opentelemetry-api==1.45.2`, which fixes
+[GHSA-mf5v-w3w3-f6cg](https://osv.dev/vulnerability/GHSA-mf5v-w3w3-f6cg)
+(CVE-2026-126360, traceparent parsing denial of service). The previous locked
+version, 1.45.1, failed the CI audit. Only that package's version and artifact
+metadata changed. Both pip-audit and the OSV scan of 51 locked public packages
+now report no known vulnerabilities. No new dependency was added.
+
+After the audit repair, the full local non-live suite passed 967 tests with
+89.15% coverage. Windows CI also exposed a fixture race: challenge detection
+can return before an iframe's image request reaches the proxy. The local test
+now defers that request and waits for its actual arrival with a five-second
+deadline. The old immediate assertion failed against the delayed fixture;
+all six transition/authentication cases pass with the bounded wait. The final
+combined frame/snapshot run passed 30 tests. Production challenge handling is
+unchanged.
+
+Review follow-up validation: the full non-live run passed 980 tests with 89.15%
+coverage. The final focused run passed 31 tests, including additional Unicode
+separator cases. The broader session/MCP regression run passed 81 tests. New
+regressions cover inline paragraph/status/value searches, ref-like visible text,
+URL and ARIA line injection, and stacked modal opening/closing order. Ruff,
+three-platform Pyright, Bandit, pip-audit, OSV, build and Twine passed. Public
+live tests were excluded; local cross-platform runtime tests remain delegated
+to CI. No benchmark numbers were regenerated for this follow-up.

webskrap-3.3.0/docs/design/mcp-snapshots.md

@@ -0,0 +1,92 @@
+# Targeted MCP accessibility reads
+
+WebSkrap keeps its native CDP DOM walker. It does not enable Chromium's
+Accessibility domain or Runtime event domain to generate snapshots.
+
+## Existing behavior and changes
+
+The starting implementation already had implicit ARIA roles, common accessible
+name rules, states, wrapper elision, text merging, depth limits, open shadow
+roots, iframe traversal, backend-node refs and character pagination. Actions
+already returned page state without an automatic snapshot.
+
+[Playwright MCP snapshots](https://playwright.dev/mcp/snapshots) describes
+subtree targets, depth limits, optional boxes and matching nodes under ancestor
+paths. WebSkrap adopts these features while keeping its selector and ref
+contracts. Playwright locator expressions are not accepted. A CSS selector
+must uniquely match an element in the main document. A frame ref selects that
+frame; an element ref also works inside an open shadow root.
+
+`browser_snapshot` adds optional `target` and `boxes` arguments after the
+existing positional arguments. Its response fields remain unchanged. Depth is
+relative to the selected root. Zero and null retain the unlimited-depth
+meaning. Negative depth returns a usage error. Boxes are rounded CSS pixels
+relative to the top-level viewport, including iframe offsets.
+
+`browser_find` returns the same snapshot-window fields, plus match counts.
+It searches semantic lines, including text and non-password values appended
+after refs, and retains their ancestor paths. Only structural ref and box
+tokens are removed for matching; ref-like text in names or content is kept.
+It omits sibling subtrees and link URL metadata. Search takes either case-insensitive literal
+text or Python regex. Literal text is escaped before compilation. Caller regex
+uses the existing subprocess worker and two-second timeout. Regex never runs
+in the page. Search has a match cap and independent character pagination.
+
+## Ref and response cost
+
+The DOM walk runs in an isolated world and returns semantic text plus remote
+node objects. Search filters the text before issuing `DOM.describeNode`.
+Only matching nodes, their ancestors and iframe owners need backend IDs.
+Iframe owners remain available while child frames are searched. A final pass
+removes nonmatching frame paths and applies the global match limit.
+
+The walker still builds each available frame's semantic tree. This avoids a
+second naming implementation and keeps semantics consistent with snapshots.
+It does not make search traversal proportional only to the number of matches.
+There is no persistent tree cache. Remote object groups are released on errors
+and successful reads. Ref replacement uses one substitution pass. Per-call
+random placeholders prevent page text that resembles a placeholder from being
+used as a remote-object index.
+
+Refs keep Chromium's backend-node identity. Repeated snapshots, find calls and
+SPA updates preserve refs when they preserve the underlying node. A detached
+node or a changed document returns `stale_ref`. There is no snapshot-generation
+counter and no model-specific ref remapping. Existing interaction mechanisms
+remain in use.
+
+Responses do not repeat snapshots after actions. A caller can find, interact,
+wait for a condition and request only the next needed control. A diff cache
+was not added: this change measures explicit targeted reads, without adding a
+second state lifetime or requiring clients to reconstruct trees.
+
+## Semantics and limits
+
+Link URL metadata uses quoted JSON strings, including escaped line separators.
+A URL cannot add structural lines or refs to the snapshot. Search still omits
+URL metadata. Dynamic ARIA state text is normalized to one line.
+
+With stacked native modal dialogs, snapshots select the modal containing
+`document.activeElement`. If focus does not identify a modal, the last modal in
+document order is used. That fallback is an approximation of top-layer order.
+Scope checks apply to the selected modal, and closing it restores the lower
+modal's context.
+
+The shared walker now also omits password input values, inert content, closed
+details content and background content behind native modal dialogs. It resolves
+`aria-labelledby` in the element's document or shadow root. Disabled fieldsets,
+expanded=false, mixed pressed state, required, readonly, invalid and busy state
+are represented. Existing compact role/name and common naming rules remain;
+this is not a complete implementation of the browser accessibility algorithm.
+
+Closed shadow roots and nested frames inside out-of-process iframes retain the
+driver's existing limits. Frame navigation during traversal can omit that frame.
+A snapshot is not an atomic DOM transaction. Continuation offsets are valid
+only while the DOM remains unchanged. A character window can split a line or
+ref. Match caps precede character caps. There are no sibling gap markers;
+ancestor paths are intentionally partial trees.
+
+Password redaction does not remove secrets from visible text, labels, URLs or
+other field values. Snapshot and find output must receive the same privacy
+handling as other browser reads.
+
+See [measured workflows](../benchmarks/mcp-snapshots.md) for results and limits.

{webskrap-3.2.4 → webskrap-3.3.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "webskrap"
-version = "3.2.4"
+version = "3.3.0"
 description = "Async Python scraping on Chromium over its own CDP client, with coherent browser profiles and session controls."
 readme = "README.md"
 requires-python = ">=3.11"