webskrap 2.2.0__tar.gz → 2.3.0__tar.gz

{webskrap-2.2.0 → webskrap-2.3.0}/CHANGELOG.md

@@ -9,6 +9,30 @@ history, so they summarize each release rather than list every change.
 
 ## [Unreleased]
 
+## [2.3.0] - 2026-09-18
+
+### Security
+
+- One-shot `fetch`/`search` calls now keep Chromium's OS sandbox by default,
+  matching persistent sessions. Opt out per call with `webskrap fetch
+  --no-sandbox`, `webskrap search --no-sandbox`, or
+  `SessionConfig(chromium_sandbox=False)`, or per host with
+  `WEBSKRAP_CHROMIUM_SANDBOX=0`. Sandbox-weakening flags (`--no-sandbox`,
+  `--disable-setuid-sandbox`, `--load-extension`, `--unsafely-*`,
+  `--remote-debugging-*`) are rejected when passed through `--launch-arg`.
+- Fetch and navigation targets must be `http`/`https` URLs without embedded
+  credentials (`data:` pages and `about:blank` stay allowed). The MCP server
+  additionally refuses hosts resolving to non-public addresses unless
+  `WEBSKRAP_ALLOW_PRIVATE_NET=1` is set.
+- MCP `browser_eval` caps expressions at 10,000 characters, logs them
+  server-side, and can be disabled with `WEBSKRAP_ALLOW_EVAL=0`.
+- Proxy credentials are redacted from `ProxyConfig` display output, embedded
+  URL credentials are scrubbed from error text, and shaped CLI/MCP fetch
+  payloads return an allowlist of response headers (no `set-cookie`).
+- MCP output and profile directories are created level-by-level `0700`,
+  symlinks along the way are rejected, and destinations are re-resolved
+  before use to narrow the resolve-to-write race.
+
 ## [2.2.0] - 2026-09-15
 
 ### Added

{webskrap-2.2.0 → webskrap-2.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: webskrap
-Version: 2.2.0
+Version: 2.3.0
 Summary: A Playwright-based Python scraping framework with coherent browser profiles and session controls.
 Project-URL: Homepage, https://github.com/kacigaya/webskrap
 Project-URL: Repository, https://github.com/kacigaya/webskrap

{webskrap-2.2.0 → webskrap-2.3.0}/SECURITY.md

@@ -46,7 +46,24 @@ into a shell, a template, a database query, or an LLM prompt without treating
 it as attacker-controlled.
 
 `browser_eval` and `webskrap browser eval` evaluate JavaScript in the page.
-Whatever they return crosses from the page into your program.
+Whatever they return crosses from the page into your program. Prefer
+snapshots, interaction, and waits over eval, and never evaluate text copied
+from a page. Over MCP, expressions are capped at 10,000 characters, logged
+server-side, and disabled entirely with `WEBSKRAP_ALLOW_EVAL=0`.
+
+### Fetch targets
+
+One-shot and persistent navigation accept `http`/`https` URLs (`data:` pages
+and `about:blank` carry no network request and stay allowed). Embedded
+`user:pass@` credentials are rejected everywhere so they never reach logs or
+state files.
+
+The MCP server additionally refuses hosts that resolve to non-public
+addresses (loopback, RFC1918, link-local, and friends), since its URLs arrive
+from a model that reads untrusted pages. Set `WEBSKRAP_ALLOW_PRIVATE_NET=1`
+to permit them. The Python API and CLI take URLs from the operator and skip
+the DNS check; point them at internal hosts deliberately, not with
+page-derived input.
 
 ### Chromium sandbox
 
@@ -60,9 +77,12 @@ enable unprivileged user namespaces, or run the browser as a non-root user
 with the sandbox intact. WebSkrap never drops the sandbox on its own, and
 never retries a failed launch without it.
 
-Note that one-shot `fetch` calls go through Playwright's own launcher, which
-defaults to `chromium_sandbox=False`. Pass `launch_args` or run under a
-hardened container if that matters for your use.
+One-shot `fetch`/`search` calls keep the sandbox by default, the same as
+persistent sessions. Opt out per call with `webskrap fetch --no-sandbox` /
+`webskrap search --no-sandbox` or `chromium_sandbox=False`, or per host with
+`WEBSKRAP_CHROMIUM_SANDBOX=0`. Sandbox-weakening flags are rejected when
+passed through `--launch-arg`: use the explicit switch so the choice stays
+visible.
 
 ### Persistent session state
 
@@ -93,10 +113,12 @@ and do not set it to a source tree, a config directory, or `$HOME`.
 Confinement is checked when the path is resolved, and the browser writes the
 file a moment later. On a host where another local account can write inside the
 output root, that gap is a race: a directory component could be replaced with a
-symlink in between. WebSkrap creates its own default root `0700` so no other
-account can plant anything there, which is as far as a local Python library can
-reasonably close it. A root you point at a world-writable directory is outside
-that guarantee.
+symlink in between. WebSkrap narrows it by creating intermediate directories
+`0700` itself, refusing to walk through symlinks, and re-resolving the
+destination immediately before returning it -- and it creates its own default
+root `0700` so no other account can plant anything there, which is as far as a
+local Python library can reasonably close it. A root you point at a
+world-writable directory is outside that guarantee.
 
 `stealth_fetch` applies the same trust-boundary rule to persistent browser
 profiles. Its `user_data_dir` is relative to `~/.webskrap/profiles` by default;
@@ -108,8 +130,11 @@ accept caller-chosen profile paths.
 ### Proxy credentials
 
 `ProxyConfig` holds proxy usernames and passwords in memory and passes them to
-Playwright. WebSkrap never writes them to disk, but they do appear in the
-model's `repr` and `model_dump`, so keep them out of logs and error reports.
+Playwright. WebSkrap never writes them to disk. Its `repr`/`str` redact them,
+error text scrubs embedded `user:pass@` authorities before it reaches a tool
+result or terminal, and shaped CLI/MCP fetch payloads carry an allowlist of
+response headers (content metadata, never `set-cookie` or credentials), so
+keep the raw `FetchResult` out of logs regardless.
 
 ### What is out of scope
 

{webskrap-2.2.0 → webskrap-2.3.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "webskrap"
-version = "2.2.0"
+version = "2.3.0"
 description = "A Playwright-based Python scraping framework with coherent browser profiles and session controls."
 readme = "README.md"
 requires-python = ">=3.11"

{webskrap-2.2.0 → webskrap-2.3.0}/references/cli.md

@@ -64,9 +64,11 @@ Sessions default to the name `default`. Profiles live below
 They contain cookies and logins and must remain mode `0700`.
 
 Keep Chromium's OS sandbox enabled. The explicit opt-outs are
-`webskrap browser open --no-sandbox` for one session and
-`WEBSKRAP_CHROMIUM_SANDBOX=0` for a host. Document that disabling it leaves a
-compromised renderer uncontained.
+`webskrap browser open --no-sandbox` for one session,
+`webskrap fetch --no-sandbox` / `webskrap search --no-sandbox` for one-shot
+calls, and `WEBSKRAP_CHROMIUM_SANDBOX=0` for a host. Document that disabling it
+leaves a compromised renderer uncontained. Never pass `--no-sandbox` (or its
+companions) through `--launch-arg`: those flags are blocked.
 
 Current limitations are one page per session, bundled Chromium only, no tabs,
 network mocking, tracing, or video, and history navigation that reloads.

{webskrap-2.2.0 → webskrap-2.3.0}/references/mcp.md

@@ -51,6 +51,13 @@ symlinks that leave it.
 The sandbox opt-out is an environment variable, never an MCP argument. Page
 content must not be able to persuade the client to disable the browser sandbox.
 
+Fetch and navigation URLs must be public `http(s)` without credentials.
+Private or local hosts are rejected unless `WEBSKRAP_ALLOW_PRIVATE_NET=1`.
+
+`browser_eval` runs page script in a possibly logged-in profile: prefer
+snapshot, interact, and wait_for, never evaluate page-controlled text, and set
+`WEBSKRAP_ALLOW_EVAL=0` where models must not run page script.
+
 ## Failures
 
 A fetch with an HTTP error status returns normally with `ok: false`. A raised

{webskrap-2.2.0 → webskrap-2.3.0}/src/webskrap/browser_session.py

@@ -14,6 +14,7 @@ from __future__ import annotations
 import asyncio
 import errno
 import json
+import logging
 import os
 import re
 import shutil
@@ -32,6 +33,9 @@ from playwright.async_api import Locator, Page, async_playwright
 from webskrap.errors import ErrorCode, WebSkrapError
 from webskrap.models import ElementState, LoadState, WaitUntil, text_window
 from webskrap.paths import secure_directory
+from webskrap.urls import validate_url
+
+logger = logging.getLogger(__name__)
 
 T = TypeVar("T")
 
@@ -145,7 +149,7 @@ def sessions_root() -> Path:
     ``WEBSKRAP_BROWSER_DIR`` overrides the default ``~/.webskrap/browser``.
     """
     if override := os.environ.get("WEBSKRAP_BROWSER_DIR"):
-        return Path(override)
+        return Path(override).expanduser()
     return Path.home() / ".webskrap" / "browser"
 
 
@@ -252,6 +256,8 @@ def session_running(directory: Path, state: dict[str, Any] | None) -> bool:
 
 def is_session_dir(path: Path) -> bool:
     """True when ``path`` looks like a session directory this tool manages."""
+    if path.is_symlink():
+        return False
     return (
         path.is_dir()
         and SESSION_NAME_PATTERN.fullmatch(path.name) is not None
@@ -484,7 +490,10 @@ def close_session(name: str, *, delete_data: bool = False) -> dict[str, Any]:
             _terminate(directory, state["pid"])
         state_path(directory).unlink(missing_ok=True)
         if delete_data:
-            shutil.rmtree(directory, ignore_errors=True)
+            try:
+                shutil.rmtree(directory, ignore_errors=False)
+            except OSError:
+                logger.debug("could not remove session directory %s", directory)
     finally:
         operation_lock.release()
     return {"session": name, "deleted_data": delete_data}
@@ -625,6 +634,7 @@ async def page_state(page: Page) -> dict[str, Any]:
 
 async def goto(page: Page, url: str, wait_until: WaitUntil) -> dict[str, Any]:
     """Navigate ``page`` to ``url`` and return its status and state."""
+    url = validate_url(url)
     response = await page.goto(url, wait_until=wait_until)
     return {"status": response.status if response else None, **await page_state(page)}
 

{webskrap-2.2.0 → webskrap-2.3.0}/src/webskrap/cli.py

@@ -22,6 +22,7 @@ from rich.markup import escape
 from rich.table import Table
 
 from webskrap.browser_cli import browser_app
+from webskrap.browser_session import sandbox_enabled
 from webskrap.cli_output import (
     OutputFormat,
     fail,
@@ -300,6 +301,13 @@ def fetch_command(
             help="Additional browser launch argument. Repeat for multiple args.",
         ),
     ] = None,
+    no_sandbox: Annotated[
+        bool,
+        typer.Option(
+            "--no-sandbox",
+            help="Disable Chromium's OS sandbox (weakens isolation; only where it cannot start).",
+        ),
+    ] = False,
     webrtc_ip_handling_policy: Annotated[
         str | None,
         typer.Option(
@@ -337,6 +345,7 @@ def fetch_command(
             reduce_fingerprint_surface=reduce_fingerprint_surface,
             mask_headless_user_agent=mask_headless_user_agent,
             launch_args=launch_args or [],
+            no_sandbox=no_sandbox,
             webrtc_ip_handling_policy=webrtc_ip_handling_policy,
         )
     )
@@ -367,6 +376,7 @@ async def _fetch(
     reduce_fingerprint_surface: bool,
     mask_headless_user_agent: bool,
     launch_args: list[str],
+    no_sandbox: bool,
     webrtc_ip_handling_policy: str | None,
 ) -> None:
     parsed_output_format = parse_output_format(output_format)
@@ -375,6 +385,7 @@ async def _fetch(
         driver="patchright",
         headless=True,
         channel=channel,
+        chromium_sandbox=sandbox_enabled(False if no_sandbox else None),
         user_data_dir=user_data_dir,
         navigation_timeout_ms=timeout_ms,
         resource_policy=resource_policy,
@@ -519,6 +530,13 @@ def search_command(
             help="Additional browser launch argument. Repeat for multiple args.",
         ),
     ] = None,
+    no_sandbox: Annotated[
+        bool,
+        typer.Option(
+            "--no-sandbox",
+            help="Disable Chromium's OS sandbox (weakens isolation; only where it cannot start).",
+        ),
+    ] = False,
     webrtc_ip_handling_policy: Annotated[
         str | None,
         typer.Option(
@@ -554,6 +572,7 @@ def search_command(
             reduce_fingerprint_surface=reduce_fingerprint_surface,
             mask_headless_user_agent=mask_headless_user_agent,
             launch_args=launch_args or [],
+            no_sandbox=no_sandbox,
             webrtc_ip_handling_policy=webrtc_ip_handling_policy,
         )
     )
@@ -576,6 +595,7 @@ async def _search(
     reduce_fingerprint_surface: bool,
     mask_headless_user_agent: bool,
     launch_args: list[str],
+    no_sandbox: bool,
     webrtc_ip_handling_policy: str | None,
 ) -> None:
     parsed_output_format = parse_output_format(output_format)
@@ -583,6 +603,7 @@ async def _search(
         driver="patchright",
         headless=True,
         channel=channel,
+        chromium_sandbox=sandbox_enabled(False if no_sandbox else None),
         user_data_dir=user_data_dir,
         navigation_timeout_ms=timeout_ms,
         resource_policy=resource_policy,

{webskrap-2.2.0 → webskrap-2.3.0}/src/webskrap/client.py

@@ -10,6 +10,7 @@ own errors (timeouts, navigation failures) propagate unchanged.
 from __future__ import annotations
 
 import asyncio
+import logging
 import shutil
 import tempfile
 import time
@@ -37,6 +38,9 @@ from webskrap.models import (
 )
 from webskrap.profiles import get_profile
 from webskrap.search import parse_results, search_url
+from webskrap.urls import validate_url
+
+logger = logging.getLogger(__name__)
 
 # Cursor jitter below is pixel offsets and sleep durations, never a token,
 # identifier, or security decision, so `random` would be adequate. It draws
@@ -228,6 +232,7 @@ class WebSkrapSession:
             WebSkrapError: If the session is closed.
         """
         self._ensure_open()
+        url = validate_url(url)
         started = time.perf_counter()
         page = await self.context.new_page()
         try:
@@ -444,7 +449,10 @@ class WebSkrapSession:
                     await self.browser.close()
             finally:
                 if self._temp_user_data_dir is not None:
-                    shutil.rmtree(self._temp_user_data_dir, ignore_errors=True)
+                    try:
+                        shutil.rmtree(self._temp_user_data_dir, ignore_errors=False)
+                    except OSError:
+                        logger.debug("could not remove temp profile %s", self._temp_user_data_dir)
                     self._temp_user_data_dir = None
                 self._closed = True
 
@@ -806,7 +814,10 @@ class WebSkrapClient:
                 with suppress(Exception):
                     await browser.close()
             if temp_user_data_dir is not None:
-                shutil.rmtree(temp_user_data_dir, ignore_errors=True)
+                try:
+                    await asyncio.to_thread(shutil.rmtree, temp_user_data_dir)
+                except OSError:
+                    logger.debug("could not remove temp profile %s", temp_user_data_dir)
             raise
         return WebSkrapSession(
             name=name,

{webskrap-2.2.0 → webskrap-2.3.0}/src/webskrap/diagnostics.py

@@ -66,5 +66,9 @@ async def diagnose() -> dict[str, Any]:
         # default" from "explicitly set to the default".
         "environment": {name: os.environ.get(name) for name in ENVIRONMENT_VARIABLES},
         "chromium_sandbox": browser_session.sandbox_enabled(),
+        # One-shot fetch/search resolve the same default through their own
+        # config: sandboxed unless the operator opted out above.
+        "chromium_sandbox_one_shot": browser_session.sandbox_enabled(None),
+        "sessions_root_symlink": browser_session.sessions_root().is_symlink(),
         "sessions": browser_session.list_sessions(),
     }

{webskrap-2.2.0 → webskrap-2.3.0}/src/webskrap/errors.py

@@ -13,6 +13,7 @@ keeping one taxonomy for WebSkrap and Playwright failures alike.
 
 from __future__ import annotations
 
+import re
 from enum import StrEnum
 
 
@@ -101,10 +102,13 @@ EXIT_CODES: dict[ErrorCode, int] = {
 }
 
 # Lowercased substrings that identify a failure whose raise site did not set a
-# code. Order is significant: the sandbox hint is appended to a launch failure
-# message, and a stale ref usually surfaces as a timeout on an `aria-ref=`
-# locator, so the more specific marker has to be tested first.
+# code. Order is significant: "is blocked" is tested first because a blocked
+# `--no-sandbox` launch flag would otherwise match the sandbox marker below,
+# and the actionable half is the rejection, not the sandbox. The sandbox hint
+# is appended to a launch failure message, and a stale ref usually surfaces as
+# a timeout on an `aria-ref=` locator, so those more specific markers come next.
 _MESSAGE_CODES: tuple[tuple[str, ErrorCode], ...] = (
+    ("is blocked", ErrorCode.USAGE),
     ("sandbox could not start", ErrorCode.SANDBOX),
     ("no usable sandbox", ErrorCode.SANDBOX),
     ("setuid sandbox", ErrorCode.SANDBOX),
@@ -176,14 +180,33 @@ def classify(exc: BaseException) -> ErrorCode:
     return ErrorCode.INTERNAL
 
 
+# Matches embedded credentials (scheme://user:pass@host) so error text handed
+# to models and terminals never carries them.
+_USERINFO_PATTERN = re.compile(r"(?P<scheme>[a-zA-Z][a-zA-Z0-9+.-]*://)[^/\s@]+@")
+
+
+def scrub_userinfo(text: str) -> str:
+    """Replace URL credentials in ``text`` with ``***``.
+
+    Args:
+        text: Message that may embed a ``user:pass@`` authority.
+
+    Returns:
+        The message with credentials replaced by ``***``.
+    """
+    return _USERINFO_PATTERN.sub(r"\g<scheme>***@", text)
+
+
 def first_line(exc: BaseException) -> str:
     """Return ``exc``'s first message line, or its type when it has none.
 
     Playwright errors carry a call log spanning dozens of lines. Only the first
     says what failed; the rest is noise in a tool result or a terminal.
+    Embedded URL credentials are scrubbed before returning.
     """
     lines = str(exc).strip().splitlines()
-    return lines[0].strip() if lines and lines[0].strip() else type(exc).__name__
+    first = lines[0].strip() if lines and lines[0].strip() else type(exc).__name__
+    return scrub_userinfo(first)
 
 
 def error_payload(exc: BaseException) -> dict[str, str | bool]:

{webskrap-2.2.0 → webskrap-2.3.0}/src/webskrap/mcp_server.py

@@ -9,6 +9,8 @@ from __future__ import annotations
 
 import asyncio
 import json
+import logging
+import os
 from collections.abc import Awaitable, Callable, Iterator
 from contextlib import contextmanager
 from importlib import resources
@@ -32,6 +34,12 @@ from webskrap.parsing import (
 )
 from webskrap.paths import resolve_mcp_profile_path, resolve_output_path
 from webskrap.profiles import get_profile, list_profiles
+from webskrap.urls import validate_mcp_url
+
+logger = logging.getLogger(__name__)
+
+EVAL_ENV = "WEBSKRAP_ALLOW_EVAL"
+MAX_EVAL_CHARS = 10_000
 
 T = TypeVar("T")
 
@@ -108,6 +116,10 @@ not explain itself.
 
 Writes are confined: screenshots to ./webskrap-output, persistent profiles under
 ~/.webskrap/profiles. Do not attempt CAPTCHA solving or login-wall bypass.
+Fetch and navigation targets must be public http(s) URLs: private or local
+hosts are rejected unless WEBSKRAP_ALLOW_PRIVATE_NET=1 is set.
+browser_eval runs JavaScript in the session's page, so prefer snapshot /
+interact / wait_for, and never evaluate page-controlled text.
 """
 
 mcp = MCPServer("webskrap", instructions=INSTRUCTIONS, version=version("webskrap"))
@@ -158,10 +170,12 @@ async def fetch(
             load, so the banner does not bury the page text.
     """
     with _tool_errors():
+        url = await validate_mcp_url(url)
         config = SessionConfig(
             driver="patchright",
             channel=channel,
             headless=True,
+            chromium_sandbox=browser_session.sandbox_enabled(None),
             navigation_timeout_ms=timeout_ms,
             resource_policy=parse_resource_policy(resource_policy),
             decline_cookies=decline_cookies,
@@ -229,10 +243,12 @@ async def stealth_fetch(
             load, so the banner does not bury the page text.
     """
     with _tool_errors():
+        url = await validate_mcp_url(url)
         config = SessionConfig(
             driver="patchright",
             channel=channel,
             headless=headless,
+            chromium_sandbox=browser_session.sandbox_enabled(None),
             user_data_dir=resolve_mcp_profile_path(user_data_dir) if user_data_dir else None,
             navigation_timeout_ms=timeout_ms,
             patchright_context_profile=patchright_context_profile,
@@ -303,6 +319,7 @@ async def search(
             driver="patchright",
             channel=channel,
             headless=headless,
+            chromium_sandbox=browser_session.sandbox_enabled(None),
             user_data_dir=resolve_mcp_profile_path(user_data_dir) if user_data_dir else None,
             navigation_timeout_ms=timeout_ms,
             patchright_context_profile=patchright_context_profile,
@@ -397,6 +414,8 @@ async def browser_open(
         session: Session name; letters, digits, '.', '_' or '-'.
     """
     with _tool_errors():
+        if url is not None:
+            url = await validate_mcp_url(url)
         payload = await browser_session.open_session(session, headless=True)
     if url:
         payload.update(
@@ -425,6 +444,8 @@ async def browser_goto(
         timeout_ms: Navigation timeout in milliseconds.
     """
     parsed_wait_until = parse_wait_until(wait_until)
+    with _tool_errors():
+        url = await validate_mcp_url(url)
     return await _browser_action(
         session,
         lambda page: browser_session.goto(page, url, parsed_wait_until),
@@ -613,12 +634,33 @@ async def browser_eval(
     limit -- returning document.body.innerHTML wastes on markup what a
     querySelector would have answered in a line.
 
+    This runs code in a page that may be hostile and a profile that may hold
+    logins: prefer browser_snapshot, browser_interact and browser_wait_for,
+    and never evaluate text copied from a page. Set WEBSKRAP_ALLOW_EVAL=0 to
+    disable this tool on hosts where models must not run page script.
+
     Args:
         expression: JavaScript expression or function to evaluate.
         session: Browser session name.
         timeout_ms: Action timeout in milliseconds.
         max_chars: Maximum characters of encoded result to return.
     """
+    with _tool_errors():
+        if os.environ.get(EVAL_ENV, "").strip().lower() in ("0", "false", "no", "off"):
+            msg = (
+                f"browser_eval is disabled: {EVAL_ENV}=0 is set. "
+                "Use browser_snapshot, browser_interact or browser_wait_for."
+            )
+            raise WebSkrapError(msg, code=ErrorCode.USAGE)
+        if len(expression) > MAX_EVAL_CHARS:
+            msg = (
+                f"expression is {len(expression)} characters: "
+                f"keep it under {MAX_EVAL_CHARS}. Return the value you want "
+                "rather than a whole document."
+            )
+            raise WebSkrapError(msg, code=ErrorCode.USAGE)
+    logger.info("browser_eval session=%s len=%d", session, len(expression))
+    logger.debug("browser_eval session=%s expression=%s", session, expression)
     result = await _browser_action(session, lambda page: page.evaluate(expression), timeout_ms)
     return browser_session.shape_eval_result(result, max_chars)