contextractor 0.4.1__tar.gz → 0.4.2__tar.gz

{contextractor-0.4.1 → contextractor-0.4.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: contextractor
-Version: 0.4.1
+Version: 0.4.2
 Summary: Drive the Contextractor Node crawler/extractor from Python — clean main-content text in txt, markdown, json, or html.
 Project-URL: Homepage, https://apify.com/glueo/contextractor
 Project-URL: Repository, https://github.com/contextractor/contextractor

{contextractor-0.4.1 → contextractor-0.4.2}/SPEC.md

@@ -48,8 +48,8 @@ reads exactly what extract wrote — no bucket names are threaded.
 - export `0` → read manifest; non-zero → raise.
 - Both runners (`_run_sync` / `_run_async`) capture raw bytes and decode stdout/stderr as UTF-8 with `errors="replace"` — never the locale codec (Windows cp1252/cp932 mojibake) and never universal-newline translation (which would corrupt `original` raw HTML).
 - Playwright "Executable doesn't exist" in stderr → `MissingBrowserError` pointing at `python -m contextractor install`.
-- Child stderr is redacted (proxy credentials masked) before being surfaced; argv is never echoed when it carries a proxy.
-- A `timeout` (sync or async) raises `ContextractorError("contextractor timed out")` — never the raw `subprocess.TimeoutExpired`, whose `cmd` would leak the `--proxy` argv.
+- Child stderr is redacted before being surfaced — proxy URLs, header values (e.g. `Authorization` tokens), and cookie values (each ≥ 4 chars) are all registered as secrets; argv is never echoed when it carries a proxy.
+- A `timeout` (sync or async) raises `ContextractorError("contextractor timed out")` — never the raw `subprocess.TimeoutExpired`, whose `cmd` would leak the `--proxy` argv. Cancelling the surrounding task in the async path also kills and reaps the child — it is never orphaned.
 
 ## Single-page orchestration (`extract_one`)
 
@@ -70,9 +70,9 @@ A single data-driven table, `OPTION_SPECS`, applied immediately before spawn (pe
 `OPTION_SPECS` keys (enforced by `tests/test_options.py`). Categories:
 
 - **scalar** → `--flag <value>` (e.g. `max_crawl_depth`, `mode`, `start_urls_file`, …).
-- **bool-pair** → `--flag` / `--no-flag`: `headless`, `block_media`, `images`.
+- **bool-pair** → `--flag` / `--no-flag`: `headless`, `block_media`, `images`, `close_cookie_modals`.
 - **negation-only** (default include; `False` emits the `--no-` flag): `links`, `comments`, `tables`.
-- **bare-switch** (`True` emits the flag): `purge`, `ignore_cors_and_csp`, `close_cookie_modals`, `ignore_https_errors`, `keep_url_fragment`, `use_sitemaps`, `respect_robots_txt`, `store_skipped_urls`, `verbose` (`-v`).
+- **bare-switch** (`True` emits the flag): `purge`, `ignore_cors_and_csp`, `ignore_https_errors`, `keep_url_fragment`, `use_sitemaps`, `respect_robots_txt`, `store_skipped_urls`, `verbose` (`-v`).
 - **repeatable** (one flag per item): `proxy`, `globs`, `exclude`, `save` (`format-destination` tokens, e.g. `markdown-kvs`).
 - **json** (`--flag <json.dumps>`): `cookies`, `headers`.
 
@@ -119,11 +119,11 @@ browsers are never bundled (`python -m contextractor install`).
 
 ## Packaging & distribution
 
-- Backend: hatchling + `hatch_build.py` (`pure_python=False`, `infer_tag=True`) → `py3-none-{platform}` wheels. Forbid maturin / scikit-build-core / uv_build.
-- `version` is static in `pyproject.toml` (the `/git:release` and `/publish:all` bump target); `__version__` is read from installed metadata.
+- Backend: hatchling + `hatch_build.py` (`pure_python=False`; explicit `tag = py3-none-{platform}`, pinned via `CONTEXTRACTOR_WHEEL_PLATFORM` in CI, inferred locally) → `py3-none-{platform}` wheels. Forbid maturin / scikit-build-core / uv_build.
+- `version` is static in `pyproject.toml` (the `/git:release` and `/publish:pypi` bump target); `__version__` is read from installed metadata.
 - `readme = "README.md"` → the PyPI project page (per `.claude/rules/user-facing-docs.md`); included in the sdist.
 - Wheel matrix: `macosx_*_arm64`, `macosx_*_x86_64`, `manylinux_2_28_x86_64`, `manylinux_2_28_aarch64`, `win_amd64`, plus an sdist. **musl is unsupported** — the napi loader throws a clear import error rather than ship a broken `.node`.
-- CI: `.github/workflows/release-pypi.yml` (cibuildwheel; `CIBW_BEFORE_ALL` stages `_vendor`; auditwheel/delocate repair disabled — there is no ELF Python extension; publish via PyPI Trusted Publishing / OIDC). It is sequenced **after** the napi-refresh PR opened by `build-napi.yml` for a `v*` tag, so wheels bundle current `.node` files — the gate is encoded in `/publish:all`.
+- CI: `.github/workflows/release-pypi.yml` (cibuildwheel; `CIBW_BEFORE_ALL` stages `_vendor`; auditwheel/delocate repair disabled — there is no ELF Python extension; publish via PyPI Trusted Publishing / OIDC). It is sequenced **after** the napi-refresh PR opened by `build-napi.yml` for a `v*` tag, so wheels bundle current `.node` files — the gate is encoded in `/publish:pypi`.
 
 ## Tests
 

{contextractor-0.4.1 → contextractor-0.4.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "contextractor"
-version = "0.4.1"
+version = "0.4.2"
 description = "Drive the Contextractor Node crawler/extractor from Python — clean main-content text in txt, markdown, json, or html."
 readme = "README.md"
 requires-python = ">=3.12"

{contextractor-0.4.1 → contextractor-0.4.2}/src/contextractor/_manifest.py

@@ -28,10 +28,12 @@ def read_summary(manifest_path: Path, output_dir: Path) -> ExtractSummary:
     """Read and tally the manifest at ``manifest_path``."""
     try:
         raw = manifest_path.read_text(encoding="utf-8")
-    except OSError as exc:
+    except FileNotFoundError as exc:
         raise ContextractorError(
             f"manifest not found at {manifest_path} — the export step wrote nothing"
         ) from exc
+    except OSError as exc:
+        raise ContextractorError(f"could not read manifest at {manifest_path}") from exc
     try:
         records = json.loads(raw)
     except json.JSONDecodeError as exc:

{contextractor-0.4.1 → contextractor-0.4.2}/src/contextractor/_options.py

@@ -73,6 +73,9 @@ OPTION_SPECS: dict[str, _Spec] = {
     "headless": _Spec("--headless", _Kind.BOOL_PAIR, "--no-headless"),
     "block_media": _Spec("--block-media", _Kind.BOOL_PAIR, "--no-block-media"),
     "images": _Spec("--images", _Kind.BOOL_PAIR, "--no-images"),
+    "close_cookie_modals": _Spec(
+        "--close-cookie-modals", _Kind.BOOL_PAIR, "--no-close-cookie-modals"
+    ),
     # --- negation-only (default include; False emits the --no- flag) ---------
     "links": _Spec("--no-links", _Kind.NEGATION_ONLY),
     "comments": _Spec("--no-comments", _Kind.NEGATION_ONLY),
@@ -80,7 +83,6 @@ OPTION_SPECS: dict[str, _Spec] = {
     # --- bare switches ------------------------------------------------------
     "purge": _Spec("--purge", _Kind.BARE_SWITCH),
     "ignore_cors_and_csp": _Spec("--ignore-cors-and-csp", _Kind.BARE_SWITCH),
-    "close_cookie_modals": _Spec("--close-cookie-modals", _Kind.BARE_SWITCH),
     "ignore_https_errors": _Spec("--ignore-https-errors", _Kind.BARE_SWITCH),
     "keep_url_fragment": _Spec("--keep-url-fragment", _Kind.BARE_SWITCH),
     "use_sitemaps": _Spec("--use-sitemaps", _Kind.BARE_SWITCH),
@@ -218,7 +220,12 @@ def validate_proxies(proxies: Sequence[str]) -> None:
     The raw URL (which carries credentials) is never echoed in the error.
     """
     for raw in proxies:
-        scheme = urlsplit(raw).scheme.lower()
+        try:
+            scheme = urlsplit(raw).scheme.lower()
+        except ValueError:
+            # urlsplit rejects e.g. malformed IPv6 brackets; stay inside the
+            # ContextractorError hierarchy and never echo the raw URL.
+            raise ProxySchemeError("malformed proxy URL") from None
         if scheme not in _ALLOWED_PROXY_SCHEMES:
             raise ProxySchemeError(
                 f"unsupported proxy scheme {scheme or '(none)'!r}; "

{contextractor-0.4.1 → contextractor-0.4.2}/src/contextractor/_run.py

@@ -53,6 +53,24 @@ def _normalize_urls(urls: list[str] | str) -> list[str]:
     return [urls] if isinstance(urls, str) else list(urls)
 
 
+def _collect_secrets(opts: dict[str, Any]) -> tuple[str, ...]:
+    """Everything the child could echo back in stderr: proxy URLs, header
+    values (e.g. Authorization tokens), and cookie values. Values shorter than
+    4 chars are skipped — they are not meaningful secrets and replacing them
+    literally would mangle the error detail."""
+    headers = opts.get("headers") or {}
+    cookies = opts.get("cookies") or []
+    return (
+        *(opts.get("proxy") or ()),
+        *(v for v in headers.values() if isinstance(v, str) and len(v) >= 4),
+        *(
+            c["value"]
+            for c in cookies
+            if isinstance(c, dict) and isinstance(c.get("value"), str) and len(c["value"]) >= 4
+        ),
+    )
+
+
 def _build_plan(
     urls: list[str] | str,
     output_dir: str | None,
@@ -76,14 +94,13 @@ def _build_plan(
         else Path(tempfile.mkdtemp(prefix="contextractor-storage-"))
     )
 
-    secrets = tuple(opts.get("proxy") or ())
     return _Plan(
         urls=url_list,
         extract_flags=extract_flags,
         output_dir=out_dir,
         storage_dir=store_dir,
         owns_storage=owns_storage,
-        secrets=secrets,
+        secrets=_collect_secrets(opts),
     )
 
 
@@ -217,9 +234,13 @@ async def _run_async(argv: list[str], timeout: float | None) -> tuple[int | None
         async with asyncio.timeout(timeout):
             out, err = await proc.communicate()
     except TimeoutError:
-        proc.kill()
-        await proc.wait()
         raise ContextractorError("contextractor timed out") from None
+    finally:
+        # Reap the child whenever communicate() did not finish — timeout or
+        # cancellation of the surrounding task — so the Node crawler is never orphaned.
+        if proc.returncode is None:
+            proc.kill()
+            await proc.wait()
     return proc.returncode, out.decode(errors="replace"), err.decode(errors="replace")
 
 
@@ -272,7 +293,7 @@ def _build_one_plan(
         url=url,
         formats=_normalize_formats(formats),
         flags=flags,
-        secrets=tuple(opts.get("proxy") or ()),
+        secrets=_collect_secrets(opts),
     )