flru-parser 0.3.0__tar.gz → 0.3.2__tar.gz

{flru_parser-0.3.0 → flru_parser-0.3.2}/CHANGELOG.md

@@ -4,6 +4,33 @@ All notable changes are documented here. The project follows [Semantic Versionin
 
 ## [Unreleased]
 
+## [0.3.2] - 2026-07-26
+
+### Fixed
+
+- Parse current FL.ru catalog cards as complete cards instead of mistaking title elements
+  for card containers, restoring descriptions, budgets, kinds, and other card fields.
+- Follow FL.ru's `/projects/page-N/` pagination so `pages="all"` traverses the catalog.
+- Default the high-level projects APIs to orders (`kind=1`), preventing vacancies from
+  appearing among orders unless explicitly requested through `types`.
+- Keep package and parser metadata versions synchronized.
+
+## [0.3.1] - 2026-07-26
+
+### Added
+
+- Block-response diagnostics: sanitized response headers and body are saved under
+  `.flru-debug/blocked-.../` when CAPTCHA or anti-bot protection is detected.
+- `BlockedError.status_code` and `BlockedError.debug_path` for programmatic diagnostics.
+- Persistent server-set cookies shared by the transport's existing clients.
+- A conservative 0.5-second minimum interval between request starts by default.
+
+### Changed
+
+- CAPTCHA responses are never retried; batches and iterators stop immediately on a block.
+- Retry handling honors `Retry-After` for both HTTP 429 and HTTP 503 responses.
+- Block detection now requires strong evidence and ignores isolated `captcha` text in normal scripts.
+
 ## [0.3.0] - 2026-07-25
 
 ### Added

{flru_parser-0.3.0 → flru_parser-0.3.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: flru-parser
-Version: 0.3.0
+Version: 0.3.2
 Summary: Simple, typed and resilient async/sync parser client for public FL.ru pages
 Author: Andrey
 License-Expression: MIT
@@ -27,6 +27,7 @@ Requires-Dist: beautifulsoup4<5,>=4.12
 Requires-Dist: httpx[http2]<1,>=0.28
 Requires-Dist: lxml<7,>=5
 Requires-Dist: pydantic<3,>=2.10
+Requires-Dist: tzdata>=2025.2
 Provides-Extra: postgres
 Requires-Dist: asyncpg<1,>=0.30; extra == "postgres"
 Provides-Extra: redis
@@ -43,9 +44,8 @@ Dynamic: license-file
 [![Python](https://img.shields.io/pypi/pyversions/flru-parser.svg)](https://pypi.org/project/flru-parser/)
 [![License](https://img.shields.io/pypi/l/flru-parser.svg)](LICENSE)
 [![Typing](https://img.shields.io/badge/typing-PEP%20561-blue)](src/flru/py.typed)
-[![Coverage](https://img.shields.io/badge/branch%20coverage-85.76%25-brightgreen)](#quality)
-[![Tests](https://img.shields.io/badge/tests-47%20passed-brightgreen)](#quality)
-[![CI](https://github.com/N3ffus/flru-parser/actions/workflows/ci.yml/badge.svg)](https://github.com/N3ffus/flru-parser/actions/workflows/ci.yml)
+[![Tests](https://github.com/N3ffus/flru-parser/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/N3ffus/flru-parser/actions/workflows/ci.yml?query=branch%3Amain)
+[![Codecov](https://codecov.io/gh/N3ffus/flru-parser/branch/main/graph/badge.svg)](https://codecov.io/gh/N3ffus/flru-parser/branch/main)
 
 A typed, resilient, read-only Python client for public pages on [FL.ru](https://www.fl.ru/).
 The default API is intentionally small; production controls remain available when needed.
@@ -324,8 +324,10 @@ See [docs/SIMPLE_API.md](docs/SIMPLE_API.md) for the complete high-level API and
 - Unclassified empty pages raise `EmptyPageError`.
 - A recognized end page contains the `catalog_end` diagnostic warning.
 - HTTP 429 can pause all concurrent workers through a shared cooldown.
+- HTTP 429 and 503 honor `Retry-After` when the server supplies it.
 - Circuit breakers are scoped by endpoint and proxy by default.
-- CAPTCHA and block pages are detected, not bypassed.
+- CAPTCHA and block pages are detected, not bypassed or retried. Their HTML and sanitized response
+  metadata are saved to `.flru-debug/blocked-.../`; this directory is ignored by Git.
 - Requests are restricted to allowed FL.ru hosts and safe redirects.
 
 ## Observability
@@ -356,7 +358,7 @@ Install `flru-parser[observability]` before using these adapters.
 
 | Exception | Meaning |
 |---|---|
-| `BlockedError` | CAPTCHA, anti-bot or access-block page detected |
+| `BlockedError` | CAPTCHA, anti-bot or access-block page detected; exposes `status_code` and `debug_path` |
 | `AuthenticationRequired` | Authenticated cookies are required |
 | `RateLimitedError` | HTTP 429 after retry policy exhaustion |
 | `CircuitOpenError` | A scoped circuit breaker is open |
@@ -368,13 +370,7 @@ Install `flru-parser[observability]` before using these adapters.
 
 ## Quality
 
-Current verified result for **0.3.0**:
-
-```text
-47 passed, 1 optional Hypothesis suite skipped locally
-Branch coverage: 85.76%
-Coverage gate: 85%
-```
+The CI and Codecov badges above always show the current `main` result.
 
 Run the complete local checks:
 
@@ -402,8 +398,8 @@ The release workflow uses PyPI Trusted Publishing and checks that the Git tag ma
 
 ```bash
 uv run python scripts/configure_project.py YOUR_GITHUB_USERNAME
-git tag -a v0.3.0 -m "flru-parser 0.3.0"
-git push origin main v0.3.0
+git tag -a v0.3.2 -m "flru-parser 0.3.2"
+git push origin main v0.3.2
 ```
 
 See [docs/PUBLISHING.md](docs/PUBLISHING.md) for the one-time PyPI and GitHub setup.

{flru_parser-0.3.0 → flru_parser-0.3.2}/README.md

@@ -4,9 +4,8 @@
 [![Python](https://img.shields.io/pypi/pyversions/flru-parser.svg)](https://pypi.org/project/flru-parser/)
 [![License](https://img.shields.io/pypi/l/flru-parser.svg)](LICENSE)
 [![Typing](https://img.shields.io/badge/typing-PEP%20561-blue)](src/flru/py.typed)
-[![Coverage](https://img.shields.io/badge/branch%20coverage-85.76%25-brightgreen)](#quality)
-[![Tests](https://img.shields.io/badge/tests-47%20passed-brightgreen)](#quality)
-[![CI](https://github.com/N3ffus/flru-parser/actions/workflows/ci.yml/badge.svg)](https://github.com/N3ffus/flru-parser/actions/workflows/ci.yml)
+[![Tests](https://github.com/N3ffus/flru-parser/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/N3ffus/flru-parser/actions/workflows/ci.yml?query=branch%3Amain)
+[![Codecov](https://codecov.io/gh/N3ffus/flru-parser/branch/main/graph/badge.svg)](https://codecov.io/gh/N3ffus/flru-parser/branch/main)
 
 A typed, resilient, read-only Python client for public pages on [FL.ru](https://www.fl.ru/).
 The default API is intentionally small; production controls remain available when needed.
@@ -285,8 +284,10 @@ See [docs/SIMPLE_API.md](docs/SIMPLE_API.md) for the complete high-level API and
 - Unclassified empty pages raise `EmptyPageError`.
 - A recognized end page contains the `catalog_end` diagnostic warning.
 - HTTP 429 can pause all concurrent workers through a shared cooldown.
+- HTTP 429 and 503 honor `Retry-After` when the server supplies it.
 - Circuit breakers are scoped by endpoint and proxy by default.
-- CAPTCHA and block pages are detected, not bypassed.
+- CAPTCHA and block pages are detected, not bypassed or retried. Their HTML and sanitized response
+  metadata are saved to `.flru-debug/blocked-.../`; this directory is ignored by Git.
 - Requests are restricted to allowed FL.ru hosts and safe redirects.
 
 ## Observability
@@ -317,7 +318,7 @@ Install `flru-parser[observability]` before using these adapters.
 
 | Exception | Meaning |
 |---|---|
-| `BlockedError` | CAPTCHA, anti-bot or access-block page detected |
+| `BlockedError` | CAPTCHA, anti-bot or access-block page detected; exposes `status_code` and `debug_path` |
 | `AuthenticationRequired` | Authenticated cookies are required |
 | `RateLimitedError` | HTTP 429 after retry policy exhaustion |
 | `CircuitOpenError` | A scoped circuit breaker is open |
@@ -329,13 +330,7 @@ Install `flru-parser[observability]` before using these adapters.
 
 ## Quality
 
-Current verified result for **0.3.0**:
-
-```text
-47 passed, 1 optional Hypothesis suite skipped locally
-Branch coverage: 85.76%
-Coverage gate: 85%
-```
+The CI and Codecov badges above always show the current `main` result.
 
 Run the complete local checks:
 
@@ -363,8 +358,8 @@ The release workflow uses PyPI Trusted Publishing and checks that the Git tag ma
 
 ```bash
 uv run python scripts/configure_project.py YOUR_GITHUB_USERNAME
-git tag -a v0.3.0 -m "flru-parser 0.3.0"
-git push origin main v0.3.0
+git tag -a v0.3.2 -m "flru-parser 0.3.2"
+git push origin main v0.3.2
 ```
 
 See [docs/PUBLISHING.md](docs/PUBLISHING.md) for the one-time PyPI and GitHub setup.

{flru_parser-0.3.0 → flru_parser-0.3.2}/docs/PUBLISHING.md

@@ -34,7 +34,7 @@ uv pip install \
   --python /tmp/flru-test/bin/python \
   --index-url https://test.pypi.org/simple/ \
   --extra-index-url https://pypi.org/simple/ \
-  flru-parser==0.3.0
+  flru-parser==0.3.2
 ```
 
 On Windows, use `/tmp/flru-test/Scripts/python.exe` instead.
@@ -47,8 +47,8 @@ On Windows, use `/tmp/flru-test/Scripts/python.exe` instead.
 4. Commit, create an annotated tag, and push it:
 
 ```bash
-git tag -a v0.3.0 -m "flru-parser 0.3.0"
-git push origin main v0.3.0
+git tag -a v0.3.2 -m "flru-parser 0.3.2"
+git push origin main v0.3.2
 ```
 
 The workflow builds distributions once, validates them, and publishes the exact same artifacts through PyPI Trusted Publishing.

{flru_parser-0.3.0 → flru_parser-0.3.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "flru-parser"
-version = "0.3.0"
+version = "0.3.2"
 description = "Simple, typed and resilient async/sync parser client for public FL.ru pages"
 readme = "README.md"
 requires-python = ">=3.11"
@@ -17,6 +17,7 @@ dependencies = [
   "httpx[http2]>=0.28,<1",
   "lxml>=5,<7",
   "pydantic>=2.10,<3",
+  "tzdata>=2025.2",
 ]
 classifiers = [
   "Development Status :: 4 - Beta",

{flru_parser-0.3.0 → flru_parser-0.3.2}/src/flru/__init__.py

@@ -67,7 +67,7 @@ from .state import (
     project_content_hash,
 )
 
-__version__ = "0.3.0"
+__version__ = "0.3.2"
 
 AsyncClient = Client
 

{flru_parser-0.3.0 → flru_parser-0.3.2}/src/flru/client.py

@@ -3,10 +3,10 @@ from __future__ import annotations
 import asyncio
 import email.utils
 import re
-from collections.abc import AsyncIterable, AsyncIterator, Iterable, Mapping, Sequence
+from collections.abc import AsyncIterable, AsyncIterator, Awaitable, Iterable, Mapping, Sequence
 from datetime import datetime, timezone
 from pathlib import Path
-from typing import Any, Self, cast
+from typing import Any, Self, TypeVar, cast
 from urllib.parse import parse_qs, urlencode, urljoin, urlsplit
 
 import httpx
@@ -15,8 +15,14 @@ from bs4 import BeautifulSoup
 from .batch import BatchError, BatchResult
 from .config import ClientConfig
 from .cookies import load_netscape_cookies
-from .exceptions import EmptyPageError, ParseError, RobotsDeniedError, SelectorDriftError
-from .filters import ProjectFilters
+from .exceptions import (
+    BlockedError,
+    EmptyPageError,
+    ParseError,
+    RobotsDeniedError,
+    SelectorDriftError,
+)
+from .filters import ProjectFilters, ProjectType
 from .models import (
     Category,
     CrawlCheckpoint,
@@ -25,6 +31,7 @@ from .models import (
     PageData,
     PortfolioItem,
     ProjectDetail,
+    ProjectKind,
     ProjectPage,
     ProjectSummary,
     RequestMetrics,
@@ -44,6 +51,26 @@ from .robots import RobotsPolicy
 from .state import CrawlStateStore, record_for
 from .transport import ResilientTransport
 
+T = TypeVar("T")
+
+
+async def _gather_batch_or_stop_on_block(
+    awaitables: Iterable[Awaitable[T]],
+) -> list[T | BaseException]:
+    """Cancel outstanding work when a block makes further scraping inappropriate."""
+    tasks = [asyncio.ensure_future(awaitable) for awaitable in awaitables]
+    if not tasks:
+        return []
+    done, pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_EXCEPTION)
+    for task in done:
+        error = task.exception()
+        if isinstance(error, BlockedError):
+            for pending_task in pending:
+                pending_task.cancel()
+            await asyncio.gather(*pending, return_exceptions=True)
+            raise error
+    return list(await asyncio.gather(*tasks, return_exceptions=True))
+
 
 class FLClient:
     """Asynchronous, read-only client/parser for public FL.ru pages."""
@@ -173,6 +200,14 @@ class FLClient:
                 timezone_name=self.config.parser_timezone,
             )
             self._validate_project_page(result)
+            if filters and filters.project_types:
+                kind_map = {
+                    ProjectType.ORDER: ProjectKind.ORDER,
+                    ProjectType.VACANCY: ProjectKind.VACANCY,
+                    ProjectType.CONTEST: ProjectKind.CONTEST,
+                }
+                requested = {kind_map[item] for item in filters.project_types}
+                result.items = [item for item in result.items if item.kind in requested]
             return result
         except ParseError as exc:
             await self._record_parse_failure(effective_url, html, exc)
@@ -216,15 +251,17 @@ class FLClient:
                     params=params,
                 )
 
-        values = await asyncio.gather(*(load(page) for page in page_numbers), return_exceptions=True)
+        values = await _gather_batch_or_stop_on_block(load(page) for page in page_numbers)
         result: BatchResult[int, ProjectPage] = BatchResult()
         for page, value in zip(page_numbers, values, strict=True):
             if isinstance(value, asyncio.CancelledError):
                 raise value
+            if isinstance(value, BlockedError):
+                raise value
             if isinstance(value, Exception):
                 result.failed.append(BatchError(page, value))
             else:
-                result.successful.append(value)
+                result.successful.append(cast(ProjectPage, value))
         return result
 
     async def get_projects_batch(
@@ -307,6 +344,16 @@ class FLClient:
         fail_fast: bool = True,
     ) -> AsyncIterator[ProjectSummary]:
         seen: set[int] = set()
+        allowed_kinds = {
+            ProjectType.ORDER: ProjectKind.ORDER,
+            ProjectType.VACANCY: ProjectKind.VACANCY,
+            ProjectType.CONTEST: ProjectKind.CONTEST,
+        }
+        requested_kinds = (
+            {allowed_kinds[item] for item in filters.project_types}
+            if filters and filters.project_types
+            else set()
+        )
         async for page in self.iter_project_pages(
             start_page=start_page,
             max_pages=max_pages,
@@ -317,6 +364,11 @@ class FLClient:
             fail_fast=fail_fast,
         ):
             for item in page.items:
+                # FL.ru can include pinned cards of another kind even when a
+                # `kind` filter is present, so enforce the requested kinds
+                # locally as well.
+                if requested_kinds and item.kind not in requested_kinds:
+                    continue
                 if deduplicate and item.id in seen:
                     continue
                 seen.add(item.id)
@@ -461,15 +513,17 @@ class FLClient:
             async with semaphore:
                 return await self.get_freelancers_page(page, category=category)
 
-        values = await asyncio.gather(*(load(page) for page in page_numbers), return_exceptions=True)
+        values = await _gather_batch_or_stop_on_block(load(page) for page in page_numbers)
         result: BatchResult[int, FreelancerPage] = BatchResult()
         for page, value in zip(page_numbers, values, strict=True):
             if isinstance(value, asyncio.CancelledError):
                 raise value
+            if isinstance(value, BlockedError):
+                raise value
             if isinstance(value, Exception):
                 result.failed.append(BatchError(page, value))
             else:
-                result.successful.append(value)
+                result.successful.append(cast(FreelancerPage, value))
         return result
 
     async def get_freelancers_batch(
@@ -555,16 +609,18 @@ class FLClient:
             async with semaphore:
                 return await self.get_project(value)
 
-        values = await asyncio.gather(*(load(project) for project in projects), return_exceptions=True)
+        values = await _gather_batch_or_stop_on_block(load(project) for project in projects)
         result: BatchResult[int | str, ProjectDetail] = BatchResult()
         for project, value in zip(projects, values, strict=True):
             key: int | str = project.id if isinstance(project, ProjectSummary) else project
             if isinstance(value, asyncio.CancelledError):
                 raise value
+            if isinstance(value, BlockedError):
+                raise value
             if isinstance(value, Exception):
                 result.failed.append(BatchError(key, value))
             else:
-                result.successful.append(value)
+                result.successful.append(cast(ProjectDetail, value))
         return result
 
     async def get_project_details(

{flru_parser-0.3.0 → flru_parser-0.3.2}/src/flru/config.py

@@ -33,7 +33,7 @@ class RateLimitConfig:
     requests_per_second: float = 1.0
     burst: int = 2
     max_concurrency: int = 5
-    min_interval: float = 0.0
+    min_interval: float = 0.5
     interval_jitter: float = 0.15
 
 
@@ -101,6 +101,8 @@ class ClientConfig:
             raise ConfigurationError("requests_per_second must be positive")
         if self.rate_limit.max_concurrency <= 0:
             raise ConfigurationError("max_concurrency must be positive")
+        if self.rate_limit.min_interval < 0:
+            raise ConfigurationError("min_interval must not be negative")
         if self.retry.max_attempts <= 0:
             raise ConfigurationError("max_attempts must be positive")
         if self.retry.total_timeout <= 0:

{flru_parser-0.3.0 → flru_parser-0.3.2}/src/flru/easy.py

@@ -444,12 +444,18 @@ def _project_filters(
     types: ProjectTypeInput,
     with_budget: bool | None,
 ) -> ProjectFilters:
+    # The public "projects" API represents FL.ru orders.  FL.ru's unfiltered
+    # catalog also contains vacancies and contests, so request orders unless
+    # callers explicitly select one or more kinds.
+    project_types = _normalize_project_types(types)
+    if types is None:
+        project_types = frozenset({ProjectType.ORDER})
     return ProjectFilters(
         query=query,
         category=category,
         budget_from=min_budget,
         budget_to=max_budget,
-        project_types=_normalize_project_types(types),
+        project_types=project_types,
         only_with_budget=with_budget,
     )
 

{flru_parser-0.3.0 → flru_parser-0.3.2}/src/flru/exceptions.py

@@ -1,5 +1,7 @@
 from __future__ import annotations
 
+from pathlib import Path
+
 
 class FLRUError(Exception):
     """Base package exception."""
@@ -27,6 +29,17 @@ class RateLimitedError(HTTPStatusError):
 class BlockedError(TransportError):
     """A CAPTCHA, anti-bot page, or access block was detected."""
 
+    def __init__(
+        self,
+        message: str,
+        *,
+        status_code: int | None = None,
+        debug_path: Path | None = None,
+    ) -> None:
+        self.status_code = status_code
+        self.debug_path = debug_path
+        super().__init__(message)
+
 
 class AuthenticationRequired(TransportError):
     """The requested page requires a valid authenticated session."""

{flru_parser-0.3.0 → flru_parser-0.3.2}/src/flru/models.py

@@ -7,7 +7,7 @@ from typing import Any
 
 from pydantic import BaseModel, ConfigDict, Field
 
-PARSER_VERSION = "0.3.0"
+PARSER_VERSION = "0.3.2"
 SCHEMA_VERSION = "1"
 
 

{flru_parser-0.3.0 → flru_parser-0.3.2}/src/flru/parsers/projects.py

@@ -39,7 +39,7 @@ from .common import (
 
 TITLE_SELECTORS = ("h1", '[itemprop="name"]', ".b-page__title", ".project-title")
 DESCRIPTION_SELECTORS = (
-    '[itemprop="description"]', ".b-layout__txt_padbot_20", ".b-post__txt",
+    '[itemprop="description"]', ".b-layout__txt_padbot_20", ".b-post__txt.text-5",
     ".project-description", '[class*="description"]',
 )
 
@@ -48,8 +48,12 @@ def _find_card(anchor: Tag) -> Tag:
     for parent in anchor.parents:
         if not isinstance(parent, Tag):
             continue
-        classes = " ".join(parent.get("class", []))
-        if parent.name == "article" or re.search(r"(?:^|[-_ ])(?:project|b-post)(?:[-_ ]|$)", classes):
+        classes = set(parent.get("class", []))
+        if (
+            parent.name == "article"
+            or classes.intersection({"b-post", "project-card", "project", "catalog-item"})
+            or str(parent.get("data-id", "")).startswith("qa-lenta-")
+        ):
             return parent
         if parent.name in {"main", "body"}:
             break
@@ -124,7 +128,10 @@ def _next_page_url(soup: BeautifulSoup, url: str, page: int) -> str | None:
         return absolute_url(url, str(explicit.get("href")))
     for anchor in soup.select("a[href]"):
         href = absolute_url(url, str(anchor.get("href")))
-        if href and parse_qs(urlsplit(href).query).get("page") == [str(page + 1)]:
+        if href and (
+            parse_qs(urlsplit(href).query).get("page") == [str(page + 1)]
+            or re.search(rf"/page-{page + 1}/?$", urlsplit(href).path)
+        ):
             return href
     return None
 

{flru_parser-0.3.0 → flru_parser-0.3.2}/src/flru/security.py

@@ -33,9 +33,7 @@ def redact_url(url: str | None) -> str | None:
     parts = urlsplit(url)
     hostname = parts.hostname or ""
     port = f":{parts.port}" if parts.port else ""
-    userinfo = ""
-    if parts.username:
-        userinfo = f"{parts.username}:***@"
+    userinfo = "***@" if parts.username else ""
     netloc = f"{userinfo}{hostname}{port}"
     query = _SECRET_QUERY_RE.sub(lambda match: f"{match.group(1)}=***", parts.query)
     return urlunsplit(SplitResult(parts.scheme, netloc, parts.path, query, parts.fragment))

{flru_parser-0.3.0 → flru_parser-0.3.2}/src/flru/transport.py

@@ -2,8 +2,10 @@ from __future__ import annotations
 
 import asyncio
 import email.utils
+import json
 import random
-from datetime import datetime, timezone
+from datetime import UTC, datetime
+from pathlib import Path
 from typing import Any
 from urllib.parse import urljoin, urlsplit
 
@@ -22,11 +24,14 @@ from .proxy import ProxyPool, ProxyState
 from .resilience import AsyncRateLimiter, CircuitBreakerRegistry, RetryBudget
 from .security import validate_url
 
-_BLOCK_MARKERS = (
-    "captcha",
-    "подтвердите, что вы не робот",
-    "доступ временно ограничен",
-    "cloudflare",
+_STRONG_BLOCK_MARKERS = (
+    "g-recaptcha",
+    "hcaptcha",
+    "captcha-container",
+    "cf-chl-",
+    "challenge-platform",
+    "<title>access denied",
+    "<title>доступ ограничен",
 )
 _AUTH_MARKERS = (
     "необходимо авторизоваться",
@@ -35,6 +40,28 @@ _AUTH_MARKERS = (
 _REDIRECT_CODES = {301, 302, 303, 307, 308}
 
 
+def _dump_blocked_response(response: httpx.Response) -> Path:
+    """Persist a blocked response without serializing request credentials."""
+    timestamp = datetime.now(UTC).strftime("%Y%m%dT%H%M%SZ")
+    directory = Path(".flru-debug") / f"blocked-{timestamp}"
+    directory.mkdir(parents=True, exist_ok=True)
+
+    (directory / "response.html").write_bytes(response.content)
+    sensitive_headers = {"authorization", "cookie", "set-cookie", "proxy-authorization"}
+    safe_headers = {
+        key: value for key, value in response.headers.items() if key.lower() not in sensitive_headers
+    }
+    metadata = {
+        "status_code": response.status_code,
+        "url": str(response.url),
+        "headers": safe_headers,
+    }
+    (directory / "metadata.json").write_text(
+        json.dumps(metadata, ensure_ascii=False, indent=2), encoding="utf-8"
+    )
+    return directory
+
+
 class ResilientTransport:
     def __init__(
         self,
@@ -145,7 +172,17 @@ class ResilientTransport:
                     bytes_received=len(response.content),
                     latency_seconds_total=elapsed,
                 )
-                self._detect_block(response)
+                try:
+                    self._detect_block(response)
+                except BlockedError as exc:
+                    debug_directory = _dump_blocked_response(response)
+                    raise BlockedError(
+                        f"{exc}. Response saved to {debug_directory}",
+                        status_code=response.status_code,
+                        debug_path=debug_directory,
+                    ) from exc
+
+                self._persist_response_cookies(response)
 
                 if response.status_code in self.config.retry.retry_statuses:
                     if response.status_code == 429:
@@ -191,7 +228,6 @@ class ResilientTransport:
                     raise
                 await self._proxy_pool.failure(proxy_state, exc)
             except BlockedError as exc:
-                last_error = exc
                 await self.metrics.mutate(endpoint=endpoint, blocked_total=1, failures_total=1)
                 await self._proxy_pool.failure(proxy_state, exc)
                 await self._emit(
@@ -206,6 +242,8 @@ class ResilientTransport:
                         error=type(exc).__name__,
                     )
                 )
+                await circuit.record_failure()
+                raise
             except httpx.TransportError as exc:
                 last_error = exc
                 await self.metrics.mutate(endpoint=endpoint, failures_total=1)
@@ -320,7 +358,7 @@ class ResilientTransport:
             self.config.retry.max_delay,
             self.config.retry.base_delay * (2 ** (attempt - 1)),
         )
-        return exponential + random.uniform(0, self.config.retry.jitter)
+        return float(exponential + random.uniform(0, self.config.retry.jitter))
 
     def _retry_after(self, response: httpx.Response) -> float | None:
         return self._parse_retry_after(response.headers.get("Retry-After"))
@@ -335,8 +373,8 @@ class ResilientTransport:
             try:
                 parsed = email.utils.parsedate_to_datetime(value)
                 if parsed.tzinfo is None:
-                    parsed = parsed.replace(tzinfo=timezone.utc)
-                return max(0.0, (parsed - datetime.now(timezone.utc)).total_seconds())
+                    parsed = parsed.replace(tzinfo=UTC)
+                return max(0.0, (parsed - datetime.now(UTC)).total_seconds())
             except (TypeError, ValueError):
                 return None
 
@@ -344,11 +382,25 @@ class ResilientTransport:
         if not self.config.detect_blocks:
             return
         sample = response.text[:200_000].casefold()
-        if any(marker in sample for marker in _BLOCK_MARKERS):
+        marker_count = sum(marker in sample for marker in _STRONG_BLOCK_MARKERS)
+        suspicious_status = response.status_code in {403, 429, 503}
+        redirected_to_challenge = any(
+            part in str(response.url).casefold() for part in ("captcha", "challenge", "blocked")
+        )
+        if redirected_to_challenge or marker_count >= 2:
             raise BlockedError(f"Anti-bot or CAPTCHA page detected at {response.url}")
+        if suspicious_status and marker_count >= 1:
+            raise BlockedError(
+                f"Possible anti-bot response ({response.status_code}) at {response.url}"
+            )
         if any(marker in sample for marker in _AUTH_MARKERS):
             raise AuthenticationRequired(f"Authentication is required for {response.url}")
 
+    def _persist_response_cookies(self, response: httpx.Response) -> None:
+        cookies = dict(response.cookies)
+        if cookies:
+            self.update_cookies(cookies)
+
     def update_cookies(self, cookies: dict[str, str]) -> None:
         self._cookies.update(cookies)
         for client in self._clients.values():