graph-crawler 5.3.7__tar.gz → 5.4.1__tar.gz

graph_crawler-5.4.1/CHANGELOG.md

@@ -0,0 +1,637 @@
+# Changelog
+
+## 5.4.1 — 2026-10
+
+### Fixed
+
+- **A page whose script rewrites the address bar is no longer treated as a redirect** (Parser `WEB_GRAF_ISSUES.md` §5,
+  NEW-1-15). Playwright provenance took `page.url` as the final URL, so `history.pushState/replaceState`
+  (e.g. `/en/vacancies/<slug>` → `/en`) looked like a JS redirect and `Graph._handle_redirect_canonical` merged the
+  scanned page into the existing `/en` node — its content was dropped (`deduped by canonical-redirect: N`).
+  `_build_playwright_provenance` now compares `page.url` with the URL of the loaded document
+  (`performance.getEntriesByType('navigation')[0].name`): if they differ, the document URL is the final one
+  (no redirect, or the server-3xx target); the rewrite is logged at INFO
+  `Address rewritten by page script (history API), not a redirect`. Real redirects (server 3xx, `location.href`)
+  are unchanged; if the document URL cannot be read, the old behaviour is kept.
+- Tests: `tests/transport/test_history_api_not_redirect.py` (real Chromium part: `GC_TEST_REAL_BROWSER=1`).
+
+### Changed
+
+- **Non-network error policies go through `resilix` (new core dependency `resilix>=1.1,<2`, zero-dependency).**
+  - `shared/utils/best_effort.close_quietly` — one best-effort close with timeout for browser/context/page/CDP
+    and the persistent engine session (`BrowserFactory._safe_close`, `close_browser`, `detach_page_cdp`,
+    PID-probe CDP detach, `PoolLifecycle._stop_persistent_session`). Log format is unchanged
+    (`<name> close timeout (<t>s)` at WARNING, other errors at DEBUG, `TargetClosedError` is silent);
+    external `CancelledError` is not swallowed.
+  - Functions whose whole body was one `try → default` are now `@protect.when(..., return_=...)`
+    (playwright `browser_factory`, `cookie_consent`, nodriver driver, captcha balance checks, storages
+    `clear`/`save_partial`, `cgroup_limits`, `url_validator.is_url_safe`, …).
+  - `try/except Exception: logger.debug("Suppressed …")` blocks are now `protect.suppress(...)` with the
+    calling function name as the log message.
+  - Batch fetches (`fetch_many`, `fetch_many_fast`) and sitemap fan-out use `resilix.gather`: results keep input
+    order, a failed URL becomes an error `FetchResponse`, concurrency is capped by `limit=max_concurrent`, and
+    an external `CancelledError` now propagates instead of being returned as a result.
+  - Fire-and-forget tasks (async middleware `setup()`, dashboard broadcast, event publishing, sitemap session
+    close) use `resilix.spawn` — no lost tasks, failures are logged.
+  - `CODE_STYLE.md` §7.3/§7.11: non-network policies — only `resilix.protect`; network retries — only
+    `RetryMiddleware`/`RetryPolicy`; `protect.timeout` only for `async def`.
+
+### Removed
+
+- Unused `graph_crawler.observability.decorators` (`retry`, `async_retry`, `timed`, `cached`, `log_calls`;
+  357 lines, no imports anywhere).
+
+## 5.4.0 — 2026-10
+
+### Fixed
+
+- **Oversized and binary responses are no longer downloaded in full** (Parser `WEB_GRAF_ISSUES.md` §1).
+  The dead `DEFAULT_MAX_CONTENT_LENGTH` constant is now enforced by `transport/core/body_limit.py` on every tier:
+  - HTTP tiers (aiohttp, curl_cffi — `fetch` and `fetch_fast`): a `Content-Length` above `max_response_bytes`
+    (default 10 MB) or a non-textual `Content-Type` (`application/octet-stream`, `video/*`, `application/zip`…,
+    `skip_non_textual_bodies=True`) skips the body entirely; without `Content-Length` the body is streamed in chunks
+    and the connection is dropped as soon as the limit is exceeded; decoded HTML is also capped by
+    `max_page_html_bytes` (default 15 MB);
+  - browser tiers: `max_page_html_bytes` caps the rendered HTML (Playwright incl. partial HTML after a timeout, and
+    nodriver); `max_page_total_bytes` (default 0 = off) caps the sum of all responses of a page and aborts further
+    non-document requests;
+  - a skipped body is returned as `html=None`, `error="response_too_large: …" / "non_textual_content: …"`,
+    `extras["body_skipped"]="too_large" | "non_textual"`; `EscalatingDriver` does not escalate it, forwards all
+    limit keys to its tiers, and the node gets `metadata["body_skipped"]` / `metadata["page_total_bytes_exceeded"]`;
+  - sitemaps keep their own larger limit: a sitemap skipped by the driver is refetched via the sitemap path.
+  - `0` disables any of the limits.
+- `AsyncUndetectedConfig.sandbox` was declared but ignored (hard-coded `False`); it is now passed to `uc.start()`
+  (default changed to `False`, i.e. the effective behaviour is unchanged).
+- Tests: `tests/transport/test_body_limits.py`.
+- **Compressed sitemaps (`.gz`) are read completely** (Parser `WEB_GRAF_ISSUES.md` §2, NEW-3-9).
+  `SitemapParser._fetch_aiohttp` used a single `response.content.read(MAX_SITEMAP_SIZE)`, which returns only the bytes
+  already received (usually the first network chunk), so a 30–300 KB `.gz` sitemap was cut to a few KB, failed to
+  decompress and was logged as `Sitemap truncated (exceeded 52428800 bytes)`. The body is now read in chunks until EOF
+  or until the limit; `truncated` is set only when the file is really larger than the limit (the log line now also
+  shows the number of bytes read). A `Content-Length` above the limit still skips the body.
+  - New parameter `max_sitemap_bytes` in `crawl_sitemap` / `async_crawl_sitemap` / `SitemapSpider`
+    (`None` → `MAX_SITEMAP_SIZE` 50 MB, `0` → no limit); previously the limit could be set only on a hand-built
+    `SitemapParser`.
+  - Tests: `tests/test_sitemap_gz_streaming.py` (local server streams a 5 000-URL `.gz` in 4 KB chunks).
+- **Redirect merge no longer logs `backend checkpoint skipped`** (Parser `WEB_GRAF_ISSUES.md` §3, NEW-3-19).
+  After merging a redirected node into an existing final node, `Graph._handle_redirect_canonical` called
+  `backend._run_async(backend.checkpoint())` without the `result_in_loop` argument added in 5.3.7, so every merge raised
+  `TypeError` (caught, logged as a warning, coroutine never run). The WAL checkpoint is not needed there —
+  `delete_node_sync` has already removed the node and its edges from the DB and the cache, and readers share the same
+  connection — so the call is removed. `SQLiteBackend._run_async` now defaults `result_in_loop=None`, which also fixes
+  the `Graph.remove_node` fallback for backends without `delete_node_sync`.
+  Tests: `tests/test_redirect_canonical_sqlite.py`.
+- **A large JSON-LD block with a `JobPosting` is no longer dropped whole** (Parser `WEB_GRAF_ISSUES.md` §4, NEW-2-12).
+  A block larger than `max_jsonld_size` (100 KB) but not larger than `max_jsonld_oversized_size` (new, default 5 MB) is
+  parsed when a quick regex finds an object of a type from `oversized_keep_types` (new, default `["JobPosting"]`;
+  plain, list and `https://schema.org/…` forms of `@type`), and only those objects are kept — also from `@graph`,
+  arrays and nested `ItemList` items; the rest of the block is discarded. Other oversized blocks are skipped as before;
+  the warning now states the size and why (`no JobPosting inside` / `contains JobPosting but exceeds hard limit N` /
+  `JobPosting mentioned but no such object found`). `oversized_keep_types=[]` restores the plain size limit.
+  Tests: `tests/unit/plugins/node/structured_data/test_jsonld_oversized.py`.
+- `LMDBEvictionStorage` had no `load_all_edges_sync`, so `Graph.load_all_from_disk()` with the LMDB backend (chosen
+  automatically when `lmdb` is installed) restored nodes but silently dropped evicted edges. Tests:
+  `tests/unit/test_p9_hotpath.py` (SQLite and LMDB).
+
+## 5.3.7 — 2026-10
+
+### Fixed
+
+- **A crawl worker no longer dies on a canonical redirect while another SQLite statement is running.**
+  `SQLiteBackend._sync_execute` treated `sqlite3.OperationalError: cannot commit transaction - SQL statements in
+  progress` as a failed write, although the DELETE/UPDATE had already run in the open transaction of the shared
+  connection. The fallback `_run_async` then ran the coroutine on a new loop in a thread (`asyncio.Lock ... is bound to
+  a different event loop`) and, catching that `RuntimeError`, called `asyncio.run()` inside the running loop — the
+  worker failed (`Graph.remove_node` → `delete_node_sync`). Now:
+  - a commit refused because of in-progress statements is deferred: the change stays in the open transaction and is
+    committed by the next async commit or by `close()`;
+  - `_run_async` inside a running loop schedules the write as a task on that loop (errors are logged) instead of
+    blocking on a foreign loop; outside a loop it still runs synchronously;
+  - `close()` awaits scheduled writes and commits before closing.
+- Tests: `tests/test_sqlite_sync_write_in_loop.py`.
+
+## 5.3.6 — 2026-06
+
+### Fixed
+
+- **A timeout on robots.txt / sitemap stops the sitemap phase of the host (1.6, daxx.com).** A down origin behind
+  Cloudflare does not always answer 52x: robots.txt came back after ~20 s and every sitemap request hung until the
+  client timeout, then the default candidates (`sitemap.xml`, `sitemap_index.xml`) and the driver's retries/tier
+  escalation repeated it (80–180 s per domain). Now:
+  - `SitemapParser._fetch` bounds the whole request by `SitemapParser.timeout` (covers driver retries and tier
+    escalation); a timeout is an answer "origin does not respond", not a reason to try again;
+  - the first timeout (no status + timeout error, incl. the driver's `tier timeout after …`) or CF 520–527 marks the
+    host `ORIGIN_DOWN` (`SitemapParser.mark_origin_down` / `origin_down_reason`); no further request to that host in
+    this sitemap phase — no default candidates, no nested sitemaps, no retries;
+  - `result["origin_down_reason"]` = `"timeout"` | `"cf_52x"`; `SitemapSpider.origin_down`,
+    `get_stats()["origin_down"]`, `SITEMAP_CRAWL_COMPLETED.data["origin_down"]`, and the robots.txt node gets
+    `error_message = "origin_down: <reason>"`. A spider-level `SITEMAP_PARSE_TIMEOUT` also marks the host down.
+  - Non-timeout network errors (DNS, connection refused) keep the old behaviour (default candidates are tried).
+  Live: daxx.com sitemap phase 80 s → 50 s (robots ~20 s + one 30 s timeout), healthy sites unchanged.
+- Tests: `tests/test_sitemap_origin_down_timeout.py` (8).
+
+## 5.3.5 — 2026-06
+
+### Fixed
+
+- **Dirty `Sitemap:` value in robots.txt no longer loses the sitemap (P19).** `SitemapParser._parse_robots_txt`
+  took the whole directive value verbatim (`line[8:].strip()`), so `Sitemap: sitemap URL https://site.com/sitemap_index.xml`
+  was glued into a broken address and the sitemap was lost. It now extracts the first `https?://\S+` from the value;
+  `urljoin` is applied only when the value is a relative path starting with `/`. New `SitemapParser._clean_sitemap_value`.
+- **A 404/410 is never a bot block (P19 side-finding).** `escalation._classify` returns `BotBlockStatus.NONE` for
+  status 404/410 right after the authoritative challenge headers. A dead sitemap candidate (or any not-found page) is
+  no longer classified as `captcha`/`challenge` and dragged through every tier (http→tls→browser). A real challenge
+  uses 403/503, not 404, so the authoritative header check still wins. Invariant: 404 is a definitive "not found"
+  from the server, not a page served instead of content. (After N3, verify & lock with a test — see Parser side.)
+- **A redirect inside the same ATS no longer blocks `domain_filter` (P21).** `CrawlCoordinator._handle_redirect_after_scan`
+  always allows a redirect to the **same host** (short ATS forms `apply.workable.com/j/X` → `apply.workable.com/<slug>/j/X`).
+  For a different host it still asks `domain_filter.is_allowed`, and when that refuses it consults a new host-alias
+  extension point `DomainFilter.allow_redirect(source, target)` (driven by a settable `redirect_host_alias` hook) — so
+  the Parser can pass its `DOMAIN_POLICIES` and allow `boards.greenhouse.io` → `job-boards.greenhouse.io` without
+  opening the whole internet.
+- **Sitemap under a Cloudflare 52x origin error stops without retries (1.6).** On the first `520`–`527` response to
+  robots.txt / sitemap, `SitemapParser` stops the sitemap phase and returns `result["origin_down"] = True` (logged as
+  `ORIGIN_DOWN`) instead of burning 100–180 s on retries/escalation against a down origin. New
+  `SitemapParser._is_origin_down`; `origin_down` is propagated through robots / default / nested sitemap paths.
+- **Nodes evicted by low-memory eviction are no longer lost in a browser full rescan (P23).**
+  `Graph.iter_nodes_async(include_evicted=True)` now also yields nodes that were evicted to disk (read-only, without
+  pulling them back into RAM), so a merge that iterates the source graph sees them. `Node.pinned` + `Graph.pin_node()`
+  / `Graph.pin_seed_nodes()` keep seed (`depth == 0`) vacancies in RAM — they were the oldest and got evicted first.
+  `Graph.eviction_stats` exposes the graph's own eviction counters (a backend-less graph evicts to LMDB/SQLite, which
+  `backend.get_stats()` reported as 0).
+- **A 403 storm without vendor markers is treated as a block (P24).** `EscalatingDriver` tracks a per-host window of
+  recent HTTP statuses; if a host that recently returned 2xx then returns 403 with no anti-bot markers and either ≥3
+  consecutive 403 or >30 % 403 in the window, the 403 is reclassified as `SOFT_BLOCK` (`escalation_reason=http_403_storm`)
+  and the tier escalates. A single 403 on a 2xx background is still treated as auth/method (no escalation). New
+  `_HostStatusWindow`.
+- **The HTML enrichment pool is priority-aware (2.1 / §15.4).** `run_in_html_pool(fn, *args, priority=...)` admits the
+  highest-priority waiter first when the pool is saturated (job-URLs and career subdomains enrich before catalog/blog
+  pages); with a free slot and empty queue it behaves exactly like the old FIFO semaphore. `Node` passes its URL-rule
+  priority. New `_PrioritySemaphore`.
+- **Cloudflare 520–527 is never a bot block.** `escalation._classify` returns `NONE` for 520–527 (as for 404/410):
+  the CF error page embeds `challenge-platform`, so a down origin was classified `challenge` and escalated
+  http→tls→browser (~20 s per tier) before `SitemapParser` could stop on `ORIGIN_DOWN`.
+- **robots.txt after a domain move keeps the new host's sitemap.** The same-origin gate and relative `Sitemap:`
+  paths now use the host that actually served robots.txt (`final_url`): `kilo.health` → `kilo.co/robots.txt` with
+  `Sitemap: … https://kilo.co/sitemap_index.xml` was dropped as cross-origin even after P19 cleaning.
+- **Cancelling a call waiting in the HTML-pool queue resumes the enrich budget.** With the priority semaphore the
+  acquire moved out of the `try`, so a cancellation in the queue left the node deadline suspended.
+- **`Node.pinned` does not leak into `user_data['_custom_fields']`** (`NodeMapper`, SQLite eviction storage base
+  fields) — it broke DTO/user-data invariants (`test_reserved_user_data_keys`, `test_node_mapper_content_depth`).
+- Tests: `tests/test_v535_fixes.py` (28, offline: P19, 404/52x, 1.6, P21 incl. end-to-end `async_crawl`, P23, P24, 2.1).
+
+### Added
+
+- `crawl` / `async_crawl` / `GraphCrawlerClient.crawl` / `CrawlerConfig`: `redirect_host_alias=` —
+  `Callable[[source_url, target_url], bool]` installed on the spider's `DomainFilter.redirect_host_alias`
+  (the extension point was otherwise unreachable through the public API).
+- `SitemapParser._clean_sitemap_value`, `SitemapParser._is_origin_down`; `result["origin_down"]` key.
+- `DomainFilter.allow_redirect(source_url, target_url)` and settable `DomainFilter.redirect_host_alias` hook.
+- `Node.pinned`; `Graph.pin_node`, `Graph.pin_seed_nodes`, `Graph.eviction_stats`; `iter_nodes_async(include_evicted=)`.
+- `run_in_html_pool(..., priority=)`.
+
+### Fixed (carried from Unreleased)
+
+- **Sitemap `<lastmod>` is no longer dropped.** `SitemapParser` reads `<lastmod>` of every `<url>` (with or without
+  namespace) into `result["lastmod"]` (`{url: value}`, merged across robots/default/nested sitemaps), and
+  `SitemapProcessor.create_url_nodes(lastmod=...)` stores it on the new `SitemapNode.lastmod` field (W3C datetime as
+  in the file). Consumers can now tell a 2021 post from a fresh page; `_parse_urlset` keeps its contract.
+
+- **An embedded captcha widget on a page with content is not a bot block.** `classify_block` follows the same
+  principle as the browser `CaptchaDetectorPlugin`: a block is a page served *instead of* content. After the
+  authoritative challenge headers (`cf-mitigated`, `x-amzn-waf-action`, `x-vercel-mitigated`), a non-blocking
+  response whose `<body>` has ≥1000 visible characters or ≥15 links is `NONE`, whatever widget or protection
+  script it embeds. Previously visible text was measured over the first 64 KB of the document (in WordPress that is
+  almost all `<head>`), and `cf-turnstile` / `challenge-platform` (Cloudflare loads its JSD script on regular pages)
+  were unconditional markers — so every page with reCAPTCHA/hCaptcha/Turnstile in a contact or apply form was
+  `captcha`, escalated HTTP→TLS and pinned the whole host to the TLS tier. `cf-turnstile`, `captcha-container`,
+  `challenge-platform`, `_incapsula_resource`, `awswafintegration` now count only on pages without content or with a
+  blocking status; markers and text are read from `<body>` up to 1 MB.
+
+## 5.3.4 — 2026-10
+
+### Fixed
+
+- **The per-node enrichment limit measures active work, not the HTML-pool queue.** `NodeScanner` wraps
+  `enrich_node` in `enrich_budget(seconds)` (`graph_crawler.domain.entities.node`): while a call waits for a thread
+  of the HTML executor in `run_in_html_pool` (parse, `_update_from_context`, hashes), the deadline is suspended and
+  resumed with the same remainder the moment the work starts. With 5.3.3 the hook and the hashes moved into the
+  shared pool, so under load a node's wall-clock depended on how many neighbours stood in front of it: a burst of
+  large pages produced a cascade of `Per-node timeout … marking as ERROR` for pages whose own processing took a
+  fraction of the limit (and every stage re-queued behind the others). A page whose **own** processing exceeds
+  `node_enrich_timeout` still times out and still returns its discovered links. The crawl-level timeout remains the
+  upper bound for a node that waits in the queue.
+- **`IsolatedExecutor.run` admits at most `max_workers` tasks at a time** (`threading.BoundedSemaphore`): the task
+  `timeout` now measures execution in the child process, not the wait behind other tasks. Previously a burst of
+  submissions could expire the timeout of a queued task, which killed the whole pool and failed every task in flight.
+
+- **Control commands are no longer dropped silently.** `GraphSpider.process_control_messages` executes only
+  `SPIDER_COMMANDS` (`STOP`, `PAUSE`, `RESUME`, `REPRIORITIZE`, `FORCE_VISIT`); any other `CrawlCommand` logs
+  `WARNING Unsupported control command '<value>' from <sender> is ignored (supported: ...)` once per command type
+  per crawl (repeats at `DEBUG`) and the crawl goes on. `REPRIORITIZE` without `url`/`priority` and `FORCE_VISIT`
+  without `url` log a `WARNING` with the received `data` instead of doing nothing.
+
+### Added
+
+- `EnrichBudget` (`pause()` / `resume()`) and `enrich_budget()` in `graph_crawler.domain.entities.node`.
+- `SPIDER_COMMANDS` (`graph_crawler.domain.interfaces`): the commands `GraphSpider` executes.
+
+### Deprecated
+
+- `CrawlCommand.SKIP_URL`, `CHANGE_STRATEGY`, `SET_DEPTH_LIMIT`, `REQUEST_STATE`, `UPDATE_GOAL` were never executed
+  by the spider. They will be removed from the enum in 5.4.0.
+
+## 5.3.3 — 2026-10
+
+### Added
+
+- **`IsolatedExecutor` / `IsolatedTaskError`** (`from graph_crawler import IsolatedExecutor`). Runs CPU-bound functions
+  in a `spawn` process pool: a native crash (SIGSEGV in libxml2/lxml, trafilatura, any C extension) or a hang of a
+  worker becomes `IsolatedTaskError(reason="crashed" | "timeout" | "unavailable")` in the caller, the pool restarts
+  and the crawl goes on. Tasks that shared the broken pool are retried once; a hung worker is killed after
+  `timeout` (default `DEFAULT_ISOLATED_TASK_TIMEOUT` = 30 s). Ordinary exceptions of the function propagate as is.
+- **`CrawlerConfig.node_enrich_timeout`** and `crawl(..., node_enrich_timeout=...)` / `async_crawl(...)` /
+  `GraphCrawlerClient.crawl(...)`: per-node enrichment limit (HTML parse, plugins, subclass hook, hashes).
+  `None` keeps `DEFAULT_NODE_ENRICH_TIMEOUT` (45 s, `shared.constants`). `NodeScanner(driver, enrich_timeout=...)`.
+- `run_in_html_pool(fn, *args)` (`graph_crawler.domain.entities.node`): runs `fn` in the HTML executor under its
+  semaphore. `Node.pop_discovered_links()` returns links extracted on `ON_HTML_PARSED` of an interrupted
+  `enrich_node`.
+- `URLRuleMatcher.rule_matches(rule, url)`.
+
+### Changed
+
+- **The `Node._update_from_context` hook and `get_content_hash`/`get_simhash` run in the HTML executor** under the
+  same semaphore as HTML parsing (they used the default `asyncio.to_thread` pool, `min(32, cpu+4)` threads, outside
+  the CPU budget). Concurrency of heavy subclass code is now bounded by `GRAPH_CRAWLER_HTML_PARSER_WORKERS`
+  (default — the cgroup CPU quota); `=1` serialises it completely. Code that is not thread-safe must use its own
+  lock or `GRAPH_CRAWLER_HTML_PARSER_WORKERS=1`; native code that can crash — `IsolatedExecutor`.
+- **One URL rule — one decision.** `CrawlScheduler`, `LinkProcessor` and `RuleBasedScheduler` share
+  `URLRuleMatcher`: first match in list order, `SmartURLRule` is matched against its `scope` (the scheduler used to
+  run `re.search` over the full URL, so `SmartURLRule(r"/help", scope=PATH, should_scan=False)` excluded
+  `https://helpware.com/`). `GraphSpider` no longer copies `url_rules` with `should_scan=False` into `PathFilter`
+  (it applied them to the path, ignoring order and scope); `excluded_paths` / `included_paths` are unchanged.
+- A URL rule whose `priority` is outside `[MIN_URL_PRIORITY, MAX_URL_PRIORITY]` (`[-100, 100]`) gets a WARNING when
+  the scheduler is created: `priority` is a queue position and is clamped; rule precedence is list order.
+- `IDriver.fetch(url, *, referer, extra_headers, cookies)` is the contract of every driver: `BaseDriver`,
+  `EscalatingDriver` (caller cookies/headers merged with the host session — the session wins on a cookie name
+  clash, the session `User-Agent` is applied on top for HTTP/TLS tiers), `PooledPlaywrightDriver` (`extra_headers`),
+  `IAsyncDriver`, `IBrowserDriver`. `BaseDriver.fetch_many` passes per-URL `extra_headers_list` / `cookies_list` to
+  `fetch` (they were dropped).
+
+### Fixed
+
+- **Per-node timeout no longer drops discovered links.** `scan_node` / `scan_batch` return the links extracted on
+  `ON_HTML_PARSED` when enrichment exceeds the limit; the node is still `ERROR` / `FAILED`. Large listing pages, the
+  first to hit the limit, keep their edges.
+
+## 5.3.2 — 2026-06
+
+### Added
+
+- **Fetch provenance on the node.** `Node.channel`, `bot_block_status`, `served_by_driver`, `served_by_engine`,
+  `escalations` and `fetch_info` read `metadata["fetch"]`; `Node.ok` (scanned, 2xx/3xx, not blocked, no error) and
+  `Node.is_blocked` separate useful pages from block and error pages.
+- **Plugin routing.** `BaseNodePlugin.domains` (domain and subdomains) and `url_pattern` (regex), on the class or via
+  `config={"domains": [...], "url_pattern": ...}`, plus an overridable `applies_to(url)`. An invalid `url_pattern`
+  raises `ConfigurationError` on `register()`. Several instances of one plugin class with different routes run
+  side by side (they used to replace each other).
+- **Typed plugin output.** `NodePluginContext.set_node_field(name, value)` writes a field declared by a `Node`
+  subclass, validated against its annotation (`"100"` → `int`); base fields are read-only. Fields are rolled back
+  together with `metadata`/`user_data` when the plugin fails.
+- **Empty extraction warning.** `BaseNodePlugin.expects_nonempty` + `extraction_key`: a WARNING when the plugin left
+  `user_data[extraction_key]` empty.
+- **`Graph.search(..., user_data_keys=("jobs",))`** searches the listed `user_data` keys as well.
+- **`Node.public_user_data`** and `RESERVED_USER_DATA_KEYS` (`_link_anchors`, `_custom_fields`, `_node_class`).
+
+### Changed
+
+- A response with status 400 or higher gets `metadata["error"] = "HTTP <code>"` and is logged as a WARNING
+  (`HTTP 404 for <url> — page not scanned`).
+- A bare 403/401/405 without vendor markers is still classified as `none`; the decision is now logged at INFO.
+- Quieter logs: import writes nothing at INFO, platform detection runs once, the start of a crawl is one line,
+  and each page is one INFO line with `[channel=…]`, `[block=…]`, `[error=…]`.
+- The temporary `_link_anchors` map is removed from the node once its edges are created (the anchor text stays in
+  `edge.metadata["anchor_text"]`). The default JSON export writes `public_user_data`.
+- A third-party plugin that overwrites a reserved `user_data` key gets a WARNING.
+
+### Fixed
+
+- `load_graph()` without `node_class=` restores a custom `Node` subclass when its module is already imported (the
+  class name is stored with the node; nothing is imported from the file). `Graph.copy()`, `union`, `intersection`,
+  `difference` and nodes created in a loaded graph keep the node class.
+- `metadata` written by a plugin on `ON_NODE_CREATED` or `ON_AFTER_SCAN` is no longer lost.
+- **Placeholder lists are detected as a JavaScript shell.** A 2xx body with five or more `placeholder` /
+  `skeleton` / `shimmer` elements makes `needs_browser` true, so `escalating` goes straight to the browser tier.
+  With `driver="http"` the node gets `metadata["js_shell_suspected"]` and a WARNING (once per host) suggests
+  `driver="escalating"` or `"playwright"`.
+- **No duplicate edges.** A link repeated on a page (menu + footer + body) created a new `Edge` each time
+  (17.5 % of edges on uk.wikipedia); `LinkProcessor` now skips existing `(source, target)` pairs. `Graph.has_edge`
+  also sees the backend edge index.
+- **`h1` children are no longer joined with spaces** (`Портал : Поточні події` → `Портал:Поточні події`).
+- **Default ports are normalized** (`http://x:80`, `https://x:443`, empty port).
+- `Graph.normalize_url()` is public; `LinkProcessor` checks the scheduler with the canonical form as well.
+- **`NetworkLogPlugin` waits for network quiet** before publishing the log (`settle_ms`=2000, `quiet_ms`=300;
+  `settle_ms=0` restores the old behaviour) — late XHR (search/listing after hydration) was lost.
+- `create_trailing_slash_rule` keeps the root path; an empty path is `/` for PATH-scope rules.
+
+## 5.3.1 — 2026-06
+
+### Added
+
+- **Hydration state globals snapshot.** The Playwright driver serialises `window.__NAME__` objects (`__NUXT__`,
+  `__APOLLO_STATE__`, `__INITIAL_STATE__`, `__PINIA__`, …) with `JSON.stringify` after the page has run and embeds
+  them into the HTML snapshot as `<script type="application/json" data-agc-state="__NAME__">`. Nuxt 2 ships its
+  state as an IIFE (`window.__NUXT__=(function(a,b){return {…}}(…))`) which is JavaScript, not JSON, and was
+  invisible to any static state parser. Globals that already exist as a JSON `<script id="__NAME__">`
+  (`__NEXT_DATA__`) are not duplicated; `PageFetcher.state_globals_max_chars` (5 MB) caps the snapshot.
+  `inject_state_globals(html, states)` is exported for tests and other drivers.
+
+## 5.3.0 — 2026-06
+
+### Added
+
+- **Page actions as public API.** `driver_config={"actions": [PageAction(...) | dict, ...]}` for the Playwright driver:
+  `click` / `fill` / `select` / `press` / `wait_for(response=…|selector=…)` / `scroll`, target by `selector`, `text`,
+  `placeholder` or `label` (`nth` among visible matches). Actions are validated when the driver is created and run
+  after content is ready and before the HTML snapshot and `CONTENT_READY` hooks — the network log is published after
+  the actions. Results land in `node.metadata["actions_log"]`, `["requests_after_actions"]` and `["actions_error"]`
+  (the first failing action stops the chain with a reason). `from graph_crawler import PageAction`.
+- `engine.page_error_types()` — page error classes of all installed browser SDKs.
+- `PageAction(type="back")`; action errors name the element that intercepts pointer events.
+
+### Fixed
+
+- Binary request bodies (gzip beacons) no longer raise `UnicodeDecodeError` inside Playwright event listeners
+  (`NetworkLogPlugin` and the action recorder read `post_data_buffer` with `errors="replace"`); the error used to break
+  the next Playwright call with `TypeError: function takes exactly 5 arguments`.
+
+### Fixed
+
+- `graph.get_node_by_url(seed)` finds the node after a redirect in the graph returned by `async_crawl`: `add_node`
+  restores the alias from `metadata["was_redirected_from"]`.
+
+## 5.2.2 — 2026-06
+
+### Fixed (test infrastructure)
+
+- **The library no longer configures logging.** `GraphCrawlerClient.__init__` called `logging.basicConfig(INFO)`
+  when its logger had no handlers, so every crawl printed to stderr and reconfigured the application's root
+  logger. Removed; `graph_crawler` gets a `NullHandler`.
+- **Noise demoted:** `HTML executor shutdown` and `lmdb not installed … Falling back to SQLiteEvictionStorage` are
+  DEBUG; `channel='chrome' not installed, falling back to bundled Chromium` is a WARNING once per process.
+- **One version source.** `graph_crawler.__version__` (and the default User-Agent) come from the installed
+  package metadata (`pyproject.toml`); `__version__.py` no longer hard-codes a number that drifted (5.1.0 vs 5.2.0).
+- **Test suite collects without optional extras.** Tests that need `psutil`, `rapidfuzz`, `stop-words`,
+  `html2md-clean` or `playwright` are skipped via `importorskip` instead of failing collection under `-x`.
+  `test_div_inline_fix.py` / `test_phase1_fixes.py` tested `html2md_clean` itself (no `graph_crawler` code) and were
+  removed from the core suite.
+- GitLab CI (`.gitlab-ci.yml`): offline suite with `[dev]` and a minimal job without extras.
+
+## 5.2.1 — 2026-06
+
+### Fixed (freelance cases)
+
+- **Captured XHR/fetch traffic could not be replayed outside the browser.** `NetworkLogPlugin` recorded the
+  URL and method of a request but not its body, so every POST/GraphQL endpoint (robota.ua lists vacancies via
+  `POST dracula.robota.ua/?q=getPublishedVacanciesList`) was unusable: a GET of the same URL returns 400.
+  Records now carry `post_data` and (on `response`) `request_headers`.
+- **GraphQL responses overwrote each other.** `body_ref` was `sha1(url|status|content-type)` — all operations
+  POSTed to one `/graphql` URL shared a key and only the last body survived. The key now includes method and
+  request body.
+- **JSON family by default.** The capture allow-list matched exact MIME types only; `application/graphql-
+response+json`, `problem+json`, `vnd.api+json`, `text/json` were dropped. Without an explicit
+  `capture_content_types` every `*/json` and `*+json` type is captured (RFC 6839). Default `max_body_kb` 64 → 256
+  (listing APIs routinely exceed 64 KB).
+- **Shadow DOM was invisible.** `page.content()` serializes only light DOM: content of open shadow roots (web
+  components, embedded price widgets) never reached the node. When a page has open shadow roots, the browser
+  driver serializes it with `Element.getHTML({shadowRoots})` and expands the declarative shadow roots into their
+  hosts (`shared/utils/shadow_dom.expose_shadow_roots`) — Lexbor, like the HTML spec, does not descend into
+  `<template>` content, so a plain declarative-shadow-DOM serialization would still hide it from selectors and link
+  extraction. Pages without shadow roots are untouched.
+- `tests/unit/domain/entities/test_graph_new_features.py` still expected the `DeprecationWarning` that
+  `export_nodes()` stopped emitting in 4aab7525; it now asserts the opposite.
+
+### Fixed (real-site tests)
+
+- **Ctrl+C during `crawl()` hung until the crawl-level timeout** (28 s with `timeout=25`; forever without
+  one). `asyncio.wait_for` in Python 3.11 swallows the outer cancellation when the awaited coroutine finishes
+  at the same moment, so some workers survived `KeyboardInterrupt` and kept crawling. `NodeScanner` and the
+  escalation tier budget now use `asyncio.timeout()`; `CrawlCoordinator` cancels its workers on
+  `CancelledError`. Ctrl+C now returns in ~0.3 s.
+- **`same_domain=False` was ignored** — without `allowed_domains` the `DomainFilter` default
+  (`domain+subdomains`) still applied and external links never entered the graph. Now maps to `allowed_domains=["*"]`.
+- **`graph.metadata["stop_reason"]` was `None` after `timeout=` / `max_pages`** — only stop conditions were
+  recorded. Now `"timeout"`, `"max_pages"` or `"limit"`.
+- **Network errors were invisible on the node**: a DNS/connection failure left `response_status=None`,
+  `scanned=True` and no message; `get_failed_nodes()` returned `[]`. The error text is now in
+  `node.metadata["error"]`, the HTTP status (if any) is kept, and `get_failed_nodes()` with default
+  arguments includes network-error nodes.
+- **`storage="json", storage_config={"path": "site.json"}` was silently ignored** (file written to
+  `crawler_data/json/graph.json`). `path` is now honoured, like for sqlite (`JSONStorage(file_name=)`).
+- `Graph.export_nodes()` / `Graph.export_edges()` are no longer deprecated: they are the public one-liners the
+  README shows and now delegate to `GraphExportUseCase` silently (no `DeprecationWarning`).
+- `tests/manual/real_sites/run_all.py -j N` runs scenarios in parallel; `tests/manual/real_sites/test_user_cases.py`
+  — 23 pytest user scenarios on Wikipedia / robota.ua / GitHub / example.com, each hard-capped at 40 s.
+
+### Changed
+
+- **Engine contract verified end-to-end**. `tests/transport/
+test_engine_contract.py` drives the real `PlaywrightDriver.fetch()` through three engines registered via the
+  public `register_engine()` on a recording fake SDK: the call sequence is byte-for-byte identical across
+  engines, stealth is injected iff `provides_stealth is False`, and `executable_path`/`ignore_default_args`/
+  launch args come only from the engine. No runtime code changed.
+- **`LexborDocument.clean_text()` no longer re-parses the page**. To strip `script/style/noscript/svg/head` without mutating the shared tree it
+  serialised the document and built a second `LexborHTMLParser` — i.e. every page with the default
+  `TextExtractorPlugin` was parsed twice (`count_parses.py`: 2.0 → **1.0** per page; with `MarkdownPlugin`
+  1 parse + 1 clone). It now uses `tree.clone()`.
+- **One `ContentDecoder` for every HTTP tier**. `AsyncDriver` (aiohttp) and `AsyncCurlCffiDriver` (curl_cffi) no longer rely on the client's own
+  `response.text`/`.text()`: they read the raw body and decode it through `ContentDecoder`
+  (`Content-Type` charset → `<meta charset>` → BOM → chardet → strict utf-8). Pages whose charset is only
+  declared in `<meta>` (e.g. windows-1251 without a header) were previously decoded as utf-8 by aiohttp,
+  failed, and silently became `html=None` with a "successful" 200. An undecodable body is now reported as
+  `FetchResponse.error` (`"ContentDecoder: cannot decode body …"`) with the status code preserved;
+  `ConnectionPoolManager.fetch_text()` goes through the same decoder. `FetchResponse.html` is a strict
+  `str` field — passing `bytes` raises `ValidationError` instead of being coerced. Browser tiers
+  (Playwright `page.content()`, nodriver `tab.get_content()`) already return `str`. Contract test:
+  `tests/transport/test_html_is_str_contract.py` (driver matrix returns `str`; cp1251 via meta/header;
+  binary → error).
+- **Lazy public facade**. `graph_crawler/__init__.py` now
+  eagerly imports only `crawl`, `async_crawl`, `Graph`, `Node`, `save_graph`, `load_graph`,
+  `BaseStopCondition`; every other name in `__all__` (drivers, storages, exceptions, settings, views,
+  celery tasks, …) resolves on first access via module `__getattr__` and is cached. `aiohttp` is no longer
+  loaded by `import graph_crawler`. `bootstrap()` no longer calls `collect_system_metrics()`
+  (`psutil.cpu_percent(interval=0.1)` blocked import for 100 ms for a DEBUG line). Import time here:
+  470 ms → 228 ms. Optional names (`PlaywrightDriver`, `celery`, `EasyDistributedCrawler`, …) still return
+  `None` without their extra.
+- **Escalation re-probe replays the solving tier's User-Agent**.
+  Playwright and nodriver now export `navigator.userAgent` as `extras["user_agent"]`;
+  `SessionState.absorb` records it (pinned once the session is solved, like `impersonate`/`proxy`) and
+  `EscalatingDriver._fetch_tier` sends it as `extra_headers={"User-Agent": …}` to HTTP/TLS tiers together
+  with the stored cookies. `cf_clearance` is bound to UA+IP+JA4, so a re-probe on Tier 1 with aiohttp's
+  default UA was rejected and the host never descended from the browser tier.
+
+### Removed
+
+- **`NodePluginContext.parser`**. Node plugins now read the
+  single Lexbor tree through the `HTMLDocument`/`HTMLNode` protocols: `LinkExtractorPlugin`
+  (`doc.css("a[href]")`, `doc.base_href()`), `MetadataExtractorPlugin`, `TextExtractorPlugin`
+  (`doc.clean_text()`, new in the protocol), `StructuredDataPlugin` (OG/Twitter/Microdata/RDFa parsers take
+  an `HTMLDocument`; `StructuredDataExtractor.extract(html=, doc=)`), `Email`/`PhoneExtractorPlugin`.
+  Migration: `context.parser.find_all(sel)` → `context.html_tree.css(sel)`, `.find()` → `.css_first()`,
+  `el.get_attribute()` → `el.attr()`. Fixed along the way in `LexborNode`: `attr()` returns `""` (not
+  `None`) for valueless boolean attributes such as `itemscope`, so nested Microdata/RDFa items are now
+  detected; `css()`/`css_first()` return descendants only (selectolax included the node itself, which made
+  Microdata recurse to `max_nesting_depth`).
+- **`PooledPlaywrightDriver` legacy batch path**: `batch_legacy.py`
+  (`LegacyBatch._fetch_batch_legacy`, `_fetch_chain`) is gone. `persistent_browser=False` keeps its
+  "launch/stop per batch" semantics through the pool path: after a batch, once no fetch is in flight, the
+  pool and the engine session are closed (`_close_pool()` + `_stop_persistent_session()`, shared with
+  `close()`). Dead `self.playwright`/`self.browsers` cleanup removed from `close()`.
+- **Escalation.** `classify_block()` rewritten to the
+  6.0 decision table: challenge/captcha vendor headers first, then unambiguous body captcha/challenge
+  markers (ahead of vendor hard-block headers), then short/blocking-page widgets, status-based verdicts
+  and empty-2xx. A bare `401/403/405` with no vendor markers is no longer a block. On the classifier corpus
+  it goes 25/37 → **37/37**.
+
+  - `needs_browser` heuristic (SPA shell: little visible text + SPA root / `<noscript>` JS hint / no
+    `<a href>`) is now the default `EscalationPolicy.needs_browser`; on a clean HTTP/TLS response it jumps
+    straight to the first browser tier (`escalation_reason="needs_browser"`).
+  - Per-tier `TierTimeouts` + `escalation_deadline_s` (60s) budget: a tier is skipped (last response
+    returned, browser never started) when the remaining budget is below its `min_budget`; each tier fetch
+    is wrapped in `asyncio.wait_for`. New `PER_NODE_FETCH_TIMEOUT` constant with a validator warning.
+  - Cross-tier session: `SessionState` / `HostSessionStore` carry solved cookies (`cf_clearance`, `__cf_bm`,
+    `datadome`, …) forward and back down to cheaper tiers on re-probe; `impersonate`/`proxy` are pinned once
+    a session is solved. curl_cffi pins the impersonate fingerprint; Playwright/nodriver export context
+    cookies via `extras["cookies"]`.
+  - `extras` contract: every driver stamps `channel`/`driver`/`engine` (via `BaseDriver`/`BaseAsyncDriver`
+    `_stamp_extras` + `_finalize_response`) even outside escalation.
+  - New Prometheus metrics `crawl_fetch_duration_seconds{channel,engine}`, `crawl_escalations_total`,
+    `crawl_escalation_deadline_hits_total`.
+
+- **HTML parsing: Lexbor (selectolax) is the only core parser**.
+  `beautifulsoup4`, `lxml`, `lxml_html_clean` removed from `[project].dependencies`;
+  `BeautifulSoupAdapter`, `LxmlAdapter`, `ScrapyAdapter` deleted. New `HTMLDocument` / `HTMLNode`
+  protocols in `domain/value_objects/html_document.py`, implemented by `LexborDocument` in
+  `infrastructure/adapters/selectolax_adapter.py`. `<template>` is renamed to `gc-template` before
+  parsing so Vue/React SSR links inside templates are not lost. `IParser.parse()` returns `HTMLDocument`;
+  `NodePluginContext.html_tree` is now a `LexborDocument`. `get_parser()` accepts `auto|selectolax|lexbor`.
+- `HTMLUtils.extract_links/extract_text/extract_metadata` accept an `HTMLDocument` (string input still
+  works but is deprecated).
+- `SitemapParser.parse_large_sitemap_streaming()` uses stdlib `iterparse` by default; lxml is an
+  optional fast-path via the new extra `sitemap-fast`.
+
+- **`html2md-clean` moved to the optional extra `markdown`** (`pip install "graph-crawler[markdown]"`);
+  `MarkdownGenerator/MarkdownOptions/MarkdownResult` re-exports in `graph_crawler.shared.utils` are lazy.
+  `MarkdownPlugin` feeds the already-parsed Lexbor tree to html2md-clean 0.5 (`generate_from_tree`) —
+  one parse per page instead of 2–4. `import graph_crawler` no longer loads bs4/lxml.
+
+- **Dependency map**. Core is now 10 packages: `requests`,
+  `httpx`, `pydantic-settings`, `pybloom-live`, `fastapi`, `tenacity`, `stop-words`, `tiktoken`,
+  `markdown-it-py` removed from `[project].dependencies`. New extras: `browser` (`playwright` kept as alias),
+  `server` (fastapi/uvicorn/websockets), `text` (tiktoken/stop-words), `captcha`; `markdown` also pulls
+  `markdown-it-py`; `all` is recursive. Extras `cloudscraper` and `undetected-sync` dropped.
+  Extra-gated imports are lazy and fail fast with `requires extra '<name>'`; `transport.PlaywrightDriver/
+RequestsDriver/CloudscraperDriver/RequestsSessionAdapter` are lazy re-exports (ImportError instead of `None`).
+  `CrawlerSettings` keeps `GC_*`/`__`/`.env` without pydantic-settings; `CrawlerConfig` is a plain `BaseModel`.
+
+- **Transport cleanup**. Deleted: sync drivers
+  (`RequestsDriver`, `CloudscraperDriver`, `UndetectedChromeDriver`, `BotasaurusDriver`, `BaseSyncDriver`),
+  `AsyncDriverHTTP2` (httpx), `AsyncCloudscraperDriver`, `tls_client.py`, `StealthHTTPDriver`,
+  `session_manager/session_adapters/session_protocol`, `shared/utils/captcha_bypass.py`,
+  `shared/utils/proxy_manager.py`, packages `transport/http` and `transport/browser`.
+  `DriverType` is now `aiohttp | tls | playwright | nodriver | escalating` (+ `asyncio` alias);
+  strings `http`/`async`/`asyncio` → `aiohttp` and `stealth`/`curl_cffi` → `tls` are accepted for one
+  release with `DeprecationWarning`. One `AsyncDriver` (former `driver_adaptive.py` is now
+  `async_http/driver.py`); request hints (`referer`, `extra_headers`, `cookies`) are honoured.
+  `requests` is gone from the package: proxy health-check, CAPTCHA solvers/bypass and `SitemapParser`
+  use aiohttp (`SitemapParser.parse_sitemap/parse_from_robots` are `asyncio.run` wrappers,
+  `http_client=` is deprecated and ignored). `BloomFilter` is pure-python (xxhash) — `pybloom-live`
+  no longer needed.
+  `import graph_crawler` no longer loads any extra (guarded by `tests/unit/test_import_surface.py`), 3.8 s → 1.7 s here.
+  `shared/constants.py` no longer imports aiohttp — `br`/`zstd` are advertised
+  by `transport.compression.runtime_browser_headers()` only when the runtime can decode them;
+  one driver registry — `application.services.driver_factory` delegates to
+  `transport.factory.DriverFactory` (`register_factory`, `registered_names`); `SitemapParser(driver=)`
+  fetches robots.txt/sitemaps through the crawl `IDriver` (same tier chain), `SitemapSpider` passes its
+  driver, aiohttp is used directly only for binary `.gz`; `tests/transport/test_all_drivers_report_redirects.py`
+  is an `IDriver` acceptance matrix (aiohttp / tls / fake playwright / fake nodriver / escalating).
+
+- **Escalation**: `EscalationPolicy.reprobe_interval` (5 min) —
+  one request per interval re-probes tier 0 for a host remembered on a higher tier (no herd of browser
+  re-escalations after TTL); `EscalationPolicy.max_hosts` (10k) — `HostTierMemory` is an LRU and drops
+  per-host locks together with entries. In strict mode (`GC_ENGINE_STRICT`) an `ImportError` raised _during_
+  `fetch` (lazy browser start without the SDK) now propagates instead of being turned into a network error.
+
+- **`extras` contract persisted.** New
+  `FetchResponse.fetch_provenance()` / `FETCH_EXTRAS_KEYS`: a JSON-safe slice
+  (`bot_block_status, channel, driver, engine, escalations` + `escalation_reason`; cookies/impersonate dropped).
+  `NodeScanner` stores it in `node.metadata["fetch"]` for every response, including blocked/error ones, so it
+  survives JSON/SQLite/MongoDB/PostgreSQL storage. Contract test `tests/transport/test_extras_contract.py`.
+
+### Fixed
+
+- Transport `extras` (`bot_block_status`, tier, escalations) were dropped after fetch and never reached the
+  node or storage; for blocked pages nothing was transferred at all.
+- `SmartPageFinder` read `html_tree.text` as a property; with Lexbor it is a method — page text was
+  silently dropped.
+- `AsyncUndetectedDriver` accepts a plain `dict` config (`AsyncUndetectedConfig.from_mapping`) — building
+  the `nodriver` tier through `DriverFactory.create(DriverType.NODRIVER, config={...})` (as `EscalatingDriver`
+  does) raised `AttributeError`.
+
+## 5.2.0 — 2026-06
+
+### Added
+
+- `infrastructure/transport/escalation.py`: `classify_block()` (Cloudflare / Akamai / DataDome /
+  PerimeterX / Incapsula markers, 403/429/503, empty 200), `EscalationPolicy`, `EscalationTier`,
+  `EscalatingDriver` — chain http → tls → browser → undetected with `max_tier`.
+  `driver="escalating"` works in `crawl()` / `async_crawl()` / `Crawler` / CLI.
+- `HostTierMemory`: adaptive per-host tier memory — TTL from the moment of decision
+  (`GC_TIER_TTL`, default 1200 s), invalidation when the remembered tier gets blocked,
+  one escalation per host at a time (per-host lock). Per process.
+- Strict engine mode: `GC_ENGINE_STRICT=1`, `{"browser_engine": {"strict": True}}` or
+  `driver_config={"engine_strict": True}` — `ImportError` instead of a silent fallback
+  `cloakbrowser → patchright → playwright` or a skipped escalation tier.
+- Browser engine registry in `playwright/engine.py`: `register_engine()`, `resolve_engine()`,
+  `registered_engines()`. Built-in engines `cloakbrowser` → `patchright` → `playwright` with
+  automatic fail-open. `GC_BROWSER_ENGINE` env selects the engine when config has none.
+  `PlaywrightBrowserEngine.effective_name` / `fallback_reason`.
+- `FetchResponse.extras` contract: `bot_block_status`, `channel`, `driver`, `engine`, `escalations`.
+- `DriverType.TLS`, `NODRIVER`, `UNDETECTED`, `BOTASAURUS`, `ESCALATING`; `DriverFactory.create()`
+  accepts extra constructor kwargs (`policy=`).
+- Extras: `tls`, `stealth`, `cloak`, `undetected`, `undetected-sync`, `antidetect`, `cloudscraper`, `system`.
+- `README.md`, `CHANGELOG.md`.
+
+### Changed
+
+- `normalize_protocol` defaults to `False`: `http://` and `https://` are different URLs.
+  The crawler follows the real redirect instead of rewriting the scheme itself, so
+  http-only hosts stay reachable. Enable it explicitly to merge both schemes on dedup.
+- `Graph.handle_redirect()` stores the alias as the real `Location` the server sent
+  (provenance); the normalized form is used only to find and merge nodes.
+- Single `DriverType` enum (`transport/base.py`); `factory.py` re-exports it.
+- `cloudscraper` moved from required dependencies to extra `cloudscraper`.
+- `psutil` is optional (`system` extra); `MemoryGuard` imports it lazily.
+- `requires-python = ">=3.11"` (upper bound removed; classifiers 3.11–3.14).
+- `pytest` runs offline by default (`-m 'not manual'`, no coverage in `addopts`).
+  Network / real-site tests live in `tests/manual/` and run with `pytest -m manual tests/manual`.
+- Project URLs point to the GitLab repository.
+
+### Fixed
+
+- `pooled_driver`: bounded walk over `redirected_from` (infinite loop on objects without a
+  `None` terminator); cookies are no longer added twice to a fresh browser.
+- Tests synced with the code: redirect aliases, `html2md-clean` 0.4.0 API, `crawl_coordinator`
+  `per_host_max_in_flight`, `.gitignore` lookup at the repository root, `tools/db_cleanup.py`
+  CLI allow-listed for `print`. Playwright fake tests skip when `playwright` is not installed.
+
+### Removed
+
+- `tests/345.py`, `tests/ДОДАТКОВІ_ТЕСТИ.py`, `tests/manual_4a_games_test.py`, `tests/google_maps_results.json`.
+- `EXTRACTION_AUDIT.md` and other analysis / plan documents that described work, not the code.
+- `graph_crawler/data/ARCHITECTURE.md`, `MIGRATION_50_STEPS.md`, `REFACTORING_PLAN.md`, `mkdocs.yml`
+  (finished plans / config without content).
+
+## 5.1.0
+
+Previous release (sync-first API, Playwright pool, adaptive HTTP drivers, storage backends).