graph-crawler 5.3.2__tar.gz → 5.3.4__tar.gz

graph_crawler-5.3.4/CHANGELOG.md

@@ -0,0 +1,429 @@
+# Changelog
+
+## 5.3.4 — 2026-10
+
+### Fixed
+
+- **The per-node enrichment limit measures active work, not the HTML-pool queue.** `NodeScanner` wraps
+  `enrich_node` in `enrich_budget(seconds)` (`graph_crawler.domain.entities.node`): while a call waits for a thread
+  of the HTML executor in `run_in_html_pool` (parse, `_update_from_context`, hashes), the deadline is suspended and
+  resumed with the same remainder the moment the work starts. With 5.3.3 the hook and the hashes moved into the
+  shared pool, so under load a node's wall-clock depended on how many neighbours stood in front of it: a burst of
+  large pages produced a cascade of `Per-node timeout … marking as ERROR` for pages whose own processing took a
+  fraction of the limit (and every stage re-queued behind the others). A page whose **own** processing exceeds
+  `node_enrich_timeout` still times out and still returns its discovered links. The crawl-level timeout remains the
+  upper bound for a node that waits in the queue.
+- **`IsolatedExecutor.run` admits at most `max_workers` tasks at a time** (`threading.BoundedSemaphore`): the task
+  `timeout` now measures execution in the child process, not the wait behind other tasks. Previously a burst of
+  submissions could expire the timeout of a queued task, which killed the whole pool and failed every task in flight.
+
+- **Control commands are no longer dropped silently.** `GraphSpider.process_control_messages` executes only
+  `SPIDER_COMMANDS` (`STOP`, `PAUSE`, `RESUME`, `REPRIORITIZE`, `FORCE_VISIT`); any other `CrawlCommand` logs
+  `WARNING Unsupported control command '<value>' from <sender> is ignored (supported: ...)` once per command type
+  per crawl (repeats at `DEBUG`) and the crawl goes on. `REPRIORITIZE` without `url`/`priority` and `FORCE_VISIT`
+  without `url` log a `WARNING` with the received `data` instead of doing nothing.
+
+### Added
+
+- `EnrichBudget` (`pause()` / `resume()`) and `enrich_budget()` in `graph_crawler.domain.entities.node`.
+- `SPIDER_COMMANDS` (`graph_crawler.domain.interfaces`): the commands `GraphSpider` executes.
+
+### Deprecated
+
+- `CrawlCommand.SKIP_URL`, `CHANGE_STRATEGY`, `SET_DEPTH_LIMIT`, `REQUEST_STATE`, `UPDATE_GOAL` were never executed
+  by the spider. They will be removed from the enum in 5.4.0.
+
+## 5.3.3 — 2026-10
+
+### Added
+
+- **`IsolatedExecutor` / `IsolatedTaskError`** (`from graph_crawler import IsolatedExecutor`). Runs CPU-bound functions
+  in a `spawn` process pool: a native crash (SIGSEGV in libxml2/lxml, trafilatura, any C extension) or a hang of a
+  worker becomes `IsolatedTaskError(reason="crashed" | "timeout" | "unavailable")` in the caller, the pool restarts
+  and the crawl goes on. Tasks that shared the broken pool are retried once; a hung worker is killed after
+  `timeout` (default `DEFAULT_ISOLATED_TASK_TIMEOUT` = 30 s). Ordinary exceptions of the function propagate as is.
+- **`CrawlerConfig.node_enrich_timeout`** and `crawl(..., node_enrich_timeout=...)` / `async_crawl(...)` /
+  `GraphCrawlerClient.crawl(...)`: per-node enrichment limit (HTML parse, plugins, subclass hook, hashes).
+  `None` keeps `DEFAULT_NODE_ENRICH_TIMEOUT` (45 s, `shared.constants`). `NodeScanner(driver, enrich_timeout=...)`.
+- `run_in_html_pool(fn, *args)` (`graph_crawler.domain.entities.node`): runs `fn` in the HTML executor under its
+  semaphore. `Node.pop_discovered_links()` returns links extracted on `ON_HTML_PARSED` of an interrupted
+  `enrich_node`.
+- `URLRuleMatcher.rule_matches(rule, url)`.
+
+### Changed
+
+- **The `Node._update_from_context` hook and `get_content_hash`/`get_simhash` run in the HTML executor** under the
+  same semaphore as HTML parsing (they used the default `asyncio.to_thread` pool, `min(32, cpu+4)` threads, outside
+  the CPU budget). Concurrency of heavy subclass code is now bounded by `GRAPH_CRAWLER_HTML_PARSER_WORKERS`
+  (default — the cgroup CPU quota); `=1` serialises it completely. Code that is not thread-safe must use its own
+  lock or `GRAPH_CRAWLER_HTML_PARSER_WORKERS=1`; native code that can crash — `IsolatedExecutor`.
+- **One URL rule — one decision.** `CrawlScheduler`, `LinkProcessor` and `RuleBasedScheduler` share
+  `URLRuleMatcher`: first match in list order, `SmartURLRule` is matched against its `scope` (the scheduler used to
+  run `re.search` over the full URL, so `SmartURLRule(r"/help", scope=PATH, should_scan=False)` excluded
+  `https://helpware.com/`). `GraphSpider` no longer copies `url_rules` with `should_scan=False` into `PathFilter`
+  (it applied them to the path, ignoring order and scope); `excluded_paths` / `included_paths` are unchanged.
+- A URL rule whose `priority` is outside `[MIN_URL_PRIORITY, MAX_URL_PRIORITY]` (`[-100, 100]`) gets a WARNING when
+  the scheduler is created: `priority` is a queue position and is clamped; rule precedence is list order.
+- `IDriver.fetch(url, *, referer, extra_headers, cookies)` is the contract of every driver: `BaseDriver`,
+  `EscalatingDriver` (caller cookies/headers merged with the host session — the session wins on a cookie name
+  clash, the session `User-Agent` is applied on top for HTTP/TLS tiers), `PooledPlaywrightDriver` (`extra_headers`),
+  `IAsyncDriver`, `IBrowserDriver`. `BaseDriver.fetch_many` passes per-URL `extra_headers_list` / `cookies_list` to
+  `fetch` (they were dropped).
+
+### Fixed
+
+- **Per-node timeout no longer drops discovered links.** `scan_node` / `scan_batch` return the links extracted on
+  `ON_HTML_PARSED` when enrichment exceeds the limit; the node is still `ERROR` / `FAILED`. Large listing pages, the
+  first to hit the limit, keep their edges.
+
+## 5.3.2 — 2026-06
+
+### Added
+
+- **Fetch provenance on the node.** `Node.channel`, `bot_block_status`, `served_by_driver`, `served_by_engine`,
+  `escalations` and `fetch_info` read `metadata["fetch"]`; `Node.ok` (scanned, 2xx/3xx, not blocked, no error) and
+  `Node.is_blocked` separate useful pages from block and error pages.
+- **Plugin routing.** `BaseNodePlugin.domains` (domain and subdomains) and `url_pattern` (regex), on the class or via
+  `config={"domains": [...], "url_pattern": ...}`, plus an overridable `applies_to(url)`. An invalid `url_pattern`
+  raises `ConfigurationError` on `register()`. Several instances of one plugin class with different routes run
+  side by side (they used to replace each other).
+- **Typed plugin output.** `NodePluginContext.set_node_field(name, value)` writes a field declared by a `Node`
+  subclass, validated against its annotation (`"100"` → `int`); base fields are read-only. Fields are rolled back
+  together with `metadata`/`user_data` when the plugin fails.
+- **Empty extraction warning.** `BaseNodePlugin.expects_nonempty` + `extraction_key`: a WARNING when the plugin left
+  `user_data[extraction_key]` empty.
+- **`Graph.search(..., user_data_keys=("jobs",))`** searches the listed `user_data` keys as well.
+- **`Node.public_user_data`** and `RESERVED_USER_DATA_KEYS` (`_link_anchors`, `_custom_fields`, `_node_class`).
+
+### Changed
+
+- A response with status 400 or higher gets `metadata["error"] = "HTTP <code>"` and is logged as a WARNING
+  (`HTTP 404 for <url> — page not scanned`).
+- A bare 403/401/405 without vendor markers is still classified as `none`; the decision is now logged at INFO.
+- Quieter logs: import writes nothing at INFO, platform detection runs once, the start of a crawl is one line,
+  and each page is one INFO line with `[channel=…]`, `[block=…]`, `[error=…]`.
+- The temporary `_link_anchors` map is removed from the node once its edges are created (the anchor text stays in
+  `edge.metadata["anchor_text"]`). The default JSON export writes `public_user_data`.
+- A third-party plugin that overwrites a reserved `user_data` key gets a WARNING.
+
+### Fixed
+
+- `load_graph()` without `node_class=` restores a custom `Node` subclass when its module is already imported (the
+  class name is stored with the node; nothing is imported from the file). `Graph.copy()`, `union`, `intersection`,
+  `difference` and nodes created in a loaded graph keep the node class.
+- `metadata` written by a plugin on `ON_NODE_CREATED` or `ON_AFTER_SCAN` is no longer lost.
+- **Placeholder lists are detected as a JavaScript shell.** A 2xx body with five or more `placeholder` /
+  `skeleton` / `shimmer` elements makes `needs_browser` true, so `escalating` goes straight to the browser tier.
+  With `driver="http"` the node gets `metadata["js_shell_suspected"]` and a WARNING (once per host) suggests
+  `driver="escalating"` or `"playwright"`.
+- **No duplicate edges.** A link repeated on a page (menu + footer + body) created a new `Edge` each time
+  (17.5 % of edges on uk.wikipedia); `LinkProcessor` now skips existing `(source, target)` pairs. `Graph.has_edge`
+  also sees the backend edge index.
+- **`h1` children are no longer joined with spaces** (`Портал : Поточні події` → `Портал:Поточні події`).
+- **Default ports are normalized** (`http://x:80`, `https://x:443`, empty port).
+- `Graph.normalize_url()` is public; `LinkProcessor` checks the scheduler with the canonical form as well.
+- **`NetworkLogPlugin` waits for network quiet** before publishing the log (`settle_ms`=2000, `quiet_ms`=300;
+  `settle_ms=0` restores the old behaviour) — late XHR (search/listing after hydration) was lost.
+- `create_trailing_slash_rule` keeps the root path; an empty path is `/` for PATH-scope rules.
+
+## 5.3.1 — 2026-06
+
+### Added
+
+- **Hydration state globals snapshot.** The Playwright driver serialises `window.__NAME__` objects (`__NUXT__`,
+  `__APOLLO_STATE__`, `__INITIAL_STATE__`, `__PINIA__`, …) with `JSON.stringify` after the page has run and embeds
+  them into the HTML snapshot as `<script type="application/json" data-agc-state="__NAME__">`. Nuxt 2 ships its
+  state as an IIFE (`window.__NUXT__=(function(a,b){return {…}}(…))`) which is JavaScript, not JSON, and was
+  invisible to any static state parser. Globals that already exist as a JSON `<script id="__NAME__">`
+  (`__NEXT_DATA__`) are not duplicated; `PageFetcher.state_globals_max_chars` (5 MB) caps the snapshot.
+  `inject_state_globals(html, states)` is exported for tests and other drivers.
+
+## 5.3.0 — 2026-06
+
+### Added
+
+- **Page actions as public API.** `driver_config={"actions": [PageAction(...) | dict, ...]}` for the Playwright driver:
+  `click` / `fill` / `select` / `press` / `wait_for(response=…|selector=…)` / `scroll`, target by `selector`, `text`,
+  `placeholder` or `label` (`nth` among visible matches). Actions are validated when the driver is created and run
+  after content is ready and before the HTML snapshot and `CONTENT_READY` hooks — the network log is published after
+  the actions. Results land in `node.metadata["actions_log"]`, `["requests_after_actions"]` and `["actions_error"]`
+  (the first failing action stops the chain with a reason). `from graph_crawler import PageAction`.
+- `engine.page_error_types()` — page error classes of all installed browser SDKs.
+- `PageAction(type="back")`; action errors name the element that intercepts pointer events.
+
+### Fixed
+
+- Binary request bodies (gzip beacons) no longer raise `UnicodeDecodeError` inside Playwright event listeners
+  (`NetworkLogPlugin` and the action recorder read `post_data_buffer` with `errors="replace"`); the error used to break
+  the next Playwright call with `TypeError: function takes exactly 5 arguments`.
+
+### Fixed
+
+- `graph.get_node_by_url(seed)` finds the node after a redirect in the graph returned by `async_crawl`: `add_node`
+  restores the alias from `metadata["was_redirected_from"]`.
+
+## 5.2.2 — 2026-06
+
+### Fixed (test infrastructure)
+
+- **The library no longer configures logging.** `GraphCrawlerClient.__init__` called `logging.basicConfig(INFO)`
+  when its logger had no handlers, so every crawl printed to stderr and reconfigured the application's root
+  logger. Removed; `graph_crawler` gets a `NullHandler`.
+- **Noise demoted:** `HTML executor shutdown` and `lmdb not installed … Falling back to SQLiteEvictionStorage` are
+  DEBUG; `channel='chrome' not installed, falling back to bundled Chromium` is a WARNING once per process.
+- **One version source.** `graph_crawler.__version__` (and the default User-Agent) come from the installed
+  package metadata (`pyproject.toml`); `__version__.py` no longer hard-codes a number that drifted (5.1.0 vs 5.2.0).
+- **Test suite collects without optional extras.** Tests that need `psutil`, `rapidfuzz`, `stop-words`,
+  `html2md-clean` or `playwright` are skipped via `importorskip` instead of failing collection under `-x`.
+  `test_div_inline_fix.py` / `test_phase1_fixes.py` tested `html2md_clean` itself (no `graph_crawler` code) and were
+  removed from the core suite.
+- GitLab CI (`.gitlab-ci.yml`): offline suite with `[dev]` and a minimal job without extras.
+
+## 5.2.1 — 2026-06
+
+### Fixed (freelance cases)
+
+- **Captured XHR/fetch traffic could not be replayed outside the browser.** `NetworkLogPlugin` recorded the
+  URL and method of a request but not its body, so every POST/GraphQL endpoint (robota.ua lists vacancies via
+  `POST dracula.robota.ua/?q=getPublishedVacanciesList`) was unusable: a GET of the same URL returns 400.
+  Records now carry `post_data` and (on `response`) `request_headers`.
+- **GraphQL responses overwrote each other.** `body_ref` was `sha1(url|status|content-type)` — all operations
+  POSTed to one `/graphql` URL shared a key and only the last body survived. The key now includes method and
+  request body.
+- **JSON family by default.** The capture allow-list matched exact MIME types only; `application/graphql-
+response+json`, `problem+json`, `vnd.api+json`, `text/json` were dropped. Without an explicit
+  `capture_content_types` every `*/json` and `*+json` type is captured (RFC 6839). Default `max_body_kb` 64 → 256
+  (listing APIs routinely exceed 64 KB).
+- **Shadow DOM was invisible.** `page.content()` serializes only light DOM: content of open shadow roots (web
+  components, embedded price widgets) never reached the node. When a page has open shadow roots, the browser
+  driver serializes it with `Element.getHTML({shadowRoots})` and expands the declarative shadow roots into their
+  hosts (`shared/utils/shadow_dom.expose_shadow_roots`) — Lexbor, like the HTML spec, does not descend into
+  `<template>` content, so a plain declarative-shadow-DOM serialization would still hide it from selectors and link
+  extraction. Pages without shadow roots are untouched.
+- `tests/unit/domain/entities/test_graph_new_features.py` still expected the `DeprecationWarning` that
+  `export_nodes()` stopped emitting in 4aab7525; it now asserts the opposite.
+
+### Fixed (real-site tests)
+
+- **Ctrl+C during `crawl()` hung until the crawl-level timeout** (28 s with `timeout=25`; forever without
+  one). `asyncio.wait_for` in Python 3.11 swallows the outer cancellation when the awaited coroutine finishes
+  at the same moment, so some workers survived `KeyboardInterrupt` and kept crawling. `NodeScanner` and the
+  escalation tier budget now use `asyncio.timeout()`; `CrawlCoordinator` cancels its workers on
+  `CancelledError`. Ctrl+C now returns in ~0.3 s.
+- **`same_domain=False` was ignored** — without `allowed_domains` the `DomainFilter` default
+  (`domain+subdomains`) still applied and external links never entered the graph. Now maps to `allowed_domains=["*"]`.
+- **`graph.metadata["stop_reason"]` was `None` after `timeout=` / `max_pages`** — only stop conditions were
+  recorded. Now `"timeout"`, `"max_pages"` or `"limit"`.
+- **Network errors were invisible on the node**: a DNS/connection failure left `response_status=None`,
+  `scanned=True` and no message; `get_failed_nodes()` returned `[]`. The error text is now in
+  `node.metadata["error"]`, the HTTP status (if any) is kept, and `get_failed_nodes()` with default
+  arguments includes network-error nodes.
+- **`storage="json", storage_config={"path": "site.json"}` was silently ignored** (file written to
+  `crawler_data/json/graph.json`). `path` is now honoured, like for sqlite (`JSONStorage(file_name=)`).
+- `Graph.export_nodes()` / `Graph.export_edges()` are no longer deprecated: they are the public one-liners the
+  README shows and now delegate to `GraphExportUseCase` silently (no `DeprecationWarning`).
+- `tests/manual/real_sites/run_all.py -j N` runs scenarios in parallel; `tests/manual/real_sites/test_user_cases.py`
+  — 23 pytest user scenarios on Wikipedia / robota.ua / GitHub / example.com, each hard-capped at 40 s.
+
+### Changed
+
+- **Engine contract verified end-to-end**. `tests/transport/
+test_engine_contract.py` drives the real `PlaywrightDriver.fetch()` through three engines registered via the
+  public `register_engine()` on a recording fake SDK: the call sequence is byte-for-byte identical across
+  engines, stealth is injected iff `provides_stealth is False`, and `executable_path`/`ignore_default_args`/
+  launch args come only from the engine. No runtime code changed.
+- **`LexborDocument.clean_text()` no longer re-parses the page**. To strip `script/style/noscript/svg/head` without mutating the shared tree it
+  serialised the document and built a second `LexborHTMLParser` — i.e. every page with the default
+  `TextExtractorPlugin` was parsed twice (`count_parses.py`: 2.0 → **1.0** per page; with `MarkdownPlugin`
+  1 parse + 1 clone). It now uses `tree.clone()`.
+- **One `ContentDecoder` for every HTTP tier**. `AsyncDriver` (aiohttp) and `AsyncCurlCffiDriver` (curl_cffi) no longer rely on the client's own
+  `response.text`/`.text()`: they read the raw body and decode it through `ContentDecoder`
+  (`Content-Type` charset → `<meta charset>` → BOM → chardet → strict utf-8). Pages whose charset is only
+  declared in `<meta>` (e.g. windows-1251 without a header) were previously decoded as utf-8 by aiohttp,
+  failed, and silently became `html=None` with a "successful" 200. An undecodable body is now reported as
+  `FetchResponse.error` (`"ContentDecoder: cannot decode body …"`) with the status code preserved;
+  `ConnectionPoolManager.fetch_text()` goes through the same decoder. `FetchResponse.html` is a strict
+  `str` field — passing `bytes` raises `ValidationError` instead of being coerced. Browser tiers
+  (Playwright `page.content()`, nodriver `tab.get_content()`) already return `str`. Contract test:
+  `tests/transport/test_html_is_str_contract.py` (driver matrix returns `str`; cp1251 via meta/header;
+  binary → error).
+- **Lazy public facade**. `graph_crawler/__init__.py` now
+  eagerly imports only `crawl`, `async_crawl`, `Graph`, `Node`, `save_graph`, `load_graph`,
+  `BaseStopCondition`; every other name in `__all__` (drivers, storages, exceptions, settings, views,
+  celery tasks, …) resolves on first access via module `__getattr__` and is cached. `aiohttp` is no longer
+  loaded by `import graph_crawler`. `bootstrap()` no longer calls `collect_system_metrics()`
+  (`psutil.cpu_percent(interval=0.1)` blocked import for 100 ms for a DEBUG line). Import time here:
+  470 ms → 228 ms. Optional names (`PlaywrightDriver`, `celery`, `EasyDistributedCrawler`, …) still return
+  `None` without their extra.
+- **Escalation re-probe replays the solving tier's User-Agent**.
+  Playwright and nodriver now export `navigator.userAgent` as `extras["user_agent"]`;
+  `SessionState.absorb` records it (pinned once the session is solved, like `impersonate`/`proxy`) and
+  `EscalatingDriver._fetch_tier` sends it as `extra_headers={"User-Agent": …}` to HTTP/TLS tiers together
+  with the stored cookies. `cf_clearance` is bound to UA+IP+JA4, so a re-probe on Tier 1 with aiohttp's
+  default UA was rejected and the host never descended from the browser tier.
+
+### Removed
+
+- **`NodePluginContext.parser`**. Node plugins now read the
+  single Lexbor tree through the `HTMLDocument`/`HTMLNode` protocols: `LinkExtractorPlugin`
+  (`doc.css("a[href]")`, `doc.base_href()`), `MetadataExtractorPlugin`, `TextExtractorPlugin`
+  (`doc.clean_text()`, new in the protocol), `StructuredDataPlugin` (OG/Twitter/Microdata/RDFa parsers take
+  an `HTMLDocument`; `StructuredDataExtractor.extract(html=, doc=)`), `Email`/`PhoneExtractorPlugin`.
+  Migration: `context.parser.find_all(sel)` → `context.html_tree.css(sel)`, `.find()` → `.css_first()`,
+  `el.get_attribute()` → `el.attr()`. Fixed along the way in `LexborNode`: `attr()` returns `""` (not
+  `None`) for valueless boolean attributes such as `itemscope`, so nested Microdata/RDFa items are now
+  detected; `css()`/`css_first()` return descendants only (selectolax included the node itself, which made
+  Microdata recurse to `max_nesting_depth`).
+- **`PooledPlaywrightDriver` legacy batch path**: `batch_legacy.py`
+  (`LegacyBatch._fetch_batch_legacy`, `_fetch_chain`) is gone. `persistent_browser=False` keeps its
+  "launch/stop per batch" semantics through the pool path: after a batch, once no fetch is in flight, the
+  pool and the engine session are closed (`_close_pool()` + `_stop_persistent_session()`, shared with
+  `close()`). Dead `self.playwright`/`self.browsers` cleanup removed from `close()`.
+- **Escalation.** `classify_block()` rewritten to the
+  6.0 decision table: challenge/captcha vendor headers first, then unambiguous body captcha/challenge
+  markers (ahead of vendor hard-block headers), then short/blocking-page widgets, status-based verdicts
+  and empty-2xx. A bare `401/403/405` with no vendor markers is no longer a block. On the classifier corpus
+  it goes 25/37 → **37/37**.
+
+  - `needs_browser` heuristic (SPA shell: little visible text + SPA root / `<noscript>` JS hint / no
+    `<a href>`) is now the default `EscalationPolicy.needs_browser`; on a clean HTTP/TLS response it jumps
+    straight to the first browser tier (`escalation_reason="needs_browser"`).
+  - Per-tier `TierTimeouts` + `escalation_deadline_s` (60s) budget: a tier is skipped (last response
+    returned, browser never started) when the remaining budget is below its `min_budget`; each tier fetch
+    is wrapped in `asyncio.wait_for`. New `PER_NODE_FETCH_TIMEOUT` constant with a validator warning.
+  - Cross-tier session: `SessionState` / `HostSessionStore` carry solved cookies (`cf_clearance`, `__cf_bm`,
+    `datadome`, …) forward and back down to cheaper tiers on re-probe; `impersonate`/`proxy` are pinned once
+    a session is solved. curl_cffi pins the impersonate fingerprint; Playwright/nodriver export context
+    cookies via `extras["cookies"]`.
+  - `extras` contract: every driver stamps `channel`/`driver`/`engine` (via `BaseDriver`/`BaseAsyncDriver`
+    `_stamp_extras` + `_finalize_response`) even outside escalation.
+  - New Prometheus metrics `crawl_fetch_duration_seconds{channel,engine}`, `crawl_escalations_total`,
+    `crawl_escalation_deadline_hits_total`.
+
+- **HTML parsing: Lexbor (selectolax) is the only core parser**.
+  `beautifulsoup4`, `lxml`, `lxml_html_clean` removed from `[project].dependencies`;
+  `BeautifulSoupAdapter`, `LxmlAdapter`, `ScrapyAdapter` deleted. New `HTMLDocument` / `HTMLNode`
+  protocols in `domain/value_objects/html_document.py`, implemented by `LexborDocument` in
+  `infrastructure/adapters/selectolax_adapter.py`. `<template>` is renamed to `gc-template` before
+  parsing so Vue/React SSR links inside templates are not lost. `IParser.parse()` returns `HTMLDocument`;
+  `NodePluginContext.html_tree` is now a `LexborDocument`. `get_parser()` accepts `auto|selectolax|lexbor`.
+- `HTMLUtils.extract_links/extract_text/extract_metadata` accept an `HTMLDocument` (string input still
+  works but is deprecated).
+- `SitemapParser.parse_large_sitemap_streaming()` uses stdlib `iterparse` by default; lxml is an
+  optional fast-path via the new extra `sitemap-fast`.
+
+- **`html2md-clean` moved to the optional extra `markdown`** (`pip install "graph-crawler[markdown]"`);
+  `MarkdownGenerator/MarkdownOptions/MarkdownResult` re-exports in `graph_crawler.shared.utils` are lazy.
+  `MarkdownPlugin` feeds the already-parsed Lexbor tree to html2md-clean 0.5 (`generate_from_tree`) —
+  one parse per page instead of 2–4. `import graph_crawler` no longer loads bs4/lxml.
+
+- **Dependency map**. Core is now 10 packages: `requests`,
+  `httpx`, `pydantic-settings`, `pybloom-live`, `fastapi`, `tenacity`, `stop-words`, `tiktoken`,
+  `markdown-it-py` removed from `[project].dependencies`. New extras: `browser` (`playwright` kept as alias),
+  `server` (fastapi/uvicorn/websockets), `text` (tiktoken/stop-words), `captcha`; `markdown` also pulls
+  `markdown-it-py`; `all` is recursive. Extras `cloudscraper` and `undetected-sync` dropped.
+  Extra-gated imports are lazy and fail fast with `requires extra '<name>'`; `transport.PlaywrightDriver/
+RequestsDriver/CloudscraperDriver/RequestsSessionAdapter` are lazy re-exports (ImportError instead of `None`).
+  `CrawlerSettings` keeps `GC_*`/`__`/`.env` without pydantic-settings; `CrawlerConfig` is a plain `BaseModel`.
+
+- **Transport cleanup**. Deleted: sync drivers
+  (`RequestsDriver`, `CloudscraperDriver`, `UndetectedChromeDriver`, `BotasaurusDriver`, `BaseSyncDriver`),
+  `AsyncDriverHTTP2` (httpx), `AsyncCloudscraperDriver`, `tls_client.py`, `StealthHTTPDriver`,
+  `session_manager/session_adapters/session_protocol`, `shared/utils/captcha_bypass.py`,
+  `shared/utils/proxy_manager.py`, packages `transport/http` and `transport/browser`.
+  `DriverType` is now `aiohttp | tls | playwright | nodriver | escalating` (+ `asyncio` alias);
+  strings `http`/`async`/`asyncio` → `aiohttp` and `stealth`/`curl_cffi` → `tls` are accepted for one
+  release with `DeprecationWarning`. One `AsyncDriver` (former `driver_adaptive.py` is now
+  `async_http/driver.py`); request hints (`referer`, `extra_headers`, `cookies`) are honoured.
+  `requests` is gone from the package: proxy health-check, CAPTCHA solvers/bypass and `SitemapParser`
+  use aiohttp (`SitemapParser.parse_sitemap/parse_from_robots` are `asyncio.run` wrappers,
+  `http_client=` is deprecated and ignored). `BloomFilter` is pure-python (xxhash) — `pybloom-live`
+  no longer needed.
+  `import graph_crawler` no longer loads any extra (guarded by `tests/unit/test_import_surface.py`), 3.8 s → 1.7 s here.
+  `shared/constants.py` no longer imports aiohttp — `br`/`zstd` are advertised
+  by `transport.compression.runtime_browser_headers()` only when the runtime can decode them;
+  one driver registry — `application.services.driver_factory` delegates to
+  `transport.factory.DriverFactory` (`register_factory`, `registered_names`); `SitemapParser(driver=)`
+  fetches robots.txt/sitemaps through the crawl `IDriver` (same tier chain), `SitemapSpider` passes its
+  driver, aiohttp is used directly only for binary `.gz`; `tests/transport/test_all_drivers_report_redirects.py`
+  is an `IDriver` acceptance matrix (aiohttp / tls / fake playwright / fake nodriver / escalating).
+
+- **Escalation**: `EscalationPolicy.reprobe_interval` (5 min) —
+  one request per interval re-probes tier 0 for a host remembered on a higher tier (no herd of browser
+  re-escalations after TTL); `EscalationPolicy.max_hosts` (10k) — `HostTierMemory` is an LRU and drops
+  per-host locks together with entries. In strict mode (`GC_ENGINE_STRICT`) an `ImportError` raised _during_
+  `fetch` (lazy browser start without the SDK) now propagates instead of being turned into a network error.
+
+- **`extras` contract persisted.** New
+  `FetchResponse.fetch_provenance()` / `FETCH_EXTRAS_KEYS`: a JSON-safe slice
+  (`bot_block_status, channel, driver, engine, escalations` + `escalation_reason`; cookies/impersonate dropped).
+  `NodeScanner` stores it in `node.metadata["fetch"]` for every response, including blocked/error ones, so it
+  survives JSON/SQLite/MongoDB/PostgreSQL storage. Contract test `tests/transport/test_extras_contract.py`.
+
+### Fixed
+
+- Transport `extras` (`bot_block_status`, tier, escalations) were dropped after fetch and never reached the
+  node or storage; for blocked pages nothing was transferred at all.
+- `SmartPageFinder` read `html_tree.text` as a property; with Lexbor it is a method — page text was
+  silently dropped.
+- `AsyncUndetectedDriver` accepts a plain `dict` config (`AsyncUndetectedConfig.from_mapping`) — building
+  the `nodriver` tier through `DriverFactory.create(DriverType.NODRIVER, config={...})` (as `EscalatingDriver`
+  does) raised `AttributeError`.
+
+## 5.2.0 — 2026-06
+
+### Added
+
+- `infrastructure/transport/escalation.py`: `classify_block()` (Cloudflare / Akamai / DataDome /
+  PerimeterX / Incapsula markers, 403/429/503, empty 200), `EscalationPolicy`, `EscalationTier`,
+  `EscalatingDriver` — chain http → tls → browser → undetected with `max_tier`.
+  `driver="escalating"` works in `crawl()` / `async_crawl()` / `Crawler` / CLI.
+- `HostTierMemory`: adaptive per-host tier memory — TTL from the moment of decision
+  (`GC_TIER_TTL`, default 1200 s), invalidation when the remembered tier gets blocked,
+  one escalation per host at a time (per-host lock). Per process.
+- Strict engine mode: `GC_ENGINE_STRICT=1`, `{"browser_engine": {"strict": True}}` or
+  `driver_config={"engine_strict": True}` — `ImportError` instead of a silent fallback
+  `cloakbrowser → patchright → playwright` or a skipped escalation tier.
+- Browser engine registry in `playwright/engine.py`: `register_engine()`, `resolve_engine()`,
+  `registered_engines()`. Built-in engines `cloakbrowser` → `patchright` → `playwright` with
+  automatic fail-open. `GC_BROWSER_ENGINE` env selects the engine when config has none.
+  `PlaywrightBrowserEngine.effective_name` / `fallback_reason`.
+- `FetchResponse.extras` contract: `bot_block_status`, `channel`, `driver`, `engine`, `escalations`.
+- `DriverType.TLS`, `NODRIVER`, `UNDETECTED`, `BOTASAURUS`, `ESCALATING`; `DriverFactory.create()`
+  accepts extra constructor kwargs (`policy=`).
+- Extras: `tls`, `stealth`, `cloak`, `undetected`, `undetected-sync`, `antidetect`, `cloudscraper`, `system`.
+- `README.md`, `CHANGELOG.md`.
+
+### Changed
+
+- `normalize_protocol` defaults to `False`: `http://` and `https://` are different URLs.
+  The crawler follows the real redirect instead of rewriting the scheme itself, so
+  http-only hosts stay reachable. Enable it explicitly to merge both schemes on dedup.
+- `Graph.handle_redirect()` stores the alias as the real `Location` the server sent
+  (provenance); the normalized form is used only to find and merge nodes.
+- Single `DriverType` enum (`transport/base.py`); `factory.py` re-exports it.
+- `cloudscraper` moved from required dependencies to extra `cloudscraper`.
+- `psutil` is optional (`system` extra); `MemoryGuard` imports it lazily.
+- `requires-python = ">=3.11"` (upper bound removed; classifiers 3.11–3.14).
+- `pytest` runs offline by default (`-m 'not manual'`, no coverage in `addopts`).
+  Network / real-site tests live in `tests/manual/` and run with `pytest -m manual tests/manual`.
+- Project URLs point to the GitLab repository.
+
+### Fixed
+
+- `pooled_driver`: bounded walk over `redirected_from` (infinite loop on objects without a
+  `None` terminator); cookies are no longer added twice to a fresh browser.
+- Tests synced with the code: redirect aliases, `html2md-clean` 0.4.0 API, `crawl_coordinator`
+  `per_host_max_in_flight`, `.gitignore` lookup at the repository root, `tools/db_cleanup.py`
+  CLI allow-listed for `print`. Playwright fake tests skip when `playwright` is not installed.
+
+### Removed
+
+- `tests/345.py`, `tests/ДОДАТКОВІ_ТЕСТИ.py`, `tests/manual_4a_games_test.py`, `tests/google_maps_results.json`.
+- `EXTRACTION_AUDIT.md` and other analysis / plan documents that described work, not the code.
+- `graph_crawler/data/ARCHITECTURE.md`, `MIGRATION_50_STEPS.md`, `REFACTORING_PLAN.md`, `mkdocs.yml`
+  (finished plans / config without content).
+
+## 5.1.0
+
+Previous release (sync-first API, Playwright pool, adaptive HTTP drivers, storage backends).

graph_crawler-5.3.4/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 GraphCrawler Team
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

graph_crawler-5.3.4/MANIFEST.in

@@ -0,0 +1,16 @@
+include README.md
+include LICENSE
+include CHANGELOG.md
+include pyproject.toml
+include graph_crawler/py.typed
+
+recursive-include graph_crawler/native *.pyx
+include graph_crawler/native/setup.py
+
+prune tests
+global-exclude __pycache__
+global-exclude *.py[cod]
+global-exclude *.so
+global-exclude *.pyd
+global-exclude *.c
+global-exclude .DS_Store