graph-crawler 5.2.0__tar.gz → 5.2.1__tar.gz

graph_crawler-5.2.1/CHANGELOG.md

@@ -0,0 +1,244 @@
+# Changelog
+
+## 5.2.1 — 2026-06
+
+### Fixed (freelance cases 33/34 — `ai-graph-crawler/docs/FREELANCE_CASES.md`)
+- **Captured XHR/fetch traffic could not be replayed outside the browser.** `NetworkLogPlugin` recorded the
+  URL and method of a request but not its body, so every POST/GraphQL endpoint (robota.ua lists vacancies via
+  `POST dracula.robota.ua/?q=getPublishedVacanciesList`) was unusable: a GET of the same URL returns 400.
+  Records now carry `post_data` and (on `response`) `request_headers`.
+- **GraphQL responses overwrote each other.** `body_ref` was `sha1(url|status|content-type)` — all operations
+  POSTed to one `/graphql` URL shared a key and only the last body survived. The key now includes method and
+  request body.
+- **JSON family by default.** The capture allow-list matched exact MIME types only; `application/graphql-
+  response+json`, `problem+json`, `vnd.api+json`, `text/json` were dropped. Without an explicit
+  `capture_content_types` every `*/json` and `*+json` type is captured (RFC 6839). Default `max_body_kb` 64 → 256
+  (listing APIs routinely exceed 64 KB).
+- **Shadow DOM was invisible.** `page.content()` serializes only light DOM: content of open shadow roots (web
+  components, embedded price widgets) never reached the node. When a page has open shadow roots, the browser
+  driver serializes it with `Element.getHTML({shadowRoots})` and expands the declarative shadow roots into their
+  hosts (`shared/utils/shadow_dom.expose_shadow_roots`) — Lexbor, like the HTML spec, does not descend into
+  `<template>` content, so a plain declarative-shadow-DOM serialization would still hide it from selectors and link
+  extraction. Pages without shadow roots are untouched.
+- `tests/unit/domain/entities/test_graph_new_features.py` still expected the `DeprecationWarning` that
+  `export_nodes()` stopped emitting in 4aab7525; it now asserts the opposite.
+
+### Fixed (real-site audit, round 2 — `docs/REAL_SITE_TESTS.md` §1.2)
+- **Ctrl+C during `crawl()` hung until the crawl-level timeout** (28 s with `timeout=25`; forever without
+  one). `asyncio.wait_for` in Python 3.11 swallows the outer cancellation when the awaited coroutine finishes
+  at the same moment, so some workers survived `KeyboardInterrupt` and kept crawling. `NodeScanner` and the
+  escalation tier budget now use `asyncio.timeout()`; `CrawlCoordinator` cancels its workers on
+  `CancelledError`. Ctrl+C now returns in ~0.3 s.
+- **`same_domain=False` was ignored** — without `allowed_domains` the `DomainFilter` default
+  (`domain+subdomains`) still applied and external links never entered the graph. Now maps to `allowed_domains=["*"]`.
+- **`graph.metadata["stop_reason"]` was `None` after `timeout=` / `max_pages`** — only stop conditions were
+  recorded. Now `"timeout"`, `"max_pages"` or `"limit"`.
+- **Network errors were invisible on the node**: a DNS/connection failure left `response_status=None`,
+  `scanned=True` and no message; `get_failed_nodes()` returned `[]`. The error text is now in
+  `node.metadata["error"]`, the HTTP status (if any) is kept, and `get_failed_nodes()` with default
+  arguments includes network-error nodes.
+- **`storage="json", storage_config={"path": "site.json"}` was silently ignored** (file written to
+  `crawler_data/json/graph.json`). `path` is now honoured, like for sqlite (`JSONStorage(file_name=)`).
+- `Graph.export_nodes()` / `Graph.export_edges()` are no longer deprecated: they are the public one-liners the
+  README shows and now delegate to `GraphExportUseCase` silently (no `DeprecationWarning`).
+- `tests/manual/real_sites/run_all.py -j N` runs scenarios in parallel; `tests/manual/real_sites/test_user_cases.py`
+  — 23 pytest user scenarios on Wikipedia / robota.ua / GitHub / example.com, each hard-capped at 40 s.
+
+### Changed
+- **Engine contract verified end-to-end** (audit `06_browser_engine.md` §4, now closed). `tests/transport/
+  test_engine_contract.py` drives the real `PlaywrightDriver.fetch()` through three engines registered via the
+  public `register_engine()` on a recording fake SDK: the call sequence is byte-for-byte identical across
+  engines, stealth is injected iff `provides_stealth is False`, and `executable_path`/`ignore_default_args`/
+  launch args come only from the engine. No runtime code changed.
+- **`LexborDocument.clean_text()` no longer re-parses the page** (audit `01_parser_audit.md` §4 "parse once",
+  `00_interim_analysis.md` §3.4). To strip `script/style/noscript/svg/head` without mutating the shared tree it
+  serialised the document and built a second `LexborHTMLParser` — i.e. every page with the default
+  `TextExtractorPlugin` was parsed twice (`count_parses.py`: 2.0 → **1.0** per page; with `MarkdownPlugin`
+  1 parse + 1 clone). It now uses `tree.clone()`. `docs/audit/benchmarks/count_parses.py` works without
+  bs4/lxml and counts `generate_from_tree` clones.
+- **One `ContentDecoder` for every HTTP tier** (audit `docs/audit/01_parser_audit.md` §5 "no parser accepts
+  bytes"). `AsyncDriver` (aiohttp) and `AsyncCurlCffiDriver` (curl_cffi) no longer rely on the client's own
+  `response.text`/`.text()`: they read the raw body and decode it through `ContentDecoder`
+  (`Content-Type` charset → `<meta charset>` → BOM → chardet → strict utf-8). Pages whose charset is only
+  declared in `<meta>` (e.g. windows-1251 without a header) were previously decoded as utf-8 by aiohttp,
+  failed, and silently became `html=None` with a "successful" 200. An undecodable body is now reported as
+  `FetchResponse.error` (`"ContentDecoder: cannot decode body …"`) with the status code preserved;
+  `ConnectionPoolManager.fetch_text()` goes through the same decoder. `FetchResponse.html` is a strict
+  `str` field — passing `bytes` raises `ValidationError` instead of being coerced. Browser tiers
+  (Playwright `page.content()`, nodriver `tab.get_content()`) already return `str`. Contract test:
+  `tests/transport/test_html_is_str_contract.py` (driver matrix returns `str`; cp1251 via meta/header;
+  binary → error).
+- **Lazy public facade** (audit `docs/audit/04_dependency_map.md` §5 п.3). `graph_crawler/__init__.py` now
+  eagerly imports only `crawl`, `async_crawl`, `Graph`, `Node`, `save_graph`, `load_graph`,
+  `BaseStopCondition`; every other name in `__all__` (drivers, storages, exceptions, settings, views,
+  celery tasks, …) resolves on first access via module `__getattr__` and is cached. `aiohttp` is no longer
+  loaded by `import graph_crawler`. `bootstrap()` no longer calls `collect_system_metrics()`
+  (`psutil.cpu_percent(interval=0.1)` blocked import for 100 ms for a DEBUG line). Import time here:
+  470 ms → 228 ms. Optional names (`PlaywrightDriver`, `celery`, `EasyDistributedCrawler`, …) still return
+  `None` without their extra.
+- **Escalation re-probe replays the solving tier's User-Agent** (audit `03_escalation.md` §3/§6).
+  Playwright and nodriver now export `navigator.userAgent` as `extras["user_agent"]`;
+  `SessionState.absorb` records it (pinned once the session is solved, like `impersonate`/`proxy`) and
+  `EscalatingDriver._fetch_tier` sends it as `extra_headers={"User-Agent": …}` to HTTP/TLS tiers together
+  with the stored cookies. `cf_clearance` is bound to UA+IP+JA4, so a re-probe on Tier 1 with aiohttp's
+  default UA was rejected and the host never descended from the browser tier.
+
+### Removed
+- **`NodePluginContext.parser`** (audit `01_parser_audit.md` §4 "parse once"). Node plugins now read the
+  single Lexbor tree through the `HTMLDocument`/`HTMLNode` protocols: `LinkExtractorPlugin`
+  (`doc.css("a[href]")`, `doc.base_href()`), `MetadataExtractorPlugin`, `TextExtractorPlugin`
+  (`doc.clean_text()`, new in the protocol), `StructuredDataPlugin` (OG/Twitter/Microdata/RDFa parsers take
+  an `HTMLDocument`; `StructuredDataExtractor.extract(html=, doc=)`), `Email`/`PhoneExtractorPlugin`.
+  Migration: `context.parser.find_all(sel)` → `context.html_tree.css(sel)`, `.find()` → `.css_first()`,
+  `el.get_attribute()` → `el.attr()`. Fixed along the way in `LexborNode`: `attr()` returns `""` (not
+  `None`) for valueless boolean attributes such as `itemscope`, so nested Microdata/RDFa items are now
+  detected; `css()`/`css_first()` return descendants only (selectolax included the node itself, which made
+  Microdata recurse to `max_nesting_depth`).
+- **`PooledPlaywrightDriver` legacy batch path** (audit `06_browser_engine.md` §3): `batch_legacy.py`
+  (`LegacyBatch._fetch_batch_legacy`, `_fetch_chain`) is gone. `persistent_browser=False` keeps its
+  "launch/stop per batch" semantics through the pool path: after a batch, once no fetch is in flight, the
+  pool and the engine session are closed (`_close_pool()` + `_stop_persistent_session()`, shared with
+  `close()`). Dead `self.playwright`/`self.browsers` cleanup removed from `close()`.
+- **Escalation 6.0 (audit `docs/audit/03_escalation.md`, Vector 2).** `classify_block()` rewritten to the
+  6.0 decision table: challenge/captcha vendor headers first, then unambiguous body captcha/challenge
+  markers (ahead of vendor hard-block headers), then short/blocking-page widgets, status-based verdicts
+  and empty-2xx. A bare `401/403/405` with no vendor markers is no longer a block. The classifier corpus
+  (`docs/audit/benchmarks/classifier_corpus.py`) goes 25/37 → **37/37**.
+  - `needs_browser` heuristic (SPA shell: little visible text + SPA root / `<noscript>` JS hint / no
+    `<a href>`) is now the default `EscalationPolicy.needs_browser`; on a clean HTTP/TLS response it jumps
+    straight to the first browser tier (`escalation_reason="needs_browser"`).
+  - Per-tier `TierTimeouts` + `escalation_deadline_s` (60s) budget: a tier is skipped (last response
+    returned, browser never started) when the remaining budget is below its `min_budget`; each tier fetch
+    is wrapped in `asyncio.wait_for`. New `PER_NODE_FETCH_TIMEOUT` constant with a validator warning.
+  - Cross-tier session: `SessionState` / `HostSessionStore` carry solved cookies (`cf_clearance`, `__cf_bm`,
+    `datadome`, …) forward and back down to cheaper tiers on re-probe; `impersonate`/`proxy` are pinned once
+    a session is solved. curl_cffi pins the impersonate fingerprint; Playwright/nodriver export context
+    cookies via `extras["cookies"]`.
+  - `extras` contract: every driver stamps `channel`/`driver`/`engine` (via `BaseDriver`/`BaseAsyncDriver`
+    `_stamp_extras` + `_finalize_response`) even outside escalation.
+  - New Prometheus metrics `crawl_fetch_duration_seconds{channel,engine}`, `crawl_escalations_total`,
+    `crawl_escalation_deadline_hits_total`.
+
+- **HTML parsing: Lexbor (selectolax) is the only core parser** (audit `docs/audit/01_parser_audit.md`).
+  `beautifulsoup4`, `lxml`, `lxml_html_clean` removed from `[project].dependencies`;
+  `BeautifulSoupAdapter`, `LxmlAdapter`, `ScrapyAdapter` deleted. New `HTMLDocument` / `HTMLNode`
+  protocols in `domain/value_objects/html_document.py`, implemented by `LexborDocument` in
+  `infrastructure/adapters/selectolax_adapter.py`. `<template>` is renamed to `gc-template` before
+  parsing so Vue/React SSR links inside templates are not lost. `IParser.parse()` returns `HTMLDocument`;
+  `NodePluginContext.html_tree` is now a `LexborDocument`. `get_parser()` accepts `auto|selectolax|lexbor`.
+- `HTMLUtils.extract_links/extract_text/extract_metadata` accept an `HTMLDocument` (string input still
+  works but is deprecated).
+- `SitemapParser.parse_large_sitemap_streaming()` uses stdlib `iterparse` by default; lxml is an
+  optional fast-path via the new extra `sitemap-fast`.
+
+- **`html2md-clean` moved to the optional extra `markdown`** (`pip install "graph-crawler[markdown]"`);
+  `MarkdownGenerator/MarkdownOptions/MarkdownResult` re-exports in `graph_crawler.shared.utils` are lazy.
+  `MarkdownPlugin` feeds the already-parsed Lexbor tree to html2md-clean 0.5 (`generate_from_tree`) —
+  one parse per page instead of 2–4. `import graph_crawler` no longer loads bs4/lxml.
+
+- **Dependency map 6.0** (audit `docs/audit/04_dependency_map.md`). Core is now 10 packages: `requests`,
+  `httpx`, `pydantic-settings`, `pybloom-live`, `fastapi`, `tenacity`, `stop-words`, `tiktoken`,
+  `markdown-it-py` removed from `[project].dependencies`. New extras: `browser` (`playwright` kept as alias),
+  `server` (fastapi/uvicorn/websockets), `text` (tiktoken/stop-words), `captcha`; `markdown` also pulls
+  `markdown-it-py`; `all` is recursive. Extras `cloudscraper` and `undetected-sync` dropped.
+  Extra-gated imports are lazy and fail fast with `requires extra '<name>'`; `transport.PlaywrightDriver/
+  RequestsDriver/CloudscraperDriver/RequestsSessionAdapter` are lazy re-exports (ImportError instead of `None`).
+  `CrawlerSettings` keeps `GC_*`/`__`/`.env` without pydantic-settings; `CrawlerConfig` is a plain `BaseModel`.
+
+- **Transport eradication 6.0** (audit `docs/audit/02_codebase_audit.md`). Deleted: sync drivers
+  (`RequestsDriver`, `CloudscraperDriver`, `UndetectedChromeDriver`, `BotasaurusDriver`, `BaseSyncDriver`),
+  `AsyncDriverHTTP2` (httpx), `AsyncCloudscraperDriver`, `tls_client.py`, `StealthHTTPDriver`,
+  `session_manager/session_adapters/session_protocol`, `shared/utils/captcha_bypass.py`,
+  `shared/utils/proxy_manager.py`, packages `transport/http` and `transport/browser`.
+  `DriverType` is now `aiohttp | tls | playwright | nodriver | escalating` (+ `asyncio` alias);
+  strings `http`/`async`/`asyncio` → `aiohttp` and `stealth`/`curl_cffi` → `tls` are accepted for one
+  release with `DeprecationWarning`. One `AsyncDriver` (former `driver_adaptive.py` is now
+  `async_http/driver.py`); request hints (`referer`, `extra_headers`, `cookies`) are honoured.
+  `requests` is gone from the package: proxy health-check, CAPTCHA solvers/bypass and `SitemapParser`
+  use aiohttp (`SitemapParser.parse_sitemap/parse_from_robots` are `asyncio.run` wrappers,
+  `http_client=` is deprecated and ignored). `BloomFilter` is pure-python (xxhash) — `pybloom-live`
+  no longer needed.
+  `import graph_crawler` no longer loads any extra (guarded by `tests/unit/test_import_surface.py`), 3.8 s → 1.7 s here.
+  Follow-up (02 §6 close-out): `shared/constants.py` no longer imports aiohttp — `br`/`zstd` are advertised
+  by `transport.compression.runtime_browser_headers()` only when the runtime can decode them;
+  one driver registry — `application.services.driver_factory` delegates to
+  `transport.factory.DriverFactory` (`register_factory`, `registered_names`); `SitemapParser(driver=)`
+  fetches robots.txt/sitemaps through the crawl `IDriver` (same tier chain), `SitemapSpider` passes its
+  driver, aiohttp is used directly only for binary `.gz`; `tests/transport/test_all_drivers_report_redirects.py`
+  is an `IDriver` acceptance matrix (aiohttp / tls / fake playwright / fake nodriver / escalating).
+
+- **Escalation (audit `docs/audit/03_escalation.md` §6/§7)**: `EscalationPolicy.reprobe_interval` (5 min) —
+  one request per interval re-probes tier 0 for a host remembered on a higher tier (no herd of browser
+  re-escalations after TTL); `EscalationPolicy.max_hosts` (10k) — `HostTierMemory` is an LRU and drops
+  per-host locks together with entries. In strict mode (`GC_ENGINE_STRICT`) an `ImportError` raised *during*
+  `fetch` (lazy browser start without the SDK) now propagates instead of being turned into a network error.
+
+- **`extras` contract persisted (audit `docs/audit/05_migration_guide.md` §4).** New
+  `FetchResponse.fetch_provenance()` / `FETCH_EXTRAS_KEYS`: a JSON-safe slice
+  (`bot_block_status, channel, driver, engine, escalations` + `escalation_reason`; cookies/impersonate dropped).
+  `NodeScanner` stores it in `node.metadata["fetch"]` for every response, including blocked/error ones, so it
+  survives JSON/SQLite/MongoDB/PostgreSQL storage. Contract test `tests/transport/test_extras_contract.py`.
+- `docs/audit/05_migration_guide.md` — 5.2 → 6.0 migration guide (extras, drivers, parser, settings, sitemap, checklist).
+
+### Fixed
+- Transport `extras` (`bot_block_status`, tier, escalations) were dropped after fetch and never reached the
+  node or storage; for blocked pages nothing was transferred at all.
+- `SmartPageFinder` read `html_tree.text` as a property; with Lexbor it is a method — page text was
+  silently dropped.
+- `AsyncUndetectedDriver` accepts a plain `dict` config (`AsyncUndetectedConfig.from_mapping`) — building
+  the `nodriver` tier through `DriverFactory.create(DriverType.NODRIVER, config={...})` (as `EscalatingDriver`
+  does) raised `AttributeError`.
+
+## 5.2.0 — 2026-06
+
+### Added
+- `infrastructure/transport/escalation.py`: `classify_block()` (Cloudflare / Akamai / DataDome /
+  PerimeterX / Incapsula markers, 403/429/503, empty 200), `EscalationPolicy`, `EscalationTier`,
+  `EscalatingDriver` — chain http → tls → browser → undetected with `max_tier`.
+  `driver="escalating"` works in `crawl()` / `async_crawl()` / `Crawler` / CLI.
+- `HostTierMemory`: adaptive per-host tier memory — TTL from the moment of decision
+  (`GC_TIER_TTL`, default 1200 s), invalidation when the remembered tier gets blocked,
+  one escalation per host at a time (per-host lock). Per process.
+- Strict engine mode: `GC_ENGINE_STRICT=1`, `{"browser_engine": {"strict": True}}` or
+  `driver_config={"engine_strict": True}` — `ImportError` instead of a silent fallback
+  `cloakbrowser → patchright → playwright` or a skipped escalation tier.
+- Browser engine registry in `playwright/engine.py`: `register_engine()`, `resolve_engine()`,
+  `registered_engines()`. Built-in engines `cloakbrowser` → `patchright` → `playwright` with
+  automatic fail-open. `GC_BROWSER_ENGINE` env selects the engine when config has none.
+  `PlaywrightBrowserEngine.effective_name` / `fallback_reason`.
+- `FetchResponse.extras` contract: `bot_block_status`, `channel`, `driver`, `engine`, `escalations`.
+- `DriverType.TLS`, `NODRIVER`, `UNDETECTED`, `BOTASAURUS`, `ESCALATING`; `DriverFactory.create()`
+  accepts extra constructor kwargs (`policy=`).
+- Extras: `tls`, `stealth`, `cloak`, `undetected`, `undetected-sync`, `antidetect`, `cloudscraper`, `system`.
+- `README.md`, `CHANGELOG.md`.
+
+### Changed
+- `normalize_protocol` defaults to `False`: `http://` and `https://` are different URLs.
+  The crawler follows the real redirect instead of rewriting the scheme itself, so
+  http-only hosts stay reachable. Enable it explicitly to merge both schemes on dedup.
+- `Graph.handle_redirect()` stores the alias as the real `Location` the server sent
+  (provenance); the normalized form is used only to find and merge nodes.
+- Single `DriverType` enum (`transport/base.py`); `factory.py` re-exports it.
+- `cloudscraper` moved from required dependencies to extra `cloudscraper`.
+- `psutil` is optional (`system` extra); `MemoryGuard` imports it lazily.
+- `requires-python = ">=3.11"` (upper bound removed; classifiers 3.11–3.14).
+- `pytest` runs offline by default (`-m 'not manual'`, no coverage in `addopts`).
+  Network / real-site tests live in `tests/manual/` and run with `pytest -m manual tests/manual`.
+- Project URLs point to the GitLab repository.
+
+### Fixed
+- `pooled_driver`: bounded walk over `redirected_from` (infinite loop on objects without a
+  `None` terminator); cookies are no longer added twice to a fresh browser.
+- Tests synced with the code: redirect aliases, `html2md-clean` 0.4.0 API, `crawl_coordinator`
+  `per_host_max_in_flight`, `.gitignore` lookup at the repository root, `tools/db_cleanup.py`
+  CLI allow-listed for `print`. Playwright fake tests skip when `playwright` is not installed.
+
+### Removed
+- `tests/345.py`, `tests/ДОДАТКОВІ_ТЕСТИ.py`, `tests/manual_4a_games_test.py`, `tests/google_maps_results.json`.
+- `EXTRACTION_AUDIT.md` and other analysis / plan documents that described work, not the code.
+- `graph_crawler/data/ARCHITECTURE.md`, `MIGRATION_50_STEPS.md`, `REFACTORING_PLAN.md`, `mkdocs.yml`
+  (finished plans / config without content).
+
+## 5.1.0
+
+Previous release (sync-first API, Playwright pool, adaptive HTTP drivers, storage backends).