graph-crawler 5.2.1__tar.gz → 5.3.3__tar.gz

{graph_crawler-5.2.1 → graph_crawler-5.3.3}/CHANGELOG.md

@@ -1,8 +1,155 @@
 # Changelog
 
+## 5.3.3 — 2026-10
+
+### Added
+
+- **`IsolatedExecutor` / `IsolatedTaskError`** (`from graph_crawler import IsolatedExecutor`). Runs CPU-bound functions
+  in a `spawn` process pool: a native crash (SIGSEGV in libxml2/lxml, trafilatura, any C extension) or a hang of a
+  worker becomes `IsolatedTaskError(reason="crashed" | "timeout" | "unavailable")` in the caller, the pool restarts
+  and the crawl goes on. Tasks that shared the broken pool are retried once; a hung worker is killed after
+  `timeout` (default `DEFAULT_ISOLATED_TASK_TIMEOUT` = 30 s). Ordinary exceptions of the function propagate as is.
+- **`CrawlerConfig.node_enrich_timeout`** and `crawl(..., node_enrich_timeout=...)` / `async_crawl(...)` /
+  `GraphCrawlerClient.crawl(...)`: per-node enrichment limit (HTML parse, plugins, subclass hook, hashes).
+  `None` keeps `DEFAULT_NODE_ENRICH_TIMEOUT` (45 s, `shared.constants`). `NodeScanner(driver, enrich_timeout=...)`.
+- `run_in_html_pool(fn, *args)` (`graph_crawler.domain.entities.node`): runs `fn` in the HTML executor under its
+  semaphore. `Node.pop_discovered_links()` returns links extracted on `ON_HTML_PARSED` of an interrupted
+  `enrich_node`.
+- `URLRuleMatcher.rule_matches(rule, url)`.
+
+### Changed
+
+- **The `Node._update_from_context` hook and `get_content_hash`/`get_simhash` run in the HTML executor** under the
+  same semaphore as HTML parsing (they used the default `asyncio.to_thread` pool, `min(32, cpu+4)` threads, outside
+  the CPU budget). Concurrency of heavy subclass code is now bounded by `GRAPH_CRAWLER_HTML_PARSER_WORKERS`
+  (default — the cgroup CPU quota); `=1` serialises it completely. Code that is not thread-safe must use its own
+  lock or `GRAPH_CRAWLER_HTML_PARSER_WORKERS=1`; native code that can crash — `IsolatedExecutor`.
+- **One URL rule — one decision.** `CrawlScheduler`, `LinkProcessor` and `RuleBasedScheduler` share
+  `URLRuleMatcher`: first match in list order, `SmartURLRule` is matched against its `scope` (the scheduler used to
+  run `re.search` over the full URL, so `SmartURLRule(r"/help", scope=PATH, should_scan=False)` excluded
+  `https://helpware.com/`). `GraphSpider` no longer copies `url_rules` with `should_scan=False` into `PathFilter`
+  (it applied them to the path, ignoring order and scope); `excluded_paths` / `included_paths` are unchanged.
+- A URL rule whose `priority` is outside `[MIN_URL_PRIORITY, MAX_URL_PRIORITY]` (`[-100, 100]`) gets a WARNING when
+  the scheduler is created: `priority` is a queue position and is clamped; rule precedence is list order.
+- `IDriver.fetch(url, *, referer, extra_headers, cookies)` is the contract of every driver: `BaseDriver`,
+  `EscalatingDriver` (caller cookies/headers merged with the host session — the session wins on a cookie name
+  clash, the session `User-Agent` is applied on top for HTTP/TLS tiers), `PooledPlaywrightDriver` (`extra_headers`),
+  `IAsyncDriver`, `IBrowserDriver`. `BaseDriver.fetch_many` passes per-URL `extra_headers_list` / `cookies_list` to
+  `fetch` (they were dropped).
+
+### Fixed
+
+- **Per-node timeout no longer drops discovered links.** `scan_node` / `scan_batch` return the links extracted on
+  `ON_HTML_PARSED` when enrichment exceeds the limit; the node is still `ERROR` / `FAILED`. Large listing pages, the
+  first to hit the limit, keep their edges.
+
+## 5.3.2 — 2026-06
+
+### Added
+
+- **Fetch provenance on the node.** `Node.channel`, `bot_block_status`, `served_by_driver`, `served_by_engine`,
+  `escalations` and `fetch_info` read `metadata["fetch"]`; `Node.ok` (scanned, 2xx/3xx, not blocked, no error) and
+  `Node.is_blocked` separate useful pages from block and error pages.
+- **Plugin routing.** `BaseNodePlugin.domains` (domain and subdomains) and `url_pattern` (regex), on the class or via
+  `config={"domains": [...], "url_pattern": ...}`, plus an overridable `applies_to(url)`. An invalid `url_pattern`
+  raises `ConfigurationError` on `register()`. Several instances of one plugin class with different routes run
+  side by side (they used to replace each other).
+- **Typed plugin output.** `NodePluginContext.set_node_field(name, value)` writes a field declared by a `Node`
+  subclass, validated against its annotation (`"100"` → `int`); base fields are read-only. Fields are rolled back
+  together with `metadata`/`user_data` when the plugin fails.
+- **Empty extraction warning.** `BaseNodePlugin.expects_nonempty` + `extraction_key`: a WARNING when the plugin left
+  `user_data[extraction_key]` empty.
+- **`Graph.search(..., user_data_keys=("jobs",))`** searches the listed `user_data` keys as well.
+- **`Node.public_user_data`** and `RESERVED_USER_DATA_KEYS` (`_link_anchors`, `_custom_fields`, `_node_class`).
+
+### Changed
+
+- A response with status 400 or higher gets `metadata["error"] = "HTTP <code>"` and is logged as a WARNING
+  (`HTTP 404 for <url> — page not scanned`).
+- A bare 403/401/405 without vendor markers is still classified as `none`; the decision is now logged at INFO.
+- Quieter logs: import writes nothing at INFO, platform detection runs once, the start of a crawl is one line,
+  and each page is one INFO line with `[channel=…]`, `[block=…]`, `[error=…]`.
+- The temporary `_link_anchors` map is removed from the node once its edges are created (the anchor text stays in
+  `edge.metadata["anchor_text"]`). The default JSON export writes `public_user_data`.
+- A third-party plugin that overwrites a reserved `user_data` key gets a WARNING.
+
+### Fixed
+
+- `load_graph()` without `node_class=` restores a custom `Node` subclass when its module is already imported (the
+  class name is stored with the node; nothing is imported from the file). `Graph.copy()`, `union`, `intersection`,
+  `difference` and nodes created in a loaded graph keep the node class.
+- `metadata` written by a plugin on `ON_NODE_CREATED` or `ON_AFTER_SCAN` is no longer lost.
+- **Placeholder lists are detected as a JavaScript shell.** A 2xx body with five or more `placeholder` /
+  `skeleton` / `shimmer` elements makes `needs_browser` true, so `escalating` goes straight to the browser tier.
+  With `driver="http"` the node gets `metadata["js_shell_suspected"]` and a WARNING (once per host) suggests
+  `driver="escalating"` or `"playwright"`.
+- **No duplicate edges.** A link repeated on a page (menu + footer + body) created a new `Edge` each time
+  (17.5 % of edges on uk.wikipedia); `LinkProcessor` now skips existing `(source, target)` pairs. `Graph.has_edge`
+  also sees the backend edge index.
+- **`h1` children are no longer joined with spaces** (`Портал : Поточні події` → `Портал:Поточні події`).
+- **Default ports are normalized** (`http://x:80`, `https://x:443`, empty port).
+- `Graph.normalize_url()` is public; `LinkProcessor` checks the scheduler with the canonical form as well.
+- **`NetworkLogPlugin` waits for network quiet** before publishing the log (`settle_ms`=2000, `quiet_ms`=300;
+  `settle_ms=0` restores the old behaviour) — late XHR (search/listing after hydration) was lost.
+- `create_trailing_slash_rule` keeps the root path; an empty path is `/` for PATH-scope rules.
+
+## 5.3.1 — 2026-06
+
+### Added
+
+- **Hydration state globals snapshot.** The Playwright driver serialises `window.__NAME__` objects (`__NUXT__`,
+  `__APOLLO_STATE__`, `__INITIAL_STATE__`, `__PINIA__`, …) with `JSON.stringify` after the page has run and embeds
+  them into the HTML snapshot as `<script type="application/json" data-agc-state="__NAME__">`. Nuxt 2 ships its
+  state as an IIFE (`window.__NUXT__=(function(a,b){return {…}}(…))`) which is JavaScript, not JSON, and was
+  invisible to any static state parser. Globals that already exist as a JSON `<script id="__NAME__">`
+  (`__NEXT_DATA__`) are not duplicated; `PageFetcher.state_globals_max_chars` (5 MB) caps the snapshot.
+  `inject_state_globals(html, states)` is exported for tests and other drivers.
+
+## 5.3.0 — 2026-06
+
+### Added
+
+- **Page actions as public API.** `driver_config={"actions": [PageAction(...) | dict, ...]}` for the Playwright driver:
+  `click` / `fill` / `select` / `press` / `wait_for(response=…|selector=…)` / `scroll`, target by `selector`, `text`,
+  `placeholder` or `label` (`nth` among visible matches). Actions are validated when the driver is created and run
+  after content is ready and before the HTML snapshot and `CONTENT_READY` hooks — the network log is published after
+  the actions. Results land in `node.metadata["actions_log"]`, `["requests_after_actions"]` and `["actions_error"]`
+  (the first failing action stops the chain with a reason). `from graph_crawler import PageAction`.
+- `engine.page_error_types()` — page error classes of all installed browser SDKs.
+- `PageAction(type="back")`; action errors name the element that intercepts pointer events.
+
+### Fixed
+
+- Binary request bodies (gzip beacons) no longer raise `UnicodeDecodeError` inside Playwright event listeners
+  (`NetworkLogPlugin` and the action recorder read `post_data_buffer` with `errors="replace"`); the error used to break
+  the next Playwright call with `TypeError: function takes exactly 5 arguments`.
+
+### Fixed
+
+- `graph.get_node_by_url(seed)` finds the node after a redirect in the graph returned by `async_crawl`: `add_node`
+  restores the alias from `metadata["was_redirected_from"]`.
+
+## 5.2.2 — 2026-06
+
+### Fixed (test infrastructure)
+
+- **The library no longer configures logging.** `GraphCrawlerClient.__init__` called `logging.basicConfig(INFO)`
+  when its logger had no handlers, so every crawl printed to stderr and reconfigured the application's root
+  logger. Removed; `graph_crawler` gets a `NullHandler`.
+- **Noise demoted:** `HTML executor shutdown` and `lmdb not installed … Falling back to SQLiteEvictionStorage` are
+  DEBUG; `channel='chrome' not installed, falling back to bundled Chromium` is a WARNING once per process.
+- **One version source.** `graph_crawler.__version__` (and the default User-Agent) come from the installed
+  package metadata (`pyproject.toml`); `__version__.py` no longer hard-codes a number that drifted (5.1.0 vs 5.2.0).
+- **Test suite collects without optional extras.** Tests that need `psutil`, `rapidfuzz`, `stop-words`,
+  `html2md-clean` or `playwright` are skipped via `importorskip` instead of failing collection under `-x`.
+  `test_div_inline_fix.py` / `test_phase1_fixes.py` tested `html2md_clean` itself (no `graph_crawler` code) and were
+  removed from the core suite.
+- GitLab CI (`.gitlab-ci.yml`): offline suite with `[dev]` and a minimal job without extras.
+
 ## 5.2.1 — 2026-06
 
-### Fixed (freelance cases 33/34 — `ai-graph-crawler/docs/FREELANCE_CASES.md`)
+### Fixed (freelance cases)
+
 - **Captured XHR/fetch traffic could not be replayed outside the browser.** `NetworkLogPlugin` recorded the
   URL and method of a request but not its body, so every POST/GraphQL endpoint (robota.ua lists vacancies via
   `POST dracula.robota.ua/?q=getPublishedVacanciesList`) was unusable: a GET of the same URL returns 400.
@@ -11,7 +158,7 @@
   POSTed to one `/graphql` URL shared a key and only the last body survived. The key now includes method and
   request body.
 - **JSON family by default.** The capture allow-list matched exact MIME types only; `application/graphql-
-  response+json`, `problem+json`, `vnd.api+json`, `text/json` were dropped. Without an explicit
+response+json`, `problem+json`, `vnd.api+json`, `text/json` were dropped. Without an explicit
   `capture_content_types` every `*/json` and `*+json` type is captured (RFC 6839). Default `max_body_kb` 64 → 256
   (listing APIs routinely exceed 64 KB).
 - **Shadow DOM was invisible.** `page.content()` serializes only light DOM: content of open shadow roots (web
@@ -23,7 +170,8 @@
 - `tests/unit/domain/entities/test_graph_new_features.py` still expected the `DeprecationWarning` that
   `export_nodes()` stopped emitting in 4aab7525; it now asserts the opposite.
 
-### Fixed (real-site audit, round 2 — `docs/REAL_SITE_TESTS.md` §1.2)
+### Fixed (real-site tests)
+
 - **Ctrl+C during `crawl()` hung until the crawl-level timeout** (28 s with `timeout=25`; forever without
   one). `asyncio.wait_for` in Python 3.11 swallows the outer cancellation when the awaited coroutine finishes
   at the same moment, so some workers survived `KeyboardInterrupt` and kept crawling. `NodeScanner` and the
@@ -45,19 +193,17 @@
   — 23 pytest user scenarios on Wikipedia / robota.ua / GitHub / example.com, each hard-capped at 40 s.
 
 ### Changed
-- **Engine contract verified end-to-end** (audit `06_browser_engine.md` §4, now closed). `tests/transport/
-  test_engine_contract.py` drives the real `PlaywrightDriver.fetch()` through three engines registered via the
+
+- **Engine contract verified end-to-end**. `tests/transport/
+test_engine_contract.py` drives the real `PlaywrightDriver.fetch()` through three engines registered via the
   public `register_engine()` on a recording fake SDK: the call sequence is byte-for-byte identical across
   engines, stealth is injected iff `provides_stealth is False`, and `executable_path`/`ignore_default_args`/
   launch args come only from the engine. No runtime code changed.
-- **`LexborDocument.clean_text()` no longer re-parses the page** (audit `01_parser_audit.md` §4 "parse once",
-  `00_interim_analysis.md` §3.4). To strip `script/style/noscript/svg/head` without mutating the shared tree it
+- **`LexborDocument.clean_text()` no longer re-parses the page**. To strip `script/style/noscript/svg/head` without mutating the shared tree it
   serialised the document and built a second `LexborHTMLParser` — i.e. every page with the default
   `TextExtractorPlugin` was parsed twice (`count_parses.py`: 2.0 → **1.0** per page; with `MarkdownPlugin`
-  1 parse + 1 clone). It now uses `tree.clone()`. `docs/audit/benchmarks/count_parses.py` works without
-  bs4/lxml and counts `generate_from_tree` clones.
-- **One `ContentDecoder` for every HTTP tier** (audit `docs/audit/01_parser_audit.md` §5 "no parser accepts
-  bytes"). `AsyncDriver` (aiohttp) and `AsyncCurlCffiDriver` (curl_cffi) no longer rely on the client's own
+  1 parse + 1 clone). It now uses `tree.clone()`.
+- **One `ContentDecoder` for every HTTP tier**. `AsyncDriver` (aiohttp) and `AsyncCurlCffiDriver` (curl_cffi) no longer rely on the client's own
   `response.text`/`.text()`: they read the raw body and decode it through `ContentDecoder`
   (`Content-Type` charset → `<meta charset>` → BOM → chardet → strict utf-8). Pages whose charset is only
   declared in `<meta>` (e.g. windows-1251 without a header) were previously decoded as utf-8 by aiohttp,
@@ -68,7 +214,7 @@
   (Playwright `page.content()`, nodriver `tab.get_content()`) already return `str`. Contract test:
   `tests/transport/test_html_is_str_contract.py` (driver matrix returns `str`; cp1251 via meta/header;
   binary → error).
-- **Lazy public facade** (audit `docs/audit/04_dependency_map.md` §5 п.3). `graph_crawler/__init__.py` now
+- **Lazy public facade**. `graph_crawler/__init__.py` now
   eagerly imports only `crawl`, `async_crawl`, `Graph`, `Node`, `save_graph`, `load_graph`,
   `BaseStopCondition`; every other name in `__all__` (drivers, storages, exceptions, settings, views,
   celery tasks, …) resolves on first access via module `__getattr__` and is cached. `aiohttp` is no longer
@@ -76,7 +222,7 @@
   (`psutil.cpu_percent(interval=0.1)` blocked import for 100 ms for a DEBUG line). Import time here:
   470 ms → 228 ms. Optional names (`PlaywrightDriver`, `celery`, `EasyDistributedCrawler`, …) still return
   `None` without their extra.
-- **Escalation re-probe replays the solving tier's User-Agent** (audit `03_escalation.md` §3/§6).
+- **Escalation re-probe replays the solving tier's User-Agent**.
   Playwright and nodriver now export `navigator.userAgent` as `extras["user_agent"]`;
   `SessionState.absorb` records it (pinned once the session is solved, like `impersonate`/`proxy`) and
   `EscalatingDriver._fetch_tier` sends it as `extra_headers={"User-Agent": …}` to HTTP/TLS tiers together
@@ -84,7 +230,8 @@
   default UA was rejected and the host never descended from the browser tier.
 
 ### Removed
-- **`NodePluginContext.parser`** (audit `01_parser_audit.md` §4 "parse once"). Node plugins now read the
+
+- **`NodePluginContext.parser`**. Node plugins now read the
   single Lexbor tree through the `HTMLDocument`/`HTMLNode` protocols: `LinkExtractorPlugin`
   (`doc.css("a[href]")`, `doc.base_href()`), `MetadataExtractorPlugin`, `TextExtractorPlugin`
   (`doc.clean_text()`, new in the protocol), `StructuredDataPlugin` (OG/Twitter/Microdata/RDFa parsers take
@@ -94,16 +241,17 @@
   `None`) for valueless boolean attributes such as `itemscope`, so nested Microdata/RDFa items are now
   detected; `css()`/`css_first()` return descendants only (selectolax included the node itself, which made
   Microdata recurse to `max_nesting_depth`).
-- **`PooledPlaywrightDriver` legacy batch path** (audit `06_browser_engine.md` §3): `batch_legacy.py`
+- **`PooledPlaywrightDriver` legacy batch path**: `batch_legacy.py`
   (`LegacyBatch._fetch_batch_legacy`, `_fetch_chain`) is gone. `persistent_browser=False` keeps its
   "launch/stop per batch" semantics through the pool path: after a batch, once no fetch is in flight, the
   pool and the engine session are closed (`_close_pool()` + `_stop_persistent_session()`, shared with
   `close()`). Dead `self.playwright`/`self.browsers` cleanup removed from `close()`.
-- **Escalation 6.0 (audit `docs/audit/03_escalation.md`, Vector 2).** `classify_block()` rewritten to the
+- **Escalation.** `classify_block()` rewritten to the
   6.0 decision table: challenge/captcha vendor headers first, then unambiguous body captcha/challenge
   markers (ahead of vendor hard-block headers), then short/blocking-page widgets, status-based verdicts
-  and empty-2xx. A bare `401/403/405` with no vendor markers is no longer a block. The classifier corpus
-  (`docs/audit/benchmarks/classifier_corpus.py`) goes 25/37 → **37/37**.
+  and empty-2xx. A bare `401/403/405` with no vendor markers is no longer a block. On the classifier corpus
+  it goes 25/37 → **37/37**.
+
   - `needs_browser` heuristic (SPA shell: little visible text + SPA root / `<noscript>` JS hint / no
     `<a href>`) is now the default `EscalationPolicy.needs_browser`; on a clean HTTP/TLS response it jumps
     straight to the first browser tier (`escalation_reason="needs_browser"`).
@@ -119,7 +267,7 @@
   - New Prometheus metrics `crawl_fetch_duration_seconds{channel,engine}`, `crawl_escalations_total`,
     `crawl_escalation_deadline_hits_total`.
 
-- **HTML parsing: Lexbor (selectolax) is the only core parser** (audit `docs/audit/01_parser_audit.md`).
+- **HTML parsing: Lexbor (selectolax) is the only core parser**.
   `beautifulsoup4`, `lxml`, `lxml_html_clean` removed from `[project].dependencies`;
   `BeautifulSoupAdapter`, `LxmlAdapter`, `ScrapyAdapter` deleted. New `HTMLDocument` / `HTMLNode`
   protocols in `domain/value_objects/html_document.py`, implemented by `LexborDocument` in
@@ -136,16 +284,16 @@
   `MarkdownPlugin` feeds the already-parsed Lexbor tree to html2md-clean 0.5 (`generate_from_tree`) —
   one parse per page instead of 2–4. `import graph_crawler` no longer loads bs4/lxml.
 
-- **Dependency map 6.0** (audit `docs/audit/04_dependency_map.md`). Core is now 10 packages: `requests`,
+- **Dependency map**. Core is now 10 packages: `requests`,
   `httpx`, `pydantic-settings`, `pybloom-live`, `fastapi`, `tenacity`, `stop-words`, `tiktoken`,
   `markdown-it-py` removed from `[project].dependencies`. New extras: `browser` (`playwright` kept as alias),
   `server` (fastapi/uvicorn/websockets), `text` (tiktoken/stop-words), `captcha`; `markdown` also pulls
   `markdown-it-py`; `all` is recursive. Extras `cloudscraper` and `undetected-sync` dropped.
   Extra-gated imports are lazy and fail fast with `requires extra '<name>'`; `transport.PlaywrightDriver/
-  RequestsDriver/CloudscraperDriver/RequestsSessionAdapter` are lazy re-exports (ImportError instead of `None`).
+RequestsDriver/CloudscraperDriver/RequestsSessionAdapter` are lazy re-exports (ImportError instead of `None`).
   `CrawlerSettings` keeps `GC_*`/`__`/`.env` without pydantic-settings; `CrawlerConfig` is a plain `BaseModel`.
 
-- **Transport eradication 6.0** (audit `docs/audit/02_codebase_audit.md`). Deleted: sync drivers
+- **Transport cleanup**. Deleted: sync drivers
   (`RequestsDriver`, `CloudscraperDriver`, `UndetectedChromeDriver`, `BotasaurusDriver`, `BaseSyncDriver`),
   `AsyncDriverHTTP2` (httpx), `AsyncCloudscraperDriver`, `tls_client.py`, `StealthHTTPDriver`,
   `session_manager/session_adapters/session_protocol`, `shared/utils/captcha_bypass.py`,
@@ -159,7 +307,7 @@
   `http_client=` is deprecated and ignored). `BloomFilter` is pure-python (xxhash) — `pybloom-live`
   no longer needed.
   `import graph_crawler` no longer loads any extra (guarded by `tests/unit/test_import_surface.py`), 3.8 s → 1.7 s here.
-  Follow-up (02 §6 close-out): `shared/constants.py` no longer imports aiohttp — `br`/`zstd` are advertised
+  `shared/constants.py` no longer imports aiohttp — `br`/`zstd` are advertised
   by `transport.compression.runtime_browser_headers()` only when the runtime can decode them;
   one driver registry — `application.services.driver_factory` delegates to
   `transport.factory.DriverFactory` (`register_factory`, `registered_names`); `SitemapParser(driver=)`
@@ -167,20 +315,20 @@
   driver, aiohttp is used directly only for binary `.gz`; `tests/transport/test_all_drivers_report_redirects.py`
   is an `IDriver` acceptance matrix (aiohttp / tls / fake playwright / fake nodriver / escalating).
 
-- **Escalation (audit `docs/audit/03_escalation.md` §6/§7)**: `EscalationPolicy.reprobe_interval` (5 min) —
+- **Escalation**: `EscalationPolicy.reprobe_interval` (5 min) —
   one request per interval re-probes tier 0 for a host remembered on a higher tier (no herd of browser
   re-escalations after TTL); `EscalationPolicy.max_hosts` (10k) — `HostTierMemory` is an LRU and drops
-  per-host locks together with entries. In strict mode (`GC_ENGINE_STRICT`) an `ImportError` raised *during*
+  per-host locks together with entries. In strict mode (`GC_ENGINE_STRICT`) an `ImportError` raised _during_
   `fetch` (lazy browser start without the SDK) now propagates instead of being turned into a network error.
 
-- **`extras` contract persisted (audit `docs/audit/05_migration_guide.md` §4).** New
+- **`extras` contract persisted.** New
   `FetchResponse.fetch_provenance()` / `FETCH_EXTRAS_KEYS`: a JSON-safe slice
   (`bot_block_status, channel, driver, engine, escalations` + `escalation_reason`; cookies/impersonate dropped).
   `NodeScanner` stores it in `node.metadata["fetch"]` for every response, including blocked/error ones, so it
   survives JSON/SQLite/MongoDB/PostgreSQL storage. Contract test `tests/transport/test_extras_contract.py`.
-- `docs/audit/05_migration_guide.md` — 5.2 → 6.0 migration guide (extras, drivers, parser, settings, sitemap, checklist).
 
 ### Fixed
+
 - Transport `extras` (`bot_block_status`, tier, escalations) were dropped after fetch and never reached the
   node or storage; for blocked pages nothing was transferred at all.
 - `SmartPageFinder` read `html_tree.text` as a property; with Lexbor it is a method — page text was
@@ -192,6 +340,7 @@
 ## 5.2.0 — 2026-06
 
 ### Added
+
 - `infrastructure/transport/escalation.py`: `classify_block()` (Cloudflare / Akamai / DataDome /
   PerimeterX / Incapsula markers, 403/429/503, empty 200), `EscalationPolicy`, `EscalationTier`,
   `EscalatingDriver` — chain http → tls → browser → undetected with `max_tier`.
@@ -213,6 +362,7 @@
 - `README.md`, `CHANGELOG.md`.
 
 ### Changed
+
 - `normalize_protocol` defaults to `False`: `http://` and `https://` are different URLs.
   The crawler follows the real redirect instead of rewriting the scheme itself, so
   http-only hosts stay reachable. Enable it explicitly to merge both schemes on dedup.
@@ -227,6 +377,7 @@
 - Project URLs point to the GitLab repository.
 
 ### Fixed
+
 - `pooled_driver`: bounded walk over `redirected_from` (infinite loop on objects without a
   `None` terminator); cookies are no longer added twice to a fresh browser.
 - Tests synced with the code: redirect aliases, `html2md-clean` 0.4.0 API, `crawl_coordinator`
@@ -234,6 +385,7 @@
   CLI allow-listed for `print`. Playwright fake tests skip when `playwright` is not installed.
 
 ### Removed
+
 - `tests/345.py`, `tests/ДОДАТКОВІ_ТЕСТИ.py`, `tests/manual_4a_games_test.py`, `tests/google_maps_results.json`.
 - `EXTRACTION_AUDIT.md` and other analysis / plan documents that described work, not the code.
 - `graph_crawler/data/ARCHITECTURE.md`, `MIGRATION_50_STEPS.md`, `REFACTORING_PLAN.md`, `mkdocs.yml`

{graph_crawler-5.2.1 → graph_crawler-5.3.3}/MANIFEST.in

@@ -1,14 +1,11 @@
 include README.md
 include LICENSE
 include CHANGELOG.md
-include CODE_STYLE.md
 include pyproject.toml
 include graph_crawler/py.typed
 
 recursive-include graph_crawler/native *.pyx
-recursive-include graph_crawler/native *.pxd
 include graph_crawler/native/setup.py
-include graph_crawler/native/README.md
 
 prune tests
 global-exclude __pycache__