graph-crawler 5.3.3__tar.gz → 5.3.5__tar.gz

{graph_crawler-5.3.3 → graph_crawler-5.3.5}/CHANGELOG.md

@@ -1,5 +1,118 @@
 # Changelog
 
+## Unreleased
+
+## 5.3.5 — 2026-06
+
+### Fixed
+
+- **Dirty `Sitemap:` value in robots.txt no longer loses the sitemap (P19).** `SitemapParser._parse_robots_txt`
+  took the whole directive value verbatim (`line[8:].strip()`), so `Sitemap: sitemap URL https://site.com/sitemap_index.xml`
+  was glued into a broken address and the sitemap was lost. It now extracts the first `https?://\S+` from the value;
+  `urljoin` is applied only when the value is a relative path starting with `/`. New `SitemapParser._clean_sitemap_value`.
+- **A 404/410 is never a bot block (P19 side-finding).** `escalation._classify` returns `BotBlockStatus.NONE` for
+  status 404/410 right after the authoritative challenge headers. A dead sitemap candidate (or any not-found page) is
+  no longer classified as `captcha`/`challenge` and dragged through every tier (http→tls→browser). A real challenge
+  uses 403/503, not 404, so the authoritative header check still wins. Invariant: 404 is a definitive "not found"
+  from the server, not a page served instead of content. (After N3, verify & lock with a test — see Parser side.)
+- **A redirect inside the same ATS no longer blocks `domain_filter` (P21).** `CrawlCoordinator._handle_redirect_after_scan`
+  always allows a redirect to the **same host** (short ATS forms `apply.workable.com/j/X` → `apply.workable.com/<slug>/j/X`).
+  For a different host it still asks `domain_filter.is_allowed`, and when that refuses it consults a new host-alias
+  extension point `DomainFilter.allow_redirect(source, target)` (driven by a settable `redirect_host_alias` hook) — so
+  the Parser can pass its `DOMAIN_POLICIES` and allow `boards.greenhouse.io` → `job-boards.greenhouse.io` without
+  opening the whole internet.
+- **Sitemap under a Cloudflare 52x origin error stops without retries (1.6).** On the first `520`–`527` response to
+  robots.txt / sitemap, `SitemapParser` stops the sitemap phase and returns `result["origin_down"] = True` (logged as
+  `ORIGIN_DOWN`) instead of burning 100–180 s on retries/escalation against a down origin. New
+  `SitemapParser._is_origin_down`; `origin_down` is propagated through robots / default / nested sitemap paths.
+- **Nodes evicted by low-memory eviction are no longer lost in a browser full rescan (P23).**
+  `Graph.iter_nodes_async(include_evicted=True)` now also yields nodes that were evicted to disk (read-only, without
+  pulling them back into RAM), so a merge that iterates the source graph sees them. `Node.pinned` + `Graph.pin_node()`
+  / `Graph.pin_seed_nodes()` keep seed (`depth == 0`) vacancies in RAM — they were the oldest and got evicted first.
+  `Graph.eviction_stats` exposes the graph's own eviction counters (a backend-less graph evicts to LMDB/SQLite, which
+  `backend.get_stats()` reported as 0).
+- **A 403 storm without vendor markers is treated as a block (P24).** `EscalatingDriver` tracks a per-host window of
+  recent HTTP statuses; if a host that recently returned 2xx then returns 403 with no anti-bot markers and either ≥3
+  consecutive 403 or >30 % 403 in the window, the 403 is reclassified as `SOFT_BLOCK` (`escalation_reason=http_403_storm`)
+  and the tier escalates. A single 403 on a 2xx background is still treated as auth/method (no escalation). New
+  `_HostStatusWindow`.
+- **The HTML enrichment pool is priority-aware (2.1 / §15.4).** `run_in_html_pool(fn, *args, priority=...)` admits the
+  highest-priority waiter first when the pool is saturated (job-URLs and career subdomains enrich before catalog/blog
+  pages); with a free slot and empty queue it behaves exactly like the old FIFO semaphore. `Node` passes its URL-rule
+  priority. New `_PrioritySemaphore`.
+- **Cloudflare 520–527 is never a bot block.** `escalation._classify` returns `NONE` for 520–527 (as for 404/410):
+  the CF error page embeds `challenge-platform`, so a down origin was classified `challenge` and escalated
+  http→tls→browser (~20 s per tier) before `SitemapParser` could stop on `ORIGIN_DOWN`.
+- **robots.txt after a domain move keeps the new host's sitemap.** The same-origin gate and relative `Sitemap:`
+  paths now use the host that actually served robots.txt (`final_url`): `kilo.health` → `kilo.co/robots.txt` with
+  `Sitemap: … https://kilo.co/sitemap_index.xml` was dropped as cross-origin even after P19 cleaning.
+- **Cancelling a call waiting in the HTML-pool queue resumes the enrich budget.** With the priority semaphore the
+  acquire moved out of the `try`, so a cancellation in the queue left the node deadline suspended.
+- **`Node.pinned` does not leak into `user_data['_custom_fields']`** (`NodeMapper`, SQLite eviction storage base
+  fields) — it broke DTO/user-data invariants (`test_reserved_user_data_keys`, `test_node_mapper_content_depth`).
+- Tests: `tests/test_v535_fixes.py` (28, offline: P19, 404/52x, 1.6, P21 incl. end-to-end `async_crawl`, P23, P24, 2.1).
+
+### Added
+
+- `crawl` / `async_crawl` / `GraphCrawlerClient.crawl` / `CrawlerConfig`: `redirect_host_alias=` —
+  `Callable[[source_url, target_url], bool]` installed on the spider's `DomainFilter.redirect_host_alias`
+  (the extension point was otherwise unreachable through the public API).
+- `SitemapParser._clean_sitemap_value`, `SitemapParser._is_origin_down`; `result["origin_down"]` key.
+- `DomainFilter.allow_redirect(source_url, target_url)` and settable `DomainFilter.redirect_host_alias` hook.
+- `Node.pinned`; `Graph.pin_node`, `Graph.pin_seed_nodes`, `Graph.eviction_stats`; `iter_nodes_async(include_evicted=)`.
+- `run_in_html_pool(..., priority=)`.
+
+### Fixed (carried from Unreleased)
+
+- **Sitemap `<lastmod>` is no longer dropped.** `SitemapParser` reads `<lastmod>` of every `<url>` (with or without
+  namespace) into `result["lastmod"]` (`{url: value}`, merged across robots/default/nested sitemaps), and
+  `SitemapProcessor.create_url_nodes(lastmod=...)` stores it on the new `SitemapNode.lastmod` field (W3C datetime as
+  in the file). Consumers can now tell a 2021 post from a fresh page; `_parse_urlset` keeps its contract.
+
+- **An embedded captcha widget on a page with content is not a bot block.** `classify_block` follows the same
+  principle as the browser `CaptchaDetectorPlugin`: a block is a page served *instead of* content. After the
+  authoritative challenge headers (`cf-mitigated`, `x-amzn-waf-action`, `x-vercel-mitigated`), a non-blocking
+  response whose `<body>` has ≥1000 visible characters or ≥15 links is `NONE`, whatever widget or protection
+  script it embeds. Previously visible text was measured over the first 64 KB of the document (in WordPress that is
+  almost all `<head>`), and `cf-turnstile` / `challenge-platform` (Cloudflare loads its JSD script on regular pages)
+  were unconditional markers — so every page with reCAPTCHA/hCaptcha/Turnstile in a contact or apply form was
+  `captcha`, escalated HTTP→TLS and pinned the whole host to the TLS tier. `cf-turnstile`, `captcha-container`,
+  `challenge-platform`, `_incapsula_resource`, `awswafintegration` now count only on pages without content or with a
+  blocking status; markers and text are read from `<body>` up to 1 MB.
+
+## 5.3.4 — 2026-10
+
+### Fixed
+
+- **The per-node enrichment limit measures active work, not the HTML-pool queue.** `NodeScanner` wraps
+  `enrich_node` in `enrich_budget(seconds)` (`graph_crawler.domain.entities.node`): while a call waits for a thread
+  of the HTML executor in `run_in_html_pool` (parse, `_update_from_context`, hashes), the deadline is suspended and
+  resumed with the same remainder the moment the work starts. With 5.3.3 the hook and the hashes moved into the
+  shared pool, so under load a node's wall-clock depended on how many neighbours stood in front of it: a burst of
+  large pages produced a cascade of `Per-node timeout … marking as ERROR` for pages whose own processing took a
+  fraction of the limit (and every stage re-queued behind the others). A page whose **own** processing exceeds
+  `node_enrich_timeout` still times out and still returns its discovered links. The crawl-level timeout remains the
+  upper bound for a node that waits in the queue.
+- **`IsolatedExecutor.run` admits at most `max_workers` tasks at a time** (`threading.BoundedSemaphore`): the task
+  `timeout` now measures execution in the child process, not the wait behind other tasks. Previously a burst of
+  submissions could expire the timeout of a queued task, which killed the whole pool and failed every task in flight.
+
+- **Control commands are no longer dropped silently.** `GraphSpider.process_control_messages` executes only
+  `SPIDER_COMMANDS` (`STOP`, `PAUSE`, `RESUME`, `REPRIORITIZE`, `FORCE_VISIT`); any other `CrawlCommand` logs
+  `WARNING Unsupported control command '<value>' from <sender> is ignored (supported: ...)` once per command type
+  per crawl (repeats at `DEBUG`) and the crawl goes on. `REPRIORITIZE` without `url`/`priority` and `FORCE_VISIT`
+  without `url` log a `WARNING` with the received `data` instead of doing nothing.
+
+### Added
+
+- `EnrichBudget` (`pause()` / `resume()`) and `enrich_budget()` in `graph_crawler.domain.entities.node`.
+- `SPIDER_COMMANDS` (`graph_crawler.domain.interfaces`): the commands `GraphSpider` executes.
+
+### Deprecated
+
+- `CrawlCommand.SKIP_URL`, `CHANGE_STRATEGY`, `SET_DEPTH_LIMIT`, `REQUEST_STATE`, `UPDATE_GOAL` were never executed
+  by the spider. They will be removed from the enum in 5.4.0.
+
 ## 5.3.3 — 2026-10
 
 ### Added

{graph_crawler-5.3.3/graph_crawler.egg-info → graph_crawler-5.3.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: graph-crawler
-Version: 5.3.3
+Version: 5.3.5
 Summary: Web crawler that builds a graph of a site. HTTP → TLS → browser escalation with pluggable browser engines.
 Author: 0-EternalJunior-0
 Maintainer: 0-EternalJunior-0
@@ -139,34 +139,20 @@ pip install "graph-crawler[browser]"      # + browser (stock Playwright)
 pip install "graph-crawler[antidetect]"   # + TLS, patchright, CloakBrowser, nodriver
 ```
 
-<<<<<<< HEAD
-| extra                                                                                         | what it adds                                                       |
-| --------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ |
-| `browser`                                                                                     | `driver="playwright"` with bundled Chromium (`playwright` = alias) |
-| `tls`                                                                                         | `curl_cffi` — TLS fingerprint tier (`driver="tls"`)                |
-| `stealth`                                                                                     | `patchright` + `playwright-stealth` — engine `"patchright"`        |
-| `cloak`                                                                                       | `cloakbrowser` — engine `"cloakbrowser"` (stealth in the binary)   |
-| `undetected`                                                                                  | `nodriver` — last tier (`driver="nodriver"`)                       |
-| `antidetect`                                                                                  | all four above                                                     |
-| `server`                                                                                      | FastAPI dashboard / REST / WebSocket (`graph_crawler.api.server`)  |
-| `text`                                                                                        | `tiktoken` + `stop-words` — `count_tokens`, `extract_keywords`     |
-| `markdown`                                                                                    | `html2md-clean` + `markdown-it-py` — `MarkdownPlugin`              |
-| `mongodb`, `postgresql`, `celery`, `viz`, `articles`, `fuzzy`, `performance`, `system`, `all` | storage backends, distributed mode, extras                         |
-=======
-| extra        | what it adds                                                        |
-|--------------|---------------------------------------------------------------------|
-| `browser`    | `driver="playwright"` (`playwright` is an alias of this extra)      |
-| `tls`        | `curl_cffi` — TLS fingerprint tier, `driver="tls"`                  |
-| `stealth`    | `patchright` + `playwright-stealth` — engine `"patchright"`         |
-| `cloak`      | `cloakbrowser` — engine `"cloakbrowser"` (stealth in the binary)    |
-| `undetected` | `nodriver` — last tier, `driver="nodriver"`                         |
-| `antidetect` | the four extras above                                               |
-| `server`     | FastAPI dashboard / REST / WebSocket                                |
-| `text`       | `tiktoken` + `stop-words`                                           |
-| `markdown`   | `html2md-clean` + `markdown-it-py` — `MarkdownPlugin`               |
-| `fuzzy`      | `rapidfuzz` — `graph.search(..., mode="fuzzy")`                     |
-| `mongodb`, `postgresql`, `celery`, `viz`, `articles`, `sitemap-fast`, `performance`, `system`, `all` | storage backends, distributed mode and other extras |
->>>>>>> d3db85cd9c8826d11c10e33988c837f3e8ffbe88
+| extra          | what it adds                                                                    |
+| -------------- | ------------------------------------------------------------------------------- |
+| `browser`      | `driver="playwright"`, stock Playwright (`playwright` is an alias of this extra) |
+| `tls`          | `curl_cffi` — TLS fingerprint tier, `driver="tls"`                              |
+| `stealth`      | `patchright` + `playwright-stealth` — engine `"patchright"`                     |
+| `cloak`        | `cloakbrowser` — engine `"cloakbrowser"` (stealth in the binary)                |
+| `undetected`   | `nodriver` — last tier, `driver="nodriver"`                                     |
+| `antidetect`   | the five extras above                                                           |
+| `server`       | FastAPI dashboard / REST / WebSocket (`graph_crawler.api.server`)               |
+| `text`         | `tiktoken` + `stop-words` — `count_tokens`, `extract_keywords`                  |
+| `markdown`     | `html2md-clean` + `markdown-it-py` — `MarkdownPlugin`                           |
+| `fuzzy`        | `rapidfuzz` — `graph.search(..., mode="fuzzy")`                                 |
+| `sitemap-fast` | `lxml` sitemap parser                                                           |
+| `mongodb`, `postgresql`, `celery`, `viz`, `articles`, `performance`, `system`, `all` | storage backends, distributed mode and other extras |
 
 After installing a browser extra, download the browser once:
 
@@ -267,23 +253,13 @@ so `node.ok` is `False` for error pages.
 
 `driver=` accepts a name or a driver instance.
 
-<<<<<<< HEAD
-| name                | tier        | needs                 | notes                                           |
-| ------------------- | ----------- | --------------------- | ----------------------------------------------- |
-| `http` / `async`    | http        | —                     | aiohttp, default                                |
-| `tls` / `curl_cffi` | tls         | `[tls]`               | Chrome TLS fingerprint, no JS                   |
-| `playwright`        | browser     | `[playwright]`        | real browser; engine chosen by `browser_engine` |
-| `nodriver`          | undetected  | `[undetected]`        | CDP without webdriver, async                    |
-| `escalating`        | all of them | whatever is installed | http → tls → browser → undetected               |
-=======
-| name         | tier        | needs                 | notes                                           |
-|--------------|-------------|-----------------------|-------------------------------------------------|
-| `aiohttp` / `http` | http  | —                     | default                                         |
-| `tls`        | tls         | `[tls]`               | Chrome TLS fingerprint, no JavaScript           |
-| `playwright` | browser     | `[browser]`           | real browser; engine chosen by `browser_engine` |
-| `nodriver`   | undetected  | `[undetected]`        | CDP without webdriver                           |
-| `escalating` | all of them | whatever is installed | http → tls → browser → undetected               |
->>>>>>> d3db85cd9c8826d11c10e33988c837f3e8ffbe88
+| name                          | tier        | needs                 | notes                                           |
+| ----------------------------- | ----------- | --------------------- | ----------------------------------------------- |
+| `aiohttp` / `http` / `async`  | http        | —                     | default                                         |
+| `tls` / `curl_cffi`           | tls         | `[tls]`               | Chrome TLS fingerprint, no JavaScript           |
+| `playwright`                  | browser     | `[browser]`           | real browser; engine chosen by `browser_engine` |
+| `nodriver`                    | undetected  | `[undetected]`        | CDP without webdriver                           |
+| `escalating`                  | all of them | whatever is installed | http → tls → browser → undetected               |
 
 `escalating` creates tiers lazily: a browser starts only when a cheaper tier is
 blocked. Tiers whose dependency is not installed are skipped.
@@ -299,13 +275,8 @@ Every response goes through `classify_block()`:
 | `none`       | normal page                                                                  |
 | `soft_block` | 429, plain 503, or 200 with an empty body                                    |
 | `challenge`  | Cloudflare "Just a moment", `cf-mitigated: challenge`, Incapsula, DDoS-Guard |
-<<<<<<< HEAD
-| `hard_block` | 403 "Access Denied", Akamai `_abck`, DataDome headers                        |
-| `captcha`    | PerimeterX, DataDome captcha, reCAPTCHA / hCaptcha / Turnstile markup        |
-=======
 | `hard_block` | 403/401/405 **with** a vendor or "Access Denied" marker (Akamai `_abck`, DataDome/Incapsula headers, `x-iinfo`, Cloudflare `server`) |
-| `captcha`    | PerimeterX, DataDome captcha, reCAPTCHA / hCaptcha / Turnstile markup   |
->>>>>>> d3db85cd9c8826d11c10e33988c837f3e8ffbe88
+| `captcha`    | PerimeterX, DataDome captcha, reCAPTCHA / hCaptcha / Turnstile markup        |
 
 Anything other than `none`, and network errors, move the request to the next tier.
 
@@ -375,25 +346,14 @@ policy = EscalationPolicy(
 
 ### Browser engines
 
-<<<<<<< HEAD
-The browser tier is stock `PlaywrightDriver`; the _engine_ decides which
-Playwright module and which Chromium binary it launches.
-
-| engine         | what it is                                                    | falls back to |
-| -------------- | ------------------------------------------------------------- | ------------- |
-| `cloakbrowser` | Chromium with fingerprint patches compiled into the binary    | `patchright`  |
-| `patchright`   | Playwright fork without automation leaks (`Runtime.enable` …) | `playwright`  |
-| `playwright`   | stock                                                         | —             |
-=======
-The browser tier is `PlaywrightDriver`; the *engine* decides which Playwright
+The browser tier is `PlaywrightDriver`; the _engine_ decides which Playwright
 module and which Chromium binary it launches.
 
 | engine         | what it is                                                    | falls back to |
-|----------------|---------------------------------------------------------------|---------------|
+| -------------- | ------------------------------------------------------------- | ------------- |
 | `cloakbrowser` | Chromium with fingerprint patches compiled into the binary    | `patchright`  |
 | `patchright`   | Playwright fork without automation leaks (`Runtime.enable` …) | `playwright`  |
 | `playwright`   | stock Playwright                                              | —             |
->>>>>>> d3db85cd9c8826d11c10e33988c837f3e8ffbe88
 
 A fallback is logged once at `INFO` with the reason. To get an `ImportError`
 instead of a silent downgrade, set `GC_ENGINE_STRICT=1`, or pass
@@ -456,6 +416,37 @@ A target is found by `selector`, `text`, `placeholder` or `label` (`nth` picks
 among visible matches). The first failing action stops the chain and is written
 to `node.metadata["actions_error"]`.
 
+## Controlling a running crawl
+
+`async_crawl(..., control_channel=channel)` (and `crawl`, `GraphCrawlerClient.crawl`)
+reads commands from the channel between pages:
+
+```python
+import graph_crawler as gc
+from graph_crawler.domain.interfaces import AsyncQueueControlChannel, force_visit_command, stop_command
+
+channel = AsyncQueueControlChannel()
+channel.send(force_visit_command("https://example.com/hidden-page"))
+graph = await gc.async_crawl("https://example.com", control_channel=channel)
+
+# from a plugin, a callback or another task while the crawl runs:
+channel.send(stop_command("enough data"))   # graph.metadata["stop_reason"] == "enough data"
+```
+
+| command        | `data`                  | effect                                         |
+| -------------- | ----------------------- | ---------------------------------------------- |
+| `STOP`         | `reason` (optional)     | stops the crawl, `stop_reason` = `reason`      |
+| `PAUSE`        | —                       | pauses until `RESUME` or `STOP`                |
+| `RESUME`       | —                       | continues a paused crawl                       |
+| `REPRIORITIZE` | `url`, `priority`       | moves a queued URL in the queue                |
+| `FORCE_VISIT`  | `url`                   | adds the URL to the queue with top priority    |
+
+`SPIDER_COMMANDS` is this set. `SKIP_URL`, `CHANGE_STRATEGY`, `SET_DEPTH_LIMIT`,
+`REQUEST_STATE` and `UPDATE_GOAL` are declared in `CrawlCommand` but not executed:
+the spider logs `WARNING Unsupported control command ...` (once per command type
+per crawl) and goes on; they will be removed in the next minor release. A supported
+command without its required `data` fields is also skipped with a `WARNING`.
+
 ## Plugins and custom nodes
 
 A node plugin runs on every fetched page and writes to `user_data`, `metadata`

{graph_crawler-5.3.3 → graph_crawler-5.3.5}/README.md

@@ -18,34 +18,20 @@ pip install "graph-crawler[browser]"      # + browser (stock Playwright)
 pip install "graph-crawler[antidetect]"   # + TLS, patchright, CloakBrowser, nodriver
 ```
 
-<<<<<<< HEAD
-| extra                                                                                         | what it adds                                                       |
-| --------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ |
-| `browser`                                                                                     | `driver="playwright"` with bundled Chromium (`playwright` = alias) |
-| `tls`                                                                                         | `curl_cffi` — TLS fingerprint tier (`driver="tls"`)                |
-| `stealth`                                                                                     | `patchright` + `playwright-stealth` — engine `"patchright"`        |
-| `cloak`                                                                                       | `cloakbrowser` — engine `"cloakbrowser"` (stealth in the binary)   |
-| `undetected`                                                                                  | `nodriver` — last tier (`driver="nodriver"`)                       |
-| `antidetect`                                                                                  | all four above                                                     |
-| `server`                                                                                      | FastAPI dashboard / REST / WebSocket (`graph_crawler.api.server`)  |
-| `text`                                                                                        | `tiktoken` + `stop-words` — `count_tokens`, `extract_keywords`     |
-| `markdown`                                                                                    | `html2md-clean` + `markdown-it-py` — `MarkdownPlugin`              |
-| `mongodb`, `postgresql`, `celery`, `viz`, `articles`, `fuzzy`, `performance`, `system`, `all` | storage backends, distributed mode, extras                         |
-=======
-| extra        | what it adds                                                        |
-|--------------|---------------------------------------------------------------------|
-| `browser`    | `driver="playwright"` (`playwright` is an alias of this extra)      |
-| `tls`        | `curl_cffi` — TLS fingerprint tier, `driver="tls"`                  |
-| `stealth`    | `patchright` + `playwright-stealth` — engine `"patchright"`         |
-| `cloak`      | `cloakbrowser` — engine `"cloakbrowser"` (stealth in the binary)    |
-| `undetected` | `nodriver` — last tier, `driver="nodriver"`                         |
-| `antidetect` | the four extras above                                               |
-| `server`     | FastAPI dashboard / REST / WebSocket                                |
-| `text`       | `tiktoken` + `stop-words`                                           |
-| `markdown`   | `html2md-clean` + `markdown-it-py` — `MarkdownPlugin`               |
-| `fuzzy`      | `rapidfuzz` — `graph.search(..., mode="fuzzy")`                     |
-| `mongodb`, `postgresql`, `celery`, `viz`, `articles`, `sitemap-fast`, `performance`, `system`, `all` | storage backends, distributed mode and other extras |
->>>>>>> d3db85cd9c8826d11c10e33988c837f3e8ffbe88
+| extra          | what it adds                                                                    |
+| -------------- | ------------------------------------------------------------------------------- |
+| `browser`      | `driver="playwright"`, stock Playwright (`playwright` is an alias of this extra) |
+| `tls`          | `curl_cffi` — TLS fingerprint tier, `driver="tls"`                              |
+| `stealth`      | `patchright` + `playwright-stealth` — engine `"patchright"`                     |
+| `cloak`        | `cloakbrowser` — engine `"cloakbrowser"` (stealth in the binary)                |
+| `undetected`   | `nodriver` — last tier, `driver="nodriver"`                                     |
+| `antidetect`   | the five extras above                                                           |
+| `server`       | FastAPI dashboard / REST / WebSocket (`graph_crawler.api.server`)               |
+| `text`         | `tiktoken` + `stop-words` — `count_tokens`, `extract_keywords`                  |
+| `markdown`     | `html2md-clean` + `markdown-it-py` — `MarkdownPlugin`                           |
+| `fuzzy`        | `rapidfuzz` — `graph.search(..., mode="fuzzy")`                                 |
+| `sitemap-fast` | `lxml` sitemap parser                                                           |
+| `mongodb`, `postgresql`, `celery`, `viz`, `articles`, `performance`, `system`, `all` | storage backends, distributed mode and other extras |
 
 After installing a browser extra, download the browser once:
 
@@ -146,23 +132,13 @@ so `node.ok` is `False` for error pages.
 
 `driver=` accepts a name or a driver instance.
 
-<<<<<<< HEAD
-| name                | tier        | needs                 | notes                                           |
-| ------------------- | ----------- | --------------------- | ----------------------------------------------- |
-| `http` / `async`    | http        | —                     | aiohttp, default                                |
-| `tls` / `curl_cffi` | tls         | `[tls]`               | Chrome TLS fingerprint, no JS                   |
-| `playwright`        | browser     | `[playwright]`        | real browser; engine chosen by `browser_engine` |
-| `nodriver`          | undetected  | `[undetected]`        | CDP without webdriver, async                    |
-| `escalating`        | all of them | whatever is installed | http → tls → browser → undetected               |
-=======
-| name         | tier        | needs                 | notes                                           |
-|--------------|-------------|-----------------------|-------------------------------------------------|
-| `aiohttp` / `http` | http  | —                     | default                                         |
-| `tls`        | tls         | `[tls]`               | Chrome TLS fingerprint, no JavaScript           |
-| `playwright` | browser     | `[browser]`           | real browser; engine chosen by `browser_engine` |
-| `nodriver`   | undetected  | `[undetected]`        | CDP without webdriver                           |
-| `escalating` | all of them | whatever is installed | http → tls → browser → undetected               |
->>>>>>> d3db85cd9c8826d11c10e33988c837f3e8ffbe88
+| name                          | tier        | needs                 | notes                                           |
+| ----------------------------- | ----------- | --------------------- | ----------------------------------------------- |
+| `aiohttp` / `http` / `async`  | http        | —                     | default                                         |
+| `tls` / `curl_cffi`           | tls         | `[tls]`               | Chrome TLS fingerprint, no JavaScript           |
+| `playwright`                  | browser     | `[browser]`           | real browser; engine chosen by `browser_engine` |
+| `nodriver`                    | undetected  | `[undetected]`        | CDP without webdriver                           |
+| `escalating`                  | all of them | whatever is installed | http → tls → browser → undetected               |
 
 `escalating` creates tiers lazily: a browser starts only when a cheaper tier is
 blocked. Tiers whose dependency is not installed are skipped.
@@ -178,13 +154,8 @@ Every response goes through `classify_block()`:
 | `none`       | normal page                                                                  |
 | `soft_block` | 429, plain 503, or 200 with an empty body                                    |
 | `challenge`  | Cloudflare "Just a moment", `cf-mitigated: challenge`, Incapsula, DDoS-Guard |
-<<<<<<< HEAD
-| `hard_block` | 403 "Access Denied", Akamai `_abck`, DataDome headers                        |
-| `captcha`    | PerimeterX, DataDome captcha, reCAPTCHA / hCaptcha / Turnstile markup        |
-=======
 | `hard_block` | 403/401/405 **with** a vendor or "Access Denied" marker (Akamai `_abck`, DataDome/Incapsula headers, `x-iinfo`, Cloudflare `server`) |
-| `captcha`    | PerimeterX, DataDome captcha, reCAPTCHA / hCaptcha / Turnstile markup   |
->>>>>>> d3db85cd9c8826d11c10e33988c837f3e8ffbe88
+| `captcha`    | PerimeterX, DataDome captcha, reCAPTCHA / hCaptcha / Turnstile markup        |
 
 Anything other than `none`, and network errors, move the request to the next tier.
 
@@ -254,25 +225,14 @@ policy = EscalationPolicy(
 
 ### Browser engines
 
-<<<<<<< HEAD
-The browser tier is stock `PlaywrightDriver`; the _engine_ decides which
-Playwright module and which Chromium binary it launches.
-
-| engine         | what it is                                                    | falls back to |
-| -------------- | ------------------------------------------------------------- | ------------- |
-| `cloakbrowser` | Chromium with fingerprint patches compiled into the binary    | `patchright`  |
-| `patchright`   | Playwright fork without automation leaks (`Runtime.enable` …) | `playwright`  |
-| `playwright`   | stock                                                         | —             |
-=======
-The browser tier is `PlaywrightDriver`; the *engine* decides which Playwright
+The browser tier is `PlaywrightDriver`; the _engine_ decides which Playwright
 module and which Chromium binary it launches.
 
 | engine         | what it is                                                    | falls back to |
-|----------------|---------------------------------------------------------------|---------------|
+| -------------- | ------------------------------------------------------------- | ------------- |
 | `cloakbrowser` | Chromium with fingerprint patches compiled into the binary    | `patchright`  |
 | `patchright`   | Playwright fork without automation leaks (`Runtime.enable` …) | `playwright`  |
 | `playwright`   | stock Playwright                                              | —             |
->>>>>>> d3db85cd9c8826d11c10e33988c837f3e8ffbe88
 
 A fallback is logged once at `INFO` with the reason. To get an `ImportError`
 instead of a silent downgrade, set `GC_ENGINE_STRICT=1`, or pass
@@ -335,6 +295,37 @@ A target is found by `selector`, `text`, `placeholder` or `label` (`nth` picks
 among visible matches). The first failing action stops the chain and is written
 to `node.metadata["actions_error"]`.
 
+## Controlling a running crawl
+
+`async_crawl(..., control_channel=channel)` (and `crawl`, `GraphCrawlerClient.crawl`)
+reads commands from the channel between pages:
+
+```python
+import graph_crawler as gc
+from graph_crawler.domain.interfaces import AsyncQueueControlChannel, force_visit_command, stop_command
+
+channel = AsyncQueueControlChannel()
+channel.send(force_visit_command("https://example.com/hidden-page"))
+graph = await gc.async_crawl("https://example.com", control_channel=channel)
+
+# from a plugin, a callback or another task while the crawl runs:
+channel.send(stop_command("enough data"))   # graph.metadata["stop_reason"] == "enough data"
+```
+
+| command        | `data`                  | effect                                         |
+| -------------- | ----------------------- | ---------------------------------------------- |
+| `STOP`         | `reason` (optional)     | stops the crawl, `stop_reason` = `reason`      |
+| `PAUSE`        | —                       | pauses until `RESUME` or `STOP`                |
+| `RESUME`       | —                       | continues a paused crawl                       |
+| `REPRIORITIZE` | `url`, `priority`       | moves a queued URL in the queue                |
+| `FORCE_VISIT`  | `url`                   | adds the URL to the queue with top priority    |
+
+`SPIDER_COMMANDS` is this set. `SKIP_URL`, `CHANGE_STRATEGY`, `SET_DEPTH_LIMIT`,
+`REQUEST_STATE` and `UPDATE_GOAL` are declared in `CrawlCommand` but not executed:
+the spider logs `WARNING Unsupported control command ...` (once per command type
+per crawl) and goes on; they will be removed in the next minor release. A supported
+command without its required `data` fields is also skipped with a `WARNING`.
+
 ## Plugins and custom nodes
 
 A node plugin runs on every fetched page and writes to `user_data`, `metadata`

{graph_crawler-5.3.3 → graph_crawler-5.3.5}/graph_crawler/api/_core.py

@@ -68,6 +68,7 @@ async def async_crawl_impl(
     worker_pool_size: int | None = None,  # A_liffe L3
     per_host_max_in_flight: int | None = None,  # ліміт сторінок на хост (None = без ліміту)
     node_enrich_timeout: float | None = None,  # per-node таймаут збагачення, с (None = 45 с)
+    redirect_host_alias: Any = None,  # Callable[[src, dst], bool] — дозволені крос-хост редіректи
     driver: DriverType | None = None,
     driver_config: dict[str, Any] | None = None,
     storage: StorageType | None = None,
@@ -243,6 +244,7 @@ async def async_crawl_impl(
             worker_pool_size=worker_pool_size,
             per_host_max_in_flight=per_host_max_in_flight,
             node_enrich_timeout=node_enrich_timeout,
+            redirect_host_alias=redirect_host_alias,
             # Low-memory режим
             low_memory_mode=low_memory_mode,
             evict_threshold=evict_threshold,

{graph_crawler-5.3.3 → graph_crawler-5.3.5}/graph_crawler/api/async_.py

@@ -56,6 +56,7 @@ async def async_crawl(
     worker_pool_size: int | None = None,  # A_liffe L3
     per_host_max_in_flight: int | None = None,
     node_enrich_timeout: float | None = None,
+    redirect_host_alias: Any = None,
     driver: DriverType | None = None,
     driver_config: dict[str, Any] | None = None,
     storage: StorageType | None = None,
@@ -134,6 +135,7 @@ async def async_crawl(
         worker_pool_size=worker_pool_size,
         per_host_max_in_flight=per_host_max_in_flight,
         node_enrich_timeout=node_enrich_timeout,
+        redirect_host_alias=redirect_host_alias,
         driver=driver,
         driver_config=driver_config,
         storage=storage,

{graph_crawler-5.3.3 → graph_crawler-5.3.5}/graph_crawler/api/client/client.py

@@ -130,6 +130,8 @@ class GraphCrawlerClient:
         per_host_max_in_flight: int | None = None,
         # Per-node таймаут збагачення, с (None = DEFAULT_NODE_ENRICH_TIMEOUT)
         node_enrich_timeout: float | None = None,
+        # Callable[[source_url, target_url], bool] — дозволені крос-хост редіректи
+        redirect_host_alias: Any = None,
         # Low-memory режим
         low_memory_mode: bool = False,
         evict_threshold: int = 500,
@@ -195,6 +197,7 @@ class GraphCrawlerClient:
             worker_pool_size=worker_pool_size,
             per_host_max_in_flight=per_host_max_in_flight,
             node_enrich_timeout=node_enrich_timeout,
+            redirect_host_alias=redirect_host_alias,
             # Low-memory режим
             low_memory_mode=low_memory_mode,
             evict_threshold=evict_threshold,

{graph_crawler-5.3.3 → graph_crawler-5.3.5}/graph_crawler/api/sync.py

@@ -50,6 +50,7 @@ def crawl(
     worker_pool_size: int | None = None,
     per_host_max_in_flight: int | None = None,
     node_enrich_timeout: float | None = None,
+    redirect_host_alias: Any = None,
     driver: DriverType | None = None,
     driver_config: dict[str, Any] | None = None,
     storage: StorageType | None = None,
@@ -98,6 +99,8 @@ def crawl(
         per_host_max_in_flight: ліміт одночасних запитів на хост (``None`` — без ліміту).
         node_enrich_timeout: ліміт збагачення однієї ноди, с (``None`` — 45 с); при перевищенні
             нода стає ERROR, лінки, витягнуті до таймауту, лишаються в графі.
+        redirect_host_alias: ``Callable[[source_url, target_url], bool]`` — дозволяє редірект
+            на хост поза ``allowed_domains`` (напр. між хостами однієї ATS); ``None`` — ні.
         driver: драйвер завантаження (``"http"``, ``"playwright"``, ``"escalating"`` тощо).
         driver_config: налаштування драйвера.
         storage: тип сховища (``"memory"``, ``"json"``, ``"sqlite"`` тощо).
@@ -178,6 +181,7 @@ def crawl(
         "worker_pool_size": worker_pool_size,
         "per_host_max_in_flight": per_host_max_in_flight,
         "node_enrich_timeout": node_enrich_timeout,
+        "redirect_host_alias": redirect_host_alias,
         "driver": driver,
         "driver_config": driver_config,
         "storage": storage,

{graph_crawler-5.3.3 → graph_crawler-5.3.5}/graph_crawler/application/dto/mappers/node_mapper.py

@@ -59,6 +59,7 @@ class NodeMapper:
             "processed_at",
             "should_scan",
             "can_create_edges",
+            "pinned",
             "created_at",
             "metadata",
             "user_data",

{graph_crawler-5.3.3 → graph_crawler-5.3.5}/graph_crawler/application/use_cases/crawling/crawl_coordinator.py

@@ -1003,11 +1003,30 @@ class CrawlCoordinator:
         # Захист від cross-domain редіректів повз same_domain фільтр:
         # драйвер може прозоро фоловити Location: на чужий домен, тож
         # питаємо domain_filter перед тим як зафіксувати редірект у графі.
+        #
+        # P21: редірект на ТОЙ САМИЙ хост дозволяємо завжди (коротка форма ATS
+        # `apply.workable.com/j/X` → `apply.workable.com/<slug>/j/X`). Для різних
+        # хостів — domain_filter; якщо він відмовив, пробуємо точку розширення
+        # «аліаси хостів» (`allow_redirect`), якою Parser передає свої
+        # DOMAIN_POLICIES (`boards.greenhouse.io` → `job-boards.greenhouse.io`).
         try:
+            from urllib.parse import urlsplit
+
+            src_host = urlsplit(original_node.url).hostname or ""
+            tgt_host = urlsplit(final_url).hostname or ""
+            same_host = bool(src_host) and src_host == tgt_host
+
             domain_filter = getattr(self.processor, "domain_filter", None)
-            if domain_filter is not None and not domain_filter.is_allowed(
-                final_url, source_url=original_node.url
-            ):
+            blocked = False
+            if not same_host and domain_filter is not None:
+                if not domain_filter.is_allowed(final_url, source_url=original_node.url):
+                    alias_hook = getattr(domain_filter, "allow_redirect", None)
+                    allowed_by_alias = bool(
+                        alias_hook and alias_hook(original_node.url, final_url)
+                    )
+                    blocked = not allowed_by_alias
+
+            if blocked:
                 logger.info(
                     "Redirect blocked by domain_filter: %s -> %s (cross-domain or disallowed)",
                     original_node.url,

{graph_crawler-5.3.3 → graph_crawler-5.3.5}/graph_crawler/application/use_cases/crawling/filters/domain_filter.py

@@ -47,6 +47,13 @@ class DomainFilter(BaseURLFilter):
         super().__init__(config.model_dump() if hasattr(config, "model_dump") else dict(config))
         self.event_bus = event_bus
 
+        # P21: точка розширення «аліаси хостів». Callable[[source_url, target_url], bool].
+        # Коли редірект веде на хост поза allowed_domains, координатор питає цей
+        # хук — так Parser дозволяє редірект між хостами однієї ATS-політики
+        # (boards.greenhouse.io → job-boards.greenhouse.io), не відкриваючи весь
+        # інтернет. None → редірект за межі allowed_domains блокується, як раніше.
+        self.redirect_host_alias = None
+
         # Парсимо спеціальні патерни
         self._parse_special_patterns()
 
@@ -92,6 +99,30 @@ class DomainFilter(BaseURLFilter):
     def name(self) -> str:
         return "domain"
 
+    def allow_redirect(self, source_url: str, target_url: str) -> bool:
+        """Точка розширення P21: чи дозволити редірект на хост поза allowed_domains.
+
+        Координатор завжди дозволяє редірект на той самий хост сам; сюди
+        потрапляють лише крос-хостові редіректи, які не пройшли ``is_allowed``.
+        За замовчуванням ``False``; якщо встановлено ``redirect_host_alias``
+        (Parser передає свої DOMAIN_POLICIES) — делегуємо рішення йому.
+
+        Args:
+            source_url: URL, що редіректив.
+            target_url: Фінальний URL редіректу.
+
+        Returns:
+            True, якщо редірект на цей чужий хост дозволено як аліас.
+        """
+        hook = self.redirect_host_alias
+        if hook is None:
+            return False
+        try:
+            return bool(hook(source_url, target_url))
+        except Exception as exc:  # noqa: BLE001 — хук не повинен валити обхід
+            logger.warning("redirect_host_alias hook failed for %s -> %s: %s", source_url, target_url, exc)
+            return False
+
     def _is_subdomain_of(self, domain: str, base_domain: str) -> bool:
         """
         Перевіряє чи domain є субдоменом base_domain або самим base_domain.