graph-crawler 5.3.3__tar.gz → 5.3.4__tar.gz

{graph_crawler-5.3.3 → graph_crawler-5.3.4}/CHANGELOG.md

@@ -1,5 +1,38 @@
 # Changelog
 
+## 5.3.4 — 2026-10
+
+### Fixed
+
+- **The per-node enrichment limit measures active work, not the HTML-pool queue.** `NodeScanner` wraps
+  `enrich_node` in `enrich_budget(seconds)` (`graph_crawler.domain.entities.node`): while a call waits for a thread
+  of the HTML executor in `run_in_html_pool` (parse, `_update_from_context`, hashes), the deadline is suspended and
+  resumed with the same remainder the moment the work starts. With 5.3.3 the hook and the hashes moved into the
+  shared pool, so under load a node's wall-clock depended on how many neighbours stood in front of it: a burst of
+  large pages produced a cascade of `Per-node timeout … marking as ERROR` for pages whose own processing took a
+  fraction of the limit (and every stage re-queued behind the others). A page whose **own** processing exceeds
+  `node_enrich_timeout` still times out and still returns its discovered links. The crawl-level timeout remains the
+  upper bound for a node that waits in the queue.
+- **`IsolatedExecutor.run` admits at most `max_workers` tasks at a time** (`threading.BoundedSemaphore`): the task
+  `timeout` now measures execution in the child process, not the wait behind other tasks. Previously a burst of
+  submissions could expire the timeout of a queued task, which killed the whole pool and failed every task in flight.
+
+- **Control commands are no longer dropped silently.** `GraphSpider.process_control_messages` executes only
+  `SPIDER_COMMANDS` (`STOP`, `PAUSE`, `RESUME`, `REPRIORITIZE`, `FORCE_VISIT`); any other `CrawlCommand` logs
+  `WARNING Unsupported control command '<value>' from <sender> is ignored (supported: ...)` once per command type
+  per crawl (repeats at `DEBUG`) and the crawl goes on. `REPRIORITIZE` without `url`/`priority` and `FORCE_VISIT`
+  without `url` log a `WARNING` with the received `data` instead of doing nothing.
+
+### Added
+
+- `EnrichBudget` (`pause()` / `resume()`) and `enrich_budget()` in `graph_crawler.domain.entities.node`.
+- `SPIDER_COMMANDS` (`graph_crawler.domain.interfaces`): the commands `GraphSpider` executes.
+
+### Deprecated
+
+- `CrawlCommand.SKIP_URL`, `CHANGE_STRATEGY`, `SET_DEPTH_LIMIT`, `REQUEST_STATE`, `UPDATE_GOAL` were never executed
+  by the spider. They will be removed from the enum in 5.4.0.
+
 ## 5.3.3 — 2026-10
 
 ### Added

{graph_crawler-5.3.3/graph_crawler.egg-info → graph_crawler-5.3.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: graph-crawler
-Version: 5.3.3
+Version: 5.3.4
 Summary: Web crawler that builds a graph of a site. HTTP → TLS → browser escalation with pluggable browser engines.
 Author: 0-EternalJunior-0
 Maintainer: 0-EternalJunior-0
@@ -139,34 +139,20 @@ pip install "graph-crawler[browser]"      # + browser (stock Playwright)
 pip install "graph-crawler[antidetect]"   # + TLS, patchright, CloakBrowser, nodriver
 ```
 
-<<<<<<< HEAD
-| extra                                                                                         | what it adds                                                       |
-| --------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ |
-| `browser`                                                                                     | `driver="playwright"` with bundled Chromium (`playwright` = alias) |
-| `tls`                                                                                         | `curl_cffi` — TLS fingerprint tier (`driver="tls"`)                |
-| `stealth`                                                                                     | `patchright` + `playwright-stealth` — engine `"patchright"`        |
-| `cloak`                                                                                       | `cloakbrowser` — engine `"cloakbrowser"` (stealth in the binary)   |
-| `undetected`                                                                                  | `nodriver` — last tier (`driver="nodriver"`)                       |
-| `antidetect`                                                                                  | all four above                                                     |
-| `server`                                                                                      | FastAPI dashboard / REST / WebSocket (`graph_crawler.api.server`)  |
-| `text`                                                                                        | `tiktoken` + `stop-words` — `count_tokens`, `extract_keywords`     |
-| `markdown`                                                                                    | `html2md-clean` + `markdown-it-py` — `MarkdownPlugin`              |
-| `mongodb`, `postgresql`, `celery`, `viz`, `articles`, `fuzzy`, `performance`, `system`, `all` | storage backends, distributed mode, extras                         |
-=======
-| extra        | what it adds                                                        |
-|--------------|---------------------------------------------------------------------|
-| `browser`    | `driver="playwright"` (`playwright` is an alias of this extra)      |
-| `tls`        | `curl_cffi` — TLS fingerprint tier, `driver="tls"`                  |
-| `stealth`    | `patchright` + `playwright-stealth` — engine `"patchright"`         |
-| `cloak`      | `cloakbrowser` — engine `"cloakbrowser"` (stealth in the binary)    |
-| `undetected` | `nodriver` — last tier, `driver="nodriver"`                         |
-| `antidetect` | the four extras above                                               |
-| `server`     | FastAPI dashboard / REST / WebSocket                                |
-| `text`       | `tiktoken` + `stop-words`                                           |
-| `markdown`   | `html2md-clean` + `markdown-it-py` — `MarkdownPlugin`               |
-| `fuzzy`      | `rapidfuzz` — `graph.search(..., mode="fuzzy")`                     |
-| `mongodb`, `postgresql`, `celery`, `viz`, `articles`, `sitemap-fast`, `performance`, `system`, `all` | storage backends, distributed mode and other extras |
->>>>>>> d3db85cd9c8826d11c10e33988c837f3e8ffbe88
+| extra          | what it adds                                                                    |
+| -------------- | ------------------------------------------------------------------------------- |
+| `browser`      | `driver="playwright"`, stock Playwright (`playwright` is an alias of this extra) |
+| `tls`          | `curl_cffi` — TLS fingerprint tier, `driver="tls"`                              |
+| `stealth`      | `patchright` + `playwright-stealth` — engine `"patchright"`                     |
+| `cloak`        | `cloakbrowser` — engine `"cloakbrowser"` (stealth in the binary)                |
+| `undetected`   | `nodriver` — last tier, `driver="nodriver"`                                     |
+| `antidetect`   | the five extras above                                                           |
+| `server`       | FastAPI dashboard / REST / WebSocket (`graph_crawler.api.server`)               |
+| `text`         | `tiktoken` + `stop-words` — `count_tokens`, `extract_keywords`                  |
+| `markdown`     | `html2md-clean` + `markdown-it-py` — `MarkdownPlugin`                           |
+| `fuzzy`        | `rapidfuzz` — `graph.search(..., mode="fuzzy")`                                 |
+| `sitemap-fast` | `lxml` sitemap parser                                                           |
+| `mongodb`, `postgresql`, `celery`, `viz`, `articles`, `performance`, `system`, `all` | storage backends, distributed mode and other extras |
 
 After installing a browser extra, download the browser once:
 
@@ -267,23 +253,13 @@ so `node.ok` is `False` for error pages.
 
 `driver=` accepts a name or a driver instance.
 
-<<<<<<< HEAD
-| name                | tier        | needs                 | notes                                           |
-| ------------------- | ----------- | --------------------- | ----------------------------------------------- |
-| `http` / `async`    | http        | —                     | aiohttp, default                                |
-| `tls` / `curl_cffi` | tls         | `[tls]`               | Chrome TLS fingerprint, no JS                   |
-| `playwright`        | browser     | `[playwright]`        | real browser; engine chosen by `browser_engine` |
-| `nodriver`          | undetected  | `[undetected]`        | CDP without webdriver, async                    |
-| `escalating`        | all of them | whatever is installed | http → tls → browser → undetected               |
-=======
-| name         | tier        | needs                 | notes                                           |
-|--------------|-------------|-----------------------|-------------------------------------------------|
-| `aiohttp` / `http` | http  | —                     | default                                         |
-| `tls`        | tls         | `[tls]`               | Chrome TLS fingerprint, no JavaScript           |
-| `playwright` | browser     | `[browser]`           | real browser; engine chosen by `browser_engine` |
-| `nodriver`   | undetected  | `[undetected]`        | CDP without webdriver                           |
-| `escalating` | all of them | whatever is installed | http → tls → browser → undetected               |
->>>>>>> d3db85cd9c8826d11c10e33988c837f3e8ffbe88
+| name                          | tier        | needs                 | notes                                           |
+| ----------------------------- | ----------- | --------------------- | ----------------------------------------------- |
+| `aiohttp` / `http` / `async`  | http        | —                     | default                                         |
+| `tls` / `curl_cffi`           | tls         | `[tls]`               | Chrome TLS fingerprint, no JavaScript           |
+| `playwright`                  | browser     | `[browser]`           | real browser; engine chosen by `browser_engine` |
+| `nodriver`                    | undetected  | `[undetected]`        | CDP without webdriver                           |
+| `escalating`                  | all of them | whatever is installed | http → tls → browser → undetected               |
 
 `escalating` creates tiers lazily: a browser starts only when a cheaper tier is
 blocked. Tiers whose dependency is not installed are skipped.
@@ -299,13 +275,8 @@ Every response goes through `classify_block()`:
 | `none`       | normal page                                                                  |
 | `soft_block` | 429, plain 503, or 200 with an empty body                                    |
 | `challenge`  | Cloudflare "Just a moment", `cf-mitigated: challenge`, Incapsula, DDoS-Guard |
-<<<<<<< HEAD
-| `hard_block` | 403 "Access Denied", Akamai `_abck`, DataDome headers                        |
-| `captcha`    | PerimeterX, DataDome captcha, reCAPTCHA / hCaptcha / Turnstile markup        |
-=======
 | `hard_block` | 403/401/405 **with** a vendor or "Access Denied" marker (Akamai `_abck`, DataDome/Incapsula headers, `x-iinfo`, Cloudflare `server`) |
-| `captcha`    | PerimeterX, DataDome captcha, reCAPTCHA / hCaptcha / Turnstile markup   |
->>>>>>> d3db85cd9c8826d11c10e33988c837f3e8ffbe88
+| `captcha`    | PerimeterX, DataDome captcha, reCAPTCHA / hCaptcha / Turnstile markup        |
 
 Anything other than `none`, and network errors, move the request to the next tier.
 
@@ -375,25 +346,14 @@ policy = EscalationPolicy(
 
 ### Browser engines
 
-<<<<<<< HEAD
-The browser tier is stock `PlaywrightDriver`; the _engine_ decides which
-Playwright module and which Chromium binary it launches.
-
-| engine         | what it is                                                    | falls back to |
-| -------------- | ------------------------------------------------------------- | ------------- |
-| `cloakbrowser` | Chromium with fingerprint patches compiled into the binary    | `patchright`  |
-| `patchright`   | Playwright fork without automation leaks (`Runtime.enable` …) | `playwright`  |
-| `playwright`   | stock                                                         | —             |
-=======
-The browser tier is `PlaywrightDriver`; the *engine* decides which Playwright
+The browser tier is `PlaywrightDriver`; the _engine_ decides which Playwright
 module and which Chromium binary it launches.
 
 | engine         | what it is                                                    | falls back to |
-|----------------|---------------------------------------------------------------|---------------|
+| -------------- | ------------------------------------------------------------- | ------------- |
 | `cloakbrowser` | Chromium with fingerprint patches compiled into the binary    | `patchright`  |
 | `patchright`   | Playwright fork without automation leaks (`Runtime.enable` …) | `playwright`  |
 | `playwright`   | stock Playwright                                              | —             |
->>>>>>> d3db85cd9c8826d11c10e33988c837f3e8ffbe88
 
 A fallback is logged once at `INFO` with the reason. To get an `ImportError`
 instead of a silent downgrade, set `GC_ENGINE_STRICT=1`, or pass
@@ -456,6 +416,37 @@ A target is found by `selector`, `text`, `placeholder` or `label` (`nth` picks
 among visible matches). The first failing action stops the chain and is written
 to `node.metadata["actions_error"]`.
 
+## Controlling a running crawl
+
+`async_crawl(..., control_channel=channel)` (and `crawl`, `GraphCrawlerClient.crawl`)
+reads commands from the channel between pages:
+
+```python
+import graph_crawler as gc
+from graph_crawler.domain.interfaces import AsyncQueueControlChannel, force_visit_command, stop_command
+
+channel = AsyncQueueControlChannel()
+channel.send(force_visit_command("https://example.com/hidden-page"))
+graph = await gc.async_crawl("https://example.com", control_channel=channel)
+
+# from a plugin, a callback or another task while the crawl runs:
+channel.send(stop_command("enough data"))   # graph.metadata["stop_reason"] == "enough data"
+```
+
+| command        | `data`                  | effect                                         |
+| -------------- | ----------------------- | ---------------------------------------------- |
+| `STOP`         | `reason` (optional)     | stops the crawl, `stop_reason` = `reason`      |
+| `PAUSE`        | —                       | pauses until `RESUME` or `STOP`                |
+| `RESUME`       | —                       | continues a paused crawl                       |
+| `REPRIORITIZE` | `url`, `priority`       | moves a queued URL in the queue                |
+| `FORCE_VISIT`  | `url`                   | adds the URL to the queue with top priority    |
+
+`SPIDER_COMMANDS` is this set. `SKIP_URL`, `CHANGE_STRATEGY`, `SET_DEPTH_LIMIT`,
+`REQUEST_STATE` and `UPDATE_GOAL` are declared in `CrawlCommand` but not executed:
+the spider logs `WARNING Unsupported control command ...` (once per command type
+per crawl) and goes on; they will be removed in the next minor release. A supported
+command without its required `data` fields is also skipped with a `WARNING`.
+
 ## Plugins and custom nodes
 
 A node plugin runs on every fetched page and writes to `user_data`, `metadata`

{graph_crawler-5.3.3 → graph_crawler-5.3.4}/README.md

@@ -18,34 +18,20 @@ pip install "graph-crawler[browser]"      # + browser (stock Playwright)
 pip install "graph-crawler[antidetect]"   # + TLS, patchright, CloakBrowser, nodriver
 ```
 
-<<<<<<< HEAD
-| extra                                                                                         | what it adds                                                       |
-| --------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ |
-| `browser`                                                                                     | `driver="playwright"` with bundled Chromium (`playwright` = alias) |
-| `tls`                                                                                         | `curl_cffi` — TLS fingerprint tier (`driver="tls"`)                |
-| `stealth`                                                                                     | `patchright` + `playwright-stealth` — engine `"patchright"`        |
-| `cloak`                                                                                       | `cloakbrowser` — engine `"cloakbrowser"` (stealth in the binary)   |
-| `undetected`                                                                                  | `nodriver` — last tier (`driver="nodriver"`)                       |
-| `antidetect`                                                                                  | all four above                                                     |
-| `server`                                                                                      | FastAPI dashboard / REST / WebSocket (`graph_crawler.api.server`)  |
-| `text`                                                                                        | `tiktoken` + `stop-words` — `count_tokens`, `extract_keywords`     |
-| `markdown`                                                                                    | `html2md-clean` + `markdown-it-py` — `MarkdownPlugin`              |
-| `mongodb`, `postgresql`, `celery`, `viz`, `articles`, `fuzzy`, `performance`, `system`, `all` | storage backends, distributed mode, extras                         |
-=======
-| extra        | what it adds                                                        |
-|--------------|---------------------------------------------------------------------|
-| `browser`    | `driver="playwright"` (`playwright` is an alias of this extra)      |
-| `tls`        | `curl_cffi` — TLS fingerprint tier, `driver="tls"`                  |
-| `stealth`    | `patchright` + `playwright-stealth` — engine `"patchright"`         |
-| `cloak`      | `cloakbrowser` — engine `"cloakbrowser"` (stealth in the binary)    |
-| `undetected` | `nodriver` — last tier, `driver="nodriver"`                         |
-| `antidetect` | the four extras above                                               |
-| `server`     | FastAPI dashboard / REST / WebSocket                                |
-| `text`       | `tiktoken` + `stop-words`                                           |
-| `markdown`   | `html2md-clean` + `markdown-it-py` — `MarkdownPlugin`               |
-| `fuzzy`      | `rapidfuzz` — `graph.search(..., mode="fuzzy")`                     |
-| `mongodb`, `postgresql`, `celery`, `viz`, `articles`, `sitemap-fast`, `performance`, `system`, `all` | storage backends, distributed mode and other extras |
->>>>>>> d3db85cd9c8826d11c10e33988c837f3e8ffbe88
+| extra          | what it adds                                                                    |
+| -------------- | ------------------------------------------------------------------------------- |
+| `browser`      | `driver="playwright"`, stock Playwright (`playwright` is an alias of this extra) |
+| `tls`          | `curl_cffi` — TLS fingerprint tier, `driver="tls"`                              |
+| `stealth`      | `patchright` + `playwright-stealth` — engine `"patchright"`                     |
+| `cloak`        | `cloakbrowser` — engine `"cloakbrowser"` (stealth in the binary)                |
+| `undetected`   | `nodriver` — last tier, `driver="nodriver"`                                     |
+| `antidetect`   | the five extras above                                                           |
+| `server`       | FastAPI dashboard / REST / WebSocket (`graph_crawler.api.server`)               |
+| `text`         | `tiktoken` + `stop-words` — `count_tokens`, `extract_keywords`                  |
+| `markdown`     | `html2md-clean` + `markdown-it-py` — `MarkdownPlugin`                           |
+| `fuzzy`        | `rapidfuzz` — `graph.search(..., mode="fuzzy")`                                 |
+| `sitemap-fast` | `lxml` sitemap parser                                                           |
+| `mongodb`, `postgresql`, `celery`, `viz`, `articles`, `performance`, `system`, `all` | storage backends, distributed mode and other extras |
 
 After installing a browser extra, download the browser once:
 
@@ -146,23 +132,13 @@ so `node.ok` is `False` for error pages.
 
 `driver=` accepts a name or a driver instance.
 
-<<<<<<< HEAD
-| name                | tier        | needs                 | notes                                           |
-| ------------------- | ----------- | --------------------- | ----------------------------------------------- |
-| `http` / `async`    | http        | —                     | aiohttp, default                                |
-| `tls` / `curl_cffi` | tls         | `[tls]`               | Chrome TLS fingerprint, no JS                   |
-| `playwright`        | browser     | `[playwright]`        | real browser; engine chosen by `browser_engine` |
-| `nodriver`          | undetected  | `[undetected]`        | CDP without webdriver, async                    |
-| `escalating`        | all of them | whatever is installed | http → tls → browser → undetected               |
-=======
-| name         | tier        | needs                 | notes                                           |
-|--------------|-------------|-----------------------|-------------------------------------------------|
-| `aiohttp` / `http` | http  | —                     | default                                         |
-| `tls`        | tls         | `[tls]`               | Chrome TLS fingerprint, no JavaScript           |
-| `playwright` | browser     | `[browser]`           | real browser; engine chosen by `browser_engine` |
-| `nodriver`   | undetected  | `[undetected]`        | CDP without webdriver                           |
-| `escalating` | all of them | whatever is installed | http → tls → browser → undetected               |
->>>>>>> d3db85cd9c8826d11c10e33988c837f3e8ffbe88
+| name                          | tier        | needs                 | notes                                           |
+| ----------------------------- | ----------- | --------------------- | ----------------------------------------------- |
+| `aiohttp` / `http` / `async`  | http        | —                     | default                                         |
+| `tls` / `curl_cffi`           | tls         | `[tls]`               | Chrome TLS fingerprint, no JavaScript           |
+| `playwright`                  | browser     | `[browser]`           | real browser; engine chosen by `browser_engine` |
+| `nodriver`                    | undetected  | `[undetected]`        | CDP without webdriver                           |
+| `escalating`                  | all of them | whatever is installed | http → tls → browser → undetected               |
 
 `escalating` creates tiers lazily: a browser starts only when a cheaper tier is
 blocked. Tiers whose dependency is not installed are skipped.
@@ -178,13 +154,8 @@ Every response goes through `classify_block()`:
 | `none`       | normal page                                                                  |
 | `soft_block` | 429, plain 503, or 200 with an empty body                                    |
 | `challenge`  | Cloudflare "Just a moment", `cf-mitigated: challenge`, Incapsula, DDoS-Guard |
-<<<<<<< HEAD
-| `hard_block` | 403 "Access Denied", Akamai `_abck`, DataDome headers                        |
-| `captcha`    | PerimeterX, DataDome captcha, reCAPTCHA / hCaptcha / Turnstile markup        |
-=======
 | `hard_block` | 403/401/405 **with** a vendor or "Access Denied" marker (Akamai `_abck`, DataDome/Incapsula headers, `x-iinfo`, Cloudflare `server`) |
-| `captcha`    | PerimeterX, DataDome captcha, reCAPTCHA / hCaptcha / Turnstile markup   |
->>>>>>> d3db85cd9c8826d11c10e33988c837f3e8ffbe88
+| `captcha`    | PerimeterX, DataDome captcha, reCAPTCHA / hCaptcha / Turnstile markup        |
 
 Anything other than `none`, and network errors, move the request to the next tier.
 
@@ -254,25 +225,14 @@ policy = EscalationPolicy(
 
 ### Browser engines
 
-<<<<<<< HEAD
-The browser tier is stock `PlaywrightDriver`; the _engine_ decides which
-Playwright module and which Chromium binary it launches.
-
-| engine         | what it is                                                    | falls back to |
-| -------------- | ------------------------------------------------------------- | ------------- |
-| `cloakbrowser` | Chromium with fingerprint patches compiled into the binary    | `patchright`  |
-| `patchright`   | Playwright fork without automation leaks (`Runtime.enable` …) | `playwright`  |
-| `playwright`   | stock                                                         | —             |
-=======
-The browser tier is `PlaywrightDriver`; the *engine* decides which Playwright
+The browser tier is `PlaywrightDriver`; the _engine_ decides which Playwright
 module and which Chromium binary it launches.
 
 | engine         | what it is                                                    | falls back to |
-|----------------|---------------------------------------------------------------|---------------|
+| -------------- | ------------------------------------------------------------- | ------------- |
 | `cloakbrowser` | Chromium with fingerprint patches compiled into the binary    | `patchright`  |
 | `patchright`   | Playwright fork without automation leaks (`Runtime.enable` …) | `playwright`  |
 | `playwright`   | stock Playwright                                              | —             |
->>>>>>> d3db85cd9c8826d11c10e33988c837f3e8ffbe88
 
 A fallback is logged once at `INFO` with the reason. To get an `ImportError`
 instead of a silent downgrade, set `GC_ENGINE_STRICT=1`, or pass
@@ -335,6 +295,37 @@ A target is found by `selector`, `text`, `placeholder` or `label` (`nth` picks
 among visible matches). The first failing action stops the chain and is written
 to `node.metadata["actions_error"]`.
 
+## Controlling a running crawl
+
+`async_crawl(..., control_channel=channel)` (and `crawl`, `GraphCrawlerClient.crawl`)
+reads commands from the channel between pages:
+
+```python
+import graph_crawler as gc
+from graph_crawler.domain.interfaces import AsyncQueueControlChannel, force_visit_command, stop_command
+
+channel = AsyncQueueControlChannel()
+channel.send(force_visit_command("https://example.com/hidden-page"))
+graph = await gc.async_crawl("https://example.com", control_channel=channel)
+
+# from a plugin, a callback or another task while the crawl runs:
+channel.send(stop_command("enough data"))   # graph.metadata["stop_reason"] == "enough data"
+```
+
+| command        | `data`                  | effect                                         |
+| -------------- | ----------------------- | ---------------------------------------------- |
+| `STOP`         | `reason` (optional)     | stops the crawl, `stop_reason` = `reason`      |
+| `PAUSE`        | —                       | pauses until `RESUME` or `STOP`                |
+| `RESUME`       | —                       | continues a paused crawl                       |
+| `REPRIORITIZE` | `url`, `priority`       | moves a queued URL in the queue                |
+| `FORCE_VISIT`  | `url`                   | adds the URL to the queue with top priority    |
+
+`SPIDER_COMMANDS` is this set. `SKIP_URL`, `CHANGE_STRATEGY`, `SET_DEPTH_LIMIT`,
+`REQUEST_STATE` and `UPDATE_GOAL` are declared in `CrawlCommand` but not executed:
+the spider logs `WARNING Unsupported control command ...` (once per command type
+per crawl) and goes on; they will be removed in the next minor release. A supported
+command without its required `data` fields is also skipped with a `WARNING`.
+
 ## Plugins and custom nodes
 
 A node plugin runs on every fetched page and writes to `user_data`, `metadata`

{graph_crawler-5.3.3 → graph_crawler-5.3.4}/graph_crawler/application/use_cases/crawling/node_scanner.py

@@ -13,7 +13,7 @@ import re
 from typing import Any
 from urllib.parse import urlsplit
 
-from graph_crawler.domain.entities.node import Node
+from graph_crawler.domain.entities.node import Node, enrich_budget
 from graph_crawler.domain.interfaces.driver import IDriver
 from graph_crawler.domain.interfaces.node_interfaces import IRetainsRawResponse
 from graph_crawler.domain.value_objects.lifecycle import NodeLifecycle
@@ -627,9 +627,9 @@ class NodeScanner:
             if result is not None and result.html is not None:
                 html = result.html
                 try:
-                    # asyncio.timeout замість wait_for: у Python 3.11 wait_for губить
-                    # зовнішнє скасування (Ctrl+C), якщо enrich завершився одночасно з ним.
-                    async with asyncio.timeout(self.enrich_timeout):
+                    # enrich_budget (над asyncio.timeout, не wait_for: у Python 3.11 wait_for
+                    # губить зовнішнє скасування). Черга html-пулу в ліміт не входить.
+                    async with enrich_budget(self.enrich_timeout):
                         links = await node.enrich_node(
                             html,
                             crawl_context=crawl_context,
@@ -799,7 +799,7 @@ class NodeScanner:
             loop = asyncio.get_running_loop()
             started = loop.time()
             try:
-                async with asyncio.timeout(self.enrich_timeout):
+                async with enrich_budget(self.enrich_timeout):
                     return await enrich_single_node(node, result)
             except TimeoutError:
                 return (node, self._on_enrich_timeout(node), None)

{graph_crawler-5.3.3 → graph_crawler-5.3.4}/graph_crawler/application/use_cases/crawling/spider.py

@@ -41,7 +41,9 @@ from graph_crawler.domain.entities.graph import Graph
 from graph_crawler.domain.entities.node import Node
 from graph_crawler.domain.events import CrawlerEvent, EventBus, EventType
 from graph_crawler.domain.interfaces.control_channel import (
+    SPIDER_COMMANDS,
     AsyncQueueControlChannel,
+    ControlMessage,
     CrawlCommand,
     IControlChannel,
 )
@@ -104,6 +106,7 @@ class GraphSpider(BaseSpider):
         self._stop_mode = stop_mode
         self._stop_reason: str | None = None
         self._control_channel = control_channel or AsyncQueueControlChannel()
+        self._warned_unsupported_commands: set[CrawlCommand] = set()
 
         # Domain-граф — приватне поле, назовні не витікає (публічний API — DTO).
         if graph is not None:
@@ -745,52 +748,85 @@ class GraphSpider(BaseSpider):
         return self._control_channel
 
     def process_control_messages(self) -> str | None:
-        """
-        Обробити всі команди з control channel.
+        """Виконує всі команди з control channel; повертає причину зупинки, якщо прийшов ``STOP``.
 
-        Returns:
-            Причина зупинки якщо отримано команду STOP, інакше None
+        Команда поза ``SPIDER_COMMANDS`` або без обов'язкових полів у ``data`` не виконується
+        і дає WARNING (непідтримувана — раз на тип команди за краул).
         """
         while self._control_channel.has_pending():
             msg = self._control_channel.receive()
             if msg is None:
                 break
-
             logger.debug("Processing control message: %s from %s", msg.command.value, msg.sender)
+            if msg.command not in SPIDER_COMMANDS:
+                self._warn_unsupported_command(msg)
+                continue
+            stop_reason = self._execute_command(msg)
+            if stop_reason is not None:
+                return stop_reason
+        return None
 
-            if msg.command == CrawlCommand.STOP:
-                reason = msg.data.get("reason", "Stop requested via control channel")
-                logger.info("Stop command received: %s", reason)
-                return reason
-
-            elif msg.command == CrawlCommand.PAUSE:
-                self._state = CrawlerState.PAUSED
-                logger.info("Pause command received")
-
-            elif msg.command == CrawlCommand.RESUME:
-                if self._state == CrawlerState.PAUSED:
-                    self._state = CrawlerState.RUNNING
-                    logger.info("Resume command received")
-
-            elif msg.command == CrawlCommand.REPRIORITIZE:
-                url = msg.data.get("url")
-                priority = msg.data.get("priority")
-                if url and priority is not None:
-                    self._scheduler.reprioritize_url(url, priority)
-                    logger.debug("Reprioritized %s to %s", url, priority)
-
-            elif msg.command == CrawlCommand.FORCE_VISIT:
-                url = msg.data.get("url")
-                if url:
-                    # Додаємо URL в чергу з високим пріоритетом
-                    node_class = self.config.custom_node_class or Node
-                    node = node_class(url=url, depth=0, plugin_manager=self.node_plugin_manager)
-                    self._graph.add_node(node)
-                    self._scheduler.add_node(node, priority=100)
-                    logger.debug("Force visit scheduled: %s", url)
-
+    def _execute_command(self, msg: ControlMessage) -> str | None:
+        if msg.command == CrawlCommand.STOP:
+            reason = msg.data.get("reason", "Stop requested via control channel")
+            logger.info("Stop command received: %s", reason)
+            return reason
+        if msg.command == CrawlCommand.PAUSE:
+            self._state = CrawlerState.PAUSED
+            logger.info("Pause command received")
+        elif msg.command == CrawlCommand.RESUME:
+            if self._state == CrawlerState.PAUSED:
+                self._state = CrawlerState.RUNNING
+                logger.info("Resume command received")
+        elif msg.command == CrawlCommand.REPRIORITIZE:
+            self._reprioritize(msg)
+        elif msg.command == CrawlCommand.FORCE_VISIT:
+            self._force_visit(msg)
         return None
 
+    def _reprioritize(self, msg: ControlMessage) -> None:
+        url = msg.data.get("url")
+        priority = msg.data.get("priority")
+        if not url or priority is None:
+            self._warn_invalid_command(msg, "requires data['url'] and data['priority']")
+            return
+        self._scheduler.reprioritize_url(url, priority)
+        logger.debug("Reprioritized %s to %s", url, priority)
+
+    def _force_visit(self, msg: ControlMessage) -> None:
+        url = msg.data.get("url")
+        if not url:
+            self._warn_invalid_command(msg, "requires data['url']")
+            return
+        node_class = self.config.custom_node_class or Node
+        node = node_class(url=url, depth=0, plugin_manager=self.node_plugin_manager)
+        self._graph.add_node(node)
+        self._scheduler.add_node(node, priority=100)
+        logger.debug("Force visit scheduled: %s", url)
+
+    def _warn_unsupported_command(self, msg: ControlMessage) -> None:
+        if msg.command in self._warned_unsupported_commands:
+            logger.debug("Unsupported control command ignored: %s from %s", msg.command.value, msg.sender)
+            return
+        self._warned_unsupported_commands.add(msg.command)
+        supported = ", ".join(sorted(command.value for command in SPIDER_COMMANDS))
+        logger.warning(
+            "Unsupported control command %r from %s is ignored (supported: %s)",
+            msg.command.value,
+            msg.sender,
+            supported,
+        )
+
+    @staticmethod
+    def _warn_invalid_command(msg: ControlMessage, problem: str) -> None:
+        logger.warning(
+            "Control command %r from %s is ignored: %s, got %r",
+            msg.command.value,
+            msg.sender,
+            problem,
+            msg.data,
+        )
+
     async def check_control_channel(self) -> str | None:
         """Асинхронна обгортка над ``process_control_messages``."""
         return self.process_control_messages()